Changelog — DisPindable

Every meaningful change ships with a version bump and an entry here. The version next to the DisPindable logo top-left always links here. — view planned work →

v0.15.238

2026-06-02 CURRENT

Gap-fill now fetches only the missing pins, not every row. It used to walk every ID from 1 to the ceiling and skip the ones already cataloged — so for a nearly-complete source it churned through ~111k present rows to reach ~9k holes, and the progress bar read “1.8% (2,174 / 120,862)”. It now computes the missing IDs up front and iterates only those; the progress bar reads in real work (“59 / 9,612 gaps”, with the already-had count shown separately). Same fetch path and rate limits, just no busywork.
Gap-fill is now automated. The 12-hour catalog task, after refreshing ceilings and scanning new pins above the ceiling, now also fills a bounded batch of the holes below it (CATALOG_AUTO_GAP_FILL_BATCH, default 300 per source per cycle). Gaps are recomputed from the DB each cycle, so filled IDs drop out and the next cycle continues with the next holes — it converges on full coverage on its own, no babysitting. It runs the sources one at a time and SKIPS entirely while a forward crawl is running, so it never doubles the request rate against a source the crawler is already hitting. A large initial catch-up is still faster via a manual full gap-fill; this is steady background top-up. Controlled by CATALOG_AUTO_GAP_FILL (on by default).

v0.15.237

2026-06-01

Extended the false-block fix to PTDB. v0.15.236 corrected the error-page name check for Pin&Pop and PinPics, but PTDB’s fetcher lives in a separate module and still used the old check — so a real PTDB pin whose name contains a bare status-code number (e.g. “...(1502)”, “Castle 2504”, “Stitch 429”) could still be wrongly flagged as an error page and blocked. All three sources now share the same corrected matcher. PTDB’s genuine Cloudflare / “403” / “Access denied” pages are still detected.

v0.15.236

2026-06-01

Fixed real pins being wrongly flagged as Pin&Pop error pages and blocked. The error-page detector matched the bare text “403” anywhere in a pin’s name — so a legitimate pin like “Silver Mickey Mouse (1403)” (Pin&Pop 65644) tripped it, because “403” sits inside “1403”. The crawler logged it as BLOCKED, hit its block cap, and disabled the source. Pin names routinely embed model numbers like (1403), LE300, or 502nd, so this could mis-block many pins. The bare HTTP status codes (403/404/500/502/503/504/429) are now matched only when they stand alone as an error title or appear with an error word — never as digits inside a longer number. Genuine error pages (“403”, “404 Not Found”, “Error 503”, “Bad Gateway”, Cloudflare titles) are still caught. Same fix applied to the PinPics detector and the junk-named-rows admin scan.
Because the affected pins were blocked and never cached (detector returns “retry, don’t cache”), no bad rows were written to the catalog — a wrongly-blocked ID just retries and clears its failure-log row on the next successful fetch. If any older build did cache such a name, the junk-named-rows scanner no longer reports legitimate model-number names as junk.

v0.15.235

2026-06-01

Gap-fill / scan now stops at the site ceiling instead of marching into 404s above it. The gap-fill worker had no end-of-data limit — it scanned exactly the range it was handed — so a scan given a top above the source’s newest real pin (from a stale cursor, a high manual cap, or a post-crash state) walked through IDs the site hasn’t issued yet, logging a 404 for each. It now clamps the upper bound to the known site ceiling. The forward crawler already did this; gap-fill now matches.
Added “Clear 404s above ceiling” to the failure log. Cleans up the phantom not_found rows a too-high scan already recorded for IDs above the ceiling (e.g. Pin&Pop 120790–120819). Those aren’t real gaps — they’re IDs that don’t exist yet — so clearing them deflates the 404 count and stops “retry not_found” from re-walking them. Real gaps below the ceiling are kept, and sources without a known ceiling are left untouched.

v0.15.234

2026-06-01

The 12-hour scheduled check now also fetches newly released pins automatically. Until now the scheduler refreshed each source’s newest-pin ceiling every 12h but didn’t pull the new pins in — that needed the “Scan for new pins now” button or a running forward crawl. Now, whenever a 12h refresh finds a source’s ceiling has grown, it immediately scans just the newly-opened band (from the old ceiling up to the new one) and fetches those pins, so daily releases land in the catalog unattended even with the crawler stopped.
Safety rails on the auto-scan: it anchors on the previous ceiling (not the forward-crawl position), so a cold or far-behind crawler can’t trigger a huge sweep; it skips entirely when there’s no prior ceiling on record (first run — establish the baseline with a manual gap-fill); it skips and logs loudly if the band is unexpectedly wide (over 250 IDs, signalling an anomaly); and because the gap-fill engine handles one job at a time, it runs the sources in sequence and steps aside if a manual gap-fill is already running. Auto-scans show up on the gap-fill status page like manual ones. Controlled by CATALOG_AUTO_SCAN_NEW_PINS (on by default).

v0.15.233

2026-06-01

Fixed Pin&Pop’s newest-pin (ceiling) detection landing a few IDs short. Pin&Pop is a Vue app: in the raw page HTML the pins aren’t links yet — they’re a JSON list embedded in the page, and the links only appear after the page’s scripts run. The old detector only read the rendered links, so a fetch that grabbed the page a moment too early could miss the very newest pins (it reported 120,785 when the real newest was 120,788). It now reads the embedded JSON directly — the authoritative, render-independent list — and only falls back to the rendered links if that JSON isn’t present.
Gap-fill’s source dropdown now shows each source’s true ceiling (the higher of the listing-page max and how far the forward crawl has reached), relabeled “max id”. It previously showed only the forward-crawl high-water mark, which could read lower than the newest pin and made it look like a blank gap-fill would stop short. The blank-bound gap-fill already scanned to the real ceiling; this just makes the label honest.

v0.15.232

2026-05-31

Split the failure-log “Retry” button into separate buttons for errored, blocked, and missing (not_found). The single button re-queued every outcome at once, so retrying a few transient errors also re-walked thousands of not_found 404s — which are generally real gaps in the ID space and almost never change on a retry. Each outcome now has its own button showing its count, so you can retry the 29 errored PTDB rows without touching the 2,394 missing ones. The “missing” button is styled as secondary and its confirm dialog notes it’s rarely worth running. (The retry endpoint already accepted a per-outcome filter; this just exposes it.)

v0.15.231

2026-05-31

Fixed PTDB site-ceiling lookups failing on a Cloudflare hiccup. The PTDB listing page (pinList.php) intermittently returns a Cloudflare “522 connection timed out” page because the big newest-first sort is slow at the origin — it loads fine in a browser on a retry. Two problems made that fatal: the headless fallback waited only for “page has a body”, which the error page also has, so it grabbed the ~7KB error page and treated it as the listing; and nothing retried the timeout. Now the refresh retries the transient error, the headless fallback waits for an actual pin tile to appear, and a page is only accepted once a real pin ID can be read from it — so an error page is never mistaken for an empty listing. Once a good fetch lands, the PTDB ceiling populates (~59,957), the crawler stops at the real end, and “Scan for new pins now” works.

v0.15.230

2026-05-31

Fixed the PTDB crawler running past the real end of the catalog. When a source’s site-ceiling lookup never succeeds (PTDB’s listing page is Cloudflare-blocked), known_max_id stays empty, and the end-of-data guard was treating “no ceiling” as “no limit” — so the crawler kept fetching IDs well past the real top (~59,957), churning certain-404s toward 100k. Added a per-source fallback ceiling so a source with no discovered ceiling now pauses just above its estimated top instead of running unbounded. A real discovered ceiling still always wins over the estimate.
Fixed the catalog-mirror progress bar showing a wildly low percentage for PTDB. It was dividing by the highest ID ever cached (~100,481, inflated by old test/stale rows above the real top) instead of the real ceiling, so it read ~58% complete when the PTDB mirror is actually ~97% complete. The bar now divides by the discovered ceiling (or the estimate when none is known yet) and ignores the stale high-water mark.

v0.15.229

2026-05-31

Fixed PTDB site-ceiling detection returning nothing — it was matching zero pin links on the live listing page. PinTradingDB’s pinList.php links each pin with a relative URL (pin/59957, no leading slash), but the extractor required a leading slash and so found no IDs at all. Result: “Refresh site max IDs” and “Scan for new pins now” both failed for PTDB with “ceiling refresh failed”, while PinPics and Pin&Pop worked. Now matches the relative form too; PinPics and Pin&Pop extraction is unchanged.
Corrected a misconception in the ceiling code: a site’s displayed pin count is not its max pin ID. Deleted and reserved IDs leave gaps, so the real ceiling sits above the count (PTDB shows 57,553 pins but the newest is pin 59,957; PinPics 148,398 vs ~187,462; Pin&Pop 111,285 vs ~120,723). Matching the count instead of the true max would have stranded ~2,400 of the newest PTDB pins below the end-of-data cutoff.

v0.15.228

2026-05-31

Added a “Scan for new pins now” button — grabs freshly-released pins on demand instead of waiting for the 12h refresh. For the chosen source it refreshes the site ceiling (known_max_id), then runs a gap-fill over ONLY the newly-opened band — the IDs above where the catalog currently ends (max_id_seen) up to the new ceiling. Reuses the existing gap-fill engine, so same rate limits and same fetch path; it just scans the narrow new-release range instead of the whole catalog.
Anchored the scan at max_id_seen rather than the forward cursor. When a source is caught up, the cursor advances into the +50 certain-404 buffer past the real max, so anchoring on it would have skipped the very pins released in that band. New pins always appear just above max_id_seen.
Taught the default gap-fill ceiling to prefer known_max_id (the listing-page ceiling added in 0.15.227) over max_id_seen, so a no-argument gap-fill can now reach pins above the forward cursor. Falls back to max_id_seen when the ceiling hasn't been refreshed yet.
If the ceiling refresh fails (listing-page fetch or regex miss), the scan refuses to start and the flash asks for the listing-page HTML so the extraction regex can be tuned. The full-range gap-fill still covers holes below the crawl frontier; this button is only the new-release band.

v0.15.227

2026-05-31

End-of-data detection: crawler now knows when it's past the last valid pin ID and stops hammering certain-404 territory. Drew confirmed all three sources behave differently past their ID ceiling: PinPics serves /pin/189000 as a 200-status “could not locate” page (caught by v0.15.225 markers); Pin&Pop serves /pins/120724 as a clean HTTP 404 (already handled); PTDB serves /pin/59999 as a fully-rendered EMPTY TEMPLATE — 200 status, all data fields blank or placeholder (name=“_”, release date = today, all $0.00 prices, “(0 users)” everywhere). The v0.15.225 empty-parse guard didn't catch PTDB because name=“_” is technically non-empty.

New module catalog_listing_pages.py: fetches each site's listing page (pinpics.com/pins/, pinandpop.com/pins, pintradingdb.com/pinList.php), extracts every /pin/N or /pins/N link href, takes max. Result is the authoritative upper bound on valid pin IDs — known_max_id in crawler state.

Crawler loop guard: before each fetch, checks if current_id > known_max_id + 50 (50-pin staleness buffer). If yes, the source pauses with stop_reason=“at end of data”, sleeps 5 min, re-checks. Does NOT disable the source — when the next scheduler refresh discovers new pin IDs and grows known_max_id, the very next iteration finds headroom and resumes automatically. This is what implements Drew's “2x/day check from last marker” design.

PTDB empty-template detection: in ptdb_sites.fetch_ptdb, after the parser produces a result, catalog_listing_pages.looks_like_ptdb_empty_template checks for the specific pattern. Three STRONG signals (any one trips): name is literally “_” / “-” / “—”; OR release date is today AND no images; OR 3+ “(0 users)” markers AND no images. Plus weak-signal scoring gated by “no images” as a hard precondition — real pins always have images, so an absent image is definitive. Verified with 7 test cases including borderline real pins (name+image+no metadata) which correctly DON'T flag.

Scheduler integration: added _check_catalog_max_ids() to the scheduler loop. Self-gated at 12-hour cadence (configurable via CATALOG_MAX_ID_REFRESH_HOURS constant). Failures (CF block, network) are silent — existing known_max_id stays in place, next tick retries. First scheduler tick after startup runs after the existing 2-minute initial-delay, so a fresh deploy picks up max-id bounds within ~2 minutes.

UI:
· Per-source card now shows “site max ID: X · N pins ahead” or “⏸ at end of data” when the cursor has caught up. ⚠ stale indicator fires if the last refresh was >24h ago.
· New “Refresh site max IDs” button in the admin actions row triggers immediate refresh without waiting for the next scheduler tick.
· POST /catalog-crawler/refresh-max-ids is the manual-trigger endpoint.

Sandbox limitation: Anthropic's sandbox can't reach pinpics.com / pintradingdb.com / pinandpop.com (network allowlist excludes them), so the listing-page extraction code is defensive best-effort based on each site's URL conventions. Verification must happen on Drew's network — after deploy, click “Refresh site max IDs” and check that known_max_id populates for each source. If extraction fails for any site, the regex patterns in _extract_max_id need tuning to that site's actual HTML structure.

Deferred (in BACKLOG.md): Drew's disneypinblog + WDWNT cross-reference idea. Different architectural shape from the catalog crawlers (content sites with article structure vs. pin DBs with numeric ID spaces). Documented as a ~2-3 day design sketch with the right model being a “pin-mention aggregator” polling RSS feeds, extracting pin names from blog post text, and cross-referencing against the catalog by fuzzy name match. Deferred until catalog mirror reaches 80%+ across all three sources — currently PinPics is at 19.95%, so the fundamental crawl-completeness work is still the higher-value lever. Surfaced here as a known next-layer feature.

v0.15.226

2026-05-30

Stopped feeding the breaker BOT_DETECT on every intermediate CF-403 that headless then recovers. Drew spotted this immediately after the v0.15.225 deploy: even with the 404-misclassification cascade fixed, the breaker was STILL tripping during normal scans — this time on the rolling-window failure-rate check after 4 consecutive successful fetches. The log told the whole story:

[pinpics] 45087: requests status=403, len=33992
[pinpics] 45087: got content via headless (36608 bytes)
[pinpics] 45088: requests status=403, len=33992
[pinpics] 45088: got content via headless (36619 bytes)
... 4x in a row ...
[breaker:pinpics] OPEN (rolling window failure rate ≥ 50%)

Root cause: PinPics's normal fetch flow is requests → CF returns 403 → headless solves the challenge and gets the real page. The headless fallback exists specifically for this. But my v0.15.224 wiring recorded BOT_DETECT on the requests-path 403 BEFORE the headless fallback ran. Every successfully-recovered fetch was logging a fake bot-detection signal to the breaker. After 4 in a row (rolling window had 4 BOT_DETECT + 1 SUCCESS = 80% failure rate), the breaker tripped — even though every single fetch had actually succeeded.

Fix: the breaker should only see the FINAL outcome of the entire fetch_pinpics call, not intermediate failures that have a fallback. Removed the two intermediate _brk.record() calls from the requests-path branch (non-200 status + RequestException). Added one new record at the “both paths failed” return point (when html is None after requests AND headless both fail) that classifies the outcome from fetch_errors: CF/challenge/403 → BOT_DETECT, 5xx → SERVER_ERROR, 429 → RATE_LIMIT, timeout → TIMEOUT, otherwise CONNECTION. Also added breaker records to two paths that previously had none: the maintenance-page guard (SERVER_ERROR — real site-down signal) and the headless-CF-challenge detection (BOT_DETECT — both paths effectively blocked).

Design principle (carry forward): record the outcome of the ENTIRE fetch call, not the outcome of intermediate steps. For sources with multiple fetch paths (PinPics has requests+headless, PTDB has the same inside _fetch_html), the breaker must see only the union of those paths, not each path individually. Pin&Pop is unaffected — it has a single requests path, so the intermediate record IS the final outcome there.

Expected behavior after deploy: normal PinPics scans (requests 403 → headless recovers) produce only SUCCESS or NOT_FOUND records to the breaker. Failure rate stays at 0% during healthy operation. The breaker only trips when BOTH paths fail for 5+ consecutive pins or sustained ≥ 50% failure rate over the rolling window — the actual site-outage signals it was designed for.

v0.15.225

2026-05-30

Fixed the “legitimate ID-space gap trips the circuit breaker” cascade Drew spotted in the v0.15.224 deploy. The breaker was correctly tripping after 5 consecutive failures, but the failures weren't real — PinPics serves its “missing pin” page as HTTP 200 with full chrome (logo, fonts, social icons, ~30KB body) and the wording isn't what the existing not_found_markers list expected. Net result: a stretch of legitimately-deleted pin IDs around 45,000 produced fake successes (cached with name = “Sorry, you do not have permission for that!”), got reclassified as errored by other code paths, tripped the breaker, and every subsequent crawler attempt during the 5-min breaker window logged as “errored breaker OPEN” — 17 spurious rows in image 1.

The real bug: PinPics changed their 404-page wording. Old list looked for “Sorry, we could not find that”. Current page says “We could not locate the item you are trying to view” in the body with title “Sorry, you do not have permission for that!” and error code “2-PINS-04”. Added all three to the marker list so the existing 404 detection path catches them and returns a stable 404 cached result. Confirmed via the actual HTML Drew pasted (pin/45076 response). Going forward, missing pins are correctly classified as not_found and the breaker is fed NOT_FOUND (ignored for health).

Breaker outcome wiring — v0.15.224 gaps: the v0.15.224 wiring missed two PinPics paths. The not_found_markers early-return and the empty-parse 404 early-return both bypassed the breaker entirely. So when a real 404 was detected, the breaker saw no record at all — not a SUCCESS, not a NOT_FOUND. Combined with other 5xx outcomes around it, the breaker had effectively zero good signal in the rolling window. Both paths now record NOT_FOUND explicitly. NOT_FOUND is in the ignored set, so it correctly says nothing about site health without polluting the failure-rate denominator either.

Breaker short-circuit log-spam fix. Pre-v0.15.225, when the crawler called fetch_X while the breaker was OPEN, the fetch returned {ok: False, _transient: True, error: ‘X breaker OPEN; next probe in Ns’}. The crawler's outcome classifier in _is_block_error / _is_not_found had no match for “OPEN”, so the row fell through to errored — same outcome bucket as a real network failure. During a single 5-min breaker window the crawler would log hundreds of these (every loop iteration, every pin ID). Added an explicit “breaker-blocked” branch at the TOP of categorization: detects the short-circuit, sleeps the breaker's recommended interval (capped at 30s so status=stopped still applies), doesn't advance current_id (pin will retry once the breaker closes), doesn't bump requests_this_run (we made no outbound HTTP), doesn't write to the failure log. Quiet pause, then resumption.

Probe targets tightened. Old probes GET'd pinpics.com/pin/1, pinandpop.com/pins/1, and pintradingdb.com/pin/35868 with the source name as marker. Two problems: (a) any of those pin IDs could be deleted upstream, making the probe permanently return a 404 page; (b) more critically, PinPics's “missing pin” 404 page contains the word “pinpics” in nav and footer chrome, so the probe marker matched and the breaker falsely CLOSED on a 404 page. New probes target stable listing/index pages (/pins/ for PinPics, homepage for the other two) with markers that appear ONLY on real pages (“Browse Pins”, “Disney pin”, “Pin Trading Database”).

JUNK_NAME_MARKERS extended. Added “do not have permission for that” and “could not locate the item” so the existing /catalog-crawler/junk-named-rows admin page surfaces any DB rows that were already cached with the junk name before this fix. Drew can click Reset on that marker group to null out hero/name and force re-fetch — the new code path will then correctly classify them as 404 cache entries.

Post-deploy cleanup steps:
1. Visit /catalog-crawler/junk-named-rows — expect a “permission for that” / “could not locate” marker group. Reset it.
2. Visit /catalog-crawler?failures=1&failures_source=pinpics&failures_outcome=errored — click “clear errored” to wipe the 17+ spurious “breaker OPEN” rows that the cascade logged.
3. Resume the crawl. The breaker won't trip on ID-space gaps anymore; if it does trip, that's a real 5xx storm worth investigating.

v0.15.224

2026-05-29

Plugged the “403 Forbidden” / “Bad Gateway” DB-pollution leak; added per-outcome failure-log cleanup; made dashboard counts clickable; shipped a site-breaker module. Drew spotted PTDB catalog rows literally named “403 Forbidden” piling up in a contiguous ID block (PTDB35854-35870+). Same vector for any future PinPics “Bad gateway Error code 502” title that parses past the upstream 5xx guard.

Root cause. The v0.15.222 error-page name guard does a substring match against JUNK_NAME_MARKERS. The tuple covered 503 / 500 / 404 / CF-block titles but had no entries for 403, “forbidden”, or 502 / “bad gateway”. PTDB's bot-detect page returns a parseable <title>403 Forbidden</title> with no image — has_name=True, has_images=False — so the empty-parse 404 guard missed it, the marker guard missed it, and the parser happily wrote “403 Forbidden” to the DB.

Fix. Added 403 / forbidden / 502 / bad gateway / 504 / 429 / cloudflare variants to JUNK_NAME_MARKERS. Existing PTDB v0.15.222 guard now refuses to cache and returns _transient=True so the next pass retries. Existing junk-named-rows scanner now surfaces “403 forbidden” as a marker group on /catalog-crawler/junk-named-rows so the poisoned rows already in the DB can be reset in one click.

Parity fix: the same name guard now runs on the PinPics fetch path (previously only PTDB and Pin&Pop had it). Defense in depth against any future outage HTML that has both a parseable title AND a parseable image (CF interstitial with logo, soft error pages, etc.).

Per-outcome cleanup. clear_failure_log now accepts an optional outcome filter. The failure-log UI on /catalog-crawler?failures=1 grew a Cleanup panel with per-source rows: clear blocked, clear errored, clear transient (blocked + errored together), and a de-emphasized clear not_found (red-flagged in the confirm dialog because those are usually real ID-space gaps). Drew's case: PTDB has 1,467 stable 404s that should never be re-crawled, but he wants to retry blocked + errored failures after a bot-detect fix. The old “Clear failure log” button wiped everything.

Clickable counts. The four count badges on each per-source status card (cached / 404 / blocked / errored) are now hyperlinks. Cached jumps to /catalog/browse?source=X; the failure-class counts jump to the failure log filtered by (source, outcome). Saves the operator three clicks every time they want to inspect a count that ticked up.

Site-breaker module (new site_breaker.py). Per-source circuit breaker with classic CLOSED / OPEN / HALF_OPEN states. Trips on 5 consecutive SERVER_ERROR / CONNECTION / TIMEOUT / BOT_DETECT outcomes OR ≥50% failure rate over a rolling 20-request, 5-minute window. Backoff starts at 5 min, doubles on each failed probe, caps at 1 hour, resets on success. Probe-based recovery: GETs a stable known URL and verifies status==200 AND marker string in body AND (if configured) image probe returns >=1 KB of image/* content. Catches CF error pages that return 200 with full HTML — a status-only check would close the breaker on a CF challenge and re-trip on the very next real request.

Not yet wired into the fetch paths in v0.15.224 — the existing last_blocked_until 30-min block-pause still handles bot-detect blocks. The breaker runs alongside as the catch-all for the 5xx-storm patterns the old logic misses (PinPics 502 hammering, Pin&Pop SSL EOF bursts). Integration plan documented in HANDOFF.md; the module is self-contained and unit-smoked. See get_breaker(name) and brk.record(Outcome.X) for the integration surface.

Fetch-path wiring (followup pass). The breaker is now actually called from all three fetch entry points: catalog_sites.fetch_pinandpop, catalog_sites.fetch_pinpics, and ptdb_sites.fetch_ptdb. At each, allow() runs before any HTTP and short-circuits with a transient error if the breaker is OPEN. After each call, record() feeds the outcome — mapping 5xx to SERVER_ERROR, 403 to BOT_DETECT, 429 to RATE_LIMIT, SSL/timeout to CONNECTION/TIMEOUT, 404 to NOT_FOUND (ignored), junk-name guard hits to BOT_DETECT (the 200-status-but-CF-block case the breaker exists to catch). Every record call is wrapped try/except so a breaker bug can never block a real fetch — worst case the breaker silently disables.

Verified with 7 end-to-end behavior scenarios mirroring Drew's actual failure patterns: PinPics 502 storm trips after 5 consec; PTDB 403 storm trips on BOT_DETECT; Pin&Pop SSL EOF burst trips on CONNECTION; 404 ID-space gaps never trip (correct); 50/50 alternating success/failure trips via rolling-window rate (consecutive streak never builds, window rate fires at 10/20 = 50%); single 429 is soft, 3 consecutive 429s trip; failed-probe backoff doubles 300→600→1200→2400→3600 then caps at 1hr.

Admin surface (new). GET /catalog-crawler/breakers returns a JSON snapshot of all per-source breaker state (state, trip count, next-probe ETA, rolling-window stats, last probe result). POST /catalog-crawler/breakers/{source}/close and /open let the operator force state if the auto-recovery is stuck or you want to halt a source you know is unhealthy. The dashboard renders a breaker panel inline on each source card: red OPEN banner with next-probe countdown + force-close button when tripped; subtle “closed · N consec fails · window X/Y” indicator when there's failure pressure but the breaker hasn't tripped yet.

PTDB 403 diagnostic + curl_cffi (new). scripts/test_ptdb_fetch.py systematically sweeps every known technique against PTDB's 403 to find one that works from Drew's network: plain requests baseline, requests + full browser headers (Referer + Sec-Fetch-*), curl_cffi with each impersonation profile (chrome131/120/116, safari17_0, firefox133, chrome131_android, edge101), curl_cffi + headers, curl_cffi sessioned (warmup homepage → pin), Playwright stealth, and curl_cffi + manually-captured cf_clearance cookie. Run with python scripts/test_ptdb_fetch.py; outputs a table per technique showing status code + body bytes + whether the expected marker (“101 Dalmatians” from pin/35868) was found. Winner gets wired as the primary fetch path in the next release.

curl_cffi added to requirements.txt as an optional dep (commented). Bypasses Cloudflare bot-detection at the TLS layer by impersonating real Chrome/Safari/Firefox JA3 handshake signatures — works where stock requests and even Playwright-stealth fail because TLS fingerprint differs from a real browser at a layer above HTTP. Diagnostic from Anthropic's sandbox showed every profile returning 403 even on the homepage with 21-byte bodies, indicating the sandbox IP is hard-blocked — the diagnostic must run from Drew's home network to be representative.

Counter consistency caveat. The per-source dashboard count badges (cached / 404 / blocked / errored) come from the in-memory crawl state counters. When you click one and jump to the failure-log filtered view, the rows shown come from the CatalogCrawlFailure DB table. These are different sources of truth and can disagree (e.g. crawler restart preserves the DB rows but state counters reset; a successful retry-failure flow deletes DB rows but the in-memory counter for the original outcome is still incremented). Not new in v0.15.224 — the filter view existed pre-this-version; the clickable links just expose the discrepancy more. If you see “errored: 4,084” on the card but the filtered view shows a different number, that's why.

PTDB 403 diagnosis. Drew confirmed pintradingdb.com/pin/35868 loads in his browser at the same IP, with the full pin record. The scraper sees 403 across every UA / Chrome 127 stealth headless attempt. This is bot detection, not IP block — almost certainly JA3 / TLS fingerprint or HTTP/2 header-order detection that pure User-Agent spoofing can't defeat. Investigation candidates listed in BACKLOG: curl_cffi for JA3 mimicry, undetected-chromedriver, manual cf_clearance cookie import, slower request cadence.

v0.15.223

2026-05-28

Junk-name purge now actually clears the name — scan no longer keeps reporting reset rows. Drew ran Purge All repeatedly and the scan kept showing the same 2,547 junk rows. The logs proved the reset WAS running (matched=1878, rows_reset=1878 then matched=669, rows_reset=669) — but the very next scan re-counted all 2,547.

Cause. reset_junk_named_rows cleared the hero image and the cache file but deliberately left the junk NAME in place (v0.15.220 cited a NOT NULL worry). The scan matches on name. So a ‘reset’ row still had “PinPics is currently unavailable” as its name and kept matching the scan — making it look like purge did nothing, even though the rows were genuinely cleared and ready for re-fetch.

Fix. CatalogPin.name is actually nullable, so the NOT NULL worry was unfounded. Reset now replaces the junk name with the placeholder “(pending re-fetch)” — clear in the UI, doesn't match any marker, and gets overwritten when the crawler re-fetches. After this, a purge immediately drops the count to zero on re-scan.

Note on re-fetch timing. Reset rows have their cache cleared, but the crawler is cursor-based — it re-fetches them when it next passes through their ID range, not instantly. The (pending re-fetch) placeholder makes it easy to see which rows are still awaiting that pass. caches_deleted=0 in the logs is expected for rows poisoned after v0.15.220 (the maintenance guard stopped writing their cache in the first place, so there's no file to delete).

v0.15.222

2026-05-28

Error-page guards extended to PTDB + Pin&Pop; full PinPics metadata capture.

1. Guard parity across all three sources. The error-page name detection from v0.15.220/221 (PinPics) now also runs for PTDB and Pin&Pop. All three share the JUNK_NAME_MARKERS list, so a maintenance page, 500/503 error, or Cloudflare block that parses with a junk title gets caught and NOT cached regardless of which source served it. PTDB already had CF-block detection; this adds the generic error-page markers on top. Pin&Pop already returned early on non-200, so this guards the 200-with-error-content edge case.

2. Full PinPics metadata. Drew asked for everything on the PinPics page except Owns/Wants/Trades counts. New parser _parse_pinpics_metadata extracts the Pin Information sidebar + Front Description:<ul><li>Origin(s) — WDW / DLR / etc.</li><li>Edition + parsed edition size (“LE 1000” → 1000)</li><li>Year on Back (distinct from the release year)</li><li>Original Price (numeric, from “$64.99/box of 3”)</li><li>SKU</li><li>Dimensions</li><li>Glitter/Shimmery + Pin-on-Pin flags (detected via the green checkmark)</li><li>Front Description (the full paragraph)</li></ul>Most map onto existing catalog columns (edition, edition_size, origin, original_price, sku, description). Five new columns added: dimensions, front_description, year_on_back, glitter_shimmery, pin_on_pin. All shown in the Catalog details panel on the pin detail page.

Backfill existing rows. The new fields are populated on every fresh crawl, but existing rows can be backfilled from their cached metadata_json WITHOUT re-scraping — the /admin/backfill-catalog-columns button replays cached scrape blobs through the same upsert path. (Note: rows crawled before v0.15.222 won't have the new fields in their metadata_json, so those specifically need a re-crawl to capture dimensions/flags/front-description. The price/edition/SKU/origin fields backfill fine since they were already in older blobs via the Pin&Pop parser.)

v0.15.221

2026-05-28

Caught a second PinPics error-page variant + added a status-based backstop. After v0.15.220 caught the 503 “currently unavailable” maintenance page, Drew found another poisoned row: “An error occurred (500 Error)”, year 2024. Same poison pattern, different PinPics failure mode (500 instead of 503).

1. More markers. Added “an error occurred”, “500 error”, “internal server error”, “something went wrong” to both the fetch-time guard and the junk-name scan list.

2. Status-based backstop (the real fix). Chasing individual error strings is whack-a-mole — PinPics can reword its error pages any time. So fetch_pinpics now tracks whether the requests attempt saw a 5xx status. If it did AND no image was parsed, the row is treated as a transient error and NOT cached, regardless of what the name says. Requires both conditions (5xx + no image) so a legitimately image-less pin fetched cleanly with a 200 isn't falsely flagged. This catches future error-page wordings we haven't seen yet.

Cleanup: the “500 Error” row (and any siblings) will now show up in /catalog-crawler/junk-named-rows under the “an error occurred” or “500 error” marker. Reset them there; they'll refetch cleanly now that PinPics is back.

v0.15.220

2026-05-28

PinPics outage poisoned ~150 rows; added detection + cleanup. PinPics went down for maintenance overnight (Cloudflare 503 serving a page titled “PinPics is currently unavailable”). The crawler's headless fallback fetched that page successfully (~20KB), the name parser extracted the maintenance title as if it were a pin name, and rows 26718-26869 got cached as ok=True with name=“PinPics is currently unavailable”, year 2009, no image. They render as junk and the crawler won't refetch them because the cache looks healthy.

1. Prevention. fetch_pinpics now checks the parsed name against maintenance/outage markers (“currently unavailable”, “under maintenance”, “service unavailable”, etc.) BEFORE caching. On a match it returns a transient error and does NOT write the cache file — so the next pass retries instead of locking in junk. The existing empty-parse 404 guard didn't catch this because the maintenance page HAS a name.

2. Cleanup tooling. New /catalog-crawler/junk-named-rows page scans every catalog row for names matching outage/error markers (maintenance pages, Cloudflare “Access denied”, “Just a moment” challenge titles, “page not found”, etc.) and groups them by marker with per-group and purge-all reset buttons. This complements the dup-hero scan: that one finds rows sharing a duplicate IMAGE, this one finds rows with a junk NAME and usually no image at all — the case the hero scan structurally can't see.

How to clean up last night's damage: go to /catalog-crawler/junk-named-rows, you'll see a group like “~150 rows match ‘currently unavailable’”. Click Reset (or Purge all). The crawler refetches them on its next pass — and now that PinPics is back up, they'll get real data.

v0.15.219

2026-05-28

VPN gate now matches the adapter DESCRIPTION, not just name/IP. Drew compared his interfaces with PIA on vs off and spotted the problem with the v0.15.215 default: the IP changes every connection (10.156.x.x one session, 10.94.x.x the next), so 10.156 was a bad qualifier. The stable signal is the adapter description — PIA's tunnel is always described “Private Internet Access Network Adapter” regardless of IP, and goes down when the VPN disconnects.

The catch. psutil keys interfaces by the generic Windows friendly-name (“Local Area Connection” for PIA), and doesn't expose the hardware description at all. So v0.15.219 adds a WMI lookup (Win32_NetworkAdapter, mapping NetConnectionID → Name) that enriches each interface with its description. The gate matcher now searches name + description + IP addresses, all case-insensitive.

New default qualifier. The baked-in launcher default changed from 10.156 (the fragile IP) to Private Internet Access (the stable description). Survives PIA IP rotation, survives reboots, no manual config. Override via NSSM as before.

Diagnostic page (/admin/network-interfaces) gains a Description column so you can see exactly what string to match, plus updated guidance explaining why description > IP > name for VPN detection.

Setup guide rewritten copy-paste-first. docs/multi-machine-setup.md is now organized as labeled parts (A: Postgres+crawler machine, B: web machine prep, C: migrate, D: image sharing, E: configure+start, F: verify) with complete command blocks and a clear PLACEHOLDER legend at the top. Every command is a full paste-ready block instead of prose-interspersed snippets.

v0.15.218

2026-05-28

Multi-machine topology enabled. Drew's main box keeps crashing despite RAM and software workarounds; WHEA-18 with the CPU at 10-15% utilization is hardware (PSU/VRM/CPU silicon). Software can't fix that, but it can get the workload off the bad machine. He has a second computer on the same LAN. This release ships the three things that enable a web-on-#1 / crawler-on-#2 split:<ul><li>CRAWLER_DISABLED=1 env var. Set via NSSM AppEnvironmentExtra on the web-only machine. Both ensure_crawler_running() and the gated ensure_crawler_running_with_gate() early-return without starting any crawler threads. Loud startup log: [crawler] CRAWLER_DISABLED is set; skipping the VPN gate AND the crawler. Default behavior unchanged when unset.</li><li>migrate_sqlite_to_postgres.py. Standalone migration script. Reads source SQLite, creates schema in target Postgres via SQLAlchemy Base.metadata.create_all, copies all 20 tables in 1000-row chunks, runs setval() on every auto-increment sequence so the next INSERT doesn't collide with existing IDs, verifies per-table row counts. Refuses to run against a non-empty target unless explicitly overridden — protects against accidental double-insert.</li><li>docs/multi-machine-setup.md. Complete walkthrough: install Postgres on machine #2, create the DB user, allow LAN connections, run the migration, SMB-share catalog_uploads/ from the crawler machine to the web machine, set DATABASE_URL + CRAWLER_DISABLED env vars via NSSM, verify the split. Includes rollback instructions for the case where something goes wrong.</li></ul>Bonus fix: cpu% column now actually shows real values. v0.15.196's telemetry created a fresh psutil.Process() each heartbeat. cpu_percent(interval=None) returns 0 on the first call against any Process instance because it has no baseline to compare against. So the cpu% column was always 0 (rendered as empty by the {{ r.cpu_pct or "" }} idiom). Module-level cache fixes this: one Process ref lives for the lifetime of the heartbeat thread, and cpu_percent() returns elapsed-since-last-call — what we actually wanted. Drew noticed when his v0.15.217 telemetry data showed every cpu% cell empty.

What this doesn't do. Doesn't fix Drew's hardware. Doesn't auto-failover if Postgres dies. Doesn't replicate. It's a deliberate, simple split — one DB, two services, shared image storage. If you want HA later that's a separate exercise.

v0.15.217

2026-05-27

Hotfix: /admin/telemetry 500'd on records without temperature data. My v0.15.216 template added a temperature column with severity-based color, but the conditional used is not none — which is True for Jinja's Undefined, the value you get when a dict key is missing. Records from before v0.15.216 (and v0.15.216 records on machines where the WMI temp read returned nothing) didn't have the cpu_temp_max_c key at all, so Undefined >= 92 raised TypeError and everyone got a 500.

Fix. Guard with is defined and is not none — the first check catches missing keys, the second catches explicit None values. Verified by isolated-render test against six cases (missing field, explicit None, cool, warm, hot, danger); all render cleanly.

Lesson learned. When templates need to do arithmetic on optional dict fields, default to is defined not is not none unless you can guarantee the field is always present. Cheap mistake; expensive symptom (every refresh of the diagnostics page broken).

v0.15.216

2026-05-27

CPU temperature in telemetry + thermal-aware crawler pause. Drew's machine keeps crashing despite the RAM-to-JEDEC change. WHEA-18 bus/interconnect errors on a CPU that's otherwise stable strongly suggest thermal or VRM stress during sustained crawler load. Software can't fix the underlying problem, but it can (a) measure it, (b) actively back off when temps get dangerous — potentially saving Drew from another crash mid-crawl.

Temperature reading. telemetry.py now records CPU temp + frequency in every 30s heartbeat. Two-stage approach: try psutil.sensors_temperatures() first (works on Linux + some Windows), fall back to WMI MSAcpi_ThermalZoneTemperature via PowerShell (works on every Windows box with ACPI, which is all of them post-XP). Reports max-core, average, and core count.

Thermal-aware pause. The crawler loop now checks the latest telemetry temperature before each polite sleep. If CPU is at or above CRAWLER_THERMAL_WARN_C (default 85°C), the crawler PAUSES until it cools to CRAWLER_THERMAL_RESUME_C (default 78°C). Hysteresis prevents yo-yo behavior. Max single pause caps at CRAWLER_THERMAL_MAX_PAUSE_S (default 600s) so a stuck-hot CPU doesn't hang the crawler forever. Each pause logs the entry temp, the cool-down progress, and the exit reason.

Disable knob. Set CRAWLER_THERMAL_DISABLED=1 via NSSM AppEnvironmentExtra to opt out (useful for machines that don't have visible sensors and where the periodic check itself is noise). Default behavior: enabled with the thresholds above — safe-by-default for Drew.

Telemetry page additions. Two new columns — temp °C and freq MHz. Temp cell color-codes by severity: yellow at 75°C+, orange at 85°C+ (warn threshold), red at 92°C+ (near-thermal-limit territory). Empty cells mean the platform/drivers didn't expose a sensor — expected on some Windows machines.

What this won't fix. If the crash is VRM voltage droop rather than CPU temperature, this gives you visibility but not protection — temp readings might look fine right up until the crash. If you see temps consistently low (sub-80°C) and the crashes continue, the next suspect is power delivery (VRM ripple/spike) rather than thermals.

v0.15.215

2026-05-27

Crawler-gate defaults baked into the launcher. Drew pointed out the obvious-in-hindsight problem with v0.15.213-214: if his machine crashes unattended at 3am, nobody is around to type the nssm set ... +CRAWLER_WAIT_FOR_INTERFACE=10.156 command. The whole point of the gate is unattended boot.

Fix. service-launch.ps1 now supplies a default CRAWLER_WAIT_FOR_INTERFACE=10.156 if NSSM's AppEnvironmentExtra didn't override it. Drew gets the gate behavior automatically on every boot, across every crash-recovery cycle, without any action. Logs the value at startup so it's clear what's in effect.

How to override. If PIA ever rotates Drew's tunnel to a non-10.156 range:<pre style="margin: 0.3rem 0; padding: 0.4rem 0.6rem; background: rgba(127,127,127,0.1); border-radius: 4px; font-size: 0.85rem;">nssm set DisPindable AppEnvironmentExtra +CRAWLER_WAIT_FOR_INTERFACE=10.x nssm restart DisPindable</pre>The NSSM-set value persists across crashes and reboots; this is a one-time configuration, not a per-boot ritual.

Timeout dump. If the 10-minute gate poll ever times out (e.g., PIA placed Drew on a different IP range), the log now dumps every up interface and its addresses so the next operator can see at a glance what to reconfigure. Without this, the previous timeout message was “giving up” with no breadcrumb to follow.

v0.15.214

2026-05-27

VPN gate now matches by IP address too, not just interface name. Drew checked his interfaces with ipconfig /all and his Private Internet Access tunnel showed up as the generic-named “Local Area Connection” (Windows can't classify PIA's adapter type, so it falls back to a meaningless friendly name). The v0.15.213 gate would only match against that name — technically workable in Drew's case since the string was unique on his system, but a fragile choice that wouldn't survive a second similarly-named adapter ever appearing.

Fix. The substring matcher now searches against both the interface name AND each of its assigned IPv4/IPv6 addresses. So CRAWLER_WAIT_FOR_INTERFACE=10.156 matches Drew's PIA tunnel (which assigned him 10.156.3.222) by IP prefix — way more distinctive and harder to false-positive than “Local Area Connection”.

Diagnostic page updated with a PIA-specific example at the top of the substring-picking list, plus a note that IP-prefix matching is supported.

v0.15.213

2026-05-27

Crawler startup-gate: optional delay + VPN-interface wait. Drew runs catalog crawls behind a VPN. On cold boot the VPN takes a few minutes to connect, but the crawler thread fires immediately on FastAPI startup — so on every reboot, the first batch of crawler requests was going out over his real IP, defeating the VPN and exposing him to more aggressive rate-limiting from PTDB. (Hardware crashes are still ongoing; the RAM-to-JEDEC fix didn't help, pointing at CPU/VRM/PSU. Until that's resolved every reboot is a real operational concern, so the gate matters.)

Two opt-in env vars.<ul><li>CRAWLER_STARTUP_DELAY_SECONDS=N — unconditional sleep before crawler threads start. Simple, deterministic; pick a number larger than your worst-case VPN connect time.</li><li>CRAWLER_WAIT_FOR_INTERFACE=name1,name2,… — case-insensitive substrings; the crawler polls every 5s until at least one OS interface whose name contains any of these is up. Times out and proceeds anyway after 10 minutes (safety so a misconfigured value doesn't permanently stall the service).</li></ul>Either, both, or neither. If both, delay runs first. If neither set, no gate (behaviour identical to v0.15.212).

Doesn't block the web server. Gate runs in a daemon thread spawned from on_startup. The web service is serving requests within seconds of process start, regardless of whether the crawler is waiting. Manual start (POST /catalog-crawler/start from the dashboard) skips the gate entirely — if the operator clicked Start they've already verified the network is where they want it.

New diagnostic page: /admin/network-interfaces. Lists every interface, IP addresses, up/down state, current env-var values, and step-by-step instructions for picking the right substring (WireGuard → wg, ProtonVPN → proton, OpenVPN TAP → tap, etc.). Plus the exact NSSM commands to set the env vars.

Loud logging. Every 30 seconds of waiting logs the current state: [crawler] startup-gate: waiting for ['wg'], currently up: ['Ethernet', 'Loopback'] (30/600s). On match: [crawler] startup-gate: interface match -- ['wg-mullvad'] is up; releasing crawler. On timeout: explicit warning that it proceeded anyway, with guidance to fix the config. No silent waiting; Drew can always tell from the service log exactly what the gate is doing.

v0.15.212

2026-05-27

Purge-all button on dup-hero scan page. Drew ran the scan at threshold 5 and got ~25 hash groups, each requiring a separate Reset button. The new “Purge all N hash group(s) at once” control sits at the top of the result list and resets every group in one click.

Safety. Strong confirmation required: operator must type PURGE (exact, uppercase) into a text field next to the button, AND a JS confirm() prompt shows the total row count and group count. Both have to pass before the endpoint fires. Without the literal PURGE in the form body, the endpoint redirects back with an error flash. Catches accidental clicks, browser autofill submits, double-form-post races.

Trusts current state, not the rendered page. The endpoint re-runs the scan internally and resets what is currently above threshold, not whatever the operator's stale page rendered. So if you opened the scan an hour ago, clicked Purge, and the catalog has changed since, you get the live answer not the cached one. The rehash-before-delete logic from v0.15.207 still protects individual rows whose hero was re-downloaded between scan and reset.

Output. Same flash banner format as the per-hash reset, with aggregate counts: “PURGE ALL at ≥5: 25 hash group(s), 1,847 row(s) reset, 1,847 file(s) deleted, 1,847 cache(s) deleted. Crawler will re-fetch on next pass.” Per-group breakdown also logged to service-stdout.log.

v0.15.211

2026-05-27

Dup-hero image previews working + deploy automation.

1. Image preview fix. The <img src> on the dup-hero scan page was constructed as /{webpath} but the static handler for catalog images is mounted at /catalog-uploads/. Drew's preview thumbnails all 404'd (showing “sample hero” alt text). Now constructs /catalog-uploads/{webpath}, normalizes Windows backslashes to forward slashes, and strips an accidentally-prefixed catalog_uploads/ if present (for tolerance of legacy stored paths).

2. deploy.bat. One-command deploy from a freshly-downloaded zip. Workflow: stop NSSM service → wait for actual stop (NSSM's stop returns before the process is gone) → back up the previous build to .last_build/ (one rollback kept) → Expand-Archive -Force the new zip over the install dir → start service → tail service-stdout.log for 10s to confirm startup. Defaults to %USERPROFILE%\Downloads\dispindable-admin.zip; pass a different path as the first arg to deploy from elsewhere. Safe to re-run; if extract fails, service is left stopped so you can fix and retry rather than ending up half-deployed.

3. restart.bat. Quick restart without extracting. Use when you've edited a file in place (e.g. tweaking a template) and want to bounce the service without going through the full deploy cycle.

4. Optional uvicorn --reload mode. service-launch.ps1 now honors the env var DISPINDABLE_AUTORELOAD=1 (set via NSSM AppEnvironmentExtra). When on, uvicorn watches all .py + .html files and restarts the worker on change — edit, save, refresh browser, no service bounce needed. Trade-offs: 5-10s reload per change (DINOv2/CLIP models re-init), restart-loop on syntax errors until fixed, possible mid-extract races during a deploy (mitigated because deploy.bat stops the service first). Default OFF; production keeps the deploy-to-restart contract clean.

v0.15.210

2026-05-27

Dup-hero-image scan: path-resolution bug + diagnostic surfaces. Drew ran the v0.15.209 inspect page successfully but the scan summary read “0 hero files hashed” on a catalog with thousands of real images on disk. The scan ran fine; it just never found any files to hash.

Root cause. hero_local_path values are stored relative to LOCAL_IMAGE_DIR (catalog_uploads/) — they look like ptdb/220/22049.jpg, not catalog_uploads/ptdb/220/22049.jpg. My v0.15.207 scanner used Path(p).exists() which resolved against the process working directory, not the image root. Every check failed, every row was silently skipped, scan returned “0 hashed” instead of an error.

Fix in find_duplicate_hero_images + reset_rows_with_hash. Resolve hero_local_path against LOCAL_IMAGE_DIR if it isn't absolute and doesn't already include the catalog_uploads/ prefix. Tolerates both formats so legacy rows (if any) keep working.

New diagnostic counters. Both the JSON result and the inspect page now surface:<ul><li>rows_with_hero_path — total rows where the column is set</li><li>rows_with_missing_file — how many of those pointed at a file that doesn't exist on disk</li><li>rows_with_hash_error — read/hash exception count</li></ul>If rows_with_hero_path > 0 AND total_files_hashed == 0, the inspect page shows a prominent warning that all paths resolved to missing files — the exact mode that bit Drew on v0.15.207 where everything looked clean and nothing was. The same info also lands in service-stdout.log as a one-line summary every time the scan runs.

Followup observed in Drew's screenshots. Two distinct bad images in the catalog: the iceberg stock photo (PTDB's Cloudflare block-page graphic) and a pure-white blank (likely an empty placeholder from a different PTDB failure mode). Both should appear as distinct entries in the next scan — one large group for the iceberg, another large group for the blank.

v0.15.209

2026-05-27

Hotfix: dup-hero-image inspect page 500'd on v0.15.207. The template I added (catalog_crawler_dup_hero.html) extended _layout.html, but DisPindable's actual base template is base.html. Jinja raised TemplateNotFound on every request to /catalog-crawler/duplicate-hero-images, returning HTTP 500.

Fix. Template now extends base.html and uses the title + content blocks like every other admin page. Added a regression test that asserts the right base extension for the dup-hero template so this can't silently re-break.

How this slipped past v0.15.207 tests. The test suite verified the route exists and the helper functions work, but didn't render the template (template rendering needs the full FastAPI app + Jinja environment with all globals registered, which the unit tests don't set up). The base-extends mismatch was an integration-level failure that only surfaced in production. Going forward I'll grep the existing templates for the correct base when adding new ones.

v0.15.208

2026-05-27

CF-stale refetch now overwrites the stale disk file. Drew found the explanation for the iceberg-persistence mystery in his own logs: ptdb pin 8040 refetched cleanly (cache name was CF-poisoned, invalidated, the page served a real image URL storedImages/12532.jpg, “1/1 saved” printed) yet /catalog/browse still showed the iceberg. The smoking gun: there was no dl OK image[0] NNNN→ 8040.jpg line anywhere in the log. The download never happened.

Root cause. The CF-stale-cache branch in ptdb_sites.fetch_ptdb invalidated the cache JSON but did NOT purge the on-disk image file. Meanwhile, catalog_sites._download_all_images has an idempotent-refresh shortcut at line ~290 that says “if the output file already exists and is >100 bytes, consider this image already cached and skip the download.” That shortcut is correct for routine repeat crawls (don't re-download an unchanged hero); it's catastrophic when the existing file is a stale iceberg from a previous CF block. Result: fresh page parse + stale file on disk = “saved” that wasn't.

Two-layer fix.<ul><li>Purge in the CF-stale branch. Now deletes {id}.jpg and {id}_1.jpg ... _12.jpg from disk before falling through to the fresh fetch. Mirrors the existing purge logic in the ‘stale URLs’ branch right above it — both branches share the same root problem (stale disk file confusing the downloader's existence-check).</li><li>overwrite_existing flag on _download_all_images. Belt-and-suspenders for the case where the purge missed a sibling file. When True, the function logs “will re-download image[i] over existing NNNNB file” and proceeds with the download instead of taking the shortcut. PTDB's fetch passes True whenever the cache was invalidated for any reason (stale URLs, CF poisoning, or force=True). Default remains False so routine repeat crawls preserve their fast idempotent path.</li></ul>What this fixes going forward. Future crawler passes over previously-iceberg'd rows will now actually replace the disk file. No more silently-stale heroes.

What this does NOT retroactively fix. Rows that have already been ‘refetched’ under the buggy path (where the cache JSON was updated but the disk file wasn't) still have iceberg JPEGs on disk. Their cache JSON says “ok=True, image_urls=[the real URL]” so the next crawler pass sees a healthy cache and won't refetch. Use v0.15.207's duplicate-hero-image cleanup to find these rows by hash and reset them. After reset, the next crawler pass will hit the now-fixed code path and produce real images.

v0.15.207

2026-05-27

Pin-ID search in /catalog/browse + duplicate-hero-image cleanup. Two fixes Drew asked for after seeing iceberg stock photos used as hero images on Pinocchio, Eeyore, Winnie The Pooh, and Victorian Carolers cards.

1. Search by source pin ID. The browse page's search box now accepts PTDB6331, PP12345, PNP12345, PinPics12345, or just a bare integer like 6331 (searches all three ID columns). Existing name/series substring search continues to work. The placeholder text updated to hint at the new syntax.

2. Duplicate-hero-image detection. Real pin images are unique; no two pins share bytes. When PTDB's CDN serves a Cloudflare block-page graphic in response to a hero image request, the crawler thinks the download succeeded and saves the same iceberg/placeholder image across every blocked pin. Detection: SHA-1 hash every hero_local_path file, group by digest, surface any hash with ≥20 affected rows.

New routes:<ul><li>GET /catalog-crawler/duplicate-hero-images — runs the scan (~30s-2min on Drew's ~60k catalog) and renders a result page with each suspicious hash, a preview thumbnail, affected row count, and a Reset button per hash.</li><li>POST /catalog-crawler/duplicate-hero-images-reset — commit: for every row whose hero hashes to the submitted target, null the hero_local_path column, delete the disk file, delete the cache JSON. The next crawler pass re-fetches.</li></ul>Belt-and-suspenders: the reset endpoint re-hashes each candidate row's file before deleting, so a stale form submit against rows whose hero has since been re-downloaded won't mass-delete fresh data.

Surfaced where Drew will find it. A red-bordered notice on /catalog/browse links to the scan, immediately adjacent to where he was already looking when he noticed the iceberg images. The scan is available from the crawler dashboard too via the same URL.

What this does NOT fix. Refetched rows can still get re-blocked by PTDB's CDN if our crawler fingerprint hasn't changed. The cleanup gives you a blank slate to refetch; the deeper question of “how do we evade the block?” is the next problem. Options include rotating IPs, headless browser for image downloads (currently only used for page HTML), or accepting that PTDB images are best-effort.

v0.15.206

2026-05-27

Dashboard now shows catalog MIRROR coverage, not cursor position. Drew called this one out: the “0.07% of estimated ID space” badge was calculated as current_id / ceiling, which meant after a state recovery (which resets cursor to first_gap) the dashboard claimed the mirror was nearly empty even though PinPics actually held 16,316 rows. That number is what people mean when they ask “how complete is the mirror?” — not where the crawler thread happens to be in ID space.

What changed. get_progress_summary now also returns row_count (live count from catalog_pins, one COUNT query per source per poll) and pct_mirrored (row_count / ceiling). The dashboard's primary progress bar + percentage label now use these. The original cursor-position info is still visible in the same line as a secondary diagnostic (“cursor at N (X% of ID range)”) because it's useful for spotting stalled threads — just no longer the headline number.

Drew's post-v0.15.205 dashboard will now show:<ul><li>pinpics: 8.7% mirrored (16,316 / ~188,000)</li><li>pinandpop: 22.3% mirrored (26,943 / ~121,000)</li><li>ptdb: 36.0% mirrored (21,586 / ~60,000)</li></ul>These are the numbers that actually tell you something.

v0.15.205

2026-05-27

Estimated catalog ceilings updated to reality. Drew checked each site directly and reported the actual ID ceilings: PinPics ~188,000, Pin&Pop ~121,000, PTDB ~60,000. The pre-v0.15.205 placeholders (250k / 150k / 200k) were guesses that ran high and made the progress badges + ETA estimates wildly optimistic / pessimistic depending on the source. PTDB was the worst offender — the 200k placeholder was over 3× the real ceiling, so the dashboard claimed PTDB was 11% done when in reality the dense crawl had already covered 38% of the real ID space.

What changes visually. The percentage in "0.07% of estimated ID space (164 / ~250,000)" on the crawler dashboard now reads against the real ceilings. ETA-days estimates also recompute against the correct total. No state file changes; this is purely a constants edit in get_progress_summary.

Important implication for cursor placement. Drew's max_id_seen for each source is now within spitting distance of the real ceiling:<ul><li>pinpics max_id=187,276 vs ceiling 188,000 — only ~700 new IDs above max</li><li>pinandpop max_id=120,251 vs ceiling 121,000 — only ~750 new IDs above max</li><li>ptdb max_id=100,481 but ceiling is 60,000 — the rows above 60k are noise/test data, not real pins</li></ul>Last session's Goal C recommendation (jump past max_id) was based on inflated ceilings that suggested tens of thousands of new IDs above. With the real numbers, Goal C yields almost nothing; the value is in the gaps below.

v0.15.204

2026-05-27

Cache-hit fast path + symmetric per-source logging. Drew tailed service-stdout.log after the v0.15.203 state recovery and noticed two things: (1) only pinandpop showed per-pin progress lines, even though all three crawlers were clearly running (their burst-pause lines were interleaved), and (2) cache hits were eating the full 3.0s polite rate limit, which meant burning through his 16k+ cached PinPics rows would take ~12 hours of dead time before the cursor reached an uncached ID.

The asymmetry. fetch_pinandpop had an explicit print("cache hit (ok=...)") on its cache-hit return; fetch_pinpics and fetch_ptdb did not. Both now emit symmetric [catalog_sites] {source} {pid}: cache hit (ok=...) lines for consistency.

The fast path. The crawler loop now classifies any sub-50ms successful fetch as a cache hit (real fetches take 300ms+ even on the happiest path). On a cache hit we:<ul><li>Don't bump requests_since_burst — burst pauses are about looking-human to the remote site, irrelevant when we never touched the remote.</li><li>Don't call _polite_sleep_for_source — the polite delay exists to be polite to a remote server we're not hitting.</li><li>Just time.sleep(0) to yield the GIL and continue the loop.</li></ul>Throttled progress log. Per-pin cache-hit lines from catalog_sites are still printed (they help with debugging individual pin issues), but the crawler loop now ALSO emits an aggregate every 200 cache hits and on streak-end:<ul><li>[crawler:pinpics] cache-hit streak: 600 pins burned through (now at id=12,847); skipping polite sleep, advancing as fast as disk allows</li><li>[crawler:pinpics] cache-hit streak ended after 847 pins; now at id=12,894, next fetch was live</li></ul>These give the operator a clear progress signal even when the per-pin log is firing at hundreds-per-second rate.

Result. Cache-hit traversal goes from 3s/pin to roughly disk-read speed (single-digit ms each). Drew's 16k cached PinPics rows that would have taken ~13 hours to replay now complete in a couple of minutes. Live fetches still respect the full rate limit + burst pauses unchanged.

v0.15.203

2026-05-27

Crawler state file: durable writes + startup auto-recovery. Drew restarted the server and the /catalog-crawler dashboard showed "0 cached" across all three sources (PinPics, Pin&Pop, PTDB) despite the local catalog mirror holding 64,845 pins. Same hardware-corruption symptom that hit at v0.15.134 where state cursors landed at 5/7/7 with 60k DB rows. Likely caused by a WHEA-18 hardlock truncating the state file mid-write. If Drew clicked Start without noticing, the crawler would re-walk ID space from 1 wasting hours of API calls on rows already cached.

Three fixes, each independent.<ul><li>save_state now fsyncs the tmp file before os.replace. Without fsync, the atomic rename was “atomic” in terms of filesystem operations but the new file's actual bytes could still be in the OS write cache when the rename hit durable storage. Hardlock between rename and cache flush = zero-byte or stale file. fsync moves the durability boundary BEFORE the rename, so by the time os.replace runs the bytes are committed.</li><li>get_state falls back to .bak if the live file is missing or unparseable, with loud logging. The rolling backup already exists (v0.15.134); now we actually use it when the live file is unreadable rather than silently returning defaults.</li><li>Startup auto-recovery when state vs DB clearly inconsistent. Before ensure_crawler_running, run detect_state_db_mismatch. If state.successful is <10% of DB row count for any source, OR cursor is at default(1) while DB has ≥100 rows AND first_gap is >100, treat as lost-state and immediately invoke recover_state_from_catalog + apply_recovered_state. Logs a prominent boxed banner with per-source diagnostics so the event is impossible to miss in the service log.</li></ul>Thresholds chosen to be conservative. Small drifts (e.g. cursor 1,234 vs DB's first_gap 1,200) DON'T trigger auto-recovery — they surface a banner for Drew to review. Only the obvious lost-state pattern (0 cached vs 18k rows) is auto-fixed.

What still needs Drew action. The current state file IS still in the lost state on the live server. After deploying this build, the auto-recovery will fire on the next startup and the dashboard will show correct cursor positions again. Alternatively Drew can hit /catalog-crawler/recover-state in a browser right now (before redeploying) to recover via the existing v0.15.134 UI — either path lands the same way.

v0.15.202

2026-05-26

Broader SerpAPI fallback for barcode lookup. Drew tested v0.15.199's diagnostic logs and demonstrated the problem cleanly: scanning 400992770866 (Food & Wine 2025 Passholder pin), the site-restricted SerpAPI query returned 0 organic results, but a manual Google search of the same barcode returned a clean yourwdwstore.net listing with the correct pin name in the title. Google indexed it; just not on pinpics/pinandpop/ptdb.

Now the lookup tries again without the site filter. When the v0.15.127 strict query returns 0 organic results, we run a second query ("{barcode}" disney pin) and look for a hit two ways:<ul><li>URL pattern match — same regex for pinpics.com/pin/N, pinandpop.com/pins/N, pintradingdb.com/pin/N, but applied to ANY URL in the results. Blog posts, forum threads, and aggregator sites often embed catalog links in their bodies and Google indexes them even when the catalog page itself doesn't rank.</li><li>Name extraction — pull the top result's title (e.g. yourwdwstore's “Disney Pin 2025 Epcot Food and Wine Festival Passholder Logo”), strip the boilerplate noise (year, the words "Disney", "Pin", "Logo", edition codes), then substring-search the local CatalogPin table for that cleaned phrase. Pick the closest length-match if more than one hit. Requires ≥12 cleaned characters to avoid false matches on overly-generic titles.</li></ul>Either route returns the same 4-tuple as the strict path, so downstream redirect logic in api_barcode_lookup is unchanged.

Cost. One additional SerpAPI call only when the strict query returned 0. Worst case 2 calls per miss instead of 1. SerpAPI free tier is 250/mo; if barcode lookups dominate usage Drew can disable the broader pass by reverting this file, but the alternative is “No match” for pins that Google clearly knows about.

Diagnostics. Every stage logs to service-stdout.log:<ul><li>broader query='...' — the actual second query string</li><li>broader query organic_results=N — how many hits the broader pass got</li><li>BROADER hit via URL pattern OR BROADER hit via name match — which route resolved it</li><li>name-extraction from top result title '...' -> cleaned='...' — lets you see what the cleaner did to the title</li><li>cleaned name too short OR name matched no local CatalogPin row — explicit rejection reasons when name-match fails</li><li>If both passes miss: dumps the broader-query URLs and titles so you can see what Google did return</li></ul>What it WON'T fix. If the pin is too new for Google to have indexed at all, or if the cleaned title doesn't substring-match anything in your local catalog (because the catalog crawl hasn't reached the pin yet), the lookup still returns no match. The logs now tell you which case it is so the next remediation is clear: wait for Google reindex, or run a targeted catalog crawl, or just use the “scan-as-hint” flow as before.

v0.15.201

2026-05-26

One-line bugfix: board overlay was rendering on the image in sensor orientation while the bboxes were in display orientation. Drew uploaded a portrait phone photo and the result page showed the board rotated 90° with one bbox landing in empty space and the bottom pin (Winnie the Pooh) sitting completely unboxed.

Root cause. Phone JPEGs typically store pixels in the sensor's native orientation (landscape) with an EXIF rotation tag telling viewers to rotate 90° for display. board_scanner.detect_pins applies ImageOps.exif_transpose before asking Gemini, so Gemini sees the photo right-side-up and returns bboxes in the right-side-up coordinate frame. The catalog-matching crop loop in main.py also applies exif_transpose. But board_scanner.render_overlay opened the raw bytes WITHOUT calling exif_transpose, so the overlay JPEG drew display-frame boxes on a sensor-frame image — the visible result was rotated relative to the box positions.

Fix. Added one line (img = ImageOps.exif_transpose(img)) in render_overlay right after the image opens. ImageOps.exif_transpose is idempotent on already-correctly-oriented input, so no risk of double-rotating client-resized uploads.

Affects. Any portrait phone photo uploaded as the original file (or with EXIF preserved). Landscape photos and photos that had their EXIF stripped by client-side downscale (which always rotates pixels and drops the tag) were never affected. This explains why some of Drew's earlier scans rendered correctly and others (higher-resolution / original-quality uploads) showed up rotated.

v0.15.200

2026-05-26

Real bug: series_name + super_series were not in the metadata-backfill mapping. Drew tested v0.15.198 and noticed that even though the eBay query logic looks up CatalogPin.series_name to build rich queries like “Mickey Mouse Icon Character Mystery Collection: Jack Skellington”, his real catalog rows show series_name=NULL — even when the Pin&Pop page CLEARLY has the series listed.

Root cause. Two different code paths write to CatalogPin:<ul><li>_upsert_catalog_pin — runs on every live scrape. Writes existing.series_name = data['series'] directly (lines 298-301).</li><li>_promote_catalog_fields — runs from the backfill endpoint (/admin/backfill-catalog-columns) that replays cached metadata_json blobs for already-crawled rows. Has a structured key-to-column mapping for original_price, sku, rarity, fac, etc. — but NOT series or super_series.</li></ul>Net effect: rows crawled BEFORE the v0.15.158 series-extraction fix (when the live scraper didn't write series_name) had the data sitting in their metadata_json blob but no mechanism to backfill into the actual column. Even pressing the existing backfill button wouldn't help.

Fix. Added ("series", "series_name", _clean_str, 255) and ("super_series", "super_series", _clean_str, 255) to the mapping loop in _promote_catalog_fields. Also updated the before/after change-detection tuple in the backfill endpoint to include series_name + super_series, so the “N rows updated” count actually reflects series fills.

To activate. Drew clicks the existing POST /catalog/backfill-catalog-columns button after deploy. Should populate series_name across the catalog in one pass. After that, board scans will build the richer “Series: Name” eBay query that v0.15.198 added and we'll see the better match rates we expected.

Clickable candidates. v0.15.198 made the TOP match clickable; alternatives were still text-only. Now every candidate's hero image, name, and external IDs (PinPics/P&P/PTDB) link out the same way the top match does. Drew can drill into any candidate to verify before clicking “Use this”.

Source-breakdown diagnostic. Drew asked “is pinpics being searched at all?” The catalog-match cache build now logs per-source counts: [catalog-match] source breakdown: pinpics=N pinandpop=N ptdb=N. If pinpics is ~0, that's the answer (no pinpics rows have DINOv2 embeddings, even though they're crawled). If it's 20k+, then pinpics IS in the cache but its stock images aren't matching real-world pin photos well — different problem, different fix. Also added cat_src=pinpics+pinandpop to the per-pin scan-assess log lines so a grep tells us how often each source surfaces as a top match.

About the “SerpAPI not configured” message. The log message is doing its job. It means os.environ.get("SERPAPI_API_KEY") returned empty when the worker thread ran. Earlier scans in this session DID hit SerpAPI successfully, so the key is probably still in api_keys.env but didn't make it into the worker's env. Things to check, in order: 1) is SERPAPI_API_KEY=... still in api_keys.env; 2) does Drew's launcher actually export it (the launcher logs [launch] api_keys.env loaded (N vars): GEMINI_API_KEY, SERPAPI_API_KEY, ... — should see SERPAPI in that list); 3) did the key get rotated by SerpAPI for hitting rate limits (250/mo free tier shared across Lens + barcode lookups). No code change to ship here — this is operational diagnosis.

v0.15.199

2026-05-26

Fixed silent barcode-lookup — now logs every path so the no-match case is debuggable. Drew scanned three real Disney pins; one resolved correctly (Mickey Halloween Wreath → P&P #104938), two left the log with ZERO [barcode-lookup] lines (Food & Wine 2025 Passholder, Jollywood Nights Happy Ollie Days). Both should clearly exist on Pin&Pop. Without logs we couldn't tell whether SerpAPI was called, what came back, or where the lookup gave up.

What now logs. Every barcode hit produces a complete trace:<ul><li>[barcode-lookup] start q='400995468890' — entry, lets us correlate against the HTTP log line</li><li>[barcode-lookup] local: exact_owned=N exact_catalog=N total_so_far=N — how many rows the local exact-ID search found</li><li>[barcode-lookup] '...': no local match, calling SerpAPI fallback OR ... skipped (reason: SERPAPI_API_KEY not set)</li><li>[barcode-lookup] _serpapi_lookup('...'): query='...' — the exact Google query</li><li>[barcode-lookup] _serpapi_lookup('...'): organic_results=N — how many hits Google returned</li><li>[barcode-lookup] _serpapi_lookup('...'): result[i] link='...' title='...' -- no catalog pattern matched — this is the key diagnostic: shows the actual URLs Google returned that DIDN'T match our 3 regex patterns (pinpics/pinandpop/ptdb)</li><li>[barcode-lookup] '...': NO MATCH (exhausted local exact + substring + SerpAPI); returning empty result — final return state</li></ul>What this will reveal. For Drew's two failing barcodes, the next scan will tell us one of three things:<ul><li>Google returned 0 results. Google hasn't indexed those barcodes against pinpics/pinandpop/ptdb yet. Could try broadening the site filter or adding eBay-listing parsing as a fourth source.</li><li>Google returned 5 results but ALL were non-catalog domains (eBay, Disney Store, Reddit, Pinterest, blogs). Our 3-source restriction missed. Consider adding fallback parsing of eBay listings that embed pinpics/pinandpop links in their description.</li><li>Google returned a pinandpop.com/pins/<slug> URL with no numeric ID. Same slug-URL problem the Lens parser has (deferred in v0.15.198). Confirming with real data tells us this is worth fixing.</li></ul>Pure logging change — no behavior modification, no perf cost, no new tests needed beyond syntax check. Diagnostic round only.

v0.15.198

2026-05-26

Four real bug fixes after Drew tested v0.15.197 on real boards.

1. Lens URL parser actually parses now. The v0.15.192 diagnostics kept reporting “Lens fired N time(s), got results but NO parseable catalog URLs” for every pin. Drew demonstrated the issue by pasting a literal crop into images.google.com and getting pinpics/pinandpop results back immediately. The bug: authenticity_check._augment_matches never had URL-pattern parsing — it returned {url, title, thumbnail, domain, classification} only, never pinpics_id / pinandpop_id / ptdb_id. The board-scan cascade was looking for those fields and finding nothing every time. Fixed: regex extraction for three URL families (pinpics.com/pin/N, pinandpop.com/pins/N, pintradingdb.com/pin/N), with PP prefix stripping for pinpics. Slug-form pinandpop URLs (no numeric ID in the path) still won't parse — that's a follow-up requiring a fetch to extract the canonical ID. But all the numeric-ID forms now work.

2. eBay queries use the full catalog row, not just the name. Drew flagged that the Nemo top match was searching eBay for bare “Nemo” (3,462 unrelated results) when the catalog had richer metadata (series name, year). The board-scan eBay enrichment now looks up the full CatalogPin row for each top match and builds “{series_name}: {name}” the same way /new-pin does. Falls back to super_series if series_name is empty. Falls back to bare name when neither is present. Plus a bare-name retry: if the rich query returns 0 listings for BOTH sold and active, we retry with the bare name (catches the “Mickey Mouse Icon Mystery Pack - Jack Skellington” case where the catalog name is too specific for how sellers list).

3. Catalog matches are clickable now. The catalog hero image and the pin name are both links to /catalog/pin/{id} (local catalog detail page, new tab). External IDs (PinPics, P&P, PTDB) link to the actual source site — click PinPics #12345 to open pinpics.com/pin/12345 in a new tab. Drilling in for verification was a hidden step before; now it's a click.

4. Per-pin assessment log lines. Drew asked for a way to track match quality across 20+ pins without manually noting each one. The worker now emits one [scan-assess:N/M] line per pin in a fixed format: verdict={VERIFIED/REJECTED/UNCERTAIN}, top_score=, d2=, clip=, src=, lens=, n_alt=, cpid=, name=. Grep scan-assess in logs/service-stdout.log to count verdicts / sources / score distributions across many scans without parsing JSON.

Gemini Pro vs Flash for verification. Drew asked if Pro would help. Verdict in the v0.15.193 verifications has been accurate (looking at his test scans, all four REJECTED were correct rejections, all VERIFIED were correct verifies). The model isn't the bottleneck. Pro would be ~10× cost, ~3× latency, no meaningful gain on this well-bounded task. Sticking with Flash.

What didn't make this round. The slug-URL case for pinandpop (where Lens returns pinandpop.com/pins/some-slug-name with no numeric ID) requires fetching the page to extract the canonical URL. Adds latency to every cascade fire; would want to gate it behind a config. Deferred. Also the 33 MB upload error needs the actual browser-console error text to diagnose — could be fetch timeout, server rejection, or proxy/CDN. Send me what console said and we'll fix it.

v0.15.197

2026-05-26

Async board scan: POST returns immediately, worker thread processes in background, browser polls for progress, push notification fires when actually done. The piece I deferred from v0.15.193 because it was bigger than the rest of that round combined. Now done.

Before (v0.15.193 sync): POST /scan/board blocked for 30+ seconds while detection + matching + verification + eBay ran serially. Push fired at the exact moment the HTTP response returned — defeating the purpose. Closing the tab during processing aborted the scan.

After (v0.15.197 async): POST saves the upload to disk, inserts a BoardScan row with status='pending', spawns a worker thread, and returns a 303 redirect to /scan/board/{id}/wait in ~50 ms. The wait page polls /scan/board/{id}/status every second; renders a progress bar with the current phase. When status flips to 'ok', browser navigates to /scan/board/{id} (the persisted result view from v0.15.193). Close the tab — scan keeps processing on the server. Push fires from the worker thread when the result is actually ready, with a link back to the result page.

Progress phases: queued → detecting (10%) → matching (35%) → verifying (60%) → fetching_prices (80%) → rendering (92%) → done (100%). Each phase has a human label in the wait page; exact progress jumps are estimates (we can't know how long Gemini will take), but the phase transition is the real signal of progress.

Schema additions to board_scans: phase, phase_progress, phase_message, upload_path, verify_matches, started_at. Migrated via _LATE_ADDITIONS so existing DBs upgrade in place. upload_path stores the relative path of the saved upload (under uploads/board_uploads/) so the worker can re-read the bytes after the request body is gone. Uploads are auto-deleted after the worker completes successfully.

Crash safety. On startup, scan_worker.cleanup_orphaned_pending_at_startup() marks all status='pending' rows as 'error: interrupted by service restart'. Stuck rows from a process that died mid-scan (like the MCE hardlocks Drew is hitting) no longer look queued-forever in the UI; user can re-upload.

Worker thread model. One daemon threading.Thread per scan, tracked in an in-process registry by scan_id. No thread pool (which would either block under load or grow unbounded). Each phase update opens its own short-lived DB session so we don't hold a connection for the entire scan duration. Daemon threads don't block clean shutdown — if the user nssm stops mid-scan, the row stays 'pending' and gets cleaned up at next startup.

What this unlocks: push notifications now MEAN something (submit, switch tabs, get pinged when done). Scan history shows in-progress scans separately from completed ones. Foundation for future features like parallel scans, cancel button, retry button, and a worker queue view on the admin page.

What did NOT change. The result page is byte-for-byte the same as v0.15.196 — same CLIP rerank scores, same Lens diagnostics, same Gemini verification badges, same candidate alternatives, same eBay links. Just rendered from the persisted JSON instead of computed inline.

v0.15.196

2026-05-26

Black-box telemetry recorder for diagnosing the 5-7 hour hardlocks during catalog crawls. Drew is hitting whole-machine freezes that require holding the power button to recover — meaning the Python process logs (in-memory until flushed) are LOST in the lockup. New telemetry.py module writes a compact JSONL heartbeat every 30 s to logs/telemetry/heartbeat.jsonl with os.fsync after each write, so the last heartbeat survives even a hardlock. The trend in the last N records before the freeze is the diagnostic.

What each heartbeat records: process RSS + VMS (the actual + virtual memory footprint), thread count, open handle count (Windows) or file descriptor count (Unix), CPU percent, system available memory, system used percent, and a COUNT OF CHROMIUM CHILD PROCESSES (Playwright leaks these on errors; the count rising monotonically is a strong leak signal). Plus catalog-crawler state injected via register_extra(): which sources are active, consecutive_blocks counts, whether the backfill thread is running. Lifecycle markers ({event: startup} and {event: shutdown}) frame each process lifetime — absence of shutdown next to a stale-then-new pair indicates a hard crash.

Viewer: /admin/telemetry. Renders the last 200 heartbeats as a table plus a trend summary at the top: RSS delta, handles delta, chromium delta, system available delta over the window. Each delta is color-coded (green / amber / red) by the magnitude that indicates a likely leak. "How to read this" panel explains what each direction means. Lets Drew (or anyone) eyeball the state RIGHT BEFORE a crash without grepping JSONL.

File rotation. Heartbeat file rotates at 50 MB or 24 h, whichever comes first, keeping the last 5 archives in logs/telemetry/heartbeat.YYYYMMDDTHHMMSS.jsonl. Old archives are pruned automatically.

New dep: psutil ≥ 5.9. Without it the heartbeat still writes (with psutil_missing: true marker) but most of the useful columns will be empty — install on first run after deploy. Tiny package (~500 KB), no native build needed on Windows.

What this won't catch. If the hardlock is a driver or kernel issue (storage driver, GPU driver, thermal cutoff) the Python heartbeats may all look healthy right up to the moment of the freeze, with no warning. In that case the SIGNAL is the absence of warning — we'll see a clean heartbeat at T-30 s and then no shutdown event, no startup event for hours, then a startup after the reboot. Pair this with Windows Event Viewer (System log, looking for Kernel-Power Event ID 41) for the OS-side picture.

v0.15.195

2026-05-26

CLIP rerank now wired into the regular /scan path too (was only board-scan in v0.15.194). Drew caught that the rerank from v0.15.194 only affected /scan/board — the single-pin /scan flow was still DINOv2-only. That meant single-pin scans (and the /new-pin prefill that feeds off them via Scan.catalog_top_matches_json) didn't benefit from the semantic correction. Fixed: submit_scan now computes CLIP for the first scanned image and passes clip_query_blob to find_catalog_matches, same as the board-scan path. Both dinov2_score and clip_score are persisted on the scan's catalog_top_matches_json so /new-pin sees them automatically when re-rendering.

UI: /scan's Catalog Matches block now shows the breakdown. Same “DINOv2 0.834 · CLIP 0.412” line and amber “⚠ disagreement” tag for big DINOv2/CLIP gaps. Looks for-real-the-same as the board-scan version so the user's eye doesn't have to re-learn the signal.

Backward compat for old scans. Pre-v0.15.194 scans persisted via catalog_top_matches_json don't have dinov2_score or clip_score keys in their JSON. Jinja's Undefined is not none returns True (gotcha!), which would have rendered empty score breakdowns for those old scans. Templates now use is defined and is not none which correctly suppresses the breakdown on old data. New scans (single-pin OR board) get the full breakdown.

What's NOT affected. The find_matches path that finds USER's own pins (not catalog) still uses DINOv2-only — that's intentional. The user's own collection is small enough that DINOv2 visual similarity is exactly what you want (“do I already have this pin?”). CLIP rerank matters when matching against the 50k+ catalog where the visually-similar-but-content-wrong problem dominates.

v0.15.194

2026-05-26

CLIP rerank of DINOv2 top candidates — the leveraged match-quality fix. Drew pointed out that CLIP embeddings were already computed and stored on every catalog row (originally to power text→image search at /catalog/text-search). They were never used for image→image board-scan matching though. v0.15.194 wires them in: for every detected pin's crop, we now compute both DINOv2 (768-d, visual similarity) AND CLIP (512-d, joint image-text semantic space), then rerank the top candidates by the ensemble score.

Why this matters. DINOv2 is great at “find the visual near-duplicate” but blind to semantic content. That's why Maleficent→Vanellope and Mickey-head Nightmare moth→Jack Skellington keep showing up — the shape silhouettes match even though the CHARACTERS are completely different. CLIP knows Maleficent and Vanellope are different characters because it was trained on image-text pairs. So CLIP score for the wrong match is much lower, dragging the ensemble down and letting the correct match (which has middling DINOv2 but high CLIP) bubble to the top.

Pipeline now. DINOv2 finds top-20 visual candidates (was top-3). CLIP cosine sims fetched for those 20 in one DB query (~10 ms). Combined score = 0.5·dinov2 + 0.5·clip. Re-sort, clamp to top-5. The whole rerank step adds ~20-30 ms total — cheaper than a single Lens call, much cheaper than Gemini verification, and runs on every pin not just the below-threshold ones.

UI: both scores visible. Each match card now shows “DINOv2 0.834 · CLIP 0.412” underneath the source IDs. When the two disagree by more than 0.25 (visual similar but semantically distant), an amber “⚠ disagreement” tag appears — that's the “DINOv2 found a Funko-shaped Vanellope but it's not actually Maleficent” signal made visible. Lower-confidence candidates show condensed “D 0.83 · C 0.41”.

How the cascade looks now. (1) DINOv2 top-20 by visual similarity, ~50 ms, free. (2) CLIP rerank to combined score, ~30 ms, free. (3) SerpAPI Lens fallback when ranked top is below the threshold, ~1-3 s, paid quota. (4) Gemini verification of final top match (v0.15.193), ~1-2 s, pennies. Each layer catches errors the previous layer missed, with cost increasing as we go deeper.

Backward compatibility. Rows without embedding_clip populated (older crawled pins before CLIP backfill ran, or pins where the CLIP encode failed) keep their DINOv2 score as combined — no penalty for missing data. The /admin/catalog/backfill-clip-embeddings route handles backfilling old rows; run it once if many of your catalog rows show “CLIP n/a” in the breakdown.

Tuning. The default ensemble weight is 50/50 (clip_alpha=0.5 in catalog_matching.find_catalog_matches). Future work could expose this as a slider on the scan-board form. If CLIP turns out to dominate the wrong-character cases, increasing CLIP's weight (alpha=0.4 or 0.3 to give CLIP 60-70%) would help. The rerank pool size (default 20) is similarly tunable; bigger pool catches more “buried correct answer” cases but adds DB lookup cost.

v0.15.193

2026-05-26

Three pieces: Gemini-verify each top match, persist every scan with a history view, fire push notification on scan completion.

1. Gemini verification of the top match. New verify_match.py module. After DINOv2 (and Lens) produce the top catalog candidate per pin, send the user's crop and the catalog hero to Gemini 2.5 Flash and ask “are these the same pin?”. Verdict per pin: VERIFIED (green), REJECTED (red), UNCERTAIN (amber), SKIPPED (when either image is missing), ERROR (when the call fails). Renders above the side-by-side compare with the model's one-sentence reason. This catches the classic DINOv2 failures — Maleficent→Vanellope was shape-similar but content-different; Gemini knows those are different characters. Parallelized via ThreadPoolExecutor (max 6 workers) so 8 pins take ~3 s instead of ~12 s serial. Opt-in via the “Gemini-verify each top match” form checkbox — DEFAULT ON since match quality is the primary problem this round. Cost is a few cents per scan.

2. Scan history. New board_scans table persists every submission: token, status, upload metadata, tunables used, counts, and the full enriched result_json (pins + bboxes + matches + Lens diagnostics + eBay prices + Gemini verdicts). New routes: GET /scan/board/history renders a list (most-recent first, thumbnail of the overlay, pin counts, click to re-open); GET /scan/board/{scan_id} re-renders an old scan from the persisted JSON, NO re-processing. Overlays persist to uploads/board_overlays/<token>.jpg so old history entries still show overlay previews after the LRU memory cache evicts them. History link in the scan-board hero. Server restarts no longer lose results.

3. Push notification on scan completion. When the device has an active push subscription, the form shows a “Notify me when this scan finishes” checkbox (opt-in). When checked, JS attaches the push endpoint URL as X-DP-Push-Endpoint; server stores it on the BoardScan row and fires push_notifications.send_to_subscription on completion. Notification body shows pin count + matched count; click opens /scan/board/{scan_id}. Caveat: the scan is still SYNCHRONOUS, so the push fires just as the HTTP response returns. Useful when the user switched apps mid-scan (push = “come back to the tab”), less useful when they're sitting on the loading overlay. A future round will go full-async (POST returns scan_id immediately, background worker, push fires later); this round delivers the persistence + push wiring so the migration is incremental.

What I deferred. True asynchronous scan processing — POST returns a scan_id immediately and a background thread does the work, with the result page polling for status. The architecture is bigger than the rest of this round combined (background worker lifecycle, restart-resume of pending rows, polling endpoint, progress phases in the UI). Shipping it half-built would be worse than shipping the rest cleanly. Next round.

v0.15.192

2026-05-26

Lens cascade diagnostics + crop resolution transparency. Two visibility upgrades for the board-scan results page so Drew can finally see WHY wrong matches (Maleficent → Vanellope, Mickey-head moth → Jack Skellington) aren't being corrected by the SerpAPI Lens fallback. Scores 0.694-0.778 are all below the 0.85 threshold and should fire Lens, but every screenshot showed “Local” badges — meaning Lens was running but producing nothing usable. Until this version we had zero visibility into which branch of the cascade failed.

1. Per-pin Lens status tracking. Each pin now carries a pin.lens dict with status enum + raw counts: raw_results (how many SerpAPI Lens matches came back), parsed_with_id (how many had URLs we could parse into pinpics/pinandpop/ptdb IDs), matched_in_db (how many of those IDs exist in your catalog). Status values: skipped_local_strong (local score ≥ threshold), skipped_disabled (SerpAPI not configured), fired_zero (Lens returned nothing), fired_no_links (Lens returned results but no parseable catalog URLs — results were on eBay/Pinterest/etc instead of pinpics), fired_no_db_match (parsed IDs but the catalog rows aren't in your DB), fired_added (success), error.

2. Aggregate Lens banner. Above the legend, color-coded summary: GREEN when Lens added catalog matches, AMBER when Lens fired but couldn't produce usable results (with the specific reason), GREY when Lens didn't fire at all (all locals strong), RED when SerpAPI isn't configured. Expandable counts panel shows per-state tallies for debugging.

3. Per-pin Lens status line. Below each pin card's header, a one-liner explains what happened for this specific pin. “Lens skipped (local 0.892 ≥ 0.85)” for strong locals, “Lens fired, 8 result(s) but NO parseable catalog URLs” when SerpAPI is producing but our URL parser can't extract IDs, etc. This is what tells you whether the Maleficent pin's Lens call completed at all and what it produced.

4. Crop dimensions in per-pin meta. Each pin card now shows “crop 845×892px” next to confidence. Lets Drew see whether v0.15.191's “Original” upload is actually producing big high-resolution crops, vs the resized 2048 path producing smaller ones. Bigger crops → better DINOv2 embeddings + better Lens matches.

5. Source-resolution clarification. The result meta now notes that Gemini sees a downscaled-to-2048 version internally (board_scanner.MAX_DETECTION_DIM) and that crops are extracted from the full source resolution. Bboxes are normalized 0-1.0 so they map cleanly between the two — this is already how it's been working; v0.15.192 just makes that visible. This is the “two-pass” architecture working as intended; once you can SEE crop dimensions in the UI you can verify it.

What this tells us next. Run a scan after deploy and look at the Lens banner. If it says “fired N times but no parseable URLs,” the fix is in the URL parser (authenticity_check._augment_matches) -- we need to teach it about additional URL formats that map to your catalog sources. If it says “fired but 0 raw results,” the crops are too small/blurry for SerpAPI to identify them, and the fix is in crop quality (try Original upload + bigger BBOX_PAD). If it says “not configured,” SerpAPI_API_KEY is missing or invalid.

v0.15.191

2026-05-26

Board-scan upload: “Original” option that skips the client-side resize entirely. Pre-v0.15.191 every upload went through createImageBitmap + canvas.toBlob('image/jpeg', 0.85) — even at the 8192 cap you got a re-encoded JPEG (your phone's native JPEG was already quality ~0.92-0.95, so this was a quality-losing double-encode). New top dropdown option “Original (no resize, best quality)” ships the raw file bytes — no decode, no resize, no re-encode. We still createImageBitmap the file once to read its dimensions for the status line, then immediately .close() the handle so the full bitmap never lands in memory. Status line shows “Uploading original (48 MB, 16320×12240, no client resize)…” so you know which path you got. Server PIL bomb cap is already disabled (v0.15.186), so 200 MP photos go through fine. Per-pin crops now come from the camera's actual pixels, not a re-encoded approximation. Use this when matching quality matters more than upload time.

v0.15.190

2026-05-26

Big board-scan UX round: catalog images now actually appear, candidate alternatives become selectable, eBay prices link to real searches, detection caching for slider tuning, bigger upload resolution default, admin push alerts on crawler blocks.

1. Catalog images: actual bug. Drew flagged every match card on the board-scan results showing “(no catalog image)”. Root cause: v0.15.189 read CatalogPin.image_filename, but that column is only set by the legacy catalog_ingest direct path. The actual crawler (which built nearly all of Drew's catalog) writes to hero_local_path via _upsert_catalog_pin. Every crawler-discovered row had image_filename = NULL even though the hero image existed on disk at the predicted path. Fix: read hero_local_path first, fall back to image_filename for legacy rows. Applied to both the local-DINOv2 match path AND the SerpAPI Lens match path.

2. Selectable candidate alternatives. Pre-v0.15.190 the “2 more candidate(s)” details block was a text-only list. When DINOv2's top match was wrong (Maleficent matched to “Vanellope #1”, Mickey-head Nightmare moth matched to “Jack Skellington Mystery Pack”), the user had no way to switch to a better candidate without going through the manual /new-pin flow. Now each alternative renders as a thumbnail strip: 60×60 catalog image, name + score + source IDs, plus a Use this button that POSTs to the same /scan/board/use-pin-match endpoint as the top match's button. Click to override the auto-pick.

3. eBay price links. Sold and Active price bands now link to the actual eBay search results page (category 38004 Disneyana > Pins, sold uses LH_Sold=1&LH_Complete=1 filter). External-link icon next to the band. Click flow: see “Sold: $12.50–$45.00”, click, land on the eBay completed listings page for that exact query.

4. Detection caching (the slider-tuning fix). The Gemini detection step takes 3-8 seconds and is independent of the tuning sliders (padding, threshold, eBay opt-in all kick in during enrichment, AFTER detection). Pre-v0.15.190 every slider tweak re-burned the Gemini call. Now: SHA-256 of the upload bytes is the cache key, 24h TTL, stored in .board_scan_detection_cache/<hash>.json. Re-scans with different sliders hit cache and only the enrichment (DINOv2 + Lens + eBay) re-runs — ~2-3 seconds instead of ~10. Cache failures are silently retried; only successful detections get cached (failures should be retried). Cached state shown in the result meta: detection time 3.4s · cached (180s ago).

5. Upload resolution: 6144 default + 8192 option. The 4096 default was conservative; modern phones produce 50-200 MP and the per-pin crops benefit from more source pixels. Bumped default to 6144 (a 50 MP shot becomes ~6 MP after resize, vs ~3 MP at 4096). 8192 added for the max-detail case. Tradeoff: 4-8 MB uploads vs 1-2 MB; client-side createImageBitmap streaming resize handles any source resolution without OOM.

6. Admin push alerts on crawler block thresholds. New opt-in per-device toggle on the Settings page's Push notifications card: “Receive admin alerts on this device.” When ON, this subscription receives pushes when the crawler auto-pauses a source (10 consecutive CF blocks). New push_notifications.notify_admins() helper with in-process 10-minute dedup keyed by (title, body) so a flapping source doesn't carpet-bomb the lock screen. Backend: new admin_alerts boolean column on PushSubscription (default False, migrated via _LATE_ADDITIONS), new GET/POST /push/admin-alerts endpoints. Wired into catalog_crawler's block-pause branch, fail-safe (push failure never blocks the crawler from applying the pause).

Deferred to next round: scan-complete push notifications. Currently scans are synchronous (POST blocks for ~30 s, returns full HTML), so a push would arrive at the same moment as the page response — defeating the purpose. Making it useful requires async-scan architecture (POST returns scan_id immediately, background worker processes, push fires when done with link to the result page). That's its own version.

v0.15.189

2026-05-26

Board scan: per-request tuning sliders, side-by-side image compare, eBay sold+active prices per pin, larger X1C-ready Mickey-head pinboard STL.

1. Live tuning sliders on the upload form. v0.15.188 made the crop padding and Lens trigger threshold env-var-tunable, but Drew wanted to dial them per-scan without restarting the service. New collapsible “Matching tuning” block below the resolution selector with two range sliders:<ul style='font-size:0.85rem;'><li>Crop padding: 0-50%, default 20% (matches the v0.15.188 env default)</li><li>SerpAPI Lens trigger: 0.00-1.01, default 0.85 (matches the v0.15.188 env default). 1.01 = always Lens, 0.00 = never Lens.</li></ul>Both submit as form fields (bbox_pad_pct / lens_threshold) and override the env defaults for the single request only -- per-call experimentation without breaking other users' defaults. The slider value displays live next to the label so the operator sees what they're committing to.

2. Side-by-side image compare on each match card. Every per-pin match card now shows the user's crop (left) next to the catalog hero (right) in a 2-column grid above the pin name and score. Quickest visual eyeball for “is this actually my pin?” -- the score alone doesn't answer that. Crop image served from /uploads/<crop_filename> (saved by the v0.15.172 Phase D pipeline); catalog hero from /catalog-uploads/<image_filename>. Required adding image_filename to each catalog_match dict (DB lookup batched per pin to avoid N+1).

3. eBay sold + active prices, opt-in. Each match card now optionally shows recent sold prices (min, max, average, count) and active listing prices (min, max, count) when the user checks “Include eBay sold + active prices” on the upload form. Reuses the existing ebay_lookup.fetch_sold_listings + search_active_listings (same 1h cache, same broaden-on-zero-results fallback). Queries fan out via ThreadPoolExecutor(max_workers=8): an 8-pin board burns 16 parallel queries that finish in roughly the time of the slowest one rather than 16-in-series. Cold cache adds ~1-2 s to the total scan time; warm cache (re-scan of the same board within an hour) is essentially free. Opt-in because the eBay Browse API tier is 5000/day and a board-spam-tester could burn through it.

4. New Mickey-head pinboard STL (mickey_pinboard_v2.stl). Drew uploaded the current pinboard and asked for a larger version sized for the Bambu X1C's 256×256 mm bed. Constraints: keep ear diameter at 82 mm (existing design), enlarge the head. The diagonal-rotation idea doesn't help here -- Mickey is already wider than tall (240×163 mm), and rotating a non-square shape on a square plate always grows the bounding box (a 45-degree rotation would push the bbox to 285×285, overshooting). At 0° rotation, the binding constraint is the 256 mm plate width: the new pinboard maxes out at 150 mm head diameter (vs 128 mm original -- ~17% larger radius, ~37% more pin surface) with ears preserved at their original 82 mm dia. Final footprint 253×176 mm. 441 pin holes (vs 309 original) on an 8 mm hex grid, 2.5 mm dia, 4 mm edge margin, 12 mm thick. Parametric rebuild rather than mesh surgery because the head/ear overlap region doesn't scale cleanly under non-uniform XY transforms.

v0.15.188

2026-05-26

Board-scan tuning: 0.85 local-confidence threshold (was 0.75), 20% bbox padding (was 15%), explicit prompt to include character bodies/extensions, both threshold and padding now env-tunable.

1. Local-confidence threshold: 0.75 → 0.85. Drew flagged that pins scoring 0.76-0.84 on DINOv2 were skipping SerpAPI Lens but getting the wrong catalog match. At 0.85 only highly-confident local matches skip Lens; everything else gets the visual second opinion. Tradeoff is more SerpAPI calls (consumes the 250/mo free tier faster) but better accuracy on the marginal cases.

2. Bbox padding: 15% → 20%. v0.15.186 shipped 15% padding, but Drew's pin 7 (maroon football) still clipped a few pixels on the right edge. 20% gives more headroom for asymmetric pin shapes. Pure padding isn't enough for cases where Gemini detected only the salient portion (see #3), but it helps the borderline cases.

3. Gemini prompt rewritten to demand full-pin bboxes. Pin 8 in Drew's scan -- Maleficent -- was boxed only at her head/torso, with her dress/feet completely outside the bbox. No amount of post-hoc padding would have caught that; Gemini interpreted “the pin” as the visually salient character upper-body. The prompt now explicitly enumerates the kinds of extensions to include:<ul style='font-size:0.85rem;'><li>Character bodies (legs, dress, cape -- not just head/torso)</li><li>Outstretched arms, wings, swords, wands, tails</li><li>Banner extensions, scrollwork, trailing ribbons</li><li>Mickey-head ear extensions jutting from a central disc</li><li>Dangling charms or chained secondary medallions</li></ul>Closes with: “If you can see metal/enamel that's clearly part of the same physical pin, it MUST be inside the bbox.”

4. Env-var tunables. Both knobs are now configurable without code edits:<ul style='font-size:0.85rem;'><li>BOARD_SCAN_LOCAL_THRESHOLD -- DINOv2 score above which to trust local and skip Lens. Default 0.85. Set to 1.01 to force Lens on every pin (burns the 250/mo quota fast); set to 0.0 to disable Lens entirely.</li><li>BOARD_SCAN_BBOX_PAD -- fraction to expand each bbox before cropping. Default 0.20. Higher = more background in crops (slightly worse for DINOv2, harmless for SerpAPI Lens).</li></ul>Both go in api_keys.env alongside the other tunables; service-launch.ps1 loads them on startup.

5. Threshold visible on the result page. The matches header now shows: “Pipeline: local DINOv2 → SerpAPI Lens when local score < 0.85 · bbox padding 20% · tune via env...”. Answers Drew's “what's the threshold?” question without him having to find the changelog.

v0.15.187

2026-05-26

CF cleanup also scans cache files (v0.15.185 only checked DB column); loading overlay on board scans.

1. CF cleanup gap closed. Drew ran /catalog-crawler/cloudflare-poisoned after the v0.15.185 deploy and got count=0, despite having visibly seen PTDB pin #9305 with “Access denied | www.pintradingdb.com used Cloudflare to restrict access | ...” in its PTDB metadata panel. Root cause: the v0.15.185 helper only inspected the CatalogPin.name DB column. But the visible metadata panel comes from .catalog_cache/ptdb_<id>.json -- a separate storage layer, written by the crawler at scrape time, read by /catalog/pin/<id> detail pages. Pre-v0.15.155, the parser would write ok=True with a CF block-page title in the cache file's name field. The DB column ended up null (some later ingest pass cleared it) but the cache file kept its poisoned name. v0.15.185 didn't look at the cache.

v0.15.187 extends the check: find_cloudflare_poisoned_cache_files() scans .catalog_cache/*.json for any file claiming ok=True whose name contains a CF signature. reset_cloudflare_poisoned_cache_files() deletes those files; the next crawler pass uses the missing cache as a re-fetch signal, and the fresh fetch goes through v0.15.155's parser-level CF detector.

Bookmarklet imports (pinpics_imported_*.json) are excluded by filename pattern -- same schema-guard principle as v0.15.183. Both inspect/reset endpoints now report two layers: db_rows and cache_files with separate counts. The flash message after reset shows both numbers.

2. Loading overlay on board scans. The v0.15.186 SerpAPI Lens cascade can add 1-3 seconds per pin with low local match confidence, so an 8-pin board with uncrawled pins can take 30+ seconds total. Pre-v0.15.187 the user saw the upload form just sitting there during processing, with only a small Processing… text next to a disabled submit button. New: full-card overlay covers the upload area during processing with: <ul style='font-size:0.85rem;'><li>Spinning indicator (CSS-only, no GIF dependency)</li><li>Phase label that updates as the pipeline progresses (preparing → uploading → detecting+matching)</li><li>Live elapsed-time counter (MM:SS)</li><li>Expandable “What's happening?” details block listing the four pipeline stages with their typical durations</li></ul>Hidden again on success (the result HTML replaces the page) or on error (the upload area returns so the user can retry). The phase label can't observe sub-second server stages without streaming responses, but the elapsed timer + clear labelling makes the wait feel deterministic.

v0.15.186

2026-05-26

Per-pin matching cascade: local DINOv2 → SerpAPI Lens. Plus crop padding, 200 MP photo support, and EXIF orientation handling.

SerpAPI Lens cascade for board scans. Pre-v0.15.186, each detected pin on a board scan only got DINOv2 nearest-neighbor against the local catalog. If the pin wasn't in the catalog yet, or DINOv2's score was inconclusive, the user saw no match -- even when Google Lens could have found the pin on PinPics / Pin&Pop / PTDB. Cascade now:<ol style='font-size:0.85rem;'><li>Local DINOv2 on the crop. Fast (~100 ms) and free. If top match score ≥ 0.75 (the new serpapi_threshold), we stop here.</li><li>SerpAPI Lens on the crop's public URL when DINOv2 confidence was low. Visual matches with parsed catalog IDs (PinPics / Pin&Pop / PTDB) get looked up in the local DB and added to the candidate list. If Lens finds a match for a pin we don't have crawled yet, it surfaces as an uncrawled catalog candidate the user can ingest manually.</li></ol>Cost: SerpAPI fires only when local fails, so a board with well-crawled pins burns zero SerpAPI calls. Per-pin provenance badge on each match card (Local vs Lens) so the user knows which path won. Combined candidates re-sorted by score so the highest-confidence match from either source is the top one.

Crop padding: 15%. Gemini returns tight rectangles; pin metal often extends beyond (Mickey-head dangly bits, banner extensions, keychain rings). DINOv2 matches the catalog hero image which usually includes those extensions. Pre-v0.15.186 crops cut off pin edges on 6/8 of Drew's first scan. Each bbox is now expanded by 15% in every direction before cropping, clamped to image bounds.

33 MB photo upload now works. Drew's Galaxy Z Fold7 in 200 MP mode produces 16320×12240 (199.8 MP, ~33 MB JPEG). PIL's default decompression bomb guard refuses anything > 178 MP with DecompressionBombError. The server logged “Could not open image” and the user saw “didn't detect” with no explanation. Image.MAX_IMAGE_PIXELS = None at board_scanner.py import. Specific DecompressionBombError handler added as defense in depth (returns a useful error message instead of the generic open failure if the guard ever gets reset).

EXIF orientation applied via ImageOps.exif_transpose. Phone JPEGs often store pixels in the sensor's native orientation with a rotation tag for display. The client-side createImageBitmap downscale honours that tag, but raw-uploaded files that bypass the resizer would land on the server in their sensor orientation -- causing Gemini to see a sideways board. Applied at both the detection-time decode (board_scanner.detect_pins) and the crop-time decode (_enrich_board_pins_with_catalog_matches) so both pipelines agree on which way is up.

v0.15.185

2026-05-26

Three fixes after Drew's first end-to-end board scan: bbox coordinate format, CF-poisoned catalog cleanup, mobile camera UX.

1. Board scan bounding boxes were in nonsense positions. Gemini correctly detected 8 pins on Drew's Mickey-head display photo, but the green rectangles were drawn floating above the actual pins. Root cause: we asked Gemini for [x1, y1, x2, y2] normalized 0-1, but Gemini is heavily fine-tuned to return its canonical box_2d = [y_min, x_min, y_max, x_max] in integers 0-1000. Per Google's own docs and the SimEdw analysis on HN, the format is encoded into the model weights via post-training -- “even a tiny change like switching the order causes performance to tank.” The parser was reading Gemini's y-major 0-1000 output as if it were x-major 0-1, which:<ul style='font-size:0.85rem;'><li>Swapped X and Y axes (every box rotated 90°)</li><li>Clamped most values to 1.0 (since 0-1000 > 1)</li><li>Made box positions effectively random</li></ul>Fix: prompt now asks for Gemini's native format explicitly. Parser reads box_2d, divides by 1000, and converts to our internal [x1, y1, x2, y2] 0-1 representation. Legacy box key still accepted with auto-detected scale for defense in depth.

2. CF-poisoned catalog rows from before v0.15.155. Drew opened a catalog pin (#9305) and saw the name was the verbatim Cloudflare block-page title: “Access denied | www.pintradingdb.com used Cloudflare to restrict access | www.pintradingdb.com | Cloudflare”. v0.15.155 added CF detection in the parser so new fetches handle this correctly, but rows written before that fix still carry the poisoned name forever -- the crawler doesn't re-fetch existing rows. Two new endpoints to clean up:<ul style='font-size:0.85rem;'><li>GET /catalog-crawler/cloudflare-poisoned -- JSON listing of matching CatalogPin rows (preview).</li><li>POST /catalog-crawler/cloudflare-poisoned-reset -- NULLs name/year/series_name/super_series and deletes the matching cache file so the next crawler pass re-fetches through the working CF detector. Image filenames and row IDs preserved (referenced by user-side Pin rows).</li></ul>Triggered by a new “Reset CF-poisoned catalog rows” button on /catalog-crawler with explanatory confirm dialog.

3. Mobile camera button on scan-board. v0.15.166 removed capture="environment" from the file input because it forced camera and blocked library uploads. The intended replacement was the OS picker offering both, but Drew's phone surfaced library-only without showing the camera option. New design: two explicit buttons (camera + library), each wired to its own hidden input. The camera button uses capture="environment" for reliable rear-camera launch; the library button is bare. A JS handler copies whichever file got picked into the main form input via DataTransfer, so submission works unchanged. A “filename (NN KB)” label below the buttons confirms what was selected.

v0.15.184

2026-05-26

Third-party API keys: load from api_keys.env file. After Drew moved from Microsoft Store Python to C:\Python310\ with a fresh venv, every third-party integration showed OFF on /admin/external-apis: Gemini, Claude, GPT, eBay, Google Vision, SerpAPI, Copyseeker. Root cause: those keys were set as Drew's Windows user environment variables, which the LocalSystem service doesn't inherit (USERNAME under LocalSystem is DESKTOP-INOBSEM$, not Drew, so user-scope env vars don't reach the process).

Fix: parallel to vapid_keys.env. The launcher now also reads api_keys.env from the app directory using the same parser (KEY=value per line, # comments, blank lines skipped). Loaded keys are exported into the process environment before uvicorn.exe launches, so all the existing os.environ.get(...) code paths work without modification. The launcher logs which keys were loaded by NAME (never values) so the operator can verify what got picked up:
[launch] api_keys.env loaded (5 vars): GEMINI_API_KEY, ANTHROPIC_API_KEY, OPENAI_API_KEY, EBAY_CLIENT_ID, EBAY_CLIENT_SECRET

New file: api_keys.env.example. Documents every supported key with a short description and URL for getting one. Operator copies to api_keys.env, fills in just the keys they use, and restarts the service. The .example version is in source control; the real api_keys.env should be added to the existing zip-exclude list along with vapid_keys.env (already done; check the build command). Missing or empty keys keep the corresponding integration OFF without any error.

Bonus UX fix: scan-board disabled-state visibility. When GEMINI_API_KEY isn't set, the file-picker and submit button were already disabled (correct behavior — detection requires Gemini), but on dark themes the disabled state was nearly invisible. Drew clicked “Choose File” and nothing happened. Now the whole upload-area dims to 55% opacity, the cursor changes to not-allowed on hover, and a title tooltip explains the cause. Pairs with the existing red “Detection not available” banner so the cause is unmissable.

v0.15.183

2026-05-26

Hot-fix: v0.15.182's clear-transient-failures endpoint crashed with 500 and would have silently deleted bookmarklet import data.

Two bugs in the same code path, both stemming from the cache directory holding multiple file types from unrelated subsystems:<ul style='font-size:0.85rem;'><li>500 Internal Server Error: the data.get('ok') dict access wasn't guarded against non-dict json.loads() results. Empty files, files containing just null, or any JSON value that isn't a dict would trigger an AttributeError that escaped the inner except (which only wrapped the parse step). Drew hit this clicking the new button on a cache directory with a couple corrupt entries from May 13.</li><li>Silent data loss risk: the glob pinpics_*.json matched not just scrape-cache results but also pinpics_imported_*.json files written by pinpics_import.py. Those store bookmarklet payloads with a completely different schema (no ok, no error, no _version keys). With the old logic, data.get('ok') returned None, data.get('error') returned None, 'does not exist' in '' was False, so the function would have happily deleted them as “transient failures”. Catastrophic. Caught before Drew triggered it for real.</li></ul>Fix in two layers.<ol style='font-size:0.85rem;'><li>Schema guard: require the file to be a dict AND have one of the known cache-version markers (_version for pinpics, _schema_version for pinandpop/ptdb) AND have an ok key. Files that don't match are counted as kept_foreign and left untouched.</li><li>One try/except wrapping the entire per-file body. Any unexpected exception (malformed JSON, weird data shapes, OS-level deletion error, race with another process) increments the errors counter and logs but never crashes the batch.</li></ol>Flash message now surfaces both kept_foreign and errors counts so the operator can verify behavior at a glance:
Cleared 56 transient failures (3245 stable 404s kept, 12 successes kept, 2 non-cache files skipped, 0 errors)

v0.15.182

2026-05-26

Pinpics cache: tiered TTL by failure type. Pin 6706 finally fetched successfully after the v0.15.181 buffering fix exposed the real culprit: pinpics' inline cache logic used a flat 14-day TTL for everything, so a transient failure during the broken-headless window earlier today (v0.15.178/179 deploy) poisoned the cache for two weeks. Every retry returned the cached failure without running the fetch code, hiding all the real fixes.

New behavior matches the existing _read_cache helper that pinandpop and ptdb already use (lines 590-591: HIT_TTL_SECONDS=30d, MISS_TTL_SECONDS=1h):<ul style='font-size:0.85rem;'><li>Success (ok=True): 14d TTL (unchanged)</li><li>Stable failure (error contains ‘does not exist’): 14d TTL — PinPics has real gaps in its ID space, no point hammering with retries</li><li>Transient failure (CF challenges, headless errors, network errors, HTTP 403/429/500, parse errors): 1h TTL — auto-recovers within one crawler iteration after the underlying problem is fixed</li></ul>Existing cache files preserved — uses the same _version=v5 key so the thousands of legit 404 caches stay valid.

New endpoint + button: POST /catalog-crawler/clear-transient-failures. Deletes cached failure entries that look transient (anything except “does not exist” 404s) across pinpics/pinandpop/ptdb caches. Successes and stable 404s are NEVER touched. Returns a summary (scanned/deleted/kept_ok/kept_stable/errors). UI button added to /catalog-crawler next to the existing “Clear all” failure-log button with a confirm dialog explaining what gets deleted and what doesn't.

Operational note. With the 1h TTL, future transient bugs self-heal — deploy a fix, wait an hour, crawler picks back up. The clear-transient button is the “don't want to wait an hour” shortcut, not a routine maintenance task.

v0.15.181

2026-05-26

Disable Python stdout buffering under NSSM so diagnostic logs actually appear in real time. Drew's diagnostic run confirmed Playwright works perfectly in the venv (5/5 steps PASS, fetched pin 6706 and got 57,879 bytes of real PinPics content). But the service's logs/service-stdout.log showed nothing after the launcher banner for 90+ seconds. None of the expected diagnostic prints — [headless] playwright-stealth detected, [catalog_sites/pinpics] 6706: requests got Cloudflare challenge, [headless] Chromium launched — were visible. Earlier runs only had partial output (the BLOCKED lines made it through but the prints leading up to them didn't).

Root cause: Python output buffering. Python defaults to BLOCK buffering for stdout/stderr when those streams are connected to pipes instead of TTYs. NSSM captures stdout via a pipe, so Python buffers print() output in 4-8 KB chunks and only flushes when the buffer fills, the process exits, or someone calls sys.stdout.flush() explicitly. That made the service look broken when it was actually fine — the diagnostic output existed but was sitting in a buffer that never got flushed during low-activity periods.

Why it didn't happen before NSSM. Interactive PowerShell: stdout is a TTY, Python uses LINE buffering (flush on newline). Under NSSM: stdout is a pipe, Python uses BLOCK buffering. The launcher's PowerShell Write-Host calls flush immediately because they go through a different code path; only the Python child's stdout was block-buffered.

Fix (two layers).<ol style='font-size:0.85rem;'><li>service-launch.ps1: sets $env:PYTHONUNBUFFERED = '1' before exec'ing uvicorn. This forces Python to use unbuffered stdin/stdout/stderr regardless of what the streams are connected to — same effect as python -u on the command line, but works through uvicorn.exe which doesn't accept interpreter flags.</li><li>main.py: existing sys.stdout.reconfigure() call now also passes line_buffering=True. Defense in depth in case the env var doesn't propagate; even with block buffering at the C level, the Python TextIOWrapper layer flushes on newline.</li></ol>After deploy. Every BLOCKED line should now have an explanatory [headless] ... log line directly above it explaining the actual failure mode (timeout / launch error / specific Playwright exception). If pin 6706 still fails after this, the failure reason will finally be visible so we can target the actual bug. Drew can also run Get-Content logs\service-stdout.log -Tail 50 -Wait -Encoding UTF8 and see prints in real time instead of waiting for buffer flushes.

v0.15.180

2026-05-26

service-launch.ps1: auto-heal SYSTEM profile env vars so LocalSystem service still finds the user's AppData. After v0.15.179's warning correctly identified the service was running as a machine account (USERNAME=DESKTOP-INOBSEM$ instead of Drew), Drew found he couldn't easily provide a Windows password for the service to switch to .\Drew — he uses Windows Hello PIN exclusively and doesn't have his password handy. Resetting the password has DPAPI side-effects (loses access to saved browser passwords, BitLocker recovery edge cases, etc.).

Fix: keep LocalSystem, override the env vars. Nothing about the service's OS identity needs to change. LocalSystem has read access to everything in C:\Users\Drew\ — it just doesn't default to looking there. So the launcher now detects when $env:LOCALAPPDATA contains system32\config\systemprofile (the SYSTEM profile signature) and, when triggered, extracts the user from the current working directory (C:\Users\<USER>\Documents\PinApp) and overrides:<ul style='font-size:0.85rem;'><li>USERPROFILE</li><li>HOMEPATH / HOMEDRIVE</li><li>LOCALAPPDATA</li><li>APPDATA</li><li>TEMP / TMP (when user's Temp dir exists)</li></ul>Any subsequent process or library that reads these env vars — Playwright finding its Chromium, pip's wheel cache, Chrome user-data-dir, the Hugging Face transformers cache, anything — now sees Drew's folders instead of SYSTEM's.

Operational note. v0.15.179's PLAYWRIGHT_BROWSERS_PATH probe is still in place as defense-in-depth, but with the auto-heal running first, $env:LOCALAPPDATA already points at Drew's path by then so the probe's first candidate hits. No change to the service ObjectName needed; Drew can leave the service registered as LocalSystem and skip the credential prompt entirely. (The v0.15.176 credential prompt is still in the installer for users who prefer running as a named account.)

v0.15.179

2026-05-26

service-launch.ps1: resolve PLAYWRIGHT_BROWSERS_PATH explicitly so headless Chromium works under NSSM. After v0.15.178 deployed, Drew found that pin 6706 (and actually every PinPics fetch requiring the headless fallback) was failing with: BrowserType.launch: Executable doesn't exist at C:\WINDOWS\system32\config\systemprofile\AppData\Local\ms-playwright\chromium_headless_shell-1223\… The pin existed (Drew confirmed); the issue was that Playwright was looking in the SYSTEM profile's AppData, not Drew's.

Root cause. When NSSM runs a service as a user account, Windows applies the user's security identity (file ownership, registry permissions) but doesn't always load the user's full profile environment. Variables like %LOCALAPPDATA% and %USERPROFILE% fall back to the SYSTEM profile paths (C:\WINDOWS\system32\config\systemprofile\…). Playwright resolves its browsers directory relative to %LOCALAPPDATA%, so it ends up looking at a path where nothing is installed and where the service user can't write anyway.

There was also a secondary issue: when Drew rebuilt the venv with python.org Python during the NSSM migration, pip install -r requirements.txt got a newer Playwright version that wanted chromium_headless_shell-1223 — a newer Chromium build than what was cached in his AppData from a previous playwright install. So even if the path resolution were correct, the version subdirectory wouldn't exist until playwright install chromium ran against the new venv.

Fix in launcher. Probes candidate paths in order: explicit PLAYWRIGHT_BROWSERS_PATH if pre-set (respects manual override) → $env:LOCALAPPDATA\ms-playwright if that doesn't point at the SYSTEM profile → reconstructed C:\Users\$env:USERNAME\AppData\Local\ms-playwright ($env:USERNAME IS set correctly even when full profile isn't loaded). First existing directory wins. If none exist, logs a loud WARNING with the exact playwright install chromium command needed to fix it — so a future fresh machine doesn't silently fail.

Drew also needs to run, in an interactive PowerShell as his user (not as service):<pre style='font-size:0.85rem;'>cd C:\Users\Drew\Documents\PinApp .venv\Scripts\playwright install chromium</pre>to populate the new chromium_headless_shell-1223 directory in his AppData. After that, nssm restart DisPindable and the headless fallback will start working again. The launcher's PLAYWRIGHT_BROWSERS_PATH detection takes care of the rest.

v0.15.178

2026-05-26

PinPics & all crawler fetches: fix the UTF-8 stdout cascade that broke every headless fetch under NSSM. Drew shipped v0.15.177, then noticed all PinPics fetches were failing with requests got Cloudflare challenge followed by headless: returned empty/None. He confirmed it wasn't an IP block (could hit the page himself in the browser). The actual cause was a dual encoding/state bug triggered by the v0.15.174 NSSM migration.

Root cause part 1: charmap codec. Windows defaults Python's stdout encoding to cp1252 when stdout is a pipe (NSSM's log file) instead of a TTY. Non-ASCII characters in print() — em-dash, smart quote, Japanese pin name, the “→” arrow in a CDN URL — crash with UnicodeEncodeError under strict error handling. The first such crash happened inside playwright's launch path, raising from _ensure_browser().

Root cause part 2: half-init cascade. The crash left module-level _playwright set but _browser None. Subsequent calls to _ensure_browser ran sync_playwright().start() a SECOND time on top of the wedged first instance, which playwright detected and refused with the (misleading) error “It looks like you are using Playwright Sync API inside the asyncio loop.” Every PinPics fetch from that point forward got the requests-CF challenge (normal) → fell to headless (permanently broken) → BLOCKED → poison-skip → same cycle on the next pin. Drew was correctly skeptical that it was actually an asyncio issue.

Why this didn't happen pre-NSSM. Interactive PowerShell: stdout is a TTY, Python uses the console code page with replace error handling — bad chars become ‘?’ silently. NSSM: stdout is a pipe, Python uses cp1252 with strict — bad chars crash. Same Python, same code, different stdout target, different encoding behavior.

Fix (3 layers).<ol style='font-size:0.85rem;'><li>service-launch.ps1: sets $env:PYTHONIOENCODING = 'utf-8' before exec'ing uvicorn, so the Python child process uses UTF-8 for all I/O encoding regardless of the host locale.</li><li>main.py (very top, before any other module's imports run): calls sys.stdout.reconfigure(encoding='utf-8', errors='replace') on both stdout AND stderr. Defensive layer in case the env var path doesn't fire (CI, scheduled-task, dev-machine PowerShell that didn't inherit the var, etc.). errors=replace means a future encoding slip degrades to ‘?’ rather than crashes — defense in depth.</li><li>headless_scrape.py: new _reset_playwright_state() helper, called from the top of _ensure_browser() when half-init is detected (_browser None but _playwright set). Recovery for any future situation where launch raises mid-flight, so a wedged session can recover on the next call instead of staying broken until process restart.</li></ol>After deploy. Drew should clear .dpb_cache\pinpics_cookies.json before restarting the service. The persisted cookies may be polluted by the failed runs (partial CF clearance, expired session tokens). Letting the crawler cold-start the next session gets a fresh handshake.

v0.15.177

2026-05-26

Gate /dashboard behind login. After NSSM started capturing real stdout, Drew noticed external IPs hitting /dashboard and getting 200 OK in the service logs (106.52.216.13, 43.166.244.66 — Tencent Cloud / China). Audit of the route showed no auth check — /dashboard was the only admin-stats route without an auth.current_user(request) guard, while peer routes (collection dashboard, settings, etc.) all guard with the canonical pattern.

Fix. Added the same 3-line guard at the top of dashboard_page: get the user, if absent redirect to /login?flash=Login+required. Identical to the pattern in user_dashboard_page and elsewhere — no new auth machinery, just bringing this route in line with the rest.

Aggregate stats (total pins, paid totals, MSRP totals) are now no longer world-readable. After deploy, the Tencent Cloud scanners will get a 303 to /login instead of everyone's collection value.

v0.15.176

2026-05-26

NSSM installer: configure service to run as the user account, not LocalSystem. Drew got the v0.15.175 installer running cleanly, then the service entered SERVICE_PAUSED state because uvicorn exited code 101 in 55 ms with empty stdout/stderr.

Root cause. NSSM defaults to LocalSystem for the service account. That account can't bootstrap a venv created under a user profile — the uvicorn.exe launcher in .venv\Scripts\ needs to find its base Python interpreter (often installed per-user at %LOCALAPPDATA%\Programs\Python\) and LocalSystem doesn't see those installation paths. The process exits before Python even starts, which is why stdout and stderr were both empty.

Fix. Installer now prompts for credentials and calls nssm set ObjectName .\<user> <password>. NSSM passes the password to the Windows Service Control Manager which stores it encrypted — it's never written to disk in plaintext and PowerShell clears its in-memory copy after the call.

UX. Default-Y prompt with the current user account (.\$env:USERNAME) pre-populated. Native Windows credential dialog via Get-Credential — same UX as any other credential prompt the user is familiar with. Skipping the prompt is permitted but produces a clear warning that the service will likely fail to start.

For Drew's in-flight installation. Two manual fix paths in the chat for the half-installed service from v0.15.175: services.msc → Log On tab, or nssm set ObjectName via PowerShell. After applying, run nssm reset DisPindable AppExit to clear the PAUSED state, then nssm start.

v0.15.175

2026-05-26

NSSM service installer: fix strict-mode interaction with nssm's “Can't open service!” stderr. Drew tried to run install-as-service.ps1 from v0.15.174 and got nssm.exe : Can't open service! as a PowerShell NativeCommandError, aborting the script before it reached the install branch.

Root cause. NSSM's status subcommand returns non-zero exit code + writes “Can't open service!” to stderr when the named service doesn't exist. The v0.15.174 script used $ErrorActionPreference = 'Stop' at the top (good for catching PowerShell-native errors like missing files), but Windows PowerShell 5.1 treats ANY native-command stderr write as a NativeCommandError under that preference. So the “is there an existing service to clean up?” probe killed the whole script on a fresh machine where no service exists yet.

Fix. Three changes to install-as-service.ps1 and uninstall-service.ps1:<ol style='font-size:0.85rem;margin-top:0.5rem;'><li>Use Get-Service -ErrorAction SilentlyContinue for existence checking (PowerShell-native, returns $null when missing, no exit-code games).</li><li>Cleanup-branch nssm calls (stop, remove) wrapped in cmd /c "… 2>&1" so the cmd shim swallows the non-zero exit before PowerShell sees it.</li><li>The whole post-install configuration block (15+ nssm set calls) wrapped in a try {} finally {} with locally-relaxed $ErrorActionPreference = 'Continue', restored at the end. Plus a final Get-Service verification so we fail loudly if anything in there actually went wrong.</li></ol>Drew can now re-run the v0.15.175 installer cleanly on a fresh box. No other changes — same logging paths, same restart policy, same Ctrl+C grace period.

v0.15.174

2026-05-26

Windows service wrapper (NSSM). After the 2026-05-26 06:22 UTC power-loss crash (Kernel-Power Event 41 with BugcheckCode=0), Drew opted to skip the UPS path and instead capture uvicorn properly via Non-Sucking Service Manager. Three new files ship with this version:

<table style='font-size:0.85rem;border-collapse:collapse;'><tr><th style='text-align:left;padding-right:1em;'>File</th><th style='text-align:left;'>Role</th></tr><tr><td>service-launch.ps1</td><td>PowerShell launcher NSSM exec's. Production-mode equivalent of start-dispindable.bat minus --reload and the interactive pause. Loads vapid_keys.env, picks HTTPS/HTTP based on cert.pem presence, execs uvicorn so its stdout flows through to NSSM's captured log files.</td></tr><tr><td>install-as-service.ps1</td><td>One-time setup. Registers the ‘DisPindable’ service, configures logs\service-stdout.log + logs\service-stderr.log with 10 MB online rotation, sets auto-start at boot, sets restart-on-crash with 5s throttle, sets graceful Ctrl+C shutdown with 30s grace period (enough for in-flight crawler row to commit). Requires admin PowerShell + NSSM installed via winget install NSSM.NSSM.</td></tr><tr><td>uninstall-service.ps1</td><td>Clean removal. Logs preserved.</td></tr><tr><td>SERVICE_SETUP.md</td><td>Operator's guide. Daily commands, deployment workflow, going-back-to-manual instructions.</td></tr></table>
What this fixes for future crashes.<ul style='font-size:0.85rem;'><li>Auto-recovery. If the 06:22 crash had happened with NSSM in place, the service would have restarted itself ~5 seconds after Windows booted at 06:22:50, instead of staying down until Drew noticed.</li><li>Forensic capture. Any FUTURE Python crash (segfault, OOM, unhandled exception) will leave a full traceback in logs\service-stderr.log. The power-loss crash had no traceback because stdout was only flowing through Windows Event Log forwarder, not captured to disk in real-time.</li><li>Survives sessions. Closing the PowerShell window, RDP-disconnecting, logging out — none of these kill the service.</li><li>Survives Windows reboots. Start=SERVICE_AUTO_START means the app comes up with the OS without manual intervention.</li></ul>What this doesn't fix. Power-loss crashes will still happen until Drew addresses whatever is causing them (the diagnosis was definitive on power loss but Drew wants to investigate the cause separately). What changes is recovery time and forensic quality: minutes-to-hours of downtime → seconds; opaque “process just died” → full traceback (when applicable).

Migration. See SERVICE_SETUP.md. start-dispindable.bat is unchanged — you can still launch manually any time. Switch to the service when you're ready; switch back via uninstall-service.ps1 any time.

v0.15.173

2026-05-25

Scan-board: fix “tells me to select smaller file” on huge phone photos + add resolution selector. Drew hit two real issues after Phase D shipped: (1) huge phone photos (200 MP) failed the client-side downscale step, with the error path showing “Upload failed: image decode failed. Try a smaller photo or different browser” — nowhere to go from there. (2) For testing detection accuracy at lower resolutions, there was no way to opt into a smaller upload.

Root cause of issue 1. The v0.15.168 downscale path used an Image element to decode the file before resizing on a canvas. For 200 MP files (which decode to ~800 MB of raw pixels), the browser exceeds its per-canvas memory limit (Chrome caps around 256 MB, Safari is lower) and silently fires onerror. Our handler reported “image decode failed,” which sounds like a bad file even though the file is perfectly valid — just too big for the browser to hold in memory at once.

Fix for issue 1. Switched to createImageBitmap() with resizeWidth/resizeHeight/resizeQuality options. The browser can stream-decode the file while downscaling, never holding the full-resolution pixel buffer in memory. Two-pass: first decode probes dimensions (browsers often keep this as a GPU-side handle, not full pixels); second decode actually resizes. Falls back to the v0.15.168 Image element path on browsers without createImageBitmap support (Firefox pre-42, Safari pre-11).

Fix for issue 2: resolution selector. New dropdown below the file picker: 4096 px (default, recommended), 2048 px (faster upload / less Phase 2 detail), or 1024 px (for testing low-res accuracy). Drew can now A/B detection quality at different resolutions to find the sweet spot. The 1024 px option is also a safe fallback if 4096 still fails for some reason.

Better error messages. When the decode fails despite the createImageBitmap path, the new error explicitly points at the resolution dropdown: “browser could not decode this image. It may be too large for your device's memory; try the 2048 or 1024 px option above.” And the top-level catch block now adds “Try selecting ‘2048 px’ or ‘1024 px’ in the resolution dropdown above” to any failure message.

v0.15.172

2026-05-25

Pinboard scan: “Use this match” per-pin button (Phase D). Phase C added per-pin catalog matching to the board scan; this ships the action button on each per-pin card. One click takes you from a detected pin to a fully pre-filled /new-pin form, skipping AI identification entirely. Lets you ingest an entire board as identified pins a few clicks at a time.

How it works. At board scan time, each detected pin's crop is now saved to UPLOADS/board_crop_{ts}_{idx}.jpg (Phase C computed crops in memory and discarded them). The filename is attached to the pin dict. Clicking “Use this match” POSTs the crop filename + chosen catalog_pin_id to /scan/board/use-pin-match, which creates a new Scan record referencing the crop as its single uploaded image, sets catalog_match_used_id (same column the v0.15.170 /scan flow uses), and redirects to /scan/{id}/new-pin. From there the existing catalog-prefill machinery takes over: the form pre-fills with the catalog row's data and the user just clicks save.

Defenses. The new endpoint validates: (a) crop_filename has no path traversal characters, (b) the file actually exists in UPLOADS, (c) catalog_pin_id resolves to a real CatalogPin row. Invalid input returns 400 with a clear message rather than 500ing. If the server was restarted between board scan and click, the user gets a specific “please re-upload the board photo” error instead of an opaque crash.

Storage cost. A typical board has 20-30 pins; each crop is ~30-80 KB at JPEG q88. So ~1-2 MB per board scan in UPLOADS. Crops persist indefinitely (a future cleanup job could prune crops that are older than N days AND not referenced by any saved Pin). Not urgent yet.

What's the user flow now?<ol style='font-size:0.85rem;margin-top:0.5rem;'><li>Upload a board photo at /scan/board.</li><li>Gemini detects pin bounding boxes; server crops each and runs the catalog match.</li><li>Per-pin cards appear below the overlay; HIGH CONFIDENCE matches (≥ 0.90) have green “Use this match” buttons.</li><li>Click “Use this match” on a pin you want to ingest. Lands on /new-pin with the catalog row pre-loaded.</li><li>Save the pin. Land on /pins/{id}.</li><li>Navigate back to /scan/board and re-upload the same photo to ingest the next pin (the board scan is transient; the crops persist but the result page state doesn't survive). A future version can persist board scans so re-uploading isn't required.</li></ol>

v0.15.171

2026-05-25

Pinboard scan: per-pin catalog matching (Phase C). After Gemini detects pin bounding boxes on a board photo, each detected pin is now cropped from the original image, embedded with DINOv2, and run through catalog_matching.find_catalog_matches from v0.15.169. The result page shows a per-pin matches panel below the bounding-box overlay: every pin that found a candidate in your local catalog is listed with its top match (name, score, source IDs) plus expandable runners-up.

How the cropping works. The bounding boxes from board_scanner.detect_pins are normalized 0–1 coordinates relative to the original image. We decode the original once via PIL, then crop each pin region to pixel coordinates, re-encode as JPEG at quality 88, and send through the standard embeddings pipeline. At the v0.15.168 client-side cap of 4096 px, typical pin crops land at ~200–400 px on a side — comparable to your catalog image dimensions and good enough for reliable DINOv2 matching.

Visual treatment. Each pin's panel has a colored left border keyed to confidence: green for ≥ 0.90 (with a HIGH pill), amber for 0.75 – 0.89, grey below. The header shows the detection index + Gemini's self-reported confidence; the body shows the match name + score + source IDs (PinPics, Pin&Pop, PTDB). Additional candidates (top-3 by default) are tucked under a “more candidate(s)” disclosure so the page doesn't explode on dense boards.

Performance. A board with 30 pins adds ~30 × ~150 ms = ~4 seconds to the response (crop + embed + catalog lookup, with the embedding-cache warm). For boards with sparse catalog coverage, most pins show no matches and the user can still rely on the bbox overlay as a starting point. Per-pin failures (degenerate bbox, decode error) are caught and the pin just gets an empty list — never blocks the rest of the result.

Next: Phase D. Each per-pin match panel gets a “Use this match” button (same pattern as the single-pin /scan flow in v0.15.170). Clicking it starts a /new-pin flow seeded with the catalog row, so the user can ingest the entire board as identified pins in a few clicks per pin instead of one full scan each.

v0.15.170

2026-05-25

Catalog visual matching (Phase B): short-circuit AI when the catalog already knows the pin. Building on v0.15.169's catalog-match infrastructure, this ships the actual short-circuit: when the local catalog has a visual match, the user can skip AI identification entirely and go straight to /new-pin with the catalog row pre-loaded.

New “Use this match” button on each card in the “📚 Catalog matches” section of /scan/{id}. HIGH CONFIDENCE matches (score ≥ 0.90) get the button colored green; others stay neutral so the user can still click but with appropriate skepticism. Clicking POSTs to a new /scan/{id}/use-catalog-match endpoint that stores the chosen catalog_pin_id on the scan and redirects to /new-pin.

Synthetic suggestion in /new-pin. When scan.catalog_match_used_id is set, the /new-pin handler fetches the catalog row and synthesizes a suggestion dict in the same shape AI providers emit (name, pinpics_id, pinandpop_id, ptdb_id, year, edition_size, etc.) with _provider_slug="catalog-match" and _vote_weight=3. Inserted at the TOP of _prefill_priority so it wins automatically, and joins merge_catalog_ids() at the top of the vote pool so the catalog IDs win the cross-reference vote too. AI buttons stay on the page so the user can still trigger them if they want to second-guess — this just changes the DEFAULT path.

Auto-use setting (default off). Two new app settings: auto_use_catalog_match (bool, default False) and catalog_match_auto_threshold (float, default 0.95). When auto-use is ON and the top catalog match in POST /scan exceeds the threshold, the user is redirected straight from /scan to /new-pin — never seeing the scan results page at all. Defaults to OFF so Drew can calibrate the threshold against real scans before turning on the fully-automatic path.

Cost savings. Each AI provider call (Gemini Pro, Claude, OpenAI deep, Opus) costs $0.01-0.10 and takes 10-30s. A typical multi-AI scan was costing $0.10-0.40 + 60-120s. For pins already in the catalog, that's now $0 + ~50ms (the catalog-match search itself). Drew's 50k catalog (going to 500k) covers most Disney pins out there — the AI fallback becomes the rare path, used only for genuinely new or low-confidence pins.

v0.15.169

2026-05-25

Catalog visual matching (Phase A): match scans against the local catalog mirror before calling AI providers. Drew has 50k catalog pins now and is on the way to 500k. When the user uploads a scan and the catalog already has a visual match, paying for AI calls (Gemini, Claude, OpenAI) is wasted money and time — everything those AIs would have told us (PinPics ID, Pin&Pop ID, name, etc.) is right there in the catalog row already.

New module: catalog_matching.py with find_catalog_matches(query_blob, top_k=5). Module-level in-memory cache loads all CatalogPin.embedding vectors into a numpy matrix on first use; subsequent queries are a single matrix-vector multiply (~20-50 ms at 50k rows). Cache self-invalidates when the catalog grows by >5% so the results stay current during active crawls. At 50k rows the matrix is ~150 MB; at 500k it'll be ~1.5 GB. We'll re-evaluate (FAISS, pgvector) around 200k rows.

POST /scan integration. After the existing user-pin matching, the handler also embeds the first uploaded image and queries the catalog. Top-5 matches (score + catalog_pin_id + source IDs + name) are stored on the Scan row in the new catalog_top_matches_json column. Failures are logged and don't break the scan.

UI: /scan/{scan_id} page now shows a “📚 Catalog matches” section alongside the existing user-pin matches. Score ≥ 0.90 gets a green “HIGH CONFIDENCE” pill; 0.75–0.89 amber; below 0.75 grey. Each card links to the local catalog detail page + the source pages on PinPics, Pin&Pop, PTDB.

Phase B (next release): “Use this catalog match” button on the scan page that skips AI entirely and goes straight to /new-pin with the catalog row pre-loaded. Threshold tuning. Phase C (later): apply catalog matching to pinboard scans — crop each detected pin from the board photo and run it through find_catalog_matches, surfacing per-pin identification suggestions as a first pass.

v0.15.168

2026-05-25

Scan-board client downscale: 2048 → 4096 px to preserve Phase 2 identification quality. Drew flagged an inconsistency in v0.15.167: the camera-tip in the UI says “shoot at 50–200 MP for better detail” while the JS was downscaling everything to 2048 px before upload — defeating the entire reason for shooting high-res in the first place.

The tradeoff matters because of two distinct phases.<table style='font-size:0.85rem;border-collapse:collapse;'><tr><th style='text-align:left;padding-right:1em;'>Phase</th><th style='text-align:left;'>What it needs</th><th style='text-align:left;'>Min source res</th></tr><tr><td>1 — Detection (live today)</td><td>Bounding boxes around each pin on the board</td><td>2048 px (Gemini)</td></tr><tr><td>2 — Identification (planned)</td><td>Crop each pin, run DINOv2/CLIP, match against catalog</td><td>4096 px+ (so crops are ~200–400 px, comparable to catalog images)</td></tr></table>
Fix. Client cap goes from 2048 → 4096. Uploads are now ~1–2 MB (vs ~400 KB at 2048, vs 8–12 MB raw from phone) — still well under the timeouts that triggered the original ERR_HTTP2_PROTOCOL_ERROR. The server continues to downscale to 2048 px internally for Gemini detection, so Phase 1 behavior is unchanged. The extra pixels are preserved for Phase 2 when it ships.

Camera-tip text updated to reflect reality: shooting beyond 50 MP doesn't help because the browser caps at 4096 px before upload anyway. 12 MP (default on most phones) is plenty for Phase 1 today and produces a 4096 px-friendly source image for Phase 2 later.

v0.15.167

2026-05-25

Scan-pin-board: client-side downscale before upload. After v0.15.166 fixed the file picker, Drew tried to upload a board photo and got ERR_HTTP2_PROTOCOL_ERROR. Root cause: phone board photos are typically 8–12 MB, big enough to trip intermediate timeouts (browser, reverse proxy, uvicorn worker keepalive) mid-upload — the HTTP/2 stream gets reset before the request body finishes. The server downscales to 2048 px before sending to Gemini anyway (board_scanner.MAX_DETECTION_DIM), so the upload bandwidth was pure waste.

Fix. Added a small JS preprocessor on /scan/board that runs before form submission: decode the file into an Image, draw onto a canvas at the same 2048 px cap the server uses, re-encode as JPEG at 85% quality, then POST that smaller blob via fetch(). Uploads drop from 8–12 MB to ~300–500 KB. Detection quality is identical because the server would have downscaled to the same resolution either way.

Progressive enhancement. If the browser lacks canvas.toBlob (very old) or can't decode the file (HEIC on a browser without support, corrupt image), the JS catches the error and shows a specific error message rather than dying silently. The form falls back to a normal multipart POST if JS is disabled entirely, so no-JS users still work the same way they did pre-v0.15.167.

UX improvements bundled. Status line next to the submit button now shows progress: “Preparing image (8.4 MB)…” → “Uploading 2048×1536 (412 KB, from 4032×3024 / 8.4 MB)…” → “Processing…”. Submit button disabled during upload to prevent double-submit. On failure, the button re-enables and the status line shows the error in red.

v0.15.166

2026-05-25

Scan-pin-board file picker: allow selecting existing photos. Drew tried to upload a previously-taken board photo via /scan-board and the page jumped straight into the camera every time, with no way to pick an existing file. The culprit: the file input had capture="environment", an HTML attribute that tells mobile browsers to skip the file picker and open the rear camera directly. Useful when you definitely want a fresh photo. Wrong here — the user often wants to upload a board shot they took earlier.

Fix. Removed the capture attribute. Mobile browsers now show the native OS picker, which has both “Take Photo or Video” and “Choose from Library” options. Desktop is unchanged (it ignored capture anyway). accept="image/*" stays so non-image files are still filtered out.

v0.15.165

2026-05-25

DB connection pool exhaustion fix. Drew tried to scan a pin while the crawler was running and got a 500 on the dashboard with sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached. Two compounding root causes.

Root cause 1: backfill loop held DB connection during network fetch. The backfill thread's row loop wrapped the whole iteration in a with db.SessionLocal() as s: block — including the catalog_sites.fetch_pinpics(force=True) call, which can take 30+ seconds when headless is solving a Cloudflare challenge. Every backfill iteration held a connection from the pool for the entire fetch. Multiply by 3 sources + status polling every 1s from the UI + user-initiated requests, and the 15-connection pool exhausted constantly.

Fix. Restructured the backfill loop into three phases: (1) open session, read row metadata into primitives, close session; (2) fetch over the network with no DB session held; (3) open a fresh session for the upsert, close. Connection is held only during the actual SQL operations — milliseconds instead of tens of seconds.

Root cause 2: pool was sized for a single-user synchronous app. Pre-v0.15.165 db.py passed no pool config to create_engine, so we got SQLAlchemy's defaults: pool_size=5, max_overflow=10, pool_timeout=30. That's 15 concurrent connections for the whole app and a 30-second wait when they're all busy — which is why the dashboard hung half a minute before erroring out.

Fix. Explicit pool config: pool_size=20, max_overflow=30, pool_timeout=10, pool_pre_ping=True, pool_recycle=3600. SQLite connections are just file handles, basically free — 50 concurrent is fine. The 10-second timeout surfaces problems fast instead of locking the UI half a minute. pool_pre_ping avoids stale-connection errors after WAL rotation. pool_recycle bounds connection lifetime so they can't accumulate stuck state forever.

Known follow-up: similar pattern in catalog_ingest._enrich_series_from_pinandpop(). That code is also user-initiated (series-expand action) and loops through fetches while holding a session. Not triggering the bug right now because it's user-action rate, not crawler rate, but should be restructured to the same read-fetch-write pattern in a future release.

v0.15.164

2026-05-25

Cloudflare clearance cookie carry-over from headless to requests. Drew flagged the deeper question after v0.15.163: are we fixing the actual root cause? Answer: no, v0.15.163 just stopped pin 6706 from pinning the crawler. The real underlying bug was that requests.get was being called bare (no session) on every fetch, so any cf_clearance cookie that headless earned by solving a Cloudflare challenge died with the playwright context. Next requests call was naked again, got challenged from scratch, fell back to headless. Headless succeeded, earned clearance, threw it away. Repeat forever — and every cycle leaked another chrome process.

The fix has three layers.<table style='font-size:0.85rem;border-collapse:collapse;'><tr><th style='text-align:left;padding-right:1em;'>Layer</th><th style='text-align:left;'>What changed</th></tr><tr><td>Module-level Session</td><td>catalog_sites._PINPICS_SESSION = requests.Session() — replaces bare requests.get. Cookies persist across calls within the uvicorn process.</td></tr><tr><td>Headless → session cookie injection</td><td>After every successful headless fetch (HTML ≥ 2KB), headless_scrape captures context.cookies() before the playwright context closes. catalog_sites pulls them via the new get_last_cookies() accessor and injects them into the session jar. Next requests.get now carries cf_clearance.</td></tr><tr><td>Disk persistence</td><td>Cookies written to .dpb_cache/pinpics_cookies.json after each injection. Loaded on module import. Most cf_clearance cookies have a ~30-min TTL, so a uvicorn restart within that window keeps the clearance instead of cold-starting back into challenges.</td></tr></table>
Expected impact. First poison-pin-like fetch still goes through headless (and earns cookies). Subsequent fetches in the same session bypass CF entirely via the persisted cookies — no headless launch, no chrome process spawn, no chrome process leak. The 21-chrome-procs-at-startup that v0.15.162 telemetry caught should drop to single-digit counts. Crawl speed on PinPics should also jump significantly (typical headless fetch is 2-5s, typical session-cookie-equipped requests.get is 100-300ms).

Gap accounting (no code change, just confirmation). Drew also asked whether the crawler accounts for “gaps” from poison-pin skips. It does: every failed fetch (including the 3 retries that led to the skip) writes a row to the catalog_crawl_failures table via _log_failure(). Visible at /catalog-crawler?failures=1. The recover_state_from_catalog path derives current_id from MAX(pin_id) + 1, so manually retrying a skipped pin and landing it in the catalog naturally fills the gap on the next state recovery. Failed pins can be re-fetched in bulk via the “Retry failures” button on the crawler page.

v0.15.163

2026-05-25

Poison-pin handling + correct consecutive_blocks semantics + structured fetch error messages. Drew watched the crawler get stuck on PinPics pin 6706 for hours. The pin loads fine in his browser, but the crawler kept logging pin 6706 BLOCKED: could not fetch page (requests blocked, headless unavailable or failed) on every retry, hit the 5/5 run cap, and disabled the source — with zero forward progress. Investigation revealed three compounding bugs.

Bug 1: blocks never advance current_id. By design — for a genuine Cloudflare wall you want to retry the same pin after backoff, not skip it. But when ONE specific pin consistently fails (a poison pin), the crawler retries it forever and never moves on.
Bug 2: consecutive_blocks counted retries, not unique pins. Pre-v0.15.163, three retries of the same pid looked identical to three different pids blocking. The safety counter (v0.15.122, “10 consecutive blocks → pause”) was firing on poison-pin spam instead of real site-wide blocks.
Bug 3: the error message was unhelpful. “requests blocked, headless unavailable or failed” doesn't tell you whether requests got a CF challenge, a 500, a thin 200, a connection refused, OR whether headless wasn't installed, timed out, crashed on launch, etc.

Fixes.<table style='font-size:0.85rem;border-collapse:collapse;'><tr><th style='text-align:left;padding-right:1em;'>Fix</th><th style='text-align:left;'>What changes</th></tr><tr><td>Poison-pin skip</td><td>New per-source state field same_pin_blocks tracks retries of the current pin. After _POISON_PIN_THRESHOLD=3 retries on the same pid, the crawler logs “poison-pin skip”, advances past it, and treats it as a permanent failure (logged in the Recent failures table for manual retry later).</td></tr><tr><td>consecutive_blocks semantics</td><td>Now counts UNIQUE pids only. Retries of the same pid no longer bump the safety counter, so the 10-block threshold fires only on real site-wide blocks. Poison-pin skips also decrement consecutive_blocks since they prove the issue was the pin, not the site.</td></tr><tr><td>Structured fetch errors</td><td>PinPics fetch now reports a pipe-separated chain of specific failures per layer, e.g. fetch failed: requests: Cloudflare challenge (520b) | headless: TimeoutError: timeout 25000ms. Distinguishes thin-200 (200 OK but suspiciously short body), non-200 HTTP, CF challenge, request exception, headless not available, headless empty return, and headless exception.</td></tr></table>
Operational note from this incident. Drew's v0.15.162 crash telemetry caught a chrome_proc_count=21 spike right after start. Those are accumulated leaked chromes from prior sessions that retried pin 6706 hundreds of times each — every headless attempt leaked at least one shell process. Once the poison-pin skip is live, future sessions will only leak (up to) 3 chromes per bad pin instead of unbounded. Separate task: investigate why the headless lifecycle in headless_scrape.py doesn't reliably clean up its chrome procs (followup needed).

v0.15.162

2026-05-25

Crash forensics telemetry for the catalog crawler. Drew reports the FastAPI app is hard-locking or rebooting his Windows machine during long crawl runs. Hard locks can't be diagnosed from logs that didn't flush before the lock, so this release adds a background sampler that captures system + process state to logs/crash_telemetry.jsonl every 10 seconds (configurable in /settings), calling os.fsync() after each write so a hard lock 100 ms later still preserves the last record.

What gets captured per sample.<table style='font-size:0.85rem;border-collapse:collapse;'><tr><th style='text-align:left;padding-right:1em;'>Field</th><th style='text-align:left;'>Why it matters</th></tr><tr><td>py_rss_mb</td><td>Python process resident memory — growth = memory leak in our code</td></tr><tr><td>py_threads, py_open_handles</td><td>Thread / handle leak detection</td></tr><tr><td>sys_mem_pct, sys_mem_avail_gb</td><td>System memory pressure — precursor to swap death spiral / OOM</td></tr><tr><td>sys_cpu_pct</td><td>CPU pegging — tight loop or thermal precursor</td></tr><tr><td>disk_pct, disk_free_gb</td><td>Cache / log / db growth filling C:\</td></tr><tr><td>net_rx_mb, net_tx_mb</td><td>Network throughput (cumulative; UI diffs adjacent samples)</td></tr><tr><td>chrome_proc_count, chrome_rss_mb</td><td>The chrome-headless-shell.exe leak watch. Same failure mode v0.15.154 partially fixed — counts processes by name + sums their RSS</td></tr><tr><td>crawler.*</td><td>Per-source current_id, blocked counter, request count — correlates system state with crawler progress at the moment of the crash</td></tr></table>
Windows Event Viewer alerts. When a metric crosses its threshold (memory ≥ 85%, chrome procs ≥ 8, disk ≥ 90%, py RSS ≥ 4 GB, threads ≥ 200), a WARNING is logged to the Windows Application event log via eventcreate.exe under Source: DisPindable. Cross-once semantics — fires on the FIRST breach, re-arms only after the value drops back below the limit, so sustained high utilization doesn't spam the event log. These entries survive a reboot and are visible in eventvwr.msc under Applications and Services Logs → Application.

Recovery view. New page at /catalog-crawler/telemetry renders the last 200 samples (configurable via ?n=<num> up to 2000) with threshold-cross highlighting. After a reboot, open this page to see what state the machine was in immediately before the lock. Heuristic in the page: if the log ends with a session_start but no matching session_end and the sampler isn't currently running, the page shows a red banner — that's the hard-lock signature. The last periodic record is the smoking gun.

Corrupt-line handling. If a write was interrupted mid-flush (e.g., the actual hard lock case), the trailing partial JSON line is preserved in the view as a corrupt_line record with the raw bytes — this is itself forensically meaningful (proves the lock happened during an I/O operation).

Sampler runs only while the crawler is running. Default ON via crash_telemetry_enabled; sampling interval via crash_telemetry_interval_s (range 1-300, default 10). Log rotates at 50 MB to bound disk usage.

v0.15.161

2026-05-25

Per-block + per-error stdout logging in the main crawler loop. Drew started a PinPics crawl with max_requests_per_run=5, watched the UI “blocked” counter climb to 5, hit the run cap, and saw the source disable itself — with zero lines in the uvicorn console explaining what was actually failing. Investigation: the block-error handler was calling _log_failure() (writes to the SQL failure-log table) but had no print(). The only stdout log fired at the 10-consecutive-blocks threshold, which a run capped at 5 can never reach. Result: blocks accumulated silently.

What changes. Both block-error and catch-all-error branches now print before mutating state. Format:
[crawler:pinpics] pin 12345 BLOCKED: Cloudflare challenge detected
[crawler:ptdb] pin 9876 ERROR: connection timed out

404s stay silent — they're too noisy at the start of a crawl (dead-zone ID ranges produce hundreds in a row) and the SQL failure-log + the contiguous-failure-ranges UI already give visibility on them. Successes also stay silent — the UI counter is the right place for forward-progress signal.

Matches the backfill thread's logging pattern from v0.15.122, which has been printing per-block lines all along. Now the main loop is consistent with it.

v0.15.160

2026-05-25

Block-threshold override now has buttons on the crawler page. v0.15.159 shipped the override behavior + a checkbox on /settings, but Drew pointed out that when blocks are piling up on /catalog-crawler, you don't want to navigate away to flip a toggle. Two new buttons sit alongside Start / Pause / Stop:

<table style='font-size:0.85rem;border-collapse:collapse;'><tr><th style='text-align:left;padding-right:1em;'>Button</th><th style='text-align:left;'>What it does</th></tr><tr><td>Block override: OFF / ON</td><td>Toggles the persistent crawler_ignore_block_threshold app_setting. Same toggle as /settings, just exposed where you're actually watching the blocks accumulate. When the env var DPB_CRAWLER_IGNORE_BLOCKS is set, the button shows “Block override: ON (env)” and is disabled, because the env var wins regardless.</td></tr><tr><td>Clear active pause & resume</td><td>One-shot action. Wipes last_blocked_until on all sources right now, so any source stuck in the 30-min auto-pause window resumes on the next loop iteration. Doesn't change the persistent override setting — useful when you've just rotated your IP and want to resume immediately but keep the safety net for next time. Only shows when at least one source actually has an active pause to clear.</td></tr></table>
The existing yellow “Hit consecutive-block threshold … auto-paused this source until …” banner now also reflects the override state: when override is ON, the banner turns green and explains that the lock is stale and being ignored. When override is OFF and a source is paused, the banner gets a tip pointing at the two new buttons.

Pinboard v3 silhouette (3D print). Re-tuned the Mickey ear proportions after Drew shared four reference photos of real Disney pin-trading boards. v3 ears were too small and absorbed into the head silhouette (too round). v4 went bigger but kept the ears inside the head radius (still merged). v5 (current) moves the ear CENTERS outside the head radius (EAR_DIST_RATIO 1.10) for prominent stuck-on-top ears with a visible neck pinch. Ear/head diameter ratio is now 0.72 (was 0.65 in v3). Head diameter is 178 mm vs v1's 164 mm; ear diameter is 128 mm. 400 holes, still 1.6 mm each, still 95% bed coverage on the X1C plate (rotated 34°).

v0.15.159

2026-05-25

Operator override for the catalog crawler's block-threshold auto-pause. Drew rotates IPs by hand when a source starts blocking (VPN endpoint switch); the v0.15.122 “10 consecutive blocks → pause the source for 30 min” safety net is over-protective for that workflow. The new toggle lets the crawler keep counting and logging blocks but skips setting last_blocked_until, so an IP rotation takes effect on the next request instead of having to wait out the lockout window.

Two ways to enable.<table style='font-size:0.85rem;border-collapse:collapse;'><tr><th style='text-align:left;padding-right:1em;'>Method</th><th style='text-align:left;'>When to use it</th></tr><tr><td>Checkbox on /settings</td><td>Persists across app restarts; flip via browser; takes effect on next loop iteration</td></tr><tr><td>Env var DPB_CRAWLER_IGNORE_BLOCKS=1</td><td>Escape hatch from Git Bash before uvicorn start; always wins over the checkbox; useful for one-off long crawl runs you don't want to commit as a persistent preference</td></tr></table>
What changes in behavior with override ON.
  1. Main crawler loop's pre-loop skip on last_blocked_until is bypassed — any stale lock from before the IP swap is ignored.
  2. Block-error handler still increments consecutive_blocks and logs each block (for visibility) but doesn't set last_blocked_until when the threshold is hit. Instead it logs a one-liner every 10 consecutive blocks so you see the situation in the console.
  3. Image-backfill thread's independent 30-min sleep loop (v0.15.122) is also bypassed.

Default is OFF. For operators without a way to rotate IPs, the threshold pause is a real safety net that prevents extending Cloudflare's suspicion window. Only flip this on if you've got a VPN, mobile hotspot, or other IP-swap mechanism ready.

v0.15.158

2026-05-25

Pin&Pop series scrape now catches series links with absolute URLs and has an og:description fallback for pages where the structured detail table is wrong. Follow-up to v0.15.157's investigation: Drew confirmed via SQL that catalog_pins.series_name was NULL for the Jack-o'-Lantern Wreath pin (P&P#104938) even though Pin&Pop's page itself shows the “Mickey Mouse Family Halloween Collection” series in the breadcrumb, og:title, og:description, body text, and URL slug.

Two things were happening. First, the series-link regex in fetch_pinandpop was ^/series/ — relative URLs only. On P&P#104938 the visible series links render as absolute https://pinandpop.com/series/... hrefs, so no match. Second, Pin&Pop's own structured Pin Details table is literally inconsistent with the rest of the page for this pin — the “Series” row reads “Not in a series” even though every other signal on the page contradicts that. Pin&Pop data-quality issue, but our scraper needs to tolerate it.

Fix. Two layers:<table style='font-size:0.85rem;border-collapse:collapse;'><tr><th style='text-align:left;padding-right:1em;'>Layer</th><th style='text-align:left;'>Behavior</th></tr><tr><td>Primary</td><td>Series-link regex now matches both /series/foo and https://pinandpop.com/series/foo (also http:// and www. variants)</td></tr><tr><td>Fallback</td><td>If no /series/ link is found anywhere in the body, parse the series name out of the og:description meta tag, which Pin&Pop's OG generator reliably writes as “... Pin from the {SERIES} Disney pin series.” for every pin that belongs to one</td></tr></table>
Cleanup for the affected row. Click “↻ Re-fetch from source” on Drew's catalog page for the Jack-o'-Lantern Wreath pin. The fresh fetch will populate catalog_pins.series_name with “Mickey Mouse Family Halloween Collection”, after which _auto_series_from_catalog (v0.15.117) will either find an existing Series with that name in your DB or create one on the fly, and pre-select it on the /new-pin form for any future scans of this pin. If you suspect other Pin&Pop-sourced rows have the same NULL series_name issue, a crawler refetch sweep will catch them all next pass.

v0.15.157

2026-05-25

Barcode-matched pins now prefill Name, Original Price, and Edition (LE) on /new-pin like a regular AI-scanned pin does. Drew flagged it on a Pin&Pop barcode match for the Mickey Mouse Jack-o'-Lantern Wreath light-up pin (P&P#104938): the catalog page correctly identified the pin and the form's Year and Set size fields came through populated, but Pin Name, Original Price, and Edition (LE) were all blank.

Cause. v0.14.56 added original_price, edition_size, and other structured fields directly into the Pin&Pop fetch output so the catalog could be source-of-truth with AI as fallback. The template was updated to use the new Pin&Pop-first chain for year, set_size, and series — but the bindings for name, original_price, and edition_size were left consulting ai_suggestion alone. On a regular AI scan that's fine because ai_suggestion is populated; on a barcode-only match the AI suggester never runs, so those three fields stayed empty even though the data was sitting in pinandpop_data waiting to be used.

Fix. Extended the same Pin&Pop-first fallback chain to the three remaining fields in scan_new_pin.html. The order matches the v0.14.56 convention — Pin&Pop as authoritative source first, AI as fallback:<pre style='font-size:0.78rem;'>value="{{ pinandpop_data.FIELD if pinandpop_data and pinandpop_data.ok and pinandpop_data.FIELD else (ai_suggestion.AI_FIELD if ai_suggestion and ai_suggestion.AI_FIELD else '') }}"</pre>Affected fields: name (pinandpop_data.name → ai_suggestion.name), original_price (pinandpop_data.original_price → ai_suggestion.original_price_usd), and edition_size (pinandpop_data.edition_size → ai_suggestion.edition_size).

Series auto-assignment is a separate path. The Series dropdown isn't bound directly to pinandpop_data.series; it pre-selects via the auto_assigned_series_id the route handler computes from _auto_series_from_catalog, which reads CatalogPin.series_name from the local catalog mirror. If that field is blank for a given pin (Pin&Pop genuinely has no series for it, or the scrape never extracted one), no series gets auto-picked. On standalone pins like the Jack-o'-Lantern Wreath this is often legitimate — Pin&Pop just doesn't categorize it under a parent series. Diagnostic SQL to check for a specific catalog row:<pre style='font-size:0.78rem;'>SELECT id, pinandpop_id, name, series_name FROM catalog_pins WHERE pinandpop_id = '104938';</pre>If series_name is NULL but Pin&Pop does show a series on the source page, click “↻ Re-fetch from source” on the catalog page to repull. If Pin&Pop genuinely has no series, the “+ New” button next to the Series dropdown creates one inline.

v0.15.156

2026-05-25

PTDB now honors DOM order for the hero image instead of letting the saturation heuristic re-rank. Drew flagged PTDB pin 9933 (DisPindable pin 28025, “Walt Disney Animation Studios Shorts Set - The Ballad of Nessie ONLY”): the catalog page showed the brass back of the pin as the hero, not the enamel front.

Diagnosis from the diag-images log + raw page source. PTDB's pin pages have no og:image, twitter:image, or JSON-LD — the <head> is bare XHTML 1.0 Transitional from circa 2014. So v0.15.139's “use og:image as the structural hero hint” fix was a no-op for PTDB. The parser correctly extracted both thumbs from #picDiv in DOM order: 16323_thumb.jpg (front) first, 16324_thumb.jpg (back) second. PTDB's own JavaScript confirms 16323 is the primary — the page hardcodes data: 'picId=16323' as the initially-loaded big picture. So the parse output had the right image at slot 0.

What broke it. After download, _promote_hero_by_saturation (v0.15.135) ran and re-ranked. Mean HSV saturation:<pre style='font-size:0.78rem;'>Front (Nessie scene — pale teal/orange/white): ~55-70 Back (brass plate with engraved text): ~85-110</pre>Brass back outscored the enamel front, got renamed into slot 0, became the hero. Drew called this exact failure mode in the v0.15.139 comment for pin 1957: “brass has enough chromatic saturation to compete with the enamel front.” Saturation was designed assuming silver/grey mesh backs (sat 5-25) where the separation from enamel fronts is clean. Gold/brass backs land in the same range as muted enamel fronts and the signal collapses.

Fix. _download_all_images gains a trust_first_image=False keyword. When True, the saturation re-rank is skipped and a one-line log explains why. PTDB's call site now passes trust_first_image=True because its DOM order is the canonical primary-first signal (no og:image to compete with). PinPics and Pin&Pop are unchanged — they have their own quirks where the source site's editorial choice isn't always the best hero (the v0.15.120 PinPics-pin-125 case), so saturation re-ranking still earns its keep there.

Cleanup for already-affected rows. The fix doesn't rewrite existing local files; it just changes behavior on the next refetch. To repair pin 9933 and any other brass-back PTDB rows, click “↻ Re-fetch from source” on the affected catalog page — the new download path will write the front into slot 0 and shift the back to {id}_1.jpg. If you suspect this hit many rows, the catalog crawler's re-fetch sweep with DOWNLOAD_ALL_IMAGES=True will pick them all up automatically next pass.

Known trade-off. If a PTDB pin has its back uploaded as the first picThumbAnchor in DOM order (rare but possible — PTDB shows the back as primary on its own site in that case), we'll now follow PTDB and pick the back too. The manual “↑ Set as primary” thumbnail buttons in the catalog UI handle these one-offs. Net: many more pins now correct automatically, a small minority shift from “sometimes accidentally right via saturation” to “needs one click of manual override.”

v0.15.155

2026-05-25

PTDB scraper now recognizes Cloudflare block pages instead of writing them into the catalog as the pin name. Drew flagged PTDB pin 9612: the catalog page showed the pin's name as “Access denied | www.pintradingdb.com used Cloudflare to restrict access | www.pintradingdb.com | Cloudflare” with seven garbage image URLs.

What went wrong. Cloudflare's rule-based 1020 “Access Denied” page is served as HTTP 200 with several KB of HTML — not 403, not the short Turnstile “Just a moment” challenge that PinPics already detects. The check in _fetch_html was if status==200 and len(text)>1000: return success, which passed the block page straight through. _parse_ptdb_html then read the page <title>, ran its “View Pin:” prefix and “pin trading database” suffix cleanup (neither matched the CF title), and emitted the entire block-page title verbatim as the pin name. _upsert_to_db wrote it into catalog_pins.

Fix is layered. <table style='font-size:0.85rem;border-collapse:collapse;'><tr><th style='text-align:left;padding-right:1em;'>Layer</th><th style='text-align:left;'>Behavior</th></tr><tr><td>_fetch_html</td><td>After both the requests path and the headless path, run _is_cloudflare_block(html) and treat detection as a fetch failure</td></tr><tr><td>_parse_ptdb_html</td><td>If the cleaned name still matches a CF block-page title pattern, drop it — defense in depth for any future fetch path that bypasses layer 1</td></tr><tr><td>fetch_ptdb cache check</td><td>Pre-v0.15.155 caches with CF-pattern names get invalidated and refetched (mirrors the v0.15.109 stale-image-URLs pattern)</td></tr></table>
Detection recognizes three CF templates: the new 1020 “Access denied” page (Drew's screenshot), the older “Attention Required! | Cloudflare” page, and the Turnstile / managed-challenge “Just a moment” page that the PinPics path already handles in catalog_sites.py.

Cleanup for the existing pin 9612 row. The cache-invalidation only purges the JSON cache; the catalog_pins DB row keeps its bad name because _upsert_to_db is only called on successful fetches (so a failing refetch can't overwrite it). One-shot cleanup in SQLite shell:<pre style='font-size:0.78rem;'>UPDATE catalog_pins SET name=NULL WHERE name LIKE 'Access denied%Cloudflare%' OR name LIKE 'Attention Required%Cloudflare%';</pre>Then a per-pin Re-fetch from source will try again with the new detection in place.

v0.15.154

2026-05-25

Closed a Playwright context leak that was OOM-crashing Drew's whole Windows machine. Drew's computer hard-rebooted at 01:30 AM on 5/25/2026. Forensics from Windows Event Viewer:<pre style='font-size: 0.78rem;'>Event 41 (Kernel-Power): The system has rebooted without cleanly shutting down first. This error could be caused if the system stopped responding, crashed, or lost power unexpectedly. Event 6008: previous shutdown at 1:30:11 AM was unexpected. No minidump file (== not a BSOD, == OOM-induced reset).</pre>tasklist | grep chrom showed 47 orphan chrome.exe processes holding ~8.5 GB of RAM — Playwright contexts that had been created but never closed. DisPindable's log ended mid-sentence at 01:30:55 while a headless browser was scraping PTDB pin 9531; no exception trace, just stopped. Classic OOM-watchdog-reset signature on Windows.

Where the leak was. In headless_scrape.fetch_rendered(), the structure was:<pre style='font-size: 0.78rem;'>context = browser.new_context(...) # creates Chromium process apply_stealth_sync(context) # could raise context.add_init_script(...) # could raise page = context.new_page() # could raise try: ... do the fetch ... finally: page.close() context.close()</pre>If any of the three lines between new_context() and the try: raised, Python unwound out of the function with the local context variable dropped — but the underlying Chromium child process kept running. Each leak = 200-400 MB stuck. Over a night of crawling, hundreds of failed fetches each leaked a context, RAM filled, Windows' watchdog hard-reset the box.

Fix. Wrapped the post-creation setup in an outer try/finally that closes the context unconditionally. page is initialized to None before the inner work so the finally can guard with if page is not None: page.close() — previous code crashed in the finally itself when page hadn't been assigned yet (which masked the underlying exception).

Verified with three tests: add_init_script raises → context closed; apply_stealth_sync raises and the fallback add_init_script also raises → context closed; happy path still closes both page and context. Negative-control check confirms the pre-v0.15.154 code shape would have leaked under the same conditions.

For Drew, right now: kill the 47 orphan Chromium processes to free ~8.5 GB before doing anything else — taskkill //F //IM chrome-headless-shell.exe is safe (Playwright-only). The full chrome.exe kill also catches normal browser windows so save any open tabs first. After restarting DisPindable on v0.15.154, the leak source is gone — new orphans should never accumulate from this code path again.

Recommendation: get a UPS. Even if v0.15.154 closes this leak vector, future bugs or unrelated browser hangs could push you to the OOM edge again. A $70 APC Back-UPS 600 also rides through power blips, which are the other common cause of Event 41 on always-on desktop machines.

v0.15.153

2026-05-24

Barcode auto-redirect now lands on /new-pin with data pre-filled, instead of bouncing to the external catalog site. Drew's UX feedback on v0.15.151: “barcode scan is working, but I don't want it to take you to the site — I want it to pre-fill the data from PinPics / Pin&Pop and land on /new-pin.”

The fix. When /api/barcode-lookup hits an exact_catalog or exact_serpapi match, it now synthesizes a minimal empty-image Scan row and returns a redirect URL of the form /scan/{N}/new-pin?override_pinpics_id=X&override_pinandpop_id=Y. The existing /scan/.../new-pin route was already designed to accept those override query params (originally added in v0.14.26 to let users click an alternative catalog candidate row), so all the prefill machinery just kicks in: catalog_sites.fetch_pinpics() + fetch_pinandpop() pull the cached scrape data, _cross_reference_catalog_ids fills in any missing source IDs from the local catalog mirror, eBay active+sold queries run with the pin's name, and the auto-tag extractor pre-populates the tags field. Drew lands on a fully-loaded new-pin form ready to save.

Synthetic Scan handling. The new helper _make_barcode_prefill_redirect(s, catalog_ids) creates a Scan(image_filenames_json='[]', notes='(synthetic scan from barcode-lookup auto-redirect)') and returns the redirect URL. The new-pin route's image-dependent code paths (CLIP embedding, visual similarity, fake-corpus reverse lookup) are all guarded by if image_filenames: checks already, so the empty-list synthesizes cleanly without crashing. The synthetic Scan rows do clutter the DB slightly — future cleanup task: prune empty-image scans older than 30 days that weren't converted into a saved Pin.

SerpAPI return shape change. _barcode_serpapi_lookup now returns a 4-tuple (link, label, source_field, source_id) instead of (link, label). The two new fields let the caller pluck the right override_*_id param without re-parsing the URL. All call sites updated.

What stays the same. Owned-pin matches still redirect to /pins/{id} (already internal — you don't need to re-add a pin you own). The manual web-search button row (Google / eBay / Disney Pins Blog) still appears in the modal when there's no auto-redirect. Substring matches still never auto-redirect — barcode “123” could name-substring-match dozens of pins, so we still surface them as a picker list instead.

v0.15.152

2026-05-24

Pin migration crashed on duplicate pins. Drew tried the admin Pin→PinOwnership migration from his /me page and hit:<pre style='font-size: 0.78rem;'>commit failed: IntegrityError: UNIQUE constraint failed: pin_ownerships.user_id, pin_ownerships.catalog_pin_id [parameters: (2, 424, 1, 0, 0, 1, ...)]</pre>The UI showed 43 created but they all rolled back when the 44th hit the constraint — whole transaction undone.

Cause. Drew has two physical copies of some pin designs (Mickey 2021 SDR Hidden Mickey #143186 was the one that fired the error), so his admin Pin table has multiple rows that resolve to the same CatalogPin. The v0.15.143 migration did session.query(PinOwnership).filter_by(...).first() to check for existing rows before inserting — but db.SessionLocal is configured with autoflush=False, so the PinOwnership just added for the first duplicate Pin wasn't visible to that query on the second Pin. Both went down the CREATE path. The UNIQUE constraint caught it at commit.

Fix. The migration now keeps an in-memory {catalog_pin_id: pending_PinOwnership} dict across the loop. First visit creates + records in the dict; subsequent visits to the same catalog_pin_id find the pending row in the dict and route to the UPDATE path. Quantity semantics differ by source: in-run duplicates SUM (two physical copies → quantity=2), pre-existing DB rows use MAX (a re-run preserves any manual increment).

Verified. The new test in test_v15_152.py sets up 3 Pin rows pointing at one CatalogPin, runs the migration twice, asserts 1 PinOwnership row with quantity=3, stable across re-runs. Drew's 63-pin run should now complete in one shot — the 20 duplicate pairs collapse into ~21 PinOwnerships with appropriate quantity values instead of crashing.

What to do. Just re-run the migration from /me. The previous failure rolled back completely, so the DB is untouched — no cleanup needed.

v0.15.151

2026-05-24

Barcode SerpAPI fallback was silently failing on every miss. Drew asked why barcode scans never seemed to fire off a search to match on PinPics / Pin&Pop. SERPAPI_API_KEY was set, local lookup correctly returned no matches, but the fallback that should have hit Google was getting swallowed by the outer try/except. Drew's uvicorn log told the whole story:<pre style='font-size: 0.78rem;'>[barcode-lookup] SerpAPI fallback failed for '400992280600': NameError: name 'requests' is not defined</pre>Cause. main.py doesn't import the requests module at top level — other call sites in the file use a local import requests as _req idiom inside their own functions. The v0.15.127 author of _barcode_serpapi_lookup wrote r = requests.get(...) without that local import, so the bare name never resolved. The outer try/except in api_barcode_lookup caught the NameError and logged it as a generic “fallback failed” line, but the modal just showed an empty matches list with the manual web-search buttons. Looked like the feature was missing; was actually crashing on every call.

Fix. One line: import requests inside _barcode_serpapi_lookup, right next to the existing import re as _re.

How this snuck past the test suite. The v0.15.127 tests mocked the SerpAPI response without ever actually calling _barcode_serpapi_lookup — the function was treated as a leaf and its source-level presence was checked via grep, not by execution. v0.15.151 adds an end-to-end test that patches requests.get and calls the function; that would have caught this in CI.

For Drew. Scan that 400992280600 barcode again after upgrading — you should see [barcode-lookup] SerpAPI hit for '400992280600': pinpics_id=... -> https://pinpics.com/pin/... in the uvicorn log and the modal should auto-redirect to the matching catalog page. If SerpAPI still returns no Disney-domain hits for that barcode, the modal falls back to the manual web-search button row as before.

v0.15.150

2026-05-24

PTDB sibling filter, take 4 — the regex bug. Drew flagged that PTDB#2101 was STILL pulling in 6+ sibling images even with v0.15.149, AND that the hero image was now showing a sibling (Ariel's green door) instead of the actual Snow White pin. He shared the actual page source — which revealed the bug immediately.

Root cause: the regex required a leading slash. Looking at the real PTDB HTML:<pre style='font-size: 0.78rem;'><h2>Pins From Same Release</h2> <a href="pin/2435"><img></a> <-- relative, no slash <a href="pin/28235"><img></a> ...</pre>But v0.15.149's helper looked for /pin/(\d+) — with a leading slash. Since every sibling href in PTDB's “Pins From Same Release” section uses the RELATIVE form (pin/N not /pin/N), the regex matched NONE of them. The helper returned False for every sibling, so every sibling image got included in the catalog. Then the hero promotion logic picked one based on saturation — Ariel's green/teal door has more saturated color than Snow White's muted backdrop, so Ariel won. Drew got a sibling as the hero.

The All Pics section. Notice PTDB's “All Pics” thumbs use <a class="picThumbAnchor" href="#"> — just a fragment, no pin/N at all. Those passed the filter correctly (no match either way), but only as a side-effect of the regex bug, not by design.

Fix in one line. Regex changed from /pin/(\d+) to (?:^|/)pin/(\d+). The (?:^|/) non-capturing group allows the leading slash to be optional, while preventing false positives from accidental substrings (spin/123, begin/456, etc. won't match).

Verified against the real HTML. Plugged Drew's page-source dump straight into a test (test_v15_150.py). Result: 2 images captured (both legitimate “All Pics” shots: Snow White through open windows + the closed shutters), 0 of 6 siblings leaked, and the Snow White image is at position 0 in image_urls so the hero promotion picks the correct shot.

For PTDB#2101 specifically. Click “Re-fetch from source” on the catalog detail page. You should see only 2 images (Snow White + closed shutters), with the Snow White image as the hero. Anything else means the v0.15.150 regex isn't loaded or PTDB changed its HTML again.

v0.15.149

2026-05-24

PTDB sibling filter, take 3 — revert v0.15.147 over-correction. Drew flagged PTDB#2101 (Princess Hinged Windows / Snow White): we got the front shot, but the second “All Pics” image (closed-windows variant) didn't come through.

Root cause. PTDB's pin pages use a two-column layout where the RIGHT column (Pin Stats, Details, Pins From Same Release) appears BEFORE the LEFT column (main image + All Pics) in DOM source order. v0.15.147's section-aware filter used find_all_next() to mark everything after the “Pins From Same Release” heading as off-limits — which on this layout includes all the legitimate “All Pics” thumbnails further down in source order. The filter was eating its own legitimate content.

Fix. Removed v0.15.147's find_all_next() section exclusion entirely. Replaced with a centralized helper is_sibling_pin_anchor(node) that walks up to 8 ancestor levels looking for an <a> with an href of /pin/N where N differs from the current page's pid. The same helper runs against both the CSS background-image walk and the body <img> walk — consolidated logic, easier to reason about. Self-anchored thumbs (<a href='/pin/2101'>) are preserved because the helper specifically requires N ≠ self_pid to filter.

Safety cap. Hard maximum of 8 images per pin. Real pins legitimately have 1–3 photos (front, back, occasionally an angle/closeup shot). When we end up with more than 8 the parser logs [ptdb] {url}: WARNING — N images exceeds safety cap of 8; trimming. Check the sibling-image filter. and truncates. That makes the 30-sibling-leak bug from before v0.15.146 impossible to silently recur.

What gets caught vs what doesn't. On real PTDB pages (as observed in the screenshots Drew shared), sibling thumbnails in the “Pins From Same Release” grid are uniformly anchored to /pin/N — the new helper catches all of them. Hypothetical edge cases that v0.15.147 was defending against (raw <img> after a heading, javascript-only hrefs, CSS-bg divs without anchor wrappers) appear to not actually exist in PTDB's real HTML — we never observed them, so removing those defenses costs nothing. The hard cap catches any future regression silently rather than relying on speculative pattern matching.

Drew's existing rows. Same as v0.15.147: the fix is in the parser, so already-cached PTDB rows still have whatever images they ingested before. Click “Re-fetch from source” on a specific pin or run a bulk re-fetch from /catalog-crawler to update them. For PTDB#2101 specifically, a single re-fetch should now give you both “All Pics” images (front + closed-windows).

v0.15.148

2026-05-24

Per-source metadata panels on the catalog detail page. Drew asked to see what each source (PinPics, Pin&Pop, PTDB) actually pulled in — not just the consolidated fields on the CatalogPin row. Useful because (a) sources sometimes disagree, and seeing each side helps diagnose; (b) source-specific fields like PinPics back descriptions or PTDB market signals are easier to read grouped by source; (c) when a pin is only on one source (e.g. the Hulu Red Carpet PinPics-only example Drew flagged), the page now surfaces that source's data even if the consolidated columns are mostly null.

Data already on disk. Every successful scrape persists to .catalog_cache/{source}_{id}.json with the full per-source result blob. The catalog detail route now reads those files (when they exist + had ok=True) and passes them to the template as a source_metadata dict keyed by source name.

Three panels, distinct accents. PinPics panel (yellow #fec400), Pin&Pop panel (blue #4079c8), PTDB panel (purple #8B6FC9) — matching the existing source-pill colors at the top of the page. Each panel renders the fields its source actually provided: common scalars (name / year / series / origin / edition / rarity / SKU / price / dates / FAC / etc.), plus source-specific extras — PTDB's market signals block (Desperation, For Keeps, For Trade, Wanted, Sellers, Buyers) only renders on the PTDB panel; back-stamp / description / fake-warning text render below the table when present; linked pins collapse into a <details> expander.

No DB schema change. Pure read-side feature — reads the JSON cache files already on disk. If a cache file is missing (older imports, manually deleted, never written), that source's panel just doesn't render. Failed fetches (ok=False) are filtered out so 404 / parse error rows don't render empty panels.

What Drew sees on existing pins. The panels start showing up immediately for any pin whose cache files are still on disk (default 30-day cache). For older pins where the cache has been evicted, click “Re-fetch from source” on the detail page — the next fetch repopulates the cache file and the panel renders on the next page load.

v0.15.147

2026-05-24

PTDB sibling-image filter, take 2 — section-aware. Drew flagged that v0.15.146's anchor-href filter wasn't catching everything. PTDB#2082 (Good Vs Evil Mystery Pack — Simba) still pulled ~30 sibling thumbnails into the catalog page despite the v0.15.146 fix.

Why v0.15.146 missed it. The fix only filtered the body <img> walk, and only based on the parent anchor's href containing /pin/N. Three escape hatches:<ul style='margin: 0.4rem 0;'><li>The CSS background-image walk (v0.15.102) had NO filter at all — siblings rendered as <div style='background-image: url(...)'> leaked through.</li><li>Sibling anchors using href="#" with onclick handlers or javascript: URLs evaded the /pin/N regex.</li><li>Raw <img> tags not wrapped in an anchor at all (PTDB uses these in some layouts) had no anchor parent to check.</li></ul>
v0.15.147 approach: find the section heading, exclude its entire subtree. Before the image walks run, locate the heading text matching any of “Pins From Same Release”, “Pins From This Release”, “More From This Release”, “Other Pins In This Release”, “Pins In This Set”, “Pins From Same Set”, or “More Pins In This”. Mark every following node (and the heading's own descendants) as off-limits via an id()-based exclusion set. Both the CSS-bg walk and the body <img> walk skip any tag whose ancestor chain hits the exclusion set.

Belt and suspenders. The v0.15.146 anchor-href filter is preserved as a backup for the rare case where a layout doesn't use any of the known section heading variants. So the parser has two independent defenses against sibling-image leak.

Drew's existing catalog rows. The fix is on the PARSER, so existing rows that were ingested with the bad data are still bad in your DB. Two options to clean them up: (1) Click ↻ Re-fetch from source on each affected pin individually (parser fix kicks in, downloads only the correct image). (2) Use the bulk re-fetch tools on /catalog-crawler to re-ingest a range of PTDB IDs. For new fetches, v0.15.147 means correct image counts from the start.

Log output. When the section heading is found, the parser logs [ptdb] {url}: found sibling-section heading ('h2': 'pins from same release') so you can confirm the filter activated. If you see PTDB pages without this log line AND with too many images, the heading text might be a new variant — share the PTDB URL and I'll add the variant to the match list.

v0.15.146

2026-05-24

Parser metadata expansion + PTDB sibling filter + multi-user UX polish. Drew flagged five separate things; addressing them all in one release.

(1) PTDB sibling-pin filter. Drew flagged that PTDB ingest was pulling too many images on pages like <a href='https://www.pintradingdb.com/pin/2062'>#2062 (Pirate's of the Caribbean) — the catalog detail page showed 10+ thumbnails because every ‘Pins From Same Release’ sibling at the bottom of the PTDB page was being downloaded as part of the current pin. Fix: _parse_ptdb_html now walks each <img>'s parent anchor; if the anchor targets /pin/N where N differs from the current page's pid, the image is filtered out (it belongs to a sibling pin, not this one). Images not inside an anchor — or inside an anchor pointing to the SAME pin (lightbox triggers, gallery thumbs) — are kept.

(2) Pin&Pop metadata expansion. Audit of Drew's example page <a href='https://pinandpop.com/pins/120251/'>#120251 (Mater) found several visible fields that weren't being captured. Added extractors and DB columns for: set_special (“Set/Series/Special” row — often blank, sometimes “Hot Topic Exclusive” / “Park Exclusive”); release_date already existed but the regex now also catches full month-name dates like “January 15, 2024”; fac (Pin&Pop FAC classification code, skips the “Unknown” placeholder); similar_design (text reference to visually-similar pins); confirmed_fakes_count (int when shown, None when “None reported”); my_pin_central_id (cross-link to mypincentral.com); full description paragraph parsed from <h2>Description</h2> onward. All surface on the catalog detail page's “Catalog details” panel.

(3) PinPics + PTDB coverage. Both parsers already captured most of the corresponding fields where the source exposes them. PinPics is genuinely sparse (just name / year / images / back / linked / pack size) — PinPics doesn't surface FAC / SKU / Confirmed Fakes / My Pin Central on its pages, so there's nothing to capture there. PTDB already had description, rarity, sku, origin, edition_size, series, release_date, retire_date, added_by from v0.15.112; the v0.15.146 sibling-image filter is the main PTDB change this release.

(4) Multi-user scope-down REVERSED for now. Drew's standing instruction: regular users should see EVERYTHING for the time being (he'll scope down later). v0.15.145's strict per-role nav gating collapsed: now both admin and regular logged-in users see the same nav links — Scan, Pins, Series, Dashboard, Collections, Top, plus the More dropdown. Routes themselves still gate where appropriate (catalog crawler stays admin-only, /admin/users stays admin-only, etc.) so a regular user clicking those gets a redirect to /me. Only the VISIBILITY opens up. Anonymous users still see the minimal Browse + Sign in nav.

(5) Sign-in / Sign-out button visibility. Drew showed two screenshots: Sign-out rendered as dim gray on the dark dropdown background (basically invisible); Sign-in wrapped onto two lines because the icon-button class forced a narrow flex container. Fixes: Sign-out button now uses the brick-red brand accent with a subtle red-tinted background and white-ish foreground; Sign-in is now a solid brand-color CTA with white-space: nowrap to prevent wrapping. Both pop visually now.

(6) Where do you set the admin env vars? Drew asked: do I edit a config file? Answer: it's not a config file — it's shell environment variables BEFORE you start uvicorn. On Windows Git Bash:<pre style='font-size: 0.82rem;'>export DISPINDABLE_ADMIN_EMAIL="[email protected]" export DISPINDABLE_ADMIN_PASSWORD="strong-password-here" uvicorn main:app --reload</pre>On first boot only, the startup hook creates the User row with is_admin=True. Subsequent boots are idempotent — won't overwrite your password. If you want this to persist across terminal sessions, add both export lines to your .bashrc or set up a .env file + source it before running uvicorn. Use DISPINDABLE_ADMIN_FORCE=1 alongside the password env var to force-reset a forgotten admin password on next boot.

Migration note. Five new columns on catalog_pins (set_special, fac, similar_design, confirmed_fakes_count, my_pin_central_id). _LATE_ADDITIONS handles the schema migration on next init_db() — no manual SQL needed. Existing rows get NULL for the new columns; re-fetching a pin or running the crawler will populate them going forward.

v0.15.145

2026-05-23

Auth-aware navigation + admin user management. Drew asked two related questions: (1) add a login indicator to the main nav with sign-out access, and (2) how do people become admin? This release answers both.

Three nav states. The top navigation now branches on the current user's auth status:<ul style='margin: 0.4rem 0;'><li>Anonymous: shows only Browse catalog + a “Sign in” CTA. No more leaking the full admin menu to logged-out visitors.</li><li>Regular user: shows My collection, Wishlist, Browse catalog. A user-badge dropdown in the top-right (display name + user icon) opens a menu with My collection, Import CSV, and Sign out.</li><li>Admin: shows the full nav (Scan / Pins / Series / Dashboard / Collections / Top / More). The user-badge dropdown adds a red ADMIN pill and a “Manage users” link. Everything Drew is used to, plus the indicator + sign-out he asked for.</li></ul>
Mobile tab-bar mirrors the same three-state pattern: regular users get Mine / Wishlist / Browse / Import tabs; admins get the existing Scan / Pins / Dashboard / Series / More tabs; anonymous gets Browse / Sign in / Register.

How users become admin. Three answers, use whichever fits:<ul style='margin: 0.4rem 0;'><li>Env var bootstrap (already in v0.15.142): set DISPINDABLE_ADMIN_EMAIL and DISPINDABLE_ADMIN_PASSWORD on first server start. Creates Drew's admin row automatically. Idempotent on subsequent boots. Use DISPINDABLE_ADMIN_FORCE=1 to reset a forgotten password.</li><li>Existing admin promotes via UI (NEW): go to /admin/users (or click “Manage users” in the user-badge dropdown). Lists every account with role badge + pin counts + last login. Per-user actions menu: Promote to admin / Demote to user, Reset password (admin sets a new one for lost-password recovery), Delete user (hard-delete with cascade to PinOwnership).</li><li>Direct DB edit (always works): UPDATE users SET is_admin = 1 WHERE email = '...';</li></ul>

Safety guards. The toggle-admin route refuses to demote yourself (would risk locking everyone out of admin tooling if you're the last admin). Same for delete-self. Both return a flash message explaining the limit; another admin can do it for you if needed. New account registrations always create regular users; admin promotion is explicit.

UX details. The user-badge dropdown shows the full email at the top so users with ambiguous display names know which account they're in. ADMIN pill uses the brick-red brand color to stand out. Sign-out is a real POST /logout form (not a GET link) so prefetch / link-scanner tools can't accidentally sign people out. Sign-in CTA on anonymous nav is icon + label so it's discoverable.

v0.15.144

2026-05-23

Missing-dep fix: itsdangerous + graceful degradation. Drew tried to start v0.15.143 on his Windows venv and got ModuleNotFoundError: No module named 'itsdangerous' at boot. Root cause: SessionMiddleware uses itsdangerous to sign session cookies, and Starlette imports it lazily — pip does NOT install it as part of the fastapi/starlette transitive dep tree. Drew's pre-v0.15.142 venv didn't have it; my test environment did, so the bug never surfaced for me.

Two-part fix. (1) Added itsdangerous>=2.0 to requirements.txt so future fresh installs pull it down. (2) Wrapped the SessionMiddleware import in a try/except. When itsdangerous is missing, the app boots normally and prints a clear actionable error (“pip install itsdangerous”) rather than crashing at import time. The non-auth routes (catalog crawler, scan, admin tooling) keep working in that degraded mode; only multi-user routes are affected.

Drew's fix right now. One command:
.venv\Scripts\pip install itsdangerous
Then restart uvicorn. Or extract v0.15.144 zip, run pip install -r requirements.txt to pick up the new dep, restart.

Bonus: Windows-friendly chmod. The os.chmod(.session_secret, 0o600) call from v0.15.142 was a no-op on Windows but could theoretically error on weird perm setups. Wrapped in try/except so any future quirks don't cascade. Also tolerant of missing SessionMiddleware in auth.current_user — request.session access raises AssertionError when the middleware isn't installed, and we now catch + return None so non-auth routes keep rendering.

v0.15.143

2026-05-23

One-time migration of admin pins to PinOwnership. Drew asked how to associate his existing Pin rows (thousands) with the v0.15.142 multi-user system. This release adds the migration tool: POST /admin/migrate-pins-to-ownership with a dry-run preview and a one-click commit.

What it does. Walks every row in the legacy pins table where any collection flag is set (is_owned, on_wishlist, or available_for_trade). For each, finds the matching CatalogPin by source ID (pinpics_id → pinandpop_id → ptdb_id priority — matches the stated catalog source preference). Creates a PinOwnership row attributed to Drew.

What carries across. Ownership flags (is_owned, on_wishlist, available_for_trade), quantity, and the acquisition metadata (acquired_at, acquired_from, acquired_via, acquired_price, condition). Combined states are preserved — a Pin that's is_owned=True AND on_wishlist=True (you own one and want another) migrates with BOTH flags set.

What does NOT. The Pin table is never modified. Read-only on the source side — admin workflow keeps working unchanged. Photos / scans / trade history / pin notes stay tied to the legacy Pin rows. Future phases can carry more across if needed; for now the PinOwnership row is the lightweight per-user “I own this catalog pin” record.

Dry-run first. Always run the dry-run preview before committing. Reports exact counts: scanned (Pin rows examined), would_create (new PinOwnership rows), would_update (existing rows with deltas), no_change (already-correct rows), no_catalog_match (Pin has a source ID but no corresponding CatalogPin yet), no_source_ids (Pin was never identified to any source — e.g. custom scans), errors. Shows the first 20 unmatched-by-catalog and no-source-id rows so Drew can decide what to do with them (let the crawler catch up, then re-run; or use scan/Re-fetch to identify them, then re-run).

Idempotent. Re-running the migration doesn't duplicate rows. UniqueConstraint on (user_id, catalog_pin_id) ensures a single PinOwnership per pin per user. Subsequent runs report no_change for previously-migrated pins. Newly-catalogued pins (e.g., a pin whose source ID had no CatalogPin row at first-run time but does now) get picked up on the next run.

User-edit safety. If a user manually edited a PinOwnership row between migrations (changed condition, added notes), the re-run never overwrites. Acquisition fields are only filled in when currently empty on the PinOwnership side. Flags are OR'd (a True flag stays True). User edits win.

Surfaced from the dashboard. Admin users see a “Migrate admin pins” button on /me (the user dashboard). Regular users don't see the button or the route — only admins have legacy Pin data to migrate.

v0.15.142

2026-05-23

Phase 1 of multi-user support — auth + per-user collection tracking. Drew asked for a system where individuals can log in and track their own collection separately, with a way to bulk-import owned pins (so far only Drew's account has contributed to the ‘Owned’ data). The local catalog DB and all admin tooling remain unchanged.

Scope of this release. Phase 1 is the auth foundation + per-user pin ownership only. NOT in this release (intentionally, to keep the change reviewable): per-user folders/series, per-user scan history, multi-user trading, friend connections. The existing Pin / Folder / Collection tables stay single-user (Drew's admin collection); regular users get the new PinOwnership table mapping to CatalogPin. A later phase can unify if needed; meanwhile this gets users on the system without rewriting hundreds of admin routes.

New models. User (email, scrypt password hash + salt, display_name, is_admin, timestamps). PinOwnership (user_id FK + catalog_pin_id FK, is_owned / on_wishlist / available_for_trade / quantity, plus acquisition fields: acquired_at / from / via / price / condition / notes). Unique constraint on (user_id, catalog_pin_id) so toggle operations update existing rows rather than creating duplicates.

Password hashing. hashlib.scrypt from stdlib — no new pip dep needed. Parameters (n=2^14, r=8, p=1, dklen=64) take ~50ms per verify on a modest server. Salt stored separately per user so we can rotate parameters later without a wholesale password reset.

Sessions. Starlette's SessionMiddleware with itsdangerous-signed cookies; secret persisted to .session_secret (0600, gitignored already since we added it to the build exclusion list). 30-day cookie lifetime, HttpOnly, SameSite=lax. https_only=False for now; flip to True once Drew's on a non-self-signed cert.

Routes (all new). GET/POST /login, GET/POST /register, POST /logout, GET /me (dashboard), GET /me/collection?view=owned|wishlist|trade, POST /me/catalog-pin/{cid}/own, POST /me/catalog-pin/{cid}/wishlist, GET/POST /me/import (CSV bulk import).

CSV import is the headline feature. Drew flagged that only one person has populated the ‘Owned’ data. New users need a fast way to declare what they own without clicking through hundreds of catalog pages. The import accepts a CSV where each row identifies a catalog pin via any of pinpics_id, pinandpop_id, ptdb_id, or a fuzzy name match (only accepted when EXACTLY one catalog pin matches to prevent wrong assignments). Optional acquisition metadata columns: acquired_from, acquired_via, acquired_price, quantity, notes. Reports created/updated/not-found/error counts + first 10 unmatched rows so users can see what didn't import.

Catalog detail page changes. Every logged-in user (admin or regular) now sees a ‘YOUR COLLECTION’ box on each catalog pin detail page with two toggles: ‘Mark as owned’ / ‘Owned’ and ‘Add to wishlist’ / ‘Wishlisted’. The admin-only buttons (Re-fetch, Diag images, Set as primary, Verify with AI) remain visible to admins as before.

Admin bootstrap. Set DISPINDABLE_ADMIN_EMAIL and DISPINDABLE_ADMIN_PASSWORD environment variables on first server boot. The startup code creates Drew's admin user automatically if no row exists for that email. Idempotent on re-runs: existing user kept, password not overwritten. Use DISPINDABLE_ADMIN_FORCE=1 to force-update the password on next restart (lost-password recovery).

What Drew should do. (1) Set env vars and restart to create the admin login. (2) Visit /login, sign in. (3) Existing PIN data is untouched — Drew's personal Pin/Folder/Collection rows stay in the admin space. The new collection system is separate. (4) Share the registration URL with other collectors who want to track their own pins.

v0.15.141

2026-05-23

Cap-not-sticking bug fixed. Drew submitted the Start form with the max/run field blank (placeholder ∞) expecting an unlimited run. Source kept capping at 10 anyway. Root cause: a chain of three issues that together made blank-means-unlimited not work.

Bug #1: form handler treated blank as ‘no change’. The /catalog-crawler/start handler only added a source to the caps dict when its form field was non-None: if pinpics_max is not None: caps["pinpics"] = int(pinpics_max). When the field was blank, FastAPI delivered None, the dict skipped that source, and a stale max_requests_per_run=10 from a previous run silently persisted in state.

Bug #2: start() respected the empty caps dict. Even reaching catalog_crawler.start(), the loop only updates max_requests_per_run when caps and s in caps. So an empty caps dict for a source ⇒ existing state untouched. Combined with #1, blank field = stale value preserved.

Bug #3: reset_cap didn't clear the cap value. Drew clicked ‘Reset cap & resume’ expecting the source to keep going. The function reset requests_this_run = 0 and run_cap_reached = False, but left max_requests_per_run = 10 as-is. Source advanced 10 requests then hit the cap again. Reset that wasn't actually a reset.

Three-part fix. Form handler now ALWAYS populates caps for all three sources, with None for blank/0 (explicit unlimited). start() handles None correctly via the existing ternary (int(cap_val) if cap_val and int(cap_val) > 0 else None). reset_cap now clears max_requests_per_run too — matches the button verb literally: ‘Reset cap’ resets everything, not just the counter.

User-visible behavior now. Blank max/run field on Start = unlimited, period. ‘Reset cap & resume’ = unlimited from this point forward, until you submit a Start form with an explicit cap. Cap value of 0 is also treated as unlimited (same as blank) for users who type 0 instead of leaving empty.

v0.15.140

2026-05-23

Empty-parse 404 detection — now symmetric across all three sources. Drew asked whether the gap-fill + recovery tooling covers all sources. Audit found one remaining asymmetry: the v0.15.137 empty-parse → 404 defense was only added to PTDB (where the problem was first observed). PinPics and Pin&Pop relied on their existing 404 paths — HTTP-status for Pin&Pop, text-marker detection for PinPics. Those paths catch the documented 404 pages, but if either site ever serves a 200 with no real pin data (template page, A/B test, region block, future layout change), the parser would return ok=True with empty fields and the crawler would count a phantom success.

Defense pattern, same as PTDB. After the parser runs, if BOTH name and image_urls are empty, fetch_pinpics and fetch_pinandpop now return ok=False with a 404-flavored error. The crawler categorizes correctly: not_found counter increments, consecutive_blocks tracks streaks, auto-pause triggers within the threshold. The v0.15.137 upsert guard is still in place as a third layer (refuses to CREATE rows with no useful data) so even if the fetch-layer check is bypassed (cache reads, direct API calls), no empty rows get created.

Coverage matrix — all source-spanning tools now uniform across PinPics, Pin&Pop, PTDB:<ul style='margin: 0.4rem 0;'><li>State recovery (v0.15.134, first-gap fix in v0.15.137)</li><li>Gap detection + mismatch banner (v0.15.137)</li><li>Gap-fill mode (v0.15.137)</li><li>Empty-row cleanup (v0.15.137)</li><li>Bulk image backfill (v0.15.138)</li><li>Saturation hero promotion (v0.15.135)</li><li>Bulk hero re-promote button (v0.15.135)</li><li>Vision-LLM hero verifier — bulk + per-pin (v0.15.136 + v0.15.139)</li><li>Structural hero hints — og:image / JSON-LD (PinPics v0.15.136, PTDB + Pin&Pop v0.15.139)</li><li>Empty-parse 404 defense (PTDB v0.15.137, PinPics + Pin&Pop v0.15.140)</li></ul>
No source-specific holes left. If Drew runs gap-fill on any one source, the same cleanup + backfill + hero verification works on the others uniformly.

v0.15.139

2026-05-23

PTDB & Pin&Pop parsers: structural hero hints. Drew flagged a PTDB row (#1957 — “A Bug's Life — Flik and Atta on a Leaf”) where the brass-toned back was selected as hero. The v0.15.135 saturation heuristic didn't flip it because brass has enough chromatic saturation to compete with the colorful enamel front. The v0.15.136 fix added structural hints (og:image, JSON-LD Product.image) to the PinPics parser but PTDB and Pin&Pop were still walking DOM order. This release fixes the gap.

PTDB parser changes. Now checks <meta property="og:image">, <meta name="twitter:image">, and JSON-LD Product.image BEFORE the existing CSS-bg / DOM walk. Each candidate still runs through looks_like_pin_image() — so the v0.15.53 concern (PTDB's old og:image pointing at the Sign-In UI icon) stays handled. UI graphics get filtered out and the parser falls through to the existing logic. Malformed JSON-LD doesn't break the parser.

Pin&Pop parser changes. Pre-v0.15.139 og:image was only a fallback for the case where the CDN-URL regex turned up nothing. Now: when og:image matches the canonical CDN pattern, its hash is extracted and PROMOTED to slot 0 of the ordered hashes list. The page-author-declared canonical image becomes image_urls[0] regardless of where it appeared in the HTML body. Other hashes in the body are preserved in insertion order behind it.

Per-pin AI hero verifier button. Drew didn't want to run the bulk batch (v0.15.136) just to fix one mis-promoted hero. The catalog pin detail page now has a small “🤖 Verify hero with AI” button (visible only when there are 2+ images) alongside the existing “Re-fetch” and “Diag images” buttons. One click runs Gemini Flash on this pin's images and swaps files if the model identifies a different image as the front. ~$0.0001 per call. Returns to the detail page with a flash describing what happened (which image got promoted + model confidence, or “current hero is correct”, or error details).

Drew's immediate fix for PTDB#1957. Either click “↑ Set as primary” on the front-image thumbnail (instant manual override, same as v0.15.135), OR click the new “🤖 Verify hero with AI” button (one-shot automatic fix). Forward fix for newly-ingested PTDB rows: this release's og:image / JSON-LD priority means PTDB's page-declared canonical image becomes the hero automatically — assuming PTDB's og:image actually points at the front, which most pin sites do.

v0.15.138

2026-05-23

Bulk image backfill for catalog rows missing their hero. Drew asked how to fix the rows from previous incidents where the catalog row has a name + source ID but no hero_local_path — common after the v0.15.134 bad-recovery incident, but also from older ingests (pre-v0.15.32 when local mirroring didn't exist) and from individual failed downloads. Existing /catalog/backfill-image-paths only patches the DB column when the file is already on disk; it doesn't re-download anything. This release fills that gap.

New module: image_backfiller.py. Background worker that walks every CatalogPin row matching name IS NOT NULL AND hero_local_path IS NULL, reads the cached fetch result from .catalog_cache/{source}_{id}.json, and re-downloads the image_urls it finds. Does NOT re-hit source sites — only the image CDNs — so it's much faster than per-pin Re-fetch and doesn't pressure source-site rate limits.

UI: /catalog/backfill-missing-images/status. Live progress page with auto-refresh, stop button (cooperative), and pre-flight pending-count badge so you know how many rows are about to be processed. Stats counters: fixed (row now has hero_local_path), no cache (no cached fetch result — needs per-pin Re-fetch instead), cache empty (cache present but image_urls is empty — original parse found no images), download failed (URLs present but couldn't fetch). Surfaced from the “Catalog gap-fill & cleanup” section on /catalog-crawler.

Source filter + limit knob. Form lets you scope the backfill to one source (e.g. PTDB only, since that's where most damage from the bad-recovery incident lives) and cap the row count for dry-run testing. Defaults: all sources, no limit, 0.5s inter-request delay.

Doesn't touch empty-stub rows. The worker explicitly filters name IS NOT NULL so the polluted “Unnamed catalog pin” rows are skipped — those should be deleted via cleanup-empty-rows first. Different problem, different tool. Running cleanup BEFORE backfill also makes the pending-count badge more accurate (no skewing from rows that should be deleted anyway).

Three-button recovery workflow. On /catalog-crawler, the cleanup section now lays out the proper order: preview empty-row cleanup, delete empty rows, then backfill missing images. Three discrete steps that together repair a catalog after a bad-recovery or failed-download incident.

v0.15.137

2026-05-23

Recovery fix: first gap, not max+1. Drew flagged the v0.15.134 recovery jumped cursors way past actual coverage. His PinPics cursor landed at 187,447 when only ~1,750 rows were cataloged — most clustered at low IDs with a handful of outliers in the 100k+ range. The recovery used max(id)+1 which lands at the outlier's successor, completely skipping 99% of the unscanned ID space. Fixed: recover_state_from_catalog now finds the FIRST GAP (lowest integer not present in catalog_pins) and sets cursor there. For Drew's case that's 1,751, not 187,447.

Upsert guard against empty rows. Drew also saw the crawler creating “Unnamed catalog pin” rows with no name, no images, no embedding — the natural consequence of the cursor being way past PTDB's actual content. PTDB's “not found” page is now >2000 chars (doesn't trip the size heuristic) and doesn't contain literal “pin not found”, so fetch_ptdb treats it as success and the empty parse result gets upserted as a real row. Fix: _upsert_catalog_pin refuses to CREATE new rows when name is empty AND image_urls is empty. Existing rows are still updatable (a partial scrape may be filled in by a re-fetch). Returns (None, False) to signal the no-op.

PTDB empty-parse now returns 404. Second-layer defense: fetch_ptdb itself now detects when a parse yields no name AND no images and returns ok=False with a 404-flavored error. This way the crawler counts the hit as a 404, not a phantom success — consecutive_blocks increments, and the source auto-pauses after the threshold instead of hammering empty ID space indefinitely. The upsert guard is still in place as a third-layer defense for fetches that bypass this path (e.g., direct cache reads).

Empty-row cleanup endpoint. POST /catalog/cleanup-empty-rows deletes rows where name, hero_local_path, and all_image_paths_json are ALL null — the polluted “Unnamed catalog pin” rows. Defaults to dry_run=True with a redirect showing the count + a sample of what would be deleted. Pass dry_run=false to commit. Surfaced as two buttons (preview, then delete) on /catalog-crawler.

Gap-fill mode — new module. gap_filler.py walks 1..max_id_seen for a chosen source, builds the set of IDs already in catalog_pins, and fetches every missing ID. Same rate limits as the main crawler (read from per-source state) so it's polite. Safe to run alongside a forward crawl on a different source. Use case: after a bad-recovery incident (like this one), gap-fill repairs the coverage holes that the forward crawl skipped past.

Gap-fill UI. GET /catalog/gap-fill/status live progress page (auto-refreshes 3s while running). Stats: checked, fetched, new rows created, skipped (already cataloged), 404s, errored. Per-source scan_end selector defaults to max_id_seen from crawler state. Cooperative stop button respects in-flight fetches.

Recommended recovery sequence for Drew right now<ol style='margin-top: 0.4rem;'><li>Visit /catalog-crawler. The state-stale banner will show again with corrected first-gap proposed cursors.</li><li>Click “Recover state from catalog DB” — preview, then commit. Cursors back up to 1,751 (PinPics) / appropriate value for Pin&Pop and PTDB.</li><li>Click “Preview empty-row cleanup” to see how many polluted rows exist, then “Delete empty rows” to remove them.</li><li>Optionally kick off “Gap-fill catalog” for each source — fills any holes in the existing range while the forward crawl picks up from the first gap.</li></ol>

v0.15.136

2026-05-23

Vision-LLM hero verifier — surgical fix for existing catalog rows. Drew flagged that v0.15.135's saturation heuristic was good but not perfect; his catalog still had a few thousand rows with the wrong hero. This release adds a second-layer fix: a background worker that sends each multi-image catalog row to Gemini Flash with a single-shot prompt — “which of these is the FRONT of the pin?” — and swaps the underlying files when the answer disagrees with the current hero.

Cost. ~$0.0001 per row (Gemini Flash, 256×256 thumbnails, 3-4 images per pin, ~1000 input tokens). For a 60k-row catalog that's ~$6 total. The new saturation_skip_threshold knob lets us skip the AI call entirely when v0.15.135's heuristic was already confident (default: skip when hero saturation beats every other image by 80+ on the 0–255 scale). Combined cost on a typical catalog: ~$1–3 because the saturation-confident majority is skipped.

UI: /catalog/verify-heroes-with-ai. Status page with live progress (auto-refreshes every 3s while running), stop button (cooperative; in-flight Gemini call finishes first), running counters for processed / swapped / skipped (saturation-confident) / skipped (single image) / errored. Last-error string is surfaced so a bad image or transient API issue is visible without digging into logs. Surfaced from the “Hero image cleanup” section on /catalog-crawler.

Settings exposed in the UI (so Drew can test on a sample first): min_confidence (default 0.6; ignore model answers below this), saturation_skip_threshold (default 80; raise to force AI on more rows, lower to trust saturation more), and limit (default unlimited; set to 100 to dry-run on a sample before committing to the full catalog).

Background worker design. Module-global state with a threading lock, single-job semantics (mirrors the catalog crawler pattern). Worker runs as a daemon thread; closing the browser tab does NOT stop it. State is in-memory; if uvicorn restarts mid-batch, the in-flight rows are lost but the DB is never modified directly — re-running is idempotent. Errors on individual rows don't poison the whole batch.

PinPics parser hardened. Drew asked whether we're trusting source-declared hero markers over document order. The answer was “Pin&Pop yes, PinPics no” — PinPics's parser just walked <img> tags in DOM order. This release fixes that. The parser now checks (in priority order) <meta property="og:image">, <meta name="twitter:image">, and JSON-LD Product.image structured data BEFORE the document-order walk. The page-author-declared canonical image goes to slot 0; DOM-order images fill subsequent slots. Graceful fallback when none of the hints are present (DOM order applies). Malformed JSON-LD doesn't break the parser.

Doesn't touch any DB rows. Like all previous hero-promotion code paths, the swap is exclusively a file rename in catalog_uploads/. all_image_paths_json ordering is preserved (slot names stay; file CONTENTS move between slots).

v0.15.135

2026-05-23

Catalog hero image — saturation-based selection. Drew flagged a real-world example where the catalog detail page for “Applause — Ariel — The Little Mermaid — 1989 Release” showed the pin's mesh BACK as the hero, with the actual Ariel character art relegated to a small thumbnail. The root cause: the v0.15.120 hero-promotion heuristic picked the largest file on disk, assuming bigger = better. But pin backs are fine mesh texture that JPEG can't compress efficiently, so back-image files often end up larger than the simpler color regions of the front. Bigger file ≠ better image.

New heuristic: mean HSV saturation. Pin backs are silver/grey monochrome (avg saturation 5–25 out of 255). Pin fronts have enamel character art with saturated colors (avg saturation 60–180+). Clean 20× separation that doesn't care about JPEG compression artifacts. _saturation_score() downscales to 128px, converts to HSV, averages the saturation channel. Drop-in replacement for the st_size comparison.

Manual override on every catalog pin page. Each non-hero thumbnail now has a small “↑ Set as primary” button. Click it and the underlying disk files are renamed so that thumbnail becomes the hero. Three-step rename to avoid name collisions; all_image_paths_json ordering is preserved (slot names stay the same, file CONTENTS move between slots). Heuristics will never be 100% reliable; manual override is the safety valve.

Bulk re-promotion of existing catalog. New POST /catalog/repromote-heroes endpoint re-runs the saturation heuristic across every multi-image row in catalog_pins. Idempotent — rows where the current hero is already the most-saturated image are skipped (no swap, no error). Surfaced as a “Hero image cleanup” section on /catalog-crawler with a confirm prompt. Doesn't touch any DB rows, only renames files on disk. Drew: run this once after ingesting v0.15.135 to fix the back-as-hero rows in your existing ~60k catalog.

Future work (NOT in this release). Saturation works but isn't perfect — e.g. a shiny metallic pin back with chromatic reflection could score artificially high. A vision-LLM pass during ingest (“which of these N images is the front?” — one Gemini Flash call at ~$0.01/pin) would be more reliable. Worth considering once we see how often the saturation heuristic still gets it wrong in practice. Manual override stays useful regardless.

v0.15.134

2026-05-23

Crawler state recovery from catalog DB. After Drew's server crashed, the dashboard showed PinPics/Pin&Pop/PTDB cursors at 5/7/7 even though the catalog_pins table still had tens of thousands of rows. The state file (.catalog_crawler_state.json) had been wiped or corrupted by the crash; the DB itself was intact. This release adds the recovery tooling so this doesn't mean “start over” ever again.

New: catalog_crawler.detect_state_db_mismatch(). Cheap startup / page-load sanity check that compares cursor positions against MAX(pinpics_id), MAX(pinandpop_id), MAX(ptdb_id) from catalog_pins. Returns per-source detail when a cursor is behind the DB.

New: recover_state_from_catalog() + apply_recovered_state(). Rebuilds cursors as (max_id_in_db + 1) per source, so the next crawl resumes one past the highest known ID. Conservative direction: cursors only move FORWARD — if the live cursor is already past the DB max (e.g. a 404 region was just crawled), the recovery preserves it. Successful row counts are also rebuilt from the DB (more honest than stale state counters).

UI — mismatch banner on /catalog-crawler. Page-load check surfaces a red warning banner when state appears stale, with a table showing per-source cursor-vs-DB delta and a “Recover state from catalog DB” button.

UI — recovery preview at /catalog-crawler/recover-state. Two-step recovery: GET shows proposed cursor changes side-by-side with current values + DB row counts. POST commits the change. Always preview before commit — Drew can verify the numbers look right before any state is written.

Crash safety — rolling backup on save_state(). Every state write now copies the live file to .catalog_crawler_state.json.bak before overwriting. Gives one round-trip of history to fall back on for transient incidents (atomic-rename was already in place via tmp+os.replace, but doesn't help if the file gets corrupted by external means — crash during open-for-read, antivirus quarantine, manual edit, etc.).

Recovery is non-destructive. The DB is never touched by recovery — only the state file. Re-running recovery is idempotent (after the first run, any_changes returns False and the page shows “Nothing to recover”).

v0.15.133

2026-05-23

Pin board scanning — Phase 1 (detection only). New /scan/board page that accepts a photo of a Disney pin trading board (cork/fabric display with multiple pins arranged in a grid) and returns bounding boxes for every pin detected. Reachable from /scan via a new “Scan a board” button next to Quick Add.

Phase 1 scope. Detection ONLY — we're validating that the vision model finds pins in the right places before investing in per-pin identification. Each submission is single-shot: photo in, bounding-box overlay out. Nothing is persisted to the database. Once Drew confirms detection quality on real board photos, Phase 2 wires the existing identify chain (Gemini Flash + SerpAPI Lens, stage 1) onto each cropped pin and adds a bulk-result UI.

Detection mechanics. The user-uploaded image is downscaled to MAX_DETECTION_DIM=2048 max dim (detection accuracy doesn't improve past ~2K and full 200 MP images blow past Gemini's input limits). The downscaled JPEG goes to Gemini Flash with a high-recall prompt per Drew's directive — bias toward flagging partial / uncertain candidates over missing real pins. Bounding boxes come back as normalized 0–1 coordinates, so Phase 2 can scale them back to full resolution for cropping. Returned boxes are validated: malformed types, wrong-length arrays, out-of-bounds coords, and pin-sized <0.5%-of-image boxes are filtered out defensively.

Render-overlay step. Detection results are drawn onto a display copy of the photo using PIL (downscaled to 1600 px max dim for fast page load). Box color codes the confidence: green ≥0.75, yellow 0.5–0.75, orange <0.5. Each box is numbered for correlation with the raw JSON, shown in a collapsible debug section. Overlay bytes live in a small in-memory LRU (8-entry cap) keyed by a random token returned in the page URL — no disk persistence.

Resolution tolerance. Works with any phone camera. UI nudges the user toward their highest resolution mode (the camera-tip banner highlights 50–200 MP) but doesn't require it. Images below MIN_INPUT_DIM=800 px on the smaller axis trigger a non-fatal warning in the result (“detection still runs but accuracy will be poor”); detection proceeds anyway. Default 12 MP phone shots are plenty for proving detection out — the per-pin resolution matters more for Phase 2 identification than for Phase 1 detection.

Camera upload mechanism. Uses <input type="file" capture="environment"> — opens the phone's native camera app, which is where the resolution mode (200 MP, etc.) is actually controllable. Browser camera APIs (getUserMedia) cap at ~4 K and can't access the full-res modes, so the file-input route is the high-res path. Server side, PIL handles 200 MP images fine (~600 MB RAM transient; freed once the downscaled version is built).

Cost. ~$0.01–0.02 per Gemini Flash detection call. Free tier covers a generous monthly volume; this won't move the bill. Phase 2 will add per-pin identification on top, where the math shifts to ~$0.02 × N pins per board.

What's next (Phase 2). After Drew validates detection on real board photos: crop each bounding box from the ORIGINAL high-res image, run authenticity check first (skip identification on obvious scrappers), then feed survivors to the existing scan chain at stage 1 only. Persist results as a new BoardScan row + per-pin partial identifications, build a bulk-action grid UI for accepting / dismissing / merging the per-pin candidates.

v0.15.132

2026-05-23

Critical: /validation crash on Windows fix. The v0.15.130 validation card redesign used the strftime("%b %-d, %Y") format specifier — the %-d token (non-zero-padded day) is a Linux/macOS-only extension. On Windows Python it raises ValueError: Invalid format string, blowing up the page render with an internal server error. Replaced with the portable %d (zero-padded: “May 05” instead of “May 5”). Same fix applied preemptively in market_top.html.

Market Top defaults reworked from research. v0.15.131 shipped reasonable first-pass defaults. After researching collector forums (PinForum, WDWMagic, DisneyPinForum), market guides (PixiePin, GoPinPro, MagicalMouseTalk), eBay live sold-price data, and fake/scrapper-density analysis, the defaults are now evidence-based:

Seed queries: 5 → 8.
· DROPPED "Disney pin hidden mickey" — multiple collector sources cite Hidden Mickey as the single most-scrapped pin category on eBay. The query was actively asking eBay to return its highest concentration of fakes.
· ADDED "Disney WDI pin" (Walt Disney Imagineering — structurally limited supply, $125–$565 sold range), "Disney DSF pin" (Disney Soda Fountain / Studio Store Hollywood Surprise Pins, $200–$500), "Disney D23 pin" (D23 Expo exclusives — pre-pandemic jumbos sell $400–$650), and "Disney pin Cast Member exclusive" ($75–$400).

Price band: widened both ends.
· MIN_PRICE $5 → $20. The $5–$20 band on eBay is overwhelmingly $0.99 starting-bid scrapper bait, bulk-lot per-pin averages, and true OE rejects. Not “trending” data.
· MAX_PRICE $500 → $1500. The $500 cap was filtering out legitimate premium sales: WDI Atlantis Milo LE 250 sold $565, D23 Haunted Mansion LE 250 Jumbo $400–$650, HKDL Maleficent Dragon Gate $995. The $1500 ceiling admits these but stops below ultra-rare 1-of-100 Disney Auctions territory ($2,500–$14,250) which isn't trending either.

New: title exclusion filter. Pre-vetting pattern matcher rejects listings whose title matches \blot\b, \bbulk\b, \bmystery\s+pack\b, \bset\s+of\s+\d+\b, \b\d+\s*pin\s*lot\b, or self-disclosed fake terms (scrapper, fake, replica, bootleg). Catches bulk-lot sneakage that gets past the catalog matcher (matcher sees title text only, doesn't understand “set of 50”). Reasoning logged into sweep stats as filtered_title.

Match threshold: 0.5 → 0.6. Observed match-quality distribution shows good matches reliably score 0.65–0.85; the 0.5–0.6 band is where noisy false-positives accumulate (e.g., “Mickey Mouse Christmas” matching “Mickey Mouse Halloween” on shared common tokens). 0.6 should drop ~10–15% of matches but the dropped ones are disproportionately wrong.

What did NOT change: LISTINGS_PER_QUERY (25 still right per eBay relevance ranking), REFRESH_INTERVAL_HOURS (weekly cadence still right — trending doesn't move that fast), per-country location filtering (rejected as too aggressive; legit HK collectors sell HKDL pins).

Considered for v2 if needed: "Disney pin POH" (Piece of History) and "Disney pin Disney Auctions" queries; item_location country filter as opt-in setting. Hold until v1 sweep performance is measured.

Backlog addition: multi-pin board scanning. Adds a new entry to the prioritized backlog: photo a trading board, detect all pins in the image, run each through the existing identify pipeline, return a list. High value (trading boards have 20–100 pins each, manually scanning each one is the current friction) but non-trivial (needs object detection / segmentation before the per-pin pipeline can fire). Tracked as research item for the next development cycle.

v0.15.131

2026-05-23

Market Top — eBay-driven trending pin discovery. New /market-top page that surfaces “what's hot in the pin market right now” from a weekly background sweep of eBay sold listings, vetted against the local CatalogPin mirror so the top list stays clean of fakes, bundles, and unrelated Disney merch.

How it works (the sweep). A scheduler-driven task fires once a week (configurable via market_top.REFRESH_INTERVAL_HOURS). Each sweep hits 5 curated seed queries against the eBay Browse API: Disney pin LE, hidden mickey, pre-production, artist proof, jumbo. For each listing returned, two filters run before persistence:
· Price sanity: only listings between $5 and $500 are kept. Above $500 we get scammer asking prices and ultra-rare items that aren't representative of trending; below $5 we get $0.01 bait listings and bulk lots.
· Catalog match: the listing title is tokenized and Jaccard-overlapped against CatalogPin names. A 0.5+ overlap is required — anything weaker would mean the listing isn't actually about a cataloged pin (could be fake, bundle, or unrelated merch). The match score is stored with each entry; the UI badges “✓ match” vs “~ fuzzy” based on a 0.75 threshold so Drew can decide whether to trust the link.

New persistence layer. MarketTopEntry model holds the per-listing results: seed_query bucket, listing title + URL + image, price, sold-date, FK to the matched CatalogPin, and the match confidence score. Each sweep replaces the previous run's rows so the table stays bounded (~25–50 entries after vetting).

UI. The page groups entries by seed query (so “Hidden Mickey” trending listings are visually separated from “Pre-Production Proof”), each as a card with the eBay image, the matched catalog name, the listing title for verification, a big green price tag, a match-confidence pill, and Pin&Pop / PinPics deep-links. Clicking the card opens the eBay listing in a new tab. Header exposes last-refresh time + a manual “Refresh now” button that bypasses the weekly cadence.

Wiring. scheduler._check_market_top() runs each loop tick and is self-gated by market_top.is_due() so it's a cheap no-op until a week has passed. New nav link surfaces the page from /top-lists; the manual refresh endpoint is POST /market-top/refresh. Sweep budget is capped at 5 minutes via MAX_SWEEP_SECONDS in case eBay stalls.

v0.15.130

2026-05-23

Validation cards redesigned from scratch. Drew flagged the v0.15.126 iteration as still “awful” and asked to scrap and redesign. The previous layout (image left, text right, inside narrow 280px cards) gave QuickAdd metadata so little horizontal room that even rich captures rendered as bare “Unidentified pin #N” with one tiny chip.

New layout, in order of visual priority:
1. Bigger image (110px desktop / 88px mobile) anchors the card — the photo IS the pin's identity until Drew identifies it.
2. acquired_from renders as the first labeled meta row (Drew specifically called this out: “primarily it has the quick add data on it if possible, namely pic and where it came from”).
3. QuickAdd fields (acquired_price, acquired_via, traded_for, notes) each get their own labeled row when populated. Cards with no metadata stay compact; cards with rich metadata feel rich. No empty fixed-height padding.
4. New “Identify this pin →” CTA footer on every card — primary affordance for clearing the queue, replacing the buried “Inbox” link.
5. Red accent retained (per Drew's ask) so the Needs Attention queue stays visually distinct from the yellow Flagged-for-follow-up section.

Smarter metadata grouping. acquired_via="Purchase" with a price renders as “Paid: $12.50 · Purchase” on one row, not two. acquired_via="Trade" with traded_for set renders as “For: <item>” — no redundant “Traded: Trade” row. Gift (or any other Via with no price/trade) gets its own “Via: Gift” row since nothing else implies it.

Grid widened. Old grid: minmax(280px, 1fr) — too narrow for richer cards. New grid: minmax(340px, 1fr) — fits the bigger thumb + multi-row metadata comfortably on desktop. Mobile collapses to a single column (unchanged behavior).

eBay backlog item retired. The “eBay sold-listing history via AI-mediated grounded search (~$0.02/pin)” item turned out to be obsolete: ebay_lookup.fetch_sold_listings already runs on the official eBay Browse API (EXT_AND_SOLD field group), retired the HTML scraper in v0.15.3. The backlog item was based on stale understanding of the current implementation. No code change needed; just a doc/backlog correction.

v0.15.129

2026-05-23

Chain runs in background thread — screen-off mid-scan no longer kills it. v0.15.128 documented this as a known limit: if the phone screen turned off DURING the AI chain (not after), the OS could kill the SSE connection, which terminated the chain itself, so no ‘done’ event ever fired and no push notification arrived. The chain had to complete entirely with the SSE connection still alive for the push to land.

What changed. The chain now runs in a daemon thread that owns its own SQLAlchemy session, independent of the request-scoped session that FastAPI tears down when the SSE client disconnects. The SSE generator's only job is to pass events from the thread's queue back to the browser. When the browser disconnects, the queue keeps filling (nobody reads it, but it doesn't block puts), the chain finishes normally, and the push notification fires from the thread itself — no SSE connection required.

SSE generator simplification. Reads from the thread's queue with a 60s timeout. On timeout, emits an SSE heartbeat (: heartbeat\n\n — valid SSE comment that proxies forward but the client ignores) to keep the connection alive through Cloudflare's ~100s idle drop. Done via a sentinel value ({"_sentinel": True}) the thread emits in its finally block, guaranteeing the generator exits even on chain errors.

Session-thread safety. The request-scoped session NEVER crosses the thread boundary (SQLAlchemy sessions aren't safe across threads). The route validates the scan exists with the request session, then hands ONLY plain data (scan_id, base_url as a string, thresholds as floats) to the thread. The thread opens its own SessionLocal(), re-fetches the scan, runs the chain, and s.close()s in finally.

Provider closures moved. chain_runner, _run_one, _run_serpapi, _tag all moved from the SSE route function into the new _run_chain_in_background_thread helper, so they bind to the thread's session and scan. The POST /ai-identify-chain route (used by the JSON-response chain endpoint) is unchanged — synchronous request handlers don't have the disconnect problem.

What this fixes in practice.
· Stage 1 (4s) chain with phone going dark at 2s — previously could kill the chain mid-stage; now completes + pushes when done.
· Stage 3 deep chain (15s+) with phone going dark immediately — previously useless (SSE killed almost instantly); now runs to completion.
· Closing the browser tab during a chain — chain still finishes server-side, push wakes the phone.

v0.15.128

2026-05-23

Web push notifications on scan-chain completion. Drew's ask: kick off a deep AI chain scan, then put the phone in pocket / let the screen time out / switch apps — have it ping back when the chain finishes so he doesn't have to guess whether it's done yet.

The push infrastructure (VAPID keys, PushSubscription model, /push/config + /push/subscribe + /push/test endpoints, push_notifications.broadcast(), service-worker push handler) all shipped in v0.15.86. This release just wires it into the chain-completion path.

What fires. The /scan/{id}/ai-identify-chain-stream SSE endpoint captures the final ‘done’ event from scan_chain.try_chain_stream as it yields, and after the SSE consumer has gotten the complete stream, calls a new helper _notify_scan_chain_complete which builds a friendly payload and calls push_notifications.broadcast():
· title: “Scan complete”
· body: “{pin name} — {confidence}% confidence”, or “Ready for review — {confidence}% confidence” when no name was settled, or “Ready for review” as final fallback. Truncated to 80 chars (Android/iOS truncate past that anyway).
· url: /scan/{scan_id} — clicking the notification opens the review page.
· tag: scan-{scan_id} — dedup-safe if the hook somehow fires twice.

What doesn't fire.
· Chain errors — an exception inside the SSE chain skips the push; pushing “Scan failed” could mislead on transient network blips that the user will retry anyway.
· No VAPID configured — the helper checks is_configured() first and returns silently. Local dev without keys is unaffected.
· No subscribers — broadcast() returns sent=0 cleanly. Single log line, no errors.
· Push send errors — swallowed in a try/except so a failing push service (rare but happens) does NOT leak into the SSE stream as an error event. The chain itself already succeeded by this point.

Known limit. If the phone screen goes off DURING the chain (not after), the SSE connection can be killed by the OS before the chain finishes, stopping the chain itself — no ‘done’ event ever fires, no push. This needs the chain to background-run independently of the SSE stream (probably a thread pool or scheduler task), which is a bigger refactor and deferred to a later version. For screen-off-AFTER-kickoff with a brief chain (Stage 1 fast = ~4s), most modern browsers will keep the tab alive long enough to finish — that's the common case.

To enable on a new device: visit /settings, click “Enable push notifications,” accept the browser permission prompt. Optional: click “Send test” to confirm the round-trip works before relying on it.

v0.15.127

2026-05-23

Barcode auto-redirect — SKU/PTDB lookup + instant navigation when match is unambiguous + SerpAPI fallback. Drew's ask: scanning a barcode currently lands on a results panel with web-search buttons even when there's a clear local hit. He wanted the modal to skip straight to the matched page when confident.

1. Coverage expansion. The /api/barcode-lookup endpoint now also queries CatalogPin.sku (the v0.15.112 column — many physical Disney pin barcodes are literal SKUs) and CatalogPin.ptdb_id (which was just missing). Net: more barcodes resolve locally without needing a web search.

2. Auto-redirect on unambiguous match. The JSON response now carries a redirect_url field, populated only when there is exactly one match AND that match was an exact ID/SKU/barcode hit (never a name-substring guess). The barcode modal honors it: shows a brief 800ms “✓ Auto-match: <name>” flash, then navigates. Substring matches still render in the list so the user can pick; multiple exact matches (rare but possible — e.g. same SKU on both an owned Pin and a catalog row) also render as a list so the user disambiguates.

3. SerpAPI fallback. When no local match AND SERPAPI_API_KEY is configured, the route issues one Google search restricted to pinpics.com / pinandpop.com / pintradingdb.com domains. The first result whose URL matches our known catalog-source patterns is parsed for its ID, the local CatalogPin row (if any) is looked up to supply a name label, and the source URL becomes the redirect target. Useful for physical-pin barcodes that Disney puts on packaging which match a SKU on the source page's HTML but aren't in our structured DB columns. Cost: 1 SerpAPI call per no-local-hit barcode (free tier 250/mo).

4. Match-quality badges. When the modal does render a list (multiple matches), each result now shows an “exact” badge if it came from an ID/SKU column rather than a name substring. Helps the user pick the right one when there's ambiguity.

5. Notification badge silhouette. Regenerated static/badge-72.png as a transparent silhouette (diamond outline + center dot, fully transparent background). Android renders the alpha channel as a white silhouette in the status bar; the previous PNG had a solid background so Android showed a white square. Lands properly when v0.15.128 web push goes live.

v0.15.126

2026-05-23

Visual fixes batch — tabs, validation cards, scan-result button parity, mobile More-sheet badge.

1. Pin-detail tabs — active state was invisible. The selected tab differed from inactive only by a 3px primary-color underline + --pico-color vs --pico-muted-color text. In dark mode both colors are near-white so the active state effectively vanished — Drew screenshotted the Overview tab with no visible indication. Now: tinted background + bold weight + accent-color text driven by an aria-selected CSS rule, so the first paint matches the JS-toggled state. Also: white-space: nowrap on tab labels so “Overview” never wraps to a hidden second line on narrow viewports.

2. Validation cards — lost data restored. The v0.15.91 swap from acquired-at date to collection-link stripped the most-actionable signal for QuickAdd pins (“when did I capture this?”). Drew flagged the cards as “horribly formatted” with most of the data gone. Restored: acquired-at if set, captured-at (created_at) otherwise — both as a small chip. Added: explicit ⚡ Quick Add badge when needs_validation=True AND no collection assigned, so QuickAdd captures are recognizable even when the “Inbox” collection name collides with a renamed series. The v0.15.91 collection link + acquired_from / acquired_via chips are all preserved — nothing previously displayed is lost.

3. Scan-result “This is it” button — slightly shorter than siblings on mobile. The button was wrapped in a <form> for the POST action; without an explicit display rule the form sized to its content (not to its flex parent), so the button inside ended up ~2px shorter than the adjacent <a role="button"> siblings. Fix: display: flex; align-items: stretch on the form forces the button to fill the form's height pixel-for-pixel.

4. Mobile More-sheet — third badge overlapping the red one. Same root cause as the v0.15.49 fix, just with a new third badge. The amber follow-up count (v0.15.76) and the red validation count both use the same .dpb-nav-count class and shared the cascade-default top-right corner, so the amber painted over the red. New rule targets the SECOND plain .dpb-nav-count (excluding the yellow-soft sibling between them) and places it bottom-right.

Deferred from this batch: the Android status-bar notification icon (Drew's image 7) needs a transparent-silhouette PNG — the existing badge-72.png ships with a solid background so Android renders a white square in the status bar. Asking Drew separately about the icon source.

v0.15.125

2026-05-23

Visual-match thumbnails + AI-resolved catalog refs panel. Two fixes for the /new-pin page after Drew screenshotted the gap.

1. Local catalog visual matches were rendering text-only. The find_similar result dict returned only image_filename, which is the pre-v0.15.33 column. Since v0.15.33 the crawler writes the path to hero_local_path instead (e.g. pinandpop/606.jpg with source subdirectory), and the legacy column stays null. Net effect: every match card showed name + similarity score but no thumbnail. Now find_similar returns both columns and the template prefers hero_local_path, falling back to image_filename for any pre-migration rows. Thumbnails are clickable too — opens full-size in a new tab. Also fixed: missing ptdb_id field in the result dict meant PTDB-sourced matches showed no link out.

2. New collapsible ‘Catalog refs’ panel above Set members. Drew couldn't visually verify which PinPics / PTDB / Pin&Pop pins the AI providers settled on without clicking out to three separate sites. The new panel pulls each AI-resolved source ID's LOCAL CatalogPin row (free — already crawled) and shows them side-by-side: thumbnail, name, year, series, ID link. Collapsed by default (the visual-matches panel above already gets prime real estate); expand to spot-check AI consensus picks. Empty source columns just don't render — if AI providers couldn't identify a PTDB ID, no PTDB card appears, no broken layout.

Both panels use the same /catalog-uploads/<path> static mount that the catalog browse page uses, so no new routes needed.

v0.15.124

2026-05-23

Periodic Chromium recycle + per-burst-pause memory diagnostics. Drew's overnight crawl froze the entire computer at 01:06 AM on the 2026-05-23 long run. App logs showed zero degradation up to the cut (embeds steady at 100-150ms, fetches steady at 0.8-1.3s), which is the signature of an OS-level event (Windows Update reboot or OS OOM) rather than gradual app slowdown. The freeze is most likely not the app's fault — but two changes harden against the next one.

1. Chromium recycle every 200 successful fetches. headless_scrape keeps a single Chromium process alive across the entire crawl lifetime; over many hours each tab/context contributes leaked DOM trees, JS heap, and network cache. After 200 successful headless renders (~30 min of PinPics crawl) the next fetch deliberately closes + relaunches the browser. Costs ~2s for the cold start, amortized across hundreds of fetches. Logged as [headless] recycle after N fetches so we can correlate with memory drops in heartbeat history.

2. Memory snapshot on every burst pause. Each _polite_sleep_for_source burst-pause now writes rss_mb=NNN.N chromium_mb=NNN.N mem_snapshot to the rolling log. Cadence: every burst_size successful requests per source (50-100 typically). After the next freeze: grep mem_snapshot logs/dispindable.log* plots RSS over time. Climbing line = leak (the recycle now buys us time); flat line = OS-level cause.

Drew has psutil installed, so RSS readouts work immediately. The Chromium child-process RSS uses psutil.Process().children(recursive=True) to sum all spawned Chromium processes — the launcher process by itself is misleadingly small because each tab is its own subprocess.

What this does NOT fix: if the next freeze is again a Windows Update auto-reboot or a kernel-level OOM, no amount of app-level recycling saves you. Check Windows Event Viewer (System log) around the freeze time for “Unexpected shutdown” or “Kernel-Power 41” events — those are OS-level and the fix is OS-side (disable auto-reboot, increase pagefile, etc).

v0.15.123

2026-05-23

Pre-computed Pin&Pop embedding lookup + catalog-search cache invalidation on row updates. Drew asked whether /scan and /new-pin actually use the v0.15.113 crawl-time embeddings now that they're populated. Audit found two real issues.

1. Pin&Pop reference embedding was being re-computed every scan. The confidence-scoring path downloads the Pin&Pop catalog photo and runs DINOv2 on it for every /new-pin render — even though v0.15.113 already pre-computes that exact embedding at crawl time and stores it on the CatalogPin row. Now we look up CatalogPin.embedding by pinandpop_id first, fall back to download+embed only when the row is missing or has no embedding. Saves ~500ms per scan when the catalog row exists. As the long crawl progresses, more and more scans will hit this fast path.

2. Catalog visual-search cache wasn't detecting updates. The existing v0.14.48 catalog visual-match panel (DINOv2 via catalog_search.find_similar) was already wired into /new-pin. Good. But its in-memory embedding-matrix cache used row count alone as the change-detection signal — which misses UPDATES to existing rows. The v0.15.121 backfill (run right now to pick up v0.15.118's PinPics full-image fix) rewrites embeddings on existing rows without changing the count, so the cache stayed stale. Now the cache also tracks max(fetched_at) among embedding-bearing rows; any backfill update bumps that timestamp and triggers reload.

What this means for the live long crawl: newly-backfilled embeddings get picked up on the next /new-pin visit without manual cache flush. If you scan a pin during the backfill, the visual match panel reflects the latest catalog state through the same second.

What I did NOT ship: I had started building a parallel find_catalog_matches function in main.py and a second visual-match panel in scan_new_pin.html — caught mid-edit that the existing catalog_search.find_similar + v0.14.48 panel already did this exact thing. Reverted the duplicate work; the existing path remains the canonical implementation.

v0.15.122

2026-05-23

Burst patterns + backfill block-detection. Drew planning a multi-day “scan them all” crawl raised the question: what gets you detected over a long window? Uniform rate + jitter alone produces a tight cluster in the request-time histogram (every interval is base ± jitter, all near base). Real users browse a few pages, pause to read, walk away for coffee, come back. Their histogram is bimodal.

1. Per-source burst pattern. Two new fields: burst_size (request count) and burst_pause_s (long-pause duration). After burst_size consecutive successful requests, the loop pauses for burst_pause_s seconds (±30% random variance, hard-coded). Persisted counter so a server restart mid-window doesn't reset the cycle and skip the pause. Both fields default 0 = disabled (legacy behavior). Form fields on the /catalog-crawler start row alongside rate and jitter.

Recommended values for long “scan all” crawls: PinPics burst=50 / pause=150, Pin&Pop burst=100 / pause=75, PTDB burst=50 / pause=150. With PinPics at rate=8 + jitter=4, that's roughly 8 minutes of crawl per burst pause + 2-3 min coffee break, to Cloudflare's session telemetry. Across 14 days that's ~200 plausible coffee breaks.

2. Shared _polite_sleep_for_source helper. Factored the per-iteration sleep logic out of the crawler loop into a helper that both the regular crawler AND the v0.15.121 backfill thread use. Net effect: any politeness tuning you set on the /catalog-crawler page (rate / jitter / burst) applies to both regular crawl and bulk backfill. No separate backfill-tuning UI needed.

3. Backfill block-detection. v0.15.121's backfill loop didn't check for Cloudflare blocks or 429s — if the source started blocking mid-backfill, the loop kept hammering. Now backfill uses the same _is_block_error + consecutive_blocks + 30-min auto-pause logic as the regular crawler. So if your overnight PinPics backfill triggers Cloudflare, it goes quiet for 30 min, then resumes — no manual intervention needed and no extending Cloudflare's suspicion window through stupid retries.

Note on legacy state: existing state files (pre-v0.15.122) get burst_size=0, burst_pause_s=0, requests_since_burst=0 backfilled silently via the standard setdefault path. Default behavior unchanged until user opts in by setting non-zero values on the form.

v0.15.121

2026-05-23

Bulk catalog backfill. After the v0.15.118 PinPics full-size extraction fix landed, every existing PinPics row in the catalog still pointed at thumbnail files saved pre-fix. Drew could click Re-fetch from source on each row, but for a catalog with thousands of pins that's impractical. v0.15.121 adds a one-click bulk backfill that walks every row of a chosen source in the background, replaying the per-pin refetch logic for each.

UI: new Catalog backfill section on /catalog-crawler (above the failure log) with three buttons (Backfill PinPics / Pin&Pop / PTDB), each labeled with the current row count for that source. While a backfill is running, the section shows live progress (completed / total, percentage, failures count, last-processed pin name) and a Stop button. Status is also surfaced via the existing /catalog-crawler/status JSON poll so the UI updates without page reload.

Concurrency model: backfill runs in its own daemon thread, separate from the regular crawler's per-source threads. Mutually exclusive with the regular crawler — both hit the same source domain and share the headless browser pool, so running them in parallel would double the request rate from PinPics's perspective. The start endpoint returns an error if the regular crawler is running, and vice-versa.

Rate-limiting: respects the per-source rate_limit_s + rate_jitter_s config that's already set on the crawler page. Don't need to configure anything new — whatever polite cadence you've dialed in for the regular crawler applies to backfill too.

Persistence + resume: backfill state (active source, total, completed, failed, last row id) lives in the same crawler state file as the regular crawler. On FastAPI startup, if backfill state shows status=running the thread is relaunched automatically — same daemon-thread + persistent-state pattern as the regular crawler. Current limitation: resume always restarts from the first row of the source rather than picking up at last_row_id. Will add row-cursor resume in a future patch if Drew needs to backfill catalogs large enough to span multiple server restarts.

Per-row code path: identical to the POST /catalog/pin/N/refetch route — purge stale files, force all-images mode (v0.15.119), fetch with force=True to bypass cache, upsert via catalog_ingest._upsert_catalog_pin. So everything that worked for per-pin refetch (including v0.15.120's promote-largest-to-hero) works for backfill.

v0.15.120

2026-05-23

Promote-largest-to-hero on PinPics multi-image pins. Drew tested v0.15.119: refetch now correctly downloads all three images (front + back + extras) at full size — 9.8KB / 246KB / 339KB — but the 9.8KB year-2000 vintage scan was being kept as the hero, so the catalog detail page kept showing the small/blurry image at the top with the sharp 240-340KB photos hidden in the extras tiles.

Why this happens: PinPics presents its original year-2000 scan as the “main” image (big, top-left, with ipsThumb_large) and the modern 2025 user-submitted photos in a carousel below (ipsThumb_medium). The parser correctly extracts all three URLs in HTML document order — front (year-2000) at index 0, modern variants at indices 1+. Pre-v0.15.120 the index-0 file always became the hero, so PinPics's editorial choice overrode visual quality.

Fix: after all downloads complete in _download_all_images, find the largest file on disk and rename it into the bare {pid}.jpg hero slot. The smaller file gets pushed to a freed numbered slot. Three-step rename via a temp file avoids name collision. The returned paths list is reordered to match, so downstream callers (_upsert_catalog_pin setting hero_local_path = paths[0]) see the new ordering automatically without any other changes.

Idempotent: if the largest file is already at index 0, no-op (the swap-needed check is up front). Swap failures (disk full, permission errors) are logged but don't block the upsert — worst case the user gets pre-v0.15.120 behavior. Pin 125 result expected: 125.jpg becomes the 339KB image, 125_2.jpg becomes the original 9.8KB year-2000 scan.

v0.15.119

2026-05-23

Hotfix: refetch route now downloads all images. v0.15.118 fixed the PinPics full-size URL extraction but Drew noticed that clicking Re-fetch from source only downloaded the hero image — pre-v0.15.118 crawler runs (via v0.15.115's all-images mode) were correctly pulling 3 images per pin (front + back + extra), but post-fix refetches only got 1.

Root cause: the module-level catalog_sites.DOWNLOAD_ALL_IMAGES flag defaults False and is only flipped True by the crawler loop. The per-pin POST /catalog/pin/N/refetch route was never wired to set the flag, so when the user clicked the button outside a running crawler session, the refetch path used _download_hero_image and stopped after the first URL.

Fix: the refetch route now brackets the fetch attempt with set_download_all_images(True) + a finally block that restores the prior value. So clicking Re-fetch now downloads everything the crawler would have downloaded, regardless of crawler state.

Note on file size: Drew also asked whether the post-v0.15.118 hero file (9.8KB) is actually full-size. Yes — the URL is pin125.jpg not _thumbpin125...jpg (verified via different byte counts: thumb was 8.4KB pre-fix). Year-2000 PinPics uploads are just small originals (~300x200px). The image looks blurry on the detail page because the <img> element is rendering it scaled-up. Newer pins (2020+) should land at 600-1200px and look noticeably sharper.

v0.15.118

2026-05-23

PinPics full-size image extraction. v0.15.115 successfully enabled all-images mode (front + back + extras now downloading), but all downloads were still thumbnails. Root cause caught by the v0.15.116 diag dump on pin 926 (PinPics #125): PinPics uses Invision Community's lightbox, which stores the full-size URL in data-fullURL (BS4 normalizes to data-fullurl). The parser was checking data-large, data-original, data-zoom, data-full, data-fullsize, data-src-large — every Disney-pin-DB attribute name except the one PinPics actually uses. Fell all the way through to src, which is the thumbnail.

One-line fix: added data-fullurl to the FRONT of the higher-res attribute priority list. Now every PinPics image lands at full resolution — usually 600-800px for the front, vs the 100-150px thumbs we were getting. Also patched the regex fallback path (the no-BS4 branch) to scan for data-fullURL= attributes before falling back to plain src, so both code paths benefit symmetrically.

Backfill: existing PinPics rows in the catalog still point at the thumbnail files saved pre-v0.15.118. Two ways to get full-size:
1. Click Re-fetch from source on the specific catalog row(s) that bother you (one-at-a-time but fastest for a handful of pins).
2. Run a fresh crawler pass with the all-images checkbox checked (will re-process pins via fetched_at cache expiry, getting full-size on the second hit). Backfill ~250k pins this way will take time — bulk option below is faster if you want it now.

v0.15.117

2026-05-23

Phase 3: auto-Series-from-catalog + auto-populate set member placeholders + reconcile-on-save. Drew asked: scan pin A from a series, then days later scan pin B from same series — does the system auto-fill the Series, add it to the checklist, and check off one slot on save? Pre-v0.15.117: no — Series picker stayed blank every time and the set-members panel was display-only. v0.15.117 makes all three work.

1. Auto-Series on /new-pin GET. When the catalog mirror knows this pin's series_name (e.g. “Cats of Epcot”), new _auto_series_from_catalog helper finds-or-creates a Collection row (UI label: Series) and preselects it in the dropdown. Case-insensitive name match catches existing Series; slug uniqueness is enforced with a counter suffix on collision. Green banner appears above the dropdown so the user knows the auto-pick was applied — click any other option to override. Rescans skip (the existing pin's collection_id wins).

2. Auto-populate set members on save. After the user's pin saves, new _auto_populate_set_members helper creates placeholder Pin rows (is_owned=False, no images) for the other members of the set. Two data sources: Pin&Pop's linked_pins_json list (most reliable when Pin&Pop knows the pin), then a series_name-scoped sibling lookup as fallback (covers PTDB-only sets). Capped at 30 placeholders per save to bound damage from bad catalog data. Pre-loaded existing Pin catalog IDs to dedupe in-memory rather than per-candidate query.

3. Reconcile-on-save. When a scanned pin matches an existing placeholder Pin row in the target Series (by any catalog ID), new _find_placeholder_pin_for_save helper detects the placeholder and the POST handler UPGRADES it in place (sets is_owned=True, fills in form data) instead of creating a duplicate Pin row. Pin id stays stable so activity log + cross-references survive. Net effect: scan pin A → its save creates a placeholder for pin B. Scan pin B days later → its save upgrades pin B's placeholder rather than creating a second row. Series's completion percentage ticks up automatically.

Two settings on /settings (both default True): auto_assign_series and auto_populate_set_members. Surfaced as toggle-row switches alongside the existing auto-run-authenticity option. Flip either off to disable the feature globally without code changes.

Caveats Drew should know about. The auto-create Series commits on GET (not save) so visiting /new-pin for a pin in a never-before-seen series creates a Series row even if the user navigates away. Mitigation: Series.name unique index prevents duplicates on revisit. If a problem Series accumulates, delete it from /series. Same-name pin across different sets: handled by scoping the placeholder lookup to collection_id, so a Maleficent placeholder in Series A won't falsely match a Maleficent scan from Series B.

v0.15.116

2026-05-23

Diagnostic route for image-extraction investigation. Drew observed PinPics is downloading thumbnails instead of full-size images. The parser tries higher-res signals (data-large attrs, parent <a href> lightbox) but must be falling back to thumb — we don't know what the actual HTML looks like for the failing pins. Pasted URLs from Drew's example show that the thumb and full hashes differ, so URL-rewriting won't work. v0.14.5-v0.14.7 already tried that and failed.

New GET /catalog/pin/<cid>/diag-images route. Read-only diagnostic. Fetches the source HTML for one pin (priority: Pin&Pop > PinPics > PTDB), then dumps to stdout: every candidate <img> tag's attributes (including data-* custom attrs that might carry the full URL), the parent <a> tag with its href and any other attrs (data-fancybox, rel, class), and 1KB of HTML context around the first candidate <img> in case the real full URL lives in nearby JS rather than on the tag itself.

Button on catalog detail page next to Re-fetch: “🔍 Diag images (log dump).” One click → check server stdout for [diag-images] lines → paste them in chat for parser fix. No state mutations, no downloads, no DB writes — safe to click on production.

v0.15.115

2026-05-23

Production crawler all-images toggle. The v0.15.33 DOWNLOAD_ALL_IMAGES mode existed but the production crawler never enabled it — only the test-batch UI did. Net effect: every catalog row in the DB has front image only. Back-of-pin images are the canonical signal for fake detection (backstamp text + Disney's serial-number format are only on the back); extras are useful for AI cross-comparison and high-res user inspection.

What changed. New per-source download_all_images field in crawler state, defaults True (in _default_source_state). The crawler loop reads it before each fetch and calls catalog_sites.set_download_all_images(value) so the next fetch_pinpics/fetch_pinandpop/fetch_ptdb call pulls the right image set. Single global checkbox on the /catalog-crawler start form (default checked) lets the user opt out for a specific run. Form field flows through to catalog_crawler.start() via a new all_images param. Backward compat: existing state files missing the field pick it up as True via the standard setdefault backfill.

Disk impact: ~3-5x per pin. Each pin has 5 image slots in the catalog_uploads sharded paths (the _download_all_images max_count cap). For a 100k-pin catalog with ~50KB hero shots, full all-images mode lands ~75GB. If that's a concern, uncheck the box on the next crawler start.

v0.15.114

2026-05-22

Hotfix: CLIP embedding compute crashed with 'BaseModelOutputWithPooling' object has no attribute 'squeeze'. Drew hit this on the v0.15.113 manual backfill: all 200 rows in the batch failed with the same error and zero embeddings were written. v0.15.113's inline crawl-time compute would have hit the same bug silently — the per-row exception handler would catch and log it, but no embeddings would ever populate.

Root cause. CLIPModel.get_image_features(**inputs) was supposed to return a bare tensor (with a .squeeze() method), but in Drew's installed transformers version it returned a BaseModelOutputWithPooling object instead. Splatting the processor's full dict (pixel_values plus version-dependent extras) was routing through CLIPModel's full forward path instead of the image-only branch, producing a ModelOutput object rather than the intended tensor.

Fix. Two layers of defense in embeddings_clip.py:
1. Explicit argument passing. Extract pixel_values from the processor output and call get_image_features(pixel_values=...) directly. Same for text: pass input_ids + attention_mask explicitly. Forces the image/text-only code path regardless of what extras the processor adds.
2. Defensive output unwrap. If the result still comes back as a ModelOutput object in some future transformers version, check for .image_embeds (or .text_embeds) as the projected vector, fall back to .pooler_output if not. Only raise a clear RuntimeError when neither attribute exists. Bare-tensor return (the documented behavior) still works unchanged.

Action items for deploy. Click the Backfill CLIP button on /catalog/browse to retry the failed batch — the v0.15.113 inline compute will start populating new crawls automatically. DINOv2 was not affected by this bug (different module, different API).

v0.15.113

2026-05-22

Phase 2: inline embedding compute at crawl time (DINOv2 + CLIP). Drew confirmed adequate CPU for both models. Previously the crawler downloaded a hero image and left both embedding columns NULL, requiring manual clicks on the /catalog/backfill-embeddings and /catalog/backfill-clip-embeddings buttons to keep visual + text search current. Users forget; 100+ clicks for a populated catalog is genuinely tedious; visual-search on /new-pin and text-search on /catalog/text-search go stale as un-embedded rows accumulate. v0.15.113 hooks both models into the crawler's image-download success path so every newly-crawled or refetched pin gets embedded automatically.

How it works. New _maybe_compute_embeddings(row) helper in catalog_ingest.py is called at the end of _upsert_catalog_pin — the same chokepoint every fetch path (crawler, refetch handler, prewarm, scan match) already lands at. When a row has a hero_local_path on disk and the corresponding embedding column is NULL, the helper loads the file, runs it through the model, stores the float32 bytes blob. Both models run independently — DINOv2 failure (e.g. OOM) doesn't block CLIP. Idempotent: re-upsert with the same hero is a fast no-op (no recompute).

Hero-swap detection. When hero_local_path changes between scrapes (rare but real — e.g. v0.15.109's PTDB thumb-promotion patch upgraded _thumb.jpg → full-size paths), the helper would otherwise keep the stale embeddings of the OLD image. Caller snapshots the prior value before the overwrite, then clears both embedding columns when the path actually differs. The helper recomputes from the new image on the next call. New rows have prior=None and don't trigger spurious clears.

Defensive on partial writes. If the on-disk hero file is missing or smaller than 100 bytes (suggesting an incomplete write), the helper skips silently. The existing image-path backfill and the cache-invalidation patches from v0.15.108+109 handle file-recovery; embeddings are downstream of healthy image storage.

Toggleable feature flags. Two module-level constants in catalog_ingest: EMBED_AT_CRAWL_TIME (DINOv2) and EMBED_CLIP_AT_CRAWL_TIME (CLIP). Both default True. If a future torch upgrade breaks inline compute, flip either to False and restart — the manual backfill buttons stay as the recovery path. Existing data is untouched either way.

Cost. ~200-500ms CPU per pin for DINOv2 + CLIP combined. The crawler already sleeps 5s between requests for politeness, so the embedding compute is invisible to throughput. Logged as [embed-at-crawl] row {id}: {Model} computed in {N}ms so you can verify timing on your CPU and catch any performance regression.

For existing pins. Inline compute applies to newly-crawled rows AND to rows revisited by Re-fetch / the per-source “Retry failures” button. Legacy rows that aren't touched by a crawl can still be embedded via the existing backfill buttons on /catalog/browse — those remain as the recovery path.

v0.15.112

2026-05-22

Phase 1: catalog column promotion + PTDB market signals extraction. Drew flagged that the parsers were extracting more data than the schema was capturing — fields like Pin&Pop's original_price / edition_size / rarity / sku, and PTDB's entire “Pin Stats” block (Desperation, For-Keeps, For-Trade, Wanted, etc.) were landing in the metadata_json blob and never again seeing daylight. Since we're already paying the HTTP cost of the scrape, there's no reason to throw that data away.

22 new columns on catalog_pins. Shared catalog fields: original_price, edition_size, rarity, sku, origin, edition, release_date, retire_date, set_size, pack_size, tags_json, linked_pins_json, description, added_by. PTDB-specific market signals: ptdb_desperation, ptdb_for_keeps_count, ptdb_for_trade_count, ptdb_wanted_count, ptdb_sellers_count, ptdb_buyers_count, ptdb_sell_price_low, ptdb_sell_price_avg. All nullable; _LATE_ADDITIONS entries added so existing DBs pick them up on next startup without a manual migration.

New _promote_catalog_fields helper in catalog_ingest centralizes the dict-key → column mapping. Called by _upsert_catalog_pin on every live scrape AND by the new backfill route to replay existing JSON blobs. Three semantic rules: (1) never overwrite a non-null string column unless the new value is meaningfully longer (later scrapes from richer sources can add detail); (2) numerics always take the latest read (market signals are time-sensitive); (3) common “no value” sentinels (Unknown!, N/A) are treated as null so columns stay clean for filtering.

PTDB parser now extracts the full Pin Stats block. Walks every <tr> table-row and <dt>/<dd> definition pair, builds a case-folded label map, then looks up values by substring match with exact-match precedence (so “origin” doesn't substring-collide with “original price”). Returned dict now carries desperation, for_keeps_count, for_trade_count, wanted_count, sellers_count, buyers_count, sell_price_low, sell_price_avg, original_price, origin, sku, release_date, retire_date, added_by, tags, and a richer description from the Details section.

One-shot backfill at POST /catalog/backfill-catalog-columns (button on /catalog/browse). Walks every catalog row in 500-row batches, replays metadata_json through _promote_catalog_fields, commits. Zero re-scraping — pure JSON read → column copy on data already on your disk. Idempotent: safe to re-run after a future schema bump. Reports a flash summary with updated / skipped / parse-error counts.

Catalog detail page (/catalog/pin/<id>) gets three new panels showing the new data: Catalog details (original price, edition size, rarity, SKU, origin, dates, set/pack sizes, tags), PTDB market signals (the full Pin Stats block with Desperation highlighted as the key trade-leverage signal), and Description (with contributor credit). Each panel only renders when at least one field is populated — sparse rows don't show empty boxes.

What this means for already-scraped pins. Nothing is lost. Every prior scrape result is preserved verbatim in metadata_json; the backfill route is a pure JSON-blob → column copy on existing on-disk data. No re-scraping required. Hit the “Backfill catalog columns” button on /catalog/browse after deploy and your existing ~83 PTDB rows will pick up their market signals immediately. PinPics + Pin&Pop rows will populate whatever they have (Pin&Pop is rich; PinPics is sparse). Future crawls auto-populate.

v0.15.111

2026-05-22

Click-to-zoom on catalog detail images. Drew flagged that there was no way to see a catalog image at its native resolution — the on-page render is constrained to the column width with aspect-ratio: 1/1 + object-fit: contain, so a 1500px PTDB hero displays at ~400px. Wrapping the hero and each extras thumbnail in <a target="_blank"> pointing at the underlying /catalog-uploads/... URL gives a one-click path to the raw file, where the browser renders it at native resolution and the user can zoom, right-click-save, or share. No JS, no lightbox library, no server-side work; cursor: zoom-in + “Open full-size in new tab” tooltip telegraph the interaction. rel="noopener" for tabnabbing protection.

Catalog-detail-only for this change. The catalog browse grid already routes clicks to the detail page (correct behavior). The scan / pin-detail pages have their own gallery interactions and aren't touched here.

v0.15.110

2026-05-22

Per-source “Retry failures” buttons on the catalog crawler page. Drew asked for the bulk equivalent of the per-pin Re-fetch button — clicking through the failure log one at a time doesn't scale once you have a fix that should retroactively heal hundreds of previously-broken rows (e.g. v0.15.104 fb.png filter, v0.15.109 thumb promotion + stale-cache invalidation).

New button per source on the Failure log section. Clicking “Retry pinpics failures” (or pinandpop / ptdb) calls catalog_crawler.queue_failure_retry(source): reads the lowest + highest failed pin_ids for that source, jumps the crawler's current_id cursor to the lowest, sets max_requests_per_run to (highest - lowest + 1) so the worker stops naturally at the end of the range, and starts the thread. All v0.15.108+109 fixes (fetched_at update on upsert, stale-cache invalidation when only blocked URLs remain, PTDB thumb promotion) apply automatically because the retry uses the normal crawler code path.

Per-source by design (Drew's explicit ask) rather than all-at-once: different sources may need different retry strategies, and clicking “retry all” while one source is mid-run on a clean ID range would disrupt healthy work. Each button has a confirm-on-submit so accidental clicks don't hijack a running crawl.

Optional outcomes form field on the route (POST /catalog-crawler/retry-failures) lets a future UI iteration restrict the retry to specific outcome types (e.g. “errored,blocked” to skip ID-space gaps marked as not_found). Default retries every logged outcome for the source.

Caveat: the crawler walks consecutive IDs, so retry will also re-visit any successful IDs between the lowest and highest failure. Those are cheap (30-day cache hits, no HTTP), so the cost is small. A “skip between failures” mode would be a real refactor of the worker loop; deferred until the cost becomes real.

v0.15.109

2026-05-22

Two PTDB fixes from Drew's v0.15.108 follow-up.

1. Full-resolution PTDB hero images (no more low-res thumbnails). Drew refetched PTDB #52 in v0.15.108 and the catalog image came back at 2.7KB — clearly a thumbnail, not the high-quality version visible on the source page. Root cause: PTDB's CDN serves two variants of each image: ptdb.co/storedImages/<n>_thumb.jpg for the gallery thumbnail and ptdb.co/storedImages/<n>.jpg for the full-resolution version. The parser's body-image walk picks up the thumbnail URLs from the gallery <img> tags because those appear above the lightbox-trigger markup in PTDB's page structure.

Fix: strip _thumb from the path immediately before the image extension on every URL the parser collects. Pattern re.sub(r'_thumb(\.(?:jpe?g|png|webp|gif))(\?|#|$)', r'\1\2', ...) — case-insensitive, preserves query strings + fragments, idempotent on already-clean URLs. Now PTDB heroes download at the full resolution PTDB's lightbox would have served.

2. Crawler now refreshes pre-v0.15.104 cached pins. Drew correctly identified that the per-pin “Re-fetch from source” button works but the catalog crawler doesn't fix the same broken rows on a re-pass over the same IDs. Root cause: the per-pin refetch handler passes force=True to fetch_ptdb, bypassing the cache. The crawler doesn't — it calls fetch_ptdb(pid) with the default force=False, hits a cached entry from a pre-v0.15.104 crawl that contains image_urls=['.../fb.png'], and _backfill_image_if_missing looks at the disk file (the bad FB-sign-in JPG, well over the 100B size threshold), declares everything fine, and walks away. Crawler “succeeds” without doing anything.

Fix: on every cache hit, scan the cached image_urls against the same blocklists the downloader uses (_DOWNLOAD_HOST_BLOCKLIST, _DOWNLOAD_URL_BLOCKLIST, _DOWNLOAD_FILENAME_BLOCKLIST). If every URL in the cache would be rejected, the cache is stale — treat as a miss, purge the disk file (same logic as the v0.15.103 per-pin refetch), and fall through to a fresh source fetch. Caches with at least one valid URL run through the normal backfill path unchanged.

Net effect: the next crawler pass over previously-broken PTDB IDs will automatically re-parse the source, get clean URLs (now promoted to full-size via fix #1), download fresh images, and update the catalog row + cache. Drew can let the crawler chew through the affected ID range instead of clicking “Re-fetch” on each pin individually.

Bonus: extracted the post-v0.15.104 filter logic into a catalog_sites._looks_unblocked(url) helper so both the downloader and the cache-staleness check share the same source of truth. Future blocklist additions land in one place.

v0.15.108

2026-05-22

The real reason refetched PTDB pins kept showing FB sign-in: fetched_at never updates on upsert. Drew refetched PTDB #53 and #54 after v0.15.107. The server log showed the parser correctly finding real image URLs from ptdb.co/storedImages/. But the catalog detail page kept rendering the FB sign-in graphic. Image 2 of the bug report holds the smoking gun: Fetched: 2026-05-20 17:28 UTC — that timestamp is from the ORIGINAL crawl, NOT from today's refetch.

_upsert_catalog_pin in catalog_ingest.py was updating last_seen_at on every upsert but never updating fetched_at — which is set only as the SQLAlchemy column default for newly-inserted rows. v0.15.106 added ?t=<fetched_at-timestamp> as a query-string cache-bust on catalog image URLs. With fetched_at frozen, the URL is identical before and after refetch — so the browser keeps serving its previous (FB sign-in) cached version regardless of what we wrote to disk. The download may have succeeded silently the whole time.

Fix: existing.fetched_at = datetime.utcnow() on every upsert. Every caller of _upsert_catalog_pin has just finished a fresh fetch from the source (crawler, refetch handler, prewarm path) — no path re-uses cached data without re-fetching — so updating is semantically correct. Cache-bust URLs now change on every refetch, forcing the browser to re-pull the image.

Bonus fix: _download_hero_image brought to parity with _download_all_images. Drew's v0.15.107 log showed [ptdb] DONE with no [catalog_sites] lines either way. v0.15.106 added success-path logging only to _download_all_images, which made it impossible to tell whether _download_hero_image (the default code path when DOWNLOAD_ALL_IMAGES=False) was silently succeeding or silently failing. Same logging now applies to both: hero dl OK 18432B → 53.jpg on success, hero HTTP 404 / hero wrong content-type on failure paths. Also added the v0.15.100/102/104 filename + host + URL-pattern blocklists (FB sign-in / fb.png / etc.) here for defense-in-depth: pre-v0.15.104 cached image_urls may still contain those URLs, and the parse-time filter doesn't run on cache hits.

v0.15.107

2026-05-22

Cache-bust catalog browse thumbnails too. v0.15.106 added ?t=<fetched_at> cache-busting on the catalog detail page (catalog_pin_detail.html) but missed the catalog browse grid (catalog_browse.html). Net effect Drew flagged: after re-fetching a PTDB pin in v0.15.106, the detail page rendered the real Jiminy Cricket image correctly, but the catalog browse grid still showed the cached FB sign-in graphic for the same row. Same fix applied to the thumbnail grid: query string changes only when fetched_at does, so unchanged rows still cache normally. The next time any pin's fetched_at updates — via refetch or a fresh crawler pass — the browse grid will pick up the new image without a hard refresh.

v0.15.106

2026-05-21

Three fixes from Drew's v0.15.105 regression run.

1. Loud diagnostics on silent download failures. Drew refetched PTDB #55 (Jiminy Cricket). v0.15.105 server log showed the parser found a real pin URL: http://www.ptdb.co/storedImages/111_thumb.jpg. The refetch endpoint purged the old FB sign-in jpg from disk and called _download_all_images. But the catalog detail page still showed the FB image after the refetch. Working hypothesis: the new URL returned a non-200 status (PTDB's CDN domain ptdb.co may be down / 404'd / geo-blocked / returning HTML), and _download_all_images silently skipped via if r.status_code != 200: continue. Net effect: no file on disk, no log line, total mystery.

Fix: every skip path inside _download_all_images now logs the HTTP status code and content type before continuing. The success path also logs file size and final filename, plus an end-of-call tally (2/3 saved) so it's easy to spot when N URLs were parsed but only M files landed. Drew can re-fetch PTDB 55 after this version and immediately see in the log what's failing — whether it's 404, content-type, or something else.

2. Cache-busting on catalog detail images. Even if downloads succeed, Chrome aggressively caches static catalog files (/catalog-uploads/...) on the URL alone — so when refetch writes a new image at the same path the browser keeps showing the old one. Adding ?t=<fetched_at-yyyyMMddhhmmss> as a query param to each image URL on /catalog/pin/<id> forces a new request whenever the row's fetched_at changes. Normal caching is preserved for unchanged rows (query string only changes when the data does).

3. Per-source crawler jitter. Drew asked for “5 seconds between scan +/- x seconds”. Added rate_jitter_s per source in catalog_crawler; actual sleep is now uniform(rate - jitter, rate + jitter), floored at 0.1s. Form field next to the existing rate input on each source row of /catalog-crawler; default 0 = legacy perfectly-regular cadence. The per-source rate badge now displays 5.0s/req ± 1.0s when jitter is non-zero so the current setting is visible at a glance.

v0.15.105

2026-05-21

Search-by-name buttons on the No-catalog-match panel. Drew flagged the use case: when Pin&Pop / PinPics returns nothing visually but the AI providers gave a partial name guess (e.g. “Lumiere Beauty and the Beast”), he wants a one-click jump to each catalog site's text-search for that name so he can browse manually and pick the right ID. SerpAPI Google Lens can't be filtered by site (no site: param on the Lens engine), so this is the next-best path.

Three buttons added at the bottom of the No-catalog-match panel — PinPics, Pin&Pop, PTDB. Uses catalog_sites.pinpics_search_url and catalog_sites.pinandpop_search_url (already registered as Jinja globals since v0.15.X) for the first two, and the canonical pintradingdb.com/pinList.php?txtSearch=<name> URL for the third. Each opens in a new tab so the user can browse without losing the scan context. Workflow note in the panel: click the right result, copy its URL, paste into the “Import from URL” field in the pin details section below.

Best-available-name resolution. The search query uses the highest-priority name across providers: chain winner (ai_suggestion.name) first, then falls back through Claude, Opus, GPT, GPT-5.5, Gemini Pro, Gemini Flash — whichever provider produced a name guess wins. If no provider produced a name at all (genuinely unidentifiable pin), the buttons are replaced with a one-line hint pointing to the pin-name field; pasting an empty query into PinPics produces a no-results page which would look broken.

Why this is the right scope. The longer-term answer to “no catalog match” cases is more local ingestion via the crawler (which Drew is actively working on across all three sources). This feature is the bridge: when the local mirror doesn't have a pin yet, the user can find it on the upstream sites in two clicks instead of opening tabs manually.

v0.15.104

2026-05-21

PTDB Facebook sign-in image — finally caught. The v0.15.103 diagnostic logging revealed the actual URL that's been getting through every filter generation since v0.15.100:

[ptdb] https://www.pintradingdb.com/pin/55: 2 pin image(s) found; first='https://www.pintradingdb.com/images/fb.png'

PTDB hosts the “Sign in with Facebook” button as <img src="/images/fb.png"> on its own domain. Every filter we built was looking for FB-flavored URLs and missed this one:<ul><li>v0.15.100 Facebook CDN host blocklist: host is pintradingdb.com, not fbcdn.net → pass</li><li>v0.15.100 UI_PATTERNS substring list (logo, login, signin, button, etc.): filename fb.png doesn't contain any of those substrings → pass</li><li>v0.15.102 download URL blocklist: same problem → pass</li></ul>The fix: filename-EQUALITY check (not substring). New UI_FILENAMES frozenset containing fb.png, facebook.png, tw.png, gp.png, etc. — the small set of bargain-bin social-button filenames that sites of PTDB's era tend to hardcode. The parser extracts the URL basename (stripping query strings + fragments) and rejects on exact match. Equality avoids the false-positive risk of e.g. fb_pinpics_hero.jpg getting nuked by a “fb” substring rule. Pre-emptively includes Twitter / Google / Instagram / YouTube equivalents since sites that hardcode fb.png usually have the matching set.

Applied in two places: ptdb_sites.looks_like_pin_image (parse-time, rejects the URL before it ever lands in image_urls) and catalog_sites._download_all_images (download-time, defense-in-depth for cached pre-v0.15.104 image_urls lists that still have fb.png in them).

Expected effect after re-fetch. PTDB #55 (Jiminy Cricket #1) should now download the real pin image. PTDB #72 (and other legitimately-empty pages) should land in the catalog with no image at all — the v0.15.103 hero_local_path = None path takes over once image_urls comes back empty.

v0.15.103

2026-05-21

Crawler checkboxes — fifth and final attempt. After v0.15.75 (width/height @ 1.15rem), v0.15.76 (transform: scale), v0.15.100 (width/height @ 1.25rem), and v0.15.102 (width/height @ 1.6rem + accent-color), the checkboxes were still rendering tiny. Drew confirmed via DevTools that bumping the box to 3rem × 2.6rem made the BOX visibly larger but the checkmark glyph inside stayed tiny. Root cause: Chrome / Firefox / Edge all draw the check glyph at the input's intrinsic font-size, not scaled with the box dimensions. width/height controls the hit-area but not the glyph rendering. accent-color only changes the color of the check, not its size.

The fix: stop fighting the native render. New .dpb-big-checkbox class in base.html uses appearance: none to suppress the browser's checkbox entirely, then draws a custom checkmark with a ::after pseudo-element — two rotated border edges in rem units so the glyph scales with the box at any size. Hover + focus-visible + disabled states all themed to match the app's primary accent. Works identically across Chrome, Edge, Firefox, and the Windows WebView; works in dark mode without any theme-specific overrides. Applied to the three crawler source checkboxes; any future checkbox that needs this treatment can use the same class.

PTDB re-fetch — actually swaps the image now. v0.15.102 added the POST /catalog/pin/<cid>/refetch endpoint but explicitly noted (in its changelog) that it didn't delete existing local image files first. Drew tested on PTDB #55 and the catalog still showed the FB sign-in image after re-fetch. The server log confirmed the fix was working at the PARSING level ([ptdb] 55: DONE in 0.5s ... images=2) — two real pin URLs were found. They just never got downloaded, because _download_all_images skips when the destination file exists with size > 100 bytes, and the OLD pre-v0.15.100 file (the FB sign-in image) was still sitting at catalog_uploads/ptdb/000/55.jpg.

v0.15.103 makes the refetch endpoint purge local image files first before triggering the fresh fetch. Walks all three sources (Pin&Pop / PinPics / PTDB) for each ID the catalog row has, deletes {prefix}/{id}.jpg and its numbered siblings _1.jpg .. _12.jpg, then proceeds with the force=True fetch. Reports the count of purged files in the flash message so you can see it actually did something.

Also: clear hero_local_path when fresh fetch returns no images. PTDB has legitimately-empty pages (Drew showed pin #72 as an example — the source page has no pin image). v0.15.102 already stopped writing the FB sign-in graphic in this case, but the catalog row's hero_local_path column kept pointing at the (now-deleted) old file. Now: a fresh fetch with no images clears the row's image columns so the catalog detail page renders “no image available” cleanly instead of a broken image link.

Parser diagnostic logging. _parse_ptdb_html now logs the first found image_url on success, in addition to the empty-result warning. So you can grep the server log for [ptdb] after a crawler run and verify the URLs look real (e.g. https://www.pintradingdb.com/images/pins/55/jiminy.jpg) vs garbage (...fb_signin.png). Cheap to write, saves time on the next round of debugging.

v0.15.102

2026-05-21

PTDB Facebook sign-in image — deeper fix + manual re-fetch. v0.15.100 added a Facebook CDN host blocklist to ptdb_sites.looks_like_pin_image; v0.15.101 went out for the fake-corpus stall; Drew came back with /catalog/browse showing — still — every PTDB pin pulling back the Facebook “Sign in” graphic. Two issues stacked here that v0.15.100 didn't catch:

1. The og:image fallback was the leak. When PTDB's body <img> walk returns zero results — which happens when the pin photo is loaded via JS after page render, or rendered inside a <div style="background-image: url(...)"> instead of an <img> tag — the parser would fall back to og:image. v0.15.53 already established that PTDB's og:image is unreliable (the “Sign in” graphic was specifically named in the v0.15.53 commit), but the fallback was still wired up as a “better than nothing” last-resort — the FB CDN check in v0.15.100 doesn't help here because the og:image URL is hosted on pintradingdb.com, not on Facebook's CDN. The og:image fallback is removed entirely. Pins where the body walk finds no images now return an empty image_urls list, the log gets a no pin images found warning, and those rows land in the catalog with no image rather than the wrong image.

2. CSS background-image walker added. Before giving up and returning empty, the parser now ALSO walks elements with style="background-image: url(...)" declarations and extracts those URLs through the same looks_like_pin_image filter. PTDB pages that render the pin photo in a styled <div> should pick up correctly going forward.

3. Defense-in-depth download blocklist. catalog_sites._download_all_images now skips any URL whose host matches the FB CDN list OR whose path contains a known UI-element substring (sign-in, signin, fb_logo, login_button, etc.). Catches the cache-backfill path where pre-v0.15.100 cached image_urls lists still have stale FB URLs — _backfill_image_if_missing reads from cache without re-parsing so the parse-time filter doesn't run there. Also catches connect.facebook.net which the v0.15.100 list missed.

4. Per-pin Re-fetch button. POST /catalog/pin/<cid>/refetch + a button on the catalog pin detail page. Force-re-scrapes the pin from its source (Pin&Pop → PinPics → PTDB priority), bypasses the 30-day cache, and runs the fresh result through the v0.15.102 filters. Use this after a scraper fix lands when you want to update a specific pin without waiting for cache expiry. Note: doesn't auto-delete the existing local image file at catalog_uploads/<source>/<prefix>/<id>.jpg — the v0.15.33 downloader skips that path if the file already exists. For the FB sign-in case specifically, you need to delete the bad file from disk first; a future v0.15.103 could auto-purge known-bad files by hash.

Crawler checkbox sizing — round 3. v0.15.100 swapped transform: scale(1.6) for direct width: 1.25rem sizing because the transform-scaled box rendered as a thin blue tick in dark mode. The 1.25rem size was correct as a hit area, but Pico/the browser draws the checkmark glyph at the input's intrinsic font-size, not the box size, so on a small box the tick reads as a sliver. v0.15.102 bumps to 1.6rem — the box is square, the glyph reads as a proper check. accent-color also added so the check tints to the app primary instead of OS default blue.

AI provider cards caption clarified. Drew asked whether a Smart-chain scan was hiding data because the results page showed fewer provider cards than a Manual scan. It's working as intended — the chain stops at the first stage that clears the confidence threshold, on purpose, to save API budget. The provider-cards summary now spells this out + points to the Manual providers section under /scan for users who want a second opinion.

v0.15.101

2026-05-21

Reddit fake-corpus scrape stall — root-caused + fixed. v0.15.100 shipped a diagnostic-only note pointing at stale .fake_corpus_state.json state; turned out the actual bug was deeper. The v0.15.88 already-running guard in _run_fake_corpus_scrape_bg was bailing on every scrape because the guard read back state that the kickoff site (POST handler / page auto-sync / scheduler) had just written via begin_running() a moment earlier — same process, same file, no real competitor in sight. From the bg task's perspective, “another scrape is already running” was true, so it returned. Net effect: the UI showed the live-progress section (because begin_running wrote the dict synchronously) but the bg worker never advanced past page 0. Drew's screenshot of the stalled banner was the visible symptom.

The fix — identity-aware guard. begin_running() now returns the started_at ISO timestamp it wrote. Each of the three kickoff sites — manual POST, page auto-sync GET, scheduler thread — captures this token and passes it forward to the bg task as expected_started_at. In the bg task, the guard bails only if the running record's started_at does not match the token (someone else's scrape genuinely owns the state) — the same-identity case proceeds. The original purpose of the guard (preventing double-clicks + scheduler vs manual races) is preserved; the false-positive same-process case is gone.

Stalled-banner button label. The button on the “Scrape may have stalled” banner used to say “Dismiss”, which read as “hide this” rather than “clear the lock so I can retry”. Renamed to Clear & retry in the stalled case; completed-run / errored cases keep the Dismiss label since those don't need a lock-clear. With v0.15.101's identity-aware guard, the Clear & retry path is rarely needed in practice — only after a real server crash leaves stale state behind — but the affordance should still communicate what it does.

v0.15.100

2026-05-21

Ten-fix UI batch + scan flow streamlining. Drew flagged a dense set of issues during the v0.15.99 review; this version cleans them up in two layers — pure UI fixes across the app, plus a streamlining of the /scan page that promotes Smart chain to the primary flow.

UI fixes.
1. Search focus border calmed. The expanded magnifying-glass pill was framed in the same accent (purple primary) used for buttons. Changed to a muted border tint so the focus state is visible but doesn't shout. The yellow/orange edge in Drew's screenshot was the accent rendering against the dark background — same hex, but reads differently next to the brand row.
2. /pins sort arrows restored. <table class="dp-sortable-table"> had its CSS living only inside the collections.html <style> block, so /pins (which uses the same class names + the global sort JS in base.html) got the behavior but never the up/down arrow indicators. The sortable-table styles moved to a global block in base.html. Folder pill styles moved with them since they have the same scope problem.
3. Scan header wrap fixed. The lede line “The app searches your reference database (132 images indexed)” was breaking inside the parenthetical — “indexed)” ended up on its own line. The count phrase is now wrapped in white-space: nowrap so the parens stay together as one unit and wrap as a whole.
4. /series + /pins delete X tightened. The destructive × button used dp-btn-sm's default 0.2rem/0.55rem padding, making it visually wider than tall. Added dp-btn-icon variant that locks the button to 1.65×1.65rem with the X glyph centered. Applied to /pins, /series, /collections, and pin detail.
5. Activity search filter chip spacing. The “Filtered to ‘X’ · clear” line was butt-up against the search input's bottom border in Drew's screenshot. It now sits on its own flex row below the input with 0.4rem of breathing room above.
6. Catalog crawler checkboxes sized. The PinPics/Pin&Pop/PTDB enable checkboxes used transform: scale(1.6) which left them as narrow blue ticks on dark mode — not recognizable as checkboxes. Replaced with direct width: 1.25rem; height: 1.25rem; sizing so they render as proper square boxes.

/scan page streamlining.
7. Smart chain promoted to primary flow. Drew flagged that the page had too many ways to identify a pin: cheap-mode toggle, three scan-preset buttons, an advanced provider grid, the smart chain, AND individual single-provider buttons. The chain has been the best path since v0.15.97 streamed real per-stage progress; everything else is now backgrounded. Specifically:
• Smart chain section is expanded by default with a primary-styled “Run smart chain” button. It's the first thing the eye lands on under the photos.
• Cheap-mode toggle removed. The chain picks providers stage-by-stage based on confidence; explicit cheap-mode is no longer the right knob. (Pricing is still shown next to each stage in the chain's output table.)
• Scan preset / advanced provider grid is now collapsed by default and labelled “Manual providers (advanced)”, parked BELOW the chain. The same checkboxes + preset buttons + cookie state are intact for anyone who relied on them; they just don't lead anymore.
• Single-provider buttons removed. “Or trigger one provider at a time” — Identify with Claude / Opus / GPT / GPT-5.5 / Gemini buttons — were kept “for A/B testing” but Drew said they had ceased being useful. Catalog-manually is preserved as a small link inside the chain section.
8. Smart chain auto-opens /new-pin on done. The SSE done handler used to render a final-summary line + a “Open /new-pin →” button the user had to click. Now window.open() fires automatically when the chain settles, mirroring the new-tab behavior the single-provider buttons used to have. The chain's stage-by-stage table stays visible in the original tab as the audit trail. The SSE endpoint already persists scan.*_suggestion_json stage-by-stage (main.py:17959) so the /new-pin tab loads with real data ready — no extra wiring needed.

/new-pin enrichment + Lens cleanup.
9. Lens catalog matches collapse beyond 2/source. Mystery-pack pins returned 5-7 Pin&Pop matches each (one per series member). The first 2 per source (PinPics / Pin&Pop / PTDB) render inline; the rest collapse into a “Show N more from <source>” <details>. Status line still shows the full count so nothing's hidden — just less scrolling on common-case Mystery results.
10. AI provider cards section default-collapsed. v0.15.13 added a default_expand_advanced app_setting to control the default state; somewhere along the way Drew's setting flipped to True and stayed there. Removed the conditional — the cards section is now always collapsed by default and the user expands when they need the per-provider breakdown.
11. Acquisition fields on /new-pin. “Acquired on / How / Paid (USD) / Acquired from / Traded for / Notes” — the fields from Quick Add — are now rendered on /new-pin too. The Pin model already had these columns; the form just never exposed them in the scan-to-new-pin flow. POST handler picks them up and writes them to both the new-pin save path and the rescan update path.

Crawler hardening.
12. PTDB Facebook-CDN image filter. All PTDB crawls were pulling back Facebook “Sign in” thumbnails as the pin photos. PTDB pages embed FB social widgets via scontent.*.fbcdn.net, fbsbx.com, and fbcdn.net hosts. The scraper's <img> walk wasn't filtering these out, so the first FB iframe image won. New filter list drops any URL on a known FB CDN host before image candidates are scored. Re-crawl your PTDB entries to pull real images now.
13. Reddit fake-corpus scrape stall. Diagnostic-only this version — the stalled-banner state Drew screenshotted (page 0/1, processed 0, added 0) means _run_fake_corpus_scrape_bg never made it past the “already running” guard. Most likely a stale running dict in .fake_corpus_state.json from a prior crash. Click Clear stuck state in the stalled banner, then retry Scrape. If it re-stalls immediately, the server log will now show [fake-corpus/bg] another scrape is already running — that's the smoking gun and points at a real concurrency bug worth a v0.15.101 dig.

v0.15.99

2026-05-21

Six UI fixes batched together. Drew flagged a stack of cosmetic and routing issues from the v0.15.98 deploy; this version cleans them up in one pass.

1. GET /collections/{id} 405 fixed. Clicking a folder pill on the Series page returned Method Not Allowed because /collections/{fid} was POST-only (used for the edit form). The same broken link lived on /pins too. Now GET /collections/{fid} redirects to /series?folder={fid} — the ?folder= filter is also new in this version and restricts the Series list to one collection's contents. When the filter is active, a clear-filter banner sits above the table so the partial result set is obvious.

2. Year column removed from the Series flat-list. Year is still available in the sort dropdown (Year asc/desc, untouched) but the column is gone — most series' year data is sparse and the column was mostly em-dashes. Net effect: the Pins / actions columns get more breathing room.

3. Sort-arrow wrap fixed. The Year and Pins headers showed their up/down arrows wrapping onto a second line below the label. The arrow is a ::after pseudo-element; right-aligned narrow columns plus no white-space rule let it wrap. Now: white-space: nowrap on every sortable th — arrow stays inline with the label regardless of column width.

4. Version badge shrunk. The v0.15.98 pill next to the brand was reading as almost as prominent as the brand name itself. Tighter padding + smaller font + reduced background contrast: it now looks like a version tag, not a button.

5. Top-nav search collapses to an icon. v0.15.96 made the input shrink from 14rem to 5rem when unfocused; that still ate 5rem of nav real estate on every page. Now the pill collapses to just the search icon (36×36 circle, matching the theme toggle) when unfocused, and smoothly expands to a 16rem search pill on focus. The icon doesn't move during the transition; the input slides in to its right.

6. Main content area widened. main max-width bumped from 1100px to 1280px (+16% horizontal real estate). The dashboard, list views, and scan grids all benefit; line-length on prose pages stays readable because most prose blocks have their own narrower max-width caps inside the 1280 container.

7. Validation card layout cleaned up. Cards without images on /validation appeared as text-only rows because the empty-thumb placeholder was a muted ? character with no border — invisible against the dark card background. Now: the empty thumb shows a Lucide pin icon at 50% opacity, every card has a min-height of 104px (88px thumb + padding) so rows align cleanly, and the grid uses align-items: stretch so cards in the same row match the tallest neighbor's height.

v0.15.98

2026-05-21

Crash detection + diagnostics. Drew flagged the server appears to be crashing his machine occasionally and wanted somewhere to look for the cause. v0.14.67 already logs to logs/dispindable.log but it truncated in-place at 5MB, kept no backups, and hard kills (OOM, SIGKILL) left no trace at all. This version makes the log infrastructure crash-aware:

1. Three log backups. RotatingFileHandler switched from backupCount=0 (truncate-in-place) to backupCount=3. When the active log hits 5MB it rotates to .1 → .2 → .3 instead of being wiped. 4× the disk ceiling (20MB total) but pre-crash context survives. Original v0.14.67 design was wrong for crash diagnosis.

2. Structured startup banner. Every uvicorn boot writes a grep-friendly === STARTUP === marker with version, PID, Python version, and — critically — the LAST line of the PREVIOUS log file, tagged graceful or INCOMPLETE. Lets you scan the file and see exactly where each restart happened, and whether the previous one died cleanly.

3. Graceful-shutdown banner. FastAPI on_event("shutdown") writes === SHUTDOWN === when the app exits cleanly (Ctrl+C with grace, normal stop). Combined with the startup banner's prev-tail check, this lets the crash detector tell apart graceful restarts from hard ones.

4. Unhandled-exception hook. sys.excepthook AND threading.excepthook route any uncaught exception through the logger with a full traceback BEFORE the process exits. Doesn't help with OOM / SIGKILL (kernel never gives Python a chance) but catches everything else — thread crashes, SSE generator errors, background task failures, etc.

5. Heartbeat file. Background thread writes logs/heartbeat.txt every 30 seconds with timestamp + RSS memory. Single line, overwritten each tick. If the server crashes hard, the file's mtime tells you when the last successful heartbeat fired — approximately when the process stopped working. Uses psutil if available, otherwise /proc on Linux, otherwise just the timestamp.

6. GET /healthz. Returns 200 JSON with uptime, version, PID, RSS memory, and the heartbeat file's age. Designed for external monitoring — point uptime-kuma, Pingdom, or a cron curl at it and get alerted when the server stops responding.

7. /admin/crashes page. Walks the rolling log files, pairs up STARTUP/SHUTDOWN markers, and classifies each restart as graceful or crash. Shows a table of probable crashes with the prior log tail (the “clue” about what went wrong) so Drew doesn't have to grep logs manually.

What to look at after the next crash:
1. Visit /admin/crashes to see the crash table.
2. Look at the “prior log tail” column for each entry — that's the last thing the server logged before dying.
3. logs/heartbeat.txt mtime = approximate time of death.
4. If dispindable.crash entries appear in the log, that's a Python-level exception you can actually debug. If the log just stops mid-line with no traceback, it was a hard kill (OOM / power / kernel) and the heartbeat file is your best clue.

v0.15.97

2026-05-21

Three fixes: section dedup on /validation, top cards restyled, real per-stage SSE streaming for the smart chain.

1. /validation section counts now sum to the nav badge. Drew flagged: nav badge says 37 unverified, but the page sections summed to 8 (no series match) + 33 (no series assigned) = 41. The two sections were overlapping by 4 pins that had BOTH no-catalog-id AND no-series. The badge uses DISTINCT counting; sections didn't. Fix: “No series assigned” section now explicitly excludes pins already in “No series match” (and in “needs validation”), so each pin appears in exactly the most-specific section that catches it. Counts now sum to the badge total.

2. /validation top cards restyled. Drew flagged the “Quick Add another pin” cards at the top of the page had two visible borders (full red outline + the v0.15.91 left-accent) and the metadata text was overflowing to the right because the vertical card body was too narrow. Rebuilt to match the layout of the lower secondary-section cards (Lumiere / Sunbathing): image LEFT, text RIGHT, single border with red left-accent. Larger thumb (88px) since these are higher-priority. Mobile breakpoint (<640px) shrinks to 72px thumb.

3. Real per-stage SSE streaming for the smart chain. Pre-v0.15.97 the /scan smart-chain button called the synchronous JSON endpoint and waited for the whole chain to finish before any feedback — which looked confusing in v0.15.95 where the JS no longer auto-advanced stages. New endpoint GET /scan/{id}/ai-identify-chain-stream returns text/event-stream and yields events as each stage completes:
· stage_start — sent when a new stage begins. UI moves the spinner to that row.
· stage_done — sent when a stage finishes with its real confidence, cost, duration. UI fills in that row's cells in-place.
· done — final summary with stopped_reason + final_suggestion. UI renders the summary line + “Open /new-pin” link.
· error — chain-level failure.
JS replaced fetch() with EventSource; cleanly closes the stream on done/error. Skipped stages render “skipped (earlier stage stopped chain)” the same way the JSON endpoint did. The synchronous JSON endpoint is preserved for any client that needs a single-shot response (tests, scripting). Cloudflare Tunnel buffering headers (X-Accel-Buffering: no, Cache-Control: no-cache) included so events deliver in real time through CF.

v0.15.96

2026-05-21

Series page restructure + sortable columns + folder thumbnails + search-bar shrink.

1. /series flat-list view. Drew wanted a list layout like /pins rather than series grouped under their parent folder in card grids. New default view (?view=list) renders all series as a sortable table: thumbnail, name, parent collection (as a colored pill linking to the folder page), year, owned/total pins, delete. The original card-grouped layout is still available via ?view=grouped + the toggle in the header. List view scales better past ~30 series; grouped view is preserved for users with curated folder layouts.

2. /collections folder thumbnails. Each folder row now shows a 48px thumbnail of the first owned pin's image from any of its series. The row felt empty before with just dot+name+count. Falls back to a folder-icon placeholder if the folder is empty or has no pin images yet.

3. Sortable columns sitewide. New .dp-sortable-table CSS class + shared JS in base.html. Header cells with data-sort-key are clickable; cells can carry data-sort-value to override the visible text for numeric sorting (e.g. “37 / 41” cell sorts by 41). Sort direction indicators rendered via CSS pseudo-elements (▲/▼). Sort state persists per-page in localStorage so reloads keep the user's choice. Applied to /pins, /series (list view).

4. Topnav search bar shrinks to 5rem until focused. The 11rem permanent width was eating horizontal space on 1024-1500px screens and pushing nav items off the right edge. Now: 5rem default (room for ~3-4 chars + the magnifying glass icon), expands to 14rem on focus or when the input has content. Smooth CSS transition. Parent pill also gets a subtle background + border treatment on focus so the active state reads clearly.

v0.15.95

2026-05-21

Smart-chain UI no longer lies about which stage is running. Drew screenshotted the chain UI mid-call showing “Escalating to Stage 3 (deep)” with timers ticking on all three stages. The server log showed the chain actually stopped at Stage 1 after Gemini Flash returned a 100/100 match — the UI was lying.

Root cause: the skeleton JS auto-advanced through stages based on cumulative est_s (4s + 6s + 15s) rather than actual server progress. When Gemini Flash took longer than 4 seconds (because of grounded search fanout), the UI would “move on” to Stage 2 and then Stage 3 even though the real call was still on Stage 1.

Fix: stop auto-advancing. The Stage 1 spinner ticks for as long as it takes. If the call goes over the 4s estimate, the progress line switches to “Stage 1 is taking longer than usual… Gemini may be doing extra grounded searches” rather than fabricating a stage transition. Once the real response lands, we replace the skeleton wholesale with the real stages_run audit trail from the server — including only the stages that actually executed.

Real streaming progress (per-stage Server-Sent Events from the route) would be a better long-term fix but requires restructuring the synchronous endpoint as an async generator. Journaled to backlog.

v0.15.94

2026-05-21

Three fixes from Drew's v0.15.93 testing.

1. Smart chain: Stage 3 no longer fires SerpAPI Lens a second time. Lens already ran in Stage 1, and Lens is deterministic for the same image — running it again in Stage 3 just burned quota with zero new info. New deep stage is opus + openai-deep only. Stage 1 still has Lens. If Drew wants Lens results, they're preserved on the scan from the Stage 1 run.

2. Gemini Flash grounded search budget capped at 3. Drew flagged Stage 1 took 65s on one scan. Log showed search_queries=21 — Gemini fired 21 grounded Google searches for one pin. The old prompt encouraged exhaustive searching (“Search PinPics. Search Pin&Pop. Check Disney Pins Blog. Check eBay…”) which Gemini interpreted literally.
New prompt explicitly caps the budget at 3 searches max and instructs combining databases into single queries. Should bring Stage 1 latency down from 60-90s to 10-20s in the worst case.
Note: this is prompt-only because Google's SDK doesn't expose a hard search-count cap on the google_search tool. The prompt is the only lever.

3. Web layout centered on wide screens. Pre-v0.15.94 main had max-width: 1100px but no margin: auto, so on 1920px+ monitors content collapsed against the left side with a huge void on the right. Added margin: 0 auto to both main and the .dpb-topnav inner row. Mobile / narrow screens unchanged (max-width is the ceiling not the floor). Content stays within 1100px of readable line length while the page itself can stretch full-width — standard web pattern.

v0.15.93

2026-05-21

PinPics-ID hygiene: validation sweep + AI hallucination prevention. Background: we confirmed in v0.15.92 that playwright-stealth defeats PinPics's CF challenge and the full fetch pipeline works against real pin URLs. But Drew's DB had accumulated bogus pinpics_id values (every test ID 404'd on the live site) — non-grounded LLM providers had been hallucinating IDs freely in their suggestions. Two fixes:

1. Validation sweep. New pinpics_id_verified_at column on Pin (DateTime, nullable). Null = unverified, set = confirmed real against the live site. New functions in catalog_sites.py:
· verify_pinpics_id(pid) — single-pin live check via the existing fetch_pinpics path.
· verify_unverified_pinpics_batch(s, max_checks=50, delay_s=10.0) — rate-limited sweep. On 404: nulls the bogus pinpics_id, stamps verified_at (so we don't re-check the same dead row), and writes an ActivityEntry audit row. On success: stamps verified_at. On ambiguous error (network / CF / parse): leaves the row alone for a retry.
· pinpics_verification_stats(s) — counts for the dashboard.
New endpoint POST /settings/pinpics-verify-sweep (form field batch_size capped at 200) fires a sweep in the background via BackgroundTasks — response returns immediately with a flash message. New settings card surfaces total / verified / unverified counts with a green “verified” block and a yellow-on-positive “unverified” block, plus the batch-trigger button.
Scheduler integration: new _check_pinpics_verification() runs once per 24h, drains 30 IDs per cycle (~5 minutes of wall time), no-ops when the backlog is empty. So even if Drew never clicks the button the queue slowly drains on its own.

2. Stop new hallucinations. New ai_identify_common.strip_ungrounded_external_ids(suggestion, *, grounded) sanitizer. When grounded=False, it pops pinpics_id and ptdb_id from the parsed dict and stashes the original values under _dropped_ungrounded so we have an audit trail of which providers were hallucinating which IDs.
Wired into all three AI providers post-parse, pre-cache-save:
· Claude — grounded = tool_uses > 0 (only treat as grounded when the model actually executed a web_search tool call).
· OpenAI — grounded = n_searches > 0 (same logic via Responses API search-tool count).
· Gemini — grounded = n_queries > 0 (google_search grounding metadata).
The cache-key already includes with_search, so cached entries from before this version won't get retroactively re-stripped — but they'll naturally expire over time, and the verification sweep handles the DB cleanup independently.

Tried system-prompt instructions first in a draft — models obeyed for the first few turns then resumed hallucinating. Their training data is full of (image → pinpics URL) associations and prompts can't consistently override that gradient. The post-hoc strip is bulletproof.

v0.15.92

2026-05-21

Push-notification quality-of-life fixes. Drew reported two issues after subscribing for the first time:
· The notification icon on Android Chrome showed as a blank white square.
· Chrome on Windows didn't display the test notification at all, while Firefox on the same machine did.

1. PNG icons replace the SVG. Web Push spec is PNG/JPEG only — Android Chrome silently fails on SVG and renders a blank icon (no DevTools warning, no console error; you just see white). Generated three PNGs from the existing favicon.svg via cairosvg:
· icon-192.png — main notification icon (shows next to the body text)
· icon-512.png — high-res variant referenced by manifest.webmanifest for PWA install
· badge-72.png — monochrome silhouette that Android tints to the device theme color for the status-bar badge
The service worker push handler + the /new-pin in-page Notification API call both now point at PNG. SW CACHE_NAME bumped to dispindable-shell-v0.15.92 so installed PWAs force-refresh the cached shell on next launch. manifest.webmanifest updated with both PNG and SVG entries (PNG marked “any maskable” so Android can crop it to the system icon shape; SVG kept for browsers that prefer vector for the address-bar icon).

2. Per-device test result + Chrome troubleshooting guide. The /push/test endpoint now returns a per_device array with one entry per subscription showing: device label (parsed from UA, e.g. “Chrome on Windows”), push service the browser registered with (FCM / Mozilla / Apple / WNS), and delivery status. The /settings test button renders this as a per-device checklist, so you can tell exactly which subscriptions the server BELIEVES it delivered to. New _friendly_device() + _identify_push_service() helpers in push_notifications.py.

The Chrome-vs-Firefox mystery: if the server reports ok: true for a Chrome subscription but no notification shows on screen, the issue is downstream of the server — either Windows Focus Assist eating the notification (the #1 cause), Chrome's site-specific permission still on “Default” not “Allow,” or chrome://flags ‘native notifications’ delegating to a Windows action center that's muted. New “Notifications not showing up?” collapsible drawer on /settings walks through these checks. Firefox uses its OWN notification system rather than Windows', which is why it works when Chrome doesn't on the same machine.

v0.15.91

2026-05-20

Nine UI/UX fixes + Gemini token-logging audit + smart-chain restructure. Mostly response to Drew's feedback after using v0.15.90 in anger.

1. Activity log searchable. /activity?q=<substring> filters across entity_name + event_type + detail. Case-insensitive. Find “everything I did with the Stitch pins” or “all delete events.” Empty-state message differentiates between “no events yet” and “nothing matched.”

2. Mobile orange follow-up dot. The “More” tab on the mobile tab-bar already showed yellow (soft-validation) and red (validation) dots; v0.15.85 added an orange follow-up dot to the desktop “…” dropdown but mobile was overlooked. Now mirrored: new .dpb-tabbar-dot-warn class at right:34px so reading order is orange→yellow→red (eventually→soon→now).

3. Gemini token-logging audit. Drew flagged the actual Gemini cost ($1.37 on Anthropic's dashboard) was way below the in-app estimate. Audit found three log_usage gaps in ai_identify_gemini.py:
· is_available()-False early return had no log entry — not-configured errors went uncounted entirely.
· no readable image paths early return also silent.
· The exception path called log_usage with input_tokens=0, output_tokens=0 even when the API had returned and consumed tokens before the parse failed. Now does best-effort token capture from the response local via locals().get() before logging the failure.
This won't reconcile past costs (the log is append-only) but future Gemini calls will be accounted correctly. The actual API-dashboard vs in-app gap is the missed tokens above; check after a day of usage to confirm.

4. Smart-chain restructured. v0.15.90 Stage 1 was Gemini Flash + Claude Haiku — but Claude Haiku's 16s tail latency bloated Stage 1 close to Stage 2 levels with marginal recall gain over Flash alone. New ladder:
· Stage 1 (fast): Gemini Flash + SerpAPI Lens — purely visual, ~$0.003, ~4s.
· Stage 2 (balanced): Claude Sonnet + GPT-5 — text-savvy mid-tier, ~$0.02, ~6s.
· Stage 3 (deep): Opus + GPT-5.5 + Lens — full escalation, ~$0.50, ~15s.
Rationale: Flash + Lens cover “what IS this pin” (Flash visual recall) plus “where else is this image” (Lens reverse search) for ~80% of common pins. If both miss or disagree, escalate to text-savvy models that can read back-stamps + edition numbers.

5. Smart-chain progressive loading UI. The endpoint is synchronous (blocks until the whole chain finishes; can take 4-90s), so a blank “Posting…” label looked broken. Now renders a skeleton table with all three stages immediately, a live timer on the current stage, and an optimistic stage-advance based on each stage's est_duration_s. When the real response lands, the skeleton is replaced wholesale with the actual data. Stages that didn't actually run (because Stage 1 passed or hit floor) render as “skipped” rows.

6. /validation quick-add cards redesigned. Three changes:
· Card outline color changed from amber #d4a017 to brick-red --dp-brick to match the notification dot color on the top nav. Two yellows for two urgencies was confusing; red signals “needs attention now.”
· H1 now has a red count badge (matching the secondary-section headings) so the queue depth is visible at a glance.
· Card meta replaces the not-actionable acquired_at date with a click-through link to the pin's series. Now you can jump from the validation queue to the series detail to find sibling un-validated pins from the same set.

7. /series count clarified. Drew flagged the “41 pins on list, 37 on detail” mystery. Root cause: the list card showed len(c.pins) (all, including wishlist) while the detail page defaults to owned-only views (37 owned of 41 total). Now the list card shows owned / total format (“37 / 41 pins”) when they differ, collapses to a single number when all are owned. Tooltip explains the gap: “37 owned of 41 total (4 wishlist or missing).”

8. Search-match indicator no longer crashes into the input. On /series and /collections, the “N matches for ‘X’” line was rendered inside the form, in the same row as the search input. The browser's autofill suggestion box rendered on top of it, producing a strikethrough effect (visible in Drew's screenshot). Moved the match indicator to its own paragraph below the form so autofill no longer overlaps.

9. /pins table: Year column → Collection. Year was less actionable than the parent Collection (folder) — Year is usually visible inside the pin name already, while the Collection link is a real navigation shortcut to sibling series. Existing year data unchanged; just not surfaced in the table column anymore. Reachable via the pin detail page.

Backlog journaled (no code this version): series page overhaul, /scan UI cleanup, “run design through Claude design,” PTDB DB improvements, investigate Google Lens auto-crop tech, more-intense indexing / scraping, web layout centering + wider-screen optimization.

v0.15.90

2026-05-20

Scan-chain wired into the UI. v0.15.89 shipped the algorithm in scan_chain.py with tests but no route. This version connects it end-to-end.

New endpoint: POST /scan/{id}/ai-identify-chain. Runs the three-stage chain in real time:
· Stage 1 fans out Gemini Flash + Claude Haiku in parallel (ThreadPoolExecutor). Saves each suggestion to the scan row using the same field names /ai-identify-multi uses, so /new-pin renders without any template change.
· Computes the chain-level confidence via scan_chain.chain_confidence_from_suggestions() — a lightweight signal based on cross-provider vote agreement on the catalog ID + confidence-note text cues. The full multi-signal compute_confidence() needs embeddings + catalog data we don't have at chain time, so we use this lighter version for the escalate-or-stop decision.
· If composite ≥ 75/100 → stop, return the result. If < 30/100 → stop early (pin is unidentifiable, escalation won't help). Otherwise → escalate to Stage 2 (Sonnet + Gemini Pro + Lens). If still below 75, escalate to Stage 3 (Opus + GPT-5.5 + Lens).
· Returns JSON with the full per-stage audit trail: confidence at each step, cost, duration, verdict (passed / below-floor / escalate). Lets Drew see exactly why the chain stopped where it did.

SerpAPI integrated as a chain provider. Lens results get translated into the same suggestion shape as the LLMs (pinandpop_id / pinpics_id / name) and participate in the cross-provider vote on equal footing. Useful as a diversity check — SerpAPI hits the same catalog sites the LLMs would have referenced anyway, but via Google Lens's reverse-image-search index rather than LLM training-data recall.

UI on /scan. New “🧮 Smart chain” collapsible card below the existing provider checkbox form. Single “⚡ Run smart chain” button. When clicked, shows a live results table with per-stage confidence / cost / time, then a final summary line + an “Open /new-pin→” link to view the rendered scan with the chain's chosen IDs already prefilled. Catalog data is prefetched via BackgroundTasks (same as the multi-identify flow) so /new-pin renders quickly.

Per-call threshold overrides. The endpoint accepts pass_threshold and retry_min form fields if you want to tune for specific pins (e.g. pass_threshold=85 to demand higher confidence before stopping). Defaults are 75/30, the values that fell out of the algorithm design discussion. Tune from there once you have data on how often Stage 1 calls produce 75-vs-actually-wrong results.

v0.15.89

2026-05-20

Three things shipped: cost-table fixes + SerpAPI quota tracking, playwright-stealth integration, and a confidence-scored scan chain (algorithm only, route wiring deferred).

1. Cost-estimate accuracy. Verified the price table against current published rates from each vendor:
· Claude (Haiku/Sonnet/Opus 4.5/4.6/4.7) at $1/$5, $3/$15, $5/$25 — correct
· GPT-5 / GPT-5 Mini at $1.25/$10 and $0.25/$2 — correct
· GPT-5.5 was wrong: had it at $2.50/$15. OpenAI doubled the price at the Apr 23 2026 launch to $5.00/$30. Updated.
· Gemini 3.1 Pro / 3 Flash at $2/$12 and $0.50/$3 — correct
· Web search grounding fees ($0.010 Claude, $0.025 OpenAI, $0.014 Gemini) — correct
Drew, your Gemini estimate-vs-actual divergence ($1.37 actual on the Anthropic dashboard) is most likely token counts not getting captured for every Gemini call, not the prices being wrong. Worth auditing ai_identify_gemini.py to confirm every code path through it logs to .usage_log.jsonl with non-zero input/output tokens. Punted to a follow-up.

2. SerpAPI quota tracking. scan_serpapi_lens now logs every call to the usage log alongside LLM calls. New summarize_usage_with_serpapi() tallies the month-to-date count and computes quota-aware cost: $0 for the first 250 searches (free tier), $0.01/each beyond. /settings Usage card now shows a quota indicator with progress bar — turns yellow at 60% and red at 90%. SERPAPI_FREE_QUOTA = 250 in ai_response_cache.py if you upgrade to a paid plan with a different limit.

3. Playwright-stealth integration. headless_scrape.py now detects playwright_stealth at module load and applies its full evasion suite (~15 properties) when available, in place of the hand-rolled 3-property init script from v0.14.94. Stealth covers chrome.app, navigator.permissions, webgl vendor, sec-ch-ua, and other fingerprintable surfaces that Cloudflare's mid-tier challenge probes. Graceful fallback to the manual script when the library isn't installed. Realistic improvement on PinPics: ~30-60% hit rate — NOT a guaranteed pass. Cloudflare also fingerprints TLS + mouse behavior; defeating those needs paid residential proxies. Added playwright-stealth>=2.0 to requirements.txt.

4. Confidence-scored scan chain v1 (scan_chain.py). New standalone module implementing the serial-fallback algorithm:
· Stage 1 (Fast): Gemini Flash + Claude Haiku, ~$0.002, ~3s. Used for 80% of pins.
· Stage 2 (Balanced): Sonnet + Gemini Pro + Lens, ~$0.02, ~5s. When Stage 1 confidence is borderline.
· Stage 3 (Deep): Opus + GPT-5.5 + Lens, ~$0.50, ~15s. When Stage 2 still can't reach the bar.
Thresholds: pass at 75/100 composite confidence (stop escalating, return); floor at 30/100 (the pin is genuinely not identifiable from this image — further escalation won't help, stop early). Both tunable per-call.
The module uses dependency-injection for the actual AI provider calls (ProviderRunner callback) so the algorithm is unit-testable without firing real providers. Route wiring is deferred to a follow-up version — this ships as a reviewable design slice. Tests cover the chain logic against mocked runners.

v0.15.88

2026-05-20

Bugfix: concurrent fake-corpus scrapes. Drew's uvicorn logs showed two distinct errors firing during a scrape:
· WinError 32: The process cannot access the file because it is being used by another process on the state file's tmp+rename atomic write.
· IntegrityError: UNIQUE constraint failed: fake_reference_posts.source, fake_reference_posts.source_id on the per-post insert for post id 1tj03r1.
Both root-caused to the same problem: two scrapes running concurrently (likely manual + scheduler, or a double-click on the Scrape button). The dedup check IS in scrape_one_page (lines 661-671) but it's a TOCTOU race — both scrapes could pass the “does this post_id exist?” query before either committed, then race to insert.

Three-layer fix:
· 1. Module-level _STATE_LOCK in fake_corpus.py. Wraps the read-modify-write cycle of the state file in a threading.Lock. Eliminates the tmp-file collision and the lost-update window where two writers trampled each other's state dicts. Applies to both _write_state (live-progress updates) and set_backfill_state (per-page cursor).
· 2. IntegrityError handling in scrape_one_page. Each commit (both the every-10 intermediate and the tail) is now wrapped in try/except. On dedup-race, rollback + decrement added count + bump skipped count + continue. Defense-in-depth so a race we didn't anticipate doesn't kill the whole page.
· 3. Already-running guard in _run_fake_corpus_scrape_bg. Before starting work, reads the state file's running dict. If a non-stale scrape is already in flight, log a rejection and return immediately without clobbering the other run's state. Uses the same 90s-staleness threshold as the UI poller.

The combined effect: it's now impossible to start a second scrape while one is running (3 catches it early), and if some other path somehow triggers concurrent inserts, 2 handles the race gracefully and 1 keeps state-file writes consistent.

v0.15.87

2026-05-20

Release-feed push notifications + vulture sweep batch 3. Two items shipped together since both were small.

1. Push notifications for matching new releases. Wires the v0.15.86 push plumbing into the existing release-feed sync flow. New flow:
· Scheduler runs the release-feed sync (every ~4h when stale, unchanged).
· After each successful sync, calls the new release_feed.notify_new_releases(s) which looks for unnotified items, matches them against the user's watch filters, and broadcasts a single batched push to all subscribed devices.
· New release_feed_items.notified_at column tracks dedup. Once notified, an item is stamped and won't re-notify even on subsequent syncs.
· Recency window: only items published in the last 7 days trigger notifications. On first install or first push-subscribe, the historical backfill (~30 items from disneypinsblog.com's feed) won't spam you — old items get silently marked notified.
· Batching: when multiple items match in a single sync, you get one notification (“3 new pin releases: Item A · Item B · +1 more”) rather than three separate ones. Phones throttle aggressive senders.

Watch filter UI on /settings — new card below the Push card. Two fields:
· A checkbox to enable / disable notifications entirely (default off).
· A multi-line textarea with one keyword pattern per line. Substring match against title + summary, case-insensitive. Lines starting with # are comments. Use * on its own line as a catch-all for every new release.
Saved via new POST /settings/release-notify route. Empty filter list = no notifications even when the toggle is on (so you can't accidentally enable notifications without picking what to watch for).

2. Vulture sweep batch 3. Fewer findings this round — batches 1+2 already cleared the big stuff. Two items removed:
· ocr.extract_text() — dropped the vestigial min_confidence parameter that was kept “for backward compatibility.” Grep confirmed no callers ever passed it. The real filter logic lives in CONF_LONG / CONF_SHORT.
· scrape_dpb._add() — dropped the path_label parameter that the function body never read. Callers passed positional labels (“img” / “picture” / “noscript”) but the value was discarded inside. Likely a leftover from when the rejection-example log tracked which extraction path tried each URL.

v0.15.86

2026-05-20

Scan presets + Web Push notifications + browser Notification on /new-pin ready. Three things shipped together since they were all about “the user shouldn't need to babysit the tab”.

1. Scan-type presets on /scan. The 7 provider checkboxes + cheap-mode toggle is consolidated behind three preset buttons:
· ⚡ Fast & cheap (Gemini Flash + Claude Haiku + Lens, cheap mode on, ~$0.002, ~3s)
· ⚖️ Balanced (Claude Sonnet + Gemini Pro + Lens, ~$0.02, ~5s) — the new default
· 🚀 Throw everything (all 7 providers, deep mode, ~$0.50, ~15s)
The original checkbox grid is still reachable via the Advanced toggle so per-provider control isn't lost. Selected preset persists in a cookie. Clicking a preset both sets the checkboxes AND auto-fires the form submit. Manual checkbox tweaks switch the active preset label to “Custom”.

2. Browser Notification on /new-pin ready. Sits alongside the existing v0.15.69 audio ping (660Hz sine pulse). Now ALSO fires a system / tab-bar Notification when the result page finishes loading — but only when the tab is backgrounded (no point notifying a tab the user is already looking at). Title includes the pin name when the AI consensus already returned one. Auto-closes after 5s so the system tray stays clean. Permission has to be granted via the Push subscribe flow on /settings; we don't auto-prompt at page load (browsers throttle that pattern).

3. Web Push notifications. Full plumbing for browser push notifications, even when the tab is closed:
· New push_notifications.py module wraps pywebpush for the send-push protocol (VAPID-signed, payload-encrypted).
· New PushSubscription model + table stores each browser's endpoint + ECDH keys + user-agent at subscribe time. Dedupes on endpoint so re-subscribing the same browser updates the keys rather than creating duplicate rows.
· New routes: GET /push/config (returns VAPID public key), POST /push/subscribe (client submits its serialized PushSubscription), POST /push/test (sends a test notification to every active subscription).
· /sw.js service worker now handles push and notificationclick events — shows the notification, focuses an existing tab on click or opens a new one.
· New Push card on /settings with Subscribe / Send test / Unsubscribe buttons. Card hides automatically when the server isn't configured (VAPID_PUBLIC_KEY / VAPID_PRIVATE_KEY / VAPID_SUBJECT env vars unset).
· iOS Safari (16.4+) requires the PWA be installed to home screen before pushManager.subscribe() works. Chrome/Edge/Firefox work in any tab.

Setup (Drew): generate the VAPID key pair once, set the three env vars, restart. The setup snippet is in push_notifications.py's module docstring.

Deferred: wiring push into the release_feed scheduler so new pin drops trigger automatic notifications. The plumbing is in place; the integration is a small follow-up (one call to push_notifications.broadcast(s, title=…) from inside the scheduler when new feed items arrive that match watch filters). Watch-filter UI is also TODO — for now, broadcasts go to all subscribed devices.

v0.15.85

2026-05-20

Four small UI fixes flagged from real usage.

1. Orange follow-up dot on the “…” dropdown. The dropdown trigger in the top nav already shows a yellow dot for soft-validation (series triage) and a red dot for hard validation (needs_validation). Now also shows an orange dot when there are follow-up-flagged pins. Position-wise it sits to the LEFT of the yellow dot when both are present. Reading left-to-right: orange (look at this eventually) → yellow (look at this soon) → red (look at this now). New class .dpb-nav-dot-warn in base.html's style block.

2. Moved “Flagged for follow-up” up on /validation. Was the LAST section on the page, after “No series match” and “No series assigned”. Promoted to be the FIRST secondary section (right after the main needs_validation grid that's linked from Quick Add). User-flagged items are more actionable than the catalog-mismatch / missing-series sections — the user already decided these need another look, so they should be the next thing you see.

3. Search box magnifier no longer crashes into placeholder text. The global input[type="search"] rule in dispindable-tokens.css had padding-left: 2rem !important which was overriding the inline 2.8rem /pins template was already setting. Bumped the global to 2.8rem !important — fixes once globally rather than chasing per-template padding-left overrides. Affects every search input in the app: /pins, /series, /collections, /catalog/browse, the top-nav search bar.

4. New “✓ Owned” filter chip on /pins. Sits immediately after the “All” chip on the toolbar. Active state is green (#226d3e) to signal “in your collection”. Clicking narrows to is_owned=True pins. New owned= query param on the /pins route also accepts no (typically wishlist / reference-only pins) and wishlist (orthogonal to owned).
Side fix: the “All” chip's active-state check was previously not filter_validation — so it would light up even when followup/missing/owned filters were active. Now correctly checks not (filter_validation or filter_followup or filter_missing or filter_owned).

v0.15.84

2026-05-20

Color tokens: WCAG-compliant muted text + semantic accent palette. First-pass on the “Better colors + scannability” backlog item.

Fixed WCAG AA failures on muted text. Audited contrast ratios on --dp-mute (the Crafted theme's muted color, fed through to Pico's --pico-muted-color override). Results:
· Light mode was #7a8497 = 3.55:1 against the cream paper bg — FAILED WCAG AA (needs 4.5:1). Bumped to #5e6878 = 5.31:1. Affects every “subtitle/small/caption” text in the app since the token feeds Pico's muted-color variable.
· Dark mode was #8392A6 = 5.73:1, barely passing AA on phone displays with reduced brightness. Bumped to #A0AFC3 = 8.14:1, comfortably above AA.
· Borders were nearly invisible in both themes (1.29-1.45:1). Lifted --dp-border to #D9CFB3 (light) and #3A4D6E (dark) so panel edges + table rows read as definite boundaries. Borders don't need WCAG contrast since they're decoration, but invisible borders are worse than visible ones.

Added semantic accent palette. Previously every panel set its own arbitrary left-border color inline: teal for visual-auth, purple for AI confidence + history, red for eBay, amber for follow-up + warnings, blue for catalog / cross-reference, green for success. Mapping was consistent in practice but undocumented — any new panel had to be eyeballed. New tokens make the meaning explicit:
· --dp-accent-info (blue) + -bg — catalog, cross-reference, links
· --dp-accent-success (green) + -bg — completed, verified, owned
· --dp-accent-warn (amber) + -bg — follow-up flag, needs review
· --dp-accent-danger (red) + -bg — errors, fake, delete
· --dp-accent-ai (purple) + -bg — AI providers, embeddings
· --dp-accent-auth (deeper purple) + -bg — authenticity checks
· --dp-accent-ebay (eBay red) + -bg — eBay-branded sections
Each has both a solid variant (for borders, text on tinted bg, badges) and a translucent -bg variant (for card-background tint). Dark-mode variants are brighter to read on the navy paper bg.

Templates not refactored this version. Inline hex colors in existing templates still work; they render the same as before since the same colors are now available as tokens. Refactoring every border-left: 3px solid #4285F4 to border-left: 3px solid var(--dp-accent-info) is a separate adoption pass (~15-20 templates affected). Worth doing if you ever want to retheme; not worth doing just for the rename. New panels going forward should use the tokens.

Classic theme unchanged. The ui_variant=classic path uses Pico defaults directly and doesn't load dispindable-tokens.css. If you switch to classic, the muted-text contrast issue comes back (Pico v2's default mute is roughly #7c8a9b). Crafted is the recommended path; this fix only helps there.

v0.15.83

2026-05-20

Five backlog items shipped: bulk wishlist, missing-metadata scanner, negative AI caching, eBay UPC API, vulture batch 2. Drew asked for the “next 6” from the backlog; 5 landed in this version and the 6th (mobile testing pass) deferred since it needs an actual phone-in-hand session rather than speculative CSS changes.

1. Bulk wishlist add. New /wishlist/bulk-add route accepts a textarea of Pin&Pop URLs or IDs (one per line) and imports them in one shot. Per-line results table shows added / skipped (duplicate) / failed status with reasons. Handles dedup both within the batch and against existing pins. Wired into both crafted + plain /wishlist header as a “+ Bulk add” button next to the single-URL one. Fetches sequentially; if cold-fetch latency becomes a problem, paste in chunks of 10-20.

2. Missing-metadata scanner. New missing= query param on /pins supports six filter values: year, ids (all three catalog IDs blank), back / front (no PinImage of that view), series (in Inbox/Unsorted/Unknown), any (year OR all-ids OR back). A new “🔍 Missing” chip on the /pins toolbar opens a dropdown with the six filter options. Bulk-cleaning a backlog of quick scans now has a dedicated entry point.

3. Smarter AI caching (negative caching slice). ai_response_cache.py now caches AI responses that succeeded but couldn't identify the pin (no name, no catalog IDs). Previously only positive results were cached, so re-running an unidentifiable pin re-spent the full provider budget on every retry. Negative entries use a separate, shorter TTL (default 1 day vs the positive 14, override via AI_CACHE_NO_RESULT_TTL_DAYS env var) so improved prompts / model versions get a chance to retry. Detection is conservative: only flagged as no-result when name AND all 3 catalog IDs are blank — a partial-info response (name but no IDs) still gets the full TTL since it's actionable. The broader caching wishlist (prompt-version-bump-survival, intermediate result reuse) is its own backlog item still.

4. eBay UPC API endpoint. New /api/ebay-upc?q={upc} returns JSON {ok, query, total, listings}. No UI yet — this is a measurement tool. Per the v0.15.82 backlog note: Drew suggested showing inline eBay matches when a barcode scan misses local-catalog. Realistic hit-rate depends on what fraction of Disney pin UPCs are actually indexed by eBay sellers. Use the endpoint to scan ~20 real UPCs (curl + jq) and decide whether the inline-eBay-on-barcode feature is worth building. If total > 0 on 12+/20, ship it; if 3/20, the current web-search buttons are the honest answer.

5. Vulture sweep batch 2. Removed six more confirmed-dead items the first batch missed:
· _media_type_for(path) in ai_identify.py — helper never called; _prepare_image_bytes always produces JPEG.
· set_typed(key, value) in app_settings.py — type-coercing setter with no callers; the settings page uses set_int/set_bool/set_str directly.
· _pinpics_fullsize_url(url) in catalog_sites.py — documented v0.14.8 as a no-op pass-through; nothing called it.
· score_label(score) in authenticity_check.py — templates render trust labels inline via Jinja conditionals, no Python callers.
· TRUST_COLORS + TRUST_LABELS constants in authenticity_check.py — never imported anywhere; templates inline the styling.
· search_by_name(query, *, limit) in catalog_search.py — /catalog/browse does its substring filter inline (with year + series narrow logic); /catalog/text-search uses CLIP semantic search instead.
Kept headless_scrape.shutdown() — tagged as a test-side hook even if no current test calls it. Net delete: ~80 lines across 5 modules.

6. Mobile testing pass — deferred. Speculative CSS changes without a real device to test against tend to make things worse on a different breakpoint than they fixed. Better as a focused session where Drew lists what's breaking on the phone and we fix targeted issues. Tagged as a next-up item rather than shipped this version.

v0.15.82

2026-05-20

Barcode scanner UI fixes + vulture sweep. Three barcode-modal bugs Drew flagged on mobile, plus a first batch of dead-code removal.

Barcode UI fixes:
· X close button now visible in both dark and light mode. Was a transparent button with a muted-color border that vanished against the dark modal card. Now has a 1.5px solid var(--pico-color) border + a translucent gray background (rgba(127,127,127,0.18)) + a thicker SVG stroke (2.5 → 3) at a larger size (14px → 18px).
· Flash on/off button readable in OFF state. Was rgba(0,0,0,0.4) (semi-transparent black over an already-dark card — blended in). Now rgba(127,127,127,0.22) muted gray with a solid 1.5px border, bumped font-weight + size. ON state still solid yellow (#f5c518) so the two states are visually distinct.
· Post-capture state is now obvious. After a successful scan, the <video> element hides and a green “✓ Captured” panel takes its place. Previously the video would freeze on the last frame — but on Android Chrome (Drew's browser) that last-frame retention is unreliable and often shows solid black, which looked like a bug. The new state is unambiguous.

Vulture sweep, batch 1. Ran vulture --min-confidence 60 across all non-test .py files. Most “unused function” hits were false positives — FastAPI routes decorated with @app.get/@app.post look unused to static analysis since the decorator's registration is runtime. Same for SQLAlchemy model attributes accessed via Jinja templates. After filtering those out, deleted five confirmed-dead items:
· StreamingResponse import in main.py — only mentioned in a docstring, never instantiated.
· OPUS_MODEL constant in ai_identify.py — declared as backwards-compat but never read anywhere. deep_model() is called directly at call sites.
· DEEP_MODEL constant in ai_identify_openai.py — same story.
· cosine_similarity() helper in embeddings_clip.py (introduced v0.15.78). The /catalog/text-search route uses vectorized np.dot(matrix, query_vec) directly — per-pair calls would be 100x slower.
· build_smart_query() in ebay_lookup.py (~87 lines). A stopword-aware query builder that pre-dated the v0.14.27 decision to use pin.name verbatim for eBay queries (which matches eBay seller naming conventions better than any smart tokenization).

Kept the ocr.py min_confidence parameter and scrape_dpb.py path_label variable that vulture flagged — the former is documented as a backwards-compat shim, the latter looks like a debugging tag that's intentional. All 35 templates are referenced by something.

Decision: leave the barcode flow mostly as-is. Drew asked whether barcode lookup could show inline matches from PinPics / Pin&Pop / eBay rather than just web-search buttons. The current “search the web” links are honest about what we can do: Disney pin packaging UPCs (like 400995468890) are NOT indexed by PinPics or Pin&Pop — those sites organize by pin name + year + series, never by retailer barcode. We literally cannot map a UPC to a Pin&Pop ID without a lookup table that doesn't exist publicly. eBay's UPC search MIGHT work for a fraction of pins but hit-rate would be low. Added a backlog item for the eBay-UPC experiment specifically — worth a future try, not worth blocking a UI fix on.

v0.15.81

2026-05-20

Cleanups + backlog audit. Housekeeping pass after the v0.15.79-v0.15.80 feature run.

Deleted the dead synchronous eBay block. v0.15.80 moved /new-pin's eBay calls to an async endpoint but left the old {% if ebay_data %} blocks (~280 lines) in scan_new_pin.html as dead code (always-False because the route stopped setting those variables). v0.15.81 deletes them entirely, along with the ebay_data + ebay_sold_data placeholder variables in the route. Only ebay_async_query remains. Caught one orphan </small></p></article>{% endif %} fragment from the cut that I fixed via a targeted str_replace; lesson logged as “run the jinja template compile() immediately after any big regex-based delete pass”.

Backlog audit. The /backlog page had drifted out of date — some “idea” items were actually shipped, and recent shipped features (character dict, franchise synonyms, CLIP text-search, cross-source cross-reference, PWA, pin detail tabs, /new-pin skeleton-hydrate, follow-up flag) were never on the list. Audit fixed both:
· Marked pintradingdb.com as alt scrapper as done (PTDB crawler shipped v0.15.35; Pin.ptdb_id first-class in v0.15.77).
· Marked Rarity score per pin as done (rarity.compute_rarity + rarity_for Jinja global wired in earlier versions).
· Promoted Mobile-responsive validation from OPEN_QUESTIONS into the BACKLOG proper as an idea-status item. OPEN_QUESTIONS now has just two questions left.
· Added 7 new BACKLOG entries with done status to capture: CLIP text-search (v0.15.78), character dict + franchise synonyms (v0.15.77 + v0.15.79), cross-source cross-reference (v0.15.79), PWA install (v0.15.79), pin detail tab split (v0.15.80), /new-pin eBay skeleton-hydrate (v0.15.80), follow-up flag (v0.15.73 + v0.15.76). These were tracked in session notes but not in the user-visible BACKLOG.

Net result: /backlog now reflects reality. The done entries serve as a quick “what shipped recently” index alongside the changelog.

Engineering hygiene backlog still open. The full vulture / ruff --select F401 sweep for unused modules / functions / templates / dead branches is still its own backlog item (Code cleanup: unused modules, scripts, tests, dead branches). The eBay-block delete in this version is one batch of that broader effort.

v0.15.80

2026-05-19

Pin detail tab split + /new-pin eBay skeleton-hydrate. The last two non-on-hold backlog items.

1. Pin detail page split into five tabs. pin_detail.html was a 2063-line monolith with everything stacked: photo gallery, edit form, AI history, authenticity check (Tier 2 + Copyseeker + fake corpus + visual auth), eBay listings, and pin history. Refactored into a tab nav at the top with five panels: Overview (edit form, catalog links, tags, AI history), Photos (image gallery, crop/rotate/replace), Authenticity (all four auth tools), Catalog (eBay market data), History (pin event timeline).

Progressive enhancement — all content still renders server-side; JS just toggles a hidden attribute on the panels. No backend changes, no new endpoints, no rendering cost. URL hash drives the active tab: #tab=photos opens Photos directly; clicking a tab updates the hash so the browser back-button works; hashchange listener responds to back/forward. Default tab is Overview when no hash present.

Header (name, summary, validation banner) and footer (AI Rescan, Delete pin buttons + JS) stay OUTSIDE the tabs since they apply across all panels. Refactor is reversible by removing the tab nav + JS — the underlying section order is unchanged.

2. /new-pin: eBay panels load asynchronously after render. The eBay active + sold fetches were the biggest visible blocker on /new-pin's initial render — 2-5s each for two external HTTP calls against a rate-limited service. Page now renders immediately with a skeleton placeholder where the eBay panels go; JS fetches the data via a new /scan/{scan_id}/ebay-async endpoint and replaces the placeholder with the rendered cards.

New endpoint: GET /scan/{scan_id}/ebay-async?query=... returns JSON {ok, query, active, sold}. Active + sold are independent: a flaky sold scrape doesn't blank the active panel and vice versa. Honors refresh_active=1 and refresh_sold=1 query params for force-refresh, replacing the old ?refresh_ebay=1 reload-the-whole-page pattern.

Template change: the old {% if ebay_data %} blocks in scan_new_pin.html are now dead code — ebay_data and ebay_sold_data are always None on /new-pin (the route skips the synchronous fetch). A new skeleton + JS hydration block above them handles all rendering. The old block is left in place under {% if false %} as a reference for the JS card layout; safe to delete in a future cleanup.

Refresh UX: “Refresh” links inside the rendered active + sold cards now re-fire the fetch with the appropriate refresh_*=1 param, all client-side. No page reload, no scroll-position loss. Spinning skeleton during refresh.

Scope honesty: only eBay is deferred this version. Other heavy /new-pin sections (AI provider fan-out result rendering, set-members fetching, visual-similarity duplicate detection) still render synchronously. The AI results are cached on Scan.serpapi_lens_json + scan_suggestion_json so they're fast on repeat loads; set-members and similarity are local-DB queries that don't need deferring. Full skeleton-hydrate of every section was the original ~3-4h scope; this shipped the highest-value slice (the external eBay calls).

Both features ship with regression tests covering tab DOM structure, hash-based activation, and the eBay endpoint's JSON shape.

v0.15.79

2026-05-19

PWA install + cross-source catalog cross-reference + franchise synonym tags. Three medium items from the backlog landed together since each was small.

1. Franchise synonym mapping (after v0.15.77 character dictionary). New FRANCHISES dict in character_dict.py maps each canonical character to its parent movie/franchise as additional tag(s). When “Stitch” matches in a pin description, the auto-tagger now also emits lilo-and-stitch; “Elsa” emits frozen; “Luke Skywalker” emits star-wars. Most characters have one franchise; a few cross-over (e.g. Huey/Dewey/Louie tag both Mickey-and-friends and DuckTales). New find_franchises(canonical_chars) helper wired into _extract_auto_tag_subjects right after the character pass. Tags are already in normalized form (lowercase, hyphen-separated) so they pass through _normalize_tags unchanged.

Why this matters: descriptions usually mention ONE character but browsing/filtering wants the franchise level (“show me all Toy Story pins”). Auto-tagging the franchise lets that filter actually work even when individual pins only name Woody or Buzz.

2. Cross-source catalog cross-reference. New helper _cross_reference_catalog_ids(s, ...) in main.py: given a partial set of catalog IDs (typically what the AI consensus returned), find the local catalog_pins mirror row by the provided ID, then search for OTHER rows in the mirror that match the seed by canonical (name + year-or-series_name). When a match is found, the matching row's IDs from other sources fill in the blanks.

Use case: AI consensus returned a PinPics ID only. The mirror has both that PinPics row and a Pin&Pop row with matching canonical name. Cross-reference surfaces the Pin&Pop ID as an inferred match. /new-pin shows the inferred fields with a blue tint + a “🔗 Cross-referenced” info banner above the IDs row explaining what was inferred and the confidence level (exact = name+year matched; fuzzy = name+series only).

Conservative by design: only fills BLANK slots (never overrides an AI consensus or user override), requires exact canonical-name match, and prefers exact year-match siblings over series-only ones. Name canonicalization (_normalize_for_crossref) strips punctuation + collapses whitespace + lowercases so “SDR - Maleficent - Villain Portraits” matches “Maleficent — Villain Portraits”.

3. PWA conversion. DisPindable can now be “installed” to a phone or desktop home screen and runs in standalone mode (no browser chrome). New static/manifest.webmanifest declares the app name + theme color (#1a1a1a) + start URL + three home-screen shortcuts (Scan, Pins, Validation). New static/sw.js service worker caches the shell (CSS, favicon, manifest) for instant cold-launch from the home screen; HTML pages and API responses are pass-through to the network so dynamic content stays fresh.

Routes /sw.js and /manifest.webmanifest in main.py serve the files from the static/ directory but at the root path — service workers are scope-locked to the path they're fetched from, so a SW at /static/sw.js would only control /static/*. Serving from /sw.js puts the scope at the origin root.

base.html gains <link rel="manifest">, theme-color + apple-mobile-web-app-* meta tags (iOS Safari still wants the pre-manifest variants), and a feature-detected navigator.serviceWorker.register call before </body>. Failures (mixed protocols, http LAN URLs without https) log silently so the current-domain LAN setup still works without surfacing errors to the user.

Install path: on Chrome / Edge mobile, the address bar gets an install icon after first page load. On iOS Safari, share menu → “Add to Home Screen”. Once installed, the icon launches into standalone mode — no URL bar, no browser tabs.

v0.15.78

2026-05-19

CLIP embeddings on catalog mirror + text→image search. The deferred-from-v0.15.77 feature. Adds a second embedding column on catalog_pins using openai/clip-vit-base-patch32, and a new text-search route that ranks catalog pins by semantic similarity to a natural-language query.

Why CLIP and not just more DINOv2. DINOv2 (the existing visual-search backbone) gives structural image-to-image similarity — great for “find the duplicate” lookups, but it doesn't understand text. CLIP puts images AND text into the same vector space, so a query like “maleficent dragon pin” can rank catalog images directly without needing a text index over scraped names. This works even when the pin's scraped name is sparse (“Disney pin 159597”) or mis-categorized.

New module embeddings_clip.py mirrors the existing embeddings.py pattern: lazy model load, _pick_device() for CUDA/MPS/CPU, L2-normalized float32 bytes for storage. Exposes embed_image_bytes() and embed_text() that produce vectors in the same shared 512-dim space. No new pip dependencies — transformers is already in the project, and CLIPModel + CLIPProcessor are part of the same package.

New column catalog_pins.embedding_clip (BLOB, LargeBinary in SQLAlchemy) with a LATE_ADDITIONS migration. Independent of the existing embedding column; both coexist and are populated by separate backfill flows.

New backfill route /catalog/backfill-clip-embeddings mirrors /catalog/backfill-embeddings: 200 rows per click, redirects back to /catalog/browse with a flash message. Re-click until the count reaches 0. The fresh-crawl flow now needs both backfills run separately (DINOv2 for visual-search, CLIP for text-search).

New search route /catalog/text-search?q=... is the main payoff. Workflow: encode query via CLIP text branch, dot-product against every embedding_clip blob in a single vectorized numpy operation, sort top-K via argpartition. At ~100k rows the full scan + sort is under 200ms on a modest machine — well within interactive-search latency. Premature to add a vector index (faiss / hnswlib) until the mirror grows past ~500k rows.

New template templates/catalog_text_search.html renders the ranked grid with score badges (green ≥0.28, yellow 0.20-0.28, gray below). Lists example queries when called with no q so the page is useful as a landing.

Discovery: /catalog/browse gains a blue callout above the filter bar linking to /catalog/text-search with example queries. Catalog-health section gains a “Backfill CLIP” button alongside the existing DINOv2 button, with its own count of rows missing the CLIP embedding.

What text-search is good at: visual concepts (“stained glass mickey”), multi-concept conjunctions (“halloween pumpkin minnie”), character + theme combinations. What it's NOT good at: exact catalog IDs or specific year filters — use /catalog/browse with the structured filters for those.

Trade-offs honestly: first /catalog/text-search request loads the CLIP model (~3-5s cold). Subsequent searches are fast. The model file is ~150MB downloaded by transformers on first use and cached in the HuggingFace cache directory. Catalog rows with no image (no hero_local_path) are silently excluded from text-search results — nothing to match against. Score interpretation: CLIP cosine scores of 0.28+ are strong matches, 0.20-0.28 are reasonable, below ~0.18 is usually noise.

v0.15.77

2026-05-19

Character dictionary, PTDB on Pin model, Reddit fake-watch in Lens, crawler docs, backlog cleanup.

1. Character dictionary for auto-tag. Largest single auto-tag recall improvement so far. New character_dict.py holds ~150 canonical Disney characters with known variants — Maleficent, Cruella de Vil / DeVil / Deville, Captain Hook / Hook, Shan Yu / Shan-Yu, Mickey Mouse, Tinker Bell / Tinkerbell, etc. The auto-tag extractor (_extract_auto_tag_subjects) now scans the full text (pin name + description) for any variant case-insensitively with word-boundary protection, and prepends matches before the existing Capitalized-noun heuristic candidates. This catches lowercase mentions (“the maleficent pin”), variant spellings, and short aliases inside longer descriptions that the old scoring-based approach missed. Adding new characters is a one-line edit to the dict; server restart picks them up.

2. PTDB pin ID promoted to primary identifier on Pin model. Before v0.15.77, PTDB IDs lived only on catalog_pins (the local mirror) and could be linked-to but not stored as a primary identifier on a saved Pin. Now Pin.ptdb_id exists with an index + LATE_ADDITIONS migration. Three downstream changes:
• New _PTDB_RE regex in ai_identify_common matches “PTDB 22107”, “Pin Trading DB #22107”, and full pintradingdb.com URLs (both modern /pin/N and legacy /pin.php?id=N forms).
• extract_catalog_ids + merge_catalog_ids + synthesize_lens_suggestion all extended for PTDB IDs with the same vote/veto machinery as PinPics + Pin&Pop. Gemini still vetoes; Lens still gets weight 2; voter lists track per-source attribution.
• New PTDB ID input field on /new-pin between Pin&Pop ID and SKU. new_pin_from_scan reads it as Form(None) and saves to the new column.

3. Reddit r/ismydisneypinfake in the Lens result panel. The SerpAPI Lens partition logic in renderResult() gains a third bucket: any visual match whose URL is on r/ismydisneypinfake, r/disneypins, or r/DisneyPinSwap gets routed to an amber-styled callout between the catalog and marketplace sections. Each card shows the subreddit, thread title, thumbnail, and a direct link. Zero extra SerpAPI quota burned — this is post-processing of the existing Lens result. When Lens visually matched the user's photo to a post on one of these subreddits, that's a strong authenticity signal worth reading before saving the pin.

4. Crawler “what is captured” disclosure. New <details> block on /catalog-crawler answers Drew's question from v0.15.75 about what fields the crawl actually persists. Walks through identifiers, display fields, hero image storage path, linked_pins JSON, Pin&Pop raw_json (counterfeit flag etc.), and explicitly calls out what isn't captured (comments, history, extra images, anything requiring a login). Closed by default.

5. Changelog pruned: ~143 entries archived. Pre-v0.14.37 entries collapsed into a single marker pointing at git log. The in-app changelog had grown to ~7000 lines of historical entries — over a third of main.py. v0.14.38 through v0.15.77 entries are kept since they cover the current session arc; older context (pre-Series-rename, early PTDB work, multi-AI plumbing) is still in git history if needed. main.py shed 922 lines (58k chars).

6. WDWNT feed backlog entry removed per Drew's explicit deprioritization. The DPB scrape and current /new-releases coverage are adequate; WDWNT is mostly parks news with pin coverage as a small filtered subset, not earning the build cost.

Deferred to v0.15.78: CLIP embeddings on catalog mirror. Genuinely 4-5h of focused work (new model dependency, new BLOB column, new backfill route, text↔image search surface). Punted from this session to keep v0.15.77 reviewable.

v0.15.76

2026-05-19

Round 2 of small fixes — v0.15.75 didn't fully land.

1. /catalog/browse 422 on empty year param. Drew confirmed the bug from his original batch still fires: /catalog/browse?q=&year=&source=ptdb&sort=newest returns 422 Unprocessable Entity with FastAPI's int_parsing error on the empty string. Root cause: the route declared year: Optional[int] = None — Pydantic tries to parse "" as int before defaulting to None, which fails. Changed to year: Optional[str] = None and manually coerce via year.strip().isdigit() in the body. Empty / non-digit values now treated as “no filter”. Template binding (value="{{ year or '' }}") still works with a string.

2. Crawler checkboxes still tiny after v0.15.75. The inline width: 1.15rem; height: 1.15rem from v0.15.75 landed but Pico CSS's [type="checkbox"] em-based dimensions were overriding the visual rendering due to display: inline-flex + font-size interactions on the surrounding labels. Switched to transform: scale(1.6) with transform-origin: left center — CSS transform can't be overridden by Pico's width/height rules, doubles the visual size regardless. Added margin-right: 0.55rem to compensate for the extra rendered width.

3. Search icon spacing still cramped. v0.15.75 set padding-left: 2.2rem on the search inputs but Pico's magnifier-icon background-position uses the same scale, leaving only ~0.45rem of breathing room between icon and placeholder text. Bumped to 2.8rem on /pins, /series, /collections.

4. Follow-up surfaces on /validation + nav badge moves there. Drew on v0.15.75: “add a part for that in /validation too, and add another # for it with another color and item count. it does not have to have # on the pins menu for count.” Implemented:
• /validation route loads followup_pins sorted by followup_flagged_at desc nullslast, id desc (latest-flagged first).
• New val_section() call in validation.html for “🚩 Flagged for follow-up” with amber #c68500 accent and the existing yellow-count-badge styling.
• Desktop top nav: third badge next to Validation, amber background, distinct from the red (needs_validation) and yellow (soft-validation) badges. Mobile more-sheet validation link gets the same.
• Removed the amber follow-up badge from the “Pins” nav link — both desktop and mobile — per Drew's “it does not have to have # on the pins menu for count”. The chip on /pins (also amber) remains for filtering.

5. SAWarning on Collection.get(None). Server log was emitting SAWarning: fully NULL primary key identity cannot load any object on every fresh pin save. The /new-pin POST handler called s.get(models.Collection, collection_id) unconditionally; when the user saved without picking a folder (collection_id = None), SQLAlchemy emitted the warning. Guarded with a conditional — s.get(..., collection_id) if collection_id else None. No behavior change; just kills the noise.

6. Known-scrapper sources: Disable + delete buttons now align. The narrow × delete button rendered visibly taller than the Disable button next to it — single-character button content has no natural width to balance padding, so the box ended up tall-and-thin. Wrapped both forms in display: inline-flex; align-items: stretch with height: 100%; min-width: 2rem on the delete button. Now they read as a pair instead of one tall pill next to one normal button.

v0.15.75

2026-05-19

Six small fixes from the deferred backlog.

1. Crawler stuck in RUNNING after all sources cap out. Drew's logs showed [crawler:pinpics] hit run cap (25/25) followed by endless /catalog-crawler/status polls with the badge still green. State-machine bug: when a source hit its per-run cap, _hit_cap() set enabled=False on that source, but the global state["status"] stayed "running". When the last enabled source capped, the thread just idled in its 2-second sleep loop and the UI sat on RUNNING forever. Fix: after every _hit_cap(), check whether any source is still enabled; if not, flip the global status to stopped with last_stop_reason = "All enabled sources reached their per-run cap...". Threads still idle and the next start() / reset_cap() revives them — just the badge flips correctly.

2. PTDB URL format on display links. Two templates were still building pintradingdb.com/pin.php?id=N (the old query-string form), which 404s for every ID. The fetch code was fixed in v0.15.52 but the display links in catalog_pin_detail.html and catalog_crawler.html (test-batch result rows) got missed. Both now use pintradingdb.com/pin/N — the canonical path-based URL PTDB serves today.

3. Catalog crawler checkboxes were nearly invisible. PinPics / Pin&Pop / PTDB toggles on /catalog-crawler were the browser-default checkbox size (~12px). Bumped all three to 1.15rem with explicit pointer cursor. Big enough to hit on a touch device, visibly registers as a control rather than a decoration.

4. Search icon spacing. Pico CSS injects a magnifier-glass icon via background-image on input[type=search]; the placeholder text was sitting right on top of it on /pins, /series, and /collections. Bumped padding-left from 0.6rem to 2.2rem on all three so the icon has breathing room before the placeholder.

5. Confirmed-fake checkbox no longer pre-checks itself. Pre-v0.15.75 the box was auto-checked when is_flagged_fake fired (Pin&Pop flagged the catalog entry as having counterfeits in circulation). Drew flagged: just because fakes EXIST for a design doesn't mean THIS scan is fake. Auto-checking risked silently mis-marking authentic pins as counterfeit. Now defaults off regardless; the “Confirmed fakes exist” banner above still surfaces the warning, but the checkbox is a deliberate user choice. The amber sub-text below the checkbox now says “Counterfeits of this design exist per Pin&Pop — visually compare your pin to the catalog reference photos before checking this box”.

6. “Catalog health” explainer disclosure. Drew asked “explain what these two things do” for the “Backfill image paths” and “Backfill embeddings” buttons on /catalog/browse. Added a <details> disclosure with a short explanation of each: the image-paths backfill is a filesystem scan that re-fills hero_local_path from JPEGs already on disk; the embeddings backfill runs each mirror image through DINOv2 to produce the 768-dim vector visual-search uses for ranking. Helps explain why one might have anything to do (e.g. a crawl got interrupted) and why visual-search results sometimes seem to miss pins (no embedding stored yet).

v0.15.74

2026-05-19

Hotfix: SerpAPI Lens card empty on page load. Drew reported the card moved to the AI provider cards section in v0.15.72 was rendering only the “Last run” timestamp + a blank box — the catalog matches list had stopped populating.

Root cause: the JS that hydrates the cached Lens result lives in a <script> block in the left sidebar (where the trigger button is). When that script runs, the target elements (#serpapi-lens-catalog-matches, etc.) are now further down the page inside the AI provider cards grid (moved in v0.15.72). The script executed immediately on parse, called getElementById for elements that hadn't streamed in yet, got null, and silently failed. Click-triggered runs worked fine because by then the page was fully loaded; only the on-load hydration broke.

Fix: wrap the hydration in a document.readyState check + DOMContentLoaded listener. If the page is still loading when the script parses, defer the render until DOM is ready; otherwise (manual cache hit on an already-loaded page) render immediately. Same pattern as the v0.15.69 page-ready ping. One-line plumbing change — no behavior change elsewhere.

v0.15.73

2026-05-19

🚩 Flag for follow-up on new-pin onboarding. Drew on v0.15.72: “add flag this pin for follow up later on new pin on boarding.” Plumbed end-to-end.

Data model. Four new columns on the pins table:
• followup_flagged (BOOL, NOT NULL DEFAULT 0) — the user's checkbox state.
• followup_reason (VARCHAR(64)) — one of bad_photo, id_uncertain, verify_authenticity, other. Stored as string so future categories don't need a migration.
• followup_note (TEXT) — free-text context.
• followup_flagged_at (DATETIME) — timestamp set when the flag is turned on, NULL otherwise. Used to sort the follow-up list newest-first.

Distinct from needs_validation, which is the AI-driven “low confidence” signal that auto-fires. This flag is purely user-driven: “I'm saving this now but want to revisit.” The two can co-exist on the same pin.

New-pin form UI. Amber-bordered checkbox block right below the “Confirmed fake / scrapper” red block on /new-pin. Unchecking the box hides the reason dropdown + note textarea via JS (no noisy empty inputs by default). Reason dropdown values match the column's expected strings exactly.

Save logic. Symmetric to the confirmed_fake handling: on a fresh save the form values flow straight into the new Pin row. On a rescan that links to an existing pin, the flag is ONLY promoted — unchecking the box during rescan does NOT silently clear an existing follow-up flag (user might have set it on a different code path; rescan shouldn't wipe state). Reason + note get overwritten when the box is checked so the latest scan's context wins.

/pins filter. New ?followup=1 query param filters the list. An amber “🚩 Follow-up” chip sits next to “All” and “Pending validation” in the filter bar. Active state colors the chip filled amber. Follow-up state also flows into has_filters + the saved-search URL computation so save-this-search works correctly.

Nav badge. Amber count badge next to “Pins” in the desktop top nav (e.g. “Pins <span style='background:#c68500;color:white;padding:0 0.4rem;border-radius:999px;font-size:0.7rem;'>3</span>”) and the mobile bottom-tab-bar (small badge bubble in the top-right corner of the pin icon). New _followup_count() helper mirrors the existing _validation_count pattern: 30s TTL, fail-soft to 0 on query error. Cheap COUNT on the indexed boolean column.

v0.15.72

2026-05-19

Lens (SerpAPI) result card joins the AI provider cards section. Drew on v0.15.71: “Looks like it's working, i want the lens to be in the same modal though as the other AI provider cards, rather than on the left side.”

Moved the SerpAPI Lens result render from the left sidebar (under the trigger buttons) into the “🧠 AI provider cards” expandable section, styled as a sibling card alongside Gemini Flash / Claude / OpenAI. The card uses a new .ai-panel.lens style with a blue accent and a magnifier-glass icon (🔍) in the header to match the icon used in the v0.15.71 “Lens contributed” pill.

Mechanics:
• The trigger button on the left sidebar stays where it was — it's the “run a new SerpAPI Lens search” control. Only the rendered result moved.
• All the original container IDs (#serpapi-lens-panel, #serpapi-lens-catalog-matches, etc.) moved with the panel, so the existing runSerpApiLens click handler + cached-result hydration both still work without JS changes.
• The card is always mounted (even with no cached run), so the click handler has somewhere to write. Pre-run state shows at opacity 0.6 with a “No Lens run yet” placeholder; the JS un-dims it when results arrive.
• Click handler now also force-opens the parent <details> when a search finishes, so the user sees the result even if they had the AI-cards section collapsed when the click fired.
• The “N active” count in the section summary now counts Lens when there's a cached result, so the tally matches what's visible.
• CONTRIBUTED pill on the Lens card header when Lens voted for either chosen ID — reuses the same lens_contributed_to_pp / lens_contributed_to_pap flags from v0.15.71. Mirrors how AI provider cards get the “active” star indicator when their suggestion is currently prefilled into the form.

v0.15.71

2026-05-19

Set members: exactly one source + Lens joins the decision loop. Two related fixes both about reducing noise in the new-pin panel.

1. Set-members panel = exactly one source (Pin&Pop primary, PinPics fallback). v0.15.70's three-fold merge was Drew's second test: the panel now showed 7 AI-only named cards mixed with 9 PinPics linked-pin cards (which included the packaging pin and adjacent-but-different sets), totalling 16 entries when the actual set is 10. Reworked to strict-priority single source:
• Tier 1: Pin&Pop series_data.pins — the canonical “Set of 10” list from the series page. When present, this fully replaces any AI-merged set_members. The earlier v0.14.56 Pin&Pop pin-page linked_pins fold (which was ALSO appending to set_members from a less-canonical source) gets clobbered by the tier-1 replacement, which is correct.
• Tier 2: PinPics linked_pins from the chosen candidate — broader than tier 1 (includes adjacent pins, sometimes the packaging) but at least has images. Only runs if tier 1 returned nothing.
• Tier 3: AI-merged set_members — name-only fallback. Only runs if both tier 1 AND tier 2 are empty. This is the “PTDB-search-only pills, no images” state Drew has been seeing; now reserved for the genuine edge case where both scrapers fail.

Local catalog mirror enrichment (v0.15.68) still runs on top of whichever tier won, so PTDB direct URLs and missing IDs get filled in regardless of source. Logged as [set-members/v0.15.71] tier N (source) → M members per request.

2. Google Lens (SerpAPI) joins the catalog-ID decision loop with double-weight voting. The Lens panel from v0.15.66-67 was a separate UI surface that couldn't affect the chosen PinPics / Pin&Pop IDs — if Gemini Flash said “Pin&Pop #29815 Duffy” and Lens said “Pin&Pop #19567 Minnie”, the AI panel still showed Duffy as chosen. Now Lens votes alongside the AI providers:
• New helper ai_identify_common.synthesize_lens_suggestion() parses the cached scan.serpapi_lens_json matches list and extracts the FIRST PinPics URL and FIRST Pin&Pop URL (Lens orders by similarity score, so earlier = better visual match). Returns a synthetic suggestion dict with the same shape AI providers emit, tagged _provider_slug = "serpapi-lens", _vote_weight = 2, _no_veto = True.
• merge_catalog_ids() updated to honor _vote_weight in the plurality pass and _no_veto in the veto pass. Gemini still goes first for vetoes (its Google Search grounding is the strongest signal for these niche catalog sites); Lens contributes only via the voting pass. Weight 2 means Lens beats ANY single AI provider but loses to 2+ concurring AI providers. Drew's design call: “visual evidence > LLM guess” but not so dominant that a single false-positive Lens result wipes out a multi-provider AI consensus.
• PinPics and Pin&Pop candidate-fetching loops add “Lens (SerpAPI)” as a candidate-source row, so when Lens picks a different ID than the AI providers it appears as a row in the disagreement panel with the same “Use this ID →” override action as the others. Visual-similarity tie-breaking still applies, but now Lens's candidate is in the pool.
• New merge_catalog_ids return field pinpics_voters / pinandpop_voters — per-ID list of provider slugs that voted. The new-pin context uses this to set lens_contributed_to_pp / lens_contributed_to_pap, which surface as a blue “🔍 Lens contributed” pill next to the “USING: GEMINI FLASH” pill at the top of /new-pin when Lens helped pick either chosen ID.

What didn't change: SerpAPI Lens still runs in parallel with the AI providers (v0.15.69 plumbing), still gets cached on scan.serpapi_lens_json, still renders the catalog-first match list in its own panel (v0.15.67) for users who want to inspect or manually override. The v0.15.71 additions are purely about making the cached result participate in the same identification pipeline as the AI providers rather than sitting in a separate panel.

v0.15.70

2026-05-19

Set-members panel: actually fold in PinPics linked_pins + Pin&Pop series_data. Drew tested v0.15.68 and the Villain Portraits set still showed 7 name-only cards with PTDB-search-only pills. Quote: “i jsut did a new search too just in case, and it did not pull in the set data”.

Root cause: even with the v0.15.68 local-catalog enrichment, the set members had nothing to enrich FROM. The /new-pin route was only folding Pin&Pop's pin-page linked_pins (which is empty for many pins — not every Pin&Pop pin page surfaces the “More from the series” strip), and was throwing away PinPics's linked_pins entirely. The Pin&Pop series page scrape (series_data) populated series_data.pins but the result was only rendered in a separate template panel — never folded back into the unified set_members list that the AI-merged panel renders.

Three new folds, run in sequence after chosen_pp_id is determined:
1. PinPics linked_pins from the chosen candidate. Preserved on pinpics_candidates_data at fetch time (previously discarded after embedding extraction). The chosen candidate's linked_pins is the most reliable set-member source because PinPics isn't JS-rendered — the “Other pins in this series” strip is in the HTML verbatim. Folded with dedup by pinpics_id and lowercase name; tags new members with sources: ["pinpics-linked"].
2. Pin&Pop series_data.pins. When fetch_pinandpop_series succeeds (the Next.js __NEXT_DATA__ parser found the canonical pins array), each pin is folded into set_members. The parent pin itself is skipped so it doesn't appear in its own “set members” list. Each fold dedupes by pinandpop_id and name. Tag: sources: ["series-data"].
3. Local catalog mirror (v0.15.68). Runs last so it can backfill ptdb_id + direct PTDB URLs on members we just folded in from steps 1 and 2.

Result: when either PinPics linked_pins OR Pin&Pop series_data succeeds (one of them almost always does), set members now arrive with images (the scraped thumbnail URLs), PP# / P&P# pills (the extracted catalog IDs), and the PTDB pill upgraded to a direct pin URL when the local mirror has it. The useless PTDB-search-only fallback only fires when all three sources fail.

Logging: each fold prints [set-fold/pinpics] +N new, M enriched (from K linked_pins on PP#X) and the equivalent for series_data, so it's visible in the server log whether the folds fired and how many members they contributed.

New diagnostic endpoint /catalog-debug. JSON dump of the raw scraper output for any pinpics_id + pinandpop_id pair. Hits the same disk cache /new-pin uses, so it's cheap to call repeatedly. Surface fields:
• _linked_pins_count_pinandpop — how many linked_pins Pin&Pop's pin-page scrape produced
• _series_pins_count — how many pins the series-page scrape produced (and the next_data_diag block inside series if you need to see which __NEXT_DATA__ arrays the parser found)
• _linked_pins_count_pinpics — how many linked_pins PinPics's scrape produced

Usage: /catalog-debug?pinpics=162531&pinandpop=74755. Drew: if a set still looks broken on /new-pin, hit this URL first to see which of the three scrapers is actually returning data. The folds in v0.15.70 only work as well as the data they have to fold — this endpoint reveals the source-of-truth.

v0.15.69

2026-05-19

SerpAPI Lens as a default-search provider + page-ready ping.

1. Google Lens (SerpAPI) joins the parallel-providers fan-out. Drew confirmed v0.15.66 + v0.15.67 worked well and asked to make it a checkbox alongside Claude / OpenAI / Gemini in the scan-page provider picker. Now wired up:
• New checkbox on /scan: “Google Lens (SerpAPI · visual)”, checked by default. Hidden when SERPAPI_API_KEY isn't configured.
• scan_progress_page's PROVIDER_INFO dict gained the serpapi-lens slug. The dict shape changed from per-row endpoint suffix (endpoint: "-opus") to per-row absolute URL (url: "/scan/{sid}/serpapi-lens") because the SerpAPI route lives outside the /ai-identify-* namespace. The other six provider rows still resolve to /scan/{sid}/ai-identify-{suffix} — just one tiny URL template change in the template + JS.
• progress.html renders data-url per row; its JS now reads row.dataset.url instead of concatenating a hardcoded /ai-identify prefix. Otherwise the orchestration logic is unchanged — SerpAPI Lens runs in parallel with the LLMs, total time is bounded by the slowest call (usually OpenAI), and the result is cached on scan.serpapi_lens_json just like a manual button click would write it.
• By the time /scan/N/new-pin loads, the SerpAPI Lens result is in cache, so the in-page panel renders immediately — same UX path as v0.15.67 but the user no longer has to click the button manually.
• All four scan.html render sites and the /new-pin context gained serpapi_lens_available from authenticity_check.google_lens_available() so the checkbox shows/hides consistently.

2. Page-ready audio ping on /scan/N/new-pin. Drew asked: can we get an alert when the page loads? Catalog + eBay + Pin&Pop fetches can take several seconds and the user often tabs away. Now plays a brief 660Hz sine pulse (~180ms, gentle attack-decay envelope) on DOMContentLoaded via Web Audio API. No audio file needed; works offline. Skipped on rescan mode (where the user is actively reviewing an existing pin rather than waiting on a fresh AI run). User-toggleable via the new new_pin_ready_ping app_setting; default on.

3. On “load faster”: the actual parallel-fetch architecture was already in place from earlier versions — the AI fan-out is via ThreadPoolExecutor in ai-identify-multi, set-member catalog verification parallelizes via ThreadPoolExecutor in v0.15.26, image optimization at /i/ is fast-pathed for small files. The big remaining serial steps are Pin&Pop / PinPics fetches and eBay listing lookup, both of which are disk-cached on repeat visits. Adding more parallelism on the /new-pin route is a bigger refactor (would need a load-skeleton-then-hydrate pattern). For now the ping closes the “is it ready yet?” UX gap with a few lines of JS rather than rebuilding the page-load model.

v0.15.68

2026-05-19

Set-members panel: enrich from local catalog mirror. Drew flagged two bugs on the Shan Yu Villain Portraits scan:
• The 7 identified set members had no images
• Their PTDB link was a name-based search query (pintradingdb.com/pinList.php?txtSearch=Maleficent) which returns junk — not a direct pin URL

Root cause: Pin&Pop's linked_pins scrape didn't populate for this pin (page structure variance), so the set members in the panel came from AI providers only. AI providers reliably identify names but rarely carry PinPics / Pin&Pop / PTDB IDs for set members. With no IDs, the template fell back to a useless search URL and couldn't render thumbnails.

Fix: new _enrich_set_members_from_local_catalog() helper does a second-pass lookup of each set member against the local catalog_pins mirror (populated by the PTDB crawler + accumulated PinPics/Pin&Pop scrapes). Match priority: pinpics_id → pinandpop_id → name (case-insensitive equality scoped to the parent pin's series_name). When a match is found, fill in (without overwriting existing values):
• ptdb_id + ptdb_url — direct pintradingdb.com/pin/N, not a search query
• image_url — /catalog-uploads/<hero_local_path> so the card renders a real thumbnail from the disk-cached mirror image
• Missing pinpics_id / pinandpop_id if the mirror row has them (cross-reference fill-in)

Name lookup is scoped to the parent pin's series_name so a name like “Maleficent” doesn't match a Maleficent pin from an unrelated set. Bound also keeps the query off a full-table scan.

Template update: when m.ptdb_url is set, render a “PTDB #N &nearr;” pill that opens the pin page directly. Falls back to the old name-search pill (greyed out now to signal it's a weaker link) when no mirror match exists. Same for image_url — the local catalog mirror image takes precedence over external URLs, so even if Pin&Pop's linked_pins fails, we still render thumbnails when the PTDB crawler has covered the set.

Single DB read for the whole panel (one IN-query for IDs, one filtered query for names) so the latency impact is negligible. Logged as [set-enrich] enriched N/M set members from local catalog mirror for diagnostic visibility.

v0.15.67

2026-05-19

SerpAPI Lens panel reshaped: catalog-first, override-ready. Drew tested v0.15.66 against a Minnie bow pin scan whose AI panel had Pin&Pop wrong (chose #29815 “Duffy”, obvious mismatch). Google Lens found both the correct PinPics and Pin&Pop catalog pages; the v0.15.66 panel rendered all 25 matches as a thumbnail grid, drowning the two useful rows in eBay/Etsy/Mercari noise. Reshaped the panel:

• Catalog matches first. Filter to URLs matching pinpics.com/pin/N, pinandpop.com/pins/N, or pintradingdb.com/pin/N. Render each as a wide row with thumbnail + title + extracted ID + status pill. Sort: PinPics, Pin&Pop, PTDB.
• Status pill per row. Green “✓ matches chosen” when the Lens ID matches what the AI panel already picked. Yellow “differs from chosen #N” when it conflicts (the actionable case Drew identified). Gray “new candidate” when there's no current ID for that source.
• One-click override for differing IDs. For PinPics and Pin&Pop, a blue “Use this ID →” button on each row links to ?override_pinpics_id=N or ?override_pinandpop_id=N — the existing override mechanism (added in v0.14.26 for the AI-disagreement panel). Page reloads with the new ID; the existing user_override_pinpics flow handles re-fetch and “(user override)” badge in the disagreement panel.
• PTDB doesn't have an override path (PTDB IDs aren't on the pins table; they live on catalog_pins as a mirror-only ID). PTDB rows show with a “View local catalog” link instead of an override button. Adding PTDB to the override flow is a separate change (would need Pin.ptdb_id column + migration + scrapper integration in catalog_sites).
• Dedup. Lens sometimes returns the same catalog pin via multiple URLs (e.g. the canonical pinpics.com/pin/N AND a thumbnail URL). Same (source, id) pairs are deduplicated, with the surviving row showing “also matched at another URL” as a hint.
• Marketplace matches collapsed. Anything not on the three catalog domains is hidden behind a <details> disclosure labeled “Show N marketplace match(es) (eBay / Mercari / Etsy / etc.)” — same rendering as before, just out of the way by default.

The server route and cache shape are unchanged from v0.15.66; the raw matches list is still stored on the Scan row in case we want to do more with it later (e.g. surface SUSPECT-domain hits as a counterfeit-existence signal). All v0.15.67 changes are in the template's rendering JS.

v0.15.66

2026-05-19

SerpAPI Google Lens button on /scan/N/new-pin. Drew asked: now that the lens.google.com handoff works, can we also pull the visual-match data back to us as JSON? Answer: not from the lens.google.com link button — that's outbound-only (user looks at Google's page, no data flows back). But we already have authenticity_check.google_lens_check() wired up to SerpAPI's engine=google_lens endpoint, which DOES return structured JSON: a list of visual matches each with URL, title, thumbnail, and domain classification.

Why this is a separate button from the existing Check authenticity flow: the heavyweight POST /pins/{pid}/check-authenticity route fires ALL FOUR providers per click (Google Vision + Google Lens + Yandex + Copyseeker), burning 2-3 SerpAPI calls and 1 Copyseeker call each time. That's right for “is this pin a fake”, wrong for “just show me Lens matches.” Drew wants something targeted to use when the AI identify panel doesn't surface the correct catalog match.

Quota correction: Drew asked if the free-tier 250 is shared between Copyseeker and SerpAPI. It isn't — they're separate services with separate API keys. He has 250 free Copyseeker searches and 250 free SerpAPI searches each month. SerpAPI's 250 is shared across all SerpAPI engines on his key (Lens, Yandex, regular Google Search, etc.), so a Lens-only button uses ~1 search per click.

What landed:
• New column on scans: serpapi_lens_json + serpapi_lens_checked_at, holding the parsed result so re-viewing the same scan doesn't re-spend quota. Migration is in _LATE_ADDITIONS.
• POST /scan/{scan_id}/serpapi-lens: calls only google_lens_check(), returns the result as JSON, caches on the Scan row. Logs every call with match count + legitimate-domain count to the uvicorn terminal.
• GET /scan/{scan_id}/serpapi-lens-cached: returns the cached result for fetch-style use.
• New “Lens (SerpAPI)” button on /scan/N/new-pin, next to the existing Lens-in-new-tab buttons. Hidden if SERPAPI_API_KEY isn't set. Tooltip warns it uses 1 search per click.
• Inline result panel: thumbnail grid of matches, each with domain badge (Legit / SUSPECT / neutral), color-coded via the existing classify_domain() logic that already knows pinpics.com, pinandpop.com, disneypinsblog.com as catalog domains and aliexpress/dhgate/temu as suspect.
• URL auto-extraction: if a match URL is pinpics.com/pin/<N> or pinandpop.com/pin/<N>, the panel surfaces a quick-nav chip linking to the local /catalog/pinpics/<N> or /catalog/pinandpop/<N> page so Drew can jump straight to the catalog row when Lens finds a match the AI identify missed.

Stale comment update: authenticity_check.py said “Free tier: 100 searches/month” for Google Lens; that was outdated — SerpAPI's free tier is 250/month per the current pricing page.

v0.15.65

2026-05-19

Lens IS WORKING as of v0.15.64. Drew's log: two clean fetches from ua='Google-Lens' at IPs 192.178.9.x (Google's crawler range). One was the 331KB fast-path (small file), one was a 1.7MB→383KB optimization. The .jpg suffix from v0.15.64 was the missing piece — Lens was rejecting our URLs at the pre-fetch validator because they didn't look like image paths.

Fake-corpus auto-sync freshness anchor. Drew asked if the scheduler is firing “every few minutes” for fake-corpus. Configured intervals say no: scheduler wakes every 30 min, fires sync only if state is >6h stale. Drew was probably reading the post-by-post [fake-corpus] skip ... log lines as separate sync events — those are individual posts being evaluated within a single sync, not separate fires. Only the [scheduler] fired fake-corpus newest-mode sync line marks an actual fire, and that appeared exactly once in his log.

While checking the wiring, I did spot a real but minor gap: end_running() wasn't refreshing last_auto_sync_at on completion. Newest-mode runs (the auto-sync flavor) also don't touch last_run_at — that's backfill-only. So the staleness window was anchored only to the kickoff timestamp from mark_auto_sync_at(). If a sync errored after kickoff, the window stayed anchored to the (failed) kickoff time. Now end_running() refreshes the anchor on successful completion, so the next 30-min scheduler tick reliably sees a recent timestamp.

v0.15.64

2026-05-19

Lens: URLs now end in .jpg + admission I've been misreading the logs.

Drew shared a HAR file of the Google Lens results page. Zero requests to dispindable.com in 25 entries. Then I went back and re-read the v0.15.58 [i] 200 log more carefully:

ua='Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome'

That's an Android Chrome user-agent. That's Drew's phone, not Google Lens. Lens's server-side fetcher would identify as Google-Lens or similar. I've been treating every [i] 200 in the log as evidence Lens fetched the URL — but Lens probably has never successfully fetched our URL in any of the versions we've tested. Every [i] 200 has been Drew manually loading the URL to verify it works.

New hypothesis (informed by comparing our URL to the working Streamlit + Catbox reference):
• Catbox URL: https://files.catbox.moe/abc123.jpg
• Our pre-v0.15.64 URL: https://dispindable.com/i/cGluXzc5X2Zyb250Xz...9Q (78-char token, no extension)

Lens probably pre-validates URLs before queueing a fetch. Our URL doesn't look like a typical image path (unusual characters, no extension), so Lens rejects it at the URL parser stage and shows “No image at the URL” without ever actually trying to fetch.

Fix: append .jpg to public_image_url's output. New URLs look like https://dispindable.com/i/TOKEN.jpg. The /i/{token} route strips the .jpg suffix before HMAC verification, so old tokens without the suffix keep working. v0.15.63's image optimization is still in effect (1600px max, JPEG q85 for files >500KB).

What to verify after deploy: the version banner should show v0.15.64. Hit /lens-debug and confirm “ACTUAL Lens button href” ends in .jpg. Then click Lens. If Lens still fails, this hypothesis is wrong and we'll move to plan C (separate subdomain with no /i/ path, or short DB-backed nonce URLs).

v0.15.63

2026-05-19

Lens fix: optimize images at /i/ serve time. Drew shared a working Streamlit reference app that successfully calls the SAME lens.google.com/uploadbyurl endpoint we've been using. Reading the reference vs what we do shows one concrete difference that matters: the reference resizes the image to 1200px max and re-encodes as JPEG quality 85 before Lens ever sees it. Our /i/ route was serving raw 1MB+ phone photos unchanged.

Strong hypothesis: Lens has an undocumented size/timeout limit. The v0.15.58 [i] 200 log line we saw showed Lens DID download the 1MB file successfully — but the UI still showed “No image at the URL.” That makes sense if Lens fetched it, then rejected it for size or decode-time. The reference app sidesteps the limit by optimizing first.

Now /i/<token> does the same optimization at serve time:
• Files under 500KB pass through unchanged (fast path)
• Larger files are opened with PIL, resized to 1600px max if needed (slightly more permissive than the reference's 1200px), and re-encoded as JPEG q85
• Result is streamed back as image/jpeg
• ?raw=1 query param bypasses optimization for diagnostics
• If PIL fails for any reason, falls back to the raw file (better to give Lens a too-big image than 500)

Log line now includes before/after sizes when optimization fires: [i] 200 fn=pin_79_front_x.jpg (optimized 1043793B->215000B, (3024, 4032)->(1200, 1600)) ...

Caveat on Gemini's explanation: Drew shared Gemini's writeup of why the Streamlit app worked. Several of Gemini's specific technical claims are wrong:
• “Localhost 403 / CSRF” — irrelevant; dispindable.com is a public domain
• “Reverse-engineered hidden API” — both implementations use lens.google.com/uploadbyurl, the same public endpoint
• “413 Payload Too Large from base64” — irrelevant; we've always used a short signed token URL
• “Browser bot detection” — the click navigates the user's browser to Lens identically in both flows

The Catbox.moe step in the reference is also a red herring — it's just where the optimized image gets hosted. The actual difference is the optimization itself, which is now in our serve path.

v0.15.62

2026-05-19

Startup version banner. Drew has been deploying version after version through this Lens debugging marathon, and we keep needing to verify “is the latest build actually running?” The uvicorn log gives no signal — “Application startup complete” looks identical for every version. Now a 60-char-wide banner prints right before db.init_db():

============================================================
DisPindable v0.15.62
PUBLIC_BASE_URL: https://dispindable.com
============================================================

Visible in the uvicorn terminal immediately after every restart, alongside the PUBLIC_BASE_URL value (also a common source of “is the env var set?” uncertainty during Lens debugging).

v0.15.61

2026-05-19

Reverted v0.15.59 URL encoding + added click-beacon diagnostics.

1. Reverted URL encoding. v0.15.59's quote(signed, safe="") was speculation; I had no proof it helped. v0.15.58 with the unencoded URL at least produced an [i] 200 log line (Lens reached our server and got the image). After v0.15.59 zero /i/ requests showed up. Either: (a) encoding broke Lens's URL parser, or (b) Lens cached the prior failure and stopped trying. The data can't distinguish these. Reverting to the format that we have evidence of working end-to-end.

2. Lens-click beacon. Drew asked “can you add more diagnostics?” Biggest gap: we can't tell “user didn't click” from “Lens never fetched after click.” The Lens button on /scan/N/new-pin and /pins/N now fires navigator.sendBeacon('/lens-click-log', {...}) on click BEFORE the browser navigates to lens.google.com. Beacon arrives even when the user immediately navigates away (that's the sendBeacon API's entire purpose). Server logs every beacon as [lens-click] cf_ip=... ua=... payload={fn,href,from}. So:
• Beacon fires + no [i] request follows ⇒ Lens got the URL but never tried to fetch (cache, URL validation, mobile quirk — all downstream of us)
• No beacon ⇒ user didn't actually click the button
• Beacon fires + [i] 200 + Lens still errors ⇒ image delivered, but Lens rejected the content (size, format, etc.)

3. Enhanced /i/ logging. Now also logs CF-Connecting-IP, CF-IPCountry, CF-Ray, and Referer. The CF-Ray is particularly useful — it's Cloudflare's per-request ID, so if Drew sees a [i] 200 with a CF-Ray, he can search for that exact ray in Cloudflare's zone analytics to see what CF did with the request.

What to look at in Cloudflare:
• Security → Events (zone-level): shows any WAF blocks. If Lens requests are being blocked at the edge before reaching the tunnel, they'd show here.
• Analytics & Logs → Traffic: aggregate request counts per path. Look for traffic to /i/* beyond the user's own browsing.
• Free tier doesn't include full HTTP logs — that's Logpush on paid plans only. But the events + aggregate views above usually answer this question.

v0.15.60

2026-05-19

/lens-debug now reflects the actual emitted href + does a server-side self-test. Drew reports that after v0.15.59's URL-encoding fix, clicking Lens produces zero /i/ logs on the server — whereas v0.15.58 showed at least one [i] 200 line per click. That's a possible regression, but it's also consistent with Lens caching a prior “No image at URL” failure and not re-fetching. Can't distinguish from the data we have.

Two issues with the v0.15.58 /lens-debug route that I caught while re-reading my own code:
• The “What this Lens button URL looks like” line was hardcoded as uploadbyurl?url={signed_url} inline. It did NOT call _lens_url_for(). So even after the v0.15.59 encoding change, /lens-debug kept showing the OLD unencoded format — the debug page was lying about what the template actually emits.
• No way to test whether the signed URL works without involving Google Lens. If Lens is buggy or caching, you can't tell from clicking Lens alone.

Fixes:
• Actual Lens href: /lens-debug now calls _lens_url_for(request, test_file) directly and prints the exact string the template emits. After deploying, hit /lens-debug and paste the “ACTUAL Lens button href” line back — that'll tell me whether v0.15.59's encoding survived deploy.
• Server-side self-test: /lens-debug now also fetches its own signed URL through the public Cloudflare origin using requests.get(). If the self-test reports HTTP 200 with a matching body length, our /i/ + CF bypass + signed URL chain is verifiably working end-to-end — any Lens failure after that is downstream of us (Lens cache, content validation, etc.). If the self-test fails, the issue is in our infra.

What I'm no longer sure about: whether v0.15.59's URL encoding actually fixed anything or made it worse. The v0.15.58 logs DID show [i] 200 after the 404s — meaning Lens reached the server eventually, but still showed “No image at URL” in the UI. That suggests the real failure was downstream of our /i/ endpoint, not in URL parsing. My v0.15.59 commit may have been speculation rather than a fix. If the v0.15.60 self-test passes but Lens still shows the error, I'll revert v0.15.59 back to unencoded since unencoded was at least known to deliver the image successfully.

v0.15.59

2026-05-19

Lens URL parameter wasn't URL-encoded. Drew used the new /lens-debug + /i/ logging from v0.15.58 to produce the smoking gun. Three log lines from a single click on the Lens button:

[i] 404 invalid-token ip=... ua='Mozilla/5.0 (Linux; Android 10; K) ...'
[i] 404 invalid-token ip=... ua='Mozilla/5.0 (Linux; Android 10; K) ...'
[i] 200 fn=pin_79_front_1779181018_00.jpg ip=... ua='Mozilla/5.0 (Linux; Android 10; K) ...'

Two 404s arrived BEFORE the eventual 200. The 404s had a specific corruption: the token was duplicated. Token VALID (length 78) became VALID + VALID[16:] (length 140). Position 16 in base64 is exactly the boundary after the filename prefix pin_79_front decoded — so something was splitting on the filename or on a special char inside it. Lens errored out with “No image at the URL” before the correct fetch even completed.

Root cause: _lens_url_for() was building https://lens.google.com/uploadbyurl?url=https://dispindable.com/i/TOKEN with the url= parameter value NOT URL-encoded. Mobile Chrome (or Lens's own URL parser on mobile) was getting confused by the unencoded :// and / inside the parameter value and producing malformed retries with doubled token segments. Standard practice is to URL-encode parameter values; we should have been doing it from the start. Now uses urllib.parse.quote(url, safe="") which encodes every non-alphanumeric character — colons become %3A, slashes become %2F, etc.

Cloudflare Access setup confirmed correct. Drew shared screenshots of his zerotrust.cloudflare.com config: app “dispindable.com” with two policies (base allow + Public images for Lens/Copyseeker BYPASS for Everyone on path /i/*). The whole infrastructure was fine all along — it was the URL-encoding miss inside the app. Should be one of those rare bug fixes where there's nothing else to verify on the user side; just pull the new build, restart uvicorn, click Lens.

v0.15.58

2026-05-19

Series sort bug, Lens diagnostic route, /i/ request logging.

1. Series sort wasn't reversing folders. Drew: “sorting works on /pins and /collections, but not on series.” The /series page renders folder-by-folder (each folder shows its series cards in sort order). Pre-v0.15.58 the folder query was hardcoded to order_by(name) regardless of the user's ?sort= choice, so “Name Z-A” only reversed series within each folder — the folder ordering itself stayed alphabetical (Animals of Disney, Beauty and the Beast, Disney Around the World, etc.), making the sort look like it wasn't doing anything. Now folder order honors name vs name_desc to match. Other sort keys (completion, recently_added) keep folders alphabetical since folders don't have those data fields of their own.

2. New /lens-debug diagnostic route + /i/ request logging. Drew reports Google Lens still shows “No image at the URL” from both LAN and the production tunnel. The v0.15.48 fix routes the Lens button through a signed /i/<token> URL that should bypass Cloudflare Access. The signing code looks correct, but Drew's uvicorn log shows zero GET /i/<token> entries from Google's IPs — strongly suggesting Google never reaches the server. Likeliest causes: (a) Cloudflare Access bypass policy for /i/* is missing or wrong, (b) Cloudflare WAF is blocking Google Lens's fetcher at the edge.

Two changes to make this diagnosable without me speculating:
• /lens-debug route picks the most recent /uploads/ file, generates a signed URL the same way the Lens button does, and renders a plain-text dump showing: the resolved PUBLIC_BASE_URL, the signed public URL, the direct local URL, and the exact Lens uploadbyurl link. Drew copies the signed URL into a new browser tab (or phone hotspot) and confirms the image actually loads. If it does, the issue is in Lens; if it doesn't, the issue is in Cloudflare config.
• /i/<token> request logging: every request to the route now prints [i] 200 fn=foo.jpg ip=... ua=... on success or [i] 404 invalid-token ip=... ua=... on failure. So when Drew clicks the Lens button and the uvicorn terminal shows zero [i] lines afterward, that confirms Google never reached the server (Cloudflare blocked it). If [i] 200 fires but Lens still complains about “No image,” the failure is downstream of our code (e.g. Lens rejecting the content-type or size).

v0.15.57

2026-05-19

One-step launcher scripts (start-dispindable.bat for cmd.exe / double-click, start-dispindable.sh for Git Bash). Both do the same thing:
• cd into the project directory (regardless of where they were invoked from)
• Activate .venv (Windows path or Unix path, auto-detected on the .sh side)
• Set PUBLIC_BASE_URL=https://dispindable.com if not already exported (so Lens URLs stay valid through the Cloudflare Access bypass on /i/)
• Auto-pick HTTPS-on-LAN vs HTTP-localhost: if cert.pem + key.pem are both present, launch uvicorn --host 0.0.0.0 --ssl-keyfile key.pem --ssl-certfile cert.pem so phones on the LAN can reach https://<lan-ip>:8000. Otherwise fall back to --host 127.0.0.1. No behavior change for the existing manual uvicorn workflow.

Both scripts pause/exec at the end so Ctrl+C-driven shutdown messages stay visible. Sanity checks for missing main.py and missing .venv with copy-pasteable setup commands in the error path.

Releases auto-sync confirmed working. Drew asked “do we have auto refresh for new releases yet?” Answer: yes, three layers since v0.15.24 + v0.15.41:
(1) scheduler.py _check_releases() fires every 30 min when the feed is >6 hours stale; (2) lazy auto-sync on /releases page visit fires a BackgroundTask if stale (same 6h threshold); (3) manual “Sync now” button on the page. Drew's screenshot showing “Last synced: 0s ago” with 32 unseen items is the lazy auto-sync working as intended — Disney Pin Blog feed is healthy. Removing the “DPB parser rewrite” entry from the open backlog since it's no longer broken.

v0.15.56

2026-05-18

Per-ID catalog crawler failure log + gap-range summarizer. Drew asked: “if we're going to spend the time to get all these, it'd be nice to know where the gaps are and what the issue was/is.” Pre-v0.15.56 the crawler only stored aggregate counters in .catalog_crawler_state.json: how many successful, 404, blocked, errored. That tells you 25,000 IDs failed but nothing about which.

New SQLite table catalog_crawl_failures logs every non-successful fetch outcome with: source, pin_id, outcome category, error message, fetch duration, attempt count, and timestamp. UNIQUE on (source, pin_id) so retries update the existing row rather than producing duplicates — table size is bounded by ID-space-size, not by total fetch attempts. Successful fetches already have a row in catalog_pins, so they're not duplicated here; a now-successful fetch on a previously-failed ID deletes the failure row.

UI additions on /catalog-crawler:
• Aggregate counts per source × outcome, always shown at the top of the new Failure log section. Cheap query, no opt-in needed.
• Recent failures table (opt-in via ?failures=1): paginated list of the 200 most recent failures with relative-time stamps. Filterable by source and by outcome category. Shows attempt count and duration alongside the error message.
• Gap-range summary (opt-in via ?ranges=1): collapses contiguous 404 ranges into single rows. So PTDB's “IDs 1-99,999 are all 404” dead zone shows up as one entry — with a Jump past button that sets current_id to end+1 via the existing /jump-to route. Single-click gap-skipping.
• Clear all button for wiping the log after a URL-pattern or parser fix (like the PTDB v0.15.52 change) makes existing entries stale.

Lazy-loaded for performance. The aggregate counts query is a single grouped SELECT (fast even at 1M+ rows). The recent-failures and gap-range queries only fire when the user clicks the opt-in buttons. Default page render stays as fast as v0.15.55. Acceptable tradeoff: one extra page load to expand the section vs. always-on for long crawls.

v0.15.55

2026-05-18

Auto-tag subject extractor finds mid-sentence subjects. Drew gave a worked example: description “The pin features Dewey, wearing his signature blue shirt and a matching blue nightcap, dipping his finger into a jar of strawberry jam...”. Asked which 3 keywords I'd pick: Dewey, strawberry jam, Hidden Mickey. Then asked whether the same logic could be added to the auto-tagger, since the existing code “doesn't account well for what is the primary subject of the pin”.

He's right. The pre-v0.15.55 _extract_auto_tag_subjects() used re.match(r'^[A-Z]...') on the description — anchored to the very start. Modern AI-generated descriptions often open with framing language (“The pin features X”, “Featured here is X”, “Shown above is X”) that pushes the actual subject past the first capitalized word, so the old regex grabbed “The”, filtered it as generic, and returned nothing. Drew's Dewey pin was a clear miss.

Rewrote the extractor as _score_description_subjects():
• Tokenize the description word-by-word.
• Group consecutive capitalized words into noun-phrase candidates.
• Strip leading structural generics (“The”, “Featured”, etc.) from each group, then take up to 2 remaining words. So “The Mad Hatter” yields “Mad Hatter”, not an orphan “Mad”.
• Score remaining candidates by (frequency, position). A name that appears 2+ times in the description is almost always the primary subject — that signal beats first-position matches of generic words. Auto-tagger now takes the top 2 description candidates.

Generic-word filter split into two sets:
• _CHAR_LEADING_GENERICS — structural words always stripped from the start of a phrase (The, A, Featured, Shown, etc.).
• _CHAR_GENERIC_ALONE — words too generic to be a tag by themselves but valid as part of a multi-word phrase (Disney, Walt, Mickey). Pre-v0.15.55 “Mickey” was in the leading set so “Mickey Mouse” got stripped to just “Mouse”. Now “Mickey Mouse”, “Hidden Mickey”, “Walt Disney” all pass; “Mickey” alone is still filtered.

What I deliberately didn't do: handle lowercase subject nouns. Drew's “strawberry jam” pick can't be extracted from this description by heuristics — it's lowercase in the source text. That needs a character/object dictionary, which is item 9 in the open backlog (“Character dictionary for auto-tag, ~2-3h, biggest quality jump”) and a bigger lift than this changelog entry warrants. The heuristic-only rewrite shipped here handles the capitalized-subject case (which is the more common AI-description pattern); the dictionary would layer on top and catch the rest.

v0.15.54

2026-05-18

Fake-corpus gallery handling + desktop More yellow dot.

1. Reddit posts missing from corpus after scrape. Drew checked /r/ismydisneypinfake against his /fake-corpus page and found 4 reddit posts newer than his latest cached row (“Tinkerbell collection” from May 17) that weren't appearing in the corpus, even after multiple scrape runs. The scrape kept reporting added=0, skipped=100.

Diagnosis: those new posts were Reddit gallery posts (is_gallery: true) and the pre-v0.15.54 image extractor only handled (a) direct image URLs in cd.url and (b) preview.images[0].source.url. Galleries on modern Reddit use a different structure — media_metadata keyed by image ID, with the gallery ordering in gallery_data.items[].media_id. Some galleries don't populate the legacy preview field at all, so they silently fell through to the “skip” branch.

Rewrote the image-extraction step as a 4-tier fallback: (1) direct image URL, (2) gallery via media_metadata + gallery_data, (3) preview.images[0] (legacy), (4) thumbnail as last resort. For galleries, the first image in gallery order is used (preserves the “single-image-per-corpus-row” invariant).

Also added per-post terminal logging so Drew can see exactly what each scrape did. Pre-v0.15.54 the scrape was almost entirely silent — only the aggregate counter in the UI showed activity, and skips gave no reason. Now every add prints [fake-corpus] +abc123 (real, 0.85): 'Title' and every skip prints [fake-corpus] skip abc123 (gallery): no image extractable from 'Title'.

2. Desktop More button now shows yellow soft-validation dot. Mirrors the mobile bottom-tab-bar pattern from v0.15.49. Pre-v0.15.54 the desktop “...” (More) summary showed only the red dot for hard-validation pins; Drew couldn't see at a glance that there were no-series-assigned pins waiting on /validation. Now yellow + red both render side-by-side when both have counts, with yellow on the left so the colors don't bleed into each other.

v0.15.53

2026-05-18

PTDB parser rewrite for the new /pin/{N} page structure + catalog crawler Enable-sources layout fix.

1. PTDB parser: name='View Pin:' for every pin + hero image is a 2 KB UI icon. Drew's v0.15.52 test batch at ID 100000-100004 returned 200 OK on the corrected /pin/{N} URL but the data was bogus: every pin's name was the literal string “View Pin:” and the hero image was 2 KB of a sign-in CTA icon rather than the actual pin photo.

Root cause: PTDB's new pin pages have og:title set to a static template token (just “View Pin:”, no pin-specific data) and og:image pointing at a UI element rather than the pin photo. The pre-v0.15.53 parser trusted both blindly and got identical-looking junk data for every pin. Confirmed via the search-result snippet from the v0.15.52 investigation: the <title> tag does contain the real name in the form “View Pin: {actual name} - Pin Trading Database” (e.g. /pin/22107 → “View Pin: SDCC 2018 - 4 Pin Fantastic Four Set - Pin Trading Database”).

Parser changes:
• Name: prefer <title> tag over og:title. Strip the “View Pin:” prefix and the “- Pin Trading Database” suffix. Skip og:title entirely if it's just the static template token. Fall back to <h1> as last resort.
• Hero image: don't trust og:image blindly. Filter through the same UI-pattern blocklist used for body images (now expanded to include “signin”, “sign-in”, “login”, “default”, “placeholder”, and a few others). Rank body images higher than og:image — only use og:image as final fallback.
• Body images: relaxed the host filter (don't require “pintradingdb.com” in src) since PTDB may serve pin images via a CDN. The UI-pattern blocklist + image-extension check do the filtering.

How to test: reset PTDB progress so the broken v0.15.52 cache doesn't persist, run a fresh Test batch with Force refresh checked at IDs 22107, 51589, or 23431 (all confirmed to exist via the v0.15.52 web search). Each should come back with a real pin name (no “View Pin:” prefix) and a non-trivially-sized image. If still broken, parser is speculative without the live HTML — share a cached file from .ptdb_cache/ and I'll tighten the selectors against it.

2. Enable-sources layout busted. Drew's screenshot showed PinPics + start of Pin&Pop on row 1, PTDB alone on row 2, Start button on row 3 — broken wrap behavior. Root cause: fieldset was display: inline-block with inline-flex labels inside, which flowed unpredictably when total width exceeded the viewport. Switched fieldset to display: flex; flex-wrap: wrap with explicit row + column gaps. Each label is now an atomic flex item that wraps as a unit; layout stays consistent across viewport widths.

v0.15.52

2026-05-18

Two real bugs from Drew's first overnight run.

1. PTDB URL pattern is wrong. Drew tested PTDB at start_id=100000 (per v0.15.51's suggested dead-zone diagnosis) and got 404s on every fetch. Verified via live web search that PTDB's URL format changed at some point from pin.php?id={N} (query string) to /pin/{N} (path). Old format now genuinely 404s for every ID; new format serves real pin pages. Confirmed three live examples mid-investigation: /pin/22107, /pin/51589, /pin/23431, all return valid pin pages. Updated ptdb_sites.fetch_ptdb URL construction accordingly. The HTML parser doesn't need changes — the rendered page structure is the same regardless of which URL pattern got it there.

2. Fake-corpus reload loop. Drew loaded /fake-corpus on his phone and watched it auto-reload every ~5 seconds indefinitely. Pattern in the access log: GET /fake-corpus, two polls of /fake-corpus/scrape-status at 2s intervals, GET /fake-corpus again (reload), two more polls, reload again, etc.

Root cause is structural rather than a single bug: the live-progress JS called location.reload() on four conditions — completion, stale, error, and !data.running. Reloading from inside a poll loop is fragile by design. Any inconsistency between page-render-time state (which determines whether the polling script even renders) and poll-response state (which determines whether reload fires) produces continuous reloads. Tried for ~45 minutes to derive the exact trigger by hand and couldn't pin it down — which is the point: even if I'd found the exact race, the pattern is too easy to re-break.

Replaced the four reload paths with in-place DOM updates via a new settle(state) function. On completion/error/stale/gone, the banner's border color and header text update without a navigation, polling stops, and a small “Refresh the page to see new posts in the grid below” hint appears. The reload loop can't happen anymore because no JS-driven navigation exists in the polling path. User has to manually refresh to see new rows render in the grid below the banner.

v0.15.51

2026-05-18

Catalog crawler: Jump-to-ID + dead-zone hint. Drew started his first real PTDB crawl with sources cap'd at 25 each. Results: PinPics 25/25 cached (every fetch needed the headless-browser Cloudflare fallback — slow but working), Pin&Pop 25/25 cached (perfect, ~0.6s per fetch, names looked great), PTDB 0 cached, 25 consecutive 404s. The source then auto-disabled itself when the run cap was reached, with no inline signal that the diagnosis was “wrong starting ID range” rather than “PTDB is broken.”

Root cause is simple: PTDB's pin.php ID space doesn't start at 1. https://www.pintradingdb.com/pin.php?id=1 through 25 all genuinely return 404 (verified the URL construction in ptdb_sites.fetch_ptdb is correct — the request is built fine, the resource just doesn't exist at those IDs). Real PTDB pins live at much higher IDs; the test-batch that worked 5/5 in v0.15.35 was presumably run at known-good IDs in the 10k-100k+ range. The production crawler started at current_id=1 and would have ticked through hundreds or thousands of 404s at 5s/req before reaching any real pins.

Two changes:

1. New ‘Jump to ID’ form on each source card. Sits next to Reset progress. Enter a new ID, click Jump, the source's current_id moves forward (or back) without resetting any other state — caches survive, run-cap counter survives, rate limit survives, observed-throughput stats survive. New helper catalog_crawler.set_current_id(source, new_id) and route POST /catalog-crawler/jump-to.

2. Dead-zone hint inline on each card. When a source has &geq;10 not-found and zero successful fetches, a warning banner appears between its stats grid and the controls: “{N} consecutive 404s, 0 cached. This source's ID space probably doesn't start where the crawler is looking. Run a Test batch above at higher starting IDs (try 10000, 50000, 100000) until you find real pin data, then use Jump to ID below.” Self-clearing — once the first real pin caches, the banner disappears.

Workflow for Drew next time he runs: Stop the crawler. Use Test batch on PTDB with start_id=10000, then 50000, then 100000, etc., until results come back with names. Use Jump to ID on the PTDB card to set current_id to wherever real pins start. Restart the crawler. PinPics and Pin&Pop don't need this since their data starts at 1.

v0.15.50

2026-05-18

Catalog source filter pills broke on mobile. The Source radio cluster on /catalog/browse rendered fine on desktop — Source: ◯ all (10) ◯ PinPics (0) ◯ Pin&Pop (10) ◯ PTDB (0), all on one line. On Drew's phone the count chips wrapped to a separate row above each radio button, producing a double-stacked visual that read like the counts belonged to the WRONG radio.

Root cause: each label was display: inline-flex with three children — the radio input, the bare-text label, and a <small> count. On narrow viewports, Pico CSS's default <small> rules combined with the anonymous text node between the input and small caused the inner flex layout to break awkwardly — the small dropped to its own visual row.

Fix: three changes on the label. (1) white-space: nowrap so the radio + label + count stay on one row no matter how cramped the viewport. (2) Replaced the bare-text label and <small> with explicit <span>s so anonymous text nodes can't produce unexpected wrap points. (3) The outer fieldset is now display: flex; flex-wrap: wrap instead of inline-flex, so the individual labels are the wrap units (not their inner elements) — if the row genuinely doesn't fit, labels wrap as whole pills to the next row, preserving the radio ↔ count association.

v0.15.49

2026-05-18

Mobile More-tab badges + validation layout + catalog ‘no embedding’ explainer. Four loosely-related papercuts in one pass.

1. Mobile More-sheet badges overlapped. The Validation tile in the mobile More bottom-sheet has two badges: red .dpb-nav-count for Quick-Add pins needing identification, yellow .dpb-nav-count-soft for soft-triage (no series match / no series assigned). Both were absolutely-positioned at top:6px / right:8px, so when both rendered, the yellow one painted over the red one and Drew thought the red badge had disappeared entirely. Fixed: red stays top-right (matches desktop convention; urgent → attention-grabbing corner), yellow moves to top-left. Distinct, visible side-by-side.

2. Mobile More-tab now shows both color dots, not just red. Same logic: the bottom tab-bar More entry had a red dot tied to validation_count() and nothing for the soft count. Added a yellow companion dot at right:24px so when both signals are non-zero, you see two distinct dots stacked horizontally next to the tab. Desktop has the same intent (red dot + yellow chip in the dropdown); mobile now matches.

3. Validation page on mobile was overpacked. The primary .val-grid uses minmax(220px, 1fr) — on a 360px viewport that produces one huge square card per row with the 1:1 thumb taking ~half the screen height before any metadata is visible. New @media (max-width: 640px) rule collapses the grid to a row-layout: 72px side thumbnail, body text right, ~6 pins per screen instead of ~1.5. Matches the secondary-section layout already used further down the page.

4. ‘no embedding’ on /catalog/browse now self-explains, plus a backfill button. Drew asked what the ‘⚠ no embedding’ badge under each catalog card meant. Two changes: (a) the badge now has a hover-title that explains the badge meaning (“no DINOv2 visual-search vector stored”) and directs to the backfill button; (b) a new top-of-page ‘Catalog health’ box surfaces n_missing_embedding and n_missing_image_path counts with one-click backfill buttons for each. The image-path backfill button was already implemented at /catalog (collapsed inside a <details>); it's now duplicated into the browse page where Drew was actually looking at the problem rows.

New helpers: catalog_ingest.count_catalog_missing_embeddings(s) and backfill_catalog_embeddings(s, limit=200) mirror the fake-corpus pair shipped in v0.15.44/46. Catalog rows store embeddings as raw bytes (LargeBinary), not JSON, so the storage shapes need separate readers/writers — hence two helpers instead of one. New route POST /catalog/backfill-embeddings wraps the loop with the same redirect-with-flash pattern used by /fake-corpus/backfill-embeddings.

v0.15.48

2026-05-18

Four bugs in one pass: Lens icon, broken Lens URLs, dead Start button on catalog crawler, sort dropdowns.

1. Lens button now has a magnifying-glass icon. v0.15.9 dropped the 🔎 emoji because color emojis rendered with intrinsic ascent/descent that made the button taller than its neighbors. Re-adding it as a Lucide-style SVG instead — same line weight as every other icon in the app, no vertical jitter. Lives in pin_detail.html and scan_new_pin.html.

2. Lens click was opening a broken Google search page with ‘No image at the URL’. The Lens button was pointing at {base}/uploads/{fn}, which on Drew's deployment sits behind Cloudflare Access. Google Lens followed the URL, got bounced to the CF login page, saw no image, and gave up. Fixed by routing Lens through the existing signed /i/<token> URL — the same CF-Access-bypass mechanism SerpAPI and Copyseeker already use. New template helper lens_url_for(request, filename) wraps the signing call. Falls back to the legacy unsigned URL only if signing raises — better to try the old URL than 500 the page.

3. Catalog crawler Start button looked like it worked but didn't crawl anything. The Start form has three checkboxes (name="sources"), and the route's sources: list[str] param defaults to []. On a fresh install (no prior run) all boxes rendered unchecked, so users would set rates + caps, hit Start, and post sources=[]. The server flipped status to running, set every source enabled=False, and flashed ‘Crawler started’ — while no threads were actually crawling anything.

Two changes: (a) template now defaults all three boxes checked on first run (any non-running state, since summary.{source}.enabled would be False at that point); (b) the /catalog-crawler/start route now rejects empty source lists with a flash redirect (‘Pick at least one source to crawl’) instead of silently no-opping.

4. Sort dropdowns weren't working on mobile. Drew reported the selection changed visually but the list stayed A-Z, on /pins, /series, and /collections. Could not reproduce server-side — TestClient hits to /collections?sort=name_desc correctly returned the folders in reverse order. The probable culprit was the inline-onchange IIFE pattern: onchange="(function(s){...})(this)". Any CSP rejection, mobile-Chrome quirk, or global JS error would silently break it without a console trace, because there's no error boundary.

Defensive rewrite: the dropdown is now wrapped in a real <form method="get"> with onchange="this.form.submit()". Works as long as forms work, which is a much smaller failure surface than inline-handler IIFEs. Hidden Apply button shows as a fallback if even onchange is blocked. The current ?q= search filter is preserved via a hidden input so sorting doesn't reset your search.

v0.15.47

2026-05-18

**/activity was 500ing on every load: no such column: activity_entries.client_ip. Drew opened the page and got the SQLAlchemy OperationalError quoted in the subject line.

Root cause: in v0.15.33 the model models.ActivityEntry (table name activity_entries) got three new columns — client_ip, user_agent, cf_country — for capturing real client IP through the Cloudflare tunnel. The matching three lines were added to db._LATE_ADDITIONS, but the table name was typed as "activity_log" instead of "activity_entries".

_ensure_columns silently skipped unknown tables (if table not in existing_tables: continue), so the migrations were a no-op for four versions. SQLAlchemy happily generated SELECTs that named the new columns; SQLite rejected them. Hadn't crashed until now because /activity wasn't loaded since the column was added.

Fixes:
1. Migration table name corrected to activity_entries. Next server start will run ALTER TABLE for the three columns and log [db] added column activity_entries.client_ip (etc.).
2. _ensure_columns now collects unknown-table references and emits a single warning at the end: [db] WARNING: _LATE_ADDITIONS references unknown table(s) [...]. Migrations for these were skipped. Check models.__tablename__ for typos against the migration list. If a future migration row references the wrong table name, this is loud at boot instead of silent until the next 500.

Regression test: test_late_additions_tables_all_exist walks every (table, column, type) tuple in _LATE_ADDITIONS and asserts the table exists in the freshly-built schema. The same typo can't pass review again.

v0.15.46

2026-05-18

Root cause of the ‘not indexed’ mystery: the embeddings call was wrong from day one. Drew clicked the v0.15.44 backfill button and watched it log module 'embeddings' has no attribute 'image_embedding' 79 times in a row, then redirect with ‘Embedded 0 posts (79 failed)’.

The real embeddings module exposes embed_bytes(image_bytes) -> bytes and deserialize(blob) -> ndarray. There has never been an image_embedding(path) function. But three call sites in fake_corpus.py (scrape_one_page, backfill_embeddings, and match_visually) all called the imaginary API. Each was wrapped in a try/except that logged-and-continued, so every scrape silently produced zero embeddings, and /new-pin reverse-lookup against the corpus silently returned zero matches. The corpus has been dead weight since v0.15.23.

Fix: new _image_path_to_vec(path) helper in fake_corpus.py that reads the file, runs embed_bytes, deserializes the blob into a numpy array, and returns .tolist() — JSON-serializable Python floats compatible with the existing embedding_json column format. All three call sites route through it. The next backfill click should report ‘Embedded N posts’.

Regression guard: new test test_image_path_to_vec_uses_real_embeddings_api pins the helper to the actual module signature. If the embeddings module API ever drifts, the test fails loudly instead of the corpus silently rotting.

Also: the imagined-API issue means the count of “not indexed” posts on /fake-corpus was always the entire corpus — not a backlog of old rows. New scrapes will index correctly from this point forward.

v0.15.45

2026-05-18

Hotfix: /fake-corpus was 500ing on every page load. The v0.15.44 backfill route shipped two new helpers in fake_corpus.py — backfill_embeddings and count_missing_embeddings — both of which referenced models.FakeReferencePost at the top of the function body. Problem: models is not imported at module scope in fake_corpus.py; it's imported locally inside scrape_one_page and match_visually to avoid a circular import. Both new functions skipped that local import.

Result: every render of /fake-corpus called count_missing_embeddings for the badge, which raised NameError: name 'models' is not defined and bubbled to 500. Fixed by adding the same import models line both functions needed.

Also: Relabel form CSS mismatch. The <select> next to the dp-btn-sm ‘Relabel’ button was inheriting Pico's default ~2.5rem form-element height while the button was locked to 1.65rem. The button looked squished and wrong sitting under a tall select. Now the select is explicitly 1.65rem tall with matching border radius and Pico form-element vars for theme-correct colors. The two elements render as a clean pair.

v0.15.44

2026-05-18

Fake-corpus embedding backfill. All 83 existing posts in Drew's corpus showed ‘not indexed’ — they have no CLIP embedding so they don't participate in /new-pin visual-match lookup. Reasons: pre-existing rows from old scrapes never had embeddings, or embedding compute failed at scrape time. Without embeddings, the corpus is dead weight: visible but not searchable.

New backfill button at top of /fake-corpus, labeled ‘Backfill embeddings — N not indexed’. Click it to walk rows where embedding_json is null and compute on the spot. Synchronous, caps at 200 per click (roughly 5-15 minutes depending on hardware). Re-click to process more. Only shows when there's work to do; disappears at 0.

Two new module helpers: backfill_embeddings(s) and count_missing_embeddings(s). The page route calls the count helper on every render so the badge stays accurate.

Suite still 163 passing.

v0.15.43

2026-05-18

Catalog mirror self-heal + Reddit retry-with-backoff. Two follow-up issues from v0.15.42 deploy.

1. Mirror badge showed ‘no img’ even with image on disk. Drew's v0.15.42 hotfix added the hero_local_path attribute to the model so /new-pin stopped crashing, but didn't fix existing rows — every row upserted pre-v0.15.42 had the buggy code set row.hero_local_path on a phantom Python attribute that SQLAlchemy never persisted to the DB. Net result: image files exist on disk, JSON cache has the path, but the DB column is NULL. Badge falls through the elif to ‘no img’.

Fix — two paths:
• Self-heal on lookup: _lookup_catalog_mirror now checks disk when the DB column is NULL, and patches the row if the expected sharded file exists. Means any row Drew looks at gets fixed on first hit, no manual intervention. Cheap (stat() per lookup that needs it). Idempotent.
• Bulk backfill button at /catalog (“Backfill image paths” expandable section). Scans every NULL-path row, patches from disk in one pass. Useful for fixing the whole table at once instead of waiting for individual lookups.

2. Reddit HTTP 500 surfaced as hard error. Drew's screenshot showed ‘Scrape failed: HTTP 500 from Reddit’. That's Reddit being flaky, not our bug — and the v0.15.41 error surfacing was working as designed. But 5xx errors clear on retry the vast majority of the time, so failing on the first try is too aggressive.

Fix: fake_corpus.fetch_listing now retries transient 5xx (500/502/503/504) and 429 up to 3 times with exponential backoff (5s, 15s, 45s). Only returns _error if all 4 attempts fail. 4xx other than 429 don't retry — real client errors don't recover on retry. Net effect: most transient Reddit hiccups are invisible to the user; only persistent issues surface.

Test coverage: 7 new tests covering disk-path resolution across all three source ID types, no-file handling, no-ID handling, Reddit 5xx retry exhaustion, 5xx-then-200 recovery, 4xx no-retry. Suite now 163 tests, all passing.

v0.15.42

2026-05-18

Hotfix: /new-pin AttributeError on catalog mirror match. Drew hit a 500 on /new-pin: 'CatalogPin' object has no attribute 'hero_local_path'. Root cause: v0.15.33 added hero_local_path and all_image_paths_json columns to the catalog_pins table via the _LATE_ADDITIONS migration in db.py, but never added the corresponding Column() definitions to the CatalogPin class in models.py. SQLAlchemy needs both: the migration creates the column on disk, the model attribute lets the ORM load it. The bug was latent until v0.15.33's _lookup_catalog_mirror helper started reading those attributes during page render.

Fix: added the missing model-class columns. Also added a regression test (test_catalog_pin_model_has_late_addition_columns) that asserts every column in _LATE_ADDITIONS for catalog_pins also exists on the model class, so this whole category of bug fails CI rather than waiting for a user to trigger it. Same check for ActivityEntry. Suite now 156 tests, all passing.

v0.15.41

2026-05-18

Per-run request cap (Drew's ask) + fake-corpus 503 surfacing + scheduler releases-sync bug fix.

1. Per-run request cap. Each catalog source now has an optional max_requests_per_run field next to its rate input on the Start form. Blank or 0 = unlimited (default behavior). Set to e.g. 100 to limit this run to 100 requests for that source. Counter resets on every Start. When cap is reached, the source disables itself with reason ‘reached max_requests_per_run (N/M)’; UI shows a yellow banner with a ‘Reset cap & resume’ button to continue from where it stopped. Each source has its own counter (crawls don't interfere). Includes a progress bar showing ‘X / N this run’ while accumulating.

Useful for staging incremental crawls: cap PinPics at 500 and let it run, evaluate quality, then either reset and continue or pause indefinitely. Avoids committing to the full 250k pin space without seeing what comes back first.

2. Fake-corpus surfaces 503/429 as errors. Drew's screenshot showed ‘Scrape complete’ (green check) when in reality Reddit returned 503 and the scrape silently exited with 0 results. Bug: the listing fetch returned {} on any HTTP error, and the scrape loop treated empty result as ‘end of pagination, done’. Fix: fetch_listing now returns {"_error": "HTTP 503"} on non-200, and scrape_one_page propagates that into its return. Background runner detects the error and calls end_running(error=...) so the UI shows a red ‘Scrape failed’ banner with the actual cause.

3. Scheduler releases-sync bug fix. Your uvicorn log line [scheduler] releases sync failed: module 'release_feed' has no attribute 'sync_now' — my v0.15.36 scheduler called a function name that doesn't exist. Actual name is sync_disney_pins_blog(session). Now fixed; the auto-sync layer can actually fire releases.

Test coverage: 5 new tests covering start() with caps, reset_cap behavior, summary exposure. Suite now 154 tests, all passing.

v0.15.40

2026-05-18

Cache-hit image backfill + force-refresh option. Drew's v0.15.39 re-test showed PinPics still at 0/5 images. Looking at the uvicorn log: every result was a cache hit (ok=True). The v0.15.37 Referer/UA fix only runs on fresh fetches; the 14-day cache TTL meant tests never reached the new code on already-fetched IDs. Cache stored ok=True from earlier runs (before image download existed) but no hero_local_path, so test results showed ‘no img’ forever.

Fix: on every cache hit, backfill missing images. New _backfill_image_if_missing helper. On a cache hit, if hero_local_path is absent OR the file on disk is missing/empty, re-runs the download path (using the v0.15.37 headers) against the cached image_urls and updates the cache. Idempotent. Wired into the cache-hit branches of fetch_pinpics, fetch_pinandpop, and fetch_ptdb. Means existing caches self-heal on the next access without losing the metadata that was already parsed.

Plus: ‘Force refresh’ checkbox in the test-batch UI as an escape hatch. When checked, the test bypasses cache entirely and re-fetches the source HTML. Useful after parser/downloader changes when you want to exercise the whole path end-to-end. test_batch accepts force_refresh param; threads through to fetch_pinpics(force=True) etc.

How to verify: redeploy, visit /catalog-crawler, run PinPics test batch (162527, count 5). On the cache hit, the backfill should trigger and download the 5 images from the previously-cached URLs. Console will log [catalog_sites] pinpics 162527: cache hit but no local image — backfilling download. After a few seconds the result UI should show all 5 with thumbnails. If for some reason backfill also fails (parser stored the wrong URL pattern, etc.), check Force refresh and re-run — that re-fetches HTML through v0.15.37's code.

Test coverage: 4 new tests in test_catalog_mirror.py — backfill downloads on missing path, no-ops when present, no-ops on no image_urls, re-downloads when on-disk file deleted. Suite now 149 tests, all passing.

v0.15.39

2026-05-18

Catalog browse UI with filters + pagination. The CatalogPin mirror is now searchable through a real browse interface, not just the bare 60-item list from before. Builds out the payoff for all the v0.15.32+ mirror plumbing.

1. /catalog/browse rewrite. New query params: q (name/series substring search), source (filter to rows with pinpics/pinandpop/ptdb ID), year, has_image (only rows with downloaded images), sort (newest/oldest/name/year_desc/year_asc), page. 48 items per page. Result count + pagination at bottom. Per-source counts shown inline in the radio button row so you can see crawl progress at a glance.

2. /catalog/pin/{id} detail view. Click any browse card to land on a dedicated detail page for that catalog row. Shows hero image + any extra downloaded images (when all-images mode was used), all three source IDs with links out, eBay sold price (if computed), fake warning (if known), crawler metadata (when fetched, embedding status). Distinct from /pins/{id} which is for OWNED pins — the catalog detail is reference-only.

3. Nav entry. Catalog browse now in the More menu (desktop + mobile).

Once the crawler runs for a few hours you'll have tens of thousands of pins here. Filters make this scale; a flat list wouldn't.

v0.15.38

2026-05-18

PTDB promoted from test-batch-only to main Start form. After the v0.15.37 fix and v0.15.35 test-batch came back 5/5 working with correct names, years, and full images, PTDB is ready for unattended crawl. New PTDB checkbox + rate input in the Start form on /catalog-crawler. Default rate 4s/req (slower than the others) since PTDB needs the headless-browser fallback for many pages. /catalog-crawler/start route now accepts ptdb_rate form param.

v0.15.37

2026-05-18

Image-download headers fix. Test-batch results in v0.15.36 showed PinPics returning ‘2 images found’ in metadata but downloading 0 of them — classic hotlink-protection symptom. Module docstring even literally warns ‘PinPics blocks bot traffic’, and the download helper was sending User-Agent: DisPindable/0.15, which is exactly the kind of bot-shaped UA that gets blocked. Embarrassing oversight.

Fix: new _download_headers helper sends real-Chrome UA + Accept: image/* + Referer pointing back at the source domain. Used by both _download_hero_image and _download_all_images. PinPics hotlink check now sees a same-origin browser request and serves the JPEG.

Pin&Pop and PTDB were already mostly working (4/5 and 5/5 in tests) but they get the better headers too — consistency wins. PTDB CDN previously responded to anything; PinPics is the strict one. Pin&Pop's lone failed download (Dewey #91559) is likely a parser issue specific to that pin's page structure; the new diagnostic logging will show the exact URL it tries to fetch so we can fix the parser next time it recurs.

Test it: visit /catalog-crawler → Test batch → PinPics, start ID 162527, count 5. Should now show all 5 with thumbnails + ~50-200KB hero sizes (instead of ‘no img’ rows). If any still fail, the uvicorn console now logs the HTTP status + exact URL for diagnosis.

v0.15.36

2026-05-18

Background scheduler + fake-corpus lookup test page. Two follow-ups to v0.15.35: makes auto-sync truly unattended (no longer requires a page visit) and gives Drew a standalone test surface to verify the fake-corpus pipeline is actually doing what it claims.

1. Background scheduler. New scheduler.py module. Single daemon thread started at app startup, wakes every 30 minutes, checks each subsystem's staleness and fires the appropriate background sync. Currently watches fake-corpus and releases (both keyed to a 6-hour staleness window). Idempotent start(), defensive against failures (one check throwing doesn't kill the loop), honors per-feature app_settings flags so each can be disabled. State persists to .scheduler_state.json with last-check timestamp + per-feature last-fire timestamps.

Behavior summary: the lazy-on-visit auto-sync from v0.15.35 still works (fires immediately when you visit a stale page), AND the scheduler now also fires unattended. Together they ensure data is at most 6h stale whether or not anyone's actively using the app. Disable via app_settings.scheduler_enabled = false; disable individual features via fake_corpus_auto_sync = false or releases_auto_sync = false.

2. Fake-corpus lookup test page. New /fake-corpus/lookup route. Upload any pin image; the server runs it through the same visual-similarity pipeline used by /new-pin and /pin-detail (CLIP embedding + cosine sim against FakeReferencePost rows), renders matches with verdict badges + similarity %. min_similarity=0.5 is more permissive than the production 0.7 default — this is for exploration, show partial matches too. Banner at top of results summarizes verdict majority: red if most matches are fakes, green if most are real. Links to each Reddit thread for context.

3. Prominent ‘Test corpus lookup’ button on /fake-corpus, plus a scheduler-state hint line that shows when the last auto-sync fired. Makes both new features discoverable without menu spelunking.

Where fake-corpus matches surface organically (reminder, unchanged): the /scan/{id}/new-pin page shows “visually-similar posts” for the scanned image; pin detail page (/pins/{id}) shows the same for any saved pin. So Drew also sees matches automatically whenever he scans or views a pin that happens to resemble something in the corpus. The new lookup page is for explicit verification + spot-checking without needing a matching pin in your collection.

Test coverage: 9 new tests in test_scheduler.py covering idempotent start, is_running reflection, fire/last-check state persistence, and that disabled app_settings flags short-circuit. Suite now 142 tests, all passing.

v0.15.35

2026-05-18

Catalog-crawler live-progress UI + fake-corpus auto-sync. Two related observability wins. Also includes the PTDB scaffolding (DB column, module, test-batch integration) from the previous WIP — PTDB is now usable via the test-batch UI but not yet wired into the main Start form, intentionally, until the parser is validated against real PTDB HTML.

1. Catalog-crawler live progress. Mirrors the fake-corpus poll pattern from v0.15.34 onto /catalog-crawler. New /catalog-crawler/status JSON endpoint returns the same summary the page uses. JS poller runs every 5s while status=running, rewriting per-source counters, progress bars, and last-fetched info in place — no page reload needed. Status flipping to paused/stopped triggers a one-time reload so the UI redraws cleanly.

2. ‘Last fetched’ line per source. Crawler loop now captures last_pin_id, last_pin_name, and last_fetched_at on every successful fetch. Surfaced in a compact strip above the stats grid: ‘Last fetched: <name> #<id>’. Same visual cue as fake-corpus's “Currently scanning” line — tells you the crawler is actively making progress rather than appearing frozen.

3. Fake-corpus auto-sync on visit. Same pattern as /releases: visiting /fake-corpus fires a one-page newest-mode scrape if the most recent run was >6 hours ago. Default on; toggleable via app_settings.fake_corpus_auto_sync (button surfaces in the auto-sync banner). Won't fire if a scrape is already in flight — the duplicate-scrape protection from v0.15.34 handles that. New helpers: fake_corpus.is_sync_stale() and mark_auto_sync_at().

4. Auto-sync banner. Green strip at top of /fake-corpus when auto-sync fires on this visit, with a “Disable auto-sync” button. Tells you why the scrape is running so it doesn't feel surprising. The live-progress section from v0.15.34 surfaces the scrape itself.

5. PTDB scaffolding (test-batch only). New ptdb_sites.py module with the standard fetch/parse/cache contract. URL pattern: pintradingdb.com/pin.php?id=N. PTDB returns 403 to bot UAs so the module falls back to headless_scrape.fetch_rendered when direct requests fail. HTML parser is a first-attempt selector set based on OpenGraph meta tags + dl/dt label patterns — expect to tweak after first test run. PTDB has its own pin ID space so the schema gains a new catalog_pins.ptdb_id column (nullable). PTDB available on the test-batch UI; main Start form does not expose it yet so the production crawler can't run PTDB unattended.

Test coverage: 10 new tests in test_v15_35.py — is_sync_stale across never-run / fresh / stale / running / completed cases, mark_auto_sync_at side effects, progress_summary surfaces last_pin_* fields on all three sources. Suite now 133 tests, all passing.

v0.15.34

2026-05-18

Live-progress UI for /fake-corpus scrapes. Drew had no visibility into in-flight scrapes — the POST blocked synchronously with no feedback, so multi-page backfills looked indistinguishable from a hung process. Architectural change: scrape now runs as a FastAPI BackgroundTask, with a live-progress UI on /fake-corpus that polls a status endpoint every 2 seconds.

1. Background task instead of blocking POST. /fake-corpus/scrape initializes running state synchronously (so the user sees the progress section immediately on the redirect), kicks off _run_fake_corpus_scrape_bg as a BackgroundTask, returns 303 to /fake-corpus. Background task uses its own DB session (the request-scoped one is closed by the time it fires).

2. Progress callback in scrape_one_page. Optional progress_callback kwarg fires after every post (added, skipped, dedup'd, or filtered) with cumulative counts and the current post title. Internal _emit() inline helper at every continue point so the UI doesn't freeze on pages full of gallery posts (all skipped, would've previously had no callback fire). Best-effort — callback exceptions don't break the scrape.

3. Live-progress state in .fake_corpus_state.json. New running sub-dict with started_at, mode, n_pages_requested, current_page, current_after, posts_processed, posts_added, posts_skipped, last_post_title, last_update_at, completed flag, optional error. New helpers: begin_running, update_running, end_running, clear_running, is_running_stale. Atomic file writes via tmp+rename.

4. Status endpoint at /fake-corpus/scrape-status. Returns {running: dict | null, stale: bool}. UI polls every 2s while not completed. Stale flag fires when last_update_at is > 90s old AND completed is not set — indicates probable background-task crash so the user gets a warning rather than infinite spinner.

5. Live UI on /fake-corpus. Renders when running exists in state. Shows: progress bar (page X/Y), mode badge, live counters (processed/added/skipped), italicized “Currently scanning: <post title>”, Reddit cursor in monospace. Pulsing dot next to header while in flight. JS poller updates figures in place every 2s; on completion or error, reloads once to render final state. Survives tab close + reopen — state persists in the file.

6. Duplicate-scrape protection. POST to /scrape now checks for an in-flight scrape and short-circuits if one's already running (unless stale, in which case the new one overrides). Prevents two concurrent scrapes from racing on the state file + DB session.

7. Dismiss button. On completion or failure, the UI offers a Dismiss button (POST to /fake-corpus/scrape-clear) to remove the running banner.

Test coverage: 12 new tests in test_scrape_progress.py — begin/update/end/clear cycle, stale detection at the boundary, completed-state-not-stale, defensive no-op when update fires without begin, lifetime-counter preservation. Suite now 123 tests, all passing in <6s.

v0.15.33

2026-05-18

Catalog mirror DB integration, multi-image download, activity log identifiers, PTDB direct URL, /new-pin local mirror badge. Also fixes a real HMAC bug in the image-signing module that's been shipping unstable URLs since v0.15.31.

1. Catalog mirror DB integration. catalog_sites.fetch_pinpics + fetch_pinandpop now upsert into the existing CatalogPin table on every successful fetch (via new _upsert_to_db bridge). The on-disk JSON cache is still the source of truth for raw scrape data; the DB table is the searchable index. Means every pin the crawler discovers becomes immediately queryable for the /new-pin mirror check and (later) for catalog browsing/search UI.

2. Multi-image download option. New _download_all_images helper + DOWNLOAD_ALL_IMAGES module flag. Checkbox in the test-batch UI (“Download all images (front + back + extras)”). When on, every image attached to a pin gets downloaded with an _N suffix ({id}.jpg, {id}_1.jpg, {id}_2.jpg...). Hero-only mode is unchanged (still the default for the production crawler — flip the checkbox in test batch when sizing disk budget). Test batch returns per-pin extra_images_downloaded count + correct cumulative disk usage.

3. Activity log identifiers. Now that the app is externally exposed via Cloudflare Tunnel, every activity entry can capture the requesting user's real IP, browser UA, and country. New activity_log.request_context(request) helper extracts CF-Connecting-IP (preferred — real client behind CF), X-Forwarded-For (fallback), and request.client.host (direct-LAN). Also pulls User-Agent + CF-IPCountry. Added as nullable columns on activity_entries. Migration via _LATE_ADDITIONS. The record() function accepts a new request= param — existing call sites still work (the fields stay null for background-task events), and call sites can opt in incrementally by passing request=request.

4. PTDB direct URL (no more Google). All four sites that linked to PinTradingDB previously went through Google site-search. Per Drew's request, switched to PTDB's own pinList.php?txtSearch= endpoint. One less click, lands on PTDB's own filtered list.

5. /new-pin local-mirror badge. When the AI/prefill resolves to a PinPics or Pin&Pop ID and that ID is already in our CatalogPin mirror, a compact green badge renders above the existing “You may already have this pin” section. Shows the cached name, year, IDs, eBay sold median (if known), and catalog-side fake warning (if any). New _lookup_catalog_mirror helper. Side-effect free.

6. HMAC token bug fix (critical). The v0.15.31 verify_token used raw.rsplit(b"|", 1) to split the decoded payload into message + signature. But the 16-byte HMAC signature can legitimately contain 0x7C (the | byte). When it does, the splitter lands in the wrong place and verification fails. Affected ~6% of generated tokens (rate the HMAC randomly contains a | byte). Symptom: occasional intermittent 404 from Google Lens / Copyseeker on the same scan that worked yesterday. Fix: parse the signature as the last 16 bytes by fixed length. Added a 200-iteration regression test so this can't recur silently.

Test coverage: 11 new tests in test_v15_33.py (all-images download paths + idempotency, toggle flag, activity_log request_context across all three IP source paths, PTDB direct URL). Plus the HMAC regression in test_image_signing.py. Suite now 111 tests, all passing in <6s.

v0.15.32

2026-05-18

Catalog mirror: local image download + pre-flight test batch. Drew wanted to build a complete local copy of PinPics and Pin&Pop (~150k pins, ~120k series) but reasonably wanted to verify the gather pipeline against a small slice before opening the floodgates. This release adds the missing piece (local image downloads) plus a synchronous test-batch UI so you can eyeball results before committing.

1. Local image mirror. catalog_sites.fetch_pinpics and fetch_pinandpop now download the hero image to catalog_uploads/{source}/{prefix}/{id}.jpg after a successful parse. Path is sharded by the first 3 digits of the pin ID to avoid 250k files in one directory. Idempotent — re-fetching an already-downloaded ID is a fast disk-check, no HTTP. Content-type validated (rejects HTML error pages disguised as 200 OK). Atomic via tmp + rename so a crash mid-download doesn't leave a half-written file. The relative path is stored in the JSON cache as hero_local_path so downstream consumers can find the local file.

2. Test batch UI. New “Test batch” panel on /catalog-crawler (collapsed by default; auto-expands after a run). Two side-by-side forms, one per source. Pick a starting ID + how many to fetch (1-20). Synchronous run — you see a spinner, then results inline with thumbnails, names, file sizes, classification (ok / 404 / blocked / errored), per-pin duration, total elapsed time, KB downloaded. Defaults: start at PP#162527 and P&P#91559 (both known-good recent IDs from Drew's prior scans).

3. Test batch isolation. Runs do NOT touch persisted crawler state. current_id stays where it was; counters don't update. Test results live in the request response only. Bounded at 20 pins so a fat-fingered count of 1000 can't self-DOS. Uses the configured rate-limit between requests so the test reflects what production will feel like.

How to use it: open /catalog-crawler → expand “Test batch” → run 5 pins from each source → check the thumbnails + names match the live sites → if everything looks right, scroll down and start the real crawl. Cost of the test: ~30 seconds and ~1-2 MB of disk.

Test coverage: 12 new tests in test_catalog_mirror.py — sharded path logic, download idempotency, content-type rejection, 404 handling (no partial file left), production-state isolation, count-cap enforcement, classification of all four result types, return-dict shape. Suite now 99 tests all passing.

v0.15.31

2026-05-18

Signed-image public URL bypass for Google Lens + Copyseeker (now that Cloudflare Tunnel is live). Drew got Cloudflare Tunnel set up so the app is reachable at https://dispindable.com, with Cloudflare Access providing email-PIN auth at the edge. Great for privacy — bad for Google Lens / Copyseeker, which need to fetch image URLs we hand them, and now hit the CF Access login page instead of the JPEG.

Solution: new image_signing.py module generates HMAC-signed, time-limited image URLs at /i/<token>. The token encodes filename + expiry timestamp + HMAC signature; default TTL 1 hour. Server-side verifies the HMAC + expiry + rejects path-traversal before serving the file. The /i/* path gets a one-time Bypass policy added in CF Access (instructions in chat) so external services skip the auth wall on that one route only.

Signing secret: 32 random bytes generated at first import, persisted to .image_signing_secret (gitignored, chmod 600 on Unix). Survives server restarts so previously-issued URLs keep working. Rotate by deleting the file — all outstanding tokens then become invalid.

PUBLIC_BASE_URL env var: new helper _public_base_url(request) prefers the PUBLIC_BASE_URL env value over request.base_url. This means you can be accessing the app via LAN (https://10.0.0.32:8000) and still hand Lens/Copyseeker a public Cloudflare URL — useful for testing without bouncing through Cloudflare every page load. Set PUBLIC_BASE_URL=https://dispindable.com in your environment.

Auto-authenticity now works externally too: the on-save hook used to skip Lens/Yandex/Copyseeker because it didn't have a Request object to call str(request.base_url) from. Now reads PUBLIC_BASE_URL directly — meaning new pin saves auto-fire all the URL-based providers if the env is set.

Test coverage: 9 tests in test_image_signing.py — sign/verify round-trip, expiry enforcement, tampered-token detection, garbage-input rejection, path-traversal rejection at both sign + verify time, URL format. Suite now 87 tests, all passing.

v0.15.30

2026-05-18

/fake-corpus/scrape: defensive error handling + flash display. Drew got a bare ‘Internal Server Error’ from POSTing /fake-corpus/scrape with no info to debug. Root cause: an uncaught exception somewhere in the scrape pipeline (most likely an unexpected Reddit response shape, image-download timeout, or the CLIP embeddings module failing to load) escaped both the per-page try and the outer route. Now wrapped in two layers of try/except:

1. Per-page exception — if one page raises, log the full traceback to server stderr, stop the loop, preserve totals from prior pages, and continue to the redirect. The flash message includes the first error's type + message so the user sees what broke without needing log access.

2. Outer route exception — catches anything from state-file writes, dependency imports, or anywhere else in the handler. Same pattern: log full trace, redirect with flash including the error class + first 200 chars of the message.

Flash messages now actually display on the /fake-corpus page. Previously they were passed via ?flash=... URL params but the template never rendered them — so success messages were invisible too. Green box for successes, red box for errors (auto-detected via ‘fail’ / ‘error’ keyword in the flash text).

What you'll see now instead of “Internal Server Error”: a redirect back to /fake-corpus with a banner like “Scrape failed: ConnectionError: HTTPSConnectionPool... Check server log for full trace.” The full traceback is in your uvicorn stderr.

v0.15.29

2026-05-18

Bulk catalog crawler for PinPics + Pin&Pop. Drew wanted to set up multi-day database scans of both big third-party catalogs to build a local mirror. New module catalog_crawler.py sequentially enumerates pin IDs through the existing catalog_sites.fetch_pinpics / fetch_pinandpop helpers (which already cache to .catalog_cache/ — meaning the cache is the corpus, no new DB tables needed).

Architecture: daemon thread per source, managed by this module. Threads sleep rate_limit_s between requests and read a shared state dict for stop/pause signals. State persists to .catalog_crawler_state.json after every fetch (atomic via tmp + rename, threading.Lock-protected). On FastAPI startup, ensure_crawler_running() checks the state file and auto-launches threads if status == 'running' — so server restarts pick up where they left off.

Auto-pause on blocks: after 10 consecutive blocked responses (Cloudflare challenge / 429 / 403), the crawler auto-pauses that source for 30 minutes before retrying. Prevents a single bad VPN endpoint or temporary anti-bot wave from cascading into a full IP ban.

Admin page at /catalog-crawler: Start / Pause / Resume / Stop controls, per-source progress bars (with rate slider), live stats (cached / 404 / blocked / errored counters), observed-rate-per-hour, ETA in days. Reset button per source for redoing a crawl — preserves the on-disk cache so re-crawled IDs return from disk in <10ms with no HTTP.

Default rates: 3.0s/req for PinPics (Cloudflare-protected, more conservative), 2.0s/req for Pin&Pop (less protected). Adjustable per source. At defaults, expect ~10 days for full PinPics, ~5 days for full Pin&Pop — longer if many auto-pauses fire.

Added to More menu (desktop dropdown + mobile sheet).

Test coverage: 12 tests in test_catalog_crawler.py covering state round-trip, partial-state backfill, start/pause/resume/stop transitions, rate override, single-source vs all reset, block / 404 classification, and progress summary math. Suite now 78 tests, all passing.

Not in this version: CLIP embeddings for crawled images. Once the cache fills up, v0.15.30+ will add the embedding pipeline + image-match search against the full crawled corpus (surfaced on /new-pin alongside the existing “you may already have this pin” panel).

v0.15.28

2026-05-18

Camera + barcode scanner torch / flashlight toggle. Drew asked whether the camera interaction could enable the phone's flashlight for low-light pin capture. Yes — added via the standard MediaStreamTrack.applyConstraints({advanced: [{torch: true}]}) API. Two modals updated:

1. Pin-photo camera modal (the “Camera” button on /scan and dropzone capture). Lightning-bolt button overlays the top-right corner of the live video feed. Tap to turn the torch on; the button glows yellow. Tap again to turn off. Constraint-toggle is async so a quick double-tap can race — logged, not user-facing.

2. Barcode/QR scanner modal (the “Scan barcode” button on /scan and pin detail). Full-width Flash button just below the video preview. Same constraint API. Resets back to “Flash off” on modal close so reopening doesn't show a stale on-state.

Browser support caveats: the button auto-hides if track.getCapabilities().torch isn't true, which means it won't appear on devices that don't support hardware torch control. The biggest gap is iOS Safari — Apple has never shipped a torch API, so iPhones (any model) won't see the button. Works fine on Chrome / Edge / Firefox / Samsung Internet on Android. On Windows desktop the button won't appear either (no torch hardware). Drew accesses primarily from a phone, so this should land where it matters.

v0.15.27

2026-05-18

Standalone PinTradingDB lookup section on /new-pin. Drew flagged that v0.15.26's PTDB pills were only visible inside the Set Members section — when set members were empty or collapsed, there was no surface area for PTDB at all. New top-level collapsible section, always renders if we have any candidate pin name (from AI suggestions or Pin&Pop scrape). Lives just after the Pin&Pop match section, just before the AI suggestions toggle — consistent with the “catalog lookups” reading order. Includes the search query in plain text so users can see exactly what we'd send to Google, plus a secondary “browse PTDB directly” link for browsing the full database. Section is collapsed by default since it's informational (no auto-match data to show); user expands when they want the search link.

Why no direct PTDB scraper yet: PTDB has its own pin numbering system that doesn't map from PinPics or Pin&Pop IDs, so we'd need a separate catalog_sites.fetch_ptdb() implementation (~3hr of work). On the backlog — will tackle when the Google-search-fallback pattern starts feeling like friction.

v0.15.26

2026-05-18

Set member catalog verification + image enrichment, “via xxx” attribution removed, PinTradingDB.com display pill.

1. Set member IDs now verified against actual catalog data. Root cause of Drew's “random data” bug: Gemini Flash (and to a lesser extent the other AI providers) hallucinate PinPics/Pin&Pop IDs for set members. The Villain Portraits set in Drew's screenshot rendered “Maleficent: PP#162533” etc, but PP#162533 was actually a different pin entirely (possibly Grumpy). New _verify_set_members_against_catalog helper: fetches each claimed PinPics + Pin&Pop ID, loose-matches the AI's claimed name against the catalog's real name. If mismatch, the ID is nulled out (name + character preserved so the member still surfaces, just without a broken link). If match, the catalog's canonical name + primary image are promoted onto the member. Runs in parallel via ThreadPoolExecutor (4 workers) so first-load latency is bounded by the slowest single fetch rather than the sum.

2. Set member thumbnails now actually display. Drew flagged this in v0.15.14 and again now. The verifier pass above also pulls each member's primary image from the PinPics or Pin&Pop catalog data and promotes it onto member.image_url. The template was already wired to render m.image_url as a 110px contain-fit thumbnail — the missing piece was that the AI providers rarely include an image URL in their set_members JSON, so the field was usually empty. Catalog enrichment fills the gap.

3. “via gemini-flash” attribution removed. The small grey provider-list at the bottom of each set member card was noise. The merged result either works or doesn't; users don't need to see which AI voted for which member. The sources field is still tracked on each member object for debugging if needed.

4. PinTradingDB.com display pill added. Drew wanted PTDB shown alongside PinPics/Pin&Pop but never used as a default source. PTDB has its own pin ID space that doesn't map from PinPics/Pin&Pop IDs, so we link via a Google site-restricted search by pin name (site:pintradingdb.com "Pin Name") rather than building a third catalog scraper. New _pintradingdb_search_url helper. Surfaces on both /new-pin set member cards and pin detail catalog links row.

Test coverage: 7 new tests for _verify_set_members_against_catalog + the PTDB URL helper. Covers: matching-name pulls image + upgrades to longer canonical name, mismatched-name drops the ID with diagnostic _dropped_pinpics_id key, catalog-fetch-failure leaves AI's ID intact (no signal ≠ bad signal), Pin&Pop path symmetric to PinPics, empty + no-IDs edge cases, URL builder format. First run caught a real bug — my _names_loose_match(claimed, "") check was always returning True via the can't-verify fallthrough, so the name-upgrade branch never fired. Tests paying for themselves again. Suite now 66 tests, all passing in <1s.

v0.15.25

2026-05-18

Auto-tag prefill bug fix, fake-corpus on /new-pin, nav menu additions, auto-tag test coverage. Four items from Drew's v0.15.24 review:

1. Auto-tag character extraction now runs at prefill time, not just save time. Drew flagged a Shan Yu scan where the description clearly read “Shan Yu from Mulan is depicted...” but the /new-pin tags field only showed villains-in-frames and 2024 — character/subject candidates were missing. Root cause: _extract_auto_tag_subjects() was only wired into the POST save handler (lines 12340 + 12468); the GET prefill at line 12201 only assembled tags from final_super_series, set_name, and year_estimate. So character tags only appeared AFTER clicking Save, which felt like a bug because the form showed empty tag candidates while the description clearly had a character name. Fix: mirror the extraction in the prefill block so users see candidates BEFORE save and can edit them in the form. Two-pass design means manual edits are preserved — the save-time auto-tag still dedupes against the existing comma list rather than overwriting.

2. r/ismydisneypinfake reverse-lookup on /new-pin (scan time, before save). Previously the fake-corpus lookup only fired on saved pin detail pages (/pins/{id}). Now also runs against the uploaded scan image on /new-pin so you see crowd verdicts BEFORE creating the pin record — useful for spotting likely-fake matches while still deciding what to do with the pin. New helper _compute_fake_corpus_matches_for_scan takes raw upload filenames from the Scan rather than PinImage rows. Surfaces between the active-AI badge and the AI provider cards; red border + warning text if any match is crowd-labeled FAKE.

3. /releases and /fake-corpus added to nav “more” menu. Both desktop dropdown and mobile sheet. Drew flagged that /releases had no menu entry — you had to know the URL. /fake-corpus same situation.

4. 12 new tests for _extract_auto_tag_subjects. Critical regression-risk path (the function silently broke in v0.15.8 and wasn't caught until v0.15.15) finally has formal coverage. Tests cover: description-based single + two-word name extraction, generic-starter rejection (“The pin shows...” should not yield “The”), name-split with venue prefix / year / generic-series-word filtering, both-None defensive behavior, and a specific regression test for Drew's Shan Yu screenshot. Required fixing the _extract_function_from_main conftest helper to handle multi-line def signatures + auto-inject referenced module-level constants (_CHAR_GENERIC_STARTERS, _NAME_VENUE_PREFIXES) by scanning the function source for _UPPER_SNAKE patterns and re-extracting matched constants. Total test suite now 59 tests, all passing in <1s.

Note on /new-releases vs /releases: these are two separate Disney Pin Blog scrapers in the codebase (/new-releases uses scrape_dpb.py, /releases uses release_feed.py added in v0.15.23). Functionally redundant. Worth consolidating in a future version — not done this release.

v0.15.24

2026-05-18

/releases auto-sync, button alignment fix, Reddit backfill cursor. Three improvements from Drew's v0.15.23 review:

1. /releases auto-syncs in the background on visit. Previously the page only refreshed when the user clicked “Sync now”. New lazy auto-sync: on each /releases visit, if the last sync is >4 hours old (or never), a FastAPI BackgroundTask fires a fresh sync_disney_pins_blog call. Page renders immediately with the existing items; a small “refreshing in background…” note appears next to the Last-Synced indicator. Reload after a few seconds to see new items. Avoids needing a separate cron / Windows Task Scheduler — the visits drive the refresh.
Sync timestamp + stats are persisted to .releases_sync_state.json so staleness checks are file-based (no DB roundtrip per visit). Manual “Sync now” still works for force-refresh.

2. Button alignment on /releases. “Sync now” and “Show seen” weren't vertically aligned — the <form> wrapping the Sync button added its own block-level height, while the anchor next to it didn't. Fixed with display: contents on the form, which makes it transparent to flex-layout while preserving the form's submit semantics. Both buttons now identical height + baseline.

3. r/ismydisneypinfake backfill mode with cursor persistence. Previously every scrape started from after=None (newest post), so running 10 pages today and 10 pages tomorrow re-fetched today's listings (DB-deduped, but wasted listing-API calls). New mode dropdown in /fake-corpus scrape control:<ul style='margin: 0.4rem 0 0.4rem 1.2rem;'><li>Newest (default) — starts from Reddit's most recent post, walks backward. Good for daily catchup. Does NOT touch the backfill cursor.</li><li>Backfill — resumes from the saved cursor (or starts from newest if no cursor exists). Saves the new cursor at end. Good for systematically working through deep Reddit history one chunk at a time.</li></ul>Cursor state lives in .fake_corpus_state.json with lifetime counters (total posts added via backfill, total pages run). UI surfaces the current cursor + last-run timestamp in the scrape control. New /fake-corpus/reset-backfill endpoint clears the cursor when Reddit's 1000-post pagination cap is hit (or when you want to start a fresh chunk).

Test coverage: 9 new tests in test_state_persistence.py — round-trip, accumulation, reset, corruption resilience for both state files. Total suite now 47 tests, all passing.

v0.15.23

2026-05-18

Big one. Three new capabilities in one release: pytest test suite, Disney Pin Blog release feed, and r/ismydisneypinfake fake-reference corpus with reverse-lookup on pin detail.

1. Pytest test suite — 38 tests across six modules covering: classify_domain (6 tests, Disney legit / AliExpress suspect / eBay neutral classification), _score_from_stats (7 tests, provenance score math incl. zero-signal and clamp behavior), _parse_catalog_url (6 tests, Pin&Pop+PinPics URL patterns), _classify_verdict (7 tests, comment-sentiment labeling incl. score weighting + mixed-thread + truncation), _strip_html/_first_image_from_html/_parse_pubdate (10 tests, RSS parsing helpers), and migration safety (2 tests, init_db idempotency + WAL pragma verification). Caught one real bug on first run — _strip_html didn't decode ' (WordPress uses it occasionally for apostrophes). Fixed.
Run with pytest tests/ from project root. New tests/conftest.py includes an _extract_function_from_main helper for testing main.py helpers without importing the module (it has heavy import-time side effects). Once main.py is refactored to push helpers into a utils.py module, that extractor disappears.

2. Disney Pin Blog release feed — new release_feed.py module scrapes the WordPress RSS at disneypinsblog.com/feed/ for new pin release announcements (typically posted days-to-weeks before pins hit PinPics/Pin&Pop catalogs). New page at /releases displays items chronologically as a card grid with hero images, summary, and timestamps. “Sync now” button triggers a fresh fetch; “Mark seen” lets you dismiss items you've already reviewed (defaults to hiding seen, toggle to show). New release_feed_items table with composite uniqueness on (source, source_guid) so re-fetching is idempotent. Parses RFC-822 pubDate, extracts first <img> for the hero, strips HTML for the summary.

3. r/ismydisneypinfake corpus + reverse-lookup — new fake_corpus.py module scrapes Reddit's public JSON endpoint (no OAuth needed for read-only) to build a labeled corpus of community-verdicted fake-pin posts. Each post gets: image downloaded locally (so the corpus survives Reddit CDN URL rotation), comment-sentiment-based auto-verdict (fake/real/uncertain with confidence score), and a CLIP-style image embedding for similarity search.
Admin page at /fake-corpus: filter by verdict, scrape control (configure pages crawled + whether to compute embeddings inline), manual relabel form for fixing miscategorized posts. Verdict labeling counts keyword hits (fake/scrapper/real/legit/etc.) in top comments weighted by score; threshold 2:1 to assign, else uncertain.
Reverse-lookup on pin detail: when viewing any pin with a front-view image, computes its embedding and nearest-neighbors it against the corpus. Matches above 70% cosine similarity surface in a new section with the verdict pill + thread link. If any match is crowd-labeled fake, the section gets red border + red banner warning. Helper _compute_fake_corpus_matches in main.py wraps the lookup with graceful fallback if the corpus is empty or embeddings module is missing.

Two new tables, both with migration safety: release_feed_items and fake_reference_posts. Both auto-create via Base.metadata.create_all() on next startup; no manual SQL needed. Migration idempotency now has test coverage (test_late_additions.py).

How to start using it:<ol style='margin: 0.4rem 0 0.4rem 1.2rem;'><li>Restart the server so the new tables get created.</li><li>Visit /releases and click “Sync now” to pull the Disney Pin Blog feed. ~5s for the initial fetch.</li><li>Visit /fake-corpus and run the scraper (start with 1 page + compute_embeddings=Yes to get the reverse-lookup working). ~5min for 100 posts with embeddings.</li><li>Open any pin with a front image — if similar posts exist in the corpus, you'll see a new “Community lookup” section showing them.</li></ol>

v0.15.22

2026-05-18

SQLite WAL mode + concurrency pragmas. Default SQLite journal mode (DELETE) locks the whole DB while writes commit, which blocks concurrent reads. Drew's deployment has background tasks (Pin&Pop enrichment, eBay refresh, embeddings warmup) writing alongside user-driven page loads — DELETE mode causes the occasional UI stall while a writer finishes.

Added a SQLAlchemy connect event listener in db.py that, on every new connection, sets:<ul style='margin: 0.4rem 0 0.4rem 1.2rem;'><li>PRAGMA journal_mode=WAL — Write-Ahead Logging. Readers and writers no longer block each other. Persistent across DB file restarts.</li><li>PRAGMA synchronous=NORMAL — the WAL-recommended fsync level. FULL is overkill with WAL, OFF risks corruption on power loss.</li><li>PRAGMA foreign_keys=ON — should always be on; SQLite ships with it OFF by default for legacy compat. Not WAL-related but worth fixing here.</li><li>PRAGMA busy_timeout=5000 — if a write IS blocked (e.g. during a periodic checkpoint), wait up to 5s before raising SQLITE_BUSY. Eliminates the rare 500s from transient locks during heavy background activity.</li></ul>What you'll notice: page-load stalls during catalog warmup / eBay refresh / background enrichment should drop noticeably. Steady-state perf is the same.

Heads up on backups: WAL mode creates two sidecar files alongside dispindable.db: dispindable.db-wal and dispindable.db-shm. They contain recent writes not yet checkpointed back to the main file. If you back up the DB by copying just the .db file mid-session, you can lose recent writes. Two safer alternatives:<ul style='margin: 0.4rem 0 0.4rem 1.2rem;'><li>Copy all three files together (.db, .db-wal, .db-shm) — simplest, copy is consistent.</li><li>Run VACUUM INTO 'backup.db' first — SQLite's atomic backup, produces a single self-contained file no matter the WAL state.</li></ul>Verified end-to-end: smoke test confirms all four pragmas apply on a fresh connection. journal_mode reads back as “wal”, synchronous=1, foreign_keys=1, busy_timeout=5000ms.

v0.15.21

2026-05-18

**Dead-code cleanup: removed the pre-v0.15.3 sold-listings HTML scraper (~492 lines, 40% of ebay_lookup.py). Deleted four functions that hadn't been called since v0.15.3 pivoted sold-listings to the eBay Browse API's EXT_AND_SOLD fieldGroup:<ul style='margin: 0.4rem 0 0.4rem 1.2rem;'><li>_title_from_itm_url() — extracted human-readable titles from eBay /itm/ URLs by slug-parsing</li><li>_normalize_whitespace_for_split() — OCR-confused-character cleanup before garble detection</li><li>_looks_garbled() — three-pattern obfuscated-title detection (a 5+ version arms race with eBay's anti-scrape)</li><li>_parse_sold_listings() — the 368-line BS4 parser for li[data-viewport] sold-listing cards</li></ul>Plus removed the _garbled_dump_done module flag that the parser used to one-shot dump garbled-card HTML to disk. The lazy from bs4 import BeautifulSoup import that lived inside _parse_sold_listings goes away with the function (no top-level BS4 import to clean up — was already lazy).

File shrinks 1220 → 722 lines. Zero behavior change — fetch_sold_listings() has been Browse-API-only since v0.15.3. Live API surface verified intact: is_available, search_active_listings, fetch_sold_listings, estimate_price, build_smart_query all still present and callable.

Note on headless_scrape.py: originally flagged in old changelogs as dead-after-v0.15.3, but actually still in use — catalog_sites.py falls back to headless Chromium when PinPics' Cloudflare challenge blocks the requests path. Kept as-is.

Resurrection path (if eBay ever pulls Browse-API sold access): git blame the terminal comment in ebay_lookup.py at line 720 and check out the v0.15.20 version of the file. The legacy scrape logic was working as of v0.15.0 with three obfuscation-pattern detectors and HTML-dump diagnostics in place.

v0.15.20

2026-05-18

Confirmed-fake red bubble next to the active-AI pill on /new-pin. When Pin&Pop's catalog scrape flags the chosen pin as having confirmed fakes in circulation, a small red “⚠️ CONFIRMED FAKES EXIST” pill renders right next to the existing “★ USING: X” sage indicator. Reuses the existing is_flagged_fake variable (same one driving the red outline on scan images + the pre-checked Confirmed-fake checkbox lower on the page), so visibility is consistent across all three spots. The full warning banner with explanation + Pin&Pop link still renders further down — this pill is the at-a-glance summary for users who are mainly looking at the AI badge to confirm prefill state.

v0.15.19

2026-05-17

Copyseeker integration wired: manual-trigger button on /pins/{id} with quota tracking, 7-day per-image cache, and publicly-fetchable catalog-URL fallback. Drew subscribed to the RapidAPI Copyseeker endpoint (free tier: 10 requests/MONTH, hard limit) and sent over the API schema, so the v0.14.15 scaffold is now real.

Endpoint: GET https://reverse-image-search-by-copyseeker.p.rapidapi.com/?imageUrl=<url> with x-rapidapi-key + x-rapidapi-host headers.

Response shape: Google Vision Web Detection wrapper — BestGuessLabel, Pages[{Url, Title, MatchingImages, Rank}], Entities[{Score, Description}], VisuallySimilar[<url>]. Parsed into the standardized provider-result dict shared with the other auth-check providers (Google Vision, SerpAPI Lens, Yandex).

Why manual-trigger only: 10 requests/month would burn out in one cataloging afternoon if auto-fired. copyseeker_check() is no longer called from run_all_checks(); instead a separate manual button on the pin detail page spends one quota slot per click.

Three layers of quota defense:<ul style='margin: 0.4rem 0 0.4rem 1.2rem;'><li>Per-image cache (.copyseeker_cache/, 7-day TTL, sha256-of-url filename). Re-clicks on the same image → 0 quota cost.</li><li>Monthly usage tracker (.copyseeker_usage.json, {month, count}). Increments on every HTTP attempt (RapidAPI counts failures against quota too). Resets automatically when month changes.</li><li>Reserve-slot limiter: blocks normal calls when remaining ≤ 1, exposes a separate “Use reserve slot” red button so a deliberate late-month retry still works.</li></ul>LAN-deployment workaround: Copyseeker's servers need a publicly-fetchable image URL, which https://10.0.0.32:8000/uploads/... isn't. Route picks the catalog reference image instead, preferring Pin&Pop's CDN over PinPics'. If neither is available (pin has no catalog IDs), shows a friendly “no public catalog image” flash rather than burning quota on a 404.

UI on /pins/{id}: new “🔎 Copyseeker reverse-image check” panel below the existing multi-provider authenticity panel. Shows X/10 used this month in the heading, “Check on Copyseeker” button (or “Use reserve slot” / “Quota exhausted” depending on state), then renders the last result's best-guess label, classified matches, detected entities, and visually-similar image grid. Up to 5 historical runs kept in a new pins.copyseeker_history_json column (migration registered in _LATE_ADDITIONS).

Test before relying on it: Click the button on one pin to confirm the integration works end-to-end before assuming it'll work for the next 9 you need. Watch the server log for the [copyseeker] prefix on errors. The Status page (/settings/api-status) shows whether COPYSEEKER_API_KEY is detected.

v0.15.18

2026-05-17

Match confidence collapses inline + catalog match boxes consolidated. Two items from Drew's v0.15.17 review:

1. Match Confidence module is one-row tall now. Previously: title row + a separate <details> below with its own line for the “Signal breakdown” summary — about 3 stacked rows of chrome. Now the <summary> IS the title row, with the “Signal breakdown” label as a small label right-aligned via margin-left: auto. Whole module collapses to one row; expanded view still shows the full per-signal table.

2. PinPics / Pin&Pop ID match boxes always render the chosen pin; larger detail panels permanently hidden. Reasoning: the candidates boxes at the top of the page (PinPics + Pin&Pop) were gated on length > 1 — only visible during AI-provider disagreement. The single-candidate happy path fell back to the larger “Pin&Pop catalog match details” / “PinPics catalog match details” collapsible panels further down, which duplicated 80% of the same info (image + name + ID + similarity).
Now: candidate boxes render whenever length >= 1. Wording scales with count — one candidate shows “PinPics match” / “Pin&Pop match”, multiple shows the existing “ID disagreement — N candidates” label with the click-to-override explanation. The two larger detail panels below are wrapped in {% if false and ... %} so they're permanently hidden but the code stays intact — flip false to a real condition to bring them back if you decide you want richer field views later (e.g. release date, retail price, edition).

v0.15.17

2026-05-17

Inline Import-from-URL on /new-pin, Collection field spacing, /series helper text trim. Three items from Drew's review of v0.15.16:

1. Collection field spacing in the inline New Series form. The Collection (folder) picker sat too close to the Name/Year grid row above it. Bumped margin-top from 0.5rem → 1rem on the label and added 0.25rem above the select; added explicit display: block + 0.3rem above the helper <small>. Whole block now reads as its own clear section.

2. Import from URL added to /new-pin inline New Series form. Drew wanted the same one-shot import the /series page has, available from the inline form so he doesn't have to leave the in-progress pin save. Built a new /api/series/import-url JSON endpoint that wraps the same scrape + create logic as POST /series/import-url. Required refactoring import_url_save → _import_url_save_impl with dict return type so both the form-redirect path and the JSON path can call into it without duplicating the ~180-line scrape/create logic. Form-redirect wrapper translates the dict to RedirectResponse with same flash messages as before; JSON wrapper returns {ok, id, name, slug, created, platform} on success, or detects duplicate-import (existing series with same source_url) and surfaces the existing series so the user's dropdown gets it selected anyway.
Inline UI: a collapsible “🔗 Import from URL” section at the top of the inline form, with one URL input and an Import & select button. Uses the folder picker from the manual form for consistency. On success: drops the new series into the dropdown, selects it, auto-closes the panel after 3s. On failure: red status text with the error from the API (the user can copy the URL into /series/import-url for a more detailed manual retry).

3. /series page helper text dropped “or a PinPics pin URL”. Drew flagged the mention as misleading on the Series-list page — PinPics single-pin import creates a one-pin series, which isn't what users land on this page expecting. The /series/import-url destination page still documents PinPics-pin as a valid input for the rare cases where a one-pin series is what you want.

v0.15.16

2026-05-17

Folder color picker + row tint, one-click destructive deletes, OCR collapsed, score helper removed. Five items from Drew's review of v0.15.15:

1. /collections (folder list) gets full-row color tint + native color picker. Two pieces: (a) folder rows now render with a soft wash of the folder's color across the entire card (via color-mix() at ~12% alpha) + a heavier left-border accent, instead of just a tiny dot beside the name. Falls back gracefully when no color is set. (b) Both the create and edit folder forms now use a native <input type="color"> picker paired with a hex text field, kept in sync via JS — no more memorizing hex codes. Type-or-pick supported in both directions; 3-char shorthand #abc expands to #aabbcc on blur.

2. “Delete this series” on the series-edit page is now a single click. Was a two-step toggle (click button to reveal panel, click panel button to commit). Drew flagged the friction. Now: one click on Delete fires a browser confirm() dialog with the full impact summary (series name + pin count). Same destructive-but-confirmable semantics as before, fewer interactions.

3. “Delete pin” on pin detail is a red button now. Was a <details> toggle with a gray-outline summary. Replaced with a direct dp-btn-danger form-submit + confirm dialog — same UX shape as item 2 above. Visually consistent with all other destructive actions in the app.

4. Removed the trust-score helper paragraph. “Composite score from image similarity, AI agreement, OCR vs catalog back-text overlap...” was visual bulk. The score label + value + the Signal breakdown details below already give all the context needed.

5. “Text read off the pin” collapsed by default. The OCR + per-AI text readings are useful as fallback signal when catalog ID is weak, but they take a lot of vertical space on a typical happy-path scan. Wrapped the section in a <details> with a count of how many text sources are inside, so the user can see “3 sources” at a glance without expanding.

v0.15.15

2026-05-17

Catalog ID pills, delete-button parity, Lens sizing, auto-tag without series. Four items from Drew's review of v0.15.14:

1. PinPics / Pin&Pop pills under Set name link directly to the catalog pin page when an ID is known. Drew flagged that the pills were firing a name-filter search (?filter_name=SDR%20Hidden%20Mickey...) when the exact pinpics_id was already on the form. Pills now carry a data-prefer-id-from attribute pointing at the f-pinpics-id / f-pinandpop-id form field. The click handler checks that field first — if populated, links to /pin/{id}/; otherwise falls back to the search URL. Super-series pills stay as search URLs since umbrella concepts don't map to a single ID.

2. Red X on /series page delete buttons. Both the per-series Delete button and the folder-header X on the series-list page (collections.html) were using secondary outline (gray). Drew wanted them red like the /collections (folder list) page. Switched both to dp-btn dp-btn-sm dp-btn-danger.

3. Lens button matches Delete button height. On /pins detail pages, the “Lens” <a role="button"> rendered noticeably taller than the “Delete” <button> next to it. Root cause: Pico's base [role="button"] rules apply different padding/line-height than its button rule, and the diff leaks through specificity. Locked .dp-btn-sm to height: 1.65rem + line-height: 1 + box-sizing: border-box so anchors and buttons render at identical sizes.

4. Auto-tag fires even when no series is chosen. Drew sent a pin with description “Donald Duck is shown standing in his classic blue sailor suit...” that got no tags. Root cause: the auto-tag block was gated on if collection and collection.auto_tag:, so pins saved without a series (allowed since v0.15.8) skipped the block entirely. Refactored to a should_auto_tag computed flag that defaults ON when no series is chosen, and respects Collection.auto_tag=False as an explicit opt-out when a series IS chosen. Series-level tag inheritance still only applies when a series is chosen (you can't inherit from a series that doesn't exist). For Drew's Donald Duck case the extractor returns [Donald Duck, Ramen] → _normalize_tags → donald-duck, ramen. Set name (“Ramen”) was already in the auto-tag chain via the existing if set_name: auto_parts.append(set_name) line; Drew asking for “the set name should be a tag” is now covered by both paths.

v0.15.14

2026-05-17

Collection +New inline-create, persistent flash, set-member thumbs (third attempt), pills fixed, smarter auto-tags, trimmed AI section. Nine items from Drew's review of v0.15.13:

1+6. + New button on the Collection picker. Drew asked twice. Mirrors the Series + New pattern: button reveals an inline form (Name + Color + Tags), POSTs to a new /api/folders JSON endpoint, then injects the new option and selects it. The JSON API is distinct from the form-redirect POST /collections (which the /collections page uses) and from /api/collections (which confusingly creates a Series due to the long-standing vocabulary mismatch — Folder=Collection in UI, Collection=Series).

2. /scan flash banner stays put. Removed the 6s auto-dismiss setTimeout. Manual X dismiss still works; the banner persists until clicked or the user navigates away.

3+4. Removed “Fields below are prefilled...” + “AI suggestions” h3. The bordered AI provider cards details from v0.15.13 reads as its own section heading; the h3 above it was redundant. The active-AI badge (“★ Using: Claude (Sonnet)”) moved to a small pill that lives just above the bordered details — same info, less chrome.

5. Removed the model-version badge. The small mono-font pill next to each provider name (“claude-sonnet-4”, “gpt-4o”, etc) is gone. Provider label is enough; exact model is still in the suggestion dict for logs / debug if needed.

7. Set member thumbnails (THIRD attempt). v0.15.11 added image_url to Pin&Pop's linked_pins scrape, but the lookup only tried data-src and src. The actual SDR Hidden Mickey pin's linked-pins strip uses a different lazy-load pattern. Expanded the attribute list to data-src → data-lazy-src → data-original → data-srcset (first URL) → src. Also added relative-URL rewriting (/images/... → https://pinandpop.com/images/...) and a debug print when extraction fails so the next miss is diagnosable from the server log. Bumped cache schema 3 → 4 to force re-fetch of any v0.15.11 cached entries.

8. Set name / Super series catalog pills work now. The PinPics ↑ / Pin&Pop ↑ pills under the Set name and Super series fields navigated to /new-pin# instead of the catalog search URL. Root cause: the click handler mutated a.href, but the browser's default navigation can resolve the href BEFORE the property assignment completes — race condition. Refactored to preventDefault() + window.open(url, '_blank') so the URL is composed and used atomically.

9. Smarter auto-tag extraction. Drew flagged that a pin with description “Dewey (in his blue hoodie) sneaks...” should pick up “dewey” as a tag, but the AI's character_or_subject was sometimes empty. New helper _extract_auto_tag_subjects(name, description) adds two fallback sources: (a) first capitalized proper-noun at the start of the description, with a stop-list (“The”, “Disney”, etc) to filter junk; (b) parts of the pin name after splitting on ' - ' (the canonical Disney-pin separator), with venue prefixes (SDR/WDW/DLR) and 4-digit years filtered out. For “SDR - Dewey - Strawberry Jam - Hidden Mickey” with Drew's description, the candidates are [Dewey, Dewey, Strawberry Jam] → _normalize_tags dedupes to dewey, strawberry-jam. Both the rescan and new-pin save paths use the helper; runs in addition to (not instead of) the existing AI character field. Unit tests in /home/claude/v_extract.py verify the heuristics on 5 representative cases.

v0.15.13

2026-05-17

Yellow nav badge + AI cards bordered + eBay parity + result-count fixes. Six items from Drew's review of v0.15.12:

1. Removed the back-description explainer paragraph. The textarea's placeholder already has a concrete example; the long “FAC number, edition number...” block was redundant. Kept the prefill-source acknowledgment since that's actionable info.

2. Yellow soft-triage badge in the nav alongside the red one. New secondary_validation_count() Jinja global counts pins where (collection_id IS NULL) OR (no PinPics AND no Pin&Pop ID AND NOT needs_validation), DISTINCT-deduped. Renders as a yellow pill right after the existing red badge in both the topnav and the More-dropdown Validation link. Same 30s cache as the red count.

3. AI provider cards section gets a visible border. Drew said the summary line alone read like plain text. Wrapped the <details> in a 1px-bordered rounded container with subtle background tint so it's obvious as a clickable card. Restored the default Pico disclosure triangle on the summary (list-style: revert) so the expand affordance is visible too.

4. eBay active panel gets the low–high range pill always, matching sold. Previously the range was conditional on count > 1; now it always renders ($X–$X for single comp is fine and consistent with the sold-listings treatment from v0.15.10).

5. Refresh link + Open-search link on sold listings panel, matching active. New ?refresh_ebay_sold=1 query param forces a fresh sold-listings fetch independently of the global refresh_ebay (which already covered both). Open-search link drops the user into the same eBay search URL the server queried, so they can browse more comps in the browser without re-typing. Active panel also got an equivalent Open-search link for symmetry.

6. More eBay results: limit bumped 6 → 10 and broaden-threshold relaxed. Drew got “1 of 1” on a query that he thought should return more. Two fixes: (a) search_active_listings(limit=10) + fetch_sold_listings(limit=10) on the /new-pin route — eBay's Browse API allows up to 200 per request so this is well within budget. (b) The broaden-retry threshold in search_active_listings changed from total == 0 to total < 3 so narrow queries that match 1-2 listings still get the broader fallback query attempted. Broadened result only wins when it has strictly more hits — strict precision is preserved when the broaden doesn't help.

If you still see “1 of 1” after this: that genuinely means eBay only has 1 listing matching both the strict and broadened phrasings. Disney pins are weirdly name-variable across sellers; a pin sometimes only has 1-2 active comps even when it's common. The Open-search link can help you verify by checking eBay manually.

v0.15.12

2026-05-17

Validation queue polish + confirmed-fake handling + AI section pared back. Four items from Drew's review of v0.15.11:

1. /validation: renamed “No catalog match” → “No series match”, added pin thumbnails, added yellow count badges. Drew asked for clearer labeling, visual recognition of pins in the triage queue, and per-section counts that stand out. Built a shared macro val_section() that renders thumbnails (prefer ‘front’ view, fall back to first image, fall back to “?” placeholder) + a yellow (#d4a017) round count badge next to the section heading. Both the no-series-match and no-series-assigned sections use it; the top needs_validation section already had thumbnails.

2. Removed the “What does Use X's suggestion actually do?” explainer. Drew flagged it as noise. Star indicator + button labels are enough affordance; users don't need a paragraph spec of which form fields get overwritten.

3. Removed the “Each provider sees the same images...” intro paragraph. The FROM CLAUDE · OPENAI pill in the h3 already lists which providers are active, so the paragraph was redundant.

4. Confirmed-fake handling on /new-pin. Three pieces:
  (a) When Pin&Pop's catalog scrape flags the matching pin as having confirmed fakes in circulation, the scan image thumbnails on the left panel get a heavy red border + glow so the user can't miss it. Gated to the chosen Pin&Pop ID (no false positives from stale scrape data).
  (b) New “Confirmed fake / scrapper” checkbox on the form, outlined red, prefilled when the same flag fires. User can override either direction.
  (c) On save, when the checkbox is set, pin.authenticity_status is set to confirmed_fake (matches the existing AUTHENTICITY_STATUSES enum). Otherwise the model default “unverified” applies. Same logic on the rescan path but with a one-way ratchet: rescan can promote to confirmed_fake, but unchecking on rescan leaves the existing status alone (so a previously-marked authentic pin doesn't silently revert).

v0.15.11

2026-05-17

Polish round: set-member thumbnails actually working, AI-active indicator more obvious, Collection field visible on /new-pin, folder tags, search padding, alignment. Seven items from Drew's review of v0.15.10:

1. /collections Edit button vertical alignment. The Edit summary sat slightly lower than the Delete X next to it because the .actions flex row had no align-items and the default stretch left the <details> child and the <form> child rendering on different baselines. Added align-items: center.

2. Folder-level tags. New folders.tags column + Tags input on both the Create-collection and Edit-collection forms. Comma-separated; _normalize_tags() handles spaces → hyphens, dedupe, lowercase. Folder tags don't auto-cascade to pins (the auto-tag chain stops at the series level), but they're useful for filtering and as a hint when creating new series under a folder.

3. AI provider card outlines + active indicator is obvious now. Two pieces: (a) base .ai-panel gets a full 1px border so cards read as distinct boxes (was just a left-border accent before), and (b) the .ai-active treatment is much louder — heavier 2px sage border all around, 3px sage halo, a star (★) prepended to the heading. The earlier ::before “ACTIVELY USING” ribbon top-right was easy to miss next to a long card heading; star + heavier outline is impossible to overlook.

4. Set member thumbnails actually work now. Root cause: Pin&Pop's linked_pins scrape in catalog_sites.py only captured {id, name} — never image_url. My v0.15.10 enrichment was matching against an image_url field that didn't exist. Fixed the scrape to also grab img.data-src (preferred) or img.src from the linked-pin anchors, bumped the Pin&Pop cache schema version (2 → 3) so old cached scrapes get re-fetched. Now the set-member cards on /new-pin render small thumbnails when Pin&Pop knew the linked pin (degrades gracefully via onerror for missing or broken URLs).

5. Collection field visible under Pin details on /new-pin. Was hidden inside the “+ New series” inline form — only visible when creating a new series, not when picking an existing one. Now there's a top-level Collection dropdown on /new-pin that auto-syncs to the chosen series's current folder (via a JS map of series_id → folder_id rendered into the page). Changing it on save re-files the series into the picked folder — which moves the series AND its other pins together (since the relationship is series → folder, not pin → folder). Empty value = Unsorted.

6. Search inputs no longer overlap their built-in icon. Browsers render a magnifying-glass icon inside <input type="search"> but the default text/placeholder offset doesn't account for it. Added a global padding-left: 2rem !important rule on input[type="search"] so the text starts cleanly past the icon on the global search bar, /series search, /collections search, and /catalog browse.

7. (Same as #2 — folder tags.)

v0.15.10

2026-05-16

/new-pin: eBay collapse symmetry, AI provider highlighting, set member thumbnails, folder in inline series, description-aware auto-tags. Five items from Drew's review of v0.15.9:

1. eBay active is now collapsed-by-default with median in the summary line. Matches the v0.15.8 sold-listings treatment. Same <details> + summary-pill pattern; expansion reveals the card grid + Refresh / See-sold links. Also: estimate_price() now accepts a single listing as a valid sample (was >=2); for rare pins one comp is still better than nothing. The summary hides the low–high range when count==1 (all three values would be identical).

2. AI provider section is its own area + the active provider is highlighted. Drew said the v0.15.8 sage-bordered container made the AI section read as a contained sub-modal rather than a page section — removed. The section header now carries a “Using: <provider>” pill that names whichever provider's data is currently prefilling the form, and that provider's card gets a heavier left border + an ACTIVELY USING ribbon. JS updates both when the user clicks a different “Use X's suggestion” button.

3. Set members: collapsed by default + thumbnails. Sets with 8–12+ pins were dominating the page. Now in a <details> with the count + set-size hint in the summary. Each member card also shows a small thumbnail when Pin&Pop's scraped linked_pins provided an image_url for that entry — enriched both during the linked-pins fold-in pass and on existing AI-merged entries (matched by pinandpop_id or close name). AI-only members without a Pin&Pop link still render without a thumbnail; everything else degrades gracefully.

4. Inline “+ New series” can pick a folder. Drew asked where the “add to collection” field was on /new-pin. The quick-create form (when you create a brand-new series inline rather than picking one) now has an optional Collection dropdown so the new series goes into the right folder up front instead of landing in Unsorted and needing a separate trip to /collections to re-file. Existing series keep their existing folder (handler doesn't mutate already-assigned ones).

5. Auto-tag picks up the character from the AI identification. Drew's example: a Dewey pin with description “Dewey (in his blue hoodie) sneaks a finger of strawberry jam...” should auto-tag “dewey”. The AI providers already return character_or_subject in their JSON; we now carry that through to the new-pin form via a hidden character field (prefilled from the chosen provider, updated by usePrefill() on swap) and append it to the auto-tag list server-side when collection.auto_tag is on. Same pattern on both the rescan path and the new-pin path. No extra UI — the AI does the noun extraction; the field stays hidden because description already carries the same info in human-readable form.

v0.15.9

2026-05-16

Polish pass: button sizing, search icons, save flow, no-series validation queue. Seven items from Drew's review of v0.15.8:

1. Lens button matches its row siblings. Dropped the 🔎 color emoji that was making the Lens button taller than the Edit/Rotate/Delete siblings on /pins/{id} and /scan/X/new-pin. Color emojis have intrinsic ascent/descent that monochrome text glyphs like ✂ (Edit) and ↻ (Rotate) don't carry. Plain “Lens” text now keeps the row uniform.

2. Series search no longer has a double icon. <input type="search"> renders its own search icon in most browsers. Adding 🔍 to the placeholder produced a visible double-icon. Stripped the emoji from placeholders on collections.html, folders.html, catalog_browse.html, and base.html's global search.

3. Collections row: Edit and Delete are compact dp-btn-sm now, Delete is red. The row-level action buttons on /collections were using secondary outline which read too neutral for the Delete and too bulky for the Edit. Now: dp-btn dp-btn-sm on Edit, dp-btn dp-btn-sm dp-btn-danger on Delete. Also added a CSS rule to suppress Pico's auto-chevron on plain summary[role="button"].dp-btn (restored only for the .dp-toggle-btn variant which renders its own ::after chevron).

4. Pack/set note helper text moves down. Adds a margin-top so the <small> helper label sits visually below the input's bottom edge rather than glued against it.

5. /series 3-button row finally renders as 3 columns. Root cause: v0.15.7 introduced .dp-toggle-row for the new buttons-in-row pattern, but that name was already taken by the settings-page toggle-row class (defined lower in dispindable-tokens.css with !important). The Settings class won the cascade and forced grid-template-columns: auto 1fr on my series buttons. Renamed to .dp-toggle-cluster-row to avoid the collision; all three buttons now render in equal columns with rotating chevrons.

6. Save pin redirects to /scan with flash. After saving on /scan/X/new-pin, the user now lands on /scan with a confirmation banner (“Pin saved: X.” + “View pin →” link) instead of the pin's detail page. Rationale: most saves happen mid-cataloging where the natural next step is scanning the next pin. Banner auto-dismisses after 6 seconds; manual X dismiss available.

7. /validation adds a “No series assigned” section. Since v0.15.8 made series optional on save, this completes the triage loop: pins saved without a series surface in /validation in their own section so they can be assigned later. Distinct from the existing needs_validation (Quick-Add-style untriaged) and no_match_pins (catalog cross-check) sections; a single pin can appear in multiple sections if multiple flags fire.

v0.15.8

2026-05-16

/new-pin workflow improvements + /scan auto-crop opt-out. Seven items from Drew's review of the post-scan review page:

1. Series field is now optional. Previously required, forcing users to either pick or create a series even for one-off pins. Now you can save a pin with no series and assign one later from /pins/{id}. Database column pins.collection_id migrated to nullable via a SQLite table rebuild — introspects the live schema to preserve every column (including any drift), updates the constraint only for the target column. Idempotent and transaction-safe; tested against synthetic schema drift.

2. Clear-all-fields button. Wipes every input, textarea, select, checkbox, and tag chip in the new-pin form after a confirm prompt. For when none of the AI / catalog prefills are right and you want a blank slate to manually enter.

3. Import-from-URL on /new-pin. Mirrors the /series page's URL-import feature. New GET /api/import-pin-from-url?url=... endpoint parses PinPics or Pin&Pop pin URLs, fetches via the existing cached catalog scrapers, and returns prefill fields. UI button on /new-pin opens a paste-URL panel; fetch & prefill overwrites form fields with whatever the catalog returns.

4. Google Lens buttons on /new-pin. One button per scan image under the image grid in the left column. Same Lens reverse-image-search pattern as on /pins/{id}; same caveat (needs public-internet reachability to the server).

5. Skip-auto-crop checkbox on /scan. Auto-crop is on by default and usually helps DINOv2 matching, but Drew flagged cases where the crop cuts out the main subject (ornate frames, set backers, partially-cropped originals). New optional skip_smart_crop checkbox on the /scan form passes through to submit_scan which uses it to toggle _save_upload(..., smart_crop=...).

6. AI suggestions section visually framed. AI suggestions area now lives inside a sage-bordered container with an explicit “AI suggestions” header (badge + provider list) so it's instantly clear which content is AI-sourced vs. catalog-sourced vs. user-entered.

7. eBay sold listings collapsed by default with median in the summary. Now we're actually getting clean Browse-API data (v0.15.3), the panel can be compact-first: the summary row shows the median price pill and low–high range without expanding. Click to expand the full card grid. Saves a chunk of vertical space on the /new-pin page.

One incidental fix: Pin.collection_id changed in models.py to nullable=True. The DB migration runs once on startup; logs print [db] relaxing NOT NULL on pins.collection_id via table rebuild then [db] pins.collection_id is now nullable (N columns preserved). Back up dispindable.db before first run as a precaution.

v0.15.7

2026-05-16

Series action cluster: buttons always visible + single chevron + tag helper spacing. Three follow-ups from v0.15.6:

1. Buttons stay in the row whether their panels are open or not. v0.15.6 used <details> where each summary doubled as both button and toggle, and opening one made it span the full grid width — pushing the other two onto the next row. Drew wanted all three always visible. Fix: new .dp-toggle-cluster pattern with .dp-toggle-btn buttons in a 3-col row plus .dp-toggle-panel siblings below. Each panel is controlled independently via data-target="#panel-id" on the button and tiny JS in base.html that flips aria-expanded + data-open. Buttons stay put; panels open/close beneath them. Multiple panels can be open at once if you want.

2. Double chevron eliminated. The summary[role="button"] Pico CSS adds its own background-image chevron, which was rendering alongside my ::after chevron from v0.15.6. Two arrows in one button. Switching from summary to plain button elements sidesteps Pico's rule entirely — only my ::after chevron renders now, on the right edge, rotating 90° when expanded. As a belt-and-suspenders, the .dp-collapse legacy class also now explicitly suppresses Pico's background-image chevron for anyone still using that pattern.

3. Tag input helper text breathes. Adds margin-bottom: 0.5rem to the chip wrapper produced by setupTagBubbleEditor. The <small> helper text below the tags input was rendering flush against the wrap's border. Now there's a small breathing gap on every page that uses the tag chip editor — series, pin detail, scan-new-pin.

v0.15.6

2026-05-16

Series action buttons: cream surface, right-side chevron, 3-column row, Delete properly red. Four UI fixes from Drew's feedback on the v0.15.5 ship:

1. Color: from filled sage to subtle cream. .dp-btn previously rendered as solid sage which competed with page typography. Now uses a low-opacity ink-soft cream surface (rgba(244,236,214,0.08) on dark, rgba(58,76,99,0.06) on light) with a thin matching border. Reads as “part of the page” rather than three glowing CTAs. Sage is kept around as .dp-btn-accent for when we want a Save/CTA button to pop.

2. Chevron: right-side and rotating. The left-side ▸/▾ chevrons I added in v0.15.5 doubled up with Pico's default right-edge marker, and the marker pointed the wrong way. Dropped both: .dp-collapse now renders its own &rsaquo; chevron via ::after on the right edge, rotating 90° on open. Pico's default marker is suppressed. One chevron, on the end, rotating correctly — matches what Drew was asking for.

3. Three buttons in a row. New .dp-btn-row utility lays out 2–3 <details> as a CSS grid (3 columns on desktop, 1-col stack on <700px). When any details opens, grid-column: 1 / -1 expands it to full width and the siblings flow to the next row — so the Edit-series form keeps a generous editing surface even when buttons are compact by default. Applied to the three Edit series / Custom thumbnail / Delete this series accordions.

4. Delete actually renders red now. v0.15.5's .dp-btn-danger didn't win against the base .dp-btn styling on <summary role="button"> elements — Pico's own summary rules added enough specificity to beat .dp-btn-danger. Fixed by explicitly including summary[role="button"].dp-btn-danger in the selector list. Brick red, dark text on dark theme, white text on light theme.

v0.15.5

2026-05-16

Series details: open by default + show/hide toggle instead of edit toggle. Drew flagged that the “Edit series” accordion on series pages implied a separate edit mode — but the form below it is just the series' own fields, nothing modal. Revealing them is a show/hide toggle, not a state change. Fix: the <details> now starts open, and the summary button shows ▾ Hide series details when open / ▸ Show series details when closed. Pure CSS swap via the new .dp-collapse class — two spans inside the summary, only the relevant one rendered based on [open] attribute. No JS needed. Default disclosure triangle suppressed (we render our own chevrons in the labels). The Custom thumbnail and Delete this series accordions stayed as-is — Custom thumbnail is task-driven, Delete needs to stay closed for safety.

v0.15.4

2026-05-16

UI polish: filled-sage action buttons + Inter system font + Lens-button height fix + tag-chip placeholder cleanup. Four related fixes Drew flagged in screenshots:

1. Lens button rendered taller than its siblings. The Edit / Rotate / Lens / Delete row on pin detail had Lens sitting visibly proud of the others. Root cause: Lens used lucide("search", 12) which returns inline SVG, wrapped in display: inline-flex; align-items: center; gap: 0.25rem;. Edit/Rotate use plain HTML entity glyphs (✂, ↻) that sit in the normal inline text baseline. The SVG's line-box metric differed enough that inline-flex bumped the button height by a few pixels. Fix: dropped the SVG + inline-flex, used 🔎 (magnifying-glass emoji) and plain text. Now all four buttons share the same inline baseline naturally.

2. .dp-btn filled-sage action button class. Replaces the thin Pico secondary outline style on action buttons that needed more presence. Filled sage (#5d8a5b light, #7BB573 dark), 6px radius, Inter system-UI font stack (Inter, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, …), darker-sage hover. Plus .dp-btn-danger variant (brick red) for Delete actions and .dp-btn-sm compact variant for inline action rows. Applied to: Edit / Rotate / Lens / Delete on pin detail thumbnails, and the Edit series / Custom thumbnail / Delete this series accordion summaries on series pages. Opt-in class — not retroactive across all buttons in the app, so existing in-progress UI work stays stable. Easy to spread further later.

3. Tag chip placeholder hidden when chips exist. The Tags input chip editor showed its long placeholder text (comma-separated, e.g. "hidden-mickey, 2014, …") wrapping awkwardly beside already-populated chips. Now: placeholder shows the full guidance text only when zero chips exist, switches to a short + tag hint once any chip is added, restores the long form when the last chip is removed. Pure JS in base.html; no template changes needed.

v0.15.3

2026-05-16

Sold-listings: replaced HTML scraping with the official eBay Browse API. Drew showed me a working Python script that calls the same Browse API endpoint we already use for active listings, just with one extra parameter (fieldGroups=EXT_AND_SOLD), and gets back real sold-listings data with real titles, real images, real prices — no scraping, no anti-bot evasion, no decoy-span detection. I went down a five-version rabbit hole (v0.14.94 through v0.15.2) building an arms-race against eBay's anti-scrape obfuscation when the official API supported sold listings the whole time. That was a wasted detour I should have caught earlier. The headless Chromium dependency I added for the scrape path (v0.14.86+) is no longer used by sold-listings — the module stays in the codebase for now in case other features need browser rendering, but this code path is pure HTTPS API calls. What this fixes: all garbled-title issues, all Akamai bot-detection issues, the “Pardon our interruption” 5xx fallback, the broadened-vs-strict bucket split, the zero-width-Unicode tokenization edge case, the temporary [ebay-sold-title] debug log — gone, all of it. Cache bumped v10 → v11. Tradeoff: the API doesn't return a per-listing “Sold <date>” field on the item_summary response, so that label no longer appears in the UI. Templates already handle null sold_date by hiding the label.

v0.15.2

2026-05-16

Garbled-title detection: normalize Unicode whitespace before tokenizing. Drew confirmed via grep that v0.15.1 was actually running on disk, cache was at v9, but the garbled-card dump file never appeared — meaning the drop path never fired despite obviously-garbled UI content. Root cause: str.split() in Python does NOT treat zero-width characters (U+200B), word joiners (U+2060), or zero-width joiners (U+200D) as whitespace. If eBay's anti-scrape lacing puts zero-width chars between letters where I expected regular spaces, split() returns ONE long token instead of many single-char tokens — defeating Signal 1's “>40% single-char tokens” check at len(tokens) < 4 early-out. Fix: (1) new _normalize_whitespace_for_split() helper uses a regex covering ASCII whitespace, NBSP, U+200B-U+200F, U+2028-U+202F, U+205F, U+2060, U+3000, U+FEFF to collapse every Unicode space flavor (including zero-width) to regular spaces before tokenizing. (2) _looks_garbled moved from nested-inside-the-loop to module scope — previously it was redefined every iteration (wasteful) and harder to reason about during debugging. Same three signals, same thresholds. (3) Debug log line added right before the garbled check: [ebay-sold-title] <title repr> garbled=True/False — repr() reveals any hidden Unicode as escape sequences. Grep [ebay-sold-title] in uvicorn output to see exactly what the parser is processing. (Temporary — will be removed once garbled-detection is verified reliable in production.) Unit-tested 8 garbled patterns (visible spaces, zero-width chars, word joiners, ZWJ, mixed) + 6 real titles. All 8 garbled fire as garbled; all 6 real pass through. Cache bumped v9 → v10.

v0.15.1

2026-05-16

Critical hotfix: v0.15.0 had a SyntaxError that prevented uvicorn from starting. In ebay_lookup.py I declared global _garbled_dump_done twice in the same function — once at the top (correct) and again inside the garbled-card loop (illegal once the name has already been written to). Python detects this at compile time and refuses to load the module. The pre-ship ast.parse() validation didn't catch it because that error fires during compile, not parse. Fix: removed the redundant inner global declaration; the top-of-function one already covers scope. Going forward I'll use compile() rather than ast.parse() for validation — same checks Python runs at import. The garbled cards Drew saw earlier were from v8 cached data served because v0.15.0 itself never booted (the crash caused uvicorn to fall back to a working older worker, which kept serving stale v8 cache entries).
Google Drive Picker integration on /scan (Option C). Adds a “📂 Drive” button next to Camera and Scan barcode. Click → native Google Drive modal opens → user picks one or many image files → client-side JS downloads bytes via the Drive API → wraps each as a File → pushes into the dropzone via the existing dropzone.addFiles() method. From there the upload flow is identical to drag-and-drop. Multi-select supported. Filters to image MIME types so the picker doesn't show PDFs or videos. Setup required: Google Cloud Console project with Picker API + Drive API enabled, browser-restricted API key, and OAuth 2.0 Web Client ID. Set GOOGLE_DRIVE_API_KEY and GOOGLE_DRIVE_OAUTH_CLIENT_ID env vars before starting uvicorn. Full setup steps in a long comment inside scan.html. Without the env vars the button stays hidden — graceful no-op rather than a broken-looking button. Why this approach: downloads happen in the browser using Drew's own OAuth token, never touching the server. No service account, no token storage, no Drive credentials in app config beyond the public API key + client ID. Anthropic-style: keep credentials with the user, not the app.

v0.15.0

2026-05-16

Sold-listings garbled-title detection: catch the grouped-chunks pattern too. v0.14.97 caught single-character decoy spans (“S K p o n 6 s o M r e 7”). Drew's next screenshot showed eBay using a different pattern — grouped chunks like “S3C 9Up QoRnso FredZ 84 8”. Only 6 tokens, only 1 single-char, ratio 16.7% — my heuristic falsely cleared it. Two new signals added to _looks_garbled: (1) tokens that mix letters AND digits (e.g. S3C, 9Up) — real Disney pin titles use pure-digit years (“2024”) and pure-letter words (“Disney”), with pin model numbers like “162531” also pure digit. Within-token alphanumeric mixing is an eBay decoy signature. Threshold: >20% of tokens. (2) Random uppercase mid-token (e.g. QoRnso, FredZ). Real titles are Title Case (upper at position 0 only) or ALL CAPS. Lowercase-then-uppercase within one token is unnatural. Threshold: >15% of tokens. Either signal alone flags the string as garbled. Unit-tested 11 cases: both old single-char and new chunked-decoy patterns fire, while 8 real Disney pin titles (including all-caps, with-numbers, and Title-Case variants) pass through cleanly. Plus a one-shot HTML dump of the first garbled-then-dropped card's outerHTML to .ebay_cache/sold_garbled_card_<ts>.html. Bounded to one file per scrape (module-level flag, reset at the start of each parse) to keep disk use predictable. Next time eBay invents a third obfuscation pattern, we have concrete DOM to grep instead of guessing. Cache bumped v8 → v9.

v0.14.99

2026-05-16

Client-side crop/rotate: cap canvas size to prevent silent empty-blob failures on phone photos. Drew's console after v0.14.98 showed the new defensive handler catching uploads as skipped (empty), with filenames ending in _cropped.jpg — that suffix is added by our own client-side crop pipeline in base.html's setupDropZone. The underlying cause was platform canvas-memory limits: canvas.toBlob() on iOS Safari (and to a lesser extent mobile Chrome) silently returns an empty blob when the canvas exceeds the platform limit (~4096×4096, less under memory pressure). Cropping into a full-resolution phone photo (typical 4032×3024) produces an output canvas large enough to trip this on most phones. Three fixes: (1) cap the crop output canvas at 1500px on the longest side — the DINOv2 embedding model resizes to 224×224 anyway, and OCR doesn't need more than 1500px, so this is generous headroom while staying well below platform limits; (2) cap the rotate-90° canvas the same way (same code path, same risk); (3) validate blob.size > 0 in both toBlob callbacks and show a useful alert if it still comes back empty — previously the empty blob got wrapped in a 0-byte File and silently submitted; (4) form-submit-time check on the /scan form that scans input.files for any zero-byte entries and refuses to submit (with an explanatory alert) if any are found. Last-line defense in case any other code path produces an empty file. The server-side v0.14.98 defensive handler stays as a backstop — the goal here is to stop 0-byte uploads from leaving the browser at all.

v0.14.98

2026-05-16

Scan upload no longer 500s on unsupported image formats (HEIC, AVIF, video, etc). Drew's console showed PIL.UnidentifiedImageError: cannot identify image file in embeddings.embed_bytes when a client at 10.0.0.45 (different IP than his desktop — likely a phone) hit POST /scan. The likely cause is iPhone HEIC: iOS defaults to HEIC for Camera Roll photos, and PIL doesn't decode HEIC without pillow-heif. The crash was a missing try/except — one bad file in a multi-file upload killed the whole request. Fix: (1) new embeddings.detect_image_format() helper that identifies images from magic bytes (JPEG, PNG, GIF, WebP, BMP, TIFF, HEIC, AVIF, MP4, MOV) plus empty/truncated/unknown states. (2) POST /scan now wraps every per-file embed_bytes call in a try/except. On UnidentifiedImageError, the parser identifies the format and skips that file with a user-facing reason (“HEIC image (iPhone default) — change Settings…”, “Looks like a video, not a photo”, etc). Other uploads in the same submission still go through. If ALL uploads fail, returns a 400 with per-file reasons instead of a 500. (3) POST /pins/{pid}/images (manual photo upload on a pin detail page) gets the same treatment + cleans up the orphan file from disk when embedding fails. Real fix for HEIC if you want it: pip install pillow-heif on the server, then add import pillow_heif; pillow_heif.register_heif_opener() near the top of embeddings.py. Out of scope for this ship (would change the install footprint); the defensive fix above stops the 500s either way.

v0.14.97

2026-05-15

eBay sold-listings: behavioral sponsored detection + img-alt title fallback + drop unrecoverable garbled cards. v0.14.96's sponsored-card filter missed cards where eBay doesn't use any of the structural markers I was checking (no sponsor in class/attr/aria-label, and the visible “Sponsored” text is rendered via decoy spans so it doesn't appear in get_text() output). Drew's screenshot showed 2 sponsored cards still slipping through with garbled titles. Real tell: sponsored placements on the sold-listings page are sourced from active listings, so they have NO “Sold <date>” line. Organic sold-comp cards always do. Switched detection to that behavioral signal — any card without a sold-date marker is treated as sponsored and dropped. This also drops accidental auction listings, which aren't useful sold comps either. Title recovery: when aria-label is missing and parsed text is garbled (the >40% single-char tokens heuristic from v0.14.96), the parser now tries: URL slug first, then <img alt="…"> (sellers set this for accessibility — often the cleanest title source). If all three fail, the card is dropped entirely — showing the user “S K p o n 6 s o M r e” gibberish is worse than fewer-but-clean results. Unit-tested against the markup pattern from Drew's screenshot; 2 sponsored cards dropped, 2 organic recovered via aria-label / img-alt, 1 organic-but-unrecoverable also dropped. Cache bumped v7 → v8.
Owned column on /pins. Per Drew's request — the list view now surfaces ownership state alongside the name/series/year columns. Green check for owned, dim dash for not. If a pin is also on the wishlist, a star appears next to the check/dash so both pieces of info fit in one narrow column.

v0.14.96

2026-05-15

eBay sold-listings: skip sponsored cards + use aria-label titles + URL-slug fallback for garbled titles. v0.14.95's parser was extracting real listings end-to-end, but two of the three results came back with garbled titles like “S p o O n s o r F e T W d 6 Z I F R” and eBay-logo thumbnails instead of pin photos. Diagnosis: those were sponsored (paid-placement) listings, and eBay decorates sponsored cards with anti-scrape decoy spans — many tiny <span> elements with random letter-separated text, hidden visually via CSS but concatenated by get_text() into nonsense. Sponsored placements are also useless for price comps regardless — they're paid spots, not organic sold prices. Three fixes: (1) Skip sponsored cards entirely via multi-signal detection: sponsor in any class on the LI or descendants, data-sponsored attribute, aria-label*='sponsored', or visible “Sponsored” text. (2) Prefer aria-label on the item anchor over text content for the title — eBay puts the real listing title there for screen-reader accessibility and the decoy spans don't touch the attribute. (3) URL-slug fallback for cards where neither aria-label nor any clean text element is available: extract title from the /itm/<slug>/<id> URL path, which is human-readable and unobfuscated (the slug appears in URLs like /itm/Disney-Trading-Pin-162531-DL-Shan-Yu-Portrait-Mulan-Villain/12345). Triggers when a parsed title has >40% single-character tokens, which is the signature of decoy-span garbage. Unit-tested the parser against synthetic sponsored + organic + garbled-title markup; only organic cards come through, all with real titles. Cache bumped v6 → v7 so cached v6 results (with the garbled sponsored entries baked in) refetch under the new parser.

v0.14.95

2026-05-15

eBay sold-listings parser: structural fallbacks for the new li[data-viewport] markup. v0.14.94 successfully got us past Akamai (2.1MB of real HTML coming back, 60 listing cards detected), but the parser still returned 0 results because all the per-field selectors inside each card were scoped to eBay's OLD s-item__* class names. The new markup dropped those entirely — the diagnostic showed 60 cards on the page with 84 price-classed and 197 title-classed elements, but none matching div.s-item__title or span.s-item__price. Fix: added structural fallbacks alongside the legacy selectors. For title: try the legacy selectors, then .s-card__title, then a[href*='/itm/'] span (every eBay item link has /itm/ in its path — that's a stable structural anchor across redesigns), then a loose [class*='title'] scoped to the card. For price: try the class-based selectors first, then scan all text in the card for the first $N.NN amount NOT preceded by ‘+’ (which would mean shipping/delivery). For URL: a[href*='/itm/'] as the fallback. For image: img[src*='ebayimg.com'] and a bare img fallback. The class names rotate every redesign; the structural relationships (each card has a title, a price, an item-URL anchor, an image) don't. Plus: when 0 results parse, also dump the raw HTML to .ebay_cache/sold_debug_<ts>.html so next time eBay rotates markup we have concrete evidence to update selectors against instead of guessing. Cache version bumped v5 → v6 so 0-listing entries from v0.14.94's run refetch with the new parser.

v0.14.94

2026-05-15

eBay sold-listings scraper: Akamai bot-detection bypass attempt + 'disney pin' query prefix. The sold-listings panel has been returning 0 results because Akamai (eBay's edge CDN) was fingerprinting our headless Chromium and serving 301-char <title>Access Denied</title> shell pages instead of the real search results. Confirmed via Akamai reference ID format in the diagnostic logs from v0.14.86. Three layered defenses added to headless_scrape.py: (1) Stealth init script — runs before any page JS on every navigation, hides navigator.webdriver, fakes navigator.plugins (real Chrome has PDF Viewer + a few others; empty list is a giveaway), and sets navigator.languages to ['en-US', 'en']. Defeats the three most common JS-side bot probes. (2) Anti-fingerprint context — locale='en-US', timezone_id='America/Chicago', 1920x1080 viewport (default 800x600 is unusual for desktop users), Accept-Language header. Headless contexts missing these read as obviously synthetic. (3) Warmup visit — new optional warmup_url parameter on fetch_rendered. When set, navigates there first in the same browser context so anti-bot cookies/localStorage are seeded, then navigates to the actual target. Akamai is more forgiving of sessions that look like they came from a homepage visit than cold deep-links. Sold-listings scrape now passes warmup_url='https://www.ebay.com/'. Plus the ‘disney pin’ query prefix from the long-pending v0.14.86 follow-up: prepended to the sold-listings URL keyword query to anchor the keyword matcher on words Disney pin sellers actually use in titles. Slightly redundant with the _sacat=38004 category filter but tightens within-category ranking. Watch for over-filtering (listings titled e.g. “Tangled Pin Pascal Mystery” have “pin” but not “disney” in the title and will be excluded by this prefix). Honest caveat: none of this beats residential-IP-based detection or full Akamai Bot Manager — those need paid scraping infrastructure (ScrapingBee, Bright Data, etc.). This stops the lazy/medium-effort detection. If logs still show Access Denied / 301-char preview after this update, we're in “live with manual eBay search buttons” territory. Bumped SOLD_CACHE_VERSION to v5 so cached v4 results (which hit the no-prefix URL) get refetched.

v0.14.93

2026-05-15

eBay active-listings search: auto-broaden on zero results. When a strict query against the Browse API returns 0 listings AND the query contains connector words (with, and, &, the, a, an, of, in, on, at, to, for, is), automatically retries once with those words stripped. Catches Pin&Pop-style names like “Dewey with Half Full Jar” where the strict query returns 0 because no seller writes “with” in their listing title, but the broadened “Dewey Half Full Jar” matches real listings. Adjective-y words (“Half”, “Full”, “Hidden”) are deliberately NOT stripped — they can be the distinctive part of a pin name. UI: when the broadened path was used, the active-listings header on /pins/{id} shows what was actually searched and notes “broadened from <original>” so price comps aren't silently misleading. Caching: broadened results live under the original query's cache key, so revisits don't re-burn API calls; 1-hour TTL unchanged. Cost: Browse API default tier is 5,000 calls/day at no cost — one extra call on miss is <0.1% of quota per pin view. Also adds: an “ok=True, listings=[]” branch on the pin page that tells the user when both queries returned 0 and suggests using the manual eBay search buttons instead (previously this case rendered nothing). Sold scraper intentionally unchanged — Akamai is blocking that path regardless of query, separate fix needed.

v0.14.92

2026-05-15

Trimmed “(USD)” off the “Original Price” label in all three places it appeared: /pins add form, /pins/{id} edit form, and the /new-pin scan-review form. The currency suffix was redundant clutter — this is a single-user app and Drew isn't tracking pins priced in yen.

v0.14.91

2026-05-15

Series checklist layout: real root-cause fix. The checklist cards weren't stacking because of a specificity issue, browser extension, or stylesheet drop (my hypotheses across v0.14.89 and v0.14.90). They were stacking because of invalid nested-<a> HTML. The card template wrapped each <article> in an outer <a class="pin-card-link">, but uncataloged pins also render small inner anchors inside the article (the “Pin&Pop ↗” and “PinPics ↗” external links). HTML5 forbids <a> inside <a>; when the parser hits the inner anchor it runs the adoption agency algorithm and pops the <article> out as a sibling of the (now-empty) outer anchor. So every rule we targeted at .pin-card-link (flex: 1 1 140px, max-width: 200px) was landing on an empty popped-out anchor, and the visible article was a separate grid/flex item taking width: 100% of its cell — one card per row. The fact that v0.14.90's grid-with-empty-tracks still showed 2-per-row instead of 4-per-row was the smoking gun (the “empty” tracks were the popped-out anchors). Confirmed by a parallel diagnosis Drew shared from another Claude session that did DOM inspection via headless Chromium. Fix: stretched-link overlay pattern. The <article> is the grid item directly — no outer wrapper anchor. A <a class="card-link-overlay"> sits inside the article, absolutely positioned with inset: 0 covering the whole card, capturing clicks for the card-level link. Interactive children (external Pin&Pop / PinPics anchors, +Own / +Wish buttons, the corner delete X) get position: relative; z-index: 2 via a .pin-card-checklist a:not(.card-link-overlay), .pin-card-checklist button selector so they float above the overlay and receive their own clicks. No nested <a>, no parser fix-up, no layout fight. Also dropped the defensive inline-style hacks from v0.14.90 — they were defending against the wrong cause and are noise now. What I missed: the “only uncataloged pins break” pattern was visible in the template I was reading the whole time — the inner anchors are gated on not pin.images. Should have caught that earlier than I did.

v0.14.90

2026-05-15

Series checklist layout: cards back in a grid, thumbnails back to full card width. Drew reported via screenshot that even after v0.14.89, his checklist cards were stacking vertically full-width instead of flowing in a grid, and the ? thumbnail placeholders were rendering as small squares in the upper-left corner of otherwise empty striped cards. Three CSS rules in the inline <style> block were not applying in his browser: .pin-grid { display: flex }, .pin-card-link { max-width: 200px }, and .pin-card-checklist .thumb-wrap { width: 100% }. Sibling rules in the same block (border, padding, background, flex centering inside the thumb-wrap) did apply, so this wasn't a generic parse failure or cache miss — specifically width / max-width / display:flex-on-container rules were getting eaten by something I couldn't identify from the code. Two fixes: (1) rewrote .pin-grid as CSS Grid with grid-template-columns: repeat(auto-fill, minmax(140px, 200px)) — auto-fill (not auto-fit) creates empty tracks for leftover space rather than stretching populated tracks, addressing the v0.14.83 over-stretch concern that pushed us off grid in the first place; (2) added defensive inline style= attributes on the .pin-grid container and .thumb-wrap div with the same values, so the layout renders even if the class rules continue to be silently dropped. Inline styles outrank class rules in the cascade without needing !important. If the broken-class-rules pattern repeats elsewhere in the app it's worth a real diagnosis — would help to see the Computed panel in DevTools next time.

v0.14.89

2026-05-15

Series-imported pins now backfill names + images correctly (two-part fix). Drew reported that pins imported from a Pin&Pop series URL render with ? thumbnail placeholders and labels like “Pin #100261” — never the real names. Root cause A: catalog_sites.fetch_pinandpop_series creates placeholder pins named Pin #<id>, but the enrichment helpers (_enrich_missing_pin_from_pinandpop and ..._from_pinpics) only recognized Pin&Pop # / PinPics # as placeholder prefixes. So placeholder evaluated to False on every series-imported pin, and the name update was silently skipped. Added Pin # to the prefix set in both helpers. Root cause B: neither /series/import-url (which redirects on duplicate source_url) nor /series/{cid}/refresh-from-url (which increments skipped_existing on ID match) re-queues enrichment for pins already in the database — so any series imported on an older version, or one where the bg task failed mid-fetch, was stuck without thumbnails forever. Fix: new POST /series/{cid}/backfill-thumbs route that iterates every pin in the series with no images and queues a fresh enrichment task (with_image=True) for each one. Reachable via a new button on the series page that only renders when missing_count > 0, so it disappears once thumbnails populate. Reports queued vs. skipped-no-catalog-ID counts via flash message.

v0.14.88

2026-05-14

make_ssl_cert.py: proper local-CA setup for Android 14+. Drew couldn't install the old self-signed cert on his Android device — even with “CA certificate” selected, the OS demanded a private key. Root cause: the old script generated an end-entity cert (CA:FALSE); Android 14+ / One UI 6+ enforces RFC 5280 strictly and refuses to install non-CA certs as CAs, falling back to the user-cert flow that needs a key. The new script generates two certs: a root CA (ca.pem with CA:TRUE and keyCertSign) plus a server cert (cert.pem) signed by that CA. Install ca.pem on the phone (no key needed); uvicorn uses cert.pem + key.pem as before. Existing CA + key reused on re-run so devices stay trusted across regenerations.

v0.14.87

2026-05-14

**Re-added _sacat=38004 (Disneyana > Pins) to the sold-listings URL. Drew correctly pointed out that the original v0.14.36 removal was based on the same 0-results evidence that turned out to be the scraper bug (fixed in v0.14.86). With the scraper actually returning real HTML, the category filter delivers the same noise reduction it does on the active path — Pascal (Blue) drops from 542 results (jeans, hockey cards, etc.) to a handful of actual Disney pins. Tradeoff: we may miss the occasional miscategorized listing where a seller put their pin under a sibling Disney category. Net positive for typical queries. Bumped sold cache version to v4 since the URL itself changed.

v0.14.86

2026-05-14

**Major: eBay sold-listings scraper was returning empty for every query. Drew correctly diagnosed that the sold panel never shows results regardless of how common the search term is. Root cause: the wait selector in ebay_lookup.fetch_sold_listings was "ul.srp-results, body". CSS comma means OR. body exists from the moment any page opens, so Playwright resolved the wait immediately — before JavaScript rendered eBay's actual results. page.content() returned a ~300-char SPA shell, and our parser reported html_len=301, item_nodes=0. Fix: (1) selector is now "ul.srp-results, #srp-river-results" — both are real eBay result containers, no body fallback; (2) fetch_rendered now sleeps wait_ms after the selector matches, giving lazy-loaded item cards time to render; (3) bumped post-selector wait to 4s; (4) navigation now uses wait_until="load" (was "domcontentloaded") so JS bundles are ready when the selector wait begins; (5) added diagnostic that dumps a preview when the HTML comes back <2000 chars (bot challenge / interstitial). This bug had been latent for many versions; the diagnostic added in v0.14.77 is what surfaced it (html_len=301 was the smoking gun).
Why active panel works while sold doesn't (Drew's question). Same query goes to both paths (pin.name verbatim), but active uses eBay's Browse API (auth'd JSON endpoint, category 38004 filter, smart tokenization) and sold uses the public web search page (web scrape, no category, treats - and () as operators). Same input, two very different parsers. v0.14.84 fixed hyphens. v0.14.86 fixes the scraper itself. Parens (e.g. Pascal (Blue)) remain a query quality issue we can iterate on once the scraper actually returns HTML.

v0.14.85

2026-05-14

Re-importing a series no longer 500s. Drew tried to import a Pin&Pop series URL that had already been imported — got a sqlite IntegrityError on collections.name UNIQUE constraint. The slug dedupe loop was suffixing the slug (...-2) but the name itself collided. Two-part fix: (1) before creating, check if any existing collection has the same source_url — if so, redirect to it with a “Already imported” flash; (2) if the URL is new but the name collides (different release, same name), suffix the name with (2) too.

v0.14.84

2026-05-14

eBay sold listings: stripped hyphens from the query. eBay's web search treats - as a NOT operator. Drew's pin name “SDR - Dewey - Strawberry Jam - Hidden Mickey” was being read by eBay as “search for SDR but EXCLUDE Dewey, Strawberry, Jam, Hidden, Mickey” — guaranteed 0 results. This only bit the sold-listings path (web scraping) because the active-listings path uses the Browse API which tokenizes differently (that's why the active panel showed 3 hits, sold showed 0, for the same query). Fix collapses " - " to " " before URL-encoding. Bumped sold-cache version to v3 so stale empty results refetch.
Hotfix: series import crashed with TypeError. v0.14.82 passed True positionally to _enrich_missing_pin_from_pinandpop, but with_image is keyword-only in the callee (*, with_image: bool = False). Every series import threw 500. Switched to with_image=True kwarg.
Pack/set note: shorter labels. Drew called out that the price-note field on /new-pin had verbose help text. Trimmed the placeholder to “e.g. /2, /pack” and the small-text caption to “Pack/set note (optional)”. Applied the same tightening to /pins/{id}.

v0.14.83

2026-05-14

Series checklist: switched grid → flex. v0.14.82 dropped the grid minmax to 140px but the cards still rendered in 3 columns with empty space to the right — CSS Grid's 1fr wasn't shrinking cells to the minmax min in practice. Replaced with display: flex; flex-wrap: wrap + per-card flex: 1 1 140px; max-width: 200px. More predictable: cards flex between 140–200px, take available row space evenly, wrap to next row when full. 4-pin series now fits one row at typical desktop widths; 8+ wraps to 2 rows.
HSTS header on HTTPS responses + HTTP-to-HTTPS redirect helper. Drew called out that some devices on the LAN tried http://10.0.0.32:8000 even after the URL was shared as https:// — uvicorn with --ssl-keyfile listens HTTPS only, so HTTP attempts get a connection error rather than a redirect. Two-part fix: (1) the FastAPI app now emits Strict-Transport-Security on every HTTPS response so browsers that reach HTTPS once remember to skip the HTTP attempt for ~24h; (2) added redirect_http_to_https.py — run it in a second terminal (python redirect_http_to_https.py) and it listens on port 8080 returning 301s to https://<host>:8000. Devices that hit http://10.0.0.32:8080 get auto-redirected, then HSTS keeps them on HTTPS thereafter.

v0.14.82

2026-05-14

Barcode field on Pin + scan button. Added Pin.barcode (VARCHAR(64), auto-migrated) plus input + scan-button on the pin edit form. Personal UPC index: scan packaging once and save it to a pin; future scans of the same barcode resolve to that pin instantly. /api/barcode-lookup now matches against this column too (exact match — barcodes are short strings, substring matching would create false positives).
Barcode scanner close X: inline SVG instead of unicode. The × / ✕ characters kept rendering as tofu boxes for Drew because his browser's font cascade didn't cover those glyphs. Replaced with an inline <svg> drawing of an X (two crossed lines). Font-independent, always renders.
Series checklist: 4-pin sets now fit one row. Grid template was repeat(auto-fill, minmax(170px, 1fr)); in the ~700px main area that only fit 2 columns, so a 4-pin series wrapped to 2x2. Dropped min to 140px so 4-5 columns fit. Larger sets still wrap naturally when they exceed the row count.
Series imports now download thumbnails automatically. Previously _schedule_catalog_prefetch only warmed the catalog cache — placeholder pins stayed imageless until the user opened each one. Series imports now schedule _enrich_missing_pin_from_pinandpop with with_image=True, so each imported pin gets its front photo downloaded as a PinImage in the background. After import, refresh the series page once and the checklist shows proper thumbs.
Save searches from /search. The page could READ saved searches but had no way to create them — you had to go to /pins to save one. Added an inline “Save this search” form that appears when a query is active. Posts to /saved-searches with the current URL prefilled.

v0.14.81

2026-05-14

Barcode scanner: web-search fallback links. Drew asked “need the option to use that barcode to search the web for matches as well?” The barcode he scanned (63761290) is a retailer UPC, not a Disney internal ID, so the local lookup against Pin.pinpics_id / Pin.pinandpop_id correctly returned 0 matches. Added five web-search fallback buttons to the scanner-result modal — Google, eBay active, eBay sold, Disney Pins Blog, and Barcode Lookup. Shown both when there ARE local matches (below them) and when there aren't. Each opens in a new tab.

v0.14.80

2026-05-14

Hotfix: /api/barcode-lookup 500'd on every scan. JSONResponse wasn't in the top-of-file fastapi.responses import — the endpoint added in v0.14.52 referenced it without ever importing. Latent until now because Drew didn't have a working barcode scanner to hit the endpoint with. With v0.14.79's HTTPS setup, the scanner finally got camera access, scanned a UPC, called the lookup endpoint, and the missing import surfaced. Added JSONResponse to the import line.

v0.14.79

2026-05-14

Pin&Pop series import: cache version invalidation. v0.14.78's new parser was running fine, but Drew kept seeing the same “no pins extracted” result + no diagnostic output. Root cause: an earlier failed import wrote pins: [] to the on-disk cache; subsequent calls short-circuited with that cached miss BEFORE the new parser code ran. Added a parser_version tag on every cached entry. Cache reader now invalidates entries with mismatched (or missing) version and re-fetches. v0.14.79 is the first release with the version tag — existing caches will all be treated as stale on first run, generating fresh scrapes that exercise the new parser.
**Added make_ssl_cert.py for HTTPS setup. Drew tried uvicorn ... --ssl-keyfile=key.pem and hit FileNotFoundError because he didn't have the cert files yet. Added a one-shot script that uses the cryptography library to generate a self-signed cert + key, with SAN entries for both localhost AND the auto-detected LAN IP so the same cert works from any device. Run python make_ssl_cert.py from the project dir, then start uvicorn with the ssl-* flags. First visit per device shows a cert-warning page — accept it — then camera APIs (barcode scanner) work normally because the page is a secure context.

v0.14.78

2026-05-14

Barcode scanner: pre-check for secure context with clear error. Drew hit “Can't enumerate devices, method not supported.” That's ZXing's message when navigator.mediaDevices isn't available — which happens because modern browsers block camera APIs on non-secure origins (HTTPS or localhost only). Drew accesses the app over LAN at http://10.0.0.32:8000, which is neither. The scanner modal now detects this BEFORE trying to load ZXing and explains the three workarounds: (1) use localhost:8000 when on the host machine, (2) set up HTTPS for the server, (3) Chrome's chrome://flags/#unsafely-treat-insecure-origin-as-secure flag.
Mobile: search icon in the top nav. Desktop has the always-visible search input in the top-right; mobile hides that to save header real estate. Added a search-icon button next to the theme toggle on mobile only — tap it to jump to /search, which has its own focused input.
Delete X alignment on /pins. The red × button rendered slightly taller than Edit because the form wrapper added an implicit baseline + the button had default padding. Now both use explicit line-height: 1.4 + display: inline-flex; align-items: center + matched padding so they sit on the same baseline.
Pin&Pop series import: rewrote __NEXT_DATA__ extraction. The v0.14.77 walker had a too-loose heuristic (“any dict with id+name+image looks like a pin”) which matched the page's “currently viewed pin” / “related pin” objects in addition to the real series.pins array — explaining why a series scrape sometimes ended up with 1 unrelated pin as its only member. Rewrote to look specifically for ARRAYS of 2+ pin-shaped dicts and pick the longest one (with a preference bonus for arrays found under keys named pins, members, linkedPins, etc.). Added a last-resort extraction from CDN image filenames (since Pin&Pop image URLs encode pin IDs). Added a diagnostic dump when 0 pins are extracted — logs final URL, page title, candidate arrays found, anchor hits, CDN image count — so we can iterate from real data when this still misses.

v0.14.77

2026-05-14

Activity log: edits now logged too. Previously only added and deleted events hit the activity feed; edited events were silent so Drew's /activity stayed empty even after several pin / series updates in a session. Added activity_log.record() calls to update_pin and update_collection. Per-pin detail history (PinHistoryEntry) still captures the fine-grained field changes; this is the global roll-up event.
Pin&Pop series import: parse __NEXT_DATA__ JSON blob. Pin&Pop renders series pin lists client-side via Next.js, so the initial HTML response often lacks <a href="/pins/{id}/..."> anchors — they get added by React after hydration. Our scraper got the un-hydrated HTML and returned “no pins extracted from URL”. Now: parses the __NEXT_DATA__ JSON blob Next.js inlines for initial state, walks it recursively for pin records, falls back to anchor-scanning when missing. Also broadened the anchor regex to match both /pins/{id}/ AND /pins/{id}- URL forms.
Original Price field: stacked layout. The price input + pack-size note input were a cramped flex row that couldn't fit $18.95 in the price column without scrolling. Stacked vertically on /new-pin and the pin edit form, with placeholder hints in each box.
Barcode scanner close button: explicit font sizing. The × character was rendering as a box in Drew's view because the button had no explicit font-family or size, so it picked up the page's Crafted-UI serif which may not have included U+00D7 in a recognizable form. Added explicit system-ui font + 1.35rem size + min 2×2rem dimensions, swapped to U+2715 (a slightly chunkier ×-shaped glyph).
eBay sold-listings parser: fallback selectors + diagnostic. Drew's eBay query showed 2 sold listings in the browser UI but our scraper returned 0. Added fallback selectors (div.s-card, li[data-viewport], others) so we catch eBay's newer markup variants on result pages. When 0 results are still parsed, the log now prints diagnostic counts (price-classed elements, title-classed elements, dollar-text occurrences) so we can see whether the page was anti-bot, empty, or just rendered with markup we don't recognize.

v0.14.76

2026-05-14

Fixed second magnifying-glass icon on the global search box. The browser was rendering its own search-input glyph in addition to the Lucide icon I draw to the left. v0.14.69 added -webkit-search-* suppression rules but those only cover WebKit pseudo-elements — Edge / Firefox / non-Chromium Chrome variants ship their own search-input glyphs that ignore -webkit-* prefixes. Swapped type="search" for type="text" on the nav search input. The Lucide icon to the left already signals “search box”, and submission works identically.
Hotfix: barcode scanner threw “listVideoInputDevices is not a function”. v0.14.74 switched the library from @zxing/browser to @zxing/library but kept the old API call: ZXing.BrowserCodeReader.listVideoInputDevices(). That static-method form is from @zxing/browser's API. In @zxing/library it's an INSTANCE method on the codeReader. Changed to codeReader.listVideoInputDevices() after instantiation. Confirmed correct via @zxing/library's current npm docs.

v0.14.75

2026-05-14

Hotfix: POST /series/import-url + /series/new-from-pinandpop returned 422. Same route-ordering bug v0.14.70 fixed for GET, but the POST side was missed. The POSTs were going to /series/{cid} with cid="import-url" → that endpoint required name + slug + valid int cid, which the form data didn't supply → 422 Unprocessable Entity. Added _import_url_save_stub + _new_from_pinandpop_save_stub at the top of main.py (before the dynamic /series/{cid} POST), delegating to the real handlers further down. Stripped the original @app.post decorators to prevent duplicate registration.
Preview badges stacked + edge-quality rescan link. The Est. rarity / Est. value / Edge quality cards on /new-pin now stack vertically with a horizontal label-value layout, so each shows its full verdict + score inline rather than truncating to a tier abbreviation. Edge quality card has a “📷 Re-photograph &nearr;” link aligned to the right — goes back to /scan so you can retake the photo if the score looks rough.
Confirmed: edge quality auto-runs on every /new-pin GET. Tier-1 of the visual-authenticity pipeline (pin_authenticity.analyze_edge_quality()) runs offline + free + ~100ms on the front scan image at page load. Score is the “Edge quality NN/100” badge in the side panel. Tier-2 (Gemini side-by-side, $0.001 per check) still requires the manual button on the saved pin's detail page.

v0.14.74

2026-05-14

Hotfix: /pins 500'd on every load. v0.14.70 added a delete-confirm dialog with {{ p.name|replace(chr(34),'') }} to strip double-quotes from the JS-embedded pin name. chr is a Python builtin, not a Jinja function — every call into the /pins template hit UndefinedError: 'chr' is undefined. Replaced with a generic confirm message: the user knows which pin they're deleting because they clicked its row's button.
Mobile tab bar: Series instead of Collections + popup More menu. Per Drew. (1) Swapped Collections (folder-level grouping, secondary) for Series (primary unit, what pins belong to) on the bottom tab bar. Collections moved into the More sheet. (2) Tapping More now opens a slide-up bottom sheet with all 9 secondary nav items (Collections, Wishlist, Top lists, New releases, Catalog, Activity, Validation, Backlog, Settings) as a 3-col icon grid. Previously it just navigated to /settings. Sheet dismisses by tapping outside, the close X, or any item. Validation count chip carries through.
Barcode scanner: switched libraries to fix the “ZXingBrowser global missing” error. @zxing/[email protected] turned out to be a thin wrapper around @zxing/library whose UMD bundle didn't actually expose window.ZXingBrowser from either jsdelivr or unpkg — the script loaded but the global never appeared. Switched to @zxing/[email protected] which is the underlying library, ships a self-contained UMD that puts everything on window.ZXing, and includes BrowserMultiFormatReader + BrowserCodeReader.listVideoInputDevices() directly. Same multi-CDN fallback (jsdelivr → unpkg) and same per-CDN timeout as v0.14.72.

v0.14.73

2026-05-14

eBay auto-refreshes on every /new-pin load. Flipped the default. Previously the on-disk 1-hour cache won unless the user clicked the Refresh button (?refresh_ebay=1); now the cache is bypassed by default and the user can opt back into cached results with ?refresh_ebay=0 if they really want to. /new-pin is the “should I keep this pin / what's it worth” decision page, so fresh pricing matters. The hourly cache still applies on /pins/{id} where reloads are typically for editing rather than for price intel.
Favicon: silenced the 404. Added static/favicon.svg matching the in-app brand glyph (gold rotated-diamond on ink-navy) + a <link rel="icon"> in base.html. Added a /favicon.ico route returning 204 No Content — browsers always request /favicon.ico regardless of the explicit link tag (backwards compat from the IE days), so the 404 was spamming the access log on every page view. 204 makes the request succeed without forcing us to maintain a binary .ico file.

v0.14.72

2026-05-14

Barcode scanner: pinned version + CDN fallback. Drew hit “Camera error: Failed to load ZXing” when trying the barcode/QR scanner. ZXing-JS is the JavaScript port of the open-source ZXing (“Zebra Crossing”) barcode library — it's loaded from a CDN on first use of the scanner. The previous URL used @zxing/browser@latest from jsdelivr only, which can fail for three reasons: jsdelivr blocked by network/adblocker/DNS filter, the @latest tag breaking when the package author bumps a major version, or the UMD path changing between releases. Now: pinned to @zxing/[email protected] (current stable), tries jsdelivr → unpkg → jsdelivr-default-entry in order with a 6s per-CDN timeout. Each fall-through is logged to the browser console ([barcode] CDN failed: ...). If all three fail, the modal shows a clear error pointing at network / adblocker rather than the generic “Failed to load ZXing”.

v0.14.71

2026-05-14

/new-pin cleanup pass. Stripped 4 panels of clutter per Drew's feedback. (1) Removed the “PinPics has richer data...” bookmarklet hint + its companion auto-fetch wiring (~110 lines incl. JS) — the Chrome extension surfaces its own UI for PinPics pull-in and the “Use PinPics data” button on the panel above handles the rest. (2) Removed the “Validate before saving” panel — the direct PinPics #/Pin&Pop # links are already in the catalog-match details panels higher up, and the eBay/DPB/Google fallback search buttons were noise. (3) Removed the “easyOCR runs automatically…” preamble + the Search PinPics / Google / Copy primary footer on the text-readings panel. (4) Dropped the fake_warning_text body from the Pin&Pop confirmed-fakes flag — it's mostly boilerplate per pin; the flag itself is the actionable signal.
eBay refresh button on /new-pin. Small ↻ Refresh affordance in the eBay panel header, both the populated and 0-found cases. Adds ?refresh_ebay=1 to the URL; the GET handler passes that through as force=True to search_active_listings + fetch_sold_listings, bypassing the 1-hour on-disk cache. Useful when sold-listings show 0 but eBay actually has matches, or when prices have moved since the last query.
Edge-quality preview on /new-pin side panel. Computes pin_authenticity.analyze_edge_quality() (Tier-1 local CV, free, ~100ms) on the front scan image at GET time. Renders next to the Est. rarity / Est. value badges as “Edge quality NN/100” with a color-coded border — green (smooth, typical of authentic pins) / amber (inconclusive) / red (rough, flag for scrapper check). Tier-2 (Gemini side-by-side, costs $0.001 per check) stays gated behind the manual button on the saved pin's page.

v0.14.70

2026-05-14

Hotfix: /series/import-url was returning 404. Route-ordering bug. The dynamic /series/{cid} route was registered first and caught everything — /series/import-url went to collection_detail(cid="import-url") → _resolve_collection returned None → 404. Added forwarding stubs for /series/import-url and /series/new-from-pinandpop registered BEFORE the dynamic route. FastAPI matches in registration order, so static paths now win.
Removed duplicate add buttons on /series. The page had two near-identical entry points (“+ Add from Pin&Pop URL” and “Import from URL”) plus a third “+ Add Collection” (folder) form that didn't belong on the series page. Now there's one “Import from URL” button (handles Pin&Pop series URLs, Pin&Pop pin URLs, and PinPics pin URLs identically) plus the manual “+ Add Series” form. Folder/Collection add moved off to the existing /collections page where it belongs.
Delete button on /pins list. Each row now has a small red × button next to Edit. Confirms with the pin name and warns the action is irreversible (Pin&Pop/PinPics catalog data is unaffected). Hooks into the existing /pins/{id}/delete handler — same code path as the per-pin detail page's delete button, just inline for fast bulk cleanup.
Pin&Pop series URL + super-series URL on saved pins. When saving a scan, the Pin&Pop scrape's series_url populates Collection.source_url (only when empty, so user edits aren't clobbered) and the super_series_url goes onto a new Pin.pinandpop_super_series_url column (auto-migrated). Pin detail page renders both as small “View series on Pin&Pop &nearr;” / “View super series on Pin&Pop &nearr;” links under their respective text fields. Falls back gracefully when only PinPics URLs are present.

v0.14.69

2026-05-14

Series: manual total-size override. New Collection.total_size_override column (auto-migrated). Editable on the series edit form: “Total pins in set (override)”. When set, the completion percentage uses this number as the denominator instead of len(pins). Useful for series like Drew's Disney Villains Framed Portrait Mystery where he knows the set is 19 but has placeholders for fewer. Clamped to >= len(pins) — an override below the actual pin count is ignored. Leave blank to revert to auto.
Series: quick-search helpers when no URL stored. The “Refresh / seed from URL” form now shows two extra buttons when the series has no source_url or pinandpop_url_override yet: “Search Pin&Pop via Google” (uses site:pinandpop.com + the series name) and “Search PinPics”. Lets the user find a URL to paste back into the field without leaving the page in guess-the-URL mode.
Series checklist alignment fix. Drew's screenshot showed cards rendering with weird gaps in rows (2 cards mid-row 1, 3 cards left-aligned row 2). The grid spec was fine but several CSS defaults were fighting: Pico's <article> margin/shadow + the <a> wrapper's inline-default width. Added explicit justify-content: start, grid-auto-flow: row dense, width: 100% on the card link, and reset Pico's article styles inside the checklist context. Cards now pack row-by-row, left-aligned.
Validation count: visible nav badge. Red dot on the “…” more-menu when any pins have needs_validation=True; numeric chip on the Validation link inside the dropdown showing the exact count. Count is cached for 30s server-side so it doesn't hammer the DB on every page render. New validation_count() Jinja global.
Extra search icon in topnav fixed. Crafted renders a Lucide search SVG to the left of the global search input. WebKit (Chrome / Safari) was overlaying its default input[type="search"] results-icon and cancel-button decorations on top, producing the “two search icons” Drew reported. Added -webkit-search-decoration / -cancel-button / -results-button / -results-decoration { appearance: none; display: none } rules.
Original price input truncation. The price field on /new-pin had min-width: 4.5rem which shrink-fit to ~5 chars at narrower viewport widths — “$18.95” rendered as “1$”. Bumped to 6.5rem so 4-digit dollar values + decimals display without clipping.
Authenticity check: inline “How does this work?” panel. Per Drew's ask. New collapsible <details> under the “Authenticity check” header on every pin page. Explains the 4 providers (Google Vision, Google Lens, Yandex Reverse, Copyseeker), per-provider scoring inputs (domain reputation, page title keywords, snippet text), and the relationship to the separate edge-roughness score from pin_authenticity.py. Tells the user what each provider call costs and where to set the API key.

v0.14.68

2026-05-14

Hotfix: NameError on pin save. v0.14.56 added sku as a form field on /new-pin and used sku=(sku or "").strip() inside the Pin constructor, but never added sku: Optional[str] = Form(None) to the handler signature. The bug was latent until this build because nothing exercised the save path with the SKU field present — once the cache invalidation in v0.14.65 started populating the field automatically via usePinandpopData() / usePinpicsData(), every save hit the NameError. Added the missing param. Also threaded SKU through the rescan-existing-pin branch of the same handler (overwrites only when non-empty, so a rescan with the field blank doesn't nuke a previously-saved SKU).

v0.14.67

2026-05-14

Lens button icon size matched to siblings. The 🔎 color emoji rendered visibly larger than the monochrome ✂ / ↻ text glyphs on the per-image action row (Edit / Rotate / Lens / Delete) even with the 0.85em scaling I added in v0.14.38. Swapped to a Lucide search SVG at 12px so all three buttons read as the same visual weight.
eBay active listings: thumbnails were silently broken. The active-listings card template referenced l.image but ebay_lookup.py returns the field as l.image_url (sold-listings cards had it right — this was a stale reference introduced when the active-listings block was added). Every active-listing card was rendering with no thumbnail. Fixed.
**Server logs persisted to <install-dir>/logs/dispindable.log. New server_logging.py module wires up a RotatingFileHandler with backupCount=0 (truncate-in-place, no .1 / .2 archive files) capped at 5 MB. Catches the root logger (FastAPI / SQLAlchemy / urllib3) plus uvicorn's access + error loggers explicitly, since uvicorn doesn't propagate to root by default. Also tees sys.stdout / sys.stderr so the print() statements throughout the app (which is most of our diagnostic output) get mirrored too — without breaking terminal output. Setup runs early in main.py so even the very first module-import banners land in the file.
Optional: auto-run authenticity check on save. New “Auto-run authenticity check after every saved pin” toggle in Settings → Preferences. Default off. When on: after a new pin is saved via /scan/{id}/new-pin, a background task fires authenticity_check.run_all_checks with the pin's front image. Result lands on the pin detail page ~15-30s later. The settings card describes the cost tradeoff inline: Google Vision is 1,000 free / month then $1.50 per 1k (effectively free for personal use); SerpAPI-backed Google Lens + Yandex share a 100 free/month pool and burn 2 calls per save, so a high-volume scanner on the free tier hits the cap in ~50 saves. Off by default for that reason.
Authenticity provider docs / costs answered. Drew asked what APIs the visual authenticity check hits. Four providers, controlled by env vars: GOOGLE_VISION_API_KEY (Google Cloud Vision Web Detection — 1k free/mo, $1.50/k after); SERPAPI_API_KEY drives both google_lens (SerpAPI Google Lens engine) and yandex_reverse (SerpAPI Yandex engine), sharing 100 free searches/mo; COPYSEEKER_API_KEY is a stub (no live integration yet, would be free per the public Copyseeker page). LAN-only deployments will see the URL-based providers (Lens / Yandex / Copyseeker) skip silently with “no URL” errors — only Google Vision (which accepts binary upload) runs. Use Tailscale Funnel / ngrok / Cloudflare Tunnel to expose the server publicly for the URL providers to function.

v0.14.66

2026-05-14

Series completion bar fix. The /series/<id> completion percentage was counting has-images rather than is-owned. After v0.14.44 added background catalog-image enrichment for missing-pin placeholders, every placeholder gained a reference image — so a series of 4 placeholders + 0 owned read as 4/4 = 100% complete. Now uses is_owned=True directly. Reference photos still attached for the checklist thumbnails, but the bar only fills when you actually own pins.
Checklist card labels fixed. Cards used to read “✓ Cataloged (1 image)” for every placeholder with a catalog reference photo, which sure looked like “you own this”. Now labels match state: “✓ Owned” when is_owned=True, “On wishlist” for wishlist, “Reference photo — not yet owned” for the placeholder case, “Not yet cataloged” for empty placeholders.
No more AI-only auto-seeding of new series. Drew's repro: scan triggers AI to misidentify the pin as part of a different set, save creates a new series auto-tagged with the wrong name, the seed step blindly creates 4 placeholder pins from the AI's set_members — resulting in a series with entirely wrong pins. The seed now uses only catalog-backed sources (Pin&Pop series scrape + PinPics linked_pins). AI set_members stay visible in the per-pin “Set Members” panel for manual review but never auto-create rows. To clean up an existing bad series, click “Delete this series” on its detail page — that removes the series + all its placeholder pins.
Dark mode contrast fix on checklist action buttons. The “+ Own” / “+ Wish” buttons were reading as near-black-on-near-black in dark mode because the Pico form-element background variable collapsed under Crafted's overrides. Replaced with explicit rgba(255,255,255,...) bg/border/text values for dark and a matching light variant via [data-theme=...] overrides + a prefers-color-scheme fallback. Buttons now legible in both themes.
Pin&Pop price/SKU/rarity will populate after re-scan. v0.14.65's cache schema invalidation is the fix — next scrape of any previously-warmed pin re-fetches Pin&Pop with the v0.14.56 field-extraction regex (validated against the actual page format for pin 59070: $18.95 + SKU 400007148895 + Rarity Common). If a re-scan still doesn't populate, the cache file may be cached upstream of fetch_pinandpop; deleting .catalog_cache/pinandpop_59070.json manually force-refreshes that specific pin.

v0.14.65

2026-05-14

Catalog cache schema-version invalidation. Pin&Pop cached scrapes from before v0.14.56 were missing all the new fields (original_price, sku, rarity, edition_size, linked_pins, id) — my fold-in logic for set members was reading pinandpop_data.linked_pins off cached blobs that didn't have it. Result: Drew rescanned the Donald Duck Offset Frames pin and still saw the wrong 4-member “Expressions” set members because the cache file for Pin&Pop #59070 was stale. Added _CACHE_SCHEMA_VERSION = 2 constant; cache files without it (or with a lower version) now count as misses and trigger a fresh fetch. _write_cache stamps the current version on every write going forward. Next fetch of any previously-warmed pin will re-scrape and get the full v0.14.56-era data — no manual cache deletion needed.
“Use PinPics data” button. Mirrors the existing “Use Pin&Pop data” button on the /new-pin page. Pulls from the chosen PinPics panel (_chosen_pp_panel) when present, or falls back to the bookmarklet-imported data (pinpics_imported) when the user ran the Chrome extension. Maps name / year / PinPics ID / set_name / original_price / edition_size / sku / back_stamp into the form fields. Only overwrites the back-stamp field when it's empty so the user's own edits aren't lost.

v0.14.64

2026-05-14

Settings toggle layout fix. The role="switch" checkboxes in Settings → Preferences were rendering with the label text wrapping behind the toggle — visible on the “Strip background…” and “Default-expand advanced panels…” rows. Restructured the label markup: toggle + wrapped text block as two grid children with a fixed gap. New .dp-toggle-row class in tokens CSS for any future toggle-with-description-text patterns.
Wishlist quick-add via Pin&Pop URL. Paste a Pin&Pop pin URL on the new /wishlist/add page — we fetch name, image, series, year, price, edition, rarity, SKU, tags, and any fake-warning flag from the catalog page, show a preview, and on confirm save a new Pin row with on_wishlist=True + the first image imported as a reference photo. New “+ Add to wishlist” button on the /wishlist header. Two-step submit (form → preview → confirm) so you see what's being saved before committing. Re-paste of an already-cataloged Pin&Pop ID surfaces the existing pin instead of duplicating; if not yet on the wishlist, offers to flip the flag. PinPics URLs not supported here (PinPics blocks server-side scraping); the scan flow handles those.
Backlog: 3 new entries. (1) Email notifications on new pin drops — AlertFilter table + nightly cron + SMTP digest, pairs with the WDWNT source. (2) Add WDWNT.com as a second new-releases feed alongside DisneyPinsBlog — new scrape_wdwnt.py module mirroring the DPB scaffold. (3) Bulk wishlist add (paste multiple URLs or pull a series's linked-pins) — power-user follow-up to the single-URL flow shipped in this version. All under a new “Wishlist & Discovery” / “New Releases & Feeds” category split.

v0.14.63

2026-05-14

Lucide-style outlined SVG icons across the nav. Per design handoff: no emoji in chrome. New lucide(name, size=18, stroke=1.6) Jinja global renders monoline SVGs with the standard Lucide paths (stroke 1.6, round linecap, fill: none, color via currentColor). Replaced emoji icons in primary nav (Scan camera, Pins pin, Series list, Dashboard grid, Collections folder, Top trophy), in the More dropdown (Wishlist star, New releases gift, Catalog book, Activity activity, Validation alert-triangle, Backlog clipboard-list, Settings gear), and on the mobile bottom tab bar. Search icon, sun/moon for theme toggle — all SVG now. Icons fade to 75% opacity by default, 100% on hover or active state.
Logo redesigned. Brand block is now a 36×36 ink-navy square with a gold-stroke rotated-diamond glyph inside, plus the “DisPindable” wordmark in Newsreader serif. Matches the handoff B_Wordmark spec.
More dropdown moved into the primary nav. Previously the “…” menu lived in the right-side toolbar (after search, before theme toggle) which visually misaligned it. Now it's the 7th item in the links list, directly to the right of Top. Cleaner visual flow + the items are read in priority order.
/ now redirects to /dashboard. Dashboard is the primary landing page. The legacy stats+scan-dropzone home lives at /home for bookmark continuity. / active-route detection updated everywhere so Dashboard highlights when you visit the root URL.

v0.14.62

2026-05-14

Nav reorganization. Collections gets promoted to the primary top nav (replacing Wishlist's slot); Wishlist moves into the “…” more-menu. Mirrored in the mobile bottom tab bar: the 5 destinations are now Scan / Pins / Dashboard / Collections / More.
Crafted is now the default UI. New installs (no AppSetting row for ui_variant) now default to Crafted. Both the app_settings.DEFAULTS entry and the _current_ui_variant() Jinja-global fallback in main.py updated. Classic still selectable in Settings → Preferences if anything looks broken — the toggle is bidirectional, no data moves.

v0.14.61

2026-05-14

Hotfix: Crafted body background was white instead of cream. The --pico-background-color override in dispindable-tokens.css's :root block was losing to Pico v2's @media (prefers-color-scheme: light) rule (higher specificity than plain :root), so the body background stayed white-ish instead of cream while the nav (with explicit background: var(--dp-paper)) rendered correctly cream — visible split right under the nav border. Added explicit html, body { background-color: var(--dp-paper) !important } rule so the cream wins regardless of what the user's OS prefers.

v0.14.60

2026-05-14

Hotfix: Crafted nav was hiding itself. The v0.14.59 CSS rule body > nav:first-of-type { display: none !important } was meant to hide Pico's default nav, but BOTH the Pico nav and the new Crafted nav are <nav> elements under <body>, so the rule was hiding the new nav too. Template-level conditional rendering ({% if ui_variant() == "crafted" %}) already ensures only one nav renders — the CSS hide layer was redundant and wrong. Removed.
Home page (/) gets Direction B page-header. index.html was the one page I missed in v0.14.59 — everything else got the eyebrow + serif H1 + lede treatment but home still showed the plain “DisPindable Admin” heading.

v0.14.59

2026-05-14

Direction B: condensed top nav + mobile bottom tab bar. The classic 14-item nav was the #1 issue in the design audit. Crafted nav drops to 6 primary destinations on desktop (Scan / Pins / Series / Dashboard / Wishlist / Top), with the rest (Collections, New, Catalog, Activity, Validation, Backlog, Settings) tucked under a “…” more-menu. Mobile (<800px) hides the top links entirely and shows a fixed bottom tab bar with 5 destinations per the handoff design (Scan / Pins / Dashboard / Wishlist / More). Search becomes a single pill-styled input in the top right. Active-route detection via request.url.path highlights the current page. Classic users see the original 14-item nav unchanged.
Page-header pattern applied to 12 templates. New .dp-page-header CSS class encodes the Direction B top-of-page block: mono-caps eyebrow + serif H1 + lede paragraph. Applied to pins, series, collections (Folders), wishlist, scan, validation, settings, top-lists, activity, backlog, new-releases, catalog-admin, changelog. Each wrapped in {% if ui_variant() == "crafted" %} so classic users see the original headings unchanged. Lede text contextualizes each page in 1-2 sentences without being marketing-y.
Reusable list-page helpers in tokens CSS. .dp-toolbar for filter/sort bars above lists; .dp-card-grid + .dp-card-tile for the standard pin/series card grid pattern. Available to any page that opts in — future template migrations will swap inline-styled grids over to these classes incrementally.
What still uses classic layout under crafted: pin_detail.html (the 1,595-line monster — needs to be split into tabs as part of its migration); the /series/{id} checklist (needs a card-grid + completion bar treatment); /quick-add. These are the next three migration targets per handoff priority.

v0.14.58

2026-05-14

Direction B dashboard migration — matches the design handoff mockups. Stage 2 of the “Crafted” UI rollout (stage 1 in v0.14.56 was the foundation: tokens CSS + fonts + Pico overrides). When ui_variant() == "crafted", dashboard.html now renders the full Direction B layout: serif Newsreader H1 (“Your collection at a glance.”) with mono “WELCOME BACK” eyebrow, ink-navy “Quick scan” hero CTA card with gold camera icon, six-stat grid with mono-caps colored labels + serif display numbers (OWNED / WISHLIST / FOR TRADE / MISSING / TOTAL PAID / MSRP VALUE), two-column body with Set completion progress bars (gold / sage when complete) + Highlights (OLDEST / NEWEST / MOST EXPENSIVE thumbnails), Recent acquisitions 2-col grid, and Top characters as pill chips (top 3 in gold-bg accent). Original layout preserved behind {% if ui_variant() == "classic" %} for users who haven't toggled. Handler unchanged — both variants pull from the same context dict so no route logic moved.
To see it: Settings → Preferences → UI theme → Crafted → reload /dashboard. Toggle back to Classic anytime — no data moves, just templates switch. Both light + dark modes supported via the theme toggle in the nav (sun / moon icon).
Next stages (per design handoff priority): migrate scan.html next (most distinctive interaction), then pin_detail.html (the 1,595-line monster — also splitting it into tabs as part of the migration), then series checklist, wishlist, settings, validation, top lists.

v0.14.57

2026-05-14

Changelog renderer: fixed raw-HTML display bug. Entries from v0.14.46 onward were authored with embedded HTML (<strong>, <code>, —) but the _inline_md filter only handled markdown backtick-code — it escaped everything else, so tags and entities rendered as literal text on /changelog. Expanded the filter to handle: (1) whitelisted inline tags <strong> / <em> / <code> / <br>; (2) <a href="..."> with href-only attrs; (3) whitelisted HTML entities (mdash, middot, hellip, etc.); (4) double-escaped &X; → &X;; (5) markdown bold and asymmetric text</strong> forms that historical entries used; (6) backtick-code as before. XSS-safe: <script> and any non-whitelisted tag stays HTML-escaped.

v0.14.56

2026-05-14

Pin&Pop catalog extraction: price, SKU, rarity, edition size, linked pins. _parse_pinandpop_html was only pulling name / series / images / tags — missing the four labelled detail-table fields visible right on the Donald Duck Offset Frames page screenshot. Added explicit regex extraction for Original Price, SKU, Rarity, and Limited Edition of N; plus walking every <a href="/pins/N/"> in the page to capture the full “More from the series” strip into linked_pins. New id field on the scrape result (parsed from the URL) lets downstream code verify the data matches the chosen pin.
Form prefill: merge AI suggestion with Pin&Pop scrape. merged_prefill dict built BEFORE the template render. For price / SKU / rarity / edition_size / year / set_size: if the chosen AI provider left it blank, fill in from Pin&Pop's scraped value. Pin&Pop is the authoritative source for these — the AI numbers are guesses, the catalog values are facts. PinPics-imported data (via bookmarklet) is a secondary fallback. New sku column on Pin model + auto-migrate; SKU form field added between PinPics ID and Pin&Pop ID; saved on scan-save.
Set members: fold in Pin&Pop's linked_pins. Drew's Donald Duck Offset Frames example: 8 linked pins on both catalog sources, only 4 came through. Root cause: v0.14.44 restricted merge_set_members to ONLY the chosen-AI provider's list (to filter Opus's wrong Silhouette Paper Cut members in the Shan Yu case), but the chosen AI doesn't always enumerate the full set. Now after the AI merge, Pin&Pop's scraped linked_pins get folded in — dedup by pinandpop_id so AI entries that covered a member aren't duplicated, missing IDs get filled in (member with AI-supplied name + Pin&Pop-supplied ID), and net-new members get appended with source pinandpop-linked.
Fake-flag mis-attribution fix (the big one). Drew's screenshot: chosen pin = Donald Duck Pin&Pop #59070, but the “Pin&Pop flag: confirmed fakes exist” banner was showing data for Peter Pan Wendy Darling #15286 (the rejected suggestion). Root cause: pinandpop_data was fetched at ~line 7065 using the merged-vote winner's ID, but the visual-similarity pass at ~line 7245 could change chosen_pap_id to a different pin AFTER that fetch. So the stale pinandpop_data still pointed at #15286 while the chosen ID was now #59070. Two-layer fix: (1) re-fetch pinandpop_data + series_data after the chosen ID stabilizes (catalog cache makes this ~0ms on a hit); (2) defensive template-side check requires pinandpop_data.id matches catalog_ids.pinandpop_id before the fake banner renders.
Direction B "Crafted" UI theme — opt-in foundation. Drew uploaded a full design handoff from a separate session (HANDOFF.md + README.md + Direction B JSX prototypes). Implementing the migration plan in stages: stage 1 (this release) drops in the foundation. New static/dispindable-tokens.css defines the Direction B palette (warm cream paper / ink-navy / gold / sage / brick) for light + dark, plus Pico variable overrides so every existing page picks up the new colors without template changes. Google Fonts (Newsreader serif + Public Sans + JetBrains Mono) load with preconnect when active. Reusable .dp-card / .dp-pill-{gold,sage,brick} / .dp-stat-value / .dp-flash-* / .dp-section-label classes for templates that adopt the new vocabulary incrementally. Toggle in Settings → Preferences (UI theme = classic | crafted). Default is classic so existing users see no change unless they opt in. Future stages: per-route templates redesigned to use the dp-* classes (dashboard first, then scan + pin detail per the handoff's prioritized order).

v0.14.55

2026-05-14

**Hotfix: collection_detail.html 500 on render. Two chr(34) / chr(39) calls in the series-delete confirm dialogs were trying to use Python's chr() builtin, which Jinja doesn't expose in its default environment. Replaced with literal quote characters via {% set safe_name = collection.name|replace('"', '')|replace("'", "") %}. Affected the series-detail page when navigated to from /dashboard or anywhere else.

v0.14.54

2026-05-14

DPB images: the real root cause was ‘ads’ matching inside ‘uploads’. Every WordPress-hosted image is served from /wp-content/uploads/..., and the v0.14.51 substring filter rejected anything containing “ads” — which happens to be a substring of “uploads”. So 100% of DPB post images were filtered as if they were ads. The log was honestly reporting img=8-17 scanned + 0 yielded, with empty rejected_examples (because the URLs were getting through _extract_img_src and dying at _add). Rewrote the substring filter with path-anchored / extension-anchored patterns: /avatar/, gravatar.com, /icons/, icon-, /logo + logo.png|jpg|svg|webp, /ads/ (explicit path, not substring), /banners/ + banner-, wp-includes/ + /themes/ for WP chrome.
User activity log. Drew's ask: a chronological “added X pin on Y date, traded a pin, deleted series” feed, distinct from per-pin history. New ActivityEntry table + activity_log.py helper with record(entity_type, entity_id, event_type, summary, metadata=) that's called from pin/series/collection create+delete paths plus the scan-save flow. Append-only: entries persist after their entity is deleted. New /activity page (nav link between Catalog and Validation) groups events by day with type/event color-coded badges; links to surviving entities. Stats header shows total + last 7 days + counts by event type. Hooked routes: POST /pins (manual create), POST /pins/{id}/delete, POST /series, POST /series/{id}/delete, POST /collections, POST /collections/{id}/delete, POST /scan/{id}/new-pin (scan-to-pin save). Existing data isn't backfilled (no timestamp on it would be honest); only changes from this version forward appear.

v0.14.53

2026-05-14

Pin&Pop “Confirmed Fakes” ingestion. Pin&Pop surfaces confirmed-counterfeit warnings on the pin page itself — we weren't reading them. _parse_pinandpop_html() now scans for known warning patterns (“Confirmed fakes of this pin exist”, “WARNING: This pin has been counterfeited”, the “Confirmed Fakes” badge) and captures the surrounding context. New CatalogPin.fake_warning + fake_warning_text columns persist the flag. scrapper_db.lookup_scrapper_flags() now pulls Pin&Pop warnings alongside community forum citations, so the red banner on pin detail picks them up automatically. On /scan/{id}/new-pin a dedicated top-of-page red banner shows the Pin&Pop warning text before the user saves the pin. Auto-syncs on every catalog re-warm.
No-catalog-match warning. A pin with no Pin&Pop ID AND no PinPics ID is a soft fakeness signal — every legit retail pin should appear in at least one catalog. Common false positives: brand-new releases, regional / convention exclusives. New list_helpers.has_no_catalog_match() helper + amber “No catalog match” badge surfaced on the /pins list. On /scan/{id}/new-pin, when neither catalog returned data AND the local visual-search mirror has nothing close, a yellow caution banner appears. New “No catalog match” section on /validation lists every unmatched pin (separate from the needs_validation queue), capped at 100.
Backlog: UI A/B test (Claude-designed variant). New entry under UX & UI Polish. Approach: ask Claude to redesign one focused page (likely /new-pin) and ship as a togglable variant behind app_settings.ui_variant, A/B in personal use, keep the winner.

v0.14.52

2026-05-14

App-settings key-value store. New AppSetting table + app_settings.py helper module with typed getters/setters. Persists user-tunable preferences without scattering env vars. Settings page got a new “⚙️ Preferences” card driving the six items below.
Background-remove pin before visual-authenticity analysis. pin_authenticity.run_full_check now optionally routes the user's front photo through rembg before edge + color analyzers. Composited onto neutral gray so transparent regions don't pollute the color palette. Sidecar cached under .pin_auth_cache/bgrm_*.png keyed by SHA-256 so repeat checks reuse the result. AI side-by-side still uses the original photo so Gemini sees full context. Toggle on Settings (default: on if rembg installed).
Sort-by across list pages. New list_helpers.py module with per-page sort catalogs + apply_*_sort helpers. /pins, /series, /collections all gained a sort dropdown next to the page title. URL param ?sort= persists across navigation; default-sort preference per page lives in Settings.
“Recently added” NEW badge. Configurable window in Settings (default 7 days). Green NEW badge on cards across /pins, /series, /collections grid views. Pairs with recently_added as a sort option.
Hide/collapse advanced UI on /new-pin. The AI provider cards panel now defaults to collapsed; the high-signal panels (PinPics candidates, Pin&Pop candidates, set members, catalog visual matches) stay expanded. New default_expand_advanced setting flips them all open for power use.
Highest-selling pins Top List. New columns on CatalogPin: ebay_sold_median + ebay_sold_sample_count + ebay_sold_fetched_at. New catalog_ingest.refresh_ebay_prices() bulk-walks every catalog pin, queries eBay sold listings, stores the median + sample count. POST /catalog/refresh-ebay from a new button on /catalog admin (skips rows refreshed <30 days). New “🔥 Hottest pins right now” section on /top-lists pulling top 15 by sold-price median. Min 2 sold listings required so a single $400 outlier doesn't crown its pin.
Barcode / QR scanning. ZXing-js browser-side scanner loaded lazily from CDN on first use (~200KB). New reusable modal + window.openBarcodeScanner() helper in base.html so any page can call it. Buttons added to the home Quick Add area and /scan upload form (data-attribute-driven; can be added to other pages with one HTML line). On detect, scans hit GET /api/barcode-lookup which searches owned Pin rows and the CatalogPin mirror by ID + name substring, returns top 5 matches. No match → user gets a “start a scan with this code as a hint” link.
Backlog: 6 items marked DONE — background-removal-for-validation, hide/collapse, sort-by, recently-added flag, highest-selling Top List, barcode scanning. Code cleanup remains the active focus next.

v0.14.51

2026-05-14

DPB post images: walk noscript / picture / source. v0.14.49's lazy-load attribute scan didn't fix the “no image found in post” problem on /new-releases. Root cause: DPB uses WP Rocket's <noscript><img src="real.jpg"></noscript> no-JS-fallback pattern alongside the lazy-load placeholder, and BeautifulSoup treats <noscript> contents as opaque text rather than parseable HTML. New _collect_images_from_element() helper walks three paths: (1) direct <img> tags via the expanded lazy-load attribute list (now 11 attrs covering Jetpack, WP Rocket, LiteSpeed, Smush, a3 Lazy Load, Cloudflare Mirage, etc.); (2) <picture><source srcset> responsive-image wrappers, taking the largest width variant; (3) <noscript> blocks, re-parsing the inner HTML for any <img> tags. Plus a diagnostic log line per post that came up empty, with sample rejected attrs so future theme variations are easy to spot.
Title-level skip filter. Drew asked to omit podcast-episode posts from /new-releases. New _filter_items_by_title() drops any item whose title contains “podcast episode” or “dpb podcast” (case-insensitive). Applied at the return of both parse_feed() (RSS path) and parse_blog_html() (fallback path) so filtered items never hit the background enrichment queue. Easy to add more substrings to the tuple if other non-pin post types show up.
Backlog: code cleanup. New “Engineering Hygiene” category + entry covering unused modules, functions, dead conditionals, stale tests, unused templates, and CHANGELOG references to removed features. Suggested approach: batches by category, one focused version bump each, so any regression caught later is easy to bisect.

v0.14.50

2026-05-14

Backlog: four more entries.
· Highest-selling pins Top List (Pricing & Value) — new ranking on /top-lists driven by eBay sold-listing medians across the local CatalogPin mirror. Notes the implementation path (background-bulk-run + cached median + sample-count + last-fetched-at).
· Phone-camera capture flow: extensive testing (UX & UI Polish) — distinct from the general mobile UI pass; covers getUserMedia, focus/exposure on close-ups, multi-image capture, retake behavior, EXIF orientation, and the existing quality heuristic as a pre-upload warning.
· Background-remove pin before visual-authenticity analysis (Fake Detection) — Tier 1 edge + color analyzers currently include the photo background in their stats, inflating roughness on busy surfaces. Fix: route the user's front photo through rembg (already optional dep) before calling the analyzers. Catalog ref is usually already BG-free, so the comparison becomes apples-to-apples.
· Test out the new visual-authenticity validation (Tier 1 + Tier 2) (UX & UI Polish) — QA task with concrete test cases (authentic, scrapper, busy background, no reference, with back photo) for the v0.14.46 panel.

v0.14.49

2026-05-14

DPB post images: handle lazy-loaded attributes. After the v0.14.47 HTML fallback got 24 posts working, the /new-releases cards still rendered empty grey boxes where the images should be. Root cause: DPB uses a WP caching plugin that puts the real image URL in data-lazy-src / data-large-file / srcset while the src attribute carries a base64 placeholder. Rewrote parse_post()'s image extraction to walk a priority list of lazy-load attrs, parse srcset (picking the largest width), and filter out placeholder URLs (data URIs, <25-char strings, known placeholder substrings).
Backlog merge. Drew dropped a backlog list; absorbed it into the BACKLOG structure with deduplication against existing items. Marked DONE: “Better auto-checklists from set membership” (v0.14.42 + v0.14.44), “Local catalog database” (v0.14.48), “Back-stamp validation” (v0.14.46). Updated “Fake pin detection” to in-progress with explicit Tier 1/2/3 status. Added new entries: mobile testing pass, end-to-end Series-add test pass, color/contrast polish, sort-by everywhere, recently-added flag, hide/collapse advanced panels, AI scan-type rationalization, smarter AI-call caching, barcode scanning, pintradingdb.com as alt source, auto-backup (promoted from OPEN_QUESTIONS). Added done status badge class to backlog.html.

v0.14.48

2026-05-14

Local catalog mirror. New catalog_ingest.py module + CatalogPin table. Pre-warm a Pin&Pop series URL and the app fetches every pin in the series, downloads the front image to catalog_uploads/, and computes a DINOv2 embedding. Per-pin commit so a crash mid-batch doesn't lose progress. Re-running is idempotent — existing rows get last_seen_at bumped + missing fields filled in, no duplicates. Stats panel on the new /catalog page shows total count, image coverage, embedding coverage, and search-ready percentage.
Local DINOv2 visual search. New catalog_search.py module loads all CatalogPin embeddings into an in-memory numpy matrix on first use and answers nearest-neighbor queries in sub-100ms. Triggered automatically on every /scan/{id}/new-pin visit — the user's scan gets embedded and the top 5 visually-similar catalog pins above 60% cosine similarity render as a new purple panel between the existing-pin matches and the AI provider cards. Free, no API calls. Cache invalidates automatically when the underlying table grows.
Known-scrapper community-flagged DB. New scrapper_db.py module + KnownScrapper + ScrapperSource tables. Configurable list of source URLs (disneypinforum threads, Reddit JSON endpoints, generic HTML pages) that get scraped for pin-ID mentions + surrounding context. Parsers for XenForo, Reddit JSON, and generic HTML, all tolerant of small theme variations. On the pin detail page, any pin whose catalog ID matches a known scrapper entry gets a prominent red banner at the top of the visual-authenticity panel with citation link(s) and context snippet(s). Default seed sources cover the “Spotting Scrappers Warning” megathread and the r/DisneyWorld pin-trading-stations-fakes Reddit post; user can add/remove/disable sources via the /catalog admin page.
Schema: new tables catalog_pins, known_scrappers, scrapper_sources (auto-created on startup via Base.metadata.create_all). New nav link “📚 Catalog” between “New” and “Validation”.
What this doesn't do: no automatic re-warm or scheduled scrapper-source refresh — both are manual actions on the /catalog page so you control when network requests happen. No auto-override of AI identifications by visual search — matches surface as a panel for the user to consider, never as a forced verdict. Tier 3 (local trained CNN) still backlogged; the data shape now supports it but no model exists.

v0.14.47

2026-05-14

DPB blog-HTML fallback: filter out nav links. The v0.14.45 anchor-sweep Strategy 2 was grabbing WP/Woo navigation slugs (/cart/, /checkout/, /products/, /my-account/, /pin-categories/, /dpb-podcast/, /registration/, etc.) along with the actual post URLs because they all match the same “single-segment slug” regex. Two filter additions: (1) a blocklist of known nav/Woo slugs (cart, checkout, my-account, etc.); (2) a minimum-hyphen-count of 3, since DPB post slugs are descriptive (“aulani-disney-resort-ocean-canoe-pin” = 5 hyphens) and nav slugs are short (“cart”, “giveaways”). The 11 spurious [fetch] calls per /new-releases visit go away.

v0.14.46

2026-05-14

Visual authenticity check (Tier 1 + Tier 2). Complements the existing reverse-image-search authenticity module with content-based scrapper detection. Adds a new pin_authenticity.py module with two layers:
  · Tier 1 (local CV) — analyze_edge_quality() uses Pillow FIND_EDGES + numpy variance to detect outline roughness (catches sprue stubs, jagged edges). compare_colors_to_reference() extracts dominant colors from the user's photo and the catalog reference, matches in LAB color space, returns average ΔE (catches wrong-hue scrappers). No API calls, no opencv dep — pure Pillow + numpy + manual LAB math.
  · Tier 2 (Gemini Flash vision) — compare_pins_via_ai() sends the user's photo + catalog reference with a structured scrapper-detection prompt (backstamp / edges / color / fine details / © text / pin posts), returns red_flags + green_flags + 1-10 score. extract_backstamp_text() OCRs the back-of-pin photo when one is present, returning © year, manufacturer code, depth quality, and trademark presence. ~$0.001 per check; results cache by image hash.
  · Composite scoring — compute_authenticity_score() merges all findings into a 0-100 score with per-marker breakdown. Weights AI explicit red flags heavier than color drift (which is too lighting-sensitive to trust on its own). Verdict labels: likely_authentic / probably_authentic / uncertain / concerning / likely_scrapper.
Pin detail UI: visual-authenticity panel. New teal-bordered article below the existing reverse-image-search panel. Score + verdict label, per-marker breakdown cards, expandable AI review with red/green flags, expandable backstamp OCR results, and an “inputs & technical notes” section listing which photos were used so the user can verify the analysis used the right inputs. Posts to /pins/{pid}/visual-authenticity. New visual_authenticity_json + visual_authenticity_checked_at columns on Pin (auto-migrate).
Tier 3 status: the data shape was kept open-ended (breakdown is a list of {marker, verdict, score_delta, detail} tuples) so a future trained-local-model finding plugs in without schema or template changes. No empty placeholder code for Tier 3 — the forward-compat is in the shape of the data, not in dead stubs. Backlogged: wdwnt.com/tag/pin-trading/ as an alt source if DPB ever goes down for good.
DPB feed: browser-like UA fixes 403 blocks. The v0.14.43 + v0.14.45 changes still left disneypinsblog.com returning 403 across every URL candidate because Cloudflare/Wordfence blocks anything self-identifying as a bot. Switched UA to a clean Chrome string with proper Accept + Accept-Language headers. Standard practice for personal-rate hobby scrapers — the “respectful crawler” convention only works on sites that explicitly whitelist bots.

v0.14.45

2026-05-14

DPB feed: try multiple URLs + HTML fallback + diagnostics. Drew's /new-releases page kept showing “No items found in the feed” even after the v0.14.43 TTL fix, because disneypinsblog.com/feed/ was returning something the parser couldn't handle (likely HTML instead of XML, or a Cloudflare challenge page). Three changes: (1) parse_feed now tries three feed URL candidates in order — /feed/, /blog/feed/, /?feed=rss2 — and uses the first one that returns parseable XML. (2) New parse_blog_html() fallback that scrapes the /blog/ index page directly when every RSS candidate fails, recovering titles + URLs from article cards. (3) parse_feed + parse_blog_html now return diagnostics (url, HTTP status, parse outcome, item count per candidate) and the /new-releases template surfaces them in a yellow callout when no items were recovered — so you can see WHY the feed is empty instead of a generic “no items” message.

v0.14.44

2026-05-14

Original Price field readability. On /new-pin the price input was using flex: 1; min-width: 0 which let it shrink to a single-digit width whenever the price_note field next to it was visible (Drew's screenshot showed “9” instead of “19.99”). Switched to flex: 1 1 6rem; min-width: 4.5rem so the price input always shows at least 4-5 digits. price_note shrunk to 4rem fixed since it's usually 2-3 chars (“/2”, “/4”).
Delete-series button on the Series detail page. Previously the only way to delete a Series was to navigate to /pins and use the bulk delete, or to manually hit the /series/{cid}/delete endpoint. Added a collapsible “🗑️ Delete this series” button between the Custom thumbnail and Completion sections. Hard-confirm modal includes the pin count so the user knows the blast radius. Cascades to all pins in the Series via the existing SQLAlchemy cascade.
Set members panel from chosen-AI only (not all six providers). Aggregating merge_set_members across all providers showed members from non-chosen providers' (often wrong) identifications. Drew's Shan Yu case: Gemini Flash correctly identified Villain Portraits with 7 valid set members; Opus identified it as Silhouette Paper Cut and added 10 wrong set members; merged panel showed all 17. Now passes only the chosen-prefill provider's suggestion to merge_set_members so the panel matches the AI that actually identified the pin. If the chosen provider has zero set members, the panel is empty — better than showing wrong ones from another provider.
Series auto-seed: name-keyed dedup, chosen-AI only, image import. Three related fixes to _seed_collection_from_scan. (1) The old dedup was by catalog ID only — meaning a pin appearing both in the Pin&Pop series scrape (P&P ID only) and in PinPics linked_pins (PinPics ID only) got created TWICE. Rewrote to name-key dedup with substring fallback, merging IDs and image URLs across the three sources into one record per pin. (2) Now uses only the chosen-prefill provider's set_members (consistent with the panel fix). (3) Schedules background image enrichment for each seeded pin via the new _enrich_missing_pin_from_pinpics (PinPics-only pins) or extended _enrich_missing_pin_from_pinandpop(with_image=True) — downloads the front catalog photo and creates a PinImage row so the checklist thumbnail renders after a refresh. Also fixed an operator-precedence bug in the name fallback that returned “Unknown pin” for some Pin&Pop-only entries.
Per-card delete on the Series checklist. Subtle × in the top-left corner of each pin card (opacity 55% default, red on hover). Posts to /pins/{pid}/delete with a next=/series/{cid} redirect so the user comes back to the Series instead of /pins. Hard confirm that names the pin and warns about image deletion. The delete_pin handler now accepts the optional next form field (same-origin only).

v0.14.43

2026-05-14

DPB feed cache now has a 1-hour TTL. The cache_get() helper in scrape_dpb.py had no expiration — once the feed was cached on first visit, it served the same N items forever (regardless of new posts on Disney Pins Blog). Added optional ttl param; feed URLs use FEED_TTL_SECONDS = 3600, individual post HTML keeps the long TTL (14 days). Also added a “🔄 Refresh feed now” button on /new-releases that hits ?refresh=1 to bust the feed cache and re-pull immediately when you don't want to wait an hour.
Search on /series and /collections. Both pages previously had no search; the only way to find a specific Series among many was scrolling. Added a ?q= free-text filter to each route, matching across name / slug / theme / tags / notes on Series, and name / slug / notes on Collections. Search box renders below the page header, matches Pico CSS look. Empty q shows everything as before. Match count + a “clear” link surface when a query is active.
“seriess” pluralization bug. The v0.14.40 rename script swapped collection to series inside {{ count }} collection{{ "s" if count != 1 else "" }}-style template expressions — producing 0 seriess / 2 seriess on the /collections page (the plural of “series” is “series”, not “seriess”). Stripped the trailing pluralization where the noun is “series”. Five template instances plus one JS-side counter in collections.html.

v0.14.42

2026-05-14

Pack-pricing auto-detect for slash notation. PinPics and Pin&Pop both surface multi-pack pricing as Original Price $19.99 / 2 (where /2 means a 2-pack). The v0.14.38 pack-size detector only caught word patterns like “Pack of N” — every slash-notation case (Shan Yu's $19.99/2, every Mystery Box, etc.) was missed. Added _parse_pack_size_from_price() in catalog_sites.py that regexes for Original Price ... $X.XX / N directly off the scraped HTML. Returns int or None; bounded 2..50 to filter noise. Wired into both fetch_pinpics and fetch_pinandpop as a new pack_size field, and the /new-pin prefill prefers the scraped value over the text-pattern detector. PinPics cache bumped to v5.
Auto-populate Series with set members on create. The inline “+ Create new series” on /new-pin used to create an empty Series, leaving the user to manually add every other pin in the set. Now passes scan_id to /api/collections, which seeds the new Series with set members from three sources (highest-priority first): Pin&Pop series scrape, AI-merged set_members, PinPics linked_pins. Each is added as a missing-pin row (is_owned=False) with whatever name + catalog IDs the scan identified. Background enrichment runs after for members that only had a Pin&Pop ID. Status message on the form reports how many were seeded.
Validate/Refresh from URL on existing Series. New collapsible form on the Series detail page: paste a Pin&Pop series URL (or a single Pin&Pop pin URL, or PinPics pin URL) and one click scrapes set members, adding any not already in the Series as missing-pin rows. Idempotent — existing pins matched by Pin&Pop or PinPics ID are left alone, so re-running is safe when Pin&Pop adds new variants to a series after import. Pre-fills the URL field from pinandpop_url_override or source_url if either is set. POSTs to /series/{cid}/refresh-from-url.
Linked-pin thumbnails no longer leak into per-pin reference images. Two related fixes. (1) Removed the “Linked pins on PinPics” panel from the /new-pin chosen-PinPics view added in v0.14.41 — the same data is now better used for Series-create seeding (above). (2) Fixed _parse_pinpics_images() which previously walked every <img> on the page including those inside “siblings/variants” anchor tags. Those thumbnails were showing up as PinPics #5 ... #11 in the reference-image checkbox grid on /new-pin (the symptom Drew spotted — villain portraits other than the pin being cataloged). Now the parser walks each <img> to its nearest ancestor anchor and skips any whose href references a different /pin/N/ than the current pin. PinPics cache bumped (v5) to re-fetch with clean lists.

v0.14.41

2026-05-14

Gemini Flash now counted for prefill + catalog ID merging. The prefill picker and merge_catalog_ids() call site had gemini_suggestion (Pro) in their priority list but were missing gemini_flash_suggestion. So when only Flash ran (which is the default cheap-mode config), Claude won the prefill by default — even when Flash had the correct PinPics + Pin&Pop IDs. Shan Yu was the catch: Flash correctly identified the Disney Villains Framed Portrait Mystery Collection with both catalog IDs, but the form prefilled with Claude's Yzma data and the eBay query went off to the wrong pin entirely. Fixed: Flash is now second in the priority list (after Pro), and also included in merge_super_series() and merge_set_members().
Linked pins from PinPics now extracted in the regular scan flow. Previously only the bookmarklet captured siblings / variants / same-series pins (rendered under the “PinPics imported data” panel). The regular AI-scan flow only got name/year/images/back_description from fetch_pinpics(). Added _parse_pinpics_linked_pins() that mirrors the bookmarklet's linkedPins() JS: walks every <a href="/pin/N/"> with a thumbnail and surfaces up to 12 as linked pins. Rendered inside the existing “PinPics catalog match details” collapsible panel, right below the reference photos. Bumped PINPICS_CACHE_VERSION to v4 so cached entries refetch with the new field. Clear .catalog_cache/pinpics_*.json if you want immediate refresh; otherwise the existing entries just don't have linked_pins until they hit their 14-day TTL.

v0.14.40

2026-05-14

Big terminology rename: folders → collections, collections → series. Folders are now Collections (the organizational grouper). Old Collections are now Series (the pin-series-level container). Routes swapped: /folders → /collections, /collections → /series. The /api/collections endpoint is preserved for backward compat (Chrome extension contract). Form fields renamed collection_id → series_id and folder_id → collection_id; route handlers accept the new names via FastAPI alias=. Internal Python model classes (models.Folder, models.Collection) and the SQLAlchemy folder_id/collection_id FK columns stay unchanged — this is purely a UI/URL rename, no DB migration.
eBay sold-listings: fall back to “Results matching fewer words” when no strict matches. Previously the v0.14.36 divider-detection stopped the parser at eBay's broadened-keyword divider, returning zero listings for very rare pins (Shan Yu was the example). Now the parser buckets into strict (above the divider) and broadened (below). UI prefers strict; when strict is empty and broadened has matches, renders them with an amber “No exact-match sold listings — showing N approximate matches” banner + amber-bordered cards. Cache version bumped to v2 so existing entries refetch.
Disagreement-panel ghost-click bug fixed + compacted. The PinPics / Pin&Pop disagreement rows had nested <a> tags (outer override link wrapped content containing the inner deep-link). Browsers parse nested anchors as broken DOM — the visible artifact was empty click boxes between rows that hijacked the layout. Swapped outer <a> for <div role="button" tabindex="0"> with onclick + onkeydown for keyboard accessibility. Also compacted: 48px image (was 70), single-line text overflow ellipsis, similarity % shrunk and stacked tight with the badge.
**/collections page CSS bug fixed. The “Show N series” expander text was rendering one character per line because the inner <details> was a direct grid child with no explicit grid-column, so auto-placement dropped it into the 14px-wide color-dot column. Changed CSS rule from details[open] to > details so the row-level details always spans the full grid width.
“Unsorted” row is now clickable. The dashed Unsorted row at the bottom of /collections now links to /series#unsorted, scrolling to the unsorted-series section on the series page.
Mobile pass. Top nav menu would overflow horizontally on phones, forcing horizontal scroll to reach Settings / theme toggle. New @media (max-width: 768px) block: nav wraps to multiple rows, search input shrinks to 6.5rem, main padding tightened, tables get overflow-x: auto, article padding tightened on phones, disagreement-panel grid gaps reduced. A 480px breakpoint shrinks nav links further and stacks the search form vertically.

v0.14.39

2026-05-14

Backlog clearance — three long-standing items wired up.
**Smart-crop wired into /scan upload. pin_crop.py was implemented but never called. The scan POST handler now runs each uploaded image through pin_crop.process_scan_image() before writing to disk — three fallback bbox-detection strategies (saturation, core-and-expand, edge-density) followed by an 8% padding crop. The saved file IS the cropped version; downstream DINOv2 embedding sees a tight pin frame instead of pin-on-busy-carpet. Detection failure is non-fatal — the original bytes get written instead.
**Optional rembg background removal. When the rembg package is installed (commented in requirements.txt — pip install rembg to enable), the smart-crop pipeline runs a second step: foreground extraction via u2net + composite onto neutral gray (#f0f0f0). Result is a clean, uniform-background pin photo. ~170MB model download on first use. Skipped silently when rembg isn't available, so the default install stays lightweight.
**New /new-releases route. Wires the long-existing scrape_dpb.py scaffold into a browsable feed. Pulls the latest 24 items from Disney Pins Blog's RSS, then enriches each from on-disk cache (.dpb_cache/) — never blocks the page on a fresh post fetch. Uncached posts get scheduled for background fetching at 2.5s polite-delay intervals, so the next page load shows their full data. Grid view with title, lead image, pub date, categories, and any extracted price / edition size / release date. Linked from the main nav as “🎉 New”.
Pipeline composability. pin_crop.process_scan_image() takes do_crop and do_bg_removal kwargs so any callsite can opt out of either step. _save_upload takes a smart_crop kwarg; today only /scan passes True. Quick-add and other upload paths stay raw — flip the flag at the callsite to enable.

v0.14.37 and earlier

—

~143 earlier changelog entries archived in v0.15.77. Drew's session arc reached ~75+ versions deep with the in-app changelog growing past 7000 lines — over a third of main.py was historical changelog text. Pruned everything from v0.14.37 and older into this single marker. Original entries remain in git history (run git log -- main.py with the appropriate range to see them). The v0.14.38 through v0.15.77 entries are kept intact since they cover the current session arc; older context (vintage of major UI changes like the Series / Collection rename in v0.14.40, the multi-AI fan-out, the initial PTDB integration, etc.) lives in git.