fix: 429 retry, sequential batching, force UI refresh after prescreen

1. prescreener.py: classify_with_deepseek now retries on 429 with
   exponential back-off (5s → 10s → 20s → 40s, up to 4 attempts);
   same back-off also covers other transient errors.

2. main.py: prescreen batches run sequentially with a 3s gap instead
   of asyncio.gather (parallel). Parallel batches caused the second
   batch to always hit the 429 rate limit, leaving most domains
   unclassified (only the smaller last batch succeeded).

3. index.html: prescreenSelected() now clears this.domains before
   calling _fetch() so Alpine re-renders the full table with the
   updated niche/type values; also updates the notify hint to mention
   the expected 1-2 min wait.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-04-17 21:52:39 +02:00
parent a30085975e
commit 468d76387d
3 changed files with 76 additions and 57 deletions

View File

@@ -196,17 +196,17 @@ async def prescreen_batch(body: dict):
counts[s] = counts.get(s, 0) + 1
# Phase 2: DeepSeek classification for live sites only
# Run batches sequentially — parallel requests cause 429 rate-limit errors.
live = [r for r in results if r.get("prescreen_status") == "live"]
classified = 0
if live:
batches = [live[i:i + DEEPSEEK_BATCH_SIZE] for i in range(0, len(live), DEEPSEEK_BATCH_SIZE)]
batch_cls = await asyncio.gather(
*[classify_with_deepseek(b) for b in batches], return_exceptions=True
)
all_cls: list = []
for bc in batch_cls:
if isinstance(bc, list):
all_cls.extend(bc)
for i, batch in enumerate(batches):
if i > 0:
await asyncio.sleep(3) # brief gap between batches
cls = await classify_with_deepseek(batch)
all_cls.extend(cls)
if all_cls:
await save_prescreen_results(all_cls)
classified = len(all_cls)