SQLite locking:
- Enable WAL journal mode in init_db (readers don't block writers)
- Set busy_timeout=30000ms in init_db
- Add timeout=30 to every aiosqlite.connect() across db.py, validator.py,
enricher.py, main.py so connections wait up to 30s instead of crashing
Error status:
- 4xx/5xx HTTP responses are now prescreen_status='error' (server alive
but broken/blocking) instead of 'live'
- Added 'error' counter to validator stats and orange Error stat box in UI
- Added ps-error CSS class (orange) and filter option in Browse tab
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Any http failure (ConnectError, ConnectTimeout, ReadTimeout, TLS error
inside a redirect chain) now falls through to an https retry instead
of immediately marking dead.
- Use one unified 8s connect / 12s read timeout for both schemes so that
http→https redirects followed inside the same client get a full TLS
handshake window (previously 4s http timeout was too short for the
https redirect hop).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Port 80 is often firewalled (drops packets → ConnectTimeout) rather than
refused (ConnectError). Previously ConnectTimeout hit the generic except
branch and broke without trying https, marking everything dead.
Now ConnectError + RemoteProtocolError + ConnectTimeout all trigger an
https retry. ReadTimeout still marks dead (server responded on connect
but was too slow).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rotate across 7 real browser UAs to avoid bot detection
- Any 2xx/3xx/4xx/5xx response = server is UP = live (only no-response = dead)
- Parking signals still checked on 200/203 body content
- Previous 403/404 responses were incorrectly marking live servers as dead
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previous fix only retried on ConnectError. Servers that accept TCP on port 80
but hang, return protocol errors, or timeout also need the https fallback.
Now any exception on http triggers https retry. Shorter http timeout (4s)
avoids wasting time on non-responsive port 80.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds rescan_dead flag that causes _filter_unvalidated to treat
previously-dead domains as needing a fresh check. Useful after
fixing the http/https detection bug.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Many modern servers refuse HTTP connections entirely. The validator was
only trying http://, causing HTTPS-only sites to be wrongly marked dead.
Now falls back to https:// on ConnectError. Also increased timeouts slightly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
