Commit Graph
9 Commits
Author SHA1 Message Date
Erik Dasque 7d3204351c fix(tiers): recognize more block/error page variants, add Tier 4 captcha parity, surface proxy/timing info
Found while running trawl against a large batch of real-world URLs: several
cases where the API returned 200 with content that was actually a blocked
page, an empty challenge stub, or Firefox's own error page. Each was a
detection gap where a tier didn't recognize the failure and reported it as a
successful scrape.

- Recognize Firefox's about:neterror/about:certerror page (browser never
  reached a server), Cloudflare's static "you have been blocked" WAF-deny
  page, and a lean CF challenge stub (blank title/body, just the bootstrap
  script) — the stub check is gated on page size since the same script
  snippet also appears on ordinary, fully-loaded CF pages as bot-management
  telemetry.
- Wire the existing isBlocked() status-code check (403/429/202) into Tiers 2
  and 3 — previously only Tier 1 checked status code, so a generic non-CF WAF
  deny that escalated to a browser tier was reported as a success.
- Bring Tier 4 up to parity with Tier 3: captcha solving and the same block
  detection. Sites that need Tier 4 for IP reputation can just as easily have
  an in-page captcha widget.
- Add proxyUsed: boolean to the response, set from the actual proxy used by
  the winning tier — previously the only signal was inferring from tier === 4,
  which doesn't distinguish "no proxy" from Tier 3's datacenter proxy.
- Attach the per-tier timings array to thrown errors via a new ScrapeError,
  and return it in /scrape's error response. The array was already being
  built in memory; it just never survived the throw, so failed requests gave
  a flat error string with no way to see which tier failed or why.
- Add process-level uncaughtException/unhandledRejection handlers. One target
  site's page threw a JS error that Camoufox/Firefox reports in a shape
  playwright-core's dispatcher doesn't expect, which crashed the entire
  process and dropped every in-flight request across all clients.
- Update the native API docs for the new response fields and error shape.

All additive — no existing fields changed shape. Full existing test suite
passes (58/58), and this is rebuilt/smoke-tested against latest dev.
2026-07-07 15:58:54 +00:00
germondai ff1804bcb7 fix(types): accept Prowlarr object form for FlareSolverrRequest.proxy 2026-07-06 13:53:53 +02:00
GermondandGitHub 8d69e660c3 Merge branch 'main' into fix/pr-8-hardening 2026-07-06 01:33:13 +02:00
germondai 7c87dc0b1a feat(types): support full HTTP method set and rename postData to body 2026-07-06 01:12:06 +02:00
germondai cd5b674c34 feat(types): add per-request proxy override field 2026-07-05 16:40:42 +02:00
whoshoe 5fde497e3a feat: add post method with postData parameter 2026-07-03 22:28:42 +08:00
germondai 990241a3ec feat(tiers): add headers field to ScrapeRequest and wire through orchestrator to tier1 2026-06-26 22:05:30 +02:00
germondai 77bd63c724 feat(api): make cmd optional on /v1, defaulting to request.get 2026-06-26 21:34:12 +02:00
germondai dddc9c1330 feat: scaffold shared types package 2026-05-31 14:22:00 +02:00