fix(tiers): recognize more block/error page variants, add Tier 4 captcha parity, surface proxy/timing info

Found while running trawl against a large batch of real-world URLs: several
cases where the API returned 200 with content that was actually a blocked
page, an empty challenge stub, or Firefox's own error page. Each was a
detection gap where a tier didn't recognize the failure and reported it as a
successful scrape.

- Recognize Firefox's about:neterror/about:certerror page (browser never
  reached a server), Cloudflare's static "you have been blocked" WAF-deny
  page, and a lean CF challenge stub (blank title/body, just the bootstrap
  script) — the stub check is gated on page size since the same script
  snippet also appears on ordinary, fully-loaded CF pages as bot-management
  telemetry.
- Wire the existing isBlocked() status-code check (403/429/202) into Tiers 2
  and 3 — previously only Tier 1 checked status code, so a generic non-CF WAF
  deny that escalated to a browser tier was reported as a success.
- Bring Tier 4 up to parity with Tier 3: captcha solving and the same block
  detection. Sites that need Tier 4 for IP reputation can just as easily have
  an in-page captcha widget.
- Add proxyUsed: boolean to the response, set from the actual proxy used by
  the winning tier — previously the only signal was inferring from tier === 4,
  which doesn't distinguish "no proxy" from Tier 3's datacenter proxy.
- Attach the per-tier timings array to thrown errors via a new ScrapeError,
  and return it in /scrape's error response. The array was already being
  built in memory; it just never survived the throw, so failed requests gave
  a flat error string with no way to see which tier failed or why.
- Add process-level uncaughtException/unhandledRejection handlers. One target
  site's page threw a JS error that Camoufox/Firefox reports in a shape
  playwright-core's dispatcher doesn't expect, which crashed the entire
  process and dropped every in-flight request across all clients.
- Update the native API docs for the new response fields and error shape.

All additive — no existing fields changed shape. Full existing test suite
passes (58/58), and this is rebuilt/smoke-tested against latest dev.
This commit is contained in:
Erik Dasque
2026-07-07 15:58:54 +00:00
parent d7476274a2
commit 7d3204351c
9 changed files with 144 additions and 11 deletions
+18
View File
@@ -5,6 +5,7 @@ import {
ProxyPool,
RequestValidationError,
requireContentTypeForBody,
ScrapeError,
SUPPORTED_METHODS,
type SupportedMethod,
sanitizeHeaders,
@@ -236,6 +237,9 @@ new Elysia()
return flareSolverrError(req.url ?? "", "Browser pool saturated, retry shortly")
}
set.status = 500
if (err instanceof ScrapeError) {
return { error: err.message, timings: err.timings }
}
return { error: err instanceof Error ? err.message : String(err) }
}
})
@@ -248,6 +252,20 @@ initPool().catch((err) => {
process.exit(1)
})
// Camoufox (Firefox) emits page-error events in a shape playwright-core's dispatcher
// doesn't expect for some target-page JS errors (e.g. missing `error.location`), which
// throws inside the library's own internal event handling — outside any try/catch we
// control, since it fires from a page-level event listener, not from our request path.
// Without this, one target site's malformed error crashes the entire process and drops
// every in-flight request across all clients, not just the one that triggered it.
process.on("uncaughtException", (err) => {
console.error("[api] uncaughtException (continuing):", err instanceof Error ? err.message : err)
})
process.on("unhandledRejection", (reason) => {
console.error("[api] unhandledRejection (continuing):", reason instanceof Error ? reason.message : reason)
})
process.on("SIGTERM", async () => {
await pool?.shutdown()
process.exit(0)