mirror of
https://github.com/germondai/trawl.git
synced 2026-08-17 12:11:23 +02:00
fix(tiers): recognize more block/error page variants, add Tier 4 captcha parity, surface proxy/timing info
Found while running trawl against a large batch of real-world URLs: several cases where the API returned 200 with content that was actually a blocked page, an empty challenge stub, or Firefox's own error page. Each was a detection gap where a tier didn't recognize the failure and reported it as a successful scrape. - Recognize Firefox's about:neterror/about:certerror page (browser never reached a server), Cloudflare's static "you have been blocked" WAF-deny page, and a lean CF challenge stub (blank title/body, just the bootstrap script) — the stub check is gated on page size since the same script snippet also appears on ordinary, fully-loaded CF pages as bot-management telemetry. - Wire the existing isBlocked() status-code check (403/429/202) into Tiers 2 and 3 — previously only Tier 1 checked status code, so a generic non-CF WAF deny that escalated to a browser tier was reported as a success. - Bring Tier 4 up to parity with Tier 3: captcha solving and the same block detection. Sites that need Tier 4 for IP reputation can just as easily have an in-page captcha widget. - Add proxyUsed: boolean to the response, set from the actual proxy used by the winning tier — previously the only signal was inferring from tier === 4, which doesn't distinguish "no proxy" from Tier 3's datacenter proxy. - Attach the per-tier timings array to thrown errors via a new ScrapeError, and return it in /scrape's error response. The array was already being built in memory; it just never survived the throw, so failed requests gave a flat error string with no way to see which tier failed or why. - Add process-level uncaughtException/unhandledRejection handlers. One target site's page threw a JS error that Camoufox/Firefox reports in a shape playwright-core's dispatcher doesn't expect, which crashed the entire process and dropped every in-flight request across all clients. - Update the native API docs for the new response fields and error shape. All additive — no existing fields changed shape. Full existing test suite passes (58/58), and this is rebuilt/smoke-tested against latest dev.
This commit is contained in:
@@ -5,6 +5,7 @@ import {
|
||||
ProxyPool,
|
||||
RequestValidationError,
|
||||
requireContentTypeForBody,
|
||||
ScrapeError,
|
||||
SUPPORTED_METHODS,
|
||||
type SupportedMethod,
|
||||
sanitizeHeaders,
|
||||
@@ -236,6 +237,9 @@ new Elysia()
|
||||
return flareSolverrError(req.url ?? "", "Browser pool saturated, retry shortly")
|
||||
}
|
||||
set.status = 500
|
||||
if (err instanceof ScrapeError) {
|
||||
return { error: err.message, timings: err.timings }
|
||||
}
|
||||
return { error: err instanceof Error ? err.message : String(err) }
|
||||
}
|
||||
})
|
||||
@@ -248,6 +252,20 @@ initPool().catch((err) => {
|
||||
process.exit(1)
|
||||
})
|
||||
|
||||
// Camoufox (Firefox) emits page-error events in a shape playwright-core's dispatcher
|
||||
// doesn't expect for some target-page JS errors (e.g. missing `error.location`), which
|
||||
// throws inside the library's own internal event handling — outside any try/catch we
|
||||
// control, since it fires from a page-level event listener, not from our request path.
|
||||
// Without this, one target site's malformed error crashes the entire process and drops
|
||||
// every in-flight request across all clients, not just the one that triggered it.
|
||||
process.on("uncaughtException", (err) => {
|
||||
console.error("[api] uncaughtException (continuing):", err instanceof Error ? err.message : err)
|
||||
})
|
||||
|
||||
process.on("unhandledRejection", (reason) => {
|
||||
console.error("[api] unhandledRejection (continuing):", reason instanceof Error ? reason.message : reason)
|
||||
})
|
||||
|
||||
process.on("SIGTERM", async () => {
|
||||
await pool?.shutdown()
|
||||
process.exit(0)
|
||||
|
||||
Reference in New Issue
Block a user