mirror of https://github.com/germondai/trawl.git synced 2026-08-17 12:11:23 +02:00

Files

T

Erik Dasque 7d3204351c fix(tiers): recognize more block/error page variants, add Tier 4 captcha parity, surface proxy/timing info

Found while running trawl against a large batch of real-world URLs: several
cases where the API returned 200 with content that was actually a blocked
page, an empty challenge stub, or Firefox's own error page. Each was a
detection gap where a tier didn't recognize the failure and reported it as a
successful scrape.

- Recognize Firefox's about:neterror/about:certerror page (browser never
  reached a server), Cloudflare's static "you have been blocked" WAF-deny
  page, and a lean CF challenge stub (blank title/body, just the bootstrap
  script) — the stub check is gated on page size since the same script
  snippet also appears on ordinary, fully-loaded CF pages as bot-management
  telemetry.
- Wire the existing isBlocked() status-code check (403/429/202) into Tiers 2
  and 3 — previously only Tier 1 checked status code, so a generic non-CF WAF
  deny that escalated to a browser tier was reported as a success.
- Bring Tier 4 up to parity with Tier 3: captcha solving and the same block
  detection. Sites that need Tier 4 for IP reputation can just as easily have
  an in-page captcha widget.
- Add proxyUsed: boolean to the response, set from the actual proxy used by
  the winning tier — previously the only signal was inferring from tier === 4,
  which doesn't distinguish "no proxy" from Tier 3's datacenter proxy.
- Attach the per-tier timings array to thrown errors via a new ScrapeError,
  and return it in /scrape's error response. The array was already being
  built in memory; it just never survived the throw, so failed requests gave
  a flat error string with no way to see which tier failed or why.
- Add process-level uncaughtException/unhandledRejection handlers. One target
  site's page threw a JS error that Camoufox/Firefox reports in a shape
  playwright-core's dispatcher doesn't expect, which crashed the entire
  process and dropped every in-flight request across all clients.
- Update the native API docs for the new response fields and error shape.

All additive — no existing fields changed shape. Full existing test suite
passes (58/58), and this is rebuilt/smoke-tested against latest dev.

2026-07-07 15:58:54 +00:00

5.1 KiB

Raw Blame History

title, description

title	description
Native API	POST /scrape — the native TRAWL endpoint with full tier control.

`POST /scrape` — Native API

The native endpoint exposes TRAWL's full feature set: tier capping, session IDs, and rich timing metadata.

Request

interface ScrapeRequest {
  url: string
  maxTimeout?: number                    // ms, default 60000
  skipHttp?: boolean                     // skip Tier 1 (plain fetch), default false
  maxTier?: 1 | 2 | 3 | 4              // cap escalation at this tier
  sessionId?: string                     // sticky session override key
  headers?: Record<string, string>       // custom headers forwarded to the target
  proxy?: string                         // per-request proxy override for Tier 3/4
}

Fields

Field	Type	Default	Description
`url`	string	—	The URL to scrape
`maxTimeout`	number	60000	Max total time in milliseconds
`skipHttp`	boolean	false	Skip Tier 1 (go straight to browser)
`maxTier`	1–4	4	Never escalate beyond this tier
`sessionId`	string	hostname	Override the Redis session key
`headers`	object	—	Custom headers forwarded to the target across all tiers — see Custom Headers
`proxy`	string	—	Proxy URL used for this request's Tier 3/4 attempts instead of the configured `PROXY_URL`/`RESIDENTIAL_PROXY_URL` pool — see Configuration § Proxies

Response

interface ScrapeResult {
  url: string
  html: string
  cookies: Cookie[]
  userAgent: string
  statusCode: number
  tier: 1 | 2 | 3 | 4        // which tier succeeded
  sessionCached: boolean       // true if a cached session was used
  timings: TierResult[]        // per-tier attempt history
  totalMs: number
  captchasSolved?: string[]    // captcha types solved on the page itself (e.g. ['turnstile'])
  proxyUsed?: boolean          // true if the winning tier routed through a proxy (Tier 3 datacenter pool or Tier 4 residential pool/override)
}

interface TierResult {
  tier: 1 | 2 | 3 | 4
  status: 'success' | 'blocked' | 'needs-js' | 'timeout' | 'error' | 'skipped'
  durationMs: number
  reason?: string
}

Examples

Minimal request

curl -s -X POST http://localhost:8191/scrape \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://nowsecure.nl" }' | jq '{tier, totalMs, sessionCached}'

Force browser only (skip plain HTTP)

curl -s -X POST http://localhost:8191/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://nowsecure.nl",
    "skipHttp": true,
    "maxTier": 3
  }'

Inspect timing breakdown

const res = await fetch('http://localhost:8191/scrape', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ url: 'https://nowsecure.nl' }),
})

const result = await res.json()

console.log(`Tier used: ${result.tier}`)
console.log(`Session cached: ${result.sessionCached}`)
console.log(`Total: ${result.totalMs}ms`)

for (const t of result.timings) {
  console.log(`  Tier ${t.tier}: ${t.status} in ${t.durationMs}ms`)
}

Example response

{
  "url": "https://nowsecure.nl",
  "html": "<!DOCTYPE html>...",
  "cookies": [
    { "name": "cf_clearance", "value": "abc123...", "domain": ".nowsecure.nl", "path": "/", "expires": 1700003600, "httpOnly": false, "secure": true }
  ],
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)...",
  "statusCode": 200,
  "tier": 2,
  "sessionCached": true,
  "timings": [
    { "tier": 1, "status": "needs-js", "durationMs": 85 },
    { "tier": 2, "status": "success", "durationMs": 512 }
  ],
  "totalMs": 600
}

Error response

HTTP status codes:

Code	Meaning
200	`tier` succeeded
400	Malformed request body
429	Pool exhausted — all browsers busy past `BROWSER_ACQUIRE_TIMEOUT_MS`
503	Browser pool initializing
500	Internal error

For 429 pool-exhaustion errors, the body is a FlareSolverr v2 envelope (same shape /v1 uses) so clients can parse both endpoints uniformly:

{
  "status": "error",
  "message": "Browser pool saturated, retry shortly",
  "startTimestamp": 1700000000000,
  "endTimestamp": 1700000015000,
  "version": "2.0.0",
  "solution": {
    "url": "https://nowsecure.nl",
    "status": 0,
    "headers": {},
    "response": "",
    "cookies": [],
    "userAgent": ""
  }
}

For 400 / 503 the body is the native shape { "error": "Human-readable message" }.

For 500 errors raised after at least one tier was attempted, the body also includes the per-tier attempt history, so a failed request is still fully diagnosable from the response alone — no need to check server logs:

{
  "error": "All tiers exhausted. Last failure: http-403",
  "timings": [
    { "tier": 1, "status": "needs-js", "durationMs": 50, "reason": "cloudflare-challenge" },
    { "tier": 3, "status": "blocked", "durationMs": 2942, "reason": "http-403" },
    { "tier": 4, "status": "blocked", "durationMs": 7890, "reason": "http-403" }
  ]
}

5.1 KiB Raw Blame History Unescape Escape

POST /scrape — Native API