External/trawl

Fork 0

mirror of https://github.com/germondai/trawl.git synced 2026-08-17 12:11:23 +02:00

Files

T

germondai c2ab109fe5 fix: report effective URL after redirects

2026-08-08 18:24:29 +02:00

6.8 KiB

Raw Permalink Blame History

title, description

title	description
Native API	POST /scrape — the native TRAWL endpoint with full tier control.

`POST /scrape` — Native API

The native endpoint exposes TRAWL's full feature set: tier capping, session IDs, and rich timing metadata.

Request

interface ScrapeRequest {
  url: string
  maxTimeout?: number                    // ms, default 60000
  skipHttp?: boolean                     // skip Tier 1 (plain fetch), default false
  maxTier?: 1 | 2 | 3 | 4              // cap escalation at this tier
  sessionId?: string                     // sticky session override key
  headers?: Record<string, string>       // custom headers forwarded to the target
  proxy?: string                         // per-request proxy override for Tier 3/4
}

Fields

Field	Type	Default	Description
`url`	string	—	The URL to scrape
`maxTimeout`	number	60000	Max total time in milliseconds
`skipHttp`	boolean	false	Skip Tier 1 (go straight to browser)
`maxTier`	1–4	4	Never escalate beyond this tier
`sessionId`	string	hostname	Override the Redis session key
`headers`	object	—	Custom headers forwarded to the target across all tiers — see Custom Headers
`proxy`	string	—	Proxy URL used for this request's Tier 3/4 attempts instead of the configured `PROXY_URL`/`RESIDENTIAL_PROXY_URL` pool — see Configuration § Proxies

Response

interface ScrapeResult {
  url: string                  // final URL after redirects
  html: string
  cookies: Cookie[]
  userAgent: string
  statusCode: number
  tier: 1 | 2 | 3 | 4        // which tier succeeded
  sessionCached: boolean       // true if a cached session was used
  timings: TierResult[]        // per-tier attempt history
  totalMs: number
  captchasSolved?: string[]    // captcha types solved on the page itself (e.g. ['turnstile'])
  proxyUsed?: boolean          // true if the winning tier routed through a proxy (Tier 3 datacenter pool or Tier 4 residential pool/override)
}

interface TierResult {
  tier: 1 | 2 | 3 | 4
  status: 'success' | 'blocked' | 'needs-js' | 'timeout' | 'error' | 'skipped'
  durationMs: number
  reason?: string
}

Examples

Minimal request

curl -s -X POST http://localhost:8191/scrape \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://nowsecure.nl" }' | jq '{tier, totalMs, sessionCached}'

Force browser only (skip plain HTTP)

curl -s -X POST http://localhost:8191/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://nowsecure.nl",
    "skipHttp": true,
    "maxTier": 3
  }'

Inspect timing breakdown

const res = await fetch('http://localhost:8191/scrape', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ url: 'https://nowsecure.nl' }),
})

const result = await res.json()

console.log(`Tier used: ${result.tier}`)
console.log(`Session cached: ${result.sessionCached}`)
console.log(`Total: ${result.totalMs}ms`)

for (const t of result.timings) {
  console.log(`  Tier ${t.tier}: ${t.status} in ${t.durationMs}ms`)
}

Example response

{
  "url": "https://nowsecure.nl",
  "html": "<!DOCTYPE html>...",
  "cookies": [
    { "name": "cf_clearance", "value": "abc123...", "domain": ".nowsecure.nl", "path": "/", "expires": 1700003600, "httpOnly": false, "secure": true }
  ],
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)...",
  "statusCode": 200,
  "tier": 2,
  "sessionCached": true,
  "timings": [
    { "tier": 1, "status": "needs-js", "durationMs": 85 },
    { "tier": 2, "status": "success", "durationMs": 512 }
  ],
  "totalMs": 600
}

Error response

HTTP status codes:

Code	Meaning
200	`tier` succeeded
400	Malformed request body
429	Pool exhausted — all browsers busy past `BROWSER_ACQUIRE_TIMEOUT_MS`
503	Browser pool initializing
500	Internal error

For 429 pool-exhaustion errors, the body is a FlareSolverr v2 envelope (same shape /v1 uses) so clients can parse both endpoints uniformly:

{
  "status": "error",
  "message": "Browser pool saturated, retry shortly",
  "startTimestamp": 1700000000000,
  "endTimestamp": 1700000015000,
  "version": "2.0.0",
  "solution": {
    "url": "https://nowsecure.nl",
    "status": 0,
    "headers": {},
    "response": "",
    "cookies": [],
    "userAgent": ""
  }
}

For 400 / 503 the body is the native shape { "error": "Human-readable message" }.

For 500 errors raised after at least one tier was attempted, the body also includes the per-tier attempt history, so a failed request is still fully diagnosable from the response alone — no need to check server logs:

{
  "error": "All tiers exhausted. Last failure: http-403",
  "timings": [
    { "tier": 1, "status": "needs-js", "durationMs": 50, "reason": "cloudflare-challenge" },
    { "tier": 3, "status": "blocked", "durationMs": 2942, "reason": "http-403" },
    { "tier": 4, "status": "blocked", "durationMs": 7890, "reason": "http-403" }
  ]
}

6.8 KiB Raw Permalink Blame History Unescape Escape

POST /scrape — Native API