mirror of
https://github.com/germondai/trawl.git
synced 2026-08-17 12:11:23 +02:00
Erik Dasque
7d3204351c
Found while running trawl against a large batch of real-world URLs: several cases where the API returned 200 with content that was actually a blocked page, an empty challenge stub, or Firefox's own error page. Each was a detection gap where a tier didn't recognize the failure and reported it as a successful scrape. - Recognize Firefox's about:neterror/about:certerror page (browser never reached a server), Cloudflare's static "you have been blocked" WAF-deny page, and a lean CF challenge stub (blank title/body, just the bootstrap script) — the stub check is gated on page size since the same script snippet also appears on ordinary, fully-loaded CF pages as bot-management telemetry. - Wire the existing isBlocked() status-code check (403/429/202) into Tiers 2 and 3 — previously only Tier 1 checked status code, so a generic non-CF WAF deny that escalated to a browser tier was reported as a success. - Bring Tier 4 up to parity with Tier 3: captcha solving and the same block detection. Sites that need Tier 4 for IP reputation can just as easily have an in-page captcha widget. - Add proxyUsed: boolean to the response, set from the actual proxy used by the winning tier — previously the only signal was inferring from tier === 4, which doesn't distinguish "no proxy" from Tier 3's datacenter proxy. - Attach the per-tier timings array to thrown errors via a new ScrapeError, and return it in /scrape's error response. The array was already being built in memory; it just never survived the throw, so failed requests gave a flat error string with no way to see which tier failed or why. - Add process-level uncaughtException/unhandledRejection handlers. One target site's page threw a JS error that Camoufox/Firefox reports in a shape playwright-core's dispatcher doesn't expect, which crashed the entire process and dropped every in-flight request across all clients. - Update the native API docs for the new response fields and error shape. All additive — no existing fields changed shape. Full existing test suite passes (58/58), and this is rebuilt/smoke-tested against latest dev.
172 lines
5.1 KiB
Markdown
172 lines
5.1 KiB
Markdown
---
|
||
title: Native API
|
||
description: POST /scrape — the native TRAWL endpoint with full tier control.
|
||
---
|
||
|
||
# `POST /scrape` — Native API
|
||
|
||
The native endpoint exposes TRAWL's full feature set: tier capping, session IDs, and rich timing metadata.
|
||
|
||
## Request
|
||
|
||
```typescript
|
||
interface ScrapeRequest {
|
||
url: string
|
||
maxTimeout?: number // ms, default 60000
|
||
skipHttp?: boolean // skip Tier 1 (plain fetch), default false
|
||
maxTier?: 1 | 2 | 3 | 4 // cap escalation at this tier
|
||
sessionId?: string // sticky session override key
|
||
headers?: Record<string, string> // custom headers forwarded to the target
|
||
proxy?: string // per-request proxy override for Tier 3/4
|
||
}
|
||
```
|
||
|
||
### Fields
|
||
|
||
| Field | Type | Default | Description |
|
||
|-------|------|---------|-------------|
|
||
| `url` | string | — | The URL to scrape |
|
||
| `maxTimeout` | number | 60000 | Max total time in milliseconds |
|
||
| `skipHttp` | boolean | false | Skip Tier 1 (go straight to browser) |
|
||
| `maxTier` | 1–4 | 4 | Never escalate beyond this tier |
|
||
| `sessionId` | string | hostname | Override the Redis session key |
|
||
| `headers` | object | — | Custom headers forwarded to the target across all tiers — see [Custom Headers](/api-reference/custom-headers) |
|
||
| `proxy` | string | — | Proxy URL used for this request's Tier 3/4 attempts instead of the configured `PROXY_URL`/`RESIDENTIAL_PROXY_URL` pool — see [Configuration § Proxies](/getting-started/configuration#proxies) |
|
||
|
||
## Response
|
||
|
||
```typescript
|
||
interface ScrapeResult {
|
||
url: string
|
||
html: string
|
||
cookies: Cookie[]
|
||
userAgent: string
|
||
statusCode: number
|
||
tier: 1 | 2 | 3 | 4 // which tier succeeded
|
||
sessionCached: boolean // true if a cached session was used
|
||
timings: TierResult[] // per-tier attempt history
|
||
totalMs: number
|
||
captchasSolved?: string[] // captcha types solved on the page itself (e.g. ['turnstile'])
|
||
proxyUsed?: boolean // true if the winning tier routed through a proxy (Tier 3 datacenter pool or Tier 4 residential pool/override)
|
||
}
|
||
|
||
interface TierResult {
|
||
tier: 1 | 2 | 3 | 4
|
||
status: 'success' | 'blocked' | 'needs-js' | 'timeout' | 'error' | 'skipped'
|
||
durationMs: number
|
||
reason?: string
|
||
}
|
||
```
|
||
|
||
## Examples
|
||
|
||
### Minimal request
|
||
|
||
```bash
|
||
curl -s -X POST http://localhost:8191/scrape \
|
||
-H "Content-Type: application/json" \
|
||
-d '{ "url": "https://nowsecure.nl" }' | jq '{tier, totalMs, sessionCached}'
|
||
```
|
||
|
||
### Force browser only (skip plain HTTP)
|
||
|
||
```bash
|
||
curl -s -X POST http://localhost:8191/scrape \
|
||
-H "Content-Type: application/json" \
|
||
-d '{
|
||
"url": "https://nowsecure.nl",
|
||
"skipHttp": true,
|
||
"maxTier": 3
|
||
}'
|
||
```
|
||
|
||
### Inspect timing breakdown
|
||
|
||
```javascript
|
||
const res = await fetch('http://localhost:8191/scrape', {
|
||
method: 'POST',
|
||
headers: { 'Content-Type': 'application/json' },
|
||
body: JSON.stringify({ url: 'https://nowsecure.nl' }),
|
||
})
|
||
|
||
const result = await res.json()
|
||
|
||
console.log(`Tier used: ${result.tier}`)
|
||
console.log(`Session cached: ${result.sessionCached}`)
|
||
console.log(`Total: ${result.totalMs}ms`)
|
||
|
||
for (const t of result.timings) {
|
||
console.log(` Tier ${t.tier}: ${t.status} in ${t.durationMs}ms`)
|
||
}
|
||
```
|
||
|
||
### Example response
|
||
|
||
```json
|
||
{
|
||
"url": "https://nowsecure.nl",
|
||
"html": "<!DOCTYPE html>...",
|
||
"cookies": [
|
||
{ "name": "cf_clearance", "value": "abc123...", "domain": ".nowsecure.nl", "path": "/", "expires": 1700003600, "httpOnly": false, "secure": true }
|
||
],
|
||
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)...",
|
||
"statusCode": 200,
|
||
"tier": 2,
|
||
"sessionCached": true,
|
||
"timings": [
|
||
{ "tier": 1, "status": "needs-js", "durationMs": 85 },
|
||
{ "tier": 2, "status": "success", "durationMs": 512 }
|
||
],
|
||
"totalMs": 600
|
||
}
|
||
```
|
||
|
||
## Error response
|
||
|
||
HTTP status codes:
|
||
|
||
| Code | Meaning |
|
||
|------|---------|
|
||
| 200 | `tier` succeeded |
|
||
| 400 | Malformed request body |
|
||
| 429 | Pool exhausted — all browsers busy past `BROWSER_ACQUIRE_TIMEOUT_MS` |
|
||
| 503 | Browser pool initializing |
|
||
| 500 | Internal error |
|
||
|
||
For 429 pool-exhaustion errors, the body is a **FlareSolverr v2 envelope** (same shape `/v1` uses) so clients can parse both endpoints uniformly:
|
||
|
||
```json
|
||
{
|
||
"status": "error",
|
||
"message": "Browser pool saturated, retry shortly",
|
||
"startTimestamp": 1700000000000,
|
||
"endTimestamp": 1700000015000,
|
||
"version": "2.0.0",
|
||
"solution": {
|
||
"url": "https://nowsecure.nl",
|
||
"status": 0,
|
||
"headers": {},
|
||
"response": "",
|
||
"cookies": [],
|
||
"userAgent": ""
|
||
}
|
||
}
|
||
```
|
||
|
||
For 400 / 503 the body is the native shape `{ "error": "Human-readable message" }`.
|
||
|
||
For 500 errors raised after at least one tier was attempted, the body also includes the
|
||
per-tier attempt history, so a failed request is still fully diagnosable from the response
|
||
alone — no need to check server logs:
|
||
|
||
```json
|
||
{
|
||
"error": "All tiers exhausted. Last failure: http-403",
|
||
"timings": [
|
||
{ "tier": 1, "status": "needs-js", "durationMs": 50, "reason": "cloudflare-challenge" },
|
||
{ "tier": 3, "status": "blocked", "durationMs": 2942, "reason": "http-403" },
|
||
{ "tier": 4, "status": "blocked", "durationMs": 7890, "reason": "http-403" }
|
||
]
|
||
}
|
||
```
|