diff --git a/apps/docs/api-reference/flaresolvr-compat.md b/apps/docs/api-reference/flaresolvr-compat.md index 982ccf1..41898ca 100644 --- a/apps/docs/api-reference/flaresolvr-compat.md +++ b/apps/docs/api-reference/flaresolvr-compat.md @@ -129,12 +129,23 @@ curl -s -X POST http://localhost:8191/v1 \ ## Error response +When the request fails, the response is still a FlareSolverr v2 envelope with `status: "error"` and an empty `solution`. The HTTP status code carries the failure class: + +| Code | Meaning | +|------|---------| +| 200 | `status: "ok"` (request succeeded) | +| 400 | Malformed request body | +| 429 | Pool exhausted — all browsers busy past `BROWSER_ACQUIRE_TIMEOUT_MS` | +| 500 | Internal error | + +Example — pool exhausted (HTTP 429): + ```json { "status": "error", - "message": "timeout", + "message": "Browser pool saturated, retry shortly", "startTimestamp": 1700000000000, - "endTimestamp": 1700000060000, + "endTimestamp": 1700000015000, "version": "2.0.0", "solution": { "url": "https://nowsecure.nl", diff --git a/apps/docs/api-reference/native-api.md b/apps/docs/api-reference/native-api.md index edd5b88..65271eb 100644 --- a/apps/docs/api-reference/native-api.md +++ b/apps/docs/api-reference/native-api.md @@ -116,3 +116,37 @@ for (const t of result.timings) { "totalMs": 600 } ``` + +## Error response + +HTTP status codes: + +| Code | Meaning | +|------|---------| +| 200 | `tier` succeeded | +| 400 | Malformed request body | +| 429 | Pool exhausted — all browsers busy past `BROWSER_ACQUIRE_TIMEOUT_MS` | +| 503 | Browser pool initializing | +| 500 | Internal error | + +For 429 pool-exhaustion errors, the body is a **FlareSolverr v2 envelope** (same shape `/v1` uses) so clients can parse both endpoints uniformly: + +```json +{ + "status": "error", + "message": "Browser pool saturated, retry shortly", + "startTimestamp": 1700000000000, + "endTimestamp": 1700000015000, + "version": "2.0.0", + "solution": { + "url": "https://nowsecure.nl", + "status": 0, + "headers": {}, + "response": "", + "cookies": [], + "userAgent": "" + } +} +``` + +For 400 / 503 / 500 the body is the native shape `{ "error": "Human-readable message" }`. diff --git a/apps/docs/api-reference/overview.md b/apps/docs/api-reference/overview.md index 2076a32..04ad31a 100644 --- a/apps/docs/api-reference/overview.md +++ b/apps/docs/api-reference/overview.md @@ -36,17 +36,24 @@ Content-Type: application/json ## Error responses -All error responses follow this shape: +Most error responses follow this shape: ```json { "error": "Human-readable message" } ``` +Pool-exhaustion errors are an exception — they return a FlareSolverr v2 envelope so `/v1` and `/scrape` produce identical bodies on saturation. See [FlareSolverr compat → Error response](/api-reference/flaresolvr-compat#error-response) for the envelope shape. + HTTP status codes: | Code | Meaning | |------|---------| | 200 | Success | | 400 | Bad request (missing/invalid fields) | +| 429 | Pool exhausted — all browsers busy past `BROWSER_ACQUIRE_TIMEOUT_MS` | | 503 | Browser pool initializing | | 500 | Internal error | + +::: info CORS +The API does **not** emit `Access-Control-Allow-Origin` headers. TRAWL is designed for direct, same-network access (e.g. Prowlarr/Jackett, internal services, your reverse proxy). If you need browser-based cross-origin access, terminate at a proxy that adds the CORS headers you need. +::: diff --git a/apps/docs/architecture/browser-pool.md b/apps/docs/architecture/browser-pool.md index 5891041..ae5ae06 100644 --- a/apps/docs/architecture/browser-pool.md +++ b/apps/docs/architecture/browser-pool.md @@ -30,11 +30,23 @@ Each entry is a `{ browser, context }` pair. `Camoufox({...})` creates the brows 1. Look for an idle browser whose `lastDomain === domain` — reuse it (its context may already have warm CF cookies) 2. Fall back to any idle browser -3. If all browsers are busy, poll every 100ms for up to 5 seconds -4. Throw `PoolExhaustedError` after 5 seconds with no idle browser +3. If all browsers are busy, poll every `pollIntervalMs` (default **100ms**) for up to `acquireTimeoutMs` (default **15000ms** = 15s) +4. Throw `PoolExhaustedError` after `acquireTimeoutMs` with no idle browser The domain match is on the **hostname only** — `https://example.com/page1` and `https://example.com/page2` both match `example.com`. +Both thresholds are configurable via the `BrowserPool` constructor: + +```typescript +new BrowserPool({ + poolSize: 3, // BROWSER_POOL_SIZE (default 3) + acquireTimeoutMs: 15000, // BROWSER_ACQUIRE_TIMEOUT_MS — 15s default + pollIntervalMs: 100, // how often to re-check for an idle browser +}) +``` + +When `acquireTimeoutMs` elapses, the API surfaces the rejection as **HTTP 429** with a FlareSolverr v2 error envelope — both on `/v1` (Prowlarr/Jackett) and on `/scrape` (native). See the [API reference](/api-reference/overview#error-responses). + ## Release `pool.release(id)` marks the browser idle and closes all open pages. `lastDomain` is updated to the domain just served. Cookies are kept to speed up the next request to the same domain. @@ -71,7 +83,7 @@ const browser = await Camoufox({ ## Memory usage -Each Camoufox instance uses ~350–500 MB. With the default pool of 5: +Each Camoufox instance uses ~350–500 MB. With the default pool of 3: | Pool size | RAM usage (browser only) | |-----------|--------------------------| diff --git a/apps/docs/architecture/tiered-execution.md b/apps/docs/architecture/tiered-execution.md index ef745d1..932444c 100644 --- a/apps/docs/architecture/tiered-execution.md +++ b/apps/docs/architecture/tiered-execution.md @@ -53,7 +53,7 @@ Reads `session:{hostname}` from Redis. If found, injects the saved cookies into ## Tier 3 — Fresh Cloudflare Challenge Solve -Acquires a browser from the pool (or waits up to 5s for one to become available). Navigates to the URL with no pre-loaded cookies. Waits for the Cloudflare challenge to resolve by polling `page.content()` every 500ms until the interstitial HTML is gone or `maxTimeout` elapses. +Acquires a browser from the pool (or waits up to `BROWSER_ACQUIRE_TIMEOUT_MS` — default 15s — for one to become available). Navigates to the URL with no pre-loaded cookies. Waits for the Cloudflare challenge to resolve by polling `page.content()` every 500ms until the interstitial HTML is gone or `maxTimeout` elapses. On success: - Extracts all cookies from the page context diff --git a/apps/docs/deployment/docker-compose.md b/apps/docs/deployment/docker-compose.md index 52bdafe..145f764 100644 --- a/apps/docs/deployment/docker-compose.md +++ b/apps/docs/deployment/docker-compose.md @@ -72,6 +72,7 @@ First run builds the web and docs images locally — takes a couple of minutes. | Variable | Default | Description | |----------|---------|-------------| | `BROWSER_POOL_SIZE` | `3` | Warm browser instances | +| `BROWSER_ACQUIRE_TIMEOUT_MS` | `15000` | How long `acquire()` polls for a free browser before returning HTTP 429 | | `REDIS_URL` | `redis://redis:6379` | Redis connection (set automatically in compose) | | `RESIDENTIAL_PROXY_URL` | — | Enables Tier 4 proxy escalation | diff --git a/apps/docs/deployment/troubleshooting.md b/apps/docs/deployment/troubleshooting.md index 38d2d6e..c3ff607 100644 --- a/apps/docs/deployment/troubleshooting.md +++ b/apps/docs/deployment/troubleshooting.md @@ -25,9 +25,21 @@ description: Common issues and how to fix them. 2. **`SESSION_TTL_SECONDS` set too low** — If it's shorter than Cloudflare's challenge interval, the cache expires before the next request. 3. **Domain key mismatch** — The key is the hostname only. `sub.example.com` and `www.example.com` are separate sessions. +## POST /v1 returns HTTP 429 with `status: "error"` + +**Symptom:** Request returns **HTTP 429** (not 500) with a FlareSolverr v2 envelope and `message: "Browser pool saturated, retry shortly"`. + +**Cause:** TRAWL polled for `BROWSER_ACQUIRE_TIMEOUT_MS` (default 15s) without finding an idle browser. With `BROWSER_POOL_SIZE=3` and 10 concurrent requests, this only fires under sustained burst pressure. + +**Fixes (in order of preference):** + +1. **Raise `BROWSER_ACQUIRE_TIMEOUT_MS`** if your upstream target legitimately takes >5s per scrape — bumps the queue wait before 429 fires. +2. **Raise `BROWSER_POOL_SIZE`** if you're consistently saturating — each browser uses ~350–500 MB RAM. +3. **Reduce incoming request rate** if you control the client (Prowlarr's indexer interval, etc.). + ## POST /v1 returns `status: "error"` with message `"timeout"` -**Symptom:** `maxTimeout` exceeded. +**Symptom:** `maxTimeout` exceeded (per-request timeout set by the client). **Causes:** diff --git a/apps/docs/getting-started/configuration.md b/apps/docs/getting-started/configuration.md index 1fa7cef..aba9a84 100644 --- a/apps/docs/getting-started/configuration.md +++ b/apps/docs/getting-started/configuration.md @@ -47,6 +47,22 @@ BROWSER_POOL_SIZE=8 # high-throughput (6+ GB host RAM) > **Note:** The API container sets `shm_size: 1gb` by default. If you raise `BROWSER_POOL_SIZE` above 5, also raise `shm_size` in your `docker-compose.yml` to at least `2gb`. +### `BROWSER_ACQUIRE_TIMEOUT_MS` + +**Default:** `15000` (15 seconds) + +How long `BrowserPool.acquire()` will poll for a free browser before rejecting with `PoolExhaustedError`. With `BROWSER_POOL_SIZE=3` and a typical Cloudflare challenge taking 5–8s per request, the 15s default lets a full burst of 10 concurrent requests drain without any 429s. + +Lower it for fail-fast client feedback (Prowlarr will see 429s sooner and retry on its own). Raise it for very heavy upstream targets or when you've bumped `BROWSER_POOL_SIZE` higher. + +```ini +BROWSER_ACQUIRE_TIMEOUT_MS=5000 # fail fast — 429s after 5s +BROWSER_ACQUIRE_TIMEOUT_MS=15000 # default — absorbs a full burst on pool=3 +BROWSER_ACQUIRE_TIMEOUT_MS=30000 # tolerate longer queueing on slow targets +``` + +When the timeout fires, both `/v1` and `/scrape` return **HTTP 429** with the FlareSolverr v2 error envelope (not a 500). + ## Session Cache ### `SESSION_TTL_SECONDS` @@ -109,7 +125,8 @@ Port the Nuxt landing page listens on. REDIS_URL=redis://localhost:6379 # ── Browser pool ────────────────────────────── -BROWSER_POOL_SIZE=5 +BROWSER_POOL_SIZE=3 +BROWSER_ACQUIRE_TIMEOUT_MS=15000 SESSION_TTL_SECONDS=3600 # ── Proxies (optional) ──────────────────────── diff --git a/apps/web/app/components/CtaSection.vue b/apps/web/app/components/CtaSection.vue index 8c4d85e..2ad237f 100644 --- a/apps/web/app/components/CtaSection.vue +++ b/apps/web/app/components/CtaSection.vue @@ -61,7 +61,7 @@ const tabs: { id: Tab; label: string; hint: string }[] = [
- single service · pool size 1 · no redis + single service · pool size 1 · 15s acquire wait · no redis 
services:
   trawl:
@@ -69,7 +69,8 @@ const tabs: { id: Tab; label: string; hint: string }[] = [
     ports: ["8191:8191"]
     shm_size: 1gb
     environment:
-      BROWSER_POOL_SIZE: 1
+ BROWSER_POOL_SIZE: 1 + BROWSER_ACQUIRE_TIMEOUT_MS: 15000
@@ -87,6 +88,7 @@ const tabs: { id: Tab; label: string; hint: string }[] = [ environment: REDIS_URL: redis://redis:6379 BROWSER_POOL_SIZE: 3 + BROWSER_ACQUIRE_TIMEOUT_MS: 15000 depends_on: [redis] volumes: redis_data: @@ -110,6 +112,7 @@ const tabs: { id: Tab; label: string; hint: string }[] = [ environment: REDIS_URL: redis://redis:6379 BROWSER_POOL_SIZE: 5 + BROWSER_ACQUIRE_TIMEOUT_MS: 15000 depends_on: [redis] healthcheck: test: wget -qO- http://localhost:8191/health