External/trawl
1
0
Fork 0
mirror of https://github.com/germondai/trawl.git synced 2026-08-17 12:11:23 +02:00
Code Issues Packages Projects Releases Wiki Activity

docs: document BROWSER_ACQUIRE_TIMEOUT_MS and HTTP 429 on pool exhaustion

Browse Source
This commit is contained in:
germondai
2026-06-30 00:22:57 +02:00
parent efaf0c96a0
commit 59ba779d3c
9 changed files with 108 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/docs/api-reference/flaresolvr-compat.md
+13 -2
View File
@@ -129,12 +129,23 @@ curl -s -X POST http://localhost:8191/v1 \
## Error response
When the request fails, the response is still a FlareSolverr v2 envelope with `status: "error"` and an empty `solution`. The HTTP status code carries the failure class:
| Code | Meaning |
|------|---------|
| 200 | `status: "ok"` (request succeeded) |
| 400 | Malformed request body |
| 429 | Pool exhausted — all browsers busy past `BROWSER_ACQUIRE_TIMEOUT_MS` |
| 500 | Internal error |
Example — pool exhausted (HTTP 429):
```json
{
"status": "error",
"message": "timeout",
"message": "Browser pool saturated, retry shortly",
"startTimestamp": 1700000000000,
"endTimestamp": 1700000060000,
"endTimestamp": 1700000015000,
"version": "2.0.0",
"solution": {
"url": "https://nowsecure.nl",
apps/docs/api-reference/native-api.md
+34
View File
@@ -116,3 +116,37 @@ for (const t of result.timings) {
"totalMs": 600
}
```
## Error response
HTTP status codes:
| Code | Meaning |
|------|---------|
| 200 | `tier` succeeded |
| 400 | Malformed request body |
| 429 | Pool exhausted — all browsers busy past `BROWSER_ACQUIRE_TIMEOUT_MS` |
| 503 | Browser pool initializing |
| 500 | Internal error |
For 429 pool-exhaustion errors, the body is a **FlareSolverr v2 envelope** (same shape `/v1` uses) so clients can parse both endpoints uniformly:
```json
{
"status": "error",
"message": "Browser pool saturated, retry shortly",
"startTimestamp": 1700000000000,
"endTimestamp": 1700000015000,
"version": "2.0.0",
"solution": {
"url": "https://nowsecure.nl",
"status": 0,
"headers": {},
"response": "",
"cookies": [],
"userAgent": ""
}
}
```
For 400 / 503 / 500 the body is the native shape `{ "error": "Human-readable message" }`.
apps/docs/api-reference/overview.md
+8 -1
View File
@@ -36,17 +36,24 @@ Content-Type: application/json
## Error responses
All error responses follow this shape:
Most error responses follow this shape:
```json
{ "error": "Human-readable message" }
```
Pool-exhaustion errors are an exception — they return a FlareSolverr v2 envelope so `/v1` and `/scrape` produce identical bodies on saturation. See [FlareSolverr compat → Error response](/api-reference/flaresolvr-compat#error-response) for the envelope shape.
HTTP status codes:
| Code | Meaning |
|------|---------|
| 200 | Success |
| 400 | Bad request (missing/invalid fields) |
| 429 | Pool exhausted — all browsers busy past `BROWSER_ACQUIRE_TIMEOUT_MS` |
| 503 | Browser pool initializing |
| 500 | Internal error |
::: info CORS
The API does **not** emit `Access-Control-Allow-Origin` headers. TRAWL is designed for direct, same-network access (e.g. Prowlarr/Jackett, internal services, your reverse proxy). If you need browser-based cross-origin access, terminate at a proxy that adds the CORS headers you need.
:::
apps/docs/architecture/browser-pool.md
+15 -3
View File
@@ -30,11 +30,23 @@ Each entry is a `{ browser, context }` pair. `Camoufox({...})` creates the brows
1. Look for an idle browser whose `lastDomain === domain` — reuse it (its context may already have warm CF cookies)
2. Fall back to any idle browser
3. If all browsers are busy, poll every 100ms for up to 5 seconds
4. Throw `PoolExhaustedError` after 5 seconds with no idle browser
3. If all browsers are busy, poll every `pollIntervalMs` (default **100ms**) for up to `acquireTimeoutMs` (default **15000ms** = 15s)
4. Throw `PoolExhaustedError` after `acquireTimeoutMs` with no idle browser
The domain match is on the **hostname only** — `https://example.com/page1` and `https://example.com/page2` both match `example.com`.
Both thresholds are configurable via the `BrowserPool` constructor:
```typescript
new BrowserPool({
poolSize: 3, // BROWSER_POOL_SIZE (default 3)
acquireTimeoutMs: 15000, // BROWSER_ACQUIRE_TIMEOUT_MS — 15s default
pollIntervalMs: 100, // how often to re-check for an idle browser
})
```
When `acquireTimeoutMs` elapses, the API surfaces the rejection as **HTTP 429** with a FlareSolverr v2 error envelope — both on `/v1` (Prowlarr/Jackett) and on `/scrape` (native). See the [API reference](/api-reference/overview#error-responses).
## Release
`pool.release(id)` marks the browser idle and closes all open pages. `lastDomain` is updated to the domain just served. Cookies are kept to speed up the next request to the same domain.
@@ -71,7 +83,7 @@ const browser = await Camoufox({
## Memory usage
Each Camoufox instance uses ~350–500 MB. With the default pool of 5:
Each Camoufox instance uses ~350–500 MB. With the default pool of 3:
| Pool size | RAM usage (browser only) |
|-----------|--------------------------|
apps/docs/architecture/tiered-execution.md
+1 -1
View File
@@ -53,7 +53,7 @@ Reads `session:{hostname}` from Redis. If found, injects the saved cookies into
## Tier 3 — Fresh Cloudflare Challenge Solve
Acquires a browser from the pool (or waits up to 5s for one to become available). Navigates to the URL with no pre-loaded cookies. Waits for the Cloudflare challenge to resolve by polling `page.content()` every 500ms until the interstitial HTML is gone or `maxTimeout` elapses.
Acquires a browser from the pool (or waits up to `BROWSER_ACQUIRE_TIMEOUT_MS` — default 15s — for one to become available). Navigates to the URL with no pre-loaded cookies. Waits for the Cloudflare challenge to resolve by polling `page.content()` every 500ms until the interstitial HTML is gone or `maxTimeout` elapses.
On success:
- Extracts all cookies from the page context
apps/docs/deployment/docker-compose.md
+1
View File
@@ -72,6 +72,7 @@ First run builds the web and docs images locally — takes a couple of minutes.
| Variable | Default | Description |
|----------|---------|-------------|
| `BROWSER_POOL_SIZE` | `3` | Warm browser instances |
| `BROWSER_ACQUIRE_TIMEOUT_MS` | `15000` | How long `acquire()` polls for a free browser before returning HTTP 429 |
| `REDIS_URL` | `redis://redis:6379` | Redis connection (set automatically in compose) |
| `RESIDENTIAL_PROXY_URL` | — | Enables Tier 4 proxy escalation |
apps/docs/deployment/troubleshooting.md
+13 -1
View File
@@ -25,9 +25,21 @@ description: Common issues and how to fix them.
2. **`SESSION_TTL_SECONDS` set too low** — If it's shorter than Cloudflare's challenge interval, the cache expires before the next request.
3. **Domain key mismatch** — The key is the hostname only. `sub.example.com` and `www.example.com` are separate sessions.
## POST /v1 returns HTTP 429 with `status: "error"`
**Symptom:** Request returns **HTTP 429** (not 500) with a FlareSolverr v2 envelope and `message: "Browser pool saturated, retry shortly"`.
**Cause:** TRAWL polled for `BROWSER_ACQUIRE_TIMEOUT_MS` (default 15s) without finding an idle browser. With `BROWSER_POOL_SIZE=3` and 10 concurrent requests, this only fires under sustained burst pressure.
**Fixes (in order of preference):**
1. **Raise `BROWSER_ACQUIRE_TIMEOUT_MS`** if your upstream target legitimately takes >5s per scrape — bumps the queue wait before 429 fires.
2. **Raise `BROWSER_POOL_SIZE`** if you're consistently saturating — each browser uses ~350–500 MB RAM.
3. **Reduce incoming request rate** if you control the client (Prowlarr's indexer interval, etc.).
## POST /v1 returns `status: "error"` with message `"timeout"`
**Symptom:** `maxTimeout` exceeded.
**Symptom:** `maxTimeout` exceeded (per-request timeout set by the client).
**Causes:**
apps/docs/getting-started/configuration.md
+18 -1
View File
@@ -47,6 +47,22 @@ BROWSER_POOL_SIZE=8 # high-throughput (6+ GB host RAM)
> **Note:** The API container sets `shm_size: 1gb` by default. If you raise `BROWSER_POOL_SIZE` above 5, also raise `shm_size` in your `docker-compose.yml` to at least `2gb`.
### `BROWSER_ACQUIRE_TIMEOUT_MS`
**Default:** `15000` (15 seconds)
How long `BrowserPool.acquire()` will poll for a free browser before rejecting with `PoolExhaustedError`. With `BROWSER_POOL_SIZE=3` and a typical Cloudflare challenge taking 5–8s per request, the 15s default lets a full burst of 10 concurrent requests drain without any 429s.
Lower it for fail-fast client feedback (Prowlarr will see 429s sooner and retry on its own). Raise it for very heavy upstream targets or when you've bumped `BROWSER_POOL_SIZE` higher.
```ini
BROWSER_ACQUIRE_TIMEOUT_MS=5000 # fail fast — 429s after 5s
BROWSER_ACQUIRE_TIMEOUT_MS=15000 # default — absorbs a full burst on pool=3
BROWSER_ACQUIRE_TIMEOUT_MS=30000 # tolerate longer queueing on slow targets
```
When the timeout fires, both `/v1` and `/scrape` return **HTTP 429** with the FlareSolverr v2 error envelope (not a 500).
## Session Cache
### `SESSION_TTL_SECONDS`
@@ -109,7 +125,8 @@ Port the Nuxt landing page listens on.
REDIS_URL=redis://localhost:6379
# ── Browser pool ──────────────────────────────
BROWSER_POOL_SIZE=5
BROWSER_POOL_SIZE=3
BROWSER_ACQUIRE_TIMEOUT_MS=15000
SESSION_TTL_SECONDS=3600
# ── Proxies (optional) ────────────────────────