Recycle browsers after temporary contexts

This commit is contained in:
CoolDotty
2026-07-06 00:47:32 -07:00
parent da8008d200
commit 68ad2e0f1a
9 changed files with 150 additions and 7 deletions
+5
View File
@@ -19,6 +19,8 @@ interface PoolEntry {
lastUsedAt?: number // unix timestamp
restartCount: number
healthy: boolean
temporaryContextUses: number
restartReason?: string
}
```
@@ -42,6 +44,7 @@ new BrowserPool({
poolSize: 3, // BROWSER_POOL_SIZE (default 3)
acquireTimeoutMs: 15000, // BROWSER_ACQUIRE_TIMEOUT_MS — 15s default
pollIntervalMs: 100, // how often to re-check for an idle browser
recycleAfterTemporaryContexts: 8,
})
```
@@ -51,6 +54,8 @@ When `acquireTimeoutMs` elapses, the API surfaces the rejection as **HTTP 429**
`pool.release(id)` marks the browser idle and closes all open pages. `lastDomain` is updated to the domain just served. Cookies are kept to speed up the next request to the same domain.
Tier 3 and Tier 4 create short-lived isolated contexts for fresh challenge solves and proxy escalation. Those contexts are closed by the tier code, but long-running Firefox/Camoufox processes can still retain child content processes after repeated solves. The pool tracks those temporary contexts and restarts the whole browser after `recycleAfterTemporaryContexts` uses so process growth stays bounded. Set `BROWSER_RECYCLE_AFTER_CONTEXTS=0` to disable this recycling.
## Self-healing
A health check runs every 30 seconds:
+1
View File
@@ -88,6 +88,7 @@ First run builds the web and docs images locally — takes a couple of minutes.
|----------|---------|-------------|
| `BROWSER_POOL_SIZE` | `3` | Warm browser instances |
| `BROWSER_ACQUIRE_TIMEOUT_MS` | `15000` | How long `acquire()` polls for a free browser before returning HTTP 429 |
| `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8` | Restart a browser after this many fresh/proxy contexts; set `0` to disable |
| `REDIS_URL` | `redis://redis:6379` | Redis connection (set automatically in compose) |
| `RESIDENTIAL_PROXY_URL` | — | Enables Tier 4 proxy escalation |
@@ -63,6 +63,17 @@ BROWSER_ACQUIRE_TIMEOUT_MS=30000 # tolerate longer queueing on slow targets
When the timeout fires, both `/v1` and `/scrape` return **HTTP 429** with the FlareSolverr v2 error envelope (not a 500).
### `BROWSER_RECYCLE_AFTER_CONTEXTS`
**Default:** `8`
How many fresh challenge/proxy contexts a pooled browser can create before TRAWL restarts the full browser process. Tier 3 and Tier 4 use short-lived isolated contexts so Cloudflare sees a clean profile, but some Camoufox/Firefox builds can leave content processes behind even after Playwright closes those contexts. Recycling the browser bounds that process growth without changing Redis session-cache TTLs.
```ini
BROWSER_RECYCLE_AFTER_CONTEXTS=8 # default - bound long-running browser process growth
BROWSER_RECYCLE_AFTER_CONTEXTS=0 # disable browser recycling
```
## Session Cache
### `SESSION_TTL_SECONDS`