mirror of
https://github.com/germondai/trawl.git
synced 2026-08-17 12:11:23 +02:00
Recycle browsers after temporary contexts
This commit is contained in:
@@ -19,6 +19,8 @@ interface PoolEntry {
|
||||
lastUsedAt?: number // unix timestamp
|
||||
restartCount: number
|
||||
healthy: boolean
|
||||
temporaryContextUses: number
|
||||
restartReason?: string
|
||||
}
|
||||
```
|
||||
|
||||
@@ -42,6 +44,7 @@ new BrowserPool({
|
||||
poolSize: 3, // BROWSER_POOL_SIZE (default 3)
|
||||
acquireTimeoutMs: 15000, // BROWSER_ACQUIRE_TIMEOUT_MS — 15s default
|
||||
pollIntervalMs: 100, // how often to re-check for an idle browser
|
||||
recycleAfterTemporaryContexts: 8,
|
||||
})
|
||||
```
|
||||
|
||||
@@ -51,6 +54,8 @@ When `acquireTimeoutMs` elapses, the API surfaces the rejection as **HTTP 429**
|
||||
|
||||
`pool.release(id)` marks the browser idle and closes all open pages. `lastDomain` is updated to the domain just served. Cookies are kept to speed up the next request to the same domain.
|
||||
|
||||
Tier 3 and Tier 4 create short-lived isolated contexts for fresh challenge solves and proxy escalation. Those contexts are closed by the tier code, but long-running Firefox/Camoufox processes can still retain child content processes after repeated solves. The pool tracks those temporary contexts and restarts the whole browser after `recycleAfterTemporaryContexts` uses so process growth stays bounded. Set `BROWSER_RECYCLE_AFTER_CONTEXTS=0` to disable this recycling.
|
||||
|
||||
## Self-healing
|
||||
|
||||
A health check runs every 30 seconds:
|
||||
|
||||
@@ -88,6 +88,7 @@ First run builds the web and docs images locally — takes a couple of minutes.
|
||||
|----------|---------|-------------|
|
||||
| `BROWSER_POOL_SIZE` | `3` | Warm browser instances |
|
||||
| `BROWSER_ACQUIRE_TIMEOUT_MS` | `15000` | How long `acquire()` polls for a free browser before returning HTTP 429 |
|
||||
| `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8` | Restart a browser after this many fresh/proxy contexts; set `0` to disable |
|
||||
| `REDIS_URL` | `redis://redis:6379` | Redis connection (set automatically in compose) |
|
||||
| `RESIDENTIAL_PROXY_URL` | — | Enables Tier 4 proxy escalation |
|
||||
|
||||
|
||||
@@ -63,6 +63,17 @@ BROWSER_ACQUIRE_TIMEOUT_MS=30000 # tolerate longer queueing on slow targets
|
||||
|
||||
When the timeout fires, both `/v1` and `/scrape` return **HTTP 429** with the FlareSolverr v2 error envelope (not a 500).
|
||||
|
||||
### `BROWSER_RECYCLE_AFTER_CONTEXTS`
|
||||
|
||||
**Default:** `8`
|
||||
|
||||
How many fresh challenge/proxy contexts a pooled browser can create before TRAWL restarts the full browser process. Tier 3 and Tier 4 use short-lived isolated contexts so Cloudflare sees a clean profile, but some Camoufox/Firefox builds can leave content processes behind even after Playwright closes those contexts. Recycling the browser bounds that process growth without changing Redis session-cache TTLs.
|
||||
|
||||
```ini
|
||||
BROWSER_RECYCLE_AFTER_CONTEXTS=8 # default - bound long-running browser process growth
|
||||
BROWSER_RECYCLE_AFTER_CONTEXTS=0 # disable browser recycling
|
||||
```
|
||||
|
||||
## Session Cache
|
||||
|
||||
### `SESSION_TTL_SECONDS`
|
||||
|
||||
Reference in New Issue
Block a user