diff --git a/CHANGELOG.md b/CHANGELOG.md index d6249fb..a9cd922 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,6 +36,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 strict `ulimit -u` can push per-user thread counts toward the cap while `/health` still reports the pool as available. Recycling returns the OS to a clean state without dropping Redis-backed session cache entries. +- `BROWSER_CONTENT_PROCESSES` env var (default `2`) caps Firefox content + processes per pooled browser via the `dom.ipc.processCount` Firefox + pref. Firefox's default of 8 lets thread count climb when Tier 3/Tier 4 + churn disposable contexts (see #13). The cap bounds the leak at the + source without needing to restart the browser. + +### Changed +- `BROWSER_RECYCLE_AFTER_CONTEXTS` no longer recycles preemptively after + every N temporary contexts. The pool now recycles only when Tier 3 or + Tier 4 returns a `blocked` / `needs-js` outcome, preserving cookies, + `cf_clearance`, and warm fingerprint state across successful solves. + This eliminates the HTTP-429 storm observed in single-browser setups + where the previous "recycle every N uses" logic left the only browser + `restarting=true` for ~13s during every recycle window. See #17. ### Security - Reserved-name header denylist prevents callers from spoofing `cf_clearance` diff --git a/README.md b/README.md index 7342b35..95a23c3 100644 --- a/README.md +++ b/README.md @@ -119,7 +119,8 @@ Synology note: many Synology NAS units (DSM 7.x on J4125 / older hardware) ship | -------------------------------- | ------------------------ | ------------------------------------------------------------------------------ | | `BROWSER_POOL_SIZE` | `3` | Warm Camoufox Firefox instances | | `BROWSER_ACQUIRE_TIMEOUT_MS` | `15000` | How long `acquire()` polls for a free browser before HTTP 429 is returned | -| `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8` | Restart a browser after this many fresh/proxy contexts; set `0` to disable | +| `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8` | Recycle a browser after this many `blocked`/`needs-js` outcomes; set `0` to disable | +| `BROWSER_CONTENT_PROCESSES` | `2` | Cap Firefox content processes per browser (`dom.ipc.processCount`); lowers RAM/CPU | | `SESSION_TTL_SECONDS` | `3600` | Redis session cache TTL (seconds) | | `REDIS_URL` | `redis://localhost:6379` | Redis connection string | | `RESIDENTIAL_PROXY_URL` | — | Enables Tier 4 proxy escalation | diff --git a/apps/docs/architecture/browser-pool.md b/apps/docs/architecture/browser-pool.md index b3e32ca..c441a02 100644 --- a/apps/docs/architecture/browser-pool.md +++ b/apps/docs/architecture/browser-pool.md @@ -45,6 +45,7 @@ new BrowserPool({ acquireTimeoutMs: 15000, // BROWSER_ACQUIRE_TIMEOUT_MS — 15s default pollIntervalMs: 100, // how often to re-check for an idle browser recycleAfterTemporaryContexts: 8, + contentProcesses: 2, // BROWSER_CONTENT_PROCESSES — caps Firefox content procs }) ``` @@ -54,7 +55,12 @@ When `acquireTimeoutMs` elapses, the API surfaces the rejection as **HTTP 429** `pool.release(id)` marks the browser idle and closes all open pages. `lastDomain` is updated to the domain just served. Cookies are kept to speed up the next request to the same domain. -Tier 3 and Tier 4 create short-lived isolated contexts for fresh challenge solves and proxy escalation. Those contexts are closed by the tier code, but long-running Firefox/Camoufox processes can still retain child content processes after repeated solves. The pool tracks those temporary contexts and restarts the whole browser after `recycleAfterTemporaryContexts` uses so process growth stays bounded. Set `BROWSER_RECYCLE_AFTER_CONTEXTS=0` to disable this recycling. +Tier 3 and Tier 4 create short-lived isolated contexts for fresh challenge solves and proxy escalation. Those contexts are closed by the tier code, but long-running Firefox/Camoufox processes can still retain child content processes after repeated solves. Two complementary defenses bound this growth: + +1. **`contentProcesses` (default `2`)** caps Firefox content processes per browser at launch via the `dom.ipc.processCount` Firefox pref. This is the primary defense — bounds thread/RAM growth at the source regardless of context churn. +2. **`recycleAfterTemporaryContexts` (default `8`)** is now **recycle-on-suspect**: the orchestrator only flags a browser for recycle when Tier 3/Tier 4 returns `blocked` / `needs-js`. Successful solves preserve cookies, `cf_clearance`, and warm fingerprint state. Set `BROWSER_RECYCLE_AFTER_CONTEXTS=0` to disable this recycling. + +See issue #13 (original bug), #17 (recycle-on-suspect trade-off discussion), and the [configuration docs](/getting-started/configuration#browser_recycle_after_contexts) for tuning. ## Self-healing diff --git a/apps/docs/getting-started/configuration.md b/apps/docs/getting-started/configuration.md index 25b996f..e2e4e06 100644 --- a/apps/docs/getting-started/configuration.md +++ b/apps/docs/getting-started/configuration.md @@ -67,11 +67,22 @@ When the timeout fires, both `/v1` and `/scrape` return **HTTP 429** with the Fl **Default:** `8` -How many fresh challenge/proxy contexts a pooled browser can create before TRAWL restarts the full browser process. Tier 3 and Tier 4 use short-lived isolated contexts so Cloudflare sees a clean profile, but some Camoufox/Firefox builds can leave content processes behind even after Playwright closes those contexts. Recycling the browser bounds that process growth without changing Redis session-cache TTLs. +How many `blocked` / `needs-js` outcomes a pooled browser can produce before TRAWL restarts the full browser process. The recycle counter only increments when Tier 3 or Tier 4 reports the upstream actively rejected the browser's profile — successful solves preserve cookies, `cf_clearance`, and warm fingerprint state. This avoids the HTTP-429 storm that occurred when the pool preemptively recycled mid-flight (issue #17). ```ini -BROWSER_RECYCLE_AFTER_CONTEXTS=8 # default - bound long-running browser process growth -BROWSER_RECYCLE_AFTER_CONTEXTS=0 # disable browser recycling +BROWSER_RECYCLE_AFTER_CONTEXTS=8 # default - recycle after 8 blocked/needs-js outcomes +BROWSER_RECYCLE_AFTER_CONTEXTS=0 # disable browser recycling entirely +``` + +### `BROWSER_CONTENT_PROCESSES` + +**Default:** `2` + +Caps Firefox content processes per pooled browser via the `dom.ipc.processCount` Firefox pref. Firefox's default of 8 lets thread count climb when Tier 3 / Tier 4 churn disposable contexts (see #13). The cap bounds the leak at the source without paying the recycle cost. Raise if specific targets fail with empty content (rare). + +```ini +BROWSER_CONTENT_PROCESSES=2 # default - conservative cap, lowest RAM/CPU +BROWSER_CONTENT_PROCESSES=4 # raise if CF/Imperva challenges stall ``` ## Session Cache