External/trawl
1
0
Fork 0
mirror of https://github.com/germondai/trawl.git synced 2026-08-17 12:11:23 +02:00
Code Issues Packages Projects Releases Wiki Activity

docs: update CHANGELOG, README, and docs for recycle-on-blocked and content-processes

Browse Source
This commit is contained in:
germondai
2026-07-06 20:11:08 +02:00
parent 2560f9157a
commit 444df08d00
4 changed files with 37 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+14
View File
@@ -36,6 +36,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
strict `ulimit -u` can push per-user thread counts toward the cap while
`/health` still reports the pool as available. Recycling returns the OS
to a clean state without dropping Redis-backed session cache entries.
- `BROWSER_CONTENT_PROCESSES` env var (default `2`) caps Firefox content
processes per pooled browser via the `dom.ipc.processCount` Firefox
pref. Firefox's default of 8 lets thread count climb when Tier 3/Tier 4
churn disposable contexts (see #13). The cap bounds the leak at the
source without needing to restart the browser.
### Changed
- `BROWSER_RECYCLE_AFTER_CONTEXTS` no longer recycles preemptively after
every N temporary contexts. The pool now recycles only when Tier 3 or
Tier 4 returns a `blocked` / `needs-js` outcome, preserving cookies,
`cf_clearance`, and warm fingerprint state across successful solves.
This eliminates the HTTP-429 storm observed in single-browser setups
where the previous "recycle every N uses" logic left the only browser
`restarting=true` for ~13s during every recycle window. See #17.
### Security
- Reserved-name header denylist prevents callers from spoofing `cf_clearance`
README.md
+2 -1
View File
@@ -119,7 +119,8 @@ Synology note: many Synology NAS units (DSM 7.x on J4125 / older hardware) ship
| -------------------------------- | ------------------------ | ------------------------------------------------------------------------------ |
| `BROWSER_POOL_SIZE` | `3` | Warm Camoufox Firefox instances |
| `BROWSER_ACQUIRE_TIMEOUT_MS` | `15000` | How long `acquire()` polls for a free browser before HTTP 429 is returned |
| `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8` | Restart a browser after this many fresh/proxy contexts; set `0` to disable |
| `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8` | Recycle a browser after this many `blocked`/`needs-js` outcomes; set `0` to disable |
| `BROWSER_CONTENT_PROCESSES` | `2` | Cap Firefox content processes per browser (`dom.ipc.processCount`); lowers RAM/CPU |
| `SESSION_TTL_SECONDS` | `3600` | Redis session cache TTL (seconds) |
| `REDIS_URL` | `redis://localhost:6379` | Redis connection string |
| `RESIDENTIAL_PROXY_URL` | — | Enables Tier 4 proxy escalation |
apps/docs/architecture/browser-pool.md
+7 -1
View File
@@ -45,6 +45,7 @@ new BrowserPool({
acquireTimeoutMs: 15000, // BROWSER_ACQUIRE_TIMEOUT_MS — 15s default
pollIntervalMs: 100, // how often to re-check for an idle browser
recycleAfterTemporaryContexts: 8,
contentProcesses: 2, // BROWSER_CONTENT_PROCESSES — caps Firefox content procs
})
```
@@ -54,7 +55,12 @@ When `acquireTimeoutMs` elapses, the API surfaces the rejection as **HTTP 429**
`pool.release(id)` marks the browser idle and closes all open pages. `lastDomain` is updated to the domain just served. Cookies are kept to speed up the next request to the same domain.
Tier 3 and Tier 4 create short-lived isolated contexts for fresh challenge solves and proxy escalation. Those contexts are closed by the tier code, but long-running Firefox/Camoufox processes can still retain child content processes after repeated solves. The pool tracks those temporary contexts and restarts the whole browser after `recycleAfterTemporaryContexts` uses so process growth stays bounded. Set `BROWSER_RECYCLE_AFTER_CONTEXTS=0` to disable this recycling.
Tier 3 and Tier 4 create short-lived isolated contexts for fresh challenge solves and proxy escalation. Those contexts are closed by the tier code, but long-running Firefox/Camoufox processes can still retain child content processes after repeated solves. Two complementary defenses bound this growth:
1. **`contentProcesses` (default `2`)** caps Firefox content processes per browser at launch via the `dom.ipc.processCount` Firefox pref. This is the primary defense — bounds thread/RAM growth at the source regardless of context churn.
2. **`recycleAfterTemporaryContexts` (default `8`)** is now **recycle-on-suspect**: the orchestrator only flags a browser for recycle when Tier 3/Tier 4 returns `blocked` / `needs-js`. Successful solves preserve cookies, `cf_clearance`, and warm fingerprint state. Set `BROWSER_RECYCLE_AFTER_CONTEXTS=0` to disable this recycling.
See issue #13 (original bug), #17 (recycle-on-suspect trade-off discussion), and the [configuration docs](/getting-started/configuration#browser_recycle_after_contexts) for tuning.
## Self-healing
apps/docs/getting-started/configuration.md
+14 -3
View File
@@ -67,11 +67,22 @@ When the timeout fires, both `/v1` and `/scrape` return **HTTP 429** with the Fl
**Default:** `8`
How many fresh challenge/proxy contexts a pooled browser can create before TRAWL restarts the full browser process. Tier 3 and Tier 4 use short-lived isolated contexts so Cloudflare sees a clean profile, but some Camoufox/Firefox builds can leave content processes behind even after Playwright closes those contexts. Recycling the browser bounds that process growth without changing Redis session-cache TTLs.
How many `blocked` / `needs-js` outcomes a pooled browser can produce before TRAWL restarts the full browser process. The recycle counter only increments when Tier 3 or Tier 4 reports the upstream actively rejected the browser's profile — successful solves preserve cookies, `cf_clearance`, and warm fingerprint state. This avoids the HTTP-429 storm that occurred when the pool preemptively recycled mid-flight (issue #17).
```ini
BROWSER_RECYCLE_AFTER_CONTEXTS=8 # default - bound long-running browser process growth
BROWSER_RECYCLE_AFTER_CONTEXTS=0 # disable browser recycling
BROWSER_RECYCLE_AFTER_CONTEXTS=8 # default - recycle after 8 blocked/needs-js outcomes
BROWSER_RECYCLE_AFTER_CONTEXTS=0 # disable browser recycling entirely
```
### `BROWSER_CONTENT_PROCESSES`
**Default:** `2`
Caps Firefox content processes per pooled browser via the `dom.ipc.processCount` Firefox pref. Firefox's default of 8 lets thread count climb when Tier 3 / Tier 4 churn disposable contexts (see #13). The cap bounds the leak at the source without paying the recycle cost. Raise if specific targets fail with empty content (rare).
```ini
BROWSER_CONTENT_PROCESSES=2 # default - conservative cap, lowest RAM/CPU
BROWSER_CONTENT_PROCESSES=4 # raise if CF/Imperva challenges stall
```
## Session Cache