Merge pull request #37 from funkypenguin/fix/bound-restart-and-reclaim-stalled

This commit is contained in:
germondai
2026-07-27 02:40:41 +02:00
14 changed files with 650 additions and 51 deletions
+12 -4
View File
@@ -46,22 +46,26 @@ Full system health check. Used by Docker Compose health checks and monitoring sy
"busy": 1,
"available": 4,
"restarts": 0,
"avgRestarts": 0
"avgRestarts": 0,
"stalled": 0,
"live": 5
}
}
```
| Field | Type | Description |
| ------------------ | ------ | ---------------------------------------- |
| `status` | `"ok"` | Always `"ok"` when the API is reachable |
| `status` | string | `"ok"` when the pool has live capacity; otherwise `"starting"` |
| `uptime` | number | Seconds since the API process started |
| `pool.total` | number | Total browser instances in the pool |
| `pool.busy` | number | Browsers currently processing a request |
| `pool.available` | number | Browsers ready to accept a request |
| `pool.restarts` | number | Total browser restarts since worker boot |
| `pool.avgRestarts` | number | Average restarts per browser |
| `pool.stalled` | number | Checked-out browsers past their deadline |
| `pool.live` | number | Connected, non-stalled browser capacity |
Pool stats are read directly from the browser pool. If the pool hasn't initialised yet, pool values will be zero.
`/health` returns HTTP 503 while the pool is warming up or has no live browser capacity. A saturated but healthy pool remains ready because active, connected requests still count as live.
### Curl
@@ -82,7 +86,9 @@ Lightweight public stats for dashboards and landing pages.
"browsers": 5,
"available": 4,
"busy": 1,
"restarts": 0
"restarts": 0,
"stalled": 0,
"live": 5
}
```
@@ -92,6 +98,8 @@ Lightweight public stats for dashboards and landing pages.
| `available` | number | Idle browsers |
| `busy` | number | Browsers in use |
| `restarts` | number | Total browser restarts since startup |
| `stalled` | number | Checked-out browsers past their deadline |
| `live` | number | Connected, non-stalled browser capacity |
### Curl
+6 -11
View File
@@ -45,7 +45,10 @@ new BrowserPool({
acquireTimeoutMs: 15000, // BROWSER_ACQUIRE_TIMEOUT_MS — 15s default
pollIntervalMs: 100, // how often to re-check for an idle browser
recycleAfterTemporaryContexts: 8,
contentProcesses: 2, // BROWSER_CONTENT_PROCESSES — caps Firefox content procs
contentProcesses: 2, // BROWSER_CONTENT_PROCESSES — caps Firefox content procs
stallAfterMs: 180000, // BROWSER_STALL_TIMEOUT_MS
closeTimeoutMs: 10000, // BROWSER_CLOSE_TIMEOUT_MS
launchTimeoutMs: 90000, // BROWSER_LAUNCH_TIMEOUT_MS
})
```
@@ -64,17 +67,9 @@ See issue #13 (original bug), #17 (recycle-on-suspect trade-off discussion), and
## Self-healing
A health check runs every 30 seconds:
A health check runs every 30 seconds. Disconnected idle browsers are relaunched in place, and checkouts that exceed the request budget plus `BROWSER_STALL_TIMEOUT_MS` are reclaimed. Lease tokens prevent a late release from an abandoned request from freeing a replacement checkout.
```typescript
for (const entry of this.entries) {
if (entry.busy) continue
const connected = entry.browser?.isConnected() ?? false
if (!connected) await this.restartEntry(entry)
}
```
`browser.isConnected()` is a synchronous check. A disconnected browser is relaunched in place. `restartCount` increments so you can monitor via `/health`.
Browser/context close and browser launch operations are bounded by `BROWSER_CLOSE_TIMEOUT_MS` and `BROWSER_LAUNCH_TIMEOUT_MS`. This keeps a wedged Firefox process from leaving a pool entry permanently stuck in restart. `/health` reports 503 when no connected, non-stalled capacity remains.
## Why Camoufox Firefox, not Chromium?
@@ -85,6 +85,16 @@ BROWSER_CONTENT_PROCESSES=2 # default - conservative cap, lowest RAM/CPU
BROWSER_CONTENT_PROCESSES=4 # raise if CF/Imperva challenges stall
```
### Browser recovery timeouts
| Variable | Default | Purpose |
| --- | ---: | --- |
| `BROWSER_STALL_TIMEOUT_MS` | `180000` | Grace period after a request's own timeout before its browser checkout is reclaimed |
| `BROWSER_CLOSE_TIMEOUT_MS` | `10000` | Maximum wait for a wedged browser or context to close |
| `BROWSER_LAUNCH_TIMEOUT_MS` | `90000` | Maximum wait for Camoufox to launch |
These bounds keep an unresponsive Firefox process from permanently consuming a pool slot. The defaults are suitable for most installations.
## Session Cache
### `SESSION_TTL_SECONDS`