mirror of
https://github.com/germondai/trawl.git
synced 2026-08-17 12:11:23 +02:00
Merge pull request #37 from funkypenguin/fix/bound-restart-and-reclaim-stalled
This commit is contained in:
@@ -46,22 +46,26 @@ Full system health check. Used by Docker Compose health checks and monitoring sy
|
||||
"busy": 1,
|
||||
"available": 4,
|
||||
"restarts": 0,
|
||||
"avgRestarts": 0
|
||||
"avgRestarts": 0,
|
||||
"stalled": 0,
|
||||
"live": 5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Description |
|
||||
| ------------------ | ------ | ---------------------------------------- |
|
||||
| `status` | `"ok"` | Always `"ok"` when the API is reachable |
|
||||
| `status` | string | `"ok"` when the pool has live capacity; otherwise `"starting"` |
|
||||
| `uptime` | number | Seconds since the API process started |
|
||||
| `pool.total` | number | Total browser instances in the pool |
|
||||
| `pool.busy` | number | Browsers currently processing a request |
|
||||
| `pool.available` | number | Browsers ready to accept a request |
|
||||
| `pool.restarts` | number | Total browser restarts since worker boot |
|
||||
| `pool.avgRestarts` | number | Average restarts per browser |
|
||||
| `pool.stalled` | number | Checked-out browsers past their deadline |
|
||||
| `pool.live` | number | Connected, non-stalled browser capacity |
|
||||
|
||||
Pool stats are read directly from the browser pool. If the pool hasn't initialised yet, pool values will be zero.
|
||||
`/health` returns HTTP 503 while the pool is warming up or has no live browser capacity. A saturated but healthy pool remains ready because active, connected requests still count as live.
|
||||
|
||||
### Curl
|
||||
|
||||
@@ -82,7 +86,9 @@ Lightweight public stats for dashboards and landing pages.
|
||||
"browsers": 5,
|
||||
"available": 4,
|
||||
"busy": 1,
|
||||
"restarts": 0
|
||||
"restarts": 0,
|
||||
"stalled": 0,
|
||||
"live": 5
|
||||
}
|
||||
```
|
||||
|
||||
@@ -92,6 +98,8 @@ Lightweight public stats for dashboards and landing pages.
|
||||
| `available` | number | Idle browsers |
|
||||
| `busy` | number | Browsers in use |
|
||||
| `restarts` | number | Total browser restarts since startup |
|
||||
| `stalled` | number | Checked-out browsers past their deadline |
|
||||
| `live` | number | Connected, non-stalled browser capacity |
|
||||
|
||||
### Curl
|
||||
|
||||
|
||||
@@ -45,7 +45,10 @@ new BrowserPool({
|
||||
acquireTimeoutMs: 15000, // BROWSER_ACQUIRE_TIMEOUT_MS — 15s default
|
||||
pollIntervalMs: 100, // how often to re-check for an idle browser
|
||||
recycleAfterTemporaryContexts: 8,
|
||||
contentProcesses: 2, // BROWSER_CONTENT_PROCESSES — caps Firefox content procs
|
||||
contentProcesses: 2, // BROWSER_CONTENT_PROCESSES — caps Firefox content procs
|
||||
stallAfterMs: 180000, // BROWSER_STALL_TIMEOUT_MS
|
||||
closeTimeoutMs: 10000, // BROWSER_CLOSE_TIMEOUT_MS
|
||||
launchTimeoutMs: 90000, // BROWSER_LAUNCH_TIMEOUT_MS
|
||||
})
|
||||
```
|
||||
|
||||
@@ -64,17 +67,9 @@ See issue #13 (original bug), #17 (recycle-on-suspect trade-off discussion), and
|
||||
|
||||
## Self-healing
|
||||
|
||||
A health check runs every 30 seconds:
|
||||
A health check runs every 30 seconds. Disconnected idle browsers are relaunched in place, and checkouts that exceed the request budget plus `BROWSER_STALL_TIMEOUT_MS` are reclaimed. Lease tokens prevent a late release from an abandoned request from freeing a replacement checkout.
|
||||
|
||||
```typescript
|
||||
for (const entry of this.entries) {
|
||||
if (entry.busy) continue
|
||||
const connected = entry.browser?.isConnected() ?? false
|
||||
if (!connected) await this.restartEntry(entry)
|
||||
}
|
||||
```
|
||||
|
||||
`browser.isConnected()` is a synchronous check. A disconnected browser is relaunched in place. `restartCount` increments so you can monitor via `/health`.
|
||||
Browser/context close and browser launch operations are bounded by `BROWSER_CLOSE_TIMEOUT_MS` and `BROWSER_LAUNCH_TIMEOUT_MS`. This keeps a wedged Firefox process from leaving a pool entry permanently stuck in restart. `/health` reports 503 when no connected, non-stalled capacity remains.
|
||||
|
||||
## Why Camoufox Firefox, not Chromium?
|
||||
|
||||
|
||||
@@ -85,6 +85,16 @@ BROWSER_CONTENT_PROCESSES=2 # default - conservative cap, lowest RAM/CPU
|
||||
BROWSER_CONTENT_PROCESSES=4 # raise if CF/Imperva challenges stall
|
||||
```
|
||||
|
||||
### Browser recovery timeouts
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
| --- | ---: | --- |
|
||||
| `BROWSER_STALL_TIMEOUT_MS` | `180000` | Grace period after a request's own timeout before its browser checkout is reclaimed |
|
||||
| `BROWSER_CLOSE_TIMEOUT_MS` | `10000` | Maximum wait for a wedged browser or context to close |
|
||||
| `BROWSER_LAUNCH_TIMEOUT_MS` | `90000` | Maximum wait for Camoufox to launch |
|
||||
|
||||
These bounds keep an unresponsive Firefox process from permanently consuming a pool slot. The defaults are suitable for most installations.
|
||||
|
||||
## Session Cache
|
||||
|
||||
### `SESSION_TTL_SECONDS`
|
||||
|
||||
Reference in New Issue
Block a user