fix(browser): bound every await in restartEntry; reclaim stalled checkouts

`BrowserPool.restartEntry` awaited `context.close()`, `browser.close()` and the
Camoufox launch with no timeout on any of them. Camoufox hangs on close when a
content process is wedged — tiers/3.ts and tiers/4.ts already guard their
*temporary* contexts against exactly this with a 5s `Promise.race` — but the
persistent context and browser the pool owns had no such guard, and launches can
hang too.

When any one of those hangs, the entry is pinned at `restarting = true` forever.
From then on the health check hits its own `if (entry.restarting) return` guard,
so every 30s tick logs "browser N disconnected, restarting" and does nothing.
The pool silently loses that slot permanently: `restartCount` never increments,
so the restart counter sits frozen while the log implies furious activity. With
enough uptime every entry ends up in this state and the pool is inert.

Changes:

  * every await in `restartEntry`, `init()` and `shutdown()` is bounded. On
    timeout the entry is left unhealthy with `restarting` cleared, so the next
    health-check tick retries it from scratch instead of wedging.
  * `runHealthCheck` reclaims checkouts past their deadline. Previously busy
    entries were skipped entirely, so an entry whose request wedged was never
    examined again.
  * a per-checkout `lease`, returned on the handle and passed back to
    `release()`, so a request that outlives its checkout cannot free — or
    recycle, via `noteTemporaryContext` — a browser the pool has since handed to
    someone else.
  * `release()` hands its in-flight page closes to `restartEntry` rather than
    racing them, since closing a context underneath in-flight `page.close()`
    calls is one way to wedge the transport in the first place.
  * abandoned launches are counted and capped. A timeout can only stop *waiting*
    for a launch, not cancel it, so retrying without a cap could pile up hung
    Firefox processes; past the cap the entry stays down and `live` reflects it.

Timeouts are configurable (`closeTimeoutMs`, `launchTimeoutMs`, `stallAfterMs`,
`healthIntervalMs`) with the API exposing them as BROWSER_*_MS env vars.

Adds regression tests for the hung close, the hung launch, stall accounting,
budget-aware stall deadlines, disconnected-but-busy entries, and stale releases.
The hung-close and hung-launch tests both fail against the unpatched pool.

apps/api/src/config.ts

@@ -17,6 +17,12 @@ export const CONTENT_PROCESSES = Number(process.env.BROWSER_CONTENT_PROCESSES ??
 // busy. A scrape's own budget is req.maxTimeout (default 60s), so 3x that is well clear
 // of anything legitimate while still catching a hung checkout within a few minutes.
 export const STALL_TIMEOUT_MS = Number(process.env.BROWSER_STALL_TIMEOUT_MS ?? "180000")
+// Upper bound on a single browser/context close during a recycle. Camoufox can hang on
+// close when a content process is wedged; past this we abandon the close and relaunch.
+export const CLOSE_TIMEOUT_MS = Number(process.env.BROWSER_CLOSE_TIMEOUT_MS ?? "10000")
+// Upper bound on a browser launch. A cold Camoufox start is a few seconds, but launches
+// have been observed to hang indefinitely — without a bound that strands the pool entry.
+export const LAUNCH_TIMEOUT_MS = Number(process.env.BROWSER_LAUNCH_TIMEOUT_MS ?? "90000")
 
 // PROXY_URL / RESIDENTIAL_PROXY_URL accept a comma-separated list of proxy URLs (a single
 // URL still works — it's just a 1-element list). *_LIST_FILE is an alternative source

apps/api/src/deps.ts

@@ -1,7 +1,9 @@
 import { BrowserPool, SessionCache } from "@trawl/browser"
 import {
   ACQUIRE_TIMEOUT_MS,
+  CLOSE_TIMEOUT_MS,
   CONTENT_PROCESSES,
+  LAUNCH_TIMEOUT_MS,
   POOL_SIZE,
   proxyPool,
   RECYCLE_AFTER_TEMPORARY_CONTEXTS,
@@ -38,6 +40,8 @@ export async function initPool() {
     recycleAfterTemporaryContexts: RECYCLE_AFTER_TEMPORARY_CONTEXTS,
     contentProcesses: CONTENT_PROCESSES,
     stallAfterMs: STALL_TIMEOUT_MS,
+    closeTimeoutMs: CLOSE_TIMEOUT_MS,
+    launchTimeoutMs: LAUNCH_TIMEOUT_MS,
   })
   await pool.init()
   pool.startHealthCheck()
@@ -49,8 +53,8 @@ export function getDeps() {
   const p = pool
   const sc = sessionCache
   return {
-    acquireBrowser: (d: string) => p.acquire(d),
-    releaseBrowser: (id: number) => p.release(id),
+    acquireBrowser: (d: string, budgetMs?: number) => p.acquire(d, budgetMs),
+    releaseBrowser: (id: number, lease?: number) => p.release(id, lease),
     // Session cache ops are no-ops when Redis is unavailable
     loadSession: (d: string) => (sc ? sc.load(d).catch(() => null) : Promise.resolve(null)),
     saveSession: (d: string, data: unknown) => (sc ? sc.save(d, data as never).catch(() => {}) : Promise.resolve()),