fix(api): gate /health on real pool capacity, not available + busy

`/health` returns 200 as soon as `pool` is non-null, which happens before
`await pool.init()` has warmed any browser — so a readiness probe on /health
passes before the process can solve anything.

The obvious fix, `available + busy > 0`, is also wrong, and fails in a much
worse way. A request that hangs mid-solve never reaches the orchestrator's
`finally`, so it never calls `release()` and its entry stays `busy` for the life
of the process. `busy` therefore counts dead entries as capacity, and /health
can report 200/"ok" indefinitely on a pool with zero usable browsers — the
failure is completely invisible to any external check.

Adds `stalled` and `live` to PoolStats:

  * an entry is `stalled` once its checkout outlives the caller's own budget
    (req.maxTimeout, threaded through acquire()) plus a grace period, so a slow
    but genuinely live request is never miscounted
  * `live` counts entries that can serve work now or are genuinely mid-request:
    idle-and-connected, plus busy-and-connected-and-not-stalled

/health now gates on `live > 0`. A fully utilised pool still reports ready, so
this does not flap under load, but a wedged one cannot report ready at all.

`isUsable()` also checks `browser.isConnected()` rather than trusting the
`healthy` flag, which is only refreshed on the 30s health-check tick and is
never refreshed at all for busy entries.

apps/api/src/config.ts

@@ -13,6 +13,10 @@ export const RECYCLE_AFTER_TEMPORARY_CONTEXTS = Number(process.env.BROWSER_RECYC
 // minimal while still allowing CF/Imperva challenges to resolve. Raise if specific
 // targets fail with empty content (rare).
 export const CONTENT_PROCESSES = Number(process.env.BROWSER_CONTENT_PROCESSES ?? "2")
+// How long a browser may stay checked out before the pool calls it wedged rather than
+// busy. A scrape's own budget is req.maxTimeout (default 60s), so 3x that is well clear
+// of anything legitimate while still catching a hung checkout within a few minutes.
+export const STALL_TIMEOUT_MS = Number(process.env.BROWSER_STALL_TIMEOUT_MS ?? "180000")
 
 // PROXY_URL / RESIDENTIAL_PROXY_URL accept a comma-separated list of proxy URLs (a single
 // URL still works — it's just a 1-element list). *_LIST_FILE is an alternative source

apps/api/src/deps.ts

@@ -8,6 +8,7 @@ import {
   REDIS_URL,
   residentialProxyPool,
   SESSION_TTL,
+  STALL_TIMEOUT_MS,
 } from "./config"
 
 // Single embedded pool — no BullMQ / worker process required.
@@ -36,6 +37,7 @@ export async function initPool() {
     acquireTimeoutMs: ACQUIRE_TIMEOUT_MS,
     recycleAfterTemporaryContexts: RECYCLE_AFTER_TEMPORARY_CONTEXTS,
     contentProcesses: CONTENT_PROCESSES,
+    stallAfterMs: STALL_TIMEOUT_MS,
   })
   await pool.init()
   pool.startHealthCheck()

apps/api/src/routes/health.ts

@@ -6,18 +6,34 @@ import { getPool } from "../deps"
 export function healthRoute() {
   return new Elysia().get("/health", ({ set }) => {
     const pool = getPool()
-    if (!pool) set.status = 503
+    const stats = pool?.getStats()
+    // `pool` is assigned before `await pool.init()` completes, so a non-null pool does
+    // not mean any browser is warm — gate readiness on real capacity instead.
+    //
+    // `available + busy > 0` looks like the right test — a busy browser is still a live
+    // browser — but it isn't. A request that hangs mid-solve never reaches the
+    // orchestrator's `finally`, so it never calls release() and its browser stays `busy`
+    // for the life of the process. `busy` therefore counts dead entries, and the pod can
+    // report 200/"ok" indefinitely with zero usable browsers.
+    //
+    // `live` excludes both restarting entries (no browser attached) and entries whose
+    // checkout has outlived the stall threshold, so it cannot be propped up by a wedge.
+    // It still counts genuinely in-flight work, so a merely saturated pool won't flap.
+    const ready = Boolean(stats && stats.live > 0)
+    if (!ready) set.status = 503
     return {
-      status: pool ? "ok" : "starting",
+      status: ready ? "ok" : "starting",
       uptime: Math.floor((Date.now() - startTime) / 1000),
       pool:
-        pool?.getStats() ??
+        stats ??
         ({
           total: 0,
           busy: 0,
           available: 0,
           restarts: 0,
           avgRestarts: 0,
+          stalled: 0,
+          live: 0,
         } satisfies PoolStats),
     }
   })

apps/api/src/routes/stats.ts

@@ -9,11 +9,15 @@ export function statsRoute() {
       available: 0,
       restarts: 0,
       avgRestarts: 0,
+      stalled: 0,
+      live: 0,
     }
     return {
       browsers: stats.total,
       available: stats.available,
       busy: stats.busy,
+      stalled: stats.stalled,
+      live: stats.live,
       restarts: stats.restarts,
       queueDepth: 0,
     }