mirror of
https://github.com/block/buzz.git
synced 2026-08-18 06:50:31 +02:00
fa1a5b1a79
## Problem Sharing compute with a large model (e.g. `gemma-4-26B`) put the desktop app into a **restart loop**: toggle Share → app appears to "download" / stall → the whole app restarts → repeat. Small models (E4B) were unaffected, which made it look model-specific and flaky. It is not model-specific and not flaky. It is a **false-positive liveness check**. ## Root cause (proven by black-box measurement) A `serve` node's OpenAI ingress (`:9337`) serializes **all** HTTP — including the `/v1/models` liveness probe — behind the current in-flight inference. It is *also* HTTP-unresponsive during model load and package-layer download. In every one of those phases the node is alive and progressing, but it cannot answer an HTTP probe. Measured on a standalone `gemma-4-26B` node (randomized ~30k-token prompt, cache-miss): | during one ~30s inference | result | |---|---| | concurrent `GET /v1/models` | **27.0s**, then 200 | | concurrent small `/chat/completions` | **28.8s**, then 200 | | `tcp_connect(:9337)` throughout | **~0ms** | Both HTTP calls simply queued behind the turn; TCP kept accepting instantly. A probe with any timeout shorter than the turn reads the node as dead. Buzz then acted on that false "dead" reading in two places, **both restart paths added in #2823**: 1. **Ingress watchdog** — after 2 consecutive `/v1/models` timeouts, evicts the node; for a serve node eviction means `app.request_restart()`. Two dead probes landing inside a prefill window → restart loop. 2. **Start / restore paths** — on a `wait_for_mesh_inference` timeout, `stop()` the node and (fresh start) `request_restart()` the app "to guarantee cleanup" — even though the node was still loading weights or downloading layers. This is the exact line in the incident log: `started node failed inference readiness … Buzz is restarting`. ## Fix Treat a **bound TCP port as alive**. Death has exactly one unambiguous signal: a *closed* port. - **Watchdog** (`recovery.rs`): only `PortClosed` may evict. A bound-but-HTTP-unresponsive `Unhealthy` port is never evicted, at any probe streak or urgency. Closed-port eviction is unchanged. - **Start / restore** (`commands/mesh_llm.rs`): install the runtime **before** probing readiness (so it is always tracked by `AppState` and can never be orphaned — which is what the restart was guarding against), and on a readiness timeout **leave it warming up** instead of stopping/restarting. Launch-restoration stays disarmed until real inference is confirmed, so a genuinely broken start is retried next launch rather than silently disabling Share Compute. ### What this deliberately does *not* do Detecting a node that is bound-but-internally-wedged needs a liveness signal that bypasses the inference lock. There is none today, so this fix cannot distinguish "wedged" from "busy" and errs toward not restarting. That gap is a mesh-llm bug, filed upstream: **Mesh-LLM/mesh-llm#1126** (lock-free `/live`+`/ready` on the ingress). A follow-up here can consume it once it lands. ## Tests - Watchdog never evicts a bound/busy port at any probe streak or urgency (the regression). - Closed-port eviction still fires (dead listener still reclaimed). - Black-box: a listener that accepts TCP then stalls HTTP classifies as `Unhealthy`, not `PortClosed`. - **Mutation-proven**: reverting the eviction rule to the old count-based logic fails the busy-node test. `cargo test` (desktop, `--features mesh-llm`) green, fmt + clippy clean. ## Not covered here The intermittent nature means I could not force the live loop deterministically on a warm machine; the proof is the measured serialization + the mutation-proven unit/black-box tests. Live behaviour (app no longer restarts while a 26B node loads/serves) still merits a manual check before merge. --------- Signed-off-by: Michael Neale <michael.neale@gmail.com> Co-authored-by: Michael Neale <michael.neale@gmail.com>