open-design/apps/daemon/DIAGNOSTICS.md

Automatic fault diagnostics (OPEND-3397)

The daemon observes Run errors, failed terminal fallbacks, automatic/model retries, OD Next blocked outcomes, and chat/resume admission errors. Main-path API failures emitted through sendApiError additionally cover runs, projects/conversations/files, uploads, artifacts, previews and exports. Its existing desktop observability route also captures renderer/child-process crashes, unclean exits and packaged startup failures. Fatal daemon handlers synchronously register evidence; active Run markers recover daemon interruptions on restart. User-stop requests are recorded before awaiting process termination, with elapsed and last-agent-activity ages, first-model-event presence, retries and cancellation origin. A repeated request and its canceled terminal fallback share one source ID. Project cleanup and unattributed cancellations are not labeled user stops. Terminal persistence write failures (including metadata refresh failure) and explicitly invalid deliverables also produce evidence, even on execution success. Healthy successful terminal callbacks do not create incidents. An error followed by its terminal callback is one fault; independent error events keep their identities.

This is background behavior controlled by the existing metrics/content preferences, including their shared UI/CLI configuration path. No new interactive capability, Vela role, query service, or download endpoint is introduced. Manual diagnostics exports include summary/automatic-log-upload.json with queue states and loss reasons. All daemon-owned paths derive from the resolved data root; see the root AGENTS.md Daemon data directory contract for ownership and launch propagation.

Delivery and privacy

storage/diagnostic-outbox.ts owns an independent SQLite incident queue with leases and version checks. Same source IDs deduplicate, stale callbacks cannot overwrite new outcomes, and upload receipts persist before content is considered delivered. Recovery updates the same incident's recoveredAt; already captured chunks remain unchanged and the relay writes a new immutable manifest generation.

packages/diagnostics writes a gzip JSONL archive as at-most-4-MiB chunks. Text is redacted before compression. Native dumps, binary attachments, environment dumps and unrelated agents' CLI logs are excluded. The incident Run is selected explicitly; host text logs and that agent's log tails supplement it. For AMR, only the records of the incident Run and the OpenCode sessions it owns are taken from the shared $AMR_HOME/logs/agent-runtime.jsonl; AMR's per-conversation OpenCode session logs are not located yet and are noted as source_not_located. A source first listed after opting in (for example after an upgrade) gets a consent boundary at its current size on the next start instead of being omitted forever. Missing/truncated sources are reported as partial. Fault summaries include version/runtime context.

Both metrics and content must be true at registration and each transport step. Malformed preferences fail closed. Disabling cancels transport, clears pending content and invalidates its leases. File watermarks prevent a later opt-in from uploading old text; an old source whose boundary cannot be proved is omitted and marked partial. A baselined file keeps its offset when rotated to another path. The consent fence admits a replacement identity with an inherited creation time only when it matches the baseline for that same source path. It saves admitted identities in the consent state, recorded at each daemon start as well as when an incident is collected, so later rotations and daemon restarts can recognize them without accepting unrelated files with matching creation times. Already transmitted remote bytes expire under the remote policy.

Pending content expires after 7 days; delivered local content after 24 hours. The queue budgets payloads plus serialized metadata within 1 GiB, reserving 100 MiB before collection. Old delivered copies go first, then oldest pending incidents. Collection uses a 96-MiB raw envelope budget and checks compressed output against 100 MiB. Small loss counters and at most 1024 recent scrubbed tombstones retain eviction reasons without an unbounded content queue. SQLite allocation/WAL overhead is separate from the content budget. Disk/DB failures leave a host warning; hard OOM, disk failure, permanent offline devices and silent unrecognized faults cannot be guaranteed lossless.

integrations/diagnostic-relay.ts derives the relay origin from the existing object or telemetry URL. It needs no Run registration, login, Vela token or Langfuse trace. Device credentials are local mode-0600 files. Backoff honors Retry-After; each retry gets a fresh short-lived grant and preserves incident/chunk identities. Losing a previously bound device identity fails explicitly instead of claiming the old scope. Completed object references go to existing observability; telemetry failure does not block R2 delivery. These records do not change the Run SLO success calculation.

Validation and rollout

Focused tests cover durable dedupe/restart, stale leases, consent revocation and re-enabling, corrupt preferences, recovery outcomes, byte/age pruning, no-run delivery, and a lost completion receipt. The Worker contract is maintained in open-design-telemetry-worker, whose DIAGNOSTICS.md contains the verified wrangler download/reassembly commands.

Before releasing a client, enable and validate the compatible Worker with a prefix-scoped 30-day R2 lifecycle rule. Its production switch remains disabled in the change until the existing CUTOVER process authorizes activation. Dry-run builds and in-memory R2 tests are not real cloud or macOS/Windows packaged acceptance.

Experience coverage boundary

The 2026-09-22 expansion reuses existing evidence hooks, without changing SLO classification or adding relay endpoints. Inactivity ages are observations, not proof of a hang. User cancellation alone is not classified as a system failure. API summaries retain route templates and error codes, not request bodies or URLs. Telemetry/diagnostic API errors are outside the selected business route families, so delivery failures cannot recursively generate more diagnostic bundles.

The frontend bridge below supplements cards, visible failures and existing white-screen/hang detectors. Legacy handlers without a visible-error or typed result signal, unavailable-daemon host compensation, and silent unrecognized failures remain coverage gaps. Existing host crash events do not prove delivery while the daemon is unavailable. Do not claim universal failure detection.

Frontend evidence bridge

The client_experience_diagnostic branch of /api/observability/event is separate from safety analytics: it validates bounded metadata, checks BOTH consent flags, and calls the same diagnostic queue. It never falls through to captureSafety. Frontend delivery is independent of PostHog setup and accepts no free-text error, URL, stack or page content. It carries category/surface/code, optional project, conversation and Run IDs, and an observation timestamp. Server registration time owns retention; client time is only evidence.

Sources: chat card (including missing Run/code), typed failed result events, start-blocked and preview failure surfaces; failed/incomplete projects in workspace tabs, recent cards and the recent rail; existing white-screen, preview runtime/ resource errors and stuck-run detectors; file version, refresh, source loading, manual edit, template save, deployment, image export and workspace upload errors. Awaiting-input, healthy success and canceled export result events do not trigger this bridge. Manual Run cancellation remains the separate daemon lifecycle hook.

Run/code and project status identities deduplicate repeated displays across reloads at the durable daemon queue. Operation/runtime events coalesce identical metadata for 10 seconds in a bounded 100-entry browser map. Local transport retries at most twice with five-second request timeouts and bounded in-memory outstanding work; transport errors never create more diagnostics. Closing the browser can lose unacknowledged events. No persistent browser queue or unavailable-daemon host uploader is claimed. Different kinds of evidence can still create distinct incidents for one Run; full cross-layer incident merging is not implemented.