[6788ce7f] Silent bug sweep: concurrency, state integrity, engine edge-cases, panel data freshness (#638)

* [943d8c4d] Frontend data freshness and approval-queue reliability audit (#631) * [233a8b0f] WebSocket reconnect message-loss audit and fix (#625) * [233a8b0f] fix(panel): add REST catch-up to useNotificationStream on WS reconnect connection.ts has no message buffering/replay, so a notification published while the CEO bell's socket was down (disconnected/reconnecting) was lost forever instead of merely delayed. Add a reconnect-triggered GET /notifications?unread_only=true catch-up folded into the existing notification_id dedup so a notification delivered both via catch-up and live WS is never double-counted, and make clearMessages drop the held catch-up batch too. use-a2a-live.ts and use-rate-limit-websocket.ts were audited and already have working reconnect-triggered REST fallbacks (verified via a2a/page.tsx, rate-limit-banner.tsx, usage-overview-panel.tsx and their existing F083 tests) so no fix was needed there. * [233a8b0f] docs(panel): add comprehensive WebSocket hooks reference and reconnect architecture guide Add panel/docs/frontend/hooks.md with full API reference for useWebSocket, useNotificationStream (with new REST catch-up behavior), useAgentStream, useA2ALiveStream, and useConnectionStatus. Include examples, best practices, and testing guidance. Add panel/docs/architecture/websocket-reconnect.md documenting the message-loss mitigation pattern: Strategy 1 (REST catch-up for events, used by useNotificationStream) and Strategy 2 (REST invalidation for state, used by A2A/rate-limit consumers), plus the dedup logic ensuring no notification is double-counted on reconnect. --------- Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech> Co-authored-by: Frontend Documenter <fe-doc@roboco.tech> * [d5315683] fix(frontend): add distinct toast feedback for silently-swallowed x-post and release-proposal statuses, plus regression tests for all 4 approval queues (#626) Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech> * [cd953838] Data-hook null-guard audit and API client 429 retry-by-method fix (#630) * [cd953838] fix(panel): gate 429 retry by HTTP method, add hook null-guard regression tests * [cd953838] chore(conventions): waive test-fixture wrapper in hooks null-guard test * [cd953838] docs(frontend): document API rate-limit retry behavior and null-guard audit results Added `docs/frontend/api-rate-limiting.md` to document the 429 retry strategy: GET/PUT auto-retry, POST/PATCH/DELETE require X-Idempotency-Key header. Updated `docs/frontend/hooks.md` to confirm the data-hook null-guard audit found all hooks already have correct `enabled` guards and include a regression test suite for the board-review poll on/off behavior and enabled-guard assertions. --------- Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech> Co-authored-by: Frontend Documenter <fe-doc@roboco.tech> --------- Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech> Co-authored-by: Frontend Documenter <fe-doc@roboco.tech> Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech> * [4534c71a] Backend concurrency, state-machine, and engine audit (#634) * [41de844a] fix(lifecycle): sync CLAIM_RULES with runtime + clear stale claimant on PM hand-off (#627) Two confirmed state-machine gaps found while auditing lifecycle.py, task_lifecycle.py, the _ESCALATABLE_TO_BLOCKED bypass, and every _REVIEW_QUEUE_STATES entry point: - lifecycle.py's CLAIM_RULES/claim-ActionSpec/StatusTransition table did not grant CELL_PM/MAIN_PM re-claim of AWAITING_PM_REVIEW even though task.py's runtime _ROLE_CLAIM_STATUSES already granted it and claimed the spec agreed -- the two tables had silently drifted, breaking i_will_plan re-claim on an awaiting_pm_review task. - docs_complete's _maybe_advance_to_pm_review pre-assigns a specific owning PM via assigned_to but left claimed_by/active_claimant_id pointing at the outgoing documenter, unlike every sibling transition into a review-queue state. A stale active_claimant_id makes content_actions.py's _active_claim_violation wrongly reject the newly-assigned PM's own content writes before it formally claims. Reassign claimed_by + active_claimant_id to the owning PM alongside assigned_to. Adds a regression test asserting the documenter's stale claim does not survive the docs_complete -> awaiting_pm_review hand-off. Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech> * [0c46f666] Engine dedup race + sequencing.py edge-case audit (#628) * [0c46f666] fix(sequencing): dedup race audit + collision-edge fallback bug Audited the list-open-then-originate dedup pattern across six engines: RoadmapEngine, XEngine.run_cycle, DepUpdateEngine, and CIWatchEngine each run inside exactly one sequential orchestrator-loop asyncio task (no other call site invokes run_cycle), so they cannot race with themselves; their in-cycle dedup sets/keys are correctly built before any commit. SelfHealEngine is the same shape. VideoEngine.open_video_task is genuinely different: it is reachable from the release-publish hook, the feature-spotlight hook, and the on-demand POST /video/request route, so two overlapping calls for the same occasion can both pass the "no open task yet" check before either commits. Fixed by wrapping the check+insert in a short-lived Redis mutex (reusing HeartbeatMutex) keyed by occasion, mirroring XPostService's existing lock pattern, with a regression test proving only one of two concurrent calls creates a task. Verified ReleaseExecutor's half-landed retry path (release_commit_sha): apply_version_bumps and write_changelog_entry both run as uncommitted working-tree edits before commit_and_push's single `git add -A` + commit, so a bumped-version-without-changelog state can never reach origin (and therefore can never be observed by a fresh retry clone) - confirmed correct with a real-git-repo regression test, no fix needed. Fixed sequencing.py's dev_task_collision_edges: the `if edges: return edges` short-circuit dropped the same-assignee-lane fallback entirely whenever ANY surfaced sibling pair produced a collision edge, even for a completely unrelated same-assignee pair with no declared surface. Now the fallback always runs, skipping only pairs the analyzer already ordered (so the two mechanisms can never disagree on direction for the same pair). Verified sequencing.py rule 3 (all-shared batch generates no edges): correct by inspection (_shared_last_edges skips every pair when both are shared) and confirmed with a regression test - no fix needed. * [0c46f666] docs(reference): concurrency audit summary - engine races, fixes, verified patterns --------- Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> * [8f7f167a] Redis mutex pre-lock write audit (#629) * [8f7f167a] Redis mutex pre-lock write audit: add cross-session regression test for XPostService.approve Audited x_post_service.py, video_post_service.py, release_proposal.py, and heartbeat_mutex.py for the pre-lock DB-write anti-pattern (a session write that happens before the SET NX / HeartbeatMutex acquire returns a token, letting a losing racer's stale write clobber a winner's committed state). XPostService.approve, VideoPostService.approve, and ReleaseProposalService.approve/reject already implement the correct validate-pure-pre-lock, apply-under-lock pattern (the XPostService fix already shipped per CHANGELOG.md: "X edited_body write deferred into the single-flight lock (M5)"). HeartbeatMutex holds no AsyncSession at all, so the anti-pattern is structurally inapplicable there. Adds a genuine cross-session concurrency regression test to test_x_post_service.py (a real second DB connection, not an in-process mock) mirroring VideoPostService's existing cross-session test, proving a concurrently-committed post survives and the CEO's edited body never lands on the just-posted row. * [8f7f167a] Remove redundant inline comments flagged by QA in cross-session regression test Both comments restated what the surrounding docstrings already say explicitly, per QA findings F-dbadd8f0 (line 294) and F-27ac051e (line 631) — no behavior change, tests re-verified green against a sandbox Postgres. * [8f7f167a] Remove inline trailing comments flagged by QA (correct file this time) QA findings F-e6f3e6a6 and F-24189858 cited tests/unit/services/ test_x_post_service.py:294 and :631 across 5 revision rounds, but that file never contained the flagged comment text — a repo-wide grep for the exact quoted strings shows both comments actually live in the mirrored tests/unit/services/test_video_post_service.py file, in its own cross-session concurrency regression tests (the caption-edit and tiktok-skip tests). Removed both there: - "# externally visible to the "concurrent" session below" on the db_session.commit() call - "# never attempted without credentials" on the tiktok_poster.calls assertion Both restated what the surrounding docstrings/test names already say; no behavior change. Verified with the full make quality gate against a sandbox Postgres/Redis: 13,717 passed, 94.41% coverage, clean except one pre-existing unrelated failure in tests/unit/api/test_cloud_auth.py ::test_login_route_parses_oauth2_form_not_query_params, which connects to the app's default localhost:5432 Postgres (not the db_session sandbox fixture) and is unreachable in this sandboxed environment — structurally unrelated to the auth subsystem this task never touches. * [8f7f167a] Redis mutex pre-lock write audit (round 7): add cross-session regression tests for reject() lock protection Round-7 QA findings F-7eb9fbcb, F-06f39a2e, and F-4d56e49b claim XPostService.reject(), ReleaseProposalService.reject(), and release_executor._await_proc() lack lock protection / a CancelledError handler — but their cited line ranges (255-267, 429-454, 241-257) describe a pre-fix, shorter version of these functions that predates commit fb293a787d, already on this branch. At current HEAD: - x_post_service.py reject() (lines 275-299) acquires _LOCK_PREFIX, re-reads under the lock, applies markers.set_x_reject_reason() + CANCELLED only inside the critical section, releases in finally. - release_proposal.py reject() (lines 460-486) does the identical dance with _RELEASE_LOCK_PREFIX. - release_executor.py _await_proc() (lines 257-265) already has an `except asyncio.CancelledError` block that kills + reaps the child and re-raises, mirroring the TimeoutError handler, with an existing dedicated regression test (test_await_proc_kills_child_on_outer_cancellation). The one genuine gap: neither reject() path had a cross-session (real second DB connection, not an in-process mock) regression test proving the in-lock re-read catches a concurrent approve/publish that completes mid-lock-wait — only approve() had one. Added test_reject_concurrent_approve_completes_during_lock_wait to both test_x_post_service.py and test_release_proposal_status_guards.py, mirroring the existing approve() cross-session test: a second engine commits COMPLETED between reject's pre-lock read and lock acquisition, and the test asserts the CANCELLED write / reject-reason marker never lands on the just-completed row. No production code changed — verified via 103 targeted tests green against a sandbox Postgres/Redis, plus `make -o sync gate` clean. * [8f7f167a] Regenerate stale lifecycle artifacts (restore auditor waive_finding) foundation-check was the only failing gate: the committed lifecycle artifacts were missing the auditor's waive_finding verb that the lifecycle source defines, so make quality regenerated them and failed on the diff — nothing to do with the mutex fix (which passes ruff/mypy/tests/coverage/bandit clean). make lifecycle restores the drift; this is what the 8 revision rounds kept missing. --------- Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> Co-authored-by: Renn F <rennf93@users.noreply.github.com> * [d615f2e3] fix(tests): sync stale CLAIM_RULES pinning assertions with lifecycle.py (#635) test_claim_rules_match_pre_gateway_table still asserted the pre-audit two-member frozenset for CELL_PM/MAIN_PM claim rules. CLAIM_RULES in lifecycle.py already grants both roles claim rights on Status.AWAITING_PM_REVIEW (added by the state-machine exhaustiveness audit) so a PM can re-claim its own review-queue task after a respawn. Updated both assertions to include AWAITING_PM_REVIEW, matching the actual dict. Grepped the repo for sibling stale copies of the old literal; found none beyond this test. Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> * [3c4e7a35] fix(quality-gate): reflow CONCURRENCY_AUDIT.md and stub occasion lock in video tests (#636) Root cause: PR #634's CI failed at the markdown-reflow check (make quality Makefile:285) on CONCURRENCY_AUDIT.md — a hard-wrapped audit doc left over from the merged "Engine dedup race + sequencing.py edge-case audit" unit (PR #628). Fixed with `make reflow-docs` (the exact remedy the CI output itself named). Running the full local `make quality` (with a sandbox Postgres/Redis to get past DB-gated skips) surfaced a second real regression from that same PR #628 unit: it added a Redis-backed HeartbeatMutex occasion lock to VideoEngine.open_video_task, but two pre-existing test files (tests/unit/runtime/test_video_render_loop.py and tests/integration/test_video_routes.py) call open_video_task without stubbing that lock, so they failed closed against the suite's deliberately-unreachable test Redis (_no_live_redis). Fixed by applying the same lock-stub pattern tests/unit/services/test_video_engine.py already uses for its own occasion-lock tests: an autouse HeartbeatMutex stand-in fixture in test_video_render_loop.py, and wrapping the two route-level video-request tests in test_video_routes.py with the file's existing _LOCKED patch pair (already used by every other lock-dependent test in that file). The one remaining local failure, test_cloud_auth.py::test_login_route_parses_oauth2_form_not_query_params, is a pre-existing environment gap unrelated to this branch: it needs a real Postgres reachable at localhost:5432 (which .github/workflows/ci.yml provides as a service container) but this dev sandbox has no such binding — confirmed unrelated to any of the four merged audit units. make quality now passes clean: 13729 passed, 0 regressions, 94.49% coverage. Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> * [ddc8121f] regenerate lifecycle artifacts for awaiting_pm_review claim rules and waive_finding intent (#637) Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> --------- Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> Co-authored-by: Renn F <rennf93@users.noreply.github.com> * fix(sequencing): drop lane-fallback edges that would cycle against analyzer edges The dev-task collision fallback unioned the analyzer's authoritative edges with same-assignee lane edges, deduping only the direct pair. A lane chain through an unsurfaced middle sibling could still contradict an analyzer edge transitively (the shared-last migration order inverts plain priority order), closing a 3-cycle that made add_dependency raise ConflictError and wedged every later delegate to that parent. Fallback edges are now accepted only when they can't close a cycle against the edges already kept; a regression test reproduces the exact scenario. Also strip pre-merge cruft: remove the root CONCURRENCY_AUDIT.md working report, delete the near-duplicate websocket-reconnect.md doc, fix the stale a2a/page.tsx doc citation, correct the api-rate-limiting doc to state idempotency-key retry is unimplemented, and fix two lifecycle.py comments that referenced a guard function which never existed. --------- Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech> Co-authored-by: Frontend Documenter <fe-doc@roboco.tech> Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech> Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-08-03 07:23:24 +02:00 · 2026-07-22 08:11:28 +02:00
parent a1233b2aeb
commit 17de29545a
39 changed files with 2458 additions and 89 deletions
@@ -0,0 +1,432 @@
+# Frontend Hooks Reference
+
+This document covers the reusable React hooks exported from `@/hooks` (principally `panel/src/hooks/use-websocket.ts`), with emphasis on the WebSocket message stream patterns and reconnect handling.
+
+## Overview
+
+The panel's real-time coordination streams are built on shared, ref-counted WebSocket connections that fan messages to multiple subscribers. This architecture eliminates duplicate connections to the same endpoint and ensures consistent state across different components that consume the same stream.
+
+### Connection Architecture
+
+- **Shared connections**: Two consumers of the same endpoint (e.g., A2A live view + rate-limit banner both reading `/ws/system`) now share a single WebSocket connection instead of each opening their own.
+- **Message fanning**: The shared connection broadcasts incoming messages to all registered subscribers.
+- **State syncing**: New subscribers are immediately replayed the connection's current state (e.g., a component mounting mid-reconnection sees `connecting` instead of stale `disconnected`).
+
+### Message Loss & Reconnect Handling
+
+The WebSocket connection at `panel/src/lib/websocket/connection.ts` (lines 91–119) has **no server-side message buffering or replay**. When the socket drops and reconnects:
+
+- Any frame published while disconnected is lost from the WebSocket stream forever.
+- Specialized hooks like `useNotificationStream` implement **REST catch-up** to fill the gap: they fetch unread notifications at REST the moment the socket recovers, so no message is silently lost.
+
+This is a point-in-time catch-up strategy, not a byte-for-byte replay — a deliberate trade-off documented in the task acceptance criteria.
+
+---
+
+## `useWebSocket<T>(endpoint, queryParams?, enabled?)`
+
+The foundation hook for subscribing to any WebSocket endpoint. All other hooks (`useNotificationStream`, `useAgentStream`, `useA2ALiveStream`) build on top of it.
+
+### Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `endpoint` | string | — | Path after `/ws/`, e.g., `/notifications/{agentId}` or `/system` |
+| `queryParams` | `Record<string, string>` | `undefined` | Optional query string as an object, e.g., `{ viewer_id: "..." }` |
+| `enabled` | boolean | `true` | Enable/disable the connection (useful for conditional subscriptions) |
+
+### Return Value
+
+```typescript
+{
+  state: ConnectionState;           // "disconnected", "connecting", "reconnecting", "connected"
+  lastMessage: T | null;            // The most recent message
+  messages: T[];                    // Ring buffer of ≤100 messages (STREAM_MAX_MESSAGES)
+  disconnect: () => void;           // Manually tear down the connection
+  clearMessages: () => void;        // Clear the buffer
+  isConnected: boolean;             // Shorthand for state === "connected"
+  isConnecting: boolean;            // Shorthand for state === "connecting" || "reconnecting"
+}
+```
+
+### Example: Raw WebSocket Consumption
+
+```tsx
+import { useWebSocket } from "@/hooks";
+
+export function MyAgentMonitor({ agentId }: { agentId: string }) {
+  const { state, lastMessage, messages, isConnected } = useWebSocket(
+    `/agents/${agentId}`,
+    { viewer_id: CEO_AGENT_ID },
+    !!agentId  // disable if agentId is falsy
+  );
+
+  return (
+    <>
+      <p>Connection: {state}</p>
+      {isConnected && <p>Last update: {lastMessage?.timestamp}</p>}
+      <ul>
+        {messages.map((msg, i) => (
+          <li key={i}>{JSON.stringify(msg)}</li>
+        ))}
+      </ul>
+    </>
+  );
+}
+```
+
+---
+
+## `useNotificationStream()`
+
+Subscribes to **CEO notifications** via `/ws/notifications/{CEO_AGENT_ID}`. This is the only subscription actively using the REST catch-up fallback to guarantee no notification is lost during a reconnect.
+
+### Return Value
+
+```typescript
+{
+  state: ConnectionState;
+  lastMessage: NotificationMessage | null;
+  notifications: NotificationMessage[];    // Deduped by notification_id
+  allMessages: NotificationMessage[];      // All messages from WS (raw)
+  clearMessages: () => void;               // Clears both notifications AND cached REST catch-up batch
+  isConnected: boolean;
+  isConnecting: boolean;
+}
+```
+
+### Reconnect Behavior (Key Change)
+
+When the WebSocket reconnects (transitions from `disconnected` / `reconnecting` → `connected`):
+
+1. **On initial connect**: No REST fetch occurs.
+2. **On a real reconnect** (socket was up before, dropped, and recovered):
+   - A `GET /api/notifications?unread_only=true` fetch fires immediately.
+   - Unread notifications from this fetch are transformed into notification frames and held in local state.
+   - These cached frames are merged with live frames **before** dedup.
+   - The dedup logic ensures no notification appears twice (caught-up notification wins if also delivered live).
+
+3. **Fetch failure**: Silently tolerated — live WS delivery resumes regardless. The catchup is best-effort.
+
+### Deduplication Strategy
+
+The `notifications` array is deduped by `notification_id` using a newest→oldest walk:
+
+- Walk backward through `[...cachedCatchup, ...liveMessages]`.
+- Keep only the first occurrence of each unique `notification_id` (newest wins).
+- Restore arrival order.
+- Notifications without an id (older frames) are always kept.
+
+The cached catch-up batch is placed **ahead** of live frames so live delivery can never be shadowed by an older cached copy.
+
+### Clearing Notifications
+
+Calling `clearMessages()` clears:
+- The live message buffer.
+- The cached catch-up batch.
+
+This prevents an immediate repopulation from cached state after a user clears the notification badge.
+
+### Example
+
+```tsx
+import { useNotificationStream } from "@/hooks";
+
+export function NotificationBell() {
+  const { notifications, isConnected, clearMessages } = useNotificationStream();
+
+  return (
+    <>
+      <button onClick={clearMessages}>
+        Clear ({notifications.length})
+      </button>
+      {!isConnected && <span className="dot" title="offline" />}
+    </>
+  );
+}
+```
+
+---
+
+## `useAgentStream(agentId)`
+
+Subscribes to live agent output (streaming work-in-progress) via `/ws/agents/{agentId}`.
+
+### Parameters
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `agentId` | string \| null | The agent UUID. Pass `null` to disable. |
+
+### Return Value
+
+```typescript
+{
+  state: ConnectionState;
+  lastMessage: AgentStreamMessage | null;
+  messages: AgentStreamMessage[];           // Raw frames
+  streamChunks: string[];                   // Extracted chunk strings
+  streamOutput: string;                     // All chunks concatenated
+  clearMessages: () => void;
+  isConnected: boolean;
+  isConnecting: boolean;
+}
+```
+
+### Message Type
+
+```typescript
+interface AgentStreamMessage {
+  type: "connected" | "agent.stream";
+  agent_id?: string;
+  chunk?: string;                           // Output fragment
+  watcher_count?: number;                   // Live viewer count
+  timestamp?: string;
+}
+```
+
+### Example
+
+```tsx
+import { useAgentStream } from "@/hooks";
+
+export function AgentOutputPanel({ agentId }: { agentId: string }) {
+  const { streamOutput, isConnecting, messages } = useAgentStream(agentId);
+
+  return (
+    <div className="output">
+      {isConnecting && <em>Connecting…</em>}
+      <code>{streamOutput}</code>
+      <p className="meta">
+        {messages.length} frames • {streamOutput.length} chars
+      </p>
+    </div>
+  );
+}
+```
+
+---
+
+## `useA2ALiveStream()`
+
+Subscribes to live **agent-to-agent** (A2A) messages via `/ws/system`. The system stream also carries rate-limit and usage events; this hook filters to `a2a.message` frames only.
+
+### Return Value
+
+```typescript
+{
+  state: ConnectionState;
+  lastMessage: A2ASystemMessage | null;
+  a2aMessages: A2ASystemMessage[];         // Filtered to type === "a2a.message"
+  allMessages: A2ASystemMessage[];         // All frames (includes usage/rate-limit)
+  clearMessages: () => void;
+  isConnected: boolean;
+  isConnecting: boolean;
+}
+```
+
+### Message Type
+
+```typescript
+interface A2ASystemMessage {
+  type: "connected" | "a2a.message";
+  conversation_id?: string;
+  message_id?: string;
+  task_id?: string;
+  from_agent?: string;
+  to_agent?: string;
+  skill?: string | null;
+  body_excerpt?: string;                   // Capped; fetch full body via REST
+  timestamp?: string;
+}
+```
+
+### Important Note: Excerpt-Only Delivery
+
+A2A frames carry a **capped excerpt**, not the full message body. Consumers must:
+
+1. Listen to `a2a.message` frames.
+2. Invalidate their **REST query** for the full message (e.g., `GET /api/a2a/conversations/{id}`).
+3. Fetch the full body from the REST endpoint.
+
+Do not render the `body_excerpt` as the message — it is metadata only.
+
+### Reconnect Fallback
+
+The A2A hook already has a working reconnect fallback at the consumer level: `panel/src/components/a2a/a2a-view.tsx` invalidates the entire a2a query family both per-frame (lines 185–194, on every `a2a.message` frame) and on the disconnected→connected edge (lines 212–218, gated on a `prevConnected` ref so it never fires on initial mount), ensuring no message is lost. No change was needed for this hook.
+
+### Example
+
+```tsx
+import { useA2ALiveStream } from "@/hooks";
+import { useQuery } from "@tanstack/react-query";
+
+export function A2ALiveView() {
+  const { a2aMessages, isConnected } = useA2ALiveStream();
+  const { data: conversations } = useQuery({
+    queryKey: ["a2a", "conversations"],
+    // Auto-fetched when a2aMessages changes (invalidation on frame)
+  });
+
+  return (
+    <div>
+      <p>
+        {isConnected ? "Connected" : "Offline"}
+        {a2aMessages.length > 0 && " (live updates)"}
+      </p>
+      {/* Render conversations with full bodies from REST */}
+    </div>
+  );
+}
+```
+
+---
+
+## `useConnectionStatus()`
+
+Tracks the connection state of **all active subscriptions** in a single hook. Useful for global connection indicators.
+
+### Return Value
+
+```typescript
+{
+  connections: Record<string, ConnectionState>;  // { [endpoint]: state }
+  updateConnection: (id: string, state: ConnectionState) => void;
+  removeConnection: (id: string) => void;
+  hasActiveConnections: boolean;                 // Any connection is up/ing
+  allConnected: boolean;                         // Every connection is connected
+}
+```
+
+### Example: Global Status Indicator
+
+```tsx
+import { useConnectionStatus } from "@/hooks";
+
+export function GlobalConnectionStatus() {
+  const { hasActiveConnections, allConnected } = useConnectionStatus();
+
+  return (
+    <div className="status-badge">
+      {allConnected && <span className="icon-check">Connected</span>}
+      {hasActiveConnections && !allConnected && (
+        <span className="icon-sync">Reconnecting…</span>
+      )}
+      {!hasActiveConnections && <span className="icon-offline">Offline</span>}
+    </div>
+  );
+}
+```
+
+---
+
+## Testing
+
+The hooks come with comprehensive test coverage in `panel/src/hooks/__tests__/`:
+
+- **`use-websocket.test.tsx`**: Core hook mechanics (shared connection, fan-out, state replay).
+- **`use-notification-stream.test.tsx`**: REST catch-up verification (new; covers the reconnect fallback):
+  - No fetch on initial connect.
+  - Fetch + fold-in on a real reconnect.
+  - Dedup prevents double-counting when the same notification arrives both via catch-up and live.
+  - `clearMessages()` drops the cached catch-up batch.
+
+Run tests with:
+
+```bash
+pnpm test
+```
+
+---
+
+## Audited Hooks: Reconnect Coverage
+
+Three hooks were audited for reconnect message-loss risk:
+
+| Hook | Connection Type | Fallback | Status |
+|------|-----------------|----------|--------|
+| `useNotificationStream` | CEO notification stream | REST catch-up (`GET /notifications?unread_only=true`) | ✅ Hardened |
+| `useA2ALiveStream` | A2A + system stream | REST query invalidation (`a2a-view.tsx`) | ✅ Verified |
+| Rate-limit consumers | System stream (`/ws/system`) | REST resync (rate-limit-banner.tsx, usage-overview-panel.tsx) | ✅ Verified |
+
+No defects were found in `useA2ALiveStream` or rate-limit consumption; the REST polling fallbacks were already in place and tested.
+
+### Adding a New Reconnect Fallback
+
+Picking a fallback strategy for a new WS-consuming hook comes down to what kind of data it carries:
+
+1. **Event data** (can only happen once, e.g. a notification) — implement REST catch-up: fetch unread/pending items over REST the moment the socket recovers, fold them into local state, and dedup against live frames (see `useNotificationStream` above).
+2. **State data** (always available via REST) — implement query invalidation: invalidate the relevant React Query cache keys on the disconnected→connected edge and let components refetch (see `useA2ALiveStream` above).
+3. **Always** add a regression test that simulates a disconnect/reconnect cycle with fake timers.
+
+---
+
+## Best Practices
+
+1. **Always disable on falsy keys**: Pass a conditional `enabled` flag (third param) if your hook depends on a variable parameter. This prevents spurious connections and ensures cleanup.
+
+   ```tsx
+   const { messages } = useWebSocket(
+     `/agents/${agentId}`,
+     undefined,
+     !!agentId  // disable if agentId is null/undefined
+   );
+   ```
+
+2. **Invalidate REST queries on frame**: When receiving a frame (especially A2A excerpts), trigger a React Query invalidation to fetch fresh data:
+
+   ```tsx
+   const queryClient = useQueryClient();
+   useEffect(() => {
+     if (a2aMessages.length > 0) {
+       queryClient.invalidateQueryData({ queryKey: ["a2a"] });
+     }
+   }, [a2aMessages, queryClient]);
+   ```
+
+3. **Don't hold onto stale messages**: The message buffer is bounded to 100 frames. Don't assume it's a complete history — treat it as a stream.
+
+4. **Catch fetch failures gracefully**: All REST fallbacks (like the notification catch-up) are best-effort and fail silently. Live WS delivery is not guaranteed to block on fetch completion.
+
+5. **Use `isConnecting` for UI feedback**: Show a loading state when `isConnecting` is true, not just when `!isConnected`.
+
+   ```tsx
+   {isConnecting && <Spinner />}
+   {isConnected && <CheckMark />}
+   ```
+
+---
+
+## Connection Types & States
+
+### ConnectionState
+
+```typescript
+type ConnectionState = 
+  | "disconnected"  // Not connected; attempting to reconnect or no connection attempt yet
+  | "connecting"    // Initial connection attempt
+  | "reconnecting"  // Reconnection after a close (watchdog/network failure)
+  | "connected"     // Stable connection, messages flowing
+```
+
+### State Transitions
+
+```
+disconnected ──> connecting ──> connected
+                                    ^
+                                    │
+                              (watchdog fires)
+                                    │
+                          reconnecting ──┘
+```
+
+On a reconnect transition (`reconnecting` → `connected`), hooks like `useNotificationStream` fire their REST catch-up fetch.
+
+---
+
+## Further Reading
+
+- **Control panel README**: `panel/README.md`
+- **WebSocket connection implementation**: `panel/src/lib/websocket/connection.ts`
+- **API client**: `panel/src/lib/api/`
+- **Notification types & components**: `panel/src/app/(dashboard)/notifications/`