Files
roboco/tests/unit/api/test_websocket_bridge.py
T
547fe444f2 [4865ff8b] Add WebSocket support to the usage dashboard (#115)
* [e7349d84] feat(dashboard): WS usage store, hook extension, status badge, and smooth animations (#111) (#113)

- Add src/store/usage-store.ts with typed UsageData interface, useUsageStore
  Zustand store, setUsageData, clearUsageData, and setWsState actions
- Export useUsageStore and UsageData from store/index.ts
- Extend use-rate-limit-websocket.ts: rename msg type to SystemWsMessage,
  add key_metrics field; add useEffect syncing wsState into useUsageStore;
  add USAGE_UPDATE/USAGE_SNAPSHOT handler dispatching to useUsageStore
  (RATE_LIMIT_HIT/LIFTED handling and onReconnect unchanged)
- Update CommandCenter to read key_metrics from useUsageStore when
  wsState === 'connected' and usageData non-null; falls back to
  useCeoOverview() (refetchInterval: 60000) when WS disconnected
- Update KeyMetricsPanel: add wsState prop, render connection status Badge
  matching AgentStreamViewer pattern (bg-green-500+Wifi / bg-yellow-500+
  Loader2 spin / bg-gray-500+WifiOff); add transition-all duration-300
  ease-in-out to metric value spans for smooth animated updates

Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev>

* [c9745ee8] feat(events): add USAGE_UPDATE/SNAPSHOT event types, throttled publisher, /ws/system usage bridge (#112) (#114)

- Add EventType.USAGE_UPDATE='usage.update' and EventType.USAGE_SNAPSHOT='usage.snapshot'
  to the EventType StrEnum in roboco/models/events.py

- Create roboco/services/usage_events.py with _UsageThrottle class (5-second per-agent
  window using time.monotonic()) and publish_usage_update() / publish_usage_snapshot()
  helpers; lazy imports prevent circular dependency with roboco.events

- Extend orchestrator._sweep_token_snapshots() to publish USAGE_UPDATE per active agent
  (throttled) and a USAGE_SNAPSHOT aggregate after each sweep cycle; wrapped in
  contextlib.suppress so event errors never abort DB snapshot operations

- Add _handle_usage_event() to websocket_bridge.py following _handle_rate_limit_event
  pattern; register USAGE_UPDATE and USAGE_SNAPSHOT subscriptions in
  register_websocket_bridge_handlers() forwarding both to /ws/system via broadcast_system()

- Add unit tests: test_usage_events.py (throttle suppression, publish helpers) and
  test_websocket_bridge.py extended with _handle_usage_event coverage and updated
  registration assertion to include USAGE_UPDATE/USAGE_SNAPSHOT

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

* fix(usage-ws): reconcile the realtime token/cost contract end-to-end

The backend and frontend halves shipped mismatched contracts, so the usage
dashboard never received live data:

- The bridge forwarded the dotted event value ("usage.update") while the panel
  switched on "USAGE_UPDATE"; map both to the UPPER_SNAKE type string the same
  way the rate-limit handler does.
- The backend emitted token/cost telemetry but the frontend read a key_metrics
  field and fed the org-metrics panel. Rewire the frontend to consume the
  USAGE_SNAPSHOT token/cost payload into the "Token Usage & Cost" panel —
  WS-first with polling fallback and a connection-status badge — and revert the
  unrelated KeyMetricsPanel / CommandCenter wiring.

Backend cleanups in the same path:

- Replace the multi-argument publish helpers with typed UsageUpdate /
  UsageSnapshot payloads, removing the too-many-arguments lint suppressions.
- Extract _fetch_agent_tokens and _persist_token_snapshot from the token sweep,
  removing the too-many-statements suppression; label the live snapshot "live".

Hardening uncovered while fixing the above:

- _finalize_spawn_session pulled the full RAG stack into the
  session-finalization path through a transcript-parse import; move the pure
  parser into a dependency-light roboco.agent_sdk.transcript_usage module so
  finalization never imports the agent SDK server.
- Reduce _finalize_spawn_session complexity by extracting
  _resolve_final_token_usage, and widen the transcript-fallback guard so a read
  error can never abort finalization.

Also align KeyMetricsPanel with the metrics /dashboard/ceo actually returns: it
read velocity_24h / avg_time_to_done / active_agents, none of which
get_key_metrics() emits, so four of five rows rendered "—". Render
velocity_weekly, completion_rate, documentation_coverage and active_blockers.

* docs: note live usage push over /ws/system on the usage dashboard

* fix(usage): finalize on self-exit and de-duplicate transcript token counts

Two bugs left token capture broken even after the transcript-read fallback
landed — surfaced by a live agent run:

- Agents that self-exit (the normal i_am_idle -> container shutdown, exit 0)
  were never finalized. _finalize_spawn_session is only called from
  stop_agent(), but a graceful self-exit goes through _handle_stopped_container,
  which set the instance OFFLINE and returned without finalizing — leaving the
  spawn-session row open with zero tokens. Finalize there for both graceful
  (exit_reason="completed") and crash (exit_reason="crashed") exits.

- sum_transcript_usage double-counted. Claude Code logs one assistant message
  as several JSONL lines (one per content block — thinking / text / tool_use),
  each repeating the same message.usage, so summing every line roughly doubled
  the totals. De-duplicate by message.id.

Verified against a live agent transcript: the raw sum (12, 1068, 62502, 115828)
vs the de-duped (6, 516, 62502, 63336), which matches the session's
authoritative result.usage exactly.

* feat(usage): fall back to the transcript in the live token sweep

The 60s token sweep read only the agent SDK's /usage/status, which races
container teardown and reports zero mid-run — so live usage (and the
USAGE_SNAPSHOT pushed to /ws/system) stayed at zero for active agents.
Extract _resolve_active_tokens: try the SDK, then fall back to the durable
transcript (the same source finalize uses) so running agents report live.

* feat(usage): add GET /usage/sessions for the dashboard's Recent Sessions

The panel's Recent Sessions table was mock-only — the backend had no sessions
endpoint, so production always showed 'No sessions recorded yet'. Add
UsageService.get_recent_sessions + a /usage/sessions route returning the most
recent spawn-session rows (token totals + cost), and point the panel client
at it.

---------

Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev>
Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-06-11 23:19:50 +02:00

403 lines
15 KiB
Python

"""websocket_bridge coverage — event handlers + bridge starter.
The handlers fan events from the Redis-stream bus to per-recipient WebSocket
connections. We don't need real Redis or sockets; we patch `manager` and the
`broadcast_*` helpers so each handler exercises its branches against
in-memory state.
"""
from __future__ import annotations
from unittest.mock import AsyncMock, patch
from uuid import uuid4
import pytest
from roboco.api.websocket_bridge import (
_handle_agent_event,
_handle_notification_sent,
_handle_rate_limit_event,
_handle_session_event,
_handle_usage_event,
register_websocket_bridge_handlers,
start_websocket_bridge,
)
from roboco.models.events import Event, EventType
def _evt(event_type: EventType, data: dict, source_agent: str | None = None) -> Event:
return Event(type=event_type, data=data, source_agent=source_agent)
# ---------------------------------------------------------------------------
# _handle_notification_sent
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_handle_notification_sent_skips_when_missing_ids() -> None:
"""Incomplete event (missing recipient/notification IDs) → log + return."""
event = _evt(EventType.NOTIFICATION_SENT, {}) # No notification_id/recipient_id
with patch("roboco.api.websocket_bridge.broadcast_notification") as bcast:
await _handle_notification_sent(event)
bcast.assert_not_called()
@pytest.mark.asyncio
async def test_handle_notification_sent_skips_invalid_uuid() -> None:
"""Invalid UUID strings → log error + return without broadcasting."""
event = _evt(
EventType.NOTIFICATION_SENT,
{"notification_id": "not-a-uuid", "recipient_id": str(uuid4())},
)
with patch("roboco.api.websocket_bridge.broadcast_notification") as bcast:
await _handle_notification_sent(event)
bcast.assert_not_called()
@pytest.mark.asyncio
async def test_handle_notification_sent_skips_when_no_connections() -> None:
"""Recipient has no WS connections → no broadcast."""
nid = uuid4()
rid = uuid4()
event = _evt(
EventType.NOTIFICATION_SENT,
{
"notification_id": str(nid),
"recipient_id": str(rid),
"type": "blocker",
"subject": "x",
"priority": "high",
},
)
with (
patch("roboco.api.websocket_bridge.broadcast_notification") as bcast,
patch("roboco.api.websocket_bridge.manager") as mgr,
):
mgr.notification_connections = {} # No connections for any agent.
await _handle_notification_sent(event)
bcast.assert_not_called()
@pytest.mark.asyncio
async def test_handle_notification_sent_broadcasts_when_connected() -> None:
"""Recipient has WS connection → broadcast_notification called."""
nid = uuid4()
rid = uuid4()
event = _evt(
EventType.NOTIFICATION_SENT,
{
"notification_id": str(nid),
"recipient_id": str(rid),
"type": "qa_ready",
"subject": "Task ready",
"priority": "normal",
},
)
bcast = AsyncMock()
with (
patch("roboco.api.websocket_bridge.broadcast_notification", bcast),
patch("roboco.api.websocket_bridge.manager") as mgr,
):
mgr.notification_connections = {rid: {"socket-1"}} # Has a connection.
await _handle_notification_sent(event)
bcast.assert_awaited_once()
call_kwargs = bcast.await_args.kwargs
assert call_kwargs["notification_id"] == nid
assert call_kwargs["agent_ids"] == [rid]
@pytest.mark.asyncio
async def test_handle_notification_acked_broadcasts_using_agent_id() -> None:
"""ACKED events carry `agent_id`, not `recipient_id`; the shared handler
must still forward (to the acking agent) rather than log 'Incomplete
notification event' on every acknowledgement."""
nid = uuid4()
aid = uuid4()
event = _evt(
EventType.NOTIFICATION_ACKED,
{"notification_id": str(nid), "agent_id": str(aid), "ack_type": "read"},
)
bcast = AsyncMock()
with (
patch("roboco.api.websocket_bridge.broadcast_notification", bcast),
patch("roboco.api.websocket_bridge.manager") as mgr,
):
mgr.notification_connections = {aid: {"socket-1"}}
await _handle_notification_sent(event)
bcast.assert_awaited_once()
call_kwargs = bcast.await_args.kwargs
assert call_kwargs["notification_id"] == nid
assert call_kwargs["agent_ids"] == [aid]
# ---------------------------------------------------------------------------
# _handle_session_event
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_handle_session_event_skips_missing_session_id() -> None:
event = _evt(EventType.SESSION_CREATED, {})
with patch("roboco.api.websocket_bridge.manager") as mgr:
mgr.broadcast_to_session = AsyncMock()
await _handle_session_event(event)
mgr.broadcast_to_session.assert_not_called()
@pytest.mark.asyncio
async def test_handle_session_event_skips_invalid_uuid() -> None:
event = _evt(EventType.SESSION_CREATED, {"session_id": "bad-uuid"})
with patch("roboco.api.websocket_bridge.manager") as mgr:
mgr.broadcast_to_session = AsyncMock()
await _handle_session_event(event)
mgr.broadcast_to_session.assert_not_called()
@pytest.mark.asyncio
async def test_handle_session_event_skips_when_no_connections() -> None:
sid = uuid4()
event = _evt(EventType.SESSION_CREATED, {"session_id": str(sid)})
with patch("roboco.api.websocket_bridge.manager") as mgr:
mgr.session_connections = {}
mgr.broadcast_to_session = AsyncMock()
await _handle_session_event(event)
mgr.broadcast_to_session.assert_not_called()
@pytest.mark.asyncio
async def test_handle_session_event_broadcasts() -> None:
sid = uuid4()
event = _evt(EventType.SESSION_CLOSED, {"session_id": str(sid)})
with patch("roboco.api.websocket_bridge.manager") as mgr:
mgr.session_connections = {sid: {"sock-1"}}
mgr.broadcast_to_session = AsyncMock()
await _handle_session_event(event)
mgr.broadcast_to_session.assert_awaited_once()
# Payload includes the trailing piece of the event-type ('closed').
call_args = mgr.broadcast_to_session.await_args
assert call_args.args[0] == sid
assert call_args.args[1]["type"] == "session.closed"
# ---------------------------------------------------------------------------
# _handle_agent_event
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_handle_agent_event_skips_when_no_agent_id() -> None:
event = _evt(EventType.AGENT_SPAWNED, {})
with patch("roboco.api.websocket_bridge.manager") as mgr:
mgr.broadcast_to_agent_watchers = AsyncMock()
await _handle_agent_event(event)
mgr.broadcast_to_agent_watchers.assert_not_called()
@pytest.mark.asyncio
async def test_handle_agent_event_skips_invalid_uuid() -> None:
event = _evt(EventType.AGENT_SPAWNED, {"agent_id": "bad"})
with patch("roboco.api.websocket_bridge.manager") as mgr:
mgr.broadcast_to_agent_watchers = AsyncMock()
await _handle_agent_event(event)
mgr.broadcast_to_agent_watchers.assert_not_called()
@pytest.mark.asyncio
async def test_handle_agent_event_uses_source_agent_fallback() -> None:
"""When data has no agent_id, falls back to event.source_agent."""
aid = uuid4()
event = _evt(EventType.AGENT_STOPPED, {}, source_agent=str(aid))
with patch("roboco.api.websocket_bridge.manager") as mgr:
mgr.agent_connections = {aid: {"sock"}}
mgr.broadcast_to_agent_watchers = AsyncMock()
await _handle_agent_event(event)
mgr.broadcast_to_agent_watchers.assert_awaited_once()
@pytest.mark.asyncio
async def test_handle_agent_event_skips_when_no_connections() -> None:
aid = uuid4()
event = _evt(EventType.AGENT_SPAWNED, {"agent_id": str(aid)})
with patch("roboco.api.websocket_bridge.manager") as mgr:
mgr.agent_connections = {}
mgr.broadcast_to_agent_watchers = AsyncMock()
await _handle_agent_event(event)
mgr.broadcast_to_agent_watchers.assert_not_called()
@pytest.mark.asyncio
async def test_handle_agent_event_broadcasts() -> None:
aid = uuid4()
event = _evt(EventType.AGENT_RESUMED, {"agent_id": str(aid)})
with patch("roboco.api.websocket_bridge.manager") as mgr:
mgr.agent_connections = {aid: {"sock"}}
mgr.broadcast_to_agent_watchers = AsyncMock()
await _handle_agent_event(event)
mgr.broadcast_to_agent_watchers.assert_awaited_once()
call_args = mgr.broadcast_to_agent_watchers.await_args
assert call_args.args[0] == aid
assert call_args.args[1]["type"] == "agent.resumed"
# ---------------------------------------------------------------------------
# _handle_rate_limit_event
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_handle_rate_limit_hit_broadcasts_to_system() -> None:
"""RATE_LIMIT_HIT → broadcast_system tagged with the type, payload intact."""
retry_after = 60.0
event = _evt(
EventType.RATE_LIMIT_HIT,
{
"provider": "anthropic",
"affectedAgents": ["be-dev-1"],
"retryAfterSeconds": retry_after,
"timestamp": "2026-06-11T00:00:00+00:00",
},
)
with patch("roboco.api.websocket_bridge.manager") as mgr:
mgr.broadcast_system = AsyncMock()
await _handle_rate_limit_event(event)
mgr.broadcast_system.assert_awaited_once()
msg = mgr.broadcast_system.await_args.args[0]
assert msg["type"] == "RATE_LIMIT_HIT"
assert msg["provider"] == "anthropic"
assert msg["affectedAgents"] == ["be-dev-1"]
assert msg["retryAfterSeconds"] == retry_after
@pytest.mark.asyncio
async def test_handle_rate_limit_lifted_broadcasts_to_system() -> None:
event = _evt(
EventType.RATE_LIMIT_LIFTED,
{"provider": "anthropic", "timestamp": "2026-06-11T00:01:00+00:00"},
)
with patch("roboco.api.websocket_bridge.manager") as mgr:
mgr.broadcast_system = AsyncMock()
await _handle_rate_limit_event(event)
msg = mgr.broadcast_system.await_args.args[0]
assert msg["type"] == "RATE_LIMIT_LIFTED"
assert msg["provider"] == "anthropic"
@pytest.mark.asyncio
async def test_handle_rate_limit_ignores_unrelated_event() -> None:
"""A non-rate-limit event type is a no-op (defensive guard)."""
event = _evt(EventType.AGENT_SPAWNED, {"provider": "anthropic"})
with patch("roboco.api.websocket_bridge.manager") as mgr:
mgr.broadcast_system = AsyncMock()
await _handle_rate_limit_event(event)
mgr.broadcast_system.assert_not_called()
# ---------------------------------------------------------------------------
# _handle_usage_event
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_handle_usage_update_broadcasts_to_system() -> None:
"""USAGE_UPDATE event → broadcast_system tagged USAGE_UPDATE + data fields."""
expected_input = 100
expected_output = 50
event = _evt(
EventType.USAGE_UPDATE,
{
"agent_id": "be-dev-1",
"task_id": "task-abc",
"input_tokens": expected_input,
"output_tokens": expected_output,
"model": "claude-sonnet-4-6",
"timestamp": "2026-06-11T00:00:00+00:00",
},
)
with patch("roboco.api.websocket_bridge.manager") as mgr:
mgr.broadcast_system = AsyncMock()
await _handle_usage_event(event)
mgr.broadcast_system.assert_awaited_once()
msg = mgr.broadcast_system.await_args.args[0]
assert msg["type"] == "USAGE_UPDATE"
assert msg["agent_id"] == "be-dev-1"
assert msg["input_tokens"] == expected_input
assert msg["output_tokens"] == expected_output
@pytest.mark.asyncio
async def test_handle_usage_snapshot_broadcasts_to_system() -> None:
"""USAGE_SNAPSHOT event → broadcast_system tagged USAGE_SNAPSHOT + aggregate."""
expected_input = 500
event = _evt(
EventType.USAGE_SNAPSHOT,
{
"period": "live",
"totals": {"input_tokens": expected_input, "output_tokens": 200},
"cost_estimate": 0.0025,
"by_agent": [
{
"agent_id": "be-dev-1",
"input_tokens": 500,
"output_tokens": 200,
"model": "sonnet",
"cost_estimate": 0.0025,
}
],
"timestamp": "2026-06-11T00:01:00+00:00",
},
)
with patch("roboco.api.websocket_bridge.manager") as mgr:
mgr.broadcast_system = AsyncMock()
await _handle_usage_event(event)
mgr.broadcast_system.assert_awaited_once()
msg = mgr.broadcast_system.await_args.args[0]
assert msg["type"] == "USAGE_SNAPSHOT"
assert msg["period"] == "live"
assert msg["totals"]["input_tokens"] == expected_input
assert len(msg["by_agent"]) == 1
# ---------------------------------------------------------------------------
# Registration + start
# ---------------------------------------------------------------------------
def test_register_websocket_bridge_handlers_subscribes_all_event_types() -> None:
"""Registration wires up all handler categories, including usage events."""
class _FakeBus:
def __init__(self) -> None:
self.subscribed: list[tuple[EventType, object]] = []
def subscribe(self, event_type: EventType, handler: object) -> None:
self.subscribed.append((event_type, handler))
fake = _FakeBus()
with patch("roboco.api.websocket_bridge.get_event_bus", return_value=fake):
register_websocket_bridge_handlers()
types = [t for t, _ in fake.subscribed]
# All 14 expected event types appear at least once.
assert EventType.NOTIFICATION_SENT in types
assert EventType.NOTIFICATION_ACKED in types
assert EventType.SESSION_CREATED in types
assert EventType.SESSION_CLOSED in types
assert EventType.SESSION_TIMEOUT in types
assert EventType.AGENT_SPAWNED in types
assert EventType.AGENT_STOPPED in types
assert EventType.AGENT_WAITING in types
assert EventType.AGENT_RESUMED in types
assert EventType.AGENT_ERROR in types
assert EventType.RATE_LIMIT_HIT in types
assert EventType.RATE_LIMIT_LIFTED in types
# Usage events forwarded to /ws/system.
assert EventType.USAGE_UPDATE in types
assert EventType.USAGE_SNAPSHOT in types
@pytest.mark.asyncio
async def test_start_websocket_bridge_registers_handlers() -> None:
"""start_websocket_bridge() calls register_websocket_bridge_handlers."""
with patch("roboco.api.websocket_bridge.register_websocket_bridge_handlers") as reg:
await start_websocket_bridge()
reg.assert_called_once()