Files
roboco/tests/unit/services/test_usage_events.py
T
547fe444f2 [4865ff8b] Add WebSocket support to the usage dashboard (#115)
* [e7349d84] feat(dashboard): WS usage store, hook extension, status badge, and smooth animations (#111) (#113)

- Add src/store/usage-store.ts with typed UsageData interface, useUsageStore
  Zustand store, setUsageData, clearUsageData, and setWsState actions
- Export useUsageStore and UsageData from store/index.ts
- Extend use-rate-limit-websocket.ts: rename msg type to SystemWsMessage,
  add key_metrics field; add useEffect syncing wsState into useUsageStore;
  add USAGE_UPDATE/USAGE_SNAPSHOT handler dispatching to useUsageStore
  (RATE_LIMIT_HIT/LIFTED handling and onReconnect unchanged)
- Update CommandCenter to read key_metrics from useUsageStore when
  wsState === 'connected' and usageData non-null; falls back to
  useCeoOverview() (refetchInterval: 60000) when WS disconnected
- Update KeyMetricsPanel: add wsState prop, render connection status Badge
  matching AgentStreamViewer pattern (bg-green-500+Wifi / bg-yellow-500+
  Loader2 spin / bg-gray-500+WifiOff); add transition-all duration-300
  ease-in-out to metric value spans for smooth animated updates

Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev>

* [c9745ee8] feat(events): add USAGE_UPDATE/SNAPSHOT event types, throttled publisher, /ws/system usage bridge (#112) (#114)

- Add EventType.USAGE_UPDATE='usage.update' and EventType.USAGE_SNAPSHOT='usage.snapshot'
  to the EventType StrEnum in roboco/models/events.py

- Create roboco/services/usage_events.py with _UsageThrottle class (5-second per-agent
  window using time.monotonic()) and publish_usage_update() / publish_usage_snapshot()
  helpers; lazy imports prevent circular dependency with roboco.events

- Extend orchestrator._sweep_token_snapshots() to publish USAGE_UPDATE per active agent
  (throttled) and a USAGE_SNAPSHOT aggregate after each sweep cycle; wrapped in
  contextlib.suppress so event errors never abort DB snapshot operations

- Add _handle_usage_event() to websocket_bridge.py following _handle_rate_limit_event
  pattern; register USAGE_UPDATE and USAGE_SNAPSHOT subscriptions in
  register_websocket_bridge_handlers() forwarding both to /ws/system via broadcast_system()

- Add unit tests: test_usage_events.py (throttle suppression, publish helpers) and
  test_websocket_bridge.py extended with _handle_usage_event coverage and updated
  registration assertion to include USAGE_UPDATE/USAGE_SNAPSHOT

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

* fix(usage-ws): reconcile the realtime token/cost contract end-to-end

The backend and frontend halves shipped mismatched contracts, so the usage
dashboard never received live data:

- The bridge forwarded the dotted event value ("usage.update") while the panel
  switched on "USAGE_UPDATE"; map both to the UPPER_SNAKE type string the same
  way the rate-limit handler does.
- The backend emitted token/cost telemetry but the frontend read a key_metrics
  field and fed the org-metrics panel. Rewire the frontend to consume the
  USAGE_SNAPSHOT token/cost payload into the "Token Usage & Cost" panel —
  WS-first with polling fallback and a connection-status badge — and revert the
  unrelated KeyMetricsPanel / CommandCenter wiring.

Backend cleanups in the same path:

- Replace the multi-argument publish helpers with typed UsageUpdate /
  UsageSnapshot payloads, removing the too-many-arguments lint suppressions.
- Extract _fetch_agent_tokens and _persist_token_snapshot from the token sweep,
  removing the too-many-statements suppression; label the live snapshot "live".

Hardening uncovered while fixing the above:

- _finalize_spawn_session pulled the full RAG stack into the
  session-finalization path through a transcript-parse import; move the pure
  parser into a dependency-light roboco.agent_sdk.transcript_usage module so
  finalization never imports the agent SDK server.
- Reduce _finalize_spawn_session complexity by extracting
  _resolve_final_token_usage, and widen the transcript-fallback guard so a read
  error can never abort finalization.

Also align KeyMetricsPanel with the metrics /dashboard/ceo actually returns: it
read velocity_24h / avg_time_to_done / active_agents, none of which
get_key_metrics() emits, so four of five rows rendered "—". Render
velocity_weekly, completion_rate, documentation_coverage and active_blockers.

* docs: note live usage push over /ws/system on the usage dashboard

* fix(usage): finalize on self-exit and de-duplicate transcript token counts

Two bugs left token capture broken even after the transcript-read fallback
landed — surfaced by a live agent run:

- Agents that self-exit (the normal i_am_idle -> container shutdown, exit 0)
  were never finalized. _finalize_spawn_session is only called from
  stop_agent(), but a graceful self-exit goes through _handle_stopped_container,
  which set the instance OFFLINE and returned without finalizing — leaving the
  spawn-session row open with zero tokens. Finalize there for both graceful
  (exit_reason="completed") and crash (exit_reason="crashed") exits.

- sum_transcript_usage double-counted. Claude Code logs one assistant message
  as several JSONL lines (one per content block — thinking / text / tool_use),
  each repeating the same message.usage, so summing every line roughly doubled
  the totals. De-duplicate by message.id.

Verified against a live agent transcript: the raw sum (12, 1068, 62502, 115828)
vs the de-duped (6, 516, 62502, 63336), which matches the session's
authoritative result.usage exactly.

* feat(usage): fall back to the transcript in the live token sweep

The 60s token sweep read only the agent SDK's /usage/status, which races
container teardown and reports zero mid-run — so live usage (and the
USAGE_SNAPSHOT pushed to /ws/system) stayed at zero for active agents.
Extract _resolve_active_tokens: try the SDK, then fall back to the durable
transcript (the same source finalize uses) so running agents report live.

* feat(usage): add GET /usage/sessions for the dashboard's Recent Sessions

The panel's Recent Sessions table was mock-only — the backend had no sessions
endpoint, so production always showed 'No sessions recorded yet'. Add
UsageService.get_recent_sessions + a /usage/sessions route returning the most
recent spawn-session rows (token totals + cost), and point the panel client
at it.

---------

Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev>
Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-06-11 23:19:50 +02:00

263 lines
8.6 KiB
Python

"""Unit tests for roboco.services.usage_events.
Covers the _UsageThrottle class and the publish_usage_update /
publish_usage_snapshot helpers. No real Redis or event bus is needed —
we use AsyncMock to assert that bus.publish is called with the right
payload and type.
The throttle suppression test is the acceptance-criterion gate:
"Server-side throttle prevents more than 1 USAGE_UPDATE publish per
agent per 5-second window."
"""
from __future__ import annotations
from datetime import UTC, datetime
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from roboco.services.usage_events import (
UsageSnapshot,
UsageUpdate,
_UsageThrottle,
publish_usage_snapshot,
publish_usage_update,
)
# ---------------------------------------------------------------------------
# _UsageThrottle
# ---------------------------------------------------------------------------
def test_throttle_allows_first_publish() -> None:
"""A fresh agent has no prior timestamp — first publish is always allowed."""
th = _UsageThrottle(window=5.0)
assert th.should_publish("be-dev-1") is True
def test_throttle_suppresses_second_publish_within_window() -> None:
"""Second call within the 5-second window returns False (suppressed)."""
th = _UsageThrottle(window=5.0)
with patch("roboco.services.usage_events.time") as mock_time:
mock_time.monotonic.return_value = 100.0
assert th.should_publish("be-dev-1") is True # first → allowed
mock_time.monotonic.return_value = 104.9 # 4.9 s later — still inside window
assert th.should_publish("be-dev-1") is False # suppressed
def test_throttle_allows_publish_after_window_expires() -> None:
"""After the full window elapses, the next publish is allowed again."""
th = _UsageThrottle(window=5.0)
with patch("roboco.services.usage_events.time") as mock_time:
mock_time.monotonic.return_value = 100.0
assert th.should_publish("be-dev-1") is True # first
mock_time.monotonic.return_value = 105.0 # exactly 5 s later
assert th.should_publish("be-dev-1") is True # window elapsed → allowed
def test_throttle_tracks_agents_independently() -> None:
"""Different agents have independent throttle windows."""
th = _UsageThrottle(window=5.0)
with patch("roboco.services.usage_events.time") as mock_time:
mock_time.monotonic.return_value = 100.0
assert th.should_publish("be-dev-1") is True
# be-dev-2 has never published, so it is always allowed.
assert th.should_publish("be-dev-2") is True
mock_time.monotonic.return_value = 101.0
# be-dev-1 is suppressed; be-dev-2 is also now suppressed.
assert th.should_publish("be-dev-1") is False
assert th.should_publish("be-dev-2") is False
def test_throttle_records_timestamp_on_allow() -> None:
"""should_publish records the current time when it returns True."""
th = _UsageThrottle(window=5.0)
recorded_at = 200.0
with patch("roboco.services.usage_events.time") as mock_time:
mock_time.monotonic.return_value = recorded_at
th.should_publish("be-dev-1")
assert th._last["be-dev-1"] == recorded_at
# ---------------------------------------------------------------------------
# publish_usage_update
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_publish_usage_update_calls_bus_publish() -> None:
"""First call in a window publishes the event and returns True."""
bus = MagicMock()
bus.publish = AsyncMock()
th = _UsageThrottle(window=5.0)
expected_input = 100
expected_output = 50
with patch("roboco.services.usage_events._throttle", th):
result = await publish_usage_update(
bus,
UsageUpdate(
agent_id="be-dev-1",
task_id="task-abc",
input_tokens=expected_input,
output_tokens=expected_output,
model="claude-sonnet-4-6",
),
)
assert result is True
bus.publish.assert_awaited_once()
event = bus.publish.await_args.args[0]
assert event.type.value == "usage.update"
assert event.data["agent_id"] == "be-dev-1"
assert event.data["task_id"] == "task-abc"
assert event.data["input_tokens"] == expected_input
assert event.data["output_tokens"] == expected_output
assert event.data["model"] == "claude-sonnet-4-6"
assert "timestamp" in event.data
@pytest.mark.asyncio
async def test_publish_usage_update_throttle_suppresses_second_call() -> None:
"""Second publish within the throttle window is suppressed (returns False)."""
bus = MagicMock()
bus.publish = AsyncMock()
th = _UsageThrottle(window=5.0)
with (
patch("roboco.services.usage_events._throttle", th),
patch("roboco.services.usage_events.time") as mock_time,
):
mock_time.monotonic.return_value = 100.0
first = await publish_usage_update(
bus,
UsageUpdate(
agent_id="be-dev-1",
task_id=None,
input_tokens=10,
output_tokens=5,
model="sonnet",
),
)
mock_time.monotonic.return_value = 102.0 # 2 s later — still suppressed
second = await publish_usage_update(
bus,
UsageUpdate(
agent_id="be-dev-1",
task_id=None,
input_tokens=20,
output_tokens=10,
model="sonnet",
),
)
assert first is True
assert second is False
# bus.publish should only have been called once.
assert bus.publish.await_count == 1
@pytest.mark.asyncio
async def test_publish_usage_update_custom_timestamp() -> None:
"""Custom timestamp is passed through to the event data."""
bus = MagicMock()
bus.publish = AsyncMock()
ts = datetime(2026, 6, 11, 12, 0, 0, tzinfo=UTC)
# Use a fresh throttle so the first publish goes through.
th = _UsageThrottle(window=5.0)
with patch("roboco.services.usage_events._throttle", th):
await publish_usage_update(
bus,
UsageUpdate(
agent_id="be-dev-1",
task_id=None,
input_tokens=0,
output_tokens=0,
model="sonnet",
timestamp=ts,
),
)
event = bus.publish.await_args.args[0]
assert event.data["timestamp"] == ts.isoformat()
# ---------------------------------------------------------------------------
# publish_usage_snapshot
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_publish_usage_snapshot_always_publishes() -> None:
"""publish_usage_snapshot has no throttle — always publishes."""
bus = MagicMock()
bus.publish = AsyncMock()
expected_input = 500
expected_cost = 0.0025
expected_agents = 2
await publish_usage_snapshot(
bus,
UsageSnapshot(
period="live",
totals={"input_tokens": expected_input, "output_tokens": 200},
cost_estimate=expected_cost,
by_agent=[
{
"agent_id": "be-dev-1",
"input_tokens": 300,
"output_tokens": 100,
"model": "sonnet",
"cost_estimate": 0.0015,
},
{
"agent_id": "be-dev-2",
"input_tokens": 200,
"output_tokens": 100,
"model": "sonnet",
"cost_estimate": 0.0010,
},
],
),
)
bus.publish.assert_awaited_once()
event = bus.publish.await_args.args[0]
assert event.type.value == "usage.snapshot"
assert event.data["period"] == "live"
assert event.data["totals"]["input_tokens"] == expected_input
assert event.data["cost_estimate"] == expected_cost
assert len(event.data["by_agent"]) == expected_agents
assert "timestamp" in event.data
@pytest.mark.asyncio
async def test_publish_usage_snapshot_twice_both_published() -> None:
"""No throttle on snapshot: two rapid calls both publish."""
bus = MagicMock()
bus.publish = AsyncMock()
expected_calls = 2
for _ in range(expected_calls):
await publish_usage_snapshot(
bus,
UsageSnapshot(
period="live",
totals={"input_tokens": 0, "output_tokens": 0},
cost_estimate=0.0,
by_agent=[],
),
)
assert bus.publish.await_count == expected_calls