[4865ff8b] Add WebSocket support to the usage dashboard (#115)

* [e7349d84] feat(dashboard): WS usage store, hook extension, status badge, and smooth animations (#111) (#113)

- Add src/store/usage-store.ts with typed UsageData interface, useUsageStore
  Zustand store, setUsageData, clearUsageData, and setWsState actions
- Export useUsageStore and UsageData from store/index.ts
- Extend use-rate-limit-websocket.ts: rename msg type to SystemWsMessage,
  add key_metrics field; add useEffect syncing wsState into useUsageStore;
  add USAGE_UPDATE/USAGE_SNAPSHOT handler dispatching to useUsageStore
  (RATE_LIMIT_HIT/LIFTED handling and onReconnect unchanged)
- Update CommandCenter to read key_metrics from useUsageStore when
  wsState === 'connected' and usageData non-null; falls back to
  useCeoOverview() (refetchInterval: 60000) when WS disconnected
- Update KeyMetricsPanel: add wsState prop, render connection status Badge
  matching AgentStreamViewer pattern (bg-green-500+Wifi / bg-yellow-500+
  Loader2 spin / bg-gray-500+WifiOff); add transition-all duration-300
  ease-in-out to metric value spans for smooth animated updates

Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev>

* [c9745ee8] feat(events): add USAGE_UPDATE/SNAPSHOT event types, throttled publisher, /ws/system usage bridge (#112) (#114)

- Add EventType.USAGE_UPDATE='usage.update' and EventType.USAGE_SNAPSHOT='usage.snapshot'
  to the EventType StrEnum in roboco/models/events.py

- Create roboco/services/usage_events.py with _UsageThrottle class (5-second per-agent
  window using time.monotonic()) and publish_usage_update() / publish_usage_snapshot()
  helpers; lazy imports prevent circular dependency with roboco.events

- Extend orchestrator._sweep_token_snapshots() to publish USAGE_UPDATE per active agent
  (throttled) and a USAGE_SNAPSHOT aggregate after each sweep cycle; wrapped in
  contextlib.suppress so event errors never abort DB snapshot operations

- Add _handle_usage_event() to websocket_bridge.py following _handle_rate_limit_event
  pattern; register USAGE_UPDATE and USAGE_SNAPSHOT subscriptions in
  register_websocket_bridge_handlers() forwarding both to /ws/system via broadcast_system()

- Add unit tests: test_usage_events.py (throttle suppression, publish helpers) and
  test_websocket_bridge.py extended with _handle_usage_event coverage and updated
  registration assertion to include USAGE_UPDATE/USAGE_SNAPSHOT

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

* fix(usage-ws): reconcile the realtime token/cost contract end-to-end

The backend and frontend halves shipped mismatched contracts, so the usage
dashboard never received live data:

- The bridge forwarded the dotted event value ("usage.update") while the panel
  switched on "USAGE_UPDATE"; map both to the UPPER_SNAKE type string the same
  way the rate-limit handler does.
- The backend emitted token/cost telemetry but the frontend read a key_metrics
  field and fed the org-metrics panel. Rewire the frontend to consume the
  USAGE_SNAPSHOT token/cost payload into the "Token Usage & Cost" panel —
  WS-first with polling fallback and a connection-status badge — and revert the
  unrelated KeyMetricsPanel / CommandCenter wiring.

Backend cleanups in the same path:

- Replace the multi-argument publish helpers with typed UsageUpdate /
  UsageSnapshot payloads, removing the too-many-arguments lint suppressions.
- Extract _fetch_agent_tokens and _persist_token_snapshot from the token sweep,
  removing the too-many-statements suppression; label the live snapshot "live".

Hardening uncovered while fixing the above:

- _finalize_spawn_session pulled the full RAG stack into the
  session-finalization path through a transcript-parse import; move the pure
  parser into a dependency-light roboco.agent_sdk.transcript_usage module so
  finalization never imports the agent SDK server.
- Reduce _finalize_spawn_session complexity by extracting
  _resolve_final_token_usage, and widen the transcript-fallback guard so a read
  error can never abort finalization.

Also align KeyMetricsPanel with the metrics /dashboard/ceo actually returns: it
read velocity_24h / avg_time_to_done / active_agents, none of which
get_key_metrics() emits, so four of five rows rendered "—". Render
velocity_weekly, completion_rate, documentation_coverage and active_blockers.

* docs: note live usage push over /ws/system on the usage dashboard

* fix(usage): finalize on self-exit and de-duplicate transcript token counts

Two bugs left token capture broken even after the transcript-read fallback
landed — surfaced by a live agent run:

- Agents that self-exit (the normal i_am_idle -> container shutdown, exit 0)
  were never finalized. _finalize_spawn_session is only called from
  stop_agent(), but a graceful self-exit goes through _handle_stopped_container,
  which set the instance OFFLINE and returned without finalizing — leaving the
  spawn-session row open with zero tokens. Finalize there for both graceful
  (exit_reason="completed") and crash (exit_reason="crashed") exits.

- sum_transcript_usage double-counted. Claude Code logs one assistant message
  as several JSONL lines (one per content block — thinking / text / tool_use),
  each repeating the same message.usage, so summing every line roughly doubled
  the totals. De-duplicate by message.id.

Verified against a live agent transcript: the raw sum (12, 1068, 62502, 115828)
vs the de-duped (6, 516, 62502, 63336), which matches the session's
authoritative result.usage exactly.

* feat(usage): fall back to the transcript in the live token sweep

The 60s token sweep read only the agent SDK's /usage/status, which races
container teardown and reports zero mid-run — so live usage (and the
USAGE_SNAPSHOT pushed to /ws/system) stayed at zero for active agents.
Extract _resolve_active_tokens: try the SDK, then fall back to the durable
transcript (the same source finalize uses) so running agents report live.

* feat(usage): add GET /usage/sessions for the dashboard's Recent Sessions

The panel's Recent Sessions table was mock-only — the backend had no sessions
endpoint, so production always showed 'No sessions recorded yet'. Add
UsageService.get_recent_sessions + a /usage/sessions route returning the most
recent spawn-session rows (token totals + cost), and point the panel client
at it.

---------

Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev>
Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
2026-06-11 23:19:50 +02:00
committed by GitHub
co-authored by Frontend Developer 1 Backend Developer 1 Renn F
parent 1d1ec88aad
commit 547fe444f2
22 changed files with 1300 additions and 187 deletions
+3 -31
View File
@@ -41,6 +41,9 @@ from roboco.agent_sdk.models import (
VerbAttemptRequest,
VerbCircuitStatus,
)
from roboco.agent_sdk.transcript_usage import (
sum_transcript_usage as _sum_transcript_usage,
)
from roboco.foundation.policy.agent_loop import DEFAULT_BUDGET as _BUDGET
from roboco.foundation.policy.agent_loop import retry_limit_for
from roboco.services.gateway.envelope import Envelope
@@ -741,37 +744,6 @@ def _token_usage_snapshot() -> TokenUsageStatus:
)
def _sum_transcript_usage(path: Path) -> tuple[int, int, int, int]:
"""Sum per-message token usage across a Claude Code JSONL transcript.
Each assistant entry carries a ``message.usage`` block with the token
counts for that API response; summing them yields the session total.
Returns ``(input, output, cache_read, cache_write)``. Malformed lines are
skipped — a single bad line must never lose the whole count.
"""
tin = tout = tcr = tcw = 0
with path.open("r", encoding="utf-8", errors="ignore") as fh:
for raw in fh:
stripped = raw.strip()
if not stripped:
continue
try:
entry = json.loads(stripped)
except (ValueError, TypeError):
continue
message = entry.get("message")
if not isinstance(message, dict):
continue
usage = message.get("usage")
if not isinstance(usage, dict):
continue
tin += int(usage.get("input_tokens", 0) or 0)
tout += int(usage.get("output_tokens", 0) or 0)
tcr += int(usage.get("cache_read_input_tokens", 0) or 0)
tcw += int(usage.get("cache_creation_input_tokens", 0) or 0)
return tin, tout, tcr, tcw
@app.post("/usage/sync", response_model=TokenUsageStatus)
async def usage_sync(req: TranscriptSyncRequest) -> TokenUsageStatus:
"""Parse the Claude Code transcript and *set* cumulative token totals.
+84
View File
@@ -0,0 +1,84 @@
"""Claude Code transcript token-usage parsing.
A dependency-light helper (only ``json`` + ``pathlib``) so callers that need
durable token counts — notably the orchestrator's session-finalization path —
can read them without importing the agent SDK server, which pulls in the
FastAPI / RAG (piragi / openai) stack.
"""
from __future__ import annotations
import json
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from pathlib import Path
def _coerce_int(value: Any) -> int:
"""Coerce a transcript usage value to int, treating null/garbage as zero."""
try:
return int(value)
except (TypeError, ValueError):
return 0
def _line_usage(line: str) -> tuple[str | None, tuple[int, int, int, int]] | None:
"""Parse one transcript line into ``(message_id, token deltas)``.
Returns ``None`` for blank lines, malformed JSON, or entries without a
``message.usage`` block. ``message_id`` lets the caller de-duplicate:
Claude Code logs a single assistant message as several lines (one per
content block — thinking, text, tool_use), each repeating the *same*
``usage``, so counting every line would multiply the totals.
"""
stripped = line.strip()
if not stripped:
return None
try:
entry = json.loads(stripped)
except (ValueError, TypeError):
return None
message = entry.get("message")
if not isinstance(message, dict):
return None
usage = message.get("usage")
if not isinstance(usage, dict):
return None
deltas = (
_coerce_int(usage.get("input_tokens")),
_coerce_int(usage.get("output_tokens")),
_coerce_int(usage.get("cache_read_input_tokens")),
_coerce_int(usage.get("cache_creation_input_tokens")),
)
return message.get("id"), deltas
def sum_transcript_usage(path: Path) -> tuple[int, int, int, int]:
"""Sum per-message token usage across a Claude Code JSONL transcript.
Each assistant message carries a ``message.usage`` block with the token
counts for that API response; summing them yields the session total.
Messages that span several lines (same ``message.id``) are counted once —
Claude Code emits one line per content block, each repeating the message's
usage, so naive summing roughly doubles the totals. Returns
``(input, output, cache_read, cache_write)``. Malformed lines are skipped —
a single bad line must never lose the whole count.
"""
tin = tout = tcr = tcw = 0
seen: set[str] = set()
with path.open("r", encoding="utf-8", errors="ignore") as fh:
for raw in fh:
parsed = _line_usage(raw)
if parsed is None:
continue
message_id, (line_in, line_out, line_cr, line_cw) = parsed
if message_id is not None:
if message_id in seen:
continue
seen.add(message_id)
tin += line_in
tout += line_out
tcr += line_cr
tcw += line_cw
return tin, tout, tcr, tcw
+21
View File
@@ -145,3 +145,24 @@ async def get_cache_efficiency(
"""
svc = get_usage_service(db)
return await svc.get_cache_efficiency(period)
# =============================================================================
# SESSIONS
# =============================================================================
@router.get("/sessions")
async def get_usage_sessions(
db: DbSession,
limit: Annotated[
int, Query(ge=1, le=200, description="Max sessions to return")
] = 50,
) -> list[dict[str, Any]]:
"""Return the most recent agent spawn sessions, newest first.
Each row carries per-session token totals (input / output / cache) and the
estimated cost — the raw rows behind the aggregate usage panels.
"""
svc = get_usage_service(db)
return await svc.get_recent_sessions(limit)
+28
View File
@@ -20,6 +20,11 @@ _RATE_LIMIT_WS_TYPES = {
EventType.RATE_LIMIT_LIFTED: "RATE_LIMIT_LIFTED",
}
_USAGE_WS_TYPES = {
EventType.USAGE_UPDATE: "USAGE_UPDATE",
EventType.USAGE_SNAPSHOT: "USAGE_SNAPSHOT",
}
# Handler for notification events
async def _handle_notification_sent(event: Event) -> None:
@@ -144,6 +149,25 @@ async def _handle_rate_limit_event(event: Event) -> None:
await manager.broadcast_system({"type": ws_type, **event.data})
async def _handle_usage_event(event: Event) -> None:
"""Forward USAGE_UPDATE/SNAPSHOT events to operator system WS clients.
Both event types carry all the fields the panel needs directly in
``event.data``; we tag them with the discriminating ``type`` string the
panel switches on (the same UPPER_SNAKE mapping the rate-limit handler
uses), so the panel can distinguish per-agent updates from aggregate
snapshots.
"""
ws_type = _USAGE_WS_TYPES.get(event.type)
if ws_type is None:
return
await manager.broadcast_system({"type": ws_type, **event.data})
logger.debug(
"Usage event forwarded to system WebSocket",
event_type=ws_type,
)
def register_websocket_bridge_handlers() -> None:
"""
Register event handlers that forward events to WebSocket clients.
@@ -172,6 +196,10 @@ def register_websocket_bridge_handlers() -> None:
bus.subscribe(EventType.RATE_LIMIT_HIT, _handle_rate_limit_event)
bus.subscribe(EventType.RATE_LIMIT_LIFTED, _handle_rate_limit_event)
# Usage events -> system WebSocket (panel dashboard)
bus.subscribe(EventType.USAGE_UPDATE, _handle_usage_event)
bus.subscribe(EventType.USAGE_SNAPSHOT, _handle_usage_event)
logger.info("WebSocket bridge handlers registered")
+4
View File
@@ -69,6 +69,10 @@ class EventType(StrEnum):
RATE_LIMIT_HIT = "rate_limit.hit"
RATE_LIMIT_LIFTED = "rate_limit.lifted"
# Usage events
USAGE_UPDATE = "usage.update"
USAGE_SNAPSHOT = "usage.snapshot"
# Question events
QUESTION_ASKED = "question.asked"
QUESTION_ANSWERED = "question.answered"
+239 -106
View File
@@ -3218,7 +3218,7 @@ class AgentOrchestrator:
fetch, which misses whenever the agent container is short-lived or
already torn down. Returns zeros when no transcript is found.
"""
from roboco.agent_sdk.server import _sum_transcript_usage
from roboco.agent_sdk.transcript_usage import sum_transcript_usage
projects = Path.home() / ".claude" / "projects"
try:
@@ -3228,12 +3228,49 @@ class AgentOrchestrator:
if d.is_dir()
for f in d.glob("*.jsonl")
]
if not jsonl:
return (0, 0, 0, 0)
newest = max(jsonl, key=lambda f: f.stat().st_mtime)
return sum_transcript_usage(newest)
except OSError:
return (0, 0, 0, 0)
if not jsonl:
return (0, 0, 0, 0)
newest = max(jsonl, key=lambda f: f.stat().st_mtime)
return _sum_transcript_usage(newest)
async def _resolve_final_token_usage(
self, agent_id: str
) -> tuple[int, int, int, int]:
"""Resolve final token counts for a stopping agent.
Tries the live SDK ``/usage/status`` first; if that misses the SDK's
in-memory counts race container teardown for short-lived agents it
falls back to the agent's Claude Code transcript, which is durable and
mounted into this container. Returns
``(input, output, cache_read, cache_write)``.
"""
tokens = (0, 0, 0, 0)
sdk_url = f"http://roboco-agent-{agent_id}:{SDK_PORT}/usage/status"
try:
async with httpx.AsyncClient(timeout=3.0) as client:
resp = await client.get(sdk_url)
if resp.status_code == http_status.HTTP_200_OK:
data = resp.json()
tokens = (
data.get("tokens_input", 0),
data.get("tokens_output", 0),
data.get("tokens_cache_read", 0),
data.get("tokens_cache_write", 0),
)
except Exception as sdk_exc:
logger.debug(
"Could not fetch final token counts from SDK",
agent_id=agent_id,
error=str(sdk_exc),
)
if not tokens[0] and not tokens[1]:
tin, tout, cr, cw = self._usage_from_transcript(agent_id)
if tin or tout:
tokens = (tin, tout, cr, cw)
return tokens
async def _finalize_spawn_session(
self,
@@ -3242,9 +3279,9 @@ class AgentOrchestrator:
) -> None:
"""Close the open agent_spawn_sessions row for this agent.
Fetches final token counts from the agent SDK's /usage/status endpoint,
calculates cost via pricing module, then updates the DB row with
ended_at, token totals, exit_reason, and estimated_cost_usd.
Resolves final token counts (live SDK, with a durable transcript
fallback), calculates cost via the pricing module, then updates the DB
row with ended_at, token totals, exit_reason, and estimated_cost_usd.
Errors are caught and logged finalization must never block stop_agent.
"""
try:
@@ -3252,44 +3289,16 @@ class AgentOrchestrator:
from roboco.db.base import get_session_factory
from roboco.db.tables import AgentSpawnSessionTable
# Fetch final token counts from the agent's SDK
sdk_url = f"http://roboco-agent-{agent_id}:{SDK_PORT}/usage/status"
tokens_input = 0
tokens_output = 0
tokens_cache_read = 0
tokens_cache_write = 0
model = "unknown"
try:
async with httpx.AsyncClient(timeout=3.0) as client:
resp = await client.get(sdk_url)
if resp.status_code == http_status.HTTP_200_OK:
data = resp.json()
tokens_input = data.get("tokens_input", 0)
tokens_output = data.get("tokens_output", 0)
tokens_cache_read = data.get("tokens_cache_read", 0)
tokens_cache_write = data.get("tokens_cache_write", 0)
except Exception as sdk_exc:
logger.debug(
"Could not fetch final token counts from SDK",
agent_id=agent_id,
error=str(sdk_exc),
)
# The live SDK fetch above races the container teardown and misses
# for short-lived agents (counts live in the SDK server's memory,
# which dies with the container). Fall back to the durable source of
# truth: the agent's Claude Code transcript, mounted into this
# container — so usage is captured regardless of container timing.
if not tokens_input and not tokens_output:
tin, tout, cr, cw = self._usage_from_transcript(agent_id)
if tin or tout:
tokens_input = tin
tokens_output = tout
tokens_cache_read = cr
tokens_cache_write = cw
# Resolve final token counts (live SDK, with transcript fallback).
(
tokens_input,
tokens_output,
tokens_cache_read,
tokens_cache_write,
) = await self._resolve_final_token_usage(agent_id)
# Look up the model and usage_session_id from the running instance config.
model = "unknown"
instance = self._instances.get(agent_id)
if instance and instance.config:
model = instance.config.model or "unknown"
@@ -3357,6 +3366,113 @@ class AgentOrchestrator:
error=str(exc),
)
@staticmethod
async def _fetch_agent_tokens(
client: httpx.AsyncClient, agent_id: str
) -> tuple[int, int, int, int] | None:
"""Fetch cumulative token counts from an agent's SDK usage endpoint.
Returns ``(input, output, cache_read, cache_write)`` or ``None`` when the
agent returns a non-200 status or has not accrued any tokens yet.
"""
sdk_url = f"http://roboco-agent-{agent_id}:{SDK_PORT}/usage/status"
resp = await client.get(sdk_url)
if resp.status_code != http_status.HTTP_200_OK:
return None
data = resp.json()
tokens = (
data.get("tokens_input", 0),
data.get("tokens_output", 0),
data.get("tokens_cache_read", 0),
data.get("tokens_cache_write", 0),
)
if sum(tokens) == 0:
return None
return tokens
async def _resolve_active_tokens(
self, client: httpx.AsyncClient, agent_id: str
) -> tuple[int, int, int, int] | None:
"""Resolve live token counts for an active agent.
Tries the agent SDK's ``/usage/status`` first; on a zero/miss falls
back to the durable transcript (the SDK can report zero mid-run, the
same race the finalize path handles). Returns ``None`` when neither
source has any usage yet.
"""
tokens = await self._fetch_agent_tokens(client, agent_id)
if tokens is not None:
return tokens
transcript = self._usage_from_transcript(agent_id)
return transcript if any(transcript) else None
@staticmethod
async def _persist_token_snapshot(
session_factory: Any,
agent_id: str,
instance: AgentInstance,
tokens: tuple[int, int, int, int],
) -> bool:
"""Insert a token_usage_snapshots row and refresh the open session totals.
Returns True when a snapshot was written; False when the agent has no
open spawn-session row to attach it to.
"""
from uuid import uuid4
from sqlalchemy import select, update
from roboco.db.tables import AgentSpawnSessionTable, TokenUsageSnapshotTable
tokens_input, tokens_output, tokens_cache_read, tokens_cache_write = tokens
async with session_factory() as db:
# Prefer a direct lookup by the session UUID captured at spawn time;
# fall back to the agent_slug heuristic for instances that pre-date
# the usage_session_id field.
if instance.usage_session_id is not None:
result = await db.execute(
select(AgentSpawnSessionTable).where(
AgentSpawnSessionTable.id == instance.usage_session_id
)
)
else:
result = await db.execute(
select(AgentSpawnSessionTable)
.where(
AgentSpawnSessionTable.agent_slug == agent_id,
AgentSpawnSessionTable.ended_at.is_(None),
)
.order_by(AgentSpawnSessionTable.started_at.desc())
.limit(1)
)
session_row = result.scalar_one_or_none()
if session_row is None:
return False
db.add(
TokenUsageSnapshotTable(
id=uuid4(),
agent_spawn_session_id=session_row.id,
snapshotted_at=datetime.now(UTC),
tokens_input=tokens_input,
tokens_output=tokens_output,
tokens_cache_read=tokens_cache_read,
tokens_cache_write=tokens_cache_write,
)
)
await db.execute(
update(AgentSpawnSessionTable)
.where(AgentSpawnSessionTable.id == session_row.id)
.values(
tokens_input=tokens_input,
tokens_output=tokens_output,
tokens_cache_read=tokens_cache_read,
tokens_cache_write=tokens_cache_write,
)
)
await db.commit()
return True
async def _sweep_token_snapshots(self) -> None:
"""Write a token_usage_snapshots row for each active agent with non-zero tokens.
@@ -3364,18 +3480,26 @@ class AgentOrchestrator:
token counts on the open agent_spawn_sessions row so the DB reflects
current progress without waiting for session close.
Errors per-agent are caught so one bad agent doesn't abort the whole sweep.
Additionally publishes USAGE_UPDATE events per agent (throttled to at most
one per 5-second window) and a USAGE_SNAPSHOT aggregate after the loop.
"""
if not self._instances:
return
try:
from roboco.db.base import get_session_factory
from roboco.db.tables import AgentSpawnSessionTable, TokenUsageSnapshotTable
except ImportError:
return
session_factory = get_session_factory()
# Accumulators for the post-loop USAGE_SNAPSHOT event.
_usage_by_agent: list[dict[str, Any]] = []
_usage_total_input = 0
_usage_total_output = 0
_usage_total_cost = 0.0
async with httpx.AsyncClient(timeout=3.0) as client:
for agent_id, instance in list(self._instances.items()):
if instance.state not in (
@@ -3384,80 +3508,60 @@ class AgentOrchestrator:
):
continue
sdk_url = f"http://roboco-agent-{agent_id}:{SDK_PORT}/usage/status"
try:
resp = await client.get(sdk_url)
if resp.status_code != http_status.HTTP_200_OK:
tokens = await self._resolve_active_tokens(client, agent_id)
if tokens is None:
continue
data = resp.json()
tokens_input = data.get("tokens_input", 0)
tokens_output = data.get("tokens_output", 0)
tokens_cache_read = data.get("tokens_cache_read", 0)
tokens_cache_write = data.get("tokens_cache_write", 0)
# Skip agents with no token usage yet
total = (
tokens_input
+ tokens_output
+ tokens_cache_read
+ tokens_cache_write
persisted = await self._persist_token_snapshot(
session_factory, agent_id, instance, tokens
)
if total == 0:
if not persisted:
continue
async with session_factory() as db:
from sqlalchemy import select, update
tokens_input, tokens_output = tokens[0], tokens[1]
model = instance.config.model if instance.config else "unknown"
# Prefer a direct lookup by the session UUID captured at
# spawn time; fall back to the agent_slug heuristic for
# instances that pre-date the usage_session_id field.
if instance.usage_session_id is not None:
result = await db.execute(
select(AgentSpawnSessionTable).where(
AgentSpawnSessionTable.id
== instance.usage_session_id
)
)
else:
result = await db.execute(
select(AgentSpawnSessionTable)
.where(
AgentSpawnSessionTable.agent_slug == agent_id,
AgentSpawnSessionTable.ended_at.is_(None),
)
.order_by(AgentSpawnSessionTable.started_at.desc())
.limit(1)
)
session_row = result.scalar_one_or_none()
if session_row is None:
continue
# Publish USAGE_UPDATE event for this agent (throttled).
with contextlib.suppress(Exception):
from roboco.events import get_event_bus
from roboco.services.usage_events import (
UsageUpdate,
publish_usage_update,
)
# Insert snapshot
from uuid import uuid4 as _uuid4
await publish_usage_update(
get_event_bus(),
UsageUpdate(
agent_id=agent_id,
task_id=instance.current_task_id,
input_tokens=tokens_input,
output_tokens=tokens_output,
model=model,
),
)
snapshot = TokenUsageSnapshotTable(
id=_uuid4(),
agent_spawn_session_id=session_row.id,
snapshotted_at=datetime.now(UTC),
# Accumulate per-agent data for the aggregate snapshot.
with contextlib.suppress(Exception):
from roboco.billing.pricing import calculate_cost
agent_cost = calculate_cost(
model=model,
tokens_input=tokens_input,
tokens_output=tokens_output,
tokens_cache_read=tokens_cache_read,
tokens_cache_write=tokens_cache_write,
)
db.add(snapshot)
# Update cumulative totals on the session row
await db.execute(
update(AgentSpawnSessionTable)
.where(AgentSpawnSessionTable.id == session_row.id)
.values(
tokens_input=tokens_input,
tokens_output=tokens_output,
tokens_cache_read=tokens_cache_read,
tokens_cache_write=tokens_cache_write,
)
_usage_by_agent.append(
{
"agent_id": agent_id,
"input_tokens": tokens_input,
"output_tokens": tokens_output,
"model": model,
"cost_estimate": agent_cost,
}
)
await db.commit()
_usage_total_input += tokens_input
_usage_total_output += tokens_output
_usage_total_cost += agent_cost
except Exception as agent_exc:
logger.debug(
@@ -3466,6 +3570,28 @@ class AgentOrchestrator:
error=str(agent_exc),
)
# Publish a USAGE_SNAPSHOT aggregate if any active agents had token data.
if _usage_by_agent:
with contextlib.suppress(Exception):
from roboco.events import get_event_bus
from roboco.services.usage_events import (
UsageSnapshot,
publish_usage_snapshot,
)
await publish_usage_snapshot(
get_event_bus(),
UsageSnapshot(
period="live",
totals={
"input_tokens": _usage_total_input,
"output_tokens": _usage_total_output,
},
cost_estimate=_usage_total_cost,
by_agent=_usage_by_agent,
),
)
async def _sweep_daily_rollup(self) -> None:
"""Upsert daily_usage_rollups from closed agent_spawn_sessions.
@@ -3910,6 +4036,13 @@ Start by:
container_id=cid,
exit_code=exit_code,
)
# The agent self-exited (a graceful i_am_idle shutdown, or a crash), so
# stop_agent() — which normally finalizes — was never called. Finalize
# here to capture token usage from the transcript; otherwise the
# spawn-session row is left open with zero tokens.
await self._finalize_spawn_session(
agent_id, exit_reason="completed" if graceful else "crashed"
)
instance.state = AgentState.OFFLINE
instance.container_id = None
if graceful:
+30
View File
@@ -35,6 +35,23 @@ def _row_tokens(row: Any) -> tuple[int, int, int, int]:
)
def _session_row(row: Any) -> dict[str, Any]:
"""Shape one spawn-session row for the dashboard's sessions table."""
tin, tout, tcr, tcw = _row_tokens(row)
return {
"id": str(row.id),
"agent_slug": row.agent_slug,
"model": row.model,
"started_at": row.started_at.isoformat(),
"ended_at": row.ended_at.isoformat() if row.ended_at else None,
"tokens_input": tin,
"tokens_output": tout,
"tokens_cache": tcr + tcw,
"total_tokens": tin + tout + tcr + tcw,
"cost": float(row.estimated_cost_usd or 0.0),
}
def _parse_period(period: str) -> tuple[datetime, int]:
"""Parse period string into (start_dt, hours).
@@ -441,6 +458,19 @@ class UsageService(BaseService):
"cost_today_usd": round(float(row.total_cost_usd or 0.0), 6),
}
async def get_recent_sessions(self, limit: int = 50) -> list[dict[str, Any]]:
"""Return the most recent spawn sessions, newest first.
These are the raw per-session rows behind the aggregate panels — the
dashboard's "Recent Sessions" table.
"""
result = await self.session.execute(
select(AgentSpawnSessionTable)
.order_by(AgentSpawnSessionTable.started_at.desc())
.limit(limit)
)
return [_session_row(row) for row in result.scalars().all()]
def get_usage_service(db: AsyncSession) -> UsageService:
"""Factory function matching the pattern used by other services."""
+129
View File
@@ -0,0 +1,129 @@
"""
Usage Event Publisher
Throttled helpers for publishing USAGE_UPDATE and USAGE_SNAPSHOT events
to the StreamEventBus. Consumed by the orchestrator token sweep and
forwarded to /ws/system WebSocket clients via the websocket_bridge.
Throttle window: one USAGE_UPDATE publish per agent per 5-second window.
Subsequent calls within the window are silently dropped so a frequent
sweep loop cannot flood the event bus or WebSocket clients.
USAGE_SNAPSHOT is always published (no per-agent throttle) — it is an
aggregate and published at most once per sweep cycle.
"""
from __future__ import annotations
import time
from dataclasses import dataclass
from datetime import UTC, datetime
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from roboco.events.stream_bus import StreamEventBus
_THROTTLE_WINDOW_SECONDS: float = 5.0
class _UsageThrottle:
"""Per-agent last-publish timestamp tracker.
Uses ``time.monotonic()`` so clock adjustments (NTP, DST) don't cause
spurious suppressions or double-fires.
"""
def __init__(self, window: float = _THROTTLE_WINDOW_SECONDS) -> None:
self._window = window
self._last: dict[str, float] = {}
def should_publish(self, agent_id: str) -> bool:
"""Return True if the agent is outside the throttle window.
Also records the current monotonic time as the new *last published*
timestamp when it returns True, so the caller does not need to call a
separate ``record()`` method.
"""
now = time.monotonic()
if now - self._last.get(agent_id, 0.0) >= self._window:
self._last[agent_id] = now
return True
return False
# Module-level singleton — shared across all callers in this process.
_throttle = _UsageThrottle()
@dataclass(frozen=True, slots=True)
class UsageUpdate:
"""Per-agent cumulative token counts carried by a USAGE_UPDATE event.
Fields map directly onto the event payload the panel consumes (the
backend emits these per active agent during the token sweep).
"""
agent_id: str
task_id: str | None
input_tokens: int
output_tokens: int
model: str
timestamp: datetime | None = None
def event_data(self) -> dict[str, Any]:
"""Render the event payload, stamping ``timestamp`` if not supplied."""
return {
"agent_id": self.agent_id,
"task_id": self.task_id,
"input_tokens": self.input_tokens,
"output_tokens": self.output_tokens,
"model": self.model,
"timestamp": (self.timestamp or datetime.now(UTC)).isoformat(),
}
@dataclass(frozen=True, slots=True)
class UsageSnapshot:
"""Aggregate token/cost totals carried by a USAGE_SNAPSHOT event.
``totals`` is ``{"input_tokens": int, "output_tokens": int}``; ``by_agent``
is the per-agent breakdown (each item carries ``agent_id``, token counts,
``model`` and ``cost_estimate``).
"""
period: str
totals: dict[str, int]
cost_estimate: float
by_agent: list[dict[str, Any]]
timestamp: datetime | None = None
def event_data(self) -> dict[str, Any]:
"""Render the event payload, stamping ``timestamp`` if not supplied."""
return {
"period": self.period,
"totals": self.totals,
"cost_estimate": self.cost_estimate,
"by_agent": self.by_agent,
"timestamp": (self.timestamp or datetime.now(UTC)).isoformat(),
}
async def publish_usage_update(bus: StreamEventBus, update: UsageUpdate) -> bool:
"""Publish a USAGE_UPDATE event if the per-agent throttle window has elapsed.
Returns True if the event was published; False if suppressed by the throttle.
"""
if not _throttle.should_publish(update.agent_id):
return False
from roboco.models.events import Event, EventType # lazy — avoids circular import
await bus.publish(Event(type=EventType.USAGE_UPDATE, data=update.event_data()))
return True
async def publish_usage_snapshot(bus: StreamEventBus, snapshot: UsageSnapshot) -> None:
"""Publish a USAGE_SNAPSHOT aggregate event (no throttle)."""
from roboco.models.events import Event, EventType # lazy — avoids circular import
await bus.publish(Event(type=EventType.USAGE_SNAPSHOT, data=snapshot.event_data()))