mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
[4865ff8b] Add WebSocket support to the usage dashboard (#115)
* [e7349d84] feat(dashboard): WS usage store, hook extension, status badge, and smooth animations (#111) (#113) - Add src/store/usage-store.ts with typed UsageData interface, useUsageStore Zustand store, setUsageData, clearUsageData, and setWsState actions - Export useUsageStore and UsageData from store/index.ts - Extend use-rate-limit-websocket.ts: rename msg type to SystemWsMessage, add key_metrics field; add useEffect syncing wsState into useUsageStore; add USAGE_UPDATE/USAGE_SNAPSHOT handler dispatching to useUsageStore (RATE_LIMIT_HIT/LIFTED handling and onReconnect unchanged) - Update CommandCenter to read key_metrics from useUsageStore when wsState === 'connected' and usageData non-null; falls back to useCeoOverview() (refetchInterval: 60000) when WS disconnected - Update KeyMetricsPanel: add wsState prop, render connection status Badge matching AgentStreamViewer pattern (bg-green-500+Wifi / bg-yellow-500+ Loader2 spin / bg-gray-500+WifiOff); add transition-all duration-300 ease-in-out to metric value spans for smooth animated updates Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev> * [c9745ee8] feat(events): add USAGE_UPDATE/SNAPSHOT event types, throttled publisher, /ws/system usage bridge (#112) (#114) - Add EventType.USAGE_UPDATE='usage.update' and EventType.USAGE_SNAPSHOT='usage.snapshot' to the EventType StrEnum in roboco/models/events.py - Create roboco/services/usage_events.py with _UsageThrottle class (5-second per-agent window using time.monotonic()) and publish_usage_update() / publish_usage_snapshot() helpers; lazy imports prevent circular dependency with roboco.events - Extend orchestrator._sweep_token_snapshots() to publish USAGE_UPDATE per active agent (throttled) and a USAGE_SNAPSHOT aggregate after each sweep cycle; wrapped in contextlib.suppress so event errors never abort DB snapshot operations - Add _handle_usage_event() to websocket_bridge.py following _handle_rate_limit_event pattern; register USAGE_UPDATE and USAGE_SNAPSHOT subscriptions in register_websocket_bridge_handlers() forwarding both to /ws/system via broadcast_system() - Add unit tests: test_usage_events.py (throttle suppression, publish helpers) and test_websocket_bridge.py extended with _handle_usage_event coverage and updated registration assertion to include USAGE_UPDATE/USAGE_SNAPSHOT Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> * fix(usage-ws): reconcile the realtime token/cost contract end-to-end The backend and frontend halves shipped mismatched contracts, so the usage dashboard never received live data: - The bridge forwarded the dotted event value ("usage.update") while the panel switched on "USAGE_UPDATE"; map both to the UPPER_SNAKE type string the same way the rate-limit handler does. - The backend emitted token/cost telemetry but the frontend read a key_metrics field and fed the org-metrics panel. Rewire the frontend to consume the USAGE_SNAPSHOT token/cost payload into the "Token Usage & Cost" panel — WS-first with polling fallback and a connection-status badge — and revert the unrelated KeyMetricsPanel / CommandCenter wiring. Backend cleanups in the same path: - Replace the multi-argument publish helpers with typed UsageUpdate / UsageSnapshot payloads, removing the too-many-arguments lint suppressions. - Extract _fetch_agent_tokens and _persist_token_snapshot from the token sweep, removing the too-many-statements suppression; label the live snapshot "live". Hardening uncovered while fixing the above: - _finalize_spawn_session pulled the full RAG stack into the session-finalization path through a transcript-parse import; move the pure parser into a dependency-light roboco.agent_sdk.transcript_usage module so finalization never imports the agent SDK server. - Reduce _finalize_spawn_session complexity by extracting _resolve_final_token_usage, and widen the transcript-fallback guard so a read error can never abort finalization. Also align KeyMetricsPanel with the metrics /dashboard/ceo actually returns: it read velocity_24h / avg_time_to_done / active_agents, none of which get_key_metrics() emits, so four of five rows rendered "—". Render velocity_weekly, completion_rate, documentation_coverage and active_blockers. * docs: note live usage push over /ws/system on the usage dashboard * fix(usage): finalize on self-exit and de-duplicate transcript token counts Two bugs left token capture broken even after the transcript-read fallback landed — surfaced by a live agent run: - Agents that self-exit (the normal i_am_idle -> container shutdown, exit 0) were never finalized. _finalize_spawn_session is only called from stop_agent(), but a graceful self-exit goes through _handle_stopped_container, which set the instance OFFLINE and returned without finalizing — leaving the spawn-session row open with zero tokens. Finalize there for both graceful (exit_reason="completed") and crash (exit_reason="crashed") exits. - sum_transcript_usage double-counted. Claude Code logs one assistant message as several JSONL lines (one per content block — thinking / text / tool_use), each repeating the same message.usage, so summing every line roughly doubled the totals. De-duplicate by message.id. Verified against a live agent transcript: the raw sum (12, 1068, 62502, 115828) vs the de-duped (6, 516, 62502, 63336), which matches the session's authoritative result.usage exactly. * feat(usage): fall back to the transcript in the live token sweep The 60s token sweep read only the agent SDK's /usage/status, which races container teardown and reports zero mid-run — so live usage (and the USAGE_SNAPSHOT pushed to /ws/system) stayed at zero for active agents. Extract _resolve_active_tokens: try the SDK, then fall back to the durable transcript (the same source finalize uses) so running agents report live. * feat(usage): add GET /usage/sessions for the dashboard's Recent Sessions The panel's Recent Sessions table was mock-only — the backend had no sessions endpoint, so production always showed 'No sessions recorded yet'. Add UsageService.get_recent_sessions + a /usage/sessions route returning the most recent spawn-session rows (token totals + cost), and point the panel client at it. --------- Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev> Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
co-authored by
Frontend Developer 1
Backend Developer 1
Renn F
parent
1d1ec88aad
commit
547fe444f2
@@ -41,6 +41,9 @@ from roboco.agent_sdk.models import (
|
||||
VerbAttemptRequest,
|
||||
VerbCircuitStatus,
|
||||
)
|
||||
from roboco.agent_sdk.transcript_usage import (
|
||||
sum_transcript_usage as _sum_transcript_usage,
|
||||
)
|
||||
from roboco.foundation.policy.agent_loop import DEFAULT_BUDGET as _BUDGET
|
||||
from roboco.foundation.policy.agent_loop import retry_limit_for
|
||||
from roboco.services.gateway.envelope import Envelope
|
||||
@@ -741,37 +744,6 @@ def _token_usage_snapshot() -> TokenUsageStatus:
|
||||
)
|
||||
|
||||
|
||||
def _sum_transcript_usage(path: Path) -> tuple[int, int, int, int]:
|
||||
"""Sum per-message token usage across a Claude Code JSONL transcript.
|
||||
|
||||
Each assistant entry carries a ``message.usage`` block with the token
|
||||
counts for that API response; summing them yields the session total.
|
||||
Returns ``(input, output, cache_read, cache_write)``. Malformed lines are
|
||||
skipped — a single bad line must never lose the whole count.
|
||||
"""
|
||||
tin = tout = tcr = tcw = 0
|
||||
with path.open("r", encoding="utf-8", errors="ignore") as fh:
|
||||
for raw in fh:
|
||||
stripped = raw.strip()
|
||||
if not stripped:
|
||||
continue
|
||||
try:
|
||||
entry = json.loads(stripped)
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
message = entry.get("message")
|
||||
if not isinstance(message, dict):
|
||||
continue
|
||||
usage = message.get("usage")
|
||||
if not isinstance(usage, dict):
|
||||
continue
|
||||
tin += int(usage.get("input_tokens", 0) or 0)
|
||||
tout += int(usage.get("output_tokens", 0) or 0)
|
||||
tcr += int(usage.get("cache_read_input_tokens", 0) or 0)
|
||||
tcw += int(usage.get("cache_creation_input_tokens", 0) or 0)
|
||||
return tin, tout, tcr, tcw
|
||||
|
||||
|
||||
@app.post("/usage/sync", response_model=TokenUsageStatus)
|
||||
async def usage_sync(req: TranscriptSyncRequest) -> TokenUsageStatus:
|
||||
"""Parse the Claude Code transcript and *set* cumulative token totals.
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
"""Claude Code transcript token-usage parsing.
|
||||
|
||||
A dependency-light helper (only ``json`` + ``pathlib``) so callers that need
|
||||
durable token counts — notably the orchestrator's session-finalization path —
|
||||
can read them without importing the agent SDK server, which pulls in the
|
||||
FastAPI / RAG (piragi / openai) stack.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def _coerce_int(value: Any) -> int:
|
||||
"""Coerce a transcript usage value to int, treating null/garbage as zero."""
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError):
|
||||
return 0
|
||||
|
||||
|
||||
def _line_usage(line: str) -> tuple[str | None, tuple[int, int, int, int]] | None:
|
||||
"""Parse one transcript line into ``(message_id, token deltas)``.
|
||||
|
||||
Returns ``None`` for blank lines, malformed JSON, or entries without a
|
||||
``message.usage`` block. ``message_id`` lets the caller de-duplicate:
|
||||
Claude Code logs a single assistant message as several lines (one per
|
||||
content block — thinking, text, tool_use), each repeating the *same*
|
||||
``usage``, so counting every line would multiply the totals.
|
||||
"""
|
||||
stripped = line.strip()
|
||||
if not stripped:
|
||||
return None
|
||||
try:
|
||||
entry = json.loads(stripped)
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
message = entry.get("message")
|
||||
if not isinstance(message, dict):
|
||||
return None
|
||||
usage = message.get("usage")
|
||||
if not isinstance(usage, dict):
|
||||
return None
|
||||
deltas = (
|
||||
_coerce_int(usage.get("input_tokens")),
|
||||
_coerce_int(usage.get("output_tokens")),
|
||||
_coerce_int(usage.get("cache_read_input_tokens")),
|
||||
_coerce_int(usage.get("cache_creation_input_tokens")),
|
||||
)
|
||||
return message.get("id"), deltas
|
||||
|
||||
|
||||
def sum_transcript_usage(path: Path) -> tuple[int, int, int, int]:
|
||||
"""Sum per-message token usage across a Claude Code JSONL transcript.
|
||||
|
||||
Each assistant message carries a ``message.usage`` block with the token
|
||||
counts for that API response; summing them yields the session total.
|
||||
Messages that span several lines (same ``message.id``) are counted once —
|
||||
Claude Code emits one line per content block, each repeating the message's
|
||||
usage, so naive summing roughly doubles the totals. Returns
|
||||
``(input, output, cache_read, cache_write)``. Malformed lines are skipped —
|
||||
a single bad line must never lose the whole count.
|
||||
"""
|
||||
tin = tout = tcr = tcw = 0
|
||||
seen: set[str] = set()
|
||||
with path.open("r", encoding="utf-8", errors="ignore") as fh:
|
||||
for raw in fh:
|
||||
parsed = _line_usage(raw)
|
||||
if parsed is None:
|
||||
continue
|
||||
message_id, (line_in, line_out, line_cr, line_cw) = parsed
|
||||
if message_id is not None:
|
||||
if message_id in seen:
|
||||
continue
|
||||
seen.add(message_id)
|
||||
tin += line_in
|
||||
tout += line_out
|
||||
tcr += line_cr
|
||||
tcw += line_cw
|
||||
return tin, tout, tcr, tcw
|
||||
@@ -145,3 +145,24 @@ async def get_cache_efficiency(
|
||||
"""
|
||||
svc = get_usage_service(db)
|
||||
return await svc.get_cache_efficiency(period)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# SESSIONS
|
||||
# =============================================================================
|
||||
|
||||
|
||||
@router.get("/sessions")
|
||||
async def get_usage_sessions(
|
||||
db: DbSession,
|
||||
limit: Annotated[
|
||||
int, Query(ge=1, le=200, description="Max sessions to return")
|
||||
] = 50,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Return the most recent agent spawn sessions, newest first.
|
||||
|
||||
Each row carries per-session token totals (input / output / cache) and the
|
||||
estimated cost — the raw rows behind the aggregate usage panels.
|
||||
"""
|
||||
svc = get_usage_service(db)
|
||||
return await svc.get_recent_sessions(limit)
|
||||
|
||||
@@ -20,6 +20,11 @@ _RATE_LIMIT_WS_TYPES = {
|
||||
EventType.RATE_LIMIT_LIFTED: "RATE_LIMIT_LIFTED",
|
||||
}
|
||||
|
||||
_USAGE_WS_TYPES = {
|
||||
EventType.USAGE_UPDATE: "USAGE_UPDATE",
|
||||
EventType.USAGE_SNAPSHOT: "USAGE_SNAPSHOT",
|
||||
}
|
||||
|
||||
|
||||
# Handler for notification events
|
||||
async def _handle_notification_sent(event: Event) -> None:
|
||||
@@ -144,6 +149,25 @@ async def _handle_rate_limit_event(event: Event) -> None:
|
||||
await manager.broadcast_system({"type": ws_type, **event.data})
|
||||
|
||||
|
||||
async def _handle_usage_event(event: Event) -> None:
|
||||
"""Forward USAGE_UPDATE/SNAPSHOT events to operator system WS clients.
|
||||
|
||||
Both event types carry all the fields the panel needs directly in
|
||||
``event.data``; we tag them with the discriminating ``type`` string the
|
||||
panel switches on (the same UPPER_SNAKE mapping the rate-limit handler
|
||||
uses), so the panel can distinguish per-agent updates from aggregate
|
||||
snapshots.
|
||||
"""
|
||||
ws_type = _USAGE_WS_TYPES.get(event.type)
|
||||
if ws_type is None:
|
||||
return
|
||||
await manager.broadcast_system({"type": ws_type, **event.data})
|
||||
logger.debug(
|
||||
"Usage event forwarded to system WebSocket",
|
||||
event_type=ws_type,
|
||||
)
|
||||
|
||||
|
||||
def register_websocket_bridge_handlers() -> None:
|
||||
"""
|
||||
Register event handlers that forward events to WebSocket clients.
|
||||
@@ -172,6 +196,10 @@ def register_websocket_bridge_handlers() -> None:
|
||||
bus.subscribe(EventType.RATE_LIMIT_HIT, _handle_rate_limit_event)
|
||||
bus.subscribe(EventType.RATE_LIMIT_LIFTED, _handle_rate_limit_event)
|
||||
|
||||
# Usage events -> system WebSocket (panel dashboard)
|
||||
bus.subscribe(EventType.USAGE_UPDATE, _handle_usage_event)
|
||||
bus.subscribe(EventType.USAGE_SNAPSHOT, _handle_usage_event)
|
||||
|
||||
logger.info("WebSocket bridge handlers registered")
|
||||
|
||||
|
||||
|
||||
@@ -69,6 +69,10 @@ class EventType(StrEnum):
|
||||
RATE_LIMIT_HIT = "rate_limit.hit"
|
||||
RATE_LIMIT_LIFTED = "rate_limit.lifted"
|
||||
|
||||
# Usage events
|
||||
USAGE_UPDATE = "usage.update"
|
||||
USAGE_SNAPSHOT = "usage.snapshot"
|
||||
|
||||
# Question events
|
||||
QUESTION_ASKED = "question.asked"
|
||||
QUESTION_ANSWERED = "question.answered"
|
||||
|
||||
+239
-106
@@ -3218,7 +3218,7 @@ class AgentOrchestrator:
|
||||
fetch, which misses whenever the agent container is short-lived or
|
||||
already torn down. Returns zeros when no transcript is found.
|
||||
"""
|
||||
from roboco.agent_sdk.server import _sum_transcript_usage
|
||||
from roboco.agent_sdk.transcript_usage import sum_transcript_usage
|
||||
|
||||
projects = Path.home() / ".claude" / "projects"
|
||||
try:
|
||||
@@ -3228,12 +3228,49 @@ class AgentOrchestrator:
|
||||
if d.is_dir()
|
||||
for f in d.glob("*.jsonl")
|
||||
]
|
||||
if not jsonl:
|
||||
return (0, 0, 0, 0)
|
||||
newest = max(jsonl, key=lambda f: f.stat().st_mtime)
|
||||
return sum_transcript_usage(newest)
|
||||
except OSError:
|
||||
return (0, 0, 0, 0)
|
||||
if not jsonl:
|
||||
return (0, 0, 0, 0)
|
||||
newest = max(jsonl, key=lambda f: f.stat().st_mtime)
|
||||
return _sum_transcript_usage(newest)
|
||||
|
||||
async def _resolve_final_token_usage(
|
||||
self, agent_id: str
|
||||
) -> tuple[int, int, int, int]:
|
||||
"""Resolve final token counts for a stopping agent.
|
||||
|
||||
Tries the live SDK ``/usage/status`` first; if that misses — the SDK's
|
||||
in-memory counts race container teardown for short-lived agents — it
|
||||
falls back to the agent's Claude Code transcript, which is durable and
|
||||
mounted into this container. Returns
|
||||
``(input, output, cache_read, cache_write)``.
|
||||
"""
|
||||
tokens = (0, 0, 0, 0)
|
||||
sdk_url = f"http://roboco-agent-{agent_id}:{SDK_PORT}/usage/status"
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=3.0) as client:
|
||||
resp = await client.get(sdk_url)
|
||||
if resp.status_code == http_status.HTTP_200_OK:
|
||||
data = resp.json()
|
||||
tokens = (
|
||||
data.get("tokens_input", 0),
|
||||
data.get("tokens_output", 0),
|
||||
data.get("tokens_cache_read", 0),
|
||||
data.get("tokens_cache_write", 0),
|
||||
)
|
||||
except Exception as sdk_exc:
|
||||
logger.debug(
|
||||
"Could not fetch final token counts from SDK",
|
||||
agent_id=agent_id,
|
||||
error=str(sdk_exc),
|
||||
)
|
||||
|
||||
if not tokens[0] and not tokens[1]:
|
||||
tin, tout, cr, cw = self._usage_from_transcript(agent_id)
|
||||
if tin or tout:
|
||||
tokens = (tin, tout, cr, cw)
|
||||
return tokens
|
||||
|
||||
async def _finalize_spawn_session(
|
||||
self,
|
||||
@@ -3242,9 +3279,9 @@ class AgentOrchestrator:
|
||||
) -> None:
|
||||
"""Close the open agent_spawn_sessions row for this agent.
|
||||
|
||||
Fetches final token counts from the agent SDK's /usage/status endpoint,
|
||||
calculates cost via pricing module, then updates the DB row with
|
||||
ended_at, token totals, exit_reason, and estimated_cost_usd.
|
||||
Resolves final token counts (live SDK, with a durable transcript
|
||||
fallback), calculates cost via the pricing module, then updates the DB
|
||||
row with ended_at, token totals, exit_reason, and estimated_cost_usd.
|
||||
Errors are caught and logged — finalization must never block stop_agent.
|
||||
"""
|
||||
try:
|
||||
@@ -3252,44 +3289,16 @@ class AgentOrchestrator:
|
||||
from roboco.db.base import get_session_factory
|
||||
from roboco.db.tables import AgentSpawnSessionTable
|
||||
|
||||
# Fetch final token counts from the agent's SDK
|
||||
sdk_url = f"http://roboco-agent-{agent_id}:{SDK_PORT}/usage/status"
|
||||
tokens_input = 0
|
||||
tokens_output = 0
|
||||
tokens_cache_read = 0
|
||||
tokens_cache_write = 0
|
||||
model = "unknown"
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=3.0) as client:
|
||||
resp = await client.get(sdk_url)
|
||||
if resp.status_code == http_status.HTTP_200_OK:
|
||||
data = resp.json()
|
||||
tokens_input = data.get("tokens_input", 0)
|
||||
tokens_output = data.get("tokens_output", 0)
|
||||
tokens_cache_read = data.get("tokens_cache_read", 0)
|
||||
tokens_cache_write = data.get("tokens_cache_write", 0)
|
||||
except Exception as sdk_exc:
|
||||
logger.debug(
|
||||
"Could not fetch final token counts from SDK",
|
||||
agent_id=agent_id,
|
||||
error=str(sdk_exc),
|
||||
)
|
||||
|
||||
# The live SDK fetch above races the container teardown and misses
|
||||
# for short-lived agents (counts live in the SDK server's memory,
|
||||
# which dies with the container). Fall back to the durable source of
|
||||
# truth: the agent's Claude Code transcript, mounted into this
|
||||
# container — so usage is captured regardless of container timing.
|
||||
if not tokens_input and not tokens_output:
|
||||
tin, tout, cr, cw = self._usage_from_transcript(agent_id)
|
||||
if tin or tout:
|
||||
tokens_input = tin
|
||||
tokens_output = tout
|
||||
tokens_cache_read = cr
|
||||
tokens_cache_write = cw
|
||||
# Resolve final token counts (live SDK, with transcript fallback).
|
||||
(
|
||||
tokens_input,
|
||||
tokens_output,
|
||||
tokens_cache_read,
|
||||
tokens_cache_write,
|
||||
) = await self._resolve_final_token_usage(agent_id)
|
||||
|
||||
# Look up the model and usage_session_id from the running instance config.
|
||||
model = "unknown"
|
||||
instance = self._instances.get(agent_id)
|
||||
if instance and instance.config:
|
||||
model = instance.config.model or "unknown"
|
||||
@@ -3357,6 +3366,113 @@ class AgentOrchestrator:
|
||||
error=str(exc),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
async def _fetch_agent_tokens(
|
||||
client: httpx.AsyncClient, agent_id: str
|
||||
) -> tuple[int, int, int, int] | None:
|
||||
"""Fetch cumulative token counts from an agent's SDK usage endpoint.
|
||||
|
||||
Returns ``(input, output, cache_read, cache_write)`` or ``None`` when the
|
||||
agent returns a non-200 status or has not accrued any tokens yet.
|
||||
"""
|
||||
sdk_url = f"http://roboco-agent-{agent_id}:{SDK_PORT}/usage/status"
|
||||
resp = await client.get(sdk_url)
|
||||
if resp.status_code != http_status.HTTP_200_OK:
|
||||
return None
|
||||
data = resp.json()
|
||||
tokens = (
|
||||
data.get("tokens_input", 0),
|
||||
data.get("tokens_output", 0),
|
||||
data.get("tokens_cache_read", 0),
|
||||
data.get("tokens_cache_write", 0),
|
||||
)
|
||||
if sum(tokens) == 0:
|
||||
return None
|
||||
return tokens
|
||||
|
||||
async def _resolve_active_tokens(
|
||||
self, client: httpx.AsyncClient, agent_id: str
|
||||
) -> tuple[int, int, int, int] | None:
|
||||
"""Resolve live token counts for an active agent.
|
||||
|
||||
Tries the agent SDK's ``/usage/status`` first; on a zero/miss falls
|
||||
back to the durable transcript (the SDK can report zero mid-run, the
|
||||
same race the finalize path handles). Returns ``None`` when neither
|
||||
source has any usage yet.
|
||||
"""
|
||||
tokens = await self._fetch_agent_tokens(client, agent_id)
|
||||
if tokens is not None:
|
||||
return tokens
|
||||
transcript = self._usage_from_transcript(agent_id)
|
||||
return transcript if any(transcript) else None
|
||||
|
||||
@staticmethod
|
||||
async def _persist_token_snapshot(
|
||||
session_factory: Any,
|
||||
agent_id: str,
|
||||
instance: AgentInstance,
|
||||
tokens: tuple[int, int, int, int],
|
||||
) -> bool:
|
||||
"""Insert a token_usage_snapshots row and refresh the open session totals.
|
||||
|
||||
Returns True when a snapshot was written; False when the agent has no
|
||||
open spawn-session row to attach it to.
|
||||
"""
|
||||
from uuid import uuid4
|
||||
|
||||
from sqlalchemy import select, update
|
||||
|
||||
from roboco.db.tables import AgentSpawnSessionTable, TokenUsageSnapshotTable
|
||||
|
||||
tokens_input, tokens_output, tokens_cache_read, tokens_cache_write = tokens
|
||||
async with session_factory() as db:
|
||||
# Prefer a direct lookup by the session UUID captured at spawn time;
|
||||
# fall back to the agent_slug heuristic for instances that pre-date
|
||||
# the usage_session_id field.
|
||||
if instance.usage_session_id is not None:
|
||||
result = await db.execute(
|
||||
select(AgentSpawnSessionTable).where(
|
||||
AgentSpawnSessionTable.id == instance.usage_session_id
|
||||
)
|
||||
)
|
||||
else:
|
||||
result = await db.execute(
|
||||
select(AgentSpawnSessionTable)
|
||||
.where(
|
||||
AgentSpawnSessionTable.agent_slug == agent_id,
|
||||
AgentSpawnSessionTable.ended_at.is_(None),
|
||||
)
|
||||
.order_by(AgentSpawnSessionTable.started_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
session_row = result.scalar_one_or_none()
|
||||
if session_row is None:
|
||||
return False
|
||||
|
||||
db.add(
|
||||
TokenUsageSnapshotTable(
|
||||
id=uuid4(),
|
||||
agent_spawn_session_id=session_row.id,
|
||||
snapshotted_at=datetime.now(UTC),
|
||||
tokens_input=tokens_input,
|
||||
tokens_output=tokens_output,
|
||||
tokens_cache_read=tokens_cache_read,
|
||||
tokens_cache_write=tokens_cache_write,
|
||||
)
|
||||
)
|
||||
await db.execute(
|
||||
update(AgentSpawnSessionTable)
|
||||
.where(AgentSpawnSessionTable.id == session_row.id)
|
||||
.values(
|
||||
tokens_input=tokens_input,
|
||||
tokens_output=tokens_output,
|
||||
tokens_cache_read=tokens_cache_read,
|
||||
tokens_cache_write=tokens_cache_write,
|
||||
)
|
||||
)
|
||||
await db.commit()
|
||||
return True
|
||||
|
||||
async def _sweep_token_snapshots(self) -> None:
|
||||
"""Write a token_usage_snapshots row for each active agent with non-zero tokens.
|
||||
|
||||
@@ -3364,18 +3480,26 @@ class AgentOrchestrator:
|
||||
token counts on the open agent_spawn_sessions row so the DB reflects
|
||||
current progress without waiting for session close.
|
||||
Errors per-agent are caught so one bad agent doesn't abort the whole sweep.
|
||||
|
||||
Additionally publishes USAGE_UPDATE events per agent (throttled to at most
|
||||
one per 5-second window) and a USAGE_SNAPSHOT aggregate after the loop.
|
||||
"""
|
||||
if not self._instances:
|
||||
return
|
||||
|
||||
try:
|
||||
from roboco.db.base import get_session_factory
|
||||
from roboco.db.tables import AgentSpawnSessionTable, TokenUsageSnapshotTable
|
||||
except ImportError:
|
||||
return
|
||||
|
||||
session_factory = get_session_factory()
|
||||
|
||||
# Accumulators for the post-loop USAGE_SNAPSHOT event.
|
||||
_usage_by_agent: list[dict[str, Any]] = []
|
||||
_usage_total_input = 0
|
||||
_usage_total_output = 0
|
||||
_usage_total_cost = 0.0
|
||||
|
||||
async with httpx.AsyncClient(timeout=3.0) as client:
|
||||
for agent_id, instance in list(self._instances.items()):
|
||||
if instance.state not in (
|
||||
@@ -3384,80 +3508,60 @@ class AgentOrchestrator:
|
||||
):
|
||||
continue
|
||||
|
||||
sdk_url = f"http://roboco-agent-{agent_id}:{SDK_PORT}/usage/status"
|
||||
try:
|
||||
resp = await client.get(sdk_url)
|
||||
if resp.status_code != http_status.HTTP_200_OK:
|
||||
tokens = await self._resolve_active_tokens(client, agent_id)
|
||||
if tokens is None:
|
||||
continue
|
||||
data = resp.json()
|
||||
tokens_input = data.get("tokens_input", 0)
|
||||
tokens_output = data.get("tokens_output", 0)
|
||||
tokens_cache_read = data.get("tokens_cache_read", 0)
|
||||
tokens_cache_write = data.get("tokens_cache_write", 0)
|
||||
|
||||
# Skip agents with no token usage yet
|
||||
total = (
|
||||
tokens_input
|
||||
+ tokens_output
|
||||
+ tokens_cache_read
|
||||
+ tokens_cache_write
|
||||
persisted = await self._persist_token_snapshot(
|
||||
session_factory, agent_id, instance, tokens
|
||||
)
|
||||
if total == 0:
|
||||
if not persisted:
|
||||
continue
|
||||
|
||||
async with session_factory() as db:
|
||||
from sqlalchemy import select, update
|
||||
tokens_input, tokens_output = tokens[0], tokens[1]
|
||||
model = instance.config.model if instance.config else "unknown"
|
||||
|
||||
# Prefer a direct lookup by the session UUID captured at
|
||||
# spawn time; fall back to the agent_slug heuristic for
|
||||
# instances that pre-date the usage_session_id field.
|
||||
if instance.usage_session_id is not None:
|
||||
result = await db.execute(
|
||||
select(AgentSpawnSessionTable).where(
|
||||
AgentSpawnSessionTable.id
|
||||
== instance.usage_session_id
|
||||
)
|
||||
)
|
||||
else:
|
||||
result = await db.execute(
|
||||
select(AgentSpawnSessionTable)
|
||||
.where(
|
||||
AgentSpawnSessionTable.agent_slug == agent_id,
|
||||
AgentSpawnSessionTable.ended_at.is_(None),
|
||||
)
|
||||
.order_by(AgentSpawnSessionTable.started_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
session_row = result.scalar_one_or_none()
|
||||
if session_row is None:
|
||||
continue
|
||||
# Publish USAGE_UPDATE event for this agent (throttled).
|
||||
with contextlib.suppress(Exception):
|
||||
from roboco.events import get_event_bus
|
||||
from roboco.services.usage_events import (
|
||||
UsageUpdate,
|
||||
publish_usage_update,
|
||||
)
|
||||
|
||||
# Insert snapshot
|
||||
from uuid import uuid4 as _uuid4
|
||||
await publish_usage_update(
|
||||
get_event_bus(),
|
||||
UsageUpdate(
|
||||
agent_id=agent_id,
|
||||
task_id=instance.current_task_id,
|
||||
input_tokens=tokens_input,
|
||||
output_tokens=tokens_output,
|
||||
model=model,
|
||||
),
|
||||
)
|
||||
|
||||
snapshot = TokenUsageSnapshotTable(
|
||||
id=_uuid4(),
|
||||
agent_spawn_session_id=session_row.id,
|
||||
snapshotted_at=datetime.now(UTC),
|
||||
# Accumulate per-agent data for the aggregate snapshot.
|
||||
with contextlib.suppress(Exception):
|
||||
from roboco.billing.pricing import calculate_cost
|
||||
|
||||
agent_cost = calculate_cost(
|
||||
model=model,
|
||||
tokens_input=tokens_input,
|
||||
tokens_output=tokens_output,
|
||||
tokens_cache_read=tokens_cache_read,
|
||||
tokens_cache_write=tokens_cache_write,
|
||||
)
|
||||
db.add(snapshot)
|
||||
|
||||
# Update cumulative totals on the session row
|
||||
await db.execute(
|
||||
update(AgentSpawnSessionTable)
|
||||
.where(AgentSpawnSessionTable.id == session_row.id)
|
||||
.values(
|
||||
tokens_input=tokens_input,
|
||||
tokens_output=tokens_output,
|
||||
tokens_cache_read=tokens_cache_read,
|
||||
tokens_cache_write=tokens_cache_write,
|
||||
)
|
||||
_usage_by_agent.append(
|
||||
{
|
||||
"agent_id": agent_id,
|
||||
"input_tokens": tokens_input,
|
||||
"output_tokens": tokens_output,
|
||||
"model": model,
|
||||
"cost_estimate": agent_cost,
|
||||
}
|
||||
)
|
||||
await db.commit()
|
||||
_usage_total_input += tokens_input
|
||||
_usage_total_output += tokens_output
|
||||
_usage_total_cost += agent_cost
|
||||
|
||||
except Exception as agent_exc:
|
||||
logger.debug(
|
||||
@@ -3466,6 +3570,28 @@ class AgentOrchestrator:
|
||||
error=str(agent_exc),
|
||||
)
|
||||
|
||||
# Publish a USAGE_SNAPSHOT aggregate if any active agents had token data.
|
||||
if _usage_by_agent:
|
||||
with contextlib.suppress(Exception):
|
||||
from roboco.events import get_event_bus
|
||||
from roboco.services.usage_events import (
|
||||
UsageSnapshot,
|
||||
publish_usage_snapshot,
|
||||
)
|
||||
|
||||
await publish_usage_snapshot(
|
||||
get_event_bus(),
|
||||
UsageSnapshot(
|
||||
period="live",
|
||||
totals={
|
||||
"input_tokens": _usage_total_input,
|
||||
"output_tokens": _usage_total_output,
|
||||
},
|
||||
cost_estimate=_usage_total_cost,
|
||||
by_agent=_usage_by_agent,
|
||||
),
|
||||
)
|
||||
|
||||
async def _sweep_daily_rollup(self) -> None:
|
||||
"""Upsert daily_usage_rollups from closed agent_spawn_sessions.
|
||||
|
||||
@@ -3910,6 +4036,13 @@ Start by:
|
||||
container_id=cid,
|
||||
exit_code=exit_code,
|
||||
)
|
||||
# The agent self-exited (a graceful i_am_idle shutdown, or a crash), so
|
||||
# stop_agent() — which normally finalizes — was never called. Finalize
|
||||
# here to capture token usage from the transcript; otherwise the
|
||||
# spawn-session row is left open with zero tokens.
|
||||
await self._finalize_spawn_session(
|
||||
agent_id, exit_reason="completed" if graceful else "crashed"
|
||||
)
|
||||
instance.state = AgentState.OFFLINE
|
||||
instance.container_id = None
|
||||
if graceful:
|
||||
|
||||
@@ -35,6 +35,23 @@ def _row_tokens(row: Any) -> tuple[int, int, int, int]:
|
||||
)
|
||||
|
||||
|
||||
def _session_row(row: Any) -> dict[str, Any]:
|
||||
"""Shape one spawn-session row for the dashboard's sessions table."""
|
||||
tin, tout, tcr, tcw = _row_tokens(row)
|
||||
return {
|
||||
"id": str(row.id),
|
||||
"agent_slug": row.agent_slug,
|
||||
"model": row.model,
|
||||
"started_at": row.started_at.isoformat(),
|
||||
"ended_at": row.ended_at.isoformat() if row.ended_at else None,
|
||||
"tokens_input": tin,
|
||||
"tokens_output": tout,
|
||||
"tokens_cache": tcr + tcw,
|
||||
"total_tokens": tin + tout + tcr + tcw,
|
||||
"cost": float(row.estimated_cost_usd or 0.0),
|
||||
}
|
||||
|
||||
|
||||
def _parse_period(period: str) -> tuple[datetime, int]:
|
||||
"""Parse period string into (start_dt, hours).
|
||||
|
||||
@@ -441,6 +458,19 @@ class UsageService(BaseService):
|
||||
"cost_today_usd": round(float(row.total_cost_usd or 0.0), 6),
|
||||
}
|
||||
|
||||
async def get_recent_sessions(self, limit: int = 50) -> list[dict[str, Any]]:
|
||||
"""Return the most recent spawn sessions, newest first.
|
||||
|
||||
These are the raw per-session rows behind the aggregate panels — the
|
||||
dashboard's "Recent Sessions" table.
|
||||
"""
|
||||
result = await self.session.execute(
|
||||
select(AgentSpawnSessionTable)
|
||||
.order_by(AgentSpawnSessionTable.started_at.desc())
|
||||
.limit(limit)
|
||||
)
|
||||
return [_session_row(row) for row in result.scalars().all()]
|
||||
|
||||
|
||||
def get_usage_service(db: AsyncSession) -> UsageService:
|
||||
"""Factory function matching the pattern used by other services."""
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
"""
|
||||
Usage Event Publisher
|
||||
|
||||
Throttled helpers for publishing USAGE_UPDATE and USAGE_SNAPSHOT events
|
||||
to the StreamEventBus. Consumed by the orchestrator token sweep and
|
||||
forwarded to /ws/system WebSocket clients via the websocket_bridge.
|
||||
|
||||
Throttle window: one USAGE_UPDATE publish per agent per 5-second window.
|
||||
Subsequent calls within the window are silently dropped so a frequent
|
||||
sweep loop cannot flood the event bus or WebSocket clients.
|
||||
|
||||
USAGE_SNAPSHOT is always published (no per-agent throttle) — it is an
|
||||
aggregate and published at most once per sweep cycle.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from roboco.events.stream_bus import StreamEventBus
|
||||
|
||||
_THROTTLE_WINDOW_SECONDS: float = 5.0
|
||||
|
||||
|
||||
class _UsageThrottle:
|
||||
"""Per-agent last-publish timestamp tracker.
|
||||
|
||||
Uses ``time.monotonic()`` so clock adjustments (NTP, DST) don't cause
|
||||
spurious suppressions or double-fires.
|
||||
"""
|
||||
|
||||
def __init__(self, window: float = _THROTTLE_WINDOW_SECONDS) -> None:
|
||||
self._window = window
|
||||
self._last: dict[str, float] = {}
|
||||
|
||||
def should_publish(self, agent_id: str) -> bool:
|
||||
"""Return True if the agent is outside the throttle window.
|
||||
|
||||
Also records the current monotonic time as the new *last published*
|
||||
timestamp when it returns True, so the caller does not need to call a
|
||||
separate ``record()`` method.
|
||||
"""
|
||||
now = time.monotonic()
|
||||
if now - self._last.get(agent_id, 0.0) >= self._window:
|
||||
self._last[agent_id] = now
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
# Module-level singleton — shared across all callers in this process.
|
||||
_throttle = _UsageThrottle()
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class UsageUpdate:
|
||||
"""Per-agent cumulative token counts carried by a USAGE_UPDATE event.
|
||||
|
||||
Fields map directly onto the event payload the panel consumes (the
|
||||
backend emits these per active agent during the token sweep).
|
||||
"""
|
||||
|
||||
agent_id: str
|
||||
task_id: str | None
|
||||
input_tokens: int
|
||||
output_tokens: int
|
||||
model: str
|
||||
timestamp: datetime | None = None
|
||||
|
||||
def event_data(self) -> dict[str, Any]:
|
||||
"""Render the event payload, stamping ``timestamp`` if not supplied."""
|
||||
return {
|
||||
"agent_id": self.agent_id,
|
||||
"task_id": self.task_id,
|
||||
"input_tokens": self.input_tokens,
|
||||
"output_tokens": self.output_tokens,
|
||||
"model": self.model,
|
||||
"timestamp": (self.timestamp or datetime.now(UTC)).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class UsageSnapshot:
|
||||
"""Aggregate token/cost totals carried by a USAGE_SNAPSHOT event.
|
||||
|
||||
``totals`` is ``{"input_tokens": int, "output_tokens": int}``; ``by_agent``
|
||||
is the per-agent breakdown (each item carries ``agent_id``, token counts,
|
||||
``model`` and ``cost_estimate``).
|
||||
"""
|
||||
|
||||
period: str
|
||||
totals: dict[str, int]
|
||||
cost_estimate: float
|
||||
by_agent: list[dict[str, Any]]
|
||||
timestamp: datetime | None = None
|
||||
|
||||
def event_data(self) -> dict[str, Any]:
|
||||
"""Render the event payload, stamping ``timestamp`` if not supplied."""
|
||||
return {
|
||||
"period": self.period,
|
||||
"totals": self.totals,
|
||||
"cost_estimate": self.cost_estimate,
|
||||
"by_agent": self.by_agent,
|
||||
"timestamp": (self.timestamp or datetime.now(UTC)).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
async def publish_usage_update(bus: StreamEventBus, update: UsageUpdate) -> bool:
|
||||
"""Publish a USAGE_UPDATE event if the per-agent throttle window has elapsed.
|
||||
|
||||
Returns True if the event was published; False if suppressed by the throttle.
|
||||
"""
|
||||
if not _throttle.should_publish(update.agent_id):
|
||||
return False
|
||||
|
||||
from roboco.models.events import Event, EventType # lazy — avoids circular import
|
||||
|
||||
await bus.publish(Event(type=EventType.USAGE_UPDATE, data=update.event_data()))
|
||||
return True
|
||||
|
||||
|
||||
async def publish_usage_snapshot(bus: StreamEventBus, snapshot: UsageSnapshot) -> None:
|
||||
"""Publish a USAGE_SNAPSHOT aggregate event (no throttle)."""
|
||||
from roboco.models.events import Event, EventType # lazy — avoids circular import
|
||||
|
||||
await bus.publish(Event(type=EventType.USAGE_SNAPSHOT, data=snapshot.event_data()))
|
||||
Reference in New Issue
Block a user