mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
[499f9eb1] Token Usage & Cost Analytics — Full-Stack Instrumentation, Persistence, and Visualization (#90)
* [cd2bf666] feat(usage): add token usage types, API client, hooks, and UI components (#87) (#88) - Append 5 TypeScript interfaces to src/types/index.ts: TokenUsageSnapshot, AgentUsageRow, UsageSession, UsageTimePoint, ModelUsageSlice - Create src/lib/api/usage.ts: Axios singleton + isMockMode guards for getUsageSnapshot, getUsageTimeSeries, getAgentUsage, getUsageSessions, getModelUsage - Create src/hooks/use-usage.ts: usageKeys factory + useUsageSnapshot, useUsageTimeSeries, useAgentUsage, useUsageSessions, useModelUsage hooks - Create UsageOverviewPanel (dashboard/usage-overview-panel.tsx): 6 metric rows with Skeleton loading state; week-over-week trend arrow for cost - Update CommandCenter: Metrics+Alerts row expanded from 2-col to 3-col grid adding UsageOverviewPanel - Create src/components/metrics/ folder: UsageTimeSeriesChart (recharts stacked AreaChart with var(--chart-1/2/3)), ModelUsageDonut (PieChart), AgentUsageChart and TeamUsageChart (BarChart), SessionsTable (sortable columns + 10-row Prev/Next pagination) - Update Metrics page: Token Usage & Costs section with 5 rows (summary cards, time series+donut, agent+team bar charts, projection+cache efficiency, sessions table) - Add usage mini-bar to AgentCard: token count + cost + progress bar; AgentGrid and Agents page pass agentUsageMap through - Install recharts 3.8.1 - Export all new symbols through their barrel index.ts files Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev> * [10372f0f] Implement full token usage instrumentation: DB migration, SDK endpoints, orchestrator hooks, analytics API, WebSocket events, dashboard integration (#86) (#89) * [10372f0f] feat(token-usage): add Alembic migration 026 for token usage tables Create agent_spawn_sessions, token_usage_snapshots, and daily_usage_rollups tables with correct BIGINT columns, indexes, and unique constraint. Chain: 025_agentrole_prompter → 026_token_usage_tables. * [10372f0f] feat(token-usage): add ORM table classes for token usage instrumentation Add AgentSpawnSessionTable, TokenUsageSnapshotTable, DailyUsageRollupTable to db/tables.py. Import BigInteger and Date from SQLAlchemy. All columns match the migration schema with BIGINT token counts and proper indexes. * [10372f0f] feat(billing): add pricing module with calculate_cost() function Create roboco/billing/__init__.py and roboco/billing/pricing.py with calculate_cost() supporting Claude opus/sonnet/haiku models with input/output/cache pricing. Unknown models return 0.0 without raising. * [10372f0f] feat(sdk): add POST /usage/report and GET /usage/status endpoints to agent SDK Extend _SessionState with token counters. Add TokenReportRequest and TokenUsageStatus models. POST /usage/report additively accumulates token counts; GET /usage/status returns current session totals for sweeper polling. * [10372f0f] feat(orchestrator): add token usage instrumentation hooks - _launch_spawn() calls _record_spawn_session() after successful container spawn - stop_agent() calls _finalize_spawn_session() before container removal - _run_sweep() calls _sweep_token_snapshots() and _sweep_daily_rollup() each tick - New methods: _record_spawn_session, _finalize_spawn_session, _sweep_token_snapshots, _sweep_daily_rollup in TOKEN USAGE section * [10372f0f] feat(api): add token usage analytics API with 7 endpoints Create roboco/services/usage.py (UsageService) and roboco/api/routes/usage.py. Endpoints: GET /api/usage/summary, /time-series, /by-agent, /by-team, /by-model, /projection, /cache-efficiency. Register in app.py. * [10372f0f] feat(dashboard): add usage_summary field to CEO dashboard Add UsageSummary schema (tokens_today, cost_today_usd) to dashboard schemas. Add usage_summary: UsageSummary | None to CEOOverview. Update get_ceo_overview() to populate usage_summary from daily_usage_rollups. * [10372f0f] fix(billing/tests): remove dead except block in _sweep_daily_rollup, add unit tests for pricing.py and services/usage.py - Remove unreachable `except Exception as e` block in orchestrator.py _sweep_daily_rollup() (lines 3376-3381) which referenced undefined `agent_id` and was copy-pasted from _sweep_token_snapshots by mistake - Add tests/unit/billing/test_pricing.py: 31 tests covering opus/sonnet/ haiku tiers with all 4 token types, unknown model → 0.0, empty string → 0.0, and substring-match priority (longer fragment wins) - Add tests/unit/services/test_usage.py: 25 tests covering get_summary trend_pct edge cases (prev=0, both=0, prev>0), get_by_agent/team/model pct_of_total summing to 100%, get_projection formula (avg_daily×30), and get_cache_efficiency hit-rate and cost_saved arithmetic - pricing.py: 100% coverage; services/usage.py: 83% coverage (>80% target) * [10372f0f] fix(usage): include cache tokens in time-series total_tokens to fix AC9 consistency violation get_time_series() previously computed total_tokens as tokens_input + tokens_output only. get_summary() includes all 4 token types (input + output + cache_read + cache_write). AC9 requires both endpoints to agree on their totals for the same period. Fix: add tokens_cache_read and tokens_cache_write to the SELECT query in get_time_series() and include them in the total_tokens calculation. Also adds 4 new unit tests in TestGetTimeSeries covering: - total_tokens includes cache_read and cache_write (the AC9 guard) - zero cache tokens still produces correct total - empty result returns empty list - required fields are present in each point * [10372f0f] fix(usage): remove unused imports and include cache tokens in breakdown totals (AC10) - Remove import math (F401 — never used) - Remove text from sqlalchemy import (F401 — never used) - Remove unused local calculate_cost import inside get_cache_efficiency (F401) - Add tokens_cache_read and tokens_cache_write to SELECT in get_by_agent, get_by_team, and get_by_model; update grand_total and per-item total to include all 4 token types so totals match get_summary() (AC10 fix) - Update test mock rows to include explicit tokens_cache_read=0 and tokens_cache_write=0 so they work with the fixed code - Add new test cases: test_cache_tokens_included_in_total_tokens and test_pct_of_total_sums_to_100_with_cache_tokens for each breakdown class --------- Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> * [44b9eb1f] feat(usage): align frontend API client, TS types, and chart components to real backend contract (#92) (#94) Update all usage-related frontend code to match the actual FastAPI backend response shapes and endpoint paths: - panel/src/lib/api/usage.ts: rewrite all 7 API functions to use correct endpoint paths (/usage/summary, /usage/by-agent, /usage/by-model, /usage/by-team, /usage/time-series, /usage/projection, /usage/cache-efficiency); send period query param (24h/7d/30d not hours); mock generators produce data matching real backend shapes exactly; getUsageSessions returns [] in prod (no /usage/sessions endpoint exists) - panel/src/types/index.ts: replace TokenUsageSnapshot with UsageSummary (tokens_input/tokens_output/total_cost_usd/trend_pct); update AgentUsageRow to use agent_slug/total_tokens/cost_usd/pct_of_total; add TeamUsageRow, UsageProjection, CacheEfficiencyResponse; update UsageTimePoint to use bucket field; update UsageSession to use agent_slug - panel/src/hooks/use-usage.ts: rewrite all hooks to match new API and types; add useTeamUsage, useUsageProjection, useCacheEfficiency hooks - panel/src/components/metrics/usage-time-series-chart.tsx: use bucket field (not timestamp) for axis labels - panel/src/components/metrics/agent-usage-chart.tsx: use agent_slug and total_tokens (not agent_name/tokens_today) - panel/src/components/metrics/team-usage-chart.tsx: rewrite to accept TeamUsageRow[] from API directly - panel/src/components/metrics/model-usage-donut.tsx: use total_tokens, cost_usd, pct_of_total (not tokens/cost/percentage) - panel/src/components/metrics/sessions-table.tsx: use agent_slug, sort keys updated - panel/src/components/dashboard/usage-overview-panel.tsx: use useUsageSummary with tokens_input/tokens_output/total_cost_usd/trend_pct - panel/src/app/(dashboard)/metrics/page.tsx: wire all new hooks, add TeamUsageChart, ProjectionCard, CacheEfficiencyCard with correct types - panel/src/app/(dashboard)/agents/page.tsx: key agentUsageMap by agent_slug - panel/src/components/agents/agent-card.tsx: use total_tokens and cost_usd Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev> * [2161b832] fix: SDK_PORT constant, stop_agent lock refactor, usage_session_id binding, rollup 7-day window (#93) (#95) - Add SDK_PORT = 9000 module-level constant to orchestrator.py; replace hardcoded 9000 in _sweep_budget_exceeded URL with SDK_PORT - Add UUID to TYPE_CHECKING imports to satisfy ruff F821 - Refactor stop_agent: call _finalize_spawn_session BEFORE acquiring self._lock so the SDK HTTP round-trip does not hold the lock - Add usage_session_id: UUID | None field to AgentInstance dataclass - Change _record_spawn_session to return UUID | None; wire return value back to instance.usage_session_id in _launch_spawn - Update _finalize_spawn_session to use WHERE id=usage_session_id for direct session row lookup when usage_session_id is not None - Add started_at >= (now_utc - 7 days) filter to _sweep_daily_rollup aggregate query to avoid re-aggregating all-time history each sweep Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> * [2e0759e1] fix: pricing accuracy, import ordering, session-id binding, rollup cleanup, write-hook tests (#97) (#98) - pricing.py: correct claude-opus-4 prices (5/25/0.50/6.25 not 15/75/1.5/3.75) and haiku family prices (1/5/0.10/1.25 not 0.8/4/0.08/0.20); add Ollama zero-cost early-return; add structlog warning for unmatched model names - app.py: move usage_router import before routes.v1 block (ruff isort fix) - orchestrator.py _sweep_daily_rollup: remove unused calculate_cost import; add blank line between stdlib (uuid4) and third-party (sqlalchemy) imports - orchestrator.py _sweep_token_snapshots: prefer direct lookup by instance.usage_session_id; fall back to agent_slug heuristic only when None - tests: add test_sweep_daily_rollup_inserts_new_row and test_stop_agent_finalizes_before_lock to test_orchestrator_write_hooks.py - usage.py, routes/usage.py, stream_bus.py, test files: ruff format/lint fixes Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> * Mypy compliance * fix(migrations,tests): linearize forked migration chain + correct ceo_reject coordination-root expectation The master merge brought in 026_completed_dependency_ids alongside the rework's 026_token_usage_tables — both off 025, forking the alembic head and breaking the enum-parity test. Rebase token-usage onto 026_completed_dependency_ids (linear chain, single head). Also: test_ceo_reject_routes_coordination_task_to_main_pm asserted the old NEEDS_REVISION behavior; the lifecycle fix correctly routes a coordination root to PENDING (Main PM's claim source). Update the assertion. --------- Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev> Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
co-authored by
Frontend Developer 1
Backend Developer 1
Renn F
Frontend Developer 1
Backend Developer 1
Renn F
parent
93c6ef8a57
commit
b3057628b0
@@ -0,0 +1,170 @@
|
||||
"""026_token_usage_tables
|
||||
|
||||
Create token usage instrumentation tables:
|
||||
- agent_spawn_sessions: tracks each agent container spawn with token totals
|
||||
- token_usage_snapshots: periodic snapshots of token usage per session
|
||||
- daily_usage_rollups: aggregated daily usage per agent/team/model
|
||||
|
||||
Revision ID: 026_token_usage_tables
|
||||
Revises: 025_agentrole_prompter
|
||||
Create Date: 2026-06-09
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects.postgresql import UUID
|
||||
|
||||
revision = "026_token_usage_tables"
|
||||
# Rebased onto 026_completed_dependency_ids so the chain stays linear: the
|
||||
# master merge brought in a second migration off 025_agentrole_prompter, which
|
||||
# forked the head. Chain is now 025 -> 026_completed_dependency_ids -> this.
|
||||
down_revision = "026_completed_dependency_ids"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# ------------------------------------------------------------------
|
||||
# agent_spawn_sessions
|
||||
# One row per container spawn. Opened on spawn, closed on stop.
|
||||
# ------------------------------------------------------------------
|
||||
op.create_table(
|
||||
"agent_spawn_sessions",
|
||||
sa.Column("id", UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("agent_slug", sa.String(100), nullable=False),
|
||||
sa.Column("team", sa.String(50), nullable=False),
|
||||
sa.Column("role", sa.String(50), nullable=False),
|
||||
sa.Column("model", sa.String(100), nullable=False),
|
||||
sa.Column("task_id", sa.String(36), nullable=True),
|
||||
sa.Column(
|
||||
"started_at",
|
||||
sa.DateTime(timezone=True),
|
||||
nullable=False,
|
||||
server_default=sa.func.now(),
|
||||
),
|
||||
sa.Column("ended_at", sa.DateTime(timezone=True), nullable=True),
|
||||
# BIGINT for token counts — they can exceed INT32 for long sessions
|
||||
sa.Column("tokens_input", sa.BigInteger, nullable=False, server_default="0"),
|
||||
sa.Column("tokens_output", sa.BigInteger, nullable=False, server_default="0"),
|
||||
sa.Column(
|
||||
"tokens_cache_read", sa.BigInteger, nullable=False, server_default="0"
|
||||
),
|
||||
sa.Column(
|
||||
"tokens_cache_write", sa.BigInteger, nullable=False, server_default="0"
|
||||
),
|
||||
sa.Column("exit_reason", sa.String(100), nullable=True),
|
||||
sa.Column("estimated_cost_usd", sa.Float, nullable=True),
|
||||
)
|
||||
|
||||
# Indexes for common query patterns
|
||||
op.create_index(
|
||||
"ix_agent_spawn_sessions_agent_slug",
|
||||
"agent_spawn_sessions",
|
||||
["agent_slug"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_agent_spawn_sessions_started_at",
|
||||
"agent_spawn_sessions",
|
||||
["started_at"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_agent_spawn_sessions_ended_at",
|
||||
"agent_spawn_sessions",
|
||||
["ended_at"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_agent_spawn_sessions_team",
|
||||
"agent_spawn_sessions",
|
||||
["team"],
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# token_usage_snapshots
|
||||
# Periodic snapshots (every ~60s) of token counts for active sessions.
|
||||
# ------------------------------------------------------------------
|
||||
op.create_table(
|
||||
"token_usage_snapshots",
|
||||
sa.Column("id", UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column(
|
||||
"agent_spawn_session_id",
|
||||
UUID(as_uuid=True),
|
||||
sa.ForeignKey(
|
||||
"agent_spawn_sessions.id", ondelete="CASCADE", name="fk_snapshot_session"
|
||||
),
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column(
|
||||
"snapshotted_at",
|
||||
sa.DateTime(timezone=True),
|
||||
nullable=False,
|
||||
server_default=sa.func.now(),
|
||||
),
|
||||
sa.Column("tokens_input", sa.BigInteger, nullable=False, server_default="0"),
|
||||
sa.Column("tokens_output", sa.BigInteger, nullable=False, server_default="0"),
|
||||
sa.Column(
|
||||
"tokens_cache_read", sa.BigInteger, nullable=False, server_default="0"
|
||||
),
|
||||
sa.Column(
|
||||
"tokens_cache_write", sa.BigInteger, nullable=False, server_default="0"
|
||||
),
|
||||
)
|
||||
|
||||
op.create_index(
|
||||
"ix_token_usage_snapshots_session_id",
|
||||
"token_usage_snapshots",
|
||||
["agent_spawn_session_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_token_usage_snapshots_snapshotted_at",
|
||||
"token_usage_snapshots",
|
||||
["snapshotted_at"],
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# daily_usage_rollups
|
||||
# Pre-aggregated daily totals per (date, agent_slug, team, model).
|
||||
# Populated by the sweeper; upserted on each sweep so re-runs are safe.
|
||||
# ------------------------------------------------------------------
|
||||
op.create_table(
|
||||
"daily_usage_rollups",
|
||||
sa.Column("id", UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("date", sa.Date, nullable=False),
|
||||
sa.Column("agent_slug", sa.String(100), nullable=False),
|
||||
sa.Column("team", sa.String(50), nullable=False),
|
||||
sa.Column("model", sa.String(100), nullable=False),
|
||||
sa.Column("tokens_input", sa.BigInteger, nullable=False, server_default="0"),
|
||||
sa.Column("tokens_output", sa.BigInteger, nullable=False, server_default="0"),
|
||||
sa.Column(
|
||||
"tokens_cache_read", sa.BigInteger, nullable=False, server_default="0"
|
||||
),
|
||||
sa.Column(
|
||||
"tokens_cache_write", sa.BigInteger, nullable=False, server_default="0"
|
||||
),
|
||||
sa.Column("total_cost_usd", sa.Float, nullable=False, server_default="0"),
|
||||
sa.Column("session_count", sa.Integer, nullable=False, server_default="0"),
|
||||
)
|
||||
|
||||
# Unique constraint enables ON CONFLICT upsert in the sweeper
|
||||
op.create_unique_constraint(
|
||||
"uq_daily_rollup_date_agent_team_model",
|
||||
"daily_usage_rollups",
|
||||
["date", "agent_slug", "team", "model"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_daily_rollups_date",
|
||||
"daily_usage_rollups",
|
||||
["date"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_daily_rollups_agent_slug",
|
||||
"daily_usage_rollups",
|
||||
["agent_slug"],
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("daily_usage_rollups")
|
||||
op.drop_table("token_usage_snapshots")
|
||||
op.drop_table("agent_spawn_sessions")
|
||||
Reference in New Issue
Block a user