[499f9eb1] Token Usage & Cost Analytics — Full-Stack Instrumentation, Persistence, and Visualization (#90)

* [cd2bf666] feat(usage): add token usage types, API client, hooks, and UI components (#87) (#88)

- Append 5 TypeScript interfaces to src/types/index.ts: TokenUsageSnapshot, AgentUsageRow, UsageSession, UsageTimePoint, ModelUsageSlice
- Create src/lib/api/usage.ts: Axios singleton + isMockMode guards for getUsageSnapshot, getUsageTimeSeries, getAgentUsage, getUsageSessions, getModelUsage
- Create src/hooks/use-usage.ts: usageKeys factory + useUsageSnapshot, useUsageTimeSeries, useAgentUsage, useUsageSessions, useModelUsage hooks
- Create UsageOverviewPanel (dashboard/usage-overview-panel.tsx): 6 metric rows with Skeleton loading state; week-over-week trend arrow for cost
- Update CommandCenter: Metrics+Alerts row expanded from 2-col to 3-col grid adding UsageOverviewPanel
- Create src/components/metrics/ folder: UsageTimeSeriesChart (recharts stacked AreaChart with var(--chart-1/2/3)), ModelUsageDonut (PieChart), AgentUsageChart and TeamUsageChart (BarChart), SessionsTable (sortable columns + 10-row Prev/Next pagination)
- Update Metrics page: Token Usage & Costs section with 5 rows (summary cards, time series+donut, agent+team bar charts, projection+cache efficiency, sessions table)
- Add usage mini-bar to AgentCard: token count + cost + progress bar; AgentGrid and Agents page pass agentUsageMap through
- Install recharts 3.8.1
- Export all new symbols through their barrel index.ts files

Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev>

* [10372f0f] Implement full token usage instrumentation: DB migration, SDK endpoints, orchestrator hooks, analytics API, WebSocket events, dashboard integration (#86) (#89)

* [10372f0f] feat(token-usage): add Alembic migration 026 for token usage tables

Create agent_spawn_sessions, token_usage_snapshots, and daily_usage_rollups
tables with correct BIGINT columns, indexes, and unique constraint.
Chain: 025_agentrole_prompter → 026_token_usage_tables.

* [10372f0f] feat(token-usage): add ORM table classes for token usage instrumentation

Add AgentSpawnSessionTable, TokenUsageSnapshotTable, DailyUsageRollupTable
to db/tables.py. Import BigInteger and Date from SQLAlchemy. All columns
match the migration schema with BIGINT token counts and proper indexes.

* [10372f0f] feat(billing): add pricing module with calculate_cost() function

Create roboco/billing/__init__.py and roboco/billing/pricing.py with
calculate_cost() supporting Claude opus/sonnet/haiku models with
input/output/cache pricing. Unknown models return 0.0 without raising.

* [10372f0f] feat(sdk): add POST /usage/report and GET /usage/status endpoints to agent SDK

Extend _SessionState with token counters. Add TokenReportRequest and
TokenUsageStatus models. POST /usage/report additively accumulates token
counts; GET /usage/status returns current session totals for sweeper polling.

* [10372f0f] feat(orchestrator): add token usage instrumentation hooks

- _launch_spawn() calls _record_spawn_session() after successful container spawn
- stop_agent() calls _finalize_spawn_session() before container removal
- _run_sweep() calls _sweep_token_snapshots() and _sweep_daily_rollup() each tick
- New methods: _record_spawn_session, _finalize_spawn_session,
  _sweep_token_snapshots, _sweep_daily_rollup in TOKEN USAGE section

* [10372f0f] feat(api): add token usage analytics API with 7 endpoints

Create roboco/services/usage.py (UsageService) and roboco/api/routes/usage.py.
Endpoints: GET /api/usage/summary, /time-series, /by-agent, /by-team,
/by-model, /projection, /cache-efficiency. Register in app.py.

* [10372f0f] feat(dashboard): add usage_summary field to CEO dashboard

Add UsageSummary schema (tokens_today, cost_today_usd) to dashboard schemas.
Add usage_summary: UsageSummary | None to CEOOverview. Update
get_ceo_overview() to populate usage_summary from daily_usage_rollups.

* [10372f0f] fix(billing/tests): remove dead except block in _sweep_daily_rollup, add unit tests for pricing.py and services/usage.py

- Remove unreachable `except Exception as e` block in orchestrator.py
  _sweep_daily_rollup() (lines 3376-3381) which referenced undefined
  `agent_id` and was copy-pasted from _sweep_token_snapshots by mistake
- Add tests/unit/billing/test_pricing.py: 31 tests covering opus/sonnet/
  haiku tiers with all 4 token types, unknown model → 0.0, empty string
  → 0.0, and substring-match priority (longer fragment wins)
- Add tests/unit/services/test_usage.py: 25 tests covering get_summary
  trend_pct edge cases (prev=0, both=0, prev>0), get_by_agent/team/model
  pct_of_total summing to 100%, get_projection formula (avg_daily×30),
  and get_cache_efficiency hit-rate and cost_saved arithmetic
- pricing.py: 100% coverage; services/usage.py: 83% coverage (>80% target)

* [10372f0f] fix(usage): include cache tokens in time-series total_tokens to fix AC9 consistency violation

get_time_series() previously computed total_tokens as tokens_input +
tokens_output only. get_summary() includes all 4 token types (input +
output + cache_read + cache_write). AC9 requires both endpoints to agree
on their totals for the same period.

Fix: add tokens_cache_read and tokens_cache_write to the SELECT query in
get_time_series() and include them in the total_tokens calculation.

Also adds 4 new unit tests in TestGetTimeSeries covering:
- total_tokens includes cache_read and cache_write (the AC9 guard)
- zero cache tokens still produces correct total
- empty result returns empty list
- required fields are present in each point

* [10372f0f] fix(usage): remove unused imports and include cache tokens in breakdown totals (AC10)

- Remove import math (F401 — never used)
- Remove text from sqlalchemy import (F401 — never used)
- Remove unused local calculate_cost import inside get_cache_efficiency (F401)
- Add tokens_cache_read and tokens_cache_write to SELECT in get_by_agent,
  get_by_team, and get_by_model; update grand_total and per-item total to
  include all 4 token types so totals match get_summary() (AC10 fix)
- Update test mock rows to include explicit tokens_cache_read=0 and
  tokens_cache_write=0 so they work with the fixed code
- Add new test cases: test_cache_tokens_included_in_total_tokens and
  test_pct_of_total_sums_to_100_with_cache_tokens for each breakdown class

---------

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

* [44b9eb1f] feat(usage): align frontend API client, TS types, and chart components to real backend contract (#92) (#94)

Update all usage-related frontend code to match the actual FastAPI backend
response shapes and endpoint paths:

- panel/src/lib/api/usage.ts: rewrite all 7 API functions to use correct
  endpoint paths (/usage/summary, /usage/by-agent, /usage/by-model,
  /usage/by-team, /usage/time-series, /usage/projection,
  /usage/cache-efficiency); send period query param (24h/7d/30d not hours);
  mock generators produce data matching real backend shapes exactly;
  getUsageSessions returns [] in prod (no /usage/sessions endpoint exists)

- panel/src/types/index.ts: replace TokenUsageSnapshot with UsageSummary
  (tokens_input/tokens_output/total_cost_usd/trend_pct); update AgentUsageRow
  to use agent_slug/total_tokens/cost_usd/pct_of_total; add TeamUsageRow,
  UsageProjection, CacheEfficiencyResponse; update UsageTimePoint to use
  bucket field; update UsageSession to use agent_slug

- panel/src/hooks/use-usage.ts: rewrite all hooks to match new API and types;
  add useTeamUsage, useUsageProjection, useCacheEfficiency hooks

- panel/src/components/metrics/usage-time-series-chart.tsx: use bucket field
  (not timestamp) for axis labels
- panel/src/components/metrics/agent-usage-chart.tsx: use agent_slug and
  total_tokens (not agent_name/tokens_today)
- panel/src/components/metrics/team-usage-chart.tsx: rewrite to accept
  TeamUsageRow[] from API directly
- panel/src/components/metrics/model-usage-donut.tsx: use total_tokens,
  cost_usd, pct_of_total (not tokens/cost/percentage)
- panel/src/components/metrics/sessions-table.tsx: use agent_slug, sort keys
  updated
- panel/src/components/dashboard/usage-overview-panel.tsx: use useUsageSummary
  with tokens_input/tokens_output/total_cost_usd/trend_pct
- panel/src/app/(dashboard)/metrics/page.tsx: wire all new hooks, add
  TeamUsageChart, ProjectionCard, CacheEfficiencyCard with correct types
- panel/src/app/(dashboard)/agents/page.tsx: key agentUsageMap by agent_slug
- panel/src/components/agents/agent-card.tsx: use total_tokens and cost_usd

Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev>

* [2161b832] fix: SDK_PORT constant, stop_agent lock refactor, usage_session_id binding, rollup 7-day window (#93) (#95)

- Add SDK_PORT = 9000 module-level constant to orchestrator.py; replace
  hardcoded 9000 in _sweep_budget_exceeded URL with SDK_PORT
- Add UUID to TYPE_CHECKING imports to satisfy ruff F821
- Refactor stop_agent: call _finalize_spawn_session BEFORE acquiring
  self._lock so the SDK HTTP round-trip does not hold the lock
- Add usage_session_id: UUID | None field to AgentInstance dataclass
- Change _record_spawn_session to return UUID | None; wire return value
  back to instance.usage_session_id in _launch_spawn
- Update _finalize_spawn_session to use WHERE id=usage_session_id for
  direct session row lookup when usage_session_id is not None
- Add started_at >= (now_utc - 7 days) filter to _sweep_daily_rollup
  aggregate query to avoid re-aggregating all-time history each sweep

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

* [2e0759e1] fix: pricing accuracy, import ordering, session-id binding, rollup cleanup, write-hook tests (#97) (#98)

- pricing.py: correct claude-opus-4 prices (5/25/0.50/6.25 not 15/75/1.5/3.75)
  and haiku family prices (1/5/0.10/1.25 not 0.8/4/0.08/0.20); add Ollama
  zero-cost early-return; add structlog warning for unmatched model names
- app.py: move usage_router import before routes.v1 block (ruff isort fix)
- orchestrator.py _sweep_daily_rollup: remove unused calculate_cost import;
  add blank line between stdlib (uuid4) and third-party (sqlalchemy) imports
- orchestrator.py _sweep_token_snapshots: prefer direct lookup by
  instance.usage_session_id; fall back to agent_slug heuristic only when None
- tests: add test_sweep_daily_rollup_inserts_new_row and
  test_stop_agent_finalizes_before_lock to test_orchestrator_write_hooks.py
- usage.py, routes/usage.py, stream_bus.py, test files: ruff format/lint fixes

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

* Mypy compliance

* fix(migrations,tests): linearize forked migration chain + correct ceo_reject coordination-root expectation

The master merge brought in 026_completed_dependency_ids alongside the rework's
026_token_usage_tables — both off 025, forking the alembic head and breaking
the enum-parity test. Rebase token-usage onto 026_completed_dependency_ids
(linear chain, single head).

Also: test_ceo_reject_routes_coordination_task_to_main_pm asserted the old
NEEDS_REVISION behavior; the lifecycle fix correctly routes a coordination root
to PENDING (Main PM's claim source). Update the assertion.

---------

Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev>
Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
2026-06-10 14:38:44 +02:00
committed by GitHub
co-authored by Frontend Developer 1 Backend Developer 1 Renn F
parent 93c6ef8a57
commit b3057628b0
40 changed files with 5039 additions and 19 deletions
+17 -1
View File
@@ -6,7 +6,8 @@ import {
useWaitingAgents,
useAgentDefinitions,
} from "@/hooks/use-agents";
import { AgentStatusResponse } from "@/types";
import { useAgentUsage } from "@/hooks/use-usage";
import { AgentStatusResponse, AgentUsageRow } from "@/types";
import { Button } from "@/components/ui/button";
import { RefreshCw } from "lucide-react";
import { OfflineState } from "@/components/ui/offline-state";
@@ -27,6 +28,7 @@ export default function AgentsPage() {
const { data: agents = [], isLoading: agentsLoading } = useAgentDefinitions();
const { data: status, isLoading, error, refetch } = useOrchestratorStatus();
const { data: waitingAgents } = useWaitingAgents();
const { data: usageRows } = useAgentUsage();
// Check if it's a connection error (backend not running)
const isOffline = error && (
@@ -46,6 +48,15 @@ export default function AgentsPage() {
return result;
}, [status]);
// Convert usage rows to a record keyed by agent_slug
const agentUsageMap = useMemo(() => {
const result: Record<string, AgentUsageRow> = {};
for (const row of usageRows ?? []) {
result[row.agent_slug] = row;
}
return result;
}, [usageRows]);
return (
<div className="space-y-6">
{/* Header */}
@@ -83,6 +94,7 @@ export default function AgentsPage() {
title="Board"
agents={getBoardAgents(agents)}
agentStatuses={agentStatuses}
agentUsage={agentUsageMap}
isLoading={(isLoading || agentsLoading) && !isOffline}
columns={4}
/>
@@ -91,6 +103,7 @@ export default function AgentsPage() {
title="Main PM"
agents={getMainPm(agents)}
agentStatuses={agentStatuses}
agentUsage={agentUsageMap}
isLoading={(isLoading || agentsLoading) && !isOffline}
columns={4}
/>
@@ -99,6 +112,7 @@ export default function AgentsPage() {
title="Backend Cell"
agents={getBackendAgents(agents)}
agentStatuses={agentStatuses}
agentUsage={agentUsageMap}
isLoading={(isLoading || agentsLoading) && !isOffline}
columns={5}
/>
@@ -107,6 +121,7 @@ export default function AgentsPage() {
title="Frontend Cell"
agents={getFrontendAgents(agents)}
agentStatuses={agentStatuses}
agentUsage={agentUsageMap}
isLoading={(isLoading || agentsLoading) && !isOffline}
columns={5}
/>
@@ -115,6 +130,7 @@ export default function AgentsPage() {
title="UX/UI Cell"
agents={getUxAgents(agents)}
agentStatuses={agentStatuses}
agentUsage={agentUsageMap}
isLoading={(isLoading || agentsLoading) && !isOffline}
columns={4}
/>
+231
View File
@@ -2,14 +2,33 @@
import { useOrchestratorStatus } from "@/hooks/use-agents";
import { useTasks } from "@/hooks/use-tasks";
import {
useUsageSummary,
useUsageTimeSeries,
useAgentUsage,
useTeamUsage,
useModelUsage,
useUsageProjection,
useCacheEfficiency,
useUsageSessions,
} from "@/hooks/use-usage";
import { TaskStatus, Team } from "@/types";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Button } from "@/components/ui/button";
import { Progress } from "@/components/ui/progress";
import { OfflineState } from "@/components/ui/offline-state";
import { Skeleton } from "@/components/ui/skeleton";
import {
UsageTimeSeriesChart,
ModelUsageDonut,
AgentUsageChart,
TeamUsageChart,
SessionsTable,
} from "@/components/metrics";
import {
Activity,
TrendingUp,
TrendingDown,
Clock,
AlertTriangle,
Users,
@@ -18,6 +37,8 @@ import {
RefreshCw,
Zap,
Timer,
Coins,
Sparkles,
} from "lucide-react";
interface MetricCardProps {
@@ -295,8 +316,218 @@ export default function MetricsPage() {
))}
</div>
</div>
{/* ─── Token Usage & Costs ─────────────────────────────────── */}
<TokenUsageCostsSection />
</>
)}
</div>
);
}
// =============================================================================
// TOKEN USAGE & COSTS SECTION
// =============================================================================
function TokenUsageCostsSection() {
const { data: summary, isLoading: loadingSnap } = useUsageSummary("24h");
const { data: timeSeries, isLoading: loadingTS } = useUsageTimeSeries("24h");
const { data: agentUsage, isLoading: loadingAgents } = useAgentUsage("24h");
const { data: teamUsage, isLoading: loadingTeams } = useTeamUsage("24h");
const { data: sessions, isLoading: loadingSessions } = useUsageSessions(100);
const { data: modelUsage, isLoading: loadingModels } = useModelUsage("24h");
const { data: projection, isLoading: loadingProj } = useUsageProjection();
const { data: cacheStats, isLoading: loadingCache } = useCacheEfficiency("24h");
const trendUp = (summary?.trend_pct ?? 0) >= 0;
return (
<div className="space-y-6">
<h2 className="text-lg font-semibold">Token Usage &amp; Costs</h2>
{/* Row 1 — Summary cards */}
<div className="grid gap-4 sm:grid-cols-2 lg:grid-cols-3 xl:grid-cols-6">
<SummaryCard
title="Tokens Input"
value={summary ? fmtTokens(summary.tokens_input) : undefined}
icon={<Zap className="h-4 w-4 text-yellow-500" />}
isLoading={loadingSnap}
/>
<SummaryCard
title="Tokens Output"
value={summary ? fmtTokens(summary.tokens_output) : undefined}
icon={<Zap className="h-4 w-4 text-blue-500" />}
isLoading={loadingSnap}
/>
<SummaryCard
title="Total Cost (24h)"
value={summary ? "$" + summary.total_cost_usd.toFixed(4) : undefined}
icon={<Coins className="h-4 w-4 text-green-500" />}
isLoading={loadingSnap}
/>
<SummaryCard
title="Trend vs Prior"
value={summary ? (trendUp ? "+" : "") + summary.trend_pct.toFixed(1) + "%" : undefined}
icon={
trendUp ? (
<TrendingUp className="h-4 w-4 text-red-500" />
) : (
<TrendingDown className="h-4 w-4 text-green-500" />
)
}
isLoading={loadingSnap}
/>
<SummaryCard
title="Total Tokens"
value={summary ? fmtTokens(summary.total_tokens) : undefined}
icon={<Activity className="h-4 w-4 text-blue-500" />}
isLoading={loadingSnap}
/>
<SummaryCard
title="Cache Saved"
value={cacheStats ? "$" + cacheStats.cost_saved_by_cache_usd.toFixed(4) : undefined}
icon={<Sparkles className="h-4 w-4 text-purple-500" />}
isLoading={loadingCache}
/>
</div>
{/* Row 2 — Time series + model donut */}
<div className="grid gap-4 lg:grid-cols-3">
<div className="lg:col-span-2">
<UsageTimeSeriesChart data={timeSeries} isLoading={loadingTS} />
</div>
<ModelUsageDonut data={modelUsage} isLoading={loadingModels} />
</div>
{/* Row 3 — Agent bar + team bar */}
<div className="grid gap-4 lg:grid-cols-2">
<AgentUsageChart data={agentUsage} isLoading={loadingAgents} />
<TeamUsageChart data={teamUsage} isLoading={loadingTeams} />
</div>
{/* Row 4 — Projection + cache efficiency */}
<div className="grid gap-4 sm:grid-cols-2">
<ProjectionCard projection={projection} isLoading={loadingProj} />
<CacheEfficiencyCard cacheStats={cacheStats} isLoading={loadingCache} />
</div>
{/* Row 5 — Sessions table (mock-mode only; empty in production) */}
<SessionsTable data={sessions} isLoading={loadingSessions} />
</div>
);
}
// ─── Helper sub-components ────────────────────────────────────────────────────
function fmtTokens(n: number): string {
if (n >= 1_000_000) return (n / 1_000_000).toFixed(2) + "M";
if (n >= 1_000) return (n / 1_000).toFixed(1) + "K";
return String(n);
}
interface SummaryCardProps {
title: string;
value: string | undefined;
icon: React.ReactNode;
trend?: { dir: "up" | "down"; label: string };
isLoading: boolean;
}
function SummaryCard({ title, value, icon, trend, isLoading }: SummaryCardProps) {
return (
<Card>
<CardHeader className="flex flex-row items-center justify-between pb-2">
<CardTitle className="text-sm font-medium text-muted-foreground">{title}</CardTitle>
{icon}
</CardHeader>
<CardContent>
{isLoading ? (
<Skeleton className="h-7 w-24" />
) : (
<>
<div className="text-2xl font-bold">{value ?? "—"}</div>
{trend && (
<p
className={
"text-xs mt-1 " +
(trend.dir === "up" ? "text-red-500" : "text-green-500")
}
>
{trend.label}
</p>
)}
</>
)}
</CardContent>
</Card>
);
}
import type { UsageProjection as UP, CacheEfficiencyResponse as CER } from "@/types";
interface ProjectionCardProps {
projection: UP | undefined;
isLoading: boolean;
}
function ProjectionCard({ projection, isLoading }: ProjectionCardProps) {
return (
<Card>
<CardHeader className="pb-2">
<CardTitle className="text-base flex items-center gap-2">
<TrendingUp className="h-4 w-4 text-blue-500" />
Monthly Projection
</CardTitle>
</CardHeader>
<CardContent>
{isLoading ? (
<Skeleton className="h-10 w-full" />
) : (
<div>
<div className="text-3xl font-bold">
{projection != null ? "$" + projection.projected_monthly_cost_usd.toFixed(2) : "—"}
</div>
<p className="text-xs text-muted-foreground mt-1">
Based on {projection?.basis_days ?? 7}-day rolling average ($
{projection?.avg_daily_cost_usd.toFixed(4) ?? "—"}/day)
</p>
</div>
)}
</CardContent>
</Card>
);
}
interface CacheEfficiencyCardProps {
cacheStats: CER | undefined;
isLoading: boolean;
}
function CacheEfficiencyCard({ cacheStats, isLoading }: CacheEfficiencyCardProps) {
const pct = cacheStats ? cacheStats.cache_hit_rate * 100 : 0;
return (
<Card>
<CardHeader className="pb-2">
<CardTitle className="text-base flex items-center gap-2">
<Sparkles className="h-4 w-4 text-purple-500" />
Cache Efficiency
</CardTitle>
</CardHeader>
<CardContent>
{isLoading ? (
<Skeleton className="h-10 w-full" />
) : (
<div>
<div className="text-3xl font-bold">{pct.toFixed(1)}%</div>
<p className="text-xs text-muted-foreground mt-1">
{cacheStats ? fmtTokens(cacheStats.tokens_cache_read) : "—"} cache reads ·
saved ${cacheStats?.cost_saved_by_cache_usd.toFixed(4) ?? "—"}
</p>
<Progress value={pct} className="mt-2" />
</div>
)}
</CardContent>
</Card>
);
}