mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
[499f9eb1] Token Usage & Cost Analytics — Full-Stack Instrumentation, Persistence, and Visualization (#90)
* [cd2bf666] feat(usage): add token usage types, API client, hooks, and UI components (#87) (#88) - Append 5 TypeScript interfaces to src/types/index.ts: TokenUsageSnapshot, AgentUsageRow, UsageSession, UsageTimePoint, ModelUsageSlice - Create src/lib/api/usage.ts: Axios singleton + isMockMode guards for getUsageSnapshot, getUsageTimeSeries, getAgentUsage, getUsageSessions, getModelUsage - Create src/hooks/use-usage.ts: usageKeys factory + useUsageSnapshot, useUsageTimeSeries, useAgentUsage, useUsageSessions, useModelUsage hooks - Create UsageOverviewPanel (dashboard/usage-overview-panel.tsx): 6 metric rows with Skeleton loading state; week-over-week trend arrow for cost - Update CommandCenter: Metrics+Alerts row expanded from 2-col to 3-col grid adding UsageOverviewPanel - Create src/components/metrics/ folder: UsageTimeSeriesChart (recharts stacked AreaChart with var(--chart-1/2/3)), ModelUsageDonut (PieChart), AgentUsageChart and TeamUsageChart (BarChart), SessionsTable (sortable columns + 10-row Prev/Next pagination) - Update Metrics page: Token Usage & Costs section with 5 rows (summary cards, time series+donut, agent+team bar charts, projection+cache efficiency, sessions table) - Add usage mini-bar to AgentCard: token count + cost + progress bar; AgentGrid and Agents page pass agentUsageMap through - Install recharts 3.8.1 - Export all new symbols through their barrel index.ts files Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev> * [10372f0f] Implement full token usage instrumentation: DB migration, SDK endpoints, orchestrator hooks, analytics API, WebSocket events, dashboard integration (#86) (#89) * [10372f0f] feat(token-usage): add Alembic migration 026 for token usage tables Create agent_spawn_sessions, token_usage_snapshots, and daily_usage_rollups tables with correct BIGINT columns, indexes, and unique constraint. Chain: 025_agentrole_prompter → 026_token_usage_tables. * [10372f0f] feat(token-usage): add ORM table classes for token usage instrumentation Add AgentSpawnSessionTable, TokenUsageSnapshotTable, DailyUsageRollupTable to db/tables.py. Import BigInteger and Date from SQLAlchemy. All columns match the migration schema with BIGINT token counts and proper indexes. * [10372f0f] feat(billing): add pricing module with calculate_cost() function Create roboco/billing/__init__.py and roboco/billing/pricing.py with calculate_cost() supporting Claude opus/sonnet/haiku models with input/output/cache pricing. Unknown models return 0.0 without raising. * [10372f0f] feat(sdk): add POST /usage/report and GET /usage/status endpoints to agent SDK Extend _SessionState with token counters. Add TokenReportRequest and TokenUsageStatus models. POST /usage/report additively accumulates token counts; GET /usage/status returns current session totals for sweeper polling. * [10372f0f] feat(orchestrator): add token usage instrumentation hooks - _launch_spawn() calls _record_spawn_session() after successful container spawn - stop_agent() calls _finalize_spawn_session() before container removal - _run_sweep() calls _sweep_token_snapshots() and _sweep_daily_rollup() each tick - New methods: _record_spawn_session, _finalize_spawn_session, _sweep_token_snapshots, _sweep_daily_rollup in TOKEN USAGE section * [10372f0f] feat(api): add token usage analytics API with 7 endpoints Create roboco/services/usage.py (UsageService) and roboco/api/routes/usage.py. Endpoints: GET /api/usage/summary, /time-series, /by-agent, /by-team, /by-model, /projection, /cache-efficiency. Register in app.py. * [10372f0f] feat(dashboard): add usage_summary field to CEO dashboard Add UsageSummary schema (tokens_today, cost_today_usd) to dashboard schemas. Add usage_summary: UsageSummary | None to CEOOverview. Update get_ceo_overview() to populate usage_summary from daily_usage_rollups. * [10372f0f] fix(billing/tests): remove dead except block in _sweep_daily_rollup, add unit tests for pricing.py and services/usage.py - Remove unreachable `except Exception as e` block in orchestrator.py _sweep_daily_rollup() (lines 3376-3381) which referenced undefined `agent_id` and was copy-pasted from _sweep_token_snapshots by mistake - Add tests/unit/billing/test_pricing.py: 31 tests covering opus/sonnet/ haiku tiers with all 4 token types, unknown model → 0.0, empty string → 0.0, and substring-match priority (longer fragment wins) - Add tests/unit/services/test_usage.py: 25 tests covering get_summary trend_pct edge cases (prev=0, both=0, prev>0), get_by_agent/team/model pct_of_total summing to 100%, get_projection formula (avg_daily×30), and get_cache_efficiency hit-rate and cost_saved arithmetic - pricing.py: 100% coverage; services/usage.py: 83% coverage (>80% target) * [10372f0f] fix(usage): include cache tokens in time-series total_tokens to fix AC9 consistency violation get_time_series() previously computed total_tokens as tokens_input + tokens_output only. get_summary() includes all 4 token types (input + output + cache_read + cache_write). AC9 requires both endpoints to agree on their totals for the same period. Fix: add tokens_cache_read and tokens_cache_write to the SELECT query in get_time_series() and include them in the total_tokens calculation. Also adds 4 new unit tests in TestGetTimeSeries covering: - total_tokens includes cache_read and cache_write (the AC9 guard) - zero cache tokens still produces correct total - empty result returns empty list - required fields are present in each point * [10372f0f] fix(usage): remove unused imports and include cache tokens in breakdown totals (AC10) - Remove import math (F401 — never used) - Remove text from sqlalchemy import (F401 — never used) - Remove unused local calculate_cost import inside get_cache_efficiency (F401) - Add tokens_cache_read and tokens_cache_write to SELECT in get_by_agent, get_by_team, and get_by_model; update grand_total and per-item total to include all 4 token types so totals match get_summary() (AC10 fix) - Update test mock rows to include explicit tokens_cache_read=0 and tokens_cache_write=0 so they work with the fixed code - Add new test cases: test_cache_tokens_included_in_total_tokens and test_pct_of_total_sums_to_100_with_cache_tokens for each breakdown class --------- Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> * [44b9eb1f] feat(usage): align frontend API client, TS types, and chart components to real backend contract (#92) (#94) Update all usage-related frontend code to match the actual FastAPI backend response shapes and endpoint paths: - panel/src/lib/api/usage.ts: rewrite all 7 API functions to use correct endpoint paths (/usage/summary, /usage/by-agent, /usage/by-model, /usage/by-team, /usage/time-series, /usage/projection, /usage/cache-efficiency); send period query param (24h/7d/30d not hours); mock generators produce data matching real backend shapes exactly; getUsageSessions returns [] in prod (no /usage/sessions endpoint exists) - panel/src/types/index.ts: replace TokenUsageSnapshot with UsageSummary (tokens_input/tokens_output/total_cost_usd/trend_pct); update AgentUsageRow to use agent_slug/total_tokens/cost_usd/pct_of_total; add TeamUsageRow, UsageProjection, CacheEfficiencyResponse; update UsageTimePoint to use bucket field; update UsageSession to use agent_slug - panel/src/hooks/use-usage.ts: rewrite all hooks to match new API and types; add useTeamUsage, useUsageProjection, useCacheEfficiency hooks - panel/src/components/metrics/usage-time-series-chart.tsx: use bucket field (not timestamp) for axis labels - panel/src/components/metrics/agent-usage-chart.tsx: use agent_slug and total_tokens (not agent_name/tokens_today) - panel/src/components/metrics/team-usage-chart.tsx: rewrite to accept TeamUsageRow[] from API directly - panel/src/components/metrics/model-usage-donut.tsx: use total_tokens, cost_usd, pct_of_total (not tokens/cost/percentage) - panel/src/components/metrics/sessions-table.tsx: use agent_slug, sort keys updated - panel/src/components/dashboard/usage-overview-panel.tsx: use useUsageSummary with tokens_input/tokens_output/total_cost_usd/trend_pct - panel/src/app/(dashboard)/metrics/page.tsx: wire all new hooks, add TeamUsageChart, ProjectionCard, CacheEfficiencyCard with correct types - panel/src/app/(dashboard)/agents/page.tsx: key agentUsageMap by agent_slug - panel/src/components/agents/agent-card.tsx: use total_tokens and cost_usd Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev> * [2161b832] fix: SDK_PORT constant, stop_agent lock refactor, usage_session_id binding, rollup 7-day window (#93) (#95) - Add SDK_PORT = 9000 module-level constant to orchestrator.py; replace hardcoded 9000 in _sweep_budget_exceeded URL with SDK_PORT - Add UUID to TYPE_CHECKING imports to satisfy ruff F821 - Refactor stop_agent: call _finalize_spawn_session BEFORE acquiring self._lock so the SDK HTTP round-trip does not hold the lock - Add usage_session_id: UUID | None field to AgentInstance dataclass - Change _record_spawn_session to return UUID | None; wire return value back to instance.usage_session_id in _launch_spawn - Update _finalize_spawn_session to use WHERE id=usage_session_id for direct session row lookup when usage_session_id is not None - Add started_at >= (now_utc - 7 days) filter to _sweep_daily_rollup aggregate query to avoid re-aggregating all-time history each sweep Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> * [2e0759e1] fix: pricing accuracy, import ordering, session-id binding, rollup cleanup, write-hook tests (#97) (#98) - pricing.py: correct claude-opus-4 prices (5/25/0.50/6.25 not 15/75/1.5/3.75) and haiku family prices (1/5/0.10/1.25 not 0.8/4/0.08/0.20); add Ollama zero-cost early-return; add structlog warning for unmatched model names - app.py: move usage_router import before routes.v1 block (ruff isort fix) - orchestrator.py _sweep_daily_rollup: remove unused calculate_cost import; add blank line between stdlib (uuid4) and third-party (sqlalchemy) imports - orchestrator.py _sweep_token_snapshots: prefer direct lookup by instance.usage_session_id; fall back to agent_slug heuristic only when None - tests: add test_sweep_daily_rollup_inserts_new_row and test_stop_agent_finalizes_before_lock to test_orchestrator_write_hooks.py - usage.py, routes/usage.py, stream_bus.py, test files: ruff format/lint fixes Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> * Mypy compliance * fix(migrations,tests): linearize forked migration chain + correct ceo_reject coordination-root expectation The master merge brought in 026_completed_dependency_ids alongside the rework's 026_token_usage_tables — both off 025, forking the alembic head and breaking the enum-parity test. Rebase token-usage onto 026_completed_dependency_ids (linear chain, single head). Also: test_ceo_reject_routes_coordination_task_to_main_pm asserted the old NEEDS_REVISION behavior; the lifecycle fix correctly routes a coordination root to PENDING (Main PM's claim source). Update the assertion. --------- Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev> Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
co-authored by
Frontend Developer 1
Backend Developer 1
Renn F
Frontend Developer 1
Backend Developer 1
Renn F
parent
93c6ef8a57
commit
b3057628b0
@@ -1,4 +1,5 @@
|
||||
export { api, API_URL } from "./client";
|
||||
export { usageApi } from "./usage";
|
||||
export { tasksApi } from "./tasks";
|
||||
export { orchestratorApi } from "./orchestrator";
|
||||
export { channelsApi } from "./channels";
|
||||
|
||||
@@ -0,0 +1,254 @@
|
||||
import api from "./client";
|
||||
import { isMockMode } from "@/lib/mock-data";
|
||||
import type {
|
||||
UsageSummary,
|
||||
AgentUsageRow,
|
||||
TeamUsageRow,
|
||||
ModelUsageSlice,
|
||||
UsageTimePoint,
|
||||
UsageProjection,
|
||||
CacheEfficiencyResponse,
|
||||
UsageSession,
|
||||
} from "@/types";
|
||||
|
||||
export type UsagePeriod = "24h" | "7d" | "30d";
|
||||
|
||||
// =============================================================================
|
||||
// MOCK DATA — shapes must exactly match the real backend response schemas
|
||||
// =============================================================================
|
||||
|
||||
function mockSummary(period: UsagePeriod = "24h"): UsageSummary {
|
||||
const scale = period === "30d" ? 30 : period === "7d" ? 7 : 1;
|
||||
const base = 124_800 * scale;
|
||||
return {
|
||||
tokens_input: Math.round(base * 0.55),
|
||||
tokens_output: Math.round(base * 0.35),
|
||||
total_tokens: base,
|
||||
total_cost_usd: parseFloat((base * 0.000030).toFixed(6)),
|
||||
trend_pct: 12.5,
|
||||
period,
|
||||
};
|
||||
}
|
||||
|
||||
function mockTimeSeries(period: UsagePeriod = "24h"): UsageTimePoint[] {
|
||||
const now = new Date();
|
||||
const points = period === "24h" ? 24 : period === "7d" ? 7 : 30;
|
||||
const step = period === "24h" ? "hour" : "day";
|
||||
return Array.from({ length: points }, (_, i) => {
|
||||
const ts = new Date(now);
|
||||
if (step === "hour") {
|
||||
ts.setHours(now.getHours() - (points - 1 - i), 0, 0, 0);
|
||||
} else {
|
||||
ts.setDate(now.getDate() - (points - 1 - i));
|
||||
ts.setHours(0, 0, 0, 0);
|
||||
}
|
||||
const base = 3_000 + Math.round(Math.random() * 4_000);
|
||||
const tokens_input = Math.round(base * 0.55);
|
||||
const tokens_output = Math.round(base * 0.35);
|
||||
const total_tokens = base;
|
||||
return {
|
||||
bucket: ts.toISOString(),
|
||||
tokens_input,
|
||||
tokens_output,
|
||||
total_tokens,
|
||||
cost_usd: parseFloat((total_tokens * 0.000030).toFixed(6)),
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
function mockAgentUsage(period: UsagePeriod = "24h"): AgentUsageRow[] {
|
||||
const scale = period === "30d" ? 30 : period === "7d" ? 7 : 1;
|
||||
const agents = [
|
||||
{ agent_slug: "be-dev-1" },
|
||||
{ agent_slug: "be-dev-2" },
|
||||
{ agent_slug: "fe-dev-1" },
|
||||
{ agent_slug: "fe-dev-2" },
|
||||
{ agent_slug: "ux-dev-1" },
|
||||
{ agent_slug: "be-qa" },
|
||||
{ agent_slug: "fe-qa" },
|
||||
{ agent_slug: "main-pm" },
|
||||
];
|
||||
const grand = agents.length * 15_000 * scale;
|
||||
return agents.map((a) => {
|
||||
const ti = Math.round((5_000 + Math.random() * 20_000) * scale);
|
||||
const to_ = Math.round(ti * 0.65);
|
||||
const total = ti + to_;
|
||||
return {
|
||||
agent_slug: a.agent_slug,
|
||||
tokens_input: ti,
|
||||
tokens_output: to_,
|
||||
total_tokens: total,
|
||||
cost_usd: parseFloat((total * 0.000030).toFixed(6)),
|
||||
pct_of_total: parseFloat(((total / grand) * 100).toFixed(2)),
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
function mockTeamUsage(period: UsagePeriod = "24h"): TeamUsageRow[] {
|
||||
const scale = period === "30d" ? 30 : period === "7d" ? 7 : 1;
|
||||
const teams = ["backend", "frontend", "ux_ui", "main_pm"];
|
||||
const grand = teams.length * 50_000 * scale;
|
||||
return teams.map((team) => {
|
||||
const ti = Math.round((30_000 + Math.random() * 40_000) * scale);
|
||||
const to_ = Math.round(ti * 0.65);
|
||||
const total = ti + to_;
|
||||
return {
|
||||
team,
|
||||
tokens_input: ti,
|
||||
tokens_output: to_,
|
||||
total_tokens: total,
|
||||
cost_usd: parseFloat((total * 0.000030).toFixed(6)),
|
||||
pct_of_total: parseFloat(((total / grand) * 100).toFixed(2)),
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
function mockModelUsage(period: UsagePeriod = "24h"): ModelUsageSlice[] {
|
||||
const scale = period === "30d" ? 30 : period === "7d" ? 7 : 1;
|
||||
const models = [
|
||||
{ model: "claude-opus-4", share: 0.548 },
|
||||
{ model: "claude-sonnet-4", share: 0.346 },
|
||||
{ model: "claude-haiku-4", share: 0.106 },
|
||||
];
|
||||
const base = 124_800 * scale;
|
||||
return models.map((m) => {
|
||||
const ti = Math.round(base * m.share * 0.55);
|
||||
const to_ = Math.round(base * m.share * 0.35);
|
||||
const total = Math.round(base * m.share);
|
||||
return {
|
||||
model: m.model,
|
||||
tokens_input: ti,
|
||||
tokens_output: to_,
|
||||
total_tokens: total,
|
||||
cost_usd: parseFloat((total * 0.000030).toFixed(6)),
|
||||
pct_of_total: parseFloat((m.share * 100).toFixed(1)),
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
function mockProjection(): UsageProjection {
|
||||
const total_cost_7d = parseFloat((124_800 * 7 * 0.000030).toFixed(6));
|
||||
return {
|
||||
total_cost_7d,
|
||||
avg_daily_cost_usd: parseFloat((total_cost_7d / 7).toFixed(6)),
|
||||
projected_monthly_cost_usd: parseFloat((total_cost_7d / 7 * 30).toFixed(4)),
|
||||
basis_days: 7,
|
||||
};
|
||||
}
|
||||
|
||||
function mockCacheEfficiency(period: UsagePeriod = "24h"): CacheEfficiencyResponse {
|
||||
return {
|
||||
cache_hit_rate: 0.3142,
|
||||
tokens_cache_read: 39_168,
|
||||
tokens_cache_write: 12_480,
|
||||
tokens_input: 85_632,
|
||||
cost_saved_by_cache_usd: parseFloat((39_168 * (3.00 - 0.30) / 1_000_000).toFixed(6)),
|
||||
period,
|
||||
};
|
||||
}
|
||||
|
||||
function mockSessions(): UsageSession[] {
|
||||
const models = ["claude-opus-4", "claude-sonnet-4", "claude-haiku-4"];
|
||||
const agentSlugs = ["be-dev-1", "be-dev-2", "fe-dev-1", "fe-qa", "main-pm"];
|
||||
return Array.from({ length: 35 }, (_, i) => {
|
||||
const agent_slug = agentSlugs[i % agentSlugs.length];
|
||||
const model = models[i % models.length];
|
||||
const input = Math.round(2_000 + Math.random() * 8_000);
|
||||
const output = Math.round(500 + Math.random() * 3_000);
|
||||
const cache = Math.round(100 + Math.random() * 1_000);
|
||||
const started = new Date(Date.now() - (i + 1) * 12 * 60_000);
|
||||
const ended = i < 3 ? null : new Date(started.getTime() + Math.round(5 + Math.random() * 55) * 60_000);
|
||||
return {
|
||||
id: `session-mock-${i + 1}`,
|
||||
agent_slug,
|
||||
started_at: started.toISOString(),
|
||||
ended_at: ended ? ended.toISOString() : null,
|
||||
tokens_input: input,
|
||||
tokens_output: output,
|
||||
tokens_cache: cache,
|
||||
total_tokens: input + output + cache,
|
||||
cost: parseFloat(((input + output) * 0.00003 + cache * 0.000003).toFixed(4)),
|
||||
model,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// API OBJECT
|
||||
// =============================================================================
|
||||
|
||||
export const usageApi = {
|
||||
/** Aggregated token usage summary — GET /usage/summary?period= */
|
||||
getUsageSummary: async (period: UsagePeriod = "24h"): Promise<UsageSummary> => {
|
||||
if (isMockMode()) return mockSummary(period);
|
||||
const { data } = await api.get<UsageSummary>("/usage/summary", {
|
||||
params: { period },
|
||||
});
|
||||
return data;
|
||||
},
|
||||
|
||||
/** Bucketed time-series — GET /usage/time-series?period= */
|
||||
getUsageTimeSeries: async (period: UsagePeriod = "24h"): Promise<UsageTimePoint[]> => {
|
||||
if (isMockMode()) return mockTimeSeries(period);
|
||||
const { data } = await api.get<UsageTimePoint[]>("/usage/time-series", {
|
||||
params: { period },
|
||||
});
|
||||
return data;
|
||||
},
|
||||
|
||||
/** Per-agent usage rows — GET /usage/by-agent?period= */
|
||||
getAgentUsage: async (period: UsagePeriod = "24h"): Promise<AgentUsageRow[]> => {
|
||||
if (isMockMode()) return mockAgentUsage(period);
|
||||
const { data } = await api.get<AgentUsageRow[]>("/usage/by-agent", {
|
||||
params: { period },
|
||||
});
|
||||
return data;
|
||||
},
|
||||
|
||||
/** Per-team usage rows — GET /usage/by-team?period= */
|
||||
getTeamUsage: async (period: UsagePeriod = "24h"): Promise<TeamUsageRow[]> => {
|
||||
if (isMockMode()) return mockTeamUsage(period);
|
||||
const { data } = await api.get<TeamUsageRow[]>("/usage/by-team", {
|
||||
params: { period },
|
||||
});
|
||||
return data;
|
||||
},
|
||||
|
||||
/** Per-model usage slices — GET /usage/by-model?period= */
|
||||
getModelUsage: async (period: UsagePeriod = "24h"): Promise<ModelUsageSlice[]> => {
|
||||
if (isMockMode()) return mockModelUsage(period);
|
||||
const { data } = await api.get<ModelUsageSlice[]>("/usage/by-model", {
|
||||
params: { period },
|
||||
});
|
||||
return data;
|
||||
},
|
||||
|
||||
/** Monthly cost projection — GET /usage/projection */
|
||||
getUsageProjection: async (): Promise<UsageProjection> => {
|
||||
if (isMockMode()) return mockProjection();
|
||||
const { data } = await api.get<UsageProjection>("/usage/projection");
|
||||
return data;
|
||||
},
|
||||
|
||||
/** Cache efficiency stats — GET /usage/cache-efficiency?period= */
|
||||
getCacheEfficiency: async (period: UsagePeriod = "24h"): Promise<CacheEfficiencyResponse> => {
|
||||
if (isMockMode()) return mockCacheEfficiency(period);
|
||||
const { data } = await api.get<CacheEfficiencyResponse>("/usage/cache-efficiency", {
|
||||
params: { period },
|
||||
});
|
||||
return data;
|
||||
},
|
||||
|
||||
/**
|
||||
* Recent inference sessions — mock-mode only.
|
||||
*
|
||||
* The backend has no /usage/sessions endpoint. In production this
|
||||
* returns an empty array so SessionsTable shows a graceful "no data"
|
||||
* state instead of throwing a 404.
|
||||
*/
|
||||
// eslint-disable-next-line @typescript-eslint/no-unused-vars
|
||||
getUsageSessions: async (_limit: number = 100): Promise<UsageSession[]> => {
|
||||
if (isMockMode()) return mockSessions();
|
||||
return [];
|
||||
},
|
||||
};
|
||||
Reference in New Issue
Block a user