mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
v0.15.0: Metrics granularity — per-member / per-task / org + CEO scorecards (#289)
* feat(metrics): capture per-session turns + tool_calls (phase 1)
Persist LLM iterations (turns) and tool invocations per agent spawn session,
the raw signal the granular per-member performance metrics build on (real
effort/iterations vs wall-clock).
- sum_transcript_usage returns a 5-tuple adding turns = unique assistant
message-id count; _usage_from_transcript + _resolve_active_tokens updated to
the 5-tuple (active-tokens keeps its 4-tuple contract by slicing).
- SDK: _SessionState.turns, set by /usage/sync; /usage/status (TokenUsageStatus)
now carries turns + tool_calls (= total_calls).
- orchestrator: new _resolve_final_turns_tools (SDK primary, transcript fallback
for turns only; Grok -> 0/0) wired into _finalize_spawn_session, which writes
turns + tool_calls to agent_spawn_sessions.
- migration 055 adds turns + tool_calls (BigInteger DEFAULT 0 -> historical/Grok
rows read 0, surfaced as n/a). Verified real alembic upgrade/downgrade.
Part of metrics-granularity (v0.15.0); recon-adjusted plan on disk.
* feat(metrics): pure compute_stage_effort helper (phase 2, part 1)
Foundation-layer overlap math (no DB): split each task status window into
active (merged wall-clock overlap of spawn stints — concurrent stints counted
once, so active <= window) vs wait (queue/review idle). Distinct from summed
effort. The per-task metrics service will feed it audit-log windows + spawn
stints. 9 unit tests (disjoint/nested/partial/merged/clamped/zero/multi-window).
* feat(metrics): per-task live metrics + GET /metrics/task/{id} (phase 2)
TaskMetrics dataclass + MetricsService.get_task_metrics: summed spawn effort
(vs wall-clock), turns/tool_calls/tokens/cost, per-stage active-vs-wait
(compute_stage_effort over audit windows x spawn stints), and who-caused-rework
(revision_count + named qa/pr fail events). Open stints and the open final
stage window close at completed_at for a terminal task (else now), so stages
don't grow past completion. Exposed at GET /dashboard/metrics/task/{task_id}
(404 if absent). Real-PG tests (compose/none/in-flight) + route tests (200/404).
* feat(metrics): CEO-as-member scorecard + ceo_reject audit regression (phase 3)
The human CEO is a measured member, read purely from audit_log (agent_role='ceo'
serializes from the CEO StrEnum): approval dwell (awaiting_ceo_approval -> a CEO
decision, incl. the coordination-root reject that lands in pending), unblock
dwell (blocked -> a CEO revive), and god-mode action count (every CEO-attributed
transition). CeoScorecard + MetricsService.get_ceo_scorecard (p50/p90 via
PERCENTILE_CONT, expanding IN for the decision sets) + GET
/dashboard/metrics/member/ceo (declared before any future member/{id} route).
The ceo_reject coordination-root audit gap the plan meant to close was already
closed by the gap-sweep (routes through admin_set_status -> agent_role='ceo'
audit); locked with a regression assertion in the existing coordination-reject
test. Real-PG tests: approval/unblock/godmode, non-ceo exclusion, empty->zeros.
* feat(metrics): audit instrumentation for escalations/blocked-others/idle (phase 4a)
The three extra per-member metrics that had no data source get durable,
in-session audit events (additive; never gate the underlying action):
- apply_escalation -> task.escalated (details.escalator_slug) on both the
normal block path and the pool-divert path -> escalations count.
- _unblock_dependents -> task.unblocked_dependents (details.count) on the
completed BLOCKER task, captured before the dependency edges are pruned ->
blocked-others count (sweeper attributes to the blocker's owner).
- mark_agent_idle -> agent.idle (details.agent_slug) -> idle/utilization (the
sweeper pairs an idle mark to the member's next spawn for idle duration).
(QA pass-rate needs no new event — reuses task.awaiting_documentation[qa] +
task.qa_fail.) Real-PG tests for each; 111 transition tests still green.
* feat(metrics): member_performance_daily rollup table + migration 056 (phase 4b)
The per-member scorecard rollup: one row per (date, member_kind, agent_slug),
CEO as a first-class member_kind='ceo' row (agent_slug='' NOT NULL so the
NULL-distinct UNIQUE keeps it unique). Full column set + the four CEO-approved
extras (qa_reviews_total/passed, escalations, blocked_others, idle_seconds) plus
blocked_seconds. Overwrite-upsert on (date, member_kind, agent_slug) for an
idempotent sweep. Migration 056 verified real up/down (24 cols, 4 indexes).
* feat(metrics): _sweep_member_performance rollup sweeper (phase 4c)
The daily per-member rollup sweep (mirrors _sweep_daily_rollup): a trailing
7-day, idempotent overwrite-upsert wired into _run_sweep. One focused query per
metric merges into a (date, agent_slug) accumulator — spawn effort/turns/tokens/
cost, completed/first-pass/revisions-received, revisions-caused (qa/pr fails),
QA pass-rate (passed + total), escalations (by escalator_slug), blocked-others
(unblocked_dependents by blocker owner), idle_seconds (idle mark -> next spawn),
blocked_seconds (blocked dwell) — plus one CEO row/day (approval/unblock dwell +
god-mode). Real-PG test asserts every facet + idempotency (a 2nd sweep
overwrites, never doubles); spawn-day != completion-day split is by-design.
* feat(metrics): member/org rollup scorecards + endpoints + live overlay (phase 5)
MemberScorecard + OrgScorecard with derived rates (FPY, effort-throughput,
turns/tool-calls per task, QA pass-rate, utilization) — all division-guarded to
None. get_member_scorecard reads member_performance_daily by slug and overlays
the member's live in-flight (non-terminal) tasks' effort via get_task_metrics
(disjoint by status: completion counts stay rollup-only, overlay only enriches
effort/turns/cost; includes_live_inflight flags it). get_org_scorecard
aggregates the cell (?team=) or whole org. Routes: GET /metrics/member/{agent_id}
(404 if absent, after the ceo literal route) + GET /metrics/org?team=. Real-PG
tests (derived rates, overlay no double-count, guards, org) + route tests.
* feat(metrics): granular CEO completion notification (phase 6)
There was no CEO completion notification at all (EventType.TASK_COMPLETED was
defined but never emitted). Add notify_ceo_of_completion in
NotificationDeliveryService — a granular body (real effort vs wall-clock +
stints/turns/tool-calls/revisions[QA/PR]/cost from get_task_metrics; degrades to
wall-clock-only, turns 'n/a', when there are no spawn sessions). Reuses the
existing ALERT type (no enum migration; the notificationtype PG enum is fixed at
001). ceo_approve now emits TASK_COMPLETED + fires the notification (best-effort
via _notify_completion — never blocks completion); complete() emits
TASK_COMPLETED too (closes the dead-code gap; the WS bridge can forward it).
Pure formatter tests + real-PG notification test.
* [metrics-granularity] Phase 7: panel Scorecards tab + dashboard overview
Add the CEO-facing metrics surfaces for the granularity feature:
- New "Scorecards" tab on the Metrics page: org rollup headline, the
CEO-as-member card (approval/unblock dwell + god-mode count), and a
per-member table (completed, first-pass yield, active effort, turns/task,
QA pass-rate, escalations, blocked-others, utilization). Each member row
self-fetches its rollup scorecard; live in-flight rows carry a "live" badge.
- New dashboard overview card (ScorecardOverviewPanel): org-wide 30-day
headline (completed, FPY, throughput/hr, active effort, cost) deep-linking
into the Scorecards tab.
- Plumbing: TaskMetrics/MemberScorecard/OrgScorecard/CeoScorecard types,
observability API client methods + empty fallbacks, and the four
useCeoScorecard/useMemberScorecard/useOrgScorecard/useTaskMetrics hooks.
Panel gate green: tsc, eslint, prettier, vitest (175 tests, +6 new).
* [metrics-granularity] test: make completion-notification robust to shared-DB CEO
test_notify_ceo_of_completion_creates_alert errored in the full suite (passed
in isolation): the session-scoped test DB is shared across the run, and the
sibling real-DB board-gate test commits a role=CEO agent (slug="ceo") without
cleanup — so my env fixture's hardcoded slug="ceo" insert hit a unique-constraint
violation, and a second role=CEO row would also make _get_ceo_agent()'s
scalar_one_or_none() raise. Reuse an existing CEO when present (the singleton the
production system actually has), else create one with a unique slug. Order-
independent. Also reflow test_metrics_instrumentation.py to ruff format.
* chore(release): 0.15.0
Metrics granularity: per-member/per-task/org + CEO-as-member scorecards,
turn/tool-call capture (migration 055), member_performance_daily rollup
(migration 056) with QA pass-rate / escalations / blocked-others / utilization,
per-task active-vs-wait metrics, granular completion notification, panel
Scorecards tab + dashboard Performance card, and the ceo_reject audit fix.
Version bump across the canonical set + CHANGELOG.
* [metrics-granularity] fix pre-tag audit findings (overlay double-count + panel error states)
Adversarial review before the v0.15.0 tag surfaced two real logical gaps:
- MAJOR (backend): the live in-flight overlay re-summed ALL sessions of every
non-terminal task via get_task_metrics, but _msweep_spawn already rolls up
every CLOSED session regardless of task status — so a closed session on a
still-open task was counted twice (rollup + overlay), permanently inflating a
member's effort/turns/tokens/cost on the common reap/respawn path. The overlay
now sums only OPEN sessions (ended_at IS NULL), which the closed-only rollup
can never contain — disjoint by construction. A just-closed session lands in
the rollup on the next ~60s sweep (no gap of note). Aggregated in SQL to mirror
_msweep_spawn. Regression test reproduces the double-count (turns 10→5).
- MAJOR (panel): the four new scorecard surfaces used `isLoading || !data` with
no isError branch, so a failed query span forever on a skeleton. They now
surface a load error. Tests added.
Also: OrgSummary active-effort formatting no longer round-trips hours→seconds→
hours; dashboard grid uses xl:grid-cols-4 (was 2xl) so 4 panels show at 1280px;
corrected the inaccurate "NULL distinct" CEO-row uniqueness comment (agent_slug
is NOT NULL; the '' tuple is simply distinct from agent rows).
make quality GREEN (cov 95.31%); panel GREEN (vitest 178).
* [metrics-granularity] fix: decode bytes stream message-id before XCLAIM
StreamEventBus._recover_stream passed the pending message id to XCLAIM via
str() on the raw bytes the client returns (redis client has no
decode_responses), producing "b'1782066556728-0'". Redis rejects that with
"Unrecognized XCLAIM option", so pending-message recovery threw on every
reclaim tick and unacked messages from crashed/slow consumers were never
reclaimed (leaking in the PEL on every stream, spamming the error log). Decode
via the existing _to_str helper — the fix the sibling claim path already uses.
Pre-existing in v0.14.0 (unrelated to metrics granularity); folded into this
release per CEO. TDD regression test + CHANGELOG entry. make quality GREEN.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "roboco-panel",
|
||||
"version": "0.14.0",
|
||||
"version": "0.15.0",
|
||||
"private": true,
|
||||
"packageManager": "pnpm@10.25.0",
|
||||
"scripts": {
|
||||
|
||||
@@ -21,6 +21,7 @@ import { Skeleton } from "@/components/ui/skeleton";
|
||||
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs";
|
||||
import { OfflineState } from "@/components/ui/offline-state";
|
||||
import { DeliveryTabContent } from "@/components/metrics/delivery-tab";
|
||||
import { ScorecardsTabContent } from "@/components/metrics/scorecards-tab";
|
||||
import {
|
||||
UsageTimeSeriesChart,
|
||||
ModelUsageDonut,
|
||||
@@ -602,12 +603,13 @@ function CacheEfficiencyCard({
|
||||
|
||||
// ─── Tab types ────────────────────────────────────────────────────────────────
|
||||
|
||||
type MetricsTab = "performance" | "token-usage" | "delivery";
|
||||
type MetricsTab = "performance" | "token-usage" | "delivery" | "scorecards";
|
||||
|
||||
const VALID_METRICS_TABS: MetricsTab[] = [
|
||||
"performance",
|
||||
"token-usage",
|
||||
"delivery",
|
||||
"scorecards",
|
||||
];
|
||||
|
||||
function isValidMetricsTab(value: string | null): value is MetricsTab {
|
||||
@@ -649,6 +651,7 @@ function MetricsPageContent() {
|
||||
<TabsTrigger value="performance">Performance</TabsTrigger>
|
||||
<TabsTrigger value="token-usage">Token Usage</TabsTrigger>
|
||||
<TabsTrigger value="delivery">Delivery</TabsTrigger>
|
||||
<TabsTrigger value="scorecards">Scorecards</TabsTrigger>
|
||||
</TabsList>
|
||||
|
||||
<TabsContent value="performance" className="mt-6">
|
||||
@@ -662,6 +665,10 @@ function MetricsPageContent() {
|
||||
<TabsContent value="delivery" className="mt-6">
|
||||
<DeliveryTabContent />
|
||||
</TabsContent>
|
||||
|
||||
<TabsContent value="scorecards" className="mt-6">
|
||||
<ScorecardsTabContent />
|
||||
</TabsContent>
|
||||
</Tabs>
|
||||
</div>
|
||||
);
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
import { describe, it, expect, vi, beforeEach } from "vitest";
|
||||
import { render, screen } from "@testing-library/react";
|
||||
|
||||
const { mockOrg } = vi.hoisted(() => ({ mockOrg: vi.fn() }));
|
||||
|
||||
vi.mock("@/hooks/use-observability", () => ({
|
||||
useOrgScorecard: mockOrg,
|
||||
}));
|
||||
|
||||
import { ScorecardOverviewPanel } from "../scorecard-overview-panel";
|
||||
|
||||
describe("ScorecardOverviewPanel", () => {
|
||||
beforeEach(() => {
|
||||
mockOrg.mockReturnValue({
|
||||
data: {
|
||||
scope: "org",
|
||||
team: null,
|
||||
member_count: 3,
|
||||
tasks_completed: 42,
|
||||
first_pass_yield: 0.75,
|
||||
effort_throughput_per_hour: 1.5,
|
||||
active_runtime_hours: 12.3,
|
||||
turns: 0,
|
||||
tool_calls: 0,
|
||||
tokens: 0,
|
||||
cost_usd: 9.5,
|
||||
revisions_caused: 0,
|
||||
revisions_received: 0,
|
||||
},
|
||||
isLoading: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("renders headline org figures", () => {
|
||||
render(<ScorecardOverviewPanel />);
|
||||
expect(screen.getByText("42")).toBeInTheDocument();
|
||||
expect(screen.getByText("75%")).toBeInTheDocument();
|
||||
expect(screen.getByText("$9.50")).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("deep-links into the Scorecards metrics tab", () => {
|
||||
render(<ScorecardOverviewPanel />);
|
||||
const link = screen.getByRole("link", { name: /scorecards/i });
|
||||
expect(link).toHaveAttribute("href", "/metrics?tab=scorecards");
|
||||
});
|
||||
|
||||
it("shows a skeleton while loading", () => {
|
||||
mockOrg.mockReturnValue({ data: undefined, isLoading: true });
|
||||
const { container } = render(<ScorecardOverviewPanel />);
|
||||
expect(container.querySelectorAll('[data-slot="skeleton"]').length).toBe(5);
|
||||
});
|
||||
|
||||
it("surfaces an error instead of an endless skeleton", () => {
|
||||
mockOrg.mockReturnValue({
|
||||
data: undefined,
|
||||
isLoading: false,
|
||||
isError: true,
|
||||
});
|
||||
const { container } = render(<ScorecardOverviewPanel />);
|
||||
expect(
|
||||
screen.getByText(/failed to load performance metrics/i),
|
||||
).toBeInTheDocument();
|
||||
expect(container.querySelectorAll('[data-slot="skeleton"]').length).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -20,6 +20,7 @@ import { StrategySignalsPanel } from "./strategy-signals-panel";
|
||||
import type { Activity } from "./activity-item";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import { UsageOverviewPanel } from "./usage-overview-panel";
|
||||
import { ScorecardOverviewPanel } from "./scorecard-overview-panel";
|
||||
import { RefreshCw, Settings, AlertCircle } from "lucide-react";
|
||||
import Link from "next/link";
|
||||
|
||||
@@ -123,14 +124,15 @@ export function CommandCenter() {
|
||||
{/* Playbook review queue (hidden when no drafts) */}
|
||||
<PlaybookReviewQueue />
|
||||
|
||||
{/* Metrics, Alerts, and Usage Row */}
|
||||
<div className="grid grid-cols-1 lg:grid-cols-3 xl:grid-cols-3 2xl:grid-cols-3 gap-6">
|
||||
{/* Metrics, Alerts, Usage, and Performance Row */}
|
||||
<div className="grid grid-cols-1 lg:grid-cols-2 xl:grid-cols-4 gap-6">
|
||||
<KeyMetricsPanel
|
||||
metrics={overview?.key_metrics}
|
||||
isLoading={loadingOverview}
|
||||
/>
|
||||
<AuditorAlertsPanel alerts={flags} isLoading={loadingFlags} />
|
||||
<UsageOverviewPanel />
|
||||
<ScorecardOverviewPanel />
|
||||
</div>
|
||||
|
||||
{/* Blockers and Activity Row */}
|
||||
|
||||
@@ -14,3 +14,4 @@ export { ReleaseProposalCard } from "./release-proposal-card";
|
||||
export { PlaybookReviewQueue } from "./playbook-review-queue";
|
||||
export { StrategySignalsPanel } from "./strategy-signals-panel";
|
||||
export { UsageOverviewPanel } from "./usage-overview-panel";
|
||||
export { ScorecardOverviewPanel } from "./scorecard-overview-panel";
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
"use client";
|
||||
|
||||
import Link from "next/link";
|
||||
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
|
||||
import { Skeleton } from "@/components/ui/skeleton";
|
||||
import { useOrgScorecard } from "@/hooks/use-observability";
|
||||
import {
|
||||
Trophy,
|
||||
CheckCircle2,
|
||||
Gauge,
|
||||
Clock,
|
||||
Coins,
|
||||
ArrowRight,
|
||||
} from "lucide-react";
|
||||
|
||||
function pctOrNa(rate: number | null): string {
|
||||
return rate === null ? "n/a" : (rate * 100).toFixed(0) + "%";
|
||||
}
|
||||
|
||||
function numOrNa(value: number | null, digits = 2): string {
|
||||
return value === null ? "n/a" : value.toFixed(digits);
|
||||
}
|
||||
|
||||
interface MetricRowProps {
|
||||
icon: React.ReactNode;
|
||||
label: string;
|
||||
value: string;
|
||||
}
|
||||
|
||||
function MetricRow({ icon, label, value }: MetricRowProps) {
|
||||
return (
|
||||
<div className="flex items-center justify-between py-1">
|
||||
<div className="text-muted-foreground flex items-center gap-2 text-sm">
|
||||
{icon}
|
||||
{label}
|
||||
</div>
|
||||
<span className="text-sm font-semibold">{value}</span>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Dashboard overview of the org-wide performance rollup (last 30 days). Headline
|
||||
* figures from useOrgScorecard with a deep-link into the full Scorecards tab.
|
||||
*/
|
||||
export function ScorecardOverviewPanel() {
|
||||
const { data, isLoading, isError } = useOrgScorecard();
|
||||
|
||||
return (
|
||||
<Card>
|
||||
<CardHeader className="pb-3">
|
||||
<div className="flex items-center justify-between">
|
||||
<CardTitle className="flex items-center gap-2 text-lg">
|
||||
<Trophy className="h-5 w-5" />
|
||||
Performance
|
||||
</CardTitle>
|
||||
<Link
|
||||
href="/metrics?tab=scorecards"
|
||||
className="text-muted-foreground hover:text-foreground flex items-center gap-1 text-xs"
|
||||
>
|
||||
Scorecards
|
||||
<ArrowRight className="h-3 w-3" />
|
||||
</Link>
|
||||
</div>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
{isError ? (
|
||||
<div className="text-muted-foreground text-sm">
|
||||
Failed to load performance metrics.
|
||||
</div>
|
||||
) : isLoading || !data ? (
|
||||
<div className="space-y-3">
|
||||
{Array.from({ length: 5 }).map((_, i) => (
|
||||
<Skeleton key={i} className="h-6" />
|
||||
))}
|
||||
</div>
|
||||
) : (
|
||||
<div className="divide-y">
|
||||
<MetricRow
|
||||
icon={<CheckCircle2 className="h-4 w-4" />}
|
||||
label="Tasks completed (30d)"
|
||||
value={String(data.tasks_completed)}
|
||||
/>
|
||||
<MetricRow
|
||||
icon={<Gauge className="h-4 w-4" />}
|
||||
label="First-pass yield"
|
||||
value={pctOrNa(data.first_pass_yield)}
|
||||
/>
|
||||
<MetricRow
|
||||
icon={<Gauge className="h-4 w-4 text-blue-500" />}
|
||||
label="Throughput / hr"
|
||||
value={numOrNa(data.effort_throughput_per_hour)}
|
||||
/>
|
||||
<MetricRow
|
||||
icon={<Clock className="h-4 w-4" />}
|
||||
label="Active effort"
|
||||
value={data.active_runtime_hours.toFixed(1) + "h"}
|
||||
/>
|
||||
<MetricRow
|
||||
icon={<Coins className="h-4 w-4" />}
|
||||
label="Cost"
|
||||
value={"$" + data.cost_usd.toFixed(2)}
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,161 @@
|
||||
import { describe, it, expect, vi, beforeEach } from "vitest";
|
||||
import { render, screen } from "@testing-library/react";
|
||||
import { AgentRole, AgentState, type Agent } from "@/types";
|
||||
import type { MemberScorecard } from "@/types";
|
||||
|
||||
const { mockOrg, mockCeo, mockMember, mockAgents } = vi.hoisted(() => ({
|
||||
mockOrg: vi.fn(),
|
||||
mockCeo: vi.fn(),
|
||||
mockMember: vi.fn(),
|
||||
mockAgents: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock("@/hooks/use-observability", () => ({
|
||||
useOrgScorecard: mockOrg,
|
||||
useCeoScorecard: mockCeo,
|
||||
useMemberScorecard: mockMember,
|
||||
}));
|
||||
|
||||
vi.mock("@/hooks/use-agents", () => ({
|
||||
useAgents: mockAgents,
|
||||
}));
|
||||
|
||||
import { ScorecardsTabContent } from "../scorecards-tab";
|
||||
|
||||
function agent(id: string, name: string, role: AgentRole): Agent {
|
||||
return {
|
||||
id,
|
||||
agent_id: id,
|
||||
name,
|
||||
role,
|
||||
team: null,
|
||||
cell: null,
|
||||
status: AgentState.IDLE,
|
||||
};
|
||||
}
|
||||
|
||||
function member(over: Partial<MemberScorecard>): MemberScorecard {
|
||||
return {
|
||||
scope: "member",
|
||||
id: "a1",
|
||||
name: "a1",
|
||||
member_kind: "agent",
|
||||
tasks_completed: 0,
|
||||
first_pass_yield: null,
|
||||
effort_throughput_per_hour: null,
|
||||
active_runtime_hours: 0,
|
||||
turns: 0,
|
||||
tool_calls: 0,
|
||||
tokens: 0,
|
||||
cost_usd: 0,
|
||||
turns_per_task: null,
|
||||
tool_calls_per_task: null,
|
||||
revisions_caused: 0,
|
||||
revisions_received: 0,
|
||||
qa_pass_rate: null,
|
||||
escalations: 0,
|
||||
blocked_others: 0,
|
||||
idle_hours: 0,
|
||||
utilization: null,
|
||||
includes_live_inflight: false,
|
||||
...over,
|
||||
};
|
||||
}
|
||||
|
||||
describe("ScorecardsTabContent", () => {
|
||||
beforeEach(() => {
|
||||
mockOrg.mockReturnValue({
|
||||
data: {
|
||||
scope: "org",
|
||||
team: null,
|
||||
member_count: 3,
|
||||
tasks_completed: 42,
|
||||
first_pass_yield: 0.75,
|
||||
effort_throughput_per_hour: 1.5,
|
||||
active_runtime_hours: 12.3,
|
||||
turns: 0,
|
||||
tool_calls: 0,
|
||||
tokens: 0,
|
||||
cost_usd: 9.5,
|
||||
revisions_caused: 0,
|
||||
revisions_received: 0,
|
||||
},
|
||||
isLoading: false,
|
||||
});
|
||||
mockCeo.mockReturnValue({
|
||||
data: {
|
||||
member_kind: "ceo",
|
||||
approval_p50_seconds: 3600,
|
||||
approval_p90_seconds: 7200,
|
||||
approval_count: 5,
|
||||
unblock_p50_seconds: 1800,
|
||||
unblock_count: 2,
|
||||
godmode_actions: 1,
|
||||
},
|
||||
isLoading: false,
|
||||
});
|
||||
mockMember.mockReturnValue({
|
||||
data: member({
|
||||
id: "dev1",
|
||||
name: "be-dev-1",
|
||||
tasks_completed: 7,
|
||||
first_pass_yield: 0.8,
|
||||
active_runtime_hours: 4.2,
|
||||
qa_pass_rate: 0.9,
|
||||
escalations: 1,
|
||||
blocked_others: 2,
|
||||
utilization: 0.6,
|
||||
}),
|
||||
isLoading: false,
|
||||
});
|
||||
mockAgents.mockReturnValue({
|
||||
data: [
|
||||
agent("dev1", "be-dev-1", AgentRole.DEVELOPER),
|
||||
agent("ceo", "Renzo", AgentRole.CEO),
|
||||
agent("sys", "system", AgentRole.SYSTEM),
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("renders the org rollup headline figures", () => {
|
||||
render(<ScorecardsTabContent />);
|
||||
expect(screen.getByText("42")).toBeInTheDocument(); // tasks_completed
|
||||
expect(screen.getByText("75%")).toBeInTheDocument(); // first-pass yield
|
||||
expect(screen.getByText("$9.50")).toBeInTheDocument(); // cost
|
||||
});
|
||||
|
||||
it("renders the CEO approval/unblock figures", () => {
|
||||
render(<ScorecardsTabContent />);
|
||||
expect(screen.getByText("Approvals")).toBeInTheDocument();
|
||||
expect(screen.getByText("5")).toBeInTheDocument(); // approval_count
|
||||
expect(screen.getByText("God-mode actions")).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("lists only non-CEO/non-system members in the table", () => {
|
||||
render(<ScorecardsTabContent />);
|
||||
expect(screen.getByText("be-dev-1")).toBeInTheDocument();
|
||||
// CEO and system are excluded from the member table.
|
||||
expect(screen.queryByText("Renzo")).not.toBeInTheDocument();
|
||||
expect(screen.queryByText("system")).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("surfaces load errors instead of an endless skeleton", () => {
|
||||
mockOrg.mockReturnValue({
|
||||
data: undefined,
|
||||
isLoading: false,
|
||||
isError: true,
|
||||
});
|
||||
mockMember.mockReturnValue({
|
||||
data: undefined,
|
||||
isLoading: false,
|
||||
isError: true,
|
||||
});
|
||||
render(<ScorecardsTabContent />);
|
||||
expect(
|
||||
screen.getByText(/failed to load organization metrics/i),
|
||||
).toBeInTheDocument();
|
||||
// The member row shows a failed marker rather than a perpetual skeleton
|
||||
// (exact lowercase text, distinct from the org card's message).
|
||||
expect(screen.getByText("failed to load")).toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,191 @@
|
||||
"use client";
|
||||
|
||||
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
|
||||
import { Skeleton } from "@/components/ui/skeleton";
|
||||
import { Badge } from "@/components/ui/badge";
|
||||
import {
|
||||
Table,
|
||||
TableBody,
|
||||
TableCell,
|
||||
TableHead,
|
||||
TableHeader,
|
||||
TableRow,
|
||||
} from "@/components/ui/table";
|
||||
import {
|
||||
useCeoScorecard,
|
||||
useMemberScorecard,
|
||||
useOrgScorecard,
|
||||
} from "@/hooks/use-observability";
|
||||
import { useAgents } from "@/hooks/use-agents";
|
||||
import { AgentRole, type Agent } from "@/types";
|
||||
|
||||
function pctOrNa(rate: number | null): string {
|
||||
return rate === null ? "n/a" : (rate * 100).toFixed(0) + "%";
|
||||
}
|
||||
|
||||
function numOrNa(value: number | null, digits = 1): string {
|
||||
return value === null ? "n/a" : value.toFixed(digits);
|
||||
}
|
||||
|
||||
function hoursOrDash(seconds: number): string {
|
||||
return (seconds / 3600).toFixed(1) + "h";
|
||||
}
|
||||
|
||||
/** One member's row — each row self-fetches its rollup scorecard. */
|
||||
function MemberRow({ agent }: { agent: Agent }) {
|
||||
const { data, isLoading, isError } = useMemberScorecard(agent.id);
|
||||
if (isError) {
|
||||
return (
|
||||
<TableRow>
|
||||
<TableCell>{agent.name || agent.slug}</TableCell>
|
||||
<TableCell colSpan={8} className="text-muted-foreground text-xs">
|
||||
failed to load
|
||||
</TableCell>
|
||||
</TableRow>
|
||||
);
|
||||
}
|
||||
if (isLoading || !data) {
|
||||
return (
|
||||
<TableRow>
|
||||
<TableCell>{agent.name || agent.slug}</TableCell>
|
||||
<TableCell colSpan={8}>
|
||||
<Skeleton className="h-4 w-full" />
|
||||
</TableCell>
|
||||
</TableRow>
|
||||
);
|
||||
}
|
||||
return (
|
||||
<TableRow>
|
||||
<TableCell className="font-medium">
|
||||
{agent.name || agent.slug}
|
||||
{data.includes_live_inflight && (
|
||||
<Badge variant="outline" className="ml-2 text-xs">
|
||||
live
|
||||
</Badge>
|
||||
)}
|
||||
</TableCell>
|
||||
<TableCell>{data.tasks_completed}</TableCell>
|
||||
<TableCell>{pctOrNa(data.first_pass_yield)}</TableCell>
|
||||
<TableCell>{data.active_runtime_hours.toFixed(1)}h</TableCell>
|
||||
<TableCell>{numOrNa(data.turns_per_task)}</TableCell>
|
||||
<TableCell>{pctOrNa(data.qa_pass_rate)}</TableCell>
|
||||
<TableCell>{data.escalations}</TableCell>
|
||||
<TableCell>{data.blocked_others}</TableCell>
|
||||
<TableCell>{pctOrNa(data.utilization)}</TableCell>
|
||||
</TableRow>
|
||||
);
|
||||
}
|
||||
|
||||
function OrgSummary() {
|
||||
const { data, isLoading, isError } = useOrgScorecard();
|
||||
if (isError)
|
||||
return (
|
||||
<div className="text-muted-foreground text-sm">
|
||||
Failed to load organization metrics.
|
||||
</div>
|
||||
);
|
||||
if (isLoading || !data) return <Skeleton className="h-24 w-full" />;
|
||||
const cells: [string, string][] = [
|
||||
["Members", String(data.member_count)],
|
||||
["Completed", String(data.tasks_completed)],
|
||||
["First-pass yield", pctOrNa(data.first_pass_yield)],
|
||||
["Throughput/hr", numOrNa(data.effort_throughput_per_hour, 2)],
|
||||
["Active effort", data.active_runtime_hours.toFixed(1) + "h"],
|
||||
["Cost", "$" + data.cost_usd.toFixed(2)],
|
||||
];
|
||||
return (
|
||||
<div className="grid grid-cols-2 gap-4 sm:grid-cols-3 lg:grid-cols-6">
|
||||
{cells.map(([k, v]) => (
|
||||
<div key={k}>
|
||||
<div className="text-2xl font-semibold">{v}</div>
|
||||
<div className="text-muted-foreground text-sm">{k}</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function CeoCard() {
|
||||
const { data, isLoading, isError } = useCeoScorecard();
|
||||
if (isError)
|
||||
return (
|
||||
<div className="text-muted-foreground text-sm">
|
||||
Failed to load CEO metrics.
|
||||
</div>
|
||||
);
|
||||
if (isLoading || !data) return <Skeleton className="h-24 w-full" />;
|
||||
const cells: [string, string][] = [
|
||||
["Approvals", String(data.approval_count)],
|
||||
["Approval p50", hoursOrDash(data.approval_p50_seconds)],
|
||||
["Approval p90", hoursOrDash(data.approval_p90_seconds)],
|
||||
["Unblocks", String(data.unblock_count)],
|
||||
["Unblock p50", hoursOrDash(data.unblock_p50_seconds)],
|
||||
["God-mode actions", String(data.godmode_actions)],
|
||||
];
|
||||
return (
|
||||
<div className="grid grid-cols-2 gap-4 sm:grid-cols-3 lg:grid-cols-6">
|
||||
{cells.map(([k, v]) => (
|
||||
<div key={k}>
|
||||
<div className="text-2xl font-semibold">{v}</div>
|
||||
<div className="text-muted-foreground text-sm">{k}</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export function ScorecardsTabContent() {
|
||||
const { data: agents } = useAgents();
|
||||
const members = (agents ?? []).filter(
|
||||
(a) => a.role !== AgentRole.CEO && a.role !== AgentRole.SYSTEM,
|
||||
);
|
||||
return (
|
||||
<div className="space-y-6">
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle>Organization</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<OrgSummary />
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle>CEO (you)</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<CeoCard />
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle>Members</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<Table>
|
||||
<TableHeader>
|
||||
<TableRow>
|
||||
<TableHead>Member</TableHead>
|
||||
<TableHead>Done</TableHead>
|
||||
<TableHead>FPY</TableHead>
|
||||
<TableHead>Effort</TableHead>
|
||||
<TableHead>Turns/task</TableHead>
|
||||
<TableHead>QA pass</TableHead>
|
||||
<TableHead>Escal.</TableHead>
|
||||
<TableHead>Blocked others</TableHead>
|
||||
<TableHead>Util.</TableHead>
|
||||
</TableRow>
|
||||
</TableHeader>
|
||||
<TableBody>
|
||||
{members.map((a) => (
|
||||
<MemberRow key={a.id} agent={a} />
|
||||
))}
|
||||
</TableBody>
|
||||
</Table>
|
||||
</CardContent>
|
||||
</Card>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -7,6 +7,10 @@ import type {
|
||||
BottleneckReport,
|
||||
ReworkReport,
|
||||
Scorecard,
|
||||
CeoScorecard,
|
||||
MemberScorecard,
|
||||
OrgScorecard,
|
||||
TaskMetrics,
|
||||
} from "@/types";
|
||||
|
||||
// =============================================================================
|
||||
@@ -23,6 +27,14 @@ export const observabilityKeys = {
|
||||
[...observabilityKeys.all, "rework", days, team ?? "all"] as const,
|
||||
teamScorecard: (team: string, days: number) =>
|
||||
[...observabilityKeys.all, "scorecard", "team", team, days] as const,
|
||||
ceoScorecard: (days: number) =>
|
||||
[...observabilityKeys.all, "scorecard", "ceo", days] as const,
|
||||
memberScorecard: (agentId: string, days: number) =>
|
||||
[...observabilityKeys.all, "scorecard", "member", agentId, days] as const,
|
||||
orgScorecard: (days: number, team?: string) =>
|
||||
[...observabilityKeys.all, "scorecard", "org", team ?? "all", days] as const,
|
||||
taskMetrics: (taskId: string) =>
|
||||
[...observabilityKeys.all, "task-metrics", taskId] as const,
|
||||
};
|
||||
|
||||
// =============================================================================
|
||||
@@ -64,3 +76,40 @@ export function useTeamScorecard(team: string, days = 7) {
|
||||
refetchInterval: 60_000,
|
||||
});
|
||||
}
|
||||
|
||||
/** The human CEO as a measured member (approval/unblock dwell + god-mode). */
|
||||
export function useCeoScorecard(days = 30) {
|
||||
return useQuery<CeoScorecard>({
|
||||
queryKey: observabilityKeys.ceoScorecard(days),
|
||||
queryFn: () => observabilityApi.getCeoScorecard(days),
|
||||
refetchInterval: 60_000,
|
||||
});
|
||||
}
|
||||
|
||||
/** Per-member rollup scorecard (+ live in-flight overlay). */
|
||||
export function useMemberScorecard(agentId: string, days = 30) {
|
||||
return useQuery<MemberScorecard>({
|
||||
queryKey: observabilityKeys.memberScorecard(agentId, days),
|
||||
queryFn: () => observabilityApi.getMemberScorecard(agentId, days),
|
||||
enabled: Boolean(agentId),
|
||||
refetchInterval: 60_000,
|
||||
});
|
||||
}
|
||||
|
||||
/** Org-wide (or per-cell) rollup aggregate. */
|
||||
export function useOrgScorecard(days = 30, team?: string) {
|
||||
return useQuery<OrgScorecard>({
|
||||
queryKey: observabilityKeys.orgScorecard(days, team),
|
||||
queryFn: () => observabilityApi.getOrgScorecard(days, team),
|
||||
refetchInterval: 60_000,
|
||||
});
|
||||
}
|
||||
|
||||
/** Granular per-task metrics (active-vs-wait drill-down). */
|
||||
export function useTaskMetrics(taskId: string) {
|
||||
return useQuery<TaskMetrics | null>({
|
||||
queryKey: observabilityKeys.taskMetrics(taskId),
|
||||
queryFn: () => observabilityApi.getTaskMetrics(taskId),
|
||||
enabled: Boolean(taskId),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -5,6 +5,10 @@ import type {
|
||||
BottleneckReport,
|
||||
ReworkReport,
|
||||
Scorecard,
|
||||
CeoScorecard,
|
||||
MemberScorecard,
|
||||
OrgScorecard,
|
||||
TaskMetrics,
|
||||
} from "@/types";
|
||||
|
||||
// =============================================================================
|
||||
@@ -39,6 +43,61 @@ function emptyScorecard(scope: string, id: string): Scorecard {
|
||||
};
|
||||
}
|
||||
|
||||
const EMPTY_CEO: CeoScorecard = {
|
||||
member_kind: "ceo",
|
||||
approval_p50_seconds: 0,
|
||||
approval_p90_seconds: 0,
|
||||
approval_count: 0,
|
||||
unblock_p50_seconds: 0,
|
||||
unblock_count: 0,
|
||||
godmode_actions: 0,
|
||||
};
|
||||
|
||||
function emptyMember(id: string): MemberScorecard {
|
||||
return {
|
||||
scope: "member",
|
||||
id,
|
||||
name: id,
|
||||
member_kind: "agent",
|
||||
tasks_completed: 0,
|
||||
first_pass_yield: null,
|
||||
effort_throughput_per_hour: null,
|
||||
active_runtime_hours: 0,
|
||||
turns: 0,
|
||||
tool_calls: 0,
|
||||
tokens: 0,
|
||||
cost_usd: 0,
|
||||
turns_per_task: null,
|
||||
tool_calls_per_task: null,
|
||||
revisions_caused: 0,
|
||||
revisions_received: 0,
|
||||
qa_pass_rate: null,
|
||||
escalations: 0,
|
||||
blocked_others: 0,
|
||||
idle_hours: 0,
|
||||
utilization: null,
|
||||
includes_live_inflight: false,
|
||||
};
|
||||
}
|
||||
|
||||
function emptyOrg(team: string | null): OrgScorecard {
|
||||
return {
|
||||
scope: team ? "team" : "org",
|
||||
team,
|
||||
member_count: 0,
|
||||
tasks_completed: 0,
|
||||
first_pass_yield: null,
|
||||
effort_throughput_per_hour: null,
|
||||
active_runtime_hours: 0,
|
||||
turns: 0,
|
||||
tool_calls: 0,
|
||||
tokens: 0,
|
||||
cost_usd: 0,
|
||||
revisions_caused: 0,
|
||||
revisions_received: 0,
|
||||
};
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// API OBJECT
|
||||
// =============================================================================
|
||||
@@ -84,4 +143,48 @@ export const observabilityApi = {
|
||||
);
|
||||
return data;
|
||||
},
|
||||
|
||||
/** CEO-as-member scorecard — GET /dashboard/metrics/member/ceo?days */
|
||||
getCeoScorecard: async (days = 30): Promise<CeoScorecard> => {
|
||||
if (isMockMode()) return EMPTY_CEO;
|
||||
const { data } = await api.get<CeoScorecard>(
|
||||
"/dashboard/metrics/member/ceo",
|
||||
{ params: { days } },
|
||||
);
|
||||
return data;
|
||||
},
|
||||
|
||||
/** Per-member rollup scorecard — GET /dashboard/metrics/member/{id}?days */
|
||||
getMemberScorecard: async (
|
||||
agentId: string,
|
||||
days = 30,
|
||||
): Promise<MemberScorecard> => {
|
||||
if (isMockMode()) return emptyMember(agentId);
|
||||
const { data } = await api.get<MemberScorecard>(
|
||||
`/dashboard/metrics/member/${agentId}`,
|
||||
{ params: { days } },
|
||||
);
|
||||
return data;
|
||||
},
|
||||
|
||||
/** Org / team rollup — GET /dashboard/metrics/org?team&days */
|
||||
getOrgScorecard: async (
|
||||
days = 30,
|
||||
team?: string,
|
||||
): Promise<OrgScorecard> => {
|
||||
if (isMockMode()) return emptyOrg(team ?? null);
|
||||
const { data } = await api.get<OrgScorecard>("/dashboard/metrics/org", {
|
||||
params: { days, ...(team ? { team } : {}) },
|
||||
});
|
||||
return data;
|
||||
},
|
||||
|
||||
/** Granular per-task metrics — GET /dashboard/metrics/task/{id} */
|
||||
getTaskMetrics: async (taskId: string): Promise<TaskMetrics | null> => {
|
||||
if (isMockMode()) return null;
|
||||
const { data } = await api.get<TaskMetrics>(
|
||||
`/dashboard/metrics/task/${taskId}`,
|
||||
);
|
||||
return data;
|
||||
},
|
||||
};
|
||||
|
||||
@@ -1431,3 +1431,77 @@ export interface Scorecard {
|
||||
tokens: number;
|
||||
cost_usd: number;
|
||||
}
|
||||
|
||||
// --- Granular per-member metrics (v0.15.0) ---
|
||||
|
||||
export interface StageEffort {
|
||||
status: string;
|
||||
active_seconds: number;
|
||||
wait_seconds: number;
|
||||
}
|
||||
|
||||
export interface TaskMetrics {
|
||||
task_id: string;
|
||||
active_runtime_seconds: number;
|
||||
wall_clock_seconds: number;
|
||||
turns: number;
|
||||
tool_calls: number;
|
||||
tokens: number;
|
||||
cost_usd: number;
|
||||
revision_count: number;
|
||||
qa_fails: number;
|
||||
pr_fails: number;
|
||||
stints: number;
|
||||
stages: StageEffort[];
|
||||
}
|
||||
|
||||
export interface MemberScorecard {
|
||||
scope: string;
|
||||
id: string;
|
||||
name: string;
|
||||
member_kind: "agent";
|
||||
tasks_completed: number;
|
||||
first_pass_yield: number | null;
|
||||
effort_throughput_per_hour: number | null;
|
||||
active_runtime_hours: number;
|
||||
turns: number;
|
||||
tool_calls: number;
|
||||
tokens: number;
|
||||
cost_usd: number;
|
||||
turns_per_task: number | null;
|
||||
tool_calls_per_task: number | null;
|
||||
revisions_caused: number;
|
||||
revisions_received: number;
|
||||
qa_pass_rate: number | null;
|
||||
escalations: number;
|
||||
blocked_others: number;
|
||||
idle_hours: number;
|
||||
utilization: number | null;
|
||||
includes_live_inflight: boolean;
|
||||
}
|
||||
|
||||
export interface OrgScorecard {
|
||||
scope: string;
|
||||
team: string | null;
|
||||
member_count: number;
|
||||
tasks_completed: number;
|
||||
first_pass_yield: number | null;
|
||||
effort_throughput_per_hour: number | null;
|
||||
active_runtime_hours: number;
|
||||
turns: number;
|
||||
tool_calls: number;
|
||||
tokens: number;
|
||||
cost_usd: number;
|
||||
revisions_caused: number;
|
||||
revisions_received: number;
|
||||
}
|
||||
|
||||
export interface CeoScorecard {
|
||||
member_kind: "ceo";
|
||||
approval_p50_seconds: number;
|
||||
approval_p90_seconds: number;
|
||||
approval_count: number;
|
||||
unblock_p50_seconds: number;
|
||||
unblock_count: number;
|
||||
godmode_actions: number;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user