v0.15.0: Metrics granularity — per-member / per-task / org + CEO scorecards (#289)

* feat(metrics): capture per-session turns + tool_calls (phase 1)

Persist LLM iterations (turns) and tool invocations per agent spawn session,
the raw signal the granular per-member performance metrics build on (real
effort/iterations vs wall-clock).

- sum_transcript_usage returns a 5-tuple adding turns = unique assistant
  message-id count; _usage_from_transcript + _resolve_active_tokens updated to
  the 5-tuple (active-tokens keeps its 4-tuple contract by slicing).
- SDK: _SessionState.turns, set by /usage/sync; /usage/status (TokenUsageStatus)
  now carries turns + tool_calls (= total_calls).
- orchestrator: new _resolve_final_turns_tools (SDK primary, transcript fallback
  for turns only; Grok -> 0/0) wired into _finalize_spawn_session, which writes
  turns + tool_calls to agent_spawn_sessions.
- migration 055 adds turns + tool_calls (BigInteger DEFAULT 0 -> historical/Grok
  rows read 0, surfaced as n/a). Verified real alembic upgrade/downgrade.

Part of metrics-granularity (v0.15.0); recon-adjusted plan on disk.

* feat(metrics): pure compute_stage_effort helper (phase 2, part 1)

Foundation-layer overlap math (no DB): split each task status window into
active (merged wall-clock overlap of spawn stints — concurrent stints counted
once, so active <= window) vs wait (queue/review idle). Distinct from summed
effort. The per-task metrics service will feed it audit-log windows + spawn
stints. 9 unit tests (disjoint/nested/partial/merged/clamped/zero/multi-window).

* feat(metrics): per-task live metrics + GET /metrics/task/{id} (phase 2)

TaskMetrics dataclass + MetricsService.get_task_metrics: summed spawn effort
(vs wall-clock), turns/tool_calls/tokens/cost, per-stage active-vs-wait
(compute_stage_effort over audit windows x spawn stints), and who-caused-rework
(revision_count + named qa/pr fail events). Open stints and the open final
stage window close at completed_at for a terminal task (else now), so stages
don't grow past completion. Exposed at GET /dashboard/metrics/task/{task_id}
(404 if absent). Real-PG tests (compose/none/in-flight) + route tests (200/404).

* feat(metrics): CEO-as-member scorecard + ceo_reject audit regression (phase 3)

The human CEO is a measured member, read purely from audit_log (agent_role='ceo'
serializes from the CEO StrEnum): approval dwell (awaiting_ceo_approval -> a CEO
decision, incl. the coordination-root reject that lands in pending), unblock
dwell (blocked -> a CEO revive), and god-mode action count (every CEO-attributed
transition). CeoScorecard + MetricsService.get_ceo_scorecard (p50/p90 via
PERCENTILE_CONT, expanding IN for the decision sets) + GET
/dashboard/metrics/member/ceo (declared before any future member/{id} route).

The ceo_reject coordination-root audit gap the plan meant to close was already
closed by the gap-sweep (routes through admin_set_status -> agent_role='ceo'
audit); locked with a regression assertion in the existing coordination-reject
test. Real-PG tests: approval/unblock/godmode, non-ceo exclusion, empty->zeros.

* feat(metrics): audit instrumentation for escalations/blocked-others/idle (phase 4a)

The three extra per-member metrics that had no data source get durable,
in-session audit events (additive; never gate the underlying action):
- apply_escalation -> task.escalated (details.escalator_slug) on both the
  normal block path and the pool-divert path -> escalations count.
- _unblock_dependents -> task.unblocked_dependents (details.count) on the
  completed BLOCKER task, captured before the dependency edges are pruned ->
  blocked-others count (sweeper attributes to the blocker's owner).
- mark_agent_idle -> agent.idle (details.agent_slug) -> idle/utilization (the
  sweeper pairs an idle mark to the member's next spawn for idle duration).
(QA pass-rate needs no new event — reuses task.awaiting_documentation[qa] +
task.qa_fail.) Real-PG tests for each; 111 transition tests still green.

* feat(metrics): member_performance_daily rollup table + migration 056 (phase 4b)

The per-member scorecard rollup: one row per (date, member_kind, agent_slug),
CEO as a first-class member_kind='ceo' row (agent_slug='' NOT NULL so the
NULL-distinct UNIQUE keeps it unique). Full column set + the four CEO-approved
extras (qa_reviews_total/passed, escalations, blocked_others, idle_seconds) plus
blocked_seconds. Overwrite-upsert on (date, member_kind, agent_slug) for an
idempotent sweep. Migration 056 verified real up/down (24 cols, 4 indexes).

* feat(metrics): _sweep_member_performance rollup sweeper (phase 4c)

The daily per-member rollup sweep (mirrors _sweep_daily_rollup): a trailing
7-day, idempotent overwrite-upsert wired into _run_sweep. One focused query per
metric merges into a (date, agent_slug) accumulator — spawn effort/turns/tokens/
cost, completed/first-pass/revisions-received, revisions-caused (qa/pr fails),
QA pass-rate (passed + total), escalations (by escalator_slug), blocked-others
(unblocked_dependents by blocker owner), idle_seconds (idle mark -> next spawn),
blocked_seconds (blocked dwell) — plus one CEO row/day (approval/unblock dwell +
god-mode). Real-PG test asserts every facet + idempotency (a 2nd sweep
overwrites, never doubles); spawn-day != completion-day split is by-design.

* feat(metrics): member/org rollup scorecards + endpoints + live overlay (phase 5)

MemberScorecard + OrgScorecard with derived rates (FPY, effort-throughput,
turns/tool-calls per task, QA pass-rate, utilization) — all division-guarded to
None. get_member_scorecard reads member_performance_daily by slug and overlays
the member's live in-flight (non-terminal) tasks' effort via get_task_metrics
(disjoint by status: completion counts stay rollup-only, overlay only enriches
effort/turns/cost; includes_live_inflight flags it). get_org_scorecard
aggregates the cell (?team=) or whole org. Routes: GET /metrics/member/{agent_id}
(404 if absent, after the ceo literal route) + GET /metrics/org?team=. Real-PG
tests (derived rates, overlay no double-count, guards, org) + route tests.

* feat(metrics): granular CEO completion notification (phase 6)

There was no CEO completion notification at all (EventType.TASK_COMPLETED was
defined but never emitted). Add notify_ceo_of_completion in
NotificationDeliveryService — a granular body (real effort vs wall-clock +
stints/turns/tool-calls/revisions[QA/PR]/cost from get_task_metrics; degrades to
wall-clock-only, turns 'n/a', when there are no spawn sessions). Reuses the
existing ALERT type (no enum migration; the notificationtype PG enum is fixed at
001). ceo_approve now emits TASK_COMPLETED + fires the notification (best-effort
via _notify_completion — never blocks completion); complete() emits
TASK_COMPLETED too (closes the dead-code gap; the WS bridge can forward it).
Pure formatter tests + real-PG notification test.

* [metrics-granularity] Phase 7: panel Scorecards tab + dashboard overview

Add the CEO-facing metrics surfaces for the granularity feature:

- New "Scorecards" tab on the Metrics page: org rollup headline, the
  CEO-as-member card (approval/unblock dwell + god-mode count), and a
  per-member table (completed, first-pass yield, active effort, turns/task,
  QA pass-rate, escalations, blocked-others, utilization). Each member row
  self-fetches its rollup scorecard; live in-flight rows carry a "live" badge.
- New dashboard overview card (ScorecardOverviewPanel): org-wide 30-day
  headline (completed, FPY, throughput/hr, active effort, cost) deep-linking
  into the Scorecards tab.
- Plumbing: TaskMetrics/MemberScorecard/OrgScorecard/CeoScorecard types,
  observability API client methods + empty fallbacks, and the four
  useCeoScorecard/useMemberScorecard/useOrgScorecard/useTaskMetrics hooks.

Panel gate green: tsc, eslint, prettier, vitest (175 tests, +6 new).

* [metrics-granularity] test: make completion-notification robust to shared-DB CEO

test_notify_ceo_of_completion_creates_alert errored in the full suite (passed
in isolation): the session-scoped test DB is shared across the run, and the
sibling real-DB board-gate test commits a role=CEO agent (slug="ceo") without
cleanup — so my env fixture's hardcoded slug="ceo" insert hit a unique-constraint
violation, and a second role=CEO row would also make _get_ceo_agent()'s
scalar_one_or_none() raise. Reuse an existing CEO when present (the singleton the
production system actually has), else create one with a unique slug. Order-
independent. Also reflow test_metrics_instrumentation.py to ruff format.

* chore(release): 0.15.0

Metrics granularity: per-member/per-task/org + CEO-as-member scorecards,
turn/tool-call capture (migration 055), member_performance_daily rollup
(migration 056) with QA pass-rate / escalations / blocked-others / utilization,
per-task active-vs-wait metrics, granular completion notification, panel
Scorecards tab + dashboard Performance card, and the ceo_reject audit fix.

Version bump across the canonical set + CHANGELOG.

* [metrics-granularity] fix pre-tag audit findings (overlay double-count + panel error states)

Adversarial review before the v0.15.0 tag surfaced two real logical gaps:

- MAJOR (backend): the live in-flight overlay re-summed ALL sessions of every
  non-terminal task via get_task_metrics, but _msweep_spawn already rolls up
  every CLOSED session regardless of task status — so a closed session on a
  still-open task was counted twice (rollup + overlay), permanently inflating a
  member's effort/turns/tokens/cost on the common reap/respawn path. The overlay
  now sums only OPEN sessions (ended_at IS NULL), which the closed-only rollup
  can never contain — disjoint by construction. A just-closed session lands in
  the rollup on the next ~60s sweep (no gap of note). Aggregated in SQL to mirror
  _msweep_spawn. Regression test reproduces the double-count (turns 10→5).

- MAJOR (panel): the four new scorecard surfaces used `isLoading || !data` with
  no isError branch, so a failed query span forever on a skeleton. They now
  surface a load error. Tests added.

Also: OrgSummary active-effort formatting no longer round-trips hours→seconds→
hours; dashboard grid uses xl:grid-cols-4 (was 2xl) so 4 panels show at 1280px;
corrected the inaccurate "NULL distinct" CEO-row uniqueness comment (agent_slug
is NOT NULL; the '' tuple is simply distinct from agent rows).

make quality GREEN (cov 95.31%); panel GREEN (vitest 178).

* [metrics-granularity] fix: decode bytes stream message-id before XCLAIM

StreamEventBus._recover_stream passed the pending message id to XCLAIM via
str() on the raw bytes the client returns (redis client has no
decode_responses), producing "b'1782066556728-0'". Redis rejects that with
"Unrecognized XCLAIM option", so pending-message recovery threw on every
reclaim tick and unacked messages from crashed/slow consumers were never
reclaimed (leaking in the PEL on every stream, spamming the error log). Decode
via the existing _to_str helper — the fix the sibling claim path already uses.

Pre-existing in v0.14.0 (unrelated to metrics granularity); folded into this
release per CEO. TDD regression test + CHANGELOG entry. make quality GREEN.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
2026-07-01 05:18:45 +02:00
committed by GitHub
co-authored by Renn F
parent 1e341c766e
commit a8cb2470ba
47 changed files with 4225 additions and 40 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "roboco-panel",
"version": "0.14.0",
"version": "0.15.0",
"private": true,
"packageManager": "pnpm@10.25.0",
"scripts": {
+8 -1
View File
@@ -21,6 +21,7 @@ import { Skeleton } from "@/components/ui/skeleton";
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs";
import { OfflineState } from "@/components/ui/offline-state";
import { DeliveryTabContent } from "@/components/metrics/delivery-tab";
import { ScorecardsTabContent } from "@/components/metrics/scorecards-tab";
import {
UsageTimeSeriesChart,
ModelUsageDonut,
@@ -602,12 +603,13 @@ function CacheEfficiencyCard({
// ─── Tab types ────────────────────────────────────────────────────────────────
type MetricsTab = "performance" | "token-usage" | "delivery";
type MetricsTab = "performance" | "token-usage" | "delivery" | "scorecards";
const VALID_METRICS_TABS: MetricsTab[] = [
"performance",
"token-usage",
"delivery",
"scorecards",
];
function isValidMetricsTab(value: string | null): value is MetricsTab {
@@ -649,6 +651,7 @@ function MetricsPageContent() {
<TabsTrigger value="performance">Performance</TabsTrigger>
<TabsTrigger value="token-usage">Token Usage</TabsTrigger>
<TabsTrigger value="delivery">Delivery</TabsTrigger>
<TabsTrigger value="scorecards">Scorecards</TabsTrigger>
</TabsList>
<TabsContent value="performance" className="mt-6">
@@ -662,6 +665,10 @@ function MetricsPageContent() {
<TabsContent value="delivery" className="mt-6">
<DeliveryTabContent />
</TabsContent>
<TabsContent value="scorecards" className="mt-6">
<ScorecardsTabContent />
</TabsContent>
</Tabs>
</div>
);
@@ -0,0 +1,65 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import { render, screen } from "@testing-library/react";
const { mockOrg } = vi.hoisted(() => ({ mockOrg: vi.fn() }));
vi.mock("@/hooks/use-observability", () => ({
useOrgScorecard: mockOrg,
}));
import { ScorecardOverviewPanel } from "../scorecard-overview-panel";
describe("ScorecardOverviewPanel", () => {
beforeEach(() => {
mockOrg.mockReturnValue({
data: {
scope: "org",
team: null,
member_count: 3,
tasks_completed: 42,
first_pass_yield: 0.75,
effort_throughput_per_hour: 1.5,
active_runtime_hours: 12.3,
turns: 0,
tool_calls: 0,
tokens: 0,
cost_usd: 9.5,
revisions_caused: 0,
revisions_received: 0,
},
isLoading: false,
});
});
it("renders headline org figures", () => {
render(<ScorecardOverviewPanel />);
expect(screen.getByText("42")).toBeInTheDocument();
expect(screen.getByText("75%")).toBeInTheDocument();
expect(screen.getByText("$9.50")).toBeInTheDocument();
});
it("deep-links into the Scorecards metrics tab", () => {
render(<ScorecardOverviewPanel />);
const link = screen.getByRole("link", { name: /scorecards/i });
expect(link).toHaveAttribute("href", "/metrics?tab=scorecards");
});
it("shows a skeleton while loading", () => {
mockOrg.mockReturnValue({ data: undefined, isLoading: true });
const { container } = render(<ScorecardOverviewPanel />);
expect(container.querySelectorAll('[data-slot="skeleton"]').length).toBe(5);
});
it("surfaces an error instead of an endless skeleton", () => {
mockOrg.mockReturnValue({
data: undefined,
isLoading: false,
isError: true,
});
const { container } = render(<ScorecardOverviewPanel />);
expect(
screen.getByText(/failed to load performance metrics/i),
).toBeInTheDocument();
expect(container.querySelectorAll('[data-slot="skeleton"]').length).toBe(0);
});
});
@@ -20,6 +20,7 @@ import { StrategySignalsPanel } from "./strategy-signals-panel";
import type { Activity } from "./activity-item";
import { Button } from "@/components/ui/button";
import { UsageOverviewPanel } from "./usage-overview-panel";
import { ScorecardOverviewPanel } from "./scorecard-overview-panel";
import { RefreshCw, Settings, AlertCircle } from "lucide-react";
import Link from "next/link";
@@ -123,14 +124,15 @@ export function CommandCenter() {
{/* Playbook review queue (hidden when no drafts) */}
<PlaybookReviewQueue />
{/* Metrics, Alerts, and Usage Row */}
<div className="grid grid-cols-1 lg:grid-cols-3 xl:grid-cols-3 2xl:grid-cols-3 gap-6">
{/* Metrics, Alerts, Usage, and Performance Row */}
<div className="grid grid-cols-1 lg:grid-cols-2 xl:grid-cols-4 gap-6">
<KeyMetricsPanel
metrics={overview?.key_metrics}
isLoading={loadingOverview}
/>
<AuditorAlertsPanel alerts={flags} isLoading={loadingFlags} />
<UsageOverviewPanel />
<ScorecardOverviewPanel />
</div>
{/* Blockers and Activity Row */}
+1
View File
@@ -14,3 +14,4 @@ export { ReleaseProposalCard } from "./release-proposal-card";
export { PlaybookReviewQueue } from "./playbook-review-queue";
export { StrategySignalsPanel } from "./strategy-signals-panel";
export { UsageOverviewPanel } from "./usage-overview-panel";
export { ScorecardOverviewPanel } from "./scorecard-overview-panel";
@@ -0,0 +1,109 @@
"use client";
import Link from "next/link";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Skeleton } from "@/components/ui/skeleton";
import { useOrgScorecard } from "@/hooks/use-observability";
import {
Trophy,
CheckCircle2,
Gauge,
Clock,
Coins,
ArrowRight,
} from "lucide-react";
function pctOrNa(rate: number | null): string {
return rate === null ? "n/a" : (rate * 100).toFixed(0) + "%";
}
function numOrNa(value: number | null, digits = 2): string {
return value === null ? "n/a" : value.toFixed(digits);
}
interface MetricRowProps {
icon: React.ReactNode;
label: string;
value: string;
}
function MetricRow({ icon, label, value }: MetricRowProps) {
return (
<div className="flex items-center justify-between py-1">
<div className="text-muted-foreground flex items-center gap-2 text-sm">
{icon}
{label}
</div>
<span className="text-sm font-semibold">{value}</span>
</div>
);
}
/**
* Dashboard overview of the org-wide performance rollup (last 30 days). Headline
* figures from useOrgScorecard with a deep-link into the full Scorecards tab.
*/
export function ScorecardOverviewPanel() {
const { data, isLoading, isError } = useOrgScorecard();
return (
<Card>
<CardHeader className="pb-3">
<div className="flex items-center justify-between">
<CardTitle className="flex items-center gap-2 text-lg">
<Trophy className="h-5 w-5" />
Performance
</CardTitle>
<Link
href="/metrics?tab=scorecards"
className="text-muted-foreground hover:text-foreground flex items-center gap-1 text-xs"
>
Scorecards
<ArrowRight className="h-3 w-3" />
</Link>
</div>
</CardHeader>
<CardContent>
{isError ? (
<div className="text-muted-foreground text-sm">
Failed to load performance metrics.
</div>
) : isLoading || !data ? (
<div className="space-y-3">
{Array.from({ length: 5 }).map((_, i) => (
<Skeleton key={i} className="h-6" />
))}
</div>
) : (
<div className="divide-y">
<MetricRow
icon={<CheckCircle2 className="h-4 w-4" />}
label="Tasks completed (30d)"
value={String(data.tasks_completed)}
/>
<MetricRow
icon={<Gauge className="h-4 w-4" />}
label="First-pass yield"
value={pctOrNa(data.first_pass_yield)}
/>
<MetricRow
icon={<Gauge className="h-4 w-4 text-blue-500" />}
label="Throughput / hr"
value={numOrNa(data.effort_throughput_per_hour)}
/>
<MetricRow
icon={<Clock className="h-4 w-4" />}
label="Active effort"
value={data.active_runtime_hours.toFixed(1) + "h"}
/>
<MetricRow
icon={<Coins className="h-4 w-4" />}
label="Cost"
value={"$" + data.cost_usd.toFixed(2)}
/>
</div>
)}
</CardContent>
</Card>
);
}
@@ -0,0 +1,161 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import { render, screen } from "@testing-library/react";
import { AgentRole, AgentState, type Agent } from "@/types";
import type { MemberScorecard } from "@/types";
const { mockOrg, mockCeo, mockMember, mockAgents } = vi.hoisted(() => ({
mockOrg: vi.fn(),
mockCeo: vi.fn(),
mockMember: vi.fn(),
mockAgents: vi.fn(),
}));
vi.mock("@/hooks/use-observability", () => ({
useOrgScorecard: mockOrg,
useCeoScorecard: mockCeo,
useMemberScorecard: mockMember,
}));
vi.mock("@/hooks/use-agents", () => ({
useAgents: mockAgents,
}));
import { ScorecardsTabContent } from "../scorecards-tab";
function agent(id: string, name: string, role: AgentRole): Agent {
return {
id,
agent_id: id,
name,
role,
team: null,
cell: null,
status: AgentState.IDLE,
};
}
function member(over: Partial<MemberScorecard>): MemberScorecard {
return {
scope: "member",
id: "a1",
name: "a1",
member_kind: "agent",
tasks_completed: 0,
first_pass_yield: null,
effort_throughput_per_hour: null,
active_runtime_hours: 0,
turns: 0,
tool_calls: 0,
tokens: 0,
cost_usd: 0,
turns_per_task: null,
tool_calls_per_task: null,
revisions_caused: 0,
revisions_received: 0,
qa_pass_rate: null,
escalations: 0,
blocked_others: 0,
idle_hours: 0,
utilization: null,
includes_live_inflight: false,
...over,
};
}
describe("ScorecardsTabContent", () => {
beforeEach(() => {
mockOrg.mockReturnValue({
data: {
scope: "org",
team: null,
member_count: 3,
tasks_completed: 42,
first_pass_yield: 0.75,
effort_throughput_per_hour: 1.5,
active_runtime_hours: 12.3,
turns: 0,
tool_calls: 0,
tokens: 0,
cost_usd: 9.5,
revisions_caused: 0,
revisions_received: 0,
},
isLoading: false,
});
mockCeo.mockReturnValue({
data: {
member_kind: "ceo",
approval_p50_seconds: 3600,
approval_p90_seconds: 7200,
approval_count: 5,
unblock_p50_seconds: 1800,
unblock_count: 2,
godmode_actions: 1,
},
isLoading: false,
});
mockMember.mockReturnValue({
data: member({
id: "dev1",
name: "be-dev-1",
tasks_completed: 7,
first_pass_yield: 0.8,
active_runtime_hours: 4.2,
qa_pass_rate: 0.9,
escalations: 1,
blocked_others: 2,
utilization: 0.6,
}),
isLoading: false,
});
mockAgents.mockReturnValue({
data: [
agent("dev1", "be-dev-1", AgentRole.DEVELOPER),
agent("ceo", "Renzo", AgentRole.CEO),
agent("sys", "system", AgentRole.SYSTEM),
],
});
});
it("renders the org rollup headline figures", () => {
render(<ScorecardsTabContent />);
expect(screen.getByText("42")).toBeInTheDocument(); // tasks_completed
expect(screen.getByText("75%")).toBeInTheDocument(); // first-pass yield
expect(screen.getByText("$9.50")).toBeInTheDocument(); // cost
});
it("renders the CEO approval/unblock figures", () => {
render(<ScorecardsTabContent />);
expect(screen.getByText("Approvals")).toBeInTheDocument();
expect(screen.getByText("5")).toBeInTheDocument(); // approval_count
expect(screen.getByText("God-mode actions")).toBeInTheDocument();
});
it("lists only non-CEO/non-system members in the table", () => {
render(<ScorecardsTabContent />);
expect(screen.getByText("be-dev-1")).toBeInTheDocument();
// CEO and system are excluded from the member table.
expect(screen.queryByText("Renzo")).not.toBeInTheDocument();
expect(screen.queryByText("system")).not.toBeInTheDocument();
});
it("surfaces load errors instead of an endless skeleton", () => {
mockOrg.mockReturnValue({
data: undefined,
isLoading: false,
isError: true,
});
mockMember.mockReturnValue({
data: undefined,
isLoading: false,
isError: true,
});
render(<ScorecardsTabContent />);
expect(
screen.getByText(/failed to load organization metrics/i),
).toBeInTheDocument();
// The member row shows a failed marker rather than a perpetual skeleton
// (exact lowercase text, distinct from the org card's message).
expect(screen.getByText("failed to load")).toBeInTheDocument();
});
});
@@ -0,0 +1,191 @@
"use client";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Skeleton } from "@/components/ui/skeleton";
import { Badge } from "@/components/ui/badge";
import {
Table,
TableBody,
TableCell,
TableHead,
TableHeader,
TableRow,
} from "@/components/ui/table";
import {
useCeoScorecard,
useMemberScorecard,
useOrgScorecard,
} from "@/hooks/use-observability";
import { useAgents } from "@/hooks/use-agents";
import { AgentRole, type Agent } from "@/types";
function pctOrNa(rate: number | null): string {
return rate === null ? "n/a" : (rate * 100).toFixed(0) + "%";
}
function numOrNa(value: number | null, digits = 1): string {
return value === null ? "n/a" : value.toFixed(digits);
}
function hoursOrDash(seconds: number): string {
return (seconds / 3600).toFixed(1) + "h";
}
/** One member's row — each row self-fetches its rollup scorecard. */
function MemberRow({ agent }: { agent: Agent }) {
const { data, isLoading, isError } = useMemberScorecard(agent.id);
if (isError) {
return (
<TableRow>
<TableCell>{agent.name || agent.slug}</TableCell>
<TableCell colSpan={8} className="text-muted-foreground text-xs">
failed to load
</TableCell>
</TableRow>
);
}
if (isLoading || !data) {
return (
<TableRow>
<TableCell>{agent.name || agent.slug}</TableCell>
<TableCell colSpan={8}>
<Skeleton className="h-4 w-full" />
</TableCell>
</TableRow>
);
}
return (
<TableRow>
<TableCell className="font-medium">
{agent.name || agent.slug}
{data.includes_live_inflight && (
<Badge variant="outline" className="ml-2 text-xs">
live
</Badge>
)}
</TableCell>
<TableCell>{data.tasks_completed}</TableCell>
<TableCell>{pctOrNa(data.first_pass_yield)}</TableCell>
<TableCell>{data.active_runtime_hours.toFixed(1)}h</TableCell>
<TableCell>{numOrNa(data.turns_per_task)}</TableCell>
<TableCell>{pctOrNa(data.qa_pass_rate)}</TableCell>
<TableCell>{data.escalations}</TableCell>
<TableCell>{data.blocked_others}</TableCell>
<TableCell>{pctOrNa(data.utilization)}</TableCell>
</TableRow>
);
}
function OrgSummary() {
const { data, isLoading, isError } = useOrgScorecard();
if (isError)
return (
<div className="text-muted-foreground text-sm">
Failed to load organization metrics.
</div>
);
if (isLoading || !data) return <Skeleton className="h-24 w-full" />;
const cells: [string, string][] = [
["Members", String(data.member_count)],
["Completed", String(data.tasks_completed)],
["First-pass yield", pctOrNa(data.first_pass_yield)],
["Throughput/hr", numOrNa(data.effort_throughput_per_hour, 2)],
["Active effort", data.active_runtime_hours.toFixed(1) + "h"],
["Cost", "$" + data.cost_usd.toFixed(2)],
];
return (
<div className="grid grid-cols-2 gap-4 sm:grid-cols-3 lg:grid-cols-6">
{cells.map(([k, v]) => (
<div key={k}>
<div className="text-2xl font-semibold">{v}</div>
<div className="text-muted-foreground text-sm">{k}</div>
</div>
))}
</div>
);
}
function CeoCard() {
const { data, isLoading, isError } = useCeoScorecard();
if (isError)
return (
<div className="text-muted-foreground text-sm">
Failed to load CEO metrics.
</div>
);
if (isLoading || !data) return <Skeleton className="h-24 w-full" />;
const cells: [string, string][] = [
["Approvals", String(data.approval_count)],
["Approval p50", hoursOrDash(data.approval_p50_seconds)],
["Approval p90", hoursOrDash(data.approval_p90_seconds)],
["Unblocks", String(data.unblock_count)],
["Unblock p50", hoursOrDash(data.unblock_p50_seconds)],
["God-mode actions", String(data.godmode_actions)],
];
return (
<div className="grid grid-cols-2 gap-4 sm:grid-cols-3 lg:grid-cols-6">
{cells.map(([k, v]) => (
<div key={k}>
<div className="text-2xl font-semibold">{v}</div>
<div className="text-muted-foreground text-sm">{k}</div>
</div>
))}
</div>
);
}
export function ScorecardsTabContent() {
const { data: agents } = useAgents();
const members = (agents ?? []).filter(
(a) => a.role !== AgentRole.CEO && a.role !== AgentRole.SYSTEM,
);
return (
<div className="space-y-6">
<Card>
<CardHeader>
<CardTitle>Organization</CardTitle>
</CardHeader>
<CardContent>
<OrgSummary />
</CardContent>
</Card>
<Card>
<CardHeader>
<CardTitle>CEO (you)</CardTitle>
</CardHeader>
<CardContent>
<CeoCard />
</CardContent>
</Card>
<Card>
<CardHeader>
<CardTitle>Members</CardTitle>
</CardHeader>
<CardContent>
<Table>
<TableHeader>
<TableRow>
<TableHead>Member</TableHead>
<TableHead>Done</TableHead>
<TableHead>FPY</TableHead>
<TableHead>Effort</TableHead>
<TableHead>Turns/task</TableHead>
<TableHead>QA pass</TableHead>
<TableHead>Escal.</TableHead>
<TableHead>Blocked others</TableHead>
<TableHead>Util.</TableHead>
</TableRow>
</TableHeader>
<TableBody>
{members.map((a) => (
<MemberRow key={a.id} agent={a} />
))}
</TableBody>
</Table>
</CardContent>
</Card>
</div>
);
}
+49
View File
@@ -7,6 +7,10 @@ import type {
BottleneckReport,
ReworkReport,
Scorecard,
CeoScorecard,
MemberScorecard,
OrgScorecard,
TaskMetrics,
} from "@/types";
// =============================================================================
@@ -23,6 +27,14 @@ export const observabilityKeys = {
[...observabilityKeys.all, "rework", days, team ?? "all"] as const,
teamScorecard: (team: string, days: number) =>
[...observabilityKeys.all, "scorecard", "team", team, days] as const,
ceoScorecard: (days: number) =>
[...observabilityKeys.all, "scorecard", "ceo", days] as const,
memberScorecard: (agentId: string, days: number) =>
[...observabilityKeys.all, "scorecard", "member", agentId, days] as const,
orgScorecard: (days: number, team?: string) =>
[...observabilityKeys.all, "scorecard", "org", team ?? "all", days] as const,
taskMetrics: (taskId: string) =>
[...observabilityKeys.all, "task-metrics", taskId] as const,
};
// =============================================================================
@@ -64,3 +76,40 @@ export function useTeamScorecard(team: string, days = 7) {
refetchInterval: 60_000,
});
}
/** The human CEO as a measured member (approval/unblock dwell + god-mode). */
export function useCeoScorecard(days = 30) {
return useQuery<CeoScorecard>({
queryKey: observabilityKeys.ceoScorecard(days),
queryFn: () => observabilityApi.getCeoScorecard(days),
refetchInterval: 60_000,
});
}
/** Per-member rollup scorecard (+ live in-flight overlay). */
export function useMemberScorecard(agentId: string, days = 30) {
return useQuery<MemberScorecard>({
queryKey: observabilityKeys.memberScorecard(agentId, days),
queryFn: () => observabilityApi.getMemberScorecard(agentId, days),
enabled: Boolean(agentId),
refetchInterval: 60_000,
});
}
/** Org-wide (or per-cell) rollup aggregate. */
export function useOrgScorecard(days = 30, team?: string) {
return useQuery<OrgScorecard>({
queryKey: observabilityKeys.orgScorecard(days, team),
queryFn: () => observabilityApi.getOrgScorecard(days, team),
refetchInterval: 60_000,
});
}
/** Granular per-task metrics (active-vs-wait drill-down). */
export function useTaskMetrics(taskId: string) {
return useQuery<TaskMetrics | null>({
queryKey: observabilityKeys.taskMetrics(taskId),
queryFn: () => observabilityApi.getTaskMetrics(taskId),
enabled: Boolean(taskId),
});
}
+103
View File
@@ -5,6 +5,10 @@ import type {
BottleneckReport,
ReworkReport,
Scorecard,
CeoScorecard,
MemberScorecard,
OrgScorecard,
TaskMetrics,
} from "@/types";
// =============================================================================
@@ -39,6 +43,61 @@ function emptyScorecard(scope: string, id: string): Scorecard {
};
}
const EMPTY_CEO: CeoScorecard = {
member_kind: "ceo",
approval_p50_seconds: 0,
approval_p90_seconds: 0,
approval_count: 0,
unblock_p50_seconds: 0,
unblock_count: 0,
godmode_actions: 0,
};
function emptyMember(id: string): MemberScorecard {
return {
scope: "member",
id,
name: id,
member_kind: "agent",
tasks_completed: 0,
first_pass_yield: null,
effort_throughput_per_hour: null,
active_runtime_hours: 0,
turns: 0,
tool_calls: 0,
tokens: 0,
cost_usd: 0,
turns_per_task: null,
tool_calls_per_task: null,
revisions_caused: 0,
revisions_received: 0,
qa_pass_rate: null,
escalations: 0,
blocked_others: 0,
idle_hours: 0,
utilization: null,
includes_live_inflight: false,
};
}
function emptyOrg(team: string | null): OrgScorecard {
return {
scope: team ? "team" : "org",
team,
member_count: 0,
tasks_completed: 0,
first_pass_yield: null,
effort_throughput_per_hour: null,
active_runtime_hours: 0,
turns: 0,
tool_calls: 0,
tokens: 0,
cost_usd: 0,
revisions_caused: 0,
revisions_received: 0,
};
}
// =============================================================================
// API OBJECT
// =============================================================================
@@ -84,4 +143,48 @@ export const observabilityApi = {
);
return data;
},
/** CEO-as-member scorecard — GET /dashboard/metrics/member/ceo?days */
getCeoScorecard: async (days = 30): Promise<CeoScorecard> => {
if (isMockMode()) return EMPTY_CEO;
const { data } = await api.get<CeoScorecard>(
"/dashboard/metrics/member/ceo",
{ params: { days } },
);
return data;
},
/** Per-member rollup scorecard — GET /dashboard/metrics/member/{id}?days */
getMemberScorecard: async (
agentId: string,
days = 30,
): Promise<MemberScorecard> => {
if (isMockMode()) return emptyMember(agentId);
const { data } = await api.get<MemberScorecard>(
`/dashboard/metrics/member/${agentId}`,
{ params: { days } },
);
return data;
},
/** Org / team rollup — GET /dashboard/metrics/org?team&days */
getOrgScorecard: async (
days = 30,
team?: string,
): Promise<OrgScorecard> => {
if (isMockMode()) return emptyOrg(team ?? null);
const { data } = await api.get<OrgScorecard>("/dashboard/metrics/org", {
params: { days, ...(team ? { team } : {}) },
});
return data;
},
/** Granular per-task metrics — GET /dashboard/metrics/task/{id} */
getTaskMetrics: async (taskId: string): Promise<TaskMetrics | null> => {
if (isMockMode()) return null;
const { data } = await api.get<TaskMetrics>(
`/dashboard/metrics/task/${taskId}`,
);
return data;
},
};
+74
View File
@@ -1431,3 +1431,77 @@ export interface Scorecard {
tokens: number;
cost_usd: number;
}
// --- Granular per-member metrics (v0.15.0) ---
export interface StageEffort {
status: string;
active_seconds: number;
wait_seconds: number;
}
export interface TaskMetrics {
task_id: string;
active_runtime_seconds: number;
wall_clock_seconds: number;
turns: number;
tool_calls: number;
tokens: number;
cost_usd: number;
revision_count: number;
qa_fails: number;
pr_fails: number;
stints: number;
stages: StageEffort[];
}
export interface MemberScorecard {
scope: string;
id: string;
name: string;
member_kind: "agent";
tasks_completed: number;
first_pass_yield: number | null;
effort_throughput_per_hour: number | null;
active_runtime_hours: number;
turns: number;
tool_calls: number;
tokens: number;
cost_usd: number;
turns_per_task: number | null;
tool_calls_per_task: number | null;
revisions_caused: number;
revisions_received: number;
qa_pass_rate: number | null;
escalations: number;
blocked_others: number;
idle_hours: number;
utilization: number | null;
includes_live_inflight: boolean;
}
export interface OrgScorecard {
scope: string;
team: string | null;
member_count: number;
tasks_completed: number;
first_pass_yield: number | null;
effort_throughput_per_hour: number | null;
active_runtime_hours: number;
turns: number;
tool_calls: number;
tokens: number;
cost_usd: number;
revisions_caused: number;
revisions_received: number;
}
export interface CeoScorecard {
member_kind: "ceo";
approval_p50_seconds: number;
approval_p90_seconds: number;
approval_count: number;
unblock_p50_seconds: number;
unblock_count: number;
godmode_actions: number;
}