v0.15.0: Metrics granularity — per-member / per-task / org + CEO scorecards (#289)

* feat(metrics): capture per-session turns + tool_calls (phase 1)

Persist LLM iterations (turns) and tool invocations per agent spawn session,
the raw signal the granular per-member performance metrics build on (real
effort/iterations vs wall-clock).

- sum_transcript_usage returns a 5-tuple adding turns = unique assistant
  message-id count; _usage_from_transcript + _resolve_active_tokens updated to
  the 5-tuple (active-tokens keeps its 4-tuple contract by slicing).
- SDK: _SessionState.turns, set by /usage/sync; /usage/status (TokenUsageStatus)
  now carries turns + tool_calls (= total_calls).
- orchestrator: new _resolve_final_turns_tools (SDK primary, transcript fallback
  for turns only; Grok -> 0/0) wired into _finalize_spawn_session, which writes
  turns + tool_calls to agent_spawn_sessions.
- migration 055 adds turns + tool_calls (BigInteger DEFAULT 0 -> historical/Grok
  rows read 0, surfaced as n/a). Verified real alembic upgrade/downgrade.

Part of metrics-granularity (v0.15.0); recon-adjusted plan on disk.

* feat(metrics): pure compute_stage_effort helper (phase 2, part 1)

Foundation-layer overlap math (no DB): split each task status window into
active (merged wall-clock overlap of spawn stints — concurrent stints counted
once, so active <= window) vs wait (queue/review idle). Distinct from summed
effort. The per-task metrics service will feed it audit-log windows + spawn
stints. 9 unit tests (disjoint/nested/partial/merged/clamped/zero/multi-window).

* feat(metrics): per-task live metrics + GET /metrics/task/{id} (phase 2)

TaskMetrics dataclass + MetricsService.get_task_metrics: summed spawn effort
(vs wall-clock), turns/tool_calls/tokens/cost, per-stage active-vs-wait
(compute_stage_effort over audit windows x spawn stints), and who-caused-rework
(revision_count + named qa/pr fail events). Open stints and the open final
stage window close at completed_at for a terminal task (else now), so stages
don't grow past completion. Exposed at GET /dashboard/metrics/task/{task_id}
(404 if absent). Real-PG tests (compose/none/in-flight) + route tests (200/404).

* feat(metrics): CEO-as-member scorecard + ceo_reject audit regression (phase 3)

The human CEO is a measured member, read purely from audit_log (agent_role='ceo'
serializes from the CEO StrEnum): approval dwell (awaiting_ceo_approval -> a CEO
decision, incl. the coordination-root reject that lands in pending), unblock
dwell (blocked -> a CEO revive), and god-mode action count (every CEO-attributed
transition). CeoScorecard + MetricsService.get_ceo_scorecard (p50/p90 via
PERCENTILE_CONT, expanding IN for the decision sets) + GET
/dashboard/metrics/member/ceo (declared before any future member/{id} route).

The ceo_reject coordination-root audit gap the plan meant to close was already
closed by the gap-sweep (routes through admin_set_status -> agent_role='ceo'
audit); locked with a regression assertion in the existing coordination-reject
test. Real-PG tests: approval/unblock/godmode, non-ceo exclusion, empty->zeros.

* feat(metrics): audit instrumentation for escalations/blocked-others/idle (phase 4a)

The three extra per-member metrics that had no data source get durable,
in-session audit events (additive; never gate the underlying action):
- apply_escalation -> task.escalated (details.escalator_slug) on both the
  normal block path and the pool-divert path -> escalations count.
- _unblock_dependents -> task.unblocked_dependents (details.count) on the
  completed BLOCKER task, captured before the dependency edges are pruned ->
  blocked-others count (sweeper attributes to the blocker's owner).
- mark_agent_idle -> agent.idle (details.agent_slug) -> idle/utilization (the
  sweeper pairs an idle mark to the member's next spawn for idle duration).
(QA pass-rate needs no new event — reuses task.awaiting_documentation[qa] +
task.qa_fail.) Real-PG tests for each; 111 transition tests still green.

* feat(metrics): member_performance_daily rollup table + migration 056 (phase 4b)

The per-member scorecard rollup: one row per (date, member_kind, agent_slug),
CEO as a first-class member_kind='ceo' row (agent_slug='' NOT NULL so the
NULL-distinct UNIQUE keeps it unique). Full column set + the four CEO-approved
extras (qa_reviews_total/passed, escalations, blocked_others, idle_seconds) plus
blocked_seconds. Overwrite-upsert on (date, member_kind, agent_slug) for an
idempotent sweep. Migration 056 verified real up/down (24 cols, 4 indexes).

* feat(metrics): _sweep_member_performance rollup sweeper (phase 4c)

The daily per-member rollup sweep (mirrors _sweep_daily_rollup): a trailing
7-day, idempotent overwrite-upsert wired into _run_sweep. One focused query per
metric merges into a (date, agent_slug) accumulator — spawn effort/turns/tokens/
cost, completed/first-pass/revisions-received, revisions-caused (qa/pr fails),
QA pass-rate (passed + total), escalations (by escalator_slug), blocked-others
(unblocked_dependents by blocker owner), idle_seconds (idle mark -> next spawn),
blocked_seconds (blocked dwell) — plus one CEO row/day (approval/unblock dwell +
god-mode). Real-PG test asserts every facet + idempotency (a 2nd sweep
overwrites, never doubles); spawn-day != completion-day split is by-design.

* feat(metrics): member/org rollup scorecards + endpoints + live overlay (phase 5)

MemberScorecard + OrgScorecard with derived rates (FPY, effort-throughput,
turns/tool-calls per task, QA pass-rate, utilization) — all division-guarded to
None. get_member_scorecard reads member_performance_daily by slug and overlays
the member's live in-flight (non-terminal) tasks' effort via get_task_metrics
(disjoint by status: completion counts stay rollup-only, overlay only enriches
effort/turns/cost; includes_live_inflight flags it). get_org_scorecard
aggregates the cell (?team=) or whole org. Routes: GET /metrics/member/{agent_id}
(404 if absent, after the ceo literal route) + GET /metrics/org?team=. Real-PG
tests (derived rates, overlay no double-count, guards, org) + route tests.

* feat(metrics): granular CEO completion notification (phase 6)

There was no CEO completion notification at all (EventType.TASK_COMPLETED was
defined but never emitted). Add notify_ceo_of_completion in
NotificationDeliveryService — a granular body (real effort vs wall-clock +
stints/turns/tool-calls/revisions[QA/PR]/cost from get_task_metrics; degrades to
wall-clock-only, turns 'n/a', when there are no spawn sessions). Reuses the
existing ALERT type (no enum migration; the notificationtype PG enum is fixed at
001). ceo_approve now emits TASK_COMPLETED + fires the notification (best-effort
via _notify_completion — never blocks completion); complete() emits
TASK_COMPLETED too (closes the dead-code gap; the WS bridge can forward it).
Pure formatter tests + real-PG notification test.

* [metrics-granularity] Phase 7: panel Scorecards tab + dashboard overview

Add the CEO-facing metrics surfaces for the granularity feature:

- New "Scorecards" tab on the Metrics page: org rollup headline, the
  CEO-as-member card (approval/unblock dwell + god-mode count), and a
  per-member table (completed, first-pass yield, active effort, turns/task,
  QA pass-rate, escalations, blocked-others, utilization). Each member row
  self-fetches its rollup scorecard; live in-flight rows carry a "live" badge.
- New dashboard overview card (ScorecardOverviewPanel): org-wide 30-day
  headline (completed, FPY, throughput/hr, active effort, cost) deep-linking
  into the Scorecards tab.
- Plumbing: TaskMetrics/MemberScorecard/OrgScorecard/CeoScorecard types,
  observability API client methods + empty fallbacks, and the four
  useCeoScorecard/useMemberScorecard/useOrgScorecard/useTaskMetrics hooks.

Panel gate green: tsc, eslint, prettier, vitest (175 tests, +6 new).

* [metrics-granularity] test: make completion-notification robust to shared-DB CEO

test_notify_ceo_of_completion_creates_alert errored in the full suite (passed
in isolation): the session-scoped test DB is shared across the run, and the
sibling real-DB board-gate test commits a role=CEO agent (slug="ceo") without
cleanup — so my env fixture's hardcoded slug="ceo" insert hit a unique-constraint
violation, and a second role=CEO row would also make _get_ceo_agent()'s
scalar_one_or_none() raise. Reuse an existing CEO when present (the singleton the
production system actually has), else create one with a unique slug. Order-
independent. Also reflow test_metrics_instrumentation.py to ruff format.

* chore(release): 0.15.0

Metrics granularity: per-member/per-task/org + CEO-as-member scorecards,
turn/tool-call capture (migration 055), member_performance_daily rollup
(migration 056) with QA pass-rate / escalations / blocked-others / utilization,
per-task active-vs-wait metrics, granular completion notification, panel
Scorecards tab + dashboard Performance card, and the ceo_reject audit fix.

Version bump across the canonical set + CHANGELOG.

* [metrics-granularity] fix pre-tag audit findings (overlay double-count + panel error states)

Adversarial review before the v0.15.0 tag surfaced two real logical gaps:

- MAJOR (backend): the live in-flight overlay re-summed ALL sessions of every
  non-terminal task via get_task_metrics, but _msweep_spawn already rolls up
  every CLOSED session regardless of task status — so a closed session on a
  still-open task was counted twice (rollup + overlay), permanently inflating a
  member's effort/turns/tokens/cost on the common reap/respawn path. The overlay
  now sums only OPEN sessions (ended_at IS NULL), which the closed-only rollup
  can never contain — disjoint by construction. A just-closed session lands in
  the rollup on the next ~60s sweep (no gap of note). Aggregated in SQL to mirror
  _msweep_spawn. Regression test reproduces the double-count (turns 10→5).

- MAJOR (panel): the four new scorecard surfaces used `isLoading || !data` with
  no isError branch, so a failed query span forever on a skeleton. They now
  surface a load error. Tests added.

Also: OrgSummary active-effort formatting no longer round-trips hours→seconds→
hours; dashboard grid uses xl:grid-cols-4 (was 2xl) so 4 panels show at 1280px;
corrected the inaccurate "NULL distinct" CEO-row uniqueness comment (agent_slug
is NOT NULL; the '' tuple is simply distinct from agent rows).

make quality GREEN (cov 95.31%); panel GREEN (vitest 178).

* [metrics-granularity] fix: decode bytes stream message-id before XCLAIM

StreamEventBus._recover_stream passed the pending message id to XCLAIM via
str() on the raw bytes the client returns (redis client has no
decode_responses), producing "b'1782066556728-0'". Redis rejects that with
"Unrecognized XCLAIM option", so pending-message recovery threw on every
reclaim tick and unacked messages from crashed/slow consumers were never
reclaimed (leaking in the PEL on every stream, spamming the error log). Decode
via the existing _to_str helper — the fix the sibling claim path already uses.

Pre-existing in v0.14.0 (unrelated to metrics granularity); folded into this
release per CEO. TDD regression test + CHANGELOG entry. make quality GREEN.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
2026-07-01 05:18:45 +02:00
committed by GitHub
co-authored by Renn F
parent 1e341c766e
commit a8cb2470ba
47 changed files with 4225 additions and 40 deletions
+13
View File
@@ -6,6 +6,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
## [Unreleased]
## [0.15.0] - 2026-07-01
### Added
- **Metrics granularity — the company is now measurable per member, per task, and as a whole, with the CEO measured as a member too.** Every agent spawn session now captures its operational shape, not just its token/cost total: LLM iterations (`turns`) and tool invocations (`tool_calls`) are parsed from the transcript, exposed over the SDK `/usage/status` + `/usage/sync`, and persisted on `agent_spawn_sessions` (migration 055) alongside the existing 4 token dimensions. A new per-task metrics endpoint (`GET /dashboard/metrics/task/{id}`) decomposes a task's lifetime into **active vs. wait** effort — merged-overlap active-runtime across every spawn stint mapped onto the audit-log stage windows (the pure `compute_stage_effort` helper) — so a slow task can be read as "the work was hard" vs. "it sat in a queue." A nightly rollup table `member_performance_daily` (migration 056) + orchestrator sweeper aggregate each member's day from data already captured: tasks completed, first-pass yield, active-effort throughput/hr, turns & tool-calls per task, revisions caused/received, **QA pass-rate, escalations raised, times-this-member-blocked-others, and idle/utilization** (idle paired from `agent.idle` audit marks to the next spawn), plus derived ratios. Member / team / org scorecards read the rollup with a **live in-flight overlay** (today's not-yet-rolled sessions are folded in so the numbers aren't a day stale), served by `GET /dashboard/metrics/member/{id}`, `/dashboard/metrics/member/ceo`, and `/dashboard/metrics/org?team=`. The human CEO is a first-class measured member: the CEO scorecard reports approval-decision and unblock latency (p50/p90 from the audit journey) and god-mode override count — reconstructed entirely from the audit log, no new hot-path writes.
- **A granular completion notification.** When a task completes, the CEO now gets a notification carrying the task's metrics breakdown (active vs. wait effort, turns, tool calls, revisions, cost) instead of a bare "done" — the completion signal doubles as a per-task scorecard.
- **Panel: a Scorecards tab and a dashboard Performance card.** The Metrics page gains a **Scorecards** tab — an org rollup headline, the CEO-as-member card (approval/unblock dwell + god-mode count), and a per-member table (completed, first-pass yield, active effort, turns/task, QA pass-rate, escalations, blocked-others, utilization) where each row self-fetches its rollup and live in-flight rows carry a "live" badge. The dashboard gains a **Performance** overview card (org-wide 30-day completed / first-pass-yield / throughput-per-hour / active-effort / cost) that deep-links into the Scorecards tab.
### Fixed
- **`ceo_reject` now leaves an audit trail.** The CEO's reject-with-changes and cancel decisions emitted no named audit event, so rework and decision-latency attribution had a hole at the CEO chokepoint. Both now emit the transition audit event at the single `_emit_status_transition_audit` chokepoint, so the CEO's decisions are reconstructable alongside every other role's.
- **Stream pending-message recovery no longer fails on every reclaim tick.** `StreamEventBus._recover_stream` decoded a Redis message id with `str()` on the raw bytes the client returns (no `decode_responses`), producing `"b'…-0'"` — which Redis rejects with "Unrecognized XCLAIM option", so unacknowledged messages from crashed/slow consumers were never reclaimed and leaked in the pending-entries list on every stream (`roboco:stream:usage` and others), spamming the error log each interval. It now decodes the id via the existing `_to_str` helper before XCLAIM.
## [0.14.0] - 2026-06-29
### Added
+1 -1
View File
@@ -129,7 +129,7 @@ Choose the registry and version with two env vars (defaults shown):
```bash
ROBOCO_REGISTRY=ghcr.io/rennf93 # or docker.io/renzof93
ROBOCO_VERSION=latest # or a pinned release, e.g. 0.14.0
ROBOCO_VERSION=latest # or a pinned release, e.g. 0.15.0
```
The orchestrator spawns the matching pre-built agent images on demand — no build toolchain or source compile on your host.
@@ -0,0 +1,39 @@
"""Add ``turns`` + ``tool_calls`` to agent_spawn_sessions.
Per-stint LLM iterations (unique assistant messages) and tool invocations,
captured at session finalize from the SDK ``/usage/status`` (``turns`` also has
a Claude-transcript fallback). They power the granular per-member performance
metrics — distinguishing real effort/iterations from wall-clock. ``DEFAULT 0``
so historical rows (and Grok agents, which have no Claude transcript) read 0,
surfaced as "n/a" in the UI rather than a misleading "0 iterations".
Revision ID: 055_spawn_session_turns
Revises: 054_a2a_message_skill
Create Date: 2026-07-01
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
revision = "055_spawn_session_turns"
down_revision = "054_a2a_message_skill"
branch_labels: dict[str, str] | None = None
depends_on: dict[str, str] | None = None
def upgrade() -> None:
op.add_column(
"agent_spawn_sessions",
sa.Column("turns", sa.BigInteger(), nullable=False, server_default="0"),
)
op.add_column(
"agent_spawn_sessions",
sa.Column("tool_calls", sa.BigInteger(), nullable=False, server_default="0"),
)
def downgrade() -> None:
op.drop_column("agent_spawn_sessions", "tool_calls")
op.drop_column("agent_spawn_sessions", "turns")
+84
View File
@@ -0,0 +1,84 @@
"""Add member_performance_daily — the granular per-member scorecard rollup.
One row per (date, member_kind, agent_slug) populated by the orchestrator's
sweeper from data already captured (agent_spawn_sessions + audit_log). Serves
the per-member / team / org scorecards without re-scanning raw task lists. The
CEO is a first-class ``member_kind='ceo'`` row (agent_slug='' — a distinct
natural-key tuple from every ``member_kind='agent'`` row, so it never collides).
Includes the four CEO-approved extra metrics (qa pass-rate counts, escalations,
blocked-others, idle_seconds) plus blocked_seconds. All columns DEFAULT 0 so a
fresh/partial row reads as zeros, never NULL.
Revision ID: 056_member_perf_daily
Revises: 055_spawn_session_turns
Create Date: 2026-07-01
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
revision = "056_member_perf_daily"
down_revision = "055_spawn_session_turns"
branch_labels: dict[str, str] | None = None
depends_on: dict[str, str] | None = None
def _int(name: str) -> sa.Column:
return sa.Column(name, sa.Integer(), nullable=False, server_default="0")
def _big(name: str) -> sa.Column:
return sa.Column(name, sa.BigInteger(), nullable=False, server_default="0")
def upgrade() -> None:
op.create_table(
"member_performance_daily",
sa.Column("id", sa.UUID(as_uuid=True), primary_key=True, nullable=False),
sa.Column("date", sa.Date(), nullable=False),
sa.Column("member_kind", sa.String(length=16), nullable=False),
sa.Column(
"agent_slug", sa.String(length=100), nullable=False, server_default=""
),
sa.Column("team", sa.String(length=50), nullable=True),
sa.Column("role", sa.String(length=50), nullable=True),
_int("tasks_completed"),
_int("tasks_first_pass"),
_int("revisions_caused"),
_int("revisions_received"),
_big("active_runtime_seconds"),
_int("turns"),
_int("tool_calls"),
_big("tokens"),
sa.Column("cost_usd", sa.Float(), nullable=False, server_default="0"),
_big("ceo_approval_dwell_seconds"),
_big("ceo_unblock_dwell_seconds"),
_int("godmode_actions"),
# The four CEO-approved extras (+ blocked_seconds).
_int("qa_reviews_total"),
_int("qa_reviews_passed"),
_int("escalations"),
_int("blocked_others"),
_big("idle_seconds"),
_big("blocked_seconds"),
sa.UniqueConstraint(
"date", "member_kind", "agent_slug", name="uq_member_perf_day"
),
)
op.create_index("ix_member_perf_date", "member_performance_daily", ["date"])
op.create_index(
"ix_member_perf_agent_slug", "member_performance_daily", ["agent_slug"]
)
op.create_index("ix_member_perf_team", "member_performance_daily", ["team"])
op.create_index("ix_member_perf_kind", "member_performance_daily", ["member_kind"])
def downgrade() -> None:
op.drop_index("ix_member_perf_kind", table_name="member_performance_daily")
op.drop_index("ix_member_perf_team", table_name="member_performance_daily")
op.drop_index("ix_member_perf_agent_slug", table_name="member_performance_daily")
op.drop_index("ix_member_perf_date", table_name="member_performance_daily")
op.drop_table("member_performance_daily")
+1 -1
View File
@@ -30,7 +30,7 @@ Two variables choose what you pull (defaults shown):
```bash
ROBOCO_REGISTRY=ghcr.io/rennf93 # or docker.io/renzof93
ROBOCO_VERSION=latest # or a pinned release, e.g. 0.14.0
ROBOCO_VERSION=latest # or a pinned release, e.g. 0.15.0
```
The orchestrator then spawns the **matching** pre-built agent images on demand (it reads `ROBOCO_AGENT_IMAGE_REGISTRY` / `ROBOCO_AGENT_IMAGE_TAG`, which the registry compose wires to the same registry and version). Pin `ROBOCO_VERSION` to a release tag in production so an upstream `latest` push can't silently change your fleet.
+1 -1
View File
@@ -25,7 +25,7 @@ A feature flag set in `.env` takes effect on the next backend restart. The env-g
| Variable | Default | Purpose |
|----------|---------|---------|
| `ROBOCO_APP_VERSION` | `0.14.0` | Reported app version. |
| `ROBOCO_APP_VERSION` | `0.15.0` | Reported app version. |
| `ROBOCO_DEBUG` | `false` | Debug mode. |
| `ROBOCO_ENVIRONMENT` | `development` | One of `development` / `staging` / `production`. Selects the JSON log renderer (prod) vs console renderer. The compose stack sets `production`. |
| `ROBOCO_HOST` | `127.0.0.1` | Bind address. Use `0.0.0.0` in containers. |
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "roboco-panel",
"version": "0.14.0",
"version": "0.15.0",
"private": true,
"packageManager": "pnpm@10.25.0",
"scripts": {
+8 -1
View File
@@ -21,6 +21,7 @@ import { Skeleton } from "@/components/ui/skeleton";
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs";
import { OfflineState } from "@/components/ui/offline-state";
import { DeliveryTabContent } from "@/components/metrics/delivery-tab";
import { ScorecardsTabContent } from "@/components/metrics/scorecards-tab";
import {
UsageTimeSeriesChart,
ModelUsageDonut,
@@ -602,12 +603,13 @@ function CacheEfficiencyCard({
// ─── Tab types ────────────────────────────────────────────────────────────────
type MetricsTab = "performance" | "token-usage" | "delivery";
type MetricsTab = "performance" | "token-usage" | "delivery" | "scorecards";
const VALID_METRICS_TABS: MetricsTab[] = [
"performance",
"token-usage",
"delivery",
"scorecards",
];
function isValidMetricsTab(value: string | null): value is MetricsTab {
@@ -649,6 +651,7 @@ function MetricsPageContent() {
<TabsTrigger value="performance">Performance</TabsTrigger>
<TabsTrigger value="token-usage">Token Usage</TabsTrigger>
<TabsTrigger value="delivery">Delivery</TabsTrigger>
<TabsTrigger value="scorecards">Scorecards</TabsTrigger>
</TabsList>
<TabsContent value="performance" className="mt-6">
@@ -662,6 +665,10 @@ function MetricsPageContent() {
<TabsContent value="delivery" className="mt-6">
<DeliveryTabContent />
</TabsContent>
<TabsContent value="scorecards" className="mt-6">
<ScorecardsTabContent />
</TabsContent>
</Tabs>
</div>
);
@@ -0,0 +1,65 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import { render, screen } from "@testing-library/react";
const { mockOrg } = vi.hoisted(() => ({ mockOrg: vi.fn() }));
vi.mock("@/hooks/use-observability", () => ({
useOrgScorecard: mockOrg,
}));
import { ScorecardOverviewPanel } from "../scorecard-overview-panel";
describe("ScorecardOverviewPanel", () => {
beforeEach(() => {
mockOrg.mockReturnValue({
data: {
scope: "org",
team: null,
member_count: 3,
tasks_completed: 42,
first_pass_yield: 0.75,
effort_throughput_per_hour: 1.5,
active_runtime_hours: 12.3,
turns: 0,
tool_calls: 0,
tokens: 0,
cost_usd: 9.5,
revisions_caused: 0,
revisions_received: 0,
},
isLoading: false,
});
});
it("renders headline org figures", () => {
render(<ScorecardOverviewPanel />);
expect(screen.getByText("42")).toBeInTheDocument();
expect(screen.getByText("75%")).toBeInTheDocument();
expect(screen.getByText("$9.50")).toBeInTheDocument();
});
it("deep-links into the Scorecards metrics tab", () => {
render(<ScorecardOverviewPanel />);
const link = screen.getByRole("link", { name: /scorecards/i });
expect(link).toHaveAttribute("href", "/metrics?tab=scorecards");
});
it("shows a skeleton while loading", () => {
mockOrg.mockReturnValue({ data: undefined, isLoading: true });
const { container } = render(<ScorecardOverviewPanel />);
expect(container.querySelectorAll('[data-slot="skeleton"]').length).toBe(5);
});
it("surfaces an error instead of an endless skeleton", () => {
mockOrg.mockReturnValue({
data: undefined,
isLoading: false,
isError: true,
});
const { container } = render(<ScorecardOverviewPanel />);
expect(
screen.getByText(/failed to load performance metrics/i),
).toBeInTheDocument();
expect(container.querySelectorAll('[data-slot="skeleton"]').length).toBe(0);
});
});
@@ -20,6 +20,7 @@ import { StrategySignalsPanel } from "./strategy-signals-panel";
import type { Activity } from "./activity-item";
import { Button } from "@/components/ui/button";
import { UsageOverviewPanel } from "./usage-overview-panel";
import { ScorecardOverviewPanel } from "./scorecard-overview-panel";
import { RefreshCw, Settings, AlertCircle } from "lucide-react";
import Link from "next/link";
@@ -123,14 +124,15 @@ export function CommandCenter() {
{/* Playbook review queue (hidden when no drafts) */}
<PlaybookReviewQueue />
{/* Metrics, Alerts, and Usage Row */}
<div className="grid grid-cols-1 lg:grid-cols-3 xl:grid-cols-3 2xl:grid-cols-3 gap-6">
{/* Metrics, Alerts, Usage, and Performance Row */}
<div className="grid grid-cols-1 lg:grid-cols-2 xl:grid-cols-4 gap-6">
<KeyMetricsPanel
metrics={overview?.key_metrics}
isLoading={loadingOverview}
/>
<AuditorAlertsPanel alerts={flags} isLoading={loadingFlags} />
<UsageOverviewPanel />
<ScorecardOverviewPanel />
</div>
{/* Blockers and Activity Row */}
+1
View File
@@ -14,3 +14,4 @@ export { ReleaseProposalCard } from "./release-proposal-card";
export { PlaybookReviewQueue } from "./playbook-review-queue";
export { StrategySignalsPanel } from "./strategy-signals-panel";
export { UsageOverviewPanel } from "./usage-overview-panel";
export { ScorecardOverviewPanel } from "./scorecard-overview-panel";
@@ -0,0 +1,109 @@
"use client";
import Link from "next/link";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Skeleton } from "@/components/ui/skeleton";
import { useOrgScorecard } from "@/hooks/use-observability";
import {
Trophy,
CheckCircle2,
Gauge,
Clock,
Coins,
ArrowRight,
} from "lucide-react";
function pctOrNa(rate: number | null): string {
return rate === null ? "n/a" : (rate * 100).toFixed(0) + "%";
}
function numOrNa(value: number | null, digits = 2): string {
return value === null ? "n/a" : value.toFixed(digits);
}
interface MetricRowProps {
icon: React.ReactNode;
label: string;
value: string;
}
function MetricRow({ icon, label, value }: MetricRowProps) {
return (
<div className="flex items-center justify-between py-1">
<div className="text-muted-foreground flex items-center gap-2 text-sm">
{icon}
{label}
</div>
<span className="text-sm font-semibold">{value}</span>
</div>
);
}
/**
* Dashboard overview of the org-wide performance rollup (last 30 days). Headline
* figures from useOrgScorecard with a deep-link into the full Scorecards tab.
*/
export function ScorecardOverviewPanel() {
const { data, isLoading, isError } = useOrgScorecard();
return (
<Card>
<CardHeader className="pb-3">
<div className="flex items-center justify-between">
<CardTitle className="flex items-center gap-2 text-lg">
<Trophy className="h-5 w-5" />
Performance
</CardTitle>
<Link
href="/metrics?tab=scorecards"
className="text-muted-foreground hover:text-foreground flex items-center gap-1 text-xs"
>
Scorecards
<ArrowRight className="h-3 w-3" />
</Link>
</div>
</CardHeader>
<CardContent>
{isError ? (
<div className="text-muted-foreground text-sm">
Failed to load performance metrics.
</div>
) : isLoading || !data ? (
<div className="space-y-3">
{Array.from({ length: 5 }).map((_, i) => (
<Skeleton key={i} className="h-6" />
))}
</div>
) : (
<div className="divide-y">
<MetricRow
icon={<CheckCircle2 className="h-4 w-4" />}
label="Tasks completed (30d)"
value={String(data.tasks_completed)}
/>
<MetricRow
icon={<Gauge className="h-4 w-4" />}
label="First-pass yield"
value={pctOrNa(data.first_pass_yield)}
/>
<MetricRow
icon={<Gauge className="h-4 w-4 text-blue-500" />}
label="Throughput / hr"
value={numOrNa(data.effort_throughput_per_hour)}
/>
<MetricRow
icon={<Clock className="h-4 w-4" />}
label="Active effort"
value={data.active_runtime_hours.toFixed(1) + "h"}
/>
<MetricRow
icon={<Coins className="h-4 w-4" />}
label="Cost"
value={"$" + data.cost_usd.toFixed(2)}
/>
</div>
)}
</CardContent>
</Card>
);
}
@@ -0,0 +1,161 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import { render, screen } from "@testing-library/react";
import { AgentRole, AgentState, type Agent } from "@/types";
import type { MemberScorecard } from "@/types";
const { mockOrg, mockCeo, mockMember, mockAgents } = vi.hoisted(() => ({
mockOrg: vi.fn(),
mockCeo: vi.fn(),
mockMember: vi.fn(),
mockAgents: vi.fn(),
}));
vi.mock("@/hooks/use-observability", () => ({
useOrgScorecard: mockOrg,
useCeoScorecard: mockCeo,
useMemberScorecard: mockMember,
}));
vi.mock("@/hooks/use-agents", () => ({
useAgents: mockAgents,
}));
import { ScorecardsTabContent } from "../scorecards-tab";
function agent(id: string, name: string, role: AgentRole): Agent {
return {
id,
agent_id: id,
name,
role,
team: null,
cell: null,
status: AgentState.IDLE,
};
}
function member(over: Partial<MemberScorecard>): MemberScorecard {
return {
scope: "member",
id: "a1",
name: "a1",
member_kind: "agent",
tasks_completed: 0,
first_pass_yield: null,
effort_throughput_per_hour: null,
active_runtime_hours: 0,
turns: 0,
tool_calls: 0,
tokens: 0,
cost_usd: 0,
turns_per_task: null,
tool_calls_per_task: null,
revisions_caused: 0,
revisions_received: 0,
qa_pass_rate: null,
escalations: 0,
blocked_others: 0,
idle_hours: 0,
utilization: null,
includes_live_inflight: false,
...over,
};
}
describe("ScorecardsTabContent", () => {
beforeEach(() => {
mockOrg.mockReturnValue({
data: {
scope: "org",
team: null,
member_count: 3,
tasks_completed: 42,
first_pass_yield: 0.75,
effort_throughput_per_hour: 1.5,
active_runtime_hours: 12.3,
turns: 0,
tool_calls: 0,
tokens: 0,
cost_usd: 9.5,
revisions_caused: 0,
revisions_received: 0,
},
isLoading: false,
});
mockCeo.mockReturnValue({
data: {
member_kind: "ceo",
approval_p50_seconds: 3600,
approval_p90_seconds: 7200,
approval_count: 5,
unblock_p50_seconds: 1800,
unblock_count: 2,
godmode_actions: 1,
},
isLoading: false,
});
mockMember.mockReturnValue({
data: member({
id: "dev1",
name: "be-dev-1",
tasks_completed: 7,
first_pass_yield: 0.8,
active_runtime_hours: 4.2,
qa_pass_rate: 0.9,
escalations: 1,
blocked_others: 2,
utilization: 0.6,
}),
isLoading: false,
});
mockAgents.mockReturnValue({
data: [
agent("dev1", "be-dev-1", AgentRole.DEVELOPER),
agent("ceo", "Renzo", AgentRole.CEO),
agent("sys", "system", AgentRole.SYSTEM),
],
});
});
it("renders the org rollup headline figures", () => {
render(<ScorecardsTabContent />);
expect(screen.getByText("42")).toBeInTheDocument(); // tasks_completed
expect(screen.getByText("75%")).toBeInTheDocument(); // first-pass yield
expect(screen.getByText("$9.50")).toBeInTheDocument(); // cost
});
it("renders the CEO approval/unblock figures", () => {
render(<ScorecardsTabContent />);
expect(screen.getByText("Approvals")).toBeInTheDocument();
expect(screen.getByText("5")).toBeInTheDocument(); // approval_count
expect(screen.getByText("God-mode actions")).toBeInTheDocument();
});
it("lists only non-CEO/non-system members in the table", () => {
render(<ScorecardsTabContent />);
expect(screen.getByText("be-dev-1")).toBeInTheDocument();
// CEO and system are excluded from the member table.
expect(screen.queryByText("Renzo")).not.toBeInTheDocument();
expect(screen.queryByText("system")).not.toBeInTheDocument();
});
it("surfaces load errors instead of an endless skeleton", () => {
mockOrg.mockReturnValue({
data: undefined,
isLoading: false,
isError: true,
});
mockMember.mockReturnValue({
data: undefined,
isLoading: false,
isError: true,
});
render(<ScorecardsTabContent />);
expect(
screen.getByText(/failed to load organization metrics/i),
).toBeInTheDocument();
// The member row shows a failed marker rather than a perpetual skeleton
// (exact lowercase text, distinct from the org card's message).
expect(screen.getByText("failed to load")).toBeInTheDocument();
});
});
@@ -0,0 +1,191 @@
"use client";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Skeleton } from "@/components/ui/skeleton";
import { Badge } from "@/components/ui/badge";
import {
Table,
TableBody,
TableCell,
TableHead,
TableHeader,
TableRow,
} from "@/components/ui/table";
import {
useCeoScorecard,
useMemberScorecard,
useOrgScorecard,
} from "@/hooks/use-observability";
import { useAgents } from "@/hooks/use-agents";
import { AgentRole, type Agent } from "@/types";
function pctOrNa(rate: number | null): string {
return rate === null ? "n/a" : (rate * 100).toFixed(0) + "%";
}
function numOrNa(value: number | null, digits = 1): string {
return value === null ? "n/a" : value.toFixed(digits);
}
function hoursOrDash(seconds: number): string {
return (seconds / 3600).toFixed(1) + "h";
}
/** One member's row — each row self-fetches its rollup scorecard. */
function MemberRow({ agent }: { agent: Agent }) {
const { data, isLoading, isError } = useMemberScorecard(agent.id);
if (isError) {
return (
<TableRow>
<TableCell>{agent.name || agent.slug}</TableCell>
<TableCell colSpan={8} className="text-muted-foreground text-xs">
failed to load
</TableCell>
</TableRow>
);
}
if (isLoading || !data) {
return (
<TableRow>
<TableCell>{agent.name || agent.slug}</TableCell>
<TableCell colSpan={8}>
<Skeleton className="h-4 w-full" />
</TableCell>
</TableRow>
);
}
return (
<TableRow>
<TableCell className="font-medium">
{agent.name || agent.slug}
{data.includes_live_inflight && (
<Badge variant="outline" className="ml-2 text-xs">
live
</Badge>
)}
</TableCell>
<TableCell>{data.tasks_completed}</TableCell>
<TableCell>{pctOrNa(data.first_pass_yield)}</TableCell>
<TableCell>{data.active_runtime_hours.toFixed(1)}h</TableCell>
<TableCell>{numOrNa(data.turns_per_task)}</TableCell>
<TableCell>{pctOrNa(data.qa_pass_rate)}</TableCell>
<TableCell>{data.escalations}</TableCell>
<TableCell>{data.blocked_others}</TableCell>
<TableCell>{pctOrNa(data.utilization)}</TableCell>
</TableRow>
);
}
function OrgSummary() {
const { data, isLoading, isError } = useOrgScorecard();
if (isError)
return (
<div className="text-muted-foreground text-sm">
Failed to load organization metrics.
</div>
);
if (isLoading || !data) return <Skeleton className="h-24 w-full" />;
const cells: [string, string][] = [
["Members", String(data.member_count)],
["Completed", String(data.tasks_completed)],
["First-pass yield", pctOrNa(data.first_pass_yield)],
["Throughput/hr", numOrNa(data.effort_throughput_per_hour, 2)],
["Active effort", data.active_runtime_hours.toFixed(1) + "h"],
["Cost", "$" + data.cost_usd.toFixed(2)],
];
return (
<div className="grid grid-cols-2 gap-4 sm:grid-cols-3 lg:grid-cols-6">
{cells.map(([k, v]) => (
<div key={k}>
<div className="text-2xl font-semibold">{v}</div>
<div className="text-muted-foreground text-sm">{k}</div>
</div>
))}
</div>
);
}
function CeoCard() {
const { data, isLoading, isError } = useCeoScorecard();
if (isError)
return (
<div className="text-muted-foreground text-sm">
Failed to load CEO metrics.
</div>
);
if (isLoading || !data) return <Skeleton className="h-24 w-full" />;
const cells: [string, string][] = [
["Approvals", String(data.approval_count)],
["Approval p50", hoursOrDash(data.approval_p50_seconds)],
["Approval p90", hoursOrDash(data.approval_p90_seconds)],
["Unblocks", String(data.unblock_count)],
["Unblock p50", hoursOrDash(data.unblock_p50_seconds)],
["God-mode actions", String(data.godmode_actions)],
];
return (
<div className="grid grid-cols-2 gap-4 sm:grid-cols-3 lg:grid-cols-6">
{cells.map(([k, v]) => (
<div key={k}>
<div className="text-2xl font-semibold">{v}</div>
<div className="text-muted-foreground text-sm">{k}</div>
</div>
))}
</div>
);
}
export function ScorecardsTabContent() {
const { data: agents } = useAgents();
const members = (agents ?? []).filter(
(a) => a.role !== AgentRole.CEO && a.role !== AgentRole.SYSTEM,
);
return (
<div className="space-y-6">
<Card>
<CardHeader>
<CardTitle>Organization</CardTitle>
</CardHeader>
<CardContent>
<OrgSummary />
</CardContent>
</Card>
<Card>
<CardHeader>
<CardTitle>CEO (you)</CardTitle>
</CardHeader>
<CardContent>
<CeoCard />
</CardContent>
</Card>
<Card>
<CardHeader>
<CardTitle>Members</CardTitle>
</CardHeader>
<CardContent>
<Table>
<TableHeader>
<TableRow>
<TableHead>Member</TableHead>
<TableHead>Done</TableHead>
<TableHead>FPY</TableHead>
<TableHead>Effort</TableHead>
<TableHead>Turns/task</TableHead>
<TableHead>QA pass</TableHead>
<TableHead>Escal.</TableHead>
<TableHead>Blocked others</TableHead>
<TableHead>Util.</TableHead>
</TableRow>
</TableHeader>
<TableBody>
{members.map((a) => (
<MemberRow key={a.id} agent={a} />
))}
</TableBody>
</Table>
</CardContent>
</Card>
</div>
);
}
+49
View File
@@ -7,6 +7,10 @@ import type {
BottleneckReport,
ReworkReport,
Scorecard,
CeoScorecard,
MemberScorecard,
OrgScorecard,
TaskMetrics,
} from "@/types";
// =============================================================================
@@ -23,6 +27,14 @@ export const observabilityKeys = {
[...observabilityKeys.all, "rework", days, team ?? "all"] as const,
teamScorecard: (team: string, days: number) =>
[...observabilityKeys.all, "scorecard", "team", team, days] as const,
ceoScorecard: (days: number) =>
[...observabilityKeys.all, "scorecard", "ceo", days] as const,
memberScorecard: (agentId: string, days: number) =>
[...observabilityKeys.all, "scorecard", "member", agentId, days] as const,
orgScorecard: (days: number, team?: string) =>
[...observabilityKeys.all, "scorecard", "org", team ?? "all", days] as const,
taskMetrics: (taskId: string) =>
[...observabilityKeys.all, "task-metrics", taskId] as const,
};
// =============================================================================
@@ -64,3 +76,40 @@ export function useTeamScorecard(team: string, days = 7) {
refetchInterval: 60_000,
});
}
/** The human CEO as a measured member (approval/unblock dwell + god-mode). */
export function useCeoScorecard(days = 30) {
return useQuery<CeoScorecard>({
queryKey: observabilityKeys.ceoScorecard(days),
queryFn: () => observabilityApi.getCeoScorecard(days),
refetchInterval: 60_000,
});
}
/** Per-member rollup scorecard (+ live in-flight overlay). */
export function useMemberScorecard(agentId: string, days = 30) {
return useQuery<MemberScorecard>({
queryKey: observabilityKeys.memberScorecard(agentId, days),
queryFn: () => observabilityApi.getMemberScorecard(agentId, days),
enabled: Boolean(agentId),
refetchInterval: 60_000,
});
}
/** Org-wide (or per-cell) rollup aggregate. */
export function useOrgScorecard(days = 30, team?: string) {
return useQuery<OrgScorecard>({
queryKey: observabilityKeys.orgScorecard(days, team),
queryFn: () => observabilityApi.getOrgScorecard(days, team),
refetchInterval: 60_000,
});
}
/** Granular per-task metrics (active-vs-wait drill-down). */
export function useTaskMetrics(taskId: string) {
return useQuery<TaskMetrics | null>({
queryKey: observabilityKeys.taskMetrics(taskId),
queryFn: () => observabilityApi.getTaskMetrics(taskId),
enabled: Boolean(taskId),
});
}
+103
View File
@@ -5,6 +5,10 @@ import type {
BottleneckReport,
ReworkReport,
Scorecard,
CeoScorecard,
MemberScorecard,
OrgScorecard,
TaskMetrics,
} from "@/types";
// =============================================================================
@@ -39,6 +43,61 @@ function emptyScorecard(scope: string, id: string): Scorecard {
};
}
const EMPTY_CEO: CeoScorecard = {
member_kind: "ceo",
approval_p50_seconds: 0,
approval_p90_seconds: 0,
approval_count: 0,
unblock_p50_seconds: 0,
unblock_count: 0,
godmode_actions: 0,
};
function emptyMember(id: string): MemberScorecard {
return {
scope: "member",
id,
name: id,
member_kind: "agent",
tasks_completed: 0,
first_pass_yield: null,
effort_throughput_per_hour: null,
active_runtime_hours: 0,
turns: 0,
tool_calls: 0,
tokens: 0,
cost_usd: 0,
turns_per_task: null,
tool_calls_per_task: null,
revisions_caused: 0,
revisions_received: 0,
qa_pass_rate: null,
escalations: 0,
blocked_others: 0,
idle_hours: 0,
utilization: null,
includes_live_inflight: false,
};
}
function emptyOrg(team: string | null): OrgScorecard {
return {
scope: team ? "team" : "org",
team,
member_count: 0,
tasks_completed: 0,
first_pass_yield: null,
effort_throughput_per_hour: null,
active_runtime_hours: 0,
turns: 0,
tool_calls: 0,
tokens: 0,
cost_usd: 0,
revisions_caused: 0,
revisions_received: 0,
};
}
// =============================================================================
// API OBJECT
// =============================================================================
@@ -84,4 +143,48 @@ export const observabilityApi = {
);
return data;
},
/** CEO-as-member scorecard — GET /dashboard/metrics/member/ceo?days */
getCeoScorecard: async (days = 30): Promise<CeoScorecard> => {
if (isMockMode()) return EMPTY_CEO;
const { data } = await api.get<CeoScorecard>(
"/dashboard/metrics/member/ceo",
{ params: { days } },
);
return data;
},
/** Per-member rollup scorecard — GET /dashboard/metrics/member/{id}?days */
getMemberScorecard: async (
agentId: string,
days = 30,
): Promise<MemberScorecard> => {
if (isMockMode()) return emptyMember(agentId);
const { data } = await api.get<MemberScorecard>(
`/dashboard/metrics/member/${agentId}`,
{ params: { days } },
);
return data;
},
/** Org / team rollup — GET /dashboard/metrics/org?team&days */
getOrgScorecard: async (
days = 30,
team?: string,
): Promise<OrgScorecard> => {
if (isMockMode()) return emptyOrg(team ?? null);
const { data } = await api.get<OrgScorecard>("/dashboard/metrics/org", {
params: { days, ...(team ? { team } : {}) },
});
return data;
},
/** Granular per-task metrics — GET /dashboard/metrics/task/{id} */
getTaskMetrics: async (taskId: string): Promise<TaskMetrics | null> => {
if (isMockMode()) return null;
const { data } = await api.get<TaskMetrics>(
`/dashboard/metrics/task/${taskId}`,
);
return data;
},
};
+74
View File
@@ -1431,3 +1431,77 @@ export interface Scorecard {
tokens: number;
cost_usd: number;
}
// --- Granular per-member metrics (v0.15.0) ---
export interface StageEffort {
status: string;
active_seconds: number;
wait_seconds: number;
}
export interface TaskMetrics {
task_id: string;
active_runtime_seconds: number;
wall_clock_seconds: number;
turns: number;
tool_calls: number;
tokens: number;
cost_usd: number;
revision_count: number;
qa_fails: number;
pr_fails: number;
stints: number;
stages: StageEffort[];
}
export interface MemberScorecard {
scope: string;
id: string;
name: string;
member_kind: "agent";
tasks_completed: number;
first_pass_yield: number | null;
effort_throughput_per_hour: number | null;
active_runtime_hours: number;
turns: number;
tool_calls: number;
tokens: number;
cost_usd: number;
turns_per_task: number | null;
tool_calls_per_task: number | null;
revisions_caused: number;
revisions_received: number;
qa_pass_rate: number | null;
escalations: number;
blocked_others: number;
idle_hours: number;
utilization: number | null;
includes_live_inflight: boolean;
}
export interface OrgScorecard {
scope: string;
team: string | null;
member_count: number;
tasks_completed: number;
first_pass_yield: number | null;
effort_throughput_per_hour: number | null;
active_runtime_hours: number;
turns: number;
tool_calls: number;
tokens: number;
cost_usd: number;
revisions_caused: number;
revisions_received: number;
}
export interface CeoScorecard {
member_kind: "ceo";
approval_p50_seconds: number;
approval_p90_seconds: number;
approval_count: number;
unblock_p50_seconds: number;
unblock_count: number;
godmode_actions: number;
}
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "roboco"
version = "0.14.0"
version = "0.15.0"
description = "AI Agents Company - A virtual organization of AI agents functioning as a software development workforce"
authors = [
{name = "Renzo Franceschini", email = "rennf93@users.noreply.github.com"}
+1 -1
View File
@@ -5,7 +5,7 @@ A virtual organization of 25 AI agents + 1 human CEO,
designed to operate as a complete software development workforce.
"""
__version__ = "0.14.0"
__version__ = "0.15.0"
# Core exports
from roboco.config import settings
+6
View File
@@ -229,6 +229,12 @@ class TokenReportRequest(BaseModel):
class TokenUsageStatus(BaseModel):
"""Current cumulative token usage for this session (GET /usage/status)."""
turns: int = Field(
default=0, description="LLM iterations (unique assistant messages)"
)
tool_calls: int = Field(
default=0, description="Tool invocations accumulated this session"
)
tokens_input: int = Field(
default=0, description="Total input tokens accumulated this session"
)
+8 -1
View File
@@ -433,6 +433,10 @@ class _SessionState:
self.tokens_output: int = 0
self.tokens_cache_read: int = 0
self.tokens_cache_write: int = 0
# LLM iterations (unique assistant message ids), set by /usage/sync from
# the transcript. tool_calls is total_calls (above). Both surfaced on
# /usage/status so the orchestrator persists them at session finalize.
self.turns: int = 0
# (size, mtime) of the last transcript parsed for /usage/sync, so a
# re-sync of an unchanged transcript skips the re-parse.
self.transcript_fingerprint: tuple[int, float] | None = None
@@ -737,6 +741,8 @@ async def usage_status() -> TokenUsageStatus:
def _token_usage_snapshot() -> TokenUsageStatus:
return TokenUsageStatus(
turns=_state.turns,
tool_calls=_state.total_calls,
tokens_input=_state.tokens_input,
tokens_output=_state.tokens_output,
tokens_cache_read=_state.tokens_cache_read,
@@ -798,11 +804,12 @@ async def usage_sync(req: TranscriptSyncRequest) -> TokenUsageStatus:
return _token_usage_snapshot()
try:
tin, tout, tcr, tcw = _sum_transcript_usage(path)
tin, tout, tcr, tcw, turns = _sum_transcript_usage(path)
except OSError as exc:
logger.warning("Transcript usage sync failed", error=str(exc))
return _token_usage_snapshot()
_state.turns = turns
_state.tokens_input = tin
_state.tokens_output = tout
_state.tokens_cache_read = tcr
+6 -5
View File
@@ -54,16 +54,17 @@ def _line_usage(line: str) -> tuple[str | None, tuple[int, int, int, int]] | Non
return message.get("id"), deltas
def sum_transcript_usage(path: Path) -> tuple[int, int, int, int]:
"""Sum per-message token usage across a Claude Code JSONL transcript.
def sum_transcript_usage(path: Path) -> tuple[int, int, int, int, int]:
"""Sum per-message token usage + turn count across a Claude Code transcript.
Each assistant message carries a ``message.usage`` block with the token
counts for that API response; summing them yields the session total.
Messages that span several lines (same ``message.id``) are counted once
Claude Code emits one line per content block, each repeating the message's
usage, so naive summing roughly doubles the totals. Returns
``(input, output, cache_read, cache_write)``. Malformed lines are skipped
a single bad line must never lose the whole count.
``(input, output, cache_read, cache_write, turns)`` where ``turns`` is the
number of UNIQUE assistant ``message.id``s (i.e. LLM iterations). Malformed
lines are skipped a single bad line must never lose the whole count.
"""
tin = tout = tcr = tcw = 0
seen: set[str] = set()
@@ -81,4 +82,4 @@ def sum_transcript_usage(path: Path) -> tuple[int, int, int, int]:
tout += line_out
tcr += line_cr
tcw += line_cw
return tin, tout, tcr, tcw
return tin, tout, tcr, tcw, len(seen)
+58
View File
@@ -608,3 +608,61 @@ async def get_team_scorecard(
status_code=status.HTTP_404_NOT_FOUND, detail="Team scorecard unavailable"
)
return card.to_dict()
@router.get("/metrics/task/{task_id}")
async def get_task_metrics(
task_id: UUID,
db: DbSession,
) -> dict[str, Any]:
"""Granular per-task metrics: real effort vs wall-clock, turns/tool-calls,
per-stage active-vs-wait, who-caused-rework, tokens + cost (live)."""
metrics_service = get_metrics_service(db)
metrics = await metrics_service.get_task_metrics(task_id)
if metrics is None:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND, detail="Task not found"
)
return metrics.to_dict()
# NOTE: /metrics/member/ceo MUST be declared before any /metrics/member/{id}
# UUID-typed route so the literal "ceo" isn't captured as a path param.
@router.get("/metrics/member/ceo")
async def get_ceo_scorecard(
db: DbSession,
days: int = Query(default=30, ge=1, le=90),
) -> dict[str, Any]:
"""The human CEO as a measured member: approval/unblock dwell + god-mode count."""
metrics_service = get_metrics_service(db)
card = await metrics_service.get_ceo_scorecard(days=days)
return card.to_dict()
@router.get("/metrics/member/{agent_id}")
async def get_member_scorecard(
agent_id: UUID,
db: DbSession,
days: int = Query(default=30, ge=1, le=90),
) -> dict[str, Any]:
"""Per-member rollup scorecard (real effort, FPY, turns, cost, rework, the
four extras) + a live in-flight overlay. 404 if the agent doesn't exist."""
metrics_service = get_metrics_service(db)
card = await metrics_service.get_member_scorecard(agent_id, days=days)
if card is None:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND, detail="Agent not found"
)
return card.to_dict()
@router.get("/metrics/org")
async def get_org_scorecard(
db: DbSession,
team: Team | None = None,
days: int = Query(default=30, ge=1, le=90),
) -> dict[str, Any]:
"""Org-wide (or per-cell with ?team=) rollup aggregate from the member table."""
metrics_service = get_metrics_service(db)
card = await metrics_service.get_org_scorecard(team=team, days=days)
return card.to_dict()
+2 -2
View File
@@ -28,7 +28,7 @@ class Settings(BaseSettings):
# ==========================================================================
# Application
# ==========================================================================
app_version: str = "0.14.0"
app_version: str = "0.15.0"
debug: bool = False
environment: str = Field(
default="development", pattern="^(development|staging|production)$"
@@ -704,7 +704,7 @@ class Settings(BaseSettings):
agent_image_tag: str = Field(
default="",
description=(
"Tag for pre-built agent images (e.g. 'latest' or '0.14.0'). Empty "
"Tag for pre-built agent images (e.g. 'latest' or '0.15.0'). Empty "
"leaves the tag implicit (':latest'); only meaningful with "
"agent_image_registry set."
),
+63
View File
@@ -2213,6 +2213,11 @@ class AgentSpawnSessionTable(Base):
tokens_cache_write: Mapped[int] = mapped_column(
BigInteger, nullable=False, default=0
)
# LLM iterations (unique assistant messages) + tool invocations for this
# stint, captured at finalize from the SDK /usage/status (turns has a
# transcript fallback). Default 0 — historical/Grok rows read 0 ("n/a").
turns: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0)
tool_calls: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0)
exit_reason: Mapped[str | None] = mapped_column(String(100), nullable=True)
estimated_cost_usd: Mapped[float | None] = mapped_column(Float, nullable=True)
@@ -2315,6 +2320,64 @@ class DailyUsageRollupTable(Base):
)
class MemberPerformanceDailyTable(Base):
"""Pre-aggregated daily per-member performance (the granular scorecard rollup).
One row per (date, member_kind, agent_slug), populated by the orchestrator
sweeper from agent_spawn_sessions + audit_log. The CEO is a first-class
``member_kind='ceo'`` row with ``agent_slug=''`` a distinct natural-key
tuple from every ``member_kind='agent'`` row, so it never collides.
Overwrite-upsert on the natural key makes the sweep idempotent. All counters
default 0.
"""
__tablename__ = "member_performance_daily"
id: Mapped[UUID] = mapped_column(
UUID(as_uuid=True), primary_key=True, default=uuid4
)
date: Mapped[Any] = mapped_column(Date, nullable=False) # datetime.date
member_kind: Mapped[str] = mapped_column(String(16), nullable=False)
agent_slug: Mapped[str] = mapped_column(String(100), nullable=False, default="")
team: Mapped[str | None] = mapped_column(String(50), nullable=True)
role: Mapped[str | None] = mapped_column(String(50), nullable=True)
tasks_completed: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
tasks_first_pass: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
revisions_caused: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
revisions_received: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
active_runtime_seconds: Mapped[int] = mapped_column(
BigInteger, nullable=False, default=0
)
turns: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
tool_calls: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
tokens: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0)
cost_usd: Mapped[float] = mapped_column(Float, nullable=False, default=0.0)
ceo_approval_dwell_seconds: Mapped[int] = mapped_column(
BigInteger, nullable=False, default=0
)
ceo_unblock_dwell_seconds: Mapped[int] = mapped_column(
BigInteger, nullable=False, default=0
)
godmode_actions: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
# The four CEO-approved extras (+ blocked_seconds).
qa_reviews_total: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
qa_reviews_passed: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
escalations: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
blocked_others: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
idle_seconds: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0)
blocked_seconds: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0)
__table_args__ = (
UniqueConstraint(
"date", "member_kind", "agent_slug", name="uq_member_perf_day"
),
Index("ix_member_perf_date", "date"),
Index("ix_member_perf_agent_slug", "agent_slug"),
Index("ix_member_perf_team", "team"),
Index("ix_member_perf_kind", "member_kind"),
)
# =============================================================================
# PROMPTER TABLES
# =============================================================================
+4 -1
View File
@@ -526,8 +526,11 @@ class StreamEventBus:
recovered = 0
for msg in pending_details:
if int(msg["time_since_delivered"]) >= idle_time_ms:
# Decode the id: xpending_range returns bytes (no
# decode_responses); str(bytes) → "b'..'", which Redis rejects
# as an unrecognized XCLAIM option, wedging pending recovery.
recovered += await self._claim_and_handle(
stream, str(msg["message_id"]), idle_time_ms
stream, self._to_str(msg["message_id"]), idle_time_ms
)
return recovered
+103
View File
@@ -0,0 +1,103 @@
"""Per-stage active-vs-wait decomposition — pure overlap math (no DB).
A task's wall-clock lifetime is a sequence of status windows (from the audit
log). During each window some agent spawn *stints* may have been running. This
splits each window into ``active`` (wall-clock during which at least one stint
was running) and ``wait`` (the remainder queue/review idle). Overlapping
stints are merged, so ``active`` can never exceed the window length and
``active + wait == window length``.
This is the wall-clock decomposition and is deliberately distinct from *summed
effort* (Σ stint durations), which can exceed wall-clock when stints run
concurrently and is computed separately at the service layer.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from collections.abc import Sequence
from datetime import datetime
# A status window: (status, entered_at, exited_at). An open final window passes
# ``now()`` as exited_at at the call site.
StageWindow = tuple[str, "datetime", "datetime"]
# A spawn stint: (started_at, ended_at). An in-flight open stint passes
# ``now()`` as ended_at at the call site.
Stint = tuple["datetime", "datetime"]
@dataclass(frozen=True)
class StageEffort:
"""Active vs wait seconds for one status window."""
status: str
active_seconds: int
wait_seconds: int
def to_dict(self) -> dict[str, object]:
return {
"status": self.status,
"active_seconds": self.active_seconds,
"wait_seconds": self.wait_seconds,
}
def _merged_overlap_seconds(
win_start: datetime, win_end: datetime, stints: Sequence[Stint]
) -> float:
"""Wall-clock seconds inside [win_start, win_end) covered by any stint.
Stints are clipped to the window and overlapping ones merged, so concurrent
stints are counted once (never exceeds the window length).
"""
clipped: list[tuple[datetime, datetime]] = []
for stint_start, stint_end in stints:
lo = max(win_start, stint_start)
hi = min(win_end, stint_end)
if hi > lo:
clipped.append((lo, hi))
if not clipped:
return 0.0
clipped.sort()
total = 0.0
cur_lo, cur_hi = clipped[0]
for lo, hi in clipped[1:]:
if lo <= cur_hi: # overlapping/adjacent — extend the current run
cur_hi = max(cur_hi, hi)
else:
total += (cur_hi - cur_lo).total_seconds()
cur_lo, cur_hi = lo, hi
total += (cur_hi - cur_lo).total_seconds()
return total
def compute_stage_effort(
windows: Sequence[StageWindow], stints: Sequence[Stint]
) -> list[StageEffort]:
"""Decompose each status window into active vs wait seconds.
``active`` = merged wall-clock overlap of the stints with the window;
``wait`` = window length minus active (clamped >= 0). A zero/negative-length
window yields ``(0, 0)``. Seconds are rounded to whole ints for a stable
API surface.
"""
result: list[StageEffort] = []
for status, start, end in windows:
span = (end - start).total_seconds()
if span <= 0:
result.append(StageEffort(status=status, active_seconds=0, wait_seconds=0))
continue
active = _merged_overlap_seconds(start, end, stints)
active = min(active, span)
wait = max(0.0, span - active)
result.append(
StageEffort(
status=status,
active_seconds=round(active),
wait_seconds=round(wait),
)
)
return result
+173
View File
@@ -8,8 +8,16 @@ from dataclasses import dataclass
from typing import Any
from uuid import UUID
from roboco.foundation.policy.stage_effort import StageEffort
from roboco.models.base import Team
# CEO decision transitions used by the CEO scorecard (audit-log to_status
# values). Approval = any CEO exit from awaiting_ceo_approval (incl. the
# coordination-root reject that lands in `pending`); unblock = a CEO revive
# out of `blocked`.
CEO_APPROVAL_DECISIONS = ("completed", "needs_revision", "cancelled", "pending")
CEO_UNBLOCK_DECISIONS = ("in_progress", "pending")
class VelocityMetrics:
"""Velocity metrics over a time period."""
@@ -247,3 +255,168 @@ class Scorecard:
"tokens": self.tokens,
"cost_usd": round(self.cost_usd, 4),
}
@dataclass
class MemberScorecard:
"""Per-member rollup scorecard (agent) with derived rates.
Served from member_performance_daily (terminal work) optionally enriched
with a live in-flight overlay (the member's non-terminal tasks' effort so
far). Completion counts stay rollup-only the two sets are disjoint by
status so ``includes_live_inflight`` only enriches effort/turns/cost.
"""
scope: str # "member"
id: str
name: str
tasks_completed: int
first_pass_yield: float | None
effort_throughput_per_hour: float | None
active_runtime_hours: float
turns: int
tool_calls: int
tokens: int
cost_usd: float
turns_per_task: float | None
tool_calls_per_task: float | None
revisions_caused: int
revisions_received: int
qa_pass_rate: float | None
escalations: int
blocked_others: int
idle_hours: float
utilization: float | None
includes_live_inflight: bool
def to_dict(self) -> dict[str, Any]:
return {
"scope": self.scope,
"id": self.id,
"name": self.name,
"member_kind": "agent",
"tasks_completed": self.tasks_completed,
"first_pass_yield": self.first_pass_yield,
"effort_throughput_per_hour": self.effort_throughput_per_hour,
"active_runtime_hours": round(self.active_runtime_hours, 2),
"turns": self.turns,
"tool_calls": self.tool_calls,
"tokens": self.tokens,
"cost_usd": round(self.cost_usd, 4),
"turns_per_task": self.turns_per_task,
"tool_calls_per_task": self.tool_calls_per_task,
"revisions_caused": self.revisions_caused,
"revisions_received": self.revisions_received,
"qa_pass_rate": self.qa_pass_rate,
"escalations": self.escalations,
"blocked_others": self.blocked_others,
"idle_hours": round(self.idle_hours, 2),
"utilization": self.utilization,
"includes_live_inflight": self.includes_live_inflight,
}
@dataclass
class OrgScorecard:
"""Team or whole-org rollup aggregate (rollup-only, no live overlay)."""
scope: str # "org" | "team"
team: str | None
member_count: int
tasks_completed: int
first_pass_yield: float | None
effort_throughput_per_hour: float | None
active_runtime_hours: float
turns: int
tool_calls: int
tokens: int
cost_usd: float
revisions_caused: int
revisions_received: int
def to_dict(self) -> dict[str, Any]:
return {
"scope": self.scope,
"team": self.team,
"member_count": self.member_count,
"tasks_completed": self.tasks_completed,
"first_pass_yield": self.first_pass_yield,
"effort_throughput_per_hour": self.effort_throughput_per_hour,
"active_runtime_hours": round(self.active_runtime_hours, 2),
"turns": self.turns,
"tool_calls": self.tool_calls,
"tokens": self.tokens,
"cost_usd": round(self.cost_usd, 4),
"revisions_caused": self.revisions_caused,
"revisions_received": self.revisions_received,
}
@dataclass
class CeoScorecard:
"""The human CEO as a measured member — read purely from the audit_log.
The CEO never runs an LLM, so token/cost/turns are n/a. Its metrics are
approval dwell (awaiting_ceo_approval -> a CEO decision), unblock dwell
(blocked -> a CEO revive), and god-mode action count (every ceo-attributed
transition in the window).
"""
approval_p50_seconds: float
approval_p90_seconds: float
approval_count: int
unblock_p50_seconds: float
unblock_count: int
godmode_actions: int
def to_dict(self) -> dict[str, Any]:
return {
"member_kind": "ceo",
"approval_p50_seconds": round(self.approval_p50_seconds, 2),
"approval_p90_seconds": round(self.approval_p90_seconds, 2),
"approval_count": self.approval_count,
"unblock_p50_seconds": round(self.unblock_p50_seconds, 2),
"unblock_count": self.unblock_count,
"godmode_actions": self.godmode_actions,
}
@dataclass
class TaskMetrics:
"""Granular per-task effort: real runtime vs wall-clock, turns, cost, rework.
``active_runtime_seconds`` is summed spawn-stint effort (can exceed
wall-clock when stints run concurrently); ``stages`` is the wall-clock
active-vs-wait decomposition per status window (active clamped to the
window). Computed live from ``agent_spawn_sessions`` (by task_id) joined
with ``audit_log`` (by target_id).
"""
task_id: str
active_runtime_seconds: int
wall_clock_seconds: int
turns: int
tool_calls: int
tokens: int
cost_usd: float
revision_count: int
qa_fails: int
pr_fails: int
stints: int
stages: list[StageEffort]
def to_dict(self) -> dict[str, Any]:
return {
"task_id": self.task_id,
"active_runtime_seconds": self.active_runtime_seconds,
"wall_clock_seconds": self.wall_clock_seconds,
"turns": self.turns,
"tool_calls": self.tool_calls,
"tokens": self.tokens,
"cost_usd": round(self.cost_usd, 4),
"revision_count": self.revision_count,
"qa_fails": self.qa_fails,
"pr_fails": self.pr_fails,
"stints": self.stints,
"stages": [s.to_dict() for s in self.stages],
}
+435 -7
View File
@@ -4980,8 +4980,8 @@ class AgentOrchestrator:
@staticmethod
def _usage_from_transcript(
agent_id: str, claude_session_id: str | None = None
) -> tuple[int, int, int, int]:
"""Sum token usage from the agent's Claude Code transcript.
) -> tuple[int, int, int, int, int]:
"""Sum token usage + turn count from the agent's Claude Code transcript.
The host ``~/.claude`` is mounted into the orchestrator, so transcripts
are readable here under ``projects/<cwd-dir>/<session-id>.jsonl``. When
@@ -5008,11 +5008,11 @@ class AgentOrchestrator:
for f in d.glob("*.jsonl")
]
if not jsonl:
return (0, 0, 0, 0)
return (0, 0, 0, 0, 0)
newest = max(jsonl, key=lambda f: f.stat().st_mtime)
return sum_transcript_usage(newest)
except OSError:
return (0, 0, 0, 0)
return (0, 0, 0, 0, 0)
def _grok_usage_json(self, agent_id: str) -> dict[str, Any] | None:
"""Read a GROK agent's ``usage.json`` (``{model, total_tokens, cost_usd}``).
@@ -5167,13 +5167,53 @@ class AgentOrchestrator:
)
if not tokens[0] and not tokens[1]:
tin, tout, cr, cw = self._usage_from_transcript(
tin, tout, cr, cw, _turns = self._usage_from_transcript(
agent_id, self._claude_session_id_for(agent_id)
)
if tin or tout:
tokens = (tin, tout, cr, cw)
return tokens
async def _resolve_final_turns_tools(self, agent_id: str) -> tuple[int, int]:
"""Resolve final ``(turns, tool_calls)`` for a stopping agent.
Primary source is the live SDK ``/usage/status`` (which carries both).
For ``turns`` only there is a durable Claude-transcript fallback (unique
assistant-message count) for short-lived agents whose SDK counts race
teardown; ``tool_calls`` has no transcript equivalent and stays 0 ("n/a")
when the SDK misses. Grok agents have neither returns ``(0, 0)``.
Best-effort: any failure degrades to zeros, never blocks finalize.
"""
from roboco.models.base import ModelProvider
if self.get_provider_for_agent(agent_id) == ModelProvider.GROK.value:
return (0, 0)
turns = tool_calls = 0
sdk_url = f"http://roboco-agent-{agent_id}:{SDK_PORT}/usage/status"
try:
async with httpx.AsyncClient(
timeout=3.0, headers=_system_api_headers()
) as client:
resp = await client.get(sdk_url)
if resp.status_code == http_status.HTTP_200_OK:
data = resp.json()
turns = int(data.get("turns", 0) or 0)
tool_calls = int(data.get("tool_calls", 0) or 0)
except Exception as sdk_exc:
logger.debug(
"Could not fetch final turns/tool_calls from SDK",
agent_id=agent_id,
error=str(sdk_exc),
)
if not turns:
*_tokens, t = self._usage_from_transcript(
agent_id, self._claude_session_id_for(agent_id)
)
turns = t
return turns, tool_calls
async def _finalize_spawn_session(
self,
agent_id: str,
@@ -5198,6 +5238,10 @@ class AgentOrchestrator:
tokens_cache_read,
tokens_cache_write,
) = await self._resolve_final_token_usage(agent_id)
# Resolve LLM iterations + tool calls (live SDK; turns has a
# transcript fallback). Separate from the token tuple so the live
# snapshot helpers keep their 4-tuple contract.
turns, tool_calls = await self._resolve_final_turns_tools(agent_id)
# Look up the model and usage_session_id from the running instance config.
model = "unknown"
@@ -5248,6 +5292,8 @@ class AgentOrchestrator:
tokens_output=tokens_output,
tokens_cache_read=tokens_cache_read,
tokens_cache_write=tokens_cache_write,
turns=turns,
tool_calls=tool_calls,
exit_reason=exit_reason,
estimated_cost_usd=cost,
)
@@ -5305,10 +5351,11 @@ class AgentOrchestrator:
tokens = await self._fetch_agent_tokens(client, agent_id)
if tokens is not None:
return tokens
transcript = self._usage_from_transcript(
tin, tout, cr, cw, _turns = self._usage_from_transcript(
agent_id, self._claude_session_id_for(agent_id)
)
return transcript if any(transcript) else None
token_counts = (tin, tout, cr, cw)
return token_counts if any(token_counts) else None
@staticmethod
async def _persist_token_snapshot(
@@ -5594,6 +5641,385 @@ class AgentOrchestrator:
else:
db.add(DailyUsageRollupTable(id=uuid4(), **key, **values))
# =========================================================================
# MEMBER-PERFORMANCE ROLLUP (granular per-member scorecards)
# =========================================================================
async def _sweep_member_performance(self) -> None:
"""Upsert member_performance_daily from spawn sessions + audit_log.
Mirrors _sweep_daily_rollup: a trailing 7-day, overwrite-upsert sweep
(idempotent re-running overwrites, never accumulates). Aggregates each
metric with one query keyed (date, agent_slug) into an accumulator, then
upserts one row per member per day plus one CEO row per day. Wrapped in
its own try/except so a bad member rollup never aborts the sweeper.
"""
try:
from roboco.db.base import get_session_factory
except ImportError:
return
try:
window_start = datetime.now(UTC) - timedelta(days=7)
session_factory = get_session_factory()
async with session_factory() as db:
acc: dict[tuple[Any, str], dict[str, Any]] = {}
await self._msweep_spawn(db, window_start, acc)
await self._msweep_delivery(db, window_start, acc)
await self._msweep_caused(db, window_start, acc)
await self._msweep_qa(db, window_start, acc)
await self._msweep_escalations(db, window_start, acc)
await self._msweep_blocked_others(db, window_start, acc)
await self._msweep_idle(db, window_start, acc)
await self._msweep_blocked_seconds(db, window_start, acc)
for (day, slug), fields in acc.items():
await self._upsert_member_perf_row(db, day, "agent", slug, fields)
await self._msweep_ceo(db, window_start)
await db.commit()
logger.debug("Member performance rollup complete", rows=len(acc))
except Exception as exc:
logger.warning("Member performance rollup failed", error=str(exc))
@staticmethod
def _merge_member(
acc: dict[tuple[Any, str], dict[str, Any]],
day: Any,
slug: str,
**fields: Any,
) -> None:
"""Merge one metric's aggregate into the (date, slug) accumulator entry."""
if not slug:
return
entry = acc.setdefault((day, slug), {})
for key, value in fields.items():
entry[key] = value
async def _msweep_spawn(
self,
db: Any,
window_start: datetime,
acc: dict[tuple[Any, str], dict[str, Any]],
) -> None:
"""Effort / turns / tool_calls / tokens / cost from closed spawn sessions."""
from sqlalchemy import text
sql = text(
"""
SELECT date(started_at) AS d, agent_slug AS slug, team, role,
COALESCE(SUM(EXTRACT(epoch FROM
(COALESCE(ended_at, now()) - started_at))), 0) AS active_s,
COALESCE(SUM(turns), 0) AS turns,
COALESCE(SUM(tool_calls), 0) AS tool_calls,
COALESCE(SUM(tokens_input + tokens_output
+ tokens_cache_read + tokens_cache_write), 0) AS tokens,
COALESCE(SUM(estimated_cost_usd), 0) AS cost
FROM agent_spawn_sessions
WHERE ended_at IS NOT NULL AND started_at >= :ws
GROUP BY date(started_at), agent_slug, team, role
"""
)
for r in (await db.execute(sql, {"ws": window_start})).all():
self._merge_member(
acc,
r.d,
r.slug,
team=r.team,
role=r.role,
active_runtime_seconds=int(r.active_s or 0),
turns=int(r.turns or 0),
tool_calls=int(r.tool_calls or 0),
tokens=int(r.tokens or 0),
cost_usd=float(r.cost or 0.0),
)
async def _msweep_delivery(
self,
db: Any,
window_start: datetime,
acc: dict[tuple[Any, str], dict[str, Any]],
) -> None:
"""Completed / first-pass / revisions-received per task owner per day."""
from sqlalchemy import text
sql = text(
"""
SELECT date(t.completed_at) AS d, ag.slug AS slug,
COUNT(*) AS completed,
COUNT(*) FILTER (WHERE COALESCE(t.revision_count, 0) = 0) AS first_pass,
COALESCE(SUM(t.revision_count), 0) AS received
FROM tasks t JOIN agents ag ON ag.id = t.assigned_to
WHERE t.status = 'completed' AND t.completed_at >= :ws
AND t.assigned_to IS NOT NULL
GROUP BY date(t.completed_at), ag.slug
"""
)
for r in (await db.execute(sql, {"ws": window_start})).all():
self._merge_member(
acc,
r.d,
r.slug,
tasks_completed=int(r.completed or 0),
tasks_first_pass=int(r.first_pass or 0),
revisions_received=int(r.received or 0),
)
async def _msweep_caused(
self,
db: Any,
window_start: datetime,
acc: dict[tuple[Any, str], dict[str, Any]],
) -> None:
"""Revisions caused — qa/pr fail events attributed to the rejector."""
from sqlalchemy import text
sql = text(
"""
SELECT date(al.timestamp) AS d, ag.slug AS slug, COUNT(*) AS caused
FROM audit_log al JOIN agents ag ON ag.id = al.agent_id
WHERE al.event_type IN ('task.qa_fail', 'task.pr_fail')
AND al.timestamp >= :ws
GROUP BY date(al.timestamp), ag.slug
"""
)
for r in (await db.execute(sql, {"ws": window_start})).all():
self._merge_member(acc, r.d, r.slug, revisions_caused=int(r.caused or 0))
async def _msweep_qa(
self,
db: Any,
window_start: datetime,
acc: dict[tuple[Any, str], dict[str, Any]],
) -> None:
"""QA pass-rate — passed (awaiting_documentation by qa) + failed (qa_fail)."""
from sqlalchemy import text
sql = text(
"""
SELECT date(al.timestamp) AS d, ag.slug AS slug,
COUNT(*) FILTER (
WHERE al.event_type = 'task.awaiting_documentation') AS passed,
COUNT(*) FILTER (WHERE al.event_type = 'task.qa_fail') AS failed
FROM audit_log al JOIN agents ag ON ag.id = al.agent_id
WHERE al.timestamp >= :ws AND (
(al.event_type = 'task.awaiting_documentation'
AND (al.details->>'agent_role') = 'qa')
OR al.event_type = 'task.qa_fail'
)
GROUP BY date(al.timestamp), ag.slug
"""
)
for r in (await db.execute(sql, {"ws": window_start})).all():
passed = int(r.passed or 0)
failed = int(r.failed or 0)
self._merge_member(
acc,
r.d,
r.slug,
qa_reviews_passed=passed,
qa_reviews_total=passed + failed,
)
async def _msweep_escalations(
self,
db: Any,
window_start: datetime,
acc: dict[tuple[Any, str], dict[str, Any]],
) -> None:
"""Escalations raised per member (keyed on details.escalator_slug)."""
from sqlalchemy import text
sql = text(
"""
SELECT date(timestamp) AS d,
(details->>'escalator_slug') AS slug, COUNT(*) AS n
FROM audit_log
WHERE event_type = 'task.escalated' AND timestamp >= :ws
AND (details->>'escalator_slug') IS NOT NULL
GROUP BY date(timestamp), (details->>'escalator_slug')
"""
)
for r in (await db.execute(sql, {"ws": window_start})).all():
self._merge_member(acc, r.d, r.slug, escalations=int(r.n or 0))
async def _msweep_blocked_others(
self,
db: Any,
window_start: datetime,
acc: dict[tuple[Any, str], dict[str, Any]],
) -> None:
"""Downstream tasks a member's completed task was blocking."""
from sqlalchemy import text
sql = text(
"""
SELECT date(al.timestamp) AS d, ag.slug AS slug,
COALESCE(SUM((al.details->>'count')::int), 0) AS n
FROM audit_log al
JOIN tasks t ON t.id = al.target_id
JOIN agents ag ON ag.id = t.assigned_to
WHERE al.event_type = 'task.unblocked_dependents' AND al.timestamp >= :ws
GROUP BY date(al.timestamp), ag.slug
"""
)
for r in (await db.execute(sql, {"ws": window_start})).all():
self._merge_member(acc, r.d, r.slug, blocked_others=int(r.n or 0))
async def _msweep_idle(
self,
db: Any,
window_start: datetime,
acc: dict[tuple[Any, str], dict[str, Any]],
) -> None:
"""Idle seconds — each idle mark to the member's next spawn (else now)."""
from sqlalchemy import text
sql = text(
"""
WITH idle AS (
SELECT date(al.timestamp) AS d,
(al.details->>'agent_slug') AS slug, al.timestamp AS idle_at
FROM audit_log al
WHERE al.event_type = 'agent.idle' AND al.timestamp >= :ws
AND (al.details->>'agent_slug') IS NOT NULL
)
SELECT i.d, i.slug,
COALESCE(SUM(EXTRACT(epoch FROM (
COALESCE((SELECT MIN(s.started_at) FROM agent_spawn_sessions s
WHERE s.agent_slug = i.slug AND s.started_at > i.idle_at),
now()) - i.idle_at))), 0) AS idle_s
FROM idle i GROUP BY i.d, i.slug
"""
)
for r in (await db.execute(sql, {"ws": window_start})).all():
self._merge_member(acc, r.d, r.slug, idle_seconds=int(r.idle_s or 0))
async def _msweep_blocked_seconds(
self,
db: Any,
window_start: datetime,
acc: dict[tuple[Any, str], dict[str, Any]],
) -> None:
"""Wall-clock a member's tasks spent in `blocked`, per owner per day."""
from sqlalchemy import text
sql = text(
"""
WITH ordered AS (
SELECT a.target_id, a.timestamp AS entered,
(a.details->>'to_status') AS status,
LEAD(a.timestamp) OVER (
PARTITION BY a.target_id ORDER BY a.timestamp) AS exited
FROM audit_log a
WHERE a.event_type LIKE 'task.%'
AND a.event_type = 'task.' || (a.details->>'to_status')
AND a.timestamp >= :ws
)
SELECT date(o.entered) AS d, ag.slug AS slug,
COALESCE(SUM(EXTRACT(epoch FROM
(COALESCE(o.exited, now()) - o.entered))), 0) AS blocked_s
FROM ordered o
JOIN tasks t ON t.id = o.target_id
JOIN agents ag ON ag.id = t.assigned_to
WHERE o.status = 'blocked'
GROUP BY date(o.entered), ag.slug
"""
)
for r in (await db.execute(sql, {"ws": window_start})).all():
self._merge_member(acc, r.d, r.slug, blocked_seconds=int(r.blocked_s or 0))
async def _msweep_ceo(self, db: Any, window_start: datetime) -> None:
"""Upsert one CEO row per day: approval/unblock dwell + god-mode count."""
from sqlalchemy import text
sql = text(
"""
WITH events AS (
SELECT target_id, timestamp, date(timestamp) AS d,
(details->>'to_status') AS to_status,
(details->>'agent_role') AS role
FROM audit_log
WHERE event_type LIKE 'task.%' AND timestamp >= :ws
),
approvals AS (
SELECT e.d, EXTRACT(epoch FROM ((
SELECT MIN(x.timestamp) FROM events x
WHERE x.target_id = e.target_id AND x.timestamp > e.timestamp
AND x.role = 'ceo'
AND x.to_status IN
('completed', 'needs_revision', 'cancelled', 'pending')
) - e.timestamp)) AS latency
FROM events e WHERE e.to_status = 'awaiting_ceo_approval'
),
unblocks AS (
SELECT e.d, EXTRACT(epoch FROM ((
SELECT MIN(x.timestamp) FROM events x
WHERE x.target_id = e.target_id AND x.timestamp > e.timestamp
AND x.role = 'ceo' AND x.to_status IN ('in_progress', 'pending')
) - e.timestamp)) AS latency
FROM events e WHERE e.to_status = 'blocked'
)
SELECT d,
COALESCE(SUM(approval_latency), 0) AS approval_s,
COALESCE(SUM(unblock_latency), 0) AS unblock_s,
COALESCE(SUM(godmode), 0) AS godmode
FROM (
SELECT d, latency AS approval_latency, 0 AS unblock_latency, 0 AS godmode
FROM approvals WHERE latency IS NOT NULL
UNION ALL
SELECT d, 0, latency, 0 FROM unblocks WHERE latency IS NOT NULL
UNION ALL
SELECT d, 0, 0, 1 FROM events WHERE role = 'ceo'
) u GROUP BY d
"""
)
for r in (await db.execute(sql, {"ws": window_start})).all():
await self._upsert_member_perf_row(
db,
r.d,
"ceo",
"",
{
"ceo_approval_dwell_seconds": int(r.approval_s or 0),
"ceo_unblock_dwell_seconds": int(r.unblock_s or 0),
"godmode_actions": int(r.godmode or 0),
},
)
async def _upsert_member_perf_row(
self, db: Any, day: Any, member_kind: str, slug: str, fields: dict[str, Any]
) -> None:
"""Overwrite-upsert one member_performance_daily row on the natural key."""
from uuid import uuid4 as _uuid4
from sqlalchemy import select, update
from roboco.db.tables import MemberPerformanceDailyTable
existing = (
await db.execute(
select(MemberPerformanceDailyTable).where(
MemberPerformanceDailyTable.date == day,
MemberPerformanceDailyTable.member_kind == member_kind,
MemberPerformanceDailyTable.agent_slug == slug,
)
)
).scalar_one_or_none()
if existing is not None:
await db.execute(
update(MemberPerformanceDailyTable)
.where(MemberPerformanceDailyTable.id == existing.id)
.values(**fields)
)
else:
db.add(
MemberPerformanceDailyTable(
id=_uuid4(),
date=day,
member_kind=member_kind,
agent_slug=slug,
**fields,
)
)
async def restore_waiting_records(self) -> int:
"""Load persisted waiting records into memory on orchestrator start.
@@ -5956,6 +6382,8 @@ Start by:
# closed sessions into the daily aggregation table.
await self._sweep_token_snapshots()
await self._sweep_daily_rollup()
# Granular per-member performance rollup (own try/except inside).
await self._sweep_member_performance()
# Prune old agent transcripts (throttled internally to ~hourly) so the
# operator's bind-mounted ~/.claude doesn't grow without bound.
+443 -1
View File
@@ -10,27 +10,35 @@ from datetime import UTC, datetime, timedelta
from typing import Any, ClassVar
from uuid import UUID
from sqlalchemy import and_, func, select, text
from sqlalchemy import and_, bindparam, func, select, text
from sqlalchemy.ext.asyncio import AsyncSession
from roboco.db.tables import (
AgentSpawnSessionTable,
AgentTable,
AuditLogTable,
MemberPerformanceDailyTable,
MessageTable,
NotificationTable,
TaskTable,
)
from roboco.foundation.policy.stage_effort import compute_stage_effort
from roboco.models.base import TaskStatus, Team
from roboco.models.metrics import (
CEO_APPROVAL_DECISIONS,
CEO_UNBLOCK_DECISIONS,
AgentMetrics,
AgentReworkRate,
BlockerMetrics,
BottleneckReport,
CeoScorecard,
MemberScorecard,
OrgScorecard,
ReworkReport,
Scorecard,
StageBottleneck,
StageTiming,
TaskMetrics,
TeamMetrics,
TeamReworkRate,
VelocityMetrics,
@@ -782,6 +790,440 @@ class MetricsService(BaseService):
).scalar() or 0.0
return float(cost)
async def _spawn_rollup_for_task(
self, task_id: UUID, close_at: datetime
) -> dict[str, Any]:
"""Per-task spawn aggregates: stints, effort, turns, tool_calls, tokens, cost.
``active_runtime`` is summed stint duration (an open stint runs to
``close_at`` completed_at for a terminal task, else now); it can exceed
wall-clock when stints overlap. ``stints`` is the list of
``(started, ended)`` intervals for the stage decomposition. ``task_id``
is bound as ``str`` the column is ``String(36)``.
"""
rows = (
await self.session.execute(
select(
AgentSpawnSessionTable.started_at,
AgentSpawnSessionTable.ended_at,
AgentSpawnSessionTable.turns,
AgentSpawnSessionTable.tool_calls,
AgentSpawnSessionTable.tokens_input,
AgentSpawnSessionTable.tokens_output,
AgentSpawnSessionTable.tokens_cache_read,
AgentSpawnSessionTable.tokens_cache_write,
AgentSpawnSessionTable.estimated_cost_usd,
).where(AgentSpawnSessionTable.task_id == str(task_id))
)
).all()
stints: list[tuple[datetime, datetime]] = []
active = turns = tool_calls = tokens = 0.0
cost = 0.0
for r in rows:
ended = r.ended_at or close_at
stints.append((r.started_at, ended))
active += max(0.0, (ended - r.started_at).total_seconds())
turns += r.turns or 0
tool_calls += r.tool_calls or 0
tokens += (
(r.tokens_input or 0)
+ (r.tokens_output or 0)
+ (r.tokens_cache_read or 0)
+ (r.tokens_cache_write or 0)
)
cost += r.estimated_cost_usd or 0.0
return {
"stints": stints,
"active_runtime_seconds": round(active),
"turns": int(turns),
"tool_calls": int(tool_calls),
"tokens": int(tokens),
"cost_usd": float(cost),
}
async def _stage_windows_for_task(
self, task_id: UUID, close_at: datetime
) -> list[tuple[str, datetime, datetime]]:
"""Ordered status windows for one task from the audit_log journey.
Single-task variant of ``get_cycle_time_by_stage``: LEAD gives each
generic ``task.<status>`` row's exit as the next row's timestamp; the
open final window (``exited_at IS NULL``) is closed at ``close_at``
(completed_at for a terminal task, else now) so an in-flight stage still
decomposes and a terminal task's final stage doesn't grow forever.
"""
sql = text(
"""
WITH ordered AS (
SELECT
(a.details->>'to_status') AS status,
a.timestamp AS entered_at,
LEAD(a.timestamp) OVER (ORDER BY a.timestamp) AS exited_at
FROM audit_log a
WHERE a.target_id = CAST(:tid AS uuid)
AND a.event_type LIKE 'task.%'
AND a.event_type = 'task.' || (a.details->>'to_status')
)
SELECT status, entered_at, exited_at FROM ordered ORDER BY entered_at
"""
)
rows = (await self.session.execute(sql, {"tid": str(task_id)})).all()
return [(r.status, r.entered_at, r.exited_at or close_at) for r in rows]
async def _task_fail_counts(self, task_id: UUID) -> tuple[int, int]:
"""(qa_fails, pr_fails) attributed to this task from named audit events."""
rows = (
await self.session.execute(
select(AuditLogTable.event_type, func.count())
.where(
AuditLogTable.target_id == task_id,
AuditLogTable.event_type.in_(["task.qa_fail", "task.pr_fail"]),
)
.group_by(AuditLogTable.event_type)
)
).all()
counts: dict[str, int] = {row[0]: row[1] for row in rows}
return counts.get("task.qa_fail", 0), counts.get("task.pr_fail", 0)
async def get_task_metrics(self, task_id: UUID) -> TaskMetrics | None:
"""Live granular metrics for one task, or None if the task doesn't exist.
Composes summed effort + turns/tool_calls/tokens/cost (spawn sessions),
the per-stage active-vs-wait decomposition (audit windows x stints), and
who-caused-rework (revision_count + named qa/pr fail events).
"""
task_row = (
await self.session.execute(
select(
TaskTable.started_at,
TaskTable.completed_at,
TaskTable.revision_count,
).where(TaskTable.id == task_id)
)
).one_or_none()
if task_row is None:
return None
started_at, completed_at, revision_count = task_row
# Close open stints / the open final stage window at completed_at for a
# terminal task (so stages don't grow past completion), else at now.
wall_end = completed_at or datetime.now(UTC)
wall_clock = (wall_end - started_at).total_seconds() if started_at else 0.0
spawn = await self._spawn_rollup_for_task(task_id, wall_end)
windows = await self._stage_windows_for_task(task_id, wall_end)
qa_fails, pr_fails = await self._task_fail_counts(task_id)
stages = compute_stage_effort(windows, spawn["stints"])
return TaskMetrics(
task_id=str(task_id),
active_runtime_seconds=spawn["active_runtime_seconds"],
wall_clock_seconds=round(max(0.0, wall_clock)),
turns=spawn["turns"],
tool_calls=spawn["tool_calls"],
tokens=spawn["tokens"],
cost_usd=spawn["cost_usd"],
revision_count=revision_count or 0,
qa_fails=qa_fails,
pr_fails=pr_fails,
stints=len(spawn["stints"]),
stages=stages,
)
async def _ceo_latency(
self, since: datetime, from_status: str, to_statuses: Sequence[str]
) -> tuple[float, float, int]:
"""(p50, p90, count) seconds from a ``from_status`` entry to the next
CEO-attributed decision (``to_statuses``) on the same task.
Reads only ``audit_log`` (agent_role='ceo' serializes from the CEO
StrEnum). The decision is the earliest ceo row after the from-event.
"""
sql = text(
"""
WITH events AS (
SELECT target_id, timestamp,
(details->>'to_status') AS to_status,
(details->>'agent_role') AS role
FROM audit_log
WHERE event_type LIKE 'task.%' AND timestamp >= :since
),
paired AS (
SELECT (
SELECT MIN(d.timestamp) FROM events d
WHERE d.target_id = e.target_id
AND d.timestamp > e.timestamp
AND d.role = 'ceo'
AND d.to_status IN :to_statuses
) - e.timestamp AS latency
FROM events e
WHERE e.to_status = :from_status
)
SELECT
PERCENTILE_CONT(0.5) WITHIN GROUP (
ORDER BY EXTRACT(epoch FROM latency))::float AS p50,
PERCENTILE_CONT(0.9) WITHIN GROUP (
ORDER BY EXTRACT(epoch FROM latency))::float AS p90,
COUNT(latency) AS n
FROM paired WHERE latency IS NOT NULL
"""
).bindparams(bindparam("to_statuses", expanding=True))
row = (
await self.session.execute(
sql,
{
"since": since,
"from_status": from_status,
"to_statuses": list(to_statuses),
},
)
).one()
return float(row.p50 or 0.0), float(row.p90 or 0.0), int(row.n or 0)
async def _ceo_godmode_count(self, since: datetime) -> int:
"""Count every CEO-attributed task transition in the window."""
sql = text(
"""
SELECT COUNT(*) AS n FROM audit_log
WHERE event_type LIKE 'task.%'
AND timestamp >= :since
AND (details->>'agent_role') = 'ceo'
"""
)
return int((await self.session.execute(sql, {"since": since})).scalar() or 0)
async def get_ceo_scorecard(self, days: int = 30) -> CeoScorecard:
"""The human CEO's scorecard — approval/unblock dwell + god-mode count."""
since = datetime.now(UTC) - timedelta(days=days)
approval_p50, approval_p90, approval_n = await self._ceo_latency(
since, "awaiting_ceo_approval", CEO_APPROVAL_DECISIONS
)
unblock_p50, _unblock_p90, unblock_n = await self._ceo_latency(
since, "blocked", CEO_UNBLOCK_DECISIONS
)
godmode = await self._ceo_godmode_count(since)
return CeoScorecard(
approval_p50_seconds=approval_p50,
approval_p90_seconds=approval_p90,
approval_count=approval_n,
unblock_p50_seconds=unblock_p50,
unblock_count=unblock_n,
godmode_actions=godmode,
)
_ROLLUP_SUM_COLUMNS: ClassVar[tuple[str, ...]] = (
"tasks_completed",
"tasks_first_pass",
"revisions_caused",
"revisions_received",
"active_runtime_seconds",
"turns",
"tool_calls",
"tokens",
"cost_usd",
"qa_reviews_total",
"qa_reviews_passed",
"escalations",
"blocked_others",
"idle_seconds",
)
async def _rollup_sums(
self, since_date: Any, *, agent_slug: str | None, team: Team | None
) -> tuple[dict[str, float], int]:
"""SUM the rollup columns over member_performance_daily agent rows.
Filters to ``member_kind='agent'`` in the window, optionally scoped to
one member (``agent_slug``) or one cell (``team``). Returns
``(sums, member_count)`` where member_count is the distinct slugs.
"""
cols = [
func.coalesce(
func.sum(getattr(MemberPerformanceDailyTable, name)), 0
).label(name)
for name in self._ROLLUP_SUM_COLUMNS
]
conds: list[Any] = [
MemberPerformanceDailyTable.member_kind == "agent",
MemberPerformanceDailyTable.date >= since_date,
]
if agent_slug is not None:
conds.append(MemberPerformanceDailyTable.agent_slug == agent_slug)
if team is not None:
conds.append(MemberPerformanceDailyTable.team == team.value)
row = (
await self.session.execute(
select(
*cols,
func.count(func.distinct(MemberPerformanceDailyTable.agent_slug)),
).where(and_(*conds))
)
).one()
sums = {
name: float(getattr(row, name) or 0) for name in self._ROLLUP_SUM_COLUMNS
}
member_count = int(row[-1] or 0)
return sums, member_count
async def _live_inflight_overlay(self, agent_id: UUID) -> dict[str, float]:
"""Effort of the member's currently-OPEN (running) spawn sessions on
non-terminal tasks the live delta the terminal-day rollup cannot hold
yet.
Disjoint from the rollup by ``ended_at``: ``_msweep_spawn`` rolls up only
CLOSED sessions (``ended_at IS NOT NULL``), so this counts only OPEN ones
(``ended_at IS NULL``). A just-closed session lands in the rollup on the
next ~60s sweep, so there is no double-count. (Summing *all* of a task's
sessions here as an earlier version did via ``get_task_metrics``
re-counted the closed sessions the rollup already holds, inflating the
member's effort on the common reap/respawn path.)
"""
overlay = {
"active_runtime_seconds": 0.0,
"turns": 0.0,
"tool_calls": 0.0,
"tokens": 0.0,
"cost_usd": 0.0,
}
ids = (
(
await self.session.execute(
select(TaskTable.id).where(
TaskTable.assigned_to == agent_id,
TaskTable.status.notin_(
[TaskStatus.COMPLETED, TaskStatus.CANCELLED]
),
)
)
)
.scalars()
.all()
)
if not ids:
return overlay
# Aggregate in SQL (mirrors _msweep_spawn); active_runtime for an open
# session runs to now(). One row back — no per-row Python branching.
s = AgentSpawnSessionTable
row = (
await self.session.execute(
select(
func.coalesce(
func.sum(func.extract("epoch", func.now() - s.started_at)),
0,
),
func.coalesce(func.sum(s.turns), 0),
func.coalesce(func.sum(s.tool_calls), 0),
func.coalesce(
func.sum(
s.tokens_input
+ s.tokens_output
+ s.tokens_cache_read
+ s.tokens_cache_write
),
0,
),
func.coalesce(func.sum(s.estimated_cost_usd), 0),
).where(
s.task_id.in_([str(i) for i in ids]),
s.ended_at.is_(None),
)
)
).one()
overlay["active_runtime_seconds"] = float(row[0] or 0)
overlay["turns"] = float(row[1] or 0)
overlay["tool_calls"] = float(row[2] or 0)
overlay["tokens"] = float(row[3] or 0)
overlay["cost_usd"] = float(row[4] or 0)
return overlay
@staticmethod
def _ratio(numerator: float, denominator: float) -> float | None:
"""Guarded ratio → None when the denominator is 0."""
return round(numerator / denominator, 4) if denominator else None
async def get_member_scorecard(
self, agent_id: UUID, days: int = 30
) -> MemberScorecard | None:
"""Per-member rollup scorecard + live in-flight overlay, or None if no
such agent."""
agent = (
await self.session.execute(
select(AgentTable.slug, AgentTable.name).where(
AgentTable.id == agent_id
)
)
).one_or_none()
if agent is None:
return None
slug, name = agent
since_date = (datetime.now(UTC) - timedelta(days=days)).date()
sums, _ = await self._rollup_sums(since_date, agent_slug=slug, team=None)
overlay = await self._live_inflight_overlay(agent_id)
includes_live = any(v for v in overlay.values())
active_runtime = (
sums["active_runtime_seconds"] + overlay["active_runtime_seconds"]
)
turns = int(sums["turns"] + overlay["turns"])
tool_calls = int(sums["tool_calls"] + overlay["tool_calls"])
tokens = int(sums["tokens"] + overlay["tokens"])
cost = sums["cost_usd"] + overlay["cost_usd"]
completed = sums["tasks_completed"]
return MemberScorecard(
scope="member",
id=str(agent_id),
name=name,
tasks_completed=int(completed),
first_pass_yield=self._ratio(sums["tasks_first_pass"], completed),
effort_throughput_per_hour=self._ratio(completed, active_runtime / 3600),
active_runtime_hours=active_runtime / 3600,
turns=turns,
tool_calls=tool_calls,
tokens=tokens,
cost_usd=cost,
turns_per_task=self._ratio(turns, completed),
tool_calls_per_task=self._ratio(tool_calls, completed),
revisions_caused=int(sums["revisions_caused"]),
revisions_received=int(sums["revisions_received"]),
qa_pass_rate=self._ratio(
sums["qa_reviews_passed"], sums["qa_reviews_total"]
),
escalations=int(sums["escalations"]),
blocked_others=int(sums["blocked_others"]),
idle_hours=sums["idle_seconds"] / 3600,
utilization=self._ratio(
sums["active_runtime_seconds"],
sums["active_runtime_seconds"] + sums["idle_seconds"],
),
includes_live_inflight=includes_live,
)
async def get_org_scorecard(
self, team: Team | None = None, days: int = 30
) -> OrgScorecard:
"""Team (or whole-org when team is None) rollup aggregate."""
since_date = (datetime.now(UTC) - timedelta(days=days)).date()
sums, member_count = await self._rollup_sums(
since_date, agent_slug=None, team=team
)
completed = sums["tasks_completed"]
active_runtime = sums["active_runtime_seconds"]
return OrgScorecard(
scope="team" if team else "org",
team=team.value if team else None,
member_count=member_count,
tasks_completed=int(completed),
first_pass_yield=self._ratio(sums["tasks_first_pass"], completed),
effort_throughput_per_hour=self._ratio(completed, active_runtime / 3600),
active_runtime_hours=active_runtime / 3600,
turns=int(sums["turns"]),
tool_calls=int(sums["tool_calls"]),
tokens=int(sums["tokens"]),
cost_usd=sums["cost_usd"],
revisions_caused=int(sums["revisions_caused"]),
revisions_received=int(sums["revisions_received"]),
)
async def get_rework_metrics(
self, team: Team | None = None, days: int = 30
) -> ReworkReport:
+54 -1
View File
@@ -12,7 +12,7 @@ Also implements the ACK system for tracking acknowledgments.
import asyncio
from dataclasses import dataclass
from datetime import UTC, datetime
from typing import ClassVar, Literal, cast
from typing import TYPE_CHECKING, ClassVar, Literal, cast
from uuid import UUID
import structlog
@@ -32,9 +32,33 @@ from roboco.services.base import BaseService, NotFoundError
from roboco.services.notification_dedup import all_recipients_recently_notified
from roboco.utils.converters import require_uuid
if TYPE_CHECKING:
from roboco.models.metrics import TaskMetrics
_log = structlog.get_logger(service="notification_delivery")
def _format_completion_body(task: TaskTable, metrics: "TaskMetrics | None") -> str:
"""Human-readable completion summary — real effort vs wall-clock, not a lone
wall-clock figure. Degrades to wall-clock-only (turns 'n/a') when there are
no spawn sessions / pre-turns-migration data."""
title = task.title or "Untitled"
if metrics is None:
return f"Task '{title}' completed."
wall_h = round(metrics.wall_clock_seconds / 3600, 1)
active_h = round(metrics.active_runtime_seconds / 3600, 1)
turns = str(metrics.turns) if metrics.turns else "n/a"
return (
f"Task '{title}' completed.\n\n"
f"Active effort: {active_h}h across {metrics.stints} stint(s) "
f"({turns} turns, {metrics.tool_calls} tool-calls)\n"
f"Wall-clock: {wall_h}h\n"
f"Revisions: {metrics.revision_count} "
f"({metrics.qa_fails} QA / {metrics.pr_fails} PR)\n"
f"Cost: ${round(metrics.cost_usd, 2)}"
)
# =============================================================================
# Deferred bus publish — transactional outbox (F107)
# =============================================================================
@@ -837,6 +861,35 @@ class NotificationDeliveryService(BaseService):
)
await self._persist_and_deliver(notification)
async def notify_ceo_of_completion(self, *, task: TaskTable, task_id: UUID) -> None:
"""CEO-facing completion notification with the granular effort breakdown.
Replaces the coarse "completed in Xh" wall-clock figure with real effort
vs wall-clock + turns/stints/revisions/cost from the per-task metrics.
Best-effort: a metrics or delivery failure must never block completion.
"""
ceo = await self._get_ceo_agent()
if not ceo:
return
from roboco.services.metrics import MetricsService
try:
metrics = await MetricsService(self.session).get_task_metrics(task_id)
except Exception: # metrics are best-effort — degrade to wall-clock-only
metrics = None
from_agent = cast("UUID", task.assigned_to) if task.assigned_to else ceo.id
notification = NotificationTable(
type=NotificationType.ALERT,
priority=NotificationPriority.NORMAL,
from_agent=from_agent,
to_agents=[ceo.id],
subject=f"Completed: {(task.title or 'Untitled')[:60]}",
body=_format_completion_body(task, metrics),
related_task_id=task_id,
requires_ack=ACK_REQUIRED_BY_TYPE[NotificationType.ALERT],
)
await self._persist_and_deliver(notification)
# ------------------------------------------------------------------
# Private helpers for recipient resolution + persist
# ------------------------------------------------------------------
+101 -1
View File
@@ -784,6 +784,33 @@ class TaskService(BaseService):
)
)
def _emit_escalation_audit(
self, task: TaskTable, *, escalator_slug: str, target_slug: str
) -> None:
"""Emit a ``task.escalated`` audit row (per-member escalation metric).
Additive to the ``task.blocked`` / pool-release audit of an escalation
it carries WHO escalated (``escalator_slug``) so the member scorecard can
charge the escalation to the escalator. Written in the caller's session
so it commits atomically with the escalation. Best-effort for metrics:
never gates the escalation itself.
"""
from roboco.db.tables import AuditLogTable
self.session.add(
AuditLogTable(
event_type="task.escalated",
agent_id=None,
target_type="task",
target_id=task.id,
severity="info",
details={
"escalator_slug": escalator_slug,
"target_slug": target_slug,
},
)
)
@staticmethod
def _audit_events_for(to_status: str, agent_role: str | None) -> list[str]:
"""Audit event types to emit for a transition.
@@ -5077,6 +5104,10 @@ class TaskService(BaseService):
await self._trigger_completion_hooks(task, agent_id)
await self._unblock_dependents(task_id)
# Emit the completion event (previously defined but never emitted) so the
# WS bridge can forward a live completion signal. The CEO-facing granular
# notification fires on the root's ceo_approve, not on each leaf complete.
await self._emit_task_event(EventType.TASK_COMPLETED, task_id, {})
return task
async def _escalation_diverts_to_pool(
@@ -5172,6 +5203,9 @@ class TaskService(BaseService):
blocked_target_slug=target_slug,
reason=reason,
)
self._emit_escalation_audit(
task, escalator_slug=escalator_slug, target_slug=target_slug
)
return True
if task.assigned_to and not task.blocker_raised_by:
task.blocker_raised_by = cast("Any", task.assigned_to)
@@ -5207,6 +5241,9 @@ class TaskService(BaseService):
agent_role=None,
audit_agent_id=pre_block_owner,
)
self._emit_escalation_audit(
task, escalator_slug=escalator_slug, target_slug=target_slug
)
self.log.info(
"Task escalated and blocked",
task_id=str(task.id),
@@ -5384,6 +5421,11 @@ class TaskService(BaseService):
# Unblock any tasks waiting on this one
await self._unblock_dependents(task_id)
# Emit the completion event (previously dead code — never emitted) + the
# CEO-facing granular completion notification (real effort vs wall-clock).
await self._emit_task_event(EventType.TASK_COMPLETED, task_id, {})
await self._notify_completion(task, task_id)
# Emit event for CEO approval
await self._emit_task_event(
EventType.TASK_CEO_APPROVED,
@@ -5979,6 +6021,26 @@ class TaskService(BaseService):
return task
async def _notify_completion(self, task: TaskTable, task_id: UUID) -> None:
"""Best-effort CEO completion notification (granular effort breakdown).
A notification/metrics failure must never block the completion, so any
error is logged and swallowed.
"""
try:
from roboco.services.notification_delivery import (
get_notification_delivery_service,
)
delivery = get_notification_delivery_service(self.session)
await delivery.notify_ceo_of_completion(task=task, task_id=task_id)
except Exception as exc:
self.log.warning(
"Completion notification failed",
task_id=str(task_id),
error=str(exc),
)
async def _emit_task_event(
self,
event_type: EventType,
@@ -6040,6 +6102,26 @@ class TaskService(BaseService):
completed_dependency=str(completed_task_id),
)
if blocked_tasks:
# Metrics (blocked-others): record how many downstream tasks this
# completed task was blocking. _unblock_dependents PRUNES the
# dependency edges above, destroying the only other record — so
# capture the count NOW as a durable audit row. The sweeper
# attributes it to the completed task's owner. target_id is the
# BLOCKER (completed) task.
from roboco.db.tables import AuditLogTable
self.session.add(
AuditLogTable(
event_type="task.unblocked_dependents",
agent_id=None,
target_type="task",
target_id=completed_task_id,
severity="info",
details={"count": len(blocked_tasks)},
)
)
await self.session.flush()
async def _revive_unblocked_dependent(self, task: TaskTable) -> None:
@@ -8146,7 +8228,13 @@ class TaskService(BaseService):
return task
async def mark_agent_idle(self, agent_id: UUID) -> None:
"""Set agent.status = IDLE."""
"""Set agent.status = IDLE + emit an ``agent.idle`` audit row.
The audit row (target = the agent, details.agent_slug) is the idle
signal the member scorecard needs it lets the sweeper compute idle
time as the gap from an idle mark to the member's next spawn. Written in
the same session so it commits with the status change.
"""
result = await self.session.execute(
select(AgentTable).where(AgentTable.id == agent_id)
)
@@ -8154,6 +8242,18 @@ class TaskService(BaseService):
if agent is None:
return
agent.status = AgentStatus.IDLE
from roboco.db.tables import AuditLogTable
self.session.add(
AuditLogTable(
event_type="agent.idle",
agent_id=agent_id,
target_type="agent",
target_id=agent_id,
severity="info",
details={"agent_slug": agent.slug},
)
)
await self.session.flush()
async def _qa_or_doc_claim(
@@ -0,0 +1,210 @@
"""CEO completion notification — granular effort breakdown (phase 6).
The pure body formatter (real effort vs wall-clock; degrades to wall-clock-only)
+ notify_ceo_of_completion end to end against real PG.
"""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
from typing import TYPE_CHECKING, Any, cast
from uuid import UUID, uuid4
import pytest
import pytest_asyncio
from roboco.db.tables import (
AgentSpawnSessionTable,
AgentTable,
NotificationTable,
ProjectTable,
TaskTable,
)
from roboco.models.base import (
AgentRole,
AgentStatus,
Complexity,
NotificationType,
TaskNature,
TaskStatus,
TaskType,
Team,
)
from roboco.models.metrics import TaskMetrics
from roboco.services.notification_delivery import (
_format_completion_body,
get_notification_delivery_service,
)
from sqlalchemy import select
if TYPE_CHECKING:
from collections.abc import AsyncIterator
from sqlalchemy.ext.asyncio import AsyncSession
def _metrics(**over: Any) -> TaskMetrics:
base: dict[str, Any] = {
"task_id": str(uuid4()),
"active_runtime_seconds": 3600,
"wall_clock_seconds": 7200,
"turns": 42,
"tool_calls": 99,
"tokens": 1000,
"cost_usd": 4.2,
"revision_count": 2,
"qa_fails": 1,
"pr_fails": 1,
"stints": 3,
"stages": [],
}
base.update(over)
return TaskMetrics(**base)
def test_format_body_with_metrics() -> None:
body = _format_completion_body(
cast("TaskTable", SimpleNamespace(title="Auth flow")), _metrics()
)
assert "Auth flow" in body
assert "Active effort: 1.0h across 3 stint(s)" in body
assert "42 turns" in body
assert "Wall-clock: 2.0h" in body
assert "2 (1 QA / 1 PR)" in body
assert "$4.2" in body
def test_format_body_turns_na_when_zero() -> None:
body = _format_completion_body(
cast("TaskTable", SimpleNamespace(title="T")), _metrics(turns=0)
)
assert "n/a turns" in body # pre-turns-migration / Grok
def test_format_body_degrades_without_metrics() -> None:
body = _format_completion_body(cast("TaskTable", SimpleNamespace(title="T")), None)
assert body == "Task 'T' completed."
@pytest_asyncio.fixture
async def env(db_session: AsyncSession) -> AsyncIterator[dict]:
# The CEO is a singleton in the real system, and `_get_ceo_agent()` resolves
# it by `role == CEO` with `scalar_one_or_none()`. The session-scoped test DB
# is shared across the run, and a sibling real-DB test commits a role=CEO
# agent (slug="ceo") without cleanup, so it can already be present here.
# Reuse an existing CEO rather than inserting a second one — creating another
# would both collide on the unique slug and make `_get_ceo_agent()` raise
# MultipleResultsFound. Order-independent: in isolation we create one.
existing_ceo = (
(
await db_session.execute(
select(AgentTable).where(AgentTable.role == AgentRole.CEO)
)
)
.scalars()
.first()
)
ceo = existing_ceo or AgentTable(
id=uuid4(),
name="CEO",
slug=f"ceo-{uuid4().hex[:6]}",
role=AgentRole.CEO,
team=None,
status=AgentStatus.ACTIVE,
model_config={},
system_prompt="x",
capabilities=[],
permissions={},
metrics={},
)
if existing_ceo is None:
db_session.add(ceo)
dev = AgentTable(
id=uuid4(),
name="dev",
slug=f"be-dev-{uuid4().hex[:6]}",
role=AgentRole.DEVELOPER,
team=Team.BACKEND,
status=AgentStatus.ACTIVE,
model_config={},
system_prompt="x",
capabilities=[],
permissions={},
metrics={},
)
db_session.add(dev)
await db_session.flush()
project = ProjectTable(
id=uuid4(),
name="P",
slug=f"p-{uuid4().hex[:6]}",
git_url="https://example.com/r.git",
assigned_cell=Team.BACKEND,
created_by=dev.id,
)
db_session.add(project)
await db_session.flush()
yield {"db": db_session, "ceo": ceo, "dev": dev, "project_id": project.id}
@pytest.mark.asyncio
async def test_notify_ceo_of_completion_creates_alert(env: dict) -> None:
db = env["db"]
base = datetime.now(UTC) - timedelta(hours=2)
task = TaskTable(
id=uuid4(),
title="Ship it",
description="d",
acceptance_criteria=["ac"],
task_type=TaskType.CODE,
nature=TaskNature.TECHNICAL,
status=TaskStatus.COMPLETED,
team=Team.BACKEND,
project_id=env["project_id"],
created_by=env["dev"].id,
assigned_to=env["dev"].id,
estimated_complexity=Complexity.MEDIUM,
started_at=base,
completed_at=base + timedelta(seconds=600),
)
db.add(task)
await db.flush()
db.add(
AgentSpawnSessionTable(
id=uuid4(),
agent_slug=env["dev"].slug,
team="backend",
role="developer",
model="claude",
task_id=str(task.id),
started_at=base,
ended_at=base + timedelta(seconds=300),
turns=7,
tool_calls=12,
tokens_input=10,
tokens_output=5,
estimated_cost_usd=0.5,
)
)
await db.flush()
delivery = get_notification_delivery_service(db)
await delivery.notify_ceo_of_completion(task=task, task_id=cast("UUID", task.id))
rows = (
(
await db.execute(
select(NotificationTable).where(
NotificationTable.related_task_id == task.id
)
)
)
.scalars()
.all()
)
assert len(rows) == 1
note = rows[0]
assert note.type == NotificationType.ALERT
assert env["ceo"].id in note.to_agents
assert "Active effort" in note.body
assert "Ship it" in note.subject
+116 -1
View File
@@ -14,10 +14,18 @@ from httpx import ASGITransport, AsyncClient
from roboco.api.deps import get_agent_context, get_db
from roboco.api.routes.dashboard import get_main_pm_kanban
from roboco.api.routes.dashboard import router as dashboard_router
from roboco.db.tables import AgentTable
from roboco.db.tables import AgentTable, ProjectTable, TaskTable
from roboco.models import AgentRole, AgentStatus
from roboco.models.base import (
Complexity,
TaskNature,
TaskStatus,
TaskType,
Team,
)
from roboco.models.permissions import AgentContext
from roboco.services.dashboard import reset_storage
from sqlalchemy import select
if TYPE_CHECKING:
from collections.abc import AsyncGenerator, AsyncIterator
@@ -408,3 +416,110 @@ async def test_get_main_pm_kanban_function_directly(
"""
result = await get_main_pm_kanban(db_session)
assert isinstance(result, dict)
@pytest.mark.asyncio
async def test_ceo_scorecard_endpoint(dashboard_client: AsyncClient) -> None:
resp = await dashboard_client.get(
"/api/dashboard/metrics/member/ceo?days=30", headers=_HDR
)
assert resp.status_code == HTTPStatus.OK
body = resp.json()
assert body["member_kind"] == "ceo"
assert set(body) >= {
"approval_p50_seconds",
"approval_count",
"unblock_p50_seconds",
"unblock_count",
"godmode_actions",
}
@pytest.mark.asyncio
async def test_member_scorecard_404_when_absent(dashboard_client: AsyncClient) -> None:
resp = await dashboard_client.get(
f"/api/dashboard/metrics/member/{uuid4()}", headers=_HDR
)
assert resp.status_code == HTTPStatus.NOT_FOUND
@pytest.mark.asyncio
async def test_ceo_route_wins_over_member_uuid_route(
dashboard_client: AsyncClient,
) -> None:
# The literal "ceo" must resolve to the CEO route, not the {agent_id} route.
resp = await dashboard_client.get("/api/dashboard/metrics/member/ceo", headers=_HDR)
assert resp.status_code == HTTPStatus.OK
assert resp.json()["member_kind"] == "ceo"
@pytest.mark.asyncio
async def test_org_scorecard_endpoint(dashboard_client: AsyncClient) -> None:
resp = await dashboard_client.get("/api/dashboard/metrics/org", headers=_HDR)
assert resp.status_code == HTTPStatus.OK
body = resp.json()
assert body["scope"] == "org"
assert set(body) >= {"member_count", "tasks_completed", "first_pass_yield"}
@pytest.mark.asyncio
async def test_task_metrics_404_for_missing_task(
dashboard_client: AsyncClient,
) -> None:
resp = await dashboard_client.get(
f"/api/dashboard/metrics/task/{uuid4()}", headers=_HDR
)
assert resp.status_code == HTTPStatus.NOT_FOUND
@pytest.mark.asyncio
async def test_task_metrics_returns_shape_for_existing_task(
db_session: AsyncSession, dashboard_client: AsyncClient
) -> None:
creator = (await db_session.execute(select(AgentTable).limit(1))).scalar_one()
project = ProjectTable(
id=uuid4(),
name="P",
slug=f"p-{uuid4().hex[:6]}",
git_url="https://example.com/r.git",
assigned_cell=Team.BACKEND,
created_by=creator.id,
)
db_session.add(project)
await db_session.flush()
task = TaskTable(
id=uuid4(),
title="t",
description="d",
acceptance_criteria=["ac"],
task_type=TaskType.CODE,
nature=TaskNature.TECHNICAL,
status=TaskStatus.IN_PROGRESS,
team=Team.BACKEND,
project_id=project.id,
created_by=creator.id,
estimated_complexity=Complexity.MEDIUM,
)
db_session.add(task)
await db_session.flush()
resp = await dashboard_client.get(
f"/api/dashboard/metrics/task/{task.id}", headers=_HDR
)
assert resp.status_code == HTTPStatus.OK
body = resp.json()
assert body["task_id"] == str(task.id)
assert set(body) >= {
"active_runtime_seconds",
"wall_clock_seconds",
"turns",
"tool_calls",
"tokens",
"cost_usd",
"revision_count",
"qa_fails",
"pr_fails",
"stints",
"stages",
}
assert isinstance(body["stages"], list)
+299
View File
@@ -0,0 +1,299 @@
"""_sweep_member_performance — the granular per-member rollup, against real PG.
Seeds a day of spawn sessions + completed tasks + the new audit events
(escalated / unblocked_dependents / agent.idle / qa pass+fail / CEO decisions),
runs the sweep, and asserts the agent + CEO rows plus idempotency (a second
sweep overwrites, never doubles).
"""
from __future__ import annotations
from contextlib import asynccontextmanager
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING, Any
from unittest.mock import patch
from uuid import uuid4
import pytest
import pytest_asyncio
from roboco.db.tables import (
AgentSpawnSessionTable,
AgentTable,
AuditLogTable,
MemberPerformanceDailyTable,
ProjectTable,
TaskTable,
)
from roboco.models.base import (
AgentRole,
AgentStatus,
Complexity,
TaskNature,
TaskStatus,
TaskType,
Team,
)
from roboco.runtime.orchestrator import AgentOrchestrator
from sqlalchemy import select
if TYPE_CHECKING:
from collections.abc import AsyncIterator
from sqlalchemy.ext.asyncio import AsyncSession
# Yesterday at noon UTC: safely inside the 7-day window and far from midnight,
# so started_at / completed_at (+1h) and all audit events share ONE date
# (the sweep legitimately splits cross-midnight work across days — not under test
# here). This keeps the seeded member's rollup on a single row.
_BASE = (datetime.now(UTC) - timedelta(days=1)).replace(
hour=12, minute=0, second=0, microsecond=0
)
_ACTIVE_SECONDS = 600
_APPROVAL_DWELL_SECONDS = 300
_COMPLETED_TASKS = 2
_BLOCKED_OTHERS = 2
_QA_TOTAL = 2
class _NoCommitSession:
"""Delegates to the real test session but turns commit() into flush() so the
per-test rollback isolation holds while the sweep still 'commits'."""
def __init__(self, real: Any) -> None:
self._real = real
def __getattr__(self, name: str) -> Any:
return getattr(self._real, name)
async def commit(self) -> None:
await self._real.flush()
def _agent(role: AgentRole, slug: str) -> AgentTable:
return AgentTable(
id=uuid4(),
name=slug,
slug=slug,
role=role,
team=Team.BACKEND,
status=AgentStatus.ACTIVE,
model_config={},
system_prompt="x",
capabilities=[],
permissions={},
metrics={},
)
def _audit(
target_id: Any,
event_type: str,
ts: datetime,
*,
agent_id: Any = None,
details: dict[str, Any] | None = None,
) -> AuditLogTable:
return AuditLogTable(
id=uuid4(),
event_type=event_type,
agent_id=agent_id,
target_type="task",
target_id=target_id,
severity="info",
details=details or {},
timestamp=ts,
)
@pytest_asyncio.fixture
async def seeded(db_session: AsyncSession) -> AsyncIterator[dict]:
dev = _agent(AgentRole.DEVELOPER, f"be-dev-{uuid4().hex[:6]}")
qa = _agent(AgentRole.QA, f"be-qa-{uuid4().hex[:6]}")
db_session.add_all([dev, qa])
await db_session.flush()
project = ProjectTable(
id=uuid4(),
name="P",
slug=f"p-{uuid4().hex[:6]}",
git_url="https://example.com/r.git",
assigned_cell=Team.BACKEND,
created_by=dev.id,
)
db_session.add(project)
await db_session.flush()
task = TaskTable(
id=uuid4(),
title="t",
description="d",
acceptance_criteria=["ac"],
task_type=TaskType.CODE,
nature=TaskNature.TECHNICAL,
status=TaskStatus.COMPLETED,
team=Team.BACKEND,
project_id=project.id,
created_by=dev.id,
assigned_to=dev.id,
revision_count=1,
estimated_complexity=Complexity.MEDIUM,
started_at=_BASE,
completed_at=_BASE + timedelta(hours=1),
)
blocker = TaskTable(
id=uuid4(),
title="b",
description="d",
acceptance_criteria=["ac"],
task_type=TaskType.CODE,
nature=TaskNature.TECHNICAL,
status=TaskStatus.COMPLETED,
team=Team.BACKEND,
project_id=project.id,
created_by=dev.id,
assigned_to=dev.id,
estimated_complexity=Complexity.MEDIUM,
started_at=_BASE,
completed_at=_BASE + timedelta(hours=1),
)
db_session.add_all([task, blocker])
await db_session.flush()
db_session.add(
AgentSpawnSessionTable(
id=uuid4(),
agent_slug=dev.slug,
team="backend",
role="developer",
model="claude",
task_id=str(task.id),
started_at=_BASE,
ended_at=_BASE + timedelta(seconds=600),
turns=5,
tool_calls=10,
tokens_input=100,
tokens_output=80,
estimated_cost_usd=1.5,
)
)
db_session.add_all(
[
_audit(
task.id,
"task.qa_fail",
_BASE,
agent_id=qa.id,
details={"agent_role": "qa"},
),
_audit(
task.id,
"task.awaiting_documentation",
_BASE + timedelta(minutes=1),
agent_id=qa.id,
details={"agent_role": "qa"},
),
_audit(
task.id,
"task.escalated",
_BASE,
details={"escalator_slug": dev.slug, "target_slug": qa.slug},
),
_audit(
blocker.id, "task.unblocked_dependents", _BASE, details={"count": 2}
),
_audit(
dev.id,
"agent.idle",
_BASE + timedelta(minutes=5),
agent_id=dev.id,
details={"agent_slug": dev.slug},
),
# CEO decision: approval + god-mode (to_status is what the pairing reads).
_audit(
task.id,
"task.awaiting_ceo_approval",
_BASE,
details={"to_status": "awaiting_ceo_approval", "agent_role": "main_pm"},
),
_audit(
task.id,
"task.completed",
_BASE + timedelta(seconds=300),
details={"to_status": "completed", "agent_role": "ceo"},
),
]
)
await db_session.flush()
yield {"db": db_session, "dev": dev.slug, "qa": qa.slug}
async def _rows(db: AsyncSession, slug: str) -> list[Any]:
return list(
(
await db.execute(
select(MemberPerformanceDailyTable).where(
MemberPerformanceDailyTable.agent_slug == slug
)
)
)
.scalars()
.all()
)
async def _run_sweep(db: AsyncSession) -> None:
orch = AgentOrchestrator.__new__(AgentOrchestrator)
@asynccontextmanager
async def _cm() -> Any:
yield _NoCommitSession(db)
with patch("roboco.db.base.get_session_factory", return_value=_cm):
await orch._sweep_member_performance()
@pytest.mark.asyncio
async def test_sweep_populates_agent_and_ceo_rows(seeded: dict) -> None:
db = seeded["db"]
await _run_sweep(db)
dev_rows = await _rows(db, seeded["dev"])
assert len(dev_rows) == 1
dev = dev_rows[0]
assert dev.member_kind == "agent"
assert dev.active_runtime_seconds == _ACTIVE_SECONDS
assert (dev.turns, dev.tool_calls, dev.tokens) == (5, 10, 180)
assert dev.cost_usd == pytest.approx(1.5)
assert dev.tasks_completed == _COMPLETED_TASKS # task + blocker
assert dev.tasks_first_pass == 1 # blocker had 0 revisions
assert dev.revisions_received == 1
assert dev.escalations == 1
assert dev.blocked_others == _BLOCKED_OTHERS
assert dev.idle_seconds > 0
qa_rows = await _rows(db, seeded["qa"])
assert len(qa_rows) == 1
qa = qa_rows[0]
assert qa.revisions_caused == 1 # the qa_fail
assert qa.qa_reviews_passed == 1
assert qa.qa_reviews_total == _QA_TOTAL # 1 pass + 1 fail
ceo_rows = await _rows(db, "")
assert len(ceo_rows) == 1
ceo = ceo_rows[0]
assert ceo.member_kind == "ceo"
assert ceo.godmode_actions == 1
assert ceo.ceo_approval_dwell_seconds == _APPROVAL_DWELL_SECONDS
@pytest.mark.asyncio
async def test_sweep_is_idempotent(seeded: dict) -> None:
db = seeded["db"]
await _run_sweep(db)
await _run_sweep(db) # second pass must overwrite, not double
dev = (await _rows(db, seeded["dev"]))[0]
assert dev.active_runtime_seconds == _ACTIVE_SECONDS # not 1200
assert dev.tasks_completed == _COMPLETED_TASKS # not 4
assert dev.escalations == 1
assert len(await _rows(db, seeded["dev"])) == 1
+118
View File
@@ -0,0 +1,118 @@
"""get_ceo_scorecard — the human CEO as a measured member (audit-log only).
Seeds CEO-attributed audit transitions and asserts approval dwell (incl. the
coordination-root reject that lands in `pending`), unblock dwell, and the
god-mode action count. The CEO never runs an LLM, so this reads only audit_log
(agent_role='ceo').
"""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING, Any
from uuid import uuid4
import pytest
import pytest_asyncio
from roboco.db.tables import AuditLogTable
from roboco.services.metrics import MetricsService
if TYPE_CHECKING:
from collections.abc import AsyncIterator
from sqlalchemy.ext.asyncio import AsyncSession
_NOW = datetime.now(UTC)
def _audit(
task_id: Any,
status: str,
ts: datetime,
*,
agent_role: str | None = None,
) -> AuditLogTable:
return AuditLogTable(
id=uuid4(),
event_type=f"task.{status}",
agent_id=None,
target_type="task",
target_id=task_id,
severity="info",
details={"to_status": status, "from_status": "prev", "agent_role": agent_role},
timestamp=ts,
)
@pytest_asyncio.fixture
async def svc(db_session: AsyncSession) -> AsyncIterator[MetricsService]:
yield MetricsService(db_session)
@pytest.mark.asyncio
async def test_empty_window_returns_zeros(svc: MetricsService) -> None:
card = await svc.get_ceo_scorecard(days=30)
assert card.approval_count == 0
assert card.unblock_count == 0
assert card.godmode_actions == 0
assert card.approval_p50_seconds == 0.0
@pytest.mark.asyncio
async def test_approval_unblock_dwell_and_godmode(
svc: MetricsService, db_session: AsyncSession
) -> None:
base = _NOW - timedelta(hours=2)
t_approve = uuid4()
t_reject = uuid4()
t_block = uuid4()
db_session.add_all(
[
# Approval: awaiting -> completed(ceo) after 300s.
_audit(t_approve, "awaiting_ceo_approval", base),
_audit(
t_approve, "completed", base + timedelta(seconds=300), agent_role="ceo"
),
# Coordination-root reject: awaiting -> pending(ceo) after 120s.
_audit(t_reject, "awaiting_ceo_approval", base),
_audit(
t_reject, "pending", base + timedelta(seconds=120), agent_role="ceo"
),
# Unblock: blocked -> in_progress(ceo) after 600s.
_audit(t_block, "blocked", base),
_audit(
t_block, "in_progress", base + timedelta(seconds=600), agent_role="ceo"
),
]
)
await db_session.flush()
card = await svc.get_ceo_scorecard(days=30)
# Two approval decisions (completed + coordination pending), median of 300/120.
expected_approvals = 2
assert card.approval_count == expected_approvals
assert card.approval_p50_seconds == pytest.approx(210.0)
# One unblock, 600s.
assert card.unblock_count == 1
assert card.unblock_p50_seconds == pytest.approx(600.0)
# God-mode = every ceo-attributed transition: completed + pending + in_progress.
expected_godmode = 3
assert card.godmode_actions == expected_godmode
@pytest.mark.asyncio
async def test_non_ceo_transitions_are_not_counted(
svc: MetricsService, db_session: AsyncSession
) -> None:
tid = uuid4()
db_session.add_all(
[
_audit(tid, "awaiting_ceo_approval", _NOW - timedelta(minutes=10)),
# A QA fail (not the CEO) must not count as an approval or god-mode.
_audit(tid, "needs_revision", _NOW - timedelta(minutes=5), agent_role="qa"),
]
)
await db_session.flush()
card = await svc.get_ceo_scorecard(days=30)
assert card.approval_count == 0
assert card.godmode_actions == 0
@@ -0,0 +1,164 @@
"""New audit instrumentation feeding the extra per-member metrics (phase 4).
- apply_escalation -> task.escalated (escalations metric)
- _unblock_dependents -> task.unblocked_dependents (blocked-others metric)
- mark_agent_idle -> agent.idle (idle/utilization metric)
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from uuid import uuid4
import pytest
import pytest_asyncio
from roboco.db.tables import AgentTable, AuditLogTable, ProjectTable, TaskTable
from roboco.models.base import (
AgentRole,
AgentStatus,
Complexity,
TaskNature,
TaskStatus,
TaskType,
Team,
)
from roboco.services.task import TaskService
from sqlalchemy import select
if TYPE_CHECKING:
from collections.abc import AsyncIterator
from sqlalchemy.ext.asyncio import AsyncSession
def _agent(role: AgentRole, slug: str) -> AgentTable:
return AgentTable(
id=uuid4(),
name=slug,
slug=slug,
role=role,
team=Team.BACKEND,
status=AgentStatus.ACTIVE,
model_config={},
system_prompt="x",
capabilities=[],
permissions={},
metrics={},
)
def _task(project_id: Any, created_by: Any, **over: Any) -> TaskTable:
base: dict[str, Any] = {
"id": uuid4(),
"title": "t",
"description": "d",
"acceptance_criteria": ["ac"],
"task_type": TaskType.CODE,
"nature": TaskNature.TECHNICAL,
"status": TaskStatus.IN_PROGRESS,
"team": Team.BACKEND,
"project_id": project_id,
"created_by": created_by,
"estimated_complexity": Complexity.MEDIUM,
}
base.update(over)
return TaskTable(**base)
async def _audit_of(db: AsyncSession, event_type: str, target_id: Any) -> list[Any]:
rows = (
(
await db.execute(
select(AuditLogTable).where(
AuditLogTable.event_type == event_type,
AuditLogTable.target_id == target_id,
)
)
)
.scalars()
.all()
)
return list(rows)
@pytest_asyncio.fixture
async def env(db_session: AsyncSession) -> AsyncIterator[dict]:
dev = _agent(AgentRole.DEVELOPER, f"be-dev-{uuid4().hex[:6]}")
pm = _agent(AgentRole.CELL_PM, f"be-pm-{uuid4().hex[:6]}")
db_session.add_all([dev, pm])
await db_session.flush()
project = ProjectTable(
id=uuid4(),
name="P",
slug=f"p-{uuid4().hex[:6]}",
git_url="https://example.com/r.git",
assigned_cell=Team.BACKEND,
created_by=dev.id,
)
db_session.add(project)
await db_session.flush()
yield {
"svc": TaskService(db_session),
"db": db_session,
"project_id": project.id,
"dev": dev,
"pm": pm,
}
@pytest.mark.asyncio
async def test_apply_escalation_emits_task_escalated(env: dict) -> None:
db = env["db"]
task = _task(
env["project_id"],
env["dev"].id,
assigned_to=env["dev"].id,
claimed_by=env["dev"].id,
)
db.add(task)
await db.flush()
ok = await env["svc"].apply_escalation(
task=task,
target_agent_id=env["pm"].id,
escalator_slug=env["dev"].slug,
target_slug=env["pm"].slug,
reason="need help with the seam",
)
assert ok is True
rows = await _audit_of(db, "task.escalated", task.id)
assert len(rows) == 1
assert rows[0].details["escalator_slug"] == env["dev"].slug
@pytest.mark.asyncio
async def test_unblock_dependents_emits_count(env: dict) -> None:
db = env["db"]
blocker = _task(env["project_id"], env["dev"].id, status=TaskStatus.COMPLETED)
db.add(blocker)
await db.flush()
dependent = _task(
env["project_id"],
env["dev"].id,
status=TaskStatus.BLOCKED,
dependency_ids=[blocker.id],
assigned_to=env["dev"].id,
claimed_by=env["dev"].id,
)
db.add(dependent)
await db.flush()
await env["svc"]._unblock_dependents(blocker.id)
rows = await _audit_of(db, "task.unblocked_dependents", blocker.id)
assert len(rows) == 1
assert rows[0].details["count"] == 1
@pytest.mark.asyncio
async def test_mark_agent_idle_emits_agent_idle(env: dict) -> None:
db = env["db"]
await env["svc"].mark_agent_idle(env["dev"].id)
rows = await _audit_of(db, "agent.idle", env["dev"].id)
assert len(rows) == 1
assert rows[0].details["agent_slug"] == env["dev"].slug
refreshed = await db.get(AgentTable, env["dev"].id)
assert refreshed is not None
assert refreshed.status == AgentStatus.IDLE
@@ -0,0 +1,316 @@
"""Member / org rollup scorecards + the live in-flight overlay (real PG).
Seeds member_performance_daily rows (the rollup source) and asserts the derived
rates (FPY, effort-throughput, turns/task, qa pass-rate, utilization), the live
in-flight overlay (enriches effort but not completion counts disjoint by
status), and the division guards.
"""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING, Any, cast
from uuid import uuid4
import pytest
import pytest_asyncio
from roboco.db.tables import (
AgentSpawnSessionTable,
AgentTable,
MemberPerformanceDailyTable,
ProjectTable,
TaskTable,
)
from roboco.models.base import (
AgentRole,
AgentStatus,
Complexity,
TaskNature,
TaskStatus,
TaskType,
Team,
)
from roboco.services.metrics import MetricsService
if TYPE_CHECKING:
from collections.abc import AsyncIterator
from uuid import UUID
from sqlalchemy.ext.asyncio import AsyncSession
_TODAY = datetime.now(UTC).date()
_TOTAL_COMPLETED = 3
_ROLLUP_COMPLETED = 2
_OVERLAY_TURNS = 3
_ROLLUP_ONLY_TURNS = 5 # closed session already in the rollup, not re-added
_ORG_MEMBERS = 2
_ORG_COMPLETED = 3
def _daily(slug: str, **over: Any) -> MemberPerformanceDailyTable:
base: dict[str, Any] = {
"id": uuid4(),
"date": _TODAY,
"member_kind": "agent",
"agent_slug": slug,
"team": Team.BACKEND.value,
"role": "developer",
}
base.update(over)
return MemberPerformanceDailyTable(**base)
def _agent(role: AgentRole, slug: str) -> AgentTable:
return AgentTable(
id=uuid4(),
name=slug,
slug=slug,
role=role,
team=Team.BACKEND,
status=AgentStatus.ACTIVE,
model_config={},
system_prompt="x",
capabilities=[],
permissions={},
metrics={},
)
@pytest_asyncio.fixture
async def svc(db_session: AsyncSession) -> AsyncIterator[MetricsService]:
yield MetricsService(db_session)
@pytest.mark.asyncio
async def test_member_scorecard_rollup_and_derived(
svc: MetricsService, db_session: AsyncSession
) -> None:
dev = _agent(AgentRole.DEVELOPER, f"be-dev-{uuid4().hex[:6]}")
db_session.add(dev)
await db_session.flush()
db_session.add_all(
[
_daily(
dev.slug,
tasks_completed=2,
tasks_first_pass=1,
active_runtime_seconds=1800,
turns=6,
tool_calls=12,
tokens=100,
cost_usd=1.0,
qa_reviews_total=3,
qa_reviews_passed=2,
escalations=1,
blocked_others=1,
idle_seconds=600,
revisions_caused=1,
revisions_received=1,
),
_daily(
dev.slug,
date=_TODAY - timedelta(days=1),
tasks_completed=1,
tasks_first_pass=1,
active_runtime_seconds=1800,
turns=4,
tool_calls=8,
tokens=50,
cost_usd=0.5,
qa_reviews_total=2,
qa_reviews_passed=2,
idle_seconds=1200,
),
]
)
await db_session.flush()
card = await svc.get_member_scorecard(cast("UUID", dev.id), days=30)
assert card is not None
assert card.tasks_completed == _TOTAL_COMPLETED
assert card.first_pass_yield == pytest.approx(2 / 3, abs=1e-4) # 2 of 3
# 3 tasks over 3600s = 1h -> 3.0/hr.
assert card.effort_throughput_per_hour == pytest.approx(3.0)
assert (card.turns, card.tool_calls) == (10, 20)
assert card.turns_per_task == pytest.approx(10 / 3, abs=1e-4)
assert card.qa_pass_rate == pytest.approx(4 / 5, abs=1e-4) # 4 of 5
assert card.escalations == 1
assert card.blocked_others == 1
# util = 3600 active / (3600 + 1800 idle) = 0.6667.
assert card.utilization == pytest.approx(3600 / 5400, abs=1e-4)
assert card.includes_live_inflight is False
@pytest.mark.asyncio
async def test_live_overlay_enriches_effort_not_completion(
svc: MetricsService, db_session: AsyncSession
) -> None:
dev = _agent(AgentRole.DEVELOPER, f"be-dev-{uuid4().hex[:6]}")
db_session.add(dev)
await db_session.flush()
project = ProjectTable(
id=uuid4(),
name="P",
slug=f"p-{uuid4().hex[:6]}",
git_url="https://example.com/r.git",
assigned_cell=Team.BACKEND,
created_by=dev.id,
)
db_session.add(project)
await db_session.flush()
db_session.add(_daily(dev.slug, tasks_completed=2, active_runtime_seconds=100))
# A non-terminal (in-flight) task with a spawn stint -> overlay effort.
inflight = TaskTable(
id=uuid4(),
title="t",
description="d",
acceptance_criteria=["ac"],
task_type=TaskType.CODE,
nature=TaskNature.TECHNICAL,
status=TaskStatus.IN_PROGRESS,
team=Team.BACKEND,
project_id=project.id,
created_by=dev.id,
assigned_to=dev.id,
estimated_complexity=Complexity.MEDIUM,
started_at=datetime.now(UTC) - timedelta(hours=1),
)
db_session.add(inflight)
await db_session.flush()
now = datetime.now(UTC)
# OPEN (still-running) session — the live delta the rollup cannot hold yet.
db_session.add(
AgentSpawnSessionTable(
id=uuid4(),
agent_slug=dev.slug,
team="backend",
role="developer",
model="claude",
task_id=str(inflight.id),
started_at=now - timedelta(seconds=200),
ended_at=None,
turns=3,
tool_calls=4,
tokens_input=10,
tokens_output=5,
estimated_cost_usd=0.2,
)
)
await db_session.flush()
card = await svc.get_member_scorecard(cast("UUID", dev.id), days=30)
assert card is not None
assert card.tasks_completed == _ROLLUP_COMPLETED # in-flight NOT completed
assert card.includes_live_inflight is True
assert card.active_runtime_hours > 100 / 3600 # rollup + overlay effort
assert card.turns == _OVERLAY_TURNS # from the overlay (rollup row had 0)
@pytest.mark.asyncio
async def test_live_overlay_excludes_closed_session_no_double_count(
svc: MetricsService, db_session: AsyncSession
) -> None:
"""A CLOSED session on a non-terminal task is already in the daily rollup
(via _msweep_spawn, which counts ended_at IS NOT NULL). The overlay must NOT
re-add it, or the member's effort/turns double-count on the common
reap/respawn path."""
dev = _agent(AgentRole.DEVELOPER, f"be-dev-{uuid4().hex[:6]}")
db_session.add(dev)
await db_session.flush()
project = ProjectTable(
id=uuid4(),
name="P",
slug=f"p-{uuid4().hex[:6]}",
git_url="https://example.com/r.git",
assigned_cell=Team.BACKEND,
created_by=dev.id,
)
db_session.add(project)
await db_session.flush()
# Rollup row already reflects the closed session (turns=5, active=300s).
db_session.add(
_daily(dev.slug, tasks_completed=0, active_runtime_seconds=300, turns=5)
)
inflight = TaskTable(
id=uuid4(),
title="t",
description="d",
acceptance_criteria=["ac"],
task_type=TaskType.CODE,
nature=TaskNature.TECHNICAL,
status=TaskStatus.IN_PROGRESS,
team=Team.BACKEND,
project_id=project.id,
created_by=dev.id,
assigned_to=dev.id,
estimated_complexity=Complexity.MEDIUM,
started_at=datetime.now(UTC) - timedelta(hours=1),
)
db_session.add(inflight)
await db_session.flush()
now = datetime.now(UTC)
db_session.add(
AgentSpawnSessionTable(
id=uuid4(),
agent_slug=dev.slug,
team="backend",
role="developer",
model="claude",
task_id=str(inflight.id),
started_at=now - timedelta(seconds=300),
ended_at=now, # CLOSED — already counted by the rollup
turns=5,
tool_calls=4,
tokens_input=10,
tokens_output=5,
estimated_cost_usd=0.2,
)
)
await db_session.flush()
card = await svc.get_member_scorecard(cast("UUID", dev.id), days=30)
assert card is not None
assert card.turns == _ROLLUP_ONLY_TURNS # closed session is NOT re-added
assert card.active_runtime_hours == pytest.approx(300 / 3600, abs=1e-4)
assert card.includes_live_inflight is False # no OPEN session
@pytest.mark.asyncio
async def test_member_scorecard_404_and_guards(
svc: MetricsService, db_session: AsyncSession
) -> None:
assert await svc.get_member_scorecard(uuid4()) is None
# An agent with no rollup rows: division guards -> None, no crash.
dev = _agent(AgentRole.DEVELOPER, f"be-dev-{uuid4().hex[:6]}")
db_session.add(dev)
await db_session.flush()
card = await svc.get_member_scorecard(cast("UUID", dev.id), days=30)
assert card is not None
assert card.tasks_completed == 0
assert card.first_pass_yield is None
assert card.effort_throughput_per_hour is None
assert card.qa_pass_rate is None
assert card.utilization is None
@pytest.mark.asyncio
async def test_org_scorecard_aggregates_members(
svc: MetricsService, db_session: AsyncSession
) -> None:
s1, s2 = f"be-dev-{uuid4().hex[:6]}", f"be-dev-{uuid4().hex[:6]}"
db_session.add_all(
[
_daily(
s1, tasks_completed=2, tasks_first_pass=2, active_runtime_seconds=3600
),
_daily(
s2, tasks_completed=1, tasks_first_pass=0, active_runtime_seconds=3600
),
]
)
await db_session.flush()
org = await svc.get_org_scorecard(team=Team.BACKEND, days=30)
assert org.scope == "team"
assert org.member_count == _ORG_MEMBERS
assert org.tasks_completed == _ORG_COMPLETED
assert org.first_pass_yield == pytest.approx(2 / 3, abs=1e-4)
+244
View File
@@ -0,0 +1,244 @@
"""get_task_metrics — granular per-task effort against a real Postgres.
Seeds a task's audit-log journey + agent spawn stints (with turns/tool_calls/
tokens/cost) + named qa/pr fail events, then asserts the composed metrics:
summed effort vs wall-clock, turns/tool_calls/tokens/cost, per-stage
active-vs-wait, and who-caused-rework.
"""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING, Any, NamedTuple
from uuid import uuid4
import pytest
import pytest_asyncio
from roboco.db.tables import (
AgentSpawnSessionTable,
AgentTable,
AuditLogTable,
ProjectTable,
TaskTable,
)
from roboco.models.base import (
AgentRole,
AgentStatus,
Complexity,
TaskNature,
TaskStatus,
TaskType,
Team,
)
from roboco.services.metrics import MetricsService
if TYPE_CHECKING:
from collections.abc import AsyncIterator
from sqlalchemy.ext.asyncio import AsyncSession
_T0 = datetime(2026, 6, 20, 12, 0, 0, tzinfo=UTC)
def _sec(n: int) -> datetime:
return _T0 + timedelta(seconds=n)
def _agent(role: AgentRole, slug: str) -> AgentTable:
return AgentTable(
id=uuid4(),
name=slug,
slug=slug,
role=role,
team=Team.BACKEND,
status=AgentStatus.ACTIVE,
model_config={},
system_prompt="x",
capabilities=[],
permissions={},
metrics={},
)
def _audit(
task_id: Any,
status: str,
ts: datetime,
*,
agent_id: Any = None,
event_type: str | None = None,
) -> AuditLogTable:
return AuditLogTable(
id=uuid4(),
event_type=event_type or f"task.{status}",
agent_id=agent_id,
target_type="task",
target_id=task_id,
severity="info",
details={"to_status": status, "from_status": "prev", "team": "backend"},
timestamp=ts,
)
class _Usage(NamedTuple):
turns: int
tool_calls: int
tokens_in: int
tokens_out: int
cost: float
def _spawn(
task_id: str,
started: datetime,
ended: datetime | None,
usage: _Usage,
) -> AgentSpawnSessionTable:
return AgentSpawnSessionTable(
id=uuid4(),
agent_slug="be-dev-1",
team="backend",
role="developer",
model="claude",
task_id=task_id,
started_at=started,
ended_at=ended,
turns=usage.turns,
tool_calls=usage.tool_calls,
tokens_input=usage.tokens_in,
tokens_output=usage.tokens_out,
estimated_cost_usd=usage.cost,
)
@pytest_asyncio.fixture
async def setup(db_session: AsyncSession) -> AsyncIterator[dict]:
dev = _agent(AgentRole.DEVELOPER, f"be-dev-{uuid4().hex[:6]}")
qa = _agent(AgentRole.QA, f"be-qa-{uuid4().hex[:6]}")
db_session.add_all([dev, qa])
await db_session.flush()
project = ProjectTable(
id=uuid4(),
name="P",
slug=f"p-{uuid4().hex[:6]}",
git_url="https://example.com/r.git",
assigned_cell=Team.BACKEND,
created_by=dev.id,
)
db_session.add(project)
await db_session.flush()
yield {
"svc": MetricsService(db_session),
"db": db_session,
"project_id": project.id,
"dev_id": dev.id,
"qa_id": qa.id,
}
@pytest.mark.asyncio
async def test_returns_none_for_missing_task(setup: dict) -> None:
assert await setup["svc"].get_task_metrics(uuid4()) is None
@pytest.mark.asyncio
async def test_composes_effort_turns_stages_and_rework(setup: dict) -> None:
db = setup["db"]
tid = uuid4()
db.add(
TaskTable(
id=tid,
title="t",
description="d",
acceptance_criteria=["ac"],
task_type=TaskType.CODE,
nature=TaskNature.TECHNICAL,
status=TaskStatus.COMPLETED,
team=Team.BACKEND,
project_id=setup["project_id"],
created_by=setup["dev_id"],
assigned_to=setup["dev_id"],
revision_count=2,
estimated_complexity=Complexity.MEDIUM,
started_at=_T0,
completed_at=_sec(7200),
)
)
db.add_all(
[
_audit(tid, "claimed", _T0),
_audit(tid, "in_progress", _sec(60)),
_audit(tid, "awaiting_qa", _sec(3660)),
_audit(tid, "completed", _sec(7200)),
_audit(tid, "needs_revision", _sec(3660), event_type="task.qa_fail"),
_audit(tid, "needs_revision", _sec(3000), event_type="task.pr_fail"),
]
)
db.add_all(
[
_spawn(str(tid), _T0, _sec(600), _Usage(5, 10, 100, 50, 1.0)),
_spawn(str(tid), _sec(3600), _sec(3660), _Usage(3, 4, 20, 10, 0.5)),
]
)
await db.flush()
m = await setup["svc"].get_task_metrics(tid)
assert m is not None
# summed effort (600 + 60) vs wall-clock (2h).
expected_active_s = 660
expected_wall_s = 7200
assert m.active_runtime_seconds == expected_active_s
assert m.wall_clock_seconds == expected_wall_s
assert (m.turns, m.tool_calls, m.tokens) == (8, 14, 180)
assert m.cost_usd == pytest.approx(1.5)
assert (m.revision_count, m.qa_fails, m.pr_fails, m.stints) == (2, 1, 1, 2)
stages = {s.status: s for s in m.stages}
# claimed [0,60): stint1 covers it fully.
assert (stages["claimed"].active_seconds, stages["claimed"].wait_seconds) == (60, 0)
# in_progress [60,3660): stint1 60..600 (540) + stint2 3600..3660 (60) = 600 active.
assert (
stages["in_progress"].active_seconds,
stages["in_progress"].wait_seconds,
) == (600, 3000)
# awaiting_qa [3660,7200): no stint running -> all wait.
assert (
stages["awaiting_qa"].active_seconds,
stages["awaiting_qa"].wait_seconds,
) == (0, 3540)
@pytest.mark.asyncio
async def test_in_flight_open_stint_and_open_window_decompose(setup: dict) -> None:
db = setup["db"]
tid = uuid4()
db.add(
TaskTable(
id=tid,
title="t",
description="d",
acceptance_criteria=["ac"],
task_type=TaskType.CODE,
nature=TaskNature.TECHNICAL,
status=TaskStatus.IN_PROGRESS,
team=Team.BACKEND,
project_id=setup["project_id"],
created_by=setup["dev_id"],
assigned_to=setup["dev_id"],
estimated_complexity=Complexity.MEDIUM,
started_at=_T0,
completed_at=None,
)
)
db.add_all([_audit(tid, "claimed", _T0), _audit(tid, "in_progress", _sec(60))])
# An OPEN stint (ended_at=None) -> runs to now.
db.add(_spawn(str(tid), _T0, None, _Usage(2, 3, 1, 1, 0.1)))
await db.flush()
m = await setup["svc"].get_task_metrics(tid)
assert m is not None
assert m.stints == 1
assert m.active_runtime_seconds > 0 # open stint ran to now
assert m.wall_clock_seconds > 0 # open task -> now
# The open final window (in_progress) still decomposes.
assert "in_progress" in {s.status for s in m.stages}
@@ -15,6 +15,7 @@ import pytest
import pytest_asyncio
from roboco.db.tables import (
AgentTable,
AuditLogTable,
JournalEntryTable,
ProductTable,
ProjectTable,
@@ -1199,6 +1200,29 @@ async def test_ceo_reject_routes_coordination_task_to_main_pm(
assert rejected.assigned_to == main_pm_id
assert rejected.claimed_by is None
# Regression (metrics-granularity Phase 3): the coordination-root reject
# routes through admin_set_status, so it MUST still emit a CEO-attributed
# audit row transitioning OUT of awaiting_ceo_approval — the signal the CEO
# scorecard pairs for approval latency + counts as a god-mode action. If a
# future refactor drops the audit (the old gap), this fails.
audit_rows = (
(
await db_session.execute(
select(AuditLogTable).where(AuditLogTable.target_id == task.id)
)
)
.scalars()
.all()
)
ceo_rows = [
a
for a in audit_rows
if (a.details or {}).get("agent_role") == "ceo"
and (a.details or {}).get("from_status") == "awaiting_ceo_approval"
]
assert ceo_rows, "coordination ceo_reject must emit a ceo-attributed audit row"
assert ceo_rows[0].details.get("to_status") == "pending"
@pytest.mark.asyncio
async def test_ceo_reject_routes_batch_umbrella_to_main_pm(
@@ -0,0 +1,95 @@
"""sum_transcript_usage — token + turn counts from a Claude Code JSONL transcript.
The 5th return value is the LLM turn count: the number of UNIQUE assistant
``message.id``s (Claude Code logs one line per content block, all sharing the
message id, so naive line-counting would inflate both tokens and turns).
"""
from __future__ import annotations
import json
from typing import TYPE_CHECKING
from roboco.agent_sdk.transcript_usage import sum_transcript_usage
if TYPE_CHECKING:
from pathlib import Path
_EXPECTED_TUPLE_LEN = 5
def _line(msg_id: str | None, **usage: int) -> str:
msg: dict[str, object] = {"usage": usage}
if msg_id is not None:
msg["id"] = msg_id
return json.dumps({"message": msg})
def _write(path: Path, lines: list[str]) -> None:
path.write_text("\n".join(lines) + "\n", encoding="utf-8")
def test_returns_five_tuple(tmp_path: Path) -> None:
f = tmp_path / "t.jsonl"
_write(f, [_line("m1", input_tokens=10, output_tokens=5)])
result = sum_transcript_usage(f)
assert len(result) == _EXPECTED_TUPLE_LEN
def test_turns_counts_unique_message_ids(tmp_path: Path) -> None:
f = tmp_path / "t.jsonl"
_write(
f,
[
_line("m1", input_tokens=10, output_tokens=5),
_line("m2", input_tokens=20, output_tokens=7),
_line("m3", input_tokens=1, output_tokens=1),
],
)
_in, _out, _cr, _cw, turns = sum_transcript_usage(f)
expected_turns = 3
assert turns == expected_turns
def test_repeated_message_id_counts_one_turn_and_one_usage(tmp_path: Path) -> None:
# Claude Code emits one line per content block of the SAME assistant message,
# each repeating the usage — must count once for tokens AND turns.
f = tmp_path / "t.jsonl"
_write(
f,
[
_line("m1", input_tokens=10, output_tokens=5),
_line("m1", input_tokens=10, output_tokens=5),
_line("m1", input_tokens=10, output_tokens=5),
],
)
tin, tout, _cr, _cw, turns = sum_transcript_usage(f)
assert (tin, tout, turns) == (10, 5, 1)
def test_malformed_lines_skipped_without_losing_turn_count(tmp_path: Path) -> None:
f = tmp_path / "t.jsonl"
_write(
f,
[
_line("m1", input_tokens=10, output_tokens=5),
"not json at all {{{",
"",
_line("m2", input_tokens=2, output_tokens=2),
],
)
tin, _out, _cr, _cw, turns = sum_transcript_usage(f)
assert (tin, turns) == (12, 2)
def test_usage_line_without_id_sums_tokens_but_not_a_turn(tmp_path: Path) -> None:
f = tmp_path / "t.jsonl"
_write(
f,
[
_line(None, input_tokens=4, output_tokens=1),
_line("m1", input_tokens=6, output_tokens=1),
],
)
tin, _out, _cr, _cw, turns = sum_transcript_usage(f)
assert (tin, turns) == (10, 1)
+25 -3
View File
@@ -139,7 +139,11 @@ def test_missing_transcript_returns_zero_without_error(
"/usage/sync", json={"transcript_path": str(tmp_path / "nope.jsonl")}
)
assert resp.status_code == _OK
assert resp.json() == _expected([])
body = resp.json()
for key, value in _expected([]).items():
assert body[key] == value
assert body["turns"] == 0
assert body["tool_calls"] == 0
def test_malformed_lines_are_skipped(client: TestClient, tmp_path: Path) -> None:
@@ -167,7 +171,7 @@ def test_parser_handles_entries_without_message(tmp_path: Path) -> None:
json.dumps({"type": "system", "subtype": "init"}),
_assistant_line(rows[0]),
)
tin, tout, cread, cwrite = srv._sum_transcript_usage(transcript)
tin, tout, cread, cwrite, turns = srv._sum_transcript_usage(transcript)
exp = _expected(rows)
assert (tin, tout, cread, cwrite) == (
exp["tokens_input"],
@@ -175,6 +179,7 @@ def test_parser_handles_entries_without_message(tmp_path: Path) -> None:
exp["tokens_cache_read"],
exp["tokens_cache_write"],
)
assert turns == 0 # _assistant_line carries no message id
def _assistant_line_with_id(row: _UsageRow, message_id: str) -> str:
@@ -214,7 +219,7 @@ def test_parser_dedupes_repeated_message_id(tmp_path: Path) -> None:
_assistant_line_with_id(msg, "msg_aaa"), # tool_use block (same id)
_assistant_line_with_id(other, "msg_bbb"),
)
tin, tout, cread, cwrite = srv._sum_transcript_usage(transcript)
tin, tout, cread, cwrite, turns = srv._sum_transcript_usage(transcript)
# Counted once per id: msg + other, NOT msg * 3 + other.
exp = _expected([msg, other])
assert (tin, tout, cread, cwrite) == (
@@ -223,3 +228,20 @@ def test_parser_dedupes_repeated_message_id(tmp_path: Path) -> None:
exp["tokens_cache_read"],
exp["tokens_cache_write"],
)
expected_turns = 2 # two unique message ids
assert turns == expected_turns
def test_sync_response_surfaces_turns(client: TestClient, tmp_path: Path) -> None:
"""/usage/sync (and thus /usage/status) reports the LLM turn count."""
transcript = tmp_path / "session.jsonl"
_write(
transcript,
_assistant_line_with_id((10, 5, 0, 0), "msg_a"),
_assistant_line_with_id((10, 5, 0, 0), "msg_a"), # same id
_assistant_line_with_id((2, 1, 0, 0), "msg_b"),
)
body = client.post("/usage/sync", json={"transcript_path": str(transcript)}).json()
expected_turns = 2
assert body["turns"] == expected_turns
assert "tool_calls" in body
@@ -0,0 +1,60 @@
"""The member_performance_daily rollup table — schema shape."""
from __future__ import annotations
from typing import TYPE_CHECKING, cast
from roboco.db.tables import MemberPerformanceDailyTable
from sqlalchemy import UniqueConstraint
if TYPE_CHECKING:
from sqlalchemy import Table
_TABLE = cast("Table", MemberPerformanceDailyTable.__table__)
def test_table_name() -> None:
assert MemberPerformanceDailyTable.__tablename__ == "member_performance_daily"
def test_has_all_metric_columns_including_extras() -> None:
cols = set(_TABLE.columns.keys())
assert {
# core
"date",
"member_kind",
"agent_slug",
"team",
"role",
"tasks_completed",
"tasks_first_pass",
"revisions_caused",
"revisions_received",
"active_runtime_seconds",
"turns",
"tool_calls",
"tokens",
"cost_usd",
"ceo_approval_dwell_seconds",
"ceo_unblock_dwell_seconds",
"godmode_actions",
# the 4 CEO-approved extras + blocked_seconds
"qa_reviews_total",
"qa_reviews_passed",
"escalations",
"blocked_others",
"idle_seconds",
"blocked_seconds",
} <= cols
def test_natural_key_is_unique() -> None:
uniques = [c for c in _TABLE.constraints if isinstance(c, UniqueConstraint)]
key_sets = [{col.name for col in u.columns} for u in uniques]
assert {"date", "member_kind", "agent_slug"} in key_sets
def test_agent_slug_not_nullable() -> None:
# NOT NULL DEFAULT '' — else the CEO row (agent_slug NULL) would duplicate
# under Postgres' NULL-distinct UNIQUE semantics.
assert _TABLE.columns["agent_slug"].nullable is False
+37
View File
@@ -267,6 +267,43 @@ async def test_undecodable_message_is_acked_and_dead_lettered() -> None:
assert invoked == []
class _FakeRecoverRedis:
"""Fake whose xpending_range returns the message id as BYTES (the real
client has no decode_responses), and which captures the ids XCLAIM gets."""
def __init__(self, message_id: bytes) -> None:
self._message_id = message_id
self.claimed_ids: list[object] = []
async def xpending(self, *args: object, **kwargs: object) -> dict:
del args, kwargs
return {"pending": 1}
async def xpending_range(self, *args: object, **kwargs: object) -> list:
del args, kwargs
return [{"message_id": self._message_id, "time_since_delivered": 10_000}]
async def xclaim(self, *args: object, **kwargs: object) -> list:
del args
self.claimed_ids = cast("list[object]", kwargs.get("message_ids") or [])
return [] # nothing claimed back → no handling
@pytest.mark.asyncio
async def test_recover_stream_decodes_bytes_message_id_for_xclaim() -> None:
"""xpending_range returns the message id as bytes; _recover_stream must
decode it before XCLAIM. A raw ``str(bytes)`` yields ``"b'1782..-0'"``,
which Redis rejects with "Unrecognized XCLAIM option", so pending-message
recovery silently fails every reclaim tick."""
bus = StreamEventBus()
fake = _FakeRecoverRedis(b"1782066556728-0")
bus._redis = cast("Redis", fake)
await bus._recover_stream("roboco:stream:usage", idle_time_ms=0)
assert fake.claimed_ids == ["1782066556728-0"] # decoded, not "b'...'"
# --- periodic reclaim: a runtime handler failure is retried without a restart ---
+100
View File
@@ -0,0 +1,100 @@
"""compute_stage_effort — split each stage window into active vs wait seconds.
Pure overlap math (no DB): given a stage's [start, end) window and the agent
spawn stints that ran during the task, ``active`` is the wall-clock time during
which AT LEAST ONE stint was running (overlapping stints merged, so active can
never exceed the window), and ``wait`` is the remainder. This is the wall-clock
decomposition distinct from summed effort (Σ stint durations), which can
exceed wall-clock when stints run concurrently.
"""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from roboco.foundation.policy.stage_effort import StageEffort, compute_stage_effort
_BASE = datetime(2026, 7, 1, 12, 0, 0, tzinfo=UTC)
def _at(seconds: int) -> datetime:
return _BASE + timedelta(seconds=seconds)
def _window(status: str, start_s: int, end_s: int) -> tuple[str, datetime, datetime]:
return (status, _at(start_s), _at(end_s))
def _stint(start_s: int, end_s: int) -> tuple[datetime, datetime]:
return (_at(start_s), _at(end_s))
def _only(windows: list, stints: list) -> StageEffort:
result = compute_stage_effort(windows, stints)
assert len(result) == 1
return result[0]
def test_disjoint_stint_is_all_wait() -> None:
eff = _only([_window("in_progress", 0, 100)], [_stint(200, 300)])
assert (eff.active_seconds, eff.wait_seconds) == (0, 100)
def test_fully_nested_stint() -> None:
eff = _only([_window("in_progress", 0, 100)], [_stint(20, 50)])
assert (eff.active_seconds, eff.wait_seconds) == (30, 70)
def test_partial_overlap_clips_to_window() -> None:
# stint runs 80..150 but window ends at 100 -> only 20s active in-window.
eff = _only([_window("in_progress", 0, 100)], [_stint(80, 150)])
assert (eff.active_seconds, eff.wait_seconds) == (20, 80)
def test_multiple_nonoverlapping_stints_sum() -> None:
eff = _only(
[_window("in_progress", 0, 100)],
[_stint(0, 10), _stint(40, 60)],
)
assert (eff.active_seconds, eff.wait_seconds) == (30, 70)
def test_overlapping_stints_are_merged_not_double_counted() -> None:
# [10,40) and [30,60) overlap -> merged union is [10,60) = 50s, NOT 60s.
eff = _only(
[_window("in_progress", 0, 100)],
[_stint(10, 40), _stint(30, 60)],
)
# merged union [10,60) = 50s active, 50s wait (NOT 60s from double-count).
assert (eff.active_seconds, eff.wait_seconds) == (50, 50)
def test_active_never_exceeds_window_length() -> None:
eff = _only(
[_window("in_progress", 0, 100)],
[_stint(-50, 500)], # stint dwarfs the window
)
assert (eff.active_seconds, eff.wait_seconds) == (100, 0)
def test_zero_length_window() -> None:
eff = _only([_window("claimed", 50, 50)], [_stint(0, 100)])
assert (eff.active_seconds, eff.wait_seconds) == (0, 0)
def test_each_window_decomposes_independently() -> None:
windows = [_window("claimed", 0, 100), _window("in_progress", 100, 300)]
stints = [_stint(50, 250)] # spans both windows
result = compute_stage_effort(windows, stints)
by_status = {e.status: e for e in result}
claimed = by_status["claimed"]
in_progress = by_status["in_progress"]
assert (claimed.active_seconds, claimed.wait_seconds) == (50, 50)
# in_progress: stint covers 100..250 of the 100..300 window.
assert (in_progress.active_seconds, in_progress.wait_seconds) == (150, 50)
def test_to_dict_shape() -> None:
eff = _only([_window("in_progress", 0, 100)], [_stint(20, 50)])
d = eff.to_dict()
assert d == {"status": "in_progress", "active_seconds": 30, "wait_seconds": 70}
@@ -269,7 +269,7 @@ async def test_finalize_spawn_session_http_error_uses_zero_tokens() -> None:
with (
patch("roboco.runtime.orchestrator.httpx.AsyncClient", _client_cls),
patch.object(orch, "_usage_from_transcript", return_value=(0, 0, 0, 0)),
patch.object(orch, "_usage_from_transcript", return_value=(0, 0, 0, 0, 0)),
patch("roboco.db.base.get_session_factory", return_value=db_factory),
patch("roboco.billing.pricing.calculate_cost", return_value=0.0) as mock_cost,
):
@@ -303,7 +303,7 @@ async def test_finalize_spawn_session_non_200_uses_zero_tokens() -> None:
with (
patch("roboco.runtime.orchestrator.httpx.AsyncClient", _client_cls),
patch.object(orch, "_usage_from_transcript", return_value=(0, 0, 0, 0)),
patch.object(orch, "_usage_from_transcript", return_value=(0, 0, 0, 0, 0)),
patch("roboco.db.base.get_session_factory", return_value=db_factory),
patch("roboco.billing.pricing.calculate_cost", return_value=0.0) as mock_cost,
):
@@ -389,7 +389,7 @@ async def test_sweep_token_snapshots_skips_zero_token_agents() -> None:
with (
patch("roboco.runtime.orchestrator.httpx.AsyncClient", _client_cls),
patch.object(orch, "_usage_from_transcript", return_value=(0, 0, 0, 0)),
patch.object(orch, "_usage_from_transcript", return_value=(0, 0, 0, 0, 0)),
patch("roboco.db.base.get_session_factory", return_value=db_factory),
):
await orch._sweep_token_snapshots()
@@ -765,7 +765,9 @@ async def test_resolve_active_tokens_falls_back_to_transcript() -> None:
)
client = _FakeHTTPClient(_handler)
with patch.object(orch, "_usage_from_transcript", return_value=(6, 514, 100, 50)):
with patch.object(
orch, "_usage_from_transcript", return_value=(6, 514, 100, 50, 3)
):
tokens = await orch._resolve_active_tokens(
cast("httpx.AsyncClient", client), _AGENT_ID
)
@@ -790,7 +792,7 @@ async def test_resolve_active_tokens_prefers_sdk() -> None:
client = _FakeHTTPClient(_handler)
with patch.object(
orch, "_usage_from_transcript", return_value=(999, 999, 999, 999)
orch, "_usage_from_transcript", return_value=(999, 999, 999, 999, 0)
) as mock_tx:
tokens = await orch._resolve_active_tokens(
cast("httpx.AsyncClient", client), _AGENT_ID
@@ -800,6 +802,50 @@ async def test_resolve_active_tokens_prefers_sdk() -> None:
mock_tx.assert_not_called()
async def test_resolve_final_turns_tools_from_sdk() -> None:
"""turns + tool_calls come from the SDK /usage/status when present."""
orch = _make_orchestrator()
def _handler(_url: str) -> Any:
return _mock_response(200, {"turns": 7, "tool_calls": 42, "tokens_input": 1})
with patch(
"roboco.runtime.orchestrator.httpx.AsyncClient",
lambda **_kw: _FakeHTTPClient(_handler),
):
turns, tool_calls = await orch._resolve_final_turns_tools(_AGENT_ID)
assert (turns, tool_calls) == (7, 42)
async def test_resolve_final_turns_tools_transcript_fallback_for_turns() -> None:
"""When the SDK reports 0 turns, fall back to the transcript turn count.
tool_calls has no transcript equivalent and stays 0 ("n/a").
"""
orch = _make_orchestrator()
def _handler(_url: str) -> Any:
return _mock_response(200, {"turns": 0, "tool_calls": 0})
transcript_turns = 9
with (
patch(
"roboco.runtime.orchestrator.httpx.AsyncClient",
lambda **_kw: _FakeHTTPClient(_handler),
),
patch.object(
orch,
"_usage_from_transcript",
return_value=(1, 2, 3, 4, transcript_turns),
),
):
turns, tool_calls = await orch._resolve_final_turns_tools(_AGENT_ID)
assert turns == transcript_turns # recovered from the transcript
assert tool_calls == 0
# ---------------------------------------------------------------------------
# _usage_from_transcript — locate by session id across any project dir
# ---------------------------------------------------------------------------
@@ -836,7 +882,7 @@ def test_usage_from_transcript_finds_by_session_id_in_shared_app_dir(
monkeypatch.setattr(Path, "home", lambda: tmp_path)
result = AgentOrchestrator._usage_from_transcript("main-pm", sid)
assert result == (exp_in, exp_out, exp_cr, exp_cw)
assert result == (exp_in, exp_out, exp_cr, exp_cw, 1) # one message => 1 turn
def test_usage_from_transcript_without_session_id_uses_slug_glob(
@@ -859,4 +905,4 @@ def test_usage_from_transcript_without_session_id_uses_slug_glob(
monkeypatch.setattr(Path, "home", lambda: tmp_path)
result = AgentOrchestrator._usage_from_transcript("be-dev-1")
assert result == (exp_in, exp_out, 0, 0)
assert result == (exp_in, exp_out, 0, 0, 1) # one message => 1 turn
Generated
+1 -1
View File
@@ -2404,7 +2404,7 @@ wheels = [
[[package]]
name = "roboco"
version = "0.14.0"
version = "0.15.0"
source = { editable = "." }
dependencies = [
{ name = "alembic" },