mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
v0.15.0: Metrics granularity — per-member / per-task / org + CEO scorecards (#289)
* feat(metrics): capture per-session turns + tool_calls (phase 1)
Persist LLM iterations (turns) and tool invocations per agent spawn session,
the raw signal the granular per-member performance metrics build on (real
effort/iterations vs wall-clock).
- sum_transcript_usage returns a 5-tuple adding turns = unique assistant
message-id count; _usage_from_transcript + _resolve_active_tokens updated to
the 5-tuple (active-tokens keeps its 4-tuple contract by slicing).
- SDK: _SessionState.turns, set by /usage/sync; /usage/status (TokenUsageStatus)
now carries turns + tool_calls (= total_calls).
- orchestrator: new _resolve_final_turns_tools (SDK primary, transcript fallback
for turns only; Grok -> 0/0) wired into _finalize_spawn_session, which writes
turns + tool_calls to agent_spawn_sessions.
- migration 055 adds turns + tool_calls (BigInteger DEFAULT 0 -> historical/Grok
rows read 0, surfaced as n/a). Verified real alembic upgrade/downgrade.
Part of metrics-granularity (v0.15.0); recon-adjusted plan on disk.
* feat(metrics): pure compute_stage_effort helper (phase 2, part 1)
Foundation-layer overlap math (no DB): split each task status window into
active (merged wall-clock overlap of spawn stints — concurrent stints counted
once, so active <= window) vs wait (queue/review idle). Distinct from summed
effort. The per-task metrics service will feed it audit-log windows + spawn
stints. 9 unit tests (disjoint/nested/partial/merged/clamped/zero/multi-window).
* feat(metrics): per-task live metrics + GET /metrics/task/{id} (phase 2)
TaskMetrics dataclass + MetricsService.get_task_metrics: summed spawn effort
(vs wall-clock), turns/tool_calls/tokens/cost, per-stage active-vs-wait
(compute_stage_effort over audit windows x spawn stints), and who-caused-rework
(revision_count + named qa/pr fail events). Open stints and the open final
stage window close at completed_at for a terminal task (else now), so stages
don't grow past completion. Exposed at GET /dashboard/metrics/task/{task_id}
(404 if absent). Real-PG tests (compose/none/in-flight) + route tests (200/404).
* feat(metrics): CEO-as-member scorecard + ceo_reject audit regression (phase 3)
The human CEO is a measured member, read purely from audit_log (agent_role='ceo'
serializes from the CEO StrEnum): approval dwell (awaiting_ceo_approval -> a CEO
decision, incl. the coordination-root reject that lands in pending), unblock
dwell (blocked -> a CEO revive), and god-mode action count (every CEO-attributed
transition). CeoScorecard + MetricsService.get_ceo_scorecard (p50/p90 via
PERCENTILE_CONT, expanding IN for the decision sets) + GET
/dashboard/metrics/member/ceo (declared before any future member/{id} route).
The ceo_reject coordination-root audit gap the plan meant to close was already
closed by the gap-sweep (routes through admin_set_status -> agent_role='ceo'
audit); locked with a regression assertion in the existing coordination-reject
test. Real-PG tests: approval/unblock/godmode, non-ceo exclusion, empty->zeros.
* feat(metrics): audit instrumentation for escalations/blocked-others/idle (phase 4a)
The three extra per-member metrics that had no data source get durable,
in-session audit events (additive; never gate the underlying action):
- apply_escalation -> task.escalated (details.escalator_slug) on both the
normal block path and the pool-divert path -> escalations count.
- _unblock_dependents -> task.unblocked_dependents (details.count) on the
completed BLOCKER task, captured before the dependency edges are pruned ->
blocked-others count (sweeper attributes to the blocker's owner).
- mark_agent_idle -> agent.idle (details.agent_slug) -> idle/utilization (the
sweeper pairs an idle mark to the member's next spawn for idle duration).
(QA pass-rate needs no new event — reuses task.awaiting_documentation[qa] +
task.qa_fail.) Real-PG tests for each; 111 transition tests still green.
* feat(metrics): member_performance_daily rollup table + migration 056 (phase 4b)
The per-member scorecard rollup: one row per (date, member_kind, agent_slug),
CEO as a first-class member_kind='ceo' row (agent_slug='' NOT NULL so the
NULL-distinct UNIQUE keeps it unique). Full column set + the four CEO-approved
extras (qa_reviews_total/passed, escalations, blocked_others, idle_seconds) plus
blocked_seconds. Overwrite-upsert on (date, member_kind, agent_slug) for an
idempotent sweep. Migration 056 verified real up/down (24 cols, 4 indexes).
* feat(metrics): _sweep_member_performance rollup sweeper (phase 4c)
The daily per-member rollup sweep (mirrors _sweep_daily_rollup): a trailing
7-day, idempotent overwrite-upsert wired into _run_sweep. One focused query per
metric merges into a (date, agent_slug) accumulator — spawn effort/turns/tokens/
cost, completed/first-pass/revisions-received, revisions-caused (qa/pr fails),
QA pass-rate (passed + total), escalations (by escalator_slug), blocked-others
(unblocked_dependents by blocker owner), idle_seconds (idle mark -> next spawn),
blocked_seconds (blocked dwell) — plus one CEO row/day (approval/unblock dwell +
god-mode). Real-PG test asserts every facet + idempotency (a 2nd sweep
overwrites, never doubles); spawn-day != completion-day split is by-design.
* feat(metrics): member/org rollup scorecards + endpoints + live overlay (phase 5)
MemberScorecard + OrgScorecard with derived rates (FPY, effort-throughput,
turns/tool-calls per task, QA pass-rate, utilization) — all division-guarded to
None. get_member_scorecard reads member_performance_daily by slug and overlays
the member's live in-flight (non-terminal) tasks' effort via get_task_metrics
(disjoint by status: completion counts stay rollup-only, overlay only enriches
effort/turns/cost; includes_live_inflight flags it). get_org_scorecard
aggregates the cell (?team=) or whole org. Routes: GET /metrics/member/{agent_id}
(404 if absent, after the ceo literal route) + GET /metrics/org?team=. Real-PG
tests (derived rates, overlay no double-count, guards, org) + route tests.
* feat(metrics): granular CEO completion notification (phase 6)
There was no CEO completion notification at all (EventType.TASK_COMPLETED was
defined but never emitted). Add notify_ceo_of_completion in
NotificationDeliveryService — a granular body (real effort vs wall-clock +
stints/turns/tool-calls/revisions[QA/PR]/cost from get_task_metrics; degrades to
wall-clock-only, turns 'n/a', when there are no spawn sessions). Reuses the
existing ALERT type (no enum migration; the notificationtype PG enum is fixed at
001). ceo_approve now emits TASK_COMPLETED + fires the notification (best-effort
via _notify_completion — never blocks completion); complete() emits
TASK_COMPLETED too (closes the dead-code gap; the WS bridge can forward it).
Pure formatter tests + real-PG notification test.
* [metrics-granularity] Phase 7: panel Scorecards tab + dashboard overview
Add the CEO-facing metrics surfaces for the granularity feature:
- New "Scorecards" tab on the Metrics page: org rollup headline, the
CEO-as-member card (approval/unblock dwell + god-mode count), and a
per-member table (completed, first-pass yield, active effort, turns/task,
QA pass-rate, escalations, blocked-others, utilization). Each member row
self-fetches its rollup scorecard; live in-flight rows carry a "live" badge.
- New dashboard overview card (ScorecardOverviewPanel): org-wide 30-day
headline (completed, FPY, throughput/hr, active effort, cost) deep-linking
into the Scorecards tab.
- Plumbing: TaskMetrics/MemberScorecard/OrgScorecard/CeoScorecard types,
observability API client methods + empty fallbacks, and the four
useCeoScorecard/useMemberScorecard/useOrgScorecard/useTaskMetrics hooks.
Panel gate green: tsc, eslint, prettier, vitest (175 tests, +6 new).
* [metrics-granularity] test: make completion-notification robust to shared-DB CEO
test_notify_ceo_of_completion_creates_alert errored in the full suite (passed
in isolation): the session-scoped test DB is shared across the run, and the
sibling real-DB board-gate test commits a role=CEO agent (slug="ceo") without
cleanup — so my env fixture's hardcoded slug="ceo" insert hit a unique-constraint
violation, and a second role=CEO row would also make _get_ceo_agent()'s
scalar_one_or_none() raise. Reuse an existing CEO when present (the singleton the
production system actually has), else create one with a unique slug. Order-
independent. Also reflow test_metrics_instrumentation.py to ruff format.
* chore(release): 0.15.0
Metrics granularity: per-member/per-task/org + CEO-as-member scorecards,
turn/tool-call capture (migration 055), member_performance_daily rollup
(migration 056) with QA pass-rate / escalations / blocked-others / utilization,
per-task active-vs-wait metrics, granular completion notification, panel
Scorecards tab + dashboard Performance card, and the ceo_reject audit fix.
Version bump across the canonical set + CHANGELOG.
* [metrics-granularity] fix pre-tag audit findings (overlay double-count + panel error states)
Adversarial review before the v0.15.0 tag surfaced two real logical gaps:
- MAJOR (backend): the live in-flight overlay re-summed ALL sessions of every
non-terminal task via get_task_metrics, but _msweep_spawn already rolls up
every CLOSED session regardless of task status — so a closed session on a
still-open task was counted twice (rollup + overlay), permanently inflating a
member's effort/turns/tokens/cost on the common reap/respawn path. The overlay
now sums only OPEN sessions (ended_at IS NULL), which the closed-only rollup
can never contain — disjoint by construction. A just-closed session lands in
the rollup on the next ~60s sweep (no gap of note). Aggregated in SQL to mirror
_msweep_spawn. Regression test reproduces the double-count (turns 10→5).
- MAJOR (panel): the four new scorecard surfaces used `isLoading || !data` with
no isError branch, so a failed query span forever on a skeleton. They now
surface a load error. Tests added.
Also: OrgSummary active-effort formatting no longer round-trips hours→seconds→
hours; dashboard grid uses xl:grid-cols-4 (was 2xl) so 4 panels show at 1280px;
corrected the inaccurate "NULL distinct" CEO-row uniqueness comment (agent_slug
is NOT NULL; the '' tuple is simply distinct from agent rows).
make quality GREEN (cov 95.31%); panel GREEN (vitest 178).
* [metrics-granularity] fix: decode bytes stream message-id before XCLAIM
StreamEventBus._recover_stream passed the pending message id to XCLAIM via
str() on the raw bytes the client returns (redis client has no
decode_responses), producing "b'1782066556728-0'". Redis rejects that with
"Unrecognized XCLAIM option", so pending-message recovery threw on every
reclaim tick and unacked messages from crashed/slow consumers were never
reclaimed (leaking in the PEL on every stream, spamming the error log). Decode
via the existing _to_str helper — the fix the sibling claim path already uses.
Pre-existing in v0.14.0 (unrelated to metrics granularity); folded into this
release per CEO. TDD regression test + CHANGELOG entry. make quality GREEN.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
@@ -6,6 +6,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.15.0] - 2026-07-01
|
||||
|
||||
### Added
|
||||
|
||||
- **Metrics granularity — the company is now measurable per member, per task, and as a whole, with the CEO measured as a member too.** Every agent spawn session now captures its operational shape, not just its token/cost total: LLM iterations (`turns`) and tool invocations (`tool_calls`) are parsed from the transcript, exposed over the SDK `/usage/status` + `/usage/sync`, and persisted on `agent_spawn_sessions` (migration 055) alongside the existing 4 token dimensions. A new per-task metrics endpoint (`GET /dashboard/metrics/task/{id}`) decomposes a task's lifetime into **active vs. wait** effort — merged-overlap active-runtime across every spawn stint mapped onto the audit-log stage windows (the pure `compute_stage_effort` helper) — so a slow task can be read as "the work was hard" vs. "it sat in a queue." A nightly rollup table `member_performance_daily` (migration 056) + orchestrator sweeper aggregate each member's day from data already captured: tasks completed, first-pass yield, active-effort throughput/hr, turns & tool-calls per task, revisions caused/received, **QA pass-rate, escalations raised, times-this-member-blocked-others, and idle/utilization** (idle paired from `agent.idle` audit marks to the next spawn), plus derived ratios. Member / team / org scorecards read the rollup with a **live in-flight overlay** (today's not-yet-rolled sessions are folded in so the numbers aren't a day stale), served by `GET /dashboard/metrics/member/{id}`, `/dashboard/metrics/member/ceo`, and `/dashboard/metrics/org?team=`. The human CEO is a first-class measured member: the CEO scorecard reports approval-decision and unblock latency (p50/p90 from the audit journey) and god-mode override count — reconstructed entirely from the audit log, no new hot-path writes.
|
||||
- **A granular completion notification.** When a task completes, the CEO now gets a notification carrying the task's metrics breakdown (active vs. wait effort, turns, tool calls, revisions, cost) instead of a bare "done" — the completion signal doubles as a per-task scorecard.
|
||||
- **Panel: a Scorecards tab and a dashboard Performance card.** The Metrics page gains a **Scorecards** tab — an org rollup headline, the CEO-as-member card (approval/unblock dwell + god-mode count), and a per-member table (completed, first-pass yield, active effort, turns/task, QA pass-rate, escalations, blocked-others, utilization) where each row self-fetches its rollup and live in-flight rows carry a "live" badge. The dashboard gains a **Performance** overview card (org-wide 30-day completed / first-pass-yield / throughput-per-hour / active-effort / cost) that deep-links into the Scorecards tab.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`ceo_reject` now leaves an audit trail.** The CEO's reject-with-changes and cancel decisions emitted no named audit event, so rework and decision-latency attribution had a hole at the CEO chokepoint. Both now emit the transition audit event at the single `_emit_status_transition_audit` chokepoint, so the CEO's decisions are reconstructable alongside every other role's.
|
||||
- **Stream pending-message recovery no longer fails on every reclaim tick.** `StreamEventBus._recover_stream` decoded a Redis message id with `str()` on the raw bytes the client returns (no `decode_responses`), producing `"b'…-0'"` — which Redis rejects with "Unrecognized XCLAIM option", so unacknowledged messages from crashed/slow consumers were never reclaimed and leaked in the pending-entries list on every stream (`roboco:stream:usage` and others), spamming the error log each interval. It now decodes the id via the existing `_to_str` helper before XCLAIM.
|
||||
|
||||
## [0.14.0] - 2026-06-29
|
||||
|
||||
### Added
|
||||
|
||||
@@ -129,7 +129,7 @@ Choose the registry and version with two env vars (defaults shown):
|
||||
|
||||
```bash
|
||||
ROBOCO_REGISTRY=ghcr.io/rennf93 # or docker.io/renzof93
|
||||
ROBOCO_VERSION=latest # or a pinned release, e.g. 0.14.0
|
||||
ROBOCO_VERSION=latest # or a pinned release, e.g. 0.15.0
|
||||
```
|
||||
|
||||
The orchestrator spawns the matching pre-built agent images on demand — no build toolchain or source compile on your host.
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
"""Add ``turns`` + ``tool_calls`` to agent_spawn_sessions.
|
||||
|
||||
Per-stint LLM iterations (unique assistant messages) and tool invocations,
|
||||
captured at session finalize from the SDK ``/usage/status`` (``turns`` also has
|
||||
a Claude-transcript fallback). They power the granular per-member performance
|
||||
metrics — distinguishing real effort/iterations from wall-clock. ``DEFAULT 0``
|
||||
so historical rows (and Grok agents, which have no Claude transcript) read 0,
|
||||
surfaced as "n/a" in the UI rather than a misleading "0 iterations".
|
||||
|
||||
Revision ID: 055_spawn_session_turns
|
||||
Revises: 054_a2a_message_skill
|
||||
Create Date: 2026-07-01
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "055_spawn_session_turns"
|
||||
down_revision = "054_a2a_message_skill"
|
||||
branch_labels: dict[str, str] | None = None
|
||||
depends_on: dict[str, str] | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"agent_spawn_sessions",
|
||||
sa.Column("turns", sa.BigInteger(), nullable=False, server_default="0"),
|
||||
)
|
||||
op.add_column(
|
||||
"agent_spawn_sessions",
|
||||
sa.Column("tool_calls", sa.BigInteger(), nullable=False, server_default="0"),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("agent_spawn_sessions", "tool_calls")
|
||||
op.drop_column("agent_spawn_sessions", "turns")
|
||||
@@ -0,0 +1,84 @@
|
||||
"""Add member_performance_daily — the granular per-member scorecard rollup.
|
||||
|
||||
One row per (date, member_kind, agent_slug) populated by the orchestrator's
|
||||
sweeper from data already captured (agent_spawn_sessions + audit_log). Serves
|
||||
the per-member / team / org scorecards without re-scanning raw task lists. The
|
||||
CEO is a first-class ``member_kind='ceo'`` row (agent_slug='' — a distinct
|
||||
natural-key tuple from every ``member_kind='agent'`` row, so it never collides).
|
||||
|
||||
Includes the four CEO-approved extra metrics (qa pass-rate counts, escalations,
|
||||
blocked-others, idle_seconds) plus blocked_seconds. All columns DEFAULT 0 so a
|
||||
fresh/partial row reads as zeros, never NULL.
|
||||
|
||||
Revision ID: 056_member_perf_daily
|
||||
Revises: 055_spawn_session_turns
|
||||
Create Date: 2026-07-01
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "056_member_perf_daily"
|
||||
down_revision = "055_spawn_session_turns"
|
||||
branch_labels: dict[str, str] | None = None
|
||||
depends_on: dict[str, str] | None = None
|
||||
|
||||
|
||||
def _int(name: str) -> sa.Column:
|
||||
return sa.Column(name, sa.Integer(), nullable=False, server_default="0")
|
||||
|
||||
|
||||
def _big(name: str) -> sa.Column:
|
||||
return sa.Column(name, sa.BigInteger(), nullable=False, server_default="0")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"member_performance_daily",
|
||||
sa.Column("id", sa.UUID(as_uuid=True), primary_key=True, nullable=False),
|
||||
sa.Column("date", sa.Date(), nullable=False),
|
||||
sa.Column("member_kind", sa.String(length=16), nullable=False),
|
||||
sa.Column(
|
||||
"agent_slug", sa.String(length=100), nullable=False, server_default=""
|
||||
),
|
||||
sa.Column("team", sa.String(length=50), nullable=True),
|
||||
sa.Column("role", sa.String(length=50), nullable=True),
|
||||
_int("tasks_completed"),
|
||||
_int("tasks_first_pass"),
|
||||
_int("revisions_caused"),
|
||||
_int("revisions_received"),
|
||||
_big("active_runtime_seconds"),
|
||||
_int("turns"),
|
||||
_int("tool_calls"),
|
||||
_big("tokens"),
|
||||
sa.Column("cost_usd", sa.Float(), nullable=False, server_default="0"),
|
||||
_big("ceo_approval_dwell_seconds"),
|
||||
_big("ceo_unblock_dwell_seconds"),
|
||||
_int("godmode_actions"),
|
||||
# The four CEO-approved extras (+ blocked_seconds).
|
||||
_int("qa_reviews_total"),
|
||||
_int("qa_reviews_passed"),
|
||||
_int("escalations"),
|
||||
_int("blocked_others"),
|
||||
_big("idle_seconds"),
|
||||
_big("blocked_seconds"),
|
||||
sa.UniqueConstraint(
|
||||
"date", "member_kind", "agent_slug", name="uq_member_perf_day"
|
||||
),
|
||||
)
|
||||
op.create_index("ix_member_perf_date", "member_performance_daily", ["date"])
|
||||
op.create_index(
|
||||
"ix_member_perf_agent_slug", "member_performance_daily", ["agent_slug"]
|
||||
)
|
||||
op.create_index("ix_member_perf_team", "member_performance_daily", ["team"])
|
||||
op.create_index("ix_member_perf_kind", "member_performance_daily", ["member_kind"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("ix_member_perf_kind", table_name="member_performance_daily")
|
||||
op.drop_index("ix_member_perf_team", table_name="member_performance_daily")
|
||||
op.drop_index("ix_member_perf_agent_slug", table_name="member_performance_daily")
|
||||
op.drop_index("ix_member_perf_date", table_name="member_performance_daily")
|
||||
op.drop_table("member_performance_daily")
|
||||
@@ -30,7 +30,7 @@ Two variables choose what you pull (defaults shown):
|
||||
|
||||
```bash
|
||||
ROBOCO_REGISTRY=ghcr.io/rennf93 # or docker.io/renzof93
|
||||
ROBOCO_VERSION=latest # or a pinned release, e.g. 0.14.0
|
||||
ROBOCO_VERSION=latest # or a pinned release, e.g. 0.15.0
|
||||
```
|
||||
|
||||
The orchestrator then spawns the **matching** pre-built agent images on demand (it reads `ROBOCO_AGENT_IMAGE_REGISTRY` / `ROBOCO_AGENT_IMAGE_TAG`, which the registry compose wires to the same registry and version). Pin `ROBOCO_VERSION` to a release tag in production so an upstream `latest` push can't silently change your fleet.
|
||||
|
||||
@@ -25,7 +25,7 @@ A feature flag set in `.env` takes effect on the next backend restart. The env-g
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|----------|---------|---------|
|
||||
| `ROBOCO_APP_VERSION` | `0.14.0` | Reported app version. |
|
||||
| `ROBOCO_APP_VERSION` | `0.15.0` | Reported app version. |
|
||||
| `ROBOCO_DEBUG` | `false` | Debug mode. |
|
||||
| `ROBOCO_ENVIRONMENT` | `development` | One of `development` / `staging` / `production`. Selects the JSON log renderer (prod) vs console renderer. The compose stack sets `production`. |
|
||||
| `ROBOCO_HOST` | `127.0.0.1` | Bind address. Use `0.0.0.0` in containers. |
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "roboco-panel",
|
||||
"version": "0.14.0",
|
||||
"version": "0.15.0",
|
||||
"private": true,
|
||||
"packageManager": "pnpm@10.25.0",
|
||||
"scripts": {
|
||||
|
||||
@@ -21,6 +21,7 @@ import { Skeleton } from "@/components/ui/skeleton";
|
||||
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs";
|
||||
import { OfflineState } from "@/components/ui/offline-state";
|
||||
import { DeliveryTabContent } from "@/components/metrics/delivery-tab";
|
||||
import { ScorecardsTabContent } from "@/components/metrics/scorecards-tab";
|
||||
import {
|
||||
UsageTimeSeriesChart,
|
||||
ModelUsageDonut,
|
||||
@@ -602,12 +603,13 @@ function CacheEfficiencyCard({
|
||||
|
||||
// ─── Tab types ────────────────────────────────────────────────────────────────
|
||||
|
||||
type MetricsTab = "performance" | "token-usage" | "delivery";
|
||||
type MetricsTab = "performance" | "token-usage" | "delivery" | "scorecards";
|
||||
|
||||
const VALID_METRICS_TABS: MetricsTab[] = [
|
||||
"performance",
|
||||
"token-usage",
|
||||
"delivery",
|
||||
"scorecards",
|
||||
];
|
||||
|
||||
function isValidMetricsTab(value: string | null): value is MetricsTab {
|
||||
@@ -649,6 +651,7 @@ function MetricsPageContent() {
|
||||
<TabsTrigger value="performance">Performance</TabsTrigger>
|
||||
<TabsTrigger value="token-usage">Token Usage</TabsTrigger>
|
||||
<TabsTrigger value="delivery">Delivery</TabsTrigger>
|
||||
<TabsTrigger value="scorecards">Scorecards</TabsTrigger>
|
||||
</TabsList>
|
||||
|
||||
<TabsContent value="performance" className="mt-6">
|
||||
@@ -662,6 +665,10 @@ function MetricsPageContent() {
|
||||
<TabsContent value="delivery" className="mt-6">
|
||||
<DeliveryTabContent />
|
||||
</TabsContent>
|
||||
|
||||
<TabsContent value="scorecards" className="mt-6">
|
||||
<ScorecardsTabContent />
|
||||
</TabsContent>
|
||||
</Tabs>
|
||||
</div>
|
||||
);
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
import { describe, it, expect, vi, beforeEach } from "vitest";
|
||||
import { render, screen } from "@testing-library/react";
|
||||
|
||||
const { mockOrg } = vi.hoisted(() => ({ mockOrg: vi.fn() }));
|
||||
|
||||
vi.mock("@/hooks/use-observability", () => ({
|
||||
useOrgScorecard: mockOrg,
|
||||
}));
|
||||
|
||||
import { ScorecardOverviewPanel } from "../scorecard-overview-panel";
|
||||
|
||||
describe("ScorecardOverviewPanel", () => {
|
||||
beforeEach(() => {
|
||||
mockOrg.mockReturnValue({
|
||||
data: {
|
||||
scope: "org",
|
||||
team: null,
|
||||
member_count: 3,
|
||||
tasks_completed: 42,
|
||||
first_pass_yield: 0.75,
|
||||
effort_throughput_per_hour: 1.5,
|
||||
active_runtime_hours: 12.3,
|
||||
turns: 0,
|
||||
tool_calls: 0,
|
||||
tokens: 0,
|
||||
cost_usd: 9.5,
|
||||
revisions_caused: 0,
|
||||
revisions_received: 0,
|
||||
},
|
||||
isLoading: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("renders headline org figures", () => {
|
||||
render(<ScorecardOverviewPanel />);
|
||||
expect(screen.getByText("42")).toBeInTheDocument();
|
||||
expect(screen.getByText("75%")).toBeInTheDocument();
|
||||
expect(screen.getByText("$9.50")).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("deep-links into the Scorecards metrics tab", () => {
|
||||
render(<ScorecardOverviewPanel />);
|
||||
const link = screen.getByRole("link", { name: /scorecards/i });
|
||||
expect(link).toHaveAttribute("href", "/metrics?tab=scorecards");
|
||||
});
|
||||
|
||||
it("shows a skeleton while loading", () => {
|
||||
mockOrg.mockReturnValue({ data: undefined, isLoading: true });
|
||||
const { container } = render(<ScorecardOverviewPanel />);
|
||||
expect(container.querySelectorAll('[data-slot="skeleton"]').length).toBe(5);
|
||||
});
|
||||
|
||||
it("surfaces an error instead of an endless skeleton", () => {
|
||||
mockOrg.mockReturnValue({
|
||||
data: undefined,
|
||||
isLoading: false,
|
||||
isError: true,
|
||||
});
|
||||
const { container } = render(<ScorecardOverviewPanel />);
|
||||
expect(
|
||||
screen.getByText(/failed to load performance metrics/i),
|
||||
).toBeInTheDocument();
|
||||
expect(container.querySelectorAll('[data-slot="skeleton"]').length).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -20,6 +20,7 @@ import { StrategySignalsPanel } from "./strategy-signals-panel";
|
||||
import type { Activity } from "./activity-item";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import { UsageOverviewPanel } from "./usage-overview-panel";
|
||||
import { ScorecardOverviewPanel } from "./scorecard-overview-panel";
|
||||
import { RefreshCw, Settings, AlertCircle } from "lucide-react";
|
||||
import Link from "next/link";
|
||||
|
||||
@@ -123,14 +124,15 @@ export function CommandCenter() {
|
||||
{/* Playbook review queue (hidden when no drafts) */}
|
||||
<PlaybookReviewQueue />
|
||||
|
||||
{/* Metrics, Alerts, and Usage Row */}
|
||||
<div className="grid grid-cols-1 lg:grid-cols-3 xl:grid-cols-3 2xl:grid-cols-3 gap-6">
|
||||
{/* Metrics, Alerts, Usage, and Performance Row */}
|
||||
<div className="grid grid-cols-1 lg:grid-cols-2 xl:grid-cols-4 gap-6">
|
||||
<KeyMetricsPanel
|
||||
metrics={overview?.key_metrics}
|
||||
isLoading={loadingOverview}
|
||||
/>
|
||||
<AuditorAlertsPanel alerts={flags} isLoading={loadingFlags} />
|
||||
<UsageOverviewPanel />
|
||||
<ScorecardOverviewPanel />
|
||||
</div>
|
||||
|
||||
{/* Blockers and Activity Row */}
|
||||
|
||||
@@ -14,3 +14,4 @@ export { ReleaseProposalCard } from "./release-proposal-card";
|
||||
export { PlaybookReviewQueue } from "./playbook-review-queue";
|
||||
export { StrategySignalsPanel } from "./strategy-signals-panel";
|
||||
export { UsageOverviewPanel } from "./usage-overview-panel";
|
||||
export { ScorecardOverviewPanel } from "./scorecard-overview-panel";
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
"use client";
|
||||
|
||||
import Link from "next/link";
|
||||
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
|
||||
import { Skeleton } from "@/components/ui/skeleton";
|
||||
import { useOrgScorecard } from "@/hooks/use-observability";
|
||||
import {
|
||||
Trophy,
|
||||
CheckCircle2,
|
||||
Gauge,
|
||||
Clock,
|
||||
Coins,
|
||||
ArrowRight,
|
||||
} from "lucide-react";
|
||||
|
||||
function pctOrNa(rate: number | null): string {
|
||||
return rate === null ? "n/a" : (rate * 100).toFixed(0) + "%";
|
||||
}
|
||||
|
||||
function numOrNa(value: number | null, digits = 2): string {
|
||||
return value === null ? "n/a" : value.toFixed(digits);
|
||||
}
|
||||
|
||||
interface MetricRowProps {
|
||||
icon: React.ReactNode;
|
||||
label: string;
|
||||
value: string;
|
||||
}
|
||||
|
||||
function MetricRow({ icon, label, value }: MetricRowProps) {
|
||||
return (
|
||||
<div className="flex items-center justify-between py-1">
|
||||
<div className="text-muted-foreground flex items-center gap-2 text-sm">
|
||||
{icon}
|
||||
{label}
|
||||
</div>
|
||||
<span className="text-sm font-semibold">{value}</span>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Dashboard overview of the org-wide performance rollup (last 30 days). Headline
|
||||
* figures from useOrgScorecard with a deep-link into the full Scorecards tab.
|
||||
*/
|
||||
export function ScorecardOverviewPanel() {
|
||||
const { data, isLoading, isError } = useOrgScorecard();
|
||||
|
||||
return (
|
||||
<Card>
|
||||
<CardHeader className="pb-3">
|
||||
<div className="flex items-center justify-between">
|
||||
<CardTitle className="flex items-center gap-2 text-lg">
|
||||
<Trophy className="h-5 w-5" />
|
||||
Performance
|
||||
</CardTitle>
|
||||
<Link
|
||||
href="/metrics?tab=scorecards"
|
||||
className="text-muted-foreground hover:text-foreground flex items-center gap-1 text-xs"
|
||||
>
|
||||
Scorecards
|
||||
<ArrowRight className="h-3 w-3" />
|
||||
</Link>
|
||||
</div>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
{isError ? (
|
||||
<div className="text-muted-foreground text-sm">
|
||||
Failed to load performance metrics.
|
||||
</div>
|
||||
) : isLoading || !data ? (
|
||||
<div className="space-y-3">
|
||||
{Array.from({ length: 5 }).map((_, i) => (
|
||||
<Skeleton key={i} className="h-6" />
|
||||
))}
|
||||
</div>
|
||||
) : (
|
||||
<div className="divide-y">
|
||||
<MetricRow
|
||||
icon={<CheckCircle2 className="h-4 w-4" />}
|
||||
label="Tasks completed (30d)"
|
||||
value={String(data.tasks_completed)}
|
||||
/>
|
||||
<MetricRow
|
||||
icon={<Gauge className="h-4 w-4" />}
|
||||
label="First-pass yield"
|
||||
value={pctOrNa(data.first_pass_yield)}
|
||||
/>
|
||||
<MetricRow
|
||||
icon={<Gauge className="h-4 w-4 text-blue-500" />}
|
||||
label="Throughput / hr"
|
||||
value={numOrNa(data.effort_throughput_per_hour)}
|
||||
/>
|
||||
<MetricRow
|
||||
icon={<Clock className="h-4 w-4" />}
|
||||
label="Active effort"
|
||||
value={data.active_runtime_hours.toFixed(1) + "h"}
|
||||
/>
|
||||
<MetricRow
|
||||
icon={<Coins className="h-4 w-4" />}
|
||||
label="Cost"
|
||||
value={"$" + data.cost_usd.toFixed(2)}
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,161 @@
|
||||
import { describe, it, expect, vi, beforeEach } from "vitest";
|
||||
import { render, screen } from "@testing-library/react";
|
||||
import { AgentRole, AgentState, type Agent } from "@/types";
|
||||
import type { MemberScorecard } from "@/types";
|
||||
|
||||
const { mockOrg, mockCeo, mockMember, mockAgents } = vi.hoisted(() => ({
|
||||
mockOrg: vi.fn(),
|
||||
mockCeo: vi.fn(),
|
||||
mockMember: vi.fn(),
|
||||
mockAgents: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock("@/hooks/use-observability", () => ({
|
||||
useOrgScorecard: mockOrg,
|
||||
useCeoScorecard: mockCeo,
|
||||
useMemberScorecard: mockMember,
|
||||
}));
|
||||
|
||||
vi.mock("@/hooks/use-agents", () => ({
|
||||
useAgents: mockAgents,
|
||||
}));
|
||||
|
||||
import { ScorecardsTabContent } from "../scorecards-tab";
|
||||
|
||||
function agent(id: string, name: string, role: AgentRole): Agent {
|
||||
return {
|
||||
id,
|
||||
agent_id: id,
|
||||
name,
|
||||
role,
|
||||
team: null,
|
||||
cell: null,
|
||||
status: AgentState.IDLE,
|
||||
};
|
||||
}
|
||||
|
||||
function member(over: Partial<MemberScorecard>): MemberScorecard {
|
||||
return {
|
||||
scope: "member",
|
||||
id: "a1",
|
||||
name: "a1",
|
||||
member_kind: "agent",
|
||||
tasks_completed: 0,
|
||||
first_pass_yield: null,
|
||||
effort_throughput_per_hour: null,
|
||||
active_runtime_hours: 0,
|
||||
turns: 0,
|
||||
tool_calls: 0,
|
||||
tokens: 0,
|
||||
cost_usd: 0,
|
||||
turns_per_task: null,
|
||||
tool_calls_per_task: null,
|
||||
revisions_caused: 0,
|
||||
revisions_received: 0,
|
||||
qa_pass_rate: null,
|
||||
escalations: 0,
|
||||
blocked_others: 0,
|
||||
idle_hours: 0,
|
||||
utilization: null,
|
||||
includes_live_inflight: false,
|
||||
...over,
|
||||
};
|
||||
}
|
||||
|
||||
describe("ScorecardsTabContent", () => {
|
||||
beforeEach(() => {
|
||||
mockOrg.mockReturnValue({
|
||||
data: {
|
||||
scope: "org",
|
||||
team: null,
|
||||
member_count: 3,
|
||||
tasks_completed: 42,
|
||||
first_pass_yield: 0.75,
|
||||
effort_throughput_per_hour: 1.5,
|
||||
active_runtime_hours: 12.3,
|
||||
turns: 0,
|
||||
tool_calls: 0,
|
||||
tokens: 0,
|
||||
cost_usd: 9.5,
|
||||
revisions_caused: 0,
|
||||
revisions_received: 0,
|
||||
},
|
||||
isLoading: false,
|
||||
});
|
||||
mockCeo.mockReturnValue({
|
||||
data: {
|
||||
member_kind: "ceo",
|
||||
approval_p50_seconds: 3600,
|
||||
approval_p90_seconds: 7200,
|
||||
approval_count: 5,
|
||||
unblock_p50_seconds: 1800,
|
||||
unblock_count: 2,
|
||||
godmode_actions: 1,
|
||||
},
|
||||
isLoading: false,
|
||||
});
|
||||
mockMember.mockReturnValue({
|
||||
data: member({
|
||||
id: "dev1",
|
||||
name: "be-dev-1",
|
||||
tasks_completed: 7,
|
||||
first_pass_yield: 0.8,
|
||||
active_runtime_hours: 4.2,
|
||||
qa_pass_rate: 0.9,
|
||||
escalations: 1,
|
||||
blocked_others: 2,
|
||||
utilization: 0.6,
|
||||
}),
|
||||
isLoading: false,
|
||||
});
|
||||
mockAgents.mockReturnValue({
|
||||
data: [
|
||||
agent("dev1", "be-dev-1", AgentRole.DEVELOPER),
|
||||
agent("ceo", "Renzo", AgentRole.CEO),
|
||||
agent("sys", "system", AgentRole.SYSTEM),
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("renders the org rollup headline figures", () => {
|
||||
render(<ScorecardsTabContent />);
|
||||
expect(screen.getByText("42")).toBeInTheDocument(); // tasks_completed
|
||||
expect(screen.getByText("75%")).toBeInTheDocument(); // first-pass yield
|
||||
expect(screen.getByText("$9.50")).toBeInTheDocument(); // cost
|
||||
});
|
||||
|
||||
it("renders the CEO approval/unblock figures", () => {
|
||||
render(<ScorecardsTabContent />);
|
||||
expect(screen.getByText("Approvals")).toBeInTheDocument();
|
||||
expect(screen.getByText("5")).toBeInTheDocument(); // approval_count
|
||||
expect(screen.getByText("God-mode actions")).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("lists only non-CEO/non-system members in the table", () => {
|
||||
render(<ScorecardsTabContent />);
|
||||
expect(screen.getByText("be-dev-1")).toBeInTheDocument();
|
||||
// CEO and system are excluded from the member table.
|
||||
expect(screen.queryByText("Renzo")).not.toBeInTheDocument();
|
||||
expect(screen.queryByText("system")).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("surfaces load errors instead of an endless skeleton", () => {
|
||||
mockOrg.mockReturnValue({
|
||||
data: undefined,
|
||||
isLoading: false,
|
||||
isError: true,
|
||||
});
|
||||
mockMember.mockReturnValue({
|
||||
data: undefined,
|
||||
isLoading: false,
|
||||
isError: true,
|
||||
});
|
||||
render(<ScorecardsTabContent />);
|
||||
expect(
|
||||
screen.getByText(/failed to load organization metrics/i),
|
||||
).toBeInTheDocument();
|
||||
// The member row shows a failed marker rather than a perpetual skeleton
|
||||
// (exact lowercase text, distinct from the org card's message).
|
||||
expect(screen.getByText("failed to load")).toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,191 @@
|
||||
"use client";
|
||||
|
||||
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
|
||||
import { Skeleton } from "@/components/ui/skeleton";
|
||||
import { Badge } from "@/components/ui/badge";
|
||||
import {
|
||||
Table,
|
||||
TableBody,
|
||||
TableCell,
|
||||
TableHead,
|
||||
TableHeader,
|
||||
TableRow,
|
||||
} from "@/components/ui/table";
|
||||
import {
|
||||
useCeoScorecard,
|
||||
useMemberScorecard,
|
||||
useOrgScorecard,
|
||||
} from "@/hooks/use-observability";
|
||||
import { useAgents } from "@/hooks/use-agents";
|
||||
import { AgentRole, type Agent } from "@/types";
|
||||
|
||||
function pctOrNa(rate: number | null): string {
|
||||
return rate === null ? "n/a" : (rate * 100).toFixed(0) + "%";
|
||||
}
|
||||
|
||||
function numOrNa(value: number | null, digits = 1): string {
|
||||
return value === null ? "n/a" : value.toFixed(digits);
|
||||
}
|
||||
|
||||
function hoursOrDash(seconds: number): string {
|
||||
return (seconds / 3600).toFixed(1) + "h";
|
||||
}
|
||||
|
||||
/** One member's row — each row self-fetches its rollup scorecard. */
|
||||
function MemberRow({ agent }: { agent: Agent }) {
|
||||
const { data, isLoading, isError } = useMemberScorecard(agent.id);
|
||||
if (isError) {
|
||||
return (
|
||||
<TableRow>
|
||||
<TableCell>{agent.name || agent.slug}</TableCell>
|
||||
<TableCell colSpan={8} className="text-muted-foreground text-xs">
|
||||
failed to load
|
||||
</TableCell>
|
||||
</TableRow>
|
||||
);
|
||||
}
|
||||
if (isLoading || !data) {
|
||||
return (
|
||||
<TableRow>
|
||||
<TableCell>{agent.name || agent.slug}</TableCell>
|
||||
<TableCell colSpan={8}>
|
||||
<Skeleton className="h-4 w-full" />
|
||||
</TableCell>
|
||||
</TableRow>
|
||||
);
|
||||
}
|
||||
return (
|
||||
<TableRow>
|
||||
<TableCell className="font-medium">
|
||||
{agent.name || agent.slug}
|
||||
{data.includes_live_inflight && (
|
||||
<Badge variant="outline" className="ml-2 text-xs">
|
||||
live
|
||||
</Badge>
|
||||
)}
|
||||
</TableCell>
|
||||
<TableCell>{data.tasks_completed}</TableCell>
|
||||
<TableCell>{pctOrNa(data.first_pass_yield)}</TableCell>
|
||||
<TableCell>{data.active_runtime_hours.toFixed(1)}h</TableCell>
|
||||
<TableCell>{numOrNa(data.turns_per_task)}</TableCell>
|
||||
<TableCell>{pctOrNa(data.qa_pass_rate)}</TableCell>
|
||||
<TableCell>{data.escalations}</TableCell>
|
||||
<TableCell>{data.blocked_others}</TableCell>
|
||||
<TableCell>{pctOrNa(data.utilization)}</TableCell>
|
||||
</TableRow>
|
||||
);
|
||||
}
|
||||
|
||||
function OrgSummary() {
|
||||
const { data, isLoading, isError } = useOrgScorecard();
|
||||
if (isError)
|
||||
return (
|
||||
<div className="text-muted-foreground text-sm">
|
||||
Failed to load organization metrics.
|
||||
</div>
|
||||
);
|
||||
if (isLoading || !data) return <Skeleton className="h-24 w-full" />;
|
||||
const cells: [string, string][] = [
|
||||
["Members", String(data.member_count)],
|
||||
["Completed", String(data.tasks_completed)],
|
||||
["First-pass yield", pctOrNa(data.first_pass_yield)],
|
||||
["Throughput/hr", numOrNa(data.effort_throughput_per_hour, 2)],
|
||||
["Active effort", data.active_runtime_hours.toFixed(1) + "h"],
|
||||
["Cost", "$" + data.cost_usd.toFixed(2)],
|
||||
];
|
||||
return (
|
||||
<div className="grid grid-cols-2 gap-4 sm:grid-cols-3 lg:grid-cols-6">
|
||||
{cells.map(([k, v]) => (
|
||||
<div key={k}>
|
||||
<div className="text-2xl font-semibold">{v}</div>
|
||||
<div className="text-muted-foreground text-sm">{k}</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function CeoCard() {
|
||||
const { data, isLoading, isError } = useCeoScorecard();
|
||||
if (isError)
|
||||
return (
|
||||
<div className="text-muted-foreground text-sm">
|
||||
Failed to load CEO metrics.
|
||||
</div>
|
||||
);
|
||||
if (isLoading || !data) return <Skeleton className="h-24 w-full" />;
|
||||
const cells: [string, string][] = [
|
||||
["Approvals", String(data.approval_count)],
|
||||
["Approval p50", hoursOrDash(data.approval_p50_seconds)],
|
||||
["Approval p90", hoursOrDash(data.approval_p90_seconds)],
|
||||
["Unblocks", String(data.unblock_count)],
|
||||
["Unblock p50", hoursOrDash(data.unblock_p50_seconds)],
|
||||
["God-mode actions", String(data.godmode_actions)],
|
||||
];
|
||||
return (
|
||||
<div className="grid grid-cols-2 gap-4 sm:grid-cols-3 lg:grid-cols-6">
|
||||
{cells.map(([k, v]) => (
|
||||
<div key={k}>
|
||||
<div className="text-2xl font-semibold">{v}</div>
|
||||
<div className="text-muted-foreground text-sm">{k}</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export function ScorecardsTabContent() {
|
||||
const { data: agents } = useAgents();
|
||||
const members = (agents ?? []).filter(
|
||||
(a) => a.role !== AgentRole.CEO && a.role !== AgentRole.SYSTEM,
|
||||
);
|
||||
return (
|
||||
<div className="space-y-6">
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle>Organization</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<OrgSummary />
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle>CEO (you)</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<CeoCard />
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle>Members</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<Table>
|
||||
<TableHeader>
|
||||
<TableRow>
|
||||
<TableHead>Member</TableHead>
|
||||
<TableHead>Done</TableHead>
|
||||
<TableHead>FPY</TableHead>
|
||||
<TableHead>Effort</TableHead>
|
||||
<TableHead>Turns/task</TableHead>
|
||||
<TableHead>QA pass</TableHead>
|
||||
<TableHead>Escal.</TableHead>
|
||||
<TableHead>Blocked others</TableHead>
|
||||
<TableHead>Util.</TableHead>
|
||||
</TableRow>
|
||||
</TableHeader>
|
||||
<TableBody>
|
||||
{members.map((a) => (
|
||||
<MemberRow key={a.id} agent={a} />
|
||||
))}
|
||||
</TableBody>
|
||||
</Table>
|
||||
</CardContent>
|
||||
</Card>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -7,6 +7,10 @@ import type {
|
||||
BottleneckReport,
|
||||
ReworkReport,
|
||||
Scorecard,
|
||||
CeoScorecard,
|
||||
MemberScorecard,
|
||||
OrgScorecard,
|
||||
TaskMetrics,
|
||||
} from "@/types";
|
||||
|
||||
// =============================================================================
|
||||
@@ -23,6 +27,14 @@ export const observabilityKeys = {
|
||||
[...observabilityKeys.all, "rework", days, team ?? "all"] as const,
|
||||
teamScorecard: (team: string, days: number) =>
|
||||
[...observabilityKeys.all, "scorecard", "team", team, days] as const,
|
||||
ceoScorecard: (days: number) =>
|
||||
[...observabilityKeys.all, "scorecard", "ceo", days] as const,
|
||||
memberScorecard: (agentId: string, days: number) =>
|
||||
[...observabilityKeys.all, "scorecard", "member", agentId, days] as const,
|
||||
orgScorecard: (days: number, team?: string) =>
|
||||
[...observabilityKeys.all, "scorecard", "org", team ?? "all", days] as const,
|
||||
taskMetrics: (taskId: string) =>
|
||||
[...observabilityKeys.all, "task-metrics", taskId] as const,
|
||||
};
|
||||
|
||||
// =============================================================================
|
||||
@@ -64,3 +76,40 @@ export function useTeamScorecard(team: string, days = 7) {
|
||||
refetchInterval: 60_000,
|
||||
});
|
||||
}
|
||||
|
||||
/** The human CEO as a measured member (approval/unblock dwell + god-mode). */
|
||||
export function useCeoScorecard(days = 30) {
|
||||
return useQuery<CeoScorecard>({
|
||||
queryKey: observabilityKeys.ceoScorecard(days),
|
||||
queryFn: () => observabilityApi.getCeoScorecard(days),
|
||||
refetchInterval: 60_000,
|
||||
});
|
||||
}
|
||||
|
||||
/** Per-member rollup scorecard (+ live in-flight overlay). */
|
||||
export function useMemberScorecard(agentId: string, days = 30) {
|
||||
return useQuery<MemberScorecard>({
|
||||
queryKey: observabilityKeys.memberScorecard(agentId, days),
|
||||
queryFn: () => observabilityApi.getMemberScorecard(agentId, days),
|
||||
enabled: Boolean(agentId),
|
||||
refetchInterval: 60_000,
|
||||
});
|
||||
}
|
||||
|
||||
/** Org-wide (or per-cell) rollup aggregate. */
|
||||
export function useOrgScorecard(days = 30, team?: string) {
|
||||
return useQuery<OrgScorecard>({
|
||||
queryKey: observabilityKeys.orgScorecard(days, team),
|
||||
queryFn: () => observabilityApi.getOrgScorecard(days, team),
|
||||
refetchInterval: 60_000,
|
||||
});
|
||||
}
|
||||
|
||||
/** Granular per-task metrics (active-vs-wait drill-down). */
|
||||
export function useTaskMetrics(taskId: string) {
|
||||
return useQuery<TaskMetrics | null>({
|
||||
queryKey: observabilityKeys.taskMetrics(taskId),
|
||||
queryFn: () => observabilityApi.getTaskMetrics(taskId),
|
||||
enabled: Boolean(taskId),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -5,6 +5,10 @@ import type {
|
||||
BottleneckReport,
|
||||
ReworkReport,
|
||||
Scorecard,
|
||||
CeoScorecard,
|
||||
MemberScorecard,
|
||||
OrgScorecard,
|
||||
TaskMetrics,
|
||||
} from "@/types";
|
||||
|
||||
// =============================================================================
|
||||
@@ -39,6 +43,61 @@ function emptyScorecard(scope: string, id: string): Scorecard {
|
||||
};
|
||||
}
|
||||
|
||||
const EMPTY_CEO: CeoScorecard = {
|
||||
member_kind: "ceo",
|
||||
approval_p50_seconds: 0,
|
||||
approval_p90_seconds: 0,
|
||||
approval_count: 0,
|
||||
unblock_p50_seconds: 0,
|
||||
unblock_count: 0,
|
||||
godmode_actions: 0,
|
||||
};
|
||||
|
||||
function emptyMember(id: string): MemberScorecard {
|
||||
return {
|
||||
scope: "member",
|
||||
id,
|
||||
name: id,
|
||||
member_kind: "agent",
|
||||
tasks_completed: 0,
|
||||
first_pass_yield: null,
|
||||
effort_throughput_per_hour: null,
|
||||
active_runtime_hours: 0,
|
||||
turns: 0,
|
||||
tool_calls: 0,
|
||||
tokens: 0,
|
||||
cost_usd: 0,
|
||||
turns_per_task: null,
|
||||
tool_calls_per_task: null,
|
||||
revisions_caused: 0,
|
||||
revisions_received: 0,
|
||||
qa_pass_rate: null,
|
||||
escalations: 0,
|
||||
blocked_others: 0,
|
||||
idle_hours: 0,
|
||||
utilization: null,
|
||||
includes_live_inflight: false,
|
||||
};
|
||||
}
|
||||
|
||||
function emptyOrg(team: string | null): OrgScorecard {
|
||||
return {
|
||||
scope: team ? "team" : "org",
|
||||
team,
|
||||
member_count: 0,
|
||||
tasks_completed: 0,
|
||||
first_pass_yield: null,
|
||||
effort_throughput_per_hour: null,
|
||||
active_runtime_hours: 0,
|
||||
turns: 0,
|
||||
tool_calls: 0,
|
||||
tokens: 0,
|
||||
cost_usd: 0,
|
||||
revisions_caused: 0,
|
||||
revisions_received: 0,
|
||||
};
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// API OBJECT
|
||||
// =============================================================================
|
||||
@@ -84,4 +143,48 @@ export const observabilityApi = {
|
||||
);
|
||||
return data;
|
||||
},
|
||||
|
||||
/** CEO-as-member scorecard — GET /dashboard/metrics/member/ceo?days */
|
||||
getCeoScorecard: async (days = 30): Promise<CeoScorecard> => {
|
||||
if (isMockMode()) return EMPTY_CEO;
|
||||
const { data } = await api.get<CeoScorecard>(
|
||||
"/dashboard/metrics/member/ceo",
|
||||
{ params: { days } },
|
||||
);
|
||||
return data;
|
||||
},
|
||||
|
||||
/** Per-member rollup scorecard — GET /dashboard/metrics/member/{id}?days */
|
||||
getMemberScorecard: async (
|
||||
agentId: string,
|
||||
days = 30,
|
||||
): Promise<MemberScorecard> => {
|
||||
if (isMockMode()) return emptyMember(agentId);
|
||||
const { data } = await api.get<MemberScorecard>(
|
||||
`/dashboard/metrics/member/${agentId}`,
|
||||
{ params: { days } },
|
||||
);
|
||||
return data;
|
||||
},
|
||||
|
||||
/** Org / team rollup — GET /dashboard/metrics/org?team&days */
|
||||
getOrgScorecard: async (
|
||||
days = 30,
|
||||
team?: string,
|
||||
): Promise<OrgScorecard> => {
|
||||
if (isMockMode()) return emptyOrg(team ?? null);
|
||||
const { data } = await api.get<OrgScorecard>("/dashboard/metrics/org", {
|
||||
params: { days, ...(team ? { team } : {}) },
|
||||
});
|
||||
return data;
|
||||
},
|
||||
|
||||
/** Granular per-task metrics — GET /dashboard/metrics/task/{id} */
|
||||
getTaskMetrics: async (taskId: string): Promise<TaskMetrics | null> => {
|
||||
if (isMockMode()) return null;
|
||||
const { data } = await api.get<TaskMetrics>(
|
||||
`/dashboard/metrics/task/${taskId}`,
|
||||
);
|
||||
return data;
|
||||
},
|
||||
};
|
||||
|
||||
@@ -1431,3 +1431,77 @@ export interface Scorecard {
|
||||
tokens: number;
|
||||
cost_usd: number;
|
||||
}
|
||||
|
||||
// --- Granular per-member metrics (v0.15.0) ---
|
||||
|
||||
export interface StageEffort {
|
||||
status: string;
|
||||
active_seconds: number;
|
||||
wait_seconds: number;
|
||||
}
|
||||
|
||||
export interface TaskMetrics {
|
||||
task_id: string;
|
||||
active_runtime_seconds: number;
|
||||
wall_clock_seconds: number;
|
||||
turns: number;
|
||||
tool_calls: number;
|
||||
tokens: number;
|
||||
cost_usd: number;
|
||||
revision_count: number;
|
||||
qa_fails: number;
|
||||
pr_fails: number;
|
||||
stints: number;
|
||||
stages: StageEffort[];
|
||||
}
|
||||
|
||||
export interface MemberScorecard {
|
||||
scope: string;
|
||||
id: string;
|
||||
name: string;
|
||||
member_kind: "agent";
|
||||
tasks_completed: number;
|
||||
first_pass_yield: number | null;
|
||||
effort_throughput_per_hour: number | null;
|
||||
active_runtime_hours: number;
|
||||
turns: number;
|
||||
tool_calls: number;
|
||||
tokens: number;
|
||||
cost_usd: number;
|
||||
turns_per_task: number | null;
|
||||
tool_calls_per_task: number | null;
|
||||
revisions_caused: number;
|
||||
revisions_received: number;
|
||||
qa_pass_rate: number | null;
|
||||
escalations: number;
|
||||
blocked_others: number;
|
||||
idle_hours: number;
|
||||
utilization: number | null;
|
||||
includes_live_inflight: boolean;
|
||||
}
|
||||
|
||||
export interface OrgScorecard {
|
||||
scope: string;
|
||||
team: string | null;
|
||||
member_count: number;
|
||||
tasks_completed: number;
|
||||
first_pass_yield: number | null;
|
||||
effort_throughput_per_hour: number | null;
|
||||
active_runtime_hours: number;
|
||||
turns: number;
|
||||
tool_calls: number;
|
||||
tokens: number;
|
||||
cost_usd: number;
|
||||
revisions_caused: number;
|
||||
revisions_received: number;
|
||||
}
|
||||
|
||||
export interface CeoScorecard {
|
||||
member_kind: "ceo";
|
||||
approval_p50_seconds: number;
|
||||
approval_p90_seconds: number;
|
||||
approval_count: number;
|
||||
unblock_p50_seconds: number;
|
||||
unblock_count: number;
|
||||
godmode_actions: number;
|
||||
}
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "roboco"
|
||||
version = "0.14.0"
|
||||
version = "0.15.0"
|
||||
description = "AI Agents Company - A virtual organization of AI agents functioning as a software development workforce"
|
||||
authors = [
|
||||
{name = "Renzo Franceschini", email = "rennf93@users.noreply.github.com"}
|
||||
|
||||
+1
-1
@@ -5,7 +5,7 @@ A virtual organization of 25 AI agents + 1 human CEO,
|
||||
designed to operate as a complete software development workforce.
|
||||
"""
|
||||
|
||||
__version__ = "0.14.0"
|
||||
__version__ = "0.15.0"
|
||||
|
||||
# Core exports
|
||||
from roboco.config import settings
|
||||
|
||||
@@ -229,6 +229,12 @@ class TokenReportRequest(BaseModel):
|
||||
class TokenUsageStatus(BaseModel):
|
||||
"""Current cumulative token usage for this session (GET /usage/status)."""
|
||||
|
||||
turns: int = Field(
|
||||
default=0, description="LLM iterations (unique assistant messages)"
|
||||
)
|
||||
tool_calls: int = Field(
|
||||
default=0, description="Tool invocations accumulated this session"
|
||||
)
|
||||
tokens_input: int = Field(
|
||||
default=0, description="Total input tokens accumulated this session"
|
||||
)
|
||||
|
||||
@@ -433,6 +433,10 @@ class _SessionState:
|
||||
self.tokens_output: int = 0
|
||||
self.tokens_cache_read: int = 0
|
||||
self.tokens_cache_write: int = 0
|
||||
# LLM iterations (unique assistant message ids), set by /usage/sync from
|
||||
# the transcript. tool_calls is total_calls (above). Both surfaced on
|
||||
# /usage/status so the orchestrator persists them at session finalize.
|
||||
self.turns: int = 0
|
||||
# (size, mtime) of the last transcript parsed for /usage/sync, so a
|
||||
# re-sync of an unchanged transcript skips the re-parse.
|
||||
self.transcript_fingerprint: tuple[int, float] | None = None
|
||||
@@ -737,6 +741,8 @@ async def usage_status() -> TokenUsageStatus:
|
||||
|
||||
def _token_usage_snapshot() -> TokenUsageStatus:
|
||||
return TokenUsageStatus(
|
||||
turns=_state.turns,
|
||||
tool_calls=_state.total_calls,
|
||||
tokens_input=_state.tokens_input,
|
||||
tokens_output=_state.tokens_output,
|
||||
tokens_cache_read=_state.tokens_cache_read,
|
||||
@@ -798,11 +804,12 @@ async def usage_sync(req: TranscriptSyncRequest) -> TokenUsageStatus:
|
||||
return _token_usage_snapshot()
|
||||
|
||||
try:
|
||||
tin, tout, tcr, tcw = _sum_transcript_usage(path)
|
||||
tin, tout, tcr, tcw, turns = _sum_transcript_usage(path)
|
||||
except OSError as exc:
|
||||
logger.warning("Transcript usage sync failed", error=str(exc))
|
||||
return _token_usage_snapshot()
|
||||
|
||||
_state.turns = turns
|
||||
_state.tokens_input = tin
|
||||
_state.tokens_output = tout
|
||||
_state.tokens_cache_read = tcr
|
||||
|
||||
@@ -54,16 +54,17 @@ def _line_usage(line: str) -> tuple[str | None, tuple[int, int, int, int]] | Non
|
||||
return message.get("id"), deltas
|
||||
|
||||
|
||||
def sum_transcript_usage(path: Path) -> tuple[int, int, int, int]:
|
||||
"""Sum per-message token usage across a Claude Code JSONL transcript.
|
||||
def sum_transcript_usage(path: Path) -> tuple[int, int, int, int, int]:
|
||||
"""Sum per-message token usage + turn count across a Claude Code transcript.
|
||||
|
||||
Each assistant message carries a ``message.usage`` block with the token
|
||||
counts for that API response; summing them yields the session total.
|
||||
Messages that span several lines (same ``message.id``) are counted once —
|
||||
Claude Code emits one line per content block, each repeating the message's
|
||||
usage, so naive summing roughly doubles the totals. Returns
|
||||
``(input, output, cache_read, cache_write)``. Malformed lines are skipped —
|
||||
a single bad line must never lose the whole count.
|
||||
``(input, output, cache_read, cache_write, turns)`` where ``turns`` is the
|
||||
number of UNIQUE assistant ``message.id``s (i.e. LLM iterations). Malformed
|
||||
lines are skipped — a single bad line must never lose the whole count.
|
||||
"""
|
||||
tin = tout = tcr = tcw = 0
|
||||
seen: set[str] = set()
|
||||
@@ -81,4 +82,4 @@ def sum_transcript_usage(path: Path) -> tuple[int, int, int, int]:
|
||||
tout += line_out
|
||||
tcr += line_cr
|
||||
tcw += line_cw
|
||||
return tin, tout, tcr, tcw
|
||||
return tin, tout, tcr, tcw, len(seen)
|
||||
|
||||
@@ -608,3 +608,61 @@ async def get_team_scorecard(
|
||||
status_code=status.HTTP_404_NOT_FOUND, detail="Team scorecard unavailable"
|
||||
)
|
||||
return card.to_dict()
|
||||
|
||||
|
||||
@router.get("/metrics/task/{task_id}")
|
||||
async def get_task_metrics(
|
||||
task_id: UUID,
|
||||
db: DbSession,
|
||||
) -> dict[str, Any]:
|
||||
"""Granular per-task metrics: real effort vs wall-clock, turns/tool-calls,
|
||||
per-stage active-vs-wait, who-caused-rework, tokens + cost (live)."""
|
||||
metrics_service = get_metrics_service(db)
|
||||
metrics = await metrics_service.get_task_metrics(task_id)
|
||||
if metrics is None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND, detail="Task not found"
|
||||
)
|
||||
return metrics.to_dict()
|
||||
|
||||
|
||||
# NOTE: /metrics/member/ceo MUST be declared before any /metrics/member/{id}
|
||||
# UUID-typed route so the literal "ceo" isn't captured as a path param.
|
||||
@router.get("/metrics/member/ceo")
|
||||
async def get_ceo_scorecard(
|
||||
db: DbSession,
|
||||
days: int = Query(default=30, ge=1, le=90),
|
||||
) -> dict[str, Any]:
|
||||
"""The human CEO as a measured member: approval/unblock dwell + god-mode count."""
|
||||
metrics_service = get_metrics_service(db)
|
||||
card = await metrics_service.get_ceo_scorecard(days=days)
|
||||
return card.to_dict()
|
||||
|
||||
|
||||
@router.get("/metrics/member/{agent_id}")
|
||||
async def get_member_scorecard(
|
||||
agent_id: UUID,
|
||||
db: DbSession,
|
||||
days: int = Query(default=30, ge=1, le=90),
|
||||
) -> dict[str, Any]:
|
||||
"""Per-member rollup scorecard (real effort, FPY, turns, cost, rework, the
|
||||
four extras) + a live in-flight overlay. 404 if the agent doesn't exist."""
|
||||
metrics_service = get_metrics_service(db)
|
||||
card = await metrics_service.get_member_scorecard(agent_id, days=days)
|
||||
if card is None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND, detail="Agent not found"
|
||||
)
|
||||
return card.to_dict()
|
||||
|
||||
|
||||
@router.get("/metrics/org")
|
||||
async def get_org_scorecard(
|
||||
db: DbSession,
|
||||
team: Team | None = None,
|
||||
days: int = Query(default=30, ge=1, le=90),
|
||||
) -> dict[str, Any]:
|
||||
"""Org-wide (or per-cell with ?team=) rollup aggregate from the member table."""
|
||||
metrics_service = get_metrics_service(db)
|
||||
card = await metrics_service.get_org_scorecard(team=team, days=days)
|
||||
return card.to_dict()
|
||||
|
||||
+2
-2
@@ -28,7 +28,7 @@ class Settings(BaseSettings):
|
||||
# ==========================================================================
|
||||
# Application
|
||||
# ==========================================================================
|
||||
app_version: str = "0.14.0"
|
||||
app_version: str = "0.15.0"
|
||||
debug: bool = False
|
||||
environment: str = Field(
|
||||
default="development", pattern="^(development|staging|production)$"
|
||||
@@ -704,7 +704,7 @@ class Settings(BaseSettings):
|
||||
agent_image_tag: str = Field(
|
||||
default="",
|
||||
description=(
|
||||
"Tag for pre-built agent images (e.g. 'latest' or '0.14.0'). Empty "
|
||||
"Tag for pre-built agent images (e.g. 'latest' or '0.15.0'). Empty "
|
||||
"leaves the tag implicit (':latest'); only meaningful with "
|
||||
"agent_image_registry set."
|
||||
),
|
||||
|
||||
@@ -2213,6 +2213,11 @@ class AgentSpawnSessionTable(Base):
|
||||
tokens_cache_write: Mapped[int] = mapped_column(
|
||||
BigInteger, nullable=False, default=0
|
||||
)
|
||||
# LLM iterations (unique assistant messages) + tool invocations for this
|
||||
# stint, captured at finalize from the SDK /usage/status (turns has a
|
||||
# transcript fallback). Default 0 — historical/Grok rows read 0 ("n/a").
|
||||
turns: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0)
|
||||
tool_calls: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0)
|
||||
exit_reason: Mapped[str | None] = mapped_column(String(100), nullable=True)
|
||||
estimated_cost_usd: Mapped[float | None] = mapped_column(Float, nullable=True)
|
||||
|
||||
@@ -2315,6 +2320,64 @@ class DailyUsageRollupTable(Base):
|
||||
)
|
||||
|
||||
|
||||
class MemberPerformanceDailyTable(Base):
|
||||
"""Pre-aggregated daily per-member performance (the granular scorecard rollup).
|
||||
|
||||
One row per (date, member_kind, agent_slug), populated by the orchestrator
|
||||
sweeper from agent_spawn_sessions + audit_log. The CEO is a first-class
|
||||
``member_kind='ceo'`` row with ``agent_slug=''`` — a distinct natural-key
|
||||
tuple from every ``member_kind='agent'`` row, so it never collides.
|
||||
Overwrite-upsert on the natural key makes the sweep idempotent. All counters
|
||||
default 0.
|
||||
"""
|
||||
|
||||
__tablename__ = "member_performance_daily"
|
||||
|
||||
id: Mapped[UUID] = mapped_column(
|
||||
UUID(as_uuid=True), primary_key=True, default=uuid4
|
||||
)
|
||||
date: Mapped[Any] = mapped_column(Date, nullable=False) # datetime.date
|
||||
member_kind: Mapped[str] = mapped_column(String(16), nullable=False)
|
||||
agent_slug: Mapped[str] = mapped_column(String(100), nullable=False, default="")
|
||||
team: Mapped[str | None] = mapped_column(String(50), nullable=True)
|
||||
role: Mapped[str | None] = mapped_column(String(50), nullable=True)
|
||||
tasks_completed: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
tasks_first_pass: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
revisions_caused: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
revisions_received: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
active_runtime_seconds: Mapped[int] = mapped_column(
|
||||
BigInteger, nullable=False, default=0
|
||||
)
|
||||
turns: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
tool_calls: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
tokens: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0)
|
||||
cost_usd: Mapped[float] = mapped_column(Float, nullable=False, default=0.0)
|
||||
ceo_approval_dwell_seconds: Mapped[int] = mapped_column(
|
||||
BigInteger, nullable=False, default=0
|
||||
)
|
||||
ceo_unblock_dwell_seconds: Mapped[int] = mapped_column(
|
||||
BigInteger, nullable=False, default=0
|
||||
)
|
||||
godmode_actions: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
# The four CEO-approved extras (+ blocked_seconds).
|
||||
qa_reviews_total: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
qa_reviews_passed: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
escalations: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
blocked_others: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
idle_seconds: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0)
|
||||
blocked_seconds: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0)
|
||||
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"date", "member_kind", "agent_slug", name="uq_member_perf_day"
|
||||
),
|
||||
Index("ix_member_perf_date", "date"),
|
||||
Index("ix_member_perf_agent_slug", "agent_slug"),
|
||||
Index("ix_member_perf_team", "team"),
|
||||
Index("ix_member_perf_kind", "member_kind"),
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# PROMPTER TABLES
|
||||
# =============================================================================
|
||||
|
||||
@@ -526,8 +526,11 @@ class StreamEventBus:
|
||||
recovered = 0
|
||||
for msg in pending_details:
|
||||
if int(msg["time_since_delivered"]) >= idle_time_ms:
|
||||
# Decode the id: xpending_range returns bytes (no
|
||||
# decode_responses); str(bytes) → "b'..'", which Redis rejects
|
||||
# as an unrecognized XCLAIM option, wedging pending recovery.
|
||||
recovered += await self._claim_and_handle(
|
||||
stream, str(msg["message_id"]), idle_time_ms
|
||||
stream, self._to_str(msg["message_id"]), idle_time_ms
|
||||
)
|
||||
return recovered
|
||||
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
"""Per-stage active-vs-wait decomposition — pure overlap math (no DB).
|
||||
|
||||
A task's wall-clock lifetime is a sequence of status windows (from the audit
|
||||
log). During each window some agent spawn *stints* may have been running. This
|
||||
splits each window into ``active`` (wall-clock during which at least one stint
|
||||
was running) and ``wait`` (the remainder — queue/review idle). Overlapping
|
||||
stints are merged, so ``active`` can never exceed the window length and
|
||||
``active + wait == window length``.
|
||||
|
||||
This is the wall-clock decomposition and is deliberately distinct from *summed
|
||||
effort* (Σ stint durations), which can exceed wall-clock when stints run
|
||||
concurrently and is computed separately at the service layer.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Sequence
|
||||
from datetime import datetime
|
||||
|
||||
# A status window: (status, entered_at, exited_at). An open final window passes
|
||||
# ``now()`` as exited_at at the call site.
|
||||
StageWindow = tuple[str, "datetime", "datetime"]
|
||||
# A spawn stint: (started_at, ended_at). An in-flight open stint passes
|
||||
# ``now()`` as ended_at at the call site.
|
||||
Stint = tuple["datetime", "datetime"]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class StageEffort:
|
||||
"""Active vs wait seconds for one status window."""
|
||||
|
||||
status: str
|
||||
active_seconds: int
|
||||
wait_seconds: int
|
||||
|
||||
def to_dict(self) -> dict[str, object]:
|
||||
return {
|
||||
"status": self.status,
|
||||
"active_seconds": self.active_seconds,
|
||||
"wait_seconds": self.wait_seconds,
|
||||
}
|
||||
|
||||
|
||||
def _merged_overlap_seconds(
|
||||
win_start: datetime, win_end: datetime, stints: Sequence[Stint]
|
||||
) -> float:
|
||||
"""Wall-clock seconds inside [win_start, win_end) covered by any stint.
|
||||
|
||||
Stints are clipped to the window and overlapping ones merged, so concurrent
|
||||
stints are counted once (never exceeds the window length).
|
||||
"""
|
||||
clipped: list[tuple[datetime, datetime]] = []
|
||||
for stint_start, stint_end in stints:
|
||||
lo = max(win_start, stint_start)
|
||||
hi = min(win_end, stint_end)
|
||||
if hi > lo:
|
||||
clipped.append((lo, hi))
|
||||
if not clipped:
|
||||
return 0.0
|
||||
clipped.sort()
|
||||
total = 0.0
|
||||
cur_lo, cur_hi = clipped[0]
|
||||
for lo, hi in clipped[1:]:
|
||||
if lo <= cur_hi: # overlapping/adjacent — extend the current run
|
||||
cur_hi = max(cur_hi, hi)
|
||||
else:
|
||||
total += (cur_hi - cur_lo).total_seconds()
|
||||
cur_lo, cur_hi = lo, hi
|
||||
total += (cur_hi - cur_lo).total_seconds()
|
||||
return total
|
||||
|
||||
|
||||
def compute_stage_effort(
|
||||
windows: Sequence[StageWindow], stints: Sequence[Stint]
|
||||
) -> list[StageEffort]:
|
||||
"""Decompose each status window into active vs wait seconds.
|
||||
|
||||
``active`` = merged wall-clock overlap of the stints with the window;
|
||||
``wait`` = window length minus active (clamped >= 0). A zero/negative-length
|
||||
window yields ``(0, 0)``. Seconds are rounded to whole ints for a stable
|
||||
API surface.
|
||||
"""
|
||||
result: list[StageEffort] = []
|
||||
for status, start, end in windows:
|
||||
span = (end - start).total_seconds()
|
||||
if span <= 0:
|
||||
result.append(StageEffort(status=status, active_seconds=0, wait_seconds=0))
|
||||
continue
|
||||
active = _merged_overlap_seconds(start, end, stints)
|
||||
active = min(active, span)
|
||||
wait = max(0.0, span - active)
|
||||
result.append(
|
||||
StageEffort(
|
||||
status=status,
|
||||
active_seconds=round(active),
|
||||
wait_seconds=round(wait),
|
||||
)
|
||||
)
|
||||
return result
|
||||
@@ -8,8 +8,16 @@ from dataclasses import dataclass
|
||||
from typing import Any
|
||||
from uuid import UUID
|
||||
|
||||
from roboco.foundation.policy.stage_effort import StageEffort
|
||||
from roboco.models.base import Team
|
||||
|
||||
# CEO decision transitions used by the CEO scorecard (audit-log to_status
|
||||
# values). Approval = any CEO exit from awaiting_ceo_approval (incl. the
|
||||
# coordination-root reject that lands in `pending`); unblock = a CEO revive
|
||||
# out of `blocked`.
|
||||
CEO_APPROVAL_DECISIONS = ("completed", "needs_revision", "cancelled", "pending")
|
||||
CEO_UNBLOCK_DECISIONS = ("in_progress", "pending")
|
||||
|
||||
|
||||
class VelocityMetrics:
|
||||
"""Velocity metrics over a time period."""
|
||||
@@ -247,3 +255,168 @@ class Scorecard:
|
||||
"tokens": self.tokens,
|
||||
"cost_usd": round(self.cost_usd, 4),
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class MemberScorecard:
|
||||
"""Per-member rollup scorecard (agent) with derived rates.
|
||||
|
||||
Served from member_performance_daily (terminal work) optionally enriched
|
||||
with a live in-flight overlay (the member's non-terminal tasks' effort so
|
||||
far). Completion counts stay rollup-only — the two sets are disjoint by
|
||||
status — so ``includes_live_inflight`` only enriches effort/turns/cost.
|
||||
"""
|
||||
|
||||
scope: str # "member"
|
||||
id: str
|
||||
name: str
|
||||
tasks_completed: int
|
||||
first_pass_yield: float | None
|
||||
effort_throughput_per_hour: float | None
|
||||
active_runtime_hours: float
|
||||
turns: int
|
||||
tool_calls: int
|
||||
tokens: int
|
||||
cost_usd: float
|
||||
turns_per_task: float | None
|
||||
tool_calls_per_task: float | None
|
||||
revisions_caused: int
|
||||
revisions_received: int
|
||||
qa_pass_rate: float | None
|
||||
escalations: int
|
||||
blocked_others: int
|
||||
idle_hours: float
|
||||
utilization: float | None
|
||||
includes_live_inflight: bool
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"scope": self.scope,
|
||||
"id": self.id,
|
||||
"name": self.name,
|
||||
"member_kind": "agent",
|
||||
"tasks_completed": self.tasks_completed,
|
||||
"first_pass_yield": self.first_pass_yield,
|
||||
"effort_throughput_per_hour": self.effort_throughput_per_hour,
|
||||
"active_runtime_hours": round(self.active_runtime_hours, 2),
|
||||
"turns": self.turns,
|
||||
"tool_calls": self.tool_calls,
|
||||
"tokens": self.tokens,
|
||||
"cost_usd": round(self.cost_usd, 4),
|
||||
"turns_per_task": self.turns_per_task,
|
||||
"tool_calls_per_task": self.tool_calls_per_task,
|
||||
"revisions_caused": self.revisions_caused,
|
||||
"revisions_received": self.revisions_received,
|
||||
"qa_pass_rate": self.qa_pass_rate,
|
||||
"escalations": self.escalations,
|
||||
"blocked_others": self.blocked_others,
|
||||
"idle_hours": round(self.idle_hours, 2),
|
||||
"utilization": self.utilization,
|
||||
"includes_live_inflight": self.includes_live_inflight,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class OrgScorecard:
|
||||
"""Team or whole-org rollup aggregate (rollup-only, no live overlay)."""
|
||||
|
||||
scope: str # "org" | "team"
|
||||
team: str | None
|
||||
member_count: int
|
||||
tasks_completed: int
|
||||
first_pass_yield: float | None
|
||||
effort_throughput_per_hour: float | None
|
||||
active_runtime_hours: float
|
||||
turns: int
|
||||
tool_calls: int
|
||||
tokens: int
|
||||
cost_usd: float
|
||||
revisions_caused: int
|
||||
revisions_received: int
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"scope": self.scope,
|
||||
"team": self.team,
|
||||
"member_count": self.member_count,
|
||||
"tasks_completed": self.tasks_completed,
|
||||
"first_pass_yield": self.first_pass_yield,
|
||||
"effort_throughput_per_hour": self.effort_throughput_per_hour,
|
||||
"active_runtime_hours": round(self.active_runtime_hours, 2),
|
||||
"turns": self.turns,
|
||||
"tool_calls": self.tool_calls,
|
||||
"tokens": self.tokens,
|
||||
"cost_usd": round(self.cost_usd, 4),
|
||||
"revisions_caused": self.revisions_caused,
|
||||
"revisions_received": self.revisions_received,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class CeoScorecard:
|
||||
"""The human CEO as a measured member — read purely from the audit_log.
|
||||
|
||||
The CEO never runs an LLM, so token/cost/turns are n/a. Its metrics are
|
||||
approval dwell (awaiting_ceo_approval -> a CEO decision), unblock dwell
|
||||
(blocked -> a CEO revive), and god-mode action count (every ceo-attributed
|
||||
transition in the window).
|
||||
"""
|
||||
|
||||
approval_p50_seconds: float
|
||||
approval_p90_seconds: float
|
||||
approval_count: int
|
||||
unblock_p50_seconds: float
|
||||
unblock_count: int
|
||||
godmode_actions: int
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"member_kind": "ceo",
|
||||
"approval_p50_seconds": round(self.approval_p50_seconds, 2),
|
||||
"approval_p90_seconds": round(self.approval_p90_seconds, 2),
|
||||
"approval_count": self.approval_count,
|
||||
"unblock_p50_seconds": round(self.unblock_p50_seconds, 2),
|
||||
"unblock_count": self.unblock_count,
|
||||
"godmode_actions": self.godmode_actions,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class TaskMetrics:
|
||||
"""Granular per-task effort: real runtime vs wall-clock, turns, cost, rework.
|
||||
|
||||
``active_runtime_seconds`` is summed spawn-stint effort (can exceed
|
||||
wall-clock when stints run concurrently); ``stages`` is the wall-clock
|
||||
active-vs-wait decomposition per status window (active clamped to the
|
||||
window). Computed live from ``agent_spawn_sessions`` (by task_id) joined
|
||||
with ``audit_log`` (by target_id).
|
||||
"""
|
||||
|
||||
task_id: str
|
||||
active_runtime_seconds: int
|
||||
wall_clock_seconds: int
|
||||
turns: int
|
||||
tool_calls: int
|
||||
tokens: int
|
||||
cost_usd: float
|
||||
revision_count: int
|
||||
qa_fails: int
|
||||
pr_fails: int
|
||||
stints: int
|
||||
stages: list[StageEffort]
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"task_id": self.task_id,
|
||||
"active_runtime_seconds": self.active_runtime_seconds,
|
||||
"wall_clock_seconds": self.wall_clock_seconds,
|
||||
"turns": self.turns,
|
||||
"tool_calls": self.tool_calls,
|
||||
"tokens": self.tokens,
|
||||
"cost_usd": round(self.cost_usd, 4),
|
||||
"revision_count": self.revision_count,
|
||||
"qa_fails": self.qa_fails,
|
||||
"pr_fails": self.pr_fails,
|
||||
"stints": self.stints,
|
||||
"stages": [s.to_dict() for s in self.stages],
|
||||
}
|
||||
|
||||
@@ -4980,8 +4980,8 @@ class AgentOrchestrator:
|
||||
@staticmethod
|
||||
def _usage_from_transcript(
|
||||
agent_id: str, claude_session_id: str | None = None
|
||||
) -> tuple[int, int, int, int]:
|
||||
"""Sum token usage from the agent's Claude Code transcript.
|
||||
) -> tuple[int, int, int, int, int]:
|
||||
"""Sum token usage + turn count from the agent's Claude Code transcript.
|
||||
|
||||
The host ``~/.claude`` is mounted into the orchestrator, so transcripts
|
||||
are readable here under ``projects/<cwd-dir>/<session-id>.jsonl``. When
|
||||
@@ -5008,11 +5008,11 @@ class AgentOrchestrator:
|
||||
for f in d.glob("*.jsonl")
|
||||
]
|
||||
if not jsonl:
|
||||
return (0, 0, 0, 0)
|
||||
return (0, 0, 0, 0, 0)
|
||||
newest = max(jsonl, key=lambda f: f.stat().st_mtime)
|
||||
return sum_transcript_usage(newest)
|
||||
except OSError:
|
||||
return (0, 0, 0, 0)
|
||||
return (0, 0, 0, 0, 0)
|
||||
|
||||
def _grok_usage_json(self, agent_id: str) -> dict[str, Any] | None:
|
||||
"""Read a GROK agent's ``usage.json`` (``{model, total_tokens, cost_usd}``).
|
||||
@@ -5167,13 +5167,53 @@ class AgentOrchestrator:
|
||||
)
|
||||
|
||||
if not tokens[0] and not tokens[1]:
|
||||
tin, tout, cr, cw = self._usage_from_transcript(
|
||||
tin, tout, cr, cw, _turns = self._usage_from_transcript(
|
||||
agent_id, self._claude_session_id_for(agent_id)
|
||||
)
|
||||
if tin or tout:
|
||||
tokens = (tin, tout, cr, cw)
|
||||
return tokens
|
||||
|
||||
async def _resolve_final_turns_tools(self, agent_id: str) -> tuple[int, int]:
|
||||
"""Resolve final ``(turns, tool_calls)`` for a stopping agent.
|
||||
|
||||
Primary source is the live SDK ``/usage/status`` (which carries both).
|
||||
For ``turns`` only there is a durable Claude-transcript fallback (unique
|
||||
assistant-message count) for short-lived agents whose SDK counts race
|
||||
teardown; ``tool_calls`` has no transcript equivalent and stays 0 ("n/a")
|
||||
when the SDK misses. Grok agents have neither — returns ``(0, 0)``.
|
||||
Best-effort: any failure degrades to zeros, never blocks finalize.
|
||||
"""
|
||||
from roboco.models.base import ModelProvider
|
||||
|
||||
if self.get_provider_for_agent(agent_id) == ModelProvider.GROK.value:
|
||||
return (0, 0)
|
||||
|
||||
turns = tool_calls = 0
|
||||
sdk_url = f"http://roboco-agent-{agent_id}:{SDK_PORT}/usage/status"
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=3.0, headers=_system_api_headers()
|
||||
) as client:
|
||||
resp = await client.get(sdk_url)
|
||||
if resp.status_code == http_status.HTTP_200_OK:
|
||||
data = resp.json()
|
||||
turns = int(data.get("turns", 0) or 0)
|
||||
tool_calls = int(data.get("tool_calls", 0) or 0)
|
||||
except Exception as sdk_exc:
|
||||
logger.debug(
|
||||
"Could not fetch final turns/tool_calls from SDK",
|
||||
agent_id=agent_id,
|
||||
error=str(sdk_exc),
|
||||
)
|
||||
|
||||
if not turns:
|
||||
*_tokens, t = self._usage_from_transcript(
|
||||
agent_id, self._claude_session_id_for(agent_id)
|
||||
)
|
||||
turns = t
|
||||
return turns, tool_calls
|
||||
|
||||
async def _finalize_spawn_session(
|
||||
self,
|
||||
agent_id: str,
|
||||
@@ -5198,6 +5238,10 @@ class AgentOrchestrator:
|
||||
tokens_cache_read,
|
||||
tokens_cache_write,
|
||||
) = await self._resolve_final_token_usage(agent_id)
|
||||
# Resolve LLM iterations + tool calls (live SDK; turns has a
|
||||
# transcript fallback). Separate from the token tuple so the live
|
||||
# snapshot helpers keep their 4-tuple contract.
|
||||
turns, tool_calls = await self._resolve_final_turns_tools(agent_id)
|
||||
|
||||
# Look up the model and usage_session_id from the running instance config.
|
||||
model = "unknown"
|
||||
@@ -5248,6 +5292,8 @@ class AgentOrchestrator:
|
||||
tokens_output=tokens_output,
|
||||
tokens_cache_read=tokens_cache_read,
|
||||
tokens_cache_write=tokens_cache_write,
|
||||
turns=turns,
|
||||
tool_calls=tool_calls,
|
||||
exit_reason=exit_reason,
|
||||
estimated_cost_usd=cost,
|
||||
)
|
||||
@@ -5305,10 +5351,11 @@ class AgentOrchestrator:
|
||||
tokens = await self._fetch_agent_tokens(client, agent_id)
|
||||
if tokens is not None:
|
||||
return tokens
|
||||
transcript = self._usage_from_transcript(
|
||||
tin, tout, cr, cw, _turns = self._usage_from_transcript(
|
||||
agent_id, self._claude_session_id_for(agent_id)
|
||||
)
|
||||
return transcript if any(transcript) else None
|
||||
token_counts = (tin, tout, cr, cw)
|
||||
return token_counts if any(token_counts) else None
|
||||
|
||||
@staticmethod
|
||||
async def _persist_token_snapshot(
|
||||
@@ -5594,6 +5641,385 @@ class AgentOrchestrator:
|
||||
else:
|
||||
db.add(DailyUsageRollupTable(id=uuid4(), **key, **values))
|
||||
|
||||
# =========================================================================
|
||||
# MEMBER-PERFORMANCE ROLLUP (granular per-member scorecards)
|
||||
# =========================================================================
|
||||
|
||||
async def _sweep_member_performance(self) -> None:
|
||||
"""Upsert member_performance_daily from spawn sessions + audit_log.
|
||||
|
||||
Mirrors _sweep_daily_rollup: a trailing 7-day, overwrite-upsert sweep
|
||||
(idempotent — re-running overwrites, never accumulates). Aggregates each
|
||||
metric with one query keyed (date, agent_slug) into an accumulator, then
|
||||
upserts one row per member per day plus one CEO row per day. Wrapped in
|
||||
its own try/except so a bad member rollup never aborts the sweeper.
|
||||
"""
|
||||
try:
|
||||
from roboco.db.base import get_session_factory
|
||||
except ImportError:
|
||||
return
|
||||
try:
|
||||
window_start = datetime.now(UTC) - timedelta(days=7)
|
||||
session_factory = get_session_factory()
|
||||
async with session_factory() as db:
|
||||
acc: dict[tuple[Any, str], dict[str, Any]] = {}
|
||||
await self._msweep_spawn(db, window_start, acc)
|
||||
await self._msweep_delivery(db, window_start, acc)
|
||||
await self._msweep_caused(db, window_start, acc)
|
||||
await self._msweep_qa(db, window_start, acc)
|
||||
await self._msweep_escalations(db, window_start, acc)
|
||||
await self._msweep_blocked_others(db, window_start, acc)
|
||||
await self._msweep_idle(db, window_start, acc)
|
||||
await self._msweep_blocked_seconds(db, window_start, acc)
|
||||
for (day, slug), fields in acc.items():
|
||||
await self._upsert_member_perf_row(db, day, "agent", slug, fields)
|
||||
await self._msweep_ceo(db, window_start)
|
||||
await db.commit()
|
||||
logger.debug("Member performance rollup complete", rows=len(acc))
|
||||
except Exception as exc:
|
||||
logger.warning("Member performance rollup failed", error=str(exc))
|
||||
|
||||
@staticmethod
|
||||
def _merge_member(
|
||||
acc: dict[tuple[Any, str], dict[str, Any]],
|
||||
day: Any,
|
||||
slug: str,
|
||||
**fields: Any,
|
||||
) -> None:
|
||||
"""Merge one metric's aggregate into the (date, slug) accumulator entry."""
|
||||
if not slug:
|
||||
return
|
||||
entry = acc.setdefault((day, slug), {})
|
||||
for key, value in fields.items():
|
||||
entry[key] = value
|
||||
|
||||
async def _msweep_spawn(
|
||||
self,
|
||||
db: Any,
|
||||
window_start: datetime,
|
||||
acc: dict[tuple[Any, str], dict[str, Any]],
|
||||
) -> None:
|
||||
"""Effort / turns / tool_calls / tokens / cost from closed spawn sessions."""
|
||||
from sqlalchemy import text
|
||||
|
||||
sql = text(
|
||||
"""
|
||||
SELECT date(started_at) AS d, agent_slug AS slug, team, role,
|
||||
COALESCE(SUM(EXTRACT(epoch FROM
|
||||
(COALESCE(ended_at, now()) - started_at))), 0) AS active_s,
|
||||
COALESCE(SUM(turns), 0) AS turns,
|
||||
COALESCE(SUM(tool_calls), 0) AS tool_calls,
|
||||
COALESCE(SUM(tokens_input + tokens_output
|
||||
+ tokens_cache_read + tokens_cache_write), 0) AS tokens,
|
||||
COALESCE(SUM(estimated_cost_usd), 0) AS cost
|
||||
FROM agent_spawn_sessions
|
||||
WHERE ended_at IS NOT NULL AND started_at >= :ws
|
||||
GROUP BY date(started_at), agent_slug, team, role
|
||||
"""
|
||||
)
|
||||
for r in (await db.execute(sql, {"ws": window_start})).all():
|
||||
self._merge_member(
|
||||
acc,
|
||||
r.d,
|
||||
r.slug,
|
||||
team=r.team,
|
||||
role=r.role,
|
||||
active_runtime_seconds=int(r.active_s or 0),
|
||||
turns=int(r.turns or 0),
|
||||
tool_calls=int(r.tool_calls or 0),
|
||||
tokens=int(r.tokens or 0),
|
||||
cost_usd=float(r.cost or 0.0),
|
||||
)
|
||||
|
||||
async def _msweep_delivery(
|
||||
self,
|
||||
db: Any,
|
||||
window_start: datetime,
|
||||
acc: dict[tuple[Any, str], dict[str, Any]],
|
||||
) -> None:
|
||||
"""Completed / first-pass / revisions-received per task owner per day."""
|
||||
from sqlalchemy import text
|
||||
|
||||
sql = text(
|
||||
"""
|
||||
SELECT date(t.completed_at) AS d, ag.slug AS slug,
|
||||
COUNT(*) AS completed,
|
||||
COUNT(*) FILTER (WHERE COALESCE(t.revision_count, 0) = 0) AS first_pass,
|
||||
COALESCE(SUM(t.revision_count), 0) AS received
|
||||
FROM tasks t JOIN agents ag ON ag.id = t.assigned_to
|
||||
WHERE t.status = 'completed' AND t.completed_at >= :ws
|
||||
AND t.assigned_to IS NOT NULL
|
||||
GROUP BY date(t.completed_at), ag.slug
|
||||
"""
|
||||
)
|
||||
for r in (await db.execute(sql, {"ws": window_start})).all():
|
||||
self._merge_member(
|
||||
acc,
|
||||
r.d,
|
||||
r.slug,
|
||||
tasks_completed=int(r.completed or 0),
|
||||
tasks_first_pass=int(r.first_pass or 0),
|
||||
revisions_received=int(r.received or 0),
|
||||
)
|
||||
|
||||
async def _msweep_caused(
|
||||
self,
|
||||
db: Any,
|
||||
window_start: datetime,
|
||||
acc: dict[tuple[Any, str], dict[str, Any]],
|
||||
) -> None:
|
||||
"""Revisions caused — qa/pr fail events attributed to the rejector."""
|
||||
from sqlalchemy import text
|
||||
|
||||
sql = text(
|
||||
"""
|
||||
SELECT date(al.timestamp) AS d, ag.slug AS slug, COUNT(*) AS caused
|
||||
FROM audit_log al JOIN agents ag ON ag.id = al.agent_id
|
||||
WHERE al.event_type IN ('task.qa_fail', 'task.pr_fail')
|
||||
AND al.timestamp >= :ws
|
||||
GROUP BY date(al.timestamp), ag.slug
|
||||
"""
|
||||
)
|
||||
for r in (await db.execute(sql, {"ws": window_start})).all():
|
||||
self._merge_member(acc, r.d, r.slug, revisions_caused=int(r.caused or 0))
|
||||
|
||||
async def _msweep_qa(
|
||||
self,
|
||||
db: Any,
|
||||
window_start: datetime,
|
||||
acc: dict[tuple[Any, str], dict[str, Any]],
|
||||
) -> None:
|
||||
"""QA pass-rate — passed (awaiting_documentation by qa) + failed (qa_fail)."""
|
||||
from sqlalchemy import text
|
||||
|
||||
sql = text(
|
||||
"""
|
||||
SELECT date(al.timestamp) AS d, ag.slug AS slug,
|
||||
COUNT(*) FILTER (
|
||||
WHERE al.event_type = 'task.awaiting_documentation') AS passed,
|
||||
COUNT(*) FILTER (WHERE al.event_type = 'task.qa_fail') AS failed
|
||||
FROM audit_log al JOIN agents ag ON ag.id = al.agent_id
|
||||
WHERE al.timestamp >= :ws AND (
|
||||
(al.event_type = 'task.awaiting_documentation'
|
||||
AND (al.details->>'agent_role') = 'qa')
|
||||
OR al.event_type = 'task.qa_fail'
|
||||
)
|
||||
GROUP BY date(al.timestamp), ag.slug
|
||||
"""
|
||||
)
|
||||
for r in (await db.execute(sql, {"ws": window_start})).all():
|
||||
passed = int(r.passed or 0)
|
||||
failed = int(r.failed or 0)
|
||||
self._merge_member(
|
||||
acc,
|
||||
r.d,
|
||||
r.slug,
|
||||
qa_reviews_passed=passed,
|
||||
qa_reviews_total=passed + failed,
|
||||
)
|
||||
|
||||
async def _msweep_escalations(
|
||||
self,
|
||||
db: Any,
|
||||
window_start: datetime,
|
||||
acc: dict[tuple[Any, str], dict[str, Any]],
|
||||
) -> None:
|
||||
"""Escalations raised per member (keyed on details.escalator_slug)."""
|
||||
from sqlalchemy import text
|
||||
|
||||
sql = text(
|
||||
"""
|
||||
SELECT date(timestamp) AS d,
|
||||
(details->>'escalator_slug') AS slug, COUNT(*) AS n
|
||||
FROM audit_log
|
||||
WHERE event_type = 'task.escalated' AND timestamp >= :ws
|
||||
AND (details->>'escalator_slug') IS NOT NULL
|
||||
GROUP BY date(timestamp), (details->>'escalator_slug')
|
||||
"""
|
||||
)
|
||||
for r in (await db.execute(sql, {"ws": window_start})).all():
|
||||
self._merge_member(acc, r.d, r.slug, escalations=int(r.n or 0))
|
||||
|
||||
async def _msweep_blocked_others(
|
||||
self,
|
||||
db: Any,
|
||||
window_start: datetime,
|
||||
acc: dict[tuple[Any, str], dict[str, Any]],
|
||||
) -> None:
|
||||
"""Downstream tasks a member's completed task was blocking."""
|
||||
from sqlalchemy import text
|
||||
|
||||
sql = text(
|
||||
"""
|
||||
SELECT date(al.timestamp) AS d, ag.slug AS slug,
|
||||
COALESCE(SUM((al.details->>'count')::int), 0) AS n
|
||||
FROM audit_log al
|
||||
JOIN tasks t ON t.id = al.target_id
|
||||
JOIN agents ag ON ag.id = t.assigned_to
|
||||
WHERE al.event_type = 'task.unblocked_dependents' AND al.timestamp >= :ws
|
||||
GROUP BY date(al.timestamp), ag.slug
|
||||
"""
|
||||
)
|
||||
for r in (await db.execute(sql, {"ws": window_start})).all():
|
||||
self._merge_member(acc, r.d, r.slug, blocked_others=int(r.n or 0))
|
||||
|
||||
async def _msweep_idle(
|
||||
self,
|
||||
db: Any,
|
||||
window_start: datetime,
|
||||
acc: dict[tuple[Any, str], dict[str, Any]],
|
||||
) -> None:
|
||||
"""Idle seconds — each idle mark to the member's next spawn (else now)."""
|
||||
from sqlalchemy import text
|
||||
|
||||
sql = text(
|
||||
"""
|
||||
WITH idle AS (
|
||||
SELECT date(al.timestamp) AS d,
|
||||
(al.details->>'agent_slug') AS slug, al.timestamp AS idle_at
|
||||
FROM audit_log al
|
||||
WHERE al.event_type = 'agent.idle' AND al.timestamp >= :ws
|
||||
AND (al.details->>'agent_slug') IS NOT NULL
|
||||
)
|
||||
SELECT i.d, i.slug,
|
||||
COALESCE(SUM(EXTRACT(epoch FROM (
|
||||
COALESCE((SELECT MIN(s.started_at) FROM agent_spawn_sessions s
|
||||
WHERE s.agent_slug = i.slug AND s.started_at > i.idle_at),
|
||||
now()) - i.idle_at))), 0) AS idle_s
|
||||
FROM idle i GROUP BY i.d, i.slug
|
||||
"""
|
||||
)
|
||||
for r in (await db.execute(sql, {"ws": window_start})).all():
|
||||
self._merge_member(acc, r.d, r.slug, idle_seconds=int(r.idle_s or 0))
|
||||
|
||||
async def _msweep_blocked_seconds(
|
||||
self,
|
||||
db: Any,
|
||||
window_start: datetime,
|
||||
acc: dict[tuple[Any, str], dict[str, Any]],
|
||||
) -> None:
|
||||
"""Wall-clock a member's tasks spent in `blocked`, per owner per day."""
|
||||
from sqlalchemy import text
|
||||
|
||||
sql = text(
|
||||
"""
|
||||
WITH ordered AS (
|
||||
SELECT a.target_id, a.timestamp AS entered,
|
||||
(a.details->>'to_status') AS status,
|
||||
LEAD(a.timestamp) OVER (
|
||||
PARTITION BY a.target_id ORDER BY a.timestamp) AS exited
|
||||
FROM audit_log a
|
||||
WHERE a.event_type LIKE 'task.%'
|
||||
AND a.event_type = 'task.' || (a.details->>'to_status')
|
||||
AND a.timestamp >= :ws
|
||||
)
|
||||
SELECT date(o.entered) AS d, ag.slug AS slug,
|
||||
COALESCE(SUM(EXTRACT(epoch FROM
|
||||
(COALESCE(o.exited, now()) - o.entered))), 0) AS blocked_s
|
||||
FROM ordered o
|
||||
JOIN tasks t ON t.id = o.target_id
|
||||
JOIN agents ag ON ag.id = t.assigned_to
|
||||
WHERE o.status = 'blocked'
|
||||
GROUP BY date(o.entered), ag.slug
|
||||
"""
|
||||
)
|
||||
for r in (await db.execute(sql, {"ws": window_start})).all():
|
||||
self._merge_member(acc, r.d, r.slug, blocked_seconds=int(r.blocked_s or 0))
|
||||
|
||||
async def _msweep_ceo(self, db: Any, window_start: datetime) -> None:
|
||||
"""Upsert one CEO row per day: approval/unblock dwell + god-mode count."""
|
||||
from sqlalchemy import text
|
||||
|
||||
sql = text(
|
||||
"""
|
||||
WITH events AS (
|
||||
SELECT target_id, timestamp, date(timestamp) AS d,
|
||||
(details->>'to_status') AS to_status,
|
||||
(details->>'agent_role') AS role
|
||||
FROM audit_log
|
||||
WHERE event_type LIKE 'task.%' AND timestamp >= :ws
|
||||
),
|
||||
approvals AS (
|
||||
SELECT e.d, EXTRACT(epoch FROM ((
|
||||
SELECT MIN(x.timestamp) FROM events x
|
||||
WHERE x.target_id = e.target_id AND x.timestamp > e.timestamp
|
||||
AND x.role = 'ceo'
|
||||
AND x.to_status IN
|
||||
('completed', 'needs_revision', 'cancelled', 'pending')
|
||||
) - e.timestamp)) AS latency
|
||||
FROM events e WHERE e.to_status = 'awaiting_ceo_approval'
|
||||
),
|
||||
unblocks AS (
|
||||
SELECT e.d, EXTRACT(epoch FROM ((
|
||||
SELECT MIN(x.timestamp) FROM events x
|
||||
WHERE x.target_id = e.target_id AND x.timestamp > e.timestamp
|
||||
AND x.role = 'ceo' AND x.to_status IN ('in_progress', 'pending')
|
||||
) - e.timestamp)) AS latency
|
||||
FROM events e WHERE e.to_status = 'blocked'
|
||||
)
|
||||
SELECT d,
|
||||
COALESCE(SUM(approval_latency), 0) AS approval_s,
|
||||
COALESCE(SUM(unblock_latency), 0) AS unblock_s,
|
||||
COALESCE(SUM(godmode), 0) AS godmode
|
||||
FROM (
|
||||
SELECT d, latency AS approval_latency, 0 AS unblock_latency, 0 AS godmode
|
||||
FROM approvals WHERE latency IS NOT NULL
|
||||
UNION ALL
|
||||
SELECT d, 0, latency, 0 FROM unblocks WHERE latency IS NOT NULL
|
||||
UNION ALL
|
||||
SELECT d, 0, 0, 1 FROM events WHERE role = 'ceo'
|
||||
) u GROUP BY d
|
||||
"""
|
||||
)
|
||||
for r in (await db.execute(sql, {"ws": window_start})).all():
|
||||
await self._upsert_member_perf_row(
|
||||
db,
|
||||
r.d,
|
||||
"ceo",
|
||||
"",
|
||||
{
|
||||
"ceo_approval_dwell_seconds": int(r.approval_s or 0),
|
||||
"ceo_unblock_dwell_seconds": int(r.unblock_s or 0),
|
||||
"godmode_actions": int(r.godmode or 0),
|
||||
},
|
||||
)
|
||||
|
||||
async def _upsert_member_perf_row(
|
||||
self, db: Any, day: Any, member_kind: str, slug: str, fields: dict[str, Any]
|
||||
) -> None:
|
||||
"""Overwrite-upsert one member_performance_daily row on the natural key."""
|
||||
from uuid import uuid4 as _uuid4
|
||||
|
||||
from sqlalchemy import select, update
|
||||
|
||||
from roboco.db.tables import MemberPerformanceDailyTable
|
||||
|
||||
existing = (
|
||||
await db.execute(
|
||||
select(MemberPerformanceDailyTable).where(
|
||||
MemberPerformanceDailyTable.date == day,
|
||||
MemberPerformanceDailyTable.member_kind == member_kind,
|
||||
MemberPerformanceDailyTable.agent_slug == slug,
|
||||
)
|
||||
)
|
||||
).scalar_one_or_none()
|
||||
if existing is not None:
|
||||
await db.execute(
|
||||
update(MemberPerformanceDailyTable)
|
||||
.where(MemberPerformanceDailyTable.id == existing.id)
|
||||
.values(**fields)
|
||||
)
|
||||
else:
|
||||
db.add(
|
||||
MemberPerformanceDailyTable(
|
||||
id=_uuid4(),
|
||||
date=day,
|
||||
member_kind=member_kind,
|
||||
agent_slug=slug,
|
||||
**fields,
|
||||
)
|
||||
)
|
||||
|
||||
async def restore_waiting_records(self) -> int:
|
||||
"""Load persisted waiting records into memory on orchestrator start.
|
||||
|
||||
@@ -5956,6 +6382,8 @@ Start by:
|
||||
# closed sessions into the daily aggregation table.
|
||||
await self._sweep_token_snapshots()
|
||||
await self._sweep_daily_rollup()
|
||||
# Granular per-member performance rollup (own try/except inside).
|
||||
await self._sweep_member_performance()
|
||||
|
||||
# Prune old agent transcripts (throttled internally to ~hourly) so the
|
||||
# operator's bind-mounted ~/.claude doesn't grow without bound.
|
||||
|
||||
+443
-1
@@ -10,27 +10,35 @@ from datetime import UTC, datetime, timedelta
|
||||
from typing import Any, ClassVar
|
||||
from uuid import UUID
|
||||
|
||||
from sqlalchemy import and_, func, select, text
|
||||
from sqlalchemy import and_, bindparam, func, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from roboco.db.tables import (
|
||||
AgentSpawnSessionTable,
|
||||
AgentTable,
|
||||
AuditLogTable,
|
||||
MemberPerformanceDailyTable,
|
||||
MessageTable,
|
||||
NotificationTable,
|
||||
TaskTable,
|
||||
)
|
||||
from roboco.foundation.policy.stage_effort import compute_stage_effort
|
||||
from roboco.models.base import TaskStatus, Team
|
||||
from roboco.models.metrics import (
|
||||
CEO_APPROVAL_DECISIONS,
|
||||
CEO_UNBLOCK_DECISIONS,
|
||||
AgentMetrics,
|
||||
AgentReworkRate,
|
||||
BlockerMetrics,
|
||||
BottleneckReport,
|
||||
CeoScorecard,
|
||||
MemberScorecard,
|
||||
OrgScorecard,
|
||||
ReworkReport,
|
||||
Scorecard,
|
||||
StageBottleneck,
|
||||
StageTiming,
|
||||
TaskMetrics,
|
||||
TeamMetrics,
|
||||
TeamReworkRate,
|
||||
VelocityMetrics,
|
||||
@@ -782,6 +790,440 @@ class MetricsService(BaseService):
|
||||
).scalar() or 0.0
|
||||
return float(cost)
|
||||
|
||||
async def _spawn_rollup_for_task(
|
||||
self, task_id: UUID, close_at: datetime
|
||||
) -> dict[str, Any]:
|
||||
"""Per-task spawn aggregates: stints, effort, turns, tool_calls, tokens, cost.
|
||||
|
||||
``active_runtime`` is summed stint duration (an open stint runs to
|
||||
``close_at`` — completed_at for a terminal task, else now); it can exceed
|
||||
wall-clock when stints overlap. ``stints`` is the list of
|
||||
``(started, ended)`` intervals for the stage decomposition. ``task_id``
|
||||
is bound as ``str`` — the column is ``String(36)``.
|
||||
"""
|
||||
rows = (
|
||||
await self.session.execute(
|
||||
select(
|
||||
AgentSpawnSessionTable.started_at,
|
||||
AgentSpawnSessionTable.ended_at,
|
||||
AgentSpawnSessionTable.turns,
|
||||
AgentSpawnSessionTable.tool_calls,
|
||||
AgentSpawnSessionTable.tokens_input,
|
||||
AgentSpawnSessionTable.tokens_output,
|
||||
AgentSpawnSessionTable.tokens_cache_read,
|
||||
AgentSpawnSessionTable.tokens_cache_write,
|
||||
AgentSpawnSessionTable.estimated_cost_usd,
|
||||
).where(AgentSpawnSessionTable.task_id == str(task_id))
|
||||
)
|
||||
).all()
|
||||
stints: list[tuple[datetime, datetime]] = []
|
||||
active = turns = tool_calls = tokens = 0.0
|
||||
cost = 0.0
|
||||
for r in rows:
|
||||
ended = r.ended_at or close_at
|
||||
stints.append((r.started_at, ended))
|
||||
active += max(0.0, (ended - r.started_at).total_seconds())
|
||||
turns += r.turns or 0
|
||||
tool_calls += r.tool_calls or 0
|
||||
tokens += (
|
||||
(r.tokens_input or 0)
|
||||
+ (r.tokens_output or 0)
|
||||
+ (r.tokens_cache_read or 0)
|
||||
+ (r.tokens_cache_write or 0)
|
||||
)
|
||||
cost += r.estimated_cost_usd or 0.0
|
||||
return {
|
||||
"stints": stints,
|
||||
"active_runtime_seconds": round(active),
|
||||
"turns": int(turns),
|
||||
"tool_calls": int(tool_calls),
|
||||
"tokens": int(tokens),
|
||||
"cost_usd": float(cost),
|
||||
}
|
||||
|
||||
async def _stage_windows_for_task(
|
||||
self, task_id: UUID, close_at: datetime
|
||||
) -> list[tuple[str, datetime, datetime]]:
|
||||
"""Ordered status windows for one task from the audit_log journey.
|
||||
|
||||
Single-task variant of ``get_cycle_time_by_stage``: LEAD gives each
|
||||
generic ``task.<status>`` row's exit as the next row's timestamp; the
|
||||
open final window (``exited_at IS NULL``) is closed at ``close_at``
|
||||
(completed_at for a terminal task, else now) so an in-flight stage still
|
||||
decomposes and a terminal task's final stage doesn't grow forever.
|
||||
"""
|
||||
sql = text(
|
||||
"""
|
||||
WITH ordered AS (
|
||||
SELECT
|
||||
(a.details->>'to_status') AS status,
|
||||
a.timestamp AS entered_at,
|
||||
LEAD(a.timestamp) OVER (ORDER BY a.timestamp) AS exited_at
|
||||
FROM audit_log a
|
||||
WHERE a.target_id = CAST(:tid AS uuid)
|
||||
AND a.event_type LIKE 'task.%'
|
||||
AND a.event_type = 'task.' || (a.details->>'to_status')
|
||||
)
|
||||
SELECT status, entered_at, exited_at FROM ordered ORDER BY entered_at
|
||||
"""
|
||||
)
|
||||
rows = (await self.session.execute(sql, {"tid": str(task_id)})).all()
|
||||
return [(r.status, r.entered_at, r.exited_at or close_at) for r in rows]
|
||||
|
||||
async def _task_fail_counts(self, task_id: UUID) -> tuple[int, int]:
|
||||
"""(qa_fails, pr_fails) attributed to this task from named audit events."""
|
||||
rows = (
|
||||
await self.session.execute(
|
||||
select(AuditLogTable.event_type, func.count())
|
||||
.where(
|
||||
AuditLogTable.target_id == task_id,
|
||||
AuditLogTable.event_type.in_(["task.qa_fail", "task.pr_fail"]),
|
||||
)
|
||||
.group_by(AuditLogTable.event_type)
|
||||
)
|
||||
).all()
|
||||
counts: dict[str, int] = {row[0]: row[1] for row in rows}
|
||||
return counts.get("task.qa_fail", 0), counts.get("task.pr_fail", 0)
|
||||
|
||||
async def get_task_metrics(self, task_id: UUID) -> TaskMetrics | None:
|
||||
"""Live granular metrics for one task, or None if the task doesn't exist.
|
||||
|
||||
Composes summed effort + turns/tool_calls/tokens/cost (spawn sessions),
|
||||
the per-stage active-vs-wait decomposition (audit windows x stints), and
|
||||
who-caused-rework (revision_count + named qa/pr fail events).
|
||||
"""
|
||||
task_row = (
|
||||
await self.session.execute(
|
||||
select(
|
||||
TaskTable.started_at,
|
||||
TaskTable.completed_at,
|
||||
TaskTable.revision_count,
|
||||
).where(TaskTable.id == task_id)
|
||||
)
|
||||
).one_or_none()
|
||||
if task_row is None:
|
||||
return None
|
||||
started_at, completed_at, revision_count = task_row
|
||||
# Close open stints / the open final stage window at completed_at for a
|
||||
# terminal task (so stages don't grow past completion), else at now.
|
||||
wall_end = completed_at or datetime.now(UTC)
|
||||
wall_clock = (wall_end - started_at).total_seconds() if started_at else 0.0
|
||||
|
||||
spawn = await self._spawn_rollup_for_task(task_id, wall_end)
|
||||
windows = await self._stage_windows_for_task(task_id, wall_end)
|
||||
qa_fails, pr_fails = await self._task_fail_counts(task_id)
|
||||
stages = compute_stage_effort(windows, spawn["stints"])
|
||||
|
||||
return TaskMetrics(
|
||||
task_id=str(task_id),
|
||||
active_runtime_seconds=spawn["active_runtime_seconds"],
|
||||
wall_clock_seconds=round(max(0.0, wall_clock)),
|
||||
turns=spawn["turns"],
|
||||
tool_calls=spawn["tool_calls"],
|
||||
tokens=spawn["tokens"],
|
||||
cost_usd=spawn["cost_usd"],
|
||||
revision_count=revision_count or 0,
|
||||
qa_fails=qa_fails,
|
||||
pr_fails=pr_fails,
|
||||
stints=len(spawn["stints"]),
|
||||
stages=stages,
|
||||
)
|
||||
|
||||
async def _ceo_latency(
|
||||
self, since: datetime, from_status: str, to_statuses: Sequence[str]
|
||||
) -> tuple[float, float, int]:
|
||||
"""(p50, p90, count) seconds from a ``from_status`` entry to the next
|
||||
CEO-attributed decision (``to_statuses``) on the same task.
|
||||
|
||||
Reads only ``audit_log`` (agent_role='ceo' serializes from the CEO
|
||||
StrEnum). The decision is the earliest ceo row after the from-event.
|
||||
"""
|
||||
sql = text(
|
||||
"""
|
||||
WITH events AS (
|
||||
SELECT target_id, timestamp,
|
||||
(details->>'to_status') AS to_status,
|
||||
(details->>'agent_role') AS role
|
||||
FROM audit_log
|
||||
WHERE event_type LIKE 'task.%' AND timestamp >= :since
|
||||
),
|
||||
paired AS (
|
||||
SELECT (
|
||||
SELECT MIN(d.timestamp) FROM events d
|
||||
WHERE d.target_id = e.target_id
|
||||
AND d.timestamp > e.timestamp
|
||||
AND d.role = 'ceo'
|
||||
AND d.to_status IN :to_statuses
|
||||
) - e.timestamp AS latency
|
||||
FROM events e
|
||||
WHERE e.to_status = :from_status
|
||||
)
|
||||
SELECT
|
||||
PERCENTILE_CONT(0.5) WITHIN GROUP (
|
||||
ORDER BY EXTRACT(epoch FROM latency))::float AS p50,
|
||||
PERCENTILE_CONT(0.9) WITHIN GROUP (
|
||||
ORDER BY EXTRACT(epoch FROM latency))::float AS p90,
|
||||
COUNT(latency) AS n
|
||||
FROM paired WHERE latency IS NOT NULL
|
||||
"""
|
||||
).bindparams(bindparam("to_statuses", expanding=True))
|
||||
row = (
|
||||
await self.session.execute(
|
||||
sql,
|
||||
{
|
||||
"since": since,
|
||||
"from_status": from_status,
|
||||
"to_statuses": list(to_statuses),
|
||||
},
|
||||
)
|
||||
).one()
|
||||
return float(row.p50 or 0.0), float(row.p90 or 0.0), int(row.n or 0)
|
||||
|
||||
async def _ceo_godmode_count(self, since: datetime) -> int:
|
||||
"""Count every CEO-attributed task transition in the window."""
|
||||
sql = text(
|
||||
"""
|
||||
SELECT COUNT(*) AS n FROM audit_log
|
||||
WHERE event_type LIKE 'task.%'
|
||||
AND timestamp >= :since
|
||||
AND (details->>'agent_role') = 'ceo'
|
||||
"""
|
||||
)
|
||||
return int((await self.session.execute(sql, {"since": since})).scalar() or 0)
|
||||
|
||||
async def get_ceo_scorecard(self, days: int = 30) -> CeoScorecard:
|
||||
"""The human CEO's scorecard — approval/unblock dwell + god-mode count."""
|
||||
since = datetime.now(UTC) - timedelta(days=days)
|
||||
approval_p50, approval_p90, approval_n = await self._ceo_latency(
|
||||
since, "awaiting_ceo_approval", CEO_APPROVAL_DECISIONS
|
||||
)
|
||||
unblock_p50, _unblock_p90, unblock_n = await self._ceo_latency(
|
||||
since, "blocked", CEO_UNBLOCK_DECISIONS
|
||||
)
|
||||
godmode = await self._ceo_godmode_count(since)
|
||||
return CeoScorecard(
|
||||
approval_p50_seconds=approval_p50,
|
||||
approval_p90_seconds=approval_p90,
|
||||
approval_count=approval_n,
|
||||
unblock_p50_seconds=unblock_p50,
|
||||
unblock_count=unblock_n,
|
||||
godmode_actions=godmode,
|
||||
)
|
||||
|
||||
_ROLLUP_SUM_COLUMNS: ClassVar[tuple[str, ...]] = (
|
||||
"tasks_completed",
|
||||
"tasks_first_pass",
|
||||
"revisions_caused",
|
||||
"revisions_received",
|
||||
"active_runtime_seconds",
|
||||
"turns",
|
||||
"tool_calls",
|
||||
"tokens",
|
||||
"cost_usd",
|
||||
"qa_reviews_total",
|
||||
"qa_reviews_passed",
|
||||
"escalations",
|
||||
"blocked_others",
|
||||
"idle_seconds",
|
||||
)
|
||||
|
||||
async def _rollup_sums(
|
||||
self, since_date: Any, *, agent_slug: str | None, team: Team | None
|
||||
) -> tuple[dict[str, float], int]:
|
||||
"""SUM the rollup columns over member_performance_daily agent rows.
|
||||
|
||||
Filters to ``member_kind='agent'`` in the window, optionally scoped to
|
||||
one member (``agent_slug``) or one cell (``team``). Returns
|
||||
``(sums, member_count)`` where member_count is the distinct slugs.
|
||||
"""
|
||||
cols = [
|
||||
func.coalesce(
|
||||
func.sum(getattr(MemberPerformanceDailyTable, name)), 0
|
||||
).label(name)
|
||||
for name in self._ROLLUP_SUM_COLUMNS
|
||||
]
|
||||
conds: list[Any] = [
|
||||
MemberPerformanceDailyTable.member_kind == "agent",
|
||||
MemberPerformanceDailyTable.date >= since_date,
|
||||
]
|
||||
if agent_slug is not None:
|
||||
conds.append(MemberPerformanceDailyTable.agent_slug == agent_slug)
|
||||
if team is not None:
|
||||
conds.append(MemberPerformanceDailyTable.team == team.value)
|
||||
row = (
|
||||
await self.session.execute(
|
||||
select(
|
||||
*cols,
|
||||
func.count(func.distinct(MemberPerformanceDailyTable.agent_slug)),
|
||||
).where(and_(*conds))
|
||||
)
|
||||
).one()
|
||||
sums = {
|
||||
name: float(getattr(row, name) or 0) for name in self._ROLLUP_SUM_COLUMNS
|
||||
}
|
||||
member_count = int(row[-1] or 0)
|
||||
return sums, member_count
|
||||
|
||||
async def _live_inflight_overlay(self, agent_id: UUID) -> dict[str, float]:
|
||||
"""Effort of the member's currently-OPEN (running) spawn sessions on
|
||||
non-terminal tasks — the live delta the terminal-day rollup cannot hold
|
||||
yet.
|
||||
|
||||
Disjoint from the rollup by ``ended_at``: ``_msweep_spawn`` rolls up only
|
||||
CLOSED sessions (``ended_at IS NOT NULL``), so this counts only OPEN ones
|
||||
(``ended_at IS NULL``). A just-closed session lands in the rollup on the
|
||||
next ~60s sweep, so there is no double-count. (Summing *all* of a task's
|
||||
sessions here — as an earlier version did via ``get_task_metrics`` —
|
||||
re-counted the closed sessions the rollup already holds, inflating the
|
||||
member's effort on the common reap/respawn path.)
|
||||
"""
|
||||
overlay = {
|
||||
"active_runtime_seconds": 0.0,
|
||||
"turns": 0.0,
|
||||
"tool_calls": 0.0,
|
||||
"tokens": 0.0,
|
||||
"cost_usd": 0.0,
|
||||
}
|
||||
ids = (
|
||||
(
|
||||
await self.session.execute(
|
||||
select(TaskTable.id).where(
|
||||
TaskTable.assigned_to == agent_id,
|
||||
TaskTable.status.notin_(
|
||||
[TaskStatus.COMPLETED, TaskStatus.CANCELLED]
|
||||
),
|
||||
)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
if not ids:
|
||||
return overlay
|
||||
# Aggregate in SQL (mirrors _msweep_spawn); active_runtime for an open
|
||||
# session runs to now(). One row back — no per-row Python branching.
|
||||
s = AgentSpawnSessionTable
|
||||
row = (
|
||||
await self.session.execute(
|
||||
select(
|
||||
func.coalesce(
|
||||
func.sum(func.extract("epoch", func.now() - s.started_at)),
|
||||
0,
|
||||
),
|
||||
func.coalesce(func.sum(s.turns), 0),
|
||||
func.coalesce(func.sum(s.tool_calls), 0),
|
||||
func.coalesce(
|
||||
func.sum(
|
||||
s.tokens_input
|
||||
+ s.tokens_output
|
||||
+ s.tokens_cache_read
|
||||
+ s.tokens_cache_write
|
||||
),
|
||||
0,
|
||||
),
|
||||
func.coalesce(func.sum(s.estimated_cost_usd), 0),
|
||||
).where(
|
||||
s.task_id.in_([str(i) for i in ids]),
|
||||
s.ended_at.is_(None),
|
||||
)
|
||||
)
|
||||
).one()
|
||||
overlay["active_runtime_seconds"] = float(row[0] or 0)
|
||||
overlay["turns"] = float(row[1] or 0)
|
||||
overlay["tool_calls"] = float(row[2] or 0)
|
||||
overlay["tokens"] = float(row[3] or 0)
|
||||
overlay["cost_usd"] = float(row[4] or 0)
|
||||
return overlay
|
||||
|
||||
@staticmethod
|
||||
def _ratio(numerator: float, denominator: float) -> float | None:
|
||||
"""Guarded ratio → None when the denominator is 0."""
|
||||
return round(numerator / denominator, 4) if denominator else None
|
||||
|
||||
async def get_member_scorecard(
|
||||
self, agent_id: UUID, days: int = 30
|
||||
) -> MemberScorecard | None:
|
||||
"""Per-member rollup scorecard + live in-flight overlay, or None if no
|
||||
such agent."""
|
||||
agent = (
|
||||
await self.session.execute(
|
||||
select(AgentTable.slug, AgentTable.name).where(
|
||||
AgentTable.id == agent_id
|
||||
)
|
||||
)
|
||||
).one_or_none()
|
||||
if agent is None:
|
||||
return None
|
||||
slug, name = agent
|
||||
since_date = (datetime.now(UTC) - timedelta(days=days)).date()
|
||||
sums, _ = await self._rollup_sums(since_date, agent_slug=slug, team=None)
|
||||
|
||||
overlay = await self._live_inflight_overlay(agent_id)
|
||||
includes_live = any(v for v in overlay.values())
|
||||
active_runtime = (
|
||||
sums["active_runtime_seconds"] + overlay["active_runtime_seconds"]
|
||||
)
|
||||
turns = int(sums["turns"] + overlay["turns"])
|
||||
tool_calls = int(sums["tool_calls"] + overlay["tool_calls"])
|
||||
tokens = int(sums["tokens"] + overlay["tokens"])
|
||||
cost = sums["cost_usd"] + overlay["cost_usd"]
|
||||
completed = sums["tasks_completed"]
|
||||
|
||||
return MemberScorecard(
|
||||
scope="member",
|
||||
id=str(agent_id),
|
||||
name=name,
|
||||
tasks_completed=int(completed),
|
||||
first_pass_yield=self._ratio(sums["tasks_first_pass"], completed),
|
||||
effort_throughput_per_hour=self._ratio(completed, active_runtime / 3600),
|
||||
active_runtime_hours=active_runtime / 3600,
|
||||
turns=turns,
|
||||
tool_calls=tool_calls,
|
||||
tokens=tokens,
|
||||
cost_usd=cost,
|
||||
turns_per_task=self._ratio(turns, completed),
|
||||
tool_calls_per_task=self._ratio(tool_calls, completed),
|
||||
revisions_caused=int(sums["revisions_caused"]),
|
||||
revisions_received=int(sums["revisions_received"]),
|
||||
qa_pass_rate=self._ratio(
|
||||
sums["qa_reviews_passed"], sums["qa_reviews_total"]
|
||||
),
|
||||
escalations=int(sums["escalations"]),
|
||||
blocked_others=int(sums["blocked_others"]),
|
||||
idle_hours=sums["idle_seconds"] / 3600,
|
||||
utilization=self._ratio(
|
||||
sums["active_runtime_seconds"],
|
||||
sums["active_runtime_seconds"] + sums["idle_seconds"],
|
||||
),
|
||||
includes_live_inflight=includes_live,
|
||||
)
|
||||
|
||||
async def get_org_scorecard(
|
||||
self, team: Team | None = None, days: int = 30
|
||||
) -> OrgScorecard:
|
||||
"""Team (or whole-org when team is None) rollup aggregate."""
|
||||
since_date = (datetime.now(UTC) - timedelta(days=days)).date()
|
||||
sums, member_count = await self._rollup_sums(
|
||||
since_date, agent_slug=None, team=team
|
||||
)
|
||||
completed = sums["tasks_completed"]
|
||||
active_runtime = sums["active_runtime_seconds"]
|
||||
return OrgScorecard(
|
||||
scope="team" if team else "org",
|
||||
team=team.value if team else None,
|
||||
member_count=member_count,
|
||||
tasks_completed=int(completed),
|
||||
first_pass_yield=self._ratio(sums["tasks_first_pass"], completed),
|
||||
effort_throughput_per_hour=self._ratio(completed, active_runtime / 3600),
|
||||
active_runtime_hours=active_runtime / 3600,
|
||||
turns=int(sums["turns"]),
|
||||
tool_calls=int(sums["tool_calls"]),
|
||||
tokens=int(sums["tokens"]),
|
||||
cost_usd=sums["cost_usd"],
|
||||
revisions_caused=int(sums["revisions_caused"]),
|
||||
revisions_received=int(sums["revisions_received"]),
|
||||
)
|
||||
|
||||
async def get_rework_metrics(
|
||||
self, team: Team | None = None, days: int = 30
|
||||
) -> ReworkReport:
|
||||
|
||||
@@ -12,7 +12,7 @@ Also implements the ACK system for tracking acknowledgments.
|
||||
import asyncio
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from typing import ClassVar, Literal, cast
|
||||
from typing import TYPE_CHECKING, ClassVar, Literal, cast
|
||||
from uuid import UUID
|
||||
|
||||
import structlog
|
||||
@@ -32,9 +32,33 @@ from roboco.services.base import BaseService, NotFoundError
|
||||
from roboco.services.notification_dedup import all_recipients_recently_notified
|
||||
from roboco.utils.converters import require_uuid
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from roboco.models.metrics import TaskMetrics
|
||||
|
||||
_log = structlog.get_logger(service="notification_delivery")
|
||||
|
||||
|
||||
def _format_completion_body(task: TaskTable, metrics: "TaskMetrics | None") -> str:
|
||||
"""Human-readable completion summary — real effort vs wall-clock, not a lone
|
||||
wall-clock figure. Degrades to wall-clock-only (turns 'n/a') when there are
|
||||
no spawn sessions / pre-turns-migration data."""
|
||||
title = task.title or "Untitled"
|
||||
if metrics is None:
|
||||
return f"Task '{title}' completed."
|
||||
wall_h = round(metrics.wall_clock_seconds / 3600, 1)
|
||||
active_h = round(metrics.active_runtime_seconds / 3600, 1)
|
||||
turns = str(metrics.turns) if metrics.turns else "n/a"
|
||||
return (
|
||||
f"Task '{title}' completed.\n\n"
|
||||
f"Active effort: {active_h}h across {metrics.stints} stint(s) "
|
||||
f"({turns} turns, {metrics.tool_calls} tool-calls)\n"
|
||||
f"Wall-clock: {wall_h}h\n"
|
||||
f"Revisions: {metrics.revision_count} "
|
||||
f"({metrics.qa_fails} QA / {metrics.pr_fails} PR)\n"
|
||||
f"Cost: ${round(metrics.cost_usd, 2)}"
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Deferred bus publish — transactional outbox (F107)
|
||||
# =============================================================================
|
||||
@@ -837,6 +861,35 @@ class NotificationDeliveryService(BaseService):
|
||||
)
|
||||
await self._persist_and_deliver(notification)
|
||||
|
||||
async def notify_ceo_of_completion(self, *, task: TaskTable, task_id: UUID) -> None:
|
||||
"""CEO-facing completion notification with the granular effort breakdown.
|
||||
|
||||
Replaces the coarse "completed in Xh" wall-clock figure with real effort
|
||||
vs wall-clock + turns/stints/revisions/cost from the per-task metrics.
|
||||
Best-effort: a metrics or delivery failure must never block completion.
|
||||
"""
|
||||
ceo = await self._get_ceo_agent()
|
||||
if not ceo:
|
||||
return
|
||||
from roboco.services.metrics import MetricsService
|
||||
|
||||
try:
|
||||
metrics = await MetricsService(self.session).get_task_metrics(task_id)
|
||||
except Exception: # metrics are best-effort — degrade to wall-clock-only
|
||||
metrics = None
|
||||
from_agent = cast("UUID", task.assigned_to) if task.assigned_to else ceo.id
|
||||
notification = NotificationTable(
|
||||
type=NotificationType.ALERT,
|
||||
priority=NotificationPriority.NORMAL,
|
||||
from_agent=from_agent,
|
||||
to_agents=[ceo.id],
|
||||
subject=f"Completed: {(task.title or 'Untitled')[:60]}",
|
||||
body=_format_completion_body(task, metrics),
|
||||
related_task_id=task_id,
|
||||
requires_ack=ACK_REQUIRED_BY_TYPE[NotificationType.ALERT],
|
||||
)
|
||||
await self._persist_and_deliver(notification)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Private helpers for recipient resolution + persist
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
+101
-1
@@ -784,6 +784,33 @@ class TaskService(BaseService):
|
||||
)
|
||||
)
|
||||
|
||||
def _emit_escalation_audit(
|
||||
self, task: TaskTable, *, escalator_slug: str, target_slug: str
|
||||
) -> None:
|
||||
"""Emit a ``task.escalated`` audit row (per-member escalation metric).
|
||||
|
||||
Additive to the ``task.blocked`` / pool-release audit of an escalation —
|
||||
it carries WHO escalated (``escalator_slug``) so the member scorecard can
|
||||
charge the escalation to the escalator. Written in the caller's session
|
||||
so it commits atomically with the escalation. Best-effort for metrics:
|
||||
never gates the escalation itself.
|
||||
"""
|
||||
from roboco.db.tables import AuditLogTable
|
||||
|
||||
self.session.add(
|
||||
AuditLogTable(
|
||||
event_type="task.escalated",
|
||||
agent_id=None,
|
||||
target_type="task",
|
||||
target_id=task.id,
|
||||
severity="info",
|
||||
details={
|
||||
"escalator_slug": escalator_slug,
|
||||
"target_slug": target_slug,
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _audit_events_for(to_status: str, agent_role: str | None) -> list[str]:
|
||||
"""Audit event types to emit for a transition.
|
||||
@@ -5077,6 +5104,10 @@ class TaskService(BaseService):
|
||||
|
||||
await self._trigger_completion_hooks(task, agent_id)
|
||||
await self._unblock_dependents(task_id)
|
||||
# Emit the completion event (previously defined but never emitted) so the
|
||||
# WS bridge can forward a live completion signal. The CEO-facing granular
|
||||
# notification fires on the root's ceo_approve, not on each leaf complete.
|
||||
await self._emit_task_event(EventType.TASK_COMPLETED, task_id, {})
|
||||
return task
|
||||
|
||||
async def _escalation_diverts_to_pool(
|
||||
@@ -5172,6 +5203,9 @@ class TaskService(BaseService):
|
||||
blocked_target_slug=target_slug,
|
||||
reason=reason,
|
||||
)
|
||||
self._emit_escalation_audit(
|
||||
task, escalator_slug=escalator_slug, target_slug=target_slug
|
||||
)
|
||||
return True
|
||||
if task.assigned_to and not task.blocker_raised_by:
|
||||
task.blocker_raised_by = cast("Any", task.assigned_to)
|
||||
@@ -5207,6 +5241,9 @@ class TaskService(BaseService):
|
||||
agent_role=None,
|
||||
audit_agent_id=pre_block_owner,
|
||||
)
|
||||
self._emit_escalation_audit(
|
||||
task, escalator_slug=escalator_slug, target_slug=target_slug
|
||||
)
|
||||
self.log.info(
|
||||
"Task escalated and blocked",
|
||||
task_id=str(task.id),
|
||||
@@ -5384,6 +5421,11 @@ class TaskService(BaseService):
|
||||
# Unblock any tasks waiting on this one
|
||||
await self._unblock_dependents(task_id)
|
||||
|
||||
# Emit the completion event (previously dead code — never emitted) + the
|
||||
# CEO-facing granular completion notification (real effort vs wall-clock).
|
||||
await self._emit_task_event(EventType.TASK_COMPLETED, task_id, {})
|
||||
await self._notify_completion(task, task_id)
|
||||
|
||||
# Emit event for CEO approval
|
||||
await self._emit_task_event(
|
||||
EventType.TASK_CEO_APPROVED,
|
||||
@@ -5979,6 +6021,26 @@ class TaskService(BaseService):
|
||||
|
||||
return task
|
||||
|
||||
async def _notify_completion(self, task: TaskTable, task_id: UUID) -> None:
|
||||
"""Best-effort CEO completion notification (granular effort breakdown).
|
||||
|
||||
A notification/metrics failure must never block the completion, so any
|
||||
error is logged and swallowed.
|
||||
"""
|
||||
try:
|
||||
from roboco.services.notification_delivery import (
|
||||
get_notification_delivery_service,
|
||||
)
|
||||
|
||||
delivery = get_notification_delivery_service(self.session)
|
||||
await delivery.notify_ceo_of_completion(task=task, task_id=task_id)
|
||||
except Exception as exc:
|
||||
self.log.warning(
|
||||
"Completion notification failed",
|
||||
task_id=str(task_id),
|
||||
error=str(exc),
|
||||
)
|
||||
|
||||
async def _emit_task_event(
|
||||
self,
|
||||
event_type: EventType,
|
||||
@@ -6040,6 +6102,26 @@ class TaskService(BaseService):
|
||||
completed_dependency=str(completed_task_id),
|
||||
)
|
||||
|
||||
if blocked_tasks:
|
||||
# Metrics (blocked-others): record how many downstream tasks this
|
||||
# completed task was blocking. _unblock_dependents PRUNES the
|
||||
# dependency edges above, destroying the only other record — so
|
||||
# capture the count NOW as a durable audit row. The sweeper
|
||||
# attributes it to the completed task's owner. target_id is the
|
||||
# BLOCKER (completed) task.
|
||||
from roboco.db.tables import AuditLogTable
|
||||
|
||||
self.session.add(
|
||||
AuditLogTable(
|
||||
event_type="task.unblocked_dependents",
|
||||
agent_id=None,
|
||||
target_type="task",
|
||||
target_id=completed_task_id,
|
||||
severity="info",
|
||||
details={"count": len(blocked_tasks)},
|
||||
)
|
||||
)
|
||||
|
||||
await self.session.flush()
|
||||
|
||||
async def _revive_unblocked_dependent(self, task: TaskTable) -> None:
|
||||
@@ -8146,7 +8228,13 @@ class TaskService(BaseService):
|
||||
return task
|
||||
|
||||
async def mark_agent_idle(self, agent_id: UUID) -> None:
|
||||
"""Set agent.status = IDLE."""
|
||||
"""Set agent.status = IDLE + emit an ``agent.idle`` audit row.
|
||||
|
||||
The audit row (target = the agent, details.agent_slug) is the idle
|
||||
signal the member scorecard needs — it lets the sweeper compute idle
|
||||
time as the gap from an idle mark to the member's next spawn. Written in
|
||||
the same session so it commits with the status change.
|
||||
"""
|
||||
result = await self.session.execute(
|
||||
select(AgentTable).where(AgentTable.id == agent_id)
|
||||
)
|
||||
@@ -8154,6 +8242,18 @@ class TaskService(BaseService):
|
||||
if agent is None:
|
||||
return
|
||||
agent.status = AgentStatus.IDLE
|
||||
from roboco.db.tables import AuditLogTable
|
||||
|
||||
self.session.add(
|
||||
AuditLogTable(
|
||||
event_type="agent.idle",
|
||||
agent_id=agent_id,
|
||||
target_type="agent",
|
||||
target_id=agent_id,
|
||||
severity="info",
|
||||
details={"agent_slug": agent.slug},
|
||||
)
|
||||
)
|
||||
await self.session.flush()
|
||||
|
||||
async def _qa_or_doc_claim(
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
"""CEO completion notification — granular effort breakdown (phase 6).
|
||||
|
||||
The pure body formatter (real effort vs wall-clock; degrades to wall-clock-only)
|
||||
+ notify_ceo_of_completion end to end against real PG.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from types import SimpleNamespace
|
||||
from typing import TYPE_CHECKING, Any, cast
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from roboco.db.tables import (
|
||||
AgentSpawnSessionTable,
|
||||
AgentTable,
|
||||
NotificationTable,
|
||||
ProjectTable,
|
||||
TaskTable,
|
||||
)
|
||||
from roboco.models.base import (
|
||||
AgentRole,
|
||||
AgentStatus,
|
||||
Complexity,
|
||||
NotificationType,
|
||||
TaskNature,
|
||||
TaskStatus,
|
||||
TaskType,
|
||||
Team,
|
||||
)
|
||||
from roboco.models.metrics import TaskMetrics
|
||||
from roboco.services.notification_delivery import (
|
||||
_format_completion_body,
|
||||
get_notification_delivery_service,
|
||||
)
|
||||
from sqlalchemy import select
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
|
||||
def _metrics(**over: Any) -> TaskMetrics:
|
||||
base: dict[str, Any] = {
|
||||
"task_id": str(uuid4()),
|
||||
"active_runtime_seconds": 3600,
|
||||
"wall_clock_seconds": 7200,
|
||||
"turns": 42,
|
||||
"tool_calls": 99,
|
||||
"tokens": 1000,
|
||||
"cost_usd": 4.2,
|
||||
"revision_count": 2,
|
||||
"qa_fails": 1,
|
||||
"pr_fails": 1,
|
||||
"stints": 3,
|
||||
"stages": [],
|
||||
}
|
||||
base.update(over)
|
||||
return TaskMetrics(**base)
|
||||
|
||||
|
||||
def test_format_body_with_metrics() -> None:
|
||||
body = _format_completion_body(
|
||||
cast("TaskTable", SimpleNamespace(title="Auth flow")), _metrics()
|
||||
)
|
||||
assert "Auth flow" in body
|
||||
assert "Active effort: 1.0h across 3 stint(s)" in body
|
||||
assert "42 turns" in body
|
||||
assert "Wall-clock: 2.0h" in body
|
||||
assert "2 (1 QA / 1 PR)" in body
|
||||
assert "$4.2" in body
|
||||
|
||||
|
||||
def test_format_body_turns_na_when_zero() -> None:
|
||||
body = _format_completion_body(
|
||||
cast("TaskTable", SimpleNamespace(title="T")), _metrics(turns=0)
|
||||
)
|
||||
assert "n/a turns" in body # pre-turns-migration / Grok
|
||||
|
||||
|
||||
def test_format_body_degrades_without_metrics() -> None:
|
||||
body = _format_completion_body(cast("TaskTable", SimpleNamespace(title="T")), None)
|
||||
assert body == "Task 'T' completed."
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def env(db_session: AsyncSession) -> AsyncIterator[dict]:
|
||||
# The CEO is a singleton in the real system, and `_get_ceo_agent()` resolves
|
||||
# it by `role == CEO` with `scalar_one_or_none()`. The session-scoped test DB
|
||||
# is shared across the run, and a sibling real-DB test commits a role=CEO
|
||||
# agent (slug="ceo") without cleanup, so it can already be present here.
|
||||
# Reuse an existing CEO rather than inserting a second one — creating another
|
||||
# would both collide on the unique slug and make `_get_ceo_agent()` raise
|
||||
# MultipleResultsFound. Order-independent: in isolation we create one.
|
||||
existing_ceo = (
|
||||
(
|
||||
await db_session.execute(
|
||||
select(AgentTable).where(AgentTable.role == AgentRole.CEO)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.first()
|
||||
)
|
||||
ceo = existing_ceo or AgentTable(
|
||||
id=uuid4(),
|
||||
name="CEO",
|
||||
slug=f"ceo-{uuid4().hex[:6]}",
|
||||
role=AgentRole.CEO,
|
||||
team=None,
|
||||
status=AgentStatus.ACTIVE,
|
||||
model_config={},
|
||||
system_prompt="x",
|
||||
capabilities=[],
|
||||
permissions={},
|
||||
metrics={},
|
||||
)
|
||||
if existing_ceo is None:
|
||||
db_session.add(ceo)
|
||||
dev = AgentTable(
|
||||
id=uuid4(),
|
||||
name="dev",
|
||||
slug=f"be-dev-{uuid4().hex[:6]}",
|
||||
role=AgentRole.DEVELOPER,
|
||||
team=Team.BACKEND,
|
||||
status=AgentStatus.ACTIVE,
|
||||
model_config={},
|
||||
system_prompt="x",
|
||||
capabilities=[],
|
||||
permissions={},
|
||||
metrics={},
|
||||
)
|
||||
db_session.add(dev)
|
||||
await db_session.flush()
|
||||
project = ProjectTable(
|
||||
id=uuid4(),
|
||||
name="P",
|
||||
slug=f"p-{uuid4().hex[:6]}",
|
||||
git_url="https://example.com/r.git",
|
||||
assigned_cell=Team.BACKEND,
|
||||
created_by=dev.id,
|
||||
)
|
||||
db_session.add(project)
|
||||
await db_session.flush()
|
||||
yield {"db": db_session, "ceo": ceo, "dev": dev, "project_id": project.id}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_notify_ceo_of_completion_creates_alert(env: dict) -> None:
|
||||
db = env["db"]
|
||||
base = datetime.now(UTC) - timedelta(hours=2)
|
||||
task = TaskTable(
|
||||
id=uuid4(),
|
||||
title="Ship it",
|
||||
description="d",
|
||||
acceptance_criteria=["ac"],
|
||||
task_type=TaskType.CODE,
|
||||
nature=TaskNature.TECHNICAL,
|
||||
status=TaskStatus.COMPLETED,
|
||||
team=Team.BACKEND,
|
||||
project_id=env["project_id"],
|
||||
created_by=env["dev"].id,
|
||||
assigned_to=env["dev"].id,
|
||||
estimated_complexity=Complexity.MEDIUM,
|
||||
started_at=base,
|
||||
completed_at=base + timedelta(seconds=600),
|
||||
)
|
||||
db.add(task)
|
||||
await db.flush()
|
||||
db.add(
|
||||
AgentSpawnSessionTable(
|
||||
id=uuid4(),
|
||||
agent_slug=env["dev"].slug,
|
||||
team="backend",
|
||||
role="developer",
|
||||
model="claude",
|
||||
task_id=str(task.id),
|
||||
started_at=base,
|
||||
ended_at=base + timedelta(seconds=300),
|
||||
turns=7,
|
||||
tool_calls=12,
|
||||
tokens_input=10,
|
||||
tokens_output=5,
|
||||
estimated_cost_usd=0.5,
|
||||
)
|
||||
)
|
||||
await db.flush()
|
||||
|
||||
delivery = get_notification_delivery_service(db)
|
||||
await delivery.notify_ceo_of_completion(task=task, task_id=cast("UUID", task.id))
|
||||
|
||||
rows = (
|
||||
(
|
||||
await db.execute(
|
||||
select(NotificationTable).where(
|
||||
NotificationTable.related_task_id == task.id
|
||||
)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
assert len(rows) == 1
|
||||
note = rows[0]
|
||||
assert note.type == NotificationType.ALERT
|
||||
assert env["ceo"].id in note.to_agents
|
||||
assert "Active effort" in note.body
|
||||
assert "Ship it" in note.subject
|
||||
@@ -14,10 +14,18 @@ from httpx import ASGITransport, AsyncClient
|
||||
from roboco.api.deps import get_agent_context, get_db
|
||||
from roboco.api.routes.dashboard import get_main_pm_kanban
|
||||
from roboco.api.routes.dashboard import router as dashboard_router
|
||||
from roboco.db.tables import AgentTable
|
||||
from roboco.db.tables import AgentTable, ProjectTable, TaskTable
|
||||
from roboco.models import AgentRole, AgentStatus
|
||||
from roboco.models.base import (
|
||||
Complexity,
|
||||
TaskNature,
|
||||
TaskStatus,
|
||||
TaskType,
|
||||
Team,
|
||||
)
|
||||
from roboco.models.permissions import AgentContext
|
||||
from roboco.services.dashboard import reset_storage
|
||||
from sqlalchemy import select
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import AsyncGenerator, AsyncIterator
|
||||
@@ -408,3 +416,110 @@ async def test_get_main_pm_kanban_function_directly(
|
||||
"""
|
||||
result = await get_main_pm_kanban(db_session)
|
||||
assert isinstance(result, dict)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ceo_scorecard_endpoint(dashboard_client: AsyncClient) -> None:
|
||||
resp = await dashboard_client.get(
|
||||
"/api/dashboard/metrics/member/ceo?days=30", headers=_HDR
|
||||
)
|
||||
assert resp.status_code == HTTPStatus.OK
|
||||
body = resp.json()
|
||||
assert body["member_kind"] == "ceo"
|
||||
assert set(body) >= {
|
||||
"approval_p50_seconds",
|
||||
"approval_count",
|
||||
"unblock_p50_seconds",
|
||||
"unblock_count",
|
||||
"godmode_actions",
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_member_scorecard_404_when_absent(dashboard_client: AsyncClient) -> None:
|
||||
resp = await dashboard_client.get(
|
||||
f"/api/dashboard/metrics/member/{uuid4()}", headers=_HDR
|
||||
)
|
||||
assert resp.status_code == HTTPStatus.NOT_FOUND
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ceo_route_wins_over_member_uuid_route(
|
||||
dashboard_client: AsyncClient,
|
||||
) -> None:
|
||||
# The literal "ceo" must resolve to the CEO route, not the {agent_id} route.
|
||||
resp = await dashboard_client.get("/api/dashboard/metrics/member/ceo", headers=_HDR)
|
||||
assert resp.status_code == HTTPStatus.OK
|
||||
assert resp.json()["member_kind"] == "ceo"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_org_scorecard_endpoint(dashboard_client: AsyncClient) -> None:
|
||||
resp = await dashboard_client.get("/api/dashboard/metrics/org", headers=_HDR)
|
||||
assert resp.status_code == HTTPStatus.OK
|
||||
body = resp.json()
|
||||
assert body["scope"] == "org"
|
||||
assert set(body) >= {"member_count", "tasks_completed", "first_pass_yield"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_task_metrics_404_for_missing_task(
|
||||
dashboard_client: AsyncClient,
|
||||
) -> None:
|
||||
resp = await dashboard_client.get(
|
||||
f"/api/dashboard/metrics/task/{uuid4()}", headers=_HDR
|
||||
)
|
||||
assert resp.status_code == HTTPStatus.NOT_FOUND
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_task_metrics_returns_shape_for_existing_task(
|
||||
db_session: AsyncSession, dashboard_client: AsyncClient
|
||||
) -> None:
|
||||
creator = (await db_session.execute(select(AgentTable).limit(1))).scalar_one()
|
||||
project = ProjectTable(
|
||||
id=uuid4(),
|
||||
name="P",
|
||||
slug=f"p-{uuid4().hex[:6]}",
|
||||
git_url="https://example.com/r.git",
|
||||
assigned_cell=Team.BACKEND,
|
||||
created_by=creator.id,
|
||||
)
|
||||
db_session.add(project)
|
||||
await db_session.flush()
|
||||
task = TaskTable(
|
||||
id=uuid4(),
|
||||
title="t",
|
||||
description="d",
|
||||
acceptance_criteria=["ac"],
|
||||
task_type=TaskType.CODE,
|
||||
nature=TaskNature.TECHNICAL,
|
||||
status=TaskStatus.IN_PROGRESS,
|
||||
team=Team.BACKEND,
|
||||
project_id=project.id,
|
||||
created_by=creator.id,
|
||||
estimated_complexity=Complexity.MEDIUM,
|
||||
)
|
||||
db_session.add(task)
|
||||
await db_session.flush()
|
||||
|
||||
resp = await dashboard_client.get(
|
||||
f"/api/dashboard/metrics/task/{task.id}", headers=_HDR
|
||||
)
|
||||
assert resp.status_code == HTTPStatus.OK
|
||||
body = resp.json()
|
||||
assert body["task_id"] == str(task.id)
|
||||
assert set(body) >= {
|
||||
"active_runtime_seconds",
|
||||
"wall_clock_seconds",
|
||||
"turns",
|
||||
"tool_calls",
|
||||
"tokens",
|
||||
"cost_usd",
|
||||
"revision_count",
|
||||
"qa_fails",
|
||||
"pr_fails",
|
||||
"stints",
|
||||
"stages",
|
||||
}
|
||||
assert isinstance(body["stages"], list)
|
||||
|
||||
@@ -0,0 +1,299 @@
|
||||
"""_sweep_member_performance — the granular per-member rollup, against real PG.
|
||||
|
||||
Seeds a day of spawn sessions + completed tasks + the new audit events
|
||||
(escalated / unblocked_dependents / agent.idle / qa pass+fail / CEO decisions),
|
||||
runs the sweep, and asserts the agent + CEO rows — plus idempotency (a second
|
||||
sweep overwrites, never doubles).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from unittest.mock import patch
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from roboco.db.tables import (
|
||||
AgentSpawnSessionTable,
|
||||
AgentTable,
|
||||
AuditLogTable,
|
||||
MemberPerformanceDailyTable,
|
||||
ProjectTable,
|
||||
TaskTable,
|
||||
)
|
||||
from roboco.models.base import (
|
||||
AgentRole,
|
||||
AgentStatus,
|
||||
Complexity,
|
||||
TaskNature,
|
||||
TaskStatus,
|
||||
TaskType,
|
||||
Team,
|
||||
)
|
||||
from roboco.runtime.orchestrator import AgentOrchestrator
|
||||
from sqlalchemy import select
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
# Yesterday at noon UTC: safely inside the 7-day window and far from midnight,
|
||||
# so started_at / completed_at (+1h) and all audit events share ONE date
|
||||
# (the sweep legitimately splits cross-midnight work across days — not under test
|
||||
# here). This keeps the seeded member's rollup on a single row.
|
||||
_BASE = (datetime.now(UTC) - timedelta(days=1)).replace(
|
||||
hour=12, minute=0, second=0, microsecond=0
|
||||
)
|
||||
|
||||
_ACTIVE_SECONDS = 600
|
||||
_APPROVAL_DWELL_SECONDS = 300
|
||||
_COMPLETED_TASKS = 2
|
||||
_BLOCKED_OTHERS = 2
|
||||
_QA_TOTAL = 2
|
||||
|
||||
|
||||
class _NoCommitSession:
|
||||
"""Delegates to the real test session but turns commit() into flush() so the
|
||||
per-test rollback isolation holds while the sweep still 'commits'."""
|
||||
|
||||
def __init__(self, real: Any) -> None:
|
||||
self._real = real
|
||||
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
return getattr(self._real, name)
|
||||
|
||||
async def commit(self) -> None:
|
||||
await self._real.flush()
|
||||
|
||||
|
||||
def _agent(role: AgentRole, slug: str) -> AgentTable:
|
||||
return AgentTable(
|
||||
id=uuid4(),
|
||||
name=slug,
|
||||
slug=slug,
|
||||
role=role,
|
||||
team=Team.BACKEND,
|
||||
status=AgentStatus.ACTIVE,
|
||||
model_config={},
|
||||
system_prompt="x",
|
||||
capabilities=[],
|
||||
permissions={},
|
||||
metrics={},
|
||||
)
|
||||
|
||||
|
||||
def _audit(
|
||||
target_id: Any,
|
||||
event_type: str,
|
||||
ts: datetime,
|
||||
*,
|
||||
agent_id: Any = None,
|
||||
details: dict[str, Any] | None = None,
|
||||
) -> AuditLogTable:
|
||||
return AuditLogTable(
|
||||
id=uuid4(),
|
||||
event_type=event_type,
|
||||
agent_id=agent_id,
|
||||
target_type="task",
|
||||
target_id=target_id,
|
||||
severity="info",
|
||||
details=details or {},
|
||||
timestamp=ts,
|
||||
)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def seeded(db_session: AsyncSession) -> AsyncIterator[dict]:
|
||||
dev = _agent(AgentRole.DEVELOPER, f"be-dev-{uuid4().hex[:6]}")
|
||||
qa = _agent(AgentRole.QA, f"be-qa-{uuid4().hex[:6]}")
|
||||
db_session.add_all([dev, qa])
|
||||
await db_session.flush()
|
||||
project = ProjectTable(
|
||||
id=uuid4(),
|
||||
name="P",
|
||||
slug=f"p-{uuid4().hex[:6]}",
|
||||
git_url="https://example.com/r.git",
|
||||
assigned_cell=Team.BACKEND,
|
||||
created_by=dev.id,
|
||||
)
|
||||
db_session.add(project)
|
||||
await db_session.flush()
|
||||
|
||||
task = TaskTable(
|
||||
id=uuid4(),
|
||||
title="t",
|
||||
description="d",
|
||||
acceptance_criteria=["ac"],
|
||||
task_type=TaskType.CODE,
|
||||
nature=TaskNature.TECHNICAL,
|
||||
status=TaskStatus.COMPLETED,
|
||||
team=Team.BACKEND,
|
||||
project_id=project.id,
|
||||
created_by=dev.id,
|
||||
assigned_to=dev.id,
|
||||
revision_count=1,
|
||||
estimated_complexity=Complexity.MEDIUM,
|
||||
started_at=_BASE,
|
||||
completed_at=_BASE + timedelta(hours=1),
|
||||
)
|
||||
blocker = TaskTable(
|
||||
id=uuid4(),
|
||||
title="b",
|
||||
description="d",
|
||||
acceptance_criteria=["ac"],
|
||||
task_type=TaskType.CODE,
|
||||
nature=TaskNature.TECHNICAL,
|
||||
status=TaskStatus.COMPLETED,
|
||||
team=Team.BACKEND,
|
||||
project_id=project.id,
|
||||
created_by=dev.id,
|
||||
assigned_to=dev.id,
|
||||
estimated_complexity=Complexity.MEDIUM,
|
||||
started_at=_BASE,
|
||||
completed_at=_BASE + timedelta(hours=1),
|
||||
)
|
||||
db_session.add_all([task, blocker])
|
||||
await db_session.flush()
|
||||
|
||||
db_session.add(
|
||||
AgentSpawnSessionTable(
|
||||
id=uuid4(),
|
||||
agent_slug=dev.slug,
|
||||
team="backend",
|
||||
role="developer",
|
||||
model="claude",
|
||||
task_id=str(task.id),
|
||||
started_at=_BASE,
|
||||
ended_at=_BASE + timedelta(seconds=600),
|
||||
turns=5,
|
||||
tool_calls=10,
|
||||
tokens_input=100,
|
||||
tokens_output=80,
|
||||
estimated_cost_usd=1.5,
|
||||
)
|
||||
)
|
||||
db_session.add_all(
|
||||
[
|
||||
_audit(
|
||||
task.id,
|
||||
"task.qa_fail",
|
||||
_BASE,
|
||||
agent_id=qa.id,
|
||||
details={"agent_role": "qa"},
|
||||
),
|
||||
_audit(
|
||||
task.id,
|
||||
"task.awaiting_documentation",
|
||||
_BASE + timedelta(minutes=1),
|
||||
agent_id=qa.id,
|
||||
details={"agent_role": "qa"},
|
||||
),
|
||||
_audit(
|
||||
task.id,
|
||||
"task.escalated",
|
||||
_BASE,
|
||||
details={"escalator_slug": dev.slug, "target_slug": qa.slug},
|
||||
),
|
||||
_audit(
|
||||
blocker.id, "task.unblocked_dependents", _BASE, details={"count": 2}
|
||||
),
|
||||
_audit(
|
||||
dev.id,
|
||||
"agent.idle",
|
||||
_BASE + timedelta(minutes=5),
|
||||
agent_id=dev.id,
|
||||
details={"agent_slug": dev.slug},
|
||||
),
|
||||
# CEO decision: approval + god-mode (to_status is what the pairing reads).
|
||||
_audit(
|
||||
task.id,
|
||||
"task.awaiting_ceo_approval",
|
||||
_BASE,
|
||||
details={"to_status": "awaiting_ceo_approval", "agent_role": "main_pm"},
|
||||
),
|
||||
_audit(
|
||||
task.id,
|
||||
"task.completed",
|
||||
_BASE + timedelta(seconds=300),
|
||||
details={"to_status": "completed", "agent_role": "ceo"},
|
||||
),
|
||||
]
|
||||
)
|
||||
await db_session.flush()
|
||||
yield {"db": db_session, "dev": dev.slug, "qa": qa.slug}
|
||||
|
||||
|
||||
async def _rows(db: AsyncSession, slug: str) -> list[Any]:
|
||||
return list(
|
||||
(
|
||||
await db.execute(
|
||||
select(MemberPerformanceDailyTable).where(
|
||||
MemberPerformanceDailyTable.agent_slug == slug
|
||||
)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
|
||||
|
||||
async def _run_sweep(db: AsyncSession) -> None:
|
||||
orch = AgentOrchestrator.__new__(AgentOrchestrator)
|
||||
|
||||
@asynccontextmanager
|
||||
async def _cm() -> Any:
|
||||
yield _NoCommitSession(db)
|
||||
|
||||
with patch("roboco.db.base.get_session_factory", return_value=_cm):
|
||||
await orch._sweep_member_performance()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sweep_populates_agent_and_ceo_rows(seeded: dict) -> None:
|
||||
db = seeded["db"]
|
||||
await _run_sweep(db)
|
||||
|
||||
dev_rows = await _rows(db, seeded["dev"])
|
||||
assert len(dev_rows) == 1
|
||||
dev = dev_rows[0]
|
||||
assert dev.member_kind == "agent"
|
||||
assert dev.active_runtime_seconds == _ACTIVE_SECONDS
|
||||
assert (dev.turns, dev.tool_calls, dev.tokens) == (5, 10, 180)
|
||||
assert dev.cost_usd == pytest.approx(1.5)
|
||||
assert dev.tasks_completed == _COMPLETED_TASKS # task + blocker
|
||||
assert dev.tasks_first_pass == 1 # blocker had 0 revisions
|
||||
assert dev.revisions_received == 1
|
||||
assert dev.escalations == 1
|
||||
assert dev.blocked_others == _BLOCKED_OTHERS
|
||||
assert dev.idle_seconds > 0
|
||||
|
||||
qa_rows = await _rows(db, seeded["qa"])
|
||||
assert len(qa_rows) == 1
|
||||
qa = qa_rows[0]
|
||||
assert qa.revisions_caused == 1 # the qa_fail
|
||||
assert qa.qa_reviews_passed == 1
|
||||
assert qa.qa_reviews_total == _QA_TOTAL # 1 pass + 1 fail
|
||||
|
||||
ceo_rows = await _rows(db, "")
|
||||
assert len(ceo_rows) == 1
|
||||
ceo = ceo_rows[0]
|
||||
assert ceo.member_kind == "ceo"
|
||||
assert ceo.godmode_actions == 1
|
||||
assert ceo.ceo_approval_dwell_seconds == _APPROVAL_DWELL_SECONDS
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sweep_is_idempotent(seeded: dict) -> None:
|
||||
db = seeded["db"]
|
||||
await _run_sweep(db)
|
||||
await _run_sweep(db) # second pass must overwrite, not double
|
||||
|
||||
dev = (await _rows(db, seeded["dev"]))[0]
|
||||
assert dev.active_runtime_seconds == _ACTIVE_SECONDS # not 1200
|
||||
assert dev.tasks_completed == _COMPLETED_TASKS # not 4
|
||||
assert dev.escalations == 1
|
||||
assert len(await _rows(db, seeded["dev"])) == 1
|
||||
@@ -0,0 +1,118 @@
|
||||
"""get_ceo_scorecard — the human CEO as a measured member (audit-log only).
|
||||
|
||||
Seeds CEO-attributed audit transitions and asserts approval dwell (incl. the
|
||||
coordination-root reject that lands in `pending`), unblock dwell, and the
|
||||
god-mode action count. The CEO never runs an LLM, so this reads only audit_log
|
||||
(agent_role='ceo').
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from roboco.db.tables import AuditLogTable
|
||||
from roboco.services.metrics import MetricsService
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
_NOW = datetime.now(UTC)
|
||||
|
||||
|
||||
def _audit(
|
||||
task_id: Any,
|
||||
status: str,
|
||||
ts: datetime,
|
||||
*,
|
||||
agent_role: str | None = None,
|
||||
) -> AuditLogTable:
|
||||
return AuditLogTable(
|
||||
id=uuid4(),
|
||||
event_type=f"task.{status}",
|
||||
agent_id=None,
|
||||
target_type="task",
|
||||
target_id=task_id,
|
||||
severity="info",
|
||||
details={"to_status": status, "from_status": "prev", "agent_role": agent_role},
|
||||
timestamp=ts,
|
||||
)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def svc(db_session: AsyncSession) -> AsyncIterator[MetricsService]:
|
||||
yield MetricsService(db_session)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_empty_window_returns_zeros(svc: MetricsService) -> None:
|
||||
card = await svc.get_ceo_scorecard(days=30)
|
||||
assert card.approval_count == 0
|
||||
assert card.unblock_count == 0
|
||||
assert card.godmode_actions == 0
|
||||
assert card.approval_p50_seconds == 0.0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_approval_unblock_dwell_and_godmode(
|
||||
svc: MetricsService, db_session: AsyncSession
|
||||
) -> None:
|
||||
base = _NOW - timedelta(hours=2)
|
||||
t_approve = uuid4()
|
||||
t_reject = uuid4()
|
||||
t_block = uuid4()
|
||||
db_session.add_all(
|
||||
[
|
||||
# Approval: awaiting -> completed(ceo) after 300s.
|
||||
_audit(t_approve, "awaiting_ceo_approval", base),
|
||||
_audit(
|
||||
t_approve, "completed", base + timedelta(seconds=300), agent_role="ceo"
|
||||
),
|
||||
# Coordination-root reject: awaiting -> pending(ceo) after 120s.
|
||||
_audit(t_reject, "awaiting_ceo_approval", base),
|
||||
_audit(
|
||||
t_reject, "pending", base + timedelta(seconds=120), agent_role="ceo"
|
||||
),
|
||||
# Unblock: blocked -> in_progress(ceo) after 600s.
|
||||
_audit(t_block, "blocked", base),
|
||||
_audit(
|
||||
t_block, "in_progress", base + timedelta(seconds=600), agent_role="ceo"
|
||||
),
|
||||
]
|
||||
)
|
||||
await db_session.flush()
|
||||
|
||||
card = await svc.get_ceo_scorecard(days=30)
|
||||
# Two approval decisions (completed + coordination pending), median of 300/120.
|
||||
expected_approvals = 2
|
||||
assert card.approval_count == expected_approvals
|
||||
assert card.approval_p50_seconds == pytest.approx(210.0)
|
||||
# One unblock, 600s.
|
||||
assert card.unblock_count == 1
|
||||
assert card.unblock_p50_seconds == pytest.approx(600.0)
|
||||
# God-mode = every ceo-attributed transition: completed + pending + in_progress.
|
||||
expected_godmode = 3
|
||||
assert card.godmode_actions == expected_godmode
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_non_ceo_transitions_are_not_counted(
|
||||
svc: MetricsService, db_session: AsyncSession
|
||||
) -> None:
|
||||
tid = uuid4()
|
||||
db_session.add_all(
|
||||
[
|
||||
_audit(tid, "awaiting_ceo_approval", _NOW - timedelta(minutes=10)),
|
||||
# A QA fail (not the CEO) must not count as an approval or god-mode.
|
||||
_audit(tid, "needs_revision", _NOW - timedelta(minutes=5), agent_role="qa"),
|
||||
]
|
||||
)
|
||||
await db_session.flush()
|
||||
card = await svc.get_ceo_scorecard(days=30)
|
||||
assert card.approval_count == 0
|
||||
assert card.godmode_actions == 0
|
||||
@@ -0,0 +1,164 @@
|
||||
"""New audit instrumentation feeding the extra per-member metrics (phase 4).
|
||||
|
||||
- apply_escalation -> task.escalated (escalations metric)
|
||||
- _unblock_dependents -> task.unblocked_dependents (blocked-others metric)
|
||||
- mark_agent_idle -> agent.idle (idle/utilization metric)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from roboco.db.tables import AgentTable, AuditLogTable, ProjectTable, TaskTable
|
||||
from roboco.models.base import (
|
||||
AgentRole,
|
||||
AgentStatus,
|
||||
Complexity,
|
||||
TaskNature,
|
||||
TaskStatus,
|
||||
TaskType,
|
||||
Team,
|
||||
)
|
||||
from roboco.services.task import TaskService
|
||||
from sqlalchemy import select
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
|
||||
def _agent(role: AgentRole, slug: str) -> AgentTable:
|
||||
return AgentTable(
|
||||
id=uuid4(),
|
||||
name=slug,
|
||||
slug=slug,
|
||||
role=role,
|
||||
team=Team.BACKEND,
|
||||
status=AgentStatus.ACTIVE,
|
||||
model_config={},
|
||||
system_prompt="x",
|
||||
capabilities=[],
|
||||
permissions={},
|
||||
metrics={},
|
||||
)
|
||||
|
||||
|
||||
def _task(project_id: Any, created_by: Any, **over: Any) -> TaskTable:
|
||||
base: dict[str, Any] = {
|
||||
"id": uuid4(),
|
||||
"title": "t",
|
||||
"description": "d",
|
||||
"acceptance_criteria": ["ac"],
|
||||
"task_type": TaskType.CODE,
|
||||
"nature": TaskNature.TECHNICAL,
|
||||
"status": TaskStatus.IN_PROGRESS,
|
||||
"team": Team.BACKEND,
|
||||
"project_id": project_id,
|
||||
"created_by": created_by,
|
||||
"estimated_complexity": Complexity.MEDIUM,
|
||||
}
|
||||
base.update(over)
|
||||
return TaskTable(**base)
|
||||
|
||||
|
||||
async def _audit_of(db: AsyncSession, event_type: str, target_id: Any) -> list[Any]:
|
||||
rows = (
|
||||
(
|
||||
await db.execute(
|
||||
select(AuditLogTable).where(
|
||||
AuditLogTable.event_type == event_type,
|
||||
AuditLogTable.target_id == target_id,
|
||||
)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
return list(rows)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def env(db_session: AsyncSession) -> AsyncIterator[dict]:
|
||||
dev = _agent(AgentRole.DEVELOPER, f"be-dev-{uuid4().hex[:6]}")
|
||||
pm = _agent(AgentRole.CELL_PM, f"be-pm-{uuid4().hex[:6]}")
|
||||
db_session.add_all([dev, pm])
|
||||
await db_session.flush()
|
||||
project = ProjectTable(
|
||||
id=uuid4(),
|
||||
name="P",
|
||||
slug=f"p-{uuid4().hex[:6]}",
|
||||
git_url="https://example.com/r.git",
|
||||
assigned_cell=Team.BACKEND,
|
||||
created_by=dev.id,
|
||||
)
|
||||
db_session.add(project)
|
||||
await db_session.flush()
|
||||
yield {
|
||||
"svc": TaskService(db_session),
|
||||
"db": db_session,
|
||||
"project_id": project.id,
|
||||
"dev": dev,
|
||||
"pm": pm,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_apply_escalation_emits_task_escalated(env: dict) -> None:
|
||||
db = env["db"]
|
||||
task = _task(
|
||||
env["project_id"],
|
||||
env["dev"].id,
|
||||
assigned_to=env["dev"].id,
|
||||
claimed_by=env["dev"].id,
|
||||
)
|
||||
db.add(task)
|
||||
await db.flush()
|
||||
ok = await env["svc"].apply_escalation(
|
||||
task=task,
|
||||
target_agent_id=env["pm"].id,
|
||||
escalator_slug=env["dev"].slug,
|
||||
target_slug=env["pm"].slug,
|
||||
reason="need help with the seam",
|
||||
)
|
||||
assert ok is True
|
||||
rows = await _audit_of(db, "task.escalated", task.id)
|
||||
assert len(rows) == 1
|
||||
assert rows[0].details["escalator_slug"] == env["dev"].slug
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unblock_dependents_emits_count(env: dict) -> None:
|
||||
db = env["db"]
|
||||
blocker = _task(env["project_id"], env["dev"].id, status=TaskStatus.COMPLETED)
|
||||
db.add(blocker)
|
||||
await db.flush()
|
||||
dependent = _task(
|
||||
env["project_id"],
|
||||
env["dev"].id,
|
||||
status=TaskStatus.BLOCKED,
|
||||
dependency_ids=[blocker.id],
|
||||
assigned_to=env["dev"].id,
|
||||
claimed_by=env["dev"].id,
|
||||
)
|
||||
db.add(dependent)
|
||||
await db.flush()
|
||||
await env["svc"]._unblock_dependents(blocker.id)
|
||||
rows = await _audit_of(db, "task.unblocked_dependents", blocker.id)
|
||||
assert len(rows) == 1
|
||||
assert rows[0].details["count"] == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mark_agent_idle_emits_agent_idle(env: dict) -> None:
|
||||
db = env["db"]
|
||||
await env["svc"].mark_agent_idle(env["dev"].id)
|
||||
rows = await _audit_of(db, "agent.idle", env["dev"].id)
|
||||
assert len(rows) == 1
|
||||
assert rows[0].details["agent_slug"] == env["dev"].slug
|
||||
refreshed = await db.get(AgentTable, env["dev"].id)
|
||||
assert refreshed is not None
|
||||
assert refreshed.status == AgentStatus.IDLE
|
||||
@@ -0,0 +1,316 @@
|
||||
"""Member / org rollup scorecards + the live in-flight overlay (real PG).
|
||||
|
||||
Seeds member_performance_daily rows (the rollup source) and asserts the derived
|
||||
rates (FPY, effort-throughput, turns/task, qa pass-rate, utilization), the live
|
||||
in-flight overlay (enriches effort but not completion counts — disjoint by
|
||||
status), and the division guards.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import TYPE_CHECKING, Any, cast
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from roboco.db.tables import (
|
||||
AgentSpawnSessionTable,
|
||||
AgentTable,
|
||||
MemberPerformanceDailyTable,
|
||||
ProjectTable,
|
||||
TaskTable,
|
||||
)
|
||||
from roboco.models.base import (
|
||||
AgentRole,
|
||||
AgentStatus,
|
||||
Complexity,
|
||||
TaskNature,
|
||||
TaskStatus,
|
||||
TaskType,
|
||||
Team,
|
||||
)
|
||||
from roboco.services.metrics import MetricsService
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import AsyncIterator
|
||||
from uuid import UUID
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
_TODAY = datetime.now(UTC).date()
|
||||
_TOTAL_COMPLETED = 3
|
||||
_ROLLUP_COMPLETED = 2
|
||||
_OVERLAY_TURNS = 3
|
||||
_ROLLUP_ONLY_TURNS = 5 # closed session already in the rollup, not re-added
|
||||
_ORG_MEMBERS = 2
|
||||
_ORG_COMPLETED = 3
|
||||
|
||||
|
||||
def _daily(slug: str, **over: Any) -> MemberPerformanceDailyTable:
|
||||
base: dict[str, Any] = {
|
||||
"id": uuid4(),
|
||||
"date": _TODAY,
|
||||
"member_kind": "agent",
|
||||
"agent_slug": slug,
|
||||
"team": Team.BACKEND.value,
|
||||
"role": "developer",
|
||||
}
|
||||
base.update(over)
|
||||
return MemberPerformanceDailyTable(**base)
|
||||
|
||||
|
||||
def _agent(role: AgentRole, slug: str) -> AgentTable:
|
||||
return AgentTable(
|
||||
id=uuid4(),
|
||||
name=slug,
|
||||
slug=slug,
|
||||
role=role,
|
||||
team=Team.BACKEND,
|
||||
status=AgentStatus.ACTIVE,
|
||||
model_config={},
|
||||
system_prompt="x",
|
||||
capabilities=[],
|
||||
permissions={},
|
||||
metrics={},
|
||||
)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def svc(db_session: AsyncSession) -> AsyncIterator[MetricsService]:
|
||||
yield MetricsService(db_session)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_member_scorecard_rollup_and_derived(
|
||||
svc: MetricsService, db_session: AsyncSession
|
||||
) -> None:
|
||||
dev = _agent(AgentRole.DEVELOPER, f"be-dev-{uuid4().hex[:6]}")
|
||||
db_session.add(dev)
|
||||
await db_session.flush()
|
||||
db_session.add_all(
|
||||
[
|
||||
_daily(
|
||||
dev.slug,
|
||||
tasks_completed=2,
|
||||
tasks_first_pass=1,
|
||||
active_runtime_seconds=1800,
|
||||
turns=6,
|
||||
tool_calls=12,
|
||||
tokens=100,
|
||||
cost_usd=1.0,
|
||||
qa_reviews_total=3,
|
||||
qa_reviews_passed=2,
|
||||
escalations=1,
|
||||
blocked_others=1,
|
||||
idle_seconds=600,
|
||||
revisions_caused=1,
|
||||
revisions_received=1,
|
||||
),
|
||||
_daily(
|
||||
dev.slug,
|
||||
date=_TODAY - timedelta(days=1),
|
||||
tasks_completed=1,
|
||||
tasks_first_pass=1,
|
||||
active_runtime_seconds=1800,
|
||||
turns=4,
|
||||
tool_calls=8,
|
||||
tokens=50,
|
||||
cost_usd=0.5,
|
||||
qa_reviews_total=2,
|
||||
qa_reviews_passed=2,
|
||||
idle_seconds=1200,
|
||||
),
|
||||
]
|
||||
)
|
||||
await db_session.flush()
|
||||
|
||||
card = await svc.get_member_scorecard(cast("UUID", dev.id), days=30)
|
||||
assert card is not None
|
||||
assert card.tasks_completed == _TOTAL_COMPLETED
|
||||
assert card.first_pass_yield == pytest.approx(2 / 3, abs=1e-4) # 2 of 3
|
||||
# 3 tasks over 3600s = 1h -> 3.0/hr.
|
||||
assert card.effort_throughput_per_hour == pytest.approx(3.0)
|
||||
assert (card.turns, card.tool_calls) == (10, 20)
|
||||
assert card.turns_per_task == pytest.approx(10 / 3, abs=1e-4)
|
||||
assert card.qa_pass_rate == pytest.approx(4 / 5, abs=1e-4) # 4 of 5
|
||||
assert card.escalations == 1
|
||||
assert card.blocked_others == 1
|
||||
# util = 3600 active / (3600 + 1800 idle) = 0.6667.
|
||||
assert card.utilization == pytest.approx(3600 / 5400, abs=1e-4)
|
||||
assert card.includes_live_inflight is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_live_overlay_enriches_effort_not_completion(
|
||||
svc: MetricsService, db_session: AsyncSession
|
||||
) -> None:
|
||||
dev = _agent(AgentRole.DEVELOPER, f"be-dev-{uuid4().hex[:6]}")
|
||||
db_session.add(dev)
|
||||
await db_session.flush()
|
||||
project = ProjectTable(
|
||||
id=uuid4(),
|
||||
name="P",
|
||||
slug=f"p-{uuid4().hex[:6]}",
|
||||
git_url="https://example.com/r.git",
|
||||
assigned_cell=Team.BACKEND,
|
||||
created_by=dev.id,
|
||||
)
|
||||
db_session.add(project)
|
||||
await db_session.flush()
|
||||
db_session.add(_daily(dev.slug, tasks_completed=2, active_runtime_seconds=100))
|
||||
# A non-terminal (in-flight) task with a spawn stint -> overlay effort.
|
||||
inflight = TaskTable(
|
||||
id=uuid4(),
|
||||
title="t",
|
||||
description="d",
|
||||
acceptance_criteria=["ac"],
|
||||
task_type=TaskType.CODE,
|
||||
nature=TaskNature.TECHNICAL,
|
||||
status=TaskStatus.IN_PROGRESS,
|
||||
team=Team.BACKEND,
|
||||
project_id=project.id,
|
||||
created_by=dev.id,
|
||||
assigned_to=dev.id,
|
||||
estimated_complexity=Complexity.MEDIUM,
|
||||
started_at=datetime.now(UTC) - timedelta(hours=1),
|
||||
)
|
||||
db_session.add(inflight)
|
||||
await db_session.flush()
|
||||
now = datetime.now(UTC)
|
||||
# OPEN (still-running) session — the live delta the rollup cannot hold yet.
|
||||
db_session.add(
|
||||
AgentSpawnSessionTable(
|
||||
id=uuid4(),
|
||||
agent_slug=dev.slug,
|
||||
team="backend",
|
||||
role="developer",
|
||||
model="claude",
|
||||
task_id=str(inflight.id),
|
||||
started_at=now - timedelta(seconds=200),
|
||||
ended_at=None,
|
||||
turns=3,
|
||||
tool_calls=4,
|
||||
tokens_input=10,
|
||||
tokens_output=5,
|
||||
estimated_cost_usd=0.2,
|
||||
)
|
||||
)
|
||||
await db_session.flush()
|
||||
|
||||
card = await svc.get_member_scorecard(cast("UUID", dev.id), days=30)
|
||||
assert card is not None
|
||||
assert card.tasks_completed == _ROLLUP_COMPLETED # in-flight NOT completed
|
||||
assert card.includes_live_inflight is True
|
||||
assert card.active_runtime_hours > 100 / 3600 # rollup + overlay effort
|
||||
assert card.turns == _OVERLAY_TURNS # from the overlay (rollup row had 0)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_live_overlay_excludes_closed_session_no_double_count(
|
||||
svc: MetricsService, db_session: AsyncSession
|
||||
) -> None:
|
||||
"""A CLOSED session on a non-terminal task is already in the daily rollup
|
||||
(via _msweep_spawn, which counts ended_at IS NOT NULL). The overlay must NOT
|
||||
re-add it, or the member's effort/turns double-count on the common
|
||||
reap/respawn path."""
|
||||
dev = _agent(AgentRole.DEVELOPER, f"be-dev-{uuid4().hex[:6]}")
|
||||
db_session.add(dev)
|
||||
await db_session.flush()
|
||||
project = ProjectTable(
|
||||
id=uuid4(),
|
||||
name="P",
|
||||
slug=f"p-{uuid4().hex[:6]}",
|
||||
git_url="https://example.com/r.git",
|
||||
assigned_cell=Team.BACKEND,
|
||||
created_by=dev.id,
|
||||
)
|
||||
db_session.add(project)
|
||||
await db_session.flush()
|
||||
# Rollup row already reflects the closed session (turns=5, active=300s).
|
||||
db_session.add(
|
||||
_daily(dev.slug, tasks_completed=0, active_runtime_seconds=300, turns=5)
|
||||
)
|
||||
inflight = TaskTable(
|
||||
id=uuid4(),
|
||||
title="t",
|
||||
description="d",
|
||||
acceptance_criteria=["ac"],
|
||||
task_type=TaskType.CODE,
|
||||
nature=TaskNature.TECHNICAL,
|
||||
status=TaskStatus.IN_PROGRESS,
|
||||
team=Team.BACKEND,
|
||||
project_id=project.id,
|
||||
created_by=dev.id,
|
||||
assigned_to=dev.id,
|
||||
estimated_complexity=Complexity.MEDIUM,
|
||||
started_at=datetime.now(UTC) - timedelta(hours=1),
|
||||
)
|
||||
db_session.add(inflight)
|
||||
await db_session.flush()
|
||||
now = datetime.now(UTC)
|
||||
db_session.add(
|
||||
AgentSpawnSessionTable(
|
||||
id=uuid4(),
|
||||
agent_slug=dev.slug,
|
||||
team="backend",
|
||||
role="developer",
|
||||
model="claude",
|
||||
task_id=str(inflight.id),
|
||||
started_at=now - timedelta(seconds=300),
|
||||
ended_at=now, # CLOSED — already counted by the rollup
|
||||
turns=5,
|
||||
tool_calls=4,
|
||||
tokens_input=10,
|
||||
tokens_output=5,
|
||||
estimated_cost_usd=0.2,
|
||||
)
|
||||
)
|
||||
await db_session.flush()
|
||||
|
||||
card = await svc.get_member_scorecard(cast("UUID", dev.id), days=30)
|
||||
assert card is not None
|
||||
assert card.turns == _ROLLUP_ONLY_TURNS # closed session is NOT re-added
|
||||
assert card.active_runtime_hours == pytest.approx(300 / 3600, abs=1e-4)
|
||||
assert card.includes_live_inflight is False # no OPEN session
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_member_scorecard_404_and_guards(
|
||||
svc: MetricsService, db_session: AsyncSession
|
||||
) -> None:
|
||||
assert await svc.get_member_scorecard(uuid4()) is None
|
||||
# An agent with no rollup rows: division guards -> None, no crash.
|
||||
dev = _agent(AgentRole.DEVELOPER, f"be-dev-{uuid4().hex[:6]}")
|
||||
db_session.add(dev)
|
||||
await db_session.flush()
|
||||
card = await svc.get_member_scorecard(cast("UUID", dev.id), days=30)
|
||||
assert card is not None
|
||||
assert card.tasks_completed == 0
|
||||
assert card.first_pass_yield is None
|
||||
assert card.effort_throughput_per_hour is None
|
||||
assert card.qa_pass_rate is None
|
||||
assert card.utilization is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_org_scorecard_aggregates_members(
|
||||
svc: MetricsService, db_session: AsyncSession
|
||||
) -> None:
|
||||
s1, s2 = f"be-dev-{uuid4().hex[:6]}", f"be-dev-{uuid4().hex[:6]}"
|
||||
db_session.add_all(
|
||||
[
|
||||
_daily(
|
||||
s1, tasks_completed=2, tasks_first_pass=2, active_runtime_seconds=3600
|
||||
),
|
||||
_daily(
|
||||
s2, tasks_completed=1, tasks_first_pass=0, active_runtime_seconds=3600
|
||||
),
|
||||
]
|
||||
)
|
||||
await db_session.flush()
|
||||
org = await svc.get_org_scorecard(team=Team.BACKEND, days=30)
|
||||
assert org.scope == "team"
|
||||
assert org.member_count == _ORG_MEMBERS
|
||||
assert org.tasks_completed == _ORG_COMPLETED
|
||||
assert org.first_pass_yield == pytest.approx(2 / 3, abs=1e-4)
|
||||
@@ -0,0 +1,244 @@
|
||||
"""get_task_metrics — granular per-task effort against a real Postgres.
|
||||
|
||||
Seeds a task's audit-log journey + agent spawn stints (with turns/tool_calls/
|
||||
tokens/cost) + named qa/pr fail events, then asserts the composed metrics:
|
||||
summed effort vs wall-clock, turns/tool_calls/tokens/cost, per-stage
|
||||
active-vs-wait, and who-caused-rework.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import TYPE_CHECKING, Any, NamedTuple
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from roboco.db.tables import (
|
||||
AgentSpawnSessionTable,
|
||||
AgentTable,
|
||||
AuditLogTable,
|
||||
ProjectTable,
|
||||
TaskTable,
|
||||
)
|
||||
from roboco.models.base import (
|
||||
AgentRole,
|
||||
AgentStatus,
|
||||
Complexity,
|
||||
TaskNature,
|
||||
TaskStatus,
|
||||
TaskType,
|
||||
Team,
|
||||
)
|
||||
from roboco.services.metrics import MetricsService
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
_T0 = datetime(2026, 6, 20, 12, 0, 0, tzinfo=UTC)
|
||||
|
||||
|
||||
def _sec(n: int) -> datetime:
|
||||
return _T0 + timedelta(seconds=n)
|
||||
|
||||
|
||||
def _agent(role: AgentRole, slug: str) -> AgentTable:
|
||||
return AgentTable(
|
||||
id=uuid4(),
|
||||
name=slug,
|
||||
slug=slug,
|
||||
role=role,
|
||||
team=Team.BACKEND,
|
||||
status=AgentStatus.ACTIVE,
|
||||
model_config={},
|
||||
system_prompt="x",
|
||||
capabilities=[],
|
||||
permissions={},
|
||||
metrics={},
|
||||
)
|
||||
|
||||
|
||||
def _audit(
|
||||
task_id: Any,
|
||||
status: str,
|
||||
ts: datetime,
|
||||
*,
|
||||
agent_id: Any = None,
|
||||
event_type: str | None = None,
|
||||
) -> AuditLogTable:
|
||||
return AuditLogTable(
|
||||
id=uuid4(),
|
||||
event_type=event_type or f"task.{status}",
|
||||
agent_id=agent_id,
|
||||
target_type="task",
|
||||
target_id=task_id,
|
||||
severity="info",
|
||||
details={"to_status": status, "from_status": "prev", "team": "backend"},
|
||||
timestamp=ts,
|
||||
)
|
||||
|
||||
|
||||
class _Usage(NamedTuple):
|
||||
turns: int
|
||||
tool_calls: int
|
||||
tokens_in: int
|
||||
tokens_out: int
|
||||
cost: float
|
||||
|
||||
|
||||
def _spawn(
|
||||
task_id: str,
|
||||
started: datetime,
|
||||
ended: datetime | None,
|
||||
usage: _Usage,
|
||||
) -> AgentSpawnSessionTable:
|
||||
return AgentSpawnSessionTable(
|
||||
id=uuid4(),
|
||||
agent_slug="be-dev-1",
|
||||
team="backend",
|
||||
role="developer",
|
||||
model="claude",
|
||||
task_id=task_id,
|
||||
started_at=started,
|
||||
ended_at=ended,
|
||||
turns=usage.turns,
|
||||
tool_calls=usage.tool_calls,
|
||||
tokens_input=usage.tokens_in,
|
||||
tokens_output=usage.tokens_out,
|
||||
estimated_cost_usd=usage.cost,
|
||||
)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def setup(db_session: AsyncSession) -> AsyncIterator[dict]:
|
||||
dev = _agent(AgentRole.DEVELOPER, f"be-dev-{uuid4().hex[:6]}")
|
||||
qa = _agent(AgentRole.QA, f"be-qa-{uuid4().hex[:6]}")
|
||||
db_session.add_all([dev, qa])
|
||||
await db_session.flush()
|
||||
project = ProjectTable(
|
||||
id=uuid4(),
|
||||
name="P",
|
||||
slug=f"p-{uuid4().hex[:6]}",
|
||||
git_url="https://example.com/r.git",
|
||||
assigned_cell=Team.BACKEND,
|
||||
created_by=dev.id,
|
||||
)
|
||||
db_session.add(project)
|
||||
await db_session.flush()
|
||||
yield {
|
||||
"svc": MetricsService(db_session),
|
||||
"db": db_session,
|
||||
"project_id": project.id,
|
||||
"dev_id": dev.id,
|
||||
"qa_id": qa.id,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_returns_none_for_missing_task(setup: dict) -> None:
|
||||
assert await setup["svc"].get_task_metrics(uuid4()) is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_composes_effort_turns_stages_and_rework(setup: dict) -> None:
|
||||
db = setup["db"]
|
||||
tid = uuid4()
|
||||
db.add(
|
||||
TaskTable(
|
||||
id=tid,
|
||||
title="t",
|
||||
description="d",
|
||||
acceptance_criteria=["ac"],
|
||||
task_type=TaskType.CODE,
|
||||
nature=TaskNature.TECHNICAL,
|
||||
status=TaskStatus.COMPLETED,
|
||||
team=Team.BACKEND,
|
||||
project_id=setup["project_id"],
|
||||
created_by=setup["dev_id"],
|
||||
assigned_to=setup["dev_id"],
|
||||
revision_count=2,
|
||||
estimated_complexity=Complexity.MEDIUM,
|
||||
started_at=_T0,
|
||||
completed_at=_sec(7200),
|
||||
)
|
||||
)
|
||||
db.add_all(
|
||||
[
|
||||
_audit(tid, "claimed", _T0),
|
||||
_audit(tid, "in_progress", _sec(60)),
|
||||
_audit(tid, "awaiting_qa", _sec(3660)),
|
||||
_audit(tid, "completed", _sec(7200)),
|
||||
_audit(tid, "needs_revision", _sec(3660), event_type="task.qa_fail"),
|
||||
_audit(tid, "needs_revision", _sec(3000), event_type="task.pr_fail"),
|
||||
]
|
||||
)
|
||||
db.add_all(
|
||||
[
|
||||
_spawn(str(tid), _T0, _sec(600), _Usage(5, 10, 100, 50, 1.0)),
|
||||
_spawn(str(tid), _sec(3600), _sec(3660), _Usage(3, 4, 20, 10, 0.5)),
|
||||
]
|
||||
)
|
||||
await db.flush()
|
||||
|
||||
m = await setup["svc"].get_task_metrics(tid)
|
||||
assert m is not None
|
||||
# summed effort (600 + 60) vs wall-clock (2h).
|
||||
expected_active_s = 660
|
||||
expected_wall_s = 7200
|
||||
assert m.active_runtime_seconds == expected_active_s
|
||||
assert m.wall_clock_seconds == expected_wall_s
|
||||
assert (m.turns, m.tool_calls, m.tokens) == (8, 14, 180)
|
||||
assert m.cost_usd == pytest.approx(1.5)
|
||||
assert (m.revision_count, m.qa_fails, m.pr_fails, m.stints) == (2, 1, 1, 2)
|
||||
|
||||
stages = {s.status: s for s in m.stages}
|
||||
# claimed [0,60): stint1 covers it fully.
|
||||
assert (stages["claimed"].active_seconds, stages["claimed"].wait_seconds) == (60, 0)
|
||||
# in_progress [60,3660): stint1 60..600 (540) + stint2 3600..3660 (60) = 600 active.
|
||||
assert (
|
||||
stages["in_progress"].active_seconds,
|
||||
stages["in_progress"].wait_seconds,
|
||||
) == (600, 3000)
|
||||
# awaiting_qa [3660,7200): no stint running -> all wait.
|
||||
assert (
|
||||
stages["awaiting_qa"].active_seconds,
|
||||
stages["awaiting_qa"].wait_seconds,
|
||||
) == (0, 3540)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_in_flight_open_stint_and_open_window_decompose(setup: dict) -> None:
|
||||
db = setup["db"]
|
||||
tid = uuid4()
|
||||
db.add(
|
||||
TaskTable(
|
||||
id=tid,
|
||||
title="t",
|
||||
description="d",
|
||||
acceptance_criteria=["ac"],
|
||||
task_type=TaskType.CODE,
|
||||
nature=TaskNature.TECHNICAL,
|
||||
status=TaskStatus.IN_PROGRESS,
|
||||
team=Team.BACKEND,
|
||||
project_id=setup["project_id"],
|
||||
created_by=setup["dev_id"],
|
||||
assigned_to=setup["dev_id"],
|
||||
estimated_complexity=Complexity.MEDIUM,
|
||||
started_at=_T0,
|
||||
completed_at=None,
|
||||
)
|
||||
)
|
||||
db.add_all([_audit(tid, "claimed", _T0), _audit(tid, "in_progress", _sec(60))])
|
||||
# An OPEN stint (ended_at=None) -> runs to now.
|
||||
db.add(_spawn(str(tid), _T0, None, _Usage(2, 3, 1, 1, 0.1)))
|
||||
await db.flush()
|
||||
|
||||
m = await setup["svc"].get_task_metrics(tid)
|
||||
assert m is not None
|
||||
assert m.stints == 1
|
||||
assert m.active_runtime_seconds > 0 # open stint ran to now
|
||||
assert m.wall_clock_seconds > 0 # open task -> now
|
||||
# The open final window (in_progress) still decomposes.
|
||||
assert "in_progress" in {s.status for s in m.stages}
|
||||
@@ -15,6 +15,7 @@ import pytest
|
||||
import pytest_asyncio
|
||||
from roboco.db.tables import (
|
||||
AgentTable,
|
||||
AuditLogTable,
|
||||
JournalEntryTable,
|
||||
ProductTable,
|
||||
ProjectTable,
|
||||
@@ -1199,6 +1200,29 @@ async def test_ceo_reject_routes_coordination_task_to_main_pm(
|
||||
assert rejected.assigned_to == main_pm_id
|
||||
assert rejected.claimed_by is None
|
||||
|
||||
# Regression (metrics-granularity Phase 3): the coordination-root reject
|
||||
# routes through admin_set_status, so it MUST still emit a CEO-attributed
|
||||
# audit row transitioning OUT of awaiting_ceo_approval — the signal the CEO
|
||||
# scorecard pairs for approval latency + counts as a god-mode action. If a
|
||||
# future refactor drops the audit (the old gap), this fails.
|
||||
audit_rows = (
|
||||
(
|
||||
await db_session.execute(
|
||||
select(AuditLogTable).where(AuditLogTable.target_id == task.id)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
ceo_rows = [
|
||||
a
|
||||
for a in audit_rows
|
||||
if (a.details or {}).get("agent_role") == "ceo"
|
||||
and (a.details or {}).get("from_status") == "awaiting_ceo_approval"
|
||||
]
|
||||
assert ceo_rows, "coordination ceo_reject must emit a ceo-attributed audit row"
|
||||
assert ceo_rows[0].details.get("to_status") == "pending"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ceo_reject_routes_batch_umbrella_to_main_pm(
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
"""sum_transcript_usage — token + turn counts from a Claude Code JSONL transcript.
|
||||
|
||||
The 5th return value is the LLM turn count: the number of UNIQUE assistant
|
||||
``message.id``s (Claude Code logs one line per content block, all sharing the
|
||||
message id, so naive line-counting would inflate both tokens and turns).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from roboco.agent_sdk.transcript_usage import sum_transcript_usage
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
_EXPECTED_TUPLE_LEN = 5
|
||||
|
||||
|
||||
def _line(msg_id: str | None, **usage: int) -> str:
|
||||
msg: dict[str, object] = {"usage": usage}
|
||||
if msg_id is not None:
|
||||
msg["id"] = msg_id
|
||||
return json.dumps({"message": msg})
|
||||
|
||||
|
||||
def _write(path: Path, lines: list[str]) -> None:
|
||||
path.write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def test_returns_five_tuple(tmp_path: Path) -> None:
|
||||
f = tmp_path / "t.jsonl"
|
||||
_write(f, [_line("m1", input_tokens=10, output_tokens=5)])
|
||||
result = sum_transcript_usage(f)
|
||||
assert len(result) == _EXPECTED_TUPLE_LEN
|
||||
|
||||
|
||||
def test_turns_counts_unique_message_ids(tmp_path: Path) -> None:
|
||||
f = tmp_path / "t.jsonl"
|
||||
_write(
|
||||
f,
|
||||
[
|
||||
_line("m1", input_tokens=10, output_tokens=5),
|
||||
_line("m2", input_tokens=20, output_tokens=7),
|
||||
_line("m3", input_tokens=1, output_tokens=1),
|
||||
],
|
||||
)
|
||||
_in, _out, _cr, _cw, turns = sum_transcript_usage(f)
|
||||
expected_turns = 3
|
||||
assert turns == expected_turns
|
||||
|
||||
|
||||
def test_repeated_message_id_counts_one_turn_and_one_usage(tmp_path: Path) -> None:
|
||||
# Claude Code emits one line per content block of the SAME assistant message,
|
||||
# each repeating the usage — must count once for tokens AND turns.
|
||||
f = tmp_path / "t.jsonl"
|
||||
_write(
|
||||
f,
|
||||
[
|
||||
_line("m1", input_tokens=10, output_tokens=5),
|
||||
_line("m1", input_tokens=10, output_tokens=5),
|
||||
_line("m1", input_tokens=10, output_tokens=5),
|
||||
],
|
||||
)
|
||||
tin, tout, _cr, _cw, turns = sum_transcript_usage(f)
|
||||
assert (tin, tout, turns) == (10, 5, 1)
|
||||
|
||||
|
||||
def test_malformed_lines_skipped_without_losing_turn_count(tmp_path: Path) -> None:
|
||||
f = tmp_path / "t.jsonl"
|
||||
_write(
|
||||
f,
|
||||
[
|
||||
_line("m1", input_tokens=10, output_tokens=5),
|
||||
"not json at all {{{",
|
||||
"",
|
||||
_line("m2", input_tokens=2, output_tokens=2),
|
||||
],
|
||||
)
|
||||
tin, _out, _cr, _cw, turns = sum_transcript_usage(f)
|
||||
assert (tin, turns) == (12, 2)
|
||||
|
||||
|
||||
def test_usage_line_without_id_sums_tokens_but_not_a_turn(tmp_path: Path) -> None:
|
||||
f = tmp_path / "t.jsonl"
|
||||
_write(
|
||||
f,
|
||||
[
|
||||
_line(None, input_tokens=4, output_tokens=1),
|
||||
_line("m1", input_tokens=6, output_tokens=1),
|
||||
],
|
||||
)
|
||||
tin, _out, _cr, _cw, turns = sum_transcript_usage(f)
|
||||
assert (tin, turns) == (10, 1)
|
||||
@@ -139,7 +139,11 @@ def test_missing_transcript_returns_zero_without_error(
|
||||
"/usage/sync", json={"transcript_path": str(tmp_path / "nope.jsonl")}
|
||||
)
|
||||
assert resp.status_code == _OK
|
||||
assert resp.json() == _expected([])
|
||||
body = resp.json()
|
||||
for key, value in _expected([]).items():
|
||||
assert body[key] == value
|
||||
assert body["turns"] == 0
|
||||
assert body["tool_calls"] == 0
|
||||
|
||||
|
||||
def test_malformed_lines_are_skipped(client: TestClient, tmp_path: Path) -> None:
|
||||
@@ -167,7 +171,7 @@ def test_parser_handles_entries_without_message(tmp_path: Path) -> None:
|
||||
json.dumps({"type": "system", "subtype": "init"}),
|
||||
_assistant_line(rows[0]),
|
||||
)
|
||||
tin, tout, cread, cwrite = srv._sum_transcript_usage(transcript)
|
||||
tin, tout, cread, cwrite, turns = srv._sum_transcript_usage(transcript)
|
||||
exp = _expected(rows)
|
||||
assert (tin, tout, cread, cwrite) == (
|
||||
exp["tokens_input"],
|
||||
@@ -175,6 +179,7 @@ def test_parser_handles_entries_without_message(tmp_path: Path) -> None:
|
||||
exp["tokens_cache_read"],
|
||||
exp["tokens_cache_write"],
|
||||
)
|
||||
assert turns == 0 # _assistant_line carries no message id
|
||||
|
||||
|
||||
def _assistant_line_with_id(row: _UsageRow, message_id: str) -> str:
|
||||
@@ -214,7 +219,7 @@ def test_parser_dedupes_repeated_message_id(tmp_path: Path) -> None:
|
||||
_assistant_line_with_id(msg, "msg_aaa"), # tool_use block (same id)
|
||||
_assistant_line_with_id(other, "msg_bbb"),
|
||||
)
|
||||
tin, tout, cread, cwrite = srv._sum_transcript_usage(transcript)
|
||||
tin, tout, cread, cwrite, turns = srv._sum_transcript_usage(transcript)
|
||||
# Counted once per id: msg + other, NOT msg * 3 + other.
|
||||
exp = _expected([msg, other])
|
||||
assert (tin, tout, cread, cwrite) == (
|
||||
@@ -223,3 +228,20 @@ def test_parser_dedupes_repeated_message_id(tmp_path: Path) -> None:
|
||||
exp["tokens_cache_read"],
|
||||
exp["tokens_cache_write"],
|
||||
)
|
||||
expected_turns = 2 # two unique message ids
|
||||
assert turns == expected_turns
|
||||
|
||||
|
||||
def test_sync_response_surfaces_turns(client: TestClient, tmp_path: Path) -> None:
|
||||
"""/usage/sync (and thus /usage/status) reports the LLM turn count."""
|
||||
transcript = tmp_path / "session.jsonl"
|
||||
_write(
|
||||
transcript,
|
||||
_assistant_line_with_id((10, 5, 0, 0), "msg_a"),
|
||||
_assistant_line_with_id((10, 5, 0, 0), "msg_a"), # same id
|
||||
_assistant_line_with_id((2, 1, 0, 0), "msg_b"),
|
||||
)
|
||||
body = client.post("/usage/sync", json={"transcript_path": str(transcript)}).json()
|
||||
expected_turns = 2
|
||||
assert body["turns"] == expected_turns
|
||||
assert "tool_calls" in body
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
"""The member_performance_daily rollup table — schema shape."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, cast
|
||||
|
||||
from roboco.db.tables import MemberPerformanceDailyTable
|
||||
from sqlalchemy import UniqueConstraint
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from sqlalchemy import Table
|
||||
|
||||
_TABLE = cast("Table", MemberPerformanceDailyTable.__table__)
|
||||
|
||||
|
||||
def test_table_name() -> None:
|
||||
assert MemberPerformanceDailyTable.__tablename__ == "member_performance_daily"
|
||||
|
||||
|
||||
def test_has_all_metric_columns_including_extras() -> None:
|
||||
cols = set(_TABLE.columns.keys())
|
||||
assert {
|
||||
# core
|
||||
"date",
|
||||
"member_kind",
|
||||
"agent_slug",
|
||||
"team",
|
||||
"role",
|
||||
"tasks_completed",
|
||||
"tasks_first_pass",
|
||||
"revisions_caused",
|
||||
"revisions_received",
|
||||
"active_runtime_seconds",
|
||||
"turns",
|
||||
"tool_calls",
|
||||
"tokens",
|
||||
"cost_usd",
|
||||
"ceo_approval_dwell_seconds",
|
||||
"ceo_unblock_dwell_seconds",
|
||||
"godmode_actions",
|
||||
# the 4 CEO-approved extras + blocked_seconds
|
||||
"qa_reviews_total",
|
||||
"qa_reviews_passed",
|
||||
"escalations",
|
||||
"blocked_others",
|
||||
"idle_seconds",
|
||||
"blocked_seconds",
|
||||
} <= cols
|
||||
|
||||
|
||||
def test_natural_key_is_unique() -> None:
|
||||
uniques = [c for c in _TABLE.constraints if isinstance(c, UniqueConstraint)]
|
||||
key_sets = [{col.name for col in u.columns} for u in uniques]
|
||||
assert {"date", "member_kind", "agent_slug"} in key_sets
|
||||
|
||||
|
||||
def test_agent_slug_not_nullable() -> None:
|
||||
# NOT NULL DEFAULT '' — else the CEO row (agent_slug NULL) would duplicate
|
||||
# under Postgres' NULL-distinct UNIQUE semantics.
|
||||
assert _TABLE.columns["agent_slug"].nullable is False
|
||||
@@ -267,6 +267,43 @@ async def test_undecodable_message_is_acked_and_dead_lettered() -> None:
|
||||
assert invoked == []
|
||||
|
||||
|
||||
class _FakeRecoverRedis:
|
||||
"""Fake whose xpending_range returns the message id as BYTES (the real
|
||||
client has no decode_responses), and which captures the ids XCLAIM gets."""
|
||||
|
||||
def __init__(self, message_id: bytes) -> None:
|
||||
self._message_id = message_id
|
||||
self.claimed_ids: list[object] = []
|
||||
|
||||
async def xpending(self, *args: object, **kwargs: object) -> dict:
|
||||
del args, kwargs
|
||||
return {"pending": 1}
|
||||
|
||||
async def xpending_range(self, *args: object, **kwargs: object) -> list:
|
||||
del args, kwargs
|
||||
return [{"message_id": self._message_id, "time_since_delivered": 10_000}]
|
||||
|
||||
async def xclaim(self, *args: object, **kwargs: object) -> list:
|
||||
del args
|
||||
self.claimed_ids = cast("list[object]", kwargs.get("message_ids") or [])
|
||||
return [] # nothing claimed back → no handling
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_recover_stream_decodes_bytes_message_id_for_xclaim() -> None:
|
||||
"""xpending_range returns the message id as bytes; _recover_stream must
|
||||
decode it before XCLAIM. A raw ``str(bytes)`` yields ``"b'1782..-0'"``,
|
||||
which Redis rejects with "Unrecognized XCLAIM option", so pending-message
|
||||
recovery silently fails every reclaim tick."""
|
||||
bus = StreamEventBus()
|
||||
fake = _FakeRecoverRedis(b"1782066556728-0")
|
||||
bus._redis = cast("Redis", fake)
|
||||
|
||||
await bus._recover_stream("roboco:stream:usage", idle_time_ms=0)
|
||||
|
||||
assert fake.claimed_ids == ["1782066556728-0"] # decoded, not "b'...'"
|
||||
|
||||
|
||||
# --- periodic reclaim: a runtime handler failure is retried without a restart ---
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
"""compute_stage_effort — split each stage window into active vs wait seconds.
|
||||
|
||||
Pure overlap math (no DB): given a stage's [start, end) window and the agent
|
||||
spawn stints that ran during the task, ``active`` is the wall-clock time during
|
||||
which AT LEAST ONE stint was running (overlapping stints merged, so active can
|
||||
never exceed the window), and ``wait`` is the remainder. This is the wall-clock
|
||||
decomposition — distinct from summed effort (Σ stint durations), which can
|
||||
exceed wall-clock when stints run concurrently.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
from roboco.foundation.policy.stage_effort import StageEffort, compute_stage_effort
|
||||
|
||||
_BASE = datetime(2026, 7, 1, 12, 0, 0, tzinfo=UTC)
|
||||
|
||||
|
||||
def _at(seconds: int) -> datetime:
|
||||
return _BASE + timedelta(seconds=seconds)
|
||||
|
||||
|
||||
def _window(status: str, start_s: int, end_s: int) -> tuple[str, datetime, datetime]:
|
||||
return (status, _at(start_s), _at(end_s))
|
||||
|
||||
|
||||
def _stint(start_s: int, end_s: int) -> tuple[datetime, datetime]:
|
||||
return (_at(start_s), _at(end_s))
|
||||
|
||||
|
||||
def _only(windows: list, stints: list) -> StageEffort:
|
||||
result = compute_stage_effort(windows, stints)
|
||||
assert len(result) == 1
|
||||
return result[0]
|
||||
|
||||
|
||||
def test_disjoint_stint_is_all_wait() -> None:
|
||||
eff = _only([_window("in_progress", 0, 100)], [_stint(200, 300)])
|
||||
assert (eff.active_seconds, eff.wait_seconds) == (0, 100)
|
||||
|
||||
|
||||
def test_fully_nested_stint() -> None:
|
||||
eff = _only([_window("in_progress", 0, 100)], [_stint(20, 50)])
|
||||
assert (eff.active_seconds, eff.wait_seconds) == (30, 70)
|
||||
|
||||
|
||||
def test_partial_overlap_clips_to_window() -> None:
|
||||
# stint runs 80..150 but window ends at 100 -> only 20s active in-window.
|
||||
eff = _only([_window("in_progress", 0, 100)], [_stint(80, 150)])
|
||||
assert (eff.active_seconds, eff.wait_seconds) == (20, 80)
|
||||
|
||||
|
||||
def test_multiple_nonoverlapping_stints_sum() -> None:
|
||||
eff = _only(
|
||||
[_window("in_progress", 0, 100)],
|
||||
[_stint(0, 10), _stint(40, 60)],
|
||||
)
|
||||
assert (eff.active_seconds, eff.wait_seconds) == (30, 70)
|
||||
|
||||
|
||||
def test_overlapping_stints_are_merged_not_double_counted() -> None:
|
||||
# [10,40) and [30,60) overlap -> merged union is [10,60) = 50s, NOT 60s.
|
||||
eff = _only(
|
||||
[_window("in_progress", 0, 100)],
|
||||
[_stint(10, 40), _stint(30, 60)],
|
||||
)
|
||||
# merged union [10,60) = 50s active, 50s wait (NOT 60s from double-count).
|
||||
assert (eff.active_seconds, eff.wait_seconds) == (50, 50)
|
||||
|
||||
|
||||
def test_active_never_exceeds_window_length() -> None:
|
||||
eff = _only(
|
||||
[_window("in_progress", 0, 100)],
|
||||
[_stint(-50, 500)], # stint dwarfs the window
|
||||
)
|
||||
assert (eff.active_seconds, eff.wait_seconds) == (100, 0)
|
||||
|
||||
|
||||
def test_zero_length_window() -> None:
|
||||
eff = _only([_window("claimed", 50, 50)], [_stint(0, 100)])
|
||||
assert (eff.active_seconds, eff.wait_seconds) == (0, 0)
|
||||
|
||||
|
||||
def test_each_window_decomposes_independently() -> None:
|
||||
windows = [_window("claimed", 0, 100), _window("in_progress", 100, 300)]
|
||||
stints = [_stint(50, 250)] # spans both windows
|
||||
result = compute_stage_effort(windows, stints)
|
||||
by_status = {e.status: e for e in result}
|
||||
claimed = by_status["claimed"]
|
||||
in_progress = by_status["in_progress"]
|
||||
assert (claimed.active_seconds, claimed.wait_seconds) == (50, 50)
|
||||
# in_progress: stint covers 100..250 of the 100..300 window.
|
||||
assert (in_progress.active_seconds, in_progress.wait_seconds) == (150, 50)
|
||||
|
||||
|
||||
def test_to_dict_shape() -> None:
|
||||
eff = _only([_window("in_progress", 0, 100)], [_stint(20, 50)])
|
||||
d = eff.to_dict()
|
||||
assert d == {"status": "in_progress", "active_seconds": 30, "wait_seconds": 70}
|
||||
@@ -269,7 +269,7 @@ async def test_finalize_spawn_session_http_error_uses_zero_tokens() -> None:
|
||||
|
||||
with (
|
||||
patch("roboco.runtime.orchestrator.httpx.AsyncClient", _client_cls),
|
||||
patch.object(orch, "_usage_from_transcript", return_value=(0, 0, 0, 0)),
|
||||
patch.object(orch, "_usage_from_transcript", return_value=(0, 0, 0, 0, 0)),
|
||||
patch("roboco.db.base.get_session_factory", return_value=db_factory),
|
||||
patch("roboco.billing.pricing.calculate_cost", return_value=0.0) as mock_cost,
|
||||
):
|
||||
@@ -303,7 +303,7 @@ async def test_finalize_spawn_session_non_200_uses_zero_tokens() -> None:
|
||||
|
||||
with (
|
||||
patch("roboco.runtime.orchestrator.httpx.AsyncClient", _client_cls),
|
||||
patch.object(orch, "_usage_from_transcript", return_value=(0, 0, 0, 0)),
|
||||
patch.object(orch, "_usage_from_transcript", return_value=(0, 0, 0, 0, 0)),
|
||||
patch("roboco.db.base.get_session_factory", return_value=db_factory),
|
||||
patch("roboco.billing.pricing.calculate_cost", return_value=0.0) as mock_cost,
|
||||
):
|
||||
@@ -389,7 +389,7 @@ async def test_sweep_token_snapshots_skips_zero_token_agents() -> None:
|
||||
|
||||
with (
|
||||
patch("roboco.runtime.orchestrator.httpx.AsyncClient", _client_cls),
|
||||
patch.object(orch, "_usage_from_transcript", return_value=(0, 0, 0, 0)),
|
||||
patch.object(orch, "_usage_from_transcript", return_value=(0, 0, 0, 0, 0)),
|
||||
patch("roboco.db.base.get_session_factory", return_value=db_factory),
|
||||
):
|
||||
await orch._sweep_token_snapshots()
|
||||
@@ -765,7 +765,9 @@ async def test_resolve_active_tokens_falls_back_to_transcript() -> None:
|
||||
)
|
||||
|
||||
client = _FakeHTTPClient(_handler)
|
||||
with patch.object(orch, "_usage_from_transcript", return_value=(6, 514, 100, 50)):
|
||||
with patch.object(
|
||||
orch, "_usage_from_transcript", return_value=(6, 514, 100, 50, 3)
|
||||
):
|
||||
tokens = await orch._resolve_active_tokens(
|
||||
cast("httpx.AsyncClient", client), _AGENT_ID
|
||||
)
|
||||
@@ -790,7 +792,7 @@ async def test_resolve_active_tokens_prefers_sdk() -> None:
|
||||
|
||||
client = _FakeHTTPClient(_handler)
|
||||
with patch.object(
|
||||
orch, "_usage_from_transcript", return_value=(999, 999, 999, 999)
|
||||
orch, "_usage_from_transcript", return_value=(999, 999, 999, 999, 0)
|
||||
) as mock_tx:
|
||||
tokens = await orch._resolve_active_tokens(
|
||||
cast("httpx.AsyncClient", client), _AGENT_ID
|
||||
@@ -800,6 +802,50 @@ async def test_resolve_active_tokens_prefers_sdk() -> None:
|
||||
mock_tx.assert_not_called()
|
||||
|
||||
|
||||
async def test_resolve_final_turns_tools_from_sdk() -> None:
|
||||
"""turns + tool_calls come from the SDK /usage/status when present."""
|
||||
orch = _make_orchestrator()
|
||||
|
||||
def _handler(_url: str) -> Any:
|
||||
return _mock_response(200, {"turns": 7, "tool_calls": 42, "tokens_input": 1})
|
||||
|
||||
with patch(
|
||||
"roboco.runtime.orchestrator.httpx.AsyncClient",
|
||||
lambda **_kw: _FakeHTTPClient(_handler),
|
||||
):
|
||||
turns, tool_calls = await orch._resolve_final_turns_tools(_AGENT_ID)
|
||||
|
||||
assert (turns, tool_calls) == (7, 42)
|
||||
|
||||
|
||||
async def test_resolve_final_turns_tools_transcript_fallback_for_turns() -> None:
|
||||
"""When the SDK reports 0 turns, fall back to the transcript turn count.
|
||||
|
||||
tool_calls has no transcript equivalent and stays 0 ("n/a").
|
||||
"""
|
||||
orch = _make_orchestrator()
|
||||
|
||||
def _handler(_url: str) -> Any:
|
||||
return _mock_response(200, {"turns": 0, "tool_calls": 0})
|
||||
|
||||
transcript_turns = 9
|
||||
with (
|
||||
patch(
|
||||
"roboco.runtime.orchestrator.httpx.AsyncClient",
|
||||
lambda **_kw: _FakeHTTPClient(_handler),
|
||||
),
|
||||
patch.object(
|
||||
orch,
|
||||
"_usage_from_transcript",
|
||||
return_value=(1, 2, 3, 4, transcript_turns),
|
||||
),
|
||||
):
|
||||
turns, tool_calls = await orch._resolve_final_turns_tools(_AGENT_ID)
|
||||
|
||||
assert turns == transcript_turns # recovered from the transcript
|
||||
assert tool_calls == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _usage_from_transcript — locate by session id across any project dir
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -836,7 +882,7 @@ def test_usage_from_transcript_finds_by_session_id_in_shared_app_dir(
|
||||
monkeypatch.setattr(Path, "home", lambda: tmp_path)
|
||||
|
||||
result = AgentOrchestrator._usage_from_transcript("main-pm", sid)
|
||||
assert result == (exp_in, exp_out, exp_cr, exp_cw)
|
||||
assert result == (exp_in, exp_out, exp_cr, exp_cw, 1) # one message => 1 turn
|
||||
|
||||
|
||||
def test_usage_from_transcript_without_session_id_uses_slug_glob(
|
||||
@@ -859,4 +905,4 @@ def test_usage_from_transcript_without_session_id_uses_slug_glob(
|
||||
monkeypatch.setattr(Path, "home", lambda: tmp_path)
|
||||
|
||||
result = AgentOrchestrator._usage_from_transcript("be-dev-1")
|
||||
assert result == (exp_in, exp_out, 0, 0)
|
||||
assert result == (exp_in, exp_out, 0, 0, 1) # one message => 1 turn
|
||||
|
||||
Reference in New Issue
Block a user