2 Commits

Include renames

Author	SHA1	Message	Date
3 people	a8cb2470ba	v0.15.0: Metrics granularity — per-member / per-task / org + CEO scorecards (#289) ... * feat(metrics): capture per-session turns + tool_calls (phase 1) Persist LLM iterations (turns) and tool invocations per agent spawn session, the raw signal the granular per-member performance metrics build on (real effort/iterations vs wall-clock). - sum_transcript_usage returns a 5-tuple adding turns = unique assistant message-id count; _usage_from_transcript + _resolve_active_tokens updated to the 5-tuple (active-tokens keeps its 4-tuple contract by slicing). - SDK: _SessionState.turns, set by /usage/sync; /usage/status (TokenUsageStatus) now carries turns + tool_calls (= total_calls). - orchestrator: new _resolve_final_turns_tools (SDK primary, transcript fallback for turns only; Grok -> 0/0) wired into _finalize_spawn_session, which writes turns + tool_calls to agent_spawn_sessions. - migration 055 adds turns + tool_calls (BigInteger DEFAULT 0 -> historical/Grok rows read 0, surfaced as n/a). Verified real alembic upgrade/downgrade. Part of metrics-granularity (v0.15.0); recon-adjusted plan on disk. * feat(metrics): pure compute_stage_effort helper (phase 2, part 1) Foundation-layer overlap math (no DB): split each task status window into active (merged wall-clock overlap of spawn stints — concurrent stints counted once, so active <= window) vs wait (queue/review idle). Distinct from summed effort. The per-task metrics service will feed it audit-log windows + spawn stints. 9 unit tests (disjoint/nested/partial/merged/clamped/zero/multi-window). * feat(metrics): per-task live metrics + GET /metrics/task/{id} (phase 2) TaskMetrics dataclass + MetricsService.get_task_metrics: summed spawn effort (vs wall-clock), turns/tool_calls/tokens/cost, per-stage active-vs-wait (compute_stage_effort over audit windows x spawn stints), and who-caused-rework (revision_count + named qa/pr fail events). Open stints and the open final stage window close at completed_at for a terminal task (else now), so stages don't grow past completion. Exposed at GET /dashboard/metrics/task/{task_id} (404 if absent). Real-PG tests (compose/none/in-flight) + route tests (200/404). * feat(metrics): CEO-as-member scorecard + ceo_reject audit regression (phase 3) The human CEO is a measured member, read purely from audit_log (agent_role='ceo' serializes from the CEO StrEnum): approval dwell (awaiting_ceo_approval -> a CEO decision, incl. the coordination-root reject that lands in pending), unblock dwell (blocked -> a CEO revive), and god-mode action count (every CEO-attributed transition). CeoScorecard + MetricsService.get_ceo_scorecard (p50/p90 via PERCENTILE_CONT, expanding IN for the decision sets) + GET /dashboard/metrics/member/ceo (declared before any future member/{id} route). The ceo_reject coordination-root audit gap the plan meant to close was already closed by the gap-sweep (routes through admin_set_status -> agent_role='ceo' audit); locked with a regression assertion in the existing coordination-reject test. Real-PG tests: approval/unblock/godmode, non-ceo exclusion, empty->zeros. * feat(metrics): audit instrumentation for escalations/blocked-others/idle (phase 4a) The three extra per-member metrics that had no data source get durable, in-session audit events (additive; never gate the underlying action): - apply_escalation -> task.escalated (details.escalator_slug) on both the normal block path and the pool-divert path -> escalations count. - _unblock_dependents -> task.unblocked_dependents (details.count) on the completed BLOCKER task, captured before the dependency edges are pruned -> blocked-others count (sweeper attributes to the blocker's owner). - mark_agent_idle -> agent.idle (details.agent_slug) -> idle/utilization (the sweeper pairs an idle mark to the member's next spawn for idle duration). (QA pass-rate needs no new event — reuses task.awaiting_documentation[qa] + task.qa_fail.) Real-PG tests for each; 111 transition tests still green. * feat(metrics): member_performance_daily rollup table + migration 056 (phase 4b) The per-member scorecard rollup: one row per (date, member_kind, agent_slug), CEO as a first-class member_kind='ceo' row (agent_slug='' NOT NULL so the NULL-distinct UNIQUE keeps it unique). Full column set + the four CEO-approved extras (qa_reviews_total/passed, escalations, blocked_others, idle_seconds) plus blocked_seconds. Overwrite-upsert on (date, member_kind, agent_slug) for an idempotent sweep. Migration 056 verified real up/down (24 cols, 4 indexes). * feat(metrics): _sweep_member_performance rollup sweeper (phase 4c) The daily per-member rollup sweep (mirrors _sweep_daily_rollup): a trailing 7-day, idempotent overwrite-upsert wired into _run_sweep. One focused query per metric merges into a (date, agent_slug) accumulator — spawn effort/turns/tokens/ cost, completed/first-pass/revisions-received, revisions-caused (qa/pr fails), QA pass-rate (passed + total), escalations (by escalator_slug), blocked-others (unblocked_dependents by blocker owner), idle_seconds (idle mark -> next spawn), blocked_seconds (blocked dwell) — plus one CEO row/day (approval/unblock dwell + god-mode). Real-PG test asserts every facet + idempotency (a 2nd sweep overwrites, never doubles); spawn-day != completion-day split is by-design. * feat(metrics): member/org rollup scorecards + endpoints + live overlay (phase 5) MemberScorecard + OrgScorecard with derived rates (FPY, effort-throughput, turns/tool-calls per task, QA pass-rate, utilization) — all division-guarded to None. get_member_scorecard reads member_performance_daily by slug and overlays the member's live in-flight (non-terminal) tasks' effort via get_task_metrics (disjoint by status: completion counts stay rollup-only, overlay only enriches effort/turns/cost; includes_live_inflight flags it). get_org_scorecard aggregates the cell (?team=) or whole org. Routes: GET /metrics/member/{agent_id} (404 if absent, after the ceo literal route) + GET /metrics/org?team=. Real-PG tests (derived rates, overlay no double-count, guards, org) + route tests. * feat(metrics): granular CEO completion notification (phase 6) There was no CEO completion notification at all (EventType.TASK_COMPLETED was defined but never emitted). Add notify_ceo_of_completion in NotificationDeliveryService — a granular body (real effort vs wall-clock + stints/turns/tool-calls/revisions[QA/PR]/cost from get_task_metrics; degrades to wall-clock-only, turns 'n/a', when there are no spawn sessions). Reuses the existing ALERT type (no enum migration; the notificationtype PG enum is fixed at 001). ceo_approve now emits TASK_COMPLETED + fires the notification (best-effort via _notify_completion — never blocks completion); complete() emits TASK_COMPLETED too (closes the dead-code gap; the WS bridge can forward it). Pure formatter tests + real-PG notification test. * [metrics-granularity] Phase 7: panel Scorecards tab + dashboard overview Add the CEO-facing metrics surfaces for the granularity feature: - New "Scorecards" tab on the Metrics page: org rollup headline, the CEO-as-member card (approval/unblock dwell + god-mode count), and a per-member table (completed, first-pass yield, active effort, turns/task, QA pass-rate, escalations, blocked-others, utilization). Each member row self-fetches its rollup scorecard; live in-flight rows carry a "live" badge. - New dashboard overview card (ScorecardOverviewPanel): org-wide 30-day headline (completed, FPY, throughput/hr, active effort, cost) deep-linking into the Scorecards tab. - Plumbing: TaskMetrics/MemberScorecard/OrgScorecard/CeoScorecard types, observability API client methods + empty fallbacks, and the four useCeoScorecard/useMemberScorecard/useOrgScorecard/useTaskMetrics hooks. Panel gate green: tsc, eslint, prettier, vitest (175 tests, +6 new). * [metrics-granularity] test: make completion-notification robust to shared-DB CEO test_notify_ceo_of_completion_creates_alert errored in the full suite (passed in isolation): the session-scoped test DB is shared across the run, and the sibling real-DB board-gate test commits a role=CEO agent (slug="ceo") without cleanup — so my env fixture's hardcoded slug="ceo" insert hit a unique-constraint violation, and a second role=CEO row would also make _get_ceo_agent()'s scalar_one_or_none() raise. Reuse an existing CEO when present (the singleton the production system actually has), else create one with a unique slug. Order- independent. Also reflow test_metrics_instrumentation.py to ruff format. * chore(release): 0.15.0 Metrics granularity: per-member/per-task/org + CEO-as-member scorecards, turn/tool-call capture (migration 055), member_performance_daily rollup (migration 056) with QA pass-rate / escalations / blocked-others / utilization, per-task active-vs-wait metrics, granular completion notification, panel Scorecards tab + dashboard Performance card, and the ceo_reject audit fix. Version bump across the canonical set + CHANGELOG. * [metrics-granularity] fix pre-tag audit findings (overlay double-count + panel error states) Adversarial review before the v0.15.0 tag surfaced two real logical gaps: - MAJOR (backend): the live in-flight overlay re-summed ALL sessions of every non-terminal task via get_task_metrics, but _msweep_spawn already rolls up every CLOSED session regardless of task status — so a closed session on a still-open task was counted twice (rollup + overlay), permanently inflating a member's effort/turns/tokens/cost on the common reap/respawn path. The overlay now sums only OPEN sessions (ended_at IS NULL), which the closed-only rollup can never contain — disjoint by construction. A just-closed session lands in the rollup on the next ~60s sweep (no gap of note). Aggregated in SQL to mirror _msweep_spawn. Regression test reproduces the double-count (turns 10→5). - MAJOR (panel): the four new scorecard surfaces used `isLoading || !data` with no isError branch, so a failed query span forever on a skeleton. They now surface a load error. Tests added. Also: OrgSummary active-effort formatting no longer round-trips hours→seconds→ hours; dashboard grid uses xl:grid-cols-4 (was 2xl) so 4 panels show at 1280px; corrected the inaccurate "NULL distinct" CEO-row uniqueness comment (agent_slug is NOT NULL; the '' tuple is simply distinct from agent rows). make quality GREEN (cov 95.31%); panel GREEN (vitest 178). * [metrics-granularity] fix: decode bytes stream message-id before XCLAIM StreamEventBus._recover_stream passed the pending message id to XCLAIM via str() on the raw bytes the client returns (redis client has no decode_responses), producing "b'1782066556728-0'". Redis rejects that with "Unrecognized XCLAIM option", so pending-message recovery threw on every reclaim tick and unacked messages from crashed/slow consumers were never reclaimed (leaking in the PEL on every stream, spamming the error log). Decode via the existing _to_str helper — the fix the sibling claim path already uses. Pre-existing in v0.14.0 (unrelated to metrics granularity); folded into this release per CEO. TDD regression test + CHANGELOG entry. make quality GREEN. --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>	2026-07-01 05:18:45 +02:00
3 people	e2f7097aab	Persist the PM-respawn counter across orchestrator restarts (#275) ... * feat(orchestrator): add respawn_tracker table + migration 051 Durable backing for AgentOrchestrator._pm_respawn_tracker (the PM-respawn loop breaker). Kept only in memory it reset to count=1 on every restart, re-burning the strike threshold against a still-wedged task. RespawnTrackerTable mirrors WaitingRecordTable: composite PK (agent_slug, task_id) matching the in-memory key; task_id is intentionally NOT a FK (the startup loader validates against live tasks so a stale counter can't resurrect). Migration 051 verified with a real alembic upgrade head + downgrade -1 + re-upgrade on Postgres. * feat(orchestrator): persist the PM-respawn counter across restarts The PM-respawn loop breaker (_pm_respawn_tracker) lived only in memory, so an orchestrator restart reset a wedged task's strike count to 1 and re-burned the whole threshold (4 spawns x container cost) before the gate fired again. Write-through each gate mutation to the respawn_tracker table via a fire-and-forget _schedule_respawn_persist (on the existing _bg_tasks strong-ref set; a DB hiccup degrades to in-memory-only, never gates/un-gates a spawn), and restore_respawn_tracker() repopulates the counter at startup, validating each row against live tasks (drops terminal/missing) so a stale counter can't resurrect against a fixed task. Best-effort + inert when the table is empty. Cannot manufacture a spawn — the counter only ever suppresses one. (_instances reconcile, the spec's other goal, already shipped as _readopt_running_agents.) * fix(types): cast Mapped[UUID] columns in project routes + self_heal A clean `mypy roboco/ tests/` run surfaces 7 pre-existing errors in files this branch doesn't touch: project-route handlers and self_heal_engine pass a ProjectTable.id (declared Mapped[UUID] against SQLAlchemy's dialect UUID, so mypy infers sqlalchemy.sql.sqltypes.UUID[Any]) where a uuid.UUID is expected. An incremental .mypy_cache had hidden them. Apply the same targeted cast unblock used for the prior batch; the deeper fix (migrating the ~88 Mapped[UUID] columns to Mapped[uuid.UUID]) remains a separate dedicated task. * docs(orchestrator): document respawn_tracker durability Add the orchestrator runtime-state durability note to CLAUDE.md (respawn_tracker write-through + restore; _instances reconciled-from-Docker) + the migration-051 narrative, and a CHANGELOG [Unreleased] Fixed entry. Also type-clean the new respawn_tracker table test (cast __table__ to Table under TYPE_CHECKING). --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>	2026-06-26 03:36:42 +02:00