2 Commits

Include renames

Author	SHA1	Message	Date
3 people	cea3e56628	feat(lifecycle): revision findings ledger — structured failure feedback, persisted and delivered down the chain (#486) ... * feat(lifecycle): revision findings ledger — structured QA/PR/PM/CEO failure feedback, persisted and delivered down the chain Every bounce used to survive only as flattened prose: rounds overwrote each other in notes_structured, request_changes persisted nothing, two raw dev_notes appends were silently destroyed by the next handoff note, and the dev prompt pointed at fields (qa_notes via evidence(), pm_notes) the API never delivered. Agents re-interpreted and re-discovered every failure before they could start fixing it. - task_review_findings (migration 071, append-only): file/line/severity/ criterion(AC-id-validated)/expected/actual/fix/evidence per finding, with origin (qa|pr_gate|pm|ceo), round, and an open->addressed->verified lifecycle (waived reserved); new tasks.pm_notes + PmReviewContent give request_changes a structured home - producers: fail_review/pr_fail/request_changes take findings=[...] (prose issues shimmed+merged for one release, deprecation-logged); ceo_reject validates its reason (no 500), lands an origin=ceo finding, and bumps round+audit on branchless coordination roots; guardrails at the verb chokepoint (nudge >5, hard reject >10, field caps, traversal-safe file); the dev_notes data-loss appends are removed; new task.request_changes + task.ceo_reject audit events close rework attribution - delivery: qa_notes/pr_reviewer_notes/pm_notes carry the deterministic [F-id8] rendering; claim briefings, evidence(), the REVISION_REQUIRED spawn prompt, PM triage bounced-blocks, and A2A bodies deliver open findings; round-N+1 QA and gate reviewers get the full prior ledger; panel Findings tab + bounced-xN chip; metrics pm_rejects/ceo_rejects + findings counts; vault task notes render a Findings section (fail-open) - resolution closes for every origin: i_am_done and submit_up/submit_root take resolved_findings gated by FINDINGS_ADDRESSED (owner-gated so a stale non-owner PM can never mutate the ledger); pass_review/pr_pass/ complete verify-stamp same-transaction; ceo_approve stamps best-effort - 24 real-DB integration tests drive the full loop through the real choreographer; full suite 12856 green * docs: revision findings ledger sweep — CLAUDE.md, map, RAG corpus - CLAUDE.md: new ledger section + corrected request_changes row - docs/map/review-findings.md (new subsystem map) + surgical updates to task-service/pr-gate-review/metrics-observability/vault/panel maps - docs/rag: producers' findings contract across qa/pr-reviewer/developer/ cell-pm/main-pm/ceo role docs (the PM docs were missing request_changes entirely), verb references, and a new architecture/review-findings.md disambiguating ledger findings from convention findings * test(e2e): resubmit resolves the pr_fail finding per the ledger contract The scripted pr_fail revision loop resubmitted submit_up without resolved_findings — correctly rejected now that FINDINGS_ADDRESSED gates the PM resubmit verbs (green locally, red only in CI since the e2e suite skips without ROBOCO_E2E_SMOKE=1). The scripted PM now reads the open ledger row pr_fail persisted (new open_finding_ids arc helper) and resolves it on resubmit, asserting the open set drains — exercising the coordinator half of the new contract end to end. --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>	2026-07-11 22:54:42 +02:00
3 people	a8cb2470ba	v0.15.0: Metrics granularity — per-member / per-task / org + CEO scorecards (#289) ... * feat(metrics): capture per-session turns + tool_calls (phase 1) Persist LLM iterations (turns) and tool invocations per agent spawn session, the raw signal the granular per-member performance metrics build on (real effort/iterations vs wall-clock). - sum_transcript_usage returns a 5-tuple adding turns = unique assistant message-id count; _usage_from_transcript + _resolve_active_tokens updated to the 5-tuple (active-tokens keeps its 4-tuple contract by slicing). - SDK: _SessionState.turns, set by /usage/sync; /usage/status (TokenUsageStatus) now carries turns + tool_calls (= total_calls). - orchestrator: new _resolve_final_turns_tools (SDK primary, transcript fallback for turns only; Grok -> 0/0) wired into _finalize_spawn_session, which writes turns + tool_calls to agent_spawn_sessions. - migration 055 adds turns + tool_calls (BigInteger DEFAULT 0 -> historical/Grok rows read 0, surfaced as n/a). Verified real alembic upgrade/downgrade. Part of metrics-granularity (v0.15.0); recon-adjusted plan on disk. * feat(metrics): pure compute_stage_effort helper (phase 2, part 1) Foundation-layer overlap math (no DB): split each task status window into active (merged wall-clock overlap of spawn stints — concurrent stints counted once, so active <= window) vs wait (queue/review idle). Distinct from summed effort. The per-task metrics service will feed it audit-log windows + spawn stints. 9 unit tests (disjoint/nested/partial/merged/clamped/zero/multi-window). * feat(metrics): per-task live metrics + GET /metrics/task/{id} (phase 2) TaskMetrics dataclass + MetricsService.get_task_metrics: summed spawn effort (vs wall-clock), turns/tool_calls/tokens/cost, per-stage active-vs-wait (compute_stage_effort over audit windows x spawn stints), and who-caused-rework (revision_count + named qa/pr fail events). Open stints and the open final stage window close at completed_at for a terminal task (else now), so stages don't grow past completion. Exposed at GET /dashboard/metrics/task/{task_id} (404 if absent). Real-PG tests (compose/none/in-flight) + route tests (200/404). * feat(metrics): CEO-as-member scorecard + ceo_reject audit regression (phase 3) The human CEO is a measured member, read purely from audit_log (agent_role='ceo' serializes from the CEO StrEnum): approval dwell (awaiting_ceo_approval -> a CEO decision, incl. the coordination-root reject that lands in pending), unblock dwell (blocked -> a CEO revive), and god-mode action count (every CEO-attributed transition). CeoScorecard + MetricsService.get_ceo_scorecard (p50/p90 via PERCENTILE_CONT, expanding IN for the decision sets) + GET /dashboard/metrics/member/ceo (declared before any future member/{id} route). The ceo_reject coordination-root audit gap the plan meant to close was already closed by the gap-sweep (routes through admin_set_status -> agent_role='ceo' audit); locked with a regression assertion in the existing coordination-reject test. Real-PG tests: approval/unblock/godmode, non-ceo exclusion, empty->zeros. * feat(metrics): audit instrumentation for escalations/blocked-others/idle (phase 4a) The three extra per-member metrics that had no data source get durable, in-session audit events (additive; never gate the underlying action): - apply_escalation -> task.escalated (details.escalator_slug) on both the normal block path and the pool-divert path -> escalations count. - _unblock_dependents -> task.unblocked_dependents (details.count) on the completed BLOCKER task, captured before the dependency edges are pruned -> blocked-others count (sweeper attributes to the blocker's owner). - mark_agent_idle -> agent.idle (details.agent_slug) -> idle/utilization (the sweeper pairs an idle mark to the member's next spawn for idle duration). (QA pass-rate needs no new event — reuses task.awaiting_documentation[qa] + task.qa_fail.) Real-PG tests for each; 111 transition tests still green. * feat(metrics): member_performance_daily rollup table + migration 056 (phase 4b) The per-member scorecard rollup: one row per (date, member_kind, agent_slug), CEO as a first-class member_kind='ceo' row (agent_slug='' NOT NULL so the NULL-distinct UNIQUE keeps it unique). Full column set + the four CEO-approved extras (qa_reviews_total/passed, escalations, blocked_others, idle_seconds) plus blocked_seconds. Overwrite-upsert on (date, member_kind, agent_slug) for an idempotent sweep. Migration 056 verified real up/down (24 cols, 4 indexes). * feat(metrics): _sweep_member_performance rollup sweeper (phase 4c) The daily per-member rollup sweep (mirrors _sweep_daily_rollup): a trailing 7-day, idempotent overwrite-upsert wired into _run_sweep. One focused query per metric merges into a (date, agent_slug) accumulator — spawn effort/turns/tokens/ cost, completed/first-pass/revisions-received, revisions-caused (qa/pr fails), QA pass-rate (passed + total), escalations (by escalator_slug), blocked-others (unblocked_dependents by blocker owner), idle_seconds (idle mark -> next spawn), blocked_seconds (blocked dwell) — plus one CEO row/day (approval/unblock dwell + god-mode). Real-PG test asserts every facet + idempotency (a 2nd sweep overwrites, never doubles); spawn-day != completion-day split is by-design. * feat(metrics): member/org rollup scorecards + endpoints + live overlay (phase 5) MemberScorecard + OrgScorecard with derived rates (FPY, effort-throughput, turns/tool-calls per task, QA pass-rate, utilization) — all division-guarded to None. get_member_scorecard reads member_performance_daily by slug and overlays the member's live in-flight (non-terminal) tasks' effort via get_task_metrics (disjoint by status: completion counts stay rollup-only, overlay only enriches effort/turns/cost; includes_live_inflight flags it). get_org_scorecard aggregates the cell (?team=) or whole org. Routes: GET /metrics/member/{agent_id} (404 if absent, after the ceo literal route) + GET /metrics/org?team=. Real-PG tests (derived rates, overlay no double-count, guards, org) + route tests. * feat(metrics): granular CEO completion notification (phase 6) There was no CEO completion notification at all (EventType.TASK_COMPLETED was defined but never emitted). Add notify_ceo_of_completion in NotificationDeliveryService — a granular body (real effort vs wall-clock + stints/turns/tool-calls/revisions[QA/PR]/cost from get_task_metrics; degrades to wall-clock-only, turns 'n/a', when there are no spawn sessions). Reuses the existing ALERT type (no enum migration; the notificationtype PG enum is fixed at 001). ceo_approve now emits TASK_COMPLETED + fires the notification (best-effort via _notify_completion — never blocks completion); complete() emits TASK_COMPLETED too (closes the dead-code gap; the WS bridge can forward it). Pure formatter tests + real-PG notification test. * [metrics-granularity] Phase 7: panel Scorecards tab + dashboard overview Add the CEO-facing metrics surfaces for the granularity feature: - New "Scorecards" tab on the Metrics page: org rollup headline, the CEO-as-member card (approval/unblock dwell + god-mode count), and a per-member table (completed, first-pass yield, active effort, turns/task, QA pass-rate, escalations, blocked-others, utilization). Each member row self-fetches its rollup scorecard; live in-flight rows carry a "live" badge. - New dashboard overview card (ScorecardOverviewPanel): org-wide 30-day headline (completed, FPY, throughput/hr, active effort, cost) deep-linking into the Scorecards tab. - Plumbing: TaskMetrics/MemberScorecard/OrgScorecard/CeoScorecard types, observability API client methods + empty fallbacks, and the four useCeoScorecard/useMemberScorecard/useOrgScorecard/useTaskMetrics hooks. Panel gate green: tsc, eslint, prettier, vitest (175 tests, +6 new). * [metrics-granularity] test: make completion-notification robust to shared-DB CEO test_notify_ceo_of_completion_creates_alert errored in the full suite (passed in isolation): the session-scoped test DB is shared across the run, and the sibling real-DB board-gate test commits a role=CEO agent (slug="ceo") without cleanup — so my env fixture's hardcoded slug="ceo" insert hit a unique-constraint violation, and a second role=CEO row would also make _get_ceo_agent()'s scalar_one_or_none() raise. Reuse an existing CEO when present (the singleton the production system actually has), else create one with a unique slug. Order- independent. Also reflow test_metrics_instrumentation.py to ruff format. * chore(release): 0.15.0 Metrics granularity: per-member/per-task/org + CEO-as-member scorecards, turn/tool-call capture (migration 055), member_performance_daily rollup (migration 056) with QA pass-rate / escalations / blocked-others / utilization, per-task active-vs-wait metrics, granular completion notification, panel Scorecards tab + dashboard Performance card, and the ceo_reject audit fix. Version bump across the canonical set + CHANGELOG. * [metrics-granularity] fix pre-tag audit findings (overlay double-count + panel error states) Adversarial review before the v0.15.0 tag surfaced two real logical gaps: - MAJOR (backend): the live in-flight overlay re-summed ALL sessions of every non-terminal task via get_task_metrics, but _msweep_spawn already rolls up every CLOSED session regardless of task status — so a closed session on a still-open task was counted twice (rollup + overlay), permanently inflating a member's effort/turns/tokens/cost on the common reap/respawn path. The overlay now sums only OPEN sessions (ended_at IS NULL), which the closed-only rollup can never contain — disjoint by construction. A just-closed session lands in the rollup on the next ~60s sweep (no gap of note). Aggregated in SQL to mirror _msweep_spawn. Regression test reproduces the double-count (turns 10→5). - MAJOR (panel): the four new scorecard surfaces used `isLoading || !data` with no isError branch, so a failed query span forever on a skeleton. They now surface a load error. Tests added. Also: OrgSummary active-effort formatting no longer round-trips hours→seconds→ hours; dashboard grid uses xl:grid-cols-4 (was 2xl) so 4 panels show at 1280px; corrected the inaccurate "NULL distinct" CEO-row uniqueness comment (agent_slug is NOT NULL; the '' tuple is simply distinct from agent rows). make quality GREEN (cov 95.31%); panel GREEN (vitest 178). * [metrics-granularity] fix: decode bytes stream message-id before XCLAIM StreamEventBus._recover_stream passed the pending message id to XCLAIM via str() on the raw bytes the client returns (redis client has no decode_responses), producing "b'1782066556728-0'". Redis rejects that with "Unrecognized XCLAIM option", so pending-message recovery threw on every reclaim tick and unacked messages from crashed/slow consumers were never reclaimed (leaking in the PEL on every stream, spamming the error log). Decode via the existing _to_str helper — the fix the sibling claim path already uses. Pre-existing in v0.14.0 (unrelated to metrics granularity); folded into this release per CEO. TDD regression test + CHANGELOG entry. make quality GREEN. --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>	2026-07-01 05:18:45 +02:00