feat(cockpit): expose first_pass_yield and a real escaped-defects metric (#709)

The Company Scorecard renders three charter objectives but the cockpit
summary only ever carried one of the metrics, so two cards read "No data
yet" permanently.

first_pass_yield is a pass-through — MetricsService.get_org_scorecard()
already computes it on the same 30d/org scope the rest of the delivery block
uses, and CockpitService.summary simply never forwarded it.

escaped_defects is new. The obvious definition — a blocker finding opened on
a task that already reached a terminal state — is unimplementable: every
producer of a task_review_findings row fires as part of a bounce whose
transition requires a non-terminal task, so it would read zero forever, and a
permanently-green card is the same fabrication the panel change removes.

What it counts instead: a blocker still at 'addressed', never 'verified', on
a task that has since completed. That is reachable because
stamp_addressed_verified only bulk-verifies rows matching its OWN origin, so
a blocker raised by one origin and never re-confirmed by that origin survives
to completion on the developer's word alone.

docs/map/metrics-observability.md documents what a zero actually means: the
one reachable trigger is a PM-origin blocker on a task escalated to the CEO
rather than completed by the PM, since escalate_to_ceo carries no
findings-resolved precondition and ceo_approve verifies only ceo-origin rows.
It also records that the count is per-finding over a rolling 30-day window,
which is not the same unit as the charter's "per release".

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
2026-07-26 19:11:27 +02:00
committed by GitHub
co-authored by Renn F
parent 0fe21b1f97
commit 80ccf415cb
7 changed files with 268 additions and 10 deletions
+11 -4
View File
@@ -10,7 +10,7 @@ The metrics & observability slice is the read-only measurement layer of RoboCo:
|---|---|---|
| `roboco/services/metrics.py` | `MetricsService` — velocity, blockers, team/agent metrics, health, cycle-time/bottleneck/rework/scorecard observability | 1521 |
| `roboco/services/dashboard.py` | `DashboardService` — auditor flags/reports (in-memory singleton), CEO overview, audit queue, agent status, recent activity | 457 |
| `roboco/services/cockpit.py` | `CockpitService` — read-only CEO "is the business winning?" summary (goals+delivery+spend+signals) | 97 |
| `roboco/services/cockpit.py` | `CockpitService` — read-only CEO "is the business winning?" summary (goals+delivery+spend+signals), delivery block now carries the 3 charter-objective metrics (`median_lead_time_hours`, `first_pass_yield`, `escaped_defects`) | 104 |
| `roboco/services/usage.py` | `UsageService` — token usage summary, time-series, by-agent/team/model, projection, cache efficiency, today summary, recent sessions | 478 |
| `roboco/services/usage_events.py` | `UsageSnapshot` dataclass + `publish_usage_snapshot` — publishes USAGE_SNAPSHOT to the StreamEventBus | 52 |
| `roboco/services/telemetry/__init__.py` | Re-export of CI telemetry source symbols | 18 |
@@ -58,8 +58,9 @@ The metrics & observability slice is the read-only measurement layer of RoboCo:
| `DashboardService.get_all_agent_status` | method | dashboard.py:365 | Agent counts by status + per-agent snapshot |
| `DashboardService.get_recent_activity` | method | dashboard.py:398 | Merged messages+task_updates feed, sorted desc |
| `CockpitService` | class | cockpit.py:36 | Read-only CEO summary + lightweight signals slice |
| `CockpitService.summary` | method | cockpit.py:41 | goals+counts+delivery+spend+projection+pitches+signals, `basis="proxy"` |
| `CockpitService.summary` | method | cockpit.py:41 | goals+counts+delivery+spend+projection+pitches+signals, `basis="proxy"`. `delivery.first_pass_yield` is a pass-through of `MetricsService.get_org_scorecard().first_pass_yield` (no new computation); `delivery.escaped_defects` is `len(ReviewFindingsRepository.escaped_defects_since(30d cutoff))` — see the Gotchas entry below for the definition. |
| `CockpitService.signals` | method | cockpit.py:81 | Strategy-engine signals only (lightweight panel slice) |
| `ReviewFindingsRepository.escaped_defects_since` | method | `services/repositories/review_findings.py` | `(task_id, origin)` pairs for blocker findings still `addressed` (never `verified`) on a `COMPLETED` task within the window — the Company Scorecard's "0 critical escaped defects" metric. See `docs/map/review-findings.md` and the Gotchas entry below. |
| `UsageService` | class | usage.py:70 | Token usage analytics over spawn sessions + rollups |
| `UsageService.get_summary` | method | usage.py:77 | Period totals + trend_pct vs previous period |
| `UsageService.get_time_series` | method | usage.py:166 | Hourly (24h) / daily (7d/30d) buckets |
@@ -122,9 +123,12 @@ graph LR
Panel -->|/api/cockpit/*| Cockpit[CockpitService]
Cockpit --> Goals[company_goals]
Cockpit --> TaskSvc[TaskService]
Cockpit --> MetricsSvc
Cockpit --> FindingsRepo[ReviewFindingsRepository.escaped_defects_since]
Cockpit --> UsageSvc
Cockpit --> Strategy[strategy_engine]
Cockpit --> Pitch[pitch service]
FindingsRepo --> TaskReviewFindings[task_review_findings]
SelfHeal[self_heal_loop] --> CISrc[GitHubCITelemetrySource]
CIWatch[ci_watch_loop] --> MultiSrc[MultiProjectCITelemetrySource]
@@ -164,7 +168,7 @@ metrics-observability
- `roboco.events.stream_bus``StreamEventBus` (TYPE_CHECKING only)
- `roboco.services.base``BaseService`
- `roboco.services.git``GitService.get_latest_ci_conclusion` (telemetry)
- `roboco.services.company_goals`, `pitch`, `strategy_engine`, `task` (cockpit)
- `roboco.services.company_goals`, `pitch`, `strategy_engine`, `task`, `metrics` (`get_metrics_service`), `repositories.review_findings` (`ReviewFindingsRepository`) (cockpit)
- `roboco.config``settings` (telemetry)
- `roboco.logging``get_logger` (telemetry)
- `roboco.utils.converters``to_python_uuid`, `require_uuid`
@@ -212,7 +216,10 @@ No flags live *inside* this slice's files, but the slice's behavior is gated/par
- **Cache-efficiency uses hardcoded sonnet pricing** (usage.py:401-404, `_FULL_INPUT_PRICE=3.00`, `_CACHE_READ_PRICE=0.30`) for the savings estimate regardless of the actual model mix — an aggregate approximation, not per-model.
- **`publish_usage_snapshot` lazy-imports `Event`/`EventType`** (usage_events.py:49) to avoid a circular import — callers must keep the bus passed in, not a module-level reference.
- **`MultiProjectCITelemetrySource.fetch` swallows per-project exceptions** (source.py:172) — one bad project never aborts the sweep, but also never surfaces beyond a warning log; a persistently failing project silently contributes no sample (treated as "unknown", not "green" — correct, but invisible).
- **`CockpitService.summary` `basis="proxy"`** (cockpit.py:57) — every payload is stamped proxy; the over_budget flag is only meaningful once the CEO greenlights real launch.
- **`CockpitService.summary` `basis="proxy"`** (cockpit.py:66) — every payload is stamped proxy; the over_budget flag is only meaningful once the CEO greenlights real launch.
- **`escaped_defects` definition — why "a finding on a terminal task" is impossible, and what it actually counts.** The Company Scorecard's third charter objective ("0 critical escaped defects per release") looks like it should mean "a blocker-severity finding opened on a task that already reached a terminal state" — but that combination can never occur: every producer of a `task_review_findings` row (`fail_review`, `pr_fail`, `request_changes`, `ceo_reject`) fires as part of a bounce whose lifecycle transition requires the task to be non-terminal at that moment (the transition itself is `* -> needs_revision`). A "terminal-task finding" query would return 0 in every window, forever — a permanently-green scorecard card is worse than none, the exact fabrication PR #704 exists to remove from the panel. The real definition, computed by `ReviewFindingsRepository.escaped_defects_since`: a `blocker`-severity finding still at status `addressed` (**never** `verified`) on a task that has since gone `COMPLETED`, within the 30-day window (`TaskTable.completed_at >= cutoff`; `cancelled` tasks are excluded on purpose — they never set `completed_at` and never ship code, so nothing "escaped" from one). This is reachable because `stamp_addressed_verified` (`services/gateway/choreographer/findings.py:306`) only bulk-verifies findings of its OWN `origin` (`row.origin == origin`) when its matching pass verb runs (`pass_review`→qa, `pr_pass`→pr_gate, `complete`→pm, `ceo_approve`→ceo — `complete` stamps `origin="pm"` via `_stamp_pm_findings_verified_or_rejection` (`services/gateway/choreographer/_impl.py:7431-7456`), a distinct verb+stamp from `ceo_approve` (`services/task.py:7289-7294`), which stamps `origin="ceo"` on its own `awaiting_ceo_approval → completed` transition) — a blocker raised by one origin, marked `addressed` by the developer, and never independently re-confirmed by that SAME origin on a later round (the task's remaining rounds routed through a different reviewer) survives all the way to `completed` still `addressed`. A non-zero value means: at least one blocker-severity concern shipped to `completed` on the developer's own word alone, with no reviewer ever re-checking the fix — a real signal of unverified risk in production, not a fabricated placeholder.
- **In practice, "pm" is the only origin that can realistically produce a non-zero reading, and even that path is narrow.** `pass_review` and `pr_pass` (qa/pr_gate origins) hard-gate their `stamp_addressed_verified` call — a stamp failure fails the verb itself (qa.py:809-823, pr_gate.py mirrors it), so a qa-origin or pr_gate-origin blocker structurally cannot reach `completed` still `addressed`: passing review IS re-verifying it. The one reachable path is: a PM raises a blocker via `request_changes` (origin=`pm`), the dev addresses it, and the PM then calls `escalate_to_ceo` instead of `complete``escalate_to_ceo`'s `ActionSpec` (`foundation/policy/lifecycle.py:657-675`) has no precondition requiring findings be resolved, and `ceo_approve` only bulk-verifies its own `ceo`-origin rows, never touching the still-`addressed` `pm`-origin one. So a fleet that rarely escalates to the CEO (the normal case — most roots complete via a PM's own `complete`) will read 0 on this metric because the triggering path is rare, not because nothing has escaped.
- **The count is per-finding, not per-task, and the 30-day window is a temporal proxy, not a release boundary.** `escaped_defects_since` returns one `(task_id, origin)` row per qualifying finding with no de-dup/grouping, and `CockpitService.summary` takes `len(escaped)` directly — a single task with three qualifying blockers contributes 3 to the count, not 1. The charter's "0 critical escaped defects per release" phrasing implies release-scoped counting, but this metric has no notion of releases at all: it's a rolling `completed_at >= now - 30d` window that will straddle zero, one, or several actual release cuts depending on cadence, so a spike right after a release and a spike from unrelated day-to-day completions look identical on this card.
## Drift from CLAUDE.md
+3 -3
View File
@@ -44,7 +44,7 @@ The revision-findings ledger: the structured replacement for prose-only QA/PR-ga
| `TaskReviewFindingTable` | ORM class | `roboco/db/tables.py` | The append-only ledger row: `task_id`, `origin`, `round`, `author_slug`, `file`/`line`/`severity`/`criterion`/`expected`/`actual`/`fix`/`evidence`, `status`, `addressed_by_commit`, `resolution_note`. `origin`/`severity`/`status` are plain `String` columns, not a native Postgres enum. |
| `Finding` | Pydantic model | `roboco/foundation/policy/content/models.py:92` | One structured finding, shared by `post_pr_review` (external PRs) and the four internal producers. Caps: `file` ≤300 (repo-relative, no `..`), `line` ≥1, `expected`/`actual` ≤300, `fix` ≤500, `evidence` ≤2000. `criterion` has **no Pydantic `max_length`** despite the DB column being `String(500)` — see Regression Risks. `file` is additionally shape-gated (`_PATH_SHAPE_RE`, #687): a value that doesn't look like a repo-relative path (prose like a PR reference, which used to validate and then doomed the panel's code-snippet fetch) is rejected with a remediate naming the file-less option for cross-cutting findings; the class admits `+`/`@` (SvelteKit route files, `@types` dirs, `@2x` assets) but excludes spaces, the prose signal. `file` remains OPTIONAL for the `issues` shim's file-less findings. |
| `PmReviewContent` | Pydantic model | `roboco/foundation/policy/content/models.py` | New content type `"pm_review"` (`summary` + `findings`, no separate `verdict` — the transition to `needs_revision` IS the verdict); mirrors to the new `tasks.pm_notes` column via `_MIRROR_COLUMN`. |
| `ReviewFindingsRepository` | class | `roboco/services/repositories/review_findings.py:32` | `insert_many` (append rows, one flush, no independent commit), `list_for_task` (default cap 500, newest round first), `status_counts_for_task` (SQL `GROUP BY (origin, status)`, whole ledger — independent of the 500 cap), `mark_addressed` (8-char-prefix match against OPEN rows, no-op on 0 or >1 matches, never raises), `mark_verified` (bulk, by full id), `mark_waived` (exists, unwired — no verb calls it). |
| `ReviewFindingsRepository` | class | `roboco/services/repositories/review_findings.py:32` | `insert_many` (append rows, one flush, no independent commit), `list_for_task` (default cap 500, newest round first), `status_counts_for_task` (SQL `GROUP BY (origin, status)`, whole ledger — independent of the 500 cap), `mark_addressed` (8-char-prefix match against OPEN rows, no-op on 0 or >1 matches, never raises), `mark_verified` (bulk, by full id), `mark_waived` (exists, unwired — no verb calls it), `escaped_defects_since` (`(task_id, origin)` for blocker findings still `addressed`, never `verified`, on a task that has since gone `COMPLETED` within a window — the Company Scorecard's `escaped_defects` metric; see `docs/map/metrics-observability.md`). |
| `findings_count_guard` / `findings_count_hint` | functions | `roboco/services/gateway/choreographer/findings.py:98,115` | Hard-reject `Envelope` above `FINDINGS_HARD_CAP=10`; non-blocking hint above `FINDINGS_NUDGE_COUNT=5`. |
| `issues_to_findings` / `merge_findings_and_issues` | functions | `roboco/services/gateway/choreographer/findings.py:57,81` | Legacy `issues: list[str]` shim → file-less `severity=major` findings (deprecation-logged); merges with any `findings` sent in the same call rather than one silently dropping the other. |
| `next_round` | function | `roboco/services/gateway/choreographer/findings.py:43` | `(task.revision_count or 0) + 1`, read BEFORE the transition — the round a finding written during this call belongs to. |
@@ -136,7 +136,7 @@ review-findings slice
- `roboco.services.gateway.envelope``Envelope`
- `roboco.services.gateway.evidence_builder``BRIEFING_LIST_CAP`
- `roboco.db.tables``TaskReviewFindingTable`, `TaskTable.pm_notes`
- Consumed by: `roboco.services.metrics` (`MetricsService`), `roboco.services.vault_assembly`/`vault_writer`, `roboco.runtime.orchestrator`, `roboco.mcp.flow_server`, `roboco.api.routes.tasks`/`v1.flow_*`, the panel's `task-detail`/`metrics` components
- Consumed by: `roboco.services.metrics` (`MetricsService`), `roboco.services.cockpit` (`CockpitService.summary`'s `escaped_defects` field), `roboco.services.vault_assembly`/`vault_writer`, `roboco.runtime.orchestrator`, `roboco.mcp.flow_server`, `roboco.api.routes.tasks`/`v1.flow_*`, the panel's `task-detail`/`metrics` components
## Entry Points
@@ -171,7 +171,7 @@ None as of this doc's authoring — CLAUDE.md was updated in the same pass to ad
- `docs/map/task-service.md``ceo_reject`, `_audit_events_for`
- `docs/map/pr-gate-review.md``pr_fail` findings wiring, gate evidence, verify-stamp
- `docs/map/metrics-observability.md` — rework-by-agent event widening, per-task findings counts
- `docs/map/metrics-observability.md` — rework-by-agent event widening, per-task findings counts, `escaped_defects` Company Scorecard metric
- `docs/map/vault.md` — task note `## Findings` section
- `docs/map/panel.md` — Findings tab, `bounced xN` chip, findings route
- `docs/internal/specs/2026-07-11-revision-findings-ledger.md` — the design spec this slice implements
+2
View File
@@ -12,6 +12,8 @@ class DeliverySummary(BaseModel):
awaiting_ceo: int
completed_30d: int = 0
median_lead_time_hours: float | None = None
first_pass_yield: float | None = None
escaped_defects: int | None = None
class SpendSummary(BaseModel):
+11
View File
@@ -11,11 +11,14 @@ CEO greenlights real external launches — every payload is stamped
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING, Any
from roboco.services.base import BaseService
from roboco.services.company_goals import get_company_goals_service
from roboco.services.metrics import get_metrics_service
from roboco.services.pitch import get_pitch_service
from roboco.services.repositories.review_findings import ReviewFindingsRepository
from roboco.services.strategy_engine import get_strategy_engine
from roboco.services.task import get_task_service
from roboco.services.usage import get_usage_service
@@ -43,6 +46,12 @@ class CockpitService(BaseService):
goals = await get_company_goals_service(self.session).get()
counts = await task_svc.count_by_status()
delivery_stats = await task_svc.get_delivery_stats_30d()
# get_org_scorecard() defaults to the same 30d/org scope as everything
# else in this block — one round trip for first_pass_yield, no new math.
scorecard = await get_metrics_service(self.session).get_org_scorecard()
escaped = await ReviewFindingsRepository(self.session).escaped_defects_since(
datetime.now(UTC) - timedelta(days=30)
)
usage_svc = get_usage_service(self.session)
spend = await usage_svc.get_summary("30d")
projection = await usage_svc.get_projection()
@@ -64,6 +73,8 @@ class CockpitService(BaseService):
"awaiting_ceo": counts.get("awaiting_ceo_approval", 0),
"completed_30d": delivery_stats["completed_30d"],
"median_lead_time_hours": delivery_stats["median_lead_time_hours"],
"first_pass_yield": scorecard.first_pass_yield,
"escaped_defects": len(escaped),
},
"spend": {
"spend_30d_usd": round(spend_30d, 2),
@@ -13,10 +13,12 @@ from typing import TYPE_CHECKING
from sqlalchemy import func, select
from roboco.db.tables import TaskReviewFindingTable
from roboco.db.tables import TaskReviewFindingTable, TaskTable
from roboco.models.base import TaskStatus
from roboco.services.repositories.base import BaseRepository
if TYPE_CHECKING:
from datetime import datetime
from uuid import UUID
from roboco.foundation.policy.content import Finding
@@ -200,6 +202,45 @@ class ReviewFindingsRepository(BaseRepository[TaskReviewFindingTable]):
result = await self.session.execute(stmt)
return [(file, count) for file, count in result.all()]
async def escaped_defects_since(self, since: datetime) -> list[tuple[UUID, str]]:
"""``(task_id, origin)`` for each blocker finding that shipped unverified:
still ``addressed`` (never independently ``verified``) on a task that
has since gone ``completed``, within the window.
``verified`` is deliberately excluded, not just ``open``/``waived``
see ``stamp_addressed_verified``, which only bulk-verifies its OWN
origin's rows. A blocker raised by one origin (e.g. ``pr_gate``),
marked ``addressed`` by the developer, and never re-checked by that
same origin (the task's later rounds routed through QA/PM instead)
survives to completion still ``addressed`` shipped without the
raiser ever confirming the fix. That's the escaped defect; see
docs/map/metrics-observability.md for the full rationale.
Scoped to ``COMPLETED`` (not every terminal state): ``cancelled``
tasks never set ``completed_at`` and never ship code, so nothing
"escaped" from one. The explicit ``status == COMPLETED`` check is
belt-and-braces on top of the ``completed_at`` conditions: today
every site that sets ``completed_at`` also transitions the task to
``COMPLETED`` in the same call (``TaskService.complete`` and
``TaskService.ceo_approve``), so the two are redundant in practice
but that pairing is an invariant of those call sites, not something
this query can see, so keep the status check rather than "simplify"
it away.
"""
stmt = (
select(TaskReviewFindingTable.task_id, TaskReviewFindingTable.origin)
.join(TaskTable, TaskTable.id == TaskReviewFindingTable.task_id)
.where(
TaskReviewFindingTable.severity == "blocker",
TaskReviewFindingTable.status == STATUS_ADDRESSED,
TaskTable.status == TaskStatus.COMPLETED,
TaskTable.completed_at.is_not(None),
TaskTable.completed_at >= since,
)
)
result = await self.session.execute(stmt)
return [(task_id, origin) for task_id, origin in result.all()]
async def list_open_findings(
self, *, limit: int = 20
) -> list[TaskReviewFindingTable]:
+24
View File
@@ -25,6 +25,8 @@ _BUDGET = 100.0
_SPEND_30D = 150.0
_COMPLETED_30D = 5
_MEDIAN_LEAD_TIME = 12.5
_FIRST_PASS_YIELD = 0.92
_ESCAPED_DEFECTS = 2
def _agent(role: AgentRole) -> AgentContext:
@@ -60,6 +62,24 @@ def _patch(monkeypatch: pytest.MonkeyPatch) -> None:
get_delivery_stats_30d=AsyncMock(return_value=delivery_stats),
),
)
monkeypatch.setattr(
cm,
"get_metrics_service",
lambda _s: MagicMock(
get_org_scorecard=AsyncMock(
return_value=MagicMock(first_pass_yield=_FIRST_PASS_YIELD)
)
),
)
monkeypatch.setattr(
cm,
"ReviewFindingsRepository",
lambda _s: MagicMock(
escaped_defects_since=AsyncMock(
return_value=[(uuid4(), "qa"), (uuid4(), "pr_gate")]
)
),
)
usage = MagicMock(
get_summary=AsyncMock(return_value={"total_cost_usd": _SPEND_30D}),
get_projection=AsyncMock(return_value={"projected_monthly_cost_usd": 200.0}),
@@ -95,6 +115,8 @@ async def test_summary_aggregates(monkeypatch: pytest.MonkeyPatch) -> None:
assert out["delivery"]["blocked"] == _BLOCKED
assert out["delivery"]["completed_30d"] == _COMPLETED_30D
assert out["delivery"]["median_lead_time_hours"] == _MEDIAN_LEAD_TIME
assert out["delivery"]["first_pass_yield"] == _FIRST_PASS_YIELD
assert out["delivery"]["escaped_defects"] == _ESCAPED_DEFECTS
assert out["spend"]["spend_30d_usd"] == _SPEND_30D
assert out["spend"]["over_budget"] is True
assert out["pending_pitches"] == 1
@@ -121,6 +143,8 @@ async def test_route_ok_for_ceo(monkeypatch: pytest.MonkeyPatch) -> None:
"awaiting_ceo": 0,
"completed_30d": 0,
"median_lead_time_hours": None,
"first_pass_yield": None,
"escaped_defects": 0,
},
"spend": {
"spend_30d_usd": 0.0,
@@ -9,6 +9,7 @@ migration replay here.
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING
from uuid import UUID, uuid4
@@ -49,17 +50,24 @@ async def _seed_agent(session: AsyncSession) -> UUID:
return UUID(str(agent.id))
async def _seed_task(session: AsyncSession, created_by: UUID) -> UUID:
async def _seed_task(
session: AsyncSession,
created_by: UUID,
*,
status: TaskStatus = TaskStatus.NEEDS_REVISION,
completed_at: datetime | None = None,
) -> UUID:
task = TaskTable(
id=uuid4(),
title="ledger seed task",
description="seed",
acceptance_criteria=["seeded"],
status=TaskStatus.NEEDS_REVISION,
status=status,
priority=2,
task_type=TaskType.CODE,
team=Team.BACKEND,
created_by=created_by,
completed_at=completed_at,
)
session.add(task)
await session.flush()
@@ -341,3 +349,168 @@ async def test_list_open_findings_excludes_non_open(
)
await repo.mark_waived(UUID(str(rows[0].id)), "nit, skip")
assert await repo.list_open_findings(limit=20) == []
# escaped_defects_since — the Company Scorecard's "0 critical escaped defects"
# metric: a blocker finding still ADDRESSED (never independently VERIFIED by
# its own raising origin) on a task that has since gone COMPLETED, in-window.
@pytest.mark.asyncio
async def test_escaped_defects_since_counts_addressed_blocker_on_completed_task(
db_session: AsyncSession,
) -> None:
agent_id = await _seed_agent(db_session)
task_id = await _seed_task(
db_session,
agent_id,
status=TaskStatus.COMPLETED,
completed_at=datetime.now(UTC),
)
repo = ReviewFindingsRepository(db_session)
rows = await repo.insert_many(
task_id=task_id,
origin="pr_gate",
round=1,
author_slug="be-dev-1",
findings=[_finding(severity=Severity.BLOCKER)],
)
await repo.mark_addressed(task_id, str(rows[0].id), commit="abc123", note="fixed")
result = await repo.escaped_defects_since(datetime.now(UTC) - timedelta(days=30))
assert result == [(task_id, "pr_gate")]
@pytest.mark.asyncio
async def test_escaped_defects_since_excludes_verified_blocker(
db_session: AsyncSession,
) -> None:
"""A blocker VERIFIED by its raising origin was actually re-confirmed —
not an escaped defect."""
agent_id = await _seed_agent(db_session)
task_id = await _seed_task(
db_session,
agent_id,
status=TaskStatus.COMPLETED,
completed_at=datetime.now(UTC),
)
repo = ReviewFindingsRepository(db_session)
rows = await repo.insert_many(
task_id=task_id,
origin="qa",
round=1,
author_slug="be-qa",
findings=[_finding(severity=Severity.BLOCKER)],
)
await repo.mark_addressed(task_id, str(rows[0].id), commit=None, note=None)
await repo.mark_verified([UUID(str(rows[0].id))])
result = await repo.escaped_defects_since(datetime.now(UTC) - timedelta(days=30))
assert result == []
@pytest.mark.asyncio
async def test_escaped_defects_since_excludes_non_blocker_severity(
db_session: AsyncSession,
) -> None:
"""Only blocker severity counts — a major finding, however unresolved,
is not a "critical escaped defect"."""
agent_id = await _seed_agent(db_session)
task_id = await _seed_task(
db_session,
agent_id,
status=TaskStatus.COMPLETED,
completed_at=datetime.now(UTC),
)
repo = ReviewFindingsRepository(db_session)
rows = await repo.insert_many(
task_id=task_id,
origin="qa",
round=1,
author_slug="be-qa",
findings=[_finding(severity=Severity.MAJOR)],
)
await repo.mark_addressed(task_id, str(rows[0].id), commit=None, note=None)
result = await repo.escaped_defects_since(datetime.now(UTC) - timedelta(days=30))
assert result == []
@pytest.mark.asyncio
async def test_escaped_defects_since_excludes_non_terminal_task(
db_session: AsyncSession,
) -> None:
"""A still-open task hasn't shipped anything yet — nothing has escaped.
(This case is actually pinned by the `completed_at IS NOT NULL` condition,
since a non-terminal task never has one set see the sibling test below
for a case that isolates the `status == COMPLETED` filter itself.)"""
agent_id = await _seed_agent(db_session)
task_id = await _seed_task(db_session, agent_id) # default: NEEDS_REVISION
repo = ReviewFindingsRepository(db_session)
rows = await repo.insert_many(
task_id=task_id,
origin="qa",
round=1,
author_slug="be-qa",
findings=[_finding(severity=Severity.BLOCKER)],
)
await repo.mark_addressed(task_id, str(rows[0].id), commit=None, note=None)
result = await repo.escaped_defects_since(datetime.now(UTC) - timedelta(days=30))
assert result == []
@pytest.mark.asyncio
async def test_escaped_defects_since_excludes_non_completed_status(
db_session: AsyncSession,
) -> None:
"""Isolates the `TaskTable.status == COMPLETED` filter: a CANCELLED task
with a (synthetic, out-of-band) `completed_at` set inside the window would
still pass the two `completed_at` conditions alone only the status
check excludes it. Without this case, deleting the status filter leaves
every other test passing (proven by mutation testing)."""
agent_id = await _seed_agent(db_session)
task_id = await _seed_task(
db_session,
agent_id,
status=TaskStatus.CANCELLED,
completed_at=datetime.now(UTC),
)
repo = ReviewFindingsRepository(db_session)
rows = await repo.insert_many(
task_id=task_id,
origin="qa",
round=1,
author_slug="be-qa",
findings=[_finding(severity=Severity.BLOCKER)],
)
await repo.mark_addressed(task_id, str(rows[0].id), commit=None, note=None)
result = await repo.escaped_defects_since(datetime.now(UTC) - timedelta(days=30))
assert result == []
@pytest.mark.asyncio
async def test_escaped_defects_since_excludes_outside_window(
db_session: AsyncSession,
) -> None:
"""A task completed before the window cutoff doesn't count toward it."""
agent_id = await _seed_agent(db_session)
task_id = await _seed_task(
db_session,
agent_id,
status=TaskStatus.COMPLETED,
completed_at=datetime.now(UTC) - timedelta(days=40),
)
repo = ReviewFindingsRepository(db_session)
rows = await repo.insert_many(
task_id=task_id,
origin="qa",
round=1,
author_slug="be-qa",
findings=[_finding(severity=Severity.BLOCKER)],
)
await repo.mark_addressed(task_id, str(rows[0].id), commit=None, note=None)
result = await repo.escaped_defects_since(datetime.now(UTC) - timedelta(days=30))
assert result == []