Files
roboco/tests/unit/gateway/test_choreographer_completion_guards.py
T
cea3e56628 feat(lifecycle): revision findings ledger — structured failure feedback, persisted and delivered down the chain (#486)
* feat(lifecycle): revision findings ledger — structured QA/PR/PM/CEO failure feedback, persisted and delivered down the chain

Every bounce used to survive only as flattened prose: rounds overwrote each
other in notes_structured, request_changes persisted nothing, two raw
dev_notes appends were silently destroyed by the next handoff note, and the
dev prompt pointed at fields (qa_notes via evidence(), pm_notes) the API
never delivered. Agents re-interpreted and re-discovered every failure
before they could start fixing it.

- task_review_findings (migration 071, append-only): file/line/severity/
  criterion(AC-id-validated)/expected/actual/fix/evidence per finding, with
  origin (qa|pr_gate|pm|ceo), round, and an open->addressed->verified
  lifecycle (waived reserved); new tasks.pm_notes + PmReviewContent give
  request_changes a structured home
- producers: fail_review/pr_fail/request_changes take findings=[...] (prose
  issues shimmed+merged for one release, deprecation-logged); ceo_reject
  validates its reason (no 500), lands an origin=ceo finding, and bumps
  round+audit on branchless coordination roots; guardrails at the verb
  chokepoint (nudge >5, hard reject >10, field caps, traversal-safe file);
  the dev_notes data-loss appends are removed; new task.request_changes +
  task.ceo_reject audit events close rework attribution
- delivery: qa_notes/pr_reviewer_notes/pm_notes carry the deterministic
  [F-id8] rendering; claim briefings, evidence(), the REVISION_REQUIRED
  spawn prompt, PM triage bounced-blocks, and A2A bodies deliver open
  findings; round-N+1 QA and gate reviewers get the full prior ledger;
  panel Findings tab + bounced-xN chip; metrics pm_rejects/ceo_rejects +
  findings counts; vault task notes render a Findings section (fail-open)
- resolution closes for every origin: i_am_done and submit_up/submit_root
  take resolved_findings gated by FINDINGS_ADDRESSED (owner-gated so a
  stale non-owner PM can never mutate the ledger); pass_review/pr_pass/
  complete verify-stamp same-transaction; ceo_approve stamps best-effort
- 24 real-DB integration tests drive the full loop through the real
  choreographer; full suite 12856 green

* docs: revision findings ledger sweep — CLAUDE.md, map, RAG corpus

- CLAUDE.md: new ledger section + corrected request_changes row
- docs/map/review-findings.md (new subsystem map) + surgical updates to
  task-service/pr-gate-review/metrics-observability/vault/panel maps
- docs/rag: producers' findings contract across qa/pr-reviewer/developer/
  cell-pm/main-pm/ceo role docs (the PM docs were missing request_changes
  entirely), verb references, and a new architecture/review-findings.md
  disambiguating ledger findings from convention findings

* test(e2e): resubmit resolves the pr_fail finding per the ledger contract

The scripted pr_fail revision loop resubmitted submit_up without
resolved_findings — correctly rejected now that FINDINGS_ADDRESSED gates
the PM resubmit verbs (green locally, red only in CI since the e2e suite
skips without ROBOCO_E2E_SMOKE=1). The scripted PM now reads the open
ledger row pr_fail persisted (new open_finding_ids arc helper) and
resolves it on resubmit, asserting the open set drains — exercising the
coordinator half of the new contract end to end.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-07-11 22:54:42 +02:00

488 lines
18 KiB
Python

"""Gate Set F: completion-time guards.
cell_pm_complete / main_pm_complete / submit_up must refuse to advance
the parent past awaiting_pm_review when any subtask is still non-
terminal. Pre-gateway location: roboco/services/task.py closure check.
These tests verify:
1. The non-terminal-subtask refusal fires.
2. The remediation NAMES the non-terminal subtasks (improvement over
the previous generic "find pending subtasks" hint).
"""
from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from unittest.mock import AsyncMock, MagicMock
from uuid import uuid4
import pytest
from roboco.services.gateway.choreographer import Choreographer, ChoreographerDeps
def _make_deps(**overrides: Any) -> ChoreographerDeps:
base: dict[str, Any] = {
"task": AsyncMock(),
"work_session": AsyncMock(),
"git": AsyncMock(),
"a2a": AsyncMock(),
"journal": AsyncMock(),
"audit": AsyncMock(),
"evidence_repo": AsyncMock(),
}
base.update(overrides)
# VerbRunner wraps composed atomic actions in
# ``task.session.begin_nested()``. AsyncMock auto-attribute access
# would return an unawaitable coroutine, breaking the
# ``async with`` protocol. Overwrite session with a MagicMock that
# implements the async-context-manager protocol explicitly.
task_dep = base["task"]
task_dep.session = MagicMock()
task_dep.session.begin_nested = MagicMock(
return_value=MagicMock(
__aenter__=AsyncMock(return_value=None),
__aexit__=AsyncMock(return_value=False),
)
)
# cell_pm_complete / main_pm_complete's pm-origin verified-stamp reads via
# session.execute (ReviewFindingsRepository.list_for_task) before merging
# — an empty scalars result (no findings) so the stamp is a no-op here.
task_dep.session.execute = AsyncMock(
return_value=MagicMock(
scalars=MagicMock(return_value=MagicMock(all=MagicMock(return_value=[])))
)
)
repo = base["evidence_repo"]
for method in (
"list_unread_a2a",
"list_unread_mentions",
"list_pending_notifications",
"task_metadata_gaps",
"recent_team_activity",
"blockers_in_lane",
"journal_highlights_for_task",
):
getattr(repo, method).return_value = []
# C8: default-fresh journal:decision so PM-decision gate passes.
# Tests that exercise the gate boundary stub their own value.
# The check matches MagicMock and AsyncMock (the two default sentinel
# types pytest's unittest.mock leaves on un-stubbed return_values).
_ldef = base["journal"].latest_decision_at.return_value
if type(_ldef).__name__ in ("MagicMock", "AsyncMock"):
base["journal"].latest_decision_at.return_value = datetime.now(UTC)
return ChoreographerDeps(**base)
# ---------------------------------------------------------------------------
# cell_pm_complete subtask gate
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_cell_pm_complete_blocks_when_subtask_pending() -> None:
pm_id = uuid4()
parent_id = uuid4()
sub_id = uuid4()
t = MagicMock(
id=parent_id,
status="awaiting_pm_review",
assigned_to=pm_id,
pr_number=10,
team="backend",
branch_name="feature/backend/abc",
)
sub = MagicMock(id=sub_id, status="pending", title="Half-done subtask")
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.all_subtasks_terminal.return_value = False
task_svc.get_subtasks.return_value = [sub]
journal_svc = AsyncMock()
journal_svc.has_decision_for_task.return_value = True
journal_svc.latest_decision_at.return_value = datetime.now(UTC)
journal_svc.has_reflect_for_task.return_value = True
deps = _make_deps(task=task_svc, journal=journal_svc)
c = Choreographer(deps)
env = await c.cell_pm_complete(
pm_id, parent_id, "reviewed cell scope and merge ready"
)
body = env.as_dict()
assert body["error"] == "tracing_gap"
# Improvement: non-terminal subtask must be named.
assert str(sub_id) in body["remediate"]
task_svc.cell_pm_complete.assert_not_awaited()
@pytest.mark.asyncio
async def test_cell_pm_complete_allows_when_all_terminal() -> None:
pm_id = uuid4()
parent_id = uuid4()
t = MagicMock(
id=parent_id,
status="awaiting_pm_review",
assigned_to=pm_id,
pr_number=10,
team="backend",
branch_name="feature/backend/abc",
parent_task_id=None,
)
after = MagicMock(**{**t.__dict__, "status": "completed"})
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.all_subtasks_terminal.return_value = True
task_svc.get_subtasks.return_value = []
task_svc.cell_pm_complete.return_value = after
journal_svc = AsyncMock()
journal_svc.has_decision_for_task.return_value = True
journal_svc.latest_decision_at.return_value = datetime.now(UTC)
journal_svc.has_reflect_for_task.return_value = True
git_svc = AsyncMock()
git_svc.pr_merge.return_value = {"merge_commit_sha": "abc"}
deps = _make_deps(task=task_svc, journal=journal_svc, git=git_svc)
c = Choreographer(deps)
env = await c.cell_pm_complete(pm_id, parent_id, "cell scope reviewed and approved")
assert env.error is None
task_svc.cell_pm_complete.assert_awaited_once()
@pytest.mark.asyncio
async def test_cell_pm_complete_allows_decision_without_separate_reflect() -> None:
"""A PM with a fresh decision but NO separate reflect can still complete —
the decision documents the close, so the reflect gate no longer loops
weak-model PMs into respawn churn."""
pm_id = uuid4()
parent_id = uuid4()
t = MagicMock(
id=parent_id,
status="awaiting_pm_review",
assigned_to=pm_id,
pr_number=10,
team="backend",
branch_name="feature/backend/abc",
parent_task_id=None,
)
after = MagicMock(**{**t.__dict__, "status": "completed"})
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.all_subtasks_terminal.return_value = True
task_svc.get_subtasks.return_value = []
task_svc.cell_pm_complete.return_value = after
journal_svc = AsyncMock()
journal_svc.has_decision_for_task.return_value = True
journal_svc.latest_decision_at.return_value = datetime.now(UTC)
journal_svc.has_reflect_for_task.return_value = False # no separate reflect
git_svc = AsyncMock()
git_svc.pr_merge.return_value = {"merge_commit_sha": "abc"}
deps = _make_deps(task=task_svc, journal=journal_svc, git=git_svc)
c = Choreographer(deps)
env = await c.cell_pm_complete(pm_id, parent_id, "cell scope reviewed and approved")
assert env.error is None
task_svc.cell_pm_complete.assert_awaited_once()
# ---------------------------------------------------------------------------
# main_pm_complete subtask gate (root-task case)
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_main_pm_complete_blocks_when_subtask_pending() -> None:
pm_id = uuid4()
root_id = uuid4()
sub_id = uuid4()
t = MagicMock(
id=root_id,
status="awaiting_pm_review",
assigned_to=pm_id,
parent_task_id=None,
pr_number=10,
team="backend",
branch_name="feature/backend/abc",
)
sub = MagicMock(id=sub_id, status="in_progress", title="Subtask still active")
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.all_subtasks_terminal.return_value = False
task_svc.get_subtasks.return_value = [sub]
journal_svc = AsyncMock()
journal_svc.has_decision_for_task.return_value = True
journal_svc.latest_decision_at.return_value = datetime.now(UTC)
journal_svc.has_reflect_for_task.return_value = True
deps = _make_deps(task=task_svc, journal=journal_svc)
c = Choreographer(deps)
env = await c.main_pm_complete(
pm_id, root_id, "root scope ready to ship to production"
)
body = env.as_dict()
assert body["error"] == "tracing_gap"
assert str(sub_id) in body["remediate"]
task_svc.escalate_to_ceo.assert_not_awaited()
# ---------------------------------------------------------------------------
# submit_up subtask gate (cell PM bubbling up to main PM)
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_submit_up_blocks_when_subtask_pending() -> None:
pm_id = uuid4()
parent_id = uuid4()
sub_id = uuid4()
t = MagicMock(
id=parent_id,
status="in_progress",
assigned_to=pm_id,
branch_name="feature/backend/abc",
team="backend",
)
sub = MagicMock(id=sub_id, status="paused", title="Paused subtask")
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.agent_for.return_value = MagicMock(role="cell_pm", team="backend")
task_svc.all_subtasks_terminal.return_value = False
task_svc.get_subtasks.return_value = [sub]
journal_svc = AsyncMock()
journal_svc.has_decision_for_task.return_value = True
journal_svc.latest_decision_at.return_value = datetime.now(UTC)
deps = _make_deps(task=task_svc, journal=journal_svc)
c = Choreographer(deps)
env = await c.submit_up(
pm_id,
parent_id,
"ready for main PM review and merge into master branch",
)
body = env.as_dict()
assert body["error"] == "tracing_gap"
assert str(sub_id) in body["remediate"]
task_svc.submit_pm_review.assert_not_awaited()
# ---------------------------------------------------------------------------
# MegaTask umbrella: no PR assembly + branchless completion
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_submit_root_rejects_batch_umbrella() -> None:
"""A MegaTask umbrella assembles no PR of its own — submit_root must
hard-reject it (each root-subtask PRs itself) so it never enters the
in-path review gate."""
pm_id = uuid4()
umbrella_id = uuid4()
t = MagicMock(
id=umbrella_id,
status="in_progress",
assigned_to=pm_id,
parent_task_id=None,
batch_id=uuid4(),
branch_name=None,
team="main_pm",
)
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.agent_for.return_value = MagicMock(role="main_pm")
deps = _make_deps(task=task_svc)
c = Choreographer(deps)
env = await c.submit_root(pm_id, umbrella_id, "all root-subtasks shipped")
body = env.as_dict()
assert body["error"] == "invalid_state"
assert "no PR" in body["message"]
assert "complete(" in body["remediate"]
@pytest.mark.asyncio
async def test_main_pm_complete_allows_batch_umbrella_from_in_progress() -> None:
"""A MegaTask umbrella is branchless: with every root-subtask terminal it
completes straight from in_progress (no submit_root / PR), walking to
awaiting_pm_review and escalating to the CEO — the PR requirement is waived."""
pm_id = uuid4()
umbrella_id = uuid4()
t = MagicMock(
id=umbrella_id,
status="in_progress",
assigned_to=pm_id,
parent_task_id=None,
batch_id=uuid4(),
branch_name=None,
team="main_pm",
)
escalated = MagicMock(**{**t.__dict__, "status": "awaiting_ceo_approval"})
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.all_subtasks_terminal.return_value = True
task_svc.get_subtasks.return_value = []
task_svc.submit_pm_review.return_value = MagicMock(status="awaiting_pm_review")
task_svc.escalate_to_ceo.return_value = escalated
journal_svc = AsyncMock()
journal_svc.has_decision_for_task.return_value = True
journal_svc.latest_decision_at.return_value = datetime.now(UTC)
journal_svc.has_reflect_for_task.return_value = True
deps = _make_deps(task=task_svc, journal=journal_svc)
c = Choreographer(deps)
env = await c.main_pm_complete(
pm_id, umbrella_id, "every root-subtask is terminal; MegaTask ready for CEO"
)
assert env.error is None
# Branchless walk in_progress -> awaiting_pm_review, then escalate. No PR.
task_svc.submit_pm_review.assert_awaited_once()
task_svc.escalate_to_ceo.assert_awaited_once()
@pytest.mark.asyncio
async def test_main_pm_complete_handles_escalate_returning_none() -> None:
"""Defense-in-depth: if escalate_to_ceo refuses (returns None), the verb
surfaces an invalid_state rejection instead of dereferencing None."""
pm_id = uuid4()
umbrella_id = uuid4()
t = MagicMock(
id=umbrella_id,
status="in_progress",
assigned_to=pm_id,
parent_task_id=None,
batch_id=uuid4(),
branch_name=None,
team="main_pm",
)
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.all_subtasks_terminal.return_value = True
task_svc.get_subtasks.return_value = []
task_svc.submit_pm_review.return_value = MagicMock(status="awaiting_pm_review")
task_svc.escalate_to_ceo.return_value = None # refused
journal_svc = AsyncMock()
journal_svc.has_decision_for_task.return_value = True
journal_svc.latest_decision_at.return_value = datetime.now(UTC)
journal_svc.has_reflect_for_task.return_value = True
deps = _make_deps(task=task_svc, journal=journal_svc)
c = Choreographer(deps)
env = await c.main_pm_complete(pm_id, umbrella_id, "ready for CEO sign-off")
assert env.error == "invalid_state"
assert "escalate_to_ceo" in env.as_dict()["message"]
@pytest.mark.asyncio
async def test_complete_escalates_batch_umbrella_from_in_progress() -> None:
"""A MegaTask umbrella is branchless by design and sits in ``in_progress``
with no branch/PR; the ``complete`` verb's spec gate
(``source_statuses={AWAITING_PM_REVIEW}``) must NOT reject it — the Main
PM routes through ``main_pm_complete``, walking in_progress ->
awaiting_pm_review -> awaiting_ceo_approval. Calling the ``complete``
ENTRY point (not ``main_pm_complete`` directly) must succeed and escalate
to the CEO; this exercises the real spec gate (``can_invoke_intent`` is
pure)."""
pm_id = uuid4()
umbrella_id = uuid4()
batch_id = uuid4()
t = MagicMock(
id=umbrella_id,
status="in_progress",
assigned_to=pm_id,
parent_task_id=None,
batch_id=batch_id,
branch_name=None,
pr_number=None,
pr_created=False,
pr_url=None,
team="main_pm",
)
after_review = MagicMock(
id=umbrella_id,
status="awaiting_pm_review",
assigned_to=pm_id,
parent_task_id=None,
batch_id=batch_id,
branch_name=None,
team="main_pm",
)
escalated = MagicMock(
id=umbrella_id,
status="awaiting_ceo_approval",
assigned_to=pm_id,
parent_task_id=None,
batch_id=batch_id,
branch_name=None,
team="main_pm",
)
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.agent_for.return_value = MagicMock(role="main_pm", slug="main-pm")
task_svc.all_subtasks_terminal.return_value = True
task_svc.get_subtasks.return_value = []
task_svc.submit_pm_review.return_value = after_review
task_svc.escalate_to_ceo.return_value = escalated
task_svc.reassign = AsyncMock()
journal_svc = AsyncMock()
journal_svc.has_decision_for_task.return_value = True
journal_svc.latest_decision_at.return_value = datetime.now(UTC)
journal_svc.has_reflect_for_task.return_value = True
deps = _make_deps(task=task_svc, journal=journal_svc)
c = Choreographer(deps)
env = await c.complete(
pm_id, umbrella_id, notes="every root-subtask terminal; MegaTask ready for CEO"
)
body = env.as_dict()
assert body.get("error") is None, body
assert body["status"] == "awaiting_ceo_approval"
task_svc.submit_pm_review.assert_awaited_once()
task_svc.escalate_to_ceo.assert_awaited_once()
@pytest.mark.asyncio
async def test_complete_rejects_non_batch_branchless_in_progress_root() -> None:
"""#30 negative pin: the ``_is_umbrella_in_progress`` bypass is gated on a
genuine batch umbrella (``is_batch_umbrella`` = batch_id set AND no parent).
A NON-batch branchless Main-PM root (a product-only delivery root,
``batch_id=None``) sitting in ``in_progress`` must NOT bypass the complete
spec gate — the gate enforces ``source_statuses={AWAITING_PM_REVIEW}`` and
rejects ``in_progress`` with ``invalid_state`` rather than falling through
to ``main_pm_complete``. A regression that loosened the predicate (e.g.
keying on ``branch_name is None`` alone) would let any branchless root
walk past the awaiting_pm_review checkpoint; this pins the tight predicate."""
pm_id = uuid4()
root_id = uuid4()
t = MagicMock(
id=root_id,
status="in_progress",
assigned_to=pm_id,
parent_task_id=None,
batch_id=None, # NOT a batch umbrella -> bypass must NOT fire.
branch_name=None,
pr_number=None,
pr_created=False,
pr_url=None,
team="main_pm",
)
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.agent_for.return_value = MagicMock(role="main_pm", slug="main-pm")
task_svc.all_subtasks_terminal.return_value = True
task_svc.get_subtasks.return_value = []
task_svc.submit_pm_review = AsyncMock()
task_svc.escalate_to_ceo = AsyncMock()
journal_svc = AsyncMock()
journal_svc.has_decision_for_task.return_value = True
journal_svc.latest_decision_at.return_value = datetime.now(UTC)
journal_svc.has_reflect_for_task.return_value = True
deps = _make_deps(task=task_svc, journal=journal_svc)
c = Choreographer(deps)
env = await c.complete(
pm_id, root_id, notes="non-batch branchless root is not awaiting_pm_review"
)
body = env.as_dict()
# The spec gate ran (not bypassed) and rejected the in_progress source status.
assert body.get("error") == "invalid_state", body
# The bypass did NOT fire -> main_pm_complete never ran.
task_svc.submit_pm_review.assert_not_awaited()
task_svc.escalate_to_ceo.assert_not_awaited()