Files
roboco/tests/unit/gateway/test_choreographer_doc.py
T
93739a9dca fix(notifications): stop tick-wide row locks + poisoned-session swallows behind the PendingRollbackError 500s (#743)
* fix(notifications): release re-escalation row locks per row; stop swallowing DB errors in notify_get

The re-escalation sweep ran one tick-wide transaction, so each CAS
claim's row lock was held across every remaining delivery until the
single commit — a concurrent mark-read UPDATE on a claimed row starved
into the 60s lock_timeout. The sweep now commits per row (claim commit
releases the lock before delivery and makes the burned slot durable),
re-fetches each row by snapshotted id so one row's rollback can't
expire the rest of the tick, and savepoints each recipient's delivery.

notify_get's bare except swallowed the resulting LockNotAvailableError
into a false "notification not found" and returned a poisoned session
to the commit-at-send middleware, which blew up with
PendingRollbackError; it now catches only the two domain outcomes.

defer_after_commit's listeners fire on SAVEPOINT release too, which
would have drained deferred telegram/bus work before real durability —
they now skip savepoint boundaries via get_nested_transaction() (the
root get_transaction() is non-None inside the listener even at a real
commit). acknowledge_for_recipient's Redis dedup-clear moved before the
flush so the row lock never spans a Redis round-trip. The five
best-effort CEO-notify swallows that persist notification rows are
savepointed.

* fix(services): contain swallowed best-effort DB write failures instead of poisoning the session

Sweep of the same class as the notify_get incident: broad
except-Exception handlers that swallow a failure whose try-body writes
through the shared session leave the session rollback-pending, and the
verb/request then dies later with PendingRollbackError at
commit-at-send. Confirmed-dangerous sites now run the write inside a
savepoint (safe since defer_after_commit skips savepoint boundaries):
ceo_approve's verified-stamp, completion/pitch/postmortem-style CEO
notifies, _inherit_upstream_base, _link_commit_to_task (covers every
commit route), board-program LEARN records, the QA/PR-gate/PM-merge
verified-stamps, and the documenter->PM handoff.
_ack_pending_wake_notifications gets the same treatment so a wake-ack
failure can't fail the A2A read. telegram_inbound's per-update loop and
intake confirm roll back explicitly instead (their success paths commit
mid-flow, so a savepoint doesn't fit).

A swallowed savepoint rollback fully expires any ORM object mutated
inside the block, and the next attribute read raises MissingGreenlet —
strictly worse than the original bug. The two paths that keep using the
object after the swallow (doc handoff's envelope build, base
inheritance's claim continuation) refresh it in the except path;
regression tests run against a real session and were verified to fail
with the refresh reverted.

* test: shape mocked session.execute results so sync accessors stop leaking unawaited coroutines

An AsyncMock's auto-created children are themselves AsyncMock, so
production code that correctly awaits session.execute() and then calls
sync accessors (.scalars().all(), .scalar_one_or_none()) on the result
was silently collecting unawaited coroutines in 22 test files — 80
RuntimeWarnings per unit run, and in test_flow_soup_guard one mock
raised a real TypeError that a coincidentally-matching invalid_state
envelope masked. Each affected fixture now returns a plain MagicMock
shaped like a real Result. Zero AsyncMock warnings remain.

* docs: document per-row sweep commits and the savepoint/refresh containment pattern

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-07-30 16:35:42 +02:00

391 lines
14 KiB
Python

"""Tests for Documenter Choreographer methods."""
from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from unittest.mock import AsyncMock, MagicMock, patch
from uuid import uuid4
import pytest
from roboco.services.gateway.choreographer import Choreographer, ChoreographerDeps
def _make_deps(**overrides: Any) -> ChoreographerDeps:
base = {
"task": AsyncMock(),
"work_session": AsyncMock(),
"git": AsyncMock(),
"a2a": AsyncMock(),
"journal": AsyncMock(),
"audit": AsyncMock(),
"evidence_repo": AsyncMock(),
}
base.update(overrides)
# Findings-ledger reads (ReviewFindingsRepository.list_for_task) go
# through session.execute — an unconfigured AsyncMock's awaited result
# is itself an AsyncMock, so a plain sync `.scalars()` call on it leaks
# an unawaited coroutine. Empty scalars result (no findings).
base["task"].session.execute = AsyncMock(
return_value=MagicMock(
scalars=MagicMock(return_value=MagicMock(all=MagicMock(return_value=[])))
)
)
repo = base["evidence_repo"]
for method in (
"list_unread_a2a",
"list_unread_mentions",
"list_pending_notifications",
"task_metadata_gaps",
"recent_team_activity",
"blockers_in_lane",
"journal_highlights_for_task",
):
getattr(repo, method).return_value = []
# C8: default-fresh journal:decision so PM-decision gate passes.
# Tests that exercise the gate boundary stub their own value.
# The check matches MagicMock and AsyncMock (the two default sentinel
# types pytest's unittest.mock leaves on un-stubbed return_values).
_ldef = base["journal"].latest_decision_at.return_value
if type(_ldef).__name__ in ("MagicMock", "AsyncMock"):
base["journal"].latest_decision_at.return_value = datetime.now(UTC)
return ChoreographerDeps(**base)
@pytest.mark.asyncio
async def test_claim_doc_task_returns_evidence() -> None:
doc_id = uuid4()
task_id = uuid4()
t = MagicMock(
id=task_id,
status="awaiting_documentation",
assigned_to=None,
pr_number=8,
pr_url="https://github.com/x/y/pull/8",
commits=[{"sha": "abc", "message": "feat: x"}],
team="backend",
branch_name="feature/backend/abc--def",
work_session_id=uuid4(),
documents=[],
dev_notes="",
acceptance_criteria=[],
acceptance_criteria_status=[],
)
after = MagicMock(**{**t.__dict__, "assigned_to": doc_id})
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.agent_for.return_value = MagicMock(role="documenter", team="backend")
task_svc.list_in_progress_for_agent.return_value = []
task_svc.list_paused_for_agent.return_value = []
task_svc.doc_claim.return_value = after
work_svc = AsyncMock()
git_svc = AsyncMock()
git_svc.diff.return_value = "+++ diff"
git_svc.list_changed_files.return_value = ["README.md"]
deps = _make_deps(task=task_svc, work_session=work_svc, git=git_svc)
c = Choreographer(deps)
env = await c.claim_doc_task(doc_id, task_id)
body = env.as_dict()
assert body["error"] is None
assert body["evidence"]["pr_url"] == "https://github.com/x/y/pull/8"
assert "README.md" in body["evidence"]["files_changed"]
@pytest.mark.asyncio
async def test_claim_doc_task_blocks_wrong_state() -> None:
doc_id = uuid4()
task_id = uuid4()
t = MagicMock(
id=task_id,
status="in_progress",
task_type="code",
team="backend",
quick_context=None,
)
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.agent_for.return_value = MagicMock(
id=doc_id, role="documenter", team="backend", slug=None
)
deps = _make_deps(task=task_svc)
c = Choreographer(deps)
env = await c.claim_doc_task(doc_id, task_id)
body = env.as_dict()
# Spec rejects: in_progress is not in `claim` action's source_statuses
# (PENDING, NEEDS_REVISION, AWAITING_QA, AWAITING_DOCUMENTATION).
assert body["error"] == "invalid_state"
@pytest.mark.asyncio
async def test_claim_doc_task_not_found() -> None:
doc_id = uuid4()
task_id = uuid4()
task_svc = AsyncMock()
task_svc.get.return_value = None
deps = _make_deps(task=task_svc)
c = Choreographer(deps)
env = await c.claim_doc_task(doc_id, task_id)
assert env.as_dict()["error"] == "not_found"
def _doc_owned_task(task_id: Any, doc_id: Any, **overrides: Any) -> MagicMock:
"""Build a doc-owned awaiting_documentation task fixture for the spec gate.
Status defaults to awaiting_documentation (which matches docs_complete's
spec source_statuses). task_type / team / quick_context defaulted so
the spec gate evaluates against real values.
"""
base = {
"id": task_id,
"status": "awaiting_documentation",
"task_type": "code",
"team": "backend",
"assigned_to": doc_id,
"quick_context": None,
}
base.update(overrides)
return MagicMock(**base)
def _doc_agent_mock(doc_id: Any) -> MagicMock:
return MagicMock(id=doc_id, role="documenter", team="backend", slug=None)
@pytest.mark.asyncio
async def test_i_documented_requires_min_notes() -> None:
doc_id = uuid4()
task_id = uuid4()
t = _doc_owned_task(task_id, doc_id)
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.agent_for.return_value = _doc_agent_mock(doc_id)
journal_svc = AsyncMock()
journal_svc.has_reflect_for_task.return_value = True
deps = _make_deps(task=task_svc, journal=journal_svc)
c = Choreographer(deps)
env = await c.i_documented(doc_id, task_id, notes="wrote the docs", files=["a.md"])
body = env.as_dict()
assert body["error"] == "tracing_gap"
assert "docs_notes>=min" in body["missing"]
@pytest.mark.asyncio
async def test_i_documented_task_not_found() -> None:
"""Line 105: task missing → not_found via _emit_rejection."""
doc_id = uuid4()
task_id = uuid4()
task_svc = AsyncMock()
task_svc.get.return_value = None
deps = _make_deps(task=task_svc)
c = Choreographer(deps)
env = await c.i_documented(doc_id, task_id, notes="x" * 30, files=["docs.md"])
assert env.as_dict()["error"] == "not_found"
@pytest.mark.asyncio
async def test_i_documented_requires_files() -> None:
doc_id = uuid4()
task_id = uuid4()
t = _doc_owned_task(task_id, doc_id)
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.agent_for.return_value = _doc_agent_mock(doc_id)
journal_svc = AsyncMock()
journal_svc.has_reflect_for_task.return_value = True
deps = _make_deps(task=task_svc, journal=journal_svc)
c = Choreographer(deps)
notes = "Wrote backend/guides/feature-x.md with usage examples."
env = await c.i_documented(doc_id, task_id, notes=notes, files=[])
body = env.as_dict()
assert body["error"] == "tracing_gap"
assert "docs_files_non_empty" in body["missing"]
@pytest.mark.asyncio
async def test_i_documented_succeeds_and_transitions() -> None:
doc_id = uuid4()
task_id = uuid4()
t = _doc_owned_task(task_id, doc_id)
after = MagicMock(
id=task_id,
status="awaiting_pm_review",
assigned_to=doc_id,
team="backend",
)
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.agent_for.return_value = _doc_agent_mock(doc_id)
task_svc.docs_complete.return_value = after
task_svc.cell_pm_for_team.return_value = MagicMock(id=uuid4())
task_svc.session = MagicMock()
task_svc.session.flush = AsyncMock()
task_svc.session.refresh = AsyncMock()
task_svc.session.begin_nested = MagicMock(
return_value=MagicMock(
__aenter__=AsyncMock(return_value=None),
__aexit__=AsyncMock(return_value=False),
)
)
a2a_svc = AsyncMock()
journal_svc = AsyncMock()
journal_svc.has_reflect_for_task.return_value = True
deps = _make_deps(task=task_svc, a2a=a2a_svc, journal=journal_svc)
c = Choreographer(deps)
notes = "Wrote backend/guides/feature-x.md with usage examples and config notes."
files = ["backend/guides/feature-x.md"]
env = await c.i_documented(doc_id, task_id, notes=notes, files=files)
assert env.error is None
assert env.status == "awaiting_pm_review"
task_svc.docs_complete.assert_awaited_once()
a2a_svc.send.assert_awaited_once()
def _doc_success_task_svc(task_id: Any, doc_id: Any) -> AsyncMock:
"""A task service stubbed for a passing i_documented (awaiting_pm_review)."""
t = _doc_owned_task(task_id, doc_id)
after = MagicMock(
id=task_id,
status="awaiting_pm_review",
assigned_to=doc_id,
team="backend",
)
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.agent_for.return_value = _doc_agent_mock(doc_id)
task_svc.docs_complete.return_value = after
task_svc.cell_pm_for_team.return_value = MagicMock(id=uuid4())
task_svc.session = MagicMock()
task_svc.session.flush = AsyncMock()
task_svc.session.refresh = AsyncMock()
task_svc.session.begin_nested = MagicMock(
return_value=MagicMock(
__aenter__=AsyncMock(return_value=None),
__aexit__=AsyncMock(return_value=False),
)
)
return task_svc
@pytest.mark.asyncio
async def test_i_documented_pushes_doc_commit() -> None:
"""The documenter's doc commit must be PUSHED so it reaches the open PR.
The PR is already open by the time docs are written; without a push the
doc commit lives only in the documenter's clone and the PM merges a PR
that excludes the docs. Mirrors the developer's _ensure_branch_pushed.
"""
doc_id = uuid4()
task_id = uuid4()
task_svc = _doc_success_task_svc(task_id, doc_id)
git_svc = AsyncMock()
git_svc.push_task_branch.return_value = 1
journal_svc = AsyncMock()
journal_svc.has_reflect_for_task.return_value = True
deps = _make_deps(task=task_svc, git=git_svc, journal=journal_svc)
c = Choreographer(deps)
notes = "Wrote backend/guides/feature-x.md with usage examples and config."
env = await c.i_documented(
doc_id, task_id, notes=notes, files=["backend/guides/feature-x.md"]
)
assert env.error is None
assert env.status == "awaiting_pm_review"
git_svc.push_task_branch.assert_awaited_once_with(doc_id, task_id)
@pytest.mark.asyncio
async def test_i_documented_push_failure_holds_task() -> None:
"""A push failure must NOT transition — the docs are local-only, so the
documenter stays in awaiting_documentation and retries (no silent drop)."""
doc_id = uuid4()
task_id = uuid4()
task_svc = _doc_success_task_svc(task_id, doc_id)
git_svc = AsyncMock()
git_svc.push_task_branch.side_effect = RuntimeError("push rejected")
journal_svc = AsyncMock()
journal_svc.has_reflect_for_task.return_value = True
deps = _make_deps(task=task_svc, git=git_svc, journal=journal_svc)
c = Choreographer(deps)
notes = "Wrote backend/guides/feature-x.md with usage examples and config."
env = await c.i_documented(
doc_id, task_id, notes=notes, files=["backend/guides/feature-x.md"]
)
assert env.as_dict()["error"] == "invalid_state"
task_svc.docs_complete.assert_not_awaited()
@pytest.mark.asyncio
async def test_i_documented_not_assigned_returns_not_authorized() -> None:
doc_id = uuid4()
other = uuid4()
task_id = uuid4()
t = _doc_owned_task(task_id, other)
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.agent_for.return_value = _doc_agent_mock(doc_id)
deps = _make_deps(task=task_svc)
c = Choreographer(deps)
env = await c.i_documented(doc_id, task_id, notes="x" * 50, files=["x.md"])
assert env.as_dict()["error"] == "not_authorized"
# ---------------------------------------------------------------------------
# H6: i_documented survives handoff failure
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_i_documented_survives_handoff_failure() -> None:
"""_handoff_to_cell_pm throws after the runner commits the docs-complete
transition. The verb must NOT 500 — the transition is committed; the
side-effect failure is logged and the envelope carries a warning."""
doc_id = uuid4()
task_id = uuid4()
t = _doc_owned_task(task_id, doc_id)
after = MagicMock(
id=task_id,
status="awaiting_pm_review",
assigned_to=doc_id,
team="backend",
)
task_svc = AsyncMock()
task_svc.get.return_value = t
task_svc.agent_for.return_value = _doc_agent_mock(doc_id)
task_svc.docs_complete.return_value = after
task_svc.cell_pm_for_team.return_value = MagicMock(id=uuid4())
task_svc.session = MagicMock()
task_svc.session.flush = AsyncMock()
task_svc.session.refresh = AsyncMock()
task_svc.session.begin_nested = MagicMock(
return_value=MagicMock(
__aenter__=AsyncMock(return_value=None),
__aexit__=AsyncMock(return_value=False),
)
)
a2a_svc = AsyncMock()
journal_svc = AsyncMock()
journal_svc.has_reflect_for_task.return_value = True
deps = _make_deps(task=task_svc, a2a=a2a_svc, journal=journal_svc)
c = Choreographer(deps)
notes = "Wrote backend/guides/feature-x.md with usage examples and config notes."
files = ["backend/guides/feature-x.md"]
with patch.object(
c,
"_handoff_to_cell_pm",
new=AsyncMock(side_effect=RuntimeError("handoff down")),
):
env = await c.i_documented(doc_id, task_id, notes=notes, files=files)
body = env.as_dict()
assert body.get("error") is None, body
assert body.get("warning") is not None
assert "handoff" in body["warning"].lower()