Files
roboco/tests/unit/gateway/test_submit_up_unchanged_pr_guard.py
T
93739a9dca fix(notifications): stop tick-wide row locks + poisoned-session swallows behind the PendingRollbackError 500s (#743)
* fix(notifications): release re-escalation row locks per row; stop swallowing DB errors in notify_get

The re-escalation sweep ran one tick-wide transaction, so each CAS
claim's row lock was held across every remaining delivery until the
single commit — a concurrent mark-read UPDATE on a claimed row starved
into the 60s lock_timeout. The sweep now commits per row (claim commit
releases the lock before delivery and makes the burned slot durable),
re-fetches each row by snapshotted id so one row's rollback can't
expire the rest of the tick, and savepoints each recipient's delivery.

notify_get's bare except swallowed the resulting LockNotAvailableError
into a false "notification not found" and returned a poisoned session
to the commit-at-send middleware, which blew up with
PendingRollbackError; it now catches only the two domain outcomes.

defer_after_commit's listeners fire on SAVEPOINT release too, which
would have drained deferred telegram/bus work before real durability —
they now skip savepoint boundaries via get_nested_transaction() (the
root get_transaction() is non-None inside the listener even at a real
commit). acknowledge_for_recipient's Redis dedup-clear moved before the
flush so the row lock never spans a Redis round-trip. The five
best-effort CEO-notify swallows that persist notification rows are
savepointed.

* fix(services): contain swallowed best-effort DB write failures instead of poisoning the session

Sweep of the same class as the notify_get incident: broad
except-Exception handlers that swallow a failure whose try-body writes
through the shared session leave the session rollback-pending, and the
verb/request then dies later with PendingRollbackError at
commit-at-send. Confirmed-dangerous sites now run the write inside a
savepoint (safe since defer_after_commit skips savepoint boundaries):
ceo_approve's verified-stamp, completion/pitch/postmortem-style CEO
notifies, _inherit_upstream_base, _link_commit_to_task (covers every
commit route), board-program LEARN records, the QA/PR-gate/PM-merge
verified-stamps, and the documenter->PM handoff.
_ack_pending_wake_notifications gets the same treatment so a wake-ack
failure can't fail the A2A read. telegram_inbound's per-update loop and
intake confirm roll back explicitly instead (their success paths commit
mid-flow, so a savepoint doesn't fit).

A swallowed savepoint rollback fully expires any ORM object mutated
inside the block, and the next attribute read raises MissingGreenlet —
strictly worse than the original bug. The two paths that keep using the
object after the swallow (doc handoff's envelope build, base
inheritance's claim continuation) refresh it in the except path;
regression tests run against a real session and were verified to fail
with the refresh reverted.

* test: shape mocked session.execute results so sync accessors stop leaking unawaited coroutines

An AsyncMock's auto-created children are themselves AsyncMock, so
production code that correctly awaits session.execute() and then calls
sync accessors (.scalars().all(), .scalar_one_or_none()) on the result
was silently collecting unawaited coroutines in 22 test files — 80
RuntimeWarnings per unit run, and in test_flow_soup_guard one mock
raised a real TypeError that a coincidentally-matching invalid_state
envelope masked. Each affected fixture now returns a plain MagicMock
shaped like a real Result. Zero AsyncMock warnings remain.

* docs: document per-row sweep commits and the savepoint/refresh containment pattern

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-07-30 16:35:42 +02:00

190 lines
7.8 KiB
Python

"""The unchanged-PR re-submit loop-stopper, applied to ``submit_up`` (cell→root).
Refuses to re-open the gate when the cell PR's head SHA equals the SHA the
last ``pr_fail`` recorded (no new dev work landed); ambiguous cases FAIL OPEN.
"""
from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from unittest.mock import AsyncMock, MagicMock
from uuid import uuid4
import pytest
from roboco.services.gateway.choreographer import Choreographer, ChoreographerDeps
SHA_OLD = "aaaa1111bbbb2222cccc3333dddd4444eeee5555"
SHA_NEW = "9999888877776666555544443333222211110000"
def _make_deps(**overrides: Any) -> ChoreographerDeps:
base: dict[str, Any] = {
"task": AsyncMock(),
"work_session": AsyncMock(),
"git": AsyncMock(),
"a2a": AsyncMock(),
"journal": AsyncMock(),
"audit": AsyncMock(),
"evidence_repo": AsyncMock(),
}
base.update(overrides)
base["journal"].has_decision_for_task.return_value = True
base["journal"].latest_decision_at.return_value = datetime.now(UTC)
base["journal"].has_reflect_for_task.return_value = True
return ChoreographerDeps(**base)
def _resubmit_cell(
*,
notes_structured: dict[str, Any] | None,
pr_number: int | None = 132,
) -> tuple[Choreographer, Any, Any]:
"""A cell-PM task re-submitted from ``in_progress`` after a ``pr_fail``.
Mirrors a live cell re-submit: the cell task is back in ``in_progress``
(re-claimed out of ``needs_revision``), carries the prior ``pr_fail``
verdict in ``notes_structured.pr_review``, and the cell→root PR is still
open. The ``_submit_up_guard`` preflight is satisfied (owned by the cell
PM, journal decision, subtasks terminal, branch present, notes long
enough) so the unchanged-PR gate is the thing under test.
"""
cell_pm_id = uuid4()
cell_task_id = uuid4()
in_prog = MagicMock(
id=cell_task_id,
status="in_progress",
assigned_to=cell_pm_id,
pr_number=pr_number,
branch_name="feature/backend/cell-task",
parent_task_id=uuid4(),
batch_id=None,
team="backend",
notes_structured=notes_structured,
)
gated = MagicMock(**{**in_prog.__dict__, "status": "awaiting_pr_review"})
task_svc = AsyncMock()
task_svc.get.return_value = in_prog
task_svc.submit_for_review.return_value = gated
task_svc.all_subtasks_terminal.return_value = True
task_svc.uncovered_parent_acceptance_criteria.return_value = []
task_svc.agent_for.return_value = MagicMock(role="cell_pm", team="backend")
task_svc.session.begin_nested = MagicMock(
return_value=MagicMock(__aenter__=AsyncMock(), __aexit__=AsyncMock())
)
# Findings-ledger reads (ReviewFindingsRepository.list_for_task) go
# through session.execute — an unconfigured AsyncMock's awaited result
# is itself an AsyncMock, so a plain sync `.scalars()` call on it leaks
# an unawaited coroutine. Empty scalars result (no findings).
task_svc.session.execute = AsyncMock(
return_value=MagicMock(
scalars=MagicMock(return_value=MagicMock(all=MagicMock(return_value=[])))
)
)
c = Choreographer(_make_deps(task=task_svc, git=AsyncMock()))
cc: Any = c
cc._project_slug_for = AsyncMock(return_value="proj-slug")
return c, cell_pm_id, cell_task_id
@pytest.mark.asyncio
async def test_submit_up_refuses_unchanged_pr_after_pr_fail() -> None:
"""The loop-stopper still holds past the one-shot exemption: prior
pr_fail stamped head SHA X, the cell PR head is still X (no new dev work
on the cell branch). The findings ledger here fail-opens to "nothing
open" (mock session, no real query) — the same signal
``_check_submit_up_gates`` upstream already reads for FINDINGS_ADDRESSED
— so the first resubmit at this head is the one-shot exemption (see
test_resubmit_unchanged_head_exemption.py for full exemption coverage);
a second resubmit at the SAME head refuses, so the loop still can't run
forever."""
c, cell_pm_id, cell_task_id = _resubmit_cell(
notes_structured={
"pr_review": {"verdict": "failed", "head_sha": SHA_OLD, "summary": "..."}
}
)
c.git.get_pr_head_sha = AsyncMock(return_value=SHA_OLD)
first = await c.submit_up(
cell_pm_id, cell_task_id, notes="re-submitting the cell after the fix"
)
assert first.error is None, first.as_dict()
env = await c.submit_up(
cell_pm_id, cell_task_id, notes="re-submitting again; still unchanged"
)
assert env.error is not None, env.as_dict()
assert env.error == "invalid_state", env.as_dict()
assert "unchanged" in (env.message or "").lower()
remediate = env.remediate or ""
assert "submit_up" in remediate
# The second attempt's PR was NOT re-opened / re-pushed — the runner ran
# only for the first (exempted) call.
c.task.submit_for_review.assert_awaited_once()
@pytest.mark.asyncio
async def test_submit_up_allows_after_cell_branch_advanced() -> None:
"""A different current head SHA ⇒ dev work landed on the cell branch ⇒ the
diff changed ⇒ allow the re-submit into the gate."""
c, cell_pm_id, cell_task_id = _resubmit_cell(
notes_structured={
"pr_review": {"verdict": "failed", "head_sha": SHA_OLD, "summary": "..."}
}
)
c.git.get_pr_head_sha = AsyncMock(return_value=SHA_NEW)
env = await c.submit_up(
cell_pm_id, cell_task_id, notes="re-submitting after the dev re-assembly"
)
assert env.error is None, env.as_dict()
assert env.status == "awaiting_pr_review"
c.task.submit_for_review.assert_awaited_once()
@pytest.mark.asyncio
async def test_submit_up_fail_open_when_no_prior_pr_fail_verdict() -> None:
"""No pr_review (first submit) ⇒ nothing to compare ⇒ allow."""
c, cell_pm_id, cell_task_id = _resubmit_cell(notes_structured=None)
env = await c.submit_up(
cell_pm_id, cell_task_id, notes="first cell submit; nothing to compare yet"
)
assert env.error is None, env.as_dict()
assert env.status == "awaiting_pr_review"
# ---------------------------------------------------------------------------
# submit_for_review returns None when the task raced out of in_progress after
# create_pr already opened the cell→root PR; the remediate must tell the PM the
# PR is open so the orphan is recoverable via create_pr's idempotent re-issue.
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_submit_up_none_remediate_names_the_already_open_pr() -> None:
"""submit_for_review returns None (raced out of in_progress) AFTER create_pr
already opened the cell→root PR. The remediate must name the open PR and point
the PM at re-fetching + reconciling, not the misleading 'PR ready' hint."""
c, cell_pm_id, cell_task_id = _resubmit_cell(notes_structured=None)
# A concurrent transition (stale-heartbeat reaper unclaim, or a racing
# i_am_blocked) moved the task out of in_progress between the precondition
# gate and the runner's composed action → submit_for_review returns None.
c.task.submit_for_review.return_value = None
env = await c.submit_up(
cell_pm_id, cell_task_id, notes="submitting the assembled cell scope"
)
assert env.error == "invalid_state", env.as_dict()
remediate = (env.remediate or "").lower()
# The create_pr pre-side-effect ran BEFORE the None transition, so the PR
# is open on GitHub — the remediate must say so. The old 'PR ready' hint
# hid this.
assert "pr" in remediate and "open" in remediate, env.as_dict()
# The misleading old hint is gone.
assert "pr ready" not in remediate, env.as_dict()
# And the PR really was opened (pre-side-effect ran before the None).
c.git.create_pr.assert_awaited_once()