fix(notifications): stop tick-wide row locks + poisoned-session swallows behind the PendingRollbackError 500s (#743)

* fix(notifications): release re-escalation row locks per row; stop swallowing DB errors in notify_get

The re-escalation sweep ran one tick-wide transaction, so each CAS
claim's row lock was held across every remaining delivery until the
single commit — a concurrent mark-read UPDATE on a claimed row starved
into the 60s lock_timeout. The sweep now commits per row (claim commit
releases the lock before delivery and makes the burned slot durable),
re-fetches each row by snapshotted id so one row's rollback can't
expire the rest of the tick, and savepoints each recipient's delivery.

notify_get's bare except swallowed the resulting LockNotAvailableError
into a false "notification not found" and returned a poisoned session
to the commit-at-send middleware, which blew up with
PendingRollbackError; it now catches only the two domain outcomes.

defer_after_commit's listeners fire on SAVEPOINT release too, which
would have drained deferred telegram/bus work before real durability —
they now skip savepoint boundaries via get_nested_transaction() (the
root get_transaction() is non-None inside the listener even at a real
commit). acknowledge_for_recipient's Redis dedup-clear moved before the
flush so the row lock never spans a Redis round-trip. The five
best-effort CEO-notify swallows that persist notification rows are
savepointed.

* fix(services): contain swallowed best-effort DB write failures instead of poisoning the session

Sweep of the same class as the notify_get incident: broad
except-Exception handlers that swallow a failure whose try-body writes
through the shared session leave the session rollback-pending, and the
verb/request then dies later with PendingRollbackError at
commit-at-send. Confirmed-dangerous sites now run the write inside a
savepoint (safe since defer_after_commit skips savepoint boundaries):
ceo_approve's verified-stamp, completion/pitch/postmortem-style CEO
notifies, _inherit_upstream_base, _link_commit_to_task (covers every
commit route), board-program LEARN records, the QA/PR-gate/PM-merge
verified-stamps, and the documenter->PM handoff.
_ack_pending_wake_notifications gets the same treatment so a wake-ack
failure can't fail the A2A read. telegram_inbound's per-update loop and
intake confirm roll back explicitly instead (their success paths commit
mid-flow, so a savepoint doesn't fit).

A swallowed savepoint rollback fully expires any ORM object mutated
inside the block, and the next attribute read raises MissingGreenlet —
strictly worse than the original bug. The two paths that keep using the
object after the swallow (doc handoff's envelope build, base
inheritance's claim continuation) refresh it in the except path;
regression tests run against a real session and were verified to fail
with the refresh reverted.

* test: shape mocked session.execute results so sync accessors stop leaking unawaited coroutines

An AsyncMock's auto-created children are themselves AsyncMock, so
production code that correctly awaits session.execute() and then calls
sync accessors (.scalars().all(), .scalar_one_or_none()) on the result
was silently collecting unawaited coroutines in 22 test files — 80
RuntimeWarnings per unit run, and in test_flow_soup_guard one mock
raised a real TypeError that a coincidentally-matching invalid_state
envelope masked. Each affected fixture now returns a plain MagicMock
shaped like a real Result. Zero AsyncMock warnings remain.

* docs: document per-row sweep commits and the savepoint/refresh containment pattern

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>

tests/integration/test_lifecycle_real_db.py

@@ -20,7 +20,7 @@ from __future__ import annotations
 
 from datetime import UTC, datetime
 from typing import TYPE_CHECKING, Any
-from unittest.mock import AsyncMock
+from unittest.mock import AsyncMock, MagicMock, patch
 from uuid import UUID, uuid4
 
 import pytest
@@ -653,6 +653,142 @@ async def test_doc_path(
     del cell_pm_agent  # asserted indirectly via cell_pm_for_team.
 
 
+@pytest.mark.asyncio
+async def test_doc_path_survives_handoff_failure_real_session(
+    db_session: AsyncSession, lifecycle_setup: dict[str, Any]
+) -> None:
+    """Round-2 regression (#doc-savepoint-expiry): _handoff_to_cell_pm's
+    `reassign()` mutates + flushes `t` inside the `begin_nested()` savepoint,
+    then `a2a.send` raises. On a REAL AsyncSession the savepoint rollback
+    fully expires every attribute of `t` — reading `t.status` /
+    `with_introspection(task=t, ...)` right after the except without
+    refreshing first raises `MissingGreenlet`, which propagates uncaught and
+    rolls back the WHOLE request (discarding the docs_complete transition the
+    warning claims survived). The equivalent unit test
+    (`test_i_documented_survives_handoff_failure`) mocks the session, so it
+    cannot reproduce this — a mock has no real ORM expiry semantics.
+    """
+    task = lifecycle_setup["task"]
+    doc_agent = lifecycle_setup["doc_agent"]
+
+    task.status = TaskStatus.AWAITING_DOCUMENTATION
+    task.pr_number = _PR_NUMBER
+    task.pr_url = _PR_URL
+    task.pr_created = True
+    task.qa_verified = True
+    task.assigned_to = None
+    task.commits = [
+        {"sha": uuid4().hex[:40], "message": "feat: /healthz", "task_id": str(task.id)}
+    ]
+    await db_session.flush()
+
+    task_service = TaskService(db_session)
+    # reassign() (a real write) runs BEFORE a2a.send inside _handoff_to_cell_pm
+    # — this raises only after that mutation has already flushed.
+    broken_a2a = AsyncMock()
+    broken_a2a.send.side_effect = RuntimeError("a2a down")
+    deps = ChoreographerDeps(
+        task=task_service,
+        work_session=_mock_work_session(),
+        git=_StubGit(db_session, task),
+        a2a=broken_a2a,
+        journal=_mock_journal_with_reflect(),
+        audit=AsyncMock(),
+        evidence_repo=_mock_evidence_repo(),
+    )
+    c = Choreographer(deps)
+
+    env = await c.claim_doc_task(doc_agent.id, task.id)
+    assert env.error is None, f"claim_doc_task failed: {env.message}"
+
+    env = await c.i_documented(
+        doc_agent.id,
+        task.id,
+        notes="Documented /healthz behaviour in docs/api/health.md",
+        files=["docs/api/health.md"],
+    )
+    body = env.as_dict()
+    # Must not 500 / propagate — this is the exact assertion that raises
+    # MissingGreenlet without the session.refresh(t) fix, since `status`
+    # reads `t.status` on the savepoint-expired object.
+    assert body["error"] is None, body
+    assert body.get("warning") is not None
+    assert "handoff" in body["warning"].lower()
+    assert body["status"] == Status.AWAITING_PM_REVIEW.value
+
+    final = await task_service.get(task.id)
+    assert final is not None
+    assert str(final.status) == Status.AWAITING_PM_REVIEW.value
+    assert final.docs_complete is True
+
+
+@pytest.mark.asyncio
+async def test_inherit_upstream_base_survives_flush_failure_real_session(
+    db_session: AsyncSession, lifecycle_setup: dict[str, Any]
+) -> None:
+    """Round-2 regression (#task-savepoint-expiry): the conflict branch
+    mutates `task` (the conflict marker + dev_notes) then `flush()`es inside
+    the `begin_nested()` savepoint. On a REAL AsyncSession a flush() failure
+    there rolls back the savepoint and fully expires every attribute of
+    `task` — the real caller (`claim_task_for_agent` ->
+    `_create_work_session_if_needed`) reads `task.project_id`/
+    `task.branch_name` right after this returns; `MissingGreenlet` is not an
+    `AttributeError`, so a `getattr` guard would not shield it, killing the
+    claim despite "never fails the claim". Forces a real, one-shot flush()
+    failure via monkeypatch — the closest realistic trigger to an actual DB
+    hiccup — mirroring `tests/unit/services/test_task_base_inheritance.py`'s
+    own project/git stubbing, but against a REAL session (that unit test's
+    mocked session cannot reproduce ORM expiry at all).
+    """
+    task = lifecycle_setup["task"]
+    project = lifecycle_setup["project"]
+    task.branch_name = "feature/backend/AAA--BBB"
+    await db_session.flush()
+
+    task_service = TaskService(db_session)
+
+    # Drive _inherit_upstream_base into the "conflict" branch (mutate +
+    # flush), mirroring the unit test's own project/git stubbing.
+    proj_svc = MagicMock()
+    proj_svc.get = AsyncMock(return_value=project)
+    git_svc = MagicMock()
+    git_svc.get_workspace = AsyncMock(return_value=MagicMock())
+    git_svc.merge_dependency_lineage = AsyncMock(
+        return_value={"status": "conflict", "files": ["a.py"]}
+    )
+    object.__setattr__(
+        task_service,
+        "_resolve_parent_branch",
+        AsyncMock(return_value="feature/main_pm/root"),
+    )
+
+    real_flush = db_session.flush
+    calls = {"n": 0}
+
+    async def _flush_once_boom() -> None:
+        calls["n"] += 1
+        if calls["n"] == 1:
+            raise RuntimeError("simulated flush failure")
+        await real_flush()
+
+    with (
+        patch(
+            "roboco.services.project.get_project_service",
+            MagicMock(return_value=proj_svc),
+        ),
+        patch("roboco.services.git.get_git_service", MagicMock(return_value=git_svc)),
+        patch.object(db_session, "flush", _flush_once_boom),
+    ):
+        await task_service._inherit_upstream_base(task, uuid4())  # must not raise
+
+    # `task` must be readable afterward — the exact access pattern
+    # `_create_work_session_if_needed` performs right after this call
+    # returns in the real claim flow. Raises MissingGreenlet without the
+    # session.refresh(task) fix in the except block.
+    assert task.project_id == project.id
+    assert task.branch_name == "feature/backend/AAA--BBB"
+
+
 # ---------------------------------------------------------------------------
 # 5. PM complete (Cell PM, simple task): awaiting_pm_review → completed
 # ---------------------------------------------------------------------------