fix(notifications): stop tick-wide row locks + poisoned-session swallows behind the PendingRollbackError 500s (#743)

* fix(notifications): release re-escalation row locks per row; stop swallowing DB errors in notify_get

The re-escalation sweep ran one tick-wide transaction, so each CAS
claim's row lock was held across every remaining delivery until the
single commit — a concurrent mark-read UPDATE on a claimed row starved
into the 60s lock_timeout. The sweep now commits per row (claim commit
releases the lock before delivery and makes the burned slot durable),
re-fetches each row by snapshotted id so one row's rollback can't
expire the rest of the tick, and savepoints each recipient's delivery.

notify_get's bare except swallowed the resulting LockNotAvailableError
into a false "notification not found" and returned a poisoned session
to the commit-at-send middleware, which blew up with
PendingRollbackError; it now catches only the two domain outcomes.

defer_after_commit's listeners fire on SAVEPOINT release too, which
would have drained deferred telegram/bus work before real durability —
they now skip savepoint boundaries via get_nested_transaction() (the
root get_transaction() is non-None inside the listener even at a real
commit). acknowledge_for_recipient's Redis dedup-clear moved before the
flush so the row lock never spans a Redis round-trip. The five
best-effort CEO-notify swallows that persist notification rows are
savepointed.

* fix(services): contain swallowed best-effort DB write failures instead of poisoning the session

Sweep of the same class as the notify_get incident: broad
except-Exception handlers that swallow a failure whose try-body writes
through the shared session leave the session rollback-pending, and the
verb/request then dies later with PendingRollbackError at
commit-at-send. Confirmed-dangerous sites now run the write inside a
savepoint (safe since defer_after_commit skips savepoint boundaries):
ceo_approve's verified-stamp, completion/pitch/postmortem-style CEO
notifies, _inherit_upstream_base, _link_commit_to_task (covers every
commit route), board-program LEARN records, the QA/PR-gate/PM-merge
verified-stamps, and the documenter->PM handoff.
_ack_pending_wake_notifications gets the same treatment so a wake-ack
failure can't fail the A2A read. telegram_inbound's per-update loop and
intake confirm roll back explicitly instead (their success paths commit
mid-flow, so a savepoint doesn't fit).

A swallowed savepoint rollback fully expires any ORM object mutated
inside the block, and the next attribute read raises MissingGreenlet —
strictly worse than the original bug. The two paths that keep using the
object after the swallow (doc handoff's envelope build, base
inheritance's claim continuation) refresh it in the except path;
regression tests run against a real session and were verified to fail
with the refresh reverted.

* test: shape mocked session.execute results so sync accessors stop leaking unawaited coroutines

An AsyncMock's auto-created children are themselves AsyncMock, so
production code that correctly awaits session.execute() and then calls
sync accessors (.scalars().all(), .scalar_one_or_none()) on the result
was silently collecting unawaited coroutines in 22 test files — 80
RuntimeWarnings per unit run, and in test_flow_soup_guard one mock
raised a real TypeError that a coincidentally-matching invalid_state
envelope masked. Each affected fixture now returns a plain MagicMock
shaped like a real Result. Zero AsyncMock warnings remain.

* docs: document per-row sweep commits and the savepoint/refresh containment pattern

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>

CLAUDE.md

@@ -253,7 +253,7 @@ Agents coordinate via **task state + task detail fields**, not a channel/session
 
 Agent learnings (`note` scope='learning') broadcast as knowledge-share notifications only to other **agents** — the human / human-driven roles (CEO, prompter, secretary) are excluded, since agent knowledge-sharing is noise in a human's inbox.
 
-**Notification re-escalation backoff (always-on).** `sweep_expired_notifications` (`roboco/services/notification_delivery.py`) re-escalates a still-unacked ack-required notification past its `expires_at` to the recipient's up-role (the PM's PM, or the CEO) — but only when a per-notification backoff schedule says it's due, not on every ~60s sweep tick forever. Each row carries `reescalation_count` / `last_reescalated_at` / `reescalation_delivered_count` (migration 079): the first re-escalation fires at expiry, each one after that doubles the wait from `ROBOCO_NOTIFICATION_REESCALATION_BASE_SECONDS` (default 1h, capped at 24h), and past `ROBOCO_NOTIFICATION_MAX_REESCALATIONS` (default 5) the row is logged once as permanently-unacked and left alone for good — the due/wait/capped decision is a pure function (`reescalation_decision`, `roboco/foundation/policy/communications.py`). The attempt slot is claimed by a compare-and-set `UPDATE ... WHERE reescalation_count = :n` BEFORE any delivery is attempted (the 60s dedup guard elsewhere does NOT backstop this — `BLOCKER_ESCALATION`, the type every re-escalation fires as, is excluded from the loop-prone dedup set), so two sweep ticks racing the same row can never both deliver. Legacy rows read as `count=0` and keep the original first-fire semantics.
+**Notification re-escalation backoff (always-on).** `sweep_expired_notifications` (`roboco/services/notification_delivery.py`) re-escalates a still-unacked ack-required notification past its `expires_at` to the recipient's up-role (the PM's PM, or the CEO) — but only when a per-notification backoff schedule says it's due, not on every ~60s sweep tick forever. Each row carries `reescalation_count` / `last_reescalated_at` / `reescalation_delivered_count` (migration 079): the first re-escalation fires at expiry, each one after that doubles the wait from `ROBOCO_NOTIFICATION_REESCALATION_BASE_SECONDS` (default 1h, capped at 24h), and past `ROBOCO_NOTIFICATION_MAX_REESCALATIONS` (default 5) the row is logged once as permanently-unacked and left alone for good — the due/wait/capped decision is a pure function (`reescalation_decision`, `roboco/foundation/policy/communications.py`). The attempt slot is claimed by a compare-and-set `UPDATE ... WHERE reescalation_count = :n` BEFORE any delivery is attempted (the 60s dedup guard elsewhere does NOT backstop this — `BLOCKER_ESCALATION`, the type every re-escalation fires as, is excluded from the loop-prone dedup set), so two sweep ticks racing the same row can never both deliver. Legacy rows read as `count=0` and keep the original first-fire semantics. **Per-row commit scope (#721/#730):** the sweep commits after each row instead of once for the whole tick — the claim commit releases the CAS row lock before delivery runs rather than holding it across the tick, which used to starve a recipient's concurrent mark-read/ack UPDATE into the 60s `lock_timeout`. A root rollback expires every object in the identity map, so the loop snapshots row ids up front and re-fetches each via `session.get` (async-safe even post-expiry) instead of iterating the original ORM instances — one row's failure rolls back only that row and never aborts the rest of the tick. Re-escalation's own delivery (`_persist_and_deliver`, wrapped in `session.begin_nested()`) rides the shared transactional-outbox primitive `defer_after_commit`: SQLAlchemy dispatches `after_commit`/`after_rollback` on a SAVEPOINT release too, not just the real root commit, so the outbox's drain/discard is guarded by `get_nested_transaction()` (non-None only at a savepoint boundary, verified live) rather than `get_transaction()` (which stays non-None at a savepoint release too, so it can't tell the two apart) — without that guard, a savepoint release would drain pending Telegram/bus work before the real per-row commit, reintroducing the phantom-notification bug the outbox exists to prevent. The same `begin_nested()` savepoint pattern now wraps every other best-effort DB write on a shared, reused session across the codebase (CEO-notify helpers, A2A wake-notification acks, the CEO-approve findings verify-stamp, commit-linking, board-program decision recording, and more) — a bare `except Exception: log(...)` around a flush/write is not enough on its own, since an unrolled-back DB error leaves the session poisoned for whatever the caller does next. `begin_nested()` is not the complete recipe by itself, though: a swallowed savepoint rollback fully EXPIRES every attribute of any ORM object mutated inside that block (proven live on this stack), so the except path must `await session.refresh(that_object)` before touching any of its attributes again — otherwise the very next read raises `MissingGreenlet` (not an `AttributeError`, so a `getattr` guard doesn't save it) and propagates uncaught, rolling back the whole request instead of the one best-effort side effect.
 
 ## Key Principles