fix(runtime): stop the chown storm from starving claims, and savepoint the PM journal auto-record

Claim-shaped verbs were failing 7/7 (claim_review) and 6/6
(claim_doc_task) as silent 120s FlowVerbTimeout 504s on the NAS: the
per-claim ownership repair walked the whole clone issuing two stat
syscalls per entry (chown_ms 39502 vs git_ms 8 in the live log), several
passes stacked per claim, and the claim transaction held the task row
the whole time — so concurrent writers queued behind it into the 60s
lock_timeout. The walk now does one stat per entry shared by the
chown-skip and chmod-skip checks, and a .git/roboco-owned sentinel
(worktree-aware via _resolve_clone_root, written only after a
zero-failure pass) skips the walk entirely when the tree is already
agent-owned. Every root-side git write invalidates the sentinel BEFORE
its subprocess runs — GitService._run_git for scope != none, plus the
three raw-subprocess paths inside WorkspaceService the adversarial pass
proved bypass it deterministically on the common respawn shape
(_worktree_git for mutating verbs, _fetch_branch_ref,
_fetch_origin_best_effort) — so a live marker can never vouch for files
a root write is about to create.

One of those queued writers was the PM journal-decision auto-record:
its INSERT hit the lock timeout, _ensure_pm_decision's catch-all
swallowed it without rollback, and the poisoned session blew up
escalate_up with PendingRollbackError (live incident). The helper's try
body now runs in a savepoint — one fix covering all seven PM verbs that
route through it — verified empirically against real Postgres in both
directions: the failure path leaves the session healthy and the task
object readable, and create_entry's internal commit inside the savepoint
drains the transactional outbox exactly once.
This commit is contained in:
Renn F
2026-07-31 01:53:07 +02:00
parent d87e2d9b4e
commit e6c9dde2a9
15 changed files with 874 additions and 42 deletions
@@ -194,6 +194,72 @@ async def test_full_scope_op_calls_full_repair_not_git_repair(
git_repair.assert_not_called()
@pytest.mark.asyncio
async def test_read_only_op_does_not_invalidate_owned_marker(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A read-only op writes nothing, so the ownership-sentinel marker
(`_ensure_agent_owned`'s root short-circuit) stays valid — invalidating
it here would force a needless full walk on the very next call."""
(tmp_path / ".git").mkdir()
monkeypatch.setattr(
"roboco.services.git.subprocess.run", lambda *_a, **_k: _ok(["status"])
)
invalidate = MagicMock()
monkeypatch.setattr("roboco.services.workspace.invalidate_owned_marker", invalidate)
await _svc()._run_git(tmp_path, ["status", "--porcelain"])
invalidate.assert_not_called()
@pytest.mark.asyncio
async def test_git_scoped_op_invalidates_owned_marker_before_running(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A `.git`-only-writing op (commit) can still create new root-owned
files, so it must invalidate the marker too, not just checkout/reset/
etc. — and it must do so BEFORE the subprocess runs, so a marker still
trusted by a concurrent `_ensure_agent_owned` call can never straddle
the write."""
(tmp_path / ".git").mkdir()
order: list[str] = []
def _run_subprocess(*_a: object, **_k: object) -> subprocess.CompletedProcess[str]:
order.append("subprocess.run")
return _ok(["commit"])
monkeypatch.setattr("roboco.services.git.subprocess.run", _run_subprocess)
monkeypatch.setattr(
"roboco.services.workspace.invalidate_owned_marker",
lambda _ws: order.append("invalidate_owned_marker"),
)
monkeypatch.setattr("roboco.services.workspace._ensure_git_dir_owned", MagicMock())
await _svc()._run_git(tmp_path, ["commit", "-m", "msg"])
assert order == ["invalidate_owned_marker", "subprocess.run"]
@pytest.mark.asyncio
async def test_full_scope_op_invalidates_owned_marker(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""checkout/reset/rebase/pull can create root-owned working-tree files
too — the marker invalidation isn't scoped to `.git`-only writes."""
(tmp_path / ".git").mkdir()
monkeypatch.setattr(
"roboco.services.git.subprocess.run", lambda *_a, **_k: _ok(["checkout"])
)
invalidate = MagicMock()
monkeypatch.setattr("roboco.services.workspace.invalidate_owned_marker", invalidate)
monkeypatch.setattr("roboco.services.workspace._ensure_agent_owned", MagicMock())
await _svc()._run_git(tmp_path, ["checkout", "some-branch"])
invalidate.assert_called_once_with(tmp_path)
@pytest.mark.asyncio
async def test_reown_after_git_op_returns_zero_ms_when_skipped() -> None:
"""The instrumentation must see a true near-zero cost for a skipped repair,