Fix the PR-divergence respawn loop: loop gate, CEO god-mode, PR conflict resolver, sequence-ordered merge (#164)

* fix(orchestrator,panel): bound the respawn loop gate and give the CEO a status override

The PM respawn loop gate could never fire on a recurring tracing_gap: every
same-status respawn that emitted a tracing_gap reset the strike counter, so a
task whose unblock can never satisfy its decision gate respawned forever. Cap
the number of tracing_gap resets (pm_respawn_max_tracing_resets) so strikes
accrue once a gap is clearly recurring rather than progressing, and route the
pm-review and blocker dispatch respawn paths through the gate so it actually
applies to those loops.

Panel: the task status dropdown was driven solely by the lifecycle graph, so a
task wedged in a terminal/blocked state offered no actionable transitions. Add
an audited admin status override (PATCH status -> admin_set_status) for every
non-in-band target, letting the human operator force any state.

* feat(git): add rebase_onto_base and close_pull_request PR-divergence primitives

Agents had no way to resolve a PR that could not merge because a sibling merged
overlapping work first: their only moves were complete (which 405s) or block
(which loops). Add the two missing operations:

- rebase_onto_base rebases a head branch onto the latest base and classifies
  the outcome: superseded (no unique commits -> safe to close), rebased (unique
  work -> force-pushed, ready to merge), or conflicts (aborted, needs a human).
- close_pull_request retires a superseded PR with an explanatory comment.

These back both the sequence-ordered merge and the conflict resolver.

* feat(gateway): auto-resolve a leaf PR that can't merge instead of looping

When a sibling lands overlapping work first, the cell PM's complete() merge
hits a GitHub 405 and the task re-blocks, respawning the PM forever (the
production wedge: one task burned 6000+ tool calls over 3 hours). The merge
now raises MergeConflictError, and cell_pm_complete resolves it:

- rebase the branch onto the current base;
- superseded (no unique commits) -> close the dead PR + complete the task
  without a redundant merge (the manual action operators kept requesting);
- rebased (unique work) -> retry the merge, then complete;
- genuine conflicts -> admin-override the task to awaiting_ceo_approval and
  alert the CEO, so it leaves agent dispatch instead of looping.

MergeConflictError subclasses GitError, so existing handlers are unaffected.

* test(git): silence unused-arg lint in close_pull_request stub

* feat(orchestrator): sequence-ordered merge for leaf siblings

Leaf siblings share one cell branch, but within-cell siblings were all left at
the default sequence 0, so two leaf PRs raced into the same branch and the
second wedged. Now:

- decomposition assigns each new sibling the next ordinal within its parent, so
  the merge order is well-defined;
- the pm-review dispatcher holds a higher-sequence leaf until its earlier
  same-team siblings are terminal, so they merge into the shared branch in order
  instead of racing.

Loop-free by construction: a gated task is simply not dispatched this tick (no
reject, no respawn). Terminal siblings never block, so a cancelled sibling can't
deadlock the rest; any sibling lookup failure degrades to dispatch.

* test: use monkeypatch.setattr instead of type:ignore in new tests

CI type-checks tests/ (the type-gated suite) which my local 'mypy roboco/' skipped.
The method-mock assignments tripped mypy method-assign/assignment; replace the
silencing comments with monkeypatch.setattr and local mock refs for assertions,
matching the project's no-type:ignore rule.

* fix(git): stop get_status misreporting an unstaged deletion as staged

git_status used stdout.strip().split() before parsing porcelain. strip() eats
the leading space on the first line, so an unstaged deletion (' D file') became
'D file' and parsed as a STAGED deletion — the false 'staged' that caused 6
wasted QA cycles when a dev deleted a file without staging it. Use splitlines(),
which preserves the index/worktree status columns.

* feat(panel): mobile sidebar hamburger + Sheet drawer (AC1)

The umbrella's AC1 was never built: on mobile the sidebar had no entry point.
Extract the nav/footer into shared SidebarNav/SidebarFooter, hide the static
sidebar below md, and add a hamburger in the header that opens the same nav in a
left Sheet drawer (closing on navigation). Desktop is unchanged.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
2026-06-14 23:18:58 +02:00
committed by GitHub
co-authored by Renn F
parent bb9d4ff12a
commit 2817ca1ceb
14 changed files with 1297 additions and 64 deletions
@@ -65,6 +65,40 @@ async def test_three_tracing_gap_responses_do_not_trip_kill() -> None:
assert fake_audit.has_recent_tracing_gap.await_count >= expected_audit_calls
@pytest.mark.asyncio
async def test_unending_tracing_gap_is_bounded_and_eventually_trips() -> None:
"""A task whose EVERY respawn trips the same tracing_gap must still die.
The rule-following reset is bounded by ``pm_respawn_max_tracing_resets``.
Before this bound a permanently-wedged task (e.g. a cold-respawned PM that
can never satisfy the unblock journal-decision gate) emitted a tracing_gap
on every spawn, reset the strike counter every time, and respawned
forever — the production bleed. With the cap, resets are exhausted and
strikes accrue until the gate fires.
"""
orch = _new_orchestrator()
task_id = str(uuid4())
task = {"id": task_id, "status": "blocked"}
fake_audit = AsyncMock()
fake_audit.has_recent_tracing_gap = AsyncMock(return_value=True)
with (
patch("roboco.services.audit.get_audit_service", return_value=fake_audit),
patch(
"roboco.services.notification.NotificationService",
return_value=AsyncMock(),
),
):
results = [
await orch._pm_respawn_should_gate("main-pm", task) for _ in range(12)
]
# The bug was that this list would be all-False forever. The gate must
# trip at least once now that the reset budget is finite.
assert any(results), "tracing_gap loop must eventually be gated"
@pytest.mark.asyncio
async def test_three_no_progress_spawns_still_trip_kill() -> None:
"""When there is NO tracing_gap envelope, the strike logic still bites.
@@ -0,0 +1,163 @@
"""Sequence-ordered merge: hold a later sibling's review until earlier ones land.
Leaf siblings share one cell branch, so merging a higher-sequence sibling before
a lower one diverges the branch and wedges the loser. The dispatcher skips a
higher-sequence task while an earlier same-team sibling is still non-terminal —
loop-free (not dispatched, not rejected). Terminal siblings never block, so a
cancelled sibling can't deadlock the rest.
"""
from __future__ import annotations
from typing import Any, cast
from unittest.mock import AsyncMock, MagicMock, patch
from uuid import uuid4
import pytest
from roboco.models.base import TaskStatus
from roboco.runtime.orchestrator import AgentOrchestrator
def _new_orchestrator() -> AgentOrchestrator:
return AgentOrchestrator.__new__(AgentOrchestrator)
def _sibling(seq: int, team: str, status: TaskStatus) -> MagicMock:
return MagicMock(id=uuid4(), sequence=seq, team=team, status=status)
def _patch_siblings(siblings: list[MagicMock]) -> Any:
"""Patch the orchestrator's direct-DB sibling lookup to return ``siblings``."""
svc = MagicMock()
svc.get_subtasks = AsyncMock(return_value=siblings)
class _CM:
async def __aenter__(self) -> MagicMock:
return MagicMock()
async def __aexit__(self, *_a: Any) -> bool:
return False
factory = MagicMock(return_value=_CM())
return (
patch("roboco.db.base.get_session_factory", return_value=factory),
patch("roboco.services.task.get_task_service", return_value=svc),
)
@pytest.mark.asyncio
async def test_blocks_when_earlier_same_team_sibling_active() -> None:
orch = _new_orchestrator()
task = {
"id": str(uuid4()),
"parent_task_id": str(uuid4()),
"sequence": 1,
"team": "frontend",
}
siblings = [_sibling(0, "frontend", TaskStatus.IN_PROGRESS)]
p1, p2 = _patch_siblings(siblings)
with p1, p2:
assert await orch._blocked_by_earlier_sibling(task) is True
@pytest.mark.asyncio
async def test_not_blocked_when_earlier_sibling_terminal() -> None:
orch = _new_orchestrator()
task = {
"id": str(uuid4()),
"parent_task_id": str(uuid4()),
"sequence": 1,
"team": "frontend",
}
# Earlier sibling completed -> no longer blocks. (Cancelled likewise.)
siblings = [
_sibling(0, "frontend", TaskStatus.COMPLETED),
_sibling(0, "frontend", TaskStatus.CANCELLED),
]
p1, p2 = _patch_siblings(siblings)
with p1, p2:
assert await orch._blocked_by_earlier_sibling(task) is False
@pytest.mark.asyncio
async def test_not_blocked_by_different_team_sibling() -> None:
orch = _new_orchestrator()
task = {
"id": str(uuid4()),
"parent_task_id": str(uuid4()),
"sequence": 1,
"team": "frontend",
}
# A backend sibling targets a different branch — never blocks the frontend leaf.
siblings = [_sibling(0, "backend", TaskStatus.IN_PROGRESS)]
p1, p2 = _patch_siblings(siblings)
with p1, p2:
assert await orch._blocked_by_earlier_sibling(task) is False
@pytest.mark.asyncio
async def test_higher_sequence_sibling_does_not_block() -> None:
orch = _new_orchestrator()
task = {
"id": str(uuid4()),
"parent_task_id": str(uuid4()),
"sequence": 0,
"team": "frontend",
}
# A LATER sibling (seq 1) must not hold up the earlier one (seq 0).
siblings = [_sibling(1, "frontend", TaskStatus.IN_PROGRESS)]
p1, p2 = _patch_siblings(siblings)
with p1, p2:
assert await orch._blocked_by_earlier_sibling(task) is False
@pytest.mark.asyncio
async def test_no_parent_returns_false_without_db() -> None:
orch = _new_orchestrator()
task = {"id": str(uuid4()), "sequence": 0, "team": "frontend"}
# No DB patch: a parentless task must short-circuit before any lookup.
assert await orch._blocked_by_earlier_sibling(task) is False
@pytest.mark.asyncio
async def test_db_failure_falls_through_to_dispatch() -> None:
orch = _new_orchestrator()
task = {
"id": str(uuid4()),
"parent_task_id": str(uuid4()),
"sequence": 1,
"team": "frontend",
}
boom = patch(
"roboco.db.base.get_session_factory", side_effect=RuntimeError("db down")
)
with boom:
# The ordering check must never wedge the dispatcher: degrade to dispatch.
assert await orch._blocked_by_earlier_sibling(task) is False
@pytest.mark.asyncio
async def test_dispatch_skips_blocked_sibling(monkeypatch: pytest.MonkeyPatch) -> None:
"""_dispatch_pm_review_work must not spawn a PM for a gated task."""
orch = _new_orchestrator()
task = {
"id": str(uuid4()),
"parent_task_id": str(uuid4()),
"sequence": 1,
"team": "frontend",
"assigned_to": str(uuid4()),
}
spawn = AsyncMock()
# monkeypatch.setattr keeps mypy's method-assign check satisfied without
# silencing it; the spawn mock is held locally so the assertion is typed.
monkeypatch.setattr(orch, "_fetch_tasks", AsyncMock(return_value=[task]))
monkeypatch.setattr(
orch, "_blocked_by_earlier_sibling", AsyncMock(return_value=True)
)
monkeypatch.setattr(orch, "spawn_agent", spawn)
monkeypatch.setattr(orch, "_resolve_agent_slug", MagicMock(return_value="fe-pm"))
monkeypatch.setattr(orch, "_is_agent_active", MagicMock(return_value=False))
await orch._dispatch_pm_review_work(cast("Any", MagicMock()))
spawn.assert_not_awaited()