Fix/run hardening prep (#263)

* fix(git): don't delete a branch that still has open dependent PRs

Root cause of the run-zombifying "integration branch gone from origin" wedge.
_delete_remote_branch_best_effort deleted a merged PR's head branch
unconditionally, so:
- merging a cell->root PR deleted the cell branch while a sibling leaf PR was
  still targeting it as base, and
- the CEO's root->master merge deleted the feature/main_pm/{root} integration
  branch.
The dependent PRs lost their base, every later git op against the vanished
branch failed, and the task zombified (a51c3d31 only made the post-merge sync
non-fatal; this removes the cause).

The remote-branch delete chokepoint (the single path all merge/close/cancel
deletions funnel through) now first checks _branch_has_open_dependents: any OPEN
PR targeting the branch as its base marks it an active integration target and
preserves it. Fails safe (any error => keep the branch; cleanup is best-effort,
stranding is not). True leaf branches with no open dependents are still cleaned
up. Adds 6 unit tests for the guard + the probe.

* fix(git): recover a drifted shared clone on resume instead of BRANCH_MISMATCH

A dev/documenter/QA clone is shared across that agent's tasks. On a
respawn/resume it can sit on a sibling task's branch, or a re-provisioned clone
can lack the task branch as a local ref (commits only on origin). The
fresh-claim path git-resets the clone clean, but resume deliberately
short-circuits before it (_dev_reentry), so the agent's next commit hit
_assert_on_task_branch's BRANCH_MISMATCH, failed, and the task wedged in a
blocked respawn loop (the documenter that could never land its doc commit).

_assert_on_task_branch now recovers instead of only rejecting: fetch + checkout
the task branch (recreating a missing local ref from origin via `git branch
<b> origin/<b>`), and raise only when the switch genuinely can't happen
(uncommitted changes block it). Never discards work — checkout, not reset — so
a resumed agent's unpushed commits are preserved. Updates the RAG troubleshooting
+ developer docs to describe the auto-recovery. Adds 5 unit tests.

* fix(runtime): re-adopt running agent containers on restart (no double-spawn)

An orchestrator restart loses the in-memory _instances registry while the agent
containers keep running. The reaper already had a Docker-liveness fallback
(_assignee_container_running), but the spawn gate (_is_agent_active) did not, so
right after a restart it saw a live agent as inactive and could launch a second
container onto work the forgotten-but-running one was already doing.

start() now calls _readopt_running_agents() after _reconcile_orphan_claims_on_startup
and before the dispatcher/reaper loops launch: it probes each known agent slug's
container (AGENT_IMAGES, reusing _inspect_container_state — the same docker
inspect the reaper uses) and registers a minimal AgentInstance(state=ACTIVE) for
any that is running and not already tracked. Inert when nothing runs (cold start
unchanged); best-effort (a probe error leaves that slot for the reaper's own
fallback). This is the gateway-health spec's Task 4 / the orchestrator-state
spec's Phase 3 (_instances reconcile). Adds 4 unit tests.

* fix(git): treat an already-merged PR as idempotent success on merge

A merge PUT against an already-merged PR returns the same 405 as a genuine
"not mergeable" conflict, so _merge_with_retry raised MergeConflictError and the
completion path tried to rebase / close-superseded / escalate a PR that had
already landed (a prior cycle, a sibling, or the CEO merged it) — the
cell_pm_complete block<->unblock respawn loop.

_merge_with_retry now disambiguates before raising: a new _pr_is_merged probe
(GET the PR, check merged==true) returns success on an already-merged PR so
completion proceeds idempotently; a genuinely-unmerged 405 still raises the
conflict. Best-effort probe (False on any error → falls through to the existing
conflict handling). Adds 4 unit tests.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
2026-06-25 18:35:52 +02:00
committed by GitHub
co-authored by Renn F
parent 99cf56dff3
commit 2c403c77a2
9 changed files with 620 additions and 25 deletions
+8
View File
@@ -8,6 +8,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
### Fixed ### Fixed
- **Completing a task whose PR is already merged no longer loops.** A merge request against an already-merged PR returns the same `405` from GitHub as a genuine "not mergeable" conflict, so the completion path treated an already-landed PR as a conflict and tried to rebase / close-superseded / escalate it — bouncing the task between blocked and unblocked forever (the case where a prior cycle, a sibling, or the CEO had already merged it). The merge now disambiguates: if the PR reports as merged, the merge is treated as idempotent success and completion proceeds; only a PR that is genuinely unmerged raises the conflict.
- **After an orchestrator restart, a still-running agent is no longer double-spawned.** The orchestrator's in-memory instance registry is lost on a restart while the agent containers keep running. The stale-claim reaper already had a Docker-liveness fallback for that, but the spawn gate (`_is_agent_active`) did not — so right after a restart it saw a live agent as inactive and could launch a second container onto the work the forgotten-but-running one was already doing. Startup now re-adopts surviving containers: it probes each known agent slug's container (the same `docker inspect` the reaper uses) and re-registers a minimal active instance for any that is running, before the dispatcher and reaper loops start. Inert when nothing is running, and best-effort (a probe error just leaves that slot for the reaper's own fallback to cover).
- **A resumed agent on a drifted shared clone no longer wedges with `BRANCH_MISMATCH`.** A dev/documenter/QA clone is shared across that agent's tasks; on a respawn/resume it can sit on a sibling task's branch, or a re-provisioned clone can lack the task branch as a local ref (its commits are only on origin). The fresh-claim path git-resets the clone clean, but resume deliberately short-circuits before it — so the agent's next `commit` hit the branch-mismatch guard, failed, and the task wedged in a blocked respawn loop (e.g. the documenter that could never land its doc commit). The guard now *recovers* instead of only rejecting: it fetches and checks out the task's branch (recreating a missing local ref from origin) and only raises when it genuinely cannot switch — i.e. uncommitted changes block it. It never discards work (checkout, not reset), so a resumed agent's unpushed commits are preserved.
- **An integration branch is no longer deleted out from under in-flight work (the "branch gone from origin" zombification).** After a PR merged, the post-merge cleanup deleted its head branch unconditionally — so merging a cell→root PR deleted the cell branch while a sibling leaf PR was still targeting it as its base, and the CEO's root→master merge deleted the `feature/main_pm/{root}` integration branch. The dependent PRs then had no base, every later git op against the vanished branch failed, and the task zombified (the symptom an earlier fix only made non-fatal). The remote-branch delete chokepoint now first checks whether any **open PR still targets the branch as its base** — an active integration target — and preserves it if so; it fails safe (on any error it keeps the branch, since cleanup is best-effort but stranding is not). True leaf branches with no open dependents are still cleaned up as before.
- **Mypy [unreachable] error in test_pr_gate_records_verdict resolved.** A test assigned `t.notes_structured = None` in the function body, causing mypy to narrow the attribute type to `None`. Since the test's helper function took the object as `Any`, mypy did not reset its narrowing after the call, treating `assert t.notes_structured is not None` as statically always-False and marking the next line as `[unreachable]`, failing the quality gate. Fixed by introducing `_TaskWithNoNotes` — a helper class that declares `notes_structured: dict[str, Any] | None = None` in `__init__` — so mypy uses the declared union type rather than a narrowed literal. All tests pass with no suppressions. This pattern is documented in the testing standards for future reference. - **Mypy [unreachable] error in test_pr_gate_records_verdict resolved.** A test assigned `t.notes_structured = None` in the function body, causing mypy to narrow the attribute type to `None`. Since the test's helper function took the object as `Any`, mypy did not reset its narrowing after the call, treating `assert t.notes_structured is not None` as statically always-False and marking the next line as `[unreachable]`, failing the quality gate. Fixed by introducing `_TaskWithNoNotes` — a helper class that declares `notes_structured: dict[str, Any] | None = None` in `__init__` — so mypy uses the declared union type rather than a narrowed literal. All tests pass with no suppressions. This pattern is documented in the testing standards for future reference.
## [0.11.1] - 2026-06-25 ## [0.11.1] - 2026-06-25
+1 -1
View File
@@ -69,7 +69,7 @@ There is **no** `roboco_git_commit / _push / _create_pr / _merge_pr / _checkout`
- Branches are auto-created on `i_will_work_on()`. - Branches are auto-created on `i_will_work_on()`.
- Don't checkout branches by hand — call the verb on the right task. - Don't checkout branches by hand — call the verb on the right task.
- If you see a `BRANCH_MISMATCH` envelope, you're on the wrong task. Use `give_me_work()` again or `unclaim` and re-pick the intended task. - A drifted clone (after a respawn/resume) is now auto-recovered onto your task branch before you commit — you normally won't see `BRANCH_MISMATCH` at all. If you still do, uncommitted changes are blocking the switch: `commit(...)` your work (or `i_am_blocked` if the changes aren't yours), then continue.
## Before Submitting to QA ## Before Submitting to QA
+7 -6
View File
@@ -33,15 +33,16 @@ Notes:
**Error envelope:** `Workspace is on '<other-branch>' but task requires '<task-branch>'` **Error envelope:** `Workspace is on '<other-branch>' but task requires '<task-branch>'`
**Cause:** You're trying to act on task A while your workspace is still on task B's branch. **Cause:** You're trying to act on task A while your workspace is still on task B's branch — usually after a respawn/resume on a shared clone, or a re-provisioned clone that no longer has your task's branch locally.
**Fix:** Don't checkout by hand — there is no `roboco_git_checkout` tool. Call the verb on the *intended* task instead: **Auto-recovery first:** a commit-time check now *recovers* this for you — it re-checks out your task's branch (recreating a missing local ref from origin) before acting, without ever discarding your local commits. So you normally won't see this error on resume at all; just continue your work.
- Devs: `i_will_work_on(task_id)` switches to that task's branch **When you still see it:** the only case left is **uncommitted changes blocking the switch** (the clone can't move off its current branch). Then:
- PMs: `i_will_plan(task_id, plan)` switches to that parent task's branch
- QA: `claim_review(task_id)` switches to the dev's branch under review
If your workspace is dirty, the verb returns an envelope telling you to either `commit(...)` first or escalate via `i_am_blocked`. - `commit(message=..., files=...)` your in-progress work, or escalate via `i_am_blocked` if the changes aren't yours / are conflicting.
- Re-call your role's claim verb on the intended task: `i_will_work_on(task_id)` (devs), `i_will_plan(task_id, plan)` (PMs), `claim_doc_task(task_id)` (documenters), `claim_review(task_id)` (QA).
Don't checkout by hand — there is no `roboco_git_checkout` tool.
## NO_COMMITS on open_pr ## NO_COMMITS on open_pr
+44
View File
@@ -813,6 +813,12 @@ class AgentOrchestrator:
# this, the next claim attempt fails non-idempotent on `git checkout -b`. # this, the next claim attempt fails non-idempotent on `git checkout -b`.
await self._reconcile_orphan_claims_on_startup() await self._reconcile_orphan_claims_on_startup()
# Re-adopt agent containers that survived this orchestrator restart, so
# the spawn gate + reaper see them as live immediately (no double-spawn,
# no over-reap). Inert when nothing is running. Must run before the
# dispatcher/reaper loops launch below.
await self._readopt_running_agents()
# Note: Per-agent settings are now generated at spawn time # Note: Per-agent settings are now generated at spawn time
# via _generate_agent_settings() - no shared settings needed # via _generate_agent_settings() - no shared settings needed
@@ -7243,6 +7249,44 @@ Start now: evidence(task_id="{task_id}")
instance = instances.get(self._resolve_agent_slug(str(owner))) instance = instances.get(self._resolve_agent_slug(str(owner)))
return instance is not None and instance.state == AgentState.ACTIVE return instance is not None and instance.state == AgentState.ACTIVE
async def _readopt_running_agents(self) -> int:
"""Re-adopt still-running agent containers into ``_instances`` at startup.
An orchestrator restart loses the in-memory ``_instances`` registry while
the agent containers keep running. The reaper has a Docker-liveness
fallback for that (``_assignee_container_running``), but the spawn gate's
``_is_agent_active`` does NOT so after a restart it sees a live agent as
inactive and can double-spawn it onto work its forgotten-but-running
container is already doing. Probe each known agent slug's container (the
same ``docker inspect`` the reaper uses) and register a minimal ACTIVE
instance for any that is running and not already tracked, so both the
reaper's live-skip and the spawn gate see the live agent immediately.
Inert when nothing is running (degrades to today's cold start) and
best-effort: a probe error leaves that slot untracked (the reaper's own
fallback still covers it). Returns the number re-adopted.
"""
readopted = 0
for slug in AGENT_IMAGES:
if slug in self._instances:
continue
try:
is_running, _ = await self._inspect_container_state(
f"roboco-agent-{slug}"
)
except Exception:
continue
if not is_running:
continue
self._instances[slug] = AgentInstance(
agent_id=slug, state=AgentState.ACTIVE
)
readopted += 1
if readopted:
logger.info(
"re-adopted running agent containers at startup", count=readopted
)
return readopted
async def _assignee_container_running(self, task: Any) -> bool: async def _assignee_container_running(self, task: Any) -> bool:
"""Docker-liveness fallback for the reaper on an instance-registry MISS. """Docker-liveness fallback for the reaper on an instance-registry MISS.
+141 -18
View File
@@ -664,21 +664,69 @@ class GitService(BaseService):
async def _assert_on_task_branch( async def _assert_on_task_branch(
self, workspace: Path, task_branch: str | None self, workspace: Path, task_branch: str | None
) -> None: ) -> None:
"""Reject ops when the workspace is on a branch other than the task's.""" """Ensure the workspace is on the task's branch, recovering a resumed
clone that drifted instead of hard-failing.
A dev/documenter/QA clone is shared across tasks; on a respawn/resume it
can sit on a sibling task's branch, or a re-provisioned clone can lack
the task branch as a local ref (commits only on origin). The
fresh-claim path git-reset-hards the clone, but resume short-circuits
before it so the agent's next commit hit BRANCH_MISMATCH and the task
wedged in a blocked respawn loop (the documented resume deadlock). This
now fetches + checks out the task branch (recreating a missing local ref
from origin) and only raises if it genuinely cannot switch (uncommitted
changes block it). It NEVER discards local work checkout, not reset, so
a resumed agent's unpushed commits are preserved.
"""
if not task_branch: if not task_branch:
return return
current_branch = await self.get_current_branch(workspace) current_branch = await self.get_current_branch(workspace)
if current_branch and current_branch != task_branch: if not current_branch or current_branch == task_branch:
raise ValidationError( return
f"BRANCH_MISMATCH: Workspace is on '{current_branch}' but " # Resumed/re-provisioned clone parked on the wrong branch — try to
f"task requires '{task_branch}'. The branch is checked out " # recover onto the task branch before rejecting.
f"into your clone by your role's claim verb: " token = await self._token_for_workspace(workspace)
f"`i_will_work_on(task_id)` (devs), " local = await self._run_git(
f"`i_will_plan(task_id, plan)` (PMs), " workspace,
f"`claim_doc_task(task_id)` (documenters), " ["rev-parse", "--verify", "--quiet", f"refs/heads/{task_branch}"],
f"`claim_review(task_id)` (QA). Re-call your role's claim " check=False,
f"verb on this task instead of switching branches by hand." )
if local.returncode != 0:
# Local ref absent (re-provisioned clone) — recover it from origin.
await self._run_git(
workspace,
["fetch", "origin", task_branch],
token=token,
check=False,
timeout=_network_git_timeout(),
) )
await self._run_git(
workspace,
["branch", task_branch, f"origin/{task_branch}"],
check=False,
)
switched = await self._run_git(
workspace, ["checkout", task_branch], check=False
)
if (
switched.returncode == 0
and await self.get_current_branch(workspace) == task_branch
):
self.log.info(
"recovered workspace onto task branch on resume",
task_branch=task_branch,
from_branch=current_branch,
)
return
raise ValidationError(
f"BRANCH_MISMATCH: Workspace is on '{current_branch}' but task "
f"requires '{task_branch}', and it could not be switched "
f"automatically (uncommitted changes likely block the switch). "
f"Re-call your role's claim verb on this task "
f"(`i_will_work_on` / `i_will_plan` / `claim_doc_task` / "
f"`claim_review`); if it persists, unclaim and re-claim to rebuild "
f"the clone, then replay your commits."
)
async def _link_commit_to_task( async def _link_commit_to_task(
self, self,
@@ -2557,17 +2605,55 @@ class GitService(BaseService):
) )
return None return None
async def _branch_has_open_dependents(
self, owner: str, repo: str, branch: str, git_token: str
) -> bool:
"""True if any OPEN PR still targets ``branch`` as its base.
Such a branch is an active integration target a leaf still merging
into its cell branch, or a cell still merging into the
``feature/main_pm/{root}`` integration branch. Deleting it strands those
in-flight child PRs (their base vanishes), which is the run-zombifying
"branch gone from origin" wedge. Fails SAFE: on any error returns True
so the branch is preserved (cleanup is best-effort; stranding is not).
"""
try:
async with httpx.AsyncClient(timeout=10.0) as client:
resp = await client.get(
f"https://api.github.com/repos/{owner}/{repo}/pulls",
params={"base": branch, "state": "open", "per_page": 1},
headers={
"Authorization": f"Bearer {git_token}",
"Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28",
},
)
if not resp.is_success:
return True
return bool(resp.json())
except httpx.HTTPError:
return True
async def _delete_remote_branch_best_effort( async def _delete_remote_branch_best_effort(
self, owner: str, repo: str, branch: str, git_token: str self, owner: str, repo: str, branch: str, git_token: str
) -> None: ) -> None:
"""Best-effort: delete a remote branch by name. """Best-effort: delete a remote branch by name.
Silently swallows errors cleanup is not critical. Skips Silently swallows errors cleanup is not critical. Skips branches that
branches that look like project defaults (main / master / look like project defaults (main / master / develop) and any branch that
develop) as a last-chance safety net against bad input. still has open dependent PRs (an active integration target deleting it
would strand in-flight child work).
""" """
if branch in ("main", "master", "develop", ""): if branch in ("main", "master", "develop", ""):
return return
if await self._branch_has_open_dependents(owner, repo, branch, git_token):
self.log.info(
"branch delete skipped: open dependent PRs target it as base",
branch=branch,
owner=owner,
repo=repo,
)
return
try: try:
async with httpx.AsyncClient(timeout=10.0) as client: async with httpx.AsyncClient(timeout=10.0) as client:
await client.delete( await client.delete(
@@ -3269,16 +3355,53 @@ class GitService(BaseService):
ctx.owner, ctx.repo, ctx.pr_number, ctx.git_token, "squash" ctx.owner, ctx.repo, ctx.pr_number, ctx.git_token, "squash"
) )
if not resp.is_success: if not resp.is_success:
# A merge refusal (typically 405 "not mergeable") means the branch # A merge PUT on an ALREADY-MERGED PR returns the same 405 as a
# conflicts with the base — a sibling landed overlapping work first. # genuine "not mergeable" conflict. An already-merged PR (a prior
# Raise the specific subclass so the completion path can rebase / # cycle, a sibling, or the CEO already landed it) is idempotent
# close-superseded / escalate instead of failing into a respawn loop. # success — NOT a conflict to rebase/escalate. Treating it as one is
# the cell_pm_complete block<->unblock respawn loop, so disambiguate
# before raising.
if await self._pr_is_merged(
ctx.owner, ctx.repo, ctx.pr_number, ctx.git_token
):
return resp
# A real merge refusal (typically 405 "not mergeable") means the
# branch conflicts with the base — a sibling landed overlapping work
# first. Raise the specific subclass so the completion path can
# rebase / close-superseded / escalate instead of respawn-looping.
raise MergeConflictError( raise MergeConflictError(
f"GitHub API refused PR merge ({resp.status_code}): {resp.text[:200]}", f"GitHub API refused PR merge ({resp.status_code}): {resp.text[:200]}",
{"owner": ctx.owner, "repo": ctx.repo, "pr": ctx.pr_number}, {"owner": ctx.owner, "repo": ctx.repo, "pr": ctx.pr_number},
) )
return resp return resp
async def _pr_is_merged(
self, owner: str, repo: str, pr_number: int, git_token: str
) -> bool:
"""True if PR ``pr_number`` is already merged on GitHub.
Disambiguates an already-merged PR from a genuine conflict (both surface
as a 405 on the merge PUT): an already-merged PR is idempotent success,
not something to rebase/escalate. Best-effort returns False on any
error so an indeterminate state falls through to the existing conflict
handling rather than masking a real failure.
"""
try:
async with httpx.AsyncClient(timeout=_default_git_timeout()) as client:
resp = await client.get(
f"https://api.github.com/repos/{owner}/{repo}/pulls/{pr_number}",
headers={
"Authorization": f"Bearer {git_token}",
"Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28",
},
)
except httpx.HTTPError:
return False
if not resp.is_success:
return False
return bool(resp.json().get("merged"))
async def pr_merge( async def pr_merge(
self, self,
pr_number: int, pr_number: int,
@@ -0,0 +1,79 @@
"""Startup re-adoption of still-running agent containers into ``_instances``.
An orchestrator restart loses the in-memory ``_instances`` registry while the
agent containers keep running. The reaper has a Docker-liveness fallback for
that (``_assignee_container_running``), but the spawn gate's ``_is_agent_active``
does not so after a restart it sees a live agent as inactive and can
double-spawn it onto work its forgotten-but-running container is already doing.
``_readopt_running_agents`` probes each known agent slug's container and
re-registers a minimal ACTIVE instance for any that is running, so both the
reaper's live-skip and the spawn gate see the live agent immediately.
"""
from __future__ import annotations
from unittest.mock import AsyncMock, MagicMock
import pytest
from roboco.runtime.orchestrator import AgentOrchestrator, AgentState
_EXPECTED_READOPTED = 2
def _orch() -> AgentOrchestrator:
orch = AgentOrchestrator.__new__(AgentOrchestrator) # bypass __init__
orch._instances = {}
return orch
@pytest.mark.asyncio
async def test_readopts_running_containers_as_active() -> None:
orch = _orch()
running = {"be-dev-1", "fe-pm"}
async def inspect(name: str) -> tuple[bool, int | None]:
slug = name.removeprefix("roboco-agent-")
return (slug in running, 0)
orch._inspect_container_state = AsyncMock(side_effect=inspect) # type: ignore[method-assign]
n = await orch._readopt_running_agents()
assert n == _EXPECTED_READOPTED
assert orch._instances["be-dev-1"].state == AgentState.ACTIVE
assert orch._instances["be-dev-1"].agent_id == "be-dev-1"
assert orch._instances["fe-pm"].state == AgentState.ACTIVE
assert "be-dev-2" not in orch._instances # probed, not running → untracked
@pytest.mark.asyncio
async def test_readopt_leaves_already_tracked_instance_untouched() -> None:
orch = _orch()
sentinel = MagicMock()
orch._instances = {"be-dev-1": sentinel}
orch._inspect_container_state = AsyncMock(return_value=(True, 0)) # type: ignore[method-assign]
await orch._readopt_running_agents()
assert orch._instances["be-dev-1"] is sentinel # not re-adopted over
@pytest.mark.asyncio
async def test_readopt_inert_when_nothing_running() -> None:
orch = _orch()
orch._inspect_container_state = AsyncMock(return_value=(False, None)) # type: ignore[method-assign]
n = await orch._readopt_running_agents()
assert n == 0
assert orch._instances == {}
@pytest.mark.asyncio
async def test_readopt_swallows_probe_errors() -> None:
orch = _orch()
orch._inspect_container_state = AsyncMock(side_effect=RuntimeError("no docker")) # type: ignore[method-assign]
n = await orch._readopt_running_agents()
assert n == 0 # best-effort: a probe failure never raises into startup
@@ -0,0 +1,121 @@
"""GitService must not delete a branch that still has open dependent PRs.
Root cause of the run-zombifying "integration branch gone from origin" wedge:
`_delete_remote_branch_best_effort` deleted a merged PR's head branch
unconditionally. Merging a cellroot PR therefore deleted the cell branch out
from under in-flight leaf PRs still targeting it (and the CEO rootmaster merge
deleted the `feature/main_pm/{root}` integration branch). The fix guards the
deletion chokepoint: a branch that is still the BASE of any open PR is an active
integration target and is preserved. Fails safe if the check can't run, the
branch is kept (cleanup is best-effort; stranding is not).
"""
from __future__ import annotations
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from roboco.services.git import GitService
def _service() -> GitService:
session = MagicMock()
session.execute = AsyncMock(return_value=None)
session.commit = AsyncMock()
return GitService(session)
def _bind(svc: GitService, name: str, value: object) -> None:
object.__setattr__(svc, name, value)
def _fake_client() -> MagicMock:
client = MagicMock()
client.__aenter__ = AsyncMock(return_value=client)
client.__aexit__ = AsyncMock(return_value=False)
client.delete = AsyncMock()
client.get = AsyncMock()
return client
# --- the deletion chokepoint guard ----------------------------------------
@pytest.mark.asyncio
async def test_delete_skips_branch_with_open_dependents() -> None:
svc = _service()
_bind(svc, "_branch_has_open_dependents", AsyncMock(return_value=True))
client = _fake_client()
with patch("roboco.services.git.httpx.AsyncClient", return_value=client):
await svc._delete_remote_branch_best_effort(
"acme", "repo", "feature/main_pm/abc123", "tok"
)
client.delete.assert_not_awaited()
@pytest.mark.asyncio
async def test_delete_removes_leaf_branch_with_no_dependents() -> None:
svc = _service()
_bind(svc, "_branch_has_open_dependents", AsyncMock(return_value=False))
client = _fake_client()
with patch("roboco.services.git.httpx.AsyncClient", return_value=client):
await svc._delete_remote_branch_best_effort(
"acme", "repo", "feature/backend/abc--cell--leaf", "tok"
)
client.delete.assert_awaited_once()
@pytest.mark.asyncio
async def test_delete_skips_default_branch_before_checking_dependents() -> None:
svc = _service()
dep = AsyncMock(return_value=False)
_bind(svc, "_branch_has_open_dependents", dep)
client = _fake_client()
with patch("roboco.services.git.httpx.AsyncClient", return_value=client):
await svc._delete_remote_branch_best_effort("acme", "repo", "master", "tok")
client.delete.assert_not_awaited()
dep.assert_not_awaited()
# --- the open-dependents probe --------------------------------------------
@pytest.mark.asyncio
async def test_has_open_dependents_true_when_open_pr_targets_base() -> None:
svc = _service()
resp = MagicMock(is_success=True)
resp.json.return_value = [{"number": 5}]
client = _fake_client()
client.get = AsyncMock(return_value=resp)
with patch("roboco.services.git.httpx.AsyncClient", return_value=client):
out = await svc._branch_has_open_dependents(
"acme", "repo", "feature/main_pm/abc123", "tok"
)
assert out is True
@pytest.mark.asyncio
async def test_has_open_dependents_false_when_none() -> None:
svc = _service()
resp = MagicMock(is_success=True)
resp.json.return_value = []
client = _fake_client()
client.get = AsyncMock(return_value=resp)
with patch("roboco.services.git.httpx.AsyncClient", return_value=client):
out = await svc._branch_has_open_dependents(
"acme", "repo", "feature/x--leaf", "tok"
)
assert out is False
@pytest.mark.asyncio
async def test_has_open_dependents_fails_safe_on_non_success() -> None:
svc = _service()
resp = MagicMock(is_success=False)
client = _fake_client()
client.get = AsyncMock(return_value=resp)
with patch("roboco.services.git.httpx.AsyncClient", return_value=client):
out = await svc._branch_has_open_dependents(
"acme", "repo", "feature/main_pm/abc123", "tok"
)
assert out is True
@@ -0,0 +1,105 @@
"""``_merge_with_retry`` treats an already-merged PR as idempotent success.
A merge PUT on an already-merged PR returns the same 405 as a genuine
"not mergeable" conflict. Treating it as a conflict made `cell_pm_complete`
try to rebase/escalate a PR that had already landed the block<->unblock
respawn loop. The merge path now disambiguates: already-merged success
(no-op), otherwise a real `MergeConflictError`.
"""
from __future__ import annotations
from pathlib import Path
from typing import Any
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from roboco.exceptions import MergeConflictError
from roboco.services.git import GitService
def _git_service() -> GitService:
svc = GitService.__new__(GitService)
svc.log = MagicMock()
return svc
def _resp(status_code: int, *, is_success: bool) -> Any:
return type(
"R",
(),
{
"status_code": status_code,
"is_success": is_success,
"text": "",
"json": lambda _self=None: {},
},
)()
def _ctx() -> Any:
return GitService._MergeContext(
owner="acme",
repo="repo",
pr_number=42,
git_token="tok",
workspace=Path("/ws"),
target="feature/main_pm/abc",
)
@pytest.mark.asyncio
async def test_merge_idempotent_when_pr_already_merged(
monkeypatch: pytest.MonkeyPatch,
) -> None:
svc = _git_service()
monkeypatch.setattr(
svc, "_call_merge_api", AsyncMock(return_value=_resp(405, is_success=False))
)
already = AsyncMock(return_value=True)
monkeypatch.setattr(svc, "_pr_is_merged", already)
# Must NOT raise — an already-merged PR is a no-op success.
await svc._merge_with_retry(_ctx())
already.assert_awaited_once()
@pytest.mark.asyncio
async def test_merge_raises_conflict_when_not_already_merged(
monkeypatch: pytest.MonkeyPatch,
) -> None:
svc = _git_service()
monkeypatch.setattr(
svc, "_call_merge_api", AsyncMock(return_value=_resp(405, is_success=False))
)
monkeypatch.setattr(svc, "_pr_is_merged", AsyncMock(return_value=False))
with pytest.raises(MergeConflictError):
await svc._merge_with_retry(_ctx())
@pytest.mark.asyncio
async def test_pr_is_merged_true_when_github_reports_merged() -> None:
svc = _git_service()
resp = type(
"R", (), {"is_success": True, "json": lambda _self=None: {"merged": True}}
)()
client = MagicMock()
client.__aenter__ = AsyncMock(return_value=client)
client.__aexit__ = AsyncMock(return_value=False)
client.get = AsyncMock(return_value=resp)
with patch("roboco.services.git.httpx.AsyncClient", return_value=client):
assert await svc._pr_is_merged("acme", "repo", 42, "tok") is True
@pytest.mark.asyncio
async def test_pr_is_merged_false_on_non_success() -> None:
svc = _git_service()
resp = type("R", (), {"is_success": False, "json": lambda _self=None: {}})()
client = MagicMock()
client.__aenter__ = AsyncMock(return_value=client)
client.__aexit__ = AsyncMock(return_value=False)
client.get = AsyncMock(return_value=resp)
with patch("roboco.services.git.httpx.AsyncClient", return_value=client):
assert await svc._pr_is_merged("acme", "repo", 42, "tok") is False
@@ -0,0 +1,114 @@
"""On resume, the branch-mismatch chokepoint recovers the clone instead of
hard-failing.
A dev/documenter/QA clone is shared across tasks; on a respawn/resume it can sit
on a sibling task's branch, or a re-provisioned clone can lack the task branch as
a local ref (commits only on origin). `_assert_on_task_branch` used to raise
BRANCH_MISMATCH in that state, so the agent's next commit failed and the task
wedged in a blocked respawn loop (the documented resume deadlock e.g. the
documenter PR #102 case). It now fetches + checks out the task branch (recreating
a missing local ref from origin) and only raises if it genuinely cannot switch
(uncommitted changes block it). It NEVER discards local commits checkout, not
reset.
"""
from __future__ import annotations
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock
import pytest
from roboco.services.base import ValidationError
from roboco.services.git import GitService
def _service() -> GitService:
session = MagicMock()
session.execute = AsyncMock(return_value=None)
return GitService(session)
def _bind(svc: GitService, name: str, value: object) -> None:
object.__setattr__(svc, name, value)
def _run_git_mock(*, local_ref_rc: int = 0, checkout_rc: int = 0) -> AsyncMock:
async def _run(_workspace: Path, args: list[str], **_kw: object) -> MagicMock:
if args[:2] == ["rev-parse", "--verify"]:
return MagicMock(returncode=local_ref_rc, stdout="")
if args[0] == "checkout":
return MagicMock(returncode=checkout_rc, stdout="")
return MagicMock(returncode=0, stdout="")
return AsyncMock(side_effect=_run)
@pytest.mark.asyncio
async def test_noop_when_already_on_task_branch() -> None:
svc = _service()
_bind(svc, "get_current_branch", AsyncMock(return_value="feature/main_pm/abc"))
run = _run_git_mock()
_bind(svc, "_run_git", run)
await svc._assert_on_task_branch(Path("/ws"), "feature/main_pm/abc")
run.assert_not_awaited() # already on it → no git work, no raise
@pytest.mark.asyncio
async def test_noop_when_task_branch_none() -> None:
svc = _service()
gcb = AsyncMock(return_value="whatever")
_bind(svc, "get_current_branch", gcb)
await svc._assert_on_task_branch(Path("/ws"), None)
gcb.assert_not_awaited()
@pytest.mark.asyncio
async def test_recovers_by_checkout_when_local_ref_present() -> None:
svc = _service()
_bind(
svc,
"get_current_branch",
AsyncMock(side_effect=["feature/other--leaf", "feature/main_pm/abc"]),
)
_bind(svc, "_token_for_workspace", AsyncMock(return_value="tok"))
run = _run_git_mock(local_ref_rc=0, checkout_rc=0)
_bind(svc, "_run_git", run)
await svc._assert_on_task_branch(Path("/ws"), "feature/main_pm/abc")
cmds = [c.args[1] for c in run.await_args_list]
assert ["checkout", "feature/main_pm/abc"] in cmds
# local ref present → no recovery fetch/branch-create
assert not any(c[0] == "fetch" for c in cmds)
@pytest.mark.asyncio
async def test_recovers_missing_local_ref_from_origin() -> None:
svc = _service()
_bind(
svc,
"get_current_branch",
AsyncMock(side_effect=["feature/other--leaf", "feature/main_pm/abc"]),
)
_bind(svc, "_token_for_workspace", AsyncMock(return_value="tok"))
run = _run_git_mock(local_ref_rc=1, checkout_rc=0) # local ref missing
_bind(svc, "_run_git", run)
await svc._assert_on_task_branch(Path("/ws"), "feature/main_pm/abc")
cmds = [c.args[1] for c in run.await_args_list]
assert ["fetch", "origin", "feature/main_pm/abc"] in cmds
assert ["branch", "feature/main_pm/abc", "origin/feature/main_pm/abc"] in cmds
assert ["checkout", "feature/main_pm/abc"] in cmds
@pytest.mark.asyncio
async def test_raises_when_cannot_switch() -> None:
svc = _service()
_bind(svc, "get_current_branch", AsyncMock(return_value="feature/other--leaf"))
_bind(svc, "_token_for_workspace", AsyncMock(return_value="tok"))
run = _run_git_mock(local_ref_rc=0, checkout_rc=1) # checkout fails (dirty tree)
_bind(svc, "_run_git", run)
with pytest.raises(ValidationError, match="BRANCH_MISMATCH"):
await svc._assert_on_task_branch(Path("/ws"), "feature/main_pm/abc")