mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
Fix/run hardening prep (#263)
* fix(git): don't delete a branch that still has open dependent PRs
Root cause of the run-zombifying "integration branch gone from origin" wedge.
_delete_remote_branch_best_effort deleted a merged PR's head branch
unconditionally, so:
- merging a cell->root PR deleted the cell branch while a sibling leaf PR was
still targeting it as base, and
- the CEO's root->master merge deleted the feature/main_pm/{root} integration
branch.
The dependent PRs lost their base, every later git op against the vanished
branch failed, and the task zombified (a51c3d31 only made the post-merge sync
non-fatal; this removes the cause).
The remote-branch delete chokepoint (the single path all merge/close/cancel
deletions funnel through) now first checks _branch_has_open_dependents: any OPEN
PR targeting the branch as its base marks it an active integration target and
preserves it. Fails safe (any error => keep the branch; cleanup is best-effort,
stranding is not). True leaf branches with no open dependents are still cleaned
up. Adds 6 unit tests for the guard + the probe.
* fix(git): recover a drifted shared clone on resume instead of BRANCH_MISMATCH
A dev/documenter/QA clone is shared across that agent's tasks. On a
respawn/resume it can sit on a sibling task's branch, or a re-provisioned clone
can lack the task branch as a local ref (commits only on origin). The
fresh-claim path git-resets the clone clean, but resume deliberately
short-circuits before it (_dev_reentry), so the agent's next commit hit
_assert_on_task_branch's BRANCH_MISMATCH, failed, and the task wedged in a
blocked respawn loop (the documenter that could never land its doc commit).
_assert_on_task_branch now recovers instead of only rejecting: fetch + checkout
the task branch (recreating a missing local ref from origin via `git branch
<b> origin/<b>`), and raise only when the switch genuinely can't happen
(uncommitted changes block it). Never discards work — checkout, not reset — so
a resumed agent's unpushed commits are preserved. Updates the RAG troubleshooting
+ developer docs to describe the auto-recovery. Adds 5 unit tests.
* fix(runtime): re-adopt running agent containers on restart (no double-spawn)
An orchestrator restart loses the in-memory _instances registry while the agent
containers keep running. The reaper already had a Docker-liveness fallback
(_assignee_container_running), but the spawn gate (_is_agent_active) did not, so
right after a restart it saw a live agent as inactive and could launch a second
container onto work the forgotten-but-running one was already doing.
start() now calls _readopt_running_agents() after _reconcile_orphan_claims_on_startup
and before the dispatcher/reaper loops launch: it probes each known agent slug's
container (AGENT_IMAGES, reusing _inspect_container_state — the same docker
inspect the reaper uses) and registers a minimal AgentInstance(state=ACTIVE) for
any that is running and not already tracked. Inert when nothing runs (cold start
unchanged); best-effort (a probe error leaves that slot for the reaper's own
fallback). This is the gateway-health spec's Task 4 / the orchestrator-state
spec's Phase 3 (_instances reconcile). Adds 4 unit tests.
* fix(git): treat an already-merged PR as idempotent success on merge
A merge PUT against an already-merged PR returns the same 405 as a genuine
"not mergeable" conflict, so _merge_with_retry raised MergeConflictError and the
completion path tried to rebase / close-superseded / escalate a PR that had
already landed (a prior cycle, a sibling, or the CEO merged it) — the
cell_pm_complete block<->unblock respawn loop.
_merge_with_retry now disambiguates before raising: a new _pr_is_merged probe
(GET the PR, check merged==true) returns success on an already-merged PR so
completion proceeds idempotently; a genuinely-unmerged 405 still raises the
conflict. Best-effort probe (False on any error → falls through to the existing
conflict handling). Adds 4 unit tests.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
@@ -8,6 +8,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|
||||||
|
- **Completing a task whose PR is already merged no longer loops.** A merge request against an already-merged PR returns the same `405` from GitHub as a genuine "not mergeable" conflict, so the completion path treated an already-landed PR as a conflict and tried to rebase / close-superseded / escalate it — bouncing the task between blocked and unblocked forever (the case where a prior cycle, a sibling, or the CEO had already merged it). The merge now disambiguates: if the PR reports as merged, the merge is treated as idempotent success and completion proceeds; only a PR that is genuinely unmerged raises the conflict.
|
||||||
|
|
||||||
|
- **After an orchestrator restart, a still-running agent is no longer double-spawned.** The orchestrator's in-memory instance registry is lost on a restart while the agent containers keep running. The stale-claim reaper already had a Docker-liveness fallback for that, but the spawn gate (`_is_agent_active`) did not — so right after a restart it saw a live agent as inactive and could launch a second container onto the work the forgotten-but-running one was already doing. Startup now re-adopts surviving containers: it probes each known agent slug's container (the same `docker inspect` the reaper uses) and re-registers a minimal active instance for any that is running, before the dispatcher and reaper loops start. Inert when nothing is running, and best-effort (a probe error just leaves that slot for the reaper's own fallback to cover).
|
||||||
|
|
||||||
|
- **A resumed agent on a drifted shared clone no longer wedges with `BRANCH_MISMATCH`.** A dev/documenter/QA clone is shared across that agent's tasks; on a respawn/resume it can sit on a sibling task's branch, or a re-provisioned clone can lack the task branch as a local ref (its commits are only on origin). The fresh-claim path git-resets the clone clean, but resume deliberately short-circuits before it — so the agent's next `commit` hit the branch-mismatch guard, failed, and the task wedged in a blocked respawn loop (e.g. the documenter that could never land its doc commit). The guard now *recovers* instead of only rejecting: it fetches and checks out the task's branch (recreating a missing local ref from origin) and only raises when it genuinely cannot switch — i.e. uncommitted changes block it. It never discards work (checkout, not reset), so a resumed agent's unpushed commits are preserved.
|
||||||
|
|
||||||
|
- **An integration branch is no longer deleted out from under in-flight work (the "branch gone from origin" zombification).** After a PR merged, the post-merge cleanup deleted its head branch unconditionally — so merging a cell→root PR deleted the cell branch while a sibling leaf PR was still targeting it as its base, and the CEO's root→master merge deleted the `feature/main_pm/{root}` integration branch. The dependent PRs then had no base, every later git op against the vanished branch failed, and the task zombified (the symptom an earlier fix only made non-fatal). The remote-branch delete chokepoint now first checks whether any **open PR still targets the branch as its base** — an active integration target — and preserves it if so; it fails safe (on any error it keeps the branch, since cleanup is best-effort but stranding is not). True leaf branches with no open dependents are still cleaned up as before.
|
||||||
|
|
||||||
- **Mypy [unreachable] error in test_pr_gate_records_verdict resolved.** A test assigned `t.notes_structured = None` in the function body, causing mypy to narrow the attribute type to `None`. Since the test's helper function took the object as `Any`, mypy did not reset its narrowing after the call, treating `assert t.notes_structured is not None` as statically always-False and marking the next line as `[unreachable]`, failing the quality gate. Fixed by introducing `_TaskWithNoNotes` — a helper class that declares `notes_structured: dict[str, Any] | None = None` in `__init__` — so mypy uses the declared union type rather than a narrowed literal. All tests pass with no suppressions. This pattern is documented in the testing standards for future reference.
|
- **Mypy [unreachable] error in test_pr_gate_records_verdict resolved.** A test assigned `t.notes_structured = None` in the function body, causing mypy to narrow the attribute type to `None`. Since the test's helper function took the object as `Any`, mypy did not reset its narrowing after the call, treating `assert t.notes_structured is not None` as statically always-False and marking the next line as `[unreachable]`, failing the quality gate. Fixed by introducing `_TaskWithNoNotes` — a helper class that declares `notes_structured: dict[str, Any] | None = None` in `__init__` — so mypy uses the declared union type rather than a narrowed literal. All tests pass with no suppressions. This pattern is documented in the testing standards for future reference.
|
||||||
|
|
||||||
## [0.11.1] - 2026-06-25
|
## [0.11.1] - 2026-06-25
|
||||||
|
|||||||
@@ -69,7 +69,7 @@ There is **no** `roboco_git_commit / _push / _create_pr / _merge_pr / _checkout`
|
|||||||
|
|
||||||
- Branches are auto-created on `i_will_work_on()`.
|
- Branches are auto-created on `i_will_work_on()`.
|
||||||
- Don't checkout branches by hand — call the verb on the right task.
|
- Don't checkout branches by hand — call the verb on the right task.
|
||||||
- If you see a `BRANCH_MISMATCH` envelope, you're on the wrong task. Use `give_me_work()` again or `unclaim` and re-pick the intended task.
|
- A drifted clone (after a respawn/resume) is now auto-recovered onto your task branch before you commit — you normally won't see `BRANCH_MISMATCH` at all. If you still do, uncommitted changes are blocking the switch: `commit(...)` your work (or `i_am_blocked` if the changes aren't yours), then continue.
|
||||||
|
|
||||||
## Before Submitting to QA
|
## Before Submitting to QA
|
||||||
|
|
||||||
|
|||||||
@@ -33,15 +33,16 @@ Notes:
|
|||||||
|
|
||||||
**Error envelope:** `Workspace is on '<other-branch>' but task requires '<task-branch>'`
|
**Error envelope:** `Workspace is on '<other-branch>' but task requires '<task-branch>'`
|
||||||
|
|
||||||
**Cause:** You're trying to act on task A while your workspace is still on task B's branch.
|
**Cause:** You're trying to act on task A while your workspace is still on task B's branch — usually after a respawn/resume on a shared clone, or a re-provisioned clone that no longer has your task's branch locally.
|
||||||
|
|
||||||
**Fix:** Don't checkout by hand — there is no `roboco_git_checkout` tool. Call the verb on the *intended* task instead:
|
**Auto-recovery first:** a commit-time check now *recovers* this for you — it re-checks out your task's branch (recreating a missing local ref from origin) before acting, without ever discarding your local commits. So you normally won't see this error on resume at all; just continue your work.
|
||||||
|
|
||||||
- Devs: `i_will_work_on(task_id)` switches to that task's branch
|
**When you still see it:** the only case left is **uncommitted changes blocking the switch** (the clone can't move off its current branch). Then:
|
||||||
- PMs: `i_will_plan(task_id, plan)` switches to that parent task's branch
|
|
||||||
- QA: `claim_review(task_id)` switches to the dev's branch under review
|
|
||||||
|
|
||||||
If your workspace is dirty, the verb returns an envelope telling you to either `commit(...)` first or escalate via `i_am_blocked`.
|
- `commit(message=..., files=...)` your in-progress work, or escalate via `i_am_blocked` if the changes aren't yours / are conflicting.
|
||||||
|
- Re-call your role's claim verb on the intended task: `i_will_work_on(task_id)` (devs), `i_will_plan(task_id, plan)` (PMs), `claim_doc_task(task_id)` (documenters), `claim_review(task_id)` (QA).
|
||||||
|
|
||||||
|
Don't checkout by hand — there is no `roboco_git_checkout` tool.
|
||||||
|
|
||||||
## NO_COMMITS on open_pr
|
## NO_COMMITS on open_pr
|
||||||
|
|
||||||
|
|||||||
@@ -813,6 +813,12 @@ class AgentOrchestrator:
|
|||||||
# this, the next claim attempt fails non-idempotent on `git checkout -b`.
|
# this, the next claim attempt fails non-idempotent on `git checkout -b`.
|
||||||
await self._reconcile_orphan_claims_on_startup()
|
await self._reconcile_orphan_claims_on_startup()
|
||||||
|
|
||||||
|
# Re-adopt agent containers that survived this orchestrator restart, so
|
||||||
|
# the spawn gate + reaper see them as live immediately (no double-spawn,
|
||||||
|
# no over-reap). Inert when nothing is running. Must run before the
|
||||||
|
# dispatcher/reaper loops launch below.
|
||||||
|
await self._readopt_running_agents()
|
||||||
|
|
||||||
# Note: Per-agent settings are now generated at spawn time
|
# Note: Per-agent settings are now generated at spawn time
|
||||||
# via _generate_agent_settings() - no shared settings needed
|
# via _generate_agent_settings() - no shared settings needed
|
||||||
|
|
||||||
@@ -7243,6 +7249,44 @@ Start now: evidence(task_id="{task_id}")
|
|||||||
instance = instances.get(self._resolve_agent_slug(str(owner)))
|
instance = instances.get(self._resolve_agent_slug(str(owner)))
|
||||||
return instance is not None and instance.state == AgentState.ACTIVE
|
return instance is not None and instance.state == AgentState.ACTIVE
|
||||||
|
|
||||||
|
async def _readopt_running_agents(self) -> int:
|
||||||
|
"""Re-adopt still-running agent containers into ``_instances`` at startup.
|
||||||
|
|
||||||
|
An orchestrator restart loses the in-memory ``_instances`` registry while
|
||||||
|
the agent containers keep running. The reaper has a Docker-liveness
|
||||||
|
fallback for that (``_assignee_container_running``), but the spawn gate's
|
||||||
|
``_is_agent_active`` does NOT — so after a restart it sees a live agent as
|
||||||
|
inactive and can double-spawn it onto work its forgotten-but-running
|
||||||
|
container is already doing. Probe each known agent slug's container (the
|
||||||
|
same ``docker inspect`` the reaper uses) and register a minimal ACTIVE
|
||||||
|
instance for any that is running and not already tracked, so both the
|
||||||
|
reaper's live-skip and the spawn gate see the live agent immediately.
|
||||||
|
Inert when nothing is running (degrades to today's cold start) and
|
||||||
|
best-effort: a probe error leaves that slot untracked (the reaper's own
|
||||||
|
fallback still covers it). Returns the number re-adopted.
|
||||||
|
"""
|
||||||
|
readopted = 0
|
||||||
|
for slug in AGENT_IMAGES:
|
||||||
|
if slug in self._instances:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
is_running, _ = await self._inspect_container_state(
|
||||||
|
f"roboco-agent-{slug}"
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
if not is_running:
|
||||||
|
continue
|
||||||
|
self._instances[slug] = AgentInstance(
|
||||||
|
agent_id=slug, state=AgentState.ACTIVE
|
||||||
|
)
|
||||||
|
readopted += 1
|
||||||
|
if readopted:
|
||||||
|
logger.info(
|
||||||
|
"re-adopted running agent containers at startup", count=readopted
|
||||||
|
)
|
||||||
|
return readopted
|
||||||
|
|
||||||
async def _assignee_container_running(self, task: Any) -> bool:
|
async def _assignee_container_running(self, task: Any) -> bool:
|
||||||
"""Docker-liveness fallback for the reaper on an instance-registry MISS.
|
"""Docker-liveness fallback for the reaper on an instance-registry MISS.
|
||||||
|
|
||||||
|
|||||||
+141
-18
@@ -664,21 +664,69 @@ class GitService(BaseService):
|
|||||||
async def _assert_on_task_branch(
|
async def _assert_on_task_branch(
|
||||||
self, workspace: Path, task_branch: str | None
|
self, workspace: Path, task_branch: str | None
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Reject ops when the workspace is on a branch other than the task's."""
|
"""Ensure the workspace is on the task's branch, recovering a resumed
|
||||||
|
clone that drifted — instead of hard-failing.
|
||||||
|
|
||||||
|
A dev/documenter/QA clone is shared across tasks; on a respawn/resume it
|
||||||
|
can sit on a sibling task's branch, or a re-provisioned clone can lack
|
||||||
|
the task branch as a local ref (commits only on origin). The
|
||||||
|
fresh-claim path git-reset-hards the clone, but resume short-circuits
|
||||||
|
before it — so the agent's next commit hit BRANCH_MISMATCH and the task
|
||||||
|
wedged in a blocked respawn loop (the documented resume deadlock). This
|
||||||
|
now fetches + checks out the task branch (recreating a missing local ref
|
||||||
|
from origin) and only raises if it genuinely cannot switch (uncommitted
|
||||||
|
changes block it). It NEVER discards local work — checkout, not reset, so
|
||||||
|
a resumed agent's unpushed commits are preserved.
|
||||||
|
"""
|
||||||
if not task_branch:
|
if not task_branch:
|
||||||
return
|
return
|
||||||
current_branch = await self.get_current_branch(workspace)
|
current_branch = await self.get_current_branch(workspace)
|
||||||
if current_branch and current_branch != task_branch:
|
if not current_branch or current_branch == task_branch:
|
||||||
raise ValidationError(
|
return
|
||||||
f"BRANCH_MISMATCH: Workspace is on '{current_branch}' but "
|
# Resumed/re-provisioned clone parked on the wrong branch — try to
|
||||||
f"task requires '{task_branch}'. The branch is checked out "
|
# recover onto the task branch before rejecting.
|
||||||
f"into your clone by your role's claim verb: "
|
token = await self._token_for_workspace(workspace)
|
||||||
f"`i_will_work_on(task_id)` (devs), "
|
local = await self._run_git(
|
||||||
f"`i_will_plan(task_id, plan)` (PMs), "
|
workspace,
|
||||||
f"`claim_doc_task(task_id)` (documenters), "
|
["rev-parse", "--verify", "--quiet", f"refs/heads/{task_branch}"],
|
||||||
f"`claim_review(task_id)` (QA). Re-call your role's claim "
|
check=False,
|
||||||
f"verb on this task instead of switching branches by hand."
|
)
|
||||||
|
if local.returncode != 0:
|
||||||
|
# Local ref absent (re-provisioned clone) — recover it from origin.
|
||||||
|
await self._run_git(
|
||||||
|
workspace,
|
||||||
|
["fetch", "origin", task_branch],
|
||||||
|
token=token,
|
||||||
|
check=False,
|
||||||
|
timeout=_network_git_timeout(),
|
||||||
)
|
)
|
||||||
|
await self._run_git(
|
||||||
|
workspace,
|
||||||
|
["branch", task_branch, f"origin/{task_branch}"],
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
switched = await self._run_git(
|
||||||
|
workspace, ["checkout", task_branch], check=False
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
switched.returncode == 0
|
||||||
|
and await self.get_current_branch(workspace) == task_branch
|
||||||
|
):
|
||||||
|
self.log.info(
|
||||||
|
"recovered workspace onto task branch on resume",
|
||||||
|
task_branch=task_branch,
|
||||||
|
from_branch=current_branch,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
raise ValidationError(
|
||||||
|
f"BRANCH_MISMATCH: Workspace is on '{current_branch}' but task "
|
||||||
|
f"requires '{task_branch}', and it could not be switched "
|
||||||
|
f"automatically (uncommitted changes likely block the switch). "
|
||||||
|
f"Re-call your role's claim verb on this task "
|
||||||
|
f"(`i_will_work_on` / `i_will_plan` / `claim_doc_task` / "
|
||||||
|
f"`claim_review`); if it persists, unclaim and re-claim to rebuild "
|
||||||
|
f"the clone, then replay your commits."
|
||||||
|
)
|
||||||
|
|
||||||
async def _link_commit_to_task(
|
async def _link_commit_to_task(
|
||||||
self,
|
self,
|
||||||
@@ -2557,17 +2605,55 @@ class GitService(BaseService):
|
|||||||
)
|
)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
async def _branch_has_open_dependents(
|
||||||
|
self, owner: str, repo: str, branch: str, git_token: str
|
||||||
|
) -> bool:
|
||||||
|
"""True if any OPEN PR still targets ``branch`` as its base.
|
||||||
|
|
||||||
|
Such a branch is an active integration target — a leaf still merging
|
||||||
|
into its cell branch, or a cell still merging into the
|
||||||
|
``feature/main_pm/{root}`` integration branch. Deleting it strands those
|
||||||
|
in-flight child PRs (their base vanishes), which is the run-zombifying
|
||||||
|
"branch gone from origin" wedge. Fails SAFE: on any error returns True
|
||||||
|
so the branch is preserved (cleanup is best-effort; stranding is not).
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient(timeout=10.0) as client:
|
||||||
|
resp = await client.get(
|
||||||
|
f"https://api.github.com/repos/{owner}/{repo}/pulls",
|
||||||
|
params={"base": branch, "state": "open", "per_page": 1},
|
||||||
|
headers={
|
||||||
|
"Authorization": f"Bearer {git_token}",
|
||||||
|
"Accept": "application/vnd.github+json",
|
||||||
|
"X-GitHub-Api-Version": "2022-11-28",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
if not resp.is_success:
|
||||||
|
return True
|
||||||
|
return bool(resp.json())
|
||||||
|
except httpx.HTTPError:
|
||||||
|
return True
|
||||||
|
|
||||||
async def _delete_remote_branch_best_effort(
|
async def _delete_remote_branch_best_effort(
|
||||||
self, owner: str, repo: str, branch: str, git_token: str
|
self, owner: str, repo: str, branch: str, git_token: str
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Best-effort: delete a remote branch by name.
|
"""Best-effort: delete a remote branch by name.
|
||||||
|
|
||||||
Silently swallows errors — cleanup is not critical. Skips
|
Silently swallows errors — cleanup is not critical. Skips branches that
|
||||||
branches that look like project defaults (main / master /
|
look like project defaults (main / master / develop) and any branch that
|
||||||
develop) as a last-chance safety net against bad input.
|
still has open dependent PRs (an active integration target — deleting it
|
||||||
|
would strand in-flight child work).
|
||||||
"""
|
"""
|
||||||
if branch in ("main", "master", "develop", ""):
|
if branch in ("main", "master", "develop", ""):
|
||||||
return
|
return
|
||||||
|
if await self._branch_has_open_dependents(owner, repo, branch, git_token):
|
||||||
|
self.log.info(
|
||||||
|
"branch delete skipped: open dependent PRs target it as base",
|
||||||
|
branch=branch,
|
||||||
|
owner=owner,
|
||||||
|
repo=repo,
|
||||||
|
)
|
||||||
|
return
|
||||||
try:
|
try:
|
||||||
async with httpx.AsyncClient(timeout=10.0) as client:
|
async with httpx.AsyncClient(timeout=10.0) as client:
|
||||||
await client.delete(
|
await client.delete(
|
||||||
@@ -3269,16 +3355,53 @@ class GitService(BaseService):
|
|||||||
ctx.owner, ctx.repo, ctx.pr_number, ctx.git_token, "squash"
|
ctx.owner, ctx.repo, ctx.pr_number, ctx.git_token, "squash"
|
||||||
)
|
)
|
||||||
if not resp.is_success:
|
if not resp.is_success:
|
||||||
# A merge refusal (typically 405 "not mergeable") means the branch
|
# A merge PUT on an ALREADY-MERGED PR returns the same 405 as a
|
||||||
# conflicts with the base — a sibling landed overlapping work first.
|
# genuine "not mergeable" conflict. An already-merged PR (a prior
|
||||||
# Raise the specific subclass so the completion path can rebase /
|
# cycle, a sibling, or the CEO already landed it) is idempotent
|
||||||
# close-superseded / escalate instead of failing into a respawn loop.
|
# success — NOT a conflict to rebase/escalate. Treating it as one is
|
||||||
|
# the cell_pm_complete block<->unblock respawn loop, so disambiguate
|
||||||
|
# before raising.
|
||||||
|
if await self._pr_is_merged(
|
||||||
|
ctx.owner, ctx.repo, ctx.pr_number, ctx.git_token
|
||||||
|
):
|
||||||
|
return resp
|
||||||
|
# A real merge refusal (typically 405 "not mergeable") means the
|
||||||
|
# branch conflicts with the base — a sibling landed overlapping work
|
||||||
|
# first. Raise the specific subclass so the completion path can
|
||||||
|
# rebase / close-superseded / escalate instead of respawn-looping.
|
||||||
raise MergeConflictError(
|
raise MergeConflictError(
|
||||||
f"GitHub API refused PR merge ({resp.status_code}): {resp.text[:200]}",
|
f"GitHub API refused PR merge ({resp.status_code}): {resp.text[:200]}",
|
||||||
{"owner": ctx.owner, "repo": ctx.repo, "pr": ctx.pr_number},
|
{"owner": ctx.owner, "repo": ctx.repo, "pr": ctx.pr_number},
|
||||||
)
|
)
|
||||||
return resp
|
return resp
|
||||||
|
|
||||||
|
async def _pr_is_merged(
|
||||||
|
self, owner: str, repo: str, pr_number: int, git_token: str
|
||||||
|
) -> bool:
|
||||||
|
"""True if PR ``pr_number`` is already merged on GitHub.
|
||||||
|
|
||||||
|
Disambiguates an already-merged PR from a genuine conflict (both surface
|
||||||
|
as a 405 on the merge PUT): an already-merged PR is idempotent success,
|
||||||
|
not something to rebase/escalate. Best-effort — returns False on any
|
||||||
|
error so an indeterminate state falls through to the existing conflict
|
||||||
|
handling rather than masking a real failure.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient(timeout=_default_git_timeout()) as client:
|
||||||
|
resp = await client.get(
|
||||||
|
f"https://api.github.com/repos/{owner}/{repo}/pulls/{pr_number}",
|
||||||
|
headers={
|
||||||
|
"Authorization": f"Bearer {git_token}",
|
||||||
|
"Accept": "application/vnd.github+json",
|
||||||
|
"X-GitHub-Api-Version": "2022-11-28",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
except httpx.HTTPError:
|
||||||
|
return False
|
||||||
|
if not resp.is_success:
|
||||||
|
return False
|
||||||
|
return bool(resp.json().get("merged"))
|
||||||
|
|
||||||
async def pr_merge(
|
async def pr_merge(
|
||||||
self,
|
self,
|
||||||
pr_number: int,
|
pr_number: int,
|
||||||
|
|||||||
@@ -0,0 +1,79 @@
|
|||||||
|
"""Startup re-adoption of still-running agent containers into ``_instances``.
|
||||||
|
|
||||||
|
An orchestrator restart loses the in-memory ``_instances`` registry while the
|
||||||
|
agent containers keep running. The reaper has a Docker-liveness fallback for
|
||||||
|
that (``_assignee_container_running``), but the spawn gate's ``_is_agent_active``
|
||||||
|
does not — so after a restart it sees a live agent as inactive and can
|
||||||
|
double-spawn it onto work its forgotten-but-running container is already doing.
|
||||||
|
``_readopt_running_agents`` probes each known agent slug's container and
|
||||||
|
re-registers a minimal ACTIVE instance for any that is running, so both the
|
||||||
|
reaper's live-skip and the spawn gate see the live agent immediately.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from unittest.mock import AsyncMock, MagicMock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from roboco.runtime.orchestrator import AgentOrchestrator, AgentState
|
||||||
|
|
||||||
|
_EXPECTED_READOPTED = 2
|
||||||
|
|
||||||
|
|
||||||
|
def _orch() -> AgentOrchestrator:
|
||||||
|
orch = AgentOrchestrator.__new__(AgentOrchestrator) # bypass __init__
|
||||||
|
orch._instances = {}
|
||||||
|
return orch
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_readopts_running_containers_as_active() -> None:
|
||||||
|
orch = _orch()
|
||||||
|
running = {"be-dev-1", "fe-pm"}
|
||||||
|
|
||||||
|
async def inspect(name: str) -> tuple[bool, int | None]:
|
||||||
|
slug = name.removeprefix("roboco-agent-")
|
||||||
|
return (slug in running, 0)
|
||||||
|
|
||||||
|
orch._inspect_container_state = AsyncMock(side_effect=inspect) # type: ignore[method-assign]
|
||||||
|
|
||||||
|
n = await orch._readopt_running_agents()
|
||||||
|
|
||||||
|
assert n == _EXPECTED_READOPTED
|
||||||
|
assert orch._instances["be-dev-1"].state == AgentState.ACTIVE
|
||||||
|
assert orch._instances["be-dev-1"].agent_id == "be-dev-1"
|
||||||
|
assert orch._instances["fe-pm"].state == AgentState.ACTIVE
|
||||||
|
assert "be-dev-2" not in orch._instances # probed, not running → untracked
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_readopt_leaves_already_tracked_instance_untouched() -> None:
|
||||||
|
orch = _orch()
|
||||||
|
sentinel = MagicMock()
|
||||||
|
orch._instances = {"be-dev-1": sentinel}
|
||||||
|
orch._inspect_container_state = AsyncMock(return_value=(True, 0)) # type: ignore[method-assign]
|
||||||
|
|
||||||
|
await orch._readopt_running_agents()
|
||||||
|
|
||||||
|
assert orch._instances["be-dev-1"] is sentinel # not re-adopted over
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_readopt_inert_when_nothing_running() -> None:
|
||||||
|
orch = _orch()
|
||||||
|
orch._inspect_container_state = AsyncMock(return_value=(False, None)) # type: ignore[method-assign]
|
||||||
|
|
||||||
|
n = await orch._readopt_running_agents()
|
||||||
|
|
||||||
|
assert n == 0
|
||||||
|
assert orch._instances == {}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_readopt_swallows_probe_errors() -> None:
|
||||||
|
orch = _orch()
|
||||||
|
orch._inspect_container_state = AsyncMock(side_effect=RuntimeError("no docker")) # type: ignore[method-assign]
|
||||||
|
|
||||||
|
n = await orch._readopt_running_agents()
|
||||||
|
|
||||||
|
assert n == 0 # best-effort: a probe failure never raises into startup
|
||||||
@@ -0,0 +1,121 @@
|
|||||||
|
"""GitService must not delete a branch that still has open dependent PRs.
|
||||||
|
|
||||||
|
Root cause of the run-zombifying "integration branch gone from origin" wedge:
|
||||||
|
`_delete_remote_branch_best_effort` deleted a merged PR's head branch
|
||||||
|
unconditionally. Merging a cell→root PR therefore deleted the cell branch out
|
||||||
|
from under in-flight leaf PRs still targeting it (and the CEO root→master merge
|
||||||
|
deleted the `feature/main_pm/{root}` integration branch). The fix guards the
|
||||||
|
deletion chokepoint: a branch that is still the BASE of any open PR is an active
|
||||||
|
integration target and is preserved. Fails safe — if the check can't run, the
|
||||||
|
branch is kept (cleanup is best-effort; stranding is not).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from unittest.mock import AsyncMock, MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from roboco.services.git import GitService
|
||||||
|
|
||||||
|
|
||||||
|
def _service() -> GitService:
|
||||||
|
session = MagicMock()
|
||||||
|
session.execute = AsyncMock(return_value=None)
|
||||||
|
session.commit = AsyncMock()
|
||||||
|
return GitService(session)
|
||||||
|
|
||||||
|
|
||||||
|
def _bind(svc: GitService, name: str, value: object) -> None:
|
||||||
|
object.__setattr__(svc, name, value)
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_client() -> MagicMock:
|
||||||
|
client = MagicMock()
|
||||||
|
client.__aenter__ = AsyncMock(return_value=client)
|
||||||
|
client.__aexit__ = AsyncMock(return_value=False)
|
||||||
|
client.delete = AsyncMock()
|
||||||
|
client.get = AsyncMock()
|
||||||
|
return client
|
||||||
|
|
||||||
|
|
||||||
|
# --- the deletion chokepoint guard ----------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_delete_skips_branch_with_open_dependents() -> None:
|
||||||
|
svc = _service()
|
||||||
|
_bind(svc, "_branch_has_open_dependents", AsyncMock(return_value=True))
|
||||||
|
client = _fake_client()
|
||||||
|
with patch("roboco.services.git.httpx.AsyncClient", return_value=client):
|
||||||
|
await svc._delete_remote_branch_best_effort(
|
||||||
|
"acme", "repo", "feature/main_pm/abc123", "tok"
|
||||||
|
)
|
||||||
|
client.delete.assert_not_awaited()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_delete_removes_leaf_branch_with_no_dependents() -> None:
|
||||||
|
svc = _service()
|
||||||
|
_bind(svc, "_branch_has_open_dependents", AsyncMock(return_value=False))
|
||||||
|
client = _fake_client()
|
||||||
|
with patch("roboco.services.git.httpx.AsyncClient", return_value=client):
|
||||||
|
await svc._delete_remote_branch_best_effort(
|
||||||
|
"acme", "repo", "feature/backend/abc--cell--leaf", "tok"
|
||||||
|
)
|
||||||
|
client.delete.assert_awaited_once()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_delete_skips_default_branch_before_checking_dependents() -> None:
|
||||||
|
svc = _service()
|
||||||
|
dep = AsyncMock(return_value=False)
|
||||||
|
_bind(svc, "_branch_has_open_dependents", dep)
|
||||||
|
client = _fake_client()
|
||||||
|
with patch("roboco.services.git.httpx.AsyncClient", return_value=client):
|
||||||
|
await svc._delete_remote_branch_best_effort("acme", "repo", "master", "tok")
|
||||||
|
client.delete.assert_not_awaited()
|
||||||
|
dep.assert_not_awaited()
|
||||||
|
|
||||||
|
|
||||||
|
# --- the open-dependents probe --------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_has_open_dependents_true_when_open_pr_targets_base() -> None:
|
||||||
|
svc = _service()
|
||||||
|
resp = MagicMock(is_success=True)
|
||||||
|
resp.json.return_value = [{"number": 5}]
|
||||||
|
client = _fake_client()
|
||||||
|
client.get = AsyncMock(return_value=resp)
|
||||||
|
with patch("roboco.services.git.httpx.AsyncClient", return_value=client):
|
||||||
|
out = await svc._branch_has_open_dependents(
|
||||||
|
"acme", "repo", "feature/main_pm/abc123", "tok"
|
||||||
|
)
|
||||||
|
assert out is True
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_has_open_dependents_false_when_none() -> None:
|
||||||
|
svc = _service()
|
||||||
|
resp = MagicMock(is_success=True)
|
||||||
|
resp.json.return_value = []
|
||||||
|
client = _fake_client()
|
||||||
|
client.get = AsyncMock(return_value=resp)
|
||||||
|
with patch("roboco.services.git.httpx.AsyncClient", return_value=client):
|
||||||
|
out = await svc._branch_has_open_dependents(
|
||||||
|
"acme", "repo", "feature/x--leaf", "tok"
|
||||||
|
)
|
||||||
|
assert out is False
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_has_open_dependents_fails_safe_on_non_success() -> None:
|
||||||
|
svc = _service()
|
||||||
|
resp = MagicMock(is_success=False)
|
||||||
|
client = _fake_client()
|
||||||
|
client.get = AsyncMock(return_value=resp)
|
||||||
|
with patch("roboco.services.git.httpx.AsyncClient", return_value=client):
|
||||||
|
out = await svc._branch_has_open_dependents(
|
||||||
|
"acme", "repo", "feature/main_pm/abc123", "tok"
|
||||||
|
)
|
||||||
|
assert out is True
|
||||||
@@ -0,0 +1,105 @@
|
|||||||
|
"""``_merge_with_retry`` treats an already-merged PR as idempotent success.
|
||||||
|
|
||||||
|
A merge PUT on an already-merged PR returns the same 405 as a genuine
|
||||||
|
"not mergeable" conflict. Treating it as a conflict made `cell_pm_complete`
|
||||||
|
try to rebase/escalate a PR that had already landed — the block<->unblock
|
||||||
|
respawn loop. The merge path now disambiguates: already-merged → success
|
||||||
|
(no-op), otherwise a real `MergeConflictError`.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
from unittest.mock import AsyncMock, MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from roboco.exceptions import MergeConflictError
|
||||||
|
from roboco.services.git import GitService
|
||||||
|
|
||||||
|
|
||||||
|
def _git_service() -> GitService:
|
||||||
|
svc = GitService.__new__(GitService)
|
||||||
|
svc.log = MagicMock()
|
||||||
|
return svc
|
||||||
|
|
||||||
|
|
||||||
|
def _resp(status_code: int, *, is_success: bool) -> Any:
|
||||||
|
return type(
|
||||||
|
"R",
|
||||||
|
(),
|
||||||
|
{
|
||||||
|
"status_code": status_code,
|
||||||
|
"is_success": is_success,
|
||||||
|
"text": "",
|
||||||
|
"json": lambda _self=None: {},
|
||||||
|
},
|
||||||
|
)()
|
||||||
|
|
||||||
|
|
||||||
|
def _ctx() -> Any:
|
||||||
|
return GitService._MergeContext(
|
||||||
|
owner="acme",
|
||||||
|
repo="repo",
|
||||||
|
pr_number=42,
|
||||||
|
git_token="tok",
|
||||||
|
workspace=Path("/ws"),
|
||||||
|
target="feature/main_pm/abc",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_merge_idempotent_when_pr_already_merged(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
svc = _git_service()
|
||||||
|
monkeypatch.setattr(
|
||||||
|
svc, "_call_merge_api", AsyncMock(return_value=_resp(405, is_success=False))
|
||||||
|
)
|
||||||
|
already = AsyncMock(return_value=True)
|
||||||
|
monkeypatch.setattr(svc, "_pr_is_merged", already)
|
||||||
|
|
||||||
|
# Must NOT raise — an already-merged PR is a no-op success.
|
||||||
|
await svc._merge_with_retry(_ctx())
|
||||||
|
|
||||||
|
already.assert_awaited_once()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_merge_raises_conflict_when_not_already_merged(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
svc = _git_service()
|
||||||
|
monkeypatch.setattr(
|
||||||
|
svc, "_call_merge_api", AsyncMock(return_value=_resp(405, is_success=False))
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(svc, "_pr_is_merged", AsyncMock(return_value=False))
|
||||||
|
|
||||||
|
with pytest.raises(MergeConflictError):
|
||||||
|
await svc._merge_with_retry(_ctx())
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_pr_is_merged_true_when_github_reports_merged() -> None:
|
||||||
|
svc = _git_service()
|
||||||
|
resp = type(
|
||||||
|
"R", (), {"is_success": True, "json": lambda _self=None: {"merged": True}}
|
||||||
|
)()
|
||||||
|
client = MagicMock()
|
||||||
|
client.__aenter__ = AsyncMock(return_value=client)
|
||||||
|
client.__aexit__ = AsyncMock(return_value=False)
|
||||||
|
client.get = AsyncMock(return_value=resp)
|
||||||
|
with patch("roboco.services.git.httpx.AsyncClient", return_value=client):
|
||||||
|
assert await svc._pr_is_merged("acme", "repo", 42, "tok") is True
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_pr_is_merged_false_on_non_success() -> None:
|
||||||
|
svc = _git_service()
|
||||||
|
resp = type("R", (), {"is_success": False, "json": lambda _self=None: {}})()
|
||||||
|
client = MagicMock()
|
||||||
|
client.__aenter__ = AsyncMock(return_value=client)
|
||||||
|
client.__aexit__ = AsyncMock(return_value=False)
|
||||||
|
client.get = AsyncMock(return_value=resp)
|
||||||
|
with patch("roboco.services.git.httpx.AsyncClient", return_value=client):
|
||||||
|
assert await svc._pr_is_merged("acme", "repo", 42, "tok") is False
|
||||||
@@ -0,0 +1,114 @@
|
|||||||
|
"""On resume, the branch-mismatch chokepoint recovers the clone instead of
|
||||||
|
hard-failing.
|
||||||
|
|
||||||
|
A dev/documenter/QA clone is shared across tasks; on a respawn/resume it can sit
|
||||||
|
on a sibling task's branch, or a re-provisioned clone can lack the task branch as
|
||||||
|
a local ref (commits only on origin). `_assert_on_task_branch` used to raise
|
||||||
|
BRANCH_MISMATCH in that state, so the agent's next commit failed and the task
|
||||||
|
wedged in a blocked respawn loop (the documented resume deadlock — e.g. the
|
||||||
|
documenter PR #102 case). It now fetches + checks out the task branch (recreating
|
||||||
|
a missing local ref from origin) and only raises if it genuinely cannot switch
|
||||||
|
(uncommitted changes block it). It NEVER discards local commits — checkout, not
|
||||||
|
reset.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import AsyncMock, MagicMock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from roboco.services.base import ValidationError
|
||||||
|
from roboco.services.git import GitService
|
||||||
|
|
||||||
|
|
||||||
|
def _service() -> GitService:
|
||||||
|
session = MagicMock()
|
||||||
|
session.execute = AsyncMock(return_value=None)
|
||||||
|
return GitService(session)
|
||||||
|
|
||||||
|
|
||||||
|
def _bind(svc: GitService, name: str, value: object) -> None:
|
||||||
|
object.__setattr__(svc, name, value)
|
||||||
|
|
||||||
|
|
||||||
|
def _run_git_mock(*, local_ref_rc: int = 0, checkout_rc: int = 0) -> AsyncMock:
|
||||||
|
async def _run(_workspace: Path, args: list[str], **_kw: object) -> MagicMock:
|
||||||
|
if args[:2] == ["rev-parse", "--verify"]:
|
||||||
|
return MagicMock(returncode=local_ref_rc, stdout="")
|
||||||
|
if args[0] == "checkout":
|
||||||
|
return MagicMock(returncode=checkout_rc, stdout="")
|
||||||
|
return MagicMock(returncode=0, stdout="")
|
||||||
|
|
||||||
|
return AsyncMock(side_effect=_run)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_noop_when_already_on_task_branch() -> None:
|
||||||
|
svc = _service()
|
||||||
|
_bind(svc, "get_current_branch", AsyncMock(return_value="feature/main_pm/abc"))
|
||||||
|
run = _run_git_mock()
|
||||||
|
_bind(svc, "_run_git", run)
|
||||||
|
await svc._assert_on_task_branch(Path("/ws"), "feature/main_pm/abc")
|
||||||
|
run.assert_not_awaited() # already on it → no git work, no raise
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_noop_when_task_branch_none() -> None:
|
||||||
|
svc = _service()
|
||||||
|
gcb = AsyncMock(return_value="whatever")
|
||||||
|
_bind(svc, "get_current_branch", gcb)
|
||||||
|
await svc._assert_on_task_branch(Path("/ws"), None)
|
||||||
|
gcb.assert_not_awaited()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_recovers_by_checkout_when_local_ref_present() -> None:
|
||||||
|
svc = _service()
|
||||||
|
_bind(
|
||||||
|
svc,
|
||||||
|
"get_current_branch",
|
||||||
|
AsyncMock(side_effect=["feature/other--leaf", "feature/main_pm/abc"]),
|
||||||
|
)
|
||||||
|
_bind(svc, "_token_for_workspace", AsyncMock(return_value="tok"))
|
||||||
|
run = _run_git_mock(local_ref_rc=0, checkout_rc=0)
|
||||||
|
_bind(svc, "_run_git", run)
|
||||||
|
|
||||||
|
await svc._assert_on_task_branch(Path("/ws"), "feature/main_pm/abc")
|
||||||
|
|
||||||
|
cmds = [c.args[1] for c in run.await_args_list]
|
||||||
|
assert ["checkout", "feature/main_pm/abc"] in cmds
|
||||||
|
# local ref present → no recovery fetch/branch-create
|
||||||
|
assert not any(c[0] == "fetch" for c in cmds)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_recovers_missing_local_ref_from_origin() -> None:
|
||||||
|
svc = _service()
|
||||||
|
_bind(
|
||||||
|
svc,
|
||||||
|
"get_current_branch",
|
||||||
|
AsyncMock(side_effect=["feature/other--leaf", "feature/main_pm/abc"]),
|
||||||
|
)
|
||||||
|
_bind(svc, "_token_for_workspace", AsyncMock(return_value="tok"))
|
||||||
|
run = _run_git_mock(local_ref_rc=1, checkout_rc=0) # local ref missing
|
||||||
|
_bind(svc, "_run_git", run)
|
||||||
|
|
||||||
|
await svc._assert_on_task_branch(Path("/ws"), "feature/main_pm/abc")
|
||||||
|
|
||||||
|
cmds = [c.args[1] for c in run.await_args_list]
|
||||||
|
assert ["fetch", "origin", "feature/main_pm/abc"] in cmds
|
||||||
|
assert ["branch", "feature/main_pm/abc", "origin/feature/main_pm/abc"] in cmds
|
||||||
|
assert ["checkout", "feature/main_pm/abc"] in cmds
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_raises_when_cannot_switch() -> None:
|
||||||
|
svc = _service()
|
||||||
|
_bind(svc, "get_current_branch", AsyncMock(return_value="feature/other--leaf"))
|
||||||
|
_bind(svc, "_token_for_workspace", AsyncMock(return_value="tok"))
|
||||||
|
run = _run_git_mock(local_ref_rc=0, checkout_rc=1) # checkout fails (dirty tree)
|
||||||
|
_bind(svc, "_run_git", run)
|
||||||
|
|
||||||
|
with pytest.raises(ValidationError, match="BRANCH_MISMATCH"):
|
||||||
|
await svc._assert_on_task_branch(Path("/ws"), "feature/main_pm/abc")
|
||||||
Reference in New Issue
Block a user