Fix/run hardening prep (#263)

* fix(git): don't delete a branch that still has open dependent PRs

Root cause of the run-zombifying "integration branch gone from origin" wedge.
_delete_remote_branch_best_effort deleted a merged PR's head branch
unconditionally, so:
- merging a cell->root PR deleted the cell branch while a sibling leaf PR was
  still targeting it as base, and
- the CEO's root->master merge deleted the feature/main_pm/{root} integration
  branch.
The dependent PRs lost their base, every later git op against the vanished
branch failed, and the task zombified (a51c3d31 only made the post-merge sync
non-fatal; this removes the cause).

The remote-branch delete chokepoint (the single path all merge/close/cancel
deletions funnel through) now first checks _branch_has_open_dependents: any OPEN
PR targeting the branch as its base marks it an active integration target and
preserves it. Fails safe (any error => keep the branch; cleanup is best-effort,
stranding is not). True leaf branches with no open dependents are still cleaned
up. Adds 6 unit tests for the guard + the probe.

* fix(git): recover a drifted shared clone on resume instead of BRANCH_MISMATCH

A dev/documenter/QA clone is shared across that agent's tasks. On a
respawn/resume it can sit on a sibling task's branch, or a re-provisioned clone
can lack the task branch as a local ref (commits only on origin). The
fresh-claim path git-resets the clone clean, but resume deliberately
short-circuits before it (_dev_reentry), so the agent's next commit hit
_assert_on_task_branch's BRANCH_MISMATCH, failed, and the task wedged in a
blocked respawn loop (the documenter that could never land its doc commit).

_assert_on_task_branch now recovers instead of only rejecting: fetch + checkout
the task branch (recreating a missing local ref from origin via `git branch
<b> origin/<b>`), and raise only when the switch genuinely can't happen
(uncommitted changes block it). Never discards work — checkout, not reset — so
a resumed agent's unpushed commits are preserved. Updates the RAG troubleshooting
+ developer docs to describe the auto-recovery. Adds 5 unit tests.

* fix(runtime): re-adopt running agent containers on restart (no double-spawn)

An orchestrator restart loses the in-memory _instances registry while the agent
containers keep running. The reaper already had a Docker-liveness fallback
(_assignee_container_running), but the spawn gate (_is_agent_active) did not, so
right after a restart it saw a live agent as inactive and could launch a second
container onto work the forgotten-but-running one was already doing.

start() now calls _readopt_running_agents() after _reconcile_orphan_claims_on_startup
and before the dispatcher/reaper loops launch: it probes each known agent slug's
container (AGENT_IMAGES, reusing _inspect_container_state — the same docker
inspect the reaper uses) and registers a minimal AgentInstance(state=ACTIVE) for
any that is running and not already tracked. Inert when nothing runs (cold start
unchanged); best-effort (a probe error leaves that slot for the reaper's own
fallback). This is the gateway-health spec's Task 4 / the orchestrator-state
spec's Phase 3 (_instances reconcile). Adds 4 unit tests.

* fix(git): treat an already-merged PR as idempotent success on merge

A merge PUT against an already-merged PR returns the same 405 as a genuine
"not mergeable" conflict, so _merge_with_retry raised MergeConflictError and the
completion path tried to rebase / close-superseded / escalate a PR that had
already landed (a prior cycle, a sibling, or the CEO merged it) — the
cell_pm_complete block<->unblock respawn loop.

_merge_with_retry now disambiguates before raising: a new _pr_is_merged probe
(GET the PR, check merged==true) returns success on an already-merged PR so
completion proceeds idempotently; a genuinely-unmerged 405 still raises the
conflict. Best-effort probe (False on any error → falls through to the existing
conflict handling). Adds 4 unit tests.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
2026-06-25 18:35:52 +02:00
committed by GitHub
co-authored by Renn F
parent 99cf56dff3
commit 2c403c77a2
9 changed files with 620 additions and 25 deletions
+8
View File
@@ -8,6 +8,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
### Fixed
- **Completing a task whose PR is already merged no longer loops.** A merge request against an already-merged PR returns the same `405` from GitHub as a genuine "not mergeable" conflict, so the completion path treated an already-landed PR as a conflict and tried to rebase / close-superseded / escalate it — bouncing the task between blocked and unblocked forever (the case where a prior cycle, a sibling, or the CEO had already merged it). The merge now disambiguates: if the PR reports as merged, the merge is treated as idempotent success and completion proceeds; only a PR that is genuinely unmerged raises the conflict.
- **After an orchestrator restart, a still-running agent is no longer double-spawned.** The orchestrator's in-memory instance registry is lost on a restart while the agent containers keep running. The stale-claim reaper already had a Docker-liveness fallback for that, but the spawn gate (`_is_agent_active`) did not — so right after a restart it saw a live agent as inactive and could launch a second container onto the work the forgotten-but-running one was already doing. Startup now re-adopts surviving containers: it probes each known agent slug's container (the same `docker inspect` the reaper uses) and re-registers a minimal active instance for any that is running, before the dispatcher and reaper loops start. Inert when nothing is running, and best-effort (a probe error just leaves that slot for the reaper's own fallback to cover).
- **A resumed agent on a drifted shared clone no longer wedges with `BRANCH_MISMATCH`.** A dev/documenter/QA clone is shared across that agent's tasks; on a respawn/resume it can sit on a sibling task's branch, or a re-provisioned clone can lack the task branch as a local ref (its commits are only on origin). The fresh-claim path git-resets the clone clean, but resume deliberately short-circuits before it — so the agent's next `commit` hit the branch-mismatch guard, failed, and the task wedged in a blocked respawn loop (e.g. the documenter that could never land its doc commit). The guard now *recovers* instead of only rejecting: it fetches and checks out the task's branch (recreating a missing local ref from origin) and only raises when it genuinely cannot switch — i.e. uncommitted changes block it. It never discards work (checkout, not reset), so a resumed agent's unpushed commits are preserved.
- **An integration branch is no longer deleted out from under in-flight work (the "branch gone from origin" zombification).** After a PR merged, the post-merge cleanup deleted its head branch unconditionally — so merging a cell→root PR deleted the cell branch while a sibling leaf PR was still targeting it as its base, and the CEO's root→master merge deleted the `feature/main_pm/{root}` integration branch. The dependent PRs then had no base, every later git op against the vanished branch failed, and the task zombified (the symptom an earlier fix only made non-fatal). The remote-branch delete chokepoint now first checks whether any **open PR still targets the branch as its base** — an active integration target — and preserves it if so; it fails safe (on any error it keeps the branch, since cleanup is best-effort but stranding is not). True leaf branches with no open dependents are still cleaned up as before.
- **Mypy [unreachable] error in test_pr_gate_records_verdict resolved.** A test assigned `t.notes_structured = None` in the function body, causing mypy to narrow the attribute type to `None`. Since the test's helper function took the object as `Any`, mypy did not reset its narrowing after the call, treating `assert t.notes_structured is not None` as statically always-False and marking the next line as `[unreachable]`, failing the quality gate. Fixed by introducing `_TaskWithNoNotes` — a helper class that declares `notes_structured: dict[str, Any] | None = None` in `__init__` — so mypy uses the declared union type rather than a narrowed literal. All tests pass with no suppressions. This pattern is documented in the testing standards for future reference.
## [0.11.1] - 2026-06-25