Leak fixes, gate green again, uv/CI hardening, e2e lifecycle smoke harness (#294)

* test: align phase1 smoke mock with the armed team-match gate

The 8e5f84c4 sweep fixed 13 test files' inconsistent-team mocks but ran
only the gateway/foundation/runtime subsets; the full gate caught this
integration mock whose parent task carried an auto-generated MagicMock
team and died on not_authorized before the incomplete_input assertion.

* fix(runtime): attribute every agent.spawned audit to its dispatcher

A rogue spawner could not be identified live (2026-07-02): agent.spawned
rows carry container/model but not which dispatch loop launched them.
spawn_agent now takes spawned_by, stamps it into the spawned/spawn_failed
audit details, every call site passes its loop name, and an AST sweep
test holds future callers to it.

* fix(api): admin-complete refuses when the task's PR is still open

PATCH status=completed on a task with an OPEN PR stranded its commits
unmerged (bit the CEO twice live 2026-07-02). The override now refuses
with the PR number/URL and the consequence before the generic hatch
text; force:true stays the deliberate, audited escape.

* fix(panel): awaiting_ceo_approval offers the working ceo-approve path

The header's only approve action was Approve & Merge (POST
/approve-and-merge, no notes) which 400s NO_PR on a branchless MegaTask
umbrella — the CEO's approve button just failed. Primary action is now
Approve & Complete via the CeoApproveDialog (POST /ceo-approve, notes
>=20 chars, proven live); Approve & Merge stays for PR-bearing tasks.

* test: stop leaking self-heal + rate-limit state into live Redis

Two test files wrote real keys into a developer's localhost Redis:
self-heal originate tests left self_heal:notified:* (2h TTL) and the
i_am_blocked rate-limited tests left a NO-TTL 'anthropic rate-limited'
tracker blob — order/state-dependent poison for anything reading the
real tracker, and the prime suspect class for the one-off
test_self_heal_engine full-run failure (not reproduced in 5x dir runs,
adversarial orders, and a green full gate). Both files now point the
computed redis_url at an unreachable port; the engines' fail-open paths
keep every assertion intact. Leaked keys scrubbed live.

* docs: changelog + map delta for the leak-fix batch; mypy-clean attribution test

The attribution test's direct method assignments tripped the full gate's
mypy (method-assign) — switched to the house monkeypatch idiom, no
suppressions.

* fix(gate): clear the ten xenon C-ranks; isolate all tests from live Redis

Master CI has been red at the phase1 smoke test, so neither CI nor a
local full gate had reached the xenon step since the team-match sweep —
whose inline 'agent_team=str(agent.team) if ...' kwarg pushed nine verb
bodies from B(10) to C(11-12) unseen. A shared actor_context_fields()
(_protocol.py) computes (actor_slug, agent_team) once per verb, restoring
all nine to B with zero behavior change; the new admin-complete override
helper extraction does the same for routes/tasks.py.

tests/conftest.py gains an autouse fixture pointing the computed
redis_url at an unreachable port for every test — the root fix for the
three families caught writing live-Redis keys (self-heal dedupe,
rate-limit tracker, notification purpose-dedupe); no test uses a real
Redis, and every production path is fail-open by design.

* refactor(runtime): delete the never-wired dispatch-time spawn cooldown

_safe_spawn / gateway_pre_spawn_check / trigger_filter had no caller in
the repo's entire history (87ef42bf only flipped the flag). Its five
rules are superseded: provider parking runs inside spawn_agent, claim
freshness is the guards+reaper, runaway respawns are the progress-aware
breaker + notification cooldown; the per-task cooldown rule would
queue-stall every normal stage handoff if wired today. gateway_triggers
table kept inert. Ratified by the CEO over wiring it.

* build: serialize uv — gate recipes never implicitly sync the venv

Every uv run re-syncs implicitly, so a background make quality plus any
foreground uv run raced two writers on one .venv and tore site-packages
apart (the recurring rich/pip/bandit ImportError corruption; bit twice
today, four times on 2026-07-02's first session). UV_NO_SYNC=1 is now
exported Makefile-wide and quality/quality-fast/gate depend on one
explicit up-front sync step.

* fix(git): PR/merge/branch REST calls honor github_api_base_url

Fifteen sites hardcoded https://api.github.com while the CI-run and
open-PR-list calls already read settings.github_api_base_url — a GHE or
test override silently applied to half the surface. One _api_base()
helper keeps them uniform; default behavior unchanged.

* ci: split the monolith — backend CI, Panel CI, E2E Smoke

ci.yml keeps its file name and the backend quality job only (self-heal /
ci-watch / release-readiness default to the ci.yml workflow); the panel
job moves to panel-ci.yml scoped to panel/**, and the new scripted-agent
lifecycle smoke gets e2e-smoke.yml + a make e2e-smoke target (env-gated
out of the default pytest run). Trade: a panel-only red now lands on
Panel CI, which the ci.yml-pinned watch engines don't see.

* feat(tests): e2e lifecycle smoke harness — scripted agents, real gates

tests/e2e_smoke stands up the real API (flow/do routers + middleware on
uvicorn) over the ephemeral test Postgres, a local bare origin standing
in for GitHub, and a fake GitHub REST layer whose merges are real git
merges. A deterministic driver reloads the real MCP flow/do modules per
agent and walks claim (real clone + worktree) -> tracing-gap -> note ->
plan gate -> commit -> PR -> the full i_am_done ladder -> QA verdicts ->
documenter -> awaiting_pm_review in ~5s. Runs via make e2e-smoke + its
own CI workflow; skipped (env-gated) in the default suite. The
freeze-lift condition's first half: scenario 1 green.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
2026-07-02 18:28:07 +02:00
committed by GitHub
co-authored by Renn F
parent fe67a630ac
commit 1c87a4e4e4
35 changed files with 1661 additions and 835 deletions
@@ -0,0 +1,134 @@
"""Spawner attribution: every ``agent.spawned`` audit row names its dispatcher.
During the 2026-07-02 live run a rogue spawner could not be identified from the
audit log — ``agent.spawned`` rows carry the container/model but not WHICH
dispatch loop launched them. ``spawn_agent`` now takes ``spawned_by`` and
stamps it into the ``agent.spawned`` / ``agent.spawn_failed`` details, and an
AST sweep holds every call site to passing it.
"""
from __future__ import annotations
import ast
import asyncio
from pathlib import Path
from typing import Any
from unittest.mock import AsyncMock
import pytest
from roboco.models.runtime import AgentInstance
from roboco.runtime.orchestrator import AgentConfig, AgentOrchestrator, AgentState
REPO_ROOT = Path(__file__).resolve().parents[3]
def _make_orchestrator(
monkeypatch: pytest.MonkeyPatch,
captured: list[dict[str, Any]],
container_result: Any,
) -> AgentOrchestrator:
orch = AgentOrchestrator.__new__(AgentOrchestrator)
orch._instances = {}
orch._lock = asyncio.Lock()
orch._bg_tasks = set()
orch._running = True
monkeypatch.setattr(orch, "_fire_audit", lambda **kw: captured.append(kw))
monkeypatch.setattr(orch, "_record_spawn_session", AsyncMock(return_value=None))
monkeypatch.setattr(orch, "_spawn_container", container_result)
return orch
def _config_and_instance() -> tuple[AgentConfig, AgentInstance]:
config = AgentConfig(
agent_id="be-dev-1",
blueprint_path=Path(),
model="opus",
provider_type="anthropic",
)
instance = AgentInstance(
agent_id="be-dev-1", state=AgentState.STARTING, config=config
)
return config, instance
@pytest.mark.asyncio
async def test_launch_spawn_audit_carries_spawned_by(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""``agent.spawned`` details must name the dispatcher that launched it."""
captured: list[dict[str, Any]] = []
orch = _make_orchestrator(
monkeypatch, captured, AsyncMock(return_value="c0ffee" * 11)
)
config, instance = _config_and_instance()
await orch._launch_spawn(
"task-1", config, instance, None, None, spawned_by="_dispatch_qa_work"
)
spawned = [c for c in captured if c["event_type"] == "agent.spawned"]
assert len(spawned) == 1
assert spawned[0]["details"]["spawned_by"] == "_dispatch_qa_work"
@pytest.mark.asyncio
async def test_spawn_failed_audit_carries_spawned_by(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A failed launch must attribute the spawner too — a rogue dispatcher
that keeps crashing containers is exactly the live-debug case."""
captured: list[dict[str, Any]] = []
orch = _make_orchestrator(
monkeypatch, captured, AsyncMock(side_effect=RuntimeError("boom"))
)
config, instance = _config_and_instance()
with pytest.raises(RuntimeError):
await orch._launch_spawn(
"task-1", config, instance, None, None, spawned_by="_spawn_pending_dev"
)
failed = [c for c in captured if c["event_type"] == "agent.spawn_failed"]
assert len(failed) == 1
assert failed[0]["details"]["spawned_by"] == "_spawn_pending_dev"
@pytest.mark.asyncio
async def test_launch_spawn_without_attribution_stamps_unspecified(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The field is always present so audit queries never KeyError."""
captured: list[dict[str, Any]] = []
orch = _make_orchestrator(
monkeypatch, captured, AsyncMock(return_value="c0ffee" * 11)
)
config, instance = _config_and_instance()
await orch._launch_spawn("task-1", config, instance, None, None)
spawned = [c for c in captured if c["event_type"] == "agent.spawned"]
assert spawned[0]["details"]["spawned_by"] == "unspecified"
def _spawn_agent_calls_missing_spawned_by(path: Path) -> list[str]:
"""Return ``file:line`` for spawn_agent() calls without a spawned_by kwarg."""
tree = ast.parse(path.read_text())
missing: list[str] = []
for node in ast.walk(tree):
if not isinstance(node, ast.Call):
continue
func = node.func
if not (isinstance(func, ast.Attribute) and func.attr == "spawn_agent"):
continue
if not any(kw.arg == "spawned_by" for kw in node.keywords):
missing.append(f"{path.name}:{node.lineno}")
return missing
def test_every_spawn_agent_call_site_passes_spawned_by() -> None:
"""Sweep-guard over the whole package: a dispatcher added without
attribution fails here, not in a 3am live-debug session."""
missing: list[str] = []
for path in sorted((REPO_ROOT / "roboco").rglob("*.py")):
missing.extend(_spawn_agent_calls_missing_spawned_by(path))
assert not missing, f"spawn_agent() calls missing spawned_by=: {missing}"