mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
Leak fixes, gate green again, uv/CI hardening, e2e lifecycle smoke harness (#294)
* test: align phase1 smoke mock with the armed team-match gate The8e5f84c4sweep fixed 13 test files' inconsistent-team mocks but ran only the gateway/foundation/runtime subsets; the full gate caught this integration mock whose parent task carried an auto-generated MagicMock team and died on not_authorized before the incomplete_input assertion. * fix(runtime): attribute every agent.spawned audit to its dispatcher A rogue spawner could not be identified live (2026-07-02): agent.spawned rows carry container/model but not which dispatch loop launched them. spawn_agent now takes spawned_by, stamps it into the spawned/spawn_failed audit details, every call site passes its loop name, and an AST sweep test holds future callers to it. * fix(api): admin-complete refuses when the task's PR is still open PATCH status=completed on a task with an OPEN PR stranded its commits unmerged (bit the CEO twice live 2026-07-02). The override now refuses with the PR number/URL and the consequence before the generic hatch text; force:true stays the deliberate, audited escape. * fix(panel): awaiting_ceo_approval offers the working ceo-approve path The header's only approve action was Approve & Merge (POST /approve-and-merge, no notes) which 400s NO_PR on a branchless MegaTask umbrella — the CEO's approve button just failed. Primary action is now Approve & Complete via the CeoApproveDialog (POST /ceo-approve, notes >=20 chars, proven live); Approve & Merge stays for PR-bearing tasks. * test: stop leaking self-heal + rate-limit state into live Redis Two test files wrote real keys into a developer's localhost Redis: self-heal originate tests left self_heal:notified:* (2h TTL) and the i_am_blocked rate-limited tests left a NO-TTL 'anthropic rate-limited' tracker blob — order/state-dependent poison for anything reading the real tracker, and the prime suspect class for the one-off test_self_heal_engine full-run failure (not reproduced in 5x dir runs, adversarial orders, and a green full gate). Both files now point the computed redis_url at an unreachable port; the engines' fail-open paths keep every assertion intact. Leaked keys scrubbed live. * docs: changelog + map delta for the leak-fix batch; mypy-clean attribution test The attribution test's direct method assignments tripped the full gate's mypy (method-assign) — switched to the house monkeypatch idiom, no suppressions. * fix(gate): clear the ten xenon C-ranks; isolate all tests from live Redis Master CI has been red at the phase1 smoke test, so neither CI nor a local full gate had reached the xenon step since the team-match sweep — whose inline 'agent_team=str(agent.team) if ...' kwarg pushed nine verb bodies from B(10) to C(11-12) unseen. A shared actor_context_fields() (_protocol.py) computes (actor_slug, agent_team) once per verb, restoring all nine to B with zero behavior change; the new admin-complete override helper extraction does the same for routes/tasks.py. tests/conftest.py gains an autouse fixture pointing the computed redis_url at an unreachable port for every test — the root fix for the three families caught writing live-Redis keys (self-heal dedupe, rate-limit tracker, notification purpose-dedupe); no test uses a real Redis, and every production path is fail-open by design. * refactor(runtime): delete the never-wired dispatch-time spawn cooldown _safe_spawn / gateway_pre_spawn_check / trigger_filter had no caller in the repo's entire history (87ef42bfonly flipped the flag). Its five rules are superseded: provider parking runs inside spawn_agent, claim freshness is the guards+reaper, runaway respawns are the progress-aware breaker + notification cooldown; the per-task cooldown rule would queue-stall every normal stage handoff if wired today. gateway_triggers table kept inert. Ratified by the CEO over wiring it. * build: serialize uv — gate recipes never implicitly sync the venv Every uv run re-syncs implicitly, so a background make quality plus any foreground uv run raced two writers on one .venv and tore site-packages apart (the recurring rich/pip/bandit ImportError corruption; bit twice today, four times on 2026-07-02's first session). UV_NO_SYNC=1 is now exported Makefile-wide and quality/quality-fast/gate depend on one explicit up-front sync step. * fix(git): PR/merge/branch REST calls honor github_api_base_url Fifteen sites hardcoded https://api.github.com while the CI-run and open-PR-list calls already read settings.github_api_base_url — a GHE or test override silently applied to half the surface. One _api_base() helper keeps them uniform; default behavior unchanged. * ci: split the monolith — backend CI, Panel CI, E2E Smoke ci.yml keeps its file name and the backend quality job only (self-heal / ci-watch / release-readiness default to the ci.yml workflow); the panel job moves to panel-ci.yml scoped to panel/**, and the new scripted-agent lifecycle smoke gets e2e-smoke.yml + a make e2e-smoke target (env-gated out of the default pytest run). Trade: a panel-only red now lands on Panel CI, which the ci.yml-pinned watch engines don't see. * feat(tests): e2e lifecycle smoke harness — scripted agents, real gates tests/e2e_smoke stands up the real API (flow/do routers + middleware on uvicorn) over the ephemeral test Postgres, a local bare origin standing in for GitHub, and a fake GitHub REST layer whose merges are real git merges. A deterministic driver reloads the real MCP flow/do modules per agent and walks claim (real clone + worktree) -> tracing-gap -> note -> plan gate -> commit -> PR -> the full i_am_done ladder -> QA verdicts -> documenter -> awaiting_pm_review in ~5s. Runs via make e2e-smoke + its own CI workflow; skipped (env-gated) in the default suite. The freeze-lift condition's first half: scenario 1 green. --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
@@ -0,0 +1,45 @@
|
||||
"""e2e lifecycle smoke harness — collection gate + the stack fixture.
|
||||
|
||||
Scripted-agent smoke: an in-process RoboCo API (real routers, real
|
||||
middleware, real gateway/choreographer/services) over the ephemeral test
|
||||
Postgres, a local bare git origin standing in for GitHub, and a fake
|
||||
GitHub REST layer whose merges are REAL git merges on that origin. A
|
||||
deterministic driver calls the REAL MCP flow/do tool functions — no LLM
|
||||
anywhere — so seam bugs (tool↔gate schema drift, squash merges, stale
|
||||
refs, workspace routing) die here instead of in a live run.
|
||||
|
||||
Gating: excluded from the default suite (`make quality`); runs via
|
||||
`make e2e-smoke` (sets ROBOCO_E2E_SMOKE=1). Needs the test Postgres
|
||||
reachable and git on PATH, nothing else.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import pytest
|
||||
from tests.e2e_smoke.harness import build_e2e_stack
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterator
|
||||
|
||||
from tests.e2e_smoke.harness import E2EStack
|
||||
|
||||
|
||||
def pytest_collection_modifyitems(
|
||||
config: pytest.Config, items: list[pytest.Item]
|
||||
) -> None:
|
||||
if os.environ.get("ROBOCO_E2E_SMOKE") == "1":
|
||||
return
|
||||
skip = pytest.mark.skip(reason="e2e smoke runs via `make e2e-smoke` only")
|
||||
for item in items:
|
||||
if "tests/e2e_smoke" in str(item.path):
|
||||
item.add_marker(skip)
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def e2e_stack(
|
||||
_test_database_url: str, tmp_path_factory: pytest.TempPathFactory
|
||||
) -> Iterator[E2EStack]:
|
||||
yield from build_e2e_stack(_test_database_url, tmp_path_factory)
|
||||
@@ -0,0 +1,472 @@
|
||||
"""e2e smoke harness — in-process RoboCo stack + scripted-agent driver.
|
||||
|
||||
Pieces (all REAL except GitHub and the LLM):
|
||||
|
||||
- The API: the real v1 flow/do routers + real middleware/exception handlers,
|
||||
served by uvicorn in a thread, over the ephemeral test Postgres (the app's
|
||||
own lazy engine is pointed at it by patching ``settings.database_*`` and
|
||||
resetting ``_DbHolder``).
|
||||
- Git: a local bare origin whose path CONTAINS ``github.com/<owner>/<repo>``
|
||||
— ``_parse_git_url`` extracts owner/repo from it while clone/fetch/push
|
||||
run tokenless over the local protocol.
|
||||
- GitHub REST: a fake ``/_github`` router mounted on the same app
|
||||
(``settings.github_api_base_url`` points at it). PR state lives in memory;
|
||||
merges perform REAL git merges (squash included) on the bare origin, so
|
||||
downstream git logic (cherry checks, freshness, branch sync) sees reality.
|
||||
- Agents: ``ScriptedAgent`` reloads the REAL ``roboco.mcp.flow_server`` /
|
||||
``do_server`` modules with that agent's env (id, role, role-scoped
|
||||
manifest built from the real ``role_config``) and calls the REAL tool
|
||||
functions, which POST to the in-process API over loopback HTTP.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import importlib
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
from contextlib import suppress
|
||||
from dataclasses import dataclass, field
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import pytest
|
||||
import uvicorn
|
||||
from cryptography.fernet import Fernet
|
||||
from fastapi import APIRouter, FastAPI, Request
|
||||
from fastapi.responses import JSONResponse
|
||||
from sqlalchemy.engine.url import make_url
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from types import ModuleType
|
||||
from uuid import UUID
|
||||
|
||||
_OWNER = "e2e-smoke"
|
||||
_REPO = "proj"
|
||||
|
||||
|
||||
def _git(cwd: Path, *args: str) -> str:
|
||||
res = subprocess.run(
|
||||
["git", "-C", str(cwd), *args],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
return res.stdout.strip()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fake GitHub REST — PR state in memory, merges as REAL git ops on the origin
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class _FakeGitHub:
|
||||
origin: Path
|
||||
admin_clone: Path
|
||||
prs: dict[int, dict[str, Any]] = field(default_factory=dict)
|
||||
comments: list[dict[str, Any]] = field(default_factory=list)
|
||||
next_number: int = 1
|
||||
|
||||
def create_pr(self, title: str, body: str, head: str, base: str) -> dict[str, Any]:
|
||||
number = self.next_number
|
||||
self.next_number += 1
|
||||
pr = {
|
||||
"number": number,
|
||||
"html_url": f"https://github.com/{_OWNER}/{_REPO}/pull/{number}",
|
||||
"title": title,
|
||||
"body": body,
|
||||
"state": "open",
|
||||
"merged": False,
|
||||
"head": {
|
||||
"ref": head,
|
||||
"sha": self._sha_of(head),
|
||||
"repo": {"full_name": f"{_OWNER}/{_REPO}"},
|
||||
},
|
||||
"base": {"ref": base},
|
||||
"user": {"login": "e2e-bot"},
|
||||
"author_association": "MEMBER",
|
||||
}
|
||||
self.prs[number] = pr
|
||||
return pr
|
||||
|
||||
def _sha_of(self, branch: str) -> str:
|
||||
try:
|
||||
return _git(self.origin, "rev-parse", branch)
|
||||
except subprocess.CalledProcessError:
|
||||
return "0" * 40
|
||||
|
||||
def merge_pr(self, number: int, merge_method: str) -> dict[str, Any]:
|
||||
pr = self.prs[number]
|
||||
head, base = pr["head"]["ref"], pr["base"]["ref"]
|
||||
admin = self.admin_clone
|
||||
_git(admin, "fetch", "origin", "--prune")
|
||||
_git(admin, "checkout", "-B", base, f"origin/{base}")
|
||||
if merge_method == "squash":
|
||||
_git(admin, "merge", "--squash", f"origin/{head}")
|
||||
_git(admin, "commit", "-m", f"{pr['title']} (#{number})")
|
||||
else:
|
||||
_git(
|
||||
admin,
|
||||
"merge",
|
||||
"--no-ff",
|
||||
"-m",
|
||||
f"Merge pull request #{number} from {head}",
|
||||
f"origin/{head}",
|
||||
)
|
||||
_git(admin, "push", "origin", base)
|
||||
sha = _git(admin, "rev-parse", "HEAD")
|
||||
pr["merged"] = True
|
||||
pr["state"] = "closed"
|
||||
return {
|
||||
"merged": True,
|
||||
"sha": sha,
|
||||
"message": "Pull Request successfully merged",
|
||||
}
|
||||
|
||||
def open_prs(self, head: str | None, base: str | None) -> list[dict[str, Any]]:
|
||||
out = []
|
||||
for pr in self.prs.values():
|
||||
if pr["state"] != "open":
|
||||
continue
|
||||
if head and pr["head"]["ref"] != head.split(":", 1)[-1]:
|
||||
continue
|
||||
if base and pr["base"]["ref"] != base:
|
||||
continue
|
||||
out.append(pr)
|
||||
return out
|
||||
|
||||
|
||||
def _fake_github_router(gh: _FakeGitHub) -> APIRouter:
|
||||
r = APIRouter(prefix="/_github")
|
||||
|
||||
@r.get("/repos/{owner}/{repo}")
|
||||
async def repo_caps(owner: str, repo: str) -> dict[str, Any]:
|
||||
return {
|
||||
"allow_squash_merge": True,
|
||||
"allow_merge_commit": True,
|
||||
"allow_rebase_merge": False,
|
||||
}
|
||||
|
||||
@r.get("/repos/{owner}/{repo}/pulls/{number}")
|
||||
async def get_pr(owner: str, repo: str, number: int) -> JSONResponse:
|
||||
pr = gh.prs.get(number)
|
||||
if pr is None:
|
||||
return JSONResponse({"message": "Not Found"}, status_code=404)
|
||||
return JSONResponse(pr)
|
||||
|
||||
@r.get("/repos/{owner}/{repo}/pulls")
|
||||
async def list_prs(
|
||||
owner: str,
|
||||
repo: str,
|
||||
head: str | None = None,
|
||||
base: str | None = None,
|
||||
state: str = "open",
|
||||
) -> list[dict[str, Any]]:
|
||||
return gh.open_prs(head, base)
|
||||
|
||||
@r.post("/repos/{owner}/{repo}/pulls", status_code=201)
|
||||
async def create_pr(owner: str, repo: str, request: Request) -> dict[str, Any]:
|
||||
body = await request.json()
|
||||
return gh.create_pr(
|
||||
body["title"], body.get("body", ""), body["head"], body["base"]
|
||||
)
|
||||
|
||||
@r.patch("/repos/{owner}/{repo}/pulls/{number}")
|
||||
async def patch_pr(
|
||||
owner: str, repo: str, number: int, request: Request
|
||||
) -> JSONResponse:
|
||||
pr = gh.prs.get(number)
|
||||
if pr is None:
|
||||
return JSONResponse({"message": "Not Found"}, status_code=404)
|
||||
body = await request.json()
|
||||
for key in ("title", "body", "state"):
|
||||
if key in body:
|
||||
pr[key] = body[key]
|
||||
return JSONResponse(pr)
|
||||
|
||||
@r.put("/repos/{owner}/{repo}/pulls/{number}/merge")
|
||||
async def merge_pr(
|
||||
owner: str, repo: str, number: int, request: Request
|
||||
) -> JSONResponse:
|
||||
if number not in gh.prs:
|
||||
return JSONResponse({"message": "Not Found"}, status_code=404)
|
||||
body = await request.json()
|
||||
try:
|
||||
result = gh.merge_pr(number, body.get("merge_method", "merge"))
|
||||
except subprocess.CalledProcessError as exc:
|
||||
return JSONResponse(
|
||||
{"message": f"Merge conflict: {exc.stderr}"}, status_code=409
|
||||
)
|
||||
return JSONResponse(result)
|
||||
|
||||
@r.post("/repos/{owner}/{repo}/pulls/{number}/requested_reviewers", status_code=201)
|
||||
async def request_reviewers(
|
||||
owner: str, repo: str, number: int, request: Request
|
||||
) -> dict[str, Any]:
|
||||
return gh.prs.get(number, {})
|
||||
|
||||
@r.post("/repos/{owner}/{repo}/issues/{number}/comments", status_code=201)
|
||||
async def comment(
|
||||
owner: str, repo: str, number: int, request: Request
|
||||
) -> dict[str, Any]:
|
||||
gh.comments.append({"number": number, "body": (await request.json())})
|
||||
return {"id": len(gh.comments)}
|
||||
|
||||
@r.delete("/repos/{owner}/{repo}/git/refs/heads/{branch:path}", status_code=204)
|
||||
async def delete_branch(owner: str, repo: str, branch: str) -> None:
|
||||
with suppress(subprocess.CalledProcessError):
|
||||
_git(gh.origin, "branch", "-D", branch)
|
||||
|
||||
return r
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Stack: settings patches + origin + app + uvicorn thread
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class E2EStack:
|
||||
base_url: str
|
||||
root: Path
|
||||
origin: Path
|
||||
workspaces_root: Path
|
||||
db_url: str
|
||||
github: _FakeGitHub
|
||||
|
||||
def workspace_of(self, project_slug: str, team: str, agent_slug: str) -> Path:
|
||||
return self.workspaces_root / project_slug / team / agent_slug
|
||||
|
||||
def run_db(self, coro_fn: Any) -> Any:
|
||||
"""Run ``coro_fn(session)`` against a fresh engine/session and return."""
|
||||
|
||||
async def _run() -> Any:
|
||||
engine = create_async_engine(self.db_url)
|
||||
factory = async_sessionmaker(engine, expire_on_commit=False)
|
||||
try:
|
||||
async with factory() as session:
|
||||
result = await coro_fn(session)
|
||||
await session.commit()
|
||||
return result
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
return asyncio.run(_run())
|
||||
|
||||
|
||||
def _free_port() -> int:
|
||||
with socket.socket() as s:
|
||||
s.bind(("127.0.0.1", 0))
|
||||
return int(s.getsockname()[1])
|
||||
|
||||
|
||||
def _seed_origin(root: Path) -> Path:
|
||||
"""Bare origin at a path _parse_git_url can read owner/repo from."""
|
||||
origin = root / "github.com" / _OWNER / f"{_REPO}.git"
|
||||
origin.parent.mkdir(parents=True)
|
||||
subprocess.run(
|
||||
["git", "init", "--bare", "--initial-branch=master", str(origin)],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
)
|
||||
seed = root / "seed-clone"
|
||||
subprocess.run(
|
||||
["git", "clone", str(origin), str(seed)], check=True, capture_output=True
|
||||
)
|
||||
_git(seed, "config", "user.name", "roboco-e2e")
|
||||
_git(seed, "config", "user.email", "e2e@roboco.local")
|
||||
(seed / "README.md").write_text("# e2e smoke project\n")
|
||||
_git(seed, "add", "README.md")
|
||||
_git(seed, "commit", "-m", "Initial commit")
|
||||
_git(seed, "push", "origin", "master")
|
||||
return origin
|
||||
|
||||
|
||||
def _make_admin_clone(root: Path, origin: Path) -> Path:
|
||||
admin = root / "gh-admin-clone"
|
||||
subprocess.run(
|
||||
["git", "clone", str(origin), str(admin)], check=True, capture_output=True
|
||||
)
|
||||
_git(admin, "config", "user.name", "fake-github")
|
||||
_git(admin, "config", "user.email", "merge@github.local")
|
||||
return admin
|
||||
|
||||
|
||||
def _build_app(gh: _FakeGitHub) -> FastAPI:
|
||||
from roboco.api.middleware import setup_middleware
|
||||
from roboco.api.routes.health import router as health_router
|
||||
from roboco.api.routes.v1 import do as do_module
|
||||
from roboco.api.routes.v1 import flow_auditor as fa
|
||||
from roboco.api.routes.v1 import flow_board as fb
|
||||
from roboco.api.routes.v1 import flow_cell_pm as fcp
|
||||
from roboco.api.routes.v1 import flow_dev as fd
|
||||
from roboco.api.routes.v1 import flow_doc as fdoc
|
||||
from roboco.api.routes.v1 import flow_main_pm as fmp
|
||||
from roboco.api.routes.v1 import flow_pr_reviewer as fpr
|
||||
from roboco.api.routes.v1 import flow_qa as fq
|
||||
|
||||
app = FastAPI(title="roboco-e2e-smoke")
|
||||
setup_middleware(app)
|
||||
app.include_router(health_router)
|
||||
for module in (fd, fq, fdoc, fcp, fmp, fb, fa, fpr):
|
||||
app.include_router(module.router)
|
||||
app.include_router(do_module.router)
|
||||
app.include_router(_fake_github_router(gh))
|
||||
return app
|
||||
|
||||
|
||||
def build_e2e_stack(
|
||||
_test_database_url: str, tmp_path_factory: pytest.TempPathFactory
|
||||
) -> Iterator[E2EStack]:
|
||||
"""Generator behind the ``e2e_stack`` fixture (defined in conftest)."""
|
||||
from roboco.config import settings
|
||||
from roboco.db import base as db_base
|
||||
|
||||
mp = pytest.MonkeyPatch()
|
||||
root = tmp_path_factory.mktemp("e2e")
|
||||
origin = _seed_origin(root)
|
||||
admin = _make_admin_clone(root, origin)
|
||||
gh = _FakeGitHub(origin=origin, admin_clone=admin)
|
||||
workspaces = root / "workspaces"
|
||||
workspaces.mkdir()
|
||||
|
||||
url = make_url(_test_database_url)
|
||||
mp.setattr(settings, "database_host", url.host or "localhost")
|
||||
mp.setattr(settings, "database_port", url.port or 5432)
|
||||
mp.setattr(settings, "database_user", url.username or "")
|
||||
mp.setattr(settings, "database_password", url.password or "")
|
||||
mp.setattr(settings, "database_name", url.database or "")
|
||||
mp.setattr(settings, "workspaces_root", str(workspaces))
|
||||
mp.setattr(settings, "workspace_auto_clone", True)
|
||||
mp.setattr(settings, "encryption_key", Fernet.generate_key().decode())
|
||||
|
||||
# The app's lazy engine must bind to the patched settings, not a leftover.
|
||||
db_base._DbHolder.engine = None
|
||||
db_base._DbHolder.session_factory = None
|
||||
|
||||
port = _free_port()
|
||||
base_url = f"http://127.0.0.1:{port}"
|
||||
mp.setattr(settings, "github_api_base_url", f"{base_url}/_github")
|
||||
|
||||
app = _build_app(gh)
|
||||
server = uvicorn.Server(
|
||||
uvicorn.Config(app, host="127.0.0.1", port=port, log_level="warning")
|
||||
)
|
||||
thread = threading.Thread(target=server.run, daemon=True)
|
||||
thread.start()
|
||||
|
||||
import httpx
|
||||
|
||||
deadline = time.time() + 30
|
||||
while time.time() < deadline:
|
||||
try:
|
||||
# Any HTTP response at all means the server thread is up.
|
||||
httpx.get(f"{base_url}/health", timeout=1)
|
||||
break
|
||||
except httpx.HTTPError:
|
||||
time.sleep(0.1)
|
||||
else:
|
||||
raise RuntimeError("e2e app server did not become ready")
|
||||
|
||||
try:
|
||||
yield E2EStack(
|
||||
base_url=base_url,
|
||||
root=root,
|
||||
origin=origin,
|
||||
workspaces_root=workspaces,
|
||||
db_url=_test_database_url,
|
||||
github=gh,
|
||||
)
|
||||
finally:
|
||||
server.should_exit = True
|
||||
thread.join(timeout=10)
|
||||
db_base._DbHolder.engine = None
|
||||
db_base._DbHolder.session_factory = None
|
||||
mp.undo()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Scripted agents — the REAL MCP tool functions, per-agent module reloads
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class ScriptedAgent:
|
||||
"""Drives the real flow/do MCP tool functions as one seeded agent."""
|
||||
|
||||
def __init__(self, stack: E2EStack, agent_id: UUID, slug: str, role: str) -> None:
|
||||
self.stack = stack
|
||||
self.agent_id = agent_id
|
||||
self.slug = slug
|
||||
self.role = role
|
||||
self._manifest_path = stack.root / f"manifest-{slug}.json"
|
||||
self._manifest_path.write_text(json.dumps(self._manifest()))
|
||||
|
||||
def _manifest(self) -> dict[str, Any]:
|
||||
from roboco.services.gateway.role_config import get_role_config
|
||||
|
||||
cfg = get_role_config(self.role)
|
||||
return {
|
||||
"agent_id": str(self.agent_id),
|
||||
"role": self.role,
|
||||
"team": "backend",
|
||||
"workspace_path": str(self.stack.workspaces_root),
|
||||
"flow_tools": list(cfg.flow_tools),
|
||||
"do_tools": list(cfg.do_tools),
|
||||
"read_tools": ["Read", "Glob", "Grep"],
|
||||
"write_tools": ["Edit", "Write"] if cfg.allows_write else [],
|
||||
"bash_allowed": True,
|
||||
"subagent_allowed": False,
|
||||
"subagent_model": None,
|
||||
"env": {},
|
||||
}
|
||||
|
||||
def _module(self, name: str) -> ModuleType:
|
||||
os.environ["ROBOCO_AGENT_ID"] = str(self.agent_id)
|
||||
os.environ["ROBOCO_AGENT_ROLE"] = self.role
|
||||
os.environ["ROBOCO_ORCHESTRATOR_URL"] = self.stack.base_url
|
||||
os.environ["ROBOCO_TOOL_MANIFEST_PATH"] = str(self._manifest_path)
|
||||
module = importlib.import_module(name)
|
||||
if getattr(module, "AGENT_ID", None) != str(self.agent_id):
|
||||
module = importlib.reload(module)
|
||||
return module
|
||||
|
||||
def flow(self, verb: str, /, **kwargs: Any) -> dict[str, Any]:
|
||||
result: dict[str, Any] = getattr(self._module("roboco.mcp.flow_server"), verb)(
|
||||
**kwargs
|
||||
)
|
||||
return result
|
||||
|
||||
def do(self, tool: str, /, **kwargs: Any) -> dict[str, Any]:
|
||||
result: dict[str, Any] = getattr(self._module("roboco.mcp.do_server"), tool)(
|
||||
**kwargs
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def expect_error(env: dict[str, Any], kind: str, context: str) -> dict[str, Any]:
|
||||
"""Assert an envelope is the EXPECTED rejection kind."""
|
||||
assert env.get("error") == kind, (
|
||||
f"{context}: expected rejection {kind!r}, got error={env.get('error')!r}\n"
|
||||
f" full: {json.dumps(env, default=str, indent=2)[:4000]}"
|
||||
)
|
||||
return env
|
||||
|
||||
|
||||
def expect_ok(env: dict[str, Any], context: str) -> dict[str, Any]:
|
||||
"""Assert an envelope is a success; on failure show the whole envelope."""
|
||||
assert isinstance(env, dict), f"{context}: non-dict envelope: {env!r}"
|
||||
assert not env.get("error"), (
|
||||
f"{context}: rejected with error={env.get('error')!r}\n"
|
||||
f" message : {env.get('message')}\n"
|
||||
f" remediate: {env.get('remediate')}\n"
|
||||
f" missing : {env.get('missing')}\n"
|
||||
f" full : {json.dumps(env, default=str, indent=2)[:4000]}"
|
||||
)
|
||||
return env
|
||||
@@ -0,0 +1,369 @@
|
||||
"""Scenario 1: a leaf dev task walks claim → work → PR → QA → docs → PM queue.
|
||||
|
||||
Every hop goes through the REAL MCP tool functions → real HTTP → real
|
||||
gateway gates → real services → real git against the local origin, with a
|
||||
fake GitHub REST layer whose merges are real git merges. No LLM: this file
|
||||
IS the agent script, and every rejection envelope is printed verbatim so a
|
||||
seam regression names itself.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
from tests.e2e_smoke.harness import (
|
||||
E2EStack,
|
||||
ScriptedAgent,
|
||||
expect_error,
|
||||
expect_ok,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
pytestmark = pytest.mark.usefixtures("e2e_stack")
|
||||
|
||||
_PROJECT_SLUG = "e2e-proj"
|
||||
|
||||
|
||||
class _Company:
|
||||
dev_id: Any
|
||||
qa_id: Any
|
||||
doc_id: Any
|
||||
cell_pm_id: Any
|
||||
project_id: Any
|
||||
task_id: Any
|
||||
|
||||
|
||||
def _seed(stack: E2EStack) -> _Company:
|
||||
from roboco.db.tables import AgentTable, ProjectTable, TaskTable
|
||||
from roboco.models import AgentRole, AgentStatus, Team
|
||||
from roboco.models.base import Complexity, TaskNature, TaskStatus, TaskType
|
||||
from roboco.utils.crypto import encrypt_token
|
||||
|
||||
out = _Company()
|
||||
|
||||
async def _run(session: AsyncSession) -> None:
|
||||
def agent(slug: str, role: AgentRole) -> AgentTable:
|
||||
row = AgentTable(
|
||||
id=uuid4(),
|
||||
name=slug,
|
||||
slug=slug,
|
||||
role=role,
|
||||
team=Team.BACKEND,
|
||||
status=AgentStatus.ACTIVE,
|
||||
model_config={},
|
||||
system_prompt=slug,
|
||||
capabilities=[],
|
||||
permissions={},
|
||||
metrics={},
|
||||
)
|
||||
session.add(row)
|
||||
return row
|
||||
|
||||
dev = agent("be-dev-1", AgentRole.DEVELOPER)
|
||||
qa = agent("be-qa", AgentRole.QA)
|
||||
doc = agent("be-doc", AgentRole.DOCUMENTER)
|
||||
pm = agent("be-pm", AgentRole.CELL_PM)
|
||||
await session.flush()
|
||||
|
||||
project = ProjectTable(
|
||||
id=uuid4(),
|
||||
name="E2E Project",
|
||||
slug=_PROJECT_SLUG,
|
||||
git_url=str(stack.origin),
|
||||
default_branch="master",
|
||||
protected_branches=["master"],
|
||||
assigned_cell=Team.BACKEND,
|
||||
created_by=pm.id,
|
||||
is_active=True,
|
||||
git_token_encrypted=encrypt_token("e2e-dummy-token"),
|
||||
)
|
||||
session.add(project)
|
||||
await session.flush()
|
||||
|
||||
task = TaskTable(
|
||||
id=uuid4(),
|
||||
title="Add the greeting module",
|
||||
description=(
|
||||
"Create greeting.txt with a friendly greeting so the smoke "
|
||||
"harness has a real file change to commit, push, and merge."
|
||||
),
|
||||
acceptance_criteria=[
|
||||
"greeting.txt exists at the repo root",
|
||||
"its content greets the reader",
|
||||
],
|
||||
status=TaskStatus.PENDING,
|
||||
priority=2,
|
||||
task_type=TaskType.CODE,
|
||||
nature=TaskNature.TECHNICAL,
|
||||
estimated_complexity=Complexity.LOW,
|
||||
project_id=project.id,
|
||||
created_by=pm.id,
|
||||
team=Team.BACKEND,
|
||||
confirmed_by_human=True,
|
||||
# The pool→agent routing lane is the orchestrator dispatcher's
|
||||
# job (not under test here); a dev container is always spawned
|
||||
# with its task already routed, which give_me_work serves via
|
||||
# the pre-assigned-pending lane.
|
||||
assigned_to=dev.id,
|
||||
)
|
||||
session.add(task)
|
||||
await session.flush()
|
||||
|
||||
out.dev_id = dev.id
|
||||
out.qa_id = qa.id
|
||||
out.doc_id = doc.id
|
||||
out.cell_pm_id = pm.id
|
||||
out.project_id = project.id
|
||||
out.task_id = task.id
|
||||
|
||||
stack.run_db(_run)
|
||||
return out
|
||||
|
||||
|
||||
def _task_state(stack: E2EStack, task_id: Any) -> dict[str, Any]:
|
||||
from roboco.db.tables import TaskTable
|
||||
from sqlalchemy import select
|
||||
|
||||
async def _run(session: AsyncSession) -> dict[str, Any]:
|
||||
row = (
|
||||
await session.execute(select(TaskTable).where(TaskTable.id == task_id))
|
||||
).scalar_one()
|
||||
return {
|
||||
"status": str(row.status),
|
||||
"branch_name": row.branch_name,
|
||||
"pr_number": row.pr_number,
|
||||
"docs_complete": row.docs_complete,
|
||||
"assigned_to": row.assigned_to,
|
||||
}
|
||||
|
||||
state: dict[str, Any] = stack.run_db(_run)
|
||||
return state
|
||||
|
||||
|
||||
def test_leaf_dev_task_reaches_pm_review(e2e_stack: E2EStack) -> None:
|
||||
stack = e2e_stack
|
||||
ids = _seed(stack)
|
||||
task_id = str(ids.task_id)
|
||||
|
||||
# --- developer: discover, claim, work, PR, submit -----------------------
|
||||
dev = ScriptedAgent(stack, ids.dev_id, "be-dev-1", "developer")
|
||||
|
||||
env = expect_ok(dev.flow("give_me_work"), "dev give_me_work")
|
||||
assert env.get("task_id") == task_id, f"expected our task, got: {env}"
|
||||
|
||||
def _claim() -> dict:
|
||||
return dev.flow(
|
||||
"i_will_work_on",
|
||||
task_id=task_id,
|
||||
plan=(
|
||||
"Create greeting.txt at the repository root containing a "
|
||||
"friendly greeting, commit it on the task branch with the "
|
||||
"task-prefixed message, push the branch to origin, open the "
|
||||
"pull request against master, and self-verify both acceptance "
|
||||
"criteria by re-reading the committed file content."
|
||||
),
|
||||
steps=[
|
||||
{
|
||||
"title": "Write greeting.txt",
|
||||
"description": (
|
||||
"Create greeting.txt at the repo root containing a "
|
||||
"friendly greeting for the reader."
|
||||
),
|
||||
},
|
||||
{
|
||||
"title": "Commit and push",
|
||||
"description": (
|
||||
"Commit the new file on the task branch with a "
|
||||
"task-prefixed message and push it to origin."
|
||||
),
|
||||
},
|
||||
{
|
||||
"title": "Open PR and self-verify",
|
||||
"description": (
|
||||
"Open the pull request against master and re-read the "
|
||||
"file to confirm both acceptance criteria hold."
|
||||
),
|
||||
},
|
||||
],
|
||||
technical_considerations=["Plain text file; no build impact."],
|
||||
risks=[
|
||||
{
|
||||
"risk": "None of substance — purely additive file.",
|
||||
"mitigation": "Self-verify the file content before submit.",
|
||||
}
|
||||
],
|
||||
open_questions=[],
|
||||
)
|
||||
|
||||
# The composed claim succeeds and STAYS; the post-claim tracing gate
|
||||
# then demands the claim-time journal note — the real agent choreography
|
||||
# is claim → tracing_gap → note (now claim-held) → retry short-circuits.
|
||||
expect_error(_claim(), "tracing_gap", "dev first i_will_work_on")
|
||||
expect_ok(
|
||||
dev.do(
|
||||
"note",
|
||||
scope="note",
|
||||
task_id=task_id,
|
||||
text=(
|
||||
"Initial assessment: a single additive text file at the repo "
|
||||
"root satisfies both acceptance criteria; no existing code is "
|
||||
"touched, so risk is minimal and the plan is a three-step "
|
||||
"write/commit/PR sequence."
|
||||
),
|
||||
),
|
||||
"dev note at claim",
|
||||
)
|
||||
expect_ok(_claim(), "dev i_will_work_on retry")
|
||||
state = _task_state(stack, ids.task_id)
|
||||
assert state["status"] in ("claimed", "in_progress"), state
|
||||
assert state["branch_name"], f"claim did not set a branch: {state}"
|
||||
|
||||
workspace = stack.workspace_of(_PROJECT_SLUG, "backend", "be-dev-1")
|
||||
assert workspace.is_dir(), f"workspace clone missing at {workspace}"
|
||||
# F123: the agent works in the per-task worktree, not the clone root.
|
||||
workdir = workspace / ".worktrees" / task_id[:8]
|
||||
assert workdir.is_dir(), f"per-task worktree missing at {workdir}"
|
||||
(workdir / "greeting.txt").write_text("Hello from the e2e smoke agent!\n")
|
||||
|
||||
expect_ok(
|
||||
dev.do(
|
||||
"commit",
|
||||
message="Add greeting.txt with a friendly greeting",
|
||||
files=["greeting.txt"],
|
||||
),
|
||||
"dev commit",
|
||||
)
|
||||
expect_ok(
|
||||
dev.do(
|
||||
"note",
|
||||
scope="note",
|
||||
task_id=task_id,
|
||||
text=(
|
||||
"greeting.txt written and committed on the task branch; "
|
||||
"opening the PR next, then self-verifying the acceptance "
|
||||
"criteria before submit."
|
||||
),
|
||||
),
|
||||
"dev progress note",
|
||||
)
|
||||
env = expect_ok(dev.flow("open_pr", task_id=task_id), "dev open_pr")
|
||||
state = _task_state(stack, ids.task_id)
|
||||
assert state["pr_number"], f"open_pr did not record a PR: {state} / {env}"
|
||||
|
||||
# The i_am_done tracing gate demands: a during-work journal entry, the
|
||||
# dev_notes handoff section, a reflect entry, and an artifact referencing
|
||||
# every acceptance criterion (quoted verbatim in the decision note).
|
||||
expect_ok(
|
||||
dev.do(
|
||||
"note",
|
||||
scope="decision",
|
||||
task_id=task_id,
|
||||
text=(
|
||||
"Verified both acceptance criteria on the branch: "
|
||||
'"greeting.txt exists at the repo root" holds (file committed '
|
||||
'at the root), and "its content greets the reader" holds '
|
||||
"(content is a friendly hello). Decision: no README change "
|
||||
"needed; the greeting file is self-contained."
|
||||
),
|
||||
),
|
||||
"dev during-work decision note",
|
||||
)
|
||||
expect_ok(
|
||||
dev.do(
|
||||
"note",
|
||||
text="Handoff summary below (section carries the content).",
|
||||
scope="handoff",
|
||||
task_id=task_id,
|
||||
section={
|
||||
"summary": (
|
||||
"Built the greeting module: greeting.txt added at the "
|
||||
"repo root with a friendly greeting. Key change is one "
|
||||
"additive file on the task branch; PR is open against "
|
||||
"master; no risks beyond trivial content review."
|
||||
)
|
||||
},
|
||||
),
|
||||
"dev handoff section",
|
||||
)
|
||||
expect_ok(
|
||||
dev.do(
|
||||
"note",
|
||||
scope="reflect",
|
||||
task_id=task_id,
|
||||
text=(
|
||||
"Reflection: implemented the greeting task exactly per plan — "
|
||||
"wrote the file, committed on the task branch, opened the PR, "
|
||||
"and self-verified both acceptance criteria against the "
|
||||
"committed content."
|
||||
),
|
||||
),
|
||||
"dev reflect note",
|
||||
)
|
||||
expect_ok(dev.flow("i_am_done", task_id=task_id), "dev i_am_done")
|
||||
assert _task_state(stack, ids.task_id)["status"] == "awaiting_qa"
|
||||
|
||||
# --- QA: claim the review, inspect, pass --------------------------------
|
||||
qa = ScriptedAgent(stack, ids.qa_id, "be-qa", "qa")
|
||||
expect_ok(qa.flow("claim_review", task_id=task_id), "qa claim_review")
|
||||
expect_ok(
|
||||
qa.do(
|
||||
"note",
|
||||
scope="learning",
|
||||
task_id=task_id,
|
||||
text=(
|
||||
"Review learning: the greeting change is a single additive "
|
||||
"file; diff inspection on the PR confirms both acceptance "
|
||||
"criteria with no side effects on existing files."
|
||||
),
|
||||
),
|
||||
"qa learning note",
|
||||
)
|
||||
expect_ok(
|
||||
qa.flow(
|
||||
"pass_review",
|
||||
task_id=task_id,
|
||||
notes=(
|
||||
"Verified the PR diff on the fake origin: greeting.txt exists "
|
||||
"at the repo root and greets the reader. Both acceptance "
|
||||
"criteria hold; no regressions in the diff, and the branch "
|
||||
"contains exactly the one additive commit described."
|
||||
),
|
||||
ac_verdicts=[
|
||||
(
|
||||
"greeting.txt exists at the repo root — verified in the "
|
||||
"PR diff: the file is added at the repository root."
|
||||
),
|
||||
(
|
||||
"its content greets the reader — verified: the committed "
|
||||
"content is a friendly hello message."
|
||||
),
|
||||
],
|
||||
),
|
||||
"qa pass_review",
|
||||
)
|
||||
assert _task_state(stack, ids.task_id)["status"] == "awaiting_documentation"
|
||||
|
||||
# --- documenter: claim, document -----------------------------------------
|
||||
doc = ScriptedAgent(stack, ids.doc_id, "be-doc", "documenter")
|
||||
expect_ok(doc.flow("claim_doc_task", task_id=task_id), "doc claim_doc_task")
|
||||
expect_ok(
|
||||
doc.flow(
|
||||
"i_documented",
|
||||
task_id=task_id,
|
||||
files=["greeting.txt"],
|
||||
notes=(
|
||||
"Documented the greeting module: greeting.txt carries the "
|
||||
"user-facing greeting; no API surface changed, README "
|
||||
"untouched by design."
|
||||
),
|
||||
),
|
||||
"doc i_documented",
|
||||
)
|
||||
|
||||
final = _task_state(stack, ids.task_id)
|
||||
assert final["status"] == "awaiting_pm_review", final
|
||||
assert final["docs_complete"] is True, final
|
||||
Reference in New Issue
Block a user