Leak fixes, gate green again, uv/CI hardening, e2e lifecycle smoke harness (#294)

* test: align phase1 smoke mock with the armed team-match gate

The 8e5f84c4 sweep fixed 13 test files' inconsistent-team mocks but ran
only the gateway/foundation/runtime subsets; the full gate caught this
integration mock whose parent task carried an auto-generated MagicMock
team and died on not_authorized before the incomplete_input assertion.

* fix(runtime): attribute every agent.spawned audit to its dispatcher

A rogue spawner could not be identified live (2026-07-02): agent.spawned
rows carry container/model but not which dispatch loop launched them.
spawn_agent now takes spawned_by, stamps it into the spawned/spawn_failed
audit details, every call site passes its loop name, and an AST sweep
test holds future callers to it.

* fix(api): admin-complete refuses when the task's PR is still open

PATCH status=completed on a task with an OPEN PR stranded its commits
unmerged (bit the CEO twice live 2026-07-02). The override now refuses
with the PR number/URL and the consequence before the generic hatch
text; force:true stays the deliberate, audited escape.

* fix(panel): awaiting_ceo_approval offers the working ceo-approve path

The header's only approve action was Approve & Merge (POST
/approve-and-merge, no notes) which 400s NO_PR on a branchless MegaTask
umbrella — the CEO's approve button just failed. Primary action is now
Approve & Complete via the CeoApproveDialog (POST /ceo-approve, notes
>=20 chars, proven live); Approve & Merge stays for PR-bearing tasks.

* test: stop leaking self-heal + rate-limit state into live Redis

Two test files wrote real keys into a developer's localhost Redis:
self-heal originate tests left self_heal:notified:* (2h TTL) and the
i_am_blocked rate-limited tests left a NO-TTL 'anthropic rate-limited'
tracker blob — order/state-dependent poison for anything reading the
real tracker, and the prime suspect class for the one-off
test_self_heal_engine full-run failure (not reproduced in 5x dir runs,
adversarial orders, and a green full gate). Both files now point the
computed redis_url at an unreachable port; the engines' fail-open paths
keep every assertion intact. Leaked keys scrubbed live.

* docs: changelog + map delta for the leak-fix batch; mypy-clean attribution test

The attribution test's direct method assignments tripped the full gate's
mypy (method-assign) — switched to the house monkeypatch idiom, no
suppressions.

* fix(gate): clear the ten xenon C-ranks; isolate all tests from live Redis

Master CI has been red at the phase1 smoke test, so neither CI nor a
local full gate had reached the xenon step since the team-match sweep —
whose inline 'agent_team=str(agent.team) if ...' kwarg pushed nine verb
bodies from B(10) to C(11-12) unseen. A shared actor_context_fields()
(_protocol.py) computes (actor_slug, agent_team) once per verb, restoring
all nine to B with zero behavior change; the new admin-complete override
helper extraction does the same for routes/tasks.py.

tests/conftest.py gains an autouse fixture pointing the computed
redis_url at an unreachable port for every test — the root fix for the
three families caught writing live-Redis keys (self-heal dedupe,
rate-limit tracker, notification purpose-dedupe); no test uses a real
Redis, and every production path is fail-open by design.

* refactor(runtime): delete the never-wired dispatch-time spawn cooldown

_safe_spawn / gateway_pre_spawn_check / trigger_filter had no caller in
the repo's entire history (87ef42bf only flipped the flag). Its five
rules are superseded: provider parking runs inside spawn_agent, claim
freshness is the guards+reaper, runaway respawns are the progress-aware
breaker + notification cooldown; the per-task cooldown rule would
queue-stall every normal stage handoff if wired today. gateway_triggers
table kept inert. Ratified by the CEO over wiring it.

* build: serialize uv — gate recipes never implicitly sync the venv

Every uv run re-syncs implicitly, so a background make quality plus any
foreground uv run raced two writers on one .venv and tore site-packages
apart (the recurring rich/pip/bandit ImportError corruption; bit twice
today, four times on 2026-07-02's first session). UV_NO_SYNC=1 is now
exported Makefile-wide and quality/quality-fast/gate depend on one
explicit up-front sync step.

* fix(git): PR/merge/branch REST calls honor github_api_base_url

Fifteen sites hardcoded https://api.github.com while the CI-run and
open-PR-list calls already read settings.github_api_base_url — a GHE or
test override silently applied to half the surface. One _api_base()
helper keeps them uniform; default behavior unchanged.

* ci: split the monolith — backend CI, Panel CI, E2E Smoke

ci.yml keeps its file name and the backend quality job only (self-heal /
ci-watch / release-readiness default to the ci.yml workflow); the panel
job moves to panel-ci.yml scoped to panel/**, and the new scripted-agent
lifecycle smoke gets e2e-smoke.yml + a make e2e-smoke target (env-gated
out of the default pytest run). Trade: a panel-only red now lands on
Panel CI, which the ci.yml-pinned watch engines don't see.

* feat(tests): e2e lifecycle smoke harness — scripted agents, real gates

tests/e2e_smoke stands up the real API (flow/do routers + middleware on
uvicorn) over the ephemeral test Postgres, a local bare origin standing
in for GitHub, and a fake GitHub REST layer whose merges are real git
merges. A deterministic driver reloads the real MCP flow/do modules per
agent and walks claim (real clone + worktree) -> tracing-gap -> note ->
plan gate -> commit -> PR -> the full i_am_done ladder -> QA verdicts ->
documenter -> awaiting_pm_review in ~5s. Runs via make e2e-smoke + its
own CI workflow; skipped (env-gated) in the default suite. The
freeze-lift condition's first half: scenario 1 green.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
2026-07-02 18:28:07 +02:00
committed by GitHub
co-authored by Renn F
parent fe67a630ac
commit 1c87a4e4e4
35 changed files with 1661 additions and 835 deletions
+10
View File
@@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
### Added
- **The e2e lifecycle smoke harness — scripted agents drive the REAL gates, no LLM anywhere.** `tests/e2e_smoke/` stands up the real API (v1 flow/do routers + middleware served by uvicorn) over the ephemeral test Postgres, a local bare git origin standing in for GitHub, and a fake GitHub REST layer whose PR merges are REAL git merges (squash included) on that origin. A deterministic driver reloads the REAL MCP `flow_server`/`do_server` modules per agent (role-scoped manifests from the real `role_config`) and walks the lifecycle through every gate: claim (real clone + per-task worktree) → tracing-gap → claim-note → plan gate → commit → PR → the full `i_am_done` ladder (during-work journal, handoff section, reflect, per-AC artifacts) → QA (learning note + per-criterion verdicts) → documenter → `awaiting_pm_review`, in ~5s. Seam bugs — tool↔gate schema drift, squash merges, stale refs, worktree routing — now die in CI (`make e2e-smoke`, its own workflow) instead of in a live run. Scenario 1 covers the leaf-dev arc; PM-merge/CEO chains extend the same harness.
- **HTTP security hardening — a fastapi-guard layer for public/cloud exposure, default-off and calibrated for RoboCo's own traffic.** RoboCo can now front its API with a [fastapi-guard](https://pypi.org/project/fastapi-guard/) `SecurityMiddleware` + per-route decorator layer (`roboco/security.py`), gated behind `ROBOCO_GUARD_ENABLED` (default **off**) — when off, `create_app` never mounts the middleware and the request path is byte-for-byte unchanged, so the decorators are harmless no-ops. Armed, it adds IP/rate controls, a signature WAF, security headers, cloud-provider/honeypot checks, and an emergency-lockdown kill switch (`ROBOCO_GUARD_EMERGENCY`), plus three RoboCo-specific custom validators the stock WAF cannot cover: **prompt-injection**, **secret-exfil**, and **internal-SSRF** scanning on the prompt-facing and agent-content surfaces. Nine distinct decorators are applied thoughtfully per-surface across ingress and sensitive routes (rate-limit, size caps, content-type, behavior analysis, cloud blocking, honeypot form-traps, usage monitoring, suspicious detection, custom validation). Exposure is env-driven — `enforce_https` follows `ROBOCO_ENVIRONMENT`, so a personal NAS deploy stays relaxed while a cloud host enforces TLS — and telemetry to a guard-core platform is separately gated (`ROBOCO_GUARD_TELEMETRY_ENABLED`, no data leaves the box while off).
- **Scanner honeytrap & auto-ban (Surface N) — two layers, matched to where traffic actually lands.** Behind nginx only `/api`, `/ws`, `/health`, `/ready` reach the orchestrator, so guard can only see (and ban) scanner probes on those paths — the classic root probes (`/.env`, `/wp-login.php`, `/phpmyadmin`, `/.git/config`) hit the panel. So Surface N is split: (1) the guard `threat_ban_config` now carries `recon` / `sensitive_file` / `cms_probing` categories, turning repeated scanner probes on `/api` paths into an adaptive per-IP auto-ban (redis-backed, 24h) once enforcement is active; and (2) nginx drops the classic root scanner paths at the edge with `444` (connection closed, no response) before they reach the panel, anchored to known scanner fingerprints so `/.well-known` and every real route are untouched. The auto-ban only fires in active mode (passive logs the recon hit) and needs redis; the nginx edge-drop is always on.
- **Guard WAF calibration — active enforcement no longer false-positives on RoboCo's own traffic.** The first end-to-end run of the guard surfaced that active enforcement would block ~50% of legitimate agent traffic: RoboCo's request bodies *are* code, SQL, unified diffs, file paths, HTML, and URLs (task specs, agent notes/commits, RAG queries, git bodies, chat), and the stock signature WAF (SQLi/XSS/path-traversal/URL detectors) flagged them as attacks. `build_security_config` now excludes RoboCo's free-text top-level body fields from WAF scanning (`excluded_detection_body_fields`, derived from the real request models — including the free-form container fields whose nested prose is stringified and scanned), which drops the false-positive rate to zero while keeping the WAF active on every structured (id/enum/slug/branch) field and leaving the custom prompt-injection / secret-exfil / SSRF validators — which run independently of the exclusion — fully in force. A new end-to-end integration test (`tests/unit/test_security_middleware.py`) mounts the real middleware, drives guard's lifespan, and fires real requests to prove: passive mode is genuinely log-only (never blocks), active mode does not false-positive on realistic agent payloads, threats are still blocked even inside excluded fields, and the WAF still fires on non-excluded fields. The NAS composes arm the guard in **passive/log-only** mode (`ROBOCO_GUARD_PASSIVE_MODE=true`, `ROBOCO_GUARD_FAIL_SECURE=false`) so a deploy calibrates against real traffic before any flip to active enforcement.
@@ -30,6 +31,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
### Fixed
- **Every `agent.spawned` audit row names its dispatcher.** A rogue spawner could not be identified live — the audit row carried container/model but not which of the ~27 dispatch loops launched it. `spawn_agent` now takes `spawned_by`, stamps it into the `agent.spawned` / `agent.spawn_failed` details (`"unspecified"` when absent so audit queries never miss the field), every call site passes its loop name, and a whole-package AST sweep test fails any future caller that omits it.
- **Admin-complete refuses while the task's PR is still open.** `PATCH status=completed` on a task whose work session records an OPEN PR stranded its commits unmerged (bit the CEO twice live). The override now refuses with the PR number/URL and the concrete consequence — merge first, or approve via `POST /tasks/{id}/ceo-approve` — checked before the generic hatch text; `force: true` stays the deliberate, audited escape, and a merged/closed PR changes nothing.
- **The panel's CEO-approve button works on every gated task.** On `awaiting_ceo_approval` the task header offered only "Approve & Merge" (`POST /approve-and-merge`, no notes), which 400s `NO_PR` on a branchless MegaTask umbrella — the CEO's approve just failed. The primary action is now "Approve & Complete" through the `CeoApproveDialog` (`POST /ceo-approve`, notes ≥ 20 chars); "Approve & Merge" remains, but only when the task actually carries a PR.
- **Tests no longer leak state into a developer's live Redis.** The self-heal originate tests wrote `self_heal:notified:*` dedupe keys (2h TTL), the `i_am_blocked` rate-limited tests wrote a NO-TTL "anthropic rate-limited" tracker blob, and notification tests left short-TTL purpose-dedupe keys in whatever Redis listens on localhost — order/state-dependent poison for any test (or local orchestrator run) reading the real instance. A root-level autouse fixture now points the computed `redis_url` at an unreachable port for every test (no test uses a real Redis; every production Redis path is fail-open by design), with explicit per-file guards kept at the two proven writers.
- **The quality gate is fully green again — ten latent xenon C-ranks cleared.** Master CI has been red at a smoke test whose mock predated the armed team-match gate, so neither CI nor a local full gate had reached the xenon step — hiding that the team-match sweep's inline `agent_team=str(agent.team) if …` kwarg had pushed nine gateway verb bodies (`i_am_done`, `resume`, `unclaim`, `submit_up`, `submit_root`, `complete`, `escalate_up`, `escalate_to_ceo`, `fail_review`) from B to C unseen. A shared `actor_context_fields()` helper computes `(actor_slug, agent_team)` once per verb — zero behavior change — and the new admin-complete open-PR check is likewise extracted to a helper so the override function stays under the threshold.
- **Declared dependencies become real edges (MegaTask + delegate).** The live S6 out-of-order break, both halves. Batch intake: each draft's `depends_on` (the CEO's declared "Depends on" list, batch indices) is now wired verbatim into the sequencing DAG — `SequencingService.analyze` unions declared edges with the derived collision rules (self/out-of-range references rejected, cycles caught by the existing toposort); previously only analyzer-derived file-overlap edges were wired and a declared wave could be silently dropped. Delegate: a `code` subtask now REQUIRES a non-empty `intends_to_touch` collision surface (new `TASK_AT_DELEGATE` completeness spec) — a no-surface code sibling is "parallel to everything" by analyzer design, which is how two devs ran explicitly-sequenced work out of order on divergent branches. Non-code delegations and REST/manual creation are unchanged. The MCP `delegate` tool now actually carries `intends_to_touch` / `adds_migration` / `touches_shared` / `depends_on` and forwards them to the gateway — the gate demanded a field the tool could not send, so every code delegation was rejected `incomplete_input` with no way to comply (live fleet-wide delegation wall); a parity test locks plan-gate fields to tool parameters.
- **Respawn circuit breaker now guards every task-keyed spawn path.** The progress-aware breaker (strike counting with status-advance reset, tracing-gap budget, DB durability, one-shot CEO notification) was consulted by only 3 dispatch paths; doc/QA/dev/PR-review/PR-gate/revision/board spawns ran unguarded at fixed cadence — a documenter with no valid verb respawned 26× in ~100 min on one task. The gate is now consulted at all 14 task-keyed spawn sites. It also catches status ping-pong: any status change used to fully reset the strike counter, so a `blocked``in_progress` oscillation — which changes status on every spawn while advancing nothing — never tripped the gate (8 spawns over two hours, live). A never-seen status still fully resets; a REVISITED status gets a bounded reset budget (`pm_respawn_max_revisit_resets`, default 2), after which strikes accrue and the gate fires.
- **Assembled-PR freshness + integrity at submit_up / submit_root.** Freshness: the assembled cell/root branch is auto-rebased onto its base when behind (children are terminal at submit time; master is never written); a rebase conflict is a clean rejection naming the files — ends the needs_revision ↔ awaiting_pr_review ping-pong of re-reviewing a stale head. Integrity: every completed child's commits must be patch-present (`git cherry`, rebase-safe) in the assembled branch before review — a completed revert whose merge was lost re-spawned the exact violation it fixed. The guard now also recognizes **squash-merged** children: `git cherry` can't patch-match N child commits against the one squashed commit, but every commit carries the `[taskid8]` prefix, so a parent commit bearing the child's marker proves the child landed (three squash-merged children read as "work missing" and every legitimate `submit_up` was refused, live). Markerless children stay flagged — the original incident the guard exists for.
@@ -43,6 +49,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
- **Admin status override now reconciles claim ownership.** Forcing a `blocked` task into a review/queue state (`needs_revision`, `awaiting_qa`, `awaiting_documentation`, `awaiting_pr_review`, `awaiting_pm_review`) previously left the stale escalation claim in place, so the next claimant was handed the task by `give_me_work`/`triage` while its `note()` writes bounced `not_authorized "you do not hold the claim"` — it re-blocked immediately. The override now clears the claim (`claimed_by`/`claimed_at`/`active_claimant_id`) and consumes the pre-block snapshot for review-state targets; the pending/in_progress owner-restore path additionally syncs `active_claimant_id` so the restored owner's content writes don't bounce either. A REST PATCH that unassigns a task (`assigned_to: null`) now releases the claim with it.
### Removed
- **The never-wired dispatch-time spawn-cooldown path.** `_safe_spawn` / `gateway_pre_spawn_check` / `trigger_filter.decide_spawn` had no caller anywhere in the repo's history — the "enable gateway cooldown logic in production" commit only flipped its flag, and the protections it promised have since shipped better elsewhere: provider parking lives inside `spawn_agent` itself, claim freshness is enforced by the claim guards + reaper, and runaway respawns are bounded by the progress-aware circuit breaker (all 14 task-keyed sites) plus the notification-spawn cooldown. Wiring it now would have re-introduced a per-task cooldown that queue-stalls every normal stage handoff (dev→QA→doc→PM spawn the same task within one window). Deleted: the orchestrator block, `trigger_filter.py`, its tests, and the dead `spawn_cooldown_seconds` / `role_spawn_rate_per_minute` settings. The `gateway_triggers` table is kept (inert; dropping it is a migration decision).
## [0.15.0] - 2026-07-01
### Added