Files
roboco/tests/unit/services/test_self_heal_originate_db.py
T
1c87a4e4e4 Leak fixes, gate green again, uv/CI hardening, e2e lifecycle smoke harness (#294)
* test: align phase1 smoke mock with the armed team-match gate

The 8e5f84c4 sweep fixed 13 test files' inconsistent-team mocks but ran
only the gateway/foundation/runtime subsets; the full gate caught this
integration mock whose parent task carried an auto-generated MagicMock
team and died on not_authorized before the incomplete_input assertion.

* fix(runtime): attribute every agent.spawned audit to its dispatcher

A rogue spawner could not be identified live (2026-07-02): agent.spawned
rows carry container/model but not which dispatch loop launched them.
spawn_agent now takes spawned_by, stamps it into the spawned/spawn_failed
audit details, every call site passes its loop name, and an AST sweep
test holds future callers to it.

* fix(api): admin-complete refuses when the task's PR is still open

PATCH status=completed on a task with an OPEN PR stranded its commits
unmerged (bit the CEO twice live 2026-07-02). The override now refuses
with the PR number/URL and the consequence before the generic hatch
text; force:true stays the deliberate, audited escape.

* fix(panel): awaiting_ceo_approval offers the working ceo-approve path

The header's only approve action was Approve & Merge (POST
/approve-and-merge, no notes) which 400s NO_PR on a branchless MegaTask
umbrella — the CEO's approve button just failed. Primary action is now
Approve & Complete via the CeoApproveDialog (POST /ceo-approve, notes
>=20 chars, proven live); Approve & Merge stays for PR-bearing tasks.

* test: stop leaking self-heal + rate-limit state into live Redis

Two test files wrote real keys into a developer's localhost Redis:
self-heal originate tests left self_heal:notified:* (2h TTL) and the
i_am_blocked rate-limited tests left a NO-TTL 'anthropic rate-limited'
tracker blob — order/state-dependent poison for anything reading the
real tracker, and the prime suspect class for the one-off
test_self_heal_engine full-run failure (not reproduced in 5x dir runs,
adversarial orders, and a green full gate). Both files now point the
computed redis_url at an unreachable port; the engines' fail-open paths
keep every assertion intact. Leaked keys scrubbed live.

* docs: changelog + map delta for the leak-fix batch; mypy-clean attribution test

The attribution test's direct method assignments tripped the full gate's
mypy (method-assign) — switched to the house monkeypatch idiom, no
suppressions.

* fix(gate): clear the ten xenon C-ranks; isolate all tests from live Redis

Master CI has been red at the phase1 smoke test, so neither CI nor a
local full gate had reached the xenon step since the team-match sweep —
whose inline 'agent_team=str(agent.team) if ...' kwarg pushed nine verb
bodies from B(10) to C(11-12) unseen. A shared actor_context_fields()
(_protocol.py) computes (actor_slug, agent_team) once per verb, restoring
all nine to B with zero behavior change; the new admin-complete override
helper extraction does the same for routes/tasks.py.

tests/conftest.py gains an autouse fixture pointing the computed
redis_url at an unreachable port for every test — the root fix for the
three families caught writing live-Redis keys (self-heal dedupe,
rate-limit tracker, notification purpose-dedupe); no test uses a real
Redis, and every production path is fail-open by design.

* refactor(runtime): delete the never-wired dispatch-time spawn cooldown

_safe_spawn / gateway_pre_spawn_check / trigger_filter had no caller in
the repo's entire history (87ef42bf only flipped the flag). Its five
rules are superseded: provider parking runs inside spawn_agent, claim
freshness is the guards+reaper, runaway respawns are the progress-aware
breaker + notification cooldown; the per-task cooldown rule would
queue-stall every normal stage handoff if wired today. gateway_triggers
table kept inert. Ratified by the CEO over wiring it.

* build: serialize uv — gate recipes never implicitly sync the venv

Every uv run re-syncs implicitly, so a background make quality plus any
foreground uv run raced two writers on one .venv and tore site-packages
apart (the recurring rich/pip/bandit ImportError corruption; bit twice
today, four times on 2026-07-02's first session). UV_NO_SYNC=1 is now
exported Makefile-wide and quality/quality-fast/gate depend on one
explicit up-front sync step.

* fix(git): PR/merge/branch REST calls honor github_api_base_url

Fifteen sites hardcoded https://api.github.com while the CI-run and
open-PR-list calls already read settings.github_api_base_url — a GHE or
test override silently applied to half the surface. One _api_base()
helper keeps them uniform; default behavior unchanged.

* ci: split the monolith — backend CI, Panel CI, E2E Smoke

ci.yml keeps its file name and the backend quality job only (self-heal /
ci-watch / release-readiness default to the ci.yml workflow); the panel
job moves to panel-ci.yml scoped to panel/**, and the new scripted-agent
lifecycle smoke gets e2e-smoke.yml + a make e2e-smoke target (env-gated
out of the default pytest run). Trade: a panel-only red now lands on
Panel CI, which the ci.yml-pinned watch engines don't see.

* feat(tests): e2e lifecycle smoke harness — scripted agents, real gates

tests/e2e_smoke stands up the real API (flow/do routers + middleware on
uvicorn) over the ephemeral test Postgres, a local bare origin standing
in for GitHub, and a fake GitHub REST layer whose merges are real git
merges. A deterministic driver reloads the real MCP flow/do modules per
agent and walks claim (real clone + worktree) -> tracing-gap -> note ->
plan gate -> commit -> PR -> the full i_am_done ladder -> QA verdicts ->
documenter -> awaiting_pm_review in ~5s. Runs via make e2e-smoke + its
own CI workflow; skipped (env-gated) in the default suite. The
freeze-lift condition's first half: scenario 1 green.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-07-02 18:28:07 +02:00

269 lines
10 KiB
Python

"""Self-heal task origination against a real Postgres DB.
The loop opens a fix task only when ``self_heal_originate_enabled``, dedupes one
open task per regression fingerprint, honors the per-cycle and rolling open-task
caps, and creates the task PENDING + assigned to the Main PM agent +
``confirmed_by_human=False`` so it is HELD for the CEO's Approve-&-Start (it
does not dispatch autonomously). Crucially the loop NEVER calls start / approve
/ merge / deploy — asserted here.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from unittest.mock import AsyncMock
import pytest
from roboco.config import settings as cfg
from roboco.db.tables import AgentTable, ProjectTable
from roboco.foundation import identity as _foundation
from roboco.models.base import AgentRole, AgentStatus, TaskStatus, Team
from roboco.services.notification import NotificationService
from roboco.services.self_heal_engine import SelfHealEngine
from roboco.services.task import TaskService, get_task_service
from roboco.services.telemetry import TelemetrySample
if TYPE_CHECKING:
from sqlalchemy.ext.asyncio import AsyncSession
SYSTEM_UUID = _foundation.AGENTS["system"].uuid
MAIN_PM_UUID = _foundation.AGENTS["main-pm"].uuid
SLUG = "roboco"
ONE = 1
class _FakeSource:
def __init__(self, samples: list[TelemetrySample]) -> None:
self._samples = samples
async def fetch(self) -> list[TelemetrySample]:
return list(self._samples)
def _breach(signal: str) -> TelemetrySample:
return TelemetrySample(
signal_name=signal,
value=1.0,
threshold=1.0,
window="latest_completed_run",
repo_hint=SLUG,
observed_at="2026-06-17T00:00:00Z",
raw_ref="https://github.com/x/roboco/actions/runs/1",
detail=f"{signal} concluded 'failure'",
)
async def _seed_project(session: AsyncSession, slug: str = SLUG) -> None:
"""Seed the system agent (FK target for created_by) + RoboCo's own project.
The system agent has a FIXED foundation uuid (origination hardcodes it as
created_by). Another test in the full suite may have already committed it
into the session-scoped DB, so get-or-create by id — a plain insert collides
on pk_agents (green in isolation, red in the full CI run).
"""
if await session.get(AgentTable, SYSTEM_UUID) is None:
session.add(
AgentTable(
id=SYSTEM_UUID,
name="System",
slug=f"system-{slug}",
role=AgentRole.SYSTEM,
team=None,
status=AgentStatus.ACTIVE,
model_config={},
system_prompt="system",
capabilities=[],
permissions={},
metrics={},
)
)
await session.flush()
# The loop assigns the fix task to the Main PM agent (an FK to agents.id), so
# that row must exist — get-or-create by its fixed foundation uuid.
if await session.get(AgentTable, MAIN_PM_UUID) is None:
session.add(
AgentTable(
id=MAIN_PM_UUID,
name="Main PM",
slug="main-pm",
role=AgentRole.MAIN_PM,
team=Team.MAIN_PM,
status=AgentStatus.ACTIVE,
model_config={},
system_prompt="pm",
capabilities=[],
permissions={},
metrics={},
)
)
await session.flush()
session.add(
ProjectTable(
name="RoboCo",
slug=slug,
git_url="https://github.com/x/roboco.git",
default_branch="master",
protected_branches=["master"],
assigned_cell=Team.BACKEND,
created_by=SYSTEM_UUID,
is_active=True,
)
)
await session.flush()
def _enable(monkeypatch: pytest.MonkeyPatch, **overrides: object) -> None:
monkeypatch.setattr(cfg, "self_heal_enabled", True)
monkeypatch.setattr(cfg, "self_heal_originate_enabled", True)
monkeypatch.setattr(cfg, "self_heal_max_open_tasks", 5)
monkeypatch.setattr(cfg, "self_heal_max_per_cycle", 5)
for key, value in overrides.items():
monkeypatch.setattr(cfg, key, value)
# Keep notification a no-op (its own session/IO is out of scope here).
monkeypatch.setattr(NotificationService, "send_ack_notification", AsyncMock())
# Point the notify-dedupe at an unreachable Redis: _already_notified fails
# open and _mark_notified swallows, so these tests neither read from nor
# leak `self_heal:notified:*` keys (2h TTL) into a developer's live Redis.
# (redis_url is a computed property — patch its inputs.)
monkeypatch.setattr(cfg, "redis_host", "127.0.0.1")
monkeypatch.setattr(cfg, "redis_port", 1)
@pytest.mark.asyncio
async def test_disabled_originate_creates_no_task(
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
) -> None:
await _seed_project(db_session)
_enable(monkeypatch, self_heal_originate_enabled=False)
engine = SelfHealEngine(db_session, source=_FakeSource([_breach("ci:roboco")]))
obs = await engine.run_cycle()
assert len(obs) == ONE # detected + notified
assert await get_task_service(db_session).list_open_self_heal_tasks() == []
@pytest.mark.asyncio
async def test_originate_creates_pending_main_pm_assigned_task(
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
) -> None:
await _seed_project(db_session)
_enable(monkeypatch)
engine = SelfHealEngine(db_session, source=_FakeSource([_breach("ci:roboco")]))
await engine.run_cycle()
open_tasks = await get_task_service(db_session).list_open_self_heal_tasks()
assert len(open_tasks) == ONE
task = open_tasks[0]
assert task.status == TaskStatus.PENDING
# Assigned to the Main PM agent up front (not just team=main_pm) so that, once
# the CEO approves it, the orchestrator dispatches it straight to that agent.
assert task.assigned_to == MAIN_PM_UUID
# held for the CEO's Approve-&-Start — NOT auto-confirmed: the orchestrator
# + give_me_work keep it out of dispatch until approve_and_start flips this.
assert task.confirmed_by_human is False
assert task.team == Team.MAIN_PM
assert task.source == "self_heal"
assert task.acceptance_criteria # non-empty (AC-guardrail)
assert (task.orchestration_markers or {}).get("self_heal_fp")
@pytest.mark.asyncio
async def test_dedupe_no_second_task_same_fingerprint(
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
) -> None:
await _seed_project(db_session)
_enable(monkeypatch)
src = _FakeSource([_breach("ci:roboco")])
await SelfHealEngine(db_session, source=src).run_cycle()
await SelfHealEngine(db_session, source=src).run_cycle() # same fingerprint
open_tasks = await get_task_service(db_session).list_open_self_heal_tasks()
assert len(open_tasks) == ONE
@pytest.mark.asyncio
async def test_per_cycle_cap_limits_origination(
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
) -> None:
await _seed_project(db_session)
_enable(monkeypatch, self_heal_max_per_cycle=1)
# Two distinct regressions in one cycle, cap = 1 → only one task opens.
src = _FakeSource([_breach("ci:roboco:a"), _breach("ci:roboco:b")])
await SelfHealEngine(db_session, source=src).run_cycle()
assert len(await get_task_service(db_session).list_open_self_heal_tasks()) == ONE
@pytest.mark.asyncio
async def test_open_task_cap_blocks_further_origination(
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
) -> None:
await _seed_project(db_session)
_enable(monkeypatch, self_heal_max_open_tasks=1)
await SelfHealEngine(
db_session, source=_FakeSource([_breach("ci:roboco:a")])
).run_cycle()
# A different regression next cycle, but one task is already open → blocked.
await SelfHealEngine(
db_session, source=_FakeSource([_breach("ci:roboco:b")])
).run_cycle()
assert len(await get_task_service(db_session).list_open_self_heal_tasks()) == ONE
@pytest.mark.asyncio
async def test_unresolved_project_is_notify_only(
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
) -> None:
await _seed_project(db_session)
_enable(monkeypatch)
ghost = TelemetrySample(
signal_name="ci:ghost",
value=1.0,
threshold=1.0,
window="latest_completed_run",
repo_hint="not-a-registered-project",
observed_at="",
raw_ref="",
)
# No crash, no task — the regression can't be tied to a repo.
await SelfHealEngine(db_session, source=_FakeSource([ghost])).run_cycle()
assert await get_task_service(db_session).list_open_self_heal_tasks() == []
@pytest.mark.asyncio
async def test_loop_never_starts_or_approves(
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
) -> None:
await _seed_project(db_session)
_enable(monkeypatch)
approve = AsyncMock()
ceo_approve = AsyncMock()
monkeypatch.setattr(TaskService, "approve_and_start", approve)
monkeypatch.setattr(TaskService, "ceo_approve", ceo_approve)
await SelfHealEngine(
db_session, source=_FakeSource([_breach("ci:roboco")])
).run_cycle()
approve.assert_not_awaited()
ceo_approve.assert_not_awaited()
open_tasks = await get_task_service(db_session).list_open_self_heal_tasks()
assert len(open_tasks) == ONE
assert open_tasks[0].status == TaskStatus.PENDING # never advanced by the loop
@pytest.mark.asyncio
async def test_originated_task_is_held_for_ceo_approve_and_start(
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The opened task is HELD (confirmed_by_human=False) for the CEO's
Approve-&-Start — the PM dispatcher does NOT pick it up autonomously; the
CEO must approve_and_start it first (F059)."""
await _seed_project(db_session)
_enable(monkeypatch)
await SelfHealEngine(
db_session, source=_FakeSource([_breach("ci:roboco")])
).run_cycle()
task = (await get_task_service(db_session).list_open_self_heal_tasks())[0]
assert task.confirmed_by_human is False # held for the CEO — no autonomous dispatch
assert task.status == TaskStatus.PENDING # not advanced by the loop
assert (
task.assigned_to == MAIN_PM_UUID
) # straight to the Main PM agent once approved