mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
fix(self-heal): dispatch fix tasks autonomously instead of stranding them
A self-heal fix task was opened confirmed_by_human=false and held out of dispatch until "Approve & Start" — but that button only renders for a board-reviewed Intake task (pending + board_review_complete + team != main_pm), never for a self-heal task (team=main_pm, no board review). So there was no way to start it: it sat in pending forever and the Main PM never picked it up. Self-heal is RoboCo healing itself, not an Intake draft — it shouldn't need a manual Approve & Start. Origination now opens the fix task confirmed + assigned to the Main PM agent, the PM dispatcher's self-heal hold is dropped, and the now-dead approve_and_start special-case is removed. The fix still ships through the normal gates (dev -> QA -> PR review -> the CEO's merge); the loop never starts, merges, or deploys.
This commit is contained in:
@@ -23,6 +23,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
- **The conventions standard now resolves for projects created before it existed.** It previously read the committed `.roboco/conventions.yml` and the repo scan from `project.workspace_path` — a field only a manual API call ever set — so an older project (or one whose workspace was cleared) showed an empty "missing" map no matter what was pushed. The service now ensures a dedicated, default-branch read clone on demand and reads from it, persisting the resolved path + HEAD (the backfill). The panel tab, the spawn-time ambient block, and the per-task constraints all resolve the committed standard with no manual setup.
|
||||
- **The conventions ambient prompt block no longer truncates mid-line.** It now lists only modules that actually constrain a kind, and when the list would exceed its budget it trims at a line boundary with a `+N more` pointer instead of cutting a module in half.
|
||||
- **The conventions read clone now stays current on a private repo.** Its refresh reused the orchestrator's token-less best-effort fetch, but the clone's remote URL is credential-stripped — so on a private repo the refresh fetch failed silently and the clone stayed frozen at clone-time, never seeing commits merged afterwards (the panel showed "auto-derived defaults" even after the standard was merged to the default branch). The refresh now performs a token-authenticated fetch + hard-reset, mirroring the clone.
|
||||
- **Self-heal fix tasks dispatch autonomously instead of being stranded.** A self-heal task was opened `confirmed_by_human=false` and held out of dispatch until an "Approve & Start" — but that button only renders for board-reviewed Intake tasks, never for a self-heal task (`team=main_pm`, no board review), so there was no way to start it and it sat in `pending` forever. Self-heal now opens the fix task confirmed + assigned to the Main PM, so the dispatcher picks it up immediately. The fix still ships through the normal gates (dev → QA → PR review → the CEO's merge); the loop never starts, merges, or deploys.
|
||||
- **Self-heal no longer reads the wrong branch and fails silently.** The CI-signal fetch filtered runs by `project.default_branch or "main"` — the only `"main"` fallback in the codebase (everywhere else falls back to `"master"`) — so a project whose default branch is `master` (like RoboCo) with an unset `default_branch` matched zero runs and the signal silently went dark: no fix task, no notification. The fallback now matches the rest of the codebase, and an armed self-heal that reads no CI signal (no/expired token, wrong branch, or a GitHub error) now logs a loud warning instead of an invisible no-op.
|
||||
- **The toolchain gate no longer passes silently on an unverifiable workspace.** A `broken` interpreter still blocks; an `unknown` status — the smoke could not confirm the suite is collectable — now emits a warning when the gate proceeds, instead of slipping through unseen.
|
||||
- **The crypto tests are hermetic.** The Fernet round-trip tests supply their own key instead of depending on `ROBOCO_ENCRYPTION_KEY` in the environment, so they pass in any gate container without the production secret being injected.
|
||||
|
||||
@@ -236,7 +236,7 @@ These gate the env-toggled capabilities. Each is inert when off. See [Optional c
|
||||
| `ROBOCO_SELF_HEAL_ENABLED` | `false` | Master switch for the self-heal loop (detect + notify the CEO). When off the loop never runs. |
|
||||
| `ROBOCO_SELF_HEAL_PROJECT_SLUG` | *(empty)* / `roboco-api` (compose) | The registered project that *is* RoboCo itself — the only repo the loop watches/originates into. |
|
||||
| `ROBOCO_SELF_HEAL_CI_WORKFLOW` | `ci.yml` | GitHub Actions workflow file to scope the CI signal to. |
|
||||
| `ROBOCO_SELF_HEAL_ORIGINATE_ENABLED` | `false` | Second opt-in: on a regression, also open a PENDING fix task. The loop never approves, merges, or deploys. |
|
||||
| `ROBOCO_SELF_HEAL_ORIGINATE_ENABLED` | `false` | Second opt-in: on a regression, also open a fix task and dispatch it to the Main PM automatically (no manual start). The loop never merges or deploys — the fix ships through the normal gates (QA, PR review, your merge). |
|
||||
| `ROBOCO_SELF_HEAL_INTERVAL_SECONDS` | `1800` | Seconds between telemetry passes. |
|
||||
| `ROBOCO_SELF_HEAL_MAX_OPEN_TASKS` | `3` | Rolling cap on concurrently-open self-heal tasks. |
|
||||
| `ROBOCO_SELF_HEAL_MAX_PER_CYCLE` | `1` | Max self-heal tasks originated in one cycle. |
|
||||
|
||||
@@ -38,7 +38,7 @@ Flip **the strategy engine** on in Settings → Feature Flags (or set `ROBOCO_ST
|
||||
|
||||
### The self-healing CI loop
|
||||
|
||||
The same shape, pointed inward: flip **self-healing** on in Settings → Feature Flags (or set `ROBOCO_SELF_HEAL_ENABLED=true`) and RoboCo begins watching its **own** repository's CI. When a run regresses it tells you. Turn on the second switch (`ROBOCO_SELF_HEAL_ORIGINATE_ENABLED=true`) and it goes one step further — it opens a fix task for the regression, but only as far as **PENDING, awaiting your approval**. It never starts, merges, or deploys that work itself: the company can notice it broke its own build and queue the repair, but the call to run it stays yours. Both switches are off by default, and it watches only the one repo you name as RoboCo itself.
|
||||
The same shape, pointed inward: flip **self-healing** on in Settings → Feature Flags (or set `ROBOCO_SELF_HEAL_ENABLED=true`) and RoboCo begins watching its **own** repository's CI. When a run regresses it tells you. Turn on the second switch (`ROBOCO_SELF_HEAL_ORIGINATE_ENABLED=true`) and it goes one step further — it opens a fix task for the regression and hands it straight to the Main PM, who coordinates the repair. It never merges or deploys that work itself: the fix still flows through the normal gates — dev, QA, PR review, and **your** merge — so the company can repair its own build autonomously while the decision to ship stays yours. Both switches are off by default, and it watches only the one repo you name as RoboCo itself.
|
||||
|
||||
## Feel the whole thing
|
||||
|
||||
|
||||
@@ -59,7 +59,7 @@ from roboco.models.runtime import (
|
||||
WaitingRecord,
|
||||
)
|
||||
from roboco.seeds.initial_data import AGENT_UUIDS
|
||||
from roboco.services.task import PR_REVIEW_SOURCES, SELF_HEAL_SOURCE
|
||||
from roboco.services.task import PR_REVIEW_SOURCES
|
||||
|
||||
logger = structlog.get_logger()
|
||||
|
||||
@@ -7705,17 +7705,11 @@ Start now: evidence(task_id="{task_id}")
|
||||
# PM hierarchy never routes or spawns them.
|
||||
if task.get("source") in PR_REVIEW_SOURCES:
|
||||
continue
|
||||
# A self-heal fix task is opened by the loop but must stay INERT until
|
||||
# the CEO Approve-&-Starts it (which flips confirmed_by_human). Until
|
||||
# then the PM hierarchy must not route, assign, or spawn it — even
|
||||
# though it's already team=main_pm. It still appears in the panel so
|
||||
# the CEO can see and approve it. This guard sits before the
|
||||
# assigned-vs-unassigned split, so it holds whether or not the task
|
||||
# carries an assignee.
|
||||
if task.get("source") == SELF_HEAL_SOURCE and not task.get(
|
||||
"confirmed_by_human"
|
||||
):
|
||||
continue
|
||||
# Self-heal fix tasks dispatch autonomously — the loop opens them
|
||||
# confirmed + assigned to the Main PM, so they flow through the
|
||||
# assigned-PM path below like any other PM task (no CEO Approve-&-
|
||||
# Start; that gate is the Intake/board flow). The fix still ships
|
||||
# through dev -> QA -> PR review -> the CEO's merge.
|
||||
assigned_to = task.get("assigned_to")
|
||||
if assigned_to:
|
||||
if self._resolve_agent_slug(assigned_to) in self._BOARD_AGENTS:
|
||||
|
||||
@@ -129,15 +129,14 @@ class SelfHealEngine(BaseService):
|
||||
Bounded + deduped: skips a regression that already has an open self-heal
|
||||
task (by fingerprint), honors the per-cycle and rolling open-task caps,
|
||||
and resolves the repo to RoboCo's own project. Each task is created
|
||||
PENDING + assigned to the Main PM + ``confirmed_by_human=False`` so it
|
||||
sits inert until the CEO Approve-&-Starts it — origination is the loop's
|
||||
last act. It is assigned to the Main PM AGENT (not merely team=main_pm)
|
||||
so that, once the CEO approves, the orchestrator dispatches it straight to
|
||||
that agent via the assigned-PM path instead of the unassigned-team
|
||||
routing, which picks up slowly or not at all. The confirmed_by_human gate
|
||||
still holds it inert until approval (the dispatcher's self-heal skip sits
|
||||
before the assigned/unassigned split). It NEVER calls start / approve /
|
||||
merge / deploy. Flushes; the caller commits.
|
||||
PENDING + assigned to the Main PM agent (not merely team=main_pm) so the
|
||||
orchestrator dispatches it straight to that agent via the assigned-PM
|
||||
path. RoboCo self-heals autonomously: the fix task dispatches WITHOUT a
|
||||
CEO Approve-&-Start (``confirmed_by_human=True`` up front) — that is the
|
||||
Intake/board flow, not this one. It is safe because the loop only OPENS
|
||||
the task; the fix itself still ships through the normal gates
|
||||
(dev -> QA -> PR review -> the CEO's merge), and the loop NEVER calls
|
||||
start / approve / merge / deploy. Flushes; the caller commits.
|
||||
"""
|
||||
task_svc = get_task_service(self.session)
|
||||
project_svc = get_project_service(self.session)
|
||||
@@ -175,8 +174,10 @@ class SelfHealEngine(BaseService):
|
||||
f"Evidence: {obs.raw_ref}\n\n"
|
||||
"Investigate and fix the regression at its root so CI "
|
||||
"returns to green. This task was opened automatically by "
|
||||
"the self-heal loop and is PENDING your Approve-&-Start; "
|
||||
"nothing runs until you approve it."
|
||||
"the self-heal loop and is READY TO START NOW — no "
|
||||
"approval needed; pick it up and coordinate the fix. It "
|
||||
"still ships through the normal gates (QA, PR review, and "
|
||||
"the CEO's merge)."
|
||||
),
|
||||
acceptance_criteria=[
|
||||
f"CI on {obs.repo_hint}'s default branch is green again",
|
||||
@@ -192,7 +193,7 @@ class SelfHealEngine(BaseService):
|
||||
project_id=cast("UUID", project.id),
|
||||
status=TaskStatus.PENDING,
|
||||
source=SELF_HEAL_SOURCE,
|
||||
confirmed_by_human=False,
|
||||
confirmed_by_human=True,
|
||||
)
|
||||
)
|
||||
# Carry the fingerprint so a later cycle sees this regression already
|
||||
|
||||
@@ -4486,14 +4486,6 @@ class TaskService(BaseService):
|
||||
# team and does not affect dispatch (which routes by assignee, not team).
|
||||
task.team = cast("Any", Team.MAIN_PM)
|
||||
|
||||
# A self-heal fix task is opened unconfirmed and held OUT of dispatch
|
||||
# until here (the PM dispatcher skips source='self_heal' while
|
||||
# confirmed_by_human is False). The CEO's Approve-&-Start IS that human
|
||||
# confirmation, so flip the gate now and the task dispatches normally.
|
||||
# Other sources don't carry this hold and are unaffected.
|
||||
if getattr(task, "source", "") == SELF_HEAL_SOURCE:
|
||||
task.confirmed_by_human = True
|
||||
|
||||
if notes:
|
||||
# Coordination metadata, not a human handoff — store as a marker so
|
||||
# quick_context carries only the structured ResumptionNote (no raw
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
"""Self-heal fix tasks dispatch autonomously through the PM dispatcher.
|
||||
|
||||
The loop opens a ``source='self_heal'`` task confirmed + assigned to the Main PM
|
||||
agent, so the dispatcher routes it through the assigned-PM path like any other
|
||||
task — there is no CEO Approve-&-Start hold (that gate is the Intake/board flow).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, cast
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
from roboco.runtime.orchestrator import AgentOrchestrator
|
||||
|
||||
|
||||
def _task(tid: str, source: str, assigned_to: str | None = None) -> dict[str, Any]:
|
||||
return {"id": tid, "source": source, "assigned_to": assigned_to}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_self_heal_task_dispatches_through_the_assigned_pm_path() -> None:
|
||||
tasks = [
|
||||
_task("A", "self_heal", assigned_to="main-pm"), # assigned → assigned-PM path
|
||||
_task("B", "self_heal"), # unassigned self-heal → routing
|
||||
_task("C", "manual"), # ordinary unassigned → routing
|
||||
]
|
||||
stub = MagicMock()
|
||||
stub._fetch_tasks = AsyncMock(return_value=tasks)
|
||||
stub._is_task_handled_this_tick = MagicMock(return_value=False)
|
||||
stub._resolve_agent_slug = MagicMock(return_value="main-pm")
|
||||
stub._BOARD_AGENTS = frozenset()
|
||||
stub._route_unassigned_pm_task = AsyncMock()
|
||||
stub._handle_pm_assigned_task = AsyncMock()
|
||||
stub._handle_board_assigned_task = AsyncMock()
|
||||
|
||||
client: Any = MagicMock()
|
||||
await AgentOrchestrator._dispatch_pm_work(cast("AgentOrchestrator", stub), client)
|
||||
|
||||
# The assigned self-heal task is handed to the assigned-PM path (spawned),
|
||||
# NOT held — it dispatches without any CEO approval.
|
||||
handled = [c.args[0]["id"] for c in stub._handle_pm_assigned_task.await_args_list]
|
||||
assert handled == ["A"]
|
||||
# Unassigned tasks (self-heal or not) route normally.
|
||||
routed = [c.args[1]["id"] for c in stub._route_unassigned_pm_task.await_args_list]
|
||||
assert set(routed) == {"B", "C"}
|
||||
@@ -1,55 +0,0 @@
|
||||
"""The PM dispatcher holds an unconfirmed self-heal task OUT of dispatch.
|
||||
|
||||
The load-bearing invariant: a source='self_heal' task the loop opened must NOT
|
||||
be routed / claimed / spawned while confirmed_by_human is False — it sits inert
|
||||
until the CEO Approve-&-Starts it (which flips the flag). Once confirmed it
|
||||
routes like any other task. Mirrors how PR-review tasks are skipped.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, cast
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
from roboco.runtime.orchestrator import AgentOrchestrator
|
||||
|
||||
|
||||
def _task(
|
||||
tid: str, source: str, confirmed: bool, assigned_to: str | None = None
|
||||
) -> dict[str, Any]:
|
||||
return {
|
||||
"id": tid,
|
||||
"source": source,
|
||||
"confirmed_by_human": confirmed,
|
||||
"assigned_to": assigned_to,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unconfirmed_self_heal_task_is_held_out_of_dispatch() -> None:
|
||||
tasks = [
|
||||
_task("A", "self_heal", False), # held — must NOT route until approved
|
||||
_task("B", "self_heal", True), # CEO-approved → routes
|
||||
_task("C", "manual", False), # ordinary task → routes
|
||||
# The loop now assigns the Main PM agent up front, so the hold must
|
||||
# survive an assignee too — the self-heal skip sits before the
|
||||
# assigned/unassigned split, so an assigned-but-unconfirmed task is
|
||||
# neither routed nor handed to the assigned-PM path.
|
||||
_task("D", "self_heal", False, assigned_to="main-pm"),
|
||||
]
|
||||
stub = MagicMock()
|
||||
stub._fetch_tasks = AsyncMock(return_value=tasks)
|
||||
stub._is_task_handled_this_tick = MagicMock(return_value=False)
|
||||
stub._route_unassigned_pm_task = AsyncMock()
|
||||
stub._handle_pm_assigned_task = AsyncMock()
|
||||
|
||||
client: Any = MagicMock()
|
||||
await AgentOrchestrator._dispatch_pm_work(cast("AgentOrchestrator", stub), client)
|
||||
|
||||
routed = [c.args[1]["id"] for c in stub._route_unassigned_pm_task.await_args_list]
|
||||
assert "A" not in routed # the unconfirmed self-heal task stays inert
|
||||
assert set(routed) == {"B", "C"}
|
||||
# The assigned-but-unconfirmed self-heal task (D) is held before the
|
||||
# assigned-PM branch — never handed to _handle_pm_assigned_task.
|
||||
stub._handle_pm_assigned_task.assert_not_awaited()
|
||||
@@ -3,15 +3,14 @@
|
||||
The loop opens a fix task only when ``self_heal_originate_enabled``, dedupes one
|
||||
open task per regression fingerprint, honors the per-cycle and rolling open-task
|
||||
caps, and creates the task PENDING + assigned to the Main PM agent +
|
||||
``confirmed_by_human=False`` so it sits inert until the CEO Approve-&-Starts it.
|
||||
Crucially it NEVER calls start / approve / merge / deploy — asserted here.
|
||||
``confirmed_by_human=True`` so it dispatches autonomously (no CEO Approve-&-Start).
|
||||
Crucially the loop NEVER calls start / approve / merge / deploy — asserted here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
from unittest.mock import AsyncMock
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
import pytest
|
||||
from roboco.config import settings as cfg
|
||||
@@ -22,7 +21,6 @@ from roboco.services.notification import NotificationService
|
||||
from roboco.services.self_heal_engine import SelfHealEngine
|
||||
from roboco.services.task import TaskService, get_task_service
|
||||
from roboco.services.telemetry import TelemetrySample
|
||||
from sqlalchemy import select
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
@@ -152,7 +150,7 @@ async def test_originate_creates_pending_main_pm_assigned_task(
|
||||
# Assigned to the Main PM agent up front (not just team=main_pm) so that, once
|
||||
# the CEO confirms it, the orchestrator dispatches it straight to that agent.
|
||||
assert task.assigned_to == MAIN_PM_UUID
|
||||
assert task.confirmed_by_human is False # still inert until Approve-&-Start
|
||||
assert task.confirmed_by_human is True # auto-confirmed → dispatches autonomously
|
||||
assert task.team == Team.MAIN_PM
|
||||
assert task.source == "self_heal"
|
||||
assert task.acceptance_criteria # non-empty (AC-guardrail)
|
||||
@@ -242,45 +240,17 @@ async def test_loop_never_starts_or_approves(
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ceo_approve_and_start_flips_the_confirmation_gate(
|
||||
async def test_originated_task_is_confirmed_for_autonomous_dispatch(
|
||||
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The opened task is unconfirmed (held out of dispatch); the CEO's
|
||||
approve_and_start flips confirmed_by_human=True so it can then dispatch."""
|
||||
"""The opened task is confirmed up front, so the PM dispatcher picks it up
|
||||
without any CEO Approve-&-Start (that gate is the Intake/board flow)."""
|
||||
await _seed_project(db_session)
|
||||
# approve_and_start reassigns to the main-pm agent — get-or-create by slug
|
||||
# (the full suite may already have committed a "main-pm" agent).
|
||||
existing_pm = (
|
||||
await db_session.execute(select(AgentTable).where(AgentTable.slug == "main-pm"))
|
||||
).scalar_one_or_none()
|
||||
if existing_pm is None:
|
||||
db_session.add(
|
||||
AgentTable(
|
||||
id=uuid4(),
|
||||
name="Main PM",
|
||||
slug="main-pm",
|
||||
role=AgentRole.MAIN_PM,
|
||||
team=Team.MAIN_PM,
|
||||
status=AgentStatus.ACTIVE,
|
||||
model_config={},
|
||||
system_prompt="pm",
|
||||
capabilities=[],
|
||||
permissions={},
|
||||
metrics={},
|
||||
)
|
||||
)
|
||||
await db_session.flush()
|
||||
_enable(monkeypatch)
|
||||
# approve_and_start emits a stream event; stub it out (no bus in the test).
|
||||
monkeypatch.setattr(TaskService, "_emit_task_event", AsyncMock())
|
||||
|
||||
await SelfHealEngine(
|
||||
db_session, source=_FakeSource([_breach("ci:roboco")])
|
||||
).run_cycle()
|
||||
task = (await get_task_service(db_session).list_open_self_heal_tasks())[0]
|
||||
assert task.confirmed_by_human is False # inert: held out of dispatch
|
||||
|
||||
started = await TaskService(db_session).approve_and_start(UUID(str(task.id)))
|
||||
assert started is not None
|
||||
assert started.confirmed_by_human is True # gate flipped → now dispatchable
|
||||
assert started.status == TaskStatus.PENDING # reassignment, not a transition
|
||||
assert task.confirmed_by_human is True # dispatches autonomously
|
||||
assert task.status == TaskStatus.PENDING # ready for the PM dispatcher
|
||||
assert task.assigned_to == MAIN_PM_UUID # straight to the Main PM agent
|
||||
|
||||
Reference in New Issue
Block a user