Feat/autonomous maintenance (#264)

* feat(ci-watch): config flags Default-off CI-watch config (mirrors self_heal_*): ci_watch_enabled, ci_watch_default_workflow (ci.yml), ci_watch_interval_seconds (1800), ci_watch_max_open_tasks (3), ci_watch_max_per_cycle (1). Registers ci_watch_enabled in the panel FEATURE_FLAGS. 4 tests. * feat(ci-watch): per-project ci_watch_enabled/workflow (migration 048) Adds projects.ci_watch_enabled (bool NOT NULL default false) + projects.ci_watch_workflow (varchar null) — the per-project opt-in for multi-repo CI-watch. ProjectTable + Pydantic Project fields + migration 048 (off 047_ws_single_active). Real upgrade->downgrade->upgrade chain verified against a throwaway Postgres; 2 ORM round-trip tests. * feat(runtime): prune dangling agent images in the background sweeper Every agent-image rebuild orphans the prior build's layers as an untagged <none> image; across deploys these pile up (the operator hit ~80). The sweeper now runs 'docker image prune -f --filter dangling=true' (dangling only — a tagged image or one backing a running container is never dangling), throttled to settings.image_prune_interval_seconds (default 6h) and gated by image_prune_enabled (default on). Best-effort: any failure is logged, never raised into the sweeper. Mirrors the transcript-retention prune. 4 tests. * feat(ci-watch): source tag + open-task dedupe query CI_WATCH_SOURCE='ci_watch' + TaskService.list_open_ci_watch_tasks(git_url=None): non-terminal ci_watch tasks (the dedupe + open-cap basis), optionally scoped to one repo by git_url — a monorepo registers several cell-projects on one git_url, so dedupe keys on the repo, not the slug. 2 real-PG tests. * feat(ci-watch): multi-project CI telemetry fan-out MultiProjectCITelemetrySource.fetch(projects) reuses the hardened per-project get_latest_ci_conclusion for each opted-in project (passing its ci_watch_workflow or the configured default). Per-project isolation: a GitHub error or absent signal yields NO sample (unknown, never read as green) and never aborts the sweep; only a real conclusion yields a sample (fail→breach, pass→non-breach). self-heal source untouched. 3 tests + self-heal regression green. * feat(ci-watch): engine — fan-out, originate, dedupe, cap CiWatchEngine.run_cycle(projects) mirrors SelfHealEngine: assess via MultiProjectCITelemetrySource, open one PENDING ci_watch fix task per red repo (team=main_pm, assigned_to=main-pm, confirmed_by_human=True so it dispatches without an Approve-&-Start — the fe029fe3 lesson), never starts/approves/merges. Dedupe per git_url (monorepo → one fix task per repo) + per-cycle/rolling caps. Default-off; disabled → no-op. 5 real-PG tests (red→one task, dedupe, cap, green/none→nothing, disabled). * feat(ci-watch): orchestrator loop tick + watch-set loader _ci_watch_loop (registered in start(), cancelled in stop(), separate from the untouched self-heal loop): dormant unless ci_watch_enabled; each interval loads the watch set (ci_watch_enabled projects, collapsed one-per-repo via the existing _projects_one_per_repo) and runs CiWatchEngine.run_cycle, committing opened tasks. _run_ci_watch_cycle extracted for testing; loud warning when enabled-but-empty. confirmed_by_human=True on the originated task means it dispatches without an Approve-&-Start (no stranding, the fe029fe3 lesson). 5 tests (disabled no-op, watch-set filter+one-per-repo, empty warn, engine run). * docs(ci-watch): CHANGELOG + CLAUDE.md for multi-repo CI-watch Document CI-watch (Added) in the CHANGELOG and the Self-Healing & Feature Flags section of CLAUDE.md — it generalizes self-heal to opted-in projects, reuses the hardened per-project CI lookup, never auto-merges, default-off. Adds the ci_watch_enabled flag to the feature-flags enumeration. * feat(dep-update): config flags Default-off dep-update config (mirrors self_heal_*/ci_watch_*): dep_update_enabled, dep_update_interval_seconds (604800 = weekly), dep_update_max_open_tasks (3), dep_update_max_per_cycle (1). Registers dep_update_enabled in FEATURE_FLAGS. 4 tests. * feat(dep-update): per-project dep_update_command/paths (migration 049) Adds projects.dep_update_command (varchar null) + dep_update_paths (varchar[] null) — the per-project opt-in for the dependency-update bot. ProjectTable + Pydantic Project fields + migration 049 (off 048_ci_watch_project_cols). Real upgrade->downgrade->upgrade chain verified on a throwaway Postgres; 2 ORM tests. * feat(dep-update): source tag + open-task dedupe query DEP_UPDATE_SOURCE='dep_update' + TaskService.list_open_dep_update_tasks(git_url=None): non-terminal dep_update tasks (dedupe + open-cap basis), optionally scoped to one repo by git_url (monorepo → one open dependency-update task per repo). 2 real-PG tests. * feat(dep-update): read-only lockfile-diff probe WorkspaceService.dry_upgrade_changes_lockfile(project): clones the project's read clone into a throwaway dir (--no-hardlinks, so the read clone is never mutated), runs project.dep_update_command (no shell, shlex.split), and reports whether any lockfile path (dep_update_paths or inferred uv.lock/pnpm-lock.yaml) is dirty. Fail-safe: null/failing command → False (don't originate on a broken probe), logged; throwaway always removed; never commits/pushes. 5 real-git tests. * feat(dep-update): engine — detect, originate, dedupe, cap DepUpdateEngine.run_cycle(projects) mirrors SelfHealEngine/CiWatchEngine: for each opted-in project (dep_update_command set) with updates available (the read-only probe), open one PENDING dep_update task (team=main_pm, assigned-to main-pm, confirmed_by_human=True), never starts/approves/merges. Cheap checks (command, per-git_url dedupe) before the expensive probe; per-cycle + rolling caps. Default-off; disabled → no-op. 6 real-PG tests. * feat(dep-update): weekly orchestrator loop tick _dep_update_loop (registered in start(), cancelled in stop(), separate from the self-heal + CI-watch loops): dormant unless dep_update_enabled; each interval (default weekly) loads projects with a dep_update_command (one-per-repo) and runs DepUpdateEngine.run_cycle, committing opened tasks. _run_dep_update_cycle extracted for testing; loud warning when enabled-but-no-commands. Refactored stop() to cancel background tasks via a shared _cancel_background_task loop (keeps it under xenon B as the loop count grows). 4 loop tests. Task 7 (anti-stranding dispatch guard) is satisfied by construction: no dispatcher skip targets source='dep_update', and the engine sets confirmed_by_human=True (the fe029fe3 lesson), asserted in the engine tests — so the originated task dispatches via the assigned-PM path, never stranded. * docs(dep-update): CHANGELOG + CLAUDE.md for the dependency-update bot Document the dep-update bot (Added) in the CHANGELOG and the Self-Healing & Feature Flags section of CLAUDE.md — read-only lockfile-diff probe, never auto-merges, per-project opt-in via dep_update_command, default-off. Adds the dep_update_enabled flag to the feature-flags enumeration. * feat(ci-watch): route fix-task notification to the project's cell PM On opening a fix task, CiWatchEngine notifies the red project's own cell PM (resolved from project.assigned_cell via foundation AGENTS — e.g. BACKEND → be-pm), not the CEO, once per project per cycle. Best-effort: a notification failure never rolls back the origination. Adds _cell_pm_slug_for + _notify_cell_pm. 1 real-PG test (asserts to_agent='be-pm', not 'ceo'). * feat(ci-watch,dep-update): expose per-project opt-ins in the project API Add ci_watch_enabled/ci_watch_workflow + dep_update_command/dep_update_paths to ProjectUpdate, ProjectUpdateRequest, the PATCH route mapping, ProjectResponse, and project_to_response — so the panel edit-project dialog can read + set the per-project autonomy opt-ins (the columns were unreachable through the API before). Also threads the previously-dropped quality_command through the update route. 1 real-PG update round-trip test. * feat(ci-watch,dep-update): panel project-edit fields for the per-project opt-ins Adds an 'Autonomous Maintenance' section to the edit-project dialog: a CI-watch enable switch + workflow input, and a dependency-update command + lockfile-paths input (comma-separated → list). Threads the four fields through the Project / ProjectUpdate TS types and the mock-mode create fixture. The global on/off toggles already live in Settings → Feature Flags; these are the per-project opt-ins. panel tsc --noEmit + eslint green. * docs(0.12): CI-watch + dep-update bot + image-prune across user docs + RAG New docs/optional/autonomous-maintenance.md (mirrors self-heal.md) covering both engines; optional/index rows; panel settings + projects-and-products notes for the Feature Flags toggles + the edit-project Autonomous Maintenance fields; resilience note for the dangling-image prune; env-reference + RAG config-reference tables for all ROBOCO_CI_WATCH_* / ROBOCO_DEP_UPDATE_* / ROBOCO_IMAGE_PRUNE_* vars; mkdocs nav entry. reflow-check green; prompts unchanged (operator-facing, not agent-facing). * chore(release): 0.12.0 Cut [Unreleased] -> [0.12.0] (CI-watch + dep-update bot + image-prune housekeeping + the post-0.11.1 run-hardening fixes). Bumps all 8 canonical version refs to 0.12.0 (pyproject / uv.lock roboco pkg / panel package.json / __init__ / config.app_version + the README / deployment / agent-image-tag examples). * fix(pr-review): repo-scope external-PR dedupe (no duplicate review on a monorepo) external_review_task_exists keyed on (project_id, pr, head_sha), but a monorepo registers several cell-projects on one git_url and the poll already collapses to one canonical project per repo — so once a review task was re-pointed to a sibling project, the next poll (checking the canonical project) no longer saw it and opened a second review of the same PR (observed: PR #131 reviewed once on guard-core-saas-frontend, once on -backend). Dedupe now spans every project sharing the PR's repo (git_url); re-review on a new head SHA still works; a genuinely different repo with the same PR number is independent. 3 real-PG tests. --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-08-03 07:23:24 +02:00 · 2026-06-25 21:11:36 +02:00
parent 2c403c77a2
commit 153723406e
49 changed files with 2828 additions and 57 deletions
@@ -0,0 +1,87 @@
+"""The orchestrator CI-watch loop: dormant when off, runs the engine when on.
+
+Dormant unless ``ci_watch_enabled``; loads the watch set (opted-in projects, one
+per repo), warns when enabled-but-empty, and runs CiWatchEngine.run_cycle each
+interval. Separate from the single-repo self-heal loop.
+"""
+
+from __future__ import annotations
+
+from contextlib import asynccontextmanager
+from typing import Any
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+from roboco.config import settings
+from roboco.runtime.orchestrator import AgentOrchestrator
+
+
+def _orch() -> AgentOrchestrator:
+    return AgentOrchestrator.__new__(AgentOrchestrator)
+
+
+@pytest.mark.asyncio
+async def test_loop_noop_when_disabled(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setattr(settings, "ci_watch_enabled", False)
+    orch = _orch()
+    cycle = AsyncMock()
+    orch._run_ci_watch_cycle = cycle  # type: ignore[method-assign]
+    await orch._ci_watch_loop()  # must return immediately, no infinite loop
+    cycle.assert_not_awaited()
+
+
+@pytest.mark.asyncio
+async def test_load_watch_set_filters_enabled_one_per_repo() -> None:
+    orch = _orch()
+    on_a = MagicMock(slug="be", git_url="https://x/a.git", ci_watch_enabled=True)
+    on_a2 = MagicMock(slug="fe", git_url="https://x/a.git", ci_watch_enabled=True)
+    off = MagicMock(slug="c", git_url="https://x/c.git", ci_watch_enabled=False)
+    svc = MagicMock()
+    svc.list_all = AsyncMock(return_value=[on_a, on_a2, off])
+    with patch("roboco.services.project.get_project_service", return_value=svc):
+        watch = await orch._load_ci_watch_set(MagicMock())
+    assert len(watch) == 1  # opt-out excluded; same-repo cell-projects collapsed
+    assert watch[0].git_url == "https://x/a.git"
+
+
+def _db_ctx(db: Any):
+    @asynccontextmanager
+    async def _ctx() -> Any:
+        yield db
+
+    return _ctx
+
+
+@pytest.mark.asyncio
+async def test_cycle_warns_and_skips_engine_when_empty() -> None:
+    orch = _orch()
+    orch._load_ci_watch_set = AsyncMock(return_value=[])  # type: ignore[method-assign]
+    get_eng = MagicMock()
+    with (
+        patch("roboco.db.get_db_context", _db_ctx(MagicMock())),
+        patch("roboco.services.ci_watch_engine.get_ci_watch_engine", get_eng),
+    ):
+        await orch._run_ci_watch_cycle()
+    get_eng.assert_not_called()  # empty watch set → no engine run
+
+
+@pytest.mark.asyncio
+async def test_cycle_runs_engine_when_watch_set_present() -> None:
+    orch = _orch()
+    watch = [MagicMock()]
+    orch._load_ci_watch_set = AsyncMock(return_value=watch)  # type: ignore[method-assign]
+    db = MagicMock()
+    db.commit = AsyncMock()
+    engine = MagicMock()
+    engine.run_cycle = AsyncMock(return_value=[])
+    with (
+        patch("roboco.db.get_db_context", _db_ctx(db)),
+        patch(
+            "roboco.services.ci_watch_engine.get_ci_watch_engine",
+            return_value=engine,
+        ) as get_eng,
+    ):
+        await orch._run_ci_watch_cycle()
+    get_eng.assert_called_once()
+    engine.run_cycle.assert_awaited_once_with(watch)
+    db.commit.assert_awaited_once()
@@ -0,0 +1,87 @@
+"""The orchestrator dep-update loop: dormant when off, runs the engine when on.
+
+Dormant unless ``dep_update_enabled``; loads the eligible set (projects with a
+dep_update_command, one per repo), warns when enabled-but-empty, and runs
+DepUpdateEngine.run_cycle each interval. Separate from self-heal/CI-watch loops.
+"""
+
+from __future__ import annotations
+
+from contextlib import asynccontextmanager
+from typing import Any
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+from roboco.config import settings
+from roboco.runtime.orchestrator import AgentOrchestrator
+
+
+def _orch() -> AgentOrchestrator:
+    return AgentOrchestrator.__new__(AgentOrchestrator)
+
+
+@pytest.mark.asyncio
+async def test_loop_noop_when_disabled(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setattr(settings, "dep_update_enabled", False)
+    orch = _orch()
+    cycle = AsyncMock()
+    orch._run_dep_update_cycle = cycle  # type: ignore[method-assign]
+    await orch._dep_update_loop()
+    cycle.assert_not_awaited()
+
+
+@pytest.mark.asyncio
+async def test_load_set_filters_command_one_per_repo() -> None:
+    orch = _orch()
+    on_a = MagicMock(slug="be", git_url="https://x/a.git", dep_update_command="uv -U")
+    on_a2 = MagicMock(slug="fe", git_url="https://x/a.git", dep_update_command="uv -U")
+    off = MagicMock(slug="c", git_url="https://x/c.git", dep_update_command=None)
+    svc = MagicMock()
+    svc.list_all = AsyncMock(return_value=[on_a, on_a2, off])
+    with patch("roboco.services.project.get_project_service", return_value=svc):
+        eligible = await orch._load_dep_update_set(MagicMock())
+    assert len(eligible) == 1  # no-command excluded; same-repo collapsed
+    assert eligible[0].git_url == "https://x/a.git"
+
+
+def _db_ctx(db: Any):
+    @asynccontextmanager
+    async def _ctx() -> Any:
+        yield db
+
+    return _ctx
+
+
+@pytest.mark.asyncio
+async def test_cycle_warns_and_skips_engine_when_empty() -> None:
+    orch = _orch()
+    orch._load_dep_update_set = AsyncMock(return_value=[])  # type: ignore[method-assign]
+    get_eng = MagicMock()
+    with (
+        patch("roboco.db.get_db_context", _db_ctx(MagicMock())),
+        patch("roboco.services.dep_update_engine.get_dep_update_engine", get_eng),
+    ):
+        await orch._run_dep_update_cycle()
+    get_eng.assert_not_called()
+
+
+@pytest.mark.asyncio
+async def test_cycle_runs_engine_when_eligible_present() -> None:
+    orch = _orch()
+    eligible = [MagicMock()]
+    orch._load_dep_update_set = AsyncMock(return_value=eligible)  # type: ignore[method-assign]
+    db = MagicMock()
+    db.commit = AsyncMock()
+    engine = MagicMock()
+    engine.run_cycle = AsyncMock(return_value=[])
+    with (
+        patch("roboco.db.get_db_context", _db_ctx(db)),
+        patch(
+            "roboco.services.dep_update_engine.get_dep_update_engine",
+            return_value=engine,
+        ) as get_eng,
+    ):
+        await orch._run_dep_update_cycle()
+    get_eng.assert_called_once()
+    engine.run_cycle.assert_awaited_once_with(eligible)
+    db.commit.assert_awaited_once()
@@ -0,0 +1,85 @@
+"""The orchestrator sweeper prunes dangling Docker images (gated + throttled).
+
+Each agent-image rebuild orphans the prior build's layers as an untagged
+``<none>`` image; over many deploys these pile up. The sweeper reclaims them
+with ``docker image prune -f --filter dangling=true`` (dangling only — never a
+tagged image or one backing a running container), gated by
+``image_prune_enabled`` and throttled to ``image_prune_interval_seconds``.
+"""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime
+from typing import Any
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+from roboco.config import settings
+from roboco.runtime.orchestrator import AgentOrchestrator
+
+
+def _orch() -> AgentOrchestrator:
+    orch = AgentOrchestrator.__new__(AgentOrchestrator)  # bypass __init__
+    orch._last_image_prune = None
+    return orch
+
+
+def _fake_proc(returncode: int = 0) -> MagicMock:
+    proc = MagicMock()
+    proc.returncode = returncode
+    proc.communicate = AsyncMock(return_value=(b"Total reclaimed space: 1.2GB", b""))
+    return proc
+
+
+@pytest.mark.asyncio
+async def test_prunes_dangling_when_enabled_and_due(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    monkeypatch.setattr(settings, "image_prune_enabled", True)
+    orch = _orch()
+    spawn = AsyncMock(return_value=_fake_proc())
+    with patch("roboco.runtime.orchestrator.asyncio.create_subprocess_exec", spawn):
+        await orch._sweep_dangling_images()
+    spawn.assert_awaited_once()
+    args: tuple[Any, ...] = spawn.await_args.args
+    assert args[:6] == (
+        "docker",
+        "image",
+        "prune",
+        "-f",
+        "--filter",
+        "dangling=true",
+    )
+    assert orch._last_image_prune is not None
+
+
+@pytest.mark.asyncio
+async def test_no_prune_when_disabled(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setattr(settings, "image_prune_enabled", False)
+    orch = _orch()
+    spawn = AsyncMock()
+    with patch("roboco.runtime.orchestrator.asyncio.create_subprocess_exec", spawn):
+        await orch._sweep_dangling_images()
+    spawn.assert_not_awaited()
+
+
+@pytest.mark.asyncio
+async def test_throttled_when_recently_pruned(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    monkeypatch.setattr(settings, "image_prune_enabled", True)
+    orch = _orch()
+    orch._last_image_prune = datetime.now(UTC)  # pruned moments ago
+    spawn = AsyncMock()
+    with patch("roboco.runtime.orchestrator.asyncio.create_subprocess_exec", spawn):
+        await orch._sweep_dangling_images()
+    spawn.assert_not_awaited()
+
+
+@pytest.mark.asyncio
+async def test_best_effort_swallows_errors(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setattr(settings, "image_prune_enabled", True)
+    orch = _orch()
+    spawn = AsyncMock(side_effect=RuntimeError("no docker"))
+    with patch("roboco.runtime.orchestrator.asyncio.create_subprocess_exec", spawn):
+        await orch._sweep_dangling_images()  # must not raise