mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
[90c9474c] Auditor revival: scheduled audit trigger and reactive alert producers (#499)
* [927e64d5] Backend slice: auditor scheduled trigger and reactive alert producers (#496) * [1f2cdb4b] Reactive alert producers at QA-fail and rework (#492) * [1f2cdb4b] feat(services): add auditor-targeted rework alert producers at QA-fail and rework chokepoints * [1f2cdb4b] test(services): fix mypy typing in auditor alert producer unit tests * [1f2cdb4b] docs(backend): document reactive auditor rework alert producers in map and role docs --------- Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> * [5173415f] Scheduled audit trigger, config, and sweep prompt (#493) * [5173415f] Add scheduled audit trigger, interval config, sweep prompt, and focused tests * [5173415f] Allow ROBOCO_AUDIT_INTERVAL_SECONDS=0 to disable scheduled sweeps * [5173415f] docs(audit): document scheduled auditor sweeps and ROBOCO_AUDIT_INTERVAL_SECONDS --------- Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> * [a26c18b9] E2E smoke test for auditor triggers (#495) * [a26c18b9] Add e2e smoke test for auditor scheduled and reactive triggers * [a26c18b9] docs(tests): add e2e smoke test catalog and changelog entry for auditor triggers --------- Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> * [3bc47cdc] Fix _fresh_orchestrator state for auditor trigger e2e tests (#497) * [3bc47cdc] fix(tests): initialize orchestrator state in _fresh_orchestrator helper * [3bc47cdc] docs(changelog): add _fresh_orchestrator test harness fix entry --------- Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> * [8323cd50] Fix e2e smoke regression on assembled cell PR #496 (#498) * [8323cd50] fix(e2e_smoke): repair auditor-trigger smoke tests and harden harness * [8323cd50] docs(tests): document e2e smoke harness hardening for PR #498 --------- Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> --------- Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> * [37e6d999] Backend: repair failing CI checks on auditor revival PR #499 (#503) * [6e79bada] Triage and fix Python quality gate and Analyze (python) failures (#501) * [6e79bada] fix(task): replace type ignore with forward-reference cast for SQLAlchemy Mapped UUID in get_all_descendants * [6e79bada] fix(notification_delivery): add generic type arguments to dict return types in get_ack_status and get_delivery_summary * [6e79bada] docs(changelog): add Python quality gate type-hygiene fixes to Unreleased --------- Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> * [50e7e104] Triage Analyze (javascript-typescript) failure on backend-only diff (#500) * [50e7e104] Split CodeQL workflow so JS/TS analyzer only runs on panel changes * [50e7e104] docs(backend): document split CodeQL workflow triggers and branch protection notes --------- Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> * [203c426b] Triage and fix e2e lifecycle smoke (scripted agents) failure (#502) * [203c426b] fix(orchestrator): pre-initialize _instances in __new__ so __init__-bypass tests survive _dispatch_audit_work; allow audit_interval_seconds=0; mount /api/notifications in e2e harness * [203c426b] fix(e2e_smoke): restore ROBOCO_AGENT_TOKEN isolation and clarify /api/notifications mount comment * [203c426b] docs(map): document orchestrator __new__ pre-init and e2e harness token isolation for auditor-revival smoke fix --------- Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> * [48cb05c2] Fix remaining e2e lifecycle smoke (scripted agents) failure on auditor-revival PR #503 (#504) * [48cb05c2] Harden AgentOrchestrator __new__ pre-init for auditor dispatch state * [48cb05c2] Document auditor-dispatch pre-init rationale in AgentOrchestrator __new__ * [48cb05c2] docs(orchestrator): extend __new__ pre-init docs for auditor-dispatch state --------- Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> --------- Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech> * [90c9474c] intake: ambient workspace note + dedupe scope clones by git_url Two intake follow-ups folded into 90c9474c's spec Notes: (a) _resolve_intake_ambient now prepends a workspace note so the intake agent knows its cwd holds clones of every project in the scope (the primary at cwd, siblings alongside under /data/workspaces) and drafts against the real trees via Grep/Glob/Read, not from memory. (b) _clone_intake_scope dedupes slugs by git_url before cloning. A multi-project scope can list several projects pointing at one repo (a monorepo's cell-projects share a git_url); cloning each produced redundant identical workspaces. Mirrors CI-watch's per-git_url dedupe: keep the first slug per non-empty git_url; a project with no/empty git_url is never collapsed onto another so distinct local repos still clone. The dedupe is a pure static helper (_dedupe_slugs_by_git_url) with unit coverage. --------- Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
co-authored by
Backend Developer 2
Backend Documenter
Backend Developer 1
Renn F
parent
69271f9e98
commit
a3524da5f8
+10
-1
@@ -189,7 +189,16 @@ async def _test_database_url() -> AsyncIterator[str]:
|
||||
pgvector_engine = create_async_engine(test_url_async, future=True)
|
||||
try:
|
||||
async with pgvector_engine.begin() as conn:
|
||||
await conn.execute(text("CREATE EXTENSION IF NOT EXISTS vector"))
|
||||
try:
|
||||
await conn.execute(text("CREATE EXTENSION IF NOT EXISTS vector"))
|
||||
except Exception:
|
||||
# Dev/sandbox Postgres may lack pgvector; the core schema does
|
||||
# not require it and the e2e smoke suite does not exercise RAG.
|
||||
# Continue so tests can run in lightweight sandboxes.
|
||||
warnings.warn(
|
||||
"pgvector extension unavailable - continuing without it",
|
||||
stacklevel=2,
|
||||
)
|
||||
finally:
|
||||
await pgvector_engine.dispose()
|
||||
|
||||
|
||||
@@ -322,6 +322,7 @@ def _make_admin_clone(root: Path, origin: Path) -> Path:
|
||||
def _build_app(gh: _FakeGitHub) -> FastAPI:
|
||||
from roboco.api.middleware import setup_middleware
|
||||
from roboco.api.routes.health import router as health_router
|
||||
from roboco.api.routes.notifications import router as notifications_router
|
||||
from roboco.api.routes.orchestrator import router as orchestrator_router
|
||||
from roboco.api.routes.settings import router as settings_router
|
||||
from roboco.api.routes.tasks import router as tasks_router
|
||||
@@ -344,6 +345,10 @@ def _build_app(gh: _FakeGitHub) -> FastAPI:
|
||||
# The REST task surface — scenario 3 drives the real CEO
|
||||
# approve-and-merge endpoint (the human gate) through it.
|
||||
app.include_router(tasks_router, prefix="/api/tasks")
|
||||
# The notifications router was already mounted here before the auditor
|
||||
# revival diff; it remains so reactive audit dispatch (``_dispatch_audit_work``)
|
||||
# can poll real ALERT rows end-to-end.
|
||||
app.include_router(notifications_router, prefix="/api/notifications")
|
||||
# Cloud-auth gate coverage smoke exercises the real _require_ceo and
|
||||
# require_panel_token dep paths on these routers.
|
||||
app.include_router(orchestrator_router, prefix="/api/orchestrator")
|
||||
@@ -472,6 +477,12 @@ class ScriptedAgent:
|
||||
os.environ["ROBOCO_AGENT_ROLE"] = self.role
|
||||
os.environ["ROBOCO_ORCHESTRATOR_URL"] = self.stack.base_url
|
||||
os.environ["ROBOCO_TOOL_MANIFEST_PATH"] = str(self._manifest_path)
|
||||
# The host agent environment may carry a real ROBOCO_AGENT_TOKEN issued
|
||||
# for the test runner's identity. flow_server reads it before each call
|
||||
# and forwards it in X-Agent-Token; the token won't match the ephemeral
|
||||
# test agent IDs and causes 401s. Drop it so tests run in the same
|
||||
# unsigned-token mode as CI.
|
||||
os.environ.pop("ROBOCO_AGENT_TOKEN", None)
|
||||
module = importlib.import_module(name)
|
||||
if getattr(module, "AGENT_ID", None) != str(self.agent_id):
|
||||
module = importlib.reload(module)
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
"""e2e smoke test for auditor triggers.
|
||||
|
||||
Exercises both the scheduled audit trigger path and the reactive alert producer
|
||||
path end-to-end, verifying they result in auditor-targeted work.
|
||||
|
||||
- Scheduled path: an in-process orchestrator instance polls the real e2e API,
|
||||
sees recent delivery activity, and calls ``spawn_agent(agent_id="auditor")``
|
||||
with the scheduled sweep prompt.
|
||||
- Reactive path: a real QA-fail POST creates an ``ALERT`` notification addressed
|
||||
to the auditor in the DB; the orchestrator's audit dispatcher fetches that
|
||||
alert and calls ``spawn_agent(agent_id="auditor")`` with the quality-alert
|
||||
prompt.
|
||||
|
||||
No real auditor container is spawned — ``spawn_agent`` is stubbed so the test
|
||||
asserts on the dispatch decision, not the LLM runtime.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from http import HTTPStatus
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import httpx
|
||||
from roboco.config import settings
|
||||
from roboco.models import NotificationType
|
||||
from roboco.models.base import TaskStatus
|
||||
from roboco.runtime.orchestrator import _SYSTEM_API_HEADERS, AgentOrchestrator
|
||||
from tests.e2e_smoke.arcs import seed_company, seed_project, seed_task
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from uuid import UUID
|
||||
|
||||
import pytest
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from tests.e2e_smoke.harness import E2EStack
|
||||
|
||||
|
||||
def _agent_headers(agent_id: Any, role: str) -> dict[str, str]:
|
||||
return {"X-Agent-ID": str(agent_id), "X-Agent-Role": role}
|
||||
|
||||
|
||||
def _seed_auditor_agent(stack: E2EStack) -> UUID:
|
||||
"""Seed the canonical auditor agent at its fixed foundation UUID.
|
||||
|
||||
``_resolve_agent_slug`` maps this UUID to ``"auditor"`` so the orchestrator
|
||||
recognises auditor-targeted notifications and spawns the right role.
|
||||
"""
|
||||
from roboco.db.tables import AgentTable
|
||||
from roboco.foundation import identity as _foundation
|
||||
from roboco.models import AgentRole, AgentStatus
|
||||
|
||||
async def _run(session: AsyncSession) -> UUID:
|
||||
auditor_id = _foundation.AGENTS["auditor"].uuid
|
||||
session.add(
|
||||
AgentTable(
|
||||
id=auditor_id,
|
||||
name="auditor",
|
||||
slug="auditor",
|
||||
role=AgentRole.AUDITOR,
|
||||
team=None,
|
||||
status=AgentStatus.ACTIVE,
|
||||
model_config={},
|
||||
system_prompt="auditor",
|
||||
capabilities=[],
|
||||
permissions={},
|
||||
metrics={},
|
||||
)
|
||||
)
|
||||
await session.flush()
|
||||
return auditor_id
|
||||
|
||||
auditor_id: UUID = stack.run_db(_run)
|
||||
return auditor_id
|
||||
|
||||
|
||||
def _notifications_for_task(
|
||||
stack: E2EStack, task_id: Any, notification_type: Any
|
||||
) -> list[dict[str, Any]]:
|
||||
from roboco.db.tables import NotificationTable
|
||||
from sqlalchemy import select
|
||||
|
||||
async def _run(session: AsyncSession) -> list[dict[str, Any]]:
|
||||
rows = (
|
||||
(
|
||||
await session.execute(
|
||||
select(NotificationTable).where(
|
||||
NotificationTable.related_task_id == task_id,
|
||||
NotificationTable.type == notification_type,
|
||||
)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
return [
|
||||
{
|
||||
"type": str(r.type),
|
||||
"related_task_id": r.related_task_id,
|
||||
"subject": r.subject,
|
||||
"priority": str(r.priority),
|
||||
"to_agents": list(r.to_agents),
|
||||
}
|
||||
for r in rows
|
||||
]
|
||||
|
||||
rows: list[dict[str, Any]] = stack.run_db(_run)
|
||||
return rows
|
||||
|
||||
|
||||
def _fresh_orchestrator(stack: E2EStack, monkeypatch: pytest.MonkeyPatch) -> Any:
|
||||
"""Return a bare orchestrator whose internal API points at the e2e app."""
|
||||
# internal_api_url is a computed property; patch its input api_url instead.
|
||||
monkeypatch.setattr(settings, "api_url", stack.base_url)
|
||||
orch: Any = AgentOrchestrator.__new__(AgentOrchestrator)
|
||||
# __new__ bypasses __init__, so the instance attributes that
|
||||
# _is_agent_active and _dispatch_audit_work read must be initialized here.
|
||||
orch._instances = {}
|
||||
orch._last_audit_spawn_at = None
|
||||
orch.spawn_agent = AsyncMock()
|
||||
return orch
|
||||
|
||||
|
||||
def test_scheduled_audit_trigger_spawns_auditor(
|
||||
e2e_stack: E2EStack, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The scheduled sweep spawns the auditor when delivery activity is recent."""
|
||||
stack = e2e_stack
|
||||
company = seed_company(stack)
|
||||
project_id, _project_slug = seed_project(stack, company)
|
||||
auditor_id = _seed_auditor_agent(stack)
|
||||
|
||||
# Any active delivery task counts as "recent activity" for the sweep gate.
|
||||
task_id = seed_task(
|
||||
stack,
|
||||
title="Scheduled audit smoke task",
|
||||
description="An in-progress task so the scheduled sweep has work to audit.",
|
||||
acceptance_criteria=["the scheduled path sees recent activity"],
|
||||
project_id=project_id,
|
||||
created_by=company.cell_pm_id,
|
||||
assigned_to=company.dev_id,
|
||||
claimed_by=company.dev_id,
|
||||
active_claimant_id=company.dev_id,
|
||||
status=TaskStatus.IN_PROGRESS,
|
||||
branch_name="feature/backend/e2e-scheduled-audit",
|
||||
)
|
||||
|
||||
orch = _fresh_orchestrator(stack, monkeypatch)
|
||||
monkeypatch.setattr(settings, "audit_interval_seconds", 60)
|
||||
|
||||
async def _dispatch() -> None:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=5.0, headers=_SYSTEM_API_HEADERS
|
||||
) as client:
|
||||
await orch._dispatch_audit_work(client)
|
||||
|
||||
asyncio.run(_dispatch())
|
||||
|
||||
orch.spawn_agent.assert_awaited_once()
|
||||
call = orch.spawn_agent.await_args
|
||||
assert call is not None
|
||||
assert call.kwargs["agent_id"] == "auditor"
|
||||
assert call.kwargs["spawned_by"] == "_dispatch_audit_work"
|
||||
prompt = call.kwargs["initial_prompt"]
|
||||
assert "SCHEDULED AUDIT SWEEP" in prompt
|
||||
|
||||
# No reactive alert should have been created for this path.
|
||||
assert _notifications_for_task(stack, task_id, NotificationType.ALERT) == []
|
||||
assert auditor_id is not None # auditor was seeded and resolved
|
||||
|
||||
|
||||
def test_reactive_alert_producer_spawns_auditor(
|
||||
e2e_stack: E2EStack, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""QA-fail emits an auditor-targeted ALERT; the dispatcher spawns the auditor."""
|
||||
stack = e2e_stack
|
||||
company = seed_company(stack)
|
||||
project_id, _project_slug = seed_project(stack, company)
|
||||
auditor_id = _seed_auditor_agent(stack)
|
||||
|
||||
task_id = seed_task(
|
||||
stack,
|
||||
title="Reactive alert smoke task",
|
||||
description="A task awaiting QA so fail_qa can emit an auditor alert.",
|
||||
acceptance_criteria=["the reactive path emits an auditor alert"],
|
||||
project_id=project_id,
|
||||
created_by=company.cell_pm_id,
|
||||
assigned_to=company.dev_id,
|
||||
claimed_by=company.dev_id,
|
||||
active_claimant_id=company.dev_id,
|
||||
status=TaskStatus.AWAITING_QA,
|
||||
branch_name="feature/backend/e2e-reactive-alert",
|
||||
)
|
||||
|
||||
resp = httpx.post(
|
||||
f"{stack.base_url}/api/tasks/{task_id}/fail-qa",
|
||||
json={"notes": "missing edge-case coverage"},
|
||||
headers=_agent_headers(company.qa_id, "qa"),
|
||||
timeout=30,
|
||||
)
|
||||
assert resp.status_code == HTTPStatus.OK, (
|
||||
f"fail-qa: {resp.status_code} {resp.text[:1500]}"
|
||||
)
|
||||
|
||||
alerts = _notifications_for_task(stack, task_id, NotificationType.ALERT)
|
||||
assert len(alerts) == 1, alerts
|
||||
alert = alerts[0]
|
||||
assert "rework alert" in alert["subject"].lower(), alert
|
||||
assert auditor_id in alert["to_agents"], alert
|
||||
|
||||
orch = _fresh_orchestrator(stack, monkeypatch)
|
||||
monkeypatch.setattr(settings, "audit_interval_seconds", 60)
|
||||
|
||||
async def _dispatch() -> None:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=5.0, headers=_SYSTEM_API_HEADERS
|
||||
) as client:
|
||||
await orch._dispatch_audit_work(client)
|
||||
|
||||
asyncio.run(_dispatch())
|
||||
|
||||
orch.spawn_agent.assert_awaited_once()
|
||||
call = orch.spawn_agent.await_args
|
||||
assert call is not None
|
||||
assert call.kwargs["agent_id"] == "auditor"
|
||||
assert call.kwargs["spawned_by"] == "_dispatch_audit_work"
|
||||
prompt = call.kwargs["initial_prompt"]
|
||||
assert "QUALITY ALERT" in prompt
|
||||
assert "missing edge-case coverage" in prompt
|
||||
@@ -19,6 +19,7 @@ from uuid import UUID, uuid4
|
||||
import pytest
|
||||
from roboco.config import settings
|
||||
from roboco.runtime.orchestrator import (
|
||||
_INTAKE_WORKSPACE_AMBIENT,
|
||||
INTAKE_AGENT_ID,
|
||||
AgentInstance,
|
||||
AgentOrchestrator,
|
||||
@@ -264,6 +265,58 @@ class TestIntakeScopeSlugs:
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _dedupe_slugs_by_git_url — a multi-project scope may list several projects
|
||||
# pointing at one repo (a monorepo's cell-projects); clone each git_url once.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestDedupeSlugsByGitUrl:
|
||||
def test_keeps_first_of_shared_git_url_drops_rest(self) -> None:
|
||||
result = AgentOrchestrator._dedupe_slugs_by_git_url(
|
||||
[
|
||||
("be", "https://git.example/o.git"),
|
||||
("fe", "https://git.example/o.git"),
|
||||
("ux", "https://git.example/o.git"),
|
||||
]
|
||||
)
|
||||
assert result == ["be"]
|
||||
|
||||
def test_distinct_git_urls_all_kept_in_order(self) -> None:
|
||||
result = AgentOrchestrator._dedupe_slugs_by_git_url(
|
||||
[
|
||||
("a", "https://git.example/a.git"),
|
||||
("b", "https://git.example/b.git"),
|
||||
]
|
||||
)
|
||||
assert result == ["a", "b"]
|
||||
|
||||
def test_empty_or_missing_git_url_never_collapsed(self) -> None:
|
||||
# Two projects with no git_url are distinct local repos — both clone.
|
||||
result = AgentOrchestrator._dedupe_slugs_by_git_url(
|
||||
[("a", None), ("b", ""), ("c", " ")]
|
||||
)
|
||||
assert result == ["a", "b", "c"]
|
||||
|
||||
def test_mixed_local_and_shared_repos(self) -> None:
|
||||
result = AgentOrchestrator._dedupe_slugs_by_git_url(
|
||||
[
|
||||
("mono-be", "https://git.example/mono.git"),
|
||||
("local-a", None),
|
||||
("mono-fe", "https://git.example/mono.git"),
|
||||
("local-b", ""),
|
||||
("mono-ux", " https://git.example/mono.git "),
|
||||
]
|
||||
)
|
||||
assert result == ["mono-be", "local-a", "local-b"]
|
||||
|
||||
def test_whitespace_only_git_url_treated_as_empty(self) -> None:
|
||||
result = AgentOrchestrator._dedupe_slugs_by_git_url(
|
||||
[("a", " "), ("b", " ")]
|
||||
)
|
||||
assert result == ["a", "b"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _resolve_history_digest_projects — the prompter-memory ambient's project scope
|
||||
# (covers all three intake scopes: project_slug, product_id, project_ids).
|
||||
@@ -639,7 +692,10 @@ class TestResolveIntakeAmbientThreadsProjectIds:
|
||||
project_ids=["11111111-1111-1111-1111-111111111111"],
|
||||
)
|
||||
|
||||
assert result == "CONVENTIONS\n\n---\n\nHISTORY"
|
||||
assert (
|
||||
result
|
||||
== f"{_INTAKE_WORKSPACE_AMBIENT}\n\n---\n\nCONVENTIONS\n\n---\n\nHISTORY"
|
||||
)
|
||||
assert conventions_calls == [
|
||||
{
|
||||
"product_id": None,
|
||||
@@ -784,8 +840,8 @@ class TestSpawnIntakeSession:
|
||||
async def test_spawn_merges_conventions_and_history_ambient(
|
||||
self, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The composed prompt's ambient is the conventions + history-digest
|
||||
blocks joined with compose_prompt's own layer separator."""
|
||||
"""The composed prompt's ambient is the workspace note + conventions +
|
||||
history-digest blocks joined with compose_prompt's own layer separator."""
|
||||
orch = _make_minimal_orchestrator()
|
||||
run_calls: list[list[str]] = []
|
||||
_wire_spawn_mocks(monkeypatch, orch, run_calls)
|
||||
@@ -809,12 +865,18 @@ class TestSpawnIntakeSession:
|
||||
|
||||
await orch.spawn_intake_session("sess-merge", project_slug="roboco")
|
||||
|
||||
assert captured["ambient"] == "CONVENTIONS BLOCK\n\n---\n\nHISTORY BLOCK"
|
||||
assert (
|
||||
captured["ambient"]
|
||||
== f"{_INTAKE_WORKSPACE_AMBIENT}\n\n---\n\nCONVENTIONS BLOCK"
|
||||
f"\n\n---\n\nHISTORY BLOCK"
|
||||
)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_spawn_ambient_none_when_both_resolvers_empty(
|
||||
async def test_spawn_ambient_is_workspace_block_when_both_resolvers_empty(
|
||||
self, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The workspace note is always present; with no conventions/history the
|
||||
ambient is just that block (never None)."""
|
||||
orch = _make_minimal_orchestrator()
|
||||
run_calls: list[list[str]] = []
|
||||
_wire_spawn_mocks(monkeypatch, orch, run_calls)
|
||||
@@ -835,7 +897,7 @@ class TestSpawnIntakeSession:
|
||||
|
||||
await orch.spawn_intake_session("sess-no-ambient", project_slug="roboco")
|
||||
|
||||
assert captured["ambient"] is None
|
||||
assert captured["ambient"] == _INTAKE_WORKSPACE_AMBIENT
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_initial_message_is_scheduled(
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
"""Scheduled audit trigger in _dispatch_audit_work.
|
||||
|
||||
Covers the interval cooldown (ROBOCO_AUDIT_INTERVAL_SECONDS) and the
|
||||
active-agent breaker that prevents auditor spawn storms.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
from roboco.config import Settings, settings
|
||||
from roboco.runtime.orchestrator import AgentOrchestrator
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def orch() -> AgentOrchestrator:
|
||||
orch = AgentOrchestrator.__new__(AgentOrchestrator)
|
||||
orch._notification_spawn_at = {}
|
||||
orch._last_audit_spawn_at = None
|
||||
return orch
|
||||
|
||||
|
||||
async def _run_dispatch(
|
||||
orch: AgentOrchestrator, *, tasks: list[dict] | None = None
|
||||
) -> MagicMock:
|
||||
"""Run _dispatch_audit_work with notifications empty and return the spawn mock."""
|
||||
client = MagicMock()
|
||||
with (
|
||||
patch.object(orch, "_fetch_notifications", new=AsyncMock(return_value=[])),
|
||||
patch.object(orch, "_fetch_tasks", new=AsyncMock(return_value=tasks or [])),
|
||||
patch.object(orch, "_is_agent_active", return_value=False),
|
||||
patch.object(orch, "spawn_agent", new=AsyncMock()) as spawn_mock,
|
||||
):
|
||||
await orch._dispatch_audit_work(client)
|
||||
return spawn_mock
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_spawns_when_overdue_and_delivery_activity(
|
||||
orch: AgentOrchestrator,
|
||||
) -> None:
|
||||
"""Interval elapsed + recent tasks -> auditor sweep is spawned."""
|
||||
now = datetime.now(UTC)
|
||||
with (
|
||||
patch.object(settings, "audit_interval_seconds", 21600),
|
||||
patch("roboco.runtime.orchestrator.datetime", wraps=datetime) as dt_mock,
|
||||
):
|
||||
dt_mock.now.return_value = now
|
||||
orch._last_audit_spawn_at = now - timedelta(seconds=21601)
|
||||
spawn_mock = await _run_dispatch(
|
||||
orch,
|
||||
tasks=[
|
||||
{
|
||||
"id": "t1",
|
||||
"status": "in_progress",
|
||||
"updated_at": "2026-07-13T00:00:00Z",
|
||||
}
|
||||
],
|
||||
)
|
||||
assert spawn_mock.await_count == 1
|
||||
call_kwargs = spawn_mock.await_args.kwargs if spawn_mock.await_args else {}
|
||||
assert call_kwargs.get("agent_id") == "auditor"
|
||||
assert call_kwargs.get("spawned_by") == "_dispatch_audit_work"
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_skips_when_interval_not_elapsed(orch: AgentOrchestrator) -> None:
|
||||
"""Cooldown: last spawn was recent -> no scheduled sweep."""
|
||||
now = datetime.now(UTC)
|
||||
with (
|
||||
patch.object(settings, "audit_interval_seconds", 21600),
|
||||
patch("roboco.runtime.orchestrator.datetime", wraps=datetime) as dt_mock,
|
||||
):
|
||||
dt_mock.now.return_value = now
|
||||
orch._last_audit_spawn_at = now - timedelta(seconds=1800)
|
||||
spawn_mock = await _run_dispatch(
|
||||
orch,
|
||||
tasks=[
|
||||
{
|
||||
"id": "t1",
|
||||
"status": "in_progress",
|
||||
"updated_at": "2026-07-13T00:00:00Z",
|
||||
}
|
||||
],
|
||||
)
|
||||
assert spawn_mock.await_count == 0
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_skips_when_no_delivery_activity(orch: AgentOrchestrator) -> None:
|
||||
"""No active tasks and no recently completed tasks -> no sweep."""
|
||||
now = datetime.now(UTC)
|
||||
with (
|
||||
patch.object(settings, "audit_interval_seconds", 21600),
|
||||
patch("roboco.runtime.orchestrator.datetime", wraps=datetime) as dt_mock,
|
||||
):
|
||||
dt_mock.now.return_value = now
|
||||
orch._last_audit_spawn_at = now - timedelta(seconds=21601)
|
||||
spawn_mock = await _run_dispatch(orch, tasks=[])
|
||||
assert spawn_mock.await_count == 0
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_breaker_skips_when_auditor_active(orch: AgentOrchestrator) -> None:
|
||||
"""Active-agent breaker prevents a second auditor container."""
|
||||
client = MagicMock()
|
||||
now = datetime.now(UTC)
|
||||
with (
|
||||
patch.object(settings, "audit_interval_seconds", 21600),
|
||||
patch("roboco.runtime.orchestrator.datetime", wraps=datetime) as dt_mock,
|
||||
patch.object(orch, "_fetch_notifications", new=AsyncMock(return_value=[])),
|
||||
patch.object(orch, "_fetch_tasks", new=AsyncMock(return_value=[])),
|
||||
patch.object(orch, "_is_agent_active", return_value=True),
|
||||
patch.object(orch, "spawn_agent", new=AsyncMock()) as spawn_mock,
|
||||
):
|
||||
dt_mock.now.return_value = now
|
||||
orch._last_audit_spawn_at = now - timedelta(seconds=21601)
|
||||
await orch._dispatch_audit_work(client)
|
||||
assert spawn_mock.await_count == 0
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_reactive_alert_stamps_last_spawn_and_blocks_scheduled(
|
||||
orch: AgentOrchestrator,
|
||||
) -> None:
|
||||
"""A reactive alert spawn records _last_audit_spawn_at and returns early."""
|
||||
client = MagicMock()
|
||||
alert = {
|
||||
"id": "a1",
|
||||
"to_agents": ["auditor"],
|
||||
"subject": "Coverage gap",
|
||||
"body": "Test",
|
||||
}
|
||||
now = datetime.now(UTC)
|
||||
with (
|
||||
patch.object(settings, "audit_interval_seconds", 21600),
|
||||
patch("roboco.runtime.orchestrator.datetime", wraps=datetime) as dt_mock,
|
||||
patch.object(orch, "_fetch_notifications", new=AsyncMock(return_value=[alert])),
|
||||
patch.object(orch, "_fetch_tasks", new=AsyncMock(return_value=[])),
|
||||
patch.object(orch, "_is_agent_active", return_value=False),
|
||||
patch.object(orch, "spawn_agent", new=AsyncMock()) as spawn_mock,
|
||||
):
|
||||
dt_mock.now.return_value = now
|
||||
await orch._dispatch_audit_work(client)
|
||||
assert spawn_mock.await_count == 1
|
||||
assert orch._last_audit_spawn_at == now
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_cooldown_zero_disables_scheduled_sweeps(orch: AgentOrchestrator) -> None:
|
||||
"""ROBOCO_AUDIT_INTERVAL_SECONDS=0 disables scheduled sweeps entirely."""
|
||||
now = datetime.now(UTC)
|
||||
with (
|
||||
patch.object(settings, "audit_interval_seconds", 0),
|
||||
patch("roboco.runtime.orchestrator.datetime", wraps=datetime) as dt_mock,
|
||||
):
|
||||
dt_mock.now.return_value = now
|
||||
orch._last_audit_spawn_at = None
|
||||
spawn_mock = await _run_dispatch(
|
||||
orch,
|
||||
tasks=[
|
||||
{
|
||||
"id": "t1",
|
||||
"status": "in_progress",
|
||||
"updated_at": "2026-07-13T00:00:00Z",
|
||||
}
|
||||
],
|
||||
)
|
||||
assert spawn_mock.await_count == 0
|
||||
|
||||
|
||||
def test_env_override_zero_is_accepted() -> None:
|
||||
"""ROBOCO_AUDIT_INTERVAL_SECONDS=0 is valid Pydantic input, matching the
|
||||
documented disable sentinel and the runtime gate in _audit_spawn_cooled().
|
||||
"""
|
||||
with patch.dict(os.environ, {"ROBOCO_AUDIT_INTERVAL_SECONDS": "0"}, clear=False):
|
||||
s = Settings()
|
||||
assert s.audit_interval_seconds == 0
|
||||
@@ -0,0 +1,303 @@
|
||||
"""Unit tests for auditor-targeted rework alert producers.
|
||||
|
||||
Covers the reactive audit dispatch path: ``NotificationDeliveryService``
|
||||
creates ``ALERT`` notifications addressed to the auditor, and ``TaskService``
|
||||
invokes that producer at the QA-fail / rework chokepoints
|
||||
(``fail_qa``, ``pr_fail``, ``request_changes``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
import pytest
|
||||
from roboco.models.base import (
|
||||
NotificationPriority,
|
||||
NotificationType,
|
||||
TaskStatus,
|
||||
)
|
||||
from roboco.services.notification_delivery import NotificationDeliveryService
|
||||
from roboco.services.task import TaskService
|
||||
from roboco.utils.converters import require_uuid
|
||||
|
||||
|
||||
def _mock_task(
|
||||
task_id: UUID | None = None, status: Any = TaskStatus.AWAITING_QA
|
||||
) -> MagicMock:
|
||||
task = MagicMock()
|
||||
task.id = task_id or uuid4()
|
||||
task.title = "Test task"
|
||||
task.status = status
|
||||
task.assigned_to = uuid4()
|
||||
task.claimed_by = None
|
||||
task.active_claimant_id = None
|
||||
task.team = MagicMock()
|
||||
task.team.value = "backend"
|
||||
task.qa_verified = True
|
||||
task.qa_notes = None
|
||||
task.dev_notes = None
|
||||
task.orchestration_markers = {}
|
||||
task.notes_structured = None
|
||||
task.revision_count = 0
|
||||
return task
|
||||
|
||||
|
||||
def _mock_agent(*, role: str = "auditor", slug: str = "auditor") -> MagicMock:
|
||||
agent = MagicMock()
|
||||
agent.id = uuid4()
|
||||
agent.role = role
|
||||
agent.slug = slug
|
||||
return agent
|
||||
|
||||
|
||||
def _session_with_agent(agent: MagicMock | None) -> MagicMock:
|
||||
"""A session whose execute().scalars().first() returns ``agent``.
|
||||
|
||||
``session.add`` assigns a fresh UUID to the notification so that
|
||||
``_persist_and_deliver`` can call ``require_uuid(notification.id)``.
|
||||
"""
|
||||
session = MagicMock()
|
||||
|
||||
def _assign_id(obj: Any) -> None:
|
||||
if getattr(obj, "id", None) is None:
|
||||
obj.id = uuid4()
|
||||
|
||||
session.add = MagicMock(side_effect=_assign_id)
|
||||
session.flush = AsyncMock()
|
||||
result = MagicMock()
|
||||
result.scalars.return_value.first.return_value = agent
|
||||
result.scalar_one_or_none.return_value = agent
|
||||
session.execute = AsyncMock(return_value=result)
|
||||
return session
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# NotificationDeliveryService.notify_auditor_of_rework
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_notify_auditor_of_rework_creates_alert_to_auditor(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The delivery service builds an ALERT notification targeted at the auditor."""
|
||||
auditor = _mock_agent(role="auditor", slug="auditor")
|
||||
actor = _mock_agent(role="qa", slug="be-qa")
|
||||
session = _session_with_agent(auditor)
|
||||
svc = NotificationDeliveryService(session)
|
||||
monkeypatch.setattr(svc, "_get_auditor_agent", AsyncMock(return_value=auditor))
|
||||
monkeypatch.setattr(svc, "_get_agent_by_id", AsyncMock(return_value=actor))
|
||||
monkeypatch.setattr(svc, "deliver", AsyncMock(return_value=True))
|
||||
|
||||
task = _mock_task()
|
||||
with patch(
|
||||
"roboco.services.notification_delivery.all_recipients_recently_notified",
|
||||
AsyncMock(return_value=False),
|
||||
):
|
||||
await svc.notify_auditor_of_rework(
|
||||
task=task,
|
||||
task_id=require_uuid(task.id),
|
||||
reason="QA review failed",
|
||||
actor_agent_id=actor.id,
|
||||
actor_role="qa",
|
||||
)
|
||||
|
||||
added = [c for c in session.add.call_args_list if c.args]
|
||||
assert len(added) == 1
|
||||
notification = added[0].args[0]
|
||||
assert notification.type == NotificationType.ALERT
|
||||
assert notification.priority == NotificationPriority.HIGH
|
||||
assert notification.to_agents == [auditor.id]
|
||||
assert notification.from_agent == actor.id
|
||||
assert "QA review failed" in notification.body
|
||||
assert "Actor role: qa" in notification.body
|
||||
assert notification.requires_ack is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_notify_auditor_of_rework_skips_when_no_auditor(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""If the auditor agent is absent, the producer is a silent no-op."""
|
||||
session = _session_with_agent(None)
|
||||
svc = NotificationDeliveryService(session)
|
||||
monkeypatch.setattr(svc, "_get_auditor_agent", AsyncMock(return_value=None))
|
||||
deliver_spy = AsyncMock()
|
||||
monkeypatch.setattr(svc, "deliver", deliver_spy)
|
||||
|
||||
task = _mock_task()
|
||||
await svc.notify_auditor_of_rework(
|
||||
task=task,
|
||||
task_id=require_uuid(task.id),
|
||||
reason="QA review failed",
|
||||
)
|
||||
|
||||
assert not session.add.called
|
||||
assert not deliver_spy.called
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TaskService._alert_auditor_of_rework
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_alert_auditor_of_rework_calls_delivery_service() -> None:
|
||||
"""TaskService delegates to the notification delivery service."""
|
||||
session = MagicMock()
|
||||
session.flush = AsyncMock()
|
||||
task = _mock_task()
|
||||
svc = TaskService(session)
|
||||
|
||||
fake_delivery = AsyncMock()
|
||||
fake_delivery.notify_auditor_of_rework = AsyncMock()
|
||||
|
||||
with patch(
|
||||
"roboco.services.notification_delivery.get_notification_delivery_service",
|
||||
lambda _s: fake_delivery,
|
||||
):
|
||||
await svc._alert_auditor_of_rework(
|
||||
task,
|
||||
reason="QA review failed",
|
||||
actor_agent_id=task.assigned_to,
|
||||
actor_role="qa",
|
||||
)
|
||||
|
||||
fake_delivery.notify_auditor_of_rework.assert_awaited_once()
|
||||
call = fake_delivery.notify_auditor_of_rework.await_args
|
||||
assert call.kwargs["task"] is task
|
||||
assert call.kwargs["reason"] == "QA review failed"
|
||||
assert call.kwargs["actor_role"] == "qa"
|
||||
assert call.kwargs["actor_agent_id"] == task.assigned_to
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_alert_auditor_of_rework_is_best_effort() -> None:
|
||||
"""A delivery failure must not raise out of the producer."""
|
||||
session = MagicMock()
|
||||
task = _mock_task()
|
||||
svc = TaskService(session)
|
||||
|
||||
with patch(
|
||||
"roboco.services.notification_delivery.get_notification_delivery_service",
|
||||
side_effect=RuntimeError("redis down"),
|
||||
):
|
||||
await svc._alert_auditor_of_rework(
|
||||
task, reason="QA review failed", actor_role="qa"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Chokepoint wiring: fail_qa, pr_fail, request_changes
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fail_qa_emits_auditor_alert(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""``fail_qa`` calls the auditor rework producer with QA attribution."""
|
||||
session = MagicMock()
|
||||
session.flush = AsyncMock()
|
||||
task = _mock_task(status=TaskStatus.AWAITING_QA)
|
||||
task.orchestration_markers = {"original_developer": str(uuid4())}
|
||||
|
||||
svc = TaskService(session)
|
||||
monkeypatch.setattr(svc, "get", AsyncMock(return_value=task))
|
||||
monkeypatch.setattr(svc, "_validate_and_set_status", MagicMock())
|
||||
alert_spy = AsyncMock()
|
||||
monkeypatch.setattr(svc, "_alert_auditor_of_rework", alert_spy)
|
||||
|
||||
with (
|
||||
patch(
|
||||
"roboco.services.task.extract_original_developer",
|
||||
return_value=str(uuid4()),
|
||||
),
|
||||
patch("roboco.services.task.asyncio.create_task", MagicMock()),
|
||||
):
|
||||
out = await svc.fail_qa(task.id, notes="missing tests")
|
||||
|
||||
assert out is task
|
||||
alert_spy.assert_awaited_once()
|
||||
call = alert_spy.await_args
|
||||
assert call is not None
|
||||
assert call.args[0] is task
|
||||
assert call.kwargs["reason"] == "missing tests"
|
||||
assert call.kwargs["actor_role"] == "qa"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pr_fail_emits_auditor_alert(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""``pr_fail`` calls the auditor rework producer with reviewer attribution."""
|
||||
session = MagicMock()
|
||||
session.flush = AsyncMock()
|
||||
reviewer_id = uuid4()
|
||||
pm_id = uuid4()
|
||||
task = _mock_task(status=TaskStatus.AWAITING_PR_REVIEW)
|
||||
task.claimed_by = reviewer_id
|
||||
|
||||
pm = MagicMock()
|
||||
pm.id = pm_id
|
||||
|
||||
svc = TaskService(session)
|
||||
monkeypatch.setattr(svc, "get", AsyncMock(return_value=task))
|
||||
monkeypatch.setattr(svc, "_validate_and_set_status", MagicMock())
|
||||
monkeypatch.setattr(svc, "_revision_pm_for_task", AsyncMock(return_value=pm))
|
||||
alert_spy = AsyncMock()
|
||||
monkeypatch.setattr(svc, "_alert_auditor_of_rework", alert_spy)
|
||||
|
||||
out = await svc.pr_fail(
|
||||
reviewer_id, task.id, notes="convention violation", issues=["mv model"]
|
||||
)
|
||||
|
||||
assert out is task
|
||||
alert_spy.assert_awaited_once()
|
||||
call = alert_spy.await_args
|
||||
assert call is not None
|
||||
assert call.args[0] is task
|
||||
assert call.kwargs["reason"] == "convention violation"
|
||||
assert call.kwargs["actor_role"] == "pr_reviewer"
|
||||
assert call.kwargs["actor_agent_id"] == reviewer_id
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_request_changes_emits_auditor_alert(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""``request_changes`` calls the auditor rework producer with PM attribution."""
|
||||
session = MagicMock()
|
||||
session.flush = AsyncMock()
|
||||
pm_id = uuid4()
|
||||
pm = MagicMock()
|
||||
pm.id = pm_id
|
||||
task = _mock_task(status=TaskStatus.AWAITING_PM_REVIEW)
|
||||
task.claimed_by = pm_id
|
||||
|
||||
svc = TaskService(session)
|
||||
monkeypatch.setattr(svc, "get", AsyncMock(return_value=task))
|
||||
monkeypatch.setattr(svc, "_validate_and_set_status", MagicMock())
|
||||
monkeypatch.setattr(svc, "_revision_pm_for_task", AsyncMock(return_value=pm))
|
||||
alert_spy = AsyncMock()
|
||||
monkeypatch.setattr(svc, "_alert_auditor_of_rework", alert_spy)
|
||||
|
||||
with patch("roboco.services.task.extract_original_developer", return_value=None):
|
||||
out = await svc.request_changes(
|
||||
pm_id,
|
||||
task.id,
|
||||
notes="AC missing",
|
||||
issues=["add test"],
|
||||
agent_role="cell_pm",
|
||||
)
|
||||
|
||||
assert out is task
|
||||
alert_spy.assert_awaited_once()
|
||||
call = alert_spy.await_args
|
||||
assert call is not None
|
||||
assert call.args[0] is task
|
||||
assert call.kwargs["reason"] == "AC missing"
|
||||
assert call.kwargs["actor_role"] == "cell_pm"
|
||||
assert call.kwargs["actor_agent_id"] == pm_id
|
||||
Reference in New Issue
Block a user