mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
* feat(observability): revision_count + audit_log query index (migration 045)
Adds tasks.revision_count (the O(1) rework counter — forward-only, existing
rows default 0) and the composite index audit_log(target_id, event_type,
timestamp) that powers the cycle-time and rework reconstruction queries.
Verified the real upgrade/downgrade/upgrade chain on a throwaway pgvector PG.
First task of the 0.10.0 observability dashboards.
* feat(observability): count reworks + attribute qa_fail/pr_fail to the rejector
Every transition into needs_revision increments tasks.revision_count at the
single audit chokepoint (exactly once per bounce, across all paths incl. pr_fail
and ceo_reject), so the rework rate is an O(1) read. A QA or PR-review bounce
also emits a named task.qa_fail / task.pr_fail audit event carrying the
rejector's agent_id, so the per-agent rework scorecard charges the rejection to
the reviewer who made it, not the developer who owns the task.
* feat(observability): cycle-time, bottleneck, rework, and scorecard metrics
MetricsService gains four read methods on the audit_log + tasks data: per-stage
cycle time reconstructed from the transition journey (excluding the named
qa_fail/pr_fail events), bottleneck distribution (cumulative dwell + live parked
counts), rework rate (overall/by-team/by-agent with rejector attribution + cost
via spawn-session task_id), and a fused per-agent/per-cell scorecard. Dataclass
models with to_dict(). Verified against a real Postgres journey.
* feat(observability): cycle-time/bottleneck/rework/scorecard read endpoints
Thin read-only routes on the dashboard router delegating to MetricsService:
/metrics/cycle-time, /metrics/bottlenecks, /metrics/rework, and
/metrics/scorecard/{agent,team}. 404 when an agent scorecard target is absent.
5 route tests (200 + shape + the agent-404 case).
* feat(panel): Delivery observability tab (cycle-time, bottlenecks, rework, scorecards)
A third Metrics tab built on the observability endpoints: a per-stage
cycle-time bar chart, a bottleneck panel (worst stage + cumulative dwell +
live parked counts), a rework panel (rate + by-team + by-agent attribution +
cost), and per-cell scorecards. Reuses Recharts + Card/Badge/Skeleton and the
React-Query hook pattern; observabilityApi mirrors usageApi with mock-mode
fallbacks. tsc + eslint clean; 113 panel tests pass.
* docs(observability): changelog + CLAUDE.md for the delivery dashboards
* feat(gateway-health): recover a broken-but-alive agent instead of protecting it
The verb-heartbeat cannot tell a quiet-healthy agent from one whose MCP gateway
is broken (a corrupted /app/.venv firing no verb) yet whose container is up — the
reaper's live-skip would shield it forever. The reaper now probes the gateway
out-of-band (docker exec: does the gateway venv import its deps?) and, once it
has been broken past gateway_health_grace_seconds (tolerating a transient probe
miss), kills + evicts the container so it falls through to release + respawn.
Probe-inconclusive or healthy spares the container. Gated by
gateway_health_enabled (default-on reliability fix; in the panel Feature Flags).
Defers the optional agent-side self-check + full registry re-adoption — the
reaper's docker-liveness fallback already recovers a broken-after-restart agent.
* docs(gateway-health): changelog + CLAUDE.md for broken-but-alive recovery
* docs(observability): user-facing docs for the Delivery dashboards + gateway-health
Documents the new Metrics -> Delivery tab (cycle-time, bottlenecks, rework with
rejector attribution, cell scorecards) in the panel guide and the operations
health-and-metrics guide, and adds the gateway-health env vars + an agent-gateway
recovery note. Published MkDocs site only; settings.md's default-off flag table
intentionally omits the default-on gateway-health flag (same as overload-break).
* chore(release): cut 0.10.0 (changelog section + version refs)
* fix(gateway): exempt PM coordinators from single-task claim guards
A Main/Cell PM plans and delegates many root tasks in parallel; the work
then runs in the delegated cells, not in the PM's own hands. But the
claim-time concurrency guards meant for developers — already_active and
paused (the latter firing after i_am_idle auto-pauses the PM's own
umbrella) — were applied to the PM too, so once it held one root it could
never plan a second: it thrashed between its claimed roots and respawned
forever, burning tokens for zero progress.
_run_claim_guards now skips already_active/paused for the coordinator PM
roles (_COORDINATOR_ROLES = {main_pm, cell_pm}); only unmet_dependency — a
real upstream sequence constraint, which parks the root back to pending —
still gates a PM. paused_tasks_guard also excludes the target task itself,
so a PM re-entering its own paused umbrella never self-blocks.
Tests: a coordinator plans a second root with one in_progress + one paused
sibling (full path + claimed-recovery path), the paused target exclusion,
and the developer guards still fire. Repurposed the pre-fix test that
asserted the now-removed PM block.
* fix(metrics): coerce SQL avg/extract hours aggregates to float (panel toFixed crash)
EXTRACT(epoch ...) returns numeric on PostgreSQL 14+, which asyncpg surfaces
as a Decimal; a Decimal serializes to a quoted JSON string, so the panel's
avg_cycle_hours.toFixed(1) (and the other hours fields) threw 'toFixed is not
a function' and blanked the Delivery tab.
A single _as_hours helper now rounds every SQL-averaged hours field to a real
float — avg_cycle_hours on the new scorecards plus the pre-existing
avg_completion_hours / avg_blocked_hours / longest_blocked_hours. Token and
cost fields were already float()-cast and are unaffected.
Regression test asserts _as_hours coerces Decimal -> float and preserves the
None/zero behavior.
* feat(panel): edit a task's sequence from the details page
A task's sequence (order within siblings, lower runs first) was display-only
with no way to change it from the UI, and TaskUpdate didn't carry the field
so PATCH couldn't set it either. The details page's Dependencies tab now has
an inline sequence editor mirroring the parent / dependency editors, and
PATCH /tasks/{id} accepts a sequence field (owner or privileged role) through
the existing generic update path.
* fix(mypy): green the full make-quality type gate
make quality runs 'mypy roboco/ tests/', which the per-module checks on the
0.10.0 branch never exercised. Two issues surfaced:
- The coordinator-exemption change added role_str to
Choreographer._run_claim_guards but not to the ChoreographerHelpers
protocol base, so the composed Choreographer had incompatible base-class
signatures. Sync the protocol signature.
- The gateway-health / stale-reaper tests stubbed methods by direct
assignment (orch._m = AsyncMock()) and typed their duck-typed task doubles
as object, tripping method-assign / assignment / attr-defined. Switch to
monkeypatch.setattr (keeping a local mock ref for the assertions) and type
the doubles as Any — no type: ignore.
Full mypy roboco/ tests/ clean (785 files); the 21 runtime tests pass.
* fix(metrics): static cycle-time SQL — clear bandit B608 (CI gate)
The cycle-time query interpolated an optional team clause into the text() SQL
via an f-string, which bandit flags as B608 (hardcoded SQL) and turned the
merge gate red. The team value was always a bound parameter, so it was a false
positive — but the f-string is the trigger. Rebuilt as one static query with
(CAST(:team AS text) IS NULL OR a.details->>'team' = :team) and an always-bound
team param (CAST, not ::text — SQLAlchemy's :param parser collides with
PostgreSQL's :: cast operator, which broke the query as a stray param).
Full make quality green vs a real pgvector PG (all 21 gate steps).
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
411 lines
13 KiB
Python
411 lines
13 KiB
Python
"""Dashboard API route coverage."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import uuid
|
|
from http import HTTPStatus
|
|
from typing import TYPE_CHECKING, cast
|
|
from uuid import uuid4
|
|
|
|
import pytest
|
|
import pytest_asyncio
|
|
from fastapi import FastAPI
|
|
from httpx import ASGITransport, AsyncClient
|
|
from roboco.api.deps import get_agent_context, get_db
|
|
from roboco.api.routes.dashboard import get_main_pm_kanban
|
|
from roboco.api.routes.dashboard import router as dashboard_router
|
|
from roboco.db.tables import AgentTable
|
|
from roboco.models import AgentRole, AgentStatus
|
|
from roboco.models.permissions import AgentContext
|
|
from roboco.services.dashboard import reset_storage
|
|
|
|
if TYPE_CHECKING:
|
|
from collections.abc import AsyncGenerator, AsyncIterator
|
|
|
|
from sqlalchemy.ext.asyncio import AsyncSession
|
|
|
|
|
|
@pytest_asyncio.fixture
|
|
async def dashboard_client(
|
|
db_session: AsyncSession,
|
|
) -> AsyncIterator[AsyncClient]:
|
|
reset_storage()
|
|
agent = AgentTable(
|
|
id=uuid4(),
|
|
name="CEO",
|
|
slug=f"ceo-{uuid4().hex[:8]}",
|
|
role=AgentRole.CEO,
|
|
team=None,
|
|
status=AgentStatus.ACTIVE,
|
|
model_config={},
|
|
system_prompt="ceo",
|
|
capabilities=[],
|
|
permissions={},
|
|
metrics={},
|
|
)
|
|
db_session.add(agent)
|
|
await db_session.flush()
|
|
|
|
app = FastAPI()
|
|
app.include_router(dashboard_router, prefix="/api/dashboard")
|
|
|
|
async def _override_db() -> AsyncGenerator[AsyncSession]:
|
|
yield db_session
|
|
|
|
async def _override_agent() -> AgentContext:
|
|
return AgentContext(
|
|
agent_id=cast("uuid.UUID", agent.id), role=AgentRole.CEO, team=None
|
|
)
|
|
|
|
app.dependency_overrides[get_db] = _override_db
|
|
app.dependency_overrides[get_agent_context] = _override_agent
|
|
|
|
transport = ASGITransport(app=app)
|
|
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
|
yield client
|
|
app.dependency_overrides.clear()
|
|
|
|
|
|
_HDR = {"X-Agent-ID": str(uuid4()), "X-Agent-Role": "ceo"}
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_create_auditor_flag(dashboard_client: AsyncClient) -> None:
|
|
response = await dashboard_client.post(
|
|
"/api/dashboard/auditor/flags",
|
|
json={
|
|
"severity": "urgent",
|
|
"category": "quality",
|
|
"title": "Bug found",
|
|
"description": "Critical issue",
|
|
},
|
|
headers=_HDR,
|
|
)
|
|
assert response.status_code == HTTPStatus.CREATED
|
|
body = response.json()
|
|
assert body["severity"] == "urgent"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_auditor_flags(dashboard_client: AsyncClient) -> None:
|
|
response = await dashboard_client.get("/api/dashboard/auditor/flags", headers=_HDR)
|
|
assert response.status_code == HTTPStatus.OK
|
|
assert isinstance(response.json(), list)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Observability endpoints (0.10.0)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_cycle_time_endpoint(dashboard_client: AsyncClient) -> None:
|
|
resp = await dashboard_client.get(
|
|
"/api/dashboard/metrics/cycle-time?days=30", headers=_HDR
|
|
)
|
|
assert resp.status_code == HTTPStatus.OK
|
|
assert isinstance(resp.json(), list)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_bottlenecks_endpoint(dashboard_client: AsyncClient) -> None:
|
|
resp = await dashboard_client.get(
|
|
"/api/dashboard/metrics/bottlenecks", headers=_HDR
|
|
)
|
|
assert resp.status_code == HTTPStatus.OK
|
|
body = resp.json()
|
|
assert "by_stage" in body and "worst_stage" in body and "active_blockers" in body
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_rework_endpoint(dashboard_client: AsyncClient) -> None:
|
|
resp = await dashboard_client.get("/api/dashboard/metrics/rework", headers=_HDR)
|
|
assert resp.status_code == HTTPStatus.OK
|
|
body = resp.json()
|
|
assert "rate" in body and "by_team" in body and "by_agent" in body
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_agent_scorecard_404_when_absent(dashboard_client: AsyncClient) -> None:
|
|
resp = await dashboard_client.get(
|
|
f"/api/dashboard/metrics/scorecard/agent/{uuid4()}", headers=_HDR
|
|
)
|
|
assert resp.status_code == HTTPStatus.NOT_FOUND
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_team_scorecard_endpoint(dashboard_client: AsyncClient) -> None:
|
|
resp = await dashboard_client.get(
|
|
"/api/dashboard/metrics/scorecard/team/backend", headers=_HDR
|
|
)
|
|
assert resp.status_code == HTTPStatus.OK
|
|
assert resp.json()["scope"] == "cell"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_resolve_auditor_flag(dashboard_client: AsyncClient) -> None:
|
|
create = await dashboard_client.post(
|
|
"/api/dashboard/auditor/flags",
|
|
json={
|
|
"severity": "warning",
|
|
"category": "quality",
|
|
"title": "Warning",
|
|
"description": "x",
|
|
},
|
|
headers=_HDR,
|
|
)
|
|
flag_id = create.json()["id"]
|
|
response = await dashboard_client.put(
|
|
f"/api/dashboard/auditor/flags/{flag_id}/resolve",
|
|
params={"notes": "fixed"},
|
|
headers=_HDR,
|
|
)
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_resolve_unknown_flag_returns_404(
|
|
dashboard_client: AsyncClient,
|
|
) -> None:
|
|
response = await dashboard_client.put(
|
|
f"/api/dashboard/auditor/flags/{uuid4()}/resolve", headers=_HDR
|
|
)
|
|
assert response.status_code == HTTPStatus.NOT_FOUND
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_create_auditor_report(dashboard_client: AsyncClient) -> None:
|
|
response = await dashboard_client.post(
|
|
"/api/dashboard/auditor/reports",
|
|
json={
|
|
"report_type": "weekly",
|
|
"title": "Q1 Report",
|
|
"summary": "Strong week",
|
|
"sections": [],
|
|
},
|
|
headers=_HDR,
|
|
)
|
|
assert response.status_code == HTTPStatus.CREATED
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_auditor_reports(dashboard_client: AsyncClient) -> None:
|
|
response = await dashboard_client.get(
|
|
"/api/dashboard/auditor/reports", headers=_HDR
|
|
)
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_kanban_for_team_known_bug(
|
|
dashboard_client: AsyncClient,
|
|
) -> None:
|
|
"""Pre-existing bug — board.team is already a string (not enum) at line 334.
|
|
|
|
The route does `team.value` on a value already coerced to a string,
|
|
raising AttributeError. We assert the bug exists so a fix flips the test.
|
|
"""
|
|
with pytest.raises(AttributeError, match="'str' object has no attribute 'value'"):
|
|
await dashboard_client.get("/api/dashboard/kanban/backend", headers=_HDR)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_all_agent_status(dashboard_client: AsyncClient) -> None:
|
|
response = await dashboard_client.get("/api/dashboard/agents/status", headers=_HDR)
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_recent_activity(dashboard_client: AsyncClient) -> None:
|
|
response = await dashboard_client.get(
|
|
"/api/dashboard/activity/recent",
|
|
headers=_HDR,
|
|
)
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_auditor_dashboard(dashboard_client: AsyncClient) -> None:
|
|
response = await dashboard_client.get("/api/dashboard/auditor", headers=_HDR)
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_send_auditor_report_not_found(
|
|
dashboard_client: AsyncClient,
|
|
) -> None:
|
|
response = await dashboard_client.post(
|
|
f"/api/dashboard/auditor/reports/{uuid4()}/send", headers=_HDR
|
|
)
|
|
assert response.status_code == HTTPStatus.NOT_FOUND
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_send_auditor_report_success(
|
|
dashboard_client: AsyncClient,
|
|
) -> None:
|
|
create = await dashboard_client.post(
|
|
"/api/dashboard/auditor/reports",
|
|
json={
|
|
"report_type": "weekly",
|
|
"title": "T",
|
|
"summary": "s",
|
|
"sections": [],
|
|
},
|
|
headers=_HDR,
|
|
)
|
|
rid = create.json()["id"]
|
|
response = await dashboard_client.post(
|
|
f"/api/dashboard/auditor/reports/{rid}/send", headers=_HDR
|
|
)
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_ceo_overview(dashboard_client: AsyncClient) -> None:
|
|
response = await dashboard_client.get("/api/dashboard/ceo", headers=_HDR)
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_ceo_team_details(dashboard_client: AsyncClient) -> None:
|
|
response = await dashboard_client.get("/api/dashboard/ceo/teams", headers=_HDR)
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_ceo_blocker_details(dashboard_client: AsyncClient) -> None:
|
|
response = await dashboard_client.get("/api/dashboard/ceo/blockers", headers=_HDR)
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_ceo_velocity(dashboard_client: AsyncClient) -> None:
|
|
response = await dashboard_client.get(
|
|
"/api/dashboard/ceo/velocity?days=14", headers=_HDR
|
|
)
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_main_pm_kanban_via_http(dashboard_client: AsyncClient) -> None:
|
|
"""`/kanban/main-pm` is now declared before `/kanban/{team}`, so it routes
|
|
correctly to `get_main_pm_kanban` instead of being matched as
|
|
`team=main-pm` (which would 422)."""
|
|
response = await dashboard_client.get("/api/dashboard/kanban/main-pm", headers=_HDR)
|
|
assert response.status_code == HTTPStatus.OK
|
|
body = response.json()
|
|
# main_pm board has columns; shape is from KanbanBoard.model_dump().
|
|
assert "columns" in body
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_velocity_metrics(dashboard_client: AsyncClient) -> None:
|
|
response = await dashboard_client.get(
|
|
"/api/dashboard/metrics/velocity?days=7", headers=_HDR
|
|
)
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_blocker_metrics(dashboard_client: AsyncClient) -> None:
|
|
response = await dashboard_client.get(
|
|
"/api/dashboard/metrics/blockers", headers=_HDR
|
|
)
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_team_metrics(dashboard_client: AsyncClient) -> None:
|
|
response = await dashboard_client.get(
|
|
"/api/dashboard/metrics/team/backend", headers=_HDR
|
|
)
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_communication_metrics(
|
|
dashboard_client: AsyncClient,
|
|
) -> None:
|
|
response = await dashboard_client.get(
|
|
"/api/dashboard/metrics/communication", headers=_HDR
|
|
)
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_health_metrics(dashboard_client: AsyncClient) -> None:
|
|
response = await dashboard_client.get("/api/dashboard/metrics/health", headers=_HDR)
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_agent_metrics_not_found(
|
|
dashboard_client: AsyncClient,
|
|
) -> None:
|
|
response = await dashboard_client.get(
|
|
f"/api/dashboard/metrics/agent/{uuid4()}", headers=_HDR
|
|
)
|
|
assert response.status_code == HTTPStatus.NOT_FOUND
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_auditor_flags_filter_severity(
|
|
dashboard_client: AsyncClient,
|
|
) -> None:
|
|
response = await dashboard_client.get(
|
|
"/api/dashboard/auditor/flags?severity=warning", headers=_HDR
|
|
)
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_auditor_reports_with_filter(
|
|
dashboard_client: AsyncClient,
|
|
) -> None:
|
|
response = await dashboard_client.get(
|
|
"/api/dashboard/auditor/reports?report_type=weekly", headers=_HDR
|
|
)
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_agent_metrics_existing_agent(
|
|
dashboard_client: AsyncClient,
|
|
db_session: AsyncSession,
|
|
) -> None:
|
|
"""Existing agent → exercise route happy path (line 494)."""
|
|
agent = AgentTable(
|
|
id=uuid4(),
|
|
name="Probe",
|
|
slug=f"probe-{uuid4().hex[:8]}",
|
|
role=AgentRole.DEVELOPER,
|
|
team=None,
|
|
status=AgentStatus.ACTIVE,
|
|
model_config={},
|
|
system_prompt="x",
|
|
capabilities=[],
|
|
permissions={},
|
|
metrics={},
|
|
)
|
|
db_session.add(agent)
|
|
await db_session.flush()
|
|
response = await dashboard_client.get(
|
|
f"/api/dashboard/metrics/agent/{agent.id}", headers=_HDR
|
|
)
|
|
# MetricsService may return None for an empty agent → 404; or a metrics
|
|
# object if there's enough data. Either way the route is exercised.
|
|
assert response.status_code in (HTTPStatus.OK, HTTPStatus.NOT_FOUND)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_main_pm_kanban_function_directly(
|
|
db_session: AsyncSession,
|
|
) -> None:
|
|
"""Route /kanban/main-pm is unreachable via HTTP (intercepted by /kanban/{team}).
|
|
|
|
Call the route function directly to cover lines 367-369.
|
|
"""
|
|
result = await get_main_pm_kanban(db_session)
|
|
assert isinstance(result, dict)
|