Files
roboco/roboco/api/middleware.py
T
06682f33c6 Fix: agent workflow hardening (#70)
* fix(gateway): push the branch before QA handoff so reviewers see the latest commits

The commit content tool commits locally without pushing; only open_pr pushed
the branch. On the first submission that was fine, but a fix committed while
addressing needs_revision never reached origin (open_pr is skipped once the PR
exists), so QA — which reviews the remote PR branch — re-reviewed the stale
remote and re-failed the task on every cycle, a loop that never converged.

i_am_done now pushes the task branch (idempotent; a no-op when nothing is
unpushed) as part of the shared submit gate, covering both the normal and
resume-from-verifying paths. A push failure blocks the handoff with a clear
remediation rather than parking the task in awaiting_qa with commits that exist
only in the developer's local workspace.

* fix(orchestrator): don't reap a stale claim while the agent's container is alive

The stale-claim reaper released any claimed/in_progress task whose
last_heartbeat_at exceeded the TTL. The heartbeat only updates on certain
gateway calls, so a developer deep in a long edit/test cycle outran the TTL and
had its claim reaped mid-work — churning the task and risking a double spawn
against the still-running container.

The reaper now skips a task whose assignee still holds a live (ACTIVE) agent
instance, trusting container liveness — the ground truth — over the heartbeat
proxy. The check is defensive on missing fields so a heartbeat-only caller (and
the reaper's existing unit tests) behave exactly as before.

* fix(gateway): refuse to unblock a task while a dependency is unfinished

A PM unblock on a dependency-gated task moved it straight to in_progress,
overriding the dependency — letting a dependent proceed without its upstream's
work (e.g. a frontend task built before its UX design lands). A dependency
block is meant to clear on its own via _unblock_dependents the moment the
upstream reaches a terminal state.

unblock now refuses while any dependency is still non-terminal, returning a
clear remediation that the block resolves automatically. Manual unblock remains
available for genuine, non-dependency blockers.

* fix(gateway): release a dependency-blocked claim to pending instead of looping

A task that reached claimed/in_progress with an unfinished dependency was left
in that state when the claim guard rejected, so the orchestrator's respawn loop
kept reviving its assignee — which could make no progress — burning work for
nothing.

The claim guard now releases such a task back to pending. claimed -> blocked is
not a legal transition, so pending — held by the dispatch dependency filter — is
the lifecycle-correct resting state: the respawn loop ignores pending tasks, and
_unblock_dependents re-dispatches it once the upstream reaches a terminal state.
release_dependency_blocked_claim shares a _force_unclaim_to_pending core with
unclaim_for_reaper so both record a truthful work-session abandon reason.

* feat(security): warn at startup in header-trust mode + document the auth posture

When ROBOCO_AGENT_AUTH_REQUIRED is not enabled the API accepts the X-Agent-Id /
X-Agent-Role headers without a signed token, so any client that can reach it may
act as any role (including 'ceo'). The API now logs a clear warning at startup
in this mode, and the README gains a Security section documenting the auth
posture and how to harden it. Acceptable only on a trusted private network — do
not expose the API to untrusted networks.

* fix(workspace): scope the refresh fetch to current + default branch

ensure_workspace's healthy short-circuit ran an all-refs 'git fetch origin' to
keep every origin/<branch> ref current. On a monorepo with many accumulated
feature/* branches that exceeds the refresh timeout, the fetch silently fails,
and the workspace keeps a stale base — so an agent builds on an out-of-date
branch.

The refresh now fetches only the workspace's current branch and the repo's
default branch (resolved via origin/HEAD), with --no-tags --prune: it transfers
near-nothing and can't time out. Readers need their own branch and the default;
the integration branch is refreshed at branch-creation time.

* fix(git): refresh a dependency-blocked task's branch off the current integration tip

A cross-cell dependent (e.g. a frontend task waiting on the UX design) was
branched off a base captured before its upstream merged into the integration
branch, and the branch was never re-synced — so the agent built on a stale
snapshot with none of the upstream's work.

Two changes close the gap:
- release_dependency_blocked_claim now clears branch_name, so the re-claim
  (after the dependency clears) re-runs branch creation.
- create_branch, when the branch is already on disk with no commits of its own,
  resets it onto the freshly-pulled base — the dependent now builds on the
  current integration tip. A branch carrying real commits is left untouched, so
  no work is discarded; the cell->leaf cascade carries the upstream down to the
  dev branch automatically.

* refactor(gateway): drop the sibling-sequence claim guard

Sibling sequence no longer gates a claim. Cross-cell ordering is
enforced by task dependencies — a cell task that depends on another is
held until its upstream reaches a terminal state, a stronger,
status-aware gate than the sequence-number check. That check was
dormant in practice anyway: every fan-out child carries sequence 0, on
which the guard short-circuited. `sequence` stays a sibling-ordering /
dispatch-priority field (list_pending ordering and the panel).

Removes sibling_sequence_guard and its _earlier_blocking_sibling
helper, the now-unused skip_sequence parameter threaded through the
claim verbs, and the sibling fetch that fed it.

* feat(gateway): sort a cross-cell dependent after its upstream

When the frontend cell task is wired to depend on its UX/UI sibling, set
its sequence to the upstream's sequence + 1 so it sorts after the design
it waits on — list_pending ordering and the panel now show UX ahead of
the implementation it gates, in either delegation order.

Adds TaskService.set_sequence (the sibling-ordering field is a service
write; it carries no claim-gating semantics — dependencies gate claims).

* feat(gateway): make the backend cell depend on UX too

UX/UI design defines the screens and API contracts both implementation
cells build against, so the backend cell — not just the frontend — waits
on the UX/UI cell task in a product fan-out and sorts after it. Wires in
either delegation order: a backend task delegated after UX gets the
dependency directly; a UX task delegated after a still-pending backend
sibling retro-wires it.

Mirrors the existing frontend wiring (_depend_backend_on_ux and
_depend_pending_backends_on_ux). Backend is held by the same dependency
gate, so it costs no extra dispatch churn.

* fix(websocket): forward notification acks instead of logging them incomplete

The bridge handler serves both notification.sent and notification.acked,
but acked events carry `agent_id` (the acking agent) rather than
`recipient_id`, so every acknowledgement tripped the missing-field guard
and logged "Incomplete notification event" instead of reaching the panel.
Accept either field as the recipient.

* feat(api): hint the full UUID when a truncated task id fails validation

Agents copy the 8-character task prefix the system shows them (the commit
prefix, task summaries) and send it as task_id, which fails UUID
validation with an opaque "invalid length" 422 and wastes a call. The
request-validation handler now detects a task_id UUID error and attaches
a `remediate` hint telling the agent to retry with the full 36-character
UUID from its task envelope.

* fix(audit): record the blocked transition when a task is escalated

Escalation sets a task to blocked by writing task.status directly, which
bypassed the validated transition helper and so never emitted a
task.blocked audit row — the lifecycle moved but the Auditor saw nothing.
Extract the audit emit from the central transition helper into
_emit_status_transition_audit and call it from the escalate path,
capturing the prior status and outgoing owner before reassignment so the
row is attributed correctly.

* fix(docs): stop doubling the docs path so design specs index into RAG

The documenter sometimes hands a doc path already rooted at docs/, and
joining it onto DOCS_BASE_PATH (/app/docs) produced /app/docs/docs/...,
so the file was never found and the spec never indexed — the frontend
cell could not retrieve the UX design over RAG. Normalize the path
before joining: trust an absolute path, otherwise strip a single
redundant leading docs/ segment.

* feat(security): let the control panel authenticate in secure mode

With ROBOCO_AGENT_AUTH_REQUIRED=true every request must carry a valid
HMAC token, which locked the human control panel out — it sends role
headers but no token. nginx, the only trusted hop between the browser
and the API, now injects the CEO token on /api and /ws, so the browser
never holds the signing secret. The injected value is just the existing
per-agent token issued for the CEO identity (issue_panel_token), so the
token-verification path is unchanged. An empty value (dev/header-trust
mode) renders to no header.

`make panel-token` prints the value; set it as ROBOCO_PANEL_AGENT_TOKEN
in .env before enabling secure mode. .env.example and the README
Security section document the flow.

* chore(compose): consolidate the two compose files into one

docker-compose.yml and docker-compose.yaml had diverged: .yml — the file
Docker actually uses — carried ROBOCO_PUBLIC_BASE_URL but was missing the
/app/manifests bind-mount, while .yaml had the manifests mount but not
the base URL. Merge the union into docker-compose.yml and delete the
duplicate so there is one source of truth and no "multiple config files"
warning.

This activates the manifests mount in the deployed file: without it the
orchestrator writes per-agent tool manifests to its ephemeral container
fs, they never reach the host for the daemon to bind-mount, and agents
fall back to all-verbs registration. Drop the stale .yaml reference from
the config.py docstring, the labeler, and the CI path filters.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-06-05 16:35:22 +02:00

384 lines
12 KiB
Python

"""
API Middleware
Request/response middleware for logging, error handling, and correlation IDs.
"""
import time
import uuid
from collections.abc import Callable, Sequence
from typing import Any, cast
import structlog
from fastapi import FastAPI, HTTPException, Request, Response
from fastapi import status as http_status
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse
from starlette.middleware.base import BaseHTTPMiddleware
from roboco.api.schemas.common import ErrorCode
from roboco.exceptions import (
AuthenticationError,
InvalidStateError,
NotFoundError,
PermissionDeniedError,
RobocoError,
ValidationError,
)
from roboco.services.base import (
ConflictError as ServiceConflictError,
)
from roboco.services.base import (
NotFoundError as ServiceNotFoundError,
)
from roboco.services.base import (
ServiceError,
ServiceUnavailableError,
)
from roboco.services.base import (
UnauthorizedError as ServiceUnauthorizedError,
)
from roboco.services.base import (
ValidationError as ServiceValidationError,
)
logger = structlog.get_logger()
# =============================================================================
# CORRELATION ID MIDDLEWARE
# =============================================================================
class CorrelationIdMiddleware(BaseHTTPMiddleware):
"""
Adds a correlation ID to each request for tracing.
The correlation ID is:
- Extracted from X-Correlation-ID header if present
- Generated if not present
- Added to response headers
- Bound to the logger context
"""
async def dispatch(self, request: Request, call_next: Callable) -> Response:
# Get or generate correlation ID
correlation_id = request.headers.get("X-Correlation-ID")
if not correlation_id:
correlation_id = str(uuid.uuid4())
# Store in request state for access in handlers
request.state.correlation_id = correlation_id
# Bind to structlog context
structlog.contextvars.clear_contextvars()
structlog.contextvars.bind_contextvars(
correlation_id=correlation_id,
path=request.url.path,
method=request.method,
)
# Process request
response = cast("Response", await call_next(request))
# Add correlation ID to response
response.headers["X-Correlation-ID"] = correlation_id
return response
# =============================================================================
# REQUEST LOGGING MIDDLEWARE
# =============================================================================
class RequestLoggingMiddleware(BaseHTTPMiddleware):
"""
Logs request/response details with timing.
"""
async def dispatch(self, request: Request, call_next: Callable) -> Response:
start_time = time.perf_counter()
# Log request
logger.info(
"Request started",
path=request.url.path,
method=request.method,
query_params=dict(request.query_params),
)
try:
response = cast("Response", await call_next(request))
duration_ms = (time.perf_counter() - start_time) * 1000
# Log response
logger.info(
"Request completed",
status_code=response.status_code,
duration_ms=round(duration_ms, 2),
)
# Add timing header
response.headers["X-Response-Time-Ms"] = str(round(duration_ms, 2))
return response
except Exception as e:
duration_ms = (time.perf_counter() - start_time) * 1000
logger.exception(
"Request failed",
duration_ms=round(duration_ms, 2),
error=str(e),
)
raise
# =============================================================================
# EXCEPTION HANDLERS
# =============================================================================
def get_status_code(exc: RobocoError) -> int:
"""Map exception type to HTTP status code."""
status_map = {
NotFoundError: 404,
ValidationError: 422,
InvalidStateError: 409,
PermissionDeniedError: 403,
AuthenticationError: 401,
}
for exc_type, status in status_map.items():
if isinstance(exc, exc_type):
return status
# Default for other RobocoError subclasses
return 400
async def roboco_exception_handler(request: Request, exc: Exception) -> JSONResponse:
"""Handle RobocoError exceptions."""
roboco_exc = cast("RobocoError", exc)
status_code = get_status_code(roboco_exc)
# Add correlation ID to error details
correlation_id = getattr(request.state, "correlation_id", None)
if correlation_id:
roboco_exc.details["correlation_id"] = correlation_id
logger.warning(
"Handled exception",
error_code=roboco_exc.code,
error_message=roboco_exc.message,
status_code=status_code,
)
return JSONResponse(
status_code=status_code,
content=roboco_exc.to_dict(),
)
# `roboco.services.base.ServiceError` is a parallel exception hierarchy that
# does NOT inherit from `RobocoError` (it extends `Exception` directly), so
# `roboco_exception_handler` never sees it and the requests fall through to
# `generic_exception_handler` as 500s. Map its subclasses to the same status
# codes used in the RobocoError handler so route-layer try/except blocks can
# surface clean 4xx codes whether the service raises from `roboco.exceptions`
# or `roboco.services.base`.
_SERVICE_ERROR_STATUS: dict[type[ServiceError], int] = {
ServiceNotFoundError: 404,
ServiceValidationError: 422,
ServiceConflictError: 409,
ServiceUnauthorizedError: 403,
ServiceUnavailableError: 503,
}
async def service_exception_handler(request: Request, exc: Exception) -> JSONResponse:
"""Handle `roboco.services.base.ServiceError` and subclasses."""
svc_exc = cast("ServiceError", exc)
status_code = 500
for exc_type, mapped_status in _SERVICE_ERROR_STATUS.items():
if isinstance(svc_exc, exc_type):
status_code = mapped_status
break
correlation_id = getattr(request.state, "correlation_id", None)
details = dict(svc_exc.details)
if correlation_id:
details["correlation_id"] = correlation_id
logger.warning(
"Handled exception",
error_type=type(svc_exc).__name__,
error_message=svc_exc.message,
status_code=status_code,
)
return JSONResponse(
status_code=status_code,
content={
"error": type(svc_exc).__name__,
"message": svc_exc.message,
"details": details,
},
)
async def generic_exception_handler(request: Request, exc: Exception) -> JSONResponse:
"""Handle unexpected exceptions."""
correlation_id = getattr(request.state, "correlation_id", None)
logger.exception(
"Unhandled exception",
error=str(exc),
error_type=type(exc).__name__,
)
return JSONResponse(
status_code=http_status.HTTP_500_INTERNAL_SERVER_ERROR,
content={
"error": {
"code": ErrorCode.INTERNAL_ERROR,
"message": "An internal error occurred",
"details": {
"correlation_id": correlation_id,
},
}
},
)
# Map HTTP status codes to string error codes
_HTTP_TO_ERROR_CODE: dict[int, str] = {
400: ErrorCode.INVALID_INPUT,
401: ErrorCode.NOT_AUTHORIZED,
403: ErrorCode.ACCESS_DENIED,
404: ErrorCode.NOT_FOUND,
409: ErrorCode.INVALID_INPUT, # Conflict
422: ErrorCode.INVALID_INPUT, # Validation error
500: ErrorCode.INTERNAL_ERROR,
}
async def http_exception_handler(request: Request, exc: Exception) -> JSONResponse:
"""
Handle FastAPI HTTPException with standardized error format.
Converts HTTP status codes to string error codes for consistency with MCP.
"""
http_exc = cast("HTTPException", exc)
correlation_id = getattr(request.state, "correlation_id", None)
# Map status code to error code
error_code = _HTTP_TO_ERROR_CODE.get(http_exc.status_code, ErrorCode.INTERNAL_ERROR)
logger.warning(
"HTTP exception",
status_code=http_exc.status_code,
error_code=error_code,
detail=http_exc.detail,
)
response_content: dict = {
"error": {
"code": error_code,
"message": str(http_exc.detail),
}
}
if correlation_id:
response_content["error"]["details"] = {"correlation_id": correlation_id}
return JSONResponse(
status_code=http_exc.status_code,
content=response_content,
)
# =============================================================================
# SETUP FUNCTION
# =============================================================================
def _uuid_field_remediation(errors: Sequence[Any]) -> str | None:
"""Spell out the fix when a truncated id is sent where a UUID is required.
Agents routinely copy the 8-character task prefix the system shows them
(e.g. the ``[cee99ecc]`` commit prefix) and send it as ``task_id``, which
fails UUID validation with an opaque "invalid length" message and wastes a
call. Detect that case and hand back an actionable remediation instead.
"""
for err in errors:
if not isinstance(err, dict):
continue
loc = err.get("loc") or ()
field = loc[-1] if loc else None
if field == "task_id" and "uuid" in str(err.get("type", "")).lower():
return (
"Use the FULL 36-character task UUID, not the 8-character short "
"form shown in commit prefixes or summaries. The full id is in "
"the `task_id` field of the envelope returned by give_me_work "
"or your most recent verb."
)
return None
async def request_validation_handler(request: Request, exc: Exception) -> JSONResponse:
"""Log the rejected body before returning the standard 422 response.
FastAPI's default 422 returns validation details to the client but
nothing lands in server logs. During smoke tests this leaves us
blind to which field actually broke. Log the body + the per-field
errors so the next 422 is debuggable in one log scan.
When the failure is a truncated ``task_id`` (the recurring agent mistake),
add a ``remediate`` hint so the agent knows to retry with the full UUID.
"""
rve = cast("RequestValidationError", exc)
body = rve.body if isinstance(rve.body, str | bytes | dict | list) else None
errors = rve.errors()
logger.warning(
"Request validation failed",
path=request.url.path,
method=request.method,
body=body,
errors=errors,
)
content: dict[str, Any] = {"detail": errors, "body": body}
remediate = _uuid_field_remediation(errors)
if remediate is not None:
content["remediate"] = remediate
return JSONResponse(
status_code=http_status.HTTP_422_UNPROCESSABLE_ENTITY,
content=content,
)
def setup_middleware(app: FastAPI) -> None:
"""
Setup all middleware for the application.
Order matters:
1. CorrelationIdMiddleware - first to set correlation ID
2. RequestLoggingMiddleware - logs with correlation ID
Exception handler priority:
1. RequestValidationError - 422s; log body + per-field errors
2. HTTPException - most common, converts to string error codes
3. RobocoError - custom domain exceptions
4. Exception - catch-all for unexpected errors
"""
# Exception handlers (order: specific to general)
app.add_exception_handler(RequestValidationError, request_validation_handler)
app.add_exception_handler(HTTPException, http_exception_handler)
app.add_exception_handler(RobocoError, roboco_exception_handler)
app.add_exception_handler(ServiceError, service_exception_handler)
app.add_exception_handler(Exception, generic_exception_handler)
# Middleware (added in reverse order due to LIFO)
app.add_middleware(RequestLoggingMiddleware)
app.add_middleware(CorrelationIdMiddleware)