Files
roboco/docs/deploy/deployment.md
T
c09cf80b40 Feature/observability gateway health (#247)
* feat(observability): revision_count + audit_log query index (migration 045)

Adds tasks.revision_count (the O(1) rework counter — forward-only, existing
rows default 0) and the composite index audit_log(target_id, event_type,
timestamp) that powers the cycle-time and rework reconstruction queries.
Verified the real upgrade/downgrade/upgrade chain on a throwaway pgvector PG.
First task of the 0.10.0 observability dashboards.

* feat(observability): count reworks + attribute qa_fail/pr_fail to the rejector

Every transition into needs_revision increments tasks.revision_count at the
single audit chokepoint (exactly once per bounce, across all paths incl. pr_fail
and ceo_reject), so the rework rate is an O(1) read. A QA or PR-review bounce
also emits a named task.qa_fail / task.pr_fail audit event carrying the
rejector's agent_id, so the per-agent rework scorecard charges the rejection to
the reviewer who made it, not the developer who owns the task.

* feat(observability): cycle-time, bottleneck, rework, and scorecard metrics

MetricsService gains four read methods on the audit_log + tasks data: per-stage
cycle time reconstructed from the transition journey (excluding the named
qa_fail/pr_fail events), bottleneck distribution (cumulative dwell + live parked
counts), rework rate (overall/by-team/by-agent with rejector attribution + cost
via spawn-session task_id), and a fused per-agent/per-cell scorecard. Dataclass
models with to_dict(). Verified against a real Postgres journey.

* feat(observability): cycle-time/bottleneck/rework/scorecard read endpoints

Thin read-only routes on the dashboard router delegating to MetricsService:
/metrics/cycle-time, /metrics/bottlenecks, /metrics/rework, and
/metrics/scorecard/{agent,team}. 404 when an agent scorecard target is absent.
5 route tests (200 + shape + the agent-404 case).

* feat(panel): Delivery observability tab (cycle-time, bottlenecks, rework, scorecards)

A third Metrics tab built on the observability endpoints: a per-stage
cycle-time bar chart, a bottleneck panel (worst stage + cumulative dwell +
live parked counts), a rework panel (rate + by-team + by-agent attribution +
cost), and per-cell scorecards. Reuses Recharts + Card/Badge/Skeleton and the
React-Query hook pattern; observabilityApi mirrors usageApi with mock-mode
fallbacks. tsc + eslint clean; 113 panel tests pass.

* docs(observability): changelog + CLAUDE.md for the delivery dashboards

* feat(gateway-health): recover a broken-but-alive agent instead of protecting it

The verb-heartbeat cannot tell a quiet-healthy agent from one whose MCP gateway
is broken (a corrupted /app/.venv firing no verb) yet whose container is up — the
reaper's live-skip would shield it forever. The reaper now probes the gateway
out-of-band (docker exec: does the gateway venv import its deps?) and, once it
has been broken past gateway_health_grace_seconds (tolerating a transient probe
miss), kills + evicts the container so it falls through to release + respawn.
Probe-inconclusive or healthy spares the container. Gated by
gateway_health_enabled (default-on reliability fix; in the panel Feature Flags).
Defers the optional agent-side self-check + full registry re-adoption — the
reaper's docker-liveness fallback already recovers a broken-after-restart agent.

* docs(gateway-health): changelog + CLAUDE.md for broken-but-alive recovery

* docs(observability): user-facing docs for the Delivery dashboards + gateway-health

Documents the new Metrics -> Delivery tab (cycle-time, bottlenecks, rework with
rejector attribution, cell scorecards) in the panel guide and the operations
health-and-metrics guide, and adds the gateway-health env vars + an agent-gateway
recovery note. Published MkDocs site only; settings.md's default-off flag table
intentionally omits the default-on gateway-health flag (same as overload-break).

* chore(release): cut 0.10.0 (changelog section + version refs)

* fix(gateway): exempt PM coordinators from single-task claim guards

A Main/Cell PM plans and delegates many root tasks in parallel; the work
then runs in the delegated cells, not in the PM's own hands. But the
claim-time concurrency guards meant for developers — already_active and
paused (the latter firing after i_am_idle auto-pauses the PM's own
umbrella) — were applied to the PM too, so once it held one root it could
never plan a second: it thrashed between its claimed roots and respawned
forever, burning tokens for zero progress.

_run_claim_guards now skips already_active/paused for the coordinator PM
roles (_COORDINATOR_ROLES = {main_pm, cell_pm}); only unmet_dependency — a
real upstream sequence constraint, which parks the root back to pending —
still gates a PM. paused_tasks_guard also excludes the target task itself,
so a PM re-entering its own paused umbrella never self-blocks.

Tests: a coordinator plans a second root with one in_progress + one paused
sibling (full path + claimed-recovery path), the paused target exclusion,
and the developer guards still fire. Repurposed the pre-fix test that
asserted the now-removed PM block.

* fix(metrics): coerce SQL avg/extract hours aggregates to float (panel toFixed crash)

EXTRACT(epoch ...) returns numeric on PostgreSQL 14+, which asyncpg surfaces
as a Decimal; a Decimal serializes to a quoted JSON string, so the panel's
avg_cycle_hours.toFixed(1) (and the other hours fields) threw 'toFixed is not
a function' and blanked the Delivery tab.

A single _as_hours helper now rounds every SQL-averaged hours field to a real
float — avg_cycle_hours on the new scorecards plus the pre-existing
avg_completion_hours / avg_blocked_hours / longest_blocked_hours. Token and
cost fields were already float()-cast and are unaffected.

Regression test asserts _as_hours coerces Decimal -> float and preserves the
None/zero behavior.

* feat(panel): edit a task's sequence from the details page

A task's sequence (order within siblings, lower runs first) was display-only
with no way to change it from the UI, and TaskUpdate didn't carry the field
so PATCH couldn't set it either. The details page's Dependencies tab now has
an inline sequence editor mirroring the parent / dependency editors, and
PATCH /tasks/{id} accepts a sequence field (owner or privileged role) through
the existing generic update path.

* fix(mypy): green the full make-quality type gate

make quality runs 'mypy roboco/ tests/', which the per-module checks on the
0.10.0 branch never exercised. Two issues surfaced:

- The coordinator-exemption change added role_str to
  Choreographer._run_claim_guards but not to the ChoreographerHelpers
  protocol base, so the composed Choreographer had incompatible base-class
  signatures. Sync the protocol signature.

- The gateway-health / stale-reaper tests stubbed methods by direct
  assignment (orch._m = AsyncMock()) and typed their duck-typed task doubles
  as object, tripping method-assign / assignment / attr-defined. Switch to
  monkeypatch.setattr (keeping a local mock ref for the assertions) and type
  the doubles as Any — no type: ignore.

Full mypy roboco/ tests/ clean (785 files); the 21 runtime tests pass.

* fix(metrics): static cycle-time SQL — clear bandit B608 (CI gate)

The cycle-time query interpolated an optional team clause into the text() SQL
via an f-string, which bandit flags as B608 (hardcoded SQL) and turned the
merge gate red. The team value was always a bound parameter, so it was a false
positive — but the f-string is the trigger. Rebuilt as one static query with
(CAST(:team AS text) IS NULL OR a.details->>'team' = :team) and an always-bound
team param (CAST, not ::text — SQLAlchemy's :param parser collides with
PostgreSQL's :: cast operator, which broke the query as a stray param).

Full make quality green vs a real pgvector PG (all 21 gate steps).

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-06-23 07:26:41 +02:00

12 KiB

Production deploy

This is the operator reference for running RoboCo on a NAS or server. If you just want it up on your laptop, the install quickstart is faster — this page assumes you've done that once and now want the durable, server-side setup: the compose files, the host mounts agents need, where data lives, how to back it up, and how to harden it.

!!! warning "Trusted network only" RoboCo is built for a private LAN or homelab. Do not expose it directly to the public internet. nginx is the single entry point, but the orchestrator's WebSocket streams and (in header-trust mode) its API assume a trusted network. Put it behind your own VPN if you need remote access.

The three compose files

There are three tracked compose files, and they are not interchangeable:

File What it does Needs a build toolchain?
docker-compose.yml Builds every image from the Dockerfiles in docker/. Yes
docker-compose.yaml Byte-identical to docker-compose.yml. Yes
docker-compose.registry.yml Pulls and runs the pre-built published images. No

docker-compose.yml and docker-compose.yaml are the same file under two names — Docker Compose picks up either, and the NAS deployment runs the .yaml. If you fork RoboCo and change a service, keep all three in sync.

Which one to run

For a server you don't intend to hack on, run the registry file — it pulls finished images and needs no source tree or compiler on the host:

docker compose -f docker-compose.registry.yml pull
docker compose -f docker-compose.registry.yml up -d

Two variables choose what you pull (defaults shown):

ROBOCO_REGISTRY=ghcr.io/rennf93   # or docker.io/renzof93
ROBOCO_VERSION=latest             # or a pinned release, e.g. 0.10.0

The orchestrator then spawns the matching pre-built agent images on demand (it reads ROBOCO_AGENT_IMAGE_REGISTRY / ROBOCO_AGENT_IMAGE_TAG, which the registry compose wires to the same registry and version). Pin ROBOCO_VERSION to a release tag in production so an upstream latest push can't silently change your fleet.

Build from source only when you're modifying RoboCo:

docker compose up -d   # builds on first run

!!! note "Agent images are build/pull-only services" The agent-*-image services in every compose file are one-shot stubs — they exist so docker compose build/pull materializes each per-role agent image up front. They never run as long-lived containers. The orchestrator spawns the actual agent containers itself, on demand, over the mounted Docker socket, and tears them down when their work is done.

The single origin

nginx (docker/nginx.conf, rendered from an envsubst template) is the only externally-exposed service. It listens on localhost:3000 and routes by path:

flowchart LR
    B[Browser :3000] --> N[nginx]
    N -->|/| P[panel:3000]
    N -->|/api/, /ws/, /health, /ready| O[orchestrator:8000]
Path Upstream
/api/, /ws/, /health, /ready roboco-orchestrator:8000
everything else roboco-panel:3000

The browser only ever sees one origin (:3000), so there's no CORS to configure — the panel uses relative /api and /ws URLs and lets nginx dispatch. The panel container is never published directly; you reach it only through nginx. /ws/ also gets a long (86400s) read timeout so live sockets stay open.

The backing services do publish host ports for direct inspection — Postgres on 15432, Redis on 16379, Ollama on 11435, and the orchestrator on 8000. You don't route browser traffic at these; they're there for psql, redis-cli, and the like.

Required host-path mounts

The orchestrator is Docker-in-Docker: it mounts /var/run/docker.sock and spawns agent containers itself. Because those agent bind-mounts resolve on the host daemon (not inside the orchestrator container), several paths must be given as absolute host paths — the orchestrator passes them straight through to docker run -v for each agent.

Variable What it points at Compose default
ROBOCO_HOST_PROJECT_DIR The RoboCo project directory on the host. /volume1/roboco
ROBOCO_HOST_CLAUDE_DIR / CLAUDE_AUTH_DIR The host ~/.claude Claude Code auth dir, mounted into the orchestrator and each agent. /home/renzof/.claude / ${HOME}/.claude
ROBOCO_HOST_DATA_DIR The host data dir handed to agents for shared volumes (workspaces, logs, grok-usage). /volume1/roboco/data
ROBOCO_DATA_DIR Host root for all persistent volumes mounted into the backing services and orchestrator (see below). ./data
ROBOCO_HOST_GROK_DIR Host ~/.grok SuperGrok auth — only needed if you run any agent on Grok. /home/renzof/.grok

!!! danger "These must be real, absolute host paths" A relative path or a path that only exists inside the orchestrator container will make agent spawns fail, because the host Docker daemon resolves the bind. On a NAS the project and data dirs usually live on the RAID volume (e.g. /volume1/roboco and /volume1/roboco/data).

The host ~/.grok is mounted read-write into the orchestrator (it rewrites the short-lived token in place to keep agents from hanging on an expired login) and read-only into each Grok agent. Run grok login on the host once before enabling Grok. Provider routing and the Grok runtime are covered in the models section.

Data persistence and backup

Everything durable lives under ROBOCO_DATA_DIR (default ./data). On a server, point this at a RAID volume:

ROBOCO_DATA_DIR=/volume1/roboco/data
Subdirectory Holds
postgres/ The entire database — tasks, projects, work sessions, journals, encrypted git tokens, the pgvector store.
redis/ Append-only cache, sessions, rate-limit + event-bus state.
ollama/ The local model cache (embedding model + local LLM) — large, but re-pullable.
workspaces/ Each agent's git clone of each project.
logs/ Per-agent run logs.
mcp-configs/, prompts-generated/, agent-settings/, briefings/, manifests/ Per-agent spawn artifacts the orchestrator writes.
grok-usage/ Per-agent Grok cost/usage capture.

For backup, the load-bearing directory is postgres/ (everything that isn't re-derivable). ollama/ and workspaces/ are reconstructible — Ollama re-pulls models, agents re-clone repos — so they're optional in a backup. Take Postgres backups with pg_dump against the published port rather than copying the live data directory:

pg_dump -h localhost -p 15432 -U roboco roboco > roboco-backup.sql

!!! danger "Back up ROBOCO_ENCRYPTION_KEY with the database" Every per-project GitHub token in the database is Fernet-encrypted with ROBOCO_ENCRYPTION_KEY. A database backup is useless without the key. If you lose or change the key, every stored token becomes undecryptable and must be re-entered project by project. Store the key with your secrets, keep it stable across restarts, and never commit .env.

Secure mode

On a trusted LAN RoboCo runs in header-trust mode by default (ROBOCO_AGENT_AUTH_REQUIRED=false): callers are identified by role headers, no token required. That's the intended homelab setup.

To harden it so one agent can't spoof another's role, turn on fail-closed auth:

ROBOCO_AGENT_AUTH_REQUIRED=true
ROBOCO_AGENT_AUTH_SECRET=<your HMAC secret>     # already required for docker compose
ROBOCO_PANEL_AGENT_TOKEN=<from make panel-token>

With auth required, every API call must carry a valid X-Agent-Token. The panel runs in your browser and can't hold the signing secret, so nginx injects the CEO's token for it: generate the token with make panel-token (it signs one using your ROBOCO_AGENT_AUTH_SECRET), put it in ROBOCO_PANEL_AGENT_TOKEN, and nginx adds it as X-Agent-Token on /api and /ws. The panel keeps working; the secret never reaches the browser.

ROBOCO_ENCRYPTION_KEY and ROBOCO_AGENT_AUTH_SECRET are both required for any docker compose run — the orchestrator service block guards them with compose :? so the stack refuses to start if either is unset. See Security for the full sandboxing model and the env reference for every knob.

Startup sequence

depends_on conditions enforce a strict boot order; the effective sequence is:

flowchart LR
    PG[postgres] --> OL[ollama]
    RD[redis] --> OL
    OL --> OI[ollama-init]
    OI --> OR[orchestrator]
    AB[agent-base-image] --> OR
    OR --> PN[panel]
    PN --> NG[nginx]
    OR --> NG
  • postgres / redis / ollama must each pass their healthcheck (pg_isready, redis-cli ping, ollama list) before anything downstream starts.
  • ollama-init is a one-shot that best-effort pulls the embedding model and the local LLM, then gates success on the models being present — a degraded model registry can't take down a fully-cached deployment.
  • orchestrator waits for postgres + redis + ollama healthy, ollama-init completed, and agent-base-image built. On startup it runs the database migrations itself (idempotently, to head) and indexes its knowledge base — you do not run alembic upgrade head by hand for the compose path.
  • panel waits for the orchestrator; nginx waits for both.

First boot is the slow one: the model pulls (the LLM is a couple of minutes) plus knowledge-base indexing. Watch it come up:

docker compose logs -f orchestrator
curl http://localhost:8000/health
docker ps --filter name=roboco

When the orchestrator reports serving, open http://localhost:3000. A boot that hangs is almost always waiting on ollama-init (model pull) or a healthcheck — check docker compose ps to see which service is still starting. Migration and data details are in Data & migrations; recurring boot symptoms are in Common issues.

Operator-relevant Makefile targets

The Makefile drives the host developer workflow (uv-based, for hacking on RoboCo itself) — it is separate from the Docker stack and needs uv on the host. The handful that matter operationally:

Target Does
make panel-token Prints a signed CEO token for ROBOCO_PANEL_AGENT_TOKEN (secure mode).
make infra Brings up only postgres + redis (make infra-down stops them) — for host-side dev against the backing services.
make migrate Runs alembic upgrade head on the host (the compose stack self-migrates; this is the host-dev path).
make run Runs the API + orchestrator on the host (no --reload); make api is the reload dev server, make dev runs both.
make quality The full merge gate: ruff format-check + lint, mypy, pytest with 80% coverage floor, complexity, security, dependency, and migration checks.
make serve-docs Serves this documentation locally with mkdocs serve.
make status / make logs Orchestrator status / recent logs against a running instance.

Run make help for the full list.

Next