Feature/observability gateway health (#247)

* feat(observability): revision_count + audit_log query index (migration 045)

Adds tasks.revision_count (the O(1) rework counter — forward-only, existing
rows default 0) and the composite index audit_log(target_id, event_type,
timestamp) that powers the cycle-time and rework reconstruction queries.
Verified the real upgrade/downgrade/upgrade chain on a throwaway pgvector PG.
First task of the 0.10.0 observability dashboards.

* feat(observability): count reworks + attribute qa_fail/pr_fail to the rejector

Every transition into needs_revision increments tasks.revision_count at the
single audit chokepoint (exactly once per bounce, across all paths incl. pr_fail
and ceo_reject), so the rework rate is an O(1) read. A QA or PR-review bounce
also emits a named task.qa_fail / task.pr_fail audit event carrying the
rejector's agent_id, so the per-agent rework scorecard charges the rejection to
the reviewer who made it, not the developer who owns the task.

* feat(observability): cycle-time, bottleneck, rework, and scorecard metrics

MetricsService gains four read methods on the audit_log + tasks data: per-stage
cycle time reconstructed from the transition journey (excluding the named
qa_fail/pr_fail events), bottleneck distribution (cumulative dwell + live parked
counts), rework rate (overall/by-team/by-agent with rejector attribution + cost
via spawn-session task_id), and a fused per-agent/per-cell scorecard. Dataclass
models with to_dict(). Verified against a real Postgres journey.

* feat(observability): cycle-time/bottleneck/rework/scorecard read endpoints

Thin read-only routes on the dashboard router delegating to MetricsService:
/metrics/cycle-time, /metrics/bottlenecks, /metrics/rework, and
/metrics/scorecard/{agent,team}. 404 when an agent scorecard target is absent.
5 route tests (200 + shape + the agent-404 case).

* feat(panel): Delivery observability tab (cycle-time, bottlenecks, rework, scorecards)

A third Metrics tab built on the observability endpoints: a per-stage
cycle-time bar chart, a bottleneck panel (worst stage + cumulative dwell +
live parked counts), a rework panel (rate + by-team + by-agent attribution +
cost), and per-cell scorecards. Reuses Recharts + Card/Badge/Skeleton and the
React-Query hook pattern; observabilityApi mirrors usageApi with mock-mode
fallbacks. tsc + eslint clean; 113 panel tests pass.

* docs(observability): changelog + CLAUDE.md for the delivery dashboards

* feat(gateway-health): recover a broken-but-alive agent instead of protecting it

The verb-heartbeat cannot tell a quiet-healthy agent from one whose MCP gateway
is broken (a corrupted /app/.venv firing no verb) yet whose container is up — the
reaper's live-skip would shield it forever. The reaper now probes the gateway
out-of-band (docker exec: does the gateway venv import its deps?) and, once it
has been broken past gateway_health_grace_seconds (tolerating a transient probe
miss), kills + evicts the container so it falls through to release + respawn.
Probe-inconclusive or healthy spares the container. Gated by
gateway_health_enabled (default-on reliability fix; in the panel Feature Flags).
Defers the optional agent-side self-check + full registry re-adoption — the
reaper's docker-liveness fallback already recovers a broken-after-restart agent.

* docs(gateway-health): changelog + CLAUDE.md for broken-but-alive recovery

* docs(observability): user-facing docs for the Delivery dashboards + gateway-health

Documents the new Metrics -> Delivery tab (cycle-time, bottlenecks, rework with
rejector attribution, cell scorecards) in the panel guide and the operations
health-and-metrics guide, and adds the gateway-health env vars + an agent-gateway
recovery note. Published MkDocs site only; settings.md's default-off flag table
intentionally omits the default-on gateway-health flag (same as overload-break).

* chore(release): cut 0.10.0 (changelog section + version refs)

* fix(gateway): exempt PM coordinators from single-task claim guards

A Main/Cell PM plans and delegates many root tasks in parallel; the work
then runs in the delegated cells, not in the PM's own hands. But the
claim-time concurrency guards meant for developers — already_active and
paused (the latter firing after i_am_idle auto-pauses the PM's own
umbrella) — were applied to the PM too, so once it held one root it could
never plan a second: it thrashed between its claimed roots and respawned
forever, burning tokens for zero progress.

_run_claim_guards now skips already_active/paused for the coordinator PM
roles (_COORDINATOR_ROLES = {main_pm, cell_pm}); only unmet_dependency — a
real upstream sequence constraint, which parks the root back to pending —
still gates a PM. paused_tasks_guard also excludes the target task itself,
so a PM re-entering its own paused umbrella never self-blocks.

Tests: a coordinator plans a second root with one in_progress + one paused
sibling (full path + claimed-recovery path), the paused target exclusion,
and the developer guards still fire. Repurposed the pre-fix test that
asserted the now-removed PM block.

* fix(metrics): coerce SQL avg/extract hours aggregates to float (panel toFixed crash)

EXTRACT(epoch ...) returns numeric on PostgreSQL 14+, which asyncpg surfaces
as a Decimal; a Decimal serializes to a quoted JSON string, so the panel's
avg_cycle_hours.toFixed(1) (and the other hours fields) threw 'toFixed is not
a function' and blanked the Delivery tab.

A single _as_hours helper now rounds every SQL-averaged hours field to a real
float — avg_cycle_hours on the new scorecards plus the pre-existing
avg_completion_hours / avg_blocked_hours / longest_blocked_hours. Token and
cost fields were already float()-cast and are unaffected.

Regression test asserts _as_hours coerces Decimal -> float and preserves the
None/zero behavior.

* feat(panel): edit a task's sequence from the details page

A task's sequence (order within siblings, lower runs first) was display-only
with no way to change it from the UI, and TaskUpdate didn't carry the field
so PATCH couldn't set it either. The details page's Dependencies tab now has
an inline sequence editor mirroring the parent / dependency editors, and
PATCH /tasks/{id} accepts a sequence field (owner or privileged role) through
the existing generic update path.

* fix(mypy): green the full make-quality type gate

make quality runs 'mypy roboco/ tests/', which the per-module checks on the
0.10.0 branch never exercised. Two issues surfaced:

- The coordinator-exemption change added role_str to
  Choreographer._run_claim_guards but not to the ChoreographerHelpers
  protocol base, so the composed Choreographer had incompatible base-class
  signatures. Sync the protocol signature.

- The gateway-health / stale-reaper tests stubbed methods by direct
  assignment (orch._m = AsyncMock()) and typed their duck-typed task doubles
  as object, tripping method-assign / assignment / attr-defined. Switch to
  monkeypatch.setattr (keeping a local mock ref for the assertions) and type
  the doubles as Any — no type: ignore.

Full mypy roboco/ tests/ clean (785 files); the 21 runtime tests pass.

* fix(metrics): static cycle-time SQL — clear bandit B608 (CI gate)

The cycle-time query interpolated an optional team clause into the text() SQL
via an f-string, which bandit flags as B608 (hardcoded SQL) and turned the
merge gate red. The team value was always a bound parameter, so it was a false
positive — but the f-string is the trigger. Rebuilt as one static query with
(CAST(:team AS text) IS NULL OR a.details->>'team' = :team) and an always-bound
team param (CAST, not ::text — SQLAlchemy's :param parser collides with
PostgreSQL's :: cast operator, which broke the query as a stray param).

Full make quality green vs a real pgvector PG (all 21 gate steps).

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>

docs/deploy/deployment.md

@@ -30,7 +30,7 @@ Two variables choose what you pull (defaults shown):
 
 ```bash
 ROBOCO_REGISTRY=ghcr.io/rennf93   # or docker.io/renzof93
-ROBOCO_VERSION=latest             # or a pinned release, e.g. 0.9.0
+ROBOCO_VERSION=latest             # or a pinned release, e.g. 0.10.0
 ```
 
 The orchestrator then spawns the **matching** pre-built agent images on demand (it reads `ROBOCO_AGENT_IMAGE_REGISTRY` / `ROBOCO_AGENT_IMAGE_TAG`, which the registry compose wires to the same registry and version). Pin `ROBOCO_VERSION` to a release tag in production so an upstream `latest` push can't silently change your fleet.

docs/deploy/env-reference.md

@@ -210,6 +210,8 @@ These gate the env-toggled capabilities. Each is inert when off. See [Optional c
 | Variable | Default | Purpose |
 |----------|---------|---------|
 | `ROBOCO_OVERLOAD_BREAK_ENABLED` | `true` | Park a provider on a persistent overload (HTTP 529/500/503) the way a 429 is parked, instead of crash-retrying. |
+| `ROBOCO_GATEWAY_HEALTH_ENABLED` | `true` | Probe a stale-heartbeat-but-live agent's gateway and kill + respawn it when the gateway is broken (a corrupted `/app` venv firing no verb), instead of the reaper protecting it forever. Off => spare live containers on verb-heartbeat liveness alone. |
+| `ROBOCO_GATEWAY_HEALTH_GRACE_SECONDS` | `180` | How long an agent gateway may probe as broken before recovery — tolerates a transient probe miss. |
 
 ### Strategy engine — default **off**
 

docs/operations/health-and-metrics.md

@@ -22,6 +22,9 @@ curl -s http://localhost:3000/api/ready
 !!! note "Startup ordering"
     The app-level root `/health` used during container startup is additionally gated on the in-house RAG engine being operational, which is why the orchestrator can take a minute to report healthy after a cold start while it indexes documents. See [deployment](../deploy/deployment.md) for the full startup sequence.
 
+!!! note "Agent gateway recovery"
+    Those probes cover the *infrastructure*; individual agents have their own liveness story. An agent whose MCP gateway breaks (a corrupted `/app` venv firing no verb) stays "up" as a container but does no work, and its verb-heartbeat goes stale. With `ROBOCO_GATEWAY_HEALTH_ENABLED` (default on) the reaper probes such a stale-but-live agent's gateway out-of-band and, once it's been broken past `ROBOCO_GATEWAY_HEALTH_GRACE_SECONDS`, kills and respawns it instead of protecting it forever — so a wedged agent self-heals without operator action. You don't monitor this directly; it shows up as a "broken-gateway agent killed and evicted" line in the orchestrator log.
+
 ## The Metrics → Performance view
 
 The **Metrics** page has a **Performance** tab driven by tasks, messages, and notifications — your read on whether work is actually flowing. See the panel walkthrough in [Metrics](../panel/metrics.md).
@@ -51,6 +54,19 @@ A team with more than 30% of its tasks blocked reads **critical**; over 15% read
 !!! tip "Health is about flow, not errors"
     This status is computed from task state, not exceptions or crashes. A green org-health with a `degraded` `/api/ready` means the infrastructure is wobbling even though the backlog looks healthy — watch both signals, they answer different questions.
 
+## The Metrics → Delivery view
+
+Where Performance answers *is work flowing*, the **Delivery** tab answers *where the time goes and how clean the work is* — reconstructed from the recorded task-transition history, so it costs no extra bookkeeping. Use it when velocity dips and you need the *why*:
+
+| Panel | Operational read |
+|-------|------------------|
+| Cycle time by stage | Average dwell per lifecycle stage — a tall stage is where work waits (e.g. review vs coding) |
+| Bottlenecks | The single worst stage by total time absorbed + how many tasks are parked in each stage right now |
+| Rework | How often work bounces to `needs_revision`, by cell and by agent (the bounce charged to the QA/PR-reviewer who sent it back), plus its token cost |
+| Cell scorecards | Per-cell completed / avg cycle / rework / cost over 7 days |
+
+The full reading guide — especially how rework attribution works — is in the panel walkthrough at [Metrics → Delivery](../panel/metrics.md#delivery).
+
 ## Next
 
 - Walk the panel surface in [Metrics](../panel/metrics.md) and [the command center](../panel/command-center.md).

docs/panel/metrics.md

@@ -1,6 +1,6 @@
 # Metrics
 
-The Metrics page (`/metrics`) is where you watch the company's throughput and its spend. Two tabs: **Performance** and **Token Usage** (the active tab is in the URL as `?tab=`).
+The Metrics page (`/metrics`) is where you watch the company's throughput and its spend. Three tabs: **Performance**, **Token Usage**, and **Delivery** (the active tab is in the URL as `?tab=`).
 
 ## Performance
 
@@ -32,6 +32,18 @@ Cost is derived from per-session token counts using provider-aware pricing — a
 !!! note "Spend against budget lives on the scorecard"
     Metrics shows raw usage and projection. Your **monthly budget cap** and whether you're over it appear on the Company Scorecard in [Business](./business.md), not here.
 
+## Delivery
+
+The Delivery tab is the flow dashboard — not *what* the company shipped or what it cost, but *how the work moved*. Every panel is reconstructed from the task lifecycle history RoboCo already records (each status transition is logged), so it needs no extra bookkeeping. Cycle-time, bottlenecks, and rework look back 30 days; the scorecards look back 7.
+
+- **Cycle Time by Stage** — the average time a task sits in each lifecycle stage (claimed, in progress, awaiting QA, awaiting documentation, awaiting PR review, awaiting PM review, …). This is where you see *where the time actually goes* — a tall "awaiting QA" bar means work waits on review, not on coding.
+- **Bottlenecks** — the same data ranked by total time absorbed, with the single **worst stage** called out and a live count of how many tasks are **parked** in each stage right now, plus the current active-blocker count. It answers "what is holding the company up today?"
+- **Rework** — how often work bounces back to `needs_revision` (the headline rate = reworked ÷ completed), broken down by cell and by agent, plus the token cost of that rework. Crucially, a bounce is attributed to the **QA or PR-reviewer who sent it back**, not the developer who owns the task — so a high `QA fails` number against a reviewer is a signal about *that reviewer's* gate, and a high rate against a developer is a signal about *their* first-pass quality.
+- **Cell scorecards** — one card per cell (Backend / Frontend / UX-UI) with its completed count, average cycle time, rework rate, and cost over the last 7 days — the quick read on which cell is moving cleanly.
+
+!!! tip "Reading rework attribution"
+    A bounce charges the reviewer who rejected it via the `task.qa_fail` / `task.pr_fail` events, while the *rate* (`reworked / completed`) is computed against the task's owner. So one agent can show a low rate (good first-pass work) while another shows many `QA fails` (an active, rejecting gate) — both are healthy. Watch for a developer with a high rate **and** a reviewer with near-zero fails: that's a gate letting work through that later needs revision.
+
 ## Next
 
 → [Cost & usage](../operations/cost-and-usage.md) for the pricing model and budget cap · [Health & metrics](../operations/health-and-metrics.md) for operational monitoring · [Command Center](./command-center.md) for the at-a-glance view.

docs/rag/workflows/task-claiming.md

@@ -29,7 +29,7 @@ The claim verb both claims and starts the task — there is no separate `start`
 
 ## Before Claiming
 
-1. Check you have capacity (one task at a time recommended)
+1. Check you have capacity (developers / QA / documenters work one task at a time; **PM coordinators are exempt** — a Main / Cell PM may hold many roots at once, gated only by sequence dependencies)
 2. Verify dependencies are completed
 3. Read task description and acceptance criteria
 
@@ -41,7 +41,7 @@ The claim verb both claims and starts the task — there is no separate `start`
 
 ## Claiming Rules
 
-- **One at a time**: Don't claim multiple in-progress tasks
+- **One at a time (workers only)**: Developers, QA, and documenters can't hold multiple in-progress tasks at once. **PM coordinators are exempt** — a Main / Cell PM plans and delegates many roots in parallel, so it may hold several at once; only a real upstream **sequence dependency** (an unfinished task it depends on) holds one of its roots back.
 - **Self-review prevention**: QA cannot `claim_review` tasks they developed
 - **Self-documentation prevention**: Documenter cannot claim tasks they developed
 - **Branch requirement**: Branch auto-created on `i_will_work_on`