v0.16.0: fastapi-guard HTTP security layer — calibrated + scanner honeytrap (#290)

* [fastapi-guard] Phase 1a: gated config flags for the HTTP security layer

Adds the ROBOCO_GUARD_* settings (all default-off / secure-default) for the
upcoming fastapi-guard 7.2.0 hardening — guard_enabled (master switch),
guard_fail_secure (fail-closed default; NAS overrides to false),
guard_telemetry_enabled + guard_agent_api_key + guard_project_id (guard-agent
telemetry, opt-in), guard_emergency + guard_emergency_whitelist (lockdown kill
switch). Inert until consumed: nothing reads them yet, so the request path is
unchanged. Foundation for v0.16.0.

* [fastapi-guard] Phase 1b: security foundation module + gated wiring

Add fastapi-guard 7.2.0 + guard-core 3.3.0 (bare, unpinned) and roboco/security.py:
- build_security_config() from settings — behind-nginx real-IP (trusted_proxies +
  trust_x_forwarded_proto), HSTS/CSP headers, threat-ban + 404-sweep rules,
  redis-backed state, exclude_paths (/ws + health + docs), env-driven
  enforce_https, fail_secure (secure default), emergency lockdown, guard-agent
  telemetry (opt-in), passive-mode calibration switch.
- guard_deco singleton (SecurityDecorator) for per-route decorators (Phase 2+).
- Three custom content validators guard's WAF can't cover: prompt-injection /
  role-override, secret-exfil / credential-in-body, internal-SSRF.
- apply_guard(app) + guarded_lifespan() wired into create_app AFTER settings.
  guard_passive_mode config flag added.

Entirely gated by ROBOCO_GUARD_ENABLED (default off): create_app mounts nothing
and returns the unchanged app when off (verified). make quality GREEN
(cov 95.32%, pip-audit clean, import-linter 2/0). 12 new unit tests.

* [fastapi-guard] Phase 2: critical-path decorators

Apply guard decorators to the highest-value endpoints (metadata-only; enforced
only when the middleware is mounted, so no-op while ROBOCO_GUARD_ENABLED is off):
- provider keys (ollama/grok/self-hosted writes): strict rate_limit +
  max_request_size + block_clouds (no datacenter IP should touch secret writes).
- settings write + release approve/reject (CEO-gated): strict rate_limit.
- intake chat (prompter start/messages/events): rate_limit + max_request_size +
  custom_validation(prompt_injection_validator) — the prompt-facing free-text
  ingress gets the injection/role-override/secret-exfil content scan.

make quality GREEN (cov 95.32%, contracts 2/0). App builds with guard off,
decorators inert (verified).

* [fastapi-guard] Phase 3: wide decorator coverage across ingress + sensitive routes

Targeted-wide application (metadata-only; no-op until ROBOCO_GUARD_ENABLED). The
global SecurityMiddleware already rate-limits + WAF-scans every request, so this
adds the custom content validators on free-text ingress + tight limits on
sensitive ops (not blanket per-route rate_limit on reads):

- agent gateway do verbs (note/say/commit/dm/pitch/progress/draft_playbook/...):
  rate_limit + max_request_size + custom_validation(secret_exfil or prompt_injection).
- a2a message/send + chat writes: rate_limit + size + prompt_injection.
- optimal/RAG (kb/search, rag/query, mentor/ask, errors/decisions/standards/
  learnings): prompt_injection on searches, secret_exfil on record writes; docs
  index → internal_ssrf.
- tasks: create/update → prompt_injection; QA/doc/PM transitions → secret_exfil;
  CEO-gated verbs → tight rate_limit.
- secretary chat → prompt_injection; research → internal_ssrf; orchestrator
  spawn/mutations → rate_limit; git ops + flow verbs → tight rate_limit.
Pure GET/reads left to the global middleware. Applied via a Sonnet workflow,
then verified: app builds with guard off (decorators inert), make quality GREEN
(cov 95.37%, contracts 2/0). Decoy/honeypot-path surface deferred (needs verified
guard ban-API integration — not rushed).

* [fastapi-guard] Phase 5: arm the NAS composes in passive/log-only mode

Arm ROBOCO_GUARD_ENABLED=true + ROBOCO_GUARD_PASSIVE_MODE=true +
ROBOCO_GUARD_FAIL_SECURE=false on the two NAS composes (docker-compose.yaml +
.yml). Passive = guard mounts and logs what it WOULD block but blocks nothing,
so the next NAS deploy calibrates against real traffic; flip PASSIVE_MODE off
after the false-positive review to enforce. fail_secure=false keeps a
guard-internal error from 500ing the personal deploy. The registry (user-facing)
compose is deliberately left unarmed so its published default stays conservative.
Phase 4 (passive calibration) is the operational step this enables.

* feat(security): Phase 3b — full-arsenal per-route guard enrichment

Stack the applicable guard decorators per surface instead of the minimal
rate_limit/max_request_size/custom_validation triad: content_type_filter on
every JSON-body write, honeypot_detection form-traps on human-facing POSTs,
block_clouds on key-writes + CEO release ops, behavior_analysis runaway-rate
rules on the agent flow/do verbs, suspicious_detection + usage_monitor on the
sensitive surfaces. Nine distinct decorators now applied thoughtfully per
endpoint. All metadata-only — no-op while ROBOCO_GUARD_ENABLED is off.

* fix(a2a): permit PR reviewer to deliver gate verdicts to the owning PM

can_a2a_direct had no pr_reviewer rule, so a reviewer (team=board, or a cell
team) fell through to the cell-member path and was cross-cell-denied when the
in-path gate delivered a pr_fail change-request to main-pm (or a cross-cell
cell-pm): "Cannot A2A main-pm ... Ask None to coordinate with None". The
delivery is best-effort, so pr_fail still transitioned but the verdict never
reached the owning PM — the blind-re-submit signal-gap the pr_fail fix closes.

Add an explicit pr_reviewer handler: it may A2A only cell_pm / main_pm (its
sole comms surface — everything else it posts on the PR itself), with a matching
route hint. The cell reviewers kept same-team access by coincidence; this scopes
every reviewer to PM-only, the correct model, with no other A2A caller affected.

Refresh uv.lock to the current resolution.

* feat(models): adopt Claude Sonnet 5 as the sonnet tier

Point the 'sonnet' alias at claude-sonnet-5 (MODEL_MAP) and give pr_reviewer
its own opus tier in ROLE_MODEL_MAP — it was falling through to the sonnet
default, and the role gates untrusted external/fork PRs plus root→master, which
warrants opus.

Price claude-sonnet-5 at the promotional 33% off Sonnet 4.6 ($2.01 / $10.05,
cache 0.201 / 0.5025) through 2026-08-31 via a dedicated pricing fragment that
beats the bare 'sonnet' alias; revert to full rate when the promo ends. Bare
'sonnet' stays full-rate as a conservative fallback (prod prices the resolved
claude-sonnet-5 id from the transcript).

Update the model docs and the billing / usage / manifest / spawn tests.

* feat(security): calibrate the guard WAF for RoboCo traffic + document the layer

The first end-to-end run of the fastapi-guard layer showed active enforcement
would block ~50% of legitimate agent traffic — RoboCo request bodies are code,
SQL, diffs, file paths, HTML, and URLs, which the stock signature WAF reads as
attacks. build_security_config now excludes RoboCo's free-text top-level body
fields (derived from the real request models, including the free-form container
fields whose nested prose is stringified and scanned) from WAF scanning,
dropping the active-mode false-positive rate to zero while keeping the WAF on
every non-excluded (id/enum/slug/branch) field and leaving the
prompt-injection / secret-exfil / internal-SSRF validators — which run
independently of the exclusion — fully in force. enable_penetration_detection
is made explicit.

Only excluded_detection_body_fields is reliable on guard 7.2.1: the per-route
categories knob is bypassed for JSON bodies, and the body scanner excludes
top-level keys only (scanning str(value) of every non-excluded field), so
free-form container fields must be excluded wholesale.

Adds tests/unit/test_security_middleware.py — the first end-to-end exercise of
the middleware (mounts it, drives guard's lifespan, fires real requests):
proves passive mode is log-only, active mode does not false-positive on
realistic agent payloads, threats are still blocked inside excluded fields, and
the WAF still fires on non-excluded fields.

Docs: CHANGELOG (Unreleased); a user-facing Optional-subsystems page + nav +
env reference for the HTTP security layer; the agent-facing RAG corpus
(what it is + why a request could be blocked); and the roboco mapping
(api-core-websocket / deployment-tooling / _complete_map).

* feat(security): Surface N — scanner honeytrap (guard /api auto-ban + nginx edge-drop)

Turns scanner probes against the scanner, in two layers matched to where
traffic lands. Behind nginx only /api, /ws, /health, /ready reach the
orchestrator, so guard can only see (and ban) scanner probes on those paths;
the classic root probes (/.env, /wp-login.php, /phpmyadmin, /.git/config) hit
the panel. So:

- build_security_config's threat_ban_config gains recon / sensitive_file /
  cms_probing categories. A scanner probing those fingerprints on an /api path
  is detected on the URL-path scan; repeated probes from one IP trip an adaptive
  per-IP auto-ban (redis-backed, 24h). Only bans in active mode (passive logs
  the recon hit) and needs redis (the 24h ban exceeds the in-memory cap). The
  spec's decoy-route file is redundant — the WAF url-path scan bans regardless
  of a registered route — so it is intentionally omitted.
- docker/nginx.conf drops the classic root scanner paths at the edge with 444
  (connection closed, no response) before they reach the panel, anchored to
  known scanner fingerprints so /.well-known and every real panel/API route are
  untouched. Always on, independent of ROBOCO_GUARD_ENABLED.

Tests: 2 unit (the exclusion set + the scanner-ban categories are present) and
2 integration (a decoy path is blocked in active mode, passes in passive). The
nginx regex was validated against 15 scanner + 19 legit paths (0 false
positives). Docs: CHANGELOG, the HTTP-security page, the roboco mapping, and the
agent-facing RAG corpus.

* Token optimization — per-role observability, compute policy, spawn preflight (#291)

* test(models): lock the sonnet→claude-sonnet-5 MODEL_MAP invariant

* feat(usage): surface cache tokens + cache_hit_rate in usage breakdowns

* feat(usage): add per-role usage breakdown endpoint

* feat(usage): add spawn-waste signal (per-role unproductive rate + respawn strikes)

* feat(panel): surface per-role cost/cache + spawn-waste on the metrics page

* feat(routing): Phase 2 per-role compute policy — qa→haiku, main_pm→sonnet, per-role effort env mechanism (default-inert)

* feat(orchestrator): Phase 3 flag-gated spawn preflight — refuse non-gateway delivery roles (respawn-forever guard)

* chore(compose): arm ROBOCO_SPAWN_PREFLIGHT_ENABLED on the NAS composes

* docs: per-role usage observability, per-role compute policy, and spawn preflight

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>

* fix(panel): pin outputFileTracingRoot so the standalone build isn't broken by stray lockfiles

* feat(routing): populate ROLE_EFFORT_MAP + wire the verified --effort flag (cell_pm/board/auditor to medium)

* feat(gateway): omit empty context_briefing sections (Phase 4 payload compaction)

* refactor(orchestrator): extract spawn chokepoint guards to restore xenon rank B on spawn_agent

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>

docs/map/api-core-websocket.md

@@ -4,7 +4,7 @@ Slice key: `api-core-websocket` Repo root: `/Users/renzof/Documents/GitHub/ZZZ/r
 
 ## Purpose
 
-The FastAPI application shell, request pipeline, and real-time WebSocket fan-out layer for RoboCo. `app.py` builds the ASGI app, wires ~40 route routers, and runs the async lifespan (DB migrations, feature-flag overlay, transcription/extraction/RAG/learning service init, ordered shutdown). `middleware.py` adds correlation IDs, request logging, and a full exception-handler chain mapping domain/service/HTTP errors to structured JSON. `websocket.py` + `websocket_bridge.py` own the live panel streams (channels, agents, sessions, notifications, system) with per-connection bounded send queues and an event-bus bridge. `deps.py` is the dependency-injection spine: agent header auth, role-gate helpers, and Choreographer/ContentActions wiring. `utils/` provides route-layer error factories and get-or-404/ownership helpers. `middleware_docs.py` enforces the docs-path permission matrix.
+The FastAPI application shell, request pipeline, and real-time WebSocket fan-out layer for RoboCo. `app.py` builds the ASGI app, wires ~40 route routers, and runs the async lifespan (DB migrations, feature-flag overlay, transcription/extraction/RAG/learning service init, ordered shutdown). `middleware.py` adds correlation IDs, request logging, and a full exception-handler chain mapping domain/service/HTTP errors to structured JSON. `websocket.py` + `websocket_bridge.py` own the live panel streams (channels, agents, sessions, notifications, system) with per-connection bounded send queues and an event-bus bridge. `deps.py` is the dependency-injection spine: agent header auth, role-gate helpers, and Choreographer/ContentActions wiring. `utils/` provides route-layer error factories and get-or-404/ownership helpers. `middleware_docs.py` enforces the docs-path permission matrix. `roboco/security.py` (outside `api/` but wired here) supplies the optional fastapi-guard HTTP security layer: `apply_guard(app)` mounts `SecurityMiddleware` last — outermost — in `create_app`, and `guarded_lifespan(lifespan)` wraps the async lifespan, both gated by `ROBOCO_GUARD_ENABLED` (default off, byte-for-byte unchanged request path while off).
 
 ## Files
 
@@ -20,6 +20,7 @@ The FastAPI application shell, request pipeline, and real-time WebSocket fan-out
 | `roboco/api/utils/errors.py` | HTTPException factories + `handle_service_error` + `service_error_handler` decorator | ~214 |
 | `roboco/api/utils/resources.py` | `get_or_404`, `get_by_field_or_404`, `require_ownership`/`require_recipient`/`require_membership` | ~180 |
 | `roboco/api/__init__.py` | Deliberately does NOT re-export `app` (circular-import guard, documented) | ~14 |
+| `roboco/security.py` | fastapi-guard 7.2.1 / guard-core 3.3.0 HTTP security layer: `SecurityMiddleware` + `guard_deco` (`SecurityDecorator`) singleton, gated by `ROBOCO_GUARD_ENABLED` (default off); wired into `create_app` via `apply_guard`/`guarded_lifespan` | ~407 |
 
 ## Key Symbols
 
@@ -74,10 +75,16 @@ The FastAPI application shell, request pipeline, and real-time WebSocket fan-out
 | `handle_service_error`/`service_error_handler` | func/deco | utils/errors.py:150/191 | ServiceError → HTTPException translation |
 | `get_or_404`/`get_by_field_or_404` | funcs | utils/resources.py:17/59 | Generic get-or-404 helpers |
 | `require_ownership`/`require_recipient`/`require_membership` | funcs | utils/resources.py:96/130/156 | Authorization checks |
+| `apply_guard` | func | security.py:378 | Mounts `SecurityMiddleware` on `app` + sets `app.state.guard_decorator`; no-op unless `settings.guard_enabled` |
+| `guarded_lifespan` | func | security.py:399 | Wraps `lifespan` with guard's `make_lifespan` (redis/geo/agent init) when armed; passthrough when off |
+| `build_security_config` | func | security.py:329 | Assembles the global `SecurityConfig` from settings: passive_mode, fail_secure, enforce_https, WAF calibration fields |
+| `security_config` / `guard_deco` | module singletons | security.py:374-375 | Built once at import (pure, no I/O); `guard_deco` is the `SecurityDecorator` route files decorate with `@guard_deco.<verb>` |
+| `prompt_injection_validator`/`secret_exfil_validator`/`internal_ssrf_validator` | async funcs | security.py:116/128/139 | Custom `@guard_deco.custom_validation` content checks the signature WAF can't cover; each returns a generic 400 (no rule detail leaked) |
+| `_WAF_FREETEXT_BODY_FIELDS` | const | security.py:211 | Top-level free-text body-field exclusion set (`excluded_detection_body_fields`) — the WAF calibration; includes free-form container fields (plan/risks/findings/section/payload/...) whose nested prose is stringified and scanned |
 
 ## Data Flow
 
-**HTTP request**: nginx → ASGI `app` → `CorrelationIdMiddleware` (binds correlation_id + path/method to structlog) → `RequestLoggingMiddleware` (start timer) → route. Route resolves `CurrentAgentContext` via `get_agent_context` (headers + HMAC verify + identity/role/team resolution), plus service deps from `get_choreographer`/`get_content_actions`. On exception, the handler chain maps: `RequestValidationError` → 422 (scrubbed log + UUID remediation hint), `HTTPException` → standardized error code, `RobocoError` → domain status, `ServiceError` → parallel-hierarchy status, `RateLimitError` → 429 + `Retry-After`, `Exception` → 500. Response gains `X-Correlation-ID` + `X-Response-Time-Ms`.
+**HTTP request**: nginx → ASGI `app` → `CorrelationIdMiddleware` (binds correlation_id + path/method to structlog) → `RequestLoggingMiddleware` (start timer) → route. Route resolves `CurrentAgentContext` via `get_agent_context` (headers + HMAC verify + identity/role/team resolution), plus service deps from `get_choreographer`/`get_content_actions`. On exception, the handler chain maps: `RequestValidationError` → 422 (scrubbed log + UUID remediation hint), `HTTPException` → standardized error code, `RobocoError` → domain status, `ServiceError` → parallel-hierarchy status, `RateLimitError` → 429 + `Retry-After`, `Exception` → 500. Response gains `X-Correlation-ID` + `X-Response-Time-Ms`. When `ROBOCO_GUARD_ENABLED` is on, `SecurityMiddleware` (mounted last in `create_app`, so outermost) runs before any of this: rate/size/WAF/custom-validator checks either block the request (enforce mode) or only log the detection (`guard_passive_mode`, the calibration posture) ahead of the correlation-id middleware; off by default, the whole path is unchanged.
 
 **Lifespan startup**: `init_db` (alembic upgrade + create_all fallback) → `apply_persisted_feature_flags` (panel settings overlay, best-effort) → `TranscriptionService.start()` + `ExtractionPipeline` → `get_optimal_service()` (BLOCKS 30-90s for RAG) → `LearningPropagationService.initialize(optimal)`. `app.state.*` holds singletons. **Shutdown**: stop orchestrator (drains bg DB writes) → `close_optimal_service` → `close_db`. The orchestrator-stop-before-DB order is load-bearing.
 
@@ -200,7 +207,8 @@ roboco/api/
 - `settings.cors_origins` / `settings.cors_allow_credentials` — CORS middleware config (app.py:218).
 - `settings.app_version` / `settings.environment` / `settings.debug` — logged at startup; docs/redoc URLs are unconditional (the `if settings.debug` is commented out, app.py:207-208).
 - `settings.host` / `settings.port` — no longer used in websocket.py (the httpx self-call was removed); still referenced elsewhere.
-- No direct ROBOCO_* feature flags live in this slice; the lifespan applies persisted flag overlays via `apply_persisted_feature_flags` but does not itself read individual subsystem flags.
+- `ROBOCO_GUARD_ENABLED` / `_PASSIVE_MODE` / `_FAIL_SECURE` / `_TELEMETRY_ENABLED` / `_AGENT_API_KEY` / `_PROJECT_ID` / `_EMERGENCY` / `_EMERGENCY_WHITELIST` — read by `roboco/security.py`, wired into `create_app` via `apply_guard(app)` (app.py:234) + `guarded_lifespan(lifespan)` (app.py:212); `ROBOCO_ENVIRONMENT` additionally drives `enforce_https` (production only).
+- Otherwise no direct ROBOCO_* feature flags live in this slice; the lifespan applies persisted flag overlays via `apply_persisted_feature_flags` but does not itself read individual subsystem flags.
 
 ## Gotchas
 
@@ -218,6 +226,8 @@ roboco/api/
 - **`roboco/api/__init__.py` deliberately does NOT re-export `app`** — importing `roboco.api.schemas.X` must not transitively load the FastAPI app + routes (circular-import cycle). The entrypoint imports `roboco.api.app:app` directly. Do not "helpfully" re-export here.
 - **`docs_url`/`redoc_url` are unconditional** (app.py:207-208) — the `if settings.debug` gating is commented out, so `/docs` and `/redoc` are always served.
 - **`apply_persisted_feature_flags` is best-effort** (app.py:115-121) — a DB failure logs a warning and continues with env defaults; startup is never blocked.
+- **fastapi-guard is a genuine no-op when off** (`ROBOCO_GUARD_ENABLED` default `false`) — `apply_guard` returns before `add_middleware`, so `create_app`'s request path is byte-for-byte unchanged; the per-route `@guard_deco.*` decorators across ~21 route files are harmless because the decorator only takes effect once `app.state.guard_decorator` is set by `apply_guard` (security.py:388).
+- **`excluded_detection_body_fields` is the only reliable WAF-calibration knob on guard 7.2.1** — the per-route `categories`/`enabled_detection_categories` config is bypassed for JSON bodies, and the body scanner excludes TOP-LEVEL keys only, scanning `str(value)` of every non-excluded field (the whole stringified subtree). A free-form container field (e.g. `plan`, `findings`) must therefore be excluded wholesale or its nested prose still trips the WAF.
 
 ## Drift from CLAUDE.md
 
@@ -229,6 +239,7 @@ roboco/api/
 - `CLAUDE.md` "Orchestrator runtime-state durability" notes the respawn_tracker DB-durable writes are drained on `stop()` — `app.py:170-186` implements the required ordering (stop before close_db). Consistent.
 - `CLAUDE.md` "Feature flags / company-in-a-box" says flags "toggle from the panel's Settings → Feature Flags card ... A toggle persists in the settings store and takes effect on the next backend restart" — `app.py:115-121` applies them in lifespan. Consistent.
 - `CLAUDE.md` does not mention the `CorrelationIdMiddleware` / `RequestLoggingMiddleware` / exception-handler chain by name; `middleware.py` is the implementation of the implied "structured error" contract. No contradiction.
+- `CLAUDE.md`'s "Feature flags / company-in-a-box" list of env-gated default-off subsystems does not mention `ROBOCO_GUARD_ENABLED` / the fastapi-guard HTTP security layer (`roboco/security.py`, wired here via `apply_guard`/`guarded_lifespan`); the doc is silent rather than contradictory.
 
 Net: **no direct contradictions with CLAUDE.md**; the one stale security docstring lives in `websocket.py` itself.
 
@@ -242,6 +253,8 @@ Diff stat: `app.py +20`, `deps.py +44`, `middleware.py +49`, `websocket.py +309/
 
 > **Post-snapshot update (2026-07-01, logical-gap sweep `536bbb64`):** `deps.py` gained `require_ceo_role` (deps.py:412) — single source-of-truth CEO-role check shared by the orchestrator router gate and the release handler, replacing two diverged inline comparisons. `websocket.py` gated `/ws/system` with `_require_panel_token` (websocket.py:621), closing the medium regression risk; all five `/ws/*` endpoints are now consistently gated.
 
+> **Local branch (not on master, NOT deployed):** `feature/fastapi-guard-hardening` (6 fastapi-guard commits `896532a3`..`99ee666e`, branched off `ab69851d`, plus 2 unrelated bundled commits) adds `roboco/security.py` and wires it into this slice — `apply_guard(app)` mounts `SecurityMiddleware` last in `create_app` (app.py:234) and `guarded_lifespan(lifespan)` wraps the async lifespan (app.py:212), both gated by `ROBOCO_GUARD_ENABLED` (default off, byte-for-byte unchanged request path when off). Per-route `@guard_deco.*` decorators (rate_limit/max_request_size/content_type_filter/behavior_analysis/block_clouds/honeypot_detection/usage_monitor/suspicious_detection/custom_validation — 9 kinds) are applied across 21 route files outside this slice (api-routes-schemas + v1 flow/do). `build_security_config` also carries a WAF false-positive calibration: `excluded_detection_body_fields` (75 free-text top-level body fields, including container fields like plan/risks/findings/section/payload) plus `enable_penetration_detection=True`, dropping active-mode false positives on RoboCo's own code/SQL/diff/URL-bearing traffic to zero while leaving the three custom validators and the WAF on non-excluded (id/enum/slug/branch) fields fully in force. New tests: `tests/unit/test_security.py` (unit) + `tests/unit/test_security_middleware.py` (integration — mounts the real middleware end-to-end). Both NAS composes (`docker-compose.yml`/`.yaml`) arm the layer passive/log-only (`c496b677`, Phase 5) — see deployment-tooling.
+
 Logic-touching changes in that commit, scoped to this slice:
 
 | Change | File:Line | IMPACT |