The FastAPI application shell, request pipeline, and real-time WebSocket fan-out layer for RoboCo. `app.py` builds the ASGI app, wires ~40 route routers, and runs the async lifespan (DB migrations, feature-flag overlay, transcription/extraction/RAG/learning service init, ordered shutdown). `middleware.py` adds correlation IDs, request logging, and a full exception-handler chain mapping domain/service/HTTP errors to structured JSON. `websocket.py` + `websocket_bridge.py` own the live panel streams (agents, notifications, system) with per-connection bounded send queues and an event-bus bridge. `deps.py` is the dependency-injection spine: agent header auth, role-gate helpers, and Choreographer/ContentActions wiring. `utils/` provides route-layer error factories and get-or-404/ownership helpers. `middleware_docs.py` enforces the docs-path permission matrix. `roboco/security.py` (outside `api/` but wired here) supplies the optional fastapi-guard HTTP security layer: `apply_guard(app)` mounts `SecurityMiddleware` last — outermost — in `create_app`, and `guarded_lifespan(lifespan)` wraps the async lifespan, both gated by `ROBOCO_GUARD_ENABLED` (default off, byte-for-byte unchanged request path while off).
| `lifespan` | async ctx mgr | app.py:82 | Startup: migrations, flag overlay, transcription/extraction/optimal/learning init; Shutdown: stop orchestrator BEFORE close_db, then close optimal, then DB |
| `create_app` | func | app.py:196 | Build FastAPI, add CORS + custom middleware, mount ~40 routers + ws_router at `/ws` |
| `_AppServices` | class | app.py:74 | Holder for transcription/extraction singletons set in lifespan |
| `DbSession` | type alias | deps.py:46 | `Annotated[AsyncSession, Depends(get_db)]` |
| `resolve_agent_id` | func | deps.py:49 | Resolve UUID-or-slug → UUID, 400 on miss |
| `_ServiceHolder` | class | deps.py:74 | Singleton store for PermissionService + orchestrator |
| `set_orchestrator`/`clear_orchestrator`/`get_orchestrator`/`get_orchestrator_or_none` | funcs | deps.py:91-119 | Global orchestrator accessors; `get_orchestrator` 503s when unset, `_or_none` used by shutdown |
| `_header_trust_agent_context` | func | deps.py:340 | The original `get_agent_context` body verbatim (header-trust); the OFF-mode path, and also what a valid agent HMAC token delegates to when cloud auth is ON |
| `_slide_session_cookie` | func | deps.py:379 | Re-mints + re-sets the session cookie on every cookie-authenticated request — the sliding 30-day window (only inactivity past `cloud_auth_cookie_max_age` logs out) |
| `_cloud_auth_agent_context` | func | deps.py:390 | Dual-path enforcement when `cloud_auth_enabled`: a valid HMAC token (any role) delegates to `_header_trust_agent_context`; otherwise a non-CEO role claim is rejected outright, and the CEO must present a valid session cookie via `resolve_session_user` |
| `get_agent_context` | func | deps.py:452 | Builds `AgentContext` from X-Agent-* headers + token (+ the `roboco_session` cookie); byte-for-byte header-trust when `cloud_auth_enabled` is False, else delegates to `_cloud_auth_agent_context` |
| `require_ceo_role` | func | deps.py:412 | Single CEO-check: raises 403 unless `role` is CEO; accepts `AgentRole`/`Role`/lowercase string; unifies orchestrator-router + release-handler CEO gates into one source of truth (`536bbb64`) |
| `_handle_notification_sent`/`_handle_agent_event`/`_handle_rate_limit_event`/`_handle_usage_event`/`_handle_a2a_message_event` | funcs | websocket_bridge.py | Event-bus → WS forwarders (`_handle_a2a_message_event` fans `A2A_MESSAGE_SENT` to `/ws/system` as an `a2a.message` frame — the CEO's live view of every agent-to-agent chat) |
| `guarded_lifespan` | func | security.py:399 | Wraps `lifespan` with guard's `make_lifespan` (redis/geo/agent init) when armed; passthrough when off |
| `build_security_config` | func | security.py:329 | Assembles the global `SecurityConfig` from settings: passive_mode, fail_secure, enforce_https, WAF calibration fields |
| `security_config` / `guard_deco` | module singletons | security.py:374-375 | Built once at import (pure, no I/O); `guard_deco` is the `SecurityDecorator` route files decorate with `@guard_deco.<verb>` |
| `prompt_injection_validator`/`secret_exfil_validator`/`internal_ssrf_validator` | async funcs | security.py:116/128/139 | Custom `@guard_deco.custom_validation` content checks the signature WAF can't cover; each returns a generic 400 (no rule detail leaked) |
| `_guard_whitelist` / `_INTERNAL_NETWORKS` | func / const | security.py | WAF/IP-ban/rate-limit exemption for the internal agent mesh: loopback + docker's default bridge pool (`172.16.0.0/12`) ONLY — deliberately excludes `10.0.0.0/8`/`192.168.0.0/16` (those would also exempt a real LAN client hitting nginx, not just the docker mesh, since a genuine LAN browser's real IP survives the one XFF hop under `trusted_proxy_depth=1`) |
| `resolve_forwarded_client_ip` | func | security.py:479 | Peels loopback/operator-named (`guard_trusted_chain_peers`) hops from the right of XFF; the remaining candidate is honored ONLY if at least one hop peeled AND the candidate is in the tailnet CGNAT range (`100.64.0.0/10`) — every other shape (direct LAN, an unnamed 172.x relay even carrying a forged prefix, all-hops, malformed) abstains to the guard's own depth-1 resolution |
| `ClientIpResolutionMiddleware` | class | security.py:521 | Pure ASGI, mounted OUTSIDE `SecurityMiddleware` (added after it, so it runs first): stamps `request.state.client_ip` (guard_core's supported override seam) with `resolve_forwarded_client_ip`'s verdict; only honors XFF when the CONNECTING peer itself is trusted to present it (`_CONNECTING_PEER_NETWORKS`, broader than and independent of the hop-peel set) |
| `_warn_unconfigured_tailnet_gateway_once` | func | security.py:453 | Logs once per distinct gateway IP (never the full XFF) when a host-proxied tailnet chain is detected behind a real docker bridge gateway but no `guard_trusted_chain_peers` is configured — the one regression left genuinely silent otherwise (traffic resolves to a whitelisted bridge IP, WAF stays inert, zero signal) |
**HTTP request**: nginx → ASGI `app` → `CorrelationIdMiddleware` (binds correlation_id + path/method to structlog) → `RequestLoggingMiddleware` (start timer) → route. Route resolves `CurrentAgentContext` via `get_agent_context` (headers + HMAC verify + identity/role/team resolution), plus service deps from `get_choreographer`/`get_content_actions`. When `ROBOCO_CLOUD_AUTH_ENABLED` is off (default) this is byte-for-byte the historical header-trust path (`_header_trust_agent_context`). When on, `_cloud_auth_agent_context` enforces a dual path: a request carrying a valid `X-Agent-Token` HMAC (any role — the agent fleet + the orchestrator's own `system` self-PATCH) is verified then delegated to the same header-trust resolution; a request with no valid token and a non-CEO role claim is rejected outright (closes the LAN header-spoof hole); the CEO alone may instead authenticate via the `roboco_session` cookie (`resolve_session_user`, `roboco.api.auth.session`), which is re-minted on every authenticated request (`_slide_session_cookie`) for a sliding 30-day window. On exception, the handler chain maps: `RequestValidationError` → 422 (scrubbed log + UUID remediation hint), `HTTPException` → standardized error code, `RobocoError` → domain status, `ServiceError` → parallel-hierarchy status, `RateLimitError` → 429 + `Retry-After`, `Exception` → 500. Response gains `X-Correlation-ID` + `X-Response-Time-Ms`. When `ROBOCO_GUARD_ENABLED` is on, `SecurityMiddleware` (mounted last in `create_app`, so outermost) runs before any of this: rate/size/WAF/custom-validator checks either block the request (enforce mode) or only log the detection (`guard_passive_mode`, the calibration posture) ahead of the correlation-id middleware; off by default, the whole path is unchanged.
**WebSocket**: panel → `wss://.../ws/{kind}/{id}` → `_require_panel_token` (CEO HMAC, except `/ws/system`) → query-param `agent_id`/`viewer_id` UUID parse (+ DB existence check for agent/notifications) → `manager.connect_*` accepts, registers in subscription set, spawns per-connection `_run_sender` task draining a bounded queue. Receive loop `await asyncio.wait_for(receive_text(), IDLE_TIMEOUT_SECONDS)`; "ping"→"pong"; timeout/disconnect → `finally disconnect`. **Broadcasts**: service code calls `manager.broadcast_to_*` → `_enqueue_or_send` per conn → either `conn.queue.put_nowait` (drop+warn on full) or legacy fire-and-forget `_send_with_timeout`. **Event-bus bridge**: `StreamEventBus` publishes `NOTIFICATION_SENT`/`AGENT_*`/`RATE_LIMIT_*`/`USAGE_SNAPSHOT`/`A2A_MESSAGE_SENT` → `websocket_bridge` handlers → `manager.broadcast_*` → panel. `A2A_MESSAGE_SENT` (published on every agent-to-agent DM) fans out to `/ws/system` as an `a2a.message` frame carrying conversation_id/message_id/task_id/from_agent/to_agent/skill + a capped body excerpt — the CEO's live view of every agent-to-agent chat, and the canonical pattern for wiring a new live event to the panel.
**External**: `fastapi` (FastAPI, APIRouter, WebSocket, HTTPException, Depends, Header, status), `starlette.middleware.base.BaseHTTPMiddleware`, `starlette` responses, `sqlalchemy` (select, async session), `structlog`, `pydantic` (via schemas), `asyncio`, `uuid`, `json`, `time`, `contextlib`. (`httpx` was REMOVED from websocket.py in the baseline→head diff — the self-call `validate_channel_access` is gone.)
## Entry Points
-`roboco.api.app:app` — the ASGI instance uvicorn/gunicorn serves; `create_app()` called at import.
-`lifespan` — FastAPI async context manager; runs on startup/shutdown.
-`setup_middleware(app)` — called from `create_app` after CORS.
-`register_websocket_bridge_handlers()` / `start_websocket_bridge()` — called by orchestrator/bootstrap after the event bus is up (NOT in `lifespan`; the lifespan does not register bridge handlers).
-`ROBOCO_CLOUD_AUTH_ENABLED` (+ `_EMAIL`/`_PASSWORD`/`_SECRET`/`_COOKIE_MAX_AGE`, default off) — `deps.get_agent_context`'s dual-path switch (`_cloud_auth_agent_context` vs byte-for-byte `_header_trust_agent_context`); the login/logout FastAPI Users router is mounted by `roboco.api.auth.routes.mount_cloud_auth` only when true, but `/api/auth/status` is always mounted (public probe for the panel's `proxy.ts`).
-`ROBOCO_DATABASE_*`, `ROBOCO_REDIS_*` — read transitively via `settings` / `init_db`.
-`settings.cors_origins` / `settings.cors_allow_credentials` — CORS middleware config (app.py:218).
-`settings.app_version` / `settings.environment` / `settings.debug` — logged at startup; docs/redoc URLs are unconditional (the `if settings.debug` is commented out, app.py:207-208).
-`settings.host` / `settings.port` — no longer used in websocket.py (the httpx self-call was removed); still referenced elsewhere.
-`ROBOCO_GUARD_TRUSTED_CHAIN_PEERS` (empty by default) — comma-separated exact IP(s), never a CIDR, trusted to appear as a recorded proxy hop inside XFF beyond loopback (e.g. the docker bridge gateway a host-proxied Tailscale Serve chain terminates behind); consumed by `resolve_forwarded_client_ip`/`ClientIpResolutionMiddleware`. See `docs/rag/architecture/http-security-guard.md`.
- Otherwise no direct ROBOCO_* feature flags live in this slice; the lifespan applies persisted flag overlays via `apply_persisted_feature_flags` but does not itself read individual subsystem flags.
- **~~`/ws/system` was the only WS endpoint WITHOUT `_require_panel_token`~~ — RESOLVED (`536bbb64`)**: `/ws/system` now calls `_require_panel_token` before subscribing (websocket.py:621); all `/ws/*` endpoints are consistently gated (five at the time of `536bbb64`; three after the comms-subsystem teardown removed `/ws/channels` + `/ws/sessions`). In strict mode a missing token closes with `WS_1008_POLICY_VIOLATION`; a forged token is rejected even in dev mode.
- **`websocket.py` module docstring (lines 8-13) is STALE** — it claims WS "validates agent_id via query params and verify the agent exists in the database. In production, this should be enhanced with proper token-based authentication." Actual security is now the CEO HMAC panel token via `_require_panel_token`, and `validate_agent_exists` is only called on `/agents`, `/notifications`. The docstring misleads.
- **`get_choreographer` passes `stream_bus=None` when no orchestrator is set** (deps.py:557) — fine, but means the rate-limit park path is inert during the startup window before bootstrap sets the orchestrator.
- **`_check_agent_auth_token` dev mode**: a missing token is allowed; a presented-but-invalid token is rejected. The header-trust warning at app.py:94-102 is the only signal. In dev, any reachable client can act as any role (including `ceo`) by setting headers.
- **`_resolve_agent_identity``system` role special-case** (deps.py:281) returns `UUID(x_agent_id)` with NO DB lookup. Combined with dev-mode no-auth, a caller can claim `role=system` with an arbitrary UUID and bypass agent resolution entirely. `ROBOCO_CLOUD_AUTH_ENABLED` does not add a DB lookup here either — it only requires that a `system`/any-role claim carry a *verified* HMAC token first (`_cloud_auth_agent_context` rejects any non-CEO role claim without one), so the identity-bypass itself is unchanged, just gated behind a valid signature.
- **`_coerce_agent_role` falls back to the DB role when the header isn't a valid enum** (deps.py:298). The X-Agent-Role header is therefore advisory when malformed — the authoritative role is the agent row's. Good for safety, but means a caller cannot escalate by header alone (the DB role wins).
- **`request_validation_handler` returns the UNSCRUBBED body to the client** (middleware.py:434). Only the server log is scrubbed (`_scrub_secrets`); the 422 response echoes whatever the client sent, including any secret fields. By design (the client sent them), but worth knowing.
- **`_run_sender` self-cancels on send error**: on a hard send error it calls `self.disconnect(ws)` which cancels `conn.sender` — the very task currently running (websocket.py:155). It returns immediately after, so the cancellation lands on an already-returning task; harmless in practice but a subtle self-cancel.
- **Lifespan shutdown order is load-bearing**: orchestrator.stop() MUST run before close_db (app.py:170-186). Reverting this order silently drops final audit-log rows + respawn_tracker upserts + agent-state finalizes. `stop()` is idempotent (bootstrap's finally re-calls it).
- **`ConnectionManager` sets are NOT mutated under a lock** — relies on asyncio single-threadedness. A broadcast iterating a set while `disconnect` mutates it is safe within one event loop, but `_run_sender`'s `disconnect(ws)` is called from a different task than the receive loop's `finally disconnect`, so two tasks can concurrently mutate the same set. `set.discard` is safe but iteration-during-mutation could raise `RuntimeError: Set changed size during iteration` in pathological cases.
- **`broadcast_notification` (websocket.py:665) bypasses the `broadcast_to_*` pattern** and reaches into `manager._enqueue_or_send` directly with a pre-serialized `data` string, while `broadcast_to_*` serialize inside. Inconsistent but works.
- **`roboco/api/__init__.py` deliberately does NOT re-export `app`** — importing `roboco.api.schemas.X` must not transitively load the FastAPI app + routes (circular-import cycle). The entrypoint imports `roboco.api.app:app` directly. Do not "helpfully" re-export here.
- **`docs_url`/`redoc_url` are unconditional** (app.py:207-208) — the `if settings.debug` gating is commented out, so `/docs` and `/redoc` are always served.
- **`apply_persisted_feature_flags` is best-effort** (app.py:115-121) — a DB failure logs a warning and continues with env defaults; startup is never blocked.
- **fastapi-guard is a genuine no-op when off** (`ROBOCO_GUARD_ENABLED` default `false`) — `apply_guard` returns before `add_middleware`, so `create_app`'s request path is byte-for-byte unchanged; the per-route `@guard_deco.*` decorators across ~21 route files are harmless because the decorator only takes effect once `app.state.guard_decorator` is set by `apply_guard` (security.py:388).
- **`excluded_detection_body_fields` is the only reliable WAF-calibration knob on guard 7.2.1** — the per-route `categories`/`enabled_detection_categories` config is bypassed for JSON bodies, and the body scanner excludes TOP-LEVEL keys only, scanning `str(value)` of every non-excluded field (the whole stringified subtree). A free-form container field (e.g. `plan`, `findings`) must therefore be excluded wholesale or its nested prose still trips the WAF.
-`CLAUDE.md` "WebSocket streams" section lists `/ws/agents/{id}`, `/ws/notifications/{id}`, `/ws/system` — **matches**`websocket.py` routes. No drift: both sides now agree the surviving set is these three. `/ws/channels/{id}` and `/ws/sessions/{id}` are removed from `websocket.py` itself as part of the comms-subsystem teardown (a real route deletion, not a stale doc omission).
-`CLAUDE.md` says "To add a new live event: define an `EventType`, publish it to the bus, add a `_handle_*` forwarder in `websocket_bridge`, and consume it on the panel via the `useWebSocket(...)` hook" — **matches** the `_handle_*` pattern. CLAUDE.md's worked example is now `A2A_MESSAGE_SENT` → `_handle_a2a_message_event` → `/ws/system` (replacing the retired `MESSAGE_SENT`/`useSessionStream` example), consistent with this slice's implementation. No drift.
-`CLAUDE.md` "Startup Sequence" says "FastAPI lifespan indexes documents using Ollama (~30-60s)" — `app.py:137-147` initializes `OptimalService` (RAG) with a 30-90s blocking init and logs "OptimalService (RAG) initialized successfully"; the "indexes documents" framing is approximate but consistent. No material drift.
-`CLAUDE.md` does NOT document the `_require_panel_token` CEO-HMAC WS gate or the HTTP `require_panel_token` dep. The `websocket.py` module docstring (lines 8-13) describes the OLD query-param model, contradicting the actual HMAC-token implementation. The CLAUDE.md "Agent Gateway" section's "Agents do not call the API or per-domain MCP tools directly" is consistent with `/ws/*` being operator-only. **Drift: in-file docstring vs actual code (websocket.py:8-13); CLAUDE.md itself is silent on WS auth, so no CLAUDE.md contradiction.**
-`CLAUDE.md` "Orchestrator runtime-state durability" notes the respawn_tracker DB-durable writes are drained on `stop()` — `app.py:170-186` implements the required ordering (stop before close_db). Consistent.
-`CLAUDE.md` "Feature flags / company-in-a-box" says flags "toggle from the panel's Settings → Feature Flags card ... A toggle persists in the settings store and takes effect on the next backend restart" — `app.py:115-121` applies them in lifespan. Consistent.
-`CLAUDE.md` does not mention the `CorrelationIdMiddleware` / `RequestLoggingMiddleware` / exception-handler chain by name; `middleware.py` is the implementation of the implied "structured error" contract. No contradiction.
-`CLAUDE.md`'s "Feature flags / company-in-a-box" list of env-gated default-off subsystems does not mention `ROBOCO_GUARD_ENABLED` / the fastapi-guard HTTP security layer (`roboco/security.py`, wired here via `apply_guard`/`guarded_lifespan`); the doc is silent rather than contradictory.
> **Post-snapshot update (2026-07-01, chat-subsystem live-delivery work `76ce53e3`):** `websocket_bridge.py` is no longer unchanged — it gained `_handle_message_event` (forwards `EventType.MESSAGE_SENT` to `/ws/sessions/{id}` + `/ws/channels/{id}` as a `message.new` frame) and a `MESSAGE_SENT` subscription in `register_websocket_bridge_handlers`. This is the live transcript-update path that was previously dead (`send_message` never broadcast). **Now removed** by the comms-subsystem teardown (`docs/internal/specs/2026-07-03-comms-teardown-trace.md`): the `/ws/sessions` + `/ws/channels` routes, the `_handle_message_event` forwarder, and its `MESSAGE_SENT` subscription are all gone; `EventType.MESSAGE_SENT` itself is left as a dead/inert enum member, not deleted. The surviving bridge worked example is `_handle_a2a_message_event` (`EventType.A2A_MESSAGE_SENT` → `/ws/system` as an `a2a.message` frame).
> **Post-snapshot update (2026-07-01, logical-gap sweep `536bbb64`):** `deps.py` gained `require_ceo_role` (deps.py:412) — single source-of-truth CEO-role check shared by the orchestrator router gate and the release handler, replacing two diverged inline comparisons. `websocket.py` gated `/ws/system` with `_require_panel_token` (websocket.py:621), closing the medium regression risk; all `/ws/*` endpoints are now consistently gated (five at the time of this commit; the comms-subsystem teardown since removed `/ws/channels` + `/ws/sessions`, leaving `/ws/agents`, `/ws/notifications`, `/ws/system`).
> **Internal-mesh whitelist scoping + tailnet client-IP resolution (#605, `07bdef3d`, #646, #650).** `bf1012e9` (#605) first exempted the internal agent mesh (agents reach the orchestrator DIRECTLY on the docker bridge, HMAC-authenticated) from WAF/IP-ban entirely via a `whitelist` of ALL RFC1918 + loopback — but that also silently exempted any real LAN client reaching the host-published nginx port, since a genuine LAN browser's real IP survives the guard's depth-1 XFF resolution. `07bdef3d` scoped the whitelist down to loopback + docker's default bridge pool ONLY (`_INTERNAL_NETWORKS`), dropping `10.0.0.0/8`/`192.168.0.0/16`. Separately, `583d43f2` (#646) + `3806317a` (#650) added `ClientIpResolutionMiddleware`/`resolve_forwarded_client_ip` (security.py:479/521) to fix a real live incident: a host-proxied Tailscale Serve chain (tailnet client → Serve on loopback → nginx) resolved under depth-1 XFF to a whitelisted bridge/loopback IP, going WAF-inert for the whole `/tg` Mini App surface. The fix is deliberately narrow — it peels ONLY loopback or an operator-named `ROBOCO_GUARD_TRUSTED_CHAIN_PEERS` entry from the right of XFF, and only trusts the result if the remaining candidate is in the tailnet CGNAT range; every other shape (a direct LAN client, an unnamed 172.x relay even carrying a forged prefix, all-hops, malformed) abstains to the guard's own baseline depth-1 logic rather than trying to be a general XFF-chain resolver. `_warn_unconfigured_tailnet_gateway_once` logs once per gateway IP when this shape is detected with nothing configured, so the residual gap isn't silent.
> **Local branch (not on master, NOT deployed):** `feature/fastapi-guard-hardening` (6 fastapi-guard commits `896532a3`..`99ee666e`, branched off `ab69851d`, plus 2 unrelated bundled commits) adds `roboco/security.py` and wires it into this slice — `apply_guard(app)` mounts `SecurityMiddleware` last in `create_app` (app.py:234) and `guarded_lifespan(lifespan)` wraps the async lifespan (app.py:212), both gated by `ROBOCO_GUARD_ENABLED` (default off, byte-for-byte unchanged request path when off). Per-route `@guard_deco.*` decorators (rate_limit/max_request_size/content_type_filter/behavior_analysis/block_clouds/honeypot_detection/usage_monitor/suspicious_detection/custom_validation — 9 kinds) are applied across 21 route files outside this slice (api-routes-schemas + v1 flow/do). `build_security_config` also carries a WAF false-positive calibration: `excluded_detection_body_fields` (75 free-text top-level body fields, including container fields like plan/risks/findings/section/payload) plus `enable_penetration_detection=True`, dropping active-mode false positives on RoboCo's own code/SQL/diff/URL-bearing traffic to zero while leaving the three custom validators and the WAF on non-excluded (id/enum/slug/branch) fields fully in force. New tests: `tests/unit/test_security.py` (unit) + `tests/unit/test_security_middleware.py` (integration — mounts the real middleware end-to-end). Both NAS composes (`docker-compose.yml`/`.yaml`) arm the layer passive/log-only (`c496b677`, Phase 5) — see deployment-tooling.
Logic-touching changes in that commit, scoped to this slice:
| Change | File:Line | IMPACT |
|--------|-----------|--------|
| Lifespan shutdown now stops orchestrator BEFORE close_db (was: DB closed first, only bootstrap's finally stopped orchestrator) | app.py:170-186 | Final audit-log rows / respawn_tracker upserts / agent-state finalizes no longer silently dropped on shutdown. `get_orchestrator_or_none()` added so lifespan doesn't 503 when no orchestrator is wired (tests/skip_orchestrator). |
| `get_orchestrator_or_none` + `clear_orchestrator` added | deps.py:96-119 | New accessor for shutdown + test teardown; `get_orchestrator` still 503s. |
| `require_panel_token` HTTP dep added (CEO HMAC for live-chat bridges) | deps.py:251-274 | New gate; mirrors WS `_require_panel_token`. Browser EventSource can't set headers, so token-only. |
| `CEO_AGENT_ID` import added to deps.py | deps.py:18 | Required by `require_panel_token` + reused by WS gate. |
| `_SECRET_FIELD_NAMES` + `_scrub_secrets` added; 422 log scrubs credential fields | middleware.py:372-404, 426 | Plaintext GitHub PAT / provider API key / bearer tokens no longer dumped to structlog on a 422. Response body unchanged. |
| WS panel-token gate (`_require_panel_token`) added to channel/agent/session/notifications streams | websocket.py:61, 371/439/503/567 | `/ws/*` now CEO-HMAC-gated in prod; dev allows missing token but rejects forged. **`/ws/system` was NOT gated** (inconsistency). |
| Per-connection bounded send queue + sender task (`_ClientConnection`, `_register_sender`, `_run_sender`, `_enqueue_or_send`, `_send_with_timeout`) | websocket.py:45-156, 244-281 | Replaced `asyncio.gather(*[conn.send_text(data)])` with non-blocking enqueue. One slow client can no longer back-pressure the fan-out; full queue drops+warns. `_run_sender` reaps dead sockets on hard send error. |
| `IDLE_TIMEOUT_SECONDS` (90s) `asyncio.wait_for` on `receive_text` | websocket.py:35, 400/471/534/586/623 | Half-open sockets from dead containers no longer block the receive loop forever; `TimeoutError` → `finally disconnect`. |
| `httpx` self-call `validate_channel_access` REMOVED; `settings` import dropped from websocket.py | websocket.py (was) | The channel WS no longer makes an HTTP round-trip to `/api/permissions/check` on the local server (deadlocks/latency risk gone). Channel access now gated only by panel token. |
| `structlog` logger (`log`) replaced the old logger; `validate_agent_exists` kept for agent/session/notifications | websocket.py:30, 337 | Logging consistent with rest of API. |
| `system_stream` endpoint + `broadcast_system` + `connect_system` (already present pre-baseline) — the commit RETAINED the no-token path for `/ws/system` | websocket.py:208-212, 316-322, 608 | Operator stream stays ungated; rate-limit/usage telemetry reachable without panel token. |
`middleware_docs.py` and `utils/*` are byte-for-byte unchanged since baseline. `websocket_bridge.py` was unchanged at the snapshot but has since been edited by the chat-subsystem live-delivery work (the `_handle_message_event` forwarder + `MESSAGE_SENT` subscription — see the post-snapshot note above).
| `/ws/system` ungated while siblings require panel token | websocket.py:608 | ~~The rate-limit/usage operator stream has no `_require_panel_token` call; if nginx does not edge-gate `/ws/system`, any reachable client gets RATE_LIMIT_HIT/LIFTED + USAGE_SNAPSHOT telemetry. The other four streams gate.~~**RESOLVED (`536bbb64`)**: `/ws/system` now calls `_require_panel_token` (websocket.py:621); all `/ws/*` endpoints are consistently gated (five then; the comms-subsystem teardown since removed `/ws/channels` + `/ws/sessions`, leaving three). | ~~medium~~ resolved |
| `_run_sender` self-cancels its own task on send error | websocket.py:155 | `self.disconnect(ws)` pops `conn.sender` and calls `.cancel()` on the task currently running `_run_sender`. The immediate `return` mitigates, but a cancellation landing on a returning task is a subtle race; under heavy churn could mask a later send or trip a CancelledError in an `except Exception` handler. | low |
| Concurrent set mutation between sender-task `disconnect` and receive-loop `finally disconnect` | websocket.py:155 / 425 | Two tasks (sender + receive loop) can call `disconnect(ws)` on the same socket concurrently. `set.discard` is idempotent, but a `broadcast_to_*` iterating the same set on the loop thread during the sender-task's `disconnect` could raise `Set changed size during iteration`. Single-loop asyncio makes this rare but not impossible. | low |
| `validate_agent_exists` opens its own DB session via `async for db in get_db()` per WS upgrade | websocket.py:347 | Each `/agents`, `/notifications` upgrade grabs a session just to confirm the viewer exists — extra session pressure under many concurrent panel connections; the check is also bypassable in dev (no token). | low |
| Lifespan shutdown hangs if `orchestrator.stop()` blocks | app.py:184 | The new stop-before-close-db order is correct, but a hung `stop()` now blocks `close_optimal_service` + `close_db` (wrapped in try/except, so a hang — not an exception — is the failure mode). Pre-baseline, a hung stop only affected bootstrap's finally. | low |
| 422 response body still echoes unscrubbed secrets | middleware.py:434 | `_scrub_secrets` only scrubs the LOG, not the response. A client sending a `git_token` that fails validation gets it back in the 422 `body` field. By design but a leak surface if logs/responses are captured. | low |
| `_resolve_agent_identity``system` role bypasses DB lookup | deps.py:281 | `role=system` returns `UUID(x_agent_id)` with no DB check. In dev (no auth) a caller can claim `system` with any UUID and get an `AgentContext` for a non-existent agent. Auth-required mode still needs a valid HMAC, mitigating prod. | low |
| `_enqueue_or_send` legacy fallback creates unbounded fire-and-forget tasks | websocket.py:266 | For an unregistered socket (shouldn't happen since every `connect_*` registers), a `_send_with_timeout` task is spawned per message with no queue cap. Only the legacy path; held in `_pending_sends` for GC safety. | low |
| `broadcast_notification` double-iterates notification connections | websocket.py:665-691 | It reads `manager.notification_connections.get(agent_id)` then calls `manager._enqueue_or_send` per conn, bypassing the `broadcast_to_*` serializer. If a disconnect happens between the snapshot and the enqueue, the enqueue targets a dead socket whose sender was cancelled — `_enqueue_or_send` falls to the legacy fire-and-forget path. Low. | low |
| Stale module docstring misleads future edits | websocket.py:8-13 | Describes query-param agent validation as the security model; actual model is CEO HMAC. A future edit trusting the docstring could weaken the gate. | low |
This slice is the API spine and is in **good structural shape**. The baseline→head refactor (`15effce0`) materially hardened it: the WebSocket fan-out no longer back-pressures on a slow client (bounded per-connection queues + sender tasks), half-open sockets are reaped by an idle timeout, the lifespan shutdown ordering fix stops silent final-write drops, 422 logs no longer leak credentials, and the WS + HTTP panel-token gates close the operator-only invariant. Post-snapshot, `536bbb64` closed the remaining medium risk by gating `/ws/system` (all `/ws/*` endpoints are now consistently gated — five then, three after the comms-subsystem teardown) and added `require_ceo_role` as a single-source CEO check; `76ce53e3` wired the live message-delivery path (`_handle_message_event`, since removed by the teardown). The event-bus bridge is clean and follows the documented `_handle_*` extension pattern. The main remaining integrity concerns are minor: the `websocket.py` module docstring is stale vs the HMAC implementation, and the concurrent-set-mutation window between the sender task's error-path `disconnect` and the receive loop's `finally disconnect` is theoretically present under asyncio single-threadedness. No logic-touching change since baseline looks broken; the regression risks above are edge-case and mostly low severity. The unchanged files (`middleware_docs.py`, `utils/*`) are stable; `websocket_bridge.py` gained `_handle_message_event` (`76ce53e3`, since removed by the comms-subsystem teardown — the surviving forwarder is `_handle_a2a_message_event`). Recommended follow-ups: refresh the `websocket.py` docstring, and consider a lock or snapshot-iteration in `ConnectionManager` broadcast paths.
