fix(release): close the 0.19.0 scan findings — sandbox mongo tag, flow-verb timeout walls, video hardening (#329)

- mongo:8-alpine → mongo:8 (tag never existed; a mongo-opted project could spawn no agents) + a Docker Hub tag-existence e2e guard for every sandbox engine
- flow-verb timeouts at both walls: shared SLOW_VERBS policy (i_am_done / submit_up / submit_root / open_pr / i_will_work_on get the 900s server budget); the MCP client now outlasts the server budget (+10s headroom, orchestrator-injected env) so agents receive the middleware's clean 504 envelope instead of dying at the old flat 30s client timeout
- cancellation safety: the quality gate kills+reaps its child on CancelledError; create_pr records the PR via a shield-with-wait-out helper so the write can neither be skipped nor race get_db's rollback
- video engine: renderer sidecar isolated on a render-only network, 2g/2cpu caps, 570s render watchdog with exit-on-hang, 512MB tar decompression cap, CEO notification on terminal render failure, reject under the approve mutex (fail-closed on Redis-down)
- dead python-jose dependency removed (drops ecdsa and its unfixable Minerva advisory PYSEC-2026-1325); panel --font-mono now a real monospace stack

Co-authored-by: Renn F <rennf93@users.noreply.github.com>

CHANGELOG.md

@@ -6,11 +6,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 ## [Unreleased]
 
-## [0.19.0] - 2026-07-05
+## [0.19.0] - 2026-07-08
 
 ### Added
 
-- **Pluggable sandbox engine registry (postgres / redis / mongo).** The per-agent-spawn sandbox service set is now a registry instead of hardcoded postgres+redis branches. A new pure module `roboco/models/sandbox.py` defines a `SandboxEngine` ABC plus three concrete engines (`_PostgresEngine` `postgres:16-alpine`, `_RedisEngine` `redis:8-alpine`, `_MongoEngine` `mongo:8-alpine`); each engine declares its image, run args, readiness probe, connection shape, and `ROBOCO_TEST_*` env emission. `VALID_SANDBOX_SERVICES = frozenset(SANDBOX_ENGINES)` is the single source of truth — derived from the registry, shared by the pydantic validators in `models/project.py` and the orchestrator-side provisioner, so they can never drift. The provisioner (`roboco/runtime/sandbox.py`) iterates the registry via a generic `_provision_engine`; the orchestrator's `_append_sandbox_env` collapses to `cmd.extend(info.emit_env())`; teardown iterates every engine. Adding a sandbox engine is now one class + one registry line — no branch edited in the provisioner or the env emitter, and the panel's edit-project dialog surfaces it via a `SANDBOX_SERVICES` catalog. Mongo rides the existing `projects.sandbox_services` opt-in (migration 057) with no new migration and no new feature flag; it adds `ROBOCO_TEST_MONGO_*` (+ `ROBOCO_TEST_MONGO_AUTH_DB=admin`). Env var names `ROBOCO_TEST_DB_*` / `ROBOCO_TEST_REDIS_*` are preserved so existing project conftests need no change. The panel's sandbox toggles became a `Set<string>` multi-select driven by the catalog.
+- **Pluggable sandbox engine registry (postgres / redis / mongo).** The per-agent-spawn sandbox service set is now a registry instead of hardcoded postgres+redis branches. A new pure module `roboco/models/sandbox.py` defines a `SandboxEngine` ABC plus three concrete engines (`_PostgresEngine` `postgres:16-alpine`, `_RedisEngine` `redis:8-alpine`, `_MongoEngine` `mongo:8`); each engine declares its image, run args, readiness probe, connection shape, and `ROBOCO_TEST_*` env emission. `VALID_SANDBOX_SERVICES = frozenset(SANDBOX_ENGINES)` is the single source of truth — derived from the registry, shared by the pydantic validators in `models/project.py` and the orchestrator-side provisioner, so they can never drift. The provisioner (`roboco/runtime/sandbox.py`) iterates the registry via a generic `_provision_engine`; the orchestrator's `_append_sandbox_env` collapses to `cmd.extend(info.emit_env())`; teardown iterates every engine. Adding a sandbox engine is now one class + one registry line — no branch edited in the provisioner or the env emitter, and the panel's edit-project dialog surfaces it via a `SANDBOX_SERVICES` catalog. Mongo rides the existing `projects.sandbox_services` opt-in (migration 057) with no new migration and no new feature flag; it adds `ROBOCO_TEST_MONGO_*` (+ `ROBOCO_TEST_MONGO_AUTH_DB=admin`). Env var names `ROBOCO_TEST_DB_*` / `ROBOCO_TEST_REDIS_*` are preserved so existing project conftests need no change. The panel's sandbox toggles became a `Set<string>` multi-select driven by the catalog.
 
 - **RoboCo video engine (default-off).** With `ROBOCO_VIDEO_ENGINE_ENABLED`, a release/feature-spotlight/on-demand CEO trigger opens a normal, assigned UX/UI authoring task (balanced across the two ux-devs) instead of a held draft — the dev builds a HyperFrames HTML composition under `motion/compositions/<id>/` and proposes its composition id + per-platform captions via the team-gated `propose_video` do-tool, then ships it through the standard commit/PR/QA/doc/review lifecycle. Once that task completes, an orchestrator render loop tars the merged `motion/` source to a new credential-free `video-renderer` sidecar, renders both the 9:16 and 1:1 MP4 cuts, and materializes a held `video_post` draft (mirroring the X-post/release-proposal shape: Secretary-owned, skipped by every dispatcher). The CEO previews, edits, approves, or rejects each draft in a new panel video queue; approving posts the rendered clip to X (native video, v2 media upload) and/or TikTok (inbox upload) under a heartbeat-renewed lock and is idempotent — an already-posted draft is a no-op. `ROBOCO_VIDEO_ON_RELEASE` / `ROBOCO_VIDEO_ON_SPOTLIGHT` gate the two automatic triggers independently of the CEO's on-demand `POST /video/request`; TikTok's OAuth2 secrets live Fernet-encrypted alongside the existing X credentials, and every unconfigured leg (renderer, X, TikTok) degrades to a graceful no-op rather than a crash. Rendered MP4s persist under `ROBOCO_VIDEO_OUTPUT_DIR` (bind-mounted in all three compose files so renders survive container recreation).
 - **Per-project video-engine opt-in.** `projects.video_engine_enabled` (migration 063, mirroring `ci_watch_enabled`): the global `ROBOCO_VIDEO_ENGINE_ENABLED` flag arms the subsystem, the per-project flag opts a repo into authoring against its `motion/` dir — `VideoEngine._opted_in_project` no-ops `open_video_task` until the operator flips it in the panel's edit-project dialog. Existing projects stay opted out.
@@ -22,6 +22,18 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 ### Fixed
 
+- **Mongo sandbox image tag existed nowhere (`mongo:8-alpine` → `mongo:8`).** MongoDB has never published an Alpine variant, so the mongo engine's pre-pull always failed and — because provisioning failure refuses the spawn by design — a project opted into a mongo sandbox could spawn no agents at all. The registry now pins `mongo:8`, the dead `MONGO_INITDB_DATABASE` env is dropped (nothing consumed it; the connection already hands agents the `admin` auth DB), and a new network-gated e2e test (`tests/e2e_smoke/test_sandbox_image_tags.py`) asserts every `SANDBOX_ENGINES` image:tag actually exists on Docker Hub — the check the fully-mocked provisioner unit tests structurally cannot make.
+
+- **Flow-verb timeouts now match what the verbs actually do — at both walls.** Live agents were abandoning tool calls at the MCP client's flat 30s `httpx` timeout (`flow_server.py`) while the server kept executing: `i_am_done` died mid-quality-gate, `i_will_work_on` died mid-workspace-clone, and planning verbs reported "timed out but succeeded". A new pure policy module (`roboco/foundation/policy/flow_timeouts.py`) defines the slow-verb set `{i_am_done, submit_up, submit_root, open_pr, i_will_work_on}` shared by both sides: the `FlowVerbTimeoutMiddleware` gives slow verbs `flow_verb_slow_timeout_seconds` (default 900) instead of the 120s default, and the agent-side client reads orchestrator-injected `ROBOCO_FLOW_VERB_TIMEOUT_SECONDS` / `ROBOCO_FLOW_VERB_SLOW_TIMEOUT_SECONDS` and always outlasts the server budget by 10s — so an agent receives the middleware's clean 504 envelope, never a raw client timeout. `i_will_plan`/`delegate` deliberately stay on the default budget: the slow wall also bounds how long a wedged verb can hold the `SELECT FOR UPDATE` task row, and `i_will_plan` is exactly the verb the row-lock wedge fix targets. The do-server `commit` tool similarly outlasts its server-side 180s git budget.
+
+- **Cancellation no longer orphans gate subprocesses or drops PR records.** `quality_gate._run_one` killed its child only on its own `TimeoutError`; an outer cancellation (the flow-verb middleware firing mid-gate) skipped the kill and left mypy/ruff running in the workspace while the agent retried into a second concurrent gate run — it now kills and reaps on `CancelledError` too. And `GitService.create_pr` shields the local `_record_pr_atomically` commit, so a cancellation landing after the PR exists on GitHub can no longer leave it locally unrecorded.
+
+- **Video engine hardening (scan follow-ups).** The `video-renderer` sidecar moves off the agent-mesh network onto a dedicated `render` bridge reachable only by the orchestrator (its headless Chrome executes agent-authored composition JS), gains `mem_limit: 2g` / `cpus: 2`, a server-side render watchdog (`RENDER_TIMEOUT_SECONDS`, default 570 — under the orchestrator's 600s client budget) that flushes a 500 and exits so Docker revives a clean container instead of accumulating wedged Chrome trees, and a 512MB tar decompression cap (`MAX_EXTRACTED_BYTES`) alongside the existing compressed-size limit. A terminally-failed render (all retry attempts spent) now sends the CEO an ack-required notification instead of dying as a log line, and `VideoPostService.reject` takes the same heartbeat mutex as `approve`, closing a narrow status-clobber race between a concurrent approve and reject.
+
+- **Dead `python-jose` dependency removed (closes PYSEC-2026-1325 exposure).** `python-jose` was declared but imported nowhere — the auth stack uses fastapi-users' pyjwt — and it transitively pinned `ecdsa`, whose Minerva timing-attack advisory (CVE-2024-23342) has no fix and never will (the maintainers consider side-channel resistance out of scope). Removing the dead dependency (+ its `types-python-jose` stubs and deptry whitelist entry) drops `ecdsa` from the tree entirely, so `pip-audit` goes green by deletion instead of by waiver.
+
+- **Panel `--font-mono` resolved to Inter.** The mono CSS var pointed at the proportional Inter font, so SHAs, task IDs, and code snippets rendered proportional; it now uses a real system monospace stack.
+
 - **Sandbox cold-pull loop + empty provisioning error string.** `docker run` pulled the sandbox image inline under a 20s run deadline, so a NAS cold pull was killed, the pull cancelled, and every retry re-pulled from scratch — a persistent loop that stranded v0.19.0 board-agent spawns with `"error": ""` (a bare `TimeoutError` stringifies to `""`). `_ensure_image` now inspects the image and pulls it under a 300s deadline before `docker run`, and provisioning failures log `f"{type(e).__name__}: {e}"` so the error is never an empty string.
 
 - **Conventions + release-readiness I/O no longer blocks the API event loop.** `ConventionsService.get_map/health/restore` and `ReleaseManagerEngine._production_assess` ran sync `git rev-parse`, filesystem walks, and yaml parses inline on the orchestrator's shared uvicorn event loop, stalling API responsiveness during conventions reads (reachable from `GET /api/projects/{id}/conventions` and the agent spawn-prepare path) and the release-manager background loop. Each blocking call is now wrapped in `asyncio.to_thread` at the async boundary; no signature changes. A concurrency audit confirmed the rest of the heavy paths (agent spawn via `docker run -d`, the video render loop, git ops via the 16-worker ThreadPoolExecutor, workspace subprocess calls) already offload correctly — no API/worker container split is warranted.

docker-compose.registry.yml

@@ -126,11 +126,16 @@ services:
     image: ${ROBOCO_REGISTRY:-ghcr.io/rennf93}/roboco-video-renderer:${ROBOCO_VERSION:-latest}
     container_name: roboco-video-renderer
     restart: unless-stopped
+    # Isolated sidecar network: headless Chrome here executes
+    # agent-authored composition HTML/JS. Only the orchestrator (also
+    # homed on `render`) can reach it; it can reach nothing else.
     networks:
-      - default
+      - render
     # Chrome headless rendering can crash under Docker's default 64MB
     # /dev/shm ("Chrome crashed"); give it real shared memory.
     shm_size: "1gb"
+    mem_limit: "2g"
+    cpus: 2
     healthcheck:
       test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:3001/health').then((r) => process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))"]
       interval: 10s
@@ -221,10 +226,12 @@ services:
     container_name: roboco-orchestrator
     restart: unless-stopped
     # Multi-homed: the agent mesh (default) for spawned agents / panel /
-    # ollama, plus the data network for postgres/redis.
+    # ollama, the data network for postgres/redis, plus render to reach the
+    # isolated video-renderer sidecar.
     networks:
       - default
       - data
+      - render
     ports:
       - "8000:8000"
     environment:
@@ -373,3 +380,9 @@ networks:
   # ports keep working.
   data:
     name: roboco_data
+  # Sidecar isolation: video-renderer executes agent-authored composition
+  # HTML/JS in headless Chrome. It lives ONLY here, reachable only by the
+  # orchestrator (multi-homed onto this network too); it can reach nothing
+  # else on roboco_default or roboco_data.
+  render:
+    name: roboco_render

docker-compose.yaml

@@ -166,11 +166,16 @@ services:
     image: roboco-video-renderer
     container_name: roboco-video-renderer
     restart: unless-stopped
+    # Isolated sidecar network: headless Chrome here executes
+    # agent-authored composition HTML/JS. Only the orchestrator (also
+    # homed on `render`) can reach it; it can reach nothing else.
     networks:
-      - default
+      - render
     # Chrome headless rendering can crash under Docker's default 64MB
     # /dev/shm ("Chrome crashed"); give it real shared memory.
     shm_size: "1gb"
+    mem_limit: "2g"
+    cpus: 2
     healthcheck:
       test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:3001/health').then((r) => process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))"]
       interval: 10s
@@ -364,10 +369,12 @@ services:
     container_name: roboco-orchestrator
     restart: unless-stopped
     # Multi-homed: the agent mesh (default) for spawned agents / panel /
-    # ollama, plus the data network for postgres/redis.
+    # ollama, the data network for postgres/redis, plus render to reach the
+    # isolated video-renderer sidecar.
     networks:
       - default
       - data
+      - render
     ports:
       - "8000:8000"
     environment:
@@ -662,6 +669,12 @@ networks:
   # ports keep working.
   data:
     name: roboco_data
+  # Sidecar isolation: video-renderer executes agent-authored composition
+  # HTML/JS in headless Chrome. It lives ONLY here, reachable only by the
+  # orchestrator (multi-homed onto this network too); it can reach nothing
+  # else on roboco_default or roboco_data.
+  render:
+    name: roboco_render
 
 volumes:
   # Named volume for MinIO — keeps rendered-video storage out of the

docker-compose.yml

@@ -166,11 +166,16 @@ services:
     image: roboco-video-renderer
     container_name: roboco-video-renderer
     restart: unless-stopped
+    # Isolated sidecar network: headless Chrome here executes
+    # agent-authored composition HTML/JS. Only the orchestrator (also
+    # homed on `render`) can reach it; it can reach nothing else.
     networks:
-      - default
+      - render
     # Chrome headless rendering can crash under Docker's default 64MB
     # /dev/shm ("Chrome crashed"); give it real shared memory.
     shm_size: "1gb"
+    mem_limit: "2g"
+    cpus: 2
     healthcheck:
       test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:3001/health').then((r) => process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))"]
       interval: 10s
@@ -364,10 +369,12 @@ services:
     container_name: roboco-orchestrator
     restart: unless-stopped
     # Multi-homed: the agent mesh (default) for spawned agents / panel /
-    # ollama, plus the data network for postgres/redis.
+    # ollama, the data network for postgres/redis, plus render to reach the
+    # isolated video-renderer sidecar.
     networks:
       - default
       - data
+      - render
     ports:
       - "8000:8000"
     environment:
@@ -662,6 +669,12 @@ networks:
   # ports keep working.
   data:
     name: roboco_data
+  # Sidecar isolation: video-renderer executes agent-authored composition
+  # HTML/JS in headless Chrome. It lives ONLY here, reachable only by the
+  # orchestrator (multi-homed onto this network too); it can reach nothing
+  # else on roboco_default or roboco_data.
+  render:
+    name: roboco_render
 
 volumes:
   # Named volume for MinIO — keeps rendered-video storage out of the

docs/map/_complete_map.md

@@ -620,7 +620,7 @@ Backend: `EventType.A2A_MESSAGE_SENT` published from `A2AService.send` (excerpt-
 ## Delta 2026-07-03 (5) — wave 3 → v0.17.0 (branch `feat/wave-3`, SDD/Sonnet 5, reviewed)
 
 Six subsystems, all default-off + additive:
-1. **Sandboxed dev DB/Redis/Mongo** (`ROBOCO_SANDBOX_DB_ENABLED`, migration 057 `projects.sandbox_services`): `SandboxProvisioner` (`roboco/runtime/sandbox.py`) `docker run`s throwaway `postgres:16-alpine`/`redis:8-alpine`/`mongo:8-alpine` sibling containers per spawn (random creds, tmpfs, labeled), injecting `ROBOCO_TEST_DB_*`/`ROBOCO_TEST_REDIS_*`/`ROBOCO_TEST_MONGO_*` in place of the prod-creds gate-env. The service set is a pluggable engine registry (`roboco/models/sandbox.py`: `SandboxEngine` ABC + `_PostgresEngine`/`_RedisEngine`/`_MongoEngine`, `SANDBOX_ENGINES` / `VALID_SANDBOX_SERVICES` derived from it); adding an engine is one class + one registry line — no provisioner or env-emitter branch. Container-tracked lifetime + orphan janitor (grace-windowed). REVIEW FIX: pre-spawn stale-clear must not tear down the just-provisioned sandbox (`teardown_sandbox=False`) + provision pre-clears stale + janitor grace; `_ensure_image` inspects+pulls (300s) before `docker run` so a NAS cold pull isn't killed at the 20s run deadline.
+1. **Sandboxed dev DB/Redis/Mongo** (`ROBOCO_SANDBOX_DB_ENABLED`, migration 057 `projects.sandbox_services`): `SandboxProvisioner` (`roboco/runtime/sandbox.py`) `docker run`s throwaway `postgres:16-alpine`/`redis:8-alpine`/`mongo:8` sibling containers per spawn (random creds, tmpfs, labeled), injecting `ROBOCO_TEST_DB_*`/`ROBOCO_TEST_REDIS_*`/`ROBOCO_TEST_MONGO_*` in place of the prod-creds gate-env. The service set is a pluggable engine registry (`roboco/models/sandbox.py`: `SandboxEngine` ABC + `_PostgresEngine`/`_RedisEngine`/`_MongoEngine`, `SANDBOX_ENGINES` / `VALID_SANDBOX_SERVICES` derived from it); adding an engine is one class + one registry line — no provisioner or env-emitter branch. Container-tracked lifetime + orphan janitor (grace-windowed). REVIEW FIX: pre-spawn stale-clear must not tear down the just-provisioned sandbox (`teardown_sandbox=False`) + provision pre-clears stale + janitor grace; `_ensure_image` inspects+pulls (300s) before `docker run` so a NAS cold pull isn't killed at the 20s run deadline.
 2. **DB network isolation** (`ROBOCO_DB_NETWORK_ISOLATED`): second `roboco_data` compose bridge = postgres+redis only, orchestrator multi-homed; agents can't reach prod DB. Suppresses `_append_gate_env` prod-creds injection. In BOTH build+registry composes (topology-coupled).
 3. **Mobile UI** (panel): `useIsMobile` (useSyncExternalStore, hydration-safe), `ResponsiveTable` table→card, snap `TabsList`, bottom tab bar, Comms/A2A single-pane drill-down below lg, vh→dvh. REVIEW FIX: `justify-center-safe` (overflow clip), memoized matchMedia subscribe.
 4. **Cloud auth** (`ROBOCO_CLOUD_AUTH_ENABLED`, migration 058 `users`): FastAPI Users, single seeded user, cookie sliding 30-day session (pwd-fingerprint JWT), `get_agent_context` dual-path (deps.py). Off = byte-identical. `proxy.ts` (Next 16). REVIEW FIX: on-mode rejects EVERY non-CEO role without a token (not just ceo — closed the PM/board :8000 spoof).

docs/map/_front.md

@@ -619,7 +619,7 @@ Backend: `EventType.A2A_MESSAGE_SENT` published from `A2AService.send` (excerpt-
 ## Delta 2026-07-03 (5) — wave 3 → v0.17.0 (branch `feat/wave-3`, SDD/Sonnet 5, reviewed)
 
 Six subsystems, all default-off + additive:
-1. **Sandboxed dev DB/Redis/Mongo** (`ROBOCO_SANDBOX_DB_ENABLED`, migration 057 `projects.sandbox_services`): `SandboxProvisioner` (`roboco/runtime/sandbox.py`) `docker run`s throwaway `postgres:16-alpine`/`redis:8-alpine`/`mongo:8-alpine` sibling containers per spawn (random creds, tmpfs, labeled), injecting `ROBOCO_TEST_DB_*`/`ROBOCO_TEST_REDIS_*`/`ROBOCO_TEST_MONGO_*` in place of the prod-creds gate-env. The service set is a pluggable engine registry (`roboco/models/sandbox.py`: `SandboxEngine` ABC + `_PostgresEngine`/`_RedisEngine`/`_MongoEngine`, `SANDBOX_ENGINES` / `VALID_SANDBOX_SERVICES` derived from it); adding an engine is one class + one registry line — no provisioner or env-emitter branch. Container-tracked lifetime + orphan janitor (grace-windowed). REVIEW FIX: pre-spawn stale-clear must not tear down the just-provisioned sandbox (`teardown_sandbox=False`) + provision pre-clears stale + janitor grace; `_ensure_image` inspects+pulls (300s) before `docker run` so a NAS cold pull isn't killed at the 20s run deadline.
+1. **Sandboxed dev DB/Redis/Mongo** (`ROBOCO_SANDBOX_DB_ENABLED`, migration 057 `projects.sandbox_services`): `SandboxProvisioner` (`roboco/runtime/sandbox.py`) `docker run`s throwaway `postgres:16-alpine`/`redis:8-alpine`/`mongo:8` sibling containers per spawn (random creds, tmpfs, labeled), injecting `ROBOCO_TEST_DB_*`/`ROBOCO_TEST_REDIS_*`/`ROBOCO_TEST_MONGO_*` in place of the prod-creds gate-env. The service set is a pluggable engine registry (`roboco/models/sandbox.py`: `SandboxEngine` ABC + `_PostgresEngine`/`_RedisEngine`/`_MongoEngine`, `SANDBOX_ENGINES` / `VALID_SANDBOX_SERVICES` derived from it); adding an engine is one class + one registry line — no provisioner or env-emitter branch. Container-tracked lifetime + orphan janitor (grace-windowed). REVIEW FIX: pre-spawn stale-clear must not tear down the just-provisioned sandbox (`teardown_sandbox=False`) + provision pre-clears stale + janitor grace; `_ensure_image` inspects+pulls (300s) before `docker run` so a NAS cold pull isn't killed at the 20s run deadline.
 2. **DB network isolation** (`ROBOCO_DB_NETWORK_ISOLATED`): second `roboco_data` compose bridge = postgres+redis only, orchestrator multi-homed; agents can't reach prod DB. Suppresses `_append_gate_env` prod-creds injection. In BOTH build+registry composes (topology-coupled).
 3. **Mobile UI** (panel): `useIsMobile` (useSyncExternalStore, hydration-safe), `ResponsiveTable` table→card, snap `TabsList`, bottom tab bar, Comms/A2A single-pane drill-down below lg, vh→dvh. REVIEW FIX: `justify-center-safe` (overflow clip), memoized matchMedia subscribe.
 4. **Cloud auth** (`ROBOCO_CLOUD_AUTH_ENABLED`, migration 058 `users`): FastAPI Users, single seeded user, cookie sliding 30-day session (pwd-fingerprint JWT), `get_agent_context` dual-path (deps.py). Off = byte-identical. `proxy.ts` (Next 16). REVIEW FIX: on-mode rejects EVERY non-CEO role without a token (not just ceo — closed the PM/board :8000 spoof).

docs/map/runtime-providers.md

@@ -10,7 +10,7 @@ This slice is the agent-runtime + LLM-provider seam plus the in-container agent
 | roboco/runtime/streaming.py | Global reasoning-stream callback holder+setter for live UI streaming | 53 |
 | roboco/runtime/transcript_retention.py | Pure selector of agent-owned old Claude transcripts to prune (never operator dirs) | 74 |
 | roboco/runtime/sandbox.py | `SandboxProvisioner` — throwaway per-agent-spawn engine sibling containers (postgres/redis/mongo via the `SANDBOX_ENGINES` registry in `roboco/models/sandbox.py`, orchestrator-side, never docker-in-agent); generic `_provision_engine` per engine, provision/teardown/janitor_sweep, standalone + unit-testable via an injected `DockerRunner` | 344 |
-| roboco/models/sandbox.py | Pure engine registry — `SandboxEngine` ABC + `_PostgresEngine` (`postgres:16-alpine`) / `_RedisEngine` (`redis:8-alpine`) / `_MongoEngine` (`mongo:8-alpine`, mongosh readiness probe, auth db `admin`); `SandboxConnection` / `SandboxInfo` (with `emit_env`); `SANDBOX_ENGINES` + `VALID_SANDBOX_SERVICES` (derived). Adding an engine = one class + one registry line — no provisioner or env-emitter branch. Lives in the models layer so `roboco/models/project.py` can derive the allowlist without importing the runtime layer. | 232 |
+| roboco/models/sandbox.py | Pure engine registry — `SandboxEngine` ABC + `_PostgresEngine` (`postgres:16-alpine`) / `_RedisEngine` (`redis:8-alpine`) / `_MongoEngine` (`mongo:8`, mongosh readiness probe, auth db `admin`); `SandboxConnection` / `SandboxInfo` (with `emit_env`); `SANDBOX_ENGINES` + `VALID_SANDBOX_SERVICES` (derived). Adding an engine = one class + one registry line — no provisioner or env-emitter branch. Lives in the models layer so `roboco/models/project.py` can derive the allowlist without importing the runtime layer. | 232 |
 | roboco/llm/__init__.py | Re-exports ToonAdapter/ToonMetrics singletons | 17 |
 | roboco/llm/metrics.py | Singleton holder for TOON token-savings metrics | 21 |
 | roboco/llm/toon_adapter.py | TOON serialization adapter for token-efficient LLM communication (JSON fallback) | 188 |

docs/rag/architecture/config-reference.md

@@ -86,7 +86,7 @@ Env-gated subsystems. Most are default-off; `ROBOCO_OVERLOAD_BREAK_ENABLED`, `RO
 | `ROBOCO_PR_GATE_AUTO_SUBMIT_ENABLED` | `true` | PR-gate turn cut: when every child of an assembled parent is terminal, run the real `submit_up` / `submit_root` system-side as the owning PM (`_try_auto_submit`) instead of spawning the PM for that turn — the submit's substance (freshness rebase, integrity check, PR open) is deterministic gate code. A gate rejection falls back to the classic PM closure spawn; the PM keeps the judgment turns (merge, revision). Each auto-submit leaves a `task.auto_submitted` audit row. Off = every closure spawns the PM to submit. |
 | `ROBOCO_SPAWN_PREFLIGHT_ENABLED` | `false` | Refuse to spawn a non-human delivery role absent from `GATEWAY_ENABLED_ROLES` (no manifest → can never claim → would respawn on the same task forever); refuse + alert the overseer once instead. Inert in practice (all delivery roles are gateway-enabled). Armed on the NAS composes. |
 | `ROBOCO_NOTIFICATION_SPAWN_COOLDOWN_SECONDS` | `600` | Cross-tick damper for notification-triggered spawns (escalation/approval/audit/a2a — task-less, so the readiness gate and respawn breaker never see them): one spawn per (agent, notification) per window; the notification stays pending so the next window retries. `0` = legacy every-tick respawn. |
-| `ROBOCO_SANDBOX_DB_ENABLED` | `false` | Sandboxed per-agent-spawn test DB/Redis/Mongo: throwaway sibling containers provisioned from the engine registry in `roboco/models/sandbox.py` (postgres:16-alpine / redis:8-alpine / mongo:8-alpine), per-project opt-in. The valid-service set is `VALID_SANDBOX_SERVICES` (registry-derived). See "Sandboxed Dev DB/Redis/Mongo" below and `docs/rag/architecture/sandbox-db.md`. |
+| `ROBOCO_SANDBOX_DB_ENABLED` | `false` | Sandboxed per-agent-spawn test DB/Redis/Mongo: throwaway sibling containers provisioned from the engine registry in `roboco/models/sandbox.py` (postgres:16-alpine / redis:8-alpine / mongo:8), per-project opt-in. The valid-service set is `VALID_SANDBOX_SERVICES` (registry-derived). See "Sandboxed Dev DB/Redis/Mongo" below and `docs/rag/architecture/sandbox-db.md`. |
 | `ROBOCO_X_ENGINE_ENABLED` | `false` | The X (Twitter) engine: draft release/mention posts, ALL held for per-post CEO approval. See "X (Twitter) Engine" below and `docs/rag/architecture/x-engine.md`. |
 | `ROBOCO_ROADMAP_ENGINE_ENABLED` | `false` | The board roadmap engine: weekly Product-Owner-authored cycle, CEO approves each item individually into BACKLOG. See "Board Roadmap Engine" below. |
 

docs/rag/architecture/sandbox-db.md

@@ -12,7 +12,7 @@ The service set is a **pluggable engine registry**, not a hardcoded postgres+red
 
 - `_PostgresEngine` — `postgres:16-alpine`, tmpfs `/var/lib/postgresql/data`, `pg_isready` probe (60s), env `ROBOCO_TEST_DB_*` (incl. `ROBOCO_TEST_DB_ADMIN_DB`).
 - `_RedisEngine` — `redis:8-alpine`, no tmpfs, `redis-cli -a … ping` probe (15s), env `ROBOCO_TEST_REDIS_*`.
-- `_MongoEngine` — `mongo:8-alpine`, tmpfs `/data/db`, `mongosh` ping against auth db `admin` (60s), env `ROBOCO_TEST_MONGO_*` (incl. `ROBOCO_TEST_MONGO_AUTH_DB=admin`).
+- `_MongoEngine` — `mongo:8` (MongoDB ships no Alpine variant), tmpfs `/data/db`, `mongosh` ping against auth db `admin` (60s), env `ROBOCO_TEST_MONGO_*` (incl. `ROBOCO_TEST_MONGO_AUTH_DB=admin`).
 
 `SANDBOX_ENGINES: dict[str, SandboxEngine]` registers them by name; `VALID_SANDBOX_SERVICES = frozenset(SANDBOX_ENGINES)` is the single source of truth the provisioner, the orchestrator's env injection, and `projects.sandbox_services` validation all consult. **Adding an engine is one class + one registry line** — no branch edited in the provisioner or the env emitter, which both iterate the registry.
 

panel/src/app/globals.css

@@ -8,7 +8,8 @@
   --color-background: var(--background);
   --color-foreground: var(--foreground);
   --font-sans: var(--font-inter);
-  --font-mono: var(--font-inter);
+  --font-mono:
+    ui-monospace, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace;
   --color-sidebar-ring: var(--sidebar-ring);
   --color-sidebar-border: var(--sidebar-border);
   --color-sidebar-accent-foreground: var(--sidebar-accent-foreground);

pyproject.toml

@@ -395,7 +395,7 @@ DEP002 = [
     "python-multipart",
     # Database migrations (CLI tool)
     "alembic",
-    # Auth libraries (used via passlib[bcrypt]; JWT via PyJWT + fastapi_users.jwt)
+    # Auth libraries (used via passlib[bcrypt])
     "passlib",
     # LLM utilities (embeddings/token counting)
     "openai",
@@ -423,7 +423,6 @@ DEP002 = [
     "rich",
     # Type stubs (used by mypy)
     "types-passlib",
-    "types-python-jose",
     "types-PyYAML",
 ]
 # DEP003: Starlette is a transitive dep of FastAPI, but BaseHTTPMiddleware is needed

roboco/api/middleware.py

@@ -30,6 +30,7 @@ from roboco.exceptions import (
     RobocoError,
     ValidationError,
 )
+from roboco.foundation.policy.flow_timeouts import SLOW_VERBS as _SLOW_VERBS
 from roboco.services.base import (
     ConflictError as ServiceConflictError,
 )
@@ -504,6 +505,13 @@ class FlowVerbTimeoutMiddleware:
 
     Reads (``evidence``) and journal writes (``note``) don't touch the task
     row, so they are unaffected; only task-row writes route through ``claim``.
+
+    ``_SLOW_VERBS`` (from ``roboco.foundation.policy.flow_timeouts`` — git
+    push + quality gate, a multi-step PR-create chain, workspace clone, or
+    planning writes) get the longer ``flow_verb_slow_timeout_seconds`` budget
+    instead of the default — routine calls to those verbs otherwise exceed
+    120s. The same set drives the agent-side MCP client's timeout
+    (``roboco/mcp/flow_server.py``) so the two walls can't drift apart.
     """
 
     def __init__(self, app: ASGIApp) -> None:
@@ -513,7 +521,12 @@ class FlowVerbTimeoutMiddleware:
         if scope["type"] != "http" or not scope["path"].startswith("/api/v1/flow/"):
             await self.app(scope, receive, send)
             return
-        timeout = settings.flow_verb_timeout_seconds
+        verb = scope["path"].rstrip("/").rsplit("/", 1)[-1]
+        timeout = (
+            settings.flow_verb_slow_timeout_seconds
+            if verb in _SLOW_VERBS
+            else settings.flow_verb_timeout_seconds
+        )
         started = False
 
         async def send_wrapper(message: Any) -> None:

roboco/config.py

@@ -1304,6 +1304,18 @@ class Settings(BaseSettings):
             "legitimate verbs are unaffected."
         ),
     )
+    flow_verb_slow_timeout_seconds: int = Field(
+        default=900,
+        ge=1,
+        description=(
+            "Server-side timeout for the slow flow verbs (i_am_done, "
+            "submit_up, submit_root, open_pr) — a git push plus a "
+            "per-command-budgeted quality gate, or a multi-step PR-create "
+            "chain, routinely exceeds flow_verb_timeout_seconds. "
+            "FlowVerbTimeoutMiddleware picks this budget for those verbs by "
+            "request path instead of the default."
+        ),
+    )
     git_commit_timeout_seconds: int = Field(
         default=180,
         ge=30,

roboco/foundation/policy/flow_timeouts.py

@@ -0,0 +1,35 @@
+"""Flow-verb timeout budgets — single source for both timeout walls.
+
+Two independent walls exist on every ``/api/v1/flow/*`` call: the server's
+``FlowVerbTimeoutMiddleware`` (``roboco/api/middleware.py``) and the agent-side
+MCP client's ``httpx.Client(timeout=...)`` (``roboco/mcp/flow_server.py``). The
+client wall must always OUTLAST the server wall — otherwise the agent's httpx
+client gives up with a raw transport error before the middleware ever gets to
+return its clean, retryable 504 ``gateway_timeout`` envelope, and the agent
+sees a Python exception instead of a directed remediation hint. Both sides
+import ``SLOW_VERBS`` from here so they can never classify a verb differently.
+"""
+
+from __future__ import annotations
+
+# Verbs whose own work routinely exceeds the default flow-verb budget:
+# i_am_done (git push + the pre-submit quality gate), submit_up / submit_root
+# / open_pr (a multi-step PR-create chain), i_will_work_on (workspace clone,
+# up to 300s). Deliberately EXCLUDES i_will_plan / delegate: the slow budget
+# also governs the server middleware wall, and i_will_plan is exactly the
+# verb that wedged in #326 holding the SELECT FOR UPDATE task-row lock — a
+# 900s budget would let a wedged planning verb block its task row for 15
+# minutes instead of 2. Healthy planning verbs are DB writes that complete
+# in seconds; their 30s+ production runs were contention symptoms.
+SLOW_VERBS = frozenset(
+    {
+        "i_am_done",
+        "submit_up",
+        "submit_root",
+        "open_pr",
+        "i_will_work_on",
+    }
+)
+
+# Client-side margin added on top of the matching server budget.
+CLIENT_HEADROOM_SECONDS = 10

roboco/mcp/do_server.py

@@ -35,6 +35,13 @@ AGENT_ID = os.environ["ROBOCO_AGENT_ID"]
 AGENT_ROLE = os.environ["ROBOCO_AGENT_ROLE"]
 
 _TIMEOUT = 30
+# commit() stages + `git commit`s in-process (no push — push is the flow
+# verb open_pr, already covered by flow_server's per-verb timeout), bounded
+# server-side by git_commit_timeout_seconds (default 180s: a large changeset,
+# e.g. the panel's hundreds of files, can legitimately take that long). The
+# shared _TIMEOUT above is tuned for fast content-tool calls (note/dm/
+# evidence) and would give up first — client must outlast the server op.
+_COMMIT_TIMEOUT = 190
 # Tight timeout for SDK loopback — local sidecar; gateway path must not stall.
 _SDK_TIMEOUT = 2.0
 # FastAPI's default missing-route status. Every /api/v1/do/* route returns
@@ -247,7 +254,9 @@ def _build_headers() -> dict[str, str]:
     return headers
 
 
-def _post(path: str, body: dict[str, Any]) -> dict[str, Any]:
+def _post(
+    path: str, body: dict[str, Any], *, timeout: float = _TIMEOUT
+) -> dict[str, Any]:
     """POST a request to the orchestrator and return the JSON envelope.
 
     Mirrors flow_server._post: surfaces the orchestrator's envelope on
@@ -260,8 +269,11 @@ def _post(path: str, body: dict[str, Any]) -> dict[str, Any]:
     REPLACED with circuit_open. Dogfooding surfaced the gap: do-server had
     no breaker and `note(scope='decision')` looped 8 times returning
     incomplete_input.
+
+    ``timeout`` overrides the default for a slow tool (e.g. commit's
+    _COMMIT_TIMEOUT) — must always outlast that tool's server-side budget.
     """
-    with httpx.Client(timeout=_TIMEOUT) as client:
+    with httpx.Client(timeout=timeout) as client:
         response = client.post(
             f"{ORCHESTRATOR_URL}{path}",
             headers=_build_headers(),
@@ -434,7 +446,11 @@ def _record_and_check_circuit(
 
 def commit(message: str, files: list[str] | None = None) -> dict[str, Any]:
     """Make a git commit. [task-id] prefix auto-applied. Validates message."""
-    return _post("/api/v1/do/commit", {"message": message, "files": files})
+    return _post(
+        "/api/v1/do/commit",
+        {"message": message, "files": files},
+        timeout=_COMMIT_TIMEOUT,
+    )
 
 
 def note(

roboco/mcp/flow_server.py

@@ -27,6 +27,7 @@ from pydantic import BeforeValidator
 
 from roboco.agents_config import get_agent_team
 from roboco.foundation.policy.content.validators import coerce_str_list
+from roboco.foundation.policy.flow_timeouts import CLIENT_HEADROOM_SECONDS, SLOW_VERBS
 
 # A ``list[str]`` field that tolerates the Claude SDK's XML-ish tool-input
 # parsing: an LLM emitting a bullet list as ``<item>…</item>`` elements arrives
@@ -50,7 +51,22 @@ SDK_URL = os.environ.get("ROBOCO_SDK_URL", "http://localhost:9000")
 AGENT_ID = os.environ["ROBOCO_AGENT_ID"]
 AGENT_ROLE = os.environ["ROBOCO_AGENT_ROLE"]
 
-_TIMEOUT = 30
+# Client wall = the matching server wall (FlowVerbTimeoutMiddleware) plus
+# headroom, so the client always outlasts the server's asyncio.timeout and
+# sees the clean 504 gateway_timeout envelope instead of a raw transport
+# timeout. This module can't read Settings (it's a subprocess in the agent
+# container), so the two server budgets are mirrored via env vars the
+# orchestrator injects at spawn from settings.flow_verb_timeout_seconds /
+# flow_verb_slow_timeout_seconds; the literal fallbacks match those settings'
+# own defaults (120 / 900).
+_SERVER_TIMEOUT_SECONDS = float(
+    os.environ.get("ROBOCO_FLOW_VERB_TIMEOUT_SECONDS", "120")
+)
+_SERVER_SLOW_TIMEOUT_SECONDS = float(
+    os.environ.get("ROBOCO_FLOW_VERB_SLOW_TIMEOUT_SECONDS", "900")
+)
+_TIMEOUT = _SERVER_TIMEOUT_SECONDS + CLIENT_HEADROOM_SECONDS
+_SLOW_TIMEOUT = _SERVER_SLOW_TIMEOUT_SECONDS + CLIENT_HEADROOM_SECONDS
 # Tight timeout for SDK loopback — the SDK is a local sidecar; anything
 # slower than 2s is unhealthy and the gateway path must not stall on it.
 _SDK_TIMEOUT = 2.0
@@ -285,6 +301,16 @@ def _build_headers() -> dict[str, str]:
     return headers
 
 
+def _client_timeout_for(verb: str) -> float:
+    """The httpx client timeout for one verb — must outlast its server wall.
+
+    Mirrors ``FlowVerbTimeoutMiddleware``'s own budget selection so the two
+    walls agree: a slow verb gets the slow server budget + headroom, every
+    other verb gets the default budget + headroom.
+    """
+    return _SLOW_TIMEOUT if verb in SLOW_VERBS else _TIMEOUT
+
+
 def _post(path: str, body: dict[str, Any]) -> dict[str, Any]:
     """POST a request to the orchestrator and return the JSON envelope.
 
@@ -301,7 +327,9 @@ def _post(path: str, body: dict[str, Any]) -> dict[str, Any]:
     being returned to the agent — preventing further hammering on a verb
     that won't succeed. Successful (ok) envelopes never touch the SDK.
     """
-    with httpx.Client(timeout=_TIMEOUT) as client:
+    # Client must outlast the server middleware budget so agents get the
+    # 504 envelope, not a raw timeout.
+    with httpx.Client(timeout=_client_timeout_for(_verb_from_path(path))) as client:
         response = client.post(
             f"{ORCHESTRATOR_URL}{path}",
             headers=_build_headers(),

roboco/models/sandbox.py

@@ -169,7 +169,7 @@ class _RedisEngine(SandboxEngine):
 
 class _MongoEngine(SandboxEngine):
     name = "mongo"
-    image = "mongo:8-alpine"
+    image = "mongo:8"
     container_port = 27017
     ready_deadline = 60.0
     tmpfs = ("/data/db",)
@@ -181,8 +181,6 @@ class _MongoEngine(SandboxEngine):
             "MONGO_INITDB_ROOT_USERNAME=sandbox",
             "-e",
             f"MONGO_INITDB_ROOT_PASSWORD={password}",
-            "-e",
-            "MONGO_INITDB_DATABASE=sandbox",
         ]
 
     def run_command(self, _password: str) -> list[str]:

roboco/runtime/orchestrator.py

@@ -3150,6 +3150,14 @@ class AgentOrchestrator:
             "ROBOCO_ORCHESTRATOR_URL": api_url,
             "ROBOCO_AGENT_ID": agent_uuid,
             "ROBOCO_AGENT_ROLE": agent_role,
+            # Mirrors the server-side FlowVerbTimeoutMiddleware budgets so the
+            # roboco-flow MCP client's per-verb timeout (flow_server.py, which
+            # can't read Settings directly) stays coherent with operator
+            # tuning of either setting.
+            "ROBOCO_FLOW_VERB_TIMEOUT_SECONDS": str(settings.flow_verb_timeout_seconds),
+            "ROBOCO_FLOW_VERB_SLOW_TIMEOUT_SECONDS": str(
+                settings.flow_verb_slow_timeout_seconds
+            ),
             # Every MCP server is launched as `uv run python -m
             # roboco.mcp.<server>` by Claude Code, with cwd = the agent's
             # WORKSPACE (not /app). Without this, `uv run` resolves a
@@ -7801,6 +7809,32 @@ Start by:
                 terminal=terminal,
                 error=str(exc),
             )
+            if terminal:
+                await self._notify_video_render_failure(task, str(exc))
+
+    async def _notify_video_render_failure(self, task: Any, last_error: str) -> None:
+        """Send one CEO alert that a video render exhausted its retries.
+
+        Best-effort, mirroring ``_notify_strategy_engine_failure`` — a
+        notification-send failure must never raise out of the render loop.
+        """
+        try:
+            from roboco.services.notification import NotificationService
+
+            await NotificationService().send_ack_notification(
+                from_agent="system",
+                to_agent="ceo",
+                body=(
+                    f"[video engine] render terminally failed for task "
+                    f"{task.title!r} ({_MAX_VIDEO_RENDER_ATTEMPTS} attempts "
+                    f"exhausted): {last_error}"
+                ),
+                task_id=task.id,
+            )
+        except Exception:
+            logger.exception(
+                "video-render failure-notify dropped", task_id=str(task.id)
+            )
 
     async def _render_both_cuts(
         self, db: Any, draft: dict[str, Any], composition_id: str, render_key: str

roboco/services/gateway/quality_gate.py

@@ -91,6 +91,14 @@ async def _run_one(workspace: Path, command: str) -> tuple[int, str]:
         # the process lingers as a transient zombie and the FDs leak.
         await proc.wait()
         return 124, f"command timed out after {_GATE_TIMEOUT_SECONDS}s"
+    except asyncio.CancelledError:
+        # An outer cancellation (e.g. FlowVerbTimeoutMiddleware's own
+        # asyncio.timeout expiring around the whole submit) throws in here
+        # instead of the wait_for's own TimeoutError above — same orphaned
+        # child + leaked FDs if left unkilled. Kill/reap, then propagate.
+        proc.kill()
+        await proc.wait()
+        raise
     rc = proc.returncode
     if rc is None:
         # communicate() returned without a recorded exit code (the process

roboco/services/git.py

@@ -25,6 +25,8 @@ from uuid import UUID
 import httpx
 
 if TYPE_CHECKING:
+    from collections.abc import Coroutine
+
     from sqlalchemy.ext.asyncio import AsyncSession
 
     # `api.schemas.git` would trigger `api/__init__.py` (which historically
@@ -97,6 +99,26 @@ def _commit_git_timeout() -> int:
     return settings.git_commit_timeout_seconds
 
 
+async def _await_shielded[T](coro: Coroutine[Any, Any, T]) -> T:
+    """Run a session write shielded from cancellation, waiting it out.
+
+    A bare ``asyncio.shield(coro)`` detaches the write on cancellation but
+    lets the CancelledError propagate immediately — the still-running write
+    then races ``get_db``'s rollback on the SAME AsyncSession and asyncpg
+    raises ``InterfaceError: another operation is in progress`` (a 500
+    instead of the middleware's clean 504). On cancellation, await the
+    in-flight write to completion BEFORE re-raising, so the session is quiet
+    by the time the rollback runs. Mirrors
+    ``VideoPostService._commit_shielded``.
+    """
+    task = asyncio.ensure_future(coro)
+    try:
+        return await asyncio.shield(task)
+    except asyncio.CancelledError:
+        await asyncio.gather(task, return_exceptions=True)
+        raise
+
+
 def _completed_branchful_children(children: list[Any]) -> list[Any]:
     """Children whose completed work can be integrity-checked: completed,
     branch-bearing, with recorded commits."""
@@ -3546,7 +3568,13 @@ class GitService(BaseService):
             if found:
                 pr_number = int(found["number"])
                 pr_url = str(found["html_url"])
-                await self._record_pr_atomically(UUID(str(task.id)), pr_number, pr_url)
+                # Shielded + waited-out: the PR already exists on GitHub, so
+                # a cancellation here must not skip recording it locally —
+                # and shield alone would leave the detached write racing
+                # get_db's rollback on the same session (see _await_shielded).
+                await _await_shielded(
+                    self._record_pr_atomically(UUID(str(task.id)), pr_number, pr_url)
+                )
                 return {
                     "pr_number": pr_number,
                     "pr_url": pr_url,
@@ -3563,7 +3591,15 @@ class GitService(BaseService):
         pr_data = resp.json()
         pr_number = int(pr_data["number"])
         pr_url = str(pr_data["html_url"])
-        await self._record_pr_atomically(UUID(str(task.id)), pr_number, pr_url)
+        # _post_pr already created the PR on GitHub — shield the local record
+        # so a cancellation landing between the POST and this commit can't
+        # leave it unrecorded (self-heals on retry via _find_existing_pr, but
+        # only after this window closes). Waited-out, not bare shield: the
+        # detached write must finish before get_db's rollback touches the
+        # same session (see _await_shielded).
+        await _await_shielded(
+            self._record_pr_atomically(UUID(str(task.id)), pr_number, pr_url)
+        )
         return {"pr_number": pr_number, "pr_url": pr_url, "is_root_pr": is_root_pr}
 
     async def _lock_parent_task_for_merge(self, parent_task_id: UUID | None) -> None:

roboco/services/learning.py

@@ -323,6 +323,11 @@ class LearningPropagationService:
         # One bulk INSERT — replaces N sequential _create_notification calls
         # each opening their own session/transaction (pool pressure).
         # id set explicitly so delivery can address each row by UUID.
+        # This bypasses NotificationService's dedup guards (_duplicate_unacked_exists
+        # + the delivery-side coalesce) entirely — safe only because KNOWLEDGE_SHARE
+        # is exempt from both (ACK_REQUIRED_BY_TYPE[KNOWLEDGE_SHARE]=False: one-shot,
+        # not ack-required). Reclassifying KNOWLEDGE_SHARE as ack-required must
+        # revisit this path or dedup silently stops applying to it.
         notification_ids = [uuid4() for _ in agents]
         rows = [
             {

roboco/services/video_post_service.py

@@ -483,7 +483,17 @@ class VideoPostService(BaseService):
         )
 
     async def reject(self, task_id: UUID, reason: str) -> TaskTable | None:
-        """Record the CEO's reason and cancel the draft (never posted)."""
+        """Record the CEO's reason and cancel the draft (never posted).
+
+        Acquires the same post-mutex ``approve()`` holds (same key, same
+        non-blocking acquire style) so a reject can't interleave with a
+        concurrent in-flight approve. Fails CLOSED like approve, both when
+        the lock is held (approve mid-post) and when Redis is unreachable:
+        an approve that took the lock while Redis was up stays authoritative
+        through the heartbeat grace window after Redis drops, so an unlocked
+        reject could CANCEL a draft that approve is mid-posting — the CEO
+        retries the reject once Redis is back.
+        """
         task = await get_task_service(self.session).get(task_id)
         if task is None or task.source != VIDEO_POST_SOURCE:
             return None
@@ -491,10 +501,37 @@ class VideoPostService(BaseService):
             raise TaskAlreadyCompletedError(
                 f"video post {task_id} already posted (COMPLETED); cannot be rejected"
             )
-        markers.set_video_reject_reason(task, reason)
-        task.status = TaskStatus.CANCELLED
-        await self.session.flush()
-        return task
+
+        mutex = HeartbeatMutex(
+            f"{_LOCK_PREFIX}{task_id}",
+            ttl_seconds=_LOCK_TTL_SECONDS,
+            heartbeat_seconds=_LOCK_HEARTBEAT_SECONDS,
+        )
+        try:
+            token = await mutex.acquire()
+        except HeartbeatLockUnavailable as exc:
+            logger.error("video-post reject lock unavailable (redis down): %s", exc)
+            return None
+        if token is None:
+            return None  # a concurrent approve is mid-post; refuse the reject
+        try:
+            # Re-read under the lock: a concurrent approve may have posted +
+            # committed COMPLETED between the pre-lock check and here.
+            self.session.expire(task)
+            locked = await get_task_service(self.session).get(task_id)
+            if locked is None:
+                return None
+            if locked.status == TaskStatus.COMPLETED:
+                raise TaskAlreadyCompletedError(
+                    f"video post {task_id} already posted (COMPLETED); "
+                    "cannot be rejected"
+                )
+            markers.set_video_reject_reason(locked, reason)
+            locked.status = TaskStatus.CANCELLED
+            await self.session.flush()
+            return locked
+        finally:
+            await mutex.release(token)
 
 
 def get_video_post_service(

tests/e2e_smoke/test_flow_verb_timeout.py

@@ -212,9 +212,11 @@ def test_flow_verb_holds_lock_when_timeout_disarmed(
     _patch_hang_in_set_plan(monkeypatch)
 
     # Prime the per-agent flow_server reload with a no-op verb (i_am_idle
-    # touches no task). The reload resets module globals (_TIMEOUT=30), so a
-    # pre-call patch would be clobbered; after this call the module is
-    # pinned to this agent and the patch below survives.
+    # touches no task). The reload resets module globals (the env-derived
+    # _TIMEOUT), so a pre-call patch would be clobbered; after this call the
+    # module is pinned to this agent and the patch below survives.
+    # i_will_plan is a default-budget verb (not in SLOW_VERBS), so the
+    # client selects _TIMEOUT for it.
     main_pm.flow("i_am_idle")
     monkeypatch.setattr(flow_server, "_TIMEOUT", _MCP_CLIENT_TIMEOUT_SECONDS)
 

tests/e2e_smoke/test_sandbox_image_tags.py

@@ -0,0 +1,48 @@
+"""Regression guard for the 2026-07-08 ``mongo:8-alpine`` ghost-tag bug.
+
+``roboco/models/sandbox.py`` pinned ``_MongoEngine.image = "mongo:8-alpine"``,
+a tag that has never existed on Docker Hub (MongoDB ships no Alpine variant).
+Every unit test mocks the docker CLI, so none of them ever touch a real
+registry and none caught it — the bug only surfaces the moment a real
+``docker run`` pulls the image. This test queries the Docker Hub registry API
+for every ``SANDBOX_ENGINES`` entry's pinned ``image:tag`` and fails if the
+tag does not actually exist, which is the check that would have caught it.
+
+Network-dependent by design; skips cleanly when the registry is unreachable
+rather than failing (mirrors ``test_background_engines.py``'s local-Redis
+reachability skip).
+"""
+
+from __future__ import annotations
+
+import httpx
+import pytest
+from roboco.models.sandbox import SANDBOX_ENGINES
+
+_REGISTRY_URL = (
+    "https://registry.hub.docker.com/v2/repositories/library/{name}/tags/{tag}"
+)
+_TIMEOUT_SECONDS = 10.0
+_HTTP_OK = 200
+
+
+def _split_image(image: str) -> tuple[str, str]:
+    name, _, tag = image.partition(":")
+    return name, tag or "latest"
+
+
+@pytest.mark.parametrize("engine_name", sorted(SANDBOX_ENGINES))
+def test_sandbox_engine_image_tag_exists_on_docker_hub(engine_name: str) -> None:
+    image = SANDBOX_ENGINES[engine_name].image
+    name, tag = _split_image(image)
+    url = _REGISTRY_URL.format(name=name, tag=tag)
+
+    try:
+        resp = httpx.get(url, timeout=_TIMEOUT_SECONDS)
+    except httpx.TransportError:
+        pytest.skip(f"Docker Hub registry unreachable, cannot verify {image!r}")
+
+    assert resp.status_code == _HTTP_OK, (
+        f"{engine_name}: pinned image {image!r} not found on Docker Hub "
+        f"(library/{name}, tag {tag!r}, status {resp.status_code}) — {url}"
+    )

tests/integration/test_video_routes.py

@@ -460,9 +460,10 @@ async def test_reject_cancels_and_records_reason(
     db_session: AsyncSession, ceo_client: AsyncClient
 ) -> None:
     task = await _seed_draft(db_session)
-    resp = await ceo_client.post(
-        f"/api/video/posts/{task.id}/reject", json={"reason": "Not our voice"}
-    )
+    with _LOCKED[0], _LOCKED[1]:
+        resp = await ceo_client.post(
+            f"/api/video/posts/{task.id}/reject", json={"reason": "Not our voice"}
+        )
     assert resp.status_code == HTTPStatus.OK
     assert resp.json()["reject_reason"] == "Not our voice"
     refreshed = await db_session.get(TaskTable, task.id)

tests/unit/api/test_middleware.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import asyncio
 from http import HTTPStatus
 from typing import Any
 
@@ -18,6 +19,7 @@ from roboco.api.middleware import (
     get_status_code,
     setup_middleware,
 )
+from roboco.config import settings
 from roboco.exceptions import (
     AuthenticationError,
     InvalidStateError,
@@ -417,3 +419,53 @@ def test_request_validation_handler_log_preserves_non_secret_fields() -> None:
     assert logged_body["title"] == "visible-title"  # non-secret preserved
     assert logged_body["git_token"] == "***REDACTED***"  # secret redacted
     assert "ghp_secret_xyz" not in str(logged_body)
+
+
+# ---------------------------------------------------------------------------
+# FlowVerbTimeoutMiddleware — per-verb budget selection
+# ---------------------------------------------------------------------------
+
+
+def _make_flow_app() -> FastAPI:
+    """Two /api/v1/flow/* routes that each sleep past the fast budget but
+    under the slow one, so the picked timeout is observable by outcome."""
+    app = FastAPI()
+
+    @app.post("/api/v1/flow/developer/give_me_work")
+    async def _normal_verb() -> Any:
+        await asyncio.sleep(0.3)
+        return {"status": "ok"}
+
+    @app.post("/api/v1/flow/developer/i_am_done")
+    async def _slow_verb() -> Any:
+        await asyncio.sleep(0.3)
+        return {"status": "ok"}
+
+    setup_middleware(app)
+    return app
+
+
+def test_flow_verb_timeout_normal_verb_uses_default_budget(
+    monkeypatch: Any,
+) -> None:
+    """A verb outside _SLOW_VERBS keeps the short default budget — a 0.3s
+    handler exceeds a 0.05s budget and comes back as a 504."""
+    monkeypatch.setattr(settings, "flow_verb_timeout_seconds", 0.05)
+    monkeypatch.setattr(settings, "flow_verb_slow_timeout_seconds", 5)
+
+    client = TestClient(_make_flow_app())
+    response = client.post("/api/v1/flow/developer/give_me_work")
+    assert response.status_code == HTTPStatus.GATEWAY_TIMEOUT
+    assert response.json()["error"] == "gateway_timeout"
+
+
+def test_flow_verb_timeout_slow_verb_uses_slow_budget(monkeypatch: Any) -> None:
+    """A _SLOW_VERBS verb gets the longer budget — the same 0.3s handler that
+    times out on the default budget completes fine under the slow one."""
+    monkeypatch.setattr(settings, "flow_verb_timeout_seconds", 0.05)
+    monkeypatch.setattr(settings, "flow_verb_slow_timeout_seconds", 5)
+
+    client = TestClient(_make_flow_app())
+    response = client.post("/api/v1/flow/developer/i_am_done")
+    assert response.status_code == HTTPStatus.OK
+    assert response.json() == {"status": "ok"}

tests/unit/gateway/test_quality_gate.py

@@ -5,9 +5,10 @@ i_am_done, blocking a red submit before it reaches QA. Full tests stay on CI.
 from __future__ import annotations
 
 import asyncio
+import os
 from types import SimpleNamespace
-from typing import TYPE_CHECKING
-from unittest.mock import AsyncMock, MagicMock
+from typing import TYPE_CHECKING, Any
+from unittest.mock import AsyncMock, MagicMock, patch
 
 if TYPE_CHECKING:
     import pathlib
@@ -128,6 +129,40 @@ async def test_run_one_reaps_killed_timeout_process(
     fake_proc.wait.assert_awaited_once()
 
 
+@pytest.mark.asyncio
+async def test_run_one_kills_child_on_outer_cancellation(
+    tmp_path: pathlib.Path,
+) -> None:
+    """An outer cancellation (e.g. FlowVerbTimeoutMiddleware's own
+    asyncio.timeout firing around the whole i_am_done submit) throws
+    CancelledError into the wait_for, bypassing the TimeoutError handler
+    above. Without a dedicated handler the child is orphaned and keeps
+    running past the cancelled request; _run_one must kill + reap it and
+    re-raise.
+    """
+    real_create_subprocess_shell = asyncio.create_subprocess_shell
+    spawned: dict[str, asyncio.subprocess.Process] = {}
+
+    async def _capturing_create(*args: Any, **kwargs: Any) -> Any:
+        proc = await real_create_subprocess_shell(*args, **kwargs)
+        spawned["proc"] = proc
+        return proc
+
+    with patch.object(asyncio, "create_subprocess_shell", _capturing_create):
+        task = asyncio.ensure_future(quality_gate._run_one(tmp_path, "sleep 30"))
+        while "proc" not in spawned:
+            await asyncio.sleep(0.01)
+        await asyncio.sleep(0.1)  # let the shell actually exec sleep
+        task.cancel()
+        with pytest.raises(asyncio.CancelledError):
+            await task
+
+    proc = spawned["proc"]
+    assert proc.returncode is not None, "child was not reaped after cancellation"
+    with pytest.raises(ProcessLookupError):
+        os.kill(proc.pid, 0)
+
+
 # --- GitService command selection -------------------------------------------
 
 

tests/unit/mcp_servers/test_do_server.py

@@ -52,13 +52,18 @@ def test_commit_posts_message_and_files(do_module: Any) -> None:
     fake_response.json.return_value = {"status": "in_progress", "task_id": "x"}
     fake_client.post.return_value = fake_response
 
-    with patch("httpx.Client", return_value=fake_client):
+    with patch("httpx.Client", return_value=fake_client) as client_cls:
         result = do_module.commit("feat(api): add /healthz", files=["foo.py"])
 
     assert result["status"] == "in_progress"
     args, kwargs = fake_client.post.call_args
     assert "/api/v1/do/commit" in args[0]
     assert kwargs["json"] == {"message": "feat(api): add /healthz", "files": ["foo.py"]}
+    # commit stages+commits a large changeset server-side (up to
+    # git_commit_timeout_seconds, default 180s) — the shared _TIMEOUT (30s)
+    # is tuned for fast content-tool calls and would give up first.
+    assert client_cls.call_args.kwargs["timeout"] == do_module._COMMIT_TIMEOUT
+    assert do_module._COMMIT_TIMEOUT > do_module._TIMEOUT
 
 
 def test_note_default_scope_note(do_module: Any) -> None:

tests/unit/mcp_servers/test_flow_server.py

@@ -504,6 +504,80 @@ def test_escalate_up_passes_reason(monkeypatch: pytest.MonkeyPatch) -> None:
     }
 
 
+# ---------------------------------------------------------------------------
+# Client-side timeout selection — must always outlast the matching server
+# wall (FlowVerbTimeoutMiddleware) so the agent sees the clean 504 envelope
+# instead of a raw httpx timeout. See roboco.foundation.policy.flow_timeouts.
+# ---------------------------------------------------------------------------
+
+
+def test_client_timeout_normal_verb_is_default_plus_headroom(
+    flow_module: types.ModuleType,
+) -> None:
+    assert flow_module._TIMEOUT == (
+        flow_module._SERVER_TIMEOUT_SECONDS + flow_module.CLIENT_HEADROOM_SECONDS
+    )
+    assert flow_module._client_timeout_for("give_me_work") == flow_module._TIMEOUT
+
+
+def test_client_timeout_slow_verb_is_slow_budget_plus_headroom(
+    flow_module: types.ModuleType,
+) -> None:
+    assert flow_module._SLOW_TIMEOUT == (
+        flow_module._SERVER_SLOW_TIMEOUT_SECONDS + flow_module.CLIENT_HEADROOM_SECONDS
+    )
+    assert flow_module._client_timeout_for("i_am_done") == flow_module._SLOW_TIMEOUT
+    # Every SLOW_VERBS member routes through the same slow budget.
+    for verb in flow_module.SLOW_VERBS:
+        assert flow_module._client_timeout_for(verb) == flow_module._SLOW_TIMEOUT
+
+
+def test_client_timeout_env_override_respected(
+    monkeypatch: pytest.MonkeyPatch, tmp_path: Path
+) -> None:
+    manifest_path = tmp_path / "tool-manifest.json"
+    manifest_path.write_text(json.dumps(_FULL_MANIFEST))
+    monkeypatch.setenv("ROBOCO_AGENT_ID", "00000000-0000-0000-0000-000000000001")
+    monkeypatch.setenv("ROBOCO_AGENT_ROLE", "developer")
+    monkeypatch.setenv("ROBOCO_ORCHESTRATOR_URL", "http://test-orchestrator:8000")
+    monkeypatch.setenv("ROBOCO_TOOL_MANIFEST_PATH", str(manifest_path))
+    monkeypatch.setenv("ROBOCO_FLOW_VERB_TIMEOUT_SECONDS", "45")
+    monkeypatch.setenv("ROBOCO_FLOW_VERB_SLOW_TIMEOUT_SECONDS", "600")
+
+    import roboco.mcp.flow_server as srv
+
+    importlib.reload(srv)
+    try:
+        assert srv._TIMEOUT == 45 + srv.CLIENT_HEADROOM_SECONDS
+        assert srv._SLOW_TIMEOUT == 600 + srv.CLIENT_HEADROOM_SECONDS
+        assert srv._client_timeout_for("give_me_work") == srv._TIMEOUT
+        assert srv._client_timeout_for("i_am_done") == srv._SLOW_TIMEOUT
+    finally:
+        importlib.reload(srv)  # restore module state for later tests
+
+
+def test_post_opens_httpx_client_with_slow_timeout_for_slow_verb(
+    flow_module: types.ModuleType,
+) -> None:
+    fake_client = _make_fake_client({"status": "awaiting_qa"})
+
+    with patch("httpx.Client", return_value=fake_client) as client_cls:
+        flow_module.i_am_done("task-abc", notes="done")
+
+    assert client_cls.call_args.kwargs["timeout"] == flow_module._SLOW_TIMEOUT
+
+
+def test_post_opens_httpx_client_with_default_timeout_for_normal_verb(
+    flow_module: types.ModuleType,
+) -> None:
+    fake_client = _make_fake_client({"status": "idle"})
+
+    with patch("httpx.Client", return_value=fake_client) as client_cls:
+        flow_module.give_me_work()
+
+    assert client_cls.call_args.kwargs["timeout"] == flow_module._TIMEOUT
+
+
 def test_escalate_to_ceo_passes_reason(monkeypatch: pytest.MonkeyPatch) -> None:
     """Board / Main PM verb forwards to /api/v1/flow/<role>/escalate_to_ceo."""
     srv = _reload_for_role(

tests/unit/runtime/test_sandbox_provisioner.py

@@ -128,7 +128,7 @@ async def test_provision_mongo_engine() -> None:
     assert mongo.user == "sandbox"
     assert mongo.database == "admin"
     run_call = next(c for c in runner.calls if c[0] == "run")
-    assert "mongo:8-alpine" in run_call
+    assert "mongo:8" in run_call
     # MONGO_INITDB_ROOT_PASSWORD env is baked into the run.
     assert any(a.startswith("MONGO_INITDB_ROOT_PASSWORD=") for a in run_call)
     # /data/db tmpfs mount for the engine.

tests/unit/runtime/test_spawn_strict_mcp.py

@@ -14,6 +14,7 @@ from pathlib import Path
 from unittest.mock import patch
 
 import pytest
+from roboco.config import settings
 from roboco.models.runtime import OrchestratorAgentConfig, SpawnGitContext
 from roboco.runtime.orchestrator import AgentOrchestrator
 
@@ -102,3 +103,19 @@ class TestMcpConfigPinsBakedVenv:
                 f"a drifted workspace-clone lock can't trigger a resync stall; "
                 f"got args={spec['args']}"
             )
+
+    @pytest.mark.asyncio
+    async def test_mcp_env_mirrors_flow_verb_timeout_settings(self) -> None:
+        """flow_server.py (a subprocess, can't read Settings) mirrors the two
+        server-side flow-verb timeout budgets via env so its client timeout
+        stays coherent with operator tuning of either setting."""
+        orch = AgentOrchestrator.__new__(AgentOrchestrator)
+        config_path = await orch._generate_mcp_config("be-dev-1")
+        config = json.loads(Path(config_path).read_text())
+        env = config["mcpServers"]["roboco-flow"]["env"]
+        assert env["ROBOCO_FLOW_VERB_TIMEOUT_SECONDS"] == str(
+            settings.flow_verb_timeout_seconds
+        )
+        assert env["ROBOCO_FLOW_VERB_SLOW_TIMEOUT_SECONDS"] == str(
+            settings.flow_verb_slow_timeout_seconds
+        )

tests/unit/runtime/test_video_render_loop.py

@@ -431,7 +431,15 @@ async def test_render_video_task_terminal_after_max_attempts(
     workspace = _fake_workspace()
     orch = _orch()
     p1, p2 = _render_patches(renderer, workspace)
-    with p1, p2:
+    notify_svc = AsyncMock()
+    with (
+        p1,
+        p2,
+        patch(
+            "roboco.services.notification.NotificationService",
+            return_value=notify_svc,
+        ),
+    ):
         await orch._render_video_task(db_session, task)  # tips to terminal
         calls_at_terminal = len(renderer.calls)
         await orch._render_video_task(db_session, task)  # now a no-op
@@ -443,3 +451,47 @@ async def test_render_video_task_terminal_after_max_attempts(
     assert len(renderer.calls) == calls_at_terminal  # not retried after terminal
     posts = await get_task_service(db_session).list_open_video_posts()
     assert posts == []
+    # Exactly one CEO alert — the second (no-op) call must not re-notify.
+    notify_svc.send_ack_notification.assert_awaited_once()
+    notify_kwargs = notify_svc.send_ack_notification.await_args.kwargs
+    assert notify_kwargs["to_agent"] == "ceo"
+    assert task.title in notify_kwargs["body"]
+    assert "render blew up" in notify_kwargs["body"]
+    assert notify_kwargs["task_id"] == task.id
+
+
+@pytest.mark.asyncio
+async def test_render_video_task_notify_failure_does_not_raise(
+    db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    """A broken notification path (e.g. the second DB connection is down)
+    must not surface out of the render loop — best-effort, like the
+    strategy-engine failure notifier."""
+    await _seed(db_session)
+    _enable(monkeypatch)
+    task = await _make_completed_video_task(
+        db_session, occasion="notify-fails", composition_id="Intro"
+    )
+    seeded = markers.get_video_draft(task) or {}
+    markers.set_video_draft(
+        task, {**seeded, "render_attempts": _MAX_VIDEO_RENDER_ATTEMPTS - 1}
+    )
+    await db_session.flush()
+
+    renderer = _FakeRenderer(fail=True)
+    workspace = _fake_workspace()
+    orch = _orch()
+    p1, p2 = _render_patches(renderer, workspace)
+    with (
+        p1,
+        p2,
+        patch(
+            "roboco.services.notification.NotificationService",
+            side_effect=RuntimeError("notification DB unreachable"),
+        ),
+    ):
+        await orch._render_video_task(db_session, task)  # must not raise
+
+    draft = markers.get_video_draft(task)
+    assert draft is not None
+    assert draft["render_status"] == "failed"

tests/unit/services/test_git.py

@@ -7,6 +7,7 @@ mock the network and filesystem boundaries.
 
 from __future__ import annotations
 
+import asyncio
 from pathlib import Path
 from typing import TYPE_CHECKING
 from unittest.mock import AsyncMock, MagicMock, patch
@@ -512,6 +513,122 @@ async def test_create_pr_returns_pr_dict() -> None:
     assert out["is_root_pr"] is True
 
 
+@pytest.mark.asyncio
+async def test_create_pr_records_pr_despite_cancellation_after_post() -> None:
+    """A cancellation landing after the GitHub POST succeeds but before the
+    local record commits must not lose the record: asyncio.shield lets
+    _record_pr_atomically finish, the cancellation still propagates."""
+    project_id = uuid4()
+    fake_task = MagicMock(
+        id=uuid4(),
+        project_id=project_id,
+        assigned_to=uuid4(),
+        title="Add login",
+        description="A short description",
+    )
+    fake_project = MagicMock(slug="roboco")
+    svc = _service()
+    _bind(svc, "_task_for_branch", AsyncMock(return_value=fake_task))
+    _bind(svc, "_workspace_for_branch", AsyncMock(return_value=Path("/tmp/ws")))
+    _bind(svc, "_get_project_token_or_raise", AsyncMock(return_value="tok"))
+    _bind(svc, "_parse_github_remote", MagicMock(return_value=("acme", "repo")))
+    _bind(svc, "_project_default_branch", AsyncMock(return_value="master"))
+
+    recorded = {"done": False}
+
+    async def _slow_record(*_args: object, **_kwargs: object) -> None:
+        await asyncio.sleep(0.05)
+        recorded["done"] = True
+
+    _bind(svc, "_record_pr_atomically", _slow_record)
+
+    fake_resp = MagicMock()
+    fake_resp.is_success = True
+    fake_resp.status_code = 201
+    fake_resp.json.return_value = {
+        "number": _EXPECTED_PR_NUMBER,
+        "html_url": f"https://github.com/acme/repo/pull/{_EXPECTED_PR_NUMBER}",
+    }
+    _bind(svc, "_post_pr", AsyncMock(return_value=fake_resp))
+
+    with _patch_project_service(fake_project):
+        task = asyncio.ensure_future(
+            svc.create_pr("feature/backend/abc12345", parent="master", is_root_pr=True)
+        )
+        await asyncio.sleep(0.01)  # let create_pr reach the shielded await
+        task.cancel()
+        with pytest.raises(asyncio.CancelledError):
+            await task
+
+    # _await_shielded waits the in-flight record out BEFORE re-raising, so
+    # by the time `await task` raised, the record had already completed.
+    assert recorded["done"] is True, (
+        "shield must let the record finish despite cancellation"
+    )
+
+
+@pytest.mark.asyncio
+async def test_create_pr_cancellation_waits_out_record_before_reraising() -> None:
+    """Ordering guard for _await_shielded: on cancellation the in-flight
+    _record_pr_atomically must run to COMPLETION before CancelledError
+    re-raises to the caller. A bare asyncio.shield detaches the write and
+    re-raises immediately — the write then races get_db's rollback on the
+    same AsyncSession and asyncpg raises InterfaceError ('another operation
+    is in progress'), escaping as a 500 instead of the middleware's 504."""
+    project_id = uuid4()
+    fake_task = MagicMock(
+        id=uuid4(),
+        project_id=project_id,
+        assigned_to=uuid4(),
+        title="Add login",
+        description="A short description",
+    )
+    fake_project = MagicMock(slug="roboco")
+    svc = _service()
+    _bind(svc, "_task_for_branch", AsyncMock(return_value=fake_task))
+    _bind(svc, "_workspace_for_branch", AsyncMock(return_value=Path("/tmp/ws")))
+    _bind(svc, "_get_project_token_or_raise", AsyncMock(return_value="tok"))
+    _bind(svc, "_parse_github_remote", MagicMock(return_value=("acme", "repo")))
+    _bind(svc, "_project_default_branch", AsyncMock(return_value="master"))
+
+    order: list[str] = []
+
+    async def _slow_record(*_args: object, **_kwargs: object) -> None:
+        await asyncio.sleep(0.05)
+        order.append("record_done")
+
+    _bind(svc, "_record_pr_atomically", _slow_record)
+
+    fake_resp = MagicMock()
+    fake_resp.is_success = True
+    fake_resp.status_code = 201
+    fake_resp.json.return_value = {
+        "number": _EXPECTED_PR_NUMBER,
+        "html_url": f"https://github.com/acme/repo/pull/{_EXPECTED_PR_NUMBER}",
+    }
+    _bind(svc, "_post_pr", AsyncMock(return_value=fake_resp))
+
+    with _patch_project_service(fake_project):
+        task = asyncio.ensure_future(
+            svc.create_pr("feature/backend/abc12345", parent="master", is_root_pr=True)
+        )
+        await asyncio.sleep(0.01)  # mid-record: cancellation lands in the shield
+        task.cancel()
+        # pytest.raises re-raises any OTHER exception type (e.g. the
+        # InterfaceError a racing rollback would surface), failing the test —
+        # that is assertion (a): CancelledError and nothing else propagates.
+        with pytest.raises(asyncio.CancelledError):
+            await task
+        # Appended synchronously right after propagation: no other coroutine
+        # can run in between, so this marker coming SECOND proves the record
+        # coroutine had already completed before the cancellation re-raised.
+        order.append("cancel_raised")
+
+    assert order == ["record_done", "cancel_raised"], (
+        f"record must complete BEFORE the cancellation re-raises; got {order}"
+    )
+
+
 @pytest.mark.asyncio
 async def test_create_pr_raises_when_branch_not_found() -> None:
     svc = _service()

tests/unit/services/test_video_post_service.py

@@ -500,14 +500,68 @@ async def test_approve_unknown_task_returns_none(db_session: AsyncSession) -> No
 @pytest.mark.asyncio
 async def test_reject_records_reason_and_cancels(db_session: AsyncSession) -> None:
     task = await _seed_video_post(db_session)
-    updated = await _svc(
-        db_session, x_poster=_StubXPoster(), tiktok_poster=_StubTikTokPoster()
-    ).reject(_id(task), "Doesn't match the release")
+    with _LOCKED[0], _LOCKED[1]:
+        updated = await _svc(
+            db_session, x_poster=_StubXPoster(), tiktok_poster=_StubTikTokPoster()
+        ).reject(_id(task), "Doesn't match the release")
     assert updated is not None
     assert updated.status == TS.CANCELLED
     assert markers.get_video_reject_reason(updated) == "Doesn't match the release"
 
 
+@pytest.mark.asyncio
+async def test_reject_takes_the_same_lock_approve_holds(
+    db_session: AsyncSession,
+) -> None:
+    """A reject under the real acquire/release path still lands (mirrors the
+    approve happy-path locking) — proves the mutex round-trip, not just the
+    mutation."""
+    task = await _seed_video_post(db_session)
+    with _LOCKED[0], _LOCKED[1]:
+        updated = await _svc(
+            db_session, x_poster=_StubXPoster(), tiktok_poster=_StubTikTokPoster()
+        ).reject(_id(task), "Doesn't match the release")
+    assert updated is not None
+    assert updated.status == TS.CANCELLED
+    _LOCKED[0].new.assert_awaited()
+    _LOCKED[1].new.assert_awaited()
+
+
+@pytest.mark.asyncio
+async def test_reject_concurrent_lock_held_refuses(db_session: AsyncSession) -> None:
+    """A reject arriving while a concurrent approve holds the lock must not
+    cancel a draft that may be mid-post — same refusal as approve's own
+    already-in-progress case."""
+    task = await _seed_video_post(db_session)
+    with patch.object(HeartbeatMutex, "acquire", AsyncMock(return_value=None)):
+        result = await _svc(
+            db_session, x_poster=_StubXPoster(), tiktok_poster=_StubTikTokPoster()
+        ).reject(_id(task), "Doesn't match the release")
+    assert result is None
+    await db_session.refresh(task)
+    assert task.status == TS.PENDING  # never cancelled while the lock was held
+
+
+@pytest.mark.asyncio
+async def test_reject_redis_unavailable_refuses(db_session: AsyncSession) -> None:
+    """Reject fails CLOSED when Redis is unreachable, mirroring approve: an
+    approve that took the lock while Redis was up stays authoritative through
+    the heartbeat grace window after Redis drops, so an unlocked reject could
+    CANCEL a draft that approve is mid-posting. The CEO retries once Redis is
+    back."""
+    task = await _seed_video_post(db_session)
+    broken = MagicMock()
+    broken.set = AsyncMock(side_effect=ConnectionError("redis down"))
+    broken.aclose = AsyncMock()
+    with patch("roboco.services.heartbeat_mutex.redis.from_url", return_value=broken):
+        result = await _svc(
+            db_session, x_poster=_StubXPoster(), tiktok_poster=_StubTikTokPoster()
+        ).reject(_id(task), "Doesn't match the release")
+    assert result is None
+    await db_session.refresh(task)
+    assert task.status == TS.PENDING  # never cancelled without the mutex
+
+
 @pytest.mark.asyncio
 async def test_list_held_video_posts_excludes_terminal(
     db_session: AsyncSession,
@@ -515,7 +569,8 @@ async def test_list_held_video_posts_excludes_terminal(
     open_task = await _seed_video_post(db_session)
     rejected_task = await _seed_video_post(db_session)
     svc = _svc(db_session, x_poster=_StubXPoster(), tiktok_poster=_StubTikTokPoster())
-    await svc.reject(_id(rejected_task), "not relevant")
+    with _LOCKED[0], _LOCKED[1]:
+        await svc.reject(_id(rejected_task), "not relevant")
     held = await svc.list_held_video_posts()
     ids = {t.id for t in held}
     assert open_task.id in ids

video-renderer/render.js

@@ -20,15 +20,61 @@ const DIMENSIONS = {
   square: { width: 1080, height: 1080 },
 };
 
+// Caps the DECOMPRESSED size (a gzip bomb inflates a tiny upload into a huge
+// tar stream); MAX_UPLOAD_BYTES in server.js only bounds the compressed
+// bytes on the wire.
+const MAX_EXTRACTED_BYTES = Number(
+  process.env.MAX_EXTRACTED_BYTES ?? 512 * 1024 * 1024,
+);
+
+/** Thrown when the tar stream's cumulative entry size crosses
+ * MAX_EXTRACTED_BYTES — server.js maps this to a 413. */
+export class ExtractedSizeExceededError extends Error {
+  constructor(maxBytes) {
+    super(`extracted archive exceeds ${maxBytes} byte cap`);
+    this.name = "ExtractedSizeExceededError";
+    this.statusCode = 413;
+  }
+}
+
 async function extractTar(tarBuffer, destDir) {
   await new Promise((resolve, reject) => {
-    const extractor = tar.extract({ cwd: destDir });
+    let extractedBytes = 0;
+    // ponytail: header-declared entry.size, summed per entry via onentry —
+    // not a byte-exact streaming cap, but tar headers carry the true
+    // (post-gunzip) size, so this catches a bomb before most of it lands.
+    const extractor = tar.extract({
+      cwd: destDir,
+      onentry: (entry) => {
+        extractedBytes += entry.size;
+        if (extractedBytes > MAX_EXTRACTED_BYTES) {
+          extractor.destroy(new ExtractedSizeExceededError(MAX_EXTRACTED_BYTES));
+        }
+      },
+    });
     extractor.on("finish", resolve);
     extractor.on("error", reject);
     Readable.from(tarBuffer).pipe(extractor);
   });
 }
 
+// Deliberately under the orchestrator's 600s client-side HTTP timeout, so
+// this fires first and the caller gets a clean error instead of an abandoned
+// connection while Chrome is still wedged server-side.
+const RENDER_TIMEOUT_SECONDS = Number(
+  process.env.RENDER_TIMEOUT_SECONDS ?? 570,
+);
+
+/** Thrown when a render exceeds RENDER_TIMEOUT_SECONDS — server.js maps
+ * this to a 500 and then hard-exits the process (see server.js for why). */
+export class RenderTimeoutError extends Error {
+  constructor(seconds) {
+    super(`render exceeded ${seconds}s timeout`);
+    this.name = "RenderTimeoutError";
+    this.statusCode = 500;
+  }
+}
+
 /** Thrown when the requested orientation's HTML file isn't present in the
  * composition dir — server.js maps this to a 400 instead of the generic 500
  * a deep-in-executeRenderJob failure would otherwise produce. The known
@@ -115,17 +161,31 @@ export async function renderComposition({
       height,
       fps: FPS,
     });
+    let timer;
     try {
-      await executeRenderJob(job, (progress) => {
-        console.log(
-          `hyperframes-renderer: ${compositionId}/${orientation} ${Math.round(
-            progress.percent * 100,
-          )}%`,
+      const timeout = new Promise((_resolve, reject) => {
+        timer = setTimeout(
+          () => reject(new RenderTimeoutError(RENDER_TIMEOUT_SECONDS)),
+          RENDER_TIMEOUT_SECONDS * 1000,
         );
       });
+      await Promise.race([
+        executeRenderJob(job, (progress) => {
+          console.log(
+            `hyperframes-renderer: ${compositionId}/${orientation} ${Math.round(
+              progress.percent * 100,
+            )}%`,
+          );
+        }),
+        timeout,
+      ]);
     } catch (err) {
       await rm(outDir, { recursive: true, force: true }).catch(() => {});
       throw err;
+    } finally {
+      // Clear on both success AND timeout-throw so a completed render never
+      // leaves a dangling timer that fires the watchdog's exit path late.
+      clearTimeout(timer);
     }
 
     return {

video-renderer/server.js

@@ -10,7 +10,12 @@ import express from "express";
 import multer from "multer";
 import rateLimit from "express-rate-limit";
 import { createReadStream } from "node:fs";
-import { renderComposition, UnknownCompositionError } from "./render.js";
+import {
+  renderComposition,
+  ExtractedSizeExceededError,
+  RenderTimeoutError,
+  UnknownCompositionError,
+} from "./render.js";
 
 const PORT = Number(process.env.PORT ?? 3001);
 
@@ -123,10 +128,24 @@ app.post("/render", renderLimiter, upload.single("source"), async (req, res) =>
     });
     stream.pipe(res);
   } catch (err) {
-    if (err instanceof UnknownCompositionError) {
+    if (
+      err instanceof UnknownCompositionError ||
+      err instanceof ExtractedSizeExceededError
+    ) {
       res.status(err.statusCode).json({ error: err.message });
       return;
     }
+    if (err instanceof RenderTimeoutError) {
+      console.error("video-renderer: render timed out", err.message);
+      res.status(500).json({ error: err.message });
+      // ponytail: Promise.race in render.js abandons the wait but can't
+      // cancel a wedged headless-Chrome render tree — hard-exiting this
+      // process is the only reliable kill. Docker's restart policy brings
+      // up a clean container; wait for the response to flush first so the
+      // caller still gets the 500 instead of a dropped connection.
+      res.on("finish", () => process.exit(1));
+      return;
+    }
     const message = err instanceof Error ? err.message : String(err);
     console.error("video-renderer: render failed", message);
     res.status(500).json({ error: `render failed: ${message}` });