feat(sandbox): pluggable per-engine registry (postgres/redis/mongo) (#324)

* feat(sandbox): pluggable per-engine registry (postgres/redis/mongo)

Replaces the hardcoded postgres+redis branches in the provisioner and the
env emitter with a registry of SandboxEngine specs (image, run args,
readiness probe, connection, ROBOCO_TEST_* env) in a pure low module
(roboco/models/sandbox.py). VALID_SANDBOX_SERVICES is derived from the
registry — single source of truth — and the provisioner + orchestrator
iterate it, so adding an engine is one class + one registry line, not
another branch. Adds a mongo:8-alpine engine (ROBOCO_TEST_MONGO_*) as the
third service alongside postgres/redis.

Also fixes the cold-pull loop that stranded v0.19.0 board agents with
empty error strings: docker run pulled inline under a 20s deadline, so a
NAS cold pull was killed, cancelled, and re-pulled from scratch forever.
_ensure_image now inspects + pulls (300s) before run; provisioning errors
log type+message so a bare TimeoutError no longer shows as "".

Panel edit-project dialog: postgres/redis toggles -> a Set<string>
multi-select driven by a SANDBOX_SERVICES catalog, so new engines appear
in the UI by adding to the catalog.

Tests: engine parity (allowlist==registry, unique slugs/images, no None
leak in env, SandboxInfo aggregates every engine), mongo provision + env
injection, plus the existing postgres/redis provision/env/spawn/janitor
suite updated to the registry shape. 821 unit / 5 skip green; ruff + mypy
(360 files) clean.

* docs(sandbox): reflect pluggable engine registry + mongo across docs

CHANGELOG (0.19.0): Added entry for the pluggable sandbox engine registry
(postgres/redis/mongo) + Fixed entry for the cold-pull loop/empty-error
strand that boarded v0.19.0 board agents.

docs/map (9 files): sandbox subsystem blurbs, SandboxProvisioner rows,
_maybe_provision_sandbox/_append_sandbox_env rows, feature-flag rows, the
migration-057 row + v0.17.0 delta, and the models.md VALID_SANDBOX_SERVICES
note — all retitled to DB/Redis/Mongo via the engine registry
(roboco/models/sandbox.py), with the one-class-one-line extension story and
the _ensure_image cold-pull fix. Production-network (roboco_data) lines left
as postgres+redis — mongo is sandbox-only, not a prod service.

docs/rag (3 files): sandbox-db.md rewritten around the registry (engine list,
generic _provision_engine, image pre-pull, ROBOCO_TEST_DB_*/REDIS_*/MONGO_*
incl. MONGO_AUTH_DB=admin, single emit_env); config-reference sandbox flag
row + subsection retitled; db-network-isolation framing broadened to
postgres/redis/mongo. preconditions-and-rejections left untouched (its hit
was an unrelated gateway see-also link).

* test(e2e): harden umbrella close terminal reads with bounded wait-for-state

The MegaTask umbrella close test flaked once on CI (ceo-approve returned
200 but the re-fetch saw awaiting_pm_review) then passed on re-run. The
production path is deterministic: complete -> main_pm_complete ->
submit_pm_review -> escalate_to_ceo -> ceo_approve -> commit, all on one
session, all awaited; the fire-and-forget completion hooks are isolated
(own session, best-effort, never touch task.status or the request session).
20 local runs could not reproduce it.

The one real surface is the read pattern: the e2e stack commits on the
uvicorn thread's loop and reads via a separate loop (run_db -> asyncio.run
with a fresh engine), so a terminal single point-read can race a
still-draining completion hook on a contended runner. Replace the two
terminal point-reads with a bounded wait_for_status poll. Strictly better
than a one-shot read: absorbs the transient, and a genuine state bug still
surfaces via the timeout branch asserting against the last-read state.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>

docs/rag/architecture/config-reference.md

@@ -86,7 +86,7 @@ Env-gated subsystems. Most are default-off; `ROBOCO_OVERLOAD_BREAK_ENABLED`, `RO
 | `ROBOCO_PR_GATE_AUTO_SUBMIT_ENABLED` | `true` | PR-gate turn cut: when every child of an assembled parent is terminal, run the real `submit_up` / `submit_root` system-side as the owning PM (`_try_auto_submit`) instead of spawning the PM for that turn — the submit's substance (freshness rebase, integrity check, PR open) is deterministic gate code. A gate rejection falls back to the classic PM closure spawn; the PM keeps the judgment turns (merge, revision). Each auto-submit leaves a `task.auto_submitted` audit row. Off = every closure spawns the PM to submit. |
 | `ROBOCO_SPAWN_PREFLIGHT_ENABLED` | `false` | Refuse to spawn a non-human delivery role absent from `GATEWAY_ENABLED_ROLES` (no manifest → can never claim → would respawn on the same task forever); refuse + alert the overseer once instead. Inert in practice (all delivery roles are gateway-enabled). Armed on the NAS composes. |
 | `ROBOCO_NOTIFICATION_SPAWN_COOLDOWN_SECONDS` | `600` | Cross-tick damper for notification-triggered spawns (escalation/approval/audit/a2a — task-less, so the readiness gate and respawn breaker never see them): one spawn per (agent, notification) per window; the notification stays pending so the next window retries. `0` = legacy every-tick respawn. |
-| `ROBOCO_SANDBOX_DB_ENABLED` | `false` | Sandboxed per-agent-spawn test DB/Redis: throwaway `postgres:16-alpine` / `redis:8-alpine` sibling containers, per-project opt-in. See "Sandboxed Dev DB/Redis" below and `docs/rag/architecture/sandbox-db.md`. |
+| `ROBOCO_SANDBOX_DB_ENABLED` | `false` | Sandboxed per-agent-spawn test DB/Redis/Mongo: throwaway sibling containers provisioned from the engine registry in `roboco/models/sandbox.py` (postgres:16-alpine / redis:8-alpine / mongo:8-alpine), per-project opt-in. The valid-service set is `VALID_SANDBOX_SERVICES` (registry-derived). See "Sandboxed Dev DB/Redis/Mongo" below and `docs/rag/architecture/sandbox-db.md`. |
 | `ROBOCO_X_ENGINE_ENABLED` | `false` | The X (Twitter) engine: draft release/mention posts, ALL held for per-post CEO approval. See "X (Twitter) Engine" below and `docs/rag/architecture/x-engine.md`. |
 | `ROBOCO_ROADMAP_ENGINE_ENABLED` | `false` | The board roadmap engine: weekly Product-Owner-authored cycle, CEO approves each item individually into BACKLOG. See "Board Roadmap Engine" below. |
 
@@ -144,13 +144,13 @@ The fan-out generalizations of self-heal — they watch any opted-in project, no
 | `ROBOCO_IMAGE_PRUNE_ENABLED` | `true` | Background sweep prunes dangling (`<none>`) Docker images from agent-image rebuilds (only dangling; ~6h throttle). Always-on safety net, not a feature flag |
 | `ROBOCO_IMAGE_PRUNE_INTERVAL_SECONDS` | `21600` | Minimum seconds between dangling-image prune passes |
 
-## Sandboxed Dev DB/Redis
+## Sandboxed Dev DB/Redis/Mongo
 
-Per-agent-spawn throwaway Postgres/Redis, replacing (never coexisting with) the legacy prod-creds gate-env injection for an opted-in project. Default-off; see `docs/rag/architecture/sandbox-db.md`.
+Per-agent-spawn throwaway Postgres/Redis/Mongo, replacing (never coexisting with) the legacy prod-creds gate-env injection for an opted-in project. Default-off; see `docs/rag/architecture/sandbox-db.md`.
 
 | Variable | Default | Description |
 |----------|---------|-------------|
-| `ROBOCO_SANDBOX_DB_ENABLED` | `false` | Master switch. Off = spawning behaves exactly as today (the legacy `_append_gate_env` prod-creds injection, itself gated by `ROBOCO_TOOLCHAIN_MATCH_ENABLED`). Only projects with their `sandbox_services` column set (migration `057`) participate even when on. |
+| `ROBOCO_SANDBOX_DB_ENABLED` | `false` | Master switch. Off = spawning behaves exactly as today (the legacy `_append_gate_env` prod-creds injection, itself gated by `ROBOCO_TOOLCHAIN_MATCH_ENABLED`). Only projects with their `sandbox_services` column set (migration `057`) participate even when on. The valid service set is `VALID_SANDBOX_SERVICES` in `roboco/models/sandbox.py` (registry-derived: postgres / redis / mongo); adding an engine is one class + one registry line, no orchestrator edit. Env injected per engine: `ROBOCO_TEST_DB_*`, `ROBOCO_TEST_REDIS_*`, `ROBOCO_TEST_MONGO_*` (incl. `ROBOCO_TEST_MONGO_AUTH_DB=admin`). |
 
 ## X (Twitter) Engine
 

docs/rag/architecture/db-network-isolation.md

@@ -2,7 +2,7 @@
 
 ## What It Is
 
-A compose-topology hardening: two user-defined Docker bridges instead of one. `roboco_default` carries the agent mesh (panel, nginx, ollama, every spawned agent container, and their sandbox DB/Redis sidecars — see `docs/rag/architecture/sandbox-db.md`). `roboco_data` carries **only** postgres + redis. The orchestrator is the sole multi-homed service (both networks) — every agent container structurally cannot resolve or TCP-reach `roboco-postgres:5432` / `roboco-redis:6379` at all. This matters because redis has no auth in this deployment: network membership *is* the containment, not a password.
+A compose-topology hardening: two user-defined Docker bridges instead of one. `roboco_default` carries the agent mesh (panel, nginx, ollama, every spawned agent container, and their sandbox sidecars — postgres / redis / mongo via the engine registry in `roboco/models/sandbox.py`; see `docs/rag/architecture/sandbox-db.md`). `roboco_data` carries **only** postgres + redis. The orchestrator is the sole multi-homed service (both networks) — every agent container structurally cannot resolve or TCP-reach `roboco-postgres:5432` / `roboco-redis:6379` at all. This matters because redis has no auth in this deployment: network membership *is* the containment, not a password.
 
 ## Enable/Disable
 
@@ -12,7 +12,7 @@ A compose-topology hardening: two user-defined Docker bridges instead of one. `r
 
 ## What flipping it changes
 
-`ROBOCO_DB_NETWORK_ISOLATED=true` suppresses the legacy `_append_gate_env` prod-creds injection (`roboco/runtime/orchestrator.py`) — the one that would otherwise hand an agent `ROBOCO_TEST_DB_HOST=roboco-postgres` credentials for a host it cannot reach. A connect timeout is worse than no credentials at all (the test suite's DB-reachability check skips cleanly on a fast refusal, but hangs on a dead-end timeout), so the flag makes that injection a no-op rather than let it happen and fail slow. Projects that need a real DB for their gate opt into the sandboxed dev DB/Redis instead (`docs/rag/architecture/sandbox-db.md`) — sandbox replaces, never coexists with, the prod-creds path.
+`ROBOCO_DB_NETWORK_ISOLATED=true` suppresses the legacy `_append_gate_env` prod-creds injection (`roboco/runtime/orchestrator.py`) — the one that would otherwise hand an agent `ROBOCO_TEST_DB_HOST=roboco-postgres` credentials for a host it cannot reach. A connect timeout is worse than no credentials at all (the test suite's DB-reachability check skips cleanly on a fast refusal, but hangs on a dead-end timeout), so the flag makes that injection a no-op rather than let it happen and fail slow. Projects that need a real DB for their gate opt into the sandboxed dev DB/Redis/Mongo instead (`docs/rag/architecture/sandbox-db.md`) — sandbox replaces, never coexists with, the prod-creds path.
 
 ## What is unaffected
 

docs/rag/architecture/sandbox-db.md

@@ -1,35 +1,50 @@
-# Sandboxed Dev DB/Redis
+# Sandboxed Dev DB/Redis/Mongo
 
 ## What It Is
 
-A per-agent-spawn throwaway Postgres/Redis pair, provisioned as **sibling containers** to the agent container (never docker-in-agent — the docker socket/CLI stay structurally absent from agent images). Implemented in `roboco/runtime/sandbox.py` (`SandboxProvisioner`), wired into the orchestrator's spawn path.
+A per-agent-spawn throwaway Postgres/Redis/Mongo set, provisioned as **sibling containers** to the agent container (never docker-in-agent — the docker socket/CLI stay structurally absent from agent images). Implemented in `roboco/runtime/sandbox.py` (`SandboxProvisioner`), wired into the orchestrator's spawn path.
 
 It replaces — never coexists with — the legacy `_append_gate_env` behavior that hands an agent RoboCo's own production Postgres credentials so its `make quality` gate can run the DB-backed test suite instead of a hollow unit-only subset.
 
+## The engine registry
+
+The service set is a **pluggable engine registry**, not a hardcoded postgres+redis pair. `roboco/models/sandbox.py` defines a `SandboxEngine` ABC (image, container port, readiness probe, tmpfs paths, env emission) and the concrete engines:
+
+- `_PostgresEngine` — `postgres:16-alpine`, tmpfs `/var/lib/postgresql/data`, `pg_isready` probe (60s), env `ROBOCO_TEST_DB_*` (incl. `ROBOCO_TEST_DB_ADMIN_DB`).
+- `_RedisEngine` — `redis:8-alpine`, no tmpfs, `redis-cli -a … ping` probe (15s), env `ROBOCO_TEST_REDIS_*`.
+- `_MongoEngine` — `mongo:8-alpine`, tmpfs `/data/db`, `mongosh` ping against auth db `admin` (60s), env `ROBOCO_TEST_MONGO_*` (incl. `ROBOCO_TEST_MONGO_AUTH_DB=admin`).
+
+`SANDBOX_ENGINES: dict[str, SandboxEngine]` registers them by name; `VALID_SANDBOX_SERVICES = frozenset(SANDBOX_ENGINES)` is the single source of truth the provisioner, the orchestrator's env injection, and `projects.sandbox_services` validation all consult. **Adding an engine is one class + one registry line** — no branch edited in the provisioner or the env emitter, which both iterate the registry.
+
 ## Enable/Disable
 
 | Variable | Default | Effect |
 |----------|---------|--------|
 | `ROBOCO_SANDBOX_DB_ENABLED` | `false` | Master switch. Off = spawning behaves exactly as today (the legacy prod-creds gate-env injection, itself gated by `ROBOCO_TOOLCHAIN_MATCH_ENABLED`). Panel-toggleable (Settings → Feature Flags). |
 
-A second, per-project gate applies even when the flag is on: only a project with its `sandbox_services` column set (e.g. `["postgres", "redis"]`; migration `057`, nullable/additive) participates. Every other project's spawns are byte-for-byte unaffected.
+A second, per-project gate applies even when the flag is on: only a project with its `sandbox_services` column set (e.g. `["postgres", "redis", "mongo"]`; migration `057`, nullable/additive) participates. Every other project's spawns are byte-for-byte unaffected. Mongo rides the same column — no new migration, no new feature flag; it is just another registry entry.
 
 ## Provisioning
 
-For an opted-in project's spawn, the orchestrator provisions before `docker run`:
+For an opted-in project's spawn, the orchestrator provisions each requested service through a single generic `_provision_engine` (no per-engine branch): it generates a random 32-hex-char password (`secrets.token_hex(16)`), pre-pulls the image, `docker run`s the sibling container, and polls the engine's readiness probe up to its deadline.
 
-- **Postgres**: `postgres:16-alpine`, named `roboco-sandbox-pg-{agent_id}`, `--tmpfs /var/lib/postgresql/data` (no disk persistence), `--memory 512m --cpus 1`, a random 32-hex-char password (`secrets.token_hex(16)`), user/db both `sandbox`. Readiness polled via `pg_isready` up to 60s.
-- **Redis**: `redis:8-alpine`, named `roboco-sandbox-redis-{agent_id}`, same memory/cpu caps, `--requirepass` with its own random password. Readiness polled via `redis-cli ping` up to 15s.
+- **Postgres**: named `roboco-sandbox-pg-{agent_id}`, `--tmpfs /var/lib/postgresql/data` (no disk persistence), `--memory 512m --cpus 1`, user/db both `sandbox`, readiness via `pg_isready` up to 60s.
+- **Redis**: named `roboco-sandbox-redis-{agent_id}`, same memory/cpu caps, `redis-server --requirepass`, readiness via `redis-cli -a … ping` up to 15s.
+- **Mongo**: named `roboco-sandbox-mongo-{agent_id}`, `--tmpfs /data/db`, same memory/cpu caps, root user/db `sandbox`/`sandbox`, readiness via `mongosh` ping (auth db `admin`) up to 60s.
 
-Both are labeled `roboco.sandbox=1` plus an owner label (`roboco.sandbox.owner=roboco-agent-{agent_id}`) so the janitor can find them. A provisioning failure is **fail-loud**: the spawn is refused (`AgentReadinessError`) rather than starting an agent whose gate can't run against a broken DB. A stale same-named sandbox left by a crash-missed teardown is pre-cleared before provisioning, so a leftover container can't collide with a fresh `docker run`.
+All are labeled `roboco.sandbox=1` plus an owner label (`roboco.sandbox.owner=roboco-agent-{agent_id}`) so the janitor can find them. A provisioning failure is **fail-loud**: the spawn is refused (`AgentReadinessError`) rather than starting an agent whose gate can't run against a broken DB, and any already-provisioned sibling is torn down before re-raising. A stale same-named sandbox left by a crash-missed teardown is pre-cleared before provisioning, so a leftover container can't collide with a fresh `docker run`.
+
+### Image pre-pull
+
+`_ensure_image` `docker image inspect`s the engine's image and, on absence, `docker pull`s it (300s timeout) **before** `docker run`. Without this a NAS cold pull would hit the 20s run timeout, get killed, and re-pull forever on every respawn. The inspect-then-pull runs per service per spawn, so an already-present image short-circuits in milliseconds.
 
 ## Injected environment
 
-Instead of the legacy `ROBOCO_TEST_DB_*` pointing at RoboCo's own production Postgres, the sandbox's own host/port/user/password are injected under the **same** `ROBOCO_TEST_DB_*` names (so an existing project's conftest needs no change) plus new `ROBOCO_TEST_REDIS_*` names. `_append_sandbox_env` runs **instead of** `_append_gate_env` whenever a sandbox was provisioned for that spawn.
+Instead of the legacy `ROBOCO_TEST_DB_*` pointing at RoboCo's own production Postgres, the sandbox's own host/port/user/password are injected. Env var names are preserved per engine so an existing project's conftest needs no change: `ROBOCO_TEST_DB_*` (postgres, incl. `ROBOCO_TEST_DB_ADMIN_DB`), `ROBOCO_TEST_REDIS_*` (redis), and `ROBOCO_TEST_MONGO_*` (mongo, incl. `ROBOCO_TEST_MONGO_AUTH_DB=admin`). The orchestrator's `_append_sandbox_env` is a single `cmd.extend(info.emit_env())` over the registry — a new engine's env lands with no orchestrator edit. It runs **instead of** `_append_gate_env` whenever a sandbox was provisioned for that spawn.
 
 ## Lifetime and teardown
 
-A sandbox's lifetime tracks its owning agent container 1:1: torn down (`stop` → `kill` fallback → `rm -f`, all best-effort and idempotent) at every container-removal path. An **orphan janitor** also runs at orchestrator startup and on each reaper tick: it lists every `roboco.sandbox=1` container, cross-references live agent containers, and removes any sandbox whose owner is gone.
+A sandbox's lifetime tracks its owning agent container 1:1: torn down (`stop` → `kill` fallback → `rm -f`, all best-effort and idempotent) at every container-removal path. Teardown iterates **all** registered engines (`SANDBOX_ENGINES.values()`) — a mongo sandbox is reaped by the same path that reaps postgres/redis, with no per-engine teardown branch. An **orphan janitor** also runs at orchestrator startup and on each reaper tick: it lists every `roboco.sandbox=1` container, cross-references live agent containers, and removes any sandbox whose owner is gone.
 
 The janitor has a **grace window** (`_JANITOR_GRACE_SECONDS`, 180s): a sandbox is provisioned *before* its agent container exists, so a sweep racing a mid-flight spawn would otherwise see "owner not live yet" and reap a fresh sandbox out from under a spawn still starting up. Owners provisioned within the grace window are skipped by that pass. The pre-spawn stale-clear (above) likewise never touches a just-provisioned sandbox.