feat(sandbox): pluggable per-engine registry (postgres/redis/mongo) (#324)

* feat(sandbox): pluggable per-engine registry (postgres/redis/mongo)

Replaces the hardcoded postgres+redis branches in the provisioner and the
env emitter with a registry of SandboxEngine specs (image, run args,
readiness probe, connection, ROBOCO_TEST_* env) in a pure low module
(roboco/models/sandbox.py). VALID_SANDBOX_SERVICES is derived from the
registry — single source of truth — and the provisioner + orchestrator
iterate it, so adding an engine is one class + one registry line, not
another branch. Adds a mongo:8-alpine engine (ROBOCO_TEST_MONGO_*) as the
third service alongside postgres/redis.

Also fixes the cold-pull loop that stranded v0.19.0 board agents with
empty error strings: docker run pulled inline under a 20s deadline, so a
NAS cold pull was killed, cancelled, and re-pulled from scratch forever.
_ensure_image now inspects + pulls (300s) before run; provisioning errors
log type+message so a bare TimeoutError no longer shows as "".

Panel edit-project dialog: postgres/redis toggles -> a Set<string>
multi-select driven by a SANDBOX_SERVICES catalog, so new engines appear
in the UI by adding to the catalog.

Tests: engine parity (allowlist==registry, unique slugs/images, no None
leak in env, SandboxInfo aggregates every engine), mongo provision + env
injection, plus the existing postgres/redis provision/env/spawn/janitor
suite updated to the registry shape. 821 unit / 5 skip green; ruff + mypy
(360 files) clean.

* docs(sandbox): reflect pluggable engine registry + mongo across docs

CHANGELOG (0.19.0): Added entry for the pluggable sandbox engine registry
(postgres/redis/mongo) + Fixed entry for the cold-pull loop/empty-error
strand that boarded v0.19.0 board agents.

docs/map (9 files): sandbox subsystem blurbs, SandboxProvisioner rows,
_maybe_provision_sandbox/_append_sandbox_env rows, feature-flag rows, the
migration-057 row + v0.17.0 delta, and the models.md VALID_SANDBOX_SERVICES
note — all retitled to DB/Redis/Mongo via the engine registry
(roboco/models/sandbox.py), with the one-class-one-line extension story and
the _ensure_image cold-pull fix. Production-network (roboco_data) lines left
as postgres+redis — mongo is sandbox-only, not a prod service.

docs/rag (3 files): sandbox-db.md rewritten around the registry (engine list,
generic _provision_engine, image pre-pull, ROBOCO_TEST_DB_*/REDIS_*/MONGO_*
incl. MONGO_AUTH_DB=admin, single emit_env); config-reference sandbox flag
row + subsection retitled; db-network-isolation framing broadened to
postgres/redis/mongo. preconditions-and-rejections left untouched (its hit
was an unrelated gateway see-also link).

* test(e2e): harden umbrella close terminal reads with bounded wait-for-state

The MegaTask umbrella close test flaked once on CI (ceo-approve returned
200 but the re-fetch saw awaiting_pm_review) then passed on re-run. The
production path is deterministic: complete -> main_pm_complete ->
submit_pm_review -> escalate_to_ceo -> ceo_approve -> commit, all on one
session, all awaited; the fire-and-forget completion hooks are isolated
(own session, best-effort, never touch task.status or the request session).
20 local runs could not reproduce it.

The one real surface is the read pattern: the e2e stack commits on the
uvicorn thread's loop and reads via a separate loop (run_db -> asyncio.run
with a fresh engine), so a terminal single point-read can race a
still-draining completion hook on a contended runner. Replace the two
terminal point-reads with a bounded wait_for_status poll. Strictly better
than a one-shot read: absorbs the transient, and a genuine state bug still
surfaces via the timeout branch asserting against the last-read state.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>

docs/map/orchestrator.md

@@ -1,5 +1,5 @@
 ## Purpose
-The AgentOrchestrator is the runtime brain of RoboCo: it owns the per-agent Docker container lifecycle, the per-tick dispatcher that matches tasks to agents, the stale-claim reaper, the provider rate-limit/overload park-and-probe recovery loop, and the default-off background engines (self-heal, CI-watch, dep-update, release-manager, strategy, external-PR poll, X-engine mentions poll, board roadmap engine, video render loop). It claims tasks on behalf of agents before spawning, injects briefings/manifests/git context at spawn time, provisions a per-spawn sandbox DB/Redis when opted in, captures per-session token usage, and persists durable runtime state (WaitingRecord, respawn_tracker) across restarts.
+The AgentOrchestrator is the runtime brain of RoboCo: it owns the per-agent Docker container lifecycle, the per-tick dispatcher that matches tasks to agents, the stale-claim reaper, the provider rate-limit/overload park-and-probe recovery loop, and the default-off background engines (self-heal, CI-watch, dep-update, release-manager, strategy, external-PR poll, X-engine mentions poll, board roadmap engine, video render loop). It claims tasks on behalf of agents before spawning, injects briefings/manifests/git context at spawn time, provisions a per-spawn sandbox (postgres/redis/mongo via the engine registry) when opted in, captures per-session token usage, and persists durable runtime state (WaitingRecord, respawn_tracker) across restarts.
 
 ## Files
 
@@ -80,8 +80,8 @@ The AgentOrchestrator is the runtime brain of RoboCo: it owns the per-agent Dock
 | AgentOrchestrator._should_skip_live_reap | method | roboco/runtime/orchestrator.py:8750 | Spare a live container from reaping UNLESS wedged (grok) or gateway-broken past grace (those kill+evict). |
 | AgentOrchestrator._assignee_is_provider_parked | method | roboco/runtime/orchestrator.py:8527 | True if a task's assignee is provider-parked; reaper skips it so the claim survives for probe-resume. |
 | AgentOrchestrator._reap_with_service | method | roboco/runtime/orchestrator.py:8770 | Inner stale-claim reaper: skip live (unless wedged/broken), skip provider-parked, unclaim_for_reaper the rest. |
-| AgentOrchestrator._maybe_provision_sandbox | method | roboco/runtime/orchestrator.py:2029 | Provision this spawn's sandbox DB/Redis via `SandboxProvisioner` when `sandbox_db_enabled` + the project's `sandbox_services` are set; None (byte-for-byte legacy path) otherwise. Fail-loud once opted in — a provisioning failure refuses the spawn. |
-| AgentOrchestrator._append_sandbox_env | staticmethod | roboco/runtime/orchestrator.py:2639 | Inject `ROBOCO_TEST_DB_*`/`ROBOCO_TEST_REDIS_*` env from `config.sandbox_info`, called INSTEAD OF `_append_gate_env` whenever a sandbox was provisioned for this spawn. |
+| AgentOrchestrator._maybe_provision_sandbox | method | roboco/runtime/orchestrator.py:2029 | Provision this spawn's sandbox engines (postgres/redis/mongo via `SandboxProvisioner` + the `SANDBOX_ENGINES` registry) when `sandbox_db_enabled` + the project's `sandbox_services` are set; None (byte-for-byte legacy path) otherwise. Fail-loud once opted in — a provisioning failure refuses the spawn. |
+| AgentOrchestrator._append_sandbox_env | staticmethod | roboco/runtime/orchestrator.py:2639 | Inject the sandbox's `ROBOCO_TEST_*` env (postgres `ROBOCO_TEST_DB_*` / redis `ROBOCO_TEST_REDIS_*` / mongo `ROBOCO_TEST_MONGO_*`) via `SandboxInfo.emit_env` over the engine registry, called INSTEAD OF `_append_gate_env` whenever a sandbox was provisioned for this spawn. Emission is registry-driven, so a new engine's env vars land here with no orchestrator change. |
 | AgentOrchestrator._sandbox_janitor_sweep | method | roboco/runtime/orchestrator.py:9426 | Best-effort: remove sandbox containers whose owner agent is gone; rides the reaper tick, error-isolated. |
 | AgentOrchestrator._x_mentions_poll_loop | method | roboco/runtime/orchestrator.py:7431 | Default-off X-engine mentions-poll tick loop (`x_engine_enabled`); release-post drafts are event-driven, not from this loop. |
 | AgentOrchestrator._run_x_mentions_cycle | method | roboco/runtime/orchestrator.py:7453 | One mentions-poll pass: `get_x_engine(db).run_cycle()` + commit; testable without the sleep. |
@@ -197,7 +197,7 @@ stateDiagram-v2
 - `ROBOCO_GROK_MAX_COST_USD` — grok budget kill-switch; `_GROK_RATE_LIMIT_EXIT_CODE=75`, `_GROK_AUTH_EXIT_CODE=78`, `_PROBE_GIVE_UP_THRESHOLD=30`.
 - `ROBOCO_CLAUDE_STUCK_KILL_SECONDS` (default 3600, min 600) — heartbeat-stale kill threshold for non-GROK agents; controls `_maybe_kill_stuck_claude`.
 - `ROBOCO_DISPATCHER_INTERVAL_SECONDS` (30), `ROBOCO_INTERACTIVE_IDLE_REAP_SECONDS`, `ROBOCO_GROK_*` backoff constants.
-- `ROBOCO_SANDBOX_DB_ENABLED` (default off) — master switch for the sandboxed per-agent-spawn Postgres/Redis provisioner (`_maybe_provision_sandbox`/`_append_sandbox_env`/`_sandbox_janitor_sweep`); a project participates only when its `sandbox_services` column is also set.
+- `ROBOCO_SANDBOX_DB_ENABLED` (default off) — master switch for the sandboxed per-agent-spawn engine provisioner (`_maybe_provision_sandbox`/`_append_sandbox_env`/`_sandbox_janitor_sweep`); a project participates only when its `sandbox_services` column is also set. The service set is the `SANDBOX_ENGINES` registry (postgres / redis / mongo); adding an engine needs no orchestrator or env-emitter change.
 - `ROBOCO_DB_NETWORK_ISOLATED` (default off; set by the compose topology that carries the `roboco_data` network) — suppresses the legacy `_append_gate_env` prod-creds injection when postgres/redis are unreachable from the agent mesh.
 - `ROBOCO_CLOUD_AUTH_ENABLED` (+ `_EMAIL`/`_PASSWORD`/`_SECRET`/`_COOKIE_MAX_AGE`, default off) — cloud auth master switch; read by `roboco.api.deps.get_agent_context`/`roboco.api.auth.*`, not the orchestrator itself, but gates whether a spawned agent's own HMAC-token identity path is the sole non-CEO auth route.
 - `ROBOCO_X_ENGINE_ENABLED` (+ `_mentions_interval_seconds` / `_mentions_max_per_cycle` / `_mentions_min_engagement` / `_max_open_posts` / `_account_user_id` / `_request_timeout_seconds`, default off) — gates `_x_mentions_poll_loop`.