roboco/docs/rag/architecture/sandbox-db.md

Sandboxed Dev DB/Redis/Mongo

What It Is

A throwaway Postgres/Redis/Mongo set, provisioned as sibling containers to the agent container (never docker-in-agent — the docker socket/CLI stay structurally absent from agent images). Implemented in roboco/runtime/sandbox.py (SandboxProvisioner) and AgentOrchestrator.ensure_sandbox.

On-demand since 2026-07-08: a sandbox is provisioned only when an agent actually asks for one, via the request_sandbox do-verb (developer + QA roles) — not eagerly at spawn. See "The request_sandbox verb" below; the design rationale lives in docs/internal/specs/2026-07-08-sandbox-on-demand.md.

It replaces — never coexists with — the legacy _append_gate_env behavior that hands an agent RoboCo's own production Postgres credentials so its make quality gate can run the DB-backed test suite instead of a hollow unit-only subset.

The engine registry

The service set is a pluggable engine registry, not a hardcoded postgres+redis pair. roboco/models/sandbox.py defines a SandboxEngine ABC (image, container port, readiness probe, tmpfs paths, env emission) and the concrete engines:

• _PostgresEngine — postgres:16-alpine, tmpfs /var/lib/postgresql/data, pg_isready probe (60s), env ROBOCO_TEST_DB_* (incl. ROBOCO_TEST_DB_ADMIN_DB).
• _RedisEngine — redis:8-alpine, no tmpfs, redis-cli -a … ping probe (15s), env ROBOCO_TEST_REDIS_*.
• _MongoEngine — mongo:8 (MongoDB ships no Alpine variant), tmpfs /data/db, mongosh ping against auth db admin (60s), env ROBOCO_TEST_MONGO_* (incl. ROBOCO_TEST_MONGO_AUTH_DB=admin).

SANDBOX_ENGINES: dict[str, SandboxEngine] registers them by name; VALID_SANDBOX_SERVICES = frozenset(SANDBOX_ENGINES) is the single source of truth the provisioner, the orchestrator's env injection, and projects.sandbox_services validation all consult. Adding an engine is one class + one registry line — no branch edited in the provisioner or the env emitter, which both iterate the registry.

Enable/Disable

Variable	Default	Effect
ROBOCO_SANDBOX_DB_ENABLED	false	Master switch. Off = spawning behaves exactly as today (the legacy prod-creds gate-env injection, itself gated by ROBOCO_TOOLCHAIN_MATCH_ENABLED), and request_sandbox refuses. Panel-toggleable (Settings → Feature Flags).

A second, per-project gate applies even when the flag is on: only a project with its sandbox_services column set (e.g. ["postgres", "redis", "mongo"]; migration 057, nullable/additive) participates. Every other project's spawns are byte-for-byte unaffected. Mongo rides the same column — no new migration, no new feature flag; it is just another registry entry. request_sandbox may request any subset of the opted-in set (or omit services for the whole set); anything outside it is rejected naming the allowed set.

The request_sandbox verb

Provisioning is on-demand: nothing is provisioned at spawn. A developer or QA agent calls the request_sandbox content tool (on roboco-do) when it actually needs a sandboxed DB, and the orchestrator provisions it inline. Wiring: roboco/api/schemas/v1/do.py RequestSandboxRequest → POST /api/v1/do/request_sandbox → ContentActions.request_sandbox (roboco/services/gateway/content_actions.py) → AgentOrchestrator.ensure_sandbox → roboco/mcp/do_server.py's request_sandbox() tool (1080s timeout — ensure_sandbox always provisions the project's full opted-in set on first call, so an all-three-engines-cold first request is the norm the timeout must cover, not a rare edge case).

request_sandbox(
    services: list[str] | None = None,
    extensions: dict[str, list[str]] | None = None,
)

services omitted means the project's whole opted-in set. extensions is an optional per-service map of extensions/modules to activate (see "Extensions and modules (on the fly)" below); it unions with the project's standing set. Guards fire in order, each with a clean invalid_state envelope + remediate:

1. ROBOCO_SANDBOX_DB_ENABLED off → refused before any DB lookup.
2. No active, project-bound task (agent hasn't give_me_work'd) → refused.
3. Project has no sandbox_services opted in → refused.
4. A requested service outside the project's opted set → refused, remediate names the allowed set.
5. A requested extension outside the per-service allowlist, or for a service not opted in → refused, remediate names the allowed extensions.
6. Orchestrator handle unavailable (e.g. mid-restart) → refused, but retryable — the one guard that isn't a permanent no.

Only past all five does it call ensure_sandbox and provision. A genuine provisioning failure (image pull, readiness timeout) also surfaces as a retryable invalid_state, never a spawn refusal — sandbox trouble can no longer block a spawn or an agent's turn.

On success, creds return in the ok-envelope's evidence, one entry per service — no env injection, no schema change:

{
  "postgres": {
    "host": "roboco-sandbox-pg-be-dev-1",
    "port": 5432,
    "user": "sandbox",
    "password": "<random>",
    "database": "sandbox",
    "env": {
      "ROBOCO_TEST_DB_HOST": "roboco-sandbox-pg-be-dev-1",
      "ROBOCO_TEST_DB_PORT": "5432",
      "ROBOCO_TEST_DB_USER": "sandbox",
      "ROBOCO_TEST_DB_PASSWORD": "<random>",
      "ROBOCO_TEST_DB_ADMIN_DB": "sandbox"
    }
  }
}

The env sub-dict (SandboxInfo.as_payload(), roboco/models/sandbox.py) carries the exact same variable names the legacy env-injection path used, so an agent can export them verbatim for gate tooling that reads ROBOCO_TEST_*. Networking needs no extra step: sandboxes join roboco_default at docker run, the same network every agent is on, so DNS resolves the moment the sibling starts.

ensure_sandbox always provisions the project's whole opted-in set on first call, regardless of what a given request_sandbox call named — so calling it again for any subset or superset of that opted set is a guaranteed cache hit (same creds, no docker calls), and a live container is never torn down mid-session by a later, broader request. services only scopes what comes back in this call's evidence; the response payload is filtered down to that subset even though the full set was provisioned. A cache hit is also re-verified live (SandboxProvisioner.is_live) before being trusted — a container OOM-killed or removed out-of-band evicts the stale entry and triggers a fresh full-set provision with new creds. Concurrent calls for the same agent (e.g. a client timeout + retry) are serialized behind a per-agent-slug asyncio.Lock so they can't race provision()/teardown() against each other. ensure_sandbox is always called with the caller's own authenticated agent slug — a caller can never reach another agent's sandbox.

Only developer and qa roles carry request_sandbox in their spawn manifest (roboco/services/gateway/role_config.py _DEV_DO / _QA_DO) — the DB-needing gate roles. It is carried unconditionally on those manifests (declarative); the real gating is the project opt-in check inside the verb itself.

Extensions and modules (on the fly)

A sandboxed DB/Redis can be built to a venture's declared extensions on the fly — "need a db? ok, extensions?" — instead of a fixed flavor. A project declares a per-service extension/module map in projects.sandbox_extensions (migration 072, jsonb null), e.g. {"postgres": ["vector", "postgis"], "redis": ["search"]}. The provisioner activates them post-ready via docker exec (CREATE EXTENSION IF NOT EXISTS <name> for pg, MODULE LOAD <so> for redis) then verifies presence, so a missing extension file fails loudly at first provision rather than silently. Design spec: docs/internal/specs/2026-07-13-sandbox-extensions-on-the-fly.md.

The allowlist is the security containment. SANDBOX_PG_EXTENSIONS = {vector, postgis, pg_trgm, citext, uuid-ossp} and SANDBOX_REDIS_MODULES = {search, json, bloom} (roboco/models/sandbox.py SANDBOX_ENGINE_FEATURES) are the only extensions/modules the system will ever activate. A plpython3u (superuser-RCE) is excluded by construction — rejected at the Project pydantic boundary before it can be persisted, and again at the verb. Mongo has no activatable features and is intentionally absent from the map. The Project model normalizes order, drops empty feature lists (a service with [] is bare), and rejects unknown service keys or unallowed features with the allowlist named.

No default set. Opters set the extensions they need explicitly; an unset service is bare. Existing opted-in projects stay byte-for-byte unchanged on the bare path — no behavior change unless a venture declares a set.

Standing set + per-call override. The project field is the trusted standing set; a request_sandbox(extensions=…) call unions a per-call override with it, bounded to the opted-in service set and the allowlist. The union reaches ensure_sandbox as the features kwarg. Recommendation: set the full set the venture needs in project settings, so agents can request subsets at runtime — a subset request is a cache hit, a feature superset re-provisions (rotates creds), mirroring the services-superset case.

Cache-by-features. A cached SandboxInfo satisfies a new ensure_sandbox call iff the services are a subset AND every requested feature per service is already cached. A feature superset re-provisions with fresh creds (the pre-clear teardown removes the now-too-small container), exactly as a services superset does.

Feature-aware image selection. A bare request (no features) uses the light upstream image (postgres:16-alpine / redis:8-alpine); a request with features pulls a kitchen-sink image that carries all extension/module files (roboco-sandbox-pg:latest — pgvector/pgvector:pg16 + postgis; redis/redis-stack-server:latest). So a bare project never pays for a heavier image, and the intersection case (pgvector AND postgis in one DB) just works — flavors are inadequate because the intersection isn't a flavor. SandboxEngine.image_for(features) picks; the provisioner pulls whichever it resolves to.

Delivery. The ok-envelope's per-service evidence entry carries available_extensions (the sorted list actually activated) so an agent doesn't guess what was turned on. The project's standing set is edited in the panel's project edit dialog (Sandbox section) — checkboxes from the allowlist grouped under each enabled service.

Provisioning

ensure_sandbox(agent_slug, requested, opted) provisions requested | opted (in practice the project's whole opted-in set) through a single generic _provision_engine (no per-engine branch): it generates a random 32-hex-char password (secrets.token_hex(16)), pre-pulls the image, docker runs the sibling container, and polls the engine's readiness probe up to its deadline.

• Postgres: named roboco-sandbox-pg-{agent_id}, --tmpfs /var/lib/postgresql/data (no disk persistence), --memory 512m --cpus 1, user/db both sandbox, readiness via pg_isready up to 60s.
• Redis: named roboco-sandbox-redis-{agent_id}, same memory/cpu caps, redis-server --requirepass, readiness via redis-cli -a … ping up to 15s.
• Mongo: named roboco-sandbox-mongo-{agent_id}, --tmpfs /data/db, same memory/cpu caps, root user/db sandbox/sandbox, readiness via mongosh ping (auth db admin) up to 60s.

All are labeled roboco.sandbox=1 plus an owner label (roboco.sandbox.owner=roboco-agent-{agent_id}) so the janitor can find them. A stale same-named sandbox left by a crash-missed teardown is pre-cleared before provisioning, so a leftover container can't collide with a fresh docker run. A provisioning failure now surfaces as a retryable envelope on the verb (see above) rather than refusing anything — there is no spawn to refuse, since the sandbox is requested well after the agent is already running.

Image pre-pull

_ensure_image docker image inspects the engine's image and, on absence, docker pulls it (300s timeout) before docker run. Without this a NAS cold pull would hit the 20s run timeout, get killed, and re-pull forever on every retry. The inspect-then-pull runs per service per call, so an already-present image short-circuits in milliseconds.

Spawn-time availability probe

Spawn itself no longer provisions anything. For an opted-in project, AgentOrchestrator._sandbox_available_services is a cheap DB lookup (best-effort — a hiccup degrades to "no sandbox" rather than blocking the spawn) that returns the project's opted-in service list, and the spawn path injects a marker env ROBOCO_SANDBOX_SERVICES_AVAILABLE=postgres,redis (never creds) plus one line in the agent's session briefing naming request_sandbox() explicitly — cheap and kills a discovery failure mode where an agent doesn't know the tool exists. AgentConfig.sandbox_info and the old eager _append_sandbox_env are gone; _append_sandbox_marker_env runs instead of _append_gate_env for an opted-in project's spawn (never both).

Lifetime and teardown

A sandbox no longer only dies with its container: AgentOrchestrator.release_sandbox(agent_slug) tears it down at the end of the caller's own engagement with its work — the Choreographer calls it (best-effort, never failing the verb) on the SUCCESSFUL exit of i_am_done, unclaim, i_am_idle, pass_review/fail_review, and i_documented, so an agent that finishes then idles or picks up unrelated work doesn't leave a sidecar running for the rest of its session. release_sandbox is a fast no-op for the common case of no cached sandbox (a single dict check, before any lock or docker call) and otherwise reuses the same teardown + cache-eviction pairing as container removal. A sandbox's lifetime STILL tracks its owning agent container 1:1 as the backstop: torn down (stop → kill fallback → rm -f, all best-effort and idempotent) at every container-removal path. Teardown iterates all registered engines (SANDBOX_ENGINES.values()) — a mongo sandbox is reaped by the same path that reaps postgres/redis, with no per-engine teardown branch. An orphan janitor also runs at orchestrator startup and on each reaper tick: it lists every roboco.sandbox=1 container, cross-references live agent containers, and removes any sandbox whose owner is gone.

The janitor has a grace window (_JANITOR_GRACE_SECONDS, 180s): a sweep racing a mid-flight request_sandbox call would otherwise see "owner not live yet" and reap a sandbox out from under a request still in progress. Owners provisioned within the grace window are skipped by that pass. The pre-spawn stale-clear likewise never touches a sandbox just requested via the verb.

Creds cache. AgentOrchestrator._sandbox_info (agent slug → last-provisioned SandboxInfo, always covering the project's full opted-in set) is what makes repeat request_sandbox calls cheap. It is evicted at every teardown path and by the janitor sweep whenever it reaps an orphan — so a torn-down sandbox's cached creds never outlive the container — and also on a failed on-demand liveness check at cache-hit time (SandboxProvisioner.is_live), which catches a container that died without going through any of those explicit teardown paths (OOM-killed, manually removed). A per-agent-slug asyncio.Lock (AgentOrchestrator._sandbox_locks) wraps the whole check-cache → provision → store section so two concurrent ensure_sandbox calls for one agent can't race each other's provision()/teardown(). Known ceiling: the cache is in-memory only; an orchestrator restart forgets it. The next request_sandbox call re-provisions (the pre-clear tears down any still-running stale container) and returns fresh creds — agents already treat creds as session-scoped, and a connection to the torn-down sandbox simply fails loudly rather than silently.

Related

• docs/internal/specs/2026-07-08-sandbox-on-demand.md — the on-demand design spec and rationale
• docs/rag/tools/task-tools.md — request_sandbox alongside the rest of the dev/QA verb surface
• docs/rag/architecture/config-reference.md — full env var table
• docs/rag/architecture/db-network-isolation.md — the network-topology change this pairs with in a NAS deploy (agents structurally can't reach production Postgres/Redis at all; sandbox is the DB-needing project's alternative)