mirror of https://github.com/rennf93/roboco.git synced 2026-08-03 07:23:24 +02:00

Files

T

8f6dde9a50 feat(sandbox): pluggable per-engine registry (postgres/redis/mongo) (#324 )

* feat(sandbox): pluggable per-engine registry (postgres/redis/mongo)

Replaces the hardcoded postgres+redis branches in the provisioner and the
env emitter with a registry of SandboxEngine specs (image, run args,
readiness probe, connection, ROBOCO_TEST_* env) in a pure low module
(roboco/models/sandbox.py). VALID_SANDBOX_SERVICES is derived from the
registry — single source of truth — and the provisioner + orchestrator
iterate it, so adding an engine is one class + one registry line, not
another branch. Adds a mongo:8-alpine engine (ROBOCO_TEST_MONGO_*) as the
third service alongside postgres/redis.

Also fixes the cold-pull loop that stranded v0.19.0 board agents with
empty error strings: docker run pulled inline under a 20s deadline, so a
NAS cold pull was killed, cancelled, and re-pulled from scratch forever.
_ensure_image now inspects + pulls (300s) before run; provisioning errors
log type+message so a bare TimeoutError no longer shows as "".

Panel edit-project dialog: postgres/redis toggles -> a Set<string>
multi-select driven by a SANDBOX_SERVICES catalog, so new engines appear
in the UI by adding to the catalog.

Tests: engine parity (allowlist==registry, unique slugs/images, no None
leak in env, SandboxInfo aggregates every engine), mongo provision + env
injection, plus the existing postgres/redis provision/env/spawn/janitor
suite updated to the registry shape. 821 unit / 5 skip green; ruff + mypy
(360 files) clean.

* docs(sandbox): reflect pluggable engine registry + mongo across docs

CHANGELOG (0.19.0): Added entry for the pluggable sandbox engine registry
(postgres/redis/mongo) + Fixed entry for the cold-pull loop/empty-error
strand that boarded v0.19.0 board agents.

docs/map (9 files): sandbox subsystem blurbs, SandboxProvisioner rows,
_maybe_provision_sandbox/_append_sandbox_env rows, feature-flag rows, the
migration-057 row + v0.17.0 delta, and the models.md VALID_SANDBOX_SERVICES
note — all retitled to DB/Redis/Mongo via the engine registry
(roboco/models/sandbox.py), with the one-class-one-line extension story and
the _ensure_image cold-pull fix. Production-network (roboco_data) lines left
as postgres+redis — mongo is sandbox-only, not a prod service.

docs/rag (3 files): sandbox-db.md rewritten around the registry (engine list,
generic _provision_engine, image pre-pull, ROBOCO_TEST_DB_*/REDIS_*/MONGO_*
incl. MONGO_AUTH_DB=admin, single emit_env); config-reference sandbox flag
row + subsection retitled; db-network-isolation framing broadened to
postgres/redis/mongo. preconditions-and-rejections left untouched (its hit
was an unrelated gateway see-also link).

* test(e2e): harden umbrella close terminal reads with bounded wait-for-state

The MegaTask umbrella close test flaked once on CI (ceo-approve returned
200 but the re-fetch saw awaiting_pm_review) then passed on re-run. The
production path is deterministic: complete -> main_pm_complete ->
submit_pm_review -> escalate_to_ceo -> ceo_approve -> commit, all on one
session, all awaited; the fire-and-forget completion hooks are isolated
(own session, best-effort, never touch task.status or the request session).
20 local runs could not reproduce it.

The one real surface is the read pattern: the e2e stack commits on the
uvicorn thread's loop and reads via a separate loop (run_db -> asyncio.run
with a fresh engine), so a terminal single point-read can race a
still-draining completion hook on a contended runner. Replace the two
terminal point-reads with a bounded wait_for_status poll. Strictly better
than a one-shot read: absorbs the transient, and a genuine state bug still
surfaces via the timeout branch asserting against the last-read state.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>

2026-07-07 13:59:53 +02:00

6.1 KiB

Raw Blame History

Sandboxed Dev DB/Redis/Mongo

What It Is

A per-agent-spawn throwaway Postgres/Redis/Mongo set, provisioned as sibling containers to the agent container (never docker-in-agent — the docker socket/CLI stay structurally absent from agent images). Implemented in roboco/runtime/sandbox.py (SandboxProvisioner), wired into the orchestrator's spawn path.

It replaces — never coexists with — the legacy _append_gate_env behavior that hands an agent RoboCo's own production Postgres credentials so its make quality gate can run the DB-backed test suite instead of a hollow unit-only subset.

The engine registry

The service set is a pluggable engine registry, not a hardcoded postgres+redis pair. roboco/models/sandbox.py defines a SandboxEngine ABC (image, container port, readiness probe, tmpfs paths, env emission) and the concrete engines:

_PostgresEngine — postgres:16-alpine, tmpfs /var/lib/postgresql/data, pg_isready probe (60s), env ROBOCO_TEST_DB_* (incl. ROBOCO_TEST_DB_ADMIN_DB).
_RedisEngine — redis:8-alpine, no tmpfs, redis-cli -a … ping probe (15s), env ROBOCO_TEST_REDIS_*.
_MongoEngine — mongo:8-alpine, tmpfs /data/db, mongosh ping against auth db admin (60s), env ROBOCO_TEST_MONGO_* (incl. ROBOCO_TEST_MONGO_AUTH_DB=admin).

SANDBOX_ENGINES: dict[str, SandboxEngine] registers them by name; VALID_SANDBOX_SERVICES = frozenset(SANDBOX_ENGINES) is the single source of truth the provisioner, the orchestrator's env injection, and projects.sandbox_services validation all consult. Adding an engine is one class + one registry line — no branch edited in the provisioner or the env emitter, which both iterate the registry.

Enable/Disable

Variable	Default	Effect
`ROBOCO_SANDBOX_DB_ENABLED`	`false`	Master switch. Off = spawning behaves exactly as today (the legacy prod-creds gate-env injection, itself gated by `ROBOCO_TOOLCHAIN_MATCH_ENABLED`). Panel-toggleable (Settings → Feature Flags).

A second, per-project gate applies even when the flag is on: only a project with its sandbox_services column set (e.g. ["postgres", "redis", "mongo"]; migration 057, nullable/additive) participates. Every other project's spawns are byte-for-byte unaffected. Mongo rides the same column — no new migration, no new feature flag; it is just another registry entry.

Provisioning

For an opted-in project's spawn, the orchestrator provisions each requested service through a single generic _provision_engine (no per-engine branch): it generates a random 32-hex-char password (secrets.token_hex(16)), pre-pulls the image, docker runs the sibling container, and polls the engine's readiness probe up to its deadline.

Postgres: named roboco-sandbox-pg-{agent_id}, --tmpfs /var/lib/postgresql/data (no disk persistence), --memory 512m --cpus 1, user/db both sandbox, readiness via pg_isready up to 60s.
Redis: named roboco-sandbox-redis-{agent_id}, same memory/cpu caps, redis-server --requirepass, readiness via redis-cli -a … ping up to 15s.
Mongo: named roboco-sandbox-mongo-{agent_id}, --tmpfs /data/db, same memory/cpu caps, root user/db sandbox/sandbox, readiness via mongosh ping (auth db admin) up to 60s.

All are labeled roboco.sandbox=1 plus an owner label (roboco.sandbox.owner=roboco-agent-{agent_id}) so the janitor can find them. A provisioning failure is fail-loud: the spawn is refused (AgentReadinessError) rather than starting an agent whose gate can't run against a broken DB, and any already-provisioned sibling is torn down before re-raising. A stale same-named sandbox left by a crash-missed teardown is pre-cleared before provisioning, so a leftover container can't collide with a fresh docker run.

Image pre-pull

_ensure_image docker image inspects the engine's image and, on absence, docker pulls it (300s timeout) before docker run. Without this a NAS cold pull would hit the 20s run timeout, get killed, and re-pull forever on every respawn. The inspect-then-pull runs per service per spawn, so an already-present image short-circuits in milliseconds.

Injected environment

Instead of the legacy ROBOCO_TEST_DB_* pointing at RoboCo's own production Postgres, the sandbox's own host/port/user/password are injected. Env var names are preserved per engine so an existing project's conftest needs no change: ROBOCO_TEST_DB_* (postgres, incl. ROBOCO_TEST_DB_ADMIN_DB), ROBOCO_TEST_REDIS_* (redis), and ROBOCO_TEST_MONGO_* (mongo, incl. ROBOCO_TEST_MONGO_AUTH_DB=admin). The orchestrator's _append_sandbox_env is a single cmd.extend(info.emit_env()) over the registry — a new engine's env lands with no orchestrator edit. It runs instead of _append_gate_env whenever a sandbox was provisioned for that spawn.

Lifetime and teardown

A sandbox's lifetime tracks its owning agent container 1:1: torn down (stop → kill fallback → rm -f, all best-effort and idempotent) at every container-removal path. Teardown iterates all registered engines (SANDBOX_ENGINES.values()) — a mongo sandbox is reaped by the same path that reaps postgres/redis, with no per-engine teardown branch. An orphan janitor also runs at orchestrator startup and on each reaper tick: it lists every roboco.sandbox=1 container, cross-references live agent containers, and removes any sandbox whose owner is gone.

The janitor has a grace window (_JANITOR_GRACE_SECONDS, 180s): a sandbox is provisioned before its agent container exists, so a sweep racing a mid-flight spawn would otherwise see "owner not live yet" and reap a fresh sandbox out from under a spawn still starting up. Owners provisioned within the grace window are skipped by that pass. The pre-spawn stale-clear (above) likewise never touches a just-provisioned sandbox.

docs/rag/architecture/config-reference.md — full env var table
docs/rag/architecture/db-network-isolation.md — the network-topology change this pairs with in a NAS deploy (agents structurally can't reach production Postgres/Redis at all; sandbox is the DB-needing project's alternative)

6.1 KiB Raw Blame History