diff --git a/CHANGELOG.md b/CHANGELOG.md index 6c9b8805..5eaf176d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added +- **Pluggable sandbox engine registry (postgres / redis / mongo).** The per-agent-spawn sandbox service set is now a registry instead of hardcoded postgres+redis branches. A new pure module `roboco/models/sandbox.py` defines a `SandboxEngine` ABC plus three concrete engines (`_PostgresEngine` `postgres:16-alpine`, `_RedisEngine` `redis:8-alpine`, `_MongoEngine` `mongo:8-alpine`); each engine declares its image, run args, readiness probe, connection shape, and `ROBOCO_TEST_*` env emission. `VALID_SANDBOX_SERVICES = frozenset(SANDBOX_ENGINES)` is the single source of truth — derived from the registry, shared by the pydantic validators in `models/project.py` and the orchestrator-side provisioner, so they can never drift. The provisioner (`roboco/runtime/sandbox.py`) iterates the registry via a generic `_provision_engine`; the orchestrator's `_append_sandbox_env` collapses to `cmd.extend(info.emit_env())`; teardown iterates every engine. Adding a sandbox engine is now one class + one registry line — no branch edited in the provisioner or the env emitter, and the panel's edit-project dialog surfaces it via a `SANDBOX_SERVICES` catalog. Mongo rides the existing `projects.sandbox_services` opt-in (migration 057) with no new migration and no new feature flag; it adds `ROBOCO_TEST_MONGO_*` (+ `ROBOCO_TEST_MONGO_AUTH_DB=admin`). Env var names `ROBOCO_TEST_DB_*` / `ROBOCO_TEST_REDIS_*` are preserved so existing project conftests need no change. The panel's sandbox toggles became a `Set` multi-select driven by the catalog. + - **RoboCo video engine (default-off).** With `ROBOCO_VIDEO_ENGINE_ENABLED`, a release/feature-spotlight/on-demand CEO trigger opens a normal, assigned UX/UI authoring task (balanced across the two ux-devs) instead of a held draft — the dev builds a HyperFrames HTML composition under `motion/compositions//` and proposes its composition id + per-platform captions via the team-gated `propose_video` do-tool, then ships it through the standard commit/PR/QA/doc/review lifecycle. Once that task completes, an orchestrator render loop tars the merged `motion/` source to a new credential-free `video-renderer` sidecar, renders both the 9:16 and 1:1 MP4 cuts, and materializes a held `video_post` draft (mirroring the X-post/release-proposal shape: Secretary-owned, skipped by every dispatcher). The CEO previews, edits, approves, or rejects each draft in a new panel video queue; approving posts the rendered clip to X (native video, v2 media upload) and/or TikTok (inbox upload) under a heartbeat-renewed lock and is idempotent — an already-posted draft is a no-op. `ROBOCO_VIDEO_ON_RELEASE` / `ROBOCO_VIDEO_ON_SPOTLIGHT` gate the two automatic triggers independently of the CEO's on-demand `POST /video/request`; TikTok's OAuth2 secrets live Fernet-encrypted alongside the existing X credentials, and every unconfigured leg (renderer, X, TikTok) degrades to a graceful no-op rather than a crash. Rendered MP4s persist under `ROBOCO_VIDEO_OUTPUT_DIR` (bind-mounted in all three compose files so renders survive container recreation). - **Per-project video-engine opt-in.** `projects.video_engine_enabled` (migration 063, mirroring `ci_watch_enabled`): the global `ROBOCO_VIDEO_ENGINE_ENABLED` flag arms the subsystem, the per-project flag opts a repo into authoring against its `motion/` dir — `VideoEngine._opted_in_project` no-ops `open_video_task` until the operator flips it in the panel's edit-project dialog. Existing projects stay opted out. - **MinIO object storage scaffolding (default-off).** `ROBOCO_MINIO_*` config (`minio_endpoint`, `minio_access_key`, `minio_secret_key`, `minio_bucket`, `minio_region`) + a `minio` service and a one-shot `minio-init` (idempotent bucket create) in the NAS compose files, on the `data` network with a named `minio-data` volume; `minio` (minio-py) added as a dependency. Empty `minio_endpoint` = disabled and the existing `FileResponse` media-serve path is byte-for-byte unchanged — this is scaffolding; the write path (PUT after local save) and serve path (`StreamingResponse` with `FileResponse` fallback) land in later chunks. The registry compose omits MinIO entirely (NAS default-on, registry default-off). @@ -20,6 +22,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Fixed +- **Sandbox cold-pull loop + empty provisioning error string.** `docker run` pulled the sandbox image inline under a 20s run deadline, so a NAS cold pull was killed, the pull cancelled, and every retry re-pulled from scratch — a persistent loop that stranded v0.19.0 board-agent spawns with `"error": ""` (a bare `TimeoutError` stringifies to `""`). `_ensure_image` now inspects the image and pulls it under a 300s deadline before `docker run`, and provisioning failures log `f"{type(e).__name__}: {e}"` so the error is never an empty string. + - **Conventions + release-readiness I/O no longer blocks the API event loop.** `ConventionsService.get_map/health/restore` and `ReleaseManagerEngine._production_assess` ran sync `git rev-parse`, filesystem walks, and yaml parses inline on the orchestrator's shared uvicorn event loop, stalling API responsiveness during conventions reads (reachable from `GET /api/projects/{id}/conventions` and the agent spawn-prepare path) and the release-manager background loop. Each blocking call is now wrapped in `asyncio.to_thread` at the async boundary; no signature changes. A concurrency audit confirmed the rest of the heavy paths (agent spawn via `docker run -d`, the video render loop, git ops via the 16-worker ThreadPoolExecutor, workspace subprocess calls) already offload correctly — no API/worker container split is warranted. - **Deleted journals de-indexed from RAG (C3).** `JournalsIndexPlugin.delete_entry` now calls `OptimalService.unindex_journal_entry(entry_id)`, which removes the entry's embedded chunks from the `chunks_journals` vector store AND drops its `indexed_documents` tracking row — a deleted (or private) entry stops surfacing in RAG answers and agent briefings. Pre-fix the chunks were orphaned, so deleted/private content bled into briefings. - **Learning `learning_id` hashes full content (M25).** The dedup `learning_id` was derived from a thin slice of the lesson, so two distinct lessons that shared the prefix collided and the second silently overwrote the first. The id now hashes the full content fields, so each distinct lesson persists as its own row. diff --git a/CLAUDE.md b/CLAUDE.md index 2707cc2b..795b300b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -381,7 +381,7 @@ Agent backends are pluggable. `roboco/llm/providers/` defines an `AgentProvider` **Organizational memory loop (default-off).** Closes the learn→reuse loop so agents stop cold-respawning blind. Three parts, all gated by `ROBOCO_ORG_MEMORY_ENABLED`: ① **capture** — at task completion `TaskService._completion_learnings_for` distills ONE high-signal lesson (Problem→Approach→Gotcha, ≤120 words) via the local model (`MemoryDistiller`, `roboco/services/memory_distiller.py`) instead of the noisy raw-notes/duration capture (flag-off keeps the legacy capture); journal indexing excludes `is_private` reflections from the shared corpus. ② **retrieve (keystone)** — on claim, `_briefing_for` injects `context_briefing["institutional_memory"]`: top-K (`ROBOCO_ORG_MEMORY_TOP_K`) relevance-floored (`ROBOCO_ORG_MEMORY_MIN_SCORE`) lessons + approved playbooks from a role-shaped query (`EvidenceRepo.similar_memory` over the LEARNINGS + PLAYBOOKS pgvector indexes); below the floor nothing is injected (no briefing bloat). ③ **playbooks** — a first-class curated procedure store: `PlaybookTable` (migration 050), the `PLAYBOOKS` OptimalService index, the `draft_playbook` content verb (delivery roles), Auditor `approve_playbook`/`reject_playbook`/`archive_playbook` curation (approval indexes it), and the panel review queue (`playbook-review-queue.tsx`; `/api/playbooks` Auditor/CEO routes). Distillation runs on the local model only — never a cloud LLM in the hot path; every step is best-effort (a failure never blocks completion or the briefing). -**Sandboxed dev DB/Redis (default-off).** Per-project opt-in (`projects.sandbox_services`, migration 057): when armed (`ROBOCO_SANDBOX_DB_ENABLED`), each opted-in project's agent spawn gets orchestrator-provisioned throwaway `postgres:16-alpine` / `redis:8-alpine` **sibling containers** (random per-sandbox creds, tmpfs pg data dir, memory/cpu-capped, labeled `roboco.sandbox=1`), injected as `ROBOCO_TEST_DB_*` / `ROBOCO_TEST_REDIS_*` **in place of** the legacy prod-creds gate-env injection (`_append_gate_env`, which points agents at RoboCo's own production Postgres under `ROBOCO_TOOLCHAIN_MATCH_ENABLED`) — sandbox replaces, never coexists with, prod creds. Lifetime tracks the agent container 1:1: teardown at every removal path plus an orphan janitor at startup + each reaper tick (grace-windowed so a sweep can't reap a sandbox whose spawn is still mid-flight; the pre-spawn stale-clear likewise spares the just-provisioned sandbox). Provisioning failure refuses the spawn (fail-loud); docker-in-agent stays structurally absent. `SandboxProvisioner` (`roboco/runtime/sandbox.py`), wired in the orchestrator spawn path. +**Sandboxed dev DB/Redis/Mongo (default-off).** Per-project opt-in (`projects.sandbox_services`, migration 057): when armed (`ROBOCO_SANDBOX_DB_ENABLED`), each opted-in project's agent spawn gets orchestrator-provisioned throwaway **sibling containers** (random per-sandbox creds, tmpfs data dir, memory/cpu-capped, labeled `roboco.sandbox=1`) — one per opted-in service — injected as `ROBOCO_TEST_DB_*` / `ROBOCO_TEST_REDIS_*` / `ROBOCO_TEST_MONGO_*` **in place of** the legacy prod-creds gate-env injection (`_append_gate_env`, which points agents at RoboCo's own production Postgres under `ROBOCO_TOOLCHAIN_MATCH_ENABLED`) — sandbox replaces, never coexists with, prod creds. Lifetime tracks the agent container 1:1: teardown at every removal path plus an orphan janitor at startup + each reaper tick (grace-windowed so a sweep can't reap a sandbox whose spawn is still mid-flight; the pre-spawn stale-clear likewise spares the just-provisioned sandbox). Provisioning failure refuses the spawn (fail-loud); docker-in-agent stays structurally absent. `SandboxProvisioner` (`roboco/runtime/sandbox.py`), wired in the orchestrator spawn path. The service set is a **pluggable engine registry** (`roboco/models/sandbox.py`): each engine declares its image, run args, readiness probe, and `ROBOCO_TEST_*` env; `VALID_SANDBOX_SERVICES` is derived from the registry, and the provisioner + orchestrator env injection iterate it, so adding an engine (e.g. mongo) is one class + one registry line — no branch edited in the provisioner or the env emitter. **Cloud auth via FastAPI Users (default-off).** Lets the panel/API be safely exposed beyond localhost without touching the CEO's local no-login flow while off. Gated by `ROBOCO_CLOUD_AUTH_ENABLED` (+ `ROBOCO_CLOUD_AUTH_EMAIL` / `_PASSWORD` / `_SECRET` / `_COOKIE_MAX_AGE`; `Settings` fails loud at startup if the flag is on with no secret). Off: `get_agent_context` (`roboco/api/deps.py`) and the WS `_require_panel_token` gate (`roboco/api/websocket.py`) are byte-for-byte unchanged (header-trust). On: header-trust is dead for humans — any agent-role claim (`ceo` OR a privileged `main_pm`/`cell_pm`/board role) with no valid HMAC token or session cookie is 401, closing the header-spoof hole on the host-published `:8000` port for every role, not just `ceo` (real agents always carry a signed token, so they're unaffected); the agent-fleet HMAC path (and the orchestrator's `system` self-PATCH) keeps working unmodified in both modes; a valid session cookie authenticates as the single seeded CEO user. New `users` table (migration 058, `UserTable` in `roboco/db/tables.py`) backs FastAPI Users' `SQLAlchemyUserDatabase`; no registration router — `roboco/api/auth/seed.py` idempotently upserts exactly one row from env at startup (by primary key, so an email change renames the row instead of duplicating it). `roboco/api/auth/backend.py` wires a **cookie** transport (httponly, secure, samesite=lax) + a `JWTStrategy` subclass that binds each token to a fingerprint of the current `hashed_password`, so rotating the seeded password invalidates every prior session. Session lifetime is **sliding**: every authenticated request through `get_agent_context` re-mints + re-sets the cookie (`_slide_session_cookie`), so an active session never expires — only genuine inactivity past `cloud_auth_cookie_max_age` (default 30 days) logs out. `GET /api/auth/status` is always mounted (public); `/api/auth/login` + `/api/auth/logout` mount only when armed (`roboco/api/auth/routes.py`, mirroring `apply_guard`'s conditional mount). Panel: `(auth)/login/page.tsx` + `proxy.ts` (the Next 16 rename of `middleware.ts`; probes `/auth/status` over the docker-internal orchestrator URL, not through nginx, and fails open to "off" on any probe error/timeout) gate the `(dashboard)` group; `client.ts` adds `withCredentials` + a 401→`/login` redirect. nginx needs no changes (`/api/auth/*` rides the existing `/api/` proxy location) — but its own static `X-Agent-Token` injection (`ROBOCO_PANEL_AGENT_TOKEN`) is itself a valid HMAC credential that bypasses login when present, so a deployment arming cloud auth for real public exposure should leave that token unset (the two are alternative human-auth tiers, not layered). diff --git a/docs/map/_complete_map.md b/docs/map/_complete_map.md index 9b0b93bf..56be795d 100644 --- a/docs/map/_complete_map.md +++ b/docs/map/_complete_map.md @@ -620,7 +620,7 @@ Backend: `EventType.A2A_MESSAGE_SENT` published from `A2AService.send` (excerpt- ## Delta 2026-07-03 (5) — wave 3 → v0.17.0 (branch `feat/wave-3`, SDD/Sonnet 5, reviewed) Six subsystems, all default-off + additive: -1. **Sandboxed dev DB/Redis** (`ROBOCO_SANDBOX_DB_ENABLED`, migration 057 `projects.sandbox_services`): `SandboxProvisioner` (`roboco/runtime/sandbox.py`) `docker run`s throwaway `postgres:16`/`redis:8` sibling containers per spawn (random creds, tmpfs, labeled), injecting `ROBOCO_TEST_DB_*`/`ROBOCO_TEST_REDIS_*` in place of the prod-creds gate-env. Container-tracked lifetime + orphan janitor (grace-windowed). REVIEW FIX: pre-spawn stale-clear must not tear down the just-provisioned sandbox (`teardown_sandbox=False`) + provision pre-clears stale + janitor grace. +1. **Sandboxed dev DB/Redis/Mongo** (`ROBOCO_SANDBOX_DB_ENABLED`, migration 057 `projects.sandbox_services`): `SandboxProvisioner` (`roboco/runtime/sandbox.py`) `docker run`s throwaway `postgres:16-alpine`/`redis:8-alpine`/`mongo:8-alpine` sibling containers per spawn (random creds, tmpfs, labeled), injecting `ROBOCO_TEST_DB_*`/`ROBOCO_TEST_REDIS_*`/`ROBOCO_TEST_MONGO_*` in place of the prod-creds gate-env. The service set is a pluggable engine registry (`roboco/models/sandbox.py`: `SandboxEngine` ABC + `_PostgresEngine`/`_RedisEngine`/`_MongoEngine`, `SANDBOX_ENGINES` / `VALID_SANDBOX_SERVICES` derived from it); adding an engine is one class + one registry line — no provisioner or env-emitter branch. Container-tracked lifetime + orphan janitor (grace-windowed). REVIEW FIX: pre-spawn stale-clear must not tear down the just-provisioned sandbox (`teardown_sandbox=False`) + provision pre-clears stale + janitor grace; `_ensure_image` inspects+pulls (300s) before `docker run` so a NAS cold pull isn't killed at the 20s run deadline. 2. **DB network isolation** (`ROBOCO_DB_NETWORK_ISOLATED`): second `roboco_data` compose bridge = postgres+redis only, orchestrator multi-homed; agents can't reach prod DB. Suppresses `_append_gate_env` prod-creds injection. In BOTH build+registry composes (topology-coupled). 3. **Mobile UI** (panel): `useIsMobile` (useSyncExternalStore, hydration-safe), `ResponsiveTable` table→card, snap `TabsList`, bottom tab bar, Comms/A2A single-pane drill-down below lg, vh→dvh. REVIEW FIX: `justify-center-safe` (overflow clip), memoized matchMedia subscribe. 4. **Cloud auth** (`ROBOCO_CLOUD_AUTH_ENABLED`, migration 058 `users`): FastAPI Users, single seeded user, cookie sliding 30-day session (pwd-fingerprint JWT), `get_agent_context` dual-path (deps.py). Off = byte-identical. `proxy.ts` (Next 16). REVIEW FIX: on-mode rejects EVERY non-CEO role without a token (not just ceo — closed the PM/board :8000 spoof). @@ -1773,7 +1773,7 @@ models/ ## Config Flags -None — pure models, no flags. (The `Project` model *carries* opt-in fields `ci_watch_enabled`, `dep_update_command`, `dep_update_paths`, `sandbox_services` (project.py:142, validated against `VALID_SANDBOX_SERVICES` — sandboxed dev DB/Redis, gated by `ROBOCO_SANDBOX_DB_ENABLED` elsewhere) that other layers gate on, and `llm_catalog` carries the "pure Ollama" defaults, but the models package itself reads no env / toggles nothing.) +None — pure models, no flags. (The `Project` model *carries* opt-in fields `ci_watch_enabled`, `dep_update_command`, `dep_update_paths`, `sandbox_services` (project.py:142, validated against `VALID_SANDBOX_SERVICES` — sandboxed dev DB/Redis/Mongo, gated by `ROBOCO_SANDBOX_DB_ENABLED` elsewhere) that other layers gate on, and `llm_catalog` carries the "pure Ollama" defaults, but the models package itself reads no env / toggles nothing.) `VALID_SANDBOX_SERVICES` is now derived from the `SANDBOX_ENGINES` registry in `roboco/models/sandbox.py` (was a hardcoded `{"postgres","redis"}` set) — mongo is just another registry entry, no new migration (rides existing 057) and no new feature flag. ## Gotchas @@ -1946,7 +1946,7 @@ The DB layer is async SQLAlchemy 2.0 over PostgreSQL+asyncpg, with pgvector for | 054 | 054_a2a_message_skill.py | `a2a_messages.skill` (String 100, nullable) — persists the capability a directed A2A message concerns; was silently dropped on send. | | 055 | 055_spawn_session_turns_tool_calls.py | `agent_spawn_sessions.turns` + `.tool_calls` (BigInteger, DEFAULT 0) — per-stint LLM iterations + tool invocations for the granular per-member performance metrics. | | 056 | 056_member_perf_daily.py | `member_performance_daily` — one row per (date, member_kind, agent_slug) scorecard rollup (incl. CEO as `member_kind='ceo'`). | -| 057 | 057_project_sandbox_services.py | `projects.sandbox_services` (ARRAY(String), nullable) — per-project opt-in for the sandboxed per-agent-spawn Postgres/Redis provisioner. | +| 057 | 057_project_sandbox_services.py | `projects.sandbox_services` (ARRAY(String), nullable) — per-project opt-in for the sandboxed per-agent-spawn engine provisioner (postgres / redis / mongo via the `SANDBOX_ENGINES` registry). | | 058 | 058_cloud_auth_users.py | `users` table (FastAPI Users schema) — the single seeded CEO login for cloud auth (`ROBOCO_CLOUD_AUTH_ENABLED`, default off). | | 059 | 059_x_credentials.py | `x_credentials` (singleton Fernet-encrypted OAuth 1.0a secrets) + `x_seen_mentions` (mentions-poll dedup ledger) — the X (Twitter) engine (`ROBOCO_X_ENGINE_ENABLED`, default off). | | 060 | 060_drop_messaging.py | Drops the channels/groups/sessions/session_tasks/messages subsystem (comms teardown — A2A is now the sole directed-message channel): `journal_entries.session_id` column, the 5 tables, and 4 enum types (`messagetype`/`sessionstatus`/`sessionscope`/`channeltype`); one-way (`downgrade()` raises `NotImplementedError`). | @@ -2108,7 +2108,7 @@ Migration chain 001..059 > Post-snapshot updates (since 2026-06-29): `536bbb64` (Chore/all/logical gaps sweep #286) — adds migration 053 (`playbooks.archived_by`/`archived_at`), two new columns on `PlaybookTable`; `d8a5bb48` ([chore] a2a hierarchy gate + skill persist) — adds migration 054 (`a2a_messages.skill`), one new column on `A2AMessageTable`, wired through `send_chat_message` and the A2AChatMessage model. > -> Delta 2026-07-03 (v0.17.0, 5 features): `055_spawn_session_turns_tool_calls` (`agent_spawn_sessions.turns`/`.tool_calls`) + `056_member_perf_daily` (`member_performance_daily`) predate this wave but were never appended to this doc; `057_project_sandbox_services` adds `projects.sandbox_services` (sandboxed dev DB/Redis, `ROBOCO_SANDBOX_DB_ENABLED`); `058_cloud_auth_users` adds `users` (`UserTable`, cloud auth, `ROBOCO_CLOUD_AUTH_ENABLED`); `059_x_credentials` adds `x_credentials` (`XCredentialsTable`) + `x_seen_mentions` (`XSeenMentionTable`) (X engine, `ROBOCO_X_ENGINE_ENABLED`). Chain head is now 059. +> Delta 2026-07-03 (v0.17.0, 5 features): `055_spawn_session_turns_tool_calls` (`agent_spawn_sessions.turns`/`.tool_calls`) + `056_member_perf_daily` (`member_performance_daily`) predate this wave but were never appended to this doc; `057_project_sandbox_services` adds `projects.sandbox_services` (sandboxed dev DB/Redis/Mongo, `ROBOCO_SANDBOX_DB_ENABLED`); `058_cloud_auth_users` adds `users` (`UserTable`, cloud auth, `ROBOCO_CLOUD_AUTH_ENABLED`); `059_x_credentials` adds `x_credentials` (`XCredentialsTable`) + `x_seen_mentions` (`XSeenMentionTable`) (X engine, `ROBOCO_X_ENGINE_ENABLED`). Chain head is now 059. > > Delta 2026-07-04 (v0.18.0): `060_drop_messaging` (the comms-teardown migration — drops `messages`/`session_tasks`/`sessions`/`groups`/`channels` + 4 enum types + `journal_entries.session_id`; A2A is now the sole directed-message channel; one-way, `downgrade()` raises `NotImplementedError`) had already landed on master but was never appended to this doc; `061_x_feature_spotlight` adds `x_seen_features` (`XSeenFeatureTable`) + `company_goals.brand_voice` (X feature-spotlight, `ROBOCO_X_FEATURE_SPOTLIGHT_ENABLED`, sub-switch of `x_engine_enabled`). Chain head is now 061. ORM table count is now 38 (verified via `grep -c '^class .*Table' roboco/db/tables.py`), up from this doc's previously-stated 37 (that figure predates 055-061 and was never recomputed). @@ -4541,7 +4541,7 @@ Panel-tunable flags defined in `services/settings.py:46` `FEATURE_FLAGS` (stored | `dep_update_enabled` | Dependency-update bot | `ROBOCO_DEP_UPDATE_ENABLED` | | `release_manager_enabled` | Gated release manager | `ROBOCO_RELEASE_MANAGER_ENABLED` | | `org_memory_enabled` | Organizational memory loop | `ROBOCO_ORG_MEMORY_ENABLED` | -| `sandbox_db_enabled` | Sandboxed per-agent test DB/Redis | `ROBOCO_SANDBOX_DB_ENABLED` | +| `sandbox_db_enabled` | Sandboxed per-agent test DB/Redis/Mongo (engine registry) | `ROBOCO_SANDBOX_DB_ENABLED` | | `x_engine_enabled` | X (Twitter) engine | `ROBOCO_X_ENGINE_ENABLED` | | `roadmap_engine_enabled` | Board roadmap engine | `ROBOCO_ROADMAP_ENGINE_ENABLED` | @@ -4609,7 +4609,7 @@ No files in this slice changed between `fd10cc86` and `HEAD`, so there are no *r This slice is a mature, mostly-stable support layer: the service-base/error hierarchy and crypto/UUID helpers are well-factored and widely reused; the Redis-Streams event bus is correctly durable (consumer groups, ACK-on-success, pending recovery) with the one real caveat that handler idempotency is the caller's job. Post-snapshot commits improved the bus (poison-pill dead-letter + periodic reclaim loop + `BaseException` marker cleanup), typed the UUID error surface (`InvalidIdentifierError`), and removed the vestigial `_find_code_patterns` call from `ProactiveKnowledgeService`. `TranscriptionService` (sync-callback + unbounded-buffer risks) remains the softest spot. Model routing's fail-safe-quietly design is intentional (a stalled spawn is worse than a wrong provider) but shifts diagnosis to logs. Overall integrity: solid, with `TranscriptionService` the one service worth either finishing or marking clearly as legacy. ## Purpose -The AgentOrchestrator is the runtime brain of RoboCo: it owns the per-agent Docker container lifecycle, the per-tick dispatcher that matches tasks to agents, the stale-claim reaper, the provider rate-limit/overload park-and-probe recovery loop, and the default-off background engines (self-heal, CI-watch, dep-update, release-manager, strategy, external-PR poll, X-engine mentions poll, board roadmap engine). It claims tasks on behalf of agents before spawning, injects briefings/manifests/git context at spawn time, provisions a per-spawn sandbox DB/Redis when opted in, captures per-session token usage, and persists durable runtime state (WaitingRecord, respawn_tracker) across restarts. +The AgentOrchestrator is the runtime brain of RoboCo: it owns the per-agent Docker container lifecycle, the per-tick dispatcher that matches tasks to agents, the stale-claim reaper, the provider rate-limit/overload park-and-probe recovery loop, and the default-off background engines (self-heal, CI-watch, dep-update, release-manager, strategy, external-PR poll, X-engine mentions poll, board roadmap engine). It claims tasks on behalf of agents before spawning, injects briefings/manifests/git context at spawn time, provisions a per-spawn sandbox (postgres/redis/mongo via the engine registry) when opted in, captures per-session token usage, and persists durable runtime state (WaitingRecord, respawn_tracker) across restarts. ## Files @@ -4690,8 +4690,8 @@ The AgentOrchestrator is the runtime brain of RoboCo: it owns the per-agent Dock | AgentOrchestrator._should_skip_live_reap | method | roboco/runtime/orchestrator.py:8750 | Spare a live container from reaping UNLESS wedged (grok) or gateway-broken past grace (those kill+evict). | | AgentOrchestrator._assignee_is_provider_parked | method | roboco/runtime/orchestrator.py:8527 | True if a task's assignee is provider-parked; reaper skips it so the claim survives for probe-resume. | | AgentOrchestrator._reap_with_service | method | roboco/runtime/orchestrator.py:8770 | Inner stale-claim reaper: skip live (unless wedged/broken), skip provider-parked, unclaim_for_reaper the rest. | -| AgentOrchestrator._maybe_provision_sandbox | method | roboco/runtime/orchestrator.py:2029 | Provision this spawn's sandbox DB/Redis via `SandboxProvisioner` when `sandbox_db_enabled` + the project's `sandbox_services` are set; None (byte-for-byte legacy path) otherwise. Fail-loud once opted in — a provisioning failure refuses the spawn. | -| AgentOrchestrator._append_sandbox_env | staticmethod | roboco/runtime/orchestrator.py:2639 | Inject `ROBOCO_TEST_DB_*`/`ROBOCO_TEST_REDIS_*` env from `config.sandbox_info`, called INSTEAD OF `_append_gate_env` whenever a sandbox was provisioned for this spawn. | +| AgentOrchestrator._maybe_provision_sandbox | method | roboco/runtime/orchestrator.py:2029 | Provision this spawn's sandbox engines (postgres/redis/mongo via `SandboxProvisioner` + the `SANDBOX_ENGINES` registry) when `sandbox_db_enabled` + the project's `sandbox_services` are set; None (byte-for-byte legacy path) otherwise. Fail-loud once opted in — a provisioning failure refuses the spawn. | +| AgentOrchestrator._append_sandbox_env | staticmethod | roboco/runtime/orchestrator.py:2639 | Inject the sandbox's `ROBOCO_TEST_*` env (postgres `ROBOCO_TEST_DB_*` / redis `ROBOCO_TEST_REDIS_*` / mongo `ROBOCO_TEST_MONGO_*`) via `SandboxInfo.emit_env` over the engine registry, called INSTEAD OF `_append_gate_env` whenever a sandbox was provisioned for this spawn. Emission is registry-driven, so a new engine's env vars land here with no orchestrator change. | | AgentOrchestrator._sandbox_janitor_sweep | method | roboco/runtime/orchestrator.py:9426 | Best-effort: remove sandbox containers whose owner agent is gone; rides the reaper tick, error-isolated. | | AgentOrchestrator._x_mentions_poll_loop | method | roboco/runtime/orchestrator.py:7431 | Default-off X-engine mentions-poll tick loop (`x_engine_enabled`); release-post drafts are event-driven, not from this loop. | | AgentOrchestrator._run_x_mentions_cycle | method | roboco/runtime/orchestrator.py:7453 | One mentions-poll pass: `get_x_engine(db).run_cycle()` + commit; testable without the sleep. | @@ -4807,7 +4807,7 @@ stateDiagram-v2 - `ROBOCO_GROK_MAX_COST_USD` — grok budget kill-switch; `_GROK_RATE_LIMIT_EXIT_CODE=75`, `_GROK_AUTH_EXIT_CODE=78`, `_PROBE_GIVE_UP_THRESHOLD=30`. - `ROBOCO_CLAUDE_STUCK_KILL_SECONDS` (default 3600, min 600) — heartbeat-stale kill threshold for non-GROK agents; controls `_maybe_kill_stuck_claude`. - `ROBOCO_DISPATCHER_INTERVAL_SECONDS` (30), `ROBOCO_INTERACTIVE_IDLE_REAP_SECONDS`, `ROBOCO_GROK_*` backoff constants. -- `ROBOCO_SANDBOX_DB_ENABLED` (default off) — master switch for the sandboxed per-agent-spawn Postgres/Redis provisioner (`_maybe_provision_sandbox`/`_append_sandbox_env`/`_sandbox_janitor_sweep`); a project participates only when its `sandbox_services` column is also set. +- `ROBOCO_SANDBOX_DB_ENABLED` (default off) — master switch for the sandboxed per-agent-spawn engine provisioner (`_maybe_provision_sandbox`/`_append_sandbox_env`/`_sandbox_janitor_sweep`); a project participates only when its `sandbox_services` column is also set. The service set is the `SANDBOX_ENGINES` registry (postgres / redis / mongo); adding an engine needs no orchestrator or env-emitter change. - `ROBOCO_DB_NETWORK_ISOLATED` (default off; set by the compose topology that carries the `roboco_data` network) — suppresses the legacy `_append_gate_env` prod-creds injection when postgres/redis are unreachable from the agent mesh. - `ROBOCO_CLOUD_AUTH_ENABLED` (+ `_EMAIL`/`_PASSWORD`/`_SECRET`/`_COOKIE_MAX_AGE`, default off) — cloud auth master switch; read by `roboco.api.deps.get_agent_context`/`roboco.api.auth.*`, not the orchestrator itself, but gates whether a spawned agent's own HMAC-token identity path is the sole non-CEO auth route. - `ROBOCO_X_ENGINE_ENABLED` (+ `_mentions_interval_seconds` / `_mentions_max_per_cycle` / `_mentions_min_engagement` / `_max_open_posts` / `_account_user_id` / `_request_timeout_seconds`, default off) — gates `_x_mentions_poll_loop`. @@ -4873,7 +4873,7 @@ This slice is the agent-runtime + LLM-provider seam plus the in-container agent | roboco/runtime/spawn_manifest.py | Builds the per-role /app/tool-manifest.json (allowed verbs/tools, env) from role_config | 85 | | roboco/runtime/streaming.py | Global reasoning-stream callback holder+setter for live UI streaming | 53 | | roboco/runtime/transcript_retention.py | Pure selector of agent-owned old Claude transcripts to prune (never operator dirs) | 74 | -| roboco/runtime/sandbox.py | `SandboxProvisioner` — throwaway per-agent-spawn Postgres/Redis sibling containers (orchestrator-side, never docker-in-agent); provision/teardown/janitor_sweep, standalone + unit-testable via an injected `DockerRunner` | 344 | +| roboco/runtime/sandbox.py | `SandboxProvisioner` — throwaway per-agent-spawn engine sibling containers (postgres/redis/mongo via the `SANDBOX_ENGINES` registry in `roboco/models/sandbox.py`, orchestrator-side, never docker-in-agent); generic `_provision_engine` per engine, provision/teardown/janitor_sweep, standalone + unit-testable via an injected `DockerRunner` | 344 | | roboco/llm/__init__.py | Re-exports ToonAdapter/ToonMetrics singletons | 17 | | roboco/llm/metrics.py | Singleton holder for TOON token-savings metrics | 21 | | roboco/llm/toon_adapter.py | TOON serialization adapter for token-efficient LLM communication (JSON fallback) | 188 | @@ -4911,7 +4911,7 @@ This slice is the agent-runtime + LLM-provider seam plus the in-container agent | stream_reasoning | function | roboco/runtime/streaming.py:39 | Stream a reasoning chunk to the registered callback if any | | is_agent_owned_dir | function | roboco/runtime/transcript_retention.py:23 | True if a ~/.claude/projects subdir was written by a spawned agent (-app or encoded workspaces root prefix, boundary-aware) | | select_prunable_transcripts | function | roboco/runtime/transcript_retention.py:59 | Pure selector of agent-owned *.jsonl transcripts older than cutoff_epoch (never operator dirs) | -| SandboxProvisioner | class | roboco/runtime/sandbox.py:90 | Per-agent-spawn throwaway Postgres/Redis provisioner; `provision`/`teardown`/`janitor_sweep`, docker plumbing is an injected `DockerRunner` callable | +| SandboxProvisioner | class | roboco/runtime/sandbox.py:90 | Per-agent-spawn throwaway engine provisioner (iterates `SANDBOX_ENGINES`); `provision`/`teardown`/`janitor_sweep`, docker plumbing is an injected `DockerRunner` callable. Engine specs live in `roboco/models/sandbox.py` (`SandboxEngine` ABC + `_PostgresEngine`/`_RedisEngine`/`_MongoEngine`) | | ToonAdapter | class | roboco/llm/toon_adapter.py:33 | TOON serialization adapter: encode/decode with JSON fallback, prompt formatting, token-savings estimate | | get_toon_adapter | function | roboco/llm/toon_adapter.py:184 | Singleton ToonAdapter accessor | | SpawnResult | dataclass | roboco/llm/providers/base.py:26 | Provider spawn result: instance_id, initial agent_state, extra metadata | @@ -8170,7 +8170,7 @@ panel/ (Next.js 16, package roboco-panel v0.14.0) - `dep_update_enabled` — dependency-update bot - `release_manager_enabled` — gated release manager - `org_memory_enabled` — organizational memory loop -- `sandbox_db_enabled` — sandboxed per-agent test DB/Redis +- `sandbox_db_enabled` — sandboxed per-agent test DB/Redis/Mongo (engine registry) - `x_engine_enabled` — X (Twitter) engine (release-post + mention-reply drafts, all CEO-held) - `roadmap_engine_enabled` — board roadmap engine (weekly Product-Owner-authored cycle) - `routing_strict` — fail-closed model routing (refuse to silently downgrade to the legacy Anthropic path on a disabled provider) @@ -8486,7 +8486,7 @@ deployment-tooling - ROBOCO_DEP_UPDATE_ENABLED / _INTERVAL_SECONDS / _MAX_OPEN_TASKS / _MAX_PER_CYCLE - ROBOCO_RELEASE_MANAGER_ENABLED / _MIN_COMMITS / _INTERVAL_SECONDS / _CI_WORKFLOW - ROBOCO_ORG_MEMORY_ENABLED / _TOP_K / _MIN_SCORE -- ROBOCO_SANDBOX_DB_ENABLED — sandboxed per-agent-spawn Postgres/Redis provisioner (`roboco/runtime/sandbox.py`); a project also needs its `sandbox_services` column set +- ROBOCO_SANDBOX_DB_ENABLED — sandboxed per-agent-spawn engine provisioner (`roboco/runtime/sandbox.py` + registry in `roboco/models/sandbox.py`); a project also needs its `sandbox_services` column set - ROBOCO_DB_NETWORK_ISOLATED — set true only by the compose topology carrying the `roboco_data` data-only network; suppresses the legacy prod-creds gate-env injection - ROBOCO_CLOUD_AUTH_ENABLED / _EMAIL / _PASSWORD / _SECRET / _COOKIE_MAX_AGE — FastAPI Users cookie login for the single seeded CEO; `Settings` fails loud at startup if armed with no secret - ROBOCO_X_ENGINE_ENABLED / _MENTIONS_INTERVAL_SECONDS / _MENTIONS_MAX_PER_CYCLE / _MENTIONS_MIN_ENGAGEMENT / _MAX_OPEN_POSTS / ROBOCO_X_ACCOUNT_USER_ID / _REQUEST_TIMEOUT_SECONDS — the X (Twitter) engine; inert without stored OAuth 1.0a credentials regardless of the flag diff --git a/docs/map/_front.md b/docs/map/_front.md index ca05a4cb..820d5217 100644 --- a/docs/map/_front.md +++ b/docs/map/_front.md @@ -619,7 +619,7 @@ Backend: `EventType.A2A_MESSAGE_SENT` published from `A2AService.send` (excerpt- ## Delta 2026-07-03 (5) — wave 3 → v0.17.0 (branch `feat/wave-3`, SDD/Sonnet 5, reviewed) Six subsystems, all default-off + additive: -1. **Sandboxed dev DB/Redis** (`ROBOCO_SANDBOX_DB_ENABLED`, migration 057 `projects.sandbox_services`): `SandboxProvisioner` (`roboco/runtime/sandbox.py`) `docker run`s throwaway `postgres:16`/`redis:8` sibling containers per spawn (random creds, tmpfs, labeled), injecting `ROBOCO_TEST_DB_*`/`ROBOCO_TEST_REDIS_*` in place of the prod-creds gate-env. Container-tracked lifetime + orphan janitor (grace-windowed). REVIEW FIX: pre-spawn stale-clear must not tear down the just-provisioned sandbox (`teardown_sandbox=False`) + provision pre-clears stale + janitor grace. +1. **Sandboxed dev DB/Redis/Mongo** (`ROBOCO_SANDBOX_DB_ENABLED`, migration 057 `projects.sandbox_services`): `SandboxProvisioner` (`roboco/runtime/sandbox.py`) `docker run`s throwaway `postgres:16-alpine`/`redis:8-alpine`/`mongo:8-alpine` sibling containers per spawn (random creds, tmpfs, labeled), injecting `ROBOCO_TEST_DB_*`/`ROBOCO_TEST_REDIS_*`/`ROBOCO_TEST_MONGO_*` in place of the prod-creds gate-env. The service set is a pluggable engine registry (`roboco/models/sandbox.py`: `SandboxEngine` ABC + `_PostgresEngine`/`_RedisEngine`/`_MongoEngine`, `SANDBOX_ENGINES` / `VALID_SANDBOX_SERVICES` derived from it); adding an engine is one class + one registry line — no provisioner or env-emitter branch. Container-tracked lifetime + orphan janitor (grace-windowed). REVIEW FIX: pre-spawn stale-clear must not tear down the just-provisioned sandbox (`teardown_sandbox=False`) + provision pre-clears stale + janitor grace; `_ensure_image` inspects+pulls (300s) before `docker run` so a NAS cold pull isn't killed at the 20s run deadline. 2. **DB network isolation** (`ROBOCO_DB_NETWORK_ISOLATED`): second `roboco_data` compose bridge = postgres+redis only, orchestrator multi-homed; agents can't reach prod DB. Suppresses `_append_gate_env` prod-creds injection. In BOTH build+registry composes (topology-coupled). 3. **Mobile UI** (panel): `useIsMobile` (useSyncExternalStore, hydration-safe), `ResponsiveTable` table→card, snap `TabsList`, bottom tab bar, Comms/A2A single-pane drill-down below lg, vh→dvh. REVIEW FIX: `justify-center-safe` (overflow clip), memoized matchMedia subscribe. 4. **Cloud auth** (`ROBOCO_CLOUD_AUTH_ENABLED`, migration 058 `users`): FastAPI Users, single seeded user, cookie sliding 30-day session (pwd-fingerprint JWT), `get_agent_context` dual-path (deps.py). Off = byte-identical. `proxy.ts` (Next 16). REVIEW FIX: on-mode rejects EVERY non-CEO role without a token (not just ceo — closed the PM/board :8000 spoof). diff --git a/docs/map/db-migrations.md b/docs/map/db-migrations.md index f6f0d3b8..289fdb57 100644 --- a/docs/map/db-migrations.md +++ b/docs/map/db-migrations.md @@ -105,7 +105,7 @@ The DB layer is async SQLAlchemy 2.0 over PostgreSQL+asyncpg, with pgvector for | 054 | 054_a2a_message_skill.py | `a2a_messages.skill` (String 100, nullable) — persists the capability a directed A2A message concerns; was silently dropped on send. | | 055 | 055_spawn_session_turns_tool_calls.py | `agent_spawn_sessions.turns` + `.tool_calls` (BigInteger, DEFAULT 0) — per-stint LLM iterations + tool invocations for the granular per-member performance metrics. | | 056 | 056_member_perf_daily.py | `member_performance_daily` — one row per (date, member_kind, agent_slug) scorecard rollup (incl. CEO as `member_kind='ceo'`). | -| 057 | 057_project_sandbox_services.py | `projects.sandbox_services` (ARRAY(String), nullable) — per-project opt-in for the sandboxed per-agent-spawn Postgres/Redis provisioner. | +| 057 | 057_project_sandbox_services.py | `projects.sandbox_services` (ARRAY(String), nullable) — per-project opt-in for the sandboxed per-agent-spawn engine provisioner (postgres / redis / mongo via the `SANDBOX_ENGINES` registry). | | 058 | 058_cloud_auth_users.py | `users` table (FastAPI Users schema) — the single seeded CEO login for cloud auth (`ROBOCO_CLOUD_AUTH_ENABLED`, default off). | | 059 | 059_x_credentials.py | `x_credentials` (singleton Fernet-encrypted OAuth 1.0a secrets) + `x_seen_mentions` (mentions-poll dedup ledger) — the X (Twitter) engine (`ROBOCO_X_ENGINE_ENABLED`, default off). | | 060 | 060_drop_messaging.py | Drops the channels/groups/sessions/session_tasks/messages subsystem (comms teardown — A2A is now the sole directed-message channel): `journal_entries.session_id` column, the 5 tables, and 4 enum types (`messagetype`/`sessionstatus`/`sessionscope`/`channeltype`); one-way (`downgrade()` raises `NotImplementedError`). | @@ -267,7 +267,7 @@ Migration chain 001..059 > Post-snapshot updates (since 2026-06-29): `536bbb64` (Chore/all/logical gaps sweep #286) — adds migration 053 (`playbooks.archived_by`/`archived_at`), two new columns on `PlaybookTable`; `d8a5bb48` ([chore] a2a hierarchy gate + skill persist) — adds migration 054 (`a2a_messages.skill`), one new column on `A2AMessageTable`, wired through `send_chat_message` and the A2AChatMessage model. > -> Delta 2026-07-03 (v0.17.0, 5 features): `055_spawn_session_turns_tool_calls` (`agent_spawn_sessions.turns`/`.tool_calls`) + `056_member_perf_daily` (`member_performance_daily`) predate this wave but were never appended to this doc; `057_project_sandbox_services` adds `projects.sandbox_services` (sandboxed dev DB/Redis, `ROBOCO_SANDBOX_DB_ENABLED`); `058_cloud_auth_users` adds `users` (`UserTable`, cloud auth, `ROBOCO_CLOUD_AUTH_ENABLED`); `059_x_credentials` adds `x_credentials` (`XCredentialsTable`) + `x_seen_mentions` (`XSeenMentionTable`) (X engine, `ROBOCO_X_ENGINE_ENABLED`). Chain head is now 059. +> Delta 2026-07-03 (v0.17.0, 5 features): `055_spawn_session_turns_tool_calls` (`agent_spawn_sessions.turns`/`.tool_calls`) + `056_member_perf_daily` (`member_performance_daily`) predate this wave but were never appended to this doc; `057_project_sandbox_services` adds `projects.sandbox_services` (sandboxed dev DB/Redis/Mongo, `ROBOCO_SANDBOX_DB_ENABLED`); `058_cloud_auth_users` adds `users` (`UserTable`, cloud auth, `ROBOCO_CLOUD_AUTH_ENABLED`); `059_x_credentials` adds `x_credentials` (`XCredentialsTable`) + `x_seen_mentions` (`XSeenMentionTable`) (X engine, `ROBOCO_X_ENGINE_ENABLED`). Chain head is now 059. Mongo rides existing 057 (no new migration) — it's just another entry in the `SANDBOX_ENGINES` registry. > > Delta 2026-07-04 (v0.18.0): `060_drop_messaging` (the comms-teardown migration — drops `messages`/`session_tasks`/`sessions`/`groups`/`channels` + 4 enum types + `journal_entries.session_id`; A2A is now the sole directed-message channel; one-way, `downgrade()` raises `NotImplementedError`) had already landed on master but was never appended to this doc; `061_x_feature_spotlight` adds `x_seen_features` (`XSeenFeatureTable`) + `company_goals.brand_voice` (X feature-spotlight, `ROBOCO_X_FEATURE_SPOTLIGHT_ENABLED`, sub-switch of `x_engine_enabled`). Chain head is now 061. ORM table count is now 38 (verified via `grep -c '^class .*Table' roboco/db/tables.py`), up from this doc's previously-stated 37 (that figure predates 055-061 and was never recomputed). diff --git a/docs/map/deployment-tooling.md b/docs/map/deployment-tooling.md index 5c0dd5da..aca12736 100644 --- a/docs/map/deployment-tooling.md +++ b/docs/map/deployment-tooling.md @@ -251,7 +251,7 @@ deployment-tooling - ROBOCO_DEP_UPDATE_ENABLED / _INTERVAL_SECONDS / _MAX_OPEN_TASKS / _MAX_PER_CYCLE - ROBOCO_RELEASE_MANAGER_ENABLED / _MIN_COMMITS / _INTERVAL_SECONDS / _CI_WORKFLOW - ROBOCO_ORG_MEMORY_ENABLED / _TOP_K / _MIN_SCORE -- ROBOCO_SANDBOX_DB_ENABLED — sandboxed per-agent-spawn Postgres/Redis provisioner (`roboco/runtime/sandbox.py`); a project also needs its `sandbox_services` column set +- ROBOCO_SANDBOX_DB_ENABLED — sandboxed per-agent-spawn engine provisioner (`roboco/runtime/sandbox.py` + registry in `roboco/models/sandbox.py`); a project also needs its `sandbox_services` column set - ROBOCO_DB_NETWORK_ISOLATED — set true only by the compose topology carrying the `roboco_data` data-only network; suppresses the legacy prod-creds gate-env injection - ROBOCO_CLOUD_AUTH_ENABLED / _EMAIL / _PASSWORD / _SECRET / _COOKIE_MAX_AGE — FastAPI Users cookie login for the single seeded CEO; `Settings` fails loud at startup if armed with no secret - ROBOCO_X_ENGINE_ENABLED / _MENTIONS_INTERVAL_SECONDS / _MENTIONS_MAX_PER_CYCLE / _MENTIONS_MIN_ENGAGEMENT / _MAX_OPEN_POSTS / ROBOCO_X_ACCOUNT_USER_ID / _REQUEST_TIMEOUT_SECONDS — the X (Twitter) engine; inert without stored OAuth 1.0a credentials regardless of the flag diff --git a/docs/map/models.md b/docs/map/models.md index be2518c3..4f23ac27 100644 --- a/docs/map/models.md +++ b/docs/map/models.md @@ -206,7 +206,7 @@ models/ ## Config Flags -None — pure models, no flags. (The `Project` model *carries* opt-in fields `ci_watch_enabled`, `dep_update_command`, `dep_update_paths`, `sandbox_services` (project.py:142, validated against `VALID_SANDBOX_SERVICES` — sandboxed dev DB/Redis, gated by `ROBOCO_SANDBOX_DB_ENABLED` elsewhere) that other layers gate on, and `llm_catalog` carries the "pure Ollama" defaults, but the models package itself reads no env / toggles nothing.) +None — pure models, no flags. (The `Project` model *carries* opt-in fields `ci_watch_enabled`, `dep_update_command`, `dep_update_paths`, `sandbox_services` (project.py:142, validated against `VALID_SANDBOX_SERVICES` — sandboxed dev DB/Redis/Mongo, gated by `ROBOCO_SANDBOX_DB_ENABLED` elsewhere) that other layers gate on, and `llm_catalog` carries the "pure Ollama" defaults, but the models package itself reads no env / toggles nothing.) `VALID_SANDBOX_SERVICES` is now derived from the `SANDBOX_ENGINES` registry in `roboco/models/sandbox.py` (was a hardcoded `{"postgres","redis"}` set) — mongo is just another registry entry, no new migration (rides existing 057) and no new feature flag. ## Gotchas diff --git a/docs/map/orchestrator.md b/docs/map/orchestrator.md index 7f38bbad..912e597e 100644 --- a/docs/map/orchestrator.md +++ b/docs/map/orchestrator.md @@ -1,5 +1,5 @@ ## Purpose -The AgentOrchestrator is the runtime brain of RoboCo: it owns the per-agent Docker container lifecycle, the per-tick dispatcher that matches tasks to agents, the stale-claim reaper, the provider rate-limit/overload park-and-probe recovery loop, and the default-off background engines (self-heal, CI-watch, dep-update, release-manager, strategy, external-PR poll, X-engine mentions poll, board roadmap engine, video render loop). It claims tasks on behalf of agents before spawning, injects briefings/manifests/git context at spawn time, provisions a per-spawn sandbox DB/Redis when opted in, captures per-session token usage, and persists durable runtime state (WaitingRecord, respawn_tracker) across restarts. +The AgentOrchestrator is the runtime brain of RoboCo: it owns the per-agent Docker container lifecycle, the per-tick dispatcher that matches tasks to agents, the stale-claim reaper, the provider rate-limit/overload park-and-probe recovery loop, and the default-off background engines (self-heal, CI-watch, dep-update, release-manager, strategy, external-PR poll, X-engine mentions poll, board roadmap engine, video render loop). It claims tasks on behalf of agents before spawning, injects briefings/manifests/git context at spawn time, provisions a per-spawn sandbox (postgres/redis/mongo via the engine registry) when opted in, captures per-session token usage, and persists durable runtime state (WaitingRecord, respawn_tracker) across restarts. ## Files @@ -80,8 +80,8 @@ The AgentOrchestrator is the runtime brain of RoboCo: it owns the per-agent Dock | AgentOrchestrator._should_skip_live_reap | method | roboco/runtime/orchestrator.py:8750 | Spare a live container from reaping UNLESS wedged (grok) or gateway-broken past grace (those kill+evict). | | AgentOrchestrator._assignee_is_provider_parked | method | roboco/runtime/orchestrator.py:8527 | True if a task's assignee is provider-parked; reaper skips it so the claim survives for probe-resume. | | AgentOrchestrator._reap_with_service | method | roboco/runtime/orchestrator.py:8770 | Inner stale-claim reaper: skip live (unless wedged/broken), skip provider-parked, unclaim_for_reaper the rest. | -| AgentOrchestrator._maybe_provision_sandbox | method | roboco/runtime/orchestrator.py:2029 | Provision this spawn's sandbox DB/Redis via `SandboxProvisioner` when `sandbox_db_enabled` + the project's `sandbox_services` are set; None (byte-for-byte legacy path) otherwise. Fail-loud once opted in — a provisioning failure refuses the spawn. | -| AgentOrchestrator._append_sandbox_env | staticmethod | roboco/runtime/orchestrator.py:2639 | Inject `ROBOCO_TEST_DB_*`/`ROBOCO_TEST_REDIS_*` env from `config.sandbox_info`, called INSTEAD OF `_append_gate_env` whenever a sandbox was provisioned for this spawn. | +| AgentOrchestrator._maybe_provision_sandbox | method | roboco/runtime/orchestrator.py:2029 | Provision this spawn's sandbox engines (postgres/redis/mongo via `SandboxProvisioner` + the `SANDBOX_ENGINES` registry) when `sandbox_db_enabled` + the project's `sandbox_services` are set; None (byte-for-byte legacy path) otherwise. Fail-loud once opted in — a provisioning failure refuses the spawn. | +| AgentOrchestrator._append_sandbox_env | staticmethod | roboco/runtime/orchestrator.py:2639 | Inject the sandbox's `ROBOCO_TEST_*` env (postgres `ROBOCO_TEST_DB_*` / redis `ROBOCO_TEST_REDIS_*` / mongo `ROBOCO_TEST_MONGO_*`) via `SandboxInfo.emit_env` over the engine registry, called INSTEAD OF `_append_gate_env` whenever a sandbox was provisioned for this spawn. Emission is registry-driven, so a new engine's env vars land here with no orchestrator change. | | AgentOrchestrator._sandbox_janitor_sweep | method | roboco/runtime/orchestrator.py:9426 | Best-effort: remove sandbox containers whose owner agent is gone; rides the reaper tick, error-isolated. | | AgentOrchestrator._x_mentions_poll_loop | method | roboco/runtime/orchestrator.py:7431 | Default-off X-engine mentions-poll tick loop (`x_engine_enabled`); release-post drafts are event-driven, not from this loop. | | AgentOrchestrator._run_x_mentions_cycle | method | roboco/runtime/orchestrator.py:7453 | One mentions-poll pass: `get_x_engine(db).run_cycle()` + commit; testable without the sleep. | @@ -197,7 +197,7 @@ stateDiagram-v2 - `ROBOCO_GROK_MAX_COST_USD` — grok budget kill-switch; `_GROK_RATE_LIMIT_EXIT_CODE=75`, `_GROK_AUTH_EXIT_CODE=78`, `_PROBE_GIVE_UP_THRESHOLD=30`. - `ROBOCO_CLAUDE_STUCK_KILL_SECONDS` (default 3600, min 600) — heartbeat-stale kill threshold for non-GROK agents; controls `_maybe_kill_stuck_claude`. - `ROBOCO_DISPATCHER_INTERVAL_SECONDS` (30), `ROBOCO_INTERACTIVE_IDLE_REAP_SECONDS`, `ROBOCO_GROK_*` backoff constants. -- `ROBOCO_SANDBOX_DB_ENABLED` (default off) — master switch for the sandboxed per-agent-spawn Postgres/Redis provisioner (`_maybe_provision_sandbox`/`_append_sandbox_env`/`_sandbox_janitor_sweep`); a project participates only when its `sandbox_services` column is also set. +- `ROBOCO_SANDBOX_DB_ENABLED` (default off) — master switch for the sandboxed per-agent-spawn engine provisioner (`_maybe_provision_sandbox`/`_append_sandbox_env`/`_sandbox_janitor_sweep`); a project participates only when its `sandbox_services` column is also set. The service set is the `SANDBOX_ENGINES` registry (postgres / redis / mongo); adding an engine needs no orchestrator or env-emitter change. - `ROBOCO_DB_NETWORK_ISOLATED` (default off; set by the compose topology that carries the `roboco_data` network) — suppresses the legacy `_append_gate_env` prod-creds injection when postgres/redis are unreachable from the agent mesh. - `ROBOCO_CLOUD_AUTH_ENABLED` (+ `_EMAIL`/`_PASSWORD`/`_SECRET`/`_COOKIE_MAX_AGE`, default off) — cloud auth master switch; read by `roboco.api.deps.get_agent_context`/`roboco.api.auth.*`, not the orchestrator itself, but gates whether a spawned agent's own HMAC-token identity path is the sole non-CEO auth route. - `ROBOCO_X_ENGINE_ENABLED` (+ `_mentions_interval_seconds` / `_mentions_max_per_cycle` / `_mentions_min_engagement` / `_max_open_posts` / `_account_user_id` / `_request_timeout_seconds`, default off) — gates `_x_mentions_poll_loop`. diff --git a/docs/map/panel.md b/docs/map/panel.md index 5aba6447..51bf3c4d 100644 --- a/docs/map/panel.md +++ b/docs/map/panel.md @@ -201,7 +201,7 @@ panel/ (Next.js 16, package roboco-panel v0.14.0) - `dep_update_enabled` — dependency-update bot - `release_manager_enabled` — gated release manager - `org_memory_enabled` — organizational memory loop -- `sandbox_db_enabled` — sandboxed per-agent test DB/Redis +- `sandbox_db_enabled` — sandboxed per-agent test DB/Redis/Mongo (engine registry) - `x_engine_enabled` — X (Twitter) engine (release-post + mention-reply drafts, all CEO-held) - `roadmap_engine_enabled` — board roadmap engine (weekly Product-Owner-authored cycle) - `routing_strict` — fail-closed model routing (refuse to silently downgrade to the legacy Anthropic path on a disabled provider) diff --git a/docs/map/runtime-providers.md b/docs/map/runtime-providers.md index 5ccc8db2..9791b7ee 100644 --- a/docs/map/runtime-providers.md +++ b/docs/map/runtime-providers.md @@ -9,7 +9,8 @@ This slice is the agent-runtime + LLM-provider seam plus the in-container agent | roboco/runtime/spawn_manifest.py | Builds the per-role /app/tool-manifest.json (allowed verbs/tools, env) from role_config | 85 | | roboco/runtime/streaming.py | Global reasoning-stream callback holder+setter for live UI streaming | 53 | | roboco/runtime/transcript_retention.py | Pure selector of agent-owned old Claude transcripts to prune (never operator dirs) | 74 | -| roboco/runtime/sandbox.py | `SandboxProvisioner` — throwaway per-agent-spawn Postgres/Redis sibling containers (orchestrator-side, never docker-in-agent); provision/teardown/janitor_sweep, standalone + unit-testable via an injected `DockerRunner` | 344 | +| roboco/runtime/sandbox.py | `SandboxProvisioner` — throwaway per-agent-spawn engine sibling containers (postgres/redis/mongo via the `SANDBOX_ENGINES` registry in `roboco/models/sandbox.py`, orchestrator-side, never docker-in-agent); generic `_provision_engine` per engine, provision/teardown/janitor_sweep, standalone + unit-testable via an injected `DockerRunner` | 344 | +| roboco/models/sandbox.py | Pure engine registry — `SandboxEngine` ABC + `_PostgresEngine` (`postgres:16-alpine`) / `_RedisEngine` (`redis:8-alpine`) / `_MongoEngine` (`mongo:8-alpine`, mongosh readiness probe, auth db `admin`); `SandboxConnection` / `SandboxInfo` (with `emit_env`); `SANDBOX_ENGINES` + `VALID_SANDBOX_SERVICES` (derived). Adding an engine = one class + one registry line — no provisioner or env-emitter branch. Lives in the models layer so `roboco/models/project.py` can derive the allowlist without importing the runtime layer. | 232 | | roboco/llm/__init__.py | Re-exports ToonAdapter/ToonMetrics singletons | 17 | | roboco/llm/metrics.py | Singleton holder for TOON token-savings metrics | 21 | | roboco/llm/toon_adapter.py | TOON serialization adapter for token-efficient LLM communication (JSON fallback) | 188 | @@ -47,7 +48,7 @@ This slice is the agent-runtime + LLM-provider seam plus the in-container agent | stream_reasoning | function | roboco/runtime/streaming.py:39 | Stream a reasoning chunk to the registered callback if any | | is_agent_owned_dir | function | roboco/runtime/transcript_retention.py:23 | True if a ~/.claude/projects subdir was written by a spawned agent (-app or encoded workspaces root prefix, boundary-aware) | | select_prunable_transcripts | function | roboco/runtime/transcript_retention.py:59 | Pure selector of agent-owned *.jsonl transcripts older than cutoff_epoch (never operator dirs) | -| SandboxProvisioner | class | roboco/runtime/sandbox.py:90 | Per-agent-spawn throwaway Postgres/Redis provisioner; `provision`/`teardown`/`janitor_sweep`, docker plumbing is an injected `DockerRunner` callable | +| SandboxProvisioner | class | roboco/runtime/sandbox.py:90 | Per-agent-spawn throwaway engine provisioner (iterates `SANDBOX_ENGINES`); `provision`/`teardown`/`janitor_sweep`, docker plumbing is an injected `DockerRunner` callable. Engine specs live in `roboco/models/sandbox.py` (`SandboxEngine` ABC + `_PostgresEngine`/`_RedisEngine`/`_MongoEngine`) | | ToonAdapter | class | roboco/llm/toon_adapter.py:33 | TOON serialization adapter: encode/decode with JSON fallback, prompt formatting, token-savings estimate | | get_toon_adapter | function | roboco/llm/toon_adapter.py:184 | Singleton ToonAdapter accessor | | SpawnResult | dataclass | roboco/llm/providers/base.py:26 | Provider spawn result: instance_id, initial agent_state, extra metadata | diff --git a/docs/map/support-services.md b/docs/map/support-services.md index 3097ed3a..934c5751 100644 --- a/docs/map/support-services.md +++ b/docs/map/support-services.md @@ -266,7 +266,7 @@ Panel-tunable flags defined in `services/settings.py:46` `FEATURE_FLAGS` (stored | `dep_update_enabled` | Dependency-update bot | `ROBOCO_DEP_UPDATE_ENABLED` | | `release_manager_enabled` | Gated release manager | `ROBOCO_RELEASE_MANAGER_ENABLED` | | `org_memory_enabled` | Organizational memory loop | `ROBOCO_ORG_MEMORY_ENABLED` | -| `sandbox_db_enabled` | Sandboxed per-agent test DB/Redis | `ROBOCO_SANDBOX_DB_ENABLED` | +| `sandbox_db_enabled` | Sandboxed per-agent test DB/Redis/Mongo (engine registry) | `ROBOCO_SANDBOX_DB_ENABLED` | | `x_engine_enabled` | X (Twitter) engine | `ROBOCO_X_ENGINE_ENABLED` | | `roadmap_engine_enabled` | Board roadmap engine | `ROBOCO_ROADMAP_ENGINE_ENABLED` | diff --git a/docs/rag/architecture/config-reference.md b/docs/rag/architecture/config-reference.md index f6692c06..22dc9176 100644 --- a/docs/rag/architecture/config-reference.md +++ b/docs/rag/architecture/config-reference.md @@ -86,7 +86,7 @@ Env-gated subsystems. Most are default-off; `ROBOCO_OVERLOAD_BREAK_ENABLED`, `RO | `ROBOCO_PR_GATE_AUTO_SUBMIT_ENABLED` | `true` | PR-gate turn cut: when every child of an assembled parent is terminal, run the real `submit_up` / `submit_root` system-side as the owning PM (`_try_auto_submit`) instead of spawning the PM for that turn — the submit's substance (freshness rebase, integrity check, PR open) is deterministic gate code. A gate rejection falls back to the classic PM closure spawn; the PM keeps the judgment turns (merge, revision). Each auto-submit leaves a `task.auto_submitted` audit row. Off = every closure spawns the PM to submit. | | `ROBOCO_SPAWN_PREFLIGHT_ENABLED` | `false` | Refuse to spawn a non-human delivery role absent from `GATEWAY_ENABLED_ROLES` (no manifest → can never claim → would respawn on the same task forever); refuse + alert the overseer once instead. Inert in practice (all delivery roles are gateway-enabled). Armed on the NAS composes. | | `ROBOCO_NOTIFICATION_SPAWN_COOLDOWN_SECONDS` | `600` | Cross-tick damper for notification-triggered spawns (escalation/approval/audit/a2a — task-less, so the readiness gate and respawn breaker never see them): one spawn per (agent, notification) per window; the notification stays pending so the next window retries. `0` = legacy every-tick respawn. | -| `ROBOCO_SANDBOX_DB_ENABLED` | `false` | Sandboxed per-agent-spawn test DB/Redis: throwaway `postgres:16-alpine` / `redis:8-alpine` sibling containers, per-project opt-in. See "Sandboxed Dev DB/Redis" below and `docs/rag/architecture/sandbox-db.md`. | +| `ROBOCO_SANDBOX_DB_ENABLED` | `false` | Sandboxed per-agent-spawn test DB/Redis/Mongo: throwaway sibling containers provisioned from the engine registry in `roboco/models/sandbox.py` (postgres:16-alpine / redis:8-alpine / mongo:8-alpine), per-project opt-in. The valid-service set is `VALID_SANDBOX_SERVICES` (registry-derived). See "Sandboxed Dev DB/Redis/Mongo" below and `docs/rag/architecture/sandbox-db.md`. | | `ROBOCO_X_ENGINE_ENABLED` | `false` | The X (Twitter) engine: draft release/mention posts, ALL held for per-post CEO approval. See "X (Twitter) Engine" below and `docs/rag/architecture/x-engine.md`. | | `ROBOCO_ROADMAP_ENGINE_ENABLED` | `false` | The board roadmap engine: weekly Product-Owner-authored cycle, CEO approves each item individually into BACKLOG. See "Board Roadmap Engine" below. | @@ -144,13 +144,13 @@ The fan-out generalizations of self-heal — they watch any opted-in project, no | `ROBOCO_IMAGE_PRUNE_ENABLED` | `true` | Background sweep prunes dangling (``) Docker images from agent-image rebuilds (only dangling; ~6h throttle). Always-on safety net, not a feature flag | | `ROBOCO_IMAGE_PRUNE_INTERVAL_SECONDS` | `21600` | Minimum seconds between dangling-image prune passes | -## Sandboxed Dev DB/Redis +## Sandboxed Dev DB/Redis/Mongo -Per-agent-spawn throwaway Postgres/Redis, replacing (never coexisting with) the legacy prod-creds gate-env injection for an opted-in project. Default-off; see `docs/rag/architecture/sandbox-db.md`. +Per-agent-spawn throwaway Postgres/Redis/Mongo, replacing (never coexisting with) the legacy prod-creds gate-env injection for an opted-in project. Default-off; see `docs/rag/architecture/sandbox-db.md`. | Variable | Default | Description | |----------|---------|-------------| -| `ROBOCO_SANDBOX_DB_ENABLED` | `false` | Master switch. Off = spawning behaves exactly as today (the legacy `_append_gate_env` prod-creds injection, itself gated by `ROBOCO_TOOLCHAIN_MATCH_ENABLED`). Only projects with their `sandbox_services` column set (migration `057`) participate even when on. | +| `ROBOCO_SANDBOX_DB_ENABLED` | `false` | Master switch. Off = spawning behaves exactly as today (the legacy `_append_gate_env` prod-creds injection, itself gated by `ROBOCO_TOOLCHAIN_MATCH_ENABLED`). Only projects with their `sandbox_services` column set (migration `057`) participate even when on. The valid service set is `VALID_SANDBOX_SERVICES` in `roboco/models/sandbox.py` (registry-derived: postgres / redis / mongo); adding an engine is one class + one registry line, no orchestrator edit. Env injected per engine: `ROBOCO_TEST_DB_*`, `ROBOCO_TEST_REDIS_*`, `ROBOCO_TEST_MONGO_*` (incl. `ROBOCO_TEST_MONGO_AUTH_DB=admin`). | ## X (Twitter) Engine diff --git a/docs/rag/architecture/db-network-isolation.md b/docs/rag/architecture/db-network-isolation.md index f8649bc0..ccf538b7 100644 --- a/docs/rag/architecture/db-network-isolation.md +++ b/docs/rag/architecture/db-network-isolation.md @@ -2,7 +2,7 @@ ## What It Is -A compose-topology hardening: two user-defined Docker bridges instead of one. `roboco_default` carries the agent mesh (panel, nginx, ollama, every spawned agent container, and their sandbox DB/Redis sidecars — see `docs/rag/architecture/sandbox-db.md`). `roboco_data` carries **only** postgres + redis. The orchestrator is the sole multi-homed service (both networks) — every agent container structurally cannot resolve or TCP-reach `roboco-postgres:5432` / `roboco-redis:6379` at all. This matters because redis has no auth in this deployment: network membership *is* the containment, not a password. +A compose-topology hardening: two user-defined Docker bridges instead of one. `roboco_default` carries the agent mesh (panel, nginx, ollama, every spawned agent container, and their sandbox sidecars — postgres / redis / mongo via the engine registry in `roboco/models/sandbox.py`; see `docs/rag/architecture/sandbox-db.md`). `roboco_data` carries **only** postgres + redis. The orchestrator is the sole multi-homed service (both networks) — every agent container structurally cannot resolve or TCP-reach `roboco-postgres:5432` / `roboco-redis:6379` at all. This matters because redis has no auth in this deployment: network membership *is* the containment, not a password. ## Enable/Disable @@ -12,7 +12,7 @@ A compose-topology hardening: two user-defined Docker bridges instead of one. `r ## What flipping it changes -`ROBOCO_DB_NETWORK_ISOLATED=true` suppresses the legacy `_append_gate_env` prod-creds injection (`roboco/runtime/orchestrator.py`) — the one that would otherwise hand an agent `ROBOCO_TEST_DB_HOST=roboco-postgres` credentials for a host it cannot reach. A connect timeout is worse than no credentials at all (the test suite's DB-reachability check skips cleanly on a fast refusal, but hangs on a dead-end timeout), so the flag makes that injection a no-op rather than let it happen and fail slow. Projects that need a real DB for their gate opt into the sandboxed dev DB/Redis instead (`docs/rag/architecture/sandbox-db.md`) — sandbox replaces, never coexists with, the prod-creds path. +`ROBOCO_DB_NETWORK_ISOLATED=true` suppresses the legacy `_append_gate_env` prod-creds injection (`roboco/runtime/orchestrator.py`) — the one that would otherwise hand an agent `ROBOCO_TEST_DB_HOST=roboco-postgres` credentials for a host it cannot reach. A connect timeout is worse than no credentials at all (the test suite's DB-reachability check skips cleanly on a fast refusal, but hangs on a dead-end timeout), so the flag makes that injection a no-op rather than let it happen and fail slow. Projects that need a real DB for their gate opt into the sandboxed dev DB/Redis/Mongo instead (`docs/rag/architecture/sandbox-db.md`) — sandbox replaces, never coexists with, the prod-creds path. ## What is unaffected diff --git a/docs/rag/architecture/sandbox-db.md b/docs/rag/architecture/sandbox-db.md index 30026330..9d3232db 100644 --- a/docs/rag/architecture/sandbox-db.md +++ b/docs/rag/architecture/sandbox-db.md @@ -1,35 +1,50 @@ -# Sandboxed Dev DB/Redis +# Sandboxed Dev DB/Redis/Mongo ## What It Is -A per-agent-spawn throwaway Postgres/Redis pair, provisioned as **sibling containers** to the agent container (never docker-in-agent — the docker socket/CLI stay structurally absent from agent images). Implemented in `roboco/runtime/sandbox.py` (`SandboxProvisioner`), wired into the orchestrator's spawn path. +A per-agent-spawn throwaway Postgres/Redis/Mongo set, provisioned as **sibling containers** to the agent container (never docker-in-agent — the docker socket/CLI stay structurally absent from agent images). Implemented in `roboco/runtime/sandbox.py` (`SandboxProvisioner`), wired into the orchestrator's spawn path. It replaces — never coexists with — the legacy `_append_gate_env` behavior that hands an agent RoboCo's own production Postgres credentials so its `make quality` gate can run the DB-backed test suite instead of a hollow unit-only subset. +## The engine registry + +The service set is a **pluggable engine registry**, not a hardcoded postgres+redis pair. `roboco/models/sandbox.py` defines a `SandboxEngine` ABC (image, container port, readiness probe, tmpfs paths, env emission) and the concrete engines: + +- `_PostgresEngine` — `postgres:16-alpine`, tmpfs `/var/lib/postgresql/data`, `pg_isready` probe (60s), env `ROBOCO_TEST_DB_*` (incl. `ROBOCO_TEST_DB_ADMIN_DB`). +- `_RedisEngine` — `redis:8-alpine`, no tmpfs, `redis-cli -a … ping` probe (15s), env `ROBOCO_TEST_REDIS_*`. +- `_MongoEngine` — `mongo:8-alpine`, tmpfs `/data/db`, `mongosh` ping against auth db `admin` (60s), env `ROBOCO_TEST_MONGO_*` (incl. `ROBOCO_TEST_MONGO_AUTH_DB=admin`). + +`SANDBOX_ENGINES: dict[str, SandboxEngine]` registers them by name; `VALID_SANDBOX_SERVICES = frozenset(SANDBOX_ENGINES)` is the single source of truth the provisioner, the orchestrator's env injection, and `projects.sandbox_services` validation all consult. **Adding an engine is one class + one registry line** — no branch edited in the provisioner or the env emitter, which both iterate the registry. + ## Enable/Disable | Variable | Default | Effect | |----------|---------|--------| | `ROBOCO_SANDBOX_DB_ENABLED` | `false` | Master switch. Off = spawning behaves exactly as today (the legacy prod-creds gate-env injection, itself gated by `ROBOCO_TOOLCHAIN_MATCH_ENABLED`). Panel-toggleable (Settings → Feature Flags). | -A second, per-project gate applies even when the flag is on: only a project with its `sandbox_services` column set (e.g. `["postgres", "redis"]`; migration `057`, nullable/additive) participates. Every other project's spawns are byte-for-byte unaffected. +A second, per-project gate applies even when the flag is on: only a project with its `sandbox_services` column set (e.g. `["postgres", "redis", "mongo"]`; migration `057`, nullable/additive) participates. Every other project's spawns are byte-for-byte unaffected. Mongo rides the same column — no new migration, no new feature flag; it is just another registry entry. ## Provisioning -For an opted-in project's spawn, the orchestrator provisions before `docker run`: +For an opted-in project's spawn, the orchestrator provisions each requested service through a single generic `_provision_engine` (no per-engine branch): it generates a random 32-hex-char password (`secrets.token_hex(16)`), pre-pulls the image, `docker run`s the sibling container, and polls the engine's readiness probe up to its deadline. -- **Postgres**: `postgres:16-alpine`, named `roboco-sandbox-pg-{agent_id}`, `--tmpfs /var/lib/postgresql/data` (no disk persistence), `--memory 512m --cpus 1`, a random 32-hex-char password (`secrets.token_hex(16)`), user/db both `sandbox`. Readiness polled via `pg_isready` up to 60s. -- **Redis**: `redis:8-alpine`, named `roboco-sandbox-redis-{agent_id}`, same memory/cpu caps, `--requirepass` with its own random password. Readiness polled via `redis-cli ping` up to 15s. +- **Postgres**: named `roboco-sandbox-pg-{agent_id}`, `--tmpfs /var/lib/postgresql/data` (no disk persistence), `--memory 512m --cpus 1`, user/db both `sandbox`, readiness via `pg_isready` up to 60s. +- **Redis**: named `roboco-sandbox-redis-{agent_id}`, same memory/cpu caps, `redis-server --requirepass`, readiness via `redis-cli -a … ping` up to 15s. +- **Mongo**: named `roboco-sandbox-mongo-{agent_id}`, `--tmpfs /data/db`, same memory/cpu caps, root user/db `sandbox`/`sandbox`, readiness via `mongosh` ping (auth db `admin`) up to 60s. -Both are labeled `roboco.sandbox=1` plus an owner label (`roboco.sandbox.owner=roboco-agent-{agent_id}`) so the janitor can find them. A provisioning failure is **fail-loud**: the spawn is refused (`AgentReadinessError`) rather than starting an agent whose gate can't run against a broken DB. A stale same-named sandbox left by a crash-missed teardown is pre-cleared before provisioning, so a leftover container can't collide with a fresh `docker run`. +All are labeled `roboco.sandbox=1` plus an owner label (`roboco.sandbox.owner=roboco-agent-{agent_id}`) so the janitor can find them. A provisioning failure is **fail-loud**: the spawn is refused (`AgentReadinessError`) rather than starting an agent whose gate can't run against a broken DB, and any already-provisioned sibling is torn down before re-raising. A stale same-named sandbox left by a crash-missed teardown is pre-cleared before provisioning, so a leftover container can't collide with a fresh `docker run`. + +### Image pre-pull + +`_ensure_image` `docker image inspect`s the engine's image and, on absence, `docker pull`s it (300s timeout) **before** `docker run`. Without this a NAS cold pull would hit the 20s run timeout, get killed, and re-pull forever on every respawn. The inspect-then-pull runs per service per spawn, so an already-present image short-circuits in milliseconds. ## Injected environment -Instead of the legacy `ROBOCO_TEST_DB_*` pointing at RoboCo's own production Postgres, the sandbox's own host/port/user/password are injected under the **same** `ROBOCO_TEST_DB_*` names (so an existing project's conftest needs no change) plus new `ROBOCO_TEST_REDIS_*` names. `_append_sandbox_env` runs **instead of** `_append_gate_env` whenever a sandbox was provisioned for that spawn. +Instead of the legacy `ROBOCO_TEST_DB_*` pointing at RoboCo's own production Postgres, the sandbox's own host/port/user/password are injected. Env var names are preserved per engine so an existing project's conftest needs no change: `ROBOCO_TEST_DB_*` (postgres, incl. `ROBOCO_TEST_DB_ADMIN_DB`), `ROBOCO_TEST_REDIS_*` (redis), and `ROBOCO_TEST_MONGO_*` (mongo, incl. `ROBOCO_TEST_MONGO_AUTH_DB=admin`). The orchestrator's `_append_sandbox_env` is a single `cmd.extend(info.emit_env())` over the registry — a new engine's env lands with no orchestrator edit. It runs **instead of** `_append_gate_env` whenever a sandbox was provisioned for that spawn. ## Lifetime and teardown -A sandbox's lifetime tracks its owning agent container 1:1: torn down (`stop` → `kill` fallback → `rm -f`, all best-effort and idempotent) at every container-removal path. An **orphan janitor** also runs at orchestrator startup and on each reaper tick: it lists every `roboco.sandbox=1` container, cross-references live agent containers, and removes any sandbox whose owner is gone. +A sandbox's lifetime tracks its owning agent container 1:1: torn down (`stop` → `kill` fallback → `rm -f`, all best-effort and idempotent) at every container-removal path. Teardown iterates **all** registered engines (`SANDBOX_ENGINES.values()`) — a mongo sandbox is reaped by the same path that reaps postgres/redis, with no per-engine teardown branch. An **orphan janitor** also runs at orchestrator startup and on each reaper tick: it lists every `roboco.sandbox=1` container, cross-references live agent containers, and removes any sandbox whose owner is gone. The janitor has a **grace window** (`_JANITOR_GRACE_SECONDS`, 180s): a sandbox is provisioned *before* its agent container exists, so a sweep racing a mid-flight spawn would otherwise see "owner not live yet" and reap a fresh sandbox out from under a spawn still starting up. Owners provisioned within the grace window are skipped by that pass. The pre-spawn stale-clear (above) likewise never touches a just-provisioned sandbox. diff --git a/panel/src/components/projects/edit-project-dialog.tsx b/panel/src/components/projects/edit-project-dialog.tsx index d7300a2f..424f179f 100644 --- a/panel/src/components/projects/edit-project-dialog.tsx +++ b/panel/src/components/projects/edit-project-dialog.tsx @@ -34,6 +34,12 @@ const cells: { value: Team; label: string }[] = [ { value: Team.UX_UI, label: "UX/UI" }, ]; +const SANDBOX_SERVICES = [ + { id: "postgres", label: "PostgreSQL" }, + { id: "redis", label: "Redis" }, + { id: "mongo", label: "MongoDB" }, +] as const; + interface EditProjectDialogProps { projectId: string; open: boolean; @@ -86,11 +92,8 @@ function EditProjectForm({ (project.dep_update_paths || []).join(", "), ); const sandboxServices = project.sandbox_services || []; - const [sandboxPostgres, setSandboxPostgres] = useState( - sandboxServices.includes("postgres"), - ); - const [sandboxRedis, setSandboxRedis] = useState( - sandboxServices.includes("redis"), + const [sandboxSet, setSandboxSet] = useState>( + new Set(sandboxServices), ); // Token handling @@ -131,10 +134,7 @@ function EditProjectForm({ .map((p) => p.trim()) .filter(Boolean) : undefined, - sandbox_services: [ - ...(sandboxPostgres ? ["postgres"] : []), - ...(sandboxRedis ? ["redis"] : []), - ], + sandbox_services: [...sandboxSet], }; // Handle token update @@ -459,31 +459,30 @@ function EditProjectForm({
-
- - -
-
- - -
+ {SANDBOX_SERVICES.map((svc) => ( +
+ + + setSandboxSet((prev) => { + const next = new Set(prev); + if (checked) next.add(svc.id); + else next.delete(svc.id); + return next; + }) + } + /> +
+ ))}

- Provision a throwaway sandbox DB/Redis per agent spawn for + Provision a throwaway sandbox DB/Redis/Mongo per agent spawn for this project instead of the production credentials.

diff --git a/roboco/models/project.py b/roboco/models/project.py index 72a7b639..ce320fe2 100644 --- a/roboco/models/project.py +++ b/roboco/models/project.py @@ -13,6 +13,7 @@ from uuid import UUID, uuid4 from pydantic import Field, field_validator from roboco.models.base import RobocoBase, Team, TimestampMixin +from roboco.models.sandbox import SANDBOX_ENGINES, VALID_SANDBOX_SERVICES class BranchReason(StrEnum): @@ -26,9 +27,9 @@ class BranchReason(StrEnum): # Sandboxed per-agent-spawn DB/Redis opt-in — the services a project may -# request. Single source of truth for both the pydantic validators below and -# the orchestrator-side provisioner (roboco/runtime/sandbox.py). -VALID_SANDBOX_SERVICES: frozenset[str] = frozenset({"postgres", "redis"}) +# request. The valid set is derived from the engine registry +# (roboco/models/sandbox.py), the single source of truth shared with the +# orchestrator-side provisioner (roboco/runtime/sandbox.py). def _normalize_sandbox_services(value: list[str] | None) -> list[str] | None: @@ -41,7 +42,7 @@ def _normalize_sandbox_services(value: list[str] | None) -> list[str] | None: f"unknown sandbox service(s) {unknown}; valid: " f"{sorted(VALID_SANDBOX_SERVICES)}" ) - return [s for s in ("postgres", "redis") if s in value] + return [s for s in SANDBOX_ENGINES if s in value] class Project(TimestampMixin): diff --git a/roboco/models/runtime.py b/roboco/models/runtime.py index f08c94fe..d5446fe7 100644 --- a/roboco/models/runtime.py +++ b/roboco/models/runtime.py @@ -11,6 +11,8 @@ from pathlib import Path from typing import Any from uuid import UUID, uuid4 +from roboco.models.sandbox import SandboxInfo + class OrchestratorAgentState(StrEnum): """Agent lifecycle states in the orchestrator.""" @@ -38,34 +40,6 @@ class SpawnGitContext: task_short_id: str | None = None -@dataclass(frozen=True) -class PostgresSandbox: - """Connection info for a per-spawn throwaway Postgres sandbox container.""" - - host: str - port: int - user: str - password: str - database: str - - -@dataclass(frozen=True) -class RedisSandbox: - """Connection info for a per-spawn throwaway Redis sandbox container.""" - - host: str - port: int - password: str - - -@dataclass(frozen=True) -class SandboxInfo: - """Sandbox container(s) provisioned for one agent spawn (services opted-in).""" - - postgres: PostgresSandbox | None = None - redis: RedisSandbox | None = None - - @dataclass class OrchestratorAgentConfig: """Configuration for an agent in the orchestrator.""" diff --git a/roboco/models/sandbox.py b/roboco/models/sandbox.py new file mode 100644 index 00000000..705f86d8 --- /dev/null +++ b/roboco/models/sandbox.py @@ -0,0 +1,232 @@ +"""Per-engine sandbox specs — image, run args, readiness probe, env emission. + +Pure (no docker): the provisioner in ``roboco/runtime/sandbox.py`` consumes the +registry and runs the containers. Lives in the models layer so +``roboco/models/project.py`` can derive the valid-service allowlist from it +without importing the runtime layer (no cycle). Adding an engine = one class + +one registry line — no branch to edit in the provisioner or the env emitter. +""" + +from __future__ import annotations + +from abc import ABC, abstractmethod +from dataclasses import dataclass + + +@dataclass(frozen=True) +class SandboxConnection: + """Connection info for one provisioned sandbox service. + + ``user`` / ``database`` are ``None`` for engines that don't expose them + (redis). For postgres ``database`` is the admin db; for mongo it is the + auth db (``admin``). + """ + + host: str + port: int + password: str + user: str | None = None + database: str | None = None + + +@dataclass(frozen=True) +class SandboxInfo: + """Sandbox container(s) provisioned for one agent spawn (services opted-in).""" + + services: dict[str, SandboxConnection] + + def emit_env(self) -> list[str]: + """Flattened ``-e KEY=VAL`` args for the agent container's ``docker run``. + + Iterates the engines so the orchestrator stays registry-agnostic. + """ + env: list[str] = [] + for name, conn in self.services.items(): + env.extend(SANDBOX_ENGINES[name].emit_env(conn)) + return env + + +class SandboxEngine(ABC): + """One sandbox service kind: how to run it, probe it, and feed its creds.""" + + name: str + image: str + container_port: int + ready_deadline: float # mutable — tests monkeypatch the instance attr + tmpfs: tuple[str, ...] + container_slug: str + + def container_name(self, agent_id: str) -> str: + return f"roboco-sandbox-{self.container_slug}-{agent_id}" + + @abstractmethod + def run_env(self, password: str) -> list[str]: + """``-e KEY=VAL`` pairs baked into the sandbox container's ``docker run``.""" + + @abstractmethod + def run_command(self, password: str) -> list[str]: + """Args after the image (e.g. ``redis-server --requirepass pw``). + + Empty for engines that need no command tail. Some engines ignore + ``password`` here; the param stays on the ABC so a password-bearing + engine needn't special-case its call. + """ + + @abstractmethod + def ready_probe(self, password: str) -> list[str]: + """``docker exec`` probe cmd; rc 0 means ready. + + Some engines ignore ``password`` (probe without auth); see ``run_command``. + """ + + @abstractmethod + def connection(self, host: str, password: str) -> SandboxConnection: + """Connection info for the agent, given the container host + password.""" + + @abstractmethod + def emit_env(self, conn: SandboxConnection) -> list[str]: + """``-e KEY=VAL`` args injecting this service's creds into the agent.""" + + +class _PostgresEngine(SandboxEngine): + name = "postgres" + image = "postgres:16-alpine" + container_port = 5432 + ready_deadline = 60.0 + tmpfs = ("/var/lib/postgresql/data",) + container_slug = "pg" + + def run_env(self, password: str) -> list[str]: + return [ + "-e", + "POSTGRES_USER=sandbox", + "-e", + f"POSTGRES_PASSWORD={password}", + "-e", + "POSTGRES_DB=sandbox", + ] + + def run_command(self, _password: str) -> list[str]: + return [] + + def ready_probe(self, _password: str) -> list[str]: + return ["pg_isready", "-U", "sandbox"] + + def connection(self, host: str, password: str) -> SandboxConnection: + return SandboxConnection( + host=host, + port=self.container_port, + password=password, + user="sandbox", + database="sandbox", + ) + + def emit_env(self, conn: SandboxConnection) -> list[str]: + return [ + "-e", + f"ROBOCO_TEST_DB_HOST={conn.host}", + "-e", + f"ROBOCO_TEST_DB_PORT={conn.port}", + "-e", + f"ROBOCO_TEST_DB_USER={conn.user}", + "-e", + f"ROBOCO_TEST_DB_PASSWORD={conn.password}", + "-e", + f"ROBOCO_TEST_DB_ADMIN_DB={conn.database}", + ] + + +class _RedisEngine(SandboxEngine): + name = "redis" + image = "redis:8-alpine" + container_port = 6379 + ready_deadline = 15.0 + tmpfs: tuple[str, ...] = () + container_slug = "redis" + + def run_env(self, _password: str) -> list[str]: + return [] + + def run_command(self, password: str) -> list[str]: + return ["redis-server", "--requirepass", password] + + def ready_probe(self, password: str) -> list[str]: + return ["redis-cli", "-a", password, "ping"] + + def connection(self, host: str, password: str) -> SandboxConnection: + return SandboxConnection(host=host, port=self.container_port, password=password) + + def emit_env(self, conn: SandboxConnection) -> list[str]: + return [ + "-e", + f"ROBOCO_TEST_REDIS_HOST={conn.host}", + "-e", + f"ROBOCO_TEST_REDIS_PORT={conn.port}", + "-e", + f"ROBOCO_TEST_REDIS_PASSWORD={conn.password}", + ] + + +class _MongoEngine(SandboxEngine): + name = "mongo" + image = "mongo:8-alpine" + container_port = 27017 + ready_deadline = 60.0 + tmpfs = ("/data/db",) + container_slug = "mongo" + + def run_env(self, password: str) -> list[str]: + return [ + "-e", + "MONGO_INITDB_ROOT_USERNAME=sandbox", + "-e", + f"MONGO_INITDB_ROOT_PASSWORD={password}", + "-e", + "MONGO_INITDB_DATABASE=sandbox", + ] + + def run_command(self, _password: str) -> list[str]: + return [] + + def ready_probe(self, password: str) -> list[str]: + return [ + "mongosh", + "--quiet", + "-u", + "sandbox", + "-p", + password, + "--authenticationDatabase", + "admin", + "--eval", + "db.runCommand({ping:1}).ok", + ] + + def connection(self, host: str, password: str) -> SandboxConnection: + return SandboxConnection( + host=host, + port=self.container_port, + password=password, + user="sandbox", + database="admin", + ) + + def emit_env(self, conn: SandboxConnection) -> list[str]: + return [ + "-e", + f"ROBOCO_TEST_MONGO_HOST={conn.host}", + "-e", + f"ROBOCO_TEST_MONGO_PORT={conn.port}", + "-e", + f"ROBOCO_TEST_MONGO_USER={conn.user}", + "-e", + f"ROBOCO_TEST_MONGO_PASSWORD={conn.password}", + "-e", + f"ROBOCO_TEST_MONGO_AUTH_DB={conn.database}", + ] + + +SANDBOX_ENGINES: dict[str, SandboxEngine] = { + e.name: e for e in (_PostgresEngine(), _RedisEngine(), _MongoEngine()) +} +VALID_SANDBOX_SERVICES: frozenset[str] = frozenset(SANDBOX_ENGINES) diff --git a/roboco/runtime/orchestrator.py b/roboco/runtime/orchestrator.py index 76ebf433..9f8646d4 100644 --- a/roboco/runtime/orchestrator.py +++ b/roboco/runtime/orchestrator.py @@ -65,10 +65,10 @@ from roboco.models.runtime import ( AgentInstance, OrchestratorAgentConfig, OrchestratorAgentState, - SandboxInfo, SpawnGitContext, WaitingRecord, ) +from roboco.models.sandbox import SandboxInfo from roboco.runtime.sandbox import SandboxProvisioner from roboco.seeds.initial_data import AGENT_UUIDS from roboco.services.task import ( @@ -2842,46 +2842,18 @@ class AgentOrchestrator: @staticmethod def _append_sandbox_env(cmd: list[str], config: AgentConfig) -> None: - """Inject sandbox DB/Redis env, in place of the prod-creds gate env. + """Inject sandbox engine env, in place of the prod-creds gate env. - Called INSTEAD OF `_append_gate_env` whenever a sandbox was - provisioned for this spawn (`config.sandbox_info` set) — sandbox - replaces, never coexists with, the production gate-env injection. - Reuses the `ROBOCO_TEST_DB_*` names so a project's conftest already - following that convention needs no change; `ROBOCO_TEST_REDIS_*` is - new. + Called INSTEAD OF `_append_gate_env` whenever a sandbox was provisioned + for this spawn (`config.sandbox_info` set) — sandbox replaces, never + coexists with, the production gate-env injection. Emission is driven by + the engine registry via `SandboxInfo.emit_env`, so a new engine's + `ROBOCO_TEST_*` vars land here with no orchestrator change. """ info = config.sandbox_info if info is None: return - if info.postgres is not None: - pg = info.postgres - cmd.extend( - [ - "-e", - f"ROBOCO_TEST_DB_HOST={pg.host}", - "-e", - f"ROBOCO_TEST_DB_PORT={pg.port}", - "-e", - f"ROBOCO_TEST_DB_USER={pg.user}", - "-e", - f"ROBOCO_TEST_DB_PASSWORD={pg.password}", - "-e", - f"ROBOCO_TEST_DB_ADMIN_DB={pg.database}", - ] - ) - if info.redis is not None: - rd = info.redis - cmd.extend( - [ - "-e", - f"ROBOCO_TEST_REDIS_HOST={rd.host}", - "-e", - f"ROBOCO_TEST_REDIS_PORT={rd.port}", - "-e", - f"ROBOCO_TEST_REDIS_PASSWORD={rd.password}", - ] - ) + cmd.extend(info.emit_env()) @staticmethod def _default_spawn_prompt() -> str: diff --git a/roboco/runtime/sandbox.py b/roboco/runtime/sandbox.py index b348928b..b2299588 100644 --- a/roboco/runtime/sandbox.py +++ b/roboco/runtime/sandbox.py @@ -1,4 +1,4 @@ -"""Per-agent-spawn sandbox provisioner: throwaway Postgres/Redis containers. +"""Per-agent-spawn sandbox provisioner: throwaway engine containers. Orchestrator-side sibling containers to the agent container — never docker-in-agent (the socket/CLI stay structurally absent from agent images). @@ -6,6 +6,10 @@ Lifetime tracks the agent container 1:1: provisioned before `docker run`, torn down whenever the agent container is stopped/removed. Standalone and unit-testable: docker plumbing is a thin injected callable, not a dependency on the orchestrator module. + +Engine specs (image, run args, readiness probe, env emission) live in the +pure registry ``roboco/models/sandbox.py``; this module only runs docker +against them. Adding an engine is one entry there — no branch edited here. """ from __future__ import annotations @@ -17,8 +21,13 @@ import time from dataclasses import dataclass from typing import TYPE_CHECKING -from roboco.models.project import VALID_SANDBOX_SERVICES -from roboco.models.runtime import PostgresSandbox, RedisSandbox, SandboxInfo +from roboco.models.sandbox import ( + SANDBOX_ENGINES, + VALID_SANDBOX_SERVICES, + SandboxConnection, + SandboxEngine, + SandboxInfo, +) if TYPE_CHECKING: from collections.abc import Awaitable, Callable @@ -39,10 +48,8 @@ _DOCKER_PS_TIMEOUT_SECONDS = 10.0 # explicitly with a generous deadline breaks it at the source. _DOCKER_PULL_TIMEOUT_SECONDS = 300.0 -# Readiness poll deadlines — pg's first-boot init (initdb + start) is slower -# than redis's near-instant start. -_PG_READY_DEADLINE_SECONDS = 60.0 -_REDIS_READY_DEADLINE_SECONDS = 15.0 +# Readiness poll interval. Per-engine deadlines live on the engine +# (`SandboxEngine.ready_deadline`); only the poll cadence is shared here. _READY_POLL_INTERVAL_SECONDS = 1.0 # Janitor grace: a sandbox is provisioned BEFORE its agent container exists, @@ -79,21 +86,13 @@ class SandboxProvisionError(RuntimeError): """Raised when a sandbox container fails to start or become ready.""" -def _pg_name(agent_id: str) -> str: - return f"roboco-sandbox-pg-{agent_id}" - - -def _redis_name(agent_id: str) -> str: - return f"roboco-sandbox-redis-{agent_id}" - - def _owner_label(agent_id: str) -> str: return f"{_AGENT_CONTAINER_PREFIX}{agent_id}" @dataclass class SandboxProvisioner: - """Provisions/tears down throwaway Postgres+Redis sibling containers. + """Provisions/tears down throwaway engine sibling containers. ``network`` is caller-supplied (the orchestrator's `AGENT_NETWORK` constant) rather than hardcoded here, so a future network-isolation @@ -140,110 +139,55 @@ class SandboxProvisioner: # spawn attempt (and a respawn-tracker strike). await self.teardown(agent_id) self._provisioned_at[_owner_label(agent_id)] = time.monotonic() - postgres: PostgresSandbox | None = None - redis: RedisSandbox | None = None + connections: dict[str, SandboxConnection] = {} try: - if "postgres" in services: - postgres = await self._provision_postgres(agent_id) - if "redis" in services: - redis = await self._provision_redis(agent_id) + for service in services: + engine = SANDBOX_ENGINES[service] + connections[service] = await self._provision_engine(agent_id, engine) except Exception: await self.teardown(agent_id) raise - return SandboxInfo(postgres=postgres, redis=redis) + return SandboxInfo(services=connections) - async def _provision_postgres(self, agent_id: str) -> PostgresSandbox: - name = _pg_name(agent_id) + async def _provision_engine( + self, agent_id: str, engine: SandboxEngine + ) -> SandboxConnection: + name = engine.container_name(agent_id) password = secrets.token_hex(16) run = self._run() - await self._ensure_image("postgres:16-alpine") - rc, _, stderr = await run( - [ - "run", - "-d", - "--name", - name, - "--network", - self.network, - "--label", - SANDBOX_LABEL, - "--label", - f"{_OWNER_LABEL_KEY}={_owner_label(agent_id)}", - "--tmpfs", - "/var/lib/postgresql/data", - "--memory", - "512m", - "--cpus", - "1", - "-e", - "POSTGRES_USER=sandbox", - "-e", - f"POSTGRES_PASSWORD={password}", - "-e", - "POSTGRES_DB=sandbox", - "postgres:16-alpine", - ], - _DOCKER_RUN_TIMEOUT_SECONDS, - ) - if rc != 0: - raise SandboxProvisionError( - f"postgres sandbox run failed for {name}: " - f"{stderr.decode(errors='replace')}" - ) - ready = await self._wait_ready( - name, ["pg_isready", "-U", "sandbox"], _PG_READY_DEADLINE_SECONDS - ) - if not ready: - raise SandboxProvisionError( - f"postgres sandbox {name} did not become ready in time" - ) - return PostgresSandbox( - host=name, port=5432, user="sandbox", password=password, database="sandbox" - ) - - async def _provision_redis(self, agent_id: str) -> RedisSandbox: - name = _redis_name(agent_id) - password = secrets.token_hex(16) - run = self._run() - await self._ensure_image("redis:8-alpine") - rc, _, stderr = await run( - [ - "run", - "-d", - "--name", - name, - "--network", - self.network, - "--label", - SANDBOX_LABEL, - "--label", - f"{_OWNER_LABEL_KEY}={_owner_label(agent_id)}", - "--memory", - "512m", - "--cpus", - "1", - "redis:8-alpine", - "redis-server", - "--requirepass", - password, - ], - _DOCKER_RUN_TIMEOUT_SECONDS, - ) - if rc != 0: - raise SandboxProvisionError( - f"redis sandbox run failed for {name}: " - f"{stderr.decode(errors='replace')}" - ) - ready = await self._wait_ready( + await self._ensure_image(engine.image) + args = [ + "run", + "-d", + "--name", name, - ["redis-cli", "-a", password, "ping"], - _REDIS_READY_DEADLINE_SECONDS, + "--network", + self.network, + "--label", + SANDBOX_LABEL, + "--label", + f"{_OWNER_LABEL_KEY}={_owner_label(agent_id)}", + ] + for mount in engine.tmpfs: + args += ["--tmpfs", mount] + args += ["--memory", "512m", "--cpus", "1"] + args += engine.run_env(password) + args.append(engine.image) + args += engine.run_command(password) + rc, _, stderr = await run(args, _DOCKER_RUN_TIMEOUT_SECONDS) + if rc != 0: + raise SandboxProvisionError( + f"{engine.name} sandbox run failed for {name}: " + f"{stderr.decode(errors='replace')}" + ) + ready = await self._wait_ready( + name, engine.ready_probe(password), engine.ready_deadline ) if not ready: raise SandboxProvisionError( - f"redis sandbox {name} did not become ready in time" + f"{engine.name} sandbox {name} did not become ready in time" ) - return RedisSandbox(host=name, port=6379, password=password) + return engine.connection(name, password) async def _wait_ready( self, container: str, probe_cmd: list[str], deadline_seconds: float @@ -263,9 +207,9 @@ class SandboxProvisioner: return False async def teardown(self, agent_id: str) -> None: - """Idempotent: stop+kill+rm both sandbox containers. Never raises.""" - for name in (_pg_name(agent_id), _redis_name(agent_id)): - await self._teardown_one(name) + """Idempotent: stop+kill+rm every engine's sandbox container. Never raises.""" + for engine in SANDBOX_ENGINES.values(): + await self._teardown_one(engine.container_name(agent_id)) async def _teardown_one(self, name: str) -> None: run = self._run() diff --git a/tests/e2e_smoke/arcs.py b/tests/e2e_smoke/arcs.py index faa4895e..4fb35d87 100644 --- a/tests/e2e_smoke/arcs.py +++ b/tests/e2e_smoke/arcs.py @@ -8,6 +8,7 @@ with unique slugs so scenarios never collide on constraints. from __future__ import annotations +import time from typing import TYPE_CHECKING, Any from uuid import uuid4 @@ -160,6 +161,36 @@ def task_state(stack: E2EStack, task_id: Any) -> dict[str, Any]: return state +def wait_for_status( + stack: E2EStack, + task_id: Any, + expected: str, + *, + timeout: float = 10.0, + interval: float = 0.25, +) -> dict[str, Any]: + """Poll ``task_state`` until ``status == expected`` or timeout. + + The e2e stack commits on the uvicorn thread's event loop and reads via a + separate loop (``run_db`` -> ``asyncio.run`` with a fresh engine). A + terminal single point-read can race a still-draining completion hook on a + contended runner and observe a pre-terminal state; the bounded poll + absorbs that transient. A genuine state bug still surfaces: the timeout + branch asserts against the last-read state, so a real regression fails + loudly with the actual (non-terminal) state instead of a misleading + one-shot mismatch. + """ + deadline = time.monotonic() + timeout + last: dict[str, Any] = {} + while True: + last = task_state(stack, task_id) + if last["status"] == expected: + return last + if time.monotonic() >= deadline: + assert last["status"] == expected, last + time.sleep(interval) + + def dispatcher_assign(stack: E2EStack, task_id: Any, agent_id: Any) -> None: """Mirror the dispatcher's claim-for-PM lane (_dispatch_pm_review_work): pr_pass clears ownership by design and the orchestrator re-claims the diff --git a/tests/e2e_smoke/test_megatask_umbrella.py b/tests/e2e_smoke/test_megatask_umbrella.py index 05557723..b9a902d0 100644 --- a/tests/e2e_smoke/test_megatask_umbrella.py +++ b/tests/e2e_smoke/test_megatask_umbrella.py @@ -41,6 +41,7 @@ from tests.e2e_smoke.arcs import ( seed_task, set_branch_name, task_state, + wait_for_status, wire_dependency, ) from tests.e2e_smoke.harness import ScriptedAgent, expect_error, expect_ok @@ -429,8 +430,7 @@ def test_megatask_umbrella_sequenced_close(e2e_stack: E2EStack) -> None: ), "main_pm complete umbrella", ) - escalated = task_state(stack, umbrella_id) - assert escalated["status"] == "awaiting_ceo_approval", escalated + escalated = wait_for_status(stack, umbrella_id, "awaiting_ceo_approval") assert escalated["pr_number"] is None, escalated resp = httpx.post( @@ -442,8 +442,7 @@ def test_megatask_umbrella_sequenced_close(e2e_stack: E2EStack) -> None: assert resp.status_code == HTTPStatus.OK, ( f"ceo-approve: {resp.status_code} {resp.text[:1500]}" ) - final = task_state(stack, umbrella_id) - assert final["status"] == "completed", final + final = wait_for_status(stack, umbrella_id, "completed") assert final["pr_number"] is None, final assert origin_file(stack, "master", "rs1.txt") assert origin_file(stack, "master", "rs2.txt") diff --git a/tests/unit/models/test_project_sandbox_services.py b/tests/unit/models/test_project_sandbox_services.py index cf82bbfd..d0a0ee2a 100644 --- a/tests/unit/models/test_project_sandbox_services.py +++ b/tests/unit/models/test_project_sandbox_services.py @@ -1,9 +1,9 @@ """Project.sandbox_services / ProjectUpdate.sandbox_services validation. -Only "postgres" and "redis" are recognized sandbox services (mirrors the -provisioner's VALID_SANDBOX_SERVICES) — an unknown value must be rejected with -a clear message rather than silently accepted and later failing at provision -time inside a container spawn. +Recognized services are whatever the engine registry exposes +(``VALID_SANDBOX_SERVICES`` in ``roboco.models.sandbox`` — postgres, redis, +mongo) — an unknown value must be rejected with a clear message rather than +silently accepted and later failing at provision time inside a container spawn. """ from __future__ import annotations @@ -32,6 +32,11 @@ def test_project_accepts_valid_sandbox_services() -> None: assert project.sandbox_services == ["postgres", "redis"] +def test_project_accepts_mongo() -> None: + project = _project(sandbox_services=["mongo"]) + assert project.sandbox_services == ["mongo"] + + def test_project_normalizes_sandbox_services_order_and_dupes() -> None: project = _project(sandbox_services=["redis", "postgres", "redis"]) assert project.sandbox_services == ["postgres", "redis"] diff --git a/tests/unit/models/test_sandbox_engines_parity.py b/tests/unit/models/test_sandbox_engines_parity.py new file mode 100644 index 00000000..be9b750d --- /dev/null +++ b/tests/unit/models/test_sandbox_engines_parity.py @@ -0,0 +1,52 @@ +"""Engine registry / allowlist parity + per-engine internal consistency. + +The valid-service allowlist is derived from the registry +(``VALID_SANDBOX_SERVICES = frozenset(SANDBOX_ENGINES)``), so the two must stay +in lockstep — a drift guard against adding an engine class without registering +it (or vice versa). Each engine's emitted env must also reference only the +connection fields it actually populates (no ``None`` leaking into an env value). +""" + +from __future__ import annotations + +from roboco.models.sandbox import ( + SANDBOX_ENGINES, + VALID_SANDBOX_SERVICES, + SandboxInfo, +) + +_ENV_HOST_PREFIX = { + "postgres": "ROBOCO_TEST_DB_HOST", + "redis": "ROBOCO_TEST_REDIS_HOST", + "mongo": "ROBOCO_TEST_MONGO_HOST", +} + + +def test_allowlist_matches_registry() -> None: + assert frozenset(SANDBOX_ENGINES) == VALID_SANDBOX_SERVICES + assert set(SANDBOX_ENGINES) == {"postgres", "redis", "mongo"} + + +def test_each_engine_has_unique_container_slug_and_image() -> None: + slugs = {e.container_slug for e in SANDBOX_ENGINES.values()} + images = {e.image for e in SANDBOX_ENGINES.values()} + assert len(slugs) == len(SANDBOX_ENGINES) + assert len(images) == len(SANDBOX_ENGINES) + + +def test_each_engine_emit_env_references_only_populated_fields() -> None: + # An engine that does not set `user`/`database` must not emit a `None` value. + for engine in SANDBOX_ENGINES.values(): + conn = engine.connection(host=f"h-{engine.name}", password="pw") + env = " ".join(engine.emit_env(conn)) + assert "None" not in env, f"{engine.name} leaked None into env: {env}" + + +def test_sandbox_info_emit_env_aggregates_every_engine() -> None: + services = { + name: engine.connection(host=f"h-{name}", password="pw") + for name, engine in SANDBOX_ENGINES.items() + } + flat = " ".join(SandboxInfo(services=services).emit_env()) + for name in SANDBOX_ENGINES: + assert _ENV_HOST_PREFIX[name] in flat diff --git a/tests/unit/runtime/test_sandbox_env.py b/tests/unit/runtime/test_sandbox_env.py index 8524784c..67843a0e 100644 --- a/tests/unit/runtime/test_sandbox_env.py +++ b/tests/unit/runtime/test_sandbox_env.py @@ -12,12 +12,8 @@ from pathlib import Path from unittest.mock import AsyncMock import pytest -from roboco.models.runtime import ( - OrchestratorAgentConfig, - PostgresSandbox, - RedisSandbox, - SandboxInfo, -) +from roboco.models.runtime import OrchestratorAgentConfig +from roboco.models.sandbox import SandboxConnection, SandboxInfo from roboco.runtime.orchestrator import AgentOrchestrator @@ -32,16 +28,18 @@ def _config(sandbox_info: SandboxInfo | None = None) -> OrchestratorAgentConfig: def test_append_sandbox_env_injects_postgres_and_redis() -> None: info = SandboxInfo( - postgres=PostgresSandbox( - host="roboco-sandbox-pg-dev-1", - port=5432, - user="sandbox", - password="pgpw", - database="sandbox", - ), - redis=RedisSandbox( - host="roboco-sandbox-redis-dev-1", port=6379, password="rdpw" - ), + services={ + "postgres": SandboxConnection( + host="roboco-sandbox-pg-dev-1", + port=5432, + password="pgpw", + user="sandbox", + database="sandbox", + ), + "redis": SandboxConnection( + host="roboco-sandbox-redis-dev-1", port=6379, password="rdpw" + ), + } ) cmd: list[str] = [] AgentOrchestrator._append_sandbox_env(cmd, _config(info)) @@ -58,13 +56,15 @@ def test_append_sandbox_env_injects_postgres_and_redis() -> None: def test_append_sandbox_env_postgres_only_omits_redis_vars() -> None: info = SandboxInfo( - postgres=PostgresSandbox( - host="roboco-sandbox-pg-dev-1", - port=5432, - user="sandbox", - password="pgpw", - database="sandbox", - ) + services={ + "postgres": SandboxConnection( + host="roboco-sandbox-pg-dev-1", + port=5432, + password="pgpw", + user="sandbox", + database="sandbox", + ) + } ) cmd: list[str] = [] AgentOrchestrator._append_sandbox_env(cmd, _config(info)) @@ -73,6 +73,29 @@ def test_append_sandbox_env_postgres_only_omits_redis_vars() -> None: assert not any(v.startswith("ROBOCO_TEST_REDIS_") for v in cmd) +def test_append_sandbox_env_injects_mongo() -> None: + info = SandboxInfo( + services={ + "mongo": SandboxConnection( + host="roboco-sandbox-mongo-dev-1", + port=27017, + password="mpw", + user="sandbox", + database="admin", + ) + } + ) + cmd: list[str] = [] + AgentOrchestrator._append_sandbox_env(cmd, _config(info)) + + assert "ROBOCO_TEST_MONGO_HOST=roboco-sandbox-mongo-dev-1" in cmd + assert "ROBOCO_TEST_MONGO_PORT=27017" in cmd + assert "ROBOCO_TEST_MONGO_USER=sandbox" in cmd + assert "ROBOCO_TEST_MONGO_PASSWORD=mpw" in cmd + assert "ROBOCO_TEST_MONGO_AUTH_DB=admin" in cmd + assert not any(v.startswith("ROBOCO_TEST_DB_") for v in cmd) + + def test_append_sandbox_env_noop_without_sandbox_info() -> None: cmd: list[str] = [] AgentOrchestrator._append_sandbox_env(cmd, _config(None)) @@ -120,9 +143,11 @@ async def test_spawn_container_uses_sandbox_env_when_sandbox_active( _stub_spawn_container_collaborators(monkeypatch, orch, calls) info = SandboxInfo( - postgres=PostgresSandbox( - host="h", port=5432, user="sandbox", password="pw", database="sandbox" - ) + services={ + "postgres": SandboxConnection( + host="h", port=5432, password="pw", user="sandbox", database="sandbox" + ) + } ) await orch._spawn_container(_config(info)) @@ -155,9 +180,11 @@ async def test_spawn_container_stale_clear_spares_fresh_sandbox( monkeypatch.setattr(orch, "_remove_container", remove) info = SandboxInfo( - postgres=PostgresSandbox( - host="h", port=5432, user="sandbox", password="pw", database="sandbox" - ) + services={ + "postgres": SandboxConnection( + host="h", port=5432, password="pw", user="sandbox", database="sandbox" + ) + } ) await orch._spawn_container(_config(info)) diff --git a/tests/unit/runtime/test_sandbox_provision_spawn.py b/tests/unit/runtime/test_sandbox_provision_spawn.py index 6b85352f..6edd1940 100644 --- a/tests/unit/runtime/test_sandbox_provision_spawn.py +++ b/tests/unit/runtime/test_sandbox_provision_spawn.py @@ -14,7 +14,7 @@ from unittest.mock import AsyncMock, MagicMock, patch import pytest from roboco.config import settings -from roboco.models.runtime import PostgresSandbox, SandboxInfo +from roboco.models.sandbox import SandboxConnection, SandboxInfo from roboco.runtime.orchestrator import AgentOrchestrator, AgentReadinessError @@ -100,9 +100,11 @@ async def test_opted_in_project_provisions_sandbox( project_service = MagicMock() project_service.get_by_slug = AsyncMock(return_value=project) info = SandboxInfo( - postgres=PostgresSandbox( - host="h", port=5432, user="sandbox", password="pw", database="sandbox" - ) + services={ + "postgres": SandboxConnection( + host="h", port=5432, password="pw", user="sandbox", database="sandbox" + ) + } ) sandbox.provision.return_value = info diff --git a/tests/unit/runtime/test_sandbox_provisioner.py b/tests/unit/runtime/test_sandbox_provisioner.py index 68f7fc3d..ebbb9f7c 100644 --- a/tests/unit/runtime/test_sandbox_provisioner.py +++ b/tests/unit/runtime/test_sandbox_provisioner.py @@ -15,6 +15,7 @@ from roboco.runtime.sandbox import SandboxProvisioner, SandboxProvisionError _NETWORK = "roboco_default" _PG_PORT = 5432 _REDIS_PORT = 6379 +_MONGO_PORT = 27017 class _FakeRunner: @@ -71,10 +72,12 @@ class _FakeRunner: @pytest.fixture(autouse=True) def _fast_readiness_deadlines(monkeypatch: pytest.MonkeyPatch) -> None: - """Shrink the polling deadlines so the timeout path is fast in tests.""" - monkeypatch.setattr(sandbox_module, "_PG_READY_DEADLINE_SECONDS", 0.05) - monkeypatch.setattr(sandbox_module, "_REDIS_READY_DEADLINE_SECONDS", 0.05) + """Shrink the polling cadence + every engine's deadline so the timeout path + is fast in tests. Deadlines live on the engine instances now (not module + constants), so monkeypatch them on the registry.""" monkeypatch.setattr(sandbox_module, "_READY_POLL_INTERVAL_SECONDS", 0.01) + for engine in sandbox_module.SANDBOX_ENGINES.values(): + monkeypatch.setattr(engine, "ready_deadline", 0.05) @pytest.mark.asyncio @@ -84,16 +87,16 @@ async def test_provision_both_services_happy_path() -> None: info = await provisioner.provision("dev-1", ["postgres", "redis"]) - assert info.postgres is not None - assert info.postgres.host == "roboco-sandbox-pg-dev-1" - assert info.postgres.port == _PG_PORT - assert info.postgres.user == "sandbox" - assert info.postgres.database == "sandbox" - assert info.redis is not None - assert info.redis.host == "roboco-sandbox-redis-dev-1" - assert info.redis.port == _REDIS_PORT + pg = info.services["postgres"] + assert pg.host == "roboco-sandbox-pg-dev-1" + assert pg.port == _PG_PORT + assert pg.user == "sandbox" + assert pg.database == "sandbox" + rd = info.services["redis"] + assert rd.host == "roboco-sandbox-redis-dev-1" + assert rd.port == _REDIS_PORT # Passwords are per-sandbox random tokens, not equal to each other. - assert info.postgres.password != info.redis.password + assert pg.password != rd.password @pytest.mark.asyncio @@ -112,6 +115,27 @@ async def test_provision_labels_are_correct() -> None: assert "roboco.sandbox.owner=roboco-agent-dev-2" in labels +@pytest.mark.asyncio +async def test_provision_mongo_engine() -> None: + runner = _FakeRunner(run_rc=0, exec_rc=0) + provisioner = SandboxProvisioner(network=_NETWORK, runner=runner) + + info = await provisioner.provision("dev-mongo", ["mongo"]) + + mongo = info.services["mongo"] + assert mongo.host == "roboco-sandbox-mongo-dev-mongo" + assert mongo.port == _MONGO_PORT + assert mongo.user == "sandbox" + assert mongo.database == "admin" + run_call = next(c for c in runner.calls if c[0] == "run") + assert "mongo:8-alpine" in run_call + # MONGO_INITDB_ROOT_PASSWORD env is baked into the run. + assert any(a.startswith("MONGO_INITDB_ROOT_PASSWORD=") for a in run_call) + # /data/db tmpfs mount for the engine. + assert "--tmpfs" in run_call + assert run_call[run_call.index("--tmpfs") + 1] == "/data/db" + + @pytest.mark.asyncio async def test_provision_readiness_timeout_tears_down_and_raises() -> None: runner = _FakeRunner(run_rc=0, exec_rc=1) # container starts, never ready