feat(sandbox): on-demand provisioning via request_sandbox verb (#338)

* feat(sandbox): on-demand request_sandbox verb replaces eager provisioning

Sandboxes were provisioned at every agent spawn for opted-in projects,
so every role paid the sidecar spin-up and a provisioning failure
refused the spawn. Provisioning now happens when an agent asks: the
request_sandbox do-verb (dev + QA) reaches the orchestrator through
ContentActionsDeps, ensure_sandbox provisions idempotently with an
in-memory per-agent cache (evicted at teardown and janitor sweep), and
creds return in the envelope payload including ready-to-export
ROBOCO_TEST_* values. Spawn now only injects a marker env naming the
available services plus a briefing line; sandbox failures can no longer
refuse a spawn. Teardown lifecycle unchanged.

* feat(sandbox): harden request_sandbox + Phase 3 wiring proof and docs

Hardening from adversarial review: ensure_sandbox now provisions the
project's full opted-in set on first request (a later superset can
never tear down a live sandbox mid-use), serializes per-agent behind an
asyncio lock (a client timeout-retry no longer races its own in-flight
provision), and verifies container liveness on every cache hit (a dead
sandbox evicts and re-provisions instead of serving dead creds). MCP
client budget 720->1080s for the full-set cold case. Phase 3: e2e smoke
wiring test (manifest grants + guard-chain envelopes over the real
API), sandbox-db/tools/map docs and CLAUDE.md rewritten for on-demand.

* feat(sandbox): release sandboxes when the agent's work ends

CEO directive: sidecars must not dangle once the agent is done. The six
work-ending verbs (i_am_done, unclaim, i_am_idle, pass_review,
fail_review, i_documented) now release the caller's sandbox best-effort
on their success path via release_sandbox (lock + teardown + cache
evict; a no-sandbox agent costs a dict lookup). Container removal and
the janitor remain the backstop; a re-request provisions fresh.

* test(sandbox): monkeypatch the release hook instead of method assignment

mypy method-assign rejected the direct AsyncMock assignments; the prior
static gate ran before this test file landed.

* test(sandbox): guard envelope evidence for mypy in verb tests

* chore(prompts): regenerate verb tables for request_sandbox

* chore: resolve merge with master (breadcrumbs + statement budget)

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>

docs/map/orchestrator.md

@@ -80,8 +80,9 @@ The AgentOrchestrator is the runtime brain of RoboCo: it owns the per-agent Dock
 | AgentOrchestrator._should_skip_live_reap | method | roboco/runtime/orchestrator.py:8750 | Spare a live container from reaping UNLESS wedged (grok) or gateway-broken past grace (those kill+evict). |
 | AgentOrchestrator._assignee_is_provider_parked | method | roboco/runtime/orchestrator.py:8527 | True if a task's assignee is provider-parked; reaper skips it so the claim survives for probe-resume. |
 | AgentOrchestrator._reap_with_service | method | roboco/runtime/orchestrator.py:8770 | Inner stale-claim reaper: skip live (unless wedged/broken), skip provider-parked, unclaim_for_reaper the rest. |
-| AgentOrchestrator._maybe_provision_sandbox | method | roboco/runtime/orchestrator.py:2029 | Provision this spawn's sandbox engines (postgres/redis/mongo via `SandboxProvisioner` + the `SANDBOX_ENGINES` registry) when `sandbox_db_enabled` + the project's `sandbox_services` are set; None (byte-for-byte legacy path) otherwise. Fail-loud once opted in — a provisioning failure refuses the spawn. |
-| AgentOrchestrator._append_sandbox_env | staticmethod | roboco/runtime/orchestrator.py:2639 | Inject the sandbox's `ROBOCO_TEST_*` env (postgres `ROBOCO_TEST_DB_*` / redis `ROBOCO_TEST_REDIS_*` / mongo `ROBOCO_TEST_MONGO_*`) via `SandboxInfo.emit_env` over the engine registry, called INSTEAD OF `_append_gate_env` whenever a sandbox was provisioned for this spawn. Emission is registry-driven, so a new engine's env vars land here with no orchestrator change. |
+| AgentOrchestrator._sandbox_available_services | method | roboco/runtime/orchestrator.py:2223 | On-demand model (2026-07-08): availability probe only — which services this spawn's project opted into (`sandbox_db_enabled` + `sandbox_services`); `[]` if off/not-opted, byte-for-byte legacy path. No provisioning happens here anymore (see `ensure_sandbox`), so a spawn never fails on sandbox infra. |
+| AgentOrchestrator.ensure_sandbox | method | roboco/runtime/orchestrator.py:2251 | On-demand model: idempotent provision called by the `request_sandbox` do-verb. Cache hit (`_sandbox_info`, keyed by agent slug) covering the requested services returns the same creds; a miss provisions via `SandboxProvisioner.provision` (pre-clear tears down stale same-named containers) and caches the result. Evicted at teardown and by the janitor sweep; in-memory only — an orchestrator restart forgets it (next call re-provisions). |
+| AgentOrchestrator._append_sandbox_marker_env | staticmethod | roboco/runtime/orchestrator.py:2846 | On-demand model: replaces the old eager `_append_sandbox_env` — injects only a cheap informational marker (`ROBOCO_SANDBOX_SERVICES_AVAILABLE=<csv>`), never creds, naming the services `request_sandbox()` will provision on demand. Called INSTEAD OF `_append_gate_env` for an opted-in project's spawn. |
 | AgentOrchestrator._sandbox_janitor_sweep | method | roboco/runtime/orchestrator.py:9426 | Best-effort: remove sandbox containers whose owner agent is gone; rides the reaper tick, error-isolated. |
 | AgentOrchestrator._x_mentions_poll_loop | method | roboco/runtime/orchestrator.py:7431 | Default-off X-engine mentions-poll tick loop (`x_engine_enabled`); release-post drafts are event-driven, not from this loop. |
 | AgentOrchestrator._run_x_mentions_cycle | method | roboco/runtime/orchestrator.py:7453 | One mentions-poll pass: `get_x_engine(db).run_cycle()` + commit; testable without the sleep. |
@@ -197,7 +198,7 @@ stateDiagram-v2
 - `ROBOCO_GROK_MAX_COST_USD` — grok budget kill-switch; `_GROK_RATE_LIMIT_EXIT_CODE=75`, `_GROK_AUTH_EXIT_CODE=78`, `_PROBE_GIVE_UP_THRESHOLD=30`.
 - `ROBOCO_CLAUDE_STUCK_KILL_SECONDS` (default 3600, min 600) — heartbeat-stale kill threshold for non-GROK agents; controls `_maybe_kill_stuck_claude`.
 - `ROBOCO_DISPATCHER_INTERVAL_SECONDS` (30), `ROBOCO_INTERACTIVE_IDLE_REAP_SECONDS`, `ROBOCO_GROK_*` backoff constants.
-- `ROBOCO_SANDBOX_DB_ENABLED` (default off) — master switch for the sandboxed per-agent-spawn engine provisioner (`_maybe_provision_sandbox`/`_append_sandbox_env`/`_sandbox_janitor_sweep`); a project participates only when its `sandbox_services` column is also set. The service set is the `SANDBOX_ENGINES` registry (postgres / redis / mongo); adding an engine needs no orchestrator or env-emitter change.
+- `ROBOCO_SANDBOX_DB_ENABLED` (default off) — master switch for the sandboxed per-agent test DB/Redis/Mongo. On-demand model (2026-07-08): nothing is provisioned at spawn — `_sandbox_available_services` only probes+names the project's opted-in set (marker env), and `ensure_sandbox` provisions idempotently when an agent calls the `request_sandbox` do-verb (`ContentActions.request_sandbox`, gateway/content_actions.py). Teardown (`_sandbox_janitor_sweep` + every container-removal path) is unchanged. A project participates only when its `sandbox_services` column is also set. The service set is the `SANDBOX_ENGINES` registry (postgres / redis / mongo); adding an engine needs no orchestrator or env-emitter change.
 - `ROBOCO_DB_NETWORK_ISOLATED` (default off; set by the compose topology that carries the `roboco_data` network) — suppresses the legacy `_append_gate_env` prod-creds injection when postgres/redis are unreachable from the agent mesh.
 - `ROBOCO_CLOUD_AUTH_ENABLED` (+ `_EMAIL`/`_PASSWORD`/`_SECRET`/`_COOKIE_MAX_AGE`, default off) — cloud auth master switch; read by `roboco.api.deps.get_agent_context`/`roboco.api.auth.*`, not the orchestrator itself, but gates whether a spawned agent's own HMAC-token identity path is the sole non-CEO auth route.
 - `ROBOCO_X_ENGINE_ENABLED` (+ `_mentions_interval_seconds` / `_mentions_max_per_cycle` / `_mentions_min_engagement` / `_max_open_posts` / `_account_user_id` / `_request_timeout_seconds`, default off) — gates `_x_mentions_poll_loop`.