A DB-durable PM-respawn counter (migration 051 / e2f7097a) wedges forever
once tripped: the only reset was a task status change, which can't happen
while the breaker blocks the spawn. So a deploy that fixes the underlying
loop (auth/prompt/schema) couldn't clear the wedge without manual DELETE
surgery on respawn_tracker — the 2026-07-06 pr-reviewer-1 loop, where the
auth fix cleared the 401 but count=63 survived restart and kept skipping
the dispatcher spawn for an external-PR task.
Freeze last_check at the trip tick and, after pm_respawn_trip_cooldown_seconds
(default 300), let ONE spawn through. A still-wedged task re-trips after the
threshold (bounded re-burn ~3 spawns per window); a fixed one advances and
the status-change path fully resets. Restore re-stamps last_check to now, so
a freshly restored row still trips immediately — durability preserved, which
is why the migration-051 persistence tests still pass.
The token was signed over the agent slug (_append_agent_auth_env) while the
MCP servers send X-Agent-ID as the agent UUID (_generate_mcp_config, since
453a7ae2 — gateway v1 parses X-Agent-ID as Annotated[UUID]). The middleware
verified HMAC(uuid:role:team) against a slug-signed token → 'signature
mismatch', token_unsigned=false. Latent for 2 months until 6ed4e139/53391f22
made the MCP servers forward the token.
The c0328971 startup heal missed it: docker exec printenv reads the
container-level ROBOCO_AGENT_ID (the slug), so the heal verified the
slug-signed token against the slug → matched → didn't kill the stale
container, which kept 401ing (its MCP server sends the UUID).
Fix: sign the token over the UUID, set the container ROBOCO_AGENT_ID to the
UUID too (so the SDK server — which inherits container env, not the MCP
manifest env — sends UUID consistently), and resolve the container-env id to
its UUID in _heal_stale_agent_tokens so pre-fix stale containers are evicted
on next restart. Regression test: test_heal_kills_slug_env_container_with_slug_signed_token.
A token is signed once at spawn. If ROBOCO_AGENT_AUTH_SECRET drifts
afterwards (a .env change, a compose recreate that reloads the
orchestrator's env without recreating agent containers, an image
redeploy), the surviving agent keeps sending its old token and the
middleware 401s every verb with 'signature mismatch'. The container
stays alive heartbeating, so the reaper never reclaims it and no fresh
agent spawns: the fleet stalls.
_heal_stale_agent_tokens runs at startup (before _readopt_running_agents)
and kills each running agent container whose baked-in token no longer
verifies against the current secret, so normal dispatch re-spawns it
with a freshly signed token. Inert when the secret is unset (dev):
verify fails for every token without a secret, so the heal would kill
the whole fleet without this gate. Best-effort: a probe failure leaves
the container alone (the reaper still covers it).
The orchestrator injects ROBOCO_AGENT_TOKEN=UNSIGNED when the HMAC secret
is unset at spawn. The API middleware rejects a presented-but-unverifiable
token with 401 'signature mismatch' even in dev mode (auth not required),
so forwarding UNSIGNED turned every flow/do/SDK/secretary/git verb into a
401 — the live pr_reviewer/i_am_idle signature-mismatch loop. Omit the
header when the token is the UNSIGNED sentinel at all five agent-side
header builders; dev accepts a missing token, prod 401s with 'Missing
X-Agent-Token' (the clear respawn-with-secret signal). Add a structlog
diagnostic on the middleware reject path so the next mismatch logs the
exact (id, role, team, token_unsigned, auth_required) inputs.
_agent_api_headers sent the UNSIGNED sentinel when ROBOCO_AGENT_AUTH_SECRET
was unset, but the dev-mode middleware rejects a presented-but-unverifiable
token with 401 signature mismatch (while accepting a missing one). The
cell-PM auto-submit self-call 401'd in every dev run, regressing
test_auto_submit_cuts_the_pm_turn. Attach the token only when a secret is
set. Also fix the FromClause.update mypy error in the per-project
video-engine opt-out test (ORM row load + flush).
The prior fix (6ed4e139) covered the flow/do MCP servers but missed four
other agent->orchestrator call sites that built the header dict by hand
and omitted X-Agent-Token and/or X-Agent-Team. With ROBOCO_AGENT_AUTH_REQUIRED
armed on the NAS, every one 401s:
- agent_sdk/server.py: the session-end post-mortem flush
(/api/journals/me/entries), A2A persistence + offline fallback
(/api/a2a/*), and the stopped-without-transition auto-substitute
(/api/tasks/auto-substitute) — all sent only X-Agent-ID/Role, so each
401'd 'Missing X-Agent-Token'. Add a shared _agent_headers() helper
(mirroring flow_server._build_headers) and route all four through it.
- agent_sdk/secretary_driver.py: _headers() sent the token but not the
team, so the HMAC gate 401'd with signature mismatch (secretary is
board-team; token signed with team='board', verified with team='').
Add the team header.
- mcp/git_readonly.py: the read-only git MCP sent only X-Agent-ID/Role
— no token, no team — so /api/git/* 401'd once auth was armed. Convert
the static _HEADERS to a _headers() helper with team + token.
- runtime/orchestrator.py: the cell-PM auto-submit self-API call acted
as a PM with a hand-built {X-Agent-ID, X-Agent-Role} dict — no token,
no team — 401ing under auth-required. Add _agent_api_headers(uuid,
role) mirroring _system_api_headers, and use it.
Tests: _agent_headers round-trip (token + team, team-omitted when None),
_agent_api_headers carries a signed PM token + team.
Mirrors ci_watch_enabled (migration 048): the global
ROBOCO_VIDEO_ENGINE_ENABLED flag arms the subsystem; the new
projects.video_engine_enabled column (migration 063) opts a repo into
authoring against its motion/ dir. VideoEngine._opted_in_project no-ops
open_video_task at the single chokepoint covering all three trigger
paths (on-release, on-spotlight, CEO on-demand) until the operator
flips it in the panel edit-project dialog. Existing projects stay
opted out (server_default=false).
flow_server._build_headers and do_server._build_headers constructed
only X-Agent-ID/Role/Correlation-ID, omitting X-Agent-Token and
X-Agent-Team (unlike ApiClient._get_agent_headers used by the other
MCP servers). Latent since the gateway refactor — surfaced when
ROBOCO_AGENT_AUTH_REQUIRED=true was armed on the NAS, 401-ing every
flow/do verb with 'Missing X-Agent-Token header'. Add both headers
(mirroring ApiClient) so the HMAC gate passes. Tests assert the
headers are now injected.
compositionId flowed unvalidated from the POST body into path.join
under extractDir/motion/compositions/, so a '../..'-style value could
escape the composition dir (CodeQL: Uncontrolled data used in path
expression). Validate at the trust boundary in server.js
(/^[A-Za-z0-9_-]+$/) and add a path.resolve + startsWith containment
check in render.js so it stays safe regardless of caller.
reflow_md.py walks the filesystem via rglob('*.md') and skips tooling dirs
(.venv, .mypy_cache, .pytest_cache, ...) but not .superpowers/ — the
superpowers SDD workflow's scratch dir (briefs, reports, progress ledger,
all gitignored). A dev running SDD locally would hit a false markdown-prose
gate failure on those transient files. Add .superpowers to SKIP_DIRS,
consistent with the existing tooling-scratch exclusions.
T7 widened _build_app's db_session param to AsyncSession | None (to drop the
4x # type: ignore[arg-type] on the DB-independent _build_app(None, ...) calls)
but left the inner _override_db fixture typed AsyncIterator[AsyncSession] —
so 'yield db_session' yielded AsyncSession | None into a declared AsyncSession,
and mypy failed at test_video_routes.py:177 ('Incompatible types in yield').
The DB-independent media tests pass db_session=None deliberately: their route
uses a monkeypatched task service and never awaits the session, so yielding
None is safe at runtime. Type the override's yield as AsyncSession | None to
match — no cast, no # type: ignore, no assert, runtime behavior unchanged.
The 3 media tests (3 passed) and the 19 db-gated tests (skipped locally) hold.
Final whole-branch review (Opus) triaged two FIX items from the SDD nits
ledger; the rest ship as-is.
- render.js: a synchronous throw from createRenderJob (post-mkdtemp, not
awaited) left an empty outDir on disk — the outer catch only reclaimed
extractDir. Reclaim outDir too when it exists, and correct the stale
comment that claimed the out dir was never created.
- {vertical,square}.html: the 4th highlights <li> lived in the DOM hidden
only by JS, so a no-JS / failed-script render would show an empty bullet.
Start it style="display:none" and reveal on populate, so an unscripted
render shows nothing instead.
Vitest smoke (release-announcement.test.js) 4/4 green; render.js syntax
checked. Python suite untouched by this fix (JS/HTML only).
* feat(video): Phase A — VideoEngine origination spine + held-source gates
New default-off engine skeleton: opens a UX/UI authoring task (source=video, assigned to a ux-dev, LOW complexity to clear the dev-needs-subtasks guard) and materializes a held CEO-approval draft (source=video_post). Excludes video_post from all three held-source skip sites; adds the video_draft marker, six config flags, and the feature-flag entries. Origination + gate behavior unit-tested.
* refactor(orchestrator): fold _dispatch_dev_work skip chain into a helper
The per-source if/continue chain grew past xenon's --max-absolute B when the video_post held source joined it. Extract _is_non_dev_dispatch_source (every held-CEO source plus the two Board exploration sources) so the dev loop's skip is one flat call. Behavior-identical.
* feat(video): Phase B — propose_video do-tool (metadata-only, team-gated)
UX/UI dev records a video's composition ref + per-platform captions onto the authoring task's video_draft marker. Team-gated at runtime via _caller_team (Role.DEVELOPER can't tell a ux-dev from a be-dev). Resolves the caller's ACTIVE task via get_active_task_for_agent, not an oldest-first scan that would clobber a second open video task. Metadata only, no render. Wired through do_server + route + schema; added to _DEV_DO.
* feat(video): Phase D — render loop + RemotionRenderer client
Orchestrator-async _video_render_loop renders a completed authoring task's merged composition to MP4 (vertical + square) via the remotion-renderer sidecar and materializes the held video_post draft. RemotionRenderer tars the read-clone's motion/ source, POSTs it, and saves the returned MP4 bytes to a TASK-scoped local path (no shared volume; a composition is reused across videos so a composition-scoped path would clobber an earlier draft). Render failures bounded-retry (read-clone catch-up window, transient sidecar) up to a cap, then terminal-fail. Client tested vs a mock transport; loop vs a mock renderer + real DB.
* feat(video): Phase C — release / spotlight / on-demand video triggers
Three entry points open a UX/UI video-authoring task via VideoEngine.open_video_task: (1) a published release drafts a companion video — best-effort in ReleaseProposalService.approve, never fails the publish; script from the CHANGELOG via the local model with a template fallback. (2) propose_feature_spotlight gains optional wants_video/video_script — best-effort, gated on video_on_spotlight, default-off leaves the spotlight flow byte-for-byte unchanged. (3) POST /video/request (CEO-only) for an on-demand brief, with clean disabled/not_opened responses. All gated on video_engine_enabled.
* fix(video): savepoint-isolate video-task inserts (F042 poisoned session)
The best-effort try/except around open_video_task (release-publish + spotlight hooks) swallowed the Python exception, but a DBAPI error at the insert flush left the shared session must-rollback — so the caller's next commit (release finalize / request boundary) threw PendingRollbackError: the release stuck 'pending' after actually publishing, or the spotlight draft + HTTP response were lost. Wrap both inserts (open_video_task, _originate_video_post) in a begin_nested savepoint (the repo's established F042 pattern) so a DB error rolls back only the insert. open_video_task returns None (every caller already handles it); _originate_video_post propagates to the render loop's handler. Regression test: an insert FK error returns None with the session left usable. Dormant while the flags were off; armed on the NAS.
* feat(video): Phase G — motion/ package + remotion-renderer sidecar + compose
In-repo Remotion v4 motion/ package (ReleaseAnnouncement composition; calculateMetadata returns 1080x1920 vertical / 1080x1080 square from inputProps.orientation) + a credential-free remotion-renderer sidecar: untar the POSTed motion/ source, bundle (LRU-cached per source sha), selectComposition + renderMedia h264, stream the MP4 bytes back — matching the RemotionRenderer client contract. docker/remotion.Dockerfile on Debian (Chrome apt deps, build-time Chrome pre-warm, ffmpeg bundled in @remotion/renderer). Wired into both compose files (roboco_default only, shm_size 1gb, /health check) + the release publish matrix. Verified via a real local render of both cuts; the Debian docker build is the CEO's to run.
* chore(video): D-hardening — video_post source_task_id + render-loop docstring
Add a source_task_id back-reference to the video_post held-draft marker (traceability from a draft to its authoring task; also makes the render loop's two-key idempotency check wireable later). Fix the render-loop test's stale docstring ('never retried' -> bounded-retry). Both from the Phase D critic's non-blocking follow-ups.
* feat(video): Phase E1 — VideoPostService + heartbeat mutex (approve->post)
CEO-approve->post service: heartbeat-renewed Redis mutex (fail-closed, grace=ttl-2*heartbeat), re-read-in-lock double-post guard, per-platform durable commits (asyncio.shield-ed, settle-before-rollback on lock-loss), all writes inside the lock (captions validated pre-lock, applied in-lock — no stale whole-column clobber), idempotent, per-platform retry-skip. Poster interfaces (X/TikTok, mocked here). Reject + list-held-drafts. Survived 3 adversarial rounds; residual = a crash in the poster->commit window (CEO-gated low-freq, documented).
* fix(video): G-hardening — renderer leaks + Share Tech Mono brand font
Sidecar: give bundle() an explicit outDir tracked + deleted on LRU eviction (was leaking ~19MB remotion-webpack-bundle-* per source); res.on('close') cleanup so an aborted/retried download no longer leaks its remotion-out-* MP4 dir. Fonts: vendor Share Tech Mono (roboco-website brand font) as the display face (self-hosted woff2, 400-weight, headline fontWeight 700->400 to avoid faux-bold) + self-hosted Inter body — no gstatic fetch at render time (lsof-verified). Extras: composition_id whitelist (400) + Multer error middleware (400/413).
* feat(video): Phase E2 — X v2 + TikTok posters, tiktok_credentials, routes
LiveXVideoPoster (X v2 chunked media upload: init/append/finalize/STATUS-poll -> tweet w/ media_ids, OAuth1 signer reused). LiveTikTokPoster (OAuth2 inbox: init -> chunked PUT with asymmetric final chunk -> status-fetch; 401 -> refresh_token grant, rotated token persisted). tiktok_credentials Fernet singleton + migration 062 (single head). Routes: CEO approve/reject + list held drafts + write-only tiktok creds, wiring real posters into VideoPostService. Residual: a lock-loss right after a token-refresh flush can discard the rotated token (same rare CEO-gated class as the documented post->commit window).
* feat(video): Phase F — panel video-post queue + TikTok creds card + flags
video-post-queue.tsx: <video> MP4 preview with 9:16/1:1 cut switch, per-platform editable captions (280/2200 counters, over-limit disables approve), approve/reject, Request-a-video dialog. tiktok-credentials-card.tsx (4 write-only OAuth2 fields). feature-flags-card inlines TikTokCredentialsForm under video_engine_enabled. Mounted in command-center. tsc/eslint clean, 273 panel tests green. NOTE: needs the GET /video/posts/{id}/media route + mp4_paths on VideoPostResponse (folded into H) for the preview source.
* feat(video): Phase H — media route + e2e smoke + NAS arming + docs
GET /video/posts/{id}/media?cut= (CEO-gated FileResponse of the rendered MP4; closes the panel preview gap) + mp4_paths on VideoPostResponse. e2e smoke tests/e2e_smoke/test_video_pipeline.py (full flow, sidecar+X/TikTok mocked; asserts dispatcher skips, render-loop materialize, propose_video team-gate, approve idempotency). NAS arming: docker-compose.yml/.yaml ROBOCO_VIDEO_ENGINE_ENABLED/ON_RELEASE/ON_SPOTLIGHT default-on (.yaml resynced to .yml); registry stays off. CLAUDE.md video-engine section + CHANGELOG. Fixed 2 pre-existing route-test pollution leaks. Full suite 11763 passed.
* fix(video): auth-carrying preview, media route confinement, VideoPost type drift
Three fixes along the video preview path:
1. panel video preview auth: the <video> element was pointed straight at
GET /video/posts/{id}/media, but a native <video src> GET carries none
of axios's X-Agent-ID/X-Agent-Role headers — so in the default
header-trust deployment the request 401s. Fetch the cut via
videoApi.getMediaBlob (axios, responseType: blob) and drive <video>
off a URL.createObjectURL result instead. The object URL is revoked
on cut-change (the previous cut's URL) and on unmount, so neither
cut switches nor row teardown leak blob URLs.
2. backend media route confinement: GET /video/posts/{id}/media now
resolves mp4_path and refuses it with 404 when it falls outside
settings.video_output_dir. Defense-in-depth against any future
writer of mp4_paths serving files from arbitrary disk locations.
3. panel VideoPost type/comment drift: added mp4_paths to the
VideoPost interface (the committed VideoPostResponse already
carries it), and corrected the stale comment on videoMediaUrl
that claimed no route served the rendered bytes — the route has
existed since the media endpoint landed; the comment now describes
why getMediaBlob exists instead of a direct <video src>.
* Persist rendered videos to data in physical storage.
* ++
* docs(video): 0.18.0 CHANGELOG entry + RAG + map reference for video engine
- Move the video engine bullet from [Unreleased] into [0.18.0] and note
the ROBOCO_VIDEO_OUTPUT_DIR bind-mount persistence.
- Add docs/rag/architecture/video-engine.md (mirrors x-engine.md shape:
enable/disable, three triggers, render loop + sidecar, CEO gate, media
route confinement, credentials).
- Reference the video render loop in docs/map/orchestrator.md's engine list.
* chore(video): re-bump to 0.19.0 + sync registry compose defaults
Version was wrongly bumped to 0.18.0; 0.18.0 is an already-released
section. Restore its 2026-07-04 date and move the video-engine CHANGELOG
bullet into a new [0.19.0] - 2026-07-05 section above it. Bump
pyproject.toml, roboco/__init__.py, roboco/config.py (app_version),
panel/package.json, and the motion/README inputProps example to 0.19.0.
docker-compose.registry.yml: add ROBOCO_VIDEO_ENGINE_ENABLED /
_VIDEO_ON_RELEASE / _VIDEO_ON_SPOTLIGHT defaulted false (NAS arms them
true), and comment out the video-renders bind mount with a short note
so the public registry image ships video off by default. Structural
sync with docker-compose.yml maintained.
* fix(video): rate-limit /render + reflow motion/README
CodeQL flagged js/missing-rate-limiting on the renderer /render route.
The sidecar is container-network-only with one trusted caller (the
orchestrator, which renders cuts serially), so this limiter is a
retry-storm ceiling (30/min, well above legit render rate), not the
primary control. Also reflows motion/README.md hard-wrapped prose that
failed the markdown quality gate.
* fix(build): finish pnpm 11 migration + regen verb tables
The panel Docker image build failed on `pnpm install --frozen-lockfile`:
node:22-alpine's corepack resolved to its bundled pnpm 11, but
panel/package.json pinned packageManager to pnpm@10.25.0, and pnpm 11
refuses to run against that pin. The Dockerfiles were already written for
pnpm 11 (comments, CI=true, strictDepBuilds); the package.json pin was the
stale outlier. Finish the migration instead of working around it:
- panel/package.json: packageManager pnpm@10.25.0 -> pnpm@11.10.0; drop the
`pnpm` field (pnpm 11 ignores it — build approval lives in
panel/pnpm-workspace.yaml's allowBuilds). Lockfile unchanged (pnpm 11
accepts it as-is); frozen-lockfile verified.
- remotion-renderer/package.json: pin packageManager pnpm@11.10.0 for
determinism (was relying on corepack's implicit default); engines.node
>=22.13 (pnpm 11 requirement).
- docker/panel.Dockerfile + docker/remotion.Dockerfile: `corepack prepare
pnpm@11.10.0 --activate` so the build uses the pinned version explicitly
instead of trusting corepack's bundled default (which a future
node:22-alpine could change).
- .github/workflows/panel-ci.yml: Node 20 -> 22 (pnpm 11 requires
Node >=22.13; Node 20 fails the engines check).
Also regenerate agents/prompts/_generated/{developer,head_marketing,verbs}.md
— the video engine added propose_video and extended propose_feature_spotlight
(wants_video, video_script) but the verb tables weren't refreshed, failing
the foundation-check quality gate.
* chore(build): approve esbuild build script in remotion pnpm-workspace.yaml
pnpm 11 generated this file with a placeholder ('set this to true or false')
during install; resolve it to true so local dev of the renderer doesn't
re-prompt. esbuild's postinstall only verifies the prebuilt platform binary
(@esbuild/<platform> is installed as an optional dep), so approving it is
safe and silences the ERR_PNPM_IGNORED_BUILDS warning.
* fix(build): copy pnpm-workspace.yaml into panel + remotion images
pnpm 11 hard-errors with [ERR_PNPM_IGNORED_BUILDS] (exit 1) when a
dependency ships a postinstall script that isn't approved in
allowBuilds. Both Dockerfiles copied only package.json + pnpm-lock.yaml,
so the build-approval map in pnpm-workspace.yaml never made it into the
image — the remotion image build died on esbuild@0.28.1's postinstall.
Copy pnpm-workspace.yaml alongside the manifests in both images. In
panel, this also drops the --config.strictDepBuilds=false workaround:
with sharp and unrs-resolver now approved, their postinstalls run and
install the platform-specific binaries (previously skipped, leaving
sharp without its @img/sharp-* binary at runtime).
Verified locally: remotion + panel `pnpm install --frozen-lockfile`
exit 0 with the workspace file present; both exit 1 without it.
* fix(perf): offload conventions + release-readiness blocking I/O off the event loop
The orchestrator runs uvicorn and the orchestration background loops on a
single shared event loop, so any sync I/O anywhere — even inside a background
loop — blocks API responsiveness for its duration. Two call sites were missing
asyncio.to_thread wrappers:
- ConventionsService.get_map/health/restore called the sync _resolve
(`git rev-parse`), _read_committed_standard (file read + yaml parse), and
_derive (filesystem walk via derive_from_scan) inline. Reachable from
GET /api/projects/{id}/conventions and from the agent spawn-prepare path.
- ReleaseManagerEngine._production_assess called gather_snapshot inline —
multiple `subprocess.run` git calls + a filesystem walk, running inside the
release-manager background loop.
Wrap each blocking call in asyncio.to_thread at the async boundary. No
signature changes; helpers stay sync. Verified: targeted tests pass
(196 passed, 36 DB-skipped), ruff + format clean.
These were the only responsiveness gaps surfaced by the concurrency audit —
the rest of the heavy paths (agent spawn via `docker run -d`, video render
loop, git ops via the 16-worker ThreadPoolExecutor, workspace subprocess
calls) already offload correctly. No API/worker container split needed.
* feat(storage): add MinIO config + dep + compose (no-op, default-off)
Chunk 1 of the MinIO video-storage plan (§1, §2, §6). No behavior change:
minio_endpoint defaults to empty = disabled, the existing FileResponse serve
path is untouched (chunk 4 wires the serve path; chunk 2 adds the client).
- pyproject.toml: add `minio` (minio-py) to dependencies; regenerate uv.lock
(resolves minio v7.2.20 + pycryptodome transitive).
- roboco/config.py: add 5 settings fields after video_output_dir
(minio_endpoint/_access_key/_secret_key/_bucket/_region). Plain str Fields
matching the existing ROBOCO_ENCRYPTION_KEY style; no SecretStr, no
presign_ttl_seconds (YAGNI — we don't presign in phase 1).
- docker-compose.yml: add `minio` service (data network only, named
minio-data volume, host ports 19000/19001 for debugging, mc healthcheck)
and a one-shot `minio-init` service mirroring the ollama-init pattern
(mc alias set + mb -p, idempotent via || true). Add ROBOCO_MINIO_* env to
the orchestrator env block (endpoint, access/secret key, bucket, region).
- docker-compose.registry.yml: intentionally omit the minio/minio-init
services and leave ROBOCO_MINIO_* unset (NAS default-on, registry
default-off — the established pattern); comment added to the orchestrator
env block noting the omission.
* docs(storage): 0.19.0 CHANGELOG + RAG + map reference for MinIO chunk 1
Backfills the release-polish docs for MinIO chunk 1 (§10 of the plan):
- docker-compose.yaml synced to docker-compose.yml (the two NAS compose files
must stay byte-identical; .yml was edited in chunk 1, .yaml was stale).
- CHANGELOG [0.19.0]: Added (MinIO scaffolding) + Fixed (event-loop I/O offload).
- docs/rag/architecture/minio-storage.md: RAG doc mirroring video-engine.md.
- docs/map/deployment-tooling.md: one-line storage reference.
* MinIO chunk 2: minio_client module (singleton + unconfigured guard) (#309)
* feat(storage): minio_client module (singleton + unconfigured guard)
Chunk 2 of the MinIO plan (§3). roboco/services/minio_client.py adds:
- get_client(): singleton minio-py Minio from settings; returns None when
minio_endpoint is empty (the disabled path used by the chunk 3/4 guards).
Parses http://... endpoint into host:port + secure flag.
- put_object(bytes, key): no-ops when unconfigured; otherwise PUTs to
settings.minio_bucket with ContentType video/mp4.
- get_object_stream(key): yields object bytes for StreamingResponse; lets
S3Error propagate so the serve route (chunk 4) can fall back to disk.
Sync calls — every call site wraps in asyncio.to_thread (chunks 3/4). One
unit test covers the unconfigured guard + endpoint scheme parsing (mocks,
no real MinIO). Not yet wired into remotion_client._save or the media route.
* MinIO chunk 3: wire write path (remotion_client._save PUT) (#310)
* feat(storage): wire MinIO write path in remotion_client._save
Chunk 3 of the MinIO plan (§3). After the local mp4 write, _save PUTs the bytes
to MinIO under key = Path(mp4_path).name (already {render_key}-{orientation}.mp4),
guarded by minio_client.get_client() (None when minio_endpoint empty) and
wrapped in asyncio.to_thread. Local disk stays the source of truth for the
poster publish path (x_video_client/tiktok_client read mp4_path from disk);
the PUT is additive. _save still returns the local path str — mp4_paths,
marker, and schema unchanged. Disabled (local-only) when MinIO unconfigured.
One test: asserts put_object is called with the basename key when configured
and the local file is still written; existing test stays green via the
unconfigured-default path. Mocks only.
* fix(storage): make MinIO PUT non-fatal in remotion_client._save
A configured-but-down MinIO made put_object raise inside the worker thread,
failing the render and retry-looping a task whose local file was already
written. Local disk is the source of truth and the serve route falls back to
FileResponse on S3Error, so a failed durable-copy PUT must never fail the
render — log and continue; the next render re-attempts the PUT.
Adds test_save_swallows_minio_put_failure (PUT raises -> _save still returns
the local path and the local file is written). Extends the CHANGELOG write-
path bullet with the non-fatal guarantee.
* MinIO chunk 4: serve path (StreamingResponse + FileResponse fallback) (#311)
* feat(storage): serve MinIO via the media route (StreamingResponse + FileResponse fallback)
Chunk 4 of the MinIO plan (§4 — the crux). GET /api/video/posts/{id}/media
derives key = Path(mp4_path).name and, when minio_endpoint is set, returns a
StreamingResponse over minio_client.get_object_stream(key), keeping
_require_ceo so auth stays end-to-end (no presigned URLs). Falls back to
FileResponse on S3Error (old render not in MinIO) or when MinIO is
unconfigured — the panel's axios-blob flow is unchanged (same URL, headers,
body, just chunked). The confinement check is kept as defense-in-depth (the
key is a basename so traversal is impossible, but the check is cheap and
protects the poster path).
Two integration tests: configured serve path streams from a stubbed
get_object_stream (CEO 200, non-CEO 403); unconfigured fallback serves the
local file via FileResponse. Mocks only — no real MinIO.
* fix(storage): eager stat_object probe so the MinIO serve fallback actually fires
The chunk-4 route wrapped StreamingResponse(get_object_stream(key), ...) in a
try/except, but get_object_stream is a lazy generator — its client.get_object
call runs on the first next(), i.e. AFTER the route returned and Starlette
started streaming. An S3Error (NoSuchKey / MinIO down) there is uncatchable;
the try/except caught nothing and the FileResponse fallback never triggered.
Add minio_client.stat_object(key): an eager existence/readiness probe that
runs INSIDE the route's try/except, so a missing object or down MinIO raises
before the StreamingResponse starts and the fallback serves the local file.
stat-then-get is two round trips; a mid-stream failure after a successful stat
is a rare race the CEO can retry (documented ceiling).
Tests: the configured test now stubs stat_object; a new test asserts the
S3Error fallback serves the local file via FileResponse and that
get_object_stream is never called. RAG doc updated to record the eager-probe
correctness detail + the non-fatal PUT.
* docs(rag): mark MinIO deployment note landed (chunk 5) (#312)
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* fix(video): offload minio stat_object off the event loop
stat_object was called inline in the async media route, blocking the
shared event loop for one sync urllib3 round-trip per preview request —
contradicting minio_client's own 'every call site wraps in to_thread'
docstring and this PR's perf-fix theme. Wrap in asyncio.to_thread; the
try/except still catches S3Error (to_thread re-raises) so the
FileResponse fallback is unchanged. Also add the trailing newline to
the minio-storage RAG doc.
* Fix red CI
* Make CI green
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* feat(video): Phase A — VideoEngine origination spine + held-source gates
New default-off engine skeleton: opens a UX/UI authoring task (source=video, assigned to a ux-dev, LOW complexity to clear the dev-needs-subtasks guard) and materializes a held CEO-approval draft (source=video_post). Excludes video_post from all three held-source skip sites; adds the video_draft marker, six config flags, and the feature-flag entries. Origination + gate behavior unit-tested.
* refactor(orchestrator): fold _dispatch_dev_work skip chain into a helper
The per-source if/continue chain grew past xenon's --max-absolute B when the video_post held source joined it. Extract _is_non_dev_dispatch_source (every held-CEO source plus the two Board exploration sources) so the dev loop's skip is one flat call. Behavior-identical.
* feat(video): Phase B — propose_video do-tool (metadata-only, team-gated)
UX/UI dev records a video's composition ref + per-platform captions onto the authoring task's video_draft marker. Team-gated at runtime via _caller_team (Role.DEVELOPER can't tell a ux-dev from a be-dev). Resolves the caller's ACTIVE task via get_active_task_for_agent, not an oldest-first scan that would clobber a second open video task. Metadata only, no render. Wired through do_server + route + schema; added to _DEV_DO.
* feat(video): Phase D — render loop + RemotionRenderer client
Orchestrator-async _video_render_loop renders a completed authoring task's merged composition to MP4 (vertical + square) via the remotion-renderer sidecar and materializes the held video_post draft. RemotionRenderer tars the read-clone's motion/ source, POSTs it, and saves the returned MP4 bytes to a TASK-scoped local path (no shared volume; a composition is reused across videos so a composition-scoped path would clobber an earlier draft). Render failures bounded-retry (read-clone catch-up window, transient sidecar) up to a cap, then terminal-fail. Client tested vs a mock transport; loop vs a mock renderer + real DB.
* feat(video): Phase C — release / spotlight / on-demand video triggers
Three entry points open a UX/UI video-authoring task via VideoEngine.open_video_task: (1) a published release drafts a companion video — best-effort in ReleaseProposalService.approve, never fails the publish; script from the CHANGELOG via the local model with a template fallback. (2) propose_feature_spotlight gains optional wants_video/video_script — best-effort, gated on video_on_spotlight, default-off leaves the spotlight flow byte-for-byte unchanged. (3) POST /video/request (CEO-only) for an on-demand brief, with clean disabled/not_opened responses. All gated on video_engine_enabled.
* fix(video): savepoint-isolate video-task inserts (F042 poisoned session)
The best-effort try/except around open_video_task (release-publish + spotlight hooks) swallowed the Python exception, but a DBAPI error at the insert flush left the shared session must-rollback — so the caller's next commit (release finalize / request boundary) threw PendingRollbackError: the release stuck 'pending' after actually publishing, or the spotlight draft + HTTP response were lost. Wrap both inserts (open_video_task, _originate_video_post) in a begin_nested savepoint (the repo's established F042 pattern) so a DB error rolls back only the insert. open_video_task returns None (every caller already handles it); _originate_video_post propagates to the render loop's handler. Regression test: an insert FK error returns None with the session left usable. Dormant while the flags were off; armed on the NAS.
* feat(video): Phase G — motion/ package + remotion-renderer sidecar + compose
In-repo Remotion v4 motion/ package (ReleaseAnnouncement composition; calculateMetadata returns 1080x1920 vertical / 1080x1080 square from inputProps.orientation) + a credential-free remotion-renderer sidecar: untar the POSTed motion/ source, bundle (LRU-cached per source sha), selectComposition + renderMedia h264, stream the MP4 bytes back — matching the RemotionRenderer client contract. docker/remotion.Dockerfile on Debian (Chrome apt deps, build-time Chrome pre-warm, ffmpeg bundled in @remotion/renderer). Wired into both compose files (roboco_default only, shm_size 1gb, /health check) + the release publish matrix. Verified via a real local render of both cuts; the Debian docker build is the CEO's to run.
* chore(video): D-hardening — video_post source_task_id + render-loop docstring
Add a source_task_id back-reference to the video_post held-draft marker (traceability from a draft to its authoring task; also makes the render loop's two-key idempotency check wireable later). Fix the render-loop test's stale docstring ('never retried' -> bounded-retry). Both from the Phase D critic's non-blocking follow-ups.
* feat(video): Phase E1 — VideoPostService + heartbeat mutex (approve->post)
CEO-approve->post service: heartbeat-renewed Redis mutex (fail-closed, grace=ttl-2*heartbeat), re-read-in-lock double-post guard, per-platform durable commits (asyncio.shield-ed, settle-before-rollback on lock-loss), all writes inside the lock (captions validated pre-lock, applied in-lock — no stale whole-column clobber), idempotent, per-platform retry-skip. Poster interfaces (X/TikTok, mocked here). Reject + list-held-drafts. Survived 3 adversarial rounds; residual = a crash in the poster->commit window (CEO-gated low-freq, documented).
* fix(video): G-hardening — renderer leaks + Share Tech Mono brand font
Sidecar: give bundle() an explicit outDir tracked + deleted on LRU eviction (was leaking ~19MB remotion-webpack-bundle-* per source); res.on('close') cleanup so an aborted/retried download no longer leaks its remotion-out-* MP4 dir. Fonts: vendor Share Tech Mono (roboco-website brand font) as the display face (self-hosted woff2, 400-weight, headline fontWeight 700->400 to avoid faux-bold) + self-hosted Inter body — no gstatic fetch at render time (lsof-verified). Extras: composition_id whitelist (400) + Multer error middleware (400/413).
* feat(video): Phase E2 — X v2 + TikTok posters, tiktok_credentials, routes
LiveXVideoPoster (X v2 chunked media upload: init/append/finalize/STATUS-poll -> tweet w/ media_ids, OAuth1 signer reused). LiveTikTokPoster (OAuth2 inbox: init -> chunked PUT with asymmetric final chunk -> status-fetch; 401 -> refresh_token grant, rotated token persisted). tiktok_credentials Fernet singleton + migration 062 (single head). Routes: CEO approve/reject + list held drafts + write-only tiktok creds, wiring real posters into VideoPostService. Residual: a lock-loss right after a token-refresh flush can discard the rotated token (same rare CEO-gated class as the documented post->commit window).
* feat(video): Phase F — panel video-post queue + TikTok creds card + flags
video-post-queue.tsx: <video> MP4 preview with 9:16/1:1 cut switch, per-platform editable captions (280/2200 counters, over-limit disables approve), approve/reject, Request-a-video dialog. tiktok-credentials-card.tsx (4 write-only OAuth2 fields). feature-flags-card inlines TikTokCredentialsForm under video_engine_enabled. Mounted in command-center. tsc/eslint clean, 273 panel tests green. NOTE: needs the GET /video/posts/{id}/media route + mp4_paths on VideoPostResponse (folded into H) for the preview source.
* feat(video): Phase H — media route + e2e smoke + NAS arming + docs
GET /video/posts/{id}/media?cut= (CEO-gated FileResponse of the rendered MP4; closes the panel preview gap) + mp4_paths on VideoPostResponse. e2e smoke tests/e2e_smoke/test_video_pipeline.py (full flow, sidecar+X/TikTok mocked; asserts dispatcher skips, render-loop materialize, propose_video team-gate, approve idempotency). NAS arming: docker-compose.yml/.yaml ROBOCO_VIDEO_ENGINE_ENABLED/ON_RELEASE/ON_SPOTLIGHT default-on (.yaml resynced to .yml); registry stays off. CLAUDE.md video-engine section + CHANGELOG. Fixed 2 pre-existing route-test pollution leaks. Full suite 11763 passed.
* fix(video): auth-carrying preview, media route confinement, VideoPost type drift
Three fixes along the video preview path:
1. panel video preview auth: the <video> element was pointed straight at
GET /video/posts/{id}/media, but a native <video src> GET carries none
of axios's X-Agent-ID/X-Agent-Role headers — so in the default
header-trust deployment the request 401s. Fetch the cut via
videoApi.getMediaBlob (axios, responseType: blob) and drive <video>
off a URL.createObjectURL result instead. The object URL is revoked
on cut-change (the previous cut's URL) and on unmount, so neither
cut switches nor row teardown leak blob URLs.
2. backend media route confinement: GET /video/posts/{id}/media now
resolves mp4_path and refuses it with 404 when it falls outside
settings.video_output_dir. Defense-in-depth against any future
writer of mp4_paths serving files from arbitrary disk locations.
3. panel VideoPost type/comment drift: added mp4_paths to the
VideoPost interface (the committed VideoPostResponse already
carries it), and corrected the stale comment on videoMediaUrl
that claimed no route served the rendered bytes — the route has
existed since the media endpoint landed; the comment now describes
why getMediaBlob exists instead of a direct <video src>.
* Persist rendered videos to data in physical storage.
* ++
* docs(video): 0.18.0 CHANGELOG entry + RAG + map reference for video engine
- Move the video engine bullet from [Unreleased] into [0.18.0] and note
the ROBOCO_VIDEO_OUTPUT_DIR bind-mount persistence.
- Add docs/rag/architecture/video-engine.md (mirrors x-engine.md shape:
enable/disable, three triggers, render loop + sidecar, CEO gate, media
route confinement, credentials).
- Reference the video render loop in docs/map/orchestrator.md's engine list.
* chore(video): re-bump to 0.19.0 + sync registry compose defaults
Version was wrongly bumped to 0.18.0; 0.18.0 is an already-released
section. Restore its 2026-07-04 date and move the video-engine CHANGELOG
bullet into a new [0.19.0] - 2026-07-05 section above it. Bump
pyproject.toml, roboco/__init__.py, roboco/config.py (app_version),
panel/package.json, and the motion/README inputProps example to 0.19.0.
docker-compose.registry.yml: add ROBOCO_VIDEO_ENGINE_ENABLED /
_VIDEO_ON_RELEASE / _VIDEO_ON_SPOTLIGHT defaulted false (NAS arms them
true), and comment out the video-renders bind mount with a short note
so the public registry image ships video off by default. Structural
sync with docker-compose.yml maintained.
* fix(video): rate-limit /render + reflow motion/README
CodeQL flagged js/missing-rate-limiting on the renderer /render route.
The sidecar is container-network-only with one trusted caller (the
orchestrator, which renders cuts serially), so this limiter is a
retry-storm ceiling (30/min, well above legit render rate), not the
primary control. Also reflows motion/README.md hard-wrapped prose that
failed the markdown quality gate.
* fix(build): finish pnpm 11 migration + regen verb tables
The panel Docker image build failed on `pnpm install --frozen-lockfile`:
node:22-alpine's corepack resolved to its bundled pnpm 11, but
panel/package.json pinned packageManager to pnpm@10.25.0, and pnpm 11
refuses to run against that pin. The Dockerfiles were already written for
pnpm 11 (comments, CI=true, strictDepBuilds); the package.json pin was the
stale outlier. Finish the migration instead of working around it:
- panel/package.json: packageManager pnpm@10.25.0 -> pnpm@11.10.0; drop the
`pnpm` field (pnpm 11 ignores it — build approval lives in
panel/pnpm-workspace.yaml's allowBuilds). Lockfile unchanged (pnpm 11
accepts it as-is); frozen-lockfile verified.
- remotion-renderer/package.json: pin packageManager pnpm@11.10.0 for
determinism (was relying on corepack's implicit default); engines.node
>=22.13 (pnpm 11 requirement).
- docker/panel.Dockerfile + docker/remotion.Dockerfile: `corepack prepare
pnpm@11.10.0 --activate` so the build uses the pinned version explicitly
instead of trusting corepack's bundled default (which a future
node:22-alpine could change).
- .github/workflows/panel-ci.yml: Node 20 -> 22 (pnpm 11 requires
Node >=22.13; Node 20 fails the engines check).
Also regenerate agents/prompts/_generated/{developer,head_marketing,verbs}.md
— the video engine added propose_video and extended propose_feature_spotlight
(wants_video, video_script) but the verb tables weren't refreshed, failing
the foundation-check quality gate.
* chore(build): approve esbuild build script in remotion pnpm-workspace.yaml
pnpm 11 generated this file with a placeholder ('set this to true or false')
during install; resolve it to true so local dev of the renderer doesn't
re-prompt. esbuild's postinstall only verifies the prebuilt platform binary
(@esbuild/<platform> is installed as an optional dep), so approving it is
safe and silences the ERR_PNPM_IGNORED_BUILDS warning.
* fix(build): copy pnpm-workspace.yaml into panel + remotion images
pnpm 11 hard-errors with [ERR_PNPM_IGNORED_BUILDS] (exit 1) when a
dependency ships a postinstall script that isn't approved in
allowBuilds. Both Dockerfiles copied only package.json + pnpm-lock.yaml,
so the build-approval map in pnpm-workspace.yaml never made it into the
image — the remotion image build died on esbuild@0.28.1's postinstall.
Copy pnpm-workspace.yaml alongside the manifests in both images. In
panel, this also drops the --config.strictDepBuilds=false workaround:
with sharp and unrs-resolver now approved, their postinstalls run and
install the platform-specific binaries (previously skipped, leaving
sharp without its @img/sharp-* binary at runtime).
Verified locally: remotion + panel `pnpm install --frozen-lockfile`
exit 0 with the workspace file present; both exit 1 without it.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
Drives the Head of Marketing calling propose_feature_spotlight through the real
do_server -> /api/v1/do route -> ContentActions -> XEngine, then asserts a held
x_feature draft is created (confirmed_by_human=False) and the exploration task
completes. Red-then-green verified: reverting the do_server registration
(3f90d382) makes this fail with the exact live symptom. This is the e2e coverage
the wiring gap should have had (standing rule: gaps become e2e smoke).
The v0.18.0 feature-spotlight verb was wired at the role-config, content-actions,
and route layers but never added to do_server's _TOOLS registry, so
_register_tools() (which registers only granted ∩ _TOOLS) silently dropped it —
a spawned Head of Marketing could NOT call propose_feature_spotlight, making the
feature non-functional as shipped. Added the do_server wrapper + _TOOLS entry
(mirroring propose_roadmap) and a regression test asserting every role-granted
do-tool is registered, so this class of gap can't recur.
Adds fable-mode, X feature-spotlight, and design-bar symbols across the map
slices (+ _complete_map mirror); backfills migrations 060 (drop_messaging) and
061 (x_feature_spotlight), both previously undocumented; corrects the ORM table
count 37->38; adds the _briefing_for include_company_goals row.
The 0.18.0 cut bumped config.py, __init__.py and panel/package.json but a
truncated grep hid pyproject.toml's version=0.17.0, so 88db8ca6 omitted it.
Corrected to 0.18.0. This chore(release) commit re-stamps pyproject.toml into
the release-readiness canonical-bump derivation; it shadows 88db8ca6, narrowing
the derived set until the next cut re-completes it (release engine is
default-off; _version_ref_gaps self-corrects by flagging the rest next time).
board_triage's idle branch built its briefing without full=True, so
company_goals (brand_voice/north_star, migration 061) never reached the Product
Owner's roadmap-exploration spawn or the Head of Marketing's feature-spotlight
spawn — both always hit the idle branch, yet both spawn prompts claim the
charter is 'already in your briefing'. Added a scoped include_company_goals flag
to _briefing_for (via a _resolve_company_goals helper; xenon B preserved) that
fetches only the cheap charter singleton without full's other heavy sections;
board_triage's idle branch opts in. Strategic branch + auditor untouched. 6 new tests.
- layout.tsx/globals.css: font-sans/font-mono pointed at undefined --font-geist-*
vars (Geist-starter leftover); now resolve to the actually-loaded Inter via
--font-inter. No visible change.
- remove orphaned company-goals-card.tsx (unmounted; live charter UI is goals-tab.tsx)
- remove dead streamApi transcription REST client (zero consumers) + its barrel
export; AgentStreamViewer (a different, live websocket-based component) untouched.
A's vendored fable.md and C's design-bar prose went in hard-wrapped, which
fails make quality's reflow-check (the no-hard-wrap gate). Reflowed to one line
per paragraph via scripts/reflow_md.py. reflow-check now clean.
The Head of Marketing now markets features, not just releases: a default-off
x_feature_spotlight loop periodically spawns the HoM to investigate what shipped
(CHANGELOG, feature flags, docs/map, KB) and draft ONE held marketing post via
propose_feature_spotlight, reviewed in the X post queue.
- New x_feature source (distinct from x_post, fixing panel mislabeling) + a
panel Feature-spotlight branch.
- brand_voice column on company_goals (migration 061, single head) as the
CEO-editable voice source, surfaced in Settings and injected into the HoM
briefing; a VOICE GUIDE baseline in head-marketing.md.
- propose_feature_spotlight verb (HoM-only), mirroring propose_roadmap.
Gated by x_feature_spotlight_enabled (default off; flag-off dormancy proven).
Also fixed two real bugs found mid-build: company_goals API schemas dropped
brand_voice on GET/PUT; the live charter UI is goals-tab.tsx, not the unmounted
company-goals-card.tsx. Full suite green (2935); migration single-head verified.
`<pipe JSON> | python3 - <<'PY'` makes both `python3 -` and the heredoc claim
stdin; the heredoc wins, so the piped JSON is silently discarded — each hook
read empty input and never triggered. Fixed to `python3 -c "$(cat <<'PY')"`
(cat consumes the heredoc, python3's stdin stays free for the pipe), matching
the already-correct fable-stop-gate-hook.sh.
Impact, all verified before/after:
- user-prompt-hook.sh: the prompt-injection guard ALLOWED injection strings
(exit 0); now correctly DENIES (exit 2). Security hole closed.
- post-tool-budget-hook.sh: every tool call hashed to {tool:unknown} — loop
detection was blind; now hashes the real tool + args.
- usage-report-hook.sh: transcript path resolved empty so its curl sync never
fired; now fires correctly.
2-line change per file; no logic/threshold/message/contract change. bash-guard
78/78 + fable-hooks 15/15 green; repo-wide grep confirms no remaining instances.
Fleet behaves more like Fable 5 on existing model tiers, behind
ROBOCO_FABLE_MODE_ENABLED (config default off; armed :-true on the NAS compose,
absent from the registry compose).
- Doctrine: vendored agents/prompts/doctrine/fable.md composed into every
agent's system prompt via fable_doctrine_layer() after base.md.
- Hooks (Claude Code): 4 non-overlapping hooks (stop-gate/bash-discipline/
honesty-nudge/precompact) appended per-agent via _fable_hook_groups(). The
make-quality + lint-suppression duplicates are deliberately NOT added (already
gate-enforced); session-start skipped.
- Hooks (grok): conservative V1 — only the non-denying honesty-nudge, since a
grok hook deny cancels the whole run.
- Flag on the feature-flags card; hook scripts shipped into the agent image.
Flag-off spawn path proven byte-identical (worktree diff, sha256 match); full
suite green (2074 unit + e2e-smoke + hook harness), mypy/xenon/ruff clean.
Fixed a real stdin bug in the vendored stop-gate hook (heredoc + pipe both
claimed stdin). Distilled from rennf93/opus-fable-playbook (MIT).
Frontend + UX/UI agents now carry a distilled design-taste bar in their team
prompts: the three dials (variance/motion/density) with dense-product-UI
defaults, plus typography/hierarchy, spacing/layout, motion, and AI-tells-to-
avoid rules, scoped to respect a project's existing design system rather than
override it. compose_prompt is unchanged (pure team/role prompt content).
developer.md gets a pointer whose heading deliberately does NOT start
'## Design bar', preserving the backend-dev negative test. New
tests/unit/agents/test_design_bar_layer.py (5 tests green).
Distilled from Leonxlnx/taste-skill (MIT).
Renn F