roboco/docs/map/video-engine.md

Purpose

The RoboCo video engine: a default-off subsystem that authors bespoke short marketing videos (release announcements, feature spotlights, on-demand CEO briefs) and distributes them to X and TikTok — nothing renders or posts without the flags on, and nothing posts without an explicit per-clip CEO approval. It mirrors the XEngine / ReleaseManagerEngine held-artifact shape, but splits across the real delivery lifecycle: a normal ASSIGNED UX/UI authoring task ships the composition through the standard commit/PR/QA/doc/review gate, then an orchestrator render loop renders the merged motion/ source via the credential-free video-renderer sidecar and materializes a held video_post draft for the CEO. Two task kinds (authoring + held post), one render pass between them.

Files

Path	Role	approx LOC
roboco/services/video_engine.py	VideoEngine — opens the ASSIGNED authoring task (open_video_task, balanced across ux-dev-1/ux-dev-2) and originates the held video_post draft once the render succeeds (_originate_video_post); release/spotlight/on-demand trigger wiring.	324
roboco/services/video_post_service.py	VideoPostService — CEO approve/reject over the held post; the ONLY caller of the X-v2 and TikTok posters; runs the critical section under a heartbeat-renewed Redis mutex, commits each platform's posted-id durably before the next, idempotent on already-COMPLETED AND on already-CANCELLED (rejected).	629
roboco/services/video_renderer_client.py	VideoRenderer — tars the merged motion/ dir, POSTs the tarball to the sidecar (ROBOCO_VIDEO_RENDERER_BASE_URL), saves the returned MP4s to video_output_dir and PUTs each to MinIO (_save). NullVideoRenderer raises on unconfigured so the render loop fails loud rather than silently no-op'ing a real trigger; get_video_renderer() factory.	188
roboco/services/minio_client.py	Singleton Minio (minio-py) with an unconfigured guard (get_client() returns None when minio_endpoint empty); put_object / get_object_stream / stat_object, sync, call sites wrapped in asyncio.to_thread.	129
roboco/services/tiktok_client.py	TikTokPoster — TikTok inbox-upload poster (v2 media, OAuth2 refresh). Fernet-encrypted singleton tiktok_credentials row (migration 062); agents never hold creds or egress.	326
roboco/services/x_video_client.py	XVideoPoster — X v2 media upload + tweet poster. NullXVideoPoster makes the unconfigured leg a graceful no-op.	266
roboco/runtime/heartbeat_mutex.py	HeartbeatMutex — Redis mutex with heartbeat-renewed TTL, shared with ReleaseProposalService's release-execute lock shape; backs VideoPostService.approve's long video-upload critical section.	—
roboco/mcp/do_server.py propose_video	Do-tool the UX/UI dev calls exactly once per authoring task to stamp the video_draft marker (composition id + per-platform captions + input props); metadata-only, does not render.	—
roboco/services/gateway/content_actions.py propose_video	Server-side action: team-gated (_caller_team rejects be-dev/fe-dev), resolves the caller's open video task, markers.set_video_draft with the metadata.	—
roboco/services/gateway/content_actions.py request_render	Do-verb (developer/QA): renders the caller's ACTUAL composition to keyframe PNGs via the sidecar's frames mode and stamps the render_preview marker — dev renders their own tree (worktree-aware, head_sha/dirty stamped), QA a read-only branch export (WorkspaceService.export_branch_motion). Frames land at the container-shared {workspaces_root}/{project}/.previews/{task8}/{orientation}/.	—
roboco/foundation/policy/tracing.py RENDER_VERIFIED	i_am_done requirement on source=video tasks: no stamped render_preview → tracing gap naming render_preview (hint: call request_render, Read every frame). Mirrored in the possibilities-matrix fast path.	—
roboco/api/routes/video.py GET /video/preview-frames/{task_id} / .../{orientation}/{filename}	CEO-gated routes serving the dev's already-rendered request_render preview frames — a source=video authoring task reaches awaiting_ceo_approval with no MP4 yet (rendering only happens post-completion), so the CEO otherwise has nothing to review. Frame listing is parsed from the self-describing .previews/{task8}/{orientation}/frame-<idx>-of-<n>-at-<t>s.png filenames (_FRAME_NAME_RE) rather than the render_preview marker (which only holds the LAST call's single orientation); the per-frame route streams a PNG behind the same _resolve_preview_path confinement guard the composition-HTML proxy already used.	—
alembic/versions/062_tiktok_credentials.py	Migration 062 — the tiktok_credentials singleton row (Fernet-encrypted OAuth2 secrets, all-or-nothing set/clear, mirroring the git-token / x_credentials pattern).	44
video-renderer/	The sidecar: server.js (HTTP+tarball boundary), render.js (@hyperframes/producer createRenderJob + executeRenderJob, system ffmpeg, headless Chromium). Credential-free and git-free — reads only what's POSTed. pnpm-managed (pnpm-lock.yaml, no npm package-lock.json); @hyperframes/producer pinned exact at 0.7.36 (0.7.60 fails every render).	—
docker/video-renderer.Dockerfile	Sidecar image (roboco-video-renderer): Node + Chromium + system ffmpeg; installs @hyperframes/producer. No RoboCo source, no creds.	—
motion/README.md ## Design bar / ## Visual design bar (demo/kit register)	Authoring craft an assigned UX/UI dev consults before building a composition: color/type/motion/layout dials for the text-card register, plus spacing, beat pacing (animation-delay, never data-start for beats), pk-chip/pk-pill semantic-variant discipline, camera+cursor+rhythm, and anti-generic tells for the kit/ demo register.	149 (file total)
motion/skills/references/{house-style,video-composition,beat-direction,motion-principles}.md	Four upstream HyperFrames craft references (palette/lazy-defaults, video-medium scale/density, per-beat rhythm planning, ease/speed/direction variance) vendored verbatim at pinned commit 9d148d28 (Apache-2.0, header in each file) — back every rule in motion/README.md's design-bar sections; re-vendor when bumping @hyperframes/producer.	462
motion/skills/hyperframes-catalog-index.md	RoboCo-authored index of the public HyperFrames catalog (109 blocks + 24 components, 133 entries) with per-category kit-mapping triage (maps onto an existing pk-* piece / a choreography engine / needs a new kit piece); read on demand when planning a beat, not injected into any agent prompt.	186
motion/skills/{hyperframes-core,hyperframes-creative,hyperframes-keyframes}.md	The vendor's own official HyperFrames agent skills (composition contract, beat planning, seek-safe keyframes across runtimes), vendored verbatim at a pinned upstream commit (Apache-2.0, header + re-vendor note in each file); motion/README.md points authoring devs at them before a new register. Primary seek-safe primitive is GSAP tweens on window.__timelines — this kit's CSS-animation register is a house pattern, and the clip-window rule is its empirically-derived companion.	382
motion/kit/kit.js choreographCursor / choreographCamera	Choreography engines: choreographCursor reads data-waypoints="t x y [click]; ..." off a .pk-cursor and generates a multi-leg eased path with fade in/out, an idle-hand sway between legs, and click rings + glyph press dips at flagged waypoints (click lands ~0.2s before the thing it triggers) — replaces the old single-glide --pk-cursor-x0/y0/x1/y1 cameo. choreographCamera reads data-shots="t x y scale; ..." off a .pk-camera wrapper and eases push-ins/pull-backs, settling at identity by the end. motion/kit/README.md documents both.	156 (diff)
roboco/runtime/orchestrator.py _is_video_authoring_spawn	Fail-closed spawn-time probe (role==developer, team==ux_ui, task.source==video) that registers the playwright MCP for the composition author — the one non-QA case, so the dev can preview the authored HTML in a real browser between renders. Gating-only: agent-ux's image already bakes the browser + wrapper entrypoint.	—

Data Flow

DETECT → AUTHOR: a release publish (ROBOCO_VIDEO_ON_RELEASE), a CEO-approved feature-spotlight draft that requests one (ROBOCO_VIDEO_ON_SPOTLIGHT), or a CEO on-demand POST /api/video/request calls VideoEngine.open_video_task, which creates a normal ASSIGNED UX/UI authoring task (source=video, confirmed_by_human=True, balanced across the two ux-devs) — NOT held, NOT in any dispatcher's skip bucket. The assigned dev authors motion/compositions/<id>/{vertical,square}.html (HyperFrames render params on <html>), reads the vendored motion/skills/hyperframes-{core,creative,keyframes}.md skills and consults kit.js's choreographCursor/choreographCamera engines for camera+cursor craft (a locked-off camera or a popping/freezing cursor is automatic revision; clip windows are for structural layers ONLY — beats ride base-hidden delayed CSS animations, since the renderer's clip scheduler drops any beat driven off its own clip window), previews the live HTML via the playwright MCP registered for this spawn (_is_video_authoring_spawn), calls the propose_video do-tool exactly once (server-side content_actions.propose_video is team-gated and stamps video_draft), then verifies the ARTIFACT: request_render renders the dev's actual working tree to keyframe PNGs the dev must Read (every scene fully visible and legible — the gate that catches an authored duration shorter than its scene list), iterating fix → re-render until the frames prove the brief; i_am_done refuses without the stamped render_preview marker (Requirement.RENDER_VERIFIED). Then commit + open_pr through the normal PR-review gate. The authoring task rides the standard QA/doc/review lifecycle to completed, with QA's claim_review evidence carrying a video_context block (the dev's preview + an instruction to request_render the branch state fresh).

RENDER: once the authoring task is completed, the orchestrator's _video_render_loop (bounded retry, _MAX_VIDEO_RENDER_ATTEMPTS) resolves the project's read-clone at the merged HEAD, tars the motion/ dir, and POSTs it to the credential-free video-renderer sidecar (ROBOCO_VIDEO_RENDERER_BASE_URL). The sidecar untars, runs @hyperframes/producer's createRenderJob + executeRenderJob per orientation (headless Chrome + system ffmpeg, ROBOCO_VIDEO_RENDER_TIMEOUT_SECONDS per render), and streams both 9:16 and 1:1 MP4s back. VideoRenderer saves them to ROBOCO_VIDEO_OUTPUT_DIR (_save also PUTs each to MinIO when minio_endpoint is set, non-fatal). On success VideoEngine._originate_video_post materializes a held video_post draft (source=video_post, confirmed_by_human=False, Secretary-owned, skipped by every dispatcher) carrying mp4_paths ({vertical, square} absolute paths) + the per-platform captions.

CEO ACT: GET /api/video/posts lists held drafts (including mp4_paths); GET /api/video/posts/{id}/media?cut=vertical|square streams the MP4 bytes for the preview player (CEO-gated, falls back to FileResponse on S3Error/unconfigured MinIO). The CEO edits captions and approves/rejects in the panel's video-post-queue.tsx. POST /api/video/posts/{id}/approve is the ONLY caller of XVideoPoster / TikTokPoster: it acquires HeartbeatMutex, re-reads the committed task state inside the lock, commits COMPLETED before releasing (so a concurrent approve can't double-post), commits each platform's posted-id durably before attempting the next (a partial failure never re-posts an already-succeeded platform on retry), and is idempotent (an already-COMPLETED draft returns the stored ids without calling a poster). A CANCELLED draft (already rejected) is refused both pre-lock and re-checked under lock, returning already_rejected — closes a hole where a stale approve (e.g. a queued Telegram button targeting the draft by id regardless of its current status) could post a draft the CEO had already rejected. POST /api/video/posts/{id}/reject cancels the draft with a reason — and, for a non-empty reason, VideoEngine.reauthor_from_rejection opens a fresh authoring task (same occasion, brief = the CEO's verbatim feedback + revise-in-place pointer at the existing composition) so the rejection feedback re-enters the delivery flow instead of dying on the cancelled draft; best-effort, never fails the reject.

Config Flags

• ROBOCO_VIDEO_ENGINE_ENABLED — master switch; off = no video-authoring task is ever opened and no render/post happens. Panel-toggleable.
• ROBOCO_VIDEO_ON_RELEASE / ROBOCO_VIDEO_ON_SPOTLIGHT — sub-switches for the two automatic triggers, independent of the master switch and of the CEO's on-demand POST /video/request.
• ROBOCO_VIDEO_RENDER_INTERVAL_SECONDS / ROBOCO_VIDEO_RENDER_TIMEOUT_SECONDS / ROBOCO_VIDEO_REQUEST_TIMEOUT_SECONDS / ROBOCO_VIDEO_OUTPUT_DIR — render loop cadence, per-render deadline, sidecar HTTP deadline, MP4 output dir (bind-mounted in all three compose files so renders survive container recreation).
• ROBOCO_VIDEO_RENDERER_BASE_URL — the sidecar endpoint (default http://roboco-video-renderer:3001).
• ROBOCO_MINIO_* — MinIO object storage (default-off; video_renderer_client._save PUTs each render after the local write; serve route streams via StreamingResponse with FileResponse fallback).

Changes Since Baseline

• 2026-07-20 (#608, a5d8c6bd, "CEO can preview a video authoring task's frames before approving"): two CEO-gated routes serve the request_render preview frames (see Files above) so the CEO has something concrete to review at awaiting_ceo_approval before any MP4 exists. The task-detail Overview gains a Video preview card — a 9:16/1:1 toggle + prev/next/scrubber frame stepper with composition id, duration, and a dirty badge — shown for a video task with preview frames or awaiting CEO approval.
• 2026-07-17 (PR #543, 3e801697): Two renderer root causes fixed — @hyperframes/producer was floating (^0.7.36, no lockfile), so image builds silently picked up 0.7.60, which fails EVERY render ("Cannot access 'rt' before initialization"); pinned exact (0.7.36, no caret) and committed a lockfile (regenerated as pnpm-lock.yaml by the immediate follow-up a12fefcb, not the npm package-lock.json this PR first wrote — this package is pnpm-managed). Second: the producer's per-clip visibility scheduler runs on a clock that lags ~50% behind the encoded timeline on a long cut, so tail scenes (past roughly the halfway mark) were silently missing from the MP4 regardless of authoring — fixed by treating class="clip" + data-start/data-duration as a structural-layer-only primitive and driving every beat with base-hidden styles + a delayed CSS animation instead (documented in motion/README.md's "Clip windows are for structural layers only" rule). Also added the two choreography engines to motion/kit/kit.js (choreographCursor / choreographCamera, see Files above) plus a "Cinematography & rhythm" section in motion/README.md and a craft-bar block in the dev video spawn prompt (roboco/runtime/orchestrator.py) so a locked-off camera or a popping/freezing cursor reads as an automatic revision.
• 2026-07-17 (PR #544, fd621f0d): The three craft capabilities wired one hop closer to the hands doing video work — vendored the vendor's own official HyperFrames agent skills (hyperframes-core/-creative/-keyframes.md, see Files above; supersedes the external-pointer-only version briefly added by the intervening 1416bd1d); registered the playwright MCP for a ux-dev spawned onto a source=video task (_is_video_authoring_spawn, fail-closed role/team/task-source probe — gating-only, agent-ux's image already bakes the browser); and added a video-mode override to the ux_ui team prompt's design bar ("video-authoring tasks are FILMS, not UI — these dials do not apply") so a video task no longer reads its own "dense product UI → motion 2-3" dial as license to ship a static slideshow.
• 2026-07-17 (Wave 6, PR #550): Authoring craft, not engine code — motion/README.md gained ## Visual design bar (demo/kit register) (spacing/hierarchy, beat pacing, pk-chip/pk-pill semantic discipline, camera+cursor+rhythm, anti-generic tells for the kit/ register), four upstream HyperFrames craft references vendored verbatim under motion/skills/references/ (fixing hyperframes-creative.md's previously-dead references/ pointers), and a new motion/skills/hyperframes-catalog-index.md (133-entry HyperFrames catalog vocabulary index, read-on-demand). No service/verb/schema change; the render/post pipeline documented above is untouched.
• 2026-07-18 (PR #570, "project-branded drafts"): the release-video script/prompt/brief builders in roboco/services/video_engine.py (_fallback_release_script, _release_video_prompt, _release_video_brief, _draft_release_script) all gained a required product_name param — draft_release_post_video (the ReleaseProposalService.approve publish-success hook, mirroring XEngine.draft_release_post) now resolves it via CompanyGoalsService.resolve_product_name(project) (the release's own project name → charter company_goals.company_name → "RoboCo" literal) instead of hardcoding "RoboCo" into the script/brief text — see docs/map/product-strategy-research-pitch.md for the shared resolver. GET /api/video/posts responses also gained project_slug/project_name (api/schemas/video.py, via the same task_project_fields helper x.py uses) so the panel's video-post-queue.tsx can render a ProjectBadge alongside the source-kind badge.

Health

Default-off, CEO-gated at two independent points (the flags, then per-clip approval). The held-draft shape mirrors the XEngine / ReleaseManagerEngine pattern, so the dispatchers never see it. The render pass is bounded retry with _MAX_VIDEO_RENDER_ATTEMPTS and a per-render deadline; NullVideoRenderer raises on unconfigured so a misflagged trigger fails loud rather than silently no-op'ing. The approve critical section is heartbeat-mutex protected so a double-click can't double-post and a partial platform failure is recoverable, and (Wave 5, PR #551) approve now also refuses a CANCELLED draft outright rather than posting it. TikTok's OAuth2 secrets live Fernet-encrypted in a singleton row (migration 062); agents never hold creds or egress — VideoPostService.approve is the only caller of the posters.

Related

• docs/rag/architecture/video-engine.md — the user-facing architecture doc
• docs/rag/architecture/minio-storage.md — the decoupled-durable render storage
• docs/map/release-manager.md — the sibling held-artifact engine whose lock shape VideoPostService.approve mirrors
• docs/map/engines-heal-ciwatch-depupdate.md — the other default-off originate-and-stop engines