mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
MinIO video storage (chunk 1: config+deps+compose) + event-loop perf fix (#308)
* feat(video): Phase A — VideoEngine origination spine + held-source gates
New default-off engine skeleton: opens a UX/UI authoring task (source=video, assigned to a ux-dev, LOW complexity to clear the dev-needs-subtasks guard) and materializes a held CEO-approval draft (source=video_post). Excludes video_post from all three held-source skip sites; adds the video_draft marker, six config flags, and the feature-flag entries. Origination + gate behavior unit-tested.
* refactor(orchestrator): fold _dispatch_dev_work skip chain into a helper
The per-source if/continue chain grew past xenon's --max-absolute B when the video_post held source joined it. Extract _is_non_dev_dispatch_source (every held-CEO source plus the two Board exploration sources) so the dev loop's skip is one flat call. Behavior-identical.
* feat(video): Phase B — propose_video do-tool (metadata-only, team-gated)
UX/UI dev records a video's composition ref + per-platform captions onto the authoring task's video_draft marker. Team-gated at runtime via _caller_team (Role.DEVELOPER can't tell a ux-dev from a be-dev). Resolves the caller's ACTIVE task via get_active_task_for_agent, not an oldest-first scan that would clobber a second open video task. Metadata only, no render. Wired through do_server + route + schema; added to _DEV_DO.
* feat(video): Phase D — render loop + RemotionRenderer client
Orchestrator-async _video_render_loop renders a completed authoring task's merged composition to MP4 (vertical + square) via the remotion-renderer sidecar and materializes the held video_post draft. RemotionRenderer tars the read-clone's motion/ source, POSTs it, and saves the returned MP4 bytes to a TASK-scoped local path (no shared volume; a composition is reused across videos so a composition-scoped path would clobber an earlier draft). Render failures bounded-retry (read-clone catch-up window, transient sidecar) up to a cap, then terminal-fail. Client tested vs a mock transport; loop vs a mock renderer + real DB.
* feat(video): Phase C — release / spotlight / on-demand video triggers
Three entry points open a UX/UI video-authoring task via VideoEngine.open_video_task: (1) a published release drafts a companion video — best-effort in ReleaseProposalService.approve, never fails the publish; script from the CHANGELOG via the local model with a template fallback. (2) propose_feature_spotlight gains optional wants_video/video_script — best-effort, gated on video_on_spotlight, default-off leaves the spotlight flow byte-for-byte unchanged. (3) POST /video/request (CEO-only) for an on-demand brief, with clean disabled/not_opened responses. All gated on video_engine_enabled.
* fix(video): savepoint-isolate video-task inserts (F042 poisoned session)
The best-effort try/except around open_video_task (release-publish + spotlight hooks) swallowed the Python exception, but a DBAPI error at the insert flush left the shared session must-rollback — so the caller's next commit (release finalize / request boundary) threw PendingRollbackError: the release stuck 'pending' after actually publishing, or the spotlight draft + HTTP response were lost. Wrap both inserts (open_video_task, _originate_video_post) in a begin_nested savepoint (the repo's established F042 pattern) so a DB error rolls back only the insert. open_video_task returns None (every caller already handles it); _originate_video_post propagates to the render loop's handler. Regression test: an insert FK error returns None with the session left usable. Dormant while the flags were off; armed on the NAS.
* feat(video): Phase G — motion/ package + remotion-renderer sidecar + compose
In-repo Remotion v4 motion/ package (ReleaseAnnouncement composition; calculateMetadata returns 1080x1920 vertical / 1080x1080 square from inputProps.orientation) + a credential-free remotion-renderer sidecar: untar the POSTed motion/ source, bundle (LRU-cached per source sha), selectComposition + renderMedia h264, stream the MP4 bytes back — matching the RemotionRenderer client contract. docker/remotion.Dockerfile on Debian (Chrome apt deps, build-time Chrome pre-warm, ffmpeg bundled in @remotion/renderer). Wired into both compose files (roboco_default only, shm_size 1gb, /health check) + the release publish matrix. Verified via a real local render of both cuts; the Debian docker build is the CEO's to run.
* chore(video): D-hardening — video_post source_task_id + render-loop docstring
Add a source_task_id back-reference to the video_post held-draft marker (traceability from a draft to its authoring task; also makes the render loop's two-key idempotency check wireable later). Fix the render-loop test's stale docstring ('never retried' -> bounded-retry). Both from the Phase D critic's non-blocking follow-ups.
* feat(video): Phase E1 — VideoPostService + heartbeat mutex (approve->post)
CEO-approve->post service: heartbeat-renewed Redis mutex (fail-closed, grace=ttl-2*heartbeat), re-read-in-lock double-post guard, per-platform durable commits (asyncio.shield-ed, settle-before-rollback on lock-loss), all writes inside the lock (captions validated pre-lock, applied in-lock — no stale whole-column clobber), idempotent, per-platform retry-skip. Poster interfaces (X/TikTok, mocked here). Reject + list-held-drafts. Survived 3 adversarial rounds; residual = a crash in the poster->commit window (CEO-gated low-freq, documented).
* fix(video): G-hardening — renderer leaks + Share Tech Mono brand font
Sidecar: give bundle() an explicit outDir tracked + deleted on LRU eviction (was leaking ~19MB remotion-webpack-bundle-* per source); res.on('close') cleanup so an aborted/retried download no longer leaks its remotion-out-* MP4 dir. Fonts: vendor Share Tech Mono (roboco-website brand font) as the display face (self-hosted woff2, 400-weight, headline fontWeight 700->400 to avoid faux-bold) + self-hosted Inter body — no gstatic fetch at render time (lsof-verified). Extras: composition_id whitelist (400) + Multer error middleware (400/413).
* feat(video): Phase E2 — X v2 + TikTok posters, tiktok_credentials, routes
LiveXVideoPoster (X v2 chunked media upload: init/append/finalize/STATUS-poll -> tweet w/ media_ids, OAuth1 signer reused). LiveTikTokPoster (OAuth2 inbox: init -> chunked PUT with asymmetric final chunk -> status-fetch; 401 -> refresh_token grant, rotated token persisted). tiktok_credentials Fernet singleton + migration 062 (single head). Routes: CEO approve/reject + list held drafts + write-only tiktok creds, wiring real posters into VideoPostService. Residual: a lock-loss right after a token-refresh flush can discard the rotated token (same rare CEO-gated class as the documented post->commit window).
* feat(video): Phase F — panel video-post queue + TikTok creds card + flags
video-post-queue.tsx: <video> MP4 preview with 9:16/1:1 cut switch, per-platform editable captions (280/2200 counters, over-limit disables approve), approve/reject, Request-a-video dialog. tiktok-credentials-card.tsx (4 write-only OAuth2 fields). feature-flags-card inlines TikTokCredentialsForm under video_engine_enabled. Mounted in command-center. tsc/eslint clean, 273 panel tests green. NOTE: needs the GET /video/posts/{id}/media route + mp4_paths on VideoPostResponse (folded into H) for the preview source.
* feat(video): Phase H — media route + e2e smoke + NAS arming + docs
GET /video/posts/{id}/media?cut= (CEO-gated FileResponse of the rendered MP4; closes the panel preview gap) + mp4_paths on VideoPostResponse. e2e smoke tests/e2e_smoke/test_video_pipeline.py (full flow, sidecar+X/TikTok mocked; asserts dispatcher skips, render-loop materialize, propose_video team-gate, approve idempotency). NAS arming: docker-compose.yml/.yaml ROBOCO_VIDEO_ENGINE_ENABLED/ON_RELEASE/ON_SPOTLIGHT default-on (.yaml resynced to .yml); registry stays off. CLAUDE.md video-engine section + CHANGELOG. Fixed 2 pre-existing route-test pollution leaks. Full suite 11763 passed.
* fix(video): auth-carrying preview, media route confinement, VideoPost type drift
Three fixes along the video preview path:
1. panel video preview auth: the <video> element was pointed straight at
GET /video/posts/{id}/media, but a native <video src> GET carries none
of axios's X-Agent-ID/X-Agent-Role headers — so in the default
header-trust deployment the request 401s. Fetch the cut via
videoApi.getMediaBlob (axios, responseType: blob) and drive <video>
off a URL.createObjectURL result instead. The object URL is revoked
on cut-change (the previous cut's URL) and on unmount, so neither
cut switches nor row teardown leak blob URLs.
2. backend media route confinement: GET /video/posts/{id}/media now
resolves mp4_path and refuses it with 404 when it falls outside
settings.video_output_dir. Defense-in-depth against any future
writer of mp4_paths serving files from arbitrary disk locations.
3. panel VideoPost type/comment drift: added mp4_paths to the
VideoPost interface (the committed VideoPostResponse already
carries it), and corrected the stale comment on videoMediaUrl
that claimed no route served the rendered bytes — the route has
existed since the media endpoint landed; the comment now describes
why getMediaBlob exists instead of a direct <video src>.
* Persist rendered videos to data in physical storage.
* ++
* docs(video): 0.18.0 CHANGELOG entry + RAG + map reference for video engine
- Move the video engine bullet from [Unreleased] into [0.18.0] and note
the ROBOCO_VIDEO_OUTPUT_DIR bind-mount persistence.
- Add docs/rag/architecture/video-engine.md (mirrors x-engine.md shape:
enable/disable, three triggers, render loop + sidecar, CEO gate, media
route confinement, credentials).
- Reference the video render loop in docs/map/orchestrator.md's engine list.
* chore(video): re-bump to 0.19.0 + sync registry compose defaults
Version was wrongly bumped to 0.18.0; 0.18.0 is an already-released
section. Restore its 2026-07-04 date and move the video-engine CHANGELOG
bullet into a new [0.19.0] - 2026-07-05 section above it. Bump
pyproject.toml, roboco/__init__.py, roboco/config.py (app_version),
panel/package.json, and the motion/README inputProps example to 0.19.0.
docker-compose.registry.yml: add ROBOCO_VIDEO_ENGINE_ENABLED /
_VIDEO_ON_RELEASE / _VIDEO_ON_SPOTLIGHT defaulted false (NAS arms them
true), and comment out the video-renders bind mount with a short note
so the public registry image ships video off by default. Structural
sync with docker-compose.yml maintained.
* fix(video): rate-limit /render + reflow motion/README
CodeQL flagged js/missing-rate-limiting on the renderer /render route.
The sidecar is container-network-only with one trusted caller (the
orchestrator, which renders cuts serially), so this limiter is a
retry-storm ceiling (30/min, well above legit render rate), not the
primary control. Also reflows motion/README.md hard-wrapped prose that
failed the markdown quality gate.
* fix(build): finish pnpm 11 migration + regen verb tables
The panel Docker image build failed on `pnpm install --frozen-lockfile`:
node:22-alpine's corepack resolved to its bundled pnpm 11, but
panel/package.json pinned packageManager to pnpm@10.25.0, and pnpm 11
refuses to run against that pin. The Dockerfiles were already written for
pnpm 11 (comments, CI=true, strictDepBuilds); the package.json pin was the
stale outlier. Finish the migration instead of working around it:
- panel/package.json: packageManager pnpm@10.25.0 -> pnpm@11.10.0; drop the
`pnpm` field (pnpm 11 ignores it — build approval lives in
panel/pnpm-workspace.yaml's allowBuilds). Lockfile unchanged (pnpm 11
accepts it as-is); frozen-lockfile verified.
- remotion-renderer/package.json: pin packageManager pnpm@11.10.0 for
determinism (was relying on corepack's implicit default); engines.node
>=22.13 (pnpm 11 requirement).
- docker/panel.Dockerfile + docker/remotion.Dockerfile: `corepack prepare
pnpm@11.10.0 --activate` so the build uses the pinned version explicitly
instead of trusting corepack's bundled default (which a future
node:22-alpine could change).
- .github/workflows/panel-ci.yml: Node 20 -> 22 (pnpm 11 requires
Node >=22.13; Node 20 fails the engines check).
Also regenerate agents/prompts/_generated/{developer,head_marketing,verbs}.md
— the video engine added propose_video and extended propose_feature_spotlight
(wants_video, video_script) but the verb tables weren't refreshed, failing
the foundation-check quality gate.
* chore(build): approve esbuild build script in remotion pnpm-workspace.yaml
pnpm 11 generated this file with a placeholder ('set this to true or false')
during install; resolve it to true so local dev of the renderer doesn't
re-prompt. esbuild's postinstall only verifies the prebuilt platform binary
(@esbuild/<platform> is installed as an optional dep), so approving it is
safe and silences the ERR_PNPM_IGNORED_BUILDS warning.
* fix(build): copy pnpm-workspace.yaml into panel + remotion images
pnpm 11 hard-errors with [ERR_PNPM_IGNORED_BUILDS] (exit 1) when a
dependency ships a postinstall script that isn't approved in
allowBuilds. Both Dockerfiles copied only package.json + pnpm-lock.yaml,
so the build-approval map in pnpm-workspace.yaml never made it into the
image — the remotion image build died on esbuild@0.28.1's postinstall.
Copy pnpm-workspace.yaml alongside the manifests in both images. In
panel, this also drops the --config.strictDepBuilds=false workaround:
with sharp and unrs-resolver now approved, their postinstalls run and
install the platform-specific binaries (previously skipped, leaving
sharp without its @img/sharp-* binary at runtime).
Verified locally: remotion + panel `pnpm install --frozen-lockfile`
exit 0 with the workspace file present; both exit 1 without it.
* fix(perf): offload conventions + release-readiness blocking I/O off the event loop
The orchestrator runs uvicorn and the orchestration background loops on a
single shared event loop, so any sync I/O anywhere — even inside a background
loop — blocks API responsiveness for its duration. Two call sites were missing
asyncio.to_thread wrappers:
- ConventionsService.get_map/health/restore called the sync _resolve
(`git rev-parse`), _read_committed_standard (file read + yaml parse), and
_derive (filesystem walk via derive_from_scan) inline. Reachable from
GET /api/projects/{id}/conventions and from the agent spawn-prepare path.
- ReleaseManagerEngine._production_assess called gather_snapshot inline —
multiple `subprocess.run` git calls + a filesystem walk, running inside the
release-manager background loop.
Wrap each blocking call in asyncio.to_thread at the async boundary. No
signature changes; helpers stay sync. Verified: targeted tests pass
(196 passed, 36 DB-skipped), ruff + format clean.
These were the only responsiveness gaps surfaced by the concurrency audit —
the rest of the heavy paths (agent spawn via `docker run -d`, video render
loop, git ops via the 16-worker ThreadPoolExecutor, workspace subprocess
calls) already offload correctly. No API/worker container split needed.
* feat(storage): add MinIO config + dep + compose (no-op, default-off)
Chunk 1 of the MinIO video-storage plan (§1, §2, §6). No behavior change:
minio_endpoint defaults to empty = disabled, the existing FileResponse serve
path is untouched (chunk 4 wires the serve path; chunk 2 adds the client).
- pyproject.toml: add `minio` (minio-py) to dependencies; regenerate uv.lock
(resolves minio v7.2.20 + pycryptodome transitive).
- roboco/config.py: add 5 settings fields after video_output_dir
(minio_endpoint/_access_key/_secret_key/_bucket/_region). Plain str Fields
matching the existing ROBOCO_ENCRYPTION_KEY style; no SecretStr, no
presign_ttl_seconds (YAGNI — we don't presign in phase 1).
- docker-compose.yml: add `minio` service (data network only, named
minio-data volume, host ports 19000/19001 for debugging, mc healthcheck)
and a one-shot `minio-init` service mirroring the ollama-init pattern
(mc alias set + mb -p, idempotent via || true). Add ROBOCO_MINIO_* env to
the orchestrator env block (endpoint, access/secret key, bucket, region).
- docker-compose.registry.yml: intentionally omit the minio/minio-init
services and leave ROBOCO_MINIO_* unset (NAS default-on, registry
default-off — the established pattern); comment added to the orchestrator
env block noting the omission.
* docs(storage): 0.19.0 CHANGELOG + RAG + map reference for MinIO chunk 1
Backfills the release-polish docs for MinIO chunk 1 (§10 of the plan):
- docker-compose.yaml synced to docker-compose.yml (the two NAS compose files
must stay byte-identical; .yml was edited in chunk 1, .yaml was stale).
- CHANGELOG [0.19.0]: Added (MinIO scaffolding) + Fixed (event-loop I/O offload).
- docs/rag/architecture/minio-storage.md: RAG doc mirroring video-engine.md.
- docs/map/deployment-tooling.md: one-line storage reference.
* MinIO chunk 2: minio_client module (singleton + unconfigured guard) (#309)
* feat(storage): minio_client module (singleton + unconfigured guard)
Chunk 2 of the MinIO plan (§3). roboco/services/minio_client.py adds:
- get_client(): singleton minio-py Minio from settings; returns None when
minio_endpoint is empty (the disabled path used by the chunk 3/4 guards).
Parses http://... endpoint into host:port + secure flag.
- put_object(bytes, key): no-ops when unconfigured; otherwise PUTs to
settings.minio_bucket with ContentType video/mp4.
- get_object_stream(key): yields object bytes for StreamingResponse; lets
S3Error propagate so the serve route (chunk 4) can fall back to disk.
Sync calls — every call site wraps in asyncio.to_thread (chunks 3/4). One
unit test covers the unconfigured guard + endpoint scheme parsing (mocks,
no real MinIO). Not yet wired into remotion_client._save or the media route.
* MinIO chunk 3: wire write path (remotion_client._save PUT) (#310)
* feat(storage): wire MinIO write path in remotion_client._save
Chunk 3 of the MinIO plan (§3). After the local mp4 write, _save PUTs the bytes
to MinIO under key = Path(mp4_path).name (already {render_key}-{orientation}.mp4),
guarded by minio_client.get_client() (None when minio_endpoint empty) and
wrapped in asyncio.to_thread. Local disk stays the source of truth for the
poster publish path (x_video_client/tiktok_client read mp4_path from disk);
the PUT is additive. _save still returns the local path str — mp4_paths,
marker, and schema unchanged. Disabled (local-only) when MinIO unconfigured.
One test: asserts put_object is called with the basename key when configured
and the local file is still written; existing test stays green via the
unconfigured-default path. Mocks only.
* fix(storage): make MinIO PUT non-fatal in remotion_client._save
A configured-but-down MinIO made put_object raise inside the worker thread,
failing the render and retry-looping a task whose local file was already
written. Local disk is the source of truth and the serve route falls back to
FileResponse on S3Error, so a failed durable-copy PUT must never fail the
render — log and continue; the next render re-attempts the PUT.
Adds test_save_swallows_minio_put_failure (PUT raises -> _save still returns
the local path and the local file is written). Extends the CHANGELOG write-
path bullet with the non-fatal guarantee.
* MinIO chunk 4: serve path (StreamingResponse + FileResponse fallback) (#311)
* feat(storage): serve MinIO via the media route (StreamingResponse + FileResponse fallback)
Chunk 4 of the MinIO plan (§4 — the crux). GET /api/video/posts/{id}/media
derives key = Path(mp4_path).name and, when minio_endpoint is set, returns a
StreamingResponse over minio_client.get_object_stream(key), keeping
_require_ceo so auth stays end-to-end (no presigned URLs). Falls back to
FileResponse on S3Error (old render not in MinIO) or when MinIO is
unconfigured — the panel's axios-blob flow is unchanged (same URL, headers,
body, just chunked). The confinement check is kept as defense-in-depth (the
key is a basename so traversal is impossible, but the check is cheap and
protects the poster path).
Two integration tests: configured serve path streams from a stubbed
get_object_stream (CEO 200, non-CEO 403); unconfigured fallback serves the
local file via FileResponse. Mocks only — no real MinIO.
* fix(storage): eager stat_object probe so the MinIO serve fallback actually fires
The chunk-4 route wrapped StreamingResponse(get_object_stream(key), ...) in a
try/except, but get_object_stream is a lazy generator — its client.get_object
call runs on the first next(), i.e. AFTER the route returned and Starlette
started streaming. An S3Error (NoSuchKey / MinIO down) there is uncatchable;
the try/except caught nothing and the FileResponse fallback never triggered.
Add minio_client.stat_object(key): an eager existence/readiness probe that
runs INSIDE the route's try/except, so a missing object or down MinIO raises
before the StreamingResponse starts and the fallback serves the local file.
stat-then-get is two round trips; a mid-stream failure after a successful stat
is a rare race the CEO can retry (documented ceiling).
Tests: the configured test now stubs stat_object; a new test asserts the
S3Error fallback serves the local file via FileResponse and that
get_object_stream is never called. RAG doc updated to record the eager-probe
correctness detail + the non-fatal PUT.
* docs(rag): mark MinIO deployment note landed (chunk 5) (#312)
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* fix(video): offload minio stat_object off the event loop
stat_object was called inline in the async media route, blocking the
shared event loop for one sync urllib3 round-trip per preview request —
contradicting minio_client's own 'every call site wraps in to_thread'
docstring and this PR's perf-fix theme. Wrap in asyncio.to_thread; the
try/except still catches S3Error (to_thread re-raises) so the
FileResponse fallback is unchanged. Also add the trailing newline to
the minio-storage RAG doc.
* Fix red CI
* Make CI green
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
@@ -11,6 +11,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
### Added
|
||||
|
||||
- **RoboCo video engine (default-off).** With `ROBOCO_VIDEO_ENGINE_ENABLED`, a release/feature-spotlight/on-demand CEO trigger opens a normal, assigned UX/UI authoring task (balanced across the two ux-devs) instead of a held draft — the dev builds a Remotion composition under `motion/` and proposes its composition id + per-platform captions via the team-gated `propose_video` do-tool, then ships it through the standard commit/PR/QA/doc/review lifecycle. Once that task completes, an orchestrator render loop tars the merged `motion/` source to a new credential-free `remotion-renderer` sidecar, renders both the 9:16 and 1:1 MP4 cuts, and materializes a held `video_post` draft (mirroring the X-post/release-proposal shape: Secretary-owned, skipped by every dispatcher). The CEO previews, edits, approves, or rejects each draft in a new panel video queue; approving posts the rendered clip to X (native video, v2 media upload) and/or TikTok (inbox upload) under a heartbeat-renewed lock and is idempotent — an already-posted draft is a no-op. `ROBOCO_VIDEO_ON_RELEASE` / `ROBOCO_VIDEO_ON_SPOTLIGHT` gate the two automatic triggers independently of the CEO's on-demand `POST /video/request`; TikTok's OAuth2 secrets live Fernet-encrypted alongside the existing X credentials, and every unconfigured leg (renderer, X, TikTok) degrades to a graceful no-op rather than a crash. Rendered MP4s persist under `ROBOCO_VIDEO_OUTPUT_DIR` (bind-mounted in all three compose files so renders survive container recreation).
|
||||
- **MinIO object storage scaffolding (default-off).** `ROBOCO_MINIO_*` config (`minio_endpoint`, `minio_access_key`, `minio_secret_key`, `minio_bucket`, `minio_region`) + a `minio` service and a one-shot `minio-init` (idempotent bucket create) in the NAS compose files, on the `data` network with a named `minio-data` volume; `minio` (minio-py) added as a dependency. Empty `minio_endpoint` = disabled and the existing `FileResponse` media-serve path is byte-for-byte unchanged — this is scaffolding; the write path (PUT after local save) and serve path (`StreamingResponse` with `FileResponse` fallback) land in later chunks. The registry compose omits MinIO entirely (NAS default-on, registry default-off).
|
||||
- **MinIO storage client.** `roboco/services/minio_client.py` — a singleton minio-py client with an unconfigured guard (`get_client()` returns `None` when `minio_endpoint` is empty), plus `put_object` and `get_object_stream`. Sync; call sites wrap in `asyncio.to_thread`. Not yet wired into the write/serve paths (chunks 3–4).
|
||||
- **MinIO write path.** `remotion_client._save` now PUTs each rendered MP4 to MinIO (key = the basename `{render_key}-{orientation}.mp4`) after the local write, guarded by `minio_endpoint`. Local disk stays the source of truth for the poster publish path; the PUT is additive and **non-fatal** — a failed PUT (MinIO down, transient 5xx) is logged and the render still succeeds, since the serve route falls back to `FileResponse` on `S3Error`. No schema, marker, or `mp4_paths` change. Disabled (local-only) when MinIO is unconfigured.
|
||||
- **MinIO serve path (the user-visible switch).** The panel video-preview media route (`GET /api/video/posts/{id}/media`) now streams the MP4 from MinIO (`StreamingResponse` over `minio_client.get_object_stream`, key = the basename) when `minio_endpoint` is set, keeping `_require_ceo` so auth stays end-to-end (no presigned URLs). Falls back to `FileResponse` from the local video-renders dir when MinIO is unconfigured OR on `S3Error` (old renders not yet in MinIO / MinIO down). The panel's axios-blob flow is unchanged — same URL, headers, body. Set `ROBOCO_MINIO_ENDPOINT` and renders start serving from MinIO.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Conventions + release-readiness I/O no longer blocks the API event loop.** `ConventionsService.get_map/health/restore` and `ReleaseManagerEngine._production_assess` ran sync `git rev-parse`, filesystem walks, and yaml parses inline on the orchestrator's shared uvicorn event loop, stalling API responsiveness during conventions reads (reachable from `GET /api/projects/{id}/conventions` and the agent spawn-prepare path) and the release-manager background loop. Each blocking call is now wrapped in `asyncio.to_thread` at the async boundary; no signature changes. A concurrency audit confirmed the rest of the heavy paths (agent spawn via `docker run -d`, the video render loop, git ops via the 16-worker ThreadPoolExecutor, workspace subprocess calls) already offload correctly — no API/worker container split is warranted.
|
||||
|
||||
## [0.18.0] - 2026-07-04
|
||||
|
||||
|
||||
@@ -295,6 +295,11 @@ services:
|
||||
ROBOCO_VIDEO_ENGINE_ENABLED: ${ROBOCO_VIDEO_ENGINE_ENABLED:-false}
|
||||
ROBOCO_VIDEO_ON_RELEASE: ${ROBOCO_VIDEO_ON_RELEASE:-false}
|
||||
ROBOCO_VIDEO_ON_SPOTLIGHT: ${ROBOCO_VIDEO_ON_SPOTLIGHT:-false}
|
||||
# MinIO object storage for rendered videos is intentionally omitted from
|
||||
# this registry compose (NAS default-on, registry default-off — the
|
||||
# established pattern). The minio/minio-init services are absent here and
|
||||
# ROBOCO_MINIO_* is left unset, so minio_endpoint defaults to empty and the
|
||||
# media route falls back to FileResponse. Arm via a custom override file.
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
- ${CLAUDE_AUTH_DIR:-${HOME}/.claude}:/root/.claude
|
||||
|
||||
@@ -47,6 +47,55 @@ services:
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
# ==========================================================================
|
||||
# MinIO - Object storage for rendered videos (NAS default-on; registry OFF)
|
||||
# ==========================================================================
|
||||
minio:
|
||||
image: minio/minio:latest
|
||||
container_name: roboco-minio
|
||||
restart: unless-stopped
|
||||
# data-only network — keeps MinIO off the agent mesh; the multi-homed
|
||||
# orchestrator reaches it via its data NIC. Host ports for debugging.
|
||||
networks:
|
||||
- data
|
||||
command: server /data --console-address ":9001"
|
||||
environment:
|
||||
MINIO_ROOT_USER: ${ROBOCO_MINIO_ACCESS_KEY:-minio}
|
||||
MINIO_ROOT_PASSWORD: ${ROBOCO_MINIO_SECRET_KEY:-minio123}
|
||||
ports:
|
||||
- "19000:9000"
|
||||
- "19001:9001"
|
||||
volumes:
|
||||
- minio-data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "mc", "ready", "local"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 10s
|
||||
|
||||
# MinIO bucket init - one-shot, mirrors the ollama-init pattern: depends on
|
||||
# minio healthy, creates the bucket idempotently, then exits. `|| true`
|
||||
# covers re-runs (bucket already exists).
|
||||
minio-init:
|
||||
image: minio/mc:latest
|
||||
container_name: roboco-minio-init
|
||||
depends_on:
|
||||
minio:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- data
|
||||
entrypoint: ["/bin/sh", "-c"]
|
||||
# Note: $$ escapes $ for docker-compose variable substitution so the
|
||||
# container's own env vars reach mc.
|
||||
command:
|
||||
- mc alias set local http://roboco-minio:9000 $$ROBOCO_MINIO_ACCESS_KEY $$ROBOCO_MINIO_SECRET_KEY && mc mb -p local/$$ROBOCO_MINIO_BUCKET || true
|
||||
environment:
|
||||
ROBOCO_MINIO_ACCESS_KEY: ${ROBOCO_MINIO_ACCESS_KEY:-minio}
|
||||
ROBOCO_MINIO_SECRET_KEY: ${ROBOCO_MINIO_SECRET_KEY:-minio123}
|
||||
ROBOCO_MINIO_BUCKET: ${ROBOCO_MINIO_BUCKET:-roboco-video-renders}
|
||||
restart: "no"
|
||||
|
||||
# ==========================================================================
|
||||
# Ollama - Local LLM and Embedding Server
|
||||
# ==========================================================================
|
||||
@@ -351,6 +400,14 @@ services:
|
||||
# itself is default-off (video_engine_enabled); this just points the
|
||||
# client at the sidecar for when it's armed.
|
||||
ROBOCO_REMOTION_BASE_URL: http://roboco-remotion:3001
|
||||
# MinIO object storage for rendered videos (use container name). Empty
|
||||
# endpoint = disabled (FileResponse fallback); armed here for the NAS
|
||||
# deploy, left OFF in docker-compose.registry.yml.
|
||||
ROBOCO_MINIO_ENDPOINT: http://roboco-minio:9000
|
||||
ROBOCO_MINIO_ACCESS_KEY: ${ROBOCO_MINIO_ACCESS_KEY:-minio}
|
||||
ROBOCO_MINIO_SECRET_KEY: ${ROBOCO_MINIO_SECRET_KEY:-minio123}
|
||||
ROBOCO_MINIO_BUCKET: ${ROBOCO_MINIO_BUCKET:-roboco-video-renders}
|
||||
ROBOCO_MINIO_REGION: ${ROBOCO_MINIO_REGION:-us-east-1}
|
||||
# Host paths for spawning agent containers (required for Docker-in-Docker)
|
||||
# IMPORTANT: These must be ABSOLUTE paths on the host filesystem
|
||||
ROBOCO_HOST_PROJECT_DIR: ${ROBOCO_HOST_PROJECT_DIR:-/volume1/roboco}
|
||||
@@ -605,3 +662,8 @@ networks:
|
||||
# ports keep working.
|
||||
data:
|
||||
name: roboco_data
|
||||
|
||||
volumes:
|
||||
# Named volume for MinIO — keeps rendered-video storage out of the
|
||||
# ${ROBOCO_DATA_DIR} bind-mount sprawl; docker-managed durable store.
|
||||
minio-data:
|
||||
|
||||
@@ -47,6 +47,55 @@ services:
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
# ==========================================================================
|
||||
# MinIO - Object storage for rendered videos (NAS default-on; registry OFF)
|
||||
# ==========================================================================
|
||||
minio:
|
||||
image: minio/minio:latest
|
||||
container_name: roboco-minio
|
||||
restart: unless-stopped
|
||||
# data-only network — keeps MinIO off the agent mesh; the multi-homed
|
||||
# orchestrator reaches it via its data NIC. Host ports for debugging.
|
||||
networks:
|
||||
- data
|
||||
command: server /data --console-address ":9001"
|
||||
environment:
|
||||
MINIO_ROOT_USER: ${ROBOCO_MINIO_ACCESS_KEY:-minio}
|
||||
MINIO_ROOT_PASSWORD: ${ROBOCO_MINIO_SECRET_KEY:-minio123}
|
||||
ports:
|
||||
- "19000:9000"
|
||||
- "19001:9001"
|
||||
volumes:
|
||||
- minio-data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "mc", "ready", "local"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 10s
|
||||
|
||||
# MinIO bucket init - one-shot, mirrors the ollama-init pattern: depends on
|
||||
# minio healthy, creates the bucket idempotently, then exits. `|| true`
|
||||
# covers re-runs (bucket already exists).
|
||||
minio-init:
|
||||
image: minio/mc:latest
|
||||
container_name: roboco-minio-init
|
||||
depends_on:
|
||||
minio:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- data
|
||||
entrypoint: ["/bin/sh", "-c"]
|
||||
# Note: $$ escapes $ for docker-compose variable substitution so the
|
||||
# container's own env vars reach mc.
|
||||
command:
|
||||
- mc alias set local http://roboco-minio:9000 $$ROBOCO_MINIO_ACCESS_KEY $$ROBOCO_MINIO_SECRET_KEY && mc mb -p local/$$ROBOCO_MINIO_BUCKET || true
|
||||
environment:
|
||||
ROBOCO_MINIO_ACCESS_KEY: ${ROBOCO_MINIO_ACCESS_KEY:-minio}
|
||||
ROBOCO_MINIO_SECRET_KEY: ${ROBOCO_MINIO_SECRET_KEY:-minio123}
|
||||
ROBOCO_MINIO_BUCKET: ${ROBOCO_MINIO_BUCKET:-roboco-video-renders}
|
||||
restart: "no"
|
||||
|
||||
# ==========================================================================
|
||||
# Ollama - Local LLM and Embedding Server
|
||||
# ==========================================================================
|
||||
@@ -351,6 +400,14 @@ services:
|
||||
# itself is default-off (video_engine_enabled); this just points the
|
||||
# client at the sidecar for when it's armed.
|
||||
ROBOCO_REMOTION_BASE_URL: http://roboco-remotion:3001
|
||||
# MinIO object storage for rendered videos (use container name). Empty
|
||||
# endpoint = disabled (FileResponse fallback); armed here for the NAS
|
||||
# deploy, left OFF in docker-compose.registry.yml.
|
||||
ROBOCO_MINIO_ENDPOINT: http://roboco-minio:9000
|
||||
ROBOCO_MINIO_ACCESS_KEY: ${ROBOCO_MINIO_ACCESS_KEY:-minio}
|
||||
ROBOCO_MINIO_SECRET_KEY: ${ROBOCO_MINIO_SECRET_KEY:-minio123}
|
||||
ROBOCO_MINIO_BUCKET: ${ROBOCO_MINIO_BUCKET:-roboco-video-renders}
|
||||
ROBOCO_MINIO_REGION: ${ROBOCO_MINIO_REGION:-us-east-1}
|
||||
# Host paths for spawning agent containers (required for Docker-in-Docker)
|
||||
# IMPORTANT: These must be ABSOLUTE paths on the host filesystem
|
||||
ROBOCO_HOST_PROJECT_DIR: ${ROBOCO_HOST_PROJECT_DIR:-/volume1/roboco}
|
||||
@@ -605,3 +662,8 @@ networks:
|
||||
# ports keep working.
|
||||
data:
|
||||
name: roboco_data
|
||||
|
||||
volumes:
|
||||
# Named volume for MinIO — keeps rendered-video storage out of the
|
||||
# ${ROBOCO_DATA_DIR} bind-mount sprawl; docker-managed durable store.
|
||||
minio-data:
|
||||
|
||||
@@ -258,6 +258,7 @@ deployment-tooling
|
||||
- ROBOCO_ROADMAP_ENGINE_ENABLED / _INTERVAL_SECONDS (default 604800) / _MIN_ITEMS_PER_CYCLE / _MAX_ITEMS_PER_CYCLE — the board roadmap engine
|
||||
- ROBOCO_X_FEATURE_SPOTLIGHT_ENABLED / _INTERVAL_SECONDS (default 259200/3d) — X-engine feature-spotlight sub-switch (requires ROBOCO_X_ENGINE_ENABLED also on), default off
|
||||
- ROBOCO_FABLE_MODE_ENABLED — opus-fable-playbook adoption (doctrine layer in the composed prompt + 5 Claude-path hook scripts + 1 grok-path hook), default off; off = byte-for-byte unchanged spawn path
|
||||
- ROBOCO_MINIO_ENDPOINT / _ACCESS_KEY / _SECRET_KEY / _BUCKET / _REGION — MinIO object storage (default-off; empty endpoint = disabled, media route falls back to `FileResponse`; when set, `remotion_client._save` PUTs each render to MinIO after the local write and `GET /api/video/posts/{id}/media` streams it via `StreamingResponse` over `minio_client.get_object_stream`, key = basename, `_require_ceo` kept so auth stays end-to-end — no presigned URLs; `S3Error` falls back to `FileResponse`); NAS compose runs `minio` + `minio-init` on the `data` network with a named `minio-data` volume, registry compose omits MinIO; see `docs/rag/architecture/minio-storage.md`
|
||||
- ROBOCO_TRANSCRIPT_RETENTION_DAYS / ROBOCO_TRANSCRIPT_PRUNE_ENABLED / _INTERVAL_SECONDS
|
||||
- ROBOCO_IMAGE_PRUNE_ENABLED / _INTERVAL_SECONDS
|
||||
- ROBOCO_GIT_COMMAND_TIMEOUT_SECONDS / _COMMIT_TIMEOUT_SECONDS / _NETWORK_TIMEOUT_SECONDS
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
# Object Storage (MinIO)
|
||||
|
||||
## What It Is
|
||||
|
||||
Rendered MP4s are written to a host bind mount (`ROBOCO_VIDEO_OUTPUT_DIR`, default `/data/video-renders`) and served back via `FileResponse` from the media route. MinIO adds decoupled, docker-managed durable object storage so renders outlive the orchestrator container without relying on the bind-mount sprawl, and gives a clean serve path that keeps app-level auth end-to-end. The client is `minio` (minio-py), sync, with every call site wrapped in `asyncio.to_thread` — same pattern as the existing `_save` in `roboco/services/remotion_client.py`.
|
||||
|
||||
## Enable/Disable
|
||||
|
||||
| Variable | Default | Effect |
|
||||
|----------|---------|--------|
|
||||
| `ROBOCO_MINIO_ENDPOINT` | `` (empty) | MinIO endpoint, e.g. `http://roboco-minio:9000`. Empty = disabled (the media route falls back to `FileResponse` from the local video-renders dir). |
|
||||
| `ROBOCO_MINIO_ACCESS_KEY` | `` | Access key. Required when endpoint is set. |
|
||||
| `ROBOCO_MINIO_SECRET_KEY` | `` | Secret key. Required when endpoint is set. |
|
||||
| `ROBOCO_MINIO_BUCKET` | `roboco-video-renders` | Bucket for rendered videos. Created idempotently by the `minio-init` one-shot service. |
|
||||
| `ROBOCO_MINIO_REGION` | `us-east-1` | MinIO region. |
|
||||
|
||||
Armed in the NAS compose (`docker-compose.yml` / `docker-compose.yaml`); intentionally omitted from `docker-compose.registry.yml` (NAS default-on, registry default-off).
|
||||
|
||||
## Current state (0.19.0 chunk 4 — serve path landed)
|
||||
|
||||
Config fields, the `minio` dependency, and the compose services (`minio` + `minio-init`) are landed (chunk 1). Chunk 2 adds `roboco/services/minio_client.py` — a singleton `Minio` with an unconfigured guard (`get_client()` returns `None` when `minio_endpoint` is empty), plus `put_object`, `get_object_stream`, and `stat_object`. Chunk 3 wires the write path: `remotion_client._save` keeps the local write (the poster publish path in `x_video_client.py` / `tiktok_client.py` still reads `Path(mp4_path).read_bytes()`) and adds a MinIO `put_object` after it, guarded by `settings.minio_endpoint` and non-fatal — a failed PUT (MinIO down) is logged and the render still succeeds, since local disk is the source of truth. Key = `Path(mp4_path).name` (already `{render_key}-{orientation}.mp4`) — no schema change, no new marker field. Chunk 4 wires the serve path (now live): the media route (`roboco/api/routes/video.py`) keeps `_require_ceo(agent)` and returns a `StreamingResponse` wrapping `minio_client.get_object_stream(key)` when configured, falling back to `FileResponse` when unconfigured OR when the eager `minio_client.stat_object(key)` probe raises (missing object / MinIO down). The probe is eager because `get_object_stream` is a lazy generator — its `get_object` call runs on first iteration, after Starlette has started streaming and the response is no longer take-back-able; the probe runs inside the route's `try/except` so the `S3Error` fallback actually fires. The key is the basename, so the existing confinement check stays as defense-in-depth. With `minio_endpoint` empty, the serve path is byte-for-byte the pre-MinIO `FileResponse`. The `minio` service runs on the `data` network only (off the agent mesh); the orchestrator reaches it via its `data` NIC. Host ports `19000:9000` / `19001:9001` are published for debugging only.
|
||||
|
||||
## Planned end state (later chunks)
|
||||
|
||||
- (none — the deployment note landed in chunk 5; see docs.roboco.tech/deploy/deployment)
|
||||
|
||||
## Why not presigned URLs
|
||||
|
||||
A presigned URL is a TTL bearer token for the object; the browser's `<video>` element making a direct GET to MinIO cannot carry `X-Agent-ID`/`X-Agent-Role`, so once issued, MinIO cannot enforce the app's CEO-role check — anyone with the URL gets the bytes for the TTL. Proxying through the authenticated route keeps the app-level auth boundary end-to-end at the cost of one loopback streaming hop on a single-host docker deploy. **Now live** — the chunk-4 serve path proxies through the route. Presigned URLs are deferred until CDN / direct-browser-to-MinIO becomes a goal and panel auth is reworked to mint short-lived tokens.
|
||||
|
||||
## Skipped (add when)
|
||||
|
||||
- Presigned URLs / `presign_ttl_seconds` — see above.
|
||||
- Storage interface / factory — one implementation, no abstraction.
|
||||
- Full MinIO-only switch (drop local disk) — when the publish path takes bytes instead of a path.
|
||||
- Lifecycle policy / bucket versioning / replication — when there's a retention or multi-site requirement.
|
||||
@@ -39,6 +39,8 @@ dependencies = [
|
||||
"passlib[bcrypt]", # Password hashing
|
||||
"tenacity", # Retry logic
|
||||
"structlog", # Structured logging
|
||||
# Storage
|
||||
"minio", # Object storage client for rendered videos (sync, to_thread-wrapped)
|
||||
# Streaming
|
||||
"sse-starlette", # Server-Sent Events for A2A streaming
|
||||
# Direct imports (promoted from transitive)
|
||||
|
||||
+54
-17
@@ -5,12 +5,13 @@ API never returns plaintext)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import APIRouter, HTTPException, status
|
||||
from fastapi.responses import FileResponse
|
||||
from fastapi.responses import FileResponse, StreamingResponse
|
||||
|
||||
from roboco.api.deps import CurrentAgentContext, DbSession, require_ceo_role
|
||||
from roboco.api.schemas.video import (
|
||||
@@ -26,6 +27,7 @@ from roboco.api.schemas.video import (
|
||||
from roboco.config import settings
|
||||
from roboco.foundation.policy.content import markers
|
||||
from roboco.security import guard_deco
|
||||
from roboco.services import minio_client
|
||||
from roboco.services.task import VIDEO_POST_SOURCE, get_task_service
|
||||
from roboco.services.tiktok_client import build_tiktok_poster
|
||||
from roboco.services.tiktok_credentials import (
|
||||
@@ -56,6 +58,25 @@ def _require_ceo(agent: CurrentAgentContext) -> None:
|
||||
require_ceo_role(agent.role, action="view or act on the video engine")
|
||||
|
||||
|
||||
def _resolve_video_cut(task: TaskTable, cut: str) -> Path:
|
||||
"""Resolve the on-disk MP4 path for ``cut`` off the task's held draft, or
|
||||
404. The ``is_relative_to`` confinement check stays even though the MinIO
|
||||
key is a basename (traversal-proof) — it also guards the ``FileResponse``
|
||||
fallback path that reads ``mp4_path`` straight from disk."""
|
||||
draft = markers.get_video_draft(task) or {}
|
||||
mp4_path = (draft.get("mp4_paths") or {}).get(cut)
|
||||
if not mp4_path or not Path(mp4_path).is_file():
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND, detail=f"No rendered {cut} cut"
|
||||
)
|
||||
output_dir = Path(settings.video_output_dir).resolve()
|
||||
if not Path(mp4_path).resolve().is_relative_to(output_dir):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND, detail=f"No rendered {cut} cut"
|
||||
)
|
||||
return Path(mp4_path)
|
||||
|
||||
|
||||
@router.post("/request", response_model=VideoRequestResponse)
|
||||
@guard_deco.rate_limit(requests=20, window=60)
|
||||
@guard_deco.block_clouds()
|
||||
@@ -144,16 +165,25 @@ async def list_video_posts(
|
||||
return [_to_response(t) for t in tasks]
|
||||
|
||||
|
||||
@router.get("/posts/{task_id}/media")
|
||||
@router.get("/posts/{task_id}/media", response_model=None)
|
||||
async def get_video_post_media(
|
||||
task_id: UUID,
|
||||
cut: str,
|
||||
db: DbSession,
|
||||
agent: CurrentAgentContext,
|
||||
) -> FileResponse:
|
||||
) -> StreamingResponse | FileResponse:
|
||||
"""Serve one rendered MP4 cut of a held video_post draft — the panel
|
||||
preview player's ``src``. 404s on a missing task/cut/file; 400 on a
|
||||
``cut`` outside {vertical, square}."""
|
||||
``cut`` outside {vertical, square}.
|
||||
|
||||
When MinIO is configured (``minio_endpoint`` set) the route streams the
|
||||
object from MinIO via ``minio_client.get_object_stream`` (key = the
|
||||
basename of ``mp4_path``). Auth stays end-to-end — ``_require_ceo`` is
|
||||
kept, no presigned URLs, no redirect — so the panel's axios-blob flow is
|
||||
unchanged (same URL, headers, body — just chunked). Falls back to
|
||||
``FileResponse`` from the local render dir when MinIO is unconfigured OR
|
||||
on ``S3Error`` (old renders not yet in MinIO / MinIO down). The local file
|
||||
existence + confinement checks stay as defense-in-depth."""
|
||||
_require_ceo(agent)
|
||||
if cut not in _VALID_CUTS:
|
||||
raise HTTPException(
|
||||
@@ -165,19 +195,26 @@ async def get_video_post_media(
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND, detail="No such video draft"
|
||||
)
|
||||
draft = markers.get_video_draft(task) or {}
|
||||
mp4_path = (draft.get("mp4_paths") or {}).get(cut)
|
||||
if not mp4_path or not Path(mp4_path).is_file():
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND, detail=f"No rendered {cut} cut"
|
||||
)
|
||||
output_dir = Path(settings.video_output_dir).resolve()
|
||||
if not Path(mp4_path).resolve().is_relative_to(output_dir):
|
||||
# Defense-in-depth: a mp4_paths entry pointing outside the configured
|
||||
# render dir is refused even though the file exists on disk.
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND, detail=f"No rendered {cut} cut"
|
||||
)
|
||||
mp4_path = _resolve_video_cut(task, cut)
|
||||
key = mp4_path.name
|
||||
if minio_client.get_client() is not None:
|
||||
try:
|
||||
# Eager probe so a missing object / down MinIO raises HERE — the
|
||||
# fallback below catches it. get_object_stream is a lazy generator,
|
||||
# so wrapping StreamingResponse(...) alone wouldn't catch S3Error:
|
||||
# it fires on the first next(), after Starlette has started
|
||||
# streaming and the response is no longer take-back-able.
|
||||
await asyncio.to_thread(minio_client.stat_object, key)
|
||||
except Exception:
|
||||
# NoSuchKey (old render not yet in MinIO) or MinIO down — fall
|
||||
# back to the local file, which is the source of truth. Auth
|
||||
# already passed; this is purely a storage-read fallback.
|
||||
pass
|
||||
else:
|
||||
return StreamingResponse(
|
||||
minio_client.get_object_stream(key),
|
||||
media_type="video/mp4",
|
||||
)
|
||||
return FileResponse(mp4_path, media_type="video/mp4")
|
||||
|
||||
|
||||
|
||||
@@ -956,6 +956,32 @@ class Settings(BaseSettings):
|
||||
"The sidecar never writes here directly — it only returns bytes."
|
||||
),
|
||||
)
|
||||
# MinIO object storage for rendered MP4s. Empty endpoint = disabled (the
|
||||
# media route falls back to FileResponse from the local video-renders dir).
|
||||
# Armed in the NAS compose; intentionally left OFF in the registry compose.
|
||||
minio_endpoint: str = Field(
|
||||
default="",
|
||||
description=(
|
||||
"MinIO endpoint, e.g. http://roboco-minio:9000. Empty = disabled "
|
||||
"(FileResponse fallback)."
|
||||
),
|
||||
)
|
||||
minio_access_key: str = Field(
|
||||
default="",
|
||||
description="MinIO access key. Required when minio_endpoint is set.",
|
||||
)
|
||||
minio_secret_key: str = Field(
|
||||
default="",
|
||||
description="MinIO secret key. Required when minio_endpoint is set.",
|
||||
)
|
||||
minio_bucket: str = Field(
|
||||
default="roboco-video-renders",
|
||||
description="MinIO bucket for rendered videos.",
|
||||
)
|
||||
minio_region: str = Field(
|
||||
default="us-east-1",
|
||||
description="MinIO region.",
|
||||
)
|
||||
|
||||
# Board roadmap engine — weekly, the Product Owner explores the company's
|
||||
# projects and proposes a themed cycle of roadmap items; the CEO approves
|
||||
|
||||
@@ -13,6 +13,7 @@ on are pure.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import subprocess
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
@@ -87,7 +88,11 @@ class ConventionsService(BaseService):
|
||||
) -> ConventionsStandard:
|
||||
"""Return the effective standard for ``project`` at its current HEAD."""
|
||||
pid = self._pid(project)
|
||||
root, head = self._resolve(project, workspace)
|
||||
# _resolve runs `git rev-parse` and _read_committed_standard/_derive
|
||||
# walk the filesystem + parse yaml — all sync I/O. Offload to a thread
|
||||
# so the shared API event loop stays responsive during conventions reads
|
||||
# (reachable from GET /api/projects/{id}/conventions and the spawn path).
|
||||
root, head = await asyncio.to_thread(self._resolve, project, workspace)
|
||||
cached = await self._cache_get(pid, head)
|
||||
# A cached ``degraded`` row is not trusted: a degraded file may have
|
||||
# been repaired in place at the same (stale) head key, and serving the
|
||||
@@ -95,7 +100,9 @@ class ConventionsService(BaseService):
|
||||
if cached is not None and cached.status != "degraded":
|
||||
return ConventionsStandard.model_validate(cached.effective_map)
|
||||
|
||||
file_standard, status = self._read_committed_standard(root)
|
||||
file_standard, status = await asyncio.to_thread(
|
||||
self._read_committed_standard, root
|
||||
)
|
||||
if status == "degraded":
|
||||
last_good = await self._latest_ok_map(pid)
|
||||
if last_good is not None:
|
||||
@@ -103,7 +110,9 @@ class ConventionsService(BaseService):
|
||||
# repaired in place), so never pin it — re-derive next call.
|
||||
return last_good
|
||||
|
||||
mapping = effective_map(self._derive(root), file_standard)
|
||||
mapping = effective_map(
|
||||
await asyncio.to_thread(self._derive, root), file_standard
|
||||
)
|
||||
if status != "degraded":
|
||||
await self._cache_put(pid, head, mapping, status)
|
||||
return mapping
|
||||
@@ -185,8 +194,8 @@ class ConventionsService(BaseService):
|
||||
if last_good is not None:
|
||||
mapping = last_good
|
||||
else:
|
||||
root, _ = self._resolve(project, workspace)
|
||||
mapping = self._derive(root)
|
||||
root, _ = await asyncio.to_thread(self._resolve, project, workspace)
|
||||
mapping = await asyncio.to_thread(self._derive, root)
|
||||
return await self._publish(
|
||||
project, render_yaml(mapping), restore=True, workspace=workspace
|
||||
)
|
||||
@@ -215,11 +224,13 @@ class ConventionsService(BaseService):
|
||||
resolvable workspace at all.
|
||||
"""
|
||||
pid = self._pid(project)
|
||||
root, head = self._resolve(project, workspace)
|
||||
root, head = await asyncio.to_thread(self._resolve, project, workspace)
|
||||
if root is None:
|
||||
status = "unknown"
|
||||
else:
|
||||
_file_standard, status = self._read_committed_standard(root)
|
||||
_file_standard, status = await asyncio.to_thread(
|
||||
self._read_committed_standard, root
|
||||
)
|
||||
last_ok = await self._latest_ok_row(pid)
|
||||
return ConventionsHealth(
|
||||
status=status,
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
"""MinIO storage client for rendered MP4s.
|
||||
|
||||
A thin wrapper around a singleton `minio.Minio` built from settings. The client
|
||||
is sync (minio-py is sync); every call site wraps the call in
|
||||
`asyncio.to_thread` — same pattern as `remotion_client._save`.
|
||||
|
||||
Unconfigured guard: when `settings.minio_endpoint` is empty, `get_client()`
|
||||
returns `None`. The write/serve paths (chunks 3/4) check `get_client()` and
|
||||
fall back to the local-disk path, so MinIO is opt-in via a single env var.
|
||||
`put_object` is defensive on the same guard (no-ops when unconfigured) so a
|
||||
caller that forgets the guard cannot crash; `get_object_stream` assumes the
|
||||
caller checked (the route's fallback catches `S3Error` for the not-found case).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
from typing import TYPE_CHECKING
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from minio import Minio
|
||||
from minio.error import S3Error # re-exported for callers (the serve route)
|
||||
|
||||
from roboco.config import settings
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterator
|
||||
|
||||
_client: Minio | None = None
|
||||
_initialised = False
|
||||
|
||||
|
||||
def _reset_client() -> None:
|
||||
"""Test-only: drop the cached singleton so the next `get_client()` rebuilds."""
|
||||
global _client, _initialised # noqa: PLW0603 - singleton cache, by design
|
||||
_client = None
|
||||
_initialised = False
|
||||
|
||||
|
||||
def get_client() -> Minio | None:
|
||||
"""Return the singleton `Minio`, or `None` when MinIO is unconfigured.
|
||||
|
||||
`None` is the disabled path: an empty `settings.minio_endpoint` means the
|
||||
write/serve paths fall back to local disk. Settings are load-time, so a
|
||||
plain module-level singleton is fine (no runtime-config drift to guard).
|
||||
"""
|
||||
global _client, _initialised # noqa: PLW0603 - singleton cache, by design
|
||||
if _initialised:
|
||||
return _client
|
||||
_initialised = True
|
||||
endpoint = settings.minio_endpoint.strip()
|
||||
if not endpoint:
|
||||
_client = None
|
||||
return None
|
||||
parsed = urlparse(endpoint if "://" in endpoint else f"//{endpoint}")
|
||||
secure = parsed.scheme == "https"
|
||||
host = parsed.netloc or endpoint # no scheme → use as-is
|
||||
_client = Minio(
|
||||
endpoint=host,
|
||||
access_key=settings.minio_access_key or None,
|
||||
secret_key=settings.minio_secret_key or None,
|
||||
secure=secure,
|
||||
region=settings.minio_region or None,
|
||||
)
|
||||
return _client
|
||||
|
||||
|
||||
def put_object(data: bytes, key: str) -> None:
|
||||
"""PUT `data` to `settings.minio_bucket` under `key` as video/mp4.
|
||||
|
||||
No-ops when MinIO is unconfigured (`get_client()` is `None`). The write path
|
||||
is guarded by `settings.minio_endpoint` upstream anyway; this is defensive
|
||||
so a caller that forgets the guard cannot crash.
|
||||
"""
|
||||
client = get_client()
|
||||
if client is None:
|
||||
return
|
||||
client.put_object(
|
||||
bucket_name=settings.minio_bucket,
|
||||
object_name=key,
|
||||
data=io.BytesIO(data),
|
||||
length=len(data),
|
||||
content_type="video/mp4",
|
||||
)
|
||||
|
||||
|
||||
def stat_object(key: str) -> None:
|
||||
"""Eager existence/readiness probe — raises if the object is missing or
|
||||
MinIO is down, so the serve route can fall back to ``FileResponse`` BEFORE
|
||||
starting a ``StreamingResponse`` it can no longer take back.
|
||||
|
||||
``get_object_stream`` is a lazy generator: its ``client.get_object`` call
|
||||
runs on the first ``next()``, i.e. after the route has returned and
|
||||
Starlette has started streaming — an ``S3Error`` there is uncatchable. This
|
||||
probe runs eagerly inside the route's ``try/except`` so the fallback
|
||||
actually fires. No-op when unconfigured (the route checks ``get_client()``
|
||||
first; this is defensive).
|
||||
|
||||
ponytail: stat-then-get is two round trips; a mid-stream failure after a
|
||||
successful stat is a rare race (object deleted / MinIO blips between the
|
||||
two calls) the CEO can retry — accept, or merge into one eager get_object
|
||||
returning the open response for a single round trip if preview latency
|
||||
ever matters.
|
||||
"""
|
||||
client = get_client()
|
||||
if client is None:
|
||||
return
|
||||
client.stat_object(bucket_name=settings.minio_bucket, object_name=key)
|
||||
|
||||
|
||||
def get_object_stream(key: str) -> Iterator[bytes]:
|
||||
"""Yield object bytes from `settings.minio_bucket`/`key` for `StreamingResponse`.
|
||||
|
||||
Assumes `get_client()` is not `None` (the route checks first). Lets
|
||||
`S3Error` propagate for the not-found / connection-refused case so the
|
||||
serve route's `try/except` can fall back to `FileResponse` from disk.
|
||||
"""
|
||||
client = get_client()
|
||||
if client is None: # defensive: caller should have checked
|
||||
raise RuntimeError("MinIO client is unconfigured (minio_endpoint empty)")
|
||||
response = client.get_object(bucket_name=settings.minio_bucket, object_name=key)
|
||||
try:
|
||||
yield from response.stream(amt=2**16)
|
||||
finally:
|
||||
response.close()
|
||||
response.release_conn()
|
||||
|
||||
|
||||
__all__ = ["S3Error", "get_client", "get_object_stream", "put_object", "stat_object"]
|
||||
@@ -22,6 +22,7 @@ Correctness is deterministic: the readiness audit lives in code
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Awaitable, Callable
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
@@ -227,7 +228,12 @@ class ReleaseManagerEngine(BaseService):
|
||||
)
|
||||
conclusion = (ci or {}).get("conclusion")
|
||||
today = datetime.now(UTC).strftime("%Y-%m-%d")
|
||||
snapshot = gather_snapshot(Path(root), master_ci_conclusion=conclusion)
|
||||
# gather_snapshot runs multiple sync `subprocess.run` git calls + a
|
||||
# filesystem walk; offload so the shared API event loop isn't blocked
|
||||
# while the release-manager background loop assesses.
|
||||
snapshot = await asyncio.to_thread(
|
||||
gather_snapshot, Path(root), master_ci_conclusion=conclusion
|
||||
)
|
||||
return assess(snapshot, today=today)
|
||||
|
||||
|
||||
|
||||
@@ -22,8 +22,12 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import structlog
|
||||
|
||||
from roboco.config import settings
|
||||
from roboco.services import minio_client
|
||||
|
||||
log = structlog.get_logger(__name__)
|
||||
|
||||
|
||||
class RemotionRendererError(Exception):
|
||||
@@ -136,11 +140,33 @@ class RemotionRenderer:
|
||||
|
||||
@staticmethod
|
||||
def _save(mp4_bytes: bytes, *, render_key: str, orientation: str) -> str:
|
||||
"""Write MP4 bytes under video_output_dir at a task-scoped path."""
|
||||
"""Write MP4 bytes under video_output_dir at a task-scoped path.
|
||||
|
||||
Durable copy to MinIO when configured. Local disk stays the source of
|
||||
truth for the poster publish path (x_video_client/tiktok_client read
|
||||
mp4_path from disk), so this is an additive PUT, not a replacement.
|
||||
Key = basename, already ``{render_key}-{orientation}.mp4`` — no
|
||||
schema/marker change. ``_save`` is wrapped in ``asyncio.to_thread`` by
|
||||
``render()``, so the sync ``put_object`` call runs in that thread.
|
||||
"""
|
||||
out_dir = Path(settings.video_output_dir)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
path = out_dir / f"{render_key}-{orientation}.mp4"
|
||||
path.write_bytes(mp4_bytes)
|
||||
if minio_client.get_client() is not None:
|
||||
# MinIO is a durable COPY, not the render's source of truth — local
|
||||
# disk is. The serve route falls back to FileResponse on S3Error, so
|
||||
# a failed PUT (MinIO down, full disk, transient 5xx) must never fail
|
||||
# the render or it'd retry-loop a task whose local file is already
|
||||
# fine. Log and continue; the next render re-attempts the PUT.
|
||||
try:
|
||||
minio_client.put_object(mp4_bytes, path.name)
|
||||
except Exception as exc: # durable copy, never fatal to the render
|
||||
log.warning(
|
||||
"minio put failed; render kept on local disk",
|
||||
key=path.name,
|
||||
error=str(exc),
|
||||
)
|
||||
return str(path)
|
||||
|
||||
|
||||
|
||||
@@ -5,6 +5,7 @@ sub-router. CEO-only throughout."""
|
||||
from __future__ import annotations
|
||||
|
||||
from http import HTTPStatus
|
||||
from types import SimpleNamespace
|
||||
from typing import TYPE_CHECKING
|
||||
from unittest.mock import AsyncMock, patch
|
||||
from uuid import UUID, uuid4
|
||||
@@ -14,6 +15,7 @@ import pytest_asyncio
|
||||
from fastapi import FastAPI
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from roboco.api.deps import get_agent_context, get_db
|
||||
from roboco.api.routes import video as video_module
|
||||
from roboco.api.routes.video import router as video_router
|
||||
from roboco.api.routes.video import tiktok_router
|
||||
from roboco.config import settings as cfg
|
||||
@@ -23,6 +25,7 @@ from roboco.foundation.policy.content import markers
|
||||
from roboco.models import AgentRole, AgentStatus, Team
|
||||
from roboco.models.base import Complexity, TaskNature, TaskStatus, TaskType
|
||||
from roboco.models.permissions import AgentContext
|
||||
from roboco.services import minio_client
|
||||
from roboco.services.heartbeat_mutex import HeartbeatMutex
|
||||
from roboco.services.task import VIDEO_POST_SOURCE, VIDEO_SOURCE, get_task_service
|
||||
from roboco.services.tiktok_credentials import get_tiktok_credentials_service
|
||||
@@ -543,3 +546,150 @@ async def test_non_ceo_is_forbidden(db_session: AsyncSession) -> None:
|
||||
assert media_resp.status_code == HTTPStatus.FORBIDDEN
|
||||
assert creds_resp.status_code == HTTPStatus.FORBIDDEN
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
# --- MinIO serve path (chunk 4) — unit-style, no DB / no real MinIO ------------
|
||||
# These two tests monkeypatch ``get_task_service`` in the video routes module
|
||||
# so they run without postgres (the ``db_session``-based tests above are
|
||||
# skipped when Postgres is unreachable). Mocks only — no testcontainers.
|
||||
|
||||
|
||||
def _stub_task_service_factory(task: object) -> object:
|
||||
"""A ``get_task_service``-shaped stub (the real one is a sync factory
|
||||
returning a service with an async ``.get``). Patched in place of
|
||||
``video_module.get_task_service`` so the route runs without postgres."""
|
||||
|
||||
class _Svc:
|
||||
async def get(self, _task_id: UUID) -> object:
|
||||
return task
|
||||
|
||||
return _Svc()
|
||||
|
||||
|
||||
def _make_task(mp4_path: str, task_id: UUID) -> SimpleNamespace:
|
||||
"""A minimal task-shaped stub carrying the video_draft marker the route
|
||||
reads — enough for the media route, no DB row needed."""
|
||||
return SimpleNamespace(
|
||||
id=task_id,
|
||||
source=VIDEO_POST_SOURCE,
|
||||
orchestration_markers={"video_draft": {"mp4_paths": {"vertical": mp4_path}}},
|
||||
)
|
||||
|
||||
|
||||
def _patch_task_service(monkeypatch: pytest.MonkeyPatch, task: object) -> None:
|
||||
monkeypatch.setattr(
|
||||
video_module,
|
||||
"get_task_service",
|
||||
lambda _db: _stub_task_service_factory(task),
|
||||
)
|
||||
|
||||
|
||||
def _minio_stream(_key: str) -> object:
|
||||
"""Stub ``get_object_stream`` yielding fixed bytes for ``StreamingResponse``."""
|
||||
return iter([b"minio-stream-bytes"])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_media_serves_from_minio_when_configured(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Configured serve path: when MinIO is configured, the media route streams
|
||||
the object via ``minio_client.get_object_stream`` (key = basename) and the
|
||||
panel-preview URL/headers stay identical. ``_require_ceo`` still 403s a
|
||||
non-CEO agent. No DB / no real MinIO — ``get_task_service`` is stubbed so
|
||||
the route runs without postgres."""
|
||||
# A real local file so the route's is_file() + confinement checks pass.
|
||||
# The served bytes come from the stubbed MinIO stream below, NOT this
|
||||
# file — that's what proves the MinIO path was taken rather than the
|
||||
# FileResponse fallback.
|
||||
monkeypatch.setattr(cfg, "video_output_dir", str(tmp_path))
|
||||
vertical = tmp_path / "clip-vertical.mp4"
|
||||
vertical.write_bytes(b"local-file-bytes")
|
||||
task_id = uuid4()
|
||||
_patch_task_service(monkeypatch, _make_task(str(vertical), task_id))
|
||||
# non-None sentinel so the route takes the MinIO branch.
|
||||
monkeypatch.setattr(minio_client, "get_client", lambda: True)
|
||||
# The route probes stat_object eagerly before streaming; stub it to pass.
|
||||
monkeypatch.setattr(minio_client, "stat_object", lambda _key: None)
|
||||
monkeypatch.setattr(minio_client, "get_object_stream", _minio_stream)
|
||||
|
||||
# CEO 200 — streamed from MinIO.
|
||||
app = _build_app(None, AgentRole.CEO, uuid4()) # type: ignore[arg-type]
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
resp = await client.get(f"/api/video/posts/{task_id}/media?cut=vertical")
|
||||
assert resp.status_code == HTTPStatus.OK
|
||||
assert resp.headers["content-type"] == "video/mp4"
|
||||
assert resp.content == b"minio-stream-bytes"
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
# Non-CEO 403 — _require_ceo still gates end-to-end (no presigned URL).
|
||||
app = _build_app(None, AgentRole.DEVELOPER, uuid4()) # type: ignore[arg-type]
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
resp = await client.get(f"/api/video/posts/{task_id}/media?cut=vertical")
|
||||
assert resp.status_code == HTTPStatus.FORBIDDEN
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_media_falls_back_to_local_file_when_minio_unconfigured(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Unconfigured fallback: with ``get_client`` returning None
|
||||
(``minio_endpoint`` empty), the media route serves the local file via
|
||||
``FileResponse`` — the body equals the local file's bytes. No DB / no
|
||||
real MinIO."""
|
||||
monkeypatch.setattr(cfg, "video_output_dir", str(tmp_path))
|
||||
vertical = tmp_path / "clip-vertical.mp4"
|
||||
vertical.write_bytes(b"local-file-bytes")
|
||||
task_id = uuid4()
|
||||
_patch_task_service(monkeypatch, _make_task(str(vertical), task_id))
|
||||
monkeypatch.setattr(minio_client, "get_client", lambda: None)
|
||||
|
||||
app = _build_app(None, AgentRole.CEO, uuid4()) # type: ignore[arg-type]
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
resp = await client.get(f"/api/video/posts/{task_id}/media?cut=vertical")
|
||||
assert resp.status_code == HTTPStatus.OK
|
||||
assert resp.headers["content-type"] == "video/mp4"
|
||||
assert resp.content == b"local-file-bytes"
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_media_falls_back_to_local_file_when_minio_missing(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""S3Error fallback: when MinIO is configured but the object is missing
|
||||
(NoSuchKey — an old render not yet in MinIO) or MinIO is down, the route's
|
||||
eager ``stat_object`` probe raises, the ``try/except`` catches it, and the
|
||||
route serves the local file via ``FileResponse``. ``get_object_stream`` is
|
||||
never called. No DB / no real MinIO."""
|
||||
monkeypatch.setattr(cfg, "video_output_dir", str(tmp_path))
|
||||
vertical = tmp_path / "clip-vertical.mp4"
|
||||
vertical.write_bytes(b"local-file-bytes")
|
||||
task_id = uuid4()
|
||||
_patch_task_service(monkeypatch, _make_task(str(vertical), task_id))
|
||||
monkeypatch.setattr(minio_client, "get_client", lambda: True)
|
||||
|
||||
def _stat_raises(_key: str) -> None:
|
||||
raise RuntimeError("minio NoSuchKey / down")
|
||||
|
||||
monkeypatch.setattr(minio_client, "stat_object", _stat_raises)
|
||||
|
||||
# If the route wrongly takes the MinIO stream branch, this would be called
|
||||
# and the assertion below would fail — guard against a regression.
|
||||
def _stream_must_not_be_called(_key: str) -> object:
|
||||
pytest.fail("get_object_stream must not be called when stat_object raises")
|
||||
|
||||
monkeypatch.setattr(minio_client, "get_object_stream", _stream_must_not_be_called)
|
||||
|
||||
app = _build_app(None, AgentRole.CEO, uuid4()) # type: ignore[arg-type]
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
resp = await client.get(f"/api/video/posts/{task_id}/media?cut=vertical")
|
||||
assert resp.status_code == HTTPStatus.OK
|
||||
assert resp.headers["content-type"] == "video/mp4"
|
||||
assert resp.content == b"local-file-bytes" # FileResponse fallback, not MinIO
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
"""MinIO client coverage: unconfigured guard + endpoint scheme parsing (mocks)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import pytest
|
||||
from roboco.config import settings as cfg
|
||||
from roboco.services import minio_client
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterator
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_singleton() -> Iterator[None]:
|
||||
"""Each test rebuilds the singleton (the test-isolation hazard)."""
|
||||
minio_client._reset_client()
|
||||
yield
|
||||
minio_client._reset_client()
|
||||
|
||||
|
||||
def test_get_client_returns_none_when_unconfigured(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr(cfg, "minio_endpoint", "")
|
||||
monkeypatch.setattr(cfg, "minio_access_key", "")
|
||||
monkeypatch.setattr(cfg, "minio_secret_key", "")
|
||||
assert minio_client.get_client() is None
|
||||
|
||||
|
||||
def test_get_client_parses_endpoint_scheme(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""http://roboco-minio:9000 → endpoint=roboco-minio:9000, secure=False."""
|
||||
built: dict[str, Any] = {}
|
||||
|
||||
class FakeMinio:
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
endpoint: str,
|
||||
access_key: str | None,
|
||||
secret_key: str | None,
|
||||
secure: bool,
|
||||
region: str | None,
|
||||
) -> None:
|
||||
built["endpoint"] = endpoint
|
||||
built["access_key"] = access_key
|
||||
built["secret_key"] = secret_key
|
||||
built["secure"] = secure
|
||||
built["region"] = region
|
||||
|
||||
monkeypatch.setattr(minio_client, "Minio", FakeMinio)
|
||||
monkeypatch.setattr(cfg, "minio_endpoint", "http://roboco-minio:9000")
|
||||
monkeypatch.setattr(cfg, "minio_access_key", "minio")
|
||||
monkeypatch.setattr(cfg, "minio_secret_key", "minio123")
|
||||
monkeypatch.setattr(cfg, "minio_region", "us-east-1")
|
||||
|
||||
minio_client.get_client()
|
||||
assert built["endpoint"] == "roboco-minio:9000"
|
||||
assert built["secure"] is False
|
||||
assert built["access_key"] == "minio"
|
||||
assert built["secret_key"] == "minio123"
|
||||
assert built["region"] == "us-east-1"
|
||||
|
||||
|
||||
def test_get_client_parses_https_endpoint(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
built: dict[str, Any] = {}
|
||||
|
||||
class FakeMinio:
|
||||
def __init__(self, **kwargs: Any) -> None:
|
||||
built["endpoint"] = kwargs["endpoint"]
|
||||
built["secure"] = kwargs["secure"]
|
||||
|
||||
monkeypatch.setattr(minio_client, "Minio", FakeMinio)
|
||||
monkeypatch.setattr(cfg, "minio_endpoint", "https://minio.example:9000")
|
||||
minio_client._reset_client()
|
||||
minio_client.get_client()
|
||||
assert built["endpoint"] == "minio.example:9000"
|
||||
assert built["secure"] is True
|
||||
|
||||
|
||||
def test_put_object_noops_when_unconfigured(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr(cfg, "minio_endpoint", "")
|
||||
minio_client._reset_client()
|
||||
# Must not raise — the write path's upstream guard is the source of truth,
|
||||
# but this stays defensive.
|
||||
minio_client.put_object(b"bytes", "key.mp4")
|
||||
|
||||
|
||||
def test_get_object_stream_raises_when_unconfigured(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr(cfg, "minio_endpoint", "")
|
||||
minio_client._reset_client()
|
||||
with pytest.raises(RuntimeError, match="unconfigured"):
|
||||
next(minio_client.get_object_stream("key.mp4"))
|
||||
|
||||
|
||||
def test_get_client_singleton_cached(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr(cfg, "minio_endpoint", "http://roboco-minio:9000")
|
||||
minio_client._reset_client()
|
||||
a = minio_client.get_client()
|
||||
b = minio_client.get_client()
|
||||
assert a is b
|
||||
@@ -9,6 +9,7 @@ from pathlib import Path
|
||||
import httpx
|
||||
import pytest
|
||||
from roboco.config import settings as cfg
|
||||
from roboco.services import minio_client
|
||||
from roboco.services.remotion_client import (
|
||||
NullRemotionRenderer,
|
||||
RemotionRenderer,
|
||||
@@ -166,3 +167,90 @@ def test_get_remotion_renderer_returns_real_client_when_set(
|
||||
renderer = get_remotion_renderer()
|
||||
assert isinstance(renderer, RemotionRenderer)
|
||||
assert not isinstance(renderer, NullRemotionRenderer)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_save_puts_to_minio_when_configured(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""When MinIO is configured, _save PUTs the bytes under the basename key
|
||||
and still writes the local mp4 file. Mocks only — no real MinIO."""
|
||||
source = _make_source(tmp_path)
|
||||
out_dir = tmp_path / "out"
|
||||
monkeypatch.setattr(cfg, "video_output_dir", str(out_dir))
|
||||
monkeypatch.setattr(cfg, "video_request_timeout_seconds", 5.0)
|
||||
monkeypatch.setattr(cfg, "video_render_timeout_seconds", 30.0)
|
||||
|
||||
minio_client._reset_client()
|
||||
# Sentinel client so get_client() returns a non-None (the guard passes).
|
||||
monkeypatch.setattr(minio_client, "get_client", object)
|
||||
put_calls: list[tuple[bytes, str]] = []
|
||||
|
||||
def fake_put_object(data: bytes, key: str) -> None:
|
||||
put_calls.append((data, key))
|
||||
|
||||
monkeypatch.setattr(minio_client, "put_object", fake_put_object)
|
||||
|
||||
def handler(_request: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(200, content=b"fake-mp4-bytes")
|
||||
|
||||
transport = httpx.MockTransport(handler)
|
||||
http_client = httpx.AsyncClient(transport=transport)
|
||||
renderer = RemotionRenderer(base_url="http://fake-remotion", client=http_client)
|
||||
|
||||
try:
|
||||
path = await renderer.render(
|
||||
source_dir=str(source),
|
||||
composition_id="Intro",
|
||||
input_props={"title": "hello"},
|
||||
orientation="vertical",
|
||||
render_key="task-77",
|
||||
)
|
||||
finally:
|
||||
await http_client.aclose()
|
||||
minio_client._reset_client()
|
||||
|
||||
# Local file still written (the poster publish path reads from disk).
|
||||
saved = Path(path)
|
||||
assert saved.exists()
|
||||
assert saved.read_bytes() == b"fake-mp4-bytes"
|
||||
assert saved.name == "task-77-vertical.mp4"
|
||||
|
||||
# One PUT, key = the basename, body = the rendered bytes (reused, not re-read).
|
||||
assert len(put_calls) == 1
|
||||
data, key = put_calls[0]
|
||||
assert key == "task-77-vertical.mp4"
|
||||
assert data == b"fake-mp4-bytes"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_save_swallows_minio_put_failure(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""A failed MinIO PUT (MinIO down, transient 5xx) must not fail the render —
|
||||
local disk is the source of truth and the serve route falls back to
|
||||
FileResponse on S3Error. _save logs and returns the local path; the local
|
||||
file is written. Mocks only."""
|
||||
out_dir = tmp_path / "out"
|
||||
monkeypatch.setattr(cfg, "video_output_dir", str(out_dir))
|
||||
|
||||
minio_client._reset_client()
|
||||
monkeypatch.setattr(minio_client, "get_client", object) # guard passes
|
||||
|
||||
def failing_put_object(_data: bytes, _key: str) -> None:
|
||||
raise RuntimeError("minio unreachable")
|
||||
|
||||
monkeypatch.setattr(minio_client, "put_object", failing_put_object)
|
||||
|
||||
try:
|
||||
# _save is a sync @staticmethod; call it directly (no httpx needed).
|
||||
path = RemotionRenderer._save(
|
||||
b"fake-mp4-bytes", render_key="task-88", orientation="square"
|
||||
)
|
||||
finally:
|
||||
minio_client._reset_client()
|
||||
|
||||
saved = Path(path)
|
||||
assert saved.exists()
|
||||
assert saved.read_bytes() == b"fake-mp4-bytes"
|
||||
assert saved.name == "task-88-square.mp4"
|
||||
|
||||
@@ -1611,6 +1611,22 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "minio"
|
||||
version = "7.2.20"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "argon2-cffi" },
|
||||
{ name = "certifi" },
|
||||
{ name = "pycryptodome" },
|
||||
{ name = "typing-extensions" },
|
||||
{ name = "urllib3" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/40/df/6dfc6540f96a74125a11653cce717603fd5b7d0001a8e847b3e54e72d238/minio-7.2.20.tar.gz", hash = "sha256:95898b7a023fbbfde375985aa77e2cd6a0762268db79cf886f002a9ea8e68598", size = 136113, upload-time = "2025-11-27T00:37:15.569Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/3e/9a/b697530a882588a84db616580f2ba5d1d515c815e11c30d219145afeec87/minio-7.2.20-py3-none-any.whl", hash = "sha256:eb33dd2fb80e04c3726a76b13241c6be3c4c46f8d81e1d58e757786f6501897e", size = 93751, upload-time = "2025-11-27T00:37:13.993Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "msgpack"
|
||||
version = "1.2.1"
|
||||
@@ -2115,6 +2131,36 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/0c/c3/44f3fbbfa403ea2a7c779186dc20772604442dde72947e7d01069cbe98e3/pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992", size = 48172, upload-time = "2026-01-21T14:26:50.693Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pycryptodome"
|
||||
version = "3.23.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/8e/a6/8452177684d5e906854776276ddd34eca30d1b1e15aa1ee9cefc289a33f5/pycryptodome-3.23.0.tar.gz", hash = "sha256:447700a657182d60338bab09fdb27518f8856aecd80ae4c6bdddb67ff5da44ef", size = 4921276, upload-time = "2025-05-17T17:21:45.242Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/04/5d/bdb09489b63cd34a976cc9e2a8d938114f7a53a74d3dd4f125ffa49dce82/pycryptodome-3.23.0-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:0011f7f00cdb74879142011f95133274741778abba114ceca229adbf8e62c3e4", size = 2495152, upload-time = "2025-05-17T17:20:20.833Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/a7/ce/7840250ed4cc0039c433cd41715536f926d6e86ce84e904068eb3244b6a6/pycryptodome-3.23.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:90460fc9e088ce095f9ee8356722d4f10f86e5be06e2354230a9880b9c549aae", size = 1639348, upload-time = "2025-05-17T17:20:23.171Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ee/f0/991da24c55c1f688d6a3b5a11940567353f74590734ee4a64294834ae472/pycryptodome-3.23.0-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4764e64b269fc83b00f682c47443c2e6e85b18273712b98aa43bcb77f8570477", size = 2184033, upload-time = "2025-05-17T17:20:25.424Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/54/16/0e11882deddf00f68b68dd4e8e442ddc30641f31afeb2bc25588124ac8de/pycryptodome-3.23.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:eb8f24adb74984aa0e5d07a2368ad95276cf38051fe2dc6605cbcf482e04f2a7", size = 2270142, upload-time = "2025-05-17T17:20:27.808Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d5/fc/4347fea23a3f95ffb931f383ff28b3f7b1fe868739182cb76718c0da86a1/pycryptodome-3.23.0-cp313-cp313t-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:d97618c9c6684a97ef7637ba43bdf6663a2e2e77efe0f863cce97a76af396446", size = 2309384, upload-time = "2025-05-17T17:20:30.765Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/6e/d9/c5261780b69ce66d8cfab25d2797bd6e82ba0241804694cd48be41add5eb/pycryptodome-3.23.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:9a53a4fe5cb075075d515797d6ce2f56772ea7e6a1e5e4b96cf78a14bac3d265", size = 2183237, upload-time = "2025-05-17T17:20:33.736Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/5a/6f/3af2ffedd5cfa08c631f89452c6648c4d779e7772dfc388c77c920ca6bbf/pycryptodome-3.23.0-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:763d1d74f56f031788e5d307029caef067febf890cd1f8bf61183ae142f1a77b", size = 2343898, upload-time = "2025-05-17T17:20:36.086Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/9a/dc/9060d807039ee5de6e2f260f72f3d70ac213993a804f5e67e0a73a56dd2f/pycryptodome-3.23.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:954af0e2bd7cea83ce72243b14e4fb518b18f0c1649b576d114973e2073b273d", size = 2269197, upload-time = "2025-05-17T17:20:38.414Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/f9/34/e6c8ca177cb29dcc4967fef73f5de445912f93bd0343c9c33c8e5bf8cde8/pycryptodome-3.23.0-cp313-cp313t-win32.whl", hash = "sha256:257bb3572c63ad8ba40b89f6fc9d63a2a628e9f9708d31ee26560925ebe0210a", size = 1768600, upload-time = "2025-05-17T17:20:40.688Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e4/1d/89756b8d7ff623ad0160f4539da571d1f594d21ee6d68be130a6eccb39a4/pycryptodome-3.23.0-cp313-cp313t-win_amd64.whl", hash = "sha256:6501790c5b62a29fcb227bd6b62012181d886a767ce9ed03b303d1f22eb5c625", size = 1799740, upload-time = "2025-05-17T17:20:42.413Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/5d/61/35a64f0feaea9fd07f0d91209e7be91726eb48c0f1bfc6720647194071e4/pycryptodome-3.23.0-cp313-cp313t-win_arm64.whl", hash = "sha256:9a77627a330ab23ca43b48b130e202582e91cc69619947840ea4d2d1be21eb39", size = 1703685, upload-time = "2025-05-17T17:20:44.388Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/db/6c/a1f71542c969912bb0e106f64f60a56cc1f0fabecf9396f45accbe63fa68/pycryptodome-3.23.0-cp37-abi3-macosx_10_9_universal2.whl", hash = "sha256:187058ab80b3281b1de11c2e6842a357a1f71b42cb1e15bce373f3d238135c27", size = 2495627, upload-time = "2025-05-17T17:20:47.139Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/6e/4e/a066527e079fc5002390c8acdd3aca431e6ea0a50ffd7201551175b47323/pycryptodome-3.23.0-cp37-abi3-macosx_10_9_x86_64.whl", hash = "sha256:cfb5cd445280c5b0a4e6187a7ce8de5a07b5f3f897f235caa11f1f435f182843", size = 1640362, upload-time = "2025-05-17T17:20:50.392Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/50/52/adaf4c8c100a8c49d2bd058e5b551f73dfd8cb89eb4911e25a0c469b6b4e/pycryptodome-3.23.0-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:67bd81fcbe34f43ad9422ee8fd4843c8e7198dd88dd3d40e6de42ee65fbe1490", size = 2182625, upload-time = "2025-05-17T17:20:52.866Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/5f/e9/a09476d436d0ff1402ac3867d933c61805ec2326c6ea557aeeac3825604e/pycryptodome-3.23.0-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c8987bd3307a39bc03df5c8e0e3d8be0c4c3518b7f044b0f4c15d1aa78f52575", size = 2268954, upload-time = "2025-05-17T17:20:55.027Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/f9/c5/ffe6474e0c551d54cab931918127c46d70cab8f114e0c2b5a3c071c2f484/pycryptodome-3.23.0-cp37-abi3-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:aa0698f65e5b570426fc31b8162ed4603b0c2841cbb9088e2b01641e3065915b", size = 2308534, upload-time = "2025-05-17T17:20:57.279Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/18/28/e199677fc15ecf43010f2463fde4c1a53015d1fe95fb03bca2890836603a/pycryptodome-3.23.0-cp37-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:53ecbafc2b55353edcebd64bf5da94a2a2cdf5090a6915bcca6eca6cc452585a", size = 2181853, upload-time = "2025-05-17T17:20:59.322Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ce/ea/4fdb09f2165ce1365c9eaefef36625583371ee514db58dc9b65d3a255c4c/pycryptodome-3.23.0-cp37-abi3-musllinux_1_2_i686.whl", hash = "sha256:156df9667ad9f2ad26255926524e1c136d6664b741547deb0a86a9acf5ea631f", size = 2342465, upload-time = "2025-05-17T17:21:03.83Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/22/82/6edc3fc42fe9284aead511394bac167693fb2b0e0395b28b8bedaa07ef04/pycryptodome-3.23.0-cp37-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:dea827b4d55ee390dc89b2afe5927d4308a8b538ae91d9c6f7a5090f397af1aa", size = 2267414, upload-time = "2025-05-17T17:21:06.72Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/59/fe/aae679b64363eb78326c7fdc9d06ec3de18bac68be4b612fc1fe8902693c/pycryptodome-3.23.0-cp37-abi3-win32.whl", hash = "sha256:507dbead45474b62b2bbe318eb1c4c8ee641077532067fec9c1aa82c31f84886", size = 1768484, upload-time = "2025-05-17T17:21:08.535Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/54/2f/e97a1b8294db0daaa87012c24a7bb714147c7ade7656973fd6c736b484ff/pycryptodome-3.23.0-cp37-abi3-win_amd64.whl", hash = "sha256:c75b52aacc6c0c260f204cbdd834f76edc9fb0d8e0da9fbf8352ef58202564e2", size = 1799636, upload-time = "2025-05-17T17:21:10.393Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/18/3d/f9441a0d798bf2b1e645adc3265e55706aead1255ccdad3856dbdcffec14/pycryptodome-3.23.0-cp37-abi3-win_arm64.whl", hash = "sha256:11eeeb6917903876f134b56ba11abe95c0b0fd5e3330def218083c7d98bbcb3c", size = 1703675, upload-time = "2025-05-17T17:21:13.146Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pydantic"
|
||||
version = "2.13.4"
|
||||
@@ -2547,6 +2593,7 @@ dependencies = [
|
||||
{ name = "hiredis" },
|
||||
{ name = "httpx" },
|
||||
{ name = "mcp" },
|
||||
{ name = "minio" },
|
||||
{ name = "openai" },
|
||||
{ name = "packaging" },
|
||||
{ name = "passlib", extra = ["bcrypt"] },
|
||||
@@ -2625,6 +2672,7 @@ requires-dist = [
|
||||
{ name = "import-linter", marker = "extra == 'dev'" },
|
||||
{ name = "ipython", marker = "extra == 'dev'" },
|
||||
{ name = "mcp" },
|
||||
{ name = "minio" },
|
||||
{ name = "mypy", marker = "extra == 'dev'" },
|
||||
{ name = "openai" },
|
||||
{ name = "packaging" },
|
||||
|
||||
Reference in New Issue
Block a user