6 Commits

Include renames

Author	SHA1	Message	Date
3 people	4923ee3ff3	MinIO video storage (chunk 1: config+deps+compose) + event-loop perf fix (#308) ... * feat(video): Phase A — VideoEngine origination spine + held-source gates New default-off engine skeleton: opens a UX/UI authoring task (source=video, assigned to a ux-dev, LOW complexity to clear the dev-needs-subtasks guard) and materializes a held CEO-approval draft (source=video_post). Excludes video_post from all three held-source skip sites; adds the video_draft marker, six config flags, and the feature-flag entries. Origination + gate behavior unit-tested. * refactor(orchestrator): fold _dispatch_dev_work skip chain into a helper The per-source if/continue chain grew past xenon's --max-absolute B when the video_post held source joined it. Extract _is_non_dev_dispatch_source (every held-CEO source plus the two Board exploration sources) so the dev loop's skip is one flat call. Behavior-identical. * feat(video): Phase B — propose_video do-tool (metadata-only, team-gated) UX/UI dev records a video's composition ref + per-platform captions onto the authoring task's video_draft marker. Team-gated at runtime via _caller_team (Role.DEVELOPER can't tell a ux-dev from a be-dev). Resolves the caller's ACTIVE task via get_active_task_for_agent, not an oldest-first scan that would clobber a second open video task. Metadata only, no render. Wired through do_server + route + schema; added to _DEV_DO. * feat(video): Phase D — render loop + RemotionRenderer client Orchestrator-async _video_render_loop renders a completed authoring task's merged composition to MP4 (vertical + square) via the remotion-renderer sidecar and materializes the held video_post draft. RemotionRenderer tars the read-clone's motion/ source, POSTs it, and saves the returned MP4 bytes to a TASK-scoped local path (no shared volume; a composition is reused across videos so a composition-scoped path would clobber an earlier draft). Render failures bounded-retry (read-clone catch-up window, transient sidecar) up to a cap, then terminal-fail. Client tested vs a mock transport; loop vs a mock renderer + real DB. * feat(video): Phase C — release / spotlight / on-demand video triggers Three entry points open a UX/UI video-authoring task via VideoEngine.open_video_task: (1) a published release drafts a companion video — best-effort in ReleaseProposalService.approve, never fails the publish; script from the CHANGELOG via the local model with a template fallback. (2) propose_feature_spotlight gains optional wants_video/video_script — best-effort, gated on video_on_spotlight, default-off leaves the spotlight flow byte-for-byte unchanged. (3) POST /video/request (CEO-only) for an on-demand brief, with clean disabled/not_opened responses. All gated on video_engine_enabled. * fix(video): savepoint-isolate video-task inserts (F042 poisoned session) The best-effort try/except around open_video_task (release-publish + spotlight hooks) swallowed the Python exception, but a DBAPI error at the insert flush left the shared session must-rollback — so the caller's next commit (release finalize / request boundary) threw PendingRollbackError: the release stuck 'pending' after actually publishing, or the spotlight draft + HTTP response were lost. Wrap both inserts (open_video_task, _originate_video_post) in a begin_nested savepoint (the repo's established F042 pattern) so a DB error rolls back only the insert. open_video_task returns None (every caller already handles it); _originate_video_post propagates to the render loop's handler. Regression test: an insert FK error returns None with the session left usable. Dormant while the flags were off; armed on the NAS. * feat(video): Phase G — motion/ package + remotion-renderer sidecar + compose In-repo Remotion v4 motion/ package (ReleaseAnnouncement composition; calculateMetadata returns 1080x1920 vertical / 1080x1080 square from inputProps.orientation) + a credential-free remotion-renderer sidecar: untar the POSTed motion/ source, bundle (LRU-cached per source sha), selectComposition + renderMedia h264, stream the MP4 bytes back — matching the RemotionRenderer client contract. docker/remotion.Dockerfile on Debian (Chrome apt deps, build-time Chrome pre-warm, ffmpeg bundled in @remotion/renderer). Wired into both compose files (roboco_default only, shm_size 1gb, /health check) + the release publish matrix. Verified via a real local render of both cuts; the Debian docker build is the CEO's to run. * chore(video): D-hardening — video_post source_task_id + render-loop docstring Add a source_task_id back-reference to the video_post held-draft marker (traceability from a draft to its authoring task; also makes the render loop's two-key idempotency check wireable later). Fix the render-loop test's stale docstring ('never retried' -> bounded-retry). Both from the Phase D critic's non-blocking follow-ups. * feat(video): Phase E1 — VideoPostService + heartbeat mutex (approve->post) CEO-approve->post service: heartbeat-renewed Redis mutex (fail-closed, grace=ttl-2*heartbeat), re-read-in-lock double-post guard, per-platform durable commits (asyncio.shield-ed, settle-before-rollback on lock-loss), all writes inside the lock (captions validated pre-lock, applied in-lock — no stale whole-column clobber), idempotent, per-platform retry-skip. Poster interfaces (X/TikTok, mocked here). Reject + list-held-drafts. Survived 3 adversarial rounds; residual = a crash in the poster->commit window (CEO-gated low-freq, documented). * fix(video): G-hardening — renderer leaks + Share Tech Mono brand font Sidecar: give bundle() an explicit outDir tracked + deleted on LRU eviction (was leaking ~19MB remotion-webpack-bundle-* per source); res.on('close') cleanup so an aborted/retried download no longer leaks its remotion-out-* MP4 dir. Fonts: vendor Share Tech Mono (roboco-website brand font) as the display face (self-hosted woff2, 400-weight, headline fontWeight 700->400 to avoid faux-bold) + self-hosted Inter body — no gstatic fetch at render time (lsof-verified). Extras: composition_id whitelist (400) + Multer error middleware (400/413). * feat(video): Phase E2 — X v2 + TikTok posters, tiktok_credentials, routes LiveXVideoPoster (X v2 chunked media upload: init/append/finalize/STATUS-poll -> tweet w/ media_ids, OAuth1 signer reused). LiveTikTokPoster (OAuth2 inbox: init -> chunked PUT with asymmetric final chunk -> status-fetch; 401 -> refresh_token grant, rotated token persisted). tiktok_credentials Fernet singleton + migration 062 (single head). Routes: CEO approve/reject + list held drafts + write-only tiktok creds, wiring real posters into VideoPostService. Residual: a lock-loss right after a token-refresh flush can discard the rotated token (same rare CEO-gated class as the documented post->commit window). * feat(video): Phase F — panel video-post queue + TikTok creds card + flags video-post-queue.tsx: <video> MP4 preview with 9:16/1:1 cut switch, per-platform editable captions (280/2200 counters, over-limit disables approve), approve/reject, Request-a-video dialog. tiktok-credentials-card.tsx (4 write-only OAuth2 fields). feature-flags-card inlines TikTokCredentialsForm under video_engine_enabled. Mounted in command-center. tsc/eslint clean, 273 panel tests green. NOTE: needs the GET /video/posts/{id}/media route + mp4_paths on VideoPostResponse (folded into H) for the preview source. * feat(video): Phase H — media route + e2e smoke + NAS arming + docs GET /video/posts/{id}/media?cut= (CEO-gated FileResponse of the rendered MP4; closes the panel preview gap) + mp4_paths on VideoPostResponse. e2e smoke tests/e2e_smoke/test_video_pipeline.py (full flow, sidecar+X/TikTok mocked; asserts dispatcher skips, render-loop materialize, propose_video team-gate, approve idempotency). NAS arming: docker-compose.yml/.yaml ROBOCO_VIDEO_ENGINE_ENABLED/ON_RELEASE/ON_SPOTLIGHT default-on (.yaml resynced to .yml); registry stays off. CLAUDE.md video-engine section + CHANGELOG. Fixed 2 pre-existing route-test pollution leaks. Full suite 11763 passed. * fix(video): auth-carrying preview, media route confinement, VideoPost type drift Three fixes along the video preview path: 1. panel video preview auth: the <video> element was pointed straight at GET /video/posts/{id}/media, but a native <video src> GET carries none of axios's X-Agent-ID/X-Agent-Role headers — so in the default header-trust deployment the request 401s. Fetch the cut via videoApi.getMediaBlob (axios, responseType: blob) and drive <video> off a URL.createObjectURL result instead. The object URL is revoked on cut-change (the previous cut's URL) and on unmount, so neither cut switches nor row teardown leak blob URLs. 2. backend media route confinement: GET /video/posts/{id}/media now resolves mp4_path and refuses it with 404 when it falls outside settings.video_output_dir. Defense-in-depth against any future writer of mp4_paths serving files from arbitrary disk locations. 3. panel VideoPost type/comment drift: added mp4_paths to the VideoPost interface (the committed VideoPostResponse already carries it), and corrected the stale comment on videoMediaUrl that claimed no route served the rendered bytes — the route has existed since the media endpoint landed; the comment now describes why getMediaBlob exists instead of a direct <video src>. * Persist rendered videos to data in physical storage. * ++ * docs(video): 0.18.0 CHANGELOG entry + RAG + map reference for video engine - Move the video engine bullet from [Unreleased] into [0.18.0] and note the ROBOCO_VIDEO_OUTPUT_DIR bind-mount persistence. - Add docs/rag/architecture/video-engine.md (mirrors x-engine.md shape: enable/disable, three triggers, render loop + sidecar, CEO gate, media route confinement, credentials). - Reference the video render loop in docs/map/orchestrator.md's engine list. * chore(video): re-bump to 0.19.0 + sync registry compose defaults Version was wrongly bumped to 0.18.0; 0.18.0 is an already-released section. Restore its 2026-07-04 date and move the video-engine CHANGELOG bullet into a new [0.19.0] - 2026-07-05 section above it. Bump pyproject.toml, roboco/__init__.py, roboco/config.py (app_version), panel/package.json, and the motion/README inputProps example to 0.19.0. docker-compose.registry.yml: add ROBOCO_VIDEO_ENGINE_ENABLED / _VIDEO_ON_RELEASE / _VIDEO_ON_SPOTLIGHT defaulted false (NAS arms them true), and comment out the video-renders bind mount with a short note so the public registry image ships video off by default. Structural sync with docker-compose.yml maintained. * fix(video): rate-limit /render + reflow motion/README CodeQL flagged js/missing-rate-limiting on the renderer /render route. The sidecar is container-network-only with one trusted caller (the orchestrator, which renders cuts serially), so this limiter is a retry-storm ceiling (30/min, well above legit render rate), not the primary control. Also reflows motion/README.md hard-wrapped prose that failed the markdown quality gate. * fix(build): finish pnpm 11 migration + regen verb tables The panel Docker image build failed on `pnpm install --frozen-lockfile`: node:22-alpine's corepack resolved to its bundled pnpm 11, but panel/package.json pinned packageManager to pnpm@10.25.0, and pnpm 11 refuses to run against that pin. The Dockerfiles were already written for pnpm 11 (comments, CI=true, strictDepBuilds); the package.json pin was the stale outlier. Finish the migration instead of working around it: - panel/package.json: packageManager pnpm@10.25.0 -> pnpm@11.10.0; drop the `pnpm` field (pnpm 11 ignores it — build approval lives in panel/pnpm-workspace.yaml's allowBuilds). Lockfile unchanged (pnpm 11 accepts it as-is); frozen-lockfile verified. - remotion-renderer/package.json: pin packageManager pnpm@11.10.0 for determinism (was relying on corepack's implicit default); engines.node >=22.13 (pnpm 11 requirement). - docker/panel.Dockerfile + docker/remotion.Dockerfile: `corepack prepare pnpm@11.10.0 --activate` so the build uses the pinned version explicitly instead of trusting corepack's bundled default (which a future node:22-alpine could change). - .github/workflows/panel-ci.yml: Node 20 -> 22 (pnpm 11 requires Node >=22.13; Node 20 fails the engines check). Also regenerate agents/prompts/_generated/{developer,head_marketing,verbs}.md — the video engine added propose_video and extended propose_feature_spotlight (wants_video, video_script) but the verb tables weren't refreshed, failing the foundation-check quality gate. * chore(build): approve esbuild build script in remotion pnpm-workspace.yaml pnpm 11 generated this file with a placeholder ('set this to true or false') during install; resolve it to true so local dev of the renderer doesn't re-prompt. esbuild's postinstall only verifies the prebuilt platform binary (@esbuild/<platform> is installed as an optional dep), so approving it is safe and silences the ERR_PNPM_IGNORED_BUILDS warning. * fix(build): copy pnpm-workspace.yaml into panel + remotion images pnpm 11 hard-errors with [ERR_PNPM_IGNORED_BUILDS] (exit 1) when a dependency ships a postinstall script that isn't approved in allowBuilds. Both Dockerfiles copied only package.json + pnpm-lock.yaml, so the build-approval map in pnpm-workspace.yaml never made it into the image — the remotion image build died on esbuild@0.28.1's postinstall. Copy pnpm-workspace.yaml alongside the manifests in both images. In panel, this also drops the --config.strictDepBuilds=false workaround: with sharp and unrs-resolver now approved, their postinstalls run and install the platform-specific binaries (previously skipped, leaving sharp without its @img/sharp-* binary at runtime). Verified locally: remotion + panel `pnpm install --frozen-lockfile` exit 0 with the workspace file present; both exit 1 without it. * fix(perf): offload conventions + release-readiness blocking I/O off the event loop The orchestrator runs uvicorn and the orchestration background loops on a single shared event loop, so any sync I/O anywhere — even inside a background loop — blocks API responsiveness for its duration. Two call sites were missing asyncio.to_thread wrappers: - ConventionsService.get_map/health/restore called the sync _resolve (`git rev-parse`), _read_committed_standard (file read + yaml parse), and _derive (filesystem walk via derive_from_scan) inline. Reachable from GET /api/projects/{id}/conventions and from the agent spawn-prepare path. - ReleaseManagerEngine._production_assess called gather_snapshot inline — multiple `subprocess.run` git calls + a filesystem walk, running inside the release-manager background loop. Wrap each blocking call in asyncio.to_thread at the async boundary. No signature changes; helpers stay sync. Verified: targeted tests pass (196 passed, 36 DB-skipped), ruff + format clean. These were the only responsiveness gaps surfaced by the concurrency audit — the rest of the heavy paths (agent spawn via `docker run -d`, video render loop, git ops via the 16-worker ThreadPoolExecutor, workspace subprocess calls) already offload correctly. No API/worker container split needed. * feat(storage): add MinIO config + dep + compose (no-op, default-off) Chunk 1 of the MinIO video-storage plan (§1, §2, §6). No behavior change: minio_endpoint defaults to empty = disabled, the existing FileResponse serve path is untouched (chunk 4 wires the serve path; chunk 2 adds the client). - pyproject.toml: add `minio` (minio-py) to dependencies; regenerate uv.lock (resolves minio v7.2.20 + pycryptodome transitive). - roboco/config.py: add 5 settings fields after video_output_dir (minio_endpoint/_access_key/_secret_key/_bucket/_region). Plain str Fields matching the existing ROBOCO_ENCRYPTION_KEY style; no SecretStr, no presign_ttl_seconds (YAGNI — we don't presign in phase 1). - docker-compose.yml: add `minio` service (data network only, named minio-data volume, host ports 19000/19001 for debugging, mc healthcheck) and a one-shot `minio-init` service mirroring the ollama-init pattern (mc alias set + mb -p, idempotent via || true). Add ROBOCO_MINIO_* env to the orchestrator env block (endpoint, access/secret key, bucket, region). - docker-compose.registry.yml: intentionally omit the minio/minio-init services and leave ROBOCO_MINIO_* unset (NAS default-on, registry default-off — the established pattern); comment added to the orchestrator env block noting the omission. * docs(storage): 0.19.0 CHANGELOG + RAG + map reference for MinIO chunk 1 Backfills the release-polish docs for MinIO chunk 1 (§10 of the plan): - docker-compose.yaml synced to docker-compose.yml (the two NAS compose files must stay byte-identical; .yml was edited in chunk 1, .yaml was stale). - CHANGELOG [0.19.0]: Added (MinIO scaffolding) + Fixed (event-loop I/O offload). - docs/rag/architecture/minio-storage.md: RAG doc mirroring video-engine.md. - docs/map/deployment-tooling.md: one-line storage reference. * MinIO chunk 2: minio_client module (singleton + unconfigured guard) (#309) * feat(storage): minio_client module (singleton + unconfigured guard) Chunk 2 of the MinIO plan (§3). roboco/services/minio_client.py adds: - get_client(): singleton minio-py Minio from settings; returns None when minio_endpoint is empty (the disabled path used by the chunk 3/4 guards). Parses http://... endpoint into host:port + secure flag. - put_object(bytes, key): no-ops when unconfigured; otherwise PUTs to settings.minio_bucket with ContentType video/mp4. - get_object_stream(key): yields object bytes for StreamingResponse; lets S3Error propagate so the serve route (chunk 4) can fall back to disk. Sync calls — every call site wraps in asyncio.to_thread (chunks 3/4). One unit test covers the unconfigured guard + endpoint scheme parsing (mocks, no real MinIO). Not yet wired into remotion_client._save or the media route. * MinIO chunk 3: wire write path (remotion_client._save PUT) (#310) * feat(storage): wire MinIO write path in remotion_client._save Chunk 3 of the MinIO plan (§3). After the local mp4 write, _save PUTs the bytes to MinIO under key = Path(mp4_path).name (already {render_key}-{orientation}.mp4), guarded by minio_client.get_client() (None when minio_endpoint empty) and wrapped in asyncio.to_thread. Local disk stays the source of truth for the poster publish path (x_video_client/tiktok_client read mp4_path from disk); the PUT is additive. _save still returns the local path str — mp4_paths, marker, and schema unchanged. Disabled (local-only) when MinIO unconfigured. One test: asserts put_object is called with the basename key when configured and the local file is still written; existing test stays green via the unconfigured-default path. Mocks only. * fix(storage): make MinIO PUT non-fatal in remotion_client._save A configured-but-down MinIO made put_object raise inside the worker thread, failing the render and retry-looping a task whose local file was already written. Local disk is the source of truth and the serve route falls back to FileResponse on S3Error, so a failed durable-copy PUT must never fail the render — log and continue; the next render re-attempts the PUT. Adds test_save_swallows_minio_put_failure (PUT raises -> _save still returns the local path and the local file is written). Extends the CHANGELOG write- path bullet with the non-fatal guarantee. * MinIO chunk 4: serve path (StreamingResponse + FileResponse fallback) (#311) * feat(storage): serve MinIO via the media route (StreamingResponse + FileResponse fallback) Chunk 4 of the MinIO plan (§4 — the crux). GET /api/video/posts/{id}/media derives key = Path(mp4_path).name and, when minio_endpoint is set, returns a StreamingResponse over minio_client.get_object_stream(key), keeping _require_ceo so auth stays end-to-end (no presigned URLs). Falls back to FileResponse on S3Error (old render not in MinIO) or when MinIO is unconfigured — the panel's axios-blob flow is unchanged (same URL, headers, body, just chunked). The confinement check is kept as defense-in-depth (the key is a basename so traversal is impossible, but the check is cheap and protects the poster path). Two integration tests: configured serve path streams from a stubbed get_object_stream (CEO 200, non-CEO 403); unconfigured fallback serves the local file via FileResponse. Mocks only — no real MinIO. * fix(storage): eager stat_object probe so the MinIO serve fallback actually fires The chunk-4 route wrapped StreamingResponse(get_object_stream(key), ...) in a try/except, but get_object_stream is a lazy generator — its client.get_object call runs on the first next(), i.e. AFTER the route returned and Starlette started streaming. An S3Error (NoSuchKey / MinIO down) there is uncatchable; the try/except caught nothing and the FileResponse fallback never triggered. Add minio_client.stat_object(key): an eager existence/readiness probe that runs INSIDE the route's try/except, so a missing object or down MinIO raises before the StreamingResponse starts and the fallback serves the local file. stat-then-get is two round trips; a mid-stream failure after a successful stat is a rare race the CEO can retry (documented ceiling). Tests: the configured test now stubs stat_object; a new test asserts the S3Error fallback serves the local file via FileResponse and that get_object_stream is never called. RAG doc updated to record the eager-probe correctness detail + the non-fatal PUT. * docs(rag): mark MinIO deployment note landed (chunk 5) (#312) Co-authored-by: Renn F <rennf93@users.noreply.github.com> --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com> --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com> --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com> * fix(video): offload minio stat_object off the event loop stat_object was called inline in the async media route, blocking the shared event loop for one sync urllib3 round-trip per preview request — contradicting minio_client's own 'every call site wraps in to_thread' docstring and this PR's perf-fix theme. Wrap in asyncio.to_thread; the try/except still catches S3Error (to_thread re-raises) so the FileResponse fallback is unchanged. Also add the trailing newline to the minio-storage RAG doc. * Fix red CI * Make CI green --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>	2026-07-05 16:12:44 +02:00
Renn F	346716b8a0	docs(map): document v0.18.0 symbols + backfill migrations 060/061 ... Adds fable-mode, X feature-spotlight, and design-bar symbols across the map slices (+ _complete_map mirror); backfills migrations 060 (drop_messaging) and 061 (x_feature_spotlight), both previously undocumented; corrects the ORM table count 37->38; adds the _briefing_for include_company_goals row.	2026-07-04 08:39:36 +02:00
3 people	7901ea419e	Retire channels/sessions/messages; A2A becomes primary agent comms (#306) ... * feat(a2a): deliver latest incoming message preview into the claim briefing list_unread_a2a now carries last_message_preview (the latest message from the OTHER agent, never the agent's own reply), fetched via a correlated subquery in the same query — no N+1 on the per-verb briefing path. * feat(a2a): read_a2a verb delivers unread message bodies to the agent A2AService.get_unread_messages returns the caller's unread INCOMING messages (never its own sends), marking exactly those rows read atomically so a message arriving mid-call is preserved. Wired as the read_a2a content verb (route + do_server tool + granted to every delivery role) — the content-bearing read the A2A inbox lacked (read_messages only zeroed the counter). * docs(rag): document read_a2a as the A2A content-read path * fix(task): backlog activation no longer requires a discussion session Removes the SessionTaskTable gate in activate() (and its dangling log field), deletes _inherit_parent_session + its create() call, and drops the now-unused SessionTaskTable import. Coordination rides task state; the session subsystem is being retired. Tests updated to the new (no-session) behavior. * fix(orchestrator): drop session sweep from _run_sweep Removes the messaging import + sweep_timed_out_sessions call. That import sat outside the try/except, so once messaging.py is deleted it would have killed the entire sweep cascade (budget kill-switch, token rollups, retention, image prune, superseded-PR reconcile). Notification sweep + all maintenance sweeps unchanged. * release-manager --no-tags read-clone fix * test: update evidence_repo unit test for a2a last_message_preview * refactor(gateway): drop session propagation on delegate Removes propagate_sessions_to_subtask from delegate(), the ChoreographerDeps messaging field + property, and the ChoreographerDeps messaging arg in deps.py (ContentActions messaging + import stay until the verbs are removed). Deletes the propagation test; strips the now-invalid messaging kwarg from ChoreographerDeps test builders. * refactor(gateway): remove say/open_session/link_session/channels verbs Removes the four channel/session verbs across content_actions (impls + ContentActionsDeps.messaging), do_server (tools + registry), role_config (grants + _CHANNEL_DISCOVERY), do.py (routes), schemas/v1/do.py (request models), and deps.py (MessagingService import + construction). Regenerates the prompt verb tables. dm/notify/read_messages/read_a2a stay. Tests deleted/updated accordingly. * uv.lock Upgrade * refactor: remove conversation RAG indexing; Secretary announces via notification Drops the CONVERSATIONS index (index_conversation, ConversationsIndexPlugin, IndexType.CONVERSATIONS enum, IndexConversationParams, mentor.py type-label, the messaging index hook) and its chunk-table manifest entries. The Secretary's ANNOUNCE/RELAY_MESSAGE now fan out a BROADCAST notification to every agent's inbox (NotificationService.broadcast) instead of posting to a dead channel. * fix(panel): label RAG health error lines by subsystem A red llm_error (e.g. the glm-5.2:cloud weekly-limit 429) rendered under the 'Embedding: ok' header with no label, reading as an embedding failure. Prefix each error line with LLM / Embedding / Vector store. * refactor: remove channel/message reads from metrics, dashboard, git, events MetricsService drops get_communication_volume + the MessageTable message-count in get_agent_metrics (and the now-dead messages_sent_week field). DashboardService drops get_channel_feeds/_compute_channel_status and the message read in get_recent_activity (task activity kept); get_auditor_metrics no longer reports communication_volume. GitService's two primary-session-id helpers always return None now (callers already treat None as "no primary session"). events/handlers.py drops the SESSION_CLOSED/SESSION_TIMEOUT subscriptions + the handle_session_boundary handler. Forced follow-on: api/routes/dashboard.py + api/schemas/dashboard.py dropped the now-dangling live_feeds/ChannelFeed surface and the /metrics/communication route, which wrapped the removed service calls directly (mypy would otherwise fail on the missing attributes). * refactor: delete MessagingService + channel seeding Edited db/__init__.py and services/__init__.py first (drop the unconditional Channel/Group/Message/Session table + MessagingService re-exports), then deleted services/messaging.py, then trimmed db/seed.py to only create_agents (create_channels/create_channel_memberships/create_initial_messages gone). Forced expansion: api/routes/{channels,groups,sessions,messages}.py import roboco.services.messaging directly (not through the package __init__), as does api/routes/tasks.py (the session-links embed on GET /tasks/{id} and the GET /{id}/sessions route). Deleting messaging.py without addressing these breaks `import roboco.api.app` immediately, since app.py eagerly imports all route modules at startup. Since the 4 CRUD route files are 100% MessagingService-backed with zero independent logic (and are wholesale deletes in the plan's later API-routes task anyway), deleted them now + unmounted from app.py/routes/__init__.py; tasks.py got the same surgical trim its later task already specified (drop session-links embed + TaskSessionLinkResponse/TaskResponse.sessions). This pulls a slice of that later work forward — the routes/schemas for channels/groups/sessions/messages still need their own pass, but their messaging-coupled parts are gone. Verified with a full-suite collection sweep (12010 tests collected, zero import errors) beyond the directly touched test dirs, given the expanded blast radius. * refactor: remove channel/session/message models, tables, and channel policy Models: deleted channel.py/group.py/session.py/messaging.py wholesale (zero external consumers besides the models/__init__.py re-export). message.py surgically trimmed: removed MessageCreate (dead) and MessageEdit (never instantiated; ExtractedMessage.edit_history retyped to list[dict[str, Any]] to match how it's actually persisted — confirmed ExtractedMessage was never written to any DB table, so MessageTable's removal carries no functional risk to the kept extraction pipeline). base.py: removed SessionStatus + ChannelType, kept MessageType. Also removed the confirmed-dead channels_read/channels_write fields from models/agent.py:AgentPermissions and models/dashboard.py:ChannelFeedData. db/tables.py: deleted ChannelTable/GroupTable/SessionTable/SessionTaskTable/ MessageTable, TaskTable.session_links, and JournalEntryTable.session_id — cascaded through models/journal.py, services/journal.py, and api/schemas+routes/journals.py (22 plumbing sites). foundation/policy/communications.py: removed the ChannelSpec/CHANNELS catalog + TEAM_SCOPED_ROLES/_CELL_*/_AUDITOR_ONLY helpers, kept the notification policy (Priority/parse_priority/NOTIFY_SENDER_ROLES/ ACK_REQUIRED_BY_TYPE). enforcement/channel_access.py deleted (confirmed fully dead in production). agents_config.py: removed CHANNEL_ACCESS (kept A2A_ALLOWED_PAIRS). seeds/initial_data.py: removed DEFAULT_CHANNELS/CHANNEL_MEMBERSHIPS/AUDITOR_SILENT_ACCESS + the never-consumed INITIAL_MESSAGES. config.py: removed session_idle_timeout_seconds (zero consumers). exceptions.py: removed dead ChannelError/ChannelAccessDeniedError/SessionClosedError. Forced expansion beyond the original file list — ChannelType cascaded into a live, mounted surface the plan didn't trace: agents_config. CHANNEL_ACCESS -> services/permissions.py's channel-RBAC methods (not models/permissions.py, which turned out to have no channel code at all) -> two real endpoints in api/routes/stream.py (GET /permissions, GET /permissions/channel/{name}) and two dependency factories in api/deps.py. Removed the channel methods + fields, deleted the channel-specific stream.py endpoint, deleted require_channel_read/write. Also deleted api/schemas/{channels,sessions}.py (hard dependency on the removed enums; already fully dead after the Task 10 route deletions) and api/schemas/messages.py (a TYPE_CHECKING-only import of the deleted MessageTable; likewise already fully dead) + its dedicated test file. Test updates: test_permissions.py -14 channel tests (matches the planned count exactly), test_communications.py / test_communications_consumers.py split to keep only notification-policy coverage, test_exceptions.py -9, test_deps.py -4, plus the journal/stream/foundation-smoke fallout. Also fixed a pre-existing (Task 7) broken assertion in test_foundation_phase3_smoke.py that inspected a `say()` method already removed from ContentActions. Verified: full-suite collection (11961 tests, zero import errors) and a complete test run (11567 passed, 394 skipped, 0 failed) in addition to the targeted suites. * migration: drop channels/groups/sessions/session_tasks/messages + enum types alembic/versions/060_drop_messaging.py: drop_column journal_entries. session_id (sidesteps hardcoding the FK constraint name — verified empirically against a live migrated DB that it's actually fk_journal_entries_session_id_sessions, but drop_column doesn't care either way); drop_table in FK order (messages -> session_tasks -> sessions -> groups -> channels); DROP TABLE IF EXISTS chunks_conversations (runtime-provisioned, not alembic-managed, would otherwise orphan); DROP TYPE IF EXISTS for messagetype/sessionstatus/sessionscope/channeltype (messagetype's Python enum stays for ExtractedMessage, but the DB type had zero live columns left once MessageTable was dropped in the prior commit). downgrade() raises NotImplementedError — one-way removal. Pruned scripts/reset_runtime_state.sql + .sh: removed the DELETE/COUNT lines for messages/session_tasks/sessions/groups/channels and the groups.active_session_id reset block. Verified end-to-end against a scratch Postgres DB: full migration chain 001->060 applies cleanly, alembic heads shows a single head, all 6 dropped tables + 4 enum types + the journal_entries.session_id column are confirmed gone, journal_entries keeps only its journal_id/task_id FKs, downgrade correctly raises NotImplementedError without corrupting DB state, and the pruned reset_runtime_state.sql runs clean (no errors) against a fully-migrated DB. * refactor(api): remove channel/session/message routes + WS streams Most of this task's file list was already forced through in earlier commits (routes/{channels,groups,sessions,messages}.py + app.py/__init__.py unmounting in the MessagingService-deletion commit; tasks.py's session-links embed + GET /{id}/sessions + schemas/tasks.py's TaskResponse.sessions in that same commit; deps.py's require_channel_read/ write + schemas/{channels,sessions}.py in the models/tables commit). This closes out what was left: - api/websocket.py: deleted the channel_stream + session_stream routes, ConnectionManager's channel_connections/session_connections dicts, connect_channel/connect_session, broadcast_to_channel/broadcast_to_session, get_channel_subscriber_count, and their cleanup lines in disconnect(). Agent streams, notification streams, and the operator system stream are untouched. - api/websocket_bridge.py: deleted _handle_session_event + _handle_message_event and their SESSION_CREATED/SESSION_CLOSED/ SESSION_TIMEOUT/MESSAGE_SENT subscriptions. The A2A live-view, rate-limit, usage, agent-lifecycle, and notification bridges are untouched. - api/schemas/websocket.py: removed NewMessageBroadcast, WSMessageNew, WSMessageEdit, WSMessageDelete, WSSessionClosed — kept the WSMessage base class (still subclassed by the kept WSAgentStream/WSNotification) plus those two. - api/schemas/groups.py: deleted (already fully orphaned since routes/ groups.py was removed; its GroupResponse/GroupDetailResponse had zero consumers). Updated the 5 websocket test files accordingly (removed the channel/ session-specific tests + fixed imports); test_websocket_bridge.py's registration-coverage test dropped the SESSION_*/MESSAGE_SENT assertions. Verified: full-suite collection (11943 tests, zero import errors) and a complete test run (11549 passed, 394 skipped, 0 failed). * docs: retire channels/sessions/messages from agent-facing docs + CLAUDE.md Rewrites docs/rag (RAG-indexed) + docs/map + CLAUDE.md to reflect A2A (dm + read_a2a) as primary agent comms; deletes the channel docs, splits messaging-tools + messaging-notification (renamed notification.md), swaps the WS worked example to A2A_MESSAGE_SENT. _complete_map.md still needs regeneration (generated file). * refactor(panel): remove Communications surface (channels/sessions) Deletes the /communications routes, message components, task-detail Sessions tab, use-channels + channel/session WS hooks, and the channels/sessions/messages/groups api clients; prunes the Channel/Session/Message/Group types + mock data. (Auditor live-feeds + dashboard.ts dead-route cleanup is a follow-up.) * refactor(panel): drop auditor channel-feed + dead communication-metric route * docs(map): regenerate _complete_map from updated slices * fix(a2a): reduce get_unread_messages complexity below xenon C + stale comments Extract the per-conversation unread-counter recompute into _reset_unread_counter (the CI quality gate flagged get_unread_messages as rank C). Also drop the deleted open_session from a content_actions comment and reword an evidence_repo docstring that cited the removed messaging._notify_mentions. --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>	2026-07-04 03:10:33 +02:00
3 people	3ccc723cd4	v0.17.0 — Wave 3: sandbox DB, DB isolation, mobile UI, cloud auth, X account, roadmap engine (#303) ... * feat(sandbox): throwaway per-agent Postgres/Redis sandbox containers Orchestrator-provisioned sibling containers per agent spawn (SandboxProvisioner, roboco/runtime/sandbox.py). Per-project opt-in via projects.sandbox_services (migration 057); master switch ROBOCO_SANDBOX_DB_ENABLED, default-off, armed in the NAS compose only. When active, ROBOCO_TEST_DB_* / ROBOCO_TEST_REDIS_* point at the sandbox and the prod-creds gate-env injection is suppressed (sandbox replaces, never coexists). Sandbox lifetime tracks the agent container: teardown at every removal path, orphan janitor at startup + each reaper tick with a grace window for mid-flight spawns. The pre-spawn stale-clear spares the just-provisioned sandbox; provision pre-clears stale same-named containers from a crash-missed teardown. Panel: per-project sandbox-service switches in the edit dialog + feature flag card entry. * docs: CLAUDE.md entry for the sandboxed dev DB/Redis subsystem * feat(security): isolate prod Postgres/Redis from agent containers (roboco_data network) Second user-defined bridge roboco_data carries postgres+redis only; the orchestrator is multi-homed (default + data). Spawned agents and their sandbox sidecars stay on roboco_default and can no longer resolve or reach roboco-postgres:5432 / roboco-redis:6379 (redis has no auth — membership is its only containment). Normal bridge, so host-published ports (15432/16379) keep working. Applied to both build composes and the registry compose; docker-compose.yml re-synced byte-identical with docker-compose.yaml (it had drifted by the sandbox flag block). ROBOCO_DB_NETWORK_ISOLATED (config default false, armed alongside the topology) suppresses the legacy _append_gate_env prod-creds injection: under isolation those creds dead-end, and unreachable creds are worse than none. DB-needing projects opt into sandbox_services instead. The flag is deliberately not a panel feature flag - it must travel with the compose networks: stanzas. Preserved by construction: agent<->agent A2A and orchestrator->agent SDK polls on :9000, MCP->orchestrator on :8000, ollama reachability, docker exec/inspect (daemon socket), host port publishing. * feat(panel): full mobile responsiveness pass Shared primitives: useIsMobile (useSyncExternalStore, hydration-safe, memoized matchMedia subscribe), ResponsiveTable table->card switch below md (single subtree mounted, no duplicated interactive rows), scrollable snap TabsList in the base primitive (justify-center-safe so the first tab stays reachable on overflow), persistent md:hidden bottom tab bar (Overview/Tasks/Kanban/Chat, safe-area padded). Applied: card lists for tasks/projects/products/work-sessions/sessions + the three raw metrics tables; CEO approval queue / release proposal / playbook review action rows stack on narrow; command-center reorders approvals above the fold on mobile; task-header metadata wraps; Communications + A2A become URL-driven single-pane drill-downs below lg (fixes the unconstrained-height ScrollArea bug) with dvh heights; recharts label density/radius adapts via useIsMobile; git diff viewer gets mobile font + wrap toggle; vh->dvh sweep; chat composers get safe-area-inset padding; dashboard main p-4 md:p-6 + pb-20 for the bar. Verified at 375px on the built app: bottom bar, drawer, approval-first overview, swipeable kanban tab strip. All gates green (eslint, tsc, vitest 249, next build 24/24 routes). * feat(auth): cloud auth via FastAPI Users (default-off, single-user cookie session) ROBOCO_CLOUD_AUTH_ENABLED (default off) lets the panel/API be exposed beyond localhost without changing the CEO's local no-login flow while off — get_agent_context and the WS gate are byte-for-byte unchanged in off-mode. On: header-trust dies for humans — any agent-role claim (ceo or a privileged PM/board role) with no valid HMAC token or session cookie is 401, closing the header-spoof hole on the host-published :8000 port for every role. The agent-fleet HMAC path and the system self-PATCH keep working unmodified in both modes. Single seeded CEO user (migration 058 users table, UserTable), no registration router — idempotent env-driven upsert at startup by PK. Cookie transport (httponly/secure/samesite=lax) + a JWTStrategy bound to a fingerprint of the current password hash (rotating the password invalidates every prior session). Sliding 30-day session: every authenticated request re-mints the cookie, so an active session never expires — no unexpected logouts. Panel: (auth)/login page + proxy.ts (Next 16 rename of middleware; probes /auth/status over the docker-internal URL, fails open to off) gate the dashboard; client.ts gets withCredentials + 401->/login. nginx unchanged. Review hardening: broadened the on-mode rejection from ceo-only to every non-CEO role without a valid token (was only closed when ROBOCO_AGENT_AUTH_REQUIRED was also armed); Next-16 proxy.ts rename to clear the middleware deprecation warning. * feat(x): RoboCo X account engine — HoM drafts, per-post CEO approval (default-off) ROBOCO_X_ENGINE_ENABLED (default off, inert without creds). Mirrors the ReleaseManagerEngine held-artifact shape: XEngine drafts a post when a release publishes (via a draft_release_post seam on ReleaseProposalService .approve) and drafts replies to meaningful mentions (dedicated poll loop, x_seen_mentions dedup ledger, per-cycle/open caps). Drafting is local-model-only, clamped to 280 chars. Nothing auto-posts — every tweet is a held task (source x_post/x_reply, confirmed_by_human=False, Secretary-owned, dispatcher-skipped) the CEO edits/approves/rejects in a panel queue. The four OAuth 1.0a secrets live Fernet-encrypted in a singleton x_credentials row (migration 059, all-or-nothing, API returns only has_credentials); decryption is server-side, agents never hold creds or egress. Hand-rolled OAuth 1.0a HMAC-SHA1 signer, no new dependency; NullXClient makes the unconfigured path a graceful no-op. XPostService.approve (CEO-only) is the sole caller of post_tweet. Review hardening: closed a double-post race — the approve path now re-reads committed task state inside the Redis lock and commits COMPLETED before releasing, so a concurrent approve that acquires the lock after the winner released can't re-post (SET-NX is non-waiting, and the route-level commit landed after the lock dropped). Added a regression test. * feat(roadmap): board roadmap engine — PO proposes themed cycles, CEO approves per-item (default-off) ROBOCO_ROADMAP_ENGINE_ENABLED (default off). Weekly, RoadmapEngine opens ONE held exploration task (source=board_roadmap, confirmed_by_human=False, Product-Owner-assigned), deduped to one open cycle. A dedicated one-shot _dispatch_roadmap_exploration spawns the PO solo (not the two-reviewer board path, which would also spawn HoM + fire Approve-&-Start). The PO explores read-only (git/KB/metrics/releases/charter/web) and makes one propose_roadmap call (PO-only content verb) authoring a themed cycle — goal + 3-7 item drafts — persisted as a roadmap_cycle marker (no table, no migration; head stays 059). The CEO acts per-item in the panel roadmap queue: approve materializes a BACKLOG task (source=roadmap, no assignee — never auto-starts), reject records a reason; all-items-terminal completes the exploration task. RoadmapService is idempotent per item. Dispatchers skip board_roadmap. Includes a real SQLAlchemy dirty-check fix (deep-copy the JSON marker before mutating, or the in-place edit + reassign compares equal to its own baseline and the UPDATE is skipped). Review hardening: create_task_from_draft now honors a draft-declared source only from a {prompter, roadmap} whitelist — drafts are LLM-authored, so an unbounded source could impersonate a privileged origin (release_manager would even wedge that engine's dedup). * chore(release): 0.17.0 Wave 3 — six default-off subsystems: sandboxed dev DB/Redis, prod Postgres/Redis network isolation, full mobile UI pass, cloud auth (FastAPI Users), the RoboCo X account engine, and the board roadmap engine. Plus the waves 1+2 work already on master since 0.16.0. Version bumped across the canonical set (config.py, __init__.py, pyproject.toml, panel/package.json, uv.lock); CHANGELOG [Unreleased] cut to [0.17.0]; docs/map delta added. Compose: every optional feature armed :-true in the NAS composes, OFF in the user-facing registry compose. Two opt-in exceptions default off (CLOUD_AUTH — needs email/password/secret + TLS, would otherwise fail startup; ROUTING_STRICT — fail-closed spawning). DB_NETWORK_ISOLATED stays on in both (coupled to the roboco_data topology). * chore(compose): arm cloud_auth + routing_strict ON in the NAS composes Every feature defaults ON in the NAS composes per policy — these two were wrongly left off. Both keep the ${VAR:-true} form so the operator controls the real runtime via .env: cloud auth needs ROBOCO_CLOUD_AUTH_EMAIL/_PASSWORD/_SECRET + TLS set there before a boot (else startup fails loud), and routing_strict is fail-closed. Registry compose keeps both off. * fix(ci): reflow board.md prose (quality gate) + document v0.17.0 env creds The roadmap section added hard-wrapped prose that failed the markdown prose gate; reflowed (token-invariant). Also brought .env.example current: cloud auth (now armed — needs SECRET or startup fails), routing strict, the X engine (panel-entered OAuth), and web research. * fix(ci): reduce cyclomatic complexity of five wave-3 blocks (xenon gate) The wave-3 subagents introduced C-rank functions the CI xenon gate rejects (my per-item reviews ran ruff/mypy/pytest but not xenon): - sandbox.janitor_sweep -> extract _list_labeled_sandboxes / _list_live_agent_containers / _prune_grace - x_client.fetch_mentions -> extract _parse_mention_items - x_engine.run_cycle -> extract _process_mentions - orchestrator._dispatch_pm_work -> extract the source-skip into a MODULE-level _is_held_ceo_source (module, not method, so the wholesale-mocked dispatcher unit tests exercise the real logic) - auth/seed.ensure_seed_user -> extract _apply_seed_updates (module avg -> A) Behavior-preserving; full suite green (11902), xenon clean. * fix(ci): declare pyjwt + fastapi-users-db-sqlalchemy as direct deps (deptry) The cloud-auth code imports jwt and fastapi_users_db_sqlalchemy directly but they were only transitive deps (via fastapi-users), which deptry (quality gate, DEP003) rejects. Declared explicitly; deptry roboco/ clean. Missed originally because local make quality stopped at earlier gates before reaching deptry. * feat(x): gate mention replies behind ROBOCO_X_REPLIES_ENABLED (default off) Per CEO decision: the X engine should only post about releases by default. Reading mentions needs a paid X API tier, so the mention-reply half is now a deliberate opt-in on top of release posting. New default-off flag x_replies_enabled gates the mentions poll loop (_x_mentions_poll_loop) and XEngine.run_cycle; release-post drafting (the release-proposal approve hook) is unaffected and still runs when x_engine_enabled + credentials are set. Added to FEATURE_FLAGS + the panel card. Tests: release posting works with replies off; run_cycle + the poll loop are no-ops with replies off. * fix: 401 only redirects to /login when cloud auth is on; panel-token strips .env quotes Two bugs that together dead-ended login in secure mode: - client.ts redirected to /login on ANY 401, so a mismatched panel token (header-trust/secure mode, cloud auth off) bounced the user to a login page whose backend route isn't mounted -> 404. Now it probes /auth/status (bare fetch, no interceptor re-entry) and only redirects when cloud_auth_enabled. - make panel-token read the .env secret with grep|cut without stripping surrounding quotes, so a quoted ROBOCO_AGENT_AUTH_SECRET produced a token signed with the quotes included — which never verifies against the orchestrator (docker-compose/pydantic unquote the secret). Now strips surrounding single/double quotes. * fix: git-log 500 on '|' in commit message; X queue shows an empty state - GET /api/git/log 500'd (ValueError: Invalid isoformat) when a commit SUBJECT contained a '|' (e.g. the 'curl|sh' lockdown commit): the fixed '|' field delimiter let the subject's pipe shift the split so author+date collapsed into one field. Switched to \x1f (Unit Separator), which can't appear in commit content. Regression test with a piped subject. - The X Post Queue returned null when empty, so there was no visible place for the X drafts. It now renders a discoverable empty state pointing at Settings -> X credentials. * docs: bring docs/rag + docs/map current for v0.17.0 (waves 1-3) Agent-facing RAG corpus and codebase map updated for every feature in the 0.17.0 span, code-verified: - wave 3: sandbox DB, DB network isolation, cloud auth, X engine (+ x_replies_enabled sub-flag), board roadmap engine — new RAG architecture pages + role/tool/config-reference updates; new symbols, migrations 057-059, panel surfaces, and the get_agent_context dual-path across the map slices. - waves 1-2: A2A live view + switchboard, prompter memory (search_past_tasks), Secretary edit access + PM-lighter scope, the PR-gate auto-submit turn cut (ROBOCO_PR_GATE_AUTO_SUBMIT_ENABLED). - correctness fix: api-routes-schemas.md no longer claims the A2A admin routes are reachable by any authenticated agent — they carry a _require_ceo gate (wave 2c). docs/internal, _front.md deltas, and the frozen _complete_map.md snapshot untouched. * fix(rag): atomic upsert for indexed-doc tracking (kills e2e segfault) The indexed-document tracking write used check-then-insert in two paths (IndexedDocumentRepository.upsert_batch and the file-source _upsert_doc_record). Under concurrent indexing both callers saw no row and both inserted, so the second violated uq_indexed_doc_source and poisoned its transaction — surfacing in CI as the intermittent _checkin_failed SIGSEGV on the failed connection's pool checkin. Both paths now use INSERT ... ON CONFLICT DO UPDATE against the constraint: coalesce keeps an existing title/preview when the new value is empty (matching the old guards) and metadata is jsonb-merged. The batch dedupes within itself first (ON CONFLICT can't touch a row twice in one statement). expire_all after the Core upsert keeps same-session ORM reads consistent with the merged DB row. --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>	2026-07-03 19:24:00 +02:00
3 people	0ca9d91b72	v0.16.0: fastapi-guard HTTP security layer — calibrated + scanner honeytrap (#290) ... * [fastapi-guard] Phase 1a: gated config flags for the HTTP security layer Adds the ROBOCO_GUARD_* settings (all default-off / secure-default) for the upcoming fastapi-guard 7.2.0 hardening — guard_enabled (master switch), guard_fail_secure (fail-closed default; NAS overrides to false), guard_telemetry_enabled + guard_agent_api_key + guard_project_id (guard-agent telemetry, opt-in), guard_emergency + guard_emergency_whitelist (lockdown kill switch). Inert until consumed: nothing reads them yet, so the request path is unchanged. Foundation for v0.16.0. * [fastapi-guard] Phase 1b: security foundation module + gated wiring Add fastapi-guard 7.2.0 + guard-core 3.3.0 (bare, unpinned) and roboco/security.py: - build_security_config() from settings — behind-nginx real-IP (trusted_proxies + trust_x_forwarded_proto), HSTS/CSP headers, threat-ban + 404-sweep rules, redis-backed state, exclude_paths (/ws + health + docs), env-driven enforce_https, fail_secure (secure default), emergency lockdown, guard-agent telemetry (opt-in), passive-mode calibration switch. - guard_deco singleton (SecurityDecorator) for per-route decorators (Phase 2+). - Three custom content validators guard's WAF can't cover: prompt-injection / role-override, secret-exfil / credential-in-body, internal-SSRF. - apply_guard(app) + guarded_lifespan() wired into create_app AFTER settings. guard_passive_mode config flag added. Entirely gated by ROBOCO_GUARD_ENABLED (default off): create_app mounts nothing and returns the unchanged app when off (verified). make quality GREEN (cov 95.32%, pip-audit clean, import-linter 2/0). 12 new unit tests. * [fastapi-guard] Phase 2: critical-path decorators Apply guard decorators to the highest-value endpoints (metadata-only; enforced only when the middleware is mounted, so no-op while ROBOCO_GUARD_ENABLED is off): - provider keys (ollama/grok/self-hosted writes): strict rate_limit + max_request_size + block_clouds (no datacenter IP should touch secret writes). - settings write + release approve/reject (CEO-gated): strict rate_limit. - intake chat (prompter start/messages/events): rate_limit + max_request_size + custom_validation(prompt_injection_validator) — the prompt-facing free-text ingress gets the injection/role-override/secret-exfil content scan. make quality GREEN (cov 95.32%, contracts 2/0). App builds with guard off, decorators inert (verified). * [fastapi-guard] Phase 3: wide decorator coverage across ingress + sensitive routes Targeted-wide application (metadata-only; no-op until ROBOCO_GUARD_ENABLED). The global SecurityMiddleware already rate-limits + WAF-scans every request, so this adds the custom content validators on free-text ingress + tight limits on sensitive ops (not blanket per-route rate_limit on reads): - agent gateway do verbs (note/say/commit/dm/pitch/progress/draft_playbook/...): rate_limit + max_request_size + custom_validation(secret_exfil or prompt_injection). - a2a message/send + chat writes: rate_limit + size + prompt_injection. - optimal/RAG (kb/search, rag/query, mentor/ask, errors/decisions/standards/ learnings): prompt_injection on searches, secret_exfil on record writes; docs index → internal_ssrf. - tasks: create/update → prompt_injection; QA/doc/PM transitions → secret_exfil; CEO-gated verbs → tight rate_limit. - secretary chat → prompt_injection; research → internal_ssrf; orchestrator spawn/mutations → rate_limit; git ops + flow verbs → tight rate_limit. Pure GET/reads left to the global middleware. Applied via a Sonnet workflow, then verified: app builds with guard off (decorators inert), make quality GREEN (cov 95.37%, contracts 2/0). Decoy/honeypot-path surface deferred (needs verified guard ban-API integration — not rushed). * [fastapi-guard] Phase 5: arm the NAS composes in passive/log-only mode Arm ROBOCO_GUARD_ENABLED=true + ROBOCO_GUARD_PASSIVE_MODE=true + ROBOCO_GUARD_FAIL_SECURE=false on the two NAS composes (docker-compose.yaml + .yml). Passive = guard mounts and logs what it WOULD block but blocks nothing, so the next NAS deploy calibrates against real traffic; flip PASSIVE_MODE off after the false-positive review to enforce. fail_secure=false keeps a guard-internal error from 500ing the personal deploy. The registry (user-facing) compose is deliberately left unarmed so its published default stays conservative. Phase 4 (passive calibration) is the operational step this enables. * feat(security): Phase 3b — full-arsenal per-route guard enrichment Stack the applicable guard decorators per surface instead of the minimal rate_limit/max_request_size/custom_validation triad: content_type_filter on every JSON-body write, honeypot_detection form-traps on human-facing POSTs, block_clouds on key-writes + CEO release ops, behavior_analysis runaway-rate rules on the agent flow/do verbs, suspicious_detection + usage_monitor on the sensitive surfaces. Nine distinct decorators now applied thoughtfully per endpoint. All metadata-only — no-op while ROBOCO_GUARD_ENABLED is off. * fix(a2a): permit PR reviewer to deliver gate verdicts to the owning PM can_a2a_direct had no pr_reviewer rule, so a reviewer (team=board, or a cell team) fell through to the cell-member path and was cross-cell-denied when the in-path gate delivered a pr_fail change-request to main-pm (or a cross-cell cell-pm): "Cannot A2A main-pm ... Ask None to coordinate with None". The delivery is best-effort, so pr_fail still transitioned but the verdict never reached the owning PM — the blind-re-submit signal-gap the pr_fail fix closes. Add an explicit pr_reviewer handler: it may A2A only cell_pm / main_pm (its sole comms surface — everything else it posts on the PR itself), with a matching route hint. The cell reviewers kept same-team access by coincidence; this scopes every reviewer to PM-only, the correct model, with no other A2A caller affected. Refresh uv.lock to the current resolution. * feat(models): adopt Claude Sonnet 5 as the sonnet tier Point the 'sonnet' alias at claude-sonnet-5 (MODEL_MAP) and give pr_reviewer its own opus tier in ROLE_MODEL_MAP — it was falling through to the sonnet default, and the role gates untrusted external/fork PRs plus root→master, which warrants opus. Price claude-sonnet-5 at the promotional 33% off Sonnet 4.6 ($2.01 / $10.05, cache 0.201 / 0.5025) through 2026-08-31 via a dedicated pricing fragment that beats the bare 'sonnet' alias; revert to full rate when the promo ends. Bare 'sonnet' stays full-rate as a conservative fallback (prod prices the resolved claude-sonnet-5 id from the transcript). Update the model docs and the billing / usage / manifest / spawn tests. * feat(security): calibrate the guard WAF for RoboCo traffic + document the layer The first end-to-end run of the fastapi-guard layer showed active enforcement would block ~50% of legitimate agent traffic — RoboCo request bodies are code, SQL, diffs, file paths, HTML, and URLs, which the stock signature WAF reads as attacks. build_security_config now excludes RoboCo's free-text top-level body fields (derived from the real request models, including the free-form container fields whose nested prose is stringified and scanned) from WAF scanning, dropping the active-mode false-positive rate to zero while keeping the WAF on every non-excluded (id/enum/slug/branch) field and leaving the prompt-injection / secret-exfil / internal-SSRF validators — which run independently of the exclusion — fully in force. enable_penetration_detection is made explicit. Only excluded_detection_body_fields is reliable on guard 7.2.1: the per-route categories knob is bypassed for JSON bodies, and the body scanner excludes top-level keys only (scanning str(value) of every non-excluded field), so free-form container fields must be excluded wholesale. Adds tests/unit/test_security_middleware.py — the first end-to-end exercise of the middleware (mounts it, drives guard's lifespan, fires real requests): proves passive mode is log-only, active mode does not false-positive on realistic agent payloads, threats are still blocked inside excluded fields, and the WAF still fires on non-excluded fields. Docs: CHANGELOG (Unreleased); a user-facing Optional-subsystems page + nav + env reference for the HTTP security layer; the agent-facing RAG corpus (what it is + why a request could be blocked); and the roboco mapping (api-core-websocket / deployment-tooling / _complete_map). * feat(security): Surface N — scanner honeytrap (guard /api auto-ban + nginx edge-drop) Turns scanner probes against the scanner, in two layers matched to where traffic lands. Behind nginx only /api, /ws, /health, /ready reach the orchestrator, so guard can only see (and ban) scanner probes on those paths; the classic root probes (/.env, /wp-login.php, /phpmyadmin, /.git/config) hit the panel. So: - build_security_config's threat_ban_config gains recon / sensitive_file / cms_probing categories. A scanner probing those fingerprints on an /api path is detected on the URL-path scan; repeated probes from one IP trip an adaptive per-IP auto-ban (redis-backed, 24h). Only bans in active mode (passive logs the recon hit) and needs redis (the 24h ban exceeds the in-memory cap). The spec's decoy-route file is redundant — the WAF url-path scan bans regardless of a registered route — so it is intentionally omitted. - docker/nginx.conf drops the classic root scanner paths at the edge with 444 (connection closed, no response) before they reach the panel, anchored to known scanner fingerprints so /.well-known and every real panel/API route are untouched. Always on, independent of ROBOCO_GUARD_ENABLED. Tests: 2 unit (the exclusion set + the scanner-ban categories are present) and 2 integration (a decoy path is blocked in active mode, passes in passive). The nginx regex was validated against 15 scanner + 19 legit paths (0 false positives). Docs: CHANGELOG, the HTTP-security page, the roboco mapping, and the agent-facing RAG corpus. * Token optimization — per-role observability, compute policy, spawn preflight (#291) * test(models): lock the sonnet→claude-sonnet-5 MODEL_MAP invariant * feat(usage): surface cache tokens + cache_hit_rate in usage breakdowns * feat(usage): add per-role usage breakdown endpoint * feat(usage): add spawn-waste signal (per-role unproductive rate + respawn strikes) * feat(panel): surface per-role cost/cache + spawn-waste on the metrics page * feat(routing): Phase 2 per-role compute policy — qa→haiku, main_pm→sonnet, per-role effort env mechanism (default-inert) * feat(orchestrator): Phase 3 flag-gated spawn preflight — refuse non-gateway delivery roles (respawn-forever guard) * chore(compose): arm ROBOCO_SPAWN_PREFLIGHT_ENABLED on the NAS composes * docs: per-role usage observability, per-role compute policy, and spawn preflight --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com> * fix(panel): pin outputFileTracingRoot so the standalone build isn't broken by stray lockfiles * feat(routing): populate ROLE_EFFORT_MAP + wire the verified --effort flag (cell_pm/board/auditor to medium) * feat(gateway): omit empty context_briefing sections (Phase 4 payload compaction) * refactor(orchestrator): extract spawn chokepoint guards to restore xenon rank B on spawn_agent --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>	2026-07-01 23:54:48 +02:00
Renn F	543dfd0305	Pushing RoboCo mapping for version tracking	2026-07-01 05:33:15 +02:00