Both deploy composes published the orchestrator API on 0.0.0.0:8000, so any
host that could reach the machine hit the control plane directly — past nginx
and, in the default header-trust posture, with no credential: read/write
runtime settings and X-Agent-Role: ceo spawn/stop. nginx reaches the API over
the internal Docker network, so a routable host publish is never needed; bind
it to 127.0.0.1. On-host debugging and normal panel operation are unchanged;
off-host access must go through nginx + cloud auth.
make quality inside the production orchestrator container fails on
~1000 clean-env assumptions (armed compose flags, live Redis, host
mounts) for a tree that is green in CI — proven live on the first
org-proposed release. The execute-time gate now re-verifies the head
rung's CI conclusion, fail-closed on absent or red with branch, sha,
and conclusion in the failure detail; the pushed release commit keeps
its own CI wait before publish.
The readiness drafter transcribed raw commit subjects even when
[Unreleased] carried curated prose, and the executor inserted its entry
below a still-populated [Unreleased] — shipping the same content twice
in two qualities. The drafter now uses the curated body as the release
entry when present (transcription stays the fallback; completeness gaps
still police curation), and the executor empties [Unreleased] as it
stamps the entry. [Unreleased] itself catches up with the feedback
round, the dense tooltip passes, and the slave-CI fix.
CLAUDE.md gains the five undocumented subsystems (env-branches ladder +
EnvSyncEngine, Telegram bridge, possibilities matrix, collision map,
PR labeler) and their flags; docs/map and the pr-creation workflow now
describe head/prod ladder resolution instead of single default_branch;
CHANGELOG's [Unreleased] covers all sixteen merged PRs plus this
hardening basket.
* [13d03d5c] Add 5 coordination-event notification producers + wire at chokepoints (#472) (#474)
* [13d03d5c] Add 5 coordination-event notification producer methods
* [13d03d5c] Wire reassignment/collision/unblock/dependency-revival notifications
* [13d03d5c] Wire stale-claim-reaped notification into orchestrator reaper
* [13d03d5c] fix(runtime): guard reaper's UUID annotation + defensive attr access
The stale-claim-reaped notification hook added a runtime-unquoted
`UUID` type annotation (only imported under TYPE_CHECKING, so the
module raised NameError on import) and a direct `t.assigned_to`
attribute access that crashes against the minimal test doubles the
existing reaper test suite uses. Quote the annotation and switch to
getattr-defensive access, matching `_assignee_is_provider_parked`'s
existing convention in the same file.
* [13d03d5c] test(notification): unit coverage for 5 coordination-event producers
One test per new send_* method (reassignment, collision-sequencing,
unblock, dependency-revival, stale-claim-reaped) following the
existing _FakeDb/_patch_db_context pattern, asserting subject/body/
related_task_id/priority/recipient-count, plus a no-recipients no-op
case for reassignment.
* [13d03d5c] test(task): prove reassign + unblock don't double-fire notifications
Two chokepoint-level tests mocking NotificationService at its defining
module: a repeated reassign() to the same already-current target skips
the notification (guarded by comparing against the pre-mutation
assignee), and a repeated unblock() on the same task only notifies
once since the second call short-circuits on the status!=BLOCKED
guard.
* [13d03d5c] style(task): ruff format the collision-sequencing wiring block
No behavior change — reflows the newly-added _notify_collision_sequencing
call site to satisfy ruff format's line-length rules.
* [13d03d5c] docs(backend): add coordination-event notification producers guide
Documented the 5 new NotificationService producers (reassignment, collision-sequencing,
unblock, dependency-revival, stale-claim-reaped) with fire conditions, double-fire
prevention mechanisms, and implementation patterns. Updated backend README to link the
new services guide for developers integrating new coordination events.
---------
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>
* [3ee8150b] Frontend: render coordination-event notifications + e2e smoke coverage (#475)
* [69777c3a] test(e2e-smoke): add coverage for soft-block + unblock coordination notifications (#471)
Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech>
* [8eb82639] Render 5 coordination-event notification types with task deep-links (#470)
* [8eb82639] feat(notifications): add APPROVAL type icon and deep-link component test
Add missing APPROVAL member to the frontend NotificationType enum to
match backend roboco/models/base.py, wire its icon into the existing
typeIcons Record in the notifications page, and add a component test
covering type rendering and the task deep-link.
* [8eb82639] docs(notifications): document 5 coordination-event types and APPROVAL enum addition
Added comprehensive reference guide explaining the 5 notification types
(TASK_ASSIGNMENT, BLOCKER_ESCALATION, REVIEW_REQUEST, DOCUMENTATION_REQUEST,
APPROVAL), their visual identities (icon + color), use cases, and
deep-linking behavior to related tasks. Updated panel README with quick
reference table. TypeScript Record pattern ensures exhaustive type coverage
at build time.
---------
Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech>
Co-authored-by: Frontend Documenter <fe-doc@roboco.tech>
---------
Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech>
Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech>
Co-authored-by: Frontend Documenter <fe-doc@roboco.tech>
* [a27de2a8] fix(docs): reflow hard-wrapped notification-types.md to pass markdown gate (#479) (#481)
The Python quality gate on assembled PR #477 was red because the newly
added docs/frontend/components/notification-types.md (introduced by the
frontend coordination-event rendering commit) had manually wrapped prose
paragraphs, which scripts/reflow_md.py --check rejects as part of make
quality. Reflowed the file with scripts/reflow_md.py --apply (whitespace
only, no content change) so the check passes. ruff format/check, mypy,
xenon, vulture, bandit, and the full pytest suite (10284 passed) all
confirmed green on this commit; notification.py, task.py, and
orchestrator.py are untouched.
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
* [705419d5] Remove duplicate unblock notification and fix its dependent tests (#485) (#488)
* [705419d5] fix(notifications): remove duplicate unblock notification, fix its tests
The /unblock route was still calling delivery.notify_assignee_of_unblock()
(TASK_ASSIGNMENT) after TaskService.unblock() already sent the
send_unblock_notification() ALERT wired in by an earlier task — a real
duplicate notification on every unblock. Delete the route-layer call and
the now-dead NotificationDeliveryService.notify_assignee_of_unblock
method, fix the integration test that mocked it, and fix/extend the e2e
notification-coordination-events test to assert the persisted ALERT rows
(exact subjects) for both the direct-unblock and dependency-revival
producers instead of the old TASK_ASSIGNMENT assertion.
* [705419d5] docs(backend): update coordination-events doc for unblock duplicate removal
---------
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>
* [6c142a73] docs(changelog): document restored coordination-event notification producers and add collision-sequencing double-fire test (#489) (#490)
Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech>
* [77719d3f] Seed system agent in e2e harness to fix unblock/dependency-revival notifications
The e2e harness's seed_company omitted the system sentinel agent that
production seeds via initial_data.py. The unblock and dependency-revival
notification producers default to from_agent="system", which
_resolve_agent_uuid looks up by slug in the DB. With no system row the
resolver returns None and _create_notification silently skips the
notification, so the two ALERT assertions got 0 rows instead of 1.
The soft-block test passed because it uses NotificationDeliveryService
which creates the notification directly with a real agent UUID as
from_agent, bypassing the slug resolution path entirely.
* [77719d3f] Use foundation UUID for system agent to avoid slug collision
The first attempt seeded the system agent with a random UUID. Other
tests (_seed_system_and_secretary, _seed_video_agents) check by the
fixed foundation UUID via session.get(AgentTable, uuid); not finding
it they INSERT their own system row, hitting ix_agents_slug. Using the
foundation UUID makes their check find the seed_company row and skip.
* [77719d3f] Fix dependency-revival notification event loop mismatch
The dependency-revival test calls _unblock_dependents directly via
stack.run_db, which creates a new asyncio event loop. Inside,
_notify_dependency_revival -> NotificationService._create_notification
opened its own session via get_db_context(), which reuses the singleton
_DbHolder engine — bound to the FastAPI server's event loop. The
asyncpg connection raised 'Future attached to a different loop' and the
exception was silently caught + logged as a warning, so the notification
never persisted and the test saw 0 rows.
Fix: add an optional db_session parameter to _create_notification and
the two send methods. When provided, use the caller's session directly
and skip the internal commit (the caller owns the transaction). The
TaskService's _notify_unblock and _notify_dependency_revival now pass
self.session, keeping the notification in the same event loop + session
as the task transition.
* [77719d3f] Scope system-agent seeding to notification tests only
Seeding the system sentinel in seed_company (commits 3bba7b32/617b7890)
fixed the 0-notification bug but caused 3 i_documented gateway_timeout
failures: every e2e test now paid notification-creation latency for
system-origin notifications that were previously silently skipped,
pushing the already-slow i_documented verb past its 120s timeout.
Move system-agent seeding out of seed_company and into a scoped
_seed_system_agent helper called only by the two coordination-event
tests that exercise send_unblock_notification /
send_dependency_revival_notification (both resolve from_agent='system'
via DB lookup). dev_lifecycle and state_machine tests revert to the
pre-fix behavior (system-origin notifications silently skipped, no extra
latency).
The event-loop fix (commit 7b95d77d: pass db_session=self.session to
_create_notification) is unchanged — dependency_revival still needs it
because stack.run_db creates a new event loop while _DbHolder.engine is
bound to the FastAPI server loop.
* [77719d3f] Fix reassignment notification deadlock + suppressed-notification commit regression
Two fixes in notification.py / task.py:
1. Cross-session self-deadlock in send_reassignment_notification:
TaskService.reassign() flushes an uncommitted row lock on the task,
then calls _notify_reassignment -> send_reassignment_notification ->
_create_notification(db_session=None) which opens a SEPARATE session
via get_db_context() and INSERTs a notification with related_task_id
FK -> tasks.id. The FK key-share lock blocks on the request session's
uncommitted exclusive lock, but the request can't commit until the
notify returns -> 120s verb hard-cut. Fix: pass db_session=self.session
so the notification joins the verb's own transaction, same pattern as
the unblock/dependency-revival fix in 7b95d77d.
2. Suppressed-notification commit regression: the 7b95d77d refactor moved
await db.commit() out of _create_notification_with_session into
_create_notification's db_session=None branch, where it ran
unconditionally — even when _create_notification_with_session returned
early (suppressed: unresolvable from_agent / no recipients /
refire-guard / dedup-hit). Fix: _create_notification_with_session now
returns bool (False at each early return, True after delivery);
_create_notification commits only when created is True.
---------
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>
Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech>
Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech>
Co-authored-by: Frontend Documenter <fe-doc@roboco.tech>
Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech>
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* [35a27c3d] UX/UI: design task-detail overhaul (#404)
* [39ea1900] docs(ux_ui): add content-readability spec for markdown, collapsible sections, timestamps (#388)
Co-authored-by: UX/UI Developer 1 <ux-dev-1@roboco.tech>
* [71f9aec6] docs(ux_ui): add task navigation/structure design spec (#400)
Adds docs/ux_ui/design/task-navigation-structure.md covering the
breadcrumb trail, prev/next sibling navigation, and a distinct visual
treatment for the read-only constraints section, grounded in the real
task-detail components and existing amber/Lock read-only tokens.
Co-authored-by: UX/UI Developer 2 <ux-dev-2@roboco.tech>
---------
Co-authored-by: UX/UI Developer 1 <ux-dev-1@roboco.tech>
Co-authored-by: UX/UI Developer 2 <ux-dev-2@roboco.tech>
* [9baa1c34] Frontend: implement task-detail overhaul (#408)
* [13b6c723] Task detail: inline timestamps + breadcrumb + prev/next navigation (#390)
* [13b6c723] feat(panel): add inline absolute timestamps, task breadcrumb, and prev/next list nav to task detail
Adds a shared formatAbsoluteTimestamp helper used inline (with tooltip)
next to relative time on progress updates and checkpoints in
tab-progress.tsx, progress-timeline.tsx, and checkpoint-card.tsx.
Adds TaskBreadcrumb (renders only when task.parent_task_id is set) and
TaskListNav, which reads a new taskListNav context in the
scroll-restoration zustand store — populated by the Tasks list page from
TaskTable's live filtered/sorted order — to move to the adjacent task.
When no list context exists for the session or the current task isn't
part of the captured order, both nav buttons render disabled with an
explanatory tooltip (the documented fallback).
* [13b6c723] docs(guide): task detail navigation, timestamps, breadcrumb, and prev/next behavior
---------
Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech>
Co-authored-by: Frontend Documenter <fe-doc@roboco.tech>
* [40acdd31] Task detail: collapsible markdown sections + distinct Constraints styling (#407)
* [40acdd31] feat(panel): collapsible task-detail sections + distinct Constraints styling
Wrap the Description, per-field Notes, and Plan cards in a new
CollapsibleSection (Radix Collapsible + tw-animate-css fade/slide, so
collapse/expand only animates opacity/transform) so a long task no longer
forces continuous scrolling. Restyle the read-only Constraints card with an
amber accent border, background tint, and ShieldAlert icon so it reads as
distinct from authored content. Existing edit/preview toggles are
force-open while active and otherwise unchanged. Adds a global
prefers-reduced-motion override in globals.css.
* [40acdd31] docs(panel): CollapsibleSection component API and usage guide
Documents the new CollapsibleSection wrapper component used for independent collapse/expand of task-detail sections (Description, Constraints, Notes, Plan). Covers component API, controlled vs. uncontrolled state patterns, animation behavior (fade+slide, transform/opacity only), prefers-reduced-motion handling, and usage examples across task-description.tsx / tab-notes.tsx / tab-plan.tsx.
---------
Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech>
Co-authored-by: Frontend Documenter <fe-doc@roboco.tech>
* [73f8311f] fix(task-table): remove exhaustive-deps suppression on visible-order effect (#409)
Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech>
---------
Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech>
Co-authored-by: Frontend Documenter <fe-doc@roboco.tech>
Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech>
* [eb417ef1] Fix: apply auto-collapse thresholds to Progress and Acceptance Criteria surfaces (#429)
* [4e855d24] Apply content-readability-spec collapse thresholds to Progress and Acceptance Criteria surfaces (#416)
* [4e855d24] feat(task-detail): auto-collapse long progress/checkpoint/AC content per readability spec
* [4e855d24] refactor(task-detail): remove inline JSX section-marker comments per no-inline-comments convention
* [4e855d24] docs(task-detail): document content-readability-spec collapse thresholds for CollapsibleSection
---------
Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech>
Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech>
Co-authored-by: Frontend Documenter <fe-doc@roboco.tech>
* [3c90ef34] Wire content-readability thresholds into CollapsibleSection, tab-progress, acceptance-criteria (#430)
* [3c90ef34] test(task-detail): add AC4 combined readability test — 30+ progress entries + long acceptance-criteria list
* [3c90ef34] docs: enhance content-readability thresholds documentation and code comments
- Enhance panel/src/lib/content-readability.ts with usage examples and clarified intent
- Enhance CollapsibleSection with auto-collapse logic explanation and precedence rules
- Enhance TabProgress's RECENT_OPEN_COUNT logic with dual-threshold explanation
- Add comprehensive architecture guide: panel/docs/CONTENT_READABILITY_THRESHOLDS.md covering thresholds, components, testing, and implementation notes
The readability feature prevents long-history tasks (30+ updates, 20+ criteria) from rendering fully expanded, keeping pages navigable. Tests confirm 32 progress updates default to 2 open, and long criteria lists collapse while short ones stay expanded.
---------
Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech>
Co-authored-by: Frontend Documenter <fe-doc@roboco.tech>
---------
Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech>
Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech>
Co-authored-by: Frontend Documenter <fe-doc@roboco.tech>
* [fc04d84a] Round-3 revision: fix 4 named gaps on task-detail overhaul, one dev leaf per fix (#455)
* [cac9b603] fix(panel): fall back to task.created_at for missing written_at stamp in tab-notes.tsx (#446)
Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech>
* [31dd4f99] Remove ArrowLeft back button from task-header.tsx (#441)
* [31dd4f99] Remove ArrowLeft back button and Link wrapper from task-header.tsx, drop now-unused imports
* [31dd4f99] docs(task-navigation): mark spec as implemented, clarify ArrowLeft button removal
Update task-navigation-structure.md to reflect v0.21.0+ implementation:
- Status changed from "proposed" to "implemented"
- Clarified that ArrowLeft back button was removed from task-header.tsx
- Noted that breadcrumb and prev/next navigation now provide all navigation
- Constraints section styling with amber tint and ShieldAlert icon is complete
- Referenced related guide documentation for task-detail-navigation features
---------
Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech>
Co-authored-by: Frontend Documenter <fe-doc@roboco.tech>
* [75fd7444] Wire content prop into EditableNoteCard's CollapsibleSection (#449)
* [75fd7444] feat(panel): wire content prop into EditableNoteCard's CollapsibleSection
Pass the note field's current value into CollapsibleSection's content
prop and derive EditableNoteCard's initial sectionOpen state from
exceedsReadabilityThreshold, so long notes default collapsed with an
expand affordance while short notes render fully expanded.
* [75fd7444] docs(panel): document EditableNoteCard's content-driven collapse pattern in collapsible-section.md
Updated docs/frontend/components/collapsible-section.md to reflect how EditableNoteCard in tab-notes.tsx uses both controlled mode (force-open while editing) and content-driven initialization (seed sectionOpen from content length). Added a new "Combined: controlled + content-driven initialization" example showing this pattern for future developers extending editable-content sections.
Pattern: long notes default collapsed with expand affordance, short notes default expanded, edit forms always visible during editing.
---------
Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech>
Co-authored-by: Frontend Documenter <fe-doc@roboco.tech>
* [18ada610] docs(ux-ui): reconcile prev/next nav design spec with shipped list-order behavior (#453)
Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech>
---------
Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech>
Co-authored-by: Frontend Documenter <fe-doc@roboco.tech>
Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech>
* [3dfc43a1] round-3 fixes: reconcile nav spec, Alt+Arrow shortcuts, CHANGELOG
The breadcrumb section of task-navigation-structure.md now describes the
shipped single-ancestor design (and drops the stale DropdownMenu claims);
Alt+ArrowLeft/Right on TaskListNav mirror the visible prev/next buttons,
suppressed while an editable element has focus, with tests; the
user-facing CHANGELOG entry lands under Unreleased. Also reflows the
round-1 content-readability-spec so the prose gate is green branch-wide.
* [3dfc43a1] blank line between Unreleased and 0.22.0 sections
---------
Co-authored-by: UX/UI Developer 1 <ux-dev-1@roboco.tech>
Co-authored-by: UX/UI Developer 2 <ux-dev-2@roboco.tech>
Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech>
Co-authored-by: Frontend Documenter <fe-doc@roboco.tech>
Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech>
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* [3e552255] Re-add Playwright chromium to QA images + prompt guidance (#395) (#405)
* [3e552255] feat(docker): re-add Playwright chromium-headless-shell to QA images
* [3e552255] ci(docker): add Playwright QA image build + headless smoke check workflow
* [3e552255] fix(ci): scope agent-image-smoke.yml trigger to paths only, add PR comment
The workflow was gated by `branches: [master]` on both push and
pull_request, but this repo's task-hierarchy PRs open against nested
parent feature branches, not master, until root->master assembly - so
the workflow never fired on a dev-level PR and produced zero evidence.
Drop the branch filter (path scoping is sufficient) and post the
size-delta table + smoke-check output as a PR comment via
actions/github-script, since no agent role has gh CLI or GitHub API
read access to pull check-run output directly.
* [3e552255] fix(ci): post agent-image-smoke PR comment even on step failure
The 'Post results as a PR comment' step only had
`if: github.event_name == 'pull_request'`, which GitHub implicitly ANDs
with success() — so if the docker build or headless-launch smoke check
failed, the PR comment (the only evidence-delivery path QA/PM has, since
no agent role can read the Checks tab) silently never posted. Added
always() so a partial report always lands on the PR.
This commit also re-lands the branches-filter removal + PR-comment step
from 747e4d74 to make sure this fix actually reaches the remote PR
branch — QA's needs_revision at 05:48 came after that commit's local
timestamp (05:43) but still saw the pre-fix workflow, indicating the
prior push never reached GitHub.
* [3e552255] docs(qa): add browser verification guide and CHANGELOG entry for Playwright chromium
- Added comprehensive QA browser verification guide at docs/backend/qa/browser-verification.md covering setup, examples for fe-qa/ux-qa, limitations, and troubleshooting
- Updated CHANGELOG.md with Unreleased entry documenting Playwright chromium-headless-shell re-add to agent-qa-fe and agent-ux images, CI smoke workflow, and links to QA guidance
- Guide explains when to use browser verification (rendered output, computed styles, a11y, visual design), how to launch headless chromium, and provides practical examples for both FE and UX QA use cases
- References built-in guidance in fe-qa.md/ux-qa.md identity prompts and CI smoke workflow verification
---------
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>
* [9aafe8f5] Revision: fix reflow-check CI failure and resolve orchestrator Playwright-allowance gap (#421)
* [9973237c] fix(docs): separate reflow-joined metadata lines in browser-verification.md (#419)
scripts/reflow_md.py --check treats two adjacent non-blank lines as one
paragraph and flags it as hard-wrapped prose needing a join. Insert a
blank line between the `**For:**` and `**Purpose:**` metadata lines so
each stays its own single-line paragraph; verified the reflow is now a
no-op and the non-whitespace token sequence is unchanged.
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
* [cc355a2b] docs(qa): add no-op analysis for orchestrator.py Playwright allowance (#422)
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
* [a506cc10] docs(qa): quote real Dockerfile Playwright snippets in no-op analysis (#424)
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
---------
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
* [2b35dd4e] Resolve merge conflict, confirm green CI, add Playwright MCP registration, re-verify 5 ACs (#447) (#450)
* [2b35dd4e] docs(changelog): resolve Unreleased/0.22.0 ordering conflict, keep Playwright entry
* [2b35dd4e] feat(runtime): register Playwright MCP server for fe-qa/ux-qa, per CEO round-3 note
Adds @playwright/mcp to agent-qa-fe and agent-ux images, wired via a wrapper
entrypoint that points the server at each image's already-baked
chromium-headless-shell instead of downloading a second browser. The
orchestrator registers the `playwright` MCP server only for the qa role on
the frontend/ux_ui teams, so be-qa and ux-dev never see it. Updates the QA
identity prompts and docs/backend/qa/browser-verification.md to document the
structured mcp__playwright__* tools in place of hand-scripted Bash+Python,
adds CI smoke coverage (binary + baked-chromium resolution + a real
panel-page screenshot from inside the ux-qa image), and records the change
in CHANGELOG.md.
---------
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
* [1197c975] type the mcpServers extraction so mypy's no-any-return passes
* [1197c975] extract role-scoped MCP registration — orchestrator back under the complexity budget
The playwright branch pushed _generate_mcp_config to rank C in the merge
ref; docs/research/playwright registrations move to one helper, behavior
identical.
* [1197c975] basename-sanitize the MCP config filename
CodeQL's path-injection query re-fired on the (moved, unchanged) config
write; agent ids are orchestrator-issued, but the filename now rides the
same os.path.basename sanitizer _grok_usage_json established.
* [1197c975] basename the agent id variable itself — the sanitizer shape CodeQL models
---------
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
The PR-gate turn cut (#295) already auto-submitted assembled tasks, but
its refusals fell back to a PM spawn silently -- in production the PM
turns the cut was meant to remove kept happening with no visible cause
(live case: an AC-coverage refusal). The flag is gone (the fallback is
the safety net), the umbrella/branchless exclusion uses the canonical
batch predicates, and every refusal reason now rides into the spawned
PM's prompt so the fallback starts informed.
ReleaseExecutor.publish_release shelled out to 'gh release create', but no
Dockerfile installs the gh CLI (verified missing in the live orchestrator
container), so an armed release manager died at publish AFTER the release
commit was pushed. Publish now POSTs /repos/{owner}/{repo}/releases with the
project's decrypted token — same auth/httpx pattern as PR creation, same
fail-closed semantics (non-201 -> structured publish_failed, CEO retries;
the 300s deadline is the httpx client timeout). Subprocess publish-timeout
test replaced with REST-path tests (201/non-201/transport-error/no-token).
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
- agent-dev-fe / agent-qa-fe: remove playwright chromium + its system libs (~770MB each; verified unused — panel tests are vitest, e2e harness is scripted Python; pnpm kept)
- agent-grok: drop the redundant chown -R that duplicated the 149MB CLI tree into a second layer (install already runs as agent)
- agent-base + orchestrator runners: split the single /app COPY into .venv-first / source-last layers so a source-only deploy re-layers ~13MB instead of ~380MB per image
- agent-base: split the 813MB apt+node+claude-code RUN so a CLI bump no longer re-downloads the OS/node layer
- .dockerignore: exclude gitignored docs/internal from the orchestrator's docs COPY; pin uv helper image to 0.11
- verified: all four images rebuilt + runtime-probed (claude/git/jq/node/uv/pnpm/grok, import roboco, docs/alembic/agents present); .venv layer proven CACHED across a source-only change
- mongo:8-alpine → mongo:8 (tag never existed; a mongo-opted project could spawn no agents) + a Docker Hub tag-existence e2e guard for every sandbox engine
- flow-verb timeouts at both walls: shared SLOW_VERBS policy (i_am_done / submit_up / submit_root / open_pr / i_will_work_on get the 900s server budget); the MCP client now outlasts the server budget (+10s headroom, orchestrator-injected env) so agents receive the middleware's clean 504 envelope instead of dying at the old flat 30s client timeout
- cancellation safety: the quality gate kills+reaps its child on CancelledError; create_pr records the PR via a shield-with-wait-out helper so the write can neither be skipped nor race get_db's rollback
- video engine: renderer sidecar isolated on a render-only network, 2g/2cpu caps, 570s render watchdog with exit-on-hang, 512MB tar decompression cap, CEO notification on terminal render failure, reject under the approve mutex (fail-closed on Redis-down)
- dead python-jose dependency removed (drops ecdsa and its unfixable Minerva advisory PYSEC-2026-1325); panel --font-mono now a real monospace stack
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* feat(sandbox): pluggable per-engine registry (postgres/redis/mongo)
Replaces the hardcoded postgres+redis branches in the provisioner and the
env emitter with a registry of SandboxEngine specs (image, run args,
readiness probe, connection, ROBOCO_TEST_* env) in a pure low module
(roboco/models/sandbox.py). VALID_SANDBOX_SERVICES is derived from the
registry — single source of truth — and the provisioner + orchestrator
iterate it, so adding an engine is one class + one registry line, not
another branch. Adds a mongo:8-alpine engine (ROBOCO_TEST_MONGO_*) as the
third service alongside postgres/redis.
Also fixes the cold-pull loop that stranded v0.19.0 board agents with
empty error strings: docker run pulled inline under a 20s deadline, so a
NAS cold pull was killed, cancelled, and re-pulled from scratch forever.
_ensure_image now inspects + pulls (300s) before run; provisioning errors
log type+message so a bare TimeoutError no longer shows as "".
Panel edit-project dialog: postgres/redis toggles -> a Set<string>
multi-select driven by a SANDBOX_SERVICES catalog, so new engines appear
in the UI by adding to the catalog.
Tests: engine parity (allowlist==registry, unique slugs/images, no None
leak in env, SandboxInfo aggregates every engine), mongo provision + env
injection, plus the existing postgres/redis provision/env/spawn/janitor
suite updated to the registry shape. 821 unit / 5 skip green; ruff + mypy
(360 files) clean.
* docs(sandbox): reflect pluggable engine registry + mongo across docs
CHANGELOG (0.19.0): Added entry for the pluggable sandbox engine registry
(postgres/redis/mongo) + Fixed entry for the cold-pull loop/empty-error
strand that boarded v0.19.0 board agents.
docs/map (9 files): sandbox subsystem blurbs, SandboxProvisioner rows,
_maybe_provision_sandbox/_append_sandbox_env rows, feature-flag rows, the
migration-057 row + v0.17.0 delta, and the models.md VALID_SANDBOX_SERVICES
note — all retitled to DB/Redis/Mongo via the engine registry
(roboco/models/sandbox.py), with the one-class-one-line extension story and
the _ensure_image cold-pull fix. Production-network (roboco_data) lines left
as postgres+redis — mongo is sandbox-only, not a prod service.
docs/rag (3 files): sandbox-db.md rewritten around the registry (engine list,
generic _provision_engine, image pre-pull, ROBOCO_TEST_DB_*/REDIS_*/MONGO_*
incl. MONGO_AUTH_DB=admin, single emit_env); config-reference sandbox flag
row + subsection retitled; db-network-isolation framing broadened to
postgres/redis/mongo. preconditions-and-rejections left untouched (its hit
was an unrelated gateway see-also link).
* test(e2e): harden umbrella close terminal reads with bounded wait-for-state
The MegaTask umbrella close test flaked once on CI (ceo-approve returned
200 but the re-fetch saw awaiting_pm_review) then passed on re-run. The
production path is deterministic: complete -> main_pm_complete ->
submit_pm_review -> escalate_to_ceo -> ceo_approve -> commit, all on one
session, all awaited; the fire-and-forget completion hooks are isolated
(own session, best-effort, never touch task.status or the request session).
20 local runs could not reproduce it.
The one real surface is the read pattern: the e2e stack commits on the
uvicorn thread's loop and reads via a separate loop (run_db -> asyncio.run
with a fresh engine), so a terminal single point-read can race a
still-draining completion hook on a contended runner. Replace the two
terminal point-reads with a bounded wait_for_status poll. Strictly better
than a one-shot read: absorbs the transient, and a genuine state bug still
surfaces via the timeout branch asserting against the last-read state.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* feat(video): rewrite sidecar render core to HyperFrames (in place)
* feat(video): convert motion compositions from Remotion TSX to HyperFrames HTML
* refactor(video): rename render client to video_renderer_client (renderer-agnostic)
* chore(video): rename remotion-renderer prose in test_video_pipeline docstrings
* chore(video): rename sidecar to video-renderer + add system ffmpeg for HyperFrames
* chore(video): rename stray remotion-renderer refs in sidecar + py docstrings (controller cleanup)
* chore(video): fix stale Remotion API names in Dockerfile comment (controller cleanup)
* docs(video): rewrite video-engine prose for HyperFrames + add map entry + folded prose fixes
* docs(video): add trailing newline to docs/map/video-engine.md (controller cleanup)
* chore(video): drop internal spec refs + minio/test suppressions (folded hygiene)
* fix(video): reclaim outDir on createRenderJob throw + hide empty 4th highlight
Final whole-branch review (Opus) triaged two FIX items from the SDD nits
ledger; the rest ship as-is.
- render.js: a synchronous throw from createRenderJob (post-mkdtemp, not
awaited) left an empty outDir on disk — the outer catch only reclaimed
extractDir. Reclaim outDir too when it exists, and correct the stale
comment that claimed the out dir was never created.
- {vertical,square}.html: the 4th highlights <li> lived in the DOM hidden
only by JS, so a no-JS / failed-script render would show an empty bullet.
Start it style="display:none" and reveal on populate, so an unscripted
render shows nothing instead.
Vitest smoke (release-announcement.test.js) 4/4 green; render.js syntax
checked. Python suite untouched by this fix (JS/HTML only).
* fix(video): type _override_db yield as AsyncSession | None
T7 widened _build_app's db_session param to AsyncSession | None (to drop the
4x # type: ignore[arg-type] on the DB-independent _build_app(None, ...) calls)
but left the inner _override_db fixture typed AsyncIterator[AsyncSession] —
so 'yield db_session' yielded AsyncSession | None into a declared AsyncSession,
and mypy failed at test_video_routes.py:177 ('Incompatible types in yield').
The DB-independent media tests pass db_session=None deliberately: their route
uses a monkeypatched task service and never awaits the session, so yielding
None is safe at runtime. Type the override's yield as AsyncSession | None to
match — no cast, no # type: ignore, no assert, runtime behavior unchanged.
The 3 media tests (3 passed) and the 19 db-gated tests (skipped locally) hold.
* chore(gate): skip .superpowers scratch in markdown prose gate
reflow_md.py walks the filesystem via rglob('*.md') and skips tooling dirs
(.venv, .mypy_cache, .pytest_cache, ...) but not .superpowers/ — the
superpowers SDD workflow's scratch dir (briefs, reports, progress ledger,
all gitignored). A dev running SDD locally would hit a false markdown-prose
gate failure on those transient files. Add .superpowers to SKIP_DIRS,
consistent with the existing tooling-scratch exclusions.
* fix(video): validate composition_id to close path traversal (CodeQL)
compositionId flowed unvalidated from the POST body into path.join
under extractDir/motion/compositions/, so a '../..'-style value could
escape the composition dir (CodeQL: Uncontrolled data used in path
expression). Validate at the trust boundary in server.js
(/^[A-Za-z0-9_-]+$/) and add a path.resolve + startsWith containment
check in render.js so it stays safe regardless of caller.
* fix(mcp): send X-Agent-Token + X-Agent-Team from flow/do servers
flow_server._build_headers and do_server._build_headers constructed
only X-Agent-ID/Role/Correlation-ID, omitting X-Agent-Token and
X-Agent-Team (unlike ApiClient._get_agent_headers used by the other
MCP servers). Latent since the gateway refactor — surfaced when
ROBOCO_AGENT_AUTH_REQUIRED=true was armed on the NAS, 401-ing every
flow/do verb with 'Missing X-Agent-Token header'. Add both headers
(mirroring ApiClient) so the HMAC gate passes. Tests assert the
headers are now injected.
* [video-engine] Per-project video_engine_enabled opt-in toggle
Mirrors ci_watch_enabled (migration 048): the global
ROBOCO_VIDEO_ENGINE_ENABLED flag arms the subsystem; the new
projects.video_engine_enabled column (migration 063) opts a repo into
authoring against its motion/ dir. VideoEngine._opted_in_project no-ops
open_video_task at the single chokepoint covering all three trigger
paths (on-release, on-spotlight, CEO on-demand) until the operator
flips it in the panel edit-project dialog. Existing projects stay
opted out (server_default=false).
* fix(auth): send X-Agent-Token + X-Agent-Team from all agent->API call sites
The prior fix (6ed4e139) covered the flow/do MCP servers but missed four
other agent->orchestrator call sites that built the header dict by hand
and omitted X-Agent-Token and/or X-Agent-Team. With ROBOCO_AGENT_AUTH_REQUIRED
armed on the NAS, every one 401s:
- agent_sdk/server.py: the session-end post-mortem flush
(/api/journals/me/entries), A2A persistence + offline fallback
(/api/a2a/*), and the stopped-without-transition auto-substitute
(/api/tasks/auto-substitute) — all sent only X-Agent-ID/Role, so each
401'd 'Missing X-Agent-Token'. Add a shared _agent_headers() helper
(mirroring flow_server._build_headers) and route all four through it.
- agent_sdk/secretary_driver.py: _headers() sent the token but not the
team, so the HMAC gate 401'd with signature mismatch (secretary is
board-team; token signed with team='board', verified with team='').
Add the team header.
- mcp/git_readonly.py: the read-only git MCP sent only X-Agent-ID/Role
— no token, no team — so /api/git/* 401'd once auth was armed. Convert
the static _HEADERS to a _headers() helper with team + token.
- runtime/orchestrator.py: the cell-PM auto-submit self-API call acted
as a PM with a hand-built {X-Agent-ID, X-Agent-Role} dict — no token,
no team — 401ing under auth-required. Add _agent_api_headers(uuid,
role) mirroring _system_api_headers, and use it.
Tests: _agent_headers round-trip (token + team, team-omitted when None),
_agent_api_headers carries a signed PM token + team.
* [auth] Omit UNSIGNED self-call token in dev mode + video-engine test mypy fix
_agent_api_headers sent the UNSIGNED sentinel when ROBOCO_AGENT_AUTH_SECRET
was unset, but the dev-mode middleware rejects a presented-but-unverifiable
token with 401 signature mismatch (while accepting a missing one). The
cell-PM auto-submit self-call 401'd in every dev run, regressing
test_auto_submit_cuts_the_pm_turn. Attach the token only when a secret is
set. Also fix the FromClause.update mypy error in the per-project
video-engine opt-out test (ORM row load + flush).
* [auth] Omit UNSIGNED agent token at every agent->API call site
The orchestrator injects ROBOCO_AGENT_TOKEN=UNSIGNED when the HMAC secret
is unset at spawn. The API middleware rejects a presented-but-unverifiable
token with 401 'signature mismatch' even in dev mode (auth not required),
so forwarding UNSIGNED turned every flow/do/SDK/secretary/git verb into a
401 — the live pr_reviewer/i_am_idle signature-mismatch loop. Omit the
header when the token is the UNSIGNED sentinel at all five agent-side
header builders; dev accepts a missing token, prod 401s with 'Missing
X-Agent-Token' (the clear respawn-with-secret signal). Add a structlog
diagnostic on the middleware reject path so the next mismatch logs the
exact (id, role, team, token_unsigned, auth_required) inputs.
* [auth] Self-heal stale agent tokens at orchestrator startup
A token is signed once at spawn. If ROBOCO_AGENT_AUTH_SECRET drifts
afterwards (a .env change, a compose recreate that reloads the
orchestrator's env without recreating agent containers, an image
redeploy), the surviving agent keeps sending its old token and the
middleware 401s every verb with 'signature mismatch'. The container
stays alive heartbeating, so the reaper never reclaims it and no fresh
agent spawns: the fleet stalls.
_heal_stale_agent_tokens runs at startup (before _readopt_running_agents)
and kills each running agent container whose baked-in token no longer
verifies against the current secret, so normal dispatch re-spawns it
with a freshly signed token. Inert when the secret is unset (dev):
verify fails for every token without a secret, so the heal would kill
the whole fleet without this gate. Best-effort: a probe failure leaves
the container alone (the reaper still covers it).
* [auth] Sign agent token over the UUID, not the slug (pr_reviewer 401 root cause)
The token was signed over the agent slug (_append_agent_auth_env) while the
MCP servers send X-Agent-ID as the agent UUID (_generate_mcp_config, since
453a7ae2 — gateway v1 parses X-Agent-ID as Annotated[UUID]). The middleware
verified HMAC(uuid:role:team) against a slug-signed token → 'signature
mismatch', token_unsigned=false. Latent for 2 months until 6ed4e139/53391f22
made the MCP servers forward the token.
The c0328971 startup heal missed it: docker exec printenv reads the
container-level ROBOCO_AGENT_ID (the slug), so the heal verified the
slug-signed token against the slug → matched → didn't kill the stale
container, which kept 401ing (its MCP server sends the UUID).
Fix: sign the token over the UUID, set the container ROBOCO_AGENT_ID to the
UUID too (so the SDK server — which inherits container env, not the MCP
manifest env — sends UUID consistently), and resolve the container-env id to
its UUID in _heal_stale_agent_tokens so pre-fix stale containers are evicted
on next restart. Regression test: test_heal_kills_slug_env_container_with_slug_signed_token.
* [scan] gate A2A/notification/stream agent-id deps under cloud auth (C1)
* [scan] omit UNSIGNED agent token from MCP server headers (H1)
* [scan] fail loud when cloud auth and nginx CEO-token are both armed (H2)
* [scan] cache last-known-good auth-probe result in panel proxy (C2)
* [respawn] Tripped breaker self-heals after a cooldown
A DB-durable PM-respawn counter (migration 051 / e2f7097a) wedges forever
once tripped: the only reset was a task status change, which can't happen
while the breaker blocks the spawn. So a deploy that fixes the underlying
loop (auth/prompt/schema) couldn't clear the wedge without manual DELETE
surgery on respawn_tracker — the 2026-07-06 pr-reviewer-1 loop, where the
auth fix cleared the 401 but count=63 survived restart and kept skipping
the dispatcher spawn for an external-PR task.
Freeze last_check at the trip tick and, after pm_respawn_trip_cooldown_seconds
(default 300), let ONE spawn through. A still-wedged task re-trips after the
threshold (bounded re-burn ~3 spawns per window); a fixed one advances and
the status-change path fully resets. Restore re-stamps last_check to now, so
a freshly restored row still trips immediately — durability preserved, which
is why the migration-051 persistence tests still pass.
* [scan] fix test_deps callsites for cloud-auth-gate signature change (C1 followup)
* [scan] per-IP rate limit on /auth/login under cloud auth (L31)
* [scan] Phase 1 auth/security fixes under 0.19.0 CHANGELOG
* [scan] secretary token signs over real team (board) not empty — fixes /api/secretary/* 401 (L31-class)
* [scan] LoginRateLimiter: key off X-Forwarded-For first hop + redis-down fail-open test
nginx is the single entry point; request.client.host is the nginx peer IP,
collapsing every external client into one limiter bucket (self-DoS amp).
Read the downstream client IP from X-Forwarded-For (first hop) / X-Real-IP,
falling back to the peer. Adds coverage for the XFF keying, the redis-down
fail-open branch, and drops a redundant asyncio marker on a sync-TestClient
test.
* [scan] nits: describe login_max_attempts + replace cast with assert in get_current_agent_slug
login_max_attempts was the only bare cloud-auth field; add a Field
description matching the surrounding idiom. Replace cast('str', ctx.slug)
with a runtime assert that fails loud if the cloud-auth ctx invariant
breaks, and drop the now-unused cast import.
* [scan] secretary token: use get_agent_team resolver + complete spawn-shutdown mock team (0dfd45ca followup)
* [scan] require agent HMAC token under cloud_auth (close v1 flow/do header-trust)
* [scan] _require_ceo accepts CEO session cookie under cloud_auth
* [scan] HTTP require_panel_token accepts session cookie under cloud_auth
* [scan] gate /api/settings behind panel token
* [scan] gate unauthenticated /api read routes (agents/a2a-tasks/kanban/usage/rate-limits)
* [scan] hoist deferred test imports to top-level (clear PLC0415)
* [scan] Phase 1b e2e smoke + CHANGELOG
* [scan] add_dependency rejects self-reference + cycle (M18)
* [scan] WorkSessionService.create translates IntegrityError to ConflictError (H10)
* [scan] _qa_or_doc_claim locks the task row FOR UPDATE (M19)
* [scan] docs_complete + mark_pr_created lock the task row FOR UPDATE (H4)
* [scan] gate complete() IN_PROGRESS on leaf/branchless only (H3)
* [scan] _unclaim_from_blocked clears stale pre-block snapshot (H5)
* [scan] admin_set_status terminal guard + skip revision bump under force (M20)
* [scan] cell_pm_complete idempotent pre-check before merge (H7)
* [scan] wrap gateway post-runner side effects in try/except (H6)
* [scan] pass_qa/fail_qa accept AWAITING_QA only (L29)
* [scan] mark_pr_created passes audit_agent_id (L30)
* [scan] phase 2 e2e smoke - one scenario per finding
* [scan] phase 2 quality gate
ruff format + check: green
mypy roboco/: green (357 files)
pytest unit+integration: 6905 passed, 10 pre-existing DB-contamination
failures (pass in isolation)
e2e smoke: 11 passed, 4 cross-scenario workspace-contamination failures
(all 6 state-machine scenarios pass individually)
Quality-gate fixes:
- move function-local imports to module top (PLC0415)
- fix M19 regression: submit_for_qa clears active_claimant_id so the
competing-claimant guard lets the QA claim through
- fix H7 regression: _StubGit gains is_pr_merged_for_task
- fix M19 unit tests: mock session.execute for the FOR UPDATE lock
- e2e H3: notes >= 20 chars; e2e H5: rich i_will_work_on inputs +
PM unclaims (block reassigns to PM)
* [scan] move active_claimant_id clear into pass_qa/fail_qa + admin_set_status (M19 follow-on)
Phase 2 opus whole-branch review found the M19 follow-on clear lived in
the gateway wrappers (qa_pass/qa_fail) not the transition methods
(pass_qa/fail_qa) themselves. The direct REST routes POST /pass-qa and
POST /fail-qa call the transitions directly, bypassing the wrappers and
leaving the QA's stale active_claimant_id set in AWAITING_DOCUMENTATION
/ NEEDS_REVISION — the competing-claimant guard then rejects the next
legitimate documenter/QA claim. admin_set_status had the same gap for a
non-blocked override into a review/queue state (IN_PROGRESS->AWAITING_QA
left the dev's id, blocking qa_claim).
Root-cause fix: move the clear INTO pass_qa and fail_qa (mirroring
submit_for_qa), add a clear in admin_set_status when
new_status in _REVIEW_QUEUE_STATES and from_status != BLOCKED, and drop
the now-redundant clears + flushes from the qa_pass/qa_fail wrappers.
Every caller is covered; the wrappers keep their actor-mismatch warnings.
Covering tests: test_pass_qa_clears_active_claimant_for_doc_claim
(asserts a subsequent doc_claim succeeds), test_fail_qa_clears_active_claimant,
test_admin_set_status_into_review_queue_clears_active_claimant,
test_admin_set_status_non_review_queue_keeps_active_claimant. Updated
the two wrapper unit tests that asserted the wrapper clears (now the
transition's job).
* [C3] unindex_journal_entry + call from delete_entry
JournalService.delete_entry deleted the DB row but never de-indexed the
RAG chunks, so deleted/private journal content bled forever into RAG
answers and claim-time briefings. Add OptimalService.unindex_journal_entry
mirroring unindex_playbook (vector-store delete_by_source + tracking-row
delete via get_db_context, both idempotent + best-effort), and call it
from delete_entry after the row commit inside a try/except so a de-index
failure never errors the delete.
* [M25] learning_id hashes full content to avoid collision
The memory distiller emits lessons with a fixed 'Problem: …' opening
shape, so two distinct lessons whose first 100 chars match collided on
learning_id = f"lrn-{md5(content[:100])[:12]}". replace_on_reingest then
routed both to the same source URI and the second ingest's replace_chunks
DELETE wiped the first lesson's chunks — silent data loss.
Hash the full content (widening the hex slice 12→16) so distinct bodies
get distinct ids and each retains its chunks.
* [H13] reject non-internal local_llm_base_url at config load
* [M28] bulk-insert learning broadcast instead of N+1
* [M27] mark_read/mark_all_read stamp only the unread rows seen at call time
mark_read and mark_all_read used to zero the unread counter FIRST, then run
a bulk UPDATE … WHERE read_at IS NULL that stamped every inbound unread row.
A send_chat_message committing between the counter-zero and the UPDATE
inserted a new read_at NULL row that the UPDATE then stamped as read — the
new message was silently consumed while the counter stayed 0.
Mirrors get_unread_messages (same file): SELECT the unread message IDs at
call time, UPDATE exactly those IDs, then recompute the unread counter from
the DB via the existing _reset_unread_counter helper. A message arriving
mid-call is not in the selected ID set, so the UPDATE skips it and the
recomputed counter keeps it unread.
* [H12] dedup: exact to_agents predicate + purpose discriminator + ack DEL
* [M23] playbook indexed_ok/indexed_at + startup reconcile of unindexed approved
* [M24] RAG indexing dead-letter + janitor reclaim + failed_index_count health
* [L23] institutional_memory_status sentinel distinguishes below-floor/empty/error/disabled
* [L26] sweep_expired_notifications re-escalates stale unacked ack-required
* [phase3] e2e smoke + CHANGELOG for 0.19.0
* [M24] _reindex_journal_entry honors is_private (C1 review fix)
Dead-letter replay mirrors the original journal._schedule_rag_index path:
a private entry is never indexed into the shared JOURNALS corpus, and a
private learning is still recorded into LEARNINGS as non-shareable.
Previously the replay always called index_journal_entry and skipped
record_learning for private learnings, leaking private content on replay
and dropping the legitimate non-shared learning. Three regression tests.
* [H11] clone via git -c http.extraheader, not URL-embedded PAT
* [H11] _sync_read_clone fetch via http.extraheader, not URL-embedded PAT
Sibling site to the clone fix: the conventions read-clone refresh ran
'git fetch --tags <https://TOKEN@host> <branch>', exposing the PAT in the
fetch argv on the orchestrator host. Mirrors the clone site's per-call
'-c http.extraheader=Authorization: Basic …' prefix + bare URL. SSH URLs
and tokenless public repos unchanged.
* [H11] release_executor clone+push via http.extraheader; delete _inject_token_into_url
* [H8] rebase_onto_base gates on clean tree like pull
* [H9] _link_commit_to_task flushes, doesn't commit out-of-band
* [M38] _pr_is_merged returns None on HTTPError; caller assumes merged
* [M39] _cherry_unmerged_entry marker grep anchored to commit-prefix
* [L1] thread actor_agent_id through update_pr_for_task
* [H8] fix rebase test mocks for clean-tree gate
H8 inserted a 'git status --porcelain' dirty-tree gate at the top of
rebase_onto_base (mirroring pull). The 3 rebase control-flow tests mocked
_run_git with a side_effect list matching the OLD call sequence (no
leading status call), so every call shifted by one and the assertions
missed. Prepend a clean-status result to each list so the gate passes
and the fetch/checkout/reset/rebase/diff/abort/push sequence aligns.
Verified: 16 passed (was 3 failed/13 passed post-H8, 16 passed pre-H8).
* [L2] push --force-with-lease instead of bare --force
* [L1] refresh stale workspace-resolution docstrings
pr_target and _workspace_for_branch still documented the actor →
assigned_to → created_by fallback chain that L1 removed from
_resolve_workspace_agent_id. Update both to the post-L1 actor →
assigned_to → None resolver (project.workspace_path as the final
fallback) so a future reader doesn't rely on a fallback that no
longer exists.
* [M37] merge_pr locks the work_session row FOR UPDATE
* [phase4] e2e smoke + CHANGELOG for 0.19.0
* [phase4] fix M37 test flake + document H8 skip
The opus whole-branch review flagged the M37 concurrency tests as
~50% flaky: both asserted caller A wins the FOR UPDATE race, but
which caller wins the lock is non-deterministic. When B won, the
'assert a_row.merged_by == a_merger' branch flipped false even
though the production code (M37) was correct — exactly one merger
recorded, audit trail intact. Assert the invariant instead: both
rows COMPLETED, both report the same merged_by, value in
{a_merger, b_merger}. Applied to both the unit test and the e2e
twin. Also documents the H8 e2e skip in the module docstring (the
report claimed it was documented there but it wasn't) and drops
the internal 'Phase 4' label from the docstring header in favor of
the public '0.19.0' version anchor.
* [H24] wait_for_ci polls through the window on non-success
* [H25,L34] release mutex orphan-sweep on start + shared redis client
* [M1] tiktok _refresh commits rotated tokens in an independent session
* [H25] drop new type:ignore in orphan-sweep test (constraint cleanup)
* [M2] feature-spotlight re-arms when exploration stale past 2x interval with no live HoM spawn
* [M6,M7] mark_seen after meaningful+project; persist since_id cursor in redis
* [M3,M5] reject() guards COMPLETED; edited_body deferred into the single-flight lock
* [M4] bound list_completed_video_tasks + ix_tasks_source_status_created index (migration 066)
* [M8,M9,L9] pass head_sha to CI gate; _run_git 30s timeout; _commits_since split maxsplit 2
* [M10,L35] dedupe dep_update by (git_url, command); fold redundant per-project queries
* [L36] gather ci_watch telemetry sweep instead of sequential iteration
* [L11] document self_heal fingerprint is stable per-signal by design
* [M11] engine-loop liveness watchdog: heartbeat + 2x-interval staleness alert
* [M21] video render loop commits per-task, not one trailing commit
* [M22] _detect_stuck_tasks skips held-CEO-source tasks
* [L6] video_renderer_client._save writes temp + atomic rename
* [phase5] e2e smoke + CHANGELOG for 0.19.0
* [M11] instrument x_mentions + roadmap engine loops with liveness heartbeats
* [phase5] fix-wave: correct e2e M11 unit-test filename + strengthen failed-cycle heartbeat assertion
* [C4] panel WS: shared /ws/system socket + long-tail retry + pong watchdog
* [H15] video-post-queue caption derived per render (mirror x-post-queue)
* [C4-fix] panel WS: discriminating long-tail tests + drop dead freeze block + evict dead shared conn on manual disconnect
Finding 1 (Critical, websocket.test.ts): the two long-tail-retry tests fired onopen between close cycles, which reset reconnectAttempts to 0 each cycle, so they passed under the pre-fix 3-attempt gate. Rewrote both to NEVER fire onopen between closes, so attempts accumulates: test 1 asserts state stays 'reconnecting' past attempt 3 (old gate would flip 'disconnected' terminal); test 2 asserts a new socket is constructed within 30000ms at attempt 7 where uncapped 5000*1.5^7 ~= 85s (old uncapped code would leave the timer unexpired). Verified both FAIL on a reverted old-shape connection.ts and PASS on the fixed code.
Finding 2 (Important, connection.ts): the 'if (raw >= cap) this.reconnectAttempts = exp' block was a no-op (exp was just read from the same field) and the unconditional increment afterwards grew the counter regardless. Deleted the dead block; kept the Math.min cap on the delay. Replaced the misleading ponytail comment with an accurate one: delay is capped, counter grows unbounded but delay is bounded.
Finding 3 (Important, use-websocket.ts): manual disconnect() tore down the shared conn for all subscribers but left the dead (manualClose=true, never reconnects) entry in _sharedSockets, so a later mount hit the reuse branch, attached a subscriber, replayed 'disconnected', and never called connect(). Added a urlRef and _sharedSockets.delete(url) in the manual disconnect callback so a later mount reopens a fresh conn.
* [H16] settings Save wired to settingsApi (persist + read back)
* [H17] tasks page passes status/team/limit to useTasks (server-side filter)
* [H18] useAgents roster re-derives on live-status change (statusEpoch in queryKey)
* [M40] useMetrics reads agent counts from useAgentStatus cache (dedupe poll)
* [H18] tighten useAgents statusEpoch comment (drop spec ref)
* [M40] drop spec ref + tighten useMetrics comment
* [M41] scorecard refetchInterval 60s -> 5min (25 req/min -> 5)
* [M42] feature-flag off-transition confirm + pending-keys Set
* [M43] X/TikTok credentials clear-behind confirm dialog
* [M44] rate-limit syncFromApi merges (keep fresher hitAt) + A2A reconnect invalidation
* [phase6] proxy.ts cookie-check comment + CHANGELOG Fixed entries
* [phase6] drop stale WS pin-attempts comment + fix tasks-page lead-in
* [H21] type DelegateRequest.estimated_complexity as Complexity (reject critical)
* [H22] type SoftBlockRequest.resolver_type as BlockerResolverType (no silent AGENT fallback)
* [H23] serialize TaskTable.documents into TaskResponse (DocRefResponse)
* [L27] delete SubstituteRequest phantom suggested_role/suggested_team fields
* [L14] Envelope.not_found defaults remediate (guide re-fetch + re-issue)
* [L28] delete unused ListResponse generic (dead code; pagination deferred)
* [H19] _delegate_static_guards allow cell_projects roots (cross-cell MegaTask)
* [M13] MegaTask confirm-batch idempotency key from session_id (SETNX guard + result sidecar)
* [M14] strip assigned_to from MegaTask drafts (no board-owned root-subtask deadlock)
* [H20] thin_routes receiver-gate add/add_all/merge (no false block on set/cache.add)
* [M16] tighten noqa code-capture to [A-Z0-9, ]+ (no false block on natural prose)
* [M45] conventions read-clone force-refetch on read (no 30s stale map window)
* [L25] conventions._resolve returns (root, sha); ORM mutated on the event loop
* [M15] open_conventions_pr force-pushes disposable scaffold branch (no silent None)
* [L24] roadmap cycle completion emits status-transition audit
* [Phase7] CHANGELOG: 15 schema/conventions/MegaTask/API fixed (H21-H23,L27,L14,L28,H19,M13,M14,H20,M16,M45,L25,M15,L24)
* [Phase7] lint gate hygiene: shorten docstring (E501), sort imports (I001), hoist AuditLogTable import (PLC0415)
* [H14] Enable the GROK provider row in _apply_grok so routing reaches the GrokCliProvider
* [M31] Route GROK active-token resolution to usage.json so live usage reflects grok agents
* [M32] Pass cache read/write tokens to calculate_cost in the usage sweep so live cost reflects Anthropic cache spend
* [M33] Park Ollama-Cloud rate limits via a marker map so a glm-5.2:cloud 429 parks instead of crash-respawning
* [M34] Sweep orphan agent_spawn_sessions at startup so crashed-run tokens roll into usage/cost summaries
* [L12] Persist revisit_resets (migration 067) so the PM-respawn breaker's revisit counter survives a restart
* [L18] Date-gate the Sonnet-5 promo revert so billing returns to list rates after 2026-08-31
* [L20] Warn when ROBOCO_GROK_RUN_LOG yields no session id instead of silently falling back to a zero-usage env id
* [phase8] CHANGELOG: LLM provider routing, usage capture, billing fixes
* [phase8] Trailing ruff format hygiene (orchestrator marker tuples, token-sweep test signatures)
* [phase8] Fix mypy: rename GROK-branch tokens var so transcript fallback stays reachable
* [M35] Add an expiring agent-token format (iat/exp) with backward-compatible verify
* [M35] Wire agent-token TTL at spawn (config + orchestrator + grok) so tokens are bounded
* [M36] Add JWT jti claim and re-mint the sliding cookie only near expiry so a stolen cookie's exp is fixed
* [M36] Redis jti revocation: read_token rejects revoked jtis and logout revokes the current jti
* [phase9] CHANGELOG: bound agent tokens + sliding-cookie re-mint window + jti revocation
* [scan-fix] mypy: type-annotate test files for make-quality gate
CI's make quality runs mypy roboco/ tests/; the scan-fix program's local
gate ran mypy roboco/ only, so test files were never type-checked. Fix all
67 errors across 23 test files with real annotations/casts/asserts/dead-code
removal — no # type: ignore / # noqa added.
* [e2e] Per-test DB isolation + dispatcher re-claim before PM complete
* [scan] Regenerate verb tables for delegate Complexity type
* [scan] Reduce 9 xenon C-ranks to B (auth, orchestrator, gateway, services)
* [scan] Restore short-circuit time.time() in verify_agent_token (security path)
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* feat(video): Phase A — VideoEngine origination spine + held-source gates
New default-off engine skeleton: opens a UX/UI authoring task (source=video, assigned to a ux-dev, LOW complexity to clear the dev-needs-subtasks guard) and materializes a held CEO-approval draft (source=video_post). Excludes video_post from all three held-source skip sites; adds the video_draft marker, six config flags, and the feature-flag entries. Origination + gate behavior unit-tested.
* refactor(orchestrator): fold _dispatch_dev_work skip chain into a helper
The per-source if/continue chain grew past xenon's --max-absolute B when the video_post held source joined it. Extract _is_non_dev_dispatch_source (every held-CEO source plus the two Board exploration sources) so the dev loop's skip is one flat call. Behavior-identical.
* feat(video): Phase B — propose_video do-tool (metadata-only, team-gated)
UX/UI dev records a video's composition ref + per-platform captions onto the authoring task's video_draft marker. Team-gated at runtime via _caller_team (Role.DEVELOPER can't tell a ux-dev from a be-dev). Resolves the caller's ACTIVE task via get_active_task_for_agent, not an oldest-first scan that would clobber a second open video task. Metadata only, no render. Wired through do_server + route + schema; added to _DEV_DO.
* feat(video): Phase D — render loop + RemotionRenderer client
Orchestrator-async _video_render_loop renders a completed authoring task's merged composition to MP4 (vertical + square) via the remotion-renderer sidecar and materializes the held video_post draft. RemotionRenderer tars the read-clone's motion/ source, POSTs it, and saves the returned MP4 bytes to a TASK-scoped local path (no shared volume; a composition is reused across videos so a composition-scoped path would clobber an earlier draft). Render failures bounded-retry (read-clone catch-up window, transient sidecar) up to a cap, then terminal-fail. Client tested vs a mock transport; loop vs a mock renderer + real DB.
* feat(video): Phase C — release / spotlight / on-demand video triggers
Three entry points open a UX/UI video-authoring task via VideoEngine.open_video_task: (1) a published release drafts a companion video — best-effort in ReleaseProposalService.approve, never fails the publish; script from the CHANGELOG via the local model with a template fallback. (2) propose_feature_spotlight gains optional wants_video/video_script — best-effort, gated on video_on_spotlight, default-off leaves the spotlight flow byte-for-byte unchanged. (3) POST /video/request (CEO-only) for an on-demand brief, with clean disabled/not_opened responses. All gated on video_engine_enabled.
* fix(video): savepoint-isolate video-task inserts (F042 poisoned session)
The best-effort try/except around open_video_task (release-publish + spotlight hooks) swallowed the Python exception, but a DBAPI error at the insert flush left the shared session must-rollback — so the caller's next commit (release finalize / request boundary) threw PendingRollbackError: the release stuck 'pending' after actually publishing, or the spotlight draft + HTTP response were lost. Wrap both inserts (open_video_task, _originate_video_post) in a begin_nested savepoint (the repo's established F042 pattern) so a DB error rolls back only the insert. open_video_task returns None (every caller already handles it); _originate_video_post propagates to the render loop's handler. Regression test: an insert FK error returns None with the session left usable. Dormant while the flags were off; armed on the NAS.
* feat(video): Phase G — motion/ package + remotion-renderer sidecar + compose
In-repo Remotion v4 motion/ package (ReleaseAnnouncement composition; calculateMetadata returns 1080x1920 vertical / 1080x1080 square from inputProps.orientation) + a credential-free remotion-renderer sidecar: untar the POSTed motion/ source, bundle (LRU-cached per source sha), selectComposition + renderMedia h264, stream the MP4 bytes back — matching the RemotionRenderer client contract. docker/remotion.Dockerfile on Debian (Chrome apt deps, build-time Chrome pre-warm, ffmpeg bundled in @remotion/renderer). Wired into both compose files (roboco_default only, shm_size 1gb, /health check) + the release publish matrix. Verified via a real local render of both cuts; the Debian docker build is the CEO's to run.
* chore(video): D-hardening — video_post source_task_id + render-loop docstring
Add a source_task_id back-reference to the video_post held-draft marker (traceability from a draft to its authoring task; also makes the render loop's two-key idempotency check wireable later). Fix the render-loop test's stale docstring ('never retried' -> bounded-retry). Both from the Phase D critic's non-blocking follow-ups.
* feat(video): Phase E1 — VideoPostService + heartbeat mutex (approve->post)
CEO-approve->post service: heartbeat-renewed Redis mutex (fail-closed, grace=ttl-2*heartbeat), re-read-in-lock double-post guard, per-platform durable commits (asyncio.shield-ed, settle-before-rollback on lock-loss), all writes inside the lock (captions validated pre-lock, applied in-lock — no stale whole-column clobber), idempotent, per-platform retry-skip. Poster interfaces (X/TikTok, mocked here). Reject + list-held-drafts. Survived 3 adversarial rounds; residual = a crash in the poster->commit window (CEO-gated low-freq, documented).
* fix(video): G-hardening — renderer leaks + Share Tech Mono brand font
Sidecar: give bundle() an explicit outDir tracked + deleted on LRU eviction (was leaking ~19MB remotion-webpack-bundle-* per source); res.on('close') cleanup so an aborted/retried download no longer leaks its remotion-out-* MP4 dir. Fonts: vendor Share Tech Mono (roboco-website brand font) as the display face (self-hosted woff2, 400-weight, headline fontWeight 700->400 to avoid faux-bold) + self-hosted Inter body — no gstatic fetch at render time (lsof-verified). Extras: composition_id whitelist (400) + Multer error middleware (400/413).
* feat(video): Phase E2 — X v2 + TikTok posters, tiktok_credentials, routes
LiveXVideoPoster (X v2 chunked media upload: init/append/finalize/STATUS-poll -> tweet w/ media_ids, OAuth1 signer reused). LiveTikTokPoster (OAuth2 inbox: init -> chunked PUT with asymmetric final chunk -> status-fetch; 401 -> refresh_token grant, rotated token persisted). tiktok_credentials Fernet singleton + migration 062 (single head). Routes: CEO approve/reject + list held drafts + write-only tiktok creds, wiring real posters into VideoPostService. Residual: a lock-loss right after a token-refresh flush can discard the rotated token (same rare CEO-gated class as the documented post->commit window).
* feat(video): Phase F — panel video-post queue + TikTok creds card + flags
video-post-queue.tsx: <video> MP4 preview with 9:16/1:1 cut switch, per-platform editable captions (280/2200 counters, over-limit disables approve), approve/reject, Request-a-video dialog. tiktok-credentials-card.tsx (4 write-only OAuth2 fields). feature-flags-card inlines TikTokCredentialsForm under video_engine_enabled. Mounted in command-center. tsc/eslint clean, 273 panel tests green. NOTE: needs the GET /video/posts/{id}/media route + mp4_paths on VideoPostResponse (folded into H) for the preview source.
* feat(video): Phase H — media route + e2e smoke + NAS arming + docs
GET /video/posts/{id}/media?cut= (CEO-gated FileResponse of the rendered MP4; closes the panel preview gap) + mp4_paths on VideoPostResponse. e2e smoke tests/e2e_smoke/test_video_pipeline.py (full flow, sidecar+X/TikTok mocked; asserts dispatcher skips, render-loop materialize, propose_video team-gate, approve idempotency). NAS arming: docker-compose.yml/.yaml ROBOCO_VIDEO_ENGINE_ENABLED/ON_RELEASE/ON_SPOTLIGHT default-on (.yaml resynced to .yml); registry stays off. CLAUDE.md video-engine section + CHANGELOG. Fixed 2 pre-existing route-test pollution leaks. Full suite 11763 passed.
* fix(video): auth-carrying preview, media route confinement, VideoPost type drift
Three fixes along the video preview path:
1. panel video preview auth: the <video> element was pointed straight at
GET /video/posts/{id}/media, but a native <video src> GET carries none
of axios's X-Agent-ID/X-Agent-Role headers — so in the default
header-trust deployment the request 401s. Fetch the cut via
videoApi.getMediaBlob (axios, responseType: blob) and drive <video>
off a URL.createObjectURL result instead. The object URL is revoked
on cut-change (the previous cut's URL) and on unmount, so neither
cut switches nor row teardown leak blob URLs.
2. backend media route confinement: GET /video/posts/{id}/media now
resolves mp4_path and refuses it with 404 when it falls outside
settings.video_output_dir. Defense-in-depth against any future
writer of mp4_paths serving files from arbitrary disk locations.
3. panel VideoPost type/comment drift: added mp4_paths to the
VideoPost interface (the committed VideoPostResponse already
carries it), and corrected the stale comment on videoMediaUrl
that claimed no route served the rendered bytes — the route has
existed since the media endpoint landed; the comment now describes
why getMediaBlob exists instead of a direct <video src>.
* Persist rendered videos to data in physical storage.
* ++
* docs(video): 0.18.0 CHANGELOG entry + RAG + map reference for video engine
- Move the video engine bullet from [Unreleased] into [0.18.0] and note
the ROBOCO_VIDEO_OUTPUT_DIR bind-mount persistence.
- Add docs/rag/architecture/video-engine.md (mirrors x-engine.md shape:
enable/disable, three triggers, render loop + sidecar, CEO gate, media
route confinement, credentials).
- Reference the video render loop in docs/map/orchestrator.md's engine list.
* chore(video): re-bump to 0.19.0 + sync registry compose defaults
Version was wrongly bumped to 0.18.0; 0.18.0 is an already-released
section. Restore its 2026-07-04 date and move the video-engine CHANGELOG
bullet into a new [0.19.0] - 2026-07-05 section above it. Bump
pyproject.toml, roboco/__init__.py, roboco/config.py (app_version),
panel/package.json, and the motion/README inputProps example to 0.19.0.
docker-compose.registry.yml: add ROBOCO_VIDEO_ENGINE_ENABLED /
_VIDEO_ON_RELEASE / _VIDEO_ON_SPOTLIGHT defaulted false (NAS arms them
true), and comment out the video-renders bind mount with a short note
so the public registry image ships video off by default. Structural
sync with docker-compose.yml maintained.
* fix(video): rate-limit /render + reflow motion/README
CodeQL flagged js/missing-rate-limiting on the renderer /render route.
The sidecar is container-network-only with one trusted caller (the
orchestrator, which renders cuts serially), so this limiter is a
retry-storm ceiling (30/min, well above legit render rate), not the
primary control. Also reflows motion/README.md hard-wrapped prose that
failed the markdown quality gate.
* fix(build): finish pnpm 11 migration + regen verb tables
The panel Docker image build failed on `pnpm install --frozen-lockfile`:
node:22-alpine's corepack resolved to its bundled pnpm 11, but
panel/package.json pinned packageManager to pnpm@10.25.0, and pnpm 11
refuses to run against that pin. The Dockerfiles were already written for
pnpm 11 (comments, CI=true, strictDepBuilds); the package.json pin was the
stale outlier. Finish the migration instead of working around it:
- panel/package.json: packageManager pnpm@10.25.0 -> pnpm@11.10.0; drop the
`pnpm` field (pnpm 11 ignores it — build approval lives in
panel/pnpm-workspace.yaml's allowBuilds). Lockfile unchanged (pnpm 11
accepts it as-is); frozen-lockfile verified.
- remotion-renderer/package.json: pin packageManager pnpm@11.10.0 for
determinism (was relying on corepack's implicit default); engines.node
>=22.13 (pnpm 11 requirement).
- docker/panel.Dockerfile + docker/remotion.Dockerfile: `corepack prepare
pnpm@11.10.0 --activate` so the build uses the pinned version explicitly
instead of trusting corepack's bundled default (which a future
node:22-alpine could change).
- .github/workflows/panel-ci.yml: Node 20 -> 22 (pnpm 11 requires
Node >=22.13; Node 20 fails the engines check).
Also regenerate agents/prompts/_generated/{developer,head_marketing,verbs}.md
— the video engine added propose_video and extended propose_feature_spotlight
(wants_video, video_script) but the verb tables weren't refreshed, failing
the foundation-check quality gate.
* chore(build): approve esbuild build script in remotion pnpm-workspace.yaml
pnpm 11 generated this file with a placeholder ('set this to true or false')
during install; resolve it to true so local dev of the renderer doesn't
re-prompt. esbuild's postinstall only verifies the prebuilt platform binary
(@esbuild/<platform> is installed as an optional dep), so approving it is
safe and silences the ERR_PNPM_IGNORED_BUILDS warning.
* fix(build): copy pnpm-workspace.yaml into panel + remotion images
pnpm 11 hard-errors with [ERR_PNPM_IGNORED_BUILDS] (exit 1) when a
dependency ships a postinstall script that isn't approved in
allowBuilds. Both Dockerfiles copied only package.json + pnpm-lock.yaml,
so the build-approval map in pnpm-workspace.yaml never made it into the
image — the remotion image build died on esbuild@0.28.1's postinstall.
Copy pnpm-workspace.yaml alongside the manifests in both images. In
panel, this also drops the --config.strictDepBuilds=false workaround:
with sharp and unrs-resolver now approved, their postinstalls run and
install the platform-specific binaries (previously skipped, leaving
sharp without its @img/sharp-* binary at runtime).
Verified locally: remotion + panel `pnpm install --frozen-lockfile`
exit 0 with the workspace file present; both exit 1 without it.
* fix(perf): offload conventions + release-readiness blocking I/O off the event loop
The orchestrator runs uvicorn and the orchestration background loops on a
single shared event loop, so any sync I/O anywhere — even inside a background
loop — blocks API responsiveness for its duration. Two call sites were missing
asyncio.to_thread wrappers:
- ConventionsService.get_map/health/restore called the sync _resolve
(`git rev-parse`), _read_committed_standard (file read + yaml parse), and
_derive (filesystem walk via derive_from_scan) inline. Reachable from
GET /api/projects/{id}/conventions and from the agent spawn-prepare path.
- ReleaseManagerEngine._production_assess called gather_snapshot inline —
multiple `subprocess.run` git calls + a filesystem walk, running inside the
release-manager background loop.
Wrap each blocking call in asyncio.to_thread at the async boundary. No
signature changes; helpers stay sync. Verified: targeted tests pass
(196 passed, 36 DB-skipped), ruff + format clean.
These were the only responsiveness gaps surfaced by the concurrency audit —
the rest of the heavy paths (agent spawn via `docker run -d`, video render
loop, git ops via the 16-worker ThreadPoolExecutor, workspace subprocess
calls) already offload correctly. No API/worker container split needed.
* feat(storage): add MinIO config + dep + compose (no-op, default-off)
Chunk 1 of the MinIO video-storage plan (§1, §2, §6). No behavior change:
minio_endpoint defaults to empty = disabled, the existing FileResponse serve
path is untouched (chunk 4 wires the serve path; chunk 2 adds the client).
- pyproject.toml: add `minio` (minio-py) to dependencies; regenerate uv.lock
(resolves minio v7.2.20 + pycryptodome transitive).
- roboco/config.py: add 5 settings fields after video_output_dir
(minio_endpoint/_access_key/_secret_key/_bucket/_region). Plain str Fields
matching the existing ROBOCO_ENCRYPTION_KEY style; no SecretStr, no
presign_ttl_seconds (YAGNI — we don't presign in phase 1).
- docker-compose.yml: add `minio` service (data network only, named
minio-data volume, host ports 19000/19001 for debugging, mc healthcheck)
and a one-shot `minio-init` service mirroring the ollama-init pattern
(mc alias set + mb -p, idempotent via || true). Add ROBOCO_MINIO_* env to
the orchestrator env block (endpoint, access/secret key, bucket, region).
- docker-compose.registry.yml: intentionally omit the minio/minio-init
services and leave ROBOCO_MINIO_* unset (NAS default-on, registry
default-off — the established pattern); comment added to the orchestrator
env block noting the omission.
* docs(storage): 0.19.0 CHANGELOG + RAG + map reference for MinIO chunk 1
Backfills the release-polish docs for MinIO chunk 1 (§10 of the plan):
- docker-compose.yaml synced to docker-compose.yml (the two NAS compose files
must stay byte-identical; .yml was edited in chunk 1, .yaml was stale).
- CHANGELOG [0.19.0]: Added (MinIO scaffolding) + Fixed (event-loop I/O offload).
- docs/rag/architecture/minio-storage.md: RAG doc mirroring video-engine.md.
- docs/map/deployment-tooling.md: one-line storage reference.
* MinIO chunk 2: minio_client module (singleton + unconfigured guard) (#309)
* feat(storage): minio_client module (singleton + unconfigured guard)
Chunk 2 of the MinIO plan (§3). roboco/services/minio_client.py adds:
- get_client(): singleton minio-py Minio from settings; returns None when
minio_endpoint is empty (the disabled path used by the chunk 3/4 guards).
Parses http://... endpoint into host:port + secure flag.
- put_object(bytes, key): no-ops when unconfigured; otherwise PUTs to
settings.minio_bucket with ContentType video/mp4.
- get_object_stream(key): yields object bytes for StreamingResponse; lets
S3Error propagate so the serve route (chunk 4) can fall back to disk.
Sync calls — every call site wraps in asyncio.to_thread (chunks 3/4). One
unit test covers the unconfigured guard + endpoint scheme parsing (mocks,
no real MinIO). Not yet wired into remotion_client._save or the media route.
* MinIO chunk 3: wire write path (remotion_client._save PUT) (#310)
* feat(storage): wire MinIO write path in remotion_client._save
Chunk 3 of the MinIO plan (§3). After the local mp4 write, _save PUTs the bytes
to MinIO under key = Path(mp4_path).name (already {render_key}-{orientation}.mp4),
guarded by minio_client.get_client() (None when minio_endpoint empty) and
wrapped in asyncio.to_thread. Local disk stays the source of truth for the
poster publish path (x_video_client/tiktok_client read mp4_path from disk);
the PUT is additive. _save still returns the local path str — mp4_paths,
marker, and schema unchanged. Disabled (local-only) when MinIO unconfigured.
One test: asserts put_object is called with the basename key when configured
and the local file is still written; existing test stays green via the
unconfigured-default path. Mocks only.
* fix(storage): make MinIO PUT non-fatal in remotion_client._save
A configured-but-down MinIO made put_object raise inside the worker thread,
failing the render and retry-looping a task whose local file was already
written. Local disk is the source of truth and the serve route falls back to
FileResponse on S3Error, so a failed durable-copy PUT must never fail the
render — log and continue; the next render re-attempts the PUT.
Adds test_save_swallows_minio_put_failure (PUT raises -> _save still returns
the local path and the local file is written). Extends the CHANGELOG write-
path bullet with the non-fatal guarantee.
* MinIO chunk 4: serve path (StreamingResponse + FileResponse fallback) (#311)
* feat(storage): serve MinIO via the media route (StreamingResponse + FileResponse fallback)
Chunk 4 of the MinIO plan (§4 — the crux). GET /api/video/posts/{id}/media
derives key = Path(mp4_path).name and, when minio_endpoint is set, returns a
StreamingResponse over minio_client.get_object_stream(key), keeping
_require_ceo so auth stays end-to-end (no presigned URLs). Falls back to
FileResponse on S3Error (old render not in MinIO) or when MinIO is
unconfigured — the panel's axios-blob flow is unchanged (same URL, headers,
body, just chunked). The confinement check is kept as defense-in-depth (the
key is a basename so traversal is impossible, but the check is cheap and
protects the poster path).
Two integration tests: configured serve path streams from a stubbed
get_object_stream (CEO 200, non-CEO 403); unconfigured fallback serves the
local file via FileResponse. Mocks only — no real MinIO.
* fix(storage): eager stat_object probe so the MinIO serve fallback actually fires
The chunk-4 route wrapped StreamingResponse(get_object_stream(key), ...) in a
try/except, but get_object_stream is a lazy generator — its client.get_object
call runs on the first next(), i.e. AFTER the route returned and Starlette
started streaming. An S3Error (NoSuchKey / MinIO down) there is uncatchable;
the try/except caught nothing and the FileResponse fallback never triggered.
Add minio_client.stat_object(key): an eager existence/readiness probe that
runs INSIDE the route's try/except, so a missing object or down MinIO raises
before the StreamingResponse starts and the fallback serves the local file.
stat-then-get is two round trips; a mid-stream failure after a successful stat
is a rare race the CEO can retry (documented ceiling).
Tests: the configured test now stubs stat_object; a new test asserts the
S3Error fallback serves the local file via FileResponse and that
get_object_stream is never called. RAG doc updated to record the eager-probe
correctness detail + the non-fatal PUT.
* docs(rag): mark MinIO deployment note landed (chunk 5) (#312)
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* fix(video): offload minio stat_object off the event loop
stat_object was called inline in the async media route, blocking the
shared event loop for one sync urllib3 round-trip per preview request —
contradicting minio_client's own 'every call site wraps in to_thread'
docstring and this PR's perf-fix theme. Wrap in asyncio.to_thread; the
try/except still catches S3Error (to_thread re-raises) so the
FileResponse fallback is unchanged. Also add the trailing newline to
the minio-storage RAG doc.
* Fix red CI
* Make CI green
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* feat(video): Phase A — VideoEngine origination spine + held-source gates
New default-off engine skeleton: opens a UX/UI authoring task (source=video, assigned to a ux-dev, LOW complexity to clear the dev-needs-subtasks guard) and materializes a held CEO-approval draft (source=video_post). Excludes video_post from all three held-source skip sites; adds the video_draft marker, six config flags, and the feature-flag entries. Origination + gate behavior unit-tested.
* refactor(orchestrator): fold _dispatch_dev_work skip chain into a helper
The per-source if/continue chain grew past xenon's --max-absolute B when the video_post held source joined it. Extract _is_non_dev_dispatch_source (every held-CEO source plus the two Board exploration sources) so the dev loop's skip is one flat call. Behavior-identical.
* feat(video): Phase B — propose_video do-tool (metadata-only, team-gated)
UX/UI dev records a video's composition ref + per-platform captions onto the authoring task's video_draft marker. Team-gated at runtime via _caller_team (Role.DEVELOPER can't tell a ux-dev from a be-dev). Resolves the caller's ACTIVE task via get_active_task_for_agent, not an oldest-first scan that would clobber a second open video task. Metadata only, no render. Wired through do_server + route + schema; added to _DEV_DO.
* feat(video): Phase D — render loop + RemotionRenderer client
Orchestrator-async _video_render_loop renders a completed authoring task's merged composition to MP4 (vertical + square) via the remotion-renderer sidecar and materializes the held video_post draft. RemotionRenderer tars the read-clone's motion/ source, POSTs it, and saves the returned MP4 bytes to a TASK-scoped local path (no shared volume; a composition is reused across videos so a composition-scoped path would clobber an earlier draft). Render failures bounded-retry (read-clone catch-up window, transient sidecar) up to a cap, then terminal-fail. Client tested vs a mock transport; loop vs a mock renderer + real DB.
* feat(video): Phase C — release / spotlight / on-demand video triggers
Three entry points open a UX/UI video-authoring task via VideoEngine.open_video_task: (1) a published release drafts a companion video — best-effort in ReleaseProposalService.approve, never fails the publish; script from the CHANGELOG via the local model with a template fallback. (2) propose_feature_spotlight gains optional wants_video/video_script — best-effort, gated on video_on_spotlight, default-off leaves the spotlight flow byte-for-byte unchanged. (3) POST /video/request (CEO-only) for an on-demand brief, with clean disabled/not_opened responses. All gated on video_engine_enabled.
* fix(video): savepoint-isolate video-task inserts (F042 poisoned session)
The best-effort try/except around open_video_task (release-publish + spotlight hooks) swallowed the Python exception, but a DBAPI error at the insert flush left the shared session must-rollback — so the caller's next commit (release finalize / request boundary) threw PendingRollbackError: the release stuck 'pending' after actually publishing, or the spotlight draft + HTTP response were lost. Wrap both inserts (open_video_task, _originate_video_post) in a begin_nested savepoint (the repo's established F042 pattern) so a DB error rolls back only the insert. open_video_task returns None (every caller already handles it); _originate_video_post propagates to the render loop's handler. Regression test: an insert FK error returns None with the session left usable. Dormant while the flags were off; armed on the NAS.
* feat(video): Phase G — motion/ package + remotion-renderer sidecar + compose
In-repo Remotion v4 motion/ package (ReleaseAnnouncement composition; calculateMetadata returns 1080x1920 vertical / 1080x1080 square from inputProps.orientation) + a credential-free remotion-renderer sidecar: untar the POSTed motion/ source, bundle (LRU-cached per source sha), selectComposition + renderMedia h264, stream the MP4 bytes back — matching the RemotionRenderer client contract. docker/remotion.Dockerfile on Debian (Chrome apt deps, build-time Chrome pre-warm, ffmpeg bundled in @remotion/renderer). Wired into both compose files (roboco_default only, shm_size 1gb, /health check) + the release publish matrix. Verified via a real local render of both cuts; the Debian docker build is the CEO's to run.
* chore(video): D-hardening — video_post source_task_id + render-loop docstring
Add a source_task_id back-reference to the video_post held-draft marker (traceability from a draft to its authoring task; also makes the render loop's two-key idempotency check wireable later). Fix the render-loop test's stale docstring ('never retried' -> bounded-retry). Both from the Phase D critic's non-blocking follow-ups.
* feat(video): Phase E1 — VideoPostService + heartbeat mutex (approve->post)
CEO-approve->post service: heartbeat-renewed Redis mutex (fail-closed, grace=ttl-2*heartbeat), re-read-in-lock double-post guard, per-platform durable commits (asyncio.shield-ed, settle-before-rollback on lock-loss), all writes inside the lock (captions validated pre-lock, applied in-lock — no stale whole-column clobber), idempotent, per-platform retry-skip. Poster interfaces (X/TikTok, mocked here). Reject + list-held-drafts. Survived 3 adversarial rounds; residual = a crash in the poster->commit window (CEO-gated low-freq, documented).
* fix(video): G-hardening — renderer leaks + Share Tech Mono brand font
Sidecar: give bundle() an explicit outDir tracked + deleted on LRU eviction (was leaking ~19MB remotion-webpack-bundle-* per source); res.on('close') cleanup so an aborted/retried download no longer leaks its remotion-out-* MP4 dir. Fonts: vendor Share Tech Mono (roboco-website brand font) as the display face (self-hosted woff2, 400-weight, headline fontWeight 700->400 to avoid faux-bold) + self-hosted Inter body — no gstatic fetch at render time (lsof-verified). Extras: composition_id whitelist (400) + Multer error middleware (400/413).
* feat(video): Phase E2 — X v2 + TikTok posters, tiktok_credentials, routes
LiveXVideoPoster (X v2 chunked media upload: init/append/finalize/STATUS-poll -> tweet w/ media_ids, OAuth1 signer reused). LiveTikTokPoster (OAuth2 inbox: init -> chunked PUT with asymmetric final chunk -> status-fetch; 401 -> refresh_token grant, rotated token persisted). tiktok_credentials Fernet singleton + migration 062 (single head). Routes: CEO approve/reject + list held drafts + write-only tiktok creds, wiring real posters into VideoPostService. Residual: a lock-loss right after a token-refresh flush can discard the rotated token (same rare CEO-gated class as the documented post->commit window).
* feat(video): Phase F — panel video-post queue + TikTok creds card + flags
video-post-queue.tsx: <video> MP4 preview with 9:16/1:1 cut switch, per-platform editable captions (280/2200 counters, over-limit disables approve), approve/reject, Request-a-video dialog. tiktok-credentials-card.tsx (4 write-only OAuth2 fields). feature-flags-card inlines TikTokCredentialsForm under video_engine_enabled. Mounted in command-center. tsc/eslint clean, 273 panel tests green. NOTE: needs the GET /video/posts/{id}/media route + mp4_paths on VideoPostResponse (folded into H) for the preview source.
* feat(video): Phase H — media route + e2e smoke + NAS arming + docs
GET /video/posts/{id}/media?cut= (CEO-gated FileResponse of the rendered MP4; closes the panel preview gap) + mp4_paths on VideoPostResponse. e2e smoke tests/e2e_smoke/test_video_pipeline.py (full flow, sidecar+X/TikTok mocked; asserts dispatcher skips, render-loop materialize, propose_video team-gate, approve idempotency). NAS arming: docker-compose.yml/.yaml ROBOCO_VIDEO_ENGINE_ENABLED/ON_RELEASE/ON_SPOTLIGHT default-on (.yaml resynced to .yml); registry stays off. CLAUDE.md video-engine section + CHANGELOG. Fixed 2 pre-existing route-test pollution leaks. Full suite 11763 passed.
* fix(video): auth-carrying preview, media route confinement, VideoPost type drift
Three fixes along the video preview path:
1. panel video preview auth: the <video> element was pointed straight at
GET /video/posts/{id}/media, but a native <video src> GET carries none
of axios's X-Agent-ID/X-Agent-Role headers — so in the default
header-trust deployment the request 401s. Fetch the cut via
videoApi.getMediaBlob (axios, responseType: blob) and drive <video>
off a URL.createObjectURL result instead. The object URL is revoked
on cut-change (the previous cut's URL) and on unmount, so neither
cut switches nor row teardown leak blob URLs.
2. backend media route confinement: GET /video/posts/{id}/media now
resolves mp4_path and refuses it with 404 when it falls outside
settings.video_output_dir. Defense-in-depth against any future
writer of mp4_paths serving files from arbitrary disk locations.
3. panel VideoPost type/comment drift: added mp4_paths to the
VideoPost interface (the committed VideoPostResponse already
carries it), and corrected the stale comment on videoMediaUrl
that claimed no route served the rendered bytes — the route has
existed since the media endpoint landed; the comment now describes
why getMediaBlob exists instead of a direct <video src>.
* Persist rendered videos to data in physical storage.
* ++
* docs(video): 0.18.0 CHANGELOG entry + RAG + map reference for video engine
- Move the video engine bullet from [Unreleased] into [0.18.0] and note
the ROBOCO_VIDEO_OUTPUT_DIR bind-mount persistence.
- Add docs/rag/architecture/video-engine.md (mirrors x-engine.md shape:
enable/disable, three triggers, render loop + sidecar, CEO gate, media
route confinement, credentials).
- Reference the video render loop in docs/map/orchestrator.md's engine list.
* chore(video): re-bump to 0.19.0 + sync registry compose defaults
Version was wrongly bumped to 0.18.0; 0.18.0 is an already-released
section. Restore its 2026-07-04 date and move the video-engine CHANGELOG
bullet into a new [0.19.0] - 2026-07-05 section above it. Bump
pyproject.toml, roboco/__init__.py, roboco/config.py (app_version),
panel/package.json, and the motion/README inputProps example to 0.19.0.
docker-compose.registry.yml: add ROBOCO_VIDEO_ENGINE_ENABLED /
_VIDEO_ON_RELEASE / _VIDEO_ON_SPOTLIGHT defaulted false (NAS arms them
true), and comment out the video-renders bind mount with a short note
so the public registry image ships video off by default. Structural
sync with docker-compose.yml maintained.
* fix(video): rate-limit /render + reflow motion/README
CodeQL flagged js/missing-rate-limiting on the renderer /render route.
The sidecar is container-network-only with one trusted caller (the
orchestrator, which renders cuts serially), so this limiter is a
retry-storm ceiling (30/min, well above legit render rate), not the
primary control. Also reflows motion/README.md hard-wrapped prose that
failed the markdown quality gate.
* fix(build): finish pnpm 11 migration + regen verb tables
The panel Docker image build failed on `pnpm install --frozen-lockfile`:
node:22-alpine's corepack resolved to its bundled pnpm 11, but
panel/package.json pinned packageManager to pnpm@10.25.0, and pnpm 11
refuses to run against that pin. The Dockerfiles were already written for
pnpm 11 (comments, CI=true, strictDepBuilds); the package.json pin was the
stale outlier. Finish the migration instead of working around it:
- panel/package.json: packageManager pnpm@10.25.0 -> pnpm@11.10.0; drop the
`pnpm` field (pnpm 11 ignores it — build approval lives in
panel/pnpm-workspace.yaml's allowBuilds). Lockfile unchanged (pnpm 11
accepts it as-is); frozen-lockfile verified.
- remotion-renderer/package.json: pin packageManager pnpm@11.10.0 for
determinism (was relying on corepack's implicit default); engines.node
>=22.13 (pnpm 11 requirement).
- docker/panel.Dockerfile + docker/remotion.Dockerfile: `corepack prepare
pnpm@11.10.0 --activate` so the build uses the pinned version explicitly
instead of trusting corepack's bundled default (which a future
node:22-alpine could change).
- .github/workflows/panel-ci.yml: Node 20 -> 22 (pnpm 11 requires
Node >=22.13; Node 20 fails the engines check).
Also regenerate agents/prompts/_generated/{developer,head_marketing,verbs}.md
— the video engine added propose_video and extended propose_feature_spotlight
(wants_video, video_script) but the verb tables weren't refreshed, failing
the foundation-check quality gate.
* chore(build): approve esbuild build script in remotion pnpm-workspace.yaml
pnpm 11 generated this file with a placeholder ('set this to true or false')
during install; resolve it to true so local dev of the renderer doesn't
re-prompt. esbuild's postinstall only verifies the prebuilt platform binary
(@esbuild/<platform> is installed as an optional dep), so approving it is
safe and silences the ERR_PNPM_IGNORED_BUILDS warning.
* fix(build): copy pnpm-workspace.yaml into panel + remotion images
pnpm 11 hard-errors with [ERR_PNPM_IGNORED_BUILDS] (exit 1) when a
dependency ships a postinstall script that isn't approved in
allowBuilds. Both Dockerfiles copied only package.json + pnpm-lock.yaml,
so the build-approval map in pnpm-workspace.yaml never made it into the
image — the remotion image build died on esbuild@0.28.1's postinstall.
Copy pnpm-workspace.yaml alongside the manifests in both images. In
panel, this also drops the --config.strictDepBuilds=false workaround:
with sharp and unrs-resolver now approved, their postinstalls run and
install the platform-specific binaries (previously skipped, leaving
sharp without its @img/sharp-* binary at runtime).
Verified locally: remotion + panel `pnpm install --frozen-lockfile`
exit 0 with the workspace file present; both exit 1 without it.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
board_triage's idle branch built its briefing without full=True, so
company_goals (brand_voice/north_star, migration 061) never reached the Product
Owner's roadmap-exploration spawn or the Head of Marketing's feature-spotlight
spawn — both always hit the idle branch, yet both spawn prompts claim the
charter is 'already in your briefing'. Added a scoped include_company_goals flag
to _briefing_for (via a _resolve_company_goals helper; xenon B preserved) that
fetches only the cheap charter singleton without full's other heavy sections;
board_triage's idle branch opts in. Strategic branch + auditor untouched. 6 new tests.
The Head of Marketing now markets features, not just releases: a default-off
x_feature_spotlight loop periodically spawns the HoM to investigate what shipped
(CHANGELOG, feature flags, docs/map, KB) and draft ONE held marketing post via
propose_feature_spotlight, reviewed in the X post queue.
- New x_feature source (distinct from x_post, fixing panel mislabeling) + a
panel Feature-spotlight branch.
- brand_voice column on company_goals (migration 061, single head) as the
CEO-editable voice source, surfaced in Settings and injected into the HoM
briefing; a VOICE GUIDE baseline in head-marketing.md.
- propose_feature_spotlight verb (HoM-only), mirroring propose_roadmap.
Gated by x_feature_spotlight_enabled (default off; flag-off dormancy proven).
Also fixed two real bugs found mid-build: company_goals API schemas dropped
brand_voice on GET/PUT; the live charter UI is goals-tab.tsx, not the unmounted
company-goals-card.tsx. Full suite green (2935); migration single-head verified.
Fleet behaves more like Fable 5 on existing model tiers, behind
ROBOCO_FABLE_MODE_ENABLED (config default off; armed :-true on the NAS compose,
absent from the registry compose).
- Doctrine: vendored agents/prompts/doctrine/fable.md composed into every
agent's system prompt via fable_doctrine_layer() after base.md.
- Hooks (Claude Code): 4 non-overlapping hooks (stop-gate/bash-discipline/
honesty-nudge/precompact) appended per-agent via _fable_hook_groups(). The
make-quality + lint-suppression duplicates are deliberately NOT added (already
gate-enforced); session-start skipped.
- Hooks (grok): conservative V1 — only the non-denying honesty-nudge, since a
grok hook deny cancels the whole run.
- Flag on the feature-flags card; hook scripts shipped into the agent image.
Flag-off spawn path proven byte-identical (worktree diff, sha256 match); full
suite green (2074 unit + e2e-smoke + hook harness), mypy/xenon/ruff clean.
Fixed a real stdin bug in the vendored stop-gate hook (heredoc + pipe both
claimed stdin). Distilled from rennf93/opus-fable-playbook (MIT).
#306 removed the channels/sessions/messages subsystem but left dangling
references. Agents were still told to call removed verbs at spawn, and a
maintenance script referenced a dropped table.
- prompts (roles/identities/teams/base): drop say/open_session/link_session/
channels() and the dead Channels sections; comms rows now teach A2A
(dm + read_a2a); renumber the PM workflow steps the removed open_session
step left behind
- scripts/reset_runtime_state.sql: drop the dropped chunks_conversations table
- models/events.py: mark the retired SESSION_*/MESSAGE_SENT enum members inert
- mcp/{do,flow}_server.py: drop the dead SESSION_CLOSED error-map key
- panel: drop channels_read/write from AgentPermissions; remove dead channel:
KB source branch
- pyproject.toml: refresh a ruff-exemption example off the removed verb kwargs
- CHANGELOG: record #306 under [Unreleased]
* feat(sandbox): throwaway per-agent Postgres/Redis sandbox containers
Orchestrator-provisioned sibling containers per agent spawn
(SandboxProvisioner, roboco/runtime/sandbox.py). Per-project opt-in via
projects.sandbox_services (migration 057); master switch
ROBOCO_SANDBOX_DB_ENABLED, default-off, armed in the NAS compose only.
When active, ROBOCO_TEST_DB_* / ROBOCO_TEST_REDIS_* point at the sandbox
and the prod-creds gate-env injection is suppressed (sandbox replaces,
never coexists). Sandbox lifetime tracks the agent container: teardown at
every removal path, orphan janitor at startup + each reaper tick with a
grace window for mid-flight spawns. The pre-spawn stale-clear spares the
just-provisioned sandbox; provision pre-clears stale same-named
containers from a crash-missed teardown.
Panel: per-project sandbox-service switches in the edit dialog + feature
flag card entry.
* docs: CLAUDE.md entry for the sandboxed dev DB/Redis subsystem
* feat(security): isolate prod Postgres/Redis from agent containers (roboco_data network)
Second user-defined bridge roboco_data carries postgres+redis only; the
orchestrator is multi-homed (default + data). Spawned agents and their
sandbox sidecars stay on roboco_default and can no longer resolve or
reach roboco-postgres:5432 / roboco-redis:6379 (redis has no auth —
membership is its only containment). Normal bridge, so host-published
ports (15432/16379) keep working. Applied to both build composes and
the registry compose; docker-compose.yml re-synced byte-identical with
docker-compose.yaml (it had drifted by the sandbox flag block).
ROBOCO_DB_NETWORK_ISOLATED (config default false, armed alongside the
topology) suppresses the legacy _append_gate_env prod-creds injection:
under isolation those creds dead-end, and unreachable creds are worse
than none. DB-needing projects opt into sandbox_services instead. The
flag is deliberately not a panel feature flag - it must travel with the
compose networks: stanzas.
Preserved by construction: agent<->agent A2A and orchestrator->agent SDK
polls on :9000, MCP->orchestrator on :8000, ollama reachability, docker
exec/inspect (daemon socket), host port publishing.
* feat(panel): full mobile responsiveness pass
Shared primitives: useIsMobile (useSyncExternalStore, hydration-safe,
memoized matchMedia subscribe), ResponsiveTable table->card switch below
md (single subtree mounted, no duplicated interactive rows), scrollable
snap TabsList in the base primitive (justify-center-safe so the first
tab stays reachable on overflow), persistent md:hidden bottom tab bar
(Overview/Tasks/Kanban/Chat, safe-area padded).
Applied: card lists for tasks/projects/products/work-sessions/sessions
+ the three raw metrics tables; CEO approval queue / release proposal /
playbook review action rows stack on narrow; command-center reorders
approvals above the fold on mobile; task-header metadata wraps;
Communications + A2A become URL-driven single-pane drill-downs below lg
(fixes the unconstrained-height ScrollArea bug) with dvh heights;
recharts label density/radius adapts via useIsMobile; git diff viewer
gets mobile font + wrap toggle; vh->dvh sweep; chat composers get
safe-area-inset padding; dashboard main p-4 md:p-6 + pb-20 for the bar.
Verified at 375px on the built app: bottom bar, drawer, approval-first
overview, swipeable kanban tab strip. All gates green (eslint, tsc,
vitest 249, next build 24/24 routes).
* feat(auth): cloud auth via FastAPI Users (default-off, single-user cookie session)
ROBOCO_CLOUD_AUTH_ENABLED (default off) lets the panel/API be exposed
beyond localhost without changing the CEO's local no-login flow while
off — get_agent_context and the WS gate are byte-for-byte unchanged in
off-mode. On: header-trust dies for humans — any agent-role claim (ceo
or a privileged PM/board role) with no valid HMAC token or session
cookie is 401, closing the header-spoof hole on the host-published
:8000 port for every role. The agent-fleet HMAC path and the system
self-PATCH keep working unmodified in both modes.
Single seeded CEO user (migration 058 users table, UserTable), no
registration router — idempotent env-driven upsert at startup by PK.
Cookie transport (httponly/secure/samesite=lax) + a JWTStrategy bound
to a fingerprint of the current password hash (rotating the password
invalidates every prior session). Sliding 30-day session: every
authenticated request re-mints the cookie, so an active session never
expires — no unexpected logouts.
Panel: (auth)/login page + proxy.ts (Next 16 rename of middleware; probes
/auth/status over the docker-internal URL, fails open to off) gate the
dashboard; client.ts gets withCredentials + 401->/login. nginx unchanged.
Review hardening: broadened the on-mode rejection from ceo-only to every
non-CEO role without a valid token (was only closed when
ROBOCO_AGENT_AUTH_REQUIRED was also armed); Next-16 proxy.ts rename to
clear the middleware deprecation warning.
* feat(x): RoboCo X account engine — HoM drafts, per-post CEO approval (default-off)
ROBOCO_X_ENGINE_ENABLED (default off, inert without creds). Mirrors the
ReleaseManagerEngine held-artifact shape: XEngine drafts a post when a
release publishes (via a draft_release_post seam on ReleaseProposalService
.approve) and drafts replies to meaningful mentions (dedicated poll loop,
x_seen_mentions dedup ledger, per-cycle/open caps). Drafting is
local-model-only, clamped to 280 chars. Nothing auto-posts — every tweet
is a held task (source x_post/x_reply, confirmed_by_human=False,
Secretary-owned, dispatcher-skipped) the CEO edits/approves/rejects in a
panel queue.
The four OAuth 1.0a secrets live Fernet-encrypted in a singleton
x_credentials row (migration 059, all-or-nothing, API returns only
has_credentials); decryption is server-side, agents never hold creds or
egress. Hand-rolled OAuth 1.0a HMAC-SHA1 signer, no new dependency;
NullXClient makes the unconfigured path a graceful no-op.
XPostService.approve (CEO-only) is the sole caller of post_tweet.
Review hardening: closed a double-post race — the approve path now
re-reads committed task state inside the Redis lock and commits COMPLETED
before releasing, so a concurrent approve that acquires the lock after the
winner released can't re-post (SET-NX is non-waiting, and the route-level
commit landed after the lock dropped). Added a regression test.
* feat(roadmap): board roadmap engine — PO proposes themed cycles, CEO approves per-item (default-off)
ROBOCO_ROADMAP_ENGINE_ENABLED (default off). Weekly, RoadmapEngine opens
ONE held exploration task (source=board_roadmap, confirmed_by_human=False,
Product-Owner-assigned), deduped to one open cycle. A dedicated one-shot
_dispatch_roadmap_exploration spawns the PO solo (not the two-reviewer
board path, which would also spawn HoM + fire Approve-&-Start). The PO
explores read-only (git/KB/metrics/releases/charter/web) and makes one
propose_roadmap call (PO-only content verb) authoring a themed cycle —
goal + 3-7 item drafts — persisted as a roadmap_cycle marker (no table,
no migration; head stays 059).
The CEO acts per-item in the panel roadmap queue: approve materializes a
BACKLOG task (source=roadmap, no assignee — never auto-starts), reject
records a reason; all-items-terminal completes the exploration task.
RoadmapService is idempotent per item. Dispatchers skip board_roadmap.
Includes a real SQLAlchemy dirty-check fix (deep-copy the JSON marker
before mutating, or the in-place edit + reassign compares equal to its
own baseline and the UPDATE is skipped).
Review hardening: create_task_from_draft now honors a draft-declared
source only from a {prompter, roadmap} whitelist — drafts are
LLM-authored, so an unbounded source could impersonate a privileged
origin (release_manager would even wedge that engine's dedup).
* chore(release): 0.17.0
Wave 3 — six default-off subsystems: sandboxed dev DB/Redis, prod
Postgres/Redis network isolation, full mobile UI pass, cloud auth
(FastAPI Users), the RoboCo X account engine, and the board roadmap
engine. Plus the waves 1+2 work already on master since 0.16.0.
Version bumped across the canonical set (config.py, __init__.py,
pyproject.toml, panel/package.json, uv.lock); CHANGELOG [Unreleased]
cut to [0.17.0]; docs/map delta added.
Compose: every optional feature armed :-true in the NAS composes, OFF
in the user-facing registry compose. Two opt-in exceptions default off
(CLOUD_AUTH — needs email/password/secret + TLS, would otherwise fail
startup; ROUTING_STRICT — fail-closed spawning). DB_NETWORK_ISOLATED
stays on in both (coupled to the roboco_data topology).
* chore(compose): arm cloud_auth + routing_strict ON in the NAS composes
Every feature defaults ON in the NAS composes per policy — these two
were wrongly left off. Both keep the ${VAR:-true} form so the operator
controls the real runtime via .env: cloud auth needs
ROBOCO_CLOUD_AUTH_EMAIL/_PASSWORD/_SECRET + TLS set there before a boot
(else startup fails loud), and routing_strict is fail-closed. Registry
compose keeps both off.
* fix(ci): reflow board.md prose (quality gate) + document v0.17.0 env creds
The roadmap section added hard-wrapped prose that failed the markdown
prose gate; reflowed (token-invariant). Also brought .env.example
current: cloud auth (now armed — needs SECRET or startup fails), routing
strict, the X engine (panel-entered OAuth), and web research.
* fix(ci): reduce cyclomatic complexity of five wave-3 blocks (xenon gate)
The wave-3 subagents introduced C-rank functions the CI xenon gate
rejects (my per-item reviews ran ruff/mypy/pytest but not xenon):
- sandbox.janitor_sweep -> extract _list_labeled_sandboxes /
_list_live_agent_containers / _prune_grace
- x_client.fetch_mentions -> extract _parse_mention_items
- x_engine.run_cycle -> extract _process_mentions
- orchestrator._dispatch_pm_work -> extract the source-skip into a
MODULE-level _is_held_ceo_source (module, not method, so the
wholesale-mocked dispatcher unit tests exercise the real logic)
- auth/seed.ensure_seed_user -> extract _apply_seed_updates (module avg -> A)
Behavior-preserving; full suite green (11902), xenon clean.
* fix(ci): declare pyjwt + fastapi-users-db-sqlalchemy as direct deps (deptry)
The cloud-auth code imports jwt and fastapi_users_db_sqlalchemy directly
but they were only transitive deps (via fastapi-users), which deptry
(quality gate, DEP003) rejects. Declared explicitly; deptry roboco/ clean.
Missed originally because local make quality stopped at earlier gates
before reaching deptry.
* feat(x): gate mention replies behind ROBOCO_X_REPLIES_ENABLED (default off)
Per CEO decision: the X engine should only post about releases by
default. Reading mentions needs a paid X API tier, so the mention-reply
half is now a deliberate opt-in on top of release posting.
New default-off flag x_replies_enabled gates the mentions poll loop
(_x_mentions_poll_loop) and XEngine.run_cycle; release-post drafting
(the release-proposal approve hook) is unaffected and still runs when
x_engine_enabled + credentials are set. Added to FEATURE_FLAGS + the
panel card. Tests: release posting works with replies off; run_cycle +
the poll loop are no-ops with replies off.
* fix: 401 only redirects to /login when cloud auth is on; panel-token strips .env quotes
Two bugs that together dead-ended login in secure mode:
- client.ts redirected to /login on ANY 401, so a mismatched panel
token (header-trust/secure mode, cloud auth off) bounced the user to a
login page whose backend route isn't mounted -> 404. Now it probes
/auth/status (bare fetch, no interceptor re-entry) and only redirects
when cloud_auth_enabled.
- make panel-token read the .env secret with grep|cut without stripping
surrounding quotes, so a quoted ROBOCO_AGENT_AUTH_SECRET produced a
token signed with the quotes included — which never verifies against
the orchestrator (docker-compose/pydantic unquote the secret). Now
strips surrounding single/double quotes.
* fix: git-log 500 on '|' in commit message; X queue shows an empty state
- GET /api/git/log 500'd (ValueError: Invalid isoformat) when a commit
SUBJECT contained a '|' (e.g. the 'curl|sh' lockdown commit): the
fixed '|' field delimiter let the subject's pipe shift the split so
author+date collapsed into one field. Switched to \x1f (Unit
Separator), which can't appear in commit content. Regression test with
a piped subject.
- The X Post Queue returned null when empty, so there was no visible
place for the X drafts. It now renders a discoverable empty state
pointing at Settings -> X credentials.
* docs: bring docs/rag + docs/map current for v0.17.0 (waves 1-3)
Agent-facing RAG corpus and codebase map updated for every feature in
the 0.17.0 span, code-verified:
- wave 3: sandbox DB, DB network isolation, cloud auth, X engine
(+ x_replies_enabled sub-flag), board roadmap engine — new RAG
architecture pages + role/tool/config-reference updates; new symbols,
migrations 057-059, panel surfaces, and the get_agent_context
dual-path across the map slices.
- waves 1-2: A2A live view + switchboard, prompter memory
(search_past_tasks), Secretary edit access + PM-lighter scope, the
PR-gate auto-submit turn cut (ROBOCO_PR_GATE_AUTO_SUBMIT_ENABLED).
- correctness fix: api-routes-schemas.md no longer claims the A2A admin
routes are reachable by any authenticated agent — they carry a
_require_ceo gate (wave 2c).
docs/internal, _front.md deltas, and the frozen _complete_map.md
snapshot untouched.
* fix(rag): atomic upsert for indexed-doc tracking (kills e2e segfault)
The indexed-document tracking write used check-then-insert in two paths
(IndexedDocumentRepository.upsert_batch and the file-source
_upsert_doc_record). Under concurrent indexing both callers saw no row
and both inserted, so the second violated uq_indexed_doc_source and
poisoned its transaction — surfacing in CI as the intermittent
_checkin_failed SIGSEGV on the failed connection's pool checkin.
Both paths now use INSERT ... ON CONFLICT DO UPDATE against the
constraint: coalesce keeps an existing title/preview when the new value
is empty (matching the old guards) and metadata is jsonb-merged. The
batch dedupes within itself first (ON CONFLICT can't touch a row twice
in one statement). expire_all after the Core upsert keeps same-session
ORM reads consistent with the merged DB row.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* fix(security): lock down shared Claude Code credential mount + curl|sh RCE
Audit of Claude Code capabilities reachable inside a spawned agent
container turned up two live gaps against the shared harness state:
- Every agent container bind-mounts the host's ~/.claude (OAuth store) and
~/.claude.json read-write (_build_mount_args) — the shared subscription
auth used by the whole fleet. Nothing denied the native Read tool or the
bash-guard hook from reading .credentials.json / .claude.json, so any
role could exfiltrate the harness's own Claude Code auth. Deny both at
the settings.json layer (absolute // form, per the #167 gotcha) and in
the bash-guard hook's credential-exfil checks (cat/grep/source/base64/
interpreter one-liners), mirroring the existing .netrc/.git-credentials
treatment.
- The bash-guard hook only blocked curl/wget to github.com or internal
hosts; `curl <any other host>/install.sh | bash` (or `bash <(curl ...)`,
`eval "$(curl ...)"`) executed untrusted remote code unchecked. New
checks deny piping a fetch into an actual shell (sh/bash/zsh/dash/ksh)
while leaving non-executing consumers (tar, jq, -o file) untouched.
Also add --disable-slash-commands to every container agent spawn: skills
resolve independently of the --tools allowlist, so a contaminated shared
~/.claude could otherwise leak host skills/plugins into an agent session.
No RoboCo role's workflow uses a Claude Code skill.
64 -> 78 shell bash-guard cases, 54 -> 71 pytest bash-guard cases, plus a
new 5-case settings/CLI test module. ruff/mypy/xenon B clean.
* docs: changelog for the CC capability lockdown
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* feat(tasks): Secretary full task access; PM lighter editing — and a closed over-permission hole
Secretary: the CEO-gated edit directive covers the full content surface
(title/description/AC/priority/team/complexity/nature + claim-aware
reassignment through the real reassign paths, enum coercion, slug or
UUID assignees), and read_task returns full detail (notes, plan,
bounded progress, PR refs). The submit_directive tool docs never
mentioned edit at all — fixed, it was undiscoverable.
PMs: scouted the PATCH route and found has_higher_perms gave PM
identities UNRESTRICTED admin (ASSIGN is not team-scoped) — wider than
'not that much'. Now: cell PMs hard-403 outside their team, and both PM
roles are capped to the content allowlist (title/description/AC/
priority) with zero status changes via this surface. CEO/Board/Auditor
keep full admin. Built subagent-driven (Sonnet 5), reviewed.
* feat(a2a): the switchboard — org-chart pair cards with live activity
70 permission-matrix-derived pair cards (cells/pm-chain/board/cross),
lighting on either direction's a2a.message frames with a 45s fade —
A2A only, never verbs, per CEO ruling. Click-through reuses the v1
transcript + chime-in drawer; v1 list stays as the mobile fallback.
One CEO-gated /a2a/chat/admin/pairs route joins the static matrix
against conversations in a single bulk query. Built subagent-driven
(Sonnet 5), reviewed; pre-existing agent-utils slug-map gap flagged.
* fix(runtime): conventions ambient covers the MegaTask project_ids scope
_resolve_intake_ambient forwarded project_ids only to the history-digest
resolver — a MegaTask intake got no architectural-conventions block even
with the flag on. The conventions resolver now takes project_ids first
(mirroring the history resolver), both share one order-preserving
_projects_by_ids helper, and a regression test pins the threading to
both sub-resolvers. Built subagent-driven (Sonnet 5), reviewed.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* feat(a2a): live view — watch fleet conversations, CEO chime-in, reply budget
A2A_MESSAGE_SENT published from A2AService.send (excerpt-capped) and
fanned through the existing /ws/system bridge; CEO-only admin REST for
conversations/messages + a reply route on the publish-bearing send path;
panel /a2a page with live transcript and a composer gated on task-linked
conversations. The matrix gains its one asymmetric rule: CEO may message
anyone, nobody may target the CEO — and agent replies inside a
CEO-opened conversation are hard-budgeted to one per CEO message
(per conversation, per agent), rejected with wait-don't-retry guidance.
Built subagent-driven (Sonnet 5), reviewed; v1 seams documented in the
map delta.
* feat(prompter): intake remembers the task history
Intake spawns now carry a per-project chronological digest of recent
tasks (capped: 15 lines/project, 4000 chars total — ~300-1000 tokens)
merged into the ambient layer, and the interviewer gets a bounded
search_past_tasks tool (one shared implementation behind the grok MCP
tool and the Claude SDK in-process tool) to check precedent
mid-conversation. Informational memory only — the sequencing analyzer
keeps ownership of ordering. Built subagent-driven (Sonnet 5), reviewed;
pre-existing conventions-ambient MegaTask-scope gap flagged, untouched.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* chore(panel): delete the five dead comms components
The comms audit found communications-view, channel-sidebar,
channel-item, message-list, and message-item exported but rendered by
no page — the live /communications page and the session detail render
their own inline content and import only MessageComposer and
MessageTypeBadge, which stay. Verified zero consumers outside the dead
cluster before deletion; panel gates green (tsc, lint, 187 tests).
* feat(tests): e2e scenario 4 — MegaTask umbrella; fix batch root-subtask completion wall
Scenario 4 seeds an umbrella + two dependency-linked root-subtasks:
sequencing hold proven (unmet_dependency on RS2's i_will_plan while RS1
lives), RS1 completed through the entire real chain to a master merge,
hold lifts, RS2 completes, umbrella closes branchless via ceo-approve
and never carries a PR.
Product fix it surfaced on first run: _main_pm_complete_guard and
escalate_to_ceo refused ANY parented task as 'not a root', but a batch
root-subtask is parented (the umbrella) BY DESIGN — both sites now
consult is_batch_root_subtask, plain subtasks stay refused. Live
root-subtasks previously needed CEO god-mode to close. Regression
tests added; built subagent-driven (Sonnet 5) and reviewed.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* feat(tests): e2e scenario 2 — the PM merge chain through the PR gate
Shared arcs extracted (arcs.py: canonical-company seeding + dev/qa/doc
segments); scenario 2 seeds a root->cell->dev hierarchy mid-flight, rides
the child through the scenario-1 arc into the cell branch (real squash
via the fake GitHub), then submit_up -> claim_gate_review/pr_pass ->
dispatcher re-claim (mirrored) -> PM complete merging cell->root. This is
the exact PM->reviewer->PM turn sequence the wave-1 turn cut shortens —
the BEFORE-net. Learned seams scripted: commit-subject validator (>=20
chars), reviewer learning-note gate, pr_pass clears ownership by design.
* feat(runtime): PR-gate turn cut — assembled parents auto-submit to the reviewer
When every child of an assembled parent is terminal, the closure
dispatcher now runs the real submit_up/submit_root through the internal
API as the owning PM (_try_auto_submit) instead of spawning the PM for
that turn — the submit's substance is deterministic gate code. Any gate
refusal falls back to the classic PM closure spawn; pr_fail routing and
the PM's final merge turn are unchanged; umbrellas never auto-submit.
ROBOCO_PR_GATE_AUTO_SUBMIT_ENABLED default-on; task.auto_submitted audit
row per cut. Proven by e2e scenario 2b (real API, real gates, real git)
against scenario 2 as the before-net.
* feat(notes): structured note sections carry a written_at trace stamp
Sections are overwrite-in-place, so without a stamp there was no way to
reconstruct WHEN a dev/qa/doc/reviewer note landed (CEO reMarkable item:
trace TIMESTAMPS). apply_structured_note stamps ISO written_at beside
the model fields; the panel notes tab renders it next to each card
title (pre-stamp rows render nothing). Progress updates, commits, and
journal entries already carried timestamps — this was the one gap.
* feat(tasks): server-side task search — title, details, and id prefix
The task list's search box only matched titles client-side, and the
trimmed summary payload deliberately carries no description — so
keyword/details/id search was impossible in the browser by design.
GET /tasks/summary gains q (ILIKE over title+description, id-prefix
match, composed with team/status and the view-permission scoping);
the panel debounces the box into the summary fetch and drops the
title-only client filter that would have hidden description matches.
* feat(wave-1): trace timestamps, real task search, Secretary task edits
- apply_structured_note stamps written_at per section; the panel notes
tab shows it (the one trace surface without a timestamp).
- GET /tasks/summary?q= searches title+description+id-prefix server-side
(summaries carry no description by design); panel debounces into the
fetch and drops the title-only client filter.
- Secretary control_task gains a CEO-gated edit action over the content
allowlist, and GET /secretary/tasks?q= resolves task names to ids for
the chat. PM-side expansion deferred per the CEO's 'not that much'.
* fix(workspace): dep-update probe scrubs the inherited venv pin
Under uv run the orchestrator's process tree carries VIRTUAL_ENV, and a
uv-based dep_update_command in the throwaway probe clone would target
that venv instead of the clone's — the same hazard _uv_subprocess_env
already guards on the install path.
* build: private per-repo uv cache — isolate from machine-wide uvx servers
Root cause of the recurring rich/pip/bandit rot, with evidence: uv cache
clean timed out on the ~/.cache/uv lock ('is another uv process
running?') — three uvx mcp-server-fetch processes (Claude Code fetch MCP,
one alive since Wednesday) share that cache and race repo syncs on it;
poisoned entries then survive venv rebuilds because rm -rf .venv never
touches the cache, and every re-link reproduces the breakage. UV_CACHE_DIR
now pins <repo>/.uv-cache (gitignored). The earlier UV_NO_SYNC
serialization stays as defense-in-depth but was not the whole story.
* feat(tests): e2e scenario 3 — pr_fail revision loop + root→CEO chain
3a: reviewer pr_fail with a concrete issue -> needs_revision ->
i_will_plan re-entry (full plan gates) -> real fix lands on the cell
branch (the unchanged-PR hard gate refuses resubmit until it does) ->
clean second pass -> merge. 3b: submit_root -> gate -> Main PM complete
escalates the root to the CEO -> the REAL approve-and-merge endpoint
squash-merges to the origin's master. Harness gains the tasks router, a
seeded CEO identity, origin_commit, and a fake GitHub whose head.sha is
recomputed live (real-GitHub semantics the unchanged gate reads). Seeds
now encode the real shape: delivery roots are team=main_pm and
planning-typed.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* test: align phase1 smoke mock with the armed team-match gate
The 8e5f84c4 sweep fixed 13 test files' inconsistent-team mocks but ran
only the gateway/foundation/runtime subsets; the full gate caught this
integration mock whose parent task carried an auto-generated MagicMock
team and died on not_authorized before the incomplete_input assertion.
* fix(runtime): attribute every agent.spawned audit to its dispatcher
A rogue spawner could not be identified live (2026-07-02): agent.spawned
rows carry container/model but not which dispatch loop launched them.
spawn_agent now takes spawned_by, stamps it into the spawned/spawn_failed
audit details, every call site passes its loop name, and an AST sweep
test holds future callers to it.
* fix(api): admin-complete refuses when the task's PR is still open
PATCH status=completed on a task with an OPEN PR stranded its commits
unmerged (bit the CEO twice live 2026-07-02). The override now refuses
with the PR number/URL and the consequence before the generic hatch
text; force:true stays the deliberate, audited escape.
* fix(panel): awaiting_ceo_approval offers the working ceo-approve path
The header's only approve action was Approve & Merge (POST
/approve-and-merge, no notes) which 400s NO_PR on a branchless MegaTask
umbrella — the CEO's approve button just failed. Primary action is now
Approve & Complete via the CeoApproveDialog (POST /ceo-approve, notes
>=20 chars, proven live); Approve & Merge stays for PR-bearing tasks.
* test: stop leaking self-heal + rate-limit state into live Redis
Two test files wrote real keys into a developer's localhost Redis:
self-heal originate tests left self_heal:notified:* (2h TTL) and the
i_am_blocked rate-limited tests left a NO-TTL 'anthropic rate-limited'
tracker blob — order/state-dependent poison for anything reading the
real tracker, and the prime suspect class for the one-off
test_self_heal_engine full-run failure (not reproduced in 5x dir runs,
adversarial orders, and a green full gate). Both files now point the
computed redis_url at an unreachable port; the engines' fail-open paths
keep every assertion intact. Leaked keys scrubbed live.
* docs: changelog + map delta for the leak-fix batch; mypy-clean attribution test
The attribution test's direct method assignments tripped the full gate's
mypy (method-assign) — switched to the house monkeypatch idiom, no
suppressions.
* fix(gate): clear the ten xenon C-ranks; isolate all tests from live Redis
Master CI has been red at the phase1 smoke test, so neither CI nor a
local full gate had reached the xenon step since the team-match sweep —
whose inline 'agent_team=str(agent.team) if ...' kwarg pushed nine verb
bodies from B(10) to C(11-12) unseen. A shared actor_context_fields()
(_protocol.py) computes (actor_slug, agent_team) once per verb, restoring
all nine to B with zero behavior change; the new admin-complete override
helper extraction does the same for routes/tasks.py.
tests/conftest.py gains an autouse fixture pointing the computed
redis_url at an unreachable port for every test — the root fix for the
three families caught writing live-Redis keys (self-heal dedupe,
rate-limit tracker, notification purpose-dedupe); no test uses a real
Redis, and every production path is fail-open by design.
* refactor(runtime): delete the never-wired dispatch-time spawn cooldown
_safe_spawn / gateway_pre_spawn_check / trigger_filter had no caller in
the repo's entire history (87ef42bf only flipped the flag). Its five
rules are superseded: provider parking runs inside spawn_agent, claim
freshness is the guards+reaper, runaway respawns are the progress-aware
breaker + notification cooldown; the per-task cooldown rule would
queue-stall every normal stage handoff if wired today. gateway_triggers
table kept inert. Ratified by the CEO over wiring it.
* build: serialize uv — gate recipes never implicitly sync the venv
Every uv run re-syncs implicitly, so a background make quality plus any
foreground uv run raced two writers on one .venv and tore site-packages
apart (the recurring rich/pip/bandit ImportError corruption; bit twice
today, four times on 2026-07-02's first session). UV_NO_SYNC=1 is now
exported Makefile-wide and quality/quality-fast/gate depend on one
explicit up-front sync step.
* fix(git): PR/merge/branch REST calls honor github_api_base_url
Fifteen sites hardcoded https://api.github.com while the CI-run and
open-PR-list calls already read settings.github_api_base_url — a GHE or
test override silently applied to half the surface. One _api_base()
helper keeps them uniform; default behavior unchanged.
* ci: split the monolith — backend CI, Panel CI, E2E Smoke
ci.yml keeps its file name and the backend quality job only (self-heal /
ci-watch / release-readiness default to the ci.yml workflow); the panel
job moves to panel-ci.yml scoped to panel/**, and the new scripted-agent
lifecycle smoke gets e2e-smoke.yml + a make e2e-smoke target (env-gated
out of the default pytest run). Trade: a panel-only red now lands on
Panel CI, which the ci.yml-pinned watch engines don't see.
* feat(tests): e2e lifecycle smoke harness — scripted agents, real gates
tests/e2e_smoke stands up the real API (flow/do routers + middleware on
uvicorn) over the ephemeral test Postgres, a local bare origin standing
in for GitHub, and a fake GitHub REST layer whose merges are real git
merges. A deterministic driver reloads the real MCP flow/do modules per
agent and walks claim (real clone + worktree) -> tracing-gap -> note ->
plan gate -> commit -> PR -> the full i_am_done ladder -> QA verdicts ->
documenter -> awaiting_pm_review in ~5s. Runs via make e2e-smoke + its
own CI workflow; skipped (env-gated) in the default suite. The
freeze-lift condition's first half: scenario 1 green.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* feat(gateway): claim-scoped context briefing — heavy sections only on context-acquisition verbs
* feat(gateway): cap unbounded LLM-facing payloads — embedded diffs, notification bodies, handoff journal content, north star
* feat(mcp): role-scope the optimal server's tool groups; index management becomes dev/test-only
* feat(mcp): cap per-result content on kb/error/learning search, mentor sources, rag citations
* refactor(gateway): extract heavy-briefing sections + clip helper to keep xenon ranks
* feat(orchestrator): cross-tick cooldown for notification-triggered spawns
* feat(usage,orchestrator): scope spawn-waste to anthropic sessions; cap agent Bash output via settings env
* docs: claim-scoped briefing, payload caps, optimal role-scoping, notification-spawn cooldown
* test(mcp): type the mixed-item cap fixture explicitly
* fix(orchestrator): lazy-init the notification-spawn cooldown store
* fix(lifecycle): admin-override claim reconciliation + PM request_changes verb (S6 postmortem B3+B4)
B3 — admin_set_status now reconciles claim ownership when leaving BLOCKED:
review/queue targets clear claimed_by/claimed_at/active_claimant_id and
consume the pre-block snapshot (a stale escalation claim was stranding the
next claimant: give_me_work handed the task out while note() bounced
not_authorized — the live b8fe0494 wedge). The pending/in_progress restore
path also syncs active_claimant_id, and a REST PATCH unassign releases the
claim with it.
B4 — new PM verb request_changes: awaiting_pm_review -> needs_revision with
concrete issues. The PM previously had no reject at merge review (only
complete/escalate), so an AC/scope violation looped i_am_blocked->escalate
4x live. Full vertical: lifecycle transition + ActionSpec + IntentSpec,
TaskService.request_changes (routes like a QA fail — original dev for a
leaf, revision PM for assembled; issues appended to dev_notes), verb-runner
compose, choreographer verb (spec gate + non-empty issues + soup check +
a2a delivery of the reject reason), HTTP routes on both PM flows, MCP tool,
journal:decision tracing, PM prompts, regenerated lifecycle artifacts.
* fix(panel): stop scorecard fetches for fallback-roster placeholder ids
useAgents() serves the static AGENT_ROSTER (ids "1".."22") while agent
definitions load; the Scorecards tab fetched a member scorecard per row
immediately, firing 22 guaranteed-422 requests per refetch cycle. Through
the browser's per-origin connection limit those queued every metrics-page
query behind them (~10s of skeletons on every tab). Gate the fetch on a
real member id (agent UUID or the "ceo" alias).
* Upgraded uv.lock
* fix(sequencing): declared deps become real edges + full loop-breaker coverage + assembled-branch freshness (S6 postmortem B1/B2/B6 + breaker)
B1a — code delegations REQUIRE a collision surface: new TASK_AT_DELEGATE
completeness spec (conditional FieldRequirement, when=('task_type','code'))
enforced at the gateway delegate gate. A no-surface code sibling is
'parallel to everything' by analyzer design, which is how two devs ran the
CEO's explicitly-ordered work out of order (f3e1afc5: seq#1 started before
seq#0, zero dependency edges). PM prompts updated; REST/manual creation
(TASK_AT_CREATE) unchanged.
B1b — the CEO's declared 'Depends on' lists become real edges: DraftSurface
gains declared_depends_on; SequencingService.analyze unions declared edges
(validated: self/out-of-range rejected) with the derived collision rules,
cycle-checked by the existing toposort. confirm_live_batch/preview_batch
map each draft's depends_on through (string indices coerced); intake tool
doc + prompter role prompt instruct verbatim copying. The live S6 root got
1 of its 3 declared in-batch edges and started alongside still-running R3.
Breaker coverage — the progress-aware respawn circuit breaker
(_pm_respawn_should_gate: strike counting, status-advance reset,
tracing-gap budget, DB durability, one-shot CEO notification) was consulted
by only 3 spawn paths; the doc/QA/dev/PR-review/PR-gate/revision/board
paths spawned unguarded at fixed cadence (the 26-respawn fe-doc loop,
~$7.20). Now consulted at every task-keyed spawn site (14 total).
B2 — assembled-branch freshness: submit_up/submit_root auto-sync the
assembled branch when it has fallen behind its base (children are terminal
at submit time, so the rebase is safe; master is never written). A rebase
conflict is a hard reject naming the files instead of a blind re-review —
kills the needs_revision↔awaiting_pr_review ping-pong of re-submitting a
stale head. Leaf i_am_done already had the behind-base gate; claim-time
fetch-fresh cut already existed.
B6 — documenter revision-pass loop: the awaiting_documentation bail
rejections (i_am_blocked/unclaim) now name the actual exit (i_documented
re-affirm) and the documenter prompt gets an explicit revision-pass rule.
* fix(orchestration): assembly-integrity gate + dispatcher heartbeat (incidents #11, #1)
Assembly integrity — submit_up/submit_root refuse when a completed child's
commits are not patch-present in the assembled branch (git cherry —
rebase-safe; branch pruned after merge or any git error fails open). Live
incident #11: a completed revert subtask's merge was lost from the cell
branch and the review gate re-flagged the exact violation the revert fixed,
spawning another revision cycle.
Dispatcher heartbeat — a dispatcher.alive audit row every 5 minutes from
the dispatch loop. The 2026-07-01 outage was 4h25m of fleet-wide silence
with no way to distinguish 'loop dead' from 'no work'; the loop's stdout
died with the container while audit_log survives. CHANGELOG for tonight's
full sweep included.
* style: ruff format for the orchestration sweep
* refactor(gateway): fold the assembled-submit guards + trim complexity under the xenon gate
_assembled_submit_guards combines the #11 integrity check and B2 freshen for
submit_up/submit_root; lifecycle's invalid-source remediate and git's
per-child cherry probe extracted into helpers. Test harnesses built via
__new__ stub the respawn tracker (the breaker now runs on their paths).
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* [fastapi-guard] Phase 1a: gated config flags for the HTTP security layer
Adds the ROBOCO_GUARD_* settings (all default-off / secure-default) for the
upcoming fastapi-guard 7.2.0 hardening — guard_enabled (master switch),
guard_fail_secure (fail-closed default; NAS overrides to false),
guard_telemetry_enabled + guard_agent_api_key + guard_project_id (guard-agent
telemetry, opt-in), guard_emergency + guard_emergency_whitelist (lockdown kill
switch). Inert until consumed: nothing reads them yet, so the request path is
unchanged. Foundation for v0.16.0.
* [fastapi-guard] Phase 1b: security foundation module + gated wiring
Add fastapi-guard 7.2.0 + guard-core 3.3.0 (bare, unpinned) and roboco/security.py:
- build_security_config() from settings — behind-nginx real-IP (trusted_proxies +
trust_x_forwarded_proto), HSTS/CSP headers, threat-ban + 404-sweep rules,
redis-backed state, exclude_paths (/ws + health + docs), env-driven
enforce_https, fail_secure (secure default), emergency lockdown, guard-agent
telemetry (opt-in), passive-mode calibration switch.
- guard_deco singleton (SecurityDecorator) for per-route decorators (Phase 2+).
- Three custom content validators guard's WAF can't cover: prompt-injection /
role-override, secret-exfil / credential-in-body, internal-SSRF.
- apply_guard(app) + guarded_lifespan() wired into create_app AFTER settings.
guard_passive_mode config flag added.
Entirely gated by ROBOCO_GUARD_ENABLED (default off): create_app mounts nothing
and returns the unchanged app when off (verified). make quality GREEN
(cov 95.32%, pip-audit clean, import-linter 2/0). 12 new unit tests.
* [fastapi-guard] Phase 2: critical-path decorators
Apply guard decorators to the highest-value endpoints (metadata-only; enforced
only when the middleware is mounted, so no-op while ROBOCO_GUARD_ENABLED is off):
- provider keys (ollama/grok/self-hosted writes): strict rate_limit +
max_request_size + block_clouds (no datacenter IP should touch secret writes).
- settings write + release approve/reject (CEO-gated): strict rate_limit.
- intake chat (prompter start/messages/events): rate_limit + max_request_size +
custom_validation(prompt_injection_validator) — the prompt-facing free-text
ingress gets the injection/role-override/secret-exfil content scan.
make quality GREEN (cov 95.32%, contracts 2/0). App builds with guard off,
decorators inert (verified).
* [fastapi-guard] Phase 3: wide decorator coverage across ingress + sensitive routes
Targeted-wide application (metadata-only; no-op until ROBOCO_GUARD_ENABLED). The
global SecurityMiddleware already rate-limits + WAF-scans every request, so this
adds the custom content validators on free-text ingress + tight limits on
sensitive ops (not blanket per-route rate_limit on reads):
- agent gateway do verbs (note/say/commit/dm/pitch/progress/draft_playbook/...):
rate_limit + max_request_size + custom_validation(secret_exfil or prompt_injection).
- a2a message/send + chat writes: rate_limit + size + prompt_injection.
- optimal/RAG (kb/search, rag/query, mentor/ask, errors/decisions/standards/
learnings): prompt_injection on searches, secret_exfil on record writes; docs
index → internal_ssrf.
- tasks: create/update → prompt_injection; QA/doc/PM transitions → secret_exfil;
CEO-gated verbs → tight rate_limit.
- secretary chat → prompt_injection; research → internal_ssrf; orchestrator
spawn/mutations → rate_limit; git ops + flow verbs → tight rate_limit.
Pure GET/reads left to the global middleware. Applied via a Sonnet workflow,
then verified: app builds with guard off (decorators inert), make quality GREEN
(cov 95.37%, contracts 2/0). Decoy/honeypot-path surface deferred (needs verified
guard ban-API integration — not rushed).
* [fastapi-guard] Phase 5: arm the NAS composes in passive/log-only mode
Arm ROBOCO_GUARD_ENABLED=true + ROBOCO_GUARD_PASSIVE_MODE=true +
ROBOCO_GUARD_FAIL_SECURE=false on the two NAS composes (docker-compose.yaml +
.yml). Passive = guard mounts and logs what it WOULD block but blocks nothing,
so the next NAS deploy calibrates against real traffic; flip PASSIVE_MODE off
after the false-positive review to enforce. fail_secure=false keeps a
guard-internal error from 500ing the personal deploy. The registry (user-facing)
compose is deliberately left unarmed so its published default stays conservative.
Phase 4 (passive calibration) is the operational step this enables.
* feat(security): Phase 3b — full-arsenal per-route guard enrichment
Stack the applicable guard decorators per surface instead of the minimal
rate_limit/max_request_size/custom_validation triad: content_type_filter on
every JSON-body write, honeypot_detection form-traps on human-facing POSTs,
block_clouds on key-writes + CEO release ops, behavior_analysis runaway-rate
rules on the agent flow/do verbs, suspicious_detection + usage_monitor on the
sensitive surfaces. Nine distinct decorators now applied thoughtfully per
endpoint. All metadata-only — no-op while ROBOCO_GUARD_ENABLED is off.
* fix(a2a): permit PR reviewer to deliver gate verdicts to the owning PM
can_a2a_direct had no pr_reviewer rule, so a reviewer (team=board, or a cell
team) fell through to the cell-member path and was cross-cell-denied when the
in-path gate delivered a pr_fail change-request to main-pm (or a cross-cell
cell-pm): "Cannot A2A main-pm ... Ask None to coordinate with None". The
delivery is best-effort, so pr_fail still transitioned but the verdict never
reached the owning PM — the blind-re-submit signal-gap the pr_fail fix closes.
Add an explicit pr_reviewer handler: it may A2A only cell_pm / main_pm (its
sole comms surface — everything else it posts on the PR itself), with a matching
route hint. The cell reviewers kept same-team access by coincidence; this scopes
every reviewer to PM-only, the correct model, with no other A2A caller affected.
Refresh uv.lock to the current resolution.
* feat(models): adopt Claude Sonnet 5 as the sonnet tier
Point the 'sonnet' alias at claude-sonnet-5 (MODEL_MAP) and give pr_reviewer
its own opus tier in ROLE_MODEL_MAP — it was falling through to the sonnet
default, and the role gates untrusted external/fork PRs plus root→master, which
warrants opus.
Price claude-sonnet-5 at the promotional 33% off Sonnet 4.6 ($2.01 / $10.05,
cache 0.201 / 0.5025) through 2026-08-31 via a dedicated pricing fragment that
beats the bare 'sonnet' alias; revert to full rate when the promo ends. Bare
'sonnet' stays full-rate as a conservative fallback (prod prices the resolved
claude-sonnet-5 id from the transcript).
Update the model docs and the billing / usage / manifest / spawn tests.
* feat(security): calibrate the guard WAF for RoboCo traffic + document the layer
The first end-to-end run of the fastapi-guard layer showed active enforcement
would block ~50% of legitimate agent traffic — RoboCo request bodies are code,
SQL, diffs, file paths, HTML, and URLs, which the stock signature WAF reads as
attacks. build_security_config now excludes RoboCo's free-text top-level body
fields (derived from the real request models, including the free-form container
fields whose nested prose is stringified and scanned) from WAF scanning,
dropping the active-mode false-positive rate to zero while keeping the WAF on
every non-excluded (id/enum/slug/branch) field and leaving the
prompt-injection / secret-exfil / internal-SSRF validators — which run
independently of the exclusion — fully in force. enable_penetration_detection
is made explicit.
Only excluded_detection_body_fields is reliable on guard 7.2.1: the per-route
categories knob is bypassed for JSON bodies, and the body scanner excludes
top-level keys only (scanning str(value) of every non-excluded field), so
free-form container fields must be excluded wholesale.
Adds tests/unit/test_security_middleware.py — the first end-to-end exercise of
the middleware (mounts it, drives guard's lifespan, fires real requests):
proves passive mode is log-only, active mode does not false-positive on
realistic agent payloads, threats are still blocked inside excluded fields, and
the WAF still fires on non-excluded fields.
Docs: CHANGELOG (Unreleased); a user-facing Optional-subsystems page + nav +
env reference for the HTTP security layer; the agent-facing RAG corpus
(what it is + why a request could be blocked); and the roboco mapping
(api-core-websocket / deployment-tooling / _complete_map).
* feat(security): Surface N — scanner honeytrap (guard /api auto-ban + nginx edge-drop)
Turns scanner probes against the scanner, in two layers matched to where
traffic lands. Behind nginx only /api, /ws, /health, /ready reach the
orchestrator, so guard can only see (and ban) scanner probes on those paths;
the classic root probes (/.env, /wp-login.php, /phpmyadmin, /.git/config) hit
the panel. So:
- build_security_config's threat_ban_config gains recon / sensitive_file /
cms_probing categories. A scanner probing those fingerprints on an /api path
is detected on the URL-path scan; repeated probes from one IP trip an adaptive
per-IP auto-ban (redis-backed, 24h). Only bans in active mode (passive logs
the recon hit) and needs redis (the 24h ban exceeds the in-memory cap). The
spec's decoy-route file is redundant — the WAF url-path scan bans regardless
of a registered route — so it is intentionally omitted.
- docker/nginx.conf drops the classic root scanner paths at the edge with 444
(connection closed, no response) before they reach the panel, anchored to
known scanner fingerprints so /.well-known and every real panel/API route are
untouched. Always on, independent of ROBOCO_GUARD_ENABLED.
Tests: 2 unit (the exclusion set + the scanner-ban categories are present) and
2 integration (a decoy path is blocked in active mode, passes in passive). The
nginx regex was validated against 15 scanner + 19 legit paths (0 false
positives). Docs: CHANGELOG, the HTTP-security page, the roboco mapping, and the
agent-facing RAG corpus.
* Token optimization — per-role observability, compute policy, spawn preflight (#291)
* test(models): lock the sonnet→claude-sonnet-5 MODEL_MAP invariant
* feat(usage): surface cache tokens + cache_hit_rate in usage breakdowns
* feat(usage): add per-role usage breakdown endpoint
* feat(usage): add spawn-waste signal (per-role unproductive rate + respawn strikes)
* feat(panel): surface per-role cost/cache + spawn-waste on the metrics page
* feat(routing): Phase 2 per-role compute policy — qa→haiku, main_pm→sonnet, per-role effort env mechanism (default-inert)
* feat(orchestrator): Phase 3 flag-gated spawn preflight — refuse non-gateway delivery roles (respawn-forever guard)
* chore(compose): arm ROBOCO_SPAWN_PREFLIGHT_ENABLED on the NAS composes
* docs: per-role usage observability, per-role compute policy, and spawn preflight
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* fix(panel): pin outputFileTracingRoot so the standalone build isn't broken by stray lockfiles
* feat(routing): populate ROLE_EFFORT_MAP + wire the verified --effort flag (cell_pm/board/auditor to medium)
* feat(gateway): omit empty context_briefing sections (Phase 4 payload compaction)
* refactor(orchestrator): extract spawn chokepoint guards to restore xenon rank B on spawn_agent
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* feat(metrics): capture per-session turns + tool_calls (phase 1)
Persist LLM iterations (turns) and tool invocations per agent spawn session,
the raw signal the granular per-member performance metrics build on (real
effort/iterations vs wall-clock).
- sum_transcript_usage returns a 5-tuple adding turns = unique assistant
message-id count; _usage_from_transcript + _resolve_active_tokens updated to
the 5-tuple (active-tokens keeps its 4-tuple contract by slicing).
- SDK: _SessionState.turns, set by /usage/sync; /usage/status (TokenUsageStatus)
now carries turns + tool_calls (= total_calls).
- orchestrator: new _resolve_final_turns_tools (SDK primary, transcript fallback
for turns only; Grok -> 0/0) wired into _finalize_spawn_session, which writes
turns + tool_calls to agent_spawn_sessions.
- migration 055 adds turns + tool_calls (BigInteger DEFAULT 0 -> historical/Grok
rows read 0, surfaced as n/a). Verified real alembic upgrade/downgrade.
Part of metrics-granularity (v0.15.0); recon-adjusted plan on disk.
* feat(metrics): pure compute_stage_effort helper (phase 2, part 1)
Foundation-layer overlap math (no DB): split each task status window into
active (merged wall-clock overlap of spawn stints — concurrent stints counted
once, so active <= window) vs wait (queue/review idle). Distinct from summed
effort. The per-task metrics service will feed it audit-log windows + spawn
stints. 9 unit tests (disjoint/nested/partial/merged/clamped/zero/multi-window).
* feat(metrics): per-task live metrics + GET /metrics/task/{id} (phase 2)
TaskMetrics dataclass + MetricsService.get_task_metrics: summed spawn effort
(vs wall-clock), turns/tool_calls/tokens/cost, per-stage active-vs-wait
(compute_stage_effort over audit windows x spawn stints), and who-caused-rework
(revision_count + named qa/pr fail events). Open stints and the open final
stage window close at completed_at for a terminal task (else now), so stages
don't grow past completion. Exposed at GET /dashboard/metrics/task/{task_id}
(404 if absent). Real-PG tests (compose/none/in-flight) + route tests (200/404).
* feat(metrics): CEO-as-member scorecard + ceo_reject audit regression (phase 3)
The human CEO is a measured member, read purely from audit_log (agent_role='ceo'
serializes from the CEO StrEnum): approval dwell (awaiting_ceo_approval -> a CEO
decision, incl. the coordination-root reject that lands in pending), unblock
dwell (blocked -> a CEO revive), and god-mode action count (every CEO-attributed
transition). CeoScorecard + MetricsService.get_ceo_scorecard (p50/p90 via
PERCENTILE_CONT, expanding IN for the decision sets) + GET
/dashboard/metrics/member/ceo (declared before any future member/{id} route).
The ceo_reject coordination-root audit gap the plan meant to close was already
closed by the gap-sweep (routes through admin_set_status -> agent_role='ceo'
audit); locked with a regression assertion in the existing coordination-reject
test. Real-PG tests: approval/unblock/godmode, non-ceo exclusion, empty->zeros.
* feat(metrics): audit instrumentation for escalations/blocked-others/idle (phase 4a)
The three extra per-member metrics that had no data source get durable,
in-session audit events (additive; never gate the underlying action):
- apply_escalation -> task.escalated (details.escalator_slug) on both the
normal block path and the pool-divert path -> escalations count.
- _unblock_dependents -> task.unblocked_dependents (details.count) on the
completed BLOCKER task, captured before the dependency edges are pruned ->
blocked-others count (sweeper attributes to the blocker's owner).
- mark_agent_idle -> agent.idle (details.agent_slug) -> idle/utilization (the
sweeper pairs an idle mark to the member's next spawn for idle duration).
(QA pass-rate needs no new event — reuses task.awaiting_documentation[qa] +
task.qa_fail.) Real-PG tests for each; 111 transition tests still green.
* feat(metrics): member_performance_daily rollup table + migration 056 (phase 4b)
The per-member scorecard rollup: one row per (date, member_kind, agent_slug),
CEO as a first-class member_kind='ceo' row (agent_slug='' NOT NULL so the
NULL-distinct UNIQUE keeps it unique). Full column set + the four CEO-approved
extras (qa_reviews_total/passed, escalations, blocked_others, idle_seconds) plus
blocked_seconds. Overwrite-upsert on (date, member_kind, agent_slug) for an
idempotent sweep. Migration 056 verified real up/down (24 cols, 4 indexes).
* feat(metrics): _sweep_member_performance rollup sweeper (phase 4c)
The daily per-member rollup sweep (mirrors _sweep_daily_rollup): a trailing
7-day, idempotent overwrite-upsert wired into _run_sweep. One focused query per
metric merges into a (date, agent_slug) accumulator — spawn effort/turns/tokens/
cost, completed/first-pass/revisions-received, revisions-caused (qa/pr fails),
QA pass-rate (passed + total), escalations (by escalator_slug), blocked-others
(unblocked_dependents by blocker owner), idle_seconds (idle mark -> next spawn),
blocked_seconds (blocked dwell) — plus one CEO row/day (approval/unblock dwell +
god-mode). Real-PG test asserts every facet + idempotency (a 2nd sweep
overwrites, never doubles); spawn-day != completion-day split is by-design.
* feat(metrics): member/org rollup scorecards + endpoints + live overlay (phase 5)
MemberScorecard + OrgScorecard with derived rates (FPY, effort-throughput,
turns/tool-calls per task, QA pass-rate, utilization) — all division-guarded to
None. get_member_scorecard reads member_performance_daily by slug and overlays
the member's live in-flight (non-terminal) tasks' effort via get_task_metrics
(disjoint by status: completion counts stay rollup-only, overlay only enriches
effort/turns/cost; includes_live_inflight flags it). get_org_scorecard
aggregates the cell (?team=) or whole org. Routes: GET /metrics/member/{agent_id}
(404 if absent, after the ceo literal route) + GET /metrics/org?team=. Real-PG
tests (derived rates, overlay no double-count, guards, org) + route tests.
* feat(metrics): granular CEO completion notification (phase 6)
There was no CEO completion notification at all (EventType.TASK_COMPLETED was
defined but never emitted). Add notify_ceo_of_completion in
NotificationDeliveryService — a granular body (real effort vs wall-clock +
stints/turns/tool-calls/revisions[QA/PR]/cost from get_task_metrics; degrades to
wall-clock-only, turns 'n/a', when there are no spawn sessions). Reuses the
existing ALERT type (no enum migration; the notificationtype PG enum is fixed at
001). ceo_approve now emits TASK_COMPLETED + fires the notification (best-effort
via _notify_completion — never blocks completion); complete() emits
TASK_COMPLETED too (closes the dead-code gap; the WS bridge can forward it).
Pure formatter tests + real-PG notification test.
* [metrics-granularity] Phase 7: panel Scorecards tab + dashboard overview
Add the CEO-facing metrics surfaces for the granularity feature:
- New "Scorecards" tab on the Metrics page: org rollup headline, the
CEO-as-member card (approval/unblock dwell + god-mode count), and a
per-member table (completed, first-pass yield, active effort, turns/task,
QA pass-rate, escalations, blocked-others, utilization). Each member row
self-fetches its rollup scorecard; live in-flight rows carry a "live" badge.
- New dashboard overview card (ScorecardOverviewPanel): org-wide 30-day
headline (completed, FPY, throughput/hr, active effort, cost) deep-linking
into the Scorecards tab.
- Plumbing: TaskMetrics/MemberScorecard/OrgScorecard/CeoScorecard types,
observability API client methods + empty fallbacks, and the four
useCeoScorecard/useMemberScorecard/useOrgScorecard/useTaskMetrics hooks.
Panel gate green: tsc, eslint, prettier, vitest (175 tests, +6 new).
* [metrics-granularity] test: make completion-notification robust to shared-DB CEO
test_notify_ceo_of_completion_creates_alert errored in the full suite (passed
in isolation): the session-scoped test DB is shared across the run, and the
sibling real-DB board-gate test commits a role=CEO agent (slug="ceo") without
cleanup — so my env fixture's hardcoded slug="ceo" insert hit a unique-constraint
violation, and a second role=CEO row would also make _get_ceo_agent()'s
scalar_one_or_none() raise. Reuse an existing CEO when present (the singleton the
production system actually has), else create one with a unique slug. Order-
independent. Also reflow test_metrics_instrumentation.py to ruff format.
* chore(release): 0.15.0
Metrics granularity: per-member/per-task/org + CEO-as-member scorecards,
turn/tool-call capture (migration 055), member_performance_daily rollup
(migration 056) with QA pass-rate / escalations / blocked-others / utilization,
per-task active-vs-wait metrics, granular completion notification, panel
Scorecards tab + dashboard Performance card, and the ceo_reject audit fix.
Version bump across the canonical set + CHANGELOG.
* [metrics-granularity] fix pre-tag audit findings (overlay double-count + panel error states)
Adversarial review before the v0.15.0 tag surfaced two real logical gaps:
- MAJOR (backend): the live in-flight overlay re-summed ALL sessions of every
non-terminal task via get_task_metrics, but _msweep_spawn already rolls up
every CLOSED session regardless of task status — so a closed session on a
still-open task was counted twice (rollup + overlay), permanently inflating a
member's effort/turns/tokens/cost on the common reap/respawn path. The overlay
now sums only OPEN sessions (ended_at IS NULL), which the closed-only rollup
can never contain — disjoint by construction. A just-closed session lands in
the rollup on the next ~60s sweep (no gap of note). Aggregated in SQL to mirror
_msweep_spawn. Regression test reproduces the double-count (turns 10→5).
- MAJOR (panel): the four new scorecard surfaces used `isLoading || !data` with
no isError branch, so a failed query span forever on a skeleton. They now
surface a load error. Tests added.
Also: OrgSummary active-effort formatting no longer round-trips hours→seconds→
hours; dashboard grid uses xl:grid-cols-4 (was 2xl) so 4 panels show at 1280px;
corrected the inaccurate "NULL distinct" CEO-row uniqueness comment (agent_slug
is NOT NULL; the '' tuple is simply distinct from agent rows).
make quality GREEN (cov 95.31%); panel GREEN (vitest 178).
* [metrics-granularity] fix: decode bytes stream message-id before XCLAIM
StreamEventBus._recover_stream passed the pending message id to XCLAIM via
str() on the raw bytes the client returns (redis client has no
decode_responses), producing "b'1782066556728-0'". Redis rejects that with
"Unrecognized XCLAIM option", so pending-message recovery threw on every
reclaim tick and unacked messages from crashed/slow consumers were never
reclaimed (leaking in the PEL on every stream, spamming the error log). Decode
via the existing _to_str helper — the fix the sibling claim path already uses.
Pre-existing in v0.14.0 (unrelated to metrics granularity); folded into this
release per CEO. TDD regression test + CHANGELOG entry. make quality GREEN.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* release-manager: fencing-token mutex + executor/readiness hardening
Closes the release-mutex TTL race (#17, HIGH) and the remaining
release-manager gaps (#88, #89, #201, #202):
- #17: the release mutex is now acquired with a uuid4 fencing token and
released via Lua compare-and-del; a background asyncio heartbeat
compare-and-expires the TTL ~every 60s while the execute owns the lock,
so a live execute no longer expires and a crashed one auto-releases
<=3000s. A second approve after TTL expiry cannot usurp and rm -rf the
in-flight clone — the fenced first-finally keeps its lock.
- #89: a Redis outage during acquire stays fail-closed (the execute never
runs) but now returns a distinct redis_unavailable result + log so the
CEO sees the cause instead of a false already_in_progress.
- #88: commit_and_push RuntimeError is wrapped into a structured
ReleaseResult(commit_failed) instead of a 500.
- #201: first-release fallback still emits untracked version-ref files as
gaps (no longer silenced by the first-release branch).
- #202: _await_proc awaits proc.wait() after kill() so a timeout cannot
leak a zombie.
TDD: tests/unit/services/test_release_proposal_concurrency.py extends
_FakeRedis with eval/get/expire and pins the fencing/heartbeat/usurper
invariants + the redis_unavailable result.
* PM/code-task creation guard + main_pm coverage + issue carve-out
Closes the creation-time role x task_type gap (the user's explicit example)
and the main_pm delegate hole:
- New pure helper `pm_cannot_own_code(role, task_type, is_issue_resolution)`
in foundation/policy/batch.py — single source of truth. Both PM roles
(cell_pm + main_pm) coordinate; a `code` task assigned/claimed by a PM is
a structural mismatch, EXCEPT a PM taking a code task in needs_revision to
resolve review/QA issues directly (the carve-out).
- Creation-time guard: TaskService.create calls the helper (closes the
create-with-cell-PM-assignee hole the team-based check misses).
- Delegate path + spec claim gate consult the same helper.
`_validate_assignee_task_type` / `_task_type_hint_for` now key on the
Role (CELL_PM OR MAIN_PM), not the cell-PM slug set — closes the
delegate-to-main-pm-as-code hole.
- identity.role_for_uuid_or_none is None-tolerant (treats None as "not a
PM" and proceeds) so a malformed/missing assignee cannot crash the guard.
- prompter.create_task_from_draft reuses the guard at draft-create.
TDD: test_batch.py (helper matrix + carve-out), test_main_pm_code_guard.py
(main_pm coverage), test_delegate_assignee_task_type.py (delegate parity),
test_lifecycle_spec.py (claim gate: rejects PM claiming code from pending,
allows from needs_revision + PM claiming planning + dev claiming code).
* task-service: completion hooks + escalation/cancel/audit hardening
Closes the task-service cluster (#21/#98, #99, #100, #101, #103, #216; #102
verified already-covered, #217 verified already-guarded):
- #21/#98: ceo_approve now closes the work session + triggers completion
hooks before worktree removal (no-op when work_session_id is None), so a
CEO-approved task lands the same close-path as PM-completed.
- #99: apply_escalation routes through the transition validator with an
enumerated escalation exemption (_ESCALATABLE_TO_BLOCKED) instead of an
arbitrary source->BLOCKED write; BACKLOG is refused.
- #100: branchless ceo_reject awaiting_ceo_approval->pending gets a real
spec edge (ceo_reject_to_pool ActionSpec + _STATUS_TRANSITIONS entry) so
future admin-override tightening can't wedge the path.
- #101: revision_count bump is documented as the single chokepoint, with
the pre-block RESTORE path undoing it when restoring a snapshotted
needs_revision (same cycle resuming, not a new rejection).
- #103: cancel cascade surfaces non-terminal orphans instead of swallowing
the role violation.
- #216: _remove_task_worktree_on_terminal escalates recurring FS/permission
failure (audit/notify after N) instead of silent-failing forever.
- #102: pinned in test_verb_runner_midverb_invalid_state.py (committed with
the choreographer cluster) — verb-runner savepoints already surface a
concurrent mid-verb state change as INVALID_STATE.
- #217: submit_for_qa claimed_by guard verified intact.
TDD: test_task.py, test_worktree_cleanup_on_complete.py,
test_escalation_board_guard.py (#99), test_task_service_* integration,
test_lifecycle_spec.py.
* choreographer: gate-claim guards + pr-gate hardening + fail-open logging
Closes the choreographer cluster (#5/#222, #29, #30, #82, #188, #189,
#192; #157/#187 verified already-fixed/pinned; #102 pin lives here):
- #5/#222: the unchanged-PR guard's fail-open head_sha lookup now logs
(warning) on a slug-resolver/git-helper error so a regression cannot
silently turn the pr_fail re-submit loop-stopper into a no-op. Stays
fail-open (never wedges the PM).
- #29: pinned (REFUTED-with-pin) — _lane_claim_guard already returns the
error envelope without releasing the claim on a transient lookup error.
- #30: pinned (REFUTED-with-negative-pin) — a non-batch branchless main_pm
root cannot bypass the complete spec gate (is_batch_umbrella requires
batch_id set).
- #82: _post_gate_review_to_pr wraps the slug-resolution call in try/except
(mirrors _capture_pr_head_sha) so a malformed cell_map AttributeError no
longer 500s the reviewer after a committed gate transition.
- #188: _is_hand_formatted_verdict anchors the header regex to line-start,
so a quoted (> ## Summary) or inline (mid-prose) header mention no longer
false-refuses a hand-formatted verdict.
- #189: pr_fail re-captures the PR head SHA after the transition commits and
re-stamps the verdict note only when it advanced (closes the stale-SHA
false-allow loop-hole); no-advance stays a single note write.
- #192: claim_gate_review skips the dev claim guards (already_active/paused/
lane) via a new skip_dev_guards param — a pr_reviewer inspecting an
assembled PR does not start work, so the single-active-task / code-lane
invariants do not apply; the dependency guard is kept, and QA's
claim_review parity is preserved.
- #157/#187: verified in tree — pr_review-only handoff is intentionally
prior-work-worth-resuming; self_review_block wiring (reviewer != dev)
holds on assembled tasks with 4 existing pin tests.
TDD: test_choreographer_*, test_pr_gate_posts_review (#82),
test_pr_review_hand_format_guard (#188), test_submit_root_unchanged_pr_guard
(#189), test_claim_gate_review_guards (#192), test_verb_runner_midverb
_invalid_state (#102 pin).
* playbook curate: guard the gating commit against a poisoned session (#55)
The explicit `session.commit()` that gates the RAG index (commit-before-index
so an uncommitted playbook cannot land in the corpus) raised PendingRollbackError
when a prior mid-verb failure had rolled the caller's session back — 500ing the
whole curation verb instead of returning a clean envelope, and (worse) risking a
fall-through to index an uncommitted playbook. Wrap the commit: on
PendingRollbackError, log + return invalid_state with a re-fetch/retry remediate
and skip the index. The happy path still commits exactly once then indexes.
TDD: test_playbook_verbs.py — poisoned-session returns a clean invalid_state and
does NOT index; clean-session still commits once + indexes (pins no fail-closed
inversion / no double-commit).
* [chore] gateway: atomic activate merge — preserve probe_failures across re-park (#156)
activate() was a blind SET that reset probe_failures to 0, so a probe-failure
increment that just landed (or was in flight) could be wiped by a concurrent
re-park — resetting the give-up / CEO-notify count mid-episode. Route activate
through a server-side Lua merge (roboco:activate_rate_limit) that refreshes the
episode metadata (kind / activated_at / retry_after / affected_agents) while
carrying over the previous probe_failures count. Indivisible w.r.t. the
increment/reset scripts (Redis single-threads an EVAL).
#56 (notify to prompter/secretary refused) verified SAFE — the pin tests
(test_notify_rejects_prompter_recipient / _secretary_recipient /
_allows_ceo_recipient) already cover the only human notify target invariant;
no legitimate send is dropped, no code change.
* [chore] foundation/policy: spec gates + QA retry-key pin (Cluster F)
#50 sync_branch composes=() so the spec gate accepted a terminal/paused/
blocked task and the handler rebased a dead/parked branch — add a
PRECONDITION_SYNC_BRANCH_STATE (claimed/in_progress/verifying/needs_revision
only), rejection_kind=invalid_state. TDD: 28 spec tests.
#148 submit_root's prose asserts 'a Main-PM root is planning-typed, never
code' but only the creation path (main_pm_cannot_own_code) backed it — add
PRECONDITION_ROOT_NOT_CODE on the submit_root IntentSpec (defense in depth),
scoped to submit_root only so the shared submit_for_review action keeps
cell_pm+code submit_up parity. Graceful on Mock/None task_type so the
choreographer Mock-task tests don't crash. TDD: 2 spec tests.
#150 VERB_RETRY_LIMITS is keyed by the MCP-exposed names (pass/fail), not
the IntentSpec-internal pass_review/fail_review — already correct; add a
pin test so a one-sided rename can't silently drop the QA-handoff cap.
#142 main_pm_cannot_own_code/pm_cannot_own_code already normalize casing
(.lower()) — no-op, pin test test_main_pm_cannot_own_code_is_case_insensitive
already in tree.
* [chore] worksession-git: 405 merge-method fallback + non-destructive close (Cluster W)
#108 _merge_with_retry hardcoded 'squash' and raised MergeConflictError on a
405 with no method fallback — wedging the PM on an open, mergeable PR whose
repo merely had the squash button off. Add a 405 fallback to a permitted
method (via _first_allowed_merge_method, exclude='squash'), mirroring the CEO
merge_pull_request path. A 405 with no permitted fallback (or a second 405)
still falls through to the already-merged disambiguation / MergeConflictError.
TDD: 2 new tests (fallback-success, no-permitted-method-raises).
#109 close_pull_request defaulted delete_branch=True, so the choreographer
supersede path deleted a superseded PR's branch while the orchestrator
supersede path explicitly preserved it — the two disagreed, and the
destructive default ran on the 'close the dead PR' path where the branch may
still be referenced / useful for audit. Flip the default to False (opt-in
deletion) and make the choreographer caller explicit (parity with the
orchestrator). TDD: 1 new test (default preserves branch); existing
deletion-when-requested test now passes delete_branch=True explicitly.
Dispositions verified against current code (no silent drops):
- #27 REFUTED/FIXED-UNDEPLOYED: work_session.merge_pr resolves by session_id
(no global pr_number lookup); the real cross-repo collision fix
(project_id scoping on pr_merge/close_pull_request/rebase_pr_for_task/
pr_target) is already in tree + tested (test_pr_merge_scopes_task_lookup_
by_project_id, test_close_pull_request_scopes_task_lookup_by_project_id,
test_git_pr_target_scoping). Verify-only.
- #106 REFUTED: a guard exists (rev-list --count {base_ref}..{branch} == 0)
before reset --hard + base_ref falls back to default_branch; tests lock
the safety (test_create_branch_never_repoints_branch_with_real_work,
test_create_branch_does_not_reset_or_checkout_shared_clone).
- #218 BY-DESIGN: the merge_pr idempotent guard intentionally preserves the
audit trail (docstring + test_merge_pr_idempotent_on_already_completed_
preserves_audit_trail); a COMPLETED session always carries attribution
(COMPLETED only via merge_pr), so the NULL-COMPLETED case is unreachable.
- #104 BY-DESIGN: agents never merge to the repo default branch in RoboCo's
model (root→master is CEO-only); the guard is a correct CEO-only rail,
locked by test_pr_merge_into_default_branch_is_ceo_only.
* [chore] llm: surface disabled-provider downgrade + scrub probe log (#20/#3/#211)
#20/#3 resolve_for_agent silently fell through to the legacy Anthropic path
when a configured provider was disabled — indistinguishable from 'no
assignment', so the operator got no signal that spawns bypassed the
provider. Surface the bypass with a warning (graceful degradation stays the
default — a stalled spawn is worse than a routing miss) and add an opt-in
ROBOCO_ROUTING_STRICT (default-off) that fail-closes instead. Wired into the
panel Feature Flags card. TDD: 3 unit tests (warn-on-disabled, strict-raises,
no-assignment-stays-silent).
#211 probe_ollama_tags logged str(exc) raw on the generic-exception branch —
structured log could carry connection internals / stack traces. Log the
exception class name only. TDD: existing generic-branch test strengthened to
assert the log kwargs don't leak the raw text.
* [chore] support/stream/optimal/playbook/comms hardening (Cluster S)
Logical-gaps sweep, Cluster S (TDD, red→green per item):
#64 notification_delivery.acknowledge published the NOTIFICATION_ACKED bus
event directly (bypassing the outbox) — a rollback left a phantom ACK. Route
it through defer_bus_publish (after_commit), mirroring deliver.
#76 playbook.archive()/reject() stamped the archiver into approved_by/
approved_at, overwriting approval provenance (and fabricating approval for a
rejected draft). Add archived_by/archived_at (migration 053 + table + model)
and write those on archive/reject, leaving approval attribution intact.
#181 vector_store.replace_chunks wiped existing index rows even when every
chunk lacked an embedding (embedder failure). Skip the wipe when chunks is
non-empty but records is empty — preserve good rows for nothing.
#182/#183 optimal.record_learning recomputed a learn-{md5(full_content)}
tracking source that never matched the URI the plugin embedded chunks under
(roboco://learnings/{doc_id}, doc_id=lrn-{hash100}). Use the plugin's
returned doc_id so de-index/lookup-by-source finds the chunk rows.
#96/#97 transcription periodic flush only peeked ready buffers (unbounded
map growth) and ran sync callbacks on the event loop (a slow callback
blocked the flush task). Flush (remove) each ready buffer after notifying,
and offload each callback to a thread.
#212 _TEAM_SCOPED_ROLES was duplicated across communications/agents_config/
seeds. Single-source it in foundation.policy.communications; consumers
reference that object (identity-tested).
#19 stream_bus._dispatch_event re-ran already-succeeded handlers on a
recover_pending replay (duplicate side effects). Add a per-(event.id,
handler) SET-NX idempotency guard: skip on a hit, clear the key on handler
failure so a replay re-runs it, fail-open when redis is unavailable.
Dispositions (no code change): #77 approve() index-write pair asserted
BY-DESIGN; #62/#63 notification DB-dedup verified pinned; #184/#185 REFUTED;
#214 REFUTED; #215 BY-DESIGN.
* [chore] db/migrations: graph-integrity guard + conftest unreachable-DB warning (Cluster D)
Logical-gaps sweep, Cluster D (TDD + real alembic upgrade head verification):
#16/#37 add tests/unit/test_migration_graph_integrity.py — a static guard that
the alembic migration graph has exactly one head, every down_revision resolves,
every revision is reachable from a root, and no revision id is duplicated. The
suite builds its DB via Base.metadata.create_all (not alembic upgrade head), so
a forked head / dangling down_revision / duplicate id would otherwise ship
silently and break a real deploy mid-stream.
Caught a real bug in the process: migration 053's revision id
"053_playbook_archived_attribution" (33 chars) exceeded alembic's
alembic_version.version_num VARCHAR(32) — a fresh `alembic upgrade head` raised
"value too long for type character varying(32)" at the 053 stamp. Renamed to
"053_playbook_archived_attr" (26 chars). Verified end-to-end on a scratch PG:
upgrade head stamps 053, downgrade -1 returns to 052. (The pre-existing
test_every_migration_revision_id_fits_the_alembic_version_column guard is now
green too; it had been red on the 33-char id.)
#90 conftest silently pytest.skip'd every DB test when Postgres was unreachable
— a non-Docker box reported a green run of all-skips. Extract the warning into
_warn_if_pg_unavailable and fire it at import so the operator sees the DB is
down (the per-test skip path is unchanged). Test: warns when unavailable, silent
when reachable (verified under -W error::UserWarning).
Dispositions (verified against real code + a fresh alembic upgrade head, no code
change): #6 REFUTED — sa.Enum(create_type=False) at 001:119/304 does NOT break a
fresh upgrade head (001→052 applied cleanly on a scratch DB); #8 REFUTED — the
upgrade passed 030/031 (RAG chunk tables) without pgvector installed; pgvector is
a runtime concern handled by roboco/db/base.py, not a migration prerequisite;
#40 REFUTED — the `|| echo` mask was already removed and partial-schema drift
reports exit 1 (only by-design unreachable/unmigrated skips remain); #204 REFUTED
— the property walk seed IS pinned (random.Random(20260504), line 97); #205/#206
REFUTED — the smoke-trace fixture IS wired via
test_lifecycle_smoke_replay.py (8 passed); no shell smoke scripts exist in the
tree to wire; #137 BY-DESIGN — pyproject version 0.14.0 is an operational note,
no code gate.
* [chore] panel: admin-override force flag + kanban subtask_count + ws cleanup + ui-store dedupe (Cluster P)
#13: kanban admin-override into a hatch state (completed / awaiting_qa /
awaiting_pm_review) now requires an explicit force=true from the panel and
emits a dedicated task.admin_override audit row server-side; non-hatch
overrides need no force. Backend gate in tasks route + admin_set_status;
panel kanban-board sends force for hatch targets; TaskUpdate carries force.
#198: kanban service threads the real subtask_count (one grouped query) into
dev + priority-swimlane + main-pm-flat boards instead of a hardcoded 0.
#79: useWebSocket cleanup clears messages/lastMessage/state on unmount or
endpoint change so a dep-change (navigating to another stream) can't leak the
prior subscription's stale snapshot as live.
#186: disambiguate the duplicate ui-store modules -- the session/scroll store
in lib/stores renamed to useScrollRestorationStore / scroll-restoration-store
(barrel + 2 consumers updated); the sidebar/theme useUIStore in @/store is now
the sole useUIStore.
#12: verified already in-tree (release-proposal-card surfaces non-404 errors
with retry; getProposal maps only 404->null). #80 by-design (handleTransportError
already resets isSending on a no-payload SSE drop). #81 docs (streamUrl docstring
records that live-intake SSE auth is session-id-based bearer-style).
Backend: ruff+mypy clean, 278 tests green. Panel: lint+typecheck clean, 159 tests.
* orchestrator: park/reaper/readopt/a2a hardening + self-heal/ci-watch dedupe (Cluster O)
Closes the orchestrator-side logical gaps from the sweep:
- #75 a2a human-only drop surfaced: _dispatch_a2a_work logs the skip
("a2a request targets a human-only role; left as a notification for the
human (not spawned)") instead of silently dropping the target — the
CEO/secretary/prompter still see the notification; only the spawn is
suppressed. (orchestrator.py)
- #72 readopt liveness: _readopt_running_agents requires a non-stale live
claim (via _agent_holds_live_claim) and skips a zombie container so a
reaped-but-restart-readopted agent isn't double-counted as active.
- #74 shutdown drain: stop() calls _flush_respawn_tracker so the durable
respawn counter write-throughs aren't lost on a clean stop.
- #71 resolve_wait active-guard + deferred liveness: a rate_limit_lifted
WaitingRecord is only confirmed-live after a _confirm_resume_liveness
probe (deferred deletion _resume_confirm_delay=30.0), and an
already-active agent short-circuits the repark. Scoped to
rate_limit_lifted records (the only ones at risk of a false lift).
- #73 stuck-Claude kill: _maybe_kill_stuck_claude + _claude_stuck_kill_ttl
(config.claude_stuck_kill_seconds) — a live container whose heartbeat is
stale past the grace AND whose gateway probe is broken is killed+evicted,
not protected forever by the reaper's live-skip.
- #230 verified FIXED-UNDEPLOYED: _gateway_broken_past_grace already
requires N consecutive false-broken probes (not one flaky streak); no
change, test added to pin the N-consecutive invariant.
- #43 self-heal per-observation dedupe: a fingerprint collapses repeat
CEO notifications for the same CI regression.
- #44 ci_watch dedupe by (git_url, workflow): a monorepo's multiple
workflows each get their own fix task (was collapsed by git_url alone).
- #49 identity.role_for_slug_or_none None-hardening: a stale/malformed
slug resolves to None and the human-only skip falls through to the safe
"not spawnable" path instead of crashing.
- #193 strategy engine: notify the CEO on a persistent assess failure
instead of failing silently in the background loop.
TDD: test_no_spawn_human_roles (a2a skip surfaced), test_orchestrator_
shutdown_drain (#74), test_provider_overload_break (#71), test_readopt_
running_agents (#72), test_resolve_wait_repark (#71), test_stale_claim_
reaper (#73/#230), test_strategy_engine_loop (#193, new),
test_self_heal_engine (#43), test_ci_watch_engine (#44), test_identity
(#49). All red->green.
* chore: make-quality green — xenon complexity refactors + mypy test fixes + lifecycle regen
No behavior changes. Brings the tree to a fully green `make quality` (the
base branch never passed the xenon B-rank gate on several blocks; the
lifecycle artifacts had drifted from the committed ceo_reject_to_pool edge).
Xenon B-rank refactors (extract a helper; preserve semantics exactly):
- api/routes/tasks.py: _apply_forced_status_override + _StatusOverride
dataclass bundle (update_task override block).
- services/task.py: _enforce_no_pm_code_on_create (create guards) +
_escalation_diverts_to_pool (collapses the two board/advisory +
main_pm+code divert branches into one predicate).
- services/prompter.py: _coerce_pm_code_to_planning (create_task_from_draft).
- services/notification.py: _duplicate_unacked_exists (_create_notification
purpose-based dedup query + ACK_REQUIRED_BY_TYPE gate).
- services/sequencing.py: _same_assignee_lane_edges (the undeclared-surface
same-assignee lane fallback at the tail of dev_task_collision_edges).
- gateway/choreographer/_impl.py: _pm_task_type_error static helper
(_validate_assignee_task_type compound PM guard).
- gateway/choreographer/pr_gate.py: _gate_review_event_verdict +
_gate_review_body static helpers (_post_gate_review_to_pr).
mypy test fixes (no type:ignore — banned; use typing.cast with quoted
strings per TC006):
- test_task_update_completeness: TaskUpdate(acceptance_criteria=None).
- test_bus: cast("Redis", _FakeRedis()); Redis import under TYPE_CHECKING.
- test_pr_merge_concurrency: capture AsyncMocks into locals before asserting.
- test_notification_delivery_phantom: cast("UUID", to_agents[0]).
Lifecycle artifact regen (owed from Cluster T #100 — the
awaiting_ceo_approval -> pending `ceo_reject_to_pool` edge was added to the
spec in 3d633084 without regenerating the derived artifacts the
foundation-check gate diffs against): docs/rag/lifecycle/intent-verbs.md,
docs/rag/lifecycle/status-transitions.md, panel/lib/lifecycle.json.
services/kanban.py: ruff format only (collapses the _load_subtask_counts
signature that drifted unformatted from Cluster P).
* [chore] logical-gaps sweep — Cluster I (intake/product/pitch)
#57/#58 prompter: preserve a top-level product_id with a 1-cell map
(prompter.py create_task_from_draft — top-level target wins over a
redundant 1-cell map instead of dropping product_id); reject — not
silently skip — a malformed project_id in the_work cell entries
(prompter.py _draft_cell_map raises ValidationError).
#59/#159 prompter: create_task_from_draft now operates on a copy
(_copy_draft) so _validate_and_coerce_draft / _clean_list never mutate
the caller's draft dict.
#160 prompter: _resolve_owning_team consults product/board routing
before forcing MAIN_PM on a multi-cell map (product root stays Board,
product+assignee-is-board stays Board).
#83/#84 github_provisioning: create_repo is idempotent by GitHub name
— a 422 "name already exists" (orphaned repo from a rolled-back prior
approval) is fetched and reused instead of erroring; pitch re-approval
now reuses the orphaned repo end-to-end.
#196 kanban: flat main-PM board has a "coordination" column for
non-cell teams (MAIN_PM/Board) instead of dropping their cards.
#197 project update: an explicit null in the PATCH body now clears the
stored field, distinct from an absent field (leave unchanged).
ProjectService.update drops exclude_none so explicit-None applies; the
PATCH route uses ProjectUpdate.model_validate(data.model_dump(
exclude_unset=True)) to preserve the request's unset-tracking (the old
field-by-field construction marked every field set and defeated the
distinction — nulling NOT-NULL git_url).
TDD: prompter 47, github_provisioning+pitch 16, kanban+project 67,
project routes 37 — all green; ruff + mypy clean.
* [chore] logical-gaps sweep — Cluster M (mcp-servers)
#60 flow_server/do_server: the circuit-breaker substitution no longer
erases the fixable rejection — the original envelope (kind/message/
remediate) is nested as inner on a copy of the SDK's circuit_open
envelope (the SDK dict is not mutated in place). The agent still sees
WHY the verb failed, not just that the breaker tripped.
#61 flow_server/do_server: a 404 carrying a *descriptive* detail
(not FastAPI's bare default {"detail":"Not Found"}) is now a
real resource not_found, surfaced as not_found so the agent
re-fetches state — instead of a misleading "server-side wiring gap"
invalid_state. The bare default and unparseable 404s still synthesize
the wiring-gap envelope; a 404 with a real Envelope (error field)
is still surfaced as-is.
#161 flow_server/do_server: dict error.code classification now uses
an exact-code map (authoritative for the codes the handlers emit) with
a substring fallback for unknown codes. Fixes the real regression:
AUTHENTICATION_REQUIRED carries no AUTHORIZED/DENIED/PERMISSION
substring, so the old substring-only rule dropped it to invalid_state
instead of not_authorized — an auth storm attributed as a state storm.
The fallback also adds AUTH so future AUTH-prefixed codes classify.
#162 flow_server/do_server: _register_tools gains a
ROBOCO_ALLOW_FULL_TOOLSET env override (default-off) so a missing
manifest falls back to the full tool set instead of raising — a
dev/test escape hatch. Production fail-loud behaviour is unchanged.
#163 intake_server: propose_batch accepts name as well as
title (intake drafts in the wild have used both), normalizing a
name-only draft onto a copy as title (caller's dict never mutated),
and reports the dropped count + reason in the return instead of
silently vanishing malformed drafts. The empty-batch hint now names
name as an alternative.
TDD: 123 mcp_servers tests green (14 new + 2 updated); ruff + mypy clean.
* [chore] Cluster N — conventions/docs logical-gaps sweep
#33: _create_new_doc/_update_existing_doc now resolve via
_resolve_contained_path (the RAG-returned update path was not containment-
checked — an escaping source could write/overwrite outside the docs dir).
#34: _commit_doc_to_repo returns committed/skipped/failed instead of
swallowing all exceptions; surfaced on DocRef.commit_status, the write
response, and the docs MCP guidance so a failed repo commit is fail-loud.
#35: write_doc only updates the similar doc when its filename matches — a
different filename creates a new file instead of collapsing onto the
similar doc's path (the dedup-overwrite defect codified by the old tests).
#129: a custom rule scoped to a language the validator never reports (a
typo) is surfaced as a warn finding on .roboco/conventions.yml via the
runner's once-per-run validation; #32 (tsx->typescript dialect) stays
BY-DESIGN.
#130: _cache_put only swallows a UNIQUE violation (23505) as a concurrent
duplicate; a non-unique IntegrityError (FK/NOT NULL/check) is log-errored
and re-raised instead of being silently misattributed.
#132: health re-reads the live file status (a cached degraded row hid an
in-place repair at a stale head key); get_map skips cached degraded rows
and stops caching degraded so a repaired file re-derives. #134 BY-DESIGN.
#133: _DB_METHODS gains stream/stream_scalars (SQLAlchemy 2.0 streaming
constructs are data access too — a route calling them is not thin).
#199: regenerate_verb_tables._annot_str strips Annotated[...] metadata
(BeforeValidator) before rendering; regenerated verbs.md + per-role
prompts so the BeforeValidator(func=...) repr (with a memory address) no
longer leaks into agent-facing prompt text.
TDD: 206 conventions/docs tests green (incl. 5 new files / appended
cases); ruff + mypy clean.
* [chore] Cluster 16 — cross-cutting hygiene logical-gaps sweep
Disposition + fix the 10 cross-cutting-hygiene gaps, TDD. make quality green
(ruff, mypy 944 files, pytest, xenon, vulture, foundation-check, enum-parity).
FIX:
- #24 /ws/system now gated by _require_panel_token (matches every sibling
/ws/* stream); rejects a missing token in strict mode and a forged token
even in dev. (roboco/api/websocket.py)
- #25 two drifted _require_ceo implementations (orchestrator router vs release
handler) unified on a single require_ceo_role helper in deps — same 403,
same role set, accepts Role/AgentRole/"ceo". (roboco/api/deps.py,
routes/orchestrator.py, routes/release.py)
- #11 a spawn session for a delivery role (developer/qa/documenter) with no
task_id now logs an unattributed-usage warning via is_unattributed_delivery_spawn.
(roboco/runtime/orchestrator.py)
- #65 pricing returns a structured CostResult(cost_usd, unpriced, is_anthropic)
so an unpriced Anthropic model (real spend we'd undercount) is flagged
instead of silently $0; calculate_cost stays a thin float wrapper.
(roboco/billing/pricing.py, billing/__init__.py)
- #67 blocker-metrics "blocked since" reads the task.blocked audit transition
(indexed on target_id/event_type/timestamp), not updated_at — which
over-counted when a blocked task was touched for a non-blocking reason.
Falls back to updated_at/created_at only with no audit row.
(roboco/services/metrics.py)
- #94 grok refresh_if_stale uses double-checked locking (_refresh_lock +
_recheck_or_refresh) so two concurrent callers don't both POST the
single-use refresh grant and burn the credential. (grok_auth.py)
DOCS (fix the doc, behavior already correct/pinned by tests):
- #66 get_summary docstring corrected — it sums raw agent_spawn_sessions rows
(sub-day precise); daily_usage_rollups/get_today_summary can diverge for
"today" until the sweeper catches up. (roboco/services/usage.py)
- #68 DashboardStorage is a documented in-memory stub; added a test pinning
that auditor flags are lost on storage reset (persisting = a migration +
service refactor, out of scope as a half-implementation).
(tests/integration/test_dashboard_service.py)
BY-DESIGN (no code change, with file:line evidence):
- #28 dashboard reads are open to the authenticated operator (dashboard.py:36
documents this); mutating auditor routes already gate via
_require_auditor_or_ceo. Role-gating reads would break the panel (no
X-Agent-ID on dashboard reads) and CEO-token-gating the router would block
the Auditor (auditor token != CEO token). nginx is the prod boundary.
REFUTED (narrowing would reintroduce a documented hang):
- #93 the ~/.grok directory mount (vs a single auth.json file) is load-bearing
— a single-file bind mount pins the inode so the atomic tmp.replace refresh
doesn't propagate to running containers (they hang at grok's login prompt).
Already documented in grok.py:161 and locked by
test_intake_grok_mounts_subscription_auth_when_present.
Incidental gate-greening (mypy errors a stale .mypy_cache had hidden in
earlier-cluster test files; xenon refactors for the new B-threshold):
- tests/unit/test_regenerate_verb_tables.py: type the dynamic-module loader.
- tests/unit/services/test_prompter.py: annotate the draft dict as dict[str,Any].
- tests/unit/services/test_conventions_cache_put.py: _FakeOrig is a real Exception
(IntegrityError's orig arg requires BaseException).
- metrics._blocked_since_map extracted from get_blocker_metrics (complexity).
- intake_server._normalize_batch_drafts extracted from propose_batch (complexity).
* Docs update
* [bug] spawn: self-heal vanished clone + branch ref before worktree ensure (be-dev-1 fatal loop)
A vanished clone_root (disk loss / /data/workspaces wipe / manual cleanup)
fatal-looped the resume path: _ensure_worktree_before_spawn ran
`git -C <missing>` and released the claim, but the reaper-style release
preserves assigned_to + branch_name so the next dispatch is a RESUME
(create_branch never re-runs to re-clone) and the same missing clone failed
every ~30s.
- workspace.py: ensure_worktree_self_heal re-attaches a present worktree +
symlinks the shared .venv; on a missing local branch ref it fetches from
origin (create_branch pushes at claim time, so pushed work survives) and
re-creates the ref, falling back to -b origin/HEAD only when the branch
was never pushed. _fetch_branch_ref is the token-aware fetch helper.
- orchestrator.py: _ensure_worktree_before_spawn health-checks the clone
and re-clones via ensure_workspace BEFORE the worktree self-heal. Fatal
git-state (WorkspaceError) still releases the claim + aborts; transient
failures abort without releasing (a fresh claim wouldn't help and
re-cloning is destructive).
TDD: 21 new + 61 related worktree/git/cancel/cleanup tests green; ruff +
mypy clean.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* Updated uv.lock
* Bunch of fixes we need to verify first..
* feat(megatask): per-cell project map root-subtasks (multi-project, multi-cell)
A MegaTask root-subtask can now target an ad-hoc per-cell project map — a
third targeting shape that mirrors the existing product fan-out root. In
RoboCo a project is per-cell (ProjectTable.assigned_cell); a monorepo is N
per-cell projects sharing one git_url. So 'multi-cell' IS 'multi-project',
and a task may mix per-cell projects across products or include OSS-library
projects not in any product.
Storage: migration 052 adds task_cell_projects (mirrors product_projects;
unique per (task, team)). TaskTable gains a cascade-delete cell_projects
relationship; TaskCreateRequest / TaskCreate / Task response carry the map.
Policy: batch.is_branchless_coordination + is_valid_batch_shape gain a
has_cell_projects param — a root-subtask targets exactly one of project /
product / cell-map; the umbrella still targets none. TaskService passes
has_cell_projects at every predicate call site and persists the rows in
create(). _ensure_branch_for_task cuts feature/main_pm/{root} per distinct
project in the map (via _distinct_projects_for_task); _require_target_or_umbrella
and _validate_batch_membership accept the map shape.
Fan-out: every distinct_project_ids site (task.py branch creation, routes
_project_for_complete + _resolve_project_for_merge, orchestrator
_ambient_projects_for_task, pr_review._project_slug_for, git._project_for_task)
generalizes to first-distinct-project-of-map-or-product. Choreographer
_resolve_subtask_project resolves a delegated subtask's cell from the parent's
cell map. The product-scoped _slugs_for_product intake helper is unchanged.
Intake: prompter._draft_cell_map extracts the per-cell map from the_work[].
_validate_batch_scope counts distinct projects across all drafts' cells
(>=2 min stays; one 2-cell draft satisfies it). create_task_from_draft
persists cell_projects for >=2-cell drafts (project_id/product_id None),
collapses a 1-cell map to the single-project shape, and leaves single-cell
top-level project_id drafts unchanged. _resolve_owning_team routes a
multi-cell map to Main PM (coordination root, like a product root — a cell
PM can't delegate cross-cell). propose_draft/propose_batch tool descriptions
declare the per-cell project_id (both Claude SDK + grok runtimes).
The umbrella stays branchless / pure-coordination / submit_root-rejected;
the CEO-escalation pr_number gate is not widened (the map root is
is_umbrella=False, mirroring a product root, so submit_root supplies it).
Single-cell root-subtasks and everything below them are byte-for-byte
unchanged. Un-run MegaTask waves (multi-cell drafts) become runnable.
* [feature] Panel per-cell project picker + pnpm format infra
MegaTask root-subtasks can fan out across cells (be+fe, fe+uxui). Since a
RoboCo project is per-cell (ProjectTable.assigned_cell), a monorepo is N
per-cell projects sharing one git_url — so multi-cell IS multi-project. The
batch-review card now shows one project Select per the_work entry, scoped to
that cell's repos, instead of one Select bound to a single top-level
project_id. confirmBatch validates each cell's project is in scope and the
batch still spans >=2 distinct projects.
- prompter.ts: CellWork gains optional project_id (the per-cell picker seam).
- batch-review-card.tsx: per-cell Selects (one per the_work entry), scoped to
the cell's projects; legacy single-cell drafts keep the one-Select path.
- use-prompter.ts: updateBatchDraftProject edits per-cell (entryIndex); confirmBatch validates every cell; batchFromEvent parses per-cell map.
Also adds the missing pnpm format infrastructure (the panel had no formatter
at all): prettier devDep + .prettierrc.json (default-style config: 80-col,
double-quote, semi, trailing-comma-all) + .prettierignore, plus format /
format:check scripts. Only the 3 changed files above were reformatted; the
~222 pre-existing non-compliant files are left untouched (a wholesale reformat
is a separate explicit decision, not bundled into this feature).
* [fix] MegaTask verification: migration 052 enum + async cell-map read
Two real bugs surfaced running the full gate against a containerized
Postgres (and the orchestrator boot log):
1. Migration 052 crashed a real orchestrator boot with
'type "team" already exists'. The generic sa.Enum(create_type=False)
does NOT set the postgres enum's create_type attribute, so op.create_table
(checkfirst=False) emitted a redundant CREATE TYPE against the pre-existing
team enum. Switched to postgresql.ENUM(create_type=False) — the postgres-
native enum whose create_type _check_for_name_in_memos actually reads, so
the CREATE TYPE is suppressed. Verified: 051->052 upgrade against a DB where
the team enum pre-existed (the exact path that crashed) now succeeds;
downgrade 052->051 drops the table and preserves the shared enum; fresh
upgrade head clean. (Migration 016 has the same latent sa.Enum pattern but
never re-runs in prod, so it's noted, not touched here.)
2. _ensure_branch_for_task read task.cell_projects (lazy=selectin to-many)
directly, tripping MissingGreenlet on a freshly-created/unqueried task —
which then poisoned the async session (PendingRollbackError). Replaced with
_task_has_cell_map: peeks InstanceState.unloaded (no IO) and reads the
already-loaded map, falling back to an awaited count query only when the
relationship is genuinely unloaded. Non-ORM stubs route to the plain
attribute. Fixes 2 integration tests; the 6 cell-map unit tests still pass.
Also: typed the self stub as Any in test_choreographer_subtask_project
(mypy tests/ wants Choreographer, not SimpleNamespace) — the codebase idiom.
Gate: ruff format/check clean; mypy roboco/ + tests/ clean; full pytest
10371 passed / 388 skipped against containerized pgvector:pg16; vulture clean.
Pre-existing xenon C-rank on reassign (from prior commit 19a474d3, not this
feature) still blocks make quality — surfaced separately.
* [refactor] Extract reassign board-advisory diversion helper (C→B complexity)
`reassign` in roboco/services/task.py hit xenon absolute complexity 11 (a
C-rank block), failing `make quality`'s --max-absolute B gate. The C-rank
originated in 19a474d3 (pre-existing, not this feature branch's work).
Extract the board/advisory → cell-task diversion into
`_maybe_divert_board_advisory_reassign` (complexity 4, A). reassign drops to
9 (B); behavior is byte-for-byte preserved — the helper runs the same
guard + pool diversion + log, returning the diverted task or None so the
caller falls through to the normal handoff. Whole-repo xenon exits 0; the 159
reassign / board-guard tests pass.
Unblocks `make quality` on feature/metrics-granularity.
* [fix] migration 016: postgresql.ENUM(create_type=False) for reused team enum
016_add_products_and_task_product_id used `sa.Enum(..., create_type=False)`
for the reused Postgres "team" enum — the same latent defect that crashed
052 on a real orchestrator boot. On the generic `sa.Enum` the
`create_type` kwarg is silently dropped, so `_check_for_name_in_memos`
never sees it and `op.create_table` (checkfirst=False) emits a redundant
`CREATE TYPE team` that fails with "type 'team' already exists" against a
DB where the enum pre-exists.
Switch to the postgres-native `postgresql.ENUM(..., create_type=False)` —
its `create_type` is a real attribute the guard reads, so the CREATE TYPE
is suppressed (and DROP TYPE on downgrade too). The member list is inert
under create_type=False (it never creates/alters the type), so it stays at
016's original six, reflecting the enum as it stood then, not the
later-widened set.
This never crashed in prod because 016 is never re-run (alembic_version is
past it), but it's the same defect class. Verified on the real boot path:
upgrade to 015 in process A (team enum created by 001), then `upgrade head`
in a fresh process B — 016 applied clean, no DuplicateObjectError; downgrade
016->015 clean, shared team enum preserved.
See project_migration_enum_create_type_gotcha.
* [chore] panel: prettier reformat across the codebase
Apply `pnpm format` (prettier 3.8.5, 80-col / double-quote / semi /
trailing-comma-all) to the 223 pre-existing panel files that predated the
prettier infra added in cb5365a4. Pure formatting — no semantic changes:
multi-line arrays/objects collapsed where they fit, trailing newlines added
(.prettierrc.json), import grouping unchanged.
Verified: `pnpm format:check` clean, `pnpm lint` clean, `pnpm typecheck`
clean, `pnpm test` 113/113 pass (7 files).
* Bunch of runtime fixes for MegaTask and other issues
* Fix different project same PR number collision problem
Fix (two layers):
1. Root cause — pr_merge and rebase_pr_for_task now take a required project_id and scope the lookup where(pr_number == X AND project_id == Y). Required so no caller can forget — the bug class can't recur. All 4 call sites updated (choreographer cell_pm_complete, the rebase-retry, the superseded close_pull_request now passes project_id, and _verb_runner._do_pr_merge).
2. Crash guard — _finalize_cell_complete None-checks the complete() return and returns a clean invalid_state envelope (with a remediate hint) instead of dereffing None → 500 → respawn loop.
* Fix: Make main_pm + task_type=code impossible
* Fix Main PM needs revision can't re delegate
* [chore] Bump local LLM glm-5→glm-5.2 + swap Ollama fleet defaults off minimax
- llm_catalog: OLLAMA_DEFAULT_MODEL minimax-m3:cloud → kimi-k2.7-code:cloud;
role defaults kimi-k2.6→kimi-k2.7-code, developer minimax→kimi, product_owner/
ceo kimi→glm-5.2, documenter glm→kimi; GLM 5.1→5.2 comment fix.
- config + .env.example + docker-compose{.yml,.yaml,.registry.yml} + docs +
memory_distiller + optimal_brain: glm-5:cloud → glm-5.2:cloud.
- panel ai-routing-card: typed SelfHostedModel/boolean annotations; drop the
stale "Minimax M3 default" string (default is now catalog-driven).
- tests: glm-5:cloud → glm-5.2:cloud in pricing + rate-limit-retry fixtures.
* [fix] submit_root: hard unchanged-PR gate stops the pr_fail re-submit loop
The 2026-06-27 infinite pr_fail loop: a Main-PM root (PR #139) was pr_fail'd,
routed to needs_revision, and re-submitted byte-identical → awaiting_pr_review
→ pr_fail again, forever. The prior hint/a2a steer was ignored by the weak
coordinator model — hints don't stop a model that won't read them. A HARD gate
refuses the re-submit when the assembled root PR's head SHA is unchanged since
the last pr_fail (no new cell work → identical diff); a different SHA ⇒ the
branch advanced ⇒ allow. Every ambiguous case fails open (no prior fail, no
recorded SHA, no pr_number, unresolvable slug, git error, closed PR) — only the
exact-unchanged case is hard-blocked.
- content/models: PrReviewContent.head_sha (optional; JSON col → no migration).
- git: get_pr_head_sha (GitHub pulls API; None on any failure → fail-open).
- pr_gate: pr_fail captures head_sha into the verdict record; pr_pass does not.
- _impl: submit_root runs _submit_root_unchanged_pr_guard after _submit_up_guard;
_current_root_pr_head_sha resolves slug + current SHA (fail-open).
- pr_review: extract module-level resolve_task_project_slug, shared by the mixin
and the gate helper (_LegacyChoreographer reaches it via cast to the
ChoreographerHelpers typed view — it doesn't inherit the helpers mixin).
- tests: test_submit_root_unchanged_pr_guard (11 — refuse/allow/6 fail-open/3
capture-side, mypy-clean via cc:Any spy idiom, zero type:ignore) +
test_pr_gate_notifies_pm capture-path stub.
* [chore] mypy tests/: clear all 15 pre-existing type errors so make quality can go green
The branch tip had 15 mypy tests/ errors in files this bundle did not author,
which blocked CI's make quality mypy step (mypy roboco/ tests/) regardless of
the bundle's own commits. Pre-existing is still existing — fix every one:
- test_schemas_v1_flow.py (8): the StrList coercion tests intentionally pass
SDK-nested list-of-strings input ([[['...']]], {'item':{'$text':'...'}}, int,
dict). Annotate those literals as list[Any] locals so mypy accepts the
coerce-able shape; the StrList BeforeValidator still flattens to list[str] at
runtime. No type:ignore.
- test_pr_gate_records_verdict.py (3): notes_structured is dict|None; narrow
with 'assert t.notes_structured is not None' before indexing (the existing
pattern at line 90).
- test_pr_review_hand_format_guard.py (1 site, 2 errors): the _verb_runner()
spy assertion — use the cc: Any = c alias idiom so assert_not_awaited
resolves; drops the now-unused type:ignore[union-attr].
- test_pr_gate_notifies_pm.py (1): drop the unused type:ignore[method-assign]
on the a2a.send reassignment.
- test_content_models.py (1): narrow coerced with isinstance(coerced,
PrReviewContent) before reading .issues (the base _Content lacks the field).
Gates: rm -rf .mypy_cache && mypy roboco/ tests/ = Success (855 files);
ruff check + format clean; 5 affected suites = 40 passed.
* [fix] fail_qa routes needs_revision back to the dev, never the pool
A dev task in needs_revision must go back to the developer, never the
pool. The pool path let a cell PM re-claim the revision (PMs can claim
needs_revision) — the live 2026-06-27 'needs revision on a dev task sent to
the cell PM' bug.
fail_qa's original_developer marker is the fast path, but it is
unreliable in practice (live observation: never persisted), so the
unassign else-branch was the load-bearing path and it dropped the task
into the pool. Add a work-session fallback (_resolve_revision_dev) that
resolves the developer who actually worked the task — the most recent
work session whose agent is a developer, the QA's own session excluded
— and reassigns to that dev instead of unassigning. Only unassign when
no developer ever touched the task. Self-heals the marker so a
subsequent re-fail takes the fast path and the QA-review index
attributes the work correctly.
* [feature] delegate carries dev-task collision surface (sequencing S1)
The cell/main PM's delegate verb now carries the dev-task collision
surface (intends_to_touch / adds_migration / touches_shared) and an
explicit depends_on override through DelegateRequest -> DelegateInputs
-> _create_subtask_from_inputs -> create_subtask, and create_subtask
forwards sequence / dependency_ids / batch_id / surfaces into the
prepared TaskCreateRequest instead of dropping them (the base create
already persists them at task.py:878-884).
This is the plumbing for the multi-level sequencing model edge kind 3
(dev-task collision DAG). Previously a dev task delegated with a
collision surface or an explicit dependency lost it before persistence
— dependency_ids was always [], so the only dev-task ordering was the
weak assignee-keyed spawn barrier (the live 2026-06-27 out-of-order
break: 40842957 started before 9b3682b8's PR merged). Phase S2 runs
SequencingService over the surfaced siblings and wires the DAG via
add_dependency.
* [feature] wire dev-task collision DAG at cell-PM delegation (sequencing S2)
Pure dev_task_collision_edges in sequencing.py turns a parent's surfaced
siblings into (depends_on_id, task_id) pairs via SequencingService. TaskService.
wire_sibling_collision_dag wires them through add_dependency (idempotent). The
choreographer calls it after each dev-task delegate so the sibling collision DAG
is built incrementally as the cell PM decomposes — file-overlap serializes,
migration chains, shared-last; stable (priority, sequence) ordering keeps edges
from flipping into reverse cycles on re-runs.
* [feature] wire cell-task wave chain + by-osmosis edge (sequencing S3)
Kind 2 (cell-task wave chain): a new cell-task under root-subtask UT_n
depends on every cell-task under every root-subtask in UT_n.dependency_ids
(the kind-1 wave-chain edges), so its branch carries the previous wave's
merged cell work. Re-derived from the root-subtask's deps, not the cell-task's
own dependency_ids (which also carry UX/product-fanout edges the by-osmosis
edge must not pick up). A root may fan to several cell-tasks (different cells),
so the previous wave's cell-task is a SET.
Kind 4 (by-osmosis): the first dev task (sequence 0) under a cell-task depends
on each predecessor cell-task's tail (max-sequence) dev task, so the new wave's
first branch carries the previous wave's fully-merged tail. Subsequent dev
tasks inherit the tail via kind 3 or the merged base.
Both wired from _create_subtask_from_inputs, dispatched on parent.team
(MAIN_PM -> kind 2; cell team -> kind 4). Pure helpers
(cell_task_wave_chain_depends_on, by_osmosis_tail_dev_tasks) unit-tested in
test_sequencing.py; TaskService methods integration-tested. Idempotent +
best-effort throughout (add_dependency dedupes; missing predecessors are
no-ops). Also fixes a latent mypy-tests gap (estimated_complexity required on
direct TaskCreateRequest calls in the S2 tests).
* [feature] sync_branch dev verb — gate-level branch rebase (Phase B1)
Raw shell git is denied to agents (Bash(git:*) base deny), so a developer
whose branch fell behind its base had no gate-level rebase — only the
CEO/PM-only /rebase HTTP route. sync_branch is the dev verb that wraps the
rebase through the gate (traced + evidenced), so the 'everything goes through
the gates' invariant holds.
- lifecycle: IntentSpec sync_branch (dev-only, ownership-gated, composes=(),
git-only — no DB transition); _next_hint_synced helper.
- GitService.sync_task_branch: rebase task.branch_name onto its resolved base
via rebase_onto_base (fetch + rebase + force-with-lease push).
- Choreographer.sync_branch + _sync_branch_preflight_rejection: not_found /
unknown-role / spec-gate / no-branch / protected-base guards, then the git
op; conflicts abort (no force-push) and steer to resolve-by-hand; git failure
steers to i_am_blocked.
- HTTP route /api/v1/flow/developer/sync_branch + SyncBranchRequest schema.
- MCP tool sync_branch(task_id) + _TOOLS registration (manifest auto-propagates
via intents_for_role(Role.DEVELOPER)).
Tests: intent spec (5), choreographer handler (8: happy/conflicts/not_found/
not_authorized/no-branch/protected-base/git-failure/audit), route (1), MCP (1).
ruff + mypy roboco/ tests/ clean; unit suite green (DB-fixture errors env-only).
* [feature] i_am_done behind-base submit gate (Phase B2)
A sibling's PR merging into the parent branch while a dev worked leaves the
dev's branch behind its base — the assembled PR then can't merge cleanly and
the sibling's changes go missing (the 2026-06-27 out-of-order dev-task break).
The behind-base gate refuses i_am_done in that state and steers the dev to
sync_branch (the Phase B1 gate-level rebase verb).
- GitService.is_behind_base: rev-list --left-right --count across
origin/{base}...origin/{head} → (behind, ahead); fetch-first so origin
reflects the pushed head. Raises on git failure (consistent with
rebase_onto_base); malformed stdout degrades to (0,0).
- Choreographer._behind_base_gate: wired into _i_am_done_gate after
_ensure_branch_pushed. behind>0 → invalid_state remediate→sync_branch.
Fail-open on git/base-resolution error (flaky fetch can't strand a task at
the submit gate — the merge layer has its own behind checks). Skipped for
branchless roots and protected bases (master/main/-prefixed).
Tests: gate (6: refuse+steer/up-to-date/branchless/protected/fail-open-base/
fail-open-git), is_behind_base (6: parse/up-to-date/malformed/argv-form/
requires-branch/missing-project). ruff + mypy roboco/ tests/ clean; unit green.
* [docs] sync_branch prompt + behind-base guidance (Phase B3)
Update every behind-base/rebase guidance surface to reflect the B1
sync_branch dev verb + B2 i_am_done behind-base gate: devs now self-rebase
through the gate instead of escalating a plain behind-base condition; PMs
still escalate cell/root integration branches (they have no rebase verb).
- developer.md: sync_branch in the verb table; 'When your branch is behind
its base' rewritten — call sync_branch, do NOT i_am_blocked a plain
behind-base; conflicts → resolve by hand, commit, sync_branch again.
- cell_pm.md: delegate signature gains intends_to_touch/adds_migration/
touches_shared/depends_on + a 'Collision surface' section (fill it on every
code subtask so sibling dev tasks that touch the same files sequence into a
conflict-free order — the 2026-06-27 out-of-order break fix); behind-base
section steers devs to sync_branch, PMs escalate only the integration branch.
- main_pm.md: behind-base section — dev leaf = dev's sync_branch; cell/root
integration branch = escalate_up.
- RAG git-errors.md / blocked-tools.md: devs sync_branch, PMs escalate.
- docs/troubleshooting/common-issues.md: leaf self-rebases; integration branch
still escalates to operator.
- CLAUDE.md verb surface: developer gains sync_branch.
- agents/prompts/_generated/*: regenerated via scripts/regenerate_verb_tables.py
— adds sync_branch to the dev table AND catches the generated tables up to
the S1/S2 delegate sequencing params + meltdown-fix note top-level params
(the derived files had drifted stale vs the already-committed schemas).
Docs/prompts only — no code. ruff + mypy roboco/ tests/ clean.
* [chore] orchestrator: refuse to spawn human-only roles (CEO/prompter/secretary)
A live 2026-06-27 incident saw a CEO agent container spawned. Root cause:
_dispatch_a2a_work iterates every A2A/notification target and spawns it
with no human-role filter, and _is_agent_active('ceo') is always false
(the CEO is never a container), so the 'skip if active' check could never
protect the CEO. Any CEO-addressed notification (board handoff, escalation)
launched a CEO container — the system acting as the human CEO: a trust
violation. The CEO is the human operator; intake (prompter) and secretary
are human-driven chats launched through their own dedicated guarded paths
(_spawn_intake_container / _spawn_secretary_container), never spawn_agent.
Fix: a single chokepoint guard at the top of spawn_agent refuses
Role.CEO / PROMPTER / SECRETARY (raises AgentReadinessError + logs). This
structurally covers every dispatcher present and future, since they all go
through spawn_agent. Plus a defense-in-depth skip in _dispatch_a2a_work so
a human-role target never even calls in (avoids error-log spam; the
notification stays for the human to read in the panel).
Safe: the dedicated human-spawn paths do not route through spawn_agent.
Regression tests: spawn_agent refuses ceo/intake-1/secretary-1, does NOT
refuse a real agent; _dispatch_a2a_work skips CEO/intake/secretary targets
and still spawns real-agent + mixed-target cases.
* [chore] orchestrator: skip human-only assignees in claimed/pm-review dispatchers
Defense-in-depth for the spawn_agent human-role chokepoint (d31d6719).
The chokepoint structurally guarantees no CEO/prompter/secretary container
can ever spawn — every dispatcher goes through spawn_agent. But two
dispatchers resolve an arbitrary assigned_to and spawn it with only a
None/unknown-role filter, so a human-assigned task would reach the
chokepoint and RAISE: caught by the per-dispatcher try/except, but it
aborts that dispatcher's whole tick (stalling other respawns behind the
mis-assigned task) and error-logs every cycle. The other dispatchers are
already safe by whitelist/hardcoded slug (blocker_resolver_slug returns
None for non-PM/non-BOARD; escalation/approval use whitelists; marketing
and audit hardcode their non-human slug).
- _claimed_task_needs_agent: return None for a CEO/prompter/secretary
assignee — no container to respawn, and do NOT release a human-owned
task to pending (that would re-route it to a PM). Leave it for the human.
- _dispatch_pm_review_work (assigned branch): skip a human-only assignee
so a CEO-assigned awaiting_pm_review task neither spawns nor aborts the
dispatcher's tick.
Audited all target-iterating dispatchers; only these two lacked a filter.
Regression tests cover both skips.
* [F002] retype board-routed MegaTask root-subtasks code->planning on activation
_activate_batch_root_subtasks flipped a held root-subtask to team=MAIN_PM
but left task_type=code (intake only coerces main_pm-team drafts, so a
board-routed code root reached activation still code-typed). The
main_pm+code combo re-introduces the 2026-06-27 meltdown. Mirror
approve_and_start's own retype via main_pm_cannot_own_code so the
activated child is a planning-typed coordination root.
TDD: RED test_activate_batch_root_subtasks_retypes_code_to_planning
watched fail (task_type stayed CODE), then GREEN after the retype.
ruff+mypy clean; 125 batch/umbrella/approve tests green, no regressions.
* [F003,F004,F014] enforce HMAC agent-token gate on do routes + WebSocket streams
F003/F014: /api/v1/do/* only required X-Agent-ID (UUID) — no token check,
unlike the flow routers' role guards. A forged X-Agent-ID passed. Added
require_any_authenticated_agent (token-only; do router serves all roles)
and applied it as a router-level dependency. Binds X-Agent-ID to a verified
HMAC token when ROBOCO_AGENT_AUTH_REQUIRED=true; rejects a forged token
even in dev mode.
F004: /ws/* per-agent streams (channels/agents/sessions/notifications)
never read the nginx-injected X-Agent-Token, so in strict mode an agent on
the Docker network could subscribe to another agent's notifications with
no auth. Added _require_panel_token verifying the CEO panel token against
the CEO identity; wired into all four per-agent streams (system stream
stays operator-only per its docstring). Same strict/dev contract.
TDD: RED tests watched fail (no gate -> 200/accept), then GREEN. ruff+mypy
clean; 399 api/mcp + 29 WS tests green, no regressions.
* [F005,F006] grok auth: directory mount + atomic-write fallback
F005: the single-file bind mount of auth.json pinned the inode, so the
orchestrator's atomic refresh (tmp+rename within ~/.grok) never reached a
running grok container — a long-lived container hung at the login prompt
when the original ~6h token expired. Mount the host ~/.grok DIRECTORY (ro)
at /home/agent/.grok-auth-ro; the entrypoint symlinks ~/.grok/auth.json at
that RO mount so grok + the --check backstop read the live credential (the
directory mount sees the host-side rename) while grok's writable state
(config.toml, sessions/) stays in the image's ~/.grok.
F006: a rotated refresh_token is single-use — xAI invalidates the old one
the instant it issues the new one. If the atomic write failed after the
rotation, the file kept the now-dead old refresh_token and the credential
was permanently lost on the next refresh. _atomic_write now falls back to a
direct write when tmp+replace fails, so the rotated token always lands on
disk (losing the write is catastrophic; losing atomicity is not).
TDD: RED tests watched fail, then GREEN. ruff+mypy clean; 32 grok tests
green, no regressions.
* [F016,F017] choreographer: surface invalid_state instead of None.status 500 on submit_root / i_am_blocked
Both verbs compose a single atomic action whose None return (the verb's
own result) flowed out of run_intent and was dereferenced as t.status,
HTTP 500-ing with no actionable rejection:
- F016 submit_root: submit_for_review returns None when the root->master
PR was already opened / the task raced out of in_progress. Post-runner
None-guard extracted into _submit_root_finalize -> invalid_state
(re-fetch; if awaiting_pr_review the PR is open, wait for reviewer;
else re-delegate fixes and retry) instead of None.status.
- F017 i_am_blocked: escalate returns None in four cases (no task, no
agent, no resolvable escalation-target slug, no target agent row) e.g.
a developer whose role has no PM above it. _run_i_am_blocked_intent
now guards updated is None -> (t, invalid_state rejection) with
remediation (re-fetch + escalate to CEO directly / retry) instead of
the caller deref'ing None.status -> 500 + respawn-loop.
TDD red->green; ruff + mypy clean; gateway suite green (58 passed).
* [F007] choreographer: cell-level unchanged-PR re-submit loop-stopper for submit_up
The root loop-stopper (F016) was root-only; a weak cell PM could re-submit
the unchanged cell->root PR after a pr_fail and loop awaiting_pr_review ->
pr_fail forever (the cell analogue of the 2026-06-27 root loop).
pr_fail stamps the assembled PR's head SHA into notes_structured.pr_review
.head_sha for cell AND root gate tasks alike (the capture is gate-verb-
level, not root-level), so the same structural refusal applies to submit_up:
if the cell PR's current head SHA equals the SHA the last pr_fail recorded,
no new dev work landed on the cell branch -> the diff is byte-identical ->
refuse, do not re-open the gate. Different SHA -> branch advanced -> allow.
- _submit_up_unchanged_pr_guard mirrors _submit_root_unchanged_pr_guard
(cell-PM remediation: re-delegate to the dev + wait for re-assembly),
wired into submit_up after _submit_up_guard passes.
- Renamed shared _current_root_pr_head_sha -> _current_pr_head_sha (both
guards use it; the lookup was never root-specific).
- Every ambiguous case FAILS OPEN (no prior fail, no recorded sha, no
pr_number, no resolvable project, git/closed-PR None) — only the exact-
unchanged case is hard-blocked.
TDD red->green; ruff + mypy clean; F007+F016 guard suites green (15 passed).
* [F008] evidence_builder: surface persisted pr_review verdict+issues in the PM task_handoff
The pr_fail a2a steer to the owning PM is fire-and-forget; a PM respawned
into needs_revision later read none of it (build_task_handoff never looked
at notes_structured), saw a generic 'needs revision' with zero concrete
change-requests, and re-submitted the same PR (the 2026-06-27 infinite
pr_fail loop on 9980d0a0 / PR #138). The signal-gap was only partially
closed by the a2a.
build_task_handoff now extracts notes_structured.pr_review
(verdict/summary/issues/head_sha — the slot pr_fail authors on every fail)
into a pr_review field on the handoff, so every PM briefing for the task
carries the concrete change-requests. A prior pr_fail alone now counts as
prior-work-worth-resuming. Type-guarded + capped; absent => no key (no
misleading empty slot).
TDD red->green; ruff + mypy clean; evidence_builder suite green (14 passed).
* [F009] notification: derive requires_ack from ACK_REQUIRED_BY_TYPE, not the True default
NotificationService._create_notification built NotificationTable without
requires_ack, so the column default (True) applied to EVERY notification -
including informational REVIEW_REQUEST / DOCUMENTATION_REQUEST /
A2A_REQUEST / KNOWLEDGE_SHARE (ACK_REQUIRED_BY_TYPE -> False) and every
@mention from MessagingService._notify_mentions. Each false ack-required
inflated the recipient's unacked set and soft-blocked i_am_idle into
respawn churn.
- _create_notification: requires_ack=ACK_REQUIRED_BY_TYPE.get(type, True)
(unmapped types default True - preserve the action-required bias).
- _notify_mentions: requires_ack=False explicit (MENTION is informational).
TDD red->green (identity is False/is True assertions - the mocked
flush doesn't apply SQLA's insert-time default, so pre-fix the attribute
was None); ruff + mypy clean; notification suite green (18 passed).
* [F010] notification: never dedup informational notifications (knowledge-share data loss)
The purpose-based dedup suppressed a same-purpose (same sender/type/task,
overlapping recipients) notification while a prior one was unacked. For
informational types (KNOWLEDGE_SHARE / MENTION / A2A_REQUEST / BROADCAST +
the pickup-proves-receipt triad) each send carries DISTINCT content (a new
learning, a new mention) and acking is voluntary, so a recipient who never
acks the prior one let the dedup permanently suppress every subsequent
same-sender broadcast - silent learning-broadcast data loss.
The dedup's anti-loop rationale (stop unacked-set inflation soft-blocking
i_am_idle) only holds for action-required signals. Gate the dedup on
ACK_REQUIRED_BY_TYPE.get(type, True): action-required types still dedup,
informational types always create. Unmapped types default True (dedup on).
TDD red->green; ruff + mypy clean; notification + dedup suites green (20).
* [F011] playbook: de-index rejected/archived playbooks from the PLAYBOOKS RAG index
* [F012] release_executor: fail-closed on git add/commit before push
* [F013] release_proposal: Redis SET NX mutex guards the ~40min execute against concurrent approves
* [F015] flow_qa/flow_doc: add i_am_blocked route (manifest-registered escape hatch was 404)
* [F018] claim_guards: treat blocked as active + broaden the guard lookup so a blocked dev can't double-claim
* [F019] git: clear orphaned .git/*.lock files after a timeout-SIGKILL'd mutation op
* [F031] identity: role_for_slug_or_none so defensive skip-guards don't crash the dispatcher tick on stale slugs
* [F032] test: unknown-assignee claim reaches release-to-pending path
F031's role_for_slug_or_none fix made the unknown-assignee release branch
in _dispatch_claimed_without_agent reachable (the human-only guard no
longer raises/short-circuits on a stale slug). Lock that reachability in:
a claimed task with an unknown-assignee UUID past grace returns the slug
(not None) so get_agent_role -> 'unknown' releases the claim to pending
for a role-matched reclaim.
* [F033] orchestrator: capture container_id at startup re-adoption
_readopt_running_agents registered re-adopted ACTIVE instances with
container_id=None. _check_health skips container_id-is-None instances, so
when a re-adopted container later exited the stopped-container handler
never ran and the task stranded under a phantom ACTIVE instance forever.
Add _resolve_container_id (docker inspect -f '{{.Id}}') and store the real
id on re-adopt. Best-effort: a probe failure degrades to None (still
ACTIVE; the reaper's Docker-liveness fallback covers it).
* [F034] orchestrator: re-stamp respawn last_check at restore
_pm_made_rule_following_retry bounds its tracing_gap audit lookup with
since = record.get('last_check'). A stale persisted last_check from before
the restart matched pre-restart tracing_gap rows, falsely resetting the
breaker on the very first post-restart spawn — exactly when a fresh strike
count should be evaluating current state.
_partition_respawn_rows now re-stamps last_check to the restore time on
every restorable entry, bounding the lookup to post-restart gaps only.
* [F035] orchestrator: probe-resume loop actually revives parked agents
_park_provider_unavailable parked the provider + offlined the instance but
never registered a WaitingRecord, so _on_probe_success -> _parked_agents_for
(always filtered on waiting_for=='rate_limit_lifted') returned [] and
resolve_wait revived nobody — recovery fell to the 600s stale-claim reaper
instead of the probe-success path the parking design relied on.
Register + persist a rate_limit_lifted WaitingRecord at park time (mirrors
mark_waiting_long, minus stop_agent — the container is already dead).
Companion reaper guard: _reap_with_service now skips provider-parked
assignees (_assignee_is_provider_parked) so the claim survives until the
probe revives the agent — otherwise the reaper releases the claim to pending
and probe-success respawns on a task the agent no longer owns.
* [F036] orchestrator: read transcript for overload detection too
The SDK server writes model-API errors (529/500/503) to /tmp/sdk-server.log,
not stdout, so an overload marker can appear only in the durable Claude
transcript — the same rationale already applied to the session-limit
detector. _provider_overload_park_target read only docker logs, so an
overload was missed and the agent crash-respawned straight back into it.
Now concatenates the transcript tail before matching, mirroring the
rate-limit path.
* [F037] orchestrator: drop bare error-NNN overload markers
The bare 'error 529'/'error 500'/'error 503' markers were broad enough to
false-match an agent that merely writes about an HTTP status code in its own
notes ('the endpoint returned error 500, retrying'), parking the whole
Anthropic fleet on a non-issue.
The SDK error formatter emits 'API Error: NNN' + a JSON error type, so the
remaining 'api error: 529/500/503' + 'overloaded_error' +
'internal_server_error' markers cover every real overload without that
false-match surface.
* [F038/F039] orchestrator: sign X-Agent-Token on self-API calls
The prior self-PATCH 401 fix only carried X-Agent-ID/X-Agent-Role. Arming
ROBOCO_AGENT_AUTH_REQUIRED=true made the middleware require a signed
X-Agent-Token, so every orchestrator self-call (auto-block / auto-resume /
auto-recover / SLA annotation) 401'd and silently no-op'd — wedging
paused/blocked parents.
Add _system_api_headers() that wraps the base headers with a signed token
for the system identity (issue_agent_token); switch all six self-call sites.
Dev fallback: no secret set => UNSIGNED sentinel + auth not required.
* [F040] orchestrator: finalize grok spawn session on cost-cap kill
_enforce_grok_cost_budget killed + evicted the container without calling
_finalize_spawn_session, so the open agent_spawn_sessions row stayed open
(ended_at IS NULL) and the burned usage/cost was never recorded in the
dashboard.
Call _finalize_spawn_session(exit_reason='cost_cap') BEFORE popping the
instance — it reads self._instances[agent_id] for the model +
usage_session_id, which the pop would lose.
* [F041] park grok exit-78 (auth missing/expired) instead of crash-retrying
A one-shot grok container whose entrypoint ran grok_auth --check and found
the token missing/expired exits 78 (EX_CONFIG). Crash-retrying 3x burns
tokens for zero progress — the agent cannot start without a valid token.
Park the provider with kind=auth_missing (same shape as the 429 exit-75
path) so the probe-resume loop revives the task once grok_auth.refresh_if_stale
mints a fresh token; if still expired, the next exit 78 re-parks (no burn).
Also fixes a latent F035 regression: _park_provider_unavailable now registers
a WaitingRecord, so the bare-__new__ rate-limit park test had to set
_waiting_records + stub _persist_waiting_record (mirrors the overload-test
fixture).
* [F042] isolate concurrent-duplicate conventions cache put in a savepoint
Two task creates for the same project/HEAD can race to populate the
conventions cache; the loser's INSERT fails the partial-unique index with
IntegrityError. A bare session.add + flush poisons the shared session (the
task-create transaction rides the same session), so every subsequent op
raises 'this session is in error state' and task creation crashes.
Run the INSERT in a savepoint (begin_nested) and swallow the IntegrityError:
only the savepoint rolls back, the outer transaction stays usable, and the
winner's row satisfies the next _cache_get.
* [F043] guard escalate_up against resurrecting terminal tasks
escalate_up had composes=() and no source-status guard, so a PM could
escalate a COMPLETED/CANCELLED task and apply_escalation set it back to
BLOCKED — bypassing the state machine's terminal-state invariant.
Defense in depth:
- spec: add PRECONDITION_NON_TERMINAL to escalate_up's extra_preconditions so
the lifecycle gate rejects terminal tasks (invalid_state) before the
journal:decision write fires; generalize _check_intent_preconditions to
honor non-tracing rejection_kind (not_authorized / invalid_state).
- service: apply_escalation (the single write primitive) returns False and
refuses to mutate a terminal task — covers the HTTP escalate route which
bypasses the spec gate. escalate() / escalate_up_to_role() return None on
refusal so the gateway emits a clean invalid_state envelope.
- route: the HTTP escalate route 409s a terminal task BEFORE sending the
escalation notification (so a finished task isn't yanked back, PM not pinged).
* [F044] pr_pass gate remediation points the reviewer at pr_fail, not i_am_blocked
The pr_pass gate runs the toolchain + conventions guards on the REVIEWER's
workspace, but their remediation text said 'call i_am_blocked' — a verb the
PR reviewer does not have. The reviewer would chase a verb they cannot call
instead of rejecting the PR.
Make the guards reviewer-aware: a reviewer=True flag (passed by _pr_pass_blocked)
switches the remediation to pr_fail(issues=[...]) — the reviewer's reject
lever, sending the PR back to needs_revision for the dev to fix the
environment / validator. The dev (i_am_done) path keeps i_am_blocked, which a
dev does have. _conventions_guard (the pr_pass path) now passes reviewer=True
through to _conventions_rejection.
* [F045] rate-limit: loud activate-failure log + in-memory orphan-probe fallback
The in-verb i_am_blocked(rate_limited) path wrapped RateLimitStateTracker.activate
in a bare contextlib.suppress. A silent activate failure stranded the fleet:
agents were parked in _waiting_records but the provider never entered the tracker,
so the tracker-driven _sweep_rate_limit_probes never probed it and no
_on_probe_success ever resumed them — parked agents stuck in WAITING_LONG.
Fix: (1) replace the bare suppress with a try/except that logs an error event
naming the provider + affected agents; (2) in _sweep_rate_limit_probes, after
probing the tracker-listed set, scan _waiting_records for any rate_limit_lifted
provider the loop did NOT cover and probe it via the time-expiry fallback (empty
state -> probe now) so _on_probe_success resumes the parked agents. The fallback
reads only local memory, so it still resumes when Redis was down at park time
(list_rate_limited_providers failure now falls through to the orphan scan instead
of returning early).
* [F046] pr_gate: guard None runner result on concurrent transition (pr_pass/pr_fail)
_gate_decision dereferenced the verb-runner result without a None guard.
run_intent returns None when a concurrent transition (cancel or a racing
reviewer) moves the task out of awaiting_pr_review between the precondition
gate and the runner's final composed action (the verb runner's documented
last-action source-status contract). The subsequent t.assigned_to /
t.status / _post_gate_review_to_pr(t, ...) dereferences then crashed the
gate with a 500 AttributeError. Add a None guard that surfaces a clean
invalid_state rejection (re-fetch + re-issue) before any dereference; no
PR post or a2a runs against a None task. TDD test_pr_gate_notifies_pm.py (+2).
* [F047] conventions: reviewer-aware block-finding remediation on pr_pass gate
The pr_pass (reviewer) conventions guard reused the dev-path block-finding
remediation: 'add a waiver to .roboco/conventions.yml in your branch'. A
pr_reviewer does not own the assembled cell->root / root->master branch and
has no commit verb on it, so the waiver remediation is unreachable — a false
positive stranded the gate with no self-recovery (the reviewer could neither
commit a waiver nor pr_pass). The fail-open content path is documented
precision-over-recall and stays as-is; the actionable gap is the remediation.
Fix: _conventions_rejection now branches the block-finding remediation on
reviewer=True (mirroring the could_not_run branch from F044). The reviewer
path points at pr_fail carrying the findings as issues so the PR returns to
needs_revision and the DEV fixes the violation or commits the waiver (the dev
CAN commit to the branch); waiver authorship is framed as the dev's action,
not the reviewer's. Dev i_am_done path wording unchanged. TDD
test_conventions_gate_pr_pass.py (+1).
* [F048] notify: reject human-only recipients (prompter/secretary) — no agent ack path
notify() only checked the SENDER role. The recipient was resolved by
NotificationService._resolve_recipients, which drops only unresolvable slugs
— it does not exclude human-only roles. The prompter (intake-1) and secretary
(secretary-1) are seeded agent rows, so they resolved, and an ack-required
ALERT addressed to them sat permanently unacked (no agent auto-acks it),
polluted the panel's pending-ack view, and — via the dedup query's
~acked_by.contains — permanently suppressed any later same-purpose
notification from the same sender to that human role. The knowledge-share
path already excludes all three human-only roles; the general notify path
did not.
Fix: a recipient-role guard in notify() via _reject_disallowed_recipient
(folds the new check into the existing CEO-dependency-block return slot so
notify stays under the PLR0911 return limit). Rejects prompter/secretary
with not_authorized; the CEO is human too but acks via the panel, so it stays
an allowed recipient (its only disallowed case, a dependency-block page, is
preserved). TDD test_notify.py (+3: reject prompter, reject secretary, allow
CEO).
* [F049] merge_pull_request: idempotent on already-merged PR (mirror _merge_with_retry)
* [F050] merge_pr_for_task: verify caller pr_number matches task's recorded PR
* [F051] open_conventions_pr: refuse dirty tree + verify checkout-base landed
* [F052] pr_target: scope task lookup by project_id (mirror close_pull_request)
* [F053] _token_for_project: log decryption failure (key rotation) with project slug
* [F054] learnings index: enforce shareable on every shared retrieval path (private-leak fix)
* [F055] messaging: recover from concurrent channel auto-create race via savepoint + re-fetch
* [F056] messaging: lock group row before session check-then-create to prevent active-session orphan race
* [F057] playbook: index/unindex as a post-commit step so the RAG corpus never leads the status transaction
* [F058] release-readiness: non-empty bump plan on first release
_canonical_bump_files derived the bump set from the previous
chore(release): commit. On the first release there is no such commit,
so it returned [] -> assess set version_bump_plan=[] -> the executor
published a tag with no files bumped (a no-op masquerading as X.Y.Z).
Fall back to the version-reference scan when no prior release commit
exists: the files currently embedding the version are exactly the set a
first release must bump, and the set the first release commit then
records as canonical for subsequent releases. Read-only derivation; the
CEO-approval gate and fail-closed executor are untouched.
* [F059] self-heal: hold fix tasks for CEO Approve-&-Start (restore dispatch gate)
The module docstring promised self-heal fix tasks 'wait for the CEO's
Approve-&-Start', but _originate created them confirmed_by_human=True and the
orchestrator dispatched them at once — a self-heal fix that re-broke CI would
trigger another cycle, open another auto-dispatched fix, and loop with no CEO
gate on dispatch.
Restore the documented gate:
* _originate opens the task confirmed_by_human=False (held for the CEO).
* The orchestrator holds a self-heal task out of both the PM and dev dispatch
paths until confirmed_by_human flips True.
* approve_and_start (the CEO's start gate) sets confirmed_by_human=True so the
held task finally dispatches (idempotent for board/intake tasks already True).
* list_pending_for_agent scopes the give_me_work hold to self-heal
(source != self_heal OR confirmed_by_human) so an already-alive PM can't grab
it pre-approval — while ordinary delegated subtasks (confirmed_by_human=False
by default, where the delegation IS the authorization) still dispatch.
The 'never self-deploys' guarantee (no merge) is unchanged.
* [F059] fix DB-integration test auth + retype self-heal root code→planning
conftest test-DB defaults matched the project's own running postgres
(roboco/roboco @ localhost:15432, the docker-compose roboco-postgres
service with CREATEDB) instead of the OS user on localhost:5432 which has
no such role — every db_session test failed with InvalidPasswordError
instead of running.
Once the DB connection worked, the self-heal origination DB test went RED
with MAIN_PM_NO_CODE: the self-heal root was task_type=CODE owned by
main_pm, the combo the main_pm_cannot_own_code guard rejects. The Main PM
coordinates the fix (delegates the code work to a cell dev); it has no
code verb. Retyped CODE→PLANNING and rewrote description/AC to
coordination-level.
* [F060] emit reversal audit row on claim-branch-failure rollback
The forward task.claimed audit row is flushed before the branch-creation
attempt, and AuditService commits on its own connection, so the rollback's
flush reverts the task row but not that audit row — the journey's last
event stayed task.claimed while the task reverted to its pre-claim status,
diverging from real state and corrupting downstream cycle-time/bottleneck
metrics. The rollback now emits a CLAIMED->original reversal audit row
(only when the forward transition was made) attributed to the claimant.
* Removing completely unnecessary files (for the repo they are unnecessary)
* [F061] audit status-transition rows now written in-session (F061/F073/F075)
_emit_status_transition_audit now writes AuditLogTable rows into
self.session synchronously (session.add) instead of dispatching
AuditService.log_task_event fire-and-forget on its own connection.
The audit row now commits/rolls back atomically with the status
transition in the caller's transaction, closing three facets at once:
- F061: audit commit no longer decoupled from the transition commit
- F073: a committed transition can no longer have NO audit row
(the row rides the same transaction; a swallowed persist can't drop it)
- F075: a transition rolled back inside a verb savepoint no longer
leaves a phantom audit row (the row is in the savepoint too)
log_task_event is now called only from this helper (narrow blast
radius verified); revision_count increment stays at this single
chokepoint. Cycle-time/bottleneck reconstruction from task.<status>
events is no longer silently corruptible.
Tests: test_emit_status_transition_audit_writes_in_session_atomically,
test_finalize_claim_rollback_emits_reversal_audit, escalation-audit
tests retargeted to in-session AuditLogTable rows.
Also: _canonical_bump_files grep-looseness follow-on (F058) -- filter
by subject, not body; git log --grep matches any message line, so a
non-release commit whose body references chore(release): shadowed the
real release commit. Test
test_canonical_bump_files_ignores_body_only_chore_release_match.
* [F061] drop type:ignore from audit-emit tests
Convention: no type:ignore/noqa. The F061 in-session audit-emit
tests used '# type: ignore[assignment]' to assign a MagicMock to
AsyncSession.add, and the F060 test assigned to .flush the same way.
Rewritten to hold a local 'session: MagicMock' variable (mypy sees
its auto-children as MagicMock, so .add.side_effect / .flush assign
cleanly with no suppression). Verified via 'mypy tests/' that both
files are now type-clean (the F060/F061 commits had skipped tests/
in mypy, masking two method-assign errors).
* [chore] clear all 64 pre-existing mypy errors in tests/ (no type:ignore)
Convention: no type:ignore/noqa, and pre-existing violations still
violate. The make-quality gate runs 'mypy roboco/ tests/', but the
prior commits' gates only ran mypy on production files, masking 64
type errors across 15 test files (method-assign, unused-ignore,
no-untyped-def, attr-defined, union-attr, has-type, index, misc).
Fixed without any type:ignore:
- method-assign (svc.session.X = / svc.method = AsyncMock()): hold a
local 'session: MagicMock'/'AsyncMock' and assert on it, or stub via
object.__setattr__ / monkeypatch / a typed '_bind' helper returning
Any, or alias 'cc: Any = c' (the pattern the file already used).
- unused 'type: ignore[assignment]' (real code was method-assign):
removed; replaced with the no-suppression patterns above.
- 'Callable[...] has no attribute assert_*': keep a typed local ref to
the AsyncMock and assert on the local, not the method-typed attr.
- no-untyped-def: annotate helper params (Any / pytest.MonkeyPatch).
- attr-defined / index / union-attr: type the helper as Any, narrow
with an 'is not None' assert, or add the missing attr to a fake.
- has-type / return-value: fix the declared return type to the tuple
the function actually returns.
- PLC0415 inline imports: hoisted to top-level.
test_pr_gate_notifies_pm._stub_gate_path converted fully to the
'cc: Any = c' alias (it already used it for one attr) so its five
'# type: ignore[method-assign]' suppressions are gone.
mypy tests/: 64 errors -> 0 (538 files). ruff check tests/: clean.
All 84 tests in the touched files pass.
* [chore] remove all remaining type:ignore suppressions from tests/
Converts 115 `# type: ignore[...]` suppressions across 23 test files to
no-suppression patterns (helper-return widening to Any, local Any aliases,
cc:Any aliases, cast at narrow call sites, typed fixtures) so the hard
no-type:ignore convention holds across tests/. No test logic or assertions
changed — only mock-wiring mechanics and type annotations.
Gate: ruff check tests/ clean; mypy tests/ (538 files) clean; 176 changed-file
tests pass. Zero real suppressions remain (the 7 grep hits are 3 hygiene-
checker string-literal test inputs and 4 prose mentions in comments).
* [F062] work_session.merge_pr: idempotency + active-status guard
merge_pr unconditionally set pr_status=merged, pr_merged_at, merged_by,
status=COMPLETED on whatever session it loaded — the only session-terminal
transition in WorkSessionService lacking both the active-status guard
(complete/abandon) and the terminal-idempotency guard (close). Two failure
modes: (1) a retried merge after a successful-but-unconfirmed GitHub merge
overwrote merged_by/pr_merged_at with the retry's actor/timestamp, corrupting
the merge audit trail; (2) merge_pr on an ABANDONED session resurrected it to
COMPLETED, undoing the single-active abandonment. Mirrors close()'s guard:
if status != ACTIVE, return the session unchanged. Both git.py callers await
merge_pr and discard the return, so the no-op is safe. TDD: 3 tests
(happy-path + both modes).
* [F063] workspace._clone_repo: rmtree half-configured clone on failure
If _configure_git raised CalledProcessError before its `remote set-url`
scrub, .git/config kept the tokenized auth URL (the project PAT) and
_assert_no_pat_leak never ran. The except clauses raised WorkspaceError
without removing the workspace, so the next ensure_workspace's health
short-circuit (valid .git with HEAD + objects) skipped past the leak —
mounting the agent on a workspace whose .git/config let it read+exfiltrate
the PAT. Both clone-failure except clauses now rmtree the workspace before
raising, so a half-configured clone is destroyed and ensure_workspace
re-clones from scratch. TDD: 2 tests (configure-failure leak + timeout).
* [F067] flow_main_pm: add missing /triage route
main_pm's manifest advertises triage (lifecycle.intents_for_role(MAIN_PM)
includes it via _PM_ROLES, alongside triage_all) but flow_main_pm.py had no
POST /triage route, so a main_pm agent calling triage hit a raw 404 that
bypassed the per-verb circuit breaker. Added the route mirroring flow_cell_pm's
/triage — wires to the existing team-scoped choreographer.triage (uses pm.team,
works for any PM role; Main PM gets its own team's blocked/awaiting tasks).
Fix direction: add-route, NOT remove-from-manifest — the manifest is spec-correct
(intents_for_role by construction); removing triage would contradict the spec
and leave main_pm with only cross-team triage_all. TDD: test_triage_route_exists_and_dispatches.
* [F068][F069] mcp servers: classify all rejection shapes + envelope 404s
F068: the do/flow-server circuit breaker only counted rejections whose
`error` field was a STRING in _CIRCUIT_REJECTION_KINDS. A 422 validation
failure (no `error` field, a `detail` list) and a 500/HTTPException
(dict-shaped `error` from the exception handlers) both bypassed the breaker
→ unbounded retries on a storm of either. Added _classify_rejection(payload)
(shared, applied to both servers) mapping all three shapes to a counted kind:
string error (existing), dict error → substring-mapped code
(*DENIED*/*AUTHORIZED*/*FORBIDDEN*/*PERMISSION*→not_authorized,
INVALID_INPUT/*VALIDATION*→incomplete_input, *NOT_FOUND*→None parity, else
→invalid_state), 422 detail→incomplete_input. The dict TypeError defence lives
in the classifier (isinstance, never dict-in-frozenset).
F069: a manifest-registered verb whose HTTP route is missing got FastAPI's raw
`{"detail":"Not Found"}` 404 body — a non-envelope payload the breaker
couldn't classify, so a storm bypassed it. _post now synthesizes an
invalid_state Envelope rejection (with a remediate hint → i_am_blocked/i_am_idle)
for a 404 status, routed through _record_and_check_circuit so the breaker counts
it. A 404 that carries a real Envelope (error field present) is surfaced as-is,
preserving test_flow_post_returns_envelope_on_404. TDD: 422/dict/404 tests in
both server test files; updated test_dict_shaped_error_does_not_crash to assert
the SDK is now called with not_authorized (replacing the pass-through assertion
that encoded the bug).
* [F064][F065][F066] websocket: non-blocking fan-out, finally-disconnect, idle timeout
F064: the bridge forwarder awaited every conn.send_text in a gather with no
per-connection queue and no send timeout — one slow WS client back-pressured
ALL event delivery to ALL clients (head-of-line blocking on the listen loop).
Each connect_* now registers a _ClientConnection (bounded asyncio.Queue(256) +
sender task); broadcasts enqueue via put_nowait (drop + structlog warn on
QueueFull) and return immediately. The sender drains the queue with each send
wrapped in wait_for(SEND_TIMEOUT=10s). Unregistered legacy sockets (set
directly into a subscription set, bypassing connect_*) get a timeout-bounded
fallback send task held in _pending_sends (ruff RUF006). disconnect cancels +
drops the sender.
F065: route handlers caught only WebSocketDisconnect with no finally — a
non-clean exit (anyio closed-resource, CancelledError, transport error)
propagated without manager.disconnect, leaking the dead socket into every
subscription set forever. Added finally: manager.disconnect(websocket) to all
5 handlers (disconnect is idempotent).
F066: no server-side heartbeat/idle timeout — a half-open socket from a dead
container blocked receive_text forever and was never reaped. receive_text now
wraps in wait_for(IDLE_TIMEOUT_SECONDS=90s); on TimeoutError, log + fall
through to the F065 finally. Named module constants (no config.py precedent for
WS tuning; callers/tests patch them).
TDD: 22 new tests across 3 files (handler cleanup, idle timeout, send queue),
non-flaky across repeats; 1 existing test adapted with a yield for the new
async fan-out (assertion unchanged). ruff/mypy clean, 421 unit/api tests pass.
No type:ignore/noqa.
* [F022][F023][F024][F025][F026] api: scrub secrets from 422 log, gate a2a/dashboard/orchestrator routes, SSE session-per-query
- middleware: redact known credential fields (git_token/api_key/token/...)
from the 422 request-validation log line; response body unchanged
- a2a: require_any_authenticated_agent on /message/send + /message/stream;
subscribe_to_task opens a short-lived session per poll instead of holding
one asyncpg connection for the full SSE lifetime (pool exhaustion) + auth
- dashboard: gate auditor flag/report mutating routes to Auditor or CEO
- orchestrator: router-level CEO gate on all control routes (spawn/stop/...)
TDD; ruff/mypy clean; 449 unit/api tests green; no type:ignore/noqa.
* [F030] conventions: typescript-scoped custom rules now apply to .tsx files
The validator tags a .tsx file as language 'tsx' (the JSX grammar needs
that tag, distinct from plain 'typescript'), but a custom rule scoped to
'typescript' — the language the scan reports for a React+TS repo — silently
skipped every .tsx file. The two suffix maps were NOT unified: the 'tsx'
tag is load-bearing (grammars.py picks the JSX grammar on it; hygiene.py
keys on it), so unifying would make .tsx fail to parse.
Fix is in check_custom: a one-directional dialect map _DIALECT_OF =
{'tsx': 'typescript'} — a typescript-scoped rule fires on a .tsx file,
but a tsx-scoped (JSX-only) rule still does not fire on plain .ts.
TDD; ruff/mypy clean; 80 unit + 38 integration conventions tests green.
* [F029] websocket: remove broken /api/permissions/check loopback from channel stream
channel_stream called validate_channel_access, which HTTP-loopbacked to
GET /api/permissions/check — a route that does not exist. Every call 404'd
-> False -> the channel stream closed with WS_1008_POLICY_VIOLATION for
EVERY client, so the real-time channel stream was dead. Removed the
function, its call site, and the now-unused httpx + settings imports.
Post-F004 the panel-token gate is the channel-stream authorization (the
CEO panel is the sole WS client and may view every channel), so the
broken loopback is removed rather than replaced with an in-process check
the CEO always passes. The legitimate enforcement.validate_channel_access
(slugs, in-process static ACL) is a different function and is untouched.
F027 is resolved-by-F004 (no code change): all three per-agent streams
gate on _require_panel_token first, so only the authorized CEO panel can
connect — 'any viewer subscribes to any target' is closed.
TDD; ruff/mypy clean; 530 unit/api+enforcement+RBAC tests green.
* [F078] release_executor: deadline every subprocess (git/make/gh/clone)
A hung git/make/gh/clone would block the CEO-gated release loop
indefinitely. Wrap each proc.communicate() in asyncio.wait_for via a
shared _await_proc helper; on expiry proc.kill() the child and return a
non-zero rc (124) so every caller's fail-closed branch fires. Mirrors the
quality-gate _run_one kill-on-timeout idiom.
Deadlines are generous (30min gate / 10min clone / 5min push+gh) so a
legitimate slow op is never wrongly aborted — floor-assertion tests pin
the floors to guard exactly that logical regression. Green path returns
the real rc unchanged.
* [F072] reaper: deadline docker inspect/exec + harden _check_health sweep
A hung Docker daemon (or a stuck container FS) froze the single asyncio
event loop: the reaper runs inline before every dispatch tick and shares
that loop with every background sweeper. Bound each docker subprocess
with asyncio.wait_for; on expiry proc.kill() the child and either raise
(inspect / resolve_container_id — callers apply their own fail-direction)
or return None (the gateway probe — inconclusive, caller declines to act,
matching its existing probe-failure contract). Deadlines generous
(10s inspect / 30s exec) so a legitimate slow docker call is never
wrongly aborted; floor-assertion tests pin the floors.
Also harden _check_health's per-agent loop so one agent's hung inspect
skips that agent, not the whole sweep — preserving the check-all-agents
invariant the timeout-then-raise would otherwise break (without this, a
hung daemon means no agent gets health-checked any tick).
* [F076] say/dm: handler guard rejects all 4 no-comms roles, not just auditor
The say()/dm() defence-in-depth guard only rejected auditor, but CLAUDE.md
mandates the same no-agent-comms invariant for pr_reviewer (posts findings
on the PR), prompter and secretary (human-only, note + evidence). For those
three the manifest was the only gate, so a call bypassing the manifest
(direct API POST, test harness, future routing change) would not be refused
at the handler — admission depended on the agent's slug happening to be
absent from the channel/a2a matrix. Extend the guard to a _NO_COMMS_ROLES
frozenset (auditor + pr_reviewer + prompter + secretary), matching the
explicit role-frozenset gates on commit/notify/pitch/playbook/open_session.
Role-appropriate remediation per role. The claimed defence-in-depth now
covers 4 of 4 silent roles, not 1 of 4.
* [F070] drain fire-and-forget _bg_tasks on shutdown (bounded, data-preserving)
Orchestrator.stop() cancelled only the named loop tasks + agents, then
returned, abandoning in-flight _schedule_bg work. An in-flight
_persist_respawn_record upsert dropped at shutdown meant the last few
gate-mutation strikes never reached the DB; restore_respawn_tracker() on
the next start repopulated a stale lower count and the dispatcher re-burned
the full 4-spawn strike threshold against a still-wedged task — the exact
re-burn the durable tracker exists to stop. Audit-log writes (load-bearing
for cycle-time/rework metrics) were similarly dropped.
Add _drain_bg_tasks(): bounded wait (5s default) lets short DB writes
commit before exit (data preserved), then cancels any stuck task past the
deadline so a hang can't wedge shutdown. return_exceptions=True so one
failing bg task doesn't crash the drain. Wrap the stop_agent loop in
try/except + logger.exception so one bad agent can't skip the drain
(re-introducing the data-loss tail). Floor test pins the deadline >= 3s
so a too-short change can't silently drop a legitimate slow write.
* [F071] abort non-blocking intake/secretary spawn on mid-spawn shutdown
The non-blocking spawn (start_intake_session / start_secretary_session)
schedules _spawn_intake_container_guarded / _spawn_secretary_container_guarded
via _schedule_bg. Those run docker run and only register in _instances at the
END. If shutdown arrived between docker run and the registration line, the
container was started but the orchestrator had no handle — stop() iterates
only _instances, so the container was orphaned (leaked, manual docker rm).
Worse, the F070 drain could let the spawn coroutine complete the
registration AFTER stop() already iterated _instances, landing a live
container into a shutting-down registry nothing tears down.
Add a post-docker-run shutdown guard in _spawn_intake_container and
_spawn_secretary_container: re-check self._running after _run_container_cmd
returns; if the orchestrator began shutting down, remove the just-started
container (by its deterministic name) and raise _SpawnAbortedDuringShutdown
WITHOUT registering. The guarded wrappers catch that BEFORE except Exception
and close the live relay silently (shutdown is not a user-facing failure,
no error pushed to the SSE stream). The F070 stop() drain awaits the bg
spawn coroutine, so the abort surfaces cleanly.
TOCTOU-safe: between the _running check and the _instances assignment there
is no await (config + instance construction are sync), so once the check
passes, registration completes before the event loop can interleave stop().
The normal running path is unchanged (sanity tests pin it).
* [F074] per-agent advisory lock closes claim TOCTOU
_run_claim_guards read the agent's other tasks via unlocked SELECTs
before claim() took its row lock, and claim()'s FOR UPDATE locked only
the TARGET row — so two concurrent i_will_work_on by the SAME agent on
TWO DIFFERENT pending tasks each locked their own row, each read an
empty in_progress set, each passed already_active, each claimed+started
→ the agent ended with two in_progress tasks (the in-process asyncio
Lock is lost on orchestrator-restart split-brain, so it wasn't a
DB-level guarantee).
Fix: TaskService.acquire_claim_lock takes a transaction-scoped
pg_advisory_xact_lock keyed by hashtextextended(agent_id). The gate
acquires it BEFORE the guard reads (for non-coordinator roles only) so
the second concurrent claim's read sees the first's committed
in_progress task and is rejected. Tx-scoped → auto-releases on
commit/rollback, can't outlive the request.
Coordinator exemption (the key logical-regression guard): cell_pm /
main_pm do NOT take the lock — the PM coordinator concurrency feature
lets a PM plan+delegate many roots in parallel, and a per-agent lock
would serialize those claims and regress it. Matches the existing
_COORDINATOR_ROLES already_active/paused guard exemption. A hash
collision only causes benign false serialization, never a false
negative.
Tests: unit (dev acquires lock before guard read; coordinator does
not) + real-PG integration (same-agent serializes, different-agent
does not, releases on rollback).
* [F021] handle SSE transport errors so the intake composer isn't stuck
openStream registered listeners for the server-sent event kinds but not
the EventSource's own transport-level error. The 'error' kind IS in
LIVE_EVENT_KINDS, so a server-sent event:error (JSON MessageEvent) was
handled — but a dropped connection / dead session fires a plain Event
with NO data, which JSON.parse(undefined) swallowed in the try/catch,
so the stream 'stayed open' (EventSource loop-reconnected a session that
no longer existed) and isSending stayed true — the composer was
permanently disabled.
Fix: route the 'error' event by payload. A MessageEvent with string
data is a server-sent error → handleEvent (unchanged). A no-data Event
is a transport error → handleTransportError: clear streamingId/activity,
set isSending false, add a 'connection lost' error message, keep a
draft/batch preview up (so the human can still act on a proposed card)
else land on 'chatting', and close the dead stream so EventSource stops
loop-reconnecting.
Tests: renderHook + a jsdom EventSource double that fires a transport
error (plain Event, no data) vs a server-sent error (MessageEvent +
JSON). RED: transport error left isSending true; GREEN: resets to
false, surfaces the message, closes the stream. The server-sent-JSON
path is unchanged. Full panel suite (129) green; eslint/typecheck/prettier clean.
* [F081] Approve dialog: label notes required (>=20 chars), not optional
The CEO Approve dialog's notes label fell into the default branch
('Notes (optional') for the approve action, but approve actually
requires substantive notes >= 20 chars — enforced client-side
(toast error on < 20) and server-side. So the CEO was told 'optional'
and only learned the real requirement from a toast after hitting
submit with empty notes.
approve and start both require >= 20 chars; reject only requires a
reason. Collapse the label to two branches: reject -> 'Reason for
rejection (required)'; everything else (approve + start) ->
'Approval notes (required, >= 20 characters)'. The approve
placeholder now also signals intent ('Why this is ready to ship...').
Tests: render the queue, click Approve, assert the notes label says
'required' + '20' and does NOT say 'optional'. RED: label read
'Notes (optional)'; GREEN: 'Approval notes (required, >= 20
characters)'. eslint/typecheck/prettier clean.
* [F082] surface release-proposal query failures instead of silent hide
The card collapsed any non-404 backend failure (500 / network drop) onto
`!proposal` and returned null, so the CEO had no idea the release-proposal
endpoint was unreachable. Distinguish the cases: isError + a Retry affordance
vs the 404 null empty state that stays hidden. Mirrors PrReviewQueue.
* [F083] clear stale usage snapshot when /ws/system leaves connected
The hook synced wsState into the store but never dropped usageData when the
stream dropped, so on reconnect wsState flipped to "connected" before any
fresh USAGE_SNAPSHOT arrived and UsageOverviewPanel rendered the prior
session's totals/cost as if they were live. Clear usageData whenever state
leaves "connected" so the panel falls back to the polling summary until a
new snapshot lands. Connected->connected is a no-op clear skip.
* [F084] scope per-control disable to the in-flight mutation, not all
FeatureFlagsCard disabled every switch while any one flag toggle was pending,
and PlaybookReviewQueue disabled every row's Approve while any one approve was
pending — so the operator couldn't act on an independent control during a
slow round-trip. Gate the disable on the in-flight mutation's variables
(matching key / id) so only the control being mutated locks; the others stay
usable. The same-flag double-tap protection is preserved.
* [F085] reject submitting both project_id and product_id
validate() only checked 'at least one of project/product', so the dialog let
both be submitted together. The server silently lets product_id win at routing
and drops project_id, recording a misleading, never-used repo. Add a validator
that refuses the ambiguous submit with a clear error. The at-least-one rule and
the single-pick submit paths are unchanged.
* [F020] kanban: confirm admin-override drags that skip lifecycle preconditions
A drag on the operator kanban routes the status move through the admin
status-override, which bypasses the in-band lifecycle validator entirely.
That override is intentional (it's how an operator recovers a wedged task)
but it also let a careless drag skip material preconditions silently —
completing a task with no open PR, QA-bypassing, finishing docs on a task
whose docs aren't complete.
Leave the override intact but make the bypass explicit: compute the
preconditions the dragged move would skip (open PR, docs complete,
self-verified + commits + progress for submit-qa, visible non-terminal
subtasks for coordination-root targets) and, when any are skipped, hold the
move behind a confirmation dialog that lists exactly what's being skipped.
Precision over recall — only warn on what the panel can verify from the
task and its in-list children; never fabricate a 'satisfied' claim, and
stay silent on benign transitions that gate on nothing we can check.
The admin status-override capability is preserved (Confirm still fires it);
this only surfaces the bypass instead of letting it happen silently. Does
not touch the master-merge invariant — the board's updateTask is the
operator override, not the Main-PM merge path.
* [F086] prompter: restore parked cell content on project toggle off/on
rebuildCellWork appended a blank {summary:'', items:[]} entry for a newly-
selected cell, so toggling a cell's project OFF then back ON in the MegaTask
review card discarded the agent-authored per-cell summary/items — the entry
was dropped on toggle-off and re-added blank on toggle-on.
Park each draft's last per-cell content in client-only BatchProposal state
(parkedCellWork, keyed by draft index — never sent to the backend; confirm
ships only title/drafts/project_ids/route, and it ride-alongs into the
localStorage persist slice so the restore survives a reload mid-review).
rebuildCellWork gains an optional priorByCell map: a re-added cell with no
live entry restores its parked summary/items (with the new project_id) in-
stead of blanking; a live entry still wins over a stale parked copy so an
in-place edit is never regressed. parkCellWork is the pure merge seam
(prevParked seeds, live work overwrites) the setBatchDraftProjects updater
calls — kept pure so the updater stays a thin caller.
Tests: rebuildCellWork restore/blank-fallback/live-wins + parkCellWork
retain/overwrite/merge (6 new), 19 GREEN. eslint/typecheck/prettier clean.
No wire-payload change, no regression to the fill/drop/one-repo-per-cell
invariants.
* Updated domain
* [F087,F088] enforce panel token on live-chat bridges (Phase 5)
Add a CEO-bound, header-token-only gate (require_panel_token) at the route
level of the prompter_live + secretary_live bridges, which were the only
panel-facing API surface that ran unauthenticated. It mirrors the WS
_require_panel_token and _check_agent_auth_token contracts: in dev
(ROBOCO_AGENT_AUTH_REQUIRED unset) a missing token is allowed; a
presented-but-forged token is rejected even in dev; in prod nginx already
injects the CEO-signed X-Agent-Token on /api/ for GET + POST, so the SSE
stream (EventSource can't set headers) and the POSTs are now checked instead
of anonymous. Applied to start/stream/status/messages/stop on both routers;
preview_live_batch switched from CurrentAgentContext+noqa to the route-level
gate (genuinely auth-only). confirm/confirm-batch/re-interview keep
CurrentAgentContext (they use agent.identity). The container->relay /events
callback is intentionally left ungated (internal Docker network, opaque
session id) — gated by a test sentinel so Option B (spawn+SDK token wiring)
is a deliberate future decision. No panel/nginx/spawn/SDK changes; master
merge invariant untouched. 22 new TDD auth tests, 492 api tests green.
* [F089] honest WorkSession agent_id nullability across the read path
The work_sessions.agent_id column is nullable=True with ondelete=SET
NULL — deleting an agent nulls the FK on every session it ever held. The
ORM annotation lied (Mapped[UUID] non-optional), the converter papered
over the lie (typing_cast to a non-optional UUID), and the response
model rejected None outright (WorkSessionResponse.agent_id: UUID). A
session whose agent had been deleted crashed the GET endpoint with a
pydantic ValidationError instead of serializing agent_id: null.
Make the read path honest end-to-end:
- WorkSessionTable.agent_id: Mapped[UUID | None] (matches the column).
- WorkSessionResponse.agent_id: UUID | None (serializes null, no crash).
- session_to_response passes agent_id via typing_cast('UUID | None', ...)
to bridge SQLAlchemy's UUID[Any] to stdlib uuid.UUID while preserving
None-ness (the cast stays for the same mypy-plugin reason every other
field uses one; it no longer narrows away None).
WorkSessionCreate.agent_id stays UUID — at create time the claiming
agent is always known. The unused WorkSession pydantic read model is
left as-is (never materialized from a DB row). task.py:_needs_revision_dev
already None-guards ws.agent_id via to_python_uuid (returns None -> skip).
* [F090] drop auditor from write_roles on main-pm-board / board-private
The auditor is a silent, read-only observer on every channel, but the
channel catalog (roboco/foundation/policy/communications.py) listed it
in write_roles for main-pm-board and board-private 'for parity' with the
legacy CHANNEL_ACCESS table, while the actual silent-observer rule was
enforced only at the say/dm guard (content_actions._NO_COMMS_ROLES) and
PermissionService.can_write_channel's auditor short-circuit.
That left the catalog-only enforcement path — the HTTP messaging route
(messages.py send_message -> validate_channel_access) — authorizing an
auditor write that both the say/dm guard and PermissionService would
have blocked. A reader of the catalog also believed the auditor could
post to those channels, which is false.
Fix: remove Role.AUDITOR from write_roles on both channels (main-pm
+ board remain writers; ceo remains a writer on board-private). The
auditor stays in read_roles, so its silent read is unchanged. silent_roles
is left empty (matches the announcements precedent: auditor reads via
read_roles, not the silent bucket) — the DB seed and silent_observers
field are untouched.
Logical-regression check: the auditor's read access on both channels
is byte-for-byte preserved (still in read_roles, so validate_channel_access
read returns True via the direct list); the legitimate writers (main-pm,
product-owner, head-marketing, ceo) are untouched; CHANNEL_ACCESS is
derived from the spec so the foundation/seed drift tests self-adjust;
PermissionService.can_write_channel already short-circuited auditor to
False everywhere, so no behavior change there; AUDITOR_SILENT_ACCESS is
unchanged (auditor not added to silent_roles -> no DB silent_observers
change -> no group-access behavior change); the say/dm _NO_COMMS_ROLES
guard is unchanged. Tests: 3 new in test_channel_access.py — auditor
write on main-pm-board/board-private now raises ChannelAccessDeniedError
(RED before: returned True), auditor read still True, main-pm/ceo still
write.
* [F091] warn at spawn time when host grok auth.json is missing
GrokCliProvider._append_grok_auth_mount silently skipped the mount when
the host ~/.grok/auth.json was absent. The spawn still succeeded (docker
run returned 0 — the container was created), so the operator had no
spawn-time signal that the agent was doomed: the entrypoint's
`python -m roboco.llm.providers.grok_auth --check` backstop then
refused to start (exit 78) and the failure only surfaced later via the
container's log markers.
Fix: emit a spawn-time WARNING (module logger) naming the missing file
and the remediation (`grok login` on the host, or set
ROBOCO_HOST_GROK_DIR) when the mount is skipped. The spawn outcome is
unchanged — the container still starts and the existing exit-78 -> park
flow (F041) still catches it — but the operator now sees the missing
credential immediately instead of diagnosing a later exit-78.
Logical-regression check: the mount-present path is byte-for-byte
unchanged (auth.json exists -> the -v bind is appended, no warning); the
spawn still succeeds when auth is absent (no raise — the existing
test_grok_spawn_omits_auth_mount_when_absent still passes: no mount, no
crash); the exit-78 entrypoint backstop and the orchestrator's
exit-78-park handling (F041) are untouched; a module-level logger adds no
side effects. Tests: new test_grok_spawn_warns_when_auth_absent uses
caplog to assert a WARNING mentioning auth.json + `grok login` is
emitted on a missing-credential spawn (RED before: no warning; GREEN
after). 102 grok tests green; ruff/mypy clean.
* [F092] decode JWT exp when refresh omits expires_in
xAI's refresh-token response sometimes omits expires_in. Without it the
new access token kept the stale pre-refresh expires_at, so is_valid /
--check forever rejected a fresh token — and the refresh loop re-rotated
the single-use refresh token every tick, killing the credential (F006).
The access token is a JWT whose exp is the authoritative expiry: decode it
when expires_in is absent. Fallback to the documented ~6h TTL + a structlog
warning when the JWT exp is unreadable, so a fresh token is treated as live
instead of stale.
* [F093] serialize concurrent live-chat spawns under a per-agent lock
The intake and secretary agent ids are each a single fixed id, so two
concurrent start_intake_session / start_secretary_session calls raced on
the container name (docker run --name roboco-agent-<id>) and the
_instances[<id>] write: both passed the reap-prior check before either
registered, both ran docker run, and the last _instances write won,
orphaning the other container + its relay.
Add _intake_spawn_lock / _secretary_spawn_lock (asyncio.Lock) and wrap the
_spawn_intake_container / _spawn_secretary_container bodies so the second
start waits for the first to fully register before its own reap-prior check
runs. Distinct from self._lock (which stop_agent takes) to avoid a
reentrancy deadlock: the spawn body holds the spawn lock then calls
stop_agent (acquires self._lock) — lock order is always spawn_lock ->
self._lock, never the reverse.
* [F094] add a persistent-probe-failure escape hatch to provider parking
_on_probe_failure only incremented the failure counter and, at 10 failures,
sent a one-shot CEO notification. It never cleared the tracker, never gave
up, never fell back to time-expiry. _do_probe returns False for any non-2xx
AND any httpx error, so a permanently unreachable probe endpoint (removed
API key, network partition to the probe host, misconfigured base URL) kept
the provider parked forever — every agent on it gated by
_provider_spawn_parked, their tasks reaped to pending but the spawn gate
queuing every spawn, sitting pending forever. The only recovery was the
operator manually clearing the Redis key.
Past _PROBE_GIVE_UP_THRESHOLD (30) persistent failures, fall back to the
same time-expiry optimism the unprobeable-provider path uses (_do_probe
returns True when there is no probe URL): clear the park and resume parked
agents. If the provider is genuinely still down the real workload attempts
re-park via the 429/5xx path, so this is bounded burn — strictly better
than a silent forever-strand. Kept above the CEO-notify threshold (10) so
the operator still gets the notification first.
* [F095] orchestrator: parked-provider spawn short-circuits before expensive prepare
spawn_agent ran the full _prepare_agent_spawn (writes blueprint/settings/
briefing/MCP files, ensures the image, registers a STARTING instance) every
dispatcher tick only to bail at the after-prepare parked-provider check —
wasting all that file I/O while the provider stayed parked and leaving a
STARTING instance registered then downgraded to OFFLINE.
Move the parked check before _prepare_agent_spawn: resolve the route cheaply
via _resolve_agent_route (only provider_type is needed) and bail with a
minimal unregistered OFFLINE instance. The existing-running check stays
first (inside the lock) so a live agent is never replaced; a TOCTOU
re-check guards the unlocked window before prepare; the after-prepare
check is kept as a rare-race defense (a park landing during prepare).
* [F096] orchestrator: serialize fire-and-forget respawn persists per commit order
_persist_respawn_record is fire-and-forget per gate mutation; a respawn loop
fires count 1->2->3->4 in quick succession, scheduling one persist per
increment for the same (agent_slug, task_id). The ON CONFLICT DO UPDATE upsert
is row-level race-free, but the fire-and-forget tasks can still COMMIT out of
order: a slow stale persist (count=2) scheduled first can resolve AFTER a fast
fresh one (count=4) scheduled second, leaving the durable row at the stale low
count and re-burning the strike threshold on restart.
Fix: acquire self._respawn_persist_lock (new asyncio.Lock) as the FIRST await
in _persist_respawn_record, so acquisition order = task creation order (FIFO
ready queue) = logical schedule order, and commits land in that order. The
durable row always ends at the latest logical value. The lock lives in the bg
task, so the dispatcher hot path never blocks; persists are best-effort and
a slow one queuing the rest just delays the durable catch-up (in-memory record
stays authoritative).
* [F097] orchestrator: back off grok re-park retry_after within a rate-limit episode
_probe_target returns (None, {}) for grok — the grok CLI's xAI endpoint is
closed and the SuperGrok OIDC access token is not a valid bearer for the metered
api.x.ai, so a real probe would either no-op or strand grok parked forever.
_do_probe treats url-is-None as success (time-expiry optimism), so once the
60s retry_after passes the probe loop optimistically clears the grok park, a
cleared park dispatches a fresh grok agent that hits the still-active xAI 429,
exits 75, and re-parks — a flat ~90s crash-retry cycle for the whole xAI
rate-limit window (each cycle costs container startup + a rejected grok call).
Fix: track _grok_repark_count + _grok_last_park_at in _park_grok_rate_limited
and back the re-park retry_after off exponentially within one episode
(60 -> 120 -> 240 -> ... capped at 2**4 = ~16min cycle) so the churn dampens. A
gap past _GROK_REPARK_EPISODE_GAP_S (25min, > the capped cycle) means no re-park
for that long => the rate limit actually lifted => a fresh episode resets the
count to the base 60s, so recovery latency isn't penalized across episodes.
The first park in a fresh episode is unchanged at 60s.
* [F098] orchestrator: keep waiting record through a re-park during probe-success resume
resolve_wait deleted the waiting record (in-memory + durable) BEFORE calling
spawn_agent. A re-park in the window between the probe-success clear and the
spawn — the provider's rate limit lifts then immediately re-limits, or a second
provider limit lands — bails spawn with an OFFLINE instance (the parked-provider
short-circuit). Deleting the record first orphaned the agent: with no record
the probe-resume loop can never revive it and the spawn gate bails every tick,
so the agent is lost until the operator intervenes.
Fix: spawn first, then tear down the record only once a container actually
launched (instance.state == ACTIVE). On an OFFLINE bail the record stays so the
next probe-success re-attempts the resume. On a spawn EXCEPTION the record is
torn down + re-raised so the probe loop doesn't keep re-resuming a task that
moved to a different state (e.g. readiness refused -> task auto-blocked) —
matching the pre-fix behavior where the record was deleted before the spawn.
* [F099] wire pr_pass/pr_fail self_review block in the spec gate
The pr_pass/pr_fail ActionSpecs carry self_review_block=True, but
_gate_preflight never populated Context.original_developer_slug, and
actor_slug was read off agent.slug — which GatewayAgentView does not
carry, so it was always None in production. The block was structurally
dormant: a reviewer who was also the original developer of the
assembled PR could pass (or fail) their own work. The service-layer
_validate_not_self_review backstop only covers qa/documenter, not
pr_reviewer, so the spec gate is the only defense.
Set actor_slug=str(reviewer_agent_id) (GatewayAgentView has no slug,
so the UUID is the identity) and original_developer_slug from the
original_developer marker (a UUID stored as a string). Both resolve to
UUID strings, so the spec's string-equality comparison fires when the
reviewer IS the recorded original developer.
The marker is never set on assembled coordination tasks (only on
dev-leaf tasks at QA/doc claim), so the block stays dormant by design
in production — but the gate is now correctly wired to fire if the
marker were ever set to the reviewer. Zero production behavior change;
the dormant-in-production state is pinned by the no-marker test.
* [F100] atomic Redis probe-failure counter via server-side Lua
increment_probe_failures / reset_probe_failures did a non-atomic
get_state (GET) -> mutate -> set (SET) in Python. A concurrent
activate() re-park writes a FRESH episode blob (probe_failures: 0 +
fresh activated_at / retry_after / affected_agents / kind); if the
stale increment's SET landed after the fresh activate's SET, the stale
blob overwrote the fresh episode metadata AND un-reset the counter
(clobbering the new episode).
Redis single-threads a Lua EVAL, so a server-side read-modify-write
is indivisible: activate's SET is serialized entirely before or after
the script, never interleaved between the script's GET and SET. The
two scripts mutate ONLY probe_failures, so every other episode field
survives the bump. activate stays a single atomic SET (a fresh episode
resetting the counter to 0 is correct semantics).
* [F101] enforce PR-open state gate on gateway open_pr (parity with HTTP path)
* [F102] make project_id mandatory on pr_target (close cross-repo pr_number collision)
* [F103] make project_id mandatory on close_pull_request (close cross-repo collision)
* [F104] fail-closed on conventions resolution errors (block gate no longer silently disabled)
* [F106] compound (timestamp, id) keyset cursor for message pagination
get_messages used strict timestamp inequalities with a non-deterministic
order_by(timestamp.desc()), so equal-timestamp messages were cut by limit
on one page and excluded (strict < T / > T) from the next — they vanished
across pages. Bundled the (timestamp, id) pair into a MessageCursor dataclass
so the next page resumes exactly past the cursor's id at the shared
timestamp (or_: strictly-older OR same-timestamp-smaller-id for before; the
mirror for after), with a deterministic order_by(timestamp.desc(), id.desc())
so the last-item cursor is unambiguous. id is None for a legacy timestamp-
only cursor (strict inequality, prior behavior). The route builds cursors
from the flat before/before_id + after/after_id HTTP params; the schema now
carries the tie-breaker ids. Also clears PLR0913 (cursors replace the
before_id/after_id params).
* [F107] defer Redis bus publish until DB commit (no phantom notifications)
deliver() and _persist_and_deliver() ran inside the caller's open
transaction: the notification row was flushed but not committed, yet
NOTIFICATION_SENT was published to the Redis bus immediately. A commit
failure (DB hiccup, constraint, asyncpg error) rolled the row back while
connected WebSocket clients had already received a push for an id that
no longer existed — a phantom notification (notify_get -> NotFoundError).
Added a deferred-publish (transactional-outbox) helper: defer_bus_publish
enqueues the event on session.info and registers one-shot after_commit /
after_rollback listeners on session.sync_session the first time it is
called for that session. On commit, the after_commit listener schedules
the async drain via asyncio.create_task on the running loop (the listener
fires synchronously inside await AsyncSession.commit, so the loop is
active); the task handles are stashed on the session so callers/tests can
await them. On rollback, after_rollback drops the pending queue — a
rolled-back txn emits nothing. deliver() now builds the per-recipient
events up front (data materialized to strings, so deferral is safe even
if the ORM object later expires) and defers each; the delivered_at DB
marker stays in-tx (rolls back with the row). The bus block stays
best-effort (try/except + log) so a bus-init failure never propagates or
rolls back the notification row — matching the prior inline semantics.
This fixes every deliver/_persist_and_deliver caller at once (the two
cited in F107 plus the orchestrator + task.py deliver sites), since they
all commit the session afterward (the deferred publish fires on that
commit; the row is durable by the time the event goes out).
* [F108] atomic replace_chunks: single-txn delete+insert closes reindex race
* [F109] playbook curation status guards: approve/reject draft-only, archive approved-only
* [F110] draft slug TOCTOU: catch IntegrityError on flush -> ConflictError (no 500)
* [F113] collapse WorkSession creation to the validated service path
_create_work_session_if_needed constructed WorkSessionTable directly,
duplicating WorkSessionService.create's validation (existing-active
check, single-active-per-task supersede, project/task existence). The
two sites had drifted. Route through WorkSessionService.create instead,
mapping ConflictError to the idempotent 'if needed' None. Remove the
now-dead _supersede_other_active_sessions (create's
supersede_active_sessions_for_task replaces it).
Fix three pre-existing RED tests surfaced by the sweep (all confirmed
failing on the F110 commit before this change):
- test_fail_qa_work_session_fallback_excludes_qa_session: inserted two
ACTIVE work_sessions per task, violating uq_work_sessions_one_active
_per_task (migration 047). The QA session is now ABANDONED — still in
the fallback query's result set (the query filters by task_id +
agent_id, not status), so the exclude filter (agent_id != qa_id) is
still exercised and the dev is resolved.
- test_ceo_reject_routes_coordination_task_to_main_pm /
test_ceo_reject_routes_batch_umbrella_to_main_pm: ceo_reject emits an
audit row keyed to CEO_AGENT_ID, but the tests never seeded the CEO
agent row (fk_audit_log_agent_id_agents). Seed the CEO agent (get-or-
create, mirroring test_ceo_reject_writes_handoff_journal).
* [F114] single-claimant guard on pr_gate_claim
pr_gate_claim delegated straight to _qa_or_doc_claim, which overwrites
claimed_by / active_claimant_id with no single-claimant check. Two
reviewers race-claiming the same awaiting_pr_review task would
last-write-wins overwrite the first claim, and the first reviewer's
subsequent pr_pass / pr_fail would actor-mismatch against the new owner
(wasting a review cycle). The orchestrator's gate dispatcher already
prevents double-reviewer-dispatch in normal flow (one task -> one team
-> one reviewer + is_agent_active + per-tick spawned set), so the race
is only reachable via direct concurrent API calls (defense-in-depth).
Add a role-aware single-claimant guard in pr_gate_claim: lock the row
FOR UPDATE (serialize concurrent claims, mirroring the dev claim path),
then refuse only when the task is already actively claimed by a
DIFFERENT PR-reviewer. The gate task is owned by the PM at entry
(submit_for_review does not clear ownership, unlike submit_for_qa), so
the guard must distinguish a PM/dev owner — which the first reviewer
legitimately overclaims — from a competing reviewer claim; checking the
existing claimant's role (pr_reviewer) does exactly that. A re-claim by
the same reviewer is idempotent (skipped by the != check). The gateway
claim_gate_review handler already maps a None return to a clean
invalid_state envelope ('it may already be claimed; give_me_work for
the next'), so no gateway change is needed.
TDD: 3 integration tests in test_task_service_basics.py — reject a second
reviewer race-claim (returns None, first claim intact), allow the first
reviewer when the PM owns the root (regression guard for the
PM-owns-at-entry model), idempotent re-claim by the same reviewer.
Confirmed the reject test RED first (race-claim succeeded, overwriting
reviewer1).
* [F115] sample monorepo per (repo,workflow)/(repo,command) not per repo
The CI-watch and dep-update loaders collapsed a monorepo's cell-projects
to one canonical entry per repo (slug-sorted-first), so a repo whose cells
each carry their OWN ci_watch_workflow / dep_update_command had only the
canonical cell's workflow/command sampled — a red on another cell's
workflow or drift on another cell's lockfile was missed (under-count).
Refactor the shared one-per-repo collapse into _projects_one_per_key, keyed
by repo identity for external-PR discovery (unchanged: one review per PR per
repo), by (repo, effective workflow) for CI-watch, and by (repo, command)
for dep-update. Each distinct workflow/command is now sampled once; the
engines' per-git_url fix-task dedup still prevents duplicate fix tasks for
the same repo. _projects_one_per_repo now delegates to _projects_one_per_key.
key_fn uses a string annotation (Callable lives under TYPE_CHECKING, like
the existing Coroutine/Iterable annotations at lines 4193/5279).
* [R115] originate ci_watch/dep_update fix tasks as PLANNING coordination roots
The Main-PM-code-impossibility guard (commit e202ce39, Thread 4 of this
audit) made team=MAIN_PM + task_type=CODE impossible — a Main PM coordinates,
it does not write code. But the ci_watch and dep_update engines still
originated their fix tasks as task_type=TaskType.CODE assigned to main-pm,
so task_svc.create raised MAIN_PM_NO_CODE and NO fix task was ever opened
— a regression introduced by the earlier audit fix (confirmed: the engine
tests pass at e202ce39~1 and fail at HEAD).
Mirror the hardened self_heal_engine precedent (self_heal_engine.py:197)
which already uses task_type=TaskType.PLANNING for its Main-PM coordination
root with an explicit 'decompose the fix and delegate the code work to a
cell dev — the Main PM does not write the fix itself' description. Both
engines now originate PLANNING coordination roots with matching delegation
guidance in the description + acceptance criteria. confirmed_by_human
stays True for both (they ride the normal delivery flow without the CEO
gate, unlike self-heal — intentional per the architecture).
The dedupe/open-cap queries (list_open_ci_watch_tasks /
list_open_dep_update_tasks) key on source + non-terminal status + git_url,
NOT task_type, so the type change does not break dedup (still one open fix
task per repo).
The two source-test fixtures (test_ci_watch_source / test_dep_update_source)
created CODE+MAIN_PM tasks directly to exercise the listing queries — same
guard violation; switched to PLANNING (the queries assert on source/status,
not task_type, so the fixture type matches the engines' corrected type).
* [F116] hold the read-clone lock across the dep-probe local clone
dry_upgrade_changes_lockfile called ensure_read_clone (which syncs the
read clone under the _meta-conventions lock then releases it) and ran
'git clone --local --no-hardlinks <read_clone>' OUTSIDE the lock. A
concurrent ensure_read_clone -> _sync_read_clone (fetch + hard-reset to
origin's default branch) could mutate the read clone's working tree /
object db mid-clone, racing the clone and producing an inconsistent or
failing probe.
Split _probe_lockfile_change into _clone_local_into (the local clone,
run under the read-clone lock) + _probe_lockfile_on_clone (the upgrade +
git status, run without the lock on the now-independent copy). The probe
acquires _ensure_lock_for(slug, '_meta-conventions') — the same lock
ensure_read_clone syncs under — and holds it only for the clone step; the
upgrade operates on the full --no-hardlinks copy and never touches the
read clone, so the lock is released before it to avoid blocking
conventions reads for the upgrade duration.
The tiny gap between ensure_read_clone releasing the lock and the probe
re-acquiring it is safe: any concurrent _sync_read_clone completes under
the lock before the probe acquires, so the clone reads a stable state.
* [F117] stop the orchestrator in lifespan shutdown BEFORE closing the DB
The lifespan shutdown closed OptimalService + the DB, and only THEN did
bootstrap's finally block call orchestrator.stop() — so stop() ran with
the DB already closed. stop() drains fire-and-forget _bg_tasks writes
(respawn_tracker upserts, audit-log rows) and stop_agent finalizes work
sessions / agent state, all needing the DB still open; closing it first
silently dropped those final writes (the durable PM-respawn counter's
last few strikes, the metrics-bearing audit trail tail).
Move orchestrator.stop() into the lifespan shutdown path, BEFORE
close_optimal_service + close_db, guarded by a new get_orchestrator_or_none()
safe accessor (no crash when no orchestrator is wired — tests,
skip_orchestrator). bootstrap's finally-block stop() becomes an idempotent
safety net: stop() gains a _stopped flag (getattr-guarded so __new__-
constructed test instances still stop) so the double-call is a clean no-op,
not a re-stop of already-stopped agents / re-drain of an empty bg set.
* [F118] coerce a lone-string where_to_look into a list
where_to_look is a list-typed handoff field like consequences/next_steps
but was the only one NOT in the _wrap_scalar_in_list field_validator. A
well-intentioned where_to_look='src/api/' 422'd at the route with no
remediation envelope, and the agent's retry loop tripped the do-server
circuit breaker — the exact failure mode the other list fields were
hardened against. Add it to the mode='before' validator so a lone string
is wrapped into a one-element list before type coercion.
* [F119] sender reaps dead sockets on send error instead of waiting for receive idle timeout
* [F120] release a stopped agent's claimed task immediately on budget-kill/shutdown
* [F122] name the already-open PR in submit_up's None-state remediate
submit_up's create_pr pre-side-effect opens the cell→root PR BEFORE
submit_for_review runs (its pr_created gate requires it — lifecycle.py:1338-1343).
When submit_for_review returns None (a concurrent state change raced the task
out of in_progress between the precondition gate and the composed action), the
old remediate ('check task state — must be in_progress with PR ready') hid
that the PR was already open on GitHub — an orphaned external artifact the PM
could not reconcile. Mirror submit_root's F016 None-envelope remediate: name
the open PR, point the PM at re-fetch + reconcile, and note create_pr is
idempotent so a re-issue re-attaches to the existing PR (no duplicate). Pure
message improvement — zero behavior change; reordering is off the table
(create_pr must precede the pr_created gate).
* [F124] re-check dependency state before releasing a dependency-blocked claim
The unmet_dependency guard read dependency state via an unlocked SELECT, then
fired release_dependency_blocked_claim (a state mutation: claimed/in_progress
-> pending, clears branch_name, abandons WorkSession) as a side-effect BEFORE
returning the rejection. An upstream dependency that reached a terminal state
(completed/cancelled) in the microseconds between the read and the release left
the task NEEDLESSLY released — its branch cleared + WorkSession abandoned +
assignee bounced, only to be re-dispatched + re-claimed when the dependency-
completion re-dispatch fired a moment later.
Re-check unmet_dependency_ids immediately before the release and skip it
(returning None — proceed) when the upstream just completed. Dependencies are
monotonic (unmet -> met only; terminal states never reopen), so a fresh read
that now finds them met stays met: safe to proceed without releasing. The
'still unmet' path is byte-for-byte the prior behavior (no regression). The
cross-task residual window (upstream completes between the re-check and the
release) is not closable by a row lock on the dependent, but the re-check
narrows the window from [first read -> release] to [re-check -> release], and
in the common case the first read already sees met (no guard fires). No
committed-work loss either way (a dependency-blocked task has none; the branch
ref + commits persist across the branch_name clear).
* [F125] serialize same-parent delegate via per-parent advisory lock
The delegate sibling-dedup guard read the parent's existing subtasks via an
unlocked get_subtasks SELECT (the dedup read) then created the subtask (the
write) with no DB serialization between them. Two concurrent delegate calls
for the same parent (PM re-delegating while a reaper re-dispatches, or two
orchestrator ticks racing) each read a duplicate-free sibling set, each passed
the dedup guard, and each created a subtask — the parent got the duplicate the
guard exists to prevent (the smoke-run runaway pattern).
Fix: a PostgreSQL transaction-scoped advisory lock keyed by the parent task
id (seed 1, disjoint from the per-agent claim lock's seed 0), acquired at the
top of the delegate body before the first get_subtasks read (the briefing
context read AND the dedup sibling read) and held through create_subtask's
flush + the outer request commit. The second concurrent same-parent delegate
blocks until the first commits, then its dedup read sees the committed
sibling and is rejected.
Per-PARENT (not per-agent): a coordinator PM legitimately delegates many
subtasks under one parent in quick succession and plans many roots in
parallel — a per-agent lock would serialize all of a PM's delegates and
regress the PM coordinator concurrency feature. The per-parent lock
serializes only same-parent delegates (the dedup invariant is per-parent)
and leaves different parents untouched.
TDD: red-first ordering test (lock acquired before first get_subtasks read
and before create_subtask) + no-regression test (create still runs).
* [F127] per-task advisory lock prevents open_pr milestone double-emit
open_pr's idempotent re-entry guard (pr_number is not None) read t.pr_number
from an unlocked fetch. Two concurrent same-task open_pr calls (the
alive-but-unresponsive respawn race) both fetched pr_number=None, both passed
the guard, both ran the runner (GitHub 422 ensures one PR), and both reached
_record_milestone_progress -> a double-emitted 70% 'opened PR #N' entry.
Fix: acquire_task_lock (pg_advisory_xact_lock, seed 2) before the fetch, held
through the runner + milestone + request commit. The second concurrent call
blocks until the first commits, then its fetch sees the committed pr_number
and the idempotent guard short-circuits without re-emitting. Per-task (single-
active-task guard means same-task concurrent open_pr is only the bug case).
* [F128] require active claim on explicit-task content posts
_verify_explicit_task_ownership checked assigned_to, which is stale
across a reap/handoff (persists until reassignment; active_claimant_id is
cleared on release). A reaped agent could keep posting say/dm/note to its
former task. Add the active-claimant check when assigned_to == caller;
assigned_to=None keep its existing allow (read-side inspection between
reassignments uses evidence, which has its own ownership path).
Existing 'active owner' test mocks passed assigned_to=agent_id without
active_claimant_id; production sets both together on claim, so the mocks
were incomplete. Updated to set both — realistic, not a behavior change.
* [F129,F130] harden quality gate _run_one exit status + timeout cleanup
F129: _run_one returned 'proc.returncode or 0', masking a None returncode
(communicate returned without a recorded exit code — process killed
out-of-band) as 0 / success. Treat None as a non-zero failure (fail-closed).
F130: on timeout, _run_one killed the subprocess but never awaited wait()
— communicate() was cancelled so it never closed the stdout/stderr pipes,
leaving a transient zombie + leaked FDs. Await wait() after kill() to reap
the process and close the transports.
* [F132] timeout the conventions validator + reap on hang
_run_conventions_validator awaited proc.communicate() with no timeout —
a hung subprocess (tree-sitter deadlock, huge repo) hung the
i_am_done/pr_pass gate forever and orphaned the python subprocess on
orchestrator restart. Wrap communicate() in wait_for(120s); on timeout
kill+wait the proc and fail closed (could_not_run=True → block gate
refuses the submit), matching the validator's own fail-loud philosophy.
* [F135] re-check activity before sweeper closes a session (TOCTOU)
sweep_timed_out_sessions read last_activity_at once at the candidate
SELECT, then closed. A message landing in that window refreshed
last_activity_at in the DB, but the sweeper closed on its stale in-memory
value — closing a just-used session. Re-read last_activity_at fresh right
before the close and skip if the session is no longer timed out.
* [F136] cancel startup indexing task on OptimalService.close()
close() cancelled only the periodic update task, then cleared the plugins.
The startup _indexing_task (background auto-index, slow Ollama / large repo)
could still be mid-flight at shutdown and write against closed/cleared
plugins. Cancel and await _indexing_task FIRST (its tail starts the periodic
task, so ordering also prevents a late periodic spawn), then the periodic
task, then clear plugins.
* [F139] scope active_task_owns_branch to the polled project
active_task_owns_branch did an unscoped WHERE branch_name = ? — a cross-project
branch_name collision (UUID-derived 8-char prefixes, theoretical) made the
internal-PR reviewer skip the WRONG project's PR (project A's leftover PR
skipped because project B happened to have an active task with the same
branch). Pass project_id (in scope at the orchestrator call site) and add
TaskTable.project_id == project_id to the WHERE. Correct for single-project
tasks and MegaTask multi-repo batches alike: each root-subtask carries its own
project_id matching its own repo, so a branch on project A's repo is owned
only by a task whose project_id == A.
* [sweep] strip Fxxx audit-ID tokens + trim bloated comments/docstrings + add behavior-change docs
Post-audit sweep over the 135 audit-fix commits since 19a474d3:
1. Stripped every # Fxxx: audit-ID token from comments AND every Fxxx token
from docstring openings across 211 blocks / ~626 lines. The CEO flagged
these twice: audit-issue IDs in code confuse future devs/agents. The
descriptive text is preserved; only the Fxxx token is removed (and bloated
narrative blocks trimmed to 1-3 lines keeping the one non-obvious invariant).
2. Trimmed bloated comments/docstrings to the concise standard (1-3 lines).
3. Added missing behavior-change docs for the audit-fix batch: prompts/roles
(documenter, pr_reviewer, qa), user-facing docs (api auth, websockets,
agent-gateway, megatask, merge-model, task-lifecycle, grok, resilience,
conventions, panel, security, troubleshooting), and the RAG corpus (cell-pm,
main-pm, pr-reviewer, qa roles; conventions; messaging-tools; escalation;
megatask; task-claiming workflows).
Comment/docstring/prose ONLY — zero code-line edits (verified: the diff
contains no def/class/return/if/for/await/assignment/call lines). Gates green:
ruff format + ruff check clean, mypy clean on roboco/. The only pytest failures
are the pre-existing sync_branch tracing-decision gap (B1, 250be5c2) — not
sweep-caused and tracked separately.
* [fix] register sync_branch in VERBS_WITHOUT_TRACING
sync_branch (B1, 250be5c2) is a git-only rebase+force-push verb (composes=(),
no DB transition, side_effects=()) but was never registered in the tracing
parity tables, so test_every_intent_verb_has_a_tracing_decision failed.
Mirrors open_pr: a mechanical git op with inline preconditions (ownership),
no journal/plan rationale required.
* chore(release): 0.14.0
* [fix] resolve 16 mypy errors across 9 test files (make quality gate)
type-clean the test files so make quality (mypy roboco/ tests/) is green:
- Any-typed locals for the two TypeError-asserting scoping tests (bypass
the required-arg check without getattr/ruff B009)
- Any-typed view for the shutdown-drain _drain_bg_tasks override (bypass
mypy method-assign without setattr/ruff B010)
- cast("uuid.UUID", ...) / cast("UUID", ...) for SQLAlchemy UUID[Any]
returns (TC006-quoted), config=None for AgentInstance stubs, None-narrowed
await_args, Iterator return on a yielding fixture, UUID annotation on the
_task helper. No type:ignore / noqa.
* [docs] regenerate lifecycle artifacts for sync_branch + branch-keyed submit_root gate
The committed artifacts were stale: lifecycle.py grew the sync_branch verb
(B1) and the branch-keyed submit_root gate description (B2/B3) but the
generated markdown/json were never regenerated. make foundation-check
enforces artifact==generator(lifecycle.py); regenerating restores that.
No source change — pure generator output.
* [refactor] reduce xenon C-rank blocks to A (behavior-preserving)
Extract helpers / flatten conditionals in 11 blocks that rated C(11)+
under xenon --max-absolute B, dropping pr_gate.py module rank B->A in
the process. Pure move-and-call refactors: each extracted helper holds
the original logic verbatim and the caller delegates to it; no control
flow, return values, or side effects changed.
Sites: validators._extract_strs, sequencing.dev_task_collision_edges,
evidence_builder.build_task_handoff, intake_driver._coerce_draft,
task.claim_task_for_agent (2 guards), prompter.create_task_from_draft
(validate+assignee), pr_gate._gate_decision (3 helpers),
orchestrator._handle_stopped_container + _reap_with_service,
_impl._create_subtask_from_inputs + complete.
_impl helper returns tuple[TaskNature, list[str]] to preserve mypy
narrowing of acceptance_criteria at the TaskCreateRequest site.
Also fix vulture: rename unused __aexit__ param tb->_tb in
test_conventions_cache_put.py (was hidden while xenon short-circuited
the gate).
* [security] bash-guard uv run --active deny + CodeQL path-traversal fixes
Fix 1 (be-dev-1 brick prevention): bash-guard now denies 'uv run --active'
and 'uv run'/'uvx' against /app targets. In the agent container
VIRTUAL_ENV=/app/.venv is baked globally, so 'uv run --active' always
resolves onto the image-baked MCP-gateway venv and uv rebuilds it,
deleting /app/.venv/bin and bricking every MCP server spawn. Bare
'uv run' (workspace .venv, cwd-relative) is untouched.
CodeQL fixes:
- docs.py: replace bypassable '..' substring guard with a
resolve-and-contain helper (_resolve_contained_path). An absolute
path made pathlib reset (base / '/etc/passwd' == '/etc/passwd'),
letting read_doc/delete_doc reach arbitrary files. Applied to both
sinks.
- orchestrator.py: _safe_agent_path_segment at the spawn_agent
chokepoint (rejects traversal-shaped agent_id before any fs op) and
inside _remove_container (slug guard before the log-dir mkdir,
defense-in-depth).
- agent_sdk/server.py: /usage/sync transcript_path now resolved and
contained under ROBOCO_TRANSCRIPT_DIR with a .jsonl suffix requirement
(was Path(raw) — unauthenticated endpoint could stat arbitrary files).
TDD RED->GREEN across all four; make quality green (4890 passed).
* [fix] enum-parity gate: drop false-green mask, skip empty/unmigrated DB
The foundation-check gate ran the enum verifier behind
`|| echo "(skipped — postgres unreachable)"`, which swallows ANY
non-zero exit — including real drift — and prints 'All quality gates
passed'. On a host with a dockerized but empty/unmigrated `roboco` DB
(0 tables: the agentrole/team enum types don't exist), the verifier
connected, found every foundation value 'missing', exited 1, and the
mask relabeled it 'skipped' → false-green.
Fix:
- scripts/verify_postgres_enums.py: move skip semantics INTO the script.
Distinguish unreachable (skip, exit 0), DB-not-migrated/both-enum-types-
absent (skip, exit 0), real drift (exit 1), match (exit 0). Extract
pure enum_drift + should_skip_for_unmigrated helpers + a type_exists
probe so an empty DB is 'no migrated target', not drift.
- Makefile: drop the `|| echo` mask — real drift now fails the gate.
TDD RED->GREEN (10 tests); make quality green (10906 passed).
* [security] docs path guard: reject '.'/empty segments for clean 400
_resolve_contained_path used an '..' substring ban, which (a) left rel='.'
passing the guard — read_doc/delete_doc then got the base DIRECTORY itself
and raised IsADirectoryError (500) instead of a clean ValidationError, and
(b) false-rejected legit filenames containing '..' like 'v1..v2.md'.
Replace the substring ban with a raw-segment check (rel.split('/')) that
rejects any '.', '..', or empty segment. Path(rel).parts was the wrong tool
— pathlib collapses '.' and empty segments on 3.13, hiding them. The split
check catches '.' / 'a/./b' / 'a//b' / '..' / 'a/../b' while allowing
'v1..v2.md' ('..' inside a filename, no bad segment). The post-resolve
parents-containment check (the real defense) is unchanged.
TDD RED->GREEN (4 new tests); make quality green (10910 passed).
Follow-up to the CodeQL path-traversal review: the two CodeQL 'High' alerts
on this guard are false-positives-on-the-fix (resolve-and-contain already
contains the bypass); this hardening closes the one genuine low residual
(rel='.' -> 500) the review surfaced, which CodeQL did not flag.
* [F123] per-task git worktrees — coordinator PM roots no longer clobber each other
A coordinator PM (Main/Cell) legitimately holds several in_progress roots at
once, but its clone is one checkout, so every fresh claim ran `git reset --hard`
+ `checkout -b` to the new branch and destroyed uncommitted tracked changes on
the still-active first root (a live run showed main-pm ping-ponging two roots
on one clone for ~13h). The reset's own comment assumed it discarded "abandoned
cruft from a finished task" — neither root was finished — and the mutation was
non-transactional with the DB claim (rollback restored DB fields, not the tree).
Each task now gets its own working tree via `git worktree add` under
`{clone_root}/.worktrees/{task-short}/` on the same clone; the F123 reset
dissolves (a fresh worktree is clean by construction). The shared clone keeps
the real `.git` store, the per-project `.venv`, and `.uv-python`; each worktree
gets a `.venv -> ../../.venv` symlink so uv resolves the shared venv (no
per-worktree re-sync), and `.uv-python/` is now gitignored. Branch-by-name ops
(push/pull/fetch/pr_merge/diff) run from the clone root; checkout/HEAD-moving
ops (create_branch/commit/rebase/checkout) target the worktree. Spawn resolves
the worktree from current_task_id each spawn and `-w`'s it; resume re-attaches a
pruned worktree before launch; claim-rollback force-removes on mid-claim failure;
terminal cancel removes the worktree (the stale-claim reaper does not — it routes
to pending for a re-claim that reuses it).
Routing-gap followup (the switch missed two checkout-dependent ops that still
resolved the clone root — both deploy-blockers):
- rebase_onto_base does `checkout <head>` + `reset --hard origin/<head>` cwd=
workspace; post-worktree the branch is checked out in the linked worktree, so a
checkout in the clone root is refused ("already checked out"). sync_task_branch
+ rebase_pr_for_task now resolve the worktree via _worktree_for_task and rebase
there (checkout becomes a no-op on the already-checked-out branch). The
destructive reset --hard is pre-existing semantics, preserved.
- conventions_check_for_task ran the validator with --root <clone root>; it reads
(root/rel).read_bytes() → default-branch content, not the dev's worktree changes
(newly-added files false-pass, conventions block gate silently disabled). Now
resolves the worktree and runs the validator there, so i_am_done/pr_pass gate
against the real diff.
Invariants untouched: only the CEO merges master (no merge/release path touched);
/app/.venv (image-baked MCP-gateway venv) stays sacred; coordinator-PM concurrency
exemption unchanged — only the workspace resolution underneath became per-task.
9085 pass / 0 fail (2272 skipped: local Docker down → test PG unavailable; 95%
with-PG on NAS). ruff/mypy(xenon/radon/vulture/bandit/pip-audit/deptry/alembic/
import-linter/foundation-check) green.
* [docs] CHANGELOG: F123 per-task worktrees + routing-gap followup
* [F-fix] preserve respawn strike count across a restart status mismatch
restore_respawn_tracker re-stamped last_check to the restore time (F034) but
left last_status at the PRE-RESTART persisted value. On the first post-restart
spawn, _pm_respawn_should_gate compares record["last_status"] to the live
status; a mismatch (a reaper/external transition in the restart gap, not
evidence the wedge cleared) reset count to 1 and disarmed the breaker — re-burning
the whole strike threshold against a still-wedged task, exactly the re-burn the
respawn_tracker table (migration 051) was built to prevent.
Re-stamp last_status to the LIVE status at restore (mirrors the last_check
re-stamp). The live progress heuristic in _pm_respawn_should_gate (status
advanced between spawns => reset) is untouched; only the restore path changes.
Conservative: if the wedge genuinely cleared during the gap, the breaker may
false-trip on the first spawn — the safe direction (pause + alert the CEO once,
never destroy state) rather than silently re-burning the budget.
* [F-fix] gate the worktree .venv symlink on the clone-root venv existing
_link_shared_venv created worktree/.venv -> ../../.venv unconditionally, so
when the clone-root venv was not yet provisioned (the near-zero gap before
install_dev_deps completes) the symlink dangled — uv then errored or re-synced
a worktree-local venv that the lexists guard could not later replace, silently
breaking the shared-venv optimization for that worktree.
Only create the symlink once clone_root/.venv exists; a later ensure (the
resume/commit path re-runs ensure_workspace -> install_dev_deps) self-heals
the link. No destructive replacement of an existing real dir at worktree/.venv
(recovery of an already-clobbered worktree venv is out of scope).
* [F-fix] ensure_worktree_before_spawn: release claim + abort on fatal ensure failure
A bare except swallowed every failure, so a pruned worktree whose branch
ref was also gone launched the container at a missing -w path. A fatal
WorkspaceError (branch ref gone, worktree unrecoverable) now releases
the claim so the next claim rebuilds via create_branch and aborts the
spawn; a transient failure aborts without releasing (next tick retries).
The swallow also masked the happy-path tests' non-awaitable mock.
* [F-fix] complete/ceo_approve: remove per-task worktree on terminal completion
Only cancel() + create_branch rollback removed per-task worktrees, so
completed/merged tasks leaked their worktree on disk until the whole
agent/project was deleted. Both terminal->completed paths (cell-PM
complete after leaf PR merge; CEO ceo_approve after root->master merge)
now remove the assignee worktree best-effort. Terminal-only (a dev task
bounces needs_revision and needs its worktree back); no-op for
branchless; best-effort so a removal failure never blocks completion.
Reaper reuse-on-reclaim unchanged.
* [F-fix] worktree-cleanup tests: preset execute on local session mock
Assigning svc.session.execute tripped mypy method-assign (session is typed
as a real AsyncSession). Build the session as a local MagicMock, preset
execute on it, then hand to TaskService — the proven _service_with idiom.
* [feature] loop-prone notification re-fire guard (sweep #4)
TASK_ASSIGNMENT/REVIEW_REQUEST/DOCUMENTATION_REQUEST/BROADCAST re-fired by a
coordinator PM every tick while a task sits in a state, flooding inboxes. The
DB purpose-dedup never fires for these (ACK_REQUIRED_BY_TYPE marks them
requires_ack=False) and _persist_and_deliver had no dedup at all.
New roboco/services/notification_dedup.all_recipients_recently_notified: a 60s
Redis SET NX window per (type, sender, recipient, task). First fire acquires
keys for fresh recipients; a later fire in the window is suppressed when no
recipient was fresh, converging the storm. Fail-open (Redis down -> never
suppress). One-shot types bypass entirely.
Wired into both creation chokepoints: NotificationService._create_notification
(after recipients resolved, before the DB dedup block) and
NotificationDeliveryService._persist_and_deliver (before session.add). TDD:
helper + both chokepoints, 15 new tests; SA UUID column type-leak peeled with
cast at the delivery path.
* [chore] sequencing: chain undeclared-surface same-assignee dev siblings + trim re-fire guard comments
wire_sibling_collision_dag only wired collision edges for dev tasks that
declared a surface; a PM delegating two dev tasks to the same developer
without declaring surfaces wired no edge, so the later task could start
while the earlier one's PR was unmerged (the out-of-order merge wedge).
dev_task_collision_edges now falls back (only with zero declared-surface
edges) to chaining each same-(project, assignee) lane by (priority,
sequence). Same-assignee scoped so cross-dev parallel work is untouched;
the edge lives in dependency_ids so it survives reassignment. Idempotent
+ incremental (stable sort, add_dependency dedupes).
TDD: 8 unit fallback tests + 1 integration test (RED-verified by neutering
the fallback). Also trims the re-fire-guard comments (no internal refs,
terse) per the comment standard.
* [chore] give_me_work/claim: enforce per-dev lane barrier (out-of-order start)
The lane order check (has_earlier_incomplete_code_sibling: a code leaf may
not start while an earlier same-assignee sibling is still open) lived only
on the orchestrator spawn path and i_am_idle, so give_me_work's pre-assigned
path and a direct i_will_work_on claim bypassed it. A dev could start a
later code leaf before the earlier one's PR merged, cutting a branch from a
base that predates the sibling's unmerged changes (the merge wedge).
give_me_work now filters pre_assigned through _pending_not_lane_held (a
lane-held leaf is dropped, not offered). _run_claim_guards refuses a direct
claim of a lane-held code task (invalid_state, parked back to pending via
release_dependency_blocked_claim) in a new _lane_claim_guard helper. The
predicate is CODE-only so coordinator PMs are inert; the claim guard is
fail-closed on a lookup error. is not True keeps both paths inert under
partial test mocks.
TDD: 7 new tests (RED-verified) + 870 gateway/lane-queue regression green.
* [chore] tests: drop a reintroduced type:ignore + fix two mypy annotations
test_notification_delivery_refire.py reassigned svc.deliver with a
type: ignore[method-assign], reintroducing a test suppression the project
purged (f8262856). Use the cc: Any alias pattern instead. Also fix
uuid4-as-a-type -> UUID in the lane barrier helper, and cast the SA
UUID column leak in the sequencing integration test. mypy roboco/ tests/
clean on all touched files.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* Updated uv.lock
* Bunch of fixes we need to verify first..
* feat(megatask): per-cell project map root-subtasks (multi-project, multi-cell)
A MegaTask root-subtask can now target an ad-hoc per-cell project map — a
third targeting shape that mirrors the existing product fan-out root. In
RoboCo a project is per-cell (ProjectTable.assigned_cell); a monorepo is N
per-cell projects sharing one git_url. So 'multi-cell' IS 'multi-project',
and a task may mix per-cell projects across products or include OSS-library
projects not in any product.
Storage: migration 052 adds task_cell_projects (mirrors product_projects;
unique per (task, team)). TaskTable gains a cascade-delete cell_projects
relationship; TaskCreateRequest / TaskCreate / Task response carry the map.
Policy: batch.is_branchless_coordination + is_valid_batch_shape gain a
has_cell_projects param — a root-subtask targets exactly one of project /
product / cell-map; the umbrella still targets none. TaskService passes
has_cell_projects at every predicate call site and persists the rows in
create(). _ensure_branch_for_task cuts feature/main_pm/{root} per distinct
project in the map (via _distinct_projects_for_task); _require_target_or_umbrella
and _validate_batch_membership accept the map shape.
Fan-out: every distinct_project_ids site (task.py branch creation, routes
_project_for_complete + _resolve_project_for_merge, orchestrator
_ambient_projects_for_task, pr_review._project_slug_for, git._project_for_task)
generalizes to first-distinct-project-of-map-or-product. Choreographer
_resolve_subtask_project resolves a delegated subtask's cell from the parent's
cell map. The product-scoped _slugs_for_product intake helper is unchanged.
Intake: prompter._draft_cell_map extracts the per-cell map from the_work[].
_validate_batch_scope counts distinct projects across all drafts' cells
(>=2 min stays; one 2-cell draft satisfies it). create_task_from_draft
persists cell_projects for >=2-cell drafts (project_id/product_id None),
collapses a 1-cell map to the single-project shape, and leaves single-cell
top-level project_id drafts unchanged. _resolve_owning_team routes a
multi-cell map to Main PM (coordination root, like a product root — a cell
PM can't delegate cross-cell). propose_draft/propose_batch tool descriptions
declare the per-cell project_id (both Claude SDK + grok runtimes).
The umbrella stays branchless / pure-coordination / submit_root-rejected;
the CEO-escalation pr_number gate is not widened (the map root is
is_umbrella=False, mirroring a product root, so submit_root supplies it).
Single-cell root-subtasks and everything below them are byte-for-byte
unchanged. Un-run MegaTask waves (multi-cell drafts) become runnable.
* [feature] Panel per-cell project picker + pnpm format infra
MegaTask root-subtasks can fan out across cells (be+fe, fe+uxui). Since a
RoboCo project is per-cell (ProjectTable.assigned_cell), a monorepo is N
per-cell projects sharing one git_url — so multi-cell IS multi-project. The
batch-review card now shows one project Select per the_work entry, scoped to
that cell's repos, instead of one Select bound to a single top-level
project_id. confirmBatch validates each cell's project is in scope and the
batch still spans >=2 distinct projects.
- prompter.ts: CellWork gains optional project_id (the per-cell picker seam).
- batch-review-card.tsx: per-cell Selects (one per the_work entry), scoped to
the cell's projects; legacy single-cell drafts keep the one-Select path.
- use-prompter.ts: updateBatchDraftProject edits per-cell (entryIndex); confirmBatch validates every cell; batchFromEvent parses per-cell map.
Also adds the missing pnpm format infrastructure (the panel had no formatter
at all): prettier devDep + .prettierrc.json (default-style config: 80-col,
double-quote, semi, trailing-comma-all) + .prettierignore, plus format /
format:check scripts. Only the 3 changed files above were reformatted; the
~222 pre-existing non-compliant files are left untouched (a wholesale reformat
is a separate explicit decision, not bundled into this feature).
* [fix] MegaTask verification: migration 052 enum + async cell-map read
Two real bugs surfaced running the full gate against a containerized
Postgres (and the orchestrator boot log):
1. Migration 052 crashed a real orchestrator boot with
'type "team" already exists'. The generic sa.Enum(create_type=False)
does NOT set the postgres enum's create_type attribute, so op.create_table
(checkfirst=False) emitted a redundant CREATE TYPE against the pre-existing
team enum. Switched to postgresql.ENUM(create_type=False) — the postgres-
native enum whose create_type _check_for_name_in_memos actually reads, so
the CREATE TYPE is suppressed. Verified: 051->052 upgrade against a DB where
the team enum pre-existed (the exact path that crashed) now succeeds;
downgrade 052->051 drops the table and preserves the shared enum; fresh
upgrade head clean. (Migration 016 has the same latent sa.Enum pattern but
never re-runs in prod, so it's noted, not touched here.)
2. _ensure_branch_for_task read task.cell_projects (lazy=selectin to-many)
directly, tripping MissingGreenlet on a freshly-created/unqueried task —
which then poisoned the async session (PendingRollbackError). Replaced with
_task_has_cell_map: peeks InstanceState.unloaded (no IO) and reads the
already-loaded map, falling back to an awaited count query only when the
relationship is genuinely unloaded. Non-ORM stubs route to the plain
attribute. Fixes 2 integration tests; the 6 cell-map unit tests still pass.
Also: typed the self stub as Any in test_choreographer_subtask_project
(mypy tests/ wants Choreographer, not SimpleNamespace) — the codebase idiom.
Gate: ruff format/check clean; mypy roboco/ + tests/ clean; full pytest
10371 passed / 388 skipped against containerized pgvector:pg16; vulture clean.
Pre-existing xenon C-rank on reassign (from prior commit 19a474d3, not this
feature) still blocks make quality — surfaced separately.
* [refactor] Extract reassign board-advisory diversion helper (C→B complexity)
`reassign` in roboco/services/task.py hit xenon absolute complexity 11 (a
C-rank block), failing `make quality`'s --max-absolute B gate. The C-rank
originated in 19a474d3 (pre-existing, not this feature branch's work).
Extract the board/advisory → cell-task diversion into
`_maybe_divert_board_advisory_reassign` (complexity 4, A). reassign drops to
9 (B); behavior is byte-for-byte preserved — the helper runs the same
guard + pool diversion + log, returning the diverted task or None so the
caller falls through to the normal handoff. Whole-repo xenon exits 0; the 159
reassign / board-guard tests pass.
Unblocks `make quality` on feature/metrics-granularity.
* [fix] migration 016: postgresql.ENUM(create_type=False) for reused team enum
016_add_products_and_task_product_id used `sa.Enum(..., create_type=False)`
for the reused Postgres "team" enum — the same latent defect that crashed
052 on a real orchestrator boot. On the generic `sa.Enum` the
`create_type` kwarg is silently dropped, so `_check_for_name_in_memos`
never sees it and `op.create_table` (checkfirst=False) emits a redundant
`CREATE TYPE team` that fails with "type 'team' already exists" against a
DB where the enum pre-exists.
Switch to the postgres-native `postgresql.ENUM(..., create_type=False)` —
its `create_type` is a real attribute the guard reads, so the CREATE TYPE
is suppressed (and DROP TYPE on downgrade too). The member list is inert
under create_type=False (it never creates/alters the type), so it stays at
016's original six, reflecting the enum as it stood then, not the
later-widened set.
This never crashed in prod because 016 is never re-run (alembic_version is
past it), but it's the same defect class. Verified on the real boot path:
upgrade to 015 in process A (team enum created by 001), then `upgrade head`
in a fresh process B — 016 applied clean, no DuplicateObjectError; downgrade
016->015 clean, shared team enum preserved.
See project_migration_enum_create_type_gotcha.
* [chore] panel: prettier reformat across the codebase
Apply `pnpm format` (prettier 3.8.5, 80-col / double-quote / semi /
trailing-comma-all) to the 223 pre-existing panel files that predated the
prettier infra added in cb5365a4. Pure formatting — no semantic changes:
multi-line arrays/objects collapsed where they fit, trailing newlines added
(.prettierrc.json), import grouping unchanged.
Verified: `pnpm format:check` clean, `pnpm lint` clean, `pnpm typecheck`
clean, `pnpm test` 113/113 pass (7 files).
* Bunch of runtime fixes for MegaTask and other issues
* Fix different project same PR number collision problem
Fix (two layers):
1. Root cause — pr_merge and rebase_pr_for_task now take a required project_id and scope the lookup where(pr_number == X AND project_id == Y). Required so no caller can forget — the bug class can't recur. All 4 call sites updated (choreographer cell_pm_complete, the rebase-retry, the superseded close_pull_request now passes project_id, and _verb_runner._do_pr_merge).
2. Crash guard — _finalize_cell_complete None-checks the complete() return and returns a clean invalid_state envelope (with a remediate hint) instead of dereffing None → 500 → respawn loop.
* Fix: Make main_pm + task_type=code impossible
* Fix Main PM needs revision can't re delegate
* [chore] Bump local LLM glm-5→glm-5.2 + swap Ollama fleet defaults off minimax
- llm_catalog: OLLAMA_DEFAULT_MODEL minimax-m3:cloud → kimi-k2.7-code:cloud;
role defaults kimi-k2.6→kimi-k2.7-code, developer minimax→kimi, product_owner/
ceo kimi→glm-5.2, documenter glm→kimi; GLM 5.1→5.2 comment fix.
- config + .env.example + docker-compose{.yml,.yaml,.registry.yml} + docs +
memory_distiller + optimal_brain: glm-5:cloud → glm-5.2:cloud.
- panel ai-routing-card: typed SelfHostedModel/boolean annotations; drop the
stale "Minimax M3 default" string (default is now catalog-driven).
- tests: glm-5:cloud → glm-5.2:cloud in pricing + rate-limit-retry fixtures.
* [fix] submit_root: hard unchanged-PR gate stops the pr_fail re-submit loop
The 2026-06-27 infinite pr_fail loop: a Main-PM root (PR #139) was pr_fail'd,
routed to needs_revision, and re-submitted byte-identical → awaiting_pr_review
→ pr_fail again, forever. The prior hint/a2a steer was ignored by the weak
coordinator model — hints don't stop a model that won't read them. A HARD gate
refuses the re-submit when the assembled root PR's head SHA is unchanged since
the last pr_fail (no new cell work → identical diff); a different SHA ⇒ the
branch advanced ⇒ allow. Every ambiguous case fails open (no prior fail, no
recorded SHA, no pr_number, unresolvable slug, git error, closed PR) — only the
exact-unchanged case is hard-blocked.
- content/models: PrReviewContent.head_sha (optional; JSON col → no migration).
- git: get_pr_head_sha (GitHub pulls API; None on any failure → fail-open).
- pr_gate: pr_fail captures head_sha into the verdict record; pr_pass does not.
- _impl: submit_root runs _submit_root_unchanged_pr_guard after _submit_up_guard;
_current_root_pr_head_sha resolves slug + current SHA (fail-open).
- pr_review: extract module-level resolve_task_project_slug, shared by the mixin
and the gate helper (_LegacyChoreographer reaches it via cast to the
ChoreographerHelpers typed view — it doesn't inherit the helpers mixin).
- tests: test_submit_root_unchanged_pr_guard (11 — refuse/allow/6 fail-open/3
capture-side, mypy-clean via cc:Any spy idiom, zero type:ignore) +
test_pr_gate_notifies_pm capture-path stub.
* [chore] mypy tests/: clear all 15 pre-existing type errors so make quality can go green
The branch tip had 15 mypy tests/ errors in files this bundle did not author,
which blocked CI's make quality mypy step (mypy roboco/ tests/) regardless of
the bundle's own commits. Pre-existing is still existing — fix every one:
- test_schemas_v1_flow.py (8): the StrList coercion tests intentionally pass
SDK-nested list-of-strings input ([[['...']]], {'item':{'$text':'...'}}, int,
dict). Annotate those literals as list[Any] locals so mypy accepts the
coerce-able shape; the StrList BeforeValidator still flattens to list[str] at
runtime. No type:ignore.
- test_pr_gate_records_verdict.py (3): notes_structured is dict|None; narrow
with 'assert t.notes_structured is not None' before indexing (the existing
pattern at line 90).
- test_pr_review_hand_format_guard.py (1 site, 2 errors): the _verb_runner()
spy assertion — use the cc: Any = c alias idiom so assert_not_awaited
resolves; drops the now-unused type:ignore[union-attr].
- test_pr_gate_notifies_pm.py (1): drop the unused type:ignore[method-assign]
on the a2a.send reassignment.
- test_content_models.py (1): narrow coerced with isinstance(coerced,
PrReviewContent) before reading .issues (the base _Content lacks the field).
Gates: rm -rf .mypy_cache && mypy roboco/ tests/ = Success (855 files);
ruff check + format clean; 5 affected suites = 40 passed.
* [fix] fail_qa routes needs_revision back to the dev, never the pool
A dev task in needs_revision must go back to the developer, never the
pool. The pool path let a cell PM re-claim the revision (PMs can claim
needs_revision) — the live 2026-06-27 'needs revision on a dev task sent to
the cell PM' bug.
fail_qa's original_developer marker is the fast path, but it is
unreliable in practice (live observation: never persisted), so the
unassign else-branch was the load-bearing path and it dropped the task
into the pool. Add a work-session fallback (_resolve_revision_dev) that
resolves the developer who actually worked the task — the most recent
work session whose agent is a developer, the QA's own session excluded
— and reassigns to that dev instead of unassigning. Only unassign when
no developer ever touched the task. Self-heals the marker so a
subsequent re-fail takes the fast path and the QA-review index
attributes the work correctly.
* [feature] delegate carries dev-task collision surface (sequencing S1)
The cell/main PM's delegate verb now carries the dev-task collision
surface (intends_to_touch / adds_migration / touches_shared) and an
explicit depends_on override through DelegateRequest -> DelegateInputs
-> _create_subtask_from_inputs -> create_subtask, and create_subtask
forwards sequence / dependency_ids / batch_id / surfaces into the
prepared TaskCreateRequest instead of dropping them (the base create
already persists them at task.py:878-884).
This is the plumbing for the multi-level sequencing model edge kind 3
(dev-task collision DAG). Previously a dev task delegated with a
collision surface or an explicit dependency lost it before persistence
— dependency_ids was always [], so the only dev-task ordering was the
weak assignee-keyed spawn barrier (the live 2026-06-27 out-of-order
break: 40842957 started before 9b3682b8's PR merged). Phase S2 runs
SequencingService over the surfaced siblings and wires the DAG via
add_dependency.
* [feature] wire dev-task collision DAG at cell-PM delegation (sequencing S2)
Pure dev_task_collision_edges in sequencing.py turns a parent's surfaced
siblings into (depends_on_id, task_id) pairs via SequencingService. TaskService.
wire_sibling_collision_dag wires them through add_dependency (idempotent). The
choreographer calls it after each dev-task delegate so the sibling collision DAG
is built incrementally as the cell PM decomposes — file-overlap serializes,
migration chains, shared-last; stable (priority, sequence) ordering keeps edges
from flipping into reverse cycles on re-runs.
* [feature] wire cell-task wave chain + by-osmosis edge (sequencing S3)
Kind 2 (cell-task wave chain): a new cell-task under root-subtask UT_n
depends on every cell-task under every root-subtask in UT_n.dependency_ids
(the kind-1 wave-chain edges), so its branch carries the previous wave's
merged cell work. Re-derived from the root-subtask's deps, not the cell-task's
own dependency_ids (which also carry UX/product-fanout edges the by-osmosis
edge must not pick up). A root may fan to several cell-tasks (different cells),
so the previous wave's cell-task is a SET.
Kind 4 (by-osmosis): the first dev task (sequence 0) under a cell-task depends
on each predecessor cell-task's tail (max-sequence) dev task, so the new wave's
first branch carries the previous wave's fully-merged tail. Subsequent dev
tasks inherit the tail via kind 3 or the merged base.
Both wired from _create_subtask_from_inputs, dispatched on parent.team
(MAIN_PM -> kind 2; cell team -> kind 4). Pure helpers
(cell_task_wave_chain_depends_on, by_osmosis_tail_dev_tasks) unit-tested in
test_sequencing.py; TaskService methods integration-tested. Idempotent +
best-effort throughout (add_dependency dedupes; missing predecessors are
no-ops). Also fixes a latent mypy-tests gap (estimated_complexity required on
direct TaskCreateRequest calls in the S2 tests).
* [feature] sync_branch dev verb — gate-level branch rebase (Phase B1)
Raw shell git is denied to agents (Bash(git:*) base deny), so a developer
whose branch fell behind its base had no gate-level rebase — only the
CEO/PM-only /rebase HTTP route. sync_branch is the dev verb that wraps the
rebase through the gate (traced + evidenced), so the 'everything goes through
the gates' invariant holds.
- lifecycle: IntentSpec sync_branch (dev-only, ownership-gated, composes=(),
git-only — no DB transition); _next_hint_synced helper.
- GitService.sync_task_branch: rebase task.branch_name onto its resolved base
via rebase_onto_base (fetch + rebase + force-with-lease push).
- Choreographer.sync_branch + _sync_branch_preflight_rejection: not_found /
unknown-role / spec-gate / no-branch / protected-base guards, then the git
op; conflicts abort (no force-push) and steer to resolve-by-hand; git failure
steers to i_am_blocked.
- HTTP route /api/v1/flow/developer/sync_branch + SyncBranchRequest schema.
- MCP tool sync_branch(task_id) + _TOOLS registration (manifest auto-propagates
via intents_for_role(Role.DEVELOPER)).
Tests: intent spec (5), choreographer handler (8: happy/conflicts/not_found/
not_authorized/no-branch/protected-base/git-failure/audit), route (1), MCP (1).
ruff + mypy roboco/ tests/ clean; unit suite green (DB-fixture errors env-only).
* [feature] i_am_done behind-base submit gate (Phase B2)
A sibling's PR merging into the parent branch while a dev worked leaves the
dev's branch behind its base — the assembled PR then can't merge cleanly and
the sibling's changes go missing (the 2026-06-27 out-of-order dev-task break).
The behind-base gate refuses i_am_done in that state and steers the dev to
sync_branch (the Phase B1 gate-level rebase verb).
- GitService.is_behind_base: rev-list --left-right --count across
origin/{base}...origin/{head} → (behind, ahead); fetch-first so origin
reflects the pushed head. Raises on git failure (consistent with
rebase_onto_base); malformed stdout degrades to (0,0).
- Choreographer._behind_base_gate: wired into _i_am_done_gate after
_ensure_branch_pushed. behind>0 → invalid_state remediate→sync_branch.
Fail-open on git/base-resolution error (flaky fetch can't strand a task at
the submit gate — the merge layer has its own behind checks). Skipped for
branchless roots and protected bases (master/main/-prefixed).
Tests: gate (6: refuse+steer/up-to-date/branchless/protected/fail-open-base/
fail-open-git), is_behind_base (6: parse/up-to-date/malformed/argv-form/
requires-branch/missing-project). ruff + mypy roboco/ tests/ clean; unit green.
* [docs] sync_branch prompt + behind-base guidance (Phase B3)
Update every behind-base/rebase guidance surface to reflect the B1
sync_branch dev verb + B2 i_am_done behind-base gate: devs now self-rebase
through the gate instead of escalating a plain behind-base condition; PMs
still escalate cell/root integration branches (they have no rebase verb).
- developer.md: sync_branch in the verb table; 'When your branch is behind
its base' rewritten — call sync_branch, do NOT i_am_blocked a plain
behind-base; conflicts → resolve by hand, commit, sync_branch again.
- cell_pm.md: delegate signature gains intends_to_touch/adds_migration/
touches_shared/depends_on + a 'Collision surface' section (fill it on every
code subtask so sibling dev tasks that touch the same files sequence into a
conflict-free order — the 2026-06-27 out-of-order break fix); behind-base
section steers devs to sync_branch, PMs escalate only the integration branch.
- main_pm.md: behind-base section — dev leaf = dev's sync_branch; cell/root
integration branch = escalate_up.
- RAG git-errors.md / blocked-tools.md: devs sync_branch, PMs escalate.
- docs/troubleshooting/common-issues.md: leaf self-rebases; integration branch
still escalates to operator.
- CLAUDE.md verb surface: developer gains sync_branch.
- agents/prompts/_generated/*: regenerated via scripts/regenerate_verb_tables.py
— adds sync_branch to the dev table AND catches the generated tables up to
the S1/S2 delegate sequencing params + meltdown-fix note top-level params
(the derived files had drifted stale vs the already-committed schemas).
Docs/prompts only — no code. ruff + mypy roboco/ tests/ clean.
* [chore] orchestrator: refuse to spawn human-only roles (CEO/prompter/secretary)
A live 2026-06-27 incident saw a CEO agent container spawned. Root cause:
_dispatch_a2a_work iterates every A2A/notification target and spawns it
with no human-role filter, and _is_agent_active('ceo') is always false
(the CEO is never a container), so the 'skip if active' check could never
protect the CEO. Any CEO-addressed notification (board handoff, escalation)
launched a CEO container — the system acting as the human CEO: a trust
violation. The CEO is the human operator; intake (prompter) and secretary
are human-driven chats launched through their own dedicated guarded paths
(_spawn_intake_container / _spawn_secretary_container), never spawn_agent.
Fix: a single chokepoint guard at the top of spawn_agent refuses
Role.CEO / PROMPTER / SECRETARY (raises AgentReadinessError + logs). This
structurally covers every dispatcher present and future, since they all go
through spawn_agent. Plus a defense-in-depth skip in _dispatch_a2a_work so
a human-role target never even calls in (avoids error-log spam; the
notification stays for the human to read in the panel).
Safe: the dedicated human-spawn paths do not route through spawn_agent.
Regression tests: spawn_agent refuses ceo/intake-1/secretary-1, does NOT
refuse a real agent; _dispatch_a2a_work skips CEO/intake/secretary targets
and still spawns real-agent + mixed-target cases.
* [chore] orchestrator: skip human-only assignees in claimed/pm-review dispatchers
Defense-in-depth for the spawn_agent human-role chokepoint (d31d6719).
The chokepoint structurally guarantees no CEO/prompter/secretary container
can ever spawn — every dispatcher goes through spawn_agent. But two
dispatchers resolve an arbitrary assigned_to and spawn it with only a
None/unknown-role filter, so a human-assigned task would reach the
chokepoint and RAISE: caught by the per-dispatcher try/except, but it
aborts that dispatcher's whole tick (stalling other respawns behind the
mis-assigned task) and error-logs every cycle. The other dispatchers are
already safe by whitelist/hardcoded slug (blocker_resolver_slug returns
None for non-PM/non-BOARD; escalation/approval use whitelists; marketing
and audit hardcode their non-human slug).
- _claimed_task_needs_agent: return None for a CEO/prompter/secretary
assignee — no container to respawn, and do NOT release a human-owned
task to pending (that would re-route it to a PM). Leave it for the human.
- _dispatch_pm_review_work (assigned branch): skip a human-only assignee
so a CEO-assigned awaiting_pm_review task neither spawns nor aborts the
dispatcher's tick.
Audited all target-iterating dispatchers; only these two lacked a filter.
Regression tests cover both skips.
* [F002] retype board-routed MegaTask root-subtasks code->planning on activation
_activate_batch_root_subtasks flipped a held root-subtask to team=MAIN_PM
but left task_type=code (intake only coerces main_pm-team drafts, so a
board-routed code root reached activation still code-typed). The
main_pm+code combo re-introduces the 2026-06-27 meltdown. Mirror
approve_and_start's own retype via main_pm_cannot_own_code so the
activated child is a planning-typed coordination root.
TDD: RED test_activate_batch_root_subtasks_retypes_code_to_planning
watched fail (task_type stayed CODE), then GREEN after the retype.
ruff+mypy clean; 125 batch/umbrella/approve tests green, no regressions.
* [F003,F004,F014] enforce HMAC agent-token gate on do routes + WebSocket streams
F003/F014: /api/v1/do/* only required X-Agent-ID (UUID) — no token check,
unlike the flow routers' role guards. A forged X-Agent-ID passed. Added
require_any_authenticated_agent (token-only; do router serves all roles)
and applied it as a router-level dependency. Binds X-Agent-ID to a verified
HMAC token when ROBOCO_AGENT_AUTH_REQUIRED=true; rejects a forged token
even in dev mode.
F004: /ws/* per-agent streams (channels/agents/sessions/notifications)
never read the nginx-injected X-Agent-Token, so in strict mode an agent on
the Docker network could subscribe to another agent's notifications with
no auth. Added _require_panel_token verifying the CEO panel token against
the CEO identity; wired into all four per-agent streams (system stream
stays operator-only per its docstring). Same strict/dev contract.
TDD: RED tests watched fail (no gate -> 200/accept), then GREEN. ruff+mypy
clean; 399 api/mcp + 29 WS tests green, no regressions.
* [F005,F006] grok auth: directory mount + atomic-write fallback
F005: the single-file bind mount of auth.json pinned the inode, so the
orchestrator's atomic refresh (tmp+rename within ~/.grok) never reached a
running grok container — a long-lived container hung at the login prompt
when the original ~6h token expired. Mount the host ~/.grok DIRECTORY (ro)
at /home/agent/.grok-auth-ro; the entrypoint symlinks ~/.grok/auth.json at
that RO mount so grok + the --check backstop read the live credential (the
directory mount sees the host-side rename) while grok's writable state
(config.toml, sessions/) stays in the image's ~/.grok.
F006: a rotated refresh_token is single-use — xAI invalidates the old one
the instant it issues the new one. If the atomic write failed after the
rotation, the file kept the now-dead old refresh_token and the credential
was permanently lost on the next refresh. _atomic_write now falls back to a
direct write when tmp+replace fails, so the rotated token always lands on
disk (losing the write is catastrophic; losing atomicity is not).
TDD: RED tests watched fail, then GREEN. ruff+mypy clean; 32 grok tests
green, no regressions.
* [F016,F017] choreographer: surface invalid_state instead of None.status 500 on submit_root / i_am_blocked
Both verbs compose a single atomic action whose None return (the verb's
own result) flowed out of run_intent and was dereferenced as t.status,
HTTP 500-ing with no actionable rejection:
- F016 submit_root: submit_for_review returns None when the root->master
PR was already opened / the task raced out of in_progress. Post-runner
None-guard extracted into _submit_root_finalize -> invalid_state
(re-fetch; if awaiting_pr_review the PR is open, wait for reviewer;
else re-delegate fixes and retry) instead of None.status.
- F017 i_am_blocked: escalate returns None in four cases (no task, no
agent, no resolvable escalation-target slug, no target agent row) e.g.
a developer whose role has no PM above it. _run_i_am_blocked_intent
now guards updated is None -> (t, invalid_state rejection) with
remediation (re-fetch + escalate to CEO directly / retry) instead of
the caller deref'ing None.status -> 500 + respawn-loop.
TDD red->green; ruff + mypy clean; gateway suite green (58 passed).
* [F007] choreographer: cell-level unchanged-PR re-submit loop-stopper for submit_up
The root loop-stopper (F016) was root-only; a weak cell PM could re-submit
the unchanged cell->root PR after a pr_fail and loop awaiting_pr_review ->
pr_fail forever (the cell analogue of the 2026-06-27 root loop).
pr_fail stamps the assembled PR's head SHA into notes_structured.pr_review
.head_sha for cell AND root gate tasks alike (the capture is gate-verb-
level, not root-level), so the same structural refusal applies to submit_up:
if the cell PR's current head SHA equals the SHA the last pr_fail recorded,
no new dev work landed on the cell branch -> the diff is byte-identical ->
refuse, do not re-open the gate. Different SHA -> branch advanced -> allow.
- _submit_up_unchanged_pr_guard mirrors _submit_root_unchanged_pr_guard
(cell-PM remediation: re-delegate to the dev + wait for re-assembly),
wired into submit_up after _submit_up_guard passes.
- Renamed shared _current_root_pr_head_sha -> _current_pr_head_sha (both
guards use it; the lookup was never root-specific).
- Every ambiguous case FAILS OPEN (no prior fail, no recorded sha, no
pr_number, no resolvable project, git/closed-PR None) — only the exact-
unchanged case is hard-blocked.
TDD red->green; ruff + mypy clean; F007+F016 guard suites green (15 passed).
* [F008] evidence_builder: surface persisted pr_review verdict+issues in the PM task_handoff
The pr_fail a2a steer to the owning PM is fire-and-forget; a PM respawned
into needs_revision later read none of it (build_task_handoff never looked
at notes_structured), saw a generic 'needs revision' with zero concrete
change-requests, and re-submitted the same PR (the 2026-06-27 infinite
pr_fail loop on 9980d0a0 / PR #138). The signal-gap was only partially
closed by the a2a.
build_task_handoff now extracts notes_structured.pr_review
(verdict/summary/issues/head_sha — the slot pr_fail authors on every fail)
into a pr_review field on the handoff, so every PM briefing for the task
carries the concrete change-requests. A prior pr_fail alone now counts as
prior-work-worth-resuming. Type-guarded + capped; absent => no key (no
misleading empty slot).
TDD red->green; ruff + mypy clean; evidence_builder suite green (14 passed).
* [F009] notification: derive requires_ack from ACK_REQUIRED_BY_TYPE, not the True default
NotificationService._create_notification built NotificationTable without
requires_ack, so the column default (True) applied to EVERY notification -
including informational REVIEW_REQUEST / DOCUMENTATION_REQUEST /
A2A_REQUEST / KNOWLEDGE_SHARE (ACK_REQUIRED_BY_TYPE -> False) and every
@mention from MessagingService._notify_mentions. Each false ack-required
inflated the recipient's unacked set and soft-blocked i_am_idle into
respawn churn.
- _create_notification: requires_ack=ACK_REQUIRED_BY_TYPE.get(type, True)
(unmapped types default True - preserve the action-required bias).
- _notify_mentions: requires_ack=False explicit (MENTION is informational).
TDD red->green (identity is False/is True assertions - the mocked
flush doesn't apply SQLA's insert-time default, so pre-fix the attribute
was None); ruff + mypy clean; notification suite green (18 passed).
* [F010] notification: never dedup informational notifications (knowledge-share data loss)
The purpose-based dedup suppressed a same-purpose (same sender/type/task,
overlapping recipients) notification while a prior one was unacked. For
informational types (KNOWLEDGE_SHARE / MENTION / A2A_REQUEST / BROADCAST +
the pickup-proves-receipt triad) each send carries DISTINCT content (a new
learning, a new mention) and acking is voluntary, so a recipient who never
acks the prior one let the dedup permanently suppress every subsequent
same-sender broadcast - silent learning-broadcast data loss.
The dedup's anti-loop rationale (stop unacked-set inflation soft-blocking
i_am_idle) only holds for action-required signals. Gate the dedup on
ACK_REQUIRED_BY_TYPE.get(type, True): action-required types still dedup,
informational types always create. Unmapped types default True (dedup on).
TDD red->green; ruff + mypy clean; notification + dedup suites green (20).
* [F011] playbook: de-index rejected/archived playbooks from the PLAYBOOKS RAG index
* [F012] release_executor: fail-closed on git add/commit before push
* [F013] release_proposal: Redis SET NX mutex guards the ~40min execute against concurrent approves
* [F015] flow_qa/flow_doc: add i_am_blocked route (manifest-registered escape hatch was 404)
* [F018] claim_guards: treat blocked as active + broaden the guard lookup so a blocked dev can't double-claim
* [F019] git: clear orphaned .git/*.lock files after a timeout-SIGKILL'd mutation op
* [F031] identity: role_for_slug_or_none so defensive skip-guards don't crash the dispatcher tick on stale slugs
* [F032] test: unknown-assignee claim reaches release-to-pending path
F031's role_for_slug_or_none fix made the unknown-assignee release branch
in _dispatch_claimed_without_agent reachable (the human-only guard no
longer raises/short-circuits on a stale slug). Lock that reachability in:
a claimed task with an unknown-assignee UUID past grace returns the slug
(not None) so get_agent_role -> 'unknown' releases the claim to pending
for a role-matched reclaim.
* [F033] orchestrator: capture container_id at startup re-adoption
_readopt_running_agents registered re-adopted ACTIVE instances with
container_id=None. _check_health skips container_id-is-None instances, so
when a re-adopted container later exited the stopped-container handler
never ran and the task stranded under a phantom ACTIVE instance forever.
Add _resolve_container_id (docker inspect -f '{{.Id}}') and store the real
id on re-adopt. Best-effort: a probe failure degrades to None (still
ACTIVE; the reaper's Docker-liveness fallback covers it).
* [F034] orchestrator: re-stamp respawn last_check at restore
_pm_made_rule_following_retry bounds its tracing_gap audit lookup with
since = record.get('last_check'). A stale persisted last_check from before
the restart matched pre-restart tracing_gap rows, falsely resetting the
breaker on the very first post-restart spawn — exactly when a fresh strike
count should be evaluating current state.
_partition_respawn_rows now re-stamps last_check to the restore time on
every restorable entry, bounding the lookup to post-restart gaps only.
* [F035] orchestrator: probe-resume loop actually revives parked agents
_park_provider_unavailable parked the provider + offlined the instance but
never registered a WaitingRecord, so _on_probe_success -> _parked_agents_for
(always filtered on waiting_for=='rate_limit_lifted') returned [] and
resolve_wait revived nobody — recovery fell to the 600s stale-claim reaper
instead of the probe-success path the parking design relied on.
Register + persist a rate_limit_lifted WaitingRecord at park time (mirrors
mark_waiting_long, minus stop_agent — the container is already dead).
Companion reaper guard: _reap_with_service now skips provider-parked
assignees (_assignee_is_provider_parked) so the claim survives until the
probe revives the agent — otherwise the reaper releases the claim to pending
and probe-success respawns on a task the agent no longer owns.
* [F036] orchestrator: read transcript for overload detection too
The SDK server writes model-API errors (529/500/503) to /tmp/sdk-server.log,
not stdout, so an overload marker can appear only in the durable Claude
transcript — the same rationale already applied to the session-limit
detector. _provider_overload_park_target read only docker logs, so an
overload was missed and the agent crash-respawned straight back into it.
Now concatenates the transcript tail before matching, mirroring the
rate-limit path.
* [F037] orchestrator: drop bare error-NNN overload markers
The bare 'error 529'/'error 500'/'error 503' markers were broad enough to
false-match an agent that merely writes about an HTTP status code in its own
notes ('the endpoint returned error 500, retrying'), parking the whole
Anthropic fleet on a non-issue.
The SDK error formatter emits 'API Error: NNN' + a JSON error type, so the
remaining 'api error: 529/500/503' + 'overloaded_error' +
'internal_server_error' markers cover every real overload without that
false-match surface.
* [F038/F039] orchestrator: sign X-Agent-Token on self-API calls
The prior self-PATCH 401 fix only carried X-Agent-ID/X-Agent-Role. Arming
ROBOCO_AGENT_AUTH_REQUIRED=true made the middleware require a signed
X-Agent-Token, so every orchestrator self-call (auto-block / auto-resume /
auto-recover / SLA annotation) 401'd and silently no-op'd — wedging
paused/blocked parents.
Add _system_api_headers() that wraps the base headers with a signed token
for the system identity (issue_agent_token); switch all six self-call sites.
Dev fallback: no secret set => UNSIGNED sentinel + auth not required.
* [F040] orchestrator: finalize grok spawn session on cost-cap kill
_enforce_grok_cost_budget killed + evicted the container without calling
_finalize_spawn_session, so the open agent_spawn_sessions row stayed open
(ended_at IS NULL) and the burned usage/cost was never recorded in the
dashboard.
Call _finalize_spawn_session(exit_reason='cost_cap') BEFORE popping the
instance — it reads self._instances[agent_id] for the model +
usage_session_id, which the pop would lose.
* [F041] park grok exit-78 (auth missing/expired) instead of crash-retrying
A one-shot grok container whose entrypoint ran grok_auth --check and found
the token missing/expired exits 78 (EX_CONFIG). Crash-retrying 3x burns
tokens for zero progress — the agent cannot start without a valid token.
Park the provider with kind=auth_missing (same shape as the 429 exit-75
path) so the probe-resume loop revives the task once grok_auth.refresh_if_stale
mints a fresh token; if still expired, the next exit 78 re-parks (no burn).
Also fixes a latent F035 regression: _park_provider_unavailable now registers
a WaitingRecord, so the bare-__new__ rate-limit park test had to set
_waiting_records + stub _persist_waiting_record (mirrors the overload-test
fixture).
* [F042] isolate concurrent-duplicate conventions cache put in a savepoint
Two task creates for the same project/HEAD can race to populate the
conventions cache; the loser's INSERT fails the partial-unique index with
IntegrityError. A bare session.add + flush poisons the shared session (the
task-create transaction rides the same session), so every subsequent op
raises 'this session is in error state' and task creation crashes.
Run the INSERT in a savepoint (begin_nested) and swallow the IntegrityError:
only the savepoint rolls back, the outer transaction stays usable, and the
winner's row satisfies the next _cache_get.
* [F043] guard escalate_up against resurrecting terminal tasks
escalate_up had composes=() and no source-status guard, so a PM could
escalate a COMPLETED/CANCELLED task and apply_escalation set it back to
BLOCKED — bypassing the state machine's terminal-state invariant.
Defense in depth:
- spec: add PRECONDITION_NON_TERMINAL to escalate_up's extra_preconditions so
the lifecycle gate rejects terminal tasks (invalid_state) before the
journal:decision write fires; generalize _check_intent_preconditions to
honor non-tracing rejection_kind (not_authorized / invalid_state).
- service: apply_escalation (the single write primitive) returns False and
refuses to mutate a terminal task — covers the HTTP escalate route which
bypasses the spec gate. escalate() / escalate_up_to_role() return None on
refusal so the gateway emits a clean invalid_state envelope.
- route: the HTTP escalate route 409s a terminal task BEFORE sending the
escalation notification (so a finished task isn't yanked back, PM not pinged).
* [F044] pr_pass gate remediation points the reviewer at pr_fail, not i_am_blocked
The pr_pass gate runs the toolchain + conventions guards on the REVIEWER's
workspace, but their remediation text said 'call i_am_blocked' — a verb the
PR reviewer does not have. The reviewer would chase a verb they cannot call
instead of rejecting the PR.
Make the guards reviewer-aware: a reviewer=True flag (passed by _pr_pass_blocked)
switches the remediation to pr_fail(issues=[...]) — the reviewer's reject
lever, sending the PR back to needs_revision for the dev to fix the
environment / validator. The dev (i_am_done) path keeps i_am_blocked, which a
dev does have. _conventions_guard (the pr_pass path) now passes reviewer=True
through to _conventions_rejection.
* [F045] rate-limit: loud activate-failure log + in-memory orphan-probe fallback
The in-verb i_am_blocked(rate_limited) path wrapped RateLimitStateTracker.activate
in a bare contextlib.suppress. A silent activate failure stranded the fleet:
agents were parked in _waiting_records but the provider never entered the tracker,
so the tracker-driven _sweep_rate_limit_probes never probed it and no
_on_probe_success ever resumed them — parked agents stuck in WAITING_LONG.
Fix: (1) replace the bare suppress with a try/except that logs an error event
naming the provider + affected agents; (2) in _sweep_rate_limit_probes, after
probing the tracker-listed set, scan _waiting_records for any rate_limit_lifted
provider the loop did NOT cover and probe it via the time-expiry fallback (empty
state -> probe now) so _on_probe_success resumes the parked agents. The fallback
reads only local memory, so it still resumes when Redis was down at park time
(list_rate_limited_providers failure now falls through to the orphan scan instead
of returning early).
* [F046] pr_gate: guard None runner result on concurrent transition (pr_pass/pr_fail)
_gate_decision dereferenced the verb-runner result without a None guard.
run_intent returns None when a concurrent transition (cancel or a racing
reviewer) moves the task out of awaiting_pr_review between the precondition
gate and the runner's final composed action (the verb runner's documented
last-action source-status contract). The subsequent t.assigned_to /
t.status / _post_gate_review_to_pr(t, ...) dereferences then crashed the
gate with a 500 AttributeError. Add a None guard that surfaces a clean
invalid_state rejection (re-fetch + re-issue) before any dereference; no
PR post or a2a runs against a None task. TDD test_pr_gate_notifies_pm.py (+2).
* [F047] conventions: reviewer-aware block-finding remediation on pr_pass gate
The pr_pass (reviewer) conventions guard reused the dev-path block-finding
remediation: 'add a waiver to .roboco/conventions.yml in your branch'. A
pr_reviewer does not own the assembled cell->root / root->master branch and
has no commit verb on it, so the waiver remediation is unreachable — a false
positive stranded the gate with no self-recovery (the reviewer could neither
commit a waiver nor pr_pass). The fail-open content path is documented
precision-over-recall and stays as-is; the actionable gap is the remediation.
Fix: _conventions_rejection now branches the block-finding remediation on
reviewer=True (mirroring the could_not_run branch from F044). The reviewer
path points at pr_fail carrying the findings as issues so the PR returns to
needs_revision and the DEV fixes the violation or commits the waiver (the dev
CAN commit to the branch); waiver authorship is framed as the dev's action,
not the reviewer's. Dev i_am_done path wording unchanged. TDD
test_conventions_gate_pr_pass.py (+1).
* [F048] notify: reject human-only recipients (prompter/secretary) — no agent ack path
notify() only checked the SENDER role. The recipient was resolved by
NotificationService._resolve_recipients, which drops only unresolvable slugs
— it does not exclude human-only roles. The prompter (intake-1) and secretary
(secretary-1) are seeded agent rows, so they resolved, and an ack-required
ALERT addressed to them sat permanently unacked (no agent auto-acks it),
polluted the panel's pending-ack view, and — via the dedup query's
~acked_by.contains — permanently suppressed any later same-purpose
notification from the same sender to that human role. The knowledge-share
path already excludes all three human-only roles; the general notify path
did not.
Fix: a recipient-role guard in notify() via _reject_disallowed_recipient
(folds the new check into the existing CEO-dependency-block return slot so
notify stays under the PLR0911 return limit). Rejects prompter/secretary
with not_authorized; the CEO is human too but acks via the panel, so it stays
an allowed recipient (its only disallowed case, a dependency-block page, is
preserved). TDD test_notify.py (+3: reject prompter, reject secretary, allow
CEO).
* [F049] merge_pull_request: idempotent on already-merged PR (mirror _merge_with_retry)
* [F050] merge_pr_for_task: verify caller pr_number matches task's recorded PR
* [F051] open_conventions_pr: refuse dirty tree + verify checkout-base landed
* [F052] pr_target: scope task lookup by project_id (mirror close_pull_request)
* [F053] _token_for_project: log decryption failure (key rotation) with project slug
* [F054] learnings index: enforce shareable on every shared retrieval path (private-leak fix)
* [F055] messaging: recover from concurrent channel auto-create race via savepoint + re-fetch
* [F056] messaging: lock group row before session check-then-create to prevent active-session orphan race
* [F057] playbook: index/unindex as a post-commit step so the RAG corpus never leads the status transaction
* [F058] release-readiness: non-empty bump plan on first release
_canonical_bump_files derived the bump set from the previous
chore(release): commit. On the first release there is no such commit,
so it returned [] -> assess set version_bump_plan=[] -> the executor
published a tag with no files bumped (a no-op masquerading as X.Y.Z).
Fall back to the version-reference scan when no prior release commit
exists: the files currently embedding the version are exactly the set a
first release must bump, and the set the first release commit then
records as canonical for subsequent releases. Read-only derivation; the
CEO-approval gate and fail-closed executor are untouched.
* [F059] self-heal: hold fix tasks for CEO Approve-&-Start (restore dispatch gate)
The module docstring promised self-heal fix tasks 'wait for the CEO's
Approve-&-Start', but _originate created them confirmed_by_human=True and the
orchestrator dispatched them at once — a self-heal fix that re-broke CI would
trigger another cycle, open another auto-dispatched fix, and loop with no CEO
gate on dispatch.
Restore the documented gate:
* _originate opens the task confirmed_by_human=False (held for the CEO).
* The orchestrator holds a self-heal task out of both the PM and dev dispatch
paths until confirmed_by_human flips True.
* approve_and_start (the CEO's start gate) sets confirmed_by_human=True so the
held task finally dispatches (idempotent for board/intake tasks already True).
* list_pending_for_agent scopes the give_me_work hold to self-heal
(source != self_heal OR confirmed_by_human) so an already-alive PM can't grab
it pre-approval — while ordinary delegated subtasks (confirmed_by_human=False
by default, where the delegation IS the authorization) still dispatch.
The 'never self-deploys' guarantee (no merge) is unchanged.
* [F059] fix DB-integration test auth + retype self-heal root code→planning
conftest test-DB defaults matched the project's own running postgres
(roboco/roboco @ localhost:15432, the docker-compose roboco-postgres
service with CREATEDB) instead of the OS user on localhost:5432 which has
no such role — every db_session test failed with InvalidPasswordError
instead of running.
Once the DB connection worked, the self-heal origination DB test went RED
with MAIN_PM_NO_CODE: the self-heal root was task_type=CODE owned by
main_pm, the combo the main_pm_cannot_own_code guard rejects. The Main PM
coordinates the fix (delegates the code work to a cell dev); it has no
code verb. Retyped CODE→PLANNING and rewrote description/AC to
coordination-level.
* [F060] emit reversal audit row on claim-branch-failure rollback
The forward task.claimed audit row is flushed before the branch-creation
attempt, and AuditService commits on its own connection, so the rollback's
flush reverts the task row but not that audit row — the journey's last
event stayed task.claimed while the task reverted to its pre-claim status,
diverging from real state and corrupting downstream cycle-time/bottleneck
metrics. The rollback now emits a CLAIMED->original reversal audit row
(only when the forward transition was made) attributed to the claimant.
* Removing completely unnecessary files (for the repo they are unnecessary)
* [F061] audit status-transition rows now written in-session (F061/F073/F075)
_emit_status_transition_audit now writes AuditLogTable rows into
self.session synchronously (session.add) instead of dispatching
AuditService.log_task_event fire-and-forget on its own connection.
The audit row now commits/rolls back atomically with the status
transition in the caller's transaction, closing three facets at once:
- F061: audit commit no longer decoupled from the transition commit
- F073: a committed transition can no longer have NO audit row
(the row rides the same transaction; a swallowed persist can't drop it)
- F075: a transition rolled back inside a verb savepoint no longer
leaves a phantom audit row (the row is in the savepoint too)
log_task_event is now called only from this helper (narrow blast
radius verified); revision_count increment stays at this single
chokepoint. Cycle-time/bottleneck reconstruction from task.<status>
events is no longer silently corruptible.
Tests: test_emit_status_transition_audit_writes_in_session_atomically,
test_finalize_claim_rollback_emits_reversal_audit, escalation-audit
tests retargeted to in-session AuditLogTable rows.
Also: _canonical_bump_files grep-looseness follow-on (F058) -- filter
by subject, not body; git log --grep matches any message line, so a
non-release commit whose body references chore(release): shadowed the
real release commit. Test
test_canonical_bump_files_ignores_body_only_chore_release_match.
* [F061] drop type:ignore from audit-emit tests
Convention: no type:ignore/noqa. The F061 in-session audit-emit
tests used '# type: ignore[assignment]' to assign a MagicMock to
AsyncSession.add, and the F060 test assigned to .flush the same way.
Rewritten to hold a local 'session: MagicMock' variable (mypy sees
its auto-children as MagicMock, so .add.side_effect / .flush assign
cleanly with no suppression). Verified via 'mypy tests/' that both
files are now type-clean (the F060/F061 commits had skipped tests/
in mypy, masking two method-assign errors).
* [chore] clear all 64 pre-existing mypy errors in tests/ (no type:ignore)
Convention: no type:ignore/noqa, and pre-existing violations still
violate. The make-quality gate runs 'mypy roboco/ tests/', but the
prior commits' gates only ran mypy on production files, masking 64
type errors across 15 test files (method-assign, unused-ignore,
no-untyped-def, attr-defined, union-attr, has-type, index, misc).
Fixed without any type:ignore:
- method-assign (svc.session.X = / svc.method = AsyncMock()): hold a
local 'session: MagicMock'/'AsyncMock' and assert on it, or stub via
object.__setattr__ / monkeypatch / a typed '_bind' helper returning
Any, or alias 'cc: Any = c' (the pattern the file already used).
- unused 'type: ignore[assignment]' (real code was method-assign):
removed; replaced with the no-suppression patterns above.
- 'Callable[...] has no attribute assert_*': keep a typed local ref to
the AsyncMock and assert on the local, not the method-typed attr.
- no-untyped-def: annotate helper params (Any / pytest.MonkeyPatch).
- attr-defined / index / union-attr: type the helper as Any, narrow
with an 'is not None' assert, or add the missing attr to a fake.
- has-type / return-value: fix the declared return type to the tuple
the function actually returns.
- PLC0415 inline imports: hoisted to top-level.
test_pr_gate_notifies_pm._stub_gate_path converted fully to the
'cc: Any = c' alias (it already used it for one attr) so its five
'# type: ignore[method-assign]' suppressions are gone.
mypy tests/: 64 errors -> 0 (538 files). ruff check tests/: clean.
All 84 tests in the touched files pass.
* [chore] remove all remaining type:ignore suppressions from tests/
Converts 115 `# type: ignore[...]` suppressions across 23 test files to
no-suppression patterns (helper-return widening to Any, local Any aliases,
cc:Any aliases, cast at narrow call sites, typed fixtures) so the hard
no-type:ignore convention holds across tests/. No test logic or assertions
changed — only mock-wiring mechanics and type annotations.
Gate: ruff check tests/ clean; mypy tests/ (538 files) clean; 176 changed-file
tests pass. Zero real suppressions remain (the 7 grep hits are 3 hygiene-
checker string-literal test inputs and 4 prose mentions in comments).
* [F062] work_session.merge_pr: idempotency + active-status guard
merge_pr unconditionally set pr_status=merged, pr_merged_at, merged_by,
status=COMPLETED on whatever session it loaded — the only session-terminal
transition in WorkSessionService lacking both the active-status guard
(complete/abandon) and the terminal-idempotency guard (close). Two failure
modes: (1) a retried merge after a successful-but-unconfirmed GitHub merge
overwrote merged_by/pr_merged_at with the retry's actor/timestamp, corrupting
the merge audit trail; (2) merge_pr on an ABANDONED session resurrected it to
COMPLETED, undoing the single-active abandonment. Mirrors close()'s guard:
if status != ACTIVE, return the session unchanged. Both git.py callers await
merge_pr and discard the return, so the no-op is safe. TDD: 3 tests
(happy-path + both modes).
* [F063] workspace._clone_repo: rmtree half-configured clone on failure
If _configure_git raised CalledProcessError before its `remote set-url`
scrub, .git/config kept the tokenized auth URL (the project PAT) and
_assert_no_pat_leak never ran. The except clauses raised WorkspaceError
without removing the workspace, so the next ensure_workspace's health
short-circuit (valid .git with HEAD + objects) skipped past the leak —
mounting the agent on a workspace whose .git/config let it read+exfiltrate
the PAT. Both clone-failure except clauses now rmtree the workspace before
raising, so a half-configured clone is destroyed and ensure_workspace
re-clones from scratch. TDD: 2 tests (configure-failure leak + timeout).
* [F067] flow_main_pm: add missing /triage route
main_pm's manifest advertises triage (lifecycle.intents_for_role(MAIN_PM)
includes it via _PM_ROLES, alongside triage_all) but flow_main_pm.py had no
POST /triage route, so a main_pm agent calling triage hit a raw 404 that
bypassed the per-verb circuit breaker. Added the route mirroring flow_cell_pm's
/triage — wires to the existing team-scoped choreographer.triage (uses pm.team,
works for any PM role; Main PM gets its own team's blocked/awaiting tasks).
Fix direction: add-route, NOT remove-from-manifest — the manifest is spec-correct
(intents_for_role by construction); removing triage would contradict the spec
and leave main_pm with only cross-team triage_all. TDD: test_triage_route_exists_and_dispatches.
* [F068][F069] mcp servers: classify all rejection shapes + envelope 404s
F068: the do/flow-server circuit breaker only counted rejections whose
`error` field was a STRING in _CIRCUIT_REJECTION_KINDS. A 422 validation
failure (no `error` field, a `detail` list) and a 500/HTTPException
(dict-shaped `error` from the exception handlers) both bypassed the breaker
→ unbounded retries on a storm of either. Added _classify_rejection(payload)
(shared, applied to both servers) mapping all three shapes to a counted kind:
string error (existing), dict error → substring-mapped code
(*DENIED*/*AUTHORIZED*/*FORBIDDEN*/*PERMISSION*→not_authorized,
INVALID_INPUT/*VALIDATION*→incomplete_input, *NOT_FOUND*→None parity, else
→invalid_state), 422 detail→incomplete_input. The dict TypeError defence lives
in the classifier (isinstance, never dict-in-frozenset).
F069: a manifest-registered verb whose HTTP route is missing got FastAPI's raw
`{"detail":"Not Found"}` 404 body — a non-envelope payload the breaker
couldn't classify, so a storm bypassed it. _post now synthesizes an
invalid_state Envelope rejection (with a remediate hint → i_am_blocked/i_am_idle)
for a 404 status, routed through _record_and_check_circuit so the breaker counts
it. A 404 that carries a real Envelope (error field present) is surfaced as-is,
preserving test_flow_post_returns_envelope_on_404. TDD: 422/dict/404 tests in
both server test files; updated test_dict_shaped_error_does_not_crash to assert
the SDK is now called with not_authorized (replacing the pass-through assertion
that encoded the bug).
* [F064][F065][F066] websocket: non-blocking fan-out, finally-disconnect, idle timeout
F064: the bridge forwarder awaited every conn.send_text in a gather with no
per-connection queue and no send timeout — one slow WS client back-pressured
ALL event delivery to ALL clients (head-of-line blocking on the listen loop).
Each connect_* now registers a _ClientConnection (bounded asyncio.Queue(256) +
sender task); broadcasts enqueue via put_nowait (drop + structlog warn on
QueueFull) and return immediately. The sender drains the queue with each send
wrapped in wait_for(SEND_TIMEOUT=10s). Unregistered legacy sockets (set
directly into a subscription set, bypassing connect_*) get a timeout-bounded
fallback send task held in _pending_sends (ruff RUF006). disconnect cancels +
drops the sender.
F065: route handlers caught only WebSocketDisconnect with no finally — a
non-clean exit (anyio closed-resource, CancelledError, transport error)
propagated without manager.disconnect, leaking the dead socket into every
subscription set forever. Added finally: manager.disconnect(websocket) to all
5 handlers (disconnect is idempotent).
F066: no server-side heartbeat/idle timeout — a half-open socket from a dead
container blocked receive_text forever and was never reaped. receive_text now
wraps in wait_for(IDLE_TIMEOUT_SECONDS=90s); on TimeoutError, log + fall
through to the F065 finally. Named module constants (no config.py precedent for
WS tuning; callers/tests patch them).
TDD: 22 new tests across 3 files (handler cleanup, idle timeout, send queue),
non-flaky across repeats; 1 existing test adapted with a yield for the new
async fan-out (assertion unchanged). ruff/mypy clean, 421 unit/api tests pass.
No type:ignore/noqa.
* [F022][F023][F024][F025][F026] api: scrub secrets from 422 log, gate a2a/dashboard/orchestrator routes, SSE session-per-query
- middleware: redact known credential fields (git_token/api_key/token/...)
from the 422 request-validation log line; response body unchanged
- a2a: require_any_authenticated_agent on /message/send + /message/stream;
subscribe_to_task opens a short-lived session per poll instead of holding
one asyncpg connection for the full SSE lifetime (pool exhaustion) + auth
- dashboard: gate auditor flag/report mutating routes to Auditor or CEO
- orchestrator: router-level CEO gate on all control routes (spawn/stop/...)
TDD; ruff/mypy clean; 449 unit/api tests green; no type:ignore/noqa.
* [F030] conventions: typescript-scoped custom rules now apply to .tsx files
The validator tags a .tsx file as language 'tsx' (the JSX grammar needs
that tag, distinct from plain 'typescript'), but a custom rule scoped to
'typescript' — the language the scan reports for a React+TS repo — silently
skipped every .tsx file. The two suffix maps were NOT unified: the 'tsx'
tag is load-bearing (grammars.py picks the JSX grammar on it; hygiene.py
keys on it), so unifying would make .tsx fail to parse.
Fix is in check_custom: a one-directional dialect map _DIALECT_OF =
{'tsx': 'typescript'} — a typescript-scoped rule fires on a .tsx file,
but a tsx-scoped (JSX-only) rule still does not fire on plain .ts.
TDD; ruff/mypy clean; 80 unit + 38 integration conventions tests green.
* [F029] websocket: remove broken /api/permissions/check loopback from channel stream
channel_stream called validate_channel_access, which HTTP-loopbacked to
GET /api/permissions/check — a route that does not exist. Every call 404'd
-> False -> the channel stream closed with WS_1008_POLICY_VIOLATION for
EVERY client, so the real-time channel stream was dead. Removed the
function, its call site, and the now-unused httpx + settings imports.
Post-F004 the panel-token gate is the channel-stream authorization (the
CEO panel is the sole WS client and may view every channel), so the
broken loopback is removed rather than replaced with an in-process check
the CEO always passes. The legitimate enforcement.validate_channel_access
(slugs, in-process static ACL) is a different function and is untouched.
F027 is resolved-by-F004 (no code change): all three per-agent streams
gate on _require_panel_token first, so only the authorized CEO panel can
connect — 'any viewer subscribes to any target' is closed.
TDD; ruff/mypy clean; 530 unit/api+enforcement+RBAC tests green.
* [F078] release_executor: deadline every subprocess (git/make/gh/clone)
A hung git/make/gh/clone would block the CEO-gated release loop
indefinitely. Wrap each proc.communicate() in asyncio.wait_for via a
shared _await_proc helper; on expiry proc.kill() the child and return a
non-zero rc (124) so every caller's fail-closed branch fires. Mirrors the
quality-gate _run_one kill-on-timeout idiom.
Deadlines are generous (30min gate / 10min clone / 5min push+gh) so a
legitimate slow op is never wrongly aborted — floor-assertion tests pin
the floors to guard exactly that logical regression. Green path returns
the real rc unchanged.
* [F072] reaper: deadline docker inspect/exec + harden _check_health sweep
A hung Docker daemon (or a stuck container FS) froze the single asyncio
event loop: the reaper runs inline before every dispatch tick and shares
that loop with every background sweeper. Bound each docker subprocess
with asyncio.wait_for; on expiry proc.kill() the child and either raise
(inspect / resolve_container_id — callers apply their own fail-direction)
or return None (the gateway probe — inconclusive, caller declines to act,
matching its existing probe-failure contract). Deadlines generous
(10s inspect / 30s exec) so a legitimate slow docker call is never
wrongly aborted; floor-assertion tests pin the floors.
Also harden _check_health's per-agent loop so one agent's hung inspect
skips that agent, not the whole sweep — preserving the check-all-agents
invariant the timeout-then-raise would otherwise break (without this, a
hung daemon means no agent gets health-checked any tick).
* [F076] say/dm: handler guard rejects all 4 no-comms roles, not just auditor
The say()/dm() defence-in-depth guard only rejected auditor, but CLAUDE.md
mandates the same no-agent-comms invariant for pr_reviewer (posts findings
on the PR), prompter and secretary (human-only, note + evidence). For those
three the manifest was the only gate, so a call bypassing the manifest
(direct API POST, test harness, future routing change) would not be refused
at the handler — admission depended on the agent's slug happening to be
absent from the channel/a2a matrix. Extend the guard to a _NO_COMMS_ROLES
frozenset (auditor + pr_reviewer + prompter + secretary), matching the
explicit role-frozenset gates on commit/notify/pitch/playbook/open_session.
Role-appropriate remediation per role. The claimed defence-in-depth now
covers 4 of 4 silent roles, not 1 of 4.
* [F070] drain fire-and-forget _bg_tasks on shutdown (bounded, data-preserving)
Orchestrator.stop() cancelled only the named loop tasks + agents, then
returned, abandoning in-flight _schedule_bg work. An in-flight
_persist_respawn_record upsert dropped at shutdown meant the last few
gate-mutation strikes never reached the DB; restore_respawn_tracker() on
the next start repopulated a stale lower count and the dispatcher re-burned
the full 4-spawn strike threshold against a still-wedged task — the exact
re-burn the durable tracker exists to stop. Audit-log writes (load-bearing
for cycle-time/rework metrics) were similarly dropped.
Add _drain_bg_tasks(): bounded wait (5s default) lets short DB writes
commit before exit (data preserved), then cancels any stuck task past the
deadline so a hang can't wedge shutdown. return_exceptions=True so one
failing bg task doesn't crash the drain. Wrap the stop_agent loop in
try/except + logger.exception so one bad agent can't skip the drain
(re-introducing the data-loss tail). Floor test pins the deadline >= 3s
so a too-short change can't silently drop a legitimate slow write.
* [F071] abort non-blocking intake/secretary spawn on mid-spawn shutdown
The non-blocking spawn (start_intake_session / start_secretary_session)
schedules _spawn_intake_container_guarded / _spawn_secretary_container_guarded
via _schedule_bg. Those run docker run and only register in _instances at the
END. If shutdown arrived between docker run and the registration line, the
container was started but the orchestrator had no handle — stop() iterates
only _instances, so the container was orphaned (leaked, manual docker rm).
Worse, the F070 drain could let the spawn coroutine complete the
registration AFTER stop() already iterated _instances, landing a live
container into a shutting-down registry nothing tears down.
Add a post-docker-run shutdown guard in _spawn_intake_container and
_spawn_secretary_container: re-check self._running after _run_container_cmd
returns; if the orchestrator began shutting down, remove the just-started
container (by its deterministic name) and raise _SpawnAbortedDuringShutdown
WITHOUT registering. The guarded wrappers catch that BEFORE except Exception
and close the live relay silently (shutdown is not a user-facing failure,
no error pushed to the SSE stream). The F070 stop() drain awaits the bg
spawn coroutine, so the abort surfaces cleanly.
TOCTOU-safe: between the _running check and the _instances assignment there
is no await (config + instance construction are sync), so once the check
passes, registration completes before the event loop can interleave stop().
The normal running path is unchanged (sanity tests pin it).
* [F074] per-agent advisory lock closes claim TOCTOU
_run_claim_guards read the agent's other tasks via unlocked SELECTs
before claim() took its row lock, and claim()'s FOR UPDATE locked only
the TARGET row — so two concurrent i_will_work_on by the SAME agent on
TWO DIFFERENT pending tasks each locked their own row, each read an
empty in_progress set, each passed already_active, each claimed+started
→ the agent ended with two in_progress tasks (the in-process asyncio
Lock is lost on orchestrator-restart split-brain, so it wasn't a
DB-level guarantee).
Fix: TaskService.acquire_claim_lock takes a transaction-scoped
pg_advisory_xact_lock keyed by hashtextextended(agent_id). The gate
acquires it BEFORE the guard reads (for non-coordinator roles only) so
the second concurrent claim's read sees the first's committed
in_progress task and is rejected. Tx-scoped → auto-releases on
commit/rollback, can't outlive the request.
Coordinator exemption (the key logical-regression guard): cell_pm /
main_pm do NOT take the lock — the PM coordinator concurrency feature
lets a PM plan+delegate many roots in parallel, and a per-agent lock
would serialize those claims and regress it. Matches the existing
_COORDINATOR_ROLES already_active/paused guard exemption. A hash
collision only causes benign false serialization, never a false
negative.
Tests: unit (dev acquires lock before guard read; coordinator does
not) + real-PG integration (same-agent serializes, different-agent
does not, releases on rollback).
* [F021] handle SSE transport errors so the intake composer isn't stuck
openStream registered listeners for the server-sent event kinds but not
the EventSource's own transport-level error. The 'error' kind IS in
LIVE_EVENT_KINDS, so a server-sent event:error (JSON MessageEvent) was
handled — but a dropped connection / dead session fires a plain Event
with NO data, which JSON.parse(undefined) swallowed in the try/catch,
so the stream 'stayed open' (EventSource loop-reconnected a session that
no longer existed) and isSending stayed true — the composer was
permanently disabled.
Fix: route the 'error' event by payload. A MessageEvent with string
data is a server-sent error → handleEvent (unchanged). A no-data Event
is a transport error → handleTransportError: clear streamingId/activity,
set isSending false, add a 'connection lost' error message, keep a
draft/batch preview up (so the human can still act on a proposed card)
else land on 'chatting', and close the dead stream so EventSource stops
loop-reconnecting.
Tests: renderHook + a jsdom EventSource double that fires a transport
error (plain Event, no data) vs a server-sent error (MessageEvent +
JSON). RED: transport error left isSending true; GREEN: resets to
false, surfaces the message, closes the stream. The server-sent-JSON
path is unchanged. Full panel suite (129) green; eslint/typecheck/prettier clean.
* [F081] Approve dialog: label notes required (>=20 chars), not optional
The CEO Approve dialog's notes label fell into the default branch
('Notes (optional') for the approve action, but approve actually
requires substantive notes >= 20 chars — enforced client-side
(toast error on < 20) and server-side. So the CEO was told 'optional'
and only learned the real requirement from a toast after hitting
submit with empty notes.
approve and start both require >= 20 chars; reject only requires a
reason. Collapse the label to two branches: reject -> 'Reason for
rejection (required)'; everything else (approve + start) ->
'Approval notes (required, >= 20 characters)'. The approve
placeholder now also signals intent ('Why this is ready to ship...').
Tests: render the queue, click Approve, assert the notes label says
'required' + '20' and does NOT say 'optional'. RED: label read
'Notes (optional)'; GREEN: 'Approval notes (required, >= 20
characters)'. eslint/typecheck/prettier clean.
* [F082] surface release-proposal query failures instead of silent hide
The card collapsed any non-404 backend failure (500 / network drop) onto
`!proposal` and returned null, so the CEO had no idea the release-proposal
endpoint was unreachable. Distinguish the cases: isError + a Retry affordance
vs the 404 null empty state that stays hidden. Mirrors PrReviewQueue.
* [F083] clear stale usage snapshot when /ws/system leaves connected
The hook synced wsState into the store but never dropped usageData when the
stream dropped, so on reconnect wsState flipped to "connected" before any
fresh USAGE_SNAPSHOT arrived and UsageOverviewPanel rendered the prior
session's totals/cost as if they were live. Clear usageData whenever state
leaves "connected" so the panel falls back to the polling summary until a
new snapshot lands. Connected->connected is a no-op clear skip.
* [F084] scope per-control disable to the in-flight mutation, not all
FeatureFlagsCard disabled every switch while any one flag toggle was pending,
and PlaybookReviewQueue disabled every row's Approve while any one approve was
pending — so the operator couldn't act on an independent control during a
slow round-trip. Gate the disable on the in-flight mutation's variables
(matching key / id) so only the control being mutated locks; the others stay
usable. The same-flag double-tap protection is preserved.
* [F085] reject submitting both project_id and product_id
validate() only checked 'at least one of project/product', so the dialog let
both be submitted together. The server silently lets product_id win at routing
and drops project_id, recording a misleading, never-used repo. Add a validator
that refuses the ambiguous submit with a clear error. The at-least-one rule and
the single-pick submit paths are unchanged.
* [F020] kanban: confirm admin-override drags that skip lifecycle preconditions
A drag on the operator kanban routes the status move through the admin
status-override, which bypasses the in-band lifecycle validator entirely.
That override is intentional (it's how an operator recovers a wedged task)
but it also let a careless drag skip material preconditions silently —
completing a task with no open PR, QA-bypassing, finishing docs on a task
whose docs aren't complete.
Leave the override intact but make the bypass explicit: compute the
preconditions the dragged move would skip (open PR, docs complete,
self-verified + commits + progress for submit-qa, visible non-terminal
subtasks for coordination-root targets) and, when any are skipped, hold the
move behind a confirmation dialog that lists exactly what's being skipped.
Precision over recall — only warn on what the panel can verify from the
task and its in-list children; never fabricate a 'satisfied' claim, and
stay silent on benign transitions that gate on nothing we can check.
The admin status-override capability is preserved (Confirm still fires it);
this only surfaces the bypass instead of letting it happen silently. Does
not touch the master-merge invariant — the board's updateTask is the
operator override, not the Main-PM merge path.
* [F086] prompter: restore parked cell content on project toggle off/on
rebuildCellWork appended a blank {summary:'', items:[]} entry for a newly-
selected cell, so toggling a cell's project OFF then back ON in the MegaTask
review card discarded the agent-authored per-cell summary/items — the entry
was dropped on toggle-off and re-added blank on toggle-on.
Park each draft's last per-cell content in client-only BatchProposal state
(parkedCellWork, keyed by draft index — never sent to the backend; confirm
ships only title/drafts/project_ids/route, and it ride-alongs into the
localStorage persist slice so the restore survives a reload mid-review).
rebuildCellWork gains an optional priorByCell map: a re-added cell with no
live entry restores its parked summary/items (with the new project_id) in-
stead of blanking; a live entry still wins over a stale parked copy so an
in-place edit is never regressed. parkCellWork is the pure merge seam
(prevParked seeds, live work overwrites) the setBatchDraftProjects updater
calls — kept pure so the updater stays a thin caller.
Tests: rebuildCellWork restore/blank-fallback/live-wins + parkCellWork
retain/overwrite/merge (6 new), 19 GREEN. eslint/typecheck/prettier clean.
No wire-payload change, no regression to the fill/drop/one-repo-per-cell
invariants.
* Updated domain
* [F087,F088] enforce panel token on live-chat bridges (Phase 5)
Add a CEO-bound, header-token-only gate (require_panel_token) at the route
level of the prompter_live + secretary_live bridges, which were the only
panel-facing API surface that ran unauthenticated. It mirrors the WS
_require_panel_token and _check_agent_auth_token contracts: in dev
(ROBOCO_AGENT_AUTH_REQUIRED unset) a missing token is allowed; a
presented-but-forged token is rejected even in dev; in prod nginx already
injects the CEO-signed X-Agent-Token on /api/ for GET + POST, so the SSE
stream (EventSource can't set headers) and the POSTs are now checked instead
of anonymous. Applied to start/stream/status/messages/stop on both routers;
preview_live_batch switched from CurrentAgentContext+noqa to the route-level
gate (genuinely auth-only). confirm/confirm-batch/re-interview keep
CurrentAgentContext (they use agent.identity). The container->relay /events
callback is intentionally left ungated (internal Docker network, opaque
session id) — gated by a test sentinel so Option B (spawn+SDK token wiring)
is a deliberate future decision. No panel/nginx/spawn/SDK changes; master
merge invariant untouched. 22 new TDD auth tests, 492 api tests green.
* [F089] honest WorkSession agent_id nullability across the read path
The work_sessions.agent_id column is nullable=True with ondelete=SET
NULL — deleting an agent nulls the FK on every session it ever held. The
ORM annotation lied (Mapped[UUID] non-optional), the converter papered
over the lie (typing_cast to a non-optional UUID), and the response
model rejected None outright (WorkSessionResponse.agent_id: UUID). A
session whose agent had been deleted crashed the GET endpoint with a
pydantic ValidationError instead of serializing agent_id: null.
Make the read path honest end-to-end:
- WorkSessionTable.agent_id: Mapped[UUID | None] (matches the column).
- WorkSessionResponse.agent_id: UUID | None (serializes null, no crash).
- session_to_response passes agent_id via typing_cast('UUID | None', ...)
to bridge SQLAlchemy's UUID[Any] to stdlib uuid.UUID while preserving
None-ness (the cast stays for the same mypy-plugin reason every other
field uses one; it no longer narrows away None).
WorkSessionCreate.agent_id stays UUID — at create time the claiming
agent is always known. The unused WorkSession pydantic read model is
left as-is (never materialized from a DB row). task.py:_needs_revision_dev
already None-guards ws.agent_id via to_python_uuid (returns None -> skip).
* [F090] drop auditor from write_roles on main-pm-board / board-private
The auditor is a silent, read-only observer on every channel, but the
channel catalog (roboco/foundation/policy/communications.py) listed it
in write_roles for main-pm-board and board-private 'for parity' with the
legacy CHANNEL_ACCESS table, while the actual silent-observer rule was
enforced only at the say/dm guard (content_actions._NO_COMMS_ROLES) and
PermissionService.can_write_channel's auditor short-circuit.
That left the catalog-only enforcement path — the HTTP messaging route
(messages.py send_message -> validate_channel_access) — authorizing an
auditor write that both the say/dm guard and PermissionService would
have blocked. A reader of the catalog also believed the auditor could
post to those channels, which is false.
Fix: remove Role.AUDITOR from write_roles on both channels (main-pm
+ board remain writers; ceo remains a writer on board-private). The
auditor stays in read_roles, so its silent read is unchanged. silent_roles
is left empty (matches the announcements precedent: auditor reads via
read_roles, not the silent bucket) — the DB seed and silent_observers
field are untouched.
Logical-regression check: the auditor's read access on both channels
is byte-for-byte preserved (still in read_roles, so validate_channel_access
read returns True via the direct list); the legitimate writers (main-pm,
product-owner, head-marketing, ceo) are untouched; CHANNEL_ACCESS is
derived from the spec so the foundation/seed drift tests self-adjust;
PermissionService.can_write_channel already short-circuited auditor to
False everywhere, so no behavior change there; AUDITOR_SILENT_ACCESS is
unchanged (auditor not added to silent_roles -> no DB silent_observers
change -> no group-access behavior change); the say/dm _NO_COMMS_ROLES
guard is unchanged. Tests: 3 new in test_channel_access.py — auditor
write on main-pm-board/board-private now raises ChannelAccessDeniedError
(RED before: returned True), auditor read still True, main-pm/ceo still
write.
* [F091] warn at spawn time when host grok auth.json is missing
GrokCliProvider._append_grok_auth_mount silently skipped the mount when
the host ~/.grok/auth.json was absent. The spawn still succeeded (docker
run returned 0 — the container was created), so the operator had no
spawn-time signal that the agent was doomed: the entrypoint's
`python -m roboco.llm.providers.grok_auth --check` backstop then
refused to start (exit 78) and the failure only surfaced later via the
container's log markers.
Fix: emit a spawn-time WARNING (module logger) naming the missing file
and the remediation (`grok login` on the host, or set
ROBOCO_HOST_GROK_DIR) when the mount is skipped. The spawn outcome is
unchanged — the container still starts and the existing exit-78 -> park
flow (F041) still catches it — but the operator now sees the missing
credential immediately instead of diagnosing a later exit-78.
Logical-regression check: the mount-present path is byte-for-byte
unchanged (auth.json exists -> the -v bind is appended, no warning); the
spawn still succeeds when auth is absent (no raise — the existing
test_grok_spawn_omits_auth_mount_when_absent still passes: no mount, no
crash); the exit-78 entrypoint backstop and the orchestrator's
exit-78-park handling (F041) are untouched; a module-level logger adds no
side effects. Tests: new test_grok_spawn_warns_when_auth_absent uses
caplog to assert a WARNING mentioning auth.json + `grok login` is
emitted on a missing-credential spawn (RED before: no warning; GREEN
after). 102 grok tests green; ruff/mypy clean.
* [F092] decode JWT exp when refresh omits expires_in
xAI's refresh-token response sometimes omits expires_in. Without it the
new access token kept the stale pre-refresh expires_at, so is_valid /
--check forever rejected a fresh token — and the refresh loop re-rotated
the single-use refresh token every tick, killing the credential (F006).
The access token is a JWT whose exp is the authoritative expiry: decode it
when expires_in is absent. Fallback to the documented ~6h TTL + a structlog
warning when the JWT exp is unreadable, so a fresh token is treated as live
instead of stale.
* [F093] serialize concurrent live-chat spawns under a per-agent lock
The intake and secretary agent ids are each a single fixed id, so two
concurrent start_intake_session / start_secretary_session calls raced on
the container name (docker run --name roboco-agent-<id>) and the
_instances[<id>] write: both passed the reap-prior check before either
registered, both ran docker run, and the last _instances write won,
orphaning the other container + its relay.
Add _intake_spawn_lock / _secretary_spawn_lock (asyncio.Lock) and wrap the
_spawn_intake_container / _spawn_secretary_container bodies so the second
start waits for the first to fully register before its own reap-prior check
runs. Distinct from self._lock (which stop_agent takes) to avoid a
reentrancy deadlock: the spawn body holds the spawn lock then calls
stop_agent (acquires self._lock) — lock order is always spawn_lock ->
self._lock, never the reverse.
* [F094] add a persistent-probe-failure escape hatch to provider parking
_on_probe_failure only incremented the failure counter and, at 10 failures,
sent a one-shot CEO notification. It never cleared the tracker, never gave
up, never fell back to time-expiry. _do_probe returns False for any non-2xx
AND any httpx error, so a permanently unreachable probe endpoint (removed
API key, network partition to the probe host, misconfigured base URL) kept
the provider parked forever — every agent on it gated by
_provider_spawn_parked, their tasks reaped to pending but the spawn gate
queuing every spawn, sitting pending forever. The only recovery was the
operator manually clearing the Redis key.
Past _PROBE_GIVE_UP_THRESHOLD (30) persistent failures, fall back to the
same time-expiry optimism the unprobeable-provider path uses (_do_probe
returns True when there is no probe URL): clear the park and resume parked
agents. If the provider is genuinely still down the real workload attempts
re-park via the 429/5xx path, so this is bounded burn — strictly better
than a silent forever-strand. Kept above the CEO-notify threshold (10) so
the operator still gets the notification first.
* [F095] orchestrator: parked-provider spawn short-circuits before expensive prepare
spawn_agent ran the full _prepare_agent_spawn (writes blueprint/settings/
briefing/MCP files, ensures the image, registers a STARTING instance) every
dispatcher tick only to bail at the after-prepare parked-provider check —
wasting all that file I/O while the provider stayed parked and leaving a
STARTING instance registered then downgraded to OFFLINE.
Move the parked check before _prepare_agent_spawn: resolve the route cheaply
via _resolve_agent_route (only provider_type is needed) and bail with a
minimal unregistered OFFLINE instance. The existing-running check stays
first (inside the lock) so a live agent is never replaced; a TOCTOU
re-check guards the unlocked window before prepare; the after-prepare
check is kept as a rare-race defense (a park landing during prepare).
* [F096] orchestrator: serialize fire-and-forget respawn persists per commit order
_persist_respawn_record is fire-and-forget per gate mutation; a respawn loop
fires count 1->2->3->4 in quick succession, scheduling one persist per
increment for the same (agent_slug, task_id). The ON CONFLICT DO UPDATE upsert
is row-level race-free, but the fire-and-forget tasks can still COMMIT out of
order: a slow stale persist (count=2) scheduled first can resolve AFTER a fast
fresh one (count=4) scheduled second, leaving the durable row at the stale low
count and re-burning the strike threshold on restart.
Fix: acquire self._respawn_persist_lock (new asyncio.Lock) as the FIRST await
in _persist_respawn_record, so acquisition order = task creation order (FIFO
ready queue) = logical schedule order, and commits land in that order. The
durable row always ends at the latest logical value. The lock lives in the bg
task, so the dispatcher hot path never blocks; persists are best-effort and
a slow one queuing the rest just delays the durable catch-up (in-memory record
stays authoritative).
* [F097] orchestrator: back off grok re-park retry_after within a rate-limit episode
_probe_target returns (None, {}) for grok — the grok CLI's xAI endpoint is
closed and the SuperGrok OIDC access token is not a valid bearer for the metered
api.x.ai, so a real probe would either no-op or strand grok parked forever.
_do_probe treats url-is-None as success (time-expiry optimism), so once the
60s retry_after passes the probe loop optimistically clears the grok park, a
cleared park dispatches a fresh grok agent that hits the still-active xAI 429,
exits 75, and re-parks — a flat ~90s crash-retry cycle for the whole xAI
rate-limit window (each cycle costs container startup + a rejected grok call).
Fix: track _grok_repark_count + _grok_last_park_at in _park_grok_rate_limited
and back the re-park retry_after off exponentially within one episode
(60 -> 120 -> 240 -> ... capped at 2**4 = ~16min cycle) so the churn dampens. A
gap past _GROK_REPARK_EPISODE_GAP_S (25min, > the capped cycle) means no re-park
for that long => the rate limit actually lifted => a fresh episode resets the
count to the base 60s, so recovery latency isn't penalized across episodes.
The first park in a fresh episode is unchanged at 60s.
* [F098] orchestrator: keep waiting record through a re-park during probe-success resume
resolve_wait deleted the waiting record (in-memory + durable) BEFORE calling
spawn_agent. A re-park in the window between the probe-success clear and the
spawn — the provider's rate limit lifts then immediately re-limits, or a second
provider limit lands — bails spawn with an OFFLINE instance (the parked-provider
short-circuit). Deleting the record first orphaned the agent: with no record
the probe-resume loop can never revive it and the spawn gate bails every tick,
so the agent is lost until the operator intervenes.
Fix: spawn first, then tear down the record only once a container actually
launched (instance.state == ACTIVE). On an OFFLINE bail the record stays so the
next probe-success re-attempts the resume. On a spawn EXCEPTION the record is
torn down + re-raised so the probe loop doesn't keep re-resuming a task that
moved to a different state (e.g. readiness refused -> task auto-blocked) —
matching the pre-fix behavior where the record was deleted before the spawn.
* [F099] wire pr_pass/pr_fail self_review block in the spec gate
The pr_pass/pr_fail ActionSpecs carry self_review_block=True, but
_gate_preflight never populated Context.original_developer_slug, and
actor_slug was read off agent.slug — which GatewayAgentView does not
carry, so it was always None in production. The block was structurally
dormant: a reviewer who was also the original developer of the
assembled PR could pass (or fail) their own work. The service-layer
_validate_not_self_review backstop only covers qa/documenter, not
pr_reviewer, so the spec gate is the only defense.
Set actor_slug=str(reviewer_agent_id) (GatewayAgentView has no slug,
so the UUID is the identity) and original_developer_slug from the
original_developer marker (a UUID stored as a string). Both resolve to
UUID strings, so the spec's string-equality comparison fires when the
reviewer IS the recorded original developer.
The marker is never set on assembled coordination tasks (only on
dev-leaf tasks at QA/doc claim), so the block stays dormant by design
in production — but the gate is now correctly wired to fire if the
marker were ever set to the reviewer. Zero production behavior change;
the dormant-in-production state is pinned by the no-marker test.
* [F100] atomic Redis probe-failure counter via server-side Lua
increment_probe_failures / reset_probe_failures did a non-atomic
get_state (GET) -> mutate -> set (SET) in Python. A concurrent
activate() re-park writes a FRESH episode blob (probe_failures: 0 +
fresh activated_at / retry_after / affected_agents / kind); if the
stale increment's SET landed after the fresh activate's SET, the stale
blob overwrote the fresh episode metadata AND un-reset the counter
(clobbering the new episode).
Redis single-threads a Lua EVAL, so a server-side read-modify-write
is indivisible: activate's SET is serialized entirely before or after
the script, never interleaved between the script's GET and SET. The
two scripts mutate ONLY probe_failures, so every other episode field
survives the bump. activate stays a single atomic SET (a fresh episode
resetting the counter to 0 is correct semantics).
* [F101] enforce PR-open state gate on gateway open_pr (parity with HTTP path)
* [F102] make project_id mandatory on pr_target (close cross-repo pr_number collision)
* [F103] make project_id mandatory on close_pull_request (close cross-repo collision)
* [F104] fail-closed on conventions resolution errors (block gate no longer silently disabled)
* [F106] compound (timestamp, id) keyset cursor for message pagination
get_messages used strict timestamp inequalities with a non-deterministic
order_by(timestamp.desc()), so equal-timestamp messages were cut by limit
on one page and excluded (strict < T / > T) from the next — they vanished
across pages. Bundled the (timestamp, id) pair into a MessageCursor dataclass
so the next page resumes exactly past the cursor's id at the shared
timestamp (or_: strictly-older OR same-timestamp-smaller-id for before; the
mirror for after), with a deterministic order_by(timestamp.desc(), id.desc())
so the last-item cursor is unambiguous. id is None for a legacy timestamp-
only cursor (strict inequality, prior behavior). The route builds cursors
from the flat before/before_id + after/after_id HTTP params; the schema now
carries the tie-breaker ids. Also clears PLR0913 (cursors replace the
before_id/after_id params).
* [F107] defer Redis bus publish until DB commit (no phantom notifications)
deliver() and _persist_and_deliver() ran inside the caller's open
transaction: the notification row was flushed but not committed, yet
NOTIFICATION_SENT was published to the Redis bus immediately. A commit
failure (DB hiccup, constraint, asyncpg error) rolled the row back while
connected WebSocket clients had already received a push for an id that
no longer existed — a phantom notification (notify_get -> NotFoundError).
Added a deferred-publish (transactional-outbox) helper: defer_bus_publish
enqueues the event on session.info and registers one-shot after_commit /
after_rollback listeners on session.sync_session the first time it is
called for that session. On commit, the after_commit listener schedules
the async drain via asyncio.create_task on the running loop (the listener
fires synchronously inside await AsyncSession.commit, so the loop is
active); the task handles are stashed on the session so callers/tests can
await them. On rollback, after_rollback drops the pending queue — a
rolled-back txn emits nothing. deliver() now builds the per-recipient
events up front (data materialized to strings, so deferral is safe even
if the ORM object later expires) and defers each; the delivered_at DB
marker stays in-tx (rolls back with the row). The bus block stays
best-effort (try/except + log) so a bus-init failure never propagates or
rolls back the notification row — matching the prior inline semantics.
This fixes every deliver/_persist_and_deliver caller at once (the two
cited in F107 plus the orchestrator + task.py deliver sites), since they
all commit the session afterward (the deferred publish fires on that
commit; the row is durable by the time the event goes out).
* [F108] atomic replace_chunks: single-txn delete+insert closes reindex race
* [F109] playbook curation status guards: approve/reject draft-only, archive approved-only
* [F110] draft slug TOCTOU: catch IntegrityError on flush -> ConflictError (no 500)
* [F113] collapse WorkSession creation to the validated service path
_create_work_session_if_needed constructed WorkSessionTable directly,
duplicating WorkSessionService.create's validation (existing-active
check, single-active-per-task supersede, project/task existence). The
two sites had drifted. Route through WorkSessionService.create instead,
mapping ConflictError to the idempotent 'if needed' None. Remove the
now-dead _supersede_other_active_sessions (create's
supersede_active_sessions_for_task replaces it).
Fix three pre-existing RED tests surfaced by the sweep (all confirmed
failing on the F110 commit before this change):
- test_fail_qa_work_session_fallback_excludes_qa_session: inserted two
ACTIVE work_sessions per task, violating uq_work_sessions_one_active
_per_task (migration 047). The QA session is now ABANDONED — still in
the fallback query's result set (the query filters by task_id +
agent_id, not status), so the exclude filter (agent_id != qa_id) is
still exercised and the dev is resolved.
- test_ceo_reject_routes_coordination_task_to_main_pm /
test_ceo_reject_routes_batch_umbrella_to_main_pm: ceo_reject emits an
audit row keyed to CEO_AGENT_ID, but the tests never seeded the CEO
agent row (fk_audit_log_agent_id_agents). Seed the CEO agent (get-or-
create, mirroring test_ceo_reject_writes_handoff_journal).
* [F114] single-claimant guard on pr_gate_claim
pr_gate_claim delegated straight to _qa_or_doc_claim, which overwrites
claimed_by / active_claimant_id with no single-claimant check. Two
reviewers race-claiming the same awaiting_pr_review task would
last-write-wins overwrite the first claim, and the first reviewer's
subsequent pr_pass / pr_fail would actor-mismatch against the new owner
(wasting a review cycle). The orchestrator's gate dispatcher already
prevents double-reviewer-dispatch in normal flow (one task -> one team
-> one reviewer + is_agent_active + per-tick spawned set), so the race
is only reachable via direct concurrent API calls (defense-in-depth).
Add a role-aware single-claimant guard in pr_gate_claim: lock the row
FOR UPDATE (serialize concurrent claims, mirroring the dev claim path),
then refuse only when the task is already actively claimed by a
DIFFERENT PR-reviewer. The gate task is owned by the PM at entry
(submit_for_review does not clear ownership, unlike submit_for_qa), so
the guard must distinguish a PM/dev owner — which the first reviewer
legitimately overclaims — from a competing reviewer claim; checking the
existing claimant's role (pr_reviewer) does exactly that. A re-claim by
the same reviewer is idempotent (skipped by the != check). The gateway
claim_gate_review handler already maps a None return to a clean
invalid_state envelope ('it may already be claimed; give_me_work for
the next'), so no gateway change is needed.
TDD: 3 integration tests in test_task_service_basics.py — reject a second
reviewer race-claim (returns None, first claim intact), allow the first
reviewer when the PM owns the root (regression guard for the
PM-owns-at-entry model), idempotent re-claim by the same reviewer.
Confirmed the reject test RED first (race-claim succeeded, overwriting
reviewer1).
* [F115] sample monorepo per (repo,workflow)/(repo,command) not per repo
The CI-watch and dep-update loaders collapsed a monorepo's cell-projects
to one canonical entry per repo (slug-sorted-first), so a repo whose cells
each carry their OWN ci_watch_workflow / dep_update_command had only the
canonical cell's workflow/command sampled — a red on another cell's
workflow or drift on another cell's lockfile was missed (under-count).
Refactor the shared one-per-repo collapse into _projects_one_per_key, keyed
by repo identity for external-PR discovery (unchanged: one review per PR per
repo), by (repo, effective workflow) for CI-watch, and by (repo, command)
for dep-update. Each distinct workflow/command is now sampled once; the
engines' per-git_url fix-task dedup still prevents duplicate fix tasks for
the same repo. _projects_one_per_repo now delegates to _projects_one_per_key.
key_fn uses a string annotation (Callable lives under TYPE_CHECKING, like
the existing Coroutine/Iterable annotations at lines 4193/5279).
* [R115] originate ci_watch/dep_update fix tasks as PLANNING coordination roots
The Main-PM-code-impossibility guard (commit e202ce39, Thread 4 of this
audit) made team=MAIN_PM + task_type=CODE impossible — a Main PM coordinates,
it does not write code. But the ci_watch and dep_update engines still
originated their fix tasks as task_type=TaskType.CODE assigned to main-pm,
so task_svc.create raised MAIN_PM_NO_CODE and NO fix task was ever opened
— a regression introduced by the earlier audit fix (confirmed: the engine
tests pass at e202ce39~1 and fail at HEAD).
Mirror the hardened self_heal_engine precedent (self_heal_engine.py:197)
which already uses task_type=TaskType.PLANNING for its Main-PM coordination
root with an explicit 'decompose the fix and delegate the code work to a
cell dev — the Main PM does not write the fix itself' description. Both
engines now originate PLANNING coordination roots with matching delegation
guidance in the description + acceptance criteria. confirmed_by_human
stays True for both (they ride the normal delivery flow without the CEO
gate, unlike self-heal — intentional per the architecture).
The dedupe/open-cap queries (list_open_ci_watch_tasks /
list_open_dep_update_tasks) key on source + non-terminal status + git_url,
NOT task_type, so the type change does not break dedup (still one open fix
task per repo).
The two source-test fixtures (test_ci_watch_source / test_dep_update_source)
created CODE+MAIN_PM tasks directly to exercise the listing queries — same
guard violation; switched to PLANNING (the queries assert on source/status,
not task_type, so the fixture type matches the engines' corrected type).
* [F116] hold the read-clone lock across the dep-probe local clone
dry_upgrade_changes_lockfile called ensure_read_clone (which syncs the
read clone under the _meta-conventions lock then releases it) and ran
'git clone --local --no-hardlinks <read_clone>' OUTSIDE the lock. A
concurrent ensure_read_clone -> _sync_read_clone (fetch + hard-reset to
origin's default branch) could mutate the read clone's working tree /
object db mid-clone, racing the clone and producing an inconsistent or
failing probe.
Split _probe_lockfile_change into _clone_local_into (the local clone,
run under the read-clone lock) + _probe_lockfile_on_clone (the upgrade +
git status, run without the lock on the now-independent copy). The probe
acquires _ensure_lock_for(slug, '_meta-conventions') — the same lock
ensure_read_clone syncs under — and holds it only for the clone step; the
upgrade operates on the full --no-hardlinks copy and never touches the
read clone, so the lock is released before it to avoid blocking
conventions reads for the upgrade duration.
The tiny gap between ensure_read_clone releasing the lock and the probe
re-acquiring it is safe: any concurrent _sync_read_clone completes under
the lock before the probe acquires, so the clone reads a stable state.
* [F117] stop the orchestrator in lifespan shutdown BEFORE closing the DB
The lifespan shutdown closed OptimalService + the DB, and only THEN did
bootstrap's finally block call orchestrator.stop() — so stop() ran with
the DB already closed. stop() drains fire-and-forget _bg_tasks writes
(respawn_tracker upserts, audit-log rows) and stop_agent finalizes work
sessions / agent state, all needing the DB still open; closing it first
silently dropped those final writes (the durable PM-respawn counter's
last few strikes, the metrics-bearing audit trail tail).
Move orchestrator.stop() into the lifespan shutdown path, BEFORE
close_optimal_service + close_db, guarded by a new get_orchestrator_or_none()
safe accessor (no crash when no orchestrator is wired — tests,
skip_orchestrator). bootstrap's finally-block stop() becomes an idempotent
safety net: stop() gains a _stopped flag (getattr-guarded so __new__-
constructed test instances still stop) so the double-call is a clean no-op,
not a re-stop of already-stopped agents / re-drain of an empty bg set.
* [F118] coerce a lone-string where_to_look into a list
where_to_look is a list-typed handoff field like consequences/next_steps
but was the only one NOT in the _wrap_scalar_in_list field_validator. A
well-intentioned where_to_look='src/api/' 422'd at the route with no
remediation envelope, and the agent's retry loop tripped the do-server
circuit breaker — the exact failure mode the other list fields were
hardened against. Add it to the mode='before' validator so a lone string
is wrapped into a one-element list before type coercion.
* [F119] sender reaps dead sockets on send error instead of waiting for receive idle timeout
* [F120] release a stopped agent's claimed task immediately on budget-kill/shutdown
* [F122] name the already-open PR in submit_up's None-state remediate
submit_up's create_pr pre-side-effect opens the cell→root PR BEFORE
submit_for_review runs (its pr_created gate requires it — lifecycle.py:1338-1343).
When submit_for_review returns None (a concurrent state change raced the task
out of in_progress between the precondition gate and the composed action), the
old remediate ('check task state — must be in_progress with PR ready') hid
that the PR was already open on GitHub — an orphaned external artifact the PM
could not reconcile. Mirror submit_root's F016 None-envelope remediate: name
the open PR, point the PM at re-fetch + reconcile, and note create_pr is
idempotent so a re-issue re-attaches to the existing PR (no duplicate). Pure
message improvement — zero behavior change; reordering is off the table
(create_pr must precede the pr_created gate).
* [F124] re-check dependency state before releasing a dependency-blocked claim
The unmet_dependency guard read dependency state via an unlocked SELECT, then
fired release_dependency_blocked_claim (a state mutation: claimed/in_progress
-> pending, clears branch_name, abandons WorkSession) as a side-effect BEFORE
returning the rejection. An upstream dependency that reached a terminal state
(completed/cancelled) in the microseconds between the read and the release left
the task NEEDLESSLY released — its branch cleared + WorkSession abandoned +
assignee bounced, only to be re-dispatched + re-claimed when the dependency-
completion re-dispatch fired a moment later.
Re-check unmet_dependency_ids immediately before the release and skip it
(returning None — proceed) when the upstream just completed. Dependencies are
monotonic (unmet -> met only; terminal states never reopen), so a fresh read
that now finds them met stays met: safe to proceed without releasing. The
'still unmet' path is byte-for-byte the prior behavior (no regression). The
cross-task residual window (upstream completes between the re-check and the
release) is not closable by a row lock on the dependent, but the re-check
narrows the window from [first read -> release] to [re-check -> release], and
in the common case the first read already sees met (no guard fires). No
committed-work loss either way (a dependency-blocked task has none; the branch
ref + commits persist across the branch_name clear).
* [F125] serialize same-parent delegate via per-parent advisory lock
The delegate sibling-dedup guard read the parent's existing subtasks via an
unlocked get_subtasks SELECT (the dedup read) then created the subtask (the
write) with no DB serialization between them. Two concurrent delegate calls
for the same parent (PM re-delegating while a reaper re-dispatches, or two
orchestrator ticks racing) each read a duplicate-free sibling set, each passed
the dedup guard, and each created a subtask — the parent got the duplicate the
guard exists to prevent (the smoke-run runaway pattern).
Fix: a PostgreSQL transaction-scoped advisory lock keyed by the parent task
id (seed 1, disjoint from the per-agent claim lock's seed 0), acquired at the
top of the delegate body before the first get_subtasks read (the briefing
context read AND the dedup sibling read) and held through create_subtask's
flush + the outer request commit. The second concurrent same-parent delegate
blocks until the first commits, then its dedup read sees the committed
sibling and is rejected.
Per-PARENT (not per-agent): a coordinator PM legitimately delegates many
subtasks under one parent in quick succession and plans many roots in
parallel — a per-agent lock would serialize all of a PM's delegates and
regress the PM coordinator concurrency feature. The per-parent lock
serializes only same-parent delegates (the dedup invariant is per-parent)
and leaves different parents untouched.
TDD: red-first ordering test (lock acquired before first get_subtasks read
and before create_subtask) + no-regression test (create still runs).
* [F127] per-task advisory lock prevents open_pr milestone double-emit
open_pr's idempotent re-entry guard (pr_number is not None) read t.pr_number
from an unlocked fetch. Two concurrent same-task open_pr calls (the
alive-but-unresponsive respawn race) both fetched pr_number=None, both passed
the guard, both ran the runner (GitHub 422 ensures one PR), and both reached
_record_milestone_progress -> a double-emitted 70% 'opened PR #N' entry.
Fix: acquire_task_lock (pg_advisory_xact_lock, seed 2) before the fetch, held
through the runner + milestone + request commit. The second concurrent call
blocks until the first commits, then its fetch sees the committed pr_number
and the idempotent guard short-circuits without re-emitting. Per-task (single-
active-task guard means same-task concurrent open_pr is only the bug case).
* [F128] require active claim on explicit-task content posts
_verify_explicit_task_ownership checked assigned_to, which is stale
across a reap/handoff (persists until reassignment; active_claimant_id is
cleared on release). A reaped agent could keep posting say/dm/note to its
former task. Add the active-claimant check when assigned_to == caller;
assigned_to=None keep its existing allow (read-side inspection between
reassignments uses evidence, which has its own ownership path).
Existing 'active owner' test mocks passed assigned_to=agent_id without
active_claimant_id; production sets both together on claim, so the mocks
were incomplete. Updated to set both — realistic, not a behavior change.
* [F129,F130] harden quality gate _run_one exit status + timeout cleanup
F129: _run_one returned 'proc.returncode or 0', masking a None returncode
(communicate returned without a recorded exit code — process killed
out-of-band) as 0 / success. Treat None as a non-zero failure (fail-closed).
F130: on timeout, _run_one killed the subprocess but never awaited wait()
— communicate() was cancelled so it never closed the stdout/stderr pipes,
leaving a transient zombie + leaked FDs. Await wait() after kill() to reap
the process and close the transports.
* [F132] timeout the conventions validator + reap on hang
_run_conventions_validator awaited proc.communicate() with no timeout —
a hung subprocess (tree-sitter deadlock, huge repo) hung the
i_am_done/pr_pass gate forever and orphaned the python subprocess on
orchestrator restart. Wrap communicate() in wait_for(120s); on timeout
kill+wait the proc and fail closed (could_not_run=True → block gate
refuses the submit), matching the validator's own fail-loud philosophy.
* [F135] re-check activity before sweeper closes a session (TOCTOU)
sweep_timed_out_sessions read last_activity_at once at the candidate
SELECT, then closed. A message landing in that window refreshed
last_activity_at in the DB, but the sweeper closed on its stale in-memory
value — closing a just-used session. Re-read last_activity_at fresh right
before the close and skip if the session is no longer timed out.
* [F136] cancel startup indexing task on OptimalService.close()
close() cancelled only the periodic update task, then cleared the plugins.
The startup _indexing_task (background auto-index, slow Ollama / large repo)
could still be mid-flight at shutdown and write against closed/cleared
plugins. Cancel and await _indexing_task FIRST (its tail starts the periodic
task, so ordering also prevents a late periodic spawn), then the periodic
task, then clear plugins.
* [F139] scope active_task_owns_branch to the polled project
active_task_owns_branch did an unscoped WHERE branch_name = ? — a cross-project
branch_name collision (UUID-derived 8-char prefixes, theoretical) made the
internal-PR reviewer skip the WRONG project's PR (project A's leftover PR
skipped because project B happened to have an active task with the same
branch). Pass project_id (in scope at the orchestrator call site) and add
TaskTable.project_id == project_id to the WHERE. Correct for single-project
tasks and MegaTask multi-repo batches alike: each root-subtask carries its own
project_id matching its own repo, so a branch on project A's repo is owned
only by a task whose project_id == A.
* [sweep] strip Fxxx audit-ID tokens + trim bloated comments/docstrings + add behavior-change docs
Post-audit sweep over the 135 audit-fix commits since 19a474d3:
1. Stripped every # Fxxx: audit-ID token from comments AND every Fxxx token
from docstring openings across 211 blocks / ~626 lines. The CEO flagged
these twice: audit-issue IDs in code confuse future devs/agents. The
descriptive text is preserved; only the Fxxx token is removed (and bloated
narrative blocks trimmed to 1-3 lines keeping the one non-obvious invariant).
2. Trimmed bloated comments/docstrings to the concise standard (1-3 lines).
3. Added missing behavior-change docs for the audit-fix batch: prompts/roles
(documenter, pr_reviewer, qa), user-facing docs (api auth, websockets,
agent-gateway, megatask, merge-model, task-lifecycle, grok, resilience,
conventions, panel, security, troubleshooting), and the RAG corpus (cell-pm,
main-pm, pr-reviewer, qa roles; conventions; messaging-tools; escalation;
megatask; task-claiming workflows).
Comment/docstring/prose ONLY — zero code-line edits (verified: the diff
contains no def/class/return/if/for/await/assignment/call lines). Gates green:
ruff format + ruff check clean, mypy clean on roboco/. The only pytest failures
are the pre-existing sync_branch tracing-decision gap (B1, 250be5c2) — not
sweep-caused and tracked separately.
* [fix] register sync_branch in VERBS_WITHOUT_TRACING
sync_branch (B1, 250be5c2) is a git-only rebase+force-push verb (composes=(),
no DB transition, side_effects=()) but was never registered in the tracing
parity tables, so test_every_intent_verb_has_a_tracing_decision failed.
Mirrors open_pr: a mechanical git op with inline preconditions (ownership),
no journal/plan rationale required.
* chore(release): 0.14.0
* [fix] resolve 16 mypy errors across 9 test files (make quality gate)
type-clean the test files so make quality (mypy roboco/ tests/) is green:
- Any-typed locals for the two TypeError-asserting scoping tests (bypass
the required-arg check without getattr/ruff B009)
- Any-typed view for the shutdown-drain _drain_bg_tasks override (bypass
mypy method-assign without setattr/ruff B010)
- cast("uuid.UUID", ...) / cast("UUID", ...) for SQLAlchemy UUID[Any]
returns (TC006-quoted), config=None for AgentInstance stubs, None-narrowed
await_args, Iterator return on a yielding fixture, UUID annotation on the
_task helper. No type:ignore / noqa.
* [docs] regenerate lifecycle artifacts for sync_branch + branch-keyed submit_root gate
The committed artifacts were stale: lifecycle.py grew the sync_branch verb
(B1) and the branch-keyed submit_root gate description (B2/B3) but the
generated markdown/json were never regenerated. make foundation-check
enforces artifact==generator(lifecycle.py); regenerating restores that.
No source change — pure generator output.
* [refactor] reduce xenon C-rank blocks to A (behavior-preserving)
Extract helpers / flatten conditionals in 11 blocks that rated C(11)+
under xenon --max-absolute B, dropping pr_gate.py module rank B->A in
the process. Pure move-and-call refactors: each extracted helper holds
the original logic verbatim and the caller delegates to it; no control
flow, return values, or side effects changed.
Sites: validators._extract_strs, sequencing.dev_task_collision_edges,
evidence_builder.build_task_handoff, intake_driver._coerce_draft,
task.claim_task_for_agent (2 guards), prompter.create_task_from_draft
(validate+assignee), pr_gate._gate_decision (3 helpers),
orchestrator._handle_stopped_container + _reap_with_service,
_impl._create_subtask_from_inputs + complete.
_impl helper returns tuple[TaskNature, list[str]] to preserve mypy
narrowing of acceptance_criteria at the TaskCreateRequest site.
Also fix vulture: rename unused __aexit__ param tb->_tb in
test_conventions_cache_put.py (was hidden while xenon short-circuited
the gate).
* [security] bash-guard uv run --active deny + CodeQL path-traversal fixes
Fix 1 (be-dev-1 brick prevention): bash-guard now denies 'uv run --active'
and 'uv run'/'uvx' against /app targets. In the agent container
VIRTUAL_ENV=/app/.venv is baked globally, so 'uv run --active' always
resolves onto the image-baked MCP-gateway venv and uv rebuilds it,
deleting /app/.venv/bin and bricking every MCP server spawn. Bare
'uv run' (workspace .venv, cwd-relative) is untouched.
CodeQL fixes:
- docs.py: replace bypassable '..' substring guard with a
resolve-and-contain helper (_resolve_contained_path). An absolute
path made pathlib reset (base / '/etc/passwd' == '/etc/passwd'),
letting read_doc/delete_doc reach arbitrary files. Applied to both
sinks.
- orchestrator.py: _safe_agent_path_segment at the spawn_agent
chokepoint (rejects traversal-shaped agent_id before any fs op) and
inside _remove_container (slug guard before the log-dir mkdir,
defense-in-depth).
- agent_sdk/server.py: /usage/sync transcript_path now resolved and
contained under ROBOCO_TRANSCRIPT_DIR with a .jsonl suffix requirement
(was Path(raw) — unauthenticated endpoint could stat arbitrary files).
TDD RED->GREEN across all four; make quality green (4890 passed).
* [fix] enum-parity gate: drop false-green mask, skip empty/unmigrated DB
The foundation-check gate ran the enum verifier behind
`|| echo "(skipped — postgres unreachable)"`, which swallows ANY
non-zero exit — including real drift — and prints 'All quality gates
passed'. On a host with a dockerized but empty/unmigrated `roboco` DB
(0 tables: the agentrole/team enum types don't exist), the verifier
connected, found every foundation value 'missing', exited 1, and the
mask relabeled it 'skipped' → false-green.
Fix:
- scripts/verify_postgres_enums.py: move skip semantics INTO the script.
Distinguish unreachable (skip, exit 0), DB-not-migrated/both-enum-types-
absent (skip, exit 0), real drift (exit 1), match (exit 0). Extract
pure enum_drift + should_skip_for_unmigrated helpers + a type_exists
probe so an empty DB is 'no migrated target', not drift.
- Makefile: drop the `|| echo` mask — real drift now fails the gate.
TDD RED->GREEN (10 tests); make quality green (10906 passed).
* [security] docs path guard: reject '.'/empty segments for clean 400
_resolve_contained_path used an '..' substring ban, which (a) left rel='.'
passing the guard — read_doc/delete_doc then got the base DIRECTORY itself
and raised IsADirectoryError (500) instead of a clean ValidationError, and
(b) false-rejected legit filenames containing '..' like 'v1..v2.md'.
Replace the substring ban with a raw-segment check (rel.split('/')) that
rejects any '.', '..', or empty segment. Path(rel).parts was the wrong tool
— pathlib collapses '.' and empty segments on 3.13, hiding them. The split
check catches '.' / 'a/./b' / 'a//b' / '..' / 'a/../b' while allowing
'v1..v2.md' ('..' inside a filename, no bad segment). The post-resolve
parents-containment check (the real defense) is unchanged.
TDD RED->GREEN (4 new tests); make quality green (10910 passed).
Follow-up to the CodeQL path-traversal review: the two CodeQL 'High' alerts
on this guard are false-positives-on-the-fix (resolve-and-contain already
contains the bypass); this hardening closes the one genuine low residual
(rel='.' -> 500) the review surfaced, which CodeQL did not flag.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* feat(orchestrator): add respawn_tracker table + migration 051
Durable backing for AgentOrchestrator._pm_respawn_tracker (the PM-respawn
loop breaker). Kept only in memory it reset to count=1 on every restart,
re-burning the strike threshold against a still-wedged task. RespawnTrackerTable
mirrors WaitingRecordTable: composite PK (agent_slug, task_id) matching the
in-memory key; task_id is intentionally NOT a FK (the startup loader validates
against live tasks so a stale counter can't resurrect). Migration 051 verified
with a real alembic upgrade head + downgrade -1 + re-upgrade on Postgres.
* feat(orchestrator): persist the PM-respawn counter across restarts
The PM-respawn loop breaker (_pm_respawn_tracker) lived only in memory, so an
orchestrator restart reset a wedged task's strike count to 1 and re-burned the
whole threshold (4 spawns x container cost) before the gate fired again.
Write-through each gate mutation to the respawn_tracker table via a
fire-and-forget _schedule_respawn_persist (on the existing _bg_tasks strong-ref
set; a DB hiccup degrades to in-memory-only, never gates/un-gates a spawn), and
restore_respawn_tracker() repopulates the counter at startup, validating each
row against live tasks (drops terminal/missing) so a stale counter can't
resurrect against a fixed task. Best-effort + inert when the table is empty.
Cannot manufacture a spawn — the counter only ever suppresses one.
(_instances reconcile, the spec's other goal, already shipped as
_readopt_running_agents.)
* fix(types): cast Mapped[UUID] columns in project routes + self_heal
A clean `mypy roboco/ tests/` run surfaces 7 pre-existing errors in files this
branch doesn't touch: project-route handlers and self_heal_engine pass a
ProjectTable.id (declared Mapped[UUID] against SQLAlchemy's dialect UUID, so
mypy infers sqlalchemy.sql.sqltypes.UUID[Any]) where a uuid.UUID is expected.
An incremental .mypy_cache had hidden them. Apply the same targeted cast unblock
used for the prior batch; the deeper fix (migrating the ~88 Mapped[UUID]
columns to Mapped[uuid.UUID]) remains a separate dedicated task.
* docs(orchestrator): document respawn_tracker durability
Add the orchestrator runtime-state durability note to CLAUDE.md (respawn_tracker
write-through + restore; _instances reconciled-from-Docker) + the migration-051
narrative, and a CHANGELOG [Unreleased] Fixed entry. Also type-clean the new
respawn_tracker table test (cast __table__ to Table under TYPE_CHECKING).
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* [57f83a44] fix(lint): remove unused imports from autonomous-maintenance code [CI run 28194267886]
The Feat/autonomous-maintenance (#264) merge introduced 4 ruff lint errors
that broke the quality gate:
F401 roboco/api/routes/project.py:7 unused `cast` import
F401 roboco/services/self_heal_engine.py:28 unused `cast` import
F401 roboco/services/self_heal_engine.py:46 unused `UUID` in TYPE_CHECKING
TC003 roboco/services/telemetry/source.py:18 `Sequence` not in TYPE_CHECKING
Root cause: automated maintenance PR added self_heal_engine.py and
ci_watch_engine.py with imports that became orphaned when the implementation
was refactored. `cast` was imported in both project.py and self_heal_engine.py
but never called. `UUID` was placed in self_heal_engine.py's TYPE_CHECKING
block but not referenced in any annotation. `Sequence` in telemetry/source.py
was imported at module level when it is only used in function-signature
annotations and therefore belongs in TYPE_CHECKING (the file has
`from __future__ import annotations` so this is runtime-safe).
The mypy type-narrowing issue in test_pr_gate_records_verdict.py (the original
AC context: in-body None assignment making subsequent assertions unreachable,
resolved via annotation-typed class attributes) was already fixed in a prior
commit before this task was opened.
Fix: remove the three unused imports; move Sequence into TYPE_CHECKING.
No suppressions, no xfail markers, no coverage threshold changes.
`ROBOCO_ENCRYPTION_KEY='...' make quality` exits 0: ruff format, ruff check,
markdown prose, mypy (0 errors, 819 files), pytest (10197 passed, 95.51%
coverage), xenon, radon mi, vulture, bandit, pip-audit, deptry, alembic
--sql, import-linter, and all foundation drift checks.
* [57f83a44] docs(changelog): document ruff lint fixes from autonomous-maintenance PR
Added comprehensive entry to CHANGELOG documenting the 4 ruff lint errors
(F401 unused imports, TC003 import placement) that were introduced by
Feat/autonomous-maintenance (#264) and subsequently fixed. Documents root
cause (orphaned imports from refactoring) and the TC003 best practice
(type-annotation-only imports belong in TYPE_CHECKING block with
`from __future__ import annotations` for runtime safety).
All quality gates pass: 10197 tests at 95.51% coverage, zero suppressions.
---------
Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>
Co-authored-by: Backend Documenter <be-doc@agents.roboco.dev>
post_pr_review (inbound external/fork PR review) derived both the recorded
notes_structured.pr_review.verdict AND the posted GitHub review event solely
from its `event` argument, which defaults to REQUEST_CHANGES — and, unlike the
in-path gate's pr_fail, it never required any findings. A reviewer that
concluded 'approve' in the summary but left event at the default filed (and
posted to the contributor's PR) a blocking 'changes requested' with nothing
cited, contradicting the approving summary the CEO saw on the PR Reviewer Notes
card.
Enforce a verdict<->findings invariant before any record or GitHub post:
- REQUEST_CHANGES must cite >=1 finding (almost always a forgotten
event='APPROVE'), mirroring pr_fail's 'at least one issue' rule;
- APPROVE may not carry a blocker/major finding.
The check is the pure policy fn pr_review_conflict() wired through the new
choreographer _verdict_consistency_gate, rejected with a clear remediate hint.
Steer the agent at the source too: the flow MCP tool + request schema now spell
out the invariant and to pass event='APPROVE' explicitly for a clean PR.
Renn F