Tiny race: dispatcher decided to spawn for closure between agent's
heartbeat and idle-pause. Spawn would land against an already-paused
parent. Gate spawn on (status != PAUSED OR last_heartbeat older than
cutoff).
State existed in the lifecycle table and the enum but no verb, route,
or service path ever set it. Removing dead state. If we need
problem-task isolation later we'll add it explicitly with a verb.
Two PMs completing different subtasks of the same parent could race
on gh API merge calls. GitHub returns 409 to one but local DB write
ordering wasn't guaranteed. Take row-level lock on parent task before
merge; retry once on 409.
Pre-gateway PMs/Board sent formal notifications (require ack); gateway
had say/dm only. Now PMs and Board can issue ack-required notifications
via NotificationService through the standard envelope path.
I1: _fetch_origin_best_effort runs as root and writes new pack files
+ ref updates that land root-owned, undoing _ensure_agent_owned that
ran before. Subsequent spawns hit Permission denied. Mirror the
fetch_branch_for_inspection pattern: re-run _ensure_agent_owned AFTER
the fetch.
I2: separate workspace_refresh_fetch_timeout_seconds (default 60s)
from workspace_clone_timeout (300s). Refresh transfers small deltas;
300s of blocking on every spawn against a hung remote is operationally
bad. 60s is enough for any sane refresh.
ensure_workspace short-circuited when the clone existed, so a
respawned PM/Doc could be reviewing arbitrarily stale diffs. Add
a best-effort 'git fetch origin' on every entry; checkout
unchanged.
X-Correlation-ID was bound to structlog but lost on the MCP→API hop.
MCP shims now forward it; Envelope carries it back to the agent;
audit_log row records it for forensic joins.
Task 13's has_recent_tracing_gap query used .astext on a generic JSON
column, which raises AttributeError at runtime. The choreographer's
exception handler swallowed it, leaving the strike-count reset
permanently inert in production. Migrate the column to JSONB (which
supports .astext + GIN indexing for future audit queries), update the
ORM, and add a real-DB integration test that would have caught this.
_pm_respawn_should_gate counted PARENT_NOT_CLAIMED rejections as
no-progress and killed PMs after 3 strikes — even when the new prompts
told them to call i_will_plan first. Reset counter when last response
was a tracing_gap (rule-following retry, not stuck).
ROBOCO_GATEWAY_ENABLED defaulted to False, leaving trigger_filter's
spawn cooldown / role-rate logic dormant. Flip to true and add the
gateway_triggers table migration if missing. Without this, respawn
rate has no server-side limit besides _pm_respawn_should_gate.
When a PM's i_will_plan and a child dev's spawn fire in the same tick,
the dev sometimes saw branch_name=None because the PM's transaction
hadn't committed yet. Auto-block fired and the dev sat blocked until
the next 30s tick. Add 3x250ms retries when parent is mid-claim.
I2: resume_for_agent did its own _validate_and_set_status call,
skipping resume()'s structlog 'Task resumed' event and the fire-and-
forget RAG lifecycle-event indexing. Gateway-driven resumes were
invisible to logs and the RAG corpus. Delegate to resume() after
the gateway-specific ownership pre-checks (mirrors pause_for_agent's
pattern).
I1: documented why Choreographer.unclaim does NOT call _touch on
its OK path - assigned_to is cleared, no claimant heartbeat to
refresh. Prevents future readers from 'fixing' the asymmetry.
i_am_idle auto-pauses owned in_progress tasks; lifecycle table allows
paused -> in_progress; no verb implemented it. Adds resume so an agent
respawned for a paused task can continue. Routes through
_validate_and_set_status mirroring the unclaim pattern (Task 9 fix).
unclaim_for_agent did direct attribute assignment, bypassing the
single point of truth for status transitions. Choreographer's pre-check
made the runtime correct today, but a future change to VALID_TRANSITIONS
or ROLE_RESTRICTED_TRANSITIONS would silently miss this path. Route
through _validate_and_set_status; assigned_to is still cleared
explicitly as the unclaim's specific side effect.
Choreographer remediate strings already pointed agents at unclaim;
the verb didn't exist. Now it does — claimed/in_progress -> pending,
clears assigned_to, available to dev/qa/doc/cell_pm/main_pm.
ContentActions.commit was stripping user-supplied prefixes but never
re-adding the canonical one. Dev prompt promised auto-prefix; code
delivered nothing. Now every gateway commit lands with [task-id-short].
Format choice: simple [task_id[:8]] (8-char), matching the dev prompt
("Auto-prefixes [task-id]") and CLAUDE.md's documented commit format.
The legacy templates/git/commit.py uses the richer
[root_short:task_short] for the commit_for_task API path with full
traceability metadata; the gateway commit path is intentionally simpler
and stays aligned with the prompt-level promise.
Previous regex used a greedy [^|]* and required at least one / before
the host, so 'curl roboco-orchestrator:8000/api' (no scheme, no slash)
slipped through. Split into two simpler checks: (a) line starts with
curl/wget/http/https/httpie, AND (b) line contains a forbidden host.
Probe-verified: scheme-ful, scheme-less, and protocol-relative forms
all denied; external URLs still allowed; GitHub-specific deny still
fires first.
Prompts told agents internal API calls were denied; the guard only
denied GitHub. Combined with task 4 (X-Agent-Role enforcement) this
closes the manifest-bypass loophole.
Choreographer takes an audit dep but never used it. Now every rejection
Envelope (invalid_state, not_authorized, tracing_gap, not_found) writes a
gateway.rejected row with verb + reason + missing fields. Forensic
signal for stuck flows; no behaviour change for happy path.
Adds AuditService.log_event() — generic free-form event_type write so
the gateway doesn't have to extend AuditEventType for every new surface.
Audit writes are best-effort: a failure logs a warning but never
propagates, since the agent's response Envelope is the contract.
I1: post_to_channel raised no error if get_agent_slug returned None
(unknown/deleted agent), and send_message's 'if agent_slug' skipped
the validate_channel_access call. Same fail-open class the prior
commit was fixing, just narrower window. Now post_to_channel raises
ChannelAccessDeniedError directly when the slug lookup fails — say()
already converts that to a clean not_authorized Envelope.
I2: 'writable channels for your role' wording was misleading because
get_agent_channels resolves by slug, not role. Replaced with
'channels you may write to' for clarity.
post_to_channel was calling send_message without agent_slug, which
disabled the validate_channel_access check. Forward the slug; convert
ChannelAccessDeniedError to a friendly not_authorized Envelope with
the agent's writable-channel list (pre-gateway behaviour).
Route layer now rejects 403 if the role doesn't match the router's
allowed set. Choreographer still re-checks role per verb where needed,
but defense in depth means a future verb that forgets the role check
doesn't leak. Auditor router also gated.
Gate E made i_am_done strict (requires pr_number set), but the only
verb that opened PRs was i_am_done_with_catchup which lives off the
dev manifest. Devs hit NO_PR with no escape. Adds submit_for_qa as
the explicit push+PR step, leaving i_am_done to do the strict submit.
i_will_work_on, i_have_committed, i_am_done, i_am_blocked, i_will_plan,
and pm_give_me_work now call task.heartbeat() so the reaper has fresh
data. Closes the loop opened by tasks 2a + 2b.
I1: claim_heartbeat_ttl_seconds (300s) overlapped semantically with the
pre-existing claim_stale_seconds (180s). Between 180-300s of silence,
trigger_filter queued duplicate spawns while the reaper hadn't yet
released the claim — exactly the dispatcher churn the reaper was
supposed to close. Collapse to one field (claim_stale_seconds, 180s);
reaper now consumes the same setting trigger_filter uses, so both
agree on 'stale' on the same tick and the reaper runs first.
I2: _task_svc injection slot on AgentOrchestrator.__init__ was
production-dead (always None) and only used by __new__-based test
instances. Drop the __init__ slot + the production branch in
_reap_stale_claims that read it. Tests still pre-bind on __new__
instances; the attribute exists per-instance, not per-class.
Dispatch loop now releases tasks whose holder has gone silent past
ROBOCO_CLAIM_HEARTBEAT_TTL_SECONDS (default 300s). Closes the
'dead container squats task forever' failure mode that the schema
hinted at but no code enforced.
Foundation for stale-claim recovery. Schema column existed since
migration 006; no writer until now. Idempotent UPDATE — no select
roundtrip — so callable from any hot verb without extra DB cost.
Service signatures are (task_id, agent_id, ...) but choreographer was
calling (agent_id, task_id). Production claim path silently returned
None; unit tests pinned the buggy order so the bug was invisible. Swap
all 7 call sites and update test assertions. Add a regression pin
that locks in the correct order.
cell_pm_complete, main_pm_complete, and submit_up already had subtask-
terminality gates inherited from the pre-gateway closure check at
roboco/services/task.py. This commit verifies the gate is preserved
and improves the remediation hint to actually NAME the non-terminal
subtasks instead of telling the PM to "call triage()".
The improvement uses a new private helper
``Choreographer._non_terminal_subtask_ids`` that queries get_subtasks
and filters to non-terminal statuses, returning a comma-separated
list of "<id> (<status>)" pairs. The PM now sees exactly which
subtasks are blocking the parent's completion.
Pre-gateway reference: roboco/services/task.py closure check
(documented in PRE_GATEWAY_LIFECYCLE.md §4.3).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
i_am_done now strictly enforces the four pre-gateway field-level gates
restored from roboco/api/routes/tasks.py:903-940 at commit 254cc93:
- NOT_SELF_VERIFIED: task.self_verified must be true.
- NO_COMMITS: task.commits must be non-empty.
- NO_PR: task.pr_number must be set.
- NO_PROGRESS: task.progress_updates must have at least one entry.
Each missing field surfaces as a tracing_gap with the matching pre-
gateway error code in the missing list, and a remediation hint that
tells the dev exactly what to do.
The previous silent-catch-up behavior is preserved as a separate
opt-in verb i_am_done_with_catchup. Existing tests that asserted the
catch-up behavior have been migrated to the new verb.
This fixes the failure mode where a dev could call i_am_done with no
commits and the gateway would silently try to push nothing, open an
empty PR, etc. — now the dev sees an explicit error.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a caller passes an explicit task_id to commit / note / say / dm /
evidence, ContentActions now verifies task.assigned_to == caller_agent_id
before allowing the side effect. Auto-fill from get_active_task_for_agent
is implicitly self-owned and does not need a re-check.
evidence() additionally allows assigned_to=None (post-handoff transient
state) so QA / documenter can inspect tasks between reassignments.
Pre-gateway, agents could not even see tasks they didn't own because the
MCP handlers resolved task from session context. The gateway exposes
task_id parameters across multiple verbs, so the explicit ownership
gate is required.
Exception: say() and dm() with NO task_id are exempt — used for channel
announcements and off-task A2A. The strict guard only applies when the
agent supplies a task_id parameter.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Choreographer.i_am_idle now refuses with INVALID_STATE when the caller
has any pending (assigned but never claimed) task. Pre-gateway this
was implicit because the orchestrator's auto-respawn would re-spawn
the agent for the assignment, leading to a tight respawn loop. The
explicit refusal lets the agent fix the state via i_will_work_on
(dev/qa/doc) or i_will_plan (pm) first.
Existing auto-pause for in_progress tasks is preserved (Gate Set C
spec calls this out as still required) — it runs AFTER the pending
guard, so an agent with a mix of pending+in_progress is told about
the pending task first instead of silently pausing in_progress and
then looping on the pending one.
Pre-gateway reference: roboco/runtime/orchestrator.py
auto-respawn loop guards (already preserved at HEAD); the explicit
agent-facing gate is new.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PARENT_NOT_CLAIMED: Choreographer.delegate now enforces that the parent
task is in_progress AND assigned to the calling PM before allowing
subtask creation. Pre-gateway this was implicit (orchestrator only
spawned PMs after they claimed their parent); the gateway exposes
delegate as a first-class verb so the gate must be explicit.
SUBTASK_CAP: hard-blocks delegation when the parent already has 12
subtasks. Pre-gateway never had this cap because PMs naturally never
created more than a handful per spawn cycle; with delegate as a verb
agents can loop, so a cap is needed.
The _delegate_guard helper was split into _delegate_role_guards,
_delegate_static_guards, and _delegate_lifecycle_guards to keep each
piece below the PLR0911 return-count threshold and make the layered
gating explicit.
Pre-gateway reference: implicit in roboco/runtime/orchestrator.py
spawn flow; restored here as explicit server-side enforcement.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ports five pre-gateway predicates that were dropped when the gateway
displaced the MCP claim handler. Predicates restored from
roboco/mcp/tasks/handlers/_helpers.py:124-204 and
roboco/mcp/tasks/handlers/claim.py:121-180 at commit 254cc93:
- SEQUENCE_ORDER_VIOLATION: a sibling task with sequence < N must be
in completed/cancelled before sibling N can be claimed.
- ALREADY_ACTIVE: agent cannot claim while owning an in_progress /
claimed / verifying task other than the one being resumed.
- PAUSED_TASKS_EXIST: agent cannot claim while paused tasks exist.
- PM_CANNOT_EXECUTE_CODE: cell_pm/main_pm cannot claim task_type=code.
- ROLE_TYPED_CLAIM: developer claim is restricted to
code/research/design; qa/documenter must use claim_review /
claim_doc_task.
All five guards run inside Choreographer._run_claim_guards before
i_will_work_on / i_will_plan / claim_review / claim_doc_task mutate
state. Skip flags isolate guards that don't apply to a verb (e.g.,
PM-code skipped on QA verb, role-typed skipped on PM verb).
The guards live in roboco/services/gateway/claim_guards.py so
choreographer.py stays focused on orchestration.
Existing tests updated to provide the new mock primings; the
permissive auto-mock behavior they relied on no longer applies.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each role file now has a six-section structure (Identity / Inputs / Your
verbs / Workflow / Anti-patterns / When the gateway returns an error).
The Identity section gives a hard role prior with explicit negative space
("you do NOT write code", "you do NOT merge"), recovering the framing
that was lost in the gateway slim-down. The Anti-patterns section names
the gateway error codes the role will hit if they step out
(PARENT_NOT_CLAIMED, SUBTASK_CAP, PM_CANNOT_EXECUTE_CODE, NO_COMMITS,
NO_PR, NOT_SELF_VERIFIED, etc.), so agents expect the system to catch
them. base.md now centralizes the envelope contract, ground rules, and
channel-slug convention so per-role files stay focused.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* choreographer.escalate_up: AttributeError when target lookup returned
None. Switched from role-based escalate_up_to_role (which mishandled
slug-shaped escalation_target like "main-pm" because AgentRole only
accepts underscore form) to slug-based task.escalate, with explicit
None-handling that returns invalid_state instead of 500.
* prompts/roles/cell_pm.md + main_pm.md: enumerate the new lifecycle
verbs (i_will_plan, delegate, submit_up, give_me_work, i_am_idle).
Without this, PM agents fell back to calling i_will_work_on (the dev
verb) and 404'd at /api/v2/flow/cell_pm/i_will_work_on. Workflow
walkthroughs included.
* messaging.get_channel_by_slug + get_or_create_channel_by_slug: strip
leading "#" so "#main-pm-board" resolves to the row stored as
"main-pm-board". Agents follow Slack convention; gateway must accept
it.
* agent_sdk session-end post-mortem hook: corrected payload shape from
{content, kind:"reflect"} to {type:"task_reflection", title, content}
so /api/journals/me/entries validates. Added pad-to-min-length so the
50-char content gate doesn't reject thin post-mortems.
* test_choreographer_pm: updated escalate_up test to assert task.escalate
is awaited, plus regression test for the None-target invalid_state path.
380 unit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* api/schemas/v2/flow.py: IWillPlanRequest, DelegateRequest,
SubmitUpRequest with min_length=1 validators where appropriate.
* api/routes/v2/flow_cell_pm.py: give_me_work routes to
pm_give_me_work; new endpoints i_will_plan, delegate, submit_up.
* api/routes/v2/flow_main_pm.py: new endpoints give_me_work,
i_will_plan, delegate.
* mcp/flow_server.py: Python wrappers for i_will_plan, delegate,
submit_up registered in _TOOLS so manifest-scoped agents can call
them.
* tests/unit/gateway/test_choreographer_pm_extras.py: 22 tests
covering happy + reject paths for each new verb plus i_am_idle's
auto-pause behavior.
* tests/unit/api/routes/v2/test_flow_cell_pm.py +
test_flow_main_pm.py: route-level tests for the new endpoints.
Test count: 352 → 381 (+29). make quality-fast green.
Adds the missing gateway verbs that PMs need to drive their parent task
through the lifecycle. Pre-Phase-4, PMs could only triage/unblock/
complete/escalate/idle — they had no way to claim+start their parent
task, no way to create subtasks (told to curl raw), no way to bubble
cell-level PRs up to the Main PM. The lifecycle stalled at pending.
This commit lands the choreographer + service layer:
* TaskService gains: list_in_progress_for_agent, pause_for_agent,
submit_pm_review (gateway alias of submit_for_pm_review),
create_subtask (TaskCreateRequest-based, infers status from
assigned_to), and main_pm_agent.
* Choreographer gains:
- i_will_plan(task_id, plan): claim + set_plan + start for PMs.
Mirrors i_will_work_on, scoped to cell_pm/main_pm.
- delegate(parent_task_id, DelegateInputs): create a subtask with
delegation-chain validation (Main PM → cell PMs; Cell PM → its
own team's devs). Resolves slug → UUID via AGENT_UUIDS.
- submit_up(task_id, notes): cell PM bubbles a finished cell-scope
task up to Main PM. Opens a cell-level PR into the parent (Main
PM) branch, transitions to awaiting_pm_review, reassigns to
Main PM. Tracing gates: notes>=20, journal:decision, all
subtasks terminal, branch present.
- pm_give_me_work: returns the PM's first assigned task in any
active status (not just dev-active states).
- i_am_idle: now auto-pauses every in_progress task this agent
owns before marking idle. Restores the pre-Phase-4 behavior the
closure dispatcher relies on.
* role_config: cell_pm gains give_me_work + i_will_plan + delegate
+ submit_up; main_pm gains give_me_work + i_will_plan + delegate.
5 spawn prompt builders (_build_dev_prompt + _get_workflow_instructions,
_build_qa_prompt, _build_doc_prompt, _build_pm_review_prompt,
_build_pm_closure_prompt) plus the recently-rewritten triage prompts
referenced MCP tools deleted in Phase 4 T9. Replace with gateway verbs
(give_me_work, i_will_work_on, claim_review, claim_doc_task,
i_will_plan, delegate, complete, submit_up, etc). Drops the curl-POST
recipe in favor of the new delegate verb. Surfaced live during NAS
smoke.
content_actions.note/dm/say/commit/evidence called write_entry/send/
post_to_channel/git.commit/git.diff(base=)/fetch_branch_for_inspection,
none of which existed on JournalService/A2AService/MessagingService/
GitService/WorkspaceService. Live smoke threw AttributeError on every
content tool. Add the matching gateway-shaped adapters on each service
(scope-string -> JournalEntryType for note; channel-by-slug -> default
group -> active session for say; UUID-or-slug recipient resolution for
dm; branch-name commit + diff(base=) for commit/evidence; project-aware
fetch_branch_for_inspection on workspace). Surfaced live.
Both _build_main_pm_triage_prompt and _build_pm_triage_prompt referenced MCP tools deleted in Phase 4 T9 (roboco_task_get/plan/start/create/activate/journal_decision/agent_idle). Main_pm + cell_pm spawned with these and tried the dead tools — minimax-m2.7 then improvised, ending up implementing the task itself instead of breaking it down.
New prompts: gateway verbs (evidence, note, say, dm, complete, escalate_*, i_am_idle) for transitions/journal/comms; explicit POST /api/tasks curl recipe for breakdown + delegation (gateway has no create_subtask verb yet). Both prompts now reinforce: PMs do not implement, do not run git, must delegate. Main PM hands to Cell PMs; Cell PMs hand to devs.
Surfaced live during NAS smoke.
_classify_code_task routed every default-complexity (medium) code task to cell_pm even when the description named no coordination work. The PM then re-delegated back to a developer, adding a useless hop and producing the smoke-test pattern where main_pm/cell_pm tried to do every lifecycle stage themselves.
Drop the complexity==medium → cell_pm branch. Cell PM now lights up only when the description carries an actual coordination keyword (coordinate, integration, cross-team, sync, planning, milestone, dependencies, review). High/critical complexity, cross-cell keywords, missing team, and team=all still route to main_pm. Adds 11 unit tests; full suite 363 passing.
Surfaced live during NAS smoke.
Adds:
- TaskService.reassign unit tests (set, clear, missing-task)
- New test_choreographer_reassignment.py covering:
i_am_done -> qa
pass_review -> documenter
i_documented -> cell_pm
main_pm_complete -> None (CEO via UI)
escalate_to_ceo (board) -> None
cell_pm_complete -> walks up to parent and reassigns when all
siblings terminal, skips otherwise
fail_review -> does NOT issue an explicit reassign (qa_fail
already restores the original developer via
quick_context)
The orchestrator polls per-agent for actionable tasks. Without
updating assigned_to as the lifecycle hands a task to the next role,
the orchestrator kept respawning the previous-stage agent (often
main-pm) for every stage, hitting role-permission rejections and
looping. Each Choreographer transition now writes the new assignee
alongside the existing A2A notification.
- Adds TaskService.reassign(task_id, new_assignee | None)
- _notify_qa: dev → qa
- pass_review: qa → documenter
- i_documented: doc → cell_pm
- main_pm_complete + escalate_to_ceo: clear assignment (CEO acts via UI)
- cell_pm_complete: walks up to parent; if all subtasks terminal,
hands the parent off to the cell PM for that team via a new
_maybe_advance_parent_to_pm_review helper
fail_review still leans on qa_fail's existing original_developer
recovery via quick_context, and unblock_with_restore restores
pre_block_assignee — both already correct, no changes needed.
10 stale remediation hints across api/routes/tasks.py + exceptions.py pointed at MCP tools deleted in Phase 4 T9 (roboco_task_start, roboco_task_qa_pass, roboco_task_qa_fail, roboco_task_progress, roboco_task_unblock, roboco_task_submit_verification, roboco_task_submit_qa, roboco_task_complete, roboco_task_claim, roboco_task_activate). Each now mentions both the gateway verb (i_will_work_on, pass, fail, complete, unblock, etc.) and the panel REST equivalent. Surfaced live during NAS smoke.
Gateway v2 endpoints declare X-Agent-ID as Annotated[UUID, Header(...)]. The MCP servers (flow_server, do_server) read os.environ['ROBOCO_AGENT_ID'] verbatim and put it in the header. We were exporting the slug ('main-pm', 'be-dev-1', ...) so every gateway call from a containerized agent 422'd on header UUID parse. Resolve to UUID via AGENT_UUIDS lookup in seeds/initial_data.py. Surfaced by the new 422 logger (de54c3b) during NAS smoke.
FastAPI's default RequestValidationError returns details to the client but nothing to server logs. Smoke test hit a 422 on /api/v2/flow/main_pm/complete with no way to tell which field failed. Add a handler that logs path/method/body/errors on every 422 so the next failure is debuggable in one log scan.
Live smoke runs showed agents reaching for `Bash curl /api/...` and
`Bash git ...` even though the slim role prompts named the gateway
verbs, and main_pm took ownership of an implementation task and tried
to commit code from the PM seat.
- base.md and every role prompt: explicit ground rule that direct
curl-to-orchestrator and raw-git invocations are forbidden — every
commit/push/PR/transition/journal/comms call goes through the gateway
verbs. The deny-list line in base.md now also covers curl/wget to the
orchestrator's /api/... (was GitHub-only before).
- main_pm.md and cell_pm.md: explicit "you do not implement tasks
yourself" rule. Implementation belongs to developers; PMs delegate.
Renn F