The purpose-based dedup suppressed a same-purpose (same sender/type/task,
overlapping recipients) notification while a prior one was unacked. For
informational types (KNOWLEDGE_SHARE / MENTION / A2A_REQUEST / BROADCAST +
the pickup-proves-receipt triad) each send carries DISTINCT content (a new
learning, a new mention) and acking is voluntary, so a recipient who never
acks the prior one let the dedup permanently suppress every subsequent
same-sender broadcast - silent learning-broadcast data loss.
The dedup's anti-loop rationale (stop unacked-set inflation soft-blocking
i_am_idle) only holds for action-required signals. Gate the dedup on
ACK_REQUIRED_BY_TYPE.get(type, True): action-required types still dedup,
informational types always create. Unmapped types default True (dedup on).
TDD red->green; ruff + mypy clean; notification + dedup suites green (20).
NotificationService._create_notification built NotificationTable without
requires_ack, so the column default (True) applied to EVERY notification -
including informational REVIEW_REQUEST / DOCUMENTATION_REQUEST /
A2A_REQUEST / KNOWLEDGE_SHARE (ACK_REQUIRED_BY_TYPE -> False) and every
@mention from MessagingService._notify_mentions. Each false ack-required
inflated the recipient's unacked set and soft-blocked i_am_idle into
respawn churn.
- _create_notification: requires_ack=ACK_REQUIRED_BY_TYPE.get(type, True)
(unmapped types default True - preserve the action-required bias).
- _notify_mentions: requires_ack=False explicit (MENTION is informational).
TDD red->green (identity is False/is True assertions - the mocked
flush doesn't apply SQLA's insert-time default, so pre-fix the attribute
was None); ruff + mypy clean; notification suite green (18 passed).
The pr_fail a2a steer to the owning PM is fire-and-forget; a PM respawned
into needs_revision later read none of it (build_task_handoff never looked
at notes_structured), saw a generic 'needs revision' with zero concrete
change-requests, and re-submitted the same PR (the 2026-06-27 infinite
pr_fail loop on 9980d0a0 / PR #138). The signal-gap was only partially
closed by the a2a.
build_task_handoff now extracts notes_structured.pr_review
(verdict/summary/issues/head_sha — the slot pr_fail authors on every fail)
into a pr_review field on the handoff, so every PM briefing for the task
carries the concrete change-requests. A prior pr_fail alone now counts as
prior-work-worth-resuming. Type-guarded + capped; absent => no key (no
misleading empty slot).
TDD red->green; ruff + mypy clean; evidence_builder suite green (14 passed).
The root loop-stopper (F016) was root-only; a weak cell PM could re-submit
the unchanged cell->root PR after a pr_fail and loop awaiting_pr_review ->
pr_fail forever (the cell analogue of the 2026-06-27 root loop).
pr_fail stamps the assembled PR's head SHA into notes_structured.pr_review
.head_sha for cell AND root gate tasks alike (the capture is gate-verb-
level, not root-level), so the same structural refusal applies to submit_up:
if the cell PR's current head SHA equals the SHA the last pr_fail recorded,
no new dev work landed on the cell branch -> the diff is byte-identical ->
refuse, do not re-open the gate. Different SHA -> branch advanced -> allow.
- _submit_up_unchanged_pr_guard mirrors _submit_root_unchanged_pr_guard
(cell-PM remediation: re-delegate to the dev + wait for re-assembly),
wired into submit_up after _submit_up_guard passes.
- Renamed shared _current_root_pr_head_sha -> _current_pr_head_sha (both
guards use it; the lookup was never root-specific).
- Every ambiguous case FAILS OPEN (no prior fail, no recorded sha, no
pr_number, no resolvable project, git/closed-PR None) — only the exact-
unchanged case is hard-blocked.
TDD red->green; ruff + mypy clean; F007+F016 guard suites green (15 passed).
Both verbs compose a single atomic action whose None return (the verb's
own result) flowed out of run_intent and was dereferenced as t.status,
HTTP 500-ing with no actionable rejection:
- F016 submit_root: submit_for_review returns None when the root->master
PR was already opened / the task raced out of in_progress. Post-runner
None-guard extracted into _submit_root_finalize -> invalid_state
(re-fetch; if awaiting_pr_review the PR is open, wait for reviewer;
else re-delegate fixes and retry) instead of None.status.
- F017 i_am_blocked: escalate returns None in four cases (no task, no
agent, no resolvable escalation-target slug, no target agent row) e.g.
a developer whose role has no PM above it. _run_i_am_blocked_intent
now guards updated is None -> (t, invalid_state rejection) with
remediation (re-fetch + escalate to CEO directly / retry) instead of
the caller deref'ing None.status -> 500 + respawn-loop.
TDD red->green; ruff + mypy clean; gateway suite green (58 passed).
F005: the single-file bind mount of auth.json pinned the inode, so the
orchestrator's atomic refresh (tmp+rename within ~/.grok) never reached a
running grok container — a long-lived container hung at the login prompt
when the original ~6h token expired. Mount the host ~/.grok DIRECTORY (ro)
at /home/agent/.grok-auth-ro; the entrypoint symlinks ~/.grok/auth.json at
that RO mount so grok + the --check backstop read the live credential (the
directory mount sees the host-side rename) while grok's writable state
(config.toml, sessions/) stays in the image's ~/.grok.
F006: a rotated refresh_token is single-use — xAI invalidates the old one
the instant it issues the new one. If the atomic write failed after the
rotation, the file kept the now-dead old refresh_token and the credential
was permanently lost on the next refresh. _atomic_write now falls back to a
direct write when tmp+replace fails, so the rotated token always lands on
disk (losing the write is catastrophic; losing atomicity is not).
TDD: RED tests watched fail, then GREEN. ruff+mypy clean; 32 grok tests
green, no regressions.
F003/F014: /api/v1/do/* only required X-Agent-ID (UUID) — no token check,
unlike the flow routers' role guards. A forged X-Agent-ID passed. Added
require_any_authenticated_agent (token-only; do router serves all roles)
and applied it as a router-level dependency. Binds X-Agent-ID to a verified
HMAC token when ROBOCO_AGENT_AUTH_REQUIRED=true; rejects a forged token
even in dev mode.
F004: /ws/* per-agent streams (channels/agents/sessions/notifications)
never read the nginx-injected X-Agent-Token, so in strict mode an agent on
the Docker network could subscribe to another agent's notifications with
no auth. Added _require_panel_token verifying the CEO panel token against
the CEO identity; wired into all four per-agent streams (system stream
stays operator-only per its docstring). Same strict/dev contract.
TDD: RED tests watched fail (no gate -> 200/accept), then GREEN. ruff+mypy
clean; 399 api/mcp + 29 WS tests green, no regressions.
_activate_batch_root_subtasks flipped a held root-subtask to team=MAIN_PM
but left task_type=code (intake only coerces main_pm-team drafts, so a
board-routed code root reached activation still code-typed). The
main_pm+code combo re-introduces the 2026-06-27 meltdown. Mirror
approve_and_start's own retype via main_pm_cannot_own_code so the
activated child is a planning-typed coordination root.
TDD: RED test_activate_batch_root_subtasks_retypes_code_to_planning
watched fail (task_type stayed CODE), then GREEN after the retype.
ruff+mypy clean; 125 batch/umbrella/approve tests green, no regressions.
Defense-in-depth for the spawn_agent human-role chokepoint (d31d6719).
The chokepoint structurally guarantees no CEO/prompter/secretary container
can ever spawn — every dispatcher goes through spawn_agent. But two
dispatchers resolve an arbitrary assigned_to and spawn it with only a
None/unknown-role filter, so a human-assigned task would reach the
chokepoint and RAISE: caught by the per-dispatcher try/except, but it
aborts that dispatcher's whole tick (stalling other respawns behind the
mis-assigned task) and error-logs every cycle. The other dispatchers are
already safe by whitelist/hardcoded slug (blocker_resolver_slug returns
None for non-PM/non-BOARD; escalation/approval use whitelists; marketing
and audit hardcode their non-human slug).
- _claimed_task_needs_agent: return None for a CEO/prompter/secretary
assignee — no container to respawn, and do NOT release a human-owned
task to pending (that would re-route it to a PM). Leave it for the human.
- _dispatch_pm_review_work (assigned branch): skip a human-only assignee
so a CEO-assigned awaiting_pm_review task neither spawns nor aborts the
dispatcher's tick.
Audited all target-iterating dispatchers; only these two lacked a filter.
Regression tests cover both skips.
A live 2026-06-27 incident saw a CEO agent container spawned. Root cause:
_dispatch_a2a_work iterates every A2A/notification target and spawns it
with no human-role filter, and _is_agent_active('ceo') is always false
(the CEO is never a container), so the 'skip if active' check could never
protect the CEO. Any CEO-addressed notification (board handoff, escalation)
launched a CEO container — the system acting as the human CEO: a trust
violation. The CEO is the human operator; intake (prompter) and secretary
are human-driven chats launched through their own dedicated guarded paths
(_spawn_intake_container / _spawn_secretary_container), never spawn_agent.
Fix: a single chokepoint guard at the top of spawn_agent refuses
Role.CEO / PROMPTER / SECRETARY (raises AgentReadinessError + logs). This
structurally covers every dispatcher present and future, since they all go
through spawn_agent. Plus a defense-in-depth skip in _dispatch_a2a_work so
a human-role target never even calls in (avoids error-log spam; the
notification stays for the human to read in the panel).
Safe: the dedicated human-spawn paths do not route through spawn_agent.
Regression tests: spawn_agent refuses ceo/intake-1/secretary-1, does NOT
refuse a real agent; _dispatch_a2a_work skips CEO/intake/secretary targets
and still spawns real-agent + mixed-target cases.
A sibling's PR merging into the parent branch while a dev worked leaves the
dev's branch behind its base — the assembled PR then can't merge cleanly and
the sibling's changes go missing (the 2026-06-27 out-of-order dev-task break).
The behind-base gate refuses i_am_done in that state and steers the dev to
sync_branch (the Phase B1 gate-level rebase verb).
- GitService.is_behind_base: rev-list --left-right --count across
origin/{base}...origin/{head} → (behind, ahead); fetch-first so origin
reflects the pushed head. Raises on git failure (consistent with
rebase_onto_base); malformed stdout degrades to (0,0).
- Choreographer._behind_base_gate: wired into _i_am_done_gate after
_ensure_branch_pushed. behind>0 → invalid_state remediate→sync_branch.
Fail-open on git/base-resolution error (flaky fetch can't strand a task at
the submit gate — the merge layer has its own behind checks). Skipped for
branchless roots and protected bases (master/main/-prefixed).
Tests: gate (6: refuse+steer/up-to-date/branchless/protected/fail-open-base/
fail-open-git), is_behind_base (6: parse/up-to-date/malformed/argv-form/
requires-branch/missing-project). ruff + mypy roboco/ tests/ clean; unit green.
Raw shell git is denied to agents (Bash(git:*) base deny), so a developer
whose branch fell behind its base had no gate-level rebase — only the
CEO/PM-only /rebase HTTP route. sync_branch is the dev verb that wraps the
rebase through the gate (traced + evidenced), so the 'everything goes through
the gates' invariant holds.
- lifecycle: IntentSpec sync_branch (dev-only, ownership-gated, composes=(),
git-only — no DB transition); _next_hint_synced helper.
- GitService.sync_task_branch: rebase task.branch_name onto its resolved base
via rebase_onto_base (fetch + rebase + force-with-lease push).
- Choreographer.sync_branch + _sync_branch_preflight_rejection: not_found /
unknown-role / spec-gate / no-branch / protected-base guards, then the git
op; conflicts abort (no force-push) and steer to resolve-by-hand; git failure
steers to i_am_blocked.
- HTTP route /api/v1/flow/developer/sync_branch + SyncBranchRequest schema.
- MCP tool sync_branch(task_id) + _TOOLS registration (manifest auto-propagates
via intents_for_role(Role.DEVELOPER)).
Tests: intent spec (5), choreographer handler (8: happy/conflicts/not_found/
not_authorized/no-branch/protected-base/git-failure/audit), route (1), MCP (1).
ruff + mypy roboco/ tests/ clean; unit suite green (DB-fixture errors env-only).
Kind 2 (cell-task wave chain): a new cell-task under root-subtask UT_n
depends on every cell-task under every root-subtask in UT_n.dependency_ids
(the kind-1 wave-chain edges), so its branch carries the previous wave's
merged cell work. Re-derived from the root-subtask's deps, not the cell-task's
own dependency_ids (which also carry UX/product-fanout edges the by-osmosis
edge must not pick up). A root may fan to several cell-tasks (different cells),
so the previous wave's cell-task is a SET.
Kind 4 (by-osmosis): the first dev task (sequence 0) under a cell-task depends
on each predecessor cell-task's tail (max-sequence) dev task, so the new wave's
first branch carries the previous wave's fully-merged tail. Subsequent dev
tasks inherit the tail via kind 3 or the merged base.
Both wired from _create_subtask_from_inputs, dispatched on parent.team
(MAIN_PM -> kind 2; cell team -> kind 4). Pure helpers
(cell_task_wave_chain_depends_on, by_osmosis_tail_dev_tasks) unit-tested in
test_sequencing.py; TaskService methods integration-tested. Idempotent +
best-effort throughout (add_dependency dedupes; missing predecessors are
no-ops). Also fixes a latent mypy-tests gap (estimated_complexity required on
direct TaskCreateRequest calls in the S2 tests).
Pure dev_task_collision_edges in sequencing.py turns a parent's surfaced
siblings into (depends_on_id, task_id) pairs via SequencingService. TaskService.
wire_sibling_collision_dag wires them through add_dependency (idempotent). The
choreographer calls it after each dev-task delegate so the sibling collision DAG
is built incrementally as the cell PM decomposes — file-overlap serializes,
migration chains, shared-last; stable (priority, sequence) ordering keeps edges
from flipping into reverse cycles on re-runs.
The cell/main PM's delegate verb now carries the dev-task collision
surface (intends_to_touch / adds_migration / touches_shared) and an
explicit depends_on override through DelegateRequest -> DelegateInputs
-> _create_subtask_from_inputs -> create_subtask, and create_subtask
forwards sequence / dependency_ids / batch_id / surfaces into the
prepared TaskCreateRequest instead of dropping them (the base create
already persists them at task.py:878-884).
This is the plumbing for the multi-level sequencing model edge kind 3
(dev-task collision DAG). Previously a dev task delegated with a
collision surface or an explicit dependency lost it before persistence
— dependency_ids was always [], so the only dev-task ordering was the
weak assignee-keyed spawn barrier (the live 2026-06-27 out-of-order
break: 40842957 started before 9b3682b8's PR merged). Phase S2 runs
SequencingService over the surfaced siblings and wires the DAG via
add_dependency.
A dev task in needs_revision must go back to the developer, never the
pool. The pool path let a cell PM re-claim the revision (PMs can claim
needs_revision) — the live 2026-06-27 'needs revision on a dev task sent to
the cell PM' bug.
fail_qa's original_developer marker is the fast path, but it is
unreliable in practice (live observation: never persisted), so the
unassign else-branch was the load-bearing path and it dropped the task
into the pool. Add a work-session fallback (_resolve_revision_dev) that
resolves the developer who actually worked the task — the most recent
work session whose agent is a developer, the QA's own session excluded
— and reassigns to that dev instead of unassigning. Only unassign when
no developer ever touched the task. Self-heals the marker so a
subsequent re-fail takes the fast path and the QA-review index
attributes the work correctly.
The branch tip had 15 mypy tests/ errors in files this bundle did not author,
which blocked CI's make quality mypy step (mypy roboco/ tests/) regardless of
the bundle's own commits. Pre-existing is still existing — fix every one:
- test_schemas_v1_flow.py (8): the StrList coercion tests intentionally pass
SDK-nested list-of-strings input ([[['...']]], {'item':{'$text':'...'}}, int,
dict). Annotate those literals as list[Any] locals so mypy accepts the
coerce-able shape; the StrList BeforeValidator still flattens to list[str] at
runtime. No type:ignore.
- test_pr_gate_records_verdict.py (3): notes_structured is dict|None; narrow
with 'assert t.notes_structured is not None' before indexing (the existing
pattern at line 90).
- test_pr_review_hand_format_guard.py (1 site, 2 errors): the _verb_runner()
spy assertion — use the cc: Any = c alias idiom so assert_not_awaited
resolves; drops the now-unused type:ignore[union-attr].
- test_pr_gate_notifies_pm.py (1): drop the unused type:ignore[method-assign]
on the a2a.send reassignment.
- test_content_models.py (1): narrow coerced with isinstance(coerced,
PrReviewContent) before reading .issues (the base _Content lacks the field).
Gates: rm -rf .mypy_cache && mypy roboco/ tests/ = Success (855 files);
ruff check + format clean; 5 affected suites = 40 passed.
The 2026-06-27 infinite pr_fail loop: a Main-PM root (PR #139) was pr_fail'd,
routed to needs_revision, and re-submitted byte-identical → awaiting_pr_review
→ pr_fail again, forever. The prior hint/a2a steer was ignored by the weak
coordinator model — hints don't stop a model that won't read them. A HARD gate
refuses the re-submit when the assembled root PR's head SHA is unchanged since
the last pr_fail (no new cell work → identical diff); a different SHA ⇒ the
branch advanced ⇒ allow. Every ambiguous case fails open (no prior fail, no
recorded SHA, no pr_number, unresolvable slug, git error, closed PR) — only the
exact-unchanged case is hard-blocked.
- content/models: PrReviewContent.head_sha (optional; JSON col → no migration).
- git: get_pr_head_sha (GitHub pulls API; None on any failure → fail-open).
- pr_gate: pr_fail captures head_sha into the verdict record; pr_pass does not.
- _impl: submit_root runs _submit_root_unchanged_pr_guard after _submit_up_guard;
_current_root_pr_head_sha resolves slug + current SHA (fail-open).
- pr_review: extract module-level resolve_task_project_slug, shared by the mixin
and the gate helper (_LegacyChoreographer reaches it via cast to the
ChoreographerHelpers typed view — it doesn't inherit the helpers mixin).
- tests: test_submit_root_unchanged_pr_guard (11 — refuse/allow/6 fail-open/3
capture-side, mypy-clean via cc:Any spy idiom, zero type:ignore) +
test_pr_gate_notifies_pm capture-path stub.
Fix (two layers):
1. Root cause — pr_merge and rebase_pr_for_task now take a required project_id and scope the lookup where(pr_number == X AND project_id == Y). Required so no caller can forget — the bug class can't recur. All 4 call sites updated (choreographer cell_pm_complete, the rebase-retry, the superseded close_pull_request now passes project_id, and _verb_runner._do_pr_merge).
2. Crash guard — _finalize_cell_complete None-checks the complete() return and returns a clean invalid_state envelope (with a remediate hint) instead of dereffing None → 500 → respawn loop.
Two real bugs surfaced running the full gate against a containerized
Postgres (and the orchestrator boot log):
1. Migration 052 crashed a real orchestrator boot with
'type "team" already exists'. The generic sa.Enum(create_type=False)
does NOT set the postgres enum's create_type attribute, so op.create_table
(checkfirst=False) emitted a redundant CREATE TYPE against the pre-existing
team enum. Switched to postgresql.ENUM(create_type=False) — the postgres-
native enum whose create_type _check_for_name_in_memos actually reads, so
the CREATE TYPE is suppressed. Verified: 051->052 upgrade against a DB where
the team enum pre-existed (the exact path that crashed) now succeeds;
downgrade 052->051 drops the table and preserves the shared enum; fresh
upgrade head clean. (Migration 016 has the same latent sa.Enum pattern but
never re-runs in prod, so it's noted, not touched here.)
2. _ensure_branch_for_task read task.cell_projects (lazy=selectin to-many)
directly, tripping MissingGreenlet on a freshly-created/unqueried task —
which then poisoned the async session (PendingRollbackError). Replaced with
_task_has_cell_map: peeks InstanceState.unloaded (no IO) and reads the
already-loaded map, falling back to an awaited count query only when the
relationship is genuinely unloaded. Non-ORM stubs route to the plain
attribute. Fixes 2 integration tests; the 6 cell-map unit tests still pass.
Also: typed the self stub as Any in test_choreographer_subtask_project
(mypy tests/ wants Choreographer, not SimpleNamespace) — the codebase idiom.
Gate: ruff format/check clean; mypy roboco/ + tests/ clean; full pytest
10371 passed / 388 skipped against containerized pgvector:pg16; vulture clean.
Pre-existing xenon C-rank on reassign (from prior commit 19a474d3, not this
feature) still blocks make quality — surfaced separately.
A MegaTask root-subtask can now target an ad-hoc per-cell project map — a
third targeting shape that mirrors the existing product fan-out root. In
RoboCo a project is per-cell (ProjectTable.assigned_cell); a monorepo is N
per-cell projects sharing one git_url. So 'multi-cell' IS 'multi-project',
and a task may mix per-cell projects across products or include OSS-library
projects not in any product.
Storage: migration 052 adds task_cell_projects (mirrors product_projects;
unique per (task, team)). TaskTable gains a cascade-delete cell_projects
relationship; TaskCreateRequest / TaskCreate / Task response carry the map.
Policy: batch.is_branchless_coordination + is_valid_batch_shape gain a
has_cell_projects param — a root-subtask targets exactly one of project /
product / cell-map; the umbrella still targets none. TaskService passes
has_cell_projects at every predicate call site and persists the rows in
create(). _ensure_branch_for_task cuts feature/main_pm/{root} per distinct
project in the map (via _distinct_projects_for_task); _require_target_or_umbrella
and _validate_batch_membership accept the map shape.
Fan-out: every distinct_project_ids site (task.py branch creation, routes
_project_for_complete + _resolve_project_for_merge, orchestrator
_ambient_projects_for_task, pr_review._project_slug_for, git._project_for_task)
generalizes to first-distinct-project-of-map-or-product. Choreographer
_resolve_subtask_project resolves a delegated subtask's cell from the parent's
cell map. The product-scoped _slugs_for_product intake helper is unchanged.
Intake: prompter._draft_cell_map extracts the per-cell map from the_work[].
_validate_batch_scope counts distinct projects across all drafts' cells
(>=2 min stays; one 2-cell draft satisfies it). create_task_from_draft
persists cell_projects for >=2-cell drafts (project_id/product_id None),
collapses a 1-cell map to the single-project shape, and leaves single-cell
top-level project_id drafts unchanged. _resolve_owning_team routes a
multi-cell map to Main PM (coordination root, like a product root — a cell
PM can't delegate cross-cell). propose_draft/propose_batch tool descriptions
declare the per-cell project_id (both Claude SDK + grok runtimes).
The umbrella stays branchless / pure-coordination / submit_root-rejected;
the CEO-escalation pr_number gate is not widened (the map root is
is_umbrella=False, mirroring a product root, so submit_root supplies it).
Single-cell root-subtasks and everything below them are byte-for-byte
unchanged. Un-run MegaTask waves (multi-cell drafts) become runnable.
* feat(orchestrator): add respawn_tracker table + migration 051
Durable backing for AgentOrchestrator._pm_respawn_tracker (the PM-respawn
loop breaker). Kept only in memory it reset to count=1 on every restart,
re-burning the strike threshold against a still-wedged task. RespawnTrackerTable
mirrors WaitingRecordTable: composite PK (agent_slug, task_id) matching the
in-memory key; task_id is intentionally NOT a FK (the startup loader validates
against live tasks so a stale counter can't resurrect). Migration 051 verified
with a real alembic upgrade head + downgrade -1 + re-upgrade on Postgres.
* feat(orchestrator): persist the PM-respawn counter across restarts
The PM-respawn loop breaker (_pm_respawn_tracker) lived only in memory, so an
orchestrator restart reset a wedged task's strike count to 1 and re-burned the
whole threshold (4 spawns x container cost) before the gate fired again.
Write-through each gate mutation to the respawn_tracker table via a
fire-and-forget _schedule_respawn_persist (on the existing _bg_tasks strong-ref
set; a DB hiccup degrades to in-memory-only, never gates/un-gates a spawn), and
restore_respawn_tracker() repopulates the counter at startup, validating each
row against live tasks (drops terminal/missing) so a stale counter can't
resurrect against a fixed task. Best-effort + inert when the table is empty.
Cannot manufacture a spawn — the counter only ever suppresses one.
(_instances reconcile, the spec's other goal, already shipped as
_readopt_running_agents.)
* fix(types): cast Mapped[UUID] columns in project routes + self_heal
A clean `mypy roboco/ tests/` run surfaces 7 pre-existing errors in files this
branch doesn't touch: project-route handlers and self_heal_engine pass a
ProjectTable.id (declared Mapped[UUID] against SQLAlchemy's dialect UUID, so
mypy infers sqlalchemy.sql.sqltypes.UUID[Any]) where a uuid.UUID is expected.
An incremental .mypy_cache had hidden them. Apply the same targeted cast unblock
used for the prior batch; the deeper fix (migrating the ~88 Mapped[UUID]
columns to Mapped[uuid.UUID]) remains a separate dedicated task.
* docs(orchestrator): document respawn_tracker durability
Add the orchestrator runtime-state durability note to CLAUDE.md (respawn_tracker
write-through + restore; _instances reconciled-from-Docker) + the migration-051
narrative, and a CHANGELOG [Unreleased] Fixed entry. Also type-clean the new
respawn_tracker table test (cast __table__ to Table under TYPE_CHECKING).
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
post_pr_review (inbound external/fork PR review) derived both the recorded
notes_structured.pr_review.verdict AND the posted GitHub review event solely
from its `event` argument, which defaults to REQUEST_CHANGES — and, unlike the
in-path gate's pr_fail, it never required any findings. A reviewer that
concluded 'approve' in the summary but left event at the default filed (and
posted to the contributor's PR) a blocking 'changes requested' with nothing
cited, contradicting the approving summary the CEO saw on the PR Reviewer Notes
card.
Enforce a verdict<->findings invariant before any record or GitHub post:
- REQUEST_CHANGES must cite >=1 finding (almost always a forgotten
event='APPROVE'), mirroring pr_fail's 'at least one issue' rule;
- APPROVE may not carry a blocker/major finding.
The check is the pure policy fn pr_review_conflict() wired through the new
choreographer _verdict_consistency_gate, rejected with a clear remediate hint.
Steer the agent at the source too: the flow MCP tool + request schema now spell
out the invariant and to pass event='APPROVE' explicitly for a clean PR.
* feat(release): add release-manager feature flag (default off)
* feat(release): change classification + semver-bump derivation
* feat(release): readiness audit (changelog/version-ref/docs/migration/gate)
* feat(release): release-manager engine proposes a gated release
* feat(release): fail-closed release executor (bump, gate, publish)
* feat(release): CEO approve/reject release-proposal surface
* docs(release): document the gated release manager
* feat(memory): add org-memory feature flags (default off)
* feat(memory): add playbooks table + status enum + migration
* feat(memory): playbook service with auditor curation transitions
* feat(memory): playbooks RAG index plugin
* feat(memory): index a playbook into RAG on approval
* feat(memory): distill a high-signal lesson at task completion
* feat(memory): keep private journal reflections out of the shared RAG corpus
* feat(memory): draft_playbook verb + auditor curation verbs
* fix(ci): resolve mypy tests/ errors blocking the gate (UUID casts, annotations)
* feat(memory): auto-inject similar lessons/playbooks into the briefing
* feat(memory): auditor playbook review queue (api + panel)
* docs(memory): document the org-memory loop + playbook verbs
* fix(provisioning): idempotent pitch provisioning (reuse product/project by slug on re-approval)
* fix(memory): add chunks_playbooks to the chunk schema + isolate release route tests
- Migration 030's CHUNK_TABLES was missing chunks_playbooks, breaking the
IndexType<->migration parity guard once the PLAYBOOKS index landed. The
upgrade is ALTER ... IF EXISTS so adding it is safe on any DB shape.
- The release-route fixture's approve/reject paths call db.commit() (real
behavior), so a held proposal outlived the per-test rollback and leaked
into engine tests that read the global list_open_release_proposals().
Tear down source=release_manager rows after each test.
- Make the gather_snapshot real-repo smoke version-agnostic (semver match)
so it stops pinning the literal repo version.
* chore(release): 0.13.0
* ++
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* feat(ci-watch): config flags
Default-off CI-watch config (mirrors self_heal_*): ci_watch_enabled,
ci_watch_default_workflow (ci.yml), ci_watch_interval_seconds (1800),
ci_watch_max_open_tasks (3), ci_watch_max_per_cycle (1). Registers
ci_watch_enabled in the panel FEATURE_FLAGS. 4 tests.
* feat(ci-watch): per-project ci_watch_enabled/workflow (migration 048)
Adds projects.ci_watch_enabled (bool NOT NULL default false) +
projects.ci_watch_workflow (varchar null) — the per-project opt-in for
multi-repo CI-watch. ProjectTable + Pydantic Project fields + migration 048
(off 047_ws_single_active). Real upgrade->downgrade->upgrade chain verified
against a throwaway Postgres; 2 ORM round-trip tests.
* feat(runtime): prune dangling agent images in the background sweeper
Every agent-image rebuild orphans the prior build's layers as an untagged
<none> image; across deploys these pile up (the operator hit ~80). The sweeper
now runs 'docker image prune -f --filter dangling=true' (dangling only — a
tagged image or one backing a running container is never dangling), throttled
to settings.image_prune_interval_seconds (default 6h) and gated by
image_prune_enabled (default on). Best-effort: any failure is logged, never
raised into the sweeper. Mirrors the transcript-retention prune. 4 tests.
* feat(ci-watch): source tag + open-task dedupe query
CI_WATCH_SOURCE='ci_watch' + TaskService.list_open_ci_watch_tasks(git_url=None):
non-terminal ci_watch tasks (the dedupe + open-cap basis), optionally scoped to
one repo by git_url — a monorepo registers several cell-projects on one git_url,
so dedupe keys on the repo, not the slug. 2 real-PG tests.
* feat(ci-watch): multi-project CI telemetry fan-out
MultiProjectCITelemetrySource.fetch(projects) reuses the hardened per-project
get_latest_ci_conclusion for each opted-in project (passing its ci_watch_workflow
or the configured default). Per-project isolation: a GitHub error or absent
signal yields NO sample (unknown, never read as green) and never aborts the
sweep; only a real conclusion yields a sample (fail→breach, pass→non-breach).
self-heal source untouched. 3 tests + self-heal regression green.
* feat(ci-watch): engine — fan-out, originate, dedupe, cap
CiWatchEngine.run_cycle(projects) mirrors SelfHealEngine: assess via
MultiProjectCITelemetrySource, open one PENDING ci_watch fix task per red repo
(team=main_pm, assigned_to=main-pm, confirmed_by_human=True so it dispatches
without an Approve-&-Start — the fe029fe3 lesson), never starts/approves/merges.
Dedupe per git_url (monorepo → one fix task per repo) + per-cycle/rolling caps.
Default-off; disabled → no-op. 5 real-PG tests (red→one task, dedupe, cap,
green/none→nothing, disabled).
* feat(ci-watch): orchestrator loop tick + watch-set loader
_ci_watch_loop (registered in start(), cancelled in stop(), separate from the
untouched self-heal loop): dormant unless ci_watch_enabled; each interval loads
the watch set (ci_watch_enabled projects, collapsed one-per-repo via the
existing _projects_one_per_repo) and runs CiWatchEngine.run_cycle, committing
opened tasks. _run_ci_watch_cycle extracted for testing; loud warning when
enabled-but-empty. confirmed_by_human=True on the originated task means it
dispatches without an Approve-&-Start (no stranding, the fe029fe3 lesson).
5 tests (disabled no-op, watch-set filter+one-per-repo, empty warn, engine run).
* docs(ci-watch): CHANGELOG + CLAUDE.md for multi-repo CI-watch
Document CI-watch (Added) in the CHANGELOG and the Self-Healing & Feature Flags
section of CLAUDE.md — it generalizes self-heal to opted-in projects, reuses the
hardened per-project CI lookup, never auto-merges, default-off. Adds the
ci_watch_enabled flag to the feature-flags enumeration.
* feat(dep-update): config flags
Default-off dep-update config (mirrors self_heal_*/ci_watch_*): dep_update_enabled,
dep_update_interval_seconds (604800 = weekly), dep_update_max_open_tasks (3),
dep_update_max_per_cycle (1). Registers dep_update_enabled in FEATURE_FLAGS. 4 tests.
* feat(dep-update): per-project dep_update_command/paths (migration 049)
Adds projects.dep_update_command (varchar null) + dep_update_paths (varchar[]
null) — the per-project opt-in for the dependency-update bot. ProjectTable +
Pydantic Project fields + migration 049 (off 048_ci_watch_project_cols). Real
upgrade->downgrade->upgrade chain verified on a throwaway Postgres; 2 ORM tests.
* feat(dep-update): source tag + open-task dedupe query
DEP_UPDATE_SOURCE='dep_update' + TaskService.list_open_dep_update_tasks(git_url=None):
non-terminal dep_update tasks (dedupe + open-cap basis), optionally scoped to one
repo by git_url (monorepo → one open dependency-update task per repo). 2 real-PG
tests.
* feat(dep-update): read-only lockfile-diff probe
WorkspaceService.dry_upgrade_changes_lockfile(project): clones the project's
read clone into a throwaway dir (--no-hardlinks, so the read clone is never
mutated), runs project.dep_update_command (no shell, shlex.split), and reports
whether any lockfile path (dep_update_paths or inferred uv.lock/pnpm-lock.yaml)
is dirty. Fail-safe: null/failing command → False (don't originate on a broken
probe), logged; throwaway always removed; never commits/pushes. 5 real-git tests.
* feat(dep-update): engine — detect, originate, dedupe, cap
DepUpdateEngine.run_cycle(projects) mirrors SelfHealEngine/CiWatchEngine: for
each opted-in project (dep_update_command set) with updates available (the
read-only probe), open one PENDING dep_update task (team=main_pm, assigned-to
main-pm, confirmed_by_human=True), never starts/approves/merges. Cheap checks
(command, per-git_url dedupe) before the expensive probe; per-cycle + rolling
caps. Default-off; disabled → no-op. 6 real-PG tests.
* feat(dep-update): weekly orchestrator loop tick
_dep_update_loop (registered in start(), cancelled in stop(), separate from the
self-heal + CI-watch loops): dormant unless dep_update_enabled; each interval
(default weekly) loads projects with a dep_update_command (one-per-repo) and runs
DepUpdateEngine.run_cycle, committing opened tasks. _run_dep_update_cycle
extracted for testing; loud warning when enabled-but-no-commands. Refactored
stop() to cancel background tasks via a shared _cancel_background_task loop
(keeps it under xenon B as the loop count grows). 4 loop tests.
Task 7 (anti-stranding dispatch guard) is satisfied by construction: no
dispatcher skip targets source='dep_update', and the engine sets
confirmed_by_human=True (the fe029fe3 lesson), asserted in the engine tests —
so the originated task dispatches via the assigned-PM path, never stranded.
* docs(dep-update): CHANGELOG + CLAUDE.md for the dependency-update bot
Document the dep-update bot (Added) in the CHANGELOG and the Self-Healing &
Feature Flags section of CLAUDE.md — read-only lockfile-diff probe, never
auto-merges, per-project opt-in via dep_update_command, default-off. Adds the
dep_update_enabled flag to the feature-flags enumeration.
* feat(ci-watch): route fix-task notification to the project's cell PM
On opening a fix task, CiWatchEngine notifies the red project's own cell PM
(resolved from project.assigned_cell via foundation AGENTS — e.g. BACKEND →
be-pm), not the CEO, once per project per cycle. Best-effort: a notification
failure never rolls back the origination. Adds _cell_pm_slug_for +
_notify_cell_pm. 1 real-PG test (asserts to_agent='be-pm', not 'ceo').
* feat(ci-watch,dep-update): expose per-project opt-ins in the project API
Add ci_watch_enabled/ci_watch_workflow + dep_update_command/dep_update_paths to
ProjectUpdate, ProjectUpdateRequest, the PATCH route mapping, ProjectResponse,
and project_to_response — so the panel edit-project dialog can read + set the
per-project autonomy opt-ins (the columns were unreachable through the API
before). Also threads the previously-dropped quality_command through the update
route. 1 real-PG update round-trip test.
* feat(ci-watch,dep-update): panel project-edit fields for the per-project opt-ins
Adds an 'Autonomous Maintenance' section to the edit-project dialog: a CI-watch
enable switch + workflow input, and a dependency-update command + lockfile-paths
input (comma-separated → list). Threads the four fields through the Project /
ProjectUpdate TS types and the mock-mode create fixture. The global on/off
toggles already live in Settings → Feature Flags; these are the per-project
opt-ins. panel tsc --noEmit + eslint green.
* docs(0.12): CI-watch + dep-update bot + image-prune across user docs + RAG
New docs/optional/autonomous-maintenance.md (mirrors self-heal.md) covering both
engines; optional/index rows; panel settings + projects-and-products notes for
the Feature Flags toggles + the edit-project Autonomous Maintenance fields;
resilience note for the dangling-image prune; env-reference + RAG config-reference
tables for all ROBOCO_CI_WATCH_* / ROBOCO_DEP_UPDATE_* / ROBOCO_IMAGE_PRUNE_*
vars; mkdocs nav entry. reflow-check green; prompts unchanged (operator-facing,
not agent-facing).
* chore(release): 0.12.0
Cut [Unreleased] -> [0.12.0] (CI-watch + dep-update bot + image-prune housekeeping
+ the post-0.11.1 run-hardening fixes). Bumps all 8 canonical version refs to
0.12.0 (pyproject / uv.lock roboco pkg / panel package.json / __init__ /
config.app_version + the README / deployment / agent-image-tag examples).
* fix(pr-review): repo-scope external-PR dedupe (no duplicate review on a monorepo)
external_review_task_exists keyed on (project_id, pr, head_sha), but a monorepo
registers several cell-projects on one git_url and the poll already collapses to
one canonical project per repo — so once a review task was re-pointed to a
sibling project, the next poll (checking the canonical project) no longer saw it
and opened a second review of the same PR (observed: PR #131 reviewed once on
guard-core-saas-frontend, once on -backend). Dedupe now spans every project
sharing the PR's repo (git_url); re-review on a new head SHA still works; a
genuinely different repo with the same PR number is independent. 3 real-PG tests.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* fix(git): don't delete a branch that still has open dependent PRs
Root cause of the run-zombifying "integration branch gone from origin" wedge.
_delete_remote_branch_best_effort deleted a merged PR's head branch
unconditionally, so:
- merging a cell->root PR deleted the cell branch while a sibling leaf PR was
still targeting it as base, and
- the CEO's root->master merge deleted the feature/main_pm/{root} integration
branch.
The dependent PRs lost their base, every later git op against the vanished
branch failed, and the task zombified (a51c3d31 only made the post-merge sync
non-fatal; this removes the cause).
The remote-branch delete chokepoint (the single path all merge/close/cancel
deletions funnel through) now first checks _branch_has_open_dependents: any OPEN
PR targeting the branch as its base marks it an active integration target and
preserves it. Fails safe (any error => keep the branch; cleanup is best-effort,
stranding is not). True leaf branches with no open dependents are still cleaned
up. Adds 6 unit tests for the guard + the probe.
* fix(git): recover a drifted shared clone on resume instead of BRANCH_MISMATCH
A dev/documenter/QA clone is shared across that agent's tasks. On a
respawn/resume it can sit on a sibling task's branch, or a re-provisioned clone
can lack the task branch as a local ref (commits only on origin). The
fresh-claim path git-resets the clone clean, but resume deliberately
short-circuits before it (_dev_reentry), so the agent's next commit hit
_assert_on_task_branch's BRANCH_MISMATCH, failed, and the task wedged in a
blocked respawn loop (the documenter that could never land its doc commit).
_assert_on_task_branch now recovers instead of only rejecting: fetch + checkout
the task branch (recreating a missing local ref from origin via `git branch
<b> origin/<b>`), and raise only when the switch genuinely can't happen
(uncommitted changes block it). Never discards work — checkout, not reset — so
a resumed agent's unpushed commits are preserved. Updates the RAG troubleshooting
+ developer docs to describe the auto-recovery. Adds 5 unit tests.
* fix(runtime): re-adopt running agent containers on restart (no double-spawn)
An orchestrator restart loses the in-memory _instances registry while the agent
containers keep running. The reaper already had a Docker-liveness fallback
(_assignee_container_running), but the spawn gate (_is_agent_active) did not, so
right after a restart it saw a live agent as inactive and could launch a second
container onto work the forgotten-but-running one was already doing.
start() now calls _readopt_running_agents() after _reconcile_orphan_claims_on_startup
and before the dispatcher/reaper loops launch: it probes each known agent slug's
container (AGENT_IMAGES, reusing _inspect_container_state — the same docker
inspect the reaper uses) and registers a minimal AgentInstance(state=ACTIVE) for
any that is running and not already tracked. Inert when nothing runs (cold start
unchanged); best-effort (a probe error leaves that slot for the reaper's own
fallback). This is the gateway-health spec's Task 4 / the orchestrator-state
spec's Phase 3 (_instances reconcile). Adds 4 unit tests.
* fix(git): treat an already-merged PR as idempotent success on merge
A merge PUT against an already-merged PR returns the same 405 as a genuine
"not mergeable" conflict, so _merge_with_retry raised MergeConflictError and the
completion path tried to rebase / close-superseded / escalate a PR that had
already landed (a prior cycle, a sibling, or the CEO merged it) — the
cell_pm_complete block<->unblock respawn loop.
_merge_with_retry now disambiguates before raising: a new _pr_is_merged probe
(GET the PR, check merged==true) returns success on an already-merged PR so
completion proceeds idempotently; a genuinely-unmerged 405 still raises the
conflict. Best-effort probe (False on any error → falls through to the existing
conflict handling). Adds 4 unit tests.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
Two in-path PR-review-gate bugs surfaced reviewing the guard-core-app recovery
roots (PR #107 / root fead4372):
1. No verdict comment reached the PR. _project_slug_for returned None whenever
project_id was None — but a Main-PM coordination root (the only task a
root->master PR ever sits on) carries just a product_id (the cell->repo map),
so _post_gate_review_to_pr resolved a None slug and silently no-op'd. It now
falls through to the product's first distinct project (mirrors
GitService._project_for_task), so the gate verdict actually lands on the PR.
2. The task's PR-reviewer notes contradicted the transition. pr_pass / pr_fail
only threaded their notes through the tracing-gate shim and posted to GitHub;
nothing wrote notes_structured.pr_review. A root passed once and later failed
kept showing verdict=passed while the real transition was pr_fail. The gate
now authors the canonical pr_review note on every decision (pr_pass -> passed,
pr_fail -> failed), best-effort so a malformed note never rolls back the gate.
Adds unit tests for the product-slug fallback and the verdict persistence.
When an integration (cell/root) branch is deleted from origin — e.g. after a sibling cell->root merge, stranding a late straggler leaf — pr_merge's post-merge _sync_target_branch ran 'git fetch origin <branch>' (check=True) and raised 'couldn't find remote ref'. That surfaced as a retryable SERVICE_ERROR, so complete() re-blocked the task and respawn-looped the PM on an already-landed merge (observed live on cell branches feature/backend/31ae12fc--0e49e04e and 7aeee245--dcfe9fc2, blocking be-pm's complete() 5+ cycles).
pr_merge reaches the post-merge sync only after the authoritative GitHub merge has already succeeded (_merge_with_retry raises otherwise), so refreshing the local workspace copy of the target branch is cosmetic. Route it through a new _sync_target_branch_best_effort that logs and returns None instead of raising. merge_pull_request (CEO path) keeps the strict sync — its target is the default branch, which always exists on origin.
make quality runs 'mypy roboco/ tests/'; the new PM-needs_revision claim test passed a SimpleNamespace where _get_valid_claim_statuses expects AgentTable | None, failing the type gate. The helper only reads agent.role, so cast the lightweight stand-in to AgentTable (no DB row, no type: ignore).
The lifecycle spec (CLAIM_RULES) grants CELL_PM/MAIN_PM claim of NEEDS_REVISION so a rejected coordination root (pr_fail / qa_fail / ceo_reject) can be re-claimed via i_will_plan and re-delegated. The runtime mapping _ROLE_CLAIM_STATUSES omitted it for PMs, so the spec gate allowed i_will_plan on a needs_revision root while the composed claim() rejected it -> returned None -> INVALID_STATE: the PM could neither plan nor idle its own rejected root and respawn-looped (observed live on cell root 0e49e04e, ~143 INVALID_STATE rejections across 11 PM sessions; the tail of the 2026-06-24/25 run).
Add NEEDS_REVISION to the cell_pm/main_pm runtime claim statuses, and a parity test locking _ROLE_CLAIM_STATUSES to lifecycle.CLAIM_RULES so the two can't drift again.
A parent's acceptance-criteria coverage is matched by stable criterion id, but
a PM may declare covers_parent_criteria on a child by EITHER the criterion's id
OR its full text (both happen in practice). _parent_ac_ref_sets unioned the raw
refs and matched by id only, so a COMPLETED child that declared coverage by text
was invisible to the matcher: the criterion read "uncovered", the roll-up gate
refused, and the PM re-delegated the already-finished work as a brand-new empty
subtask (0 commits, no PR) that can never close — looping for hours and burning
tokens (observed live: a parent's xenon work completed + merged via one child,
then re-delegated 2h later as an empty phantom).
Normalize every child ref to the criterion id (text -> id via the parent's own
criteria) in a small _normalize_ac_refs helper, so coverage counts regardless of
how it was declared. Fixes existing mismatched data and future declarations; all
three consumers (uncovered/unclaimed/parent_ac_coverage) share the builder. An
unknown ref (neither id nor a current criterion text) passes through and matches
nothing, exactly as before. Adds two regression tests.
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* [831988ba] fix(lifecycle): add rejection_kind to Precondition, PRECONDITION_OWNERSHIP uses not_authorized
Add rejection_kind: RejectionKind = 'tracing_gap' field to the Precondition
frozen dataclass. PRECONDITION_OWNERSHIP now carries rejection_kind='not_authorized'
so ownership failures surface as authorization issues rather than tracing gaps.
Update _check_intent_preconditions to dispatch Decision.reject(kind='not_authorized')
when the first failing precondition has rejection_kind='not_authorized' — for all
other rejection_kinds the existing Decision.tracing_gap path applies.
Update test_can_invoke_intent_open_pr_rejects_non_owner to assert not_authorized
instead of tracing_gap (90 parity tests in test_lifecycle_consumer_parity.py
now agree: choreographer and spec both return not_authorized for owned=False).
All 4871 foundation tests pass, 3264 unit tests pass, ruff/mypy green.
* [831988ba] docs(architecture): document preconditions and rejection kinds in lifecycle spec
Add comprehensive guide explaining how Precondition rejection_kind field works in
the lifecycle spec. Documents the distinction between tracing_gap (missing artifact)
and not_authorized (identity/role boundary) rejections, includes the dispatch logic
in _check_intent_preconditions, and explains agent-visible impact of the change.
This context is essential for agents to understand why PRECONDITION_OWNERSHIP failures
now return not_authorized instead of tracing_gap, and when to use each rejection_kind
for new preconditions.
---------
Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>
Co-authored-by: Backend Documenter <be-doc@agents.roboco.dev>
Generalize the GROK-only _grok_spawn_parked guard to _provider_spawn_parked.
spawn_agent now consults the RateLimitStateTracker for every provider, so
Anthropic session/overload parking suppresses container launches instead of
letting the dispatcher re-spawn every tick. Fail-open on tracker errors.
- Rename _grok_spawn_parked -> _provider_spawn_parked (any provider)
- Update spawn_agent log + gate
- Update grok-rate-limit tests to cover general provider behavior
The SDK server writes runtime output to /tmp/sdk-server.log inside the agent
container, so the session-limit markers never appeared in docker logs. Read
the newest durable Claude transcript from ~/.claude/projects as a fallback so
the provider gets parked and auto-revived instead of crash-retrying.
- Add _transcript_tail_text to read the agent's transcript tail
- Use it in _provider_rate_limit_park_target alongside docker logs
- Add regression test for transcript-only detection
Forensic triage of a 24h run reconstructed the dominant gateway.rejected
loops from the audit_log. After earlier deploys fixed the i_will_plan crash
and the open_pr push-gap, three real, recurring-capable burn loops remained.
This fixes them at the architecture level, not by prompt-nagging.
journal:decision write-then-gate (the dominant completion-path blocker):
PM decision-point verbs required a separate note(scope='decision') call
before the verb, which loaded/weak models forget to chain — so complete and
unblock hit a tracing_gap (journal:decision missing) and respawn-looped,
stranding finished tasks forever. Each verb now auto-records its OWN
rationale as the journal:decision before the gate runs (the proven
i_am_blocked -> write_struggle pattern), so the gate passes off real,
persisted reasoning. unblock gains a required `reason` (threaded MCP tool ->
request schema -> routes -> choreographer); delegate derives the decision
from its title + description; complete/submit_up/submit_root/escalate_up/
escalate_to_ceo reuse their existing notes/reason. The gate still runs as
defense-in-depth; the auto-record is idempotent within the decision window
and best-effort. Adds JournalService.write_decision and
Choreographer._ensure_pm_decision.
open_pr empty-diff 422: an overlapping-decomposition leaf with zero commits
vs its base makes GitHub 422 "No commits between ...". The generic
invalid_state "retry" looped the dev 15x on one task. open_pr now steers to a
terminal i_am_blocked hand-off so the PM completes or cancels the redundant
leaf.
owns_task stale-agent loop (41x): a superseded agent (task reassigned away)
calling i_am_done/open_pr got a PRECONDITION_OWNERSHIP tracing_gap it read as
a fixable precondition and retried forever. Both verbs now short-circuit with
the clear not_authorized "no longer yours -> give_me_work" steer that
resume/unclaim already use.
RAG docs updated for the new unblock(reason) signature; CHANGELOG entries
added under 0.11.0 (unreleased). open_pr refactored into
_open_pr_preflight_rejection + _open_pr_failure_env to stay within the
return-count and complexity budgets.
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
_sync_target_branch previously did a bare 'git checkout <target>' with no
fallback. In the shared-agent-clone model the leaf developer's workspace
often has only the task branch locally; the parent/cell branch exists only
on origin. That produced a SERVICE_ERROR which cycled the task back to
blocked every time the PM retried complete().
Now, when checkout fails, we fetch the target branch from origin and create a
tracking branch before the pull. Includes regression tests for the local,
fallback, and origin-missing paths.
* fix(run-hardening): stop three blocked-task respawn loops
Three independent fixes for blocked-task respawn loops observed in the live
run (the bleeders behind a wedged near-complete run):
- verb runner: re-check the working task after EACH composed atomic action,
not just at entry. A concurrent transition between a verb's precondition
gate and execution (e.g. a racing i_am_blocked moving a root from
needs_revision to blocked) made claim() return None mid-sequence; the next
composed step dereferenced None.id and crashed with the opaque
"'NoneType' object has no attribute 'id'", looping the PM. Now fails fast
with an actionable INVALID_STATE; the savepoint rolls the partial run back.
- blocker dispatch: never dispatch a Board role (product-owner / head-
marketing) as a blocker resolver. Board roles have no unblock verb, so the
dispatcher respawned one forever to "resolve" a blocker it could only
notify/triage about — one incident burned ~6400 tool calls on a single
mis-owned root. _blocker_resolver_slug now returns None for a Board
assignee so the dispatch skips it.
- git push: recover a missing local task-branch ref from origin/<branch>
before push-by-name. A re-provisioned shared clone can lack the branch
locally though its commits are on origin, so push died on
"src refspec <branch> does not match any" and the task wedged at i_am_done.
Now materializes the ref (no-op push when already on origin) or fails loud
with an unclaim+reclaim instruction when the work is on neither.
Adds regression tests for all three. Full no-DB gate green (ruff, reflow,
mypy, xenon); pytest+coverage validated by CI.
* fix(verb-runner): only raise on an INTERMEDIATE composed None, not the last
The mid-composition None-guard was too aggressive: it raised for a None
returned by the LAST composed action too (e.g. start()), preempting the
caller's existing `if task is None` handler that surfaces the verb-specific
message ("start failed for task ...", the board verb's decline envelope).
Three tests asserting those messages broke in CI.
Only an INTERMEDIATE None is fatal (the next action would deref None.id). A
None from the last action is the verb's own result and must flow out as the
runner's return value. Guard now fires only for position > 0, before the
next dispatch — still prevents the crash, preserves the last-action contract.
* fix(escalation): never hand a Main-PM coordination root to the Board
The upstream cause of the board catch-22 (which the orchestrator-side
blocker-dispatch guard only backstopped): the escalation chain points
main-pm -> product-owner, and i_am_blocked/escalate REASSIGNS the task to
that chain target. apply_escalation's board-advisory guard only refused
descendant cell tasks (both predicates require parent_task_id), so a
top-level Main-PM coordination root slipped through and the whole root was
reassigned to the Product Owner + marked blocked. The board has no unblock
verb, so it spam-notified the CEO and respawn-looped (~6400 tool calls on
one root).
Add _is_coordination_task (team == main_pm — covers a delivery root AND a
MegaTask root-subtask) and a shared _board_cannot_own predicate, applied at
all four board-refusal sites (escalation, reassign, reassign_active_claim,
dependency-revival). A main_pm coordination task escalated/reassigned onto a
board role is now diverted to the pool for a role-matched (Main-PM) reclaim.
Complements the blocker-dispatch backstop in the prior commits (defense in
depth). Tests: coordination-root predicate cases + apply_escalation divert;
existing teamless-root / board-root behavior unchanged.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
A developer's single clone is shared across all of their tasks, so by the
QA-submission / open_pr boundary the workspace is usually parked on a LATER
task's branch. push_task_branch / push_for_task asserted the current branch
and pushed it, and create_pull_request used get_current_branch as the PR head
— so for an earlier task the push was rejected (BRANCH_MISMATCH) and the
locally-committed work never reached origin, leaving the task branch empty and
open_pr failing with GitHub's "No commits between" 422. The work was committed
correctly on the local task branch, just never pushed.
Operate on the task's recorded branch by name, independent of the checkout:
- push() takes an explicit branch and pushes that named ref
- push_task_branch / push_for_task push the task's branch by name (drop the
assert-on-current-branch gate that rejected the shared-clone case)
- create_pull_request uses the task's branch_name as the PR head via the new
_pr_head_branch helper
Regression tests assert push and PR-head target the task branch from any
checkout. 45 git + 735 gateway + 59 git-integration tests green; ruff/mypy/
xenon clean.
Two failures the full make-quality flagged after 06adf978 landed on master:
- mypy: _second_agent (test helper) now returns the agent's UUID, so
WorkSessionCreate(agent_id=...) receives a real uuid.UUID rather than the
ORM column type.
- xenon: the single-active-per-task supersede is extracted out of
_create_work_session_if_needed into _supersede_other_active_sessions,
bringing the former back under complexity rank B.
No behavior change; 1566 work-session/gateway/service tests green.
A task re-claimed by a different agent (pool release, reaper unclaim,
escalation redirect) left the prior holder's active work session open.
WorkSessionService.get_active_for_task then ran a one-row query over the
duplicates and raised MultipleResultsFound; the caught failure surfaced
as the cryptic "'NoneType' object has no attribute 'id'" that crashed the
claim/plan/start flow — so the task could never advance, the orchestrator
re-spawned its PM every ~30s forever, and its dependents stayed blocked.
Fixed at three layers:
- active-session lookups return the most-recent session instead of raising
- claiming a task supersedes any other agent's stale active session
(the single-active-per-task invariant), in both WorkSessionService.create
and TaskService._create_work_session_if_needed
- a partial unique index (migration 047, which de-duplicates existing rows
keeping the most recent) enforces it at the DB level; mirrored on the model
Verified: 1614 tests green (work_session + gateway + services), ruff/mypy
clean, migration chain applies + reverses, dedup proven on the real schema.
Renn F