increment_probe_failures / reset_probe_failures did a non-atomic
get_state (GET) -> mutate -> set (SET) in Python. A concurrent
activate() re-park writes a FRESH episode blob (probe_failures: 0 +
fresh activated_at / retry_after / affected_agents / kind); if the
stale increment's SET landed after the fresh activate's SET, the stale
blob overwrote the fresh episode metadata AND un-reset the counter
(clobbering the new episode).
Redis single-threads a Lua EVAL, so a server-side read-modify-write
is indivisible: activate's SET is serialized entirely before or after
the script, never interleaved between the script's GET and SET. The
two scripts mutate ONLY probe_failures, so every other episode field
survives the bump. activate stays a single atomic SET (a fresh episode
resetting the counter to 0 is correct semantics).
A hung git/make/gh/clone would block the CEO-gated release loop
indefinitely. Wrap each proc.communicate() in asyncio.wait_for via a
shared _await_proc helper; on expiry proc.kill() the child and return a
non-zero rc (124) so every caller's fail-closed branch fires. Mirrors the
quality-gate _run_one kill-on-timeout idiom.
Deadlines are generous (30min gate / 10min clone / 5min push+gh) so a
legitimate slow op is never wrongly aborted — floor-assertion tests pin
the floors to guard exactly that logical regression. Green path returns
the real rc unchanged.
If _configure_git raised CalledProcessError before its `remote set-url`
scrub, .git/config kept the tokenized auth URL (the project PAT) and
_assert_no_pat_leak never ran. The except clauses raised WorkspaceError
without removing the workspace, so the next ensure_workspace's health
short-circuit (valid .git with HEAD + objects) skipped past the leak —
mounting the agent on a workspace whose .git/config let it read+exfiltrate
the PAT. Both clone-failure except clauses now rmtree the workspace before
raising, so a half-configured clone is destroyed and ensure_workspace
re-clones from scratch. TDD: 2 tests (configure-failure leak + timeout).
merge_pr unconditionally set pr_status=merged, pr_merged_at, merged_by,
status=COMPLETED on whatever session it loaded — the only session-terminal
transition in WorkSessionService lacking both the active-status guard
(complete/abandon) and the terminal-idempotency guard (close). Two failure
modes: (1) a retried merge after a successful-but-unconfirmed GitHub merge
overwrote merged_by/pr_merged_at with the retry's actor/timestamp, corrupting
the merge audit trail; (2) merge_pr on an ABANDONED session resurrected it to
COMPLETED, undoing the single-active abandonment. Mirrors close()'s guard:
if status != ACTIVE, return the session unchanged. Both git.py callers await
merge_pr and discard the return, so the no-op is safe. TDD: 3 tests
(happy-path + both modes).
Converts 115 `# type: ignore[...]` suppressions across 23 test files to
no-suppression patterns (helper-return widening to Any, local Any aliases,
cc:Any aliases, cast at narrow call sites, typed fixtures) so the hard
no-type:ignore convention holds across tests/. No test logic or assertions
changed — only mock-wiring mechanics and type annotations.
Gate: ruff check tests/ clean; mypy tests/ (538 files) clean; 176 changed-file
tests pass. Zero real suppressions remain (the 7 grep hits are 3 hygiene-
checker string-literal test inputs and 4 prose mentions in comments).
Convention: no type:ignore/noqa, and pre-existing violations still
violate. The make-quality gate runs 'mypy roboco/ tests/', but the
prior commits' gates only ran mypy on production files, masking 64
type errors across 15 test files (method-assign, unused-ignore,
no-untyped-def, attr-defined, union-attr, has-type, index, misc).
Fixed without any type:ignore:
- method-assign (svc.session.X = / svc.method = AsyncMock()): hold a
local 'session: MagicMock'/'AsyncMock' and assert on it, or stub via
object.__setattr__ / monkeypatch / a typed '_bind' helper returning
Any, or alias 'cc: Any = c' (the pattern the file already used).
- unused 'type: ignore[assignment]' (real code was method-assign):
removed; replaced with the no-suppression patterns above.
- 'Callable[...] has no attribute assert_*': keep a typed local ref to
the AsyncMock and assert on the local, not the method-typed attr.
- no-untyped-def: annotate helper params (Any / pytest.MonkeyPatch).
- attr-defined / index / union-attr: type the helper as Any, narrow
with an 'is not None' assert, or add the missing attr to a fake.
- has-type / return-value: fix the declared return type to the tuple
the function actually returns.
- PLC0415 inline imports: hoisted to top-level.
test_pr_gate_notifies_pm._stub_gate_path converted fully to the
'cc: Any = c' alias (it already used it for one attr) so its five
'# type: ignore[method-assign]' suppressions are gone.
mypy tests/: 64 errors -> 0 (538 files). ruff check tests/: clean.
All 84 tests in the touched files pass.
Convention: no type:ignore/noqa. The F061 in-session audit-emit
tests used '# type: ignore[assignment]' to assign a MagicMock to
AsyncSession.add, and the F060 test assigned to .flush the same way.
Rewritten to hold a local 'session: MagicMock' variable (mypy sees
its auto-children as MagicMock, so .add.side_effect / .flush assign
cleanly with no suppression). Verified via 'mypy tests/' that both
files are now type-clean (the F060/F061 commits had skipped tests/
in mypy, masking two method-assign errors).
_emit_status_transition_audit now writes AuditLogTable rows into
self.session synchronously (session.add) instead of dispatching
AuditService.log_task_event fire-and-forget on its own connection.
The audit row now commits/rolls back atomically with the status
transition in the caller's transaction, closing three facets at once:
- F061: audit commit no longer decoupled from the transition commit
- F073: a committed transition can no longer have NO audit row
(the row rides the same transaction; a swallowed persist can't drop it)
- F075: a transition rolled back inside a verb savepoint no longer
leaves a phantom audit row (the row is in the savepoint too)
log_task_event is now called only from this helper (narrow blast
radius verified); revision_count increment stays at this single
chokepoint. Cycle-time/bottleneck reconstruction from task.<status>
events is no longer silently corruptible.
Tests: test_emit_status_transition_audit_writes_in_session_atomically,
test_finalize_claim_rollback_emits_reversal_audit, escalation-audit
tests retargeted to in-session AuditLogTable rows.
Also: _canonical_bump_files grep-looseness follow-on (F058) -- filter
by subject, not body; git log --grep matches any message line, so a
non-release commit whose body references chore(release): shadowed the
real release commit. Test
test_canonical_bump_files_ignores_body_only_chore_release_match.
The forward task.claimed audit row is flushed before the branch-creation
attempt, and AuditService commits on its own connection, so the rollback's
flush reverts the task row but not that audit row — the journey's last
event stayed task.claimed while the task reverted to its pre-claim status,
diverging from real state and corrupting downstream cycle-time/bottleneck
metrics. The rollback now emits a CLAIMED->original reversal audit row
(only when the forward transition was made) attributed to the claimant.
The module docstring promised self-heal fix tasks 'wait for the CEO's
Approve-&-Start', but _originate created them confirmed_by_human=True and the
orchestrator dispatched them at once — a self-heal fix that re-broke CI would
trigger another cycle, open another auto-dispatched fix, and loop with no CEO
gate on dispatch.
Restore the documented gate:
* _originate opens the task confirmed_by_human=False (held for the CEO).
* The orchestrator holds a self-heal task out of both the PM and dev dispatch
paths until confirmed_by_human flips True.
* approve_and_start (the CEO's start gate) sets confirmed_by_human=True so the
held task finally dispatches (idempotent for board/intake tasks already True).
* list_pending_for_agent scopes the give_me_work hold to self-heal
(source != self_heal OR confirmed_by_human) so an already-alive PM can't grab
it pre-approval — while ordinary delegated subtasks (confirmed_by_human=False
by default, where the delegation IS the authorization) still dispatch.
The 'never self-deploys' guarantee (no merge) is unchanged.
_canonical_bump_files derived the bump set from the previous
chore(release): commit. On the first release there is no such commit,
so it returned [] -> assess set version_bump_plan=[] -> the executor
published a tag with no files bumped (a no-op masquerading as X.Y.Z).
Fall back to the version-reference scan when no prior release commit
exists: the files currently embedding the version are exactly the set a
first release must bump, and the set the first release commit then
records as canonical for subsequent releases. Read-only derivation; the
CEO-approval gate and fail-closed executor are untouched.
escalate_up had composes=() and no source-status guard, so a PM could
escalate a COMPLETED/CANCELLED task and apply_escalation set it back to
BLOCKED — bypassing the state machine's terminal-state invariant.
Defense in depth:
- spec: add PRECONDITION_NON_TERMINAL to escalate_up's extra_preconditions so
the lifecycle gate rejects terminal tasks (invalid_state) before the
journal:decision write fires; generalize _check_intent_preconditions to
honor non-tracing rejection_kind (not_authorized / invalid_state).
- service: apply_escalation (the single write primitive) returns False and
refuses to mutate a terminal task — covers the HTTP escalate route which
bypasses the spec gate. escalate() / escalate_up_to_role() return None on
refusal so the gateway emits a clean invalid_state envelope.
- route: the HTTP escalate route 409s a terminal task BEFORE sending the
escalation notification (so a finished task isn't yanked back, PM not pinged).
Two task creates for the same project/HEAD can race to populate the
conventions cache; the loser's INSERT fails the partial-unique index with
IntegrityError. A bare session.add + flush poisons the shared session (the
task-create transaction rides the same session), so every subsequent op
raises 'this session is in error state' and task creation crashes.
Run the INSERT in a savepoint (begin_nested) and swallow the IntegrityError:
only the savepoint rolls back, the outer transaction stays usable, and the
winner's row satisfies the next _cache_get.
NotificationService._create_notification built NotificationTable without
requires_ack, so the column default (True) applied to EVERY notification -
including informational REVIEW_REQUEST / DOCUMENTATION_REQUEST /
A2A_REQUEST / KNOWLEDGE_SHARE (ACK_REQUIRED_BY_TYPE -> False) and every
@mention from MessagingService._notify_mentions. Each false ack-required
inflated the recipient's unacked set and soft-blocked i_am_idle into
respawn churn.
- _create_notification: requires_ack=ACK_REQUIRED_BY_TYPE.get(type, True)
(unmapped types default True - preserve the action-required bias).
- _notify_mentions: requires_ack=False explicit (MENTION is informational).
TDD red->green (identity is False/is True assertions - the mocked
flush doesn't apply SQLA's insert-time default, so pre-fix the attribute
was None); ruff + mypy clean; notification suite green (18 passed).
A sibling's PR merging into the parent branch while a dev worked leaves the
dev's branch behind its base — the assembled PR then can't merge cleanly and
the sibling's changes go missing (the 2026-06-27 out-of-order dev-task break).
The behind-base gate refuses i_am_done in that state and steers the dev to
sync_branch (the Phase B1 gate-level rebase verb).
- GitService.is_behind_base: rev-list --left-right --count across
origin/{base}...origin/{head} → (behind, ahead); fetch-first so origin
reflects the pushed head. Raises on git failure (consistent with
rebase_onto_base); malformed stdout degrades to (0,0).
- Choreographer._behind_base_gate: wired into _i_am_done_gate after
_ensure_branch_pushed. behind>0 → invalid_state remediate→sync_branch.
Fail-open on git/base-resolution error (flaky fetch can't strand a task at
the submit gate — the merge layer has its own behind checks). Skipped for
branchless roots and protected bases (master/main/-prefixed).
Tests: gate (6: refuse+steer/up-to-date/branchless/protected/fail-open-base/
fail-open-git), is_behind_base (6: parse/up-to-date/malformed/argv-form/
requires-branch/missing-project). ruff + mypy roboco/ tests/ clean; unit green.
Kind 2 (cell-task wave chain): a new cell-task under root-subtask UT_n
depends on every cell-task under every root-subtask in UT_n.dependency_ids
(the kind-1 wave-chain edges), so its branch carries the previous wave's
merged cell work. Re-derived from the root-subtask's deps, not the cell-task's
own dependency_ids (which also carry UX/product-fanout edges the by-osmosis
edge must not pick up). A root may fan to several cell-tasks (different cells),
so the previous wave's cell-task is a SET.
Kind 4 (by-osmosis): the first dev task (sequence 0) under a cell-task depends
on each predecessor cell-task's tail (max-sequence) dev task, so the new wave's
first branch carries the previous wave's fully-merged tail. Subsequent dev
tasks inherit the tail via kind 3 or the merged base.
Both wired from _create_subtask_from_inputs, dispatched on parent.team
(MAIN_PM -> kind 2; cell team -> kind 4). Pure helpers
(cell_task_wave_chain_depends_on, by_osmosis_tail_dev_tasks) unit-tested in
test_sequencing.py; TaskService methods integration-tested. Idempotent +
best-effort throughout (add_dependency dedupes; missing predecessors are
no-ops). Also fixes a latent mypy-tests gap (estimated_complexity required on
direct TaskCreateRequest calls in the S2 tests).
Pure dev_task_collision_edges in sequencing.py turns a parent's surfaced
siblings into (depends_on_id, task_id) pairs via SequencingService. TaskService.
wire_sibling_collision_dag wires them through add_dependency (idempotent). The
choreographer calls it after each dev-task delegate so the sibling collision DAG
is built incrementally as the cell PM decomposes — file-overlap serializes,
migration chains, shared-last; stable (priority, sequence) ordering keeps edges
from flipping into reverse cycles on re-runs.
Fix (two layers):
1. Root cause — pr_merge and rebase_pr_for_task now take a required project_id and scope the lookup where(pr_number == X AND project_id == Y). Required so no caller can forget — the bug class can't recur. All 4 call sites updated (choreographer cell_pm_complete, the rebase-retry, the superseded close_pull_request now passes project_id, and _verb_runner._do_pr_merge).
2. Crash guard — _finalize_cell_complete None-checks the complete() return and returns a clean invalid_state envelope (with a remediate hint) instead of dereffing None → 500 → respawn loop.
Two real bugs surfaced running the full gate against a containerized
Postgres (and the orchestrator boot log):
1. Migration 052 crashed a real orchestrator boot with
'type "team" already exists'. The generic sa.Enum(create_type=False)
does NOT set the postgres enum's create_type attribute, so op.create_table
(checkfirst=False) emitted a redundant CREATE TYPE against the pre-existing
team enum. Switched to postgresql.ENUM(create_type=False) — the postgres-
native enum whose create_type _check_for_name_in_memos actually reads, so
the CREATE TYPE is suppressed. Verified: 051->052 upgrade against a DB where
the team enum pre-existed (the exact path that crashed) now succeeds;
downgrade 052->051 drops the table and preserves the shared enum; fresh
upgrade head clean. (Migration 016 has the same latent sa.Enum pattern but
never re-runs in prod, so it's noted, not touched here.)
2. _ensure_branch_for_task read task.cell_projects (lazy=selectin to-many)
directly, tripping MissingGreenlet on a freshly-created/unqueried task —
which then poisoned the async session (PendingRollbackError). Replaced with
_task_has_cell_map: peeks InstanceState.unloaded (no IO) and reads the
already-loaded map, falling back to an awaited count query only when the
relationship is genuinely unloaded. Non-ORM stubs route to the plain
attribute. Fixes 2 integration tests; the 6 cell-map unit tests still pass.
Also: typed the self stub as Any in test_choreographer_subtask_project
(mypy tests/ wants Choreographer, not SimpleNamespace) — the codebase idiom.
Gate: ruff format/check clean; mypy roboco/ + tests/ clean; full pytest
10371 passed / 388 skipped against containerized pgvector:pg16; vulture clean.
Pre-existing xenon C-rank on reassign (from prior commit 19a474d3, not this
feature) still blocks make quality — surfaced separately.
A MegaTask root-subtask can now target an ad-hoc per-cell project map — a
third targeting shape that mirrors the existing product fan-out root. In
RoboCo a project is per-cell (ProjectTable.assigned_cell); a monorepo is N
per-cell projects sharing one git_url. So 'multi-cell' IS 'multi-project',
and a task may mix per-cell projects across products or include OSS-library
projects not in any product.
Storage: migration 052 adds task_cell_projects (mirrors product_projects;
unique per (task, team)). TaskTable gains a cascade-delete cell_projects
relationship; TaskCreateRequest / TaskCreate / Task response carry the map.
Policy: batch.is_branchless_coordination + is_valid_batch_shape gain a
has_cell_projects param — a root-subtask targets exactly one of project /
product / cell-map; the umbrella still targets none. TaskService passes
has_cell_projects at every predicate call site and persists the rows in
create(). _ensure_branch_for_task cuts feature/main_pm/{root} per distinct
project in the map (via _distinct_projects_for_task); _require_target_or_umbrella
and _validate_batch_membership accept the map shape.
Fan-out: every distinct_project_ids site (task.py branch creation, routes
_project_for_complete + _resolve_project_for_merge, orchestrator
_ambient_projects_for_task, pr_review._project_slug_for, git._project_for_task)
generalizes to first-distinct-project-of-map-or-product. Choreographer
_resolve_subtask_project resolves a delegated subtask's cell from the parent's
cell map. The product-scoped _slugs_for_product intake helper is unchanged.
Intake: prompter._draft_cell_map extracts the per-cell map from the_work[].
_validate_batch_scope counts distinct projects across all drafts' cells
(>=2 min stays; one 2-cell draft satisfies it). create_task_from_draft
persists cell_projects for >=2-cell drafts (project_id/product_id None),
collapses a 1-cell map to the single-project shape, and leaves single-cell
top-level project_id drafts unchanged. _resolve_owning_team routes a
multi-cell map to Main PM (coordination root, like a product root — a cell
PM can't delegate cross-cell). propose_draft/propose_batch tool descriptions
declare the per-cell project_id (both Claude SDK + grok runtimes).
The umbrella stays branchless / pure-coordination / submit_root-rejected;
the CEO-escalation pr_number gate is not widened (the map root is
is_umbrella=False, mirroring a product root, so submit_root supplies it).
Single-cell root-subtasks and everything below them are byte-for-byte
unchanged. Un-run MegaTask waves (multi-cell drafts) become runnable.
* feat(release): add release-manager feature flag (default off)
* feat(release): change classification + semver-bump derivation
* feat(release): readiness audit (changelog/version-ref/docs/migration/gate)
* feat(release): release-manager engine proposes a gated release
* feat(release): fail-closed release executor (bump, gate, publish)
* feat(release): CEO approve/reject release-proposal surface
* docs(release): document the gated release manager
* feat(memory): add org-memory feature flags (default off)
* feat(memory): add playbooks table + status enum + migration
* feat(memory): playbook service with auditor curation transitions
* feat(memory): playbooks RAG index plugin
* feat(memory): index a playbook into RAG on approval
* feat(memory): distill a high-signal lesson at task completion
* feat(memory): keep private journal reflections out of the shared RAG corpus
* feat(memory): draft_playbook verb + auditor curation verbs
* fix(ci): resolve mypy tests/ errors blocking the gate (UUID casts, annotations)
* feat(memory): auto-inject similar lessons/playbooks into the briefing
* feat(memory): auditor playbook review queue (api + panel)
* docs(memory): document the org-memory loop + playbook verbs
* fix(provisioning): idempotent pitch provisioning (reuse product/project by slug on re-approval)
* fix(memory): add chunks_playbooks to the chunk schema + isolate release route tests
- Migration 030's CHUNK_TABLES was missing chunks_playbooks, breaking the
IndexType<->migration parity guard once the PLAYBOOKS index landed. The
upgrade is ALTER ... IF EXISTS so adding it is safe on any DB shape.
- The release-route fixture's approve/reject paths call db.commit() (real
behavior), so a held proposal outlived the per-test rollback and leaked
into engine tests that read the global list_open_release_proposals().
Tear down source=release_manager rows after each test.
- Make the gather_snapshot real-repo smoke version-agnostic (semver match)
so it stops pinning the literal repo version.
* chore(release): 0.13.0
* ++
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* feat(ci-watch): config flags
Default-off CI-watch config (mirrors self_heal_*): ci_watch_enabled,
ci_watch_default_workflow (ci.yml), ci_watch_interval_seconds (1800),
ci_watch_max_open_tasks (3), ci_watch_max_per_cycle (1). Registers
ci_watch_enabled in the panel FEATURE_FLAGS. 4 tests.
* feat(ci-watch): per-project ci_watch_enabled/workflow (migration 048)
Adds projects.ci_watch_enabled (bool NOT NULL default false) +
projects.ci_watch_workflow (varchar null) — the per-project opt-in for
multi-repo CI-watch. ProjectTable + Pydantic Project fields + migration 048
(off 047_ws_single_active). Real upgrade->downgrade->upgrade chain verified
against a throwaway Postgres; 2 ORM round-trip tests.
* feat(runtime): prune dangling agent images in the background sweeper
Every agent-image rebuild orphans the prior build's layers as an untagged
<none> image; across deploys these pile up (the operator hit ~80). The sweeper
now runs 'docker image prune -f --filter dangling=true' (dangling only — a
tagged image or one backing a running container is never dangling), throttled
to settings.image_prune_interval_seconds (default 6h) and gated by
image_prune_enabled (default on). Best-effort: any failure is logged, never
raised into the sweeper. Mirrors the transcript-retention prune. 4 tests.
* feat(ci-watch): source tag + open-task dedupe query
CI_WATCH_SOURCE='ci_watch' + TaskService.list_open_ci_watch_tasks(git_url=None):
non-terminal ci_watch tasks (the dedupe + open-cap basis), optionally scoped to
one repo by git_url — a monorepo registers several cell-projects on one git_url,
so dedupe keys on the repo, not the slug. 2 real-PG tests.
* feat(ci-watch): multi-project CI telemetry fan-out
MultiProjectCITelemetrySource.fetch(projects) reuses the hardened per-project
get_latest_ci_conclusion for each opted-in project (passing its ci_watch_workflow
or the configured default). Per-project isolation: a GitHub error or absent
signal yields NO sample (unknown, never read as green) and never aborts the
sweep; only a real conclusion yields a sample (fail→breach, pass→non-breach).
self-heal source untouched. 3 tests + self-heal regression green.
* feat(ci-watch): engine — fan-out, originate, dedupe, cap
CiWatchEngine.run_cycle(projects) mirrors SelfHealEngine: assess via
MultiProjectCITelemetrySource, open one PENDING ci_watch fix task per red repo
(team=main_pm, assigned_to=main-pm, confirmed_by_human=True so it dispatches
without an Approve-&-Start — the fe029fe3 lesson), never starts/approves/merges.
Dedupe per git_url (monorepo → one fix task per repo) + per-cycle/rolling caps.
Default-off; disabled → no-op. 5 real-PG tests (red→one task, dedupe, cap,
green/none→nothing, disabled).
* feat(ci-watch): orchestrator loop tick + watch-set loader
_ci_watch_loop (registered in start(), cancelled in stop(), separate from the
untouched self-heal loop): dormant unless ci_watch_enabled; each interval loads
the watch set (ci_watch_enabled projects, collapsed one-per-repo via the
existing _projects_one_per_repo) and runs CiWatchEngine.run_cycle, committing
opened tasks. _run_ci_watch_cycle extracted for testing; loud warning when
enabled-but-empty. confirmed_by_human=True on the originated task means it
dispatches without an Approve-&-Start (no stranding, the fe029fe3 lesson).
5 tests (disabled no-op, watch-set filter+one-per-repo, empty warn, engine run).
* docs(ci-watch): CHANGELOG + CLAUDE.md for multi-repo CI-watch
Document CI-watch (Added) in the CHANGELOG and the Self-Healing & Feature Flags
section of CLAUDE.md — it generalizes self-heal to opted-in projects, reuses the
hardened per-project CI lookup, never auto-merges, default-off. Adds the
ci_watch_enabled flag to the feature-flags enumeration.
* feat(dep-update): config flags
Default-off dep-update config (mirrors self_heal_*/ci_watch_*): dep_update_enabled,
dep_update_interval_seconds (604800 = weekly), dep_update_max_open_tasks (3),
dep_update_max_per_cycle (1). Registers dep_update_enabled in FEATURE_FLAGS. 4 tests.
* feat(dep-update): per-project dep_update_command/paths (migration 049)
Adds projects.dep_update_command (varchar null) + dep_update_paths (varchar[]
null) — the per-project opt-in for the dependency-update bot. ProjectTable +
Pydantic Project fields + migration 049 (off 048_ci_watch_project_cols). Real
upgrade->downgrade->upgrade chain verified on a throwaway Postgres; 2 ORM tests.
* feat(dep-update): source tag + open-task dedupe query
DEP_UPDATE_SOURCE='dep_update' + TaskService.list_open_dep_update_tasks(git_url=None):
non-terminal dep_update tasks (dedupe + open-cap basis), optionally scoped to one
repo by git_url (monorepo → one open dependency-update task per repo). 2 real-PG
tests.
* feat(dep-update): read-only lockfile-diff probe
WorkspaceService.dry_upgrade_changes_lockfile(project): clones the project's
read clone into a throwaway dir (--no-hardlinks, so the read clone is never
mutated), runs project.dep_update_command (no shell, shlex.split), and reports
whether any lockfile path (dep_update_paths or inferred uv.lock/pnpm-lock.yaml)
is dirty. Fail-safe: null/failing command → False (don't originate on a broken
probe), logged; throwaway always removed; never commits/pushes. 5 real-git tests.
* feat(dep-update): engine — detect, originate, dedupe, cap
DepUpdateEngine.run_cycle(projects) mirrors SelfHealEngine/CiWatchEngine: for
each opted-in project (dep_update_command set) with updates available (the
read-only probe), open one PENDING dep_update task (team=main_pm, assigned-to
main-pm, confirmed_by_human=True), never starts/approves/merges. Cheap checks
(command, per-git_url dedupe) before the expensive probe; per-cycle + rolling
caps. Default-off; disabled → no-op. 6 real-PG tests.
* feat(dep-update): weekly orchestrator loop tick
_dep_update_loop (registered in start(), cancelled in stop(), separate from the
self-heal + CI-watch loops): dormant unless dep_update_enabled; each interval
(default weekly) loads projects with a dep_update_command (one-per-repo) and runs
DepUpdateEngine.run_cycle, committing opened tasks. _run_dep_update_cycle
extracted for testing; loud warning when enabled-but-no-commands. Refactored
stop() to cancel background tasks via a shared _cancel_background_task loop
(keeps it under xenon B as the loop count grows). 4 loop tests.
Task 7 (anti-stranding dispatch guard) is satisfied by construction: no
dispatcher skip targets source='dep_update', and the engine sets
confirmed_by_human=True (the fe029fe3 lesson), asserted in the engine tests —
so the originated task dispatches via the assigned-PM path, never stranded.
* docs(dep-update): CHANGELOG + CLAUDE.md for the dependency-update bot
Document the dep-update bot (Added) in the CHANGELOG and the Self-Healing &
Feature Flags section of CLAUDE.md — read-only lockfile-diff probe, never
auto-merges, per-project opt-in via dep_update_command, default-off. Adds the
dep_update_enabled flag to the feature-flags enumeration.
* feat(ci-watch): route fix-task notification to the project's cell PM
On opening a fix task, CiWatchEngine notifies the red project's own cell PM
(resolved from project.assigned_cell via foundation AGENTS — e.g. BACKEND →
be-pm), not the CEO, once per project per cycle. Best-effort: a notification
failure never rolls back the origination. Adds _cell_pm_slug_for +
_notify_cell_pm. 1 real-PG test (asserts to_agent='be-pm', not 'ceo').
* feat(ci-watch,dep-update): expose per-project opt-ins in the project API
Add ci_watch_enabled/ci_watch_workflow + dep_update_command/dep_update_paths to
ProjectUpdate, ProjectUpdateRequest, the PATCH route mapping, ProjectResponse,
and project_to_response — so the panel edit-project dialog can read + set the
per-project autonomy opt-ins (the columns were unreachable through the API
before). Also threads the previously-dropped quality_command through the update
route. 1 real-PG update round-trip test.
* feat(ci-watch,dep-update): panel project-edit fields for the per-project opt-ins
Adds an 'Autonomous Maintenance' section to the edit-project dialog: a CI-watch
enable switch + workflow input, and a dependency-update command + lockfile-paths
input (comma-separated → list). Threads the four fields through the Project /
ProjectUpdate TS types and the mock-mode create fixture. The global on/off
toggles already live in Settings → Feature Flags; these are the per-project
opt-ins. panel tsc --noEmit + eslint green.
* docs(0.12): CI-watch + dep-update bot + image-prune across user docs + RAG
New docs/optional/autonomous-maintenance.md (mirrors self-heal.md) covering both
engines; optional/index rows; panel settings + projects-and-products notes for
the Feature Flags toggles + the edit-project Autonomous Maintenance fields;
resilience note for the dangling-image prune; env-reference + RAG config-reference
tables for all ROBOCO_CI_WATCH_* / ROBOCO_DEP_UPDATE_* / ROBOCO_IMAGE_PRUNE_*
vars; mkdocs nav entry. reflow-check green; prompts unchanged (operator-facing,
not agent-facing).
* chore(release): 0.12.0
Cut [Unreleased] -> [0.12.0] (CI-watch + dep-update bot + image-prune housekeeping
+ the post-0.11.1 run-hardening fixes). Bumps all 8 canonical version refs to
0.12.0 (pyproject / uv.lock roboco pkg / panel package.json / __init__ /
config.app_version + the README / deployment / agent-image-tag examples).
* fix(pr-review): repo-scope external-PR dedupe (no duplicate review on a monorepo)
external_review_task_exists keyed on (project_id, pr, head_sha), but a monorepo
registers several cell-projects on one git_url and the poll already collapses to
one canonical project per repo — so once a review task was re-pointed to a
sibling project, the next poll (checking the canonical project) no longer saw it
and opened a second review of the same PR (observed: PR #131 reviewed once on
guard-core-saas-frontend, once on -backend). Dedupe now spans every project
sharing the PR's repo (git_url); re-review on a new head SHA still works; a
genuinely different repo with the same PR number is independent. 3 real-PG tests.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* fix(git): don't delete a branch that still has open dependent PRs
Root cause of the run-zombifying "integration branch gone from origin" wedge.
_delete_remote_branch_best_effort deleted a merged PR's head branch
unconditionally, so:
- merging a cell->root PR deleted the cell branch while a sibling leaf PR was
still targeting it as base, and
- the CEO's root->master merge deleted the feature/main_pm/{root} integration
branch.
The dependent PRs lost their base, every later git op against the vanished
branch failed, and the task zombified (a51c3d31 only made the post-merge sync
non-fatal; this removes the cause).
The remote-branch delete chokepoint (the single path all merge/close/cancel
deletions funnel through) now first checks _branch_has_open_dependents: any OPEN
PR targeting the branch as its base marks it an active integration target and
preserves it. Fails safe (any error => keep the branch; cleanup is best-effort,
stranding is not). True leaf branches with no open dependents are still cleaned
up. Adds 6 unit tests for the guard + the probe.
* fix(git): recover a drifted shared clone on resume instead of BRANCH_MISMATCH
A dev/documenter/QA clone is shared across that agent's tasks. On a
respawn/resume it can sit on a sibling task's branch, or a re-provisioned clone
can lack the task branch as a local ref (commits only on origin). The
fresh-claim path git-resets the clone clean, but resume deliberately
short-circuits before it (_dev_reentry), so the agent's next commit hit
_assert_on_task_branch's BRANCH_MISMATCH, failed, and the task wedged in a
blocked respawn loop (the documenter that could never land its doc commit).
_assert_on_task_branch now recovers instead of only rejecting: fetch + checkout
the task branch (recreating a missing local ref from origin via `git branch
<b> origin/<b>`), and raise only when the switch genuinely can't happen
(uncommitted changes block it). Never discards work — checkout, not reset — so
a resumed agent's unpushed commits are preserved. Updates the RAG troubleshooting
+ developer docs to describe the auto-recovery. Adds 5 unit tests.
* fix(runtime): re-adopt running agent containers on restart (no double-spawn)
An orchestrator restart loses the in-memory _instances registry while the agent
containers keep running. The reaper already had a Docker-liveness fallback
(_assignee_container_running), but the spawn gate (_is_agent_active) did not, so
right after a restart it saw a live agent as inactive and could launch a second
container onto work the forgotten-but-running one was already doing.
start() now calls _readopt_running_agents() after _reconcile_orphan_claims_on_startup
and before the dispatcher/reaper loops launch: it probes each known agent slug's
container (AGENT_IMAGES, reusing _inspect_container_state — the same docker
inspect the reaper uses) and registers a minimal AgentInstance(state=ACTIVE) for
any that is running and not already tracked. Inert when nothing runs (cold start
unchanged); best-effort (a probe error leaves that slot for the reaper's own
fallback). This is the gateway-health spec's Task 4 / the orchestrator-state
spec's Phase 3 (_instances reconcile). Adds 4 unit tests.
* fix(git): treat an already-merged PR as idempotent success on merge
A merge PUT against an already-merged PR returns the same 405 as a genuine
"not mergeable" conflict, so _merge_with_retry raised MergeConflictError and the
completion path tried to rebase / close-superseded / escalate a PR that had
already landed (a prior cycle, a sibling, or the CEO merged it) — the
cell_pm_complete block<->unblock respawn loop.
_merge_with_retry now disambiguates before raising: a new _pr_is_merged probe
(GET the PR, check merged==true) returns success on an already-merged PR so
completion proceeds idempotently; a genuinely-unmerged 405 still raises the
conflict. Best-effort probe (False on any error → falls through to the existing
conflict handling). Adds 4 unit tests.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
When an integration (cell/root) branch is deleted from origin — e.g. after a sibling cell->root merge, stranding a late straggler leaf — pr_merge's post-merge _sync_target_branch ran 'git fetch origin <branch>' (check=True) and raised 'couldn't find remote ref'. That surfaced as a retryable SERVICE_ERROR, so complete() re-blocked the task and respawn-looped the PM on an already-landed merge (observed live on cell branches feature/backend/31ae12fc--0e49e04e and 7aeee245--dcfe9fc2, blocking be-pm's complete() 5+ cycles).
pr_merge reaches the post-merge sync only after the authoritative GitHub merge has already succeeded (_merge_with_retry raises otherwise), so refreshing the local workspace copy of the target branch is cosmetic. Route it through a new _sync_target_branch_best_effort that logs and returns None instead of raising. merge_pull_request (CEO path) keeps the strict sync — its target is the default branch, which always exists on origin.
make quality runs 'mypy roboco/ tests/'; the new PM-needs_revision claim test passed a SimpleNamespace where _get_valid_claim_statuses expects AgentTable | None, failing the type gate. The helper only reads agent.role, so cast the lightweight stand-in to AgentTable (no DB row, no type: ignore).
The lifecycle spec (CLAIM_RULES) grants CELL_PM/MAIN_PM claim of NEEDS_REVISION so a rejected coordination root (pr_fail / qa_fail / ceo_reject) can be re-claimed via i_will_plan and re-delegated. The runtime mapping _ROLE_CLAIM_STATUSES omitted it for PMs, so the spec gate allowed i_will_plan on a needs_revision root while the composed claim() rejected it -> returned None -> INVALID_STATE: the PM could neither plan nor idle its own rejected root and respawn-looped (observed live on cell root 0e49e04e, ~143 INVALID_STATE rejections across 11 PM sessions; the tail of the 2026-06-24/25 run).
Add NEEDS_REVISION to the cell_pm/main_pm runtime claim statuses, and a parity test locking _ROLE_CLAIM_STATUSES to lifecycle.CLAIM_RULES so the two can't drift again.
A parent's acceptance-criteria coverage is matched by stable criterion id, but
a PM may declare covers_parent_criteria on a child by EITHER the criterion's id
OR its full text (both happen in practice). _parent_ac_ref_sets unioned the raw
refs and matched by id only, so a COMPLETED child that declared coverage by text
was invisible to the matcher: the criterion read "uncovered", the roll-up gate
refused, and the PM re-delegated the already-finished work as a brand-new empty
subtask (0 commits, no PR) that can never close — looping for hours and burning
tokens (observed live: a parent's xenon work completed + merged via one child,
then re-delegated 2h later as an empty phantom).
Normalize every child ref to the criterion id (text -> id via the parent's own
criteria) in a small _normalize_ac_refs helper, so coverage counts regardless of
how it was declared. Fixes existing mismatched data and future declarations; all
three consumers (uncovered/unclaimed/parent_ac_coverage) share the builder. An
unknown ref (neither id nor a current criterion text) passes through and matches
nothing, exactly as before. Adds two regression tests.
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
Forensic triage of a 24h run reconstructed the dominant gateway.rejected
loops from the audit_log. After earlier deploys fixed the i_will_plan crash
and the open_pr push-gap, three real, recurring-capable burn loops remained.
This fixes them at the architecture level, not by prompt-nagging.
journal:decision write-then-gate (the dominant completion-path blocker):
PM decision-point verbs required a separate note(scope='decision') call
before the verb, which loaded/weak models forget to chain — so complete and
unblock hit a tracing_gap (journal:decision missing) and respawn-looped,
stranding finished tasks forever. Each verb now auto-records its OWN
rationale as the journal:decision before the gate runs (the proven
i_am_blocked -> write_struggle pattern), so the gate passes off real,
persisted reasoning. unblock gains a required `reason` (threaded MCP tool ->
request schema -> routes -> choreographer); delegate derives the decision
from its title + description; complete/submit_up/submit_root/escalate_up/
escalate_to_ceo reuse their existing notes/reason. The gate still runs as
defense-in-depth; the auto-record is idempotent within the decision window
and best-effort. Adds JournalService.write_decision and
Choreographer._ensure_pm_decision.
open_pr empty-diff 422: an overlapping-decomposition leaf with zero commits
vs its base makes GitHub 422 "No commits between ...". The generic
invalid_state "retry" looped the dev 15x on one task. open_pr now steers to a
terminal i_am_blocked hand-off so the PM completes or cancels the redundant
leaf.
owns_task stale-agent loop (41x): a superseded agent (task reassigned away)
calling i_am_done/open_pr got a PRECONDITION_OWNERSHIP tracing_gap it read as
a fixable precondition and retried forever. Both verbs now short-circuit with
the clear not_authorized "no longer yours -> give_me_work" steer that
resume/unclaim already use.
RAG docs updated for the new unblock(reason) signature; CHANGELOG entries
added under 0.11.0 (unreleased). open_pr refactored into
_open_pr_preflight_rejection + _open_pr_failure_env to stay within the
return-count and complexity budgets.
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
_sync_target_branch previously did a bare 'git checkout <target>' with no
fallback. In the shared-agent-clone model the leaf developer's workspace
often has only the task branch locally; the parent/cell branch exists only
on origin. That produced a SERVICE_ERROR which cycled the task back to
blocked every time the PM retried complete().
Now, when checkout fails, we fetch the target branch from origin and create a
tracking branch before the pull. Includes regression tests for the local,
fallback, and origin-missing paths.
* fix(run-hardening): stop three blocked-task respawn loops
Three independent fixes for blocked-task respawn loops observed in the live
run (the bleeders behind a wedged near-complete run):
- verb runner: re-check the working task after EACH composed atomic action,
not just at entry. A concurrent transition between a verb's precondition
gate and execution (e.g. a racing i_am_blocked moving a root from
needs_revision to blocked) made claim() return None mid-sequence; the next
composed step dereferenced None.id and crashed with the opaque
"'NoneType' object has no attribute 'id'", looping the PM. Now fails fast
with an actionable INVALID_STATE; the savepoint rolls the partial run back.
- blocker dispatch: never dispatch a Board role (product-owner / head-
marketing) as a blocker resolver. Board roles have no unblock verb, so the
dispatcher respawned one forever to "resolve" a blocker it could only
notify/triage about — one incident burned ~6400 tool calls on a single
mis-owned root. _blocker_resolver_slug now returns None for a Board
assignee so the dispatch skips it.
- git push: recover a missing local task-branch ref from origin/<branch>
before push-by-name. A re-provisioned shared clone can lack the branch
locally though its commits are on origin, so push died on
"src refspec <branch> does not match any" and the task wedged at i_am_done.
Now materializes the ref (no-op push when already on origin) or fails loud
with an unclaim+reclaim instruction when the work is on neither.
Adds regression tests for all three. Full no-DB gate green (ruff, reflow,
mypy, xenon); pytest+coverage validated by CI.
* fix(verb-runner): only raise on an INTERMEDIATE composed None, not the last
The mid-composition None-guard was too aggressive: it raised for a None
returned by the LAST composed action too (e.g. start()), preempting the
caller's existing `if task is None` handler that surfaces the verb-specific
message ("start failed for task ...", the board verb's decline envelope).
Three tests asserting those messages broke in CI.
Only an INTERMEDIATE None is fatal (the next action would deref None.id). A
None from the last action is the verb's own result and must flow out as the
runner's return value. Guard now fires only for position > 0, before the
next dispatch — still prevents the crash, preserves the last-action contract.
* fix(escalation): never hand a Main-PM coordination root to the Board
The upstream cause of the board catch-22 (which the orchestrator-side
blocker-dispatch guard only backstopped): the escalation chain points
main-pm -> product-owner, and i_am_blocked/escalate REASSIGNS the task to
that chain target. apply_escalation's board-advisory guard only refused
descendant cell tasks (both predicates require parent_task_id), so a
top-level Main-PM coordination root slipped through and the whole root was
reassigned to the Product Owner + marked blocked. The board has no unblock
verb, so it spam-notified the CEO and respawn-looped (~6400 tool calls on
one root).
Add _is_coordination_task (team == main_pm — covers a delivery root AND a
MegaTask root-subtask) and a shared _board_cannot_own predicate, applied at
all four board-refusal sites (escalation, reassign, reassign_active_claim,
dependency-revival). A main_pm coordination task escalated/reassigned onto a
board role is now diverted to the pool for a role-matched (Main-PM) reclaim.
Complements the blocker-dispatch backstop in the prior commits (defense in
depth). Tests: coordination-root predicate cases + apply_escalation divert;
existing teamless-root / board-root behavior unchanged.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
Renn F