increment_probe_failures / reset_probe_failures did a non-atomic
get_state (GET) -> mutate -> set (SET) in Python. A concurrent
activate() re-park writes a FRESH episode blob (probe_failures: 0 +
fresh activated_at / retry_after / affected_agents / kind); if the
stale increment's SET landed after the fresh activate's SET, the stale
blob overwrote the fresh episode metadata AND un-reset the counter
(clobbering the new episode).
Redis single-threads a Lua EVAL, so a server-side read-modify-write
is indivisible: activate's SET is serialized entirely before or after
the script, never interleaved between the script's GET and SET. The
two scripts mutate ONLY probe_failures, so every other episode field
survives the bump. activate stays a single atomic SET (a fresh episode
resetting the counter to 0 is correct semantics).
The pr_pass/pr_fail ActionSpecs carry self_review_block=True, but
_gate_preflight never populated Context.original_developer_slug, and
actor_slug was read off agent.slug — which GatewayAgentView does not
carry, so it was always None in production. The block was structurally
dormant: a reviewer who was also the original developer of the
assembled PR could pass (or fail) their own work. The service-layer
_validate_not_self_review backstop only covers qa/documenter, not
pr_reviewer, so the spec gate is the only defense.
Set actor_slug=str(reviewer_agent_id) (GatewayAgentView has no slug,
so the UUID is the identity) and original_developer_slug from the
original_developer marker (a UUID stored as a string). Both resolve to
UUID strings, so the spec's string-equality comparison fires when the
reviewer IS the recorded original developer.
The marker is never set on assembled coordination tasks (only on
dev-leaf tasks at QA/doc claim), so the block stays dormant by design
in production — but the gate is now correctly wired to fire if the
marker were ever set to the reviewer. Zero production behavior change;
the dormant-in-production state is pinned by the no-marker test.
resolve_wait deleted the waiting record (in-memory + durable) BEFORE calling
spawn_agent. A re-park in the window between the probe-success clear and the
spawn — the provider's rate limit lifts then immediately re-limits, or a second
provider limit lands — bails spawn with an OFFLINE instance (the parked-provider
short-circuit). Deleting the record first orphaned the agent: with no record
the probe-resume loop can never revive it and the spawn gate bails every tick,
so the agent is lost until the operator intervenes.
Fix: spawn first, then tear down the record only once a container actually
launched (instance.state == ACTIVE). On an OFFLINE bail the record stays so the
next probe-success re-attempts the resume. On a spawn EXCEPTION the record is
torn down + re-raised so the probe loop doesn't keep re-resuming a task that
moved to a different state (e.g. readiness refused -> task auto-blocked) —
matching the pre-fix behavior where the record was deleted before the spawn.
_probe_target returns (None, {}) for grok — the grok CLI's xAI endpoint is
closed and the SuperGrok OIDC access token is not a valid bearer for the metered
api.x.ai, so a real probe would either no-op or strand grok parked forever.
_do_probe treats url-is-None as success (time-expiry optimism), so once the
60s retry_after passes the probe loop optimistically clears the grok park, a
cleared park dispatches a fresh grok agent that hits the still-active xAI 429,
exits 75, and re-parks — a flat ~90s crash-retry cycle for the whole xAI
rate-limit window (each cycle costs container startup + a rejected grok call).
Fix: track _grok_repark_count + _grok_last_park_at in _park_grok_rate_limited
and back the re-park retry_after off exponentially within one episode
(60 -> 120 -> 240 -> ... capped at 2**4 = ~16min cycle) so the churn dampens. A
gap past _GROK_REPARK_EPISODE_GAP_S (25min, > the capped cycle) means no re-park
for that long => the rate limit actually lifted => a fresh episode resets the
count to the base 60s, so recovery latency isn't penalized across episodes.
The first park in a fresh episode is unchanged at 60s.
_persist_respawn_record is fire-and-forget per gate mutation; a respawn loop
fires count 1->2->3->4 in quick succession, scheduling one persist per
increment for the same (agent_slug, task_id). The ON CONFLICT DO UPDATE upsert
is row-level race-free, but the fire-and-forget tasks can still COMMIT out of
order: a slow stale persist (count=2) scheduled first can resolve AFTER a fast
fresh one (count=4) scheduled second, leaving the durable row at the stale low
count and re-burning the strike threshold on restart.
Fix: acquire self._respawn_persist_lock (new asyncio.Lock) as the FIRST await
in _persist_respawn_record, so acquisition order = task creation order (FIFO
ready queue) = logical schedule order, and commits land in that order. The
durable row always ends at the latest logical value. The lock lives in the bg
task, so the dispatcher hot path never blocks; persists are best-effort and
a slow one queuing the rest just delays the durable catch-up (in-memory record
stays authoritative).
spawn_agent ran the full _prepare_agent_spawn (writes blueprint/settings/
briefing/MCP files, ensures the image, registers a STARTING instance) every
dispatcher tick only to bail at the after-prepare parked-provider check —
wasting all that file I/O while the provider stayed parked and leaving a
STARTING instance registered then downgraded to OFFLINE.
Move the parked check before _prepare_agent_spawn: resolve the route cheaply
via _resolve_agent_route (only provider_type is needed) and bail with a
minimal unregistered OFFLINE instance. The existing-running check stays
first (inside the lock) so a live agent is never replaced; a TOCTOU
re-check guards the unlocked window before prepare; the after-prepare
check is kept as a rare-race defense (a park landing during prepare).
_on_probe_failure only incremented the failure counter and, at 10 failures,
sent a one-shot CEO notification. It never cleared the tracker, never gave
up, never fell back to time-expiry. _do_probe returns False for any non-2xx
AND any httpx error, so a permanently unreachable probe endpoint (removed
API key, network partition to the probe host, misconfigured base URL) kept
the provider parked forever — every agent on it gated by
_provider_spawn_parked, their tasks reaped to pending but the spawn gate
queuing every spawn, sitting pending forever. The only recovery was the
operator manually clearing the Redis key.
Past _PROBE_GIVE_UP_THRESHOLD (30) persistent failures, fall back to the
same time-expiry optimism the unprobeable-provider path uses (_do_probe
returns True when there is no probe URL): clear the park and resume parked
agents. If the provider is genuinely still down the real workload attempts
re-park via the 429/5xx path, so this is bounded burn — strictly better
than a silent forever-strand. Kept above the CEO-notify threshold (10) so
the operator still gets the notification first.
The intake and secretary agent ids are each a single fixed id, so two
concurrent start_intake_session / start_secretary_session calls raced on
the container name (docker run --name roboco-agent-<id>) and the
_instances[<id>] write: both passed the reap-prior check before either
registered, both ran docker run, and the last _instances write won,
orphaning the other container + its relay.
Add _intake_spawn_lock / _secretary_spawn_lock (asyncio.Lock) and wrap the
_spawn_intake_container / _spawn_secretary_container bodies so the second
start waits for the first to fully register before its own reap-prior check
runs. Distinct from self._lock (which stop_agent takes) to avoid a
reentrancy deadlock: the spawn body holds the spawn lock then calls
stop_agent (acquires self._lock) — lock order is always spawn_lock ->
self._lock, never the reverse.
xAI's refresh-token response sometimes omits expires_in. Without it the
new access token kept the stale pre-refresh expires_at, so is_valid /
--check forever rejected a fresh token — and the refresh loop re-rotated
the single-use refresh token every tick, killing the credential (F006).
The access token is a JWT whose exp is the authoritative expiry: decode it
when expires_in is absent. Fallback to the documented ~6h TTL + a structlog
warning when the JWT exp is unreadable, so a fresh token is treated as live
instead of stale.
GrokCliProvider._append_grok_auth_mount silently skipped the mount when
the host ~/.grok/auth.json was absent. The spawn still succeeded (docker
run returned 0 — the container was created), so the operator had no
spawn-time signal that the agent was doomed: the entrypoint's
`python -m roboco.llm.providers.grok_auth --check` backstop then
refused to start (exit 78) and the failure only surfaced later via the
container's log markers.
Fix: emit a spawn-time WARNING (module logger) naming the missing file
and the remediation (`grok login` on the host, or set
ROBOCO_HOST_GROK_DIR) when the mount is skipped. The spawn outcome is
unchanged — the container still starts and the existing exit-78 -> park
flow (F041) still catches it — but the operator now sees the missing
credential immediately instead of diagnosing a later exit-78.
Logical-regression check: the mount-present path is byte-for-byte
unchanged (auth.json exists -> the -v bind is appended, no warning); the
spawn still succeeds when auth is absent (no raise — the existing
test_grok_spawn_omits_auth_mount_when_absent still passes: no mount, no
crash); the exit-78 entrypoint backstop and the orchestrator's
exit-78-park handling (F041) are untouched; a module-level logger adds no
side effects. Tests: new test_grok_spawn_warns_when_auth_absent uses
caplog to assert a WARNING mentioning auth.json + `grok login` is
emitted on a missing-credential spawn (RED before: no warning; GREEN
after). 102 grok tests green; ruff/mypy clean.
The auditor is a silent, read-only observer on every channel, but the
channel catalog (roboco/foundation/policy/communications.py) listed it
in write_roles for main-pm-board and board-private 'for parity' with the
legacy CHANNEL_ACCESS table, while the actual silent-observer rule was
enforced only at the say/dm guard (content_actions._NO_COMMS_ROLES) and
PermissionService.can_write_channel's auditor short-circuit.
That left the catalog-only enforcement path — the HTTP messaging route
(messages.py send_message -> validate_channel_access) — authorizing an
auditor write that both the say/dm guard and PermissionService would
have blocked. A reader of the catalog also believed the auditor could
post to those channels, which is false.
Fix: remove Role.AUDITOR from write_roles on both channels (main-pm
+ board remain writers; ceo remains a writer on board-private). The
auditor stays in read_roles, so its silent read is unchanged. silent_roles
is left empty (matches the announcements precedent: auditor reads via
read_roles, not the silent bucket) — the DB seed and silent_observers
field are untouched.
Logical-regression check: the auditor's read access on both channels
is byte-for-byte preserved (still in read_roles, so validate_channel_access
read returns True via the direct list); the legitimate writers (main-pm,
product-owner, head-marketing, ceo) are untouched; CHANNEL_ACCESS is
derived from the spec so the foundation/seed drift tests self-adjust;
PermissionService.can_write_channel already short-circuited auditor to
False everywhere, so no behavior change there; AUDITOR_SILENT_ACCESS is
unchanged (auditor not added to silent_roles -> no DB silent_observers
change -> no group-access behavior change); the say/dm _NO_COMMS_ROLES
guard is unchanged. Tests: 3 new in test_channel_access.py — auditor
write on main-pm-board/board-private now raises ChannelAccessDeniedError
(RED before: returned True), auditor read still True, main-pm/ceo still
write.
The work_sessions.agent_id column is nullable=True with ondelete=SET
NULL — deleting an agent nulls the FK on every session it ever held. The
ORM annotation lied (Mapped[UUID] non-optional), the converter papered
over the lie (typing_cast to a non-optional UUID), and the response
model rejected None outright (WorkSessionResponse.agent_id: UUID). A
session whose agent had been deleted crashed the GET endpoint with a
pydantic ValidationError instead of serializing agent_id: null.
Make the read path honest end-to-end:
- WorkSessionTable.agent_id: Mapped[UUID | None] (matches the column).
- WorkSessionResponse.agent_id: UUID | None (serializes null, no crash).
- session_to_response passes agent_id via typing_cast('UUID | None', ...)
to bridge SQLAlchemy's UUID[Any] to stdlib uuid.UUID while preserving
None-ness (the cast stays for the same mypy-plugin reason every other
field uses one; it no longer narrows away None).
WorkSessionCreate.agent_id stays UUID — at create time the claiming
agent is always known. The unused WorkSession pydantic read model is
left as-is (never materialized from a DB row). task.py:_needs_revision_dev
already None-guards ws.agent_id via to_python_uuid (returns None -> skip).
Add a CEO-bound, header-token-only gate (require_panel_token) at the route
level of the prompter_live + secretary_live bridges, which were the only
panel-facing API surface that ran unauthenticated. It mirrors the WS
_require_panel_token and _check_agent_auth_token contracts: in dev
(ROBOCO_AGENT_AUTH_REQUIRED unset) a missing token is allowed; a
presented-but-forged token is rejected even in dev; in prod nginx already
injects the CEO-signed X-Agent-Token on /api/ for GET + POST, so the SSE
stream (EventSource can't set headers) and the POSTs are now checked instead
of anonymous. Applied to start/stream/status/messages/stop on both routers;
preview_live_batch switched from CurrentAgentContext+noqa to the route-level
gate (genuinely auth-only). confirm/confirm-batch/re-interview keep
CurrentAgentContext (they use agent.identity). The container->relay /events
callback is intentionally left ungated (internal Docker network, opaque
session id) — gated by a test sentinel so Option B (spawn+SDK token wiring)
is a deliberate future decision. No panel/nginx/spawn/SDK changes; master
merge invariant untouched. 22 new TDD auth tests, 492 api tests green.
rebuildCellWork appended a blank {summary:'', items:[]} entry for a newly-
selected cell, so toggling a cell's project OFF then back ON in the MegaTask
review card discarded the agent-authored per-cell summary/items — the entry
was dropped on toggle-off and re-added blank on toggle-on.
Park each draft's last per-cell content in client-only BatchProposal state
(parkedCellWork, keyed by draft index — never sent to the backend; confirm
ships only title/drafts/project_ids/route, and it ride-alongs into the
localStorage persist slice so the restore survives a reload mid-review).
rebuildCellWork gains an optional priorByCell map: a re-added cell with no
live entry restores its parked summary/items (with the new project_id) in-
stead of blanking; a live entry still wins over a stale parked copy so an
in-place edit is never regressed. parkCellWork is the pure merge seam
(prevParked seeds, live work overwrites) the setBatchDraftProjects updater
calls — kept pure so the updater stays a thin caller.
Tests: rebuildCellWork restore/blank-fallback/live-wins + parkCellWork
retain/overwrite/merge (6 new), 19 GREEN. eslint/typecheck/prettier clean.
No wire-payload change, no regression to the fill/drop/one-repo-per-cell
invariants.
A drag on the operator kanban routes the status move through the admin
status-override, which bypasses the in-band lifecycle validator entirely.
That override is intentional (it's how an operator recovers a wedged task)
but it also let a careless drag skip material preconditions silently —
completing a task with no open PR, QA-bypassing, finishing docs on a task
whose docs aren't complete.
Leave the override intact but make the bypass explicit: compute the
preconditions the dragged move would skip (open PR, docs complete,
self-verified + commits + progress for submit-qa, visible non-terminal
subtasks for coordination-root targets) and, when any are skipped, hold the
move behind a confirmation dialog that lists exactly what's being skipped.
Precision over recall — only warn on what the panel can verify from the
task and its in-list children; never fabricate a 'satisfied' claim, and
stay silent on benign transitions that gate on nothing we can check.
The admin status-override capability is preserved (Confirm still fires it);
this only surfaces the bypass instead of letting it happen silently. Does
not touch the master-merge invariant — the board's updateTask is the
operator override, not the Main-PM merge path.
validate() only checked 'at least one of project/product', so the dialog let
both be submitted together. The server silently lets product_id win at routing
and drops project_id, recording a misleading, never-used repo. Add a validator
that refuses the ambiguous submit with a clear error. The at-least-one rule and
the single-pick submit paths are unchanged.
FeatureFlagsCard disabled every switch while any one flag toggle was pending,
and PlaybookReviewQueue disabled every row's Approve while any one approve was
pending — so the operator couldn't act on an independent control during a
slow round-trip. Gate the disable on the in-flight mutation's variables
(matching key / id) so only the control being mutated locks; the others stay
usable. The same-flag double-tap protection is preserved.
The hook synced wsState into the store but never dropped usageData when the
stream dropped, so on reconnect wsState flipped to "connected" before any
fresh USAGE_SNAPSHOT arrived and UsageOverviewPanel rendered the prior
session's totals/cost as if they were live. Clear usageData whenever state
leaves "connected" so the panel falls back to the polling summary until a
new snapshot lands. Connected->connected is a no-op clear skip.
The card collapsed any non-404 backend failure (500 / network drop) onto
`!proposal` and returned null, so the CEO had no idea the release-proposal
endpoint was unreachable. Distinguish the cases: isError + a Retry affordance
vs the 404 null empty state that stays hidden. Mirrors PrReviewQueue.
The CEO Approve dialog's notes label fell into the default branch
('Notes (optional') for the approve action, but approve actually
requires substantive notes >= 20 chars — enforced client-side
(toast error on < 20) and server-side. So the CEO was told 'optional'
and only learned the real requirement from a toast after hitting
submit with empty notes.
approve and start both require >= 20 chars; reject only requires a
reason. Collapse the label to two branches: reject -> 'Reason for
rejection (required)'; everything else (approve + start) ->
'Approval notes (required, >= 20 characters)'. The approve
placeholder now also signals intent ('Why this is ready to ship...').
Tests: render the queue, click Approve, assert the notes label says
'required' + '20' and does NOT say 'optional'. RED: label read
'Notes (optional)'; GREEN: 'Approval notes (required, >= 20
characters)'. eslint/typecheck/prettier clean.
openStream registered listeners for the server-sent event kinds but not
the EventSource's own transport-level error. The 'error' kind IS in
LIVE_EVENT_KINDS, so a server-sent event:error (JSON MessageEvent) was
handled — but a dropped connection / dead session fires a plain Event
with NO data, which JSON.parse(undefined) swallowed in the try/catch,
so the stream 'stayed open' (EventSource loop-reconnected a session that
no longer existed) and isSending stayed true — the composer was
permanently disabled.
Fix: route the 'error' event by payload. A MessageEvent with string
data is a server-sent error → handleEvent (unchanged). A no-data Event
is a transport error → handleTransportError: clear streamingId/activity,
set isSending false, add a 'connection lost' error message, keep a
draft/batch preview up (so the human can still act on a proposed card)
else land on 'chatting', and close the dead stream so EventSource stops
loop-reconnecting.
Tests: renderHook + a jsdom EventSource double that fires a transport
error (plain Event, no data) vs a server-sent error (MessageEvent +
JSON). RED: transport error left isSending true; GREEN: resets to
false, surfaces the message, closes the stream. The server-sent-JSON
path is unchanged. Full panel suite (129) green; eslint/typecheck/prettier clean.
_run_claim_guards read the agent's other tasks via unlocked SELECTs
before claim() took its row lock, and claim()'s FOR UPDATE locked only
the TARGET row — so two concurrent i_will_work_on by the SAME agent on
TWO DIFFERENT pending tasks each locked their own row, each read an
empty in_progress set, each passed already_active, each claimed+started
→ the agent ended with two in_progress tasks (the in-process asyncio
Lock is lost on orchestrator-restart split-brain, so it wasn't a
DB-level guarantee).
Fix: TaskService.acquire_claim_lock takes a transaction-scoped
pg_advisory_xact_lock keyed by hashtextextended(agent_id). The gate
acquires it BEFORE the guard reads (for non-coordinator roles only) so
the second concurrent claim's read sees the first's committed
in_progress task and is rejected. Tx-scoped → auto-releases on
commit/rollback, can't outlive the request.
Coordinator exemption (the key logical-regression guard): cell_pm /
main_pm do NOT take the lock — the PM coordinator concurrency feature
lets a PM plan+delegate many roots in parallel, and a per-agent lock
would serialize those claims and regress it. Matches the existing
_COORDINATOR_ROLES already_active/paused guard exemption. A hash
collision only causes benign false serialization, never a false
negative.
Tests: unit (dev acquires lock before guard read; coordinator does
not) + real-PG integration (same-agent serializes, different-agent
does not, releases on rollback).
The non-blocking spawn (start_intake_session / start_secretary_session)
schedules _spawn_intake_container_guarded / _spawn_secretary_container_guarded
via _schedule_bg. Those run docker run and only register in _instances at the
END. If shutdown arrived between docker run and the registration line, the
container was started but the orchestrator had no handle — stop() iterates
only _instances, so the container was orphaned (leaked, manual docker rm).
Worse, the F070 drain could let the spawn coroutine complete the
registration AFTER stop() already iterated _instances, landing a live
container into a shutting-down registry nothing tears down.
Add a post-docker-run shutdown guard in _spawn_intake_container and
_spawn_secretary_container: re-check self._running after _run_container_cmd
returns; if the orchestrator began shutting down, remove the just-started
container (by its deterministic name) and raise _SpawnAbortedDuringShutdown
WITHOUT registering. The guarded wrappers catch that BEFORE except Exception
and close the live relay silently (shutdown is not a user-facing failure,
no error pushed to the SSE stream). The F070 stop() drain awaits the bg
spawn coroutine, so the abort surfaces cleanly.
TOCTOU-safe: between the _running check and the _instances assignment there
is no await (config + instance construction are sync), so once the check
passes, registration completes before the event loop can interleave stop().
The normal running path is unchanged (sanity tests pin it).
Orchestrator.stop() cancelled only the named loop tasks + agents, then
returned, abandoning in-flight _schedule_bg work. An in-flight
_persist_respawn_record upsert dropped at shutdown meant the last few
gate-mutation strikes never reached the DB; restore_respawn_tracker() on
the next start repopulated a stale lower count and the dispatcher re-burned
the full 4-spawn strike threshold against a still-wedged task — the exact
re-burn the durable tracker exists to stop. Audit-log writes (load-bearing
for cycle-time/rework metrics) were similarly dropped.
Add _drain_bg_tasks(): bounded wait (5s default) lets short DB writes
commit before exit (data preserved), then cancels any stuck task past the
deadline so a hang can't wedge shutdown. return_exceptions=True so one
failing bg task doesn't crash the drain. Wrap the stop_agent loop in
try/except + logger.exception so one bad agent can't skip the drain
(re-introducing the data-loss tail). Floor test pins the deadline >= 3s
so a too-short change can't silently drop a legitimate slow write.
The say()/dm() defence-in-depth guard only rejected auditor, but CLAUDE.md
mandates the same no-agent-comms invariant for pr_reviewer (posts findings
on the PR), prompter and secretary (human-only, note + evidence). For those
three the manifest was the only gate, so a call bypassing the manifest
(direct API POST, test harness, future routing change) would not be refused
at the handler — admission depended on the agent's slug happening to be
absent from the channel/a2a matrix. Extend the guard to a _NO_COMMS_ROLES
frozenset (auditor + pr_reviewer + prompter + secretary), matching the
explicit role-frozenset gates on commit/notify/pitch/playbook/open_session.
Role-appropriate remediation per role. The claimed defence-in-depth now
covers 4 of 4 silent roles, not 1 of 4.
A hung Docker daemon (or a stuck container FS) froze the single asyncio
event loop: the reaper runs inline before every dispatch tick and shares
that loop with every background sweeper. Bound each docker subprocess
with asyncio.wait_for; on expiry proc.kill() the child and either raise
(inspect / resolve_container_id — callers apply their own fail-direction)
or return None (the gateway probe — inconclusive, caller declines to act,
matching its existing probe-failure contract). Deadlines generous
(10s inspect / 30s exec) so a legitimate slow docker call is never
wrongly aborted; floor-assertion tests pin the floors.
Also harden _check_health's per-agent loop so one agent's hung inspect
skips that agent, not the whole sweep — preserving the check-all-agents
invariant the timeout-then-raise would otherwise break (without this, a
hung daemon means no agent gets health-checked any tick).
A hung git/make/gh/clone would block the CEO-gated release loop
indefinitely. Wrap each proc.communicate() in asyncio.wait_for via a
shared _await_proc helper; on expiry proc.kill() the child and return a
non-zero rc (124) so every caller's fail-closed branch fires. Mirrors the
quality-gate _run_one kill-on-timeout idiom.
Deadlines are generous (30min gate / 10min clone / 5min push+gh) so a
legitimate slow op is never wrongly aborted — floor-assertion tests pin
the floors to guard exactly that logical regression. Green path returns
the real rc unchanged.
channel_stream called validate_channel_access, which HTTP-loopbacked to
GET /api/permissions/check — a route that does not exist. Every call 404'd
-> False -> the channel stream closed with WS_1008_POLICY_VIOLATION for
EVERY client, so the real-time channel stream was dead. Removed the
function, its call site, and the now-unused httpx + settings imports.
Post-F004 the panel-token gate is the channel-stream authorization (the
CEO panel is the sole WS client and may view every channel), so the
broken loopback is removed rather than replaced with an in-process check
the CEO always passes. The legitimate enforcement.validate_channel_access
(slugs, in-process static ACL) is a different function and is untouched.
F027 is resolved-by-F004 (no code change): all three per-agent streams
gate on _require_panel_token first, so only the authorized CEO panel can
connect — 'any viewer subscribes to any target' is closed.
TDD; ruff/mypy clean; 530 unit/api+enforcement+RBAC tests green.
The validator tags a .tsx file as language 'tsx' (the JSX grammar needs
that tag, distinct from plain 'typescript'), but a custom rule scoped to
'typescript' — the language the scan reports for a React+TS repo — silently
skipped every .tsx file. The two suffix maps were NOT unified: the 'tsx'
tag is load-bearing (grammars.py picks the JSX grammar on it; hygiene.py
keys on it), so unifying would make .tsx fail to parse.
Fix is in check_custom: a one-directional dialect map _DIALECT_OF =
{'tsx': 'typescript'} — a typescript-scoped rule fires on a .tsx file,
but a tsx-scoped (JSX-only) rule still does not fire on plain .ts.
TDD; ruff/mypy clean; 80 unit + 38 integration conventions tests green.
- middleware: redact known credential fields (git_token/api_key/token/...)
from the 422 request-validation log line; response body unchanged
- a2a: require_any_authenticated_agent on /message/send + /message/stream;
subscribe_to_task opens a short-lived session per poll instead of holding
one asyncpg connection for the full SSE lifetime (pool exhaustion) + auth
- dashboard: gate auditor flag/report mutating routes to Auditor or CEO
- orchestrator: router-level CEO gate on all control routes (spawn/stop/...)
TDD; ruff/mypy clean; 449 unit/api tests green; no type:ignore/noqa.
F064: the bridge forwarder awaited every conn.send_text in a gather with no
per-connection queue and no send timeout — one slow WS client back-pressured
ALL event delivery to ALL clients (head-of-line blocking on the listen loop).
Each connect_* now registers a _ClientConnection (bounded asyncio.Queue(256) +
sender task); broadcasts enqueue via put_nowait (drop + structlog warn on
QueueFull) and return immediately. The sender drains the queue with each send
wrapped in wait_for(SEND_TIMEOUT=10s). Unregistered legacy sockets (set
directly into a subscription set, bypassing connect_*) get a timeout-bounded
fallback send task held in _pending_sends (ruff RUF006). disconnect cancels +
drops the sender.
F065: route handlers caught only WebSocketDisconnect with no finally — a
non-clean exit (anyio closed-resource, CancelledError, transport error)
propagated without manager.disconnect, leaking the dead socket into every
subscription set forever. Added finally: manager.disconnect(websocket) to all
5 handlers (disconnect is idempotent).
F066: no server-side heartbeat/idle timeout — a half-open socket from a dead
container blocked receive_text forever and was never reaped. receive_text now
wraps in wait_for(IDLE_TIMEOUT_SECONDS=90s); on TimeoutError, log + fall
through to the F065 finally. Named module constants (no config.py precedent for
WS tuning; callers/tests patch them).
TDD: 22 new tests across 3 files (handler cleanup, idle timeout, send queue),
non-flaky across repeats; 1 existing test adapted with a yield for the new
async fan-out (assertion unchanged). ruff/mypy clean, 421 unit/api tests pass.
No type:ignore/noqa.
F068: the do/flow-server circuit breaker only counted rejections whose
`error` field was a STRING in _CIRCUIT_REJECTION_KINDS. A 422 validation
failure (no `error` field, a `detail` list) and a 500/HTTPException
(dict-shaped `error` from the exception handlers) both bypassed the breaker
→ unbounded retries on a storm of either. Added _classify_rejection(payload)
(shared, applied to both servers) mapping all three shapes to a counted kind:
string error (existing), dict error → substring-mapped code
(*DENIED*/*AUTHORIZED*/*FORBIDDEN*/*PERMISSION*→not_authorized,
INVALID_INPUT/*VALIDATION*→incomplete_input, *NOT_FOUND*→None parity, else
→invalid_state), 422 detail→incomplete_input. The dict TypeError defence lives
in the classifier (isinstance, never dict-in-frozenset).
F069: a manifest-registered verb whose HTTP route is missing got FastAPI's raw
`{"detail":"Not Found"}` 404 body — a non-envelope payload the breaker
couldn't classify, so a storm bypassed it. _post now synthesizes an
invalid_state Envelope rejection (with a remediate hint → i_am_blocked/i_am_idle)
for a 404 status, routed through _record_and_check_circuit so the breaker counts
it. A 404 that carries a real Envelope (error field present) is surfaced as-is,
preserving test_flow_post_returns_envelope_on_404. TDD: 422/dict/404 tests in
both server test files; updated test_dict_shaped_error_does_not_crash to assert
the SDK is now called with not_authorized (replacing the pass-through assertion
that encoded the bug).
main_pm's manifest advertises triage (lifecycle.intents_for_role(MAIN_PM)
includes it via _PM_ROLES, alongside triage_all) but flow_main_pm.py had no
POST /triage route, so a main_pm agent calling triage hit a raw 404 that
bypassed the per-verb circuit breaker. Added the route mirroring flow_cell_pm's
/triage — wires to the existing team-scoped choreographer.triage (uses pm.team,
works for any PM role; Main PM gets its own team's blocked/awaiting tasks).
Fix direction: add-route, NOT remove-from-manifest — the manifest is spec-correct
(intents_for_role by construction); removing triage would contradict the spec
and leave main_pm with only cross-team triage_all. TDD: test_triage_route_exists_and_dispatches.
If _configure_git raised CalledProcessError before its `remote set-url`
scrub, .git/config kept the tokenized auth URL (the project PAT) and
_assert_no_pat_leak never ran. The except clauses raised WorkspaceError
without removing the workspace, so the next ensure_workspace's health
short-circuit (valid .git with HEAD + objects) skipped past the leak —
mounting the agent on a workspace whose .git/config let it read+exfiltrate
the PAT. Both clone-failure except clauses now rmtree the workspace before
raising, so a half-configured clone is destroyed and ensure_workspace
re-clones from scratch. TDD: 2 tests (configure-failure leak + timeout).
merge_pr unconditionally set pr_status=merged, pr_merged_at, merged_by,
status=COMPLETED on whatever session it loaded — the only session-terminal
transition in WorkSessionService lacking both the active-status guard
(complete/abandon) and the terminal-idempotency guard (close). Two failure
modes: (1) a retried merge after a successful-but-unconfirmed GitHub merge
overwrote merged_by/pr_merged_at with the retry's actor/timestamp, corrupting
the merge audit trail; (2) merge_pr on an ABANDONED session resurrected it to
COMPLETED, undoing the single-active abandonment. Mirrors close()'s guard:
if status != ACTIVE, return the session unchanged. Both git.py callers await
merge_pr and discard the return, so the no-op is safe. TDD: 3 tests
(happy-path + both modes).
Converts 115 `# type: ignore[...]` suppressions across 23 test files to
no-suppression patterns (helper-return widening to Any, local Any aliases,
cc:Any aliases, cast at narrow call sites, typed fixtures) so the hard
no-type:ignore convention holds across tests/. No test logic or assertions
changed — only mock-wiring mechanics and type annotations.
Gate: ruff check tests/ clean; mypy tests/ (538 files) clean; 176 changed-file
tests pass. Zero real suppressions remain (the 7 grep hits are 3 hygiene-
checker string-literal test inputs and 4 prose mentions in comments).
Convention: no type:ignore/noqa, and pre-existing violations still
violate. The make-quality gate runs 'mypy roboco/ tests/', but the
prior commits' gates only ran mypy on production files, masking 64
type errors across 15 test files (method-assign, unused-ignore,
no-untyped-def, attr-defined, union-attr, has-type, index, misc).
Fixed without any type:ignore:
- method-assign (svc.session.X = / svc.method = AsyncMock()): hold a
local 'session: MagicMock'/'AsyncMock' and assert on it, or stub via
object.__setattr__ / monkeypatch / a typed '_bind' helper returning
Any, or alias 'cc: Any = c' (the pattern the file already used).
- unused 'type: ignore[assignment]' (real code was method-assign):
removed; replaced with the no-suppression patterns above.
- 'Callable[...] has no attribute assert_*': keep a typed local ref to
the AsyncMock and assert on the local, not the method-typed attr.
- no-untyped-def: annotate helper params (Any / pytest.MonkeyPatch).
- attr-defined / index / union-attr: type the helper as Any, narrow
with an 'is not None' assert, or add the missing attr to a fake.
- has-type / return-value: fix the declared return type to the tuple
the function actually returns.
- PLC0415 inline imports: hoisted to top-level.
test_pr_gate_notifies_pm._stub_gate_path converted fully to the
'cc: Any = c' alias (it already used it for one attr) so its five
'# type: ignore[method-assign]' suppressions are gone.
mypy tests/: 64 errors -> 0 (538 files). ruff check tests/: clean.
All 84 tests in the touched files pass.
Convention: no type:ignore/noqa. The F061 in-session audit-emit
tests used '# type: ignore[assignment]' to assign a MagicMock to
AsyncSession.add, and the F060 test assigned to .flush the same way.
Rewritten to hold a local 'session: MagicMock' variable (mypy sees
its auto-children as MagicMock, so .add.side_effect / .flush assign
cleanly with no suppression). Verified via 'mypy tests/' that both
files are now type-clean (the F060/F061 commits had skipped tests/
in mypy, masking two method-assign errors).
_emit_status_transition_audit now writes AuditLogTable rows into
self.session synchronously (session.add) instead of dispatching
AuditService.log_task_event fire-and-forget on its own connection.
The audit row now commits/rolls back atomically with the status
transition in the caller's transaction, closing three facets at once:
- F061: audit commit no longer decoupled from the transition commit
- F073: a committed transition can no longer have NO audit row
(the row rides the same transaction; a swallowed persist can't drop it)
- F075: a transition rolled back inside a verb savepoint no longer
leaves a phantom audit row (the row is in the savepoint too)
log_task_event is now called only from this helper (narrow blast
radius verified); revision_count increment stays at this single
chokepoint. Cycle-time/bottleneck reconstruction from task.<status>
events is no longer silently corruptible.
Tests: test_emit_status_transition_audit_writes_in_session_atomically,
test_finalize_claim_rollback_emits_reversal_audit, escalation-audit
tests retargeted to in-session AuditLogTable rows.
Also: _canonical_bump_files grep-looseness follow-on (F058) -- filter
by subject, not body; git log --grep matches any message line, so a
non-release commit whose body references chore(release): shadowed the
real release commit. Test
test_canonical_bump_files_ignores_body_only_chore_release_match.
The forward task.claimed audit row is flushed before the branch-creation
attempt, and AuditService commits on its own connection, so the rollback's
flush reverts the task row but not that audit row — the journey's last
event stayed task.claimed while the task reverted to its pre-claim status,
diverging from real state and corrupting downstream cycle-time/bottleneck
metrics. The rollback now emits a CLAIMED->original reversal audit row
(only when the forward transition was made) attributed to the claimant.
conftest test-DB defaults matched the project's own running postgres
(roboco/roboco @ localhost:15432, the docker-compose roboco-postgres
service with CREATEDB) instead of the OS user on localhost:5432 which has
no such role — every db_session test failed with InvalidPasswordError
instead of running.
Once the DB connection worked, the self-heal origination DB test went RED
with MAIN_PM_NO_CODE: the self-heal root was task_type=CODE owned by
main_pm, the combo the main_pm_cannot_own_code guard rejects. The Main PM
coordinates the fix (delegates the code work to a cell dev); it has no
code verb. Retyped CODE→PLANNING and rewrote description/AC to
coordination-level.
The module docstring promised self-heal fix tasks 'wait for the CEO's
Approve-&-Start', but _originate created them confirmed_by_human=True and the
orchestrator dispatched them at once — a self-heal fix that re-broke CI would
trigger another cycle, open another auto-dispatched fix, and loop with no CEO
gate on dispatch.
Restore the documented gate:
* _originate opens the task confirmed_by_human=False (held for the CEO).
* The orchestrator holds a self-heal task out of both the PM and dev dispatch
paths until confirmed_by_human flips True.
* approve_and_start (the CEO's start gate) sets confirmed_by_human=True so the
held task finally dispatches (idempotent for board/intake tasks already True).
* list_pending_for_agent scopes the give_me_work hold to self-heal
(source != self_heal OR confirmed_by_human) so an already-alive PM can't grab
it pre-approval — while ordinary delegated subtasks (confirmed_by_human=False
by default, where the delegation IS the authorization) still dispatch.
The 'never self-deploys' guarantee (no merge) is unchanged.
_canonical_bump_files derived the bump set from the previous
chore(release): commit. On the first release there is no such commit,
so it returned [] -> assess set version_bump_plan=[] -> the executor
published a tag with no files bumped (a no-op masquerading as X.Y.Z).
Fall back to the version-reference scan when no prior release commit
exists: the files currently embedding the version are exactly the set a
first release must bump, and the set the first release commit then
records as canonical for subsequent releases. Read-only derivation; the
CEO-approval gate and fail-closed executor are untouched.
notify() only checked the SENDER role. The recipient was resolved by
NotificationService._resolve_recipients, which drops only unresolvable slugs
— it does not exclude human-only roles. The prompter (intake-1) and secretary
(secretary-1) are seeded agent rows, so they resolved, and an ack-required
ALERT addressed to them sat permanently unacked (no agent auto-acks it),
polluted the panel's pending-ack view, and — via the dedup query's
~acked_by.contains — permanently suppressed any later same-purpose
notification from the same sender to that human role. The knowledge-share
path already excludes all three human-only roles; the general notify path
did not.
Fix: a recipient-role guard in notify() via _reject_disallowed_recipient
(folds the new check into the existing CEO-dependency-block return slot so
notify stays under the PLR0911 return limit). Rejects prompter/secretary
with not_authorized; the CEO is human too but acks via the panel, so it stays
an allowed recipient (its only disallowed case, a dependency-block page, is
preserved). TDD test_notify.py (+3: reject prompter, reject secretary, allow
CEO).
The pr_pass (reviewer) conventions guard reused the dev-path block-finding
remediation: 'add a waiver to .roboco/conventions.yml in your branch'. A
pr_reviewer does not own the assembled cell->root / root->master branch and
has no commit verb on it, so the waiver remediation is unreachable — a false
positive stranded the gate with no self-recovery (the reviewer could neither
commit a waiver nor pr_pass). The fail-open content path is documented
precision-over-recall and stays as-is; the actionable gap is the remediation.
Fix: _conventions_rejection now branches the block-finding remediation on
reviewer=True (mirroring the could_not_run branch from F044). The reviewer
path points at pr_fail carrying the findings as issues so the PR returns to
needs_revision and the DEV fixes the violation or commits the waiver (the dev
CAN commit to the branch); waiver authorship is framed as the dev's action,
not the reviewer's. Dev i_am_done path wording unchanged. TDD
test_conventions_gate_pr_pass.py (+1).
_gate_decision dereferenced the verb-runner result without a None guard.
run_intent returns None when a concurrent transition (cancel or a racing
reviewer) moves the task out of awaiting_pr_review between the precondition
gate and the runner's final composed action (the verb runner's documented
last-action source-status contract). The subsequent t.assigned_to /
t.status / _post_gate_review_to_pr(t, ...) dereferences then crashed the
gate with a 500 AttributeError. Add a None guard that surfaces a clean
invalid_state rejection (re-fetch + re-issue) before any dereference; no
PR post or a2a runs against a None task. TDD test_pr_gate_notifies_pm.py (+2).
The in-verb i_am_blocked(rate_limited) path wrapped RateLimitStateTracker.activate
in a bare contextlib.suppress. A silent activate failure stranded the fleet:
agents were parked in _waiting_records but the provider never entered the tracker,
so the tracker-driven _sweep_rate_limit_probes never probed it and no
_on_probe_success ever resumed them — parked agents stuck in WAITING_LONG.
Fix: (1) replace the bare suppress with a try/except that logs an error event
naming the provider + affected agents; (2) in _sweep_rate_limit_probes, after
probing the tracker-listed set, scan _waiting_records for any rate_limit_lifted
provider the loop did NOT cover and probe it via the time-expiry fallback (empty
state -> probe now) so _on_probe_success resumes the parked agents. The fallback
reads only local memory, so it still resumes when Redis was down at park time
(list_rate_limited_providers failure now falls through to the orphan scan instead
of returning early).
The pr_pass gate runs the toolchain + conventions guards on the REVIEWER's
workspace, but their remediation text said 'call i_am_blocked' — a verb the
PR reviewer does not have. The reviewer would chase a verb they cannot call
instead of rejecting the PR.
Make the guards reviewer-aware: a reviewer=True flag (passed by _pr_pass_blocked)
switches the remediation to pr_fail(issues=[...]) — the reviewer's reject
lever, sending the PR back to needs_revision for the dev to fix the
environment / validator. The dev (i_am_done) path keeps i_am_blocked, which a
dev does have. _conventions_guard (the pr_pass path) now passes reviewer=True
through to _conventions_rejection.
escalate_up had composes=() and no source-status guard, so a PM could
escalate a COMPLETED/CANCELLED task and apply_escalation set it back to
BLOCKED — bypassing the state machine's terminal-state invariant.
Defense in depth:
- spec: add PRECONDITION_NON_TERMINAL to escalate_up's extra_preconditions so
the lifecycle gate rejects terminal tasks (invalid_state) before the
journal:decision write fires; generalize _check_intent_preconditions to
honor non-tracing rejection_kind (not_authorized / invalid_state).
- service: apply_escalation (the single write primitive) returns False and
refuses to mutate a terminal task — covers the HTTP escalate route which
bypasses the spec gate. escalate() / escalate_up_to_role() return None on
refusal so the gateway emits a clean invalid_state envelope.
- route: the HTTP escalate route 409s a terminal task BEFORE sending the
escalation notification (so a finished task isn't yanked back, PM not pinged).
Two task creates for the same project/HEAD can race to populate the
conventions cache; the loser's INSERT fails the partial-unique index with
IntegrityError. A bare session.add + flush poisons the shared session (the
task-create transaction rides the same session), so every subsequent op
raises 'this session is in error state' and task creation crashes.
Run the INSERT in a savepoint (begin_nested) and swallow the IntegrityError:
only the savepoint rolls back, the outer transaction stays usable, and the
winner's row satisfies the next _cache_get.
A one-shot grok container whose entrypoint ran grok_auth --check and found
the token missing/expired exits 78 (EX_CONFIG). Crash-retrying 3x burns
tokens for zero progress — the agent cannot start without a valid token.
Park the provider with kind=auth_missing (same shape as the 429 exit-75
path) so the probe-resume loop revives the task once grok_auth.refresh_if_stale
mints a fresh token; if still expired, the next exit 78 re-parks (no burn).
Also fixes a latent F035 regression: _park_provider_unavailable now registers
a WaitingRecord, so the bare-__new__ rate-limit park test had to set
_waiting_records + stub _persist_waiting_record (mirrors the overload-test
fixture).
_enforce_grok_cost_budget killed + evicted the container without calling
_finalize_spawn_session, so the open agent_spawn_sessions row stayed open
(ended_at IS NULL) and the burned usage/cost was never recorded in the
dashboard.
Call _finalize_spawn_session(exit_reason='cost_cap') BEFORE popping the
instance — it reads self._instances[agent_id] for the model +
usage_session_id, which the pop would lose.
The prior self-PATCH 401 fix only carried X-Agent-ID/X-Agent-Role. Arming
ROBOCO_AGENT_AUTH_REQUIRED=true made the middleware require a signed
X-Agent-Token, so every orchestrator self-call (auto-block / auto-resume /
auto-recover / SLA annotation) 401'd and silently no-op'd — wedging
paused/blocked parents.
Add _system_api_headers() that wraps the base headers with a signed token
for the system identity (issue_agent_token); switch all six self-call sites.
Dev fallback: no secret set => UNSIGNED sentinel + auth not required.
The bare 'error 529'/'error 500'/'error 503' markers were broad enough to
false-match an agent that merely writes about an HTTP status code in its own
notes ('the endpoint returned error 500, retrying'), parking the whole
Anthropic fleet on a non-issue.
The SDK error formatter emits 'API Error: NNN' + a JSON error type, so the
remaining 'api error: 529/500/503' + 'overloaded_error' +
'internal_server_error' markers cover every real overload without that
false-match surface.
The SDK server writes model-API errors (529/500/503) to /tmp/sdk-server.log,
not stdout, so an overload marker can appear only in the durable Claude
transcript — the same rationale already applied to the session-limit
detector. _provider_overload_park_target read only docker logs, so an
overload was missed and the agent crash-respawned straight back into it.
Now concatenates the transcript tail before matching, mirroring the
rate-limit path.
_park_provider_unavailable parked the provider + offlined the instance but
never registered a WaitingRecord, so _on_probe_success -> _parked_agents_for
(always filtered on waiting_for=='rate_limit_lifted') returned [] and
resolve_wait revived nobody — recovery fell to the 600s stale-claim reaper
instead of the probe-success path the parking design relied on.
Register + persist a rate_limit_lifted WaitingRecord at park time (mirrors
mark_waiting_long, minus stop_agent — the container is already dead).
Companion reaper guard: _reap_with_service now skips provider-parked
assignees (_assignee_is_provider_parked) so the claim survives until the
probe revives the agent — otherwise the reaper releases the claim to pending
and probe-success respawns on a task the agent no longer owns.
_pm_made_rule_following_retry bounds its tracing_gap audit lookup with
since = record.get('last_check'). A stale persisted last_check from before
the restart matched pre-restart tracing_gap rows, falsely resetting the
breaker on the very first post-restart spawn — exactly when a fresh strike
count should be evaluating current state.
_partition_respawn_rows now re-stamps last_check to the restore time on
every restorable entry, bounding the lookup to post-restart gaps only.
_readopt_running_agents registered re-adopted ACTIVE instances with
container_id=None. _check_health skips container_id-is-None instances, so
when a re-adopted container later exited the stopped-container handler
never ran and the task stranded under a phantom ACTIVE instance forever.
Add _resolve_container_id (docker inspect -f '{{.Id}}') and store the real
id on re-adopt. Best-effort: a probe failure degrades to None (still
ACTIVE; the reaper's Docker-liveness fallback covers it).
F031's role_for_slug_or_none fix made the unknown-assignee release branch
in _dispatch_claimed_without_agent reachable (the human-only guard no
longer raises/short-circuits on a stale slug). Lock that reachability in:
a claimed task with an unknown-assignee UUID past grace returns the slug
(not None) so get_agent_role -> 'unknown' releases the claim to pending
for a role-matched reclaim.
The purpose-based dedup suppressed a same-purpose (same sender/type/task,
overlapping recipients) notification while a prior one was unacked. For
informational types (KNOWLEDGE_SHARE / MENTION / A2A_REQUEST / BROADCAST +
the pickup-proves-receipt triad) each send carries DISTINCT content (a new
learning, a new mention) and acking is voluntary, so a recipient who never
acks the prior one let the dedup permanently suppress every subsequent
same-sender broadcast - silent learning-broadcast data loss.
The dedup's anti-loop rationale (stop unacked-set inflation soft-blocking
i_am_idle) only holds for action-required signals. Gate the dedup on
ACK_REQUIRED_BY_TYPE.get(type, True): action-required types still dedup,
informational types always create. Unmapped types default True (dedup on).
TDD red->green; ruff + mypy clean; notification + dedup suites green (20).
NotificationService._create_notification built NotificationTable without
requires_ack, so the column default (True) applied to EVERY notification -
including informational REVIEW_REQUEST / DOCUMENTATION_REQUEST /
A2A_REQUEST / KNOWLEDGE_SHARE (ACK_REQUIRED_BY_TYPE -> False) and every
@mention from MessagingService._notify_mentions. Each false ack-required
inflated the recipient's unacked set and soft-blocked i_am_idle into
respawn churn.
- _create_notification: requires_ack=ACK_REQUIRED_BY_TYPE.get(type, True)
(unmapped types default True - preserve the action-required bias).
- _notify_mentions: requires_ack=False explicit (MENTION is informational).
TDD red->green (identity is False/is True assertions - the mocked
flush doesn't apply SQLA's insert-time default, so pre-fix the attribute
was None); ruff + mypy clean; notification suite green (18 passed).
The pr_fail a2a steer to the owning PM is fire-and-forget; a PM respawned
into needs_revision later read none of it (build_task_handoff never looked
at notes_structured), saw a generic 'needs revision' with zero concrete
change-requests, and re-submitted the same PR (the 2026-06-27 infinite
pr_fail loop on 9980d0a0 / PR #138). The signal-gap was only partially
closed by the a2a.
build_task_handoff now extracts notes_structured.pr_review
(verdict/summary/issues/head_sha — the slot pr_fail authors on every fail)
into a pr_review field on the handoff, so every PM briefing for the task
carries the concrete change-requests. A prior pr_fail alone now counts as
prior-work-worth-resuming. Type-guarded + capped; absent => no key (no
misleading empty slot).
TDD red->green; ruff + mypy clean; evidence_builder suite green (14 passed).
The root loop-stopper (F016) was root-only; a weak cell PM could re-submit
the unchanged cell->root PR after a pr_fail and loop awaiting_pr_review ->
pr_fail forever (the cell analogue of the 2026-06-27 root loop).
pr_fail stamps the assembled PR's head SHA into notes_structured.pr_review
.head_sha for cell AND root gate tasks alike (the capture is gate-verb-
level, not root-level), so the same structural refusal applies to submit_up:
if the cell PR's current head SHA equals the SHA the last pr_fail recorded,
no new dev work landed on the cell branch -> the diff is byte-identical ->
refuse, do not re-open the gate. Different SHA -> branch advanced -> allow.
- _submit_up_unchanged_pr_guard mirrors _submit_root_unchanged_pr_guard
(cell-PM remediation: re-delegate to the dev + wait for re-assembly),
wired into submit_up after _submit_up_guard passes.
- Renamed shared _current_root_pr_head_sha -> _current_pr_head_sha (both
guards use it; the lookup was never root-specific).
- Every ambiguous case FAILS OPEN (no prior fail, no recorded sha, no
pr_number, no resolvable project, git/closed-PR None) — only the exact-
unchanged case is hard-blocked.
TDD red->green; ruff + mypy clean; F007+F016 guard suites green (15 passed).
Both verbs compose a single atomic action whose None return (the verb's
own result) flowed out of run_intent and was dereferenced as t.status,
HTTP 500-ing with no actionable rejection:
- F016 submit_root: submit_for_review returns None when the root->master
PR was already opened / the task raced out of in_progress. Post-runner
None-guard extracted into _submit_root_finalize -> invalid_state
(re-fetch; if awaiting_pr_review the PR is open, wait for reviewer;
else re-delegate fixes and retry) instead of None.status.
- F017 i_am_blocked: escalate returns None in four cases (no task, no
agent, no resolvable escalation-target slug, no target agent row) e.g.
a developer whose role has no PM above it. _run_i_am_blocked_intent
now guards updated is None -> (t, invalid_state rejection) with
remediation (re-fetch + escalate to CEO directly / retry) instead of
the caller deref'ing None.status -> 500 + respawn-loop.
TDD red->green; ruff + mypy clean; gateway suite green (58 passed).
F005: the single-file bind mount of auth.json pinned the inode, so the
orchestrator's atomic refresh (tmp+rename within ~/.grok) never reached a
running grok container — a long-lived container hung at the login prompt
when the original ~6h token expired. Mount the host ~/.grok DIRECTORY (ro)
at /home/agent/.grok-auth-ro; the entrypoint symlinks ~/.grok/auth.json at
that RO mount so grok + the --check backstop read the live credential (the
directory mount sees the host-side rename) while grok's writable state
(config.toml, sessions/) stays in the image's ~/.grok.
F006: a rotated refresh_token is single-use — xAI invalidates the old one
the instant it issues the new one. If the atomic write failed after the
rotation, the file kept the now-dead old refresh_token and the credential
was permanently lost on the next refresh. _atomic_write now falls back to a
direct write when tmp+replace fails, so the rotated token always lands on
disk (losing the write is catastrophic; losing atomicity is not).
TDD: RED tests watched fail, then GREEN. ruff+mypy clean; 32 grok tests
green, no regressions.
F003/F014: /api/v1/do/* only required X-Agent-ID (UUID) — no token check,
unlike the flow routers' role guards. A forged X-Agent-ID passed. Added
require_any_authenticated_agent (token-only; do router serves all roles)
and applied it as a router-level dependency. Binds X-Agent-ID to a verified
HMAC token when ROBOCO_AGENT_AUTH_REQUIRED=true; rejects a forged token
even in dev mode.
F004: /ws/* per-agent streams (channels/agents/sessions/notifications)
never read the nginx-injected X-Agent-Token, so in strict mode an agent on
the Docker network could subscribe to another agent's notifications with
no auth. Added _require_panel_token verifying the CEO panel token against
the CEO identity; wired into all four per-agent streams (system stream
stays operator-only per its docstring). Same strict/dev contract.
TDD: RED tests watched fail (no gate -> 200/accept), then GREEN. ruff+mypy
clean; 399 api/mcp + 29 WS tests green, no regressions.
_activate_batch_root_subtasks flipped a held root-subtask to team=MAIN_PM
but left task_type=code (intake only coerces main_pm-team drafts, so a
board-routed code root reached activation still code-typed). The
main_pm+code combo re-introduces the 2026-06-27 meltdown. Mirror
approve_and_start's own retype via main_pm_cannot_own_code so the
activated child is a planning-typed coordination root.
TDD: RED test_activate_batch_root_subtasks_retypes_code_to_planning
watched fail (task_type stayed CODE), then GREEN after the retype.
ruff+mypy clean; 125 batch/umbrella/approve tests green, no regressions.
Defense-in-depth for the spawn_agent human-role chokepoint (d31d6719).
The chokepoint structurally guarantees no CEO/prompter/secretary container
can ever spawn — every dispatcher goes through spawn_agent. But two
dispatchers resolve an arbitrary assigned_to and spawn it with only a
None/unknown-role filter, so a human-assigned task would reach the
chokepoint and RAISE: caught by the per-dispatcher try/except, but it
aborts that dispatcher's whole tick (stalling other respawns behind the
mis-assigned task) and error-logs every cycle. The other dispatchers are
already safe by whitelist/hardcoded slug (blocker_resolver_slug returns
None for non-PM/non-BOARD; escalation/approval use whitelists; marketing
and audit hardcode their non-human slug).
- _claimed_task_needs_agent: return None for a CEO/prompter/secretary
assignee — no container to respawn, and do NOT release a human-owned
task to pending (that would re-route it to a PM). Leave it for the human.
- _dispatch_pm_review_work (assigned branch): skip a human-only assignee
so a CEO-assigned awaiting_pm_review task neither spawns nor aborts the
dispatcher's tick.
Audited all target-iterating dispatchers; only these two lacked a filter.
Regression tests cover both skips.
A live 2026-06-27 incident saw a CEO agent container spawned. Root cause:
_dispatch_a2a_work iterates every A2A/notification target and spawns it
with no human-role filter, and _is_agent_active('ceo') is always false
(the CEO is never a container), so the 'skip if active' check could never
protect the CEO. Any CEO-addressed notification (board handoff, escalation)
launched a CEO container — the system acting as the human CEO: a trust
violation. The CEO is the human operator; intake (prompter) and secretary
are human-driven chats launched through their own dedicated guarded paths
(_spawn_intake_container / _spawn_secretary_container), never spawn_agent.
Fix: a single chokepoint guard at the top of spawn_agent refuses
Role.CEO / PROMPTER / SECRETARY (raises AgentReadinessError + logs). This
structurally covers every dispatcher present and future, since they all go
through spawn_agent. Plus a defense-in-depth skip in _dispatch_a2a_work so
a human-role target never even calls in (avoids error-log spam; the
notification stays for the human to read in the panel).
Safe: the dedicated human-spawn paths do not route through spawn_agent.
Regression tests: spawn_agent refuses ceo/intake-1/secretary-1, does NOT
refuse a real agent; _dispatch_a2a_work skips CEO/intake/secretary targets
and still spawns real-agent + mixed-target cases.
Update every behind-base/rebase guidance surface to reflect the B1
sync_branch dev verb + B2 i_am_done behind-base gate: devs now self-rebase
through the gate instead of escalating a plain behind-base condition; PMs
still escalate cell/root integration branches (they have no rebase verb).
- developer.md: sync_branch in the verb table; 'When your branch is behind
its base' rewritten — call sync_branch, do NOT i_am_blocked a plain
behind-base; conflicts → resolve by hand, commit, sync_branch again.
- cell_pm.md: delegate signature gains intends_to_touch/adds_migration/
touches_shared/depends_on + a 'Collision surface' section (fill it on every
code subtask so sibling dev tasks that touch the same files sequence into a
conflict-free order — the 2026-06-27 out-of-order break fix); behind-base
section steers devs to sync_branch, PMs escalate only the integration branch.
- main_pm.md: behind-base section — dev leaf = dev's sync_branch; cell/root
integration branch = escalate_up.
- RAG git-errors.md / blocked-tools.md: devs sync_branch, PMs escalate.
- docs/troubleshooting/common-issues.md: leaf self-rebases; integration branch
still escalates to operator.
- CLAUDE.md verb surface: developer gains sync_branch.
- agents/prompts/_generated/*: regenerated via scripts/regenerate_verb_tables.py
— adds sync_branch to the dev table AND catches the generated tables up to
the S1/S2 delegate sequencing params + meltdown-fix note top-level params
(the derived files had drifted stale vs the already-committed schemas).
Docs/prompts only — no code. ruff + mypy roboco/ tests/ clean.
A sibling's PR merging into the parent branch while a dev worked leaves the
dev's branch behind its base — the assembled PR then can't merge cleanly and
the sibling's changes go missing (the 2026-06-27 out-of-order dev-task break).
The behind-base gate refuses i_am_done in that state and steers the dev to
sync_branch (the Phase B1 gate-level rebase verb).
- GitService.is_behind_base: rev-list --left-right --count across
origin/{base}...origin/{head} → (behind, ahead); fetch-first so origin
reflects the pushed head. Raises on git failure (consistent with
rebase_onto_base); malformed stdout degrades to (0,0).
- Choreographer._behind_base_gate: wired into _i_am_done_gate after
_ensure_branch_pushed. behind>0 → invalid_state remediate→sync_branch.
Fail-open on git/base-resolution error (flaky fetch can't strand a task at
the submit gate — the merge layer has its own behind checks). Skipped for
branchless roots and protected bases (master/main/-prefixed).
Tests: gate (6: refuse+steer/up-to-date/branchless/protected/fail-open-base/
fail-open-git), is_behind_base (6: parse/up-to-date/malformed/argv-form/
requires-branch/missing-project). ruff + mypy roboco/ tests/ clean; unit green.
Raw shell git is denied to agents (Bash(git:*) base deny), so a developer
whose branch fell behind its base had no gate-level rebase — only the
CEO/PM-only /rebase HTTP route. sync_branch is the dev verb that wraps the
rebase through the gate (traced + evidenced), so the 'everything goes through
the gates' invariant holds.
- lifecycle: IntentSpec sync_branch (dev-only, ownership-gated, composes=(),
git-only — no DB transition); _next_hint_synced helper.
- GitService.sync_task_branch: rebase task.branch_name onto its resolved base
via rebase_onto_base (fetch + rebase + force-with-lease push).
- Choreographer.sync_branch + _sync_branch_preflight_rejection: not_found /
unknown-role / spec-gate / no-branch / protected-base guards, then the git
op; conflicts abort (no force-push) and steer to resolve-by-hand; git failure
steers to i_am_blocked.
- HTTP route /api/v1/flow/developer/sync_branch + SyncBranchRequest schema.
- MCP tool sync_branch(task_id) + _TOOLS registration (manifest auto-propagates
via intents_for_role(Role.DEVELOPER)).
Tests: intent spec (5), choreographer handler (8: happy/conflicts/not_found/
not_authorized/no-branch/protected-base/git-failure/audit), route (1), MCP (1).
ruff + mypy roboco/ tests/ clean; unit suite green (DB-fixture errors env-only).
Kind 2 (cell-task wave chain): a new cell-task under root-subtask UT_n
depends on every cell-task under every root-subtask in UT_n.dependency_ids
(the kind-1 wave-chain edges), so its branch carries the previous wave's
merged cell work. Re-derived from the root-subtask's deps, not the cell-task's
own dependency_ids (which also carry UX/product-fanout edges the by-osmosis
edge must not pick up). A root may fan to several cell-tasks (different cells),
so the previous wave's cell-task is a SET.
Kind 4 (by-osmosis): the first dev task (sequence 0) under a cell-task depends
on each predecessor cell-task's tail (max-sequence) dev task, so the new wave's
first branch carries the previous wave's fully-merged tail. Subsequent dev
tasks inherit the tail via kind 3 or the merged base.
Both wired from _create_subtask_from_inputs, dispatched on parent.team
(MAIN_PM -> kind 2; cell team -> kind 4). Pure helpers
(cell_task_wave_chain_depends_on, by_osmosis_tail_dev_tasks) unit-tested in
test_sequencing.py; TaskService methods integration-tested. Idempotent +
best-effort throughout (add_dependency dedupes; missing predecessors are
no-ops). Also fixes a latent mypy-tests gap (estimated_complexity required on
direct TaskCreateRequest calls in the S2 tests).
Pure dev_task_collision_edges in sequencing.py turns a parent's surfaced
siblings into (depends_on_id, task_id) pairs via SequencingService. TaskService.
wire_sibling_collision_dag wires them through add_dependency (idempotent). The
choreographer calls it after each dev-task delegate so the sibling collision DAG
is built incrementally as the cell PM decomposes — file-overlap serializes,
migration chains, shared-last; stable (priority, sequence) ordering keeps edges
from flipping into reverse cycles on re-runs.
The cell/main PM's delegate verb now carries the dev-task collision
surface (intends_to_touch / adds_migration / touches_shared) and an
explicit depends_on override through DelegateRequest -> DelegateInputs
-> _create_subtask_from_inputs -> create_subtask, and create_subtask
forwards sequence / dependency_ids / batch_id / surfaces into the
prepared TaskCreateRequest instead of dropping them (the base create
already persists them at task.py:878-884).
This is the plumbing for the multi-level sequencing model edge kind 3
(dev-task collision DAG). Previously a dev task delegated with a
collision surface or an explicit dependency lost it before persistence
— dependency_ids was always [], so the only dev-task ordering was the
weak assignee-keyed spawn barrier (the live 2026-06-27 out-of-order
break: 40842957 started before 9b3682b8's PR merged). Phase S2 runs
SequencingService over the surfaced siblings and wires the DAG via
add_dependency.
A dev task in needs_revision must go back to the developer, never the
pool. The pool path let a cell PM re-claim the revision (PMs can claim
needs_revision) — the live 2026-06-27 'needs revision on a dev task sent to
the cell PM' bug.
fail_qa's original_developer marker is the fast path, but it is
unreliable in practice (live observation: never persisted), so the
unassign else-branch was the load-bearing path and it dropped the task
into the pool. Add a work-session fallback (_resolve_revision_dev) that
resolves the developer who actually worked the task — the most recent
work session whose agent is a developer, the QA's own session excluded
— and reassigns to that dev instead of unassigning. Only unassign when
no developer ever touched the task. Self-heals the marker so a
subsequent re-fail takes the fast path and the QA-review index
attributes the work correctly.
The branch tip had 15 mypy tests/ errors in files this bundle did not author,
which blocked CI's make quality mypy step (mypy roboco/ tests/) regardless of
the bundle's own commits. Pre-existing is still existing — fix every one:
- test_schemas_v1_flow.py (8): the StrList coercion tests intentionally pass
SDK-nested list-of-strings input ([[['...']]], {'item':{'$text':'...'}}, int,
dict). Annotate those literals as list[Any] locals so mypy accepts the
coerce-able shape; the StrList BeforeValidator still flattens to list[str] at
runtime. No type:ignore.
- test_pr_gate_records_verdict.py (3): notes_structured is dict|None; narrow
with 'assert t.notes_structured is not None' before indexing (the existing
pattern at line 90).
- test_pr_review_hand_format_guard.py (1 site, 2 errors): the _verb_runner()
spy assertion — use the cc: Any = c alias idiom so assert_not_awaited
resolves; drops the now-unused type:ignore[union-attr].
- test_pr_gate_notifies_pm.py (1): drop the unused type:ignore[method-assign]
on the a2a.send reassignment.
- test_content_models.py (1): narrow coerced with isinstance(coerced,
PrReviewContent) before reading .issues (the base _Content lacks the field).
Gates: rm -rf .mypy_cache && mypy roboco/ tests/ = Success (855 files);
ruff check + format clean; 5 affected suites = 40 passed.
The 2026-06-27 infinite pr_fail loop: a Main-PM root (PR #139) was pr_fail'd,
routed to needs_revision, and re-submitted byte-identical → awaiting_pr_review
→ pr_fail again, forever. The prior hint/a2a steer was ignored by the weak
coordinator model — hints don't stop a model that won't read them. A HARD gate
refuses the re-submit when the assembled root PR's head SHA is unchanged since
the last pr_fail (no new cell work → identical diff); a different SHA ⇒ the
branch advanced ⇒ allow. Every ambiguous case fails open (no prior fail, no
recorded SHA, no pr_number, unresolvable slug, git error, closed PR) — only the
exact-unchanged case is hard-blocked.
- content/models: PrReviewContent.head_sha (optional; JSON col → no migration).
- git: get_pr_head_sha (GitHub pulls API; None on any failure → fail-open).
- pr_gate: pr_fail captures head_sha into the verdict record; pr_pass does not.
- _impl: submit_root runs _submit_root_unchanged_pr_guard after _submit_up_guard;
_current_root_pr_head_sha resolves slug + current SHA (fail-open).
- pr_review: extract module-level resolve_task_project_slug, shared by the mixin
and the gate helper (_LegacyChoreographer reaches it via cast to the
ChoreographerHelpers typed view — it doesn't inherit the helpers mixin).
- tests: test_submit_root_unchanged_pr_guard (11 — refuse/allow/6 fail-open/3
capture-side, mypy-clean via cc:Any spy idiom, zero type:ignore) +
test_pr_gate_notifies_pm capture-path stub.
Renn F