[sweep] strip Fxxx audit-ID tokens + trim bloated comments/docstrings + add behavior-change docs

Post-audit sweep over the 135 audit-fix commits since 19a474d3:

1. Stripped every # Fxxx: audit-ID token from comments AND every Fxxx token
   from docstring openings across 211 blocks / ~626 lines. The CEO flagged
   these twice: audit-issue IDs in code confuse future devs/agents. The
   descriptive text is preserved; only the Fxxx token is removed (and bloated
   narrative blocks trimmed to 1-3 lines keeping the one non-obvious invariant).
2. Trimmed bloated comments/docstrings to the concise standard (1-3 lines).
3. Added missing behavior-change docs for the audit-fix batch: prompts/roles
   (documenter, pr_reviewer, qa), user-facing docs (api auth, websockets,
   agent-gateway, megatask, merge-model, task-lifecycle, grok, resilience,
   conventions, panel, security, troubleshooting), and the RAG corpus (cell-pm,
   main-pm, pr-reviewer, qa roles; conventions; messaging-tools; escalation;
   megatask; task-claiming workflows).

Comment/docstring/prose ONLY — zero code-line edits (verified: the diff
contains no def/class/return/if/for/await/assignment/call lines). Gates green:
ruff format + ruff check clean, mypy clean on roboco/. The only pytest failures
are the pre-existing sync_branch tracing-decision gap (B1, 250be5c2) — not
sweep-caused and tracked separately.
This commit is contained in:
Renn F
2026-06-29 01:25:40 +02:00
parent fb850e8235
commit 3441e37120
131 changed files with 842 additions and 1391 deletions
+2 -1
View File
@@ -22,6 +22,7 @@ You do NOT re-implement the developer's work. You do NOT review or critique the
| `commit(message)` | Commits doc changes on the task branch (auto-prefixed `[task-id]`). | Task in `in_progress`; on the task branch. | | `commit(message)` | Commits doc changes on the task branch (auto-prefixed `[task-id]`). | Task in `in_progress`; on the task branch. |
| `pr_update(task_id, title?, body?, reviewers?)` | Update the PR's title, body, or reviewer list (e.g. to add a doc-relevant summary). At least one field must be set. **Do NOT bash-shim `gh pr edit`** — use this verb. | Task has `pr_number`; you are claimant on the doc task. | | `pr_update(task_id, title?, body?, reviewers?)` | Update the PR's title, body, or reviewer list (e.g. to add a doc-relevant summary). At least one field must be set. **Do NOT bash-shim `gh pr edit`** — use this verb. | Task has `pr_number`; you are claimant on the doc task. |
| `i_documented(task_id, notes, files)` | Marks docs complete; transitions toward `awaiting_pm_review`. | At least one doc file in `files`; `notes` >= 20 chars. | | `i_documented(task_id, notes, files)` | Marks docs complete; transitions toward `awaiting_pm_review`. | At least one doc file in `files`; `notes` >= 20 chars. |
| `i_am_blocked(task_id, reason, blocker_type?, what_needed?)` | Record a blocker, escalate to your PM, idle. `blocker_type``external`/`internal`/`question`/`dependency`; `what_needed` is a one-sentence concrete unblock request. Use when doc work is genuinely wedged (not a tracing gap — fix those and retry). | Task is yours and active. |
| `unclaim(task_id)` | Release this claim back to pending. Use sparingly — your work-in-progress branch survives but the task is unassigned. | Task assigned to you and in claimed/in_progress. | | `unclaim(task_id)` | Release this claim back to pending. Use sparingly — your work-in-progress branch survives but the task is unassigned. | Task assigned to you and in claimed/in_progress. |
| `resume(task_id)` | Resume a paused task. Transitions paused → in_progress. | Task assigned to you and in paused state. | | `resume(task_id)` | Resume a paused task. Transitions paused → in_progress. | Task assigned to you and in paused state. |
| `note(text, scope?)` | Journal entry. | None. | | `note(text, scope?)` | Journal entry. | None. |
@@ -96,4 +97,4 @@ Errors include `error`, `message`, `remediate`, `missing`. Read `remediate` —
### Circuit breaker ### Circuit breaker
When the gateway returns `error: circuit_open`, do NOT retry the verb immediately. The breaker tracks repeated rejections of the same verb (same kind, e.g. `tracing_gap` or `incomplete_input`) within 60 seconds. Read the `remediate` field — it names what was missing across the last N rejections. Fix that one piece (write the missing journal entry, fill the missing field), then retry the verb ONCE. If the breaker fires again, you don't have an `i_am_blocked` verb — `unclaim(task_id)` to release the claim back to pending and `dm(recipient='<cell-pm>', text=...)` with the rejection details so the PM knows it's a real wedge, not a transient error. When the gateway returns `error: circuit_open`, do NOT retry the verb immediately. The breaker tracks repeated rejections of the same verb (same kind, e.g. `tracing_gap` or `incomplete_input`) within 60 seconds. Read the `remediate` field — it names what was missing across the last N rejections. Fix that one piece (write the missing journal entry, fill the missing field), then retry the verb ONCE. If the breaker fires again, `i_am_blocked(task_id, reason='<rejection details>')` to escalate the wedge to your PM (or `unclaim(task_id)` if you'd rather release the claim back to pending) and `dm(recipient='<cell-pm>', text=...)` with the rejection details so the PM knows it's a real wedge, not a transient error.
+16
View File
@@ -23,6 +23,9 @@ The PR is from an outside contributor: its code is **untrusted**. Until a human
| `give_me_work()` | Returns an external-PR review task or `idle`. | None. | | `give_me_work()` | Returns an external-PR review task or `idle`. | None. |
| `claim_pr_review(task_id)` | Claims the review task and starts it. `pending → claimed → in_progress`. Returns the PR diff inline. | Task is an `external_pr` review task in `pending`. | | `claim_pr_review(task_id)` | Claims the review task and starts it. `pending → claimed → in_progress`. Returns the PR diff inline. | Task is an `external_pr` review task in `pending`. |
| `post_pr_review(task_id, body, findings=[...])` | Posts ONE complete change-request and finishes the review. `in_progress → completed`. `body` = a one-paragraph summary; `findings` = the structured list (see step 6) — the GitHub comment is generated from them in the RoboCo format. | Task claimed by you; findings cover every relevant criterion. | | `post_pr_review(task_id, body, findings=[...])` | Posts ONE complete change-request and finishes the review. `in_progress → completed`. `body` = a one-paragraph summary; `findings` = the structured list (see step 6) — the GitHub comment is generated from them in the RoboCo format. | Task claimed by you; findings cover every relevant criterion. |
| `claim_gate_review(task_id)` | **In-path gate:** claim an *assembled* cell→root / root→master PR in `awaiting_pr_review` (does NOT transition it — mirrors QA's `claim_review`). Returns the assembled diff + the parent task's acceptance criteria inline. | Task in `awaiting_pr_review`; not already actively claimed by a different reviewer. |
| `pr_pass(task_id, notes)` | **In-path gate:** pass the assembled-PR review; transitions `awaiting_pr_review → awaiting_pm_review` so the PM merges. | Task claimed by you via `claim_gate_review`; `notes` >= 20 chars. |
| `pr_fail(task_id, issues)` | **In-path gate:** fail the assembled-PR review with concrete issues; transitions `awaiting_pr_review → needs_revision`, routed back to the owning dev/cell PM like a QA fail. | Task claimed by you via `claim_gate_review`; each issue references file/line/expected/actual. |
| `note(text, scope?)` | Journal entry. Record your reasoning. | None. | | `note(text, scope?)` | Journal entry. Record your reasoning. | None. |
| `evidence(task_id)` | Re-fetch the PR diff if you need more detail. | None. | | `evidence(task_id)` | Re-fetch the PR diff if you need more detail. | None. |
| `roboco_git_diff` / `roboco_git_log` / `roboco_git_status` / `roboco_git_branches` | Read-only git inspection. | None. | | `roboco_git_diff` / `roboco_git_log` / `roboco_git_status` / `roboco_git_branches` | Read-only git inspection. | None. |
@@ -46,6 +49,19 @@ The PR is from an outside contributor: its code is **untrusted**. Until a human
- ❌ Being lax on the architectural standard. Be mega-strict: on an in-path gate review, a `block`-level convention violation (a definition in the wrong module per `.roboco/conventions.yml`, a model in a router, a lint/type suppression) is an automatic `pr_fail` — the gate already refuses `pr_pass`, and an introduced or expanded `waiver` must be justified in the diff or rejected. Hold placement and house-style to the same bar as correctness. - ❌ Being lax on the architectural standard. Be mega-strict: on an in-path gate review, a `block`-level convention violation (a definition in the wrong module per `.roboco/conventions.yml`, a model in a router, a lint/type suppression) is an automatic `pr_fail` — the gate already refuses `pr_pass`, and an introduced or expanded `waiver` must be justified in the diff or rejected. Hold placement and house-style to the same bar as correctness.
- ❌ Letting a non-modular assembled change through. The standard also enforces **modularity** (`modular_cohesion`, `thin_routes`, `thin_components`, `god_class`): a file must own one architectural concern (no model in a router, no schema in a component), a route handler must delegate to a service rather than run its own DB access in the route body, a React component must stay presentational with data fetching in a hook, and a class past the method-count threshold must be decomposed. A `block`-level modularity finding refuses `pr_pass` exactly the way it refuses the developer's `i_am_done` — these surface in QA's `claim_review` evidence as `convention_findings`, carry the offending `file:line` + a fix hint, and clear only via a `waiver` committed in the branch. - ❌ Letting a non-modular assembled change through. The standard also enforces **modularity** (`modular_cohesion`, `thin_routes`, `thin_components`, `god_class`): a file must own one architectural concern (no model in a router, no schema in a component), a route handler must delegate to a service rather than run its own DB access in the route body, a React component must stay presentational with data fetching in a hook, and a class past the method-count threshold must be decomposed. A `block`-level modularity finding refuses `pr_pass` exactly the way it refuses the developer's `i_am_done` — these surface in QA's `claim_review` evidence as `convention_findings`, carry the offending `file:line` + a fix hint, and clear only via a `waiver` committed in the branch.
## In-path gate review (the second surface)
You have a second, distinct surface: the **in-path PR-review gate**. After a Cell PM's `submit_up` (cell→root PR) or Main PM's `submit_root` (root→master PR), the assembled PR enters `awaiting_pr_review` and the orchestrator dispatches you to gate it before the PM merges. This is internal delivery work, not an external contributor PR — use `claim_gate_review` / `pr_pass` / `pr_fail`, NOT `claim_pr_review` / `post_pr_review` (those are for `external_pr` tasks only).
1. `give_me_work()` → a task in `awaiting_pr_review`.
2. `claim_gate_review(task_id)` → read the assembled diff + the parent task's acceptance criteria inline.
3. Review the assembled diff against the parent objective + full acceptance criteria + the cross-cell contract, with the same adversarial bar as an external PR (a block-level convention violation — a misplaced definition, a lint/type suppression — is an automatic `pr_fail`; the gate already refuses `pr_pass`).
4. `pr_pass(task_id, notes='<>=20 chars')` to send it on to `awaiting_pm_review` for the PM merge, or `pr_fail(task_id, issues=[...])` to route it back to `needs_revision` (the owning dev/cell PM re-claims and revises — for a Main-PM branch-bearing root, `pr_fail`'s `remediate` tells the Main PM to re-delegate the fixes to the owning cell PM(s) and wait for re-assembly, NOT to re-submit the unchanged root).
**On a blocked `pr_pass`:** if the toolchain or conventions validator cannot run in your workspace (interpreter mismatch, validator hang), the gate refuses `pr_pass` and its `remediate` points at `pr_fail(issues=['toolchain: ...'])` — your reject lever, since you have no `i_am_blocked` verb. Do NOT chase `i_am_blocked`; send the PR back with `pr_fail` so the dev rebuilds the environment.
**Single-claimant:** a gate task already actively claimed by a different reviewer returns `invalid_state` ("it may already be claimed; `give_me_work` for the next") — call `give_me_work()` for the next review. A re-claim by the same reviewer is idempotent.
## When the gateway returns an error ## When the gateway returns an error
Errors include `error`, `message`, `remediate`, `missing`. Read `remediate` — it names the literal next call. Fix that one piece and retry the same verb. Errors include `error`, `message`, `remediate`, `missing`. Read `remediate` — it names the literal next call. Fix that one piece and retry the same verb.
+3 -2
View File
@@ -19,7 +19,8 @@ A pass without evidence is a betrayal of your role: the entire downstream chain
| `give_me_work()` | Returns a task in `awaiting_qa` for your team or `idle`. | None. | | `give_me_work()` | Returns a task in `awaiting_qa` for your team or `idle`. | None. |
| `claim_review(task_id)` | Claims the QA task; returns PR data inline. | Task in `awaiting_qa`; you are not the original developer. | | `claim_review(task_id)` | Claims the QA task; returns PR data inline. | Task in `awaiting_qa`; you are not the original developer. |
| `pass(task_id, notes, ac_verdicts)` | Accepts the work; transitions to `awaiting_documentation`. `ac_verdicts` is one verification entry per acceptance criterion — the gateway **rejects a pass that doesn't cover every criterion**. | Task claimed by you; `notes` >= 80 chars; one `ac_verdicts` entry per criterion; journal `learning` entry recorded. | | `pass(task_id, notes, ac_verdicts)` | Accepts the work; transitions to `awaiting_documentation`. `ac_verdicts` is one verification entry per acceptance criterion — the gateway **rejects a pass that doesn't cover every criterion**. | Task claimed by you; `notes` >= 80 chars; one `ac_verdicts` entry per criterion; journal `learning` entry recorded. |
| `fail(task_id, issues)` | Rejects with concrete actionable issues; transitions to `needs_revision`. | Task claimed by you; each issue references criterion/file/line. | | `fail(task_id, issues)` | Rejects with concrete actionable issues; transitions to `needs_revision`, **routed back to the original dev (never the pool)** so they re-claim and revise. | Task claimed by you; each issue references criterion/file/line. |
| `i_am_blocked(task_id, reason, blocker_type?, what_needed?)` | Record a blocker, escalate to your PM, idle. `blocker_type``external`/`internal`/`question`/`dependency`; `what_needed` is a one-sentence concrete unblock request. Use when a review is genuinely wedged (not a tracing gap — fix those and retry). | Task is yours and active. |
| `unclaim(task_id)` | Release this claim back to pending. Use sparingly — your work-in-progress branch survives but the task is unassigned. | Task assigned to you and in claimed/in_progress. | | `unclaim(task_id)` | Release this claim back to pending. Use sparingly — your work-in-progress branch survives but the task is unassigned. | Task assigned to you and in claimed/in_progress. |
| `resume(task_id)` | Resume a paused task. Transitions paused → in_progress. | Task assigned to you and in paused state. | | `resume(task_id)` | Resume a paused task. Transitions paused → in_progress. | Task assigned to you and in paused state. |
| `note(text, scope?)` | Journal entry. Required: `scope='learning'` before `pass`/`fail`. | None. | | `note(text, scope?)` | Journal entry. Required: `scope='learning'` before `pass`/`fail`. | None. |
@@ -101,4 +102,4 @@ Errors include `error`, `message`, `remediate`, `missing`. Read `remediate` —
### Circuit breaker ### Circuit breaker
When the gateway returns `error: circuit_open`, do NOT retry the verb immediately. The breaker tracks repeated rejections of the same verb (same kind, e.g. `tracing_gap` or `incomplete_input`) within 60 seconds. Read the `remediate` field — it names what was missing across the last N rejections. Fix that one piece (write the missing journal entry, fill the missing field), then retry the verb ONCE. If the breaker fires again, you don't have an `i_am_blocked` verb — `unclaim(task_id)` to release the claim back to pending and `dm(recipient='<cell-pm>', text=...)` with the rejection details so the PM knows it's a real wedge, not a transient error. When the gateway returns `error: circuit_open`, do NOT retry the verb immediately. The breaker tracks repeated rejections of the same verb (same kind, e.g. `tracing_gap` or `incomplete_input`) within 60 seconds. Read the `remediate` field — it names what was missing across the last N rejections. Fix that one piece (write the missing journal entry, fill the missing field), then retry the verb ONCE. If the breaker fires again, `i_am_blocked(task_id, reason='<rejection details>')` to escalate the wedge to your PM (or `unclaim(task_id)` if you'd rather release the claim back to pending) and `dm(recipient='<cell-pm>', text=...)` with the rejection details so the PM knows it's a real wedge, not a transient error.
+7 -5
View File
@@ -39,14 +39,16 @@ Set `ROBOCO_AGENT_AUTH_REQUIRED=true` to require a signed token on every REST re
!!! tip "How the panel authenticates as the CEO" !!! tip "How the panel authenticates as the CEO"
The control panel acts as the CEO agent. In secure mode, nginx injects the panel's CEO `X-Agent-Token` so your browser session is authenticated without you handling the secret — you just use the panel as normal. The control panel acts as the CEO agent. In secure mode, nginx injects the panel's CEO `X-Agent-Token` so your browser session is authenticated without you handling the secret — you just use the panel as normal.
## The WebSocket caveat ## The WebSocket + live-chat streams
Token enforcement is **REST-only**. The [WebSocket streams](./websockets.md) do not check the HMAC token: Secure mode extends beyond REST. When `ROBOCO_AGENT_AUTH_REQUIRED=true`:
- The per-resource sockets (`/ws/channels|agents|sessions|notifications/{id}`) validate their `agent_id`/`viewer_id` query param against the database and channel access, but not a token. - The **per-resource WebSocket streams** (`/ws/channels|agents|sessions|notifications/{id}`) require the **CEO panel token** — the same signed `X-Agent-Token` nginx injects for the panel. An agent on the Docker network can no longer subscribe to another agent's notifications with no auth. They still validate `agent_id`/`viewer_id` against the DB and channel access on top.
- `/ws/system` is fully unauthenticated. - The **`/api/v1/do/*` content routes** require a valid per-agent HMAC token bound to `X-Agent-ID` (the do router serves every role, so the gate is token-only, not role-specific).
- The **live-chat bridges** (`/prompter/live/*`, `/secretary/live/*`) — the prompter/secretary intake chats — require the CEO panel token on their start/stream/status/messages/stop endpoints. They were the last panel-facing API surface that ran unauthenticated.
- **`/ws/system`** stays operator-only and read-only by design (it carries system telemetry and accepts nothing from the client); it is not token-gated.
The streams are read-only and carry no control surface or secrets, so this isn't a privilege-escalation path the way the REST headers are — but it does mean the orchestrator port should stay trusted-network-only until WebSocket auth lands, even when you've enabled secure-mode REST. A presented-but-forged token is rejected even in dev (header-trust) mode, so you can roll out tokens before flipping the switch without breaking anything. The container→relay internal callback is left ungated by design (internal Docker network, opaque session id).
## What to do ## What to do
+7 -7
View File
@@ -8,16 +8,16 @@ There are four per-resource streams plus one operator-wide stream:
| Endpoint | Stream | Auth | | Endpoint | Stream | Auth |
|----------|--------|------| |----------|--------|------|
| `/ws/channels/{channel_id}` | Live messages in a team channel | `agent_id` query param, validated against the DB + channel access | | `/ws/channels/{channel_id}` | Live messages in a team channel | `agent_id` query param, validated against the DB + channel access; **CEO panel token required in secure mode** |
| `/ws/agents/{agent_id}` | An agent's output and lifecycle events | `viewer_id`/`agent_id` query param, validated against the DB | | `/ws/agents/{agent_id}` | An agent's output and lifecycle events | `viewer_id`/`agent_id` query param, validated against the DB; **CEO panel token required in secure mode** |
| `/ws/sessions/{session_id}` | Messages in a communication session | `agent_id` query param, validated | | `/ws/sessions/{session_id}` | Messages in a communication session | `agent_id` query param, validated; **CEO panel token required in secure mode** |
| `/ws/notifications/{agent_id}` | An agent's notifications | `agent_id` query param, validated | | `/ws/notifications/{agent_id}` | An agent's notifications | `agent_id` query param, validated; **CEO panel token required in secure mode** |
| `/ws/system` | Operator/system-wide stream — no per-agent keying | **Unauthenticated, read-only** | | `/ws/system` | Operator/system-wide stream — no per-agent keying | **Unauthenticated, read-only** (operator-only by design; not token-gated) |
All sockets support a `ping`/`pong` keepalive: send `{"type": "ping"}` and you'll get a `pong` back. All sockets support a `ping`/`pong` keepalive: send `{"type": "ping"}` and you'll get a `pong` back.
!!! warning "WebSocket auth is not the REST auth" !!! info "Secure mode now covers the per-agent streams"
The per-resource sockets validate their `agent_id`/`viewer_id` query param against the database (and channel access via the permissions layer), but they do **not** enforce the HMAC `X-Agent-Token` that secure-mode REST requires — token enforcement is REST-only. `/ws/system` is intentionally fully unauthenticated. None of the streams carry a control surface or secrets, so they're read-only by design, but the orchestrator port should be treated as trusted-network-only until WebSocket auth lands. See [Authentication](./auth.md) and [Security](../troubleshooting/security.md). When `ROBOCO_AGENT_AUTH_REQUIRED=true`, the four per-resource sockets require the **CEO panel token** (the signed `X-Agent-Token` nginx injects for the panel) on top of their `agent_id`/`viewer_id` DB validation — an agent on the Docker network can no longer subscribe to another agent's stream unauthenticated. `/ws/system` is intentionally left operator-only and read-only. A forged token is rejected even in dev mode. See [Authentication](./auth.md).
## How events reach the sockets ## How events reach the sockets
+5 -2
View File
@@ -17,8 +17,8 @@ Two more read-only servers give agents a read-only view of git (`status`, `log`,
At spawn, every agent is handed a **manifest** listing exactly the verbs its role may call — and nothing else. The manifest is built from a server-side role configuration and mounted read-only into the container. The result is that the lifecycle's role rules aren't just policy, they're *unreachable code* for the wrong role: At spawn, every agent is handed a **manifest** listing exactly the verbs its role may call — and nothing else. The manifest is built from a server-side role configuration and mounted read-only into the container. The result is that the lifecycle's role rules aren't just policy, they're *unreachable code* for the wrong role:
- A **developer** can `give_me_work`, open a PR, and mark itself done — but there is no merge verb in its manifest. - A **developer** can `give_me_work`, open a PR, mark itself done, and `sync_branch` (rebase its branch onto its base through the gate) — but there is no merge verb in its manifest.
- **QA** can claim a review and pass or fail it — but it has no `commit`. - **QA** can claim a review and pass or fail it — but it has no `commit`. QA and Documenters also get `i_am_blocked` as their escape hatch when they're stuck.
- A **PR reviewer** can pass or fail an assembled PR and post its review on the PR — but it never gets agent chat verbs. - A **PR reviewer** can pass or fail an assembled PR and post its review on the PR — but it never gets agent chat verbs.
- The **Auditor** is restricted to leaving a private note and reading evidence; it cannot `say` or `dm`. It observes; it does not participate. - The **Auditor** is restricted to leaving a private note and reading evidence; it cannot `say` or `dm`. It observes; it does not participate.
@@ -38,6 +38,9 @@ That `next` / `remediate` contract is why agents move through the lifecycle reli
A few more protections run by construction, the same way on every backend (Claude or Grok): A few more protections run by construction, the same way on every backend (Claude or Grok):
- **Claim-locking** serializes work, so two agents can't grab the same task or race a merge. - **Claim-locking** serializes work, so two agents can't grab the same task or race a merge.
- **Content posts require an active claim.** `commit`, `note`, `say`, `dm`, and `evidence` on a specific task are refused unless the agent holds that task's active claim — an agent can't write to a task it hasn't locked.
- **Human-only roles are never spawned.** The CEO, the Intake (prompter), and the Secretary are human-driven, so `spawn_agent` structurally refuses them — a notification addressed to the CEO can never launch a CEO container that acts as the human. Intake and Secretary run through their own dedicated, guarded chat paths instead.
- **Notifications can't target human-only roles.** `notify` rejects the CEO/prompter/secretary as recipients — there is no agent acknowledgement path for them, so a notification to them is a no-op rather than a stuck ack.
- **The token never enters the container.** Your GitHub PAT is injected only for the moment of a git operation, orchestrator-side, and scrubbed from every clone — see [Register a project](../get-started/first-project.md#what-happens-under-the-hood). - **The token never enters the container.** Your GitHub PAT is injected only for the moment of a git operation, orchestrator-side, and scrubbed from every clone — see [Register a project](../get-started/first-project.md#what-happens-under-the-hood).
- **A prompt-injection guard** screens task prompts, and a bash guard blocks credential-exfiltration and identity-forgery patterns. - **A prompt-injection guard** screens task prompts, and a bash guard blocks credential-exfiltration and identity-forgery patterns.
- **Rate limits and overloads park, they don't crash-loop.** If a provider returns a 429 or a persistent overload, RoboCo *queues* that agent's work and probes for recovery instead of burning tokens retrying. You'll see an amber banner; the work resumes automatically when the provider does. - **Rate limits and overloads park, they don't crash-loop.** If a provider returns a 429 or a persistent overload, RoboCo *queues* that agent's work and probes for recovery instead of burning tokens retrying. You'll see an amber banner; the work resumes automatically when the provider does.
+2
View File
@@ -25,6 +25,8 @@ For each task it proposes, the agent declares a small **collision surface**: whi
The waves are just ordinary task dependencies, so the same dependency-gate that already paces the rest of the company runs them: a wave starts only once the previous wave's tasks have reached a terminal state — normally each one's pull request is merged (a cancelled task releases the next wave too). The waves are just ordinary task dependencies, so the same dependency-gate that already paces the rest of the company runs them: a wave starts only once the previous wave's tasks have reached a terminal state — normally each one's pull request is merged (a cancelled task releases the next wave too).
The same collision-aware sequencing follows the work **down the chain**, not just at the top level. When a cell PM delegates a root-subtask into developer tasks, the dev-task collision surfaces flow through the same DAG — file-overlap serializes, migration-adders chain, shared-surface edits wait their turn — and cell tasks themselves wave-chain off their sibling root-subtasks. So a batch that spans a shared codebase stays ordered all the way to the leaves, not only at the umbrella. The task hierarchy is capped at four layers (umbrella → root → cell → dev) to fit this MegaTask shape.
## What gets created ## What gets created
When you confirm, RoboCo creates one **umbrella** task that groups the batch, and one **root-subtask** per piece of work: When you confirm, RoboCo creates one **umbrella** task that groups the batch, and one **root-subtask** per piece of work:
+9
View File
@@ -40,6 +40,15 @@ graph BT
Each of those assembled pull requests passes through the [in-path PR-review gate](task-lifecycle.md#the-in-path-pr-review-gate) before its PM merges it. Each of those assembled pull requests passes through the [in-path PR-review gate](task-lifecycle.md#the-in-path-pr-review-gate) before its PM merges it.
## Submit gates that keep the chain clean
Two gate-level checks stop a stale branch from sneaking through:
- **Behind-base gate on `i_am_done`.** If a sibling's PR merged into the parent branch while the developer worked, the dev's branch is now behind its base and the assembled PR won't merge cleanly. The gate refuses `i_am_done` in that state and steers the developer to `sync_branch` — the gate-level rebase verb that rebases the branch onto its base (raw shell git is denied to agents, so the rebase goes through the gate, traced and evidenced). Conflicts abort with no force-push and point the dev at resolve-by-hand. The gate fails open on a flaky fetch so a transient git error can't strand a task at the submit gate.
- **Unchanged-PR gate on `submit_root`.** When a Main-PM root PR is `pr_fail`'d and re-submitted byte-identical, the loop would repeat forever. The gate refuses the re-submit when the assembled root PR's head SHA is unchanged since the last `pr_fail` (no new cell work → identical diff); a different SHA means the branch advanced and the submit proceeds. Every ambiguous case fails open.
PR operations are also **scoped per project** — `open_pr`, `pr_target`, `close_pull_request`, and `merge_pr` all require the project and resolve the PR number within it, so two tasks in different repos that happen to share a PR number can never collide and merge the wrong repository's PR.
## Only the CEO merges to master ## Only the CEO merges to master
The final pull request — root → master — is the one place the company stops and hands the decision back to you. It lands in your **CEO Approval Queue** and waits. The final pull request — root → master — is the one place the company stops and hands the decision back to you. It lands in your **CEO Approval Queue** and waits.
+3
View File
@@ -54,6 +54,8 @@ stateDiagram-v2
Rejection isn't a dead end — it's a loop. When **QA fails** a task, or a **PR reviewer rejects** an assembled pull request, the task drops back to `needs_revision`, the developer reworks it, and it re-enters the flow. The same is true when *you* request changes from the CEO Approval Queue. Nothing is lost; the task carries its history, branch, and pull request with it the whole way around. Rejection isn't a dead end — it's a loop. When **QA fails** a task, or a **PR reviewer rejects** an assembled pull request, the task drops back to `needs_revision`, the developer reworks it, and it re-enters the flow. The same is true when *you* request changes from the CEO Approval Queue. Nothing is lost; the task carries its history, branch, and pull request with it the whole way around.
A failed developer task is routed back to **the developer who worked it** (resolved from the work session), not the pool — so the revision lands with whoever has the context, rather than being re-claimed cold by a cell PM. Only a task no developer ever touched falls back to the pool.
## The in-path PR-review gate ## The in-path PR-review gate
Most leaf developer tasks are reviewed by QA and never need a separate PR review. But when work is **assembled and pushed up the chain as a pull request**, it stops for a dedicated review before any PM merges it: Most leaf developer tasks are reviewed by QA and never need a separate PR review. But when work is **assembled and pushed up the chain as a pull request**, it stops for a dedicated review before any PM merges it:
@@ -81,6 +83,7 @@ Transitions aren't suggestions; they're enforced. A handful of the rules:
- **`pr_pass` / `pr_fail`** are PR-reviewer-only. - **`pr_pass` / `pr_fail`** are PR-reviewer-only.
- **Merging** (`awaiting_pm_review → completed`) is PM-only; **escalating to the CEO** and the final **approve / request-changes / cancel** are CEO-only. - **Merging** (`awaiting_pm_review → completed`) is PM-only; **escalating to the CEO** and the final **approve / request-changes / cancel** are CEO-only.
- **Cancelling** is PM-only. - **Cancelling** is PM-only.
- **A Main-PM coordination root can never be `task_type=code`.** The Main PM coordinates; it doesn't write code itself, so the combination is rejected at creation — a structural guard, not a hint.
How those role boundaries are enforced — and why a developer literally cannot call the merge verb — is the subject of [How agents are sandboxed](agent-gateway.md). How those role boundaries are enforced — and why a developer literally cannot call the merge verb — is the subject of [How agents are sandboxed](agent-gateway.md).
+1 -1
View File
@@ -28,7 +28,7 @@ The Grok access token has a fixed ~6-hour, server-set lifetime, and the CLI can'
- As a backstop, each agent's entrypoint runs `python -m roboco.llm.providers.grok_auth --check` and **refuses to start** on a missing or expired token instead of hanging. - As a backstop, each agent's entrypoint runs `python -m roboco.llm.providers.grok_auth --check` and **refuses to start** on a missing or expired token instead of hanging.
!!! warning "The orchestrator's `~/.grok` mount must be writable" !!! warning "The orchestrator's `~/.grok` mount must be writable"
The orchestrator rewrites `auth.json` when it refreshes the token, so the orchestrator's own mount of `~/.grok` must be **read-write**. (The per-agent mount stays read-only — agents only read the credential.) If the orchestrator can't write it, the token will expire and Grok agents will fail their start-up `--check`. The orchestrator rewrites `auth.json` when it refreshes the token, so the orchestrator's own mount of `~/.grok` must be **read-write**. (The per-agent mount stays read-only — agents only read the credential.) If the orchestrator can't write it, the token will expire and Grok agents will fail their start-up `--check`. If the host `auth.json` is missing entirely at spawn time, the orchestrator logs a loud warning — a missing credential is the most common Grok misconfiguration, so it's surfaced early rather than as a fleet of failed starts.
## Per-fleet tuning ## Per-fleet tuning
+3
View File
@@ -38,6 +38,9 @@ The crucial property: **work is queued, never dropped.** Parked tasks wait; the
!!! tip "Parked is not stuck" !!! tip "Parked is not stuck"
If a run goes quiet, check the banner before assuming something broke. A parked provider with a counting-down timer is RoboCo waiting out a rate limit on purpose. The work is held and will resume — there's nothing for you to do. If a run goes quiet, check the banner before assuming something broke. A parked provider with a counting-down timer is RoboCo waiting out a rate limit on purpose. The work is held and will resume — there's nothing for you to do.
!!! info "Escape hatch for a probe that never recovers"
Park-and-probe assumes the provider comes back. If a provider's probe fails persistently (the secret was rotated, the endpoint moved), an escape hatch releases the parked work back to the pool instead of holding it forever — so a permanently-dead provider doesn't strand its tasks. Grok auth-missing (exit 78) is parked the same way rather than crash-retried straight back into the same missing-credential failure.
## Disk housekeeping: dangling-image prune ## Disk housekeeping: dangling-image prune
Every agent-image rebuild leaves the previous build behind as a dangling (`<none>`) Docker image. Left alone they pile up and eat disk. The orchestrator's background sweep prunes them on a throttle (~6h): it removes **only** dangling images — a tagged image, or one still backing a running container, is never touched. It is gated by `ROBOCO_IMAGE_PRUNE_ENABLED`, which is **on by default**. This isn't a feature flag you opt into; it's an always-on safety net you can disable if you'd rather manage image cleanup yourself. Every agent-image rebuild leaves the previous build behind as a dangling (`<none>`) Docker image. Left alone they pile up and eat disk. The orchestrator's background sweep prunes them on a throttle (~6h): it removes **only** dangling images — a tagged image, or one still backing a running container, is never touched. It is gated by `ROBOCO_IMAGE_PRUNE_ENABLED`, which is **on by default**. This isn't a feature flag you opt into; it's an always-on safety net you can disable if you'd rather manage image cleanup yourself.
+2 -2
View File
@@ -14,7 +14,7 @@ The rules live in a per-project `.roboco/conventions.yml` with four curated part
|------|-----------| |------|-----------|
| **Module map** | Path prefixes mapped to a human purpose and the definition *kinds* forbidden there (`model`, `route`, `helper`, `business_logic`, `component`). "`routers/` is for HTTP routes — no models, no helpers." | | **Module map** | Path prefixes mapped to a human purpose and the definition *kinds* forbidden there (`model`, `route`, `helper`, `business_logic`, `component`). "`routers/` is for HTTP routes — no models, no helpers." |
| **Rules** | A toggleable rule set. Each rule fires at `warn` (advisory, never blocks) or `block` (refuses the gate). | | **Rules** | A toggleable rule set. Each rule fires at `warn` (advisory, never blocks) or `block` (refuses the gate). |
| **Custom rules** | Project-specific regex rules — a pattern, a message, and a level, optionally scoped to languages. | | **Custom rules** | Project-specific regex rules — a pattern, a message, and a level, optionally scoped to languages. TypeScript-scoped custom rules apply to both `.ts` and `.tsx` files. |
| **Waivers** | Accountable per-`(path, rule)` escape hatches with a written reason — the sanctioned way to relieve a false positive, reviewed in the PR. | | **Waivers** | Accountable per-`(path, rule)` escape hatches with a written reason — the sanctioned way to relieve a false positive, reviewed in the PR. |
### Placement, hygiene, and modularity checks ### Placement, hygiene, and modularity checks
@@ -34,7 +34,7 @@ The validator runs four check families over each changed file:
| `god_class` | A class grows past 15 methods (single-responsibility smell) | `warn` | | `god_class` | A class grows past 15 methods (single-responsibility smell) | `warn` |
!!! info "Precision over recall" !!! info "Precision over recall"
Every check fires only on a confident, structural signal, and abstains when it is uncertain — so a `block`-level gate is never tripped by a guess. If the validator genuinely *cannot* run on a diff (a parse or grammar error), it is **fail-loud**: it exits non-zero and the gate blocks rather than passing silently. Every check fires only on a confident, structural signal, and abstains when it is uncertain — so a `block`-level gate is never tripped by a guess. If the validator genuinely *cannot* run on a diff (a parse or grammar error), it is **fail-loud**: it exits non-zero and the gate blocks rather than passing silently. The validator is also **time-bounded** — a hung run (a tree-sitter deadlock, an enormous repo) is killed after 120s and treated as `could_not_run`, so a stuck subprocess can't hang the `i_am_done` / `pr_pass` gate forever or orphan a process on restart. And if the *effective map itself* can't be resolved (a conventions-service error), the gate **fails closed** rather than silently disabling the standard for that task.
## The effective map: defaults, present, absent, or partial ## The effective map: defaults, present, absent, or partial
+1 -1
View File
@@ -52,7 +52,7 @@ A swim-lane board of the delivery pipeline, switched with the `?view=` query par
| **PR Review** | `pr-review` | assembled PRs at the in-path review gate | | **PR Review** | `pr-review` | assembled PRs at the in-path review gate |
| **PM** | `pm` | tasks awaiting PM review and merge | | **PM** | `pm` | tasks awaiting PM review and merge |
Each board is a read-at-a-glance view of where work sits in the [lifecycle](../company/task-lifecycle.md). Switching tabs updates the URL, so a specific board is shareable. Each board is a read-at-a-glance view of where work sits in the [lifecycle](../company/task-lifecycle.md). Switching tabs updates the URL, so a specific board is shareable. A drag that would skip a lifecycle precondition (moving a task past a gate it hasn't passed) opens a confirmation dialog first, so an accidental drop can't silently bypass the flow.
## Next ## Next
+17 -1
View File
@@ -150,12 +150,28 @@ notify(target="be-dev-1", text="Please prioritise task X by EOD.",
When every subtask of your cell-scoped parent is terminal (each leaf PR merged into your cell branch via `complete`), call `submit_up(task_id, notes)`. This opens the **cell→root PR** and moves the parent into the in-path PR-review gate (`awaiting_pr_review`), where your cell's **PR reviewer** reviews the assembled diff: When every subtask of your cell-scoped parent is terminal (each leaf PR merged into your cell branch via `complete`), call `submit_up(task_id, notes)`. This opens the **cell→root PR** and moves the parent into the in-path PR-review gate (`awaiting_pr_review`), where your cell's **PR reviewer** reviews the assembled diff:
- `pr_pass` → the parent moves to `awaiting_pm_review`; you then `complete(task_id, notes)` to merge the cell→root PR into the root branch. - `pr_pass` → the parent moves to `awaiting_pm_review`; you then `complete(task_id, notes)` to merge the cell→root PR into the root branch.
- `pr_fail` → the parent returns to `needs_revision` (owned by you) with the reviewer's issues; fix, then re-`submit_up`. - `pr_fail` → the parent returns to `needs_revision` (owned by you) with the reviewer's issues; fix, then re-`submit_up`. The reviewer's verdict + issues are carried in your task handoff, so you are not blind on the rework.
Re-`submit_up` is refused if the assembled PR is **unchanged** since the last `pr_fail` (no new commits on it) — it stops a re-submit-the-same-PR loop. Fix the issues and commit before re-submitting.
You merge your own cell→root PR — the Main PM does **not** merge your cell branch. The Main PM owns the **root** task: once every cell's parent is terminal, it runs the same gate one level up (`submit_root` → main reviewer → escalate to CEO) and only the CEO merges to `master`. You never open or merge a master PR yourself. You merge your own cell→root PR — the Main PM does **not** merge your cell branch. The Main PM owns the **root** task: once every cell's parent is terminal, it runs the same gate one level up (`submit_root` → main reviewer → escalate to CEO) and only the CEO merges to `master`. You never open or merge a master PR yourself.
`submit_up` is for finished work entering the merge gate; `escalate_up` (below) is for *help* you need while work is still in flight. `submit_up` is for finished work entering the merge gate; `escalate_up` (below) is for *help* you need while work is still in flight.
### Sequencing dev-task collisions
When you `delegate` a dev subtask you may pass the collision surface so the sequencing DAG orders siblings that touch the same files:
```python
delegate(parent_task_id=..., ...,
intends_to_touch=["roboco/api/routes/*.py"], # file globs
adds_migration=False, # adds a DB migration
touches_shared=True, # edits a shared module
depends_on=["<sibling-task-id>"]) # explicit ordering
```
Siblings whose `intends_to_touch` globs overlap are serialized (more-important first); migration-adders chain serially; a shared-surface edit runs after each non-shared task it overlaps. Omit these and only the weak assignee-keyed spawn barrier orders your dev tasks (the 2026-06-27 out-of-order break).
## Escalating to Main PM ## Escalating to Main PM
Use `escalate_up(task_id, reason)` when: Use `escalate_up(task_id, reason)` when:
+1 -1
View File
@@ -111,7 +111,7 @@ master ← feature/main_pm/{root} ← feature/{cell}/{root}/{cell-pm} ←
``` ```
- A cell PM's `complete` merges a leaf PR into its cell branch; after the cell gate, its `complete` merges the cell→root PR into your root branch. You do not merge cell branches. - A cell PM's `complete` merges a leaf PR into its cell branch; after the cell gate, its `complete` merges the cell→root PR into your root branch. You do not merge cell branches.
- Once every cell's parent is terminal, **`submit_root(root_task_id, notes)`** opens the root→master PR and enters the in-path gate (`awaiting_pr_review`). The **main PR reviewer** checks the assembled root diff: `pr_pass``awaiting_pm_review`; `pr_fail``needs_revision` (owned by you, fix + re-`submit_root`). - Once every cell's parent is terminal, **`submit_root(root_task_id, notes)`** opens the root→master PR and enters the in-path gate (`awaiting_pr_review`). The **main PR reviewer** checks the assembled root diff: `pr_pass``awaiting_pm_review`; `pr_fail``needs_revision` (owned by you, fix + re-`submit_root`). The reviewer's verdict + issues are carried in your task handoff, and re-`submit_root` is refused if the root PR is **unchanged** since the last `pr_fail` — fix and commit before re-submitting.
- After `pr_pass`, `complete(root_task_id, notes)` escalates the root to the CEO (`awaiting_ceo_approval`) — it does **not** merge. A branchless coordination root (product fan-out, no repo) skips the gate and `complete` escalates directly. - After `pr_pass`, `complete(root_task_id, notes)` escalates the root to the CEO (`awaiting_ceo_approval`) — it does **not** merge. A branchless coordination root (product fan-out, no repo) skips the gate and `complete` escalates directly.
- The CEO approves and merges the root→master PR from the panel. Only the CEO ever merges to `master`. - The CEO approves and merges the root→master PR from the panel. Only the CEO ever merges to `master`.
+3 -1
View File
@@ -21,7 +21,9 @@ The `pr_reviewer` role also runs the **in-path gate** on the org's OWN assembled
### Gate enforcement ### Gate enforcement
When the architectural-conventions standard is enabled, `pr_pass` is refused on any block-level convention finding, the same way the developer's `i_am_done` is. When toolchain matching is enabled, `pr_pass` is likewise refused on a "broken" toolchain status. Your verdict note is a mandatory structured field (`pr_reviewer_notes`) written at `pr_pass` / `pr_fail`; it is persisted structured with a derived text mirror. When the architectural-conventions standard is enabled, `pr_pass` is refused on any block-level convention finding, the same way the developer's `i_am_done` is — the remediation hint points you at the offending `file:line` + the `pr_fail` verb (not `i_am_blocked`). When toolchain matching is enabled, `pr_pass` is likewise refused on a "broken" toolchain status. Your verdict note is a mandatory structured field (`pr_reviewer_notes`) written at `pr_pass` / `pr_fail`; it is persisted structured with a derived text mirror.
You cannot `pr_pass` / `pr_fail` an assembled PR you authored (self-review guard, same shape as QA's). A `claim_gate_review` on your own work returns `not_authorized`.
## What You CAN Do ## What You CAN Do
+5 -2
View File
@@ -22,6 +22,7 @@
- Read-only inspect git via `roboco_git_status / _log / _diff / _branch_list` - Read-only inspect git via `roboco_git_status / _log / _diff / _branch_list`
- Search the knowledge base via `roboco_ask_mentor` / `roboco_kb_search` - Search the knowledge base via `roboco_ask_mentor` / `roboco_kb_search`
- Note evidence via `note(text=..., scope="...")` and `evidence(...)` - Note evidence via `note(text=..., scope="...")` and `evidence(...)`
- Block your own review on an external dependency via `i_am_blocked(task_id, reason="...")` (Cell PM unblocks)
## What You CANNOT Do ## What You CANNOT Do
@@ -41,6 +42,8 @@ claim_review(task_id) → claim for review
pass(task_id, notes) → moves to awaiting_documentation pass(task_id, notes) → moves to awaiting_documentation
fail(task_id, issues=[...]) → moves to needs_revision; the dev's fail(task_id, issues=[...]) → moves to needs_revision; the dev's
original assignee gets it back original assignee gets it back
i_am_blocked(task_id, reason=...) → external blocker (broken env, can't
reproduce); Cell PM unblocks
unclaim(task_id) / resume(task_id) / i_am_idle() unclaim(task_id) / resume(task_id) / i_am_idle()
``` ```
@@ -48,7 +51,7 @@ unclaim(task_id) / resume(task_id) / i_am_idle()
| MCP server | Verbs you can call | | MCP server | Verbs you can call |
|-----------------------|--------------------| |-----------------------|--------------------|
| `roboco-flow` | `give_me_work`, `claim_review`, `pass`, `fail`, `unclaim`, `resume`, `i_am_idle` | | `roboco-flow` | `give_me_work`, `claim_review`, `pass`, `fail`, `i_am_blocked`, `unclaim`, `resume`, `i_am_idle` |
| `roboco-do` | `note`, `say`, `dm`, `evidence` (no `commit`, no `notify`) | | `roboco-do` | `note`, `say`, `dm`, `evidence` (no `commit`, no `notify`) |
| `roboco-git-readonly` | `roboco_git_status`, `roboco_git_log`, `roboco_git_diff`, `roboco_git_branch_list` | | `roboco-git-readonly` | `roboco_git_status`, `roboco_git_log`, `roboco_git_diff`, `roboco_git_branch_list` |
| `roboco-optimal` | `roboco_ask_mentor`, `roboco_kb_search` | | `roboco-optimal` | `roboco_ask_mentor`, `roboco_kb_search` |
@@ -118,4 +121,4 @@ dm(recipient="be-pm",
task_id="...") task_id="...")
``` ```
If the situation is unresolvable from the QA side (e.g. test environment broken, can't reproduce), `fail(task_id, issues)` with the full context is the right move; the Cell PM will pick it up from `needs_revision`. For an external blocker (test environment broken, can't reproduce, missing infra), use `i_am_blocked(task_id, reason="...")` — your Cell PM is notified and `unblock`s you. If the work itself is wrong, `fail(task_id, issues)` with the full context is the right move; the Cell PM picks it up from `needs_revision`.
+1 -1
View File
@@ -54,7 +54,7 @@ A single Python CLI classifies every changed definition with tree-sitter (Python
python -m roboco.conventions check --root <repo> --files <a> <b> ... python -m roboco.conventions check --root <repo> --files <a> <b> ...
``` ```
It favours precision over recall — it abstains when it cannot classify a definition, so a `block` gate can never strand a task on a guess — and it fails loud: a validator that cannot run exits non-zero so the gate blocks rather than silently passing. It favours precision over recall — it abstains when it cannot classify a definition, so a `block` gate can never strand a task on a guess — and it fails loud: a validator that cannot run exits non-zero so the gate blocks rather than silently passing. A **hung** validator is reaped after a timeout and the gate blocks the same way, and a conventions **resolution error** (effective-map build failure) fails closed — the gate is never silently disabled by an upstream error.
## Modularity ## Modularity
+4
View File
@@ -18,6 +18,8 @@ Don't invent channel slugs. Call `channels()` first if unsure:
channels() # -> {"writable": [...], "readable": [...]} channels() # -> {"writable": [...], "readable": [...]}
``` ```
**Active-claim required (explicit `task_id`):** when you pass an explicit `task_id`, `say` / `dm` / `note` check that you are the task's **active claimant** — not just `assigned_to`, which goes stale across a reap/handoff. A reaped or reassigned agent can no longer post to a former task; if you see `not_authorized` on a content post, re-`claim` the task first (or drop the explicit `task_id` for a general channel post).
Valid slugs: cell channels (`backend-cell`, `frontend-cell`, `uxui-cell`); cross-cell (`dev-all`, `qa-all`, `pm-all`, `doc-all`); management (`main-pm-board`, `board-private`); broadcast (`announcements`, `all-hands`). Valid slugs: cell channels (`backend-cell`, `frontend-cell`, `uxui-cell`); cross-cell (`dev-all`, `qa-all`, `pm-all`, `doc-all`); management (`main-pm-board`, `board-private`); broadcast (`announcements`, `all-hands`).
## Direct message (A2A) — `dm` ## Direct message (A2A) — `dm`
@@ -40,6 +42,8 @@ notify(target="be-dev-1", text="Task ready for you", priority="normal", task_id=
`priority` is `normal | high | urgent`. `task_id` auto-injects from the active task when omitted. `priority` is `normal | high | urgent`. `task_id` auto-injects from the active task when omitted.
`notify` rejects **human-only recipients** (`prompter`, `secretary`) — they have no agent ack path, so an ack-required alert to them would sit unacked forever. The CEO is allowed (acks via the panel).
## Receiving notifications ## Receiving notifications
Every role with an inbox gets these (so `i_am_idle()` doesn't soft-block on unread items): Every role with an inbox gets these (so `i_am_idle()` doesn't soft-block on unread items):
+2
View File
@@ -29,6 +29,8 @@ escalate_up(
Auto-routes to your escalation target (you cannot choose it). Auto-routes to your escalation target (you cannot choose it).
`escalate_up` is refused on a **terminal** task (`completed` / `cancelled`) — it returns `invalid_state` rather than resurrecting a finished task. Escalate live work only.
## When to Escalate ## When to Escalate
| Situation | Escalate To | | Situation | Escalate To |
+1 -1
View File
@@ -14,7 +14,7 @@ Hierarchy: Umbrella (Main PM) → Root-subtasks (Main PM) → Cell tasks (cell P
- The umbrella does **no git**. It is exempt from the branch gate (it reaches `in_progress` with no branch) and you must **not** call `submit_root` on it — it assembles no PR. Each root-subtask opens and is reviewed on its own PR. - The umbrella does **no git**. It is exempt from the branch gate (it reaches `in_progress` with no branch) and you must **not** call `submit_root` on it — it assembles no PR. Each root-subtask opens and is reviewed on its own PR.
- The umbrella **completes** only when every root-subtask is terminal; then it escalates to the CEO (PR requirement waived). - The umbrella **completes** only when every root-subtask is terminal; then it escalates to the CEO (PR requirement waived).
- The root-subtasks are sequenced: a wave's tasks dispatch only once the previous wave's tasks reach a terminal state (ordinary dependency-gating). You do not reorder them — the analyzer set the order at create time. - The root-subtasks are sequenced: a wave's tasks dispatch only once the previous wave's tasks reach a terminal state (ordinary dependency-gating). You do not reorder them — the analyzer set the order at create time.
- On the Board route the root-subtasks are held in `backlog` until the CEO approves the umbrella, then released to `pending`. On the Approve & Start route they start immediately. - On the Board route the root-subtasks are held in `backlog` until the CEO approves the umbrella, then released to `pending`. On the Approve & Start route they start immediately. On Board-route activation a `code`-typed root-subtask is **retyped to `planning`** — a Main PM never owns a `code` task (the `main_pm + code` combo is the 2026-06-27 meltdown trigger).
## For the Main PM ## For the Main PM
+1 -1
View File
@@ -41,7 +41,7 @@ The claim verb both claims and starts the task — there is no separate `start`
## Claiming Rules ## Claiming Rules
- **One at a time (workers only)**: Developers, QA, and documenters can't hold multiple in-progress tasks at once. **PM coordinators are exempt** — a Main / Cell PM plans and delegates many roots in parallel, so it may hold several at once; only a real upstream **sequence dependency** (an unfinished task it depends on) holds one of its roots back. - **One at a time (workers only)**: Developers, QA, and documenters can't hold multiple in-progress tasks at once. A **blocked** task still counts as active — a blocked dev cannot `claim` a second task; unblock or `unclaim` first. **PM coordinators are exempt** — a Main / Cell PM plans and delegates many roots in parallel, so it may hold several at once; only a real upstream **sequence dependency** (an unfinished task it depends on) holds one of its roots back.
- **Self-review prevention**: QA cannot `claim_review` tasks they developed - **Self-review prevention**: QA cannot `claim_review` tasks they developed
- **Self-documentation prevention**: Documenter cannot claim tasks they developed - **Self-documentation prevention**: Documenter cannot claim tasks they developed
- **Branch requirement**: Branch auto-created on `i_will_work_on` - **Branch requirement**: Branch auto-created on `i_will_work_on`
+9 -2
View File
@@ -45,9 +45,16 @@ Project GitHub tokens are **encrypted the moment you save them** (with `ROBOCO_E
This is the guarantee that makes it safe to hand RoboCo a private repo: **your GitHub PAT is never present inside an agent container.** The orchestrator decrypts the token only at the moment of a git operation, injects it for that operation, and immediately after cloning **scrubs the token out of the clone's git config** — then verifies no token byte survives anywhere under `.git/`, destroying the workspace if one did. A compromised or misbehaving agent has nothing to exfiltrate, because the credential was never on its disk. The clone scrub is described in [Register a project](../get-started/first-project.md#what-happens-under-the-hood), and the broader sandboxing model in [the gateway](../company/agent-gateway.md). This is the guarantee that makes it safe to hand RoboCo a private repo: **your GitHub PAT is never present inside an agent container.** The orchestrator decrypts the token only at the moment of a git operation, injects it for that operation, and immediately after cloning **scrubs the token out of the clone's git config** — then verifies no token byte survives anywhere under `.git/`, destroying the workspace if one did. A compromised or misbehaving agent has nothing to exfiltrate, because the credential was never on its disk. The clone scrub is described in [Register a project](../get-started/first-project.md#what-happens-under-the-hood), and the broader sandboxing model in [the gateway](../company/agent-gateway.md).
## WebSocket auth caveat ## WebSocket + live-chat auth
The per-resource WebSocket streams are keyed to a resource, but the operator stream is not authenticated. **`/ws/system` carries no per-agent keying and no token** even when `ROBOCO_AGENT_AUTH_REQUIRED=true` — secure mode does not extend to it. It is **read-only** (it carries system events like rate-limit lifecycle and usage snapshots; it accepts nothing from the client), so the exposure is limited to a reader seeing system telemetry. It is, however, one more reason the system must sit on a trusted network: anyone who can open that socket can watch the operator stream. Secure mode (`ROBOCO_AGENT_AUTH_REQUIRED=true`) extends to the live streams and the content routes, not just REST:
- The **per-resource WebSocket streams** (`/ws/channels|agents|sessions|notifications/{id}`) require the **CEO panel token** — the signed `X-Agent-Token` nginx injects for the panel — on top of their `agent_id`/`viewer_id` DB validation. An agent on the Docker network can no longer subscribe to another agent's notifications with no auth.
- The **`/api/v1/do/*` content routes** require a per-agent HMAC token bound to `X-Agent-ID`.
- The **live-chat bridges** (`/prompter/live/*`, `/secretary/live/*`) require the CEO panel token — they were the last panel-facing surface that ran unauthenticated.
- **`/ws/system`** is the one exception: it stays operator-only and read-only by design (system telemetry, no client input), and is not token-gated. It is one more reason the system must sit on a trusted network: anyone who can open that socket can watch the operator stream.
A presented-but-forged token is rejected even in header-trust (dev) mode, so rolling out tokens before flipping the switch breaks nothing.
## Next ## Next
+7 -8
View File
@@ -297,11 +297,10 @@ async def subscribe_to_task(
Opens a persistent connection that streams task state changes Opens a persistent connection that streams task state changes
until the task reaches a terminal state or client disconnects. until the task reaches a terminal state or client disconnects.
F024: each poll opens a SHORT-LIVED session via ``get_session_factory`` Each poll opens a SHORT-LIVED session via ``get_session_factory`` and
and closes it before the next ``asyncio.sleep`` — never holding one closes it before the next ``asyncio.sleep`` — never holding one asyncpg
asyncpg connection across the full SSE lifetime (up to 1 hour / 720 connection across the full SSE lifetime (up to 1 hour / 720 polls). The
polls), which previously exhausted the pool one connection per connected route takes no ``db: DbSession`` for the same reason.
client. The route takes no ``db: DbSession`` for the same reason.
""" """
session_factory = get_session_factory() session_factory = get_session_factory()
@@ -324,9 +323,9 @@ async def subscribe_to_task(
if await request.is_disconnected(): if await request.is_disconnected():
break break
# F024: refresh task state from a per-poll session that is # Refresh task state from a per-poll session released before the
# released before the sleep below — never held across the poll # sleep — never held across the poll interval, so the asyncpg pool
# interval, so the asyncpg pool is free between queries. # is free between queries.
async with session_factory() as session: async with session_factory() as session:
task = await A2AService(session).get_task(task_id) task = await A2AService(session).get_task(task_id)
if task is None: if task is None:
+4 -5
View File
@@ -2024,11 +2024,10 @@ async def escalate_task(
raise HTTPException( raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND, detail="Task not found" status_code=status.HTTP_404_NOT_FOUND, detail="Task not found"
) )
# F043: a terminal task (completed / cancelled) must not be resurrected to # A terminal task (completed / cancelled) must not be resurrected to BLOCKED
# BLOCKED via escalation. Refuse BEFORE sending the escalation notification # via escalation — refuse BEFORE sending the notification so a finished task
# so a finished/cancelled task isn't yanked back into the workflow (and the # isn't yanked back into the workflow. apply_escalation guards this too
# PM isn't pinged about a task that's already done). The single write # (defense in depth).
# primitive apply_escalation guards this too — defense in depth.
if task.status in (TaskStatus.COMPLETED, TaskStatus.CANCELLED): if task.status in (TaskStatus.COMPLETED, TaskStatus.CANCELLED):
raise HTTPException( raise HTTPException(
status_code=status.HTTP_409_CONFLICT, status_code=status.HTTP_409_CONFLICT,
+1 -1
View File
@@ -62,7 +62,7 @@ require_pr_reviewer = _require_roles(frozenset({Role.PR_REVIEWER}))
def _require_authenticated_agent() -> params.Depends: def _require_authenticated_agent() -> params.Depends:
"""Token-only guard for the content-tool (do) router (F003/F014). """Token-only guard for the content-tool (do) router.
The do router serves every role content tools are role-uniform, with The do router serves every role content tools are role-uniform, with
per-role removal handled in the spawn manifest so, unlike the flow per-role removal handled in the spawn manifest so, unlike the flow
+2 -3
View File
@@ -104,9 +104,8 @@ async def i_am_blocked(
x_agent_id: _AgentIdHeader, x_agent_id: _AgentIdHeader,
choreographer: _ChoreographerDep, choreographer: _ChoreographerDep,
) -> dict: ) -> dict:
"""F015: the documenter manifest registers ``i_am_blocked`` — surface the """Surface the ``i_am_blocked`` route so a blocked documenter's escape
route so a blocked documenter's escape hatch returns an envelope instead of hatch returns an envelope instead of a 404."""
a 404."""
env = await choreographer.i_am_blocked( env = await choreographer.i_am_blocked(
x_agent_id, x_agent_id,
body.task_id, body.task_id,
+2 -2
View File
@@ -116,8 +116,8 @@ async def i_am_blocked(
x_agent_id: _AgentIdHeader, x_agent_id: _AgentIdHeader,
choreographer: _ChoreographerDep, choreographer: _ChoreographerDep,
) -> dict: ) -> dict:
"""F015: the QA manifest registers ``i_am_blocked`` — surface the route so a """Surface the ``i_am_blocked`` route so a blocked QA agent's escape hatch
blocked QA agent's escape hatch returns an envelope instead of a 404.""" returns an envelope instead of a 404."""
env = await choreographer.i_am_blocked( env = await choreographer.i_am_blocked(
x_agent_id, x_agent_id,
body.task_id, body.task_id,
+47 -65
View File
@@ -29,25 +29,21 @@ from roboco.services.repositories import resolve_agent_uuid
router = APIRouter() router = APIRouter()
log = structlog.get_logger() log = structlog.get_logger()
# F066: server-side idle timeout for WS receive loops. A half-open socket # Server-side idle timeout for WS receive loops. A half-open socket (dead
# (dead agent container, silent client) blocks ``receive_text()`` forever; # agent container, silent client) blocks ``receive_text()`` forever;
# wrapping it in ``asyncio.wait_for`` reaps the socket after this many # ``asyncio.wait_for`` reaps the socket after this many seconds of silence.
# seconds of silence. No env-var/config precedent exists in ``config.py``
# for WS tuning, so this is a module constant — callers/tests patch it.
IDLE_TIMEOUT_SECONDS: float = 90.0 IDLE_TIMEOUT_SECONDS: float = 90.0
# F064: per-connection send queue + send timeout. Each registered connection # Per-connection send queue + send timeout. Each registered connection owns a
# owns a bounded ``asyncio.Queue`` drained by a sender task, so a slow client # bounded ``asyncio.Queue`` drained by a sender task, so a slow client can't
# can't back-pressure the fan-out: broadcast enqueues (non-blocking) and # back-pressure the fan-out: broadcast enqueues (non-blocking) and returns
# returns immediately. When the queue is full the message is dropped + logged # immediately; a full queue drops + logs (client lagging, not the fan-out).
# (the client is lagging, not the whole fan-out). ``send_text`` itself is
# wrapped in ``wait_for`` so a stuck transport doesn't wedge the sender.
MAX_SEND_QUEUE: int = 256 MAX_SEND_QUEUE: int = 256
SEND_TIMEOUT_SECONDS: float = 10.0 SEND_TIMEOUT_SECONDS: float = 10.0
class _ClientConnection: class _ClientConnection:
"""F064: per-connection send queue + sender task. """Per-connection send queue + sender task.
Holds the bounded outbound queue drained by ``sender``; broadcast enqueues Holds the bounded outbound queue drained by ``sender``; broadcast enqueues
here instead of awaiting ``send_text`` directly, so one slow client cannot here instead of awaiting ``send_text`` directly, so one slow client cannot
@@ -63,20 +59,13 @@ class _ClientConnection:
async def _require_panel_token(websocket: WebSocket) -> bool: async def _require_panel_token(websocket: WebSocket) -> bool:
"""F004: bind a per-agent WS upgrade to the panel/CEO HMAC token. """Bind a per-agent WS upgrade to the panel/CEO HMAC token.
The /ws/* streams are operator-only the control panel is the sole WS /ws/* streams are operator-only (the panel is the sole WS client; agents
client (agents use MCP verbs, not WS), and nginx injects the CEO panel use MCP verbs). nginx injects the CEO panel token as ``X-Agent-Token``.
token as ``X-Agent-Token`` on /ws/ upgrades. Without verifying it the In strict mode (``ROBOCO_AGENT_AUTH_REQUIRED=true``) the token is required
per-agent endpoints (channels/agents/sessions/notifications) accepted a + verified against the CEO identity; a presented-but-forged token is
bare ``agent_id`` query param with no auth, so in strict mode rejected even in dev mode. Returns True to proceed, False to close.
(``ROBOCO_AGENT_AUTH_REQUIRED=true``) an agent on the Docker network
could hit e.g. ``/ws/notifications/{id}`` directly and subscribe to
another agent's notifications. This gate requires + verifies the token
against the CEO identity in strict mode, and rejects a presented-but-
forged token even in dev mode the same contract as the HTTP
``_check_agent_auth_token`` role gates. Returns True to proceed, False
to close with a policy violation (caller closes the socket).
""" """
token = websocket.headers.get("x-agent-token") token = websocket.headers.get("x-agent-token")
if _auth_required() and not token: if _auth_required() and not token:
@@ -119,15 +108,15 @@ class ConnectionManager:
# websocket -> agent_id (for tracking who is connected) # websocket -> agent_id (for tracking who is connected)
self.connection_agents: dict[WebSocket, UUID] = {} self.connection_agents: dict[WebSocket, UUID] = {}
# F064: websocket -> per-connection send queue + sender task. Every # websocket -> per-connection send queue + sender task. Every connect_*
# connect_* registers here; disconnect cancels + removes. Broadcast # registers here; disconnect cancels + removes. Broadcast enqueues into
# enqueues into these queues instead of awaiting send_text directly so # these queues instead of awaiting send_text directly so one slow client
# one slow client can't block the fan-out. # can't block the fan-out.
self.connection_senders: dict[WebSocket, _ClientConnection] = {} self.connection_senders: dict[WebSocket, _ClientConnection] = {}
# F064: fire-and-forget fallback send tasks for unregistered sockets # Fire-and-forget fallback send tasks for unregistered sockets (legacy
# (legacy path). Held only to satisfy ruff RUF006 + to allow clean # path). Held to satisfy ruff RUF006 + allow clean shutdown; each task
# shutdown; each task removes itself on completion. # removes itself on completion.
self._pending_sends: set[asyncio.Task[None]] = set() self._pending_sends: set[asyncio.Task[None]] = set()
def _register_sender(self, websocket: WebSocket) -> _ClientConnection: def _register_sender(self, websocket: WebSocket) -> _ClientConnection:
@@ -246,21 +235,19 @@ class ConnectionManager:
# Remove from tracking # Remove from tracking
self.connection_agents.pop(websocket, None) self.connection_agents.pop(websocket, None)
# F064: cancel + drop the per-connection sender task so a slow/stale # Cancel + drop the per-connection sender task so a slow/stale client's
# client's queue doesn't leak after the socket is removed. # queue doesn't leak after the socket is removed.
conn = self.connection_senders.pop(websocket, None) conn = self.connection_senders.pop(websocket, None)
if conn is not None and conn.sender is not None: if conn is not None and conn.sender is not None:
conn.sender.cancel() conn.sender.cancel()
def _enqueue_or_send(self, websocket: WebSocket, data: str) -> None: def _enqueue_or_send(self, websocket: WebSocket, data: str) -> None:
"""F064: fan out one message to one connection without blocking. """Fan out one message to one connection without blocking.
Registered connections (created via ``connect_*``) get the message Registered connections get the message enqueued into their bounded send
enqueued into their bounded send queue non-blocking, drop + warn on queue (non-blocking, drop + warn on overflow). An unregistered socket
overflow. An unregistered socket (legacy path: present in a falls back to a timeout-bounded ``send_text`` scheduled on the loop, so
subscription set but not in ``connection_senders``) falls back to a the broadcast never blocks on a single slow client.
timeout-bounded ``send_text`` scheduled on the loop, so the broadcast
still never blocks on a single slow client.
""" """
conn = self.connection_senders.get(websocket) conn = self.connection_senders.get(websocket)
if conn is not None: if conn is not None:
@@ -380,7 +367,7 @@ async def channel_stream(
Clients receive real-time messages for the channel. Clients receive real-time messages for the channel.
""" """
# F004: verify the panel/CEO token before any subject lookup. # Verify the panel/CEO token before any subject lookup.
if not await _require_panel_token(websocket): if not await _require_panel_token(websocket):
await websocket.close(code=status.WS_1008_POLICY_VIOLATION) await websocket.close(code=status.WS_1008_POLICY_VIOLATION)
return return
@@ -428,10 +415,9 @@ async def channel_stream(
# exit path (anyio closed-resource, CancelledError, transport errors). # exit path (anyio closed-resource, CancelledError, transport errors).
pass pass
except TimeoutError: except TimeoutError:
# F066: idle timeout — the client has been silent for # Idle timeout — the client has been silent for IDLE_TIMEOUT_SECONDS
# IDLE_TIMEOUT_SECONDS (likely a half-open socket from a dead # (likely a half-open socket from a dead container). Fall through to
# container). Log and fall through to the finally so the socket is # the finally so the socket is removed from every subscription set.
# removed from every subscription set.
log.warning( log.warning(
"WebSocket idle timeout — disconnecting", timeout=IDLE_TIMEOUT_SECONDS "WebSocket idle timeout — disconnecting", timeout=IDLE_TIMEOUT_SECONDS
) )
@@ -449,7 +435,7 @@ async def agent_stream(
Clients receive real-time LLM output from the agent. Clients receive real-time LLM output from the agent.
""" """
# F004: verify the panel/CEO token before any subject lookup. # Verify the panel/CEO token before any subject lookup.
if not await _require_panel_token(websocket): if not await _require_panel_token(websocket):
await websocket.close(code=status.WS_1008_POLICY_VIOLATION) await websocket.close(code=status.WS_1008_POLICY_VIOLATION)
return return
@@ -494,10 +480,9 @@ async def agent_stream(
# exit path (anyio closed-resource, CancelledError, transport errors). # exit path (anyio closed-resource, CancelledError, transport errors).
pass pass
except TimeoutError: except TimeoutError:
# F066: idle timeout — the client has been silent for # Idle timeout — the client has been silent for IDLE_TIMEOUT_SECONDS
# IDLE_TIMEOUT_SECONDS (likely a half-open socket from a dead # (likely a half-open socket from a dead container). Fall through to
# container). Log and fall through to the finally so the socket is # the finally so the socket is removed from every subscription set.
# removed from every subscription set.
log.warning( log.warning(
"WebSocket idle timeout — disconnecting", timeout=IDLE_TIMEOUT_SECONDS "WebSocket idle timeout — disconnecting", timeout=IDLE_TIMEOUT_SECONDS
) )
@@ -515,7 +500,7 @@ async def session_stream(
Clients receive real-time messages for a specific session. Clients receive real-time messages for a specific session.
""" """
# F004: verify the panel/CEO token before any subject lookup. # Verify the panel/CEO token before any subject lookup.
if not await _require_panel_token(websocket): if not await _require_panel_token(websocket):
await websocket.close(code=status.WS_1008_POLICY_VIOLATION) await websocket.close(code=status.WS_1008_POLICY_VIOLATION)
return return
@@ -558,10 +543,9 @@ async def session_stream(
# exit path (anyio closed-resource, CancelledError, transport errors). # exit path (anyio closed-resource, CancelledError, transport errors).
pass pass
except TimeoutError: except TimeoutError:
# F066: idle timeout — the client has been silent for # Idle timeout — the client has been silent for IDLE_TIMEOUT_SECONDS
# IDLE_TIMEOUT_SECONDS (likely a half-open socket from a dead # (likely a half-open socket from a dead container). Fall through to
# container). Log and fall through to the finally so the socket is # the finally so the socket is removed from every subscription set.
# removed from every subscription set.
log.warning( log.warning(
"WebSocket idle timeout — disconnecting", timeout=IDLE_TIMEOUT_SECONDS "WebSocket idle timeout — disconnecting", timeout=IDLE_TIMEOUT_SECONDS
) )
@@ -579,7 +563,7 @@ async def notification_stream(
Agents receive real-time notifications via this stream. Agents receive real-time notifications via this stream.
""" """
# F004: verify the panel/CEO token before any subject lookup. # Verify the panel/CEO token before any subject lookup.
if not await _require_panel_token(websocket): if not await _require_panel_token(websocket):
await websocket.close(code=status.WS_1008_POLICY_VIOLATION) await websocket.close(code=status.WS_1008_POLICY_VIOLATION)
return return
@@ -611,10 +595,9 @@ async def notification_stream(
# exit path (anyio closed-resource, CancelledError, transport errors). # exit path (anyio closed-resource, CancelledError, transport errors).
pass pass
except TimeoutError: except TimeoutError:
# F066: idle timeout — the client has been silent for # Idle timeout — the client has been silent for IDLE_TIMEOUT_SECONDS
# IDLE_TIMEOUT_SECONDS (likely a half-open socket from a dead # (likely a half-open socket from a dead container). Fall through to
# container). Log and fall through to the finally so the socket is # the finally so the socket is removed from every subscription set.
# removed from every subscription set.
log.warning( log.warning(
"WebSocket idle timeout — disconnecting", timeout=IDLE_TIMEOUT_SECONDS "WebSocket idle timeout — disconnecting", timeout=IDLE_TIMEOUT_SECONDS
) )
@@ -649,10 +632,9 @@ async def system_stream(websocket: WebSocket) -> None:
# exit path (anyio closed-resource, CancelledError, transport errors). # exit path (anyio closed-resource, CancelledError, transport errors).
pass pass
except TimeoutError: except TimeoutError:
# F066: idle timeout — the client has been silent for # Idle timeout — the client has been silent for IDLE_TIMEOUT_SECONDS
# IDLE_TIMEOUT_SECONDS (likely a half-open socket from a dead # (likely a half-open socket from a dead container). Fall through to
# container). Log and fall through to the finally so the socket is # the finally so the socket is removed from every subscription set.
# removed from every subscription set.
log.warning( log.warning(
"WebSocket idle timeout — disconnecting", timeout=IDLE_TIMEOUT_SECONDS "WebSocket idle timeout — disconnecting", timeout=IDLE_TIMEOUT_SECONDS
) )
+10 -16
View File
@@ -56,15 +56,11 @@ GROK_AUTH_HOST_PATH = os.environ.get("ROBOCO_HOST_GROK_DIR", str(Path.home() / "
# In-container paths. # In-container paths.
_MCP_CONFIG_IN_CONTAINER = "/app/mcp-config.json" _MCP_CONFIG_IN_CONTAINER = "/app/mcp-config.json"
# F005: the host ~/.grok DIRECTORY is mounted read-only here (NOT the single # The host ~/.grok DIRECTORY (not a single auth.json file) is mounted RO here:
# auth.json file). A single-file bind mount pins the inode, so the # a single-file bind mount pins the inode so the orchestrator's atomic
# orchestrator's atomic auth.json refresh (tmp+rename within the dir) never # tmp+rename refresh never reaches a running container. The entrypoint
# reached a running container — a long-lived grok container hung at the login # symlinks ~/.grok/auth.json -> this RO mount; grok's writable state lives in
# prompt when the original ~6h token expired. A directory mount sees the # the image's ~/.grok.
# rename, so the refreshed token propagates to running containers. The
# entrypoint symlinks ~/.grok/auth.json -> this RO mount so grok (and the
# --check backstop) read the live credential while grok's own writable state
# (config.toml, sessions/) still lands in the image's ~/.grok.
_GROK_AUTH_DIR_IN_CONTAINER = "/home/agent/.grok-auth-ro" _GROK_AUTH_DIR_IN_CONTAINER = "/home/agent/.grok-auth-ro"
# Per-agent data dir (the host side is reused from the shared assembly): the # Per-agent data dir (the host side is reused from the shared assembly): the
# entrypoint writes the captured token usage here so the orchestrator reads it # entrypoint writes the captured token usage here so the orchestrator reads it
@@ -166,18 +162,16 @@ class GrokCliProvider(AgentProvider):
def _append_grok_auth_mount(cmd: list[str]) -> None: def _append_grok_auth_mount(cmd: list[str]) -> None:
"""Mount the host's SuperGrok ``~/.grok`` directory (read-only). """Mount the host's SuperGrok ``~/.grok`` directory (read-only).
F005: the mount is the DIRECTORY, not the single ``auth.json`` file. The mount is the DIRECTORY, not the single ``auth.json`` file: a
A single-file bind mount pins the inode, so when the orchestrator single-file bind mount pins the inode, so when the orchestrator
atomically refreshes the token (``tmp.replace`` = rename within the atomically refreshes the token (``tmp.replace`` = rename within the
host ``~/.grok``), a running container kept reading the stale inode and host ``~/.grok``), a running container kept reading the stale inode and
hung at grok's login prompt once the original ~6h token expired. A hung at grok's login prompt once the original ~6h token expired. A
directory bind mount sees the rename, so the refreshed ``auth.json`` directory bind mount sees the rename, so the refreshed ``auth.json``
propagates to running containers. The entrypoint symlinks propagates to running containers. The entrypoint symlinks
``~/.grok/auth.json`` at this RO directory mount, so grok (and the ``~/.grok/auth.json`` at this RO directory mount; grok's own writable
``--check`` backstop) read the live credential while grok's own state (``config.toml``, ``sessions/``) lands in the image's ``~/.grok``.
writable state (``config.toml``, ``sessions/``) still lands in the Read-only so concurrent containers can't corrupt the shared credential.
image's ``~/.grok``. Read-only so concurrent containers can't corrupt
the shared subscription credential.
""" """
auth_dir = Path(GROK_AUTH_HOST_PATH) auth_dir = Path(GROK_AUTH_HOST_PATH)
if (auth_dir / "auth.json").exists(): if (auth_dir / "auth.json").exists():
+17 -21
View File
@@ -38,7 +38,7 @@ _SDK_TIMEOUT = 2.0
# FastAPI's default missing-route status. Every /api/v1/do/* route returns # FastAPI's default missing-route status. Every /api/v1/do/* route returns
# 200 with an Envelope (including not_found rejections), so a 404 from the # 200 with an Envelope (including not_found rejections), so a 404 from the
# orchestrator is always a manifest-registered tool whose HTTP route is # orchestrator is always a manifest-registered tool whose HTTP route is
# missing — F069 synthesizes an invalid_state Envelope for it. # missing — synthesize an invalid_state Envelope for it.
_MISSING_ROUTE_STATUS = 404 _MISSING_ROUTE_STATUS = 404
# Envelope error kinds that count toward the per-verb circuit breaker. # Envelope error kinds that count toward the per-verb circuit breaker.
@@ -54,8 +54,8 @@ _CIRCUIT_REJECTION_KINDS: frozenset[str] = frozenset(
# Dict-shaped `error.code` values (from FastAPI's exception handlers — # Dict-shaped `error.code` values (from FastAPI's exception handlers —
# `roboco_exception_handler` / `http_exception_handler` / `generic_exception_handler`) # `roboco_exception_handler` / `http_exception_handler` / `generic_exception_handler`)
# mapped to the counted breaker kind they are semantically equivalent to. F068: # mapped to the counted breaker kind they are semantically equivalent to. A
# a 422 / 500 / 4xx-exception storm is retry-storm-worthy but the response body # 422 / 500 / 4xx-exception storm is retry-storm-worthy but the response body
# carries `error` as a DICT (not a string kind), so the breaker's string-only # carries `error` as a DICT (not a string kind), so the breaker's string-only
# check skipped it — unbounded retries. We classify by `error.code` so the SDK # check skipped it — unbounded retries. We classify by `error.code` so the SDK
# actually records the attempt. Kinds not in `_CIRCUIT_REJECTION_KINDS` are # actually records the attempt. Kinds not in `_CIRCUIT_REJECTION_KINDS` are
@@ -88,7 +88,7 @@ def _classify_rejection(payload: dict[str, Any]) -> str | None:
The breaker only counts rejections whose kind is in The breaker only counts rejections whose kind is in
``_CIRCUIT_REJECTION_KINDS`` (the SDK's authoritative catalog). Three ``_CIRCUIT_REJECTION_KINDS`` (the SDK's authoritative catalog). Three
reachable rejection shapes must all map to a counted kind so a storm of reachable rejection shapes must all map to a counted kind so a storm of
any of them trips the breaker (F068): any of them trips the breaker:
1. Envelope rejection: ``error`` is a STRING kind. Forward it if in 1. Envelope rejection: ``error`` is a STRING kind. Forward it if in
the counted set (existing behaviour). Uncounted string kinds (e.g. the counted set (existing behaviour). Uncounted string kinds (e.g.
@@ -156,18 +156,14 @@ def _post(path: str, body: dict[str, Any]) -> dict[str, Any]:
headers=_build_headers(), headers=_build_headers(),
json=body, json=body,
) )
# F069: a 404 here means a manifest-registered content tool has no # A 404 here means a manifest-registered content tool has no matching
# matching route on the orchestrator (every /api/v1/do/* route # route on the orchestrator: every /api/v1/do/* route returns 200 with
# returns 200 with an Envelope including not_found rejections so # an Envelope (including not_found rejections), so FastAPI's default
# a 404 status with FastAPI's default body (``{"detail": "Not # 404 body (no ``error`` field) is always a missing route, never a legit
# Found"}``, no ``error`` field) is always a missing route, never a # Envelope. Synthesize an ``invalid_state`` Envelope so the breaker
# legit Envelope). That body is a non-envelope payload the breaker # counts it (via ``_classify_rejection``) and the agent gets a
# can't classify, so a storm of these bypassed the circuit breaker → # remediation hint instead of a raw ``detail`` body. A 404 carrying a
# unbounded retries on a tool that can never succeed. Synthesize an # real Envelope (``error`` field) is surfaced as-is. Mirrors
# ``invalid_state`` Envelope rejection so the breaker counts it (via
# ``_classify_rejection``) and the agent gets a remediation hint
# instead of a raw ``detail`` body. A 404 that DOES carry a real
# Envelope (an ``error`` field) is surfaced as-is. Mirrors
# flow_server._post. # flow_server._post.
if response.status_code == _MISSING_ROUTE_STATUS: if response.status_code == _MISSING_ROUTE_STATUS:
try: try:
@@ -244,11 +240,11 @@ def _record_and_check_circuit(
# Gateway envelopes use a string `error` (kind); RobocoError-derived # Gateway envelopes use a string `error` (kind); RobocoError-derived
# exceptions surface a dict-shaped error via FastAPI's middleware, and # exceptions surface a dict-shaped error via FastAPI's middleware, and
# 422 validation failures carry a `detail` list with no `error` field # 422 validation failures carry a `detail` list with no `error` field
# at all. F068: classify all three rejection shapes so a storm of 500s # at all. Classify all three rejection shapes so a storm of 500s or 422s
# or 422s counts toward the breaker (previously bypassed → unbounded # counts toward the breaker (previously bypassed → unbounded retries). The
# retries). The dict-shape defence against `TypeError: unhashable type: # dict-shape defence against `TypeError: unhashable type: 'dict'` lives in
# 'dict'` lives in `_classify_rejection` (isinstance checks, never a # `_classify_rejection` (isinstance checks, never a `dict in frozenset`
# `dict in frozenset` membership test). # membership test).
rejection_kind = _classify_rejection(payload) rejection_kind = _classify_rejection(payload)
if rejection_kind is None: if rejection_kind is None:
return payload return payload
+18 -22
View File
@@ -56,7 +56,7 @@ _SDK_TIMEOUT = 2.0
# FastAPI's default missing-route status. Every gateway route returns 200 # FastAPI's default missing-route status. Every gateway route returns 200
# with an Envelope (including not_found rejections), so a 404 from the # with an Envelope (including not_found rejections), so a 404 from the
# orchestrator is always a manifest-registered verb whose HTTP route is # orchestrator is always a manifest-registered verb whose HTTP route is
# missing — F069 synthesizes an invalid_state Envelope for it. # missing — synthesize an invalid_state Envelope for it.
_MISSING_ROUTE_STATUS = 404 _MISSING_ROUTE_STATUS = 404
# Envelope error kinds that count toward the per-verb circuit breaker. # Envelope error kinds that count toward the per-verb circuit breaker.
@@ -70,8 +70,8 @@ _CIRCUIT_REJECTION_KINDS: frozenset[str] = frozenset(
# Dict-shaped `error.code` values (from FastAPI's exception handlers — # Dict-shaped `error.code` values (from FastAPI's exception handlers —
# `roboco_exception_handler` / `http_exception_handler` / `generic_exception_handler`) # `roboco_exception_handler` / `http_exception_handler` / `generic_exception_handler`)
# mapped to the counted breaker kind they are semantically equivalent to. F068: # mapped to the counted breaker kind they are semantically equivalent to. A
# a 422 / 500 / 4xx-exception storm is retry-storm-worthy but the response body # 422 / 500 / 4xx-exception storm is retry-storm-worthy but the response body
# carries `error` as a DICT (not a string kind), so the breaker's string-only # carries `error` as a DICT (not a string kind), so the breaker's string-only
# check skipped it — unbounded retries. We classify by `error.code` so the SDK # check skipped it — unbounded retries. We classify by `error.code` so the SDK
# actually records the attempt. Kinds not in `_CIRCUIT_REJECTION_KINDS` are # actually records the attempt. Kinds not in `_CIRCUIT_REJECTION_KINDS` are
@@ -104,7 +104,7 @@ def _classify_rejection(payload: dict[str, Any]) -> str | None:
The breaker only counts rejections whose kind is in The breaker only counts rejections whose kind is in
``_CIRCUIT_REJECTION_KINDS`` (the SDK's authoritative catalog). Three ``_CIRCUIT_REJECTION_KINDS`` (the SDK's authoritative catalog). Three
reachable rejection shapes must all map to a counted kind so a storm of reachable rejection shapes must all map to a counted kind so a storm of
any of them trips the breaker (F068): any of them trips the breaker:
1. Envelope rejection: ``error`` is a STRING kind. Forward it if in 1. Envelope rejection: ``error`` is a STRING kind. Forward it if in
the counted set (existing behaviour). Uncounted string kinds (e.g. the counted set (existing behaviour). Uncounted string kinds (e.g.
@@ -176,19 +176,15 @@ def _post(path: str, body: dict[str, Any]) -> dict[str, Any]:
headers=_build_headers(), headers=_build_headers(),
json=body, json=body,
) )
# F069: a 404 here means a manifest-registered verb has no matching # A 404 here means a manifest-registered verb has no matching route on
# route on the orchestrator (every gateway route returns 200 with an # the orchestrator: every gateway route returns 200 with an Envelope
# Envelope — including not_found rejections — so a 404 status with # (including not_found rejections), so FastAPI's default 404 body (no
# FastAPI's default body (``{"detail": "Not Found"}``, no ``error`` # ``error`` field) is always a missing route, never a legit Envelope.
# field) is always a missing route, never a legit Envelope). That # Synthesize an ``invalid_state`` Envelope so the breaker counts it
# body is a non-envelope payload the breaker can't classify, so a # (via ``_classify_rejection``) and the agent gets a remediation hint
# storm of these bypassed the circuit breaker → unbounded retries on # instead of a raw ``detail`` body. A 404 carrying a real Envelope (an
# a verb that can never succeed. Synthesize an ``invalid_state`` # ``error`` field — e.g. a proxy re-status a 200 rejection to 404) is
# Envelope rejection so the breaker counts it (via # surfaced as-is.
# ``_classify_rejection``) and the agent gets a remediation hint
# instead of a raw ``detail`` body. A 404 that DOES carry a real
# Envelope (an ``error`` field — e.g. a proxy re-status a 200
# rejection to 404) is surfaced as-is.
if response.status_code == _MISSING_ROUTE_STATUS: if response.status_code == _MISSING_ROUTE_STATUS:
try: try:
body_404 = response.json() body_404 = response.json()
@@ -270,11 +266,11 @@ def _record_and_check_circuit(
# Gateway envelopes use a string `error` (kind); RobocoError-derived # Gateway envelopes use a string `error` (kind); RobocoError-derived
# exceptions surface a dict-shaped error via FastAPI's middleware, and # exceptions surface a dict-shaped error via FastAPI's middleware, and
# 422 validation failures carry a `detail` list with no `error` field # 422 validation failures carry a `detail` list with no `error` field
# at all. F068: classify all three rejection shapes so a storm of 500s # at all. Classify all three rejection shapes so a storm of 500s or 422s
# or 422s counts toward the breaker (previously bypassed → unbounded # counts toward the breaker (previously bypassed → unbounded retries). The
# retries). The dict-shape defence against `TypeError: unhashable type: # dict-shape defence against `TypeError: unhashable type: 'dict'` lives in
# 'dict'` lives in `_classify_rejection` (isinstance checks, never a # `_classify_rejection` (isinstance checks, never a `dict in frozenset`
# `dict in frozenset` membership test). # membership test).
rejection_kind = _classify_rejection(payload) rejection_kind = _classify_rejection(payload)
if rejection_kind is None: if rejection_kind is None:
return payload return payload
+45 -62
View File
@@ -304,16 +304,13 @@ _GROK_INTERACTIVE_DOCKERFILES = {
# the retry window (unknown-provider time-expiry fallback in _probe_target). # the retry window (unknown-provider time-expiry fallback in _probe_target).
_GROK_RATE_LIMIT_EXIT_CODE = 75 _GROK_RATE_LIMIT_EXIT_CODE = 75
_GROK_RATE_LIMIT_RETRY_AFTER_S = 60.0 _GROK_RATE_LIMIT_RETRY_AFTER_S = 60.0
# F097: grok has no real recovery probe (the grok CLI's xAI endpoint is closed # Grok has no real recovery probe (the SuperGrok OIDC token is not a valid
# and the SuperGrok OIDC access token is not a valid bearer for the metered # bearer for the metered api.x.ai, so a probe would no-op or strand grok
# api.x.ai, so a probe would either no-op or strand grok parked forever). So # parked). The probe loop clears a grok park on a timer; the fresh agent hits
# the probe loop clears a grok park optimistically on a timer, a cleared park # the still-active xAI 429, exits 75, and re-parks. Back the re-park retry_after
# dispatches a fresh grok agent that immediately hits the still-active xAI 429, # off exponentially within one episode so the churn dampens (60 -> 120 -> 240
# exits 75, and re-parks — a flat ~90s crash-retry cycle for the whole xAI # -> ... capped) instead of spinning flat. The episode gap resets the count
# window. Back the re-park retry_after off exponentially within one episode so # once the rate limit has actually lifted.
# the churn dampens (60 -> 120 -> 240 -> ... capped) instead of spinning flat.
# Cap bounds the cycle; the episode gap (> the max cycle) resets the count once
# the rate limit has actually lifted (no re-park for the gap => fresh episode).
_GROK_REPARK_BACKOFF_CAP = 4 # max 2**4 = 16x base (~16min cycle) _GROK_REPARK_BACKOFF_CAP = 4 # max 2**4 = 16x base (~16min cycle)
_GROK_REPARK_EPISODE_GAP_S = 1500.0 # 25min — > the capped ~16min cycle _GROK_REPARK_EPISODE_GAP_S = 1500.0 # 25min — > the capped ~16min cycle
# A one-shot Grok container exits with this code (EX_CONFIG) when the # A one-shot Grok container exits with this code (EX_CONFIG) when the
@@ -925,14 +922,10 @@ class AgentOrchestrator:
# kill-switch parity (the grok CLI exposes no live usage hook). 0 disables. # kill-switch parity (the grok CLI exposes no live usage hook). 0 disables.
# See _enforce_grok_cost_budget. # See _enforce_grok_cost_budget.
self._grok_max_cost_usd: float = settings.grok_max_cost_usd self._grok_max_cost_usd: float = settings.grok_max_cost_usd
# F097: grok re-park backoff state. Grok has no real recovery probe, so # Grok re-park backoff state. Track the re-park count within one episode
# the probe loop clears a grok park optimistically on a timer; a cleared # so retry_after can back off exponentially (dampening the ~90s
# park respawns a grok agent that hits the still-active xAI 429 and # crash-retry churn), and the last park time so a gap (the rate limit
# re-parks. Track the re-park count within one episode so the retry_after # actually lifted) resets the count for the next episode.
# can back off exponentially (dampening the ~90s crash-retry churn), and
# the last park time so a gap (the rate limit actually lifted) resets
# the count for the next episode. Per-provider state would be cleaner,
# but grok is a single provider key, so a scalar suffices.
self._grok_last_park_at: datetime | None = None self._grok_last_park_at: datetime | None = None
self._grok_repark_count: int = 0 self._grok_repark_count: int = 0
@@ -4848,11 +4841,9 @@ class AgentOrchestrator:
error=str(exc), error=str(exc),
) )
continue continue
# F040: finalize the spawn session BEFORE popping the instance so # Finalize the spawn session BEFORE popping the instance so the
# the captured usage/cost is recorded in the DB/dashboard. # captured usage/cost is recorded; popping first would lose the
# _finalize_spawn_session reads self._instances[agent_id] for the # model + usage_session_id and leave the session row open.
# model + usage_session_id; popping first would lose them and leave
# the session row open (ended_at IS NULL) — the burn invisible.
with contextlib.suppress(Exception): with contextlib.suppress(Exception):
await self._finalize_spawn_session(agent_id, exit_reason="cost_cap") await self._finalize_spawn_session(agent_id, exit_reason="cost_cap")
self._instances.pop(agent_id, None) self._instances.pop(agent_id, None)
@@ -6834,11 +6825,10 @@ Start by:
error=str(e), error=str(e),
) )
# F045 orphan fallback: resume agents parked for a provider the # Orphan fallback: resume agents parked for a provider the
# tracker-listed loop above did not cover (activate failed silently or # tracker-listed loop above did not cover (activate failed silently or
# Redis was down at park time). Empty state => probe immediately; on # Redis was down at park time). On probe success ``_on_probe_success``
# success ``_on_probe_success`` clears the tracker (self-healing) and # clears the tracker (self-healing) and resumes the parked agents.
# resumes the parked agents.
orphan_providers: set[str] = set() orphan_providers: set[str] = set()
for record in self._waiting_records.values(): for record in self._waiting_records.values():
if record.waiting_for != "rate_limit_lifted": if record.waiting_for != "rate_limit_lifted":
@@ -6984,11 +6974,10 @@ Start by:
if provider_type not in (None, ModelProvider.ANTHROPIC.value): if provider_type not in (None, ModelProvider.ANTHROPIC.value):
return None return None
tail = await self._tail_container_logs(f"roboco-agent-{agent_id}") tail = await self._tail_container_logs(f"roboco-agent-{agent_id}")
# F036: the SDK server writes model-API errors to /tmp/sdk-server.log, # The SDK server writes model-API errors to /tmp/sdk-server.log, not
# not stdout, so the overload marker (529/500/503) may appear only in # stdout, so the overload marker may appear only in the durable Claude
# the durable Claude transcript — the same rationale already applied to # transcript; without it an overload is missed and the agent
# the session-limit detector. Without the transcript an overload is # crash-respawns straight back into it.
# missed and the agent crash-respawns straight back into it.
transcript_tail = self._transcript_tail_text(agent_id) transcript_tail = self._transcript_tail_text(agent_id)
lowered = (tail + "\n" + transcript_tail).lower() lowered = (tail + "\n" + transcript_tail).lower()
if any(marker in lowered for marker in _ANTHROPIC_OVERLOAD_MARKERS): if any(marker in lowered for marker in _ANTHROPIC_OVERLOAD_MARKERS):
@@ -7058,16 +7047,13 @@ Start by:
kind=kind, kind=kind,
error=str(exc), error=str(exc),
) )
# F035: register a WaitingRecord so the probe-resume loop can revive # Register a WaitingRecord so the probe-resume loop can revive this
# this agent when the provider recovers. ``_on_probe_success`` reads # agent when the provider recovers; without it recovery falls to the
# ``_waiting_records`` filtered by ``waiting_for == "rate_limit_lifted"`` # 600s stale-claim reaper instead of the probe-success path the parking
# + ``context.provider``; without a record here it resumes nobody and # design relies on. Persisted so a restart still resolves the wait.
# recovery falls to the 600s stale-claim reaper instead of the # We do NOT call ``mark_waiting_long`` — the container is already dead,
# probe-success path the parking design relies on. Persisted (mirrors # and parking keeps OFFLINE so the reaper's live-skip / health loop
# ``mark_waiting_long``) so a restart still resolves the wait. We do NOT # ignore it.
# call ``mark_waiting_long`` itself — the container is already dead (it
# exited), so there is nothing to stop, and parking keeps OFFLINE (not
# WAITING_LONG) so the reaper's live-skip / health loop ignore it.
task_id = str(instance.current_task_id) if instance.current_task_id else None task_id = str(instance.current_task_id) if instance.current_task_id else None
record = WaitingRecord( record = WaitingRecord(
agent_id=agent_id, agent_id=agent_id,
@@ -8429,13 +8415,11 @@ Start now: evidence(task_id="{task_id}")
continue continue
if not is_running: if not is_running:
continue continue
# F033: capture the real container id. _check_health skips # Capture the real container id: ``_check_health`` skips instances
# ``container_id is None`` instances, so a re-adopted instance # with ``container_id is None``, so a re-adopted instance without
# without the id would be invisible to the health loop — when the # the id would be invisible to the health loop and strand the task
# container later exits the stopped-container handler never runs # under a phantom ACTIVE instance. Best-effort — a probe failure
# and the task strands under a phantom ACTIVE instance. Best-effort: # degrades to None (reaper's Docker-liveness fallback covers it).
# a probe failure degrades to the prior None (still re-adopted as
# ACTIVE; the reaper's Docker-liveness fallback covers it).
container_id: str | None = None container_id: str | None = None
try: try:
container_id = await self._resolve_container_id(f"roboco-agent-{slug}") container_id = await self._resolve_container_id(f"roboco-agent-{slug}")
@@ -8643,11 +8627,11 @@ Start now: evidence(task_id="{task_id}")
and not await self._maybe_recover_broken_gateway(t) and not await self._maybe_recover_broken_gateway(t)
): ):
continue continue
# F035: a provider-parked agent (session-limit / overload / # A provider-parked agent (session-limit / overload / grok-429)
# grok-429) is OFFLINE with a dead container and a # is OFFLINE with a dead container and a ``rate_limit_lifted``
# ``rate_limit_lifted`` WaitingRecord. The probe-resume loop # WaitingRecord. The probe-resume loop owns its recovery — do
# owns its recovery — do NOT reap the claim, or probe-success # NOT reap the claim, or probe-success would respawn the agent
# would later respawn the agent on a task it no longer owns. # on a task it no longer owns.
if self._assignee_is_provider_parked(t): if self._assignee_is_provider_parked(t):
continue continue
task_id = require_uuid(t.id) task_id = require_uuid(t.id)
@@ -9225,12 +9209,11 @@ Start now: evidence(task_id="{task_id}")
# are acted on by the release routes + executor, never dispatched. # are acted on by the release routes + executor, never dispatched.
if task.get("source") == RELEASE_MANAGER_SOURCE: if task.get("source") == RELEASE_MANAGER_SOURCE:
continue continue
# F059: a self-heal fix task is HELD for the CEO's Approve-&-Start # A self-heal fix task is HELD for the CEO's Approve-&-Start
# (confirmed_by_human=False at origination). It must NOT dispatch # (confirmed_by_human=False at origination). It must NOT dispatch
# autonomously — the loop only OPENS it; the CEO's approve_and_start # autonomously — the CEO's approve_and_start flips
# flips confirmed_by_human True, after which it flows through the # confirmed_by_human True, after which it flows through the
# assigned-PM path below like any other PM task. (The fix still ships # assigned-PM path below like any other PM task.
# through dev -> QA -> PR review -> the CEO's merge.)
if task.get("source") == SELF_HEAL_SOURCE and not task.get( if task.get("source") == SELF_HEAL_SOURCE and not task.get(
"confirmed_by_human" "confirmed_by_human"
): ):
@@ -9616,9 +9599,9 @@ Never `commit`, never write code, never run `git`. PMs coordinate.
# Release proposals are CEO-gated artifacts, never dev work. # Release proposals are CEO-gated artifacts, never dev work.
if task.get("source") == RELEASE_MANAGER_SOURCE: if task.get("source") == RELEASE_MANAGER_SOURCE:
continue continue
# F059: a self-heal fix task held for the CEO's Approve-&-Start is # A self-heal fix task held for the CEO's Approve-&-Start is not dev
# not dev work yet — it must not route to its assigned_to as a dev # work yet — it must not route to its assigned_to as a dev before
# before the CEO approves it. # the CEO approves it.
if task.get("source") == SELF_HEAL_SOURCE and not task.get( if task.get("source") == SELF_HEAL_SOURCE and not task.get(
"confirmed_by_human" "confirmed_by_human"
): ):
+38 -72
View File
@@ -1919,7 +1919,7 @@ class Choreographer:
``reviewer=True`` for the pr_pass gate: a PR reviewer has no ``reviewer=True`` for the pr_pass gate: a PR reviewer has no
``i_am_blocked`` verb, so the remediation points at ``pr_fail`` (their ``i_am_blocked`` verb, so the remediation points at ``pr_fail`` (their
reject lever, sending the PR back to needs_revision for the dev to fix reject lever, sending the PR back to needs_revision for the dev to fix
the environment) instead of a verb they cannot call (F044). the environment) instead of a verb they cannot call.
""" """
from roboco.config import settings as _settings from roboco.config import settings as _settings
@@ -2019,7 +2019,7 @@ class Choreographer:
``file:line`` + fix hint. ``warn`` findings never block. Inert when the ``file:line`` + fix hint. ``warn`` findings never block. Inert when the
flag is off. This is the pr_pass (reviewer) path the remediation is flag is off. This is the pr_pass (reviewer) path the remediation is
reviewer-aware (``pr_fail``, not ``i_am_blocked`` which a reviewer reviewer-aware (``pr_fail``, not ``i_am_blocked`` which a reviewer
lacks) via ``_conventions_rejection(..., reviewer=True)`` (F044). lacks) via ``_conventions_rejection(..., reviewer=True)``.
""" """
from roboco.config import settings as _settings from roboco.config import settings as _settings
@@ -2036,7 +2036,7 @@ class Choreographer:
``reviewer=True`` for the pr_pass gate: a reviewer has no ``reviewer=True`` for the pr_pass gate: a reviewer has no
``i_am_blocked`` verb, so the could_not_run remediation points at ``i_am_blocked`` verb, so the could_not_run remediation points at
``pr_fail`` instead (F044). ``pr_fail`` instead.
""" """
if result.get("could_not_run"): if result.get("could_not_run"):
if reviewer: if reviewer:
@@ -2067,15 +2067,10 @@ class Choreographer:
f"- {f.get('file')}:{f.get('line')}{f.get('fix_hint')}" for f in blocks f"- {f.get('file')}:{f.get('line')}{f.get('fix_hint')}" for f in blocks
) )
if reviewer: if reviewer:
# F047: the pr_pass gate runs this on the REVIEWER, who does not own # The pr_pass gate runs on the REVIEWER, who has no commit verb on
# the assembled cell→root / root→master branch and has no commit # the assembled branch; the only lever is pr_fail — bounce the PR
# verb on it. The dev-path remediation ("add a waiver in your # back to needs_revision with the findings as issues so the dev
# branch") is unreachable by the reviewer and would strand the gate # fixes the violation or commits a waiver.
# on every false positive with no self-recovery. The reviewer's only
# lever is pr_fail — bounce the PR back to needs_revision carrying
# the findings as issues so the dev fixes the violation or commits
# the waiver (the dev CAN commit to the branch). Waiver authorship is
# framed as the dev's action, not the reviewer's.
remediate = ( remediate = (
"the assembled PR carries block-level architectural-convention" "the assembled PR carries block-level architectural-convention"
" violations. call pr_fail(issues=[<file:line — fix_hint>, ...])" " violations. call pr_fail(issues=[<file:line — fix_hint>, ...])"
@@ -2913,14 +2908,9 @@ class Choreographer:
task_id=task_id, task_id=task_id,
verb="i_am_blocked", verb="i_am_blocked",
) )
# F017: ``block`` is the LAST composed action, so a ``None`` return # ``block`` is the LAST composed action, so a ``None`` return (escalate
# (TaskService.escalate resolved no escalation target — missing task, # resolved no target) re-binds ``t`` to ``None``; guard the deref with
# agent, escalation-target slug, or target agent row) flows out of # an invalid_state so the agent gets a retryable rejection, not a 500.
# ``run_intent`` as the verb's result. Without this guard the caller
# re-binds ``t`` to ``None`` and dereferences ``t.status`` building the
# success envelope → AttributeError → HTTP 500, and the agent
# respawn-loops with no actionable rejection. Surface invalid_state
# instead, pointing the agent at a direct CEO escalation or a retry.
if updated is None: if updated is None:
return t, await self._emit_rejection( return t, await self._emit_rejection(
Envelope.invalid_state( Envelope.invalid_state(
@@ -3014,16 +3004,9 @@ class Choreographer:
# provider is "unknown" (orchestrator not wired or not tracking the # provider is "unknown" (orchestrator not wired or not tracking the
# agent) to avoid polluting the tracker with meaningless keys. # agent) to avoid polluting the tracker with meaningless keys.
# #
# F045: an activate() failure is logged loudly, NOT bare-suppressed. # An activate() failure is logged loudly, NOT bare-suppressed: the
# The probe-resume loop is tracker-driven — it iterates # probe-resume loop is tracker-driven, so a silent failure strands
# ``list_rate_limited_providers()`` — so a silent activate failure here # every parked agent waiting on a provider the tracker never learned.
# leaves every parked agent in ``_waiting_records`` with a provider the
# tracker never learned about, and no probe ever runs to resume them
# (the stranded-fleet blind spot). The orchestrator's
# ``_sweep_rate_limit_probes`` has an in-memory ``_waiting_records``
# fallback that resumes them when the provider recovers even without
# tracker state; this error log makes the condition visible to
# operators either way.
if provider != "unknown": if provider != "unknown":
try: try:
from roboco.services.gateway.rate_limit_tracker import ( from roboco.services.gateway.rate_limit_tracker import (
@@ -3968,10 +3951,9 @@ class Choreographer:
reality. Checkpoint failure is swallowed; it must never block the pause. reality. Checkpoint failure is swallowed; it must never block the pause.
""" """
in_progress = await self.task.list_in_progress_for_agent(agent_id) in_progress = await self.task.list_in_progress_for_agent(agent_id)
# F018: the lookup now also returns blocked tasks (so the claim guard # The lookup also returns blocked tasks (so the claim guard sees them);
# sees them). i_am_idle only auto-pauses genuinely in_progress tasks — # i_am_idle only auto-pauses genuinely in_progress ones — a blocked task
# a blocked task is waiting on an external dep, not on the agent, so it # waits on an external dep, not the agent, so it stays blocked.
# stays blocked (and isn't reported as paused for the agent to resume).
from roboco.models.base import TaskStatus from roboco.models.base import TaskStatus
paused_ids: list[str] = [] paused_ids: list[str] = []
@@ -5381,10 +5363,9 @@ class Choreographer:
# + subtasks-terminal + branch-present. None of these are modelled by # + subtasks-terminal + branch-present. None of these are modelled by
# the spec yet — keep them in the verb body. # the spec yet — keep them in the verb body.
guard = await self._submit_up_guard(pm_agent_id, task_id, t, notes) guard = await self._submit_up_guard(pm_agent_id, task_id, t, notes)
# F007: the cell-level unchanged-PR loop-stopper. Only consult it once # Cell-level unchanged-PR loop-stopper. Consulted only after the state
# the state guard above has passed (ownership/tracing/branch all OK) — # guard above passes; a prior preflight reject short-circuits before the
# mirroring submit_root, a prior preflight reject short-circuits before # head-sha comparison runs (mirroring submit_root).
# the head-sha comparison runs.
if guard is None: if guard is None:
guard = await self._submit_up_unchanged_pr_guard(t, briefing) guard = await self._submit_up_unchanged_pr_guard(t, briefing)
if guard is not None: if guard is not None:
@@ -6173,7 +6154,7 @@ class Choreographer:
async def _current_pr_head_sha(self, t: Any) -> str | None: async def _current_pr_head_sha(self, t: Any) -> str | None:
"""Best-effort current head SHA of the task's assembled PR (fail-open). """Best-effort current head SHA of the task's assembled PR (fail-open).
The lookup both unchanged-PR gates (submit_root F016 + submit_up F007) The lookup both unchanged-PR gates (submit_root + submit_up)
compare against ``pr_fail`` stamps the head SHA for cell AND root compare against ``pr_fail`` stamps the head SHA for cell AND root
gate tasks alike (the capture is gate-verb-level, not root-level), so gate tasks alike (the capture is gate-verb-level, not root-level), so
one resolver serves both. Returns ``None`` on every ambiguous case (no one resolver serves both. Returns ``None`` on every ambiguous case (no
@@ -6201,18 +6182,12 @@ class Choreographer:
async def _submit_up_unchanged_pr_guard( async def _submit_up_unchanged_pr_guard(
self, t: Any, briefing: dict[str, Any] self, t: Any, briefing: dict[str, Any]
) -> Envelope | None: ) -> Envelope | None:
"""F007 — the cell-level analogue of ``_submit_root_unchanged_pr_guard``. """Cell-level analogue of ``_submit_root_unchanged_pr_guard``.
The root loop-stopper was root-only; a weak cell PM could re-submit the Refuses re-submit when the cell PR's current head SHA equals the SHA
unchanged cellroot PR after a ``pr_fail`` and loop ``pr_fail`` recorded in ``notes_structured.pr_review.head_sha`` (no new
``awaiting_pr_review`` ``pr_fail`` forever. ``pr_fail`` stamps the dev work landed byte-identical diff). Ambiguous cases FAIL OPEN via
assembled PR's head SHA into ``notes_structured.pr_review.head_sha`` ``_current_pr_head_sha``; only the exact-unchanged case is hard-blocked.
for cell gate tasks too (the capture is gate-verb-level), so the same
structural refusal applies: if the cell PR's current head SHA equals
the SHA the last ``pr_fail`` recorded, no new dev work landed on the
cell branch the diff is byte-identical refuse. Every ambiguous case
FAILS OPEN (shared ``_current_pr_head_sha``) only the exact-unchanged
case is hard-blocked; the rest fall through to the cell reviewer.
""" """
pr_review = (getattr(t, "notes_structured", None) or {}).get("pr_review") or {} pr_review = (getattr(t, "notes_structured", None) or {}).get("pr_review") or {}
if pr_review.get("verdict") != "failed": if pr_review.get("verdict") != "failed":
@@ -6334,16 +6309,10 @@ class Choreographer:
task_id=task_id, task_id=task_id,
verb="submit_root", verb="submit_root",
) )
# F016: submit_for_review returns None when the root->master PR was # submit_for_review returns None when the root->master PR was already
# already opened (the task raced out of in_progress, or a prior call # opened (race out of in_progress / prior call). The PR exists but the
# already transitioned it to awaiting_pr_review). The create_root_pr # transition did not — guard the None deref with an invalid_state so
# pre-side-effect already ran, so the PR exists, but the transition # the PM re-fetches and reconciles instead of crashing.
# did not happen — dereferencing t.status here 500'd. Surface an
# actionable invalid_state so the PM re-fetches and reconciles (if the
# task is already awaiting_pr_review the PR is open — wait for the
# reviewer; otherwise re-delegate the fixes and retry) instead of a
# crash. The None-guard + success envelope share a finalize helper so
# submit_root's own return count stays under the branch-limit.
return await self._submit_root_finalize( return await self._submit_root_finalize(
main_pm_agent_id, task_id, t, role_str, briefing main_pm_agent_id, task_id, t, role_str, briefing
) )
@@ -6358,7 +6327,7 @@ class Choreographer:
) -> Envelope: ) -> Envelope:
"""Build the submit_root result envelope after the verb runner returns. """Build the submit_root result envelope after the verb runner returns.
``None`` (F016) invalid_state rejection (the root->master PR was ``None`` invalid_state rejection (the root->master PR was
already opened / the task raced out of in_progress); otherwise the already opened / the task raced out of in_progress); otherwise the
success envelope keyed off the post-transition status. success envelope keyed off the post-transition status.
""" """
@@ -6627,17 +6596,14 @@ class Choreographer:
verb="complete", verb="complete",
): ):
return soup return soup
# F001: a MegaTask umbrella is branchless by design and never goes # A MegaTask umbrella is branchless by design and never goes through
# through submit_root / pr_pass, so it sits in in_progress with no # submit_root / pr_pass, so it sits in in_progress with no branch/PR.
# branch/PR. The ``complete`` action's source_statuses= # The ``complete`` action's source_statuses={AWAITING_PM_REVIEW} spec
# {AWAITING_PM_REVIEW} spec gate would reject it before # gate would reject it before main_pm_complete's branchless-aware guard
# main_pm_complete's branchless-aware guard can run. Skip the spec # can run; skip the spec gate for an in_progress batch umbrella and fall
# gate for an in_progress batch umbrella and fall through to # through to main_pm_complete (CEO merges the root PR; no agent touches
# main_pm_complete, which walks in_progress -> awaiting_pm_review -> # master). Role membership is preserved; main_pm_complete re-checks
# awaiting_ceo_approval (the CEO merges the root PR; no agent touches # assignment, subtasks-terminal, and the journal:decision gate.
# master). Role membership is preserved (role_str == "main_pm");
# main_pm_complete's own guard re-checks assignment, subtasks-
# terminal, and the journal:decision gate.
umbrella_in_progress = ( umbrella_in_progress = (
role_str == "main_pm" role_str == "main_pm"
and str(t.status) == "in_progress" and str(t.status) == "in_progress"
@@ -282,16 +282,10 @@ class PRGateMixin(_Base):
task_id=task_id, task_id=task_id,
verb=verb, verb=verb,
) )
# F046: a concurrent transition (cancel, or a racing reviewer) between # A concurrent transition (cancel, racing reviewer) between the
# the precondition gate and the runner's final composed action makes # precondition gate and the runner's final action makes run_intent
# the source-status check fail mid-flight and run_intent returns None # return None; guard the dereferences below with a clean rejection so
# (the verb runner's documented contract for a last-action source-status # the reviewer re-fetches and re-issues.
# failure). Without this guard the dereferences below (t.assigned_to,
# t.status, _post_gate_review_to_pr(t, ...)) crash the gate with a 500
# AttributeError. Surface a clean invalid_state rejection so the
# reviewer re-fetches with evidence(task_id) and re-issues — the
# already-authored verdict note is harmless (the task is no longer in
# the gate state) and no PR post / a2a runs against a None task.
if t is None: if t is None:
return await self._emit_rejection( return await self._emit_rejection(
Envelope.invalid_state( Envelope.invalid_state(
+3 -7
View File
@@ -24,13 +24,9 @@ from roboco.services.gateway.envelope import Envelope
if TYPE_CHECKING: if TYPE_CHECKING:
from uuid import UUID from uuid import UUID
# Statuses that count as "still actively worked" — pre-gateway # Statuses that count as "still actively worked".
# _helpers.py:check_blocking_tasks 134-152. # ``blocked`` is included: it is still owned by the dev and resumes to
# # ``in_progress``; excluding it let a dev hold two in_progress tasks at once.
# F018: ``blocked`` is included — a blocked task is still owned by the dev and
# ``unblock_with_restore`` resumes it to ``in_progress``. Excluding it let a dev
# claim a second task while blocked, then end up with TWO ``in_progress``
# tasks once the first was unblocked, violating the one-active-task invariant.
_ACTIVE_BLOCKING_STATUSES: frozenset[str] = frozenset( _ACTIVE_BLOCKING_STATUSES: frozenset[str] = frozenset(
{"claimed", "in_progress", "verifying", "blocked"} {"claimed", "in_progress", "verifying", "blocked"}
) )
+3 -4
View File
@@ -1208,10 +1208,9 @@ class ContentActions:
# A dependency block is a "wait silently" situation — never a CEO signal. # A dependency block is a "wait silently" situation — never a CEO signal.
# An agent must not page the CEO to relax or escalate a task that is # An agent must not page the CEO to relax or escalate a task that is
# simply waiting on an unfinished upstream; that wait clears on its own. # simply waiting on an unfinished upstream; that wait clears on its own.
# F048: also reject human-only recipients (prompter/secretary) — they # Also reject human-only recipients (prompter/secretary) — they have no
# have no agent ack path, so an ack-required signal would sit permanently # agent ack path, so an ack-required signal would sit permanently unacked
# unacked and suppress later same-purpose notifications via the dedup # and suppress later same-purpose notifications via the dedup query.
# query. The CEO acks via the panel and stays an allowed recipient.
if reject := await self._reject_disallowed_recipient(target, task_id): if reject := await self._reject_disallowed_recipient(target, task_id):
return reject return reject
await self.notifications.send_ack_notification( await self.notifications.send_ack_notification(
+4 -8
View File
@@ -106,14 +106,10 @@ def build_task_handoff(
# Upstream dependencies that completed and were cleared — present only on a # Upstream dependencies that completed and were cleared — present only on a
# just-unblocked task, so the revived dependent knows what it can build on. # just-unblocked task, so the revived dependent knows what it can build on.
completed_deps = _typed(getattr(task, "completed_dependency_ids", None), list, []) completed_deps = _typed(getattr(task, "completed_dependency_ids", None), list, [])
# F008 — the persisted in-path PR-review gate verdict + concrete issues. # The persisted in-path PR-review gate verdict + concrete issues.
# ``pr_fail`` authors ``notes_structured.pr_review`` (verdict / summary / # ``pr_fail`` writes ``notes_structured.pr_review``; surfacing it here puts
# issues / head_sha) on every fail, but the a2a steer to the owning PM is # the concrete issues in every PM briefing so a respawned PM doesn't
# fire-and-forget — a PM respawned into ``needs_revision`` later read none # re-submit the same PR blind.
# of it (build_task_handoff never looked at notes_structured), saw a generic
# "needs revision" with zero change-requests, and re-submitted the same PR
# (the 2026-06-27 infinite pr_fail loop on 9980d0a0 / PR #138). Surfacing it
# here puts the concrete issues in every PM briefing for the task.
pr_review = _extract_pr_review(getattr(task, "notes_structured", None)) pr_review = _extract_pr_review(getattr(task, "notes_structured", None))
has_prior = bool( has_prior = bool(
commits commits
+2 -3
View File
@@ -277,9 +277,8 @@ class GitService(BaseService):
try: try:
result = await loop.run_in_executor(_GIT_EXECUTOR, _run) result = await loop.run_in_executor(_GIT_EXECUTOR, _run)
except subprocess.TimeoutExpired as e: except subprocess.TimeoutExpired as e:
# F019: the timed-out git process was SIGKILL'd mid-mutation and may # Timed-out git process was SIGKILL'd mid-mutation and may orphan
# have orphaned .git/*.lock files; clear them so the workspace isn't # .git/*.lock files; clear them so the workspace isn't wedged.
# wedged for the next op (incl. the next fresh-claim reset --hard).
await loop.run_in_executor( await loop.run_in_executor(
_GIT_EXECUTOR, _remove_stale_git_locks, workspace _GIT_EXECUTOR, _remove_stale_git_locks, workspace
) )
+3 -4
View File
@@ -1495,10 +1495,9 @@ class MessagingService(BaseService):
subject=f"You were mentioned in #{channel_slug}", subject=f"You were mentioned in #{channel_slug}",
body=message.content[:500], # Truncate for notification body=message.content[:500], # Truncate for notification
related_task_id=message.task_id, related_task_id=message.task_id,
# F009: MENTION is informational (ACK_REQUIRED_BY_TYPE -> False). # MENTION is informational (ACK_REQUIRED_BY_TYPE -> False); the
# The column default True made every @mention require an ack, # column default True would inflate unacked sets and soft-block
# inflating the recipient's unacked set and soft-blocking # i_am_idle.
# i_am_idle into respawn churn.
requires_ack=False, requires_ack=False,
) )
self.session.add(notification) self.session.add(notification)
+6 -17
View File
@@ -493,15 +493,9 @@ class NotificationService:
# already acked all go through. Body text is NOT compared, so # already acked all go through. Body text is NOT compared, so
# rewording cannot defeat the guard. # rewording cannot defeat the guard.
# #
# F010: the dedup only applies to ACTION-REQUIRED types # Dedup only applies to ACTION-REQUIRED types; informational types
# (ACK_REQUIRED_BY_TYPE -> True). Informational types # carry distinct content per send and acking is voluntary, so
# (KNOWLEDGE_SHARE / MENTION / A2A_REQUEST / BROADCAST / the # deduping them would silently drop broadcasts.
# pickup-proves-receipt triad) carry distinct content per send — a
# new learning, a new mention — and acking them is voluntary, so a
# recipient who never acks would let the dedup permanently suppress
# every subsequent same-sender broadcast (silent learning-broadcast
# data loss). The anti-loop rationale only holds for ack-required
# signals; informational ones are not deduped.
related = params.related_task_id related = params.related_task_id
is_ack_required = ACK_REQUIRED_BY_TYPE.get(params.notification_type, True) is_ack_required = ACK_REQUIRED_BY_TYPE.get(params.notification_type, True)
if is_ack_required: if is_ack_required:
@@ -535,14 +529,9 @@ class NotificationService:
subject=params.subject, subject=params.subject,
body=params.body, body=params.body,
related_task_id=params.related_task_id, related_task_id=params.related_task_id,
# F009: requires_ack follows ACK_REQUIRED_BY_TYPE (the spec's # requires_ack follows ACK_REQUIRED_BY_TYPE (action-required vs
# action-required vs informational split), not the column's True # informational), not the column's True default; default True
# default. Without this every notification — including # for an unmapped type preserves the safe action-required bias.
# informational REVIEW_REQUEST / DOCUMENTATION_REQUEST /
# A2A_REQUEST / MENTION / KNOWLEDGE_SHARE — became requires_ack,
# inflating recipients' unacked sets and soft-blocking
# i_am_idle into respawn churn. Default to True for an unmapped
# type (preserve the safe action-required bias).
requires_ack=ACK_REQUIRED_BY_TYPE.get(params.notification_type, True), requires_ack=ACK_REQUIRED_BY_TYPE.get(params.notification_type, True),
) )
db.add(notification) db.add(notification)
+2 -3
View File
@@ -255,9 +255,8 @@ class _GitReleaseOps:
"commit", "-S", "-m", f"chore(release): {version}" "commit", "-S", "-m", f"chore(release): {version}"
) )
if commit_rc != 0: if commit_rc != 0:
# F012: a failed commit (gpgsign/pre-commit reject/no-op bump) must # A failed commit (gpgsign/pre-commit reject/no-op bump) must abort
# abort BEFORE rev-parse+push — otherwise the pre-bump base gets # before push — otherwise the pre-bump base gets tagged as the release.
# pushed and tagged as the new version.
logger.error("release commit failed", error=commit_out.strip()[:300]) logger.error("release commit failed", error=commit_out.strip()[:300])
raise RuntimeError(f"release commit failed: {commit_out.strip()[:200]}") raise RuntimeError(f"release commit failed: {commit_out.strip()[:200]}")
_, out = await self._git("rev-parse", "HEAD") _, out = await self._git("rev-parse", "HEAD")
+2 -4
View File
@@ -33,10 +33,8 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
# F013: a Redis mutex guarding the ~40min release execute against concurrent # Redis mutex guarding the ~40min release execute against concurrent
# approves (CEO double-click / panel retry). TTL is a backstop above the CI # approves; TTL backstops a crash, lock is released on completion.
# poll ceiling (ReleaseExecutor ~40min) so a crashed process can't hold the
# release hostage forever; the lock is released explicitly on completion.
_RELEASE_LOCK_PREFIX = "roboco:release_proposal:" _RELEASE_LOCK_PREFIX = "roboco:release_proposal:"
_RELEASE_LOCK_TTL_SECONDS = 3000 # 50 min > 40 min CI ceiling _RELEASE_LOCK_TTL_SECONDS = 3000 # 50 min > 40 min CI ceiling
+3 -6
View File
@@ -438,12 +438,9 @@ def _canonical_bump_files(root: Path, version: str) -> list[str]:
return sorted( return sorted(
line.strip() for line in files_raw.splitlines() if line.strip() line.strip() for line in files_raw.splitlines() if line.strip()
) )
# F058: the FIRST release has no prior ``chore(release):`` commit, so the # First release has no prior ``chore(release):`` commit, so derivation
# historical derivation returns ``[]`` and the executor would publish a tag # returns [] — fall back to the version-reference scan: files embedding the
# with no files bumped (a no-op release masquerading as X.Y.Z). Fall back to # version are exactly the set a first release must bump. Read-only.
# the version-reference scan — the files currently embedding the version are
# exactly the set a first release must bump, and the set the first release
# commit then records as canonical for every subsequent release. Read-only.
return _tracked_files_with_version(root, version) return _tracked_files_with_version(root, version)
+18 -26
View File
@@ -2316,19 +2316,13 @@ class TaskService(BaseService):
task.last_heartbeat_at = original_heartbeat task.last_heartbeat_at = original_heartbeat
task.active_claimant_id = original_claimant_id task.active_claimant_id = original_claimant_id
await self.session.flush() await self.session.flush()
# F060: emit the reversal audit row so the journey doesn't # emit the reversal audit row so the journey doesn't diverge
# diverge from real state. The forward ``task.claimed`` row # from real state. The forward ``task.claimed`` audit row was
# (emitted above via ``_validate_and_set_status``) was already # already committed on the audit service's own connection; this
# committed by the audit service on its OWN connection — this
# rollback's flush reverts the task row but NOT that audit row. # rollback's flush reverts the task row but NOT that audit row.
# Without a matching reversal row the journey's last event stays # Without a matching reversal row, downstream metrics
# ``task.claimed`` while the task is back to its pre-claim # (cycle time, bottlenecks) reconstructed from ``task.<status>``
# status, corrupting every downstream metric reconstructed from # events would be corrupted.
# ``task.<status>`` events (cycle time, bottlenecks). Emitted
# only when the forward transition was made (the original status
# was claimable) and attributed to the claimant via the explicit
# ``audit_agent_id`` (``claimed_by`` was just rolled back to
# ``None``).
if original_status in self._CLAIMABLE_STATUSES: if original_status in self._CLAIMABLE_STATUSES:
self._emit_status_transition_audit( self._emit_status_transition_audit(
task, task,
@@ -5248,12 +5242,12 @@ class TaskService(BaseService):
already = task.assigned_to == main_pm.id already = task.assigned_to == main_pm.id
task.assigned_to = cast("Any", main_pm.id) task.assigned_to = cast("Any", main_pm.id)
# F059: this is the CEO's start gate — approving the task confirms it for # this is the CEO's start gate — approving the task confirms it for
# dispatch. A self-heal fix task is opened held (confirmed_by_human=False) # dispatch. A self-heal fix task is opened held
# so the orchestrator + give_me_work keep it out of dispatch until now; # (confirmed_by_human=False) so dispatch skips it until now; flipping
# flipping it True lifts that hold. Idempotent for board/intake tasks, # it True lifts that hold. Idempotent for board/intake tasks (already
# which are already confirmed at creation. (The release-manager proposal # confirmed at creation). The release-manager proposal is not routed
# is not routed through approve_and_start — it has its own CEO routes.) # here — it has its own CEO routes.
task.confirmed_by_human = cast("Any", True) task.confirmed_by_human = cast("Any", True)
# The board-reviewed coordination task now belongs to Main PM, who will # The board-reviewed coordination task now belongs to Main PM, who will
# delegate it to the cells. Leaving team="board" is misleading once it's # delegate it to the cells. Leaving team="board" is misleading once it's
@@ -5317,14 +5311,12 @@ class TaskService(BaseService):
if child.batch_id is not None and child.status == TaskStatus.BACKLOG: if child.batch_id is not None and child.status == TaskStatus.BACKLOG:
child.status = TaskStatus.PENDING child.status = TaskStatus.PENDING
child.team = cast("Any", Team.MAIN_PM) child.team = cast("Any", Team.MAIN_PM)
# F002: a board-routed root-subtask is created in BACKLOG with # a board-routed root-subtask is created in BACKLOG with
# team=board and task_type=code (intake only coerces main_pm-team # team=board and task_type=code. Now that team is flipped to
# drafts, so a board-routed code root reaches activation still # MAIN_PM, leaving task_type=code would re-introduce the
# code-typed). Now that team is flipped to MAIN_PM, leaving # main_pm+code meltdown — retype code->planning so the activated
# task_type=code would re-introduce the 2026-06-27 main_pm+code # child is a planning-typed coordination root the Main PM
# meltdown. Retype code->planning, mirroring approve_and_start's # delegates to the cells.
# own retype above — the activated child is a planning-typed
# coordination root the Main PM delegates to the cells.
if main_pm_cannot_own_code(team=child.team, task_type=child.task_type): if main_pm_cannot_own_code(team=child.team, task_type=child.task_type):
self.log.info( self.log.info(
"activate_batch_root_subtasks retyped main-pm code " "activate_batch_root_subtasks retyped main-pm code "
+5 -8
View File
@@ -1049,14 +1049,11 @@ class WorkspaceService:
workspace=str(workspace), workspace=str(workspace),
) )
except subprocess.CalledProcessError as e: except subprocess.CalledProcessError as e:
# F063: a failure anywhere in clone/configure/leakcheck/own leaves # a failure anywhere in clone/configure/leakcheck/own leaves a
# a half-configured workspace on disk. If _configure_git raised # half-configured workspace whose .git/config may still carry the
# before its `remote set-url` scrub, .git/config still carries the # tokenized auth URL (the project PAT). Destroy it so the next
# tokenized auth URL (the project PAT); _assert_no_pat_leak never # ensure_workspace re-clones from scratch — fail-closed against
# ran, and the next ensure_workspace's health short-circuit would # PAT exfiltration.
# skip straight past the leak — mounting the agent on a workspace
# whose .git/config lets it read+exfiltrate the PAT. Destroy the
# workspace so the next ensure_workspace re-clones from scratch.
shutil.rmtree(workspace, ignore_errors=True) shutil.rmtree(workspace, ignore_errors=True)
raise WorkspaceError( raise WorkspaceError(
f"Failed to clone repository: {e.stderr or e.stdout}" f"Failed to clone repository: {e.stderr or e.stdout}"
+1 -1
View File
@@ -232,7 +232,7 @@ def test_team_for_slug() -> None:
def test_role_for_slug_or_none_unknown_returns_none() -> None: def test_role_for_slug_or_none_unknown_returns_none() -> None:
"""F019/F031: a safe variant for defensive skip-guards — an unknown/stale """A safe variant for defensive skip-guards — an unknown/stale
slug returns None instead of raising KeyError, so a stale assignee or slug returns None instead of raising KeyError, so a stale assignee or
notification-target slug can't crash the whole dispatcher tick.""" notification-target slug can't crash the whole dispatcher tick."""
assert identity.role_for_slug_or_none("nonexistent-slug") is None assert identity.role_for_slug_or_none("nonexistent-slug") is None
+7 -15
View File
@@ -567,7 +567,7 @@ def test_can_invoke_intent_developer_open_pr_no_commits_tracing_gap() -> None:
# --------------------------------------------------------------------------- # # --------------------------------------------------------------------------- #
# F101: open_pr must enforce the PR-open state gate (parity with the HTTP path) # open_pr must enforce the PR-open state gate (parity with the HTTP path)
# --------------------------------------------------------------------------- # # --------------------------------------------------------------------------- #
@@ -584,10 +584,8 @@ def _owned_task(**overrides: Any) -> SimpleNamespace:
def test_open_pr_rejected_on_claimed_task() -> None: def test_open_pr_rejected_on_claimed_task() -> None:
"""F101: ``open_pr`` has ``composes=()`` so the spec gate applied NO """``open_pr`` must be rejected from ``claimed`` — only ``in_progress`` may
source-status check a dev could open a PR from ``claimed`` (before open a PR (mirrors the HTTP path's ``_assert_pr_create_allowed``)."""
``in_progress``), skipping the active-dev state the HTTP path's
``_assert_pr_create_allowed`` enforces. The state gate now rejects it."""
actor = uuid4() actor = uuid4()
d = spec.can_invoke_intent( d = spec.can_invoke_intent(
spec.Role.DEVELOPER, spec.Role.DEVELOPER,
@@ -673,14 +671,8 @@ def test_open_pr_state_gate_takes_priority_over_unowned() -> None:
def test_escalate_up_rejected_on_completed_task() -> None: def test_escalate_up_rejected_on_completed_task() -> None:
"""F043: a PM must not resurrect a COMPLETED task via escalate_up. """A PM must not resurrect a COMPLETED task via ``escalate_up`` — the spec
gate rejects terminal tasks before the journal:decision write fires."""
escalate_up has composes=() and historically no source-status guard, so the
spec gate accepted it on a terminal task and apply_escalation set it back to
BLOCKED bypassing the state machine's terminal-state invariant. The spec
now rejects terminal tasks (completed / cancelled) before the journal:decision
write fires.
"""
d = spec.can_invoke_intent( d = spec.can_invoke_intent(
spec.Role.CELL_PM, spec.Role.CELL_PM,
"escalate_up", "escalate_up",
@@ -692,7 +684,7 @@ def test_escalate_up_rejected_on_completed_task() -> None:
def test_escalate_up_rejected_on_cancelled_task() -> None: def test_escalate_up_rejected_on_cancelled_task() -> None:
"""F043: cancelled is terminal — escalate_up must not resurrect it either.""" """Cancelled is terminal — escalate_up must not resurrect it either."""
d = spec.can_invoke_intent( d = spec.can_invoke_intent(
spec.Role.MAIN_PM, spec.Role.MAIN_PM,
"escalate_up", "escalate_up",
@@ -704,7 +696,7 @@ def test_escalate_up_rejected_on_cancelled_task() -> None:
def test_escalate_up_allowed_on_blocked_task() -> None: def test_escalate_up_allowed_on_blocked_task() -> None:
"""F043: the terminal guard must not over-restrict — BLOCKED is the natural """The terminal guard must not over-restrict — BLOCKED is the natural
escalation source and must still be allowed.""" escalation source and must still be allowed."""
d = spec.can_invoke_intent( d = spec.can_invoke_intent(
spec.Role.CELL_PM, spec.Role.CELL_PM,
@@ -102,13 +102,10 @@ async def test_explicit_dep_update_paths_scope(tmp_path: Path) -> None:
async def test_probe_holds_read_clone_lock_across_local_clone( async def test_probe_holds_read_clone_lock_across_local_clone(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None: ) -> None:
"""F116: the dep-update probe must hold the read-clone lock for the """The dep-update probe holds the read-clone lock across the local
duration of the local ``git clone --local`` from the read clone, so a ``git clone --local`` so a concurrent ``_sync_read_clone`` cannot mutate the
concurrent ``ensure_read_clone`` ``_sync_read_clone`` (fetch + hard-reset read clone mid-clone; released before the upgrade (which runs on an
to origin's default branch) cannot mutate the read clone mid-clone. The independent copy)."""
lock is released before the upgrade runs on the independent copy (the
upgrade never touches the read clone, so holding the lock past the clone
would needlessly block conventions reads for the upgrade duration)."""
read_clone = _make_read_clone(tmp_path) read_clone = _make_read_clone(tmp_path)
svc = _svc(read_clone) svc = _svc(read_clone)
# Unique slug → a fresh lock not shared with any other test. # Unique slug → a fresh lock not shared with any other test.
@@ -1,19 +1,8 @@
"""F074 — real-Postgres proof that ``TaskService.acquire_claim_lock`` serializes """Real-Postgres proof that ``TaskService.acquire_claim_lock`` serializes
concurrent claims by the SAME agent (the one-task-per-agent invariant) while NOT concurrent claims by the SAME agent (one-task-per-agent) while NOT serializing
serializing claims by DIFFERENT agents. different agents. Skips when Postgres is unreachable; each ``acquire_claim_lock``
uses a throwaway session so cancellation mid-checkout can't poison the rest of
The choreographer-level ordering + coordinator-exemption is covered by the unit the test.
suite (``test_choreographer_claim_lock.py``); this test pins the DB-level
contract the unit suite mocks out: that ``pg_advisory_xact_lock`` keyed by
``hashtextextended(agent_id)`` actually blocks a second transaction trying to
acquire the same agent's lock until the first commits/rolls back, and that a
different agent's lock is uncontended. Skips when Postgres is unreachable.
Each ``acquire_claim_lock`` call uses its own fresh session/engine. The
blocking call's session is single-use: ``asyncio.wait_for`` cancelling an
in-flight asyncpg query leaves the SQLAlchemy session mid-connection-checkout
("provisioning a new connection"), so a throwaway session per timed acquire
keeps the rest of the test on clean connections.
""" """
from __future__ import annotations from __future__ import annotations
+2 -2
View File
@@ -1198,7 +1198,7 @@ async def _seed_messages_same_timestamp(
async def test_get_messages_compound_before_cursor_no_skip_on_equal_timestamps( async def test_get_messages_compound_before_cursor_no_skip_on_equal_timestamps(
msg_setup: dict, msg_setup: dict,
) -> None: ) -> None:
"""Equal-timestamp messages must not be skipped across pages (F106). """Equal-timestamp messages must not be skipped across pages.
With a strict ``timestamp < before`` cursor and ``order_by(timestamp.desc())``, With a strict ``timestamp < before`` cursor and ``order_by(timestamp.desc())``,
messages sharing the page's last timestamp are cut by ``limit`` on page 1 messages sharing the page's last timestamp are cut by ``limit`` on page 1
@@ -1239,7 +1239,7 @@ async def test_get_messages_compound_after_cursor_no_skip_on_equal_timestamps(
) -> None: ) -> None:
"""Forward pagination (``after``) with the compound ``(timestamp, id)`` """Forward pagination (``after``) with the compound ``(timestamp, id)``
cursor tie-breaks on id so newer-direction pagination across equal cursor tie-breaks on id so newer-direction pagination across equal
timestamps skips nothing either (F106). timestamps skips nothing either.
With a strict ``timestamp > after`` cursor, every row sharing the cursor's With a strict ``timestamp > after`` cursor, every row sharing the cursor's
timestamp is EXCLUDED so forward-paginating from a middle message would timestamp is EXCLUDED so forward-paginating from a middle message would
@@ -1,17 +1,8 @@
"""F107 — Redis bus publish must be deferred until the DB commit lands. """Redis bus publish must be deferred until the DB commit lands so a rollback
drops the event (no phantom notification for a row that never became durable).
`NotificationDeliveryService.deliver` historically published Integration tests against the migrated Postgres DB: the deferral uses
``NOTIFICATION_SENT`` to the Redis event bus *before* the caller committed SQLAlchemy ``after_commit`` events and a recording bus stand-in.
the notification row. A commit failure (DB hiccup, constraint, asyncpg error)
rolled the row back but left the bus event behind connected WebSocket
clients received a push for an id that no longer existed (a phantom
notification). The fix defers the bus publish to the session's
``after_commit`` so a rollback drops it; the row is durable by the time the
event fires.
These tests need a real ``AsyncSession`` (the deferral uses SQLAlchemy
session commit/rollback events) plus a recording bus stand-in, so they are
integration tests against the migrated Postgres test DB.
""" """
from __future__ import annotations from __future__ import annotations
@@ -129,12 +120,8 @@ async def _seed_agents_and_notification(
async def test_deliver_does_not_publish_before_commit( async def test_deliver_does_not_publish_before_commit(
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
) -> None: ) -> None:
"""The bus event must NOT fire until the session commits (F107). """The bus event must NOT fire until the session commits — ``deliver``
only schedules; the event fires on commit."""
Currently RED: ``deliver`` publishes immediately, so the bus is non-empty
before any commit the phantom window. With the deferred-publish fix,
``deliver`` only schedules; the event fires on commit.
"""
bus = _RecordingBus() bus = _RecordingBus()
monkeypatch.setattr( monkeypatch.setattr(
"roboco.services.notification_delivery.get_event_bus", lambda: bus "roboco.services.notification_delivery.get_event_bus", lambda: bus
@@ -152,7 +139,7 @@ async def test_deliver_does_not_publish_before_commit(
async def test_deliver_publishes_after_commit( async def test_deliver_publishes_after_commit(
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
) -> None: ) -> None:
"""Commit drains the deferred publish — one event per recipient (F107).""" """Commit drains the deferred publish — one event per recipient."""
bus = _RecordingBus() bus = _RecordingBus()
monkeypatch.setattr( monkeypatch.setattr(
"roboco.services.notification_delivery.get_event_bus", lambda: bus "roboco.services.notification_delivery.get_event_bus", lambda: bus
@@ -179,7 +166,7 @@ async def test_deliver_rollback_drops_phantom(
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
) -> None: ) -> None:
"""A rollback instead of commit drops the pending publish — no phantom """A rollback instead of commit drops the pending publish — no phantom
event for a row that never became durable (F107).""" event for a row that never became durable."""
bus = _RecordingBus() bus = _RecordingBus()
monkeypatch.setattr( monkeypatch.setattr(
"roboco.services.notification_delivery.get_event_bus", lambda: bus "roboco.services.notification_delivery.get_event_bus", lambda: bus
+8 -10
View File
@@ -1467,10 +1467,9 @@ async def _gate_task(task_setup: dict, db_session: AsyncSession) -> Any:
async def test_pr_gate_claim_rejects_second_reviewer_race( async def test_pr_gate_claim_rejects_second_reviewer_race(
task_setup: dict, db_session: AsyncSession task_setup: dict, db_session: AsyncSession
) -> None: ) -> None:
"""F114: a second PR-reviewer race-claiming a gate task already claimed by a """A second PR-reviewer race-claiming a gate task already claimed by another
reviewer must be refused (last-write-wins would otherwise overwrite the reviewer is refused, so the first reviewer's claim and subsequent
first reviewer's claim and the first reviewer's pr_pass/pr_fail would pr_pass/pr_fail actor-checks are not overwritten."""
actor-mismatch)."""
svc = task_setup["svc"] svc = task_setup["svc"]
reviewer1 = _reviewer("R1") reviewer1 = _reviewer("R1")
reviewer2 = _reviewer("R2") reviewer2 = _reviewer("R2")
@@ -1497,10 +1496,9 @@ async def test_pr_gate_claim_rejects_second_reviewer_race(
async def test_pr_gate_claim_allows_first_reviewer_when_pm_owns_root( async def test_pr_gate_claim_allows_first_reviewer_when_pm_owns_root(
task_setup: dict, db_session: AsyncSession task_setup: dict, db_session: AsyncSession
) -> None: ) -> None:
"""F114 regression guard: the gate task is owned by the PM at entry """The first reviewer can still claim a gate task owned by the PM at entry
(submit_for_review does not clear ownership), so the FIRST reviewer must (submit_for_review does not clear ownership); the guard only rejects a
still be allowed to claim the guard only rejects a competing REVIEWER competing REVIEWER claim, not the PM owner."""
claim, not the PM owner."""
svc = task_setup["svc"] svc = task_setup["svc"]
pm = _pm("PM") pm = _pm("PM")
reviewer = _reviewer("R") reviewer = _reviewer("R")
@@ -1526,8 +1524,8 @@ async def test_pr_gate_claim_allows_first_reviewer_when_pm_owns_root(
async def test_pr_gate_claim_idempotent_for_same_reviewer( async def test_pr_gate_claim_idempotent_for_same_reviewer(
task_setup: dict, db_session: AsyncSession task_setup: dict, db_session: AsyncSession
) -> None: ) -> None:
"""F114: a reviewer re-claiming its OWN gate claim is idempotent (allowed), """A reviewer re-claiming its own gate claim is idempotent (allowed); the
not rejected the guard only refuses a DIFFERENT reviewer.""" guard only refuses a different reviewer."""
svc = task_setup["svc"] svc = task_setup["svc"]
reviewer = _reviewer("R") reviewer = _reviewer("R")
db_session.add(reviewer) db_session.add(reviewer)
@@ -453,14 +453,10 @@ async def test_create_work_session_no_project_returns_none(
async def test_create_work_session_delegates_to_service_create( async def test_create_work_session_delegates_to_service_create(
task_setup: dict, db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch task_setup: dict, db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
) -> None: ) -> None:
"""F113: the claim path must create the WorkSession through the validated """The claim path must create the WorkSession via ``WorkSessionService.create``
``WorkSessionService.create`` (the single source of truth), not construct a (single source of truth) rather than constructing a ``WorkSessionTable``
``WorkSessionTable`` directly. Two divergent creation sites had drifted and directly, so service-layer validation (existing-active check, supersede
bypassed the service-layer validation (existing-active check, supersede invariant) is not bypassed."""
invariant, project/task existence). Routing through ``create`` collapses
them to one validated path. The derived target_branch (parent branch for
subtasks, project default for roots) is passed in via ``WorkSessionCreate``.
"""
svc = task_setup["svc"] svc = task_setup["svc"]
task = await svc.create(_req(task_setup)) task = await svc.create(_req(task_setup))
task.branch_name = "feature/backend/delegate" task.branch_name = "feature/backend/delegate"
@@ -1657,11 +1657,9 @@ async def test_submit_for_pm_review_advances_with_notes(
async def test_submit_for_pm_review_waives_branch_pr_for_batch_umbrella( async def test_submit_for_pm_review_waives_branch_pr_for_batch_umbrella(
task_setup: dict, db_session: AsyncSession task_setup: dict, db_session: AsyncSession
) -> None: ) -> None:
"""F001: a MegaTask umbrella is branchless by design (no branch/PR) yet """A MegaTask umbrella is branchless by design yet must walk
must walk in_progress -> awaiting_pm_review so main_pm_complete can in_progress -> awaiting_pm_review; submit_for_pm_review waives the
escalate it to the CEO. submit_for_pm_review must waive the branch+PR branch+PR requirement for a batch umbrella so completion does not deadlock."""
requirement for a batch umbrella, or umbrella completion deadlocks in
in_progress forever (the Main PM loops on `complete` -> invalid_state)."""
svc = task_setup["svc"] svc = task_setup["svc"]
task = await svc.create(_req(task_setup)) task = await svc.create(_req(task_setup))
task.status = TaskStatus.IN_PROGRESS task.status = TaskStatus.IN_PROGRESS
@@ -1682,13 +1680,10 @@ async def test_submit_for_pm_review_waives_branch_pr_for_batch_umbrella(
async def test_activate_batch_root_subtasks_retypes_code_to_planning( async def test_activate_batch_root_subtasks_retypes_code_to_planning(
task_setup: dict, db_session: AsyncSession task_setup: dict, db_session: AsyncSession
) -> None: ) -> None:
"""F002: a board-routed MegaTask root-subtask is created in BACKLOG with """A board-routed MegaTask root-subtask is created in BACKLOG with
team=board and task_type=code (intake only coerces main_pm-team drafts, so a task_type=code; _activate_batch_root_subtasks must retype it code->planning
board-routed code root-subtask reaches activation still code-typed). When when flipping team to main_pm, mirroring approve_and_start, or the
the CEO approves the umbrella, _activate_batch_root_subtasks flips the held main_pm+code combo recurs."""
child to team=main_pm but if it leaves task_type=code the combo
re-introduces the 2026-06-27 main_pm+code meltdown. The activation must
retype code->planning, mirroring approve_and_start's own retype."""
svc = task_setup["svc"] svc = task_setup["svc"]
# approve_and_start resolves the main-pm agent by slug — seed it. # approve_and_start resolves the main-pm agent by slug — seed it.
main_pm = AgentTable( main_pm = AgentTable(
+1 -1
View File
@@ -1,4 +1,4 @@
"""F003/F014: /api/v1/do/* must enforce the same HMAC agent-token gate as """/api/v1/do/* must enforce the same HMAC agent-token gate as
the /api/v1/flow/* routers. the /api/v1/flow/* routers.
The do router serves every role (content tools are role-uniform), so it has The do router serves every role (content tools are role-uniform), so it has
+2 -4
View File
@@ -194,10 +194,8 @@ async def test_resume_dispatches() -> None:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_i_am_blocked_dispatches_to_choreographer() -> None: async def test_i_am_blocked_dispatches_to_choreographer() -> None:
"""F015: POST /api/v1/flow/documenter/i_am_blocked must exist (the """POST /api/v1/flow/documenter/i_am_blocked returns an envelope (the
documenter manifest registers i_am_blocked) and return an envelope, not 404 documenter manifest registers i_am_blocked) rather than a raw 404."""
with a non-envelope body. Without this route a blocked documenter's escape
hatch 404s."""
mock_chore = MagicMock() mock_chore = MagicMock()
mock_chore.i_am_blocked = AsyncMock( mock_chore.i_am_blocked = AsyncMock(
return_value=_make_envelope(status="blocked", task_id=_TASK_ID) return_value=_make_envelope(status="blocked", task_id=_TASK_ID)
@@ -366,13 +366,9 @@ async def test_resume_dispatches() -> None:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_triage_route_exists_and_dispatches() -> None: async def test_triage_route_exists_and_dispatches() -> None:
"""F067: POST /api/v1/flow/main_pm/triage must exist and wire to """POST /api/v1/flow/main_pm/triage wires to choreographer.triage (the
choreographer.triage. The main_pm manifest (from lifecycle.intents_for_role) main_pm manifest advertises `triage` alongside `triage_all`); the
advertises `triage` alongside `triage_all`, so a main_pm agent calling team-scoped choreographer.triage impl works for any PM role."""
`triage` must hit a real route not a raw 404 that bypasses the circuit
breaker. Mirrors flow_cell_pm's /triage route (the choreographer.triage
impl is team-scoped and works for any PM role).
"""
mock_chore = MagicMock() mock_chore = MagicMock()
mock_chore.triage = AsyncMock(return_value=_make_envelope(status="idle")) mock_chore.triage = AsyncMock(return_value=_make_envelope(status="idle"))
client = TestClient(_build_app(mock_chore)) client = TestClient(_build_app(mock_chore))
+2 -3
View File
@@ -223,9 +223,8 @@ async def test_i_am_idle_dispatches_agent_id() -> None:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_i_am_blocked_dispatches_to_choreographer() -> None: async def test_i_am_blocked_dispatches_to_choreographer() -> None:
"""F015: POST /api/v1/flow/qa/i_am_blocked must exist (the QA manifest """POST /api/v1/flow/qa/i_am_blocked returns an envelope (the QA manifest
registers i_am_blocked) and return an envelope, not 404 with a non-envelope registers i_am_blocked) rather than a raw 404."""
body. Without this route a blocked QA agent's escape hatch 404s."""
mock_chore = MagicMock() mock_chore = MagicMock()
mock_chore.i_am_blocked = AsyncMock( mock_chore.i_am_blocked = AsyncMock(
return_value=_make_envelope(status="blocked", task_id=_TASK_ID) return_value=_make_envelope(status="blocked", task_id=_TASK_ID)
@@ -117,11 +117,8 @@ def test_note_request_coerces_string_next_steps_to_list() -> None:
def test_note_request_coerces_string_where_to_look_to_list() -> None: def test_note_request_coerces_string_where_to_look_to_list() -> None:
"""F118: a single string for where_to_look is wrapped into a one-element """A single string for where_to_look is wrapped into a one-element list,
list. It is a list-typed handoff field like consequences/next_steps and mirroring consequences/next_steps, so a lone scalar does not 422 the route."""
must tolerate a lone scalar without this a well-intentioned
``where_to_look="src/api/"`` 422'd at the route (no remediation envelope)
and the agent's retry loop tripped the do-server circuit breaker."""
req = NoteRequest.model_validate( req = NoteRequest.model_validate(
{"text": "x", "scope": "handoff", "where_to_look": "src/api/auth.py"} {"text": "x", "scope": "handoff", "where_to_look": "src/api/auth.py"}
) )
+3 -11
View File
@@ -1,14 +1,6 @@
"""F023: POST /api/a2a/message/send and /message/stream must enforce the same """POST /api/a2a/message/send and /message/stream enforce the same HMAC
HMAC agent-token gate as the /api/v1/do/* router (F003). agent-token gate as the /api/v1/do/* router (``require_any_authenticated_agent``,
token-only, DB-free, no role assertion the a2a router serves every role).
Both routes previously took only ``request: SendMessageRequest, db: DbSession``
no auth dependency. The sender was self-declared in the request body
(``metadata.from_agent``), so any caller could impersonate any agent and
inject A2A notifications that the orchestrator dispatcher picks up to spawn
target agents. The fix reuses F003's ``require_any_authenticated_agent``
(token-only, DB-free, no role assertion the a2a router serves every role).
In dev (header-trust) mode a missing token is a no-op; a presented-but-forged
token is still rejected, exactly as the do router does.
""" """
from __future__ import annotations from __future__ import annotations
+12 -19
View File
@@ -1,12 +1,8 @@
"""F024: the SSE ``subscribe_to_task`` endpoint must (a) be authenticated """SSE ``subscribe_to_task`` is authenticated like the rest of the a2a
like the rest of the a2a message surface (F023) and (b) acquire a SHORT-LIVED message surface and opens a SHORT-LIVED DB session per poll iteration
DB session per poll iteration instead of holding the request-scoped (via ``get_session_factory()``) instead of holding the request-scoped
``db: DbSession`` for the full SSE lifetime (up to 1 hour / 720 polls), which ``db: DbSession`` for the full SSE lifetime, which exhausted the asyncpg
exhausted the asyncpg pool one connection per connected client. pool one connection per connected client.
The fix mirrors F003's ``require_any_authenticated_agent`` for auth and uses
``get_session_factory()`` inside the generator so each poll opens, queries,
and closes its own session no connection is held across ``asyncio.sleep``.
""" """
from __future__ import annotations from __future__ import annotations
@@ -112,9 +108,9 @@ async def test_subscribe_accepts_valid_token_then_404s_unknown_task(
def test_subscribe_route_does_not_hold_request_scoped_db() -> None: def test_subscribe_route_does_not_hold_request_scoped_db() -> None:
"""F024: the route must NOT depend on ``get_db`` — the request-scoped """The route must NOT depend on ``get_db`` — the request-scoped session
session would be held for the full SSE lifetime (up to 1 hour). Each would be held for the full SSE lifetime (up to 1 hour). Each poll opens
poll must open its own short-lived session via ``get_session_factory``. its own short-lived session via ``get_session_factory``.
""" """
subscribe_route = cast( subscribe_route = cast(
"APIRoute", "APIRoute",
@@ -146,13 +142,10 @@ def test_subscribe_route_does_not_hold_request_scoped_db() -> None:
async def test_subscribe_opens_a_short_lived_session_per_poll( async def test_subscribe_opens_a_short_lived_session_per_poll(
a2a_client: AsyncClient, monkeypatch: pytest.MonkeyPatch a2a_client: AsyncClient, monkeypatch: pytest.MonkeyPatch
) -> None: ) -> None:
"""F024: each poll iteration opens its own session and closes it before """Each poll iteration opens its own session and closes it before the next
the next ``asyncio.sleep`` never holding one connection across the full ``asyncio.sleep`` never holding one connection across the full SSE
SSE lifetime. We patch ``get_session_factory`` to count session opens, lifetime. Asserts more than one session open (one per poll, not one for
patch ``A2AService.get_task`` to return a non-terminal task, patch the lifetime)."""
``asyncio.sleep`` to no-op, and make ``request.is_disconnected`` return
True after a few polls to terminate the stream quickly. The count of
session opens must exceed 1 (one per poll, not one for the lifetime)."""
monkeypatch.setenv("ROBOCO_AGENT_AUTH_SECRET", _SECRET) monkeypatch.setenv("ROBOCO_AGENT_AUTH_SECRET", _SECRET)
monkeypatch.setenv("ROBOCO_AGENT_AUTH_REQUIRED", "true") monkeypatch.setenv("ROBOCO_AGENT_AUTH_REQUIRED", "true")
+4 -6
View File
@@ -147,12 +147,10 @@ async def test_lifespan_startup_and_shutdown_happy_path() -> None:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_lifespan_stops_orchestrator_before_closing_db_and_optimal() -> None: async def test_lifespan_stops_orchestrator_before_closing_db_and_optimal() -> None:
"""F117: orchestrator.stop() must run BEFORE close_optimal_service / close_db """orchestrator.stop() runs BEFORE close_optimal_service / close_db on
on shutdown. stop() drains fire-and-forget DB writes (respawn_tracker shutdown stop() drains fire-and-forget DB writes (respawn_tracker
upserts, audit-log rows) and stop_agent finalizes work sessions / agent upserts, audit-log rows) and finalizes work sessions, all needing the
state all needing the DB still open. Closing the DB first (the old order, DB still open."""
where only bootstrap's finally called stop() after lifespan had already
closed the DB) silently dropped those final writes."""
order: list[str] = [] order: list[str] = []
def _record(label: str) -> AsyncMock: def _record(label: str) -> AsyncMock:
@@ -1,12 +1,7 @@
"""F025: dashboard auditor flag/report mutating routes must be gated to the """Dashboard auditor flag/report mutating routes (``create_auditor_flag``,
Auditor or CEO. ``resolve_auditor_flag``, ``create_auditor_report``, ``send_auditor_report``)
are gated to AUDITOR or CEO via a ``CurrentAgentContext`` dependency plus a
``create_auditor_flag`` / ``resolve_auditor_flag`` / ``create_auditor_report`` coarse role gate, mirroring ``roboco/api/routes/playbooks.py::_require_curator``.
/ ``send_auditor_report`` previously took only ``db: DbSession`` no
``CurrentAgentContext``, no role check so any unauthenticated caller could
create/resolve flags and mark reports as sent to the CEO. The fix mirrors
``roboco/api/routes/playbooks.py::_require_curator``: a ``CurrentAgentContext``
dependency plus a coarse role gate that admits only ``AUDITOR`` and ``CEO``.
""" """
from __future__ import annotations from __future__ import annotations
+7 -7
View File
@@ -279,7 +279,7 @@ def test_request_validation_handler_returns_422_with_details() -> None:
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# F022: secret scrubbing in the 422 log line # secret scrubbing in the 422 log line
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -296,10 +296,10 @@ class _SecretBody(BaseModel):
def test_request_validation_handler_scrubs_secrets_from_log() -> None: def test_request_validation_handler_scrubs_secrets_from_log() -> None:
"""F022: a 422 on a secret-bearing request must not dump the plaintext """A 422 on a secret-bearing request must not dump the plaintext secret
secret into the log line only the redacted placeholder. The 422 into the log line only the redacted placeholder. The 422 response body
response body is unchanged (the client sent those values; the server is unchanged (the client sent those values; the server only redacts its
only redacts its own log).""" own log)."""
app = FastAPI() app = FastAPI()
setup_middleware(app) setup_middleware(app)
@@ -354,8 +354,8 @@ def test_request_validation_handler_scrubs_secrets_from_log() -> None:
def test_request_validation_handler_log_preserves_non_secret_fields() -> None: def test_request_validation_handler_log_preserves_non_secret_fields() -> None:
"""F022: non-secret fields in the body are still logged in full — only """Non-secret fields in the body are still logged in full — only the
the known credential-looking field names are redacted.""" known credential-looking field names are redacted."""
app = FastAPI() app = FastAPI()
setup_middleware(app) setup_middleware(app)
+5 -11
View File
@@ -1,14 +1,8 @@
"""F026: orchestrator control routes (/api/orchestrator/*) must be gated to """Orchestrator control routes (/api/orchestrator/*) are gated to the
the CEO/operator identity. CEO/operator identity: the presented ``X-Agent-ID`` is bound to a verified
HMAC token (DB-free panel-token guard) and the role asserted as CEO. In dev
``spawn_agent`` / ``stop_agent`` / ``resolve_wait`` / ``mark_waiting`` previously (header-trust) mode a missing token is a no-op; a presented-but-forged token
took no auth dependency at all any client that could reach the API could is still rejected.
spawn, stop, mark-waiting, or resolve-wait any agent. The fix mirrors the
F004 panel-token guard (DB-free): bind the presented ``X-Agent-ID`` to a
verified HMAC token and assert the role is CEO. In dev (header-trust) mode a
missing token is a no-op (the panel/operator flow keeps working), but a
presented-but-forged token is still rejected same contract as the v1 flow
role guards and the do router (F003).
""" """
from __future__ import annotations from __future__ import annotations
+5 -10
View File
@@ -1,13 +1,8 @@
"""F004: WebSocket streams must enforce the HMAC panel/CEO token gate when """WebSocket streams (/ws/*, operator-only — the panel is the sole WS client)
ROBOCO_AGENT_AUTH_REQUIRED=true. enforce the HMAC panel/CEO token gate when ROBOCO_AGENT_AUTH_REQUIRED=true:
each per-agent WS upgrade requires + verifies the CEO token in strict mode
The /ws/* streams are operator-only (the panel is the sole WS client; agents and rejects a forged token even in dev mode (same contract as the HTTP role
use MCP verbs, not WS). nginx injects the CEO panel token as X-Agent-Token on gates).
/ws/ upgrades, but the endpoints never read or verified it so in strict mode
an agent on the Docker network could hit /ws/notifications/{id} directly and
subscribe to another agent's notifications with no auth. The fix binds each
per-agent WS upgrade to the CEO token: require + verify it in strict mode, and
reject a forged token even in dev mode (same contract as the HTTP role gates).
""" """
from __future__ import annotations from __future__ import annotations
@@ -1,21 +1,11 @@
"""F065: WS route handlers must disconnect on ANY exit path, not just """WS route handlers must disconnect on ANY exit path, not just
WebSocketDisconnect. WebSocketDisconnect.
The old handlers were ``try: ... while True: receive_text() ... except Each handler adds ``finally: manager.disconnect(websocket)``; ``disconnect``
WebSocketDisconnect: manager.disconnect(websocket)`` with NO ``finally``. is idempotent (``set.discard`` / ``dict.pop`` with default), so the
If ``receive_text()`` raised anything else (anyio closed-resource during clean-disconnect path and the finally both calling it is safe. Tests use
shutdown, ``asyncio.CancelledError``, transport errors), the exception mock sockets (no real app/Redis) and an isolated ``ConnectionManager``
propagated WITHOUT calling ``manager.disconnect(websocket)``, so the dead patched in for the module-global ``manager``.
socket stayed in the subscription set + ``connection_agents`` forever and
was still fanned out to on every broadcast.
The fix adds ``finally: manager.disconnect(websocket)`` to every handler.
``disconnect`` is idempotent (``set.discard`` / ``dict.pop`` with default),
so the clean-disconnect path (still caught by ``except WebSocketDisconnect``
for clarity) and the new finally both calling it is safe.
These tests use mock sockets (no real app/Redis) and an isolated
``ConnectionManager`` patched in for the module-global ``manager``.
""" """
from __future__ import annotations from __future__ import annotations
+7 -15
View File
@@ -1,19 +1,11 @@
"""F066: server-side idle timeout reaps half-open WS sockets. """Server-side idle timeout reaps half-open WS sockets.
The keepalive was client-driven (respond to ``"ping"`` with ``"pong"``); the Each handler wraps ``receive_text()`` in
server never sent its own ping and never timed out a silent client. If an ``asyncio.wait_for(..., timeout=IDLE_TIMEOUT_SECONDS)``; on timeout the
agent container died leaving the TCP socket half-open, ``receive_text()`` handler's ``finally`` disconnects the idle socket. ``IDLE_TIMEOUT_SECONDS``
blocked forever and ``disconnect`` was never called. is a named module constant (ruff PLR2004). Tests patch it to a tiny value
and use a never-resolved ``Future`` so assertions hold in well under a
The fix wraps ``receive_text()`` in second, never relying on real wall-clock timing.
``asyncio.wait_for(..., timeout=IDLE_TIMEOUT_SECONDS)`` per handler; on
``asyncio.TimeoutError`` the finally (from F065) disconnects the idle
socket. ``IDLE_TIMEOUT_SECONDS`` is a named module constant (ruff PLR2004).
Deterministic: the slow-socket test patches ``IDLE_TIMEOUT_SECONDS`` to a
tiny value (0.05s) and uses a ``receive_text`` that returns a never-resolved
``Future`` so the test asserts a prompt disconnect in well under a second,
never relying on real wall-clock timing of the default timeout.
""" """
from __future__ import annotations from __future__ import annotations
+18 -33
View File
@@ -1,24 +1,13 @@
"""F064: per-connection send queue + send timeout — one slow WS client must """Per-connection send queue + send timeout — one slow WS client must not
not back-pressure ALL event delivery to ALL clients. back-pressure ALL event delivery to ALL clients.
The old broadcast did ``await asyncio.gather(*[conn.send_text(data) for conn Each registered connection gets a bounded send queue + a sender coroutine
in connections], return_exceptions=True)`` with no per-connection send queue that drains it, with ``send_text`` behind
and no send timeout. If one client was slow to drain, ``conn.send_text(data)``
awaited indefinitely on the transport, blocking the gather the bridge
handler ``_dispatch_event`` the whole ``_listen_loop`` for every event
type and recipient.
The fix gives each registered connection a bounded send queue + a sender
coroutine that drains it, with ``send_text`` behind
``asyncio.wait_for(..., timeout=SEND_TIMEOUT_SECONDS)``. Broadcasts become ``asyncio.wait_for(..., timeout=SEND_TIMEOUT_SECONDS)``. Broadcasts become
fire-and-enqueue: a slow client's queue fills, then drops/overflows (logged fire-and-enqueue: a slow client's queue fills, then drops/overflows (logged)
as a warning) instead of blocking the fan-out. The listen loop is never instead of blocking the fan-out. Tests patch ``SEND_TIMEOUT_SECONDS`` to a
blocked on a single client. tiny value and use a never-resolved ``Future`` so assertions hold in well
under a second, never relying on real wall-clock timing.
Determinism: every slow-send test uses a ``receive``/``send_text`` that
awaits a never-resolved ``Future`` and patches ``SEND_TIMEOUT_SECONDS`` to a
tiny value, so assertions hold in well under a second and never rely on real
wall-clock timing of the default timeout.
""" """
from __future__ import annotations from __future__ import annotations
@@ -223,21 +212,17 @@ async def test_broadcast_send_timeout_protects_legacy_unregistered_socket() -> N
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Send-side failure proactively reaps the dead socket (F119) # Send-side failure proactively reaps the dead socket.
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_sender_triggers_disconnect_on_send_error() -> None: async def test_sender_triggers_disconnect_on_send_error() -> None:
"""F119: when send_text raises (transport closed / dead socket), the sender """When ``send_text`` raises (transport closed / dead socket), the sender
task must proactively disconnect the socket from every subscription set task must proactively disconnect the socket from every subscription set
not just stop sending and wait for the receive loop's idle timeout to reap rather than wait for the receive loop's idle timeout — otherwise a
it. Without this a send-side-detected dead socket lingers in the sets and send-side-detected dead socket lingers and broadcasts keep enqueuing into
broadcasts keep enqueuing into a queue whose consumer has exited a queue whose consumer has exited."""
(queue-overflow log spam, then silent drops) for up to IDLE_TIMEOUT_SECONDS
until the receive loop's idle timeout finally fires. The send path provably
failed, so it should clean up immediately the receive-side-only reap was
the gap the audit flagged."""
mgr = ConnectionManager() mgr = ConnectionManager()
dead_ws = _make_ws(send_side_effect=ConnectionError("transport closed")) dead_ws = _make_ws(send_side_effect=ConnectionError("transport closed"))
await mgr.connect_system(dead_ws) await mgr.connect_system(dead_ws)
@@ -257,10 +242,10 @@ async def test_sender_triggers_disconnect_on_send_error() -> None:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_sender_keeps_live_socket_on_send_timeout_only() -> None: async def test_sender_keeps_live_socket_on_send_timeout_only() -> None:
"""F119: a send TIMEOUT alone (slow client, not a dead socket) must NOT """A send TIMEOUT alone (slow client, not a dead socket) must NOT disconnect
disconnect the socket only a hard send Exception (transport closed) does. the socket only a hard send Exception (transport closed) does. A
A slow-but-live client should keep receiving once it drains; timing it out slow-but-live client should keep receiving once it drains; timing it out
is the existing F064 graceful-degradation path, not a reap trigger.""" is the graceful-degradation path, not a reap trigger."""
mgr = ConnectionManager() mgr = ConnectionManager()
hang: asyncio.Future[None] = asyncio.Future() hang: asyncio.Future[None] = asyncio.Future()
slow_ws = _make_ws(send_side_effect=hang) slow_ws = _make_ws(send_side_effect=hang)
@@ -1,28 +1,14 @@
"""F074 — the one-task-per-agent invariant had no DB-level enforcement. """The one-task-per-agent invariant is enforced at the DB level by a
PostgreSQL transaction-scoped advisory lock keyed by agent_id, acquired in
``_claim_plan_start_gate`` BEFORE the guard reads (non-coordinator roles
only) and held until the request transaction commits, so a second concurrent
claim's guard read sees the first's committed in_progress task and is
rejected.
``_run_claim_guards`` read the agent's other tasks via unlocked SELECTs CRITICAL regression guard: the lock is acquired ONLY for non-coordinator
(``list_in_progress_for_agent`` / ``list_paused_for_agent``) BEFORE ``claim()`` roles acquiring it for a coordinator would serialize a cell_pm / main_pm's
took its row lock, and ``claim()``'s ``FOR UPDATE`` locked only the TARGET row parallel root planning and regress coordinator concurrency (matches the
not the agent-wide invariant. So two concurrent ``i_will_work_on`` calls by ``_COORDINATOR_ROLES`` guard exemption).
the SAME agent on TWO DIFFERENT pending tasks each locked their own target row
(no contention), each read an empty in_progress set, each passed
``already_active_guard``, and each claim+start succeeded the agent ended with
two in_progress tasks. The in-process ``asyncio.Lock`` serializes container
spawns per agent but is lost on orchestrator-restart split-brain, so it is not
a DB-level guarantee.
The fix: a PostgreSQL transaction-scoped advisory lock keyed by agent_id,
acquired in ``_claim_plan_start_gate`` BEFORE the guard reads (for non-
coordinator roles only). Held until the request transaction commits, it spans
the guard read + the savepoint + the claim write, so the second concurrent
claim's guard read sees the first's committed in_progress task and is rejected.
CRITICAL logical-regression guard: the advisory lock is acquired ONLY for non-
coordinator roles. The PM coordinator concurrency feature (CLAUDE.md) lets a
cell_pm / main_pm plan + delegate many roots in parallel acquiring a per-
agent advisory lock for a coordinator would serialize those claims and
REGRESS that feature. So coordinators are exempt (matching the existing
``_COORDINATOR_ROLES`` guard exemption for ``already_active`` / ``paused``).
""" """
from __future__ import annotations from __future__ import annotations
@@ -281,22 +267,17 @@ async def test_coordinator_claim_does_not_acquire_lock() -> None:
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# F124: the unmet_dependency guard reads dependency state via an unlocked # the unmet_dependency guard reads dependency state via an unlocked SELECT,
# SELECT, then fires release_dependency_blocked_claim (a state mutation: # then fires release_dependency_blocked_claim (claimed/in_progress -> pending,
# claimed/in_progress -> pending, clears branch_name, abandons WorkSession) # clears branch_name, abandons WorkSession) BEFORE returning the rejection. If
# as a side-effect BEFORE returning the rejection. If an upstream dependency # the upstream completes in the microseconds between the read and the release,
# completes (transitions to completed/cancelled) in the microseconds between # the task is NEEDLESSLY released. Dependencies are monotonic (unmet -> met,
# the read and the release, the task is NEEDLESSLY released — its branch # terminal never reopen), so a fresh re-read that now finds them met stays met:
# cleared + WorkSession abandoned + assignee bounced, only to be re-dispatched # safe to proceed without releasing. The fix re-checks unmet_dependency_ids
# + re-claimed when the dependency-completion re-dispatch fires. Dependencies # immediately before the release and skips it when the upstream just completed.
# are monotonic (unmet -> met, terminal: completed/cancelled never reopen), so
# a fresh re-read that now finds them met stays met: safe to proceed without
# releasing. The fix re-checks unmet_dependency_ids immediately before the
# release and skips it (returning None — proceed) when the upstream just
# completed. The "still unmet" path is byte-for-byte the prior behavior.
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Initial dependency read + the re-check before release (F124). # Initial dependency read + the re-check before release.
_DEP_READ_INITIAL_PLUS_RECHECK = 2 _DEP_READ_INITIAL_PLUS_RECHECK = 2
@@ -320,12 +301,9 @@ def _dep_task_svc(agent_id: object, task_id: object, dep_id: object) -> AsyncMoc
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_dependency_guard_skips_release_when_upstream_just_completed() -> None: async def test_dependency_guard_skips_release_when_upstream_just_completed() -> None:
"""F124: the first dependency read sees the upstream still unmet, but by the """The first dependency read sees the upstream still unmet, but the re-check
re-check (a few microseconds later) it has completed. The guard must NOT sees it completed; the guard must NOT release the task (the dependency is
release the task the dependency is now met, so the task can proceed. now met). Returns None (proceed), no release."""
Releasing would needlessly clear its branch + abandon its WorkSession only
to be re-dispatched + re-claimed when the dependency-completion re-dispatch
fires. Returns None (proceed), no release."""
agent_id = uuid4() agent_id = uuid4()
task_id = uuid4() task_id = uuid4()
dep_id = uuid4() dep_id = uuid4()
@@ -353,10 +331,10 @@ async def test_dependency_guard_skips_release_when_upstream_just_completed() ->
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_dependency_guard_releases_when_still_unmet_no_regression() -> None: async def test_dependency_guard_releases_when_still_unmet_no_regression() -> None:
"""F124 no-regression: both the first read AND the re-check see the upstream """No-regression: both the first read AND the re-check see the upstream
still unmet. The guard releases the task to pending (stopping respawn churn still unmet; the guard releases the task to pending and returns the
into a blocked task) and returns the rejection byte-for-byte the prior rejection, so the re-check must not weaken the genuine-blocked release
behavior. The re-check must not weaken the genuine-blocked release path.""" path."""
agent_id = uuid4() agent_id = uuid4()
task_id = uuid4() task_id = uuid4()
dep_id = uuid4() dep_id = uuid4()
@@ -363,15 +363,14 @@ async def test_main_pm_complete_handles_escalate_returning_none() -> None:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_complete_escalates_batch_umbrella_from_in_progress() -> None: async def test_complete_escalates_batch_umbrella_from_in_progress() -> None:
"""F001: a MegaTask umbrella is branchless by design and sits in """A MegaTask umbrella is branchless by design and sits in ``in_progress``
in_progress with no branch/PR. The ``complete`` verb's spec gate with no branch/PR; the ``complete`` verb's spec gate
(``complete`` action source_statuses={AWAITING_PM_REVIEW}) must NOT (``source_statuses={AWAITING_PM_REVIEW}``) must NOT reject it the Main
reject it the Main PM routes through main_pm_complete, which walks PM routes through ``main_pm_complete``, walking in_progress ->
in_progress -> awaiting_pm_review -> awaiting_ceo_approval. Calling the awaiting_pm_review -> awaiting_ceo_approval. Calling the ``complete``
``complete`` ENTRY point (not main_pm_complete directly) must succeed ENTRY point (not ``main_pm_complete`` directly) must succeed and escalate
and escalate to the CEO. This exercises the real spec gate to the CEO; this exercises the real spec gate (``can_invoke_intent`` is
(can_invoke_intent is pure) the prior test mocked submit_pm_review and pure)."""
called main_pm_complete directly, bypassing the gate (false green)."""
pm_id = uuid4() pm_id = uuid4()
umbrella_id = uuid4() umbrella_id = uuid4()
batch_id = uuid4() batch_id = uuid4()
@@ -1,12 +1,6 @@
"""F018 — ``already_active_guard`` must treat a ``blocked`` task as active. """``already_active_guard`` must treat a ``blocked`` task as active. A blocked
task is still owned and will resume to ``in_progress`` on unblock, so it must
``_ACTIVE_BLOCKING_STATUSES`` excluded ``blocked``, so a developer with a block a new claim (preserves the one-active-task-per-dev invariant).
blocked task could claim a second task (the guard passed). When the blocked
task was later unblocked via ``unblock_with_restore`` it resumed to
``in_progress`` leaving the dev silently holding TWO ``in_progress`` tasks,
violating the one-active-task-per-dev invariant the guard exists to enforce.
A blocked task is still owned and will resume to active, so it must block a
new claim.
""" """
from __future__ import annotations from __future__ import annotations
@@ -25,7 +19,7 @@ def _task(*, status: str) -> MagicMock:
def test_already_active_guard_blocks_when_agent_has_blocked_task() -> None: def test_already_active_guard_blocks_when_agent_has_blocked_task() -> None:
"""A blocked task the dev still owns must block a new claim (F018).""" """A blocked task the dev still owns must block a new claim."""
target_id = uuid4() target_id = uuid4()
blocked = _task(status="blocked") blocked = _task(status="blocked")
env = already_active_guard([blocked], target_id) env = already_active_guard([blocked], target_id)
@@ -82,9 +82,9 @@ async def test_pr_pass_guard_blocks_when_validator_cannot_run(
async def test_pr_pass_guard_could_not_run_remediation_uses_pr_fail( async def test_pr_pass_guard_could_not_run_remediation_uses_pr_fail(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
# F044: _conventions_guard is the pr_pass (reviewer) path. A reviewer has no # _conventions_guard is the pr_pass (reviewer) path. A reviewer has no
# i_am_blocked verb, so the could_not_run remediation must point at pr_fail # i_am_blocked verb, so the could_not_run remediation must point at pr_fail
# (the reviewer's reject lever) not tell them to call a verb they lack. # (the reviewer's reject lever), not a verb they lack.
monkeypatch.setattr(settings, "conventions_enabled", True) monkeypatch.setattr(settings, "conventions_enabled", True)
c = _make_choreographer(check_result={"findings": [], "could_not_run": True}) c = _make_choreographer(check_result={"findings": [], "could_not_run": True})
env = await c._conventions_guard(uuid4(), MagicMock(), {}) env = await c._conventions_guard(uuid4(), MagicMock(), {})
@@ -98,13 +98,11 @@ async def test_pr_pass_guard_could_not_run_remediation_uses_pr_fail(
async def test_pr_pass_guard_block_remediation_uses_pr_fail_not_reviewer_waiver( async def test_pr_pass_guard_block_remediation_uses_pr_fail_not_reviewer_waiver(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
# F047: on the pr_pass (reviewer) path a block-level finding's remediation # on the pr_pass (reviewer) path a block-level finding's remediation must
# must point at pr_fail (the reviewer's only lever) and frame the waiver as # point at pr_fail (the reviewer's only lever) and frame the waiver as the
# the DEV's action — NOT tell the reviewer to "add a waiver to # DEV's action — a pr_reviewer does not own the assembled branch and has no
# .roboco/conventions.yml in your branch". A pr_reviewer does not own the # commit verb on it, so the dev-path waiver remediation would strand the
# assembled cell→root / root→master branch and has no commit verb on it, so # gate on every false positive.
# the shared dev-path waiver remediation is unreachable and would strand the
# gate on every false positive (no self-recovery).
monkeypatch.setattr(settings, "conventions_enabled", True) monkeypatch.setattr(settings, "conventions_enabled", True)
c = _make_choreographer(check_result=_BLOCK_RESULT) c = _make_choreographer(check_result=_BLOCK_RESULT)
env = await c._conventions_guard(uuid4(), MagicMock(), {}) env = await c._conventions_guard(uuid4(), MagicMock(), {})
+10 -28
View File
@@ -1,32 +1,14 @@
"""F125 — the delegate sibling-dedup guard had a read/write TOCTOU. """The delegate sibling-dedup guard is serialized by a PostgreSQL
transaction-scoped advisory lock keyed by the parent task id, acquired at the
TOP of the delegate body (before the first ``get_subtasks`` read) and held
through ``create_subtask``'s flush + the outer request commit. Different
parents hash to different keys (seed ``1``, disjoint from the per-agent claim
lock's seed ``0``) so cross-parent delegates are not serialized.
``_delegate_sibling_dedup_guard`` reads the parent's existing subtasks via an CRITICAL regression guard: the lock is per-PARENT, not per-agent. A per-agent
unlocked ``get_subtasks`` SELECT (the dedup read), then the verb body calls lock would serialize all of a coordinator PM's delegates and regress
``create_subtask`` (the write) with no DB serialization between the two. Two coordinator concurrency; the dedup invariant is per-parent, so only same-parent
concurrent ``delegate`` calls for the SAME parent (a PM re-delegating while a delegates serialize.
stale-heartbeat reaper unclaims + re-dispatches, or two orchestrator ticks
racing) each read an empty/duplicate-free sibling set, each pass the dedup
guard, and each create a subtask the parent gets the duplicate the guard
exists to prevent (the smoke-run runaway pattern the guard was built for).
The fix: a PostgreSQL transaction-scoped advisory lock keyed by the parent
task id, acquired at the TOP of the delegate body before the first
``get_subtasks`` read (the briefing's context read AND the dedup guard's
sibling read) and held through ``create_subtask``'s flush + the outer request
commit. The second concurrent same-parent delegate blocks on the lock until
the first commits; its dedup read then sees the first's committed sibling and
is rejected. Different parents hash to different keys (seed ``1``, disjoint
from the per-agent claim lock's seed ``0``) so cross-parent delegates are not
serialized the PM coordinator concurrency feature (parallel root planning)
is preserved.
CRITICAL logical-regression guard: the lock is per-PARENT, not per-agent. A
single cell_pm / main_pm legitimately delegates many subtasks under one parent
in quick succession (a per-dev sequenced queue), and a coordinator PM plans
many roots in parallel. A per-agent lock would serialize all of a PM's
delegates and regress coordinator concurrency; a per-parent lock serializes
only same-parent delegates (the actual dedup invariant is per-parent) and
leaves different parents untouched.
""" """
from __future__ import annotations from __future__ import annotations
+2 -4
View File
@@ -190,10 +190,8 @@ class TestTaskHandoff:
class TestPrReviewSurface: class TestPrReviewSurface:
"""F008 — the persisted pr_fail verdict + issues must surface in the PM """The persisted pr_fail verdict + issues must surface in the PM
briefing's task_handoff, not just the fire-and-forget a2a. A PM respawned briefing's task_handoff, not just the fire-and-forget a2a."""
into ``needs_revision`` after a pr_fail otherwise sees a generic "needs
revision" with zero concrete change-requests and re-submits the same PR."""
def test_surfaces_pr_fail_verdict_and_issues(self) -> None: def test_surfaces_pr_fail_verdict_and_issues(self) -> None:
t = _task(pr_number=138, commits=[{"sha": "abc", "message": "feat: x"}]) t = _task(pr_number=138, commits=[{"sha": "abc", "message": "feat: x"}])
@@ -1,21 +1,7 @@
"""F017 — ``i_am_blocked`` must surface ``invalid_state`` instead of a 500. """``i_am_blocked`` must surface ``invalid_state`` instead of 500 when the
block action returns ``None`` (no escalation target resolvable) the
The bug: ``i_am_blocked`` (any non-``rate_limited`` reason) composes the choreographer emits a re-fetch + escalate-to-CEO rejection rather than
single ``(block,)`` atomic action, whose handler calls dereferencing ``None.status``.
``TaskService.escalate``. ``escalate`` returns ``None`` in four cases
(no task, no agent, no resolvable escalation-target slug, no target agent
row) e.g. a developer whose role has no PM above it in
``get_escalation_target``. Because ``block`` is the LAST composed action,
its ``None`` return flows out of ``run_intent`` as the verb's result. The
choreographer then re-binds ``t`` to that ``None`` and dereferences
``t.status`` building the success envelope ``'NoneType' object has no
attribute 'status'`` HTTP 500. The agent gets no actionable rejection
and respawn-loops.
The fix mirrors F016's ``submit_root`` guard: in
``_run_i_am_blocked_intent``, when the runner returns ``None``, emit an
``invalid_state`` rejection (re-fetch + escalate-to-CEO directly) instead
of letting the caller dereference ``None.status``.
""" """
from __future__ import annotations from __future__ import annotations
@@ -71,7 +57,7 @@ def _make_task_svc(agent_id: object, task_id: object) -> AsyncMock:
team="backend", team="backend",
slug="be-dev-1", slug="be-dev-1",
) )
# F017: escalate resolves no escalation target for this role → None. # escalate resolves no escalation target for this role → None.
task_svc.escalate.return_value = None task_svc.escalate.return_value = None
return task_svc return task_svc
@@ -508,12 +508,9 @@ class TestRateLimitTrackerActivateOnParking:
assert env.status == "in_progress" assert env.status == "in_progress"
async def test_activate_failure_is_logged_not_silent(self) -> None: async def test_activate_failure_is_logged_not_silent(self) -> None:
"""F045: an activate() failure must be logged loudly, not bare-suppressed. """An activate() failure must be logged loudly, not bare-suppressed
the probe-resume loop is tracker-driven, so a silent failure strands
The probe-resume loop is tracker-driven, so a silent activate failure every parked agent in WAITING_LONG with no probe ever running.
strands every parked agent in WAITING_LONG with no probe ever running.
A loud error log makes the stranded-fleet condition visible to
operators (and pairs with the orchestrator's in-memory fallback sweep).
""" """
agent_id = uuid4() agent_id = uuid4()
task_id = uuid4() task_id = uuid4()
+7 -9
View File
@@ -234,11 +234,9 @@ async def test_notify_auditor_rejected_with_not_authorized() -> None:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_notify_rejects_prompter_recipient() -> None: async def test_notify_rejects_prompter_recipient() -> None:
"""F048: the prompter (intake-1) is a human-only role with no agent ack """The prompter (intake-1) is human-only with no agent ack path, so an
path. An ack-required ALERT sent to it sits permanently unacked and via ack-required ALERT to it would sit unacked and dedup-suppress later
the dedup query's ``~acked_by.contains`` — permanently suppresses any same-purpose notifications notify must reject it at the handler."""
later same-purpose notification to that role. The notify verb must reject
a prompter recipient at the handler, not deliver an un-ackable signal."""
agent_id = uuid4() agent_id = uuid4()
task_svc = AsyncMock() task_svc = AsyncMock()
task_svc.get_active_task_for_agent.return_value = None task_svc.get_active_task_for_agent.return_value = None
@@ -264,7 +262,7 @@ async def test_notify_rejects_prompter_recipient() -> None:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_notify_rejects_secretary_recipient() -> None: async def test_notify_rejects_secretary_recipient() -> None:
"""F048: the secretary (secretary-1) is human-only with no agent ack path — """The secretary (secretary-1) is human-only with no agent ack path —
same un-ackable-signal + dedup-suppression hazard as the prompter.""" same un-ackable-signal + dedup-suppression hazard as the prompter."""
agent_id = uuid4() agent_id = uuid4()
task_svc = AsyncMock() task_svc = AsyncMock()
@@ -292,9 +290,9 @@ async def test_notify_rejects_secretary_recipient() -> None:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_notify_allows_ceo_recipient() -> None: async def test_notify_allows_ceo_recipient() -> None:
"""F048: the CEO is human-only too, but the human acks via the panel, so a """The CEO is human-only too, but acks via the panel, so a
non-dependency-block CEO notification is a valid ack-required target. The non-dependency-block CEO notification is a valid ack-required target
recipient guard must NOT over-exclude the CEO (only prompter/secretary).""" the guard must NOT over-exclude the CEO (only prompter/secretary)."""
agent_id = uuid4() agent_id = uuid4()
task_svc = AsyncMock() task_svc = AsyncMock()
task_svc.get_active_task_for_agent.return_value = None task_svc.get_active_task_for_agent.return_value = None
@@ -1,35 +1,20 @@
"""F127 — open_pr's idempotent re-entry guard was a read-then-act with no DB """open_pr's idempotent re-entry guard reads ``t.pr_number`` from an unlocked
serialization, so a CONCURRENT (respawn-race) retry double-emitted the fetch, so two CONCURRENT (respawn-race) retries both pass the guard and both
"opened PR #N" milestone progress entry. emit the 70% "opened PR #N" milestone progress entry double-counting one
PR-open event in the Progress tab + cycle-time metrics (no PR duplication;
GitHub's 422 'already exists' guard holds).
The sequential-retry guard at ``open_pr`` (``if t.pr_number is not None and The fix: a PostgreSQL transaction-scoped advisory lock keyed by the task id
t.assigned_to == agent_id: return Envelope.ok(...)``) short-circuits BEFORE (seed ``2``, disjoint from the per-agent claim lock seed ``0`` and the
the runner and BEFORE ``_open_pr_success_envelope`` so a SECOND call AFTER per-parent delegate lock seed ``1``) acquired at the top of ``open_pr``
the first completed does NOT re-emit the 70% milestone. But this guard reads BEFORE the ``t = await self.task.get(...)`` fetch and held through
``t.pr_number`` from an unlocked fetch. Two CONCURRENT ``open_pr`` calls from
the same agent (the alive-but-unresponsive respawn race CLAUDE.md documents)
both fetch ``t`` with ``pr_number=None``, both pass the guard, both run the
runner (``create_pr``'s GitHub 422 'already exists' path ensures only one PR
is created no double PR), and both then reach
``_open_pr_success_envelope`` ``_record_milestone_progress`` (the 70%
"opened PR #N" entry). Result: TWO milestone progress entries for one PR
the Progress tab + audit reconstruction double-count one PR-open event, and
cycle-time/milestone metrics are skewed. No PR duplication (GitHub 422 guard
holds) and no state corruption purely a double-emission under the narrow
concurrent-retry case.
The fix: a PostgreSQL transaction-scoped advisory lock keyed by the task id,
acquired at the TOP of ``open_pr`` BEFORE the ``t = await self.task.get(...)``
fetch (the read the idempotent guard consults) and held through the runner +
``_record_milestone_progress`` + the outer request commit. The second ``_record_milestone_progress`` + the outer request commit. The second
concurrent same-task ``open_pr`` blocks on the lock until the first commits; same-task concurrent ``open_pr`` blocks until the first commits; its fetch
its fetch then sees the first's committed ``pr_number``, the idempotent guard then sees the committed ``pr_number``, the idempotent guard fires, and it
fires, and it short-circuits WITHOUT re-emitting the milestone. Per-TASK (not short-circuits without re-emitting. Per-TASK (not per-agent): the
per-agent): the single-active-task guard means a dev has one task at a time, single-active-task guard means a dev has one task at a time, so concurrent
so concurrent ``open_pr`` on the SAME task is purely the respawn-race bug case ``open_pr`` on the SAME task is purely the respawn-race case no legitimate
no legitimate concurrency is regressed. Seed ``2`` keeps this in a disjoint concurrency is regressed.
key space from the per-agent claim lock (seed ``0``) and the per-parent
delegate lock (seed ``1``).
""" """
from __future__ import annotations from __future__ import annotations
+4 -4
View File
@@ -115,7 +115,7 @@ async def test_approve_playbook_for_auditor(monkeypatch: pytest.MonkeyPatch) ->
assert env.error is None assert env.error is None
assert env.status == "playbook_approved" assert env.status == "playbook_approved"
svc.approve.assert_awaited_once() svc.approve.assert_awaited_once()
# F057: the status commit gates the index — commit then index, never index # the status commit gates the index — commit then index, never index
# before commit (the index write auto-commits on its own connection). # before commit (the index write auto-commits on its own connection).
actions.task.session.commit.assert_awaited_once() actions.task.session.commit.assert_awaited_once()
svc.index_approved.assert_awaited_once_with(approved) svc.index_approved.assert_awaited_once_with(approved)
@@ -138,7 +138,7 @@ async def test_reject_playbook_archives_for_auditor(
) )
assert env.status == "playbook_archived" assert env.status == "playbook_archived"
svc.reject.assert_awaited_once() svc.reject.assert_awaited_once()
# F057: de-index is the post-commit step (commit gates it). # de-index is the post-commit step (commit gates it).
actions.task.session.commit.assert_awaited_once() actions.task.session.commit.assert_awaited_once()
svc.unindex_playbook.assert_awaited_once_with(archived) svc.unindex_playbook.assert_awaited_once_with(archived)
@@ -147,7 +147,7 @@ async def test_reject_playbook_archives_for_auditor(
async def test_archive_playbook_retires_approved_for_auditor( async def test_archive_playbook_retires_approved_for_auditor(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
"""archive_playbook is the distinct APPROVED->archived retire path (F109): """archive_playbook is the distinct APPROVED->archived retire path:
it calls ``svc.archive`` (NOT ``svc.reject``), commits, then de-indexes.""" it calls ``svc.archive`` (NOT ``svc.reject``), commits, then de-indexes."""
archived = MagicMock() archived = MagicMock()
archived.id = uuid4() archived.id = uuid4()
@@ -172,7 +172,7 @@ async def test_approve_playbook_invalid_state_envelope(
) -> None: ) -> None:
"""A status-precondition ConflictError from the service becomes a clean """A status-precondition ConflictError from the service becomes a clean
invalid_state envelope (not a 500) the agent gets a remediate hint to invalid_state envelope (not a 500) the agent gets a remediate hint to
re-fetch the playbook's current status before re-trying (F109).""" re-fetch the playbook's current status before re-trying."""
svc = MagicMock() svc = MagicMock()
svc.approve = AsyncMock( svc.approve = AsyncMock(
side_effect=ConflictError("not draft", resource_type="playbook") side_effect=ConflictError("not draft", resource_type="playbook")
@@ -244,13 +244,10 @@ async def test_pr_fail_a2a_failure_is_swallowed() -> None:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_pr_fail_returns_invalid_state_when_runner_returns_none() -> None: async def test_pr_fail_returns_invalid_state_when_runner_returns_none() -> None:
"""F046: if a concurrent transition (cancel or a racing reviewer) moved the """A concurrent transition (cancel or racing reviewer) moving the task
task out of ``awaiting_pr_review`` between the precondition gate and the out of ``awaiting_pr_review`` after the gate makes ``run_intent`` return
runner's final composed action, ``run_intent`` returns None (the verb None; ``_gate_decision`` must surface ``invalid_state`` rather than
runner's documented contract for a last-action source-status failure). dereference None and 500 on ``t.assigned_to`` / ``t.status``.
``_gate_decision`` must surface a clean ``invalid_state`` rejection so the
reviewer re-fetches and re-issues NOT dereference None and crash the
gate with a 500 AttributeError on ``t.assigned_to`` / ``t.status``.
""" """
reviewer_id = uuid4() reviewer_id = uuid4()
task_id = uuid4() task_id = uuid4()
@@ -279,7 +276,7 @@ async def test_pr_fail_returns_invalid_state_when_runner_returns_none() -> None:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_pr_pass_returns_invalid_state_when_runner_returns_none() -> None: async def test_pr_pass_returns_invalid_state_when_runner_returns_none() -> None:
"""F046: the same None-guard covers pr_pass — a concurrent cancel between """The same None-guard covers pr_pass — a concurrent cancel between
gate and runner must surface invalid_state, not crash on ``str(t.status)``. gate and runner must surface invalid_state, not crash on ``str(t.status)``.
""" """
reviewer_id = uuid4() reviewer_id = uuid4()
@@ -407,17 +407,16 @@ async def test_pr_pass_does_not_capture_head_sha() -> None:
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# F016 — submit_root must not 500 when submit_for_review returns None # submit_root must not 500 when submit_for_review returns None
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_submit_root_invalid_state_when_submit_for_review_returns_none() -> None: async def test_submit_root_invalid_state_when_submit_for_review_returns_none() -> None:
"""F016: submit_for_review returns None when the root->master PR was already """submit_for_review returns None when the root->master PR was already
opened (the task raced out of in_progress, or a prior call already opened (task raced out of in_progress, or a prior call transitioned it).
transitioned it). create_root_pr already ran as the pre-side-effect, so the submit_root must surface ``invalid_state``, not dereference None.status
PR exists, but the transition did not happen. submit_root must surface an and 500."""
invalid_state envelope, not dereference None.status and 500."""
c, main_pm_id, root_task_id = _resubmit_root(notes_structured=None) c, main_pm_id, root_task_id = _resubmit_root(notes_structured=None)
# The transition did not happen (PR already opened / task raced). # The transition did not happen (PR already opened / task raced).
c.task.submit_for_review.return_value = None c.task.submit_for_review.return_value = None
@@ -1,17 +1,7 @@
"""F007 — the unchanged-PR re-submit loop-stopper is root-only; ``submit_up`` """The unchanged-PR re-submit loop-stopper, applied to ``submit_up`` (cell→root).
(cellroot) had no head_sha guard, so a weak cell PM could re-submit the
unchanged cell PR and loop ``awaiting_pr_review`` ``pr_fail`` forever
(the cell-level analogue of the 2026-06-27 root loop F016 closes).
``pr_fail`` stamps the assembled PR's head SHA into Refuses to re-open the gate when the cell PR's head SHA equals the SHA the
``notes_structured.pr_review.head_sha`` for BOTH cell and root gate tasks last ``pr_fail`` recorded (no new dev work landed); ambiguous cases FAIL OPEN.
(``pr_gate._capture_pr_head_sha`` / ``_record_gate_verdict`` are
gate-verb-level, not root-level). So the same structural refusal applies
to ``submit_up``: if the cell PR's current head SHA equals the SHA the
last ``pr_fail`` recorded, no new dev work landed on the cell branch
the diff is byte-identical refuse, do not re-open the gate. Every
ambiguous case FAILS OPEN, identical to the root guard (shared
``_current_pr_head_sha``).
""" """
from __future__ import annotations from __future__ import annotations
@@ -145,25 +135,17 @@ async def test_submit_up_fail_open_when_no_prior_pr_fail_verdict() -> None:
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# F122: when submit_for_review returns None (a concurrent state change moved # submit_for_review returns None when the task raced out of in_progress after
# the task out of in_progress AFTER the create_pr pre-side-effect already # create_pr already opened the cell→root PR; the remediate must tell the PM the
# opened the cell→root PR), the invalid_state remediate must TELL the cell PM # PR is open so the orphan is recoverable via create_pr's idempotent re-issue.
# the PR is already open. The old remediate ('must be in_progress with PR
# ready') hid that the PR exists — so the agent could not tell an orphaned PR
# was sitting on GitHub. The orphan is inherent to the correct pre-side-effect
# ordering (submit_for_review's pr_created gate requires create_pr first, see
# lifecycle.py:1338-1343) and is recoverable via create_pr's idempotent re-issue
# — but only if the agent KNOWS the PR is open. Mirrors submit_root's F016
# remediate (_impl.py:6305-6310).
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_submit_up_none_remediate_names_the_already_open_pr() -> None: async def test_submit_up_none_remediate_names_the_already_open_pr() -> None:
"""F122: submit_for_review returns None (raced out of in_progress) AFTER """submit_for_review returns None (raced out of in_progress) AFTER create_pr
create_pr already opened the cellroot PR. The rejection remediate must already opened the cellroot PR. The remediate must name the open PR and point
name the open PR and point the PM at re-fetching + reconciling, not the the PM at re-fetching + reconciling, not the misleading 'PR ready' hint."""
misleading 'must be in_progress with PR ready' that hides the PR exists."""
c, cell_pm_id, cell_task_id = _resubmit_cell(notes_structured=None) c, cell_pm_id, cell_task_id = _resubmit_cell(notes_structured=None)
# A concurrent transition (stale-heartbeat reaper unclaim, or a racing # A concurrent transition (stale-heartbeat reaper unclaim, or a racing
# i_am_blocked) moved the task out of in_progress between the precondition # i_am_blocked) moved the task out of in_progress between the precondition
+5 -8
View File
@@ -99,11 +99,9 @@ async def test_guard_silent_when_no_marker(monkeypatch: pytest.MonkeyPatch) -> N
async def test_guard_reviewer_remediation_uses_pr_fail_not_i_am_blocked( async def test_guard_reviewer_remediation_uses_pr_fail_not_i_am_blocked(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
# F044: the pr_pass gate runs this guard on the REVIEWER's workspace. A PR # pr_pass runs this guard on the REVIEWER's workspace; the remediation must
# reviewer has no i_am_blocked verb, so the dev-path remediation ("call # use pr_fail (not i_am_blocked — a reviewer has no i_am_blocked verb) so the
# i_am_blocked(reason='toolchain')") sends them to a verb they cannot call. # PR returns to needs_revision for the dev to fix the environment.
# The reviewer's reject lever is pr_fail — the remediation must point there
# so the PR goes back to needs_revision for the dev to fix the environment.
monkeypatch.setattr(settings, "toolchain_match_enabled", True) monkeypatch.setattr(settings, "toolchain_match_enabled", True)
c = _make_choreographer(status="broken") c = _make_choreographer(status="broken")
env = await c._toolchain_broken_guard(uuid4(), MagicMock(), reviewer=True) env = await c._toolchain_broken_guard(uuid4(), MagicMock(), reviewer=True)
@@ -118,9 +116,8 @@ async def test_guard_reviewer_remediation_uses_pr_fail_not_i_am_blocked(
async def test_guard_dev_remediation_still_uses_i_am_blocked( async def test_guard_dev_remediation_still_uses_i_am_blocked(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
# F044: the dev (i_am_done) path keeps i_am_blocked — a dev DOES have that # the dev (i_am_done) path keeps i_am_blocked — a dev has that verb, so the
# verb, so the original remediation is correct there. The reviewer flag must # reviewer flag must not change the dev-path wording.
# not change the dev-path wording.
monkeypatch.setattr(settings, "toolchain_match_enabled", True) monkeypatch.setattr(settings, "toolchain_match_enabled", True)
c = _make_choreographer(status="broken") c = _make_choreographer(status="broken")
env = await c._toolchain_broken_guard(uuid4(), MagicMock()) env = await c._toolchain_broken_guard(uuid4(), MagicMock())
+11 -15
View File
@@ -112,11 +112,10 @@ def test_refresh_mints_new_token_when_stale(tmp_path: Path) -> None:
def test_refresh_omitting_expires_in_still_marks_token_valid(tmp_path: Path) -> None: def test_refresh_omitting_expires_in_still_marks_token_valid(tmp_path: Path) -> None:
"""F092: if xAI's refresh response omits ``expires_in``, the access token's """If xAI's refresh response omits ``expires_in``, the access token's JWT
JWT ``exp`` claim is the authoritative expiry decode it so a fresh token ``exp`` claim is the authoritative expiry decode it so a fresh token isn't
isn't left with the stale pre-refresh ``expires_at`` (which would make left with the stale pre-refresh ``expires_at`` (which would re-rotate the
``is_valid`` / ``--check`` forever reject it and the refresh loop re-rotate single-use refresh token every tick)."""
the single-use refresh token every tick)."""
path = tmp_path / "auth.json" path = tmp_path / "auth.json"
_write(path, _bundle(_PAST)) _write(path, _bundle(_PAST))
exp_unix = int((datetime.now(UTC) + timedelta(hours=6)).timestamp()) exp_unix = int((datetime.now(UTC) + timedelta(hours=6)).timestamp())
@@ -136,10 +135,9 @@ def test_refresh_omitting_expires_in_still_marks_token_valid(tmp_path: Path) ->
def test_refresh_omitting_expires_in_with_unreadable_jwt_defaults_ttl( def test_refresh_omitting_expires_in_with_unreadable_jwt_defaults_ttl(
tmp_path: Path, tmp_path: Path,
) -> None: ) -> None:
"""F092 fallback: expires_in missing AND the access token isn't a JWT with a """Fallback when ``expires_in`` is missing AND the access token isn't a JWT
readable ``exp`` default to the documented ~6h TTL so a fresh token is with a readable ``exp``: default to the documented ~6h TTL so a fresh token
treated as live instead of stale, rather than forever rejected (the warning is treated as live instead of forever rejected."""
is emitted via structlog, visible in the captured stdout)."""
path = tmp_path / "auth.json" path = tmp_path / "auth.json"
_write(path, _bundle(_PAST)) _write(path, _bundle(_PAST))
@@ -185,12 +183,10 @@ def test_refresh_failed_when_no_access_token(tmp_path: Path) -> None:
def test_refresh_persists_rotated_token_when_atomic_write_fails( def test_refresh_persists_rotated_token_when_atomic_write_fails(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None: ) -> None:
"""F006: a rotated refresh_token is single-use — xAI invalidates the old one """A rotated refresh_token is single-use — xAI invalidates the old one on
the moment it issues the new one. If the atomic write (tmp+replace) fails rotation. If the atomic write (tmp+replace) fails after rotation, the file
after the rotation, the file keeps the now-dead old refresh_token and the keeps the now-dead refresh_token and the credential is permanently lost; the
credential is permanently lost on the next refresh. The write must fall back write must fall back to a direct write so the rotated token survives."""
to a direct write so the rotated refresh_token survives even when the atomic
replace can't."""
path = tmp_path / "auth.json" path = tmp_path / "auth.json"
_write(path, _bundle(_PAST)) _write(path, _bundle(_PAST))
+3 -4
View File
@@ -242,10 +242,9 @@ async def test_grok_spawn_mounts_auth_when_present(_isolate_grok_auth: Path) ->
) as exec_mock: ) as exec_mock:
await provider.spawn(_config()) await provider.spawn(_config())
cmd = list(exec_mock.call_args.args) cmd = list(exec_mock.call_args.args)
# F005: mount the host ~/.grok DIRECTORY (ro), not the single auth.json # mount the host ~/.grok DIRECTORY (ro), not the single auth.json file — a
# file — a single-file bind mount pins the inode, so the orchestrator's # single-file bind mount pins the inode, so the orchestrator's atomic
# atomic auth.json refresh (rename) never reaches a running container. # auth.json refresh (rename) never reaches a running container.
# The entrypoint symlinks ~/.grok/auth.json at this RO dir mount.
expected = f"{_isolate_grok_auth}:/home/agent/.grok-auth-ro:ro" expected = f"{_isolate_grok_auth}:/home/agent/.grok-auth-ro:ro"
assert expected in cmd assert expected in cmd
@@ -265,17 +265,10 @@ def test_verb_extracted_from_path(do_module: types.ModuleType) -> None:
def test_dict_shaped_error_does_not_crash(do_module: types.ModuleType) -> None: def test_dict_shaped_error_does_not_crash(do_module: types.ModuleType) -> None:
"""A RobocoError.to_dict()-shaped response must not TypeError the breaker. """A RobocoError.to_dict()-shaped response must not TypeError the breaker.
Smoke-7: A2AAccessDeniedError escaped to middleware and was rendered as A dict-shaped `error` is a retry-storm-worthy rejection (the orchestrator's
{'error': {'code': ..., 'message': ..., 'details': ...}}. The circuit exception handlers surface this shape on 4xx/5xx), so the breaker must count
breaker's `error in frozenset` check then crashed with it via the classifier rather than passing it through silently. The original
`TypeError: unhashable type: 'dict'`. dict payload still reaches the agent (the breaker only substitutes when open).
F068: a dict-shaped `error` is a retry-storm-worthy rejection (the
orchestrator's exception handlers all surface this shape on 4xx/5xx),
so the breaker must COUNT it mapped to a counted kind by the
classifier rather than passing it through silently. The original
dict payload still reaches the agent (the breaker only substitutes
when open). No TypeError may be raised either way.
""" """
factory, captured = _make_client( factory, captured = _make_client(
orchestrator_response={ orchestrator_response={
@@ -311,10 +304,9 @@ def test_dict_shaped_error_does_not_crash(do_module: types.ModuleType) -> None:
def test_422_validation_failure_counts_as_incomplete_input( def test_422_validation_failure_counts_as_incomplete_input(
do_module: types.ModuleType, do_module: types.ModuleType,
) -> None: ) -> None:
"""F068: a 422 validation-failure body (`{"detail": [...], "body": ...}`, """A 422 validation-failure body (`{"detail": [...], "body": ...}`, no `error`
no `error` field) must count toward the breaker a storm of 422s is field) must count toward the breaker a storm of 422s is retry-storm-worthy.
retry-storm-worthy (the agent keeps re-submitting malformed input). Mapped to `incomplete_input`.
Mapped to `incomplete_input` (the agent's input was incomplete/invalid).
""" """
factory, captured = _make_client( factory, captured = _make_client(
orchestrator_response={ orchestrator_response={
@@ -347,9 +339,8 @@ def test_422_validation_failure_counts_as_incomplete_input(
def test_dict_shaped_internal_error_counts_as_invalid_state( def test_dict_shaped_internal_error_counts_as_invalid_state(
do_module: types.ModuleType, do_module: types.ModuleType,
) -> None: ) -> None:
"""F068: a 500 INTERNAL_ERROR dict-shaped response (generic_exception_handler) """A 500 INTERNAL_ERROR dict-shaped response (generic_exception_handler) must
must count toward the breaker as `invalid_state` a storm of 500s is count toward the breaker as `invalid_state` a storm of 500s is retry-storm-worthy.
retry-storm-worthy and previously bypassed the breaker entirely.
""" """
factory, captured = _make_client( factory, captured = _make_client(
orchestrator_response={ orchestrator_response={
@@ -379,7 +370,7 @@ def test_dict_shaped_internal_error_counts_as_invalid_state(
def test_dict_shaped_invalid_input_counts_as_incomplete_input( def test_dict_shaped_invalid_input_counts_as_incomplete_input(
do_module: types.ModuleType, do_module: types.ModuleType,
) -> None: ) -> None:
"""F068: a dict-shaped INVALID_INPUT (mapped from 422 by http_exception_handler) """A dict-shaped INVALID_INPUT (mapped from 422 by http_exception_handler)
counts as `incomplete_input` semantically the agent's input was invalid. counts as `incomplete_input` semantically the agent's input was invalid.
""" """
factory, captured = _make_client( factory, captured = _make_client(
@@ -404,19 +395,18 @@ def test_dict_shaped_invalid_input_counts_as_incomplete_input(
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# F069 — a manifest-registered content tool whose route is missing must # A manifest-registered content tool whose route is missing must return an
# return an envelope rejection (not a raw 404 body) so the breaker counts it. # envelope rejection (not a raw 404 body) so the breaker counts it.
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
def test_missing_route_404_returns_envelope_and_counts( def test_missing_route_404_returns_envelope_and_counts(
do_module: types.ModuleType, do_module: types.ModuleType,
) -> None: ) -> None:
"""F069: a 404 from the orchestrator (manifest-registered tool with no """A 404 from the orchestrator (manifest-registered tool with no route) must
route) must surface as a proper `invalid_state` Envelope rejection not surface as a proper `invalid_state` Envelope rejection not FastAPI's raw
FastAPI's raw ``{"detail": "Not Found"}`` body — and the breaker must ``{"detail": "Not Found"}`` body and the breaker must count it. Mirrors
count it. Without this, the agent retries the missing tool forever and flow_server's 404 handling.
the breaker never trips. Mirrors flow_server's 404 handling.
""" """
captured: list[tuple[str, dict[str, Any] | None]] = [] captured: list[tuple[str, dict[str, Any] | None]] = []
@@ -432,17 +432,16 @@ def test_task_id_none_is_forwarded_as_null(flow_module: types.ModuleType) -> Non
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# F068 — non-string-error / no-error-field rejection shapes are counted # Non-string-error / no-error-field rejection shapes are counted
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
def test_422_validation_failure_counts_as_incomplete_input( def test_422_validation_failure_counts_as_incomplete_input(
flow_module: types.ModuleType, flow_module: types.ModuleType,
) -> None: ) -> None:
"""F068: a 422 validation-failure body (`{"detail": [...]}`, no `error`) """A 422 validation-failure body (`{"detail": [...]}`, no `error`) must count
must count toward the breaker as `incomplete_input` a storm of 422s is toward the breaker as `incomplete_input` a storm of 422s is retry-storm-worthy.
retry-storm-worthy. Mirrors do_server's classifier (the two servers share Mirrors do_server's classifier (the two servers share the same breaker logic).
the same breaker logic and must stay in parity).
""" """
factory, captured = _make_client( factory, captured = _make_client(
orchestrator_response={ orchestrator_response={
@@ -475,8 +474,8 @@ def test_422_validation_failure_counts_as_incomplete_input(
def test_dict_shaped_internal_error_counts_as_invalid_state( def test_dict_shaped_internal_error_counts_as_invalid_state(
flow_module: types.ModuleType, flow_module: types.ModuleType,
) -> None: ) -> None:
"""F068: a 500 INTERNAL_ERROR dict-shaped response (generic_exception_handler) """A 500 INTERNAL_ERROR dict-shaped response (generic_exception_handler) counts
counts as `invalid_state` a storm of 500s previously bypassed the breaker. as `invalid_state` a storm of 500s is retry-storm-worthy.
""" """
factory, captured = _make_client( factory, captured = _make_client(
orchestrator_response={ orchestrator_response={
@@ -506,9 +505,8 @@ def test_dict_shaped_internal_error_counts_as_invalid_state(
def test_dict_shaped_not_found_does_not_count( def test_dict_shaped_not_found_does_not_count(
flow_module: types.ModuleType, flow_module: types.ModuleType,
) -> None: ) -> None:
"""F068: a dict-shaped NOT_FOUND (404 family) does NOT count — parity with """A dict-shaped NOT_FOUND (404 family) does NOT count — parity with the
the string-error contract that a `not_found` rejection isn't counted string-error contract that a `not_found` rejection isn't counted.
(retrying a missing resource won't help until state changes).
""" """
factory, captured = _make_client( factory, captured = _make_client(
orchestrator_response={ orchestrator_response={
@@ -524,8 +522,8 @@ def test_dict_shaped_not_found_does_not_count(
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# F069 — a manifest-registered verb whose route is missing must return an # A manifest-registered verb whose route is missing must return an envelope
# envelope rejection (not a raw 404 body) so the breaker counts it. # rejection (not a raw 404 body) so the breaker counts it.
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -570,11 +568,9 @@ def _make_404_client() -> tuple[Any, list[tuple[str, dict[str, Any] | None]]]:
def test_missing_route_404_returns_envelope_and_counts( def test_missing_route_404_returns_envelope_and_counts(
flow_module: types.ModuleType, flow_module: types.ModuleType,
) -> None: ) -> None:
"""F069: a 404 from the orchestrator (manifest-registered verb with no """A 404 from the orchestrator (manifest-registered verb with no route) must
route) must surface as a proper `invalid_state` Envelope rejection not surface as a proper `invalid_state` Envelope rejection not FastAPI's raw
FastAPI's raw ``{"detail": "Not Found"}`` body — and the breaker must ``{"detail": "Not Found"}`` body and the breaker must count it.
count it. Without this, the agent retries the missing route forever and
the breaker never trips.
""" """
factory, captured = _make_404_client() factory, captured = _make_404_client()
with patch("httpx.Client", side_effect=factory): with patch("httpx.Client", side_effect=factory):
@@ -196,16 +196,9 @@ def test_in_progress_task_with_no_agent_returns_assignee() -> None:
def test_claimed_task_with_unknown_assignee_returns_slug_for_release() -> None: def test_claimed_task_with_unknown_assignee_returns_slug_for_release() -> None:
# F032: a claimed/in_progress task whose assignee is a stale/unknown UUID # A claimed/in_progress task whose assignee is a stale/unknown UUID (no
# (no seeded agent) must reach the release-to-pending path. The human-only # seeded agent) must reach the release-to-pending path: the human-only guard
# guard (role_for_slug_or_none) returns None for an unknown slug, and # returns None for unknown slugs, so the slug falls through and is released.
# ``None in (CEO, PROMPTER, SECRETARY)`` is False — so it does NOT
# short-circuit, the slug falls through the grace window, and the resolver
# returns the unknown slug. _dispatch_claimed_without_agent then sees
# get_agent_role(slug) == "unknown" and releases the claim to pending for
# a role-matched reclaim. Before F031's role_for_slug_or_none fix the guard
# raised KeyError on the unknown slug, crashing the whole tick before the
# release path could run.
orch = _orch() orch = _orch()
unknown_uuid = str(uuid4()) unknown_uuid = str(uuid4())
task: dict[str, Any] = { task: dict[str, Any] = {
+5 -5
View File
@@ -58,11 +58,11 @@ async def test_load_watch_set_filters_enabled_one_per_repo() -> None:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_load_watch_set_keeps_distinct_workflows_per_repo() -> None: async def test_load_watch_set_keeps_distinct_workflows_per_repo() -> None:
"""F115: a monorepo's several cell-projects each carrying their OWN """A monorepo's several cell-projects each carrying their OWN
``ci_watch_workflow`` must ALL be watched collapsing to the canonical ``ci_watch_workflow`` must ALL be watched collapsing to the canonical cell's
cell's workflow would miss a red on the other cells' workflows (under-count). workflow would miss a red on the other cells' workflows (under-count). Same
Same repo, DIFFERENT workflows one entry per (repo, workflow). The engine's repo, DIFFERENT workflows one entry per (repo, workflow); per-git_url dedup
per-git_url fix-task dedup still prevents duplicate fix tasks for the repo.""" still prevents duplicate fix tasks for the repo."""
orch = _orch() orch = _orch()
be = MagicMock( be = MagicMock(
slug="be", slug="be",
+6 -6
View File
@@ -46,12 +46,12 @@ async def test_load_set_filters_command_one_per_repo() -> None:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_load_set_keeps_distinct_commands_per_repo() -> None: async def test_load_set_keeps_distinct_commands_per_repo() -> None:
"""F115: a monorepo's several cell-projects each carrying their OWN """A monorepo's several cell-projects each carrying their OWN
``dep_update_command`` (different ecosystems different lockfiles) must ``dep_update_command`` (different ecosystems different lockfiles) must ALL
ALL be probed collapsing to the canonical cell's command would miss the be probed collapsing to the canonical cell's command would miss the other
other cells' lockfile drift (under-count). Same repo, DIFFERENT commands → cells' lockfile drift (under-count). Same repo, DIFFERENT commands → one
one entry per (repo, command). The engine's per-git_url open-task dedup entry per (repo, command); per-git_url open-task dedup still prevents
still prevents duplicate update tasks for the repo.""" duplicate update tasks for the repo."""
orch = _orch() orch = _orch()
be = MagicMock( be = MagicMock(
slug="be", git_url="https://x/a.git", dep_update_command="uv lock --upgrade" slug="be", git_url="https://x/a.git", dep_update_command="uv lock --upgrade"
+3 -5
View File
@@ -60,11 +60,9 @@ async def test_cost_over_cap_kills_and_evicts(monkeypatch: pytest.MonkeyPatch) -
async def test_cost_over_cap_finalizes_spawn_session_before_evict( async def test_cost_over_cap_finalizes_spawn_session_before_evict(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
# F040: a cost-cap-killed grok container must finalize its spawn session so # Cost-cap-killed grok container must finalize its spawn session BEFORE the
# the captured usage/cost is recorded in the DB/dashboard — otherwise the # instance is popped: _finalize_spawn_session reads _instances[agent_id] for
# session row stays open (ended_at IS NULL) and the burn is invisible. # the model + usage_session_id; otherwise the burn stays invisible.
# Finalization must run BEFORE the instance is popped: _finalize_spawn_session
# reads self._instances[agent_id] for the model + usage_session_id.
orch, _remove_mock = _orch(monkeypatch, cap=5.0, cost=7.5) orch, _remove_mock = _orch(monkeypatch, cap=5.0, cost=7.5)
finalize = AsyncMock() finalize = AsyncMock()
monkeypatch.setattr(orch, "_finalize_spawn_session", finalize) monkeypatch.setattr(orch, "_finalize_spawn_session", finalize)
+8 -10
View File
@@ -116,7 +116,7 @@ async def test_park_grok_rate_limited_activates_and_offlines(
# needs the dict + persist stub to exercise that without AttributeError. # needs the dict + persist stub to exercise that without AttributeError.
orch._waiting_records = {} orch._waiting_records = {}
orch._rate_limit_ceo_notified = set() orch._rate_limit_ceo_notified = set()
# F097 backoff state — the constructor (skipped here) initializes these. # Backoff state — the constructor (skipped here) initializes these.
orch._grok_last_park_at = None orch._grok_last_park_at = None
orch._grok_repark_count = 0 orch._grok_repark_count = 0
inst = _grok_instance() inst = _grok_instance()
@@ -159,7 +159,7 @@ async def test_handle_stopped_container_parks_on_grok_429(
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# F041: exit 78 (auth missing/expired) parks instead of crash-retrying # Exit 78 (auth missing/expired) parks instead of crash-retrying
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -178,11 +178,9 @@ def test_is_grok_auth_exit() -> None:
async def test_handle_stopped_container_parks_on_grok_auth_exit( async def test_handle_stopped_container_parks_on_grok_auth_exit(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
# F041: a grok container whose entrypoint ran `grok_auth --check` and found # A grok container whose entrypoint ran `grok_auth --check` and found the
# the token missing/expired exits 78 (EX_CONFIG). Crash-retrying 3x burns # token missing/expired exits 78 (EX_CONFIG); park it (like the 429 exit-75
# tokens for zero progress (the agent can't start without a valid token); # path) so the probe-resume loop revives the task once a fresh token is minted.
# park it like the 429 exit-75 path so the probe-resume loop revives the
# task once grok_auth.refresh_if_stale mints a fresh token.
orch = AgentOrchestrator.__new__(AgentOrchestrator) orch = AgentOrchestrator.__new__(AgentOrchestrator)
inst = _grok_instance() inst = _grok_instance()
park = AsyncMock() park = AsyncMock()
@@ -224,9 +222,9 @@ async def test_park_grok_auth_unavailable_activates_with_auth_missing_kind(
# --------------------------------------------------------------------------- # # --------------------------------------------------------------------------- #
# F097 — grok has no real probe, so an optimistic clear respawns into a still- # Grok has no real probe, so an optimistic clear respawns into a still-active
# active xAI 429 every ~90s. Back off the re-park retry_after within one rate- # xAI 429 every ~90s; back off the re-park retry_after within one rate-limit
# limit episode so the churn dampens instead of spinning flat at 60s. # episode so the churn dampens instead of spinning flat at 60s.
# --------------------------------------------------------------------------- # # --------------------------------------------------------------------------- #
+5 -7
View File
@@ -37,8 +37,8 @@ def _make_minimal_orchestrator() -> AgentOrchestrator:
# (F071); without this the post-docker-run guard would AttributeError on # (F071); without this the post-docker-run guard would AttributeError on
# the constructor-skipped instance. # the constructor-skipped instance.
orch._running = True orch._running = True
# F093: concurrent intake starts serialize on this lock; the constructor # Concurrent intake starts serialize on this lock; the constructor (skipped
# (skipped here) initializes it. # here) initializes it.
orch._intake_spawn_lock = asyncio.Lock() orch._intake_spawn_lock = asyncio.Lock()
return orch return orch
@@ -572,11 +572,9 @@ class TestDeliverWhenReady:
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# F071 — non-blocking intake spawn must not orphan a container if shutdown # Non-blocking intake spawn must not orphan a container if shutdown arrives
# arrives between ``docker run`` and the _instances registration. The guarded # between ``docker run`` and _instances registration: without a post-docker-run
# wrapper runs concurrently with stop(); without a post-docker-run shutdown # shutdown check the just-started container is never recorded so leaks.
# check, the just-started container is never recorded in _instances (which
# stop() already iterated) so nothing tears it down — a leaked container.
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -82,7 +82,7 @@ def test_intake_grok_mounts_subscription_auth_when_present(
cmd = AgentOrchestrator._build_intake_run_cmd( cmd = AgentOrchestrator._build_intake_run_cmd(
_intake_spec("grok", base_url="https://api.x.ai/v1", token="xai-key") _intake_spec("grok", base_url="https://api.x.ai/v1", token="xai-key")
) )
# F005: directory mount (ro), not the single-file inode-pinning mount. # directory mount (ro), not the single-file inode-pinning mount.
assert f"{grok_dir}:/home/agent/.grok-auth-ro:ro" in cmd assert f"{grok_dir}:/home/agent/.grok-auth-ro:ro" in cmd
@@ -1,22 +1,10 @@
"""F070 — fire-and-forget ``_bg_tasks`` (respawn_tracker upserts, audit-log """Drain ``_bg_tasks`` on shutdown so fire-and-forget writes (respawn_tracker
writes, intake first-message delivery) were never cancelled or drained on upserts, audit-log writes, intake first-message delivery) are not abandoned.
shutdown. ``Orchestrator.stop()`` cancelled only the named loop tasks and the
agents, then returned, abandoning any in-flight ``_schedule_bg`` work.
The data-loss tail: an in-flight ``_persist_respawn_record`` upsert dropped at Invariant: ``Orchestrator.stop()`` drains ``_bg_tasks`` with a bounded timeout
shutdown means the last few gate-mutation strikes never reach the DB. The short DB writes finish before the process exits (data preserved), a stuck task
in-memory counter dies with the process; ``restore_respawn_tracker()`` on the is cancelled once the deadline passes (can't hang shutdown). The ``stop_agent``
next start repopulates a stale lower count and the dispatcher re-burns the loop is wrapped so one agent's stop error can't skip the drain.
full strike threshold (4 spawns) against a still-wedged task the exact
re-burn the durable tracker exists to stop. Audit-log writes (load-bearing for
the cycle-time / rework metrics) are similarly dropped.
The fix DRAINs ``_bg_tasks`` with a bounded timeout on shutdown short DB
writes finish before the process exits (data preserved), while a stuck task
can't hang shutdown (it is cancelled once the drain deadline passes). Cancels
outright would lose the data (the opposite of the goal), so the drain tries to
let work complete first. The ``stop_agent`` loop is also wrapped so one agent's
stop error can't skip the drain (which would still drop the data).
""" """
from __future__ import annotations from __future__ import annotations
@@ -152,10 +140,9 @@ async def test_stop_failing_agent_does_not_skip_drain() -> None:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_stop_is_idempotent_double_call_is_noop() -> None: async def test_stop_is_idempotent_double_call_is_noop() -> None:
"""F117: stop() is idempotent. The lifespan shutdown path now stops the """stop() is idempotent: the lifespan path and bootstrap's finally block both
orchestrator before closing the DB, and bootstrap's finally block re-calls call it, so the second call must be a clean no-op not a re-drain or re-stop
stop() as a safety net. The second call must be a clean no-op not a of already-stopped agents guarded by ``_stopped``."""
re-drain, not a re-stop of already-stopped agents guarded by ``_stopped``."""
orch = _make_orchestrator() orch = _make_orchestrator()
real_drain = orch._drain_bg_tasks real_drain = orch._drain_bg_tasks
drain_calls = 0 drain_calls = 0
@@ -568,10 +568,9 @@ def _stop_agent_patches(orch: AgentOrchestrator) -> Any:
async def test_stop_agent_releases_claim_when_release_claim_true() -> None: async def test_stop_agent_releases_claim_when_release_claim_true() -> None:
"""F120: stop_agent(release_claim=True) hands the agent's claimed task back """stop_agent(release_claim=True) releases the agent's claimed task to the
to the pool immediately. A SIGTERM/budget-kill mid-verb otherwise leaves pool immediately, so a mid-verb SIGTERM/budget-kill doesn't strand the task
the task CLAIMED/IN_PROGRESS with no running agent for up to CLAIMED/IN_PROGRESS until the reaper's heartbeat TTL expires."""
stale_claim_reap_seconds (the reaper's heartbeat TTL)."""
orch = _make_orchestrator() orch = _make_orchestrator()
instance = _make_instance(_AGENT_ID) instance = _make_instance(_AGENT_ID)
instance.current_task_id = str(uuid4()) instance.current_task_id = str(uuid4())
@@ -631,10 +630,9 @@ async def test_stop_agent_does_not_release_claim_by_default() -> None:
async def test_stop_agent_skips_release_for_provider_parked_agent() -> None: async def test_stop_agent_skips_release_for_provider_parked_agent() -> None:
"""F120: a provider-parked agent (rate_limit_lifted WaitingRecord) must NOT """A provider-parked agent (rate_limit_lifted WaitingRecord) must NOT have
have its claim released even when release_claim=True. The probe-resume loop its claim released even when release_claim=True the probe-resume loop
owns its recovery and the claim must survive so probe-success revives the revives the SAME agent on the SAME task, so reaping would lose the claim."""
SAME agent on the SAME task reaping would let another agent claim it."""
orch = _make_orchestrator() orch = _make_orchestrator()
instance = _make_instance(_AGENT_ID) instance = _make_instance(_AGENT_ID)
instance.current_task_id = str(uuid4()) instance.current_task_id = str(uuid4())
@@ -108,11 +108,9 @@ async def test_clean_output_is_not_overload(
async def test_detects_overload_marker_in_transcript( async def test_detects_overload_marker_in_transcript(
orch: AgentOrchestrator, monkeypatch: pytest.MonkeyPatch orch: AgentOrchestrator, monkeypatch: pytest.MonkeyPatch
) -> None: ) -> None:
# F036: the SDK server writes model-API errors to /tmp/sdk-server.log, not # The overload marker may appear only in the durable Claude transcript, not
# stdout, so the overload marker (529/500/503) may appear only in the durable # stdout; without reading it an overload is missed and the agent
# Claude transcript — exactly the rationale already applied to the # crash-respawns straight back into it.
# session-limit detector. Without reading the transcript here an overload
# is missed and the agent crash-respawns straight back into it.
monkeypatch.setattr(settings, "overload_break_enabled", True) monkeypatch.setattr(settings, "overload_break_enabled", True)
monkeypatch.setattr(orch, "_tail_container_logs", AsyncMock(return_value="")) monkeypatch.setattr(orch, "_tail_container_logs", AsyncMock(return_value=""))
monkeypatch.setattr( monkeypatch.setattr(
@@ -128,10 +126,9 @@ async def test_detects_overload_marker_in_transcript(
async def test_agent_writing_about_error_500_does_not_park( async def test_agent_writing_about_error_500_does_not_park(
orch: AgentOrchestrator, monkeypatch: pytest.MonkeyPatch orch: AgentOrchestrator, monkeypatch: pytest.MonkeyPatch
) -> None: ) -> None:
# F037: an agent that merely writes about an HTTP error code in its own # An agent merely writing about an HTTP error code in its own notes must NOT
# notes ("the endpoint returned error 500, retrying") must NOT trip the # trip the detector and park the whole fleet — markers must be specific to
# overload detector and park the whole Anthropic fleet. Markers must be # the API error formatter, not bare "error NNN".
# specific to the API error formatter, not bare "error NNN".
monkeypatch.setattr(settings, "overload_break_enabled", True) monkeypatch.setattr(settings, "overload_break_enabled", True)
agent_note = ( agent_note = (
"be-dev-1: the /health endpoint returned error 500 on retry; " "be-dev-1: the /health endpoint returned error 500 on retry; "
@@ -203,11 +200,10 @@ async def test_park_offlines_and_activates_with_kind(
async def test_park_registers_waiting_record_so_probe_can_resume( async def test_park_registers_waiting_record_so_probe_can_resume(
orch: AgentOrchestrator, monkeypatch: pytest.MonkeyPatch orch: AgentOrchestrator, monkeypatch: pytest.MonkeyPatch
) -> None: ) -> None:
# F035: the probe-resume loop reads _waiting_records filtered by # The probe-resume loop reads _waiting_records filtered by
# waiting_for == "rate_limit_lifted" + context.provider. Without a record # waiting_for == "rate_limit_lifted" + context.provider; without a record
# here, _parked_agents_for(provider) returns [] and _on_probe_success # here recovery falls to the 600s stale-claim reaper instead of the
# resumes nobody — recovery falls to the 600s stale-claim reaper instead of # probe-success path.
# the probe-success path the parking design relies on.
orch._waiting_records = {} orch._waiting_records = {}
inst = _instance() inst = _instance()
inst.current_task_id = "task-1" inst.current_task_id = "task-1"
@@ -232,7 +228,7 @@ async def test_park_registers_waiting_record_so_probe_can_resume(
async def test_probe_success_respawns_parked_agent( async def test_probe_success_respawns_parked_agent(
orch: AgentOrchestrator, monkeypatch: pytest.MonkeyPatch orch: AgentOrchestrator, monkeypatch: pytest.MonkeyPatch
) -> None: ) -> None:
# F035: once the probe succeeds, the parked agent must be respawned via # Once the probe succeeds, the parked agent must be respawned via
# resolve_wait — not left stranded for the 600s reaper. # resolve_wait — not left stranded for the 600s reaper.
orch._waiting_records = { orch._waiting_records = {
"be-dev-1": WaitingRecord( "be-dev-1": WaitingRecord(
+5 -7
View File
@@ -570,13 +570,11 @@ class TestCEONotificationThreshold:
class TestOrphanProviderFallback: class TestOrphanProviderFallback:
"""F045: an activate() failure in the in-verb ``i_am_blocked(rate_limited)`` """An activate() failure in the ``i_am_blocked(rate_limited)`` path parks
path leaves agents parked in ``_waiting_records`` but the provider never agents in ``_waiting_records`` without entering the tracker, so the
makes it into the tracker so the tracker-driven loop never probes it and tracker-driven loop never probes them. The sweep must scan the in-memory
the parked agents strand in WAITING_LONG forever. The sweep must scan the records for any ``rate_limit_lifted`` provider the tracker missed and probe
in-memory records for any ``rate_limit_lifted`` provider the tracker-listed it via the time-expiry fallback so ``_on_probe_success`` can resume them.
set did NOT cover and probe it via the time-expiry fallback so
``_on_probe_success`` can resume them.
""" """
async def test_orphan_parked_agent_resumed_when_tracker_lacks_provider( async def test_orphan_parked_agent_resumed_when_tracker_lacks_provider(

Some files were not shown because too many files have changed in this diff Show More