Fix: agent workflow hardening (#70)

* fix(gateway): push the branch before QA handoff so reviewers see the latest commits

The commit content tool commits locally without pushing; only open_pr pushed
the branch. On the first submission that was fine, but a fix committed while
addressing needs_revision never reached origin (open_pr is skipped once the PR
exists), so QA — which reviews the remote PR branch — re-reviewed the stale
remote and re-failed the task on every cycle, a loop that never converged.

i_am_done now pushes the task branch (idempotent; a no-op when nothing is
unpushed) as part of the shared submit gate, covering both the normal and
resume-from-verifying paths. A push failure blocks the handoff with a clear
remediation rather than parking the task in awaiting_qa with commits that exist
only in the developer's local workspace.

* fix(orchestrator): don't reap a stale claim while the agent's container is alive

The stale-claim reaper released any claimed/in_progress task whose
last_heartbeat_at exceeded the TTL. The heartbeat only updates on certain
gateway calls, so a developer deep in a long edit/test cycle outran the TTL and
had its claim reaped mid-work — churning the task and risking a double spawn
against the still-running container.

The reaper now skips a task whose assignee still holds a live (ACTIVE) agent
instance, trusting container liveness — the ground truth — over the heartbeat
proxy. The check is defensive on missing fields so a heartbeat-only caller (and
the reaper's existing unit tests) behave exactly as before.

* fix(gateway): refuse to unblock a task while a dependency is unfinished

A PM unblock on a dependency-gated task moved it straight to in_progress,
overriding the dependency — letting a dependent proceed without its upstream's
work (e.g. a frontend task built before its UX design lands). A dependency
block is meant to clear on its own via _unblock_dependents the moment the
upstream reaches a terminal state.

unblock now refuses while any dependency is still non-terminal, returning a
clear remediation that the block resolves automatically. Manual unblock remains
available for genuine, non-dependency blockers.

* fix(gateway): release a dependency-blocked claim to pending instead of looping

A task that reached claimed/in_progress with an unfinished dependency was left
in that state when the claim guard rejected, so the orchestrator's respawn loop
kept reviving its assignee — which could make no progress — burning work for
nothing.

The claim guard now releases such a task back to pending. claimed -> blocked is
not a legal transition, so pending — held by the dispatch dependency filter — is
the lifecycle-correct resting state: the respawn loop ignores pending tasks, and
_unblock_dependents re-dispatches it once the upstream reaches a terminal state.
release_dependency_blocked_claim shares a _force_unclaim_to_pending core with
unclaim_for_reaper so both record a truthful work-session abandon reason.

* feat(security): warn at startup in header-trust mode + document the auth posture

When ROBOCO_AGENT_AUTH_REQUIRED is not enabled the API accepts the X-Agent-Id /
X-Agent-Role headers without a signed token, so any client that can reach it may
act as any role (including 'ceo'). The API now logs a clear warning at startup
in this mode, and the README gains a Security section documenting the auth
posture and how to harden it. Acceptable only on a trusted private network — do
not expose the API to untrusted networks.

* fix(workspace): scope the refresh fetch to current + default branch

ensure_workspace's healthy short-circuit ran an all-refs 'git fetch origin' to
keep every origin/<branch> ref current. On a monorepo with many accumulated
feature/* branches that exceeds the refresh timeout, the fetch silently fails,
and the workspace keeps a stale base — so an agent builds on an out-of-date
branch.

The refresh now fetches only the workspace's current branch and the repo's
default branch (resolved via origin/HEAD), with --no-tags --prune: it transfers
near-nothing and can't time out. Readers need their own branch and the default;
the integration branch is refreshed at branch-creation time.

* fix(git): refresh a dependency-blocked task's branch off the current integration tip

A cross-cell dependent (e.g. a frontend task waiting on the UX design) was
branched off a base captured before its upstream merged into the integration
branch, and the branch was never re-synced — so the agent built on a stale
snapshot with none of the upstream's work.

Two changes close the gap:
- release_dependency_blocked_claim now clears branch_name, so the re-claim
  (after the dependency clears) re-runs branch creation.
- create_branch, when the branch is already on disk with no commits of its own,
  resets it onto the freshly-pulled base — the dependent now builds on the
  current integration tip. A branch carrying real commits is left untouched, so
  no work is discarded; the cell->leaf cascade carries the upstream down to the
  dev branch automatically.

* refactor(gateway): drop the sibling-sequence claim guard

Sibling sequence no longer gates a claim. Cross-cell ordering is
enforced by task dependencies — a cell task that depends on another is
held until its upstream reaches a terminal state, a stronger,
status-aware gate than the sequence-number check. That check was
dormant in practice anyway: every fan-out child carries sequence 0, on
which the guard short-circuited. `sequence` stays a sibling-ordering /
dispatch-priority field (list_pending ordering and the panel).

Removes sibling_sequence_guard and its _earlier_blocking_sibling
helper, the now-unused skip_sequence parameter threaded through the
claim verbs, and the sibling fetch that fed it.

* feat(gateway): sort a cross-cell dependent after its upstream

When the frontend cell task is wired to depend on its UX/UI sibling, set
its sequence to the upstream's sequence + 1 so it sorts after the design
it waits on — list_pending ordering and the panel now show UX ahead of
the implementation it gates, in either delegation order.

Adds TaskService.set_sequence (the sibling-ordering field is a service
write; it carries no claim-gating semantics — dependencies gate claims).

* feat(gateway): make the backend cell depend on UX too

UX/UI design defines the screens and API contracts both implementation
cells build against, so the backend cell — not just the frontend — waits
on the UX/UI cell task in a product fan-out and sorts after it. Wires in
either delegation order: a backend task delegated after UX gets the
dependency directly; a UX task delegated after a still-pending backend
sibling retro-wires it.

Mirrors the existing frontend wiring (_depend_backend_on_ux and
_depend_pending_backends_on_ux). Backend is held by the same dependency
gate, so it costs no extra dispatch churn.

* fix(websocket): forward notification acks instead of logging them incomplete

The bridge handler serves both notification.sent and notification.acked,
but acked events carry `agent_id` (the acking agent) rather than
`recipient_id`, so every acknowledgement tripped the missing-field guard
and logged "Incomplete notification event" instead of reaching the panel.
Accept either field as the recipient.

* feat(api): hint the full UUID when a truncated task id fails validation

Agents copy the 8-character task prefix the system shows them (the commit
prefix, task summaries) and send it as task_id, which fails UUID
validation with an opaque "invalid length" 422 and wastes a call. The
request-validation handler now detects a task_id UUID error and attaches
a `remediate` hint telling the agent to retry with the full 36-character
UUID from its task envelope.

* fix(audit): record the blocked transition when a task is escalated

Escalation sets a task to blocked by writing task.status directly, which
bypassed the validated transition helper and so never emitted a
task.blocked audit row — the lifecycle moved but the Auditor saw nothing.
Extract the audit emit from the central transition helper into
_emit_status_transition_audit and call it from the escalate path,
capturing the prior status and outgoing owner before reassignment so the
row is attributed correctly.

* fix(docs): stop doubling the docs path so design specs index into RAG

The documenter sometimes hands a doc path already rooted at docs/, and
joining it onto DOCS_BASE_PATH (/app/docs) produced /app/docs/docs/...,
so the file was never found and the spec never indexed — the frontend
cell could not retrieve the UX design over RAG. Normalize the path
before joining: trust an absolute path, otherwise strip a single
redundant leading docs/ segment.

* feat(security): let the control panel authenticate in secure mode

With ROBOCO_AGENT_AUTH_REQUIRED=true every request must carry a valid
HMAC token, which locked the human control panel out — it sends role
headers but no token. nginx, the only trusted hop between the browser
and the API, now injects the CEO token on /api and /ws, so the browser
never holds the signing secret. The injected value is just the existing
per-agent token issued for the CEO identity (issue_panel_token), so the
token-verification path is unchanged. An empty value (dev/header-trust
mode) renders to no header.

`make panel-token` prints the value; set it as ROBOCO_PANEL_AGENT_TOKEN
in .env before enabling secure mode. .env.example and the README
Security section document the flow.

* chore(compose): consolidate the two compose files into one

docker-compose.yml and docker-compose.yaml had diverged: .yml — the file
Docker actually uses — carried ROBOCO_PUBLIC_BASE_URL but was missing the
/app/manifests bind-mount, while .yaml had the manifests mount but not
the base URL. Merge the union into docker-compose.yml and delete the
duplicate so there is one source of truth and no "multiple config files"
warning.

This activates the manifests mount in the deployed file: without it the
orchestrator writes per-agent tool manifests to its ephemeral container
fs, they never reach the host for the daemon to bind-mount, and agents
fall back to all-verbs registration. Drop the stale .yaml reference from
the config.py docstring, the labeler, and the CI path filters.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
2026-06-05 16:35:22 +02:00
committed by GitHub
co-authored by Renn F
parent 97533f769b
commit 06682f33c6
40 changed files with 1300 additions and 653 deletions
+15 -1
View File
@@ -33,7 +33,7 @@ from typing import Final
from roboco.foundation import identity as _foundation
from roboco.foundation.policy import communications as _comms
from roboco.models.base import NotificationPriority, NotificationType
from roboco.seeds.initial_data import AGENT_UUIDS
from roboco.seeds.initial_data import AGENT_UUIDS, CEO_AGENT_ID
# Env var containing the HMAC secret used to sign agent auth tokens.
# Must be set in orchestrator + API container environments; if missing,
@@ -91,6 +91,20 @@ def verify_agent_token(token: str, agent_id: str, role: str, team: str = "") ->
return hmac.compare_digest(expected, token)
def issue_panel_token() -> str:
"""Mint the token the control panel presents to act as the CEO.
The panel calls the API as the CEO identity — ``X-Agent-Id`` = the CEO
UUID, ``X-Agent-Role`` = ``ceo``, and no team header — so the token is
signed for exactly those values (empty team). In secure mode nginx injects
it as ``X-Agent-Token`` so the browser never holds the signing secret;
this is just the existing per-agent token issued for the CEO identity, so
the verification path is unchanged. Returns ``UNSIGNED`` when the secret is
unset (same fail-closed contract as ``issue_agent_token``).
"""
return issue_agent_token(CEO_AGENT_ID, "ceo", "")
# Reverse mapping: UUID -> slug (computed from seeds)
_UUID_TO_SLUG: Final[dict[str, str]] = {
uuid: slug for slug, uuid in AGENT_UUIDS.items()
+11
View File
@@ -11,6 +11,7 @@ from contextlib import asynccontextmanager
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from roboco.api.deps import _auth_required
from roboco.api.middleware import setup_middleware
from roboco.api.routes.a2a import router as a2a_router
from roboco.api.routes.a2a import wellknown_router as a2a_wellknown_router
@@ -76,6 +77,16 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None]:
environment=settings.environment,
)
if not _auth_required():
logger.warning(
"Agent auth is in HEADER-TRUST mode (ROBOCO_AGENT_AUTH_REQUIRED is "
"not set to true): the API accepts X-Agent-Id / X-Agent-Role without "
"verifying a signed token, so any client that can reach it may act as "
"any role, including 'ceo'. Acceptable only on a trusted private "
"network. Set ROBOCO_AGENT_AUTH_REQUIRED=true and do NOT expose this "
"API to untrusted networks.",
)
# Startup: apply Alembic migrations (+ create_all fallback for fresh DBs).
# init_db runs on every environment now — migrations are idempotent via
# alembic_version, and this is the only way new schema (e.g. enum value
+35 -4
View File
@@ -6,8 +6,8 @@ Request/response middleware for logging, error handling, and correlation IDs.
import time
import uuid
from collections.abc import Callable
from typing import cast
from collections.abc import Callable, Sequence
from typing import Any, cast
import structlog
from fastapi import FastAPI, HTTPException, Request, Response
@@ -303,6 +303,29 @@ async def http_exception_handler(request: Request, exc: Exception) -> JSONRespon
# =============================================================================
def _uuid_field_remediation(errors: Sequence[Any]) -> str | None:
"""Spell out the fix when a truncated id is sent where a UUID is required.
Agents routinely copy the 8-character task prefix the system shows them
(e.g. the ``[cee99ecc]`` commit prefix) and send it as ``task_id``, which
fails UUID validation with an opaque "invalid length" message and wastes a
call. Detect that case and hand back an actionable remediation instead.
"""
for err in errors:
if not isinstance(err, dict):
continue
loc = err.get("loc") or ()
field = loc[-1] if loc else None
if field == "task_id" and "uuid" in str(err.get("type", "")).lower():
return (
"Use the FULL 36-character task UUID, not the 8-character short "
"form shown in commit prefixes or summaries. The full id is in "
"the `task_id` field of the envelope returned by give_me_work "
"or your most recent verb."
)
return None
async def request_validation_handler(request: Request, exc: Exception) -> JSONResponse:
"""Log the rejected body before returning the standard 422 response.
@@ -310,19 +333,27 @@ async def request_validation_handler(request: Request, exc: Exception) -> JSONRe
nothing lands in server logs. During smoke tests this leaves us
blind to which field actually broke. Log the body + the per-field
errors so the next 422 is debuggable in one log scan.
When the failure is a truncated ``task_id`` (the recurring agent mistake),
add a ``remediate`` hint so the agent knows to retry with the full UUID.
"""
rve = cast("RequestValidationError", exc)
body = rve.body if isinstance(rve.body, str | bytes | dict | list) else None
errors = rve.errors()
logger.warning(
"Request validation failed",
path=request.url.path,
method=request.method,
body=body,
errors=rve.errors(),
errors=errors,
)
content: dict[str, Any] = {"detail": errors, "body": body}
remediate = _uuid_field_remediation(errors)
if remediate is not None:
content["remediate"] = remediate
return JSONResponse(
status_code=http_status.HTTP_422_UNPROCESSABLE_ENTITY,
content={"detail": rve.errors(), "body": body},
content=content,
)
+5 -1
View File
@@ -22,7 +22,11 @@ async def _handle_notification_sent(event: Event) -> None:
data = event.data
notification_id_str = data.get("notification_id")
recipient_id_str = data.get("recipient_id")
# SENT events carry `recipient_id`; ACKED events carry `agent_id` (the
# agent who acknowledged). This handler serves both, so accept either —
# otherwise every acknowledgement logged a spurious "Incomplete
# notification event" and never reached the panel.
recipient_id_str = data.get("recipient_id") or data.get("agent_id")
notification_type = data.get("type", "unknown")
subject = data.get("subject", "")
priority = data.get("priority", "normal")
+1 -1
View File
@@ -323,7 +323,7 @@ class Settings(BaseSettings):
description=(
"Orchestrator-side directory where per-agent tool manifests are "
"written. Must be a path that's bind-mounted from the host "
"(see docker-compose.yaml) so the docker daemon can in turn mount "
"(see docker-compose.yml) so the docker daemon can in turn mount "
"the file into spawned agent containers as /app/tool-manifest.json."
),
)
+26 -1
View File
@@ -3990,12 +3990,35 @@ Start now: evidence(task_id="{task_id}")
await self._reap_with_service(svc)
await db.commit()
def _assignee_has_active_instance(self, task: Any) -> bool:
"""True if the task's assignee currently holds a live (ACTIVE) container.
The heartbeat only approximates liveness. A developer deep in an
edit/test cycle can go longer than the heartbeat TTL between gateway
calls, so a heartbeat-only reaper releases claims out from under agents
that are alive and working churning the task (and risking a double
spawn against the still-running container). The agent-instance registry
is the ground truth; defer to it when present. Defensive on missing
fields so a heartbeat-only caller (and the reaper's own unit tests)
behave exactly as before.
"""
owner = getattr(task, "assigned_to", None) or getattr(task, "claimed_by", None)
if not owner:
return False
instances = getattr(self, "_instances", None)
if not instances:
return False
instance = instances.get(self._resolve_agent_slug(str(owner)))
return instance is not None and instance.state == AgentState.ACTIVE
async def _reap_with_service(self, svc: "TaskService") -> None:
"""Inner reap loop, parameterized by the TaskService to use.
Wraps each ``unclaim_for_reaper`` in try/except so a single bad row
doesn't abort the dispatch tick — the reaper must keep ticking even
if one task's release somehow fails.
if one task's release somehow fails. A claim whose assignee still has
a live container is skipped: the heartbeat is a stale proxy there, and
reaping a working agent only churns the task.
"""
from roboco.utils.converters import require_uuid
@@ -4004,6 +4027,8 @@ Start now: evidence(task_id="{task_id}")
for t in candidates:
ts = t.last_heartbeat_at
if ts is None or ts < cutoff:
if self._assignee_has_active_instance(t):
continue
task_id = require_uuid(t.id)
try:
await svc.unclaim_for_reaper(task_id)
+125 -39
View File
@@ -24,7 +24,6 @@ from roboco.services.gateway.choreographer._verb_runner import VerbRunner
from roboco.services.gateway.claim_guards import (
already_active_guard,
paused_tasks_guard,
sibling_sequence_guard,
unmet_dependency_guard,
)
from roboco.services.gateway.envelope import Envelope
@@ -704,7 +703,6 @@ class Choreographer:
*,
agent_id: UUID,
task: Any,
skip_sequence: bool = False,
) -> Envelope | None:
"""Run concurrency-invariant claim guards. Returns rejection or None.
@@ -714,11 +712,7 @@ class Choreographer:
in the verb's spec gate; the former role-typed and
pm_cannot_execute_code guards have been deleted (Task 27, 2026-05-10).
Pre-gateway location: _helpers.py:124-204 + claim.py:121-180.
``skip_sequence`` lets resumption-of-already-claimed-task call sites
skip the sibling-sequence check (the sequence was already validated
on the original claim).
Pre-gateway location: _helpers.py:124-204.
"""
in_progress = await self.task.list_in_progress_for_agent(agent_id)
if guard := already_active_guard(in_progress, task.id):
@@ -730,26 +724,15 @@ class Choreographer:
if dep_ids:
unmet = await self.task.unmet_dependency_ids(dep_ids)
if guard := unmet_dependency_guard(task, unmet):
return guard
if not skip_sequence:
siblings = await self._fetch_siblings(task)
if guard := sibling_sequence_guard(task, siblings):
# Park the dependency-gated task back to pending so the
# orchestrator stops respawning its assignee (the respawn loop
# targets only claimed/in_progress) and the dispatch dependency
# filter holds it until the upstream completes. No-op unless the
# task is currently claimed/in_progress.
await self.task.release_dependency_blocked_claim(task.id)
return guard
return None
async def _fetch_siblings(self, task: Any) -> list[Any]:
"""Fetch sibling tasks for the sequence-order guard.
Returns ``[]`` when the task has no parent (root task) so the
guard short-circuits. Otherwise returns the parent's subtasks via
``TaskService.get_subtasks``.
"""
parent_id = getattr(task, "parent_task_id", None)
if parent_id is None:
return []
siblings: list[Any] = await self.task.get_subtasks(parent_id)
return siblings
async def _non_terminal_subtask_ids(self, parent_task_id: UUID) -> str:
"""Return a human-readable comma-separated list of non-terminal subtasks.
@@ -833,13 +816,11 @@ class Choreographer:
task_id=task_id,
verb=verb_name,
)
# Concurrency guards still apply (paused / already-active in another
# task). Sibling sequence is skipped on resumption — see
# _run_claim_guards docstring.
# Concurrency guards still apply on resumption (paused / already-active
# in another task).
if guard := await self._run_claim_guards(
agent_id=agent_id,
task=t,
skip_sequence=True,
):
return await self._emit_rejection(
self._with_briefing(guard, briefing).with_introspection(
@@ -899,7 +880,7 @@ class Choreographer:
"""Run all gates for an ``i_will_work_on`` / ``i_will_plan`` call.
Order: spec.can_invoke_intent -> behavioral claim guards
(already_active / paused / sibling_sequence). Any rejection
(already_active / paused / unmet_dependency). Any rejection
short-circuits with the appropriate envelope.
Per-role claim authority (CLAIM_RULES) is enforced inside
@@ -921,7 +902,7 @@ class Choreographer:
# Behavioral pre-flight guards the spec doesn't yet model:
# - already_active: agent has another in_progress task elsewhere
# - paused_tasks: agent has a paused task they should resume first
# - sibling_sequence: an earlier-numbered sibling is still open
# - unmet_dependency: an upstream dependency is still non-terminal
# The role/state/task_type checks already passed via the spec gate
# above. These migrate into spec.extra_preconditions in a later
# task; until then, keep them imperative so concurrency invariants
@@ -1475,7 +1456,10 @@ class Choreographer:
async def _i_am_done_gate(self, ctx: _IAmDoneContext) -> Envelope | None:
"""Run defense-in-depth tracing + field-level gates the spec doesn't model.
Returns the rejection envelope if any gate fails; None on pass.
Also pushes the branch to origin so a task cannot reach awaiting_qa
with commits that exist only in the developer's local workspace.
Returns the rejection envelope if any gate fails; None on pass. Shared
by the normal and resume-from-verifying paths so both push.
"""
if rejection := await self._check_tracing_gates(
ctx.agent_id, ctx.task_id, ctx.task
@@ -1485,12 +1469,38 @@ class Choreographer:
ctx.agent_id, ctx.task_id, ctx.task
):
return await self._reject_i_am_done(ctx, rejection)
if rejection := await self._ensure_branch_pushed(ctx):
return await self._reject_i_am_done(ctx, rejection)
# Wave C5 (2026-05-12) — pre-gateway parity. Persist per-criterion
# status now that all gates have passed. The write runs AFTER the
# verdict so it cannot change i_am_done's rejection behavior.
await self._write_criteria_status(ctx.agent_id, ctx.task_id, ctx.task)
return None
async def _ensure_branch_pushed(self, ctx: _IAmDoneContext) -> Envelope | None:
"""Push the task branch to origin before it reaches awaiting_qa.
QA reviews the remote PR branch. A fix committed during a revision
cycle lives only in the developer's local workspace until pushed —
without this, QA re-reviews the stale remote and fails the same task
every cycle (a non-converging loop). Idempotent: a no-op when nothing
is unpushed, so first-submit (already pushed by open_pr) is unaffected.
"""
try:
await self.git.push_task_branch(ctx.agent_id, ctx.task_id)
except Exception as exc:
return Envelope.invalid_state(
message=f"could not push your branch to origin: {exc}",
remediate=(
"your latest commits are local-only and QA reviews the "
"pushed PR branch. resolve the push error (often a "
"transient network / fetch timeout) and call i_am_done "
"again."
),
context_briefing=ctx.briefing,
)
return None
@staticmethod
def _extract_first_commit_sha(t: Any) -> str | None:
"""Read the first commit sha off the task, dict or model alike."""
@@ -3349,13 +3359,13 @@ class Choreographer:
return value.value if hasattr(value, "value") else str(value)
async def _wire_ux_frontend_dependency(self, new_task: Any, parent: Any) -> None:
"""Cross-cell sequencing: in a product fan-out the FRONTEND cell task
depends on the UX/UI cell task UX design is upstream of frontend
implementation, while backend runs in parallel. Wires the dependency in
either delegation order. A dev/code subtask delegated under a cell task
that is itself still waiting on that dependency inherits it, so the
developer is held until UX is done instead of coding ahead of the
design. Best-effort: never breaks delegate.
"""Cross-cell sequencing: in a product fan-out the implementation cells
(FRONTEND and BACKEND) depend on the UX/UI cell task UX design defines
the screens and API contracts both cells build against, so it is upstream
of implementation. Wires the dependency in either delegation order. A
dev/code subtask delegated under a cell task that is itself still waiting
on that dependency inherits it, so the developer is held until UX is done
instead of coding ahead of the design. Best-effort: never breaks delegate.
"""
if parent is None or getattr(parent, "product_id", None) is None:
return
@@ -3368,11 +3378,14 @@ class Choreographer:
await self.task.inherit_unmet_dependencies(new_task.id, parent.id)
if nt_team == Team.FRONTEND.value:
await self._depend_frontend_on_ux(new_task, parent.id)
elif nt_team == Team.BACKEND.value:
await self._depend_backend_on_ux(new_task, parent.id)
elif nt_team == Team.UX_UI.value:
await self._depend_pending_frontends_on_ux(new_task, parent.id)
await self._depend_pending_backends_on_ux(new_task, parent.id)
except Exception as exc:
logger.warning(
"cross-cell UX->FE sequencing wiring failed",
"cross-cell UX->implementation sequencing wiring failed",
error=str(exc),
parent_task_id=str(getattr(parent, "id", None)),
)
@@ -3396,6 +3409,32 @@ class Choreographer:
)
if ux is not None:
await self.task.add_dependency(fe_task.id, ux.id)
await self.task.set_sequence(
fe_task.id, (getattr(ux, "sequence", 0) or 0) + 1
)
async def _depend_backend_on_ux(self, be_task: Any, parent_id: Any) -> None:
"""Make a new BACKEND cell task wait on its non-terminal UX/UI sibling."""
from roboco.foundation.identity import Team
from roboco.models.base import TaskStatus
terminal = {TaskStatus.COMPLETED, TaskStatus.CANCELLED}
siblings = await self.task.get_subtasks(parent_id)
ux = next(
(
s
for s in siblings
if self._team_value(s.team) == Team.UX_UI.value
and s.id != be_task.id
and s.status not in terminal
),
None,
)
if ux is not None:
await self.task.add_dependency(be_task.id, ux.id)
await self.task.set_sequence(
be_task.id, (getattr(ux, "sequence", 0) or 0) + 1
)
async def _depend_pending_frontends_on_ux(
self, ux_task: Any, parent_id: Any
@@ -3405,6 +3444,7 @@ class Choreographer:
from roboco.models.base import TaskStatus
not_started = {TaskStatus.BACKLOG, TaskStatus.PENDING}
ux_sequence = (getattr(ux_task, "sequence", 0) or 0) + 1
siblings = await self.task.get_subtasks(parent_id)
for fe in siblings:
if (
@@ -3413,6 +3453,26 @@ class Choreographer:
and fe.status in not_started
):
await self.task.add_dependency(fe.id, ux_task.id)
await self.task.set_sequence(fe.id, ux_sequence)
async def _depend_pending_backends_on_ux(
self, ux_task: Any, parent_id: Any
) -> None:
"""Retro-wire not-yet-started BACKEND siblings onto a new UX/UI task."""
from roboco.foundation.identity import Team
from roboco.models.base import TaskStatus
not_started = {TaskStatus.BACKLOG, TaskStatus.PENDING}
ux_sequence = (getattr(ux_task, "sequence", 0) or 0) + 1
siblings = await self.task.get_subtasks(parent_id)
for be in siblings:
if (
self._team_value(be.team) == Team.BACKEND.value
and be.id != ux_task.id
and be.status in not_started
):
await self.task.add_dependency(be.id, ux_task.id)
await self.task.set_sequence(be.id, ux_sequence)
async def _resolve_subtask_project(
self, parent: Any, inputs: DelegateInputs
@@ -3901,6 +3961,32 @@ class Choreographer:
verb="unblock",
)
# A dependency block must not be cleared by hand. It auto-clears via
# _unblock_dependents the moment its last dependency reaches a terminal
# state; forcing it now would let the dependent proceed without the
# upstream's work (e.g. a frontend task built before its UX design lands).
dep_ids = list(t.dependency_ids or [])
unmet = await self.task.unmet_dependency_ids(dep_ids) if dep_ids else []
if unmet:
return await self._emit_rejection(
Envelope.invalid_state(
message=(
f"task {task_id} still depends on {len(unmet)} "
"unfinished task(s); a dependency block clears on its "
"own once the upstream work completes"
),
remediate=(
"don't force this — let the dependency finish; the task "
"auto-unblocks the moment its last dependency reaches "
"completed/cancelled"
),
context_briefing=await self._briefing_for(pm_agent_id, task_id),
).with_introspection(task=t, role=role),
agent_id=pm_agent_id,
task_id=task_id,
verb="unblock",
)
if env := await self._check_pm_decision_required(
"unblock", pm_agent_id, task_id, t
):
@@ -72,7 +72,6 @@ class ChoreographerHelpers:
*,
agent_id: UUID,
task: Any,
skip_sequence: bool = False,
) -> Envelope | None:
raise NotImplementedError
@@ -178,7 +178,6 @@ class DocMixin(_Base):
guard = await self._run_claim_guards(
agent_id=doc_agent_id,
task=t,
skip_sequence=True,
)
if guard:
guard.with_introspection(task=t, role=role_str)
@@ -152,7 +152,6 @@ class QAMixin(_Base):
guard = await self._run_claim_guards(
agent_id=qa_agent_id,
task=t,
skip_sequence=True,
)
if guard:
guard.with_introspection(task=t, role=role_str)
-51
View File
@@ -13,7 +13,6 @@ through ``spec.can_invoke_action``'s CLAIM_RULES + ``ActionSpec
Pre-gateway location at commit 0c3d15a:
roboco/mcp/tasks/handlers/_helpers.py:124-204
roboco/mcp/tasks/handlers/claim.py:121-180 (sibling sequence)
"""
from __future__ import annotations
@@ -31,10 +30,6 @@ _ACTIVE_BLOCKING_STATUSES: frozenset[str] = frozenset(
{"claimed", "in_progress", "verifying"}
)
# Terminal statuses that satisfy the sibling-sequence check —
# pre-gateway claim.py:153.
_TERMINAL_STATUSES: frozenset[str] = frozenset({"completed", "cancelled"})
def already_active_guard(
in_progress_tasks: list[Any], target_task_id: UUID
@@ -107,49 +102,3 @@ def unmet_dependency_guard(
"completed/cancelled before claiming this task"
),
)
def _earlier_blocking_sibling(
target_task: Any, siblings: list[Any], my_sequence: int
) -> Any | None:
"""Return the first non-terminal sibling with a lower sequence, else None."""
for sib in siblings:
if sib.id == target_task.id:
continue
sib_seq = getattr(sib, "sequence", 0) or 0
sib_status = str(getattr(sib, "status", ""))
if sib_seq < my_sequence and sib_status not in _TERMINAL_STATUSES:
return sib
return None
def sibling_sequence_guard(target_task: Any, siblings: list[Any]) -> Envelope | None:
"""Refuse claim if any earlier-sequence sibling is non-terminal.
Pre-gateway: claim.py:_validate_sibling_sequence 121-180.
A task with sequence=N is blocked while any sibling with sequence<N is
not in (completed, cancelled). Tasks without a parent_task_id (root) or
sequence==0 (first in line) are always allowed.
"""
parent_id = getattr(target_task, "parent_task_id", None)
if parent_id is None:
return None
my_sequence = getattr(target_task, "sequence", 0) or 0
if my_sequence == 0:
return None
blocker = _earlier_blocking_sibling(target_task, siblings, my_sequence)
if blocker is None:
return None
sib_seq = getattr(blocker, "sequence", 0) or 0
sib_status = str(getattr(blocker, "status", ""))
return Envelope.invalid_state(
message=(
f"sequence {my_sequence} blocked: earlier sibling "
f"{blocker.id} (sequence {sib_seq}) is in {sib_status}"
),
remediate=(
f"wait for sibling {blocker.id} (sequence {sib_seq}) to "
"reach completed/cancelled before claiming this task"
),
)
+36
View File
@@ -848,6 +848,22 @@ class GitService(BaseService):
)
if created.returncode != 0:
await self._run_git(workspace, ["checkout", branch_name])
# The branch already existed on disk. If it carries no commits of
# its own — a dependency-blocked task branched before its upstream
# merged into the integration branch, then released and re-claimed —
# re-point it at the freshly-pulled base so the agent builds on the
# current integration tip, not a stale snapshot. Guarded on "no
# commits unique to the branch": a branch with real work is left
# exactly as-is.
unique = await self._run_git(
workspace,
["rev-list", "--count", f"{base_branch}..{branch_name}"],
check=False,
)
if unique.returncode == 0 and unique.stdout.strip() == "0":
await self._run_git(
workspace, ["reset", "--hard", base_branch], check=False
)
await self._run_git(
workspace,
["push", "-u", "origin", branch_name],
@@ -1065,6 +1081,26 @@ class GitService(BaseService):
return await self.push(workspace, getattr(data, "force", False))
async def push_task_branch(self, agent_id: UUID, task_id: UUID) -> int:
"""Idempotently push a task's branch to origin; return commits pushed.
Reviewers see the remote PR branch, not the developer's workspace. A
fix committed during a revision cycle lives only in that local clone
until it is pushed so without an explicit push at the QA-submission
boundary, QA re-reviews the stale remote and fails the same task on
every cycle. Self-resolves the project/workspace from the task so the
choreographer can call it with just (agent, task). A no-op when there
is nothing unpushed; raises typed service errors on a real failure.
"""
task = await self._assert_task_owned_with_branch(task_id, agent_id)
project = await self._project_for_task(task)
if project is None:
return 0
workspace = await self.get_workspace(project.slug, agent_id)
await self._assert_on_task_branch(workspace, task.branch_name)
_branch, pushed = await self.push(workspace)
return pushed
# =========================================================================
# PR METHODS
# =========================================================================
+141 -23
View File
@@ -407,22 +407,44 @@ class TaskService(BaseService):
error=str(e),
)
# Fire-and-forget audit write. Critical: we must hold a strong
# reference to the Task object (via `_background_tasks`) — the event
# loop only weak-refs tasks, so without this the audit write can be
# garbage-collected before it commits. That's why audit_log was
# coming up empty even though the log call ran.
self._emit_status_transition_audit(
task,
from_status=current,
to_status=target,
agent_role=agent_role,
audit_agent_id=audit_agent_id,
)
def _emit_status_transition_audit(
self,
task: TaskTable,
*,
from_status: str,
to_status: str,
agent_role: str | None,
audit_agent_id: str | UUID | None,
) -> None:
"""Emit the ``task.<status>`` audit row for a status transition.
Extracted from ``_validate_and_set_status`` so transition paths that
set ``task.status`` directly e.g. ``apply_escalation``, which blocks a
task without routing through the strict transition validator record
the same audit event. No status change may bypass the audit log.
Fire-and-forget, but we hold a strong reference to the background task
(via ``_background_tasks``): the event loop only weak-refs tasks, so
without it the audit write can be garbage-collected before it commits.
The explicit ``audit_agent_id`` (capture-before-mutate) wins: callers
like ``submit_for_qa`` clear ``task.claimed_by`` before transitioning
but still want the row attributed to the outgoing agent. Otherwise fall
back to ``task.claimed_by``.
"""
import asyncio
import contextlib
from roboco.services.audit import get_audit_service
# Prefer the explicit `audit_agent_id` when the caller passed one
# (capture-before-mutate pattern: callers like `submit_for_qa` and
# `pass_qa` clear `task.claimed_by` BEFORE calling us so the next
# role can claim, but still want the audit row attributed to the
# outgoing agent). Fall back to `task.claimed_by` for transitions
# where the assignment didn't change (claim, start_work, etc.).
if audit_agent_id is not None:
resolved_audit_agent_id: str | None = str(audit_agent_id)
elif task.claimed_by is not None:
@@ -434,12 +456,12 @@ class TaskService(BaseService):
with contextlib.suppress(RuntimeError):
bg = asyncio.get_running_loop().create_task(
audit.log_task_event(
event_type=f"task.{target}",
event_type=f"task.{to_status}",
task_id=str(task.id),
agent_id=resolved_audit_agent_id,
details={
"from_status": current,
"to_status": target,
"from_status": from_status,
"to_status": to_status,
"agent_role": agent_role,
"team": (
task.team.value
@@ -1697,13 +1719,33 @@ class TaskService(BaseService):
error=str(e),
)
@staticmethod
def _resolve_doc_abspath(rel_path: str) -> str:
"""Resolve a documenter-supplied doc path to its on-disk absolute path.
Docs live under ``DOCS_BASE_PATH`` (``/app/docs``). Agents sometimes
hand a path already rooted at ``docs/`` (or an absolute path); joining
``DOCS_BASE_PATH`` with a ``docs/``-prefixed relative path doubles the
segment (``/app/docs/docs/...``), so the file is never found and the
docs never index into RAG. Normalize: trust an absolute path; otherwise
strip a single redundant leading ``docs/`` before joining.
"""
from pathlib import Path
from roboco.services.docs import DOCS_BASE_PATH
path = Path(rel_path)
if path.is_absolute():
return str(path)
parts = path.parts
if parts and parts[0] == DOCS_BASE_PATH.name:
path = Path(*parts[1:]) if len(parts) > 1 else Path()
return str(DOCS_BASE_PATH / path)
async def _index_docs_background(
self, task_id: UUID, documents: list[dict[str, Any]]
) -> None:
"""Index documentation from completed doc task (fire-and-forget)."""
from pathlib import Path
from roboco.services.docs import DOCS_BASE_PATH
from roboco.services.optimal import get_optimal_service
try:
@@ -1714,8 +1756,7 @@ class TaskService(BaseService):
for d in documents:
rel_path = d.get("path")
if rel_path:
absolute_path = str(DOCS_BASE_PATH / Path(rel_path))
doc_paths.append(absolute_path)
doc_paths.append(self._resolve_doc_abspath(rel_path))
if doc_paths:
count = await optimal.index_documentation(doc_paths, project="roboco")
@@ -2074,24 +2115,65 @@ class TaskService(BaseService):
ownership/role checks because the holder is provably dead (no
heartbeat past TTL).
"""
await self._force_unclaim_to_pending(task_id, reason="reaper-unclaim")
async def release_dependency_blocked_claim(self, task_id: UUID) -> None:
"""Release a claimed/in_progress task whose dependency is still unmet.
A task assigned with an unfinished dependency cannot proceed, but while
it sits claimed/in_progress the orchestrator keeps respawning its
assignee (the respawn loop targets only claimed/in_progress). Releasing
it to pending stops that churn: the dispatch dependency filter holds it
un-spawned, and ``_unblock_dependents`` clears the dependency once the
upstream completes so it re-dispatches on its own. ``claimed -> blocked``
is not a legal transition, so pending (held by the dependency filter) is
the lifecycle-correct resting state. No-op when not in a releasable state.
Also forgets ``branch_name`` so the eventual re-claim re-runs branch
creation and cuts the branch fresh off the current integration tip
which by then includes the upstream's merged work — instead of reusing a
snapshot taken before the dependency landed. A dependency-blocked task
has done no work of its own, so nothing is lost; ``create_branch``
leaves any branch carrying real commits intact.
"""
if not await self._force_unclaim_to_pending(task_id, reason="dependency-unmet"):
return
task = await self.get(task_id)
if task is not None and task.branch_name:
task.branch_name = None
await self.session.flush()
async def _force_unclaim_to_pending(self, task_id: UUID, *, reason: str) -> bool:
"""Force a claimed/in_progress task back to pending (system action).
Shared core of ``unclaim_for_reaper`` and
``release_dependency_blocked_claim``. Routes through
``_validate_and_set_status`` so the state machine records the
transition, clears assignee/heartbeat/claimant, and abandons the active
WorkSession (best-effort, tagged with ``reason``) so a re-claim doesn't
trip the uniqueness constraint. Bypasses ownership/role checks the
system itself is performing the transition. Returns True iff the task
was actually released (False when missing or not in a releasable state).
"""
task = await self.get(task_id)
if task is None:
return
return False
if task.status not in (TaskStatus.CLAIMED, TaskStatus.IN_PROGRESS):
return
return False
try:
self._validate_and_set_status(task, TaskStatus.PENDING, None)
except TaskLifecycleError:
return
return False
if task.work_session_id:
await self._abandon_work_session_best_effort(
task.work_session_id, reason="reaper-unclaim"
task.work_session_id, reason=reason
)
task.work_session_id = cast("Any", None)
task.assigned_to = cast("Any", None)
task.last_heartbeat_at = None
task.active_claimant_id = cast("Any", None)
await self.session.flush()
return True
async def _abandon_work_session_best_effort(
self, session_id: Any, *, reason: str
@@ -3346,6 +3428,15 @@ class TaskService(BaseService):
return
if task.assigned_to and not task.blocker_raised_by:
task.blocker_raised_by = cast("Any", task.assigned_to)
# Capture before mutating: the audit row must record the real prior
# status and attribute the block to the outgoing owner, not the
# escalation target we are about to assign.
pre_block_status = (
task.status.value
if isinstance(task.status, TaskStatus)
else str(task.status)
)
pre_block_owner = cast("Any", task.claimed_by)
task.assigned_to = cast("Any", target_agent_id)
task.claimed_by = cast("Any", target_agent_id)
task.status = TaskStatus.BLOCKED
@@ -3355,6 +3446,16 @@ class TaskService(BaseService):
)
task.dev_notes = existing_notes + escalation_note
await self.session.flush()
# This path sets BLOCKED directly (bypassing the strict transition
# validator), so emit the task.blocked audit explicitly — no status
# change may skip the audit log.
self._emit_status_transition_audit(
task,
from_status=pre_block_status,
to_status=TaskStatus.BLOCKED.value,
agent_role=None,
audit_agent_id=pre_block_owner,
)
self.log.info(
"Task escalated and blocked",
task_id=str(task.id),
@@ -4253,6 +4354,23 @@ class TaskService(BaseService):
task.dependency_ids = [*task.dependency_ids, depends_on_id]
await self.session.flush()
async def set_sequence(self, task_id: UUID, sequence: int) -> None:
"""Set a task's sibling-ordering sequence (lower = first).
`sequence` is a display / dispatch-priority field only it orders
siblings in `list_pending`, `list_for_team`, and the panel and carries
no claim-gating semantics (dependencies gate claims). Cross-cell
fan-out uses it so an upstream design task sorts ahead of the
implementation tasks that depend on it. No-op if the task is gone or
already at `sequence`.
"""
task = await self.get(task_id)
if task is None:
return
if task.sequence != sequence:
task.sequence = sequence
await self.session.flush()
async def unmet_dependency_ids(self, dependency_ids: list[UUID]) -> list[UUID]:
"""Return the subset of dependency IDs whose status is non-terminal.
+31 -6
View File
@@ -422,11 +422,14 @@ class WorkspaceService:
Called from `ensure_workspace`'s healthy short-circuit so that a
respawned PM/Doc reads fresh `origin/<branch>` refs instead of
whatever the previous spawn left on disk. We deliberately omit a
positional refspec `git fetch origin` (no args after `origin`)
updates every branch under `refs/remotes/origin/`, which is what
downstream `git diff origin/<branch>` and `git log origin/<branch>`
readers want.
whatever the previous spawn left on disk. The fetch is SCOPED to the
workspace's current branch + the repo's default branch (with
`--no-tags --prune`). An all-refs `git fetch origin` transfers every
accumulated `feature/*` on a monorepo and blows past the timeout, after
which the workspace silently keeps a stale base and the agent builds on
it. The refs a workspace's `git diff/log origin/<branch>` readers need
are its own branch and the default; the integration branch is refreshed
at branch-creation time (`create_branch_for_task`), not here.
No `-c http.extraheader=` token injection: the orchestrator did
the original clone with a token but `_configure_git()` already
@@ -442,9 +445,31 @@ class WorkspaceService:
remote is operationally bad.
"""
def _git(*args: str) -> subprocess.CompletedProcess[str]:
return subprocess.run(
["git", *args],
cwd=str(workspace),
capture_output=True,
text=True,
check=False,
)
def _scoped_refs() -> list[str]:
"""The current branch + the repo's default branch, deduped."""
current = _git("rev-parse", "--abbrev-ref", "HEAD").stdout.strip()
origin_head = _git(
"symbolic-ref", "--short", "refs/remotes/origin/HEAD"
).stdout.strip()
default = origin_head.split("/", 1)[1] if "/" in origin_head else "master"
refs: list[str] = []
for ref in (current, default):
if ref and ref != "HEAD" and ref not in refs:
refs.append(ref)
return refs or ["master"]
def _do_fetch() -> subprocess.CompletedProcess[str]:
return subprocess.run(
["git", "fetch", "origin"],
["git", "fetch", "--no-tags", "--prune", "origin", *_scoped_refs()],
cwd=str(workspace),
capture_output=True,
text=True,