mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
fix(sequencing): reachability-aware claim bar + sequence_held surfacing (#681)
Three coupled claim-path bugs from the 2026-07-24 live incident, fixed at the shared root: - The edge-agnostic sequence bar phantom-held a task behind an unrelated, never-connected same-parent sibling that coincidentally shared a lower raw sequence (stamp_wave_sequence stamps from a partial per-task view). _claim_blocked_by_sequence now branches on is_batch_root_subtask: a MegaTask root-subtask (globally-computed Kahn wave, a deliberate staged-release barrier) keeps the strict rule unchanged; every other same-parent context routes through the pure sequence_blocker_id, which only blocks on a real transitive predecessor via dependency_ids UNIONED with completed_dependency_ids. A task with no same-parent dependency edge at all falls back to the raw bar unchanged (#452 preserved). - The hold surfaced as claim()'s bare None and was misdiagnosed by the verb runner as a concurrent-transition invalid_state. New sequence_hold_reason + a proactive _sequencing_claim_guard return a dedicated Envelope.sequence_held naming the blocker, on both the PENDING and NEEDS_REVISION reclaim paths. - give_me_work offered tasks the claim gate then rejected: both offer paths (list_pending_for_agent, _drop_dependency_held) now consult the bar via the exact claim predicate (is_pending_claim_blocked, extended to NEEDS_REVISION). Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
@@ -218,7 +218,9 @@ backlog -> pending -> claimed -> in_progress -> [blocked|paused] -> verifying
|
||||
|
||||
**In-path PR-review gate** (`awaiting_pr_review`): each assembled PR is reviewed before the PM merges. The cell PM's `submit_up` opens the cell→root PR and the Main PM's `submit_root` opens the root→master PR; both enter `awaiting_pr_review`, where a reviewer `pr_pass`es it on to `awaiting_pm_review` or `pr_fail`s it back to `needs_revision` — the merge-level reject the PM otherwise lacks. Leaf dev tasks and branchless coordination roots skip the gate. `pr_pass` additionally refuses while the assembled PR's own CI (its head commit's checks, `GitService.get_pr_ci_status`) is failing, pending, or unresolvable — a repo with no CI configured passes through with an evidence note; `pr_fail` stays available regardless so a reviewer is never stuck waiting on CI. The reviewer prompt requires a per-AC `file:line` walk (a silently dropped deliverable is an automatic fail) and the gate's diff/conventions base resolves the task's REAL parent branch (`resolve_parent_branch`, the parent task's own `branch_name`) instead of deriving it from the branch-name string, so a cross-team hop (e.g. cell→root) no longer attributes inherited base-branch content to the task under review.
|
||||
|
||||
**Sequence is the bar.** A task with a parent and effective sequence N (`COALESCE(sequence, 0)`) cannot be claimed while any same-parent sibling with a strictly lower effective sequence is non-terminal — assignee-blind, independent of and stricter than `dependency_ids`, enforced in `TaskService._validate_claim_preconditions` (the `claim` chokepoint itself) so every claim path crosses it. Ties run parallel; cancelled siblings never block; sequence `0` and parentless tasks are unaffected. Delegation stamps sequence from the collision DAG (`stamp_wave_sequence`: `1 + max` same-parent dependency sequence, or `0` when independent) instead of a raw per-sibling ordinal, so fully independent siblings tie and run in parallel while colliding/ordered work ascends — PM-authored sequences are never rewritten. `tasks.parent_task_id` is indexed (migration 069) since the guard's sibling probe runs on every claim; the dispatcher pre-filters dependency/sequence-held tasks (`TaskService.is_pending_claim_blocked`) before attempting a doomed claim.
|
||||
**Sequence is the bar.** A task with a parent and effective sequence N (`COALESCE(sequence, 0)`) cannot be claimed while any same-parent sibling with a strictly lower effective sequence is non-terminal — assignee-blind and, for MegaTask batch root-subtasks, independent of and stricter than `dependency_ids` (every other same-parent context is reachability-aware — see the sequence-drift fix below), enforced in `TaskService._validate_claim_preconditions` (the `claim` chokepoint itself) so every claim path crosses it. Ties run parallel; cancelled siblings never block; sequence `0` and parentless tasks are unaffected. Delegation stamps sequence from the collision DAG (`stamp_wave_sequence`: `1 + max` same-parent dependency sequence, or `0` when independent) instead of a raw per-sibling ordinal, so fully independent siblings tie and run in parallel while colliding/ordered work ascends — PM-authored sequences are never rewritten. `tasks.parent_task_id` is indexed (migration 069) since the guard's sibling probe runs on every claim; the dispatcher pre-filters dependency/sequence-held tasks (`TaskService.is_pending_claim_blocked`) before attempting a doomed claim.
|
||||
|
||||
**Sequence-drift fix (2026-07-24): the bar is reachability-aware outside MegaTask batches.** `stamp_wave_sequence` stamps each new sibling's wave from a partial, per-task view of the graph at delegate time — fine within one connected chain, but two INDEPENDENT dev-task streams under the same parent (unconnected, `stamp_wave_sequence`d incrementally over time) can land on the same raw sequence number by coincidence, and the old edge-agnostic bar phantom-held a wave-N sibling behind a totally unrelated wave-(N-1) sibling from a different stream. `_claim_blocked_by_sequence` now branches on `is_batch_root_subtask(task.batch_id, task.parent_task_id)`: a MegaTask root-subtask (whose `sequence` is a one-shot, globally-computed Kahn wave index from `PrompterService._build_confirm_batch` — a deliberate staged-release barrier) keeps the original strict, edge-agnostic rule unchanged; every other same-parent context routes through the pure `sequence_blocker_id` (`roboco/services/sequencing.py`), which only lets a lower-sequence candidate block when it's a real (transitive) predecessor via `dependency_ids` UNIONED with `completed_dependency_ids` (the union matters — `_unblock_dependents` prunes a completed dependency's edge into `completed_dependency_ids` the moment it lands, almost always before the dependent is ever claimed) — a task with NO dependency edge onto any same-parent sibling at all still falls back to the original raw bar unchanged (preserves the #452 edge-less-PM-delegation scenario exactly). The hold also now surfaces cleanly: `TaskService.sequence_hold_reason` + a proactive `_sequencing_claim_guard` in the gateway's `_run_claim_guards` return a dedicated `Envelope.sequence_held` (naming the blocker) on both the PENDING and NEEDS_REVISION reclaim paths, instead of `claim()`'s bare `None` return reaching the verb runner and getting misdiagnosed as a "concurrent transition" `invalid_state`. `give_me_work`'s two offer paths (`TaskService.list_pending_for_agent` and the Choreographer's `_drop_dependency_held` over `list_assigned_for_agent`) both now also consult the sequence bar (the latter via `is_pending_claim_blocked`, extended to NEEDS_REVISION) so the dispatcher never offers a task the claim gate is about to reject.
|
||||
|
||||
**PM-turn elimination (auto-submit).** When every child of an assembled, PR-bearing parent goes terminal, the orchestrator's closure dispatcher (`_maybe_spawn_pm_closure` → `_closure_handled_without_pm` → `_try_auto_submit`) runs the real `submit_up`/`submit_root` gate system-side as the owning PM instead of spawning the PM for that turn — same verb, same guards (ownership, notes, journal:decision, subtasks-terminal, parent-AC coverage, branch), authorized via the internal API with the PM's own identity headers. This is unconditional — the turn cut IS the flow, no kill-switch. Success lands the task on `awaiting_pr_review` with an audited `task.auto_submitted` row and no PM spawn; ANY refusal (branchless/umbrella parent, a gate rejection — freshness, AC coverage, a subtask-terminal race — or a transport error) falls back to spawning the PM exactly as before — that fallback is the sole safety net — with the refusal reason threaded into the PM's closure prompt so it isn't rediscovering it blind.
|
||||
|
||||
|
||||
@@ -37,6 +37,7 @@ from roboco.services.gateway.claim_guards import (
|
||||
already_active_guard,
|
||||
paused_tasks_guard,
|
||||
project_budget_exceeded_guard,
|
||||
sequence_held_guard,
|
||||
unmet_dependency_guard,
|
||||
)
|
||||
from roboco.services.gateway.envelope import Envelope
|
||||
@@ -829,23 +830,35 @@ class Choreographer:
|
||||
return f"call i_will_work_on(task_id='{tid}', plan='<plan>') to start"
|
||||
|
||||
async def _drop_dependency_held(self, tasks: list[Any]) -> list[Any]:
|
||||
"""Drop pre-assigned PENDING tasks whose non-terminal dependencies are
|
||||
still unresolved.
|
||||
"""Drop PENDING/NEEDS_REVISION tasks the claim gate would refuse
|
||||
right now — an unmet dependency or a same-parent sequence hold.
|
||||
|
||||
``give_me_work``'s ``list_assigned_for_agent`` fallback includes PENDING
|
||||
rows with no dependency filter, so without this a held pre-assigned
|
||||
subtask (e.g. a frontend dev's task waiting on the UX/UI design) would
|
||||
still be offered and the agent only bounced at claim time. Mirrors the
|
||||
gate in ``TaskService.list_pending_for_agent`` and ``_run_claim_guards``.
|
||||
Only PENDING rows are gated — an already-claimed task is past the gate.
|
||||
``give_me_work``'s ``list_assigned_for_agent`` fallback spans every
|
||||
active status with no hold filter, so without this a held task
|
||||
(e.g. a frontend dev's subtask waiting on the UX/UI design, or a
|
||||
needs_revision reclaim behind a lower-sequence sibling delegated
|
||||
after the first claim) was still offered here and only bounced at
|
||||
claim time — the give_me_work/i_will_work_on offer-then-reject loop
|
||||
the 2026-07-24 incident hit on the needs_revision path.
|
||||
``is_pending_claim_blocked`` wraps the EXACT predicate
|
||||
``TaskService.claim()`` enforces (dependency + sequence), so this
|
||||
can't drift from the claim gate. Scoped to PENDING and
|
||||
NEEDS_REVISION — the only statuses that guard reads
|
||||
(``_claim_blocked_by_sequencing``); every other status already
|
||||
passed it at an earlier claim. ``is True`` (not a bare truthy
|
||||
check) keeps this inert under partial test mocks (an unstubbed
|
||||
AsyncMock method returns a truthy mock object, not a real bool) —
|
||||
mirrors ``_pending_not_lane_held``'s identical ``is not True`` guard.
|
||||
"""
|
||||
offerable: list[Any] = []
|
||||
for task in tasks:
|
||||
dep_ids = list(getattr(task, "dependency_ids", []) or [])
|
||||
if (
|
||||
str(task.status) == "pending"
|
||||
and dep_ids
|
||||
and await self._deps.task.unmet_dependency_ids(dep_ids)
|
||||
str(task.status)
|
||||
in (
|
||||
"pending",
|
||||
"needs_revision",
|
||||
)
|
||||
and await self._deps.task.is_pending_claim_blocked(task.id) is True
|
||||
):
|
||||
continue
|
||||
offerable.append(task)
|
||||
@@ -1192,7 +1205,7 @@ class Choreographer:
|
||||
paused = await self.task.list_paused_for_agent(agent_id)
|
||||
if guard := paused_tasks_guard(paused, task.id):
|
||||
return guard
|
||||
if guard := await self._dependency_claim_guard(task):
|
||||
if guard := await self._sequencing_claim_guard(task):
|
||||
return guard
|
||||
if check_project_budget and (
|
||||
guard := await self._project_budget_claim_guard(task)
|
||||
@@ -1202,6 +1215,37 @@ class Choreographer:
|
||||
return None
|
||||
return await self._lane_claim_guard(task)
|
||||
|
||||
async def _sequencing_claim_guard(self, task: Any) -> Envelope | None:
|
||||
"""Both halves of the claim-time sequencing bar in one call — an
|
||||
unmet dependency or a same-parent sequence hold — mirroring
|
||||
``TaskService._claim_blocked_by_sequencing``'s own composition.
|
||||
Collapsed into one ``_run_claim_guards`` return so the xenon
|
||||
return-statement budget holds.
|
||||
"""
|
||||
if guard := await self._dependency_claim_guard(task):
|
||||
return guard
|
||||
return await self._sequence_claim_guard(task)
|
||||
|
||||
async def _sequence_claim_guard(self, task: Any) -> Envelope | None:
|
||||
"""Refuse claim while a same-parent sibling with a lower sequence is
|
||||
still non-terminal — surfaced as a clean, named ``sequence_held``
|
||||
BEFORE the composed claim verb runs (see ``sequence_held_guard``).
|
||||
|
||||
Runs for every ``_run_claim_guards`` caller (``i_will_work_on`` /
|
||||
``i_will_plan``, both the fresh-claim and stuck-``claimed``-recovery
|
||||
paths), so both the PENDING and NEEDS_REVISION reclaim routes agree
|
||||
with ``TaskService.claim()``'s own sequence bar instead of it
|
||||
surfacing deep inside the verb runner as a misdiagnosed
|
||||
"concurrent transition". ``isinstance(..., str)`` keeps this inert
|
||||
under partial test mocks (an unstubbed AsyncMock method returns a
|
||||
truthy mock object, not ``None`` or a real string) — mirrors
|
||||
``_pending_not_lane_held``'s identical mock-safety guard.
|
||||
"""
|
||||
blocked_by = await self.task.sequence_hold_reason(task)
|
||||
if not isinstance(blocked_by, str):
|
||||
return None
|
||||
return sequence_held_guard(task, blocked_by)
|
||||
|
||||
async def _dependency_claim_guard(self, task: Any) -> Envelope | None:
|
||||
"""Refuse claim while the task has non-terminal dependencies.
|
||||
|
||||
|
||||
@@ -109,6 +109,26 @@ def project_budget_exceeded_guard(
|
||||
)
|
||||
|
||||
|
||||
def sequence_held_guard(target_task: Any, blocked_by: str | None) -> Envelope | None:
|
||||
"""Refuse claim while a same-parent sibling with a lower effective
|
||||
sequence is still non-terminal (CLAUDE.md "sequence is the bar").
|
||||
|
||||
``blocked_by`` is the caller-resolved blocker description (it requires a
|
||||
DB read of the sibling set — ``TaskService.sequence_hold_reason``) so
|
||||
this predicate stays pure, mirroring ``unmet_dependency_guard``. Runs
|
||||
BEFORE the composed claim verb so the hold surfaces as a clean, distinct
|
||||
``sequence_held`` envelope naming the blocker instead of
|
||||
``TaskService.claim()``'s bare ``None`` return being misdiagnosed by the
|
||||
verb runner as a concurrent-transition race (the 2026-07-24 incident:
|
||||
repeated identical ``invalid_state`` rejections on a needs_revision
|
||||
reclaim that was never touched by anything concurrent — it was
|
||||
sequence-held the whole time).
|
||||
"""
|
||||
if blocked_by is None:
|
||||
return None
|
||||
return Envelope.sequence_held(blocked_by=blocked_by, task_id=str(target_task.id))
|
||||
|
||||
|
||||
def unmet_dependency_guard(
|
||||
target_task: Any, unmet_dependency_ids: list[UUID]
|
||||
) -> Envelope | None:
|
||||
|
||||
@@ -141,6 +141,37 @@ class Envelope:
|
||||
context_briefing=context_briefing or {},
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def sequence_held(
|
||||
cls,
|
||||
*,
|
||||
blocked_by: str,
|
||||
task_id: str | None = None,
|
||||
context_briefing: dict[str, Any] | None = None,
|
||||
) -> Envelope:
|
||||
"""A same-parent sibling with a lower effective sequence is still
|
||||
non-terminal (CLAUDE.md "sequence is the bar"). Distinct from
|
||||
`invalid_state` so an agent (and the audit log) can tell a real,
|
||||
named hold from a genuine concurrent-transition race — the bare
|
||||
`None` `TaskService.claim()` used to return for this case surfaced
|
||||
as a cryptic "concurrent transition" `invalid_state` (the
|
||||
2026-07-24 misdiagnosis: 6 identical rejections on a stably-held
|
||||
needs_revision reclaim nothing concurrent ever touched).
|
||||
"""
|
||||
return cls(
|
||||
error="sequence_held",
|
||||
task_id=task_id,
|
||||
message=(
|
||||
f"blocked by {blocked_by} — a same-parent sibling with a "
|
||||
"lower sequence is still in progress."
|
||||
),
|
||||
remediate=(
|
||||
"wait for the blocking sibling to reach completed/cancelled, "
|
||||
"or call give_me_work() for other available work meanwhile"
|
||||
),
|
||||
context_briefing=context_briefing or {},
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def not_authorized(
|
||||
cls,
|
||||
|
||||
@@ -460,3 +460,56 @@ def by_osmosis_tail_dev_tasks(
|
||||
)
|
||||
tails.append(getattr(tail, "id", tail))
|
||||
return tails
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Claim-gate sequence-blocker resolution — the pure decision behind
|
||||
# TaskService._claim_blocked_by_sequence's non-batch branch.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def sequence_blocker_id(
|
||||
task_dependency_ids: list[object],
|
||||
candidate_ids: list[object],
|
||||
sibling_dependency_ids: dict[object, list[object]],
|
||||
) -> object | None:
|
||||
"""Pick the same-parent sibling (if any) that really blocks a claim.
|
||||
|
||||
``candidate_ids`` are same-parent siblings with a strictly lower
|
||||
effective sequence that are still non-terminal (the caller has already
|
||||
applied both filters) — this only decides WHICH of them, if any,
|
||||
actually blocks. ``sibling_dependency_ids`` maps every same-parent
|
||||
sibling's id to its own ``dependency_ids`` (candidates and non-candidates
|
||||
alike — a non-candidate can still be an intermediate hop in a chain).
|
||||
|
||||
``dependency_ids`` is the stable, authoritative ordering; ``sequence`` is
|
||||
a derived redundancy that drifts once a completed dependency's edge is
|
||||
pruned (``_unblock_dependents``) or a later sibling is stamped from a
|
||||
different partial view of the graph (``stamp_wave_sequence`` runs
|
||||
per-task at delegate time, not as one coordinated plan) — the live
|
||||
2026-07-24 incident: an unrelated, never-connected sibling in a
|
||||
different stream phantom-held a claim purely because it shared a lower
|
||||
raw sequence number. So a candidate only blocks when it is a real
|
||||
(transitive) dependency-graph predecessor of this task — UNLESS the task
|
||||
carries NO dependency edge onto ANY same-parent sibling at all, in which
|
||||
case there is no graph to consult and the original raw bar applies
|
||||
unchanged (a PM delegating siblings 0..N with zero wired edges between
|
||||
them still holds strict order — the #452 incident this guard exists
|
||||
for). Batch root-subtasks (MegaTask) never reach this — see
|
||||
``_claim_blocked_by_sequence``.
|
||||
"""
|
||||
dep_ids: set[object] = set(task_dependency_ids)
|
||||
if not (dep_ids & sibling_dependency_ids.keys()):
|
||||
return candidate_ids[0] if candidate_ids else None
|
||||
candidates: set[object] = set(candidate_ids)
|
||||
seen: set[object] = set()
|
||||
frontier: list[object] = list(dep_ids)
|
||||
while frontier:
|
||||
node = frontier.pop()
|
||||
if node in seen:
|
||||
continue
|
||||
seen.add(node)
|
||||
if node in candidates:
|
||||
return node
|
||||
frontier.extend(sibling_dependency_ids.get(node, []))
|
||||
return None
|
||||
|
||||
+140
-24
@@ -3171,16 +3171,36 @@ class TaskService(BaseService):
|
||||
CEO directive: sequence is the bar, independent of ``dependency_ids``
|
||||
— a PM can delegate siblings with ``sequence`` 0..N and wire no
|
||||
dependency edges between them, and claim order must still hold (the
|
||||
4-revision-subtask live failure this guards). STRICTER than
|
||||
dependency edges where both exist: a wave-N sibling waits for EVERY
|
||||
wave-(N-1) sibling, not just its edge targets — no edges-exist
|
||||
exemption. Effective sequence is ``COALESCE(sequence, 0)`` on both
|
||||
sides; ties run in parallel. Effective sequence 0 or no parent is
|
||||
unaffected. Scoped to PENDING and NEEDS_REVISION claims — wider than
|
||||
``_claim_blocked_by_dependencies`` (PENDING only), deliberately: a
|
||||
lower-sequence sibling delegated AFTER the first claim must still
|
||||
hold a needs_revision reclaim; the dep guard's identical
|
||||
needs_revision gap is pre-existing and left as-is.
|
||||
4-revision-subtask live failure this guards). Effective sequence is
|
||||
``COALESCE(sequence, 0)`` on both sides; ties run in parallel.
|
||||
Effective sequence 0 or no parent is unaffected. Scoped to PENDING
|
||||
and NEEDS_REVISION claims — wider than ``_claim_blocked_by_dependencies``
|
||||
(PENDING only), deliberately: a lower-sequence sibling delegated
|
||||
AFTER the first claim must still hold a needs_revision reclaim; the
|
||||
dep guard's identical needs_revision gap is pre-existing and left
|
||||
as-is.
|
||||
|
||||
Two scopes, deliberately different (the live 2026-07-24 sequence-drift
|
||||
incident): a MegaTask root-subtask's ``sequence`` is a one-shot,
|
||||
globally-computed Kahn wave index (``PrompterService._build_confirm_batch``)
|
||||
— a deliberate staged-release barrier, so it stays STRICTER than
|
||||
dependency edges: a wave-N root-subtask waits for EVERY wave-(N-1)
|
||||
root-subtask, not just its wired edge targets, no edges-exist
|
||||
exemption. Every other same-parent context (dev-task collision-DAG
|
||||
streams stamped incrementally by ``stamp_wave_sequence``, or a
|
||||
hand-authored coordination subtask) instead routes through
|
||||
``sequence_blocker_id``: a candidate only blocks when it is a real
|
||||
(transitive) predecessor via ``dependency_ids`` UNIONED with
|
||||
``completed_dependency_ids`` (a real edge to an already-terminal
|
||||
predecessor is pruned from the live column by
|
||||
``_unblock_dependents`` — the union keeps that graph fact visible
|
||||
so the fallback below never misfires on it) — because ``sequence``
|
||||
is stamped from a partial, per-task view of the graph at delegate
|
||||
time, an unrelated sibling from a different, never-connected stream
|
||||
must not phantom-hold a claim purely for sharing a lower raw
|
||||
number. A task with NO dependency edge (live or completed) onto ANY
|
||||
same-parent sibling falls back to the raw bar unchanged (preserves
|
||||
the #452 edge-less scenario exactly).
|
||||
"""
|
||||
if (
|
||||
task.status not in (TaskStatus.PENDING, TaskStatus.NEEDS_REVISION)
|
||||
@@ -3190,21 +3210,16 @@ class TaskService(BaseService):
|
||||
seq = task.sequence or 0
|
||||
if seq == 0:
|
||||
return None
|
||||
terminal = (TaskStatus.COMPLETED, TaskStatus.CANCELLED)
|
||||
result = await self.session.execute(
|
||||
select(TaskTable.title, TaskTable.sequence)
|
||||
.where(
|
||||
TaskTable.parent_task_id == task.parent_task_id,
|
||||
TaskTable.id != task.id,
|
||||
func.coalesce(TaskTable.sequence, 0) < seq,
|
||||
TaskTable.status.notin_(terminal),
|
||||
)
|
||||
.limit(1)
|
||||
candidates, detail, sibling_deps = await self._sequence_sibling_candidates(
|
||||
task, seq
|
||||
)
|
||||
row = result.first()
|
||||
if row is None:
|
||||
if not candidates:
|
||||
return None
|
||||
blocker = f"{row.title!r} (sequence {row.sequence or 0})"
|
||||
blocker_id = self._resolve_sequence_blocker(task, candidates, sibling_deps)
|
||||
if blocker_id is None:
|
||||
return None
|
||||
title, blk_seq = detail[blocker_id]
|
||||
blocker = f"{title!r} (sequence {blk_seq})"
|
||||
self.log.warning(
|
||||
"Cannot claim task - sequence_held",
|
||||
task_id=str(task.id),
|
||||
@@ -3212,6 +3227,100 @@ class TaskService(BaseService):
|
||||
)
|
||||
return blocker
|
||||
|
||||
async def _sequence_sibling_candidates(
|
||||
self, task: TaskTable, seq: int
|
||||
) -> tuple[
|
||||
list[UUID],
|
||||
dict[UUID, tuple[str, int]],
|
||||
dict[UUID, list[UUID]],
|
||||
]:
|
||||
"""Fetch same-parent siblings, split into (candidates, detail,
|
||||
dependency-graph). ``candidates`` are strictly-lower-sequence,
|
||||
non-terminal siblings; ``sibling_deps`` unions each sibling's live +
|
||||
completed dependency ids (every sibling, not just candidates — a
|
||||
BFS hop through an already-terminal intermediate must still reach a
|
||||
further, still-open predecessor beyond it)."""
|
||||
terminal = (TaskStatus.COMPLETED, TaskStatus.CANCELLED)
|
||||
result = await self.session.execute(
|
||||
select(
|
||||
TaskTable.id,
|
||||
TaskTable.title,
|
||||
TaskTable.sequence,
|
||||
TaskTable.status,
|
||||
TaskTable.dependency_ids,
|
||||
TaskTable.completed_dependency_ids,
|
||||
)
|
||||
.where(
|
||||
TaskTable.parent_task_id == task.parent_task_id,
|
||||
TaskTable.id != task.id,
|
||||
)
|
||||
.order_by(TaskTable.sequence, TaskTable.created_at)
|
||||
)
|
||||
candidates: list[UUID] = []
|
||||
detail: dict[UUID, tuple[str, int]] = {}
|
||||
sibling_deps: dict[UUID, list[UUID]] = {}
|
||||
for row in result.all():
|
||||
sibling_deps[row.id] = [
|
||||
*(row.dependency_ids or []),
|
||||
*(row.completed_dependency_ids or []),
|
||||
]
|
||||
if (row.sequence or 0) < seq and row.status not in terminal:
|
||||
candidates.append(row.id)
|
||||
detail[row.id] = (row.title, row.sequence or 0)
|
||||
return candidates, detail, sibling_deps
|
||||
|
||||
def _resolve_sequence_blocker(
|
||||
self,
|
||||
task: TaskTable,
|
||||
candidates: list[UUID],
|
||||
sibling_deps: dict[UUID, list[UUID]],
|
||||
) -> UUID | None:
|
||||
"""Which candidate (if any) really blocks — batch root-subtasks keep
|
||||
the raw, edge-agnostic bar; everything else routes through the
|
||||
dependency-graph-aware ``sequence_blocker_id`` (see
|
||||
``_claim_blocked_by_sequence``'s docstring for the full rationale)."""
|
||||
if is_batch_root_subtask(
|
||||
batch_id=task.batch_id, parent_task_id=task.parent_task_id
|
||||
):
|
||||
return candidates[0]
|
||||
from roboco.services.sequencing import sequence_blocker_id
|
||||
|
||||
# Union live + completed dependency ids: a real edge to an
|
||||
# already-terminal predecessor is pruned from `dependency_ids` by
|
||||
# `_unblock_dependents` (moved into `completed_dependency_ids`
|
||||
# instead) the moment that predecessor completes — almost always
|
||||
# BEFORE this dependent is ever claimed. Without the union, a task
|
||||
# whose sole real edge already resolved would look edge-less and
|
||||
# wrongly fall back to the raw bar, reviving the exact phantom-hold
|
||||
# this fix removes.
|
||||
own_dep_ids: list[object] = [
|
||||
*(task.dependency_ids or []),
|
||||
*(task.completed_dependency_ids or []),
|
||||
]
|
||||
return cast(
|
||||
"UUID | None",
|
||||
sequence_blocker_id(
|
||||
task_dependency_ids=own_dep_ids,
|
||||
candidate_ids=cast("list[object]", candidates),
|
||||
sibling_dependency_ids=cast("dict[object, list[object]]", sibling_deps),
|
||||
),
|
||||
)
|
||||
|
||||
async def sequence_hold_reason(self, task: TaskTable) -> str | None:
|
||||
"""Public accessor for the sequence claim-gate's blocker naming.
|
||||
|
||||
Thin wrapper over ``_claim_blocked_by_sequence`` so gateway callers
|
||||
can surface a clean, distinct ``sequence_held`` envelope BEFORE
|
||||
running the composed claim verb — instead of letting the hold
|
||||
surface as ``TaskService.claim()``'s bare ``None`` return, which the
|
||||
verb runner's intermediate-``None`` guard misdiagnoses as a
|
||||
concurrent-transition race (the 2026-07-24 incident: repeated
|
||||
identical ``invalid_state`` rejections on a ``needs_revision``
|
||||
reclaim that nothing concurrent ever touched — it was sequence-held
|
||||
the whole time).
|
||||
"""
|
||||
return await self._claim_blocked_by_sequence(task)
|
||||
|
||||
async def _claim_blocked_by_sequencing(self, task: TaskTable) -> bool:
|
||||
"""True when either sequencing guard blocks this PENDING claim.
|
||||
|
||||
@@ -9440,7 +9549,12 @@ class TaskService(BaseService):
|
||||
dependency that has not resolved (e.g. a frontend dev coding before
|
||||
the UX/UI design lands). The pre-assigned path bypasses
|
||||
`list_pending(filter_by_dependencies=True)`, so the dependency gate
|
||||
must be applied here too.
|
||||
must be applied here too — and so must the sequence claim-gate
|
||||
(`_claim_blocked_by_sequence`): without it, give_me_work offered a
|
||||
sequence-held pre-assigned task that `i_will_work_on`'s claim() then
|
||||
rejected (the give_me_work/i_will_work_on disagreement this closes
|
||||
on the PENDING side; `_drop_dependency_held` closes the same class
|
||||
of gap for the `list_assigned_for_agent` NEEDS_REVISION fallback).
|
||||
|
||||
F059: a self-heal fix task held for the CEO's Approve-&-Start
|
||||
(``source=self_heal`` + ``confirmed_by_human=False``) is NOT offered
|
||||
@@ -9485,6 +9599,8 @@ class TaskService(BaseService):
|
||||
for task in tasks:
|
||||
if await self.unmet_dependency_ids(list(task.dependency_ids)):
|
||||
continue
|
||||
if await self._claim_blocked_by_sequence(task) is not None:
|
||||
continue
|
||||
available.append(task)
|
||||
return available
|
||||
|
||||
|
||||
@@ -117,7 +117,7 @@ def test_sibling_sequence_blocks_claim_until_earlier_sibling_terminal(
|
||||
# No wire_dependency() call anywhere in this test — sequence alone must
|
||||
# hold the order; seq0 stays PENDING (open, non-terminal).
|
||||
|
||||
expect_error(
|
||||
env = expect_error(
|
||||
main_pm.flow(
|
||||
"i_will_plan",
|
||||
task_id=str(seq1_id),
|
||||
@@ -125,9 +125,10 @@ def test_sibling_sequence_blocks_claim_until_earlier_sibling_terminal(
|
||||
approach=_APPROACH,
|
||||
sub_tasks=_SUB_TASKS,
|
||||
),
|
||||
"invalid_state",
|
||||
"sequence_held",
|
||||
"main_pm i_will_plan seq-1 while seq-0 open (no dependency edge)",
|
||||
)
|
||||
assert "Revision 0" in (env.get("message") or "")
|
||||
assert task_state(stack, seq1_id)["status"] == "pending"
|
||||
|
||||
_cancel(stack, seq0_id)
|
||||
|
||||
@@ -1737,11 +1737,17 @@ async def test_is_pending_claim_blocked_false_for_missing_task(
|
||||
async def test_claim_batch_wave_blocked_by_all_wave0_siblings_no_edges(
|
||||
task_setup: dict, db_session: AsyncSession
|
||||
) -> None:
|
||||
"""STRICTER than dependency edges where both exist: a wave-1 root-subtask
|
||||
waits for EVERY wave-0 sibling, not just the one edge target the
|
||||
collision analyzer happened to wire — no edges-exist exemption."""
|
||||
"""STRICTER than dependency edges where both exist: a MegaTask wave-1
|
||||
root-subtask waits for EVERY wave-0 root-subtask, not just the one edge
|
||||
target the collision analyzer happened to wire — no edges-exist
|
||||
exemption. Scoped to `batch_id`-bearing root-subtasks specifically
|
||||
(`_build_confirm_batch` stamps `sequence` as a one-shot, globally
|
||||
computed Kahn wave index — a deliberate staged-release barrier); a
|
||||
plain (non-batch) same-parent sibling instead needs a real dependency
|
||||
path — see `test_claim_not_blocked_by_unconnected_sibling_in_different_stream`."""
|
||||
svc = task_setup["svc"]
|
||||
umbrella = await svc.create(_req(task_setup, title="umbrella"))
|
||||
batch_id = uuid4()
|
||||
wave0_a = await svc.create(
|
||||
_req(task_setup, title="wave0-a", parent_task_id=umbrella.id, sequence=0)
|
||||
)
|
||||
@@ -1756,6 +1762,12 @@ async def test_claim_batch_wave_blocked_by_all_wave0_siblings_no_edges(
|
||||
# never wires an edge to it.
|
||||
await svc.add_dependency(wave1.id, wave0_a.id)
|
||||
wave0_a.status = TaskStatus.COMPLETED
|
||||
# Direct ORM stamp (mirrors `_build_confirm_batch`'s BatchPlacement,
|
||||
# bypassing create-time batch-shape validation the same way the rest of
|
||||
# this test pokes `.status` directly).
|
||||
wave0_a.batch_id = batch_id
|
||||
wave0_b.batch_id = batch_id
|
||||
wave1.batch_id = batch_id
|
||||
await db_session.flush()
|
||||
|
||||
# The edge is satisfied, but wave0_b is still open with a lower sequence.
|
||||
@@ -1770,6 +1782,94 @@ async def test_claim_batch_wave_blocked_by_all_wave0_siblings_no_edges(
|
||||
assert claimed.status == TaskStatus.CLAIMED
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_claim_not_blocked_by_unconnected_sibling_in_different_stream(
|
||||
task_setup: dict, db_session: AsyncSession
|
||||
) -> None:
|
||||
"""The 2026-07-24 live incident: a frontend cell with 4 independent
|
||||
dev-task streams — stream1-b (stamped wave 1, a real dependency on
|
||||
stream1-a) must not be phantom-held by stream4-b (wave 0, in_progress)
|
||||
just because they share a same parent and a lower raw sequence number.
|
||||
Unlike the MegaTask-batch case above (no `batch_id` here), a plain
|
||||
same-parent sibling only blocks via a real dependency path."""
|
||||
svc = task_setup["svc"]
|
||||
cell = await svc.create(_req(task_setup, title="cell"))
|
||||
stream1_a = await svc.create(
|
||||
_req(task_setup, title="stream1-a", parent_task_id=cell.id)
|
||||
)
|
||||
stream1_b = await svc.create(
|
||||
_req(task_setup, title="stream1-b", parent_task_id=cell.id)
|
||||
)
|
||||
stream4_b = await svc.create(
|
||||
_req(task_setup, title="stream4-b", parent_task_id=cell.id)
|
||||
)
|
||||
await svc.add_dependency(stream1_b.id, stream1_a.id)
|
||||
await svc.stamp_wave_sequence(stream1_a.id)
|
||||
await svc.stamp_wave_sequence(stream1_b.id)
|
||||
await svc.stamp_wave_sequence(stream4_b.id)
|
||||
assert (stream1_a.sequence, stream1_b.sequence, stream4_b.sequence) == (0, 1, 0)
|
||||
|
||||
# stream1-a (the REAL predecessor) is done; stream4-b (an unconnected
|
||||
# sibling in a different stream) is still open with a lower sequence.
|
||||
stream1_a.status = TaskStatus.COMPLETED
|
||||
stream4_b.status = TaskStatus.IN_PROGRESS
|
||||
stream1_b.branch_name = "feature/frontend/aaaa9999"
|
||||
await db_session.flush()
|
||||
|
||||
claimed = await svc.claim(stream1_b.id, task_setup["agent_id"])
|
||||
assert claimed is not None, (
|
||||
"an unconnected sibling in a different stream must not phantom-hold"
|
||||
)
|
||||
assert claimed.status == TaskStatus.CLAIMED
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_claim_not_blocked_after_real_dependency_pruned_on_completion(
|
||||
task_setup: dict, db_session: AsyncSession
|
||||
) -> None:
|
||||
"""The precise drift mechanic: `_unblock_dependents` strips a completed
|
||||
dependency from the live `dependency_ids` (moving it to
|
||||
`completed_dependency_ids`) the moment it completes — almost always
|
||||
BEFORE the dependent is ever claimed. The sequence claim-gate must
|
||||
still recognize the pruned edge as real graph info (via
|
||||
`completed_dependency_ids`) rather than treating the now-edge-less task
|
||||
as a manually-sequenced, edge-less chain and reviving the raw
|
||||
strictly-lower-sequence bar against an unrelated sibling."""
|
||||
svc = task_setup["svc"]
|
||||
cell = await svc.create(_req(task_setup, title="cell"))
|
||||
real_predecessor = await svc.create(
|
||||
_req(task_setup, title="real predecessor", parent_task_id=cell.id)
|
||||
)
|
||||
dependent = await svc.create(
|
||||
_req(task_setup, title="dependent", parent_task_id=cell.id)
|
||||
)
|
||||
unrelated = await svc.create(
|
||||
_req(task_setup, title="unrelated stream", parent_task_id=cell.id)
|
||||
)
|
||||
await svc.add_dependency(dependent.id, real_predecessor.id)
|
||||
await svc.stamp_wave_sequence(real_predecessor.id)
|
||||
await svc.stamp_wave_sequence(dependent.id)
|
||||
await svc.stamp_wave_sequence(unrelated.id)
|
||||
assert dependent.sequence == 1
|
||||
|
||||
# Simulate `_unblock_dependents`'s exact effect: the completed
|
||||
# predecessor's edge is pruned from the live column and moved to the
|
||||
# completed ledger.
|
||||
real_predecessor.status = TaskStatus.COMPLETED
|
||||
dependent.dependency_ids = []
|
||||
dependent.completed_dependency_ids = [real_predecessor.id]
|
||||
unrelated.status = TaskStatus.IN_PROGRESS
|
||||
dependent.branch_name = "feature/frontend/bbbb8888"
|
||||
await db_session.flush()
|
||||
|
||||
claimed = await svc.claim(dependent.id, task_setup["agent_id"])
|
||||
assert claimed is not None, (
|
||||
"a pruned-but-once-real edge must still count as graph info, not "
|
||||
"revert to the raw edge-less sequence bar"
|
||||
)
|
||||
assert claimed.status == TaskStatus.CLAIMED
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_needs_revision_reclaim_blocked_by_lower_sequence_sibling(
|
||||
task_setup: dict, db_session: AsyncSession
|
||||
|
||||
@@ -22,6 +22,7 @@ from roboco.services.sequencing import (
|
||||
by_osmosis_tail_dev_tasks,
|
||||
cell_task_wave_chain_depends_on,
|
||||
dev_task_collision_edges,
|
||||
sequence_blocker_id,
|
||||
)
|
||||
|
||||
|
||||
@@ -594,3 +595,75 @@ def test_declared_cycle_rejected() -> None:
|
||||
]
|
||||
with pytest.raises(SequencingError):
|
||||
SequencingService().analyze(s, _backend, {"backend": 2})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# sequence_blocker_id — the claim-gate's non-batch reachability decision
|
||||
# (the 2026-07-24 phantom cross-stream serialization fix).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_sequence_blocker_no_graph_info_falls_back_to_raw_first_candidate() -> None:
|
||||
"""No dependency edge onto ANY same-parent sibling at all — the #452
|
||||
manually-sequenced, edge-less scenario — keeps the strict raw bar: the
|
||||
first (lowest-sequence) candidate blocks unconditionally."""
|
||||
a, b = uuid4(), uuid4()
|
||||
assert (
|
||||
sequence_blocker_id(
|
||||
task_dependency_ids=[],
|
||||
candidate_ids=[a, b],
|
||||
sibling_dependency_ids={a: [], b: []},
|
||||
)
|
||||
== a
|
||||
)
|
||||
|
||||
|
||||
def test_sequence_blocker_ignores_unconnected_sibling() -> None:
|
||||
"""A same-parent sibling reachable via NO edge (a different stream) must
|
||||
not block once real graph info exists elsewhere."""
|
||||
real_predecessor, unrelated = uuid4(), uuid4()
|
||||
assert (
|
||||
sequence_blocker_id(
|
||||
task_dependency_ids=[real_predecessor],
|
||||
candidate_ids=[unrelated],
|
||||
sibling_dependency_ids={real_predecessor: [], unrelated: []},
|
||||
)
|
||||
is None
|
||||
)
|
||||
|
||||
|
||||
def test_sequence_blocker_finds_direct_predecessor() -> None:
|
||||
predecessor = uuid4()
|
||||
assert (
|
||||
sequence_blocker_id(
|
||||
task_dependency_ids=[predecessor],
|
||||
candidate_ids=[predecessor],
|
||||
sibling_dependency_ids={predecessor: []},
|
||||
)
|
||||
== predecessor
|
||||
)
|
||||
|
||||
|
||||
def test_sequence_blocker_transitive_two_hop() -> None:
|
||||
"""A candidate two hops away (through an already-terminal, non-candidate
|
||||
intermediate) is still found — a genuine ordering must still hold."""
|
||||
intermediate, root_blocker = uuid4(), uuid4()
|
||||
assert (
|
||||
sequence_blocker_id(
|
||||
task_dependency_ids=[intermediate],
|
||||
candidate_ids=[root_blocker],
|
||||
sibling_dependency_ids={intermediate: [root_blocker], root_blocker: []},
|
||||
)
|
||||
== root_blocker
|
||||
)
|
||||
|
||||
|
||||
def test_sequence_blocker_no_candidates_is_none() -> None:
|
||||
assert (
|
||||
sequence_blocker_id(
|
||||
task_dependency_ids=[uuid4()],
|
||||
candidate_ids=[],
|
||||
sibling_dependency_ids={},
|
||||
)
|
||||
is None
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user