fix(orchestrator): task-scoped oscillation breaker for escalate/unblock ping-pong (#685)

* fix(orchestrator): task-scoped oscillation breaker for escalate/unblock ping-pong

An escalation ping-pong oscillates a task between two agents (cell PM
escalate_up -> BLOCKED -> main PM unblock -> restored -> respawn ->
escalate again). The per-(agent, task) respawn gate never trips on it:
the restored side is dispatched by _dispatch_claimed_without_agent,
which consults no respawn counter at all, so one side of the round trip
always has fuel regardless of the other's strikes — and even a tripped
main-PM counter only stalls the task silently at blocked instead of
surfacing the oscillation.

- Strikes are counted task-scoped at the unblock() chokepoint
  (agent-agnostic; legitimate needs_revision rework never calls
  unblock, so it structurally cannot trip this), durable in the
  existing orchestration_markers column — no migration.
- Progress between round-trips (commits / revision_count advancing)
  resets the count: real forward motion is not an oscillation.
- On trip: the task is blocked with a HUMAN resolver (the budget-breach
  posture), both dispatchers stop respawning onto it, further unblock()
  refuses until an admin override clears the marker, and the CEO
  notification names both agents and the cycle count.
- _notification_has_live_work now treats a HITL-blocked related task as
  no live work, closing the same loop for the admin-route escalation
  path.

* fix(orchestrator): wire the oscillation trip to the dispatchers and make recovery reachable

- TaskResponse serializes blocker_resolver_type: the dispatchers' HITL-blocked
  skip and the notification-path live-work check now actually fire over the
  wire instead of only against in-process rows.
- The oscillation marker clears on every human transition out of BLOCKED
  (snapshot or not), and the human unblock route treats a tripped task as
  the requested intervention: clears the marker and proceeds, while the
  agent gateway verb keeps refusing.
- The progress fingerprint includes the terminal-children count, so a
  coordination root whose children advanced between escalations resets
  instead of accruing toward a false trip.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
2026-07-24 17:20:20 +02:00
committed by GitHub
co-authored by Renn F
parent 23ae0ca217
commit eb0dcb6ecb
11 changed files with 990 additions and 66 deletions
+95 -1
View File
@@ -32,8 +32,17 @@ from roboco.api.schemas.tasks import (
task_to_response,
transform_update_data,
)
from roboco.models.base import Complexity, TaskNature, TaskStatus, TaskType, Team
from roboco.db.tables import TaskTable
from roboco.models.base import (
BlockerResolverType,
Complexity,
TaskNature,
TaskStatus,
TaskType,
Team,
)
from roboco.models.product import ProductCellMapping
from roboco.runtime.orchestrator import AgentOrchestrator
_ORDER_DEFAULT = 0
@@ -311,6 +320,7 @@ def _stub_task(*, with_project: bool = False) -> Any:
description="d",
acceptance_criteria=["a"],
status=TaskStatus.PENDING,
blocker_resolver_type=None,
priority=1,
sequence=0,
nature=TaskNature.TECHNICAL,
@@ -488,6 +498,90 @@ def test_task_list_to_response_returns_list() -> None:
assert len(out) == len(stubs)
# ---------------------------------------------------------------------------
# blocker_resolver_type — wire-shaped regression. A hand-rolled stub (like
# _stub_task above) can't catch a field TaskResponse silently drops; this
# builds a REAL TaskTable row and pushes it through the actual serialization
# path the orchestrator's dispatchers see over HTTP.
# ---------------------------------------------------------------------------
def _hitl_blocked_task_row() -> TaskTable:
"""A real TaskTable instance (never added to a session) in the exact
shape `unblock`'s oscillation-breaker trip leaves behind: BLOCKED with
blocker_resolver_type=HUMAN."""
return TaskTable(
id=uuid4(),
title="t",
description="d",
acceptance_criteria=["a"],
acceptance_criteria_ids=[],
parent_ac_refs=[],
status=TaskStatus.BLOCKED,
blocker_resolver_type=BlockerResolverType.HUMAN,
priority=2,
sequence=0,
nature=TaskNature.TECHNICAL,
task_type=TaskType.CODE,
project_id=None,
product_id=None,
docs_complete=False,
pr_created=False,
board_review_complete=False,
team=Team.BACKEND,
created_by=uuid4(),
assigned_to=None,
parent_task_id=None,
dependency_ids=[],
blocker_ids=[],
batch_id=None,
created_at=datetime.now(UTC),
updated_at=None,
claimed_at=None,
claimed_by=None,
started_at=None,
completed_at=None,
target_date=None,
estimated_complexity=Complexity.LOW,
plan=None,
checkpoints=[],
progress_updates=[],
commits=[],
documents=[],
dev_notes=None,
qa_notes=None,
auditor_notes=None,
self_verified=False,
qa_verified=None,
branch_name=None,
pr_number=None,
pr_url=None,
source="manual",
confirmed_by_human=True,
)
def test_task_to_response_serializes_blocker_resolver_type() -> None:
"""A real ORM row's blocker_resolver_type must round-trip — the field was
silently missing from TaskResponse, so every dispatcher reading it over
the wire saw None regardless of the DB value."""
row = _hitl_blocked_task_row()
resp = task_to_response(row)
assert resp.blocker_resolver_type == BlockerResolverType.HUMAN
def test_wire_shaped_hitl_blocked_task_trips_is_hitl_blocked() -> None:
"""End-to-end wire simulation: real TaskTable -> task_to_response ->
JSON-mode serialization (what httpx.json() hands the orchestrator) ->
AgentOrchestrator._is_hitl_blocked. Before the fix this always read
None over the wire and never fired."""
row = _hitl_blocked_task_row()
resp = task_to_response(row)
wire_dict = resp.model_dump(mode="json")
assert wire_dict["blocker_resolver_type"] == "human"
assert AgentOrchestrator._is_hitl_blocked(wire_dict) is True
# ---------------------------------------------------------------------------
# enrich_task_with_context — covers the work_session + project lookup branches.
# ---------------------------------------------------------------------------