Persist the PM-respawn counter across orchestrator restarts (#275)

* feat(orchestrator): add respawn_tracker table + migration 051

Durable backing for AgentOrchestrator._pm_respawn_tracker (the PM-respawn
loop breaker). Kept only in memory it reset to count=1 on every restart,
re-burning the strike threshold against a still-wedged task. RespawnTrackerTable
mirrors WaitingRecordTable: composite PK (agent_slug, task_id) matching the
in-memory key; task_id is intentionally NOT a FK (the startup loader validates
against live tasks so a stale counter can't resurrect). Migration 051 verified
with a real alembic upgrade head + downgrade -1 + re-upgrade on Postgres.

* feat(orchestrator): persist the PM-respawn counter across restarts

The PM-respawn loop breaker (_pm_respawn_tracker) lived only in memory, so an
orchestrator restart reset a wedged task's strike count to 1 and re-burned the
whole threshold (4 spawns x container cost) before the gate fired again.

Write-through each gate mutation to the respawn_tracker table via a
fire-and-forget _schedule_respawn_persist (on the existing _bg_tasks strong-ref
set; a DB hiccup degrades to in-memory-only, never gates/un-gates a spawn), and
restore_respawn_tracker() repopulates the counter at startup, validating each
row against live tasks (drops terminal/missing) so a stale counter can't
resurrect against a fixed task. Best-effort + inert when the table is empty.
Cannot manufacture a spawn — the counter only ever suppresses one.

(_instances reconcile, the spec's other goal, already shipped as
_readopt_running_agents.)

* fix(types): cast Mapped[UUID] columns in project routes + self_heal

A clean `mypy roboco/ tests/` run surfaces 7 pre-existing errors in files this
branch doesn't touch: project-route handlers and self_heal_engine pass a
ProjectTable.id (declared Mapped[UUID] against SQLAlchemy's dialect UUID, so
mypy infers sqlalchemy.sql.sqltypes.UUID[Any]) where a uuid.UUID is expected.
An incremental .mypy_cache had hidden them. Apply the same targeted cast unblock
used for the prior batch; the deeper fix (migrating the ~88 Mapped[UUID]
columns to Mapped[uuid.UUID]) remains a separate dedicated task.

* docs(orchestrator): document respawn_tracker durability

Add the orchestrator runtime-state durability note to CLAUDE.md (respawn_tracker
write-through + restore; _instances reconciled-from-Docker) + the migration-051
narrative, and a CHANGELOG [Unreleased] Fixed entry. Also type-clean the new
respawn_tracker table test (cast __table__ to Table under TYPE_CHECKING).

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
2026-06-26 03:36:42 +02:00
committed by GitHub
co-authored by Renn F
parent 6f4c601ddf
commit e2f7097aab
10 changed files with 658 additions and 10 deletions
@@ -0,0 +1,52 @@
"""The respawn_tracker table — durable backing for the PM-respawn counter.
Mirrors WaitingRecordTable: a composite-PK row per (agent_slug, task_id) the
orchestrator's loop-breaker counter is keyed on, so it survives a restart
instead of resetting to count=1 and re-burning the strike threshold.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, cast
from roboco.db.tables import RespawnTrackerTable
if TYPE_CHECKING:
from sqlalchemy import Table
_TABLE = cast("Table", RespawnTrackerTable.__table__)
def test_table_name() -> None:
assert RespawnTrackerTable.__tablename__ == "respawn_tracker"
def test_composite_primary_key_is_agent_slug_and_task_id() -> None:
pk_cols = {col.name for col in _TABLE.primary_key.columns}
assert pk_cols == {"agent_slug", "task_id"}
def test_payload_columns_present() -> None:
cols = set(_TABLE.columns.keys())
assert {
"agent_slug",
"task_id",
"count",
"last_status",
"last_check",
"tracing_resets",
"notified",
"updated_at",
} <= cols
def test_task_id_has_no_foreign_key() -> None:
# Deliberately NOT a FK to tasks: the startup loader validates against live
# tasks instead, so a cascade can never silently resurrect/erase a counter.
task_id = _TABLE.columns["task_id"]
assert task_id.foreign_keys == set()
def test_last_check_index_present() -> None:
index_names = {idx.name for idx in _TABLE.indexes}
assert "ix_respawn_tracker_last_check" in index_names