mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
Persist the PM-respawn counter across orchestrator restarts (#275)
* feat(orchestrator): add respawn_tracker table + migration 051 Durable backing for AgentOrchestrator._pm_respawn_tracker (the PM-respawn loop breaker). Kept only in memory it reset to count=1 on every restart, re-burning the strike threshold against a still-wedged task. RespawnTrackerTable mirrors WaitingRecordTable: composite PK (agent_slug, task_id) matching the in-memory key; task_id is intentionally NOT a FK (the startup loader validates against live tasks so a stale counter can't resurrect). Migration 051 verified with a real alembic upgrade head + downgrade -1 + re-upgrade on Postgres. * feat(orchestrator): persist the PM-respawn counter across restarts The PM-respawn loop breaker (_pm_respawn_tracker) lived only in memory, so an orchestrator restart reset a wedged task's strike count to 1 and re-burned the whole threshold (4 spawns x container cost) before the gate fired again. Write-through each gate mutation to the respawn_tracker table via a fire-and-forget _schedule_respawn_persist (on the existing _bg_tasks strong-ref set; a DB hiccup degrades to in-memory-only, never gates/un-gates a spawn), and restore_respawn_tracker() repopulates the counter at startup, validating each row against live tasks (drops terminal/missing) so a stale counter can't resurrect against a fixed task. Best-effort + inert when the table is empty. Cannot manufacture a spawn — the counter only ever suppresses one. (_instances reconcile, the spec's other goal, already shipped as _readopt_running_agents.) * fix(types): cast Mapped[UUID] columns in project routes + self_heal A clean `mypy roboco/ tests/` run surfaces 7 pre-existing errors in files this branch doesn't touch: project-route handlers and self_heal_engine pass a ProjectTable.id (declared Mapped[UUID] against SQLAlchemy's dialect UUID, so mypy infers sqlalchemy.sql.sqltypes.UUID[Any]) where a uuid.UUID is expected. An incremental .mypy_cache had hidden them. Apply the same targeted cast unblock used for the prior batch; the deeper fix (migrating the ~88 Mapped[UUID] columns to Mapped[uuid.UUID]) remains a separate dedicated task. * docs(orchestrator): document respawn_tracker durability Add the orchestrator runtime-state durability note to CLAUDE.md (respawn_tracker write-through + restore; _instances reconciled-from-Docker) + the migration-051 narrative, and a CHANGELOG [Unreleased] Fixed entry. Also type-clean the new respawn_tracker table test (cast __table__ to Table under TYPE_CHECKING). --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
@@ -0,0 +1,52 @@
|
||||
"""The respawn_tracker table — durable backing for the PM-respawn counter.
|
||||
|
||||
Mirrors WaitingRecordTable: a composite-PK row per (agent_slug, task_id) the
|
||||
orchestrator's loop-breaker counter is keyed on, so it survives a restart
|
||||
instead of resetting to count=1 and re-burning the strike threshold.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, cast
|
||||
|
||||
from roboco.db.tables import RespawnTrackerTable
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from sqlalchemy import Table
|
||||
|
||||
_TABLE = cast("Table", RespawnTrackerTable.__table__)
|
||||
|
||||
|
||||
def test_table_name() -> None:
|
||||
assert RespawnTrackerTable.__tablename__ == "respawn_tracker"
|
||||
|
||||
|
||||
def test_composite_primary_key_is_agent_slug_and_task_id() -> None:
|
||||
pk_cols = {col.name for col in _TABLE.primary_key.columns}
|
||||
assert pk_cols == {"agent_slug", "task_id"}
|
||||
|
||||
|
||||
def test_payload_columns_present() -> None:
|
||||
cols = set(_TABLE.columns.keys())
|
||||
assert {
|
||||
"agent_slug",
|
||||
"task_id",
|
||||
"count",
|
||||
"last_status",
|
||||
"last_check",
|
||||
"tracing_resets",
|
||||
"notified",
|
||||
"updated_at",
|
||||
} <= cols
|
||||
|
||||
|
||||
def test_task_id_has_no_foreign_key() -> None:
|
||||
# Deliberately NOT a FK to tasks: the startup loader validates against live
|
||||
# tasks instead, so a cascade can never silently resurrect/erase a counter.
|
||||
task_id = _TABLE.columns["task_id"]
|
||||
assert task_id.foreign_keys == set()
|
||||
|
||||
|
||||
def test_last_check_index_present() -> None:
|
||||
index_names = {idx.name for idx in _TABLE.indexes}
|
||||
assert "ix_respawn_tracker_last_check" in index_names
|
||||
Reference in New Issue
Block a user