mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
e2f7097aab
* feat(orchestrator): add respawn_tracker table + migration 051 Durable backing for AgentOrchestrator._pm_respawn_tracker (the PM-respawn loop breaker). Kept only in memory it reset to count=1 on every restart, re-burning the strike threshold against a still-wedged task. RespawnTrackerTable mirrors WaitingRecordTable: composite PK (agent_slug, task_id) matching the in-memory key; task_id is intentionally NOT a FK (the startup loader validates against live tasks so a stale counter can't resurrect). Migration 051 verified with a real alembic upgrade head + downgrade -1 + re-upgrade on Postgres. * feat(orchestrator): persist the PM-respawn counter across restarts The PM-respawn loop breaker (_pm_respawn_tracker) lived only in memory, so an orchestrator restart reset a wedged task's strike count to 1 and re-burned the whole threshold (4 spawns x container cost) before the gate fired again. Write-through each gate mutation to the respawn_tracker table via a fire-and-forget _schedule_respawn_persist (on the existing _bg_tasks strong-ref set; a DB hiccup degrades to in-memory-only, never gates/un-gates a spawn), and restore_respawn_tracker() repopulates the counter at startup, validating each row against live tasks (drops terminal/missing) so a stale counter can't resurrect against a fixed task. Best-effort + inert when the table is empty. Cannot manufacture a spawn — the counter only ever suppresses one. (_instances reconcile, the spec's other goal, already shipped as _readopt_running_agents.) * fix(types): cast Mapped[UUID] columns in project routes + self_heal A clean `mypy roboco/ tests/` run surfaces 7 pre-existing errors in files this branch doesn't touch: project-route handlers and self_heal_engine pass a ProjectTable.id (declared Mapped[UUID] against SQLAlchemy's dialect UUID, so mypy infers sqlalchemy.sql.sqltypes.UUID[Any]) where a uuid.UUID is expected. An incremental .mypy_cache had hidden them. Apply the same targeted cast unblock used for the prior batch; the deeper fix (migrating the ~88 Mapped[UUID] columns to Mapped[uuid.UUID]) remains a separate dedicated task. * docs(orchestrator): document respawn_tracker durability Add the orchestrator runtime-state durability note to CLAUDE.md (respawn_tracker write-through + restore; _instances reconciled-from-Docker) + the migration-051 narrative, and a CHANGELOG [Unreleased] Fixed entry. Also type-clean the new respawn_tracker table test (cast __table__ to Table under TYPE_CHECKING). --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>
52 lines
1.9 KiB
Python
52 lines
1.9 KiB
Python
"""Add the respawn_tracker table — durable PM-respawn loop counter.
|
|
|
|
``AgentOrchestrator._pm_respawn_tracker`` is the circuit breaker against
|
|
respawning the same PM on the same task forever. Kept only in memory it reset
|
|
to ``count=1`` on every orchestrator restart, re-burning the whole strike
|
|
threshold against a still-wedged task. This table is its write-through mirror,
|
|
restored at startup. Composite PK ``(agent_slug, task_id)`` matches the
|
|
in-memory dict key. ``task_id`` is deliberately NOT a FK to ``tasks``: the
|
|
startup loader validates against live tasks instead, so a stale counter cannot
|
|
resurrect against a fixed/deleted task.
|
|
|
|
Revision ID: 051_respawn_tracker
|
|
Revises: 050_playbooks
|
|
Create Date: 2026-06-26
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import sqlalchemy as sa
|
|
from alembic import op
|
|
|
|
revision = "051_respawn_tracker"
|
|
down_revision = "050_playbooks"
|
|
branch_labels = None
|
|
depends_on = None
|
|
|
|
|
|
def upgrade() -> None:
|
|
op.create_table(
|
|
"respawn_tracker",
|
|
sa.Column("agent_slug", sa.String(length=64), nullable=False),
|
|
sa.Column("task_id", sa.UUID(as_uuid=True), nullable=False),
|
|
sa.Column("count", sa.Integer(), nullable=False, server_default="1"),
|
|
sa.Column("last_status", sa.String(length=64), nullable=True),
|
|
sa.Column("last_check", sa.DateTime(timezone=True), nullable=False),
|
|
sa.Column("tracing_resets", sa.Integer(), nullable=False, server_default="0"),
|
|
sa.Column("notified", sa.Boolean(), nullable=False, server_default=sa.false()),
|
|
sa.Column(
|
|
"updated_at",
|
|
sa.DateTime(timezone=True),
|
|
nullable=False,
|
|
server_default=sa.text("now()"),
|
|
),
|
|
sa.PrimaryKeyConstraint("agent_slug", "task_id"),
|
|
)
|
|
op.create_index("ix_respawn_tracker_last_check", "respawn_tracker", ["last_check"])
|
|
|
|
|
|
def downgrade() -> None:
|
|
op.drop_index("ix_respawn_tracker_last_check", table_name="respawn_tracker")
|
|
op.drop_table("respawn_tracker")
|