mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
7c8453e210
* fix(notifications): exponential backoff + CAS claim for expired-unacked re-escalation The sweep re-escalated every expired unacked ack-required notification on every ~60s tick, forever — the live incident: 3 fresh blocker escalations + Telegram DMs per minute from a static stale pile. Now each notification carries reescalation_count / last_reescalated_at / reescalation_delivered_count (migration 079): first fire at expiry, then doubling intervals from 1h capped at 24h, hard stop after ROBOCO_NOTIFICATION_MAX_REESCALATIONS (default 5) with one permanent log carrying attempts-vs-delivered so 'seen and ignored' is distinguishable from 'route never worked'. The due/wait/capped decision is a pure function in foundation/policy/communications.py. Per adversarial review, the attempt slot is claimed by compare-and-set (UPDATE ... WHERE reescalation_count = :n) BEFORE delivery — the previous draft leaned on the 60s dedup window, which never engages for BLOCKER_ESCALATION (_LOOP_PRONE_TYPES excludes it), so concurrent sweeps would have double-delivered. A lost claim skips delivery outright. Legacy rows read as count=0 and keep today's first-fire semantics. 61 tests incl. a two-session CAS race and a real alembic upgrade/downgrade round trip. * feat(budgets): per-task and per-project cost budgets (flag-gated) tasks.budget_usd + projects.monthly_budget_usd (migration 080, chained on 079; adds ix_agent_spawn_sessions_task_id since both enforcement seams filter on bare task_id). Behind ROBOCO_TASK_BUDGETS_ENABLED (default off, feature-flags card) — verifiably inert when off. Claim-time: a project-month-spend guard applies to WORK-STARTING claims only (i_will_work_on / i_will_plan) — per adversarial review, review/ doc/gate/inbound-PR claims are exempt so in-flight work can always finish reviewing and merging at cap. Spend counts closed sessions' estimated_cost_usd PLUS open sessions priced live from token snapshots (the original closed-only sum read parallel long sessions as $0). Sweep-side: the existing budget sweep also prices the active task's spend vs budget_usd (TaskType defaults when null); on breach the task is BLOCKED (HUMAN resolver, budget marker) BEFORE the graceful stop so the unclaim no-ops and the dispatcher never respawns onto it, and the CEO notification names both recovery steps. unblock on a budget-blocked task re-checks live spend and refuses while still over — no silent re-breach loop. Panel: budget inputs in both dialogs (0 rejected — a zero budget silently blocks everything), spend logic consolidated in TaskService.task_spend_usd. 42 new tests incl. a real-DB spend-query suite and a two-tick non-refire sweep test. --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>
60 lines
2.2 KiB
Python
60 lines
2.2 KiB
Python
"""Per-task and per-project cost budgets (feature-flagged, default off).
|
|
|
|
``tasks.budget_usd`` caps one task's own accumulated agent-spawn spend;
|
|
``projects.monthly_budget_usd`` caps a project's calendar-month spend across
|
|
all its tasks. Both nullable and additive — null means "no cap" and is a pure
|
|
no-op regardless of ``ROBOCO_TASK_BUDGETS_ENABLED`` (the flag itself gates
|
|
whether the caps are ever consulted at all). Mirrors the ``ci_watch_enabled``
|
|
per-project opt-in shape (migration 048): the column exists unconditionally,
|
|
the feature flag decides whether anything reads it.
|
|
|
|
``ix_agent_spawn_sessions_task_id`` rides along: both the claim-time monthly-
|
|
spend query (``TaskService.project_month_spend_usd``) and the orchestrator's
|
|
per-tick task-spend sweep (``_task_spend_usd``) filter this table by
|
|
``task_id`` with no existing index — an unbounded per-row scan every minute
|
|
once the flag is armed. Added here rather than a separate migration since it
|
|
exists to serve these same two new read paths.
|
|
|
|
Revision ID: 080_task_project_budgets
|
|
Revises: 079_notification_backoff
|
|
Create Date: 2026-07-22
|
|
|
|
NOTE: down_revision was re-chained from 078_project_codegen_command to
|
|
079_notification_backoff (PR #652 landed first at integration) — the
|
|
079_task_project_budgets -> 080_task_project_budgets rename + re-chain is
|
|
exactly the "may be re-chained at integration" case flagged in the original
|
|
docstring, not a fork of the tree.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import sqlalchemy as sa
|
|
from alembic import op
|
|
|
|
revision = "080_task_project_budgets"
|
|
down_revision = "079_notification_backoff"
|
|
branch_labels: dict[str, str] | None = None
|
|
depends_on: dict[str, str] | None = None
|
|
|
|
|
|
def upgrade() -> None:
|
|
op.add_column(
|
|
"tasks",
|
|
sa.Column("budget_usd", sa.Float(), nullable=True),
|
|
)
|
|
op.add_column(
|
|
"projects",
|
|
sa.Column("monthly_budget_usd", sa.Float(), nullable=True),
|
|
)
|
|
op.create_index(
|
|
"ix_agent_spawn_sessions_task_id",
|
|
"agent_spawn_sessions",
|
|
["task_id"],
|
|
)
|
|
|
|
|
|
def downgrade() -> None:
|
|
op.drop_index("ix_agent_spawn_sessions_task_id", table_name="agent_spawn_sessions")
|
|
op.drop_column("projects", "monthly_budget_usd")
|
|
op.drop_column("tasks", "budget_usd")
|