mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
[aaac85d2] Rate limit guardrails for Anthropic and Ollama providers (#104)
* [25aa5b24] Implement rate-limit Zustand store, Axios interceptor, WebSocket hook, banner component, and page-load sync (#99) (#101) * [25aa5b24] feat(rate-limits): add types, Zustand store, Axios 429 interceptor, WS hook, sync hook, and banner component - panel/src/types/rate-limits.ts: RateLimitEntry, RateLimitHitEvent, RateLimitLiftedEvent, RateLimitApiResponse - panel/src/store/rate-limit-store.ts: useRateLimitStore with Map state, hitRateLimit/liftRateLimit/syncFromApi - panel/src/lib/api/rate-limits.ts: GET /api/system/rate-limits with isMockMode guard - panel/src/lib/api/client.ts: 429 interceptor dispatches to store first, Sonner toast on retry exhaustion - panel/src/hooks/use-rate-limit-websocket.ts: RATE_LIMIT_HIT/LIFTED events + onReconnect callback - panel/src/hooks/use-rate-limit-sync.ts: mount sync + no-op with console.warn when endpoint unavailable - panel/src/components/rate-limit/rate-limit-banner.tsx: amber rows with countdown, no dismiss button - panel/src/app/(dashboard)/layout.tsx: RateLimitBanner mounted below Header - store/index.ts, hooks/index.ts: export new store and hooks * [25aa5b24] fix(rate-limit-banner): use lint-clean countdown pattern (computeSecondsLeft outside render) * [25aa5b24] fix(client): add real retry loop to 429 interceptor so Sonner toast fires on exhaustion - Increment error.config._retryCount and return api(error.config) when retryCount < RATE_LIMIT_MAX_RETRIES, actually retrying the request. - Toast fires only when retryCount >= RATE_LIMIT_MAX_RETRIES (3 attempts). - Fixes AC4: toast was dead code because without return api(error.config) every 429 saw retryCount=1, permanently below the threshold of 3. --------- Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev> * [4112cd34] feat(rate-limit): add RateLimitError with 5-retry exponential backoff at all LLM call sites (#102) (#103) - Create roboco/services/exceptions.py with RateLimitError(provider, retry_after), HTTP_TOO_MANY_REQUESTS, MAX_RATE_LIMIT_RETRIES constants, and parse_retry_after_header() helper - extraction.py: extract _call_anthropic_with_retry() helper; retry Anthropic call 5x on 429 with exponential backoff; re-raise RateLimitError from outer except instead of swallowing it - ollama_embedder.py: 5-retry outer loop (429) wrapping existing 3-retry inner loop (ConnectError/Timeout) for all 4 call sites; two concerns kept isolated - indexes/base.py, mentor.py, validator.py: replace magic 429 literals with HTTP_TOO_MANY_REQUESTS; 5-retry loop on 429 for LLM calls - middleware.py: add rate_limit_exception_handler returning HTTP 429 with Retry-After response header - tests/unit/services/test_rate_limit_retry.py: 28 tests covering exhaustion, Retry-After header sleep, partial retries then success, ConnectError isolation Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> * [18107054] feat(rate-limit): Redis rate-limit state tracker + i_am_blocked rate_limited path (#105) (#106) - Add RateLimitStateTracker in roboco/services/gateway/rate_limit_tracker.py with activate(), clear(), is_rate_limited(), get_state(), increment_probe_failures(), reset_probe_failures() backed by redis.asyncio - Add RATE_LIMIT_HIT = "rate_limit.hit" to EventType StrEnum in events.py - Add _handle_rate_limited_parking() to Choreographer: intercepts i_am_blocked(reason='rate_limited') before block state transition, parks all active agents sharing affected provider via mark_waiting_long, publishes RATE_LIMIT_HIT event to StreamEventBus, task stays in_progress - Add get_provider_for_agent() and get_active_agent_slugs_for_provider() helper methods to AgentOrchestrator - Wire orchestrator and stream_bus into ChoreographerDeps via deps.py - Add test_rate_limit_tracker.py (basic ops, probe failures, cross-reconnection persistence, provider isolation) and test_i_am_blocked_rate_limited.py (AC3/AC4/AC5 coverage: task stays in_progress, mark_waiting_long call count equals active agent count, RATE_LIMIT_HIT event payload structure) Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> * [5501e4b4] Wire RateLimitStateTracker into live orchestrator paths — 4 CEO-identified integration gaps (#109) * [8451ca50] feat(gateway): wire RateLimitStateTracker.activate() into i_am_blocked rate-limited path and add provider-rate-limit gate to decide_spawn() (#107) - Add provider/provider_rate_limited optional fields to TriggerContext (backward-compatible defaults) - Insert rule 2 in decide_spawn(): QUEUE when trigger.provider_rate_limited is True with reason 'provider X rate-limited' - Call RateLimitStateTracker(provider).activate() in _handle_rate_limited_parking() after mark_waiting_long loop (wrapped in contextlib.suppress for Redis fault tolerance) - Extend gateway_pre_spawn_check() with optional provider param; check RateLimitStateTracker.is_rate_limited() when provider is known - Pass provider=self.get_provider_for_agent(agent_id) from orchestrator call site - Add TestProviderRateLimitGate (6 tests) to test_trigger_filter.py - Add TestRateLimitTrackerActivateOnParking (6 tests) to test_i_am_blocked_rate_limited.py - All 38 unit tests pass; ruff and mypy clean on changed files Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> * [e9cef0f0] feat(rate-limits): sweeper probe loop, CEO notification, and GET /api/system/rate-limits endpoint (AC4, AC8, AC9) (#108) - Add RATE_LIMIT_LIFTED event type to EventType enum in models/events.py - Add RateLimitStateTracker.list_rate_limited_providers() classmethod to scan Redis for all currently rate-limited providers (used by the new endpoint) - Add orchestrator._rate_limit_probe_loop(): background task started/stopped in start()/stop(), runs _sweep_rate_limit_probes() every 30s - Add orchestrator._probe_one_provider(): checks estimated_lift_at gate, calls _do_probe(); on success: tracker.clear(), resolve_wait() for all parked agents with waiting_for='rate_limit_lifted' matching the provider, publishes RATE_LIMIT_LIFTED event; on failure: increments probe_failures counter, sends CEO notification at threshold 10 (once per episode via _rate_limit_ceo_notified) - Add orchestrator._make_tracker(): injectable factory for RateLimitStateTracker - Add orchestrator._do_probe(): overridable async bool probe (default: True) - Add orchestrator._notify_rate_limit_ceo(): high-priority notification to CEO containing provider name, duration since activation, and paused agent count - Add roboco/api/routes/system.py with GET /rate-limits endpoint (AC9) - Register system_router in app.py under /api/system prefix - Add 17 unit tests in tests/unit/runtime/test_rate_limit_sweep.py covering all AC4/AC8/AC9 paths: probe success/failure, CEO threshold, endpoint schema Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> --------- Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> --------- Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev> Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
co-authored by
Backend Developer 1
Frontend Developer 1
Renn F
Backend Developer 1
Frontend Developer 1
Renn F
parent
cc4ccb7ea3
commit
98e618c243
@@ -219,6 +219,17 @@ class ChoreographerDeps:
|
||||
# callsites / tests that don't exercise Product routing don't have to plumb
|
||||
# it in; when None, delegate falls back to parent-project inheritance.
|
||||
product: Any = None
|
||||
# Orchestrator access for the rate-limited i_am_blocked path.
|
||||
# Implements get_provider_for_agent(slug) -> str | None,
|
||||
# get_active_agent_slugs_for_provider(provider) -> list[str], and
|
||||
# async mark_waiting_long(slug, waiting_for, task_id, context).
|
||||
# Optional: when None the parking step is skipped (e.g. in unit tests
|
||||
# that don't need to verify orchestrator interactions).
|
||||
orchestrator: Any = None
|
||||
# StreamEventBus for publishing RATE_LIMIT_HIT events.
|
||||
# Optional so existing callsites that don't exercise the rate-limit path
|
||||
# don't have to plumb it in.
|
||||
stream_bus: Any = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -360,6 +371,14 @@ class Choreographer:
|
||||
def product(self) -> Any:
|
||||
return self._deps.product
|
||||
|
||||
@property
|
||||
def orchestrator(self) -> Any:
|
||||
return self._deps.orchestrator
|
||||
|
||||
@property
|
||||
def stream_bus(self) -> Any:
|
||||
return self._deps.stream_bus
|
||||
|
||||
async def _touch(self, task_id: UUID | None) -> None:
|
||||
"""Best-effort heartbeat write; silent on missing task."""
|
||||
if task_id is not None:
|
||||
@@ -2188,6 +2207,114 @@ class Choreographer:
|
||||
)
|
||||
return updated, None
|
||||
|
||||
@staticmethod
|
||||
def _parse_retry_after(what_needed: str | None) -> float | None:
|
||||
"""Extract a retry-after seconds value from ``what_needed``, or None.
|
||||
|
||||
Agents may embed the Retry-After seconds in the ``what_needed``
|
||||
field as a numeric string (e.g. ``"30"`` or ``"60.5"``). This
|
||||
helper tries to parse it; any non-numeric or absent value returns
|
||||
``None``, which maps to the nullable ``retryAfterSeconds`` in the
|
||||
RATE_LIMIT_HIT event.
|
||||
"""
|
||||
if what_needed is None:
|
||||
return None
|
||||
try:
|
||||
return float(what_needed.strip())
|
||||
except (ValueError, AttributeError):
|
||||
return None
|
||||
|
||||
async def _handle_rate_limited_parking(
|
||||
self,
|
||||
agent_id: UUID,
|
||||
task_id: UUID,
|
||||
t: Any,
|
||||
agent: Any,
|
||||
role_str: str,
|
||||
briefing: dict[str, Any],
|
||||
what_needed: str | None,
|
||||
) -> Envelope:
|
||||
"""Rate-limited fast path: park agents, publish event, persist state.
|
||||
|
||||
Called from ``i_am_blocked`` when ``reason == 'rate_limited'``.
|
||||
The task stays in its current status (``in_progress``) — no block
|
||||
transition occurs. Instead, every orchestrator-tracked active agent
|
||||
sharing the same provider as the calling agent is parked via
|
||||
``mark_waiting_long(waiting_for='rate_limit_lifted')``. A
|
||||
``RATE_LIMIT_HIT`` event is published so downstream consumers (the
|
||||
orchestrator backpressure layer, the panel) can react.
|
||||
"""
|
||||
from roboco.models.events import Event, EventType
|
||||
|
||||
agent_slug: str | None = (
|
||||
getattr(agent, "slug", None) if agent is not None else None
|
||||
)
|
||||
|
||||
provider: str = "unknown"
|
||||
affected_agents: list[str] = []
|
||||
|
||||
orch = self.orchestrator
|
||||
if orch is not None and agent_slug is not None:
|
||||
with contextlib.suppress(Exception):
|
||||
prov = orch.get_provider_for_agent(agent_slug)
|
||||
if prov:
|
||||
provider = prov
|
||||
with contextlib.suppress(Exception):
|
||||
affected_agents = list(
|
||||
orch.get_active_agent_slugs_for_provider(provider)
|
||||
)
|
||||
for slug in affected_agents:
|
||||
with contextlib.suppress(Exception):
|
||||
await orch.mark_waiting_long(
|
||||
slug,
|
||||
waiting_for="rate_limit_lifted",
|
||||
task_id=str(task_id),
|
||||
context={"provider": provider, "triggered_by": agent_slug},
|
||||
)
|
||||
|
||||
retry_after_seconds = self._parse_retry_after(what_needed)
|
||||
|
||||
# Persist rate-limit state to Redis so downstream decide_spawn()
|
||||
# calls can gate new spawns for this provider. Skipped when the
|
||||
# provider is "unknown" (orchestrator not wired or not tracking the
|
||||
# agent) to avoid polluting the tracker with meaningless keys.
|
||||
if provider != "unknown":
|
||||
with contextlib.suppress(Exception):
|
||||
from roboco.services.gateway.rate_limit_tracker import (
|
||||
RateLimitStateTracker,
|
||||
)
|
||||
|
||||
await RateLimitStateTracker(provider).activate(
|
||||
retry_after=retry_after_seconds,
|
||||
affected_agents=affected_agents,
|
||||
)
|
||||
|
||||
bus = self.stream_bus
|
||||
if bus is not None:
|
||||
event = Event(
|
||||
type=EventType.RATE_LIMIT_HIT,
|
||||
data={
|
||||
"provider": provider,
|
||||
"affectedAgents": affected_agents,
|
||||
"retryAfterSeconds": retry_after_seconds,
|
||||
"timestamp": datetime.now(UTC).isoformat(),
|
||||
},
|
||||
source_agent=str(agent_id),
|
||||
)
|
||||
with contextlib.suppress(Exception):
|
||||
await bus.publish(event)
|
||||
|
||||
await self._touch(task_id)
|
||||
return Envelope.ok(
|
||||
status=str(t.status),
|
||||
task_id=str(task_id),
|
||||
next=(
|
||||
"agent parked waiting for rate_limit_lifted; "
|
||||
"will be respawned when the limit clears"
|
||||
),
|
||||
context_briefing=briefing,
|
||||
).with_introspection(task=t, role=role_str)
|
||||
|
||||
async def i_am_blocked(
|
||||
self,
|
||||
agent_id: UUID,
|
||||
@@ -2202,8 +2329,15 @@ class Choreographer:
|
||||
membership (developer/qa/documenter) and the source-status
|
||||
constraint of the composed ``block`` action (in_progress only).
|
||||
After the spec gate accepts, the journal:struggle entry is written
|
||||
from the verb body, then ``VerbRunner.run_intent("i_am_blocked", ...)``
|
||||
dispatches the (block,) atomic chain wrapped in a savepoint.
|
||||
from the verb body, then either:
|
||||
|
||||
- ``reason == 'rate_limited'``: the task is **not** transitioned to
|
||||
``blocked``; instead every active agent on the same provider is
|
||||
parked via ``mark_waiting_long(waiting_for='rate_limit_lifted')``
|
||||
and a ``RATE_LIMIT_HIT`` event is published.
|
||||
- any other reason: ``VerbRunner.run_intent("i_am_blocked", ...)``
|
||||
dispatches the ``(block,)`` atomic chain wrapped in a savepoint,
|
||||
transitioning the task to ``blocked``.
|
||||
"""
|
||||
t = await self.task.get(task_id)
|
||||
if t is None:
|
||||
@@ -2253,6 +2387,20 @@ class Choreographer:
|
||||
task_id=task_id,
|
||||
content=self._build_struggle_body(reason, blocker_type, what_needed),
|
||||
)
|
||||
|
||||
# Rate-limited fast path: skip the block state transition and park
|
||||
# all affected agents instead.
|
||||
if reason.strip().lower() == "rate_limited":
|
||||
return await self._handle_rate_limited_parking(
|
||||
agent_id=agent_id,
|
||||
task_id=task_id,
|
||||
t=t,
|
||||
agent=agent,
|
||||
role_str=role_str,
|
||||
briefing=briefing,
|
||||
what_needed=what_needed,
|
||||
)
|
||||
|
||||
t, rejection = await self._run_i_am_blocked_intent(
|
||||
agent_id, task_id, t, agent, spec_ctx, role_str, briefing
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user