mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
Feat/provider overload break (#242)
* feat(conventions): standard schema models + effective-map merge * feat(orchestrator): park provider on persistent server overload (529/500) A 429 rate limit already parks a provider — queue its spawns, probe until it recovers — but a persistent 529/500/503 overload had no such break: the run died and the orchestrator crash-retried straight back into the overload, burning tokens in a respawn loop. Generalize the park to provider-unavailability. On a non-graceful Anthropic agent exit, match the API's overload markers (overloaded_error / internal_server_error / "API Error: 5xx") against the dead container's own output and park the provider with kind="overloaded"; the existing spawn gate already queues any parked provider, and the probe-resume loop revives the task when it recovers. Grok keeps its exit-75 path; both now route through one _park_provider_unavailable helper. Markers are kept specific so an agent that merely writes about HTTP 500/529 can't trip the break. Fix the recovery probe to require a 2xx: it treated any non-429 as recovered, so a probe that itself got a 529 would have resumed agents straight back into the overload — wrong for the new path and for a 429 that lifts into a 5xx. Gated by ROBOCO_OVERLOAD_BREAK_ENABLED (default on; off => crash-retry). --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
@@ -35,10 +35,17 @@ class _FakeTracker:
|
||||
async def is_rate_limited(self) -> bool:
|
||||
return self._limited
|
||||
|
||||
async def activate(self, *, retry_after: float, affected_agents: list[str]) -> None:
|
||||
async def activate(
|
||||
self,
|
||||
*,
|
||||
retry_after: float,
|
||||
affected_agents: list[str],
|
||||
kind: str = "rate_limited",
|
||||
) -> None:
|
||||
self.activated_with = {
|
||||
"retry_after": retry_after,
|
||||
"affected_agents": affected_agents,
|
||||
"kind": kind,
|
||||
}
|
||||
|
||||
|
||||
@@ -115,6 +122,7 @@ async def test_park_grok_rate_limited_activates_and_offlines(
|
||||
assert tracker.activated_with == {
|
||||
"retry_after": pytest.approx(60.0),
|
||||
"affected_agents": ["be-dev-1"],
|
||||
"kind": "rate_limited",
|
||||
}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user