Feat/provider overload break (#242)

* feat(conventions): standard schema models + effective-map merge

* feat(orchestrator): park provider on persistent server overload (529/500)

A 429 rate limit already parks a provider — queue its spawns, probe until it
recovers — but a persistent 529/500/503 overload had no such break: the run
died and the orchestrator crash-retried straight back into the overload,
burning tokens in a respawn loop.

Generalize the park to provider-unavailability. On a non-graceful Anthropic
agent exit, match the API's overload markers (overloaded_error /
internal_server_error / "API Error: 5xx") against the dead container's own
output and park the provider with kind="overloaded"; the existing spawn gate
already queues any parked provider, and the probe-resume loop revives the task
when it recovers. Grok keeps its exit-75 path; both now route through one
_park_provider_unavailable helper. Markers are kept specific so an agent that
merely writes about HTTP 500/529 can't trip the break.

Fix the recovery probe to require a 2xx: it treated any non-429 as recovered,
so a probe that itself got a 529 would have resumed agents straight back into
the overload — wrong for the new path and for a 429 that lifts into a 5xx.

Gated by ROBOCO_OVERLOAD_BREAK_ENABLED (default on; off => crash-retry).

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
2026-06-22 03:49:45 +02:00
committed by GitHub
co-authored by Renn F
parent ba8b877a05
commit 01e10ad693
14 changed files with 826 additions and 29 deletions
+9 -1
View File
@@ -35,10 +35,17 @@ class _FakeTracker:
async def is_rate_limited(self) -> bool:
return self._limited
async def activate(self, *, retry_after: float, affected_agents: list[str]) -> None:
async def activate(
self,
*,
retry_after: float,
affected_agents: list[str],
kind: str = "rate_limited",
) -> None:
self.activated_with = {
"retry_after": retry_after,
"affected_agents": affected_agents,
"kind": kind,
}
@@ -115,6 +122,7 @@ async def test_park_grok_rate_limited_activates_and_offlines(
assert tracker.activated_with == {
"retry_after": pytest.approx(60.0),
"affected_agents": ["be-dev-1"],
"kind": "rate_limited",
}