Feat/provider overload break (#242)

* feat(conventions): standard schema models + effective-map merge

* feat(orchestrator): park provider on persistent server overload (529/500)

A 429 rate limit already parks a provider — queue its spawns, probe until it
recovers — but a persistent 529/500/503 overload had no such break: the run
died and the orchestrator crash-retried straight back into the overload,
burning tokens in a respawn loop.

Generalize the park to provider-unavailability. On a non-graceful Anthropic
agent exit, match the API's overload markers (overloaded_error /
internal_server_error / "API Error: 5xx") against the dead container's own
output and park the provider with kind="overloaded"; the existing spawn gate
already queues any parked provider, and the probe-resume loop revives the task
when it recovers. Grok keeps its exit-75 path; both now route through one
_park_provider_unavailable helper. Markers are kept specific so an agent that
merely writes about HTTP 500/529 can't trip the break.

Fix the recovery probe to require a 2xx: it treated any non-429 as recovered,
so a probe that itself got a 529 would have resumed agents straight back into
the overload — wrong for the new path and for a 429 that lifts into a 5xx.

Gated by ROBOCO_OVERLOAD_BREAK_ENABLED (default on; off => crash-retry).

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
2026-06-22 03:49:45 +02:00
committed by GitHub
co-authored by Renn F
parent ba8b877a05
commit 01e10ad693
14 changed files with 826 additions and 29 deletions
@@ -108,6 +108,21 @@ class TestActivateAndRead:
state = await tracker.get_state()
assert state["probe_failures"] == 0
async def test_activate_defaults_kind_to_rate_limited(self) -> None:
mock = _make_redis_mock()
tracker = _make_tracker(redis_mock=mock)
await tracker.activate()
state = await tracker.get_state()
assert state["kind"] == "rate_limited"
async def test_activate_stores_overloaded_kind(self) -> None:
mock = _make_redis_mock()
tracker = _make_tracker(redis_mock=mock)
await tracker.activate(kind="overloaded")
state = await tracker.get_state()
assert state["kind"] == "overloaded"
assert await tracker.is_rate_limited() is True # gates spawns either way
async def test_clear_removes_state(self) -> None:
mock = _make_redis_mock()
tracker = _make_tracker(redis_mock=mock)