mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
Feat/provider overload break (#242)
* feat(conventions): standard schema models + effective-map merge * feat(orchestrator): park provider on persistent server overload (529/500) A 429 rate limit already parks a provider — queue its spawns, probe until it recovers — but a persistent 529/500/503 overload had no such break: the run died and the orchestrator crash-retried straight back into the overload, burning tokens in a respawn loop. Generalize the park to provider-unavailability. On a non-graceful Anthropic agent exit, match the API's overload markers (overloaded_error / internal_server_error / "API Error: 5xx") against the dead container's own output and park the provider with kind="overloaded"; the existing spawn gate already queues any parked provider, and the probe-resume loop revives the task when it recovers. Grok keeps its exit-75 path; both now route through one _park_provider_unavailable helper. Markers are kept specific so an agent that merely writes about HTTP 500/529 can't trip the break. Fix the recovery probe to require a 2xx: it treated any non-429 as recovered, so a probe that itself got a 529 would have resumed agents straight back into the overload — wrong for the new path and for a 429 that lifts into a 5xx. Gated by ROBOCO_OVERLOAD_BREAK_ENABLED (default on; off => crash-retry). --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
@@ -108,6 +108,21 @@ class TestActivateAndRead:
|
||||
state = await tracker.get_state()
|
||||
assert state["probe_failures"] == 0
|
||||
|
||||
async def test_activate_defaults_kind_to_rate_limited(self) -> None:
|
||||
mock = _make_redis_mock()
|
||||
tracker = _make_tracker(redis_mock=mock)
|
||||
await tracker.activate()
|
||||
state = await tracker.get_state()
|
||||
assert state["kind"] == "rate_limited"
|
||||
|
||||
async def test_activate_stores_overloaded_kind(self) -> None:
|
||||
mock = _make_redis_mock()
|
||||
tracker = _make_tracker(redis_mock=mock)
|
||||
await tracker.activate(kind="overloaded")
|
||||
state = await tracker.get_state()
|
||||
assert state["kind"] == "overloaded"
|
||||
assert await tracker.is_rate_limited() is True # gates spawns either way
|
||||
|
||||
async def test_clear_removes_state(self) -> None:
|
||||
mock = _make_redis_mock()
|
||||
tracker = _make_tracker(redis_mock=mock)
|
||||
|
||||
Reference in New Issue
Block a user