Commit Graph
3 Commits
Author SHA1 Message Date
Renn F 75788f519c [067ce5d1] fix(runtime): detect session-limit 429 in Claude transcript for provider parking
The SDK server writes runtime output to /tmp/sdk-server.log inside the agent
container, so the session-limit markers never appeared in docker logs. Read
the newest durable Claude transcript from ~/.claude/projects as a fallback so
the provider gets parked and auto-revived instead of crash-retrying.

- Add _transcript_tail_text to read the agent's transcript tail
- Use it in _provider_rate_limit_park_target alongside docker logs
- Add regression test for transcript-only detection
2026-06-25 03:28:03 +02:00
40d685bd9f fix(run-hardening): park the workforce on a session-limit + PR-review verdict colour (#249)
* fix(orchestrator): park the provider on a Claude session-limit 429, not crash-loop

When the org Claude usage ("5-hour") session limit is hit, an agent container
exits non-zero with a 0-token 429 rejection. The provider-unavailable break only
recognized 5xx overload signatures (529/500/503), so a session-limit crash fell
through to the normal crash-retry path — the orchestrator respawned the agent
straight back into the limit, fleet-wide, until the window reset.

Add a sibling detector _provider_rate_limit_park_target that matches the
session-limit markers ("hit your session limit", "five_hour") in the dead
container's output and parks the provider with kind="rate_limited" (a longer
probe cadence), checked before the overload path in _handle_stopped_container.
Reuses the existing park-and-probe machinery, so the background probe loop
revives the parked tasks when the quota resets — no churn. Gated by the same
overload_break_enabled flag.

Also backfills the CHANGELOG Fixed entry for the orchestrator self-call auth fix
(merged in #248 without one).

* fix(panel): PR Reviewer Notes card colour reflects the verdict

The card was hardcoded teal/green regardless of the review verdict, so a Failed
review sat inside a green card and read as passing at a glance. Derive the card
background from the verdict (red on failed, green on approved/passed, amber on
changes-requested, neutral teal before a verdict) — mirroring the QA Notes card.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-06-24 03:20:14 +02:00
01e10ad693 Feat/provider overload break (#242)
* feat(conventions): standard schema models + effective-map merge

* feat(orchestrator): park provider on persistent server overload (529/500)

A 429 rate limit already parks a provider — queue its spawns, probe until it
recovers — but a persistent 529/500/503 overload had no such break: the run
died and the orchestrator crash-retried straight back into the overload,
burning tokens in a respawn loop.

Generalize the park to provider-unavailability. On a non-graceful Anthropic
agent exit, match the API's overload markers (overloaded_error /
internal_server_error / "API Error: 5xx") against the dead container's own
output and park the provider with kind="overloaded"; the existing spawn gate
already queues any parked provider, and the probe-resume loop revives the task
when it recovers. Grok keeps its exit-75 path; both now route through one
_park_provider_unavailable helper. Markers are kept specific so an agent that
merely writes about HTTP 500/529 can't trip the break.

Fix the recovery probe to require a 2xx: it treated any non-429 as recovered,
so a probe that itself got a 529 would have resumed agents straight back into
the overload — wrong for the new path and for a 429 that lifts into a 5xx.

Gated by ROBOCO_OVERLOAD_BREAK_ENABLED (default on; off => crash-retry).

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-06-22 03:49:45 +02:00