Post-audit sweep over the 135 audit-fix commits since 19a474d3:
1. Stripped every # Fxxx: audit-ID token from comments AND every Fxxx token
from docstring openings across 211 blocks / ~626 lines. The CEO flagged
these twice: audit-issue IDs in code confuse future devs/agents. The
descriptive text is preserved; only the Fxxx token is removed (and bloated
narrative blocks trimmed to 1-3 lines keeping the one non-obvious invariant).
2. Trimmed bloated comments/docstrings to the concise standard (1-3 lines).
3. Added missing behavior-change docs for the audit-fix batch: prompts/roles
(documenter, pr_reviewer, qa), user-facing docs (api auth, websockets,
agent-gateway, megatask, merge-model, task-lifecycle, grok, resilience,
conventions, panel, security, troubleshooting), and the RAG corpus (cell-pm,
main-pm, pr-reviewer, qa roles; conventions; messaging-tools; escalation;
megatask; task-claiming workflows).
Comment/docstring/prose ONLY — zero code-line edits (verified: the diff
contains no def/class/return/if/for/await/assignment/call lines). Gates green:
ruff format + ruff check clean, mypy clean on roboco/. The only pytest failures
are the pre-existing sync_branch tracing-decision gap (B1, 250be5c2) — not
sweep-caused and tracked separately.
Convention: no type:ignore/noqa, and pre-existing violations still
violate. The make-quality gate runs 'mypy roboco/ tests/', but the
prior commits' gates only ran mypy on production files, masking 64
type errors across 15 test files (method-assign, unused-ignore,
no-untyped-def, attr-defined, union-attr, has-type, index, misc).
Fixed without any type:ignore:
- method-assign (svc.session.X = / svc.method = AsyncMock()): hold a
local 'session: MagicMock'/'AsyncMock' and assert on it, or stub via
object.__setattr__ / monkeypatch / a typed '_bind' helper returning
Any, or alias 'cc: Any = c' (the pattern the file already used).
- unused 'type: ignore[assignment]' (real code was method-assign):
removed; replaced with the no-suppression patterns above.
- 'Callable[...] has no attribute assert_*': keep a typed local ref to
the AsyncMock and assert on the local, not the method-typed attr.
- no-untyped-def: annotate helper params (Any / pytest.MonkeyPatch).
- attr-defined / index / union-attr: type the helper as Any, narrow
with an 'is not None' assert, or add the missing attr to a fake.
- has-type / return-value: fix the declared return type to the tuple
the function actually returns.
- PLC0415 inline imports: hoisted to top-level.
test_pr_gate_notifies_pm._stub_gate_path converted fully to the
'cc: Any = c' alias (it already used it for one attr) so its five
'# type: ignore[method-assign]' suppressions are gone.
mypy tests/: 64 errors -> 0 (538 files). ruff check tests/: clean.
All 84 tests in the touched files pass.
The bare 'error 529'/'error 500'/'error 503' markers were broad enough to
false-match an agent that merely writes about an HTTP status code in its own
notes ('the endpoint returned error 500, retrying'), parking the whole
Anthropic fleet on a non-issue.
The SDK error formatter emits 'API Error: NNN' + a JSON error type, so the
remaining 'api error: 529/500/503' + 'overloaded_error' +
'internal_server_error' markers cover every real overload without that
false-match surface.
The SDK server writes model-API errors (529/500/503) to /tmp/sdk-server.log,
not stdout, so an overload marker can appear only in the durable Claude
transcript — the same rationale already applied to the session-limit
detector. _provider_overload_park_target read only docker logs, so an
overload was missed and the agent crash-respawned straight back into it.
Now concatenates the transcript tail before matching, mirroring the
rate-limit path.
_park_provider_unavailable parked the provider + offlined the instance but
never registered a WaitingRecord, so _on_probe_success -> _parked_agents_for
(always filtered on waiting_for=='rate_limit_lifted') returned [] and
resolve_wait revived nobody — recovery fell to the 600s stale-claim reaper
instead of the probe-success path the parking design relied on.
Register + persist a rate_limit_lifted WaitingRecord at park time (mirrors
mark_waiting_long, minus stop_agent — the container is already dead).
Companion reaper guard: _reap_with_service now skips provider-parked
assignees (_assignee_is_provider_parked) so the claim survives until the
probe revives the agent — otherwise the reaper releases the claim to pending
and probe-success respawns on a task the agent no longer owns.
The SDK server writes runtime output to /tmp/sdk-server.log inside the agent
container, so the session-limit markers never appeared in docker logs. Read
the newest durable Claude transcript from ~/.claude/projects as a fallback so
the provider gets parked and auto-revived instead of crash-retrying.
- Add _transcript_tail_text to read the agent's transcript tail
- Use it in _provider_rate_limit_park_target alongside docker logs
- Add regression test for transcript-only detection
* fix(orchestrator): park the provider on a Claude session-limit 429, not crash-loop
When the org Claude usage ("5-hour") session limit is hit, an agent container
exits non-zero with a 0-token 429 rejection. The provider-unavailable break only
recognized 5xx overload signatures (529/500/503), so a session-limit crash fell
through to the normal crash-retry path — the orchestrator respawned the agent
straight back into the limit, fleet-wide, until the window reset.
Add a sibling detector _provider_rate_limit_park_target that matches the
session-limit markers ("hit your session limit", "five_hour") in the dead
container's output and parks the provider with kind="rate_limited" (a longer
probe cadence), checked before the overload path in _handle_stopped_container.
Reuses the existing park-and-probe machinery, so the background probe loop
revives the parked tasks when the quota resets — no churn. Gated by the same
overload_break_enabled flag.
Also backfills the CHANGELOG Fixed entry for the orchestrator self-call auth fix
(merged in #248 without one).
* fix(panel): PR Reviewer Notes card colour reflects the verdict
The card was hardcoded teal/green regardless of the review verdict, so a Failed
review sat inside a green card and read as passing at a glance. Derive the card
background from the verdict (red on failed, green on approved/passed, amber on
changes-requested, neutral teal before a verdict) — mirroring the QA Notes card.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
* feat(conventions): standard schema models + effective-map merge
* feat(orchestrator): park provider on persistent server overload (529/500)
A 429 rate limit already parks a provider — queue its spawns, probe until it
recovers — but a persistent 529/500/503 overload had no such break: the run
died and the orchestrator crash-retried straight back into the overload,
burning tokens in a respawn loop.
Generalize the park to provider-unavailability. On a non-graceful Anthropic
agent exit, match the API's overload markers (overloaded_error /
internal_server_error / "API Error: 5xx") against the dead container's own
output and park the provider with kind="overloaded"; the existing spawn gate
already queues any parked provider, and the probe-resume loop revives the task
when it recovers. Grok keeps its exit-75 path; both now route through one
_park_provider_unavailable helper. Markers are kept specific so an agent that
merely writes about HTTP 500/529 can't trip the break.
Fix the recovery probe to require a 2xx: it treated any non-429 as recovered,
so a probe that itself got a 529 would have resumed agents straight back into
the overload — wrong for the new path and for a 429 that lifts into a 5xx.
Gated by ROBOCO_OVERLOAD_BREAK_ENABLED (default on; off => crash-retry).
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
Renn F