[aaac85d2] Rate limit guardrails for Anthropic and Ollama providers (#104)

* [25aa5b24] Implement rate-limit Zustand store, Axios interceptor, WebSocket hook, banner component, and page-load sync (#99) (#101) * [25aa5b24] feat(rate-limits): add types, Zustand store, Axios 429 interceptor, WS hook, sync hook, and banner component - panel/src/types/rate-limits.ts: RateLimitEntry, RateLimitHitEvent, RateLimitLiftedEvent, RateLimitApiResponse - panel/src/store/rate-limit-store.ts: useRateLimitStore with Map state, hitRateLimit/liftRateLimit/syncFromApi - panel/src/lib/api/rate-limits.ts: GET /api/system/rate-limits with isMockMode guard - panel/src/lib/api/client.ts: 429 interceptor dispatches to store first, Sonner toast on retry exhaustion - panel/src/hooks/use-rate-limit-websocket.ts: RATE_LIMIT_HIT/LIFTED events + onReconnect callback - panel/src/hooks/use-rate-limit-sync.ts: mount sync + no-op with console.warn when endpoint unavailable - panel/src/components/rate-limit/rate-limit-banner.tsx: amber rows with countdown, no dismiss button - panel/src/app/(dashboard)/layout.tsx: RateLimitBanner mounted below Header - store/index.ts, hooks/index.ts: export new store and hooks * [25aa5b24] fix(rate-limit-banner): use lint-clean countdown pattern (computeSecondsLeft outside render) * [25aa5b24] fix(client): add real retry loop to 429 interceptor so Sonner toast fires on exhaustion - Increment error.config._retryCount and return api(error.config) when retryCount < RATE_LIMIT_MAX_RETRIES, actually retrying the request. - Toast fires only when retryCount >= RATE_LIMIT_MAX_RETRIES (3 attempts). - Fixes AC4: toast was dead code because without return api(error.config) every 429 saw retryCount=1, permanently below the threshold of 3. --------- Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev> * [4112cd34] feat(rate-limit): add RateLimitError with 5-retry exponential backoff at all LLM call sites (#102) (#103) - Create roboco/services/exceptions.py with RateLimitError(provider, retry_after), HTTP_TOO_MANY_REQUESTS, MAX_RATE_LIMIT_RETRIES constants, and parse_retry_after_header() helper - extraction.py: extract _call_anthropic_with_retry() helper; retry Anthropic call 5x on 429 with exponential backoff; re-raise RateLimitError from outer except instead of swallowing it - ollama_embedder.py: 5-retry outer loop (429) wrapping existing 3-retry inner loop (ConnectError/Timeout) for all 4 call sites; two concerns kept isolated - indexes/base.py, mentor.py, validator.py: replace magic 429 literals with HTTP_TOO_MANY_REQUESTS; 5-retry loop on 429 for LLM calls - middleware.py: add rate_limit_exception_handler returning HTTP 429 with Retry-After response header - tests/unit/services/test_rate_limit_retry.py: 28 tests covering exhaustion, Retry-After header sleep, partial retries then success, ConnectError isolation Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> * [18107054] feat(rate-limit): Redis rate-limit state tracker + i_am_blocked rate_limited path (#105) (#106) - Add RateLimitStateTracker in roboco/services/gateway/rate_limit_tracker.py with activate(), clear(), is_rate_limited(), get_state(), increment_probe_failures(), reset_probe_failures() backed by redis.asyncio - Add RATE_LIMIT_HIT = "rate_limit.hit" to EventType StrEnum in events.py - Add _handle_rate_limited_parking() to Choreographer: intercepts i_am_blocked(reason='rate_limited') before block state transition, parks all active agents sharing affected provider via mark_waiting_long, publishes RATE_LIMIT_HIT event to StreamEventBus, task stays in_progress - Add get_provider_for_agent() and get_active_agent_slugs_for_provider() helper methods to AgentOrchestrator - Wire orchestrator and stream_bus into ChoreographerDeps via deps.py - Add test_rate_limit_tracker.py (basic ops, probe failures, cross-reconnection persistence, provider isolation) and test_i_am_blocked_rate_limited.py (AC3/AC4/AC5 coverage: task stays in_progress, mark_waiting_long call count equals active agent count, RATE_LIMIT_HIT event payload structure) Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> * [5501e4b4] Wire RateLimitStateTracker into live orchestrator paths — 4 CEO-identified integration gaps (#109) * [8451ca50] feat(gateway): wire RateLimitStateTracker.activate() into i_am_blocked rate-limited path and add provider-rate-limit gate to decide_spawn() (#107) - Add provider/provider_rate_limited optional fields to TriggerContext (backward-compatible defaults) - Insert rule 2 in decide_spawn(): QUEUE when trigger.provider_rate_limited is True with reason 'provider X rate-limited' - Call RateLimitStateTracker(provider).activate() in _handle_rate_limited_parking() after mark_waiting_long loop (wrapped in contextlib.suppress for Redis fault tolerance) - Extend gateway_pre_spawn_check() with optional provider param; check RateLimitStateTracker.is_rate_limited() when provider is known - Pass provider=self.get_provider_for_agent(agent_id) from orchestrator call site - Add TestProviderRateLimitGate (6 tests) to test_trigger_filter.py - Add TestRateLimitTrackerActivateOnParking (6 tests) to test_i_am_blocked_rate_limited.py - All 38 unit tests pass; ruff and mypy clean on changed files Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> * [e9cef0f0] feat(rate-limits): sweeper probe loop, CEO notification, and GET /api/system/rate-limits endpoint (AC4, AC8, AC9) (#108) - Add RATE_LIMIT_LIFTED event type to EventType enum in models/events.py - Add RateLimitStateTracker.list_rate_limited_providers() classmethod to scan Redis for all currently rate-limited providers (used by the new endpoint) - Add orchestrator._rate_limit_probe_loop(): background task started/stopped in start()/stop(), runs _sweep_rate_limit_probes() every 30s - Add orchestrator._probe_one_provider(): checks estimated_lift_at gate, calls _do_probe(); on success: tracker.clear(), resolve_wait() for all parked agents with waiting_for='rate_limit_lifted' matching the provider, publishes RATE_LIMIT_LIFTED event; on failure: increments probe_failures counter, sends CEO notification at threshold 10 (once per episode via _rate_limit_ceo_notified) - Add orchestrator._make_tracker(): injectable factory for RateLimitStateTracker - Add orchestrator._do_probe(): overridable async bool probe (default: True) - Add orchestrator._notify_rate_limit_ceo(): high-priority notification to CEO containing provider name, duration since activation, and paused agent count - Add roboco/api/routes/system.py with GET /rate-limits endpoint (AC9) - Register system_router in app.py under /api/system prefix - Add 17 unit tests in tests/unit/runtime/test_rate_limit_sweep.py covering all AC4/AC8/AC9 paths: probe success/failure, CEO threshold, endpoint schema Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> --------- Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> --------- Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev> Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-08-03 07:23:24 +02:00 · 2026-06-11 09:21:26 +02:00
parent cc4ccb7ea3
commit 98e618c243
30 changed files with 3919 additions and 273 deletions
@@ -0,0 +1,550 @@
+"""Unit tests for the rate-limit sweeper probe loop (AC4, AC8).
+
+Tests cover:
+- probe-success path: tracker.clear() + resolve_wait + RATE_LIMIT_LIFTED event
+- probe-failure path: increment_probe_failures is called
+- CEO notification fires at threshold 10 exactly once per episode
+- ``_do_probe`` / ``_make_tracker`` are injectable boundaries for mocking
+"""
+
+from __future__ import annotations
+
+import fnmatch
+import json
+from datetime import UTC, datetime
+from typing import Any
+from unittest.mock import AsyncMock, MagicMock, patch
+from uuid import uuid4
+
+from httpx import ASGITransport, AsyncClient
+from roboco.api.app import create_app
+from roboco.models.events import EventType
+from roboco.models.runtime import WaitingRecord
+from roboco.runtime.orchestrator import AgentOrchestrator
+from roboco.services.gateway.rate_limit_tracker import RateLimitStateTracker
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def _make_redis_mock(initial_store: dict[str, Any] | None = None) -> AsyncMock:
+    """Fake redis.asyncio.Redis backed by a plain dict."""
+    store: dict[str, Any] = initial_store if initial_store is not None else {}
+
+    async def _get(key: str) -> bytes | None:
+        val = store.get(key)
+        if val is None:
+            return None
+        return str(val).encode() if not isinstance(val, bytes) else val
+
+    async def _set(key: str, value: Any) -> None:
+        store[key] = value
+
+    async def _delete(key: str) -> int:
+        return 1 if store.pop(key, None) is not None else 0
+
+    async def _scan(
+        _cursor: int, match: str = "*", count: int = 100  # noqa: ARG001
+    ) -> tuple[int, list[bytes]]:
+        # Simple in-memory scan: return all matching keys in one shot
+        matches = [k.encode() for k in store if fnmatch.fnmatch(k, match)]
+        return (0, matches)
+
+    async def _aclose() -> None:
+        pass
+
+    mock = AsyncMock()
+    mock.get = AsyncMock(side_effect=_get)
+    mock.set = AsyncMock(side_effect=_set)
+    mock.delete = AsyncMock(side_effect=_delete)
+    mock.scan = AsyncMock(side_effect=_scan)
+    mock.aclose = AsyncMock(side_effect=_aclose)
+    mock._store = store
+    return mock
+
+
+def _make_orchestrator() -> AgentOrchestrator:
+    """Build a minimal orchestrator via __new__ (no __init__ side-effects)."""
+    orch = AgentOrchestrator.__new__(AgentOrchestrator)
+    orch._running = True
+    orch._waiting_records: dict[str, WaitingRecord] = {}
+    orch._instances: dict[str, Any] = {}
+    orch._rate_limit_ceo_notified: set[str] = set()
+    return orch
+
+
+def _make_tracker_mock(failure_return: int = 1) -> AsyncMock:
+    """Create an async mock RateLimitStateTracker instance."""
+    mock = AsyncMock()
+    mock.clear = AsyncMock()
+    mock.increment_probe_failures = AsyncMock(return_value=failure_return)
+    mock.reset_probe_failures = AsyncMock()
+    return mock
+
+
+def _make_active_state(
+    _provider: str = "anthropic",
+    retry_after: float | None = None,
+    probe_failures: int = 0,
+    activated_at: datetime | None = None,
+) -> dict[str, Any]:
+    """Build a tracker state dict."""
+    at = activated_at or datetime.now(UTC)
+    return {
+        "rate_limited": True,
+        "activated_at": at.isoformat(),
+        "retry_after": retry_after,
+        "affected_agents": ["be-dev-1"],
+        "probe_failures": probe_failures,
+    }
+
+
+def _waiting_record(
+    agent_id: str,
+    provider: str = "anthropic",
+    task_id: str | None = None,
+) -> WaitingRecord:
+    return WaitingRecord(
+        agent_id=agent_id,
+        task_id=task_id or str(uuid4()),
+        waiting_for="rate_limit_lifted",
+        waiting_since=datetime.now(UTC),
+        context={"provider": provider},
+    )
+
+
+# ---------------------------------------------------------------------------
+# Tests: probe-success path (AC4)
+# ---------------------------------------------------------------------------
+
+
+class TestProbeSuccessPath:
+    """When _do_probe returns True the rate limit should be cleared and
+    all parked agents resolved."""
+
+    async def test_tracker_clear_called_on_success(self) -> None:
+        """tracker.clear() is invoked when the probe succeeds."""
+        orch = _make_orchestrator()
+        provider = "anthropic"
+        state = _make_active_state(provider, retry_after=None)
+
+        tracker_mock = _make_tracker_mock()
+        orch._make_tracker = MagicMock(return_value=tracker_mock)  # type: ignore[method-assign]
+        orch.resolve_wait = AsyncMock(return_value=None)
+
+        async def fake_do_probe(_p: str) -> bool:
+            return True
+
+        orch._do_probe = fake_do_probe  # type: ignore[method-assign]
+
+        with patch("roboco.events.get_event_bus") as mock_bus_fn:
+            bus_mock = AsyncMock()
+            bus_mock.publish = AsyncMock()
+            mock_bus_fn.return_value = bus_mock
+
+            await orch._probe_one_provider(provider, state)
+
+        tracker_mock.clear.assert_awaited_once()
+
+    async def test_resolve_wait_called_for_parked_agents(self) -> None:
+        """resolve_wait is called for each agent waiting for rate_limit_lifted."""
+        orch = _make_orchestrator()
+        provider = "anthropic"
+        state = _make_active_state(provider, retry_after=None)
+
+        agent1 = "be-dev-1"
+        agent2 = "be-dev-2"
+        orch._waiting_records = {
+            agent1: _waiting_record(agent1, provider),
+            agent2: _waiting_record(agent2, provider),
+            "be-qa-1": _waiting_record(
+                "be-qa-1", "other-provider"
+            ),  # different provider
+        }
+
+        orch.resolve_wait = AsyncMock(return_value=None)
+
+        tracker_mock = _make_tracker_mock()
+        orch._make_tracker = MagicMock(return_value=tracker_mock)  # type: ignore[method-assign]
+
+        async def fake_do_probe(_p: str) -> bool:
+            return True
+
+        orch._do_probe = fake_do_probe  # type: ignore[method-assign]
+
+        with patch("roboco.events.get_event_bus") as mock_bus_fn:
+            bus_mock = AsyncMock()
+            bus_mock.publish = AsyncMock()
+            mock_bus_fn.return_value = bus_mock
+
+            await orch._probe_one_provider(provider, state)
+
+        # Only the two anthropic-parked agents should be resolved
+        assert orch.resolve_wait.await_count == 2  # noqa: PLR2004
+        resolved_ids = {call.args[0] for call in orch.resolve_wait.call_args_list}
+        assert agent1 in resolved_ids
+        assert agent2 in resolved_ids
+        assert "be-qa-1" not in resolved_ids
+
+    async def test_rate_limit_lifted_event_published(self) -> None:
+        """RATE_LIMIT_LIFTED event is published to the bus on probe success."""
+        orch = _make_orchestrator()
+        provider = "anthropic"
+        state = _make_active_state(provider, retry_after=None)
+
+        orch.resolve_wait = AsyncMock(return_value=None)
+
+        tracker_mock = _make_tracker_mock()
+        orch._make_tracker = MagicMock(return_value=tracker_mock)  # type: ignore[method-assign]
+
+        published_events: list[Any] = []
+
+        async def fake_do_probe(_p: str) -> bool:
+            return True
+
+        orch._do_probe = fake_do_probe  # type: ignore[method-assign]
+
+        with patch("roboco.events.get_event_bus") as mock_bus_fn:
+            bus_mock = AsyncMock()
+            bus_mock.publish = AsyncMock(side_effect=published_events.append)
+            mock_bus_fn.return_value = bus_mock
+
+            await orch._probe_one_provider(provider, state)
+
+        assert len(published_events) == 1
+        event = published_events[0]
+        assert event.type == EventType.RATE_LIMIT_LIFTED
+        assert event.data["provider"] == provider
+
+    async def test_ceo_notified_flag_cleared_on_success(self) -> None:
+        """_rate_limit_ceo_notified is cleared when probe succeeds."""
+        orch = _make_orchestrator()
+        provider = "anthropic"
+        orch._rate_limit_ceo_notified.add(provider)  # simulates prior episode
+        state = _make_active_state(provider, retry_after=None)
+
+        orch.resolve_wait = AsyncMock(return_value=None)
+
+        tracker_mock = _make_tracker_mock()
+        orch._make_tracker = MagicMock(return_value=tracker_mock)  # type: ignore[method-assign]
+
+        async def fake_do_probe(_p: str) -> bool:
+            return True
+
+        orch._do_probe = fake_do_probe  # type: ignore[method-assign]
+
+        with patch("roboco.events.get_event_bus") as mock_bus_fn:
+            bus_mock = AsyncMock()
+            bus_mock.publish = AsyncMock()
+            mock_bus_fn.return_value = bus_mock
+
+            await orch._probe_one_provider(provider, state)
+
+        assert provider not in orch._rate_limit_ceo_notified
+
+    async def test_probe_skipped_before_estimated_lift_at(self) -> None:
+        """When retry_after has not elapsed yet the probe is skipped entirely."""
+        orch = _make_orchestrator()
+        provider = "anthropic"
+        # Set activated_at to now; retry_after = 300s → estimated lift in future
+        state = _make_active_state(
+            provider,
+            retry_after=300.0,
+            activated_at=datetime.now(UTC),
+        )
+
+        probe_called = []
+
+        async def fake_do_probe(_p: str) -> bool:
+            probe_called.append(_p)
+            return True
+
+        orch._do_probe = fake_do_probe  # type: ignore[method-assign]
+
+        await orch._probe_one_provider(provider, state)
+
+        assert probe_called == []  # probe was gated by time
+
+
+# ---------------------------------------------------------------------------
+# Tests: probe-failure path (AC4)
+# ---------------------------------------------------------------------------
+
+
+class TestProbeFailurePath:
+    """When _do_probe returns False the failure counter should be incremented."""
+
+    async def test_increment_probe_failures_called_on_failure(self) -> None:
+        """increment_probe_failures is called when the probe fails."""
+        orch = _make_orchestrator()
+        provider = "anthropic"
+        state = _make_active_state(provider, retry_after=None)
+
+        tracker_mock = _make_tracker_mock(failure_return=1)
+        orch._make_tracker = MagicMock(return_value=tracker_mock)  # type: ignore[method-assign]
+
+        async def fake_do_probe(_p: str) -> bool:
+            return False
+
+        orch._do_probe = fake_do_probe  # type: ignore[method-assign]
+        orch._notify_rate_limit_ceo = AsyncMock()
+
+        await orch._probe_one_provider(provider, state)
+
+        tracker_mock.increment_probe_failures.assert_awaited_once()
+
+    async def test_clear_not_called_on_failure(self) -> None:
+        """tracker.clear() must NOT be called when the probe fails."""
+        orch = _make_orchestrator()
+        provider = "anthropic"
+        state = _make_active_state(provider, retry_after=None)
+
+        tracker_mock = _make_tracker_mock(failure_return=1)
+        orch._make_tracker = MagicMock(return_value=tracker_mock)  # type: ignore[method-assign]
+
+        async def fake_do_probe(_p: str) -> bool:
+            return False
+
+        orch._do_probe = fake_do_probe  # type: ignore[method-assign]
+        orch._notify_rate_limit_ceo = AsyncMock()
+
+        await orch._probe_one_provider(provider, state)
+
+        tracker_mock.clear.assert_not_awaited()
+
+
+# ---------------------------------------------------------------------------
+# Tests: CEO notification threshold (AC8)
+# ---------------------------------------------------------------------------
+
+
+class TestCEONotificationThreshold:
+    """CEO notification fires at count==10 exactly once per episode."""
+
+    async def test_notification_fires_at_exactly_10_failures(self) -> None:
+        """_notify_rate_limit_ceo is called when failure count hits 10."""
+        orch = _make_orchestrator()
+        provider = "anthropic"
+        state = _make_active_state(provider, retry_after=None)
+
+        # simulate already at 9 failures; next increment returns 10
+        tracker_mock = _make_tracker_mock(failure_return=10)
+        orch._make_tracker = MagicMock(return_value=tracker_mock)  # type: ignore[method-assign]
+        orch._notify_rate_limit_ceo = AsyncMock()
+
+        async def fake_do_probe(_p: str) -> bool:
+            return False
+
+        orch._do_probe = fake_do_probe  # type: ignore[method-assign]
+
+        await orch._probe_one_provider(provider, state)
+
+        orch._notify_rate_limit_ceo.assert_awaited_once()
+
+    async def test_notification_not_fired_before_threshold(self) -> None:
+        """No CEO notification below threshold 10."""
+        orch = _make_orchestrator()
+        provider = "anthropic"
+        state = _make_active_state(provider, retry_after=None)
+
+        tracker_mock = _make_tracker_mock(failure_return=9)
+        orch._make_tracker = MagicMock(return_value=tracker_mock)  # type: ignore[method-assign]
+        orch._notify_rate_limit_ceo = AsyncMock()
+
+        async def fake_do_probe(_p: str) -> bool:
+            return False
+
+        orch._do_probe = fake_do_probe  # type: ignore[method-assign]
+
+        await orch._probe_one_provider(provider, state)
+
+        orch._notify_rate_limit_ceo.assert_not_awaited()
+
+    async def test_notification_sent_only_once_per_episode(self) -> None:
+        """Even if failures keep accumulating, the CEO is notified only once."""
+        orch = _make_orchestrator()
+        provider = "anthropic"
+        state = _make_active_state(provider, retry_after=None)
+
+        # Mark this episode as already notified
+        orch._rate_limit_ceo_notified.add(provider)
+
+        tracker_mock = _make_tracker_mock(failure_return=15)
+        orch._make_tracker = MagicMock(return_value=tracker_mock)  # type: ignore[method-assign]
+        orch._notify_rate_limit_ceo = AsyncMock()
+
+        async def fake_do_probe(_p: str) -> bool:
+            return False
+
+        orch._do_probe = fake_do_probe  # type: ignore[method-assign]
+
+        await orch._probe_one_provider(provider, state)
+
+        orch._notify_rate_limit_ceo.assert_not_awaited()
+
+    async def test_new_episode_allows_new_notification(self) -> None:
+        """After a rate-limit clears (success) a new episode starts fresh."""
+        orch = _make_orchestrator()
+        provider = "anthropic"
+        # Episode 1: had a notification
+        orch._rate_limit_ceo_notified.add(provider)
+
+        success_state = _make_active_state(provider, retry_after=None)
+        orch.resolve_wait = AsyncMock(return_value=None)
+
+        tracker_mock = _make_tracker_mock(failure_return=10)
+        orch._make_tracker = MagicMock(return_value=tracker_mock)  # type: ignore[method-assign]
+        notify_mock = AsyncMock()
+        orch._notify_rate_limit_ceo = notify_mock
+
+        async def fake_do_probe_success(_p: str) -> bool:
+            return True
+
+        orch._do_probe = fake_do_probe_success  # type: ignore[method-assign]
+
+        with patch("roboco.events.get_event_bus") as mock_bus_fn:
+            bus_mock = AsyncMock()
+            bus_mock.publish = AsyncMock()
+            mock_bus_fn.return_value = bus_mock
+
+            # Success clears the episode flag
+            await orch._probe_one_provider(provider, success_state)
+
+        assert provider not in orch._rate_limit_ceo_notified
+
+        # Episode 2: simulate a new failure reaching threshold 10
+        async def fake_do_probe_fail(_p: str) -> bool:
+            return False
+
+        orch._do_probe = fake_do_probe_fail  # type: ignore[method-assign]
+
+        failure_state = _make_active_state(provider, retry_after=None)
+        await orch._probe_one_provider(provider, failure_state)
+
+        # Notification SHOULD fire for the new episode
+        notify_mock.assert_awaited_once()
+
+
+# ---------------------------------------------------------------------------
+# Tests: list_rate_limited_providers
+# ---------------------------------------------------------------------------
+
+
+class TestListRateLimitedProviders:
+    """list_rate_limited_providers scans Redis for active rate-limit keys."""
+
+    async def test_returns_empty_when_no_keys(self) -> None:
+        redis_mock = _make_redis_mock()
+        with patch("redis.asyncio.from_url", return_value=redis_mock):
+            result = await RateLimitStateTracker.list_rate_limited_providers()
+        assert result == []
+
+    async def test_returns_active_provider(self) -> None:
+        state = {
+            "rate_limited": True,
+            "activated_at": datetime.now(UTC).isoformat(),
+            "retry_after": 60.0,
+            "affected_agents": ["be-dev-1"],
+            "probe_failures": 0,
+        }
+        store = {"roboco:rate_limit:anthropic:state": json.dumps(state).encode()}
+        redis_mock = _make_redis_mock(store)
+
+        with patch("redis.asyncio.from_url", return_value=redis_mock):
+            result = await RateLimitStateTracker.list_rate_limited_providers()
+
+        assert len(result) == 1
+        provider, returned_state = result[0]
+        assert provider == "anthropic"
+        assert returned_state["rate_limited"] is True
+
+    async def test_ignores_cleared_providers(self) -> None:
+        state = {
+            "rate_limited": False,
+            "activated_at": datetime.now(UTC).isoformat(),
+            "retry_after": 60.0,
+            "affected_agents": [],
+            "probe_failures": 2,
+        }
+        store = {"roboco:rate_limit:anthropic:state": json.dumps(state).encode()}
+        redis_mock = _make_redis_mock(store)
+
+        with patch("redis.asyncio.from_url", return_value=redis_mock):
+            result = await RateLimitStateTracker.list_rate_limited_providers()
+
+        assert result == []
+
+
+# ---------------------------------------------------------------------------
+# Tests: GET /api/system/rate-limits endpoint schema (AC9)
+# ---------------------------------------------------------------------------
+
+
+class TestRateLimitsEndpoint:
+    """GET /api/system/rate-limits returns correct schema."""
+
+    async def test_returns_empty_list_when_no_rate_limits(self) -> None:
+        app = create_app()
+
+        with patch(
+            "roboco.api.routes.system.RateLimitStateTracker"
+            ".list_rate_limited_providers",
+            new_callable=AsyncMock,
+            return_value=[],
+        ):
+            async with AsyncClient(
+                transport=ASGITransport(app=app), base_url="http://test"
+            ) as client:
+                resp = await client.get("/api/system/rate-limits")
+
+        assert resp.status_code == 200  # noqa: PLR2004
+        assert resp.json() == []
+
+    async def test_returns_provider_state_when_rate_limited(self) -> None:
+        app = create_app()
+
+        state = {
+            "rate_limited": True,
+            "activated_at": "2026-06-11T00:00:00+00:00",
+            "retry_after": 60.0,
+            "affected_agents": ["be-dev-1"],
+            "probe_failures": 3,
+        }
+
+        with patch(
+            "roboco.api.routes.system.RateLimitStateTracker"
+            ".list_rate_limited_providers",
+            new_callable=AsyncMock,
+            return_value=[("anthropic", state)],
+        ):
+            async with AsyncClient(
+                transport=ASGITransport(app=app), base_url="http://test"
+            ) as client:
+                resp = await client.get("/api/system/rate-limits")
+
+        assert resp.status_code == 200  # noqa: PLR2004
+        data = resp.json()
+        assert len(data) == 1
+        entry = data[0]
+        assert entry["provider"] == "anthropic"
+        assert entry["rate_limited"] is True
+        assert entry["probe_failures"] == 3  # noqa: PLR2004
+        assert entry["retry_after"] == 60.0  # noqa: PLR2004
+
+    async def test_endpoint_not_404(self) -> None:
+        """The endpoint must be registered in app.py — no 404."""
+        app = create_app()
+
+        with patch(
+            "roboco.api.routes.system.RateLimitStateTracker"
+            ".list_rate_limited_providers",
+            new_callable=AsyncMock,
+            return_value=[],
+        ):
+            async with AsyncClient(
+                transport=ASGITransport(app=app), base_url="http://test"
+            ) as client:
+                resp = await client.get("/api/system/rate-limits")
+
+        assert resp.status_code != 404  # noqa: PLR2004