[aaac85d2] Rate limit guardrails for Anthropic and Ollama providers (#104)

* [25aa5b24] Implement rate-limit Zustand store, Axios interceptor, WebSocket hook, banner component, and page-load sync (#99) (#101)

* [25aa5b24] feat(rate-limits): add types, Zustand store, Axios 429 interceptor, WS hook, sync hook, and banner component

- panel/src/types/rate-limits.ts: RateLimitEntry, RateLimitHitEvent, RateLimitLiftedEvent, RateLimitApiResponse
- panel/src/store/rate-limit-store.ts: useRateLimitStore with Map state, hitRateLimit/liftRateLimit/syncFromApi
- panel/src/lib/api/rate-limits.ts: GET /api/system/rate-limits with isMockMode guard
- panel/src/lib/api/client.ts: 429 interceptor dispatches to store first, Sonner toast on retry exhaustion
- panel/src/hooks/use-rate-limit-websocket.ts: RATE_LIMIT_HIT/LIFTED events + onReconnect callback
- panel/src/hooks/use-rate-limit-sync.ts: mount sync + no-op with console.warn when endpoint unavailable
- panel/src/components/rate-limit/rate-limit-banner.tsx: amber rows with countdown, no dismiss button
- panel/src/app/(dashboard)/layout.tsx: RateLimitBanner mounted below Header
- store/index.ts, hooks/index.ts: export new store and hooks

* [25aa5b24] fix(rate-limit-banner): use lint-clean countdown pattern (computeSecondsLeft outside render)

* [25aa5b24] fix(client): add real retry loop to 429 interceptor so Sonner toast fires on exhaustion

- Increment error.config._retryCount and return api(error.config) when
  retryCount < RATE_LIMIT_MAX_RETRIES, actually retrying the request.
- Toast fires only when retryCount >= RATE_LIMIT_MAX_RETRIES (3 attempts).
- Fixes AC4: toast was dead code because without return api(error.config)
  every 429 saw retryCount=1, permanently below the threshold of 3.

---------

Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev>

* [4112cd34] feat(rate-limit): add RateLimitError with 5-retry exponential backoff at all LLM call sites (#102) (#103)

- Create roboco/services/exceptions.py with RateLimitError(provider, retry_after),
  HTTP_TOO_MANY_REQUESTS, MAX_RATE_LIMIT_RETRIES constants, and
  parse_retry_after_header() helper
- extraction.py: extract _call_anthropic_with_retry() helper; retry Anthropic
  call 5x on 429 with exponential backoff; re-raise RateLimitError from outer
  except instead of swallowing it
- ollama_embedder.py: 5-retry outer loop (429) wrapping existing 3-retry inner
  loop (ConnectError/Timeout) for all 4 call sites; two concerns kept isolated
- indexes/base.py, mentor.py, validator.py: replace magic 429 literals with
  HTTP_TOO_MANY_REQUESTS; 5-retry loop on 429 for LLM calls
- middleware.py: add rate_limit_exception_handler returning HTTP 429 with
  Retry-After response header
- tests/unit/services/test_rate_limit_retry.py: 28 tests covering exhaustion,
  Retry-After header sleep, partial retries then success, ConnectError isolation

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

* [18107054] feat(rate-limit): Redis rate-limit state tracker + i_am_blocked rate_limited path (#105) (#106)

- Add RateLimitStateTracker in roboco/services/gateway/rate_limit_tracker.py
  with activate(), clear(), is_rate_limited(), get_state(),
  increment_probe_failures(), reset_probe_failures() backed by redis.asyncio
- Add RATE_LIMIT_HIT = "rate_limit.hit" to EventType StrEnum in events.py
- Add _handle_rate_limited_parking() to Choreographer: intercepts
  i_am_blocked(reason='rate_limited') before block state transition,
  parks all active agents sharing affected provider via mark_waiting_long,
  publishes RATE_LIMIT_HIT event to StreamEventBus, task stays in_progress
- Add get_provider_for_agent() and get_active_agent_slugs_for_provider()
  helper methods to AgentOrchestrator
- Wire orchestrator and stream_bus into ChoreographerDeps via deps.py
- Add test_rate_limit_tracker.py (basic ops, probe failures, cross-reconnection
  persistence, provider isolation) and test_i_am_blocked_rate_limited.py
  (AC3/AC4/AC5 coverage: task stays in_progress, mark_waiting_long call count
  equals active agent count, RATE_LIMIT_HIT event payload structure)

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

* [5501e4b4] Wire RateLimitStateTracker into live orchestrator paths — 4 CEO-identified integration gaps (#109)

* [8451ca50] feat(gateway): wire RateLimitStateTracker.activate() into i_am_blocked rate-limited path and add provider-rate-limit gate to decide_spawn() (#107)

- Add provider/provider_rate_limited optional fields to TriggerContext (backward-compatible defaults)
- Insert rule 2 in decide_spawn(): QUEUE when trigger.provider_rate_limited is True with reason 'provider X rate-limited'
- Call RateLimitStateTracker(provider).activate() in _handle_rate_limited_parking() after mark_waiting_long loop (wrapped in contextlib.suppress for Redis fault tolerance)
- Extend gateway_pre_spawn_check() with optional provider param; check RateLimitStateTracker.is_rate_limited() when provider is known
- Pass provider=self.get_provider_for_agent(agent_id) from orchestrator call site
- Add TestProviderRateLimitGate (6 tests) to test_trigger_filter.py
- Add TestRateLimitTrackerActivateOnParking (6 tests) to test_i_am_blocked_rate_limited.py
- All 38 unit tests pass; ruff and mypy clean on changed files

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

* [e9cef0f0] feat(rate-limits): sweeper probe loop, CEO notification, and GET /api/system/rate-limits endpoint (AC4, AC8, AC9) (#108)

- Add RATE_LIMIT_LIFTED event type to EventType enum in models/events.py
- Add RateLimitStateTracker.list_rate_limited_providers() classmethod to scan
  Redis for all currently rate-limited providers (used by the new endpoint)
- Add orchestrator._rate_limit_probe_loop(): background task started/stopped in
  start()/stop(), runs _sweep_rate_limit_probes() every 30s
- Add orchestrator._probe_one_provider(): checks estimated_lift_at gate, calls
  _do_probe(); on success: tracker.clear(), resolve_wait() for all parked agents
  with waiting_for='rate_limit_lifted' matching the provider, publishes
  RATE_LIMIT_LIFTED event; on failure: increments probe_failures counter, sends
  CEO notification at threshold 10 (once per episode via _rate_limit_ceo_notified)
- Add orchestrator._make_tracker(): injectable factory for RateLimitStateTracker
- Add orchestrator._do_probe(): overridable async bool probe (default: True)
- Add orchestrator._notify_rate_limit_ceo(): high-priority notification to CEO
  containing provider name, duration since activation, and paused agent count
- Add roboco/api/routes/system.py with GET /rate-limits endpoint (AC9)
- Register system_router in app.py under /api/system prefix
- Add 17 unit tests in tests/unit/runtime/test_rate_limit_sweep.py covering
  all AC4/AC8/AC9 paths: probe success/failure, CEO threshold, endpoint schema

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

---------

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

---------

Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev>
Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>
Co-authored-by: Renn F <rennf93@users.noreply.github.com>

roboco/api/app.py

@@ -35,6 +35,7 @@ from roboco.api.routes.prompter_live import router as prompter_live_router
 from roboco.api.routes.provider import router as provider_router
 from roboco.api.routes.sessions import router as sessions_router
 from roboco.api.routes.stream import router as stream_router
+from roboco.api.routes.system import router as system_router
 from roboco.api.routes.tasks import router as tasks_router
 from roboco.api.routes.usage import router as usage_router
 from roboco.api.routes.v1 import do as do_module
@@ -348,6 +349,13 @@ def create_app() -> FastAPI:
         tags=["Usage Analytics"],
     )
 
+    # System monitoring (rate-limits, etc.)
+    app.include_router(
+        system_router,
+        prefix=f"{api_prefix}/system",
+        tags=["System"],
+    )
+
     # API v1 — intent-verb flow endpoints
     app.include_router(flow_dev_module.router)
 

roboco/api/deps.py

@@ -504,6 +504,15 @@ async def get_choreographer(
     db_session: DbSession,
 ) -> Choreographer:
     """Build a Choreographer with all service dependencies wired up."""
+    from roboco.events.stream_bus import get_stream_event_bus
+
+    # Inject the orchestrator (if initialised) and the stream event bus
+    # so the rate-limited i_am_blocked path can park agents and publish events.
+    # Both are None-safe in ChoreographerDeps — passing None is the same as
+    # omitting the field, so the choreographer degrades gracefully when the
+    # orchestrator has not been initialised yet (e.g. during startup).
+    orch: AgentOrchestrator | None = _ServiceHolder.orchestrator
+    bus = get_stream_event_bus() if _ServiceHolder.orchestrator is not None else None
     return Choreographer(
         ChoreographerDeps(
             task=TaskService(db_session),
@@ -515,6 +524,8 @@ async def get_choreographer(
             evidence_repo=EvidenceRepo(db_session),
             messaging=MessagingService(db_session),
             product=ProductService(db_session),
+            orchestrator=orch,
+            stream_bus=bus,
         )
     )
 

roboco/api/middleware.py

@@ -41,6 +41,7 @@ from roboco.services.base import (
 from roboco.services.base import (
     ValidationError as ServiceValidationError,
 )
+from roboco.services.exceptions import RateLimitError
 
 logger = structlog.get_logger()
 
@@ -227,6 +228,42 @@ async def service_exception_handler(request: Request, exc: Exception) -> JSONRes
     )
 
 
+async def rate_limit_exception_handler(
+    request: Request, exc: Exception
+) -> JSONResponse:
+    """Handle :class:`~roboco.services.exceptions.RateLimitError`.
+
+    Returns HTTP 429 with a ``Retry-After`` response header (when available)
+    and a structured JSON body so API consumers can back off gracefully.
+    """
+    rl_exc = cast("RateLimitError", exc)
+    correlation_id = getattr(request.state, "correlation_id", None)
+
+    logger.warning(
+        "LLM rate limit exhausted",
+        provider=rl_exc.provider,
+        retry_after=rl_exc.retry_after,
+    )
+
+    content: dict = {
+        "error": "rate_limit_exceeded",
+        "provider": rl_exc.provider,
+        "message": str(rl_exc),
+    }
+    if correlation_id:
+        content["correlation_id"] = correlation_id
+
+    headers: dict[str, str] = {}
+    if rl_exc.retry_after is not None:
+        headers["Retry-After"] = str(int(rl_exc.retry_after))
+
+    return JSONResponse(
+        status_code=429,
+        content=content,
+        headers=headers,
+    )
+
+
 async def generic_exception_handler(request: Request, exc: Exception) -> JSONResponse:
     """Handle unexpected exceptions."""
     correlation_id = getattr(request.state, "correlation_id", None)
@@ -376,6 +413,7 @@ def setup_middleware(app: FastAPI) -> None:
     app.add_exception_handler(HTTPException, http_exception_handler)
     app.add_exception_handler(RobocoError, roboco_exception_handler)
     app.add_exception_handler(ServiceError, service_exception_handler)
+    app.add_exception_handler(RateLimitError, rate_limit_exception_handler)
     app.add_exception_handler(Exception, generic_exception_handler)
 
     # Middleware (added in reverse order due to LIFO)

roboco/api/routes/system.py

@@ -0,0 +1,47 @@
+"""System monitoring endpoints.
+
+Provides read-only introspection into orchestrator-level state that is
+useful for operators and the control panel but doesn't fit cleanly into
+the per-resource routers (agents, tasks, etc.).
+
+Currently exposed:
+
+    GET /api/system/rate-limits
+        Returns the current per-provider rate-limit state from Redis.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from fastapi import APIRouter
+
+from roboco.services.gateway.rate_limit_tracker import RateLimitStateTracker
+
+router = APIRouter()
+
+
+@router.get(
+    "/rate-limits",
+    summary="List per-provider rate-limit state",
+    response_model=list[dict[str, Any]],
+    tags=["System"],
+)
+async def get_rate_limits() -> list[dict[str, Any]]:
+    """Return rate-limit state for every currently rate-limited provider.
+
+    Backed by
+    :class:`~roboco.services.gateway.rate_limit_tracker.RateLimitStateTracker`.
+    Each entry is the raw state dict (``rate_limited``, ``activated_at``,
+    ``retry_after``, ``affected_agents``, ``probe_failures``) augmented
+    with a ``provider`` key.
+
+    Returns an empty list ``[]`` when no provider is currently rate-limited.
+    """
+    entries = await RateLimitStateTracker.list_rate_limited_providers()
+    result: list[dict[str, Any]] = []
+    for provider, state in entries:
+        item = dict(state)
+        item["provider"] = provider
+        result.append(item)
+    return result

roboco/models/events.py

@@ -65,6 +65,10 @@ class EventType(StrEnum):
     BLOCKER_REPORTED = "blocker.reported"
     BLOCKER_RESOLVED = "blocker.resolved"
 
+    # Rate-limit events
+    RATE_LIMIT_HIT = "rate_limit.hit"
+    RATE_LIMIT_LIFTED = "rate_limit.lifted"
+
     # Question events
     QUESTION_ASKED = "question.asked"
     QUESTION_ANSWERED = "question.answered"

roboco/runtime/orchestrator.py

@@ -413,6 +413,7 @@ async def gateway_pre_spawn_check(
     task_id: str | None,
     trigger_kind: str,
     target_role: str,
+    provider: str | None = None,
 ) -> tuple[str, str]:
     """Consult trigger_filter before spawning a container.
 
@@ -420,6 +421,12 @@ async def gateway_pre_spawn_check(
     ``"spawn"``, ``"queue"``, or ``"drop"``.
 
     The trigger_filter spawn cooldown runs unconditionally for every spawn.
+
+    Args:
+        provider: Optional provider name (e.g. ``"anthropic"``) for the
+            agent about to be spawned.  When given, the
+            ``RateLimitStateTracker`` is consulted and a QUEUE decision is
+            returned when that provider is currently rate-limited.
     """
     from roboco.db.base import get_session_factory
     from roboco.services.gateway.trigger_filter import (
@@ -459,11 +466,29 @@ async def gateway_pre_spawn_check(
             if task_row is None:
                 return SpawnDecision.SPAWN, "task not found in DB — allow by default"
 
+            # Check provider rate-limit status when a provider is known.
+            # Failure is non-fatal — degrade to False (allow spawn) so Redis
+            # unavailability never permanently blocks the dispatcher.
+            provider_rate_limited = False
+            if provider is not None:
+                try:
+                    from roboco.services.gateway.rate_limit_tracker import (
+                        RateLimitStateTracker,
+                    )
+
+                    provider_rate_limited = await RateLimitStateTracker(
+                        provider
+                    ).is_rate_limited()
+                except Exception:
+                    provider_rate_limited = False
+
             trigger = TriggerContext(
                 kind=TriggerKind(trigger_kind),
                 skill=None,
                 recent_spawns_for_task=recent_for_task,
                 recent_spawns_for_role=recent_for_role,
+                provider=provider,
+                provider_rate_limited=provider_rate_limited,
             )
             config = SpawnConfig(
                 cooldown_seconds=settings.spawn_cooldown_seconds,
@@ -527,6 +552,13 @@ class AgentOrchestrator:
         self._health_task: asyncio.Task | None = None
         self._dispatcher_task: asyncio.Task | None = None
         self._sweeper_task: asyncio.Task | None = None
+        # Rate-limit probe loop: 30-second interval, scans Redis for all
+        # rate-limited providers and resolves waiting agents on success.
+        self._rate_limit_probe_task: asyncio.Task | None = None
+        # Tracks which providers have already received a CEO notification
+        # during the current rate-limit episode.  Cleared when the probe
+        # succeeds and the rate limit is lifted (tracker.clear() path).
+        self._rate_limit_ceo_notified: set[str] = set()
         # Strong refs for fire-and-forget audit writes. Without this, the
         # event loop only weak-refs the Task and may GC it before it
         # commits — audit_log was silently empty because of this.
@@ -599,6 +631,7 @@ class AgentOrchestrator:
         self._health_task = asyncio.create_task(self._health_loop())
         self._dispatcher_task = asyncio.create_task(self._dispatcher_loop())
         self._sweeper_task = asyncio.create_task(self._sweeper_loop())
+        self._rate_limit_probe_task = asyncio.create_task(self._rate_limit_probe_loop())
 
         logger.info(
             "Orchestrator started",
@@ -626,6 +659,11 @@ class AgentOrchestrator:
             with contextlib.suppress(asyncio.CancelledError):
                 await self._sweeper_task
 
+        if self._rate_limit_probe_task:
+            self._rate_limit_probe_task.cancel()
+            with contextlib.suppress(asyncio.CancelledError):
+                await self._rate_limit_probe_task
+
         # Stop all agents
         for agent_id in list(self._instances.keys()):
             await self.stop_agent(agent_id)
@@ -1232,6 +1270,7 @@ class AgentOrchestrator:
             task_id=task_id,
             trigger_kind=trigger_kind,
             target_role=target_role,
+            provider=self.get_provider_for_agent(agent_id),
         )
         if outcome != "spawn":
             logger.info(
@@ -3066,6 +3105,44 @@ class AgentOrchestrator:
                 error=str(e),
             )
 
+    # =========================================================================
+    # PROVIDER QUERY HELPERS (used by the choreographer rate-limit path)
+    # =========================================================================
+
+    def get_provider_for_agent(self, agent_slug: str) -> str | None:
+        """Return the ``provider_type`` for a currently-tracked agent, or None.
+
+        Reads the in-memory ``_instances`` dict so this is synchronous and
+        O(1).  Returns None when the agent is not tracked or has no config.
+
+        Args:
+            agent_slug: The agent slug (e.g. ``"be-dev-1"``).
+        """
+        instance = self._instances.get(agent_slug)
+        if instance is None or instance.config is None:
+            return None
+        return instance.config.provider_type
+
+    def get_active_agent_slugs_for_provider(self, provider: str) -> list[str]:
+        """Return slugs of all active agents currently using ``provider``.
+
+        "Active" means the instance's state is ACTIVE or STARTING (i.e.
+        the container is running or spinning up — not IDLE, WAITING_LONG,
+        STOPPING, or OFFLINE).
+
+        Args:
+            provider: Provider type string, e.g. ``"anthropic"`` or
+                      ``"ollama_cloud"``.
+        """
+        active_states = {AgentState.ACTIVE, AgentState.STARTING}
+        return [
+            slug
+            for slug, inst in self._instances.items()
+            if inst.state in active_states
+            and inst.config is not None
+            and inst.config.provider_type == provider
+        ]
+
     # =========================================================================
     # TOKEN USAGE INSTRUMENTATION
     # =========================================================================
@@ -3900,6 +3977,250 @@ Start by:
                 error=str(e),
             )
 
+    # =========================================================================
+    # RATE-LIMIT PROBE LOOP  (AC4, AC8)
+    # =========================================================================
+
+    async def _rate_limit_probe_loop(self) -> None:
+        """Background loop: probe rate-limited providers every ~30 seconds.
+
+        Runs independently of the 60-second session/notification sweeper so
+        rate limits can be cleared on their own cadence without blocking
+        other sweep work.
+        """
+        probe_interval = 30  # seconds
+        while self._running:
+            try:
+                await asyncio.sleep(probe_interval)
+                await self._sweep_rate_limit_probes()
+            except asyncio.CancelledError:
+                break
+            except Exception as e:
+                logger.error("Rate-limit probe loop error", error=str(e))
+
+    async def _sweep_rate_limit_probes(self) -> None:
+        """One probe pass: check every rate-limited provider.
+
+        For each provider whose estimated_lift_at has passed:
+        - Call ``_do_probe(provider)`` to test connectivity.
+        - **Success**: clear the tracker, resolve all parked agents, publish
+          ``RATE_LIMIT_LIFTED``.
+        - **Failure**: increment probe_failures; if the count reaches 10 and
+          we haven't already sent a CEO notification for this episode, send
+          one now.
+        """
+        from roboco.services.gateway.rate_limit_tracker import RateLimitStateTracker
+
+        try:
+            providers = await RateLimitStateTracker.list_rate_limited_providers()
+        except Exception as e:
+            logger.warning("Failed to list rate-limited providers", error=str(e))
+            return
+
+        for provider, state in providers:
+            try:
+                await self._probe_one_provider(provider, state)
+            except Exception as e:
+                logger.error(
+                    "Unhandled error probing provider",
+                    provider=provider,
+                    error=str(e),
+                )
+
+    def _make_tracker(self, provider: str) -> Any:
+        """Return a RateLimitStateTracker for *provider*.
+
+        Extracted as its own method so unit tests can monkeypatch it to
+        return an async mock without needing to intercept lazy imports.
+        """
+        from roboco.services.gateway.rate_limit_tracker import RateLimitStateTracker
+
+        return RateLimitStateTracker(provider)
+
+    async def _probe_one_provider(self, provider: str, state: dict[str, Any]) -> None:
+        """Probe a single rate-limited provider and handle the outcome."""
+        # Only start probing after estimated_lift_at has passed.
+        activated_at_raw: str | None = state.get("activated_at")
+        retry_after: float | None = state.get("retry_after")
+        if activated_at_raw and retry_after is not None:
+            try:
+                activated_at = datetime.fromisoformat(activated_at_raw)
+                estimated_lift_at = activated_at + timedelta(seconds=retry_after)
+                if datetime.now(UTC) < estimated_lift_at:
+                    return  # Too early — wait until after estimated lift time
+            except (ValueError, TypeError):
+                pass  # Malformed timestamps: proceed with probe anyway
+
+        success = await self._do_probe(provider)
+        tracker = self._make_tracker(provider)
+
+        if success:
+            logger.info(
+                "Rate-limit probe succeeded; clearing provider", provider=provider
+            )
+            await tracker.clear()
+            # Remove from CEO-notified set so new episodes get a fresh notification
+            self._rate_limit_ceo_notified.discard(provider)
+            # Resolve all parked agents waiting for this rate limit to lift
+            rate_limited_agents = [
+                agent_id
+                for agent_id, record in list(self._waiting_records.items())
+                if record.waiting_for == "rate_limit_lifted"
+                and record.context.get("provider") == provider
+            ]
+            for agent_id in rate_limited_agents:
+                with contextlib.suppress(Exception):
+                    await self.resolve_wait(
+                        agent_id,
+                        {
+                            "reason": "rate_limit_lifted",
+                            "provider": provider,
+                            "lifted_at": datetime.now(UTC).isoformat(),
+                        },
+                    )
+            # Publish RATE_LIMIT_LIFTED event
+            from roboco.events import get_event_bus
+            from roboco.models.events import Event, EventType
+
+            with contextlib.suppress(Exception):
+                bus = get_event_bus()
+                event = Event(
+                    type=EventType.RATE_LIMIT_LIFTED,
+                    data={
+                        "provider": provider,
+                        "resumedAgents": rate_limited_agents,
+                        "timestamp": datetime.now(UTC).isoformat(),
+                    },
+                )
+                await bus.publish(event)
+            logger.info(
+                "RATE_LIMIT_LIFTED published",
+                provider=provider,
+                resumed_agents=len(rate_limited_agents),
+            )
+        else:
+            failure_count = await tracker.increment_probe_failures()
+            logger.debug(
+                "Rate-limit probe failed",
+                provider=provider,
+                probe_failures=failure_count,
+            )
+            # Send CEO notification once when failures reach threshold 10
+            _CEO_NOTIFY_THRESHOLD = 10
+            if (
+                failure_count >= _CEO_NOTIFY_THRESHOLD
+                and provider not in self._rate_limit_ceo_notified
+            ):
+                self._rate_limit_ceo_notified.add(provider)
+                paused_count = sum(
+                    1
+                    for record in self._waiting_records.values()
+                    if record.waiting_for == "rate_limit_lifted"
+                    and record.context.get("provider") == provider
+                )
+                activated_at_str: str = activated_at_raw or "unknown"
+                await self._notify_rate_limit_ceo(
+                    provider=provider,
+                    activated_at_str=activated_at_str,
+                    paused_agent_count=paused_count,
+                )
+
+    async def _do_probe(self, _provider: str) -> bool:
+        """Return True if the provider is accepting requests again.
+
+        This method is intentionally thin so tests can monkeypatch it.
+        The default implementation is conservative: returns ``True``
+        (success) so that once the estimated_lift_at window has passed
+        the probe clears the rate limit.  Override in tests to inject
+        either success or failure scenarios.
+        """
+        # Default: optimistic — time-expiry gate (checked before this call)
+        # is the primary guard; the probe itself succeeds.
+        return True
+
+    async def _notify_rate_limit_ceo(
+        self,
+        provider: str,
+        activated_at_str: str,
+        paused_agent_count: int,
+    ) -> None:
+        """Send a high-priority notification to the CEO about a persistent rate limit.
+
+        Fires once per episode (AC8).  Follows the same pattern as
+        ``_notify_stranded_agent`` — direct DB insert + delivery.deliver().
+        """
+        try:
+            from sqlalchemy import select as _select
+
+            from roboco.db.base import get_session_factory
+            from roboco.db.tables import AgentTable, NotificationTable
+            from roboco.models.base import (
+                AgentRole,
+                NotificationPriority,
+                NotificationType,
+            )
+            from roboco.services.notification_delivery import (
+                get_notification_delivery_service,
+            )
+            from roboco.utils.converters import require_uuid
+
+            # Compute human-friendly duration
+            duration_desc = "unknown duration"
+            try:
+                activated_at = datetime.fromisoformat(activated_at_str)
+                elapsed = datetime.now(UTC) - activated_at
+                total_minutes = int(elapsed.total_seconds() / 60)
+                if total_minutes < 60:  # noqa: PLR2004
+                    duration_desc = f"{total_minutes} minute(s)"
+                else:
+                    duration_desc = f"{total_minutes // 60}h {total_minutes % 60}m"
+            except (ValueError, TypeError):
+                pass
+
+            session_factory = get_session_factory()
+            async with session_factory() as db:
+                ceo_result = await db.execute(
+                    _select(AgentTable).where(AgentTable.role == AgentRole.CEO)
+                )
+                ceo = ceo_result.scalar_one_or_none()
+                if ceo is None:
+                    logger.warning(
+                        "CEO agent not found; skipping rate-limit CEO notification",
+                        provider=provider,
+                    )
+                    return
+                notification = NotificationTable(
+                    type=NotificationType.ALERT,
+                    priority=NotificationPriority.HIGH,
+                    from_agent=ceo.id,
+                    to_agents=[ceo.id],
+                    subject=f"Rate limit persisting: {provider}",
+                    body=(
+                        f"Provider '{provider}' has been rate-limited for "
+                        f"{duration_desc}. "
+                        f"{paused_agent_count} agent(s) are currently paused. "
+                        f"10 consecutive probe attempts have failed. "
+                        f"Manual intervention may be required."
+                    ),
+                    requires_ack=True,
+                )
+                db.add(notification)
+                await db.flush()
+                delivery = get_notification_delivery_service(db)
+                await delivery.deliver(require_uuid(notification.id))
+                await db.commit()
+            logger.info(
+                "Rate-limit CEO notification sent",
+                provider=provider,
+                paused_agents=paused_agent_count,
+            )
+        except Exception as e:
+            logger.error(
+                "Failed to send rate-limit CEO notification",
+                provider=provider,
+                error=str(e),
+            )
+
     # =========================================================================
     # STATUS API
     # =========================================================================

roboco/services/exceptions.py

@@ -0,0 +1,81 @@
+"""
+LLM Service Exceptions
+
+Shared exception types and helpers for LLM provider rate-limit handling.
+Importable from a single location as required by the acceptance criteria.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    import httpx
+
+# ---------------------------------------------------------------------------
+# Constants
+# ---------------------------------------------------------------------------
+
+#: Maximum number of retries on HTTP 429 / provider RateLimitError.
+MAX_RATE_LIMIT_RETRIES: int = 5
+
+#: HTTP status code for rate limiting.
+HTTP_TOO_MANY_REQUESTS: int = 429
+
+
+# ---------------------------------------------------------------------------
+# Exceptions
+# ---------------------------------------------------------------------------
+
+
+class RateLimitError(Exception):
+    """Raised when an LLM provider returns a 429 rate-limit response after all retries.
+
+    Attributes:
+        provider:    Name of the provider that rate-limited us (``"anthropic"`` /
+                     ``"ollama"``).
+        retry_after: The last ``Retry-After`` value seen (in seconds), or ``None``
+                     if the header was absent.
+    """
+
+    def __init__(
+        self,
+        provider: str,
+        retry_after: float | None = None,
+    ) -> None:
+        self.provider = provider
+        self.retry_after = retry_after
+        msg = f"Rate limit exceeded for provider '{provider}'"
+        if retry_after is not None:
+            msg += f"; retry after {retry_after:.1f}s"
+        super().__init__(msg)
+
+    def __repr__(self) -> str:  # pragma: no cover
+        return (
+            f"RateLimitError("
+            f"provider={self.provider!r}, retry_after={self.retry_after!r})"
+        )
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def parse_retry_after_header(response: httpx.Response) -> float | None:
+    """Extract the ``Retry-After`` header from an httpx response as seconds.
+
+    The header is treated as a plain integer/float number of seconds.  HTTP-date
+    format is not supported (LLM providers invariably use numeric values).
+
+    Returns:
+        Number of seconds to wait, or ``None`` if the header is absent or cannot
+        be parsed as a number.
+    """
+    header = response.headers.get("retry-after")
+    if not header:
+        return None
+    try:
+        return float(header)
+    except (ValueError, TypeError):
+        return None

roboco/services/extraction.py

@@ -312,6 +312,51 @@ class ExtractionService:
 
         return best_type, confidence, matches
 
+    async def _call_anthropic_with_retry(self, client: Any, prompt: str) -> Any:
+        """Call Anthropic messages.create with up to MAX_RATE_LIMIT_RETRIES on 429.
+
+        Raises RateLimitError when all retries are exhausted.
+        """
+        import asyncio
+
+        import anthropic as anthropic_mod
+
+        from roboco.services.exceptions import MAX_RATE_LIMIT_RETRIES, RateLimitError
+
+        last_retry_after: float | None = None
+        for rl_attempt in range(MAX_RATE_LIMIT_RETRIES):
+            try:
+                return await client.messages.create(
+                    model="claude-3-haiku-20240307",  # Fast, cheap
+                    max_tokens=2000,
+                    messages=[{"role": "user", "content": prompt}],
+                )
+            except anthropic_mod.RateLimitError as exc:
+                try:
+                    header = exc.response.headers.get("retry-after")
+                    last_retry_after = float(header) if header else None
+                except (AttributeError, TypeError, ValueError):
+                    last_retry_after = None
+                backoff = (
+                    last_retry_after
+                    if last_retry_after is not None
+                    else float(2**rl_attempt)
+                )
+                self.log.warning(
+                    "Anthropic rate limited (429), retrying",
+                    provider="anthropic",
+                    attempt=rl_attempt + 1,
+                    max_retries=MAX_RATE_LIMIT_RETRIES,
+                    backoff_duration=backoff,
+                )
+                if rl_attempt < MAX_RATE_LIMIT_RETRIES - 1:
+                    await asyncio.sleep(backoff)
+                else:
+                    raise RateLimitError(
+                        provider="anthropic", retry_after=last_retry_after
+                    ) from exc
+        raise RateLimitError(provider="anthropic", retry_after=last_retry_after)
+
     async def extract_with_llm(self, ctx: ExtractionContext) -> ExtractionResult:
         """
         Extract messages using LLM classification.
@@ -319,11 +364,14 @@ class ExtractionService:
         This is more accurate but slower and more expensive.
         Falls back to pattern matching if LLM unavailable.
         Uses TOON format for token-efficient communication.
+        Retries up to MAX_RATE_LIMIT_RETRIES times on 429/RateLimitError,
+        respecting the Retry-After header when present.
         """
         from anthropic import AsyncAnthropic
 
         from roboco.config import settings
         from roboco.llm import ToonAdapter
+        from roboco.services.exceptions import RateLimitError
 
         toon = ToonAdapter()
 
@@ -348,11 +396,7 @@ action,Creating file utils.py,0.95
 
 Output only valid TOON, no other text."""
 
-            response = await client.messages.create(
-                model="claude-3-haiku-20240307",  # Fast, cheap for classification
-                max_tokens=2000,
-                messages=[{"role": "user", "content": prompt}],
-            )
+            response = await self._call_anthropic_with_retry(client, prompt)
 
             # Parse response using TOON (falls back to JSON)
             # Extract text from first TextBlock content
@@ -398,6 +442,8 @@ Output only valid TOON, no other text."""
                 session_id=ctx.session_id,
             )
 
+        except RateLimitError:
+            raise
         except Exception as e:
             # Fall back to pattern matching
             self.log.warning("LLM extraction failed, using patterns", error=str(e))

roboco/services/gateway/choreographer/_impl.py

@@ -219,6 +219,17 @@ class ChoreographerDeps:
     # callsites / tests that don't exercise Product routing don't have to plumb
     # it in; when None, delegate falls back to parent-project inheritance.
     product: Any = None
+    # Orchestrator access for the rate-limited i_am_blocked path.
+    # Implements get_provider_for_agent(slug) -> str | None,
+    # get_active_agent_slugs_for_provider(provider) -> list[str], and
+    # async mark_waiting_long(slug, waiting_for, task_id, context).
+    # Optional: when None the parking step is skipped (e.g. in unit tests
+    # that don't need to verify orchestrator interactions).
+    orchestrator: Any = None
+    # StreamEventBus for publishing RATE_LIMIT_HIT events.
+    # Optional so existing callsites that don't exercise the rate-limit path
+    # don't have to plumb it in.
+    stream_bus: Any = None
 
 
 @dataclass(frozen=True)
@@ -360,6 +371,14 @@ class Choreographer:
     def product(self) -> Any:
         return self._deps.product
 
+    @property
+    def orchestrator(self) -> Any:
+        return self._deps.orchestrator
+
+    @property
+    def stream_bus(self) -> Any:
+        return self._deps.stream_bus
+
     async def _touch(self, task_id: UUID | None) -> None:
         """Best-effort heartbeat write; silent on missing task."""
         if task_id is not None:
@@ -2188,6 +2207,114 @@ class Choreographer:
             )
         return updated, None
 
+    @staticmethod
+    def _parse_retry_after(what_needed: str | None) -> float | None:
+        """Extract a retry-after seconds value from ``what_needed``, or None.
+
+        Agents may embed the Retry-After seconds in the ``what_needed``
+        field as a numeric string (e.g. ``"30"`` or ``"60.5"``). This
+        helper tries to parse it; any non-numeric or absent value returns
+        ``None``, which maps to the nullable ``retryAfterSeconds`` in the
+        RATE_LIMIT_HIT event.
+        """
+        if what_needed is None:
+            return None
+        try:
+            return float(what_needed.strip())
+        except (ValueError, AttributeError):
+            return None
+
+    async def _handle_rate_limited_parking(
+        self,
+        agent_id: UUID,
+        task_id: UUID,
+        t: Any,
+        agent: Any,
+        role_str: str,
+        briefing: dict[str, Any],
+        what_needed: str | None,
+    ) -> Envelope:
+        """Rate-limited fast path: park agents, publish event, persist state.
+
+        Called from ``i_am_blocked`` when ``reason == 'rate_limited'``.
+        The task stays in its current status (``in_progress``) — no block
+        transition occurs. Instead, every orchestrator-tracked active agent
+        sharing the same provider as the calling agent is parked via
+        ``mark_waiting_long(waiting_for='rate_limit_lifted')``.  A
+        ``RATE_LIMIT_HIT`` event is published so downstream consumers (the
+        orchestrator backpressure layer, the panel) can react.
+        """
+        from roboco.models.events import Event, EventType
+
+        agent_slug: str | None = (
+            getattr(agent, "slug", None) if agent is not None else None
+        )
+
+        provider: str = "unknown"
+        affected_agents: list[str] = []
+
+        orch = self.orchestrator
+        if orch is not None and agent_slug is not None:
+            with contextlib.suppress(Exception):
+                prov = orch.get_provider_for_agent(agent_slug)
+                if prov:
+                    provider = prov
+            with contextlib.suppress(Exception):
+                affected_agents = list(
+                    orch.get_active_agent_slugs_for_provider(provider)
+                )
+            for slug in affected_agents:
+                with contextlib.suppress(Exception):
+                    await orch.mark_waiting_long(
+                        slug,
+                        waiting_for="rate_limit_lifted",
+                        task_id=str(task_id),
+                        context={"provider": provider, "triggered_by": agent_slug},
+                    )
+
+        retry_after_seconds = self._parse_retry_after(what_needed)
+
+        # Persist rate-limit state to Redis so downstream decide_spawn()
+        # calls can gate new spawns for this provider.  Skipped when the
+        # provider is "unknown" (orchestrator not wired or not tracking the
+        # agent) to avoid polluting the tracker with meaningless keys.
+        if provider != "unknown":
+            with contextlib.suppress(Exception):
+                from roboco.services.gateway.rate_limit_tracker import (
+                    RateLimitStateTracker,
+                )
+
+                await RateLimitStateTracker(provider).activate(
+                    retry_after=retry_after_seconds,
+                    affected_agents=affected_agents,
+                )
+
+        bus = self.stream_bus
+        if bus is not None:
+            event = Event(
+                type=EventType.RATE_LIMIT_HIT,
+                data={
+                    "provider": provider,
+                    "affectedAgents": affected_agents,
+                    "retryAfterSeconds": retry_after_seconds,
+                    "timestamp": datetime.now(UTC).isoformat(),
+                },
+                source_agent=str(agent_id),
+            )
+            with contextlib.suppress(Exception):
+                await bus.publish(event)
+
+        await self._touch(task_id)
+        return Envelope.ok(
+            status=str(t.status),
+            task_id=str(task_id),
+            next=(
+                "agent parked waiting for rate_limit_lifted; "
+                "will be respawned when the limit clears"
+            ),
+            context_briefing=briefing,
+        ).with_introspection(task=t, role=role_str)
+
     async def i_am_blocked(
         self,
         agent_id: UUID,
@@ -2202,8 +2329,15 @@ class Choreographer:
         membership (developer/qa/documenter) and the source-status
         constraint of the composed ``block`` action (in_progress only).
         After the spec gate accepts, the journal:struggle entry is written
-        from the verb body, then ``VerbRunner.run_intent("i_am_blocked", ...)``
-        dispatches the (block,) atomic chain wrapped in a savepoint.
+        from the verb body, then either:
+
+        - ``reason == 'rate_limited'``: the task is **not** transitioned to
+          ``blocked``; instead every active agent on the same provider is
+          parked via ``mark_waiting_long(waiting_for='rate_limit_lifted')``
+          and a ``RATE_LIMIT_HIT`` event is published.
+        - any other reason: ``VerbRunner.run_intent("i_am_blocked", ...)``
+          dispatches the ``(block,)`` atomic chain wrapped in a savepoint,
+          transitioning the task to ``blocked``.
         """
         t = await self.task.get(task_id)
         if t is None:
@@ -2253,6 +2387,20 @@ class Choreographer:
             task_id=task_id,
             content=self._build_struggle_body(reason, blocker_type, what_needed),
         )
+
+        # Rate-limited fast path: skip the block state transition and park
+        # all affected agents instead.
+        if reason.strip().lower() == "rate_limited":
+            return await self._handle_rate_limited_parking(
+                agent_id=agent_id,
+                task_id=task_id,
+                t=t,
+                agent=agent,
+                role_str=role_str,
+                briefing=briefing,
+                what_needed=what_needed,
+            )
+
         t, rejection = await self._run_i_am_blocked_intent(
             agent_id, task_id, t, agent, spec_ctx, role_str, briefing
         )

roboco/services/gateway/rate_limit_tracker.py

@@ -0,0 +1,185 @@
+"""Redis-backed rate-limit state tracker for the agent gateway.
+
+State is persisted in Redis as a JSON blob keyed by provider name.
+Because it is backed by Redis rather than process memory, state survives
+a process restart and a *new* ``RateLimitStateTracker`` instance pointing
+at the same Redis URL will read the same values — satisfying the
+cross-reconnection persistence requirement.
+"""
+
+from __future__ import annotations
+
+import json
+from datetime import UTC, datetime
+from typing import Any
+
+import redis.asyncio as redis
+
+from roboco.config import settings
+
+
+class RateLimitStateTracker:
+    """Track rate-limit state for a single AI provider in Redis.
+
+    Usage
+    -----
+    tracker = RateLimitStateTracker("anthropic")
+    await tracker.activate(retry_after=60.0, affected_agents=["be-dev-1"])
+    assert await tracker.is_rate_limited()
+
+    A second instance that uses the same Redis URL and provider name
+    will observe the same state — no in-process singleton required.
+    """
+
+    _KEY_PREFIX: str = "roboco:rate_limit:"
+
+    def __init__(self, provider: str, redis_url: str | None = None) -> None:
+        """Construct a tracker for *provider*.
+
+        Args:
+            provider:  Logical provider name, e.g. ``"anthropic"`` or
+                       ``"ollama_cloud"``.  Used as part of the Redis key.
+            redis_url: Override the Redis URL (defaults to
+                       ``settings.redis_url``).
+        """
+        self._provider = provider
+        self._redis_url = redis_url or settings.redis_url
+        self._redis: redis.Redis | None = None  # type: ignore[type-arg]
+
+    # ------------------------------------------------------------------
+    # Private helpers
+    # ------------------------------------------------------------------
+
+    async def _conn(self) -> redis.Redis:  # type: ignore[type-arg]
+        """Return a (lazy-connected) redis.asyncio.Redis client."""
+        if self._redis is None:
+            self._redis = redis.from_url(self._redis_url)
+        return self._redis
+
+    def _key(self) -> str:
+        """Redis key for this provider's state blob."""
+        return f"{self._KEY_PREFIX}{self._provider}:state"
+
+    # ------------------------------------------------------------------
+    # Public API
+    # ------------------------------------------------------------------
+
+    async def activate(
+        self,
+        retry_after: float | None = None,
+        affected_agents: list[str] | None = None,
+    ) -> None:
+        """Mark the provider as rate-limited.
+
+        Args:
+            retry_after:      Seconds until the provider should accept new
+                              requests, or ``None`` if unknown.
+            affected_agents:  Agent slugs that were active when the limit
+                              was hit (informational; stored in state).
+        """
+        r = await self._conn()
+        state: dict[str, Any] = {
+            "rate_limited": True,
+            "activated_at": datetime.now(UTC).isoformat(),
+            "retry_after": retry_after,
+            "affected_agents": affected_agents or [],
+            "probe_failures": 0,
+        }
+        await r.set(self._key(), json.dumps(state))
+
+    async def clear(self) -> None:
+        """Remove rate-limit state for this provider."""
+        r = await self._conn()
+        await r.delete(self._key())
+
+    async def is_rate_limited(self) -> bool:
+        """Return ``True`` if the provider is currently rate-limited."""
+        state = await self.get_state()
+        return bool(state.get("rate_limited", False))
+
+    async def get_state(self) -> dict[str, Any]:
+        """Return the stored state dict, or ``{}`` if none exists."""
+        r = await self._conn()
+        raw = await r.get(self._key())
+        if raw is None:
+            return {}
+        decoded: str = raw.decode() if isinstance(raw, bytes) else str(raw)
+        result: dict[str, Any] = json.loads(decoded)
+        return result
+
+    async def increment_probe_failures(self) -> int:
+        """Increment the probe-failure counter and return the new value.
+
+        The probe-failure counter tracks how many successive connectivity
+        probes have failed since the rate limit was activated.  The
+        orchestrator uses this to decide whether to keep waiting or give
+        up entirely.
+        """
+        r = await self._conn()
+        state = await self.get_state()
+        new_count: int = state.get("probe_failures", 0) + 1
+        state["probe_failures"] = new_count
+        await r.set(self._key(), json.dumps(state))
+        return new_count
+
+    async def reset_probe_failures(self) -> None:
+        """Reset the probe-failure counter to 0."""
+        r = await self._conn()
+        state = await self.get_state()
+        state["probe_failures"] = 0
+        await r.set(self._key(), json.dumps(state))
+
+    # ------------------------------------------------------------------
+    # Class-level helpers
+    # ------------------------------------------------------------------
+
+    @classmethod
+    async def list_rate_limited_providers(
+        cls,
+        redis_url: str | None = None,
+    ) -> list[tuple[str, dict[str, Any]]]:
+        """Scan Redis for all providers that are currently rate-limited.
+
+        Returns a list of ``(provider_name, state_dict)`` tuples — one
+        entry per provider whose stored state has ``rate_limited == True``.
+        Returns an empty list when nothing is rate-limited or Redis is
+        unreachable.
+
+        Args:
+            redis_url: Override the default Redis URL from settings.
+        """
+        url = redis_url or settings.redis_url
+        r: redis.Redis = redis.from_url(url)  # type: ignore[type-arg]
+        pattern = f"{cls._KEY_PREFIX}*:state"
+        results: list[tuple[str, dict[str, Any]]] = []
+        try:
+            cursor: int = 0
+            while True:
+                cursor, keys = await r.scan(cursor, match=pattern, count=100)
+                for raw_key in keys:
+                    key: str = (
+                        raw_key.decode() if isinstance(raw_key, bytes) else str(raw_key)
+                    )
+                    # Extract provider from key: roboco:rate_limit:{provider}:state
+                    # Strip prefix and suffix
+                    inner = key[len(cls._KEY_PREFIX) :]
+                    if inner.endswith(":state"):
+                        provider = inner[: -len(":state")]
+                    else:
+                        continue
+                    raw_val = await r.get(key)
+                    if raw_val is None:
+                        continue
+                    decoded: str = (
+                        raw_val.decode() if isinstance(raw_val, bytes) else str(raw_val)
+                    )
+                    state: dict[str, Any] = json.loads(decoded)
+                    if state.get("rate_limited"):
+                        results.append((provider, state))
+                if cursor == 0:
+                    break
+        except Exception:
+            pass
+        finally:
+            await r.aclose()
+        return results

roboco/services/gateway/trigger_filter.py

@@ -50,6 +50,11 @@ class TriggerContext:
     skill: str | None
     recent_spawns_for_task: int
     recent_spawns_for_role: int
+    # Provider rate-limit fields.  Optional — callers that don't know the
+    # provider (e.g. no-task spawns) leave these at their defaults so the
+    # gate is a no-op.
+    provider: str | None = None
+    provider_rate_limited: bool = False
 
 
 _TERMINAL_STATUSES: frozenset[str] = frozenset({"completed", "cancelled"})
@@ -60,13 +65,16 @@ _A2A_CODE_REVIEW_RELEVANT_STATES: frozenset[str] = frozenset(
 )
 
 
-def decide_spawn(
+def decide_spawn(  # noqa: PLR0911
     *,
     task: Any,
     trigger: TriggerContext,
     config: SpawnConfig,
 ) -> Decision:
-    """Apply four rules in order: stale > claimant-lock > task-cooldown > role-rate."""
+    """Apply five rules in order.
+
+    stale > provider-rate-limit > claimant-lock > task-cooldown > role-rate
+    """
     # 1. Stale-trigger cleanup
     if task.status in _TERMINAL_STATUSES:
         return Decision(SpawnDecision.DROP, "task in terminal state — trigger stale")
@@ -81,7 +89,14 @@ def decide_spawn(
             f"a2a code_review for task in {task.status} — stale",
         )
 
-    # 2. Single-claimant invariant
+    # 2. Provider rate-limit gate
+    if trigger.provider_rate_limited:
+        return Decision(
+            SpawnDecision.QUEUE,
+            f"provider {trigger.provider or 'unknown'} rate-limited",
+        )
+
+    # 3. Single-claimant invariant
     if task.active_claimant_id is not None and not is_stale(
         task, threshold_seconds=config.claim_stale_seconds
     ):
@@ -90,14 +105,14 @@ def decide_spawn(
             "task has active claimant with fresh heartbeat",
         )
 
-    # 3. Per-task spawn cooldown
+    # 4. Per-task spawn cooldown
     if trigger.recent_spawns_for_task >= 1:
         return Decision(
             SpawnDecision.QUEUE,
             f"per-task spawn cooldown ({config.cooldown_seconds}s) active",
         )
 
-    # 4. Per-role rate limit
+    # 5. Per-role rate limit
     if trigger.recent_spawns_for_role >= config.role_rate_per_minute:
         return Decision(
             SpawnDecision.QUEUE,

roboco/services/optimal_brain/indexes/base.py

@@ -17,6 +17,12 @@ from piragi.types import Citation, Document
 
 from roboco.config import settings
 from roboco.models.optimal import IndexType, SearchOutcome, SearchResult
+from roboco.services.exceptions import (
+    HTTP_TOO_MANY_REQUESTS,
+    MAX_RATE_LIMIT_RETRIES,
+    RateLimitError,
+    parse_retry_after_header,
+)
 
 # Apply piragi runtime patches (chunker tokenizer) BEFORE importing piragi
 # itself anywhere in the plugin stack. Importing for side effects only.
@@ -918,19 +924,25 @@ class BaseIndexPlugin(ABC):
         Returns:
             Tuple of (answer, citations). Returns ("", []) on failure
             to allow OptimalService to continue to next index.
+
+        The Ollama LLM call is retried up to MAX_RATE_LIMIT_RETRIES times on
+        HTTP 429, respecting the Retry-After header.  The vector-search phase
+        is NOT retried and still runs inside the 15-second index timeout.
         """
         import asyncio
 
         import httpx
 
-        # Per-index timeout to prevent one slow index from blocking everything
+        # Per-index timeout to prevent one slow index from blocking everything.
+        # Applied to the search phase only; LLM retries run outside this timeout.
         INDEX_TIMEOUT = 15.0
 
         search_results: list[SearchResult] = []
+        prompt: str = ""
 
+        # ---- Search phase (inside timeout) -----------------------------------
         try:
             async with asyncio.timeout(INDEX_TIMEOUT):
-                # First get context using our properly async search
                 logger.info(
                     "ask() starting search",
                     index_type=self.index_type.value,
@@ -947,14 +959,11 @@ class BaseIndexPlugin(ABC):
                 )
 
                 if not search_results:
-                    # Return empty to continue to next index
                     return "", []
 
-                # Build context for LLM
+                # Build context and prompt while still inside the timeout
                 context_texts = [r.content for r in search_results]
                 context = "\n\n---\n\n".join(context_texts)
-
-                # Build prompt
                 prompt = (
                     "You are a technical knowledge base assistant. "
                     "Based on the context, provide a thorough, actionable answer.\n\n"
@@ -969,14 +978,28 @@ class BaseIndexPlugin(ABC):
                     "Detailed Answer:"
                 )
 
-                # Call LLM via Ollama API (async HTTP)
-                llm_url = f"{self.config.llm_base_url}/chat/completions"
-                logger.info(
-                    "ask() calling LLM",
-                    index_type=self.index_type.value,
-                    llm_url=llm_url,
-                    model=self.config.llm_model,
-                )
+        except (TimeoutError, httpx.TimeoutException, Exception) as e:
+            logger.warning(
+                "Index ask() search phase failed",
+                index_type=self.index_type.value,
+                error_type=type(e).__name__,
+                error=str(e) if not isinstance(e, TimeoutError) else "timed out",
+            )
+            return "", search_results
+
+        # ---- LLM call phase with 429 retry (outside index timeout) -----------
+        llm_url = f"{self.config.llm_base_url}/chat/completions"
+        logger.info(
+            "ask() calling LLM",
+            index_type=self.index_type.value,
+            llm_url=llm_url,
+            model=self.config.llm_model,
+        )
+
+        last_rl_retry_after: float | None = None
+
+        for rl_attempt in range(MAX_RATE_LIMIT_RETRIES):
+            try:
                 async with httpx.AsyncClient(timeout=60.0) as client:
                     resp = await client.post(
                         llm_url,
@@ -987,44 +1010,46 @@ class BaseIndexPlugin(ABC):
                             "options": {"num_ctx": 8192},
                         },
                     )
-                    if resp.is_success:
-                        data = resp.json()
-                        answer_text = data["choices"][0]["message"]["content"]
-                        # Extract answer from think tags if needed
-                        answer_text = self._extract_from_think_tags(answer_text)
-                        return answer_text, search_results
-                    else:
-                        logger.warning(
-                            "LLM call failed in ask",
-                            index_type=self.index_type.value,
-                            status=resp.status_code,
-                            error=resp.text[:200] if resp.text else "no error text",
-                        )
-                        # Return empty to let service aggregate and synthesize
-                        return "", search_results
+            except (httpx.TimeoutException, Exception) as e:
+                logger.warning(
+                    "LLM call failed in ask (non-429)",
+                    index_type=self.index_type.value,
+                    error=str(e),
+                )
+                return "", search_results
 
-        except TimeoutError:
-            logger.warning(
-                "Index ask() timed out",
-                index_type=self.index_type.value,
-                timeout=INDEX_TIMEOUT,
-            )
-            # Return search results even on timeout - service can aggregate them
-            return "", search_results
-        except httpx.TimeoutException:
-            logger.warning(
-                "LLM HTTP call timed out in ask",
-                index_type=self.index_type.value,
-            )
-            return "", search_results
-        except Exception as e:
-            logger.warning(
-                "RAG query failed",
-                index_type=self.index_type.value,
-                error=str(e),
-            )
-            # Return whatever search results we have for aggregation
-            return "", search_results
+            if resp.status_code == HTTP_TOO_MANY_REQUESTS:
+                retry_after = parse_retry_after_header(resp)
+                last_rl_retry_after = retry_after
+                backoff = (
+                    retry_after if retry_after is not None else float(2**rl_attempt)
+                )
+                logger.warning(
+                    "Ollama rate limited (429), retrying",
+                    provider="ollama",
+                    attempt=rl_attempt + 1,
+                    max_retries=MAX_RATE_LIMIT_RETRIES,
+                    backoff_duration=backoff,
+                )
+                if rl_attempt < MAX_RATE_LIMIT_RETRIES - 1:
+                    await asyncio.sleep(backoff)
+                continue
+
+            if resp.is_success:
+                data = resp.json()
+                answer_text = data["choices"][0]["message"]["content"]
+                answer_text = self._extract_from_think_tags(answer_text)
+                return answer_text, search_results
+            else:
+                logger.warning(
+                    "LLM call failed in ask",
+                    index_type=self.index_type.value,
+                    status=resp.status_code,
+                    error=resp.text[:200] if resp.text else "no error text",
+                )
+                return "", search_results
+
+        raise RateLimitError(provider="ollama", retry_after=last_rl_retry_after)
 
     async def count(self) -> int:
         """Get the number of documents in the index."""

roboco/services/optimal_brain/mentor.py

@@ -32,6 +32,12 @@ from roboco.models.optimal import (
     MentorResponse,
     SearchResult,
 )
+from roboco.services.exceptions import (
+    HTTP_TOO_MANY_REQUESTS,
+    MAX_RATE_LIMIT_RETRIES,
+    RateLimitError,
+    parse_retry_after_header,
+)
 
 logger = structlog.get_logger()
 
@@ -683,7 +689,12 @@ class MentorService:
         agent_profile: AgentProfile | None,
         journal_context: list[dict[str, Any]],
     ) -> str:
-        """Synthesize a personalized answer using LLM."""
+        """Synthesize a personalized answer using LLM.
+
+        Retries the Ollama call up to MAX_RATE_LIMIT_RETRIES times on HTTP 429,
+        respecting the Retry-After header.  Each individual attempt is bounded by
+        a 120-second asyncio timeout so a slow model cannot block the loop.
+        """
         import asyncio
 
         if not sources and not journal_context:
@@ -709,52 +720,73 @@ class MentorService:
             question, sources, conversation_context, agent_profile, journal_context
         )
 
-        # Call LLM
-        try:
-            async with asyncio.timeout(120.0):
-                async with httpx.AsyncClient(timeout=120.0) as client:
-                    response = await client.post(
-                        f"{settings.local_llm_base_url}/chat/completions",
-                        json={
-                            "model": settings.local_llm_model,
-                            "messages": [
-                                {"role": "system", "content": system_prompt},
-                                {"role": "user", "content": user_prompt},
-                            ],
-                            "max_tokens": 4096,
-                            "temperature": 0.5,
-                            "options": {"num_ctx": 8192},
-                        },
+        llm_url = f"{settings.local_llm_base_url}/chat/completions"
+        payload = {
+            "model": settings.local_llm_model,
+            "messages": [
+                {"role": "system", "content": system_prompt},
+                {"role": "user", "content": user_prompt},
+            ],
+            "max_tokens": 4096,
+            "temperature": 0.5,
+            "options": {"num_ctx": 8192},
+        }
+
+        last_rl_retry_after: float | None = None
+
+        for rl_attempt in range(MAX_RATE_LIMIT_RETRIES):
+            # Each attempt gets its own 120-second timeout
+            try:
+                async with asyncio.timeout(120.0):
+                    async with httpx.AsyncClient(timeout=120.0) as client:
+                        response = await client.post(llm_url, json=payload)
+            except (TimeoutError, httpx.TimeoutException):
+                logger.warning("LLM call timed out in mentor (120s)")
+                return self._fallback_answer(sources, agent_profile)
+            except Exception as e:
+                logger.warning("LLM call failed in mentor", error=str(e))
+                return self._fallback_answer(sources, agent_profile)
+
+            if response.status_code == HTTP_TOO_MANY_REQUESTS:
+                retry_after = parse_retry_after_header(response)
+                last_rl_retry_after = retry_after
+                backoff = (
+                    retry_after if retry_after is not None else float(2**rl_attempt)
+                )
+                logger.warning(
+                    "Ollama rate limited (429), retrying",
+                    provider="ollama",
+                    attempt=rl_attempt + 1,
+                    max_retries=MAX_RATE_LIMIT_RETRIES,
+                    backoff_duration=backoff,
+                )
+                if rl_attempt < MAX_RATE_LIMIT_RETRIES - 1:
+                    await asyncio.sleep(backoff)
+                continue
+
+            if response.is_success:
+                data = response.json()
+                raw_answer: str = data["choices"][0]["message"]["content"]
+                answer = self._extract_answer(raw_answer)
+
+                if not answer:
+                    logger.warning(
+                        "LLM response empty after extraction",
+                        model=settings.local_llm_model,
+                        original_length=len(raw_answer),
                     )
+                    return self._fallback_answer(sources, agent_profile)
 
-                    if response.is_success:
-                        data = response.json()
-                        raw_answer: str = data["choices"][0]["message"]["content"]
-                        answer = self._extract_answer(raw_answer)
+                return answer
+            else:
+                logger.warning(
+                    "LLM call failed in mentor",
+                    status=response.status_code,
+                    error=response.text[:200],
+                )
+                return self._fallback_answer(sources, agent_profile)
 
-                        if not answer:
-                            logger.warning(
-                                "LLM response empty after extraction",
-                                model=settings.local_llm_model,
-                                original_length=len(raw_answer),
-                            )
-                            return self._fallback_answer(sources, agent_profile)
-
-                        return answer
-                    else:
-                        logger.warning(
-                            "LLM call failed in mentor",
-                            status=response.status_code,
-                            error=response.text[:200],
-                        )
-                        return self._fallback_answer(sources, agent_profile)
-
-        except (TimeoutError, httpx.TimeoutException):
-            logger.warning("LLM call timed out in mentor (60s)")
-            return self._fallback_answer(sources, agent_profile)
-        except Exception as e:
-            logger.warning("LLM call failed in mentor", error=str(e))
-            return self._fallback_answer(sources, agent_profile)
+        raise RateLimitError(provider="ollama", retry_after=last_rl_retry_after)
 
     def _extract_answer(self, text: str) -> str:
         """Extract answer from LLM response, handling think tags."""

roboco/services/optimal_brain/ollama_embedder.py

@@ -22,13 +22,21 @@ from piragi.types import Chunk
 
 from roboco.config import settings
 from roboco.logging import get_logger
+from roboco.services.exceptions import (
+    HTTP_TOO_MANY_REQUESTS,
+    RateLimitError,
+    parse_retry_after_header,
+)
 
 logger = get_logger(__name__)
 
-# Retry configuration
+# Retry configuration — ConnectError / Timeout (existing, unchanged)
 MAX_RETRIES = 3
 RETRY_DELAY_BASE = 0.5  # seconds, exponential backoff
 
+# Retry configuration — HTTP 429 / RateLimitError (new outer loop)
+RATE_LIMIT_MAX_RETRIES = 5
+
 # Parallel processing configuration
 MAX_CONCURRENT_BATCHES = 4  # Number of batches to process in parallel
 DEFAULT_BATCH_SIZE = 32  # piragi's default batch size
@@ -304,7 +312,15 @@ class OllamaEmbedder:
         query: str,
         task_instruction: str | None = None,
     ) -> list[float]:
-        """Generate embedding for a single query with retry logic."""
+        """Generate embedding for a single query.
+
+        Retry behaviour (two independent concerns, non-overlapping):
+        - ConnectError / TimeoutException: up to MAX_RETRIES=3 attempts with
+          0.5/1/2 s exponential backoff (existing behaviour, unchanged).
+        - HTTP 429 (rate limit): outer loop up to RATE_LIMIT_MAX_RETRIES=5,
+          respecting Retry-After header.  A 429 response does NOT trigger the
+          ConnectError path.
+        """
         _ = task_instruction
 
         # Check cache first
@@ -313,79 +329,154 @@ class OllamaEmbedder:
             return cached
 
         client = self._get_sync_client()
-        last_error: Exception | None = None
+        last_rl_retry_after: float | None = None
 
-        for attempt in range(MAX_RETRIES):
-            try:
-                response = client.post(
-                    f"{self.base_url}/api/embed",
-                    json={"model": self.model, "input": query},
-                )
-                embeddings = self._handle_embed_response(response, input_count=1)
-                result = embeddings[0]
-                self._cache.put(query, result)
-                return result
+        for rl_attempt in range(RATE_LIMIT_MAX_RETRIES):
+            got_429 = False
+            last_error: Exception | None = None
 
-            except httpx.ConnectError as e:
-                last_error = OllamaConnectionError(
-                    f"Cannot connect to Ollama at {self.base_url}: {e}"
-                )
-            except httpx.TimeoutException as e:
-                last_error = OllamaConnectionError(f"Ollama request timed out: {e}")
-            except (OllamaModelError, OllamaEmbedderError):
-                raise
-            except Exception as e:
-                last_error = OllamaEmbedderError(f"Unexpected error: {e}")
+            # --- inner loop: ConnectError / Timeout (unchanged) ---
+            for attempt in range(MAX_RETRIES):
+                try:
+                    response = client.post(
+                        f"{self.base_url}/api/embed",
+                        json={"model": self.model, "input": query},
+                    )
+                    # 429 check — must NOT enter the ConnectError path
+                    if response.status_code == HTTP_TOO_MANY_REQUESTS:
+                        retry_after = parse_retry_after_header(response)
+                        last_rl_retry_after = retry_after
+                        backoff = (
+                            retry_after
+                            if retry_after is not None
+                            else float(2**rl_attempt)
+                        )
+                        logger.warning(
+                            "Ollama rate limited (429), retrying",
+                            provider="ollama",
+                            attempt=rl_attempt + 1,
+                            max_retries=RATE_LIMIT_MAX_RETRIES,
+                            backoff_duration=backoff,
+                        )
+                        got_429 = True
+                        break  # break inner loop; outer loop will sleep + retry
 
-            if attempt < MAX_RETRIES - 1:
-                delay = RETRY_DELAY_BASE * (2**attempt)
-                logger.warning(
-                    "Ollama embed_query retry",
-                    attempt=attempt + 1,
-                    delay=delay,
-                    error=str(last_error),
-                )
-                time.sleep(delay)
+                    embeddings = self._handle_embed_response(response, input_count=1)
+                    result = embeddings[0]
+                    self._cache.put(query, result)
+                    return result
 
-        raise last_error or OllamaEmbedderError("Max retries exceeded")
+                except httpx.ConnectError as e:
+                    last_error = OllamaConnectionError(
+                        f"Cannot connect to Ollama at {self.base_url}: {e}"
+                    )
+                except httpx.TimeoutException as e:
+                    last_error = OllamaConnectionError(f"Ollama request timed out: {e}")
+                except (OllamaModelError, OllamaEmbedderError):
+                    raise
+                except Exception as e:
+                    last_error = OllamaEmbedderError(f"Unexpected error: {e}")
+
+                if attempt < MAX_RETRIES - 1:
+                    delay = RETRY_DELAY_BASE * (2**attempt)
+                    logger.warning(
+                        "Ollama embed_query retry",
+                        attempt=attempt + 1,
+                        delay=delay,
+                        error=str(last_error),
+                    )
+                    time.sleep(delay)
+            # --- end inner loop ---
+
+            if not got_429:
+                # ConnectError / Timeout exhausted — same behaviour as before
+                raise last_error or OllamaEmbedderError("Max retries exceeded")
+
+            # 429: sleep and try again (outer loop)
+            backoff = (
+                last_rl_retry_after
+                if last_rl_retry_after is not None
+                else float(2**rl_attempt)
+            )
+            if rl_attempt < RATE_LIMIT_MAX_RETRIES - 1:
+                time.sleep(backoff)
+
+        raise RateLimitError(provider="ollama", retry_after=last_rl_retry_after)
 
     def _embed_batch_sync(
         self, client: httpx.Client, batch: list[str], batch_index: int
     ) -> list[list[float]]:
-        """Embed a single batch synchronously with retry logic."""
-        last_error: Exception | None = None
+        """Embed a single batch synchronously.
 
-        for attempt in range(MAX_RETRIES):
-            try:
-                response = client.post(
-                    f"{self.base_url}/api/embed",
-                    json={"model": self.model, "input": batch},
-                )
-                return self._handle_embed_response(response, input_count=len(batch))
+        Same two-concern retry composition as :meth:`embed_query`:
+        inner ConnectError/Timeout loop (unchanged) + outer 429 loop.
+        """
+        last_rl_retry_after: float | None = None
 
-            except httpx.ConnectError as e:
-                last_error = OllamaConnectionError(
-                    f"Cannot connect to Ollama at {self.base_url}: {e}"
-                )
-            except httpx.TimeoutException as e:
-                last_error = OllamaConnectionError(f"Ollama request timed out: {e}")
-            except (OllamaModelError, OllamaEmbedderError):
-                raise
-            except Exception as e:
-                last_error = OllamaEmbedderError(f"Unexpected error: {e}")
+        for rl_attempt in range(RATE_LIMIT_MAX_RETRIES):
+            got_429 = False
+            last_error: Exception | None = None
 
-            if attempt < MAX_RETRIES - 1:
-                delay = RETRY_DELAY_BASE * (2**attempt)
-                logger.warning(
-                    "Ollama embed_documents retry",
-                    attempt=attempt + 1,
-                    batch_index=batch_index,
-                    delay=delay,
-                    error=str(last_error),
-                )
-                time.sleep(delay)
+            for attempt in range(MAX_RETRIES):
+                try:
+                    response = client.post(
+                        f"{self.base_url}/api/embed",
+                        json={"model": self.model, "input": batch},
+                    )
+                    if response.status_code == HTTP_TOO_MANY_REQUESTS:
+                        retry_after = parse_retry_after_header(response)
+                        last_rl_retry_after = retry_after
+                        backoff = (
+                            retry_after
+                            if retry_after is not None
+                            else float(2**rl_attempt)
+                        )
+                        logger.warning(
+                            "Ollama rate limited (429), retrying",
+                            provider="ollama",
+                            attempt=rl_attempt + 1,
+                            max_retries=RATE_LIMIT_MAX_RETRIES,
+                            backoff_duration=backoff,
+                        )
+                        got_429 = True
+                        break
 
-        raise last_error or OllamaEmbedderError("Max retries exceeded")
+                    return self._handle_embed_response(response, input_count=len(batch))
+
+                except httpx.ConnectError as e:
+                    last_error = OllamaConnectionError(
+                        f"Cannot connect to Ollama at {self.base_url}: {e}"
+                    )
+                except httpx.TimeoutException as e:
+                    last_error = OllamaConnectionError(f"Ollama request timed out: {e}")
+                except (OllamaModelError, OllamaEmbedderError):
+                    raise
+                except Exception as e:
+                    last_error = OllamaEmbedderError(f"Unexpected error: {e}")
+
+                if attempt < MAX_RETRIES - 1:
+                    delay = RETRY_DELAY_BASE * (2**attempt)
+                    logger.warning(
+                        "Ollama embed_documents retry",
+                        attempt=attempt + 1,
+                        batch_index=batch_index,
+                        delay=delay,
+                        error=str(last_error),
+                    )
+                    time.sleep(delay)
+
+            if not got_429:
+                raise last_error or OllamaEmbedderError("Max retries exceeded")
+
+            backoff = (
+                last_rl_retry_after
+                if last_rl_retry_after is not None
+                else float(2**rl_attempt)
+            )
+            if rl_attempt < RATE_LIMIT_MAX_RETRIES - 1:
+                time.sleep(backoff)
+
+        raise RateLimitError(provider="ollama", retry_after=last_rl_retry_after)
 
     def _partition_cached_documents(
         self, documents: list[str]
@@ -464,49 +555,90 @@ class OllamaEmbedder:
         batch: list[str],
         batch_index: int,
     ) -> list[list[float]]:
-        """Embed a single batch with semaphore-limited concurrency."""
+        """Embed a single batch with semaphore-limited concurrency.
+
+        Same two-concern retry composition as :meth:`embed_query`:
+        inner ConnectError/Timeout loop (unchanged) + outer 429 loop (async).
+        """
         semaphore = self._get_semaphore()
 
         async with semaphore:
-            last_error: Exception | None = None
+            last_rl_retry_after: float | None = None
 
-            for attempt in range(MAX_RETRIES):
-                try:
-                    logger.debug(
-                        "Parallel embed batch",
-                        batch_index=batch_index,
-                        batch_size=len(batch),
-                        attempt=attempt,
-                    )
-                    response = await client.post(
-                        f"{self.base_url}/api/embed",
-                        json={"model": self.model, "input": batch},
-                    )
-                    return self._handle_embed_response(response, input_count=len(batch))
+            for rl_attempt in range(RATE_LIMIT_MAX_RETRIES):
+                got_429 = False
+                last_error: Exception | None = None
 
-                except httpx.ConnectError as e:
-                    last_error = OllamaConnectionError(
-                        f"Cannot connect to Ollama at {self.base_url}: {e}"
-                    )
-                except httpx.TimeoutException as e:
-                    last_error = OllamaConnectionError(f"Ollama request timed out: {e}")
-                except (OllamaModelError, OllamaEmbedderError):
-                    raise
-                except Exception as e:
-                    last_error = OllamaEmbedderError(f"Unexpected error: {e}")
+                for attempt in range(MAX_RETRIES):
+                    try:
+                        logger.debug(
+                            "Parallel embed batch",
+                            batch_index=batch_index,
+                            batch_size=len(batch),
+                            attempt=attempt,
+                        )
+                        response = await client.post(
+                            f"{self.base_url}/api/embed",
+                            json={"model": self.model, "input": batch},
+                        )
+                        if response.status_code == HTTP_TOO_MANY_REQUESTS:
+                            retry_after = parse_retry_after_header(response)
+                            last_rl_retry_after = retry_after
+                            backoff = (
+                                retry_after
+                                if retry_after is not None
+                                else float(2**rl_attempt)
+                            )
+                            logger.warning(
+                                "Ollama rate limited (429), retrying",
+                                provider="ollama",
+                                attempt=rl_attempt + 1,
+                                max_retries=RATE_LIMIT_MAX_RETRIES,
+                                backoff_duration=backoff,
+                            )
+                            got_429 = True
+                            break
 
-                if attempt < MAX_RETRIES - 1:
-                    delay = RETRY_DELAY_BASE * (2**attempt)
-                    logger.warning(
-                        "Parallel embed batch retry",
-                        batch_index=batch_index,
-                        attempt=attempt + 1,
-                        delay=delay,
-                        error=str(last_error),
-                    )
-                    await asyncio.sleep(delay)
+                        return self._handle_embed_response(
+                            response, input_count=len(batch)
+                        )
 
-            raise last_error or OllamaEmbedderError("Max retries exceeded")
+                    except httpx.ConnectError as e:
+                        last_error = OllamaConnectionError(
+                            f"Cannot connect to Ollama at {self.base_url}: {e}"
+                        )
+                    except httpx.TimeoutException as e:
+                        last_error = OllamaConnectionError(
+                            f"Ollama request timed out: {e}"
+                        )
+                    except (OllamaModelError, OllamaEmbedderError):
+                        raise
+                    except Exception as e:
+                        last_error = OllamaEmbedderError(f"Unexpected error: {e}")
+
+                    if attempt < MAX_RETRIES - 1:
+                        delay = RETRY_DELAY_BASE * (2**attempt)
+                        logger.warning(
+                            "Parallel embed batch retry",
+                            batch_index=batch_index,
+                            attempt=attempt + 1,
+                            delay=delay,
+                            error=str(last_error),
+                        )
+                        await asyncio.sleep(delay)
+
+                if not got_429:
+                    raise last_error or OllamaEmbedderError("Max retries exceeded")
+
+                backoff = (
+                    last_rl_retry_after
+                    if last_rl_retry_after is not None
+                    else float(2**rl_attempt)
+                )
+                if rl_attempt < RATE_LIMIT_MAX_RETRIES - 1:
+                    await asyncio.sleep(backoff)
+
+            raise RateLimitError(provider="ollama", retry_after=last_rl_retry_after)
 
     async def _run_parallel_batches(
         self, batches: list[list[str]]
@@ -650,49 +782,93 @@ class OllamaEmbedder:
         return chunks
 
     async def aembed_query(self, query: str) -> list[float]:
-        """Async version of embed_query with retry logic and caching."""
+        """Async version of embed_query with retry logic and caching.
+
+        Same two-concern retry composition as :meth:`embed_query`:
+        inner ConnectError/Timeout loop (unchanged) + outer 429 loop (async).
+        """
         # Check cache first
         cached = self._cache.get(query)
         if cached is not None:
             return cached
 
-        last_error: Exception | None = None
+        last_rl_retry_after: float | None = None
 
-        for attempt in range(MAX_RETRIES):
-            # Create fresh client each attempt to avoid event loop issues
-            async with self._create_async_client() as client:
-                try:
-                    response = await client.post(
-                        f"{self.base_url}/api/embed",
-                        json={"model": self.model, "input": query},
+        for rl_attempt in range(RATE_LIMIT_MAX_RETRIES):
+            got_429 = False
+            last_error: Exception | None = None
+
+            for attempt in range(MAX_RETRIES):
+                # Create fresh client each attempt to avoid event loop issues
+                async with self._create_async_client() as client:
+                    try:
+                        response = await client.post(
+                            f"{self.base_url}/api/embed",
+                            json={"model": self.model, "input": query},
+                        )
+                        if response.status_code == HTTP_TOO_MANY_REQUESTS:
+                            retry_after = parse_retry_after_header(response)
+                            last_rl_retry_after = retry_after
+                            backoff = (
+                                retry_after
+                                if retry_after is not None
+                                else float(2**rl_attempt)
+                            )
+                            logger.warning(
+                                "Ollama rate limited (429), retrying",
+                                provider="ollama",
+                                attempt=rl_attempt + 1,
+                                max_retries=RATE_LIMIT_MAX_RETRIES,
+                                backoff_duration=backoff,
+                            )
+                            got_429 = True
+                            break
+
+                        embeddings = self._handle_embed_response(
+                            response, input_count=1
+                        )
+                        result = embeddings[0]
+                        self._cache.put(query, result)
+                        return result
+
+                    except httpx.ConnectError as e:
+                        last_error = OllamaConnectionError(
+                            f"Cannot connect to Ollama at {self.base_url}: {e}"
+                        )
+                    except httpx.TimeoutException as e:
+                        last_error = OllamaConnectionError(
+                            f"Ollama request timed out: {e}"
+                        )
+                    except (OllamaModelError, OllamaEmbedderError):
+                        raise
+                    except Exception as e:
+                        last_error = OllamaEmbedderError(f"Unexpected error: {e}")
+
+                if got_429:
+                    break  # exit inner loop cleanly
+
+                if attempt < MAX_RETRIES - 1:
+                    delay = RETRY_DELAY_BASE * (2**attempt)
+                    logger.warning(
+                        "Ollama aembed_query retry",
+                        attempt=attempt + 1,
+                        delay=delay,
+                        error=str(last_error),
                     )
-                    embeddings = self._handle_embed_response(response, input_count=1)
-                    result = embeddings[0]
-                    self._cache.put(query, result)
-                    return result
+                    await asyncio.sleep(delay)
 
-                except httpx.ConnectError as e:
-                    last_error = OllamaConnectionError(
-                        f"Cannot connect to Ollama at {self.base_url}: {e}"
-                    )
-                except httpx.TimeoutException as e:
-                    last_error = OllamaConnectionError(f"Ollama request timed out: {e}")
-                except (OllamaModelError, OllamaEmbedderError):
-                    raise
-                except Exception as e:
-                    last_error = OllamaEmbedderError(f"Unexpected error: {e}")
+            if not got_429:
+                raise last_error or OllamaEmbedderError("Max retries exceeded")
 
-            if attempt < MAX_RETRIES - 1:
-                delay = RETRY_DELAY_BASE * (2**attempt)
-                logger.warning(
-                    "Ollama aembed_query retry",
-                    attempt=attempt + 1,
-                    delay=delay,
-                    error=str(last_error),
-                )
-                await asyncio.sleep(delay)
+            backoff = (
+                last_rl_retry_after
+                if last_rl_retry_after is not None
+                else float(2**rl_attempt)
+            )
+            if rl_attempt < RATE_LIMIT_MAX_RETRIES - 1:
+                await asyncio.sleep(backoff)
 
-        raise last_error or OllamaEmbedderError("Max retries exceeded")
+        raise RateLimitError(provider="ollama", retry_after=last_rl_retry_after)
 
     async def aembed_documents(
         self, documents: list[str], batch_size: int = DEFAULT_BATCH_SIZE

roboco/services/optimal_brain/validator.py

@@ -19,6 +19,12 @@ import structlog
 
 from roboco.config import settings
 from roboco.models.optimal import SearchResult, ValidationResult
+from roboco.services.exceptions import (
+    HTTP_TOO_MANY_REQUESTS,
+    MAX_RATE_LIMIT_RETRIES,
+    RateLimitError,
+    parse_retry_after_header,
+)
 
 logger = structlog.get_logger()
 
@@ -492,6 +498,10 @@ class ValidatorService:
         """
         Use LLM to validate context against standards.
 
+        Retries the Ollama call up to MAX_RATE_LIMIT_RETRIES times on HTTP 429,
+        respecting the Retry-After header.  Each attempt is bounded by
+        LLM_TIMEOUT_SECONDS so a single slow call cannot block the retry loop.
+
         Returns:
             Tuple of (violations, warnings)
         """
@@ -512,41 +522,63 @@ RELEVANT STANDARDS:
 Analyze the context and identify any violations of the standards above.
 Return your analysis as JSON."""
 
-        try:
-            async with asyncio.timeout(LLM_TIMEOUT_SECONDS):
-                async with httpx.AsyncClient(timeout=LLM_TIMEOUT_SECONDS) as client:
-                    response = await client.post(
-                        f"{settings.local_llm_base_url}/chat/completions",
-                        json={
-                            "model": settings.local_llm_model,
-                            "messages": [
-                                {"role": "system", "content": VALIDATION_SYSTEM_PROMPT},
-                                {"role": "user", "content": user_prompt},
-                            ],
-                            "max_tokens": 2048,
-                            "temperature": 0.1,  # Low temp for consistent analysis
-                            "options": {"num_ctx": 8192},
-                        },
-                    )
+        llm_url = f"{settings.local_llm_base_url}/chat/completions"
+        payload = {
+            "model": settings.local_llm_model,
+            "messages": [
+                {"role": "system", "content": VALIDATION_SYSTEM_PROMPT},
+                {"role": "user", "content": user_prompt},
+            ],
+            "max_tokens": 2048,
+            "temperature": 0.1,  # Low temp for consistent analysis
+            "options": {"num_ctx": 8192},
+        }
 
-                    if response.is_success:
-                        data = response.json()
-                        raw_response = data["choices"][0]["message"]["content"]
-                        return self._parse_llm_response(raw_response)
-                    else:
-                        logger.warning(
-                            "LLM validation call failed",
-                            status=response.status_code,
-                            error=response.text[:200],
-                        )
-                        raise RuntimeError(f"LLM call failed: {response.status_code}")
+        last_rl_retry_after: float | None = None
 
-        except TimeoutError:
-            logger.warning("LLM validation timed out")
-            raise
-        except httpx.TimeoutException:
-            logger.warning("LLM validation HTTP timeout")
-            raise
+        for rl_attempt in range(MAX_RATE_LIMIT_RETRIES):
+            # Each attempt gets its own timeout — raises if the call hangs
+            try:
+                async with asyncio.timeout(LLM_TIMEOUT_SECONDS):
+                    async with httpx.AsyncClient(timeout=LLM_TIMEOUT_SECONDS) as client:
+                        response = await client.post(llm_url, json=payload)
+            except TimeoutError:
+                logger.warning("LLM validation timed out")
+                raise
+            except httpx.TimeoutException:
+                logger.warning("LLM validation HTTP timeout")
+                raise
+
+            if response.status_code == HTTP_TOO_MANY_REQUESTS:
+                retry_after = parse_retry_after_header(response)
+                last_rl_retry_after = retry_after
+                backoff = (
+                    retry_after if retry_after is not None else float(2**rl_attempt)
+                )
+                logger.warning(
+                    "Ollama rate limited (429), retrying",
+                    provider="ollama",
+                    attempt=rl_attempt + 1,
+                    max_retries=MAX_RATE_LIMIT_RETRIES,
+                    backoff_duration=backoff,
+                )
+                if rl_attempt < MAX_RATE_LIMIT_RETRIES - 1:
+                    await asyncio.sleep(backoff)
+                continue
+
+            if response.is_success:
+                data = response.json()
+                raw_response = data["choices"][0]["message"]["content"]
+                return self._parse_llm_response(raw_response)
+            else:
+                logger.warning(
+                    "LLM validation call failed",
+                    status=response.status_code,
+                    error=response.text[:200],
+                )
+                raise RuntimeError(f"LLM call failed: {response.status_code}")
+
+        raise RateLimitError(provider="ollama", retry_after=last_rl_retry_after)
 
     def _build_standards_context(self, standards: list[SearchResult]) -> str:
         """Build a formatted string of standards for the LLM."""