mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
Fix: rate limit real probe (#110)
* fix(rate-limit): real provider liveness probe instead of time-based stub
The rate-limit recovery sweeper cleared a provider and resumed parked agents
purely on elapsed time — _do_probe was a stub that always returned True once
the retry_after window passed, so it never confirmed the provider had actually
stopped rate-limiting us. Under a sustained limit that resumes agents straight
into another 429, re-parking them: avoidable churn.
Make the probe real. _do_probe now issues a free, unmetered liveness call —
Anthropic GET /v1/models or Ollama GET /api/tags — and treats any non-429
response as the limit having lifted. A 429 keeps the provider parked; a
network error keeps it parked too (retry next sweep). When the provider can't
be probed (no API key, or an unrecognized provider), it falls back to the
prior time-expiry optimism rather than stranding agents. _probe_target keeps
URL/header resolution separate and testable, and _do_probe stays a
monkeypatchable boundary so the existing sweep tests are unaffected.
Also drop two acceptance-criteria-number labels from comments in this file.
* chore(rate-limit): clear merged gate debt in rate-limit tests + deps lint
The rate-limit PR landed with ruff violations the full gate flags but the
authors' runs missed: test_rate_limit_sweep.py was unformatted, and
test_rate_limit_tracker.py had unsorted/unused imports and magic-value
comparisons. Format the sweep test, drop the dead imports, and bind the
magic comparison values to locals. Also strip acceptance-criteria-number
labels from comments/docstrings across the three rate-limit test files
(leaving genuine acceptance_criteria=[...] test data untouched), and add
api/deps.py to the PLC0415 per-file-ignore — it is the DI wiring hub and
defers a couple of service imports to call time to avoid import cycles,
the same rationale already applied to api/routes, runtime, and services.
* fix(rate-limit): resolve redis type errors in RateLimitStateTracker
A cold mypy run (the gate's true state — prior passes were warm-cache only)
flagged four redis-typing errors in rate_limit_tracker.py that the merge
missed: three unused type:ignore[type-arg] on redis.Redis, and an
aclose() the bundled redis type stub doesn't expose.
Drop the now-unused ignores, and close the scan client via
'async with redis.from_url(...) as r:' instead of a finally-block
aclose(). The context manager closes the client on exit using the modern
redis.asyncio API — no deprecated close(), no stub-missing aclose(), no
suppression. Extend the test's redis mock to model the async
context-manager protocol so it returns itself on enter.
* test(prompter): pass route='main_pm' in the product main-PM routing test
Pre-existing master failure, unrelated to the rate-limit work. The test is
named ...product_routes_to_main_pm and asserts team=MAIN_PM, but called
confirm_live_draft without a route, so it got the 'board' default — which
assigns the Product Owner and yields team=BOARD by design (the board-review
path keeps the root at team=board until the CEO approves). The Main-PM path
is selected with route='main_pm', exactly as the sibling
...main_pm_route_assigns_main_pm test does. Add the missing kwarg so the test
verifies the path it names; behaviour under test is unchanged.
* Updated uv.lock
* refactor(complexity): bring all rank-C blocks under the xenon B ceiling
The full quality gate's xenon step (--max-absolute B --max-modules A
--max-average A) failed on eight rank-C blocks plus the extraction module
average — debt the rate-limit and token-analytics merges deferred. Reduce
each by extracting cohesive helpers, behaviour unchanged:
- orchestrator._probe_one_provider: split into _too_early_to_probe,
_on_probe_success, _on_probe_failure, _parked_agents_for.
- rate_limit_tracker.list_rate_limited_providers: extract _read_rate_limited_entry
and a _decode helper.
- trigger_filter.decide_spawn: extract _stale_trigger_decision (drops the
PLR0911 suppression too).
- ollama_embedder (embed_query, _embed_batch_sync, aembed_query,
_embed_batch_async): share _rl_backoff / _map_embed_error / _log_429 /
_sleep_connect_retry / _asleep_connect_retry; remove a dead post-loop guard
in aembed_query.
- mentor._synthesize_answer: extract _select_system_prompt and
_answer_from_response.
- indexes/base.ask: extract the 429-retried LLM call into _ask_llm.
- extraction.__init__: extract _compile_patterns so the module average
lands at rank A.
xenon now exits 0; rate-limit, optimal_brain, extraction, and events suites
all green.
* chore(deps): drop obsolete types-redis stub; honor redis 8.0 inline types
types-redis 4.6 (typed for redis 4.x) shadowed redis 8.0's own inline types,
which both masked real annotation mismatches in stream_bus.py and forced
awkward workarounds elsewhere. The stale stub is why the mypy gate only ever
passed warm-cached: a cold run under the wrong stub disagreed with the code.
Remove types-redis (and its orphaned transitive stubs) so mypy uses redis's
shipped types. That surfaces that xreadgroup/xclaim return bytes-keyed records
while _handle_message is annotated str — the code already decodes bytes
defensively, so this is an annotation gap, not a runtime bug. Make the types
honest: cast each result to its concrete shape and decode the stream name and
message id to str at the dispatch boundary via a _to_str helper.
mypy roboco/ is now clean cold (247 files) against redis's real types; events
suite green.
* Updated uv.lock
* fix(workspace): install the dev extra so agents can run make quality
Agent workspaces were set up with plain `uv sync`, which installs only the
project's default dependency group (pytest) — not the `dev` *extra* where the
gate tools live (ruff, mypy, xenon, radon, vulture, bandit, deptry). So an
agent's .venv had pytest but no linters, and `make quality` died immediately
on `ruff: command not found`. Agents literally could not lint, type-check, or
complexity-check their own work, which is how format/mypy/xenon debt merged
unseen. Sync the `dev` extra (`uv sync --extra dev`) so the workspace gets the
full toolchain the setup's own docstring already promised.
* fix(panel): rate-limit endpoint shape + websocket path
Two panel-facing breakages from the rate-limit rework:
- GET /api/system/rate-limits returned a raw list, but the panel store reads
response.entries — so `r.entries is not iterable` crashed the banner sync on
page load. Return the panel's contract: a { entries: [...] } envelope whose
items are camelCase {provider, affectedAgents, hitAt, resumeAt,
retryAfterSeconds}, derived from the raw Redis state (resumeAt = hitAt +
retryAfter).
- The rate-limit websocket hook passed "/ws/system" while getWebSocketUrl()
already supplies the "/ws" base, producing the doubled "/ws/ws/system" URL.
Pass "/system" to match the agents/channels/notifications hooks.
Note: the backend /ws/system endpoint itself does not yet exist (the rework
shipped the panel hook only); the REST fix keeps the banner correct on load
and reconnect until that endpoint is built.
* test(workspace): assert uv sync installs the dev extra
Follow the workspace setup change: the dependency-install command is now
`uv sync --extra dev` so the agent workspace gets the lint/type/complexity
toolchain. Update the three assertions that pinned the old `uv sync`.
* feat(ws): add /ws/system stream and bridge rate-limit events to the panel
The rate-limit rework shipped the panel's websocket hook but no backend: there
was no /ws/system endpoint and nothing forwarded RATE_LIMIT_HIT/LIFTED to a
socket, so the banner got no live updates.
Build the missing half:
- ConnectionManager grows a system-wide connection set with connect_system /
broadcast_system, and disconnect() now clears it.
- A /ws/system websocket endpoint (operator stream, no per-agent keying) with
the same connected + ping/pong lifecycle as the other streams.
- websocket_bridge subscribes RATE_LIMIT_HIT/LIFTED and forwards each to
broadcast_system tagged with the type the panel switches on. Both events
ride the same StreamEventBus singleton, and the subscriptions register
before start_listening(), so the consumer reads their streams.
Pairs with the panel hook now passing '/system' (getWebSocketUrl supplies the
'/ws' base). Covered by handler, manager, and endpoint-lifecycle tests.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
+56
-15
@@ -7,41 +7,82 @@ the per-resource routers (agents, tasks, etc.).
|
||||
Currently exposed:
|
||||
|
||||
GET /api/system/rate-limits
|
||||
Returns the current per-provider rate-limit state from Redis.
|
||||
Returns the current per-provider rate-limit state from Redis,
|
||||
shaped for the control panel's rate-limit store.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel, ConfigDict
|
||||
from pydantic.alias_generators import to_camel
|
||||
|
||||
from roboco.services.gateway.rate_limit_tracker import RateLimitStateTracker
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
class _CamelModel(BaseModel):
|
||||
"""Serialize with camelCase aliases so the panel consumes fields directly."""
|
||||
|
||||
model_config = ConfigDict(alias_generator=to_camel, populate_by_name=True)
|
||||
|
||||
|
||||
class RateLimitEntry(_CamelModel):
|
||||
"""A single provider's active rate-limit state, in the panel's shape."""
|
||||
|
||||
provider: str
|
||||
affected_agents: list[str]
|
||||
hit_at: str | None
|
||||
resume_at: str | None
|
||||
retry_after_seconds: float | None
|
||||
|
||||
|
||||
class RateLimitListResponse(_CamelModel):
|
||||
"""The envelope the panel's rate-limit store expects: ``{ "entries": [...] }``."""
|
||||
|
||||
entries: list[RateLimitEntry]
|
||||
|
||||
|
||||
def _resume_at(hit_at: str | None, retry_after: float | None) -> str | None:
|
||||
"""Estimated lift time = hit_at + retry_after, ISO; falls back to hit_at."""
|
||||
if not hit_at or retry_after is None:
|
||||
return hit_at
|
||||
try:
|
||||
lifted = datetime.fromisoformat(hit_at) + timedelta(seconds=retry_after)
|
||||
except (ValueError, TypeError):
|
||||
return hit_at
|
||||
return lifted.isoformat()
|
||||
|
||||
|
||||
@router.get(
|
||||
"/rate-limits",
|
||||
summary="List per-provider rate-limit state",
|
||||
response_model=list[dict[str, Any]],
|
||||
response_model=RateLimitListResponse,
|
||||
tags=["System"],
|
||||
)
|
||||
async def get_rate_limits() -> list[dict[str, Any]]:
|
||||
async def get_rate_limits() -> RateLimitListResponse:
|
||||
"""Return rate-limit state for every currently rate-limited provider.
|
||||
|
||||
Backed by
|
||||
:class:`~roboco.services.gateway.rate_limit_tracker.RateLimitStateTracker`.
|
||||
Each entry is the raw state dict (``rate_limited``, ``activated_at``,
|
||||
``retry_after``, ``affected_agents``, ``probe_failures``) augmented
|
||||
with a ``provider`` key.
|
||||
Shaped as the panel's rate-limit store consumes it — a
|
||||
``{ "entries": [...] }`` envelope where each entry is
|
||||
``{provider, affectedAgents, hitAt, resumeAt, retryAfterSeconds}``.
|
||||
|
||||
Returns an empty list ``[]`` when no provider is currently rate-limited.
|
||||
``entries`` is empty when no provider is currently rate-limited.
|
||||
"""
|
||||
entries = await RateLimitStateTracker.list_rate_limited_providers()
|
||||
result: list[dict[str, Any]] = []
|
||||
for provider, state in entries:
|
||||
item = dict(state)
|
||||
item["provider"] = provider
|
||||
result.append(item)
|
||||
return result
|
||||
states = await RateLimitStateTracker.list_rate_limited_providers()
|
||||
entries = [
|
||||
RateLimitEntry(
|
||||
provider=provider,
|
||||
affected_agents=state.get("affected_agents", []),
|
||||
hit_at=state.get("activated_at"),
|
||||
resume_at=_resume_at(state.get("activated_at"), state.get("retry_after")),
|
||||
retry_after_seconds=state.get("retry_after"),
|
||||
)
|
||||
for provider, state in states
|
||||
]
|
||||
return RateLimitListResponse(entries=entries)
|
||||
|
||||
@@ -56,6 +56,9 @@ class ConnectionManager:
|
||||
# agent_id -> set of websockets (for notifications)
|
||||
self.notification_connections: dict[UUID, set[WebSocket]] = {}
|
||||
|
||||
# Operator/system-wide stream (rate limits, etc.) — no per-agent keying.
|
||||
self.system_connections: set[WebSocket] = set()
|
||||
|
||||
# websocket -> agent_id (for tracking who is connected)
|
||||
self.connection_agents: dict[WebSocket, UUID] = {}
|
||||
|
||||
@@ -105,6 +108,11 @@ class ConnectionManager:
|
||||
self.notification_connections[agent_id].add(websocket)
|
||||
self.connection_agents[websocket] = agent_id
|
||||
|
||||
async def connect_system(self, websocket: WebSocket) -> None:
|
||||
"""Connect to the operator/system-wide stream (rate limits, etc.)."""
|
||||
await websocket.accept()
|
||||
self.system_connections.add(websocket)
|
||||
|
||||
def disconnect(self, websocket: WebSocket) -> None:
|
||||
"""Remove a websocket from all subscriptions."""
|
||||
# Remove from channel connections
|
||||
@@ -123,6 +131,9 @@ class ConnectionManager:
|
||||
for connections in self.notification_connections.values():
|
||||
connections.discard(websocket)
|
||||
|
||||
# Remove from the system-wide stream
|
||||
self.system_connections.discard(websocket)
|
||||
|
||||
# Remove from tracking
|
||||
self.connection_agents.pop(websocket, None)
|
||||
|
||||
@@ -168,6 +179,17 @@ class ConnectionManager:
|
||||
return_exceptions=True,
|
||||
)
|
||||
|
||||
async def broadcast_system(self, message: dict[str, Any]) -> None:
|
||||
"""Broadcast a message to all operator/system-wide subscribers."""
|
||||
if not self.system_connections:
|
||||
return
|
||||
|
||||
data = json.dumps(message, default=str)
|
||||
await asyncio.gather(
|
||||
*[conn.send_text(data) for conn in self.system_connections],
|
||||
return_exceptions=True,
|
||||
)
|
||||
|
||||
def get_channel_subscriber_count(self, channel_id: UUID) -> int:
|
||||
"""Get number of subscribers to a channel."""
|
||||
return len(self.channel_connections.get(channel_id, set()))
|
||||
@@ -412,6 +434,29 @@ async def notification_stream(
|
||||
manager.disconnect(websocket)
|
||||
|
||||
|
||||
@router.websocket("/system")
|
||||
async def system_stream(websocket: WebSocket) -> None:
|
||||
"""Operator/system-wide WebSocket stream.
|
||||
|
||||
Carries system-level events for the control panel — currently the
|
||||
rate-limit lifecycle (``RATE_LIMIT_HIT`` / ``RATE_LIMIT_LIFTED``), bridged
|
||||
from the event bus by ``websocket_bridge``. No per-agent keying or auth:
|
||||
it's a read-only operator stream behind the panel's own access controls.
|
||||
"""
|
||||
await manager.connect_system(websocket)
|
||||
|
||||
try:
|
||||
await websocket.send_json({"type": "connected"})
|
||||
|
||||
while True:
|
||||
data = await websocket.receive_text()
|
||||
if data == "ping":
|
||||
await websocket.send_text("pong")
|
||||
|
||||
except WebSocketDisconnect:
|
||||
manager.disconnect(websocket)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Helper Functions for Broadcasting
|
||||
# =============================================================================
|
||||
|
||||
@@ -15,6 +15,11 @@ from roboco.events import Event, EventType, get_event_bus
|
||||
|
||||
logger = structlog.get_logger()
|
||||
|
||||
_RATE_LIMIT_WS_TYPES = {
|
||||
EventType.RATE_LIMIT_HIT: "RATE_LIMIT_HIT",
|
||||
EventType.RATE_LIMIT_LIFTED: "RATE_LIMIT_LIFTED",
|
||||
}
|
||||
|
||||
|
||||
# Handler for notification events
|
||||
async def _handle_notification_sent(event: Event) -> None:
|
||||
@@ -126,6 +131,19 @@ async def _handle_agent_event(event: Event) -> None:
|
||||
)
|
||||
|
||||
|
||||
async def _handle_rate_limit_event(event: Event) -> None:
|
||||
"""Forward RATE_LIMIT_HIT/LIFTED events to operator system WS clients.
|
||||
|
||||
The published payload already carries the panel's fields
|
||||
(``provider``, ``affectedAgents``, ``retryAfterSeconds``, ``timestamp``);
|
||||
we only tag it with the discriminating ``type`` the panel switches on.
|
||||
"""
|
||||
ws_type = _RATE_LIMIT_WS_TYPES.get(event.type)
|
||||
if ws_type is None:
|
||||
return
|
||||
await manager.broadcast_system({"type": ws_type, **event.data})
|
||||
|
||||
|
||||
def register_websocket_bridge_handlers() -> None:
|
||||
"""
|
||||
Register event handlers that forward events to WebSocket clients.
|
||||
@@ -150,6 +168,10 @@ def register_websocket_bridge_handlers() -> None:
|
||||
bus.subscribe(EventType.AGENT_RESUMED, _handle_agent_event)
|
||||
bus.subscribe(EventType.AGENT_ERROR, _handle_agent_event)
|
||||
|
||||
# Rate-limit lifecycle -> system WebSocket (panel banner)
|
||||
bus.subscribe(EventType.RATE_LIMIT_HIT, _handle_rate_limit_event)
|
||||
bus.subscribe(EventType.RATE_LIMIT_LIFTED, _handle_rate_limit_event)
|
||||
|
||||
logger.info("WebSocket bridge handlers registered")
|
||||
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ import contextlib
|
||||
import os
|
||||
import socket
|
||||
from collections.abc import Callable, Coroutine
|
||||
from typing import Any
|
||||
from typing import Any, cast
|
||||
|
||||
import redis.asyncio as redis
|
||||
import structlog
|
||||
@@ -230,21 +230,33 @@ class StreamEventBus:
|
||||
logger.error("Error in stream event loop", error=str(e))
|
||||
await asyncio.sleep(1)
|
||||
|
||||
@staticmethod
|
||||
def _to_str(value: object) -> str:
|
||||
"""Decode a Redis stream/key value (bytes or str) to str."""
|
||||
return value.decode() if isinstance(value, bytes) else str(value)
|
||||
|
||||
async def _listen_tick(self, stream_dict: dict[str, str]) -> None:
|
||||
"""Block for one XREADGROUP cycle and dispatch any messages."""
|
||||
assert self._redis is not None
|
||||
results = await self._redis.xreadgroup(
|
||||
# redis returns bytes-keyed records (no decode_responses); the concrete
|
||||
# shape is list[(stream, [(id, fields)])]. _handle_message takes str, so
|
||||
# decode the stream name and message id at this boundary.
|
||||
raw = await self._redis.xreadgroup(
|
||||
self.group_name,
|
||||
self.consumer_name,
|
||||
stream_dict,
|
||||
cast("dict[Any, Any]", stream_dict),
|
||||
count=10,
|
||||
block=5000,
|
||||
)
|
||||
if not results:
|
||||
if not raw:
|
||||
return
|
||||
results = cast(
|
||||
"list[tuple[bytes, list[tuple[bytes, dict[bytes, bytes]]]]]", raw
|
||||
)
|
||||
for stream_name, messages in results:
|
||||
stream_str = self._to_str(stream_name)
|
||||
for message_id, data in messages:
|
||||
await self._handle_message(stream_name, message_id, data)
|
||||
await self._handle_message(stream_str, self._to_str(message_id), data)
|
||||
|
||||
async def _handle_response_error(
|
||||
self, exc: ResponseError, streams: list[str]
|
||||
@@ -345,17 +357,18 @@ class StreamEventBus:
|
||||
"""Claim a single idle message and process it; return count recovered."""
|
||||
if self._redis is None:
|
||||
raise RuntimeError("Invariant: self._redis must be set — guarded by caller")
|
||||
claimed = await self._redis.xclaim(
|
||||
raw = await self._redis.xclaim(
|
||||
stream,
|
||||
self.group_name,
|
||||
self.consumer_name,
|
||||
min_idle_time=idle_time_ms,
|
||||
message_ids=[msg_id],
|
||||
)
|
||||
if not claimed:
|
||||
if not raw:
|
||||
return 0
|
||||
claimed = cast("list[tuple[bytes, dict[bytes, bytes]]]", raw)
|
||||
for claim_id, data in claimed:
|
||||
await self._handle_message(stream, claim_id, data)
|
||||
await self._handle_message(stream, self._to_str(claim_id), data)
|
||||
return 1
|
||||
|
||||
async def _recover_stream(self, stream: str, idle_time_ms: int) -> int:
|
||||
|
||||
+131
-86
@@ -75,6 +75,15 @@ AGENT_BASE_IMAGE = "roboco-agent-base"
|
||||
# _sweep_budget_exceeded) to build the SDK health/usage URL.
|
||||
SDK_PORT: int = 9000
|
||||
|
||||
# Rate-limit recovery probe: a free, unmetered liveness call confirms a
|
||||
# provider has stopped rate-limiting us before parked agents are resumed.
|
||||
# Listing models / tags costs no tokens; a non-429 response means lifted.
|
||||
_ANTHROPIC_PROBE_BASE = "https://api.anthropic.com"
|
||||
_PROBE_TIMEOUT_SECONDS = 10.0
|
||||
_HTTP_TOO_MANY_REQUESTS = 429
|
||||
# Consecutive failed recovery probes before the CEO is notified once per episode.
|
||||
_CEO_NOTIFY_THRESHOLD = 10
|
||||
|
||||
# The intake (prompter) agent: a single seeded, board-adjacent interviewer.
|
||||
# Unlike delivery agents it is never dispatched and runs ONE persistent
|
||||
# container at a time (single CEO → one live chat). See the INTAKE section
|
||||
@@ -3978,7 +3987,7 @@ Start by:
|
||||
)
|
||||
|
||||
# =========================================================================
|
||||
# RATE-LIMIT PROBE LOOP (AC4, AC8)
|
||||
# RATE-LIMIT PROBE LOOP
|
||||
# =========================================================================
|
||||
|
||||
async def _rate_limit_probe_loop(self) -> None:
|
||||
@@ -4037,106 +4046,142 @@ Start by:
|
||||
|
||||
return RateLimitStateTracker(provider)
|
||||
|
||||
async def _probe_one_provider(self, provider: str, state: dict[str, Any]) -> None:
|
||||
"""Probe a single rate-limited provider and handle the outcome."""
|
||||
# Only start probing after estimated_lift_at has passed.
|
||||
activated_at_raw: str | None = state.get("activated_at")
|
||||
retry_after: float | None = state.get("retry_after")
|
||||
if activated_at_raw and retry_after is not None:
|
||||
try:
|
||||
activated_at = datetime.fromisoformat(activated_at_raw)
|
||||
estimated_lift_at = activated_at + timedelta(seconds=retry_after)
|
||||
if datetime.now(UTC) < estimated_lift_at:
|
||||
return # Too early — wait until after estimated lift time
|
||||
except (ValueError, TypeError):
|
||||
pass # Malformed timestamps: proceed with probe anyway
|
||||
@staticmethod
|
||||
def _too_early_to_probe(state: dict[str, Any]) -> bool:
|
||||
"""True while the estimated lift time (activated_at + retry_after) is future.
|
||||
|
||||
success = await self._do_probe(provider)
|
||||
tracker = self._make_tracker(provider)
|
||||
Missing or malformed timestamps fall through to allow the probe.
|
||||
"""
|
||||
activated_at_raw = state.get("activated_at")
|
||||
retry_after = state.get("retry_after")
|
||||
if not activated_at_raw or retry_after is None:
|
||||
return False
|
||||
try:
|
||||
activated_at = datetime.fromisoformat(activated_at_raw)
|
||||
except (ValueError, TypeError):
|
||||
return False
|
||||
return datetime.now(UTC) < activated_at + timedelta(seconds=retry_after)
|
||||
|
||||
if success:
|
||||
logger.info(
|
||||
"Rate-limit probe succeeded; clearing provider", provider=provider
|
||||
)
|
||||
await tracker.clear()
|
||||
# Remove from CEO-notified set so new episodes get a fresh notification
|
||||
self._rate_limit_ceo_notified.discard(provider)
|
||||
# Resolve all parked agents waiting for this rate limit to lift
|
||||
rate_limited_agents = [
|
||||
agent_id
|
||||
for agent_id, record in list(self._waiting_records.items())
|
||||
if record.waiting_for == "rate_limit_lifted"
|
||||
and record.context.get("provider") == provider
|
||||
]
|
||||
for agent_id in rate_limited_agents:
|
||||
with contextlib.suppress(Exception):
|
||||
await self.resolve_wait(
|
||||
agent_id,
|
||||
{
|
||||
"reason": "rate_limit_lifted",
|
||||
"provider": provider,
|
||||
"lifted_at": datetime.now(UTC).isoformat(),
|
||||
},
|
||||
)
|
||||
# Publish RATE_LIMIT_LIFTED event
|
||||
def _parked_agents_for(self, provider: str) -> list[str]:
|
||||
"""Agent slugs parked waiting for *provider*'s rate limit to lift."""
|
||||
return [
|
||||
agent_id
|
||||
for agent_id, record in list(self._waiting_records.items())
|
||||
if record.waiting_for == "rate_limit_lifted"
|
||||
and record.context.get("provider") == provider
|
||||
]
|
||||
|
||||
async def _on_probe_success(self, provider: str, tracker: Any) -> None:
|
||||
"""Clear the limit, resume parked agents, publish RATE_LIMIT_LIFTED."""
|
||||
logger.info("Rate-limit probe succeeded; clearing provider", provider=provider)
|
||||
await tracker.clear()
|
||||
# New episodes should get a fresh CEO notification.
|
||||
self._rate_limit_ceo_notified.discard(provider)
|
||||
resumed = self._parked_agents_for(provider)
|
||||
for agent_id in resumed:
|
||||
with contextlib.suppress(Exception):
|
||||
await self.resolve_wait(
|
||||
agent_id,
|
||||
{
|
||||
"reason": "rate_limit_lifted",
|
||||
"provider": provider,
|
||||
"lifted_at": datetime.now(UTC).isoformat(),
|
||||
},
|
||||
)
|
||||
with contextlib.suppress(Exception):
|
||||
from roboco.events import get_event_bus
|
||||
from roboco.models.events import Event, EventType
|
||||
|
||||
with contextlib.suppress(Exception):
|
||||
bus = get_event_bus()
|
||||
event = Event(
|
||||
await get_event_bus().publish(
|
||||
Event(
|
||||
type=EventType.RATE_LIMIT_LIFTED,
|
||||
data={
|
||||
"provider": provider,
|
||||
"resumedAgents": rate_limited_agents,
|
||||
"resumedAgents": resumed,
|
||||
"timestamp": datetime.now(UTC).isoformat(),
|
||||
},
|
||||
)
|
||||
await bus.publish(event)
|
||||
logger.info(
|
||||
"RATE_LIMIT_LIFTED published",
|
||||
provider=provider,
|
||||
resumed_agents=len(rate_limited_agents),
|
||||
)
|
||||
logger.info(
|
||||
"RATE_LIMIT_LIFTED published",
|
||||
provider=provider,
|
||||
resumed_agents=len(resumed),
|
||||
)
|
||||
|
||||
async def _on_probe_failure(
|
||||
self, provider: str, tracker: Any, activated_at_raw: str | None
|
||||
) -> None:
|
||||
"""Count a failed probe; notify the CEO once at the failure threshold."""
|
||||
failure_count = await tracker.increment_probe_failures()
|
||||
logger.debug(
|
||||
"Rate-limit probe failed", provider=provider, probe_failures=failure_count
|
||||
)
|
||||
if (
|
||||
failure_count >= _CEO_NOTIFY_THRESHOLD
|
||||
and provider not in self._rate_limit_ceo_notified
|
||||
):
|
||||
self._rate_limit_ceo_notified.add(provider)
|
||||
await self._notify_rate_limit_ceo(
|
||||
provider=provider,
|
||||
activated_at_str=activated_at_raw or "unknown",
|
||||
paused_agent_count=len(self._parked_agents_for(provider)),
|
||||
)
|
||||
|
||||
async def _probe_one_provider(self, provider: str, state: dict[str, Any]) -> None:
|
||||
"""Probe a single rate-limited provider and handle the outcome."""
|
||||
if self._too_early_to_probe(state):
|
||||
return # Wait until after the estimated lift time.
|
||||
tracker = self._make_tracker(provider)
|
||||
if await self._do_probe(provider):
|
||||
await self._on_probe_success(provider, tracker)
|
||||
else:
|
||||
failure_count = await tracker.increment_probe_failures()
|
||||
logger.debug(
|
||||
"Rate-limit probe failed",
|
||||
provider=provider,
|
||||
probe_failures=failure_count,
|
||||
)
|
||||
# Send CEO notification once when failures reach threshold 10
|
||||
_CEO_NOTIFY_THRESHOLD = 10
|
||||
if (
|
||||
failure_count >= _CEO_NOTIFY_THRESHOLD
|
||||
and provider not in self._rate_limit_ceo_notified
|
||||
):
|
||||
self._rate_limit_ceo_notified.add(provider)
|
||||
paused_count = sum(
|
||||
1
|
||||
for record in self._waiting_records.values()
|
||||
if record.waiting_for == "rate_limit_lifted"
|
||||
and record.context.get("provider") == provider
|
||||
)
|
||||
activated_at_str: str = activated_at_raw or "unknown"
|
||||
await self._notify_rate_limit_ceo(
|
||||
provider=provider,
|
||||
activated_at_str=activated_at_str,
|
||||
paused_agent_count=paused_count,
|
||||
)
|
||||
await self._on_probe_failure(provider, tracker, state.get("activated_at"))
|
||||
|
||||
async def _do_probe(self, _provider: str) -> bool:
|
||||
"""Return True if the provider is accepting requests again.
|
||||
@staticmethod
|
||||
def _probe_target(provider: str) -> tuple[str | None, dict[str, str]]:
|
||||
"""Resolve the (url, headers) for a free liveness probe of ``provider``.
|
||||
|
||||
This method is intentionally thin so tests can monkeypatch it.
|
||||
The default implementation is conservative: returns ``True``
|
||||
(success) so that once the estimated_lift_at window has passed
|
||||
the probe clears the rate limit. Override in tests to inject
|
||||
either success or failure scenarios.
|
||||
Returns ``(None, {})`` when the provider can't be probed — an unknown
|
||||
provider, or Anthropic with no API key configured. The caller then
|
||||
falls back to time-expiry optimism rather than parking forever.
|
||||
"""
|
||||
# Default: optimistic — time-expiry gate (checked before this call)
|
||||
# is the primary guard; the probe itself succeeds.
|
||||
return True
|
||||
p = provider.lower()
|
||||
if p == "anthropic":
|
||||
key = settings.anthropic_api_key
|
||||
if not key:
|
||||
return None, {}
|
||||
return (
|
||||
f"{_ANTHROPIC_PROBE_BASE}/v1/models",
|
||||
{"x-api-key": key, "anthropic-version": "2023-06-01"},
|
||||
)
|
||||
if p.startswith("ollama"):
|
||||
return f"{settings.ollama_base_url.rstrip('/')}/api/tags", {}
|
||||
return None, {}
|
||||
|
||||
async def _do_probe(self, provider: str) -> bool:
|
||||
"""Return True if ``provider`` is accepting requests again (not 429).
|
||||
|
||||
Makes a free, unmetered liveness call — Anthropic ``GET /v1/models``
|
||||
or Ollama ``GET /api/tags`` — and treats any non-429 response as the
|
||||
rate limit having lifted. A 429 keeps the provider parked; a network
|
||||
error stays parked too (retry next sweep). When the provider can't be
|
||||
probed (no key / unknown), fall back to time-expiry optimism: the
|
||||
caller only reaches this after ``estimated_lift_at`` has passed.
|
||||
|
||||
Injectable boundary — tests monkeypatch this to force outcomes.
|
||||
"""
|
||||
url, headers = self._probe_target(provider)
|
||||
if url is None:
|
||||
return True # cannot probe — trust the elapsed retry_after window
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=_PROBE_TIMEOUT_SECONDS) as client:
|
||||
resp = await client.get(url, headers=headers)
|
||||
except httpx.HTTPError as exc:
|
||||
logger.debug(
|
||||
"Rate-limit probe request failed", provider=provider, error=str(exc)
|
||||
)
|
||||
return False # unreachable — stay parked, retry on the next sweep
|
||||
return resp.status_code != _HTTP_TOO_MANY_REQUESTS
|
||||
|
||||
async def _notify_rate_limit_ceo(
|
||||
self,
|
||||
@@ -4146,7 +4191,7 @@ Start by:
|
||||
) -> None:
|
||||
"""Send a high-priority notification to the CEO about a persistent rate limit.
|
||||
|
||||
Fires once per episode (AC8). Follows the same pattern as
|
||||
Fires once per rate-limit episode. Follows the same pattern as
|
||||
``_notify_stranded_agent`` — direct DB insert + delivery.deliver().
|
||||
"""
|
||||
try:
|
||||
|
||||
@@ -145,12 +145,10 @@ class ExtractionService:
|
||||
await store_message(message)
|
||||
"""
|
||||
|
||||
def __init__(self, config: ExtractionConfig | None = None) -> None:
|
||||
self.config = config or ExtractionConfig()
|
||||
self.log = logger.bind(component="extraction")
|
||||
|
||||
# Compile patterns
|
||||
self._compiled_patterns: dict[MessageType, list[re.Pattern]] = {
|
||||
@staticmethod
|
||||
def _compile_patterns() -> dict[MessageType, list[re.Pattern]]:
|
||||
"""Pre-compile the per-message-type regex pattern lists."""
|
||||
return {
|
||||
MessageType.REASONING: [re.compile(p) for p in REASONING_PATTERNS],
|
||||
MessageType.DIALOGUE: [re.compile(p) for p in DIALOGUE_PATTERNS],
|
||||
MessageType.DECISION: [re.compile(p) for p in DECISION_PATTERNS],
|
||||
@@ -159,7 +157,10 @@ class ExtractionService:
|
||||
MessageType.TECHNICAL: [re.compile(p) for p in TECHNICAL_PATTERNS],
|
||||
}
|
||||
|
||||
# Mention pattern
|
||||
def __init__(self, config: ExtractionConfig | None = None) -> None:
|
||||
self.config = config or ExtractionConfig()
|
||||
self.log = logger.bind(component="extraction")
|
||||
self._compiled_patterns = self._compile_patterns()
|
||||
self._mention_pattern = re.compile(r"@(\w+)")
|
||||
|
||||
async def extract(self, ctx: ExtractionContext) -> ExtractionResult:
|
||||
|
||||
@@ -44,13 +44,13 @@ class RateLimitStateTracker:
|
||||
"""
|
||||
self._provider = provider
|
||||
self._redis_url = redis_url or settings.redis_url
|
||||
self._redis: redis.Redis | None = None # type: ignore[type-arg]
|
||||
self._redis: redis.Redis | None = None
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Private helpers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _conn(self) -> redis.Redis: # type: ignore[type-arg]
|
||||
async def _conn(self) -> redis.Redis:
|
||||
"""Return a (lazy-connected) redis.asyncio.Redis client."""
|
||||
if self._redis is None:
|
||||
self._redis = redis.from_url(self._redis_url)
|
||||
@@ -149,37 +149,46 @@ class RateLimitStateTracker:
|
||||
redis_url: Override the default Redis URL from settings.
|
||||
"""
|
||||
url = redis_url or settings.redis_url
|
||||
r: redis.Redis = redis.from_url(url) # type: ignore[type-arg]
|
||||
pattern = f"{cls._KEY_PREFIX}*:state"
|
||||
results: list[tuple[str, dict[str, Any]]] = []
|
||||
try:
|
||||
cursor: int = 0
|
||||
while True:
|
||||
cursor, keys = await r.scan(cursor, match=pattern, count=100)
|
||||
for raw_key in keys:
|
||||
key: str = (
|
||||
raw_key.decode() if isinstance(raw_key, bytes) else str(raw_key)
|
||||
)
|
||||
# Extract provider from key: roboco:rate_limit:{provider}:state
|
||||
# Strip prefix and suffix
|
||||
inner = key[len(cls._KEY_PREFIX) :]
|
||||
if inner.endswith(":state"):
|
||||
provider = inner[: -len(":state")]
|
||||
else:
|
||||
continue
|
||||
raw_val = await r.get(key)
|
||||
if raw_val is None:
|
||||
continue
|
||||
decoded: str = (
|
||||
raw_val.decode() if isinstance(raw_val, bytes) else str(raw_val)
|
||||
)
|
||||
state: dict[str, Any] = json.loads(decoded)
|
||||
if state.get("rate_limited"):
|
||||
results.append((provider, state))
|
||||
if cursor == 0:
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
finally:
|
||||
await r.aclose()
|
||||
# `async with` closes the client on exit (modern redis.asyncio API),
|
||||
# avoiding a deprecated explicit close in a finally block.
|
||||
async with redis.from_url(url) as r:
|
||||
try:
|
||||
cursor: int = 0
|
||||
while True:
|
||||
cursor, keys = await r.scan(cursor, match=pattern, count=100)
|
||||
for raw_key in keys:
|
||||
entry = await cls._read_rate_limited_entry(r, raw_key)
|
||||
if entry is not None:
|
||||
results.append(entry)
|
||||
if cursor == 0:
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
return results
|
||||
|
||||
@staticmethod
|
||||
def _decode(value: Any) -> str:
|
||||
"""Decode a Redis value (bytes or str) to str."""
|
||||
return value.decode() if isinstance(value, bytes) else str(value)
|
||||
|
||||
@classmethod
|
||||
async def _read_rate_limited_entry(
|
||||
cls, r: Any, raw_key: Any
|
||||
) -> tuple[str, dict[str, Any]] | None:
|
||||
"""``(provider, state)`` for a scan key, iff it holds a rate-limited record.
|
||||
|
||||
Returns None for keys that don't match the ``...:{provider}:state`` shape,
|
||||
have no stored value, or whose state is not currently rate-limited.
|
||||
"""
|
||||
key = cls._decode(raw_key)
|
||||
inner = key[len(cls._KEY_PREFIX) :]
|
||||
if not inner.endswith(":state"):
|
||||
return None
|
||||
provider = inner[: -len(":state")]
|
||||
raw_val = await r.get(key)
|
||||
if raw_val is None:
|
||||
return None
|
||||
state: dict[str, Any] = json.loads(cls._decode(raw_val))
|
||||
return (provider, state) if state.get("rate_limited") else None
|
||||
|
||||
@@ -65,7 +65,23 @@ _A2A_CODE_REVIEW_RELEVANT_STATES: frozenset[str] = frozenset(
|
||||
)
|
||||
|
||||
|
||||
def decide_spawn( # noqa: PLR0911
|
||||
def _stale_trigger_decision(task: Any, trigger: TriggerContext) -> Decision | None:
|
||||
"""DROP decision for a trigger that no longer applies to the task, else None."""
|
||||
if task.status in _TERMINAL_STATUSES:
|
||||
return Decision(SpawnDecision.DROP, "task in terminal state — trigger stale")
|
||||
if (
|
||||
trigger.kind is TriggerKind.A2A
|
||||
and trigger.skill == "code_review"
|
||||
and task.status not in _A2A_CODE_REVIEW_RELEVANT_STATES
|
||||
):
|
||||
return Decision(
|
||||
SpawnDecision.DROP,
|
||||
f"a2a code_review for task in {task.status} — stale",
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def decide_spawn(
|
||||
*,
|
||||
task: Any,
|
||||
trigger: TriggerContext,
|
||||
@@ -76,18 +92,9 @@ def decide_spawn( # noqa: PLR0911
|
||||
stale > provider-rate-limit > claimant-lock > task-cooldown > role-rate
|
||||
"""
|
||||
# 1. Stale-trigger cleanup
|
||||
if task.status in _TERMINAL_STATUSES:
|
||||
return Decision(SpawnDecision.DROP, "task in terminal state — trigger stale")
|
||||
|
||||
if (
|
||||
trigger.kind is TriggerKind.A2A
|
||||
and trigger.skill == "code_review"
|
||||
and task.status not in _A2A_CODE_REVIEW_RELEVANT_STATES
|
||||
):
|
||||
return Decision(
|
||||
SpawnDecision.DROP,
|
||||
f"a2a code_review for task in {task.status} — stale",
|
||||
)
|
||||
stale = _stale_trigger_decision(task, trigger)
|
||||
if stale is not None:
|
||||
return stale
|
||||
|
||||
# 2. Provider rate-limit gate
|
||||
if trigger.provider_rate_limited:
|
||||
|
||||
@@ -988,6 +988,20 @@ class BaseIndexPlugin(ABC):
|
||||
return "", search_results
|
||||
|
||||
# ---- LLM call phase with 429 retry (outside index timeout) -----------
|
||||
return await self._ask_llm(prompt, search_results)
|
||||
|
||||
async def _ask_llm(
|
||||
self, prompt: str, search_results: list[SearchResult]
|
||||
) -> tuple[str, list[SearchResult]]:
|
||||
"""Run the 429-retried LLM synthesis for :meth:`ask`.
|
||||
|
||||
Returns ``(answer, citations)``; ``("", citations)`` on any non-429
|
||||
failure. Raises RateLimitError only when every 429 retry is exhausted.
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
import httpx
|
||||
|
||||
llm_url = f"{self.config.llm_base_url}/chat/completions"
|
||||
logger.info(
|
||||
"ask() calling LLM",
|
||||
@@ -1019,10 +1033,11 @@ class BaseIndexPlugin(ABC):
|
||||
return "", search_results
|
||||
|
||||
if resp.status_code == HTTP_TOO_MANY_REQUESTS:
|
||||
retry_after = parse_retry_after_header(resp)
|
||||
last_rl_retry_after = retry_after
|
||||
last_rl_retry_after = parse_retry_after_header(resp)
|
||||
backoff = (
|
||||
retry_after if retry_after is not None else float(2**rl_attempt)
|
||||
last_rl_retry_after
|
||||
if last_rl_retry_after is not None
|
||||
else float(2**rl_attempt)
|
||||
)
|
||||
logger.warning(
|
||||
"Ollama rate limited (429), retrying",
|
||||
@@ -1037,17 +1052,17 @@ class BaseIndexPlugin(ABC):
|
||||
|
||||
if resp.is_success:
|
||||
data = resp.json()
|
||||
answer_text = data["choices"][0]["message"]["content"]
|
||||
answer_text = self._extract_from_think_tags(answer_text)
|
||||
return answer_text, search_results
|
||||
else:
|
||||
logger.warning(
|
||||
"LLM call failed in ask",
|
||||
index_type=self.index_type.value,
|
||||
status=resp.status_code,
|
||||
error=resp.text[:200] if resp.text else "no error text",
|
||||
answer_text = self._extract_from_think_tags(
|
||||
data["choices"][0]["message"]["content"]
|
||||
)
|
||||
return "", search_results
|
||||
return answer_text, search_results
|
||||
logger.warning(
|
||||
"LLM call failed in ask",
|
||||
index_type=self.index_type.value,
|
||||
status=resp.status_code,
|
||||
error=resp.text[:200] if resp.text else "no error text",
|
||||
)
|
||||
return "", search_results
|
||||
|
||||
raise RateLimitError(provider="ollama", retry_after=last_rl_retry_after)
|
||||
|
||||
|
||||
@@ -681,6 +681,44 @@ class MentorService:
|
||||
parts.append(f"\nQuestion: {question}")
|
||||
return "\n".join(parts)
|
||||
|
||||
@staticmethod
|
||||
def _select_system_prompt(agent_profile: AgentProfile | None) -> str:
|
||||
"""Role-specific synthesis prompt when available, else the generic one."""
|
||||
if agent_profile and agent_profile.role in ROLE_PROMPTS:
|
||||
return ROLE_PROMPTS[agent_profile.role]
|
||||
return (
|
||||
"Answer the question based on the knowledge base context provided below. "
|
||||
"Synthesize a clear, thorough answer. Do NOT just copy text - explain in "
|
||||
"your own words. If the context doesn't fully answer the question, say "
|
||||
"what you can based on what's available."
|
||||
)
|
||||
|
||||
def _answer_from_response(
|
||||
self,
|
||||
response: httpx.Response,
|
||||
sources: list[SearchResult],
|
||||
agent_profile: AgentProfile | None,
|
||||
) -> str:
|
||||
"""Parse a successful LLM response into an answer, else fall back."""
|
||||
if not response.is_success:
|
||||
logger.warning(
|
||||
"LLM call failed in mentor",
|
||||
status=response.status_code,
|
||||
error=response.text[:200],
|
||||
)
|
||||
return self._fallback_answer(sources, agent_profile)
|
||||
data = response.json()
|
||||
raw_answer: str = data["choices"][0]["message"]["content"]
|
||||
answer = self._extract_answer(raw_answer)
|
||||
if not answer:
|
||||
logger.warning(
|
||||
"LLM response empty after extraction",
|
||||
model=settings.local_llm_model,
|
||||
original_length=len(raw_answer),
|
||||
)
|
||||
return self._fallback_answer(sources, agent_profile)
|
||||
return answer
|
||||
|
||||
async def _synthesize_answer(
|
||||
self,
|
||||
question: str,
|
||||
@@ -703,17 +741,7 @@ class MentorService:
|
||||
"Try rephrasing your question or asking about a different topic."
|
||||
)
|
||||
|
||||
# Get role-specific system prompt
|
||||
base_prompt = (
|
||||
"Answer the question based on the knowledge base context provided below. "
|
||||
"Synthesize a clear, thorough answer. Do NOT just copy text - explain in "
|
||||
"your own words. If the context doesn't fully answer the question, say "
|
||||
"what you can based on what's available."
|
||||
)
|
||||
if agent_profile and agent_profile.role in ROLE_PROMPTS:
|
||||
system_prompt = ROLE_PROMPTS[agent_profile.role]
|
||||
else:
|
||||
system_prompt = base_prompt
|
||||
system_prompt = self._select_system_prompt(agent_profile)
|
||||
|
||||
# Build user prompt
|
||||
user_prompt = self._build_user_prompt(
|
||||
@@ -764,27 +792,7 @@ class MentorService:
|
||||
await asyncio.sleep(backoff)
|
||||
continue
|
||||
|
||||
if response.is_success:
|
||||
data = response.json()
|
||||
raw_answer: str = data["choices"][0]["message"]["content"]
|
||||
answer = self._extract_answer(raw_answer)
|
||||
|
||||
if not answer:
|
||||
logger.warning(
|
||||
"LLM response empty after extraction",
|
||||
model=settings.local_llm_model,
|
||||
original_length=len(raw_answer),
|
||||
)
|
||||
return self._fallback_answer(sources, agent_profile)
|
||||
|
||||
return answer
|
||||
else:
|
||||
logger.warning(
|
||||
"LLM call failed in mentor",
|
||||
status=response.status_code,
|
||||
error=response.text[:200],
|
||||
)
|
||||
return self._fallback_answer(sources, agent_profile)
|
||||
return self._answer_from_response(response, sources, agent_profile)
|
||||
|
||||
raise RateLimitError(provider="ollama", retry_after=last_rl_retry_after)
|
||||
|
||||
|
||||
@@ -307,6 +307,58 @@ class OllamaEmbedder:
|
||||
|
||||
return embeddings
|
||||
|
||||
@staticmethod
|
||||
def _rl_backoff(retry_after: float | None, rl_attempt: int) -> float:
|
||||
"""Backoff seconds for a 429: honor Retry-After, else exponential."""
|
||||
return retry_after if retry_after is not None else float(2**rl_attempt)
|
||||
|
||||
@staticmethod
|
||||
def _map_embed_error(e: Exception, base_url: str) -> Exception:
|
||||
"""Map a raw request exception to the appropriate Ollama embedder error."""
|
||||
if isinstance(e, httpx.ConnectError):
|
||||
return OllamaConnectionError(f"Cannot connect to Ollama at {base_url}: {e}")
|
||||
if isinstance(e, httpx.TimeoutException):
|
||||
return OllamaConnectionError(f"Ollama request timed out: {e}")
|
||||
return OllamaEmbedderError(f"Unexpected error: {e}")
|
||||
|
||||
@staticmethod
|
||||
def _log_429(rl_attempt: int, backoff: float) -> None:
|
||||
"""Log an Ollama 429 rate-limit retry."""
|
||||
logger.warning(
|
||||
"Ollama rate limited (429), retrying",
|
||||
provider="ollama",
|
||||
attempt=rl_attempt + 1,
|
||||
max_retries=RATE_LIMIT_MAX_RETRIES,
|
||||
backoff_duration=backoff,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _sleep_connect_retry(
|
||||
attempt: int, last_error: Exception | None, label: str, **extra: Any
|
||||
) -> None:
|
||||
"""Exponential backoff between ConnectError/Timeout retries.
|
||||
|
||||
No sleep on the final attempt — the caller raises ``last_error`` then.
|
||||
"""
|
||||
if attempt < MAX_RETRIES - 1:
|
||||
delay = RETRY_DELAY_BASE * (2**attempt)
|
||||
logger.warning(
|
||||
label, attempt=attempt + 1, delay=delay, error=str(last_error), **extra
|
||||
)
|
||||
time.sleep(delay)
|
||||
|
||||
@staticmethod
|
||||
async def _asleep_connect_retry(
|
||||
attempt: int, last_error: Exception | None, label: str, **extra: Any
|
||||
) -> None:
|
||||
"""Async counterpart of :meth:`_sleep_connect_retry`."""
|
||||
if attempt < MAX_RETRIES - 1:
|
||||
delay = RETRY_DELAY_BASE * (2**attempt)
|
||||
logger.warning(
|
||||
label, attempt=attempt + 1, delay=delay, error=str(last_error), **extra
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
def embed_query(
|
||||
self,
|
||||
query: str,
|
||||
@@ -344,19 +396,10 @@ class OllamaEmbedder:
|
||||
)
|
||||
# 429 check — must NOT enter the ConnectError path
|
||||
if response.status_code == HTTP_TOO_MANY_REQUESTS:
|
||||
retry_after = parse_retry_after_header(response)
|
||||
last_rl_retry_after = retry_after
|
||||
backoff = (
|
||||
retry_after
|
||||
if retry_after is not None
|
||||
else float(2**rl_attempt)
|
||||
)
|
||||
logger.warning(
|
||||
"Ollama rate limited (429), retrying",
|
||||
provider="ollama",
|
||||
attempt=rl_attempt + 1,
|
||||
max_retries=RATE_LIMIT_MAX_RETRIES,
|
||||
backoff_duration=backoff,
|
||||
last_rl_retry_after = parse_retry_after_header(response)
|
||||
self._log_429(
|
||||
rl_attempt,
|
||||
self._rl_backoff(last_rl_retry_after, rl_attempt),
|
||||
)
|
||||
got_429 = True
|
||||
break # break inner loop; outer loop will sleep + retry
|
||||
@@ -366,26 +409,14 @@ class OllamaEmbedder:
|
||||
self._cache.put(query, result)
|
||||
return result
|
||||
|
||||
except httpx.ConnectError as e:
|
||||
last_error = OllamaConnectionError(
|
||||
f"Cannot connect to Ollama at {self.base_url}: {e}"
|
||||
)
|
||||
except httpx.TimeoutException as e:
|
||||
last_error = OllamaConnectionError(f"Ollama request timed out: {e}")
|
||||
except (OllamaModelError, OllamaEmbedderError):
|
||||
raise
|
||||
except Exception as e:
|
||||
last_error = OllamaEmbedderError(f"Unexpected error: {e}")
|
||||
last_error = self._map_embed_error(e, self.base_url)
|
||||
|
||||
if attempt < MAX_RETRIES - 1:
|
||||
delay = RETRY_DELAY_BASE * (2**attempt)
|
||||
logger.warning(
|
||||
"Ollama embed_query retry",
|
||||
attempt=attempt + 1,
|
||||
delay=delay,
|
||||
error=str(last_error),
|
||||
)
|
||||
time.sleep(delay)
|
||||
self._sleep_connect_retry(
|
||||
attempt, last_error, "Ollama embed_query retry"
|
||||
)
|
||||
# --- end inner loop ---
|
||||
|
||||
if not got_429:
|
||||
@@ -393,13 +424,8 @@ class OllamaEmbedder:
|
||||
raise last_error or OllamaEmbedderError("Max retries exceeded")
|
||||
|
||||
# 429: sleep and try again (outer loop)
|
||||
backoff = (
|
||||
last_rl_retry_after
|
||||
if last_rl_retry_after is not None
|
||||
else float(2**rl_attempt)
|
||||
)
|
||||
if rl_attempt < RATE_LIMIT_MAX_RETRIES - 1:
|
||||
time.sleep(backoff)
|
||||
time.sleep(self._rl_backoff(last_rl_retry_after, rl_attempt))
|
||||
|
||||
raise RateLimitError(provider="ollama", retry_after=last_rl_retry_after)
|
||||
|
||||
@@ -424,57 +450,33 @@ class OllamaEmbedder:
|
||||
json={"model": self.model, "input": batch},
|
||||
)
|
||||
if response.status_code == HTTP_TOO_MANY_REQUESTS:
|
||||
retry_after = parse_retry_after_header(response)
|
||||
last_rl_retry_after = retry_after
|
||||
backoff = (
|
||||
retry_after
|
||||
if retry_after is not None
|
||||
else float(2**rl_attempt)
|
||||
)
|
||||
logger.warning(
|
||||
"Ollama rate limited (429), retrying",
|
||||
provider="ollama",
|
||||
attempt=rl_attempt + 1,
|
||||
max_retries=RATE_LIMIT_MAX_RETRIES,
|
||||
backoff_duration=backoff,
|
||||
last_rl_retry_after = parse_retry_after_header(response)
|
||||
self._log_429(
|
||||
rl_attempt,
|
||||
self._rl_backoff(last_rl_retry_after, rl_attempt),
|
||||
)
|
||||
got_429 = True
|
||||
break
|
||||
|
||||
return self._handle_embed_response(response, input_count=len(batch))
|
||||
|
||||
except httpx.ConnectError as e:
|
||||
last_error = OllamaConnectionError(
|
||||
f"Cannot connect to Ollama at {self.base_url}: {e}"
|
||||
)
|
||||
except httpx.TimeoutException as e:
|
||||
last_error = OllamaConnectionError(f"Ollama request timed out: {e}")
|
||||
except (OllamaModelError, OllamaEmbedderError):
|
||||
raise
|
||||
except Exception as e:
|
||||
last_error = OllamaEmbedderError(f"Unexpected error: {e}")
|
||||
last_error = self._map_embed_error(e, self.base_url)
|
||||
|
||||
if attempt < MAX_RETRIES - 1:
|
||||
delay = RETRY_DELAY_BASE * (2**attempt)
|
||||
logger.warning(
|
||||
"Ollama embed_documents retry",
|
||||
attempt=attempt + 1,
|
||||
batch_index=batch_index,
|
||||
delay=delay,
|
||||
error=str(last_error),
|
||||
)
|
||||
time.sleep(delay)
|
||||
self._sleep_connect_retry(
|
||||
attempt,
|
||||
last_error,
|
||||
"Ollama embed_documents retry",
|
||||
batch_index=batch_index,
|
||||
)
|
||||
|
||||
if not got_429:
|
||||
raise last_error or OllamaEmbedderError("Max retries exceeded")
|
||||
|
||||
backoff = (
|
||||
last_rl_retry_after
|
||||
if last_rl_retry_after is not None
|
||||
else float(2**rl_attempt)
|
||||
)
|
||||
if rl_attempt < RATE_LIMIT_MAX_RETRIES - 1:
|
||||
time.sleep(backoff)
|
||||
time.sleep(self._rl_backoff(last_rl_retry_after, rl_attempt))
|
||||
|
||||
raise RateLimitError(provider="ollama", retry_after=last_rl_retry_after)
|
||||
|
||||
@@ -582,19 +584,10 @@ class OllamaEmbedder:
|
||||
json={"model": self.model, "input": batch},
|
||||
)
|
||||
if response.status_code == HTTP_TOO_MANY_REQUESTS:
|
||||
retry_after = parse_retry_after_header(response)
|
||||
last_rl_retry_after = retry_after
|
||||
backoff = (
|
||||
retry_after
|
||||
if retry_after is not None
|
||||
else float(2**rl_attempt)
|
||||
)
|
||||
logger.warning(
|
||||
"Ollama rate limited (429), retrying",
|
||||
provider="ollama",
|
||||
attempt=rl_attempt + 1,
|
||||
max_retries=RATE_LIMIT_MAX_RETRIES,
|
||||
backoff_duration=backoff,
|
||||
last_rl_retry_after = parse_retry_after_header(response)
|
||||
self._log_429(
|
||||
rl_attempt,
|
||||
self._rl_backoff(last_rl_retry_after, rl_attempt),
|
||||
)
|
||||
got_429 = True
|
||||
break
|
||||
@@ -603,40 +596,25 @@ class OllamaEmbedder:
|
||||
response, input_count=len(batch)
|
||||
)
|
||||
|
||||
except httpx.ConnectError as e:
|
||||
last_error = OllamaConnectionError(
|
||||
f"Cannot connect to Ollama at {self.base_url}: {e}"
|
||||
)
|
||||
except httpx.TimeoutException as e:
|
||||
last_error = OllamaConnectionError(
|
||||
f"Ollama request timed out: {e}"
|
||||
)
|
||||
except (OllamaModelError, OllamaEmbedderError):
|
||||
raise
|
||||
except Exception as e:
|
||||
last_error = OllamaEmbedderError(f"Unexpected error: {e}")
|
||||
last_error = self._map_embed_error(e, self.base_url)
|
||||
|
||||
if attempt < MAX_RETRIES - 1:
|
||||
delay = RETRY_DELAY_BASE * (2**attempt)
|
||||
logger.warning(
|
||||
"Parallel embed batch retry",
|
||||
batch_index=batch_index,
|
||||
attempt=attempt + 1,
|
||||
delay=delay,
|
||||
error=str(last_error),
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
await self._asleep_connect_retry(
|
||||
attempt,
|
||||
last_error,
|
||||
"Parallel embed batch retry",
|
||||
batch_index=batch_index,
|
||||
)
|
||||
|
||||
if not got_429:
|
||||
raise last_error or OllamaEmbedderError("Max retries exceeded")
|
||||
|
||||
backoff = (
|
||||
last_rl_retry_after
|
||||
if last_rl_retry_after is not None
|
||||
else float(2**rl_attempt)
|
||||
)
|
||||
if rl_attempt < RATE_LIMIT_MAX_RETRIES - 1:
|
||||
await asyncio.sleep(backoff)
|
||||
await asyncio.sleep(
|
||||
self._rl_backoff(last_rl_retry_after, rl_attempt)
|
||||
)
|
||||
|
||||
raise RateLimitError(provider="ollama", retry_after=last_rl_retry_after)
|
||||
|
||||
@@ -807,19 +785,10 @@ class OllamaEmbedder:
|
||||
json={"model": self.model, "input": query},
|
||||
)
|
||||
if response.status_code == HTTP_TOO_MANY_REQUESTS:
|
||||
retry_after = parse_retry_after_header(response)
|
||||
last_rl_retry_after = retry_after
|
||||
backoff = (
|
||||
retry_after
|
||||
if retry_after is not None
|
||||
else float(2**rl_attempt)
|
||||
)
|
||||
logger.warning(
|
||||
"Ollama rate limited (429), retrying",
|
||||
provider="ollama",
|
||||
attempt=rl_attempt + 1,
|
||||
max_retries=RATE_LIMIT_MAX_RETRIES,
|
||||
backoff_duration=backoff,
|
||||
last_rl_retry_after = parse_retry_after_header(response)
|
||||
self._log_429(
|
||||
rl_attempt,
|
||||
self._rl_backoff(last_rl_retry_after, rl_attempt),
|
||||
)
|
||||
got_429 = True
|
||||
break
|
||||
@@ -831,42 +800,21 @@ class OllamaEmbedder:
|
||||
self._cache.put(query, result)
|
||||
return result
|
||||
|
||||
except httpx.ConnectError as e:
|
||||
last_error = OllamaConnectionError(
|
||||
f"Cannot connect to Ollama at {self.base_url}: {e}"
|
||||
)
|
||||
except httpx.TimeoutException as e:
|
||||
last_error = OllamaConnectionError(
|
||||
f"Ollama request timed out: {e}"
|
||||
)
|
||||
except (OllamaModelError, OllamaEmbedderError):
|
||||
raise
|
||||
except Exception as e:
|
||||
last_error = OllamaEmbedderError(f"Unexpected error: {e}")
|
||||
last_error = self._map_embed_error(e, self.base_url)
|
||||
|
||||
if got_429:
|
||||
break # exit inner loop cleanly
|
||||
|
||||
if attempt < MAX_RETRIES - 1:
|
||||
delay = RETRY_DELAY_BASE * (2**attempt)
|
||||
logger.warning(
|
||||
"Ollama aembed_query retry",
|
||||
attempt=attempt + 1,
|
||||
delay=delay,
|
||||
error=str(last_error),
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
# A 429 already broke the inner loop above; otherwise back off.
|
||||
await self._asleep_connect_retry(
|
||||
attempt, last_error, "Ollama aembed_query retry"
|
||||
)
|
||||
|
||||
if not got_429:
|
||||
raise last_error or OllamaEmbedderError("Max retries exceeded")
|
||||
|
||||
backoff = (
|
||||
last_rl_retry_after
|
||||
if last_rl_retry_after is not None
|
||||
else float(2**rl_attempt)
|
||||
)
|
||||
if rl_attempt < RATE_LIMIT_MAX_RETRIES - 1:
|
||||
await asyncio.sleep(backoff)
|
||||
await asyncio.sleep(self._rl_backoff(last_rl_retry_after, rl_attempt))
|
||||
|
||||
raise RateLimitError(provider="ollama", retry_after=last_rl_retry_after)
|
||||
|
||||
|
||||
@@ -270,8 +270,13 @@ def _detect_dep_commands(workspace: Path) -> list[tuple[str, list[str]]]:
|
||||
Detects project ecosystems by lockfile/manifest and returns
|
||||
``(label, argv)`` tuples to run from the workspace root:
|
||||
|
||||
- Python: `pyproject.toml` → ``uv sync`` (installs dev deps into a
|
||||
`.venv` next to the project, giving the agent its own ruff/mypy/pytest).
|
||||
- Python: `pyproject.toml` → ``uv sync --extra dev`` (installs the project
|
||||
plus its ``dev`` extra into a `.venv` next to the project, giving the
|
||||
agent its own ruff/mypy/xenon/pytest for the `make quality` gate). Plain
|
||||
``uv sync`` would install only the default dependency group, leaving the
|
||||
lint/type/complexity tools — which live in the ``dev`` *extra* — absent,
|
||||
so `make quality` dies on ``ruff: command not found`` and the agent
|
||||
cannot gate its own work.
|
||||
- Node/TS: `pnpm-lock.yaml` → ``pnpm install``;
|
||||
``package-lock.json`` → ``npm ci``; bare `package.json` → ``npm install``.
|
||||
|
||||
@@ -280,7 +285,7 @@ def _detect_dep_commands(workspace: Path) -> list[tuple[str, list[str]]]:
|
||||
commands: list[tuple[str, list[str]]] = []
|
||||
|
||||
if (workspace / "pyproject.toml").is_file():
|
||||
commands.append(("uv sync", ["uv", "sync"]))
|
||||
commands.append(("uv sync --extra dev", ["uv", "sync", "--extra", "dev"]))
|
||||
|
||||
if (workspace / "pnpm-lock.yaml").is_file():
|
||||
commands.append(("pnpm install", ["pnpm", "install", "--frozen-lockfile"]))
|
||||
|
||||
Reference in New Issue
Block a user