mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
* fix(rate-limit): real provider liveness probe instead of time-based stub
The rate-limit recovery sweeper cleared a provider and resumed parked agents
purely on elapsed time — _do_probe was a stub that always returned True once
the retry_after window passed, so it never confirmed the provider had actually
stopped rate-limiting us. Under a sustained limit that resumes agents straight
into another 429, re-parking them: avoidable churn.
Make the probe real. _do_probe now issues a free, unmetered liveness call —
Anthropic GET /v1/models or Ollama GET /api/tags — and treats any non-429
response as the limit having lifted. A 429 keeps the provider parked; a
network error keeps it parked too (retry next sweep). When the provider can't
be probed (no API key, or an unrecognized provider), it falls back to the
prior time-expiry optimism rather than stranding agents. _probe_target keeps
URL/header resolution separate and testable, and _do_probe stays a
monkeypatchable boundary so the existing sweep tests are unaffected.
Also drop two acceptance-criteria-number labels from comments in this file.
* chore(rate-limit): clear merged gate debt in rate-limit tests + deps lint
The rate-limit PR landed with ruff violations the full gate flags but the
authors' runs missed: test_rate_limit_sweep.py was unformatted, and
test_rate_limit_tracker.py had unsorted/unused imports and magic-value
comparisons. Format the sweep test, drop the dead imports, and bind the
magic comparison values to locals. Also strip acceptance-criteria-number
labels from comments/docstrings across the three rate-limit test files
(leaving genuine acceptance_criteria=[...] test data untouched), and add
api/deps.py to the PLC0415 per-file-ignore — it is the DI wiring hub and
defers a couple of service imports to call time to avoid import cycles,
the same rationale already applied to api/routes, runtime, and services.
* fix(rate-limit): resolve redis type errors in RateLimitStateTracker
A cold mypy run (the gate's true state — prior passes were warm-cache only)
flagged four redis-typing errors in rate_limit_tracker.py that the merge
missed: three unused type:ignore[type-arg] on redis.Redis, and an
aclose() the bundled redis type stub doesn't expose.
Drop the now-unused ignores, and close the scan client via
'async with redis.from_url(...) as r:' instead of a finally-block
aclose(). The context manager closes the client on exit using the modern
redis.asyncio API — no deprecated close(), no stub-missing aclose(), no
suppression. Extend the test's redis mock to model the async
context-manager protocol so it returns itself on enter.
* test(prompter): pass route='main_pm' in the product main-PM routing test
Pre-existing master failure, unrelated to the rate-limit work. The test is
named ...product_routes_to_main_pm and asserts team=MAIN_PM, but called
confirm_live_draft without a route, so it got the 'board' default — which
assigns the Product Owner and yields team=BOARD by design (the board-review
path keeps the root at team=board until the CEO approves). The Main-PM path
is selected with route='main_pm', exactly as the sibling
...main_pm_route_assigns_main_pm test does. Add the missing kwarg so the test
verifies the path it names; behaviour under test is unchanged.
* Updated uv.lock
* refactor(complexity): bring all rank-C blocks under the xenon B ceiling
The full quality gate's xenon step (--max-absolute B --max-modules A
--max-average A) failed on eight rank-C blocks plus the extraction module
average — debt the rate-limit and token-analytics merges deferred. Reduce
each by extracting cohesive helpers, behaviour unchanged:
- orchestrator._probe_one_provider: split into _too_early_to_probe,
_on_probe_success, _on_probe_failure, _parked_agents_for.
- rate_limit_tracker.list_rate_limited_providers: extract _read_rate_limited_entry
and a _decode helper.
- trigger_filter.decide_spawn: extract _stale_trigger_decision (drops the
PLR0911 suppression too).
- ollama_embedder (embed_query, _embed_batch_sync, aembed_query,
_embed_batch_async): share _rl_backoff / _map_embed_error / _log_429 /
_sleep_connect_retry / _asleep_connect_retry; remove a dead post-loop guard
in aembed_query.
- mentor._synthesize_answer: extract _select_system_prompt and
_answer_from_response.
- indexes/base.ask: extract the 429-retried LLM call into _ask_llm.
- extraction.__init__: extract _compile_patterns so the module average
lands at rank A.
xenon now exits 0; rate-limit, optimal_brain, extraction, and events suites
all green.
* chore(deps): drop obsolete types-redis stub; honor redis 8.0 inline types
types-redis 4.6 (typed for redis 4.x) shadowed redis 8.0's own inline types,
which both masked real annotation mismatches in stream_bus.py and forced
awkward workarounds elsewhere. The stale stub is why the mypy gate only ever
passed warm-cached: a cold run under the wrong stub disagreed with the code.
Remove types-redis (and its orphaned transitive stubs) so mypy uses redis's
shipped types. That surfaces that xreadgroup/xclaim return bytes-keyed records
while _handle_message is annotated str — the code already decodes bytes
defensively, so this is an annotation gap, not a runtime bug. Make the types
honest: cast each result to its concrete shape and decode the stream name and
message id to str at the dispatch boundary via a _to_str helper.
mypy roboco/ is now clean cold (247 files) against redis's real types; events
suite green.
* Updated uv.lock
* fix(workspace): install the dev extra so agents can run make quality
Agent workspaces were set up with plain `uv sync`, which installs only the
project's default dependency group (pytest) — not the `dev` *extra* where the
gate tools live (ruff, mypy, xenon, radon, vulture, bandit, deptry). So an
agent's .venv had pytest but no linters, and `make quality` died immediately
on `ruff: command not found`. Agents literally could not lint, type-check, or
complexity-check their own work, which is how format/mypy/xenon debt merged
unseen. Sync the `dev` extra (`uv sync --extra dev`) so the workspace gets the
full toolchain the setup's own docstring already promised.
* fix(panel): rate-limit endpoint shape + websocket path
Two panel-facing breakages from the rate-limit rework:
- GET /api/system/rate-limits returned a raw list, but the panel store reads
response.entries — so `r.entries is not iterable` crashed the banner sync on
page load. Return the panel's contract: a { entries: [...] } envelope whose
items are camelCase {provider, affectedAgents, hitAt, resumeAt,
retryAfterSeconds}, derived from the raw Redis state (resumeAt = hitAt +
retryAfter).
- The rate-limit websocket hook passed "/ws/system" while getWebSocketUrl()
already supplies the "/ws" base, producing the doubled "/ws/ws/system" URL.
Pass "/system" to match the agents/channels/notifications hooks.
Note: the backend /ws/system endpoint itself does not yet exist (the rework
shipped the panel hook only); the REST fix keeps the banner correct on load
and reconnect until that endpoint is built.
* test(workspace): assert uv sync installs the dev extra
Follow the workspace setup change: the dependency-install command is now
`uv sync --extra dev` so the agent workspace gets the lint/type/complexity
toolchain. Update the three assertions that pinned the old `uv sync`.
* feat(ws): add /ws/system stream and bridge rate-limit events to the panel
The rate-limit rework shipped the panel's websocket hook but no backend: there
was no /ws/system endpoint and nothing forwarded RATE_LIMIT_HIT/LIFTED to a
socket, so the banner got no live updates.
Build the missing half:
- ConnectionManager grows a system-wide connection set with connect_system /
broadcast_system, and disconnect() now clears it.
- A /ws/system websocket endpoint (operator stream, no per-agent keying) with
the same connected + ping/pong lifecycle as the other streams.
- websocket_bridge subscribes RATE_LIMIT_HIT/LIFTED and forwards each to
broadcast_system tagged with the type the panel switches on. Both events
ride the same StreamEventBus singleton, and the subscriptions register
before start_listening(), so the consumer reads their streams.
Pairs with the panel hook now passing '/system' (getWebSocketUrl supplies the
'/ws' base). Covered by handler, manager, and endpoint-lifecycle tests.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
187 lines
5.9 KiB
Python
187 lines
5.9 KiB
Python
"""
|
|
WebSocket Event Bridge
|
|
|
|
Consumes events from Redis Streams and forwards them to WebSocket clients.
|
|
This enables real-time updates to connected clients while maintaining
|
|
durable message delivery through the stream.
|
|
"""
|
|
|
|
from uuid import UUID
|
|
|
|
import structlog
|
|
|
|
from roboco.api.websocket import broadcast_notification, manager
|
|
from roboco.events import Event, EventType, get_event_bus
|
|
|
|
logger = structlog.get_logger()
|
|
|
|
_RATE_LIMIT_WS_TYPES = {
|
|
EventType.RATE_LIMIT_HIT: "RATE_LIMIT_HIT",
|
|
EventType.RATE_LIMIT_LIFTED: "RATE_LIMIT_LIFTED",
|
|
}
|
|
|
|
|
|
# Handler for notification events
|
|
async def _handle_notification_sent(event: Event) -> None:
|
|
"""Handle NOTIFICATION_SENT events and forward to WebSocket."""
|
|
data = event.data
|
|
|
|
notification_id_str = data.get("notification_id")
|
|
# SENT events carry `recipient_id`; ACKED events carry `agent_id` (the
|
|
# agent who acknowledged). This handler serves both, so accept either —
|
|
# otherwise every acknowledgement logged a spurious "Incomplete
|
|
# notification event" and never reached the panel.
|
|
recipient_id_str = data.get("recipient_id") or data.get("agent_id")
|
|
notification_type = data.get("type", "unknown")
|
|
subject = data.get("subject", "")
|
|
priority = data.get("priority", "normal")
|
|
|
|
if not notification_id_str or not recipient_id_str:
|
|
logger.warning(
|
|
"Incomplete notification event",
|
|
event_id=str(event.id),
|
|
)
|
|
return
|
|
|
|
try:
|
|
notification_id = UUID(notification_id_str)
|
|
recipient_id = UUID(recipient_id_str)
|
|
except ValueError as e:
|
|
logger.error("Invalid UUID in notification event", error=str(e))
|
|
return
|
|
|
|
# Check if recipient has WebSocket connections
|
|
connections = manager.notification_connections.get(recipient_id, set())
|
|
if connections:
|
|
await broadcast_notification(
|
|
agent_ids=[recipient_id],
|
|
notification_id=notification_id,
|
|
notification_type=notification_type,
|
|
subject=subject,
|
|
priority=priority,
|
|
)
|
|
logger.debug(
|
|
"Notification forwarded to WebSocket",
|
|
notification_id=notification_id_str,
|
|
recipient=recipient_id_str,
|
|
connection_count=len(connections),
|
|
)
|
|
|
|
|
|
async def _handle_session_event(event: Event) -> None:
|
|
"""Handle session lifecycle events and forward to WebSocket."""
|
|
data = event.data
|
|
|
|
session_id_str = data.get("session_id")
|
|
if not session_id_str:
|
|
return
|
|
|
|
try:
|
|
session_id = UUID(session_id_str)
|
|
except ValueError:
|
|
return
|
|
|
|
connections = manager.session_connections.get(session_id, set())
|
|
if not connections:
|
|
return
|
|
|
|
# Forward event to session subscribers
|
|
event_payload = {
|
|
"type": f"session.{event.type.value.split('.')[-1]}",
|
|
"session_id": session_id_str,
|
|
"data": data,
|
|
}
|
|
|
|
await manager.broadcast_to_session(session_id, event_payload)
|
|
logger.debug(
|
|
"Session event forwarded to WebSocket",
|
|
event_type=event.type.value,
|
|
session_id=session_id_str,
|
|
)
|
|
|
|
|
|
async def _handle_agent_event(event: Event) -> None:
|
|
"""Handle agent lifecycle events and forward to WebSocket."""
|
|
data = event.data
|
|
|
|
agent_id_str = data.get("agent_id") or event.source_agent
|
|
if not agent_id_str:
|
|
return
|
|
|
|
try:
|
|
agent_id = UUID(agent_id_str)
|
|
except ValueError:
|
|
return
|
|
|
|
connections = manager.agent_connections.get(agent_id, set())
|
|
if not connections:
|
|
return
|
|
|
|
event_payload = {
|
|
"type": f"agent.{event.type.value.split('.')[-1]}",
|
|
"agent_id": agent_id_str,
|
|
"data": data,
|
|
}
|
|
|
|
await manager.broadcast_to_agent_watchers(agent_id, event_payload)
|
|
logger.debug(
|
|
"Agent event forwarded to WebSocket",
|
|
event_type=event.type.value,
|
|
agent_id=agent_id_str,
|
|
)
|
|
|
|
|
|
async def _handle_rate_limit_event(event: Event) -> None:
|
|
"""Forward RATE_LIMIT_HIT/LIFTED events to operator system WS clients.
|
|
|
|
The published payload already carries the panel's fields
|
|
(``provider``, ``affectedAgents``, ``retryAfterSeconds``, ``timestamp``);
|
|
we only tag it with the discriminating ``type`` the panel switches on.
|
|
"""
|
|
ws_type = _RATE_LIMIT_WS_TYPES.get(event.type)
|
|
if ws_type is None:
|
|
return
|
|
await manager.broadcast_system({"type": ws_type, **event.data})
|
|
|
|
|
|
def register_websocket_bridge_handlers() -> None:
|
|
"""
|
|
Register event handlers that forward events to WebSocket clients.
|
|
|
|
Call this during application startup after the event bus is initialized.
|
|
"""
|
|
bus = get_event_bus()
|
|
|
|
# Notification events -> WebSocket
|
|
bus.subscribe(EventType.NOTIFICATION_SENT, _handle_notification_sent)
|
|
bus.subscribe(EventType.NOTIFICATION_ACKED, _handle_notification_sent)
|
|
|
|
# Session events -> WebSocket
|
|
bus.subscribe(EventType.SESSION_CREATED, _handle_session_event)
|
|
bus.subscribe(EventType.SESSION_CLOSED, _handle_session_event)
|
|
bus.subscribe(EventType.SESSION_TIMEOUT, _handle_session_event)
|
|
|
|
# Agent events -> WebSocket
|
|
bus.subscribe(EventType.AGENT_SPAWNED, _handle_agent_event)
|
|
bus.subscribe(EventType.AGENT_STOPPED, _handle_agent_event)
|
|
bus.subscribe(EventType.AGENT_WAITING, _handle_agent_event)
|
|
bus.subscribe(EventType.AGENT_RESUMED, _handle_agent_event)
|
|
bus.subscribe(EventType.AGENT_ERROR, _handle_agent_event)
|
|
|
|
# Rate-limit lifecycle -> system WebSocket (panel banner)
|
|
bus.subscribe(EventType.RATE_LIMIT_HIT, _handle_rate_limit_event)
|
|
bus.subscribe(EventType.RATE_LIMIT_LIFTED, _handle_rate_limit_event)
|
|
|
|
logger.info("WebSocket bridge handlers registered")
|
|
|
|
|
|
async def start_websocket_bridge() -> None:
|
|
"""
|
|
Start the WebSocket bridge.
|
|
|
|
This registers handlers and ensures they're connected to the event stream.
|
|
Should be called during application startup.
|
|
"""
|
|
register_websocket_bridge_handlers()
|
|
logger.info("WebSocket bridge started")
|