Files
roboco/roboco/api/websocket_bridge.py
T
303c2db289 Fix: rate limit real probe (#110)
* fix(rate-limit): real provider liveness probe instead of time-based stub

The rate-limit recovery sweeper cleared a provider and resumed parked agents
purely on elapsed time — _do_probe was a stub that always returned True once
the retry_after window passed, so it never confirmed the provider had actually
stopped rate-limiting us. Under a sustained limit that resumes agents straight
into another 429, re-parking them: avoidable churn.

Make the probe real. _do_probe now issues a free, unmetered liveness call —
Anthropic GET /v1/models or Ollama GET /api/tags — and treats any non-429
response as the limit having lifted. A 429 keeps the provider parked; a
network error keeps it parked too (retry next sweep). When the provider can't
be probed (no API key, or an unrecognized provider), it falls back to the
prior time-expiry optimism rather than stranding agents. _probe_target keeps
URL/header resolution separate and testable, and _do_probe stays a
monkeypatchable boundary so the existing sweep tests are unaffected.

Also drop two acceptance-criteria-number labels from comments in this file.

* chore(rate-limit): clear merged gate debt in rate-limit tests + deps lint

The rate-limit PR landed with ruff violations the full gate flags but the
authors' runs missed: test_rate_limit_sweep.py was unformatted, and
test_rate_limit_tracker.py had unsorted/unused imports and magic-value
comparisons. Format the sweep test, drop the dead imports, and bind the
magic comparison values to locals. Also strip acceptance-criteria-number
labels from comments/docstrings across the three rate-limit test files
(leaving genuine acceptance_criteria=[...] test data untouched), and add
api/deps.py to the PLC0415 per-file-ignore — it is the DI wiring hub and
defers a couple of service imports to call time to avoid import cycles,
the same rationale already applied to api/routes, runtime, and services.

* fix(rate-limit): resolve redis type errors in RateLimitStateTracker

A cold mypy run (the gate's true state — prior passes were warm-cache only)
flagged four redis-typing errors in rate_limit_tracker.py that the merge
missed: three unused type:ignore[type-arg] on redis.Redis, and an
aclose() the bundled redis type stub doesn't expose.

Drop the now-unused ignores, and close the scan client via
'async with redis.from_url(...) as r:' instead of a finally-block
aclose(). The context manager closes the client on exit using the modern
redis.asyncio API — no deprecated close(), no stub-missing aclose(), no
suppression. Extend the test's redis mock to model the async
context-manager protocol so it returns itself on enter.

* test(prompter): pass route='main_pm' in the product main-PM routing test

Pre-existing master failure, unrelated to the rate-limit work. The test is
named ...product_routes_to_main_pm and asserts team=MAIN_PM, but called
confirm_live_draft without a route, so it got the 'board' default — which
assigns the Product Owner and yields team=BOARD by design (the board-review
path keeps the root at team=board until the CEO approves). The Main-PM path
is selected with route='main_pm', exactly as the sibling
...main_pm_route_assigns_main_pm test does. Add the missing kwarg so the test
verifies the path it names; behaviour under test is unchanged.

* Updated uv.lock

* refactor(complexity): bring all rank-C blocks under the xenon B ceiling

The full quality gate's xenon step (--max-absolute B --max-modules A
--max-average A) failed on eight rank-C blocks plus the extraction module
average — debt the rate-limit and token-analytics merges deferred. Reduce
each by extracting cohesive helpers, behaviour unchanged:

- orchestrator._probe_one_provider: split into _too_early_to_probe,
  _on_probe_success, _on_probe_failure, _parked_agents_for.
- rate_limit_tracker.list_rate_limited_providers: extract _read_rate_limited_entry
  and a _decode helper.
- trigger_filter.decide_spawn: extract _stale_trigger_decision (drops the
  PLR0911 suppression too).
- ollama_embedder (embed_query, _embed_batch_sync, aembed_query,
  _embed_batch_async): share _rl_backoff / _map_embed_error / _log_429 /
  _sleep_connect_retry / _asleep_connect_retry; remove a dead post-loop guard
  in aembed_query.
- mentor._synthesize_answer: extract _select_system_prompt and
  _answer_from_response.
- indexes/base.ask: extract the 429-retried LLM call into _ask_llm.
- extraction.__init__: extract _compile_patterns so the module average
  lands at rank A.

xenon now exits 0; rate-limit, optimal_brain, extraction, and events suites
all green.

* chore(deps): drop obsolete types-redis stub; honor redis 8.0 inline types

types-redis 4.6 (typed for redis 4.x) shadowed redis 8.0's own inline types,
which both masked real annotation mismatches in stream_bus.py and forced
awkward workarounds elsewhere. The stale stub is why the mypy gate only ever
passed warm-cached: a cold run under the wrong stub disagreed with the code.

Remove types-redis (and its orphaned transitive stubs) so mypy uses redis's
shipped types. That surfaces that xreadgroup/xclaim return bytes-keyed records
while _handle_message is annotated str — the code already decodes bytes
defensively, so this is an annotation gap, not a runtime bug. Make the types
honest: cast each result to its concrete shape and decode the stream name and
message id to str at the dispatch boundary via a _to_str helper.

mypy roboco/ is now clean cold (247 files) against redis's real types; events
suite green.

* Updated uv.lock

* fix(workspace): install the dev extra so agents can run make quality

Agent workspaces were set up with plain `uv sync`, which installs only the
project's default dependency group (pytest) — not the `dev` *extra* where the
gate tools live (ruff, mypy, xenon, radon, vulture, bandit, deptry). So an
agent's .venv had pytest but no linters, and `make quality` died immediately
on `ruff: command not found`. Agents literally could not lint, type-check, or
complexity-check their own work, which is how format/mypy/xenon debt merged
unseen. Sync the `dev` extra (`uv sync --extra dev`) so the workspace gets the
full toolchain the setup's own docstring already promised.

* fix(panel): rate-limit endpoint shape + websocket path

Two panel-facing breakages from the rate-limit rework:

- GET /api/system/rate-limits returned a raw list, but the panel store reads
  response.entries — so `r.entries is not iterable` crashed the banner sync on
  page load. Return the panel's contract: a { entries: [...] } envelope whose
  items are camelCase {provider, affectedAgents, hitAt, resumeAt,
  retryAfterSeconds}, derived from the raw Redis state (resumeAt = hitAt +
  retryAfter).
- The rate-limit websocket hook passed "/ws/system" while getWebSocketUrl()
  already supplies the "/ws" base, producing the doubled "/ws/ws/system" URL.
  Pass "/system" to match the agents/channels/notifications hooks.

Note: the backend /ws/system endpoint itself does not yet exist (the rework
shipped the panel hook only); the REST fix keeps the banner correct on load
and reconnect until that endpoint is built.

* test(workspace): assert uv sync installs the dev extra

Follow the workspace setup change: the dependency-install command is now
`uv sync --extra dev` so the agent workspace gets the lint/type/complexity
toolchain. Update the three assertions that pinned the old `uv sync`.

* feat(ws): add /ws/system stream and bridge rate-limit events to the panel

The rate-limit rework shipped the panel's websocket hook but no backend: there
was no /ws/system endpoint and nothing forwarded RATE_LIMIT_HIT/LIFTED to a
socket, so the banner got no live updates.

Build the missing half:
- ConnectionManager grows a system-wide connection set with connect_system /
  broadcast_system, and disconnect() now clears it.
- A /ws/system websocket endpoint (operator stream, no per-agent keying) with
  the same connected + ping/pong lifecycle as the other streams.
- websocket_bridge subscribes RATE_LIMIT_HIT/LIFTED and forwards each to
  broadcast_system tagged with the type the panel switches on. Both events
  ride the same StreamEventBus singleton, and the subscriptions register
  before start_listening(), so the consumer reads their streams.

Pairs with the panel hook now passing '/system' (getWebSocketUrl supplies the
'/ws' base). Covered by handler, manager, and endpoint-lifecycle tests.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-06-11 18:16:20 +02:00

187 lines
5.9 KiB
Python

"""
WebSocket Event Bridge
Consumes events from Redis Streams and forwards them to WebSocket clients.
This enables real-time updates to connected clients while maintaining
durable message delivery through the stream.
"""
from uuid import UUID
import structlog
from roboco.api.websocket import broadcast_notification, manager
from roboco.events import Event, EventType, get_event_bus
logger = structlog.get_logger()
_RATE_LIMIT_WS_TYPES = {
EventType.RATE_LIMIT_HIT: "RATE_LIMIT_HIT",
EventType.RATE_LIMIT_LIFTED: "RATE_LIMIT_LIFTED",
}
# Handler for notification events
async def _handle_notification_sent(event: Event) -> None:
"""Handle NOTIFICATION_SENT events and forward to WebSocket."""
data = event.data
notification_id_str = data.get("notification_id")
# SENT events carry `recipient_id`; ACKED events carry `agent_id` (the
# agent who acknowledged). This handler serves both, so accept either —
# otherwise every acknowledgement logged a spurious "Incomplete
# notification event" and never reached the panel.
recipient_id_str = data.get("recipient_id") or data.get("agent_id")
notification_type = data.get("type", "unknown")
subject = data.get("subject", "")
priority = data.get("priority", "normal")
if not notification_id_str or not recipient_id_str:
logger.warning(
"Incomplete notification event",
event_id=str(event.id),
)
return
try:
notification_id = UUID(notification_id_str)
recipient_id = UUID(recipient_id_str)
except ValueError as e:
logger.error("Invalid UUID in notification event", error=str(e))
return
# Check if recipient has WebSocket connections
connections = manager.notification_connections.get(recipient_id, set())
if connections:
await broadcast_notification(
agent_ids=[recipient_id],
notification_id=notification_id,
notification_type=notification_type,
subject=subject,
priority=priority,
)
logger.debug(
"Notification forwarded to WebSocket",
notification_id=notification_id_str,
recipient=recipient_id_str,
connection_count=len(connections),
)
async def _handle_session_event(event: Event) -> None:
"""Handle session lifecycle events and forward to WebSocket."""
data = event.data
session_id_str = data.get("session_id")
if not session_id_str:
return
try:
session_id = UUID(session_id_str)
except ValueError:
return
connections = manager.session_connections.get(session_id, set())
if not connections:
return
# Forward event to session subscribers
event_payload = {
"type": f"session.{event.type.value.split('.')[-1]}",
"session_id": session_id_str,
"data": data,
}
await manager.broadcast_to_session(session_id, event_payload)
logger.debug(
"Session event forwarded to WebSocket",
event_type=event.type.value,
session_id=session_id_str,
)
async def _handle_agent_event(event: Event) -> None:
"""Handle agent lifecycle events and forward to WebSocket."""
data = event.data
agent_id_str = data.get("agent_id") or event.source_agent
if not agent_id_str:
return
try:
agent_id = UUID(agent_id_str)
except ValueError:
return
connections = manager.agent_connections.get(agent_id, set())
if not connections:
return
event_payload = {
"type": f"agent.{event.type.value.split('.')[-1]}",
"agent_id": agent_id_str,
"data": data,
}
await manager.broadcast_to_agent_watchers(agent_id, event_payload)
logger.debug(
"Agent event forwarded to WebSocket",
event_type=event.type.value,
agent_id=agent_id_str,
)
async def _handle_rate_limit_event(event: Event) -> None:
"""Forward RATE_LIMIT_HIT/LIFTED events to operator system WS clients.
The published payload already carries the panel's fields
(``provider``, ``affectedAgents``, ``retryAfterSeconds``, ``timestamp``);
we only tag it with the discriminating ``type`` the panel switches on.
"""
ws_type = _RATE_LIMIT_WS_TYPES.get(event.type)
if ws_type is None:
return
await manager.broadcast_system({"type": ws_type, **event.data})
def register_websocket_bridge_handlers() -> None:
"""
Register event handlers that forward events to WebSocket clients.
Call this during application startup after the event bus is initialized.
"""
bus = get_event_bus()
# Notification events -> WebSocket
bus.subscribe(EventType.NOTIFICATION_SENT, _handle_notification_sent)
bus.subscribe(EventType.NOTIFICATION_ACKED, _handle_notification_sent)
# Session events -> WebSocket
bus.subscribe(EventType.SESSION_CREATED, _handle_session_event)
bus.subscribe(EventType.SESSION_CLOSED, _handle_session_event)
bus.subscribe(EventType.SESSION_TIMEOUT, _handle_session_event)
# Agent events -> WebSocket
bus.subscribe(EventType.AGENT_SPAWNED, _handle_agent_event)
bus.subscribe(EventType.AGENT_STOPPED, _handle_agent_event)
bus.subscribe(EventType.AGENT_WAITING, _handle_agent_event)
bus.subscribe(EventType.AGENT_RESUMED, _handle_agent_event)
bus.subscribe(EventType.AGENT_ERROR, _handle_agent_event)
# Rate-limit lifecycle -> system WebSocket (panel banner)
bus.subscribe(EventType.RATE_LIMIT_HIT, _handle_rate_limit_event)
bus.subscribe(EventType.RATE_LIMIT_LIFTED, _handle_rate_limit_event)
logger.info("WebSocket bridge handlers registered")
async def start_websocket_bridge() -> None:
"""
Start the WebSocket bridge.
This registers handlers and ensures they're connected to the event stream.
Should be called during application startup.
"""
register_websocket_bridge_handlers()
logger.info("WebSocket bridge started")