Files
roboco/roboco/api/websocket.py
T
303c2db289 Fix: rate limit real probe (#110)
* fix(rate-limit): real provider liveness probe instead of time-based stub

The rate-limit recovery sweeper cleared a provider and resumed parked agents
purely on elapsed time — _do_probe was a stub that always returned True once
the retry_after window passed, so it never confirmed the provider had actually
stopped rate-limiting us. Under a sustained limit that resumes agents straight
into another 429, re-parking them: avoidable churn.

Make the probe real. _do_probe now issues a free, unmetered liveness call —
Anthropic GET /v1/models or Ollama GET /api/tags — and treats any non-429
response as the limit having lifted. A 429 keeps the provider parked; a
network error keeps it parked too (retry next sweep). When the provider can't
be probed (no API key, or an unrecognized provider), it falls back to the
prior time-expiry optimism rather than stranding agents. _probe_target keeps
URL/header resolution separate and testable, and _do_probe stays a
monkeypatchable boundary so the existing sweep tests are unaffected.

Also drop two acceptance-criteria-number labels from comments in this file.

* chore(rate-limit): clear merged gate debt in rate-limit tests + deps lint

The rate-limit PR landed with ruff violations the full gate flags but the
authors' runs missed: test_rate_limit_sweep.py was unformatted, and
test_rate_limit_tracker.py had unsorted/unused imports and magic-value
comparisons. Format the sweep test, drop the dead imports, and bind the
magic comparison values to locals. Also strip acceptance-criteria-number
labels from comments/docstrings across the three rate-limit test files
(leaving genuine acceptance_criteria=[...] test data untouched), and add
api/deps.py to the PLC0415 per-file-ignore — it is the DI wiring hub and
defers a couple of service imports to call time to avoid import cycles,
the same rationale already applied to api/routes, runtime, and services.

* fix(rate-limit): resolve redis type errors in RateLimitStateTracker

A cold mypy run (the gate's true state — prior passes were warm-cache only)
flagged four redis-typing errors in rate_limit_tracker.py that the merge
missed: three unused type:ignore[type-arg] on redis.Redis, and an
aclose() the bundled redis type stub doesn't expose.

Drop the now-unused ignores, and close the scan client via
'async with redis.from_url(...) as r:' instead of a finally-block
aclose(). The context manager closes the client on exit using the modern
redis.asyncio API — no deprecated close(), no stub-missing aclose(), no
suppression. Extend the test's redis mock to model the async
context-manager protocol so it returns itself on enter.

* test(prompter): pass route='main_pm' in the product main-PM routing test

Pre-existing master failure, unrelated to the rate-limit work. The test is
named ...product_routes_to_main_pm and asserts team=MAIN_PM, but called
confirm_live_draft without a route, so it got the 'board' default — which
assigns the Product Owner and yields team=BOARD by design (the board-review
path keeps the root at team=board until the CEO approves). The Main-PM path
is selected with route='main_pm', exactly as the sibling
...main_pm_route_assigns_main_pm test does. Add the missing kwarg so the test
verifies the path it names; behaviour under test is unchanged.

* Updated uv.lock

* refactor(complexity): bring all rank-C blocks under the xenon B ceiling

The full quality gate's xenon step (--max-absolute B --max-modules A
--max-average A) failed on eight rank-C blocks plus the extraction module
average — debt the rate-limit and token-analytics merges deferred. Reduce
each by extracting cohesive helpers, behaviour unchanged:

- orchestrator._probe_one_provider: split into _too_early_to_probe,
  _on_probe_success, _on_probe_failure, _parked_agents_for.
- rate_limit_tracker.list_rate_limited_providers: extract _read_rate_limited_entry
  and a _decode helper.
- trigger_filter.decide_spawn: extract _stale_trigger_decision (drops the
  PLR0911 suppression too).
- ollama_embedder (embed_query, _embed_batch_sync, aembed_query,
  _embed_batch_async): share _rl_backoff / _map_embed_error / _log_429 /
  _sleep_connect_retry / _asleep_connect_retry; remove a dead post-loop guard
  in aembed_query.
- mentor._synthesize_answer: extract _select_system_prompt and
  _answer_from_response.
- indexes/base.ask: extract the 429-retried LLM call into _ask_llm.
- extraction.__init__: extract _compile_patterns so the module average
  lands at rank A.

xenon now exits 0; rate-limit, optimal_brain, extraction, and events suites
all green.

* chore(deps): drop obsolete types-redis stub; honor redis 8.0 inline types

types-redis 4.6 (typed for redis 4.x) shadowed redis 8.0's own inline types,
which both masked real annotation mismatches in stream_bus.py and forced
awkward workarounds elsewhere. The stale stub is why the mypy gate only ever
passed warm-cached: a cold run under the wrong stub disagreed with the code.

Remove types-redis (and its orphaned transitive stubs) so mypy uses redis's
shipped types. That surfaces that xreadgroup/xclaim return bytes-keyed records
while _handle_message is annotated str — the code already decodes bytes
defensively, so this is an annotation gap, not a runtime bug. Make the types
honest: cast each result to its concrete shape and decode the stream name and
message id to str at the dispatch boundary via a _to_str helper.

mypy roboco/ is now clean cold (247 files) against redis's real types; events
suite green.

* Updated uv.lock

* fix(workspace): install the dev extra so agents can run make quality

Agent workspaces were set up with plain `uv sync`, which installs only the
project's default dependency group (pytest) — not the `dev` *extra* where the
gate tools live (ruff, mypy, xenon, radon, vulture, bandit, deptry). So an
agent's .venv had pytest but no linters, and `make quality` died immediately
on `ruff: command not found`. Agents literally could not lint, type-check, or
complexity-check their own work, which is how format/mypy/xenon debt merged
unseen. Sync the `dev` extra (`uv sync --extra dev`) so the workspace gets the
full toolchain the setup's own docstring already promised.

* fix(panel): rate-limit endpoint shape + websocket path

Two panel-facing breakages from the rate-limit rework:

- GET /api/system/rate-limits returned a raw list, but the panel store reads
  response.entries — so `r.entries is not iterable` crashed the banner sync on
  page load. Return the panel's contract: a { entries: [...] } envelope whose
  items are camelCase {provider, affectedAgents, hitAt, resumeAt,
  retryAfterSeconds}, derived from the raw Redis state (resumeAt = hitAt +
  retryAfter).
- The rate-limit websocket hook passed "/ws/system" while getWebSocketUrl()
  already supplies the "/ws" base, producing the doubled "/ws/ws/system" URL.
  Pass "/system" to match the agents/channels/notifications hooks.

Note: the backend /ws/system endpoint itself does not yet exist (the rework
shipped the panel hook only); the REST fix keeps the banner correct on load
and reconnect until that endpoint is built.

* test(workspace): assert uv sync installs the dev extra

Follow the workspace setup change: the dependency-install command is now
`uv sync --extra dev` so the agent workspace gets the lint/type/complexity
toolchain. Update the three assertions that pinned the old `uv sync`.

* feat(ws): add /ws/system stream and bridge rate-limit events to the panel

The rate-limit rework shipped the panel's websocket hook but no backend: there
was no /ws/system endpoint and nothing forwarded RATE_LIMIT_HIT/LIFTED to a
socket, so the banner got no live updates.

Build the missing half:
- ConnectionManager grows a system-wide connection set with connect_system /
  broadcast_system, and disconnect() now clears it.
- A /ws/system websocket endpoint (operator stream, no per-agent keying) with
  the same connected + ping/pong lifecycle as the other streams.
- websocket_bridge subscribes RATE_LIMIT_HIT/LIFTED and forwards each to
  broadcast_system tagged with the type the panel switches on. Both events
  ride the same StreamEventBus singleton, and the subscriptions register
  before start_listening(), so the consumer reads their streams.

Pairs with the panel hook now passing '/system' (getWebSocketUrl supplies the
'/ws' base). Covered by handler, manager, and endpoint-lifecycle tests.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-06-11 18:16:20 +02:00

509 lines
16 KiB
Python

"""
WebSocket Handlers
Real-time communication via WebSocket connections for:
- Channel streams (all messages in a channel)
- Agent streams (individual agent output)
- Session streams (messages in a session)
Security Note:
WebSocket connections validate agent_id via query params and verify
the agent exists in the database. In production, this should be
enhanced with proper token-based authentication (JWT, etc.).
"""
import asyncio
import json
from datetime import UTC, datetime
from typing import Any
from uuid import UUID
import httpx
from fastapi import APIRouter, WebSocket, WebSocketDisconnect, status
from roboco.config import settings
from roboco.db.base import get_db
from roboco.services.repositories import resolve_agent_uuid
router = APIRouter()
# =============================================================================
# Connection Manager
# =============================================================================
class ConnectionManager:
"""
Manages WebSocket connections organized by type and ID.
Supports:
- Channel subscriptions
- Agent output streams
- Session streams
"""
def __init__(self) -> None:
# channel_id -> set of websockets
self.channel_connections: dict[UUID, set[WebSocket]] = {}
# agent_id -> set of websockets
self.agent_connections: dict[UUID, set[WebSocket]] = {}
# session_id -> set of websockets
self.session_connections: dict[UUID, set[WebSocket]] = {}
# agent_id -> set of websockets (for notifications)
self.notification_connections: dict[UUID, set[WebSocket]] = {}
# Operator/system-wide stream (rate limits, etc.) — no per-agent keying.
self.system_connections: set[WebSocket] = set()
# websocket -> agent_id (for tracking who is connected)
self.connection_agents: dict[WebSocket, UUID] = {}
async def connect_channel(
self, websocket: WebSocket, channel_id: UUID, agent_id: UUID
) -> None:
"""Connect to a channel stream."""
await websocket.accept()
if channel_id not in self.channel_connections:
self.channel_connections[channel_id] = set()
self.channel_connections[channel_id].add(websocket)
self.connection_agents[websocket] = agent_id
async def connect_agent(
self, websocket: WebSocket, target_agent_id: UUID, viewer_agent_id: UUID
) -> None:
"""Connect to an agent's output stream."""
await websocket.accept()
if target_agent_id not in self.agent_connections:
self.agent_connections[target_agent_id] = set()
self.agent_connections[target_agent_id].add(websocket)
self.connection_agents[websocket] = viewer_agent_id
async def connect_session(
self, websocket: WebSocket, session_id: UUID, agent_id: UUID
) -> None:
"""Connect to a session stream."""
await websocket.accept()
if session_id not in self.session_connections:
self.session_connections[session_id] = set()
self.session_connections[session_id].add(websocket)
self.connection_agents[websocket] = agent_id
async def connect_notifications(self, websocket: WebSocket, agent_id: UUID) -> None:
"""Connect to an agent's notification stream."""
await websocket.accept()
if agent_id not in self.notification_connections:
self.notification_connections[agent_id] = set()
self.notification_connections[agent_id].add(websocket)
self.connection_agents[websocket] = agent_id
async def connect_system(self, websocket: WebSocket) -> None:
"""Connect to the operator/system-wide stream (rate limits, etc.)."""
await websocket.accept()
self.system_connections.add(websocket)
def disconnect(self, websocket: WebSocket) -> None:
"""Remove a websocket from all subscriptions."""
# Remove from channel connections
for connections in self.channel_connections.values():
connections.discard(websocket)
# Remove from agent connections
for connections in self.agent_connections.values():
connections.discard(websocket)
# Remove from session connections
for connections in self.session_connections.values():
connections.discard(websocket)
# Remove from notification connections
for connections in self.notification_connections.values():
connections.discard(websocket)
# Remove from the system-wide stream
self.system_connections.discard(websocket)
# Remove from tracking
self.connection_agents.pop(websocket, None)
async def broadcast_to_channel(
self, channel_id: UUID, message: dict[str, Any]
) -> None:
"""Broadcast a message to all channel subscribers."""
connections = self.channel_connections.get(channel_id, set())
if not connections:
return
data = json.dumps(message, default=str)
await asyncio.gather(
*[conn.send_text(data) for conn in connections],
return_exceptions=True,
)
async def broadcast_to_agent_watchers(
self, agent_id: UUID, message: dict[str, Any]
) -> None:
"""Broadcast a message to all watching an agent's stream."""
connections = self.agent_connections.get(agent_id, set())
if not connections:
return
data = json.dumps(message, default=str)
await asyncio.gather(
*[conn.send_text(data) for conn in connections],
return_exceptions=True,
)
async def broadcast_to_session(
self, session_id: UUID, message: dict[str, Any]
) -> None:
"""Broadcast a message to all session subscribers."""
connections = self.session_connections.get(session_id, set())
if not connections:
return
data = json.dumps(message, default=str)
await asyncio.gather(
*[conn.send_text(data) for conn in connections],
return_exceptions=True,
)
async def broadcast_system(self, message: dict[str, Any]) -> None:
"""Broadcast a message to all operator/system-wide subscribers."""
if not self.system_connections:
return
data = json.dumps(message, default=str)
await asyncio.gather(
*[conn.send_text(data) for conn in self.system_connections],
return_exceptions=True,
)
def get_channel_subscriber_count(self, channel_id: UUID) -> int:
"""Get number of subscribers to a channel."""
return len(self.channel_connections.get(channel_id, set()))
def get_agent_watcher_count(self, agent_id: UUID) -> int:
"""Get number of watchers of an agent's stream."""
return len(self.agent_connections.get(agent_id, set()))
# Global connection manager
manager = ConnectionManager()
async def validate_agent_exists(agent_id: UUID | str) -> bool:
"""
Validate that an agent exists in the database.
This provides basic security by ensuring the claimed agent_id
is a valid agent, not just a valid UUID format.
TODO: Enhance with token-based authentication (JWT) for production.
"""
try:
async for db in get_db():
result = await resolve_agent_uuid(db, str(agent_id))
return result is not None
except Exception:
return False
return False
async def validate_channel_access(channel_id: UUID, agent_id: UUID) -> bool:
"""
Validate that an agent has access to a channel.
Calls the permissions API to check read access.
"""
try:
url = f"http://{settings.host}:{settings.port}/api/permissions/check"
async with httpx.AsyncClient(timeout=10.0) as client:
response = await client.get(
url,
params={
"agent_id": str(agent_id),
"channel_id": str(channel_id),
"action": "read",
},
)
if response.status_code == status.HTTP_200_OK:
data = response.json()
return bool(data.get("allowed", False))
return False
except Exception:
# On error, deny access (fail closed)
return False
# =============================================================================
# WebSocket Routes
# =============================================================================
@router.websocket("/channels/{channel_id}")
async def channel_stream(
websocket: WebSocket,
channel_id: UUID,
) -> None:
"""
WebSocket endpoint for channel message streams.
Clients receive real-time messages for the channel.
"""
# Get agent ID from query params (or auth in production)
agent_id_str = websocket.query_params.get("agent_id")
if not agent_id_str:
await websocket.close(code=status.WS_1008_POLICY_VIOLATION)
return
try:
agent_id = UUID(agent_id_str)
except ValueError:
await websocket.close(code=status.WS_1008_POLICY_VIOLATION)
return
# Validate agent access to channel
has_access = await validate_channel_access(channel_id, agent_id)
if not has_access:
await websocket.close(code=status.WS_1008_POLICY_VIOLATION)
return
await manager.connect_channel(websocket, channel_id, agent_id)
try:
# Send connection confirmation
await websocket.send_json(
{
"type": "connected",
"channel_id": str(channel_id),
"subscriber_count": manager.get_channel_subscriber_count(channel_id),
}
)
# Keep connection alive and handle incoming messages
while True:
data = await websocket.receive_text()
# Handle ping/pong for keepalive
if data == "ping":
await websocket.send_text("pong")
continue
# Handle other client messages if needed
# For now, channels are primarily for receiving
except WebSocketDisconnect:
manager.disconnect(websocket)
@router.websocket("/agents/{agent_id}")
async def agent_stream(
websocket: WebSocket,
agent_id: UUID,
) -> None:
"""
WebSocket endpoint for an agent's output stream.
Clients receive real-time LLM output from the agent.
"""
# Get viewer agent ID
viewer_id_str = websocket.query_params.get("viewer_id")
if not viewer_id_str:
await websocket.close(code=status.WS_1008_POLICY_VIOLATION)
return
try:
viewer_id = UUID(viewer_id_str)
except ValueError:
await websocket.close(code=status.WS_1008_POLICY_VIOLATION)
return
# Validate viewer agent exists in database
if not await validate_agent_exists(viewer_id):
await websocket.close(code=status.WS_1008_POLICY_VIOLATION)
return
await manager.connect_agent(websocket, agent_id, viewer_id)
try:
await websocket.send_json(
{
"type": "connected",
"agent_id": str(agent_id),
"watcher_count": manager.get_agent_watcher_count(agent_id),
}
)
while True:
data = await websocket.receive_text()
if data == "ping":
await websocket.send_text("pong")
except WebSocketDisconnect:
manager.disconnect(websocket)
@router.websocket("/sessions/{session_id}")
async def session_stream(
websocket: WebSocket,
session_id: UUID,
) -> None:
"""
WebSocket endpoint for session message streams.
Clients receive real-time messages for a specific session.
"""
agent_id_str = websocket.query_params.get("agent_id")
if not agent_id_str:
await websocket.close(code=status.WS_1008_POLICY_VIOLATION)
return
try:
agent_id = UUID(agent_id_str)
except ValueError:
await websocket.close(code=status.WS_1008_POLICY_VIOLATION)
return
# Validate agent exists in database
if not await validate_agent_exists(agent_id):
await websocket.close(code=status.WS_1008_POLICY_VIOLATION)
return
await manager.connect_session(websocket, session_id, agent_id)
try:
await websocket.send_json(
{
"type": "connected",
"session_id": str(session_id),
}
)
while True:
data = await websocket.receive_text()
if data == "ping":
await websocket.send_text("pong")
except WebSocketDisconnect:
manager.disconnect(websocket)
@router.websocket("/notifications/{agent_id}")
async def notification_stream(
websocket: WebSocket,
agent_id: UUID,
) -> None:
"""
WebSocket endpoint for agent notifications.
Agents receive real-time notifications via this stream.
"""
# Validate agent exists in database
if not await validate_agent_exists(agent_id):
await websocket.close(code=status.WS_1008_POLICY_VIOLATION)
return
await manager.connect_notifications(websocket, agent_id)
try:
await websocket.send_json(
{
"type": "connected",
"agent_id": str(agent_id),
}
)
while True:
data = await websocket.receive_text()
if data == "ping":
await websocket.send_text("pong")
except WebSocketDisconnect:
manager.disconnect(websocket)
@router.websocket("/system")
async def system_stream(websocket: WebSocket) -> None:
"""Operator/system-wide WebSocket stream.
Carries system-level events for the control panel — currently the
rate-limit lifecycle (``RATE_LIMIT_HIT`` / ``RATE_LIMIT_LIFTED``), bridged
from the event bus by ``websocket_bridge``. No per-agent keying or auth:
it's a read-only operator stream behind the panel's own access controls.
"""
await manager.connect_system(websocket)
try:
await websocket.send_json({"type": "connected"})
while True:
data = await websocket.receive_text()
if data == "ping":
await websocket.send_text("pong")
except WebSocketDisconnect:
manager.disconnect(websocket)
# =============================================================================
# Helper Functions for Broadcasting
# =============================================================================
async def broadcast_agent_chunk(
agent_id: str, chunk: str, metadata: dict[str, Any]
) -> None:
"""Broadcast an agent stream chunk to watchers."""
event = {
"type": "agent.stream",
"agent_id": agent_id,
"chunk": chunk,
"timestamp": datetime.now(UTC).isoformat(),
**metadata,
}
await manager.broadcast_to_agent_watchers(UUID(agent_id), event)
async def broadcast_notification(
agent_ids: list[UUID],
notification_id: UUID,
notification_type: str,
subject: str,
priority: str,
) -> None:
"""
Broadcast notification to specific agents.
Sends to all agents that have notification websocket connections.
"""
event = {
"type": "notification",
"notification_id": str(notification_id),
"notification_type": notification_type,
"subject": subject,
"priority": priority,
"timestamp": datetime.now(UTC).isoformat(),
}
data = json.dumps(event)
for agent_id in agent_ids:
connections = manager.notification_connections.get(agent_id, set())
if connections:
await asyncio.gather(
*[conn.send_text(data) for conn in connections],
return_exceptions=True,
)