3 Commits

Include renames

Author	SHA1	Message	Date
5 people	547fe444f2	[4865ff8b] Add WebSocket support to the usage dashboard (#115) ... * [e7349d84] feat(dashboard): WS usage store, hook extension, status badge, and smooth animations (#111) (#113) - Add src/store/usage-store.ts with typed UsageData interface, useUsageStore Zustand store, setUsageData, clearUsageData, and setWsState actions - Export useUsageStore and UsageData from store/index.ts - Extend use-rate-limit-websocket.ts: rename msg type to SystemWsMessage, add key_metrics field; add useEffect syncing wsState into useUsageStore; add USAGE_UPDATE/USAGE_SNAPSHOT handler dispatching to useUsageStore (RATE_LIMIT_HIT/LIFTED handling and onReconnect unchanged) - Update CommandCenter to read key_metrics from useUsageStore when wsState === 'connected' and usageData non-null; falls back to useCeoOverview() (refetchInterval: 60000) when WS disconnected - Update KeyMetricsPanel: add wsState prop, render connection status Badge matching AgentStreamViewer pattern (bg-green-500+Wifi / bg-yellow-500+ Loader2 spin / bg-gray-500+WifiOff); add transition-all duration-300 ease-in-out to metric value spans for smooth animated updates Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev> * [c9745ee8] feat(events): add USAGE_UPDATE/SNAPSHOT event types, throttled publisher, /ws/system usage bridge (#112) (#114) - Add EventType.USAGE_UPDATE='usage.update' and EventType.USAGE_SNAPSHOT='usage.snapshot' to the EventType StrEnum in roboco/models/events.py - Create roboco/services/usage_events.py with _UsageThrottle class (5-second per-agent window using time.monotonic()) and publish_usage_update() / publish_usage_snapshot() helpers; lazy imports prevent circular dependency with roboco.events - Extend orchestrator._sweep_token_snapshots() to publish USAGE_UPDATE per active agent (throttled) and a USAGE_SNAPSHOT aggregate after each sweep cycle; wrapped in contextlib.suppress so event errors never abort DB snapshot operations - Add _handle_usage_event() to websocket_bridge.py following _handle_rate_limit_event pattern; register USAGE_UPDATE and USAGE_SNAPSHOT subscriptions in register_websocket_bridge_handlers() forwarding both to /ws/system via broadcast_system() - Add unit tests: test_usage_events.py (throttle suppression, publish helpers) and test_websocket_bridge.py extended with _handle_usage_event coverage and updated registration assertion to include USAGE_UPDATE/USAGE_SNAPSHOT Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> * fix(usage-ws): reconcile the realtime token/cost contract end-to-end The backend and frontend halves shipped mismatched contracts, so the usage dashboard never received live data: - The bridge forwarded the dotted event value ("usage.update") while the panel switched on "USAGE_UPDATE"; map both to the UPPER_SNAKE type string the same way the rate-limit handler does. - The backend emitted token/cost telemetry but the frontend read a key_metrics field and fed the org-metrics panel. Rewire the frontend to consume the USAGE_SNAPSHOT token/cost payload into the "Token Usage & Cost" panel — WS-first with polling fallback and a connection-status badge — and revert the unrelated KeyMetricsPanel / CommandCenter wiring. Backend cleanups in the same path: - Replace the multi-argument publish helpers with typed UsageUpdate / UsageSnapshot payloads, removing the too-many-arguments lint suppressions. - Extract _fetch_agent_tokens and _persist_token_snapshot from the token sweep, removing the too-many-statements suppression; label the live snapshot "live". Hardening uncovered while fixing the above: - _finalize_spawn_session pulled the full RAG stack into the session-finalization path through a transcript-parse import; move the pure parser into a dependency-light roboco.agent_sdk.transcript_usage module so finalization never imports the agent SDK server. - Reduce _finalize_spawn_session complexity by extracting _resolve_final_token_usage, and widen the transcript-fallback guard so a read error can never abort finalization. Also align KeyMetricsPanel with the metrics /dashboard/ceo actually returns: it read velocity_24h / avg_time_to_done / active_agents, none of which get_key_metrics() emits, so four of five rows rendered "—". Render velocity_weekly, completion_rate, documentation_coverage and active_blockers. * docs: note live usage push over /ws/system on the usage dashboard * fix(usage): finalize on self-exit and de-duplicate transcript token counts Two bugs left token capture broken even after the transcript-read fallback landed — surfaced by a live agent run: - Agents that self-exit (the normal i_am_idle -> container shutdown, exit 0) were never finalized. _finalize_spawn_session is only called from stop_agent(), but a graceful self-exit goes through _handle_stopped_container, which set the instance OFFLINE and returned without finalizing — leaving the spawn-session row open with zero tokens. Finalize there for both graceful (exit_reason="completed") and crash (exit_reason="crashed") exits. - sum_transcript_usage double-counted. Claude Code logs one assistant message as several JSONL lines (one per content block — thinking / text / tool_use), each repeating the same message.usage, so summing every line roughly doubled the totals. De-duplicate by message.id. Verified against a live agent transcript: the raw sum (12, 1068, 62502, 115828) vs the de-duped (6, 516, 62502, 63336), which matches the session's authoritative result.usage exactly. * feat(usage): fall back to the transcript in the live token sweep The 60s token sweep read only the agent SDK's /usage/status, which races container teardown and reports zero mid-run — so live usage (and the USAGE_SNAPSHOT pushed to /ws/system) stayed at zero for active agents. Extract _resolve_active_tokens: try the SDK, then fall back to the durable transcript (the same source finalize uses) so running agents report live. * feat(usage): add GET /usage/sessions for the dashboard's Recent Sessions The panel's Recent Sessions table was mock-only — the backend had no sessions endpoint, so production always showed 'No sessions recorded yet'. Add UsageService.get_recent_sessions + a /usage/sessions route returning the most recent spawn-session rows (token totals + cost), and point the panel client at it. --------- Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev> Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> Co-authored-by: Renn F <rennf93@users.noreply.github.com>	2026-06-11 23:19:50 +02:00
3 people	303c2db289	Fix: rate limit real probe (#110) ... * fix(rate-limit): real provider liveness probe instead of time-based stub The rate-limit recovery sweeper cleared a provider and resumed parked agents purely on elapsed time — _do_probe was a stub that always returned True once the retry_after window passed, so it never confirmed the provider had actually stopped rate-limiting us. Under a sustained limit that resumes agents straight into another 429, re-parking them: avoidable churn. Make the probe real. _do_probe now issues a free, unmetered liveness call — Anthropic GET /v1/models or Ollama GET /api/tags — and treats any non-429 response as the limit having lifted. A 429 keeps the provider parked; a network error keeps it parked too (retry next sweep). When the provider can't be probed (no API key, or an unrecognized provider), it falls back to the prior time-expiry optimism rather than stranding agents. _probe_target keeps URL/header resolution separate and testable, and _do_probe stays a monkeypatchable boundary so the existing sweep tests are unaffected. Also drop two acceptance-criteria-number labels from comments in this file. * chore(rate-limit): clear merged gate debt in rate-limit tests + deps lint The rate-limit PR landed with ruff violations the full gate flags but the authors' runs missed: test_rate_limit_sweep.py was unformatted, and test_rate_limit_tracker.py had unsorted/unused imports and magic-value comparisons. Format the sweep test, drop the dead imports, and bind the magic comparison values to locals. Also strip acceptance-criteria-number labels from comments/docstrings across the three rate-limit test files (leaving genuine acceptance_criteria=[...] test data untouched), and add api/deps.py to the PLC0415 per-file-ignore — it is the DI wiring hub and defers a couple of service imports to call time to avoid import cycles, the same rationale already applied to api/routes, runtime, and services. * fix(rate-limit): resolve redis type errors in RateLimitStateTracker A cold mypy run (the gate's true state — prior passes were warm-cache only) flagged four redis-typing errors in rate_limit_tracker.py that the merge missed: three unused type:ignore[type-arg] on redis.Redis, and an aclose() the bundled redis type stub doesn't expose. Drop the now-unused ignores, and close the scan client via 'async with redis.from_url(...) as r:' instead of a finally-block aclose(). The context manager closes the client on exit using the modern redis.asyncio API — no deprecated close(), no stub-missing aclose(), no suppression. Extend the test's redis mock to model the async context-manager protocol so it returns itself on enter. * test(prompter): pass route='main_pm' in the product main-PM routing test Pre-existing master failure, unrelated to the rate-limit work. The test is named ...product_routes_to_main_pm and asserts team=MAIN_PM, but called confirm_live_draft without a route, so it got the 'board' default — which assigns the Product Owner and yields team=BOARD by design (the board-review path keeps the root at team=board until the CEO approves). The Main-PM path is selected with route='main_pm', exactly as the sibling ...main_pm_route_assigns_main_pm test does. Add the missing kwarg so the test verifies the path it names; behaviour under test is unchanged. * Updated uv.lock * refactor(complexity): bring all rank-C blocks under the xenon B ceiling The full quality gate's xenon step (--max-absolute B --max-modules A --max-average A) failed on eight rank-C blocks plus the extraction module average — debt the rate-limit and token-analytics merges deferred. Reduce each by extracting cohesive helpers, behaviour unchanged: - orchestrator._probe_one_provider: split into _too_early_to_probe, _on_probe_success, _on_probe_failure, _parked_agents_for. - rate_limit_tracker.list_rate_limited_providers: extract _read_rate_limited_entry and a _decode helper. - trigger_filter.decide_spawn: extract _stale_trigger_decision (drops the PLR0911 suppression too). - ollama_embedder (embed_query, _embed_batch_sync, aembed_query, _embed_batch_async): share _rl_backoff / _map_embed_error / _log_429 / _sleep_connect_retry / _asleep_connect_retry; remove a dead post-loop guard in aembed_query. - mentor._synthesize_answer: extract _select_system_prompt and _answer_from_response. - indexes/base.ask: extract the 429-retried LLM call into _ask_llm. - extraction.__init__: extract _compile_patterns so the module average lands at rank A. xenon now exits 0; rate-limit, optimal_brain, extraction, and events suites all green. * chore(deps): drop obsolete types-redis stub; honor redis 8.0 inline types types-redis 4.6 (typed for redis 4.x) shadowed redis 8.0's own inline types, which both masked real annotation mismatches in stream_bus.py and forced awkward workarounds elsewhere. The stale stub is why the mypy gate only ever passed warm-cached: a cold run under the wrong stub disagreed with the code. Remove types-redis (and its orphaned transitive stubs) so mypy uses redis's shipped types. That surfaces that xreadgroup/xclaim return bytes-keyed records while _handle_message is annotated str — the code already decodes bytes defensively, so this is an annotation gap, not a runtime bug. Make the types honest: cast each result to its concrete shape and decode the stream name and message id to str at the dispatch boundary via a _to_str helper. mypy roboco/ is now clean cold (247 files) against redis's real types; events suite green. * Updated uv.lock * fix(workspace): install the dev extra so agents can run make quality Agent workspaces were set up with plain `uv sync`, which installs only the project's default dependency group (pytest) — not the `dev` *extra* where the gate tools live (ruff, mypy, xenon, radon, vulture, bandit, deptry). So an agent's .venv had pytest but no linters, and `make quality` died immediately on `ruff: command not found`. Agents literally could not lint, type-check, or complexity-check their own work, which is how format/mypy/xenon debt merged unseen. Sync the `dev` extra (`uv sync --extra dev`) so the workspace gets the full toolchain the setup's own docstring already promised. * fix(panel): rate-limit endpoint shape + websocket path Two panel-facing breakages from the rate-limit rework: - GET /api/system/rate-limits returned a raw list, but the panel store reads response.entries — so `r.entries is not iterable` crashed the banner sync on page load. Return the panel's contract: a { entries: [...] } envelope whose items are camelCase {provider, affectedAgents, hitAt, resumeAt, retryAfterSeconds}, derived from the raw Redis state (resumeAt = hitAt + retryAfter). - The rate-limit websocket hook passed "/ws/system" while getWebSocketUrl() already supplies the "/ws" base, producing the doubled "/ws/ws/system" URL. Pass "/system" to match the agents/channels/notifications hooks. Note: the backend /ws/system endpoint itself does not yet exist (the rework shipped the panel hook only); the REST fix keeps the banner correct on load and reconnect until that endpoint is built. * test(workspace): assert uv sync installs the dev extra Follow the workspace setup change: the dependency-install command is now `uv sync --extra dev` so the agent workspace gets the lint/type/complexity toolchain. Update the three assertions that pinned the old `uv sync`. * feat(ws): add /ws/system stream and bridge rate-limit events to the panel The rate-limit rework shipped the panel's websocket hook but no backend: there was no /ws/system endpoint and nothing forwarded RATE_LIMIT_HIT/LIFTED to a socket, so the banner got no live updates. Build the missing half: - ConnectionManager grows a system-wide connection set with connect_system / broadcast_system, and disconnect() now clears it. - A /ws/system websocket endpoint (operator stream, no per-agent keying) with the same connected + ping/pong lifecycle as the other streams. - websocket_bridge subscribes RATE_LIMIT_HIT/LIFTED and forwards each to broadcast_system tagged with the type the panel switches on. Both events ride the same StreamEventBus singleton, and the subscriptions register before start_listening(), so the consumer reads their streams. Pairs with the panel hook now passing '/system' (getWebSocketUrl supplies the '/ws' base). Covered by handler, manager, and endpoint-lifecycle tests. --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>	2026-06-11 18:16:20 +02:00
5 people	98e618c243	[aaac85d2] Rate limit guardrails for Anthropic and Ollama providers (#104) ... * [25aa5b24] Implement rate-limit Zustand store, Axios interceptor, WebSocket hook, banner component, and page-load sync (#99) (#101) * [25aa5b24] feat(rate-limits): add types, Zustand store, Axios 429 interceptor, WS hook, sync hook, and banner component - panel/src/types/rate-limits.ts: RateLimitEntry, RateLimitHitEvent, RateLimitLiftedEvent, RateLimitApiResponse - panel/src/store/rate-limit-store.ts: useRateLimitStore with Map state, hitRateLimit/liftRateLimit/syncFromApi - panel/src/lib/api/rate-limits.ts: GET /api/system/rate-limits with isMockMode guard - panel/src/lib/api/client.ts: 429 interceptor dispatches to store first, Sonner toast on retry exhaustion - panel/src/hooks/use-rate-limit-websocket.ts: RATE_LIMIT_HIT/LIFTED events + onReconnect callback - panel/src/hooks/use-rate-limit-sync.ts: mount sync + no-op with console.warn when endpoint unavailable - panel/src/components/rate-limit/rate-limit-banner.tsx: amber rows with countdown, no dismiss button - panel/src/app/(dashboard)/layout.tsx: RateLimitBanner mounted below Header - store/index.ts, hooks/index.ts: export new store and hooks * [25aa5b24] fix(rate-limit-banner): use lint-clean countdown pattern (computeSecondsLeft outside render) * [25aa5b24] fix(client): add real retry loop to 429 interceptor so Sonner toast fires on exhaustion - Increment error.config._retryCount and return api(error.config) when retryCount < RATE_LIMIT_MAX_RETRIES, actually retrying the request. - Toast fires only when retryCount >= RATE_LIMIT_MAX_RETRIES (3 attempts). - Fixes AC4: toast was dead code because without return api(error.config) every 429 saw retryCount=1, permanently below the threshold of 3. --------- Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev> * [4112cd34] feat(rate-limit): add RateLimitError with 5-retry exponential backoff at all LLM call sites (#102) (#103) - Create roboco/services/exceptions.py with RateLimitError(provider, retry_after), HTTP_TOO_MANY_REQUESTS, MAX_RATE_LIMIT_RETRIES constants, and parse_retry_after_header() helper - extraction.py: extract _call_anthropic_with_retry() helper; retry Anthropic call 5x on 429 with exponential backoff; re-raise RateLimitError from outer except instead of swallowing it - ollama_embedder.py: 5-retry outer loop (429) wrapping existing 3-retry inner loop (ConnectError/Timeout) for all 4 call sites; two concerns kept isolated - indexes/base.py, mentor.py, validator.py: replace magic 429 literals with HTTP_TOO_MANY_REQUESTS; 5-retry loop on 429 for LLM calls - middleware.py: add rate_limit_exception_handler returning HTTP 429 with Retry-After response header - tests/unit/services/test_rate_limit_retry.py: 28 tests covering exhaustion, Retry-After header sleep, partial retries then success, ConnectError isolation Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> * [18107054] feat(rate-limit): Redis rate-limit state tracker + i_am_blocked rate_limited path (#105) (#106) - Add RateLimitStateTracker in roboco/services/gateway/rate_limit_tracker.py with activate(), clear(), is_rate_limited(), get_state(), increment_probe_failures(), reset_probe_failures() backed by redis.asyncio - Add RATE_LIMIT_HIT = "rate_limit.hit" to EventType StrEnum in events.py - Add _handle_rate_limited_parking() to Choreographer: intercepts i_am_blocked(reason='rate_limited') before block state transition, parks all active agents sharing affected provider via mark_waiting_long, publishes RATE_LIMIT_HIT event to StreamEventBus, task stays in_progress - Add get_provider_for_agent() and get_active_agent_slugs_for_provider() helper methods to AgentOrchestrator - Wire orchestrator and stream_bus into ChoreographerDeps via deps.py - Add test_rate_limit_tracker.py (basic ops, probe failures, cross-reconnection persistence, provider isolation) and test_i_am_blocked_rate_limited.py (AC3/AC4/AC5 coverage: task stays in_progress, mark_waiting_long call count equals active agent count, RATE_LIMIT_HIT event payload structure) Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> * [5501e4b4] Wire RateLimitStateTracker into live orchestrator paths — 4 CEO-identified integration gaps (#109) * [8451ca50] feat(gateway): wire RateLimitStateTracker.activate() into i_am_blocked rate-limited path and add provider-rate-limit gate to decide_spawn() (#107) - Add provider/provider_rate_limited optional fields to TriggerContext (backward-compatible defaults) - Insert rule 2 in decide_spawn(): QUEUE when trigger.provider_rate_limited is True with reason 'provider X rate-limited' - Call RateLimitStateTracker(provider).activate() in _handle_rate_limited_parking() after mark_waiting_long loop (wrapped in contextlib.suppress for Redis fault tolerance) - Extend gateway_pre_spawn_check() with optional provider param; check RateLimitStateTracker.is_rate_limited() when provider is known - Pass provider=self.get_provider_for_agent(agent_id) from orchestrator call site - Add TestProviderRateLimitGate (6 tests) to test_trigger_filter.py - Add TestRateLimitTrackerActivateOnParking (6 tests) to test_i_am_blocked_rate_limited.py - All 38 unit tests pass; ruff and mypy clean on changed files Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> * [e9cef0f0] feat(rate-limits): sweeper probe loop, CEO notification, and GET /api/system/rate-limits endpoint (AC4, AC8, AC9) (#108) - Add RATE_LIMIT_LIFTED event type to EventType enum in models/events.py - Add RateLimitStateTracker.list_rate_limited_providers() classmethod to scan Redis for all currently rate-limited providers (used by the new endpoint) - Add orchestrator._rate_limit_probe_loop(): background task started/stopped in start()/stop(), runs _sweep_rate_limit_probes() every 30s - Add orchestrator._probe_one_provider(): checks estimated_lift_at gate, calls _do_probe(); on success: tracker.clear(), resolve_wait() for all parked agents with waiting_for='rate_limit_lifted' matching the provider, publishes RATE_LIMIT_LIFTED event; on failure: increments probe_failures counter, sends CEO notification at threshold 10 (once per episode via _rate_limit_ceo_notified) - Add orchestrator._make_tracker(): injectable factory for RateLimitStateTracker - Add orchestrator._do_probe(): overridable async bool probe (default: True) - Add orchestrator._notify_rate_limit_ceo(): high-priority notification to CEO containing provider name, duration since activation, and paused agent count - Add roboco/api/routes/system.py with GET /rate-limits endpoint (AC9) - Register system_router in app.py under /api/system prefix - Add 17 unit tests in tests/unit/runtime/test_rate_limit_sweep.py covering all AC4/AC8/AC9 paths: probe success/failure, CEO threshold, endpoint schema Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> --------- Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> --------- Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev> Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev> Co-authored-by: Renn F <rennf93@users.noreply.github.com>	2026-06-11 09:21:26 +02:00