Files
roboco/roboco/api/deps.py
T
98e618c243 [aaac85d2] Rate limit guardrails for Anthropic and Ollama providers (#104)
* [25aa5b24] Implement rate-limit Zustand store, Axios interceptor, WebSocket hook, banner component, and page-load sync (#99) (#101)

* [25aa5b24] feat(rate-limits): add types, Zustand store, Axios 429 interceptor, WS hook, sync hook, and banner component

- panel/src/types/rate-limits.ts: RateLimitEntry, RateLimitHitEvent, RateLimitLiftedEvent, RateLimitApiResponse
- panel/src/store/rate-limit-store.ts: useRateLimitStore with Map state, hitRateLimit/liftRateLimit/syncFromApi
- panel/src/lib/api/rate-limits.ts: GET /api/system/rate-limits with isMockMode guard
- panel/src/lib/api/client.ts: 429 interceptor dispatches to store first, Sonner toast on retry exhaustion
- panel/src/hooks/use-rate-limit-websocket.ts: RATE_LIMIT_HIT/LIFTED events + onReconnect callback
- panel/src/hooks/use-rate-limit-sync.ts: mount sync + no-op with console.warn when endpoint unavailable
- panel/src/components/rate-limit/rate-limit-banner.tsx: amber rows with countdown, no dismiss button
- panel/src/app/(dashboard)/layout.tsx: RateLimitBanner mounted below Header
- store/index.ts, hooks/index.ts: export new store and hooks

* [25aa5b24] fix(rate-limit-banner): use lint-clean countdown pattern (computeSecondsLeft outside render)

* [25aa5b24] fix(client): add real retry loop to 429 interceptor so Sonner toast fires on exhaustion

- Increment error.config._retryCount and return api(error.config) when
  retryCount < RATE_LIMIT_MAX_RETRIES, actually retrying the request.
- Toast fires only when retryCount >= RATE_LIMIT_MAX_RETRIES (3 attempts).
- Fixes AC4: toast was dead code because without return api(error.config)
  every 429 saw retryCount=1, permanently below the threshold of 3.

---------

Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev>

* [4112cd34] feat(rate-limit): add RateLimitError with 5-retry exponential backoff at all LLM call sites (#102) (#103)

- Create roboco/services/exceptions.py with RateLimitError(provider, retry_after),
  HTTP_TOO_MANY_REQUESTS, MAX_RATE_LIMIT_RETRIES constants, and
  parse_retry_after_header() helper
- extraction.py: extract _call_anthropic_with_retry() helper; retry Anthropic
  call 5x on 429 with exponential backoff; re-raise RateLimitError from outer
  except instead of swallowing it
- ollama_embedder.py: 5-retry outer loop (429) wrapping existing 3-retry inner
  loop (ConnectError/Timeout) for all 4 call sites; two concerns kept isolated
- indexes/base.py, mentor.py, validator.py: replace magic 429 literals with
  HTTP_TOO_MANY_REQUESTS; 5-retry loop on 429 for LLM calls
- middleware.py: add rate_limit_exception_handler returning HTTP 429 with
  Retry-After response header
- tests/unit/services/test_rate_limit_retry.py: 28 tests covering exhaustion,
  Retry-After header sleep, partial retries then success, ConnectError isolation

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

* [18107054] feat(rate-limit): Redis rate-limit state tracker + i_am_blocked rate_limited path (#105) (#106)

- Add RateLimitStateTracker in roboco/services/gateway/rate_limit_tracker.py
  with activate(), clear(), is_rate_limited(), get_state(),
  increment_probe_failures(), reset_probe_failures() backed by redis.asyncio
- Add RATE_LIMIT_HIT = "rate_limit.hit" to EventType StrEnum in events.py
- Add _handle_rate_limited_parking() to Choreographer: intercepts
  i_am_blocked(reason='rate_limited') before block state transition,
  parks all active agents sharing affected provider via mark_waiting_long,
  publishes RATE_LIMIT_HIT event to StreamEventBus, task stays in_progress
- Add get_provider_for_agent() and get_active_agent_slugs_for_provider()
  helper methods to AgentOrchestrator
- Wire orchestrator and stream_bus into ChoreographerDeps via deps.py
- Add test_rate_limit_tracker.py (basic ops, probe failures, cross-reconnection
  persistence, provider isolation) and test_i_am_blocked_rate_limited.py
  (AC3/AC4/AC5 coverage: task stays in_progress, mark_waiting_long call count
  equals active agent count, RATE_LIMIT_HIT event payload structure)

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

* [5501e4b4] Wire RateLimitStateTracker into live orchestrator paths — 4 CEO-identified integration gaps (#109)

* [8451ca50] feat(gateway): wire RateLimitStateTracker.activate() into i_am_blocked rate-limited path and add provider-rate-limit gate to decide_spawn() (#107)

- Add provider/provider_rate_limited optional fields to TriggerContext (backward-compatible defaults)
- Insert rule 2 in decide_spawn(): QUEUE when trigger.provider_rate_limited is True with reason 'provider X rate-limited'
- Call RateLimitStateTracker(provider).activate() in _handle_rate_limited_parking() after mark_waiting_long loop (wrapped in contextlib.suppress for Redis fault tolerance)
- Extend gateway_pre_spawn_check() with optional provider param; check RateLimitStateTracker.is_rate_limited() when provider is known
- Pass provider=self.get_provider_for_agent(agent_id) from orchestrator call site
- Add TestProviderRateLimitGate (6 tests) to test_trigger_filter.py
- Add TestRateLimitTrackerActivateOnParking (6 tests) to test_i_am_blocked_rate_limited.py
- All 38 unit tests pass; ruff and mypy clean on changed files

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

* [e9cef0f0] feat(rate-limits): sweeper probe loop, CEO notification, and GET /api/system/rate-limits endpoint (AC4, AC8, AC9) (#108)

- Add RATE_LIMIT_LIFTED event type to EventType enum in models/events.py
- Add RateLimitStateTracker.list_rate_limited_providers() classmethod to scan
  Redis for all currently rate-limited providers (used by the new endpoint)
- Add orchestrator._rate_limit_probe_loop(): background task started/stopped in
  start()/stop(), runs _sweep_rate_limit_probes() every 30s
- Add orchestrator._probe_one_provider(): checks estimated_lift_at gate, calls
  _do_probe(); on success: tracker.clear(), resolve_wait() for all parked agents
  with waiting_for='rate_limit_lifted' matching the provider, publishes
  RATE_LIMIT_LIFTED event; on failure: increments probe_failures counter, sends
  CEO notification at threshold 10 (once per episode via _rate_limit_ceo_notified)
- Add orchestrator._make_tracker(): injectable factory for RateLimitStateTracker
- Add orchestrator._do_probe(): overridable async bool probe (default: True)
- Add orchestrator._notify_rate_limit_ceo(): high-priority notification to CEO
  containing provider name, duration since activation, and paused agent count
- Add roboco/api/routes/system.py with GET /rate-limits endpoint (AC9)
- Register system_router in app.py under /api/system prefix
- Add 17 unit tests in tests/unit/runtime/test_rate_limit_sweep.py covering
  all AC4/AC8/AC9 paths: probe success/failure, CEO threshold, endpoint schema

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

---------

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

---------

Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev>
Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-06-11 09:21:26 +02:00

569 lines
19 KiB
Python

"""
API Dependencies
Shared dependencies for FastAPI routes.
"""
from __future__ import annotations
import contextlib
import os
from typing import TYPE_CHECKING, Annotated, Any
from uuid import UUID
from fastapi import Depends, Header, HTTPException, status
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from roboco.agents_config import verify_agent_token
from roboco.api.schemas.optimal import PaginationParams
from roboco.db.base import get_db
from roboco.db.tables import AgentTable
from roboco.foundation.identity import BOARD_ROLES, DEV_ROLES, PM_ROLES, Role
from roboco.models import AgentRole, Team
from roboco.runtime import AgentOrchestrator
from roboco.services.a2a import A2AService
from roboco.services.audit import get_audit_service
from roboco.services.gateway.choreographer import Choreographer, ChoreographerDeps
from roboco.services.gateway.content_actions import ContentActions, ContentActionsDeps
from roboco.services.gateway.evidence_repo import EvidenceRepo
from roboco.services.git import GitService
from roboco.services.journal import JournalService
from roboco.services.messaging import MessagingService
from roboco.services.notification import NotificationService
from roboco.services.notification_delivery import NotificationDeliveryService
from roboco.services.permissions import AgentContext, PermissionService
from roboco.services.product import ProductService
from roboco.services.repositories import resolve_agent_identity, resolve_agent_uuid
from roboco.services.task import TaskService
from roboco.services.work_session import WorkSessionService
from roboco.services.workspace import WorkspaceService
if TYPE_CHECKING:
from collections.abc import Callable, Coroutine
# Type alias for database session dependency
DbSession = Annotated[AsyncSession, Depends(get_db)]
async def resolve_agent_id(agent_id_str: str, db: AsyncSession) -> UUID:
"""
Resolve agent ID from string (UUID or slug).
Args:
agent_id_str: Either a UUID string or agent slug (e.g., "be-dev-1")
db: Database session
Returns:
UUID of the agent
Raises:
HTTPException: If agent not found or invalid format
"""
result = await resolve_agent_uuid(db, agent_id_str)
if result is None:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f"Agent not found: {agent_id_str}",
)
return result
class _ServiceHolder:
"""Holder for singleton service instances."""
permission_service: PermissionService | None = None
orchestrator: AgentOrchestrator | None = None
def get_permission_service() -> PermissionService:
"""Get or create the permission service singleton."""
if _ServiceHolder.permission_service is None:
_ServiceHolder.permission_service = PermissionService()
return _ServiceHolder.permission_service
PermissionServiceDep = Annotated[PermissionService, Depends(get_permission_service)]
def set_orchestrator(orchestrator: AgentOrchestrator) -> None:
"""Set the global orchestrator instance."""
_ServiceHolder.orchestrator = orchestrator
def get_orchestrator() -> AgentOrchestrator:
"""Get the global orchestrator instance."""
if _ServiceHolder.orchestrator is None:
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail="Orchestrator not initialized",
)
return _ServiceHolder.orchestrator
OrchestratorDep = Annotated[AgentOrchestrator, Depends(get_orchestrator)]
async def get_current_agent_id(
db: DbSession,
x_agent_id: Annotated[str | None, Header()] = None,
) -> UUID:
"""
Get the current agent ID from request headers.
Accepts either a UUID string or agent slug (e.g., "be-dev-1").
In production, this would validate a JWT token and extract the agent ID.
For now, we use a simple header-based approach for development.
Args:
x_agent_id: Agent ID (UUID or slug) from X-Agent-ID header
db: Database session for slug resolution
Returns:
UUID of the current agent
Raises:
HTTPException: If agent ID is missing or invalid/not found
"""
if not x_agent_id:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Missing X-Agent-ID header",
)
return await resolve_agent_id(x_agent_id, db)
# Type alias for current agent dependency
CurrentAgentId = Annotated[UUID, Depends(get_current_agent_id)]
async def get_current_agent_slug(
x_agent_id: Annotated[str | None, Header()] = None,
) -> str:
"""
Get the current agent slug from request headers.
Unlike get_current_agent_id, this returns the slug directly without
resolving to UUID. Useful for A2A where we work with agent slugs.
Args:
x_agent_id: Agent slug from X-Agent-ID header
Returns:
Agent slug string
Raises:
HTTPException: If agent ID header is missing
"""
if not x_agent_id:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Missing X-Agent-ID header",
)
return x_agent_id
# Type alias for agent slug dependency
CurrentAgentSlug = Annotated[str, Depends(get_current_agent_slug)]
async def get_optional_agent_id(
db: DbSession,
x_agent_id: Annotated[str | None, Header()] = None,
) -> UUID | None:
"""
Get the current agent ID if provided.
Accepts either a UUID string or agent slug (e.g., "be-dev-1").
Unlike get_current_agent_id, this doesn't raise an error if missing.
"""
if not x_agent_id:
return None
try:
return await resolve_agent_id(x_agent_id, db)
except HTTPException:
return None
OptionalAgentId = Annotated[UUID | None, Depends(get_optional_agent_id)]
def _auth_required() -> bool:
"""True when agent HMAC auth is mandatory (prod-ish) vs opt-in (dev)."""
val = os.environ.get("ROBOCO_AGENT_AUTH_REQUIRED", "").strip().lower()
return val in ("1", "true", "yes")
def _check_agent_auth_token(
x_agent_id: str,
x_agent_role: str,
x_agent_team: str | None,
x_agent_token: str | None,
) -> None:
"""Enforce HMAC token when required; reject invalid tokens even in dev."""
# Token verification: stops an agent on the Docker network from
# spoofing another agent's role by setting headers directly. When
# ROBOCO_AGENT_AUTH_REQUIRED is true, every request must carry a
# token matching HMAC(id:role:team, secret). In dev it's optional
# (so the panel / curl-for-debugging keep working), but any token
# that IS presented is still verified — you can't bypass by
# supplying an invalid token.
if _auth_required() and not x_agent_token:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Missing X-Agent-Token header (auth required)",
)
if x_agent_token and not verify_agent_token(
x_agent_token,
x_agent_id,
x_agent_role,
x_agent_team or "",
):
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail=(
"Invalid X-Agent-Token — signature mismatch. Header "
"values do not match the token issued for this agent."
),
)
async def _resolve_agent_identity(
db: DbSession, x_agent_id: str, x_agent_role: str
) -> tuple[UUID, str]:
"""Return (agent_id, slug), handling the special `system` role."""
if x_agent_role.lower() == "system":
try:
return UUID(x_agent_id), "system"
except ValueError as e:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f"Invalid system agent UUID: {x_agent_id}",
) from e
identity = await resolve_agent_identity(db, x_agent_id)
if identity is None:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f"Agent not found: {x_agent_id}",
)
return identity
async def _coerce_agent_role(
db: DbSession, x_agent_role: str, agent_id: UUID, x_agent_id: str
) -> AgentRole:
"""Parse the role header; fall back to the DB role if it's a slug."""
try:
return AgentRole(x_agent_role.lower())
except ValueError:
# Panel/clients sometimes pass the agent slug (e.g. "main-pm") instead
# of the role value ("main_pm"). If the header isn't a valid enum
# value, fall back to the authoritative role on the agent row we
# already resolved above.
role_row = await db.execute(
select(AgentTable.role).where(AgentTable.id == agent_id)
)
db_role = role_row.scalar_one_or_none()
if db_role is None:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=(
f"Invalid agent role '{x_agent_role}' and no role on "
f"record for agent {x_agent_id}"
),
) from None
return db_role
def _coerce_agent_team(x_agent_team: str | None) -> Team | None:
"""Parse the team header; return None for empty/invalid values."""
if not x_agent_team:
return None
with contextlib.suppress(ValueError):
return Team(x_agent_team.lower())
return None
async def get_agent_context(
db: DbSession,
x_agent_id: Annotated[str | None, Header()] = None,
x_agent_role: Annotated[str | None, Header()] = None,
x_agent_team: Annotated[str | None, Header()] = None,
x_agent_token: Annotated[str | None, Header()] = None,
) -> AgentContext:
"""
Get the current agent context from request headers.
Headers:
X-Agent-ID: UUID or slug of the agent (e.g., "be-dev-1")
X-Agent-Role: Role (e.g., 'developer', 'cell_pm')
X-Agent-Team: (optional) Team (e.g., 'backend', 'frontend')
X-Agent-Token: (required when ROBOCO_AGENT_AUTH_REQUIRED=true)
HMAC of "agent_id:role:team" signed with
ROBOCO_AGENT_AUTH_SECRET. Orchestrator issues this at spawn.
"""
if not x_agent_id:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Missing X-Agent-ID header",
)
if not x_agent_role:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Missing X-Agent-Role header",
)
_check_agent_auth_token(x_agent_id, x_agent_role, x_agent_team, x_agent_token)
agent_id, slug = await _resolve_agent_identity(db, x_agent_id, x_agent_role)
role = await _coerce_agent_role(db, x_agent_role, agent_id, x_agent_id)
team = _coerce_agent_team(x_agent_team)
return AgentContext(
agent_id=agent_id,
role=role,
team=team,
slug=slug,
)
CurrentAgentContext = Annotated[AgentContext, Depends(get_agent_context)]
# =============================================================================
# ROLE-GATE HELPERS
#
# Small HTTP-layer guards for routes that need a coarse "PM or above" /
# "developer or above" check. They raise HTTPException directly because
# the check IS the HTTP authorization decision — no service-side logic,
# no translation layer needed.
# =============================================================================
# Role-sets derive from foundation so renaming a role lives in one file.
# HEAD_MARKETING is intentionally excluded from every "above" set — the role is
# a marketing spokesperson, not a workflow approver. StrEnum membership means
# the sets compare equal against both Role.* and the lowercase header string.
_PM_OR_ABOVE_ROLES: frozenset[Role] = (
PM_ROLES | (BOARD_ROLES - {Role.HEAD_MARKETING}) | {Role.CEO}
)
_DEVELOPER_OR_ABOVE_ROLES: frozenset[Role] = DEV_ROLES | _PM_OR_ABOVE_ROLES
def _role_value(role: Any) -> str:
"""AgentRole or str → plain string for set membership checks."""
return role.value if hasattr(role, "value") else str(role)
def require_pm_or_above(role: Any, action: str) -> None:
"""Raise 403 unless caller is PM-or-above (cell_pm/main_pm/board/CEO)."""
if _role_value(role) not in _PM_OR_ABOVE_ROLES:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=f"Only PMs and management can {action}",
)
def require_developer_or_above(role: Any, action: str) -> None:
"""Raise 403 unless caller is developer-or-above."""
if _role_value(role) not in _DEVELOPER_OR_ABOVE_ROLES:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=f"Only developers and above can {action}",
)
_GLOBAL_CELL_ACCESS_ROLES: frozenset[Role] = (BOARD_ROLES - {Role.HEAD_MARKETING}) | {
Role.MAIN_PM,
Role.CEO,
}
def require_cell_access(agent: AgentContext, cell: Team, action: str) -> None:
"""Raise 403 unless caller can act in the given cell.
Main PM, board, and CEO can act across all cells. Cell PMs and their
members are restricted to their own cell.
"""
if _role_value(agent.role) in _GLOBAL_CELL_ACCESS_ROLES:
return
if agent.team != cell:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=f"Cannot {action} projects in {cell.value} cell",
)
def require_channel_read(
channel_name: str,
) -> Callable[..., Coroutine[Any, Any, None]]:
"""
Dependency factory that requires read access to a channel.
Usage:
@router.get("/channels/{channel_id}/messages")
async def get_messages(
agent: CurrentAgentContext,
_: Annotated[None, Depends(require_channel_read("backend-cell"))],
):
...
"""
async def check_permission(
agent: CurrentAgentContext,
permissions: PermissionServiceDep,
) -> None:
if not permissions.can_read_channel(agent, channel_name):
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=f"No read access to channel: {channel_name}",
)
return check_permission
def require_channel_write(
channel_name: str,
) -> Callable[..., Coroutine[Any, Any, None]]:
"""
Dependency factory that requires write access to a channel.
"""
async def check_permission(
agent: CurrentAgentContext,
permissions: PermissionServiceDep,
) -> None:
if not permissions.can_write_channel(agent, channel_name):
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=f"No write access to channel: {channel_name}",
)
return check_permission
def require_notification_permission() -> Callable[..., Coroutine[Any, Any, None]]:
"""
Dependency that requires the agent can send notifications.
"""
async def check_permission(
agent: CurrentAgentContext,
permissions: PermissionServiceDep,
) -> None:
if not permissions.can_send_notifications(agent):
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="Not authorized to send notifications",
)
return check_permission
def require_task_action(
action: str, task_team: Team | None = None
) -> Callable[..., Coroutine[Any, Any, None]]:
"""
Dependency factory that requires permission for a task action.
Args:
action: The task action (from TaskAction constants)
task_team: Optional team context for team-specific checks
Usage:
@router.post("/tasks")
async def create_task(
agent: CurrentAgentContext,
_: Annotated[None, Depends(require_task_action("create"))],
):
...
"""
async def check_permission(
agent: CurrentAgentContext,
permissions: PermissionServiceDep,
) -> None:
if not permissions.can_perform_task_action(agent, action, task_team):
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=f"Not authorized to perform task action: {action}",
)
return check_permission
# =============================================================================
# GATEWAY / CHOREOGRAPHER DEPENDENCIES
# =============================================================================
async def get_choreographer(
db_session: DbSession,
) -> Choreographer:
"""Build a Choreographer with all service dependencies wired up."""
from roboco.events.stream_bus import get_stream_event_bus
# Inject the orchestrator (if initialised) and the stream event bus
# so the rate-limited i_am_blocked path can park agents and publish events.
# Both are None-safe in ChoreographerDeps — passing None is the same as
# omitting the field, so the choreographer degrades gracefully when the
# orchestrator has not been initialised yet (e.g. during startup).
orch: AgentOrchestrator | None = _ServiceHolder.orchestrator
bus = get_stream_event_bus() if _ServiceHolder.orchestrator is not None else None
return Choreographer(
ChoreographerDeps(
task=TaskService(db_session),
work_session=WorkSessionService(db_session),
git=GitService(db_session),
a2a=A2AService(db_session),
journal=JournalService(db_session),
audit=get_audit_service(),
evidence_repo=EvidenceRepo(db_session),
messaging=MessagingService(db_session),
product=ProductService(db_session),
orchestrator=orch,
stream_bus=bus,
)
)
async def get_content_actions(
db_session: DbSession,
) -> ContentActions:
"""Build a ContentActions with all service dependencies wired up."""
return ContentActions(
ContentActionsDeps(
task=TaskService(db_session),
git=GitService(db_session),
messaging=MessagingService(db_session),
a2a=A2AService(db_session),
journal=JournalService(db_session),
workspace=WorkspaceService(db_session),
notifications=NotificationService(),
notification_delivery=NotificationDeliveryService(db_session),
evidence_repo=EvidenceRepo(db_session),
)
)
# =============================================================================
# PAGINATION DEPENDENCIES
# =============================================================================
def get_pagination(
limit: int = 50,
offset: int = 0,
) -> PaginationParams:
"""Dependency for pagination parameters."""
# Enforce constraints
limit = max(1, min(100, limit))
offset = max(0, offset)
return PaginationParams(limit=limit, offset=offset)
PaginationDep = Annotated[PaginationParams, Depends(get_pagination)]