mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
303c2db289
* fix(rate-limit): real provider liveness probe instead of time-based stub
The rate-limit recovery sweeper cleared a provider and resumed parked agents
purely on elapsed time — _do_probe was a stub that always returned True once
the retry_after window passed, so it never confirmed the provider had actually
stopped rate-limiting us. Under a sustained limit that resumes agents straight
into another 429, re-parking them: avoidable churn.
Make the probe real. _do_probe now issues a free, unmetered liveness call —
Anthropic GET /v1/models or Ollama GET /api/tags — and treats any non-429
response as the limit having lifted. A 429 keeps the provider parked; a
network error keeps it parked too (retry next sweep). When the provider can't
be probed (no API key, or an unrecognized provider), it falls back to the
prior time-expiry optimism rather than stranding agents. _probe_target keeps
URL/header resolution separate and testable, and _do_probe stays a
monkeypatchable boundary so the existing sweep tests are unaffected.
Also drop two acceptance-criteria-number labels from comments in this file.
* chore(rate-limit): clear merged gate debt in rate-limit tests + deps lint
The rate-limit PR landed with ruff violations the full gate flags but the
authors' runs missed: test_rate_limit_sweep.py was unformatted, and
test_rate_limit_tracker.py had unsorted/unused imports and magic-value
comparisons. Format the sweep test, drop the dead imports, and bind the
magic comparison values to locals. Also strip acceptance-criteria-number
labels from comments/docstrings across the three rate-limit test files
(leaving genuine acceptance_criteria=[...] test data untouched), and add
api/deps.py to the PLC0415 per-file-ignore — it is the DI wiring hub and
defers a couple of service imports to call time to avoid import cycles,
the same rationale already applied to api/routes, runtime, and services.
* fix(rate-limit): resolve redis type errors in RateLimitStateTracker
A cold mypy run (the gate's true state — prior passes were warm-cache only)
flagged four redis-typing errors in rate_limit_tracker.py that the merge
missed: three unused type:ignore[type-arg] on redis.Redis, and an
aclose() the bundled redis type stub doesn't expose.
Drop the now-unused ignores, and close the scan client via
'async with redis.from_url(...) as r:' instead of a finally-block
aclose(). The context manager closes the client on exit using the modern
redis.asyncio API — no deprecated close(), no stub-missing aclose(), no
suppression. Extend the test's redis mock to model the async
context-manager protocol so it returns itself on enter.
* test(prompter): pass route='main_pm' in the product main-PM routing test
Pre-existing master failure, unrelated to the rate-limit work. The test is
named ...product_routes_to_main_pm and asserts team=MAIN_PM, but called
confirm_live_draft without a route, so it got the 'board' default — which
assigns the Product Owner and yields team=BOARD by design (the board-review
path keeps the root at team=board until the CEO approves). The Main-PM path
is selected with route='main_pm', exactly as the sibling
...main_pm_route_assigns_main_pm test does. Add the missing kwarg so the test
verifies the path it names; behaviour under test is unchanged.
* Updated uv.lock
* refactor(complexity): bring all rank-C blocks under the xenon B ceiling
The full quality gate's xenon step (--max-absolute B --max-modules A
--max-average A) failed on eight rank-C blocks plus the extraction module
average — debt the rate-limit and token-analytics merges deferred. Reduce
each by extracting cohesive helpers, behaviour unchanged:
- orchestrator._probe_one_provider: split into _too_early_to_probe,
_on_probe_success, _on_probe_failure, _parked_agents_for.
- rate_limit_tracker.list_rate_limited_providers: extract _read_rate_limited_entry
and a _decode helper.
- trigger_filter.decide_spawn: extract _stale_trigger_decision (drops the
PLR0911 suppression too).
- ollama_embedder (embed_query, _embed_batch_sync, aembed_query,
_embed_batch_async): share _rl_backoff / _map_embed_error / _log_429 /
_sleep_connect_retry / _asleep_connect_retry; remove a dead post-loop guard
in aembed_query.
- mentor._synthesize_answer: extract _select_system_prompt and
_answer_from_response.
- indexes/base.ask: extract the 429-retried LLM call into _ask_llm.
- extraction.__init__: extract _compile_patterns so the module average
lands at rank A.
xenon now exits 0; rate-limit, optimal_brain, extraction, and events suites
all green.
* chore(deps): drop obsolete types-redis stub; honor redis 8.0 inline types
types-redis 4.6 (typed for redis 4.x) shadowed redis 8.0's own inline types,
which both masked real annotation mismatches in stream_bus.py and forced
awkward workarounds elsewhere. The stale stub is why the mypy gate only ever
passed warm-cached: a cold run under the wrong stub disagreed with the code.
Remove types-redis (and its orphaned transitive stubs) so mypy uses redis's
shipped types. That surfaces that xreadgroup/xclaim return bytes-keyed records
while _handle_message is annotated str — the code already decodes bytes
defensively, so this is an annotation gap, not a runtime bug. Make the types
honest: cast each result to its concrete shape and decode the stream name and
message id to str at the dispatch boundary via a _to_str helper.
mypy roboco/ is now clean cold (247 files) against redis's real types; events
suite green.
* Updated uv.lock
* fix(workspace): install the dev extra so agents can run make quality
Agent workspaces were set up with plain `uv sync`, which installs only the
project's default dependency group (pytest) — not the `dev` *extra* where the
gate tools live (ruff, mypy, xenon, radon, vulture, bandit, deptry). So an
agent's .venv had pytest but no linters, and `make quality` died immediately
on `ruff: command not found`. Agents literally could not lint, type-check, or
complexity-check their own work, which is how format/mypy/xenon debt merged
unseen. Sync the `dev` extra (`uv sync --extra dev`) so the workspace gets the
full toolchain the setup's own docstring already promised.
* fix(panel): rate-limit endpoint shape + websocket path
Two panel-facing breakages from the rate-limit rework:
- GET /api/system/rate-limits returned a raw list, but the panel store reads
response.entries — so `r.entries is not iterable` crashed the banner sync on
page load. Return the panel's contract: a { entries: [...] } envelope whose
items are camelCase {provider, affectedAgents, hitAt, resumeAt,
retryAfterSeconds}, derived from the raw Redis state (resumeAt = hitAt +
retryAfter).
- The rate-limit websocket hook passed "/ws/system" while getWebSocketUrl()
already supplies the "/ws" base, producing the doubled "/ws/ws/system" URL.
Pass "/system" to match the agents/channels/notifications hooks.
Note: the backend /ws/system endpoint itself does not yet exist (the rework
shipped the panel hook only); the REST fix keeps the banner correct on load
and reconnect until that endpoint is built.
* test(workspace): assert uv sync installs the dev extra
Follow the workspace setup change: the dependency-install command is now
`uv sync --extra dev` so the agent workspace gets the lint/type/complexity
toolchain. Update the three assertions that pinned the old `uv sync`.
* feat(ws): add /ws/system stream and bridge rate-limit events to the panel
The rate-limit rework shipped the panel's websocket hook but no backend: there
was no /ws/system endpoint and nothing forwarded RATE_LIMIT_HIT/LIFTED to a
socket, so the banner got no live updates.
Build the missing half:
- ConnectionManager grows a system-wide connection set with connect_system /
broadcast_system, and disconnect() now clears it.
- A /ws/system websocket endpoint (operator stream, no per-agent keying) with
the same connected + ping/pong lifecycle as the other streams.
- websocket_bridge subscribes RATE_LIMIT_HIT/LIFTED and forwards each to
broadcast_system tagged with the type the panel switches on. Both events
ride the same StreamEventBus singleton, and the subscriptions register
before start_listening(), so the consumer reads their streams.
Pairs with the panel hook now passing '/system' (getWebSocketUrl supplies the
'/ws' base). Covered by handler, manager, and endpoint-lifecycle tests.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
508 lines
19 KiB
Python
508 lines
19 KiB
Python
"""Unit tests for the rate-limited path in Choreographer.i_am_blocked.
|
|
|
|
Behaviours verified here:
|
|
- i_am_blocked(reason='rate_limited') calls RateLimitStateTracker.activate()
|
|
and stores affected agent IDs; all active agents on the rate-limited
|
|
provider are subsequently marked waiting-long.
|
|
- POST /v1/i_am_blocked with reason='rate_limited' does NOT transition the
|
|
task to 'blocked'; the task remains in its current status (in_progress) and
|
|
the calling agent is parked via mark_waiting_long(waiting_for='rate_limit_lifted').
|
|
- mark_waiting_long is called for every orchestrator-tracked active agent
|
|
sharing the affected provider — call count equals active agent count.
|
|
- A RATE_LIMIT_HIT event is published to the StreamEventBus with fields
|
|
provider, affectedAgents, retryAfterSeconds, and timestamp.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from unittest.mock import AsyncMock, MagicMock, patch
|
|
from uuid import uuid4
|
|
|
|
from roboco.models.events import EventType
|
|
from roboco.services.gateway.choreographer import Choreographer, ChoreographerDeps
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Helpers
|
|
# ---------------------------------------------------------------------------
|
|
|
|
_ACTIVE_AGENTS = ["be-dev-1", "be-dev-2", "be-qa"]
|
|
_PROVIDER = "anthropic"
|
|
|
|
|
|
def _make_evidence_repo() -> AsyncMock:
|
|
repo = AsyncMock()
|
|
for method in (
|
|
"list_unread_a2a",
|
|
"list_unread_mentions",
|
|
"list_pending_notifications",
|
|
"task_metadata_gaps",
|
|
"recent_team_activity",
|
|
"blockers_in_lane",
|
|
"journal_highlights_for_task",
|
|
):
|
|
getattr(repo, method).return_value = []
|
|
return repo
|
|
|
|
|
|
def _make_task_svc(agent_id: object, task_id: object) -> AsyncMock:
|
|
t = MagicMock(
|
|
id=task_id,
|
|
status="in_progress",
|
|
assigned_to=agent_id,
|
|
pre_block_state=None,
|
|
task_type="code",
|
|
team="backend",
|
|
# Avoid issues with spec iteration in claim guards
|
|
dependency_ids=[],
|
|
# acceptance_criteria needed by some paths
|
|
acceptance_criteria=[],
|
|
quick_context=None,
|
|
)
|
|
task_svc = AsyncMock()
|
|
task_svc.session = MagicMock()
|
|
task_svc.session.begin_nested = MagicMock(
|
|
return_value=MagicMock(
|
|
__aenter__=AsyncMock(return_value=None),
|
|
__aexit__=AsyncMock(return_value=False),
|
|
)
|
|
)
|
|
task_svc.get.return_value = t
|
|
task_svc.agent_for.return_value = MagicMock(
|
|
id=agent_id,
|
|
role="developer",
|
|
team="backend",
|
|
slug="be-dev-1", # calling agent's slug
|
|
)
|
|
return task_svc
|
|
|
|
|
|
def _make_orchestrator(
|
|
active_agents: list[str] | None = None,
|
|
provider: str = _PROVIDER,
|
|
) -> MagicMock:
|
|
"""Build a synchronous/async orchestrator mock."""
|
|
agents = active_agents if active_agents is not None else _ACTIVE_AGENTS
|
|
orch = MagicMock()
|
|
orch.get_provider_for_agent = MagicMock(return_value=provider)
|
|
orch.get_active_agent_slugs_for_provider = MagicMock(return_value=agents)
|
|
orch.mark_waiting_long = AsyncMock(return_value=None)
|
|
return orch
|
|
|
|
|
|
def _make_stream_bus() -> AsyncMock:
|
|
bus = AsyncMock()
|
|
bus.publish = AsyncMock(return_value="msg-id-1")
|
|
return bus
|
|
|
|
|
|
def _make_deps(
|
|
agent_id: object,
|
|
task_id: object,
|
|
orchestrator: MagicMock | None = None,
|
|
stream_bus: AsyncMock | None = None,
|
|
) -> ChoreographerDeps:
|
|
return ChoreographerDeps(
|
|
task=_make_task_svc(agent_id, task_id),
|
|
work_session=AsyncMock(),
|
|
git=AsyncMock(),
|
|
a2a=AsyncMock(),
|
|
journal=AsyncMock(),
|
|
audit=AsyncMock(),
|
|
evidence_repo=_make_evidence_repo(),
|
|
orchestrator=orchestrator,
|
|
stream_bus=stream_bus,
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Task stays in in_progress, agent parked via mark_waiting_long
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestRateLimitedDoesNotBlockTask:
|
|
async def test_task_status_remains_in_progress(self) -> None:
|
|
"""reason='rate_limited' must NOT transition the task to 'blocked'."""
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
orch = _make_orchestrator()
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch)
|
|
c = Choreographer(deps)
|
|
|
|
env = await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
assert env.error is None
|
|
assert env.status == "in_progress"
|
|
|
|
async def test_verb_runner_block_action_not_called(self) -> None:
|
|
"""The `block` action (task.escalate) must NOT run on rate_limited path."""
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
deps = _make_deps(agent_id, task_id)
|
|
c = Choreographer(deps)
|
|
|
|
await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
# The VerbRunner calls task.escalate for the normal block path.
|
|
# In the rate-limited path this must NOT happen.
|
|
deps.task.escalate.assert_not_awaited()
|
|
|
|
async def test_calling_agent_parked_via_mark_waiting_long(self) -> None:
|
|
"""mark_waiting_long must be called with waiting_for='rate_limit_lifted'."""
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
orch = _make_orchestrator(active_agents=["be-dev-1"])
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch)
|
|
c = Choreographer(deps)
|
|
|
|
await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
# Verify that at least one mark_waiting_long call uses the right reason.
|
|
# The implementation calls mark_waiting_long(slug, waiting_for=..., ...)
|
|
# so waiting_for is always a keyword argument.
|
|
waiting_for_values = [
|
|
c.kwargs.get("waiting_for") for c in orch.mark_waiting_long.call_args_list
|
|
]
|
|
assert "rate_limit_lifted" in waiting_for_values
|
|
|
|
async def test_case_insensitive_reason_match(self) -> None:
|
|
"""reason='Rate_Limited' (any case) should trigger the special path."""
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
orch = _make_orchestrator(active_agents=["be-dev-1"])
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch)
|
|
c = Choreographer(deps)
|
|
|
|
env = await c.i_am_blocked(agent_id, task_id, "Rate_Limited")
|
|
|
|
assert env.error is None
|
|
assert env.status == "in_progress"
|
|
|
|
async def test_struggle_journal_still_written(self) -> None:
|
|
"""journal.write_struggle must still be written on the rate_limited path."""
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
deps = _make_deps(agent_id, task_id)
|
|
c = Choreographer(deps)
|
|
|
|
await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
deps.journal.write_struggle.assert_awaited_once()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# mark_waiting_long called for every active agent on affected provider
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestMarkWaitingLongCallCount:
|
|
async def test_call_count_equals_active_agent_count(self) -> None:
|
|
"""mark_waiting_long must be called once per active agent."""
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
active = ["be-dev-1", "be-dev-2", "be-dev-3"]
|
|
orch = _make_orchestrator(active_agents=active)
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch)
|
|
c = Choreographer(deps)
|
|
|
|
await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
assert orch.mark_waiting_long.call_count == len(active)
|
|
|
|
async def test_call_count_with_single_active_agent(self) -> None:
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
orch = _make_orchestrator(active_agents=["be-dev-1"])
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch)
|
|
c = Choreographer(deps)
|
|
|
|
await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
assert orch.mark_waiting_long.call_count == 1
|
|
|
|
async def test_no_calls_when_no_active_agents(self) -> None:
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
orch = _make_orchestrator(active_agents=[])
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch)
|
|
c = Choreographer(deps)
|
|
|
|
await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
assert orch.mark_waiting_long.call_count == 0
|
|
|
|
async def test_no_calls_when_orchestrator_is_none(self) -> None:
|
|
"""When orchestrator is not wired in, no parking happens but no crash."""
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
deps = _make_deps(agent_id, task_id, orchestrator=None)
|
|
c = Choreographer(deps)
|
|
|
|
env = await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
# Should still succeed; no orchestrator = no parking
|
|
assert env.error is None
|
|
assert env.status == "in_progress"
|
|
|
|
async def test_mark_waiting_long_receives_waiting_for_arg(self) -> None:
|
|
"""Every mark_waiting_long call must carry waiting_for='rate_limit_lifted'."""
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
active = ["be-dev-1", "be-qa"]
|
|
orch = _make_orchestrator(active_agents=active)
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch)
|
|
c = Choreographer(deps)
|
|
|
|
await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
for c_args in orch.mark_waiting_long.call_args_list:
|
|
# mark_waiting_long(slug, waiting_for=..., ...) — waiting_for is a kwarg
|
|
assert c_args.kwargs.get("waiting_for") == "rate_limit_lifted"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# RATE_LIMIT_HIT event published with correct payload structure
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestRateLimitHitEventPublished:
|
|
async def test_stream_bus_publish_called_once(self) -> None:
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
orch = _make_orchestrator()
|
|
bus = _make_stream_bus()
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch, stream_bus=bus)
|
|
c = Choreographer(deps)
|
|
|
|
await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
bus.publish.assert_awaited_once()
|
|
|
|
async def test_event_type_is_rate_limit_hit(self) -> None:
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
orch = _make_orchestrator()
|
|
bus = _make_stream_bus()
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch, stream_bus=bus)
|
|
c = Choreographer(deps)
|
|
|
|
await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
event = bus.publish.call_args.args[0]
|
|
assert event.type == EventType.RATE_LIMIT_HIT
|
|
|
|
async def test_event_data_has_provider_field(self) -> None:
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
orch = _make_orchestrator(provider="anthropic")
|
|
bus = _make_stream_bus()
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch, stream_bus=bus)
|
|
c = Choreographer(deps)
|
|
|
|
await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
event = bus.publish.call_args.args[0]
|
|
assert "provider" in event.data
|
|
assert event.data["provider"] == "anthropic"
|
|
|
|
async def test_event_data_has_affected_agents_list(self) -> None:
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
active = ["be-dev-1", "be-dev-2"]
|
|
orch = _make_orchestrator(active_agents=active)
|
|
bus = _make_stream_bus()
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch, stream_bus=bus)
|
|
c = Choreographer(deps)
|
|
|
|
await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
event = bus.publish.call_args.args[0]
|
|
assert "affectedAgents" in event.data
|
|
assert isinstance(event.data["affectedAgents"], list)
|
|
assert event.data["affectedAgents"] == active
|
|
|
|
async def test_event_data_has_retry_after_seconds_null_by_default(self) -> None:
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
orch = _make_orchestrator()
|
|
bus = _make_stream_bus()
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch, stream_bus=bus)
|
|
c = Choreographer(deps)
|
|
|
|
await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
event = bus.publish.call_args.args[0]
|
|
assert "retryAfterSeconds" in event.data
|
|
assert event.data["retryAfterSeconds"] is None
|
|
|
|
async def test_event_data_retry_after_parsed_from_what_needed(self) -> None:
|
|
"""If what_needed is a numeric string, it becomes retryAfterSeconds."""
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
orch = _make_orchestrator()
|
|
bus = _make_stream_bus()
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch, stream_bus=bus)
|
|
c = Choreographer(deps)
|
|
|
|
await c.i_am_blocked(agent_id, task_id, "rate_limited", what_needed="30")
|
|
|
|
event = bus.publish.call_args.args[0]
|
|
assert event.data["retryAfterSeconds"] == float("30")
|
|
|
|
async def test_event_data_has_timestamp_iso_string(self) -> None:
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
orch = _make_orchestrator()
|
|
bus = _make_stream_bus()
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch, stream_bus=bus)
|
|
c = Choreographer(deps)
|
|
|
|
await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
event = bus.publish.call_args.args[0]
|
|
assert "timestamp" in event.data
|
|
# ISO string: must be a non-empty string
|
|
ts = event.data["timestamp"]
|
|
assert isinstance(ts, str) and len(ts) > 0
|
|
|
|
async def test_no_publish_when_stream_bus_is_none(self) -> None:
|
|
"""When stream_bus is not wired in, no publish is attempted."""
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
orch = _make_orchestrator()
|
|
# stream_bus=None: no bus
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch, stream_bus=None)
|
|
c = Choreographer(deps)
|
|
|
|
env = await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
# Should still succeed
|
|
assert env.error is None
|
|
assert env.status == "in_progress"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# RateLimitStateTracker.activate() called on rate_limited path
|
|
# ---------------------------------------------------------------------------
|
|
|
|
_TRACKER_PATCH = "roboco.services.gateway.rate_limit_tracker.RateLimitStateTracker"
|
|
|
|
|
|
class TestRateLimitTrackerActivateOnParking:
|
|
"""Verify that _handle_rate_limited_parking() calls activate()."""
|
|
|
|
async def test_activate_called_when_provider_known(self) -> None:
|
|
"""activate() must be called once when provider != 'unknown'."""
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
orch = _make_orchestrator(active_agents=["be-dev-1"], provider=_PROVIDER)
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch)
|
|
c = Choreographer(deps)
|
|
|
|
mock_tracker = AsyncMock()
|
|
mock_tracker.activate = AsyncMock(return_value=None)
|
|
mock_tracker_cls = MagicMock(return_value=mock_tracker)
|
|
|
|
with patch(_TRACKER_PATCH, mock_tracker_cls):
|
|
await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
mock_tracker_cls.assert_called_once_with(_PROVIDER)
|
|
mock_tracker.activate.assert_awaited_once()
|
|
|
|
async def test_activate_receives_affected_agents(self) -> None:
|
|
"""activate() must be called with the affected_agents list."""
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
active = ["be-dev-1", "be-dev-2"]
|
|
orch = _make_orchestrator(active_agents=active, provider=_PROVIDER)
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch)
|
|
c = Choreographer(deps)
|
|
|
|
mock_tracker = AsyncMock()
|
|
mock_tracker.activate = AsyncMock(return_value=None)
|
|
mock_tracker_cls = MagicMock(return_value=mock_tracker)
|
|
|
|
with patch(_TRACKER_PATCH, mock_tracker_cls):
|
|
await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
call_kwargs = mock_tracker.activate.call_args.kwargs
|
|
assert call_kwargs.get("affected_agents") == active
|
|
|
|
async def test_activate_receives_retry_after_from_what_needed(self) -> None:
|
|
"""activate() must receive retry_after parsed from what_needed."""
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
orch = _make_orchestrator(active_agents=["be-dev-1"], provider=_PROVIDER)
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch)
|
|
c = Choreographer(deps)
|
|
|
|
mock_tracker = AsyncMock()
|
|
mock_tracker.activate = AsyncMock(return_value=None)
|
|
mock_tracker_cls = MagicMock(return_value=mock_tracker)
|
|
|
|
with patch(_TRACKER_PATCH, mock_tracker_cls):
|
|
await c.i_am_blocked(agent_id, task_id, "rate_limited", what_needed="45")
|
|
|
|
call_kwargs = mock_tracker.activate.call_args.kwargs
|
|
assert call_kwargs.get("retry_after") == float("45")
|
|
|
|
async def test_activate_retry_after_none_when_what_needed_not_numeric(self) -> None:
|
|
"""activate() must receive retry_after=None when what_needed is not a number."""
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
orch = _make_orchestrator(active_agents=["be-dev-1"], provider=_PROVIDER)
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch)
|
|
c = Choreographer(deps)
|
|
|
|
mock_tracker = AsyncMock()
|
|
mock_tracker.activate = AsyncMock(return_value=None)
|
|
mock_tracker_cls = MagicMock(return_value=mock_tracker)
|
|
|
|
with patch(_TRACKER_PATCH, mock_tracker_cls):
|
|
await c.i_am_blocked(
|
|
agent_id, task_id, "rate_limited", what_needed="retry soon"
|
|
)
|
|
|
|
call_kwargs = mock_tracker.activate.call_args.kwargs
|
|
assert call_kwargs.get("retry_after") is None
|
|
|
|
async def test_activate_skipped_when_provider_unknown(self) -> None:
|
|
"""activate() must NOT be called when provider resolves to 'unknown'."""
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
# get_provider_for_agent returns None → provider stays 'unknown'
|
|
orch = MagicMock()
|
|
orch.get_provider_for_agent = MagicMock(return_value=None)
|
|
orch.get_active_agent_slugs_for_provider = MagicMock(return_value=[])
|
|
orch.mark_waiting_long = AsyncMock(return_value=None)
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch)
|
|
c = Choreographer(deps)
|
|
|
|
mock_tracker = AsyncMock()
|
|
mock_tracker.activate = AsyncMock(return_value=None)
|
|
mock_tracker_cls = MagicMock(return_value=mock_tracker)
|
|
|
|
with patch(_TRACKER_PATCH, mock_tracker_cls):
|
|
env = await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
# No crash, no activate call
|
|
assert env.error is None
|
|
mock_tracker.activate.assert_not_awaited()
|
|
|
|
async def test_activate_failure_does_not_crash_path(self) -> None:
|
|
"""If activate() raises, _handle_rate_limited_parking must still succeed."""
|
|
agent_id = uuid4()
|
|
task_id = uuid4()
|
|
orch = _make_orchestrator(active_agents=["be-dev-1"], provider=_PROVIDER)
|
|
deps = _make_deps(agent_id, task_id, orchestrator=orch)
|
|
c = Choreographer(deps)
|
|
|
|
mock_tracker = AsyncMock()
|
|
mock_tracker.activate = AsyncMock(side_effect=RuntimeError("redis down"))
|
|
mock_tracker_cls = MagicMock(return_value=mock_tracker)
|
|
|
|
with patch(_TRACKER_PATCH, mock_tracker_cls):
|
|
env = await c.i_am_blocked(agent_id, task_id, "rate_limited")
|
|
|
|
assert env.error is None
|
|
assert env.status == "in_progress"
|