Files
roboco/tests/unit/gateway/test_evidence_builder.py
T
0ca9d91b72 v0.16.0: fastapi-guard HTTP security layer — calibrated + scanner honeytrap (#290)
* [fastapi-guard] Phase 1a: gated config flags for the HTTP security layer

Adds the ROBOCO_GUARD_* settings (all default-off / secure-default) for the
upcoming fastapi-guard 7.2.0 hardening — guard_enabled (master switch),
guard_fail_secure (fail-closed default; NAS overrides to false),
guard_telemetry_enabled + guard_agent_api_key + guard_project_id (guard-agent
telemetry, opt-in), guard_emergency + guard_emergency_whitelist (lockdown kill
switch). Inert until consumed: nothing reads them yet, so the request path is
unchanged. Foundation for v0.16.0.

* [fastapi-guard] Phase 1b: security foundation module + gated wiring

Add fastapi-guard 7.2.0 + guard-core 3.3.0 (bare, unpinned) and roboco/security.py:
- build_security_config() from settings — behind-nginx real-IP (trusted_proxies +
  trust_x_forwarded_proto), HSTS/CSP headers, threat-ban + 404-sweep rules,
  redis-backed state, exclude_paths (/ws + health + docs), env-driven
  enforce_https, fail_secure (secure default), emergency lockdown, guard-agent
  telemetry (opt-in), passive-mode calibration switch.
- guard_deco singleton (SecurityDecorator) for per-route decorators (Phase 2+).
- Three custom content validators guard's WAF can't cover: prompt-injection /
  role-override, secret-exfil / credential-in-body, internal-SSRF.
- apply_guard(app) + guarded_lifespan() wired into create_app AFTER settings.
  guard_passive_mode config flag added.

Entirely gated by ROBOCO_GUARD_ENABLED (default off): create_app mounts nothing
and returns the unchanged app when off (verified). make quality GREEN
(cov 95.32%, pip-audit clean, import-linter 2/0). 12 new unit tests.

* [fastapi-guard] Phase 2: critical-path decorators

Apply guard decorators to the highest-value endpoints (metadata-only; enforced
only when the middleware is mounted, so no-op while ROBOCO_GUARD_ENABLED is off):
- provider keys (ollama/grok/self-hosted writes): strict rate_limit +
  max_request_size + block_clouds (no datacenter IP should touch secret writes).
- settings write + release approve/reject (CEO-gated): strict rate_limit.
- intake chat (prompter start/messages/events): rate_limit + max_request_size +
  custom_validation(prompt_injection_validator) — the prompt-facing free-text
  ingress gets the injection/role-override/secret-exfil content scan.

make quality GREEN (cov 95.32%, contracts 2/0). App builds with guard off,
decorators inert (verified).

* [fastapi-guard] Phase 3: wide decorator coverage across ingress + sensitive routes

Targeted-wide application (metadata-only; no-op until ROBOCO_GUARD_ENABLED). The
global SecurityMiddleware already rate-limits + WAF-scans every request, so this
adds the custom content validators on free-text ingress + tight limits on
sensitive ops (not blanket per-route rate_limit on reads):

- agent gateway do verbs (note/say/commit/dm/pitch/progress/draft_playbook/...):
  rate_limit + max_request_size + custom_validation(secret_exfil or prompt_injection).
- a2a message/send + chat writes: rate_limit + size + prompt_injection.
- optimal/RAG (kb/search, rag/query, mentor/ask, errors/decisions/standards/
  learnings): prompt_injection on searches, secret_exfil on record writes; docs
  index → internal_ssrf.
- tasks: create/update → prompt_injection; QA/doc/PM transitions → secret_exfil;
  CEO-gated verbs → tight rate_limit.
- secretary chat → prompt_injection; research → internal_ssrf; orchestrator
  spawn/mutations → rate_limit; git ops + flow verbs → tight rate_limit.
Pure GET/reads left to the global middleware. Applied via a Sonnet workflow,
then verified: app builds with guard off (decorators inert), make quality GREEN
(cov 95.37%, contracts 2/0). Decoy/honeypot-path surface deferred (needs verified
guard ban-API integration — not rushed).

* [fastapi-guard] Phase 5: arm the NAS composes in passive/log-only mode

Arm ROBOCO_GUARD_ENABLED=true + ROBOCO_GUARD_PASSIVE_MODE=true +
ROBOCO_GUARD_FAIL_SECURE=false on the two NAS composes (docker-compose.yaml +
.yml). Passive = guard mounts and logs what it WOULD block but blocks nothing,
so the next NAS deploy calibrates against real traffic; flip PASSIVE_MODE off
after the false-positive review to enforce. fail_secure=false keeps a
guard-internal error from 500ing the personal deploy. The registry (user-facing)
compose is deliberately left unarmed so its published default stays conservative.
Phase 4 (passive calibration) is the operational step this enables.

* feat(security): Phase 3b — full-arsenal per-route guard enrichment

Stack the applicable guard decorators per surface instead of the minimal
rate_limit/max_request_size/custom_validation triad: content_type_filter on
every JSON-body write, honeypot_detection form-traps on human-facing POSTs,
block_clouds on key-writes + CEO release ops, behavior_analysis runaway-rate
rules on the agent flow/do verbs, suspicious_detection + usage_monitor on the
sensitive surfaces. Nine distinct decorators now applied thoughtfully per
endpoint. All metadata-only — no-op while ROBOCO_GUARD_ENABLED is off.

* fix(a2a): permit PR reviewer to deliver gate verdicts to the owning PM

can_a2a_direct had no pr_reviewer rule, so a reviewer (team=board, or a cell
team) fell through to the cell-member path and was cross-cell-denied when the
in-path gate delivered a pr_fail change-request to main-pm (or a cross-cell
cell-pm): "Cannot A2A main-pm ... Ask None to coordinate with None". The
delivery is best-effort, so pr_fail still transitioned but the verdict never
reached the owning PM — the blind-re-submit signal-gap the pr_fail fix closes.

Add an explicit pr_reviewer handler: it may A2A only cell_pm / main_pm (its
sole comms surface — everything else it posts on the PR itself), with a matching
route hint. The cell reviewers kept same-team access by coincidence; this scopes
every reviewer to PM-only, the correct model, with no other A2A caller affected.

Refresh uv.lock to the current resolution.

* feat(models): adopt Claude Sonnet 5 as the sonnet tier

Point the 'sonnet' alias at claude-sonnet-5 (MODEL_MAP) and give pr_reviewer
its own opus tier in ROLE_MODEL_MAP — it was falling through to the sonnet
default, and the role gates untrusted external/fork PRs plus root→master, which
warrants opus.

Price claude-sonnet-5 at the promotional 33% off Sonnet 4.6 ($2.01 / $10.05,
cache 0.201 / 0.5025) through 2026-08-31 via a dedicated pricing fragment that
beats the bare 'sonnet' alias; revert to full rate when the promo ends. Bare
'sonnet' stays full-rate as a conservative fallback (prod prices the resolved
claude-sonnet-5 id from the transcript).

Update the model docs and the billing / usage / manifest / spawn tests.

* feat(security): calibrate the guard WAF for RoboCo traffic + document the layer

The first end-to-end run of the fastapi-guard layer showed active enforcement
would block ~50% of legitimate agent traffic — RoboCo request bodies are code,
SQL, diffs, file paths, HTML, and URLs, which the stock signature WAF reads as
attacks. build_security_config now excludes RoboCo's free-text top-level body
fields (derived from the real request models, including the free-form container
fields whose nested prose is stringified and scanned) from WAF scanning,
dropping the active-mode false-positive rate to zero while keeping the WAF on
every non-excluded (id/enum/slug/branch) field and leaving the
prompt-injection / secret-exfil / internal-SSRF validators — which run
independently of the exclusion — fully in force. enable_penetration_detection
is made explicit.

Only excluded_detection_body_fields is reliable on guard 7.2.1: the per-route
categories knob is bypassed for JSON bodies, and the body scanner excludes
top-level keys only (scanning str(value) of every non-excluded field), so
free-form container fields must be excluded wholesale.

Adds tests/unit/test_security_middleware.py — the first end-to-end exercise of
the middleware (mounts it, drives guard's lifespan, fires real requests):
proves passive mode is log-only, active mode does not false-positive on
realistic agent payloads, threats are still blocked inside excluded fields, and
the WAF still fires on non-excluded fields.

Docs: CHANGELOG (Unreleased); a user-facing Optional-subsystems page + nav +
env reference for the HTTP security layer; the agent-facing RAG corpus
(what it is + why a request could be blocked); and the roboco mapping
(api-core-websocket / deployment-tooling / _complete_map).

* feat(security): Surface N — scanner honeytrap (guard /api auto-ban + nginx edge-drop)

Turns scanner probes against the scanner, in two layers matched to where
traffic lands. Behind nginx only /api, /ws, /health, /ready reach the
orchestrator, so guard can only see (and ban) scanner probes on those paths;
the classic root probes (/.env, /wp-login.php, /phpmyadmin, /.git/config) hit
the panel. So:

- build_security_config's threat_ban_config gains recon / sensitive_file /
  cms_probing categories. A scanner probing those fingerprints on an /api path
  is detected on the URL-path scan; repeated probes from one IP trip an adaptive
  per-IP auto-ban (redis-backed, 24h). Only bans in active mode (passive logs
  the recon hit) and needs redis (the 24h ban exceeds the in-memory cap). The
  spec's decoy-route file is redundant — the WAF url-path scan bans regardless
  of a registered route — so it is intentionally omitted.
- docker/nginx.conf drops the classic root scanner paths at the edge with 444
  (connection closed, no response) before they reach the panel, anchored to
  known scanner fingerprints so /.well-known and every real panel/API route are
  untouched. Always on, independent of ROBOCO_GUARD_ENABLED.

Tests: 2 unit (the exclusion set + the scanner-ban categories are present) and
2 integration (a decoy path is blocked in active mode, passes in passive). The
nginx regex was validated against 15 scanner + 19 legit paths (0 false
positives). Docs: CHANGELOG, the HTTP-security page, the roboco mapping, and the
agent-facing RAG corpus.

* Token optimization — per-role observability, compute policy, spawn preflight (#291)

* test(models): lock the sonnet→claude-sonnet-5 MODEL_MAP invariant

* feat(usage): surface cache tokens + cache_hit_rate in usage breakdowns

* feat(usage): add per-role usage breakdown endpoint

* feat(usage): add spawn-waste signal (per-role unproductive rate + respawn strikes)

* feat(panel): surface per-role cost/cache + spawn-waste on the metrics page

* feat(routing): Phase 2 per-role compute policy — qa→haiku, main_pm→sonnet, per-role effort env mechanism (default-inert)

* feat(orchestrator): Phase 3 flag-gated spawn preflight — refuse non-gateway delivery roles (respawn-forever guard)

* chore(compose): arm ROBOCO_SPAWN_PREFLIGHT_ENABLED on the NAS composes

* docs: per-role usage observability, per-role compute policy, and spawn preflight

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>

* fix(panel): pin outputFileTracingRoot so the standalone build isn't broken by stray lockfiles

* feat(routing): populate ROLE_EFFORT_MAP + wire the verified --effort flag (cell_pm/board/auditor to medium)

* feat(gateway): omit empty context_briefing sections (Phase 4 payload compaction)

* refactor(orchestrator): extract spawn chokepoint guards to restore xenon rank B on spawn_agent

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-07-01 23:54:48 +02:00

244 lines
9.0 KiB
Python

"""Tests for evidence-payload + context-briefing assembly."""
from __future__ import annotations
from typing import Any
from unittest.mock import MagicMock
from uuid import uuid4
from roboco.services.gateway.evidence_builder import (
BRIEFING_LIST_CAP,
BriefingInputs,
build_context_briefing,
build_evidence_for_task,
build_task_handoff,
)
def _task(
*,
pr_number: int | None = 8,
pr_url: str | None = "https://github.com/x/y/pull/8",
commits: list[dict] | None = None,
dev_notes: str = "did stuff",
) -> MagicMock:
t = MagicMock()
t.id = uuid4()
t.pr_number = pr_number
t.pr_url = pr_url
t.commits = commits or [{"sha": "abc123", "message": "feat: x"}]
t.dev_notes = dev_notes
t.acceptance_criteria_status = []
return t
class TestEvidence:
def test_basic_payload(self) -> None:
t = _task()
ev = build_evidence_for_task(
t, journal_highlights=[], files_changed=["README.md"]
)
assert ev.pr_url == "https://github.com/x/y/pull/8"
assert ev.commits == [{"sha": "abc123", "message": "feat: x"}]
assert "README.md" in ev.files_changed
def test_no_pr_returns_empty_url(self) -> None:
t = _task(pr_number=None, pr_url=None)
ev = build_evidence_for_task(t, journal_highlights=[], files_changed=[])
assert ev.pr_url is None
assert ev.pr_number is None
class TestContextBriefing:
def test_empty_briefing(self) -> None:
inputs = BriefingInputs(
unread_a2a=[],
unread_mentions=[],
pending_notifications=[],
task_metadata_gaps=[],
recent_team_activity=[],
blockers_in_my_lane=[],
)
b = build_context_briefing(inputs)
# Empty sections are omitted to keep the per-verb payload compact — an
# all-empty briefing collapses to {}.
assert b == {}
def test_omits_empty_keeps_nonempty(self) -> None:
inputs = BriefingInputs(
unread_a2a=[{"id": "a1"}],
unread_mentions=[],
pending_notifications=[],
task_metadata_gaps=[],
recent_team_activity=[],
blockers_in_my_lane=[],
)
assert build_context_briefing(inputs) == {"unread_a2a": [{"id": "a1"}]}
def test_lists_capped_at_10(self) -> None:
twenty = [{"i": i} for i in range(20)]
inputs = BriefingInputs(
unread_a2a=twenty,
unread_mentions=twenty,
pending_notifications=twenty,
task_metadata_gaps=[],
recent_team_activity=twenty,
blockers_in_my_lane=twenty,
)
b = build_context_briefing(inputs)
assert len(b["unread_a2a"]) == BRIEFING_LIST_CAP
assert len(b["unread_mentions"]) == BRIEFING_LIST_CAP
assert len(b["pending_notifications"]) == BRIEFING_LIST_CAP
assert len(b["recent_team_activity"]) == BRIEFING_LIST_CAP
assert len(b["blockers_in_my_lane"]) == BRIEFING_LIST_CAP
def test_task_handoff_defaults_none_and_surfaces_in_briefing(self) -> None:
inputs = BriefingInputs(
unread_a2a=[],
unread_mentions=[],
pending_notifications=[],
task_metadata_gaps=[],
recent_team_activity=[],
blockers_in_my_lane=[],
)
# A None handoff is an empty section → omitted from the briefing.
assert "task_handoff" not in build_context_briefing(inputs)
with_handoff = BriefingInputs(
unread_a2a=[],
unread_mentions=[],
pending_notifications=[],
task_metadata_gaps=[],
recent_team_activity=[],
blockers_in_my_lane=[],
task_handoff={"pr_number": 8},
)
assert build_context_briefing(with_handoff)["task_handoff"] == {"pr_number": 8}
def test_company_goals_defaults_none_and_surfaces_in_briefing(self) -> None:
inputs = BriefingInputs(
unread_a2a=[],
unread_mentions=[],
pending_notifications=[],
task_metadata_gaps=[],
recent_team_activity=[],
blockers_in_my_lane=[],
)
# None goals is an empty section → omitted from the briefing.
assert "company_goals" not in build_context_briefing(inputs)
with_goals = BriefingInputs(
unread_a2a=[],
unread_mentions=[],
pending_notifications=[],
task_metadata_gaps=[],
recent_team_activity=[],
blockers_in_my_lane=[],
company_goals={"north_star": "win"},
)
assert build_context_briefing(with_goals)["company_goals"] == {
"north_star": "win"
}
class TestTaskHandoff:
def test_none_task_returns_none(self) -> None:
assert build_task_handoff(None, []) is None
def test_no_prior_work_returns_none(self) -> None:
t = _task(pr_number=None, pr_url=None, dev_notes="")
t.commits = [] # _task's `commits or [...]` default would re-seed one
t.acceptance_criteria_status = []
assert build_task_handoff(t, []) is None
def test_digest_from_prior_work(self) -> None:
pr = 8
t = _task(
pr_number=pr,
commits=[{"sha": "abc", "message": "feat: x"}],
dev_notes="implemented the parser",
)
t.branch_name = "feature/backend/abc"
t.acceptance_criteria_status = [{"criterion": "parses", "met": True}]
digest = build_task_handoff(t, [{"summary": "chose recursive descent"}])
assert digest is not None
assert digest["pr_number"] == pr
assert digest["branch_name"] == "feature/backend/abc"
assert digest["commit_count"] == 1
assert digest["dev_summary"] == "implemented the parser"
assert digest["journal_highlights"] == [{"summary": "chose recursive descent"}]
def test_surfaces_completed_dependencies(self) -> None:
dep_id = uuid4()
t = _task(pr_number=None, pr_url=None, dev_notes="")
t.commits = []
t.acceptance_criteria_status = []
t.completed_dependency_ids = [dep_id]
digest = build_task_handoff(t, [])
# A just-unblocked task with no other prior work still surfaces the dep.
assert digest is not None
assert digest["completed_dependency_ids"] == [str(dep_id)]
def test_caps_lists_and_type_guards(self) -> None:
thirty = [{"sha": str(i)} for i in range(30)]
t = _task(pr_number=7, commits=thirty)
# Non-list / mismatched-type attributes degrade safely, never leak.
t.acceptance_criteria_status = object()
t.pr_url = object()
t.branch_name = None
not_a_list: Any = object()
digest = build_task_handoff(t, not_a_list)
assert digest is not None
assert len(digest["recent_commits"]) == BRIEFING_LIST_CAP
assert digest["acceptance_criteria_status"] == []
assert digest["journal_highlights"] == []
assert digest["pr_url"] is None
assert digest["branch_name"] is None
class TestPrReviewSurface:
"""The persisted pr_fail verdict + issues must surface in the PM
briefing's task_handoff, not just the fire-and-forget a2a."""
def test_surfaces_pr_fail_verdict_and_issues(self) -> None:
t = _task(pr_number=138, commits=[{"sha": "abc", "message": "feat: x"}])
t.notes_structured = {
"pr_review": {
"verdict": "failed",
"summary": "In-path PR-review gate requested changes.",
"issues": ["missing null guard", "no test for the edge case"],
"head_sha": "aaaa1111bbbb2222",
}
}
digest = build_task_handoff(t, [])
assert digest is not None
pr_review = digest["pr_review"]
assert pr_review["verdict"] == "failed"
assert pr_review["issues"] == [
"missing null guard",
"no test for the edge case",
]
assert pr_review["head_sha"] == "aaaa1111bbbb2222"
def test_no_pr_review_field_when_none_present(self) -> None:
t = _task(pr_number=8, commits=[{"sha": "abc", "message": "feat: x"}])
t.notes_structured = None
digest = build_task_handoff(t, [])
assert digest is not None
# Absent pr_review ⇒ no key (not a None-valued key) so a PM without a
# prior gate verdict doesn't see a misleading empty slot.
assert "pr_review" not in digest
def test_pr_review_alone_is_prior_work_worth_resuming(self) -> None:
"""A task with no commits/dev-notes but a prior pr_fail verdict still
surfaces the handoff so the owning PM reads the change-requests."""
t = _task(pr_number=None, pr_url=None, dev_notes="")
t.commits = []
t.acceptance_criteria_status = []
t.notes_structured = {
"pr_review": {"verdict": "failed", "issues": ["fix the off-by-one"]}
}
digest = build_task_handoff(t, [])
assert digest is not None
assert digest["pr_review"]["issues"] == ["fix the off-by-one"]