feat(eval): golden-task eval harness + doctrine cohort stamp (#655)

* fix(notifications): exponential backoff + CAS claim for expired-unacked re-escalation

The sweep re-escalated every expired unacked ack-required notification
on every ~60s tick, forever — the live incident: 3 fresh blocker
escalations + Telegram DMs per minute from a static stale pile. Now
each notification carries reescalation_count / last_reescalated_at /
reescalation_delivered_count (migration 079): first fire at expiry,
then doubling intervals from 1h capped at 24h, hard stop after
ROBOCO_NOTIFICATION_MAX_REESCALATIONS (default 5) with one permanent
log carrying attempts-vs-delivered so 'seen and ignored' is
distinguishable from 'route never worked'. The due/wait/capped decision
is a pure function in foundation/policy/communications.py.

Per adversarial review, the attempt slot is claimed by compare-and-set
(UPDATE ... WHERE reescalation_count = :n) BEFORE delivery — the
previous draft leaned on the 60s dedup window, which never engages for
BLOCKER_ESCALATION (_LOOP_PRONE_TYPES excludes it), so concurrent
sweeps would have double-delivered. A lost claim skips delivery
outright. Legacy rows read as count=0 and keep today's first-fire
semantics. 61 tests incl. a two-session CAS race and a real alembic
upgrade/downgrade round trip.

* feat(budgets): per-task and per-project cost budgets (flag-gated)

tasks.budget_usd + projects.monthly_budget_usd (migration 080, chained
on 079; adds ix_agent_spawn_sessions_task_id since both enforcement
seams filter on bare task_id). Behind ROBOCO_TASK_BUDGETS_ENABLED
(default off, feature-flags card) — verifiably inert when off.

Claim-time: a project-month-spend guard applies to WORK-STARTING claims
only (i_will_work_on / i_will_plan) — per adversarial review, review/
doc/gate/inbound-PR claims are exempt so in-flight work can always
finish reviewing and merging at cap. Spend counts closed sessions'
estimated_cost_usd PLUS open sessions priced live from token snapshots
(the original closed-only sum read parallel long sessions as $0).

Sweep-side: the existing budget sweep also prices the active task's
spend vs budget_usd (TaskType defaults when null); on breach the task
is BLOCKED (HUMAN resolver, budget marker) BEFORE the graceful stop so
the unclaim no-ops and the dispatcher never respawns onto it, and the
CEO notification names both recovery steps. unblock on a budget-blocked
task re-checks live spend and refuses while still over — no silent
re-breach loop. Panel: budget inputs in both dialogs (0 rejected — a
zero budget silently blocks everything), spend logic consolidated in
TaskService.task_spend_usd. 42 new tests incl. a real-DB spend-query
suite and a two-tick non-refire sweep test.

* feat(eval): golden-task eval harness + doctrine cohort stamp

roboco/eval: 6 BenchTaskSpec fixtures run through the real lifecycle in
a disposable environment (the e2e_smoke harness's fake GitHub + local
git origin + throwaway DB catalog — real isolation, not convention),
scored deterministically (terminal status, revision_count, cycle time,
tokens/cost via the agent_spawn_sessions task_id join) plus a local-
model judge whose output is nested under a non_deterministic-marked
object so cohort diffs don't read judge noise as regression. CLI:
python -m roboco.eval run --role <slug> --cohort <name>. Source-
checkout-only by declared posture (deptry-scoped ignore + a hard
ImportError guard naming why; tests/ never ships in images or wheels).

agent_spawn_sessions.doctrine_version (migration 081, chained on 080)
is stamped at spawn-session finalize from the composed prompt layers —
with the session's model column it identifies a cohort durably.

Per adversarial review: bench runs patch the vault flags off (they were
writing real markdown into the operator's vault), and the real-spawn
OrchestratorStageSpawner is deliberately cut to NotImplementedError —
spawned containers' MCP wiring resolves to the production orchestrator
under real agent UUIDs, so real spawns wait for a dedicated follow-up;
the injectable scripted spawner is the working path. Full suite 13852
passed / 94% coverage in the source worktree; deptry/mypy/xenon clean.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>

tests/e2e_smoke/harness.py

@@ -45,8 +45,20 @@ if TYPE_CHECKING:
     from collections.abc import Iterator
     from pathlib import Path
     from types import ModuleType
+    from typing import Protocol
     from uuid import UUID
 
+    class TmpPathFactory(Protocol):
+        """Structural stand-in for the one ``pytest.TempPathFactory`` method
+        ``build_e2e_stack`` uses. pytest's real fixture value already
+        satisfies this shape, so it needs no adapter — but it lets
+        ``roboco/eval/runner.py`` (an offline CLI, not a pytest session) drive
+        this same stack-building machinery with a plain temp-dir factory
+        instead of constructing a real ``pytest.Config``."""
+
+        def mktemp(self, basename: str, numbered: bool = True) -> Path: ...
+
+
 _OWNER = "e2e-smoke"
 _REPO = "proj"
 
@@ -360,9 +372,15 @@ def _build_app(gh: _FakeGitHub) -> FastAPI:
 
 
 def build_e2e_stack(
-    _test_database_url: str, tmp_path_factory: pytest.TempPathFactory
+    _test_database_url: str, tmp_path_factory: TmpPathFactory
 ) -> Iterator[E2EStack]:
-    """Generator behind the ``e2e_stack`` fixture (defined in conftest)."""
+    """Generator behind the ``e2e_stack`` fixture (defined in conftest).
+
+    ``tmp_path_factory`` only needs ``.mktemp()`` (see ``TmpPathFactory``
+    above) — pytest's real fixture satisfies it structurally, and
+    ``roboco/eval/runner.py`` drives this same function with a plain
+    non-pytest factory to reuse this stack outside a test session.
+    """
     from roboco.config import settings
     from roboco.db import base as db_base
 

tests/e2e_smoke/test_eval_bench.py

@@ -0,0 +1,204 @@
+"""Integration test for the eval bench's own orchestration/scoring plumbing.
+
+The real ``StageSpawner`` (``OrchestratorStageSpawner``) drives a REAL agent
+container via ``AgentOrchestrator.spawn_agent`` and needs a Docker daemon +
+built agent images — it cannot run here (see ``roboco/eval/runner.py``'s
+module docstring). This test substitutes a scripted stand-in that drives the
+SAME real MCP flow/do tool functions ``tests.e2e_smoke.harness.ScriptedAgent``
+uses (via the existing ``dev_arc`` / ``qa_arc`` / ``doc_arc`` helpers, plus a
+PM ``complete`` call) so it proves the runner's OWN code — its throwaway-DB +
+disposable-project setup, its status-driven stage loop, its PM pre-claim,
+its deterministic scoring, its JSON/table output — without touching Docker.
+
+Runs the smallest fixture (a single-file bug fix) end to end: PENDING ->
+awaiting_qa -> awaiting_documentation -> awaiting_pm_review -> completed.
+
+Gating: like every other module here, this is skipped unless
+``ROBOCO_E2E_SMOKE=1`` (see ``tests/e2e_smoke/conftest.py``'s
+``pytest_collection_modifyitems``) — it needs the real test Postgres, which
+``EvalRunner`` provisions its own throwaway copy of (see
+``roboco/eval/runner.py``'s ``_scratch_database``), independent of this
+package's shared session-scoped ``e2e_stack`` fixture.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from typing import TYPE_CHECKING, Any
+from uuid import UUID
+
+from roboco.config import settings
+from roboco.eval.fixtures import FIXTURES
+from roboco.eval.runner import BenchJudge, EvalRunner, JudgeVerdict, _bench_environment
+from tests.e2e_smoke.arcs import Company, dev_arc, doc_arc, qa_arc
+from tests.e2e_smoke.harness import ScriptedAgent
+
+if TYPE_CHECKING:
+    import pytest
+    from roboco.eval.fixtures import BenchTaskSpec
+    from tests.e2e_smoke.harness import E2EStack
+
+_FIXTURE_KEY = "bugfix-off-by-one"
+_FIXED_FIX = (
+    "def paginate(items, page, size):\n"
+    "    start = (page - 1) * size\n"
+    "    end = page * size\n"
+    "    return items[start:end]\n"
+)
+
+
+def _fixture() -> BenchTaskSpec:
+    for f in FIXTURES:
+        if f.key == _FIXTURE_KEY:
+            return f
+    raise AssertionError(f"{_FIXTURE_KEY!r} fixture not found in FIXTURES")
+
+
+def _stub_company() -> Company:
+    """A ``Company`` carrying the FIXED uuids ``EvalRunner``'s own company
+    seeding uses (not fresh random ones, unlike ``arcs.seed_company``) — the
+    "be-*" slugs are hardcoded inside ``dev_arc`` / ``qa_arc`` / ``doc_arc``
+    themselves, so this only needs to supply the matching ids."""
+    from roboco.foundation import identity as _foundation
+
+    company = Company()
+    company.dev_id = _foundation.AGENTS["be-dev-1"].uuid
+    company.qa_id = _foundation.AGENTS["be-qa"].uuid
+    company.doc_id = _foundation.AGENTS["be-doc"].uuid
+    company.cell_pm_id = _foundation.AGENTS["be-pm"].uuid
+    return company
+
+
+class _ScriptedBenchSpawner:
+    """Test-only ``StageSpawner``: applies the KNOWN correct fix via the real
+    MCP flow/do tool functions, standing in for a real container spawn.
+
+    ``dev_arc`` / ``qa_arc`` / ``doc_arc`` (and ``ScriptedAgent`` itself) call
+    ``E2EStack.run_db``, which runs its own ``asyncio.run()`` per call — fine
+    from a plain sync pytest test, but ``run_stage`` is awaited from inside
+    ``_drive_task_to_terminal``'s own event loop, where a nested
+    ``asyncio.run()`` raises. Running the scripted turn on a worker thread
+    (``asyncio.to_thread``) gives it a thread with no running loop, exactly
+    like the sync test functions those helpers were written for.
+    """
+
+    def __init__(self, stack: E2EStack) -> None:
+        self._stack = stack
+        self._company = _stub_company()
+
+    async def run_stage(self, *, task: dict[str, Any], agent_slug: str) -> None:
+        await asyncio.to_thread(self._run_stage_sync, task, agent_slug)
+
+    def _run_stage_sync(self, task: dict[str, Any], agent_slug: str) -> None:
+        from roboco.agents_config import get_agent_role
+
+        role = get_agent_role(agent_slug)
+        task_id = UUID(task["id"])
+        if role == "developer":
+            dev_arc(
+                self._stack,
+                self._company,
+                task["project_slug"],
+                task_id,
+                work=(f"bench/{_FIXTURE_KEY}/paginate.py", _FIXED_FIX),
+            )
+        elif role == "qa":
+            qa_arc(self._stack, self._company, task_id)
+        elif role == "documenter":
+            doc_arc(
+                self._stack,
+                self._company,
+                task_id,
+                filename=f"bench/{_FIXTURE_KEY}/paginate.py",
+            )
+        elif role == "cell_pm":
+            pm = ScriptedAgent(
+                self._stack, self._company.cell_pm_id, agent_slug, "cell_pm"
+            )
+            pm.flow(
+                "complete",
+                task_id=str(task_id),
+                notes="Scripted bench completion: QA passed, docs complete.",
+            )
+        else:
+            raise AssertionError(f"unexpected role for the scripted bench: {role!r}")
+
+
+_EXPECTED_JUDGE_SCORE = 5
+
+
+class _FakeJudge(BenchJudge):
+    """Deterministic stand-in for the local-model judge — no network."""
+
+    async def score(
+        self, *, fixture: BenchTaskSpec, diff: str, notes: str
+    ) -> JudgeVerdict:
+        return JudgeVerdict(
+            score=_EXPECTED_JUDGE_SCORE, rationale="scripted test: assumed correct"
+        )
+
+
+def test_eval_runner_drives_a_fixture_to_completion_with_a_scripted_spawn() -> None:
+    runner = EvalRunner(
+        make_spawner=_ScriptedBenchSpawner,
+        judge=_FakeJudge(),
+        fixture_timeout_seconds=60.0,
+    )
+
+    cohort = runner.run_cohort(
+        "be-dev-1", "scripted-test", fixtures=[_fixture()], json_out=None
+    )
+
+    assert cohort.role_slug == "be-dev-1"
+    assert len(cohort.fixtures) == 1
+    result = cohort.fixtures[0]
+    assert result.fixture_key == _FIXTURE_KEY
+    assert result.metrics.final_status == "completed"
+    assert result.metrics.stalled is False
+    assert result.passed is True
+    assert result.metrics.revision_count == 0
+    assert result.judge.score == _EXPECTED_JUDGE_SCORE
+    assert cohort.pass_rate == 1.0
+    # No real container spawned, so no agent_spawn_sessions rows accrued for
+    # this task — the scripted stand-in proves the runner's DB/polling/
+    # scoring plumbing, not token/cost accounting (that needs a real spawn;
+    # see the module docstring).
+    assert result.metrics.total_tokens == 0
+    assert result.metrics.estimated_cost_usd == 0.0
+
+
+def test_bench_environment_disables_vault_writes_even_when_ambient_flags_are_armed(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """A bench run must never write into the operator's REAL Obsidian vault.
+    Simulates the compose-default posture (every vault flag armed True) and
+    asserts `_bench_environment` forces them all off for its duration, then
+    restores the prior values on exit — the exact leak an adversarial review
+    flagged (TaskService.create / JournalService / A2AService all gate on
+    obsidian_vault_enabled first, so patching it is the load-bearing part;
+    the three sub-flags are patched too for defense-in-depth)."""
+    monkeypatch.setattr(settings, "obsidian_vault_enabled", True)
+    monkeypatch.setattr(settings, "vault_intake_enabled", True)
+    monkeypatch.setattr(settings, "vault_kb_enabled", True)
+    monkeypatch.setattr(settings, "vault_report_enabled", True)
+    armed = (
+        settings.obsidian_vault_enabled,
+        settings.vault_intake_enabled,
+        settings.vault_kb_enabled,
+        settings.vault_report_enabled,
+    )
+
+    with _bench_environment("be-dev-1"):
+        assert settings.obsidian_vault_enabled is False
+        assert settings.vault_intake_enabled is False
+        assert settings.vault_kb_enabled is False
+        assert settings.vault_report_enabled is False
+
+    # Restored to the (simulated ambient) armed state once the bench exits.
+    restored = (
+        settings.obsidian_vault_enabled,
+        settings.vault_intake_enabled,
+        settings.vault_kb_enabled,
+        settings.vault_report_enabled,
+    )
+    assert restored == armed == (True, True, True, True)

tests/unit/eval/test_fixtures.py

@@ -0,0 +1,74 @@
+"""Schema checks for the golden-task fixtures (roboco/eval/fixtures.py).
+
+Nothing here touches a DB or the network — these are pure sanity checks on
+the static FIXTURES tuple so a malformed fixture (a duplicate key, a fixture
+file that escapes its own bench/<key>/ namespace and could collide with
+another fixture's repo state, an empty brief) is caught before it ever
+reaches the runner.
+"""
+
+from __future__ import annotations
+
+import dataclasses
+from typing import Any, cast
+
+import pytest
+from roboco.eval.fixtures import FIXTURES, BenchTaskSpec
+
+_MIN_FIXTURES = 5
+_MAX_FIXTURES = 8
+
+
+def test_fixture_keys_are_unique() -> None:
+    keys = [f.key for f in FIXTURES]
+    assert len(keys) == len(set(keys)), f"duplicate fixture keys: {keys}"
+
+
+def test_at_least_five_fixtures() -> None:
+    # The task calls for 5-8 canonical fixtures.
+    assert _MIN_FIXTURES <= len(FIXTURES) <= _MAX_FIXTURES, len(FIXTURES)
+
+
+def test_every_fixture_has_a_non_empty_brief() -> None:
+    for f in FIXTURES:
+        assert f.title.strip(), f.key
+        assert f.description.strip(), f.key
+        assert f.acceptance_criteria, f"{f.key} has no acceptance criteria"
+        assert all(c.strip() for c in f.acceptance_criteria), f.key
+        assert f.expectations.strip(), f"{f.key} has no judge expectations note"
+
+
+def test_repo_files_are_namespaced_under_bench_key() -> None:
+    """Every fixture's seeded file lives under bench/<its own key>/ so
+    sequential fixtures sharing one project's git history never collide."""
+    for f in FIXTURES:
+        assert f.repo_files, f"{f.key} seeds no repo files"
+        prefix = f"bench/{f.key}/"
+        for rel_path, content in f.repo_files:
+            assert rel_path.startswith(prefix), (
+                f"{f.key}: {rel_path!r} escapes its own {prefix!r} namespace"
+            )
+            assert ".." not in rel_path, f"{f.key}: {rel_path!r} looks like a traversal"
+            assert content, f"{f.key}: {rel_path!r} has empty content"
+
+
+def test_repo_file_paths_within_a_fixture_are_unique() -> None:
+    for f in FIXTURES:
+        paths = [rel_path for rel_path, _content in f.repo_files]
+        assert len(paths) == len(set(paths)), f"{f.key}: duplicate paths {paths}"
+
+
+def test_target_role_is_developer_for_every_fixture() -> None:
+    """Matches EvalRunner.run_cohort's current scope cut (see runner.py's
+    module docstring) — every fixture must be runnable by the one role the
+    bench supports today."""
+    for f in FIXTURES:
+        assert f.target_role == "developer", f.key
+
+
+def test_bench_task_spec_is_frozen() -> None:
+    spec = FIXTURES[0]
+    assert isinstance(spec, BenchTaskSpec)
+    mutable_view = cast("Any", spec)
+    with pytest.raises(dataclasses.FrozenInstanceError):
+        mutable_view.title = "mutated"

tests/unit/eval/test_scoring.py

@@ -0,0 +1,221 @@
+"""Unit tests for the eval bench's scorer math (roboco/eval/runner.py).
+
+Pure dataclass/aggregate-property tests — no DB, no network, no asyncio.
+`_build_judge_prompt` and `BenchJudge`'s score-parsing regex are covered too
+since both are pure string logic with no I/O.
+"""
+
+from __future__ import annotations
+
+import pytest
+from roboco.eval.fixtures import FIXTURES
+from roboco.eval.runner import (
+    _JUDGE_SCORE_RE,
+    CohortResult,
+    DeterministicMetrics,
+    FixtureResult,
+    JudgeVerdict,
+    OrchestratorStageSpawner,
+    _build_judge_prompt,
+)
+
+_EXPECTED_TOTAL_TOKENS = 180
+_HALF_PASS_RATE = 0.5
+_COHORT_TOTAL_TOKENS = 600
+_COHORT_MEAN_CYCLE_SECONDS = 20.0
+_COHORT_MEAN_JUDGE_SCORE = 5.0
+_PASSING_JUDGE_SCORE = 4
+
+
+def _metrics(
+    *,
+    final_status: str = "completed",
+    stalled: bool = False,
+    cycle_time_seconds: float = 10.0,
+    tokens_input: int = 100,
+    tokens_output: int = 50,
+    tokens_cache_read: int = 0,
+    tokens_cache_write: int = 0,
+    estimated_cost_usd: float = 0.01,
+) -> DeterministicMetrics:
+    return DeterministicMetrics(
+        final_status=final_status,
+        stalled=stalled,
+        revision_count=0,
+        cycle_time_seconds=cycle_time_seconds,
+        tokens_input=tokens_input,
+        tokens_output=tokens_output,
+        tokens_cache_read=tokens_cache_read,
+        tokens_cache_write=tokens_cache_write,
+        estimated_cost_usd=estimated_cost_usd,
+    )
+
+
+def test_deterministic_metrics_total_tokens_sums_all_four_buckets() -> None:
+    m = _metrics(
+        tokens_input=100, tokens_output=50, tokens_cache_read=25, tokens_cache_write=5
+    )
+    assert m.total_tokens == _EXPECTED_TOTAL_TOKENS
+
+
+def test_fixture_result_passed_requires_completed_and_not_stalled() -> None:
+    passed = FixtureResult(
+        fixture_key="a",
+        metrics=_metrics(final_status="completed", stalled=False),
+        judge=JudgeVerdict(score=None, rationale=None),
+    )
+    assert passed.passed is True
+
+    cancelled = FixtureResult(
+        fixture_key="b",
+        metrics=_metrics(final_status="cancelled", stalled=False),
+        judge=JudgeVerdict(score=None, rationale=None),
+    )
+    assert cancelled.passed is False
+
+    # A stall that happens to leave the row at "completed" is still not a
+    # pass — `stalled` overrides the status.
+    stalled_completed = FixtureResult(
+        fixture_key="c",
+        metrics=_metrics(final_status="completed", stalled=True),
+        judge=JudgeVerdict(score=None, rationale=None),
+    )
+    assert stalled_completed.passed is False
+
+
+def _sample_cohort() -> CohortResult:
+    fixtures = [
+        FixtureResult(
+            fixture_key="a",
+            metrics=_metrics(
+                final_status="completed",
+                cycle_time_seconds=10.0,
+                estimated_cost_usd=0.10,
+                tokens_input=100,
+                tokens_output=100,
+            ),
+            judge=JudgeVerdict(score=5, rationale="great"),
+        ),
+        FixtureResult(
+            fixture_key="b",
+            metrics=_metrics(
+                final_status="needs_revision",
+                stalled=True,
+                cycle_time_seconds=30.0,
+                estimated_cost_usd=0.20,
+                tokens_input=200,
+                tokens_output=200,
+            ),
+            judge=JudgeVerdict(score=None, rationale="judge unavailable"),
+        ),
+    ]
+    return CohortResult(role_slug="be-dev-1", cohort_name="baseline", fixtures=fixtures)
+
+
+def test_cohort_pass_rate_and_totals() -> None:
+    cohort = _sample_cohort()
+
+    assert cohort.pass_rate == _HALF_PASS_RATE
+    assert cohort.total_cost_usd == pytest.approx(0.3)
+    assert cohort.total_tokens == _COHORT_TOTAL_TOKENS
+    assert cohort.mean_cycle_time_seconds == _COHORT_MEAN_CYCLE_SECONDS
+    # Only fixture "a" has a judge score; "b"'s None is excluded from the mean.
+    assert cohort.mean_judge_score == _COHORT_MEAN_JUDGE_SCORE
+
+
+def test_cohort_mean_judge_score_is_none_when_no_fixture_was_scored() -> None:
+    fixtures = [
+        FixtureResult(
+            fixture_key="a",
+            metrics=_metrics(),
+            judge=JudgeVerdict(score=None, rationale="judge unavailable"),
+        )
+    ]
+    cohort = CohortResult(role_slug="be-dev-1", cohort_name="x", fixtures=fixtures)
+    assert cohort.mean_judge_score is None
+
+
+def test_cohort_with_no_fixtures_is_a_zero_result_not_a_crash() -> None:
+    cohort = CohortResult(role_slug="be-dev-1", cohort_name="x", fixtures=[])
+    assert cohort.pass_rate == 0.0
+    assert cohort.total_cost_usd == 0.0
+    assert cohort.total_tokens == 0
+    assert cohort.mean_cycle_time_seconds == 0.0
+    assert cohort.mean_judge_score is None
+
+
+def test_cohort_as_dict_round_trips_every_fixture() -> None:
+    fixtures = [
+        FixtureResult(
+            fixture_key="a",
+            metrics=_metrics(),
+            judge=JudgeVerdict(_PASSING_JUDGE_SCORE, "solid"),
+        ),
+    ]
+    cohort = CohortResult(role_slug="be-dev-1", cohort_name="x", fixtures=fixtures)
+    payload = cohort.as_dict()
+
+    assert payload["role_slug"] == "be-dev-1"
+    assert payload["cohort_name"] == "x"
+    assert payload["aggregate"]["fixture_count"] == 1
+    assert payload["aggregate"]["pass_rate"] == 1.0
+    # Judge fields live under their own nested, explicitly-marked object —
+    # never flat beside deterministic metrics — so a naive diff can't read
+    # judge noise as a regression.
+    assert "mean_judge_score" not in payload["aggregate"]
+    assert payload["judge"] == {
+        "mean_score": _PASSING_JUDGE_SCORE,
+        "non_deterministic": True,
+    }
+    assert len(payload["fixtures"]) == 1
+    assert payload["fixtures"][0]["fixture_key"] == "a"
+    assert "judge_score" not in payload["fixtures"][0]
+    assert payload["fixtures"][0]["judge"] == {
+        "score": _PASSING_JUDGE_SCORE,
+        "rationale": "solid",
+        "non_deterministic": True,
+    }
+
+
+def test_judge_score_regex_parses_the_required_reply_shape() -> None:
+    reply = "Score: 4\nRationale: matches the expectation closely.\n"
+    match = _JUDGE_SCORE_RE.search(reply)
+    assert match is not None
+    assert int(match.group(1)) == _PASSING_JUDGE_SCORE
+
+
+def test_judge_score_regex_is_case_insensitive_and_tolerates_spacing() -> None:
+    assert _JUDGE_SCORE_RE.search("score:5") is not None
+    assert _JUDGE_SCORE_RE.search("SCORE :   3") is not None
+
+
+def test_judge_score_regex_rejects_out_of_range_scores() -> None:
+    assert _JUDGE_SCORE_RE.search("Score: 0") is None
+    assert _JUDGE_SCORE_RE.search("Score: 6") is None
+
+
+def test_build_judge_prompt_includes_the_expectation_and_acceptance_criteria() -> None:
+    fixture = FIXTURES[0]
+    prompt = _build_judge_prompt(fixture, diff="+ fixed line", notes="dev notes here")
+
+    assert fixture.title in prompt
+    assert fixture.expectations in prompt
+    for criterion in fixture.acceptance_criteria:
+        assert criterion in prompt
+    assert "+ fixed line" in prompt
+    assert "dev notes here" in prompt
+
+
+def test_build_judge_prompt_handles_empty_diff_and_notes() -> None:
+    fixture = FIXTURES[0]
+    prompt = _build_judge_prompt(fixture, diff="", notes="")
+    assert "(empty diff)" in prompt
+    assert "(no notes)" in prompt
+
+
+def test_orchestrator_stage_spawner_is_cut_and_refuses_to_construct() -> None:
+    """The real-spawn path is deliberately disabled this release (its MCP
+    wiring would authenticate against the REAL production orchestrator) —
+    this is the one runnable check that the cut stays in place."""
+    with pytest.raises(NotImplementedError, match="cut from this release"):
+        OrchestratorStageSpawner()