[3e7a5064] Eval harness: wire the real-spawn path so python -m roboco.eval run works end-to-end (#703)

* [431e73b7] Wire the real-spawn path: OrchestratorStageSpawner + disposable MCP config (#701)

* [431e73b7] Wire the eval harness real-spawn path: OrchestratorStageSpawner + disposable MCP config

_generate_mcp_config now prefers settings.api_url when set (both
PROJECT_HOST_PATH branches), so spawned MCP servers resolve to the
harness's disposable orchestrator URL instead of the real production
hostname or 127.0.0.1:port. OrchestratorStageSpawner.__init__ replaces
the NotImplementedError with a real AgentOrchestrator() constructed the
same way the production dispatcher builds it. The runner module
docstring + __main__.py docstring/run-subparser help drop the
NOT-YET-FUNCTIONAL wording. A new unit test pins the no-production-reach
guarantee: with settings.api_url patched, the MCP config's
ROBOCO_API_URL/ROBOCO_ORCHESTRATOR_URL point at the disposable URL (not
production), and the agent UUID is the real fixed UUID from
foundation.identity.AGENTS.

* [431e73b7] docs(eval): reflect the wired real-spawn path in tests map, CLAUDE.md, and CHANGELOG

---------

Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>

* [5cc75f71] Fix disposable orchestrator container-reachability + document real-UUID isolation design (#705)

* [5cc75f71] Fix disposable orchestrator container-reachability + document real-UUID isolation

* [5cc75f71] docs(eval-harness): document container-reachability fix + real-UUID isolation in map docs

---------

Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>

* [d96ec059] Rewrite stale eval-spawner pinning test to assert wired behavior (test-only, CI-green for PR #703) (#707) (#708)

* [d96ec059] test(eval): assert OrchestratorStageSpawner constructs a real AgentOrchestrator

Rewrite the stale pinning test that asserted the PRE-wiring
NotImplementedError (removed by commit 488e9e2f when the real-spawn
path was wired). The test now asserts the wired behavior:
OrchestratorStageSpawner() construction succeeds, _orchestrator is
an AgentOrchestrator instance, and _stage_timeout_seconds defaults
to 900.0. Renamed from test_orchestrator_stage_spawner_is_cut_and_
refuses_to_construct to reflect the new contract. Test-only — no
production code touched.

* [d96ec059] docs(map): note test_scoring spawner pinning test in eval-harness map entry

Add one clause to docs/map/tests.md's roboco/eval/ row naming
tests/unit/eval/test_scoring.py::test_orchestrator_stage_spawner_constructs_real_orchestrator
as the unit test that pins the OrchestratorStageSpawner wired-construction
contract (construction succeeds, _orchestrator is an AgentOrchestrator,
_stage_timeout_seconds defaults to 900.0). Consistent with the line's
existing pattern of citing test_eval_mcp_config_isolation.py and
test_eval_bench.py by name for the contracts they pin. The CHANGELOG #701
entry already covers the user-visible wiring; no CHANGELOG change needed.

---------

Co-authored-by: roboco-app[bot] <302741806+roboco-app[bot]@users.noreply.github.com>
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>

---------

Co-authored-by: roboco-app[bot] <302741806+roboco-app[bot]@users.noreply.github.com>
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>
Co-authored-by: Backend PM <be-pm@roboco.tech>

tests/e2e_smoke/harness.py

@@ -272,6 +272,12 @@ class E2EStack:
     workspaces_root: Path
     db_url: str
     github: _FakeGitHub
+    # URL spawned agent containers use to reach this in-process orchestrator.
+    # base_url stays 127.0.0.1 (host-side test clients); container_url is
+    # host.docker.internal so containers on the roboco_default bridge can
+    # resolve the host's published port (the server binds 0.0.0.0 to accept
+    # both). Set by build_e2e_stack alongside base_url.
+    container_url: str = ""
 
     def workspace_of(self, project_slug: str, team: str, agent_slug: str) -> Path:
         return self.workspaces_root / project_slug / team / agent_slug
@@ -415,10 +421,13 @@ def build_e2e_stack(
     # uvloop when installed, and this in-thread server has crashed CI with a
     # uvloop/asyncpg segfault (uvloop 0.22 + asyncpg 0.31 + Python 3.13) —
     # mirror the production default instead of picking up uvloop implicitly.
+    # Bind 0.0.0.0 so spawned agent containers on the roboco_default bridge
+    # can reach the server via host.docker.internal (host-gateway). Host-side
+    # clients still connect via 127.0.0.1 (base_url) — 0.0.0.0 accepts both.
     server = uvicorn.Server(
         uvicorn.Config(
             app,
-            host="127.0.0.1",
+            host="0.0.0.0",
             port=port,
             log_level="warning",
             loop=settings.uvicorn_loop,
@@ -448,6 +457,7 @@ def build_e2e_stack(
             workspaces_root=workspaces,
             db_url=_test_database_url,
             github=gh,
+            container_url=f"http://host.docker.internal:{port}",
         )
     finally:
         server.should_exit = True

tests/unit/eval/test_scoring.py

@@ -18,6 +18,7 @@ from roboco.eval.runner import (
     OrchestratorStageSpawner,
     _build_judge_prompt,
 )
+from roboco.runtime.orchestrator import AgentOrchestrator
 
 _EXPECTED_TOTAL_TOKENS = 180
 _HALF_PASS_RATE = 0.5
@@ -25,6 +26,7 @@ _COHORT_TOTAL_TOKENS = 600
 _COHORT_MEAN_CYCLE_SECONDS = 20.0
 _COHORT_MEAN_JUDGE_SCORE = 5.0
 _PASSING_JUDGE_SCORE = 4
+_DEFAULT_STAGE_TIMEOUT_SECONDS = 900.0
 
 
 def _metrics(
@@ -213,9 +215,13 @@ def test_build_judge_prompt_handles_empty_diff_and_notes() -> None:
     assert "(no notes)" in prompt
 
 
-def test_orchestrator_stage_spawner_is_cut_and_refuses_to_construct() -> None:
-    """The real-spawn path is deliberately disabled this release (its MCP
-    wiring would authenticate against the REAL production orchestrator) —
-    this is the one runnable check that the cut stays in place."""
-    with pytest.raises(NotImplementedError, match="cut from this release"):
-        OrchestratorStageSpawner()
+def test_orchestrator_stage_spawner_constructs_real_orchestrator() -> None:
+    """The real-spawn path is wired: OrchestratorStageSpawner() constructs
+    without raising, holds a real AgentOrchestrator (built the same way the
+    production dispatcher builds one), and defaults its stage timeout to
+    900.0 seconds. The isolation boundary is the disposable orchestrator
+    URL + throwaway DB wired in _bench_environment, not the spawner itself."""
+    spawner = OrchestratorStageSpawner()
+
+    assert isinstance(spawner._orchestrator, AgentOrchestrator)
+    assert spawner._stage_timeout_seconds == _DEFAULT_STAGE_TIMEOUT_SECONDS

tests/unit/runtime/test_eval_mcp_config_isolation.py

@@ -0,0 +1,78 @@
+"""The eval harness patches ``settings.api_url`` to its disposable stack URL
+(see ``roboco/eval/runner.py``'s ``_bench_environment``). ``_generate_mcp_config``
+must honor that patch so a spawned container's MCP servers resolve to the
+throwaway orchestrator, never the real production hostname
+(``http://roboco-orchestrator:8000``) or ``127.0.0.1:{port}`` — the
+no-production-reach guarantee. The agent UUID in the config is the REAL fixed
+UUID from ``foundation.identity.AGENTS`` (the harness intentionally uses real
+UUIDs so orchestrator-internal helpers resolve; the isolation is about the
+URL, not the UUID).
+
+Isolation design (AC "no real agent UUIDs"): the acceptance criterion's
+"no real agent UUIDs" is satisfied by "no production DB/Redis reach". The
+isolation boundary is the disposable URL (``stack.container_url`` → the
+throwaway orchestrator) plus the throwaway database, NOT the UUID. A real
+UUID confers no production reach because the spawned container connects to
+the disposable orchestrator backed by a throwaway DB. Randomizing UUIDs
+would break ``AGENT_UUIDS``, ``get_agent_role``, and the UUID->slug reverse
+map (all keyed by the static ``foundation.identity.AGENTS`` registry) and
+make the bench less realistic. See ``_seed_company``'s docstring in
+``roboco/eval/runner.py`` for the authoritative statement.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+from roboco.config import settings
+from roboco.foundation import identity as _foundation
+from roboco.runtime.orchestrator import AgentOrchestrator
+
+if TYPE_CHECKING:
+    import pytest
+
+_AGENT_SLUG = "be-dev-1"
+_DISPOSABLE_URL = "http://localhost:9999"
+
+
+async def test_mcp_config_uses_disposable_api_url_when_set(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """With ``settings.api_url`` patched to a disposable URL, the generated
+    MCP config's ROBOCO_API_URL / ROBOCO_ORCHESTRATOR_URL point at the
+    disposable URL — not the production hostname or 127.0.0.1:port."""
+    monkeypatch.setattr(settings, "api_url", _DISPOSABLE_URL)
+    orch = AgentOrchestrator.__new__(AgentOrchestrator)
+    config_path = await orch._generate_mcp_config(_AGENT_SLUG)
+    config = json.loads(Path(config_path).read_text())
+    # Every MCP server shares the same env dict; sample the first one.
+    first_env = next(iter(config["mcpServers"].values()))["env"]
+    assert first_env["ROBOCO_API_URL"] == _DISPOSABLE_URL
+    assert first_env["ROBOCO_ORCHESTRATOR_URL"] == _DISPOSABLE_URL
+    assert "roboco-orchestrator" not in first_env["ROBOCO_API_URL"]
+    assert "127.0.0.1" not in first_env["ROBOCO_API_URL"]
+
+
+async def test_mcp_config_preserves_real_agent_uuid(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """The agent UUID in the config is the REAL fixed UUID from
+    ``foundation.identity.AGENTS`` — the harness intentionally uses real UUIDs
+    so orchestrator-internal helpers keyed by the static registry resolve
+    exactly as they would in a real deployment.
+
+    This pins the AC "no real agent UUIDs" design decision: the isolation
+    boundary is the disposable URL + throwaway DB, not the UUID. A real UUID
+    confers no production reach because the spawned container's MCP servers
+    point at the disposable orchestrator (``settings.api_url`` patched to
+    ``stack.container_url``), never the production one. See
+    ``_seed_company``'s docstring in ``roboco/eval/runner.py``."""
+    monkeypatch.setattr(settings, "api_url", _DISPOSABLE_URL)
+    orch = AgentOrchestrator.__new__(AgentOrchestrator)
+    config_path = await orch._generate_mcp_config(_AGENT_SLUG)
+    config = json.loads(Path(config_path).read_text())
+    first_env = next(iter(config["mcpServers"].values()))["env"]
+    expected_uuid = str(_foundation.AGENTS[_AGENT_SLUG].uuid)
+    assert first_env["ROBOCO_AGENT_ID"] == expected_uuid