[3e7a5064] Eval harness: wire the real-spawn path so python -m roboco.eval run works end-to-end (#703)

* [431e73b7] Wire the real-spawn path: OrchestratorStageSpawner + disposable MCP config (#701)

* [431e73b7] Wire the eval harness real-spawn path: OrchestratorStageSpawner + disposable MCP config

_generate_mcp_config now prefers settings.api_url when set (both
PROJECT_HOST_PATH branches), so spawned MCP servers resolve to the
harness's disposable orchestrator URL instead of the real production
hostname or 127.0.0.1:port. OrchestratorStageSpawner.__init__ replaces
the NotImplementedError with a real AgentOrchestrator() constructed the
same way the production dispatcher builds it. The runner module
docstring + __main__.py docstring/run-subparser help drop the
NOT-YET-FUNCTIONAL wording. A new unit test pins the no-production-reach
guarantee: with settings.api_url patched, the MCP config's
ROBOCO_API_URL/ROBOCO_ORCHESTRATOR_URL point at the disposable URL (not
production), and the agent UUID is the real fixed UUID from
foundation.identity.AGENTS.

* [431e73b7] docs(eval): reflect the wired real-spawn path in tests map, CLAUDE.md, and CHANGELOG

---------

Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>

* [5cc75f71] Fix disposable orchestrator container-reachability + document real-UUID isolation design (#705)

* [5cc75f71] Fix disposable orchestrator container-reachability + document real-UUID isolation

* [5cc75f71] docs(eval-harness): document container-reachability fix + real-UUID isolation in map docs

---------

Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>

* [d96ec059] Rewrite stale eval-spawner pinning test to assert wired behavior (test-only, CI-green for PR #703) (#707) (#708)

* [d96ec059] test(eval): assert OrchestratorStageSpawner constructs a real AgentOrchestrator

Rewrite the stale pinning test that asserted the PRE-wiring
NotImplementedError (removed by commit 488e9e2f when the real-spawn
path was wired). The test now asserts the wired behavior:
OrchestratorStageSpawner() construction succeeds, _orchestrator is
an AgentOrchestrator instance, and _stage_timeout_seconds defaults
to 900.0. Renamed from test_orchestrator_stage_spawner_is_cut_and_
refuses_to_construct to reflect the new contract. Test-only — no
production code touched.

* [d96ec059] docs(map): note test_scoring spawner pinning test in eval-harness map entry

Add one clause to docs/map/tests.md's roboco/eval/ row naming
tests/unit/eval/test_scoring.py::test_orchestrator_stage_spawner_constructs_real_orchestrator
as the unit test that pins the OrchestratorStageSpawner wired-construction
contract (construction succeeds, _orchestrator is an AgentOrchestrator,
_stage_timeout_seconds defaults to 900.0). Consistent with the line's
existing pattern of citing test_eval_mcp_config_isolation.py and
test_eval_bench.py by name for the contracts they pin. The CHANGELOG #701
entry already covers the user-visible wiring; no CHANGELOG change needed.

---------

Co-authored-by: roboco-app[bot] <302741806+roboco-app[bot]@users.noreply.github.com>
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>

---------

Co-authored-by: roboco-app[bot] <302741806+roboco-app[bot]@users.noreply.github.com>
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>
Co-authored-by: Backend PM <be-pm@roboco.tech>
This commit is contained in:
roboco-app[bot]
2026-07-26 17:14:09 +00:00
committed by GitHub
co-authored by roboco-app[bot] <302741806+roboco-app[bot]@users.noreply.github.com> Backend Developer 1 Backend Documenter roboco-app[bot] <302741806+roboco-app[bot]@users.noreply.github.com> Backend PM
parent b3dda00e41
commit 66f0287d11
10 changed files with 183 additions and 69 deletions
+11 -1
View File
@@ -272,6 +272,12 @@ class E2EStack:
workspaces_root: Path
db_url: str
github: _FakeGitHub
# URL spawned agent containers use to reach this in-process orchestrator.
# base_url stays 127.0.0.1 (host-side test clients); container_url is
# host.docker.internal so containers on the roboco_default bridge can
# resolve the host's published port (the server binds 0.0.0.0 to accept
# both). Set by build_e2e_stack alongside base_url.
container_url: str = ""
def workspace_of(self, project_slug: str, team: str, agent_slug: str) -> Path:
return self.workspaces_root / project_slug / team / agent_slug
@@ -415,10 +421,13 @@ def build_e2e_stack(
# uvloop when installed, and this in-thread server has crashed CI with a
# uvloop/asyncpg segfault (uvloop 0.22 + asyncpg 0.31 + Python 3.13) —
# mirror the production default instead of picking up uvloop implicitly.
# Bind 0.0.0.0 so spawned agent containers on the roboco_default bridge
# can reach the server via host.docker.internal (host-gateway). Host-side
# clients still connect via 127.0.0.1 (base_url) — 0.0.0.0 accepts both.
server = uvicorn.Server(
uvicorn.Config(
app,
host="127.0.0.1",
host="0.0.0.0",
port=port,
log_level="warning",
loop=settings.uvicorn_loop,
@@ -448,6 +457,7 @@ def build_e2e_stack(
workspaces_root=workspaces,
db_url=_test_database_url,
github=gh,
container_url=f"http://host.docker.internal:{port}",
)
finally:
server.should_exit = True