mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
[431e73b7] Wire the real-spawn path: OrchestratorStageSpawner + disposable MCP config (#701)
* [431e73b7] Wire the eval harness real-spawn path: OrchestratorStageSpawner + disposable MCP config _generate_mcp_config now prefers settings.api_url when set (both PROJECT_HOST_PATH branches), so spawned MCP servers resolve to the harness's disposable orchestrator URL instead of the real production hostname or 127.0.0.1:port. OrchestratorStageSpawner.__init__ replaces the NotImplementedError with a real AgentOrchestrator() constructed the same way the production dispatcher builds it. The runner module docstring + __main__.py docstring/run-subparser help drop the NOT-YET-FUNCTIONAL wording. A new unit test pins the no-production-reach guarantee: with settings.api_url patched, the MCP config's ROBOCO_API_URL/ROBOCO_ORCHESTRATOR_URL point at the disposable URL (not production), and the agent UUID is the real fixed UUID from foundation.identity.AGENTS. * [431e73b7] docs(eval): reflect the wired real-spawn path in tests map, CLAUDE.md, and CHANGELOG --------- Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech>
This commit is contained in:
roboco-app[bot]
co-authored by
Backend Developer 1
Backend Documenter
Backend Developer 1
Backend Documenter
parent
401f8a2cc9
commit
488e9e2f11
@@ -8,6 +8,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
|
||||
### Added
|
||||
|
||||
- **The eval harness real-spawn path is wired (#701).** `python -m roboco.eval run` now works end-to-end for a developer-role cohort: `OrchestratorStageSpawner.__init__` replaces its `NotImplementedError` with a real `AgentOrchestrator()` constructed the same way the production dispatcher builds it, so `run_stage` drives a real `spawn_agent` per turn. The no-production-reach guarantee lives in `_generate_mcp_config`, which now prefers `settings.api_url` when set (both `PROJECT_HOST_PATH` branches) — the harness's `_bench_environment` patches `settings.api_url` to the disposable stack URL, so a spawned container's MCP servers resolve to the throwaway orchestrator instead of the real production hostname or `127.0.0.1:{port}`, even though `_seed_company` seeds agents under their real production UUIDs (correct — orchestrator-internal helpers keyed by the static registry resolve as in a real deployment; the isolation is the URL, not the UUID). A new `tests/unit/runtime/test_eval_mcp_config_isolation.py` pins the guarantee without a Docker daemon. The injectable scripted `StageSpawner` (`tests/e2e_smoke/test_eval_bench.py`) is unaffected — it injects its own `make_spawner` — and remains the unit-test fallback. The real spawn needs a Docker daemon + built agent images.
|
||||
|
||||
- **The Board Program registry — the board finally works (#689, #699 + the Phase-2/3 program train).** Board dormancy root-caused and fixed (#684: the research key never reached the container, the pitch verb was never prompted, auditor playbook curation had no dispatch path), then one generic registry/engine/loop (`BoardProgram`, `BoardProgramEngine`, migrations 087-088) replaces the bespoke roadmap/spotlight loops behavior-preserved, adds a LEARN ledger (per-item CEO decisions feed the next cycle's exploration prompt), per-project scoping with dual polarity (`projects.board_programs`: plain keys opt into project-scoped programs, `!key` opts out of an org-scoped program's outputs), and a single settings-store arming chokepoint (`program_armed`) so the panel toggle can never be silently vetoed by a legacy boot flag. Twelve programs ride it: **Pest Control** (PO bug hunts over the findings ledger and rework hotspots, rework-spike accelerated), **Spackle** (PO gap-fill audits of half-shipped surface area), **Scales** (PO monthly portfolio rebalance whose approvals reprioritize or cancel live backlog tasks), **Dogfood** (the PO walks the product with task-scoped browser tools), **Periscope** (HoM market briefs with mandatory source citations, feeding the roadmap prompt), **Megaphone** (HoM editorial calendar into the X queue), **Mirror** (HoM quarterly positioning audits), **Barfly** (HoM replies to adjacent X conversations, injection-screened, candidate-id-bound), **War Room** (HoM release campaigns as manual-cadence held drafts), **Coroner** (Auditor postmortems on bounce/cancel/budget events, drafting process-change playbooks), **Librarian** (Auditor playbook mining from recurring learnings), and **Sentinel** (Auditor weekly drift-watch quality reports). Every artifact is held; the CEO remains the only path to materialization; nothing auto-posts or auto-starts.
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -436,7 +436,7 @@ The fourteen programs: **Printer**/`roadmap` (PO, weekly cron, org) — the pre-
|
||||
|
||||
**Ponytail (bundled with Fable-mode).** Rides `ROBOCO_FABLE_MODE_ENABLED` — no separate flag. Vendors the ponytail "lazy senior dev" build-laziness doctrine (`agents/prompts/doctrine/ponytail.md` + ethos sibling, MIT, Copyright (c) 2026 DietrichGebert — trimmed, YAML frontmatter stripped) into every composed system prompt via `ponytail_doctrine_layer` (`roboco/agents/factories/_base.py`), slotted immediately after the Fable doctrine layer and gated on the same flag. Role-scoped: developers (`AgentRole.DEVELOPER`) get the full ladder (YAGNI → reuse-in-this-codebase → stdlib → native-platform → installed-dep → one-line → minimal); every other role gets the ethos-only cut (`ponytail-ethos.md`) — the code-mechanics rungs are dropped so they can't leak into prose artifacts (task plans, review notes, docs). Both files carry a 5-point RoboCo preamble (the ethos sibling adds a 6th: free-text field obligations) that makes the ladder yield to the Architectural Conventions Standard (placement), the 80% coverage gate + QA review + self-verification, the per-team design bar, task hygiene (everything-is-a-task / commits-linked / state-is-sacred), and reviewer feedback (`needs_revision` / `pr_fail` / `request_changes`) — the overlap mitigation is scoping, not deletion, and it rides ponytail's own "when NOT to be lazy" clause. Developer intensity is tunable via `ROBOCO_PONYTAIL_INTENSITY` (`lite` / `full` / `ultra`, default `full`; `roboco/config.py` `ponytail_intensity`, a string value — not a feature flag): `full` enforces the ladder, `lite` builds what's asked and names the lazier alternative, `ultra` is YAGNI-extremist (deletion before addition, challenge the requirement). Non-developers get no dial — `ultra` is wrong for prose artifacts, so the ethos runs a fixed restrained stance. Prompt-only: no hooks, no grok-path changes — ponytail adds no hook surface, so bundling it under the Fable flag changes only the composed prompt, not the spawn hooks. The Fable flag's description in `roboco/config.py` names both doctrines.
|
||||
|
||||
**Golden-task eval harness (source-checkout-only offline CLI).** `roboco/eval/` replays a fixed set of `BenchTaskSpec` fixtures (`roboco/eval/fixtures.py`) through the REAL delivery lifecycle in a disposable environment reused from `tests/e2e_smoke/harness.py` (fake GitHub REST, a real local git origin, a throwaway DB) — real isolation, not a mock. `EvalRunner.run_cohort` (`roboco/eval/runner.py`) scores each fixture on deterministic metrics (final status, `revision_count`, cycle time, tokens/cost via the `agent_spawn_sessions` task_id join) plus a local-model judge comparing the final PR diff + notes against the fixture's checked-in expectations, nested under a `"non_deterministic": true`-marked `"judge"` object so a naive cohort diff never mistakes judge noise for a real regression. `agent_spawn_sessions.doctrine_version` (migration 081) is stamped at spawn-session finalize from the composed prompt layers, so a cohort's model + doctrine combination (e.g. Fable-mode on vs. off) is durably identifiable after the fact. **Real-spawn is cut for this release**: `OrchestratorStageSpawner` raises `NotImplementedError` at construction — a real spawn's MCP wiring would resolve to the REAL production orchestrator under real agent UUIDs, unsafe for a bench run — so `python -m roboco.eval run` is wired but not yet functional; the only working path today is driving `EvalRunner` with an injected scripted `StageSpawner` from Python (see `tests/e2e_smoke/test_eval_bench.py`). Scoped to developer-role fixtures only (`run_cohort` refuses any other role) and only runs from a source checkout (`tests/e2e_smoke` isn't shipped in containers or wheels); bench runs also patch every vault flag off so a bench task/note/journal write never lands in the operator's real Obsidian vault.
|
||||
**Golden-task eval harness (source-checkout-only offline CLI).** `roboco/eval/` replays a fixed set of `BenchTaskSpec` fixtures (`roboco/eval/fixtures.py`) through the REAL delivery lifecycle in a disposable environment reused from `tests/e2e_smoke/harness.py` (fake GitHub REST, a real local git origin, a throwaway DB) — real isolation, not a mock. `EvalRunner.run_cohort` (`roboco/eval/runner.py`) scores each fixture on deterministic metrics (final status, `revision_count`, cycle time, tokens/cost via the `agent_spawn_sessions` task_id join) plus a local-model judge comparing the final PR diff + notes against the fixture's checked-in expectations, nested under a `"non_deterministic": true`-marked `"judge"` object so a naive cohort diff never mistakes judge noise for a real regression. `agent_spawn_sessions.doctrine_version` (migration 081) is stamped at spawn-session finalize from the composed prompt layers, so a cohort's model + doctrine combination (e.g. Fable-mode on vs. off) is durably identifiable after the fact. **Real-spawn is wired**: `OrchestratorStageSpawner` drives a real `AgentOrchestrator.spawn_agent` per turn (constructed the same way the production dispatcher builds it), and `_generate_mcp_config` honors the patched `settings.api_url` (set to the harness's disposable stack URL in `_bench_environment`) so a spawned container's MCP servers resolve to the throwaway orchestrator, never the REAL production one — even though `_seed_company` seeds agents under their REAL production UUIDs (correct: orchestrator-internal helpers keyed by the static registry resolve exactly as in a real deployment; the isolation is about the URL, not the UUID). `python -m roboco.eval run` works end-to-end for a developer-role cohort; it needs a Docker daemon + built agent images for the real spawn path. The injectable scripted `StageSpawner` (see `tests/e2e_smoke/test_eval_bench.py`) remains the unit-test fallback that proves the runner's polling/scoring/DB plumbing without touching Docker; `tests/unit/runtime/test_eval_mcp_config_isolation.py` pins the no-production-reach guarantee without a Docker daemon. Scoped to developer-role fixtures only (`run_cohort` refuses any other role) and only runs from a source checkout (`tests/e2e_smoke` isn't shipped in containers or wheels); bench runs also patch every vault flag off so a bench task/note/journal write never lands in the operator's real Obsidian vault.
|
||||
|
||||
**Env-branches ladder + EnvSyncEngine (default-off `ROBOCO_ENV_SYNC_ENABLED`).** Replaces a project's single `default_branch` with an ordered environment ladder: nullable `projects.environments` JSONB (migration 073), an ordered `list[{name, branch}]` where index 0 is the **head** rung (where dev/cell/leaf PRs land) and index -1 is the **prod** rung (where the gated release executor commits + tags); middle rungs are intermediates (qa/stag). A null ladder degenerates to a single-branch ladder synthesized from `default_branch` at read time (`roboco/models/env_branches.py`: `head_branch` / `prod_branch` / `ladder_pairs` / `promotion_chain`) — no backfill, byte-for-byte legacy behavior until the CEO declares a real split. Every former `default_branch` consumer now routes through the shim: the PR target and per-agent clone (`WorkspaceService.ensure_workspace` / `ensure_read_clone`), the CI branch, the release executor's clone/commit/tag target (`_ReleaseContext.prod_branch`) plus its full-chain head→…→prod promotion before bumping (`promote_env_chain`, fail-closed `promotion_failed` on a merge conflict), and `release_readiness`'s diff baseline (`prod..head` instead of `last_tag..HEAD`) with a tag-drift cross-check (`_tag_drift_gaps` — the last tag's commit vs. prod tip disagreeing flags a hotfix that landed on prod after the tag). `EnvSyncEngine` (`roboco/services/env_sync_engine.py`) cascades the ladder prod→…→head via GitHub's merges API: a clean merge auto-pushes straight to the lower rung, a conflict opens ONE idempotent sync PR + a Main-PM coordination task and stops that project's cascade for the cycle — the cascade's target is never the prod rung by construction, so "only the CEO merges master" still holds. Bounded + deduped per repo (one open env_sync task at a time). Panel: an environment-ladder editor on the project edit dialog.
|
||||
|
||||
|
||||
+1
-1
@@ -28,7 +28,7 @@ The pytest test suite for RoboCo: 571 test_*.py files across tests/foundation, t
|
||||
| tests/e2e_smoke/harness.py | E2E harness: E2EStack app + orchestrator client + per-test agent manifests; used by the e2e_smoke tier | ~520 |
|
||||
| tests/e2e_smoke/test_gitea_live.py | Live-Gitea contract suite for `GiteaProvider` — fully self-seeding (creates its own uniquely-named repo, pushes real commits) against a real Gitea instance; skipped unless `ROBOCO_GITEA_E2E_URL`/`ROBOCO_GITEA_E2E_TOKEN` are both set; exercises PR open → duplicate-409→422 reshape → list/filter → diff → comment review → commit-status CI reshape → squash merge → branch delete → release, plus the git-CLI Basic-auth extraheader claim | 250 |
|
||||
| tests/e2e_smoke/test_gitlab_live.py | Live-GitLab contract suite for `GitLabProvider` — mirrors `test_gitea_live.py`, self-seeding against a real GitLab instance (gitlab.com works, project deleted afterward best-effort); skipped unless `ROBOCO_GITLAB_E2E_URL`/`ROBOCO_GITLAB_E2E_TOKEN` are both set; exercises MR open → duplicate reshape → GitHub-shape adaptation → diff reassembly → note review → commit-status CI reshape → squash merge → branch delete → release → the oauth2 Basic-auth git-CLI claim | 265 |
|
||||
| roboco/eval/ (`fixtures.py`/`runner.py`/`__main__.py`) | Golden-task offline CLI bench harness (source-checkout-only, NOT a pytest suite) — replays fixed `BenchTaskSpec` fixtures through the real delivery lifecycle in a disposable environment REUSED from `tests/e2e_smoke/harness.py` (fake GitHub REST, a real local git origin, a throwaway DB), scoring on deterministic metrics + a local-model judge. `python -m roboco.eval run` is wired but not functional this release — `OrchestratorStageSpawner` raises `NotImplementedError` at construction (a real spawn would resolve to the production orchestrator under real agent UUIDs, unsafe for a bench run); the only working path today is driving `EvalRunner` with an injected scripted `StageSpawner` from Python. Scoped to developer-role fixtures only; only runs from a source checkout. See `tests/e2e_smoke/test_eval_bench.py`. | — |
|
||||
| roboco/eval/ (`fixtures.py`/`runner.py`/`__main__.py`) | Golden-task offline CLI bench harness (source-checkout-only, NOT a pytest suite) — replays fixed `BenchTaskSpec` fixtures through the real delivery lifecycle in a disposable environment REUSED from `tests/e2e_smoke/harness.py` (fake GitHub REST, a real local git origin, a throwaway DB), scoring on deterministic metrics + a local-model judge. `python -m roboco.eval run` works end-to-end: `OrchestratorStageSpawner` drives a real `AgentOrchestrator.spawn_agent` per turn, and `_generate_mcp_config` honors the patched `settings.api_url` (set to the harness's disposable stack URL in `_bench_environment`) so a spawned container's MCP servers resolve to the throwaway orchestrator, never the real production one — even though `_seed_company` seeds agents under their REAL production UUIDs (correct: orchestrator-internal helpers keyed by the static registry resolve as in a real deployment). Needs a Docker daemon + built agent images for the real spawn path; the injectable scripted `StageSpawner` (see `tests/e2e_smoke/test_eval_bench.py`) remains the unit-test fallback that proves the runner's plumbing without touching Docker. Scoped to developer-role fixtures only; only runs from a source checkout. | — |
|
||||
|
||||
## E2E smoke harness
|
||||
|
||||
|
||||
+9
-13
@@ -3,15 +3,14 @@
|
||||
python -m roboco.eval run --role <slug> --cohort <name> \\
|
||||
[--fixtures a,b] [--json-out path]
|
||||
|
||||
NOT YET FUNCTIONAL: the real-spawn path (``OrchestratorStageSpawner``) is
|
||||
deliberately cut — see ``roboco/eval/runner.py``'s module docstring's
|
||||
"Real-spawn status" section — because a real container spawn's MCP wiring
|
||||
would authenticate against the REAL production orchestrator, not this
|
||||
harness's disposable one. ``run`` will raise ``NotImplementedError`` once it
|
||||
reaches the first fixture. The only working path today is driving
|
||||
``EvalRunner`` with an injected scripted ``StageSpawner`` from Python (see
|
||||
``tests/e2e_smoke/test_eval_bench.py``); this CLI is wired for the day the
|
||||
follow-up lands, not for use today.
|
||||
Replays the golden-task fixtures against one agent through the REAL delivery
|
||||
lifecycle: ``OrchestratorStageSpawner`` drives a real ``AgentOrchestrator``
|
||||
container spawn per turn, and ``_generate_mcp_config`` honors the patched
|
||||
``settings.api_url`` so spawned MCP servers resolve to the harness's
|
||||
disposable orchestrator, never the real production one. Needs a Docker daemon
|
||||
+ built agent images. The injectable scripted ``StageSpawner`` (see
|
||||
``tests/e2e_smoke/test_eval_bench.py``) remains the unit-test fallback that
|
||||
proves the runner's plumbing without touching Docker.
|
||||
|
||||
Offline dev/ops tool: no panel surface, no feature flag. Also runs from a
|
||||
source checkout only (needs ``tests/e2e_smoke``, not shipped in containers
|
||||
@@ -35,10 +34,7 @@ def _parse_args(argv: list[str]) -> argparse.Namespace:
|
||||
|
||||
run = subparsers.add_parser(
|
||||
"run",
|
||||
help=(
|
||||
"Replay the golden-task fixtures against one agent "
|
||||
"[NOT YET FUNCTIONAL — real-spawn path is cut, see module docstring]"
|
||||
),
|
||||
help="Replay the golden-task fixtures against one agent",
|
||||
)
|
||||
run.add_argument(
|
||||
"--role", required=True, help="Agent slug under test (e.g. be-dev-1)"
|
||||
|
||||
+37
-42
@@ -28,23 +28,22 @@ Vault safety: ``_bench_environment`` also patches ``obsidian_vault_enabled``
|
||||
bench task/note/journal write never lands in the operator's real Obsidian
|
||||
vault even when the ambient deployment has vault flags armed.
|
||||
|
||||
Real-spawn status: CUT for this release. ``StageSpawner`` is the seam
|
||||
between a real container spawn and a scripted stand-in, and
|
||||
``OrchestratorStageSpawner`` — what would be the default, real
|
||||
implementation — raises ``NotImplementedError`` at construction: a spawned
|
||||
container's MCP servers resolve their orchestrator URL via
|
||||
``_generate_mcp_config`` (``PROJECT_HOST_PATH`` -> the REAL production
|
||||
hostname, or ``settings.port``), never the patched ``settings.api_url`` this
|
||||
harness's disposable stack listens on — combined with ``_seed_company``
|
||||
seeding agents under their REAL production UUIDs, a real spawn here could
|
||||
authenticate as e.g. be-dev-1 against the production orchestrator and act on
|
||||
real tasks. Fixing the spawn-env wiring is a dedicated follow-up. The ONLY
|
||||
working ``StageSpawner`` today is an injected scripted one (see
|
||||
``tests/e2e_smoke/test_eval_bench.py``) that drives the SAME real MCP flow/do
|
||||
tool functions e2e_smoke's ``ScriptedAgent`` uses — proving the runner's
|
||||
Real-spawn status: ``OrchestratorStageSpawner`` is the default, real
|
||||
``StageSpawner``: it drives one turn via the REAL
|
||||
``AgentOrchestrator.spawn_agent`` — the exact method the production
|
||||
dispatcher calls. ``_generate_mcp_config`` honors the patched
|
||||
``settings.api_url`` (set to the harness's disposable stack URL in
|
||||
``_bench_environment``), so a spawned container's MCP servers resolve to the
|
||||
throwaway orchestrator, never the real production one — even though
|
||||
``_seed_company`` seeds agents under their REAL production UUIDs (which is
|
||||
correct: orchestrator-internal helpers keyed by the static registry resolve
|
||||
exactly as they would in a real deployment). The injectable scripted
|
||||
``StageSpawner`` (see ``tests/e2e_smoke/test_eval_bench.py``) remains the
|
||||
unit-test fallback — it drives the SAME real MCP flow/do tool functions
|
||||
e2e_smoke's ``ScriptedAgent`` uses, proving the runner's
|
||||
polling/scoring/DB plumbing without touching Docker. ``python -m roboco.eval
|
||||
run`` therefore does not work yet; it is wired for the day the follow-up
|
||||
lands, not for use today.
|
||||
run`` works end to end for a developer-role cohort; it needs a Docker daemon
|
||||
+ built agent images for the real spawn path.
|
||||
|
||||
Scope cut: only developer-role fixtures are supported (``run_cohort``
|
||||
refuses any other role). QA/documenter/cell-PM only ever pick up a task a
|
||||
@@ -491,40 +490,36 @@ class StageSpawner(Protocol):
|
||||
|
||||
|
||||
class OrchestratorStageSpawner:
|
||||
"""CUT for this release — do not construct. See the ``NotImplementedError``
|
||||
raised below for exactly why, and the module docstring's "Real-spawn
|
||||
status" section.
|
||||
|
||||
This was meant to be the default, real ``StageSpawner``: drive one turn
|
||||
via the REAL ``AgentOrchestrator.spawn_agent`` — the exact method the
|
||||
production dispatcher calls — reusing its own ``_get_prompt_for_agent`` /
|
||||
"""The default, real ``StageSpawner``: drive one turn via the REAL
|
||||
``AgentOrchestrator.spawn_agent`` — the exact method the production
|
||||
dispatcher calls — reusing its own ``_get_prompt_for_agent`` /
|
||||
``_task_git_context`` helpers so the prompt and workspace mount are
|
||||
byte-for-byte what a real dispatch tick would build, then wait for the
|
||||
container to exit (or the stage timeout). The ``run_stage`` body below is
|
||||
otherwise correct and is left in place for the follow-up that fixes the
|
||||
wiring (see ``__init__``) rather than deleted — re-enable it there by
|
||||
removing the raise.
|
||||
container to exit (or the stage timeout).
|
||||
|
||||
Safe because ``_generate_mcp_config`` honors the patched
|
||||
``settings.api_url`` (set to the harness's disposable stack URL in
|
||||
``_bench_environment``), so a spawned container's MCP servers resolve to
|
||||
the throwaway orchestrator, never the real production one — even though
|
||||
``_seed_company`` seeds agents under their REAL production UUIDs (which
|
||||
is correct: orchestrator-internal helpers keyed by the static registry
|
||||
resolve exactly as they would in a real deployment).
|
||||
"""
|
||||
|
||||
_orchestrator: Any
|
||||
_stage_timeout_seconds: float
|
||||
|
||||
def __init__(self, stage_timeout_seconds: float = 900.0) -> None:
|
||||
raise NotImplementedError(
|
||||
"OrchestratorStageSpawner (the real-spawn path) is cut from this "
|
||||
"release: a spawned container's MCP servers connect via "
|
||||
"_generate_mcp_config, which resolves the orchestrator URL from "
|
||||
"PROJECT_HOST_PATH ('http://roboco-orchestrator:8000', the REAL "
|
||||
"production hostname) or settings.port — NEVER the patched "
|
||||
"settings.api_url this harness's disposable stack listens on. "
|
||||
"Combined with _seed_company seeding agents under their REAL "
|
||||
"production UUIDs, a real spawn here would authenticate as e.g. "
|
||||
"be-dev-1 against the production orchestrator and could act on "
|
||||
"real tasks. Fixing this belongs in a dedicated follow-up that "
|
||||
"makes the spawn env honor the patched stack; until then only "
|
||||
"the injectable scripted StageSpawner (see "
|
||||
"tests/e2e_smoke/test_eval_bench.py) is a working path."
|
||||
)
|
||||
from roboco.runtime.orchestrator import AgentOrchestrator
|
||||
|
||||
self._stage_timeout_seconds = stage_timeout_seconds
|
||||
# Constructed the same way the production dispatcher does
|
||||
# (bootstrap.py: ``AgentOrchestrator()``); the harness's
|
||||
# ``_bench_environment`` has already patched ``settings.database_*``
|
||||
# to the throwaway DB and ``settings.api_url`` to the disposable
|
||||
# stack URL, so the orchestrator's DB + MCP-config wiring resolve to
|
||||
# the bench's own environment, not production.
|
||||
self._orchestrator = AgentOrchestrator()
|
||||
|
||||
async def run_stage(self, *, task: dict[str, Any], agent_slug: str) -> None:
|
||||
from roboco.models.runtime import OrchestratorAgentState
|
||||
|
||||
@@ -3889,8 +3889,16 @@ class AgentOrchestrator:
|
||||
not-authorized error rather than 404. Git context is forwarded
|
||||
only as a fallback for tools that resolve project/branch from env.
|
||||
"""
|
||||
# MCP servers run inside agent containers, need to connect via Docker network
|
||||
if PROJECT_HOST_PATH:
|
||||
# MCP servers run inside agent containers, need to connect to the
|
||||
# orchestrator API. Prefer an explicit settings.api_url override —
|
||||
# production sets it to the container hostname, and the eval harness
|
||||
# patches it to its disposable in-process stack (see runner.py's
|
||||
# _bench_environment) so a spawned container's MCP servers resolve to
|
||||
# the throwaway orchestrator, never the real production one. Fall back
|
||||
# to the PROJECT_HOST_PATH / settings.port logic when it is unset.
|
||||
if settings.api_url:
|
||||
api_url = settings.api_url
|
||||
elif PROJECT_HOST_PATH:
|
||||
api_url = "http://roboco-orchestrator:8000"
|
||||
else:
|
||||
api_url = f"http://127.0.0.1:{settings.port}"
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
"""The eval harness patches ``settings.api_url`` to its disposable stack URL
|
||||
(see ``roboco/eval/runner.py``'s ``_bench_environment``). ``_generate_mcp_config``
|
||||
must honor that patch so a spawned container's MCP servers resolve to the
|
||||
throwaway orchestrator, never the real production hostname
|
||||
(``http://roboco-orchestrator:8000``) or ``127.0.0.1:{port}`` — the
|
||||
no-production-reach guarantee. The agent UUID in the config is the REAL fixed
|
||||
UUID from ``foundation.identity.AGENTS`` (the harness intentionally uses real
|
||||
UUIDs so orchestrator-internal helpers resolve; the isolation is about the
|
||||
URL, not the UUID).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from roboco.config import settings
|
||||
from roboco.foundation import identity as _foundation
|
||||
from roboco.runtime.orchestrator import AgentOrchestrator
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import pytest
|
||||
|
||||
_AGENT_SLUG = "be-dev-1"
|
||||
_DISPOSABLE_URL = "http://localhost:9999"
|
||||
|
||||
|
||||
async def test_mcp_config_uses_disposable_api_url_when_set(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""With ``settings.api_url`` patched to a disposable URL, the generated
|
||||
MCP config's ROBOCO_API_URL / ROBOCO_ORCHESTRATOR_URL point at the
|
||||
disposable URL — not the production hostname or 127.0.0.1:port."""
|
||||
monkeypatch.setattr(settings, "api_url", _DISPOSABLE_URL)
|
||||
orch = AgentOrchestrator.__new__(AgentOrchestrator)
|
||||
config_path = await orch._generate_mcp_config(_AGENT_SLUG)
|
||||
config = json.loads(Path(config_path).read_text())
|
||||
# Every MCP server shares the same env dict; sample the first one.
|
||||
first_env = next(iter(config["mcpServers"].values()))["env"]
|
||||
assert first_env["ROBOCO_API_URL"] == _DISPOSABLE_URL
|
||||
assert first_env["ROBOCO_ORCHESTRATOR_URL"] == _DISPOSABLE_URL
|
||||
assert "roboco-orchestrator" not in first_env["ROBOCO_API_URL"]
|
||||
assert "127.0.0.1" not in first_env["ROBOCO_API_URL"]
|
||||
|
||||
|
||||
async def test_mcp_config_preserves_real_agent_uuid(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The agent UUID in the config is the REAL fixed UUID from
|
||||
``foundation.identity.AGENTS`` — the harness intentionally uses real UUIDs
|
||||
so orchestrator-internal helpers keyed by the static registry resolve
|
||||
exactly as they would in a real deployment."""
|
||||
monkeypatch.setattr(settings, "api_url", _DISPOSABLE_URL)
|
||||
orch = AgentOrchestrator.__new__(AgentOrchestrator)
|
||||
config_path = await orch._generate_mcp_config(_AGENT_SLUG)
|
||||
config = json.loads(Path(config_path).read_text())
|
||||
first_env = next(iter(config["mcpServers"].values()))["env"]
|
||||
expected_uuid = str(_foundation.AGENTS[_AGENT_SLUG].uuid)
|
||||
assert first_env["ROBOCO_AGENT_ID"] == expected_uuid
|
||||
Reference in New Issue
Block a user