2 Commits

Include renames

Author	SHA1	Message	Date
7 people	66f0287d11	[3e7a5064] Eval harness: wire the real-spawn path so python -m roboco.eval run works end-to-end (#703) ... * [431e73b7] Wire the real-spawn path: OrchestratorStageSpawner + disposable MCP config (#701) * [431e73b7] Wire the eval harness real-spawn path: OrchestratorStageSpawner + disposable MCP config _generate_mcp_config now prefers settings.api_url when set (both PROJECT_HOST_PATH branches), so spawned MCP servers resolve to the harness's disposable orchestrator URL instead of the real production hostname or 127.0.0.1:port. OrchestratorStageSpawner.__init__ replaces the NotImplementedError with a real AgentOrchestrator() constructed the same way the production dispatcher builds it. The runner module docstring + __main__.py docstring/run-subparser help drop the NOT-YET-FUNCTIONAL wording. A new unit test pins the no-production-reach guarantee: with settings.api_url patched, the MCP config's ROBOCO_API_URL/ROBOCO_ORCHESTRATOR_URL point at the disposable URL (not production), and the agent UUID is the real fixed UUID from foundation.identity.AGENTS. * [431e73b7] docs(eval): reflect the wired real-spawn path in tests map, CLAUDE.md, and CHANGELOG --------- Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> * [5cc75f71] Fix disposable orchestrator container-reachability + document real-UUID isolation design (#705) * [5cc75f71] Fix disposable orchestrator container-reachability + document real-UUID isolation * [5cc75f71] docs(eval-harness): document container-reachability fix + real-UUID isolation in map docs --------- Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> * [d96ec059] Rewrite stale eval-spawner pinning test to assert wired behavior (test-only, CI-green for PR #703) (#707) (#708) * [d96ec059] test(eval): assert OrchestratorStageSpawner constructs a real AgentOrchestrator Rewrite the stale pinning test that asserted the PRE-wiring NotImplementedError (removed by commit 488e9e2f when the real-spawn path was wired). The test now asserts the wired behavior: OrchestratorStageSpawner() construction succeeds, _orchestrator is an AgentOrchestrator instance, and _stage_timeout_seconds defaults to 900.0. Renamed from test_orchestrator_stage_spawner_is_cut_and_ refuses_to_construct to reflect the new contract. Test-only — no production code touched. * [d96ec059] docs(map): note test_scoring spawner pinning test in eval-harness map entry Add one clause to docs/map/tests.md's roboco/eval/ row naming tests/unit/eval/test_scoring.py::test_orchestrator_stage_spawner_constructs_real_orchestrator as the unit test that pins the OrchestratorStageSpawner wired-construction contract (construction succeeds, _orchestrator is an AgentOrchestrator, _stage_timeout_seconds defaults to 900.0). Consistent with the line's existing pattern of citing test_eval_mcp_config_isolation.py and test_eval_bench.py by name for the contracts they pin. The CHANGELOG #701 entry already covers the user-visible wiring; no CHANGELOG change needed. --------- Co-authored-by: roboco-app[bot] <302741806+roboco-app[bot]@users.noreply.github.com> Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> --------- Co-authored-by: roboco-app[bot] <302741806+roboco-app[bot]@users.noreply.github.com> Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech> Co-authored-by: Backend Documenter <be-doc@roboco.tech> Co-authored-by: Backend PM <be-pm@roboco.tech>	2026-07-26 17:14:09 +00:00
3 people	10f039c36f	feat(eval): golden-task eval harness + doctrine cohort stamp (#655) ... * fix(notifications): exponential backoff + CAS claim for expired-unacked re-escalation The sweep re-escalated every expired unacked ack-required notification on every ~60s tick, forever — the live incident: 3 fresh blocker escalations + Telegram DMs per minute from a static stale pile. Now each notification carries reescalation_count / last_reescalated_at / reescalation_delivered_count (migration 079): first fire at expiry, then doubling intervals from 1h capped at 24h, hard stop after ROBOCO_NOTIFICATION_MAX_REESCALATIONS (default 5) with one permanent log carrying attempts-vs-delivered so 'seen and ignored' is distinguishable from 'route never worked'. The due/wait/capped decision is a pure function in foundation/policy/communications.py. Per adversarial review, the attempt slot is claimed by compare-and-set (UPDATE ... WHERE reescalation_count = :n) BEFORE delivery — the previous draft leaned on the 60s dedup window, which never engages for BLOCKER_ESCALATION (_LOOP_PRONE_TYPES excludes it), so concurrent sweeps would have double-delivered. A lost claim skips delivery outright. Legacy rows read as count=0 and keep today's first-fire semantics. 61 tests incl. a two-session CAS race and a real alembic upgrade/downgrade round trip. * feat(budgets): per-task and per-project cost budgets (flag-gated) tasks.budget_usd + projects.monthly_budget_usd (migration 080, chained on 079; adds ix_agent_spawn_sessions_task_id since both enforcement seams filter on bare task_id). Behind ROBOCO_TASK_BUDGETS_ENABLED (default off, feature-flags card) — verifiably inert when off. Claim-time: a project-month-spend guard applies to WORK-STARTING claims only (i_will_work_on / i_will_plan) — per adversarial review, review/ doc/gate/inbound-PR claims are exempt so in-flight work can always finish reviewing and merging at cap. Spend counts closed sessions' estimated_cost_usd PLUS open sessions priced live from token snapshots (the original closed-only sum read parallel long sessions as $0). Sweep-side: the existing budget sweep also prices the active task's spend vs budget_usd (TaskType defaults when null); on breach the task is BLOCKED (HUMAN resolver, budget marker) BEFORE the graceful stop so the unclaim no-ops and the dispatcher never respawns onto it, and the CEO notification names both recovery steps. unblock on a budget-blocked task re-checks live spend and refuses while still over — no silent re-breach loop. Panel: budget inputs in both dialogs (0 rejected — a zero budget silently blocks everything), spend logic consolidated in TaskService.task_spend_usd. 42 new tests incl. a real-DB spend-query suite and a two-tick non-refire sweep test. * feat(eval): golden-task eval harness + doctrine cohort stamp roboco/eval: 6 BenchTaskSpec fixtures run through the real lifecycle in a disposable environment (the e2e_smoke harness's fake GitHub + local git origin + throwaway DB catalog — real isolation, not convention), scored deterministically (terminal status, revision_count, cycle time, tokens/cost via the agent_spawn_sessions task_id join) plus a local- model judge whose output is nested under a non_deterministic-marked object so cohort diffs don't read judge noise as regression. CLI: python -m roboco.eval run --role <slug> --cohort <name>. Source- checkout-only by declared posture (deptry-scoped ignore + a hard ImportError guard naming why; tests/ never ships in images or wheels). agent_spawn_sessions.doctrine_version (migration 081, chained on 080) is stamped at spawn-session finalize from the composed prompt layers — with the session's model column it identifies a cohort durably. Per adversarial review: bench runs patch the vault flags off (they were writing real markdown into the operator's vault), and the real-spawn OrchestratorStageSpawner is deliberately cut to NotImplementedError — spawned containers' MCP wiring resolves to the production orchestrator under real agent UUIDs, so real spawns wait for a dedicated follow-up; the injectable scripted spawner is the working path. Full suite 13852 passed / 94% coverage in the source worktree; deptry/mypy/xenon clean. --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>	2026-07-23 00:06:50 +02:00