# tests slice ## Purpose The pytest test suite for RoboCo: 571 test_*.py files across tests/foundation, tests/integration (+services/+v1), tests/property, and tests/unit (mirroring the roboco/ package layout). This slice covers the test INFRASTRUCTURE only — the two conftest.py layers, the JSON smoke-trace fixture, the two standalone shell smoke scripts, the pyproject pytest/coverage config, and the Makefile gates — plus the categorical structure of the 571 test files (NOT every individual test, per the slice scope). The suite is async-first (pytest-asyncio auto mode), coverage-gated at 80% via sys.monitoring, and DB tests run against an ephemeral real Postgres+pgvector DB created per session. ## Files | Path | Role | LOC | |---|---|---| | tests/conftest.py | Top-level conftest: provisions ephemeral Postgres+pgvector test DB, db_session + smoke_test_batch fixtures, skip-when-PG-unreachable hook | 510 | | tests/unit/services/optimal_brain/conftest.py | Single-line docstring conftest for optimal_brain unit tests (no fixtures) | 1 | | tests/__init__.py | Empty package marker (all 14 tests/*/__init__.py are empty) | 0 | | tests/fixtures/2026-05-08-smoke-trace.json | Synthesized 11-bug smoke trace fixture consumed by test_lifecycle_smoke_replay; pins (verb,role,task setup,expected envelope) per bug with a replay_kind taxonomy | 194 | | tests/integration/test_bash_guard_message.sh | Standalone bash smoke: asserts docker/scripts/bash-guard-hook.sh denial blocks are <=3 lines (token-budget guard) | 43 | | tests/integration/test_stop_hook_verb_names.sh | Standalone bash smoke: asserts stop-hook.sh + bash-guard-hook.sh list only current gateway verbs (no pre-gateway legacy verb names) | 36 | | pyproject.toml | [tool.pytest.ini_options] (asyncio_mode=auto, testpaths, cov addopts) + [tool.coverage.run] (sysmon core, omit list) — the coverage-architecture source of truth | 450 | | Makefile | quality/quality-fast/panel-gate/test/test-3.10..3.14 targets wiring pytest+ruff+mypy+xenon+vulture into the merge gate | 547 | | tests/foundation/ | 21 structural/parity tests: agents_config parity, lifecycle consumer/generator parity, role-set parity, seed-orchestrator parity, tracing-verb parity, route-guard consolidation, smoke replay, lifecycle spec, pr_review_gate, identity, communications, journaling, cell_teams, agent_loop, validate, task_completeness | | | tests/integration/ | 100 real-DB + FastAPI integration tests: full lifecycle real DB (_StubGit), task_service_*, *_routes, migrations 013-049, sequencing, conventions e2e, metrics, release, ci_watch, dep_update, prompter_live, secretary | | | tests/integration/services/ | 10 autonomy-engine integration tests: ci_watch_engine/notify/source, dep_update_engine/probe/source, docs_sync_engine, external_pr_repo_dedup, project_autonomy_update, active_task_owns_branch_scoping | | | tests/integration/v1/ | 1 FastAPI TestClient e2e (test_full_pending_to_completed) exercising all six v1 routers via stateful mocked Choreographer/ContentActions through the full hand-off chain | | | tests/property/ | 2 deterministic property tests (no hypothesis dep): state-machine invariants (orphan/terminal/reachability/random-walk) + tracing-completeness over smoke_test_batch | | | tests/unit/ | ~430 unit tests mirroring roboco/: agents, api (+routes/+schemas), billing, config, conventions, db, enforcement, events, foundation/policy, gateway (105), llm, mcp_servers, migrations, models, runtime (64), scripts, services (120), templates, utils | | | tests/unit/gateway/ | 105 Choreographer/verb-runner unit tests — the largest single cluster: every intent verb's guards, envelopes, evidence, claim locks, lane barrier, pr gate, conventions gate, content actions | | | tests/unit/runtime/ | 64 orchestrator unit tests: spawn/manifest/cwd/worktree, reaper, respawn persistence, rate-limit/overload sweeps, ci_watch/dep_update/release/self_heal loops, no_spawn_human_roles, per_dev_lane_queue, readopt_running_agents | | | tests/unit/services/ | 120+ service unit tests: task, git (+worktree), workspace, work_session, release_executor/readiness/manager, sequencing, conventions, playbook, notification, rate_limit_tracker, optimal_brain/ (10), forge/ (`test_gitea_provider.py`, `test_gitlab_provider.py`, `test_router.py` — mocked-transport coverage of the forge package, distinct from the live contract suites below) | | | tests/e2e_smoke/ | Scripted-agent smoke tests against the live in-process orchestrator; separate from the default pytest collection (run by the PR/NAS smoke gate) | | | tests/e2e_smoke/harness.py | E2E harness: E2EStack app + orchestrator client + per-test agent manifests; used by the e2e_smoke tier | ~520 | | tests/e2e_smoke/test_gitea_live.py | Live-Gitea contract suite for `GiteaProvider` — fully self-seeding (creates its own uniquely-named repo, pushes real commits) against a real Gitea instance; skipped unless `ROBOCO_GITEA_E2E_URL`/`ROBOCO_GITEA_E2E_TOKEN` are both set; exercises PR open → duplicate-409→422 reshape → list/filter → diff → comment review → commit-status CI reshape → squash merge → branch delete → release, plus the git-CLI Basic-auth extraheader claim | 250 | | tests/e2e_smoke/test_gitlab_live.py | Live-GitLab contract suite for `GitLabProvider` — mirrors `test_gitea_live.py`, self-seeding against a real GitLab instance (gitlab.com works, project deleted afterward best-effort); skipped unless `ROBOCO_GITLAB_E2E_URL`/`ROBOCO_GITLAB_E2E_TOKEN` are both set; exercises MR open → duplicate reshape → GitHub-shape adaptation → diff reassembly → note review → commit-status CI reshape → squash merge → branch delete → release → the oauth2 Basic-auth git-CLI claim | 265 | | roboco/eval/ (`fixtures.py`/`runner.py`/`__main__.py`) | Golden-task offline CLI bench harness (source-checkout-only, NOT a pytest suite) — replays fixed `BenchTaskSpec` fixtures through the real delivery lifecycle in a disposable environment REUSED from `tests/e2e_smoke/harness.py` (fake GitHub REST, a real local git origin, a throwaway DB), scoring on deterministic metrics + a local-model judge. `python -m roboco.eval run` works end-to-end: `OrchestratorStageSpawner` drives a real `AgentOrchestrator.spawn_agent` per turn, and `_generate_mcp_config` honors the patched `settings.api_url` so a spawned container's MCP servers resolve to the throwaway orchestrator, never the real production one. **Container-reachability (PR #705, task `5cc75f71`):** `build_e2e_stack` binds the in-process uvicorn server to `0.0.0.0` (not `127.0.0.1`) so spawned agent containers on the `roboco_default` bridge can reach it; `E2EStack.container_url` is `http://host.docker.internal:{port}` (host-side `base_url` stays `127.0.0.1`), `_bench_environment` patches `settings.api_url` to `stack.container_url`, and `_build_mount_args` adds `--add-host host.docker.internal:host-gateway` to every agent `docker run` argv (Docker 20.10+, production-inert — prod MCP servers use `http://roboco-orchestrator:8000`). **Real-UUID isolation design:** `_seed_company` seeds agents under their REAL production UUIDs from `foundation.identity.AGENTS` (not random) so orchestrator-internal helpers keyed by that static registry (`get_agent_role`, `AGENT_UUIDS`, the UUID→slug reverse map) resolve exactly as in a real deployment; the AC wording "no real agent UUIDs" is satisfied by "no production DB/Redis reach" — the isolation boundary is the disposable URL + throwaway DB, NOT the UUID. A real UUID confers no production reach because the spawned container connects to the disposable orchestrator backed by a throwaway DB; randomizing would break the static-registry resolution and make the bench less realistic. `tests/unit/runtime/test_eval_mcp_config_isolation.py` pins this. Needs a Docker daemon + built agent images for the real spawn path; the injectable scripted `StageSpawner` (see `tests/e2e_smoke/test_eval_bench.py`) remains the unit-test fallback that proves the runner's plumbing without touching Docker. `tests/unit/eval/test_scoring.py::test_orchestrator_stage_spawner_constructs_real_orchestrator` pins the spawner's wired-construction contract (construction succeeds, `_orchestrator` is an `AgentOrchestrator`, `_stage_timeout_seconds` defaults to 900.0). Scoped to developer-role fixtures only; only runs from a source checkout. | — | ## E2E smoke harness The `tests/e2e_smoke/` tier runs scripted agents against a real in-process FastAPI app and orchestrator. It is **not** collected by the default `uv run pytest` invocation; it is exercised separately by the PR gate / NAS smoke run. The harness in `tests/e2e_smoke/harness.py` builds an `E2EStack`, mounts the orchestrator client, and gives each test a per-agent manifest. **Live-forge contract suites** (`test_gitea_live.py` / `test_gitlab_live.py`) are a different animal from the scripted-agent harness above — no `E2EStack`, no orchestrator, no manifests. Each drives its `GitProvider` implementation directly against a REAL forge instance (a dockerized `gitea/gitea` or real `gitlab.com`), fully self-seeding (creates its own repo/project, pushes real commits via subprocess `git`, tears down after). Both are `pytest.mark.skipif`'d off unless their pair of `ROBOCO_{GITEA,GITLAB}_E2E_{URL,TOKEN}` env vars is set, so a normal CI/local run never attempts network I/O against a real forge. These are what caught the slash-encoding (branch names with `/` need `quote()`-encoding before hitting Gitea's router) and the http-vs-https scheme gaps that the mocked-transport unit tests in `tests/unit/services/forge/` could not have found, since those mock the HTTP layer entirely. Recent harness hardening for the auditor-revival slice (PR #498, task `8323cd50`): - `tests/conftest.py` now catches a missing `pgvector` extension when creating the ephemeral test database and continues with a warning. The core schema does not require it and the e2e smoke suite does not exercise RAG, so lightweight Postgres sandboxes can run the suite without the pgvector package. - `tests/e2e_smoke/harness.py` clears any host-supplied `ROBOCO_AGENT_TOKEN` from the environment before a `ScriptedAgent` re-imports `roboco.mcp.flow_server`. The host agent may carry a real token for the test runner's identity; `flow_server` reads it before each call and forwards it in `X-Agent-Token`. That token does not match the ephemeral test agent IDs and causes `401 Unauthorized`, so `ScriptedAgent._module` pops it to keep scripted agents in unsigned-token mode. This guard was accidentally dropped in an earlier auditor-revival commit and restored by PR #502. - `tests/e2e_smoke/test_auditor_triggers.py` exercises both scheduled and reactive auditor dispatch through a `_fresh_orchestrator` helper. Because the helper constructs `AgentOrchestrator` via `__new__` to bypass normal initialization, `AgentOrchestrator.__new__` now pre-initializes the private attributes `_instances`, `_last_audit_spawn_at`, and `_notification_spawn_at` that `_is_agent_active`, `_dispatch_audit_work`, and notification-cooldown paths read. The helper still explicitly sets `_instances = {}` and `_last_audit_spawn_at = None` for clarity, but this is no longer required for correctness. The two tests are intentionally **synchronous** (no `@pytest.mark.asyncio`) because the harness already enters an async loop via `run_db`; the test body uses `asyncio.run` on an inner coroutine. Internal API calls use the raw `_SYSTEM_API_HEADERS` (system identity only) rather than the production `_system_api_headers` helper, because dev mode rejects an `UNSIGNED` `X-Agent-Token`. Settings are patched at `api_url` (the input to the computed `internal_api_url` property), and notifications are matched with `NotificationType.ALERT` instead of the string `"ALERT"`. ## Key Symbols | Name | Kind | File:Line | Responsibility | |---|---|---|---| | _TEST_DB_HOST/_TEST_DB_PORT/_TEST_DB_USER/_TEST_DB_PASSWORD/_TEST_DB_ADMIN_DB | module constants | tests/conftest.py:84 | Env-overridable Postgres test endpoint discovery; defaults host=localhost port=15432 user=roboco password=roboco admin-db=postgres (changed from :5432/$USER since baseline) | | _postgres_reachable | function | tests/conftest.py:91 | TCP-probe the configured Postgres endpoint with a 1s timeout; returns False on OSError | | _PG_AVAILABLE | module constant | tests/conftest.py:114 | Module-level liveness flag computed once at import; gates fixture self-skip + collection-time skip marking | | _warn_if_pg_unavailable | function | tests/conftest.py:101 | Added in 536bbb64: emits a `warnings.warn` at import time when PG is unreachable, so an all-skips green run surfaces a visible warning instead of passing silently; tested by tests/unit/test_conftest_db_unreachable_warning.py | | _build_url | function | tests/conftest.py:108 | Build a postgresql+asyncpg:// URL with optional password for a given database name | | _test_database_url | fixture (session, async) | tests/conftest.py:118 | Session-scoped: CREATE DATABASE roboco_test__, enable pgvector, Base.metadata.create_all + manually ADD COLUMN acceptance_criteria_status/qa_evidence_inspected (migration 006, not on ORM), yield URL, DROP DATABASE on teardown (force-terminate stragglers first); self-skips if PG unreachable | | db_session | fixture (function, async) | tests/conftest.py:222 | Per-test AsyncSession bound to the ephemeral DB; no pool_pre_ping (avoids un-awaited cancel RuntimeWarning); rollback + dispose on teardown | | smoke_test_batch | fixture (function, async) | tests/conftest.py:247 | Seeds 1 project + system agent + dev/qa/pm agents + 1 COMPLETED task + 3 audit_log rows + 3 journals/entries (reflect/learning/decision) + raw-SQL sets migration-006 columns; the only fixture that COMMITs; yields [task_id] | | pytest_collection_modifyitems | hook | tests/conftest.py:490 | Collection-time skip marker for any item whose fixturenames intersect {db_session, smoke_test_batch} when _PG_AVAILABLE is False — does NOT cover _test_database_url direct requesters | ## Data Flow pytest invocation (make quality/make test/uv run pytest) reads [tool.pytest.ini_options] in pyproject.toml: asyncio_mode=auto makes every async def test_ run on the asyncio loop, testpaths=["tests"] scopes collection, addopts="--cov=roboco --cov-report=term-missing" attaches coverage with core=sysmon. At import time tests/conftest.py probes Postgres via _postgres_reachable() and sets _PG_AVAILABLE. The session fixture _test_database_url opens an asyncpg admin connection, CREATEs an ephemeral DB roboco_test__, enables pgvector, builds schema via Base.metadata.create_all (NOT alembic — sidesteps documented drift in migration 001 enum casing and 008 nonexistent agents.skills), manually ALTERs the migration-006 columns the ORM doesn't map, yields the URL, and DROPs the DB on session teardown. Per-test, db_session builds a fresh engine+AsyncSession against that URL, yields it, rolls back + disposes. smoke_test_batch seeds a full tracing-contract-compliant completed task (committing). pytest_collection_modifyitems marks db_session/smoke_test_batch requesters as skipped when PG is down. Coverage flows: sys.monitoring core counts async-aware line coverage, the omit list excludes infra-bound modules (orchestrator.py, git.py, workspace.py, mcp/*, agents/*, optimal.py, etc.) from the 80% --cov-fail-under gate, deferring their coverage to NAS smoke runs. The shell scripts are NOT invoked by pytest — they are standalone smoke scripts run manually/CI against docker/scripts/*.sh. The JSON fixture is loaded by tests/foundation/test_lifecycle_smoke_replay.py via _load_fixture() and dispatched per record by replay_kind into spec.Decision assertions. ## Mermaid ```mermaid flowchart TD A["make quality/uv run pytest"] --> P["pyproject.toml
[tool.pytest.ini_options]
asyncio_mode=auto
testpaths=tests
--cov=roboco"] P --> C["tests/conftest.py (import)"] C --> R["_postgres_reachable()
→ _PG_AVAILABLE"] R -- "False" --> S["pytest_collection_modifyitems
skip db_session/smoke_test_batch items"] R -- "True" --> F["_test_database_url fixture (session)"] F --> F1["asyncpg admin: CREATE DB roboco_test_pid_rand"] F1 --> F2["enable pgvector"] F2 --> F3["Base.metadata.create_all
+ ALTER migration-006 cols"] F3 --> F4["yield URL"] F4 --> D["db_session fixture (per test)"] F4 --> SB["smoke_test_batch fixture (seeds + COMMITs)"] F4 --> U["_test_database_url direct requesters
(claim_lock, board_review_gate)"] D --> T["tests/integration, tests/property
real-DB tests"] SB --> PT["test_tracing_completeness
6-bullet contract"] F4 -- teardown --> X["pg_terminate_backend + DROP DATABASE"] subgraph cov["Coverage architecture [tool.coverage.run]"] CO["core=sysmon (async-accurate)"] OM["omit: orchestrator, git, workspace,
mcp, agents, optimal, websocket,
stream_bus, cli, alembic"] G["--cov-fail-under=80 gate"] end P --> cov subgraph tiers["Test tiers"] TF["tests/foundation (21) — parity + smoke replay"] TI["tests/integration (100) — real DB + TestClient"] TP["tests/property (2) — deterministic invariants"] TU["tests/unit (~430) — mirror roboco/"] end T --> tiers S --> SK["skip: PG unreachable"] ``` ## Logical Tree ``` tests/ ├── __init__.py (empty package marker) ├── conftest.py [TOP-LEVEL — DB provisioning, db_session, smoke_test_batch, skip hook] ├── fixtures/ │ └── 2026-05-08-smoke-trace.json [11 synthesized bug records, replay_kind taxonomy] ├── e2e_smoke/ (scripted-agent smoke tests against live orchestrator; not collected by default pytest run) │ ├── harness.py [E2EStack + orchestrator client + per-test agent manifests] │ ├── arcs.py [seed helpers for company/project/task] │ └── test_auditor_triggers.py [scheduled + reactive auditor dispatch smoke tests] ├── foundation/ (21 tests — structural/parity gates) │ ├── test_lifecycle_smoke_replay.py [consumes the JSON fixture, dispatch by replay_kind] │ ├── test_lifecycle_spec.py/test_lifecycle_generators.py/test_lifecycle_consumer_parity.py │ ├── test_agents_config_parity.py/test_role_set_parity.py/test_seed_orchestrator_parity.py │ ├── test_tracing_verb_parity.py/test_tracing.py │ ├── test_route_guard_consolidation.py/test_role_reexport.py │ ├── test_pr_review_gate.py/test_task_completeness.py/test_validate.py │ ├── test_identity.py/test_cell_teams.py/test_agent_loop.py │ └── test_communications.py/test_communications_consumers.py/test_journaling.py/test_journaling_consumers.py ├── integration/ (100 + 9 services + 1 v1) │ ├── _StubGit pattern (test_full_lifecycle_real_db, test_lifecycle_real_db) │ ├── test_foundation_phase1..4_smoke.py [tiered package-layout gates] │ ├── test_migration_013/014/016/028 + batch_intake/ci_watch/dep_update/observability │ ├── test_*_routes.py (agents, dashboard, docs, git, journal, kanban, notifications, pitch, product, project, release, research, secretary, stream, tasks, work_session, orchestrator, prompter_live) │ ├── test_task_service_* [basics, transitions, lifecycle_misc, misc, background, no_silent_fallback, route_orchestration] │ ├── services/ [ci_watch_engine/notify/source, dep_update_engine/probe/source, docs_sync_engine, external_pr_repo_dedup, project_autonomy_update, active_task_owns_branch_scoping] │ ├── v1/test_full_pending_to_completed.py [TestClient e2e, all 6 v1 routers, stateful mocks] │ ├── test_bash_guard_message.sh [standalone smoke] │ └── test_stop_hook_verb_names.sh [standalone smoke] ├── property/ (2 — deterministic, no hypothesis) │ ├── test_state_machine_invariants.py [6 invariants: orphan/terminal/reachability/random-walk/self-loop] │ └── test_tracing_completeness.py [6-bullet tracing contract over smoke_test_batch] ├── e2e_smoke/ (scripted-agent lifecycle smoke — env-gated) │ ├── conftest.py [collection gate: skips unless ROBOCO_E2E_SMOKE=1] │ ├── harness.py [in-process API, ephemeral Postgres, bare git origin, fake GitHub REST] │ ├── arcs.py [canonical-company seeding + dev/qa/pm/reviewer lifecycle helpers] │ ├── test_dev_lifecycle.py [scenario 1: leaf dev arc → awaiting_pm_review] │ ├── test_pm_merge_chain.py [scenarios 2/2b: PR-gate turn cut + serial root merges] │ ├── test_root_ceo_chain.py [scenario 3: pr_fail → rework → real approve-and-merge] │ ├── test_megatask_umbrella.py [scenario 4: MegaTask sequencing + umbrella closure] │ ├── test_notification_coordination_events.py [DB-truth checks for restored coordination-event ALERTs] │ ├── test_auditor_triggers.py [scheduled sweep + reactive QA-fail alert paths spawn auditor] │ ├── test_auth_gate_coverage.py, test_background_engines.py, test_data_integrity.py, test_feature_spotlight.py, test_flow_verb_timeout.py, test_git_workflow.py, test_sandbox_image_tags.py, test_sandbox_on_demand.py, test_state_machine.py, test_video_pipeline.py [other e2e smoke scenarios] │ └── test_bash_guard_message.sh / test_stop_hook_verb_names.sh [NOT here — these live in tests/integration/] └── unit/ (~430 — mirrors roboco/) ├── test_* (11 top-level: agents_config, bootstrap, config_properties, enum_migration_parity, exceptions, logging, no_deleted_tool_names, notification_dedup, notification_dedup_refire, notification_delivery_refire, toolchain_flag) ├── agent_sdk/ (11) — grok_cli/intake/secretary/manifest/prompt_guard/usage_sync/verb_circuit_breaker ├── agents/ (4) — autogen_prompt_layer, briefing_cluster, conventions_ambient_injection, tool_load_directive ├── api/ (30) + routes/ (3) + routes/v1/ (11) + schemas/ (2) + schemas/v1/ (1) — middleware, deps, errors, websocket_*, schemas, v1 flow router tests ├── billing/ (1) — pricing ├── config/ (6) — ci_watch/conventions/dep_update/docs_sync/org_memory/release_manager flag tests ├── conventions/ (10) — classify_python/ts, cli, cli_smoke, custom, hygiene, modularity, placement, runner, scan ├── db/ (1) — respawn_tracker_table ├── enforcement/ (4) — a2a_access, journal_perms, task_lifecycle, task_ownership ├── events/ (2) — bus, handlers ├── foundation/policy/ (2) + content/ (6) + conventions/ (3) — pure policy models ├── gateway/ (105) — Choreographer/verb-runner guard + envelope + evidence surface ├── llm/ (4) + providers/ (3) — metrics_holder, probe_ollama_tags, providers, toon_adapter, grok_auth/cli_config/cli_usage ├── mcp_servers/ (10) — flow/do/search/intake/secretary servers, circuit breakers, envelope_on_4xx, intent_public_mapping ├── migrations/ (1) — structured_content_backfill ├── models/ (7) — events, journal, llm, misc, product, task_create_completeness, transcription ├── runtime/ (64) — orchestrator spawn/reaper/loops; per_dev_lane_queue, respawn_persistence, readopt_running_agents, gateway_health ├── scripts/ (2) — bash_guard, verify_postgres_enums ├── services/ (~120) + optimal_brain/ (10 + conftest) — task, git (+worktree family), workspace, work_session, release_* executor/readiness/manager/proposal + docs_sync hook, sequencing, conventions, playbook, notification, rate_limit, optimal_brain ├── templates/ (2) — pr_internal, pr_root └── utils/ (2) — converters, crypto ``` ## Dependencies - Internal: roboco.db.tables (AgentTable, AuditLogTable, JournalEntryTable, JournalTable, ProjectTable, TaskTable), roboco.db.base.Base (metadata.create_all), roboco.models.base (AgentRole, AgentStatus, TaskStatus, TaskType, Team, Complexity, JournalEntryType, TaskNature), roboco.foundation.policy.lifecycle (VALID_TRANSITIONS, get_valid_transitions, is_terminal_state, can_invoke_intent, intents_for_role, _INTENT_VERBS), roboco.foundation.policy.tracing (VERB_REQUIREMENTS, VERBS_WITHOUT_TRACING), roboco.foundation.identity/roboco.foundation.AGENTS, roboco.enforcement.task_lifecycle (shim), roboco.agents_config (_BOARD_ROLES, TASK_CREATOR_ROLES, AGENT_ROLE_MAP, AGENT_TEAM_MAP), roboco.runtime.orchestrator.AgentOrchestrator, roboco.api.deps/roboco.api.routes.v1.*/roboco.services.gateway.envelope.Envelope, scripts/verify_postgres_enums.py (loaded via importlib.util), docker/scripts/bash-guard-hook.sh + docker/scripts/stop-hook.sh (shell smoke targets) - External: pytest, pytest-asyncio, pytest-cov, pytest-xdist, asyncpg, sqlalchemy[asyncio] (create_async_engine, async_sessionmaker, AsyncSession, text, select), fastapi.testclient.TestClient, unittest.mock (AsyncMock, MagicMock, patch), hypothesis NOT a dependency (property tests are deterministic) ## Entry Points | Name | File | Trigger | |---|---|---| | make quality | Makefile | developer/CI merge gate — runs ruff format-check + ruff check + mypy roboco/ tests/ + pytest -q --cov=roboco --cov-report=term-missing --cov-fail-under=80 + xenon + vulture | | make quality-fast | Makefile | pre-submit fast gate — ruff + mypy + pytest -q -x --no-cov (no coverage threshold) | | make e2e-smoke | Makefile | scripted-agent lifecycle smoke — `ROBOCO_E2E_SMOKE=1 uv run pytest tests/e2e_smoke -q --no-cov`; needs test Postgres + git on PATH | | make gate (panel-gate) | Makefile | panel pnpm lint + typecheck + vitest | | make test/test-3.10..3.14/test-all | Makefile | docker compose run roboco pytest -v --cov=. across Python versions | | uv run pytest | pyproject.toml | direct invocation — auto-applies addopts (--cov=roboco --cov-report=term-missing), testpaths=tests, asyncio_mode=auto | | tests/integration/test_bash_guard_message.sh | tests/integration/test_bash_guard_message.sh | standalone bash smoke (NOT pytest-collected); run manually/CI to assert bash-guard denial <=3 lines | | tests/integration/test_stop_hook_verb_names.sh | tests/integration/test_stop_hook_verb_names.sh | standalone bash smoke (NOT pytest-collected); asserts no pre-gateway legacy verb names in stop/bash hooks | ## Config Flags - ROBOCO_TEST_DB_HOST (default localhost) - ROBOCO_TEST_DB_PORT (default 15432; was 5432 at baseline) - ROBOCO_TEST_DB_USER (default roboco; was $USER at baseline) - ROBOCO_TEST_DB_PASSWORD (default roboco; was empty at baseline) - ROBOCO_TEST_DB_ADMIN_DB (default postgres) - ROBOCO_AGENT_AUTH_SECRET/ROBOCO_AGENT_AUTH_REQUIRED/ROBOCO_AGENT_ID/ROBOCO_AGENT_ROLE/ROBOCO_AGENT_TOKEN/ROBOCO_AGENT_SESSION_ID/ROBOCO_AGENT_MODEL (gateway auth context set by tests via monkeypatch) - ROBOCO_TOOL_MANIFEST_PATH/ROBOCO_ORCHESTRATOR_URL/ROBOCO_API_URL/ROBOCO_SDK_URL/ROBOCO_PUBLIC_BASE_URL/ROBOCO_MCP_CONFIG (agent-container env stubbed in unit tests) - ROBOCO_GROK_REASONING_EFFORT/ROBOCO_GROK_TURN_TIMEOUT_SECONDS/ROBOCO_GROK_USAGE_FILE/ROBOCO_GROK_RUN_LOG/ROBOCO_GROK_RUN_CWD (grok provider tests) - ROBOCO_GATEWAY_ENABLED/ROBOCO_GUARD_SKIP_GIT (gateway/guard toggles exercised in runtime+gateway tests) - ROBOCO_CI_WATCH_ENABLED (tests/unit/config/test_ci_watch_flag.py + integration/services/test_ci_watch_*) - ROBOCO_DEP_UPDATE_ENABLED (tests/unit/config/test_dep_update_flag.py + integration/services/test_dep_update_*) - ROBOCO_CONVENTIONS_ENABLED (tests/unit/config/test_conventions_flag.py + conventions/* + gateway conventions-gate tests) - ROBOCO_ORG_MEMORY_ENABLED/ROBOCO_ORG_MEMORY_TOP_K (tests/unit/config/test_org_memory_flag.py + gateway/test_briefing_memory.py) - ROBOCO_RELEASE_MANAGER_ENABLED/ROBOCO_RELEASE_MIN_COMMITS (tests/unit/config/test_release_manager_flag.py + release_* service/runtime tests) - ROBOCO_TOOLCHAIN_MATCH_ENABLED (tests/unit/test_toolchain_flag.py + runtime/test_toolchain_guard.py) - ROBOCO_AGENT_LOOP_THRESHOLD/ROBOCO_AGENT_LOOP_WINDOW/ROBOCO_AGENT_TOOL_CALL_HALT/ROBOCO_AGENT_TOOL_CALL_WARN (agent_loop unit tests) - ROBOCO_TRANSCRIPT_DIR/ROBOCO_LOG_DIR/ROBOCO_DATA_DIR/ROBOCO_WORKSPACE/ROBOCO_ENCRYPTION_KEY/ROBOCO_PROMPTER_SESSION_ID/ROBOCO_SECRETARY_SESSION_ID/ROBOCO_INITIAL_PROMPT/ROBOCO_GIT_AGENT/ROBOCO_TEST_MARKER (misc env stubs) ## Gotchas - conftest uses Base.metadata.create_all, NOT alembic upgrade head, by design — migration 001's agentrole enum uses lowercase values while the ORM StrEnum binds uppercase NAMES, and migration 008 UPDATEs a nonexistent agents.skills column. A fresh alembic upgrade head would fail today. Any new migration that adds a NOT NULL column without a server_default AND is mapped on the ORM will break create_all for EVERY db_session test. - Migration-006 columns acceptance_criteria_status + qa_evidence_inspected are deliberately NOT mapped on TaskTable; conftest adds them via raw ALTER and smoke_test_batch sets them via raw SQL. Any test reading them must use raw SQL (text()), not the ORM. - smoke_test_batch is the ONLY fixture that calls db_session.commit() (line 477). Its data persists for the test lifetime; db_session teardown rollback is a no-op against committed rows. Safe because the whole DB is dropped at session end. - pytest_collection_modifyitems only skips items whose fixturenames intersect {db_session, smoke_test_batch}. Tests that request _test_database_url DIRECTLY (test_claim_lock_serialization, test_board_review_gate_timeline_real_db) are NOT covered by the collection skip — they rely on the fixture's own internal pytest.skip(). If _test_database_url's internal skip is ever removed, those tests will hard-fail with connection errors instead of skipping when PG is down. - _test_database_url force-terminates straggler backends (pg_terminate_backend) before DROP DATABASE — needed because asyncpg pool connections can outlive the engine.dispose() race; a flaky DROP could leak `roboco_test__` DBs. - Coverage core=sysmon is Python 3.12+ only; the legacy pytrace core under-counts async routes ~30% on 3.13. Running the suite on <3.12 would silently mis-measure coverage. - The omit list excludes roboco/runtime/orchestrator.py, git.py, workspace.py, mcp/*, agents/*, optimal.py from the 80% gate — these are covered by NAS smoke runs, NOT unit coverage. A regression in those modules will not fail make quality. - Property tests are NOT hypothesis-driven (hypothesis is not a dependency); they use deterministic exhaustive enumeration + a bounded random walk seeded by stdlib random. A state-machine hole could hide if the random walk doesn't exercise the violating path. - The JSON fixture is a SYNTHESIS, not a verbatim trace — the original /tmp/audit-trace.txt was wiped. replay_kind classifies each record; schema_only_skip/audit_only_skip/behavioral_skip records are NOT re-asserted at the spec layer (documented gaps). - The two .sh smoke scripts are NOT collected by pytest (no pytest collection of .sh); they must be invoked directly. They are not referenced in Makefile or .github/ workflows — likely run manually or in an unwired CI step. - _test_database_url is session-scoped with loop_scope=session; db_session is function-scoped with asyncio_default_fixture_loop_scope=function (pyproject). Mixing session+function loop scopes is supported but a session-scoped async fixture holds one event loop for the whole session. - conftest default port 15432 points at Docker's roboco-postgres (down on a bare-metal dev box). Per project memory the local-PG workflow is ROBOCO_TEST_DB_PORT=55432 ROBOCO_TEST_DB_USER=renzof ROBOCO_TEST_DB_PASSWORD= — without these env vars, ALL db_session tests silently skip on a non-Docker dev machine. - `tests/e2e_smoke/test_auditor_triggers.py` tests are sync wrappers around an async helper. The two auditor-trigger smoke tests are plain `def` tests that call `asyncio.run()` on an inner coroutine, because the harness already starts an async loop via `run_db`. Marking them with `@pytest.mark.asyncio` would conflict with that nested loop. - `tests/conftest.py` now tolerates a missing `pgvector` extension. `_test_database_url` catches `CREATE EXTENSION IF NOT EXISTS vector` failures and warns instead of failing. This lets the suite run in lightweight sandboxes, but it also means a sandbox with missing pgvector will pass with a warning rather than failing loudly. - `tests/e2e_smoke/harness.py` clears host `ROBOCO_AGENT_TOKEN` before scripted agents load `flow_server`. A `ScriptedAgent` pops `ROBOCO_AGENT_TOKEN` from `os.environ` before re-importing `roboco.mcp.flow_server`. Without this, a real host token is forwarded for an ephemeral test agent identity and causes 401s. Any new harness path that imports `flow_server` inside a scripted agent must do the same. ## Drift from CLAUDE.md - CLAUDE.md MEMORY note says 'conftest defaults roboco@localhost:15432' and that :5432 has the roboco role but NO pgvector, and :55432 is the local test PG. The conftest.py code (line 84-87) defaults to host=localhost port=15432 user=roboco password=roboco — matches the Docker-default claim, but CLAUDE.md's body text never documents the ROBOCO_TEST_DB_* env knobs or the create_all-vs-alembic drift; that lives only in the conftest docstring. - CLAUDE.md states coverage target 80% — matches Makefile --cov-fail-under=80. No drift. - CLAUDE.md does not mention the coverage omit list (orchestrator.py, git.py, workspace.py, mcp/*, agents/*, optimal.py, websocket.py, stream_bus.py, cli.py, alembic/*) — so an 80% green gate does NOT actually cover the orchestrator/runtime/git surface, which CLAUDE.md implies is tested. The omit list is a conftest/pyproject-level architectural fact absent from CLAUDE.md. - CLAUDE.md does not mention that the property tests are deterministic (no hypothesis) or that the bash smoke scripts are outside pytest — not a contradiction, just undocumented scope. ## Changes Since Baseline | SHA | Subject | Impact | |---|---|---| | 15effce0 | Chore: 141 Gaps fill-in (#283) | 165 test files changed (+17360/-276). Added 82 new test files including the worktree-routing/claim-rollback/cancel-cleanup, respawn_persistence, per_dev_lane_queue, notification_dedup_refire/delivery_refire, rate_limit_tracker_atomic, release_proposal_concurrency/readiness_first_release/executor_commit_fail_closed/subprocess_timeout, playbook_index_ordering/slug_race/unindex, main_pm_code_guard, task_assignment_invariants, workspace_uv_python_install_dir/uv_resolves_clone_venv/worktree_lifecycle/worktree_paths, sequencing expansion, git_resolve_git_dir/token_decryption_log, messaging_channel/session_race. Modified tests/conftest.py: test-DB default port 5432->15432, user $USER->roboco, password empty->roboco (so the integration suite runs out-of-the-box against Docker postgres). Tightened sequencing, task, work_session, pr_merge_concurrency, self_heal_originate_db, notification_dedup tests. | | 3aff6e04 | Chore: Close gaps (#285) | 21 test files changed (+2889/-132). MegaTask per-cell project map root-subtask tests (batch shape, branch creation, intake cell-map), pr_gate_notifies_pm/records_verdict/hand_format_guard/structured/verdict_consistency/self_review narrowed, schemas_v1_flow StrList coercion, content_models isinstance narrowing, channel_access auditor cases, grok_spawn auth-mount absence, fail_qa work-session fallback exclusion, ceo_reject coordination/umbrella routing + handoff journal, task_service_basics second-active-session rejection, ci_watch/dep_update source parity, tracing-verb parity table realignment after new verbs. No conftest change in this commit. | > Post-snapshot updates (since 2026-06-29): > - **536bbb64** Chore/all/logical gaps sweep (#286): 169 files changed; 13 new test files added (tests/unit/gateway/test_claim_gate_review_guards.py, test_verb_runner_midverb_invalid_state.py; tests/unit/runtime/test_spawn_session_attribution.py, test_strategy_engine_loop.py; tests/unit/mcp_servers/test_classify_dict_error_code.py, test_register_tools_override.py; tests/unit/llm/test_routing_downgrade.py; tests/unit/services/test_transcription_flush.py; tests/unit/conventions/test_custom_languages.py; tests/unit/api/test_deps_ceo_gate.py; tests/unit/test_conftest_db_unreachable_warning.py, test_migration_graph_integrity.py, test_regenerate_verb_tables.py). conftest.py gained `_warn_if_pg_unavailable` (import-time warning when PG unreachable — see Key Symbols and Regression Risks). 79 existing test files also modified (coverage expansion, mypy-type tightening, behavior corrections). > - **b49337e7** logical-gaps route-layer force/privileged-field gate: added tests/integration/test_tasks_route_privileged_fields.py. > - **115061f3** notification_delivery over-fetch fix: added tests/integration/test_notification_system_list.py. > - **321e68d7** proactive dead-code sweep: added tests/unit/services/test_proactive_code_patterns_dead.py. > - **9b29c071** migration 052 unique constraint: added tests/integration/test_task_cell_projects_unique.py. > - **49526f55** test-suite quality gate unblock: fixed 12 mypy errors across 5 test files + 2 behavior corrections (child task state for cascade-cancel assertion; test_a2a_message_auth mocked to DB-free path). > - **76ce53e3** chat MESSAGE_SENT fix: test_websocket_bridge.py gained 3 new test functions for `_handle_message_event` (skips missing ids, skips invalid uuid, broadcasts to session+channel). > - **77958c1e** chat session/group/message read IDOR fix: test_messaging_service.py extended with `_make_agent` helper + IDOR access-control test coverage for get_session/get_group/list_messages. > - **1f129199** auditor-trigger e2e smoke test: added `tests/e2e_smoke/test_auditor_triggers.py` exercising scheduled sweep and reactive QA-fail alert paths end-to-end, plus harness mount of `/api/notifications` so `_dispatch_audit_work` can poll ALERT rows. > - **babffe0a** fix(e2e_smoke): repair auditor-trigger smoke tests and harden harness (#498): fixed `tests/e2e_smoke/test_auditor_triggers.py` so scheduled/reactive auditor-trigger tests reach their spawn assertions, hardened `tests/conftest.py` to tolerate missing pgvector, and cleared leaked `ROBOCO_AGENT_TOKEN` in `tests/e2e_smoke/harness.py` before scripted agents load `flow_server`. See the E2E smoke harness section above for the exact patterns. > - **f081a574** (PR #502, 2026-07-13): Follow-up e2e lifecycle smoke fix. Restored the `ROBOCO_AGENT_TOKEN` pop in `tests/e2e_smoke/harness.py:ScriptedAgent._module` after it was accidentally removed, and clarified the `/api/notifications` mount comment so it no longer implies the router was newly added. The orchestrator `__new__` pre-init from `89b68786` means `_fresh_orchestrator` no longer needs to manually set `_instances`. > - **`10f039c3`** (#655, "golden-task eval harness + doctrine cohort stamp"): adds `roboco/eval/` (see Files above), a bench harness reusing `tests/e2e_smoke/harness.py`'s disposable environment rather than a mock — real isolation. `agent_spawn_sessions.doctrine_version` (migration 081) is stamped at spawn-session finalize from the composed prompt layers, so a cohort's model+doctrine combination (e.g. Fable-mode on vs. off) is durably identifiable after the fact — see `docs/map/db-migrations.md`. Bench runs also patch every vault flag off so a bench task/note/journal write never lands in the operator's real Obsidian vault. > - **`9882ebc6`** (#705, task `5cc75f71`, "fix disposable orchestrator container-reachability + document real-UUID isolation"): the in-process uvicorn server in `build_e2e_stack` now binds `0.0.0.0` (was `127.0.0.1`) so spawned agent containers on the `roboco_default` bridge can reach it; `E2EStack` gained a `container_url` field (`http://host.docker.internal:{port}`, set alongside `base_url` which stays `127.0.0.1` for host-side test clients); `_bench_environment` patches `settings.api_url` to `stack.container_url` (was `stack.base_url`); `_build_mount_args` adds `--add-host host.docker.internal:host-gateway` to every agent `docker run` argv (production-inert — prod MCP servers use `http://roboco-orchestrator:8000`; Docker 20.10+ supports host-gateway on Linux). F2 (docs-only): `_seed_company`'s docstring + `tests/unit/runtime/test_eval_mcp_config_isolation.py`'s module/test docstrings now state explicitly that the AC "no real agent UUIDs" is satisfied by "no production DB/Redis reach" — the isolation boundary is the disposable URL + throwaway DB, not the UUID; real UUIDs are intentional (orchestrator-internal helpers keyed by the static registry resolve as in a real deployment) and randomizing them would make the bench less realistic. > - **Forge providers, 2026-07-18/19 (#569/#571/#575/#579/#581)**: adds `tests/unit/services/forge/` (`test_gitea_provider.py` 297 lines/17 tests, `test_gitlab_provider.py` 713 lines/33 tests, `test_router.py` 90 lines/9 tests — all mocked-transport) + `tests/unit/foundation/policy/test_forge.py` (132 lines/23 tests, `extract_host`/`detect_provider`/`validate_project_forge`) + the live-forge contract suites `test_gitea_live.py`/`test_gitlab_live.py` (see the "Live-forge contract suites" section above). See `docs/map/worksession-git.md` for the forge package itself. ## Regression Risks | Title | File:Line | Claim | Severity | |---|---|---|---| | conftest DB-default change silently skips all DB tests on non-Docker dev boxes | tests/conftest.py:85 | Default port changed 5432->15432 (Docker roboco-postgres). On a bare-metal dev box without Docker, _postgres_reachable() returns False and EVERY db_session/smoke_test_batch test skips — a developer can see a green run with zero real-DB coverage and not notice. **Partially mitigated in 536bbb64**: `_warn_if_pg_unavailable` now emits a `warnings.warn` at import time when PG is unreachable, so the all-skips run surfaces a visible pytest warning. The per-test skip itself is unchanged. Previously (5432/$USER) it errored loudly with InvalidPasswordError, surfacing the misconfiguration. | medium | | _test_database_url direct requesters bypass the collection-time skip | tests/conftest.py:506 | pytest_collection_modifyitems only adds the skip marker for fixturenames in {db_session, smoke_test_batch}. test_claim_lock_serialization and test_board_review_gate_timeline_real_db request _test_database_url directly; they skip only because the fixture self-skips at line 126. If a refactor removes that internal skip (e.g. moving provisioning into a different fixture), these tests will hard-fail with connection errors instead of skipping when PG is unreachable. | low | | create_all schema drift — new NOT NULL ORM-mapped columns break all DB tests | tests/conftest.py:182 | Schema is built via Base.metadata.create_all, not alembic. Any new migration (post-052) adding a NOT NULL column WITHOUT a server_default that IS mapped on the ORM will make create_all fail for every db_session test, since create_all emits the column with no default. The conftest only manually backfills the two migration-006 columns; it does not replay later migrations. The 141-gaps commit added migrations 046-052 (batch/ci_watch/dep_update/playbook/respawn_tracker/cell_projects); if any of those columns is NOT NULL without default and ORM-mapped, db_session tests break. | high | | smoke_test_batch commits but db_session teardown only rolls back — cross-test leakage within a session | tests/conftest.py:477 | smoke_test_batch calls db_session.commit() (the only fixture that commits). db_session teardown does session.rollback() which is a no-op against already-committed rows. Because the DB is session-scoped and dropped at end, this is safe ONLY if no other test in the same session re-uses the committed rows expecting a clean state. A test that requests db_session AFTER smoke_test_batch in the same session could see residual committed rows if the seeding fixture ran in the same DB. Function-scoped fixtures re-seed each call, but the commit breaks the per-test isolation invariant the rollback teardown assumes. | medium | | Coverage omit list hides orchestrator/git/workspace regressions from the 80% gate | pyproject.toml:259 | [tool.coverage.run].omit excludes roboco/runtime/orchestrator.py, services/git.py, services/workspace.py, mcp/*, agents/*, optimal.py, websocket.py, stream_bus.py, cli.py. A regression in any of these (e.g. the respawn-tracker upsert race, the pr_merge cross-repo scoping, the worktree routing) will NOT fail make quality's --cov-fail-under=80 gate. The 141-gaps and Close-gaps commits added many runtime/git tests, but the modules remain omitted, so coverage is measured only on the non-omitted surface. | medium | | Property random-walk seed is not pinned — state-machine holes can hide | tests/property/test_state_machine_invariants.py:95 | test_random_walks_stay_within_declared_states uses stdlib random without an explicit seed. A state-machine transition added by the gap-fill commits that violates an invariant on a rarely-walked path could pass CI on most runs and fail intermittently. Without hypothesis and without a fixed seed, the walk coverage is non-deterministic. | low | | JSON smoke-trace fixture is a synthesis — skip-classified bugs are not re-asserted | tests/fixtures/2026-05-08-smoke-trace.json:1 | 3 of 11 records are *_skip (schema_only_skip, audit_only_skip, behavioral_skip) and are documented NOT re-asserted at the spec layer in test_lifecycle_smoke_replay. A regression in the bug those records document (e.g. delegate.task_type default, the audit-layer fix) would not be caught by the smoke-replay test; it relies on other tests covering those layers. If those other tests were removed, the regression window re-opens silently. | low | | Shell smoke scripts are not wired into any CI/Makefile target | tests/integration/test_stop_hook_verb_names.sh:1 | test_bash_guard_message.sh and test_stop_hook_verb_names.sh are not referenced in Makefile or .github/workflows. They guard against pre-gateway legacy verb names leaking back into stop-hook.sh/bash-guard-hook.sh and against denial-message bloat. A regression (re-introducing an old verb name, or an 8-line denial) would not be caught by make quality or pytest; the scripts must be run manually. | low | | Missing pgvector is now a warning, not a failure | tests/conftest.py:189 | `_test_database_url` catches `CREATE EXTENSION IF NOT EXISTS vector` failures. A sandbox that accidentally omits pgvector will green-run the e2e smoke suite with a warning, so a regression in pgvector-dependent code could pass locally and only fail in CI. | low | | Host `ROBOCO_AGENT_TOKEN` can leak into scripted-agent calls | tests/e2e_smoke/harness.py:476 | The harness clears the token before loading `flow_server`, but any new scripted-agent bootstrap path that forgets this step will forward the test runner's real token and get 401s for ephemeral agent IDs. | low | | `_fresh_orchestrator` bypasses `__init__` and must stay in sync with private attributes | tests/e2e_smoke/test_auditor_triggers.py:115 | The helper constructs `AgentOrchestrator` via `__new__`. `AgentOrchestrator.__new__` now pre-initializes `_instances`, `_last_audit_spawn_at`, and `_notification_spawn_at`, so the helper's explicit initialization of `_instances` and `_last_audit_spawn_at` is defensive rather than required. A refactor of `AgentOrchestrator.__init__` that adds new instance attributes read by `_dispatch_audit_work` without also adding them to `__new__` will break the smoke tests silently until the helper is updated. | low | ## Health The test slice is structurally healthy and well-tiered (foundation parity/integration real-DB/property invariants/unit mirror), with a single load-bearing conftest that honestly documents its own drift from the alembic chain. The 571-file suite is async-first and coverage-gated, but two architectural facts temper confidence: (1) the coverage omit list excludes the orchestrator, git, workspace, mcp, and agents surface from the 80% gate, so make quality green does NOT mean those hot paths are covered — their regressions are deferred to NAS smoke runs; (2) the conftest builds schema via Base.metadata.create_all, which is correct today but creates a standing trap for any future NOT NULL ORM-mapped migration column. The since-baseline gap-fill commits added substantial real coverage (worktree lifecycle, respawn persistence, lane barrier, sequencing, release executor fail-closed, rate-limit atomicity), and the conftest DB-default change (5432/$USER -> 15432/roboco) fixed a real out-of-the-box failure mode but introduced silent-skip behavior on non-Docker dev boxes. The property tests lack hypothesis and a pinned seed, and the JSON smoke fixture is a synthesis with 3/11 records not re-asserted — both are documented limitations, not defects. Overall the slice is solid for a merge gate but should be supplemented by the NAS smoke run before any deploy claim, and the shell smoke scripts need wiring into CI to actually guard what they assert.