diff --git a/.gitignore b/.gitignore index 6a0a7012..6d20476a 100644 --- a/.gitignore +++ b/.gitignore @@ -66,6 +66,9 @@ dmypy.json # Ruff .ruff_cache/ +# import-linter (lint-imports) +.import_linter_cache/ + # Environment .env .env.local diff --git a/Makefile b/Makefile index 0b294997..aeaf5c8e 100644 --- a/Makefile +++ b/Makefile @@ -338,7 +338,7 @@ quality-fast: sync @uv run mypy roboco/ tests/ @uv run pytest -q -x --no-cov -# Fast pre-submit gate: format-check + lint + types + complexity, NO tests. +# Fast pre-submit gate: format-check + lint + types + complexity + import boundaries, NO tests. # This is the command a project points `quality_command` at, so the agent # pre-submit gate (run at i_am_done) executes it in the dev's workspace and # catches lint/type/complexity at the desk. The test suite stays on CI. @@ -348,6 +348,7 @@ gate: sync @uv run ruff check . @uv run mypy roboco/ tests/ @uv run xenon --max-absolute B --max-modules A --max-average A roboco/ + @uv run lint-imports # Panel (Next.js) fast gate: lint + type-check + vitest. # Run locally before submitting panel changes; mirrors the CI panel job exactly. diff --git a/agents/prompts/teams/backend.md b/agents/prompts/teams/backend.md index 62b3ba60..7d397f47 100644 --- a/agents/prompts/teams/backend.md +++ b/agents/prompts/teams/backend.md @@ -23,7 +23,7 @@ # The Makefile sets UV_NO_SYNC=1 + a private UV_CACHE_DIR to prevent venv # corruption; bare `uv run` bypasses both. make lint # ruff format + ruff check + mypy + vulture (formats in place) -make gate # fast pre-submit: ruff format --check + ruff check + mypy + xenon +make gate # fast pre-submit: ruff format --check + ruff check + mypy + xenon + lint-imports make quality # full merge gate (lint+types+tests+cov+xenon+bandit+audit+...) make test # pytest with coverage diff --git a/docs/map/_complete_map.md b/docs/map/_complete_map.md index 6bbc6b90..ede8106b 100644 --- a/docs/map/_complete_map.md +++ b/docs/map/_complete_map.md @@ -939,7 +939,7 @@ This slice is the packaging, build, and runtime-tooling layer of RoboCo: the Doc | docker-compose.yaml | Build-from-source compose: postgres/redis/ollama/ollama-init, 14 agent-*-image builders, orchestrator, panel, nginx, `backup` pg_dump sidecar on the `roboco_data` DB-isolation network; NAS prod env vars + volume mounts, `ROBOCO_OBSIDIAN_VAULT_ENABLED`/`ROBOCO_VAULT_INTAKE_ENABLED` default `true` | 556 | | docker-compose.yml | Byte-identical copy of docker-compose.yaml (kept for the canonical name compose picks up by default) | 556 | | docker-compose.registry.yml | Pull-and-run compose using pre-built GHCR/Docker Hub images (ROBOCO_REGISTRY + ROBOCO_VERSION); infra services byte-identical to build compose (incl. `backup` + vault defaults), agent-* services are one-shot pre-pulls, own `roboco_data` network | 334 | -| Makefile | Ops + quality targets: infra, dev/run/orchestrator, quality gate (ruff/mypy/pytest/xenon/radon/vulture/bandit/pip-audit/deptry/lint-imports/alembic/foundation-check), per-Python test matrix, docs, lifecycle regen | 548 | +| Makefile | Ops + quality targets: infra, dev/run/orchestrator, quality gate (ruff/mypy/pytest/xenon/radon/vulture/bandit/pip-audit/deptry/lint-imports/alembic/foundation-check), fast pre-submit `gate` target (ruff format/check, mypy, xenon, lint-imports), per-Python test matrix, docs, lifecycle regen | 599 | | pyproject.toml | Project + dependency manifest: requires-python >=3.13,<3.15, deps, dev/docs extras, console scripts, ruff/mypy/pytest/coverage/vulture/bandit/radon/xenon/deptry/importlinter config | 451 | | roboco/config.py | Pydantic Settings (env prefix ROBOCO_, cached via lru_cache); every tunable: DB, Redis, RAG, LLM/Ollama, workspaces, agent guardrails, gateway thresholds, autonomy-engine flags | 1022 | | roboco/__init__.py | Package root: __version__ + re-exports settings, exceptions, logging helpers | 39 | @@ -1148,7 +1148,8 @@ deployment-tooling | python -m roboco.cli / roboco console script | roboco/cli.py | orchestrator container ENTRYPOINT (docker/orchestrator.Dockerfile:98); `make orchestrator`; `make dev`; `make db-init` (--db-only) | | roboco-bootstrap console script | roboco/bootstrap.py | pyproject [project.scripts] alias (points at roboco.bootstrap:cli which does not exist — see drift) | | uvicorn roboco.api.app:app | roboco/bootstrap.py | spawned as asyncio task inside bootstrap.main (make api / make run run it directly) | -| make quality / foundation-check / lifecycle | Makefile | CI merge gate + local pre-submit | +| make quality / foundation-check / lifecycle | Makefile | CI merge gate | +| make gate | Makefile | Fast local pre-submit (`quality_command`, runs at dev `i_am_done`): ruff format/check, mypy, xenon, lint-imports — no tests | | scripts/build_lifecycle_artifacts.py | scripts/build_lifecycle_artifacts.py | make lifecycle (foundation-check runs it) | | scripts/verify_postgres_enums.py | scripts/verify_postgres_enums.py | make foundation-check (final step) | | scripts/reflow_md.py --check | scripts/reflow_md.py | make reflow-check / make quality | @@ -1195,6 +1196,7 @@ deployment-tooling - ROBOCO_X_ENGINE_ENABLED / _MENTIONS_INTERVAL_SECONDS / _MENTIONS_MAX_PER_CYCLE / _MENTIONS_MIN_ENGAGEMENT / _MAX_OPEN_POSTS / ROBOCO_X_ACCOUNT_USER_ID / _REQUEST_TIMEOUT_SECONDS — the X (Twitter) engine; inert without stored OAuth 1.0a credentials regardless of the flag - ROBOCO_ROADMAP_ENGINE_ENABLED / _INTERVAL_SECONDS (default 604800) / _MIN_ITEMS_PER_CYCLE / _MAX_ITEMS_PER_CYCLE — the board roadmap engine - ROBOCO_X_FEATURE_SPOTLIGHT_ENABLED / _INTERVAL_SECONDS (default 259200/3d) — X-engine feature-spotlight sub-switch (requires ROBOCO_X_ENGINE_ENABLED also on), default off +- ROBOCO_PEST_REWORK_THRESHOLD (default 0.3, config.py ~1432) — the only compose-settable knob for any of the twelve new Board Programs (Pest Control's off-schedule rework-rate-spike accelerator); every other new program is armed exclusively via its per-program settings-store row (`board_program.{key}.enabled`, toggled from the Board Programs panel page), not an env flag — see `docs/rag/architecture/board-programs.md` / CLAUDE.md's "Board Program registry" - ROBOCO_OBSIDIAN_VAULT_ENABLED / ROBOCO_VAULT_PATH (default `/data/vault`) — Obsidian vault V1 projection master switch, config-default off but both compose files set it `true`; ROBOCO_VAULT_INTAKE_ENABLED / _INTERVAL_SECONDS / _DIR / _MAX_PER_CYCLE / _MAX_OPEN_DRAFTS — the independently-gated `#roboco`-tag inbox watcher - ROBOCO_FABLE_MODE_ENABLED — opus-fable-playbook adoption (doctrine layer in the composed prompt + 5 Claude-path hook scripts + 1 grok-path hook), default off; off = byte-for-byte unchanged spawn path - `ROBOCO_BACKUP_MIRROR_DIR` (unset by default) — arms the `backup` sidecar's off-disk mirror step; the host path should live on a DIFFERENT disk (external/remote mount) — a same-disk mirror protects nothing. Compose only sets the container env when the `.env` variable is set. @@ -3260,6 +3262,7 @@ The AgentOrchestrator is the runtime brain of RoboCo: it owns the per-agent Dock | AgentOrchestrator._generate_composed_prompt | method | roboco/runtime/orchestrator.py:2946 | Compose the spawn prompt (identity + task briefing + ambient conventions block + tool-load block) via compose_prompt. | | AgentOrchestrator._resolve_conventions_ambient | method | roboco/runtime/orchestrator.py:2998 | Resolve the in-scope projects for the ambient architectural-conventions block (single repo / product / ad-hoc cell map). | | AgentOrchestrator._readiness_gate | method | roboco/runtime/orchestrator.py:3104 | Pre-flight refusal: missing project/cell-map, missing AC, role/status mismatch, missing git token, bad task shape, unmet dependencies. | +| AgentOrchestrator._build_mount_args | staticmethod | roboco/runtime/orchestrator.py:3162 | Compose the `docker run -v/-e` mount + env argv for an agent: `--name`, `--network AGENT_NETWORK`, `--add-host host.docker.internal:host-gateway`, Claude auth/JSON mounts, optional host mounts, then role-scoped core volumes + env via `_core_volume_and_env_args`. The `--add-host host.docker.internal:host-gateway` (PR #705, task `5cc75f71`) lets spawned containers resolve `host.docker.internal` so the eval harness's disposable orchestrator (bound `0.0.0.0` on the host) is reachable via `host.docker.internal:` — **production-inert**: prod MCP servers use `http://roboco-orchestrator:8000` (the container hostname), never `host.docker.internal`. Docker 20.10+ (May 2021) supports `host-gateway` on Linux. Added unconditionally (simpler than threading `spawned_by="eval_bench"` through); no production spawn observes it. | | AgentOrchestrator._write_agent_briefing | method | roboco/runtime/orchestrator.py:3431 | Fetch task + institutional memory + workflow state and render the per-agent briefing markdown at the cwd path. | | AgentOrchestrator.start_intake_session | method | roboco/runtime/orchestrator.py:3612 | Open the intake relay + schedule the guarded spawn of the single persistent intake container. | | AgentOrchestrator._spawn_intake_container | method | roboco/runtime/orchestrator.py:3715 | Spawn the intake container under _intake_spawn_lock: clone intake scope, build cmd, run, abort-if-shutdown, register instance, record usage session. | @@ -7967,7 +7970,7 @@ The revision-findings ledger: the structured replacement for prose-only QA/PR-ga | `TaskReviewFindingTable` | ORM class | `roboco/db/tables.py` | The append-only ledger row: `task_id`, `origin`, `round`, `author_slug`, `file`/`line`/`severity`/`criterion`/`expected`/`actual`/`fix`/`evidence`, `status`, `addressed_by_commit`, `resolution_note`. `origin`/`severity`/`status` are plain `String` columns, not a native Postgres enum. | | `Finding` | Pydantic model | `roboco/foundation/policy/content/models.py:92` | One structured finding, shared by `post_pr_review` (external PRs) and the four internal producers. Caps: `file` ≤300 (repo-relative, no `..`), `line` ≥1, `expected`/`actual` ≤300, `fix` ≤500, `evidence` ≤2000. `criterion` has **no Pydantic `max_length`** despite the DB column being `String(500)` — see Regression Risks. `file` is additionally shape-gated (`_PATH_SHAPE_RE`, #687): a value that doesn't look like a repo-relative path (prose like a PR reference, which used to validate and then doomed the panel's code-snippet fetch) is rejected with a remediate naming the file-less option for cross-cutting findings; the class admits `+`/`@` (SvelteKit route files, `@types` dirs, `@2x` assets) but excludes spaces, the prose signal. `file` remains OPTIONAL for the `issues` shim's file-less findings. | | `PmReviewContent` | Pydantic model | `roboco/foundation/policy/content/models.py` | New content type `"pm_review"` (`summary` + `findings`, no separate `verdict` — the transition to `needs_revision` IS the verdict); mirrors to the new `tasks.pm_notes` column via `_MIRROR_COLUMN`. | -| `ReviewFindingsRepository` | class | `roboco/services/repositories/review_findings.py:32` | `insert_many` (append rows, one flush, no independent commit), `list_for_task` (default cap 500, newest round first), `status_counts_for_task` (SQL `GROUP BY (origin, status)`, whole ledger — independent of the 500 cap), `mark_addressed` (8-char-prefix match against OPEN rows, no-op on 0 or >1 matches, never raises), `mark_verified` (bulk, by full id), `mark_waived` (exists, unwired — no verb calls it). | +| `ReviewFindingsRepository` | class | `roboco/services/repositories/review_findings.py:32` | `insert_many` (append rows, one flush, no independent commit), `list_for_task` (default cap 500, newest round first), `status_counts_for_task` (SQL `GROUP BY (origin, status)`, whole ledger — independent of the 500 cap), `mark_addressed` (8-char-prefix match against OPEN rows, no-op on 0 or >1 matches, never raises), `mark_verified` (bulk, by full id), `mark_waived` (exists, unwired — no verb calls it), `escaped_defects_since` (`(task_id, origin)` for blocker findings still `addressed`, never `verified`, on a task that has since gone `COMPLETED` within a window — the Company Scorecard's `escaped_defects` metric; see `docs/map/metrics-observability.md`). | | `findings_count_guard` / `findings_count_hint` | functions | `roboco/services/gateway/choreographer/findings.py:98,115` | Hard-reject `Envelope` above `FINDINGS_HARD_CAP=10`; non-blocking hint above `FINDINGS_NUDGE_COUNT=5`. | | `issues_to_findings` / `merge_findings_and_issues` | functions | `roboco/services/gateway/choreographer/findings.py:57,81` | Legacy `issues: list[str]` shim → file-less `severity=major` findings (deprecation-logged); merges with any `findings` sent in the same call rather than one silently dropping the other. | | `next_round` | function | `roboco/services/gateway/choreographer/findings.py:43` | `(task.revision_count or 0) + 1`, read BEFORE the transition — the round a finding written during this call belongs to. | @@ -8059,7 +8062,7 @@ review-findings slice - `roboco.services.gateway.envelope` — `Envelope` - `roboco.services.gateway.evidence_builder` — `BRIEFING_LIST_CAP` - `roboco.db.tables` — `TaskReviewFindingTable`, `TaskTable.pm_notes` -- Consumed by: `roboco.services.metrics` (`MetricsService`), `roboco.services.vault_assembly`/`vault_writer`, `roboco.runtime.orchestrator`, `roboco.mcp.flow_server`, `roboco.api.routes.tasks`/`v1.flow_*`, the panel's `task-detail`/`metrics` components +- Consumed by: `roboco.services.metrics` (`MetricsService`), `roboco.services.cockpit` (`CockpitService.summary`'s `escaped_defects` field), `roboco.services.vault_assembly`/`vault_writer`, `roboco.runtime.orchestrator`, `roboco.mcp.flow_server`, `roboco.api.routes.tasks`/`v1.flow_*`, the panel's `task-detail`/`metrics` components ## Entry Points @@ -8094,7 +8097,7 @@ None as of this doc's authoring — CLAUDE.md was updated in the same pass to ad - `docs/map/task-service.md` — `ceo_reject`, `_audit_events_for` - `docs/map/pr-gate-review.md` — `pr_fail` findings wiring, gate evidence, verify-stamp -- `docs/map/metrics-observability.md` — rework-by-agent event widening, per-task findings counts +- `docs/map/metrics-observability.md` — rework-by-agent event widening, per-task findings counts, `escaped_defects` Company Scorecard metric - `docs/map/vault.md` — task note `## Findings` section - `docs/map/panel.md` — Findings tab, `bounced xN` chip, findings route - `docs/internal/specs/2026-07-11-revision-findings-ledger.md` — the design spec this slice implements @@ -8113,7 +8116,7 @@ The metrics & observability slice is the read-only measurement layer of RoboCo: |---|---|---| | `roboco/services/metrics.py` | `MetricsService` — velocity, blockers, team/agent metrics, health, cycle-time/bottleneck/rework/scorecard observability | 1521 | | `roboco/services/dashboard.py` | `DashboardService` — auditor flags/reports (in-memory singleton), CEO overview, audit queue, agent status, recent activity | 457 | -| `roboco/services/cockpit.py` | `CockpitService` — read-only CEO "is the business winning?" summary (goals+delivery+spend+signals) | 97 | +| `roboco/services/cockpit.py` | `CockpitService` — read-only CEO "is the business winning?" summary (goals+delivery+spend+signals), delivery block now carries the 3 charter-objective metrics (`median_lead_time_hours`, `first_pass_yield`, `escaped_defects`) | 104 | | `roboco/services/usage.py` | `UsageService` — token usage summary, time-series, by-agent/team/model, projection, cache efficiency, today summary, recent sessions | 478 | | `roboco/services/usage_events.py` | `UsageSnapshot` dataclass + `publish_usage_snapshot` — publishes USAGE_SNAPSHOT to the StreamEventBus | 52 | | `roboco/services/telemetry/__init__.py` | Re-export of CI telemetry source symbols | 18 | @@ -8161,8 +8164,9 @@ The metrics & observability slice is the read-only measurement layer of RoboCo: | `DashboardService.get_all_agent_status` | method | dashboard.py:365 | Agent counts by status + per-agent snapshot | | `DashboardService.get_recent_activity` | method | dashboard.py:398 | Merged messages+task_updates feed, sorted desc | | `CockpitService` | class | cockpit.py:36 | Read-only CEO summary + lightweight signals slice | -| `CockpitService.summary` | method | cockpit.py:41 | goals+counts+delivery+spend+projection+pitches+signals, `basis="proxy"` | +| `CockpitService.summary` | method | cockpit.py:41 | goals+counts+delivery+spend+projection+pitches+signals, `basis="proxy"`. `delivery.first_pass_yield` is a pass-through of `MetricsService.get_org_scorecard().first_pass_yield` (no new computation); `delivery.escaped_defects` is `len(ReviewFindingsRepository.escaped_defects_since(30d cutoff))` — see the Gotchas entry below for the definition. | | `CockpitService.signals` | method | cockpit.py:81 | Strategy-engine signals only (lightweight panel slice) | +| `ReviewFindingsRepository.escaped_defects_since` | method | `services/repositories/review_findings.py` | `(task_id, origin)` pairs for blocker findings still `addressed` (never `verified`) on a `COMPLETED` task within the window — the Company Scorecard's "0 critical escaped defects" metric. See `docs/map/review-findings.md` and the Gotchas entry below. | | `UsageService` | class | usage.py:70 | Token usage analytics over spawn sessions + rollups | | `UsageService.get_summary` | method | usage.py:77 | Period totals + trend_pct vs previous period | | `UsageService.get_time_series` | method | usage.py:166 | Hourly (24h) / daily (7d/30d) buckets | @@ -8225,9 +8229,12 @@ graph LR Panel -->|/api/cockpit/*| Cockpit[CockpitService] Cockpit --> Goals[company_goals] Cockpit --> TaskSvc[TaskService] + Cockpit --> MetricsSvc + Cockpit --> FindingsRepo[ReviewFindingsRepository.escaped_defects_since] Cockpit --> UsageSvc Cockpit --> Strategy[strategy_engine] Cockpit --> Pitch[pitch service] + FindingsRepo --> TaskReviewFindings[task_review_findings] SelfHeal[self_heal_loop] --> CISrc[GitHubCITelemetrySource] CIWatch[ci_watch_loop] --> MultiSrc[MultiProjectCITelemetrySource] @@ -8267,7 +8274,7 @@ metrics-observability - `roboco.events.stream_bus` — `StreamEventBus` (TYPE_CHECKING only) - `roboco.services.base` — `BaseService` - `roboco.services.git` — `GitService.get_latest_ci_conclusion` (telemetry) -- `roboco.services.company_goals`, `pitch`, `strategy_engine`, `task` (cockpit) +- `roboco.services.company_goals`, `pitch`, `strategy_engine`, `task`, `metrics` (`get_metrics_service`), `repositories.review_findings` (`ReviewFindingsRepository`) (cockpit) - `roboco.config` — `settings` (telemetry) - `roboco.logging` — `get_logger` (telemetry) - `roboco.utils.converters` — `to_python_uuid`, `require_uuid` @@ -8315,7 +8322,10 @@ No flags live *inside* this slice's files, but the slice's behavior is gated/par - **Cache-efficiency uses hardcoded sonnet pricing** (usage.py:401-404, `_FULL_INPUT_PRICE=3.00`, `_CACHE_READ_PRICE=0.30`) for the savings estimate regardless of the actual model mix — an aggregate approximation, not per-model. - **`publish_usage_snapshot` lazy-imports `Event`/`EventType`** (usage_events.py:49) to avoid a circular import — callers must keep the bus passed in, not a module-level reference. - **`MultiProjectCITelemetrySource.fetch` swallows per-project exceptions** (source.py:172) — one bad project never aborts the sweep, but also never surfaces beyond a warning log; a persistently failing project silently contributes no sample (treated as "unknown", not "green" — correct, but invisible). -- **`CockpitService.summary` `basis="proxy"`** (cockpit.py:57) — every payload is stamped proxy; the over_budget flag is only meaningful once the CEO greenlights real launch. +- **`CockpitService.summary` `basis="proxy"`** (cockpit.py:66) — every payload is stamped proxy; the over_budget flag is only meaningful once the CEO greenlights real launch. +- **`escaped_defects` definition — why "a finding on a terminal task" is impossible, and what it actually counts.** The Company Scorecard's third charter objective ("0 critical escaped defects per release") looks like it should mean "a blocker-severity finding opened on a task that already reached a terminal state" — but that combination can never occur: every producer of a `task_review_findings` row (`fail_review`, `pr_fail`, `request_changes`, `ceo_reject`) fires as part of a bounce whose lifecycle transition requires the task to be non-terminal at that moment (the transition itself is `* -> needs_revision`). A "terminal-task finding" query would return 0 in every window, forever — a permanently-green scorecard card is worse than none, the exact fabrication PR #704 exists to remove from the panel. The real definition, computed by `ReviewFindingsRepository.escaped_defects_since`: a `blocker`-severity finding still at status `addressed` (**never** `verified`) on a task that has since gone `COMPLETED`, within the 30-day window (`TaskTable.completed_at >= cutoff`; `cancelled` tasks are excluded on purpose — they never set `completed_at` and never ship code, so nothing "escaped" from one). This is reachable because `stamp_addressed_verified` (`services/gateway/choreographer/findings.py:306`) only bulk-verifies findings of its OWN `origin` (`row.origin == origin`) when its matching pass verb runs (`pass_review`→qa, `pr_pass`→pr_gate, `complete`→pm, `ceo_approve`→ceo — `complete` stamps `origin="pm"` via `_stamp_pm_findings_verified_or_rejection` (`services/gateway/choreographer/_impl.py:7431-7456`), a distinct verb+stamp from `ceo_approve` (`services/task.py:7289-7294`), which stamps `origin="ceo"` on its own `awaiting_ceo_approval → completed` transition) — a blocker raised by one origin, marked `addressed` by the developer, and never independently re-confirmed by that SAME origin on a later round (the task's remaining rounds routed through a different reviewer) survives all the way to `completed` still `addressed`. A non-zero value means: at least one blocker-severity concern shipped to `completed` on the developer's own word alone, with no reviewer ever re-checking the fix — a real signal of unverified risk in production, not a fabricated placeholder. +- **In practice, "pm" is the only origin that can realistically produce a non-zero reading, and even that path is narrow.** `pass_review` and `pr_pass` (qa/pr_gate origins) hard-gate their `stamp_addressed_verified` call — a stamp failure fails the verb itself (qa.py:809-823, pr_gate.py mirrors it), so a qa-origin or pr_gate-origin blocker structurally cannot reach `completed` still `addressed`: passing review IS re-verifying it. The one reachable path is: a PM raises a blocker via `request_changes` (origin=`pm`), the dev addresses it, and the PM then calls `escalate_to_ceo` instead of `complete` — `escalate_to_ceo`'s `ActionSpec` (`foundation/policy/lifecycle.py:657-675`) has no precondition requiring findings be resolved, and `ceo_approve` only bulk-verifies its own `ceo`-origin rows, never touching the still-`addressed` `pm`-origin one. So a fleet that rarely escalates to the CEO (the normal case — most roots complete via a PM's own `complete`) will read 0 on this metric because the triggering path is rare, not because nothing has escaped. +- **The count is per-finding, not per-task, and the 30-day window is a temporal proxy, not a release boundary.** `escaped_defects_since` returns one `(task_id, origin)` row per qualifying finding with no de-dup/grouping, and `CockpitService.summary` takes `len(escaped)` directly — a single task with three qualifying blockers contributes 3 to the count, not 1. The charter's "0 critical escaped defects per release" phrasing implies release-scoped counting, but this metric has no notion of releases at all: it's a rolling `completed_at >= now - 30d` window that will straddle zero, one, or several actual release cuts depending on cadence, so a spike right after a release and a spike from unrelated day-to-day completions look identical on this card. ## Drift from CLAUDE.md @@ -8661,6 +8671,21 @@ The product / strategy / research / pitch slice covers the "company layer" above | `roboco/services/x_credentials.py` | Singleton Fernet-encrypted OAuth 1.0a credential CRUD; decrypts server-side only | 140 | | `roboco/api/routes/x.py` | CEO-only routes: list open X posts, approve/reject one draft | 164 | | `roboco/api/schemas/x.py` | `XPostResponse` + `XMentionRefModel` / `XFeatureRefModel` response shapes | 73 | +| `roboco/foundation/policy/board_programs.py` | Board Program registry: `BoardProgram` dataclass, `PROGRAMS` (14 entries), `program_due`/`project_participates`/`validate_board_programs_field` (pure) | 295 | +| `roboco/services/board_programs.py` | `BoardProgramEngine` — trigger/dedup/originate/LEARN over every registered program; `program_armed` settings-store chokepoint; `pick_rotation_target` for project-scoped round-robin | 566 | +| `roboco/api/routes/board_programs.py` | CEO-only routes: list every program's live status, `POST /{key}/run-now` | 103 | +| `roboco/services/pest_control_engine.py` | `PestControlEngine` — weekly+metric cron, project-scoped; evidence context (rework hotspots, recurring/waived findings) | 199 | +| `roboco/services/spackle_engine.py` | `SpackleEngine` — biweekly cron, project-scoped gap-fill audit | 142 | +| `roboco/services/scales_engine.py` | `ScalesEngine` — monthly cron, org-scoped portfolio rebalance; stale-backlog snapshot | 176 | +| `roboco/services/dogfood_engine.py` | `DogfoodEngine` — event-only, project-scoped; real `_ORIGINATORS` binding (unlike Coroner's stub) | 152 | +| `roboco/services/periscope_engine.py` | `PeriscopeEngine` — weekly cron, org-scoped market-research brief; `latest_brief_context` feeds Printer | 157 | +| `roboco/services/megaphone_engine.py` | `MegaphoneEngine` — 3-day cron, org-scoped editorial calendar; shipped-this-week digest + Unreleased changelog | 191 | +| `roboco/services/mirror_engine.py` | `MirrorEngine` — quarterly cron, project-scoped messaging-drift audit | 143 | +| `roboco/services/barfly_engine.py` | `BarflyEngine` — 2-day cron, org-scoped; screens X search candidates through `injection_guard`, marks seen | 223 | +| `roboco/services/war_room_engine.py` | `WarRoomEngine` — event-triggered (`open_for_release` release hook + CEO run-now); campaign posts w/ `publish_after` | 231 | +| `roboco/services/coroner_engine.py` | `CoronerEngine` — event-only (`open_for_incident`, no `run_cycle`/cron path); incident + transition-history context | 237 | +| `roboco/services/sentinel_engine.py` | `SentinelEngine` — weekly cron, org-scoped drift report (waivers/findings/conventions/spend) | 261 | +| `roboco/services/librarian_engine.py` | `LibrarianEngine` — biweekly cron, org-scoped proactive playbook mining | 232 | ## Key Symbols @@ -8746,6 +8771,30 @@ The product / strategy / research / pitch slice covers the "company layer" above | `get_x_engine` | factory | x_engine.py:869 | Session-bound constructor (optional injected `XClient` for tests) | | `CompanyGoalsService.resolve_product_name` | method | company_goals.py:79 | The shared product-name fallback chain: `project.name` if set, else the charter's `company_name`, else the "RoboCo" literal — single source so `XEngine`/`VideoEngine` can't drift apart on branding | | `task_project_fields` | func | api/schemas/project_fields.py:19 | `(project_slug, project_name)` or `(None, None)` for a task response — `sa_inspect(task).unloaded` guard before touching `task.project` (a freshly-created task can have an unloaded relationship); shared by the X and video queue response builders so a multi-project CEO can tell drafts apart via the panel's `ProjectBadge` | +| `BoardProgram` | dataclass | foundation/policy/board_programs.py:31 | Frozen registry entry: `key`/`role`/`trigger`/`source`/`default_interval_seconds`/`max_items_per_cycle`/`scope` | +| `PROGRAMS` | dict | foundation/policy/board_programs.py:45 | All 14 registered programs, keyed by `key` | +| `program_due` | func | foundation/policy/board_programs.py:225 | Pure cron-due check; METRIC/EVENT programs always return False (opened by their own hooks, never the loop) | +| `project_participates` | func | foundation/policy/board_programs.py:241 | Dual-polarity scope predicate — affirmative opt-in for `scope="project"`, opt-out (`"!key"`) for `scope="org"` | +| `validate_board_programs_field` | func | foundation/policy/board_programs.py:258 | Rejects an unknown key or a polarity mismatched to the program's own scope | +| `BoardProgramEngine` | class | services/board_programs.py:293 | Trigger/dedup/originate/LEARN over every registered program | +| `BoardProgramEngine.run_due_programs` | method | services/board_programs.py:302 | Originates a cycle for every enabled+due CRON program, then every metric predicate that fires off-schedule; one program's failure never blocks the rest | +| `BoardProgramEngine.open_program_cycle` | method | services/board_programs.py:373 | Enabled+scope+dedup only, no cron-due check — the CEO "run now" / strategy-engine idle-trigger seam | +| `BoardProgramEngine.record_decision` / `prior_cycle_context` | method | services/board_programs.py:422 / 466 | LEARN: accrue a CEO approve/reject onto the cycle row; render the last N closed cycles for the next exploration prompt | +| `program_armed` | func | services/board_programs.py:274 | THE arming chokepoint — settings-store `board_program.{key}.enabled`, falling back to a legacy flag only for `roadmap`/`x_feature` | +| `pick_rotation_target` | func | services/board_programs.py:191 | Shared round-robin for project-scoped programs (Pest Control/Spackle/Mirror/Dogfood): never-explored first, else oldest `last_opened_at` | +| `PestControlEngine.run_cycle` / `evidence_context` | method | pest_control_engine.py:79 / 152 | Weekly+metric cron; server-assembles rework-hotspot/recurring/waived-finding evidence for the PO's prompt | +| `SpackleEngine.run_cycle` | method | spackle_engine.py:71 | Biweekly cron, project-scoped gap-fill audit | +| `ScalesEngine.run_cycle` / `_stale_backlog_snapshot` | method | scales_engine.py:70 / 140 | Monthly cron; snapshots BACKLOG/PENDING tasks older than 30 days for the rebalance prompt | +| `DogfoodEngine.run_cycle` | method | dogfood_engine.py:78 | Event-only; real originator (unlike Coroner's stub) — needs no external incident id, picks the next opted-in project via `pick_rotation_target` | +| `PeriscopeEngine.run_cycle` / `latest_brief_context` | method | periscope_engine.py:71 / 126 | Weekly cron; the latest closed brief is injected into Printer's own exploration prompt | +| `MegaphoneEngine.run_cycle` / `digest_context` | method | megaphone_engine.py:74 / 133 | 3-day cron; server-assembles the shipped-this-week digest + Unreleased CHANGELOG section | +| `MirrorEngine.run_cycle` | method | mirror_engine.py:71 | Quarterly cron, project-scoped messaging-drift audit | +| `BarflyEngine.run_cycle` / `_screen_and_mark` | method | barfly_engine.py:88 / 145 | 2-day cron; screens each X search candidate through `injection_guard.screen_external_text` before it reaches the HoM's prompt | +| `WarRoomEngine.run_cycle` / `open_for_release` | method | war_room_engine.py:111 / 120 | `run_cycle` is the CEO on-demand blank-brief path (reachable via `open_program_cycle`); `open_for_release` bypasses `_ORIGINATORS` entirely, called from the release-publish hook with pre-curated highlights | +| `CoronerEngine.open_for_incident` / `incident_context` | method | coroner_engine.py:75 / 195 | The ONLY way a Coroner cycle opens — called directly from three chokepoints (bounce>=3, cancel-after-work, budget-block), never the cron loop; `_ORIGINATORS["coroner"]` is an always-`None` stub that only exists so the dict covers the registry 1:1 | +| `SentinelEngine.run_cycle` / `evidence_context` | method | sentinel_engine.py:79 / 131 | Weekly cron; server-assembles waiver-trend/open-findings/conventions-hotspot/spend evidence | +| `LibrarianEngine.run_cycle` / `mining_context` | method | librarian_engine.py:89 / 140 | Biweekly cron; server-assembles recurring learning-journal topics + existing playbook titles to mine against | +| `get_pest_control_engine` / `get_spackle_engine` / `get_scales_engine` / `get_dogfood_engine` / `get_periscope_engine` / `get_megaphone_engine` / `get_mirror_engine` / `get_barfly_engine` / `get_war_room_engine` / `get_coroner_engine` / `get_sentinel_engine` / `get_librarian_engine` / `get_board_program_engine` | factory | each engine's own file | Session-bound constructors, one per engine | ## Data Flow @@ -8763,6 +8812,12 @@ Two distinct flows originate work into the delivery lifecycle: **X (Twitter) flow (three originators, one held queue, default off).** Unlike every other engine on this page, `XEngine` never spawns an agent for release posts or mention replies — `draft_release_post` (event hook off `ReleaseProposalService.approve`'s publish-success branch) and `run_cycle` (periodic mentions poll, `Orchestrator._x_mentions_poll_loop`) both draft via a raw local-model chat completion, never a cloud LLM. The feature-spotlight half is the exception: `Orchestrator._x_feature_spotlight_loop` (dormant unless BOTH `x_engine_enabled` AND `x_feature_spotlight_enabled`) opens a DB context each `x_feature_spotlight_interval_seconds` and calls `XEngine.open_feature_spotlight_exploration`, which no-ops on the usual guards (creds, one-open-cycle dedup, the shared `x_max_open_posts` cap, project resolvability) or else opens ONE held PENDING exploration task (`source=x_feature_exploration`) assigned to the Head of Marketing, carrying a snapshot of already-covered feature slugs (`x_seen_features` marker). The board dispatcher's `_dispatch_pm_work` special-cases this source (mirroring `ROADMAP_SOURCE`) to call `_dispatch_feature_spotlight_exploration`, a one-shot spawn of the real Head-of-Marketing agent (full read tools) who investigates CHANGELOG.md/feature-flags/docs/map/charter/KB and calls the `propose_feature_spotlight` do-tool exactly once; that verb materializes a brand-new held draft task (`source=x_feature`) and completes the exploration task as a side effect — a deliberate asymmetry from `propose_roadmap`, which instead writes a marker onto the SAME task and leaves it open. Every draft from all three paths — release, reply, spotlight — lands in the identical held-task shape (`TaskTable`, `confirmed_by_human=False`, `assigned_to=secretary-1`, body in `orchestration_markers.x_draft_body`) rendered by the panel's X Post Queue and acted on only by `XPostService.approve`/`.reject`; nothing here ever calls `x_client.post_tweet` itself. `XEngine._voice_guide` (a live `CompanyGoalsService.get()` read, never hardcoded) feeds a baseline house-voice constant plus the CEO's optional `brand_voice` charter sample into every one of the two local-model prompts, and the Head of Marketing's own identity prompt points it at the same charter field for its cloud-LLM-authored spotlight body. +**Board Program flow (registry, no master flag, default off per program).** The orchestrator's `_board_program_loop` ticks `BoardProgramEngine.run_due_programs` on a floor interval (shortest registered cadence, clamped 300s-3600s). Per CRON program: `program_armed` (settings-store `board_program.{key}.enabled`, falling back to a legacy flag only for `roadmap`/`x_feature`) → `_scope_gate` (a `scope="project"` program needs at least one project with the key in `projects.board_programs`) → dedup against `board_program_cycles` (migration `087`, one open row per program, auto-closed once its exploration task goes terminal) → `program_due` → `_ORIGINATORS[key]` calls that program's own `run_cycle`, which opens ONE held PENDING exploration task assigned to the program's role (Product Owner: Pest Control/Spackle/Scales/Dogfood; Head of Marketing: Periscope/Megaphone/Mirror/Barfly/War Room; Auditor: Sentinel/Librarian) and records a fresh `board_program_cycles` row. `run_due_programs` separately evaluates every registered metric predicate (`_METRIC_PREDICATES`, today only Pest Control's 7-day rework-rate check against `ROBOCO_PEST_REWORK_THRESHOLD`) after the same scope/dedup gates, so an off-schedule accelerator never re-pays a multi-query metric check on a tick that was always going to be rejected. `open_program_cycle(key)` is the same path minus cron-due — used by the CEO panel's "run now" (`POST /api/board-programs/{key}/run-now`), the Strategy Engine's `idle` observation (Printer only — the design's `stranded_blocked` → Coroner fold was never wired), and Dogfood's release-publish hook. Coroner is the exception to the whole loop: its `trigger=event` means `program_due` always refuses it, and its ONLY real entry point is `CoronerEngine.open_for_incident`, called directly from three chokepoints — `TaskService`'s bounce-past-`revision_count>=3` transition, `TaskService`'s cancel-after-work path, and the orchestrator's budget-block path — never the cron loop. War Room's release cycle similarly bypasses `_ORIGINATORS` via `open_for_release`, called from the same release-publish hook as `draft_release_post`/Dogfood, carrying pre-curated highlights so campaign posts never invent a feature. + +Every exploration task dispatches through `_dispatch_board_program_exploration` — a dict-dispatch table (not an `if`/`elif` chain, xenon budget) keyed by `task['source']`, routing to a dedicated one-shot spawner (`_dispatch_pest_control_exploration`, etc.) that bypasses `_handle_board_assigned_task`'s two-reviewer board-review gate entirely; every dispatcher shares the `_board_dispatched` one-shot tracker + respawn breaker. The agent calls its program's ONE proposal verb (`propose_bug_hunt`/`propose_gap_fill`/`propose_rebalance`/`propose_friction_fixes` for the PO; `propose_market_brief`/`propose_editorial_post`/`propose_messaging_fixes`/`propose_campaign`/`propose_conversation_replies` for the HoM; `propose_postmortem`/`propose_playbook_drafts`/`propose_quality_report` for the Auditor — all in `roboco/services/gateway/content_actions.py`) exactly once. Materialization varies by program: most (Pest Control/Spackle/Mirror/roadmap) create BACKLOG tasks with a per-item CEO decision identical to the roadmap flow; Scales instead MUTATES a live task in place on approval (reprioritize or cancel — never creates one); Periscope/Sentinel complete their exploration task in the same call as a held report with no per-item queue; Megaphone/Barfly/War Room/spotlight land in the existing X held-draft queue; Coroner materializes a held process-change item or drafts straight into the pending-playbook queue (`kind='playbook'`); Librarian drafts 1-3 real DRAFT playbooks directly via `PlaybookService`, bypassing `draft_playbook` entirely (an explicit invariant: the Auditor curates but does not draft, except here). LEARN closes the loop: `BoardProgramEngine.record_decision` accrues each CEO verdict onto the cycle row's `decisions` jsonb, and `prior_cycle_context` renders the last two closed cycles back into the NEXT cycle's exploration prompt. + +Project-scoped programs (Pest Control/Spackle/Mirror/Dogfood) additionally use `pick_rotation_target` to round-robin across their opted-in projects — never-explored beats explored, else oldest `last_opened_at` wins, read from the programs' own exploration tasks (not the LEARN ledger, since a project-scoped engine's `run_cycle` can be called directly, outside the loop). `projects.board_programs` (migration `088`) governs opt-in/opt-out with dual polarity per `project_participates` — a plain key for a `scope="project"` program, `"!key"` to exclude a project from a `scope="org"` program's default-eligible output. + **Read-only views.** `KanbanService` builds role-specific boards from `TaskTable` queries on demand for the kanban API; `CompanyGoalsService.get` is read by the briefing injector into every agent's `context_briefing`. ## Mermaid @@ -8821,6 +8876,30 @@ flowchart TD CEO -->|approve/reject /api/x/posts| XSvc[XPostService] XSvc -->|approve, single-flight lock| Tweet[(x_client.post_tweet)] end + + subgraph BoardProgramLoop["Board Program registry — 14 entries, per-program settings-store arming"] + BPLoop[Orchestrator._board_program_loop] -->|floor interval| RunDue[BoardProgramEngine.run_due_programs] + RunDue --> Armed{program_armed settings-store} + Armed -->|CRON, scope+dedup+due| Origin[program._ORIGINATORS run_cycle] + RunDue --> MetricCheck[_run_due_metric_predicates: pest_control rework-spike] + MetricCheck --> Origin + CoronerHook["TaskService bounce/cancel + budget-block hook"] --> CoronerOpen[CoronerEngine.open_for_incident] + ReleaseHook[ReleaseProposalService.approve publish] --> WarRoomOpen[WarRoomEngine.open_for_release] + ReleaseHook --> DogfoodRun[DogfoodEngine.run_cycle] + Origin -->|held PENDING task| Explorer["PO / HoM / Auditor spawn (solo, board-review gate bypassed)"] + CoronerOpen -->|held PENDING task| Explorer + WarRoomOpen -->|held PENDING task| Explorer + Explorer -->|ONE propose_* verb| ProposeVerb[ContentActions.propose_*] + ProposeVerb --> Materialize{materializer} + Materialize -->|backlog tasks, per-item decision| Backlog[(BACKLOG task)] + Materialize -->|held report, no queue| Report[(CEO report)] + Materialize -->|held X draft| XQueue2[(X post queue)] + Materialize -->|mutate live task| LiveTask[(reprioritize / cancel)] + Materialize -->|playbook draft| PlaybookQ[(pending-playbook queue)] + CEO -->|approve/reject per item| Learn[BoardProgramEngine.record_decision] + Learn -->|LEARN| Ledger[(board_program_cycles.decisions)] + Ledger -->|prior_cycle_context| Origin + end ``` ## Logical Tree @@ -8889,9 +8968,29 @@ product-strategy-research-pitch │ └── reject (record reason, cancel draft) ├── x_client.py — XClient ABC / NullXClient / LiveXClient │ └── build_x_client (creds present → LiveXClient, else NullXClient) -└── x_credentials.py — XCredentialsService (singleton, Fernet-encrypted) - ├── set_credentials (all-or-nothing) - └── get_decrypted (server-side only) +├── x_credentials.py — XCredentialsService (singleton, Fernet-encrypted) +│ ├── set_credentials (all-or-nothing) +│ └── get_decrypted (server-side only) +├── foundation/policy/board_programs.py — pure registry (no IO) +│ ├── BoardProgram (frozen dataclass) + PROGRAMS (14 entries) +│ ├── program_due (cron-due check) +│ ├── project_participates (dual-polarity scope predicate) +│ └── validate_board_programs_field +├── services/board_programs.py — BoardProgramEngine +│ ├── run_due_programs / _run_due_metric_predicates (cron + metric pass) +│ ├── open_program_cycle (enabled+scope+dedup, no cron-due — "run now"/idle-trigger seam) +│ ├── _scope_gate / opted_in_projects +│ ├── _dedup_state / _maybe_close / _latest_cycle (board_program_cycles ledger) +│ ├── record_decision / prior_cycle_context (LEARN) +│ ├── program_armed (settings-store arming chokepoint) +│ └── pick_rotation_target (shared project-scoped round-robin) +├── pest_control_engine.py / spackle_engine.py / scales_engine.py / dogfood_engine.py — Product Owner programs +│ └── each: run_cycle (CRON) + a program-specific evidence/context builder; dogfood_engine also binds a real _ORIGINATORS entry despite being event-only +├── periscope_engine.py / megaphone_engine.py / mirror_engine.py / barfly_engine.py / war_room_engine.py — Head of Marketing programs +│ └── each: run_cycle + context builder; war_room_engine also exposes open_for_release (release-hook bypass of _ORIGINATORS); barfly_engine screens candidates through injection_guard +├── coroner_engine.py / sentinel_engine.py / librarian_engine.py — Auditor programs +│ └── coroner_engine: open_for_incident is the ONLY real entry point (event-only, no run_cycle path through the loop); sentinel_engine/librarian_engine: run_cycle (CRON) + context builder +└── api/routes/board_programs.py — CEO-only status + run-now routes ``` ## Dependencies @@ -8915,7 +9014,11 @@ product-strategy-research-pitch - `roboco.services.notification` — `StrategyEngine.run_cycle`. - `roboco.services.github_provisioning` — `PitchService.approve`. - `roboco.services.project` / `product` — `PitchService`. -- `roboco.runtime.orchestrator` — runs `_strategy_engine_loop` + `_roadmap_engine_loop`/`_dispatch_roadmap_exploration`; mounts `roboco-search` MCP when `research_enabled`. +- `roboco.runtime.orchestrator` — runs `_strategy_engine_loop` + `_roadmap_engine_loop`/`_dispatch_roadmap_exploration` + `_board_program_loop`/`_dispatch_board_program_exploration`; mounts `roboco-search` MCP when `research_enabled`, `playwright` MCP task-scoped for Dogfood. +- `roboco.services.gateway.content_actions.ContentActions` — the fourteen `propose_*` do-verbs (one per program) that author each program's proposal; `roboco.api.schemas.v1.do` — the matching `*Input`/`Propose*Request` pydantic schemas. +- `roboco.services.metrics.MetricsService.get_rework_metrics` — Pest Control's off-schedule metric predicate. +- `roboco.foundation.policy.injection_guard.screen_external_text` — Barfly screens every candidate conversation through it before the HoM's prompt sees it. +- `roboco.services.playbook.PlaybookService` — Coroner (`kind='playbook'`) and Librarian both draft directly into it, never through the `draft_playbook` do-tool. **External:** - `sqlalchemy` (async ext) — all DB-backed services. @@ -8937,9 +9040,11 @@ product-strategy-research-pitch - `dashboard.py` — `get_product_service` / `get_project_service` for dashboard views. - `roadmap.py` — `GET /api/roadmap/cycles`, `POST /cycles/{id}/items/{id}/{approve,reject}` (CEO-only) → `get_roadmap_service`. - `x.py` — `GET /api/x/posts`, `POST /posts/{id}/{approve,reject}` (CEO-only) → `get_x_post_service`. -- **Orchestrator loop tick:** `_strategy_engine_loop` (orchestrator.py:6360) — created at `start()` (line 1010), cancelled in shutdown (line 1075); ticks every `strategy_engine_interval_seconds`, calls `StrategyEngine.run_cycle`. `_roadmap_engine_loop` (orchestrator.py:7462) — same lifecycle shape, ticks every `roadmap_interval_seconds` (default weekly), calls `RoadmapEngine.run_cycle`; `_dispatch_roadmap_exploration` (orchestrator.py:10284) spawns the Product Owner once per open exploration task. `_x_mentions_poll_loop` (orchestrator.py:7509) ticks every `x_mentions_interval_seconds`, calls `XEngine.run_cycle`. `_x_feature_spotlight_loop` (orchestrator.py:7571) — same lifecycle shape, dormant unless BOTH `x_engine_enabled` AND `x_feature_spotlight_enabled`, ticks every `x_feature_spotlight_interval_seconds` (default 3 days), calls `XEngine.open_feature_spotlight_exploration`; `_dispatch_feature_spotlight_exploration` (orchestrator.py:10424) spawns the Head of Marketing once per open exploration task — `_dispatch_pm_work` routes `source=x_feature_exploration` to it BEFORE the generic `_BOARD_AGENTS` check (mirroring the roadmap source's own early branch), so it never falls into the two-reviewer board-review gate. -- **MCP mount (orchestrator spawn):** `roboco-search` MCP mounted into Board/PM agent containers only when `research_enabled` (orchestrator.py:2914); the MCP server calls the `/api/research/*` routes. -- **Service-to-service:** `ProjectService` called by `WorkspaceService`, `GitService`, `PitchService`, `task`, `docs`, `cockpit`, `secretary`, gateway choreographer; `ProductService.project_for` called from gateway delegate path; `CompanyGoalsService.get` called by briefing injector. + - `board_programs.py` — `GET /api/board-programs` (list all 14 with live status), `POST /api/board-programs/{key}/run-now` (CEO-only) → `get_board_program_engine`. +- **Orchestrator loop tick:** `_strategy_engine_loop` (orchestrator.py:6360) — created at `start()` (line 1010), cancelled in shutdown (line 1075); ticks every `strategy_engine_interval_seconds`, calls `StrategyEngine.run_cycle`. `_roadmap_engine_loop` (orchestrator.py:7462) — same lifecycle shape, ticks every `roadmap_interval_seconds` (default weekly), calls `RoadmapEngine.run_cycle`; `_dispatch_roadmap_exploration` (orchestrator.py:10284) spawns the Product Owner once per open exploration task. `_x_mentions_poll_loop` (orchestrator.py:7509) ticks every `x_mentions_interval_seconds`, calls `XEngine.run_cycle`. `_x_feature_spotlight_loop` (orchestrator.py:7571) — same lifecycle shape, dormant unless BOTH `x_engine_enabled` AND `x_feature_spotlight_enabled`, ticks every `x_feature_spotlight_interval_seconds` (default 3 days), calls `XEngine.open_feature_spotlight_exploration`; `_dispatch_feature_spotlight_exploration` (orchestrator.py:10424) spawns the Head of Marketing once per open exploration task — `_dispatch_pm_work` routes `source=x_feature_exploration` to it BEFORE the generic `_BOARD_AGENTS` check (mirroring the roadmap source's own early branch), so it never falls into the two-reviewer board-review gate. `_board_program_loop` (orchestrator.py:9224) — same lifecycle shape, ticks on a floor interval (`_board_program_interval_seconds`: shortest registered program cadence, clamped 300s-3600s), calls `BoardProgramEngine.run_due_programs`; `_dispatch_board_program_exploration` (a module-level dict-dispatch function, not a method — orchestrator.py:948) routes each program's held exploration task to its own one-shot dispatcher (`_dispatch_pest_control_exploration`, `_dispatch_periscope_exploration`, etc.), each spawning its program's role solo, bypassing the two-reviewer board-review gate exactly like the roadmap/spotlight dispatchers already did. +- **MCP mount (orchestrator spawn):** `roboco-search` MCP mounted into Board/PM agent containers only when `research_enabled` (orchestrator.py:2914); the MCP server calls the `/api/research/*` routes. `playwright` MCP mounted task-scoped (not role-blanket) for a `board_dogfood` spawn only, via `_is_dogfood_spawn` (orchestrator.py:3834). +- **Event hooks (bypass the loop entirely):** `TaskService`'s bounce-into-`needs_revision` transition and cancel-after-work path both call `CoronerEngine.open_for_incident` directly (`services/task.py:812` / `:1392`); the orchestrator's budget-block path calls it too (`orchestrator.py:8311`); `ReleaseProposalService.approve`'s publish-success branch calls `WarRoomEngine.open_for_release` (`services/release_proposal.py:330`) alongside the pre-existing `XEngine.draft_release_post` hook. +- **Service-to-service:** `ProjectService` called by `WorkspaceService`, `GitService`, `PitchService`, `task`, `docs`, `cockpit`, `secretary`, gateway choreographer; `ProductService.project_for` called from gateway delegate path; `CompanyGoalsService.get` called by briefing injector; `BoardProgramEngine` called by every program's own engine (dedup/record) and by `StrategyEngine.run_cycle` (Printer's `idle` trigger). - **No CLI / lifespan entry points** for this slice. ## Config Flags @@ -8971,6 +9076,7 @@ product-strategy-research-pitch | `ROBOCO_ROADMAP_INTERVAL_SECONDS` | `604800` | config.py:875 | Seconds between roadmap-exploration cycles (default weekly) | | `ROBOCO_ROADMAP_MIN_ITEMS_PER_CYCLE` | `3` | config.py:880 | Minimum item drafts `propose_roadmap` must submit for a themed cycle | | `ROBOCO_ROADMAP_MAX_ITEMS_PER_CYCLE` | `7` | config.py:885 | Maximum item drafts per cycle | +| `ROBOCO_PEST_REWORK_THRESHOLD` | `0.3` | config.py:1432 | 7-day rework rate above which Pest Control's metric predicate opens a cycle off-schedule, on top of its weekly cron. The ONLY env-settable knob among the twelve new Board Programs — every other one arms exclusively via its own settings-store row (`board_program.{key}.enabled`, no `ROBOCO_*_ENABLED` flag exists for them) | ## Gotchas @@ -8986,6 +9092,7 @@ product-strategy-research-pitch - **`build_provider` returns `NullProvider` for an unknown provider name (research.py:326- 328).** A typo in `ROBOCO_RESEARCH_PROVIDER` (validated by pydantic pattern, so unlikely) would silently degrade to empty results rather than erroring. - **`GitHubProvisioningService.enabled` requires master + token + org (github_provisioning.py:64).** `provisioning_enabled` defaults `True`, so the flag alone is not enough — an operator who toggles the flag without setting token/org still gets `enabled=False` and `approve` raises `ProvisioningDisabledError`. - **`PitchService._seed_main_pm_task` requires a `main-pm` agent row (pitch.py:241-243).** If the agent slug is missing it raises `ValidationError` after provisioning has already happened — another partial-failure window (repos + Product created, no seed task). +- **The Strategy Engine's `stranded_blocked` → Coroner fold was designed but never wired.** The internal design spec (`docs/internal/specs/2026-07-24-board-programs-design.md` §3) proposed both `StrategyEngine` signals becoming Board Program triggers — `idle` → Printer (roadmap) and `stranded_blocked` → Coroner. Only the `idle`→roadmap half shipped (`strategy_engine.py:95-101`'s own docstring: "`stranded_blocked` stays notify-only (Coroner is Phase 2 — its event hook lands then)"). Coroner is reachable only through its own three chokepoints (bounce/cancel/budget-block); a long-stranded blocked task never triggers an autopsy on its own. Not a bug — a deliberately scoped-down Phase 1, but a real gap between the design doc and the shipped code worth knowing before assuming the fold is complete. - **`XPostService.approve` did NOT check for a CANCELLED (already-rejected) task before Wave 5 (`11915f36`, PR #551).** Before the fix, approving a draft the CEO had already rejected would proceed straight to posting it — reachable via the Telegram inbound bridge's inline Approve button (targets a draft by id regardless of its current status) and equally via a replayed HTTP `POST /api/x/posts/{id}/approve`. The guard now returns `already_rejected` both before acquiring the lock and again after re-reading the task under lock. ## Drift from CLAUDE.md @@ -8997,6 +9104,7 @@ product-strategy-research-pitch - **CLAUDE.md does not mention `ROBOCO_PROTECTED_GIT_URLS`** (the project denylist, config.py:770, project.py:40). Doc omission. - **CLAUDE.md's service table does not list `KanbanService`, `CompanyGoalsService`, `StrategyEngine`, `ResearchService`, `PitchService`, `GitHubProvisioningService`.** The CLAUDE.md "Services" table is explicitly a non-exhaustive "Core services" list, so this is an acknowledged omission rather than drift. - **No contradictions between CLAUDE.md claims and actual code were found in this slice.** All documented flags, defaults, and behaviors (default-off strategy engine, server-side- only keys, notify-only engine, pitch→provision→normal-lifecycle, CEO-only approve) match the code. +- **CLAUDE.md's "Board Program registry" entry documents the shipped scope accurately, including the `stranded_blocked`→Coroner gap.** Code matches: `program_armed` has no master flag (services/board_programs.py:274), the strategy-engine fold is `idle`-only (strategy_engine.py:95-101). No drift. ## Changes Since Baseline @@ -9016,6 +9124,7 @@ product-strategy-research-pitch > - `7e01c0ce` (PR #570, "project-branded drafts + project badges", 2026-07-18): migration 075 adds `company_goals.company_name`; `CompanyGoalsService.resolve_product_name` (company_goals.py:79) is the new single fallback chain (project name → charter `company_name` → "RoboCo") consumed by both `XEngine._voice_guide`/`draft_release_post` and `VideoEngine` (see `docs/map/video-engine.md`) so their prompt builders stop hardcoding "RoboCo". New `roboco/api/schemas/project_fields.py`'s `task_project_fields` helper adds `project_slug`/`project_name` to the X and video post-queue API responses (`api/routes/x.py`, `api/routes/video.py`); the panel renders them via a shared `ProjectBadge` — see `docs/map/panel.md`. > - `461a6e1a`+`96401f4c`+`5f32d876` (Phases 1/2-3/4, 2026-07-18/19, #571/#575/#581) — Phase 4 makes `GitHubProvisioningService` provider-aware: `_build_provider` (github_provisioning.py:68) dispatches to `GitHubProvider`/`GiteaProvider`/`GitLabProvider` by `ROBOCO_PROVISIONING_PROVIDER`, `.enabled` additionally requires `ROBOCO_PROVISIONING_HOST` for gitlab/gitea, and `_is_already_exists` (github_provisioning.py:61) matches the "already exists" idempotency signal across all three forges' differing status codes/phrasing. The forge transport package itself (`GitProvider`/`ForgeRouter`/provider implementations) is documented in `docs/map/worksession-git.md` — this slice only covers the provisioning consumer. > - `a0baf94b` ("agnosticism-residue", agnosticism audit items B6/B8): `x_engine.py`'s remaining hardcoded `"RoboCo"` literals (the reply-prompt builder and the feature-spotlight exploration description — `draft_release_post`/`_voice_guide` were already fixed by `7e01c0ce` above) are threaded out: `_reply_prompt` gains a `product_name` param, `_FEATURE_EXPLORATION_DESCRIPTION` (a module constant) becomes `_feature_exploration_description(product_name)` (a function), and `run_cycle`/`open_feature_spotlight_exploration` each resolve `product_name` once via `resolve_product_name` and thread it through. +> - **Board Program registry (2026-07-24, #689/#699 + the Phase 2/3 program train).** The single largest addition to this slice since the baseline: `foundation/policy/board_programs.py` (`BoardProgram`/`PROGRAMS`/`program_due`/`project_participates`) + `services/board_programs.py` (`BoardProgramEngine`) + `api/routes/board_programs.py` generalize the roadmap/spotlight shape into one registry-driven engine (migrations `087` `board_program_cycles` LEARN ledger, `088` `projects.board_programs` scoping column), migrating `roadmap` and `x_feature` onto it byte-for-byte (Phase 1) before adding twelve new programs across all three Board roles (Phase 2/3): Pest Control/Spackle/Scales/Dogfood (`pest_control_engine.py`/`spackle_engine.py`/`scales_engine.py`/`dogfood_engine.py`, Product Owner), Periscope/Megaphone/Mirror/Barfly/War Room (`periscope_engine.py`/`megaphone_engine.py`/`mirror_engine.py`/`barfly_engine.py`/`war_room_engine.py`, Head of Marketing), and Coroner/Sentinel/Librarian (`coroner_engine.py`/`sentinel_engine.py`/`librarian_engine.py`, Auditor). Arming has no master flag — `program_armed` reads a per-program settings-store row exclusively, except `roadmap`/`x_feature`'s legacy env-flag fallback. `StrategyEngine.run_cycle`'s `idle` observation now also triggers a Printer cycle via `BoardProgramEngine.open_program_cycle("roadmap")` (the `stranded_blocked`→Coroner half of the same design was NOT built — see Gotchas). Fourteen new `propose_*` do-verbs land in `content_actions.py` + `api/schemas/v1/do.py`; the Playwright MCP grant is task-scoped to Dogfood only, not a role-wide product_owner grant. ## Regression Risks @@ -9353,6 +9462,7 @@ The Next.js 16 control panel (`panel/`, package `roboco-panel` v0.25.0) is the s | `WorkSessionsView` | comp | `components/work-sessions/work-sessions-view.tsx` | Git page's "Work Sessions" tab body; search/status filters are LOCAL `useState`, not URL params | | `SessionTrendChart` | comp | `components/work-sessions/session-trend-chart.tsx` | Active-session start-time histogram (hourly/daily bucketing); honestly labeled active-only, no history beyond `GET /work-sessions` | | `CostTrendChart` / `SpendTrendChart` | comp | `components/dashboard/cost-trend-chart.tsx` / `components/business/spend-trend-chart.tsx` | Daily-spend area charts off `GET /usage/time-series`; 7d on Overview (`CommandCenter`), 30d on the Business scorecard (`CompanyScorecardCard`) | +| `CompanyScorecardCard` / `ObjectivesSection` | comp | `components/business/company-scorecard-card.tsx` | Business page Scorecard tab body; four sections off one `cockpitApi.summary()` call. `ObjectivesSection` renders three positional charter objective cards (`first_pass_yield` 90%, `median_lead_time_hours` <24h, `escaped_defects` 0) — positional-by-convention mapping documented in a `ponytail:` code comment; "No data yet" fallback for null/undefined metrics (mirrors `SpeedSection`). `CockpitSummary` gained optional `first_pass_yield?: number\|null` and `escaped_defects?: number\|null` (backend companion item not yet shipped — see `panel/docs/frontend/company-scorecard-card.md`). | | `ProductCardGrid` | comp | `components/products/product-card-grid.tsx` | Workstation Products card-grid view; reuses `CellsList`/`ProgressCell` (exported from `product-table.tsx`) | | `ProjectCardGrid` | comp | `components/projects/project-card-grid.tsx` | Workstation Projects card-grid view; reuses `getExternalUrl`/badge renderers (exported from `project-table.tsx`) | | `sortProducts` | fn | `components/products/products-view.tsx` | Pure client-side sort (name/cell count) for the Products card grid; direction rides a comparator multiplier, not sort-then-reverse, so ties keep their relative order | @@ -9603,7 +9713,7 @@ The pytest test suite for RoboCo: 571 test_*.py files across tests/foundation, t | tests/e2e_smoke/harness.py | E2E harness: E2EStack app + orchestrator client + per-test agent manifests; used by the e2e_smoke tier | ~520 | | tests/e2e_smoke/test_gitea_live.py | Live-Gitea contract suite for `GiteaProvider` — fully self-seeding (creates its own uniquely-named repo, pushes real commits) against a real Gitea instance; skipped unless `ROBOCO_GITEA_E2E_URL`/`ROBOCO_GITEA_E2E_TOKEN` are both set; exercises PR open → duplicate-409→422 reshape → list/filter → diff → comment review → commit-status CI reshape → squash merge → branch delete → release, plus the git-CLI Basic-auth extraheader claim | 250 | | tests/e2e_smoke/test_gitlab_live.py | Live-GitLab contract suite for `GitLabProvider` — mirrors `test_gitea_live.py`, self-seeding against a real GitLab instance (gitlab.com works, project deleted afterward best-effort); skipped unless `ROBOCO_GITLAB_E2E_URL`/`ROBOCO_GITLAB_E2E_TOKEN` are both set; exercises MR open → duplicate reshape → GitHub-shape adaptation → diff reassembly → note review → commit-status CI reshape → squash merge → branch delete → release → the oauth2 Basic-auth git-CLI claim | 265 | -| roboco/eval/ (`fixtures.py`/`runner.py`/`__main__.py`) | Golden-task offline CLI bench harness (source-checkout-only, NOT a pytest suite) — replays fixed `BenchTaskSpec` fixtures through the real delivery lifecycle in a disposable environment REUSED from `tests/e2e_smoke/harness.py` (fake GitHub REST, a real local git origin, a throwaway DB), scoring on deterministic metrics + a local-model judge. `python -m roboco.eval run` is wired but not functional this release — `OrchestratorStageSpawner` raises `NotImplementedError` at construction (a real spawn would resolve to the production orchestrator under real agent UUIDs, unsafe for a bench run); the only working path today is driving `EvalRunner` with an injected scripted `StageSpawner` from Python. Scoped to developer-role fixtures only; only runs from a source checkout. See `tests/e2e_smoke/test_eval_bench.py`. | — | +| roboco/eval/ (`fixtures.py`/`runner.py`/`__main__.py`) | Golden-task offline CLI bench harness (source-checkout-only, NOT a pytest suite) — replays fixed `BenchTaskSpec` fixtures through the real delivery lifecycle in a disposable environment REUSED from `tests/e2e_smoke/harness.py` (fake GitHub REST, a real local git origin, a throwaway DB), scoring on deterministic metrics + a local-model judge. `python -m roboco.eval run` works end-to-end: `OrchestratorStageSpawner` drives a real `AgentOrchestrator.spawn_agent` per turn, and `_generate_mcp_config` honors the patched `settings.api_url` so a spawned container's MCP servers resolve to the throwaway orchestrator, never the real production one. **Container-reachability (PR #705, task `5cc75f71`):** `build_e2e_stack` binds the in-process uvicorn server to `0.0.0.0` (not `127.0.0.1`) so spawned agent containers on the `roboco_default` bridge can reach it; `E2EStack.container_url` is `http://host.docker.internal:{port}` (host-side `base_url` stays `127.0.0.1`), `_bench_environment` patches `settings.api_url` to `stack.container_url`, and `_build_mount_args` adds `--add-host host.docker.internal:host-gateway` to every agent `docker run` argv (Docker 20.10+, production-inert — prod MCP servers use `http://roboco-orchestrator:8000`). **Real-UUID isolation design:** `_seed_company` seeds agents under their REAL production UUIDs from `foundation.identity.AGENTS` (not random) so orchestrator-internal helpers keyed by that static registry (`get_agent_role`, `AGENT_UUIDS`, the UUID→slug reverse map) resolve exactly as in a real deployment; the AC wording "no real agent UUIDs" is satisfied by "no production DB/Redis reach" — the isolation boundary is the disposable URL + throwaway DB, NOT the UUID. A real UUID confers no production reach because the spawned container connects to the disposable orchestrator backed by a throwaway DB; randomizing would break the static-registry resolution and make the bench less realistic. `tests/unit/runtime/test_eval_mcp_config_isolation.py` pins this. Needs a Docker daemon + built agent images for the real spawn path; the injectable scripted `StageSpawner` (see `tests/e2e_smoke/test_eval_bench.py`) remains the unit-test fallback that proves the runner's plumbing without touching Docker. `tests/unit/eval/test_scoring.py::test_orchestrator_stage_spawner_constructs_real_orchestrator` pins the spawner's wired-construction contract (construction succeeds, `_orchestrator` is an `AgentOrchestrator`, `_stage_timeout_seconds` defaults to 900.0). Scoped to developer-role fixtures only; only runs from a source checkout. | — | ## E2E smoke harness @@ -9821,6 +9931,7 @@ tests/ > - **babffe0a** fix(e2e_smoke): repair auditor-trigger smoke tests and harden harness (#498): fixed `tests/e2e_smoke/test_auditor_triggers.py` so scheduled/reactive auditor-trigger tests reach their spawn assertions, hardened `tests/conftest.py` to tolerate missing pgvector, and cleared leaked `ROBOCO_AGENT_TOKEN` in `tests/e2e_smoke/harness.py` before scripted agents load `flow_server`. See the E2E smoke harness section above for the exact patterns. > - **f081a574** (PR #502, 2026-07-13): Follow-up e2e lifecycle smoke fix. Restored the `ROBOCO_AGENT_TOKEN` pop in `tests/e2e_smoke/harness.py:ScriptedAgent._module` after it was accidentally removed, and clarified the `/api/notifications` mount comment so it no longer implies the router was newly added. The orchestrator `__new__` pre-init from `89b68786` means `_fresh_orchestrator` no longer needs to manually set `_instances`. > - **`10f039c3`** (#655, "golden-task eval harness + doctrine cohort stamp"): adds `roboco/eval/` (see Files above), a bench harness reusing `tests/e2e_smoke/harness.py`'s disposable environment rather than a mock — real isolation. `agent_spawn_sessions.doctrine_version` (migration 081) is stamped at spawn-session finalize from the composed prompt layers, so a cohort's model+doctrine combination (e.g. Fable-mode on vs. off) is durably identifiable after the fact — see `docs/map/db-migrations.md`. Bench runs also patch every vault flag off so a bench task/note/journal write never lands in the operator's real Obsidian vault. +> - **`9882ebc6`** (#705, task `5cc75f71`, "fix disposable orchestrator container-reachability + document real-UUID isolation"): the in-process uvicorn server in `build_e2e_stack` now binds `0.0.0.0` (was `127.0.0.1`) so spawned agent containers on the `roboco_default` bridge can reach it; `E2EStack` gained a `container_url` field (`http://host.docker.internal:{port}`, set alongside `base_url` which stays `127.0.0.1` for host-side test clients); `_bench_environment` patches `settings.api_url` to `stack.container_url` (was `stack.base_url`); `_build_mount_args` adds `--add-host host.docker.internal:host-gateway` to every agent `docker run` argv (production-inert — prod MCP servers use `http://roboco-orchestrator:8000`; Docker 20.10+ supports host-gateway on Linux). F2 (docs-only): `_seed_company`'s docstring + `tests/unit/runtime/test_eval_mcp_config_isolation.py`'s module/test docstrings now state explicitly that the AC "no real agent UUIDs" is satisfied by "no production DB/Redis reach" — the isolation boundary is the disposable URL + throwaway DB, not the UUID; real UUIDs are intentional (orchestrator-internal helpers keyed by the static registry resolve as in a real deployment) and randomizing them would make the bench less realistic. > - **Forge providers, 2026-07-18/19 (#569/#571/#575/#579/#581)**: adds `tests/unit/services/forge/` (`test_gitea_provider.py` 297 lines/17 tests, `test_gitlab_provider.py` 713 lines/33 tests, `test_router.py` 90 lines/9 tests — all mocked-transport) + `tests/unit/foundation/policy/test_forge.py` (132 lines/23 tests, `extract_host`/`detect_provider`/`validate_project_forge`) + the live-forge contract suites `test_gitea_live.py`/`test_gitlab_live.py` (see the "Live-forge contract suites" section above). See `docs/map/worksession-git.md` for the forge package itself. ## Regression Risks diff --git a/docs/map/deployment-tooling.md b/docs/map/deployment-tooling.md index 9a768e43..96e76ec7 100644 --- a/docs/map/deployment-tooling.md +++ b/docs/map/deployment-tooling.md @@ -8,7 +8,7 @@ This slice is the packaging, build, and runtime-tooling layer of RoboCo: the Doc | docker-compose.yaml | Build-from-source compose: postgres/redis/ollama/ollama-init, 14 agent-*-image builders, orchestrator, panel, nginx, `backup` pg_dump sidecar on the `roboco_data` DB-isolation network; NAS prod env vars + volume mounts, `ROBOCO_OBSIDIAN_VAULT_ENABLED`/`ROBOCO_VAULT_INTAKE_ENABLED` default `true` | 556 | | docker-compose.yml | Byte-identical copy of docker-compose.yaml (kept for the canonical name compose picks up by default) | 556 | | docker-compose.registry.yml | Pull-and-run compose using pre-built GHCR/Docker Hub images (ROBOCO_REGISTRY + ROBOCO_VERSION); infra services byte-identical to build compose (incl. `backup` + vault defaults), agent-* services are one-shot pre-pulls, own `roboco_data` network | 334 | -| Makefile | Ops + quality targets: infra, dev/run/orchestrator, quality gate (ruff/mypy/pytest/xenon/radon/vulture/bandit/pip-audit/deptry/lint-imports/alembic/foundation-check), per-Python test matrix, docs, lifecycle regen | 548 | +| Makefile | Ops + quality targets: infra, dev/run/orchestrator, quality gate (ruff/mypy/pytest/xenon/radon/vulture/bandit/pip-audit/deptry/lint-imports/alembic/foundation-check), fast pre-submit `gate` target (ruff format/check, mypy, xenon, lint-imports), per-Python test matrix, docs, lifecycle regen | 599 | | pyproject.toml | Project + dependency manifest: requires-python >=3.13,<3.15, deps, dev/docs extras, console scripts, ruff/mypy/pytest/coverage/vulture/bandit/radon/xenon/deptry/importlinter config | 451 | | roboco/config.py | Pydantic Settings (env prefix ROBOCO_, cached via lru_cache); every tunable: DB, Redis, RAG, LLM/Ollama, workspaces, agent guardrails, gateway thresholds, autonomy-engine flags | 1022 | | roboco/__init__.py | Package root: __version__ + re-exports settings, exceptions, logging helpers | 39 | @@ -217,7 +217,8 @@ deployment-tooling | python -m roboco.cli / roboco console script | roboco/cli.py | orchestrator container ENTRYPOINT (docker/orchestrator.Dockerfile:98); `make orchestrator`; `make dev`; `make db-init` (--db-only) | | roboco-bootstrap console script | roboco/bootstrap.py | pyproject [project.scripts] alias (points at roboco.bootstrap:cli which does not exist — see drift) | | uvicorn roboco.api.app:app | roboco/bootstrap.py | spawned as asyncio task inside bootstrap.main (make api / make run run it directly) | -| make quality / foundation-check / lifecycle | Makefile | CI merge gate + local pre-submit | +| make quality / foundation-check / lifecycle | Makefile | CI merge gate | +| make gate | Makefile | Fast local pre-submit (`quality_command`, runs at dev `i_am_done`): ruff format/check, mypy, xenon, lint-imports — no tests | | scripts/build_lifecycle_artifacts.py | scripts/build_lifecycle_artifacts.py | make lifecycle (foundation-check runs it) | | scripts/verify_postgres_enums.py | scripts/verify_postgres_enums.py | make foundation-check (final step) | | scripts/reflow_md.py --check | scripts/reflow_md.py | make reflow-check / make quality |