mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
feat(vault): Obsidian vault V2 — janitor, archival, weekly report, KB ingest, Bases + sync runbook (#482)
* feat(vault): V2 — create-seam + drift janitor, archival, weekly org-report, KB ingest, Bases views + sync runbook Implements the vault V2 canonical spec end to end (the splice guard shipped separately and is reused at KB-ingest time): - materialize-on-create: TaskService.create writes each task's note best-effort from the moment it exists; the transition-touch stops no-oping on live work - drift janitor (services/vault_janitor.py + hourly _vault_janitor_loop): daily changed-task re-projection, random drift sample, archival pass — restart-proof via RoboCo/_meta/.janitor_state.json, 200/cycle caps, per-item isolation, processed-only resume markers, self-repairing state file - archival: vault_archive_days (30, 0=off) moves old terminal tasks' notes to RoboCo/Archive/<year>/Tasks/<project>/ — one write_task code path for janitor and rebuild, id8 lookup across Tasks/+Archive/, alias links keep moves safe - weekly org-report: VaultWriter.write_org_report renders Reports/<ISO-week>.md from MetricsService/UsageService (numbers duplicated into frontmatter for trend queries), once per ISO week, with a best-effort CEO notification - KB ingest: IndexType.VAULT_NOTES + VaultNotesIndexPlugin + _vault_kb_loop embed the CEO's RoboCo/Notes into the RAG corpus — injection guard as a hard gate (flagged notes quarantined with an idempotent callout), traversal- and symlink-contained at both config and engine layers, content-hash dedup, 50-ingest/cycle cap, frontmatter stripped; reaches roboco_kb_search, the mentor default domain, claim-time briefings (kind vault_note), and the panel KB browser; no migration (chunks table auto-creates; migration 030's CHUNK_TABLES tuple appended per the chunks_playbooks precedent) - Bases views (Task Board.base, Reports.base — schema verified against the Obsidian docs) + the Mac sync runbook vault asset - config/flags/compose: vault_archive_days, vault_report_enabled (flags card), vault_kb_enabled (flags card; NAS compose arms it, registry ships it off), vault_kb_dirs (+ overlap/traversal validator), vault_kb_interval_seconds - e2e smoke (tests/e2e_smoke/test_vault_v2.py): real create-seam, real janitor cycle incl. archival + state, real KB engine + real guard * docs: vault V2 sweep — map, RAG corpus, CLAUDE.md - docs/map/vault.md: V1+V2 — janitor/archival/report/KB data flows, new files, config, health posture - docs/map/orchestrator.md + task-service.md: the two new loops, the create seam, the three janitor queries - docs/rag/architecture/obsidian-vault.md: agent-facing what-changed (notes from creation, archive link-safety, CEO notes retrievable, weekly report) - docs/rag/architecture/config-reference.md: the five new settings - CLAUDE.md: vault paragraph covers V1+V2; flags-card list mentions the vault report/KB flags --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
@@ -347,7 +347,7 @@ Each agent gets a **spawn manifest** at `/app/tool-manifest.json` listing the ve
|
||||
| prompter | (none beyond `i_am_idle` — not a delivery-lifecycle role; intake interviewer, human-only) |
|
||||
| secretary | (none beyond `i_am_idle` — human-only chief-of-staff; reads company state + runs gated CEO directives) |
|
||||
|
||||
Content tools (do_server) — most roles: `commit`, `note`, `dm`, `read_a2a`, `evidence`. Delivery roles (developer / qa / documenter / cell_pm / main_pm) also get `draft_playbook` (draft a curated playbook for the KB). Product Owner additionally gets `propose_roadmap` (product_owner-only, authors the weekly board-roadmap cycle) and Head of Marketing additionally gets `propose_feature_spotlight` (head_marketing-only, drafts a feature-spotlight X post) — see "Board roadmap engine" / "RoboCo X account" below. Auditor is restricted to `note` (scope=reflect) + `evidence`, plus the playbook-curation verbs `approve_playbook` / `reject_playbook` / `archive_playbook` (a bounded, deliberate expansion — KB curation, not agent comms, so its no-`dm` restriction holds) and, when the Obsidian vault is armed, `curate_vault` (writes one narrative paragraph onto a just-completed root's vault note — see "Obsidian vault V1" below). The `pr_reviewer` posts its change-request on the PR itself (no agent comms). The `prompter` (intake) and `secretary` are restricted to `note` + `evidence` — human-only, no `dm`/`notify`. The `note`/journal write returns as soon as the entry is persisted; RAG indexing (Ollama embedding) runs fire-and-forget, so the tool no longer times out under concurrent load.
|
||||
Content tools (do_server) — most roles: `commit`, `note`, `dm`, `read_a2a`, `evidence`. Delivery roles (developer / qa / documenter / cell_pm / main_pm) also get `draft_playbook` (draft a curated playbook for the KB). Product Owner additionally gets `propose_roadmap` (product_owner-only, authors the weekly board-roadmap cycle) and Head of Marketing additionally gets `propose_feature_spotlight` (head_marketing-only, drafts a feature-spotlight X post) — see "Board roadmap engine" / "RoboCo X account" below. Auditor is restricted to `note` (scope=reflect) + `evidence`, plus the playbook-curation verbs `approve_playbook` / `reject_playbook` / `archive_playbook` (a bounded, deliberate expansion — KB curation, not agent comms, so its no-`dm` restriction holds) and, when the Obsidian vault is armed, `curate_vault` (writes one narrative paragraph onto a just-completed root's vault note — see "Obsidian vault V1+V2" below). The `pr_reviewer` posts its change-request on the PR itself (no agent comms). The `prompter` (intake) and `secretary` are restricted to `note` + `evidence` — human-only, no `dm`/`notify`. The `note`/journal write returns as soon as the entry is persisted; RAG indexing (Ollama embedding) runs fire-and-forget, so the tool no longer times out under concurrent load.
|
||||
|
||||
### MCP servers running per agent container
|
||||
|
||||
@@ -396,13 +396,13 @@ Agent backends are pluggable. `roboco/llm/providers/` defines an `AgentProvider`
|
||||
|
||||
**Board roadmap engine (default-off).** The Board originating strategic work: on a weekly interval (`ROBOCO_ROADMAP_ENGINE_ENABLED` + `_INTERVAL_SECONDS` / `_MIN_ITEMS_PER_CYCLE` / `_MAX_ITEMS_PER_CYCLE`) `RoadmapEngine` (`roboco/services/roadmap_engine.py`) opens ONE held **exploration** task (`source="board_roadmap"`, `confirmed_by_human=False`, PENDING, Product-Owner-assigned, `Team.BOARD`), deduped to one open cycle at a time. A dedicated one-shot `_dispatch_roadmap_exploration` spawns the Product Owner **solo** — deliberately NOT `_handle_board_assigned_task` (which would also spawn Head of Marketing and fire the Approve-&-Start handoff, both wrong for a PO-authored cycle) — reusing the `_board_dispatched` one-shot tracker + respawn breaker, and short-circuiting once the cycle is authored. The PO explores (read-only git, KB/RAG, metrics, releases, charter, optional web research) and makes ONE `propose_roadmap` call (a content verb gated to `product_owner` only, `_ROADMAP_ROLES`; wired through the do_server/Choreographer like `pitch`) authoring a **themed cycle** — a one-line goal + 3-7 item drafts — persisted as a `roadmap_cycle` marker on the exploration task (no table/migration). The CEO acts per-item in the panel roadmap queue (`roadmap-review-queue.tsx`; `/api/roadmap/cycles{,/items/{id}/approve,/items/{id}/reject}`, CEO-only): approve materializes that item as a BACKLOG task (`source="roadmap"`, no assignee — never auto-starts; normal PM activation picks it up) via `PrompterService.create_task_from_draft`, reject records a reason; when every item is terminal the exploration task completes (`RoadmapService`, idempotent per item). Dispatchers skip `board_roadmap` (never delivery work). `create_task_from_draft` honors a draft-declared `source` only from a `{prompter, roadmap}` whitelist — an LLM-authored draft can't impersonate a privileged origin.
|
||||
|
||||
**Obsidian vault V1 (default-off).** The org's human-readable memory palace as a rebuildable DB projection — tasks, journals, and A2A digests as wikilinked markdown — gated by `ROBOCO_OBSIDIAN_VAULT_ENABLED` + `ROBOCO_VAULT_PATH` (default off, but both compose files arm it `true`). `VaultWriter` (`roboco/services/vault_writer.py`) is a pure, DB-free materializer under `RoboCo/{Tasks/<project-slug>,Journals/<agent-slug>,A2A,Agents}/`; every note carries a stable `aliases: [<id8>]` so a title rename never breaks a `[[id8|title]]` cross-link, and private journals are excluded. Three best-effort event seams (`TaskService`'s status-transition frontmatter touch, `JournalService`, `A2AService`) patch/append on the relevant transition — a vault write failure never blocks the real action. `python -m roboco.vault rebuild` re-projects every entity from the DB (preserving an existing task's Auditor-authored `## Narrative`) and materializes the shipped `.obsidian/` config (Dataview, Kanban, graph groups) + `RoboCo/_meta/` dashboards from `roboco/vault_assets/`; `relocate <path>` moves the tree, grafting `RoboCo/` into an existing personal vault without touching its own config. The Auditor gets a one-shot `curate_vault(task_id, narrative)` do-tool, spawned by the orchestrator on each completed root task, writing the `## Narrative` section a deterministic write otherwise leaves as a placeholder. A second, independently-gated `ROBOCO_VAULT_INTAKE_ENABLED` watcher (`VaultIntakeEngine`) turns `#roboco`-tagged notes under the vault's inbox folder into PENDING, Product-Owner-assigned board-review drafts (`source=vault_note`) — the identical board-review path a chat-confirmed draft takes, never straight into delivery. Extraction runs on the local model with a deterministic fallback; the note body is screened through `foundation/policy/injection_guard.screen_external_text` (the same untrusted-content envelope `XEngine` applies to X mentions — flags an injection-pattern line inline, never removes content) before it reaches the prompt or the fallback. Deduped per `(vault-relative path, content hash)` via `vault_seen_notes`, so an edit re-qualifies a previously-seen note.
|
||||
**Obsidian vault V1+V2 (default-off).** The org's human-readable memory palace as a rebuildable DB projection — tasks, journals, and A2A digests as wikilinked markdown — gated by `ROBOCO_OBSIDIAN_VAULT_ENABLED` + `ROBOCO_VAULT_PATH` (default off, but both compose files arm it `true`). `VaultWriter` (`roboco/services/vault_writer.py`) is a pure, DB-free materializer under `RoboCo/{Tasks/<project-slug>,Journals/<agent-slug>,A2A,Agents,Archive/<year>/Tasks/<project-slug>,Reports}/`; every note carries a stable `aliases: [<id8>]` so a title rename (or an archival move) never breaks a `[[id8|title]]` cross-link, and private journals are excluded. Four best-effort event seams (`TaskService.create`'s materialize-on-create, `TaskService`'s status-transition frontmatter touch, `JournalService`, `A2AService`) patch/append on the relevant transition — a vault write failure never blocks the real action; materialize-on-create means a task's note exists from the moment it's created, not just at curation/rebuild. `python -m roboco.vault rebuild` re-projects every entity from the DB (preserving an existing task's Auditor-authored `## Narrative`, archive-aware so an old terminal task lands directly in `Archive/<year>/`) and materializes the shipped `.obsidian/` config (Dataview, Kanban, graph groups) + `RoboCo/_meta/` dashboards + `.base` Bases views from `roboco/vault_assets/`; `relocate <path>` moves the tree, grafting `RoboCo/` into an existing personal vault without touching its own config. The Auditor gets a one-shot `curate_vault(task_id, narrative)` do-tool, spawned by the orchestrator on each completed root task, writing the `## Narrative` section a deterministic write otherwise leaves as a placeholder. A second, independently-gated `ROBOCO_VAULT_INTAKE_ENABLED` watcher (`VaultIntakeEngine`) turns `#roboco`-tagged notes under the vault's inbox folder into PENDING, Product-Owner-assigned board-review drafts (`source=vault_note`) — the identical board-review path a chat-confirmed draft takes, never straight into delivery. Extraction runs on the local model with a deterministic fallback; the note body is screened through `foundation/policy/injection_guard.screen_external_text` (the same untrusted-content envelope `XEngine` applies to X mentions — flags an injection-pattern line inline, never removes content) before it reaches the prompt or the fallback. Deduped per `(vault-relative path, content hash)` via `vault_seen_notes`, so an edit re-qualifies a previously-seen note — the same hashing convention (every RoboCo feedback callout stripped first, `foundation/policy/vault_notes.py`) is now shared with the KB engine below. V2 adds three things on top: a **drift janitor** (`services/vault_janitor.py`, `_vault_janitor_loop`) hourly-ticked but gated by a `RoboCo/_meta/.janitor_state.json` state file so real work (a daily changed-task re-projection + random-sample drift check + archival pass, each capped at 200/cycle and per-item isolated so one bad row never wedges the sweep) and a weekly org-report (`vault_report_enabled`, default true — `VaultWriter.write_org_report` from `MetricsService`/`UsageService`, best-effort CEO notification) each fire exactly once per elapsed period regardless of loop/restart cadence; **archival** (`vault_archive_days`, default 30, `0`=off) moving old terminal tasks' notes into `RoboCo/Archive/<year>/Tasks/<project>/` during the sweep, alias links making the move free and the shipped Dataview/graph assets `Archive/`-aware; and **KB ingest** (`vault_kb_enabled`, default false — NAS compose arms it `true`, registry compose leaves it `false`) embedding the CEO's own `RoboCo/Notes/` (config `vault_kb_dirs`, csv, load-time-validated against traversal/overlap with reserved projection dirs) into a new `IndexType.VAULT_NOTES` corpus via `_vault_kb_loop` (`services/vault_kb_engine.py`, default 900s), with every note re-checked for symlink/path-escape at read time and screened through the injection guard as a hard GATE (a flagged note is quarantined — skipped, logged, callout-marked, never embedded) rather than the intake watcher's screen-and-still-process posture — reaching `roboco_kb_search`, `MentorService`'s default domain, and `EvidenceRepo.similar_memory` (claim-time briefings, relevance-floored, labeled `vault_note`) so the CEO's own writing finally becomes fleet-retrievable institutional memory.
|
||||
|
||||
**Fable-mode (default-off).** Full opus-fable-playbook adoption: makes the fleet behave more like Fable 5 on the existing model tiers (the tiers stay — Fable 5 the model is not an option). Two levers, both gated by `ROBOCO_FABLE_MODE_ENABLED`: ① **doctrine** — `fable_doctrine_layer()` (`roboco/agents/factories/_base.py`) composes the vendored behavioral doctrine (`agents/prompts/doctrine/fable.md`, from `github.com/rennf93/opus-fable-playbook` MIT `output-styles/fable.md`, YAML frontmatter stripped) into `compose_prompt`'s layer tuple immediately after `base.md` — universal cross-role doctrine, the same tier as the base rules, ahead of role/team/identity layers so those keep their specificity precedence. ② **hooks** — 5 vendored scripts under `docker/scripts/fable-*.sh` (stop-gate, bash-discipline, honesty-nudge, prompt-nudge, precompact; `session-start.sh` deliberately SKIPPED — its doctrine card is redundant with ① and its output-style check is inapplicable here) are installed alongside RoboCo's own hooks, never replacing them: `AgentOrchestrator._fable_hook_groups()` appends them AFTER RoboCo's own per-event entries in the Claude-path settings.json (isolated into its own helper to protect `_generate_agent_settings`'s xenon budget); the grok path installs only `honesty-nudge` (`write_grok_fable_hooks`, `roboco/llm/providers/grok_cli_config.py`) — a deliberately conservative V1 scope, since a grok `PreToolUse`/`Stop` hook deny cancels the entire run (verified live) while `PostToolUse` never denies. Off by default: the spawn path (composed prompt, settings.json, grok hooks) is byte-for-byte unchanged when the flag is off. No new eval harness — measurement rides the existing rework/spawn-waste/`revision_count` dashboard (see "Delivery observability" below). Armed on the NAS deploy like the rest; left OFF in `docker-compose.registry.yml`.
|
||||
|
||||
**Ponytail (bundled with Fable-mode).** Rides `ROBOCO_FABLE_MODE_ENABLED` — no separate flag. Vendors the ponytail "lazy senior dev" build-laziness doctrine (`agents/prompts/doctrine/ponytail.md` + ethos sibling, MIT, Copyright (c) 2026 DietrichGebert — trimmed, YAML frontmatter stripped) into every composed system prompt via `ponytail_doctrine_layer` (`roboco/agents/factories/_base.py`), slotted immediately after the Fable doctrine layer and gated on the same flag. Role-scoped: developers (`AgentRole.DEVELOPER`) get the full ladder (YAGNI → reuse-in-this-codebase → stdlib → native-platform → installed-dep → one-line → minimal); every other role gets the ethos-only cut (`ponytail-ethos.md`) — the code-mechanics rungs are dropped so they can't leak into prose artifacts (task plans, review notes, docs). Both files carry a 5-point RoboCo preamble (the ethos sibling adds a 6th: free-text field obligations) that makes the ladder yield to the Architectural Conventions Standard (placement), the 80% coverage gate + QA review + self-verification, the per-team design bar, task hygiene (everything-is-a-task / commits-linked / state-is-sacred), and reviewer feedback (`needs_revision` / `pr_fail` / `request_changes`) — the overlap mitigation is scoping, not deletion, and it rides ponytail's own "when NOT to be lazy" clause. Developer intensity is tunable via `ROBOCO_PONYTAIL_INTENSITY` (`lite` / `full` / `ultra`, default `full`; `roboco/config.py` `ponytail_intensity`, a string value — not a feature flag): `full` enforces the ladder, `lite` builds what's asked and names the lazier alternative, `ultra` is YAGNI-extremist (deletion before addition, challenge the requirement). Non-developers get no dial — `ultra` is wrong for prose artifacts, so the ethos runs a fixed restrained stance. Prompt-only: no hooks, no grok-path changes — ponytail adds no hook surface, so bundling it under the Fable flag changes only the composed prompt, not the spawn hooks. The Fable flag's description in `roboco/config.py` names both doctrines.
|
||||
|
||||
**Feature flags / company-in-a-box.** Env-gated, default-off subsystems toggle from the panel's Settings → Feature Flags card (`panel/src/components/settings/feature-flags-card.tsx`) instead of hand-editing env: web research (`ROBOCO_RESEARCH_ENABLED`), the strategy engine (`ROBOCO_STRATEGY_ENGINE_ENABLED`), pitch provisioning (`ROBOCO_PROVISIONING_*`), external / internal PR review, the agent-runtime toolchain match (`ROBOCO_TOOLCHAIN_MATCH_ENABLED`), the architectural-conventions standard (`ROBOCO_CONVENTIONS_ENABLED`), gateway-health recovery (`ROBOCO_GATEWAY_HEALTH_ENABLED`), multi-repo CI-watch (`ROBOCO_CI_WATCH_ENABLED`), the dependency-update bot (`ROBOCO_DEP_UPDATE_ENABLED`), the gated release manager (`ROBOCO_RELEASE_MANAGER_ENABLED`), the organizational memory loop (`ROBOCO_ORG_MEMORY_ENABLED`), the sandboxed dev DB/Redis (`ROBOCO_SANDBOX_DB_ENABLED`), the RoboCo X account (`ROBOCO_X_ENGINE_ENABLED`), the RoboCo video engine (`ROBOCO_VIDEO_ENGINE_ENABLED`), the board roadmap engine (`ROBOCO_ROADMAP_ENGINE_ENABLED`), Fable-mode (`ROBOCO_FABLE_MODE_ENABLED`), and the self-heal flags above. Cloud auth (`ROBOCO_CLOUD_AUTH_ENABLED`) is deliberately NOT on this card — like `ROBOCO_DB_NETWORK_ISOLATED`, it's a compose/env-coupled flag a runtime toggle can't safely flip mid-session. A toggle persists in the settings store and takes effect on the next backend restart; an unset flag falls back to its environment / config default.
|
||||
**Feature flags / company-in-a-box.** Env-gated, default-off subsystems toggle from the panel's Settings → Feature Flags card (`panel/src/components/settings/feature-flags-card.tsx`) instead of hand-editing env: web research (`ROBOCO_RESEARCH_ENABLED`), the strategy engine (`ROBOCO_STRATEGY_ENGINE_ENABLED`), pitch provisioning (`ROBOCO_PROVISIONING_*`), external / internal PR review, the agent-runtime toolchain match (`ROBOCO_TOOLCHAIN_MATCH_ENABLED`), the architectural-conventions standard (`ROBOCO_CONVENTIONS_ENABLED`), gateway-health recovery (`ROBOCO_GATEWAY_HEALTH_ENABLED`), multi-repo CI-watch (`ROBOCO_CI_WATCH_ENABLED`), the dependency-update bot (`ROBOCO_DEP_UPDATE_ENABLED`), the gated release manager (`ROBOCO_RELEASE_MANAGER_ENABLED`), the organizational memory loop (`ROBOCO_ORG_MEMORY_ENABLED`), the sandboxed dev DB/Redis (`ROBOCO_SANDBOX_DB_ENABLED`), the RoboCo X account (`ROBOCO_X_ENGINE_ENABLED`), the RoboCo video engine (`ROBOCO_VIDEO_ENGINE_ENABLED`), the board roadmap engine (`ROBOCO_ROADMAP_ENGINE_ENABLED`), Fable-mode (`ROBOCO_FABLE_MODE_ENABLED`), the vault weekly report + KB ingest (`ROBOCO_VAULT_REPORT_ENABLED` / `ROBOCO_VAULT_KB_ENABLED`), and the self-heal flags above. Cloud auth (`ROBOCO_CLOUD_AUTH_ENABLED`) is deliberately NOT on this card — like `ROBOCO_DB_NETWORK_ISOLATED`, it's a compose/env-coupled flag a runtime toggle can't safely flip mid-session. A toggle persists in the settings store and takes effect on the next backend restart; an unset flag falls back to its environment / config default.
|
||||
|
||||
## Architectural Conventions Standard
|
||||
|
||||
|
||||
Reference in New Issue
Block a user