mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
feat(vault): Obsidian vault V2 — janitor, archival, weekly report, KB ingest, Bases + sync runbook (#482)
* feat(vault): V2 — create-seam + drift janitor, archival, weekly org-report, KB ingest, Bases views + sync runbook Implements the vault V2 canonical spec end to end (the splice guard shipped separately and is reused at KB-ingest time): - materialize-on-create: TaskService.create writes each task's note best-effort from the moment it exists; the transition-touch stops no-oping on live work - drift janitor (services/vault_janitor.py + hourly _vault_janitor_loop): daily changed-task re-projection, random drift sample, archival pass — restart-proof via RoboCo/_meta/.janitor_state.json, 200/cycle caps, per-item isolation, processed-only resume markers, self-repairing state file - archival: vault_archive_days (30, 0=off) moves old terminal tasks' notes to RoboCo/Archive/<year>/Tasks/<project>/ — one write_task code path for janitor and rebuild, id8 lookup across Tasks/+Archive/, alias links keep moves safe - weekly org-report: VaultWriter.write_org_report renders Reports/<ISO-week>.md from MetricsService/UsageService (numbers duplicated into frontmatter for trend queries), once per ISO week, with a best-effort CEO notification - KB ingest: IndexType.VAULT_NOTES + VaultNotesIndexPlugin + _vault_kb_loop embed the CEO's RoboCo/Notes into the RAG corpus — injection guard as a hard gate (flagged notes quarantined with an idempotent callout), traversal- and symlink-contained at both config and engine layers, content-hash dedup, 50-ingest/cycle cap, frontmatter stripped; reaches roboco_kb_search, the mentor default domain, claim-time briefings (kind vault_note), and the panel KB browser; no migration (chunks table auto-creates; migration 030's CHUNK_TABLES tuple appended per the chunks_playbooks precedent) - Bases views (Task Board.base, Reports.base — schema verified against the Obsidian docs) + the Mac sync runbook vault asset - config/flags/compose: vault_archive_days, vault_report_enabled (flags card), vault_kb_enabled (flags card; NAS compose arms it, registry ships it off), vault_kb_dirs (+ overlap/traversal validator), vault_kb_interval_seconds - e2e smoke (tests/e2e_smoke/test_vault_v2.py): real create-seam, real janitor cycle incl. archival + state, real KB engine + real guard * docs: vault V2 sweep — map, RAG corpus, CLAUDE.md - docs/map/vault.md: V1+V2 — janitor/archival/report/KB data flows, new files, config, health posture - docs/map/orchestrator.md + task-service.md: the two new loops, the create seam, the three janitor queries - docs/rag/architecture/obsidian-vault.md: agent-facing what-changed (notes from creation, archive link-safety, CEO notes retrievable, weekly report) - docs/rag/architecture/config-reference.md: the five new settings - CLAUDE.md: vault paragraph covers V1+V2; flags-card list mentions the vault report/KB flags --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
@@ -92,6 +92,9 @@ The PR-gate turn cut (when every child of an assembled parent is terminal, `_try
|
||||
| `ROBOCO_ROADMAP_ENGINE_ENABLED` | `false` | The board roadmap engine: weekly Product-Owner-authored cycle, CEO approves each item individually into BACKLOG. See "Board Roadmap Engine" below. |
|
||||
| `ROBOCO_OBSIDIAN_VAULT_ENABLED` | `false` (both compose files set `true`) | The Obsidian vault projection: tasks/journals/A2A become wikilinked markdown, rebuildable from the DB. See `docs/rag/architecture/obsidian-vault.md`. |
|
||||
| `ROBOCO_VAULT_INTAKE_ENABLED` | `false` (both compose files set `true`) | The vault's `#roboco`-tag inbox watcher — requires `ROBOCO_OBSIDIAN_VAULT_ENABLED` also on. See `docs/rag/architecture/obsidian-vault.md`. |
|
||||
| `ROBOCO_VAULT_ARCHIVE_DAYS` | `30` (`0` disables) | Age (past its terminal timestamp) a completed/cancelled task's note must reach before the vault janitor moves it to `RoboCo/Archive/<year>/`. |
|
||||
| `ROBOCO_VAULT_REPORT_ENABLED` | `true` | The vault janitor's weekly `RoboCo/Reports/<ISO-week>.md` org-report note + CEO notification (deterministic, no LLM). Needs `ROBOCO_OBSIDIAN_VAULT_ENABLED` also on. |
|
||||
| `ROBOCO_VAULT_KB_ENABLED` | `false` (NAS compose sets `true`; registry compose leaves `false`) | Master switch for vault KB ingest: embeds `ROBOCO_VAULT_KB_DIRS` note folders (default `RoboCo/Notes`) into `IndexType.VAULT_NOTES`, screened for injection attempts before indexing. Requires `ROBOCO_OBSIDIAN_VAULT_ENABLED` also on. `ROBOCO_VAULT_KB_DIRS` (CSV, default `RoboCo/Notes`) and `ROBOCO_VAULT_KB_INTERVAL_SECONDS` (default `900`) tune scope and cadence. See `docs/rag/architecture/obsidian-vault.md`. |
|
||||
|
||||
The company-in-a-box subsystems toggle the same way: web research (`ROBOCO_RESEARCH_ENABLED`, default **on** — see "Web Research" below), the strategy engine (`ROBOCO_STRATEGY_ENGINE_ENABLED`, default off), and pitch provisioning (`ROBOCO_PROVISIONING_ENABLED`, default on but inert without a token/org configured).
|
||||
|
||||
|
||||
@@ -1,55 +1,78 @@
|
||||
# Obsidian Vault (V1)
|
||||
# Obsidian Vault
|
||||
|
||||
## What It Is
|
||||
|
||||
RoboCo can project its own state into a human-readable [Obsidian](https://obsidian.md) vault — tasks, journal entries, and A2A thread digests as wikilinked markdown notes — so the CEO can browse the org's memory with a normal notes app instead of the panel alone. It is a **rebuildable projection**, not a second source of truth: every note is derived from DB state and can always be regenerated from scratch. Implemented in `roboco/services/vault_writer.py` (pure markdown materializer), `roboco/services/vault_assembly.py` (DB → materializer dataclasses), `roboco/services/vault_intake_engine.py` (the inbox watcher), and `roboco/vault.py` (the `rebuild`/`relocate` CLI).
|
||||
RoboCo can project its own state into a human-readable [Obsidian](https://obsidian.md) vault — tasks, journal entries, and A2A thread digests as wikilinked markdown notes — so the CEO can browse the org's memory with a normal notes app instead of the panel alone. It is a **rebuildable projection**, not a second source of truth: every note is derived from DB state and can always be regenerated from scratch. Implemented in `roboco/services/vault_writer.py` (pure markdown materializer), `roboco/services/vault_assembly.py` (DB → materializer dataclasses), `roboco/services/vault_intake_engine.py` (the inbox watcher), `roboco/services/vault_janitor.py` (drift repair, archival, weekly report), `roboco/services/vault_kb_engine.py` (KB ingest), and `roboco/vault.py` (the `rebuild`/`relocate` CLI).
|
||||
|
||||
## Enable/Disable
|
||||
|
||||
| Variable | Default | Effect |
|
||||
|----------|---------|--------|
|
||||
| `ROBOCO_OBSIDIAN_VAULT_ENABLED` | `false` (both compose files set `true`) | Master switch. Off = no note is ever written, `curate_vault` and `python -m roboco.vault` both refuse. |
|
||||
| `ROBOCO_OBSIDIAN_VAULT_ENABLED` | `false` (both compose files set `true`) | Master switch. Off = no note is ever written, `curate_vault` and `python -m roboco.vault` both refuse, and neither the janitor nor the KB engine run. |
|
||||
| `ROBOCO_VAULT_PATH` | `/data/vault` | Root directory the vault materializes into. |
|
||||
| `ROBOCO_VAULT_INTAKE_ENABLED` | `false` (both compose files set `true`) | The `#roboco`-tag inbox watcher (see below) — requires the master switch also on. |
|
||||
| `ROBOCO_VAULT_ARCHIVE_DAYS` | `30` (`0` disables) | Age past which a completed/cancelled task's note moves into the vault's archive during the janitor's daily sweep. |
|
||||
| `ROBOCO_VAULT_REPORT_ENABLED` | `true` | The janitor's weekly org-report note + CEO notification. |
|
||||
| `ROBOCO_VAULT_KB_ENABLED` | `false` (NAS compose sets `true`; registry compose leaves `false`) | KB ingest of the CEO's own `RoboCo/Notes/` into `IndexType.VAULT_NOTES` — see below. Requires the master switch also on. |
|
||||
|
||||
## What agents actually touch
|
||||
|
||||
Almost everything here is transparent to a working agent — notes get written or patched as a side effect of normal verbs (a status transition, a `note()` call, a `dm()`), never something you call yourself. There are exactly two places an agent interacts with the vault directly:
|
||||
Almost everything here is transparent to a working agent — notes get written or patched as a side effect of normal verbs (a status transition, a `note()` call, a `dm()`), never something you call yourself. What changed for you since V1:
|
||||
|
||||
1. **The Auditor's `curate_vault(task_id, narrative)`** — spawned once per completed root task to write the one piece of vault content that isn't mechanically derivable from DB columns: a narrative paragraph on the task's note. See `docs/rag/roles/auditor.md`.
|
||||
2. **The vault-intake watcher turning a CEO-authored note into a task you might get delegated.** If the CEO tags a note `#roboco` in the vault's inbox folder, a default-off watcher (`ROBOCO_VAULT_INTAKE_ENABLED`) drafts it into a board-review task (`source=vault_note`) — same shape as a chat-confirmed intake draft, Product-Owner-assigned, `team=board`. It never starts work directly: the board reviews it and only the CEO's `approve_and_start` hands it to the Main PM for real delegation. If you end up working a task with `source=vault_note`, its origin was a note the CEO wrote in their own vault, not a chat.
|
||||
1. **Your task has a note from the moment it's created**, not just once curated or rebuilt — `TaskService.create` now materializes it directly. You'll never see a task the CEO can look up in the vault yet you can't.
|
||||
2. **A task that gets archived (old + terminal) doesn't lose its links.** The vault moves an old completed/cancelled task's note into `RoboCo/Archive/<year>/Tasks/<project>/`, but every wikilink to it (`[[id8|title]]`) is alias-based, so nothing pointing at it breaks — you'd never notice unless you went looking at the raw file path.
|
||||
3. **If `vault_kb_enabled` is on, the CEO's own vault notes are retrievable by you.** Anything the CEO writes under `RoboCo/Notes/` (screened for injection attempts first) is embedded into the knowledge base like any other corpus — it shows up in `roboco_kb_search` / `roboco_ask_mentor` results and in your claim-time institutional-memory briefing, labeled `vault_note` (or "Vault Notes" in mentor output). Treat it exactly like a learning or a playbook: institutional memory, not a directive to follow blindly.
|
||||
4. **A weekly org-report note exists** (`RoboCo/Reports/<ISO-week>.md`) — velocity, cycle time by stage, bottlenecks, rework, cost. Deterministic (no LLM), for the CEO's browsing; not something you're expected to author or reference.
|
||||
5. **The Auditor's `curate_vault(task_id, narrative)`** — unchanged: spawned once per completed root task to write the one piece of vault content that isn't mechanically derivable from DB columns, a narrative paragraph. See `docs/rag/roles/auditor.md`.
|
||||
6. **The vault-intake watcher turning a CEO-authored note into a task you might get delegated.** If the CEO tags a note `#roboco` in the vault's inbox folder, a default-off watcher (`ROBOCO_VAULT_INTAKE_ENABLED`) drafts it into a board-review task (`source=vault_note`) — same shape as a chat-confirmed intake draft, Product-Owner-assigned, `team=board`. It never starts work directly: the board reviews it and only the CEO's `approve_and_start` hands it to the Main PM for real delegation.
|
||||
|
||||
Everything else — note layout, link stability, the rebuild CLI — is infrastructure you don't need to reason about to do your job; the rest of this doc is here for completeness, not because you'll call any of it.
|
||||
Everything else — note layout, link stability, the janitor's internals, the rebuild CLI — is infrastructure you don't need to reason about to do your job; the rest of this doc is here for completeness, not because you'll call any of it.
|
||||
|
||||
## Layout and link stability
|
||||
|
||||
```
|
||||
RoboCo/
|
||||
Tasks/<project-slug>/<title> (<id8>).md
|
||||
Archive/<year>/Tasks/<project-slug>/<title> (<id8>).md (old terminal tasks)
|
||||
Journals/<agent-slug>/<date> <title> (<id8>).md
|
||||
A2A/<date> <agents> (<thread-id8>).md
|
||||
Agents/<slug>.md
|
||||
_meta/ (shipped Dataview/Kanban/graph-group dashboards)
|
||||
Reports/<ISO-week>.md (weekly org-report)
|
||||
Notes/ (CEO's own writing — KB ingest scope)
|
||||
_meta/ (shipped Dataview/Kanban/graph-group dashboards + Bases views)
|
||||
```
|
||||
|
||||
Every note carries `aliases: [<id8>]` in its frontmatter, so a cross-link is always written as `[[<id8>|<title>]]` — Obsidian resolves it by alias regardless of the target's current filename. A task title edit updates the note's own title line without ever renaming the file or breaking a link elsewhere in the vault. Private journal entries (`is_private`) are excluded from the projection, same as the shared RAG corpus.
|
||||
Every note carries `aliases: [<id8>]` in its frontmatter, so a cross-link is always written as `[[<id8>|<title>]]` — Obsidian resolves it by alias regardless of the target's current filename or folder. A task title edit updates the note's own title line without ever renaming the file or breaking a link elsewhere in the vault; an archival move works the same way — it relocates the file, not the identity agents/links reference. Private journal entries (`is_private`) are excluded from the projection, same as the shared RAG corpus, and stay excluded from KB ingest scope too (KB ingest only ever covers `RoboCo/Notes/`, never `Journals`/`Tasks`/`A2A`/`Agents`/`Archive`/`Reports`).
|
||||
|
||||
## What triggers a write
|
||||
|
||||
Best-effort event seams patch or append on the relevant transition — never a gate, never something that can block the real action:
|
||||
|
||||
- A task status transition patches the note's frontmatter (status/team/PR) in place, if the note already exists.
|
||||
- **Task creation** materializes the note immediately (deterministic template; the narrative is a placeholder until the Auditor curates it).
|
||||
- A task status transition patches the note's frontmatter (status/team/PR) in place.
|
||||
- A `note()` (journal entry) writes one immutable file per entry.
|
||||
- A `dm()` (A2A message) appends to a per-thread digest file.
|
||||
- An hourly-ticked, daily-gated janitor sweep catches anything the above seams missed: re-projects tasks changed since the last sweep, verifies a random sample of older ones, and archives old terminal tasks. This is a backstop, not something your own verb calls need to think about.
|
||||
|
||||
None of these fail your call if the vault write itself fails — it's logged and swallowed.
|
||||
|
||||
## KB ingest — the CEO's notes become retrievable
|
||||
|
||||
When `vault_kb_enabled` is on (alongside the master switch), notes the CEO writes under `RoboCo/Notes/` are embedded into a dedicated `IndexType.VAULT_NOTES` corpus on a ~15-minute cycle. Before anything is indexed, the note body is screened for prompt-injection patterns — a flagged note is quarantined (never embedded, a warning callout is appended to it so the CEO sees why) rather than silently indexed. A clean note is retrievable through:
|
||||
|
||||
- `roboco_kb_search` (pass no filter, or filter to `vault_notes` explicitly)
|
||||
- `roboco_ask_mentor` — company/general domain queries include it, formatted as "Vault Notes"
|
||||
- Your claim-time institutional-memory briefing, alongside distilled learnings and approved playbooks, subject to the same relevance floor (nothing injected below it)
|
||||
|
||||
Content you retrieve this way is the CEO's own writing — background, decisions, preferences — not a verified fact or a directive. Weigh it the same way you'd weigh a learning: useful context, not gospel.
|
||||
|
||||
## Rebuild and relocate
|
||||
|
||||
`python -m roboco.vault rebuild` re-projects every live entity from the DB from scratch (preserving any existing task's Auditor-authored narrative, since that isn't derivable) and materializes the shipped `.obsidian/` config + `_meta/` dashboards if not already present. `python -m roboco.vault relocate <path>` moves the tree — useful for grafting `RoboCo/` into an existing personal vault without touching that vault's own config. Both are operator/CEO commands, not something an agent runs.
|
||||
`python -m roboco.vault rebuild` re-projects every live entity from the DB from scratch (preserving any existing task's Auditor-authored narrative, archive-aware so an old terminal task lands directly in `Archive/<year>/`) and materializes the shipped `.obsidian/` config + `_meta/` dashboards/Bases views if not already present. `python -m roboco.vault relocate <path>` moves the tree. Both are operator/CEO commands, not something an agent runs.
|
||||
|
||||
## Related
|
||||
|
||||
- `docs/rag/roles/auditor.md` — the `curate_vault` verb
|
||||
- `docs/rag/architecture/config-reference.md` — full env var table
|
||||
- `docs/rag/architecture/x-engine.md` — the sibling engine whose injection-screening guard (`screen_external_text`) the vault-intake watcher shares
|
||||
- `docs/rag/architecture/x-engine.md` — the sibling engine whose injection-screening guard (`screen_external_text`) both the vault-intake watcher and the KB engine share
|
||||
- `docs/map/vault.md` — the code-level map (files, data flow, config, health)
|
||||
|
||||
Reference in New Issue
Block a user