External/roboco
1
0
Fork 0
mirror of https://github.com/rennf93/roboco.git synced 2026-08-03 07:23:24 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
a80790499925da55d40917afc4dc85107c44382b
roboco/docs/map/db-migrations.md
T
Renzo FGitHubRenn F
 3ccc723cd4 v0.17.0 — Wave 3: sandbox DB, DB isolation, mobile UI, cloud auth, X account, roadmap engine (#303) 
* feat(sandbox): throwaway per-agent Postgres/Redis sandbox containers

Orchestrator-provisioned sibling containers per agent spawn
(SandboxProvisioner, roboco/runtime/sandbox.py). Per-project opt-in via
projects.sandbox_services (migration 057); master switch
ROBOCO_SANDBOX_DB_ENABLED, default-off, armed in the NAS compose only.

When active, ROBOCO_TEST_DB_* / ROBOCO_TEST_REDIS_* point at the sandbox
and the prod-creds gate-env injection is suppressed (sandbox replaces,
never coexists). Sandbox lifetime tracks the agent container: teardown at
every removal path, orphan janitor at startup + each reaper tick with a
grace window for mid-flight spawns. The pre-spawn stale-clear spares the
just-provisioned sandbox; provision pre-clears stale same-named
containers from a crash-missed teardown.

Panel: per-project sandbox-service switches in the edit dialog + feature
flag card entry.

* docs: CLAUDE.md entry for the sandboxed dev DB/Redis subsystem

* feat(security): isolate prod Postgres/Redis from agent containers (roboco_data network)

Second user-defined bridge roboco_data carries postgres+redis only; the
orchestrator is multi-homed (default + data). Spawned agents and their
sandbox sidecars stay on roboco_default and can no longer resolve or
reach roboco-postgres:5432 / roboco-redis:6379 (redis has no auth —
membership is its only containment). Normal bridge, so host-published
ports (15432/16379) keep working. Applied to both build composes and
the registry compose; docker-compose.yml re-synced byte-identical with
docker-compose.yaml (it had drifted by the sandbox flag block).

ROBOCO_DB_NETWORK_ISOLATED (config default false, armed alongside the
topology) suppresses the legacy _append_gate_env prod-creds injection:
under isolation those creds dead-end, and unreachable creds are worse
than none. DB-needing projects opt into sandbox_services instead. The
flag is deliberately not a panel feature flag - it must travel with the
compose networks: stanzas.

Preserved by construction: agent<->agent A2A and orchestrator->agent SDK
polls on :9000, MCP->orchestrator on :8000, ollama reachability, docker
exec/inspect (daemon socket), host port publishing.

* feat(panel): full mobile responsiveness pass

Shared primitives: useIsMobile (useSyncExternalStore, hydration-safe,
memoized matchMedia subscribe), ResponsiveTable table->card switch below
md (single subtree mounted, no duplicated interactive rows), scrollable
snap TabsList in the base primitive (justify-center-safe so the first
tab stays reachable on overflow), persistent md:hidden bottom tab bar
(Overview/Tasks/Kanban/Chat, safe-area padded).

Applied: card lists for tasks/projects/products/work-sessions/sessions
+ the three raw metrics tables; CEO approval queue / release proposal /
playbook review action rows stack on narrow; command-center reorders
approvals above the fold on mobile; task-header metadata wraps;
Communications + A2A become URL-driven single-pane drill-downs below lg
(fixes the unconstrained-height ScrollArea bug) with dvh heights;
recharts label density/radius adapts via useIsMobile; git diff viewer
gets mobile font + wrap toggle; vh->dvh sweep; chat composers get
safe-area-inset padding; dashboard main p-4 md:p-6 + pb-20 for the bar.

Verified at 375px on the built app: bottom bar, drawer, approval-first
overview, swipeable kanban tab strip. All gates green (eslint, tsc,
vitest 249, next build 24/24 routes).

* feat(auth): cloud auth via FastAPI Users (default-off, single-user cookie session)

ROBOCO_CLOUD_AUTH_ENABLED (default off) lets the panel/API be exposed
beyond localhost without changing the CEO's local no-login flow while
off — get_agent_context and the WS gate are byte-for-byte unchanged in
off-mode. On: header-trust dies for humans — any agent-role claim (ceo
or a privileged PM/board role) with no valid HMAC token or session
cookie is 401, closing the header-spoof hole on the host-published
:8000 port for every role. The agent-fleet HMAC path and the system
self-PATCH keep working unmodified in both modes.

Single seeded CEO user (migration 058 users table, UserTable), no
registration router — idempotent env-driven upsert at startup by PK.
Cookie transport (httponly/secure/samesite=lax) + a JWTStrategy bound
to a fingerprint of the current password hash (rotating the password
invalidates every prior session). Sliding 30-day session: every
authenticated request re-mints the cookie, so an active session never
expires — no unexpected logouts.

Panel: (auth)/login page + proxy.ts (Next 16 rename of middleware; probes
/auth/status over the docker-internal URL, fails open to off) gate the
dashboard; client.ts gets withCredentials + 401->/login. nginx unchanged.

Review hardening: broadened the on-mode rejection from ceo-only to every
non-CEO role without a valid token (was only closed when
ROBOCO_AGENT_AUTH_REQUIRED was also armed); Next-16 proxy.ts rename to
clear the middleware deprecation warning.

* feat(x): RoboCo X account engine — HoM drafts, per-post CEO approval (default-off)

ROBOCO_X_ENGINE_ENABLED (default off, inert without creds). Mirrors the
ReleaseManagerEngine held-artifact shape: XEngine drafts a post when a
release publishes (via a draft_release_post seam on ReleaseProposalService
.approve) and drafts replies to meaningful mentions (dedicated poll loop,
x_seen_mentions dedup ledger, per-cycle/open caps). Drafting is
local-model-only, clamped to 280 chars. Nothing auto-posts — every tweet
is a held task (source x_post/x_reply, confirmed_by_human=False,
Secretary-owned, dispatcher-skipped) the CEO edits/approves/rejects in a
panel queue.

The four OAuth 1.0a secrets live Fernet-encrypted in a singleton
x_credentials row (migration 059, all-or-nothing, API returns only
has_credentials); decryption is server-side, agents never hold creds or
egress. Hand-rolled OAuth 1.0a HMAC-SHA1 signer, no new dependency;
NullXClient makes the unconfigured path a graceful no-op.

XPostService.approve (CEO-only) is the sole caller of post_tweet.

Review hardening: closed a double-post race — the approve path now
re-reads committed task state inside the Redis lock and commits COMPLETED
before releasing, so a concurrent approve that acquires the lock after the
winner released can't re-post (SET-NX is non-waiting, and the route-level
commit landed after the lock dropped). Added a regression test.

* feat(roadmap): board roadmap engine — PO proposes themed cycles, CEO approves per-item (default-off)

ROBOCO_ROADMAP_ENGINE_ENABLED (default off). Weekly, RoadmapEngine opens
ONE held exploration task (source=board_roadmap, confirmed_by_human=False,
Product-Owner-assigned), deduped to one open cycle. A dedicated one-shot
_dispatch_roadmap_exploration spawns the PO solo (not the two-reviewer
board path, which would also spawn HoM + fire Approve-&-Start). The PO
explores read-only (git/KB/metrics/releases/charter/web) and makes one
propose_roadmap call (PO-only content verb) authoring a themed cycle —
goal + 3-7 item drafts — persisted as a roadmap_cycle marker (no table,
no migration; head stays 059).

The CEO acts per-item in the panel roadmap queue: approve materializes a
BACKLOG task (source=roadmap, no assignee — never auto-starts), reject
records a reason; all-items-terminal completes the exploration task.
RoadmapService is idempotent per item. Dispatchers skip board_roadmap.

Includes a real SQLAlchemy dirty-check fix (deep-copy the JSON marker
before mutating, or the in-place edit + reassign compares equal to its
own baseline and the UPDATE is skipped).

Review hardening: create_task_from_draft now honors a draft-declared
source only from a {prompter, roadmap} whitelist — drafts are
LLM-authored, so an unbounded source could impersonate a privileged
origin (release_manager would even wedge that engine's dedup).

* chore(release): 0.17.0

Wave 3 — six default-off subsystems: sandboxed dev DB/Redis, prod
Postgres/Redis network isolation, full mobile UI pass, cloud auth
(FastAPI Users), the RoboCo X account engine, and the board roadmap
engine. Plus the waves 1+2 work already on master since 0.16.0.

Version bumped across the canonical set (config.py, __init__.py,
pyproject.toml, panel/package.json, uv.lock); CHANGELOG [Unreleased]
cut to [0.17.0]; docs/map delta added.

Compose: every optional feature armed :-true in the NAS composes, OFF
in the user-facing registry compose. Two opt-in exceptions default off
(CLOUD_AUTH — needs email/password/secret + TLS, would otherwise fail
startup; ROUTING_STRICT — fail-closed spawning). DB_NETWORK_ISOLATED
stays on in both (coupled to the roboco_data topology).

* chore(compose): arm cloud_auth + routing_strict ON in the NAS composes

Every feature defaults ON in the NAS composes per policy — these two
were wrongly left off. Both keep the ${VAR:-true} form so the operator
controls the real runtime via .env: cloud auth needs
ROBOCO_CLOUD_AUTH_EMAIL/_PASSWORD/_SECRET + TLS set there before a boot
(else startup fails loud), and routing_strict is fail-closed. Registry
compose keeps both off.

* fix(ci): reflow board.md prose (quality gate) + document v0.17.0 env creds

The roadmap section added hard-wrapped prose that failed the markdown
prose gate; reflowed (token-invariant). Also brought .env.example
current: cloud auth (now armed — needs SECRET or startup fails), routing
strict, the X engine (panel-entered OAuth), and web research.

* fix(ci): reduce cyclomatic complexity of five wave-3 blocks (xenon gate)

The wave-3 subagents introduced C-rank functions the CI xenon gate
rejects (my per-item reviews ran ruff/mypy/pytest but not xenon):
- sandbox.janitor_sweep -> extract _list_labeled_sandboxes /
  _list_live_agent_containers / _prune_grace
- x_client.fetch_mentions -> extract _parse_mention_items
- x_engine.run_cycle -> extract _process_mentions
- orchestrator._dispatch_pm_work -> extract the source-skip into a
  MODULE-level _is_held_ceo_source (module, not method, so the
  wholesale-mocked dispatcher unit tests exercise the real logic)
- auth/seed.ensure_seed_user -> extract _apply_seed_updates (module avg -> A)

Behavior-preserving; full suite green (11902), xenon clean.

* fix(ci): declare pyjwt + fastapi-users-db-sqlalchemy as direct deps (deptry)

The cloud-auth code imports jwt and fastapi_users_db_sqlalchemy directly
but they were only transitive deps (via fastapi-users), which deptry
(quality gate, DEP003) rejects. Declared explicitly; deptry roboco/ clean.
Missed originally because local make quality stopped at earlier gates
before reaching deptry.

* feat(x): gate mention replies behind ROBOCO_X_REPLIES_ENABLED (default off)

Per CEO decision: the X engine should only post about releases by
default. Reading mentions needs a paid X API tier, so the mention-reply
half is now a deliberate opt-in on top of release posting.

New default-off flag x_replies_enabled gates the mentions poll loop
(_x_mentions_poll_loop) and XEngine.run_cycle; release-post drafting
(the release-proposal approve hook) is unaffected and still runs when
x_engine_enabled + credentials are set. Added to FEATURE_FLAGS + the
panel card. Tests: release posting works with replies off; run_cycle +
the poll loop are no-ops with replies off.

* fix: 401 only redirects to /login when cloud auth is on; panel-token strips .env quotes

Two bugs that together dead-ended login in secure mode:
- client.ts redirected to /login on ANY 401, so a mismatched panel
  token (header-trust/secure mode, cloud auth off) bounced the user to a
  login page whose backend route isn't mounted -> 404. Now it probes
  /auth/status (bare fetch, no interceptor re-entry) and only redirects
  when cloud_auth_enabled.
- make panel-token read the .env secret with grep|cut without stripping
  surrounding quotes, so a quoted ROBOCO_AGENT_AUTH_SECRET produced a
  token signed with the quotes included — which never verifies against
  the orchestrator (docker-compose/pydantic unquote the secret). Now
  strips surrounding single/double quotes.

* fix: git-log 500 on '|' in commit message; X queue shows an empty state

- GET /api/git/log 500'd (ValueError: Invalid isoformat) when a commit
  SUBJECT contained a '|' (e.g. the 'curl|sh' lockdown commit): the
  fixed '|' field delimiter let the subject's pipe shift the split so
  author+date collapsed into one field. Switched to \x1f (Unit
  Separator), which can't appear in commit content. Regression test with
  a piped subject.
- The X Post Queue returned null when empty, so there was no visible
  place for the X drafts. It now renders a discoverable empty state
  pointing at Settings -> X credentials.

* docs: bring docs/rag + docs/map current for v0.17.0 (waves 1-3)

Agent-facing RAG corpus and codebase map updated for every feature in
the 0.17.0 span, code-verified:
- wave 3: sandbox DB, DB network isolation, cloud auth, X engine
  (+ x_replies_enabled sub-flag), board roadmap engine — new RAG
  architecture pages + role/tool/config-reference updates; new symbols,
  migrations 057-059, panel surfaces, and the get_agent_context
  dual-path across the map slices.
- waves 1-2: A2A live view + switchboard, prompter memory
  (search_past_tasks), Secretary edit access + PM-lighter scope, the
  PR-gate auto-submit turn cut (ROBOCO_PR_GATE_AUTO_SUBMIT_ENABLED).
- correctness fix: api-routes-schemas.md no longer claims the A2A admin
  routes are reachable by any authenticated agent — they carry a
  _require_ceo gate (wave 2c).

docs/internal, _front.md deltas, and the frozen _complete_map.md
snapshot untouched.

* fix(rag): atomic upsert for indexed-doc tracking (kills e2e segfault)

The indexed-document tracking write used check-then-insert in two paths
(IndexedDocumentRepository.upsert_batch and the file-source
_upsert_doc_record). Under concurrent indexing both callers saw no row
and both inserted, so the second violated uq_indexed_doc_source and
poisoned its transaction — surfacing in CI as the intermittent
_checkin_failed SIGSEGV on the failed connection's pool checkin.

Both paths now use INSERT ... ON CONFLICT DO UPDATE against the
constraint: coalesce keeps an existing title/preview when the new value
is empty (matching the old guards) and metadata is jsonb-merged. The
batch dedupes within itself first (ON CONFLICT can't touch a row twice
in one statement). expire_all after the Core upsert keeps same-session
ORM reads consistent with the merged DB row.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-07-03 19:24:00 +02:00

280 lines
22 KiB
Markdown
Raw Blame History

# db-migrations slice
## Purpose
The DB layer is async SQLAlchemy 2.0 over PostgreSQL+asyncpg, with pgvector for the in-house RAG engine. Schema evolution is owned by an Alembic chain (001→059) that runs on every boot via `init_db()`; `Base.metadata.create_all` is no longer the source of truth — migration 017 reconciled the drift the other way. The ORM tables live in one fat module `roboco/db/tables.py` (~2.5k lines, 37 tables).
## Files
| Path | Role |
|------|------|
| `roboco/db/__init__.py` | Re-exports `Base`, session helpers, `bootstrap_database`, table classes. |
| `roboco/db/base.py` | `Base` (DeclarativeBase + naming convention), `get_engine`, `get_session_factory`, `get_db` (FastAPI dep), `get_db_context`, `run_migrations`, `init_db`, `_db_has_*` probes. Stamps a pre-Alembic DB at 001 then upgrades head. |
| `roboco/db/tables.py` | All 37 ORM table classes (single module). |
| `roboco/db/seed.py` | `bootstrap_database()` — runs `init_db` then seeds agents, channels, groups, initial messages. |
| `alembic/env.py` | Async Alembic env; imports `roboco.db.tables` to register metadata, overrides `sqlalchemy.url` from settings, `compare_type` + `compare_server_default` on. |
| `alembic.ini` | Standard config; `script_location=alembic`, `prepend_sys_path=.`, no URL (set in env.py). |
| `alembic/versions/` | 59 migration files 001..059 (two share number 026 — chained, not a collision). |
## Key Symbols
| Name | Kind | File:Line | Responsibility |
|------|------|-----------|----------------|
| `Base` | class | db/base.py:38 | DeclarativeBase + MetaData naming convention. |
| `get_engine` | fn | db/base.py:46 | Lazy singleton async engine (pool_pre_ping). |
| `get_db` | fn | db/base.py:70 | FastAPI async session dependency. |
| `get_db_context` | fn | db/base.py | Out-of-request async session context. |
| `init_db` | fn | db/base.py:180 | Boot entry: stamp pre-Alembic DB at 001 then `run_migrations`. |
| `run_migrations` | fn | db/base.py:141 | Runs `alembic upgrade head` via `command.upgrade` in a thread. |
| `bootstrap_database` | fn | db/seed.py:282 | `init_db` + seed default agents/channels/groups/messages. |
| `TaskTable` | class | tables.py:157 | Core task entity (largest table, drives lifecycle). |
| `WorkSessionTable` | class | tables.py:798 | Per-claim session; single-active enforced by 047 partial-unique index. |
| `AgentTable` | class | tables.py:95 | Agent identity, role, team, model provider assignment. |
| `ProjectTable` | class | tables.py:475 | Git repo config + CI/watch/dep-update/quality_command/`sandbox_services` (057) cols. |
| `AuditLogTable` | class | tables.py:1940 | Transition journey; `details` JSONB (010); composite query index (045). |
| `AgentSpawnSessionTable` | class | tables.py:2170 | Per-spawn token totals; feeds usage dashboard. |
| `ProjectConventionsCacheTable` | class | tables.py:2442 | Effective conventions map per (project, HEAD sha). |
| `PlaybookTable` | class | tables.py:721 | Curated procedures (draft→approved→indexed). |
| `RespawnTrackerTable` | class | tables.py:1902 | Durable PM-respawn circuit breaker mirror. |
| `TaskCellProjectTable` | class | tables.py:632 | Per-cell project map for a MegaTask root-subtask (052). |
| `WaitingRecordTable` | class | tables.py:1872 | Persisted dispatcher waiting records (restore at start). |
| `IndexedDocumentTable` | class | tables.py:1651 | RAG corpus docs (added to chain by 017). |
| `UserTable` | class | tables.py:2603 | Cloud-auth (FastAPI Users) single seeded CEO login row (058). |
| `XCredentialsTable` | class | tables.py:2650 | Singleton Fernet-encrypted OAuth 1.0a secrets for the X engine (059). |
| `XSeenMentionTable` | class | tables.py:2675 | X mentions-poll dedup ledger, keyed by mention id (059). |
| `run_async_migrations` | fn | env.py | Async online migration runner (NullPool). |
## Migration Chain
| Num | File | What it adds/changes |
|-----|------|---------------------|
| 001 | 001_initial_schema.py | All initial tables (agents, tasks, work_sessions, channels, sessions, messages, notifications, journals, audit_log, a2a_*). |
| 002 | 002_persistence_tables.py | Persistence tables + `NotificationType.APPROVAL`. |
| 003 | 003_blocker_resolver_type.py | `tasks.blocker_resolver_type` + `blockerresolvertype` enum. |
| 004 | 004_provider_routing.py | `provider_configs` + `model_assignments`; `modelprovider`/`assignmentscope` enums (create_type=False). |
| 005 | 005_blocker_raised_by.py | `tasks.blocker_raised_by`. |
| 006 | 006_gateway_columns.py | Gateway cols: claimant lock, heartbeat, pre-block snapshot, AC status, qa evidence flag. |
| 007 | 007_gateway_triggers_table.py | `gateway_triggers` (dispatcher decision log). |
| 008 | 008_align_skills.py | No-op (skills alignment done statically). |
| 009 | 009_enum_reconcile.py | Reconcile every postgres enum with ORM StrEnum (lowercase) + new members. |
| 010 | 010_audit_log_details_jsonb.py | `audit_log.details` JSON→JSONB. |
| 011 | 011_drop_quarantined_state.py | Drop `quarantined` from taskstatus enum (phantom state, audit D15). |
| 012 | 012_align_agentrole_foundation.py | Add agentrole/team enum values foundation declares. |
| 013 | 013_drop_role_enum.py | Drop stray `role` enum (smoke run 2). |
| 014 | 014_drop_pm_approvals.py | Drop unused `tasks.pm_approvals`. |
| 015 | 015_drop_task_execution_outputs.py | Drop unused `execution_log`/`outputs`. |
| 016 | 016_add_products_and_task_product_id.py | `products` + `product_projects`; `tasks.product_id` (team enum create_type=False). |
| 017 | 017_reconcile_orm_schema_drift.py | Add ORM tables/columns the chain never had (e.g. `indexed_documents`). |
| 018 | 018_task_project_id_nullable.py | `tasks.project_id` nullable (board/fan-out tasks carry product_id). |
| 019 | 019_seed_default_providers.py | Idempotent seed of default model providers. |
| 020 | 020_backfill_enum_values.py | Backfill ORM enum values the chain never added. |
| 021 | 021_task_board_review_complete.py | `tasks.board_review_complete` (board-review handoff flag). |
| 022 | 022_default_branch_master.py | Flip `projects.default_branch` default `main`→`master`. |
| 023 | 023_prompter_tracking_columns.py | `tasks.source` + `confirmed_by_human` (prompter origin). |
| 024 | 024_add_prompter_tables.py | `prompter_sessions`, `prompter_messages`, `task_drafts`. |
| 025 | 025_agentrole_prompter.py | Add `prompter` to agentrole enum. |
| 026a | 026_completed_dependency_ids.py | `tasks.completed_dependency_ids`. |
| 026b | 026_token_usage_tables.py | `agent_spawn_sessions` + `token_usage_snapshots` (chained off 026a). |
| 027 | 027_system_settings.py | `system_settings` key-value store. |
| 028 | 028_seed_self_hosted_provider.py | Seed Self-Hosted (Ollama LOCAL) provider row. |
| 029 | 029_project_quality_command.py | `projects.quality_command` (fast pre-submit gate). |
| 030 | 030_rag_chunks_content_schema.py | Align RAG chunk tables with vector-store schema. |
| 031 | 031_rag_chunks_fulltext.py | tsvector + GIN index on every chunk table (hybrid retrieval). |
| 032 | 032_company_goals.py | `company_goals` singleton charter. |
| 033 | 033_pitches.py | `pitches` (Board proposals → auto-provision). |
| 034 | 034_agentrole_secretary.py | Add `secretary` to agentrole enum. |
| 035 | 035_secretary_directives.py | `secretary_directives` (command audit + gate queue). |
| 036 | 036_ac_ids_and_parent_refs.py | Per-criterion AC ids + child→parent AC linkage. |
| 037 | 037_agentrole_pr_reviewer.py | Add `pr_reviewer` to agentrole enum. |
| 038 | 038_modelprovider_grok.py | Add `grok` to modelprovider enum. |
| 039 | 039_seed_grok_provider.py | Seed Grok (xAI) provider row. |
| 040 | 040_awaiting_pr_review.py | Add `awaiting_pr_review` to taskstatus enum (PR-review gate). |
| 041 | 041_structured_content_columns.py | `pr_reviewer_notes`, machine-marker split, structured content cols. |
| 042 | 042_worksession_toolchain.py | `work_sessions` toolchain matching cols. |
| 043 | 043_conventions_cache.py | `project_conventions_cache`. |
| 044 | 044_convention_findings.py | `project_convention_findings` (violations feed). |
| 045 | 045_observability_rework.py | `tasks.revision_count` + audit_log composite query index. |
| 046 | 046_batch_intake.py | `tasks.batch_id` + collision descriptors (intends_to_touch, adds_migration, touches_shared). |
| 047 | 047_ws_single_active.py | Partial-unique index: one ACTIVE work_session per task. |
| 048 | 048_ci_watch_project_cols.py | Per-project CI-watch opt-in cols. |
| 049 | 049_dep_update_project_cols.py | Per-project dep-update bot opt-in cols. |
| 050 | 050_playbooks.py | `playbooks` table (curated procedures). |
| 051 | 051_respawn_tracker.py | `respawn_tracker` (durable PM-respawn counter). |
| 052 | 052_task_cell_projects.py | `task_cell_projects` (per-cell project map for MegaTask root-subtask; reuses team enum create_type=False). |
| 053 | 053_playbook_archived_attr.py | `playbooks.archived_by` (UUID) + `playbooks.archived_at` (DateTime) — distinct retirement attribution; keeps `approved_by`/`approved_at` as approval-only provenance. |
| 054 | 054_a2a_message_skill.py | `a2a_messages.skill` (String 100, nullable) — persists the capability a directed A2A message concerns; was silently dropped on send. |
| 055 | 055_spawn_session_turns_tool_calls.py | `agent_spawn_sessions.turns` + `.tool_calls` (BigInteger, DEFAULT 0) — per-stint LLM iterations + tool invocations for the granular per-member performance metrics. |
| 056 | 056_member_perf_daily.py | `member_performance_daily` — one row per (date, member_kind, agent_slug) scorecard rollup (incl. CEO as `member_kind='ceo'`). |
| 057 | 057_project_sandbox_services.py | `projects.sandbox_services` (ARRAY(String), nullable) — per-project opt-in for the sandboxed per-agent-spawn Postgres/Redis provisioner. |
| 058 | 058_cloud_auth_users.py | `users` table (FastAPI Users schema) — the single seeded CEO login for cloud auth (`ROBOCO_CLOUD_AUTH_ENABLED`, default off). |
| 059 | 059_x_credentials.py | `x_credentials` (singleton Fernet-encrypted OAuth 1.0a secrets) + `x_seen_mentions` (mentions-poll dedup ledger) — the X (Twitter) engine (`ROBOCO_X_ENGINE_ENABLED`, default off). |
## Data Flow
On boot, `init_db()` probes for application tables and `alembic_version`; if a pre-Alembic DB exists it stamps it at revision 001, then always runs `run_migrations()` → `alembic upgrade head` (in a thread via `asyncio.to_thread`). `env.py` imports `roboco.db.tables` so `Base.metadata` is fully populated, overrides `sqlalchemy.url` from `settings.database_url`, and runs online with an async NullPool engine. `compare_type` + `compare_server_default` are on so autogenerate drift is detectable. `tables.py` classes are the ORM mapping the migrations build; the domain layer reads them through `roboco/models/` dataclasses, not the tables directly.
## Mermaid
```mermaid
graph LR
001-->002-->003-->004-->005-->006-->007-->008-->009-->010
010-->011-->012-->013-->014-->015-->016-->017-->018-->019
019-->020-->021-->022-->023-->024-->025-->026a-->026b-->027
027-->028-->029-->030-->031-->032-->033-->034-->035-->036
036-->037-->038-->039-->040-->041-->042-->043-->044-->045
045-->046-->047-->048-->049-->050-->051-->052-->053-->054
054-->055-->056-->057-->058-->059
```
## Logical Tree
```
Migration chain 001..059
├── Initial schema
│ └── 001 initial schema (agents, tasks, work_sessions, channels, sessions, messages, notifications, journals, audit_log, a2a_*)
├── Persistence
│ └── 002 persistence tables + NotificationType.APPROVAL
├── Blocker metadata
│ ├── 003 blocker_resolver_type + blockerresolvertype enum
│ └── 005 blocker_raised_by
├── Provider routing & model assignments
│ ├── 004 provider_configs + model_assignments (modelprovider/assignmentscope enums)
│ ├── 019 seed default model providers
│ ├── 028 seed Self-Hosted (Ollama LOCAL) provider
│ ├── 038 add grok to modelprovider enum
│ └── 039 seed Grok (xAI) provider
├── Gateway
│ ├── 006 gateway columns (claimant lock, heartbeat, pre-block snapshot, AC status, qa evidence)
│ └── 007 gateway_triggers (dispatcher decision log)
├── Enum reconcile / widening
│ ├── 009 reconcile postgres enums with ORM StrEnum
│ ├── 011 drop quarantined from taskstatus enum
│ ├── 012 align agentrole/team enums with foundation
│ ├── 013 drop stray role enum
│ ├── 020 backfill ORM enum values
│ ├── 025 add prompter to agentrole enum
│ ├── 034 add secretary to agentrole enum
│ └── 037 add pr_reviewer to agentrole enum
├── Audit log
│ ├── 010 audit_log.details JSON→JSONB
│ └── 045 tasks.revision_count + audit_log composite query index
├── Cleanup / drops
│ ├── 014 drop unused tasks.pm_approvals
│ ├── 015 drop unused execution_log/outputs
│ └── 008 no-op (skills alignment done statically)
├── Products
│ ├── 016 products + product_projects; tasks.product_id (team enum create_type=False)
│ └── 018 tasks.project_id nullable
├── ORM drift reconcile
│ └── 017 add ORM tables/columns the chain never had (indexed_documents)
├── Board review
│ └── 021 tasks.board_review_complete
├── Project defaults
│ └── 022 flip projects.default_branch default main→master
├── Prompter tracking
│ ├── 023 tasks.source + confirmed_by_human
│ └── 024 prompter_sessions, prompter_messages, task_drafts
├── Dependency / token usage
│ ├── 026a tasks.completed_dependency_ids
│ └── 026b agent_spawn_sessions + token_usage_snapshots (chained off 026a)
├── System settings
│ └── 027 system_settings key-value store
├── Project quality
│ └── 029 projects.quality_command (fast pre-submit gate)
├── RAG
│ ├── 030 align RAG chunk tables with vector-store schema
│ └── 031 tsvector + GIN index on chunk tables (hybrid retrieval)
├── Strategy / provisioning
│ ├── 032 company_goals singleton charter
│ └── 033 pitches (Board proposals → auto-provision)
├── Secretary
│ └── 035 secretary_directives (command audit + gate queue)
├── Acceptance criteria
│ └── 036 per-criterion AC ids + child→parent AC linkage
├── PR review
│ ├── 040 add awaiting_pr_review to taskstatus enum
│ └── 041 pr_reviewer_notes, machine-marker split, structured content cols
├── Worksession toolchain
│ └── 042 work_sessions toolchain matching cols
├── Conventions standard
│ ├── 043 project_conventions_cache
│ └── 044 project_convention_findings (violations feed)
├── MegaTask / batch intake
│ ├── 046 tasks.batch_id + collision descriptors
│ └── 052 task_cell_projects (per-cell project map for MegaTask root-subtask)
├── WorkSession single-active
│ └── 047 partial-unique index: one ACTIVE work_session per task
├── Autonomous maintenance
│ ├── 048 per-project CI-watch opt-in cols
│ └── 049 per-project dep-update bot opt-in cols
├── Organizational memory
│ ├── 050 playbooks table (curated procedures)
│ └── 053 playbooks.archived_by + archived_at (distinct retirement attribution from approval)
├── Orchestrator runtime durability
│ └── 051 respawn_tracker (durable PM-respawn counter)
├── A2A messaging
│ └── 054 a2a_messages.skill (nullable; persists directed-A2A capability context)
├── Per-member performance metrics
│ ├── 055 agent_spawn_sessions.turns + .tool_calls (DEFAULT 0)
│ └── 056 member_performance_daily (per date/member_kind/agent_slug rollup)
├── Sandboxed dev DB/Redis
│ └── 057 projects.sandbox_services (per-project opt-in array)
├── Cloud auth
│ └── 058 users (FastAPI Users; single seeded CEO login)
└── X (Twitter) engine
└── 059 x_credentials (singleton encrypted OAuth 1.0a) + x_seen_mentions (dedup ledger)
```
## Dependencies
- PostgreSQL 15+ (NULLS NOT DISTINCT) — actually pgvector image on PG 16.
- `pgvector` extension for RAG cosine similarity (`chunks_*` tables, `indexed_documents`).
- `asyncpg` driver; SQLAlchemy 2.0 async.
- Alembic; migrations run on every orchestrator boot.
## Entry Points
- `init_db()` / `run_migrations()` in `roboco/db/base.py` — boot-time `alembic upgrade head`.
- `bootstrap_database()` in `roboco/db/seed.py` — init + seed.
- `alembic upgrade head` (manual, in orchestrator container).
- `conftest` (tests) — ephemeral DB per test; runs migrations or `create_all` depending on PG availability.
## Config Flags
- `ROBOCO_DATABASE_*` (host/port/user/password/name) — `settings.database_url`.
- `ROBOCO_DATABASE_ECHO`, pool size/timeout/recycle.
- No DB-specific feature flag; migrations always run. Feature flags (`ROBOCO_CONVENTIONS_ENABLED`, `ROBOCO_CI_WATCH_ENABLED`, `ROBOCO_DEP_UPDATE_ENABLED`, `ROBOCO_RELEASE_MANAGER_ENABLED`, `ROBOCO_ORG_MEMORY_ENABLED`) gate *use* of tables the migrations already added.
## Gotchas
- **`sa.Enum(create_type=False)` is silently ignored** — the flag only works on `postgresql.ENUM`, not generic `sa.Enum`. Using `sa.Enum` re-emits CREATE TYPE and fails with "type already exists". 001/016/052 carry the live gotcha comment; 004/016 use the correct `postgresql.ENUM(create_type=False)`. 001 itself uses `sa.Enum(..., create_type=False)` in spots — latent on a clean re-apply.
- **Enum-parity gate can false-green** — `make quality` runs `scripts/verify_postgres_enums.py` only against a migrated DB; an empty `roboco` DB (conftest ephemeral) or `|| echo` masking hides drift. Fixed in 957fb522 but the gate is only as good as the DB it points at.
- **016 latent** — the `team` enum member list under create_type=False is the *original* set, not the later-widened set; inert but misleading.
- **Two files numbered 026** — not a collision: `026_token_usage_tables` chains off `026_completed_dependency_ids`. Renaming is risky (breaks down_revision refs).
- **052 reuses the `team` enum** with `create_type=False` correctly — no new enum added; safe.
- **017 reconciled drift the other way** — added ORM tables the chain had missed; `create_all` is no longer authoritative.
## Drift from CLAUDE.md
- CLAUDE.md says "52 migrations 001..052" — now stale; chain is 001..059 (59 files). Does not mention the two 026 files (chained, not a conflict).
- CLAUDE.md cites migrations 043/046/047/048/049/050/051 by number in feature sections — all present and consistent.
- No factual drift found in the DB layer description.
## Changes Since Baseline
`git log fd10cc862c2020b3f639cdb686d427b0198a2441..HEAD -- alembic/ roboco/db/`:
- `15effce0` Chore: 141 Gaps fill-in (#283) — adds migration 052 (`task_cell_projects`) + `TaskCellProjectTable`; logic-touching.
(Only one commit in range touches these paths.)
> Post-snapshot updates (since 2026-06-29): `536bbb64` (Chore/all/logical gaps sweep #286) — adds migration 053 (`playbooks.archived_by`/`archived_at`), two new columns on `PlaybookTable`; `d8a5bb48` ([chore] a2a hierarchy gate + skill persist) — adds migration 054 (`a2a_messages.skill`), one new column on `A2AMessageTable`, wired through `send_chat_message` and the A2AChatMessage model.
>
> Delta 2026-07-03 (v0.17.0, 5 features): `055_spawn_session_turns_tool_calls` (`agent_spawn_sessions.turns`/`.tool_calls`) + `056_member_perf_daily` (`member_performance_daily`) predate this wave but were never appended to this doc; `057_project_sandbox_services` adds `projects.sandbox_services` (sandboxed dev DB/Redis, `ROBOCO_SANDBOX_DB_ENABLED`); `058_cloud_auth_users` adds `users` (`UserTable`, cloud auth, `ROBOCO_CLOUD_AUTH_ENABLED`); `059_x_credentials` adds `x_credentials` (`XCredentialsTable`) + `x_seen_mentions` (`XSeenMentionTable`) (X engine, `ROBOCO_X_ENGINE_ENABLED`). Chain head is now 059.
## Regression Risks
| Title | File:Line | Claim | Severity |
|-------|-----------|-------|----------|
| sa.Enum create_type silently dropped | alembic/versions/001_initial_schema.py:127,312 | 001 uses `sa.Enum(..., create_type=False)` which is a no-op on the generic Enum; a fresh re-apply on a clean DB can double-emit CREATE TYPE. | High |
| Enum-parity gate false-green | Makefile:540 + scripts/verify_postgres_enums.py | Gate skips on no-migrated-DB; an empty/mismatched `roboco` DB hides postgres-enum drift until a smoke run. | High |
| 016 team enum stale member list | alembic/versions/016_add_products_and_task_product_id.py:38 | `postgresql.ENUM(create_type=False)` member list is frozen at the original set; inert but masks later widening. | Medium |
| Missing pgvector extension blocks RAG | roboco/db/tables.py (chunks_*/indexed_documents) | Migrations assume pgvector installed; on a plain PG the vector columns fail and init_db aborts. | High |
| Two 026 files — rename hazard | alembic/versions/026_*.py | Renaming either 026 file breaks `down_revision` chain; autogenerate may mis-order. | Medium |
| 047 partial-unique index assumes single-active | alembic/versions/047_ws_single_active.py | A duplicate ACTIVE session raises on the partial-unique index; service-layer guard must run first or claim crashes. | Medium |
| 052 reuses team enum — order-dependent | alembic/versions/052_task_cell_projects.py:44 | Depends on `team` enum already existing (from 001/016); a partial chain replay to 052 without 016 would fail. | Low |
| Single-head violation on re-apply | alembic/versions/017_reconcile_orm_schema_drift.py | 017 adds tables/columns that `create_all` had created; on a DB built by `create_all` then stamped, 017 may double-create. | Medium |
## Health
The chain is linear and complete (001→059), with `init_db` running `upgrade head` on every boot so deployed schemas stay current. The two structural risks are the `sa.Enum(create_type=False)` no-op in 001 (latent on clean re-applies) and the enum-parity gate's dependence on a populated migrated DB. New migrations consistently use the `postgresql.ENUM(create_type=False)` pattern and `ALTER TYPE ... ADD VALUE IF NOT EXISTS` for enum widening, so recent additions are safe.
Reference in New Issue View Git Blame Copy Permalink