feat(web): database config (file/sqlite/postgres) + docs (phase 3)

Wire a `database` block in bdrive web config: {driver: file|sqlite|postgres, dsn}. Default file, unchanged (existing per-registry paths honored). sqlite or postgres routes all five hub registries through the SQL MetaStore. Verified: signup + project + invite round-trip survives a restart on BOTH file and sqlite; the sqlite DB holds real relational tables/rows (not JSON blobs). Docs: a "choosing a database" section (README), the persistence architecture note (CLAUDE.md), and a config mention (SKILL.md). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R7Q9ZKSZRTdvrSJkYLUmYs
2026-08-25 08:08:08 +02:00 · 2026-07-10 06:54:49 -07:00
parent 0fa83e4532
commit 62175e29a9
4 changed files with 94 additions and 9 deletions
@@ -39,7 +39,7 @@ Package roles (`internal/`):
 - **`syncer`** — the heart: `Session.Cycle()` runs one pass: scan → commit local ops → pull peer journals → preserve conflict copies → materialize merged state → push blobs + own journal. Read the package doc comment in `syncer.go` first. `ignore.go` holds the path filter (`.bdriveignore` rules + the `.bdrive` include list), applied symmetrically in scan and materialize; a newly filtered path is dropped from the cache *without* a delete op so opting out locally never deletes remotely.
 - **`daemon`** — per-mount background loop (detached process, `daemon.pid`/`daemon.log` in the mount's volume dir). Scans every `--scan-interval` (3s), talks to the remote every `--remote-interval` (10s) or immediately after local edits. Re-reads `.bdrive/config.json` each tick; if it vanishes (folder moved/renamed/deleted) the daemon **exits cleanly without propagating deletes** — the next bdrive command at the new location resumes it (self-heal on next touch).
 - **`config`** — global state under `$BDRIVE_HOME` (default `~/.bdrive`): device identity (`device.json`), settings (`settings.json`: default server + device token + signed-in account), and the mount registry (`mounts.json`, keyed by **stable mount id**, holding only each mount's last-known path). The per-folder `.bdrive/` directory (`project.go`) holds `config.json` with the mount id + volume/remote/include; **nothing is keyed by the folder path**, so renames/moves are free — `ResolveMount` self-heals the registry path, and the volume store lives at `~/.bdrive/volumes/<mount-id>/`. `.bdrive/` is never synced and holds no credentials.
- **`webapp`** — the `bdrive web` server, in two modes. Single-volume: `Source` is a `DirSource` (plain folder from disk) or `RemoteSource` (folds journals into a file tree with per-file provenance). Hub: `Root` + `Projects` host many projects on one storage root, each under `<root>/<project-id>/` via `remote.Prefixed`; `ProjectDB` (`projects.go`) is a file-backed registry (JSON, loaded at open, rewritten atomically per change) with create-or-join-by-name semantics, name-scoped per organization. Orgs (`orgs.go`, file-backed `orgs.json`) wall projects by membership (email → owner|member): every per-project route — viewer APIs, uploads, history, shares management, the `/store/*` sync proxy — 403s for non-members, `/api/projects` lists only your orgs' projects, owners mint expiring multi-use invite links (`/join/<token>`), and a pre-org hub migrates all projects into a "default" org (all existing accounts join, oldest owns) at startup. `QuotaProvider` (`quota.go`) is the plan-enforcement seam mirroring `AuthProvider` — CheckWrite/RecordUsage on every write path, CheckSeat on invite redemption; OSS ships only `UnlimitedQuota`, managed deployments swap the provider. Renders markdown (goldmark + Obsidian `[[wikilinks]]`). With `--upload` it accepts writes: browser uploads (`upload.go` — direct-to-storage via presigned URLs when the backend implements `remote.PutSigner`, relayed otherwise; ops journaled under the server's own device) and the per-project `/api/p/<id>/store/*` proxy (`store.go`) that whole devices sync through — the `https://` remote backend (`remote/http.go`) is its client; journals are never presigned, only immutable blobs. Frontend is dependency-free vanilla JS embedded via `go:embed static`; it learns everything from `/api/config` (+ `/api/projects` in hub mode) and never sees storage info or credentials. It uses native History-API path routing (`/<project-id>/<path>` in hub mode, `/<path>` in volume mode, `/join/<token>` for invites — no `#`, slashes stay literal); `Server.frontend` serves `index.html` as the SPA fallback for any non-asset, non-API/auth/share route so deep links and refreshes resolve, and all client API/asset URLs are root-absolute so a deep path doesn't break relative resolution.
+- **`webapp`** — the `bdrive web` server, in two modes. Single-volume: `Source` is a `DirSource` (plain folder from disk) or `RemoteSource` (folds journals into a file tree with per-file provenance). Hub: `Root` + `Projects` host many projects on one storage root, each under `<root>/<project-id>/` via `remote.Prefixed`; `ProjectDB` (`projects.go`) is a file-backed registry (JSON, loaded at open, rewritten atomically per change) with create-or-join-by-name semantics, name-scoped per organization. Orgs (`orgs.go`, file-backed `orgs.json`) wall projects by membership (email → owner|member): every per-project route — viewer APIs, uploads, history, shares management, the `/store/*` sync proxy — 403s for non-members, `/api/projects` lists only your orgs' projects, owners mint expiring multi-use invite links (`/join/<token>`), and a pre-org hub migrates all projects into a "default" org (all existing accounts join, oldest owns) at startup. `QuotaProvider` (`quota.go`) is the plan-enforcement seam mirroring `AuthProvider` — CheckWrite/RecordUsage on every write path, CheckSeat on invite redemption; OSS ships only `UnlimitedQuota`, managed deployments swap the provider. Renders markdown (goldmark + Obsidian `[[wikilinks]]`). With `--upload` it accepts writes: browser uploads (`upload.go` — direct-to-storage via presigned URLs when the backend implements `remote.PutSigner`, relayed otherwise; ops journaled under the server's own device) and the per-project `/api/p/<id>/store/*` proxy (`store.go`) that whole devices sync through — the `https://` remote backend (`remote/http.go`) is its client; journals are never presigned, only immutable blobs. Frontend is dependency-free vanilla JS embedded via `go:embed static`; it learns everything from `/api/config` (+ `/api/projects` in hub mode) and never sees storage info or credentials. It uses native History-API path routing (`/<project-id>/<path>` in hub mode, `/<path>` in volume mode, `/join/<token>` for invites — no `#`, slashes stay literal); `Server.frontend` serves `index.html` as the SPA fallback for any non-asset, non-API/auth/share route so deep links and refreshes resolve, and all client API/asset URLs are root-absolute so a deep path doesn't break relative resolution. **Hub metadata persistence** (accounts, projects, orgs+invites, shares, devices — never blobs or journals) sits behind a pluggable `MetaStore` of typed repos (`db.go`): the service structs (`BuiltinAuth`, `OrgDB`, `ProjectDB`, `ShareDB`, `DeviceRegistry`) keep their in-memory maps + logic and persist each change as one record through a repo. Two backends — `db_file.go` (the historical JSON files, still the zero-dep default, reached via the `Open*(path)` constructors) and `db_sql.go` (one `database/sql` impl over pure-Go drivers: `modernc.org/sqlite` locally, `jackc/pgx` for Postgres/Supabase, portable schema + idempotent migrations + transactional multi-row writes). `web.go`'s `database` config (`{driver:file|sqlite|postgres, dsn}`) selects it; file is default and untouched. `db_conformance_test.go` runs the same service ops against every backend.

 `cmd/bdrive/` is a thin cobra CLI over these packages (`login`, `init`, `stop`, `sync`, `status`, `log`, `remote`, `web`, `whoami`, `daemon`, `version` — `mnt`/`umnt` are gone; `init` is the front door and `stop` pauses). `bdrive login` signs the device in (bare form uses the remembered server or `config.DefaultServer` = beardrive.ai; loopback-callback browser flow in `login.go`, `--device` for headless) and stores server+token+account in `settings.json`. `bdrive init` is interactive on a TTY (survey menus: create-new vs connect-existing with a project list; whole-folder vs `--shared <dir>`, which becomes the include list) with full flag bypass (`--name/--project/--shared/--yes`) and never prompts without a TTY; it runs the login flow first when there is no session, writes `.bdrive/config.json`, seeds `.bdriveignore`, and starts sync via `startSync`; re-running it resumes — including after a folder move. `bdrive web -c config.json` configures the server from a file, explicit flags winning.

@@ -348,6 +348,33 @@ non-localhost address. Internally all of this sits behind an
 email/password provider, and alternative identity backends can be swapped
 in without touching the CLI or the API.

+### Choosing a database
+
+A hub keeps a little **metadata** — accounts, projects, orgs, invites,
+shares, devices — separate from your files. (File content and the sync
+journals always live in the object store; the database never holds them.)
+You choose where that metadata lives with the `database` block:
+
+```jsonc
+"database": { "driver": "file" }                       // default — JSON under BDRIVE_HOME
+"database": { "driver": "sqlite",   "dsn": "/var/lib/bdrive/hub.db" }
+"database": { "driver": "postgres", "dsn": "postgres://…@…pooler.supabase.com:6543/postgres" }
+```
+
+- **file** (default): zero dependencies, human-readable JSON, perfect for a
+  laptop or a small self-hosted hub.
+- **sqlite**: one embedded database file — a real DB locally with no server
+  to run.
+- **postgres**: a managed Postgres such as **Supabase** for production —
+  just point `dsn` at its connection string (use the transaction pooler for
+  many connections). Since Supabase *is* Postgres, this stays fully
+  open-source with no managed-only lock-in.
+
+`file` and `sqlite` are single-writer (run one hub instance); Postgres is
+transactional and can back more than one instance. Switching backends
+doesn't migrate existing data — pick one when you set the hub up. Both SQL
+drivers are pure Go, so the binary stays a CGO-free static build.
+
 ### Uploads

 The browser client is deliberately storage-blind: it never sees the remote
@@ -50,6 +50,13 @@ type webConfig struct {
 			From string `json:"from,omitempty"`
 		} `json:"smtp,omitempty"`
 	} `json:"auth,omitempty"`
+	// Database selects where hub metadata (accounts, projects, orgs, invites,
+	// shares, devices) lives. Default "file" (JSON under BDRIVE_HOME). Blobs
+	// and journals always stay in the object store, never the database.
+	Database *struct {
+		Driver string `json:"driver,omitempty"` // file (default) | sqlite | postgres
+		DSN    string `json:"dsn,omitempty"`    // sqlite file path, or a Postgres/Supabase URL
+	} `json:"database,omitempty"`
 }

 func loadWebConfig(path string) (webConfig, error) {
@@ -162,6 +169,7 @@ credentials); otherwise it is relayed through this server.`,
 				ShareRPM: cfg.ShareRPM,
 			}
 			var display string
+			var meta webapp.MetaStore // hub metadata store; nil means the file backend
 			if dir != "" {
 				// Single-volume viewer over a plain folder.
 				abs, err := filepath.Abs(dir)
@@ -188,7 +196,31 @@ credentials); otherwise it is relayed through this server.`,
 					}
 					projectsDB = filepath.Join(home, "projects.json")
 				}
-				db, err := webapp.OpenProjectDB(projectsDB)
+				// Pick the metadata backend: file (default) or a SQL database
+				// (sqlite locally, Postgres/Supabase in production).
+				if cfg.Database != nil {
+					switch cfg.Database.Driver {
+					case "", "file":
+					case "sqlite", "postgres", "pgx":
+						drv := cfg.Database.Driver
+						if drv == "postgres" {
+							drv = "pgx"
+						}
+						meta, err = webapp.OpenSQLStore(drv, cfg.Database.DSN)
+						if err != nil {
+							return fmt.Errorf("open database: %w", err)
+						}
+						defer meta.Close()
+					default:
+						return fmt.Errorf("unknown database driver %q (use file, sqlite, or postgres)", cfg.Database.Driver)
+					}
+				}
+				var db *webapp.ProjectDB
+				if meta != nil {
+					db, err = webapp.NewProjectDB(meta.Projects())
+				} else {
+					db, err = webapp.OpenProjectDB(projectsDB)
+				}
 				if err != nil {
 					return fmt.Errorf("open project registry: %w", err)
 				}
@@ -236,7 +268,12 @@ credentials); otherwise it is relayed through this server.`,
 				if usersDB == "" {
 					usersDB = filepath.Join(home, "auth.json")
 				}
-				auth, err := webapp.OpenBuiltinAuth(usersDB, allowSignup, mail)
+				var auth *webapp.BuiltinAuth
+				if meta != nil {
+					auth, err = webapp.NewBuiltinAuth(meta.Accounts(), allowSignup, mail)
+				} else {
+					auth, err = webapp.OpenBuiltinAuth(usersDB, allowSignup, mail)
+				}
 				if err != nil {
 					return fmt.Errorf("open account registry: %w", err)
 				}
@@ -266,7 +303,12 @@ credentials); otherwise it is relayed through this server.`,
 					return err
 				}
 				srv.Auth = auth
-				orgs, err := webapp.OpenOrgDB(filepath.Join(filepath.Dir(projectsDB), "orgs.json"))
+				var orgs *webapp.OrgDB
+				if meta != nil {
+					orgs, err = webapp.NewOrgDB(meta.Orgs())
+				} else {
+					orgs, err = webapp.OpenOrgDB(filepath.Join(filepath.Dir(projectsDB), "orgs.json"))
+				}
 				if err != nil {
 					return fmt.Errorf("open org registry: %w", err)
 				}
@@ -278,17 +320,31 @@ credentials); otherwise it is relayed through this server.`,
 				if err := webapp.MigrateOrgs(srv.Projects, orgs, auth.Accounts()); err != nil {
 					return fmt.Errorf("migrate projects into orgs: %w", err)
 				}
-				devices, err := webapp.OpenDeviceRegistry(filepath.Join(filepath.Dir(projectsDB), "devices.json"))
+				var devices *webapp.DeviceRegistry
+				if meta != nil {
+					devices, err = webapp.NewDeviceRegistry(meta.Devices())
+				} else {
+					devices, err = webapp.OpenDeviceRegistry(filepath.Join(filepath.Dir(projectsDB), "devices.json"))
+				}
 				if err != nil {
 					return fmt.Errorf("open device registry: %w", err)
 				}
 				srv.Devices = devices
-				shares, err := webapp.OpenShareDB(filepath.Join(filepath.Dir(projectsDB), "shares.json"))
+				var shares *webapp.ShareDB
+				if meta != nil {
+					shares, err = webapp.NewShareDB(meta.Shares())
+				} else {
+					shares, err = webapp.OpenShareDB(filepath.Join(filepath.Dir(projectsDB), "shares.json"))
+				}
 				if err != nil {
 					return fmt.Errorf("open share registry: %w", err)
 				}
 				srv.Shares = shares
-				display += " (auth: " + usersDB + ")"
+				if meta != nil {
+					display += " (db: " + cfg.Database.Driver + ")"
+				} else {
+					display += " (auth: " + usersDB + ")"
+				}
 			}

 			shown := addr
@@ -25,7 +25,7 @@ Use this skill whenever the user is working with the `bdrive` CLI: initializing
 | Share a synced file publicly by URL | `bdrive share <file>` — prints a link anyone can open (HTML renders as a page, markdown rendered, PDFs inline; sandboxed; always the latest content; no account needed). `--expires 24h` for self-destructing links; `--list` / `--revoke <token-or-url>` to manage. Put generated reports in the shared folder, sync, then share. |
 | Set up a project for a Claude Code team | `/beardrive:install` — installs the CLI, signs in, runs init (whole/shared folder), offers a CLAUDE.md section about the shared folder, and registers project-level hooks (blocking pull at prompt-submit, async push after Write/Edit) in `.claude/settings.json` |
 | Per-file / folder change history in the web UI | History button (file versions or project feed) and per-folder ⌚ — each entry: account, time, device (name/OS/IP), view/download of that exact version. API: `GET /api/p/<id>/history?path=\|prefix=`, `GET /api/p/<id>/blob?sha=` |
-| Web server: viewer + multi-project sync hub (read-only unless `--upload`) | `bdrive web [<folder> \| <storage-root-url>]` (serves cwd by default, `--addr :4173`; `-c config.json` reads remote/addr/upload/projects_db settings from a file, explicit flags win; a storage root URL makes it a hub hosting many projects at `<root>/<project-id>/`, registry in `--projects-db` file, default `$BDRIVE_HOME/projects.json`; `--upload` lets browsers add files, client devices push, and projects be created — direct to storage via expiring presigned URLs on S3/GCS, relayed through the server for `file://`; `--upload-ttl 15m`; clients never see the remote URL or credentials; hub projects are walled by org membership — invite teammates from the web UI; the viewer has a ⌘K palette for fuzzy file search, project switching, and quick actions) |
+| Web server: viewer + multi-project sync hub (read-only unless `--upload`) | `bdrive web [<folder> \| <storage-root-url>]` (serves cwd by default, `--addr :4173`; `-c config.json` reads remote/addr/upload/projects_db/database/auth settings from a file, explicit flags win; a storage root URL makes it a hub hosting many projects at `<root>/<project-id>/`, registry in `--projects-db` file, default `$BDRIVE_HOME/projects.json`; `--upload` lets browsers add files, client devices push, and projects be created — direct to storage via expiring presigned URLs on S3/GCS, relayed through the server for `file://`; `--upload-ttl 15m`; clients never see the remote URL or credentials; hub projects are walled by org membership — invite teammates from the web UI; the viewer has a ⌘K palette for fuzzy file search, project switching, and quick actions) |

 `<folder>` is created if missing. Omitting it on `sync`/`status`/`log` defaults to the current working directory.

@@ -93,7 +93,9 @@ cd ~/agent-workspace && bdrive init --name agent-workspace

 Devices connecting the same project (by name or id) converge through the hub. Direct-to-bucket setups (no hub) remain possible via `bdrive remote set <folder> s3://…` after an offline init.

-Hub projects belong to an **organization**: only members of the project's org can see or sync it (project names are scoped per org too). Your first `bdrive init` creates your org automatically. **Hubs are invite-only by default** — the safe posture for a public URL. To give a teammate access, an org **owner** opens the web UI and clicks **Invite** in the sidebar footer — it mints an expiring join link (`…/join/<token>`); the teammate opens it and creates an account through the link (invites bootstrap signup even when public self-signup is closed), and is in. An admin can instead open self-service signup with a gate (admin approval, or allowed-domains + email verification) under **Admin → Signup & access** / the config's `auth` block. If a teammate's `bdrive init --project <id>` gets 403/404 or the project list looks empty, the missing invite is the reason. Public share links (`bdrive share`) intentionally bypass the org wall.
+Hub projects belong to an **organization**: only members of the project's org can see or sync it (project names are scoped per org too). Your first `bdrive init` creates your org automatically. **Hubs are invite-only by default** — the safe posture for a public URL. To give a teammate access, an org **owner** opens the web UI and clicks **Invite** in the sidebar footer — it mints an expiring join link (`…/join/<token>`); the teammate opens it and creates an account through the link (invites bootstrap signup even when public self-signup is closed), and is in. An admin can instead open self-service signup with a gate (admin approval, or allowed-domains + email verification) under **Admin → Signup & access** / the config's `auth` block.
+
+A hub stores its metadata (accounts, projects, orgs, invites, shares, devices — never files or journals, which stay in object storage) in a database chosen by the config's `database` block: `{"driver":"file"}` (default, JSON under `$BDRIVE_HOME`), `{"driver":"sqlite","dsn":"…/hub.db"}`, or `{"driver":"postgres","dsn":"postgres://…"}` for a managed Postgres such as Supabase. file/sqlite are single-writer; Postgres backs multiple instances. If a teammate's `bdrive init --project <id>` gets 403/404 or the project list looks empty, the missing invite is the reason. Public share links (`bdrive share`) intentionally bypass the org wall.

 ### Renames and moves