mirror of
https://github.com/snapotter-hq/SnapOtter.git
synced 2026-08-03 07:46:42 +02:00
* feat(api): parse DATA_DIR from env for 1.x import auto-detection Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * test(migrator): build 1.17.2 fixtures by replaying legacy migrations Discovered the legacy migrations seed a Default team (0005) and builtin roles (0007), so the replayed fixture carries them. Seed uses a distinct custom team. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * fix(migrator): self-adjusting column copy, jobs.status map, drop sessions, advisory lock The importer now inserts only the intersection of source and live target columns, so the three analytics_* columns 2.x dropped no longer break the first users INSERT (and future dropped columns are handled generically). jobs.status is mapped onto the 2.x enum (error->failed). Sessions are no longer migrated. A pg_advisory_xact_lock serializes concurrent replicas. Includes login-after-migrate and library assertions. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * test(migrator): CI drift guard fails when a required column is unfillable from 1.17.2 Introspects every NOT-NULL-no-default column of each migrated table in the current schema and asserts the engine can fill it from a real 1.17.2 source. Turns a future breaking schema change into a PR-time failure instead of a production import break. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * feat(migrator): orchestrator with detection, boot states, marker, blob count sqlite-import.ts owns source resolution (explicit path, 'off' sentinel, DATA_DIR probe), the four boot states (import/leftover/locked/none), the persisted sqlite_import marker, and a read-only library-blob count. runBootImport wires them together and catches TargetNonEmptyError as a benign multi-replica skip. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * feat(api): route boot through the 1.x import orchestrator; hide marker from non-admins index.ts now calls runBootImport (which owns detection + the four boot states) instead of the inline SQLITE_MIGRATE_PATH block. The sqlite_import marker is added to SENSITIVE_KEYS (but not REDACTED_KEYS) so admins see the counts for the banner while non-admins don't see the key at all. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * feat(migrator): add analyzeSqlite + dry-run/verify CLI analyzeSqlite is a read-only pre-flight (no live Postgres): per-table row counts, library-blob presence, and out-of-enum job statuses. The migrate:sqlite CLI now lives in the orchestrator and supports --dry-run/--verify (prints the analysis and exits without writing) alongside the existing import and --force. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * docs: add 1.x to 2.0 upgrade guide; fix volume-name casing New apps/docs upgrade guide covering auto-detect, the SQLITE_MIGRATE_PATH override + off opt-out, the dry-run, what carries over, locked-state recovery, and non-destructive rollback. Leads with 'back up the WHOLE /data volume, not just snapotter.db' because 1.x WAL mode leaves data in snapotter.db-wal (surfaced by the real-image upgrade test). Standardizes README/DOCKERHUB compose volume names on the canonical SnapOtter-data casing so they match the repo compose and don't orphan an upgrader's volume. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * feat(web): admin 1.x migration banner + 21-locale strings A one-time admin banner reads the sqlite_import marker from /v1/settings and shows the import result (user + saved-file counts) on success, or a warning when a 1.x database was found but not imported. Dismissal persists to a sqlite_import.dismissedAt settings key. shouldShowMigrationBanner/parseMigrationMarker sit in feedback.ts with the other shouldShow helpers; strings added to en.ts and all 20 other locales. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * style(landing): biome-format Hero.astro trustBadges array Pre-existing formatting drift on main (its Lint check was skipped on the merge that introduced it); this PR's full Lint run surfaced it. Formatting-only, applied via the repo's own biome formatter to unblock the required Lint check. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w
183 lines
6.1 KiB
Markdown
183 lines
6.1 KiB
Markdown
---
|
|
description: PostgreSQL database schema, tables, migrations, and backup procedures for SnapOtter.
|
|
---
|
|
|
|
# Database
|
|
|
|
SnapOtter uses PostgreSQL 17 with [Drizzle ORM](https://orm.drizzle.team/) (pg-core / node-postgres) for data persistence. The schema is defined in `apps/api/src/db/schema.ts`.
|
|
|
|
The connection is configured via the `DATABASE_URL` environment variable (default `postgres://snapotter:snapotter@postgres:5432/snapotter`). In Docker Compose, the Postgres container stores its data in the `SnapOtter-pgdata` named volume.
|
|
|
|
## Tables
|
|
|
|
### users
|
|
|
|
Stores user accounts. Created automatically on first run from `DEFAULT_USERNAME` and `DEFAULT_PASSWORD`.
|
|
|
|
| Column | Type | Notes |
|
|
|---|---|---|
|
|
| `id` | uuid | Primary key |
|
|
| `username` | varchar | Unique, required |
|
|
| `passwordHash` | varchar | scrypt hash |
|
|
| `role` | varchar | `admin`, `editor`, or `user` |
|
|
| `mustChangePassword` | boolean | Forced password reset flag |
|
|
| `createdAt` | timestamp | Creation time |
|
|
| `updatedAt` | timestamp | Last update time |
|
|
|
|
### sessions
|
|
|
|
Active login sessions. Each row ties a session token to a user.
|
|
|
|
| Column | Type | Notes |
|
|
|---|---|---|
|
|
| `id` | varchar | Primary key (session token) |
|
|
| `userId` | uuid | Foreign key to `users.id` |
|
|
| `expiresAt` | timestamp | Expiry time |
|
|
| `createdAt` | timestamp | Creation time |
|
|
|
|
### teams
|
|
|
|
Groups for organizing users. Admins can assign users to teams.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| `id` | uuid | Primary key |
|
|
| `name` | varchar (unique, max 50 chars) | Team name |
|
|
| `createdAt` | timestamp | Creation time |
|
|
|
|
### api_keys
|
|
|
|
API keys for programmatic access. The raw key is shown once on creation; only the hash is stored.
|
|
|
|
| Column | Type | Notes |
|
|
|---|---|---|
|
|
| `id` | uuid | Primary key |
|
|
| `userId` | uuid | Foreign key to `users.id` |
|
|
| `keyHash` | varchar | scrypt hash of the key |
|
|
| `name` | varchar | User-provided label |
|
|
| `createdAt` | timestamp | Creation time |
|
|
| `lastUsedAt` | timestamp | Updated on each authenticated request |
|
|
|
|
Keys are prefixed with `si_` followed by 96 hex characters (48 random bytes).
|
|
|
|
### pipelines
|
|
|
|
Saved tool chains that users create in the UI.
|
|
|
|
| Column | Type | Notes |
|
|
|---|---|---|
|
|
| `id` | uuid | Primary key |
|
|
| `name` | varchar | Pipeline name |
|
|
| `description` | varchar | Optional description |
|
|
| `steps` | jsonb | Array of `{ toolId, settings }` objects |
|
|
| `createdAt` | timestamp | Creation time |
|
|
|
|
### user_files
|
|
|
|
Persistent file library with version chain tracking. Each processing step that saves a result creates a new row linked to its parent via `parentId`, forming a version tree.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| `id` | uuid | Primary key |
|
|
| `userId` | uuid | FK to users (CASCADE DELETE) |
|
|
| `originalName` | varchar | Original upload filename |
|
|
| `storedName` | varchar | Filename on disk |
|
|
| `mimeType` | varchar | MIME type |
|
|
| `size` | integer | File size in bytes |
|
|
| `width` | integer | Image width in px |
|
|
| `height` | integer | Image height in px |
|
|
| `version` | integer | Version number (1 = original) |
|
|
| `parentId` | uuid or null | FK to user_files (parent version) |
|
|
| `toolChain` | jsonb | Tool IDs applied in order to produce this version |
|
|
| `createdAt` | timestamp | Creation time |
|
|
|
|
### jobs
|
|
|
|
Tracks processing jobs for progress reporting and cleanup.
|
|
|
|
| Column | Type | Notes |
|
|
|---|---|---|
|
|
| `id` | uuid | Primary key |
|
|
| `type` | varchar | Tool or pipeline identifier |
|
|
| `status` | varchar | `queued`, `processing`, `completed`, or `failed` |
|
|
| `progress` | real | 0.0-1.0 fraction |
|
|
| `inputFiles` | jsonb | Array of input file paths |
|
|
| `outputPath` | varchar | Path to the result file |
|
|
| `settings` | jsonb | Tool settings used |
|
|
| `error` | varchar | Error message if failed |
|
|
| `createdAt` | timestamp | Creation time |
|
|
| `completedAt` | timestamp | Completion time |
|
|
|
|
### settings
|
|
|
|
Key-value store for server-wide settings that admins can change from the UI.
|
|
|
|
| Column | Type | Notes |
|
|
|---|---|---|
|
|
| `key` | varchar | Primary key |
|
|
| `value` | varchar | Setting value |
|
|
| `updatedAt` | timestamp | Last update time |
|
|
|
|
### roles
|
|
|
|
Custom roles with granular permissions.
|
|
|
|
| Column | Type | Notes |
|
|
|---|---|---|
|
|
| `id` | uuid | Primary key |
|
|
| `name` | varchar | Unique role name |
|
|
| `description` | varchar | Optional description |
|
|
| `permissions` | jsonb | Array of permission strings |
|
|
| `createdAt` | timestamp | Creation time |
|
|
|
|
### audit_log
|
|
|
|
Security-relevant action log.
|
|
|
|
| Column | Type | Notes |
|
|
|---|---|---|
|
|
| `id` | uuid | Primary key |
|
|
| `userId` | uuid | FK to users |
|
|
| `action` | varchar | Action type |
|
|
| `details` | jsonb | Action-specific data |
|
|
| `createdAt` | timestamp | Action time |
|
|
|
|
## Migrations
|
|
|
|
Drizzle handles schema migrations. Migration files live in `apps/api/drizzle/`. During development:
|
|
|
|
```bash
|
|
cd apps/api
|
|
npx drizzle-kit generate # generate a migration from schema changes
|
|
npx drizzle-kit migrate # apply pending migrations
|
|
```
|
|
|
|
In production, pending migrations are applied automatically on startup.
|
|
|
|
## Backup and restore
|
|
|
|
The relational database lives in the Postgres container's `SnapOtter-pgdata` volume, not the app's `/data` volume.
|
|
|
|
**Option 1: pg_dump (recommended)**
|
|
|
|
```bash
|
|
# Dump the database while the stack is running
|
|
docker exec SnapOtter-postgres pg_dump -U snapotter snapotter > backup.sql
|
|
|
|
# Restore into a fresh database
|
|
cat backup.sql | docker exec -i SnapOtter-postgres psql -U snapotter snapotter
|
|
```
|
|
|
|
**Option 2: Volume snapshot**
|
|
|
|
```bash
|
|
# Stop the stack, then snapshot the pgdata volume
|
|
docker compose down
|
|
docker run --rm -v SnapOtter-pgdata:/data -v $(pwd)/backup:/backup \
|
|
alpine tar czf /backup/snapotter-pgdata.tar.gz -C /data .
|
|
```
|
|
|
|
### Migrating from 1.x (SQLite)
|
|
|
|
Upgrading from SnapOtter 1.x has its own guide: see [Upgrading from 1.x to 2.0](./upgrading). In short, reuse your existing `/data` volume and 2.0 auto-detects and imports `/data/snapotter.db` on first boot (or set `SQLITE_MIGRATE_PATH` to point at it explicitly). Back up the whole `/data` volume first, not just `snapotter.db`: 1.x uses SQLite WAL mode, so a stopped container often leaves most of its data in `snapotter.db-wal` beside an almost-empty `snapotter.db`.
|