fix(migrator): correct and harden the 1.x to 2.0 SQLite import (#434)

* feat(api): parse DATA_DIR from env for 1.x import auto-detection

Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w

* test(migrator): build 1.17.2 fixtures by replaying legacy migrations

Discovered the legacy migrations seed a Default team (0005) and builtin roles
(0007), so the replayed fixture carries them. Seed uses a distinct custom team.

Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w

* fix(migrator): self-adjusting column copy, jobs.status map, drop sessions, advisory lock

The importer now inserts only the intersection of source and live target columns,
so the three analytics_* columns 2.x dropped no longer break the first users INSERT
(and future dropped columns are handled generically). jobs.status is mapped onto the
2.x enum (error->failed). Sessions are no longer migrated. A pg_advisory_xact_lock
serializes concurrent replicas. Includes login-after-migrate and library assertions.

Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w

* test(migrator): CI drift guard fails when a required column is unfillable from 1.17.2

Introspects every NOT-NULL-no-default column of each migrated table in the current
schema and asserts the engine can fill it from a real 1.17.2 source. Turns a future
breaking schema change into a PR-time failure instead of a production import break.

Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w

* feat(migrator): orchestrator with detection, boot states, marker, blob count

sqlite-import.ts owns source resolution (explicit path, 'off' sentinel, DATA_DIR
probe), the four boot states (import/leftover/locked/none), the persisted
sqlite_import marker, and a read-only library-blob count. runBootImport wires them
together and catches TargetNonEmptyError as a benign multi-replica skip.

Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w

* feat(api): route boot through the 1.x import orchestrator; hide marker from non-admins

index.ts now calls runBootImport (which owns detection + the four boot states)
instead of the inline SQLITE_MIGRATE_PATH block. The sqlite_import marker is added
to SENSITIVE_KEYS (but not REDACTED_KEYS) so admins see the counts for the banner
while non-admins don't see the key at all.

Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w

* feat(migrator): add analyzeSqlite + dry-run/verify CLI

analyzeSqlite is a read-only pre-flight (no live Postgres): per-table row counts,
library-blob presence, and out-of-enum job statuses. The migrate:sqlite CLI now
lives in the orchestrator and supports --dry-run/--verify (prints the analysis and
exits without writing) alongside the existing import and --force.

Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w

* docs: add 1.x to 2.0 upgrade guide; fix volume-name casing

New apps/docs upgrade guide covering auto-detect, the SQLITE_MIGRATE_PATH override +
off opt-out, the dry-run, what carries over, locked-state recovery, and non-destructive
rollback. Leads with 'back up the WHOLE /data volume, not just snapotter.db' because
1.x WAL mode leaves data in snapotter.db-wal (surfaced by the real-image upgrade test).
Standardizes README/DOCKERHUB compose volume names on the canonical SnapOtter-data
casing so they match the repo compose and don't orphan an upgrader's volume.

Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w

* feat(web): admin 1.x migration banner + 21-locale strings

A one-time admin banner reads the sqlite_import marker from /v1/settings and shows
the import result (user + saved-file counts) on success, or a warning when a 1.x
database was found but not imported. Dismissal persists to a sqlite_import.dismissedAt
settings key. shouldShowMigrationBanner/parseMigrationMarker sit in feedback.ts with
the other shouldShow helpers; strings added to en.ts and all 20 other locales.

Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w

* style(landing): biome-format Hero.astro trustBadges array

Pre-existing formatting drift on main (its Lint check was skipped on the merge that
introduced it); this PR's full Lint run surfaced it. Formatting-only, applied via
the repo's own biome formatter to unblock the required Lint check.

Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w
This commit is contained in:
SnapOtter
2026-07-04 15:15:39 +00:00
committed by GitHub
parent dc589fd0a1
commit dadf766899
43 changed files with 1206 additions and 87 deletions
+7 -7
View File
@@ -51,7 +51,7 @@ services:
DATABASE_URL: postgres://snapotter:snapotter@postgres:5432/snapotter
REDIS_URL: redis://redis:6379
volumes:
- snapotter-data:/data
- SnapOtter-data:/data
depends_on: [postgres, redis]
restart: unless-stopped
postgres:
@@ -60,16 +60,16 @@ services:
POSTGRES_USER: snapotter
POSTGRES_PASSWORD: snapotter
POSTGRES_DB: snapotter
volumes: ["snapotter-pgdata:/var/lib/postgresql/data"]
volumes: ["SnapOtter-pgdata:/var/lib/postgresql/data"]
restart: unless-stopped
redis:
image: redis:8-alpine
volumes: ["snapotter-redisdata:/data"]
volumes: ["SnapOtter-redisdata:/data"]
restart: unless-stopped
volumes:
snapotter-data:
snapotter-pgdata:
snapotter-redisdata:
SnapOtter-data:
SnapOtter-pgdata:
SnapOtter-redisdata:
```
Then start the stack:
@@ -143,7 +143,7 @@ OIDC, SSO, S3 storage, and the full variable reference are documented in [Config
| `/data` | AI models and persistent user files; in single-container mode also the embedded PostgreSQL and Redis data. Back this up. |
| `/tmp/workspace` | Temporary processing files (auto-cleaned). |
In the Compose stack, PostgreSQL and Redis keep their own volumes (`snapotter-pgdata`, `snapotter-redisdata`).
In the Compose stack, PostgreSQL and Redis keep their own volumes (`SnapOtter-pgdata`, `SnapOtter-redisdata`).
## Ports
+6 -6
View File
@@ -70,7 +70,7 @@ services:
DATABASE_URL: postgres://snapotter:snapotter@postgres:5432/snapotter
REDIS_URL: redis://redis:6379
volumes:
- snapotter-data:/data
- SnapOtter-data:/data
depends_on: [postgres, redis]
restart: unless-stopped
postgres:
@@ -79,16 +79,16 @@ services:
POSTGRES_USER: snapotter
POSTGRES_PASSWORD: snapotter
POSTGRES_DB: snapotter
volumes: ["snapotter-pgdata:/var/lib/postgresql/data"]
volumes: ["SnapOtter-pgdata:/var/lib/postgresql/data"]
restart: unless-stopped
redis:
image: redis:8-alpine
volumes: ["snapotter-redisdata:/data"]
volumes: ["SnapOtter-redisdata:/data"]
restart: unless-stopped
volumes:
snapotter-data:
snapotter-pgdata:
snapotter-redisdata:
SnapOtter-data:
SnapOtter-pgdata:
SnapOtter-redisdata:
```
Then start the stack:
+1 -1
View File
@@ -10,7 +10,7 @@
"lint": "biome check src/",
"typecheck": "tsc --noEmit",
"clean": "rm -rf dist",
"migrate:sqlite": "tsx src/db/migrate-from-sqlite.ts"
"migrate:sqlite": "tsx src/db/sqlite-import.ts"
},
"dependencies": {
"@fastify/busboy": "^3.2.0",
+100 -41
View File
@@ -2,11 +2,17 @@ import { sql } from "drizzle-orm";
import { db } from "./index.js";
import { runMigrations } from "./migrate.js";
// Advisory lock: 7_421_xxx reserved for SnapOtter app locks (7_421_001 = schema migrate).
const SQLITE_IMPORT_LOCK_KEY = 7_421_002;
type SqliteRow = Record<string, unknown>;
export interface MigrationResult {
tables: Record<string, number>;
}
/** Thrown when the target became non-empty (e.g. another replica imported first). */
export class TargetNonEmptyError extends Error {}
// columns storing epoch-seconds integers in 1.x
const TS = new Set(["created_at", "updated_at", "expires_at", "completed_at", "last_used_at"]);
// columns storing 0/1 booleans in 1.x
@@ -20,13 +26,29 @@ const JSONB: Record<string, Set<string>> = {
audit_log: new Set(["details"]),
user_files: new Set(["tool_chain"]),
};
// FK-safe copy order
const TABLE_ORDER = [
// Map 1.x job status values onto the 2.x job_status enum. Anything unrecognized
// is coerced to "failed" so no single row can abort the transaction on an enum error.
const STATUS_MAP: Record<string, string> = {
queued: "queued",
processing: "processing",
running: "processing",
completed: "completed",
complete: "completed",
failed: "failed",
error: "failed",
canceled: "canceled",
cancelled: "canceled",
};
const VALID_STATUS = new Set(["queued", "processing", "completed", "failed", "canceled"]);
// FK-safe copy order. Sessions are intentionally NOT migrated (users re-auth once;
// credentials are unchanged so the same login works).
export const MIGRATED_TABLES = [
"users",
"teams",
"settings",
"roles",
"sessions",
"api_keys",
"pipelines",
"jobs",
@@ -34,32 +56,61 @@ const TABLE_ORDER = [
"user_files",
] as const;
// Column renames between 1.x and 2.x (source name -> target name). None today.
const RENAMES: Record<string, Record<string, string>> = {};
/**
* Target columns the engine can populate for `table` given the 1.x row's source
* columns. Used by the CI drift guard so it agrees with the actual copy logic.
*/
export function columnsEngineCanFill(table: string, sourceColumns: string[]): Set<string> {
const rename = RENAMES[table] ?? {};
const out = new Set(sourceColumns.map((c) => rename[c] ?? c));
if (table === "jobs") {
// jobs remap discards source file paths and produces these target columns:
out.delete("input_files");
out.delete("output_path");
out.add("input_refs");
out.add("output_refs");
// progress/error/status keep their source names.
}
return out;
}
function convertRow(table: string, row: SqliteRow): SqliteRow {
const out: SqliteRow = {};
for (const [col, raw] of Object.entries(row)) {
// Jobs table: remap removed 1.x columns to new spine columns
const rename = RENAMES[table] ?? {};
for (const [rawCol, raw] of Object.entries(row)) {
const col = rename[rawCol] ?? rawCol;
// Jobs table: remap removed/renamed 1.x columns to new spine columns.
if (table === "jobs") {
if (col === "input_files") {
if (rawCol === "input_files") {
// 1.x refs are dead workspace paths; discard content, store empty array
out.input_refs = [];
continue;
}
if (col === "output_path") {
if (rawCol === "output_path") {
// Replaced by output_refs; 1.x paths are dead
out.output_refs = [];
continue;
}
if (col === "progress") {
if (rawCol === "progress") {
// real 0-1 becomes jsonb {percent}
const p = typeof raw === "number" ? raw : 0;
out.progress = { percent: Math.round(p * 100) };
continue;
}
if (col === "error") {
if (rawCol === "error") {
// text becomes jsonb {message}
out.error = raw ? { message: String(raw) } : null;
continue;
}
if (rawCol === "status") {
// Map 1.x status onto the 2.x enum; unknown -> failed.
const mapped = STATUS_MAP[String(raw).toLowerCase()] ?? "failed";
out.status = VALID_STATUS.has(mapped) ? mapped : "failed";
continue;
}
}
if (raw === null || raw === undefined) {
@@ -83,6 +134,18 @@ function convertRow(table: string, row: SqliteRow): SqliteRow {
return out;
}
/** Live target columns for a public table (drizzle transaction handle). */
async function targetColumns(
tx: { execute: typeof db.execute },
table: string,
): Promise<Set<string>> {
const res = await tx.execute(
sql`SELECT column_name FROM information_schema.columns
WHERE table_schema = 'public' AND table_name = ${table}`,
);
return new Set(res.rows.map((r) => r.column_name as string));
}
export async function migrateFromSqlite(
sqlitePath: string,
opts: { force: boolean },
@@ -92,22 +155,38 @@ export async function migrateFromSqlite(
// so the CLI works standalone; do not remove.
await runMigrations();
const existing = await db.execute(sql`SELECT count(*)::int AS n FROM users`);
if ((existing.rows[0].n as number) > 0 && !opts.force) {
throw new Error(
"Target Postgres database is non-empty; refusing to migrate. Re-run with --force to attempt inserting 1.x rows into the existing database. This will FAIL and roll back if any primary key or unique value (username, team name, role name) collides with existing data.",
);
}
const sqlite = new Database(sqlitePath, { readonly: true, fileMustExist: true });
const result: MigrationResult = { tables: {} };
try {
await db.transaction(async (tx) => {
for (const table of TABLE_ORDER) {
const rows = sqlite.prepare(`SELECT * FROM ${table}`).all() as SqliteRow[];
// Serialize concurrent replicas: only one import proceeds; losers re-check below.
await tx.execute(sql`SELECT pg_advisory_xact_lock(${SQLITE_IMPORT_LOCK_KEY})`);
const existing = await tx.execute(sql`SELECT count(*)::int AS n FROM users`);
if ((existing.rows[0].n as number) > 0 && !opts.force) {
throw new TargetNonEmptyError(
"Target Postgres database is non-empty; refusing to migrate. Re-run with --force to attempt inserting 1.x rows into the existing database. This will FAIL and roll back if any primary key or unique value (username, team name, role name) collides with existing data.",
);
}
for (const table of MIGRATED_TABLES) {
let rows: SqliteRow[];
try {
rows = sqlite.prepare(`SELECT * FROM ${table}`).all() as SqliteRow[];
} catch (e) {
// Older-than-1.17.2 files may lack a table. Skip rather than fatal.
if (/no such table/i.test((e as Error).message)) {
result.tables[table] = 0;
continue;
}
throw e;
}
const target = await targetColumns(tx, table);
for (const row of rows) {
const converted = convertRow(table, row);
const cols = Object.keys(converted);
// Self-adjusting: insert only columns that exist in the live target, so a
// column 1.x has but 2.x dropped (analytics_*) is skipped generically.
const cols = Object.keys(converted).filter((c) => target.has(c));
const colList = sql.raw(cols.map((c) => `"${c}"`).join(", "));
const values = sql.join(
cols.map((c) => {
@@ -139,25 +218,5 @@ export async function migrateFromSqlite(
}
return result;
}
// CLI entry: pnpm --filter @snapotter/api migrate:sqlite -- <path> [--force]
const invokedDirectly = /migrate-from-sqlite\.[tj]s$/.test(process.argv[1] ?? "");
if (invokedDirectly) {
// pnpm forwards "--" as a literal arg; skip it and any flags to find the positional path
const args = process.argv.slice(2);
const path = args.find((a) => a !== "--" && !a.startsWith("--"));
const force = args.includes("--force");
if (!path) {
console.error("Usage: migrate-from-sqlite <path-to-1.x-sqlite-db> [--force]");
process.exit(1);
}
migrateFromSqlite(path, { force })
.then((r) => {
console.log("Migration complete:", JSON.stringify(r.tables));
process.exit(0);
})
.catch((err) => {
console.error("Migration FAILED (no partial state; transaction rolled back):", err.message);
process.exit(1);
});
}
// The CLI (including --dry-run) lives in sqlite-import.ts, the orchestrator that
// wraps this engine. `pnpm --filter @snapotter/api migrate:sqlite` runs that.
+240
View File
@@ -0,0 +1,240 @@
import { existsSync } from "node:fs";
import { join } from "node:path";
import { sql } from "drizzle-orm";
import { getSettingString, upsertSetting } from "../lib/settings-helpers.js";
import { db } from "./index.js";
import { MIGRATED_TABLES, migrateFromSqlite, TargetNonEmptyError } from "./migrate-from-sqlite.js";
export const IMPORT_MARKER_KEY = "sqlite_import";
export type ImportStatus = "completed" | "detected_locked";
export interface ImportMarker {
status: ImportStatus;
tables?: Record<string, number>;
blobs?: { present: number; missing: number };
sourcePath?: string;
at?: string;
}
export type BootState = "import" | "leftover" | "locked" | "none";
/** Resolve which SQLite file to consider. Explicit path wins; "off" disables; else probe DATA_DIR. */
export function resolveSource(env: {
SQLITE_MIGRATE_PATH: string;
DATA_DIR: string;
}): string | null {
const explicit = env.SQLITE_MIGRATE_PATH.trim();
if (explicit.toLowerCase() === "off") return null;
if (explicit) return explicit;
const probed = join(env.DATA_DIR, "snapotter.db");
return existsSync(probed) ? probed : null;
}
/**
* Decide the boot action from the three inputs. `import` runs the copy; `leftover`
* means we already imported (source file is a harmless leftover); `locked` means a
* source is present but the instance already has data, so we cannot auto-import.
*/
export function evaluateBootState(input: {
usersCount: number;
source: string | null;
marker: { status: ImportStatus } | null;
}): BootState {
if (!input.source) return "none";
if (input.marker?.status === "completed") return "leftover";
if (input.usersCount === 0) return "import";
return "locked";
}
/**
* Count how many user_files.stored_name blobs exist under filesStoragePath.
* Read-only; only ever called during an actual import or the CLI, never on
* leftover/locked boots (a large library must not stat-scan on every boot).
*/
export async function countLibraryBlobs(
sqlitePath: string,
filesStoragePath: string,
): Promise<{ present: number; missing: number }> {
const { default: Database } = await import("better-sqlite3");
const s = new Database(sqlitePath, { readonly: true });
let present = 0;
let missing = 0;
try {
const rows = s.prepare("SELECT stored_name FROM user_files").all() as Array<{
stored_name: string;
}>;
for (const r of rows) {
if (existsSync(join(filesStoragePath, r.stored_name))) present++;
else missing++;
}
} catch {
/* no user_files table on very old files */
} finally {
s.close();
}
return { present, missing };
}
const VALID_JOB_STATUS = new Set(["queued", "processing", "completed", "failed", "canceled"]);
export interface AnalyzeReport {
tables: Record<string, number>;
blobs: { present: number; missing: number };
badStatuses: string[];
}
/**
* Read-only pre-flight analysis of a 1.x SQLite file. Needs no live Postgres, so an
* operator can run it before starting the stack. Reports per-table row counts,
* library-blob presence, and any job status values outside the 2.x enum (which the
* importer maps to "failed"/"processing").
*/
export async function analyzeSqlite(
sqlitePath: string,
filesStoragePath: string,
): Promise<AnalyzeReport> {
const { default: Database } = await import("better-sqlite3");
const s = new Database(sqlitePath, { readonly: true, fileMustExist: true });
const tables: Record<string, number> = {};
const badStatuses = new Set<string>();
try {
for (const t of MIGRATED_TABLES) {
try {
tables[t] = (s.prepare(`SELECT count(*) AS n FROM ${t}`).get() as { n: number }).n;
} catch {
tables[t] = 0; // table absent in an older 1.x file
}
}
try {
const rows = s.prepare("SELECT DISTINCT status FROM jobs").all() as Array<{ status: string }>;
for (const r of rows) {
if (!VALID_JOB_STATUS.has(String(r.status).toLowerCase())) {
badStatuses.add(String(r.status));
}
}
} catch {
/* no jobs table */
}
} finally {
s.close();
}
const blobs = await countLibraryBlobs(sqlitePath, filesStoragePath);
return { tables, blobs, badStatuses: [...badStatuses] };
}
export async function getImportMarker(): Promise<ImportMarker | null> {
const raw = await getSettingString(IMPORT_MARKER_KEY, "");
if (!raw) return null;
try {
return JSON.parse(raw) as ImportMarker;
} catch {
return null;
}
}
async function writeImportMarker(marker: ImportMarker): Promise<void> {
await upsertSetting(IMPORT_MARKER_KEY, JSON.stringify(marker));
}
/**
* Boot entry point. Resolves the source, evaluates the state, and acts, keeping
* the "import before any seeding" ordering the caller relies on. Never deletes
* the source. Returns the resolved state for logging.
*/
export async function runBootImport(env: {
SQLITE_MIGRATE_PATH: string;
DATA_DIR: string;
FILES_STORAGE_PATH: string;
}): Promise<BootState> {
const source = resolveSource(env);
const marker = await getImportMarker();
const { rows } = await db.execute(sql`SELECT count(*)::int AS n FROM users`);
const state = evaluateBootState({ usersCount: rows[0].n as number, source, marker });
if (state === "import" && source) {
try {
const result = await migrateFromSqlite(source, { force: false });
const blobs = await countLibraryBlobs(source, env.FILES_STORAGE_PATH);
await writeImportMarker({
status: "completed",
tables: result.tables,
blobs,
sourcePath: source,
at: new Date().toISOString(),
});
console.log(
"Imported 1.x SQLite database:",
JSON.stringify({ tables: result.tables, blobs }),
);
} catch (err) {
if (err instanceof TargetNonEmptyError) {
// Another replica imported first; benign.
console.log("1.x import skipped: another instance populated the database first.");
return "leftover";
}
console.error(
`FATAL: 1.x SQLite import failed from ${source}: ${(err as Error).message}. No partial data was written.`,
);
process.exit(1);
}
} else if (state === "locked" && source) {
console.warn(
`WARNING: a 1.x database was found at ${source} but this instance already has data, so it was NOT imported. ` +
"Importing 1.x data requires an empty instance. See the upgrade guide.",
);
await writeImportMarker({ status: "detected_locked", sourcePath: source });
} else if (state === "leftover" && source) {
console.log(`1.x import already applied; you may delete ${source}.`);
}
return state;
}
// CLI: pnpm --filter @snapotter/api migrate:sqlite -- <path> [--dry-run|--verify] [--force]
const invokedDirectly = /sqlite-import\.[tj]s$/.test(process.argv[1] ?? "");
if (invokedDirectly) {
const args = process.argv.slice(2);
const cliPath = args.find((a) => a !== "--" && !a.startsWith("--"));
const dryRun = args.includes("--dry-run") || args.includes("--verify");
const force = args.includes("--force");
if (!cliPath) {
console.error("Usage: migrate:sqlite -- <path-to-1.x-sqlite-db> [--dry-run] [--force]");
process.exit(1);
}
const filesStoragePath = process.env.FILES_STORAGE_PATH || "./data/files";
if (dryRun) {
analyzeSqlite(cliPath, filesStoragePath)
.then((r) => {
const total = Object.values(r.tables).reduce((a, b) => a + b, 0);
console.log("1.x import dry run (no writes):");
console.log(" rows per table:", JSON.stringify(r.tables));
console.log(` library blobs: ${r.blobs.present} present, ${r.blobs.missing} missing`);
if (r.blobs.missing > 0) {
console.log(" WARNING: some saved-library files are missing from the files directory.");
}
if (r.badStatuses.length) {
console.log(
` job statuses to be mapped: ${r.badStatuses.join(", ")} -> failed/processing`,
);
}
console.log(` would import ${total} rows total (sessions are intentionally skipped).`);
process.exit(0);
})
.catch((err) => {
console.error("Dry run failed:", (err as Error).message);
process.exit(1);
});
} else {
migrateFromSqlite(cliPath, { force })
.then((r) => {
console.log("Migration complete:", JSON.stringify(r.tables));
process.exit(0);
})
.catch((err) => {
console.error(
"Migration FAILED (no partial state; transaction rolled back):",
(err as Error).message,
);
process.exit(1);
});
}
}
+10 -17
View File
@@ -103,23 +103,16 @@ try {
process.exit(1);
}
// Auto-import 1.x SQLite database on first boot (before default user creation)
if (env.SQLITE_MIGRATE_PATH) {
const { rows } = await db.execute(sql`SELECT count(*)::int AS n FROM users`);
if ((rows[0].n as number) === 0) {
try {
const { migrateFromSqlite } = await import("./db/migrate-from-sqlite.js");
const result = await migrateFromSqlite(env.SQLITE_MIGRATE_PATH, { force: false });
console.log("Imported 1.x SQLite database:", JSON.stringify(result.tables));
} catch (err) {
console.error(
`FATAL: 1.x SQLite import failed from ${env.SQLITE_MIGRATE_PATH}: ${(err as Error).message}. No partial data was written.`,
);
process.exit(1);
}
} else {
console.log("SQLITE_MIGRATE_PATH set but target is not empty; skipping import");
}
// Auto-import / detect a 1.x SQLite database on boot (before default user creation).
// The orchestrator owns detection (explicit path, "off" sentinel, or DATA_DIR probe),
// the four boot states, and the persisted marker. See db/sqlite-import.ts.
{
const { runBootImport } = await import("./db/sqlite-import.js");
await runBootImport({
SQLITE_MIGRATE_PATH: env.SQLITE_MIGRATE_PATH,
DATA_DIR: env.DATA_DIR,
FILES_STORAGE_PATH: env.FILES_STORAGE_PATH,
});
}
// Seed built-in roles (admin, editor, user) that legacy SQLite migrations
+1
View File
@@ -36,6 +36,7 @@ const envSchema = z
API_KEYS_RATE_LIMIT_PER_MIN: z.coerce.number().default(30),
DATABASE_URL: z.string().default("postgres://snapotter:snapotter@localhost:5432/snapotter"),
SQLITE_MIGRATE_PATH: z.string().default(""),
DATA_DIR: z.string().default("./data"),
FILES_STORAGE_PATH: z.string().default("./data/files"),
WORKSPACE_PATH: z.string().default("./tmp/workspace"),
DEFAULT_THEME: z.enum(["light", "dark", "system"]).default("light"),
+1
View File
@@ -27,6 +27,7 @@ const SENSITIVE_KEYS = new Set([
"scim_token_hash",
"siem_config",
"siem_webhook_auth",
"sqlite_import",
"webhook_destinations",
]);
+1
View File
@@ -134,6 +134,7 @@ export default defineConfig({
{ text: "SCIM Provisioning", link: "/guide/scim" },
{ text: "Users, Roles & Permissions", link: "/guide/users-roles" },
{ text: "Database", link: "/guide/database" },
{ text: "Upgrading from 1.x", link: "/guide/upgrading" },
{ text: "Deployment", link: "/guide/deployment" },
{ text: "Security & Hardening", link: "/guide/security" },
{ text: "What SnapOtter collects", link: "/guide/telemetry" },
+1 -1
View File
@@ -179,4 +179,4 @@ docker run --rm -v SnapOtter-pgdata:/data -v $(pwd)/backup:/backup \
### Migrating from 1.x (SQLite)
If you are upgrading from SnapOtter 1.x, set `SQLITE_MIGRATE_PATH` to the path of your old `snapotter.db` file on first boot. The migration runs once and imports users, settings, pipelines, and files into Postgres. Remove the variable after migration succeeds.
Upgrading from SnapOtter 1.x has its own guide: see [Upgrading from 1.x to 2.0](./upgrading). In short, reuse your existing `/data` volume and 2.0 auto-detects and imports `/data/snapotter.db` on first boot (or set `SQLITE_MIGRATE_PATH` to point at it explicitly). Back up the whole `/data` volume first, not just `snapotter.db`: 1.x uses SQLite WAL mode, so a stopped container often leaves most of its data in `snapotter.db-wal` beside an almost-empty `snapotter.db`.
+108
View File
@@ -0,0 +1,108 @@
# Upgrading from 1.x to 2.0
SnapOtter 1.x stored everything in a single SQLite file and ran as one container. SnapOtter 2.0 uses PostgreSQL and Redis. This guide walks through moving a 1.x install to 2.0 without losing data.
The short version: reuse your existing `/data` volume, and 2.0 imports your 1.x database automatically on first boot. Your users, saved files, settings, API keys, and pipelines come across. The old database is never modified, so you can always roll back.
## Before you start: back up the whole `/data` volume
Do this first, every time. Back up the **entire** `/data` volume, not just the `snapotter.db` file.
Here is why it matters. 1.x runs SQLite in WAL mode, so a stopped 1.x container routinely leaves most of its committed data in `snapotter.db-wal` beside an almost-empty `snapotter.db`. Copying only `snapotter.db` captures an empty database and silently loses everything. The volume carries `snapotter.db`, `snapotter.db-wal`, `snapotter.db-shm`, and your `files/` directory together, and they must travel as a set.
```bash
# Adjust the volume name to match yours (see "Check your volume name" below).
docker run --rm -v SnapOtter-data:/data -v "$PWD":/backup \
alpine tar czf /backup/snapotter-1x-data.tgz -C /data .
```
## Upgrade to 1.17.2 first
Upgrade your 1.x install to the latest 1.x release (1.17.2) before moving to 2.0. That lets 1.x run its own final schema migrations, so 2.0 imports from a known, complete schema. Upgrading from an older 1.x straight to 2.0 is not supported.
## Check your volume name
The importer only sees your data if the 2.0 stack mounts the same volume your 1.x install used. Docker volume names are case sensitive, and older README snippets used a lowercase `snapotter-data` while the Compose files use `SnapOtter-data`. Confirm which one you have:
```bash
docker volume ls | grep -i snapotter
```
Use that exact name in your 2.0 configuration.
## Path A: single container (quickest)
If you run SnapOtter with a single `docker run`, keep doing that. 2.0 boots an embedded PostgreSQL and Redis inside the container when you do not set `DATABASE_URL` or `REDIS_URL`, and it auto-detects and imports `/data/snapotter.db` on first boot.
```bash
docker run -d --name snapotter -p 1349:1349 \
-v SnapOtter-data:/data \
snapotter/snapotter:latest
```
Watch the logs for a line like:
```
Imported 1.x SQLite database: {"tables":{"users":2,"teams":1,...},"blobs":{"present":1,"missing":0}}
```
That is it. Log in with your existing credentials.
## Path B: Compose (recommended for production)
The 2.0 Compose stack runs three services (app, Postgres, Redis). Reuse your 1.x `/data` volume for the app service. The app auto-detects `/data/snapotter.db` and imports it into Postgres on first boot.
```yaml
services:
SnapOtter:
image: snapotter/snapotter:latest
volumes:
- SnapOtter-data:/data # your existing 1.x volume
- SnapOtter-workspace:/tmp/workspace
environment:
- DATABASE_URL=postgres://snapotter:snapotter@postgres:5432/snapotter
- REDIS_URL=redis://:snapotter@redis:6379
# ...
```
If you would rather point at the old database explicitly, set `SQLITE_MIGRATE_PATH=/data/snapotter.db`. An explicit path always wins over auto-detect.
## Preview the import first (optional)
To see exactly what would be imported without writing anything, run a dry run against your database file:
```bash
pnpm --filter @snapotter/api migrate:sqlite -- /path/to/snapotter.db --dry-run
```
It prints the row counts per table, how many saved-library files it found on disk, and any job statuses it will normalize. It needs no running Postgres.
## What carries over, and what does not
Carried over:
- Users, and the ability to log in. Password hashes are unchanged, so the same username and password work.
- Teams, settings (including your instance identity), roles, API keys (they keep working), and saved pipelines.
- Job history records.
- Your saved-file library, both the records and the actual files, because `/data/files` is preserved on the volume.
Not carried over:
- Login sessions. Everyone signs in once after the upgrade. Credentials are unchanged, so it is a single re-login, nothing more.
- The input and output files of old processing jobs. Those lived in a temporary workspace and are gone by design. The job history records remain.
- Per-user analytics-consent flags from 1.x, which have no 2.0 equivalent (2.0 analytics is an instance-level setting).
## Turning the import off
If you deliberately want a fresh database even though a `snapotter.db` is present on the volume, set `SQLITE_MIGRATE_PATH=off`.
## If you already have data in the 2.0 instance
The importer only runs into an empty database. If you started 2.0 fresh (creating data), then later mounted an old `snapotter.db`, 2.0 will detect it but will not import, because merging two datasets can collide on IDs. You will see a warning in the logs. To import the 1.x data you need an empty instance:
- If the 2.0 instance only holds the default admin (you have not really used it), stop the stack, remove the Postgres volume (`SnapOtter-pgdata`), and boot again with the old `/data` present. It will import cleanly. This wipes only the throwaway Postgres data, not your 1.x database.
- If the 2.0 instance holds real data you want to keep, the two datasets cannot be auto-merged. Export what you need and import the 1.x data into a separate fresh deployment.
## Rolling back
The upgrade never modifies or deletes your 1.x `snapotter.db`. If you need to go back to 1.x, redeploy the 1.x image against the same volume. Anything you created in 2.0 after the upgrade lives in Postgres and would not be in the 1.x database, so roll back promptly if you are going to.
+1 -6
View File
@@ -5,12 +5,7 @@ import CategoryCards from "./CategoryCards.astro";
import HeroSearch from "./HeroSearch.astro";
import TrustSignals from "./TrustSignals.astro";
const trustBadges = [
"Self-hosted",
"Open source",
"Air-gap capable",
"Compliance-friendly",
];
const trustBadges = ["Self-hosted", "Open source", "Air-gap capable", "Compliance-friendly"];
---
<section class="relative overflow-hidden px-6 pt-28 pb-12 md:pt-32 md:pb-16">
+2
View File
@@ -4,6 +4,7 @@ import { BrowserRouter, Navigate, Route, Routes, useLocation } from "react-route
import { Toaster, toast } from "sonner";
import { ConnectionMonitor } from "./components/common/connection-monitor";
import { KeyboardShortcutProvider } from "./components/common/keyboard-shortcut-provider";
import { MigrationBanner } from "./components/common/migration-banner";
import { RouteAnnouncer } from "./components/common/route-announcer";
import { UsageSurveyOverlay } from "./components/onboarding/usage-survey-overlay";
import { I18nProvider } from "./contexts/i18n-context";
@@ -193,6 +194,7 @@ export function App() {
<RouteAnnouncer />
<KeyboardShortcutProvider>
<AuthGuard>
<MigrationBanner />
<UsageSurveyOverlay />
<Suspense fallback={<PageLoader />}>
<Routes>
@@ -0,0 +1,95 @@
import { CheckCircle2, TriangleAlert, X } from "lucide-react";
import { useEffect, useState } from "react";
import { useTranslation } from "@/contexts/i18n-context";
import { useAuth } from "@/hooks/use-auth";
import { apiGet, apiPut } from "@/lib/api";
import { parseMigrationMarker, shouldShowMigrationBanner } from "@/lib/feedback";
import { format } from "@/lib/format";
import { withTimeout } from "@/lib/with-timeout";
// A hung write must not leave the dismiss button stuck; time it out so the admin
// can retry (worst case the banner reappears next load, which is harmless).
const WRITE_TIMEOUT_MS = 15_000;
/**
* One-time admin banner announcing the result of a 1.x SQLite import (or warning
* that a 1.x database was found but not imported). Reads the `sqlite_import`
* marker from the settings payload and persists dismissal to a settings key,
* mirroring the usage-survey overlay pattern.
*/
export function MigrationBanner() {
const { t } = useTranslation();
const { role } = useAuth();
const [settings, setSettings] = useState<Record<string, string> | null>(null);
const [dismissing, setDismissing] = useState(false);
useEffect(() => {
if (role !== "admin") return;
apiGet<{ settings: Record<string, string> }>("/v1/settings")
.then((data) => setSettings(data.settings))
.catch(() => {
// Fail closed: no settings means we cannot know the import state, so we
// simply do not show the banner for this load.
});
}, [role]);
if (settings === null || !shouldShowMigrationBanner({ settings, role })) return null;
const marker = parseMigrationMarker(settings.sqlite_import);
if (!marker) return null;
const locked = marker.status === "detected_locked";
const users = marker.tables?.users ?? 0;
const files = marker.tables?.user_files ?? 0;
async function dismiss() {
if (dismissing) return;
setDismissing(true);
const value = new Date().toISOString();
try {
await withTimeout(
apiPut("/v1/settings", { "sqlite_import.dismissedAt": value }),
WRITE_TIMEOUT_MS,
);
setSettings((current) => ({ ...(current ?? {}), "sqlite_import.dismissedAt": value }));
} catch {
// Leave it visible next load; a failed dismiss is low-stakes.
} finally {
setDismissing(false);
}
}
return (
<div
role="status"
aria-live="polite"
className={`fixed top-0 left-0 right-0 z-[55] flex items-center justify-center gap-3 px-4 py-2 text-sm font-medium ${
locked
? "bg-amber-500 text-amber-950 dark:bg-amber-600 dark:text-amber-50"
: "bg-emerald-500 text-emerald-950 dark:bg-emerald-600 dark:text-emerald-50"
}`}
>
{locked ? (
<TriangleAlert className="h-4 w-4 shrink-0" aria-hidden="true" />
) : (
<CheckCircle2 className="h-4 w-4 shrink-0" aria-hidden="true" />
)}
<span className="text-center">
<span className="font-semibold">
{locked ? t.migrationBanner.warningTitle : t.migrationBanner.successTitle}
</span>{" "}
{locked
? t.migrationBanner.warningBody
: format(t.migrationBanner.successBody, { users, files })}
</span>
<button
type="button"
onClick={dismiss}
disabled={dismissing}
aria-label={t.migrationBanner.dismiss}
className="shrink-0 rounded p-0.5 hover:bg-black/10 disabled:opacity-50"
>
<X className="h-4 w-4" aria-hidden="true" />
</button>
</div>
);
}
+34
View File
@@ -152,6 +152,40 @@ export function shouldShowUsageSurvey({
);
}
export interface MigrationMarker {
status: "completed" | "detected_locked";
tables?: Record<string, number>;
blobs?: { present: number; missing: number };
}
/** Parse the `sqlite_import` settings marker written by the 1.x import on boot. */
export function parseMigrationMarker(raw: string | undefined): MigrationMarker | null {
if (!raw) return null;
try {
const marker = JSON.parse(raw) as MigrationMarker;
return marker.status === "completed" || marker.status === "detected_locked" ? marker : null;
} catch {
return null;
}
}
/**
* Show the 1.x migration banner to admins when an import marker exists and has
* not been dismissed. The marker key is admin-only (SENSITIVE_KEYS on the API),
* so non-admins never receive it, and we gate on role here as well.
*/
export function shouldShowMigrationBanner({
settings,
role,
}: {
settings: Record<string, string>;
role: string | null;
}): boolean {
if (role !== "admin") return false;
if (settings["sqlite_import.dismissedAt"]) return false;
return parseMigrationMarker(settings.sqlite_import) !== null;
}
export async function submitFeedback(payload: FeedbackPayload): Promise<FeedbackResponse> {
return apiPost<FeedbackResponse>("/v1/feedback", payload);
}
+8
View File
@@ -1,6 +1,14 @@
import type { TranslationKeys } from "./en.js";
export const ar: TranslationKeys = {
migrationBanner: {
successTitle: "تم استيراد بيانات SnapOtter 1.x.",
successBody: "تم نقل {users} مستخدمين و {files} ملفات محفوظة من إصدارك السابق.",
warningTitle: "تم العثور على قاعدة بيانات 1.x ولكن لم يتم استيرادها.",
warningBody:
"يحتوي هذا المثيل على بيانات بالفعل، لذلك لم يتم استيراد قاعدة بيانات 1.x. يتطلب الاستيراد مثيلاً فارغاً. راجع دليل الترقية.",
dismiss: "إغلاق",
},
common: {
upload: "رفع من الجهاز",
process: "معالجة",
+9
View File
@@ -1,6 +1,15 @@
import type { TranslationKeys } from "./en.js";
export const de: TranslationKeys = {
migrationBanner: {
successTitle: "SnapOtter 1.x-Daten importiert.",
successBody:
"{users} Benutzer und {files} gespeicherte Dateien aus deiner vorherigen Version übernommen.",
warningTitle: "1.x-Datenbank gefunden, aber nicht importiert.",
warningBody:
"Diese Instanz enthält bereits Daten, daher wurde deine 1.x-Datenbank nicht importiert. Der Import erfordert eine leere Instanz. Siehe die Upgrade-Anleitung.",
dismiss: "Schließen",
},
common: {
upload: "Vom Computer hochladen",
process: "Verarbeiten",
+8
View File
@@ -1,4 +1,12 @@
export const en = {
migrationBanner: {
successTitle: "SnapOtter 1.x data imported.",
successBody: "Brought over {users} users and {files} saved files from your previous version.",
warningTitle: "1.x database found but not imported.",
warningBody:
"This instance already has data, so your 1.x database was not imported. Importing requires an empty instance. See the upgrade guide.",
dismiss: "Dismiss",
},
common: {
upload: "Upload from computer",
process: "Process",
+9
View File
@@ -1,6 +1,15 @@
import type { TranslationKeys } from "./en.js";
export const es: TranslationKeys = {
migrationBanner: {
successTitle: "Datos de SnapOtter 1.x importados.",
successBody:
"Se migraron {users} usuarios y {files} archivos guardados de tu versión anterior.",
warningTitle: "Se encontró una base de datos 1.x, pero no se importó.",
warningBody:
"Esta instancia ya tiene datos, por lo que tu base de datos 1.x no se importó. La importación requiere una instancia vacía. Consulta la guía de actualización.",
dismiss: "Descartar",
},
common: {
upload: "Subir desde la computadora",
process: "Procesar",
+9
View File
@@ -1,6 +1,15 @@
import type { TranslationKeys } from "./en.js";
export const fr: TranslationKeys = {
migrationBanner: {
successTitle: "Données SnapOtter 1.x importées.",
successBody:
"{users} utilisateurs et {files} fichiers enregistrés ont été migrés depuis votre version précédente.",
warningTitle: "Base de données 1.x détectée, mais non importée.",
warningBody:
"Cette instance contient déjà des données, votre base de données 1.x n'a donc pas été importée. L'importation nécessite une instance vide. Consultez le guide de mise à niveau.",
dismiss: "Ignorer",
},
common: {
upload: "Importer depuis l'ordinateur",
process: "Traiter",
+8
View File
@@ -1,6 +1,14 @@
import type { TranslationKeys } from "./en.js";
export const hi: TranslationKeys = {
migrationBanner: {
successTitle: "SnapOtter 1.x डेटा आयात किया गया।",
successBody: "आपके पिछले संस्करण से {users} उपयोगकर्ता और {files} सहेजी गई फ़ाइलें स्थानांतरित की गईं।",
warningTitle: "1.x डेटाबेस मिला, लेकिन आयात नहीं किया गया।",
warningBody:
"इस इंस्टेंस में पहले से डेटा है, इसलिए आपका 1.x डेटाबेस आयात नहीं किया गया। आयात के लिए खाली इंस्टेंस आवश्यक है। अपग्रेड गाइड देखें।",
dismiss: "बंद करें",
},
common: {
upload: "कंप्यूटर से अपलोड करें",
process: "प्रोसेस करें",
+8
View File
@@ -1,6 +1,14 @@
import type { TranslationKeys } from "./en.js";
export const id: TranslationKeys = {
migrationBanner: {
successTitle: "Data SnapOtter 1.x diimpor.",
successBody: "Memindahkan {users} pengguna dan {files} file tersimpan dari versi sebelumnya.",
warningTitle: "Database 1.x ditemukan, tetapi tidak diimpor.",
warningBody:
"Instans ini sudah memiliki data, jadi database 1.x Anda tidak diimpor. Impor memerlukan instans kosong. Lihat panduan peningkatan.",
dismiss: "Tutup",
},
common: {
upload: "Unggah dari komputer",
process: "Proses",
+8
View File
@@ -1,6 +1,14 @@
import type { TranslationKeys } from "./en.js";
export const it: TranslationKeys = {
migrationBanner: {
successTitle: "Dati di SnapOtter 1.x importati.",
successBody: "Migrati {users} utenti e {files} file salvati dalla versione precedente.",
warningTitle: "Database 1.x rilevato, ma non importato.",
warningBody:
"Questa istanza contiene già dati, quindi il database 1.x non è stato importato. L'importazione richiede un'istanza vuota. Consulta la guida all'aggiornamento.",
dismiss: "Ignora",
},
common: {
upload: "Carica dal computer",
process: "Elabora",
+9
View File
@@ -1,6 +1,15 @@
import type { TranslationKeys } from "./en.js";
export const ja: TranslationKeys = {
migrationBanner: {
successTitle: "SnapOtter 1.x のデータをインポートしました。",
successBody:
"以前のバージョンから {users} 人のユーザーと {files} 件の保存済みファイルを移行しました。",
warningTitle: "1.x データベースが見つかりましたが、インポートされていません。",
warningBody:
"このインスタンスには既にデータがあるため、1.x データベースはインポートされませんでした。インポートには空のインスタンスが必要です。アップグレードガイドをご覧ください。",
dismiss: "閉じる",
},
common: {
upload: "パソコンからアップロード",
process: "処理",
+8
View File
@@ -1,6 +1,14 @@
import type { TranslationKeys } from "./en.js";
export const ko: TranslationKeys = {
migrationBanner: {
successTitle: "SnapOtter 1.x 데이터를 가져왔습니다.",
successBody: "이전 버전에서 사용자 {users}명과 저장된 파일 {files}개를 이전했습니다.",
warningTitle: "1.x 데이터베이스를 찾았지만 가져오지 않았습니다.",
warningBody:
"이 인스턴스에 이미 데이터가 있어 1.x 데이터베이스를 가져오지 않았습니다. 가져오려면 빈 인스턴스가 필요합니다. 업그레이드 가이드를 참조하세요.",
dismiss: "닫기",
},
common: {
upload: "컴퓨터에서 업로드",
process: "처리",
+9
View File
@@ -1,6 +1,15 @@
import type { TranslationKeys } from "./en.js";
export const nl: TranslationKeys = {
migrationBanner: {
successTitle: "SnapOtter 1.x-gegevens geïmporteerd.",
successBody:
"{users} gebruikers en {files} opgeslagen bestanden overgezet vanuit je vorige versie.",
warningTitle: "1.x-database gevonden, maar niet geïmporteerd.",
warningBody:
"Deze instantie bevat al gegevens, dus je 1.x-database is niet geïmporteerd. Voor importeren is een lege instantie nodig. Zie de upgradehandleiding.",
dismiss: "Sluiten",
},
common: {
upload: "Upload van computer",
process: "Verwerken",
+9
View File
@@ -1,6 +1,15 @@
import type { TranslationKeys } from "./en.js";
export const pl: TranslationKeys = {
migrationBanner: {
successTitle: "Zaimportowano dane SnapOtter 1.x.",
successBody:
"Przeniesiono {users} użytkowników i {files} zapisanych plików z poprzedniej wersji.",
warningTitle: "Znaleziono bazę danych 1.x, ale jej nie zaimportowano.",
warningBody:
"Ta instancja zawiera już dane, więc baza danych 1.x nie została zaimportowana. Import wymaga pustej instancji. Zobacz przewodnik aktualizacji.",
dismiss: "Zamknij",
},
common: {
upload: "Prześlij z komputera",
process: "Przetwórz",
+8
View File
@@ -1,6 +1,14 @@
import type { TranslationKeys } from "./en.js";
export const ptBR: TranslationKeys = {
migrationBanner: {
successTitle: "Dados do SnapOtter 1.x importados.",
successBody: "Migramos {users} usuários e {files} arquivos salvos da sua versão anterior.",
warningTitle: "Banco de dados 1.x encontrado, mas não importado.",
warningBody:
"Esta instância já tem dados, então seu banco de dados 1.x não foi importado. A importação exige uma instância vazia. Consulte o guia de atualização.",
dismiss: "Dispensar",
},
common: {
upload: "Enviar do computador",
process: "Processar",
+9
View File
@@ -1,6 +1,15 @@
import type { TranslationKeys } from "./en.js";
export const ru: TranslationKeys = {
migrationBanner: {
successTitle: "Данные SnapOtter 1.x импортированы.",
successBody:
"Перенесено пользователей: {users}, сохранённых файлов: {files} из предыдущей версии.",
warningTitle: "База данных 1.x найдена, но не импортирована.",
warningBody:
"В этом экземпляре уже есть данные, поэтому база данных 1.x не была импортирована. Для импорта нужен пустой экземпляр. См. руководство по обновлению.",
dismiss: "Закрыть",
},
common: {
upload: "Загрузить с компьютера",
process: "Обработать",
+8
View File
@@ -1,6 +1,14 @@
import type { TranslationKeys } from "./en.js";
export const sv: TranslationKeys = {
migrationBanner: {
successTitle: "SnapOtter 1.x-data importerad.",
successBody: "Överförde {users} användare och {files} sparade filer från din tidigare version.",
warningTitle: "1.x-databas hittades men importerades inte.",
warningBody:
"Den här instansen har redan data, så din 1.x-databas importerades inte. Import kräver en tom instans. Se uppgraderingsguiden.",
dismiss: "Stäng",
},
common: {
upload: "Ladda upp från dator",
process: "Bearbeta",
+8
View File
@@ -1,6 +1,14 @@
import type { TranslationKeys } from "./en.js";
export const th: TranslationKeys = {
migrationBanner: {
successTitle: "นำเข้าข้อมูล SnapOtter 1.x แล้ว",
successBody: "ย้าย {users} ผู้ใช้และ {files} ไฟล์ที่บันทึกไว้จากเวอร์ชันก่อนหน้าของคุณ",
warningTitle: "พบฐานข้อมูล 1.x แต่ยังไม่ได้นำเข้า",
warningBody:
"อินสแตนซ์นี้มีข้อมูลอยู่แล้ว จึงไม่ได้นำเข้าฐานข้อมูล 1.x ของคุณ การนำเข้าต้องใช้อินสแตนซ์ที่ว่างเปล่า ดูคู่มือการอัปเกรด",
dismiss: "ปิด",
},
common: {
upload: "อัปโหลดจากคอมพิวเตอร์",
process: "ประมวลผล",
+8
View File
@@ -1,6 +1,14 @@
import type { TranslationKeys } from "./en.js";
export const tr: TranslationKeys = {
migrationBanner: {
successTitle: "SnapOtter 1.x verileri içe aktarıldı.",
successBody: "Önceki sürümünüzden {users} kullanıcı ve {files} kayıtlı dosya taşındı.",
warningTitle: "1.x veritabanı bulundu ancak içe aktarılmadı.",
warningBody:
"Bu örnekte zaten veri var, bu yüzden 1.x veritabanınız içe aktarılmadı. İçe aktarma boş bir örnek gerektirir. Yükseltme kılavuzuna bakın.",
dismiss: "Kapat",
},
common: {
upload: "Bilgisayardan yükle",
process: "İşle",
+9
View File
@@ -1,6 +1,15 @@
import type { TranslationKeys } from "./en.js";
export const uk: TranslationKeys = {
migrationBanner: {
successTitle: "Дані SnapOtter 1.x імпортовано.",
successBody:
"Перенесено {users} користувачів і {files} збережених файлів із попередньої версії.",
warningTitle: "Базу даних 1.x знайдено, але не імпортовано.",
warningBody:
"Цей екземпляр уже містить дані, тому базу даних 1.x не було імпортовано. Для імпорту потрібен порожній екземпляр. Див. посібник з оновлення.",
dismiss: "Закрити",
},
common: {
upload: "Завантажити з комп'ютера",
process: "Обробити",
+8
View File
@@ -1,6 +1,14 @@
import type { TranslationKeys } from "./en.js";
export const vi: TranslationKeys = {
migrationBanner: {
successTitle: "Đã nhập dữ liệu SnapOtter 1.x.",
successBody: "Đã chuyển {users} người dùng và {files} tệp đã lưu từ phiên bản trước của bạn.",
warningTitle: "Đã tìm thấy cơ sở dữ liệu 1.x nhưng chưa nhập.",
warningBody:
"Phiên bản này đã có dữ liệu nên cơ sở dữ liệu 1.x của bạn chưa được nhập. Việc nhập yêu cầu một phiên bản trống. Xem hướng dẫn nâng cấp.",
dismiss: "Đóng",
},
common: {
upload: "Tải lên từ máy tính",
process: "Xử lý",
+7
View File
@@ -1,6 +1,13 @@
import type { TranslationKeys } from "./en.js";
export const zhCN: TranslationKeys = {
migrationBanner: {
successTitle: "已导入 SnapOtter 1.x 数据。",
successBody: "已从旧版本迁移 {users} 个用户和 {files} 个已保存文件。",
warningTitle: "发现 1.x 数据库,但未导入。",
warningBody: "此实例已有数据,因此未导入你的 1.x 数据库。导入需要一个空实例。请参阅升级指南。",
dismiss: "关闭",
},
common: {
upload: "从电脑上传",
process: "处理",
+8
View File
@@ -1,6 +1,14 @@
import type { TranslationKeys } from "./en.js";
export const zhTW: TranslationKeys = {
migrationBanner: {
successTitle: "已匯入 SnapOtter 1.x 資料。",
successBody: "已從舊版本移轉 {users} 位使用者與 {files} 個已儲存檔案。",
warningTitle: "偵測到 1.x 資料庫,但未匯入。",
warningBody:
"此執行個體已有資料,因此未匯入您的 1.x 資料庫。匯入需要空白的執行個體。請參閱升級指南。",
dismiss: "關閉",
},
common: {
upload: "從電腦上傳",
process: "處理",
+111
View File
@@ -0,0 +1,111 @@
import { readdirSync, readFileSync } from "node:fs";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
import Database from "better-sqlite3";
import { hashPassword } from "../../apps/api/src/plugins/auth.js";
const LEGACY_DIR = join(
dirname(fileURLToPath(import.meta.url)),
"../../apps/api/drizzle-sqlite-legacy",
);
/**
* Rebuild the final 1.17.2 SQLite schema by replaying every archived legacy
* migration in filename order. Drizzle separates statements with the marker
* "--> statement-breakpoint"; each chunk is a single statement, so prepare().run()
* applies it. Standalone comment lines (some migrations, e.g. 0012, prefix a
* statement with them) are stripped first. Using the real migrations (not a
* hand-written schema) guarantees the fixture matches what a real 1.x instance
* has, including columns 2.x later dropped.
*/
export function buildLegacySqlite(path: string): void {
const files = readdirSync(LEGACY_DIR)
.filter((f) => f.endsWith(".sql"))
.sort();
const s = new Database(path);
try {
s.pragma("foreign_keys = OFF");
for (const file of files) {
const sql = readFileSync(join(LEGACY_DIR, file), "utf8");
for (const chunk of sql.split("--> statement-breakpoint")) {
// Drop standalone comment lines, trailing semicolon, and surrounding whitespace.
const stmt = chunk
.replace(/^\s*--.*$/gm, "")
.trim()
.replace(/;\s*$/, "");
if (!stmt) continue;
s.prepare(stmt).run();
}
}
} finally {
s.close();
}
}
/**
* Insert a small but representative 1.17.2 dataset: a user with a REAL scrypt
* password hash (so login-after-migrate can be verified), populated analytics
* columns (the drop-me case), a saved file, a pipeline, jobs whose statuses
* include the out-of-2.x-enum value "error", and an unexpired session (to prove
* sessions are NOT copied). Returns the plaintext password and admin id to assert on.
*/
export async function seedRealistic1xData(
path: string,
): Promise<{ password: string; adminId: string }> {
const password = "correct horse battery staple";
const passwordHash = await hashPassword(password);
const now = 1748000000; // epoch seconds, as 1.x stored
const s = new Database(path);
try {
s.prepare(
`INSERT INTO users (id, username, password_hash, role, team, must_change_password,
auth_provider, external_id, email, created_at, updated_at,
analytics_enabled, analytics_consent_shown_at, analytics_consent_remind_at)
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)`,
).run(
"u-admin",
"admin",
passwordHash,
"admin",
"Default",
0,
"local",
null,
"admin@example.com",
now,
now,
1,
now,
null,
);
// The "Default" team is already seeded by legacy migration 0005; add a
// distinct custom team so the copy exercises a non-default row too.
s.prepare("INSERT INTO teams (id, name, created_at) VALUES (?,?,?)").run(
"tm-eng",
"Engineering",
now,
);
s.prepare(
"INSERT INTO pipelines (id, user_id, name, steps, created_at) VALUES (?,?,?,?,?)",
).run("p-1", "u-admin", "Shrink", '[{"toolId":"compress","settings":{"quality":70}}]', now);
// job with an out-of-enum status "error" (must map to "failed")
s.prepare(
"INSERT INTO jobs (id, type, status, progress, input_files, created_at) VALUES (?,?,?,?,?,?)",
).run("j-err", "single", "error", 0.5, '["a.png"]', now);
s.prepare(
"INSERT INTO jobs (id, type, status, progress, input_files, created_at, completed_at) VALUES (?,?,?,?,?,?,?)",
).run("j-ok", "single", "completed", 1.0, '["b.png"]', now, now);
s.prepare(
"INSERT INTO user_files (id, user_id, original_name, stored_name, mime_type, size, version, created_at) VALUES (?,?,?,?,?,?,?,?)",
).run("uf-1", "u-admin", "photo.png", "abc123.png", "image/png", 1024, 1, now);
s.prepare("INSERT INTO sessions (id, user_id, expires_at, created_at) VALUES (?,?,?,?)").run(
"ses-1",
"u-admin",
4102444800,
now,
); // expires year 2100
} finally {
s.close();
}
return { password, adminId: "u-admin" };
}
@@ -5,7 +5,11 @@ import Database from "better-sqlite3";
import { sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { db } from "../../../apps/api/src/db/index.js";
import { migrateFromSqlite } from "../../../apps/api/src/db/migrate-from-sqlite.js";
import {
MIGRATED_TABLES,
migrateFromSqlite,
} from "../../../apps/api/src/db/migrate-from-sqlite.js";
import { buildLegacySqlite, seedRealistic1xData } from "../../helpers/legacy-sqlite-fixture.js";
function buildFixtureSqlite(path: string): void {
const s = new Database(path);
@@ -456,7 +460,6 @@ describe("migrate-from-sqlite (representative 1.x database)", () => {
expect(result.tables.teams).toBe(3);
expect(result.tables.settings).toBe(3);
expect(result.tables.roles).toBe(2);
expect(result.tables.sessions).toBe(2);
expect(result.tables.api_keys).toBe(2);
expect(result.tables.pipelines).toBe(2);
expect(result.tables.jobs).toBe(4);
@@ -580,10 +583,57 @@ describe("migrate-from-sqlite (representative 1.x database)", () => {
expect(maxUpload?.value).toBe("50");
});
it("sessions: id_token null and non-null", async () => {
const [ses1] = (await db.execute(sql`SELECT * FROM sessions WHERE id = 'ses-1'`)).rows;
expect(ses1.id_token).toBeNull();
const [ses2] = (await db.execute(sql`SELECT * FROM sessions WHERE id = 'ses-2'`)).rows;
expect(ses2.id_token).toBe("eyJhbGciOiJSUzI1NiJ9.fake-jwt-token");
// Note: sessions are intentionally NOT migrated (see the real-1.17.2 suite below).
});
describe("migrate-from-sqlite (real 1.17.2 schema)", () => {
const realDir = mkdtempSync(join(tmpdir(), "snapotter-migrator-real-"));
const realPath = join(realDir, "real-1x.db");
let seeded: { password: string; adminId: string };
beforeAll(async () => {
buildLegacySqlite(realPath);
seeded = await seedRealistic1xData(realPath);
await db.execute(
sql`TRUNCATE user_files, audit_log, jobs, pipelines, api_keys, sessions, roles, settings, teams, users CASCADE`,
);
});
afterAll(async () => {
await db.execute(
sql`TRUNCATE user_files, audit_log, jobs, pipelines, api_keys, sessions, roles, settings, teams, users CASCADE`,
);
});
it("excludes sessions from the migrated set", () => {
expect(MIGRATED_TABLES).not.toContain("sessions");
});
it("imports a real 1.17.2 database (analytics columns do not break it)", async () => {
const result = await migrateFromSqlite(realPath, { force: false });
expect(result.tables.users).toBe(1);
expect(result.tables).not.toHaveProperty("sessions");
const [u] = (await db.execute(sql`SELECT * FROM users WHERE id = 'u-admin'`)).rows;
expect(u.username).toBe("admin");
expect(u).not.toHaveProperty("analytics_enabled");
});
it("maps the out-of-enum job status 'error' to 'failed'", async () => {
const [j] = (await db.execute(sql`SELECT status FROM jobs WHERE id = 'j-err'`)).rows;
expect(j.status).toBe("failed");
});
it("does not copy the sessions table", async () => {
const { rows } = await db.execute(sql`SELECT count(*)::int AS n FROM sessions`);
expect(rows[0].n).toBe(0);
});
it("migrated user can still log in and the library row is intact", async () => {
const { verifyPassword } = await import("../../../apps/api/src/plugins/auth.js");
const [u] = (
await db.execute(sql`SELECT password_hash FROM users WHERE id = ${seeded.adminId}`)
).rows;
expect(await verifyPassword(seeded.password, u.password_hash as string)).toBe(true);
const [f] = (await db.execute(sql`SELECT stored_name FROM user_files WHERE id = 'uf-1'`)).rows;
expect(f.stored_name).toBe("abc123.png");
});
});
@@ -0,0 +1,59 @@
import { mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import Database from "better-sqlite3";
import { sql } from "drizzle-orm";
import { beforeAll, describe, expect, it } from "vitest";
import { db } from "../../../apps/api/src/db/index.js";
import { runMigrations } from "../../../apps/api/src/db/migrate.js";
import {
columnsEngineCanFill,
MIGRATED_TABLES,
} from "../../../apps/api/src/db/migrate-from-sqlite.js";
import { buildLegacySqlite } from "../../helpers/legacy-sqlite-fixture.js";
describe("migrator schema drift guard", () => {
let sourceColumns: Record<string, string[]>;
beforeAll(async () => {
await runMigrations(); // bring the test DB to the current schema
const dir = mkdtempSync(join(tmpdir(), "drift-"));
const path = join(dir, "legacy.db");
buildLegacySqlite(path);
const s = new Database(path, { readonly: true });
sourceColumns = {};
for (const table of MIGRATED_TABLES) {
const info = s.prepare(`PRAGMA table_info(${table})`).all() as Array<{ name: string }>;
sourceColumns[table] = info.map((c) => c.name);
}
s.close();
});
it("every required target column is fillable from a real 1.17.2 source", async () => {
const unfillable: string[] = [];
for (const table of MIGRATED_TABLES) {
const required = await db.execute(
sql`SELECT column_name FROM information_schema.columns
WHERE table_schema = 'public' AND table_name = ${table}
AND is_nullable = 'NO' AND column_default IS NULL`,
);
const canFill = columnsEngineCanFill(table, sourceColumns[table] ?? []);
for (const row of required.rows) {
const col = row.column_name as string;
if (!canFill.has(col)) unfillable.push(`${table}.${col}`);
}
}
// If this fails, a schema change added a required column the 1.x importer
// cannot populate. Give it a default, make it nullable, or add a computed
// default / rename mapping in migrate-from-sqlite.ts.
expect(unfillable).toEqual([]);
});
it("the guard flags a column the engine cannot fill (self-check)", () => {
const canFill = columnsEngineCanFill("users", ["id", "username"]);
expect(canFill.has("username")).toBe(true);
// A hypothetical future required column absent from the 1.17.2 source is not
// fillable, so the drift check above would report it and fail CI.
expect(canFill.has("some_future_required_col")).toBe(false);
});
});
@@ -0,0 +1,81 @@
import { mkdirSync, mkdtempSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import {
analyzeSqlite,
countLibraryBlobs,
evaluateBootState,
resolveSource,
} from "../../../apps/api/src/db/sqlite-import.js";
import { buildLegacySqlite, seedRealistic1xData } from "../../helpers/legacy-sqlite-fixture.js";
describe("resolveSource", () => {
it("returns null for the 'off' sentinel", () => {
expect(resolveSource({ SQLITE_MIGRATE_PATH: "off", DATA_DIR: "/data" })).toBeNull();
expect(resolveSource({ SQLITE_MIGRATE_PATH: "OFF", DATA_DIR: "/data" })).toBeNull();
});
it("prefers an explicit path", () => {
expect(resolveSource({ SQLITE_MIGRATE_PATH: "/x/foo.db", DATA_DIR: "/data" })).toBe(
"/x/foo.db",
);
});
it("probes DATA_DIR/snapotter.db when unset", () => {
const dir = mkdtempSync(join(tmpdir(), "src-"));
writeFileSync(join(dir, "snapotter.db"), "x");
expect(resolveSource({ SQLITE_MIGRATE_PATH: "", DATA_DIR: dir })).toBe(
join(dir, "snapotter.db"),
);
});
it("returns null when nothing is present", () => {
const dir = mkdtempSync(join(tmpdir(), "src-"));
expect(resolveSource({ SQLITE_MIGRATE_PATH: "", DATA_DIR: dir })).toBeNull();
});
});
describe("evaluateBootState", () => {
it("import when users empty, source present, no marker", () => {
expect(evaluateBootState({ usersCount: 0, source: "/a.db", marker: null })).toBe("import");
});
it("leftover when a completed marker exists", () => {
expect(
evaluateBootState({ usersCount: 5, source: "/a.db", marker: { status: "completed" } }),
).toBe("leftover");
});
it("locked when source present, users non-empty, no marker", () => {
expect(evaluateBootState({ usersCount: 1, source: "/a.db", marker: null })).toBe("locked");
});
it("none when no source", () => {
expect(evaluateBootState({ usersCount: 0, source: null, marker: null })).toBe("none");
});
});
describe("countLibraryBlobs", () => {
it("counts present and missing library blobs", async () => {
const dir = mkdtempSync(join(tmpdir(), "blob-"));
const dbPath = join(dir, "legacy.db");
buildLegacySqlite(dbPath);
await seedRealistic1xData(dbPath); // seeds user_files uf-1 -> stored_name abc123.png
const filesDir = join(dir, "files");
mkdirSync(filesDir, { recursive: true });
writeFileSync(join(filesDir, "abc123.png"), "img");
expect(await countLibraryBlobs(dbPath, filesDir)).toEqual({ present: 1, missing: 0 });
});
});
describe("analyzeSqlite", () => {
it("reports counts, blobs, and out-of-enum statuses without a live Postgres", async () => {
const dir = mkdtempSync(join(tmpdir(), "analyze-"));
const dbPath = join(dir, "legacy.db");
buildLegacySqlite(dbPath);
await seedRealistic1xData(dbPath); // 1 user, 2 jobs (one "error"), 1 user_file
const filesDir = join(dir, "files");
mkdirSync(filesDir, { recursive: true });
writeFileSync(join(filesDir, "abc123.png"), "img"); // the uf-1 blob
const report = await analyzeSqlite(dbPath, filesDir);
expect(report.tables.users).toBe(1);
expect(report.tables.jobs).toBe(2);
expect(report.blobs).toEqual({ present: 1, missing: 0 });
expect(report.badStatuses).toContain("error"); // will be mapped to "failed" on import
});
});
+25
View File
@@ -0,0 +1,25 @@
import { describe, expect, it } from "vitest";
import { loadEnv } from "../../apps/api/src/lib/env.js";
describe("env DATA_DIR", () => {
it("defaults to ./data when unset", () => {
const prev = process.env.DATA_DIR;
delete process.env.DATA_DIR;
try {
expect(loadEnv().DATA_DIR).toBe("./data");
} finally {
if (prev !== undefined) process.env.DATA_DIR = prev;
}
});
it("honors an explicit DATA_DIR", () => {
const prev = process.env.DATA_DIR;
process.env.DATA_DIR = "/data";
try {
expect(loadEnv().DATA_DIR).toBe("/data");
} finally {
if (prev === undefined) delete process.env.DATA_DIR;
else process.env.DATA_DIR = prev;
}
});
});
+36
View File
@@ -0,0 +1,36 @@
import { mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import Database from "better-sqlite3";
import { describe, expect, it } from "vitest";
import { buildLegacySqlite, seedRealistic1xData } from "../helpers/legacy-sqlite-fixture.js";
describe("legacy sqlite fixture builder", () => {
it("reconstructs the 1.17.2 schema including dropped-in-2.x columns", () => {
const dir = mkdtempSync(join(tmpdir(), "legacy-fixture-"));
const path = join(dir, "legacy.db");
buildLegacySqlite(path);
const s = new Database(path, { readonly: true });
const cols = (s.prepare("PRAGMA table_info(users)").all() as Array<{ name: string }>).map(
(c) => c.name,
);
s.close();
// These three exist in 1.17.2 (legacy migration 0009) but NOT in the 2.x schema.
expect(cols).toContain("analytics_enabled");
expect(cols).toContain("analytics_consent_shown_at");
expect(cols).toContain("analytics_consent_remind_at");
});
it("seeds realistic rows including an out-of-enum job status", async () => {
const dir = mkdtempSync(join(tmpdir(), "legacy-fixture-"));
const path = join(dir, "legacy.db");
buildLegacySqlite(path);
await seedRealistic1xData(path);
const s = new Database(path, { readonly: true });
const statuses = (s.prepare("SELECT status FROM jobs").all() as Array<{ status: string }>).map(
(r) => r.status,
);
s.close();
expect(statuses).toContain("error"); // not a 2.x enum member — must be mapped on import
});
});
+54 -1
View File
@@ -1,5 +1,10 @@
import { describe, expect, it } from "vitest";
import { shouldShowInstallFeedbackCard, shouldShowUsageSurvey } from "@/lib/feedback";
import {
parseMigrationMarker,
shouldShowInstallFeedbackCard,
shouldShowMigrationBanner,
shouldShowUsageSurvey,
} from "@/lib/feedback";
const NOW = new Date("2026-01-15T00:00:00Z").getTime();
@@ -150,3 +155,51 @@ describe("shouldShowUsageSurvey", () => {
).toBe(false);
});
});
describe("shouldShowMigrationBanner", () => {
const marker = JSON.stringify({
status: "completed",
tables: { users: 2, user_files: 1 },
blobs: { present: 1, missing: 0 },
});
it("shows for admins when a marker exists and is not dismissed", () => {
expect(shouldShowMigrationBanner({ settings: { sqlite_import: marker }, role: "admin" })).toBe(
true,
);
});
it("is hidden for non-admins even with a marker", () => {
expect(shouldShowMigrationBanner({ settings: { sqlite_import: marker }, role: "user" })).toBe(
false,
);
});
it("is hidden once dismissed", () => {
expect(
shouldShowMigrationBanner({
settings: { sqlite_import: marker, "sqlite_import.dismissedAt": "2026-01-14T00:00:00Z" },
role: "admin",
}),
).toBe(false);
});
it("is hidden when there is no marker", () => {
expect(shouldShowMigrationBanner({ settings: {}, role: "admin" })).toBe(false);
});
});
describe("parseMigrationMarker", () => {
it("parses completed and detected_locked markers", () => {
expect(parseMigrationMarker(JSON.stringify({ status: "completed" }))?.status).toBe("completed");
expect(parseMigrationMarker(JSON.stringify({ status: "detected_locked" }))?.status).toBe(
"detected_locked",
);
});
it("returns null for missing, invalid, or unknown-status input", () => {
expect(parseMigrationMarker(undefined)).toBeNull();
expect(parseMigrationMarker("not json")).toBeNull();
expect(parseMigrationMarker(JSON.stringify({ status: "bogus" }))).toBeNull();
});
});