Uninstalling a bundle only deletes its downloaded model weights, never the
shared venv's site-packages, so self-hosters who already hit an AI bundle
conflict (e.g. the scipy ABI strand) have no clean self-service path via
uninstall+reinstall: reinstalling just overlays corrected files on top of
stale ones. Adds POST /api/v1/admin/features/reset, which wipes
/data/ai/{venv,models,pip-cache}, resets installed.json, and reseeds a real
working venv from the image's baked /opt/venv (extracted docker/reseed-ai-venv.sh,
now shared with entrypoint.sh's existing base-venv-upgrade bootstrap instead
of duplicating that logic) -- leaving an empty venv directory here would
make the very next install fail with "spawn .../python3 ENOENT", caught by
testing this live rather than assuming it. Ships with a matching Settings UI
section (inline confirm, same pattern as per-bundle uninstall) and strings
across all 21 locales.
Verified against a real snapotter/snapotter:1.17.2 image migrated to 2.0.0,
with real multi-GB bundles installed (background-removal + OCR): confirmed
the migrated instance's inherited python3.11 venv (2.0.0 itself uses 3.12)
still imports the fixed scipy/numpy/paddleocr correctly, then reset + real
reinstall + actual tool execution (remove-background, verified output image)
all worked end-to-end.
The container dropped privileges to the non-root snapotter user via gosu
(external) and s6-setuidgid (embedded), both of which preserve the
environment without setting HOME. The app therefore kept root's HOME=/root,
which is not writable by snapotter, and PaddleOCR died with
PermissionError: '/root/.paddlex/temp' -- breaking the ocr tool at default
quality in every non-root deployment. Prior GPU QA ran the app as root, which
masked it.
Fix: export HOME=/data/.home (persistent, writable, hidden) at every
privilege-drop point:
- entrypoint.sh external gosu path and non-root tini path (the latter uses
$DD/.home so a DATA_DIR override stays consistent).
- the s6 snapotter/run service (scoped there, not globally before /init, so
postgres/redis do not inherit a snapotter-owned HOME).
The root preflight creates /data/.home and the existing chown sweep owns it as
the PUID/PGID-remapped snapotter; the dir is added to both ensure_writable
probes so an unwritable HOME fails fast with the storage-permission guidance
instead of crashing late. The Dockerfile passwd home moves from /app
(read-only) to /data/.home as the getpwuid fallback when HOME is unset.
Because bridge.ts forwards HOME to the Python sidecar, this also repairs the
expanduser("~") caches in inpaint/outpaint/restore/noise_removal/remove_bg,
not just PaddleOCR.
Also fixes a test-harness inconsistency: tool-default-settings passport-photo
countryCode "us" -> "US" (the route exact-matches uppercase PASSPORT_SPECS
codes; the UI already sends "US", so users were never affected).
Claude-Session: https://claude.ai/code/session_01XGB4pGvTvb7sUX4JN745U7
AI feature installs now keep copied Python venv metadata (bin/pip shebang,
bin/activate, pyvenv.cfg) pointed at /data/ai/venv, so scripts no longer
silently fall back to the baked, read-only /opt/venv after the venv is
bootstrapped into /data. Fixes#127 (AI tools incompatible with PUID/PGID).
The entrypoint repairs both fresh bootstraps and already-stamped runtime
venvs (self-heals existing deployments on next restart, no reinstall
needed), with regression coverage for literal path replacement and binary
file safety.
Independently reviewed and verified: traced chown/gosu ordering in
entrypoint.sh to confirm no permission regression, reproduced the exact
issue #127 scenario (custom PUID + manual venv activation) in a live
container both before and after the fix, and ran the PR's own test suite
locally (16/16 passing).
Co-authored-by: SyntaxSawdust
Embedded Postgres 17 + Redis via s6-overlay when DATABASE_URL/REDIS_URL are unset; restores the one-command docker run for 2.0. EMBEDDED=0 disables; Compose stays the production path. Verified arm64 (14/14 lifecycle + Compose regression) and amd64 (build + embedded smoke).
The entrypoint only fixed volume permissions when started as root (chown +
gosu-drop to snapotter). Launched under a non-root/foreign UID (TrueNAS app
user, Kubernetes runAsUser, OpenShift) it did no permission setup, so /data and
/tmp/workspace -- owned by uid 999 from the image -- were not writable by the
running user. Uploads and processing then failed with a cryptic EACCES
("workspace folder is not writable") and AI bundle installs failed the same way,
while health checks still reported the container healthy.
- entrypoint: source new entrypoint-lib.sh; verify writability up front when
non-root, and as snapotter after chown when root (catches root-squashed
mounts), failing fast with an actionable message (which dir, uid/gid, how to
fix) instead of a late, cryptic EACCES
- Dockerfile: own /data and /tmp/workspace as snapotter:0, group-writable with
setgid, so an arbitrary UID with the root supplementary group (OpenShift /
Kubernetes fsGroup) can write; keep /opt/venv world-readable for the AI venv
bootstrap under arbitrary UIDs
- api: assert storage writability at boot (lib/storage-writable.ts), failing
fast with the same guidance even when the entrypoint is bypassed
- docs: add a Storage permissions section (named volumes, bind mounts, TrueNAS,
Kubernetes/OpenShift) and cross-link it from the security guide
Fixes#230
- Lower LOGIN_ATTEMPT_LIMIT default from 30 to 10 (brute-force protection)
- Lower RATE_LIMIT_PER_MIN default from 1000 to 300
- Add Redis authentication (requirepass) with REDIS_PASSWORD env var
- Add Redis maxmemory 512mb cap to prevent unbounded growth
- Add mem_limit: 1g to Postgres and Redis containers
- Strip internal file paths from all error responses (defense-in-depth)
- Add startup warnings for default admin/Postgres/Redis credentials
- Update security test expectations for new defaults
On 1.x-to-2.0 upgrade the /data/ai/venv already existed with stale
packages, so the entrypoint skipped the cp from /opt/venv and 2.0
Python tools failed with ImportError (#85).
Dockerfile now writes a SHA-256 of `pip freeze` to /opt/venv/.venv-version
at build time. The entrypoint compares that stamp against the copy on the
volume; on mismatch it does a clean nuke + re-copy and resets
installed.json so the UI correctly shows bundles as needing reinstall.
Models in /data/ai/models survive the refresh, so bundle reinstall only
reruns pip (model downloads are idempotent).
Backward compatible: if the image has no stamp (old build), the
entrypoint falls through to the existing skip-if-exists behavior.
* feat(infra): add dev compose stack with postgres and redis
* fix(infra): comment dev env defaults until wired; harden dev compose restart and start_period
* chore(deps): add pg driver and testcontainers for postgres migration
* feat(db): translate schema to drizzle pg-core (timestamptz, boolean, pgEnum, jsonb)
Schema translation (apps/api/src/db/schema.ts):
- sqlite-core -> pg-core, all 10 tables preserved 1:1
- integer(mode:'timestamp') -> timestamp({ withTimezone: true })
- integer(mode:'boolean') -> boolean
- jobs.status text enum -> pgEnum('job_status') with same 4 values
- 7 columns changed from text to jsonb: jobs.inputFiles, jobs.settings,
pipelines.steps, apiKeys.permissions, roles.permissions,
auditLog.details, userFiles.toolChain
- settings.value stays text, jobs.error stays text, jobs.progress stays real
jsonb call-site sweep (removed JSON.stringify on writes, JSON.parse on reads):
- apps/api/src/routes/roles.ts: permissions read/write (3 sites)
- apps/api/src/routes/api-keys.ts: permissions write + read (2 sites)
- apps/api/src/routes/audit-log.ts: details read (1 site)
- apps/api/src/routes/pipeline.ts: steps write + read (2 sites)
- apps/api/src/routes/progress.ts: inputFiles write (2 sites)
- apps/api/src/routes/tool-factory.ts: toolChain read + write (2 sites)
- apps/api/src/routes/user-files.ts: toolChain read + write (4 sites)
- apps/api/src/permissions.ts: roles.permissions read (1 site)
- apps/api/src/lib/audit.ts: details write (1 site)
- apps/api/src/plugins/auth.ts: apiKeys.permissions read (1 site)
* refactor(db): type jsonb columns via $type and note raw CTE conversion requirements
* feat(db): archive sqlite migrations and generate postgres baseline
* chore(db): dockerignore legacy migrations, add archive breadcrumb, fix trailing newline
* feat(db): pg pool connection, advisory-locked boot migrations, DATABASE_URL config
* fix(db): friendly fatal on unreachable postgres, idempotent closeDb, lock-key convention note
* refactor(db): async drizzle calls in plugins, lib, permissions
* fix(api): analytics never throws, typed permission guard, single-query session invalidation
* refactor(db): async drizzle calls across all routes and bootstrap
Convert every route file and index.ts from sync SQLite drizzle
patterns to async node-postgres drizzle:
- .all() removed (bare await on select)
- .get() converted to destructured [row] = await ...
- .run() removed (bare await on insert/update/delete)
- .changes replaced with .rowCount (null-guarded) in progress.ts
- sqlite import removed from user-files.ts; raw CTEs converted to
await db.execute(sql`...`) with postgres-dialect recursive CTEs
- ChainRow types updated: tool_chain is parsed jsonb (string[] | null),
created_at is Date (timestamptz) with no * 1000 conversion
- All requirePermission() guard calls awaited (security: unawaited
async guard returns truthy Promise, bypassing permission check)
- All hasEffectivePermission() and getPermissions() calls awaited
- All auditLog() calls awaited (preserves write-before-response order)
- trackEvent() and captureException() left un-awaited (fire-and-forget
by design, guaranteed never-throw)
- ensureAnonymousUser(), startCleanupCron(), recoverStaleJobs() awaited
in bootstrap sequence
- ensureInstanceId() and ensureDefaultSettings() made async
Files converted: 14 (index.ts + 12 route files + tools/index.ts)
* fix(db): await async checkStorageQuota in user-files upload/save routes
* fix(db): await checkStorageQuota in save-result route (missed second call site)
* feat(db): sqlite-to-postgres migrator with CLI and first-boot import
* fix(db): migrator error context, honest force semantics, boot-hook fatal, null-variance tests
* test: run suite against per-file postgres databases via testcontainers
- Add tests/global-setup.ts: spins up a Postgres testcontainer,
creates a migrated template database once per vitest run.
- Rewrite tests/setup/per-fork-env.ts: each test file (forks pool)
clones the template into its own database via CREATE DATABASE ...
TEMPLATE, preserving the same per-file isolation granularity.
- Update vitest.config.ts: add globalSetup, pg alias, update comment.
- Fix tests/integration/test-server.ts: remove DB_PATH mkdir, async
runMigrations, async db operations, remove SQLite WAL checkpoint.
- Fix 21 unit test db/index mocks: add pool and closeDb exports.
- Fix 8 unit test files: add async/await for now-async permission,
audit, and analytics functions.
- Fix 18 integration test files: convert sync .run()/.all()/.get()
to async drizzle patterns, add async to callbacks.
- Production change: apps/api/src/routes/teams.ts: cast COUNT(*)
to ::int so Postgres returns a number instead of bigint string.
* fix(db): seed built-in roles, reject NUL bytes, cast COUNT, serialize job persists
- Seed built-in roles (admin, editor, user) at boot via ensureBuiltinRoles()
with onConflictDoNothing, restoring data that legacy SQLite migration 0007
provided via INSERT statements (the pg baseline is DDL-only).
- Reject NUL bytes in login credentials with 401 (postgres rejects \x00 in
text columns; valid usernames never contain NUL, matching 1.x behavior).
- Cast COUNT(*)::int in user-files, audit-log, and roles listing queries so
postgres returns a JS number instead of bigint-as-string.
- Serialize fire-and-forget job progress DB writes per jobId so the final
"completed" status is never overwritten by a late-arriving "processing"
write (race condition exposed by async postgres round-trips).
* test: fix teams race, seed roles in test server, poll for job status
- Add missing await to resetTeams() in teams PUT beforeEach (the async
delete raced with the subsequent insert under postgres).
- Call ensureBuiltinRoles() in test server bootstrap so integration tests
have the same built-in roles as production.
- Replace fixed 100ms flushPersist delay with a polling helper that waits
for terminal job status, eliminating timing-dependent failures caused by
postgres network round-trip latency.
* test: make heic temp-file cleanup assertion resilient to concurrent workers
Use a set-based diff instead of raw file count when checking that
decodeHeic cleans up temp files. Other concurrent test workers can
create heic-in-*/heic-out-* files in the shared tmpdir, inflating the
"after" count and causing spurious failures under full-suite load.
* fix(db): align builtin-role seed to post-0010 legacy state; test polish
* feat(docker): three-container compose (app, postgres, redis) with boot wait and migrations
* fix(docker): set TEST_DATABASE_URL so containerized tests skip testcontainers
* chore(docker): test compose project name, clearer 1.x upgrade comment, unref probe timer
* feat(enterprise): enforce D15 license boundary; move s3 storage into packages/enterprise
* fix(enterprise): restore lazy aws-sdk loading; community installs load no s3 code at boot
* fix(enterprise): boundary check catches dynamic imports; document getS3 concurrency
* feat(db)!: SnapOtter 2.0 phase 1 foundation: postgres, migrator, compose stack
BREAKING CHANGE: SQLite is no longer the runtime database. Deployments now
require Postgres (and Redis, used from phase 2). Existing installs migrate
with SQLITE_MIGRATE_PATH or 'pnpm --filter @snapotter/api migrate:sqlite'.
* fix(ci): postgres service + fresh e2e database per run; ignore unfixable torch CVE-2025-3000
Support reading secrets from mounted files instead of plain-text
environment variables, following the standard Docker/Kubernetes
convention used by MariaDB, Postgres, and Stirling-PDF.
Supported vars: DEFAULT_PASSWORD, S3_ACCESS_KEY_ID,
S3_SECRET_ACCESS_KEY, OIDC_CLIENT_SECRET, COOKIE_SECRET,
SNAPOTTER_LICENSE_KEY.
Auth: login rate limit 30/min (was 500), global rate limit 1000/min (was
unlimited), password/username max lengths on all Zod schemas, session
invalidation on role change, API key legacy scan bounded to 100 keys.
SVG: hardened regex sanitizer with CDATA stripping, XML entity decoding,
set/animate/iframe/embed blocking, comprehensive data: URI blocking,
use element external href blocking. 11 attack payload fixtures added.
SSRF: fixed DNS rebinding TOCTOU by pinning resolved IPs via custom
HTTP/HTTPS agents. Added 6to4 and NAT64 to blocked IPv6 ranges.
Docker: capability dropping (cap_drop ALL + minimal cap_add), resource
limits (4g/8g mem, 512/1024 pids), healthcheck timeout, password
removed from startup banner, default password warning comments.
Network: CSP and HSTS applied in all environments (not just production),
stack traces removed from all error responses, internal paths stripped
from error details, per-route rate limits on uploads (60/min) and URL
fetches (200/hour).
Files: exclusive temp file creation (O_EXCL), disk space circuit
breaker, per-user storage quotas, settings payload 64KB size guard.
Python sidecar: script name allowlist in dispatcher, minimal environment
for subprocess spawns.
Dependencies: fixed 6 production CVEs (drizzle-orm, fastify, fast-uri,
@fastify/static, next, archiver/lodash). Pinned all GitHub Actions to
SHA hashes.
114 security tests added. Full OWASP Top 10 penetration test matrix
verified against production Docker container (30/30 pass after
hardening).
- Add Cloudflare Pages deployment for landing page (snapotter.com) and
docs (docs.snapotter.com)
- Create deploy-landing.yml and update deploy-docs.yml workflows
- Update CI to ignore apps/landing/** paths
- Fix logo transparency (remove white background) across all apps
- Recreate social-preview.png with SnapOtter branding
- Update all docs URLs from GitHub Pages to docs.snapotter.com
- Update VitePress config: light theme default, fix llms.txt paths
- Add .vitepress/cache/ and .env.* to gitignore
Phase 1 — Docker Artifact Optimization:
- Replace broad `COPY . .` with targeted frontend source copies (API/Python
changes no longer bust the frontend build cache)
- Replace build-essential with gcc/g++ (leaner runtime)
- Fix LOG_LEVEL=debug → info for production
- Harden .dockerignore (exclude worktrees, IDE, CI, test artifacts)
Phase 2 — State & Persistence:
- Add PUID/PGID support in entrypoint.sh for bind mount compatibility
- Guard against PUID=0/PGID=0 to prevent accidental root execution
- Evict conflicting system users (e.g. node:1000) before UID remap
Phase 3 — Security:
- Always register @fastify/rate-limit so login brute-force protection
works even when global rate limit is disabled (RATE_LIMIT_PER_MIN=0)
- Add trustProxy support (TRUST_PROXY env var, default true) so rate
limiting and audit logs use real client IPs behind reverse proxies
- Strip stack traces from 500 error responses in production
- Fix FSTDEP022 deprecation: maxParamLength → routerOptions
- Add multi-file guard on single-file tool endpoint with clear error
message pointing to the /batch endpoint
Phase 4 — Graceful Degradation:
- Add consolidated hardware detection startup banner (GPU, rate limit,
upload limit, proxy status)
- Add ConnectionMonitor component with health polling and reconnecting
overlay that auto-dismisses when the server comes back
Phase 5 — Deployment Docs:
- Rewrite deployment.md with copy-paste CPU and GPU compose templates
- Add hardware requirements table (minimum, recommended, heavy workloads)
- Add PUID/PGID bind mount documentation
- Add complete env var reference table
- Add reverse proxy guides for Nginx, Nginx Proxy Manager, Traefik,
and Cloudflare Tunnels
Remove all ML pip installs (onnxruntime, rembg, realesrgan, paddlepaddle,
mediapipe, codeformer), model downloads, and post-install fixups from the
Dockerfile. The base image now ships only Node.js + Sharp + Python with
numpy/Pillow/opencv. AI features are installed on-demand at runtime via
the feature manifest and install_feature.py script.
Key changes:
- Remove SKIP_MODEL_DOWNLOADS build arg (no longer needed)
- Remove apt-get purge of build-essential (needed for runtime pip installs)
- Remove PaddleX symlinks and facexlib weight directory setup
- Add COPY of feature-manifest.json and install_feature.py
- Update PYTHON_VENV_PATH to /data/ai/venv, add MODELS_PATH and DATA_DIR
- Entrypoint bootstraps AI venv from /opt/venv on first container start
with crash-safe temp directory pattern
Set U2NET_HOME=/opt/models/rembg so rembg models pre-downloaded at
build time as root are found at runtime by the non-root ashim user.
Without this every fresh container re-downloaded the 973 MB BiRefNet
models on first background-removal request.
Apply the same fix to PaddleOCR: download to /opt/models/paddlex and
symlink into both /root/.paddlex and /app/.paddlex so PaddleX finds
models regardless of which HOME gosu resolves at runtime.
Fall back to per-request spawning in bridge.ts when the persistent
dispatcher crashes mid-request (e.g. OOM loading a large ONNX model),
so the operation succeeds instead of surfacing "Python dispatcher
exited unexpectedly" to the user.
Improve entrypoint.sh permission warning to mention Windows bind mounts
as the likely cause.
- Fix "Cannot access 'a' before initialization" TDZ error after login
caused by manualChunks splitting react-vendor + lucide icons into
circular ES-module chunks. Removed manualChunks entirely.
- Replace `import * as icons from "lucide-react"` (pulls all ~1000 icons)
with a targeted icon-map of ~50 icons actually used by tool definitions.
Reduces shared icons chunk from 745KB to 62KB (132KB→16KB gzip).
- Exclude static files from @fastify/rate-limit via allowList so rapid
page navigations don't 429 on JS/CSS chunk requests.
- Move Docker auth defaults (AUTH_ENABLED, DEFAULT_USERNAME,
DEFAULT_PASSWORD) from Dockerfile ENV to entrypoint.sh runtime exports
to avoid SecretsUsedInArgOrEnv warnings.
- Fix Docker CMD to use pnpm --filter for workspace-scoped tsx binary.
- Set COREPACK_HOME system-wide so non-root user can access pnpm cache.
- Lazy-load all pages in App.tsx and all controls in
pipeline-step-settings.tsx to keep main bundle under 300KB.
Adds a gosu-based entrypoint that starts as root, fixes ownership of
/data and /tmp/workspace for the stirling user, then drops privileges.
This fixes "SQLITE database not found" errors when users bind-mount
host directories.
