External/SnapOtter
1
0
Fork 0
mirror of https://github.com/snapotter-hq/SnapOtter.git synced 2026-08-03 07:46:42 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix: release QA hardening across processing, media, security, and CI gates (#649)

Browse Source 
A release-readiness QA pass over the whole product. The commits split into
defects a user would hit and gates that were reporting green while measuring
nothing.

## Fixes that change behaviour

Rate limiting was bypassable on every install: TRUST_PROXY defaulted to true, so
request.ip came from a client-set header and a forged X-Forwarded-For got past
the login limiter. The default is now a private-network trust list.

A transient Postgres outage stranded in-flight jobs, leaving finished output on
disk with no row pointing at it. A reconciler now resolves those rows and adopts
the bytes rather than dropping the work.

A Redis connection that moved to a new address wedged every read-blocked
consumer, so completions stopped signalling while health still answered 200.
Socket timeouts plus subscriber pings recover it.

Installing more than one AI bundle left the shared venv multi-versioned and
silently broke three tools. The installer now reconciles distributions to one
version each.

Converting an image to JXL at quality 1 through 4 returned a 500, because
libjxl 0.7 rejects the distance those values compute. The quality is floored at
what the encoder honours. A missing ffmpeg was also reported to the user as a
corrupt upload; it now says the engine is unavailable.

RAW uploads reached an unpatched LibRaw on arm64, so it is built from source at
0.22.2, and the release scan was split so it can fail on an unfixed critical
instead of hiding it behind ignore-unfixed.

## Gates that could not fail

Two mutation lanes ran zero mutants because Stryker crawled the gitignored docs
build; coverage discarded its whole report on any failing test; the lint gate
skipped root tests, scripts, and two workspaces; and several generated matrices
counted a host missing ffmpeg as a passing tool. Each now measures what it
claims.

Full evidence and the outstanding release items are tracked locally and are not
part of this branch.
This commit is contained in:
SnapOtter
2026-07-27 15:37:30 +08:00
committed by GitHub
parent bc32f86a07
commit d10d0f544f
855 changed files with 54564 additions and 13092 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/docs/guide/configuration.md
+35 -13
View File
@@ -16,28 +16,51 @@ All configuration is done through environment variables. Every variable has a se
| `RATE_LIMIT_PER_MIN` | `1000` | Maximum requests per minute per IP. Set to 0 to disable rate limiting. |
| `CORS_ORIGIN` | (empty) | Comma-separated allowed origins for CORS, or empty for same-origin only. |
| `LOG_LEVEL` | `info` | Log verbosity. One of: `fatal`, `error`, `warn`, `info`, `debug`, `trace`. |
| `TRUST_PROXY` | `true` | Trust `X-Forwarded-For` headers from a reverse proxy. Set to `false` if not behind a proxy. |
| `TRUST_PROXY` | `loopback,linklocal,uniquelocal` | Which peers may set the client IP through `X-Forwarded-For`. The default believes only a private-network peer, so a reverse proxy on a Docker network or a LAN is trusted and a public client's forged header is not. Set `true` only when a proxy you control sits in front on a public address. |
### Authentication {#authentication}
The two booleans below accept only `true` and `false`. Anything else, `1` or `yes` or `on`, fails validation and the server exits before it starts listening.
| Variable | Default | Description |
|---|---|---|
| `AUTH_ENABLED` | `false` | Set to `true` to require login. The Docker image defaults to `true`. |
| `AUTH_ENABLED` | `true` | Require a login. Set to `false` to run with no accounts at all, which grants every request admin rights, so keep that to a trusted network. |
| `DEFAULT_USERNAME` | `admin` | Username for the initial admin account. Only used on first run. |
| `DEFAULT_PASSWORD` | `admin` | Password for the initial admin account. Change this after first login. |
| `MAX_USERS` | `0` (unlimited) | Maximum number of registered user accounts. Set to 0 for unlimited. |
| `SESSION_DURATION_HOURS` | `168` | Login session lifetime in hours (default is 7 days). |
| `SKIP_MUST_CHANGE_PASSWORD` | - | Set to any non-empty value to bypass the forced password-change prompt on first login |
| `SKIP_MUST_CHANGE_PASSWORD` | `false` | Set to `true` to skip the forced password-change prompt on first login. |
### Storage {#storage}
| Variable | Default | Description |
|---|---|---|
| `STORAGE_MODE` | `local` | `local` or `s3`. S3/MinIO requires a license with the s3_storage feature. |
| `DATABASE_URL` | `postgres://snapotter:snapotter@postgres:5432/snapotter` | PostgreSQL connection string. |
| `REDIS_URL` | `redis://redis:6379` | Redis connection string (used for BullMQ job queues). |
| `WORKSPACE_PATH` | `./tmp/workspace` | Directory for temporary files during processing. Cleaned up automatically. |
| `FILES_STORAGE_PATH` | `./data/files` | Directory for persistent user files (uploaded images, saved results). |
| `STORAGE_MODE` | `local` | `local` or `s3`. S3 and MinIO need a license with the s3_storage feature plus the `S3_*` variables below. |
| `DATABASE_URL` | `postgres://snapotter:snapotter@localhost:5432/snapotter` | PostgreSQL connection string. The Compose stack points this at its `postgres` service; leave it unset (together with `REDIS_URL`) to get embedded mode. |
| `REDIS_URL` | `redis://localhost:6379` | Redis connection string (used for BullMQ job queues). Compose points this at its `redis` service. |
| `WORKSPACE_PATH` | `./tmp/workspace` | Directory for temporary files during processing. Cleaned up automatically. The image sets `/tmp/workspace`. |
| `FILES_STORAGE_PATH` | `./data/files` | Directory for persistent user files (uploaded images, saved results). The image sets `/data/files`. |
### S3 object storage {#s3-object-storage}
Only read when `STORAGE_MODE=s3`. Miss any of the three required ones and startup fails with the name of the variable you left out.
| Variable | Default | Description |
|---|---|---|
| `S3_BUCKET` | (empty) | Bucket that holds uploads and outputs. Required. |
| `S3_ACCESS_KEY_ID` | (empty) | Access key. Required. In the container you can mount it instead, via `S3_ACCESS_KEY_ID_FILE`. |
| `S3_SECRET_ACCESS_KEY` | (empty) | Secret key. Required. Same file convention: `S3_SECRET_ACCESS_KEY_FILE`. |
| `S3_REGION` | `us-east-1` | Bucket region. |
| `S3_ENDPOINT` | (empty) | Custom endpoint for MinIO, R2, Backblaze, and other S3-compatible stores. Empty means AWS. |
| `S3_FORCE_PATH_STYLE` | `false` | Set to `true` for MinIO and anything else that wants `endpoint/bucket/key` instead of virtual-host addressing. |
| `S3_PREFIX` | (empty) | Key prefix, so one bucket can hold several instances. |
### Encryption at rest {#encryption-at-rest}
| Variable | Default | Description |
|---|---|---|
| `DATA_ENCRYPTION_KEY` | (empty) | 64 hex characters (32 bytes). Encrypts sensitive settings stored in the database. Anything that is not 64 hex characters is rejected at startup. |
| `DATA_ENCRYPTION_KEY_PREVIOUS` | (empty) | The key you are rotating away from, same format. Set both during a rotation so existing rows still decrypt, then drop this one. |
### Embedded mode {#embedded-mode}
@@ -56,16 +79,15 @@ Telemetry note: embedded mode inherits the image's analytics default like any ot
| Variable | Default | Description |
|---|---|---|
| `MAX_UPLOAD_SIZE_MB` | `100` | Maximum file size per upload in megabytes. Set to 0 for unlimited. |
| `MAX_BATCH_SIZE` | `100` | Maximum number of files in a single batch request. Set to 0 for unlimited. |
| `MAX_UPLOAD_SIZE_MB` | `0` (unlimited) | Maximum file size per upload in megabytes. Set to 0 for unlimited. The published image ships `0`; a source build starts at 100. |
| `MAX_BATCH_SIZE` | `0` (unlimited) | Maximum number of files in a single batch request. Set to 0 for unlimited. The published image ships `0`; a source build starts at 100. |
| `CONCURRENT_JOBS` | `0` (auto) | Number of batch jobs that run in parallel. Set to 0 to auto-detect based on available CPU cores. |
| `MAX_MEGAPIXELS` | `0` (unlimited) | Maximum image resolution allowed in megapixels. Set to 0 for unlimited. |
| `MAX_WORKER_THREADS` | `0` (auto) | Maximum worker threads for image processing. Set to 0 to auto-detect based on available CPU cores. |
| `PROCESSING_TIMEOUT_S` | `0` (no limit) | Maximum processing time per request in seconds. Set to 0 for no timeout. |
| `MAX_PIPELINE_STEPS` | `20` | Maximum number of steps in a pipeline. Set to 0 for no limit. |
| `MAX_CANVAS_PIXELS` | `0` (no limit) | Maximum canvas size in pixels for output images. Set to 0 for no limit. |
| `MAX_SVG_SIZE_MB` | `0` (unlimited) | Maximum SVG file size in megabytes. Set to 0 for unlimited. |
| `MAX_SPLIT_GRID` | `100` | Maximum grid dimension for the image split tool. |
| `MAX_SVG_SIZE_MB` | `50` | Largest SVG accepted before sanitizing, in megabytes. `0` behaves differently here than in the rows around it. It removes the pre-parse size cap entirely rather than raising it, so leave this one set. |
| `MAX_PDF_PAGES` | `0` (unlimited) | Maximum number of PDF pages for PDF-to-image conversion. Set to 0 for unlimited. |
### Cleanup {#cleanup}
@@ -79,7 +101,7 @@ Telemetry note: embedded mode inherits the image's analytics default like any ot
| Variable | Default | Description |
|---|---|---|
| `DEFAULT_THEME` | `light` | Default theme for new sessions. `light` or `dark`. |
| `DEFAULT_THEME` | `light` | Default theme for new sessions. `light`, `dark`, or `system`. |
| `DEFAULT_LOCALE` | `en` | Default interface language. |
| `DEFAULT_TOOL_VIEW` | `sidebar` | Default tool layout. `sidebar` or `fullscreen`. |
apps/docs/guide/contributing.md
+2 -2
View File
@@ -50,7 +50,7 @@ If you are contributing on behalf of your employer and your employer retains IP
### Prerequisites {#prerequisites}
- Node.js 22+
- Node.js 22.22+
- pnpm 9+
- Python 3.11+ (only for AI tools)
- Docker (optional, for full integration testing)
@@ -68,7 +68,7 @@ docker compose -f docker-compose.dev.yml up -d
# Install dependencies
pnpm install
# Start dev servers (web on :1349, API on :13490)
# Start dev servers (web on :1351, API on :13490)
pnpm dev
```
apps/docs/guide/database.md
+29 -10
View File
@@ -142,6 +142,17 @@ Security-relevant action log.
| `details` | jsonb | Action-specific data |
| `createdAt` | timestamp | Action time |
### user_preferences {#user-preferences}
Per-user UI state, keyed by preference name. Backs the dashboard's pinned tools through `PUT /api/v1/preferences`.
| Column | Type | Notes |
|---|---|---|
| `userId` | text | FK to users, cascades on delete. Primary key with `key` |
| `key` | text | Preference name. Primary key with `userId` |
| `value` | jsonb | Preference payload |
| `updatedAt` | timestamp | Last write |
## Migrations {#migrations}
Drizzle handles schema migrations. Migration files live in `apps/api/drizzle/`. During development:
@@ -158,25 +169,33 @@ In production, pending migrations are applied automatically on startup.
The relational database lives in the Postgres container's `SnapOtter-pgdata` volume, not the app's `/data` volume.
**Option 1: pg_dump (recommended)**
**Logical backup with validation (recommended)**
```bash
# Dump the database while the stack is running
docker exec SnapOtter-postgres pg_dump -U snapotter snapotter > backup.sql
# Dump into PostgreSQL's portable custom archive format
docker exec SnapOtter-postgres \
pg_dump --format=custom --no-owner -U snapotter snapotter > snapotter.dump
test -s snapotter.dump
docker exec -i SnapOtter-postgres pg_restore --list < snapotter.dump >/dev/null
# Restore into a fresh database
cat backup.sql | docker exec -i SnapOtter-postgres psql -U snapotter snapotter
# Restore into a fresh/disposable target first and fail on the first SQL error
docker exec -i SnapOtter-postgres \
pg_restore --exit-on-error --clean --if-exists --no-owner \
-U snapotter -d snapotter < snapotter.dump
```
**Option 2: Volume snapshot**
This database dump does not contain saved library objects in `/data/files` or durable BullMQ state in Redis. Back up and restore those with the coordinated procedure in [Security & Hardening](/guide/security#backup-and-recovery).
**Cold volume snapshot**
```bash
# Stop the stack, then snapshot the pgdata volume
docker compose down
docker run --rm -v SnapOtter-pgdata:/data -v $(pwd)/backup:/backup \
alpine tar czf /backup/snapotter-pgdata.tar.gz -C /data .
# Stop every service first, then use your storage platform to snapshot the
# PostgreSQL, app-data, and Redis volumes as one crash-consistent set.
docker compose -f docker/docker-compose.yml stop
```
Do not copy a live PostgreSQL data directory with `tar`. Compose prefixes volume names by project, so resolve the mounted volume IDs from `docker inspect` or your storage platform rather than assuming the literal label `SnapOtter-pgdata`.
### Migrating from 1.x (SQLite) {#migrating-from-1-x-sqlite}
Upgrading from SnapOtter 1.x has its own guide: see [Upgrading from 1.x to 2.0](./upgrading). In short, reuse your existing `/data` volume and 2.0 auto-detects and imports `/data/snapotter.db` on first boot (or set `SQLITE_MIGRATE_PATH` to point at it explicitly). Back up the whole `/data` volume first, not just `snapotter.db`: 1.x uses SQLite WAL mode, so a stopped container often leaves most of its data in `snapotter.db-wal` beside an almost-empty `snapotter.db`.
apps/docs/guide/deployment.md
+9 -5
View File
@@ -38,7 +38,7 @@ services:
# - MAX_USERS=0 # Max user accounts
# --- Networking ---
# - TRUST_PROXY=true # Trust X-Forwarded-For headers (set false if not behind a proxy)
# - TRUST_PROXY=loopback,linklocal,uniquelocal # Which peers may set the client IP via X-Forwarded-For (default shown)
# --- Bind mount permissions ---
# - PUID=1000 # Match your host user's UID (run: id -u)
@@ -431,11 +431,11 @@ The startup error names the exact UID to use, so the quickest path is to start t
| `AUTH_ENABLED` | `true` | Enable/disable login requirement |
| `DEFAULT_USERNAME` | `admin` | Initial admin username |
| `DEFAULT_PASSWORD` | `admin` | Initial admin password (forced change on first login) |
| `MAX_UPLOAD_SIZE_MB` | `100` | Per-file upload limit |
| `MAX_BATCH_SIZE` | `100` | Max files per batch request |
| `MAX_UPLOAD_SIZE_MB` | `0` (unlimited) | Per-file upload limit in MB. The image ships `0`; a source build starts at 100 |
| `MAX_BATCH_SIZE` | `0` (unlimited) | Max files per batch request. The image ships `0`; a source build starts at 100 |
| `RATE_LIMIT_PER_MIN` | `1000` | API requests per minute per IP (set 0 to disable) |
| `MAX_USERS` | `0` (unlimited) | Maximum user accounts |
| `TRUST_PROXY` | `true` | Trust X-Forwarded-For headers from reverse proxy |
| `TRUST_PROXY` | `loopback,linklocal,uniquelocal` | Which peers may set the client IP through `X-Forwarded-For`. Private networks only by default |
| `PUID` | `999` | Run as this UID (for bind mount permissions) |
| `PGID` | `999` | Run as this GID (for bind mount permissions) |
| `LOG_LEVEL` | `info` | Log verbosity: fatal, error, warn, info, debug, trace |
@@ -478,7 +478,11 @@ curl http://localhost:1349/api/v1/health
## Reverse Proxy {#reverse-proxy}
SnapOtter sets `TRUST_PROXY=true` by default so rate limiting and logging use the real client IP from `X-Forwarded-For` headers.
`TRUST_PROXY` defaults to `loopback,linklocal,uniquelocal`, so SnapOtter believes `X-Forwarded-For` only from a peer on a private network. A reverse proxy on the same host, on a Docker network, or on your LAN is trusted out of the box, which means rate limiting, the login brute-force limiter, the audit log, and the enterprise IP allowlist all see the real client IP with no configuration.
Set `TRUST_PROXY=true` only when the proxy in front reaches SnapOtter from a **public** address, a cloud load balancer on a different network for instance. On a directly exposed instance that value makes `request.ip` attacker-controlled, because a caller who rotates the header gets a fresh rate-limit bucket per request.
Two things to know before you go measuring client IPs. Docker Desktop on macOS and Windows serves a published port through a userland proxy that rewrites every source address to the VM gateway `192.168.65.1`, so no value of `TRUST_PROXY` recovers the real client there; deploy on Linux for anything internet-facing. And on any platform, reaching a published port over `localhost` is observed as the bridge gateway rather than as your client, so a localhost test tells you nothing about how a real client is attributed. The full table of `TRUST_PROXY` values and the Docker Desktop caveat are in [SECURITY.md](https://github.com/snapotter-hq/SnapOtter/blob/main/SECURITY.md#client-ip-resolution-trust_proxy).
Two things matter for every proxy below: allow large request bodies (uploads), and do not buffer responses. A response-buffering proxy breaks SSE progress and, more visibly, makes a large file download "start but never finish", because the proxy holds the whole file before passing it on. SnapOtter sends `X-Accel-Buffering: no` on downloads so nginx streams them even if buffering is left on elsewhere, but proxies other than nginx need response buffering disabled explicitly (shown in each config below). If a download stalls partway, a buffering proxy in front is the first thing to check.
apps/docs/guide/developer.md
+15 -4
View File
@@ -8,12 +8,12 @@ How to set up a local development environment and contribute code to SnapOtter.
## Prerequisites {#prerequisites}
- [Node.js](https://nodejs.org/) 22+
- [Node.js](https://nodejs.org/) 22.22+
- [pnpm](https://pnpm.io/) 9+ (`corepack enable && corepack prepare pnpm@latest --activate`)
- [Docker](https://www.docker.com/) (required for local Postgres + Redis, container builds, and AI features)
- Git
Python 3.10+ is only needed if you are working on the AI/ML sidecar (background removal, upscaling, OCR).
Python 3.11+ is only needed if you are working on the AI/ML sidecar (background removal, upscaling, OCR).
## Setup {#setup}
@@ -29,10 +29,10 @@ This starts two dev servers:
| Service | URL | Notes |
|----------|--------------------------|------------------------------------|
| Frontend | http://localhost:1349 | Vite dev server, proxies /api |
| Frontend | http://localhost:1351 | Vite dev server, proxies /api |
| Backend | http://localhost:13490 | Fastify API (accessed via proxy) |
Open http://localhost:1349 in your browser. Login with `admin` / `admin`. You will be prompted to change the password on first login.
Open http://localhost:1351 in your browser. Login with `admin` / `admin`. You will be prompted to change the password on first login.
## Project structure {#project-structure}
@@ -217,6 +217,17 @@ Use BuildKit cache mounts for faster rebuilds:
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile -t snapotter:latest .
```
## Release version domains {#release-version-domains}
SnapOtter intentionally has three version domains. Do not copy one domain into another during a release:
- The application release version covers the root manifest, all private workspace packages and `APP_VERSION`. Semantic-release supplies this value, and `pnpm version:sync <version>` updates every workspace before an application release.
- OpenAPI `info.version` is the stable public API-major contract. All localized specifications stay on `<major>.0.0` for compatible application releases and change only when the API contract moves to a new major version.
- `docker/feature-manifest.json` keeps `imageVersion: 2.0.0` as the immutable legacy feature-bundle storage epoch. Those v2 archive paths are not application package versions. Accurate OCR uses runtime format v3 and records its application release provenance separately.
`tests/unit/infra/release-version-policy.test.ts` enforces these boundaries. A new version domain or migration must update that contract and the relevant artifact migration design together.
The independent API and legacy-bundle values live in `config/release-version-policy.json`; application version synchronization must never rewrite that policy file implicitly.
## Environment variables {#environment-variables}
See the [Configuration guide](/guide/configuration) for the full list. Key ones for development:
apps/docs/guide/docker-tags.md
+3 -3
View File
@@ -137,9 +137,9 @@ For NVIDIA CUDA acceleration via Docker Compose, add the deploy section to the S
| Tag | Description |
|-----|------------|
| `latest` | Latest release |
| `1.11.0` | Exact version |
| `1.11` | Latest patch in 1.11.x |
| `1` | Latest minor in 1.x |
| `2.1.0` | Exact version |
| `2.1` | Latest patch in 2.1.x |
| `2` | Latest minor in 2.x |
## Platforms {#platforms}
apps/docs/guide/getting-started.md
+21 -55
View File
@@ -14,7 +14,7 @@ Explore the full UI at [demo.snapotter.com](https://demo.snapotter.com) - no sig
docker run -d --name SnapOtter -p 1349:1349 -v SnapOtter-data:/data snapotter/snapotter:latest
```
This single container runs everything it needs: with no `DATABASE_URL` set, it starts its own PostgreSQL and Redis on the loopback interface (embedded mode) and keeps all data in the `SnapOtter-data` volume. It is the fastest way to try SnapOtter or self-host on a homelab. For production, run the [Docker Compose](#docker-compose) stack below, which keeps PostgreSQL and Redis in their own containers. Embedded mode runs as root (the default) and turns off automatically as soon as you set `DATABASE_URL`.
This single container runs everything it needs: with no `DATABASE_URL` set, it starts its own PostgreSQL and Redis on the loopback interface (embedded mode) and keeps all data in the `SnapOtter-data` volume. It is the fastest way to try SnapOtter or self-host on a homelab. For production, use the [canonical Docker Compose stack](#docker-compose), which keeps PostgreSQL and Redis in their own containers. Embedded mode runs as root (the default) and turns off automatically as soon as you set `DATABASE_URL`.
Installing on a Raspberry Pi, an old laptop, or a small VPS? See [Low-Resource Setups](/guide/low-resource) for a tuned walkthrough and what to expect from constrained hardware.
@@ -50,65 +50,31 @@ Both registries publish the same image on every release.
## Docker Compose {#docker-compose}
```yaml
services:
SnapOtter:
image: snapotter/snapotter:latest # or ghcr.io/snapotter-hq/snapotter:latest
ports:
- "1349:1349"
volumes:
- SnapOtter-data:/data
environment:
- AUTH_ENABLED=true
- DEFAULT_USERNAME=admin
- DEFAULT_PASSWORD=admin
- DATABASE_URL=postgres://snapotter:snapotter@postgres:5432/snapotter
- REDIS_URL=redis://redis:6379
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
restart: unless-stopped
Use the production file maintained and tested with each release instead of copying an abbreviated Compose example from this page:
postgres:
image: postgres:17-alpine
environment:
POSTGRES_USER: snapotter
POSTGRES_PASSWORD: snapotter # Change this for non-local deployments
POSTGRES_DB: snapotter
volumes:
- SnapOtter-pgdata:/var/lib/postgresql/data
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "pg_isready -U snapotter -d snapotter"]
interval: 10s
timeout: 5s
retries: 12
```bash
install -d -m 700 snapotter && cd snapotter
curl --proto '=https' --tlsv1.2 -fsSLo docker-compose.yml \
https://raw.githubusercontent.com/snapotter-hq/SnapOtter/v2.1.0/docker/docker-compose.yml
redis:
image: redis:8-alpine
command: ["redis-server", "--maxmemory-policy", "noeviction", "--appendonly", "yes"]
volumes:
- SnapOtter-redisdata:/data
restart: unless-stopped
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
timeout: 5s
retries: 12
# Keep generated service credentials out of shell history and world-readable files.
umask 077
POSTGRES_PASSWORD="$(openssl rand -hex 32)"
REDIS_PASSWORD="$(openssl rand -hex 32)"
printf 'POSTGRES_PASSWORD=%s\nREDIS_PASSWORD=%s\n' \
"$POSTGRES_PASSWORD" "$REDIS_PASSWORD" > .env
volumes:
SnapOtter-data:
SnapOtter-pgdata:
SnapOtter-redisdata:
docker compose -f docker-compose.yml pull
docker compose -f docker-compose.yml up -d --no-build
```
See [Configuration](/guide/configuration) for all environment variables.
The canonical [`docker/docker-compose.yml`](https://github.com/snapotter-hq/SnapOtter/blob/v2.1.0/docker/docker-compose.yml) includes all four runtime volumes, health checks, resource limits, durable Redis configuration, pinned database/cache images, and the current container hardening. Change the default admin password immediately after first login. For a reproducible deployment, pin the SnapOtter application image to the release tag or digest you verified instead of following `latest`.
See [Configuration](/guide/configuration) for all environment variables and [Security & Hardening](/guide/security) for secrets, network policy, and backup guidance.
## Build from Source {#build-from-source}
**Prerequisites:** Node.js 22+, pnpm 9+, Docker (for Postgres + Redis), Python 3.10+ (for AI features), Git.
**Prerequisites:** Node.js 22.22+, pnpm 9+, Docker (for Postgres + Redis), Python 3.11+ (for AI features), Git.
```bash
git clone https://github.com/snapotter-hq/SnapOtter.git
@@ -118,7 +84,7 @@ pnpm install
pnpm dev
```
- Frontend: [http://localhost:1349](http://localhost:1349)
- Frontend: [http://localhost:1351](http://localhost:1351)
- Backend: [http://localhost:13490](http://localhost:13490)
## What You Can Do {#what-you-can-do}
@@ -130,8 +96,8 @@ pnpm dev
| **Image** | 107 | Resize, Crop, Compress, Convert, Remove Background, Upscale, OCR, Watermark, Collage, Colorize, GIF Tools, format presets |
| **Video** | 57 | Trim, Crop, Compress, Convert, Merge, Extract Audio, Auto Subtitles, Video to GIF, Resize, Stabilize, format presets |
| **Audio** | 27 | Trim, Merge, Convert, Normalize, Noise Reduction, Transcribe, Pitch Shift, Fade, Ringtone Maker, format presets |
| **PDF / Document** | 42 | Merge, Split, Compress, OCR, Watermark, Redact, Word to PDF, Excel to PDF, Rotate, Protect, Repair |
| **Files** | 10 | CSV to JSON, JSON to XML, Merge CSVs, Split CSV, Create ZIP, Extract ZIP, Chart Maker, YAML/JSON |
| **PDF / Document** | 29 | Merge, Split, Compress, OCR, Watermark, Redact, Word to PDF, Excel to PDF, Rotate, Protect, Repair |
| **Files** | 23 | CSV to JSON, JSON to XML, Merge CSVs, Split CSV, Create ZIP, Extract ZIP, Chart Maker, YAML/JSON |
### Pipelines {#pipelines}
apps/docs/guide/security.md
+70 -143
View File
@@ -10,105 +10,17 @@ The container runs as a dedicated non-root user (`snapotter`) with all Linux cap
## Container Hardening {#container-hardening}
The [default docker-compose.yml](https://github.com/snapotter-hq/SnapOtter/blob/main/docker/docker-compose.yml) includes production security hardening. Here is a breakdown of each option and why it matters:
The canonical [CPU](https://github.com/snapotter-hq/SnapOtter/blob/main/docker/docker-compose.yml) and [GPU](https://github.com/snapotter-hq/SnapOtter/blob/main/docker/docker-compose-gpu.yml) Compose files are the source of truth. Do not copy an abbreviated example into production; deploy the file from the release tag you verified.
```yaml
services:
SnapOtter:
image: snapotter/snapotter:latest
ports:
# Bind to localhost only for internet-facing deployments:
- "127.0.0.1:1349:1349"
volumes:
- SnapOtter-data:/data
- SnapOtter-workspace:/tmp/workspace
environment:
- AUTH_ENABLED=true
- DEFAULT_PASSWORD=change-me-immediately
- RATE_LIMIT_PER_MIN=1000
- DATABASE_URL=postgres://snapotter:snapotter@postgres:5432/snapotter
- REDIS_URL=redis://redis:6379
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
Both stacks apply the following controls:
# --- Resource limits ---
mem_limit: 6g # Prevents runaway memory from crashing the host
memswap_limit: 6g # No swap - fail fast instead of degrading the host
cpus: 4 # Cap CPU usage to 4 cores
pids_limit: 512 # Prevents fork bombs
- Memory, swap, CPU, and PID limits contain runaway native processing.
- Every service drops all Linux capabilities. The application adds back only `CHOWN, SETUID, SETGID, DAC_OVERRIDE, FOWNER, KILL` for volume ownership, the one-way `gosu` identity drop, and graceful signal forwarding. PostgreSQL and Redis receive only the subset their official entrypoints need.
- `security_opt: [no-new-privileges:true]` prevents processes in the application, PostgreSQL, and Redis containers from gaining additional privileges. This remains compatible with `gosu`: the entrypoint begins as root, prepares the volumes, and only drops to the dedicated `snapotter` user.
- PostgreSQL and Redis image inputs are pinned by digest. The application should likewise be pinned to a verified release tag or digest rather than `latest`.
- Health checks, bounded JSON log rotation, durable Redis AOF, and restart policy are defined centrally in the canonical files.
# --- Capability restrictions ---
cap_drop:
- ALL # Drop ALL Linux capabilities first
cap_add:
- CHOWN # Needed for volume permission setup
- SETUID # Needed for gosu privilege drop (root -> snapotter)
- SETGID # Needed for gosu privilege drop
- DAC_OVERRIDE # Needed for volume permission setup
- FOWNER # Needed for volume permission setup
# --- Logging ---
logging:
driver: json-file
options:
max-size: "50m" # Rotate logs at 50 MB
max-file: "5" # Keep 5 rotated log files
# --- Health check ---
healthcheck:
test: ["CMD", "curl", "-sf", "--max-time", "5", "http://localhost:1349/api/v1/health"]
interval: 30s
timeout: 5s
start_period: 60s
retries: 3
shm_size: "2gb" # Required for Python ML shared memory
restart: unless-stopped
postgres:
image: postgres:17-alpine
environment:
POSTGRES_USER: snapotter
POSTGRES_PASSWORD: snapotter # Change this for non-local deployments
POSTGRES_DB: snapotter
volumes:
- SnapOtter-pgdata:/var/lib/postgresql/data
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "pg_isready -U snapotter -d snapotter"]
interval: 10s
timeout: 5s
retries: 12
start_period: 15s
redis:
image: redis:8-alpine
command: ["redis-server", "--maxmemory-policy", "noeviction", "--appendonly", "yes"]
volumes:
- SnapOtter-redisdata:/data
restart: unless-stopped
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
timeout: 5s
retries: 12
start_period: 10s
volumes:
SnapOtter-data:
SnapOtter-workspace:
SnapOtter-pgdata:
SnapOtter-redisdata:
```
### Why `no-new-privileges` Is Not Set {#why-no-new-privileges-is-not-set}
`security_opt: [no-new-privileges:true]` is intentionally omitted. The entrypoint starts as root to fix volume ownership, then drops to the `snapotter` user via [gosu](https://github.com/tianon/gosu), which requires setuid. Once the privilege drop completes, the process runs as `snapotter` with all capabilities except the five listed above removed.
If you use Kubernetes or Docker's `--user` flag to run as non-root directly (bypassing gosu), `no-new-privileges` is safe to enable.
For an internet-facing deployment, bind port 1349 to loopback and terminate TLS at a maintained reverse proxy. Generate unique PostgreSQL and Redis credentials, store secrets in protected files or a secret manager, and change the initial administrator password immediately.
### Why `read_only` Is Not Set {#why-read-only-is-not-set}
@@ -116,21 +28,18 @@ If you use Kubernetes or Docker's `--user` flag to run as non-root directly (byp
## Network Isolation {#network-isolation}
During normal operation, the container makes **zero outbound network connections**. All file processing happens locally using bundled libraries.
File processing is local, but a default installation is **not an egress-free system**. Anonymous product analytics use PostHog and crash reporting uses Sentry when telemetry is enabled. Set `SNAPOTTER_TELEMETRY=0` (or disable analytics under Settings > System > Privacy) to turn off both. SnapOtter never includes uploaded files, file names, OCR output, document text, or other file contents in those events.
```
Browser --> Reverse Proxy (TLS) --> SnapOtter container --> (nothing)
```
The only exception is **AI model downloads**: when a user installs an AI feature bundle through the UI, the container downloads the pre-built bundle archive from Hugging Face, plus a few individual model files from GitHub Releases, Google Storage, and PyPI. These downloads happen once per bundle and are stored in the `/data` volume.
Other outbound traffic is feature-driven: AI bundle/model installation downloads signed release inputs; URL import fetches a user-requested public URL; and explicitly configured OIDC, SAML, OpenTelemetry, webhooks, S3-compatible storage, or similar integrations contact the destinations chosen by the administrator. Runtime model downloads are disabled by default. Set `SNAPOTTER_ALLOW_MODEL_DOWNLOAD=1` only to explicitly opt into automatic fallback downloads. An [offline bundle import](/guide/deployment) can provision AI features without runtime model egress.
**Firewall recommendations:**
| Scenario | Outbound rule |
|---|---|
| Air-gapped (no AI) | Block all outbound traffic from the container |
| AI bundles needed | Allow HTTPS to `huggingface.co`, `*.xethub.hf.co`, `cdn-lfs.huggingface.co`, `github.com`, `objects.githubusercontent.com`, `storage.googleapis.com`, `pypi.org`, `files.pythonhosted.org` during install, then block |
| After AI install | Block all outbound traffic - models are cached locally |
| Air-gapped | Set `SNAPOTTER_TELEMETRY=0` and `SNAPOTTER_ALLOW_MODEL_DOWNLOAD=0`, use offline AI bundle import, disable URL import and external integrations, then block egress |
| Default telemetry | Allow the PostHog and Sentry endpoints listed by your browser/network logs; disable telemetry if policy does not permit them |
| AI bundles needed | During installation, allow HTTPS to `huggingface.co, *.xethub.hf.co, cdn-lfs.huggingface.co, github.com, objects.githubusercontent.com, storage.googleapis.com, pypi.org, files.pythonhosted.org`; then block those hosts |
| External integrations | Allow only the exact administrator-configured OIDC/SAML/OTLP/webhook/object-storage destinations |
Bundle archives are served from Hugging Face's Xet storage, which transfers over the `*.xethub.hf.co` endpoints in parallel and is what makes multi-GB bundle downloads fast. If your firewall allows `huggingface.co` but blocks `*.xethub.hf.co`, installs still succeed but fall back to a slower single-stream download, so allowlist the Xet hosts to stay on the fast path. Fully offline installs can skip all of this and use [Offline Bundle Import](/guide/deployment) instead.
@@ -254,51 +163,58 @@ For resource sizing, see [Hardware Requirements](/guide/deployment#hardware-requ
## Backup and Recovery {#backup-and-recovery}
Persistent state is split across two volumes:
The production Compose stack defines four volumes. Stop ingress and let active jobs finish before taking a coordinated backup so PostgreSQL, Redis, and file state describe the same point in time.
| Volume | Contents | Critical? |
| Volume | Contents | Recovery treatment |
|---|---|---|
| `SnapOtter-pgdata` | PostgreSQL database (users, settings, pipelines, jobs, audit log) | Yes |
| `/data` (app volume) | User-uploaded files, AI models, Python venv | Partially (see below) |
| `SnapOtter-pgdata` | PostgreSQL users, settings, pipelines, jobs, file metadata, and audit log | Critical; use a fail-fast logical dump for portable recovery |
| `SnapOtter-data` | Saved library objects, logs, and AI state (`/data/files, /data/logs, /data/ai, /data/ai/venv`) | Back up the whole volume; to save space, deliberately omit all AI state and reinstall its bundles |
| `SnapOtter-redisdata` | Redis AOF for durable BullMQ queue state | Back up after pausing the app and forcing `SAVE`; required to resume queued work exactly |
| `SnapOtter-workspace` | Temporary object-storage keys (`/tmp/workspace/uploads, /tmp/workspace/outputs`) | Do not back up after all jobs are drained or cancelled; never discard it while jobs are active |
Within the `/data` volume:
| Path | Contents | Critical? |
|---|---|---|
| `/data/uploads/`, `/data/outputs/` | User files and processing results | Yes |
| `/data/ai/` | Downloaded AI model files | No (re-downloadable) |
| `/data/venv/` | Python virtual environment | No (rebuilt on start) |
Compose normally prefixes volume names with the project name. Resolve the real source volume from the mounted container instead of assuming that a display name such as `SnapOtter-data` is the Docker volume name.
### Database backup {#database-backup}
Use `pg_dump` to back up the database while the stack is running:
Use PostgreSQL's custom archive format and verify the archive before treating the backup as complete:
```bash
# Dump the database
docker exec SnapOtter-postgres pg_dump -U snapotter snapotter > backup.sql
docker exec SnapOtter-postgres \
pg_dump --format=custom --no-owner -U snapotter snapotter > snapotter.dump
test -s snapotter.dump
docker exec -i SnapOtter-postgres pg_restore --list < snapotter.dump >/dev/null
# Restore into a fresh database
cat backup.sql | docker exec -i SnapOtter-postgres psql -U snapotter snapotter
# Restore only into a fresh/disposable target first; any SQL error fails the command.
docker exec -i SnapOtter-postgres \
pg_restore --exit-on-error --clean --if-exists --no-owner \
-U snapotter -d snapotter < snapotter.dump
```
Alternatively, stop the stack and snapshot the `SnapOtter-pgdata` volume:
Test every backup by restoring it into an isolated stack, checking database records and file checksums, and starting the application. The repository's `tests/qa/backup-restore-drill.sh` automates that release gate against an explicit `QA_IMAGE`.
If your platform takes crash-consistent volume snapshots instead, stop the entire stack first and snapshot all critical volumes as one set. A raw PostgreSQL data-directory copy from a running container is not a supported logical backup.
### File and queue backup {#file-and-queue-backup}
Pause the application before capturing file and queue volumes. Use `docker inspect` to resolve the actual volume name, force Redis to persist its current state, and archive with ownership and permissions preserved:
```bash
docker compose down
docker run --rm -v SnapOtter-pgdata:/data -v $(pwd)/backup:/backup \
alpine tar czf /backup/snapotter-pgdata.tar.gz -C /data .
docker stop SnapOtter
docker exec SnapOtter-redis redis-cli -a "$REDIS_PASSWORD" --no-auth-warning SAVE
docker stop SnapOtter-redis
DATA_VOLUME="$(docker inspect SnapOtter --format '{{range .Mounts}}{{if eq .Destination "/data"}}{{.Name}}{{end}}{{end}}')"
REDIS_VOLUME="$(docker inspect SnapOtter-redis --format '{{range .Mounts}}{{if eq .Destination "/data"}}{{.Name}}{{end}}{{end}}')"
install -d -m 700 backup
docker run --rm -v "$DATA_VOLUME:/source:ro" -v "$PWD/backup:/backup" \
alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce tar czf /backup/snapotter-data.tar.gz -C /source .
docker run --rm -v "$REDIS_VOLUME:/source:ro" -v "$PWD/backup:/backup" \
alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce tar czf /backup/snapotter-redis.tar.gz -C /source .
sha256sum backup/snapotter-*.tar.gz > backup/SHA256SUMS
```
### User files backup {#user-files-backup}
```bash
# Snapshot the app data volume (excluding re-downloadable AI models)
docker run --rm -v SnapOtter-data:/data -v $(pwd)/backup:/backup \
alpine tar czf /backup/snapotter-files.tar.gz \
--exclude='ai' --exclude='venv' -C /data .
```
AI models total up to about 24 GB across all bundles. Since they are re-downloadable, exclude `/data/ai/` and `/data/venv/` from backups to save space. Only the database and user files are critical.
Restart Redis before the application. If you intentionally exclude `/data/ai`, remove the whole AI subtree rather than preserving an `installed.json` record without its models or virtual environment. Keep backup files encrypted, access-controlled, and separate from the host running SnapOtter.
## Compliance Artifacts {#compliance-artifacts}
@@ -306,9 +222,10 @@ Each SnapOtter release includes the following security artifacts:
| Artifact | Format | Where to find it |
|---|---|---|
| SBOM (CycloneDX) | JSON | [GitHub Release](https://github.com/snapotter-hq/SnapOtter/releases) asset: `snapotter-v{version}-sbom.cdx.json` |
| SBOM (SPDX) | JSON | [GitHub Release](https://github.com/snapotter-hq/SnapOtter/releases) asset: `snapotter-v{version}-sbom.spdx.json` |
| Vulnerability scan | Trivy JSON | [GitHub Release](https://github.com/snapotter-hq/SnapOtter/releases) asset: `snapotter-v{version}-trivy.json` |
| Release subject binding | Canonical JSON + GitHub attestation | [GitHub Release](https://github.com/snapotter-hq/SnapOtter/releases) asset: `snapotter-v{version}-release-subjects.json` |
| Archive SBOM | CycloneDX and SPDX JSON | Release assets: `snapotter-v{version}-archive-linux-{arch}-sbom.{cdx,spdx}.json` |
| Image SBOM | CycloneDX and SPDX JSON | Release assets: `snapotter-v{version}-image-linux-{arch}-sbom.{cdx,spdx}.json` |
| Vulnerability scans | Trivy JSON | Release assets with matching `archive-linux-{arch}` or `image-linux-{arch}` prefixes |
| Vulnerability scan | SARIF | [GitHub Security](https://github.com/snapotter-hq/SnapOtter/security) tab |
| Static analysis | CodeQL (JS/TS + Python) | [GitHub Security](https://github.com/snapotter-hq/SnapOtter/security) tab, runs weekly + per PR |
| Dependency review | GitHub native | Per-PR check, fails on high-severity additions |
@@ -318,19 +235,29 @@ Each SnapOtter release includes the following security artifacts:
**Running your own scan:**
Download the SBOM from the release and scan it with your preferred tool:
Download the release-subject manifest and verify that it was attested by the release workflow:
```bash
gh attestation verify snapotter-v2.1.0-release-subjects.json \
--repo snapotter-hq/SnapOtter \
--signer-workflow snapotter-hq/SnapOtter/.github/workflows/release.yml
```
The manifest records `releaseTag`, `releaseCommit`, and `workflowTriggerCommit` separately. Verify that `releaseCommit` is the commit peeled from the immutable tag, then verify the SHA-256 digest of the archive, image, SBOM, or scan you consume against its entry in `subjects`. This distinction is intentional: checking out a newly created release commit does not change the commit identity in the workflow's OIDC credential.
You can also scan a downloaded SBOM or the image directly:
```bash
# Scan with Grype using the CycloneDX SBOM
grype sbom:snapotter-v1.17.2-sbom.cdx.json
grype sbom:snapotter-v2.1.0-image-linux-amd64-sbom.cdx.json
# Scan with Trivy using the SPDX SBOM
trivy sbom snapotter-v1.17.2-sbom.spdx.json
trivy sbom snapotter-v2.1.0-image-linux-amd64-sbom.spdx.json
# Scan the Docker image directly
trivy image snapotter/snapotter:1.17.2
trivy image snapotter/snapotter:2.1.0
```
::: info
The SBOM and vulnerability scan reflect the exact image published for that release. AI model bundles installed after deployment are not included in the SBOM since they are downloaded at runtime.
Image SBOMs and scans reflect the exact architecture-specific image published for that release. Archive SBOMs and scans describe the prebuilt archive separately. AI model bundles installed after deployment are not included in these SBOMs because they are downloaded at runtime.
:::
apps/docs/guide/supported-formats.md
+6 -2
View File
@@ -8,7 +8,7 @@ SnapOtter processes files across five modalities: image, video, audio, PDF, and
## Image Formats {#image-formats}
SnapOtter supports 55+ image formats for input and 13 formats for output.
SnapOtter supports 55+ image formats for input and 17 formats for output.
## Input Formats {#input-formats}
@@ -101,7 +101,7 @@ SnapOtter supports 55+ image formats for input and 13 formats for output.
| PAM | .pam | Sharp (native) | Arbitrary map |
| PFM | .pfm | Sharp (native) | Float map |
## Output Formats (13) {#output-formats-13}
## Output Formats (17) {#output-formats-13}
| Format | Encoder | Quality Control | Available In |
|--------|---------|----------------|-------------|
@@ -118,6 +118,10 @@ SnapOtter supports 55+ image formats for input and 13 formats for output.
| ICO | ImageMagick CLI | Lossless | Convert tool |
| JP2 | opj_compress CLI | Compression ratio | Convert tool |
| QOI | Inline codec | Lossless | Convert tool |
| PSD | ImageMagick CLI | Lossless | Convert tool |
| PPM | ImageMagick CLI | Lossless | Convert tool |
| EPS | ImageMagick CLI | Lossless | Convert tool |
| TGA | ImageMagick CLI | Lossless | Convert tool |
## Video Formats {#video-formats}