feat: all-in-one embedded single-container mode (#377)

Embedded Postgres 17 + Redis via s6-overlay when DATABASE_URL/REDIS_URL are unset; restores the one-command docker run for 2.0. EMBEDDED=0 disables; Compose stays the production path. Verified arm64 (14/14 lifecycle + Compose regression) and amd64 (build + embedded smoke).
This commit is contained in:
SnapOtter
2026-06-29 15:49:09 +08:00
committed by GitHub
parent 0cdd560ac4
commit 084a9c9faa
36 changed files with 656 additions and 24 deletions
+33
View File
@@ -0,0 +1,33 @@
name: Embedded Image Lifecycle
# Heavy and manual on purpose. This builds the full production image (~8GB on
# amd64 because of the CUDA base) and boots it through the embedded single
# container lifecycle. That is too large and slow for PR or scheduled CI, so it
# runs only from the Actions tab. Run it when touching docker/ (entrypoint, the
# s6 tree, embedded-lib, the Dockerfile) or tests/e2e-docker/embedded-mode.mjs.
# It frees runner disk first so the image fits on a standard ubuntu-latest
# runner; if that ever stops being enough, move this job to a self-hosted runner.
on:
workflow_dispatch:
permissions:
contents: read
jobs:
embedded-image:
name: Build + embedded lifecycle (amd64)
runs-on: ubuntu-latest
timeout-minutes: 90
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Free runner disk space
run: |
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc \
/opt/hostedtoolcache/CodeQL /usr/local/share/boost
sudo docker image prune -af || true
df -h /
- name: Build production image (amd64, CPU)
run: docker build -f docker/Dockerfile -t snapotter:embed-ci .
- name: Embedded-mode lifecycle test
run: SNAPOTTER_IMAGE=snapotter:embed-ci node tests/e2e-docker/embedded-mode.mjs
+8 -2
View File
@@ -41,13 +41,19 @@ Stirling-PDF stops at PDFs. ConvertX stops at conversions. SnapOtter runs all fi
- **21 languages:** English, Arabic, Chinese (Simplified & Traditional), Dutch, French, German, Hindi, Indonesian, Italian, Japanese, Korean, Polish, Portuguese, Russian, Spanish, Swedish, Thai, Turkish, Ukrainian, Vietnamese. RTL support for Arabic
- **Pipelines:** Chain tools into reusable workflows with unlimited steps. Import/export as JSON. Batch process unlimited files at once
- **REST API:** Every tool available via API with API key auth. Interactive docs at `/api/docs`
- **Self-hosted stack:** SnapOtter + Postgres 17 + Redis 8, run together with one `docker compose up`. No external SaaS dependencies
- **Self-hosted:** one `docker run` for a single-container quick start (embedded Postgres + Redis), or a Postgres 17 + Redis 8 Compose stack for production. No external SaaS dependencies
- **Multi-arch:** Runs on AMD64 and ARM64 (Intel, Apple Silicon, Raspberry Pi)
- **Privacy first:** Your files never leave your network. Basic analytics help us catch bugs and improve tools -- disable anytime by rebuilding with `SNAPOTTER_ANALYTICS=off` ([Here's how to do it](https://docs.snapotter.com/guide/deployment.html#analytics))
## Quick Start
SnapOtter runs as a small Docker Compose stack (app + Postgres 17 + Redis 8). Save this as `compose.yaml`:
One container, no setup. It starts an embedded Postgres + Redis on loopback and stores data in the `SnapOtter-data` volume:
```bash
docker run -d --name SnapOtter -p 1349:1349 -v SnapOtter-data:/data snapotter/snapotter:latest
```
For production, run the 3-container Compose stack (app + Postgres 17 + Redis 8). Save this as `compose.yaml`:
```yaml
services:
+13
View File
@@ -39,6 +39,19 @@ All configuration is done through environment variables. Every variable has a se
| `WORKSPACE_PATH` | `./tmp/workspace` | Directory for temporary files during processing. Cleaned up automatically. |
| `FILES_STORAGE_PATH` | `./data/files` | Directory for persistent user files (uploaded images, saved results). |
### Embedded mode
Run the image with no `DATABASE_URL` and no `REDIS_URL` and it starts its own PostgreSQL 17 and Redis inside the container, bound to loopback, with all data on the `/data` volume. This restores the single-command `docker run` experience for quick start, homelab, and upgrades from 1.x. It is a convenience path, not a production deployment: for production, run the 3-container Compose stack with separate PostgreSQL and Redis. Embedded mode requires running the container as root and is incompatible with arbitrary-UID runtimes (OpenShift, Kubernetes `runAsNonRoot`); use Compose there.
| Variable | Default | Description |
|---|---|---|
| `EMBEDDED` | `auto` | Auto-enabled when both `DATABASE_URL` and `REDIS_URL` are unset. Set to `0` to disable it (the app then fails fast if no external `DATABASE_URL`/`REDIS_URL` is set, rather than silently starting an in-container database). |
| `REDIS_MAXMEMORY` | `512mb` | Memory cap for the embedded Redis (embedded mode only). Lower it on memory-constrained hosts such as a Raspberry Pi. |
Upgrading from 1.x: put your old `snapotter.db` at `/data/snapotter.db` in the volume and embedded mode imports it into the embedded PostgreSQL on first boot. The import runs once; later boots skip it.
Telemetry note: embedded mode inherits the image's analytics default like any other configuration. The published image ships with analytics on; build with `--build-arg SNAPOTTER_ANALYTICS=off`, or use the in-app admin opt-out, to disable it.
### Processing limits
| Variable | Default | Description |
+3 -1
View File
@@ -4,7 +4,7 @@ description: SnapOtter Docker image tags, GPU benchmarks, version pinning, and m
# Docker Image
SnapOtter ships as a Docker image that runs alongside PostgreSQL 17 and Redis 8 in a Compose stack. The app image works on all platforms.
SnapOtter ships as a single Docker image. Run it on its own and it starts an embedded PostgreSQL 17 and Redis on the loopback interface (embedded mode); for production, run it alongside separate PostgreSQL 17 and Redis 8 containers with Compose. The app image works on all platforms.
## Quick start
@@ -12,6 +12,8 @@ SnapOtter ships as a Docker image that runs alongside PostgreSQL 17 and Redis 8
docker run -d --name SnapOtter -p 1349:1349 -v SnapOtter-data:/data snapotter/snapotter:latest
```
With no `DATABASE_URL` set, this runs in embedded mode: PostgreSQL and Redis start inside the container on loopback, with all data under the `SnapOtter-data` volume. Set `DATABASE_URL` and `REDIS_URL` (as the [Compose](#docker-compose) stack does) to use external services instead. See [Configuration](/guide/configuration#embedded-mode).
## GPU acceleration
The image includes CUDA support on amd64. If you have an NVIDIA GPU with the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) installed, add `--gpus all`:
+2
View File
@@ -14,6 +14,8 @@ Explore the full UI at [demo.snapotter.com](https://demo.snapotter.com) - no sig
docker run -d --name SnapOtter -p 1349:1349 -v SnapOtter-data:/data snapotter/snapotter:latest
```
This single container runs everything it needs: with no `DATABASE_URL` set, it starts its own PostgreSQL and Redis on the loopback interface (embedded mode) and keeps all data in the `SnapOtter-data` volume. It is the fastest way to try SnapOtter or self-host on a homelab. For production, run the [Docker Compose](#docker-compose) stack below, which keeps PostgreSQL and Redis in their own containers. Embedded mode runs as root (the default) and turns off automatically as soon as you set `DATABASE_URL`.
You will be asked to change your password on first login.
::: tip NVIDIA GPU acceleration
+59 -4
View File
@@ -242,6 +242,7 @@ RUN for i in 1 2 3; do apt-get -o Acquire::Retries=3 update && break || sleep $(
libopenjp2-tools \
curl \
gosu \
xz-utils \
libde265-0 \
libimage-exiftool-perl \
python3 python3-pip python3-venv python3-dev \
@@ -273,6 +274,42 @@ RUN for i in 1 2 3; do apt-get -o Acquire::Retries=3 update && break || sleep $(
fi \
&& rm -rf /var/lib/apt/lists/*
# Embedded-mode databases: PostgreSQL 17 (PGDG, to match the Compose postgres:17
# major for a clean data handoff) + Redis. Shipped in every image (single tag);
# unused in external/Compose mode, ~tens of MB against the multi-GB base. They
# enter the Trivy CVE surface and ride the existing apt-get upgrade patching.
RUN install -d /usr/share/postgresql-common/pgdg \
&& curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \
-o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc \
&& . /etc/os-release \
&& echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt ${VERSION_CODENAME}-pgdg main" \
> /etc/apt/sources.list.d/pgdg.list \
&& for i in 1 2 3; do apt-get -o Acquire::Retries=3 update && break || sleep $((i * 15)); done \
&& apt-get install -y --no-install-recommends postgresql-17 postgresql-client-17 redis-server \
&& rm -rf /var/lib/apt/lists/*
# s6-overlay supervises the embedded service tree (postgres + redis + app).
# Pinned and checksum-verified against the upstream-published .sha256, consistent
# with the repo's digest-pinning posture. For stricter supply-chain pinning,
# replace the .sha256 fetch with a literal hash checked via
# `echo "<hash> <file>" | sha256sum -c -`.
ARG S6_OVERLAY_VERSION=3.2.0.2
RUN set -e; \
case "$TARGETARCH" in \
amd64) S6_ARCH=x86_64 ;; \
arm64) S6_ARCH=aarch64 ;; \
*) echo "unsupported TARGETARCH=$TARGETARCH" >&2; exit 1 ;; \
esac; \
cd /tmp; \
base="https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}"; \
for f in "s6-overlay-noarch.tar.xz" "s6-overlay-${S6_ARCH}.tar.xz"; do \
curl -fsSL -O "${base}/${f}"; \
curl -fsSL -O "${base}/${f}.sha256"; \
sha256sum -c "${f}.sha256"; \
tar -C / -Jxpf "${f}"; \
done; \
rm -f /tmp/s6-overlay-*
# Allow ImageMagick to use Ghostscript delegate for EPS (read for decode,
# write for the convert tool's EPS output). PS/PDF/XPS stay read-only.
RUN POLICY_FILE=$(find /etc/ImageMagick* -name policy.xml 2>/dev/null | head -1) && \
@@ -440,6 +477,13 @@ ENV SENTRY_RELEASE=${SENTRY_RELEASE}
ENV NVIDIA_VISIBLE_DEVICES=all \
NVIDIA_DRIVER_CAPABILITIES=compute,utility
# s6-overlay (embedded mode): propagate the runtime-exported environment (the
# 127.0.0.1 URLs, resolved _FILE secrets, auth defaults) into supervised
# services, and never time out waiting for first-boot readiness (initdb can take
# minutes). Inert in external mode, which never invokes s6.
ENV S6_KEEP_ENV=1 \
S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0
# Suppress noisy ML library output in docker logs
ENV PYTHONWARNINGS=default \
TF_CPP_MIN_LOG_LEVEL=3 \
@@ -463,14 +507,25 @@ RUN chown -R snapotter:snapotter /app /opt/venv && \
# entrypoint-lib.sh holds the writability helpers it sources at startup.
COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh
COPY docker/entrypoint-lib.sh /usr/local/bin/entrypoint-lib.sh
COPY docker/embedded-lib.sh /usr/local/bin/embedded-lib.sh
COPY docker/embedded/postgres-bootstrap.sh /usr/local/bin/embedded-postgres-bootstrap.sh
COPY docker/wait-for-postgres.mjs /app/docker/wait-for-postgres.mjs
RUN chmod +x /usr/local/bin/entrypoint.sh
# s6-overlay reads its service tree from /etc/s6-overlay/s6-rc.d (embedded mode)
COPY docker/s6/s6-rc.d /etc/s6-overlay/s6-rc.d
RUN chmod +x /usr/local/bin/entrypoint.sh /usr/local/bin/embedded-postgres-bootstrap.sh \
&& chmod +x /etc/s6-overlay/s6-rc.d/postgres/run /etc/s6-overlay/s6-rc.d/redis/run \
/etc/s6-overlay/s6-rc.d/snapotter/run \
/etc/s6-overlay/s6-rc.d/postgres-init/up /etc/s6-overlay/s6-rc.d/postgres-ready/up \
/etc/s6-overlay/s6-rc.d/redis-ready/up
EXPOSE 1349
HEALTHCHECK --interval=30s --timeout=5s --start-period=60s --retries=3 \
HEALTHCHECK --interval=30s --timeout=5s --start-period=180s --retries=3 \
CMD curl -sf --max-time 5 http://localhost:1349/api/v1/health || exit 1
# tini as PID 1 for zombie reaping + signal forwarding
ENTRYPOINT ["tini", "--", "entrypoint.sh"]
# entrypoint.sh runs as PID 1 and re-execs the right init: `tini` for external
# mode (zombie reaping + signal forwarding for the gosu-dropped app, same end
# state as before), or s6-overlay's /init as PID 1 for embedded mode (which
# s6-overlay-suexec requires).
ENTRYPOINT ["entrypoint.sh"]
CMD ["pnpm", "--filter", "@snapotter/api", "run", "start"]
+87
View File
@@ -0,0 +1,87 @@
#!/bin/sh
# Shared helpers for SnapOtter embedded mode (in-container Postgres + Redis).
# Sourced by docker/entrypoint.sh and the s6 service scripts. Kept in its own
# file so the decision logic can be unit-tested directly
# (tests/unit/security/embedded-mode.test.ts) rather than mirrored. Sourcing has
# no side effects, only function definitions. POSIX sh only (no bashisms).
#
# Functions use _-prefixed locals (sh has no portable `local`).
# decide_run_mode
# Echoes "embedded" or "external" and returns 0, OR prints a fatal partial-config
# error to stderr and returns 2. Embedded requires BOTH DATABASE_URL and
# REDIS_URL unset and EMBEDDED != 0. Exactly one URL set is an ambiguous
# misconfiguration and is rejected.
decide_run_mode() {
if [ "${EMBEDDED:-auto}" = "0" ]; then
echo "external"
return 0
fi
if [ -z "${DATABASE_URL:-}" ] && [ -z "${REDIS_URL:-}" ]; then
echo "embedded"
return 0
fi
if [ -n "${DATABASE_URL:-}" ] && [ -n "${REDIS_URL:-}" ]; then
echo "external"
return 0
fi
echo "FATAL: set BOTH DATABASE_URL and REDIS_URL (external mode), or NEITHER (embedded mode)." >&2
echo "Exactly one is set, which is ambiguous. Refusing to guess." >&2
return 2
}
# embedded_requires_root <uid>
# Embedded mode needs root to initdb, chown PGDATA, run Postgres as the postgres
# user, and s6-setuidgid per service. Arbitrary-UID runtimes (OpenShift,
# Kubernetes runAsNonRoot, `docker run --user`) cannot do this. Returns 0 when
# uid is 0, otherwise prints guidance and returns 1.
embedded_requires_root() {
_err_uid="$1"
if [ "$_err_uid" = "0" ]; then
return 0
fi
echo "FATAL: embedded mode needs root to run the in-container database (uid=$_err_uid)." >&2
echo "Run the container as root (the default), or use the Compose 3-container stack," >&2
echo "or set DATABASE_URL + REDIS_URL to point at external services." >&2
return 1
}
# sqlite_autodetect_path <data_dir>
# Echoes the SQLite path the app should import on first boot, or empty. An
# explicit SQLITE_MIGRATE_PATH always wins. Otherwise, if <data_dir>/snapotter.db
# exists (a 1.x single-container database), echo it so embedded mode upgrades in
# place. The importer itself no-ops when the target Postgres is non-empty, so a
# second boot does not re-import.
sqlite_autodetect_path() {
_sap_dir="$1"
if [ -n "${SQLITE_MIGRATE_PATH:-}" ]; then
echo "$SQLITE_MIGRATE_PATH"
return 0
fi
if [ -f "$_sap_dir/snapotter.db" ]; then
echo "$_sap_dir/snapotter.db"
return 0
fi
echo ""
}
# check_pg_version <pgdata> <installed_major>
# Guards against a silent major-version mismatch. If <pgdata>/PG_VERSION exists
# and its major differs from <installed_major>, print actionable guidance and
# return 1 (never auto-pg_upgrade, never overwrite). Returns 0 when it matches or
# when the data dir is fresh (no PG_VERSION yet).
check_pg_version() {
_cpv_data="$1"
_cpv_installed="$2"
if [ ! -f "$_cpv_data/PG_VERSION" ]; then
return 0
fi
_cpv_found="$(tr -d '[:space:]' < "$_cpv_data/PG_VERSION" 2>/dev/null)"
if [ "$_cpv_found" = "$_cpv_installed" ]; then
return 0
fi
echo "FATAL: $_cpv_data was created by PostgreSQL $_cpv_found, but this image ships PostgreSQL $_cpv_installed." >&2
echo "Major-version upgrades are a manual procedure (pg_dump from the old major, restore into the new)." >&2
echo "See the embedded-mode upgrade docs. Refusing to start to avoid data corruption." >&2
return 1
}
+52
View File
@@ -0,0 +1,52 @@
#!/bin/sh
# First-boot initializer for the embedded Postgres. Runs as ROOT inside the s6
# `postgres-init` oneshot, before the `postgres` longrun starts. Idempotent: on a
# data dir that already exists it only guards the version and fixes ownership.
set -e
. /usr/local/bin/embedded-lib.sh
PGDATA=/data/postgres
PGBIN=/usr/lib/postgresql/17/bin
INSTALLED_MAJOR=17
TMP=/data/postgres.bootstrapping # same filesystem as PGDATA so the mv is atomic
# Clean any interrupted previous bootstrap.
rm -rf "$TMP"
# Existing data dir: guard the major version, fix ownership, done.
if [ -f "$PGDATA/PG_VERSION" ]; then
check_pg_version "$PGDATA" "$INSTALLED_MAJOR" || exit 1
chown -R postgres:postgres "$PGDATA"
echo "Embedded Postgres: existing data dir OK (major $INSTALLED_MAJOR)."
exit 0
fi
echo "Embedded Postgres: first-boot initdb..."
install -d -o postgres -g postgres -m 700 "$TMP"
# initdb: C locale (byte-ordered, libc-independent collation, so the data dir is
# safe across the glibc/musl handoff to a Compose postgres:17-alpine), trust auth
# on loopback (the only reachable interface), bootstrap superuser `snapotter` so
# the role in DATABASE_URL already exists.
s6-setuidgid postgres "$PGBIN/initdb" -D "$TMP" \
--username=snapotter --encoding=UTF8 --locale=C \
--auth-local=trust --auth-host=trust
# Loopback only, and avoid the 64MB /dev/shm for parallel workers.
{
echo "listen_addresses = '127.0.0.1'"
echo "dynamic_shared_memory_type = mmap"
} >> "$TMP/postgresql.conf"
# Create the application database and set the role password via single-user mode:
# no socket, no listener, no /var/run/postgresql, auth bypassed. The snapotter
# superuser already exists from initdb --username. The password is harmless under
# trust auth but lets a future scram flip work without a reinit.
echo "CREATE DATABASE snapotter OWNER snapotter;" | \
s6-setuidgid postgres "$PGBIN/postgres" --single -D "$TMP" postgres
echo "ALTER ROLE snapotter WITH PASSWORD 'snapotter';" | \
s6-setuidgid postgres "$PGBIN/postgres" --single -D "$TMP" postgres
# Atomic publish: a crash before this leaves only the throwaway temp dir.
mv "$TMP" "$PGDATA"
echo "Embedded Postgres: initialized."
+61 -14
View File
@@ -4,6 +4,9 @@ set -e
# Shared permission helpers (dir_writable, ensure_writable). Lives beside this
# script in the image; sourcing only defines functions (no side effects).
. /usr/local/bin/entrypoint-lib.sh
# Embedded-mode helpers (decide_run_mode, embedded_requires_root,
# sqlite_autodetect_path, check_pg_version). Same no-side-effects contract.
. /usr/local/bin/embedded-lib.sh
# --- Docker secret file convention (_FILE suffix) ---
# For each supported var, if VAR_FILE is set, read the secret from that file
@@ -49,6 +52,25 @@ export AUTH_ENABLED="${AUTH_ENABLED:-true}"
export DEFAULT_USERNAME="${DEFAULT_USERNAME:-admin}"
export DEFAULT_PASSWORD="${DEFAULT_PASSWORD:-admin}"
# Embedded mode detection (in-container Postgres + Redis). Decide the run mode
# before any DB-dependent step. EMBEDDED_MODE is exported so later branches (the
# wait loop, security warnings, chown, final exec) can key off it.
RUN_MODE="$(decide_run_mode)" || exit $? # exits 2 on ambiguous partial config
if [ "$RUN_MODE" = "embedded" ]; then
embedded_requires_root "$(id -u)" || exit 1
EMBEDDED_MODE=1
export EMBEDDED_MODE
export DATABASE_URL="postgres://snapotter:snapotter@127.0.0.1:5432/snapotter"
export REDIS_URL="redis://127.0.0.1:6379"
# 1.x single-container upgrade: auto-import /data/snapotter.db on first boot
# unless the operator set an explicit path. Importer no-ops on a non-empty DB.
SQLITE_MIGRATE_PATH="$(sqlite_autodetect_path "${DATA_DIR:-/data}")"
export SQLITE_MIGRATE_PATH
printf '\n \033[1;33m🦦 SnapOtter embedded mode\033[0m\n' >&2
printf ' \033[2mRunning an in-container Postgres + Redis for quick start.\033[0m\n' >&2
printf ' \033[2mFor production, use the Compose 3-container stack. See docs.\033[0m\n\n' >&2
fi
# Writable directories the runtime needs: WS = processing scratch, DD = data
# root (holds files, logs, and the AI venv/models). Honor env overrides.
WS="${WORKSPACE_PATH:-/tmp/workspace}"
@@ -110,8 +132,9 @@ if [ -d "/opt/venv" ]; then
fi
fi
# Wait for Postgres to be reachable before starting the app
if [ -n "${DATABASE_URL:-}" ]; then
# Wait for Postgres to be reachable before starting the app (external mode only;
# in embedded mode s6 starts Postgres and gates the app on a pg_isready oneshot).
if [ -z "${EMBEDDED_MODE:-}" ] && [ -n "${DATABASE_URL:-}" ]; then
echo "Waiting for Postgres..."
i=0
until node /app/docker/wait-for-postgres.mjs; do
@@ -126,11 +149,15 @@ print_security_warnings() {
if [ "${DEFAULT_PASSWORD}" = "admin" ]; then
printf ' \033[33mWARNING:%b Default admin password is still "admin". Change it for any non-local deployment.\n' '\033[0m' >&2
fi
if echo "${DATABASE_URL:-}" | grep -q "snapotter:snapotter@"; then
printf ' \033[33mWARNING:%b Default Postgres credentials in use. Set POSTGRES_PASSWORD for production.\n' '\033[0m' >&2
fi
if echo "${REDIS_URL:-}" | grep -q ":snapotter@"; then
printf ' \033[33mWARNING:%b Default Redis password in use. Set REDIS_PASSWORD for production.\n' '\033[0m' >&2
# Embedded Postgres/Redis are loopback-only and never network-exposed, so the
# default-credential warnings below only apply to external (Compose) mode.
if [ -z "${EMBEDDED_MODE:-}" ]; then
if echo "${DATABASE_URL:-}" | grep -q "snapotter:snapotter@"; then
printf ' \033[33mWARNING:%b Default Postgres credentials in use. Set POSTGRES_PASSWORD for production.\n' '\033[0m' >&2
fi
if echo "${REDIS_URL:-}" | grep -q ":snapotter@"; then
printf ' \033[33mWARNING:%b Default Redis password in use. Set REDIS_PASSWORD for production.\n' '\033[0m' >&2
fi
fi
}
@@ -187,11 +214,23 @@ if [ "$(id -u)" = "0" ]; then
# Ensure all writable subdirectories exist before chown
mkdir -p /data/files /data/logs /data/ai/models /data/ai/pip-cache /data/ai/venv /tmp/workspace
[ -n "${EMBEDDED_MODE:-}" ] && mkdir -p /data/redis
# Chown writable directories (/data is the persistent volume, /tmp/workspace is ephemeral).
# /app and /opt/venv are read-only at runtime -- no chown needed.
chown -R snapotter:snapotter /data /tmp/workspace 2>&1 || \
echo "WARNING: Could not fix volume permissions. Use named volumes (not Windows bind mounts) to avoid this. See docs for details." >&2
# Chown writable directories (/data is the persistent volume, /tmp/workspace is
# ephemeral). /app and /opt/venv are read-only at runtime, so no chown needed.
# Embedded carve-out: /data/postgres must stay owned by the postgres user
# (Postgres refuses a foreign-owned PGDATA), so exclude it from the snapotter
# sweep and chown it separately when it already exists.
if [ -n "${EMBEDDED_MODE:-}" ]; then
find /data -mindepth 1 -maxdepth 1 ! -name postgres -exec chown -R snapotter:snapotter {} + 2>&1 || \
echo "WARNING: Could not fix /data permissions. Use named volumes to avoid this. See docs." >&2
chown snapotter:snapotter /data 2>/dev/null || true
chown -R snapotter:snapotter /tmp/workspace 2>&1 || true
[ -d /data/postgres ] && chown -R postgres:postgres /data/postgres 2>/dev/null || true
else
chown -R snapotter:snapotter /data /tmp/workspace 2>&1 || \
echo "WARNING: Could not fix volume permissions. Use named volumes (not Windows bind mounts) to avoid this. See docs for details." >&2
fi
# Root can write anywhere, so verify as the unprivileged snapotter user that
# actually runs the app. This catches root-squashed or foreign-owned mounts
@@ -201,9 +240,17 @@ if [ "$(id -u)" = "0" ]; then
fi
print_banner
exec gosu snapotter "$@"
if [ -n "${EMBEDDED_MODE:-}" ]; then
# Hand off to s6-overlay as PID 1 (s6-overlay-suexec requires PID 1); it
# supervises postgres + redis + the app and drops privileges per service.
exec /init
fi
# External mode: tini becomes PID 1 (reaps zombies, forwards signals) and runs
# the app as snapotter, the same end state as the prior tini ENTRYPOINT.
exec tini -- gosu snapotter "$@"
fi
# Already running as snapotter (e.g. Kubernetes runAsUser)
# Already running as snapotter (e.g. Kubernetes runAsUser). External mode only:
# embedded mode requires root and exited earlier. tini becomes PID 1.
print_banner
exec "$@"
exec tini -- "$@"
+1
View File
@@ -0,0 +1 @@
oneshot
+2
View File
@@ -0,0 +1,2 @@
#!/command/with-contenv sh
/usr/local/bin/embedded-postgres-bootstrap.sh
+1
View File
@@ -0,0 +1 @@
oneshot
+1
View File
@@ -0,0 +1 @@
/command/with-contenv sh -c "until pg_isready -h 127.0.0.1 -p 5432 -q; do sleep 1; done"
+1
View File
@@ -0,0 +1 @@
SIGINT
+2
View File
@@ -0,0 +1,2 @@
#!/command/with-contenv sh
exec s6-setuidgid postgres /usr/lib/postgresql/17/bin/postgres -D /data/postgres
+1
View File
@@ -0,0 +1 @@
longrun
+1
View File
@@ -0,0 +1 @@
oneshot
+1
View File
@@ -0,0 +1 @@
/command/with-contenv sh -c "until redis-cli -h 127.0.0.1 -p 6379 ping 2>/dev/null | grep -q PONG; do sleep 1; done"
+8
View File
@@ -0,0 +1,8 @@
#!/command/with-contenv sh
exec s6-setuidgid snapotter redis-server \
--dir /data/redis \
--bind 127.0.0.1 \
--port 6379 \
--maxmemory-policy noeviction \
--appendonly yes \
--maxmemory "${REDIS_MAXMEMORY:-512mb}"
+1
View File
@@ -0,0 +1 @@
longrun
+3
View File
@@ -0,0 +1,3 @@
#!/command/with-contenv sh
cd /app
exec s6-setuidgid snapotter pnpm --filter @snapotter/api run start
+1
View File
@@ -0,0 +1 @@
longrun
+7 -3
View File
@@ -1,12 +1,16 @@
# SnapOtter
Open-source, self-hostable file processing suite with 200+ tools across image, video, audio, PDF, and files, plus a layer-based image editor and local AI. Runs as a small Docker Compose stack, no external services. Dual-licensed AGPLv3 and commercial.
Open-source, self-hostable file processing suite with 200+ tools across image, video, audio, PDF, and files, plus a layer-based image editor and local AI. Runs as a single container with embedded Postgres + Redis for quick start, or a Docker Compose stack for production. No external services. Dual-licensed AGPLv3 and commercial.
All processing happens locally. Files never leave your infrastructure.
## Quick Start (Docker Compose)
## Quick Start
SnapOtter runs alongside PostgreSQL 17 and Redis 8. Minimal compose.yaml:
Single container (embedded Postgres + Redis), the fastest way to start:
docker run -d --name SnapOtter -p 1349:1349 -v SnapOtter-data:/data snapotter/snapotter:latest
For production, run alongside PostgreSQL 17 and Redis 8 with Compose. Minimal compose.yaml:
services:
snapotter:
+181
View File
@@ -0,0 +1,181 @@
#!/usr/bin/env node
// Embedded-mode container lifecycle tests. Requires Docker and a built image.
// Usage: SNAPOTTER_IMAGE=snapotter:embed-wip node tests/e2e-docker/embedded-mode.mjs
//
// Unlike the Playwright e2e-docker specs (which talk to an already-running
// container), these drive `docker run`/`stop` directly to exercise the container
// lifecycle: bare-run boot, restart persistence, clean shutdown, the non-root and
// partial-config fail-fast guards, and the 1.x SQLite auto-detect upgrade.
import { execFileSync, spawnSync } from "node:child_process";
const IMAGE = process.env.SNAPOTTER_IMAGE || "snapotter:embed-wip";
const NAME = "so-embed-test";
const VOL = "so-embed-test-data";
const PORT = process.env.SNAPOTTER_TEST_PORT || "13492";
let failures = 0;
const ok = (m) => console.log(` PASS ${m}`);
const bad = (m) => {
failures++;
console.error(` FAIL ${m}`);
};
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
// Run docker, returning stdout. Throws on non-zero exit (callers that expect a
// non-zero exit wrap in try/catch and read combined output from the error).
const docker = (args) =>
execFileSync("docker", args, { encoding: "utf-8", stdio: ["ignore", "pipe", "pipe"] });
const quiet = (args) => {
try {
return docker(args);
} catch {
return "";
}
};
const combined = (args) => {
try {
return docker(args);
} catch (e) {
return `${e.stdout || ""}${e.stderr || ""}`;
}
};
// `docker logs` writes the container's stdout and stderr to docker's own stdout
// and stderr respectively. The embedded banner (entrypoint >&2) and Postgres
// logs go to stderr, so capture BOTH streams or those assertions miss them.
const dockerLogs = (name) => {
const res = spawnSync("docker", ["logs", name], { encoding: "utf-8" });
return `${res.stdout || ""}${res.stderr || ""}`;
};
const cleanup = () => {
quiet(["rm", "-f", NAME]);
quiet(["volume", "rm", VOL]);
};
async function waitHealthy(timeoutMs) {
const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) {
try {
const res = await fetch(`http://localhost:${PORT}/api/v1/health`);
if (res.ok) {
const body = await res.json();
if (body.status === "healthy") return true;
}
} catch {
// not accepting connections yet
}
await sleep(3000);
}
return false;
}
const countMatches = (haystack, re) => (haystack.match(re) || []).length;
async function main() {
console.log(`Embedded-mode tests against ${IMAGE}`);
cleanup();
// 1. Bare `docker run` with no DB env boots and becomes healthy.
console.log("\n[1] bare docker run boots healthy");
docker(["run", "-d", "--name", NAME, "-p", `${PORT}:1349`, "-v", `${VOL}:/data`, IMAGE]);
const healthy = await waitHealthy(300000);
healthy
? ok("embedded container reached healthy")
: bad("embedded container never became healthy");
const logs1 = dockerLogs(NAME);
logs1.includes("embedded mode") ? ok("embedded banner printed") : bad("no embedded banner");
countMatches(logs1, /first-boot initdb/g) === 1
? ok("initdb ran exactly once")
: bad("initdb did not run exactly once");
// 2. Restart reuses PGDATA (no second initdb), stays healthy.
console.log("\n[2] restart persists data, no re-init");
docker(["restart", NAME]);
(await waitHealthy(180000)) ? ok("healthy after restart") : bad("unhealthy after restart");
countMatches(dockerLogs(NAME), /first-boot initdb/g) === 1
? ok("no second initdb on restart")
: bad("re-initialized on restart (data loss risk)");
// 3. Clean shutdown: docker stop returns fast, Postgres logs a clean stop.
console.log("\n[3] clean shutdown");
const t0 = Date.now();
docker(["stop", NAME]);
const stopMs = Date.now() - t0;
stopMs < 9000
? ok(`stopped in ${stopMs}ms (before kill timeout)`)
: bad(`stop took ${stopMs}ms (likely SIGKILL)`);
dockerLogs(NAME).includes("database system is shut down")
? ok("Postgres logged a clean shutdown")
: bad("no clean Postgres shutdown line");
cleanup();
// 4. Non-root fails fast.
console.log("\n[4] non-root fail-fast");
combined(["run", "--rm", "--user", "1000:1000", IMAGE]).includes("embedded mode needs root")
? ok("non-root rejected with guidance")
: bad("non-root not rejected");
// 5. Partial config fails fast.
console.log("\n[5] partial-config fail-fast");
combined(["run", "--rm", "-e", "REDIS_URL=redis://x:6379", IMAGE]).includes(
"set BOTH DATABASE_URL and REDIS_URL",
)
? ok("partial config rejected")
: bad("partial config not rejected");
// 6. 1.x SQLite auto-detect upgrade: a /data/snapotter.db present with no
// SQLITE_MIGRATE_PATH set is auto-imported on first boot. Seed all 10 tables
// the importer reads (empty) so the import succeeds without column-mismatch;
// this proves the auto-detect wiring (row migration is covered elsewhere).
console.log("\n[6] 1.x SQLite auto-detect upgrade");
cleanup();
const tables = [
"users",
"teams",
"settings",
"roles",
"sessions",
"api_keys",
"pipelines",
"jobs",
"audit_log",
"user_files",
];
const seedSql = tables.map((t) => `CREATE TABLE ${t}(id TEXT);`).join(" ");
docker([
"run",
"--rm",
"-v",
`${VOL}:/data`,
"alpine:3.20",
"sh",
"-c",
`apk add --no-cache sqlite >/dev/null 2>&1 && sqlite3 /data/snapotter.db "${seedSql}"`,
]);
docker(["run", "-d", "--name", NAME, "-p", `${PORT}:1349`, "-v", `${VOL}:/data`, IMAGE]);
(await waitHealthy(300000))
? ok("healthy after upgrade boot")
: bad("unhealthy after upgrade boot");
dockerLogs(NAME).includes("Imported 1.x SQLite database")
? ok("auto-detected and imported the 1.x SQLite DB")
: bad("did not auto-import the 1.x SQLite DB");
// Second boot must NOT re-import (target Postgres now non-empty).
docker(["restart", NAME]);
(await waitHealthy(180000))
? ok("healthy after second boot")
: bad("unhealthy after second boot");
countMatches(dockerLogs(NAME), /Imported 1\.x SQLite database/g) === 1
? ok("did not re-import on the second boot")
: bad("re-imported on the second boot");
cleanup();
console.log(`\n${failures === 0 ? "ALL PASSED" : `${failures} FAILED`}`);
process.exit(failures === 0 ? 0 : 1);
}
main().catch((e) => {
console.error(e);
cleanup();
process.exit(1);
});
+126
View File
@@ -0,0 +1,126 @@
import { spawnSync } from "node:child_process";
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { dirname, join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
// Exercises the REAL docker/embedded-lib.sh (sourced, not mirrored) so the test
// cannot drift from what ships in the image. Mirrors entrypoint-permissions.test.ts.
const here = dirname(fileURLToPath(import.meta.url));
const LIB = resolve(here, "../../../docker/embedded-lib.sh");
// Run `snippet` with env after sourcing the lib; capture status + output.
function runLib(
snippet: string,
env: Record<string, string> = {},
): { status: number; stdout: string; stderr: string } {
const res = spawnSync("/bin/sh", ["-c", `. "${LIB}"\n${snippet}`], {
encoding: "utf-8",
env: { PATH: process.env.PATH ?? "", ...env },
});
return { status: res.status ?? 1, stdout: (res.stdout ?? "").trim(), stderr: res.stderr ?? "" };
}
describe("embedded-lib.sh decide_run_mode", () => {
it("embedded when both URLs unset and EMBEDDED unset", () => {
const r = runLib("decide_run_mode");
expect(r.status).toBe(0);
expect(r.stdout).toBe("embedded");
});
it("fails fast when only DATABASE_URL is set (partial config)", () => {
const r = runLib("decide_run_mode", { DATABASE_URL: "postgres://x@db/y" });
expect(r.status).toBe(2);
expect(r.stderr).toContain("BOTH");
});
it("external when both URLs are set", () => {
const r = runLib("decide_run_mode", {
DATABASE_URL: "postgres://x@db/y",
REDIS_URL: "redis://r",
});
expect(r.status).toBe(0);
expect(r.stdout).toBe("external");
});
it("external (not embedded) when EMBEDDED=0 even with no URLs", () => {
const r = runLib("decide_run_mode", { EMBEDDED: "0" });
expect(r.status).toBe(0);
expect(r.stdout).toBe("external");
});
it("fails fast when exactly one URL is set (partial config)", () => {
const r = runLib("decide_run_mode", { REDIS_URL: "redis://r" });
expect(r.status).toBe(2);
expect(r.stderr).toContain("BOTH");
});
});
describe("embedded-lib.sh embedded_requires_root", () => {
it("succeeds when uid is 0", () => {
expect(runLib("embedded_requires_root 0").status).toBe(0);
});
it("fails with actionable guidance when uid is non-zero", () => {
const r = runLib("embedded_requires_root 1000");
expect(r.status).toBe(1);
expect(r.stderr.toLowerCase()).toContain("root");
expect(r.stderr).toContain("Compose");
});
});
describe("embedded-lib.sh sqlite_autodetect_path", () => {
let dir: string;
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), "sqlite-detect-"));
});
afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});
it("echoes empty when no snapotter.db is present", () => {
expect(runLib(`sqlite_autodetect_path '${dir}'`).stdout).toBe("");
});
it("echoes the db path when snapotter.db exists and no override is set", () => {
writeFileSync(join(dir, "snapotter.db"), "x");
expect(runLib(`sqlite_autodetect_path '${dir}'`).stdout).toBe(join(dir, "snapotter.db"));
});
it("honors an explicit SQLITE_MIGRATE_PATH over auto-detect", () => {
writeFileSync(join(dir, "snapotter.db"), "x");
const r = runLib(`sqlite_autodetect_path '${dir}'`, {
SQLITE_MIGRATE_PATH: "/custom/legacy.db",
});
expect(r.stdout).toBe("/custom/legacy.db");
});
});
describe("embedded-lib.sh check_pg_version", () => {
let dir: string;
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), "pgver-"));
});
afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});
it("succeeds when PG_VERSION matches the installed major", () => {
writeFileSync(join(dir, "PG_VERSION"), "17\n");
expect(runLib(`check_pg_version '${dir}' 17`).status).toBe(0);
});
it("fails loudly when PG_VERSION is a different major", () => {
writeFileSync(join(dir, "PG_VERSION"), "16\n");
const r = runLib(`check_pg_version '${dir}' 17`);
expect(r.status).toBe(1);
expect(r.stderr).toContain("16");
expect(r.stderr).toContain("17");
expect(r.stderr.toLowerCase()).toContain("manual");
});
it("succeeds (no-op) when PG_VERSION is absent (fresh data dir)", () => {
expect(runLib(`check_pg_version '${dir}' 17`).status).toBe(0);
});
});