Files
SnapOtter/docker/docker-compose-gpu.yml
T
SnapOtterandGitHub 35e18d8b79 fix: GPU deployment robustness (6 fixes from end-to-end testing on an RTX 4070) (#334)
* fix(docker): pin CUDA base to 12.6 so the GPU image starts on R560+ drivers

The amd64 base nvidia/cuda:12.9.2-cudnn-runtime bakes a cuda>=12.9 driver gate enforced by nvidia-container-toolkit at container start, so the image fails to launch on common production drivers (e.g. 570.x / CUDA 12.8). The AI bundles are all cu126 wheels and the image installs libcublas-12-6, so 12.9 was misaligned with the workload. Pin to nvidia/cuda:12.6.3-cudnn-runtime-ubuntu24.04 to match the wheels and lower the driver floor to R560+.

* fix(ai): broaden OOM detection so the rembg lighter-model fallback fires

onnxruntime/CUDA allocation failures surface as 'Failed to allocate memory for requested buffer', CUBLAS_STATUS_ALLOC_FAILED, or bad_alloc, not just 'out of memory'. The background-removal and transparency-fixer fallback-to-lighter-model paths only matched the literal 'out of memory', so the fallback was dead code and transparency-fixer (default birefnet-hr-matting) always failed with an allocation error. Add isMemoryAllocError() and use it in both checks.

* fix(ai): use bundled PaddleOCR models so OCR runs offline

ocr.py passed no model dirs to PaddleOCR, so PaddleX resolved models from ~/.paddlex and downloaded them from HuggingFace at runtime (slow first use, broken air-gapped), ignoring the models the OCR bundle ships in MODELS_PATH; it also pulled doc-orientation/unwarping models that are not bundled. Pin detection, recognition and textline models to the bundled dirs in MODELS_PATH (per language) and disable use_doc_orientation_classify / use_doc_unwarping, with per-component fallback when a model is absent. Verified: OCR runs with zero HuggingFace requests.

* fix(docker): add CAP_KILL so container shutdown is graceful

cap_drop: ALL without re-adding KILL meant tini (PID 1, root) could not forward SIGTERM to the gosu-dropped snapotter process (root minus CAP_KILL cannot signal a different UID). docker stop logged '[FATAL tini] forwarding signal: Operation not permitted', never delivered the signal, and fell back to SIGKILL after the 10s timeout. Add KILL to cap_add in both compose files. Verified: docker stop completes in 0s with SIGTERM delivered (exit 143) and no FATAL tini.

* fix(ai): serialize bundle installs against AI jobs to prevent sidecar segfault

A feature bundle install rewrites the shared Python venv (pip + copytree of site-packages/*.so) as a background subprocess, with no coordination against AI tool jobs that dlopen native libs (torch / onnxruntime CUDA) from the same venv; a job loading a shared object while it is overwritten segfaults the sidecar. Add a process-wide async mutex (venv-lock.ts): bridge.run() acquires it before every AI script and the install route holds it across the installer subprocess. Both run in the same Node process so a module-level lock suffices. Verified: concurrent install + AI job produces zero segfaults and the job serializes behind the install.

* fix(ai): make the venv lock read/write so concurrent AI jobs are not serialized

The first cut used an exclusive mutex, which (a) deferred the dispatcher spawn by a microtask and broke unit tests that synchronously drive the mocked spawn, and (b) serialized AI jobs against each other, removing the dispatcher's by-id request multiplexing. Make it a writer-preferring read/write lock: AI jobs are shared readers (with a synchronous fast path so spawn still happens in-tick) and a bundle install is the exclusive writer. Verified: all 764 AI unit tests pass.

* fix(ai): degrade OCR to Tesseract on CPU-only hosts instead of segfaulting

The amd64 AI bundle ships paddlepaddle-gpu, whose native libs dlopen
libcuda.so.1 at import and segfault on a host without a GPU (libcuda is the
driver lib, injected only by nvidia-container-toolkit on GPU hosts). The
segfault crashed the shared long-lived AI dispatcher and, after a few attempts,
tripped the bridge crash-recovery permanent-disable, wedging all AI until a
container restart. The standalone ocr tool defaults to quality=balanced
(PaddleOCR), so it hit this on every CPU-only deployment; ocr-pdf already
hardcoded Tesseract and was unaffected.

ocr.py now gates the PaddleOCR tiers on gpu_available(): balanced/best
transparently fall back to fast (Tesseract, CPU-capable) when no usable GPU is
present, and run_paddleocr_v5/run_paddleocr_vl refuse before importing paddle so
the GPU build is never dlopen'd on CPU. GPU hosts are unchanged.

Verified on a CPU-only Windows/WSL2 box: ocr returns Tesseract text across
repeated runs with the dispatcher staying healthy (no wedge).
2026-06-23 18:39:51 +08:00

169 lines
5.9 KiB
YAML

name: SnapOtter
# NVIDIA GPU deployment — requires nvidia-container-toolkit.
# Install: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
# Usage: docker compose -f docker-compose-gpu.yml up -d
# Verify: docker logs SnapOtter 2>&1 | grep GPU
services:
SnapOtter:
build:
context: ..
dockerfile: docker/Dockerfile
image: snapotter/snapotter:latest
container_name: SnapOtter
ports:
# For internet-facing deployments, bind to localhost only:
# - "127.0.0.1:1349:1349"
- "1349:1349"
volumes:
- SnapOtter-data:/data # AI models, user files
- SnapOtter-workspace:/tmp/workspace # Temp processing (auto-cleaned)
environment:
- AUTH_ENABLED=true
- DEFAULT_USERNAME=admin
# Set a strong password. Default is 'admin' -- CHANGE THIS for any non-local deployment.
# - DEFAULT_PASSWORD=your-strong-password-here
- DEFAULT_PASSWORD=admin
- SKIP_MUST_CHANGE_PASSWORD=${SKIP_MUST_CHANGE_PASSWORD:-false}
- MAX_UPLOAD_SIZE_MB=${MAX_UPLOAD_SIZE_MB:-0}
- MAX_BATCH_SIZE=${MAX_BATCH_SIZE:-0}
- MAX_MEGAPIXELS=${MAX_MEGAPIXELS:-0}
- CONCURRENT_JOBS=${CONCURRENT_JOBS:-0}
- MAX_WORKER_THREADS=${MAX_WORKER_THREADS:-0}
- PROCESSING_TIMEOUT_S=${PROCESSING_TIMEOUT_S:-0}
- MAX_PIPELINE_STEPS=${MAX_PIPELINE_STEPS:-20}
- RATE_LIMIT_PER_MIN=${RATE_LIMIT_PER_MIN:-300}
- MAX_USERS=${MAX_USERS:-0}
- SESSION_DURATION_HOURS=${SESSION_DURATION_HOURS:-168}
- TRUST_PROXY=${TRUST_PROXY:-true}
- DATABASE_URL=postgres://${POSTGRES_USER:-snapotter}:${POSTGRES_PASSWORD:-snapotter}@postgres:5432/${POSTGRES_DB:-snapotter}
- REDIS_URL=redis://:${REDIS_PASSWORD:-snapotter}@redis:6379
# 1.x upgrade: uncomment to import the old SQLite database on first boot;
# re-comment after the migration succeeds.
# - SQLITE_MIGRATE_PATH=/data/snapotter.db
# OIDC Authentication (optional)
# - EXTERNAL_URL=https://photos.example.com
# - OIDC_ENABLED=false
# - OIDC_ISSUER_URL=
# - OIDC_CLIENT_ID=
# - OIDC_CLIENT_SECRET=
# - OIDC_SCOPES=openid profile email
# - OIDC_AUTO_CREATE_USERS=true
# - OIDC_DEFAULT_ROLE=user
# - OIDC_AUTO_LINK_USERS=false
# - OIDC_PROVIDER_NAME=
# - OIDC_USERNAME_CLAIM=preferred_username
# - OIDC_CLOCK_TOLERANCE=30
# - COOKIE_SECRET=
#
# Docker secrets (_FILE convention): mount secrets as files instead of
# passing them as plain-text env vars. Supported for sensitive vars only.
# - DEFAULT_PASSWORD_FILE=/run/secrets/snapotter_password
# - S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key
# - S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_key
# - OIDC_CLIENT_SECRET_FILE=/run/secrets/oidc_secret
# - COOKIE_SECRET_FILE=/run/secrets/cookie_secret
# - SNAPOTTER_LICENSE_KEY_FILE=/run/secrets/license_key
restart: unless-stopped
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
# --- Security hardening ---
mem_limit: 8g
memswap_limit: 8g
cpus: 8
pids_limit: 1024
cap_drop:
- ALL
cap_add:
- CHOWN
- SETUID
- SETGID
- DAC_OVERRIDE
- FOWNER
# KILL lets tini (PID 1, root) forward SIGTERM to the gosu-dropped
# snapotter process on shutdown. Without it, root minus CAP_KILL cannot
# signal a different-UID process, so docker stop is ungraceful
# ("[FATAL tini] forwarding signal: Operation not permitted" -> SIGKILL).
- KILL
# NOTE: security_opt: [no-new-privileges:true] is intentionally omitted.
# gosu requires setuid to drop from root to the snapotter user.
# Mitigation: cap_drop: ALL limits available capabilities after privilege drop.
# NOTE: read_only: true is not set because PUID/PGID remapping requires
# writing to /etc/passwd and /etc/group. Consider using Docker --user flag
# instead of PUID/PGID for read-only rootfs support.
healthcheck:
test: ["CMD", "curl", "-sf", "--max-time", "5", "http://localhost:1349/api/v1/health"]
interval: 30s
timeout: 5s
start_period: 60s
retries: 3
shm_size: "2gb" # Required for PyTorch CUDA shared memory
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
logging:
driver: json-file
options:
max-size: "50m"
max-file: "5"
postgres:
image: postgres:17-alpine
container_name: SnapOtter-postgres
environment:
POSTGRES_USER: ${POSTGRES_USER:-snapotter}
# Set a strong password -- CHANGE THIS for any non-local deployment.
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-snapotter}
POSTGRES_DB: ${POSTGRES_DB:-snapotter}
volumes:
- SnapOtter-pgdata:/var/lib/postgresql/data
restart: unless-stopped
mem_limit: 1g
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-snapotter}"]
interval: 10s
timeout: 5s
retries: 12
start_period: 15s
redis:
image: redis:8-alpine
container_name: SnapOtter-redis
command: >-
redis-server
--maxmemory-policy noeviction
--maxmemory 512mb
--appendonly yes
--requirepass ${REDIS_PASSWORD:-snapotter}
volumes:
- SnapOtter-redisdata:/data
restart: unless-stopped
mem_limit: 1g
healthcheck:
test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD:-snapotter}", "--no-auth-warning", "ping"]
interval: 10s
timeout: 5s
retries: 12
start_period: 10s
# Uncomment to use Docker secrets (requires Docker Swarm or compose v2.23+):
# secrets:
# snapotter_password:
# file: ./secrets/snapotter_password.txt
# oidc_secret:
# file: ./secrets/oidc_secret.txt
volumes:
SnapOtter-data:
SnapOtter-workspace:
SnapOtter-pgdata:
SnapOtter-redisdata: