pnpm test:docker ran pnpm test:ci (vitest --coverage), but the lean test image deliberately skips binary-gated tools (AI model bundles, LibreOffice, etc.), so it can never meet the host-calibrated coverage thresholds -- the container exited non-zero on coverage even with zero test failures. Point the compose command at vitest run so test:docker is a clean functional pass/fail gate; coverage stays enforced on host CI where every tool is present.
Two test-image gaps surfaced by a full pnpm test:ci run:
- s3-storage.test.ts imports @aws-sdk/client-s3 (an enterprise dependency) at module load, but Dockerfile.test never copied packages/enterprise/package.json before pnpm install, so the suite failed to collect. Copy it so the dep installs; the suite then skips cleanly when MinIO is absent.
- EPS batch decode returned 422: ImageMagick reads EPS through the Ghostscript PS coder, but policy.xml left PS/PS2/PS3 at rights=none (only EPS was opened), so convert refused with a policy error before Ghostscript ran. Open the PostScript coders too.
tests/setup/per-fork-env.ts hardcoded SYNC_WAIT_MS=30000 on every fork, overriding whatever the container set, so the docker test image could never grant heavy ops a wider sync window. A 12MP stress-image enhance takes ~34s on the macOS Docker VM (Sharp runs 2-3x slower there), just past the 30s window, so the factory returned 202 and three sync-asserting image-enhancement tests failed.
Honor a higher SYNC_WAIT_MS when provided (30s floor preserved for host/CI), raise it to 120s in docker-compose.test.yml, and make the vitest test/hook timeouts env-overridable so a slow-but-correct job returns 200 rather than tripping a framework timeout. Host and CI behavior is unchanged.
Make the full pnpm test:docker suite pass the env-dependent tests (~85 failures):
- Dockerfile.test: ENV LD_LIBRARY_PATH=/usr/local/lib so the built libheif 1.21 is not shadowed by the base image's older system libheif (heif-dec failed with an undefined-symbol error -> 'No HEIF decoder found' on 72 HEIF tests); add libjxl-tools (JXL) and ghostscript + the ImageMagick policy.xml EPS allow-edit.
- docker-compose.test.yml: SYNC_WAIT_MS=30000 so sync-wait image tools do not fall back to 202 under single-container contention (10 tests).
- install_feature.py: guard tarfile.extractall(filter='data') behind Python>=3.12 (bookworm ships 3.11); the manual entry guards already protect.
- feature-status.test.ts / docker-file-secrets.test.ts: skip the two cases that cannot hold inside the container (/.dockerenv always present; root bypasses chmod). Verified on host: all still pass.
- Use /opt/venv directly when --entrypoint bash bypasses entrypoint.sh
- Use sys.executable for all pip calls (not bare pip)
- Override entrypoint in CI workflow to avoid startup banner
- Fix Biome formatting (template literals, try/catch blocks)
On 1.x-to-2.0 upgrade the /data/ai/venv already existed with stale
packages, so the entrypoint skipped the cp from /opt/venv and 2.0
Python tools failed with ImportError (#85).
Dockerfile now writes a SHA-256 of `pip freeze` to /opt/venv/.venv-version
at build time. The entrypoint compares that stamp against the copy on the
volume; on mismatch it does a clean nuke + re-copy and resets
installed.json so the UI correctly shows bundles as needing reinstall.
Models in /data/ai/models survive the refresh, so bundle reinstall only
reruns pip (model downloads are idempotent).
Backward compatible: if the image has no stamp (old build), the
entrypoint falls through to the existing skip-if-exists behavior.
* feat(infra): add dev compose stack with postgres and redis
* fix(infra): comment dev env defaults until wired; harden dev compose restart and start_period
* chore(deps): add pg driver and testcontainers for postgres migration
* feat(db): translate schema to drizzle pg-core (timestamptz, boolean, pgEnum, jsonb)
Schema translation (apps/api/src/db/schema.ts):
- sqlite-core -> pg-core, all 10 tables preserved 1:1
- integer(mode:'timestamp') -> timestamp({ withTimezone: true })
- integer(mode:'boolean') -> boolean
- jobs.status text enum -> pgEnum('job_status') with same 4 values
- 7 columns changed from text to jsonb: jobs.inputFiles, jobs.settings,
pipelines.steps, apiKeys.permissions, roles.permissions,
auditLog.details, userFiles.toolChain
- settings.value stays text, jobs.error stays text, jobs.progress stays real
jsonb call-site sweep (removed JSON.stringify on writes, JSON.parse on reads):
- apps/api/src/routes/roles.ts: permissions read/write (3 sites)
- apps/api/src/routes/api-keys.ts: permissions write + read (2 sites)
- apps/api/src/routes/audit-log.ts: details read (1 site)
- apps/api/src/routes/pipeline.ts: steps write + read (2 sites)
- apps/api/src/routes/progress.ts: inputFiles write (2 sites)
- apps/api/src/routes/tool-factory.ts: toolChain read + write (2 sites)
- apps/api/src/routes/user-files.ts: toolChain read + write (4 sites)
- apps/api/src/permissions.ts: roles.permissions read (1 site)
- apps/api/src/lib/audit.ts: details write (1 site)
- apps/api/src/plugins/auth.ts: apiKeys.permissions read (1 site)
* refactor(db): type jsonb columns via $type and note raw CTE conversion requirements
* feat(db): archive sqlite migrations and generate postgres baseline
* chore(db): dockerignore legacy migrations, add archive breadcrumb, fix trailing newline
* feat(db): pg pool connection, advisory-locked boot migrations, DATABASE_URL config
* fix(db): friendly fatal on unreachable postgres, idempotent closeDb, lock-key convention note
* refactor(db): async drizzle calls in plugins, lib, permissions
* fix(api): analytics never throws, typed permission guard, single-query session invalidation
* refactor(db): async drizzle calls across all routes and bootstrap
Convert every route file and index.ts from sync SQLite drizzle
patterns to async node-postgres drizzle:
- .all() removed (bare await on select)
- .get() converted to destructured [row] = await ...
- .run() removed (bare await on insert/update/delete)
- .changes replaced with .rowCount (null-guarded) in progress.ts
- sqlite import removed from user-files.ts; raw CTEs converted to
await db.execute(sql`...`) with postgres-dialect recursive CTEs
- ChainRow types updated: tool_chain is parsed jsonb (string[] | null),
created_at is Date (timestamptz) with no * 1000 conversion
- All requirePermission() guard calls awaited (security: unawaited
async guard returns truthy Promise, bypassing permission check)
- All hasEffectivePermission() and getPermissions() calls awaited
- All auditLog() calls awaited (preserves write-before-response order)
- trackEvent() and captureException() left un-awaited (fire-and-forget
by design, guaranteed never-throw)
- ensureAnonymousUser(), startCleanupCron(), recoverStaleJobs() awaited
in bootstrap sequence
- ensureInstanceId() and ensureDefaultSettings() made async
Files converted: 14 (index.ts + 12 route files + tools/index.ts)
* fix(db): await async checkStorageQuota in user-files upload/save routes
* fix(db): await checkStorageQuota in save-result route (missed second call site)
* feat(db): sqlite-to-postgres migrator with CLI and first-boot import
* fix(db): migrator error context, honest force semantics, boot-hook fatal, null-variance tests
* test: run suite against per-file postgres databases via testcontainers
- Add tests/global-setup.ts: spins up a Postgres testcontainer,
creates a migrated template database once per vitest run.
- Rewrite tests/setup/per-fork-env.ts: each test file (forks pool)
clones the template into its own database via CREATE DATABASE ...
TEMPLATE, preserving the same per-file isolation granularity.
- Update vitest.config.ts: add globalSetup, pg alias, update comment.
- Fix tests/integration/test-server.ts: remove DB_PATH mkdir, async
runMigrations, async db operations, remove SQLite WAL checkpoint.
- Fix 21 unit test db/index mocks: add pool and closeDb exports.
- Fix 8 unit test files: add async/await for now-async permission,
audit, and analytics functions.
- Fix 18 integration test files: convert sync .run()/.all()/.get()
to async drizzle patterns, add async to callbacks.
- Production change: apps/api/src/routes/teams.ts: cast COUNT(*)
to ::int so Postgres returns a number instead of bigint string.
* fix(db): seed built-in roles, reject NUL bytes, cast COUNT, serialize job persists
- Seed built-in roles (admin, editor, user) at boot via ensureBuiltinRoles()
with onConflictDoNothing, restoring data that legacy SQLite migration 0007
provided via INSERT statements (the pg baseline is DDL-only).
- Reject NUL bytes in login credentials with 401 (postgres rejects \x00 in
text columns; valid usernames never contain NUL, matching 1.x behavior).
- Cast COUNT(*)::int in user-files, audit-log, and roles listing queries so
postgres returns a JS number instead of bigint-as-string.
- Serialize fire-and-forget job progress DB writes per jobId so the final
"completed" status is never overwritten by a late-arriving "processing"
write (race condition exposed by async postgres round-trips).
* test: fix teams race, seed roles in test server, poll for job status
- Add missing await to resetTeams() in teams PUT beforeEach (the async
delete raced with the subsequent insert under postgres).
- Call ensureBuiltinRoles() in test server bootstrap so integration tests
have the same built-in roles as production.
- Replace fixed 100ms flushPersist delay with a polling helper that waits
for terminal job status, eliminating timing-dependent failures caused by
postgres network round-trip latency.
* test: make heic temp-file cleanup assertion resilient to concurrent workers
Use a set-based diff instead of raw file count when checking that
decodeHeic cleans up temp files. Other concurrent test workers can
create heic-in-*/heic-out-* files in the shared tmpdir, inflating the
"after" count and causing spurious failures under full-suite load.
* fix(db): align builtin-role seed to post-0010 legacy state; test polish
* feat(docker): three-container compose (app, postgres, redis) with boot wait and migrations
* fix(docker): set TEST_DATABASE_URL so containerized tests skip testcontainers
* chore(docker): test compose project name, clearer 1.x upgrade comment, unref probe timer
* feat(enterprise): enforce D15 license boundary; move s3 storage into packages/enterprise
* fix(enterprise): restore lazy aws-sdk loading; community installs load no s3 code at boot
* fix(enterprise): boundary check catches dynamic imports; document getS3 concurrency
* feat(db)!: SnapOtter 2.0 phase 1 foundation: postgres, migrator, compose stack
BREAKING CHANGE: SQLite is no longer the runtime database. Deployments now
require Postgres (and Redis, used from phase 2). Existing installs migrate
with SQLITE_MIGRATE_PATH or 'pnpm --filter @snapotter/api migrate:sqlite'.
* fix(ci): postgres service + fresh e2e database per run; ignore unfixable torch CVE-2025-3000
Closes the "e2e never runs in CI" hole. Adds per-PR e2e smoke gate,
nightly full-suite workflows, parallel vitest forks (per-fork DBs),
Playwright parallel/serial/visual projects against production builds,
metadata-generated test suites (drift guards, hostile inputs, format
matrix, pairwise settings, property-based fuzz), Stryker mutation
testing, Schemathesis API fuzz, coverage ratchet, and fixes for three
session-poisoning bugs that caused 200+ serial-bucket failures.
Bug fix included: favicon/split/bulk-rename could hang clients forever
when ZIP streaming failed after reply.hijack().
rembg 2.0.75 pulls in numpy>=2.3.0 which conflicts with our pinned
numpy==1.26.4 and would break the entire AI dependency chain. The two
rembg CVEs (SSRF + path traversal) are in its server/CLI components
which we don't use; they're already in the pip-audit ignore list.
- Bump Go from 1.23 to 1.24 to fix CVE-2025-68121 (CRITICAL:
crypto/tls certificate validation) and CVE-2025-61726 (HIGH:
net/url memory exhaustion) in the caire binary.
- Bump Pillow from 11.1.0 to 12.1.1 to fix CVE-2026-25990 (HIGH:
out-of-bounds write via crafted image).
- fix(ocr): change paddlepaddle-gpu from --extra-index-url to --index-url
for the CUDA 12.6 package index. With --extra-index-url, pip could
resolve from PyPI (CUDA 11 build) instead of the cu126 index, causing
"libcusolver.so.11: undefined symbol" errors on CUDA 12 containers.
- revert version to 1.17.1 (v1.17.2 release was deleted)
- fix(rate-limit): treat RATE_LIMIT_PER_MIN=0 as unlimited (50k/min)
instead of blocking all requests. @fastify/rate-limit interprets
max:0 as "allow zero requests," breaking fresh container startups.
- fix(docker): add libgles2 for MediaPipe face detection tools.
blur-faces, red-eye-removal, enhance-faces, and passport-photo
failed with "libGLESv2.so.2 not found" on all headless containers.
- fix(docker/arm64): remove conflicting system libheif1 to avoid
ABI symbol mismatch with our custom libheif 1.21.2 build.
heif-convert failed with "undefined symbol: heif_get_plugin_directories."
- fix(docker/arm64): pre-install wheel+setuptools in base Python venv
so basicsr can build from source on arm64 (no pre-built wheel).
This unblocks upscale-enhance and photo-restoration bundles.
* fix(security): harden SVG sanitizer, rate limiting, and analytics defaults
- SVG: add control-char stripping in href values to block whitespace/null-byte
obfuscated javascript: URIs; block <feImage> with external href (SSRF via
SVG filter primitives); expand test suite to 32 inline bypass payloads
- Rate limiting: add per-route limits on tool endpoints (60/min) and batch
(20/min); fix compose files defaulting RATE_LIMIT_PER_MIN to 0 which mapped
to 50,000 in code; simplify rate limit registration to use env.ts default
- Analytics: default ANALYTICS_ENABLED to false so self-hosters do not
unknowingly send telemetry
- Docker: add --max-time 5 and -s flags to compose healthcheck curl commands
* fix: remove stale login limit bypass, reduce error log noise, clean up fixtures
- Fix getLoginAttemptLimit() ignoring LOGIN_ATTEMPT_LIMIT when global rate
limit exceeded 1000/min, which let the global limit override the stricter
per-route login brute-force protection
- Downgrade rate limit 429 responses from error to warn level in the global
error handler to avoid log noise and unnecessary Sentry reports
- Log 4xx client errors at warn level instead of error level
- Remove 11 orphaned SVG attack fixture files replaced by inline test payloads
Support reading secrets from mounted files instead of plain-text
environment variables, following the standard Docker/Kubernetes
convention used by MariaDB, Postgres, and Stirling-PDF.
Supported vars: DEFAULT_PASSWORD, S3_ACCESS_KEY_ID,
S3_SECRET_ACCESS_KEY, OIDC_CLIENT_SECRET, COOKIE_SECRET,
SNAPOTTER_LICENSE_KEY.
Add pre-built release archives (Linux amd64/arm64) to the release
workflow, published as GitHub Release assets. Each archive is a
self-contained tar.gz (~240MB) with built frontend, API source,
and production node_modules. Users extract and run without needing
pnpm build.
Also includes AI install manifest fixes for Proxmox/bare-metal users:
- Pin setuptools<75 for Python 3.13 basicsr compatibility
- Pre-install basicsr with --no-build-isolation before realesrgan
- Loosen mediapipe pins from == to >= for Python 3.13 wheels
- Add retry logic to HuggingFace model downloads
iPhone 15 Pro (iOS 18+) HEIC files include HDR gain maps as auxiliary
image references. Distro-packaged libheif (1.15-1.17) rejects these
with "Too many auxiliary image references". Build libheif v1.21.2 from
source in a new Dockerfile stage to fix decoding.
- Add libheif-builder stage with platform-matched bases (debian:bookworm
for arm64, ubuntu:24.04 for amd64) to avoid shared-library ABI
mismatches
- Replace libheif-examples distro package with source-built binaries
- Update Dockerfile.test with same libheif source build
- No application code changes needed (heic-converter.ts CLI interface
is stable across versions)
Closes#183
Token-Permissions (0 -> 10): Set permissions: {} at workflow top level
across all 7 workflows, moved write scopes to per-job minimum.
SAST (0 -> 10): Added CodeQL workflow for JavaScript/TypeScript and
Python analysis on push, PR, and weekly schedule.
Vulnerabilities (0 -> ~8): Added 13 pnpm overrides to patch transitive
dependency vulnerabilities (38 -> 2 remaining, both in dev-only tools).
Pinned-Dependencies (5 -> 8-9): Pinned all Docker FROM images to SHA
digests, pinned pip-audit version in CI, pinned pip version in
Dockerfile.
Handle OOM kills (exit code 137) with actionable memory guidance,
filter ANSI/progress noise from error output, add --no-cache-dir to
pip installs, reduce download concurrency to 2, and bump default
container memory from 4g to 6g.
When AUTH_ENABLED=false, the anonymous user was assigned the "user" role
which lacks settings:write permission, making all settings saves return
403. Since no admin exists when auth is disabled, settings were
permanently read-only. Promote the anonymous user to "admin" so the
single user has full control of the instance.
Also adds DEFAULT_TOOL_VIEW env var (sidebar|fullscreen) following the
existing DEFAULT_THEME pattern, seeded via ensureDefaultSettings() on
first boot.
Closes#135
- LOGIN_ATTEMPT_LIMIT: 500 -> 30 in Dockerfile (matching code default)
- RATE_LIMIT_PER_MIN=0 now means no global limit (50K ceiling) instead
of 1 req/min, preserving backward compatibility with existing deploys
while keeping per-route limits (login brute-force) active
Auth: login rate limit 30/min (was 500), global rate limit 1000/min (was
unlimited), password/username max lengths on all Zod schemas, session
invalidation on role change, API key legacy scan bounded to 100 keys.
SVG: hardened regex sanitizer with CDATA stripping, XML entity decoding,
set/animate/iframe/embed blocking, comprehensive data: URI blocking,
use element external href blocking. 11 attack payload fixtures added.
SSRF: fixed DNS rebinding TOCTOU by pinning resolved IPs via custom
HTTP/HTTPS agents. Added 6to4 and NAT64 to blocked IPv6 ranges.
Docker: capability dropping (cap_drop ALL + minimal cap_add), resource
limits (4g/8g mem, 512/1024 pids), healthcheck timeout, password
removed from startup banner, default password warning comments.
Network: CSP and HSTS applied in all environments (not just production),
stack traces removed from all error responses, internal paths stripped
from error details, per-route rate limits on uploads (60/min) and URL
fetches (200/hour).
Files: exclusive temp file creation (O_EXCL), disk space circuit
breaker, per-user storage quotas, settings payload 64KB size guard.
Python sidecar: script name allowlist in dispatcher, minimal environment
for subprocess spawns.
Dependencies: fixed 6 production CVEs (drizzle-orm, fastify, fast-uri,
@fastify/static, next, archiver/lodash). Pinned all GitHub Actions to
SHA hashes.
114 security tests added. Full OWASP Top 10 penetration test matrix
verified against production Docker container (30/30 pass after
hardening).
- Fix SSE write-after-end crash in progress.ts (remove callback before ending stream)
- Fix blob URL memory leaks: revoke processedPreviewUrl and old HEIC preview URLs
- Add AbortController to batch fetch in use-tool-processor and use-pipeline-processor
- Fix TGA format misidentified as CUR (extension overrides magic bytes)
- Add libheif-plugin-libde265 to Docker for HEIC/HEIF decode support
- Remove unused imports and state (AppLayout, setSampledColor, useEffect)
- Fix non-null assertions in meme-text-renderer and meme-generator
- Fix confusing void type in meme-templates
- Remove unnecessary useEffect deps in adjustments-panel
- Fix Playwright strict mode violations in 5 E2E tests
