Workflow-level paths-ignore never creates the check runs on docs-only
PRs, which would deadlock required status checks (contexts stay
"expected" forever). A new always-on changes job diffs against the
base commit and every downstream job skips when only README,
CONTRIBUTING, branding, apps/docs, apps/landing, or docs changed.
Skipped jobs report a conclusion, which satisfies branch protection,
so main can now require the full CI matrix without blocking
docs-only changes. Fails open: unknown base (force push, dispatch,
new branch) runs the full pipeline.
Prerequisite for enabling required status checks on main.
Claude-Session: https://claude.ai/code/session_01XGB4pGvTvb7sUX4JN745U7
Lands five integrated branches: pipeline templates (#355), analytics opt-out (#354), 83 conversion presets bringing the catalog to 240 tools (#356), self-hosted positioning (#353), and e2e modernization (#351).
Integration fixes: aligned stale web analytics tests with the opt-out/allow-list model, closed 3 CodeQL incomplete-sanitization alerts in the i18n generator, resolved settings/index/docs/format-matrix conflicts, and corrected tool counts to 240.
Disable the redundant ryuk reaper (tests/global-setup.ts stops its containers
explicitly and CI runners are ephemeral) and pre-pull postgres/redis with retry,
so a transient Docker Hub 500 -- as hit pulling ryuk on #320's post-merge run --
cannot fail the suite. Covers ci.yml (unit + integration) and nightly.yml.
RAW (DNG) processing crashed on ImageMagick's deprecated ufraw-batch
delegate, which fails on modern formats such as iPhone ProRAW DNG.
Root cause: the dcraw_emu (LibRaw) decode tier read the wrong output path.
dcraw_emu APPENDS the output extension (raw-in-X.dng -> raw-in-X.dng.tiff)
but the code looked for raw-in-X.tiff (replaced extension), so readFile threw
on every RAW, the tier silently fell through to ufraw, and the 24MB TIFF
leaked into the temp dir on each attempt.
- Repair the dcraw_emu output path; clean it up in finally (fixes the leak)
- Prefer LibRaw full decode over embedded-preview extraction so a
full-resolution RAW is never silently returned as a reduced-size preview
(sample DNG: was 1024x683 preview, now 3474x2314 full)
- Add RAW decode regression tests (DNG full-resolution + all 6 RAW formats);
these were absent, which let the bug ship
- Install libraw-bin on CI test runners so dcraw_emu is actually exercised
- fixture-integrity: probe media via media-engine probeMedia (resolves the
bundled static ffmpeg) instead of bare system ffprobe, which is ENOENT in CI;
gate on ffmpegAvailable() like the other media tests
- a11y: regenerate a11y-baseline.json to include the mobile device keys (the
baseline only had desktop keys, so the mobile a11y scan saw them as new)
- device-visual: tag @visual and exclude it from the PR mobile-smoke gate
(darwin-only screenshots cannot pass on linux; nightly + update-visual-baselines
still run it to seed linux goldens)
Fixes all integration CI failures on the 2.0 branch.
## What was broken
Two independent root causes:
1. **202 assertion failures** -- Under 4-fork CI parallel load, the 30s
`SYNC_WAIT_MS` sync window can expire before a BullMQ worker finishes a
heavy encode (avif, heic), returning a legitimate `202 {jobId, async: true}`
instead of `200`. Tests that hard-asserted `200` were spuriously failing.
2. **Vitest timeout race** -- `SYNC_WAIT_MS` (30s) and the default Vitest
`testTimeout` (also 30s) fired simultaneously. Vitest won the race,
reporting "Test timed out in 30000ms" instead of the test receiving the
202 response.
## Fixes
- Added `isAsyncFallback()` helper to four integration test files; validates
the `{async: true, jobId}` body shape and returns early so the synchronous
200 path runs full assertions only when warranted.
- Set `vi.setConfig({ testTimeout: 60_000 })` at module level in
`image-enhancement.test.ts` and `format-matrix-comprehensive.test.ts`,
giving a 30s buffer between when `waitForJob()` returns 202 and when
Vitest gives up.
- Bumped explicit matrix timeouts in `format-matrix.test.ts` and
`new-formats.test.ts` from 30s to 60s for the same reason.
- Installed missing CI doc-engine binaries (qpdf, pandoc, libreoffice,
pdfcpu) that were causing unrelated integration failures.
- Fixed E2E smoke specs for 2.0 UI changes (modality selector, tool routes,
validation behavior).
* feat(infra): add dev compose stack with postgres and redis
* fix(infra): comment dev env defaults until wired; harden dev compose restart and start_period
* chore(deps): add pg driver and testcontainers for postgres migration
* feat(db): translate schema to drizzle pg-core (timestamptz, boolean, pgEnum, jsonb)
Schema translation (apps/api/src/db/schema.ts):
- sqlite-core -> pg-core, all 10 tables preserved 1:1
- integer(mode:'timestamp') -> timestamp({ withTimezone: true })
- integer(mode:'boolean') -> boolean
- jobs.status text enum -> pgEnum('job_status') with same 4 values
- 7 columns changed from text to jsonb: jobs.inputFiles, jobs.settings,
pipelines.steps, apiKeys.permissions, roles.permissions,
auditLog.details, userFiles.toolChain
- settings.value stays text, jobs.error stays text, jobs.progress stays real
jsonb call-site sweep (removed JSON.stringify on writes, JSON.parse on reads):
- apps/api/src/routes/roles.ts: permissions read/write (3 sites)
- apps/api/src/routes/api-keys.ts: permissions write + read (2 sites)
- apps/api/src/routes/audit-log.ts: details read (1 site)
- apps/api/src/routes/pipeline.ts: steps write + read (2 sites)
- apps/api/src/routes/progress.ts: inputFiles write (2 sites)
- apps/api/src/routes/tool-factory.ts: toolChain read + write (2 sites)
- apps/api/src/routes/user-files.ts: toolChain read + write (4 sites)
- apps/api/src/permissions.ts: roles.permissions read (1 site)
- apps/api/src/lib/audit.ts: details write (1 site)
- apps/api/src/plugins/auth.ts: apiKeys.permissions read (1 site)
* refactor(db): type jsonb columns via $type and note raw CTE conversion requirements
* feat(db): archive sqlite migrations and generate postgres baseline
* chore(db): dockerignore legacy migrations, add archive breadcrumb, fix trailing newline
* feat(db): pg pool connection, advisory-locked boot migrations, DATABASE_URL config
* fix(db): friendly fatal on unreachable postgres, idempotent closeDb, lock-key convention note
* refactor(db): async drizzle calls in plugins, lib, permissions
* fix(api): analytics never throws, typed permission guard, single-query session invalidation
* refactor(db): async drizzle calls across all routes and bootstrap
Convert every route file and index.ts from sync SQLite drizzle
patterns to async node-postgres drizzle:
- .all() removed (bare await on select)
- .get() converted to destructured [row] = await ...
- .run() removed (bare await on insert/update/delete)
- .changes replaced with .rowCount (null-guarded) in progress.ts
- sqlite import removed from user-files.ts; raw CTEs converted to
await db.execute(sql`...`) with postgres-dialect recursive CTEs
- ChainRow types updated: tool_chain is parsed jsonb (string[] | null),
created_at is Date (timestamptz) with no * 1000 conversion
- All requirePermission() guard calls awaited (security: unawaited
async guard returns truthy Promise, bypassing permission check)
- All hasEffectivePermission() and getPermissions() calls awaited
- All auditLog() calls awaited (preserves write-before-response order)
- trackEvent() and captureException() left un-awaited (fire-and-forget
by design, guaranteed never-throw)
- ensureAnonymousUser(), startCleanupCron(), recoverStaleJobs() awaited
in bootstrap sequence
- ensureInstanceId() and ensureDefaultSettings() made async
Files converted: 14 (index.ts + 12 route files + tools/index.ts)
* fix(db): await async checkStorageQuota in user-files upload/save routes
* fix(db): await checkStorageQuota in save-result route (missed second call site)
* feat(db): sqlite-to-postgres migrator with CLI and first-boot import
* fix(db): migrator error context, honest force semantics, boot-hook fatal, null-variance tests
* test: run suite against per-file postgres databases via testcontainers
- Add tests/global-setup.ts: spins up a Postgres testcontainer,
creates a migrated template database once per vitest run.
- Rewrite tests/setup/per-fork-env.ts: each test file (forks pool)
clones the template into its own database via CREATE DATABASE ...
TEMPLATE, preserving the same per-file isolation granularity.
- Update vitest.config.ts: add globalSetup, pg alias, update comment.
- Fix tests/integration/test-server.ts: remove DB_PATH mkdir, async
runMigrations, async db operations, remove SQLite WAL checkpoint.
- Fix 21 unit test db/index mocks: add pool and closeDb exports.
- Fix 8 unit test files: add async/await for now-async permission,
audit, and analytics functions.
- Fix 18 integration test files: convert sync .run()/.all()/.get()
to async drizzle patterns, add async to callbacks.
- Production change: apps/api/src/routes/teams.ts: cast COUNT(*)
to ::int so Postgres returns a number instead of bigint string.
* fix(db): seed built-in roles, reject NUL bytes, cast COUNT, serialize job persists
- Seed built-in roles (admin, editor, user) at boot via ensureBuiltinRoles()
with onConflictDoNothing, restoring data that legacy SQLite migration 0007
provided via INSERT statements (the pg baseline is DDL-only).
- Reject NUL bytes in login credentials with 401 (postgres rejects \x00 in
text columns; valid usernames never contain NUL, matching 1.x behavior).
- Cast COUNT(*)::int in user-files, audit-log, and roles listing queries so
postgres returns a JS number instead of bigint-as-string.
- Serialize fire-and-forget job progress DB writes per jobId so the final
"completed" status is never overwritten by a late-arriving "processing"
write (race condition exposed by async postgres round-trips).
* test: fix teams race, seed roles in test server, poll for job status
- Add missing await to resetTeams() in teams PUT beforeEach (the async
delete raced with the subsequent insert under postgres).
- Call ensureBuiltinRoles() in test server bootstrap so integration tests
have the same built-in roles as production.
- Replace fixed 100ms flushPersist delay with a polling helper that waits
for terminal job status, eliminating timing-dependent failures caused by
postgres network round-trip latency.
* test: make heic temp-file cleanup assertion resilient to concurrent workers
Use a set-based diff instead of raw file count when checking that
decodeHeic cleans up temp files. Other concurrent test workers can
create heic-in-*/heic-out-* files in the shared tmpdir, inflating the
"after" count and causing spurious failures under full-suite load.
* fix(db): align builtin-role seed to post-0010 legacy state; test polish
* feat(docker): three-container compose (app, postgres, redis) with boot wait and migrations
* fix(docker): set TEST_DATABASE_URL so containerized tests skip testcontainers
* chore(docker): test compose project name, clearer 1.x upgrade comment, unref probe timer
* feat(enterprise): enforce D15 license boundary; move s3 storage into packages/enterprise
* fix(enterprise): restore lazy aws-sdk loading; community installs load no s3 code at boot
* fix(enterprise): boundary check catches dynamic imports; document getS3 concurrency
* feat(db)!: SnapOtter 2.0 phase 1 foundation: postgres, migrator, compose stack
BREAKING CHANGE: SQLite is no longer the runtime database. Deployments now
require Postgres (and Redis, used from phase 2). Existing installs migrate
with SQLITE_MIGRATE_PATH or 'pnpm --filter @snapotter/api migrate:sqlite'.
* fix(ci): postgres service + fresh e2e database per run; ignore unfixable torch CVE-2025-3000
Closes the "e2e never runs in CI" hole. Adds per-PR e2e smoke gate,
nightly full-suite workflows, parallel vitest forks (per-fork DBs),
Playwright parallel/serial/visual projects against production builds,
metadata-generated test suites (drift guards, hostile inputs, format
matrix, pairwise settings, property-based fuzz), Stryker mutation
testing, Schemathesis API fuzz, coverage ratchet, and fixes for three
session-poisoning bugs that caused 200+ serial-bucket failures.
Bug fix included: favicon/split/bulk-rename could hang clients forever
when ZIP streaming failed after reply.hijack().
No fixed versions available yet for:
- torch 2.12.0: 11 PYSEC advisories (transitive dep from ML packages)
- joblib 1.5.3: PYSEC-2024-277
- markdown 3.10.2: PYSEC-2026-89
Token-Permissions (0 -> 10): Set permissions: {} at workflow top level
across all 7 workflows, moved write scopes to per-job minimum.
SAST (0 -> 10): Added CodeQL workflow for JavaScript/TypeScript and
Python analysis on push, PR, and weekly schedule.
Vulnerabilities (0 -> ~8): Added 13 pnpm overrides to patch transitive
dependency vulnerabilities (38 -> 2 remaining, both in dev-only tools).
Pinned-Dependencies (5 -> 8-9): Pinned all Docker FROM images to SHA
digests, pinned pip-audit version in CI, pinned pip version in
Dockerfile.
- Separate unit tests (fast, no system deps) from integration test
shards to prevent vi.mock db leakage across test types
- Fix hero test: update expected subtitle to match current copy
- Fix FAQ test: add missing json-ld alias in vitest config and mock
- Integration tests run in 4 parallel shards (30min timeout each)
The full test suite takes ~100 minutes sequentially. Split into 4
parallel shards using Vitest's --shard flag so each completes in
~25 minutes. Removed coverage from CI (was causing overhead without
being reported anywhere). 30-minute timeout per shard as safety net.
The sample.heif was 8736x5856 (2.5MB), causing each processing
operation to take 17-38s in CI. Resized to 1432x960 (224KB) and added
a 15-minute timeout to the test job.
Auth: login rate limit 30/min (was 500), global rate limit 1000/min (was
unlimited), password/username max lengths on all Zod schemas, session
invalidation on role change, API key legacy scan bounded to 100 keys.
SVG: hardened regex sanitizer with CDATA stripping, XML entity decoding,
set/animate/iframe/embed blocking, comprehensive data: URI blocking,
use element external href blocking. 11 attack payload fixtures added.
SSRF: fixed DNS rebinding TOCTOU by pinning resolved IPs via custom
HTTP/HTTPS agents. Added 6to4 and NAT64 to blocked IPv6 ranges.
Docker: capability dropping (cap_drop ALL + minimal cap_add), resource
limits (4g/8g mem, 512/1024 pids), healthcheck timeout, password
removed from startup banner, default password warning comments.
Network: CSP and HSTS applied in all environments (not just production),
stack traces removed from all error responses, internal paths stripped
from error details, per-route rate limits on uploads (60/min) and URL
fetches (200/hour).
Files: exclusive temp file creation (O_EXCL), disk space circuit
breaker, per-user storage quotas, settings payload 64KB size guard.
Python sidecar: script name allowlist in dispatcher, minimal environment
for subprocess spawns.
Dependencies: fixed 6 production CVEs (drizzle-orm, fastify, fast-uri,
@fastify/static, next, archiver/lodash). Pinned all GitHub Actions to
SHA hashes.
114 security tests added. Full OWASP Top 10 penetration test matrix
verified against production Docker container (30/30 pass after
hardening).
Install ImageMagick, Ghostscript, libjxl-tools, and libopenjp2-tools in
CI so exotic format decoder tests (PSD, EPS, HDR, ICO, JP2, etc.) can
run. Relax ImageMagick EPS/PS security policy to match the Dockerfile.
Replace fragile vi.mock() of the SSRF module in fetch-urls tests with an
env-var guard (SSRF_ALLOW_PRIVATE) that bypasses private-IP checks in
the test environment. The vi.mock approach broke under V8 coverage
instrumentation in CI.
Pillow 11.1.0 has CVE-2026-42308, CVE-2026-42310, CVE-2026-42311
(fixed in 12.2.0). Upgrading to Pillow 12.x requires validating
compatibility with rembg, realesrgan, mediapipe, and codeformer.
Add .vscode/ with Biome formatter, Tailwind, Vitest, Playwright, and
Python debug configs. Extract shared pnpm/Node setup into a composite
GitHub Action and add Dependabot and dependency-review workflows.
rembg 2.0.62 has both CVE-2026-40086 and GHSA-55v6-g8pm-pw4c
(same vulnerability, different ID sources). Both need ignoring
since upgrading rembg to 2.0.75 breaks the dependency tree.
Pillow 12.x conflicts with pinned numpy 1.26.4, rembg, realesrgan,
and mediapipe. Revert to working 11.1.0 pins and ignore the CVEs
in pip-audit instead — they require a coordinated major version
upgrade across the entire ML stack (Pillow, numpy, torch, basicsr).
Ignored CVEs:
- CVE-2024-27763 (basicsr, no fix available)
- CVE-2026-40086 (rembg, fix needs Pillow 12)
- CVE-2026-25990 (Pillow, fix is 12.1.1)
- CVE-2026-40192 (Pillow, fix is 12.2.0)
- Increase QR generate max-size test timeout to 120s (10000x10000
PNG generation exceeds 30s default on CI runners)
- Update Pillow 11.1.0 → >=12.2.0 (CVE-2026-25990, CVE-2026-40192)
- Update rembg 2.0.62 → >=2.0.75 (CVE-2026-40086)
- Update opencv-python-headless to flexible range >=4.10,<4.12
- Ignore CVE-2024-27763 in pip-audit (basicsr transitive dep from
realesrgan, no fix available upstream)
- Align requirements-gpu.txt and Dockerfile with same versions
Closes#17, #18, #19, #31, #32, #33, #34
Format preservation (#17, #18, #19):
- Add resolveOutputFormat to rotate, resize, text-overlay, watermark-text,
border, replace-color, blur-faces, upscale, erase-object, restore-photo
- Alpha-aware fallback: border with corner radius/shadow and replace-color
with makeTransparent fall back to PNG for non-alpha formats (JPEG)
- Python sidecar tools (blur-faces, upscale, erase-object) now convert
PNG output back to input format, matching restore-photo/colorize pattern
- Upscale and erase-object default to "auto" format detection instead of PNG
Dispatcher stability (#31, #32):
- Add gc.collect() and torch.cuda.empty_cache() after each dispatcher request
- Add configurable max_requests (default 50) for periodic dispatcher restart
- Add exponential backoff to dispatcher crash recovery in bridge.ts
- Circuit breaker: 5 crashes within 60s permanently disables dispatcher
- Reset crash counter on successful dispatcher startup
Health & security (#33, #34):
- Export getDispatcherStatus() from @snapotter/ai with running/ready/failed/
gpu/pid/consecutiveCrashes fields
- Admin health endpoint now includes full dispatcher status
- Add pip-audit job to CI workflow for Python dependency scanning
- Add Cloudflare Pages deployment for landing page (snapotter.com) and
docs (docs.snapotter.com)
- Create deploy-landing.yml and update deploy-docs.yml workflows
- Update CI to ignore apps/landing/** paths
- Fix logo transparency (remove white background) across all apps
- Recreate social-preview.png with SnapOtter branding
- Update all docs URLs from GitHub Pages to docs.snapotter.com
- Update VitePress config: light theme default, fix llms.txt paths
- Add .vitepress/cache/ and .env.* to gitignore
- Parallelize all 14 model downloads using ThreadPoolExecutor (6 workers)
Downloads were sequential (~30 min), now concurrent (~5-10 min)
- Switch Docker cache from type=gha to type=registry (GHCR)
GHA cache has 10 GB limit causing blob eviction and corrupted builds
Registry cache has no size limit and persists across runner instances
- Add pip download cache mounts to all pip install layers
Prevents re-downloading packages when layers rebuild
- ci.yml: skip lint/test/docker on docs-only and markdown-only pushes
- deploy-docs.yml: only rebuild GitHub Pages when apps/docs/** changes
- README.md: updated key features and content
- images: updated dashboard screenshots, removed stale images
Replace CGO_ENABLED=0 (which fails because gioui.org requires CGO on Linux)
with a proper C cross-compiler approach using Debian multi-arch packages.
Running caire-builder with --platform=\$BUILDPLATFORM avoids QEMU crashes on
Apple Silicon; the C cross-compiler bridges the CGO gap for the target arch.
Also adds SKIP_MODEL_DOWNLOADS=true to the CI docker build job to prevent
HuggingFace CDN 504s in CI (image structure is what matters there).
urlretrieve against the HuggingFace CDN was consistently returning HTTP
504 in GitHub Actions runners for LaMa, NAFNet, and the OpenCV caffemodel.
The huggingface_hub library has built-in retry logic, resumable downloads,
and better CDN routing than bare urlretrieve.
- download_lama_model: urlretrieve → hf_hub_download (Carve/LaMa-ONNX)
- download_nafnet_model: urlretrieve → hf_hub_download (mikestealth/nafnet-models)
- download_opencv_colorize_models: caffemodel → hf_hub_download (space repo_type)
- _urlretrieve: retry count 3→5, flat 10s delay → exponential backoff (10/20/40/80s)
- Also reverts the SKIP_MODEL_DOWNLOADS=true from CI workflow (wrong approach)
The Docker Build Test was consistently failing because HuggingFace CDN
returns 504 Gateway Timeout when downloading the LaMa ONNX model (~200MB)
from GitHub Actions runners. Model availability is an external dependency,
not something CI can control.
Added SKIP_MODEL_DOWNLOADS build arg (default: false). When set to true,
the download_models.py step is skipped entirely. CI only needs to verify
the image structure builds — Python deps install, Node build runs, app
code is copied — not that every ML model CDN is reachable.
Production builds (docker build without the arg) still download all models
as before.
The edit-metadata integration tests require exiftool (libimage-exiftool-perl)
which was missing from the CI test runner, causing 4 tests to fail with 422.
Merge CPU, CUDA, and lite Docker images into a single unified image.
One tag (latest) works on all platforms: amd64 (NVIDIA CUDA) and arm64 (CPU).
GPU auto-detected at runtime. All ML models and packages baked in.
Key changes:
- Platform-conditional Dockerfile (nvidia/cuda on amd64, node on arm64)
- tini as PID 1 for proper signal handling
- Fix FILES_STORAGE_PATH data loss bug
- Fix RealESRGAN upscaler (was broken, always fell back to Lanczos)
- Fix PaddleOCR language codes and stdout corruption
- Simplified CI/CD (single build, single tag)
- Expanded model pre-download with verification
- Shutdown timeout, improved health endpoint
- Remove unused lama-cleaner
Add a :cuda Docker image tag that auto-detects NVIDIA GPU at runtime
and falls back gracefully to CPU. Same pattern as Immich.
- New gpu.py shared utility for cached CUDA detection
- Background removal (rembg): pass CUDAExecutionProvider to ONNX Runtime
- Upscaling (Real-ESRGAN): use CUDA device + FP16 when GPU available
- OCR (PaddleOCR): enable use_gpu when CUDA detected
- Dispatcher reports GPU status at startup via readiness signal
- Admin health endpoint exposes GPU availability
- Dockerfile uses ARG GPU=false with conditional NVIDIA CUDA base image
- docker-compose.gpu.yml override for GPU users
- CI/CD workflows build and publish :cuda tag (amd64 only)
Three tags: :latest (CPU), :lite (no AI), :cuda (GPU with CPU fallback)
Ubuntu 24.04 uses plugin-based libheif codecs. Added libheif-plugin-x265
(HEVC encoder) and libheif-plugin-libde265 (HEVC decoder) to the CI test
job. Debian bookworm (Docker) bundles these in libheif1 directly.
- Remove @fastify/swagger and @fastify/swagger-ui (API docs live on GitHub Pages)
- Run typecheck, build, and docker CI jobs in parallel instead of sequentially
