* feat(infra): add dev compose stack with postgres and redis
* fix(infra): comment dev env defaults until wired; harden dev compose restart and start_period
* chore(deps): add pg driver and testcontainers for postgres migration
* feat(db): translate schema to drizzle pg-core (timestamptz, boolean, pgEnum, jsonb)
Schema translation (apps/api/src/db/schema.ts):
- sqlite-core -> pg-core, all 10 tables preserved 1:1
- integer(mode:'timestamp') -> timestamp({ withTimezone: true })
- integer(mode:'boolean') -> boolean
- jobs.status text enum -> pgEnum('job_status') with same 4 values
- 7 columns changed from text to jsonb: jobs.inputFiles, jobs.settings,
pipelines.steps, apiKeys.permissions, roles.permissions,
auditLog.details, userFiles.toolChain
- settings.value stays text, jobs.error stays text, jobs.progress stays real
jsonb call-site sweep (removed JSON.stringify on writes, JSON.parse on reads):
- apps/api/src/routes/roles.ts: permissions read/write (3 sites)
- apps/api/src/routes/api-keys.ts: permissions write + read (2 sites)
- apps/api/src/routes/audit-log.ts: details read (1 site)
- apps/api/src/routes/pipeline.ts: steps write + read (2 sites)
- apps/api/src/routes/progress.ts: inputFiles write (2 sites)
- apps/api/src/routes/tool-factory.ts: toolChain read + write (2 sites)
- apps/api/src/routes/user-files.ts: toolChain read + write (4 sites)
- apps/api/src/permissions.ts: roles.permissions read (1 site)
- apps/api/src/lib/audit.ts: details write (1 site)
- apps/api/src/plugins/auth.ts: apiKeys.permissions read (1 site)
* refactor(db): type jsonb columns via $type and note raw CTE conversion requirements
* feat(db): archive sqlite migrations and generate postgres baseline
* chore(db): dockerignore legacy migrations, add archive breadcrumb, fix trailing newline
* feat(db): pg pool connection, advisory-locked boot migrations, DATABASE_URL config
* fix(db): friendly fatal on unreachable postgres, idempotent closeDb, lock-key convention note
* refactor(db): async drizzle calls in plugins, lib, permissions
* fix(api): analytics never throws, typed permission guard, single-query session invalidation
* refactor(db): async drizzle calls across all routes and bootstrap
Convert every route file and index.ts from sync SQLite drizzle
patterns to async node-postgres drizzle:
- .all() removed (bare await on select)
- .get() converted to destructured [row] = await ...
- .run() removed (bare await on insert/update/delete)
- .changes replaced with .rowCount (null-guarded) in progress.ts
- sqlite import removed from user-files.ts; raw CTEs converted to
await db.execute(sql`...`) with postgres-dialect recursive CTEs
- ChainRow types updated: tool_chain is parsed jsonb (string[] | null),
created_at is Date (timestamptz) with no * 1000 conversion
- All requirePermission() guard calls awaited (security: unawaited
async guard returns truthy Promise, bypassing permission check)
- All hasEffectivePermission() and getPermissions() calls awaited
- All auditLog() calls awaited (preserves write-before-response order)
- trackEvent() and captureException() left un-awaited (fire-and-forget
by design, guaranteed never-throw)
- ensureAnonymousUser(), startCleanupCron(), recoverStaleJobs() awaited
in bootstrap sequence
- ensureInstanceId() and ensureDefaultSettings() made async
Files converted: 14 (index.ts + 12 route files + tools/index.ts)
* fix(db): await async checkStorageQuota in user-files upload/save routes
* fix(db): await checkStorageQuota in save-result route (missed second call site)
* feat(db): sqlite-to-postgres migrator with CLI and first-boot import
* fix(db): migrator error context, honest force semantics, boot-hook fatal, null-variance tests
* test: run suite against per-file postgres databases via testcontainers
- Add tests/global-setup.ts: spins up a Postgres testcontainer,
creates a migrated template database once per vitest run.
- Rewrite tests/setup/per-fork-env.ts: each test file (forks pool)
clones the template into its own database via CREATE DATABASE ...
TEMPLATE, preserving the same per-file isolation granularity.
- Update vitest.config.ts: add globalSetup, pg alias, update comment.
- Fix tests/integration/test-server.ts: remove DB_PATH mkdir, async
runMigrations, async db operations, remove SQLite WAL checkpoint.
- Fix 21 unit test db/index mocks: add pool and closeDb exports.
- Fix 8 unit test files: add async/await for now-async permission,
audit, and analytics functions.
- Fix 18 integration test files: convert sync .run()/.all()/.get()
to async drizzle patterns, add async to callbacks.
- Production change: apps/api/src/routes/teams.ts: cast COUNT(*)
to ::int so Postgres returns a number instead of bigint string.
* fix(db): seed built-in roles, reject NUL bytes, cast COUNT, serialize job persists
- Seed built-in roles (admin, editor, user) at boot via ensureBuiltinRoles()
with onConflictDoNothing, restoring data that legacy SQLite migration 0007
provided via INSERT statements (the pg baseline is DDL-only).
- Reject NUL bytes in login credentials with 401 (postgres rejects \x00 in
text columns; valid usernames never contain NUL, matching 1.x behavior).
- Cast COUNT(*)::int in user-files, audit-log, and roles listing queries so
postgres returns a JS number instead of bigint-as-string.
- Serialize fire-and-forget job progress DB writes per jobId so the final
"completed" status is never overwritten by a late-arriving "processing"
write (race condition exposed by async postgres round-trips).
* test: fix teams race, seed roles in test server, poll for job status
- Add missing await to resetTeams() in teams PUT beforeEach (the async
delete raced with the subsequent insert under postgres).
- Call ensureBuiltinRoles() in test server bootstrap so integration tests
have the same built-in roles as production.
- Replace fixed 100ms flushPersist delay with a polling helper that waits
for terminal job status, eliminating timing-dependent failures caused by
postgres network round-trip latency.
* test: make heic temp-file cleanup assertion resilient to concurrent workers
Use a set-based diff instead of raw file count when checking that
decodeHeic cleans up temp files. Other concurrent test workers can
create heic-in-*/heic-out-* files in the shared tmpdir, inflating the
"after" count and causing spurious failures under full-suite load.
* fix(db): align builtin-role seed to post-0010 legacy state; test polish
* feat(docker): three-container compose (app, postgres, redis) with boot wait and migrations
* fix(docker): set TEST_DATABASE_URL so containerized tests skip testcontainers
* chore(docker): test compose project name, clearer 1.x upgrade comment, unref probe timer
* feat(enterprise): enforce D15 license boundary; move s3 storage into packages/enterprise
* fix(enterprise): restore lazy aws-sdk loading; community installs load no s3 code at boot
* fix(enterprise): boundary check catches dynamic imports; document getS3 concurrency
* feat(db)!: SnapOtter 2.0 phase 1 foundation: postgres, migrator, compose stack
BREAKING CHANGE: SQLite is no longer the runtime database. Deployments now
require Postgres (and Redis, used from phase 2). Existing installs migrate
with SQLITE_MIGRATE_PATH or 'pnpm --filter @snapotter/api migrate:sqlite'.
* fix(ci): postgres service + fresh e2e database per run; ignore unfixable torch CVE-2025-3000
Closes the "e2e never runs in CI" hole. Adds per-PR e2e smoke gate,
nightly full-suite workflows, parallel vitest forks (per-fork DBs),
Playwright parallel/serial/visual projects against production builds,
metadata-generated test suites (drift guards, hostile inputs, format
matrix, pairwise settings, property-based fuzz), Stryker mutation
testing, Schemathesis API fuzz, coverage ratchet, and fixes for three
session-poisoning bugs that caused 200+ serial-bucket failures.
Bug fix included: favicon/split/bulk-rename could hang clients forever
when ZIP streaming failed after reply.hijack().
The Trivy scan finds HIGH CVEs in pnpm's own transitive dependencies
(glob, minimatch, tar, picomatch) which are build-time only and not
in the runtime image. These block manifest creation unnecessarily.
Scan results still upload to GitHub Security tab via SARIF.
- Add v1.17.2 entry to docs/changelog.md
- Add "Update docs changelog" step to release workflow that
auto-prepends .release-notes.md to the docs changelog on
each release, then commits and pushes to trigger docs deploy
Deleting .github breaks the Post Run cleanup for composite actions,
causing the entire job to fail. Use tar --exclude instead so the
working directory stays intact for GitHub Actions cleanup.
The archive step removes .git before creating the tarball, but
gh release upload needs git context to resolve the repo. Pass
--repo explicitly to avoid "not a git repository" errors.
Generate .sha256 checksum files alongside each prebuilt tarball for
download integrity verification. Stop stripping the LICENSE file from
archives to comply with AGPL-3.0 distribution requirements.
Add pre-built release archives (Linux amd64/arm64) to the release
workflow, published as GitHub Release assets. Each archive is a
self-contained tar.gz (~240MB) with built frontend, API source,
and production node_modules. Users extract and run without needing
pnpm build.
Also includes AI install manifest fixes for Proxmox/bare-metal users:
- Pin setuptools<75 for Python 3.13 basicsr compatibility
- Pre-install basicsr with --no-build-isolation before realesrgan
- Loosen mediapipe pins from == to >= for Python 3.13 wheels
- Add retry logic to HuggingFace model downloads
No fixed versions available yet for:
- torch 2.12.0: 11 PYSEC advisories (transitive dep from ML packages)
- joblib 1.5.3: PYSEC-2024-277
- markdown 3.10.2: PYSEC-2026-89
Token-Permissions (0 -> 10): Set permissions: {} at workflow top level
across all 7 workflows, moved write scopes to per-job minimum.
SAST (0 -> 10): Added CodeQL workflow for JavaScript/TypeScript and
Python analysis on push, PR, and weekly schedule.
Vulnerabilities (0 -> ~8): Added 13 pnpm overrides to patch transitive
dependency vulnerabilities (38 -> 2 remaining, both in dev-only tools).
Pinned-Dependencies (5 -> 8-9): Pinned all Docker FROM images to SHA
digests, pinned pip-audit version in CI, pinned pip version in
Dockerfile.
- Separate unit tests (fast, no system deps) from integration test
shards to prevent vi.mock db leakage across test types
- Fix hero test: update expected subtitle to match current copy
- Fix FAQ test: add missing json-ld alias in vitest config and mock
- Integration tests run in 4 parallel shards (30min timeout each)
The full test suite takes ~100 minutes sequentially. Split into 4
parallel shards using Vitest's --shard flag so each completes in
~25 minutes. Removed coverage from CI (was causing overhead without
being reported anywhere). 30-minute timeout per shard as safety net.
The sample.heif was 8736x5856 (2.5MB), causing each processing
operation to take 17-38s in CI. Resized to 1432x960 (224KB) and added
a 15-minute timeout to the test job.
- CODEOWNERS: use @snapotter-hq username instead of non-existent team
- PR template: use absolute URLs (relative links break in PR body)
- CLA: strengthen entity definition with successor/assignee clause
- CLA workflow: store signatures on dedicated branch, add owner to allowlist
- CONTRIBUTING: clarify approval means the "approved" label, reword CCLA
- Bug template: allow "running from source" for developer contributors
Add full contribution infrastructure: CLA with broad sublicensing
rights for dual-licensing, CONTRIBUTING.md with scope rules and dev
setup, CLA Assistant workflow, CODEOWNERS, SUPPORT.md, PR template,
updated issue templates with contribution prompts, and declarative
label config.
Auth: login rate limit 30/min (was 500), global rate limit 1000/min (was
unlimited), password/username max lengths on all Zod schemas, session
invalidation on role change, API key legacy scan bounded to 100 keys.
SVG: hardened regex sanitizer with CDATA stripping, XML entity decoding,
set/animate/iframe/embed blocking, comprehensive data: URI blocking,
use element external href blocking. 11 attack payload fixtures added.
SSRF: fixed DNS rebinding TOCTOU by pinning resolved IPs via custom
HTTP/HTTPS agents. Added 6to4 and NAT64 to blocked IPv6 ranges.
Docker: capability dropping (cap_drop ALL + minimal cap_add), resource
limits (4g/8g mem, 512/1024 pids), healthcheck timeout, password
removed from startup banner, default password warning comments.
Network: CSP and HSTS applied in all environments (not just production),
stack traces removed from all error responses, internal paths stripped
from error details, per-route rate limits on uploads (60/min) and URL
fetches (200/hour).
Files: exclusive temp file creation (O_EXCL), disk space circuit
breaker, per-user storage quotas, settings payload 64KB size guard.
Python sidecar: script name allowlist in dispatcher, minimal environment
for subprocess spawns.
Dependencies: fixed 6 production CVEs (drizzle-orm, fastify, fast-uri,
@fastify/static, next, archiver/lodash). Pinned all GitHub Actions to
SHA hashes.
114 security tests added. Full OWASP Top 10 penetration test matrix
verified against production Docker container (30/30 pass after
hardening).
Install ImageMagick, Ghostscript, libjxl-tools, and libopenjp2-tools in
CI so exotic format decoder tests (PSD, EPS, HDR, ICO, JP2, etc.) can
run. Relax ImageMagick EPS/PS security policy to match the Dockerfile.
Replace fragile vi.mock() of the SSRF module in fetch-urls tests with an
env-var guard (SSRF_ALLOW_PRIVATE) that bypasses private-IP checks in
the test environment. The vi.mock approach broke under V8 coverage
instrumentation in CI.
Scans the amd64 Docker image for CRITICAL/HIGH CVEs with available fixes
before publishing multi-arch manifests. Results upload to GitHub Security tab
as SARIF. Blocks release if fixable vulnerabilities are found.
Pillow 11.1.0 has CVE-2026-42308, CVE-2026-42310, CVE-2026-42311
(fixed in 12.2.0). Upgrading to Pillow 12.x requires validating
compatibility with rembg, realesrgan, mediapipe, and codeformer.
Add .vscode/ with Biome formatter, Tailwind, Vitest, Playwright, and
Python debug configs. Extract shared pnpm/Node setup into a composite
GitHub Action and add Dependabot and dependency-review workflows.
rembg 2.0.62 has both CVE-2026-40086 and GHSA-55v6-g8pm-pw4c
(same vulnerability, different ID sources). Both need ignoring
since upgrading rembg to 2.0.75 breaks the dependency tree.
Pillow 12.x conflicts with pinned numpy 1.26.4, rembg, realesrgan,
and mediapipe. Revert to working 11.1.0 pins and ignore the CVEs
in pip-audit instead — they require a coordinated major version
upgrade across the entire ML stack (Pillow, numpy, torch, basicsr).
Ignored CVEs:
- CVE-2024-27763 (basicsr, no fix available)
- CVE-2026-40086 (rembg, fix needs Pillow 12)
- CVE-2026-25990 (Pillow, fix is 12.1.1)
- CVE-2026-40192 (Pillow, fix is 12.2.0)
- Increase QR generate max-size test timeout to 120s (10000x10000
PNG generation exceeds 30s default on CI runners)
- Update Pillow 11.1.0 → >=12.2.0 (CVE-2026-25990, CVE-2026-40192)
- Update rembg 2.0.62 → >=2.0.75 (CVE-2026-40086)
- Update opencv-python-headless to flexible range >=4.10,<4.12
- Ignore CVE-2024-27763 in pip-audit (basicsr transitive dep from
realesrgan, no fix available upstream)
- Align requirements-gpu.txt and Dockerfile with same versions
Closes#17, #18, #19, #31, #32, #33, #34
Format preservation (#17, #18, #19):
- Add resolveOutputFormat to rotate, resize, text-overlay, watermark-text,
border, replace-color, blur-faces, upscale, erase-object, restore-photo
- Alpha-aware fallback: border with corner radius/shadow and replace-color
with makeTransparent fall back to PNG for non-alpha formats (JPEG)
- Python sidecar tools (blur-faces, upscale, erase-object) now convert
PNG output back to input format, matching restore-photo/colorize pattern
- Upscale and erase-object default to "auto" format detection instead of PNG
Dispatcher stability (#31, #32):
- Add gc.collect() and torch.cuda.empty_cache() after each dispatcher request
- Add configurable max_requests (default 50) for periodic dispatcher restart
- Add exponential backoff to dispatcher crash recovery in bridge.ts
- Circuit breaker: 5 crashes within 60s permanently disables dispatcher
- Reset crash counter on successful dispatcher startup
Health & security (#33, #34):
- Export getDispatcherStatus() from @snapotter/ai with running/ready/failed/
gpu/pid/consecutiveCrashes fields
- Admin health endpoint now includes full dispatcher status
- Add pip-audit job to CI workflow for Python dependency scanning
- Fix test-with-exif.jpg Software field from "ashim Test" to
"SnapOtter Test" to match test expectations
- Replace cloudflare/wrangler-action with npx wrangler to avoid
pnpm workspace root install error
- Add Cloudflare Pages deployment for landing page (snapotter.com) and
docs (docs.snapotter.com)
- Create deploy-landing.yml and update deploy-docs.yml workflows
- Update CI to ignore apps/landing/** paths
- Fix logo transparency (remove white background) across all apps
- Recreate social-preview.png with SnapOtter branding
- Update all docs URLs from GitHub Pages to docs.snapotter.com
- Update VitePress config: light theme default, fix llms.txt paths
- Add .vitepress/cache/ and .env.* to gitignore
- Parallelize all 14 model downloads using ThreadPoolExecutor (6 workers)
Downloads were sequential (~30 min), now concurrent (~5-10 min)
- Switch Docker cache from type=gha to type=registry (GHCR)
GHA cache has 10 GB limit causing blob eviction and corrupted builds
Registry cache has no size limit and persists across runner instances
- Add pip download cache mounts to all pip install layers
Prevents re-downloading packages when layers rebuild
Allows re-triggering the release workflow after a Docker push failure
without needing new commits. If semantic-release produces no new version
the workflow now uses the latest existing git tag for the Docker build.
- ci.yml: skip lint/test/docker on docs-only and markdown-only pushes
- deploy-docs.yml: only rebuild GitHub Pages when apps/docs/** changes
- README.md: updated key features and content
- images: updated dashboard screenshots, removed stale images
Replace CGO_ENABLED=0 (which fails because gioui.org requires CGO on Linux)
with a proper C cross-compiler approach using Debian multi-arch packages.
Running caire-builder with --platform=\$BUILDPLATFORM avoids QEMU crashes on
Apple Silicon; the C cross-compiler bridges the CGO gap for the target arch.
Also adds SKIP_MODEL_DOWNLOADS=true to the CI docker build job to prevent
HuggingFace CDN 504s in CI (image structure is what matters there).
urlretrieve against the HuggingFace CDN was consistently returning HTTP
504 in GitHub Actions runners for LaMa, NAFNet, and the OpenCV caffemodel.
The huggingface_hub library has built-in retry logic, resumable downloads,
and better CDN routing than bare urlretrieve.
- download_lama_model: urlretrieve → hf_hub_download (Carve/LaMa-ONNX)
- download_nafnet_model: urlretrieve → hf_hub_download (mikestealth/nafnet-models)
- download_opencv_colorize_models: caffemodel → hf_hub_download (space repo_type)
- _urlretrieve: retry count 3→5, flat 10s delay → exponential backoff (10/20/40/80s)
- Also reverts the SKIP_MODEL_DOWNLOADS=true from CI workflow (wrong approach)
The Docker Build Test was consistently failing because HuggingFace CDN
returns 504 Gateway Timeout when downloading the LaMa ONNX model (~200MB)
from GitHub Actions runners. Model availability is an external dependency,
not something CI can control.
Added SKIP_MODEL_DOWNLOADS build arg (default: false). When set to true,
the download_models.py step is skipped entirely. CI only needs to verify
the image structure builds — Python deps install, Node build runs, app
code is copied — not that every ML model CDN is reachable.
Production builds (docker build without the arg) still download all models
as before.
SnapOtter