RAW (DNG) processing crashed on ImageMagick's deprecated ufraw-batch
delegate, which fails on modern formats such as iPhone ProRAW DNG.
Root cause: the dcraw_emu (LibRaw) decode tier read the wrong output path.
dcraw_emu APPENDS the output extension (raw-in-X.dng -> raw-in-X.dng.tiff)
but the code looked for raw-in-X.tiff (replaced extension), so readFile threw
on every RAW, the tier silently fell through to ufraw, and the 24MB TIFF
leaked into the temp dir on each attempt.
- Repair the dcraw_emu output path; clean it up in finally (fixes the leak)
- Prefer LibRaw full decode over embedded-preview extraction so a
full-resolution RAW is never silently returned as a reduced-size preview
(sample DNG: was 1024x683 preview, now 3474x2314 full)
- Add RAW decode regression tests (DNG full-resolution + all 6 RAW formats);
these were absent, which let the bug ship
- Install libraw-bin on CI test runners so dcraw_emu is actually exercised
The verify job runs between build and publish, gating HuggingFace
upload on per-bundle smoke tests. Each of the 14 bundle variants
(7 bundles x 2 arches) is verified independently: SHA256 checksum,
tarball extraction, Python import checks, and functional inference.
- fixture-integrity: probe media via media-engine probeMedia (resolves the
bundled static ffmpeg) instead of bare system ffprobe, which is ENOENT in CI;
gate on ffmpegAvailable() like the other media tests
- a11y: regenerate a11y-baseline.json to include the mobile device keys (the
baseline only had desktop keys, so the mobile a11y scan saw them as new)
- device-visual: tag @visual and exclude it from the PR mobile-smoke gate
(darwin-only screenshots cannot pass on linux; nightly + update-visual-baselines
still run it to seed linux goldens)
Fixes all integration CI failures on the 2.0 branch.
## What was broken
Two independent root causes:
1. **202 assertion failures** -- Under 4-fork CI parallel load, the 30s
`SYNC_WAIT_MS` sync window can expire before a BullMQ worker finishes a
heavy encode (avif, heic), returning a legitimate `202 {jobId, async: true}`
instead of `200`. Tests that hard-asserted `200` were spuriously failing.
2. **Vitest timeout race** -- `SYNC_WAIT_MS` (30s) and the default Vitest
`testTimeout` (also 30s) fired simultaneously. Vitest won the race,
reporting "Test timed out in 30000ms" instead of the test receiving the
202 response.
## Fixes
- Added `isAsyncFallback()` helper to four integration test files; validates
the `{async: true, jobId}` body shape and returns early so the synchronous
200 path runs full assertions only when warranted.
- Set `vi.setConfig({ testTimeout: 60_000 })` at module level in
`image-enhancement.test.ts` and `format-matrix-comprehensive.test.ts`,
giving a 30s buffer between when `waitForJob()` returns 202 and when
Vitest gives up.
- Bumped explicit matrix timeouts in `format-matrix.test.ts` and
`new-formats.test.ts` from 30s to 60s for the same reason.
- Installed missing CI doc-engine binaries (qpdf, pandoc, libreoffice,
pdfcpu) that were causing unrelated integration failures.
- Fixed E2E smoke specs for 2.0 UI changes (modality selector, tool routes,
validation behavior).
The Release workflow failed at startup ("workflow file issue", no jobs
created) whenever dispatched. Root cause: release.yml sets top-level
`permissions: {}`, so the ai-bundles reusable-workflow call inherited an
empty scope set. ai-bundles.yml's jobs declare `contents: read` /
`packages: read`, and GitHub rejects a called workflow requesting scopes
the caller never granted -- validated at startup, before any job runs, so
the whole run fails immediately regardless of the job's `if:`.
Grant the two scopes on the call job. Confirmed on a throwaway branch: a
neutralized dispatch went from startup_failure to a clean run with
ai-bundles correctly skipped.
This unblocks every real release; main's release.yml could not start.
Overhaul the bug report form and restructure issue intake for 2.0:
- Bug report: split actual vs expected behavior, add Affected area and
Specific tool fields for triage, replace the self-defeating required
"latest version" checkbox with a Version field that accepts a tag,
release, or commit (always answerable from any install), modernize
Docker guidance to the Compose stack, add Host OS, and fix the
previously required image-tag field that source users could not fill.
- Move feature requests to GitHub Discussions: delete the feature_request
issue form, rename the orphaned discussion form to ideas.yml so it binds
to the built-in Ideas category, and point config.yml there.
- Add Translation and Documentation issue forms routed to existing labels.
- Remove the roadmap-update discussion form (roadmap is private).
- Reconcile CONTRIBUTING.md, SUPPORT.md, and the published docs
contributing guide with the new routing.
- Use /opt/venv directly when --entrypoint bash bypasses entrypoint.sh
- Use sys.executable for all pip calls (not bare pip)
- Override entrypoint in CI workflow to avoid startup banner
- Fix Biome formatting (template literals, try/catch blocks)
* feat(infra): add dev compose stack with postgres and redis
* fix(infra): comment dev env defaults until wired; harden dev compose restart and start_period
* chore(deps): add pg driver and testcontainers for postgres migration
* feat(db): translate schema to drizzle pg-core (timestamptz, boolean, pgEnum, jsonb)
Schema translation (apps/api/src/db/schema.ts):
- sqlite-core -> pg-core, all 10 tables preserved 1:1
- integer(mode:'timestamp') -> timestamp({ withTimezone: true })
- integer(mode:'boolean') -> boolean
- jobs.status text enum -> pgEnum('job_status') with same 4 values
- 7 columns changed from text to jsonb: jobs.inputFiles, jobs.settings,
pipelines.steps, apiKeys.permissions, roles.permissions,
auditLog.details, userFiles.toolChain
- settings.value stays text, jobs.error stays text, jobs.progress stays real
jsonb call-site sweep (removed JSON.stringify on writes, JSON.parse on reads):
- apps/api/src/routes/roles.ts: permissions read/write (3 sites)
- apps/api/src/routes/api-keys.ts: permissions write + read (2 sites)
- apps/api/src/routes/audit-log.ts: details read (1 site)
- apps/api/src/routes/pipeline.ts: steps write + read (2 sites)
- apps/api/src/routes/progress.ts: inputFiles write (2 sites)
- apps/api/src/routes/tool-factory.ts: toolChain read + write (2 sites)
- apps/api/src/routes/user-files.ts: toolChain read + write (4 sites)
- apps/api/src/permissions.ts: roles.permissions read (1 site)
- apps/api/src/lib/audit.ts: details write (1 site)
- apps/api/src/plugins/auth.ts: apiKeys.permissions read (1 site)
* refactor(db): type jsonb columns via $type and note raw CTE conversion requirements
* feat(db): archive sqlite migrations and generate postgres baseline
* chore(db): dockerignore legacy migrations, add archive breadcrumb, fix trailing newline
* feat(db): pg pool connection, advisory-locked boot migrations, DATABASE_URL config
* fix(db): friendly fatal on unreachable postgres, idempotent closeDb, lock-key convention note
* refactor(db): async drizzle calls in plugins, lib, permissions
* fix(api): analytics never throws, typed permission guard, single-query session invalidation
* refactor(db): async drizzle calls across all routes and bootstrap
Convert every route file and index.ts from sync SQLite drizzle
patterns to async node-postgres drizzle:
- .all() removed (bare await on select)
- .get() converted to destructured [row] = await ...
- .run() removed (bare await on insert/update/delete)
- .changes replaced with .rowCount (null-guarded) in progress.ts
- sqlite import removed from user-files.ts; raw CTEs converted to
await db.execute(sql`...`) with postgres-dialect recursive CTEs
- ChainRow types updated: tool_chain is parsed jsonb (string[] | null),
created_at is Date (timestamptz) with no * 1000 conversion
- All requirePermission() guard calls awaited (security: unawaited
async guard returns truthy Promise, bypassing permission check)
- All hasEffectivePermission() and getPermissions() calls awaited
- All auditLog() calls awaited (preserves write-before-response order)
- trackEvent() and captureException() left un-awaited (fire-and-forget
by design, guaranteed never-throw)
- ensureAnonymousUser(), startCleanupCron(), recoverStaleJobs() awaited
in bootstrap sequence
- ensureInstanceId() and ensureDefaultSettings() made async
Files converted: 14 (index.ts + 12 route files + tools/index.ts)
* fix(db): await async checkStorageQuota in user-files upload/save routes
* fix(db): await checkStorageQuota in save-result route (missed second call site)
* feat(db): sqlite-to-postgres migrator with CLI and first-boot import
* fix(db): migrator error context, honest force semantics, boot-hook fatal, null-variance tests
* test: run suite against per-file postgres databases via testcontainers
- Add tests/global-setup.ts: spins up a Postgres testcontainer,
creates a migrated template database once per vitest run.
- Rewrite tests/setup/per-fork-env.ts: each test file (forks pool)
clones the template into its own database via CREATE DATABASE ...
TEMPLATE, preserving the same per-file isolation granularity.
- Update vitest.config.ts: add globalSetup, pg alias, update comment.
- Fix tests/integration/test-server.ts: remove DB_PATH mkdir, async
runMigrations, async db operations, remove SQLite WAL checkpoint.
- Fix 21 unit test db/index mocks: add pool and closeDb exports.
- Fix 8 unit test files: add async/await for now-async permission,
audit, and analytics functions.
- Fix 18 integration test files: convert sync .run()/.all()/.get()
to async drizzle patterns, add async to callbacks.
- Production change: apps/api/src/routes/teams.ts: cast COUNT(*)
to ::int so Postgres returns a number instead of bigint string.
* fix(db): seed built-in roles, reject NUL bytes, cast COUNT, serialize job persists
- Seed built-in roles (admin, editor, user) at boot via ensureBuiltinRoles()
with onConflictDoNothing, restoring data that legacy SQLite migration 0007
provided via INSERT statements (the pg baseline is DDL-only).
- Reject NUL bytes in login credentials with 401 (postgres rejects \x00 in
text columns; valid usernames never contain NUL, matching 1.x behavior).
- Cast COUNT(*)::int in user-files, audit-log, and roles listing queries so
postgres returns a JS number instead of bigint-as-string.
- Serialize fire-and-forget job progress DB writes per jobId so the final
"completed" status is never overwritten by a late-arriving "processing"
write (race condition exposed by async postgres round-trips).
* test: fix teams race, seed roles in test server, poll for job status
- Add missing await to resetTeams() in teams PUT beforeEach (the async
delete raced with the subsequent insert under postgres).
- Call ensureBuiltinRoles() in test server bootstrap so integration tests
have the same built-in roles as production.
- Replace fixed 100ms flushPersist delay with a polling helper that waits
for terminal job status, eliminating timing-dependent failures caused by
postgres network round-trip latency.
* test: make heic temp-file cleanup assertion resilient to concurrent workers
Use a set-based diff instead of raw file count when checking that
decodeHeic cleans up temp files. Other concurrent test workers can
create heic-in-*/heic-out-* files in the shared tmpdir, inflating the
"after" count and causing spurious failures under full-suite load.
* fix(db): align builtin-role seed to post-0010 legacy state; test polish
* feat(docker): three-container compose (app, postgres, redis) with boot wait and migrations
* fix(docker): set TEST_DATABASE_URL so containerized tests skip testcontainers
* chore(docker): test compose project name, clearer 1.x upgrade comment, unref probe timer
* feat(enterprise): enforce D15 license boundary; move s3 storage into packages/enterprise
* fix(enterprise): restore lazy aws-sdk loading; community installs load no s3 code at boot
* fix(enterprise): boundary check catches dynamic imports; document getS3 concurrency
* feat(db)!: SnapOtter 2.0 phase 1 foundation: postgres, migrator, compose stack
BREAKING CHANGE: SQLite is no longer the runtime database. Deployments now
require Postgres (and Redis, used from phase 2). Existing installs migrate
with SQLITE_MIGRATE_PATH or 'pnpm --filter @snapotter/api migrate:sqlite'.
* fix(ci): postgres service + fresh e2e database per run; ignore unfixable torch CVE-2025-3000
Closes the "e2e never runs in CI" hole. Adds per-PR e2e smoke gate,
nightly full-suite workflows, parallel vitest forks (per-fork DBs),
Playwright parallel/serial/visual projects against production builds,
metadata-generated test suites (drift guards, hostile inputs, format
matrix, pairwise settings, property-based fuzz), Stryker mutation
testing, Schemathesis API fuzz, coverage ratchet, and fixes for three
session-poisoning bugs that caused 200+ serial-bucket failures.
Bug fix included: favicon/split/bulk-rename could hang clients forever
when ZIP streaming failed after reply.hijack().
The Trivy scan finds HIGH CVEs in pnpm's own transitive dependencies
(glob, minimatch, tar, picomatch) which are build-time only and not
in the runtime image. These block manifest creation unnecessarily.
Scan results still upload to GitHub Security tab via SARIF.
- Add v1.17.2 entry to docs/changelog.md
- Add "Update docs changelog" step to release workflow that
auto-prepends .release-notes.md to the docs changelog on
each release, then commits and pushes to trigger docs deploy
Deleting .github breaks the Post Run cleanup for composite actions,
causing the entire job to fail. Use tar --exclude instead so the
working directory stays intact for GitHub Actions cleanup.
The archive step removes .git before creating the tarball, but
gh release upload needs git context to resolve the repo. Pass
--repo explicitly to avoid "not a git repository" errors.
Generate .sha256 checksum files alongside each prebuilt tarball for
download integrity verification. Stop stripping the LICENSE file from
archives to comply with AGPL-3.0 distribution requirements.
Add pre-built release archives (Linux amd64/arm64) to the release
workflow, published as GitHub Release assets. Each archive is a
self-contained tar.gz (~240MB) with built frontend, API source,
and production node_modules. Users extract and run without needing
pnpm build.
Also includes AI install manifest fixes for Proxmox/bare-metal users:
- Pin setuptools<75 for Python 3.13 basicsr compatibility
- Pre-install basicsr with --no-build-isolation before realesrgan
- Loosen mediapipe pins from == to >= for Python 3.13 wheels
- Add retry logic to HuggingFace model downloads
No fixed versions available yet for:
- torch 2.12.0: 11 PYSEC advisories (transitive dep from ML packages)
- joblib 1.5.3: PYSEC-2024-277
- markdown 3.10.2: PYSEC-2026-89
Token-Permissions (0 -> 10): Set permissions: {} at workflow top level
across all 7 workflows, moved write scopes to per-job minimum.
SAST (0 -> 10): Added CodeQL workflow for JavaScript/TypeScript and
Python analysis on push, PR, and weekly schedule.
Vulnerabilities (0 -> ~8): Added 13 pnpm overrides to patch transitive
dependency vulnerabilities (38 -> 2 remaining, both in dev-only tools).
Pinned-Dependencies (5 -> 8-9): Pinned all Docker FROM images to SHA
digests, pinned pip-audit version in CI, pinned pip version in
Dockerfile.
Remove Ko-fi from FUNDING.yml since GitHub Sponsors is now active.
Add sponsor badge to README badge row and support section. Add
sponsor button to landing page open-source section and footer.
Add heart icon social link to docs site nav bar.
- Separate unit tests (fast, no system deps) from integration test
shards to prevent vi.mock db leakage across test types
- Fix hero test: update expected subtitle to match current copy
- Fix FAQ test: add missing json-ld alias in vitest config and mock
- Integration tests run in 4 parallel shards (30min timeout each)
The full test suite takes ~100 minutes sequentially. Split into 4
parallel shards using Vitest's --shard flag so each completes in
~25 minutes. Removed coverage from CI (was causing overhead without
being reported anywhere). 30-minute timeout per shard as safety net.
The sample.heif was 8736x5856 (2.5MB), causing each processing
operation to take 17-38s in CI. Resized to 1432x960 (224KB) and added
a 15-minute timeout to the test job.
- CODEOWNERS: use @snapotter-hq username instead of non-existent team
- PR template: use absolute URLs (relative links break in PR body)
- CLA: strengthen entity definition with successor/assignee clause
- CLA workflow: store signatures on dedicated branch, add owner to allowlist
- CONTRIBUTING: clarify approval means the "approved" label, reword CCLA
- Bug template: allow "running from source" for developer contributors
Add full contribution infrastructure: CLA with broad sublicensing
rights for dual-licensing, CONTRIBUTING.md with scope rules and dev
setup, CLA Assistant workflow, CODEOWNERS, SUPPORT.md, PR template,
updated issue templates with contribution prompts, and declarative
label config.
Auth: login rate limit 30/min (was 500), global rate limit 1000/min (was
unlimited), password/username max lengths on all Zod schemas, session
invalidation on role change, API key legacy scan bounded to 100 keys.
SVG: hardened regex sanitizer with CDATA stripping, XML entity decoding,
set/animate/iframe/embed blocking, comprehensive data: URI blocking,
use element external href blocking. 11 attack payload fixtures added.
SSRF: fixed DNS rebinding TOCTOU by pinning resolved IPs via custom
HTTP/HTTPS agents. Added 6to4 and NAT64 to blocked IPv6 ranges.
Docker: capability dropping (cap_drop ALL + minimal cap_add), resource
limits (4g/8g mem, 512/1024 pids), healthcheck timeout, password
removed from startup banner, default password warning comments.
Network: CSP and HSTS applied in all environments (not just production),
stack traces removed from all error responses, internal paths stripped
from error details, per-route rate limits on uploads (60/min) and URL
fetches (200/hour).
Files: exclusive temp file creation (O_EXCL), disk space circuit
breaker, per-user storage quotas, settings payload 64KB size guard.
Python sidecar: script name allowlist in dispatcher, minimal environment
for subprocess spawns.
Dependencies: fixed 6 production CVEs (drizzle-orm, fastify, fast-uri,
@fastify/static, next, archiver/lodash). Pinned all GitHub Actions to
SHA hashes.
114 security tests added. Full OWASP Top 10 penetration test matrix
verified against production Docker container (30/30 pass after
hardening).
