Commit Graph
93 Commits
Author SHA1 Message Date
SnapOtterandGitHub 5d36ac06d8 feat(analytics): build-time bake + telemetry depth (#336)
Bake PostHog + Sentry into the published Docker image (SNAPOTTER_ANALYTICS
build arg, codegen script). Delete entire consent system. Move event emission
to BullMQ worker. Add cross-tier identity stitching, Sentry performance
tracing on both tiers, frontend funnel events. Fix stateful regex bug.

86 files changed, 1593 insertions(+), 3747 deletions(-)
2026-06-24 11:05:39 +08:00
SnapOtterandGitHub 35e18d8b79 fix: GPU deployment robustness (6 fixes from end-to-end testing on an RTX 4070) (#334)
* fix(docker): pin CUDA base to 12.6 so the GPU image starts on R560+ drivers

The amd64 base nvidia/cuda:12.9.2-cudnn-runtime bakes a cuda>=12.9 driver gate enforced by nvidia-container-toolkit at container start, so the image fails to launch on common production drivers (e.g. 570.x / CUDA 12.8). The AI bundles are all cu126 wheels and the image installs libcublas-12-6, so 12.9 was misaligned with the workload. Pin to nvidia/cuda:12.6.3-cudnn-runtime-ubuntu24.04 to match the wheels and lower the driver floor to R560+.

* fix(ai): broaden OOM detection so the rembg lighter-model fallback fires

onnxruntime/CUDA allocation failures surface as 'Failed to allocate memory for requested buffer', CUBLAS_STATUS_ALLOC_FAILED, or bad_alloc, not just 'out of memory'. The background-removal and transparency-fixer fallback-to-lighter-model paths only matched the literal 'out of memory', so the fallback was dead code and transparency-fixer (default birefnet-hr-matting) always failed with an allocation error. Add isMemoryAllocError() and use it in both checks.

* fix(ai): use bundled PaddleOCR models so OCR runs offline

ocr.py passed no model dirs to PaddleOCR, so PaddleX resolved models from ~/.paddlex and downloaded them from HuggingFace at runtime (slow first use, broken air-gapped), ignoring the models the OCR bundle ships in MODELS_PATH; it also pulled doc-orientation/unwarping models that are not bundled. Pin detection, recognition and textline models to the bundled dirs in MODELS_PATH (per language) and disable use_doc_orientation_classify / use_doc_unwarping, with per-component fallback when a model is absent. Verified: OCR runs with zero HuggingFace requests.

* fix(docker): add CAP_KILL so container shutdown is graceful

cap_drop: ALL without re-adding KILL meant tini (PID 1, root) could not forward SIGTERM to the gosu-dropped snapotter process (root minus CAP_KILL cannot signal a different UID). docker stop logged '[FATAL tini] forwarding signal: Operation not permitted', never delivered the signal, and fell back to SIGKILL after the 10s timeout. Add KILL to cap_add in both compose files. Verified: docker stop completes in 0s with SIGTERM delivered (exit 143) and no FATAL tini.

* fix(ai): serialize bundle installs against AI jobs to prevent sidecar segfault

A feature bundle install rewrites the shared Python venv (pip + copytree of site-packages/*.so) as a background subprocess, with no coordination against AI tool jobs that dlopen native libs (torch / onnxruntime CUDA) from the same venv; a job loading a shared object while it is overwritten segfaults the sidecar. Add a process-wide async mutex (venv-lock.ts): bridge.run() acquires it before every AI script and the install route holds it across the installer subprocess. Both run in the same Node process so a module-level lock suffices. Verified: concurrent install + AI job produces zero segfaults and the job serializes behind the install.

* fix(ai): make the venv lock read/write so concurrent AI jobs are not serialized

The first cut used an exclusive mutex, which (a) deferred the dispatcher spawn by a microtask and broke unit tests that synchronously drive the mocked spawn, and (b) serialized AI jobs against each other, removing the dispatcher's by-id request multiplexing. Make it a writer-preferring read/write lock: AI jobs are shared readers (with a synchronous fast path so spawn still happens in-tick) and a bundle install is the exclusive writer. Verified: all 764 AI unit tests pass.

* fix(ai): degrade OCR to Tesseract on CPU-only hosts instead of segfaulting

The amd64 AI bundle ships paddlepaddle-gpu, whose native libs dlopen
libcuda.so.1 at import and segfault on a host without a GPU (libcuda is the
driver lib, injected only by nvidia-container-toolkit on GPU hosts). The
segfault crashed the shared long-lived AI dispatcher and, after a few attempts,
tripped the bridge crash-recovery permanent-disable, wedging all AI until a
container restart. The standalone ocr tool defaults to quality=balanced
(PaddleOCR), so it hit this on every CPU-only deployment; ocr-pdf already
hardcoded Tesseract and was unaffected.

ocr.py now gates the PaddleOCR tiers on gpu_available(): balanced/best
transparently fall back to fast (Tesseract, CPU-capable) when no usable GPU is
present, and run_paddleocr_v5/run_paddleocr_vl refuse before importing paddle so
the GPU build is never dlopen'd on CPU. GPU hosts are unchanged.

Verified on a CPU-only Windows/WSL2 box: ocr returns Tesseract text across
repeated runs with the dispatcher staying healthy (no wedge).
2026-06-23 18:39:51 +08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1a6c3a25ac chore(deps): bump node from 2d178f2 to e0d149b in /docker (#294)
Bumps node from `2d178f2` to `e0d149b`.

---
updated-dependencies:
- dependency-name: node
  dependency-version: 22-bookworm
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-22 21:42:21 +08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
b3a2996f83 chore(deps): bump mwader/static-ffmpeg from 8.1.1 to 8.1.2 in /docker (#293)
Bumps mwader/static-ffmpeg from 8.1.1 to 8.1.2.

---
updated-dependencies:
- dependency-name: mwader/static-ffmpeg
  dependency-version: 8.1.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-22 21:42:17 +08:00
SnapOtterandGitHub f74f648e47 fix(docker): copy patches/ before pnpm install so cold builds succeed (#300)
The builder and production stages run `pnpm install` without first COPYing the
patches/ directory, so package.json's patchedDependencies makes pnpm abort with
`ENOENT: ... patches/gray-matter@4.0.3.patch` on any build whose pnpm-store
layer cache is cold (e.g. a fresh CI runner or `docker compose build`). The
existing image only built because that layer happened to be cached.

Copy patches/ ahead of both `pnpm install` invocations. Verified: a clean
`docker build` of docker/Dockerfile now completes end to end.
2026-06-22 16:59:10 +08:00
SnapOtterandGitHub 1fec97111b fix(docker): make storage writable under non-root/foreign UIDs (TrueNAS, OpenShift) (#299)
The entrypoint only fixed volume permissions when started as root (chown +
gosu-drop to snapotter). Launched under a non-root/foreign UID (TrueNAS app
user, Kubernetes runAsUser, OpenShift) it did no permission setup, so /data and
/tmp/workspace -- owned by uid 999 from the image -- were not writable by the
running user. Uploads and processing then failed with a cryptic EACCES
("workspace folder is not writable") and AI bundle installs failed the same way,
while health checks still reported the container healthy.

- entrypoint: source new entrypoint-lib.sh; verify writability up front when
  non-root, and as snapotter after chown when root (catches root-squashed
  mounts), failing fast with an actionable message (which dir, uid/gid, how to
  fix) instead of a late, cryptic EACCES
- Dockerfile: own /data and /tmp/workspace as snapotter:0, group-writable with
  setgid, so an arbitrary UID with the root supplementary group (OpenShift /
  Kubernetes fsGroup) can write; keep /opt/venv world-readable for the AI venv
  bootstrap under arbitrary UIDs
- api: assert storage writability at boot (lib/storage-writable.ts), failing
  fast with the same guidance even when the entrypoint is bypassed
- docs: add a Storage permissions section (named volumes, bind mounts, TrueNAS,
  Kubernetes/OpenShift) and cross-link it from the security guide

Fixes #230
2026-06-22 16:58:59 +08:00
SnapOtterandGitHub c203267866 fix(docker): patch OS + pip image CVEs, document accepted Trivy residuals (#288)
Reduces the container-image CVE surface flagged by Trivy.

Genuinely fixed on every rebuild:
- apt-get upgrade in the production stage pulls Ubuntu security patches
  for base-image packages (libgnutls30t64 3.8.3-1.1ubuntu3.5 -> ubuntu3.6,
  libgcrypt20, liblzma5), closing ~15 OS-package CVEs.
- pip 25.1.1 -> 26.1.2 closes 4 pip CVEs (CVE-2025-8869, 2026-1703,
  2026-3219, 2026-6357).

Accepted via .trivyignore (canonical, reviewed):
- 6 newly surfaced pnpm 9.x build-tool CVEs (fixed only in pnpm 10.x, a
  major migration tracked separately; pnpm runs at install/start only).
- caire's bundled golang.org/x/image (esimov/caire v1.5.0 is latest and
  still pins x/image v0.18.0; no upstream fix).
- brace-expansion 2.x ReDoS (transitive of glob; patched 5.0.6 already
  present; not reachable from user input).

Already resolved in the current tree (clear on next scan): picomatch
4.0.4 (override), ip-address removed.

Verification note: the Trivy job in release.yml depends on the
intentionally gated-off docker build/publish job, so these cannot be
re-scanned in CI without enabling image publishing. The image is not
currently shipped.
2026-06-21 23:23:03 +08:00
SnapOtterandGitHub 19d9ed181a fix(audio): low-samplerate ogg encode + post-2.0 QA hygiene
Quality-VBR ogg (libvorbis -q:a) fixes 8 kHz 'encoder setup failed' in both ogg paths; drop empty COOKIE_SECRET ENV (app auto-generates); emit real bundle extractedSize; fix stale image-pad/compress-pdf QA specs.
2026-06-20 10:41:50 +08:00
SnapOtterandGitHub d8cf979d4b fix: resolve 18 QA-discovered bugs across tools, previews, and the AI pipeline (#242)
Exhaustive QA sweep of all 157 tools. Fixes: CSP blob media, csv-excel ExcelJS interop, ocr-pdf segfault, chart-maker upload, non-PDF doc preview, RAW decode, merge-tool multi-file path, html-to-image chromium, ogv/wma/amr/ac3 preview fallbacks, meme/gif/stabilize codecs, nav+home a11y. Plus orphan-format and test-debt cleanup, the AI bundle build script, and a reusable Playwright QA harness under tests/qa/.
2026-06-15 22:26:24 +08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
771a884ac5 chore(deps): bump debian from ed4fcc4 to 49ba348 in /docker (#235)
Bumps debian from `ed4fcc4` to `49ba348`.

---
updated-dependencies:
- dependency-name: debian
  dependency-version: bookworm
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-15 15:02:14 +08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
eedecbb67e chore(deps): bump mwader/static-ffmpeg from 8.0 to 8.1.1 in /docker (#234)
Bumps mwader/static-ffmpeg from 8.0 to 8.1.1.

---
updated-dependencies:
- dependency-name: mwader/static-ffmpeg
  dependency-version: 8.1.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-15 15:02:11 +08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
21ed984c92 chore(deps): bump node from 1031993 to 2d178f2 in /docker (#233)
Bumps node from `1031993` to `2d178f2`.

---
updated-dependencies:
- dependency-name: node
  dependency-version: 22-bookworm
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-15 15:02:03 +08:00
SnapOtter 39543d6512 fix(docker): make venv bootstrap upgrade-aware with pip-freeze stamp
On 1.x-to-2.0 upgrade the /data/ai/venv already existed with stale
packages, so the entrypoint skipped the cp from /opt/venv and 2.0
Python tools failed with ImportError (#85).

Dockerfile now writes a SHA-256 of `pip freeze` to /opt/venv/.venv-version
at build time. The entrypoint compares that stamp against the copy on the
volume; on mismatch it does a clean nuke + re-copy and resets
installed.json so the UI correctly shows bundles as needing reinstall.
Models in /data/ai/models survive the refresh, so bundle reinstall only
reruns pip (model downloads are idempotent).

Backward compatible: if the image has no stamp (old build), the
entrypoint falls through to the existing skip-if-exists behavior.
2026-06-13 10:31:22 +08:00
SnapOtter 6e1b9865f1 feat(tools): 2.0 phase 5 wave 5a - image gap-fill (11 tools) (#225) 2026-06-13 10:19:16 +08:00
SnapOtter fc7c1f850e feat(tools): 2.0 phase 5 wave 4 - office, ebooks, data, archives (14 tools) (#224) 2026-06-13 10:19:11 +08:00
SnapOtter 2f39e38162 feat(tools): 2.0 phase 5 wave 2 - pdf depth (21 tools) (#220) 2026-06-13 10:18:55 +08:00
SnapOtter d647d8ed19 feat(modality)!: SnapOtter 2.0 phase 3 modality framework: media/doc engines, pool routing, display modes (#218) 2026-06-13 10:18:39 +08:00
SnapOtter c451b939c7 feat(jobs)!: SnapOtter 2.0 phase 2 job spine: async queues, worker pools, object storage, admin dashboard (#217) 2026-06-13 10:17:13 +08:00
SnapOtterandGitHub 1c724d5d21 feat(db)!: SnapOtter 2.0 phase 1 foundation: postgres, migrator, compose stack (#216)
* feat(infra): add dev compose stack with postgres and redis

* fix(infra): comment dev env defaults until wired; harden dev compose restart and start_period

* chore(deps): add pg driver and testcontainers for postgres migration

* feat(db): translate schema to drizzle pg-core (timestamptz, boolean, pgEnum, jsonb)

Schema translation (apps/api/src/db/schema.ts):
- sqlite-core -> pg-core, all 10 tables preserved 1:1
- integer(mode:'timestamp') -> timestamp({ withTimezone: true })
- integer(mode:'boolean') -> boolean
- jobs.status text enum -> pgEnum('job_status') with same 4 values
- 7 columns changed from text to jsonb: jobs.inputFiles, jobs.settings,
  pipelines.steps, apiKeys.permissions, roles.permissions,
  auditLog.details, userFiles.toolChain
- settings.value stays text, jobs.error stays text, jobs.progress stays real

jsonb call-site sweep (removed JSON.stringify on writes, JSON.parse on reads):
- apps/api/src/routes/roles.ts: permissions read/write (3 sites)
- apps/api/src/routes/api-keys.ts: permissions write + read (2 sites)
- apps/api/src/routes/audit-log.ts: details read (1 site)
- apps/api/src/routes/pipeline.ts: steps write + read (2 sites)
- apps/api/src/routes/progress.ts: inputFiles write (2 sites)
- apps/api/src/routes/tool-factory.ts: toolChain read + write (2 sites)
- apps/api/src/routes/user-files.ts: toolChain read + write (4 sites)
- apps/api/src/permissions.ts: roles.permissions read (1 site)
- apps/api/src/lib/audit.ts: details write (1 site)
- apps/api/src/plugins/auth.ts: apiKeys.permissions read (1 site)

* refactor(db): type jsonb columns via $type and note raw CTE conversion requirements

* feat(db): archive sqlite migrations and generate postgres baseline

* chore(db): dockerignore legacy migrations, add archive breadcrumb, fix trailing newline

* feat(db): pg pool connection, advisory-locked boot migrations, DATABASE_URL config

* fix(db): friendly fatal on unreachable postgres, idempotent closeDb, lock-key convention note

* refactor(db): async drizzle calls in plugins, lib, permissions

* fix(api): analytics never throws, typed permission guard, single-query session invalidation

* refactor(db): async drizzle calls across all routes and bootstrap

Convert every route file and index.ts from sync SQLite drizzle
patterns to async node-postgres drizzle:

- .all() removed (bare await on select)
- .get() converted to destructured [row] = await ...
- .run() removed (bare await on insert/update/delete)
- .changes replaced with .rowCount (null-guarded) in progress.ts
- sqlite import removed from user-files.ts; raw CTEs converted to
  await db.execute(sql`...`) with postgres-dialect recursive CTEs
- ChainRow types updated: tool_chain is parsed jsonb (string[] | null),
  created_at is Date (timestamptz) with no * 1000 conversion
- All requirePermission() guard calls awaited (security: unawaited
  async guard returns truthy Promise, bypassing permission check)
- All hasEffectivePermission() and getPermissions() calls awaited
- All auditLog() calls awaited (preserves write-before-response order)
- trackEvent() and captureException() left un-awaited (fire-and-forget
  by design, guaranteed never-throw)
- ensureAnonymousUser(), startCleanupCron(), recoverStaleJobs() awaited
  in bootstrap sequence
- ensureInstanceId() and ensureDefaultSettings() made async

Files converted: 14 (index.ts + 12 route files + tools/index.ts)

* fix(db): await async checkStorageQuota in user-files upload/save routes

* fix(db): await checkStorageQuota in save-result route (missed second call site)

* feat(db): sqlite-to-postgres migrator with CLI and first-boot import

* fix(db): migrator error context, honest force semantics, boot-hook fatal, null-variance tests

* test: run suite against per-file postgres databases via testcontainers

- Add tests/global-setup.ts: spins up a Postgres testcontainer,
  creates a migrated template database once per vitest run.
- Rewrite tests/setup/per-fork-env.ts: each test file (forks pool)
  clones the template into its own database via CREATE DATABASE ...
  TEMPLATE, preserving the same per-file isolation granularity.
- Update vitest.config.ts: add globalSetup, pg alias, update comment.
- Fix tests/integration/test-server.ts: remove DB_PATH mkdir, async
  runMigrations, async db operations, remove SQLite WAL checkpoint.
- Fix 21 unit test db/index mocks: add pool and closeDb exports.
- Fix 8 unit test files: add async/await for now-async permission,
  audit, and analytics functions.
- Fix 18 integration test files: convert sync .run()/.all()/.get()
  to async drizzle patterns, add async to callbacks.
- Production change: apps/api/src/routes/teams.ts: cast COUNT(*)
  to ::int so Postgres returns a number instead of bigint string.

* fix(db): seed built-in roles, reject NUL bytes, cast COUNT, serialize job persists

- Seed built-in roles (admin, editor, user) at boot via ensureBuiltinRoles()
  with onConflictDoNothing, restoring data that legacy SQLite migration 0007
  provided via INSERT statements (the pg baseline is DDL-only).
- Reject NUL bytes in login credentials with 401 (postgres rejects \x00 in
  text columns; valid usernames never contain NUL, matching 1.x behavior).
- Cast COUNT(*)::int in user-files, audit-log, and roles listing queries so
  postgres returns a JS number instead of bigint-as-string.
- Serialize fire-and-forget job progress DB writes per jobId so the final
  "completed" status is never overwritten by a late-arriving "processing"
  write (race condition exposed by async postgres round-trips).

* test: fix teams race, seed roles in test server, poll for job status

- Add missing await to resetTeams() in teams PUT beforeEach (the async
  delete raced with the subsequent insert under postgres).
- Call ensureBuiltinRoles() in test server bootstrap so integration tests
  have the same built-in roles as production.
- Replace fixed 100ms flushPersist delay with a polling helper that waits
  for terminal job status, eliminating timing-dependent failures caused by
  postgres network round-trip latency.

* test: make heic temp-file cleanup assertion resilient to concurrent workers

Use a set-based diff instead of raw file count when checking that
decodeHeic cleans up temp files. Other concurrent test workers can
create heic-in-*/heic-out-* files in the shared tmpdir, inflating the
"after" count and causing spurious failures under full-suite load.

* fix(db): align builtin-role seed to post-0010 legacy state; test polish

* feat(docker): three-container compose (app, postgres, redis) with boot wait and migrations

* fix(docker): set TEST_DATABASE_URL so containerized tests skip testcontainers

* chore(docker): test compose project name, clearer 1.x upgrade comment, unref probe timer

* feat(enterprise): enforce D15 license boundary; move s3 storage into packages/enterprise

* fix(enterprise): restore lazy aws-sdk loading; community installs load no s3 code at boot

* fix(enterprise): boundary check catches dynamic imports; document getS3 concurrency

* feat(db)!: SnapOtter 2.0 phase 1 foundation: postgres, migrator, compose stack

BREAKING CHANGE: SQLite is no longer the runtime database. Deployments now
require Postgres (and Redis, used from phase 2). Existing installs migrate
with SQLITE_MIGRATE_PATH or 'pnpm --filter @snapotter/api migrate:sqlite'.

* fix(ci): postgres service + fresh e2e database per run; ignore unfixable torch CVE-2025-3000
2026-06-13 10:15:23 +08:00
SnapOtter d2fff1f87d fix(security): bump Go 1.25 and Pillow 12.2.0 for new Trivy CVEs
- Go 1.24 -> 1.25: fixes CVE-2026-25679 (HIGH: net/url IPv6 parsing)
- Pillow 12.1.1 -> 12.2.0: fixes CVE-2026-40192 (HIGH: decompression bomb DoS)
2026-06-08 17:32:34 +08:00
SnapOtter 0c58db2417 fix(security): update Go 1.24 and Pillow 12.1.1 for Trivy CVEs
- Bump Go from 1.23 to 1.24 to fix CVE-2025-68121 (CRITICAL:
  crypto/tls certificate validation) and CVE-2025-61726 (HIGH:
  net/url memory exhaustion) in the caire binary.
- Bump Pillow from 11.1.0 to 12.1.1 to fix CVE-2026-25990 (HIGH:
  out-of-bounds write via crafted image).
2026-06-08 17:06:07 +08:00
SnapOtter e0eb2e2075 fix(ci): add curl retry for libheif download and update release notes
- Add --retry 3 --retry-delay 5 to libheif curl in Dockerfile to
  handle transient GitHub CDN 504 errors on arm64 CI runners.
- Update release notes: use @ mentions, remove double-dash separators.
2026-06-08 16:17:28 +08:00
SnapOtter 9a1d3d25f4 fix(docker): resolve 4 release-blocking issues from validation
- fix(rate-limit): treat RATE_LIMIT_PER_MIN=0 as unlimited (50k/min)
  instead of blocking all requests. @fastify/rate-limit interprets
  max:0 as "allow zero requests," breaking fresh container startups.

- fix(docker): add libgles2 for MediaPipe face detection tools.
  blur-faces, red-eye-removal, enhance-faces, and passport-photo
  failed with "libGLESv2.so.2 not found" on all headless containers.

- fix(docker/arm64): remove conflicting system libheif1 to avoid
  ABI symbol mismatch with our custom libheif 1.21.2 build.
  heif-convert failed with "undefined symbol: heif_get_plugin_directories."

- fix(docker/arm64): pre-install wheel+setuptools in base Python venv
  so basicsr can build from source on arm64 (no pre-built wheel).
  This unblocks upscale-enhance and photo-restoration bundles.
2026-06-08 14:07:55 +08:00
SnapOtter ace41168bc fix(security): harden API against pentest findings
- Default TRUST_PROXY=false to prevent XFF rate limit bypass (PT-01)
- Return 400 instead of 500 on malformed JSON input (PT-03)
- Default MAX_PIPELINE_STEPS=20 to prevent DoS (PT-04)
- Validate clientJobId length (max 128) across all routes (PT-06)
- Add security headers to all reply.hijack() streaming responses (PT-07)
- Sanitize usernames in audit log to prevent stored XSS (PT-08)
- Block TRACE method with 405 response (PT-10)
- Add 429 RateLimited response to OpenAPI spec (PT-12)
- Default MAX_SVG_SIZE_MB=50 to limit SVGZ decompression (PT-13)
- Pin Dockerfile base images by digest
- Sanitize OIDC IdP error and sub claim in audit log
- Sync Docker compose/Dockerfile defaults with env.ts
2026-06-07 21:54:27 +08:00
SnapOtter 4d7f67d2af feat: add playwright chromium to docker image for html-to-image 2026-06-06 21:45:39 +08:00
SnapOtterandGitHub c365fde599 fix: build libheif 1.21.2 from source for iPhone HEIC support (#183) (#199)
iPhone 15 Pro (iOS 18+) HEIC files include HDR gain maps as auxiliary
image references. Distro-packaged libheif (1.15-1.17) rejects these
with "Too many auxiliary image references". Build libheif v1.21.2 from
source in a new Dockerfile stage to fix decoding.

- Add libheif-builder stage with platform-matched bases (debian:bookworm
  for arm64, ubuntu:24.04 for amd64) to avoid shared-library ABI
  mismatches
- Replace libheif-examples distro package with source-built binaries
- Update Dockerfile.test with same libheif source build
- No application code changes needed (heic-converter.ts CLI interface
  is stable across versions)

Closes #183
2026-06-05 17:05:01 +08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
e4608991ef chore(deps): bump nvidia/cuda in /docker (#185)
Bumps nvidia/cuda from 12.6.3-cudnn-runtime-ubuntu24.04 to 12.9.2-cudnn-runtime-ubuntu24.04.

---
updated-dependencies:
- dependency-name: nvidia/cuda
  dependency-version: 12.9.2-cudnn-runtime-ubuntu24.04
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-04 22:16:35 +08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
7af141be02 chore(deps): bump node from e3ca095 to 1031993 in /docker (#184)
Bumps node from `e3ca095` to `1031993`.

---
updated-dependencies:
- dependency-name: node
  dependency-version: 22-bookworm
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-04 21:48:35 +08:00
SnapOtterandGitHub ec6ff3d8a8 chore: harden OpenSSF Scorecard from 4.3 to ~7.0 (#142)
Token-Permissions (0 -> 10): Set permissions: {} at workflow top level
across all 7 workflows, moved write scopes to per-job minimum.

SAST (0 -> 10): Added CodeQL workflow for JavaScript/TypeScript and
Python analysis on push, PR, and weekly schedule.

Vulnerabilities (0 -> ~8): Added 13 pnpm overrides to patch transitive
dependency vulnerabilities (38 -> 2 remaining, both in dev-only tools).

Pinned-Dependencies (5 -> 8-9): Pinned all Docker FROM images to SHA
digests, pinned pip-audit version in CI, pinned pip version in
Dockerfile.
2026-05-18 15:57:17 +08:00
SnapOtter f86ef124c2 fix: grant admin role to anonymous user and add DEFAULT_TOOL_VIEW env var
When AUTH_ENABLED=false, the anonymous user was assigned the "user" role
which lacks settings:write permission, making all settings saves return
403. Since no admin exists when auth is disabled, settings were
permanently read-only. Promote the anonymous user to "admin" so the
single user has full control of the instance.

Also adds DEFAULT_TOOL_VIEW env var (sidebar|fullscreen) following the
existing DEFAULT_THEME pattern, seeded via ensureDefaultSettings() on
first boot.

Closes #135
2026-05-16 11:37:28 +08:00
SnapOtter b449bd1a56 fix(security): correct Docker rate limit defaults
- LOGIN_ATTEMPT_LIMIT: 500 -> 30 in Dockerfile (matching code default)
- RATE_LIMIT_PER_MIN=0 now means no global limit (50K ceiling) instead
  of 1 req/min, preserving backward compatibility with existing deploys
  while keeping per-route limits (login brute-force) active
2026-05-14 23:02:25 +08:00
SnapOtter ca2ef5b3f4 feat: add OIDC/SSO authentication (#3)
Add OpenID Connect (OIDC) authentication alongside existing
username/password login. Users can log in via any standards-compliant
OIDC provider (Keycloak, Authentik, Authelia, Google, Azure AD, Okta)
while preserving full backward compatibility.

- OIDC Fastify plugin with lazy discovery, PKCE, cookie-based sessions
- Login page OIDC button, auth hook updates, settings dialog badges
- 28 integration tests, OIDC setup guide with provider examples
- Fix pre-existing test failures (content-aware-crop, watermark, SVGZ)
- WAL checkpoint fix for SQLite test stability

Closes #3

# Conflicts:
#	apps/api/src/lib/env.ts
#	apps/api/src/routes/tools/watermark-image.ts
#	pnpm-lock.yaml
#	tests/integration/color-palette.test.ts
#	tests/integration/compare.test.ts
#	tests/integration/watermark-image.test.ts
2026-05-14 22:31:26 +08:00
SnapOtter 4e64ee2779 fix(security): comprehensive security audit and hardening
Auth: login rate limit 30/min (was 500), global rate limit 1000/min (was
unlimited), password/username max lengths on all Zod schemas, session
invalidation on role change, API key legacy scan bounded to 100 keys.

SVG: hardened regex sanitizer with CDATA stripping, XML entity decoding,
set/animate/iframe/embed blocking, comprehensive data: URI blocking,
use element external href blocking. 11 attack payload fixtures added.

SSRF: fixed DNS rebinding TOCTOU by pinning resolved IPs via custom
HTTP/HTTPS agents. Added 6to4 and NAT64 to blocked IPv6 ranges.

Docker: capability dropping (cap_drop ALL + minimal cap_add), resource
limits (4g/8g mem, 512/1024 pids), healthcheck timeout, password
removed from startup banner, default password warning comments.

Network: CSP and HSTS applied in all environments (not just production),
stack traces removed from all error responses, internal paths stripped
from error details, per-route rate limits on uploads (60/min) and URL
fetches (200/hour).

Files: exclusive temp file creation (O_EXCL), disk space circuit
breaker, per-user storage quotas, settings payload 64KB size guard.

Python sidecar: script name allowlist in dispatcher, minimal environment
for subprocess spawns.

Dependencies: fixed 6 production CVEs (drizzle-orm, fastify, fast-uri,
@fastify/static, next, archiver/lodash). Pinned all GitHub Actions to
SHA hashes.

114 security tests added. Full OWASP Top 10 penetration test matrix
verified against production Docker container (30/30 pass after
hardening).
2026-05-13 21:33:50 +08:00
SnapOtter 71496db6c5 feat(oidc): add OIDC environment variables to Docker configuration 2026-05-13 19:07:33 +08:00
SnapOtter 7f131d99a6 fix: QA sweep -- SSE crash, memory leaks, HEIC Docker decode, TGA detection, lint cleanup
- Fix SSE write-after-end crash in progress.ts (remove callback before ending stream)
- Fix blob URL memory leaks: revoke processedPreviewUrl and old HEIC preview URLs
- Add AbortController to batch fetch in use-tool-processor and use-pipeline-processor
- Fix TGA format misidentified as CUR (extension overrides magic bytes)
- Add libheif-plugin-libde265 to Docker for HEIC/HEIF decode support
- Remove unused imports and state (AppLayout, setSampledColor, useEffect)
- Fix non-null assertions in meme-text-renderer and meme-generator
- Fix confusing void type in meme-templates
- Remove unnecessary useEffect deps in adjustments-panel
- Fix Playwright strict mode violations in 5 E2E tests
2026-05-09 13:55:00 +08:00
SnapOtter aa773addb4 merge: resolve conflict with main branch in tool-registry.tsx 2026-05-08 18:55:28 +08:00
SnapOtter f9e0e90897 fix: include apps/api/static in Docker image for meme templates and fonts 2026-05-08 16:59:00 +08:00
SnapOtter 136a4a0752 Merge branch 'feat/format-support-expansion'
# Conflicts:
#	README.md
#	apps/web/src/components/layout/app-layout.tsx
2026-05-08 16:21:11 +08:00
SnapOtter 6f3792a9c7 feat: add ghostscript and openjpeg to Docker, fix ImageMagick policy for EPS 2026-05-07 23:54:25 +08:00
SnapOtter 16af9c573a feat: remove app name and logo customization feature
Users can no longer customize the app name or logo. The branding API
endpoints, permission, frontend UI, env vars (APP_NAME, MAX_LOGO_SIZE_KB),
and all related tests are removed. Includes a migration to clean up
branding data from existing databases.
2026-05-07 19:41:30 +08:00
SnapOtter 4486cf926f fix: revert Pillow/rembg upgrades that break dependency tree
Pillow 12.x conflicts with pinned numpy 1.26.4, rembg, realesrgan,
and mediapipe. Revert to working 11.1.0 pins and ignore the CVEs
in pip-audit instead — they require a coordinated major version
upgrade across the entire ML stack (Pillow, numpy, torch, basicsr).

Ignored CVEs:
- CVE-2024-27763 (basicsr, no fix available)
- CVE-2026-40086 (rembg, fix needs Pillow 12)
- CVE-2026-25990 (Pillow, fix is 12.1.1)
- CVE-2026-40192 (Pillow, fix is 12.2.0)
2026-04-27 01:23:25 +08:00
SnapOtter b926e5d1be fix: CI failures — QR test timeout and Python dependency CVEs
- Increase QR generate max-size test timeout to 120s (10000x10000
  PNG generation exceeds 30s default on CI runners)
- Update Pillow 11.1.0 → >=12.2.0 (CVE-2026-25990, CVE-2026-40192)
- Update rembg 2.0.62 → >=2.0.75 (CVE-2026-40086)
- Update opencv-python-headless to flexible range >=4.10,<4.12
- Ignore CVE-2024-27763 in pip-audit (basicsr transitive dep from
  realesrgan, no fix available upstream)
- Align requirements-gpu.txt and Dockerfile with same versions
2026-04-27 01:19:25 +08:00
SnapOtter 86db131198 fix: JXL decode fallback and Playwright remote container support
Add djxl (libjxl-tools) as primary JXL decoder with ImageMagick
fallback — fixes JXL format failures on Ubuntu where stock ImageMagick
lacks a JXL delegate. Also make Playwright Docker config respect
BASE_URL env var for testing against remote containers.
2026-04-26 02:53:02 +08:00
SnapOtter 291e4092cd fix: make libmagickcore-extra conditional for multi-arch Docker builds
arm64 uses Debian bookworm (libmagickcore-6.q16-6-extra) while amd64
uses Ubuntu 24.04 (libmagickcore-6.q16-7-extra). Probing both variants
prevents build failures on the arm64 platform.
2026-04-25 07:32:02 +08:00
SnapOtter bd84728588 docs: comprehensive API sync and documentation audit
- Rewrite OpenAPI spec to match actual code (988 lines changed):
  - Fix ToolResponse schema (add previewUrl, savedFileId)
  - Fix Error schema shape ({error, details} not {statusCode, error, message})
  - Fix POST /api/auth/register URL (was /api/auth/users)
  - Fix login/session responses (7 missing user fields + expiresAt)
  - Fix 8 endpoints returning 204 → 200 with {ok: true}
  - Fix pipeline execute field name (steps → pipeline)
  - Fix API keys response key (keys → apiKeys)
  - Fix settings response wrapper, teams UUID type
  - Rewrite 7 major tool response schemas (info, barcode-read,
    find-duplicates, compare, remove-background, upscale, ocr, blur-faces)
  - Fix files/save-result (JSON → multipart), files/upload (201 + array)
  - Fix SSE progress schema (integers not arrays)
  - Add 422/501 error responses to AI and processing tools
  - Fix settings required → optional on 29 tool endpoints
  - Add 5 missing color adjustment fields to alias endpoints
- Rewrite rest.md tool parameter descriptions (12 tools fixed)
- Add Tool Sub-Routes section to rest.md (11 endpoints)
- Fix file library, settings, pipeline, auth docs in rest.md
- Fix API key hashing description (SHA-256 → scrypt)
- Fix "GitHub Pages" → "Cloudflare Pages" in architecture + deployment docs
- Fix tool count "45+" → "47" across all doc surfaces
- Fix branding endpoint paths in rest.md (/branding/logo → /settings/logo)
2026-04-25 07:17:45 +08:00
SnapOtter 8633dba431 fix: sync API docs, register content-aware-resize, normalize tool counts
- Fix 23 OpenAPI schema discrepancies across 16+ tools (wrong ranges,
  missing fields, incorrect schemas for gif-tools/collage/ocr)
- Add content-aware-resize to canonical TOOLS array and landing BentoGrid
- Normalize tool count to 47 across README, docs, landing, i18n, OpenAPI
- Remove dead "automation" ToolCategory variant
- Add BMP and JPEG XL format decoding via ImageMagick
- Add libopenexr-dev to Docker runtime image
- Update e2e test selectors for current pipeline builder UI
2026-04-24 23:27:08 +08:00
SnapOtter 7f62bc32db test: expand coverage to 3,382 tests across all layers
- Unit: 1,353 tests (42 files) — +256 new tests covering AI bridge
  modules, image-engine sharpen/optimize-for-web, Zustand stores, and
  icon-map validation
- Integration: 1,640 tests (57 files) — +826 new tests across all
  tool routes, pipeline/progress/batch infrastructure, user-files,
  edit-metadata, and a 321-test cross-format matrix
- E2E-Docker: 389 passing (20 spec files) — 6 new spec files for
  batch processing, format conversion, layout, optimization,
  watermark/overlay, and pipeline chains. Tests verified against fresh
  Docker container with all 6 AI bundles installed.

Bug fixes discovered during testing:
- fix(compress): SVG/BMP/exotic formats crashed Sharp encoder — added
  format-safety fallback to PNG
- fix(rate-limit): increase default login attempt limit from 10 to 500
  per minute — previous value caused false test failures and is too
  restrictive for a self-hosted app
- fix(auth.setup): wait for consent button visibility before clicking
  to prevent flaky E2E-Docker auth setup
2026-04-24 22:43:14 +08:00
SnapOtter 0309e0f680 chore: deploy to Cloudflare Pages and update branding
- Add Cloudflare Pages deployment for landing page (snapotter.com) and
  docs (docs.snapotter.com)
- Create deploy-landing.yml and update deploy-docs.yml workflows
- Update CI to ignore apps/landing/** paths
- Fix logo transparency (remove white background) across all apps
- Recreate social-preview.png with SnapOtter branding
- Update all docs URLs from GitHub Pages to docs.snapotter.com
- Update VitePress config: light theme default, fix llms.txt paths
- Add .vitepress/cache/ and .env.* to gitignore
2026-04-24 18:06:29 +08:00
AshimandGitHub 4c9dc6e38e fix: Docker hardening, security, and deployment readiness for V1 (#82)
Phase 1 — Docker Artifact Optimization:
- Replace broad `COPY . .` with targeted frontend source copies (API/Python
  changes no longer bust the frontend build cache)
- Replace build-essential with gcc/g++ (leaner runtime)
- Fix LOG_LEVEL=debug → info for production
- Harden .dockerignore (exclude worktrees, IDE, CI, test artifacts)

Phase 2 — State & Persistence:
- Add PUID/PGID support in entrypoint.sh for bind mount compatibility
- Guard against PUID=0/PGID=0 to prevent accidental root execution
- Evict conflicting system users (e.g. node:1000) before UID remap

Phase 3 — Security:
- Always register @fastify/rate-limit so login brute-force protection
  works even when global rate limit is disabled (RATE_LIMIT_PER_MIN=0)
- Add trustProxy support (TRUST_PROXY env var, default true) so rate
  limiting and audit logs use real client IPs behind reverse proxies
- Strip stack traces from 500 error responses in production
- Fix FSTDEP022 deprecation: maxParamLength → routerOptions
- Add multi-file guard on single-file tool endpoint with clear error
  message pointing to the /batch endpoint

Phase 4 — Graceful Degradation:
- Add consolidated hardware detection startup banner (GPU, rate limit,
  upload limit, proxy status)
- Add ConnectionMonitor component with health polling and reconnecting
  overlay that auto-dismisses when the server comes back

Phase 5 — Deployment Docs:
- Rewrite deployment.md with copy-paste CPU and GPU compose templates
- Add hardware requirements table (minimum, recommended, heavy workloads)
- Add PUID/PGID bind mount documentation
- Add complete env var reference table
- Add reverse proxy guides for Nginx, Nginx Proxy Manager, Traefik,
  and Cloudflare Tunnels
2026-04-21 10:19:08 +08:00
ashim-hq 7daf2377f0 feat: unlimited by default — env vars default to 0 (unlimited)
Add 10 new configurable env vars (MAX_WORKER_THREADS, PROCESSING_TIMEOUT_S,
MAX_PIPELINE_STEPS, MAX_CANVAS_PIXELS, MAX_SVG_SIZE_MB, MAX_LOGO_SIZE_KB,
MAX_SPLIT_GRID, MAX_PDF_PAGES, SESSION_DURATION_HOURS, LOGIN_ATTEMPT_LIMIT).

Change existing defaults: MAX_UPLOAD_SIZE_MB=0, MAX_BATCH_SIZE=0,
MAX_MEGAPIXELS=0, RATE_LIMIT_PER_MIN=0, MAX_USERS=0, CONCURRENT_JOBS=0.

Zero means unlimited/auto-detect throughout the application.
2026-04-20 21:35:15 +08:00