Commit Graph
105 Commits
Author SHA1 Message Date
SnapOtterandGitHub f3342a1e57 fix: harden Docker image and async job responses
Harden Docker runtime packaging, preserve async job response semantics, fix Redis subscriber startup connections, clear lint warnings, and harden enterprise S3 object body handling.
2026-07-01 12:32:33 +08:00
SnapOtterandGitHub 37dc0098ba docs: sync API and documentation coverage (#379)
* docs: sync API and docs coverage

* ci: pin pandoc for sandboxed conversions
2026-06-29 22:35:05 +08:00
SnapOtter 3c37444bd1 chore(deps): merge cuda image digest bump (#373) 2026-06-29 16:01:30 +08:00
SnapOtter da026d9a13 chore(deps): merge node image digest bump (#372) 2026-06-29 16:01:25 +08:00
SnapOtter b9c7ae2bcb chore(deps): merge debian image digest bump (#371) 2026-06-29 16:01:21 +08:00
dependabot[bot]andGitHub 66447350ae chore(deps): bump node from e0d149b to c601a46 in /docker
Bumps node from `e0d149b` to `c601a46`.

---
updated-dependencies:
- dependency-name: node
  dependency-version: 22-bookworm
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-06-29 07:51:07 +00:00
SnapOtterandGitHub 084a9c9faa feat: all-in-one embedded single-container mode (#377)
Embedded Postgres 17 + Redis via s6-overlay when DATABASE_URL/REDIS_URL are unset; restores the one-command docker run for 2.0. EMBEDDED=0 disables; Compose stays the production path. Verified arm64 (14/14 lifecycle + Compose regression) and amd64 (build + embedded smoke).
2026-06-29 15:49:09 +08:00
dependabot[bot]andGitHub 2f35e012a9 chore(deps): bump nvidia/cuda in /docker
Bumps nvidia/cuda from 12.6.3-cudnn-runtime-ubuntu24.04 to 12.9.2-cudnn-runtime-ubuntu24.04.

---
updated-dependencies:
- dependency-name: nvidia/cuda
  dependency-version: 12.9.2-cudnn-runtime-ubuntu24.04
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-06-29 05:35:28 +00:00
dependabot[bot]andGitHub 4d4851ee42 chore(deps): bump debian from 49ba348 to 30482e8 in /docker
Bumps debian from `49ba348` to `30482e8`.

---
updated-dependencies:
- dependency-name: debian
  dependency-version: bookworm
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-06-29 05:34:59 +00:00
SnapOtterandGitHub 1c202c6ef0 feat(analytics): upload web source maps to Sentry + tie release to build (#369)
* feat(analytics): upload web source maps to Sentry + tie release to build

Web crash reports were unusable: the bundle ships minified with no source
maps uploaded, and every build reported as the frozen APP_VERSION, so a
Sentry error showed an unreadable stack under a single release.

- Add @sentry/vite-plugin: emit hidden source maps and upload them by debug
  id when SENTRY_AUTH_TOKEN is present (published Docker build only), then
  delete the maps so they never ship. No-op for dev and the source archive.
- Set the Sentry release from SENTRY_RELEASE / VITE_SENTRY_RELEASE (the Docker
  build passes the release version), falling back to APP_VERSION.
- Relax beforeSend so app bundle frames keep a host-stripped path (Sentry needs
  it to match the uploaded map) while the instance hostname, error message, and
  PII stay stripped. Filesystem paths still collapse to the basename.
- Wire the Dockerfile (sentry_auth_token build secret + SENTRY_RELEASE arg/env)
  and the release docker job.

* fix(analytics): point source map upload at the snapotter org (project node)
2026-06-29 10:47:30 +08:00
SnapOtterandGitHub 9819c5885e feat(analytics): inject Sentry DSN + PostHog key from build env (#367)
* fix(analytics): bake real Sentry DSN and lower trace sampling

The bake script emitted a placeholder Sentry DSN even in on mode, so every
published image initialized Sentry against a dead endpoint and no events ever
reached the project. Point it at the real snapotter project DSN.

Also drop tracesSampleRate from 1 to 0.1. It governs only performance
transactions (errors are always captured), so 100% fleet-wide tracing would
drain Sentry quota for no benefit.

* fix(analytics): point baked Sentry DSN at the snapotter org

* refactor(analytics): inject Sentry DSN + PostHog key from build env

#336 replaced the analytics creds with placeholders but never added a way to
put real values back at build time, so any image built from the repo since then
ships dead analytics (the live fleet only still reports because publishing is
paused and it runs a pre-placeholder image).

Restore the pipeline the clean way: bake-analytics.mjs reads SNAPOTTER_SENTRY_DSN
and SNAPOTTER_POSTHOG_KEY from the environment; the official image's CI supplies
them from repo secrets via build args. A build with neither stays disabled, so
building from source never phones home. Both values are public (they ship in the
browser bundle), so this is about not making source builds report, not secrecy.

Supersedes the hardcoded DSN: real creds are no longer committed to the repo.
2026-06-29 10:41:58 +08:00
SnapOtterandGitHub 8f4235d2c6 fix(enterprise): ship enterprise package in prod image + S3, analytics, tracing, queue fixes (#342)
* fix(enterprise): ship enterprise pkg in prod image, full license features, tracing key fallback

docker/Dockerfile: COPY packages/enterprise manifest+src into the production stage.
Without it, apps/api's workspace link to @snapotter/enterprise dangles and every
import() throws (silently caught), so all 19 enterprise features failed closed
(enterprise.active=false) regardless of a valid license.

scripts/generate-license.mjs: sync PLAN_FEATURES with packages/enterprise/src/license.ts
so a --plan enterprise license unlocks all 19 features (was 8) and team unlocks 8.

apps/api/src/tracing.ts: accept SNAPOTTER_LICENSE_KEY as a fallback to LICENSE_KEY so
distributed_tracing activates with the same key as the rest of the app.

* fix(docker): keep scripts/bake-analytics.mjs in build context

.dockerignore excluded the whole scripts/ dir (PR #82, V1 hardening), but
docker/Dockerfile later added 'COPY scripts/bake-analytics.mjs' for the analytics
bake step. A clean production image build therefore fails with
'scripts/bake-analytics.mjs: not found'. The published image build is gated off in
CI so this latent break went unnoticed. Exclude scripts/* but re-include the one
file the Dockerfile needs.

* fix: S3 upload stream, analytics bake reaches API, dedupe retention field, reconcile orphan jobs

storage-s3.ts: wrap the upload AsyncIterable in Readable.from() so @aws-sdk/lib-storage
accepts it. STORAGE_MODE=s3 file uploads failed with 'Body Data is unsupported format'
for every tool because a bare async generator is not a Readable.

docker/Dockerfile: COPY the builder-baked analytics baked.ts into the API runtime stage.
The API re-copied the committed (off) baked.ts from the build context, so the
SNAPOTTER_ANALYTICS build arg had no effect on the API -- and since the SPA reads
/api/v1/config/analytics, analytics was off everywhere regardless of the arg.

settings-dialog.tsx: remove the duplicate tempFileMaxAgeHours control under Data
Retention; it bound the same setting key as the File Management control with a different
default, so editing either silently overwrote the other.

apps/api/src/index.ts: reconcile orphaned job rows (empty tool_id, never enqueued to
BullMQ) at boot so they don't sit in processing/queued forever and inflate the per-user
concurrent-job count and the upgrade-check in-flight gate.

* fix(web): style the SSO login buttons (they referenced undefined theme tokens)

The OIDC/SAML 'Sign in with <provider>' buttons used bg-secondary /
text-secondary-foreground, which the web theme never defines (it has primary,
background, foreground, muted, border, card, primary-subtle). Those classes resolved
to nothing, so the buttons rendered as bare unstyled text on the login page.

Restyle: the optional (non-enforced) buttons become white-card outline buttons with a
key icon and an orange hover tint, secondary to the primary Login button; the
SSO-enforced buttons become solid primary with the icon.

* fix: gate S3 behind license, custom-role enterprise perms, wire retention UI, cleanup

S3 is a licensed feature, but shipping packages/enterprise in every image removed the
implicit gate, so STORAGE_MODE=s3 worked without a license. Enforce
isFeatureEnabled('s3_storage') at boot and fail fast if unlicensed.

Custom roles can now be granted security:manage / compliance:manage / webhooks:manage
(roles.ts ALL_PERMISSIONS + the Roles UI) so admins can build least-privilege
compliance/security roles instead of only the built-in admin role.

retentionSweep now reads the jobsRetentionDays / auditRetentionDays DB settings the
System Settings UI writes (env vars become the fallback default), mirroring how the
temp-file sweep reads tempFileMaxAgeHours. Previously those two UI controls were no-ops.

Cleanup: drop the never-set snapotter_storage_bytes gauge and the unused
MAX_WORKSPACE_SIZE_GB env var; emit tool_client_error to PostHog from the web
ErrorBoundary (client crashes were not reaching analytics); add the Python
OpenTelemetry packages so the innermost sidecar.<script> span exports; fix the stale
'only local storage' line in the docs; delete two e2e-analytics specs that tested the
removed consent UI.

* fix(env): restore MAX_WORKSPACE_SIZE_GB default

security-auth-hardening.test.ts asserts env.MAX_WORKSPACE_SIZE_GB defaults to 10, so
the var is an intentional (tested) default, not dead code. Removing it in the cleanup
commit broke that unit test. Keep the declaration.
2026-06-24 17:27:59 +08:00
SnapOtterandGitHub 5d36ac06d8 feat(analytics): build-time bake + telemetry depth (#336)
Bake PostHog + Sentry into the published Docker image (SNAPOTTER_ANALYTICS
build arg, codegen script). Delete entire consent system. Move event emission
to BullMQ worker. Add cross-tier identity stitching, Sentry performance
tracing on both tiers, frontend funnel events. Fix stateful regex bug.

86 files changed, 1593 insertions(+), 3747 deletions(-)
2026-06-24 11:05:39 +08:00
SnapOtterandGitHub 35e18d8b79 fix: GPU deployment robustness (6 fixes from end-to-end testing on an RTX 4070) (#334)
* fix(docker): pin CUDA base to 12.6 so the GPU image starts on R560+ drivers

The amd64 base nvidia/cuda:12.9.2-cudnn-runtime bakes a cuda>=12.9 driver gate enforced by nvidia-container-toolkit at container start, so the image fails to launch on common production drivers (e.g. 570.x / CUDA 12.8). The AI bundles are all cu126 wheels and the image installs libcublas-12-6, so 12.9 was misaligned with the workload. Pin to nvidia/cuda:12.6.3-cudnn-runtime-ubuntu24.04 to match the wheels and lower the driver floor to R560+.

* fix(ai): broaden OOM detection so the rembg lighter-model fallback fires

onnxruntime/CUDA allocation failures surface as 'Failed to allocate memory for requested buffer', CUBLAS_STATUS_ALLOC_FAILED, or bad_alloc, not just 'out of memory'. The background-removal and transparency-fixer fallback-to-lighter-model paths only matched the literal 'out of memory', so the fallback was dead code and transparency-fixer (default birefnet-hr-matting) always failed with an allocation error. Add isMemoryAllocError() and use it in both checks.

* fix(ai): use bundled PaddleOCR models so OCR runs offline

ocr.py passed no model dirs to PaddleOCR, so PaddleX resolved models from ~/.paddlex and downloaded them from HuggingFace at runtime (slow first use, broken air-gapped), ignoring the models the OCR bundle ships in MODELS_PATH; it also pulled doc-orientation/unwarping models that are not bundled. Pin detection, recognition and textline models to the bundled dirs in MODELS_PATH (per language) and disable use_doc_orientation_classify / use_doc_unwarping, with per-component fallback when a model is absent. Verified: OCR runs with zero HuggingFace requests.

* fix(docker): add CAP_KILL so container shutdown is graceful

cap_drop: ALL without re-adding KILL meant tini (PID 1, root) could not forward SIGTERM to the gosu-dropped snapotter process (root minus CAP_KILL cannot signal a different UID). docker stop logged '[FATAL tini] forwarding signal: Operation not permitted', never delivered the signal, and fell back to SIGKILL after the 10s timeout. Add KILL to cap_add in both compose files. Verified: docker stop completes in 0s with SIGTERM delivered (exit 143) and no FATAL tini.

* fix(ai): serialize bundle installs against AI jobs to prevent sidecar segfault

A feature bundle install rewrites the shared Python venv (pip + copytree of site-packages/*.so) as a background subprocess, with no coordination against AI tool jobs that dlopen native libs (torch / onnxruntime CUDA) from the same venv; a job loading a shared object while it is overwritten segfaults the sidecar. Add a process-wide async mutex (venv-lock.ts): bridge.run() acquires it before every AI script and the install route holds it across the installer subprocess. Both run in the same Node process so a module-level lock suffices. Verified: concurrent install + AI job produces zero segfaults and the job serializes behind the install.

* fix(ai): make the venv lock read/write so concurrent AI jobs are not serialized

The first cut used an exclusive mutex, which (a) deferred the dispatcher spawn by a microtask and broke unit tests that synchronously drive the mocked spawn, and (b) serialized AI jobs against each other, removing the dispatcher's by-id request multiplexing. Make it a writer-preferring read/write lock: AI jobs are shared readers (with a synchronous fast path so spawn still happens in-tick) and a bundle install is the exclusive writer. Verified: all 764 AI unit tests pass.

* fix(ai): degrade OCR to Tesseract on CPU-only hosts instead of segfaulting

The amd64 AI bundle ships paddlepaddle-gpu, whose native libs dlopen
libcuda.so.1 at import and segfault on a host without a GPU (libcuda is the
driver lib, injected only by nvidia-container-toolkit on GPU hosts). The
segfault crashed the shared long-lived AI dispatcher and, after a few attempts,
tripped the bridge crash-recovery permanent-disable, wedging all AI until a
container restart. The standalone ocr tool defaults to quality=balanced
(PaddleOCR), so it hit this on every CPU-only deployment; ocr-pdf already
hardcoded Tesseract and was unaffected.

ocr.py now gates the PaddleOCR tiers on gpu_available(): balanced/best
transparently fall back to fast (Tesseract, CPU-capable) when no usable GPU is
present, and run_paddleocr_v5/run_paddleocr_vl refuse before importing paddle so
the GPU build is never dlopen'd on CPU. GPU hosts are unchanged.

Verified on a CPU-only Windows/WSL2 box: ocr returns Tesseract text across
repeated runs with the dispatcher staying healthy (no wedge).
2026-06-23 18:39:51 +08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1a6c3a25ac chore(deps): bump node from 2d178f2 to e0d149b in /docker (#294)
Bumps node from `2d178f2` to `e0d149b`.

---
updated-dependencies:
- dependency-name: node
  dependency-version: 22-bookworm
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-22 21:42:21 +08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
b3a2996f83 chore(deps): bump mwader/static-ffmpeg from 8.1.1 to 8.1.2 in /docker (#293)
Bumps mwader/static-ffmpeg from 8.1.1 to 8.1.2.

---
updated-dependencies:
- dependency-name: mwader/static-ffmpeg
  dependency-version: 8.1.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-22 21:42:17 +08:00
SnapOtterandGitHub f74f648e47 fix(docker): copy patches/ before pnpm install so cold builds succeed (#300)
The builder and production stages run `pnpm install` without first COPYing the
patches/ directory, so package.json's patchedDependencies makes pnpm abort with
`ENOENT: ... patches/gray-matter@4.0.3.patch` on any build whose pnpm-store
layer cache is cold (e.g. a fresh CI runner or `docker compose build`). The
existing image only built because that layer happened to be cached.

Copy patches/ ahead of both `pnpm install` invocations. Verified: a clean
`docker build` of docker/Dockerfile now completes end to end.
2026-06-22 16:59:10 +08:00
SnapOtterandGitHub 1fec97111b fix(docker): make storage writable under non-root/foreign UIDs (TrueNAS, OpenShift) (#299)
The entrypoint only fixed volume permissions when started as root (chown +
gosu-drop to snapotter). Launched under a non-root/foreign UID (TrueNAS app
user, Kubernetes runAsUser, OpenShift) it did no permission setup, so /data and
/tmp/workspace -- owned by uid 999 from the image -- were not writable by the
running user. Uploads and processing then failed with a cryptic EACCES
("workspace folder is not writable") and AI bundle installs failed the same way,
while health checks still reported the container healthy.

- entrypoint: source new entrypoint-lib.sh; verify writability up front when
  non-root, and as snapotter after chown when root (catches root-squashed
  mounts), failing fast with an actionable message (which dir, uid/gid, how to
  fix) instead of a late, cryptic EACCES
- Dockerfile: own /data and /tmp/workspace as snapotter:0, group-writable with
  setgid, so an arbitrary UID with the root supplementary group (OpenShift /
  Kubernetes fsGroup) can write; keep /opt/venv world-readable for the AI venv
  bootstrap under arbitrary UIDs
- api: assert storage writability at boot (lib/storage-writable.ts), failing
  fast with the same guidance even when the entrypoint is bypassed
- docs: add a Storage permissions section (named volumes, bind mounts, TrueNAS,
  Kubernetes/OpenShift) and cross-link it from the security guide

Fixes #230
2026-06-22 16:58:59 +08:00
SnapOtterandGitHub c203267866 fix(docker): patch OS + pip image CVEs, document accepted Trivy residuals (#288)
Reduces the container-image CVE surface flagged by Trivy.

Genuinely fixed on every rebuild:
- apt-get upgrade in the production stage pulls Ubuntu security patches
  for base-image packages (libgnutls30t64 3.8.3-1.1ubuntu3.5 -> ubuntu3.6,
  libgcrypt20, liblzma5), closing ~15 OS-package CVEs.
- pip 25.1.1 -> 26.1.2 closes 4 pip CVEs (CVE-2025-8869, 2026-1703,
  2026-3219, 2026-6357).

Accepted via .trivyignore (canonical, reviewed):
- 6 newly surfaced pnpm 9.x build-tool CVEs (fixed only in pnpm 10.x, a
  major migration tracked separately; pnpm runs at install/start only).
- caire's bundled golang.org/x/image (esimov/caire v1.5.0 is latest and
  still pins x/image v0.18.0; no upstream fix).
- brace-expansion 2.x ReDoS (transitive of glob; patched 5.0.6 already
  present; not reachable from user input).

Already resolved in the current tree (clear on next scan): picomatch
4.0.4 (override), ip-address removed.

Verification note: the Trivy job in release.yml depends on the
intentionally gated-off docker build/publish job, so these cannot be
re-scanned in CI without enabling image publishing. The image is not
currently shipped.
2026-06-21 23:23:03 +08:00
SnapOtterandGitHub 19d9ed181a fix(audio): low-samplerate ogg encode + post-2.0 QA hygiene
Quality-VBR ogg (libvorbis -q:a) fixes 8 kHz 'encoder setup failed' in both ogg paths; drop empty COOKIE_SECRET ENV (app auto-generates); emit real bundle extractedSize; fix stale image-pad/compress-pdf QA specs.
2026-06-20 10:41:50 +08:00
SnapOtterandGitHub d8cf979d4b fix: resolve 18 QA-discovered bugs across tools, previews, and the AI pipeline (#242)
Exhaustive QA sweep of all 157 tools. Fixes: CSP blob media, csv-excel ExcelJS interop, ocr-pdf segfault, chart-maker upload, non-PDF doc preview, RAW decode, merge-tool multi-file path, html-to-image chromium, ogv/wma/amr/ac3 preview fallbacks, meme/gif/stabilize codecs, nav+home a11y. Plus orphan-format and test-debt cleanup, the AI bundle build script, and a reusable Playwright QA harness under tests/qa/.
2026-06-15 22:26:24 +08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
771a884ac5 chore(deps): bump debian from ed4fcc4 to 49ba348 in /docker (#235)
Bumps debian from `ed4fcc4` to `49ba348`.

---
updated-dependencies:
- dependency-name: debian
  dependency-version: bookworm
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-15 15:02:14 +08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
eedecbb67e chore(deps): bump mwader/static-ffmpeg from 8.0 to 8.1.1 in /docker (#234)
Bumps mwader/static-ffmpeg from 8.0 to 8.1.1.

---
updated-dependencies:
- dependency-name: mwader/static-ffmpeg
  dependency-version: 8.1.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-15 15:02:11 +08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
21ed984c92 chore(deps): bump node from 1031993 to 2d178f2 in /docker (#233)
Bumps node from `1031993` to `2d178f2`.

---
updated-dependencies:
- dependency-name: node
  dependency-version: 22-bookworm
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-15 15:02:03 +08:00
SnapOtter 39543d6512 fix(docker): make venv bootstrap upgrade-aware with pip-freeze stamp
On 1.x-to-2.0 upgrade the /data/ai/venv already existed with stale
packages, so the entrypoint skipped the cp from /opt/venv and 2.0
Python tools failed with ImportError (#85).

Dockerfile now writes a SHA-256 of `pip freeze` to /opt/venv/.venv-version
at build time. The entrypoint compares that stamp against the copy on the
volume; on mismatch it does a clean nuke + re-copy and resets
installed.json so the UI correctly shows bundles as needing reinstall.
Models in /data/ai/models survive the refresh, so bundle reinstall only
reruns pip (model downloads are idempotent).

Backward compatible: if the image has no stamp (old build), the
entrypoint falls through to the existing skip-if-exists behavior.
2026-06-13 10:31:22 +08:00
SnapOtter 6e1b9865f1 feat(tools): 2.0 phase 5 wave 5a - image gap-fill (11 tools) (#225) 2026-06-13 10:19:16 +08:00
SnapOtter fc7c1f850e feat(tools): 2.0 phase 5 wave 4 - office, ebooks, data, archives (14 tools) (#224) 2026-06-13 10:19:11 +08:00
SnapOtter 2f39e38162 feat(tools): 2.0 phase 5 wave 2 - pdf depth (21 tools) (#220) 2026-06-13 10:18:55 +08:00
SnapOtter d647d8ed19 feat(modality)!: SnapOtter 2.0 phase 3 modality framework: media/doc engines, pool routing, display modes (#218) 2026-06-13 10:18:39 +08:00
SnapOtter c451b939c7 feat(jobs)!: SnapOtter 2.0 phase 2 job spine: async queues, worker pools, object storage, admin dashboard (#217) 2026-06-13 10:17:13 +08:00
SnapOtterandGitHub 1c724d5d21 feat(db)!: SnapOtter 2.0 phase 1 foundation: postgres, migrator, compose stack (#216)
* feat(infra): add dev compose stack with postgres and redis

* fix(infra): comment dev env defaults until wired; harden dev compose restart and start_period

* chore(deps): add pg driver and testcontainers for postgres migration

* feat(db): translate schema to drizzle pg-core (timestamptz, boolean, pgEnum, jsonb)

Schema translation (apps/api/src/db/schema.ts):
- sqlite-core -> pg-core, all 10 tables preserved 1:1
- integer(mode:'timestamp') -> timestamp({ withTimezone: true })
- integer(mode:'boolean') -> boolean
- jobs.status text enum -> pgEnum('job_status') with same 4 values
- 7 columns changed from text to jsonb: jobs.inputFiles, jobs.settings,
  pipelines.steps, apiKeys.permissions, roles.permissions,
  auditLog.details, userFiles.toolChain
- settings.value stays text, jobs.error stays text, jobs.progress stays real

jsonb call-site sweep (removed JSON.stringify on writes, JSON.parse on reads):
- apps/api/src/routes/roles.ts: permissions read/write (3 sites)
- apps/api/src/routes/api-keys.ts: permissions write + read (2 sites)
- apps/api/src/routes/audit-log.ts: details read (1 site)
- apps/api/src/routes/pipeline.ts: steps write + read (2 sites)
- apps/api/src/routes/progress.ts: inputFiles write (2 sites)
- apps/api/src/routes/tool-factory.ts: toolChain read + write (2 sites)
- apps/api/src/routes/user-files.ts: toolChain read + write (4 sites)
- apps/api/src/permissions.ts: roles.permissions read (1 site)
- apps/api/src/lib/audit.ts: details write (1 site)
- apps/api/src/plugins/auth.ts: apiKeys.permissions read (1 site)

* refactor(db): type jsonb columns via $type and note raw CTE conversion requirements

* feat(db): archive sqlite migrations and generate postgres baseline

* chore(db): dockerignore legacy migrations, add archive breadcrumb, fix trailing newline

* feat(db): pg pool connection, advisory-locked boot migrations, DATABASE_URL config

* fix(db): friendly fatal on unreachable postgres, idempotent closeDb, lock-key convention note

* refactor(db): async drizzle calls in plugins, lib, permissions

* fix(api): analytics never throws, typed permission guard, single-query session invalidation

* refactor(db): async drizzle calls across all routes and bootstrap

Convert every route file and index.ts from sync SQLite drizzle
patterns to async node-postgres drizzle:

- .all() removed (bare await on select)
- .get() converted to destructured [row] = await ...
- .run() removed (bare await on insert/update/delete)
- .changes replaced with .rowCount (null-guarded) in progress.ts
- sqlite import removed from user-files.ts; raw CTEs converted to
  await db.execute(sql`...`) with postgres-dialect recursive CTEs
- ChainRow types updated: tool_chain is parsed jsonb (string[] | null),
  created_at is Date (timestamptz) with no * 1000 conversion
- All requirePermission() guard calls awaited (security: unawaited
  async guard returns truthy Promise, bypassing permission check)
- All hasEffectivePermission() and getPermissions() calls awaited
- All auditLog() calls awaited (preserves write-before-response order)
- trackEvent() and captureException() left un-awaited (fire-and-forget
  by design, guaranteed never-throw)
- ensureAnonymousUser(), startCleanupCron(), recoverStaleJobs() awaited
  in bootstrap sequence
- ensureInstanceId() and ensureDefaultSettings() made async

Files converted: 14 (index.ts + 12 route files + tools/index.ts)

* fix(db): await async checkStorageQuota in user-files upload/save routes

* fix(db): await checkStorageQuota in save-result route (missed second call site)

* feat(db): sqlite-to-postgres migrator with CLI and first-boot import

* fix(db): migrator error context, honest force semantics, boot-hook fatal, null-variance tests

* test: run suite against per-file postgres databases via testcontainers

- Add tests/global-setup.ts: spins up a Postgres testcontainer,
  creates a migrated template database once per vitest run.
- Rewrite tests/setup/per-fork-env.ts: each test file (forks pool)
  clones the template into its own database via CREATE DATABASE ...
  TEMPLATE, preserving the same per-file isolation granularity.
- Update vitest.config.ts: add globalSetup, pg alias, update comment.
- Fix tests/integration/test-server.ts: remove DB_PATH mkdir, async
  runMigrations, async db operations, remove SQLite WAL checkpoint.
- Fix 21 unit test db/index mocks: add pool and closeDb exports.
- Fix 8 unit test files: add async/await for now-async permission,
  audit, and analytics functions.
- Fix 18 integration test files: convert sync .run()/.all()/.get()
  to async drizzle patterns, add async to callbacks.
- Production change: apps/api/src/routes/teams.ts: cast COUNT(*)
  to ::int so Postgres returns a number instead of bigint string.

* fix(db): seed built-in roles, reject NUL bytes, cast COUNT, serialize job persists

- Seed built-in roles (admin, editor, user) at boot via ensureBuiltinRoles()
  with onConflictDoNothing, restoring data that legacy SQLite migration 0007
  provided via INSERT statements (the pg baseline is DDL-only).
- Reject NUL bytes in login credentials with 401 (postgres rejects \x00 in
  text columns; valid usernames never contain NUL, matching 1.x behavior).
- Cast COUNT(*)::int in user-files, audit-log, and roles listing queries so
  postgres returns a JS number instead of bigint-as-string.
- Serialize fire-and-forget job progress DB writes per jobId so the final
  "completed" status is never overwritten by a late-arriving "processing"
  write (race condition exposed by async postgres round-trips).

* test: fix teams race, seed roles in test server, poll for job status

- Add missing await to resetTeams() in teams PUT beforeEach (the async
  delete raced with the subsequent insert under postgres).
- Call ensureBuiltinRoles() in test server bootstrap so integration tests
  have the same built-in roles as production.
- Replace fixed 100ms flushPersist delay with a polling helper that waits
  for terminal job status, eliminating timing-dependent failures caused by
  postgres network round-trip latency.

* test: make heic temp-file cleanup assertion resilient to concurrent workers

Use a set-based diff instead of raw file count when checking that
decodeHeic cleans up temp files. Other concurrent test workers can
create heic-in-*/heic-out-* files in the shared tmpdir, inflating the
"after" count and causing spurious failures under full-suite load.

* fix(db): align builtin-role seed to post-0010 legacy state; test polish

* feat(docker): three-container compose (app, postgres, redis) with boot wait and migrations

* fix(docker): set TEST_DATABASE_URL so containerized tests skip testcontainers

* chore(docker): test compose project name, clearer 1.x upgrade comment, unref probe timer

* feat(enterprise): enforce D15 license boundary; move s3 storage into packages/enterprise

* fix(enterprise): restore lazy aws-sdk loading; community installs load no s3 code at boot

* fix(enterprise): boundary check catches dynamic imports; document getS3 concurrency

* feat(db)!: SnapOtter 2.0 phase 1 foundation: postgres, migrator, compose stack

BREAKING CHANGE: SQLite is no longer the runtime database. Deployments now
require Postgres (and Redis, used from phase 2). Existing installs migrate
with SQLITE_MIGRATE_PATH or 'pnpm --filter @snapotter/api migrate:sqlite'.

* fix(ci): postgres service + fresh e2e database per run; ignore unfixable torch CVE-2025-3000
2026-06-13 10:15:23 +08:00
SnapOtter d2fff1f87d fix(security): bump Go 1.25 and Pillow 12.2.0 for new Trivy CVEs
- Go 1.24 -> 1.25: fixes CVE-2026-25679 (HIGH: net/url IPv6 parsing)
- Pillow 12.1.1 -> 12.2.0: fixes CVE-2026-40192 (HIGH: decompression bomb DoS)
2026-06-08 17:32:34 +08:00
SnapOtter 0c58db2417 fix(security): update Go 1.24 and Pillow 12.1.1 for Trivy CVEs
- Bump Go from 1.23 to 1.24 to fix CVE-2025-68121 (CRITICAL:
  crypto/tls certificate validation) and CVE-2025-61726 (HIGH:
  net/url memory exhaustion) in the caire binary.
- Bump Pillow from 11.1.0 to 12.1.1 to fix CVE-2026-25990 (HIGH:
  out-of-bounds write via crafted image).
2026-06-08 17:06:07 +08:00
SnapOtter e0eb2e2075 fix(ci): add curl retry for libheif download and update release notes
- Add --retry 3 --retry-delay 5 to libheif curl in Dockerfile to
  handle transient GitHub CDN 504 errors on arm64 CI runners.
- Update release notes: use @ mentions, remove double-dash separators.
2026-06-08 16:17:28 +08:00
SnapOtter 9a1d3d25f4 fix(docker): resolve 4 release-blocking issues from validation
- fix(rate-limit): treat RATE_LIMIT_PER_MIN=0 as unlimited (50k/min)
  instead of blocking all requests. @fastify/rate-limit interprets
  max:0 as "allow zero requests," breaking fresh container startups.

- fix(docker): add libgles2 for MediaPipe face detection tools.
  blur-faces, red-eye-removal, enhance-faces, and passport-photo
  failed with "libGLESv2.so.2 not found" on all headless containers.

- fix(docker/arm64): remove conflicting system libheif1 to avoid
  ABI symbol mismatch with our custom libheif 1.21.2 build.
  heif-convert failed with "undefined symbol: heif_get_plugin_directories."

- fix(docker/arm64): pre-install wheel+setuptools in base Python venv
  so basicsr can build from source on arm64 (no pre-built wheel).
  This unblocks upscale-enhance and photo-restoration bundles.
2026-06-08 14:07:55 +08:00
SnapOtter ace41168bc fix(security): harden API against pentest findings
- Default TRUST_PROXY=false to prevent XFF rate limit bypass (PT-01)
- Return 400 instead of 500 on malformed JSON input (PT-03)
- Default MAX_PIPELINE_STEPS=20 to prevent DoS (PT-04)
- Validate clientJobId length (max 128) across all routes (PT-06)
- Add security headers to all reply.hijack() streaming responses (PT-07)
- Sanitize usernames in audit log to prevent stored XSS (PT-08)
- Block TRACE method with 405 response (PT-10)
- Add 429 RateLimited response to OpenAPI spec (PT-12)
- Default MAX_SVG_SIZE_MB=50 to limit SVGZ decompression (PT-13)
- Pin Dockerfile base images by digest
- Sanitize OIDC IdP error and sub claim in audit log
- Sync Docker compose/Dockerfile defaults with env.ts
2026-06-07 21:54:27 +08:00
SnapOtter 4d7f67d2af feat: add playwright chromium to docker image for html-to-image 2026-06-06 21:45:39 +08:00
SnapOtterandGitHub c365fde599 fix: build libheif 1.21.2 from source for iPhone HEIC support (#183) (#199)
iPhone 15 Pro (iOS 18+) HEIC files include HDR gain maps as auxiliary
image references. Distro-packaged libheif (1.15-1.17) rejects these
with "Too many auxiliary image references". Build libheif v1.21.2 from
source in a new Dockerfile stage to fix decoding.

- Add libheif-builder stage with platform-matched bases (debian:bookworm
  for arm64, ubuntu:24.04 for amd64) to avoid shared-library ABI
  mismatches
- Replace libheif-examples distro package with source-built binaries
- Update Dockerfile.test with same libheif source build
- No application code changes needed (heic-converter.ts CLI interface
  is stable across versions)

Closes #183
2026-06-05 17:05:01 +08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
e4608991ef chore(deps): bump nvidia/cuda in /docker (#185)
Bumps nvidia/cuda from 12.6.3-cudnn-runtime-ubuntu24.04 to 12.9.2-cudnn-runtime-ubuntu24.04.

---
updated-dependencies:
- dependency-name: nvidia/cuda
  dependency-version: 12.9.2-cudnn-runtime-ubuntu24.04
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-04 22:16:35 +08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
7af141be02 chore(deps): bump node from e3ca095 to 1031993 in /docker (#184)
Bumps node from `e3ca095` to `1031993`.

---
updated-dependencies:
- dependency-name: node
  dependency-version: 22-bookworm
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-04 21:48:35 +08:00
SnapOtterandGitHub ec6ff3d8a8 chore: harden OpenSSF Scorecard from 4.3 to ~7.0 (#142)
Token-Permissions (0 -> 10): Set permissions: {} at workflow top level
across all 7 workflows, moved write scopes to per-job minimum.

SAST (0 -> 10): Added CodeQL workflow for JavaScript/TypeScript and
Python analysis on push, PR, and weekly schedule.

Vulnerabilities (0 -> ~8): Added 13 pnpm overrides to patch transitive
dependency vulnerabilities (38 -> 2 remaining, both in dev-only tools).

Pinned-Dependencies (5 -> 8-9): Pinned all Docker FROM images to SHA
digests, pinned pip-audit version in CI, pinned pip version in
Dockerfile.
2026-05-18 15:57:17 +08:00
SnapOtter f86ef124c2 fix: grant admin role to anonymous user and add DEFAULT_TOOL_VIEW env var
When AUTH_ENABLED=false, the anonymous user was assigned the "user" role
which lacks settings:write permission, making all settings saves return
403. Since no admin exists when auth is disabled, settings were
permanently read-only. Promote the anonymous user to "admin" so the
single user has full control of the instance.

Also adds DEFAULT_TOOL_VIEW env var (sidebar|fullscreen) following the
existing DEFAULT_THEME pattern, seeded via ensureDefaultSettings() on
first boot.

Closes #135
2026-05-16 11:37:28 +08:00
SnapOtter b449bd1a56 fix(security): correct Docker rate limit defaults
- LOGIN_ATTEMPT_LIMIT: 500 -> 30 in Dockerfile (matching code default)
- RATE_LIMIT_PER_MIN=0 now means no global limit (50K ceiling) instead
  of 1 req/min, preserving backward compatibility with existing deploys
  while keeping per-route limits (login brute-force) active
2026-05-14 23:02:25 +08:00
SnapOtter ca2ef5b3f4 feat: add OIDC/SSO authentication (#3)
Add OpenID Connect (OIDC) authentication alongside existing
username/password login. Users can log in via any standards-compliant
OIDC provider (Keycloak, Authentik, Authelia, Google, Azure AD, Okta)
while preserving full backward compatibility.

- OIDC Fastify plugin with lazy discovery, PKCE, cookie-based sessions
- Login page OIDC button, auth hook updates, settings dialog badges
- 28 integration tests, OIDC setup guide with provider examples
- Fix pre-existing test failures (content-aware-crop, watermark, SVGZ)
- WAL checkpoint fix for SQLite test stability

Closes #3

# Conflicts:
#	apps/api/src/lib/env.ts
#	apps/api/src/routes/tools/watermark-image.ts
#	pnpm-lock.yaml
#	tests/integration/color-palette.test.ts
#	tests/integration/compare.test.ts
#	tests/integration/watermark-image.test.ts
2026-05-14 22:31:26 +08:00
SnapOtter 4e64ee2779 fix(security): comprehensive security audit and hardening
Auth: login rate limit 30/min (was 500), global rate limit 1000/min (was
unlimited), password/username max lengths on all Zod schemas, session
invalidation on role change, API key legacy scan bounded to 100 keys.

SVG: hardened regex sanitizer with CDATA stripping, XML entity decoding,
set/animate/iframe/embed blocking, comprehensive data: URI blocking,
use element external href blocking. 11 attack payload fixtures added.

SSRF: fixed DNS rebinding TOCTOU by pinning resolved IPs via custom
HTTP/HTTPS agents. Added 6to4 and NAT64 to blocked IPv6 ranges.

Docker: capability dropping (cap_drop ALL + minimal cap_add), resource
limits (4g/8g mem, 512/1024 pids), healthcheck timeout, password
removed from startup banner, default password warning comments.

Network: CSP and HSTS applied in all environments (not just production),
stack traces removed from all error responses, internal paths stripped
from error details, per-route rate limits on uploads (60/min) and URL
fetches (200/hour).

Files: exclusive temp file creation (O_EXCL), disk space circuit
breaker, per-user storage quotas, settings payload 64KB size guard.

Python sidecar: script name allowlist in dispatcher, minimal environment
for subprocess spawns.

Dependencies: fixed 6 production CVEs (drizzle-orm, fastify, fast-uri,
@fastify/static, next, archiver/lodash). Pinned all GitHub Actions to
SHA hashes.

114 security tests added. Full OWASP Top 10 penetration test matrix
verified against production Docker container (30/30 pass after
hardening).
2026-05-13 21:33:50 +08:00
SnapOtter 71496db6c5 feat(oidc): add OIDC environment variables to Docker configuration 2026-05-13 19:07:33 +08:00
SnapOtter 7f131d99a6 fix: QA sweep -- SSE crash, memory leaks, HEIC Docker decode, TGA detection, lint cleanup
- Fix SSE write-after-end crash in progress.ts (remove callback before ending stream)
- Fix blob URL memory leaks: revoke processedPreviewUrl and old HEIC preview URLs
- Add AbortController to batch fetch in use-tool-processor and use-pipeline-processor
- Fix TGA format misidentified as CUR (extension overrides magic bytes)
- Add libheif-plugin-libde265 to Docker for HEIC/HEIF decode support
- Remove unused imports and state (AppLayout, setSampledColor, useEffect)
- Fix non-null assertions in meme-text-renderer and meme-generator
- Fix confusing void type in meme-templates
- Remove unnecessary useEffect deps in adjustments-panel
- Fix Playwright strict mode violations in 5 E2E tests
2026-05-09 13:55:00 +08:00
SnapOtter aa773addb4 merge: resolve conflict with main branch in tool-registry.tsx 2026-05-08 18:55:28 +08:00
SnapOtter f9e0e90897 fix: include apps/api/static in Docker image for meme templates and fonts 2026-05-08 16:59:00 +08:00
SnapOtter 136a4a0752 Merge branch 'feat/format-support-expansion'
# Conflicts:
#	README.md
#	apps/web/src/components/layout/app-layout.tsx
2026-05-08 16:21:11 +08:00