* fix: remove all automatic third-party egress (OSM tiles, Scalar fonts, editor Google Fonts, AI model download fallbacks)
Phone-home audit follow-up. The product no longer makes any automatic
third-party request; user-initiated click-outs stay, and production now
fails closed on missing AI models.
1. GPS leak via OSM tiles: the strip-metadata panel auto-loaded
tile.openstreetmap.org tiles encoding the photo's GPS position. The
Leaflet mini-map is gone; coordinates render as text plus an explicit
View on map link (openstreetmap.org, opens on click only). Removed
tile.openstreetmap.org from the CSP img-src, dropped the leaflet
dependency, added the viewOnMap i18n key to all 21 locales.
2. Scalar docs fonts: /api/docs loaded Inter and JetBrains Mono from
fonts.scalar.com. Scalar now renders with withDefaultFonts: false and
both --scalar-font and --scalar-font-code pinned to system stacks;
fonts.scalar.com removed from the docs CSP font-src. Verified by
injecting GET /api/docs/: config carries withDefaultFonts false and
the served page has no fonts.scalar.com reference.
3. Editor Google Fonts: the editor font picker built
fonts.googleapis.com stylesheet URLs for 25 web fonts the served CSP
already blocked. The remote loading path is deleted; the picker now
offers system fonts only, with a SELF_HOSTED_FONTS seam (FontFace API,
same origin) for bundling fonts later. Unknown families saved in old
documents fall back to the browser default.
4. Python sidecar fails closed on model downloads: new
packages/ai/python/offline_guard.py gates every runtime download
fallback (inpaint, outpaint, restore, noise_removal, detect_faces,
enhance_faces, face_landmarks, red_eye_removal, remove_bg, ocr,
transcribe, upscale) behind SNAPOTTER_ALLOW_MODEL_DOWNLOAD=1 with an
actionable error. Bundled models keep working untouched.
5. OCR and transcription library-internal downloads: unbundled PaddleOCR
language and detection fallbacks now raise the guard error naming the
language instead of resolving models over the network; faster-whisper
gets local_files_only when downloads are off.
6. GFPGAN and CodeFormer cwd-relative weights: facexlib and
codeformer-pip resolve helper weights relative to the process cwd and
fetch them from GitHub when absent. They are now symlinked from the
installed bundle files under MODELS_PATH/gfpgan/facelib before the
libraries load, failing closed when unresolvable.
Defense in depth: HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 are set in
the runtime image and in the sidecar spawn env; install_feature.py lifts
them for user-initiated bundle installs and restores them afterwards
(it can run in-process inside the dispatcher). SNAPOTTER_ALLOW_MODEL_DOWNLOAD
is documented in .env.example, default off.
Validation: typecheck 9/9 workspaces, Biome clean on touched files,
5178 unit tests pass, py_compile on all touched scripts, guard behavior
exercised in both dispatcher exec and per-request import modes, zero
remaining runtime references to the three hosts. Docker build and live
AI inference need post-merge verification on the GPU host.
Claude-Session: https://claude.ai/code/session_01XGB4pGvTvb7sUX4JN745U7
* fix: allow AI model downloads by default, make strict offline mode opt-in
Product call: ease of use first. The download gating from the previous
commit inverts its default: runtime model fetches (public model weights
only, never user data) are allowed out of the box so AI tools self-heal,
and SNAPOTTER_ALLOW_MODEL_DOWNLOAD=0 becomes the explicit strict offline
mode for airgapped deployments, where every fallback raises the
actionable error instead of fetching.
Changes: offline_guard blocks only on an explicit 0/false; the
unconditional HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE image ENV is removed
and bridge.ts sets those flags for the sidecar only in strict mode;
.env.example documents the new default; install_feature's lift/restore
stays. All bundled-path preferences, pre-existence checks, and symlink
pre-placement remain, so installed bundles never trigger a download.
The OSM, Scalar font, and editor font fixes are unchanged.
Validation rerun: typecheck 9/9, Biome clean on touched files, 5178
unit tests pass, py_compile on touched scripts, guard behavior verified
for unset/1 (allowed) and 0/false (blocked with the new message).
Claude-Session: https://claude.ai/code/session_01XGB4pGvTvb7sUX4JN745U7
Fixes found by manually testing a fresh install end to end:
- auth: the must-change-password gate returned 403 on public routes
including /api/v1/health, so every fresh install showed a false
"Reconnecting to server" banner on the forced password change
screen. Public routes are now exempt (they need no session at all).
Adds the gate's first direct tests.
- multipart: @fastify/multipart's parts() iterator (9.4.0 and 10.0.0)
ends on the request stream's "close", which on a reused keep-alive
connection fires while an earlier part is still streaming to storage,
silently dropping the parts behind it. The object eraser lost its
mask file on every second POST per connection. Replaced with a
busboy-driven iterator (lib/multipart-parts.ts) that ends on busboy's
own "finish", installed for all routes via a preValidation hook;
the tool-factory field-recovery workaround for the same bug is now
unnecessary and removed.
- eraser: the mask canvas backing store is natural resolution, but
"absolute inset-0" does not stretch replaced elements, so the
canvas rendered at intrinsic size and the brush ring, strokes, and
exported mask were all misscaled on photos larger than the viewport.
The canvas now gets an explicit CSS box at the fitted size.
- compare slider: solid white divider with a dark halo so it stays
visible over light images; still initialised at the painted region.
- tool page: the AI bundle install prompt now centers in the content
area instead of hugging the top.
- api docs: disabled Scalar's cloud features (Ask AI, Generate MCP,
Open API Client, dev toolbar), hid the "Powered by Scalar" footer
link, and set the page title to "SnapOtter API Reference". The docs
CSP blocks those cloud calls by design, so the buttons were dead UI.
- docker: embedded Redis comes from packages.redis.io pinned to the
8.x major (was Debian's 7.0.15), matching the Compose stack and the
documented claim. Build fails fast if the major ever drifts.
- docs: DOCKERHUB.md quick start now leads with the one-command docker
run (matching the README) with Compose as the production path;
README says embedded Postgres 17 + Redis 8.
Claude-Session: https://claude.ai/code/session_01XGB4pGvTvb7sUX4JN745U7
* fix(api): correct format/filename/container handling across tool routes
Found during a comprehensive QA sweep exercising every tool against its
full accepted-format matrix:
- watermark-image, compose: preserve the requested output format and a
matching download filename/extension instead of always emitting the
source format
- compose: crop oversized overlays to the visible base area instead of
crashing Sharp's composite, and reject only overlays fully outside the
base image instead of any oversized one
- compare, vectorize: switch to the shared image input handler so
filenames and formats like .svgz/.tga/RAW survive validation instead
of being rejected pre-processing
- tool-factory, images-to-video: normalize frames through Sharp before
handing them to FFmpeg, fixing GIF/AVIF/RAW image-to-video jobs that
previously failed or hung
- media-tool, replace-audio, embed-subtitles: fix legacy container
MIME/codec handling for MPEG sources and subtitle remux cases
- files: expand download MIME mapping for text/data/document/video/audio
outputs that were falling back to a generic content type
- convert-document/presentation/spreadsheet: same-format conversions now
return the original validated file instead of erroring or producing
corrupt tiny output
Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG
* fix(web): dropzone a11y, stale localStorage getter, dead code
- dropzone: stop making the whole drop-zone section clickable/focusable.
A section acting as an interactive element around a real upload button
is a nested-interactive-element anti-pattern that confuses screen
readers; drag-and-drop doesn't need focus semantics, only the button
fallback does. Keeps that button semantic and keyboard-reachable.
Updates the two e2e call sites that clicked the section directly.
- api, use-auth: read through window.localStorage via the existing API
storage helper instead of the bare global, which resolves to Node's
experimental localStorage getter under Vitest and threw
- find-duplicates-settings, info-settings, login-page: remove dead code
(unused zip-download handler, a stale mount-only effect dependency
that left cached info stuck at reused indices, an unused response
variable)
Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG
* fix(i18n): pt-BR, zh-CN, zh-TW were silently falling back to English
The locale loader looked up dynamic-import exports by the raw locale
code (mod["pt-BR"], mod["zh-CN"], mod["zh-TW"]), but those three modules
export camelCased bindings (ptBR, zhCN, zhTW) since identifiers can't
contain hyphens. The lookup returned undefined and every consumer
silently fell back to English for these three locales. Replaces the
generic lookup with explicit per-locale loaders so the mapping can't
drift out of sync again.
Also updates the dropzone helper copy across all 21 locales to match
the drag-only dropzone wording from the previous commit.
Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG
* fix(docs): clear build warnings in the VitePress site
- config.mts: add an onwarn handler for the @vueuse INVALID_ANNOTATION
warnings emitted during the docs build
- deployment.md: the caddyfile code fence language isn't a shiki grammar
VitePress ships with, so it warned on every build; use txt instead
Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG
* test(qa): update QA harness for the drag-only dropzone and regen metadata
- api-sweep, qa-helpers, verify-ai: add JSON-body tools, multi-input
secondary fixtures, async polling for slow valid jobs, 501
FEATURE_NOT_INSTALLED skip handling, and safer per-tool settings
- input-preview, pipeline-ui specs: update upload flow for the
drag-only dropzone surface
- add tests/fixtures/data/valid/chart.json, a valid chart fixture the
updated helpers route to
- regenerate tools-meta.json against current TOOLS[]
Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG
* fix(security): close a login timing side-channel, harden zip-slip tests
Found during a black-box security sweep of the real auth-enabled
production container: a nonexistent username returned 401 in ~3-10ms,
while a wrong password for a real user took ~35-42ms, because scrypt
verification only ran when a user row existed. That timing gap lets an
attacker enumerate valid usernames without ever guessing a password.
Now runs verification against a cached dummy hash on the unknown-user
path too, so both cases cost the same regardless of outcome.
extract-zip already had a relative-traversal regression test
(../evil.txt), but its absolute-path rejection branches
(name.startsWith("/") / startsWith("\\")) had none. Added the three
missing cases: deep relative traversal, absolute Unix path, and
Windows-style absolute path.
Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG
* test(qa): add UI-driven AI bundle install scripts
QA_PROMPT.md's Phase 2 requires installing AI models the way a user
does -- through the UI, on demand from HuggingFace -- and treats the
curl-based admin install endpoint as fallback-only. Nothing in the
harness actually drove that flow; tests/qa/seed-ai-models.sh installs
via docker exec + pip, which is further from a real user than even the
API fallback.
install-ai-bundles-ui.mts logs in, opens Settings > AI Features,
screenshots the pre-install state, clicks Install All, and screenshots
progress -- then exits, since installs continue server-side once
triggered. verify-ai-install-complete.mts polls bundle status,
screenshots the completed state, and runs one real tool per installed
bundle to prove the freshly-downloaded model actually executes.
Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG
* fix(qa): correct the apiToolPath import in the AI verify script
Dynamic import of the package name failed under tsx's module resolution
from apps/api's node_modules context; use the same relative-path import
api-sweep.mts already uses successfully.
Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG
* fix(web): correct AI bundle size estimates shown before install
Measured real downloads during GPU-node QA verification: photo-restoration
pulls ~4.4GB (was advertised as 800MB-1GB, off by 4-5x) and ocr pulls
~5.5GB (was advertised as 3-4GB). Both estimates only accounted for model
weights, not the pip dependencies (torch/paddle) that come down with them.
Updated to reflect actual total download size, since that's what a user
deciding whether they have the disk/bandwidth actually needs to know.
Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG
* fix(web): make desktop Settings reachable when auth is disabled
AvatarDropdown (the only desktop entry point to Settings) was gated
behind `!isMobile && authEnabled`. With AUTH_ENABLED=false the synthetic
anonymous admin user should have full Settings access per how auth.ts
documents this mode -- and the mobile bottom nav already worked this way,
showing Settings unconditionally. Desktop just had a stray extra gate the
component doesn't need: AvatarDropdown already resolves its own username
internally (falling back to "admin") and reads authEnabled itself where
it actually matters (hiding the Logout button). Removed the outer gate;
verified end-to-end against a fresh AUTH_ENABLED=false instance -- avatar
now renders, Settings opens, shows the anonymous/Admin identity correctly.
Also documents (not changes) a related finding in install_feature.py:
detect_arch() always resolves amd64 hosts to the GPU-bundled archive
variant regardless of actual GPU presence, since no CPU-only amd64
archive is published to the bundle repo yet. Left as a code comment
rather than a behavior change, since requesting an unpublished archive
key would hard-fail installs entirely -- worse than the current
oversized-but-working download. Full detail in the QA report.
Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG
* fix(ai): stop logging expected dispatcher reloads as crashes
After each AI bundle install the Python dispatcher reloads because the
venv changed, and after every app shutdown it's SIGTERMed. Both took the
close handler's `code !== 0` branch (SIGTERM makes the exit code null),
so they were counted as crashes -- producing an alarming "crash" line in
the logs and a pointless ~1s recovery backoff after each of 7 installs.
A `stopping` flag set in shutdown() lets the close handler tell an
intentional stop apart from a real crash. The request-timeout kill path
deliberately does not set it, so a genuinely hung script still records a
crash and the 5-in-60s permanent-disable threshold is untouched.
Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG
* fix(api): return a clean message when content-aware resize times out
Carving a very high-resolution image down to a tiny target could exceed
the caire subprocess timeout, and the raw error forwarded to the user was
caire's terminal output -- ANSI color codes and progress-spinner control
characters -- instead of anything actionable. Now: the timeout path
throws a clear "timed out; try a smaller image or larger target" message
(keeping the raw stderr as `cause` for server logs); friendlyError()
strips ANSI/control chars centrally so any subprocess dump surfaced
through the shared sanitizer is plain text; and the content-aware-resize
route (a custom route that bypassed the sanitizer) now routes its error
paths through friendlyError like every other tool.
Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG
* fix(ai): stop bundle installs from exhausting host disk
Installing an AI bundle on a tight-disk host could push the root
filesystem to zero bytes free after the preflight check had already
passed. Two root causes:
- move_tree used copytree+rmtree, so during the move the extracted
payload existed in both staging and the venv at once -- a full
transient doubling on disk. Rewrote it to rename entries (a cheap
metadata op on the same filesystem, no copy), falling back to a copy
only across filesystems.
- the preflight budget used the manifest's extractedSize verbatim, which
is 0 for several archives, collapsing the estimate to just the
compressed size. Added a conservative fallback (3x compressed) so a
missing value can't under-reserve.
Also added a real-on-disk re-check immediately before the first
destructive venv write (measuring the actual extracted payload and
whether the move needs extra space for a cross-filesystem copy), which
also now covers the offline-import path that previously skipped the disk
check entirely; wrapped the moves so an out-of-space failure returns a
clean actionable error instead of a traceback; and made the disk check
resolve the nearest existing ancestor so it never throws on a
not-yet-created venv path.
Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG
* feat(web): show the real per-arch AI bundle download size
The bundle cards and install prompt showed a hardcoded, architecture-blind
estimatedSize string. That's misleading: amd64 hosts always pull the
CUDA-inclusive archive (there's no CPU-only amd64 variant published), so a
bundle labelled "1-2 GB" can actually download several times that, while
arm64 pulls a much smaller archive for the same label. The manifest
already carries the real per-arch compressedSize (and extractedSize where
measured), so surface those: a new optional downloadBytes/installedBytes
on FeatureBundleState, populated in getFeatureStates() for this host's
arch (resolver mirrors install_feature.py detect_arch), shown by the UI
when present with estimatedSize kept as the fallback label. Also nudged
upscale-enhance's fallback string (4-5 -> 5-6 GB) to match its real
compressed size, consistent with the earlier photo-restoration/ocr fixes.
Fields are optional so demo/mock and existing tests stay compiling; the
manifest's extractedSize is 0 for a few archives, which now surfaces as
null rather than a bogus 0.
Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG
* fix(web): move the AI install queue to the server so it survives tab close
Installing multiple bundles could silently lose all but the first. The
server rejected a concurrent install with 409, so the client worked
around it by queueing the rest in browser-local state and only POSTing
each once it saw the previous finish. A single POSTed install is durable
(the installer child is detached from the request), but a queued one had
zero server footprint -- close the tab mid-queue and those installs
vanished with no error, while the UI still showed them "Queued". The
client "mutex" didn't even serialize: the queued bundles' local waits all
resolved at once and raced into concurrent POSTs that 409'd each other.
Now the queue lives on the server (a small in-memory FIFO leaf module).
The install endpoint enqueues instead of 409-ing and returns
202 {jobId, queued}; a pump starts the next bundle when the current one's
child exits (and after an offline import releases the lock), all behind
the existing venv + file locks, which are unchanged. The client just
POSTs every bundle immediately and reflects the server-reported
queued/installing status; Install All fires all POSTs and lets the server
serialize them, keeping the one-shot retry-on-failure. Adds "queued" to
FeatureStatus (the bundle card already rendered that state) and surfaces
it from getFeatureStates. In-memory is deliberate: it matches the
existing contract (survives a tab close, not a server restart, which
already clears the lock on boot).
Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG
* fix(qa): don't log env-derived credentials in the AI-install script
CodeQL flagged clear-text logging of sensitive information: the login
status line interpolated the QA base URL and username (both read from
the process environment) into a console.log. Replaced with a static
message. QA helper only, but it's a real hygiene issue and cleared the
high-severity code-scanning alert on the PR.
Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG
AI feature installs now keep copied Python venv metadata (bin/pip shebang,
bin/activate, pyvenv.cfg) pointed at /data/ai/venv, so scripts no longer
silently fall back to the baked, read-only /opt/venv after the venv is
bootstrapped into /data. Fixes#127 (AI tools incompatible with PUID/PGID).
The entrypoint repairs both fresh bootstraps and already-stamped runtime
venvs (self-heals existing deployments on next restart, no reinstall
needed), with regression coverage for literal path replacement and binary
file safety.
Independently reviewed and verified: traced chown/gosu ordering in
entrypoint.sh to confirm no permission regression, reproduced the exact
issue #127 scenario (custom PUID + manual venv activation) in a live
container both before and after the fix, and ran the PR's own test suite
locally (16/16 passing).
Co-authored-by: SyntaxSawdust
Adds a prefilled 'Request a tool' affordance to the home search empty state and beneath weak results. Opens the in-app feedback dialog with a new search_miss source and a structured search_query when analytics is on; links to a prefilled GitHub Discussions (Ideas) post when off, so a request is never silently dropped. Reuses the existing feedback pipe, dialog, and analytics gate; no new storage. i18n across all 21 locales.
* fix(landing): correct PDF tool count to 28 in alternatives copy
The pdf section has 28 tools (section.test.ts asserts bySection('pdf')=28). PR #363 corrected the docs breakdown but the alternatives pages still said 40 PDF tools (the document-modality count, not the pdf section) with 200 for the rest. Update to 28 PDF tools and 212 for the non-PDF remainder.
* chore: use "200+ tools" for the tool-count claim across public surfaces
Replaces the exact '240 tools' count (which drifts as tools are added) with the stable '200+ tools' on README, the Docker Hub overview, landing pages, the docs site (meta, homepage, search), the API self-description, the demo OG tag, llms.txt, package.json, the branding readme, and the en/nl app strings. Per-modality breakdown tables stay exact. Leaves the architecture doc's technical 'tool routes' figure, an internal vitest comment, and a QA report line unchanged. Updates the two tests that assert the docs strings.
Lands five integrated branches: pipeline templates (#355), analytics opt-out (#354), 83 conversion presets bringing the catalog to 240 tools (#356), self-hosted positioning (#353), and e2e modernization (#351).
Integration fixes: aligned stale web analytics tests with the opt-out/allow-list model, closed 3 CodeQL incomplete-sanitization alerts in the i18n generator, resolved settings/index/docs/format-matrix conflicts, and corrected tool counts to 240.
The entrypoint only fixed volume permissions when started as root (chown +
gosu-drop to snapotter). Launched under a non-root/foreign UID (TrueNAS app
user, Kubernetes runAsUser, OpenShift) it did no permission setup, so /data and
/tmp/workspace -- owned by uid 999 from the image -- were not writable by the
running user. Uploads and processing then failed with a cryptic EACCES
("workspace folder is not writable") and AI bundle installs failed the same way,
while health checks still reported the container healthy.
- entrypoint: source new entrypoint-lib.sh; verify writability up front when
non-root, and as snapotter after chown when root (catches root-squashed
mounts), failing fast with an actionable message (which dir, uid/gid, how to
fix) instead of a late, cryptic EACCES
- Dockerfile: own /data and /tmp/workspace as snapotter:0, group-writable with
setgid, so an arbitrary UID with the root supplementary group (OpenShift /
Kubernetes fsGroup) can write; keep /opt/venv world-readable for the AI venv
bootstrap under arbitrary UIDs
- api: assert storage writability at boot (lib/storage-writable.ts), failing
fast with the same guidance even when the entrypoint is bypassed
- docs: add a Storage permissions section (named volumes, bind mounts, TrueNAS,
Kubernetes/OpenShift) and cross-link it from the security guide
Fixes#230
RAW (DNG) processing crashed on ImageMagick's deprecated ufraw-batch
delegate, which fails on modern formats such as iPhone ProRAW DNG.
Root cause: the dcraw_emu (LibRaw) decode tier read the wrong output path.
dcraw_emu APPENDS the output extension (raw-in-X.dng -> raw-in-X.dng.tiff)
but the code looked for raw-in-X.tiff (replaced extension), so readFile threw
on every RAW, the tier silently fell through to ufraw, and the 24MB TIFF
leaked into the temp dir on each attempt.
- Repair the dcraw_emu output path; clean it up in finally (fixes the leak)
- Prefer LibRaw full decode over embedded-preview extraction so a
full-resolution RAW is never silently returned as a reduced-size preview
(sample DNG: was 1024x683 preview, now 3474x2314 full)
- Add RAW decode regression tests (DNG full-resolution + all 6 RAW formats);
these were absent, which let the bug ship
- Install libraw-bin on CI test runners so dcraw_emu is actually exercised
Replace textual prefix matching in isPrivateIPv6() with numeric bit-prefix matching via ipaddr.js. Closes reachable classifier bypasses for IPv4-compatible IPv6 (::a.b.c.d), site-local fec0::/10, and the full fe80::/10 link-local span, on top of the IPv4-mapped forms. Embedded IPv4 (mapped and compatible) is run through the full isPrivateIPv4 classifier; unparseable input fails closed.
Follow-up hardening to f64cbdda.
Reported-by: tonghuaroot
Fixes a backlog of integration/unit specs that asserted pre-2.0 behavior and
were failing CI (not caused by recent feature work):
- modality-aware empty-input error is 'No file(s) provided', not /no image/i
(rotate, border, crop, resize, smart-crop, edge-cases, adversarial-extended,
api, tool-factory-route)
- input validation rejects pre-enqueue with a clean 400 in 'error' (was a worker
422 in 'details'): create-zip, extract-zip, merge-csvs
- resolveToolPool defaults unknown tools to the system pool (pool-routing)
- /upload and fetch-urls accept non-image content, validated per-tool at process
time (api, fetch-urls)
- color-adjust legacy aliases were consolidated into adjust-colors: drop the
removed-alias tests; retarget the format-preservation tests
- xml-to-csv gracefully converts a single non-repeating record to a 1-row CSV
- dropzone is multimodal; image-only filtering is opt-in via fileFilter
- factory-multi-input: register the synthetic test tools in the catalog so they
route correctly (file modality for concat; image for the validation-prefix test)
Verified locally: unit 4546 passed, integration 8332 passed, typecheck + lint green.
The full unit suite surfaced two issues from earlier commits on this branch. (1) Importing the app logger into media-input.ts pulled logger.ts -- which builds its pino file transport at module load via join(env.LOG_DIR, ...) -- into the unit-test import graph, throwing at collection time wherever LOG_DIR is unset (integration tests set it; unit tests do not). A low-level modality handler should not depend on the app logger, and a corrupt upload is an expected user error, so drop the import and keep the clean validation message. (2) tool-factory-route.test.ts mocked errors.js without the new friendlyError export; add it.
* docs: rebrand from image-only to multi-modality across docs and metadata
SnapOtter expanded from image-only to 157 tools across 5 modalities
(image, video, audio, document/PDF, data). Update all product-level
copy, metadata, and i18n that still framed it as an image-only tool.
- README, package.json, root llms.txt: multi-modality framing, 157 tools
- OpenAPI info + tags, generated /llms.txt tagline (docs.ts)
- VitePress docs site: hero, getting-started, architecture, security,
deployment, configuration, developer, supported-formats
- i18n: 10 product keys across all 21 locales (hero, app description,
privacy notes, AI features, progress messages, getting-started)
- web/demo/landing meta + privacy copy, COMMUNITY_GUIDE, .env.example
Stale tool counts (53/50+/52/70+/35) corrected to 157 throughout.
Database/container deployment claims left unchanged (out of scope).
* docs: fix stale post-rebrand test assertions and README language list
- tests/e2e-docs/homepage.spec.ts: assert the current docs homepage (file toolkit, 157 tools, 5 modalities) instead of the old image-only strings
- tests/unit/api/docs-route.test.ts: sync the reproduced llms.txt tagline with docs.ts
- README.md: 21 languages with the correct list (add Swedish and Chinese Traditional, drop Czech which is not supported)
* docs: correct 2.0 architecture references (Postgres 17 + Redis 8, 3-container stack)
The docs and metadata still described the 1.x stack (SQLite, single container, p-queue). Update them to the current 2.0 reality.
- README: replace the broken single-container `docker run` quick-start with the real Docker Compose stack (app + Postgres 17 + Redis 8); fix the "no Redis, no Postgres" feature bullet
- package.json: description no longer claims a single container
- apps/docs: rewrite database.md for Postgres; configuration.md DB_PATH -> DATABASE_URL + REDIS_URL; architecture.md SQLite/p-queue/better-sqlite3 -> Postgres/BullMQ/pg and add media-engine + doc-engine; developer/security/deployment/docker-tags/getting-started/contributing compose examples now include postgres + redis; index.md + api/ai.md AI count 16 -> 19
- SECURITY.md: Drizzle (SQLite) -> (PostgreSQL)
- landing: enterprise/FeatureHighlights single-container wording; TrustSignals/ToolGrid 150+ -> 157 (dynamic); Pricing/FAQ 15 -> 19 AI tools
* docs(api): document all video, audio, document, and data tool endpoints in OpenAPI
The spec covered only image tools; the Scalar UI and the generated /llms.txt and /llms-full.txt inherited that gap. Add the 104 missing tool endpoints so the API docs match the code.
- Video: 29 endpoints (most long/async; auto-subtitles is AI)
- Audio: 17 (transcribe-audio is AI)
- Document/PDF: 36 (ocr-pdf is AI; conversions are long/async)
- Data: 10
- Image: 12 newer tools (background-replace, blur-background AI; histogram/lqip-placeholder/sprite-sheet custom responses; barcode-generate uses a JSON body)
Each schema is derived from the tool's Zod validator and executionHint (fast -> 200, long -> 202+SSE, AI adds 501 FeatureNotInstalledError, multi-file inputs as arrays), referencing the existing shared schemas. Tool path entries: 64 -> 168. Spec parses as valid YAML with no duplicate paths and only known $refs.
SnapOtter spans five modalities now, but several code paths still assumed image input.
- dropzone: default to accept-all when no fileFilter is given (image tools still pass one); neutral "supported file types" error text instead of "image files"
- automate (pipelines): accept any modality in the file pickers and dropzones; render modality-aware previews (video player, audio waveform, document/data card) instead of always using ImageViewer/BeforeAfterSlider
- filename sanitizer: extend the double-extension allowlist beyond image extensions to video/audio/document/data so e.g. "report.csv.php" becomes "report.csv"; add tests
- thumbnail route: return 422 for non-rasterisable files (audio, data, non-PDF docs) instead of attempting a doomed Sharp decode
- pool: unknown tools fall back to the "system" pool, not the image pool
- a11y labels: "Previous/Next image", "Image viewer/area/controls/drop zone" are now modality-neutral, across all 21 locales
- copy: bulk-rename default, find-duplicates ZIP name, SSRF user-agent, fetch-urls fallback name, file-details MIME label, URL-import placeholder, help dialog
* feat(tracing): add OpenTelemetry dependencies and --import preload flag
* feat(enterprise): add distributed_tracing feature gate
* feat(tracing): add SDK bootstrap with enterprise gating
* fix(tracing): correct test coverage for enterprise-unavailable path and prevent double-init
Test 2 now mocks @snapotter/enterprise to throw an import error, exercising
the catch block in the preload. Test 3 imports with no endpoint so the preload
is a no-op, avoiding leaked SDK from double-initialization. Added idempotency
guard to initTracing() as a safety net.
* feat(tracing): add Pino trace mixin and shared logger
When OTel tracing is active, every Pino log line now includes traceId,
spanId, and traceFlags fields for log-to-trace correlation. The mixin
is a no-op when no SDK is registered (community users).
* feat(tracing): add _otel to ToolJobData and inject trace context at enqueue
Add optional _otel carrier field to ToolJobData for W3C trace context
propagation across BullMQ job boundaries. When an active OTel span exists,
propagation.inject() writes traceparent/tracestate into the job data before
queue.add(). When no SDK is registered (community edition), the carrier
stays empty and _otel remains undefined -- zero overhead.
* feat(tracing): extract trace context and create spans in BullMQ worker
* feat(tracing): inject trace context into Python sidecar calls
* feat(tracing): add trace context extraction to Python sidecar
* feat(tracing): add shutdownTracing to graceful shutdown sequence
* feat(tracing): enrich HTTP spans with tool_id and user_id attributes
* docs: add OpenTelemetry env var documentation to .env.example
* test(tracing): add lifecycle integration tests for trace propagation
* fix(tracing): inject trace context into pipeline and batch flow jobs
* fix(tracing): add sidecar.execute Node-side span and remove unnecessary comment
Wraps PythonDispatcher.run() with a sidecar.execute span on the Node
side so traces show the full round-trip (Node span -> Python span).
Also removes an obvious comment from logger.ts.
Move user resolution logic (match by externalId, auto-link by email,
auto-create with user limit check) into a shared module that both
OIDC and SAML callbacks can use. Includes sanitizeUsername and
findUniqueUsername helpers. Preserves all existing OIDC behavior
and audit events.
Closes#71. Several error paths in the worker could leak internal
filesystem paths (/tmp/workspace, /data/ai/venv, /app) through SSE
frames, resultPayload objects, and Redis batch-error lists. The
existing stripInternalPaths call at worker.ts line 345 only covered
the single-file processToolJob catch block.
Wrapped 6 additional call sites:
- processPipelineStep: prevError from DB and catch errorMsg
- processPipelineFinalize: composed errorMsg reaching SSE,
recordChildOutcome, and resultPayload
- processBatchChild: catch error reaching recordChildOutcome
and resultPayload
- processBatchFinalize: manifest errorMsg from DB rows
- recordChildOutcome (batch-progress.ts): defense-in-depth
strip before Redis rpush
Added 10 unit tests for stripInternalPaths covering /tmp, /data,
/app, /opt, /home, /workspace, multi-path messages, safe passthrough,
and pipeline-step wrapping.
* feat(infra): add dev compose stack with postgres and redis
* fix(infra): comment dev env defaults until wired; harden dev compose restart and start_period
* chore(deps): add pg driver and testcontainers for postgres migration
* feat(db): translate schema to drizzle pg-core (timestamptz, boolean, pgEnum, jsonb)
Schema translation (apps/api/src/db/schema.ts):
- sqlite-core -> pg-core, all 10 tables preserved 1:1
- integer(mode:'timestamp') -> timestamp({ withTimezone: true })
- integer(mode:'boolean') -> boolean
- jobs.status text enum -> pgEnum('job_status') with same 4 values
- 7 columns changed from text to jsonb: jobs.inputFiles, jobs.settings,
pipelines.steps, apiKeys.permissions, roles.permissions,
auditLog.details, userFiles.toolChain
- settings.value stays text, jobs.error stays text, jobs.progress stays real
jsonb call-site sweep (removed JSON.stringify on writes, JSON.parse on reads):
- apps/api/src/routes/roles.ts: permissions read/write (3 sites)
- apps/api/src/routes/api-keys.ts: permissions write + read (2 sites)
- apps/api/src/routes/audit-log.ts: details read (1 site)
- apps/api/src/routes/pipeline.ts: steps write + read (2 sites)
- apps/api/src/routes/progress.ts: inputFiles write (2 sites)
- apps/api/src/routes/tool-factory.ts: toolChain read + write (2 sites)
- apps/api/src/routes/user-files.ts: toolChain read + write (4 sites)
- apps/api/src/permissions.ts: roles.permissions read (1 site)
- apps/api/src/lib/audit.ts: details write (1 site)
- apps/api/src/plugins/auth.ts: apiKeys.permissions read (1 site)
* refactor(db): type jsonb columns via $type and note raw CTE conversion requirements
* feat(db): archive sqlite migrations and generate postgres baseline
* chore(db): dockerignore legacy migrations, add archive breadcrumb, fix trailing newline
* feat(db): pg pool connection, advisory-locked boot migrations, DATABASE_URL config
* fix(db): friendly fatal on unreachable postgres, idempotent closeDb, lock-key convention note
* refactor(db): async drizzle calls in plugins, lib, permissions
* fix(api): analytics never throws, typed permission guard, single-query session invalidation
* refactor(db): async drizzle calls across all routes and bootstrap
Convert every route file and index.ts from sync SQLite drizzle
patterns to async node-postgres drizzle:
- .all() removed (bare await on select)
- .get() converted to destructured [row] = await ...
- .run() removed (bare await on insert/update/delete)
- .changes replaced with .rowCount (null-guarded) in progress.ts
- sqlite import removed from user-files.ts; raw CTEs converted to
await db.execute(sql`...`) with postgres-dialect recursive CTEs
- ChainRow types updated: tool_chain is parsed jsonb (string[] | null),
created_at is Date (timestamptz) with no * 1000 conversion
- All requirePermission() guard calls awaited (security: unawaited
async guard returns truthy Promise, bypassing permission check)
- All hasEffectivePermission() and getPermissions() calls awaited
- All auditLog() calls awaited (preserves write-before-response order)
- trackEvent() and captureException() left un-awaited (fire-and-forget
by design, guaranteed never-throw)
- ensureAnonymousUser(), startCleanupCron(), recoverStaleJobs() awaited
in bootstrap sequence
- ensureInstanceId() and ensureDefaultSettings() made async
Files converted: 14 (index.ts + 12 route files + tools/index.ts)
* fix(db): await async checkStorageQuota in user-files upload/save routes
* fix(db): await checkStorageQuota in save-result route (missed second call site)
* feat(db): sqlite-to-postgres migrator with CLI and first-boot import
* fix(db): migrator error context, honest force semantics, boot-hook fatal, null-variance tests
* test: run suite against per-file postgres databases via testcontainers
- Add tests/global-setup.ts: spins up a Postgres testcontainer,
creates a migrated template database once per vitest run.
- Rewrite tests/setup/per-fork-env.ts: each test file (forks pool)
clones the template into its own database via CREATE DATABASE ...
TEMPLATE, preserving the same per-file isolation granularity.
- Update vitest.config.ts: add globalSetup, pg alias, update comment.
- Fix tests/integration/test-server.ts: remove DB_PATH mkdir, async
runMigrations, async db operations, remove SQLite WAL checkpoint.
- Fix 21 unit test db/index mocks: add pool and closeDb exports.
- Fix 8 unit test files: add async/await for now-async permission,
audit, and analytics functions.
- Fix 18 integration test files: convert sync .run()/.all()/.get()
to async drizzle patterns, add async to callbacks.
- Production change: apps/api/src/routes/teams.ts: cast COUNT(*)
to ::int so Postgres returns a number instead of bigint string.
* fix(db): seed built-in roles, reject NUL bytes, cast COUNT, serialize job persists
- Seed built-in roles (admin, editor, user) at boot via ensureBuiltinRoles()
with onConflictDoNothing, restoring data that legacy SQLite migration 0007
provided via INSERT statements (the pg baseline is DDL-only).
- Reject NUL bytes in login credentials with 401 (postgres rejects \x00 in
text columns; valid usernames never contain NUL, matching 1.x behavior).
- Cast COUNT(*)::int in user-files, audit-log, and roles listing queries so
postgres returns a JS number instead of bigint-as-string.
- Serialize fire-and-forget job progress DB writes per jobId so the final
"completed" status is never overwritten by a late-arriving "processing"
write (race condition exposed by async postgres round-trips).
* test: fix teams race, seed roles in test server, poll for job status
- Add missing await to resetTeams() in teams PUT beforeEach (the async
delete raced with the subsequent insert under postgres).
- Call ensureBuiltinRoles() in test server bootstrap so integration tests
have the same built-in roles as production.
- Replace fixed 100ms flushPersist delay with a polling helper that waits
for terminal job status, eliminating timing-dependent failures caused by
postgres network round-trip latency.
* test: make heic temp-file cleanup assertion resilient to concurrent workers
Use a set-based diff instead of raw file count when checking that
decodeHeic cleans up temp files. Other concurrent test workers can
create heic-in-*/heic-out-* files in the shared tmpdir, inflating the
"after" count and causing spurious failures under full-suite load.
* fix(db): align builtin-role seed to post-0010 legacy state; test polish
* feat(docker): three-container compose (app, postgres, redis) with boot wait and migrations
* fix(docker): set TEST_DATABASE_URL so containerized tests skip testcontainers
* chore(docker): test compose project name, clearer 1.x upgrade comment, unref probe timer
* feat(enterprise): enforce D15 license boundary; move s3 storage into packages/enterprise
* fix(enterprise): restore lazy aws-sdk loading; community installs load no s3 code at boot
* fix(enterprise): boundary check catches dynamic imports; document getS3 concurrency
* feat(db)!: SnapOtter 2.0 phase 1 foundation: postgres, migrator, compose stack
BREAKING CHANGE: SQLite is no longer the runtime database. Deployments now
require Postgres (and Redis, used from phase 2). Existing installs migrate
with SQLITE_MIGRATE_PATH or 'pnpm --filter @snapotter/api migrate:sqlite'.
* fix(ci): postgres service + fresh e2e database per run; ignore unfixable torch CVE-2025-3000
