mirror of
https://github.com/snapotter-hq/SnapOtter.git
synced 2026-08-03 07:46:42 +02:00
e7ffb37e981c553cf17db88155ac12667f104062
204
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 SnapOtterandGitHub
|
43334324c4 |
fix(api): contain library stored-name path traversal (#600)
The library file-storage helpers joined FILES_STORAGE_PATH with a database stored_name and never checked containment, so a crafted name could read or delete files outside the storage root after a malicious 1.x SQLite import (which copies stored_name verbatim). Add assertSafeStoredName() and apply it in every helper that resolves a stored name to a path, matching the containment guard object-storage already uses. Reported by Alpesh Bhagwatkar. |
||
|
SnapOtterandGitHub
|
37c915c3da |
fix(jobs): make timeout messages tool-agnostic and CPU-aware (#596)
Replace the job timeout message that hardcoded "background-removal" for every tool with a tool-agnostic one that sets the CPU-vs-GPU expectation, the usual reason heavy AI times out on modest hardware. The client-side SSE stall message gets the same treatment. Both stay under friendlyError's 280-char limit so the guidance reaches the user instead of collapsing to a generic "Processing failed". Refs #591 |
||
|
SnapOtterandGitHub
|
bda4db3f35 |
fix(ocr): unblock and harden accurate-OCR install (#552)
Two OCR-install fixes surfaced while verifying the accurate-OCR (v3 runtime) path end to end: - Installer timeout must be a safe integer, not a performance.now() float. With the default INSTALL_MAX_MS this failed every accurate-OCR install via the app right after the download (masked by the unpublished runtime; CI drives install_runtime.py directly so it never surfaced). Fixed via remainingInstallerTimeoutMs(). - Classify an absent or forbidden runtime index (401/403/404/410) as OcrRuntimeNotPublishedError with a clear "Fast OCR still works" message instead of a raw HTTP 404, without retrying. Refs #552 |
||
|
SnapOtterandGitHub
|
86251434b5 |
feat(telemetry): Sentry + PostHog quality pass (#546)
Comprehensive telemetry quality improvements across Sentry and PostHog, grounded in an audit of the live data plus current best-practice research. Sentry: job_id/instance_id tags, operational fingerprinting, PII-safe settings context on bug events, web tag population + extension-noise filtering, an early-crash buffer, http status/method kept on breadcrumbs, and a gated-off-by-default performance-tracing re-enable (tracesSampler that zeroes db/redis/queue-poll root spans + drops the Redis integration) with worker job spans and canonical-host cron monitors. PostHog: history_change SPA pageviews, instance_id super property for fleet rollups, enriched tool_used (formats, byte sizes, is_batch, execution_hint, real error_kind taxonomy), the previously-dead result_saved/batch_processed/ai_bundle_prompted events fired, search click-through, editor + Automate authoring + auth instrumentation, a before_send PII boundary, and minimal opt-in landing-site pageviews. |
||
|
SnapOtterandGitHub
|
4ac89fe650 |
fix(api): wait for Postgres and Redis at startup instead of crash-looping (#537)
Briefly retry Postgres and Redis connectivity at boot (waitForService, DB_STARTUP_TIMEOUT_MS) so an ordered-but-not-yet-ready dependency recovers cleanly instead of crash-looping. |
||
|
SnapOtterandGitHub
|
281b4a06e3 |
feat(telemetry): add a safe input_format tag to worker error reports (#541)
Every worker tool error now carries an input_format tag (file extension only, never the filename) for triage, derived once at the worker error path and added to the scrubber allowlist. |
||
|
SnapOtterandGitHub
|
631d82eaae |
fix(telemetry): classify environmental database errors as operational (#540)
Postgres auth (28xxx), permission (42501), resource (class 53), and operator-intervention (class 57) failures now classify as operational via a cause-chain walk, not bug. pg query bugs (e.g. 42601) stay bugs. |
||
|
SnapOtterandGitHub
|
9cccbc9576 |
fix(image-tools): surface Sharp encode failures instead of "Error: Error" (#534)
Wrap convert and gif-tools process functions so a Sharp .toBuffer() failure carries an authored SafeError title (and the original as cause) rather than a scrubbed "Error: Error". |
||
|
SnapOtterandGitHub
|
991c981529 |
fix: make OCR portable and reliable across AMD64 and ARM64 (#519)
* fix: make OCR portable and reliable * fix: harden OCR installation portability * fix: pin OCR partials across downloads * fix: make OCR execution reliably asynchronous * fix: harden OCR portability and docs routes * fix: preserve decoder and docs safeguards |
||
|
SnapOtterandGitHub
|
e1b8c24e5d |
feat(analytics): instance census, full capture, richer error context (#511)
Add a once-per-boot instance_started event (arch, os, deploy_mode, gpu_present) so the fleet architecture mix is measurable. It reuses the existing per-instance instance_id and is exempt from the volume sample rate, since a census that fires once per boot must not be thinned. Restore useful capture depth now that the sponsored plan removes the quota pressure behind the earlier hardening: - PostHog sample rate 0.1 to 1.0 (full analytics when enabled); the property allowlist still blocks file data. - Sentry per-instance ceiling 20 to 500/hr, breadcrumb trail restored (sanitized: urls/paths redacted, data payloads dropped), full stack paths kept; local vars, request bodies, and PII still dropped. Both api and web. Honor ANALYTICS_ENABLED=false as an opt-out alias: it was documented on the Docker Hub README but never wired in 2.x, so anyone who set it was still tracked. All capture stays behind the analytics opt-out gate. |
||
|
SnapOtterandGitHub
|
cb5db59f77 |
feat(tools): remove background from animated GIFs (WebP, APNG) (#502)
Adds a dedicated remove-gif-background AI tool: removes the background from an animated GIF, WebP, or APNG frame by frame and reassembles a transparent (or composited) animation in WebP, APNG, or GIF, with full per-frame effects. Reuses the background-removal bundle. Verified end-to-end with the real rembg model. Closes #496. |
||
|
SnapOtterandGitHub
|
b457596649 |
fix(telemetry): sharpen Sentry signal for v2.1.0 residual defects (#498)
Follow-ups to the v2.1.0 Sentry telemetry overhaul, found by reviewing live release:2.1.0 events: - error_code tag was empty because reportError read only the top-level err.code; add extractErrorCode() to walk the cause chain (pg SQLSTATE, node E-code, else first short code). - InputValidationError from a tool's processV2 in the worker was logged as error_class=bug; classify it as expected for any source. Worker-side ZodError stays a bug (schema drift). - AI dispatcher timeouts rejected with a bare Error, which the sanitizer scrubbed to a message-less "Error: Error"; reject with an operational SafeError (code "timeout") at both timeout sites. Each fix written failing-test-first; affected and adjacent unit suites green plus full CI (integration + e2e). |
||
|
SnapOtterandGitHub
|
ae6a4c8b7c |
fix: error-only Sentry telemetry, storm-proof capture, and crash fixes (#476)
Removes Sentry tracing entirely (BullMQ idle polling burned 4.8M transactions in 2 days at the baked 0.1 rate), decouples PostHog sampling, and replaces the type-only error scrub with a vetted-field sanitizer plus SafeError/ToolInputError contracts. One classified capture path with per-signature throttles and a per-process ceiling makes storms impossible (NODE-1E was 4,541 events from one 30s loop). Browser errors move to a dedicated web Sentry project with their own source maps. Adds the SNAPOTTER_TELEMETRY runtime kill switch and silences test fleets. Crash fixes: remote 204/304 SSRF process kill (NODE-20), conversion-preset boot crash loop (NODE-21), Redis version preflight + unhandled subscribe rejection (NODE-1T), Sign PDF on plain-http origins (NODE-1K/1M), wavesurfer/pdf.js teardown rejections (NODE-1P/1N), bundle-import ZlibError to 400 (NODE-1Z), chart-maker input errors declassified (NODE-1H/1J), asset requests skip the session DB lookup (NODE-1D). |
||
|
SnapOtterandGitHub
|
a731c3d1fe |
fix: reliable, self-healing AI feature-bundle installs (#472)
Make on-demand AI feature-bundle installs reliable and self-healing, closing the failure modes behind most "some tool doesn't work" reports. Multi-bundle installs: tools needing more than one bundle (Passport Photo, Enhance Faces) install every required bundle from one action and stay not-installed until all are present. Verified across all 19 AI tools. Downloads: self-heal the accelerated Hugging Face (Xet) client so an upgraded venv no longer silently falls back to slow urllib; restart instead of corrupting a resumed partial when a proxy ignores Range and returns 200; verify the completed size; fail fast on disk-full and HTTP 4xx; retry transient errors five times; add hf_transfer fallback and document Xet egress. Install integrity: crash-atomic venv writes so a killed or out-of-space install can no longer tear the shared venv and break other tools; a boot breadcrumb reseeds a torn venv to a clean state automatically; a post-install smoke import test refuses to record a bundle whose libraries cannot load; an install watchdog stops a wedged installer that would otherwise hold the venv writer lock forever. Adds unit and end-to-end tests for every failure mode above. |
||
|
SnapOtterandGitHub
|
fb96cf8743 |
feat(ai): add a Reset AI Environment admin feature for the upgrade gap (#459)
Uninstalling a bundle only deletes its downloaded model weights, never the
shared venv's site-packages, so self-hosters who already hit an AI bundle
conflict (e.g. the scipy ABI strand) have no clean self-service path via
uninstall+reinstall: reinstalling just overlays corrected files on top of
stale ones. Adds POST /api/v1/admin/features/reset, which wipes
/data/ai/{venv,models,pip-cache}, resets installed.json, and reseeds a real
working venv from the image's baked /opt/venv (extracted docker/reseed-ai-venv.sh,
now shared with entrypoint.sh's existing base-venv-upgrade bootstrap instead
of duplicating that logic) -- leaving an empty venv directory here would
make the very next install fail with "spawn .../python3 ENOENT", caught by
testing this live rather than assuming it. Ships with a matching Settings UI
section (inline confirm, same pattern as per-bundle uninstall) and strings
across all 21 locales.
Verified against a real snapotter/snapotter:1.17.2 image migrated to 2.0.0,
with real multi-GB bundles installed (background-removal + OCR): confirmed
the migrated instance's inherited python3.11 venv (2.0.0 itself uses 3.12)
still imports the fixed scipy/numpy/paddleocr correctly, then reset + real
reinstall + actual tool execution (remove-background, verified output image)
all worked end-to-end.
|
||
|
SnapOtterandGitHub
|
60d01ab2dd |
fix: release-acceptance QA follow-ups (upload crash, scipy ABI conflict, rate limit, OCR fallback) (#458)
* fix(api): prevent a crash when an over-limit upload stream has no consumer yet busboy's "limit" handler destroyed the file stream with an error but never attached its own error listener, relying entirely on whatever consumes part.file downstream to do so. On a fast enough connection (or a fully buffered body, e.g. Fastify inject()), busboy can process enough bytes to hit the size limit before the route handler's receiveUpload() call has attached its own stream listener, leaving the resulting "error" event with zero listeners -- which crashes the whole process by default in Node. Surfaced by tonight's FULL_MATRIX+FUZZ integration run (880 uncaught exceptions, all the same root cause). Reproduces deterministically in isolation; unrelated to this release's actual code delta (file untouched since PR #413, well before the baseline QA pass). Fix: attach a baseline no-op error listener the moment the stream is created, guaranteeing at least one listener always exists. EventEmitter delivers "error" to every registered listener, so the real consumer's own error handling is unaffected. * fix(ai-bundles): rebuild upscale-enhance and photo-restoration to reconcile scipy ABI upscale-enhance and photo-restoration both depend on codeformer-pip, whose transitive closure (basicsr -> realesrgan -> gfpgan) pulls in an unpinned scipy. Both bundles were last built ~June 18-19, before PR #437 added the manifest's `constraints` array (numpy==1.26.4, scipy==1.12.0, etc.) to pin exactly this kind of dependency during bundle builds. Only the ocr bundle was rebuilt after that fix landed. install_feature.py has no pip install step -- it's a raw tarfile extraction with no cross-bundle conflict resolution, so installing OCR alongside either stale bundle left three incompatible scipy versions' files mixed in the same site-packages directory (a compiled _rotation.*.so from one release next to Python files expecting a different release's API), breaking the `upscale` tool and OCR's higher-quality tiers with an ImportError. Rebuilt both bundles for amd64-gpu and arm64-cpu from the current manifest, verified scipy/scikit-learn/scikit-image/pandas all resolve to the pinned versions in the tarballs themselves, then verified end-to-end on real hardware (Mac arm64 CPU and ubuntu_gpu .248 RTX 4070): installing all affected bundles together now yields exactly one version of each constrained package, `upscale` produces correct output, and OCR's balanced/best tiers correctly use PaddleOCR-GPU instead of erroring out. Published the rebuilt tarballs to the public deepsafe/feature-bundles HuggingFace repo and updated this manifest's sha256/sizes to match. Also adds verify-bundle-compatibility.sh: verify-bundle.sh checks each bundle in isolation (a fresh venv per bundle), which is exactly why this shipped twice -- nothing ever checked that bundles built at different times agree once layered into the one shared venv real installs use. The new script installs every bundle for an arch into one venv and asserts each constrained package has exactly one, correct version. Known follow-up (not fixed here, needs separate discussion): uninstalling a bundle only removes its downloaded model weights, never the site-packages it added, so existing installations that already hit this bug have no clean self-service fix via uninstall+reinstall -- they need a full AI-venv wipe. * fix(docker): bake a real rate limit default for the all-in-one one-liner The documented single-container `docker run` install had RATE_LIMIT_PER_MIN=0 (effectively unlimited, ~50k/min) baked in, since only docker-compose.yml carried a hardened override. A self-hoster following the one-liner path got no meaningful throttling anywhere, including auth-adjacent routes with no dedicated per-route limit. Bakes a generous-but-real 1000/min default into the Dockerfile, raises both compose files' fallback to match so the two documented install paths converge on the same posture, and updates the Zod schema default plus docs that quoted the old value. * fix(api): boot log undercounted tool routes by the conversion-preset total The "Tool routes: N active" line logged before registerConversionPresets(app) ran, so it only ever reported the base 158 tools, 83 short of the real 241-tool total. Presets have to register after the base loop (they delegate to each base tool's own processV2), so the fix moves the log line to after that call and has registerConversionPresets return its count instead of reordering the dependency. * fix(ai): forward {info}/{warning} stderr JSON instead of dropping it The dispatcher stderr parser only recognized {ready} and {progress,stage} shaped JSON lines; anything else that parsed as valid JSON (like ocr.py's GPU-to-tesseract downgrade notice, an {"info": ...} line) matched neither branch and fell through silently, never reaching docker logs. Adds explicit {info}/{warning} handling that forwards to console.log/console.warn, same as the existing [prefix]-tagged non-JSON path. * fix(api): fall back to a lower OCR tier when PaddleOCR itself is unusable ocr.ts already retries lower quality tiers on a crashed dispatcher, but the condition only matched crash-style messages (segfault, exited unexpectedly). ocr.py's own ImportError/exception handlers already produce messages telling the caller to use a lower tier (e.g. on the scipy ABI conflict class of bug), but nothing ever acted on them, so a broken PaddleOCR hard-failed with 422 instead of degrading to Tesseract like ocr-pdf effectively does. Broadens the retry condition to also catch PaddleOCR-engine-unusable messages. Note: ocr-pdf's tesseract-only behavior turned out to be an unrelated, pre-existing, deliberate design choice (PaddleOCR segfaults on rasterized PDF pages on arm64), not a graceful-fallback mechanism to copy -- the two tools weren't actually solving the same problem, so this fixes ocr.ts's own gap rather than trying to mirror ocr-pdf. |
||
|
SnapOtterandGitHub
|
dadf766899 |
fix(migrator): correct and harden the 1.x to 2.0 SQLite import (#434)
* feat(api): parse DATA_DIR from env for 1.x import auto-detection Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * test(migrator): build 1.17.2 fixtures by replaying legacy migrations Discovered the legacy migrations seed a Default team (0005) and builtin roles (0007), so the replayed fixture carries them. Seed uses a distinct custom team. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * fix(migrator): self-adjusting column copy, jobs.status map, drop sessions, advisory lock The importer now inserts only the intersection of source and live target columns, so the three analytics_* columns 2.x dropped no longer break the first users INSERT (and future dropped columns are handled generically). jobs.status is mapped onto the 2.x enum (error->failed). Sessions are no longer migrated. A pg_advisory_xact_lock serializes concurrent replicas. Includes login-after-migrate and library assertions. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * test(migrator): CI drift guard fails when a required column is unfillable from 1.17.2 Introspects every NOT-NULL-no-default column of each migrated table in the current schema and asserts the engine can fill it from a real 1.17.2 source. Turns a future breaking schema change into a PR-time failure instead of a production import break. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * feat(migrator): orchestrator with detection, boot states, marker, blob count sqlite-import.ts owns source resolution (explicit path, 'off' sentinel, DATA_DIR probe), the four boot states (import/leftover/locked/none), the persisted sqlite_import marker, and a read-only library-blob count. runBootImport wires them together and catches TargetNonEmptyError as a benign multi-replica skip. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * feat(api): route boot through the 1.x import orchestrator; hide marker from non-admins index.ts now calls runBootImport (which owns detection + the four boot states) instead of the inline SQLITE_MIGRATE_PATH block. The sqlite_import marker is added to SENSITIVE_KEYS (but not REDACTED_KEYS) so admins see the counts for the banner while non-admins don't see the key at all. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * feat(migrator): add analyzeSqlite + dry-run/verify CLI analyzeSqlite is a read-only pre-flight (no live Postgres): per-table row counts, library-blob presence, and out-of-enum job statuses. The migrate:sqlite CLI now lives in the orchestrator and supports --dry-run/--verify (prints the analysis and exits without writing) alongside the existing import and --force. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * docs: add 1.x to 2.0 upgrade guide; fix volume-name casing New apps/docs upgrade guide covering auto-detect, the SQLITE_MIGRATE_PATH override + off opt-out, the dry-run, what carries over, locked-state recovery, and non-destructive rollback. Leads with 'back up the WHOLE /data volume, not just snapotter.db' because 1.x WAL mode leaves data in snapotter.db-wal (surfaced by the real-image upgrade test). Standardizes README/DOCKERHUB compose volume names on the canonical SnapOtter-data casing so they match the repo compose and don't orphan an upgrader's volume. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * feat(web): admin 1.x migration banner + 21-locale strings A one-time admin banner reads the sqlite_import marker from /v1/settings and shows the import result (user + saved-file counts) on success, or a warning when a 1.x database was found but not imported. Dismissal persists to a sqlite_import.dismissedAt settings key. shouldShowMigrationBanner/parseMigrationMarker sit in feedback.ts with the other shouldShow helpers; strings added to en.ts and all 20 other locales. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w * style(landing): biome-format Hero.astro trustBadges array Pre-existing formatting drift on main (its Lint check was skipped on the merge that introduced it); this PR's full Lint run surfaced it. Formatting-only, applied via the repo's own biome formatter to unblock the required Lint check. Claude-Session: https://claude.ai/code/session_01721WHAUGxnVk22qEeTub7w |
||
|
SnapOtterandGitHub
|
6e3a14ec6b |
fix: remove automatic third-party egress of user data + optional strict offline mode (OSM tiles, Scalar fonts, editor fonts, AI model downloads) (#422)
* fix: remove all automatic third-party egress (OSM tiles, Scalar fonts, editor Google Fonts, AI model download fallbacks) Phone-home audit follow-up. The product no longer makes any automatic third-party request; user-initiated click-outs stay, and production now fails closed on missing AI models. 1. GPS leak via OSM tiles: the strip-metadata panel auto-loaded tile.openstreetmap.org tiles encoding the photo's GPS position. The Leaflet mini-map is gone; coordinates render as text plus an explicit View on map link (openstreetmap.org, opens on click only). Removed tile.openstreetmap.org from the CSP img-src, dropped the leaflet dependency, added the viewOnMap i18n key to all 21 locales. 2. Scalar docs fonts: /api/docs loaded Inter and JetBrains Mono from fonts.scalar.com. Scalar now renders with withDefaultFonts: false and both --scalar-font and --scalar-font-code pinned to system stacks; fonts.scalar.com removed from the docs CSP font-src. Verified by injecting GET /api/docs/: config carries withDefaultFonts false and the served page has no fonts.scalar.com reference. 3. Editor Google Fonts: the editor font picker built fonts.googleapis.com stylesheet URLs for 25 web fonts the served CSP already blocked. The remote loading path is deleted; the picker now offers system fonts only, with a SELF_HOSTED_FONTS seam (FontFace API, same origin) for bundling fonts later. Unknown families saved in old documents fall back to the browser default. 4. Python sidecar fails closed on model downloads: new packages/ai/python/offline_guard.py gates every runtime download fallback (inpaint, outpaint, restore, noise_removal, detect_faces, enhance_faces, face_landmarks, red_eye_removal, remove_bg, ocr, transcribe, upscale) behind SNAPOTTER_ALLOW_MODEL_DOWNLOAD=1 with an actionable error. Bundled models keep working untouched. 5. OCR and transcription library-internal downloads: unbundled PaddleOCR language and detection fallbacks now raise the guard error naming the language instead of resolving models over the network; faster-whisper gets local_files_only when downloads are off. 6. GFPGAN and CodeFormer cwd-relative weights: facexlib and codeformer-pip resolve helper weights relative to the process cwd and fetch them from GitHub when absent. They are now symlinked from the installed bundle files under MODELS_PATH/gfpgan/facelib before the libraries load, failing closed when unresolvable. Defense in depth: HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 are set in the runtime image and in the sidecar spawn env; install_feature.py lifts them for user-initiated bundle installs and restores them afterwards (it can run in-process inside the dispatcher). SNAPOTTER_ALLOW_MODEL_DOWNLOAD is documented in .env.example, default off. Validation: typecheck 9/9 workspaces, Biome clean on touched files, 5178 unit tests pass, py_compile on all touched scripts, guard behavior exercised in both dispatcher exec and per-request import modes, zero remaining runtime references to the three hosts. Docker build and live AI inference need post-merge verification on the GPU host. Claude-Session: https://claude.ai/code/session_01XGB4pGvTvb7sUX4JN745U7 * fix: allow AI model downloads by default, make strict offline mode opt-in Product call: ease of use first. The download gating from the previous commit inverts its default: runtime model fetches (public model weights only, never user data) are allowed out of the box so AI tools self-heal, and SNAPOTTER_ALLOW_MODEL_DOWNLOAD=0 becomes the explicit strict offline mode for airgapped deployments, where every fallback raises the actionable error instead of fetching. Changes: offline_guard blocks only on an explicit 0/false; the unconditional HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE image ENV is removed and bridge.ts sets those flags for the sidecar only in strict mode; .env.example documents the new default; install_feature's lift/restore stays. All bundled-path preferences, pre-existence checks, and symlink pre-placement remain, so installed bundles never trigger a download. The OSM, Scalar font, and editor font fixes are unchanged. Validation rerun: typecheck 9/9, Biome clean on touched files, 5178 unit tests pass, py_compile on touched scripts, guard behavior verified for unset/1 (allowed) and 0/false (blocked with the new message). Claude-Session: https://claude.ai/code/session_01XGB4pGvTvb7sUX4JN745U7 |
||
|
SnapOtterandGitHub
|
8b3f1e6884 |
fix: stamp SnapOtter as Producer on generated PDFs (#416)
Conversion engines wrote their own names into PDF metadata: LibreOffice, Ghostscript, pdfcpu, WeasyPrint, and PDFKit all stamped Producer/Creator on generated files. A new doc_scrub_meta docs-profile script (PyMuPDF) rewrites both fields to SnapOtter and drops the stale XMP copy; the worker applies it to the 25 PDF-generating tools before outputs reach object storage. Best effort by design: any failure keeps the original bytes and only logs a warning. Deliberately untouched: tools that edit the user's own PDF and preserve its metadata (qpdf edits, sign, flatten), encrypted outputs (copied through), and pdfa-convert, where a metadata rewrite risks PDF/A conformance. Claude-Session: https://claude.ai/code/session_01XGB4pGvTvb7sUX4JN745U7 |
||
|
SnapOtterandGitHub
|
bf417a509e |
fix: first-run QA sweep of the single-container image (#413)
Fixes found by manually testing a fresh install end to end: - auth: the must-change-password gate returned 403 on public routes including /api/v1/health, so every fresh install showed a false "Reconnecting to server" banner on the forced password change screen. Public routes are now exempt (they need no session at all). Adds the gate's first direct tests. - multipart: @fastify/multipart's parts() iterator (9.4.0 and 10.0.0) ends on the request stream's "close", which on a reused keep-alive connection fires while an earlier part is still streaming to storage, silently dropping the parts behind it. The object eraser lost its mask file on every second POST per connection. Replaced with a busboy-driven iterator (lib/multipart-parts.ts) that ends on busboy's own "finish", installed for all routes via a preValidation hook; the tool-factory field-recovery workaround for the same bug is now unnecessary and removed. - eraser: the mask canvas backing store is natural resolution, but "absolute inset-0" does not stretch replaced elements, so the canvas rendered at intrinsic size and the brush ring, strokes, and exported mask were all misscaled on photos larger than the viewport. The canvas now gets an explicit CSS box at the fitted size. - compare slider: solid white divider with a dark halo so it stays visible over light images; still initialised at the painted region. - tool page: the AI bundle install prompt now centers in the content area instead of hugging the top. - api docs: disabled Scalar's cloud features (Ask AI, Generate MCP, Open API Client, dev toolbar), hid the "Powered by Scalar" footer link, and set the page title to "SnapOtter API Reference". The docs CSP blocks those cloud calls by design, so the buttons were dead UI. - docker: embedded Redis comes from packages.redis.io pinned to the 8.x major (was Debian's 7.0.15), matching the Compose stack and the documented claim. Build fails fast if the major ever drifts. - docs: DOCKERHUB.md quick start now leads with the one-command docker run (matching the README) with Compose as the production path; README says embedded Postgres 17 + Redis 8. Claude-Session: https://claude.ai/code/session_01XGB4pGvTvb7sUX4JN745U7 |
||
|
SnapOtterandGitHub
|
b4375e558d |
fix: harden install queue/dispatcher lifecycle and repair review-sweep regressions (#395)
Fixes 15 defects found by a max-effort multi-agent review of the last 6 merged PRs (#388, #390, #391, #392, #393, #394), all adversarially verified before fixing. Install queue + dispatcher (the serious cluster): - features.ts: finalize the installer child exactly once. A failed spawn fires both "error" and "close", and the second event released the file lock and active slot that pump() had just handed to the next queued bundle, letting two pip processes write the same venv concurrently. Outcome recording now happens before pump() so the next bundle's first progress frame cannot race the previous install's bookkeeping. - feature-status.ts: keep failed-install errors in a per-bundle map instead of the single progress slot. With the queue auto-starting the next install, the slot was overwritten within seconds and a failed install vanished without ever surfacing to GET /features. - bridge.ts: scope child lifecycle per process (stopped-children set + request generation tags) instead of an instance-wide shuttingDown flag that the next spawn reset. A stale SIGTERMed child's late close event could record a phantom crash (5 of which permanently disable the dispatcher), null out the freshly spawned child, and reject the new child's pending requests. The request-timeout kill path still counts as a real crash. - install_feature.py: the pre-write disk re-check measured ai_dir's filesystem even when budgeting the cross-filesystem copy that lands on the venv's disk; now each budget is checked against the filesystem the bytes actually land on, so ENOSPC cannot strike mid-write and leave site-packages half overwritten. Behavior regressions: - embed-subtitles: preserve pre-existing subtitle tracks (0:s?) and MKV attachments (0:t?) that the -map 0:v:0/0:a? rewrite silently dropped; data streams stay unmapped on purpose (the actual MPEG remux fix). The new subtitle maps first so the language tag hits the right stream. - usage-survey-overlay: fail closed when the settings fetch fails; the fail-open path rendered the blocking survey against an unhealthy API and soft-locked admins, the lock-out class #392 fixed. - features-store: queued bundles poll instead of each holding an SSE connection (Install All could pin 7 EventSources and exhaust the browser's 6-per-origin HTTP/1.1 limit, hanging the whole app); listenToProgress closes any prior stream and stops any poll before subscribing; installAll skips bundles already installing or queued. Contracts, tests, i18n: - openapi.yaml: add "queued" to the features status enum and document downloadBytes/installedBytes (Schemathesis conformance). - feature-lifecycle e2e: queue transcription (~0.5 GB) instead of ocr (~6 GB) and give the test a budget that covers both install drains (the stacked waits exceeded the old 900s timeout). - docker-compose.qa.yml: parameterize the host port (QA_APP_PORT) so QA_PROJECT_NAME concurrent stacks can actually bind. - compare + watermark-image: restore per-input error attribution ("Invalid first/second image", "Invalid watermark image") lost in the shared-handler migration. - ai-features-section: the "{size} on disk" suffix now goes through i18n; key added to all 21 locales. - watermark-image + content-aware-resize: migrate to the shared inputHandlerFor("image") chain like compare/vectorize/compose, fixing drift in the inline copies (no SVG sanitize, no RAW extension hint, no AVIF probe). Verified: typecheck across 9 workspaces, Biome clean on all changed files, 584 targeted unit tests and 249 integration tests green (including real-ffmpeg embed-subtitles runs). One unit test updated to the new poll-while-queued contract with a single-EventSource assertion. Claude-Session: https://claude.ai/code/session_017mR1HiHaf3a1BmUtrHX4j3 |
||
|
SnapOtterandGitHub
|
b37faed95f |
fix: QA sweep - tool routes, security, i18n, a11y, + AI bundle install hardening (#393)
* fix(api): correct format/filename/container handling across tool routes Found during a comprehensive QA sweep exercising every tool against its full accepted-format matrix: - watermark-image, compose: preserve the requested output format and a matching download filename/extension instead of always emitting the source format - compose: crop oversized overlays to the visible base area instead of crashing Sharp's composite, and reject only overlays fully outside the base image instead of any oversized one - compare, vectorize: switch to the shared image input handler so filenames and formats like .svgz/.tga/RAW survive validation instead of being rejected pre-processing - tool-factory, images-to-video: normalize frames through Sharp before handing them to FFmpeg, fixing GIF/AVIF/RAW image-to-video jobs that previously failed or hung - media-tool, replace-audio, embed-subtitles: fix legacy container MIME/codec handling for MPEG sources and subtitle remux cases - files: expand download MIME mapping for text/data/document/video/audio outputs that were falling back to a generic content type - convert-document/presentation/spreadsheet: same-format conversions now return the original validated file instead of erroring or producing corrupt tiny output Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG * fix(web): dropzone a11y, stale localStorage getter, dead code - dropzone: stop making the whole drop-zone section clickable/focusable. A section acting as an interactive element around a real upload button is a nested-interactive-element anti-pattern that confuses screen readers; drag-and-drop doesn't need focus semantics, only the button fallback does. Keeps that button semantic and keyboard-reachable. Updates the two e2e call sites that clicked the section directly. - api, use-auth: read through window.localStorage via the existing API storage helper instead of the bare global, which resolves to Node's experimental localStorage getter under Vitest and threw - find-duplicates-settings, info-settings, login-page: remove dead code (unused zip-download handler, a stale mount-only effect dependency that left cached info stuck at reused indices, an unused response variable) Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG * fix(i18n): pt-BR, zh-CN, zh-TW were silently falling back to English The locale loader looked up dynamic-import exports by the raw locale code (mod["pt-BR"], mod["zh-CN"], mod["zh-TW"]), but those three modules export camelCased bindings (ptBR, zhCN, zhTW) since identifiers can't contain hyphens. The lookup returned undefined and every consumer silently fell back to English for these three locales. Replaces the generic lookup with explicit per-locale loaders so the mapping can't drift out of sync again. Also updates the dropzone helper copy across all 21 locales to match the drag-only dropzone wording from the previous commit. Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG * fix(docs): clear build warnings in the VitePress site - config.mts: add an onwarn handler for the @vueuse INVALID_ANNOTATION warnings emitted during the docs build - deployment.md: the caddyfile code fence language isn't a shiki grammar VitePress ships with, so it warned on every build; use txt instead Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG * test(qa): update QA harness for the drag-only dropzone and regen metadata - api-sweep, qa-helpers, verify-ai: add JSON-body tools, multi-input secondary fixtures, async polling for slow valid jobs, 501 FEATURE_NOT_INSTALLED skip handling, and safer per-tool settings - input-preview, pipeline-ui specs: update upload flow for the drag-only dropzone surface - add tests/fixtures/data/valid/chart.json, a valid chart fixture the updated helpers route to - regenerate tools-meta.json against current TOOLS[] Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG * fix(security): close a login timing side-channel, harden zip-slip tests Found during a black-box security sweep of the real auth-enabled production container: a nonexistent username returned 401 in ~3-10ms, while a wrong password for a real user took ~35-42ms, because scrypt verification only ran when a user row existed. That timing gap lets an attacker enumerate valid usernames without ever guessing a password. Now runs verification against a cached dummy hash on the unknown-user path too, so both cases cost the same regardless of outcome. extract-zip already had a relative-traversal regression test (../evil.txt), but its absolute-path rejection branches (name.startsWith("/") / startsWith("\\")) had none. Added the three missing cases: deep relative traversal, absolute Unix path, and Windows-style absolute path. Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG * test(qa): add UI-driven AI bundle install scripts QA_PROMPT.md's Phase 2 requires installing AI models the way a user does -- through the UI, on demand from HuggingFace -- and treats the curl-based admin install endpoint as fallback-only. Nothing in the harness actually drove that flow; tests/qa/seed-ai-models.sh installs via docker exec + pip, which is further from a real user than even the API fallback. install-ai-bundles-ui.mts logs in, opens Settings > AI Features, screenshots the pre-install state, clicks Install All, and screenshots progress -- then exits, since installs continue server-side once triggered. verify-ai-install-complete.mts polls bundle status, screenshots the completed state, and runs one real tool per installed bundle to prove the freshly-downloaded model actually executes. Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG * fix(qa): correct the apiToolPath import in the AI verify script Dynamic import of the package name failed under tsx's module resolution from apps/api's node_modules context; use the same relative-path import api-sweep.mts already uses successfully. Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG * fix(web): correct AI bundle size estimates shown before install Measured real downloads during GPU-node QA verification: photo-restoration pulls ~4.4GB (was advertised as 800MB-1GB, off by 4-5x) and ocr pulls ~5.5GB (was advertised as 3-4GB). Both estimates only accounted for model weights, not the pip dependencies (torch/paddle) that come down with them. Updated to reflect actual total download size, since that's what a user deciding whether they have the disk/bandwidth actually needs to know. Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG * fix(web): make desktop Settings reachable when auth is disabled AvatarDropdown (the only desktop entry point to Settings) was gated behind `!isMobile && authEnabled`. With AUTH_ENABLED=false the synthetic anonymous admin user should have full Settings access per how auth.ts documents this mode -- and the mobile bottom nav already worked this way, showing Settings unconditionally. Desktop just had a stray extra gate the component doesn't need: AvatarDropdown already resolves its own username internally (falling back to "admin") and reads authEnabled itself where it actually matters (hiding the Logout button). Removed the outer gate; verified end-to-end against a fresh AUTH_ENABLED=false instance -- avatar now renders, Settings opens, shows the anonymous/Admin identity correctly. Also documents (not changes) a related finding in install_feature.py: detect_arch() always resolves amd64 hosts to the GPU-bundled archive variant regardless of actual GPU presence, since no CPU-only amd64 archive is published to the bundle repo yet. Left as a code comment rather than a behavior change, since requesting an unpublished archive key would hard-fail installs entirely -- worse than the current oversized-but-working download. Full detail in the QA report. Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG * fix(ai): stop logging expected dispatcher reloads as crashes After each AI bundle install the Python dispatcher reloads because the venv changed, and after every app shutdown it's SIGTERMed. Both took the close handler's `code !== 0` branch (SIGTERM makes the exit code null), so they were counted as crashes -- producing an alarming "crash" line in the logs and a pointless ~1s recovery backoff after each of 7 installs. A `stopping` flag set in shutdown() lets the close handler tell an intentional stop apart from a real crash. The request-timeout kill path deliberately does not set it, so a genuinely hung script still records a crash and the 5-in-60s permanent-disable threshold is untouched. Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG * fix(api): return a clean message when content-aware resize times out Carving a very high-resolution image down to a tiny target could exceed the caire subprocess timeout, and the raw error forwarded to the user was caire's terminal output -- ANSI color codes and progress-spinner control characters -- instead of anything actionable. Now: the timeout path throws a clear "timed out; try a smaller image or larger target" message (keeping the raw stderr as `cause` for server logs); friendlyError() strips ANSI/control chars centrally so any subprocess dump surfaced through the shared sanitizer is plain text; and the content-aware-resize route (a custom route that bypassed the sanitizer) now routes its error paths through friendlyError like every other tool. Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG * fix(ai): stop bundle installs from exhausting host disk Installing an AI bundle on a tight-disk host could push the root filesystem to zero bytes free after the preflight check had already passed. Two root causes: - move_tree used copytree+rmtree, so during the move the extracted payload existed in both staging and the venv at once -- a full transient doubling on disk. Rewrote it to rename entries (a cheap metadata op on the same filesystem, no copy), falling back to a copy only across filesystems. - the preflight budget used the manifest's extractedSize verbatim, which is 0 for several archives, collapsing the estimate to just the compressed size. Added a conservative fallback (3x compressed) so a missing value can't under-reserve. Also added a real-on-disk re-check immediately before the first destructive venv write (measuring the actual extracted payload and whether the move needs extra space for a cross-filesystem copy), which also now covers the offline-import path that previously skipped the disk check entirely; wrapped the moves so an out-of-space failure returns a clean actionable error instead of a traceback; and made the disk check resolve the nearest existing ancestor so it never throws on a not-yet-created venv path. Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG * feat(web): show the real per-arch AI bundle download size The bundle cards and install prompt showed a hardcoded, architecture-blind estimatedSize string. That's misleading: amd64 hosts always pull the CUDA-inclusive archive (there's no CPU-only amd64 variant published), so a bundle labelled "1-2 GB" can actually download several times that, while arm64 pulls a much smaller archive for the same label. The manifest already carries the real per-arch compressedSize (and extractedSize where measured), so surface those: a new optional downloadBytes/installedBytes on FeatureBundleState, populated in getFeatureStates() for this host's arch (resolver mirrors install_feature.py detect_arch), shown by the UI when present with estimatedSize kept as the fallback label. Also nudged upscale-enhance's fallback string (4-5 -> 5-6 GB) to match its real compressed size, consistent with the earlier photo-restoration/ocr fixes. Fields are optional so demo/mock and existing tests stay compiling; the manifest's extractedSize is 0 for a few archives, which now surfaces as null rather than a bogus 0. Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG * fix(web): move the AI install queue to the server so it survives tab close Installing multiple bundles could silently lose all but the first. The server rejected a concurrent install with 409, so the client worked around it by queueing the rest in browser-local state and only POSTing each once it saw the previous finish. A single POSTed install is durable (the installer child is detached from the request), but a queued one had zero server footprint -- close the tab mid-queue and those installs vanished with no error, while the UI still showed them "Queued". The client "mutex" didn't even serialize: the queued bundles' local waits all resolved at once and raced into concurrent POSTs that 409'd each other. Now the queue lives on the server (a small in-memory FIFO leaf module). The install endpoint enqueues instead of 409-ing and returns 202 {jobId, queued}; a pump starts the next bundle when the current one's child exits (and after an offline import releases the lock), all behind the existing venv + file locks, which are unchanged. The client just POSTs every bundle immediately and reflects the server-reported queued/installing status; Install All fires all POSTs and lets the server serialize them, keeping the one-shot retry-on-failure. Adds "queued" to FeatureStatus (the bundle card already rendered that state) and surfaces it from getFeatureStates. In-memory is deliberate: it matches the existing contract (survives a tab close, not a server restart, which already clears the lock on boot). Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG * fix(qa): don't log env-derived credentials in the AI-install script CodeQL flagged clear-text logging of sensitive information: the login status line interpolated the QA base URL and username (both read from the process environment) into a console.log. Replaced with a static message. QA helper only, but it's a real hygiene issue and cleared the high-severity code-scanning alert on the PR. Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG |
||
|
SnapOtterandGitHub
|
ca076f91fd |
fix: critical first-login soft-lock in usage survey overlay (#392)
* fix: prevent UsageSurveyOverlay from soft-locking the first-login password-change flow Claude-Session: https://claude.ai/code/session_01KAC9Lbx8AmebAnj9WQZXHp * fix: prevent double feedback submission when the settings write fails Claude-Session: https://claude.ai/code/session_01KAC9Lbx8AmebAnj9WQZXHp * refactor: consolidate feedback enums into packages/shared as a single source of truth Claude-Session: https://claude.ai/code/session_01KAC9Lbx8AmebAnj9WQZXHp * feat: add ARIA semantics, dismiss-button guard, and shared auth-route list to UsageSurveyOverlay Claude-Session: https://claude.ai/code/session_01KAC9Lbx8AmebAnj9WQZXHp * test: cover the submit-failure retry path and a persona-only minimal payload Claude-Session: https://claude.ai/code/session_01KAC9Lbx8AmebAnj9WQZXHp |
||
|
SnapOtterandGitHub
|
a0d1c70172 |
feat: add usage onboarding survey overlay (#388)
* feat: add usage-survey feedback types and gating function Claude-Session: https://claude.ai/code/session_01KAC9Lbx8AmebAnj9WQZXHp * feat: add onboarding usage-survey i18n strings to all locales Relabels three ambiguous feedback.usageTypes values (personal/team_internal/ business_workflow) and adds a new onboarding namespace (4 keys) across the reference locale and all 20 translations, so the tree compiles at every commit instead of only after both locale groups land. Claude-Session: https://claude.ai/code/session_01KAC9Lbx8AmebAnj9WQZXHp * feat: add UsageSurveyOverlay component * feat: mount UsageSurveyOverlay inside AuthGuard Claude-Session: https://claude.ai/code/session_01KAC9Lbx8AmebAnj9WQZXHp * fix: use text-start instead of text-left for RTL support in UsageSurveyOverlay * refactor: drop redundant usage-type field from the admin feedback dialog Claude-Session: https://claude.ai/code/session_01KAC9Lbx8AmebAnj9WQZXHp * feat: accept onboarding source and survey id in the feedback route Claude-Session: https://claude.ai/code/session_01KAC9Lbx8AmebAnj9WQZXHp * test: cover the onboarding source in the feedback route integration test Claude-Session: https://claude.ai/code/session_01KAC9Lbx8AmebAnj9WQZXHp * chore: remove orphaned usageTypeLabel i18n key * refactor: derive feedback source/survey_id enums from a single source of truth * perf: skip the settings fetch in UsageSurveyOverlay for non-admin users |
||
|
Matt Van HornandGitHub
|
9052da27f3 |
fix: adopt sharp 0.35.2+ by centralizing the FormatEnum key type (#362)
Centralizes the sharp format key type into a single `SharpFormat` alias derived from `toFormat()`'s signature (exported from @snapotter/image-engine, imported by the API consumers), replacing the duplicated `keyof FormatEnum` definitions. Adds convert/compress format round-trip tests covering webp/avif/png/jpeg. Part of #325. |
||
|
SnapOtterandGitHub
|
c68297d5a4 |
feat: request a tool when home search finds nothing (#385)
Adds a prefilled 'Request a tool' affordance to the home search empty state and beneath weak results. Opens the in-app feedback dialog with a new search_miss source and a structured search_query when analytics is on; links to a prefilled GitHub Discussions (Ideas) post when off, so a request is never silently dropped. Reuses the existing feedback pipe, dialog, and analytics gate; no new storage. i18n across all 21 locales. |
||
|
SnapOtterandGitHub
|
f3342a1e57 |
fix: harden Docker image and async job responses
Harden Docker runtime packaging, preserve async job response semantics, fix Redis subscriber startup connections, clear lint warnings, and harden enterprise S3 object body handling. |
||
|
SnapOtter
|
1d99acf9ee | feat: merge PostHog customer feedback | ||
|
SnapOtter
|
649e65b035 | feat: add PostHog customer feedback | ||
|
SnapOtter
|
c6319cf8a9 | fix(security): harden auth and outbound fetches | ||
|
SnapOtter
|
6f85b3d12a | fix(api): support sharp 0.35 types | ||
|
SnapOtterandGitHub
|
63a03d26f2 |
feat: pipeline templates, analytics opt-out, 83 conversion presets, positioning + e2e modernization
Lands five integrated branches: pipeline templates (#355), analytics opt-out (#354), 83 conversion presets bringing the catalog to 240 tools (#356), self-hosted positioning (#353), and e2e modernization (#351). Integration fixes: aligned stale web analytics tests with the opt-out/allow-list model, closed 3 CodeQL incomplete-sanitization alerts in the i18n generator, resolved settings/index/docs/format-matrix conflicts, and corrected tool counts to 240. |
||
|
SnapOtterandGitHub
|
8f4235d2c6 |
fix(enterprise): ship enterprise package in prod image + S3, analytics, tracing, queue fixes (#342)
* fix(enterprise): ship enterprise pkg in prod image, full license features, tracing key fallback docker/Dockerfile: COPY packages/enterprise manifest+src into the production stage. Without it, apps/api's workspace link to @snapotter/enterprise dangles and every import() throws (silently caught), so all 19 enterprise features failed closed (enterprise.active=false) regardless of a valid license. scripts/generate-license.mjs: sync PLAN_FEATURES with packages/enterprise/src/license.ts so a --plan enterprise license unlocks all 19 features (was 8) and team unlocks 8. apps/api/src/tracing.ts: accept SNAPOTTER_LICENSE_KEY as a fallback to LICENSE_KEY so distributed_tracing activates with the same key as the rest of the app. * fix(docker): keep scripts/bake-analytics.mjs in build context .dockerignore excluded the whole scripts/ dir (PR #82, V1 hardening), but docker/Dockerfile later added 'COPY scripts/bake-analytics.mjs' for the analytics bake step. A clean production image build therefore fails with 'scripts/bake-analytics.mjs: not found'. The published image build is gated off in CI so this latent break went unnoticed. Exclude scripts/* but re-include the one file the Dockerfile needs. * fix: S3 upload stream, analytics bake reaches API, dedupe retention field, reconcile orphan jobs storage-s3.ts: wrap the upload AsyncIterable in Readable.from() so @aws-sdk/lib-storage accepts it. STORAGE_MODE=s3 file uploads failed with 'Body Data is unsupported format' for every tool because a bare async generator is not a Readable. docker/Dockerfile: COPY the builder-baked analytics baked.ts into the API runtime stage. The API re-copied the committed (off) baked.ts from the build context, so the SNAPOTTER_ANALYTICS build arg had no effect on the API -- and since the SPA reads /api/v1/config/analytics, analytics was off everywhere regardless of the arg. settings-dialog.tsx: remove the duplicate tempFileMaxAgeHours control under Data Retention; it bound the same setting key as the File Management control with a different default, so editing either silently overwrote the other. apps/api/src/index.ts: reconcile orphaned job rows (empty tool_id, never enqueued to BullMQ) at boot so they don't sit in processing/queued forever and inflate the per-user concurrent-job count and the upgrade-check in-flight gate. * fix(web): style the SSO login buttons (they referenced undefined theme tokens) The OIDC/SAML 'Sign in with <provider>' buttons used bg-secondary / text-secondary-foreground, which the web theme never defines (it has primary, background, foreground, muted, border, card, primary-subtle). Those classes resolved to nothing, so the buttons rendered as bare unstyled text on the login page. Restyle: the optional (non-enforced) buttons become white-card outline buttons with a key icon and an orange hover tint, secondary to the primary Login button; the SSO-enforced buttons become solid primary with the icon. * fix: gate S3 behind license, custom-role enterprise perms, wire retention UI, cleanup S3 is a licensed feature, but shipping packages/enterprise in every image removed the implicit gate, so STORAGE_MODE=s3 worked without a license. Enforce isFeatureEnabled('s3_storage') at boot and fail fast if unlicensed. Custom roles can now be granted security:manage / compliance:manage / webhooks:manage (roles.ts ALL_PERMISSIONS + the Roles UI) so admins can build least-privilege compliance/security roles instead of only the built-in admin role. retentionSweep now reads the jobsRetentionDays / auditRetentionDays DB settings the System Settings UI writes (env vars become the fallback default), mirroring how the temp-file sweep reads tempFileMaxAgeHours. Previously those two UI controls were no-ops. Cleanup: drop the never-set snapotter_storage_bytes gauge and the unused MAX_WORKSPACE_SIZE_GB env var; emit tool_client_error to PostHog from the web ErrorBoundary (client crashes were not reaching analytics); add the Python OpenTelemetry packages so the innermost sidecar.<script> span exports; fix the stale 'only local storage' line in the docs; delete two e2e-analytics specs that tested the removed consent UI. * fix(env): restore MAX_WORKSPACE_SIZE_GB default security-auth-hardening.test.ts asserts env.MAX_WORKSPACE_SIZE_GB defaults to 10, so the var is an intentional (tested) default, not dead code. Removing it in the cleanup commit broke that unit test. Keep the declaration. |
||
|
SnapOtterandGitHub
|
5d36ac06d8 |
feat(analytics): build-time bake + telemetry depth (#336)
Bake PostHog + Sentry into the published Docker image (SNAPOTTER_ANALYTICS build arg, codegen script). Delete entire consent system. Move event emission to BullMQ worker. Add cross-tier identity stitching, Sentry performance tracing on both tiers, frontend funnel events. Fix stateful regex bug. 86 files changed, 1593 insertions(+), 3747 deletions(-) |
||
|
SnapOtterandGitHub
|
32c1192d63 |
fix(passport-photo): require the face-detection bundle, not just background-removal (#329)
* fix(passport-photo): require the face-detection bundle, not just background-removal Passport Photo runs face-landmark detection (face_landmarks.py, gated to the face-detection bundle) before background removal (background-removal bundle), but it was only declared under and guarded against background-removal. A user who installed only Background Removal passed every JS-side check, then hit a late "feature_not_installed" from the Python dispatcher gate when the analyze step ran face landmarks, and the UI never told them Face Detection was needed. - shared: add TOOL_EXTRA_BUNDLES + getRequiredBundlesForTool so a tool can declare more than one required bundle (passport-photo needs background-removal and face-detection). enablesTools is untouched, so the one-tool-per-bundle invariant still holds. - api: isToolInstalled() now checks every required bundle; add getFirstMissingBundleForTool() so the analyze and base routes, pipeline (both guards) and batch report the bundle the user actually still needs. - web: the proactive install prompt (tool-page) and features-store treat a tool as installed only when all required bundles are present, and point the prompt at the first missing one (sequential install, no new UI). Refs #327 * test(passport-photo): deterministic integration coverage for the two-bundle guard Boots the real API with an isolated DATA_DIR and controls installed.json to prove the HTTP route behavior end-to-end: - nothing installed -> 501 naming background-removal - only background-removal installed -> 501 naming face-detection (issue #327) - both installed -> guard passes (not 501) - base route reports face-detection too Refs #327 |
||
|
SnapOtterandGitHub
|
8952e9ba47 |
fix: harden against three production Sentry crashes (#328)
Three production crashes from the snapotter/node Sentry project.
feature-status (NODE-12): a valid-JSON-but-wrong-shape installed.json
crashed boot via Object.keys(data.bundles). readInstalled() now
normalizes any unusable shape to { bundles: {} }, and the boot recovery
call is wrapped so cleanup can never fatal startup.
image-viewer (NODE-15/17/18): drag-to-pan read .x off an undefined
use-gesture memo on pointerUp or a pinch-into-pan. A guarded pure helper
(resolvePanStart) now falls back to the live pan offset.
Fastify (NODE-14): raised pluginTimeout to 60s so slow self-hosted boots
do not fatal at @fastify/static.
|
||
|
SnapOtterandGitHub
|
1fec97111b |
fix(docker): make storage writable under non-root/foreign UIDs (TrueNAS, OpenShift) (#299)
The entrypoint only fixed volume permissions when started as root (chown +
gosu-drop to snapotter). Launched under a non-root/foreign UID (TrueNAS app
user, Kubernetes runAsUser, OpenShift) it did no permission setup, so /data and
/tmp/workspace -- owned by uid 999 from the image -- were not writable by the
running user. Uploads and processing then failed with a cryptic EACCES
("workspace folder is not writable") and AI bundle installs failed the same way,
while health checks still reported the container healthy.
- entrypoint: source new entrypoint-lib.sh; verify writability up front when
non-root, and as snapotter after chown when root (catches root-squashed
mounts), failing fast with an actionable message (which dir, uid/gid, how to
fix) instead of a late, cryptic EACCES
- Dockerfile: own /data and /tmp/workspace as snapotter:0, group-writable with
setgid, so an arbitrary UID with the root supplementary group (OpenShift /
Kubernetes fsGroup) can write; keep /opt/venv world-readable for the AI venv
bootstrap under arbitrary UIDs
- api: assert storage writability at boot (lib/storage-writable.ts), failing
fast with the same guidance even when the entrypoint is bypassed
- docs: add a Storage permissions section (named volumes, bind mounts, TrueNAS,
Kubernetes/OpenShift) and cross-link it from the security guide
Fixes #230
|
||
|
SnapOtterandGitHub
|
3d9ff1e0d2 |
fix(api): decode RAW via LibRaw first so DNG processes at full resolution (#289) (#290)
RAW (DNG) processing crashed on ImageMagick's deprecated ufraw-batch delegate, which fails on modern formats such as iPhone ProRAW DNG. Root cause: the dcraw_emu (LibRaw) decode tier read the wrong output path. dcraw_emu APPENDS the output extension (raw-in-X.dng -> raw-in-X.dng.tiff) but the code looked for raw-in-X.tiff (replaced extension), so readFile threw on every RAW, the tier silently fell through to ufraw, and the 24MB TIFF leaked into the temp dir on each attempt. - Repair the dcraw_emu output path; clean it up in finally (fixes the leak) - Prefer LibRaw full decode over embedded-preview extraction so a full-resolution RAW is never silently returned as a reduced-size preview (sample DNG: was 1024x683 preview, now 3474x2314 full) - Add RAW decode regression tests (DNG full-resolution + all 6 RAW formats); these were absent, which let the bug ship - Install libraw-bin on CI test runners so dcraw_emu is actually exercised |
||
|
SnapOtterandGitHub
|
f75cc328ac |
fix(security): numeric CIDR matching for IPv6 SSRF allow/deny
Replace textual prefix matching in isPrivateIPv6() with numeric bit-prefix matching via ipaddr.js. Closes reachable classifier bypasses for IPv4-compatible IPv6 (::a.b.c.d), site-local fec0::/10, and the full fe80::/10 link-local span, on top of the IPv4-mapped forms. Embedded IPv4 (mapped and compatible) is run through the full isPrivateIPv4 classifier; unparseable input fails closed.
Follow-up hardening to
|
||
|
SnapOtter
|
bdadb843d8 |
fix(security): close remaining high-severity CodeQL alerts
- svg-sanitize.ts: strip each dangerous element repeatedly until stable with whitespace-tolerant end tags, defeating nested/overlapping tags (closes 5 incomplete-multi-character-sanitization + 1 bad-tag-filter; the prior single-pass regex could leave a residual <script>/<iframe>). - file-preview.ts: add a resolve()+containment barrier (the path-traversal guard CodeQL recognizes) on top of the id charset check (closes 9 path-injection). - metadata.ts: bound the XMP namespace:name key segments so parseXmp cannot backtrack polynomially (closes js/polynomial-redos). - analytics-disabled.spec.ts: match analytics by URL host, not substring (closes 4 incomplete-url-substring-sanitization). typecheck + lint green; svg (119), preview (22), metadata (164) tests pass. |
||
|
SnapOtter
|
8001ec97f1 | Merge branch 'fix/biome-lint-debt' into chore/consolidate-v2.0.0 | ||
|
SnapOtter
|
37841e3369 | Merge branch 'security/hardening-audit' into chore/consolidate-v2.0.0 | ||
|
SnapOtter
|
7561f2a8c3 | fix(lint): make husky hook executable; rename backend useS3 to isS3Enabled (clears 17 false-positive useHookAtTopLevel) | ||
|
SnapOtter
|
0acc8ca751 | chore: post-2.0 QA hygiene across api hardening and qa metadata | ||
|
SnapOtterandGitHub
|
19d9ed181a |
fix(audio): low-samplerate ogg encode + post-2.0 QA hygiene
Quality-VBR ogg (libvorbis -q:a) fixes 8 kHz 'encoder setup failed' in both ogg paths; drop empty COOKIE_SECRET ENV (app auto-generates); emit real bundle extractedSize; fix stale image-pad/compress-pdf QA specs. |
||
|
SnapOtter
|
d61226496b |
fix(security): harden rate limits, Redis auth, resource caps, and error sanitization
- Lower LOGIN_ATTEMPT_LIMIT default from 30 to 10 (brute-force protection) - Lower RATE_LIMIT_PER_MIN default from 1000 to 300 - Add Redis authentication (requirepass) with REDIS_PASSWORD env var - Add Redis maxmemory 512mb cap to prevent unbounded growth - Add mem_limit: 1g to Postgres and Redis containers - Strip internal file paths from all error responses (defense-in-depth) - Add startup warnings for default admin/Postgres/Redis credentials - Update security test expectations for new defaults |
||
|
SnapOtter
|
6220885e0c |
fix(api): narrow friendlyError matching to avoid collapsing valid messages
The raw-dump detector matched broad content keywords ('conversion failed', 'pixel format', bare 'ffprobe') that appear in legitimate validation messages (e.g. 'SVG conversion failed'), which would wrongly collapse them to the generic error. Narrow it to the unambiguous 'ffmpeg/ffprobe exited N:' prefix, python tracebacks, and crashes; longer/multi-line raw dumps are still caught by the length/line-count check. Found during self-review.
|
||
|
SnapOtter
|
4af4bfa8eb |
fix(api): return user-safe processing errors, keep raw stderr in logs
Add friendlyError() which collapses raw external-tool failure output (ffmpeg/ffprobe/LibreOffice/qpdf/etc.) into one generic sentence while preserving intentional validation messages and scrubbing internal paths. Apply it at every client-facing error surface in the tool factory and job worker (sync 422, async SSE, pipeline + batch finalize). The full error is still recorded server-side via request.log.error / logger.error and telemetry. |
||
|
SnapOtterandGitHub
|
8eee17aeea |
fix: handle non-image modalities across uploads, previews, and filenames (#255)
SnapOtter spans five modalities now, but several code paths still assumed image input. - dropzone: default to accept-all when no fileFilter is given (image tools still pass one); neutral "supported file types" error text instead of "image files" - automate (pipelines): accept any modality in the file pickers and dropzones; render modality-aware previews (video player, audio waveform, document/data card) instead of always using ImageViewer/BeforeAfterSlider - filename sanitizer: extend the double-extension allowlist beyond image extensions to video/audio/document/data so e.g. "report.csv.php" becomes "report.csv"; add tests - thumbnail route: return 422 for non-rasterisable files (audio, data, non-PDF docs) instead of attempting a doomed Sharp decode - pool: unknown tools fall back to the "system" pool, not the image pool - a11y labels: "Previous/Next image", "Image viewer/area/controls/drop zone" are now modality-neutral, across all 21 locales - copy: bulk-rename default, find-duplicates ZIP name, SSRF user-agent, fetch-urls fallback name, file-details MIME label, URL-import placeholder, help dialog |
||
|
SnapOtterandGitHub
|
d50e8e42a7 |
QA + image-tool depth pass: codec/eraser/PDF fixes, modality renames, 13 image tools deepened (#249)
* fix(media): mux container-correct codecs in video tools Video tools hardcoded H.264 (and AAC) while keeping the input's container extension, so a .webm input produced an invalid file (ffmpeg exit 234: H.264 cannot be muxed into WebM). Add shared videoEncodeArgsForContainer/audioEncodeArgsForContainer helpers (vp9+opus for webm, theora+vorbis for ogv, h264+aac otherwise) and apply them across 14 tools; re-encode audio to AAC in burn-subtitles (forced mp4). Adds a webm regression test for change-fps. * fix(eraser): recover Object Eraser when its progress SSE drops The eraser used a bespoke EventSource with no recovery, so a dropped SSE left the UI stuck at ~25% forever even though the backend job had finished and saved its result. Add a resilient subscription (reconnect on tab refocus, which replays the cached terminal frame; 5-minute stall timeout) mirroring the standard processor's PR #203/#204 recovery. * feat(ui): rename the Documents modality to PDF and Data to Files Updates modality display names, the home-page tabs, the tool-page breadcrumb, and the homePage.documents/data + modalities labels across all 21 locales. URL slugs are unchanged for link stability. * feat(compress-pdf): add quality and target-size compression modes Mirror the image Compress tool: a quality slider (1-100) and a target file size, replacing the screen/ebook/printer preset. Adds gsCompressPdfQuality to doc-engine (quality maps to image downsample DPI, the dominant size lever for PDFs); target-size binary-searches the DPI for the highest quality under the target. The frontend reuses the shared CompressControls component, so no new translation strings are needed. * feat(ocr-pdf): show the PDF preview and extracted text side by side ocr-pdf fell back to the image viewer, which cannot render a PDF, so the right pane showed 'Preview not available' and the extracted text was only a download. It now uses a custom results view (custom-results display mode) rendering the input PDF via pdf.js (DocumentView gains an inputOnly prop, since the tool's output is a .txt) next to the extracted OCR text, with a copy button. * feat(ui): link the modality breadcrumb to its tools tab The modality segment of the tool breadcrumb (PDF, Image, Video, Audio, Files) is now a link to /?modality=<tab>. The home page reads the param, activates the matching tab, and cleans the URL, so it returns to the existing Tools page filtered to that modality without a new page. Handles the file modality whose tab key is 'data'. * feat(circle-crop): add zoom/offset framing, border, background, and output size Upgrade the circle-crop tool from a bare centered crop into a framing and styling tool. New settings (all backward-compatible with the old empty payload): - zoom (1-5x) plus offsetX/offsetY (0-1) to control how tight the circle is and where it sits in the source image - borderWidth (0-200px) plus borderColor for an optional ring - background: transparent (clear corners) or a hex fill - outputSize for a square output; omitted keeps native size The settings panel gains an inline draggable circular preview that mirrors the framing live, a zoom slider, a border slider with color, a transparent/color background toggle, and an output-size field. Adds an integration test covering output size, border, and a solid background. * feat(image-tools): flesh out five thin tools (gif-webp, histogram, favicon, color-palette, lqip) Tier A of the image-tool depth pass. Each of these was as bare as the old circle-crop (empty settings, opaque or invisible output). Now: - gif-webp: quality, lossless, and resize-percent controls; shows before/after size - histogram: returns full per-channel bins + stats; the settings panel renders an inline interactive histogram with R/G/B/Luma toggles, linear/log scale, and a mean/median/stdev readout (server PNG still downloadable) - favicon: background fill, padding, corner-radius, theme color, and a per-size checklist, with a live preview grid; the route applies the styling and honors the size filter - color-palette: count (2-16) and hex/rgb/hsl format controls, median-cut extraction, a palette strip, and CSS/JSON export - lqip-placeholder: blur/pixelate/solid strategies, format and quality; the output panel now surfaces the data URI with copy plus HTML/CSS snippets and a preview (previously the deliverable was never shown) Also expose resultPayload from useToolProcessor so a tool can render the route's extra result fields (histogram bins, lqip data URI) in its own panel. Updates the five integration tests to cover the new settings. * feat(image-tools): deepen five thin tools (duotone, vignette, pixelate, background-replace, blur-background) Tier B of the image-tool depth pass. - duotone: preset palettes, an intensity slider that blends the duotone with the original, and a true live duotone preview (a self-contained grayscale + lighten/darken overlay so the pane filter cannot wash it out) - vignette: radius, softness, roundness, and center-x/y controls driving a rebuilt radial gradient, with a matching live overlay - pixelate: a selection mode that exposes the route's region support via a draggable box over the image plus width/height sliders, so a face or plate can be pixelated in isolation - background-replace: gradient backgrounds, edge feather, and webp output on top of the existing solid color; now shown before/after - blur-background: edge feather and webp output; now shown before/after The live previews for duotone and vignette needed onImageStyle to mount the overlay branch in image-viewer. The duotone intensity blend and both AI tools' edge feather were rewritten to splice the alpha channel through raw buffers; joinChannel did not reliably re-tag the merged channel as alpha and a raw-without-encoder buffer broke the next decode. Updates the five integration tests. * fix(data): rename Files modality to Data + 20 Data-tool bug fixes (#247) * fix(ui): restore the Data modality name (revert Files rename) The 'file' modality reverts to the 'Data' label in modality.ts, the home-page tab, and the tools.data + documentsAndFiles i18n keys across all 21 locales. The separate Documents to PDF rename is kept. The URL slug was already /data, so name and slug realign; the tool breadcrumb follows modality.ts automatically. * fix(create-zip): require at least two files before enabling submit create-zip enabled its submit button with a single file, but the backend rejects fewer than two files ('Zipping needs at least two files'), producing a 422 error. Gate the button on files.length >= 2 to match the sibling merge-csvs tool. Found during the Data-modality QA sweep. * fix(data): resolve 17 bugs found in a deeper Data-tool review Crashes (threw an internal error on otherwise-valid input): - csv-json: a primitive JSON array like [1,2,3] threw "Unable to serialize"; now a clear error. - json-xml: a null or primitive JSON root crashed the XML builder; now a clear 4xx. - yaml-json: an empty or comment-only YAML returned undefined and threw on Buffer.from; now emits null. Data loss / wrong output: - csv-json: nested objects rendered as "[object Object]" (now serialized to JSON); heterogeneous objects dropped columns (now the union of all keys). - xml-to-csv: leaked fast-xml-parser markers ("@_" on attributes, "#text") into CSV headers (now cleaned); a single-record XML failed to tabulate (now a 1-row table); heterogeneous records dropped columns (now the union of all keys). - csv-excel: xlsx date cells were rendered in the server timezone via Date.toString (now ISO 8601, round-trippable). - create-zip and extract-zip: filename/basename collisions overwrote zip entries and silently lost a file; dedup now checks generated names and guarantees uniqueness. - chart-maker: negative values produced invalid/degenerate SVG that Sharp silently dropped; now rejected with a clear message. Empty output / validation: - split-csv: a header-only CSV produced an empty zip; now errors with "No data rows to split". - extract-zip: a directory-only zip produced an empty zip; now errors with "No extractable files found". - create-zip and merge-csvs: a single-file request fell through to the worker and returned 422; the factory now supports minInputs and returns 400 pre-enqueue. UI: - review-panel: the result card showed "Saved +X%" when the output grew; the savings row now appears only when the file is actually smaller (Original/Processed sizes always shown). Found via two adversarial code-review passes over the 10 Data routes. All 24 fix + regression checks pass against a fresh Docker stack on :1359. * fix(data): clean 400 for unsafe-zip entries; drop header on split keepHeader=false - tool-factory: add an opt-in preValidate hook that runs after input prep and before enqueue. Throwing InputValidationError there returns its statusCode (400) instead of the worker's generic 422. BullMQ loses the error class across the job boundary, so InputValidationErrors thrown in the worker cannot be mapped to their status; pre-enqueue validation can. - extract-zip: validate entry paths via preValidate, rejecting path-traversal and absolute-path archives (and unreadable/corrupt zips) with a clear 400. The processV2 guards remain as defense-in-depth for the pipeline/batch path. - split-csv: keepHeader=false now drops the header (parts contain only data rows) instead of keeping it as the first data row of part-1. Verified against a fresh Docker stack: unsafe / absolute / corrupt zips -> 400, normal zip still 200; split keepHeader=false drops the header while true repeats it in each part. No regressions across 51 fix + scenario checks. * feat(image-tools): deepen image-pad and sprite-sheet, fix sprite-sheet multi-file submit Tier C of the image-tool depth pass. - image-pad: a custom W:H ratio alongside the presets, a background mode (solid color, transparent, or an Instagram-style blurred cover fill), and an extra padding margin. The settings panel gains a real live preview of the padded canvas (it previously declared live-preview but rendered nothing) via onImageStyle + onImageOverlay. - sprite-sheet: PNG/WebP/JPEG output with a quality control, and the coordinate map it already computes is now returned and surfaced as Copy CSS (per-frame background-position rules) and Copy JSON exports. Also fix a pre-existing sprite-sheet bug: with more than one image the panel called processAllFiles, fanning out to the per-file batch route (422). It now calls processFiles, which packs all images into a single sheet request (it is a MULTI_FILE tool). Updates both integration tests. * fix(media): preserve source sample rate after loudnorm (#243) ffmpeg's loudnorm filter runs internally at 192 kHz and emits 192 kHz unless the chain resamples back. normalize-audio and video-loudnorm therefore produced 192 kHz output (4.3x larger files) regardless of the input rate. Append aresample to restore the input's sample rate. runMediaTool now exposes the input audio sample rate to its args callback. * fix(color-palette): collapse solid-color images to one swatch The median-cut bucket selector started bestRange at -1, so a uniform bucket (range 0) still satisfied the > comparison and kept splitting, yielding N identical swatches for a solid-color image. Start at 0 so only buckets with real color spread are split. * fix(lint): annotate implicit-any lets in saml and user-files biome noImplicitAnyLet flagged the bare let in saml.ts (profile) and user-files.ts (stream); add derived type annotations (type-only, no behavior change). Latent on main via the turbo lint cache; surfaced when the Data changes busted the apps/api lint cache. |