Briefly retry Postgres and Redis connectivity at boot (waitForService, DB_STARTUP_TIMEOUT_MS) so an ordered-but-not-yet-ready dependency recovers cleanly instead of crash-looping.
Add a once-per-boot instance_started event (arch, os, deploy_mode,
gpu_present) so the fleet architecture mix is measurable. It reuses the
existing per-instance instance_id and is exempt from the volume sample
rate, since a census that fires once per boot must not be thinned.
Restore useful capture depth now that the sponsored plan removes the
quota pressure behind the earlier hardening:
- PostHog sample rate 0.1 to 1.0 (full analytics when enabled); the
property allowlist still blocks file data.
- Sentry per-instance ceiling 20 to 500/hr, breadcrumb trail restored
(sanitized: urls/paths redacted, data payloads dropped), full stack
paths kept; local vars, request bodies, and PII still dropped. Both
api and web.
Honor ANALYTICS_ENABLED=false as an opt-out alias: it was documented on
the Docker Hub README but never wired in 2.x, so anyone who set it was
still tracked.
All capture stays behind the analytics opt-out gate.
Adds a dedicated remove-gif-background AI tool: removes the background from an animated GIF, WebP, or APNG frame by frame and reassembles a transparent (or composited) animation in WebP, APNG, or GIF, with full per-frame effects. Reuses the background-removal bundle. Verified end-to-end with the real rembg model.
Closes#496.
Removes Sentry tracing entirely (BullMQ idle polling burned 4.8M transactions in 2 days at the baked 0.1 rate), decouples PostHog sampling, and replaces the type-only error scrub with a vetted-field sanitizer plus SafeError/ToolInputError contracts. One classified capture path with per-signature throttles and a per-process ceiling makes storms impossible (NODE-1E was 4,541 events from one 30s loop). Browser errors move to a dedicated web Sentry project with their own source maps. Adds the SNAPOTTER_TELEMETRY runtime kill switch and silences test fleets.
Crash fixes: remote 204/304 SSRF process kill (NODE-20), conversion-preset boot crash loop (NODE-21), Redis version preflight + unhandled subscribe rejection (NODE-1T), Sign PDF on plain-http origins (NODE-1K/1M), wavesurfer/pdf.js teardown rejections (NODE-1P/1N), bundle-import ZlibError to 400 (NODE-1Z), chart-maker input errors declassified (NODE-1H/1J), asset requests skip the session DB lookup (NODE-1D).
* fix: remove all automatic third-party egress (OSM tiles, Scalar fonts, editor Google Fonts, AI model download fallbacks)
Phone-home audit follow-up. The product no longer makes any automatic
third-party request; user-initiated click-outs stay, and production now
fails closed on missing AI models.
1. GPS leak via OSM tiles: the strip-metadata panel auto-loaded
tile.openstreetmap.org tiles encoding the photo's GPS position. The
Leaflet mini-map is gone; coordinates render as text plus an explicit
View on map link (openstreetmap.org, opens on click only). Removed
tile.openstreetmap.org from the CSP img-src, dropped the leaflet
dependency, added the viewOnMap i18n key to all 21 locales.
2. Scalar docs fonts: /api/docs loaded Inter and JetBrains Mono from
fonts.scalar.com. Scalar now renders with withDefaultFonts: false and
both --scalar-font and --scalar-font-code pinned to system stacks;
fonts.scalar.com removed from the docs CSP font-src. Verified by
injecting GET /api/docs/: config carries withDefaultFonts false and
the served page has no fonts.scalar.com reference.
3. Editor Google Fonts: the editor font picker built
fonts.googleapis.com stylesheet URLs for 25 web fonts the served CSP
already blocked. The remote loading path is deleted; the picker now
offers system fonts only, with a SELF_HOSTED_FONTS seam (FontFace API,
same origin) for bundling fonts later. Unknown families saved in old
documents fall back to the browser default.
4. Python sidecar fails closed on model downloads: new
packages/ai/python/offline_guard.py gates every runtime download
fallback (inpaint, outpaint, restore, noise_removal, detect_faces,
enhance_faces, face_landmarks, red_eye_removal, remove_bg, ocr,
transcribe, upscale) behind SNAPOTTER_ALLOW_MODEL_DOWNLOAD=1 with an
actionable error. Bundled models keep working untouched.
5. OCR and transcription library-internal downloads: unbundled PaddleOCR
language and detection fallbacks now raise the guard error naming the
language instead of resolving models over the network; faster-whisper
gets local_files_only when downloads are off.
6. GFPGAN and CodeFormer cwd-relative weights: facexlib and
codeformer-pip resolve helper weights relative to the process cwd and
fetch them from GitHub when absent. They are now symlinked from the
installed bundle files under MODELS_PATH/gfpgan/facelib before the
libraries load, failing closed when unresolvable.
Defense in depth: HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 are set in
the runtime image and in the sidecar spawn env; install_feature.py lifts
them for user-initiated bundle installs and restores them afterwards
(it can run in-process inside the dispatcher). SNAPOTTER_ALLOW_MODEL_DOWNLOAD
is documented in .env.example, default off.
Validation: typecheck 9/9 workspaces, Biome clean on touched files,
5178 unit tests pass, py_compile on all touched scripts, guard behavior
exercised in both dispatcher exec and per-request import modes, zero
remaining runtime references to the three hosts. Docker build and live
AI inference need post-merge verification on the GPU host.
Claude-Session: https://claude.ai/code/session_01XGB4pGvTvb7sUX4JN745U7
* fix: allow AI model downloads by default, make strict offline mode opt-in
Product call: ease of use first. The download gating from the previous
commit inverts its default: runtime model fetches (public model weights
only, never user data) are allowed out of the box so AI tools self-heal,
and SNAPOTTER_ALLOW_MODEL_DOWNLOAD=0 becomes the explicit strict offline
mode for airgapped deployments, where every fallback raises the
actionable error instead of fetching.
Changes: offline_guard blocks only on an explicit 0/false; the
unconditional HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE image ENV is removed
and bridge.ts sets those flags for the sidecar only in strict mode;
.env.example documents the new default; install_feature's lift/restore
stays. All bundled-path preferences, pre-existence checks, and symlink
pre-placement remain, so installed bundles never trigger a download.
The OSM, Scalar font, and editor font fixes are unchanged.
Validation rerun: typecheck 9/9, Biome clean on touched files, 5178
unit tests pass, py_compile on touched scripts, guard behavior verified
for unset/1 (allowed) and 0/false (blocked with the new message).
Claude-Session: https://claude.ai/code/session_01XGB4pGvTvb7sUX4JN745U7
* docs: rebrand from image-only to multi-modality across docs and metadata
SnapOtter expanded from image-only to 157 tools across 5 modalities
(image, video, audio, document/PDF, data). Update all product-level
copy, metadata, and i18n that still framed it as an image-only tool.
- README, package.json, root llms.txt: multi-modality framing, 157 tools
- OpenAPI info + tags, generated /llms.txt tagline (docs.ts)
- VitePress docs site: hero, getting-started, architecture, security,
deployment, configuration, developer, supported-formats
- i18n: 10 product keys across all 21 locales (hero, app description,
privacy notes, AI features, progress messages, getting-started)
- web/demo/landing meta + privacy copy, COMMUNITY_GUIDE, .env.example
Stale tool counts (53/50+/52/70+/35) corrected to 157 throughout.
Database/container deployment claims left unchanged (out of scope).
* docs: fix stale post-rebrand test assertions and README language list
- tests/e2e-docs/homepage.spec.ts: assert the current docs homepage (file toolkit, 157 tools, 5 modalities) instead of the old image-only strings
- tests/unit/api/docs-route.test.ts: sync the reproduced llms.txt tagline with docs.ts
- README.md: 21 languages with the correct list (add Swedish and Chinese Traditional, drop Czech which is not supported)
* docs: correct 2.0 architecture references (Postgres 17 + Redis 8, 3-container stack)
The docs and metadata still described the 1.x stack (SQLite, single container, p-queue). Update them to the current 2.0 reality.
- README: replace the broken single-container `docker run` quick-start with the real Docker Compose stack (app + Postgres 17 + Redis 8); fix the "no Redis, no Postgres" feature bullet
- package.json: description no longer claims a single container
- apps/docs: rewrite database.md for Postgres; configuration.md DB_PATH -> DATABASE_URL + REDIS_URL; architecture.md SQLite/p-queue/better-sqlite3 -> Postgres/BullMQ/pg and add media-engine + doc-engine; developer/security/deployment/docker-tags/getting-started/contributing compose examples now include postgres + redis; index.md + api/ai.md AI count 16 -> 19
- SECURITY.md: Drizzle (SQLite) -> (PostgreSQL)
- landing: enterprise/FeatureHighlights single-container wording; TrustSignals/ToolGrid 150+ -> 157 (dynamic); Pricing/FAQ 15 -> 19 AI tools
* docs(api): document all video, audio, document, and data tool endpoints in OpenAPI
The spec covered only image tools; the Scalar UI and the generated /llms.txt and /llms-full.txt inherited that gap. Add the 104 missing tool endpoints so the API docs match the code.
- Video: 29 endpoints (most long/async; auto-subtitles is AI)
- Audio: 17 (transcribe-audio is AI)
- Document/PDF: 36 (ocr-pdf is AI; conversions are long/async)
- Data: 10
- Image: 12 newer tools (background-replace, blur-background AI; histogram/lqip-placeholder/sprite-sheet custom responses; barcode-generate uses a JSON body)
Each schema is derived from the tool's Zod validator and executionHint (fast -> 200, long -> 202+SSE, AI adds 501 FeatureNotInstalledError, multi-file inputs as arrays), referencing the existing shared schemas. Tool path entries: 64 -> 168. Spec parses as valid YAML with no duplicate paths and only known $refs.
* feat(tracing): add OpenTelemetry dependencies and --import preload flag
* feat(enterprise): add distributed_tracing feature gate
* feat(tracing): add SDK bootstrap with enterprise gating
* fix(tracing): correct test coverage for enterprise-unavailable path and prevent double-init
Test 2 now mocks @snapotter/enterprise to throw an import error, exercising
the catch block in the preload. Test 3 imports with no endpoint so the preload
is a no-op, avoiding leaked SDK from double-initialization. Added idempotency
guard to initTracing() as a safety net.
* feat(tracing): add Pino trace mixin and shared logger
When OTel tracing is active, every Pino log line now includes traceId,
spanId, and traceFlags fields for log-to-trace correlation. The mixin
is a no-op when no SDK is registered (community users).
* feat(tracing): add _otel to ToolJobData and inject trace context at enqueue
Add optional _otel carrier field to ToolJobData for W3C trace context
propagation across BullMQ job boundaries. When an active OTel span exists,
propagation.inject() writes traceparent/tracestate into the job data before
queue.add(). When no SDK is registered (community edition), the carrier
stays empty and _otel remains undefined -- zero overhead.
* feat(tracing): extract trace context and create spans in BullMQ worker
* feat(tracing): inject trace context into Python sidecar calls
* feat(tracing): add trace context extraction to Python sidecar
* feat(tracing): add shutdownTracing to graceful shutdown sequence
* feat(tracing): enrich HTTP spans with tool_id and user_id attributes
* docs: add OpenTelemetry env var documentation to .env.example
* test(tracing): add lifecycle integration tests for trace propagation
* fix(tracing): inject trace context into pipeline and batch flow jobs
* fix(tracing): add sidecar.execute Node-side span and remove unnecessary comment
Wraps PythonDispatcher.run() with a sidecar.execute span on the Node
side so traces show the full round-trip (Node span -> Python span).
Also removes an obvious comment from logger.ts.
* feat(infra): add dev compose stack with postgres and redis
* fix(infra): comment dev env defaults until wired; harden dev compose restart and start_period
* chore(deps): add pg driver and testcontainers for postgres migration
* feat(db): translate schema to drizzle pg-core (timestamptz, boolean, pgEnum, jsonb)
Schema translation (apps/api/src/db/schema.ts):
- sqlite-core -> pg-core, all 10 tables preserved 1:1
- integer(mode:'timestamp') -> timestamp({ withTimezone: true })
- integer(mode:'boolean') -> boolean
- jobs.status text enum -> pgEnum('job_status') with same 4 values
- 7 columns changed from text to jsonb: jobs.inputFiles, jobs.settings,
pipelines.steps, apiKeys.permissions, roles.permissions,
auditLog.details, userFiles.toolChain
- settings.value stays text, jobs.error stays text, jobs.progress stays real
jsonb call-site sweep (removed JSON.stringify on writes, JSON.parse on reads):
- apps/api/src/routes/roles.ts: permissions read/write (3 sites)
- apps/api/src/routes/api-keys.ts: permissions write + read (2 sites)
- apps/api/src/routes/audit-log.ts: details read (1 site)
- apps/api/src/routes/pipeline.ts: steps write + read (2 sites)
- apps/api/src/routes/progress.ts: inputFiles write (2 sites)
- apps/api/src/routes/tool-factory.ts: toolChain read + write (2 sites)
- apps/api/src/routes/user-files.ts: toolChain read + write (4 sites)
- apps/api/src/permissions.ts: roles.permissions read (1 site)
- apps/api/src/lib/audit.ts: details write (1 site)
- apps/api/src/plugins/auth.ts: apiKeys.permissions read (1 site)
* refactor(db): type jsonb columns via $type and note raw CTE conversion requirements
* feat(db): archive sqlite migrations and generate postgres baseline
* chore(db): dockerignore legacy migrations, add archive breadcrumb, fix trailing newline
* feat(db): pg pool connection, advisory-locked boot migrations, DATABASE_URL config
* fix(db): friendly fatal on unreachable postgres, idempotent closeDb, lock-key convention note
* refactor(db): async drizzle calls in plugins, lib, permissions
* fix(api): analytics never throws, typed permission guard, single-query session invalidation
* refactor(db): async drizzle calls across all routes and bootstrap
Convert every route file and index.ts from sync SQLite drizzle
patterns to async node-postgres drizzle:
- .all() removed (bare await on select)
- .get() converted to destructured [row] = await ...
- .run() removed (bare await on insert/update/delete)
- .changes replaced with .rowCount (null-guarded) in progress.ts
- sqlite import removed from user-files.ts; raw CTEs converted to
await db.execute(sql`...`) with postgres-dialect recursive CTEs
- ChainRow types updated: tool_chain is parsed jsonb (string[] | null),
created_at is Date (timestamptz) with no * 1000 conversion
- All requirePermission() guard calls awaited (security: unawaited
async guard returns truthy Promise, bypassing permission check)
- All hasEffectivePermission() and getPermissions() calls awaited
- All auditLog() calls awaited (preserves write-before-response order)
- trackEvent() and captureException() left un-awaited (fire-and-forget
by design, guaranteed never-throw)
- ensureAnonymousUser(), startCleanupCron(), recoverStaleJobs() awaited
in bootstrap sequence
- ensureInstanceId() and ensureDefaultSettings() made async
Files converted: 14 (index.ts + 12 route files + tools/index.ts)
* fix(db): await async checkStorageQuota in user-files upload/save routes
* fix(db): await checkStorageQuota in save-result route (missed second call site)
* feat(db): sqlite-to-postgres migrator with CLI and first-boot import
* fix(db): migrator error context, honest force semantics, boot-hook fatal, null-variance tests
* test: run suite against per-file postgres databases via testcontainers
- Add tests/global-setup.ts: spins up a Postgres testcontainer,
creates a migrated template database once per vitest run.
- Rewrite tests/setup/per-fork-env.ts: each test file (forks pool)
clones the template into its own database via CREATE DATABASE ...
TEMPLATE, preserving the same per-file isolation granularity.
- Update vitest.config.ts: add globalSetup, pg alias, update comment.
- Fix tests/integration/test-server.ts: remove DB_PATH mkdir, async
runMigrations, async db operations, remove SQLite WAL checkpoint.
- Fix 21 unit test db/index mocks: add pool and closeDb exports.
- Fix 8 unit test files: add async/await for now-async permission,
audit, and analytics functions.
- Fix 18 integration test files: convert sync .run()/.all()/.get()
to async drizzle patterns, add async to callbacks.
- Production change: apps/api/src/routes/teams.ts: cast COUNT(*)
to ::int so Postgres returns a number instead of bigint string.
* fix(db): seed built-in roles, reject NUL bytes, cast COUNT, serialize job persists
- Seed built-in roles (admin, editor, user) at boot via ensureBuiltinRoles()
with onConflictDoNothing, restoring data that legacy SQLite migration 0007
provided via INSERT statements (the pg baseline is DDL-only).
- Reject NUL bytes in login credentials with 401 (postgres rejects \x00 in
text columns; valid usernames never contain NUL, matching 1.x behavior).
- Cast COUNT(*)::int in user-files, audit-log, and roles listing queries so
postgres returns a JS number instead of bigint-as-string.
- Serialize fire-and-forget job progress DB writes per jobId so the final
"completed" status is never overwritten by a late-arriving "processing"
write (race condition exposed by async postgres round-trips).
* test: fix teams race, seed roles in test server, poll for job status
- Add missing await to resetTeams() in teams PUT beforeEach (the async
delete raced with the subsequent insert under postgres).
- Call ensureBuiltinRoles() in test server bootstrap so integration tests
have the same built-in roles as production.
- Replace fixed 100ms flushPersist delay with a polling helper that waits
for terminal job status, eliminating timing-dependent failures caused by
postgres network round-trip latency.
* test: make heic temp-file cleanup assertion resilient to concurrent workers
Use a set-based diff instead of raw file count when checking that
decodeHeic cleans up temp files. Other concurrent test workers can
create heic-in-*/heic-out-* files in the shared tmpdir, inflating the
"after" count and causing spurious failures under full-suite load.
* fix(db): align builtin-role seed to post-0010 legacy state; test polish
* feat(docker): three-container compose (app, postgres, redis) with boot wait and migrations
* fix(docker): set TEST_DATABASE_URL so containerized tests skip testcontainers
* chore(docker): test compose project name, clearer 1.x upgrade comment, unref probe timer
* feat(enterprise): enforce D15 license boundary; move s3 storage into packages/enterprise
* fix(enterprise): restore lazy aws-sdk loading; community installs load no s3 code at boot
* fix(enterprise): boundary check catches dynamic imports; document getS3 concurrency
* feat(db)!: SnapOtter 2.0 phase 1 foundation: postgres, migrator, compose stack
BREAKING CHANGE: SQLite is no longer the runtime database. Deployments now
require Postgres (and Redis, used from phase 2). Existing installs migrate
with SQLITE_MIGRATE_PATH or 'pnpm --filter @snapotter/api migrate:sqlite'.
* fix(ci): postgres service + fresh e2e database per run; ignore unfixable torch CVE-2025-3000
- Add Cloudflare Pages deployment for landing page (snapotter.com) and
docs (docs.snapotter.com)
- Create deploy-landing.yml and update deploy-docs.yml workflows
- Update CI to ignore apps/landing/** paths
- Fix logo transparency (remove white background) across all apps
- Recreate social-preview.png with SnapOtter branding
- Update all docs URLs from GitHub Pages to docs.snapotter.com
- Update VitePress config: light theme default, fix llms.txt paths
- Add .vitepress/cache/ and .env.* to gitignore
Remove hardcoded --platform=linux/amd64 from Dockerfile so buildx produces
native arm64 images for Apple Silicon and Raspberry Pi. Add audit logging
for auth events, harden file storage with extension whitelists and
double-extension attack prevention, reject null-byte buffers in validation,
add data-testid attributes to all tool settings components, update
deployment docs with architecture notes and correct CI workflow references,
and fix unit test mock to match throwWithMessage error extraction.
Remove docs/superpowers/, .claude/ config, and PRD.md from version
control (kept locally via .gitignore). Update README, CHANGELOG,
VitePress docs, and .env.example to reflect recent features: Files
page, teams, admin settings, persistent storage, and various API
improvements.
Consolidate all access to localhost:1349 — Vite dev server serves on 1349
and proxies API calls to an internal dev port (13490). Production API
defaults to 1349. Also includes strip-metadata improvements, UI component
updates, and compress operation fixes.
