Move user resolution logic (match by externalId, auto-link by email,
auto-create with user limit check) into a shared module that both
OIDC and SAML callbacks can use. Includes sanitizeUsername and
findUniqueUsername helpers. Preserves all existing OIDC behavior
and audit events.
Add isToolAuditEnabled() helper that checks the auditToolOperations
DB setting (off by default) or falls back to the enterprise
audit_export feature flag. The createToolRoute factory now emits a
TOOL_EXECUTED audit entry on successful tool execution when enabled,
using a fire-and-forget pattern so a failed audit write never blocks
the tool response.
Static data-driven test covering 90 tools with 551 variations. For each
tool that has configurable settings, every enum value, boundary number,
and boolean flag is exercised through the HTTP endpoint. The test asserts
no 500s occur and validates response shape for 200/202 results.
Coverage spans image (29 tools), video (20), audio (15), PDF/document
(21), and data (5) modalities.
Closes#71. Several error paths in the worker could leak internal
filesystem paths (/tmp/workspace, /data/ai/venv, /app) through SSE
frames, resultPayload objects, and Redis batch-error lists. The
existing stripInternalPaths call at worker.ts line 345 only covered
the single-file processToolJob catch block.
Wrapped 6 additional call sites:
- processPipelineStep: prevError from DB and catch errorMsg
- processPipelineFinalize: composed errorMsg reaching SSE,
recordChildOutcome, and resultPayload
- processBatchChild: catch error reaching recordChildOutcome
and resultPayload
- processBatchFinalize: manifest errorMsg from DB rows
- recordChildOutcome (batch-progress.ts): defense-in-depth
strip before Redis rpush
Added 10 unit tests for stripInternalPaths covering /tmp, /data,
/app, /opt, /home, /workspace, multi-path messages, safe passthrough,
and pipeline-step wrapping.
On 1.x-to-2.0 upgrade the /data/ai/venv already existed with stale
packages, so the entrypoint skipped the cp from /opt/venv and 2.0
Python tools failed with ImportError (#85).
Dockerfile now writes a SHA-256 of `pip freeze` to /opt/venv/.venv-version
at build time. The entrypoint compares that stamp against the copy on the
volume; on mismatch it does a clean nuke + re-copy and resets
installed.json so the UI correctly shows bundles as needing reinstall.
Models in /data/ai/models survive the refresh, so bundle reinstall only
reruns pip (model downloads are idempotent).
Backward compatible: if the image has no stamp (old build), the
entrypoint falls through to the existing skip-if-exists behavior.
* feat(infra): add dev compose stack with postgres and redis
* fix(infra): comment dev env defaults until wired; harden dev compose restart and start_period
* chore(deps): add pg driver and testcontainers for postgres migration
* feat(db): translate schema to drizzle pg-core (timestamptz, boolean, pgEnum, jsonb)
Schema translation (apps/api/src/db/schema.ts):
- sqlite-core -> pg-core, all 10 tables preserved 1:1
- integer(mode:'timestamp') -> timestamp({ withTimezone: true })
- integer(mode:'boolean') -> boolean
- jobs.status text enum -> pgEnum('job_status') with same 4 values
- 7 columns changed from text to jsonb: jobs.inputFiles, jobs.settings,
pipelines.steps, apiKeys.permissions, roles.permissions,
auditLog.details, userFiles.toolChain
- settings.value stays text, jobs.error stays text, jobs.progress stays real
jsonb call-site sweep (removed JSON.stringify on writes, JSON.parse on reads):
- apps/api/src/routes/roles.ts: permissions read/write (3 sites)
- apps/api/src/routes/api-keys.ts: permissions write + read (2 sites)
- apps/api/src/routes/audit-log.ts: details read (1 site)
- apps/api/src/routes/pipeline.ts: steps write + read (2 sites)
- apps/api/src/routes/progress.ts: inputFiles write (2 sites)
- apps/api/src/routes/tool-factory.ts: toolChain read + write (2 sites)
- apps/api/src/routes/user-files.ts: toolChain read + write (4 sites)
- apps/api/src/permissions.ts: roles.permissions read (1 site)
- apps/api/src/lib/audit.ts: details write (1 site)
- apps/api/src/plugins/auth.ts: apiKeys.permissions read (1 site)
* refactor(db): type jsonb columns via $type and note raw CTE conversion requirements
* feat(db): archive sqlite migrations and generate postgres baseline
* chore(db): dockerignore legacy migrations, add archive breadcrumb, fix trailing newline
* feat(db): pg pool connection, advisory-locked boot migrations, DATABASE_URL config
* fix(db): friendly fatal on unreachable postgres, idempotent closeDb, lock-key convention note
* refactor(db): async drizzle calls in plugins, lib, permissions
* fix(api): analytics never throws, typed permission guard, single-query session invalidation
* refactor(db): async drizzle calls across all routes and bootstrap
Convert every route file and index.ts from sync SQLite drizzle
patterns to async node-postgres drizzle:
- .all() removed (bare await on select)
- .get() converted to destructured [row] = await ...
- .run() removed (bare await on insert/update/delete)
- .changes replaced with .rowCount (null-guarded) in progress.ts
- sqlite import removed from user-files.ts; raw CTEs converted to
await db.execute(sql`...`) with postgres-dialect recursive CTEs
- ChainRow types updated: tool_chain is parsed jsonb (string[] | null),
created_at is Date (timestamptz) with no * 1000 conversion
- All requirePermission() guard calls awaited (security: unawaited
async guard returns truthy Promise, bypassing permission check)
- All hasEffectivePermission() and getPermissions() calls awaited
- All auditLog() calls awaited (preserves write-before-response order)
- trackEvent() and captureException() left un-awaited (fire-and-forget
by design, guaranteed never-throw)
- ensureAnonymousUser(), startCleanupCron(), recoverStaleJobs() awaited
in bootstrap sequence
- ensureInstanceId() and ensureDefaultSettings() made async
Files converted: 14 (index.ts + 12 route files + tools/index.ts)
* fix(db): await async checkStorageQuota in user-files upload/save routes
* fix(db): await checkStorageQuota in save-result route (missed second call site)
* feat(db): sqlite-to-postgres migrator with CLI and first-boot import
* fix(db): migrator error context, honest force semantics, boot-hook fatal, null-variance tests
* test: run suite against per-file postgres databases via testcontainers
- Add tests/global-setup.ts: spins up a Postgres testcontainer,
creates a migrated template database once per vitest run.
- Rewrite tests/setup/per-fork-env.ts: each test file (forks pool)
clones the template into its own database via CREATE DATABASE ...
TEMPLATE, preserving the same per-file isolation granularity.
- Update vitest.config.ts: add globalSetup, pg alias, update comment.
- Fix tests/integration/test-server.ts: remove DB_PATH mkdir, async
runMigrations, async db operations, remove SQLite WAL checkpoint.
- Fix 21 unit test db/index mocks: add pool and closeDb exports.
- Fix 8 unit test files: add async/await for now-async permission,
audit, and analytics functions.
- Fix 18 integration test files: convert sync .run()/.all()/.get()
to async drizzle patterns, add async to callbacks.
- Production change: apps/api/src/routes/teams.ts: cast COUNT(*)
to ::int so Postgres returns a number instead of bigint string.
* fix(db): seed built-in roles, reject NUL bytes, cast COUNT, serialize job persists
- Seed built-in roles (admin, editor, user) at boot via ensureBuiltinRoles()
with onConflictDoNothing, restoring data that legacy SQLite migration 0007
provided via INSERT statements (the pg baseline is DDL-only).
- Reject NUL bytes in login credentials with 401 (postgres rejects \x00 in
text columns; valid usernames never contain NUL, matching 1.x behavior).
- Cast COUNT(*)::int in user-files, audit-log, and roles listing queries so
postgres returns a JS number instead of bigint-as-string.
- Serialize fire-and-forget job progress DB writes per jobId so the final
"completed" status is never overwritten by a late-arriving "processing"
write (race condition exposed by async postgres round-trips).
* test: fix teams race, seed roles in test server, poll for job status
- Add missing await to resetTeams() in teams PUT beforeEach (the async
delete raced with the subsequent insert under postgres).
- Call ensureBuiltinRoles() in test server bootstrap so integration tests
have the same built-in roles as production.
- Replace fixed 100ms flushPersist delay with a polling helper that waits
for terminal job status, eliminating timing-dependent failures caused by
postgres network round-trip latency.
* test: make heic temp-file cleanup assertion resilient to concurrent workers
Use a set-based diff instead of raw file count when checking that
decodeHeic cleans up temp files. Other concurrent test workers can
create heic-in-*/heic-out-* files in the shared tmpdir, inflating the
"after" count and causing spurious failures under full-suite load.
* fix(db): align builtin-role seed to post-0010 legacy state; test polish
* feat(docker): three-container compose (app, postgres, redis) with boot wait and migrations
* fix(docker): set TEST_DATABASE_URL so containerized tests skip testcontainers
* chore(docker): test compose project name, clearer 1.x upgrade comment, unref probe timer
* feat(enterprise): enforce D15 license boundary; move s3 storage into packages/enterprise
* fix(enterprise): restore lazy aws-sdk loading; community installs load no s3 code at boot
* fix(enterprise): boundary check catches dynamic imports; document getS3 concurrency
* feat(db)!: SnapOtter 2.0 phase 1 foundation: postgres, migrator, compose stack
BREAKING CHANGE: SQLite is no longer the runtime database. Deployments now
require Postgres (and Redis, used from phase 2). Existing installs migrate
with SQLITE_MIGRATE_PATH or 'pnpm --filter @snapotter/api migrate:sqlite'.
* fix(ci): postgres service + fresh e2e database per run; ignore unfixable torch CVE-2025-3000
Closes the "e2e never runs in CI" hole. Adds per-PR e2e smoke gate,
nightly full-suite workflows, parallel vitest forks (per-fork DBs),
Playwright parallel/serial/visual projects against production builds,
metadata-generated test suites (drift guards, hostile inputs, format
matrix, pairwise settings, property-based fuzz), Stryker mutation
testing, Schemathesis API fuzz, coverage ratchet, and fixes for three
session-poisoning bugs that caused 200+ serial-bucket failures.
Bug fix included: favicon/split/bulk-rename could hang clients forever
when ZIP streaming failed after reply.hijack().
Five unit tests crashed because they imported modules that transitively
reached db/index.ts, which eagerly calls `new Database()` at module load
time. Added vi.mock for db/index.js matching the pattern used by 16
other passing API unit tests.
Moved cleanup.test.ts to tests/integration/ since it uses a real SQLite
database, runs migrations, and inserts rows.
- Create 53 per-tool VitePress documentation pages with accurate
parameters from Zod schemas, example requests, and response formats
- Add root llms.txt for LLM-friendly repo browsing
- Fix OpenAPI spec: add auth and 422 error schemas to
edit-metadata/inspect and strip-metadata/inspect sub-routes
- Fix tool count inconsistency (52 -> 53) across landing site,
e2e tests, and local docs
- Rename color-adjustments.ts to adjust-colors.ts to match tool ID
- Update VitePress sidebar with all 8 tool categories and top nav
* feat(a11y): add i18n keys for ARIA labels and screen reader text
* fix(security): harden API against pentest findings
- Default TRUST_PROXY=false to prevent XFF rate limit bypass (PT-01)
- Return 400 instead of 500 on malformed JSON input (PT-03)
- Default MAX_PIPELINE_STEPS=20 to prevent DoS (PT-04)
- Validate clientJobId length (max 128) across all routes (PT-06)
- Add security headers to all reply.hijack() streaming responses (PT-07)
- Sanitize usernames in audit log to prevent stored XSS (PT-08)
- Block TRACE method with 405 response (PT-10)
- Add 429 RateLimited response to OpenAPI spec (PT-12)
- Default MAX_SVG_SIZE_MB=50 to limit SVGZ decompression (PT-13)
- Pin Dockerfile base images by digest
- Sanitize OIDC IdP error and sub claim in audit log
- Sync Docker compose/Dockerfile defaults with env.ts
* feat(a11y): convert all hardcoded aria-labels to i18n keys
Replace 49 hardcoded aria-label="..." strings across 25 files with
their corresponding t.a11y.* and t.common.* i18n references. Add
useTranslation import and hook call to 15 components that lacked it.
Zero hardcoded aria-labels remain in the codebase.
* feat(a11y): add aria-labels to icon-only buttons, aria-hidden on decorative icons, sr-only status text
* feat(a11y): add aria-live regions for processing status announcements
* feat(a11y): add skip-nav link, route announcer, main content landmark, and page h1 elements
* feat(a11y): add prefers-reduced-motion support, preserve functional spinners
* feat(a11y): add useFocusTrap hook for modal focus management
* feat(a11y): add focus trapping and dialog roles to all modals
* feat(a11y): add toggle switch roles, form labels, and error association
* fix(a11y): fix contrast failures, touch targets, and add nav landmark to sidebar
* fix(a11y): add role=switch to remaining toggle buttons found in verification sweep
- Fix selective metadata stripping (P1): use Sharp's keepExif()/keepIccProfile()
instead of broken withMetadata({}) that preserved everything
- Fix meme font mapping (P1): add ArchivoBlack and ComicNeue fonts, map
arial-black and comic-sans to correct TTF files instead of Anton
- Fix meme contentType (P2): detect actual output format from Sharp metadata
instead of hardcoding image/png
- Fix info/text-overlay/color-palette i18n (P2): wire up existing translation
keys that were imported but never used
- Fix info and color-palette displayMode (P2): change from before-after to
no-comparison since neither tool produces a processed image
- Add missing i18n keys across all 21 locales
- Update displayMode test assertions
* fix(security): harden SVG sanitizer, rate limiting, and analytics defaults
- SVG: add control-char stripping in href values to block whitespace/null-byte
obfuscated javascript: URIs; block <feImage> with external href (SSRF via
SVG filter primitives); expand test suite to 32 inline bypass payloads
- Rate limiting: add per-route limits on tool endpoints (60/min) and batch
(20/min); fix compose files defaulting RATE_LIMIT_PER_MIN to 0 which mapped
to 50,000 in code; simplify rate limit registration to use env.ts default
- Analytics: default ANALYTICS_ENABLED to false so self-hosters do not
unknowingly send telemetry
- Docker: add --max-time 5 and -s flags to compose healthcheck curl commands
* fix: remove stale login limit bypass, reduce error log noise, clean up fixtures
- Fix getLoginAttemptLimit() ignoring LOGIN_ATTEMPT_LIMIT when global rate
limit exceeded 1000/min, which let the global limit override the stricter
per-route login brute-force protection
- Downgrade rate limit 429 responses from error to warn level in the global
error handler to avoid log noise and unnecessary Sentry reports
- Log 4xx client errors at warn level instead of error level
- Remove 11 orphaned SVG attack fixture files replaced by inline test payloads
- Apply biome formatting fixes to web app components
- Add required S3 credentials to loadEnv test when STORAGE_MODE=s3
- Update bento-grid test tool counts from 52 to 53 for html-to-image
SnapOtter