mirror of
https://github.com/snapotter-hq/SnapOtter.git
synced 2026-08-03 07:46:42 +02:00
main
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 SnapOtterandGitHub
|
d10d0f544f |
fix: release QA hardening across processing, media, security, and CI gates (#649)
A release-readiness QA pass over the whole product. The commits split into defects a user would hit and gates that were reporting green while measuring nothing. ## Fixes that change behaviour Rate limiting was bypassable on every install: TRUST_PROXY defaulted to true, so request.ip came from a client-set header and a forged X-Forwarded-For got past the login limiter. The default is now a private-network trust list. A transient Postgres outage stranded in-flight jobs, leaving finished output on disk with no row pointing at it. A reconciler now resolves those rows and adopts the bytes rather than dropping the work. A Redis connection that moved to a new address wedged every read-blocked consumer, so completions stopped signalling while health still answered 200. Socket timeouts plus subscriber pings recover it. Installing more than one AI bundle left the shared venv multi-versioned and silently broke three tools. The installer now reconciles distributions to one version each. Converting an image to JXL at quality 1 through 4 returned a 500, because libjxl 0.7 rejects the distance those values compute. The quality is floored at what the encoder honours. A missing ffmpeg was also reported to the user as a corrupt upload; it now says the engine is unavailable. RAW uploads reached an unpatched LibRaw on arm64, so it is built from source at 0.22.2, and the release scan was split so it can fail on an unfixed critical instead of hiding it behind ignore-unfixed. ## Gates that could not fail Two mutation lanes ran zero mutants because Stryker crawled the gitignored docs build; coverage discarded its whole report on any failing test; the lint gate skipped root tests, scripts, and two workspaces; and several generated matrices counted a host missing ffmpeg as a passing tool. Each now measures what it claims. Full evidence and the outstanding release items are tracked locally and are not part of this branch. |
||
|
SnapOtterandGitHub
|
8f4235d2c6 |
fix(enterprise): ship enterprise package in prod image + S3, analytics, tracing, queue fixes (#342)
* fix(enterprise): ship enterprise pkg in prod image, full license features, tracing key fallback docker/Dockerfile: COPY packages/enterprise manifest+src into the production stage. Without it, apps/api's workspace link to @snapotter/enterprise dangles and every import() throws (silently caught), so all 19 enterprise features failed closed (enterprise.active=false) regardless of a valid license. scripts/generate-license.mjs: sync PLAN_FEATURES with packages/enterprise/src/license.ts so a --plan enterprise license unlocks all 19 features (was 8) and team unlocks 8. apps/api/src/tracing.ts: accept SNAPOTTER_LICENSE_KEY as a fallback to LICENSE_KEY so distributed_tracing activates with the same key as the rest of the app. * fix(docker): keep scripts/bake-analytics.mjs in build context .dockerignore excluded the whole scripts/ dir (PR #82, V1 hardening), but docker/Dockerfile later added 'COPY scripts/bake-analytics.mjs' for the analytics bake step. A clean production image build therefore fails with 'scripts/bake-analytics.mjs: not found'. The published image build is gated off in CI so this latent break went unnoticed. Exclude scripts/* but re-include the one file the Dockerfile needs. * fix: S3 upload stream, analytics bake reaches API, dedupe retention field, reconcile orphan jobs storage-s3.ts: wrap the upload AsyncIterable in Readable.from() so @aws-sdk/lib-storage accepts it. STORAGE_MODE=s3 file uploads failed with 'Body Data is unsupported format' for every tool because a bare async generator is not a Readable. docker/Dockerfile: COPY the builder-baked analytics baked.ts into the API runtime stage. The API re-copied the committed (off) baked.ts from the build context, so the SNAPOTTER_ANALYTICS build arg had no effect on the API -- and since the SPA reads /api/v1/config/analytics, analytics was off everywhere regardless of the arg. settings-dialog.tsx: remove the duplicate tempFileMaxAgeHours control under Data Retention; it bound the same setting key as the File Management control with a different default, so editing either silently overwrote the other. apps/api/src/index.ts: reconcile orphaned job rows (empty tool_id, never enqueued to BullMQ) at boot so they don't sit in processing/queued forever and inflate the per-user concurrent-job count and the upgrade-check in-flight gate. * fix(web): style the SSO login buttons (they referenced undefined theme tokens) The OIDC/SAML 'Sign in with <provider>' buttons used bg-secondary / text-secondary-foreground, which the web theme never defines (it has primary, background, foreground, muted, border, card, primary-subtle). Those classes resolved to nothing, so the buttons rendered as bare unstyled text on the login page. Restyle: the optional (non-enforced) buttons become white-card outline buttons with a key icon and an orange hover tint, secondary to the primary Login button; the SSO-enforced buttons become solid primary with the icon. * fix: gate S3 behind license, custom-role enterprise perms, wire retention UI, cleanup S3 is a licensed feature, but shipping packages/enterprise in every image removed the implicit gate, so STORAGE_MODE=s3 worked without a license. Enforce isFeatureEnabled('s3_storage') at boot and fail fast if unlicensed. Custom roles can now be granted security:manage / compliance:manage / webhooks:manage (roles.ts ALL_PERMISSIONS + the Roles UI) so admins can build least-privilege compliance/security roles instead of only the built-in admin role. retentionSweep now reads the jobsRetentionDays / auditRetentionDays DB settings the System Settings UI writes (env vars become the fallback default), mirroring how the temp-file sweep reads tempFileMaxAgeHours. Previously those two UI controls were no-ops. Cleanup: drop the never-set snapotter_storage_bytes gauge and the unused MAX_WORKSPACE_SIZE_GB env var; emit tool_client_error to PostHog from the web ErrorBoundary (client crashes were not reaching analytics); add the Python OpenTelemetry packages so the innermost sidecar.<script> span exports; fix the stale 'only local storage' line in the docs; delete two e2e-analytics specs that tested the removed consent UI. * fix(env): restore MAX_WORKSPACE_SIZE_GB default security-auth-hardening.test.ts asserts env.MAX_WORKSPACE_SIZE_GB defaults to 10, so the var is an intentional (tested) default, not dead code. Removing it in the cleanup commit broke that unit test. Keep the declaration. |
||
|
SnapOtterandGitHub
|
1c724d5d21 |
feat(db)!: SnapOtter 2.0 phase 1 foundation: postgres, migrator, compose stack (#216)
* feat(infra): add dev compose stack with postgres and redis
* fix(infra): comment dev env defaults until wired; harden dev compose restart and start_period
* chore(deps): add pg driver and testcontainers for postgres migration
* feat(db): translate schema to drizzle pg-core (timestamptz, boolean, pgEnum, jsonb)
Schema translation (apps/api/src/db/schema.ts):
- sqlite-core -> pg-core, all 10 tables preserved 1:1
- integer(mode:'timestamp') -> timestamp({ withTimezone: true })
- integer(mode:'boolean') -> boolean
- jobs.status text enum -> pgEnum('job_status') with same 4 values
- 7 columns changed from text to jsonb: jobs.inputFiles, jobs.settings,
pipelines.steps, apiKeys.permissions, roles.permissions,
auditLog.details, userFiles.toolChain
- settings.value stays text, jobs.error stays text, jobs.progress stays real
jsonb call-site sweep (removed JSON.stringify on writes, JSON.parse on reads):
- apps/api/src/routes/roles.ts: permissions read/write (3 sites)
- apps/api/src/routes/api-keys.ts: permissions write + read (2 sites)
- apps/api/src/routes/audit-log.ts: details read (1 site)
- apps/api/src/routes/pipeline.ts: steps write + read (2 sites)
- apps/api/src/routes/progress.ts: inputFiles write (2 sites)
- apps/api/src/routes/tool-factory.ts: toolChain read + write (2 sites)
- apps/api/src/routes/user-files.ts: toolChain read + write (4 sites)
- apps/api/src/permissions.ts: roles.permissions read (1 site)
- apps/api/src/lib/audit.ts: details write (1 site)
- apps/api/src/plugins/auth.ts: apiKeys.permissions read (1 site)
* refactor(db): type jsonb columns via $type and note raw CTE conversion requirements
* feat(db): archive sqlite migrations and generate postgres baseline
* chore(db): dockerignore legacy migrations, add archive breadcrumb, fix trailing newline
* feat(db): pg pool connection, advisory-locked boot migrations, DATABASE_URL config
* fix(db): friendly fatal on unreachable postgres, idempotent closeDb, lock-key convention note
* refactor(db): async drizzle calls in plugins, lib, permissions
* fix(api): analytics never throws, typed permission guard, single-query session invalidation
* refactor(db): async drizzle calls across all routes and bootstrap
Convert every route file and index.ts from sync SQLite drizzle
patterns to async node-postgres drizzle:
- .all() removed (bare await on select)
- .get() converted to destructured [row] = await ...
- .run() removed (bare await on insert/update/delete)
- .changes replaced with .rowCount (null-guarded) in progress.ts
- sqlite import removed from user-files.ts; raw CTEs converted to
await db.execute(sql`...`) with postgres-dialect recursive CTEs
- ChainRow types updated: tool_chain is parsed jsonb (string[] | null),
created_at is Date (timestamptz) with no * 1000 conversion
- All requirePermission() guard calls awaited (security: unawaited
async guard returns truthy Promise, bypassing permission check)
- All hasEffectivePermission() and getPermissions() calls awaited
- All auditLog() calls awaited (preserves write-before-response order)
- trackEvent() and captureException() left un-awaited (fire-and-forget
by design, guaranteed never-throw)
- ensureAnonymousUser(), startCleanupCron(), recoverStaleJobs() awaited
in bootstrap sequence
- ensureInstanceId() and ensureDefaultSettings() made async
Files converted: 14 (index.ts + 12 route files + tools/index.ts)
* fix(db): await async checkStorageQuota in user-files upload/save routes
* fix(db): await checkStorageQuota in save-result route (missed second call site)
* feat(db): sqlite-to-postgres migrator with CLI and first-boot import
* fix(db): migrator error context, honest force semantics, boot-hook fatal, null-variance tests
* test: run suite against per-file postgres databases via testcontainers
- Add tests/global-setup.ts: spins up a Postgres testcontainer,
creates a migrated template database once per vitest run.
- Rewrite tests/setup/per-fork-env.ts: each test file (forks pool)
clones the template into its own database via CREATE DATABASE ...
TEMPLATE, preserving the same per-file isolation granularity.
- Update vitest.config.ts: add globalSetup, pg alias, update comment.
- Fix tests/integration/test-server.ts: remove DB_PATH mkdir, async
runMigrations, async db operations, remove SQLite WAL checkpoint.
- Fix 21 unit test db/index mocks: add pool and closeDb exports.
- Fix 8 unit test files: add async/await for now-async permission,
audit, and analytics functions.
- Fix 18 integration test files: convert sync .run()/.all()/.get()
to async drizzle patterns, add async to callbacks.
- Production change: apps/api/src/routes/teams.ts: cast COUNT(*)
to ::int so Postgres returns a number instead of bigint string.
* fix(db): seed built-in roles, reject NUL bytes, cast COUNT, serialize job persists
- Seed built-in roles (admin, editor, user) at boot via ensureBuiltinRoles()
with onConflictDoNothing, restoring data that legacy SQLite migration 0007
provided via INSERT statements (the pg baseline is DDL-only).
- Reject NUL bytes in login credentials with 401 (postgres rejects \x00 in
text columns; valid usernames never contain NUL, matching 1.x behavior).
- Cast COUNT(*)::int in user-files, audit-log, and roles listing queries so
postgres returns a JS number instead of bigint-as-string.
- Serialize fire-and-forget job progress DB writes per jobId so the final
"completed" status is never overwritten by a late-arriving "processing"
write (race condition exposed by async postgres round-trips).
* test: fix teams race, seed roles in test server, poll for job status
- Add missing await to resetTeams() in teams PUT beforeEach (the async
delete raced with the subsequent insert under postgres).
- Call ensureBuiltinRoles() in test server bootstrap so integration tests
have the same built-in roles as production.
- Replace fixed 100ms flushPersist delay with a polling helper that waits
for terminal job status, eliminating timing-dependent failures caused by
postgres network round-trip latency.
* test: make heic temp-file cleanup assertion resilient to concurrent workers
Use a set-based diff instead of raw file count when checking that
decodeHeic cleans up temp files. Other concurrent test workers can
create heic-in-*/heic-out-* files in the shared tmpdir, inflating the
"after" count and causing spurious failures under full-suite load.
* fix(db): align builtin-role seed to post-0010 legacy state; test polish
* feat(docker): three-container compose (app, postgres, redis) with boot wait and migrations
* fix(docker): set TEST_DATABASE_URL so containerized tests skip testcontainers
* chore(docker): test compose project name, clearer 1.x upgrade comment, unref probe timer
* feat(enterprise): enforce D15 license boundary; move s3 storage into packages/enterprise
* fix(enterprise): restore lazy aws-sdk loading; community installs load no s3 code at boot
* fix(enterprise): boundary check catches dynamic imports; document getS3 concurrency
* feat(db)!: SnapOtter 2.0 phase 1 foundation: postgres, migrator, compose stack
BREAKING CHANGE: SQLite is no longer the runtime database. Deployments now
require Postgres (and Redis, used from phase 2). Existing installs migrate
with SQLITE_MIGRATE_PATH or 'pnpm --filter @snapotter/api migrate:sqlite'.
* fix(ci): postgres service + fresh e2e database per run; ignore unfixable torch CVE-2025-3000
|
||
|
AshimandGitHub
|
4c9dc6e38e |
fix: Docker hardening, security, and deployment readiness for V1 (#82)
Phase 1 — Docker Artifact Optimization: - Replace broad `COPY . .` with targeted frontend source copies (API/Python changes no longer bust the frontend build cache) - Replace build-essential with gcc/g++ (leaner runtime) - Fix LOG_LEVEL=debug → info for production - Harden .dockerignore (exclude worktrees, IDE, CI, test artifacts) Phase 2 — State & Persistence: - Add PUID/PGID support in entrypoint.sh for bind mount compatibility - Guard against PUID=0/PGID=0 to prevent accidental root execution - Evict conflicting system users (e.g. node:1000) before UID remap Phase 3 — Security: - Always register @fastify/rate-limit so login brute-force protection works even when global rate limit is disabled (RATE_LIMIT_PER_MIN=0) - Add trustProxy support (TRUST_PROXY env var, default true) so rate limiting and audit logs use real client IPs behind reverse proxies - Strip stack traces from 500 error responses in production - Fix FSTDEP022 deprecation: maxParamLength → routerOptions - Add multi-file guard on single-file tool endpoint with clear error message pointing to the /batch endpoint Phase 4 — Graceful Degradation: - Add consolidated hardware detection startup banner (GPU, rate limit, upload limit, proxy status) - Add ConnectionMonitor component with health polling and reconnecting overlay that auto-dismisses when the server comes back Phase 5 — Deployment Docs: - Rewrite deployment.md with copy-paste CPU and GPU compose templates - Add hardware requirements table (minimum, recommended, heavy workloads) - Add PUID/PGID bind mount documentation - Add complete env var reference table - Add reverse proxy guides for Nginx, Nginx Proxy Manager, Traefik, and Cloudflare Tunnels |
||
|
Siddharth Kumar Sah
|
80e536bcf8 |
chore: remove dead code, add test infrastructure, update docs
- Delete 3 dead files: use-batch-processor.ts, use-i18n.ts, smart-crop.ts (AI package) - Remove dead getJobProgress function and unused runPythonScript wrapper - Remove 6 unused imports across API and web apps - Remove unused shared types (ImageFormat, AppConfig, ApiError, HealthResponse, JobProgress) and constants (SUPPORTED_INPUT_FORMATS/OUTPUT_FORMATS, DEFAULT_OUTPUT_FORMAT) - Remove unused store method (setOriginalBlobUrl) and clean AI package re-exports - Add test infrastructure: vitest config, unit/integration/e2e tests, fixtures, screenshots - Add Docker test infrastructure: Dockerfile.test, docker-compose.test.yml - Add download_models.py for pre-baking AI model weights in Docker - Add filename sanitization utility (apps/api/src/lib/filename.ts) - Update .gitignore to exclude coverage/, *.tsbuildinfo, .superpowers/, test artifacts - Update .dockerignore to exclude test/coverage/IDE artifacts from builds - Update docs: remove smart crop from AI docs (uses Sharp directly), update bridge docs |
||
|
Siddharth Kumar Sah
|
ce03aad10f |
feat: production Docker, Playwright tests, settings API, and bug fixes
- Add user management endpoints (register, list, delete, change password) - Add API key management (create, list, delete) - Add settings persistence endpoints (get, put) - Wire settings dialog to real backend (People, API Keys, System, Security) - Fix login auth flow (window.location.href for full reload) - Fix download URLs returning 401 (make public since UUIDs are unguessable) - Fix border tool shadowColor validation (accept 6-8 hex digits) - Fix remove-bg alpha matting fallback (retry without on failure) - Fix AI tool silent fallbacks (report errors instead of no-ops) - Add checkerboard background to before/after slider for transparency - Add progress bars to all AI tool components - Add Playwright E2E test suite (131 tests across 9 test files) - Rewrite Dockerfile for production (tsx runtime, pre-baked AI models) - Add .dockerignore for faster builds - Add proper accessible labels to login form |