Commit Graph
17 Commits
Author SHA1 Message Date
SnapOtterandGitHub 5d36ac06d8 feat(analytics): build-time bake + telemetry depth (#336)
Bake PostHog + Sentry into the published Docker image (SNAPOTTER_ANALYTICS
build arg, codegen script). Delete entire consent system. Move event emission
to BullMQ worker. Add cross-tier identity stitching, Sentry performance
tracing on both tiers, frontend funnel events. Fix stateful regex bug.

86 files changed, 1593 insertions(+), 3747 deletions(-)
2026-06-24 11:05:39 +08:00
SnapOtterandGitHub 35e18d8b79 fix: GPU deployment robustness (6 fixes from end-to-end testing on an RTX 4070) (#334)
* fix(docker): pin CUDA base to 12.6 so the GPU image starts on R560+ drivers

The amd64 base nvidia/cuda:12.9.2-cudnn-runtime bakes a cuda>=12.9 driver gate enforced by nvidia-container-toolkit at container start, so the image fails to launch on common production drivers (e.g. 570.x / CUDA 12.8). The AI bundles are all cu126 wheels and the image installs libcublas-12-6, so 12.9 was misaligned with the workload. Pin to nvidia/cuda:12.6.3-cudnn-runtime-ubuntu24.04 to match the wheels and lower the driver floor to R560+.

* fix(ai): broaden OOM detection so the rembg lighter-model fallback fires

onnxruntime/CUDA allocation failures surface as 'Failed to allocate memory for requested buffer', CUBLAS_STATUS_ALLOC_FAILED, or bad_alloc, not just 'out of memory'. The background-removal and transparency-fixer fallback-to-lighter-model paths only matched the literal 'out of memory', so the fallback was dead code and transparency-fixer (default birefnet-hr-matting) always failed with an allocation error. Add isMemoryAllocError() and use it in both checks.

* fix(ai): use bundled PaddleOCR models so OCR runs offline

ocr.py passed no model dirs to PaddleOCR, so PaddleX resolved models from ~/.paddlex and downloaded them from HuggingFace at runtime (slow first use, broken air-gapped), ignoring the models the OCR bundle ships in MODELS_PATH; it also pulled doc-orientation/unwarping models that are not bundled. Pin detection, recognition and textline models to the bundled dirs in MODELS_PATH (per language) and disable use_doc_orientation_classify / use_doc_unwarping, with per-component fallback when a model is absent. Verified: OCR runs with zero HuggingFace requests.

* fix(docker): add CAP_KILL so container shutdown is graceful

cap_drop: ALL without re-adding KILL meant tini (PID 1, root) could not forward SIGTERM to the gosu-dropped snapotter process (root minus CAP_KILL cannot signal a different UID). docker stop logged '[FATAL tini] forwarding signal: Operation not permitted', never delivered the signal, and fell back to SIGKILL after the 10s timeout. Add KILL to cap_add in both compose files. Verified: docker stop completes in 0s with SIGTERM delivered (exit 143) and no FATAL tini.

* fix(ai): serialize bundle installs against AI jobs to prevent sidecar segfault

A feature bundle install rewrites the shared Python venv (pip + copytree of site-packages/*.so) as a background subprocess, with no coordination against AI tool jobs that dlopen native libs (torch / onnxruntime CUDA) from the same venv; a job loading a shared object while it is overwritten segfaults the sidecar. Add a process-wide async mutex (venv-lock.ts): bridge.run() acquires it before every AI script and the install route holds it across the installer subprocess. Both run in the same Node process so a module-level lock suffices. Verified: concurrent install + AI job produces zero segfaults and the job serializes behind the install.

* fix(ai): make the venv lock read/write so concurrent AI jobs are not serialized

The first cut used an exclusive mutex, which (a) deferred the dispatcher spawn by a microtask and broke unit tests that synchronously drive the mocked spawn, and (b) serialized AI jobs against each other, removing the dispatcher's by-id request multiplexing. Make it a writer-preferring read/write lock: AI jobs are shared readers (with a synchronous fast path so spawn still happens in-tick) and a bundle install is the exclusive writer. Verified: all 764 AI unit tests pass.

* fix(ai): degrade OCR to Tesseract on CPU-only hosts instead of segfaulting

The amd64 AI bundle ships paddlepaddle-gpu, whose native libs dlopen
libcuda.so.1 at import and segfault on a host without a GPU (libcuda is the
driver lib, injected only by nvidia-container-toolkit on GPU hosts). The
segfault crashed the shared long-lived AI dispatcher and, after a few attempts,
tripped the bridge crash-recovery permanent-disable, wedging all AI until a
container restart. The standalone ocr tool defaults to quality=balanced
(PaddleOCR), so it hit this on every CPU-only deployment; ocr-pdf already
hardcoded Tesseract and was unaffected.

ocr.py now gates the PaddleOCR tiers on gpu_available(): balanced/best
transparently fall back to fast (Tesseract, CPU-capable) when no usable GPU is
present, and run_paddleocr_v5/run_paddleocr_vl refuse before importing paddle so
the GPU build is never dlopen'd on CPU. GPU hosts are unchanged.

Verified on a CPU-only Windows/WSL2 box: ocr returns Tesseract text across
repeated runs with the dispatcher staying healthy (no wedge).
2026-06-23 18:39:51 +08:00
SnapOtter d61226496b fix(security): harden rate limits, Redis auth, resource caps, and error sanitization
- Lower LOGIN_ATTEMPT_LIMIT default from 30 to 10 (brute-force protection)
- Lower RATE_LIMIT_PER_MIN default from 1000 to 300
- Add Redis authentication (requirepass) with REDIS_PASSWORD env var
- Add Redis maxmemory 512mb cap to prevent unbounded growth
- Add mem_limit: 1g to Postgres and Redis containers
- Strip internal file paths from all error responses (defense-in-depth)
- Add startup warnings for default admin/Postgres/Redis credentials
- Update security test expectations for new defaults
2026-06-20 00:53:11 +08:00
SnapOtterandGitHub 9b64a96bcd fix(docker): restore Postgres/Redis in GPU compose stack + pause image publishing
Restore Postgres 17 + Redis 8 services in docker-compose-gpu.yml; re-add if:false publish pause on the release.yml docker job (main had lost it).
2026-06-19 18:39:35 +08:00
SnapOtter ace41168bc fix(security): harden API against pentest findings
- Default TRUST_PROXY=false to prevent XFF rate limit bypass (PT-01)
- Return 400 instead of 500 on malformed JSON input (PT-03)
- Default MAX_PIPELINE_STEPS=20 to prevent DoS (PT-04)
- Validate clientJobId length (max 128) across all routes (PT-06)
- Add security headers to all reply.hijack() streaming responses (PT-07)
- Sanitize usernames in audit log to prevent stored XSS (PT-08)
- Block TRACE method with 405 response (PT-10)
- Add 429 RateLimited response to OpenAPI spec (PT-12)
- Default MAX_SVG_SIZE_MB=50 to limit SVGZ decompression (PT-13)
- Pin Dockerfile base images by digest
- Sanitize OIDC IdP error and sub claim in audit log
- Sync Docker compose/Dockerfile defaults with env.ts
2026-06-07 21:54:27 +08:00
SnapOtterandGitHub 5a32e29b8c fix(security): security audit and hardening (#207)
* fix(security): harden SVG sanitizer, rate limiting, and analytics defaults

- SVG: add control-char stripping in href values to block whitespace/null-byte
  obfuscated javascript: URIs; block <feImage> with external href (SSRF via
  SVG filter primitives); expand test suite to 32 inline bypass payloads
- Rate limiting: add per-route limits on tool endpoints (60/min) and batch
  (20/min); fix compose files defaulting RATE_LIMIT_PER_MIN to 0 which mapped
  to 50,000 in code; simplify rate limit registration to use env.ts default
- Analytics: default ANALYTICS_ENABLED to false so self-hosters do not
  unknowingly send telemetry
- Docker: add --max-time 5 and -s flags to compose healthcheck curl commands

* fix: remove stale login limit bypass, reduce error log noise, clean up fixtures

- Fix getLoginAttemptLimit() ignoring LOGIN_ATTEMPT_LIMIT when global rate
  limit exceeded 1000/min, which let the global limit override the stricter
  per-route login brute-force protection
- Downgrade rate limit 429 responses from error to warn level in the global
  error handler to avoid log noise and unnecessary Sentry reports
- Log 4xx client errors at warn level instead of error level
- Remove 11 orphaned SVG attack fixture files replaced by inline test payloads
2026-06-07 10:43:52 +08:00
SnapOtterandGitHub 10a85cf37e feat: add Docker _FILE secret convention for sensitive env vars (#205)
Support reading secrets from mounted files instead of plain-text
environment variables, following the standard Docker/Kubernetes
convention used by MariaDB, Postgres, and Stirling-PDF.

Supported vars: DEFAULT_PASSWORD, S3_ACCESS_KEY_ID,
S3_SECRET_ACCESS_KEY, OIDC_CLIENT_SECRET, COOKIE_SECRET,
SNAPOTTER_LICENSE_KEY.
2026-06-06 21:18:14 +08:00
SnapOtterandGitHub 32055e0b32 fix: correct Docker image name in compose files (#198)
Closes #182
2026-06-05 17:04:52 +08:00
SnapOtter ca2ef5b3f4 feat: add OIDC/SSO authentication (#3)
Add OpenID Connect (OIDC) authentication alongside existing
username/password login. Users can log in via any standards-compliant
OIDC provider (Keycloak, Authentik, Authelia, Google, Azure AD, Okta)
while preserving full backward compatibility.

- OIDC Fastify plugin with lazy discovery, PKCE, cookie-based sessions
- Login page OIDC button, auth hook updates, settings dialog badges
- 28 integration tests, OIDC setup guide with provider examples
- Fix pre-existing test failures (content-aware-crop, watermark, SVGZ)
- WAL checkpoint fix for SQLite test stability

Closes #3

# Conflicts:
#	apps/api/src/lib/env.ts
#	apps/api/src/routes/tools/watermark-image.ts
#	pnpm-lock.yaml
#	tests/integration/color-palette.test.ts
#	tests/integration/compare.test.ts
#	tests/integration/watermark-image.test.ts
2026-05-14 22:31:26 +08:00
SnapOtter 4e64ee2779 fix(security): comprehensive security audit and hardening
Auth: login rate limit 30/min (was 500), global rate limit 1000/min (was
unlimited), password/username max lengths on all Zod schemas, session
invalidation on role change, API key legacy scan bounded to 100 keys.

SVG: hardened regex sanitizer with CDATA stripping, XML entity decoding,
set/animate/iframe/embed blocking, comprehensive data: URI blocking,
use element external href blocking. 11 attack payload fixtures added.

SSRF: fixed DNS rebinding TOCTOU by pinning resolved IPs via custom
HTTP/HTTPS agents. Added 6to4 and NAT64 to blocked IPv6 ranges.

Docker: capability dropping (cap_drop ALL + minimal cap_add), resource
limits (4g/8g mem, 512/1024 pids), healthcheck timeout, password
removed from startup banner, default password warning comments.

Network: CSP and HSTS applied in all environments (not just production),
stack traces removed from all error responses, internal paths stripped
from error details, per-route rate limits on uploads (60/min) and URL
fetches (200/hour).

Files: exclusive temp file creation (O_EXCL), disk space circuit
breaker, per-user storage quotas, settings payload 64KB size guard.

Python sidecar: script name allowlist in dispatcher, minimal environment
for subprocess spawns.

Dependencies: fixed 6 production CVEs (drizzle-orm, fastify, fast-uri,
@fastify/static, next, archiver/lodash). Pinned all GitHub Actions to
SHA hashes.

114 security tests added. Full OWASP Top 10 penetration test matrix
verified against production Docker container (30/30 pass after
hardening).
2026-05-13 21:33:50 +08:00
SnapOtter 71496db6c5 feat(oidc): add OIDC environment variables to Docker configuration 2026-05-13 19:07:33 +08:00
SnapOtter 0309e0f680 chore: deploy to Cloudflare Pages and update branding
- Add Cloudflare Pages deployment for landing page (snapotter.com) and
  docs (docs.snapotter.com)
- Create deploy-landing.yml and update deploy-docs.yml workflows
- Update CI to ignore apps/landing/** paths
- Fix logo transparency (remove white background) across all apps
- Recreate social-preview.png with SnapOtter branding
- Update all docs URLs from GitHub Pages to docs.snapotter.com
- Update VitePress config: light theme default, fix llms.txt paths
- Add .vitepress/cache/ and .env.* to gitignore
2026-04-24 18:06:29 +08:00
AshimandGitHub 4c9dc6e38e fix: Docker hardening, security, and deployment readiness for V1 (#82)
Phase 1 — Docker Artifact Optimization:
- Replace broad `COPY . .` with targeted frontend source copies (API/Python
  changes no longer bust the frontend build cache)
- Replace build-essential with gcc/g++ (leaner runtime)
- Fix LOG_LEVEL=debug → info for production
- Harden .dockerignore (exclude worktrees, IDE, CI, test artifacts)

Phase 2 — State & Persistence:
- Add PUID/PGID support in entrypoint.sh for bind mount compatibility
- Guard against PUID=0/PGID=0 to prevent accidental root execution
- Evict conflicting system users (e.g. node:1000) before UID remap

Phase 3 — Security:
- Always register @fastify/rate-limit so login brute-force protection
  works even when global rate limit is disabled (RATE_LIMIT_PER_MIN=0)
- Add trustProxy support (TRUST_PROXY env var, default true) so rate
  limiting and audit logs use real client IPs behind reverse proxies
- Strip stack traces from 500 error responses in production
- Fix FSTDEP022 deprecation: maxParamLength → routerOptions
- Add multi-file guard on single-file tool endpoint with clear error
  message pointing to the /batch endpoint

Phase 4 — Graceful Degradation:
- Add consolidated hardware detection startup banner (GPU, rate limit,
  upload limit, proxy status)
- Add ConnectionMonitor component with health polling and reconnecting
  overlay that auto-dismisses when the server comes back

Phase 5 — Deployment Docs:
- Rewrite deployment.md with copy-paste CPU and GPU compose templates
- Add hardware requirements table (minimum, recommended, heavy workloads)
- Add PUID/PGID bind mount documentation
- Add complete env var reference table
- Add reverse proxy guides for Nginx, Nginx Proxy Manager, Traefik,
  and Cloudflare Tunnels
2026-04-21 10:19:08 +08:00
ashim-hq ee8e9861a7 feat: docker unlimited defaults + i18n strings for admin UI
- docker-compose: log rotation 10m×3 → 50m×5 for power users
- docker-compose: add shm_size 2gb for AI workloads
- i18n: add strings for Limits & Resources admin panel
2026-04-20 21:52:07 +08:00
ashim-hq 7daf2377f0 feat: unlimited by default — env vars default to 0 (unlimited)
Add 10 new configurable env vars (MAX_WORKER_THREADS, PROCESSING_TIMEOUT_S,
MAX_PIPELINE_STEPS, MAX_CANVAS_PIXELS, MAX_SVG_SIZE_MB, MAX_LOGO_SIZE_KB,
MAX_SPLIT_GRID, MAX_PDF_PAGES, SESSION_DURATION_HOURS, LOGIN_ATTEMPT_LIMIT).

Change existing defaults: MAX_UPLOAD_SIZE_MB=0, MAX_BATCH_SIZE=0,
MAX_MEGAPIXELS=0, RATE_LIMIT_PER_MIN=0, MAX_USERS=0, CONCURRENT_JOBS=0.

Zero means unlimited/auto-detect throughout the application.
2026-04-20 21:35:15 +08:00
ashim-hq ac5fdfb841 fix: update e2e tests for current UI, increase timeouts for CPU environments, and fix auth bypass bug
- Fix SKIP_MUST_CHANGE_PASSWORD not affecting login/session API responses,
  causing frontend redirect even when the env var was set after user creation
- Increase Docker Playwright timeouts (test: 600s, expect: 60s, AI processing: 300s)
  to support CPU-only self-hosted environments
- Increase default rate limit from 100 to 50000 req/min for self-hosted deployments
- Fix OCR tests: use filechooser pattern (Dropzone has no static file input),
  correct enhance checkbox default, rewrite for actual fixture behavior
- Fix remove-bg tests: update quality labels (Balanced→HD, Best→Max)
- Fix noise-removal skip guard: use waitFor() instead of instant isVisible()
- Fix automate pipeline save test: clean up stale E2E pipelines before assertion
2026-04-20 15:03:56 +08:00
ashim-hq e20418c3e1 feat: enhance noise removal tool tests and update Docker configurations for GPU support 2026-04-20 12:27:07 +08:00