Follow-ups to the v2.1.0 Sentry telemetry overhaul, found by reviewing live release:2.1.0 events:
- error_code tag was empty because reportError read only the top-level err.code; add extractErrorCode() to walk the cause chain (pg SQLSTATE, node E-code, else first short code).
- InputValidationError from a tool's processV2 in the worker was logged as error_class=bug; classify it as expected for any source. Worker-side ZodError stays a bug (schema drift).
- AI dispatcher timeouts rejected with a bare Error, which the sanitizer scrubbed to a message-less "Error: Error"; reject with an operational SafeError (code "timeout") at both timeout sites.
Each fix written failing-test-first; affected and adjacent unit suites green plus full CI (integration + e2e).
Removes Sentry tracing entirely (BullMQ idle polling burned 4.8M transactions in 2 days at the baked 0.1 rate), decouples PostHog sampling, and replaces the type-only error scrub with a vetted-field sanitizer plus SafeError/ToolInputError contracts. One classified capture path with per-signature throttles and a per-process ceiling makes storms impossible (NODE-1E was 4,541 events from one 30s loop). Browser errors move to a dedicated web Sentry project with their own source maps. Adds the SNAPOTTER_TELEMETRY runtime kill switch and silences test fleets.
Crash fixes: remote 204/304 SSRF process kill (NODE-20), conversion-preset boot crash loop (NODE-21), Redis version preflight + unhandled subscribe rejection (NODE-1T), Sign PDF on plain-http origins (NODE-1K/1M), wavesurfer/pdf.js teardown rejections (NODE-1P/1N), bundle-import ZlibError to 400 (NODE-1Z), chart-maker input errors declassified (NODE-1H/1J), asset requests skip the session DB lookup (NODE-1D).
Fixes 15 defects found by a max-effort multi-agent review of the last 6
merged PRs (#388, #390, #391, #392, #393, #394), all adversarially
verified before fixing.
Install queue + dispatcher (the serious cluster):
- features.ts: finalize the installer child exactly once. A failed spawn
fires both "error" and "close", and the second event released the file
lock and active slot that pump() had just handed to the next queued
bundle, letting two pip processes write the same venv concurrently.
Outcome recording now happens before pump() so the next bundle's first
progress frame cannot race the previous install's bookkeeping.
- feature-status.ts: keep failed-install errors in a per-bundle map
instead of the single progress slot. With the queue auto-starting the
next install, the slot was overwritten within seconds and a failed
install vanished without ever surfacing to GET /features.
- bridge.ts: scope child lifecycle per process (stopped-children set +
request generation tags) instead of an instance-wide shuttingDown flag
that the next spawn reset. A stale SIGTERMed child's late close event
could record a phantom crash (5 of which permanently disable the
dispatcher), null out the freshly spawned child, and reject the new
child's pending requests. The request-timeout kill path still counts
as a real crash.
- install_feature.py: the pre-write disk re-check measured ai_dir's
filesystem even when budgeting the cross-filesystem copy that lands on
the venv's disk; now each budget is checked against the filesystem the
bytes actually land on, so ENOSPC cannot strike mid-write and leave
site-packages half overwritten.
Behavior regressions:
- embed-subtitles: preserve pre-existing subtitle tracks (0:s?) and MKV
attachments (0:t?) that the -map 0:v:0/0:a? rewrite silently dropped;
data streams stay unmapped on purpose (the actual MPEG remux fix). The
new subtitle maps first so the language tag hits the right stream.
- usage-survey-overlay: fail closed when the settings fetch fails; the
fail-open path rendered the blocking survey against an unhealthy API
and soft-locked admins, the lock-out class #392 fixed.
- features-store: queued bundles poll instead of each holding an SSE
connection (Install All could pin 7 EventSources and exhaust the
browser's 6-per-origin HTTP/1.1 limit, hanging the whole app);
listenToProgress closes any prior stream and stops any poll before
subscribing; installAll skips bundles already installing or queued.
Contracts, tests, i18n:
- openapi.yaml: add "queued" to the features status enum and document
downloadBytes/installedBytes (Schemathesis conformance).
- feature-lifecycle e2e: queue transcription (~0.5 GB) instead of ocr
(~6 GB) and give the test a budget that covers both install drains
(the stacked waits exceeded the old 900s timeout).
- docker-compose.qa.yml: parameterize the host port (QA_APP_PORT) so
QA_PROJECT_NAME concurrent stacks can actually bind.
- compare + watermark-image: restore per-input error attribution
("Invalid first/second image", "Invalid watermark image") lost in the
shared-handler migration.
- ai-features-section: the "{size} on disk" suffix now goes through
i18n; key added to all 21 locales.
- watermark-image + content-aware-resize: migrate to the shared
inputHandlerFor("image") chain like compare/vectorize/compose, fixing
drift in the inline copies (no SVG sanitize, no RAW extension hint,
no AVIF probe).
Verified: typecheck across 9 workspaces, Biome clean on all changed
files, 584 targeted unit tests and 249 integration tests green
(including real-ffmpeg embed-subtitles runs). One unit test updated to
the new poll-while-queued contract with a single-EventSource assertion.
Claude-Session: https://claude.ai/code/session_017mR1HiHaf3a1BmUtrHX4j3
Found and fixed during a full local Docker build validation (amd64/arm64, all
four fleet targets, AI bundle installs, QA harness) and the follow-up bug
sweep requested afterward. None of the affected scripts run in CI, so these
had been silently broken indefinitely.
- docker/feature-manifest.json: pythonVersion was a flat "3.11", but the
amd64 base (Ubuntu 24.04) ships Python 3.12 while arm64 (Debian bookworm)
ships 3.11. Changed to a per-arch object matching the file's existing
convention.
- tests/qa/api-sweep.mts and verify-ai.mts: bare "@snapotter/shared" import
can't resolve since tests/ is not a pnpm workspace member, making both
silently unrunnable via their own documented command on any fresh
checkout. Switched to a relative import.
- tests/qa/generate-ledger.mts: wrote to docs/qa/ without creating the
directory first; docs/ is gitignored except COMMUNITY_GUIDE.md, so a fresh
checkout threw ENOENT.
- Seven QA Playwright spec files (input-preview, settings,
settings-extended, multifile, output-preview, pipeline-ui, smoke) had
~115 fixture() calls using directory names that don't exist. Resolved
every call programmatically against the real fixture tree.
- packages/ai/src/bridge.ts: AI dispatcher restart (happens on every bundle
install) was falsely counted as a crash, risking permanent dispatcher
disable after enough legitimate restarts within the crash window. Added a
shuttingDown flag checked at all three recordCrash() call sites.
- packages/image-engine/src/operations/auto-enhance.ts: image-enhancement
hung 40+ seconds on large RAW photos (confirmed on a real 20.2MP file) in
Sharp's .clahe() step, whose cost scales with total pixel count regardless
of tile size. Added a 16-megapixel cap above which CLAHE is skipped;
verified against the real file (40+s -> 2.0s) with no regression to other
RAW formats or normal-sized images. Fixing this surfaced a second,
smaller bug where the saturation step's CLAHE compensation boost was
keyed off the raw toggle instead of whether CLAHE actually ran.
- Two QA-harness robustness gaps closed per "fix everything, even the small
bugs": the passport-photo/erase-object input-preview tests now skip
cleanly with a clear reason on a container without their AI bundle
installed, and docker-compose.qa.yml's hardcoded project/container name
(the actual root cause of a mid-validation container swap between two
concurrent sessions) is now parameterized via QA_PROJECT_NAME.
Full validation report is local-only per repo convention.
The persistent Python dispatcher rejects scripts whose feature bundle is not
installed, but the per-request fallback (used when the dispatcher is down, e.g.
restarting right after a model repair) spawned scripts directly and bypassed
that gate. Behavior was therefore inconsistent: a gated script would fail under
the dispatcher but run under the fallback -- the "works once after a repair"
symptom from the original report.
- add packages/ai/src/feature-gate.ts: SCRIPT_BUNDLE_MAP + missingBundleForScript,
mirroring TOOL_BUNDLE_MAP in dispatcher.py, reading the same installed.json and
failing closed exactly like dispatcher._get_installed_bundles()
- runPerRequest now rejects with "feature_not_installed" (the same message the
dispatcher path surfaces) when a gated script's bundle is not installed
- unit tests for the gate, plus a drift test pinning the TS map to dispatcher.py
Closes#327
The upscale function called runPythonWithProgress without a timeout parameter,
defaulting to the bridge's 10-minute hard limit. On CPU-only systems like
Synology NAS devices, Real-ESRGAN 4x upscaling easily exceeds this for modest
images. Additionally, when the timeout fired on the dispatcher path, the Python
process was left running and blocked all subsequent AI operations.
This fix adds an adaptive timeout based on input megapixels, scale factor, and
GPU availability (180s/effective-MP on CPU, 30s/effective-MP on GPU, floor of
10 minutes). It also kills the dispatcher on timeout so subsequent requests can
proceed via a fresh restart.
Closes#119
The dispatcher was lazy-initialized on first AI request, but a race
condition meant the first call always missed it (dispatcherReady still
false) and fell through to cold per-request Python. initDispatcher()
starts the dispatcher eagerly and returns a Promise that resolves with
GPU status once ready (or after a timeout).
The close handler called recordCrash() unconditionally, even for exit
code 0 (normal MAX_REQUESTS restart). After 5 normal cycles within 60s
the dispatcher was permanently disabled. Now only non-zero exits count.
Add ~500 new E2E tests and ~300 new integration tests covering:
- 24 new GUI E2E specs: navigation, responsive layout, keyboard shortcuts,
tool UI for all 35 non-AI tools, batch/pipeline workflows, settings/RBAC,
visual regression, accessibility, and performance budgets
- 3 new E2E-Docker specs: batch workflows, advanced pipelines, cross-format
- 1 new adversarial integration test: memory pressure, corrupted files,
unicode filenames, extreme dimensions, pipeline/batch edge cases
- 29 expanded integration test files: HEIC/HEIF input, large files, parameter
boundaries, batch processing, format edge cases across all tools
- Cross-format matrix expanded: 641 tests covering every tool x 18 formats
- AI bridge unit tests expanded: lifecycle, tool modules, error propagation
- Unit test gaps filled: analytics, tool-registry, web stores
Also fixes:
- vitest.config.ts: exclude e2e-docs and e2e-landing from Vitest runner
- AI E2E specs: add sidecar health check to skip gracefully when Python
AI backend is not running instead of timing out
- Unit: 1,353 tests (42 files) — +256 new tests covering AI bridge
modules, image-engine sharpen/optimize-for-web, Zustand stores, and
icon-map validation
- Integration: 1,640 tests (57 files) — +826 new tests across all
tool routes, pipeline/progress/batch infrastructure, user-files,
edit-metadata, and a 321-test cross-format matrix
- E2E-Docker: 389 passing (20 spec files) — 6 new spec files for
batch processing, format conversion, layout, optimization,
watermark/overlay, and pipeline chains. Tests verified against fresh
Docker container with all 6 AI bundles installed.
Bug fixes discovered during testing:
- fix(compress): SVG/BMP/exotic formats crashed Sharp encoder — added
format-safety fallback to PNG
- fix(rate-limit): increase default login attempt limit from 10 to 500
per minute — previous value caused false test failures and is too
restrictive for a self-hosted app
- fix(auth.setup): wait for consent button visibility before clicking
to prevent flaky E2E-Docker auth setup
