Fix the Unit Tests CI job: bundleRepo now asserts deepsafe/feature-bundles (intentional, temporary); extractedSize relaxed to >= 0 (best-effort field, build script does not measure uncompressed size). sha256 + compressedSize remain strict. Full unit suite: 4546 passed.
Fixes all integration CI failures on the 2.0 branch.
## What was broken
Two independent root causes:
1. **202 assertion failures** -- Under 4-fork CI parallel load, the 30s
`SYNC_WAIT_MS` sync window can expire before a BullMQ worker finishes a
heavy encode (avif, heic), returning a legitimate `202 {jobId, async: true}`
instead of `200`. Tests that hard-asserted `200` were spuriously failing.
2. **Vitest timeout race** -- `SYNC_WAIT_MS` (30s) and the default Vitest
`testTimeout` (also 30s) fired simultaneously. Vitest won the race,
reporting "Test timed out in 30000ms" instead of the test receiving the
202 response.
## Fixes
- Added `isAsyncFallback()` helper to four integration test files; validates
the `{async: true, jobId}` body shape and returns early so the synchronous
200 path runs full assertions only when warranted.
- Set `vi.setConfig({ testTimeout: 60_000 })` at module level in
`image-enhancement.test.ts` and `format-matrix-comprehensive.test.ts`,
giving a 30s buffer between when `waitForJob()` returns 202 and when
Vitest gives up.
- Bumped explicit matrix timeouts in `format-matrix.test.ts` and
`new-formats.test.ts` from 30s to 60s for the same reason.
- Installed missing CI doc-engine binaries (qpdf, pandoc, libreoffice,
pdfcpu) that were causing unrelated integration failures.
- Fixed E2E smoke specs for 2.0 UI changes (modality selector, tool routes,
validation behavior).
rest.md: correct the tool response shape (JSON {jobId, downloadUrl, ...} for fast / 202 {jobId, async} for long, not the raw file), fix the SSE jobId source (body, with X-Job-Id only on batch/pipeline ZIP streams), complete the permissions list to all 17, and add ai-canvas-expand.
ai.md: reconcile ~14 AI tool parameter tables to the real Zod schemas (remove-background, upscale, ocr, enhance-faces, colorize, noise-removal, red-eye-removal, restore-photo, passport-photo, erase-object, smart-crop, content-aware-resize, transparency-fixer, image-enhancement), add the 6 missing AI tools (background-replace, blur-background, ocr-pdf, ai-canvas-expand, transcribe-audio, auto-subtitles), and fix the dispatcher script diagram and feature-bundle table.
image-engine.md: add 4 undocumented operations (sharpen, sharpen-advanced, color-blindness, edit-metadata) and fix flip/crop/rotate/convert/compress/strip-metadata/color-channels params, output-format count (13), and the info JSON example. supported-formats.md: output count 14 -> 13. Verified with a clean docs build.
Reconcile every tool page's parameters, defaults, and response shape against the tool's Zod settings schema and executionHint in code. Notable fixes: color-palette (add count + format params, hex output, median-cut algorithm), favicon (add 5 params, was documented as having none), qr-generate (add logoDataUri), convert (add ppm/eps/tga formats), video-loudnorm (-16 LUFS not -14), smart-crop (async 202 not sync 200), images-to-video (1080x1080 square), and several output-filename and behavior-note corrections.
Also normalize API endpoint paths to /api/v1/tools/<id> (no modality segment) and standardize curl examples on the Docker API port 1349. Verified with a clean docs build.
Move all 157 tool pages into image/video/audio/pdf/data subfolders so URLs read /tools/<modality>/<id> (e.g. /tools/image/crop). Nest the image sub-categories under an Image group in the sidebar so the nav reads by modality. Add public/_redirects (301, clean + .html forms) mapping every old flat /tools/<id> URL to its new path so inbound links keep working. Rewrite all internal /tools links. Verified with a clean docs build (no dead links).
The Release workflow failed at startup ("workflow file issue", no jobs
created) whenever dispatched. Root cause: release.yml sets top-level
`permissions: {}`, so the ai-bundles reusable-workflow call inherited an
empty scope set. ai-bundles.yml's jobs declare `contents: read` /
`packages: read`, and GitHub rejects a called workflow requesting scopes
the caller never granted -- validated at startup, before any job runs, so
the whole run fails immediately regardless of the job's `if:`.
Grant the two scopes on the call job. Confirmed on a throwaway branch: a
neutralized dispatch went from startup_failure to a clean run with
ai-bundles correctly skipped.
This unblocks every real release; main's release.yml could not start.
Fixes a backlog of integration/unit specs that asserted pre-2.0 behavior and
were failing CI (not caused by recent feature work):
- modality-aware empty-input error is 'No file(s) provided', not /no image/i
(rotate, border, crop, resize, smart-crop, edge-cases, adversarial-extended,
api, tool-factory-route)
- input validation rejects pre-enqueue with a clean 400 in 'error' (was a worker
422 in 'details'): create-zip, extract-zip, merge-csvs
- resolveToolPool defaults unknown tools to the system pool (pool-routing)
- /upload and fetch-urls accept non-image content, validated per-tool at process
time (api, fetch-urls)
- color-adjust legacy aliases were consolidated into adjust-colors: drop the
removed-alias tests; retarget the format-preservation tests
- xml-to-csv gracefully converts a single non-repeating record to a 1-row CSV
- dropzone is multimodal; image-only filtering is opt-in via fileFilter
- factory-multi-input: register the synthetic test tools in the catalog so they
route correctly (file modality for concat; image for the validation-prefix test)
Verified locally: unit 4546 passed, integration 8332 passed, typecheck + lint green.
rembg 2.0.70+ requires numpy>=2.3.0, but the AI bundle pins numpy==1.26.4
(mediapipe, realesrgan/basicsr, codeformer, paddle all need numpy<2). The
unresolvable rembg==2.0.75 + numpy==1.26.4 combination broke pip-audit's
dependency resolution (CI red) and the background-removal bundle build. 2.0.69
is the newest rembg with an unconstrained numpy requirement. Verified: pip-audit
resolves with no unignored vulnerabilities on Python 3.11.
The batch endpoint became modality-aware, so an empty request returns the
generic "No files provided" rather than the legacy image-specific message.
The assertion still matched /no image/i (a pre-existing stale spec on main),
so it failed against the correct behavior. Match /no files/i instead.
Bump all workspace package versions and APP_VERSION to 2.0.0, marking the
official 2.0 release. Removes the stale 1.x .release-notes.md artifact
(semantic-release regenerates release notes). The 2.0/multimodality docs
and rebrand already landed on main via #254 and #261, so this carries only
the version designation forward from the rebrand branch.
BREAKING CHANGE: SnapOtter 2.0 - the platform re-architecture (Postgres 17 +
Redis 8 + BullMQ durable jobs, 157 tools across five modalities) is the 2.0
release line, replacing the 1.x SQLite single-container architecture.
These suites only ever skipped (pandoc is absent in every test env), which hid three latent bugs once the binaries are reachable:
- epub-convert and to-epub are executionHint "long" (async: 202 + jobId), but the tests asserted a synchronous 200 and read the response downloadUrl. Rewrite to the 202 -> poll job row -> download pattern.
- The job-row reads selected schema.jobs.errorMessage, which is not a column (the field is `error` jsonb {message,details}); drizzle threw on the bad select. Read error/error.message instead. (The pdf-chain test had the same bug.)
- The SSRF test asserted the remote <img> URL passes through, but pandoc strips the unmanifested remote ref. Assert the security-relevant property instead: the conversion completes from the book's own content and never fetches+inlines the remote resource (no data: URI).
Verified by building the test image with pandoc: all epub/to-epub tests pass.
The tool factory created its per-request validation scratch dir at tmpdir()/snapotter-scratch/<jobId> -- the same path the BullMQ worker uses for job processing (worker.ts scratchRoot()/<jobId>) whenever SCRATCH_PATH is unset. For sync tools this is harmless because the factory blocks on the worker before its finally cleanup runs. For "long" tools the factory returns 202 immediately and then rm's that dir, racing the worker that is writing the job's input there -- so tools that stage input in the worker scratch dir (epub-convert, to-epub) intermittently failed with ENOENT. Production never hit this because it sets SCRATCH_PATH, diverging the two paths. Give the factory's validation dir a -prep suffix so its cleanup can never touch the worker's job dir.
pnpm test:docker ran pnpm test:ci (vitest --coverage), but the lean test image deliberately skips binary-gated tools (AI model bundles, LibreOffice, etc.), so it can never meet the host-calibrated coverage thresholds -- the container exited non-zero on coverage even with zero test failures. Point the compose command at vitest run so test:docker is a clean functional pass/fail gate; coverage stays enforced on host CI where every tool is present.
Two test-image gaps surfaced by a full pnpm test:ci run:
- s3-storage.test.ts imports @aws-sdk/client-s3 (an enterprise dependency) at module load, but Dockerfile.test never copied packages/enterprise/package.json before pnpm install, so the suite failed to collect. Copy it so the dep installs; the suite then skips cleanly when MinIO is absent.
- EPS batch decode returned 422: ImageMagick reads EPS through the Ghostscript PS coder, but policy.xml left PS/PS2/PS3 at rights=none (only EPS was opened), so convert refused with a policy error before Ghostscript ran. Open the PostScript coders too.
tests/setup/per-fork-env.ts hardcoded SYNC_WAIT_MS=30000 on every fork, overriding whatever the container set, so the docker test image could never grant heavy ops a wider sync window. A 12MP stress-image enhance takes ~34s on the macOS Docker VM (Sharp runs 2-3x slower there), just past the 30s window, so the factory returned 202 and three sync-asserting image-enhancement tests failed.
Honor a higher SYNC_WAIT_MS when provided (30s floor preserved for host/CI), raise it to 120s in docker-compose.test.yml, and make the vitest test/hook timeouts env-overridable so a slow-but-correct job returns 200 rather than tripping a framework timeout. Host and CI behavior is unchanged.
Make the full pnpm test:docker suite pass the env-dependent tests (~85 failures):
- Dockerfile.test: ENV LD_LIBRARY_PATH=/usr/local/lib so the built libheif 1.21 is not shadowed by the base image's older system libheif (heif-dec failed with an undefined-symbol error -> 'No HEIF decoder found' on 72 HEIF tests); add libjxl-tools (JXL) and ghostscript + the ImageMagick policy.xml EPS allow-edit.
- docker-compose.test.yml: SYNC_WAIT_MS=30000 so sync-wait image tools do not fall back to 202 under single-container contention (10 tests).
- install_feature.py: guard tarfile.extractall(filter='data') behind Python>=3.12 (bookworm ships 3.11); the manual entry guards already protect.
- feature-status.test.ts / docker-file-secrets.test.ts: skip the two cases that cannot hold inside the container (/.dockerenv always present; root bypasses chmod). Verified on host: all still pass.
A bare 'vitest run' (pnpm test:ci) collected tests/qa/*.qa.spec.ts (Playwright specs importing @playwright/test -> 'test.describe() called here') and tests/unit/landing/*.test.tsx (React tests importing @landing/app/* and @landing/components/* paths that no longer exist after the landing Next.js->Astro migration). Exclude both. Surfaced by the full pnpm test:docker run; the host unit suite is now green.
acceptedInputs is [.pdf], but pdf-to-image validated by calling mupdf.openDocument(buf, 'application/pdf'); mupdf sniffs the real format and opens JPEGs/PNGs/etc. as 1-page image-documents, returning 200. So non-PDF (incl. truncated/hostile) inputs were accepted, violating the contract and the hostile-input robustness check -- the one pre-existing failure surfaced by the full integration run. Gate all three endpoints (convert/info/preview) on the %PDF- magic bytes. Verified: truncated.jpg -> 400, valid PDF -> 200; the hostile-inputs test passes.
The earlier worker error-logging change logged every job failure at error level, including expected InputValidationErrors (e.g. 'needs at least two audio files') -- flooding error logs with non-actionable user-input rejections (visible across the integration run). Skip validation errors (matched by name, which survives the BullMQ boundary); genuine faults still log at error, and all failures still reach the OTel span.
The full unit suite surfaced two issues from earlier commits on this branch. (1) Importing the app logger into media-input.ts pulled logger.ts -- which builds its pino file transport at module load via join(env.LOG_DIR, ...) -- into the unit-test import graph, throwing at collection time wherever LOG_DIR is unset (integration tests set it; unit tests do not). A low-level modality handler should not depend on the app logger, and a corrupt upload is an expected user error, so drop the import and keep the clean validation message. (2) tool-factory-route.test.ts mocked errors.js without the new friendlyError export; add it.
friendlyError unit test (incl. the false-positive guard); gated integration tests for multi-file video batch and a multi-step video pipeline (regression for the modality-aware batch/pipeline fix). All pass locally; existing image batch (36) and pipeline (37) suites remain green, and the existing gif-to-video webm test now passes with the pix_fmt fix.
The raw-dump detector matched broad content keywords ('conversion failed', 'pixel format', bare 'ffprobe') that appear in legitimate validation messages (e.g. 'SVG conversion failed'), which would wrongly collapse them to the generic error. Narrow it to the unambiguous 'ffmpeg/ffprobe exited N:' prefix, python tracebacks, and crashes; longer/multi-line raw dumps are still caught by the length/line-count check. Found during self-review.
Main already reuses the per-modality input handlers in batch and pipeline
(#244), so the modality-aware reuse from the video QA sweep was redundant.
Port only the remaining unique piece: pass lenient: skipStructuralValidation
to the handler so batched/pipelined PDF tools that opt out of structural
validation behave like the single-file factory path. Keeps main's safer
explicit image-decode chain (HEIC/RAW/SVG/autoOrient) and AI fileId threading.
An unrecognized/corrupt media upload threw an InputValidationError whose message embedded the raw ffprobe stderr ('ffprobe exited 1: ...'). Return a clean 'Unrecognized <kind> file' message to the client and log the raw probe failure via logger.warn instead. Found via the round-2 hostile-input gap test.
Add friendlyError() which collapses raw external-tool failure output (ffmpeg/ffprobe/LibreOffice/qpdf/etc.) into one generic sentence while preserving intentional validation messages and scrubbing internal paths. Apply it at every client-facing error surface in the tool factory and job worker (sync 422, async SSE, pipeline + batch finalize). The full error is still recorded server-side via request.log.error / logger.error and telemetry.
GIFs decode to bgra/gbrap (alpha); libvpx-vp9 rejects those pixel formats so the encoder never opened (ffmpeg exit 234), breaking GIF->WebM for essentially every GIF. Flatten to yuv420p in the webm branch, matching what the mp4 branch already does.
While getting the editor e2e suite green, three "stale test" failures turned
out to be real bugs (per the reporter's hunch that tests might be catching
real issues):
- Layer effects (drop shadow, glows) never applied. The panel wrote effects
into `attrs.effects` through updateObject, but the panel and renderer both
read the object's top-level `effects`, so the toggle never persisted. Add a
dedicated `setObjectEffects` store action and route the panel through it.
- Object flip (transform tool) did nothing. No object renderer applied
`scaleX`/`scaleY`, and the flip negated scale without compensating position.
Apply scale in the renderers and flip in place: mirror points for stroke
objects, negate scale + shift position for sized objects.
(The paint-bucket / pixel-tool coordinate bug and the broken-at-non-100%-zoom
export were fixed in the preceding #259 change.)
Also adds a small "Beta" badge to the editor (welcome heading + nav link) and
repairs ~18 stale editor e2e specs whose selectors/assertions had drifted from
the current UI: the options bar is `h-9` not `h-10` (added a stable
`data-testid`), the menu bar is `h-8`/`bg-background`, the flip button
aria-labels are lowercase, the welcome "Image Editor" heading collides with an
sr-only `<h1>`, the color-picker tabs need a role-scoped selector, and the
magic-wand / flip tests now use deterministic setup and assert the actual
effect instead of fragile screenshot diffs.
Every editor tool that reads or writes raster pixels exported the stage with `stage.toCanvas({ x: 0, y: 0, width, height })`, which bakes in the stage's zoom/pan transform. The captured buffer was the *viewport* (the document scaled and offset by the current zoom/pan), not the document in its own coordinate space, so tools sampled and wrote the wrong pixels: the paint bucket produced a misplaced black rectangle instead of flood-filling the click, the eyedropper read the wrong colour, the magic wand selected the wrong region, and PNG/clipboard export silently produced a scaled/offset image at any zoom other than 100%.
Add `captureDocumentCanvas()`, which normalizes the stage to the document size with an identity transform, renders, captures, and restores -- all synchronously, so there is no visible flicker. Route every pixel capture through it: fill, magic wand, clone stamp, eyedropper, dodge/burn, blur/sharpen/smudge, the adjustments histogram, and the exporter.
The 'rulers render as black bars' part of #259 was fixed in the preceding editor-layout change (#258).
Adds editor-tool-coordinates.spec.ts asserting the paint bucket fills at the clicked location.
The image editor canvas only used part of the viewport, and the right sidebar was a fixed width that could clip its controls on shorter screens.
- Canvas: the canvas container used `flex-1`, but its parent wrapper in editor-page.tsx was not a flex container, so it collapsed to the Konva Stage's content height (~600px), leaving a large inert region below. Make the wrapper a flex container so the canvas fills the available area.
- Rulers: ruler background/ticks were set via `ctx.fillStyle = "var(--color-card)"`, which canvas 2D cannot parse, so the default black fill remained and painted the rulers as solid black bars. Resolve the theme tokens to concrete colors from computed style at draw time (theme-aware).
- Right panel: add a left-edge drag handle to resize the panel (240-480px, persisted to localStorage) and `min-h-0` so the tab content scrolls internally instead of pushing the color controls off-screen.
Adds editor-layout.spec.ts (canvas-fill + resize) and a ruler-not-black regression test. All 7 targeted editor e2e tests pass.
Overhaul the bug report form and restructure issue intake for 2.0:
- Bug report: split actual vs expected behavior, add Affected area and
Specific tool fields for triage, replace the self-defeating required
"latest version" checkbox with a Version field that accepts a tag,
release, or commit (always answerable from any install), modernize
Docker guidance to the Compose stack, add Host OS, and fix the
previously required image-tag field that source users could not fill.
- Move feature requests to GitHub Discussions: delete the feature_request
issue form, rename the orphaned discussion form to ideas.yml so it binds
to the built-in Ideas category, and point config.yml there.
- Add Translation and Documentation issue forms routed to existing labels.
- Remove the roadmap-update discussion form (roadmap is private).
- Reconcile CONTRIBUTING.md, SUPPORT.md, and the published docs
contributing guide with the new routing.
* fix(docs): keep gray-matter on js-yaml 3 so the docs site builds
The js-yaml >=4.2.0 override from #257 forced js-yaml 4 onto gray-matter (used by vitepress and vitepress-plugin-llms), which calls the removed yaml.safeLoad and broke `vitepress build`. Scope a gray-matter>js-yaml ^3.14.1 override so gray-matter keeps the v3 API (build-time, trusted frontmatter only) while app code stays on js-yaml 4.2.0+.
* docs: add per-tool reference pages for all 157 tools, with a modality sidebar
Generate /tools/<id> pages for the 104 tools that lacked one (video 29, audio 17, document 36, data 10, and 12 newer image tools), matching the existing page format (API endpoint, parameters from the OpenAPI spec, curl example, response, notes). Async/AI tools document the 202+SSE flow and feature-bundle requirement.
Sidebar: add Video / Audio / PDF & Documents / Data groups with per-tool links, fold the 12 new image tools into the existing image categories, and replace the placeholder rest.md-anchor group. Docs site builds cleanly (157 pages, no dead links).
SnapOtter