mirror of
https://github.com/snapotter-hq/SnapOtter.git
synced 2026-08-03 07:46:42 +02:00
SnapOtterandGitHub
2d8b57c57f
Three defects from #645, all of which let the server report something that was not true. Tool access was enforced per route, so it drifted. createToolRoute calls requireToolAccess and the factory tools were fine, but all 45 hand-written routes had to remember the same call and none of them did. A role without tools:use could run image-to-pdf, svg-to-raster, erase-object, favicon, qr-generate, upscale, sign-pdf and the rest. The issue described this as affecting two routes; it was every one of them. The check now lives in a single preHandler keyed off the tool the router matched, so it covers sub-paths (/batch, /info, /preview, /analyze, /inspect) and any route added later without that route opting in. Ids no tool claims stay unresolved, which keeps an unknown or misfiled tool a 404 rather than telling an unauthorized caller which ids exist. Resolution reads request.routeOptions.url, the pattern the app itself registered, rather than parsing request.url a second time. find-my-way decodes before matching, so an independent parse disagrees with the router and the router wins: `/api/v1/tools/image/%66avicon` ran favicon while the gate saw no tool at all. Absolute-form request targets slipped it the same way. Taking the router's own answer removes the disagreement. A ZIP stream that failed after the 200 headers were out called reply.raw.end(). On a chunked response that is indistinguishable from success, so a client kept an archive with no central directory believing it whole. Worse, a source stream that errored had no listener: the request hung until it timed out and the error surfaced as unhandled. A poisoned-storage probe reproduced both. The socket is destroyed instead, and every source stream is listened to. svg-to-raster additionally ran its append loop past the hijack with no try/catch, where a throw leaves Fastify logging and walking away with the socket neither ended nor destroyed. pdf-to-image is fixed alongside the other two: it shipped in #643 with the destroy half but not the listener, so it hung the same way. A zero-byte upload was dropped during parsing. The client pairs results with its own file list by index, so every later result shifted onto the wrong file: one document's output was presented as another's, under another's name, while the file that actually converted was marked "not found in batch results". Empty parts now keep their slot and fail in place with a reason. Two tests in adversarial-extended.test.ts asserted the old zero-byte behavior, including a comment that batch "silently skips zero-byte parts". They now pin the replacement: still rejected, nothing processed, but the caller is told which files were empty instead of being told it sent none. A guard walks the whole catalog and fails if any of the 241 tools answers anything but 403 for a role without tools:use, so a tool cannot escape the gate by being registered in a shape nobody thought to sample. Fixes #645
SnapOtter Test Suite
Directory Structure
tests/
unit/ # Vitest unit tests (~185 files)
ai/ # AI sidecar bridge + bundle tests
api/ # API route/config/middleware tests
doc-engine/ # doc-engine package tests
features/ # Feature flag + enterprise tests
fixtures/ # Fixture guard tests (resolve/budget/manifest)
image-engine/ # image-engine operation tests
media-engine/ # media-engine FFmpeg wrapper tests
security/ # Security-focused unit tests
shared/ # Shared constants/i18n/permissions tests
web/ # React component tests (tsx)
integration/ # Full-API integration tests (~250 files)
fixtures/ # Fixture integrity tests (decode through Sharp/ffprobe/qpdf)
generated/ # Generated test matrices (format x tool, pairwise settings)
platform/ # Platform tests (jobs, pipelines, batch, auth, OIDC)
security/ # Security integration tests (SSRF, XXE, path traversal)
tools/ # Per-tool integration tests
e2e/ # Playwright specs (main app, Chromium primary)
e2e-analytics/ # Analytics flow e2e specs
e2e-docker/ # Docker container e2e specs
e2e-docs/ # VitePress docs site e2e specs
e2e-editor/ # Konva image editor e2e specs
e2e-landing/ # Astro landing site e2e specs
fixtures/ # Shared test fixtures (see below)
helpers/ # Test utilities (pairwise, zod-pict, tool defaults)
setup/ # Vitest setup (per-fork DB clone, BullMQ prefix isolation)
baseline/ # Phase 0 baseline snapshots for parity verification
benchmark/ # Performance benchmarks
qa/ # QA checklists and manual test plans
Fixtures
Fixtures live in tests/fixtures/ in a modality-first layout:
tests/fixtures/
index.ts # Typed registry -- import paths from here, not raw strings
manifest.json # sha256 + bytes + provenance for all valid/ assets
gen-manifest.mjs # Re-stamps manifest hashes from disk
gen-synthetic-content.mjs # Regenerates CC0 synthetic content (QR, barcode, OCR, etc.)
LICENSES.md # Provenance audit trail
image/
valid/ # Real images for depth tests (portraits, scenes, OCR, etc.)
formats/ # Tier-1 tiny format samples (sample.png, sample.arw, ...)
edge/ # Edge-case images (1x1, blank, extreme aspect)
hostile/ # Malformed/bomb/polyglot files for rejection tests
video/
valid/ # Real videos (Big Buck Bunny CC-BY heroes, TTS speech)
formats/ # Tier-1 tiny format samples (tiny.mp4, tiny.webm, ...)
hostile/ # Truncated/malformed video files
audio/
valid/ # Real audio (TTS speech in 6 formats, tagged MP3)
formats/ # Tier-1 tiny format samples (tiny.mp3, tiny.wav, ...)
hostile/ # Zero-byte/malformed audio files
document/
valid/ # PDFs (3-page, 6-page, encrypted, OCR-scanned)
formats/ # Tier-1 tiny doc samples (tiny.docx, tiny.epub, ...)
edge/ # Edge-case docs (remote-img.html for SSRF tests)
hostile/ # Truncated/garbage documents
data/
valid/ # Data files (CSV, JSON, XML, YAML, TSV, ZIP)
security/ # XXE/SSRF test vectors (SVG)
Two-Tier Fixture Strategy
Tier-1: Tiny Synthetics (in formats/, edge/, hostile/)
Small, project-generated files for format-compatibility matrices and fast unit tests.
These are always-regeneratable by the generator scripts and are used for breadth.
Tier-2: Real Heroes (in valid/)
Content-representative files for integration and fidelity tests. Includes:
- Big Buck Bunny video clips (CC-BY, Blender Foundation)
- macOS TTS speech in 6 audio formats (CC0)
- Project-generated synthetics (QR codes, barcodes, OCR text images, PDFs)
- Portrait/scene photos for AI tool tests (some UNVERIFIED-REVIEW, see manifest)
Every valid/ file is tracked in manifest.json with sha256 hash, byte count,
source URL, and license. The manifest guard test (fixture-manifest.test.ts)
verifies disk matches manifest on every run.
Typed Fixture Registry
Always import fixture paths from tests/fixtures/index.ts:
import { fixtures, readFixture, fixtureDir } from "../../fixtures/index.js";
// Named paths (type-checked, IDE autocomplete)
const pngPath = fixtures.image.base.png200;
const heroMp4 = fixtures.video.hero.mp4;
// Read file bytes
const buffer = readFixture(fixtures.image.base.png200);
// Format accessor (for matrix iteration)
const arwPath = fixtures.image.formats("arw");
const mp4Path = fixtures.video.tiny("mp4");
// Directory paths (for glob/scan patterns)
const hostileDir = fixtureDir.image.hostile;
Guard Tests
Four guard tests protect fixture integrity:
| Guard | Location | What it checks |
|---|---|---|
| fixture-resolve | unit/fixtures/ |
Every registry path exists and is non-empty |
| fixture-budget | unit/fixtures/ |
Per-extension size caps prevent bloat |
| fixture-manifest | unit/fixtures/ |
sha256/bytes in manifest.json match disk |
| fixture-integrity | integration/fixtures/ |
Real heroes decode through Sharp/ffprobe/qpdf |
Additional parity guards:
| Guard | Location | What it checks |
|---|---|---|
| tool-registry-drift | unit/shared/ |
Frontend registry matches shared TOOLS catalog |
| tool-route-drift | integration/ |
API routes match shared TOOLS catalog |
| i18n-completeness | unit/shared/ |
All 21 locales have all required keys |
Adding a Fixture
- Add the file to the appropriate modality directory (
image/valid/,video/formats/, etc.) - Add a named key in
tests/fixtures/index.ts - Run
node tests/fixtures/gen-manifest.mjs(forvalid/files) - Fill in
sourceUrlandlicenseinmanifest.json - Update
LICENSES.mdif the file has third-party origin - Run guards:
pnpm vitest run tests/unit/fixtures/
Generator Scripts
Three generators produce deterministic synthetics. All are idempotent and skip
files that already exist (to avoid breaking manifest hashes from encoder-version
differences). Pass --force to gen-synthetic-content.mjs to overwrite.
| Script | Scope | Outputs |
|---|---|---|
scripts/generate-test-fixtures.mjs |
Tier-1 format samples | video/formats/tiny.mp4, audio/formats/tiny.{mp3,wav}, document/formats/tiny.{docx,xlsx,epub,html,md}, document/valid/encrypted.pdf |
scripts/generate-hostile-fixtures.mjs |
Hostile inputs | image/hostile/{truncated.jpg,zero-byte.png,garbage.jpg,png-bytes.jpg,bomb-50000x50000.png} |
tests/fixtures/gen-synthetic-content.mjs |
CC0 synthetics | QR codes, barcodes, OCR text images, PDFs, SVG logo, multi-face placeholder, animated GIF, tagged MP3, metadata MP4 |
The generators do NOT touch committed real heroes (BBB videos, TTS speech, UNVERIFIED-REVIEW photos).
Running Tests
# Full suite
pnpm test # All unit + integration tests
pnpm test:unit # Unit tests only
pnpm test:integration # Integration tests only
# Single file
pnpm vitest run tests/unit/my-test.test.ts
pnpm vitest run tests/integration/my-test.test.ts
# Fixture guards only
pnpm vitest run tests/unit/fixtures
# E2E (Playwright)
pnpm test:e2e # Main app (chromium + serial + visual projects)
pnpm test:e2e:landing # Landing site
pnpm test:e2e:docs # VitePress docs
pnpm test:e2e:analytics # Analytics flows
pnpm test:docker # Full Docker container test
# Editor e2e (no root alias)
pnpm playwright test --config playwright.editor.config.ts
# Single e2e spec
pnpm playwright test tests/e2e/my-test.spec.ts
Environment Requirements
Unit and integration tests need Postgres + Redis:
docker compose -f docker-compose.dev.yml up -d
export DATA_DIR=/tmp/so-test-data && mkdir -p "$DATA_DIR"
Each Vitest fork gets its own Postgres DB clone + workspace + isolated BULLMQ_PREFIX
via tests/setup/per-fork-env.ts (testcontainers for Postgres). 30s test timeouts.
E2E Projects
The main Playwright config defines three projects:
- chromium: Parallel-safe specs, 2 workers (override via
PW_WORKERS) - chromium-serial: Global-state specs, 1 worker
- chromium-visual: Screenshot comparison specs (platform-suffixed baselines)
Auth setup project logs in and saves storage state. The webServer block spins up
both dev servers with a fresh test DB.
Environment Switches
| Variable | Effect |
|---|---|
FULL_MATRIX=1 |
Run all format x tool combinations (nightly) |
FUZZ=1 + FUZZ_RUNS=N |
Property-based fuzz testing (nightly) |
PW_WORKERS=N |
Override Playwright worker count |
CI Pipeline
PRs run: lint, typecheck, unit, integration (4 shards), e2e smoke. Nightly: full e2e, cross-browser, docker suite, full matrix + fuzz, coverage.