mirror of
https://github.com/snapotter-hq/SnapOtter.git
synced 2026-08-03 07:46:42 +02:00
SnapOtterandGitHub
935861bced
Two release-pipeline defects found while pre-flighting 2.2.0, plus the image hardening that the second one exposed. The release job would have died immediately after pushing the v2.2.0 tag. draftRelease was turned on in #649 and never executed, and GitHub's /releases/tags/{tag} endpoint does not return draft releases, so all nine tag lookups in release.yml would have 404'd against the draft semantic-release had just created. Verified against this repo with a throwaway draft: the tag endpoint 404s while gh release view reads it and /releases/{id} returns the same REST shape. Every site now resolves the numeric id first, so existing jq expressions are untouched. The unfixed-vulnerability gate was measuring almost nothing. The blocking Trivy steps run ignore-unfixed, and trivy-unfixed-gate.mjs was meant to cover the remainder but defaults to CRITICAL with neither call site passing --severity. An unfixed HIGH was gated by nothing, and the arm64 image carried 79 of them while the summary read clean. Rather than document 79 findings, the image lost what it did not need: libde265 1.1.1 and libheif 1.23.1 are now built from source (the old libheif pin was itself affected by CVE-2026-3950, and Debian's libde265 1.0.11 was the decoder every .heic upload actually reached), and xvfb, wget and openssh-client are purged. 15 CVEs left the image outright and the HIGH gap fell to 65, each now carrying a rationale verified against the running container. curl gets its own section: bookworm-backports has a fixed 8.14.1, so claiming no fix was available would have been false. It is recorded as a declined fix. Verified on both architectures: gate exits 0, the source-built libde265 is the one libheif links, and HEIC, RAW, ImageMagick, Sharp AVIF and headless chromium all still work after the purge.
SnapOtter Test Suite
Directory Structure
tests/
unit/ # Vitest unit tests (~185 files)
ai/ # AI sidecar bridge + bundle tests
api/ # API route/config/middleware tests
doc-engine/ # doc-engine package tests
features/ # Feature flag + enterprise tests
fixtures/ # Fixture guard tests (resolve/budget/manifest)
image-engine/ # image-engine operation tests
media-engine/ # media-engine FFmpeg wrapper tests
security/ # Security-focused unit tests
shared/ # Shared constants/i18n/permissions tests
web/ # React component tests (tsx)
integration/ # Full-API integration tests (~250 files)
fixtures/ # Fixture integrity tests (decode through Sharp/ffprobe/qpdf)
generated/ # Generated test matrices (format x tool, pairwise settings)
platform/ # Platform tests (jobs, pipelines, batch, auth, OIDC)
security/ # Security integration tests (SSRF, XXE, path traversal)
tools/ # Per-tool integration tests
e2e/ # Playwright specs (main app, Chromium primary)
e2e-analytics/ # Analytics flow e2e specs
e2e-docker/ # Docker container e2e specs
e2e-docs/ # VitePress docs site e2e specs
e2e-editor/ # Konva image editor e2e specs
e2e-landing/ # Astro landing site e2e specs
fixtures/ # Shared test fixtures (see below)
helpers/ # Test utilities (pairwise, zod-pict, tool defaults)
setup/ # Vitest setup (per-fork DB clone, BullMQ prefix isolation)
baseline/ # Phase 0 baseline snapshots for parity verification
benchmark/ # Performance benchmarks
qa/ # QA checklists and manual test plans
Fixtures
Fixtures live in tests/fixtures/ in a modality-first layout:
tests/fixtures/
index.ts # Typed registry -- import paths from here, not raw strings
manifest.json # sha256 + bytes + provenance for all valid/ assets
gen-manifest.mjs # Re-stamps manifest hashes from disk
gen-synthetic-content.mjs # Regenerates CC0 synthetic content (QR, barcode, OCR, etc.)
LICENSES.md # Provenance audit trail
image/
valid/ # Real images for depth tests (portraits, scenes, OCR, etc.)
formats/ # Tier-1 tiny format samples (sample.png, sample.arw, ...)
edge/ # Edge-case images (1x1, blank, extreme aspect)
hostile/ # Malformed/bomb/polyglot files for rejection tests
video/
valid/ # Real videos (Big Buck Bunny CC-BY heroes, TTS speech)
formats/ # Tier-1 tiny format samples (tiny.mp4, tiny.webm, ...)
hostile/ # Truncated/malformed video files
audio/
valid/ # Real audio (TTS speech in 6 formats, tagged MP3)
formats/ # Tier-1 tiny format samples (tiny.mp3, tiny.wav, ...)
hostile/ # Zero-byte/malformed audio files
document/
valid/ # PDFs (3-page, 6-page, encrypted, OCR-scanned)
formats/ # Tier-1 tiny doc samples (tiny.docx, tiny.epub, ...)
edge/ # Edge-case docs (remote-img.html for SSRF tests)
hostile/ # Truncated/garbage documents
data/
valid/ # Data files (CSV, JSON, XML, YAML, TSV, ZIP)
security/ # XXE/SSRF test vectors (SVG)
Two-Tier Fixture Strategy
Tier-1: Tiny Synthetics (in formats/, edge/, hostile/)
Small, project-generated files for format-compatibility matrices and fast unit tests.
These are always-regeneratable by the generator scripts and are used for breadth.
Tier-2: Real Heroes (in valid/)
Content-representative files for integration and fidelity tests. Includes:
- Big Buck Bunny video clips (CC-BY, Blender Foundation)
- macOS TTS speech in 6 audio formats (CC0)
- Project-generated synthetics (QR codes, barcodes, OCR text images, PDFs)
- Portrait/scene photos for AI tool tests (some UNVERIFIED-REVIEW, see manifest)
Every valid/ file is tracked in manifest.json with sha256 hash, byte count,
source URL, and license. The manifest guard test (fixture-manifest.test.ts)
verifies disk matches manifest on every run.
Typed Fixture Registry
Always import fixture paths from tests/fixtures/index.ts:
import { fixtures, readFixture, fixtureDir } from "../../fixtures/index.js";
// Named paths (type-checked, IDE autocomplete)
const pngPath = fixtures.image.base.png200;
const heroMp4 = fixtures.video.hero.mp4;
// Read file bytes
const buffer = readFixture(fixtures.image.base.png200);
// Format accessor (for matrix iteration)
const arwPath = fixtures.image.formats("arw");
const mp4Path = fixtures.video.tiny("mp4");
// Directory paths (for glob/scan patterns)
const hostileDir = fixtureDir.image.hostile;
Guard Tests
Four guard tests protect fixture integrity:
| Guard | Location | What it checks |
|---|---|---|
| fixture-resolve | unit/fixtures/ |
Every registry path exists and is non-empty |
| fixture-budget | unit/fixtures/ |
Per-extension size caps prevent bloat |
| fixture-manifest | unit/fixtures/ |
sha256/bytes in manifest.json match disk |
| fixture-integrity | integration/fixtures/ |
Real heroes decode through Sharp/ffprobe/qpdf |
Additional parity guards:
| Guard | Location | What it checks |
|---|---|---|
| tool-registry-drift | unit/shared/ |
Frontend registry matches shared TOOLS catalog |
| tool-route-drift | integration/ |
API routes match shared TOOLS catalog |
| i18n-completeness | unit/shared/ |
All 21 locales have all required keys |
Adding a Fixture
- Add the file to the appropriate modality directory (
image/valid/,video/formats/, etc.) - Add a named key in
tests/fixtures/index.ts - Run
node tests/fixtures/gen-manifest.mjs(forvalid/files) - Fill in
sourceUrlandlicenseinmanifest.json - Update
LICENSES.mdif the file has third-party origin - Run guards:
pnpm vitest run tests/unit/fixtures/
Generator Scripts
Three generators produce deterministic synthetics. All are idempotent and skip
files that already exist (to avoid breaking manifest hashes from encoder-version
differences). Pass --force to gen-synthetic-content.mjs to overwrite.
| Script | Scope | Outputs |
|---|---|---|
scripts/generate-test-fixtures.mjs |
Tier-1 format samples | video/formats/tiny.mp4, audio/formats/tiny.{mp3,wav}, document/formats/tiny.{docx,xlsx,epub,html,md}, document/valid/encrypted.pdf |
scripts/generate-hostile-fixtures.mjs |
Hostile inputs | image/hostile/{truncated.jpg,zero-byte.png,garbage.jpg,png-bytes.jpg,bomb-50000x50000.png} |
tests/fixtures/gen-synthetic-content.mjs |
CC0 synthetics | QR codes, barcodes, OCR text images, PDFs, SVG logo, multi-face placeholder, animated GIF, tagged MP3, metadata MP4 |
The generators do NOT touch committed real heroes (BBB videos, TTS speech, UNVERIFIED-REVIEW photos).
Running Tests
# Full suite
pnpm test # All unit + integration tests
pnpm test:unit # Unit tests only
pnpm test:integration # Integration tests only
# Single file
pnpm vitest run tests/unit/my-test.test.ts
pnpm vitest run tests/integration/my-test.test.ts
# Fixture guards only
pnpm vitest run tests/unit/fixtures
# E2E (Playwright)
pnpm test:e2e # Main app (chromium + serial + visual projects)
pnpm test:e2e:landing # Landing site
pnpm test:e2e:docs # VitePress docs
pnpm test:e2e:analytics # Analytics flows
pnpm test:docker # Full Docker container test
# Editor e2e (no root alias)
pnpm playwright test --config playwright.editor.config.ts
# Single e2e spec
pnpm playwright test tests/e2e/my-test.spec.ts
Environment Requirements
Unit and integration tests need Postgres + Redis:
docker compose -f docker-compose.dev.yml up -d
export DATA_DIR=/tmp/so-test-data && mkdir -p "$DATA_DIR"
Each Vitest fork gets its own Postgres DB clone + workspace + isolated BULLMQ_PREFIX
via tests/setup/per-fork-env.ts (testcontainers for Postgres). 30s test timeouts.
E2E Projects
The main Playwright config defines three projects:
- chromium: Parallel-safe specs, 2 workers (override via
PW_WORKERS) - chromium-serial: Global-state specs, 1 worker
- chromium-visual: Screenshot comparison specs (platform-suffixed baselines)
Auth setup project logs in and saves storage state. The webServer block spins up
both dev servers with a fresh test DB.
Environment Switches
| Variable | Effect |
|---|---|
FULL_MATRIX=1 |
Run all format x tool combinations (nightly) |
FUZZ=1 + FUZZ_RUNS=N |
Property-based fuzz testing (nightly) |
PW_WORKERS=N |
Override Playwright worker count |
CI Pipeline
PRs run: lint, typecheck, unit, integration (4 shards), e2e smoke. Nightly: full e2e, cross-browser, docker suite, full matrix + fuzz, coverage.