mirror of
https://github.com/snapotter-hq/SnapOtter.git
synced 2026-08-03 07:46:42 +02:00
test: add test suite README and final verification baseline (phase 8)
- Write tests/README.md covering directory structure, fixture registry, two-tier strategy, guard tests, generators, and run commands - Update tests/fixtures/README.md to reflect modality-first layout - Add final.json parity baseline: 13,382 passing, 0 dropped from Phase 0 - Write verification report to docs/superpowers/plans/ (local-only)
This commit is contained in:
SnapOtter
+216
@@ -0,0 +1,216 @@
|
|||||||
|
# SnapOtter Test Suite
|
||||||
|
|
||||||
|
## Directory Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
tests/
|
||||||
|
unit/ # Vitest unit tests (~185 files)
|
||||||
|
ai/ # AI sidecar bridge + bundle tests
|
||||||
|
api/ # API route/config/middleware tests
|
||||||
|
doc-engine/ # doc-engine package tests
|
||||||
|
features/ # Feature flag + enterprise tests
|
||||||
|
fixtures/ # Fixture guard tests (resolve/budget/manifest)
|
||||||
|
image-engine/ # image-engine operation tests
|
||||||
|
media-engine/ # media-engine FFmpeg wrapper tests
|
||||||
|
security/ # Security-focused unit tests
|
||||||
|
shared/ # Shared constants/i18n/permissions tests
|
||||||
|
web/ # React component tests (tsx)
|
||||||
|
integration/ # Full-API integration tests (~250 files)
|
||||||
|
fixtures/ # Fixture integrity tests (decode through Sharp/ffprobe/qpdf)
|
||||||
|
generated/ # Generated test matrices (format x tool, pairwise settings)
|
||||||
|
platform/ # Platform tests (jobs, pipelines, batch, auth, OIDC)
|
||||||
|
security/ # Security integration tests (SSRF, XXE, path traversal)
|
||||||
|
tools/ # Per-tool integration tests
|
||||||
|
e2e/ # Playwright specs (main app, Chromium primary)
|
||||||
|
e2e-analytics/ # Analytics flow e2e specs
|
||||||
|
e2e-docker/ # Docker container e2e specs
|
||||||
|
e2e-docs/ # VitePress docs site e2e specs
|
||||||
|
e2e-editor/ # Konva image editor e2e specs
|
||||||
|
e2e-landing/ # Astro landing site e2e specs
|
||||||
|
fixtures/ # Shared test fixtures (see below)
|
||||||
|
helpers/ # Test utilities (pairwise, zod-pict, tool defaults)
|
||||||
|
setup/ # Vitest setup (per-fork DB clone, BullMQ prefix isolation)
|
||||||
|
baseline/ # Phase 0 baseline snapshots for parity verification
|
||||||
|
benchmark/ # Performance benchmarks
|
||||||
|
qa/ # QA checklists and manual test plans
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fixtures
|
||||||
|
|
||||||
|
Fixtures live in `tests/fixtures/` in a modality-first layout:
|
||||||
|
|
||||||
|
```
|
||||||
|
tests/fixtures/
|
||||||
|
index.ts # Typed registry -- import paths from here, not raw strings
|
||||||
|
manifest.json # sha256 + bytes + provenance for all valid/ assets
|
||||||
|
gen-manifest.mjs # Re-stamps manifest hashes from disk
|
||||||
|
gen-synthetic-content.mjs # Regenerates CC0 synthetic content (QR, barcode, OCR, etc.)
|
||||||
|
LICENSES.md # Provenance audit trail
|
||||||
|
image/
|
||||||
|
valid/ # Real images for depth tests (portraits, scenes, OCR, etc.)
|
||||||
|
formats/ # Tier-1 tiny format samples (sample.png, sample.arw, ...)
|
||||||
|
edge/ # Edge-case images (1x1, blank, extreme aspect)
|
||||||
|
hostile/ # Malformed/bomb/polyglot files for rejection tests
|
||||||
|
video/
|
||||||
|
valid/ # Real videos (Big Buck Bunny CC-BY heroes, TTS speech)
|
||||||
|
formats/ # Tier-1 tiny format samples (tiny.mp4, tiny.webm, ...)
|
||||||
|
hostile/ # Truncated/malformed video files
|
||||||
|
audio/
|
||||||
|
valid/ # Real audio (TTS speech in 6 formats, tagged MP3)
|
||||||
|
formats/ # Tier-1 tiny format samples (tiny.mp3, tiny.wav, ...)
|
||||||
|
hostile/ # Zero-byte/malformed audio files
|
||||||
|
document/
|
||||||
|
valid/ # PDFs (3-page, 6-page, encrypted, OCR-scanned)
|
||||||
|
formats/ # Tier-1 tiny doc samples (tiny.docx, tiny.epub, ...)
|
||||||
|
edge/ # Edge-case docs (remote-img.html for SSRF tests)
|
||||||
|
hostile/ # Truncated/garbage documents
|
||||||
|
data/
|
||||||
|
valid/ # Data files (CSV, JSON, XML, YAML, TSV, ZIP)
|
||||||
|
security/ # XXE/SSRF test vectors (SVG)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Two-Tier Fixture Strategy
|
||||||
|
|
||||||
|
**Tier-1: Tiny Synthetics** (in `formats/`, `edge/`, `hostile/`)
|
||||||
|
Small, project-generated files for format-compatibility matrices and fast unit tests.
|
||||||
|
These are always-regeneratable by the generator scripts and are used for breadth.
|
||||||
|
|
||||||
|
**Tier-2: Real Heroes** (in `valid/`)
|
||||||
|
Content-representative files for integration and fidelity tests. Includes:
|
||||||
|
- Big Buck Bunny video clips (CC-BY, Blender Foundation)
|
||||||
|
- macOS TTS speech in 6 audio formats (CC0)
|
||||||
|
- Project-generated synthetics (QR codes, barcodes, OCR text images, PDFs)
|
||||||
|
- Portrait/scene photos for AI tool tests (some UNVERIFIED-REVIEW, see manifest)
|
||||||
|
|
||||||
|
Every `valid/` file is tracked in `manifest.json` with sha256 hash, byte count,
|
||||||
|
source URL, and license. The manifest guard test (`fixture-manifest.test.ts`)
|
||||||
|
verifies disk matches manifest on every run.
|
||||||
|
|
||||||
|
### Typed Fixture Registry
|
||||||
|
|
||||||
|
Always import fixture paths from `tests/fixtures/index.ts`:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import { fixtures, readFixture, fixtureDir } from "../../fixtures/index.js";
|
||||||
|
|
||||||
|
// Named paths (type-checked, IDE autocomplete)
|
||||||
|
const pngPath = fixtures.image.base.png200;
|
||||||
|
const heroMp4 = fixtures.video.hero.mp4;
|
||||||
|
|
||||||
|
// Read file bytes
|
||||||
|
const buffer = readFixture(fixtures.image.base.png200);
|
||||||
|
|
||||||
|
// Format accessor (for matrix iteration)
|
||||||
|
const arwPath = fixtures.image.formats("arw");
|
||||||
|
const mp4Path = fixtures.video.tiny("mp4");
|
||||||
|
|
||||||
|
// Directory paths (for glob/scan patterns)
|
||||||
|
const hostileDir = fixtureDir.image.hostile;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Guard Tests
|
||||||
|
|
||||||
|
Four guard tests protect fixture integrity:
|
||||||
|
|
||||||
|
| Guard | Location | What it checks |
|
||||||
|
|-------|----------|----------------|
|
||||||
|
| fixture-resolve | `unit/fixtures/` | Every registry path exists and is non-empty |
|
||||||
|
| fixture-budget | `unit/fixtures/` | Per-extension size caps prevent bloat |
|
||||||
|
| fixture-manifest | `unit/fixtures/` | sha256/bytes in manifest.json match disk |
|
||||||
|
| fixture-integrity | `integration/fixtures/` | Real heroes decode through Sharp/ffprobe/qpdf |
|
||||||
|
|
||||||
|
Additional parity guards:
|
||||||
|
|
||||||
|
| Guard | Location | What it checks |
|
||||||
|
|-------|----------|----------------|
|
||||||
|
| tool-registry-drift | `unit/shared/` | Frontend registry matches shared TOOLS catalog |
|
||||||
|
| tool-route-drift | `integration/` | API routes match shared TOOLS catalog |
|
||||||
|
| i18n-completeness | `unit/shared/` | All 21 locales have all required keys |
|
||||||
|
|
||||||
|
### Adding a Fixture
|
||||||
|
|
||||||
|
1. Add the file to the appropriate modality directory (`image/valid/`, `video/formats/`, etc.)
|
||||||
|
2. Add a named key in `tests/fixtures/index.ts`
|
||||||
|
3. Run `node tests/fixtures/gen-manifest.mjs` (for `valid/` files)
|
||||||
|
4. Fill in `sourceUrl` and `license` in `manifest.json`
|
||||||
|
5. Update `LICENSES.md` if the file has third-party origin
|
||||||
|
6. Run guards: `pnpm vitest run tests/unit/fixtures/`
|
||||||
|
|
||||||
|
## Generator Scripts
|
||||||
|
|
||||||
|
Three generators produce deterministic synthetics. All are idempotent and skip
|
||||||
|
files that already exist (to avoid breaking manifest hashes from encoder-version
|
||||||
|
differences). Pass `--force` to `gen-synthetic-content.mjs` to overwrite.
|
||||||
|
|
||||||
|
| Script | Scope | Outputs |
|
||||||
|
|--------|-------|---------|
|
||||||
|
| `scripts/generate-test-fixtures.mjs` | Tier-1 format samples | `video/formats/tiny.mp4`, `audio/formats/tiny.{mp3,wav}`, `document/formats/tiny.{docx,xlsx,epub,html,md}`, `document/valid/encrypted.pdf` |
|
||||||
|
| `scripts/generate-hostile-fixtures.mjs` | Hostile inputs | `image/hostile/{truncated.jpg,zero-byte.png,garbage.jpg,png-bytes.jpg,bomb-50000x50000.png}` |
|
||||||
|
| `tests/fixtures/gen-synthetic-content.mjs` | CC0 synthetics | QR codes, barcodes, OCR text images, PDFs, SVG logo, multi-face placeholder, animated GIF, tagged MP3, metadata MP4 |
|
||||||
|
|
||||||
|
The generators do NOT touch committed real heroes (BBB videos, TTS speech, UNVERIFIED-REVIEW photos).
|
||||||
|
|
||||||
|
## Running Tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Full suite
|
||||||
|
pnpm test # All unit + integration tests
|
||||||
|
pnpm test:unit # Unit tests only
|
||||||
|
pnpm test:integration # Integration tests only
|
||||||
|
|
||||||
|
# Single file
|
||||||
|
pnpm vitest run tests/unit/my-test.test.ts
|
||||||
|
pnpm vitest run tests/integration/my-test.test.ts
|
||||||
|
|
||||||
|
# Fixture guards only
|
||||||
|
pnpm vitest run tests/unit/fixtures
|
||||||
|
|
||||||
|
# E2E (Playwright)
|
||||||
|
pnpm test:e2e # Main app (chromium + serial + visual projects)
|
||||||
|
pnpm test:e2e:landing # Landing site
|
||||||
|
pnpm test:e2e:docs # VitePress docs
|
||||||
|
pnpm test:e2e:analytics # Analytics flows
|
||||||
|
pnpm test:docker # Full Docker container test
|
||||||
|
|
||||||
|
# Editor e2e (no root alias)
|
||||||
|
pnpm playwright test --config playwright.editor.config.ts
|
||||||
|
|
||||||
|
# Single e2e spec
|
||||||
|
pnpm playwright test tests/e2e/my-test.spec.ts
|
||||||
|
```
|
||||||
|
|
||||||
|
### Environment Requirements
|
||||||
|
|
||||||
|
Unit and integration tests need Postgres + Redis:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f docker-compose.dev.yml up -d
|
||||||
|
export DATA_DIR=/tmp/so-test-data && mkdir -p "$DATA_DIR"
|
||||||
|
```
|
||||||
|
|
||||||
|
Each Vitest fork gets its own Postgres DB clone + workspace + isolated `BULLMQ_PREFIX`
|
||||||
|
via `tests/setup/per-fork-env.ts` (testcontainers for Postgres). 30s test timeouts.
|
||||||
|
|
||||||
|
### E2E Projects
|
||||||
|
|
||||||
|
The main Playwright config defines three projects:
|
||||||
|
|
||||||
|
- **chromium**: Parallel-safe specs, 2 workers (override via `PW_WORKERS`)
|
||||||
|
- **chromium-serial**: Global-state specs, 1 worker
|
||||||
|
- **chromium-visual**: Screenshot comparison specs (platform-suffixed baselines)
|
||||||
|
|
||||||
|
Auth setup project logs in and saves storage state. The `webServer` block spins up
|
||||||
|
both dev servers with a fresh test DB.
|
||||||
|
|
||||||
|
### Environment Switches
|
||||||
|
|
||||||
|
| Variable | Effect |
|
||||||
|
|----------|--------|
|
||||||
|
| `FULL_MATRIX=1` | Run all format x tool combinations (nightly) |
|
||||||
|
| `FUZZ=1` + `FUZZ_RUNS=N` | Property-based fuzz testing (nightly) |
|
||||||
|
| `PW_WORKERS=N` | Override Playwright worker count |
|
||||||
|
|
||||||
|
### CI Pipeline
|
||||||
|
|
||||||
|
PRs run: lint, typecheck, unit, integration (4 shards), e2e smoke.
|
||||||
|
Nightly: full e2e, cross-browser, docker suite, full matrix + fuzz, coverage.
|
||||||
File diff suppressed because it is too large
Load Diff
Vendored
+39
-24
@@ -1,37 +1,47 @@
|
|||||||
# Test Fixtures
|
# Test Fixtures
|
||||||
|
|
||||||
Shared test fixtures for the SnapOtter test suite. Organized in two tiers:
|
Shared test fixtures for the SnapOtter test suite. Organized by modality in two tiers.
|
||||||
|
|
||||||
## Two-Tier Structure
|
## Two-Tier Structure
|
||||||
|
|
||||||
**Tier 1: Tiny Synthetics (matrices/format coverage)**
|
**Tier 1: Tiny Synthetics (matrices/format coverage)**
|
||||||
Small, project-generated files for format-compatibility matrices and fast unit tests.
|
Small, project-generated files for format-compatibility matrices and fast unit tests.
|
||||||
Located in `media/`, `documents/`, `data/`, `formats/`, and root-level `test-*` files.
|
Located in `*/formats/`, `*/edge/`, `*/hostile/`, and `security/`.
|
||||||
|
|
||||||
**Tier 2: Real Heroes (depth/fidelity testing)**
|
**Tier 2: Real Heroes (depth/fidelity testing)**
|
||||||
Larger, content-representative files for integration and fidelity tests.
|
Larger, content-representative files for integration and fidelity tests.
|
||||||
Located in `content/`. Each must be tracked in `manifest.json` with sha256 + provenance.
|
Located in `*/valid/`. Each must be tracked in `manifest.json` with sha256 + provenance.
|
||||||
|
|
||||||
## Layout (Modality-First, Phase 6 Target)
|
## Modality-First Layout
|
||||||
|
|
||||||
```
|
```
|
||||||
tests/fixtures/
|
tests/fixtures/
|
||||||
index.ts # Typed registry (import paths from here, not raw strings)
|
index.ts # Typed registry (import paths from here, not raw strings)
|
||||||
manifest.json # sha256 + bytes + provenance for content/ heroes
|
manifest.json # sha256 + bytes + provenance for all valid/ assets
|
||||||
gen-manifest.mjs # Script to re-stamp manifest hashes
|
gen-manifest.mjs # Script to re-stamp manifest hashes
|
||||||
|
gen-synthetic-content.mjs # Regenerate CC0 synthetics (QR, barcode, OCR, etc.)
|
||||||
LICENSES.md # Provenance audit trail
|
LICENSES.md # Provenance audit trail
|
||||||
content/ # Tier-2 real heroes
|
image/
|
||||||
formats/ # Tier-1 format samples (35 formats incl. RAW)
|
valid/ # Base test images, portraits, OCR, barcodes, QR codes
|
||||||
media/ # Tier-1 tiny audio/video samples
|
formats/ # 35 format samples (sample.png, sample.arw, sample.psd, ...)
|
||||||
documents/ # Tier-1 tiny document samples
|
edge/ # Edge cases (1x1, blank, extreme aspect, fake transparency)
|
||||||
data/ # Tier-1 tiny data samples (csv, json, xml, etc.)
|
hostile/ # Truncated, zero-byte, garbage, bomb, extension-mismatch
|
||||||
hostile/ # Malformed/bomb/polyglot files for rejection tests
|
video/
|
||||||
security/ # XXE, SSRF test vectors
|
valid/ # BBB heroes (mov/webm/mkv/avi), TTS speech, metadata MP4
|
||||||
image/ # (Phase 6) modality-first target dir
|
formats/ # tiny.{mp4,webm,avi,...}, subtitle files (srt/vtt/ass)
|
||||||
video/ # (Phase 6) modality-first target dir
|
hostile/ # Truncated MP4
|
||||||
audio/ # (Phase 6) modality-first target dir
|
audio/
|
||||||
document/ # (Phase 6) modality-first target dir
|
valid/ # TTS speech (wav/flac/ogg/m4a/aac/opus), tagged MP3
|
||||||
data/ # (Phase 6) modality-first target dir
|
formats/ # tiny.{mp3,wav,flac,...}, tone-stereo, tone-gap
|
||||||
|
hostile/ # Zero-byte WAV
|
||||||
|
document/
|
||||||
|
valid/ # PDFs (3-page, 6-page, encrypted, OCR-scanned)
|
||||||
|
formats/ # tiny.{docx,xlsx,epub,html,md}
|
||||||
|
edge/ # remote-img.html (SSRF test vector)
|
||||||
|
hostile/ # Truncated DOCX, garbage PDF
|
||||||
|
data/
|
||||||
|
valid/ # tiny.{csv,json,xml,yaml,tsv,zip}
|
||||||
|
security/ # SVG XXE test vectors (file-read, SSRF)
|
||||||
```
|
```
|
||||||
|
|
||||||
## Using the Registry
|
## Using the Registry
|
||||||
@@ -54,18 +64,18 @@ const mp4Path = fixtures.video.tiny("mp4");
|
|||||||
|
|
||||||
## Adding a Fixture
|
## Adding a Fixture
|
||||||
|
|
||||||
1. Add the file to the appropriate directory
|
1. Add the file to the appropriate modality directory
|
||||||
2. Add a key in `tests/fixtures/index.ts` pointing to it
|
2. Add a key in `tests/fixtures/index.ts` pointing to it
|
||||||
3. Run `node tests/fixtures/gen-manifest.mjs` if the file is in `content/`
|
3. Run `node tests/fixtures/gen-manifest.mjs` (for `valid/` files)
|
||||||
4. Fill in `sourceUrl` and `license` in `manifest.json` (required for Phase 2+)
|
4. Fill in `sourceUrl` and `license` in `manifest.json` (required)
|
||||||
5. Update `LICENSES.md` with the provenance record
|
5. Update `LICENSES.md` with the provenance record
|
||||||
6. Run the guards: `pnpm vitest run tests/unit/fixtures/ tests/integration/fixtures/`
|
6. Run the guards: `pnpm vitest run tests/unit/fixtures/`
|
||||||
|
|
||||||
## Licensing Gate
|
## Licensing Gate
|
||||||
|
|
||||||
Phase 1 allows `UNVERIFIED` provenance. Phase 2 enforces that every `content/` hero
|
Every `valid/` hero must have a verified license (CC0, CC-BY, CC-BY-SA, or
|
||||||
has a verified license (CC0, CC-BY, CC-BY-SA, or public-domain). See `LICENSES.md`
|
public-domain). Assets with `UNVERIFIED-REVIEW` need license review before
|
||||||
for the current audit status.
|
release. The manifest guard test reports the current count.
|
||||||
|
|
||||||
## Guard Tests
|
## Guard Tests
|
||||||
|
|
||||||
@@ -75,3 +85,8 @@ Four guard tests protect fixture integrity:
|
|||||||
- **fixture-budget** (unit): per-extension size caps prevent bloat
|
- **fixture-budget** (unit): per-extension size caps prevent bloat
|
||||||
- **fixture-manifest** (unit): sha256/bytes in manifest.json match disk
|
- **fixture-manifest** (unit): sha256/bytes in manifest.json match disk
|
||||||
- **fixture-integrity** (integration): real heroes decode through Sharp/ffprobe/qpdf
|
- **fixture-integrity** (integration): real heroes decode through Sharp/ffprobe/qpdf
|
||||||
|
|
||||||
|
## Generator Scripts
|
||||||
|
|
||||||
|
Three generators produce deterministic synthetics. All are idempotent and skip
|
||||||
|
existing files to avoid breaking manifest hashes. See `tests/README.md` for details.
|
||||||
|
|||||||
Reference in New Issue
Block a user