test: add test suite README and final verification baseline (phase 8)

- Write tests/README.md covering directory structure, fixture registry,
  two-tier strategy, guard tests, generators, and run commands
- Update tests/fixtures/README.md to reflect modality-first layout
- Add final.json parity baseline: 13,382 passing, 0 dropped from Phase 0
- Write verification report to docs/superpowers/plans/ (local-only)
This commit is contained in:
SnapOtter
2026-06-20 06:10:45 +08:00
parent f100ba1b71
commit 6b2023b2b4
3 changed files with 27247 additions and 24 deletions
+39 -24
View File
@@ -1,37 +1,47 @@
# Test Fixtures
Shared test fixtures for the SnapOtter test suite. Organized in two tiers:
Shared test fixtures for the SnapOtter test suite. Organized by modality in two tiers.
## Two-Tier Structure
**Tier 1: Tiny Synthetics (matrices/format coverage)**
Small, project-generated files for format-compatibility matrices and fast unit tests.
Located in `media/`, `documents/`, `data/`, `formats/`, and root-level `test-*` files.
Located in `*/formats/`, `*/edge/`, `*/hostile/`, and `security/`.
**Tier 2: Real Heroes (depth/fidelity testing)**
Larger, content-representative files for integration and fidelity tests.
Located in `content/`. Each must be tracked in `manifest.json` with sha256 + provenance.
Located in `*/valid/`. Each must be tracked in `manifest.json` with sha256 + provenance.
## Layout (Modality-First, Phase 6 Target)
## Modality-First Layout
```
tests/fixtures/
index.ts # Typed registry (import paths from here, not raw strings)
manifest.json # sha256 + bytes + provenance for content/ heroes
manifest.json # sha256 + bytes + provenance for all valid/ assets
gen-manifest.mjs # Script to re-stamp manifest hashes
gen-synthetic-content.mjs # Regenerate CC0 synthetics (QR, barcode, OCR, etc.)
LICENSES.md # Provenance audit trail
content/ # Tier-2 real heroes
formats/ # Tier-1 format samples (35 formats incl. RAW)
media/ # Tier-1 tiny audio/video samples
documents/ # Tier-1 tiny document samples
data/ # Tier-1 tiny data samples (csv, json, xml, etc.)
hostile/ # Malformed/bomb/polyglot files for rejection tests
security/ # XXE, SSRF test vectors
image/ # (Phase 6) modality-first target dir
video/ # (Phase 6) modality-first target dir
audio/ # (Phase 6) modality-first target dir
document/ # (Phase 6) modality-first target dir
data/ # (Phase 6) modality-first target dir
image/
valid/ # Base test images, portraits, OCR, barcodes, QR codes
formats/ # 35 format samples (sample.png, sample.arw, sample.psd, ...)
edge/ # Edge cases (1x1, blank, extreme aspect, fake transparency)
hostile/ # Truncated, zero-byte, garbage, bomb, extension-mismatch
video/
valid/ # BBB heroes (mov/webm/mkv/avi), TTS speech, metadata MP4
formats/ # tiny.{mp4,webm,avi,...}, subtitle files (srt/vtt/ass)
hostile/ # Truncated MP4
audio/
valid/ # TTS speech (wav/flac/ogg/m4a/aac/opus), tagged MP3
formats/ # tiny.{mp3,wav,flac,...}, tone-stereo, tone-gap
hostile/ # Zero-byte WAV
document/
valid/ # PDFs (3-page, 6-page, encrypted, OCR-scanned)
formats/ # tiny.{docx,xlsx,epub,html,md}
edge/ # remote-img.html (SSRF test vector)
hostile/ # Truncated DOCX, garbage PDF
data/
valid/ # tiny.{csv,json,xml,yaml,tsv,zip}
security/ # SVG XXE test vectors (file-read, SSRF)
```
## Using the Registry
@@ -54,18 +64,18 @@ const mp4Path = fixtures.video.tiny("mp4");
## Adding a Fixture
1. Add the file to the appropriate directory
1. Add the file to the appropriate modality directory
2. Add a key in `tests/fixtures/index.ts` pointing to it
3. Run `node tests/fixtures/gen-manifest.mjs` if the file is in `content/`
4. Fill in `sourceUrl` and `license` in `manifest.json` (required for Phase 2+)
3. Run `node tests/fixtures/gen-manifest.mjs` (for `valid/` files)
4. Fill in `sourceUrl` and `license` in `manifest.json` (required)
5. Update `LICENSES.md` with the provenance record
6. Run the guards: `pnpm vitest run tests/unit/fixtures/ tests/integration/fixtures/`
6. Run the guards: `pnpm vitest run tests/unit/fixtures/`
## Licensing Gate
Phase 1 allows `UNVERIFIED` provenance. Phase 2 enforces that every `content/` hero
has a verified license (CC0, CC-BY, CC-BY-SA, or public-domain). See `LICENSES.md`
for the current audit status.
Every `valid/` hero must have a verified license (CC0, CC-BY, CC-BY-SA, or
public-domain). Assets with `UNVERIFIED-REVIEW` need license review before
release. The manifest guard test reports the current count.
## Guard Tests
@@ -75,3 +85,8 @@ Four guard tests protect fixture integrity:
- **fixture-budget** (unit): per-extension size caps prevent bloat
- **fixture-manifest** (unit): sha256/bytes in manifest.json match disk
- **fixture-integrity** (integration): real heroes decode through Sharp/ffprobe/qpdf
## Generator Scripts
Three generators produce deterministic synthetics. All are idempotent and skip
existing files to avoid breaking manifest hashes. See `tests/README.md` for details.