mirror of
https://github.com/snapotter-hq/SnapOtter.git
synced 2026-08-03 07:46:42 +02:00
test: add test suite README and final verification baseline (phase 8)
- Write tests/README.md covering directory structure, fixture registry, two-tier strategy, guard tests, generators, and run commands - Update tests/fixtures/README.md to reflect modality-first layout - Add final.json parity baseline: 13,382 passing, 0 dropped from Phase 0 - Write verification report to docs/superpowers/plans/ (local-only)
This commit is contained in:
Vendored
+39
-24
@@ -1,37 +1,47 @@
|
||||
# Test Fixtures
|
||||
|
||||
Shared test fixtures for the SnapOtter test suite. Organized in two tiers:
|
||||
Shared test fixtures for the SnapOtter test suite. Organized by modality in two tiers.
|
||||
|
||||
## Two-Tier Structure
|
||||
|
||||
**Tier 1: Tiny Synthetics (matrices/format coverage)**
|
||||
Small, project-generated files for format-compatibility matrices and fast unit tests.
|
||||
Located in `media/`, `documents/`, `data/`, `formats/`, and root-level `test-*` files.
|
||||
Located in `*/formats/`, `*/edge/`, `*/hostile/`, and `security/`.
|
||||
|
||||
**Tier 2: Real Heroes (depth/fidelity testing)**
|
||||
Larger, content-representative files for integration and fidelity tests.
|
||||
Located in `content/`. Each must be tracked in `manifest.json` with sha256 + provenance.
|
||||
Located in `*/valid/`. Each must be tracked in `manifest.json` with sha256 + provenance.
|
||||
|
||||
## Layout (Modality-First, Phase 6 Target)
|
||||
## Modality-First Layout
|
||||
|
||||
```
|
||||
tests/fixtures/
|
||||
index.ts # Typed registry (import paths from here, not raw strings)
|
||||
manifest.json # sha256 + bytes + provenance for content/ heroes
|
||||
manifest.json # sha256 + bytes + provenance for all valid/ assets
|
||||
gen-manifest.mjs # Script to re-stamp manifest hashes
|
||||
gen-synthetic-content.mjs # Regenerate CC0 synthetics (QR, barcode, OCR, etc.)
|
||||
LICENSES.md # Provenance audit trail
|
||||
content/ # Tier-2 real heroes
|
||||
formats/ # Tier-1 format samples (35 formats incl. RAW)
|
||||
media/ # Tier-1 tiny audio/video samples
|
||||
documents/ # Tier-1 tiny document samples
|
||||
data/ # Tier-1 tiny data samples (csv, json, xml, etc.)
|
||||
hostile/ # Malformed/bomb/polyglot files for rejection tests
|
||||
security/ # XXE, SSRF test vectors
|
||||
image/ # (Phase 6) modality-first target dir
|
||||
video/ # (Phase 6) modality-first target dir
|
||||
audio/ # (Phase 6) modality-first target dir
|
||||
document/ # (Phase 6) modality-first target dir
|
||||
data/ # (Phase 6) modality-first target dir
|
||||
image/
|
||||
valid/ # Base test images, portraits, OCR, barcodes, QR codes
|
||||
formats/ # 35 format samples (sample.png, sample.arw, sample.psd, ...)
|
||||
edge/ # Edge cases (1x1, blank, extreme aspect, fake transparency)
|
||||
hostile/ # Truncated, zero-byte, garbage, bomb, extension-mismatch
|
||||
video/
|
||||
valid/ # BBB heroes (mov/webm/mkv/avi), TTS speech, metadata MP4
|
||||
formats/ # tiny.{mp4,webm,avi,...}, subtitle files (srt/vtt/ass)
|
||||
hostile/ # Truncated MP4
|
||||
audio/
|
||||
valid/ # TTS speech (wav/flac/ogg/m4a/aac/opus), tagged MP3
|
||||
formats/ # tiny.{mp3,wav,flac,...}, tone-stereo, tone-gap
|
||||
hostile/ # Zero-byte WAV
|
||||
document/
|
||||
valid/ # PDFs (3-page, 6-page, encrypted, OCR-scanned)
|
||||
formats/ # tiny.{docx,xlsx,epub,html,md}
|
||||
edge/ # remote-img.html (SSRF test vector)
|
||||
hostile/ # Truncated DOCX, garbage PDF
|
||||
data/
|
||||
valid/ # tiny.{csv,json,xml,yaml,tsv,zip}
|
||||
security/ # SVG XXE test vectors (file-read, SSRF)
|
||||
```
|
||||
|
||||
## Using the Registry
|
||||
@@ -54,18 +64,18 @@ const mp4Path = fixtures.video.tiny("mp4");
|
||||
|
||||
## Adding a Fixture
|
||||
|
||||
1. Add the file to the appropriate directory
|
||||
1. Add the file to the appropriate modality directory
|
||||
2. Add a key in `tests/fixtures/index.ts` pointing to it
|
||||
3. Run `node tests/fixtures/gen-manifest.mjs` if the file is in `content/`
|
||||
4. Fill in `sourceUrl` and `license` in `manifest.json` (required for Phase 2+)
|
||||
3. Run `node tests/fixtures/gen-manifest.mjs` (for `valid/` files)
|
||||
4. Fill in `sourceUrl` and `license` in `manifest.json` (required)
|
||||
5. Update `LICENSES.md` with the provenance record
|
||||
6. Run the guards: `pnpm vitest run tests/unit/fixtures/ tests/integration/fixtures/`
|
||||
6. Run the guards: `pnpm vitest run tests/unit/fixtures/`
|
||||
|
||||
## Licensing Gate
|
||||
|
||||
Phase 1 allows `UNVERIFIED` provenance. Phase 2 enforces that every `content/` hero
|
||||
has a verified license (CC0, CC-BY, CC-BY-SA, or public-domain). See `LICENSES.md`
|
||||
for the current audit status.
|
||||
Every `valid/` hero must have a verified license (CC0, CC-BY, CC-BY-SA, or
|
||||
public-domain). Assets with `UNVERIFIED-REVIEW` need license review before
|
||||
release. The manifest guard test reports the current count.
|
||||
|
||||
## Guard Tests
|
||||
|
||||
@@ -75,3 +85,8 @@ Four guard tests protect fixture integrity:
|
||||
- **fixture-budget** (unit): per-extension size caps prevent bloat
|
||||
- **fixture-manifest** (unit): sha256/bytes in manifest.json match disk
|
||||
- **fixture-integrity** (integration): real heroes decode through Sharp/ffprobe/qpdf
|
||||
|
||||
## Generator Scripts
|
||||
|
||||
Three generators produce deterministic synthetics. All are idempotent and skip
|
||||
existing files to avoid breaking manifest hashes. See `tests/README.md` for details.
|
||||
|
||||
Reference in New Issue
Block a user