test(buzz-media): add audio metadata-strip verification harness

The inline-audio feature splits one invariant across two lanes: the
desktop client strips audio metadata before upload, and the relay
validates the stripped shape. Both sides have to agree on exact bytes,
so both sides should test against the same evidence rather than against
separately-invented synthetic packets.

Adds a generator for the fixtures that actually discriminate, a
four-instrument checker for a candidate stripper, and an independent
oracle for the canonical Ogg page layout.

Fixtures are generated by ffmpeg rather than committed: they are
deterministic, and bigart.ogg is 127 KB of synthetic album art that
does not belong in git history.

Two fixtures encode findings that are easy to miss:

- mpeg2_22k.mp3 / mpeg25_11k.mp3 have frame syncs FF F3 and FF E3.
  infer 0.19.0's is_mp3 matches only ID3 magic or the MPEG-1 sync
  FF FB, so these files are recognised solely by their ID3 header.
  Strip it and infer::get() returns None, which silently downgrades a
  valid mp3 to a generic attachment with no player and no error. Any
  acceptance path keying on infer for mp3 has this bug.

- bigart.ogg's Vorbis comment packet exhausts one page's 255 lacing
  values and continues onto the next, so the comment cannot be excised
  in place; the header region must be demuxed and re-paginated with
  sequence numbers renumbered and page CRCs recomputed.

check_strip.sh runs four instruments because the obvious two are not
sufficient: an Ogg rewrite that dropped continuation pages left 58 KB
of payload in place while still decoding cleanly and still reporting
the correct duration. Only the PCM comparison catches a lossy strip,
and only the marker grep catches a strip that did nothing -- so the
marker check reports itself blind when its negative control is absent
rather than passing vacuously.

verify_contract.py is written from RFC 3533 rather than from any
particular stripper, so it does not grade its own homework.

Harness self-tested against known-broken inputs before use: a no-op
copy fails on the marker check, and the dropped-continuation-page Ogg
bug fails on decode, duration, and PCM. Known-good strippers pass all
fixtures.

Signed-off-by: tlongwell-block <109685178+tlongwell-block@users.noreply.github.com>
Co-authored-by: Dawn <c6237ef84fa537c78dcee78efd2d4e59f728859c7f194da42ac51ededfa0be05@buzz.block.builderlab.xyz>
This commit is contained in:
tlongwell-block
2026-08-14 11:42:56 -04:00
co-authored by Dawn
parent 5e4d0fe925
commit 1f87c19d84
5 changed files with 178 additions and 0 deletions
@@ -0,0 +1,3 @@
# Generated by make_fixtures.sh — regenerate locally, never commit.
# bigart.ogg alone is 127 KB of synthetic album art.
fixtures/
+77
View File
@@ -0,0 +1,77 @@
# Audio metadata-strip verification harness
Shared evidence for the inline-audio work: the desktop client strips audio
metadata before upload, and the relay validates the stripped shape. Both
sides must agree on exact bytes, so both sides test against these fixtures.
Unlike the `ios/` and `android/` fixtures, the files here are **generated,
not committed** — they come from `ffmpeg` deterministically, and one of them
is 127 KB of synthetic album art that has no business in git history.
## Generating
```sh
./make_fixtures.sh [output-dir] # defaults to ./fixtures
```
Requires `ffmpeg`/`ffprobe` on PATH. Every fixture embeds a known location
marker (`GPS 37.7749N 122.4194W`), so a stripper that does nothing cannot
pass — see the negative control in `check_strip.sh`.
## The fixtures that carry weight
| Fixture | Why it exists |
|---|---|
| `tagged.{mp3,ogg,wav}` | Baseline: title + location comment in each container. |
| `mpeg2_22k.mp3` | 22.05 kHz MPEG-2 mp3. Frame sync is `FF F3`. |
| `mpeg25_11k.mp3` | 11.025 kHz MPEG-2.5 mp3. Frame sync is `FF E3`. |
| `bigart.ogg` | 120 KB comment value, so the Vorbis comment packet **spans pages**. |
| `long.ogg` | 30 s, multiple audio pages after setup. |
`infer::is_mp3` (infer 0.19.0, `src/matchers/audio.rs`) matches only ID3
magic or the MPEG-1 sync `FF FB`. The two MPEG-2/2.5 fixtures are therefore
recognised *solely* by their ID3 header: strip it and `infer::get()` returns
`None`. Any acceptance path that consults `infer` for mp3 will silently
downgrade these files to generic attachments, so both the client sniff and
the relay validator must key on MPEG frame structure instead.
`bigart.ogg` is the Ogg correctness case. Its comment packet exhausts one
page's 255 lacing values and continues onto the next (continuation bit set in
the following page header), so the comment cannot be excised in place — the
header region must be demuxed into packets and re-paginated, with
`page_sequence_number` renumbered and every page CRC recomputed
(RFC 3533 §6, polynomial `0x04c11db7`).
## Checking a stripper
```sh
./check_strip.sh <original> <stripped> [marker-regex]
```
Four independent instruments, all of which must pass:
1. **Marker grep**, with a negative control on the original — did it strip?
(If the marker is absent from the original the check reports itself blind
rather than passing.)
2. **`ffmpeg -f null -`** — does it still decode?
3. **`ffprobe` duration** — did it truncate?
4. **PCM SHA compare** — decode both files to raw `s16le` and compare. This
is the only instrument that proves the strip is *lossless*.
Instruments 2 and 3 both pass on a file that still contains the metadata you
believe you removed; that failure was observed during development, on an Ogg
rewrite that dropped continuation pages while leaving 58 KB of payload behind
and still reporting the correct duration. Do not rely on decodability alone.
## Checking Ogg page structure
```sh
python3 verify_contract.py <file.ogg>...
```
Asserts the canonical layout the client emits: one logical stream, page
sequence contiguous from 0, every page CRC recomputed and re-verified, the
identification packet alone on the BOS page, the canonical empty comment
packet alone on page 1 with a single lacing value, and the setup packet
isolated on its own page(s). Written from RFC 3533 rather than from any
particular stripper implementation, so it is an independent oracle.
+32
View File
@@ -0,0 +1,32 @@
#!/usr/bin/env bash
# Four independent instruments. A strip is correct only if ALL four pass.
# Usage: check_strip.sh <original> <stripped> [marker]
set -uo pipefail
ORIG="$1"; STRIP="$2"; MARK="${3:-GPS\|TITLE\|Lavf\|XXXXXXXX}"
fail=0
# (1) metadata actually gone. Run against the ORIGINAL first as a negative
# control -- if the marker isn't in the original, this instrument is blind.
oh=$(strings -a "$ORIG" | grep -ci "$MARK" || true)
sh=$(strings -a "$STRIP" | grep -ci "$MARK" || true)
if [ "$oh" -eq 0 ]; then echo " [!] BLIND: marker absent from original, test proves nothing"; fail=1
elif [ "$sh" -ne 0 ]; then echo " [x] metadata STILL PRESENT ($sh hits)"; fail=1
else echo " [ok] metadata gone (original had $oh hits)"; fi
# (2) still decodes
if ffmpeg -loglevel error -i "$STRIP" -f null - 2>/dev/null; then echo " [ok] decodes"
else echo " [x] DECODE FAILED"; fail=1; fi
# (3) duration preserved
d1=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$ORIG")
d2=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$STRIP")
if [ "$d1" = "$d2" ]; then echo " [ok] duration $d2"; else echo " [x] duration $d1 -> $d2"; fail=1; fi
# (4) THE ONE THAT MATTERS: audio bit-exact. Decode both to raw PCM, compare.
# (2) and (3) both pass on a file that still contains the metadata.
a=$(ffmpeg -loglevel error -i "$ORIG" -f s16le - 2>/dev/null | shasum | cut -d' ' -f1)
b=$(ffmpeg -loglevel error -i "$STRIP" -f s16le - 2>/dev/null | shasum | cut -d' ' -f1)
if [ "$a" = "$b" ]; then echo " [ok] PCM bit-identical"; else echo " [x] PCM DIFFERS -- strip is lossy"; fail=1; fi
echo " => $([ $fail -eq 0 ] && echo PASS || echo FAIL)"
exit $fail
+30
View File
@@ -0,0 +1,30 @@
#!/usr/bin/env bash
# Fixture generator for the audio metadata-strip work (block/buzz audio inline).
# Every fixture carries a known location marker so a no-op "stripper" cannot pass.
set -euo pipefail
OUT="${1:-fixtures}"
mkdir -p "$OUT"; cd "$OUT"
MARK="GPS 37.7749N 122.4194W"
SINE="sine=frequency=440:duration=2"
# 1. baseline tagged files, one per container
ffmpeg -loglevel error -f lavfi -i "$SINE" -metadata title="TITLE" -metadata comment="$MARK" -y tagged.mp3
ffmpeg -loglevel error -f lavfi -i "$SINE" -metadata title="TITLE" -metadata comment="$MARK" -y tagged.ogg
ffmpeg -loglevel error -f lavfi -i "$SINE" -metadata title="TITLE" -metadata comment="$MARK" -y tagged.wav
# 2. THE ONE THAT BREAKS infer: MPEG-2 (22.05kHz) -> 0xFF 0xF3 sync, not 0xFF 0xFB.
# Strip its ID3 and infer::get() returns None.
ffmpeg -loglevel error -f lavfi -i "$SINE" -ar 22050 -b:a 32k -metadata title="T" -y mpeg2_22k.mp3
# MPEG-2.5 (11.025kHz) -> 0xFF 0xE3
ffmpeg -loglevel error -f lavfi -i "$SINE" -ar 11025 -b:a 32k -metadata title="T" -y mpeg25_11k.mp3
# 3. album art large enough that the Ogg comment packet SPANS pages
# (page1 hits 255 lacing values, page2 sets the continuation bit).
BIG=$(python3 -c 'print("X"*120000)')
ffmpeg -loglevel error -f lavfi -i "$SINE" -metadata comment="$BIG" -y bigart.ogg
# 4. longer file: multiple audio pages after setup
ffmpeg -loglevel error -f lavfi -i "sine=frequency=440:duration=30" -y long.ogg
echo "fixtures written to $(pwd)"
ls -la
@@ -0,0 +1,36 @@
import struct,sys
def crc32_ogg(data):
crc=0
for b in data:
crc^=b<<24
for _ in range(8):
crc=((crc<<1)^0x04c11db7)&0xFFFFFFFF if crc&0x80000000 else (crc<<1)&0xFFFFFFFF
return crc
def check(path):
d=open(path,'rb').read(); off=0; i=0; errs=[]; serials=set()
while off<len(d) and d[off:off+4]==b'OggS':
nseg=d[off+26];seg=d[off+27:off+27+nseg];hdr=27+nseg;body=hdr+sum(seg)
page=d[off:off+body];pay=d[off+hdr:off+body]
seq=struct.unpack('<I',page[18:22])[0]
stored=struct.unpack('<I',page[22:26])[0]
calc=crc32_ogg(page[:22]+b'\x00'*4+page[26:])
serials.add(page[14:18])
if stored!=calc: errs.append(f"page{i}: CRC mismatch")
if seq!=i: errs.append(f"page{i}: seq={seq} not contiguous")
if i==0:
if not (page[5]&0x02): errs.append("page0: BOS flag not set")
if pay[:7]!=b'\x01vorbis': errs.append("page0: not identification")
if len(seg)!=1: errs.append(f"page0: {len(seg)} packets, want ident alone")
if i==1:
if pay[:7]!=b'\x03vorbis': errs.append("page1: not comment")
if len(seg)!=1: errs.append(f"page1: {len(seg)} lacing values, contract wants ONE")
if pay!=b'\x03vorbis'+struct.pack('<I',0)+struct.pack('<I',0)+b'\x01':
errs.append("page1: comment not canonical-empty")
if i==2:
if pay[:7]!=b'\x05vorbis': errs.append("page2: setup not isolated at page2")
off+=body;i+=1
if len(serials)!=1: errs.append(f"{len(serials)} logical streams, want 1")
print(f"{path}: pages={i} serials={len(serials)}")
for e in errs: print(" VIOLATION:",e)
if not errs: print(" conforms to locked contract")
for p in sys.argv[1:]: check(p)