mirror of
https://github.com/block/buzz.git
synced 2026-08-18 06:50:31 +02:00
test(buzz-media): add audio metadata-strip verification harness
The inline-audio feature splits one invariant across two lanes: the desktop client strips audio metadata before upload, and the relay validates the stripped shape. Both sides have to agree on exact bytes, so both sides should test against the same evidence rather than against separately-invented synthetic packets. Adds a generator for the fixtures that actually discriminate, a four-instrument checker for a candidate stripper, and an independent oracle for the canonical Ogg page layout. Fixtures are generated by ffmpeg rather than committed: they are deterministic, and bigart.ogg is 127 KB of synthetic album art that does not belong in git history. Two fixtures encode findings that are easy to miss: - mpeg2_22k.mp3 / mpeg25_11k.mp3 have frame syncs FF F3 and FF E3. infer 0.19.0's is_mp3 matches only ID3 magic or the MPEG-1 sync FF FB, so these files are recognised solely by their ID3 header. Strip it and infer::get() returns None, which silently downgrades a valid mp3 to a generic attachment with no player and no error. Any acceptance path keying on infer for mp3 has this bug. - bigart.ogg's Vorbis comment packet exhausts one page's 255 lacing values and continues onto the next, so the comment cannot be excised in place; the header region must be demuxed and re-paginated with sequence numbers renumbered and page CRCs recomputed. check_strip.sh runs four instruments because the obvious two are not sufficient: an Ogg rewrite that dropped continuation pages left 58 KB of payload in place while still decoding cleanly and still reporting the correct duration. Only the PCM comparison catches a lossy strip, and only the marker grep catches a strip that did nothing -- so the marker check reports itself blind when its negative control is absent rather than passing vacuously. verify_contract.py is written from RFC 3533 rather than from any particular stripper, so it does not grade its own homework. Harness self-tested against known-broken inputs before use: a no-op copy fails on the marker check, and the dropped-continuation-page Ogg bug fails on decode, duration, and PCM. Known-good strippers pass all fixtures. Signed-off-by: tlongwell-block <109685178+tlongwell-block@users.noreply.github.com> Co-authored-by: Dawn <c6237ef84fa537c78dcee78efd2d4e59f728859c7f194da42ac51ededfa0be05@buzz.block.builderlab.xyz>
This commit is contained in:
@@ -0,0 +1,3 @@
|
||||
# Generated by make_fixtures.sh — regenerate locally, never commit.
|
||||
# bigart.ogg alone is 127 KB of synthetic album art.
|
||||
fixtures/
|
||||
@@ -0,0 +1,77 @@
|
||||
# Audio metadata-strip verification harness
|
||||
|
||||
Shared evidence for the inline-audio work: the desktop client strips audio
|
||||
metadata before upload, and the relay validates the stripped shape. Both
|
||||
sides must agree on exact bytes, so both sides test against these fixtures.
|
||||
|
||||
Unlike the `ios/` and `android/` fixtures, the files here are **generated,
|
||||
not committed** — they come from `ffmpeg` deterministically, and one of them
|
||||
is 127 KB of synthetic album art that has no business in git history.
|
||||
|
||||
## Generating
|
||||
|
||||
```sh
|
||||
./make_fixtures.sh [output-dir] # defaults to ./fixtures
|
||||
```
|
||||
|
||||
Requires `ffmpeg`/`ffprobe` on PATH. Every fixture embeds a known location
|
||||
marker (`GPS 37.7749N 122.4194W`), so a stripper that does nothing cannot
|
||||
pass — see the negative control in `check_strip.sh`.
|
||||
|
||||
## The fixtures that carry weight
|
||||
|
||||
| Fixture | Why it exists |
|
||||
|---|---|
|
||||
| `tagged.{mp3,ogg,wav}` | Baseline: title + location comment in each container. |
|
||||
| `mpeg2_22k.mp3` | 22.05 kHz MPEG-2 mp3. Frame sync is `FF F3`. |
|
||||
| `mpeg25_11k.mp3` | 11.025 kHz MPEG-2.5 mp3. Frame sync is `FF E3`. |
|
||||
| `bigart.ogg` | 120 KB comment value, so the Vorbis comment packet **spans pages**. |
|
||||
| `long.ogg` | 30 s, multiple audio pages after setup. |
|
||||
|
||||
`infer::is_mp3` (infer 0.19.0, `src/matchers/audio.rs`) matches only ID3
|
||||
magic or the MPEG-1 sync `FF FB`. The two MPEG-2/2.5 fixtures are therefore
|
||||
recognised *solely* by their ID3 header: strip it and `infer::get()` returns
|
||||
`None`. Any acceptance path that consults `infer` for mp3 will silently
|
||||
downgrade these files to generic attachments, so both the client sniff and
|
||||
the relay validator must key on MPEG frame structure instead.
|
||||
|
||||
`bigart.ogg` is the Ogg correctness case. Its comment packet exhausts one
|
||||
page's 255 lacing values and continues onto the next (continuation bit set in
|
||||
the following page header), so the comment cannot be excised in place — the
|
||||
header region must be demuxed into packets and re-paginated, with
|
||||
`page_sequence_number` renumbered and every page CRC recomputed
|
||||
(RFC 3533 §6, polynomial `0x04c11db7`).
|
||||
|
||||
## Checking a stripper
|
||||
|
||||
```sh
|
||||
./check_strip.sh <original> <stripped> [marker-regex]
|
||||
```
|
||||
|
||||
Four independent instruments, all of which must pass:
|
||||
|
||||
1. **Marker grep**, with a negative control on the original — did it strip?
|
||||
(If the marker is absent from the original the check reports itself blind
|
||||
rather than passing.)
|
||||
2. **`ffmpeg -f null -`** — does it still decode?
|
||||
3. **`ffprobe` duration** — did it truncate?
|
||||
4. **PCM SHA compare** — decode both files to raw `s16le` and compare. This
|
||||
is the only instrument that proves the strip is *lossless*.
|
||||
|
||||
Instruments 2 and 3 both pass on a file that still contains the metadata you
|
||||
believe you removed; that failure was observed during development, on an Ogg
|
||||
rewrite that dropped continuation pages while leaving 58 KB of payload behind
|
||||
and still reporting the correct duration. Do not rely on decodability alone.
|
||||
|
||||
## Checking Ogg page structure
|
||||
|
||||
```sh
|
||||
python3 verify_contract.py <file.ogg>...
|
||||
```
|
||||
|
||||
Asserts the canonical layout the client emits: one logical stream, page
|
||||
sequence contiguous from 0, every page CRC recomputed and re-verified, the
|
||||
identification packet alone on the BOS page, the canonical empty comment
|
||||
packet alone on page 1 with a single lacing value, and the setup packet
|
||||
isolated on its own page(s). Written from RFC 3533 rather than from any
|
||||
particular stripper implementation, so it is an independent oracle.
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
#!/usr/bin/env bash
|
||||
# Four independent instruments. A strip is correct only if ALL four pass.
|
||||
# Usage: check_strip.sh <original> <stripped> [marker]
|
||||
set -uo pipefail
|
||||
ORIG="$1"; STRIP="$2"; MARK="${3:-GPS\|TITLE\|Lavf\|XXXXXXXX}"
|
||||
fail=0
|
||||
|
||||
# (1) metadata actually gone. Run against the ORIGINAL first as a negative
|
||||
# control -- if the marker isn't in the original, this instrument is blind.
|
||||
oh=$(strings -a "$ORIG" | grep -ci "$MARK" || true)
|
||||
sh=$(strings -a "$STRIP" | grep -ci "$MARK" || true)
|
||||
if [ "$oh" -eq 0 ]; then echo " [!] BLIND: marker absent from original, test proves nothing"; fail=1
|
||||
elif [ "$sh" -ne 0 ]; then echo " [x] metadata STILL PRESENT ($sh hits)"; fail=1
|
||||
else echo " [ok] metadata gone (original had $oh hits)"; fi
|
||||
|
||||
# (2) still decodes
|
||||
if ffmpeg -loglevel error -i "$STRIP" -f null - 2>/dev/null; then echo " [ok] decodes"
|
||||
else echo " [x] DECODE FAILED"; fail=1; fi
|
||||
|
||||
# (3) duration preserved
|
||||
d1=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$ORIG")
|
||||
d2=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$STRIP")
|
||||
if [ "$d1" = "$d2" ]; then echo " [ok] duration $d2"; else echo " [x] duration $d1 -> $d2"; fail=1; fi
|
||||
|
||||
# (4) THE ONE THAT MATTERS: audio bit-exact. Decode both to raw PCM, compare.
|
||||
# (2) and (3) both pass on a file that still contains the metadata.
|
||||
a=$(ffmpeg -loglevel error -i "$ORIG" -f s16le - 2>/dev/null | shasum | cut -d' ' -f1)
|
||||
b=$(ffmpeg -loglevel error -i "$STRIP" -f s16le - 2>/dev/null | shasum | cut -d' ' -f1)
|
||||
if [ "$a" = "$b" ]; then echo " [ok] PCM bit-identical"; else echo " [x] PCM DIFFERS -- strip is lossy"; fail=1; fi
|
||||
|
||||
echo " => $([ $fail -eq 0 ] && echo PASS || echo FAIL)"
|
||||
exit $fail
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
#!/usr/bin/env bash
|
||||
# Fixture generator for the audio metadata-strip work (block/buzz audio inline).
|
||||
# Every fixture carries a known location marker so a no-op "stripper" cannot pass.
|
||||
set -euo pipefail
|
||||
OUT="${1:-fixtures}"
|
||||
mkdir -p "$OUT"; cd "$OUT"
|
||||
MARK="GPS 37.7749N 122.4194W"
|
||||
SINE="sine=frequency=440:duration=2"
|
||||
|
||||
# 1. baseline tagged files, one per container
|
||||
ffmpeg -loglevel error -f lavfi -i "$SINE" -metadata title="TITLE" -metadata comment="$MARK" -y tagged.mp3
|
||||
ffmpeg -loglevel error -f lavfi -i "$SINE" -metadata title="TITLE" -metadata comment="$MARK" -y tagged.ogg
|
||||
ffmpeg -loglevel error -f lavfi -i "$SINE" -metadata title="TITLE" -metadata comment="$MARK" -y tagged.wav
|
||||
|
||||
# 2. THE ONE THAT BREAKS infer: MPEG-2 (22.05kHz) -> 0xFF 0xF3 sync, not 0xFF 0xFB.
|
||||
# Strip its ID3 and infer::get() returns None.
|
||||
ffmpeg -loglevel error -f lavfi -i "$SINE" -ar 22050 -b:a 32k -metadata title="T" -y mpeg2_22k.mp3
|
||||
# MPEG-2.5 (11.025kHz) -> 0xFF 0xE3
|
||||
ffmpeg -loglevel error -f lavfi -i "$SINE" -ar 11025 -b:a 32k -metadata title="T" -y mpeg25_11k.mp3
|
||||
|
||||
# 3. album art large enough that the Ogg comment packet SPANS pages
|
||||
# (page1 hits 255 lacing values, page2 sets the continuation bit).
|
||||
BIG=$(python3 -c 'print("X"*120000)')
|
||||
ffmpeg -loglevel error -f lavfi -i "$SINE" -metadata comment="$BIG" -y bigart.ogg
|
||||
|
||||
# 4. longer file: multiple audio pages after setup
|
||||
ffmpeg -loglevel error -f lavfi -i "sine=frequency=440:duration=30" -y long.ogg
|
||||
|
||||
echo "fixtures written to $(pwd)"
|
||||
ls -la
|
||||
@@ -0,0 +1,36 @@
|
||||
import struct,sys
|
||||
def crc32_ogg(data):
|
||||
crc=0
|
||||
for b in data:
|
||||
crc^=b<<24
|
||||
for _ in range(8):
|
||||
crc=((crc<<1)^0x04c11db7)&0xFFFFFFFF if crc&0x80000000 else (crc<<1)&0xFFFFFFFF
|
||||
return crc
|
||||
def check(path):
|
||||
d=open(path,'rb').read(); off=0; i=0; errs=[]; serials=set()
|
||||
while off<len(d) and d[off:off+4]==b'OggS':
|
||||
nseg=d[off+26];seg=d[off+27:off+27+nseg];hdr=27+nseg;body=hdr+sum(seg)
|
||||
page=d[off:off+body];pay=d[off+hdr:off+body]
|
||||
seq=struct.unpack('<I',page[18:22])[0]
|
||||
stored=struct.unpack('<I',page[22:26])[0]
|
||||
calc=crc32_ogg(page[:22]+b'\x00'*4+page[26:])
|
||||
serials.add(page[14:18])
|
||||
if stored!=calc: errs.append(f"page{i}: CRC mismatch")
|
||||
if seq!=i: errs.append(f"page{i}: seq={seq} not contiguous")
|
||||
if i==0:
|
||||
if not (page[5]&0x02): errs.append("page0: BOS flag not set")
|
||||
if pay[:7]!=b'\x01vorbis': errs.append("page0: not identification")
|
||||
if len(seg)!=1: errs.append(f"page0: {len(seg)} packets, want ident alone")
|
||||
if i==1:
|
||||
if pay[:7]!=b'\x03vorbis': errs.append("page1: not comment")
|
||||
if len(seg)!=1: errs.append(f"page1: {len(seg)} lacing values, contract wants ONE")
|
||||
if pay!=b'\x03vorbis'+struct.pack('<I',0)+struct.pack('<I',0)+b'\x01':
|
||||
errs.append("page1: comment not canonical-empty")
|
||||
if i==2:
|
||||
if pay[:7]!=b'\x05vorbis': errs.append("page2: setup not isolated at page2")
|
||||
off+=body;i+=1
|
||||
if len(serials)!=1: errs.append(f"{len(serials)} logical streams, want 1")
|
||||
print(f"{path}: pages={i} serials={len(serials)}")
|
||||
for e in errs: print(" VIOLATION:",e)
|
||||
if not errs: print(" conforms to locked contract")
|
||||
for p in sys.argv[1:]: check(p)
|
||||
Reference in New Issue
Block a user