The file-cleaners layer covered 15 formats -- all image, document, or
text -- and zero audio/video. That gap gets more expensive every month:
Sora, Veo, ElevenLabs, and Suno all embed provenance through the same
mechanisms image generators do, just in different containers.
New av_meta.py adds inspect/clean for:
- MP4/MOV/M4A/M4V: top-level C2PA (jumb/c2pa box) and XMP (uuid box)
detection/stripping reuse inspect_isobmff()/strip_isobmff() from
image_meta.py unchanged -- that's exactly the mechanism the C2PA spec
defines for ISOBMFF-family containers, already proven for AVIF/HEIC.
moov/udta (where generator/tool tags live) is handled separately since
it's MP4-specific.
- WAV: RIFF LIST INFO chunk + embedded id3 chunk.
- MP3: ID3v2 frames, per-frame for v2.3/v2.4, whole-tag fallback for
v2.2 (3-byte frame IDs are detected but not decomposed, so a partial
rewrite is never attempted there).
Every box/chunk/frame is either kept byte-identical or dropped whole --
nothing does a partial in-place rewrite of a payload, so a container can
never come out semantically mangled. Default strip_all_metadata=True
matches this project's existing default (privacy-first: drop everything,
--keep-non-ai-metadata narrows to only AI-flagged content), same as the
image cleaners.
Wired through the full dispatch stack so the feature isn't a half
integration: format_dispatch.py (new "av" Kind), inspect_file.py /
clean_file.py (--as av), audit_lib.py (so audit_dir.py's CI/SARIF path
and the pre-commit hooks from #135 both cover audio/video too), and
server.py (HTTP /inspect and /clean).
Closes#134
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
CI gating for AI provenance marks already exists (audit_dir.py's -j
concurrency + SARIF export from #101), but that only runs after a
marked file has already been committed and pushed. Catch it at commit
time instead, using git's own hook point.
Adds two hooks via .pre-commit-hooks.yaml:
- watermarks-remover-check: fails the commit and lists findings when
staged files carry AI/C2PA marks. Wraps audit_lib.scan_file() /
is_actionable() -- the exact per-file logic audit_dir.py already
uses for CI, so the pre-commit gate and the CI gate agree on what
counts as actionable.
- watermarks-remover-clean (opt-in): rewrites staged files in place by
shelling out to clean_file.py --in-place per file (no duplicated
cleaning logic), then exits 1 so the developer reviews the diff and
re-stages -- the same convention as auto-fixing hooks like ruff --fix.
.pre-commit-hooks.yaml needed an explicit allow-rule in the deny-by-
default .gitignore, same as every other root-level config file already
listed there.
Closes#135
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Directory-scale cleaning already exists in the CLI (audit_dir.py, -j
concurrency, SARIF export from #101), but the HTTP service handled one
file per request. Any web app or CI step talking to the service over
HTTP instead of the CLI paid N full round trips to clean N files.
Extract the single-file /inspect and /clean logic into _inspect_payload
and _clean_payload so both the existing single-file endpoints and the
new batch endpoints run the identical pipeline — no duplicated cleaning
logic. A malformed entry in a batch (bad base64, unknown option,
unrecognized format) surfaces as that entry's "ok": false with an
"error" string instead of aborting the rest of the batch.
Capped at WATERMARKS_MAX_BATCH_FILES per request (default 50) as
defense-in-depth against a request packing many tiny files into one
call; the existing MAX_BODY_BYTES envelope cap already bounds total
payload size the same as a single-file request.
/openapi.json picks up both routes automatically since the spec is
generated from the route table.
Closes#136
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
clean_markdown() had two bugs in the same loop:
1. It crashed with IndexError on any frontmatter containing a blank
line, because line[0] was evaluated on an empty string.
2. It kept the nested children of a dropped top-level key, so dropping
`model:` left `name: claude-opus` behind and produced invalid YAML,
while still reporting the key as removed.
Rewrite the loop with a `dropping` flag that tracks whether the current
top-level key was dropped, and guard blank/comment lines before
indexing. inspect_markdown() is unchanged (already guards line[0]).
Adds 4 regression tests: blank-line crash, nested-key leak, inspect
round-trip, and comment/list preservation.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Layer A already preserves emoji ZWJ/VS glue, but still stripped other
load-bearing invisibles, silently corrupting real text:
- ZWNJ/ZWJ inside complex scripts (Persian میروم, Devanagari क्ष)
- flag emoji tag sequences (🏴 -> 🏴)
- orthographic Arabic/Syriac Cf marks (U+0600, U+06DD, U+070F, ...)
Extend the existing _decide()/glue machinery: keep ZWNJ/ZWJ when a
neighbour is a complex-script letter, keep tag chars after an emoji
base, and allowlist the orthographic Cf codepoints. The same characters
between plain ASCII stay carriers and are still stripped.
--strip-emoji-glue continues to strip all of them (paranoid mode).
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
clean_file.py and clean_image.py computed the failure exit code inside
the human-output branch, so `--json` always exited 0 even when the clean
left C2PA/AI signals behind. A script gating on `clean_file --json` would
treat a still-marked file as clean.
Move the residual (and degraded-PDF) decision out of the output branch in
both entry points so the exit code is the same regardless of --json.
Human output is unchanged; degraded best-effort PDF copies stay
non-failures.
Adds tests asserting json and human modes return the same exit code for
residual, clean, and degraded cases.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
