* feat: optional MarkLLM text-watermark verification harness
Add an optional external backend wrapping THU-BPM/MarkLLM (Apache-2.0)
so a specific statistical text-watermark scheme (KGW / SynthID-Text)
can be verified before/after a Layer B rewrite.
- detect_text_watermark.py: detect/watermark subcommands, external
checkout at a pinned commit, exit codes 0/1/2/3, --json
- rewrite_text.py --markllm-scheme: before/after detection around the
rewrite, reports a `cleared` flag; never fails the rewrite when the
backend is unavailable
- setup_markllm.sh + requirements-markllm.txt (pinned deps) +
Dockerfile.markllm + Makefile bootstrap/smoke/docker targets
- tests/test_markllm_detect.py: 16 mock-based cases (no torch in CI)
- Docs: verification-harness caveat (same-config-only, not a
vendor-detector oracle) in README, SKILL.md, removal-matrix, vendor-notes
* chore: tidy merged Unreleased changelog list
* security: harden the MarkLLM harness (offline, caps, supply-chain)
Addresses the PR security review:
- detect_text_watermark.py: --offline loads the scoring model from the HF
cache only (local_files_only + HF_HUB_OFFLINE, no remote code), and the
algorithm config is capped at 1 MiB so a crafted huge file is refused
before either this script or upstream reads it into memory
- rewrite_text.py: WATERMARKS_MARKLLM_RLIMIT_AS (env, POSIX) optionally
applies RLIMIT_AS to the MarkLLM subprocess; off by default because
torch/CUDA needs large address spaces
- Dockerfile.markllm: drop the unpinned torch install (it is pinned in
requirements-markllm.txt) and verify the cloned upstream commit SHA
- tests: offline flag, config-too-large, and preexec/rlimit cases
- docs: hardening knobs in README + SKILL.md; changelog updated
setup_ctrlregen.sh and setup_synthid.sh are POSIX-only: they invoke
"$DIR/.venv/bin/python", a layout `python -m venv` never produces on Windows,
so both abort right after the clone. image_meta.py already resolves
`.venv\Scripts\python.exe` under `os.name == "nt"`, so only the bootstrap was
missing.
setup_ctrlregen.ps1 and setup_synthid.ps1 mirror the shell scripts: same
pinned upstream commits and SHA verification, same sparse-checkout paths, same
`pip==26.2.1` pin, same flags (as -Dir / -Ref / -Full / -Python).
One deliberate difference in setup_ctrlregen.ps1. The shell script derives the
torch wheel tag from the CUDA version nvidia-smi prints, but that is the
maximum the *driver* supports, not what to install - drivers are backward
compatible, so a cu126 wheel runs fine on a 13.0 driver. Worse, the naive tag
breaks on older cards: cu128+ and CUDA 13 dropped Maxwell/Pascal/Volta, so a
GTX 1080 Ti (sm_61) would get a cu130 torch with no kernels for it and fail at
runtime with "no kernel image is available for execution". Checking that the
index URL exists does not catch this - every index returns 200. The port keys
off the GPU's compute capability instead, forces cu126 below 7.5, and verifies
the installed wheel with torch.cuda.get_arch_list().
Both scripts parse clean; the CUDA detection was checked against a real
compute-capability 6.1 GPU. The .sh scripts are left untouched.
Co-authored-by: uniendodimensionesgm <153029553+uniendodimensionesgm@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
`exiftool -all=` on a PDF appends a %BeginExifToolUpdate block that frees the
Info object and drops /Info from the trailer, but the original metadata bytes
stay in the file verbatim - exiftool itself reverts the edit with
`-PDF-update:all=`. clean_pdf reported "exiftool -all= (rc=0)" and nothing
else, so a PDF still carrying `/Producer (Claude Opus)` in plain bytes read as
cleaned. The output file gets *larger*, which is the tell.
Follow the exiftool pass with `qpdf --linearize`, which re-serializes the
document from its object graph and so drops the now-unreferenced objects.
Guarded on which("qpdf"): without qpdf the behaviour is unchanged except for
an explicit warning that the metadata bytes remain recoverable.
Tests cover the no-qpdf warning, the rebuild replacing the exiftool output,
qpdf's rc=3 "succeeded with warnings", a failed rewrite leaving the exiftool
output in place, and an end-to-end check (skipped without the real tools)
asserting no readable metadata survives in the output bytes. That last one
fails on the leak itself when the rewrite is reverted.
Co-authored-by: uniendodimensionesgm <153029553+uniendodimensionesgm@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* Refuse binary input in the text-only tools
inspect_text.py, clean_text.py and rewrite_text.py accept any path and decode
it with errors="surrogateescape". Pointed at a .docx - a zip - they walk
deflate-compressed bytes and report whatever codepoints fall out of them. The
counts look like findings but track the compression, not the content: in one
sample set a document with nothing hidden in its text reported 12 "suspicious"
characters, while another with 54 real no-break spaces reported 11, none of
which were the no-break spaces.
clean_text.py is worse than misleading. It writes the mangled decode back, so
`clean_text.py report.docx` reports "removed=1" and silently corrupts the
document - the output still passes zipfile.is_zipfile() because the end-of-
central-directory record survives, but reading a member raises.
common.looks_binary() now sniffs magic numbers plus a control-byte ratio, and
guard_binary() refuses with a message naming the tool that does handle the
format. The ratio test is deliberately conservative so text in encodings other
than UTF-8 keeps working, and every entry point takes --force-text to override.
clean_file.py gets the same check on the branch where classify() falls back to
"text" for unrecognised bytes.
Adds tests covering magic-number and heuristic detection, the override, refusal
without writing or backing up, and that clean_file.py still routes a .docx to
the container path.
* Address review: backup ordering, stdin sniff, router advice
Three fixes from the review on #24.
clean_file.py sniffed after --in-place had already taken the backup, so
`clean_file.py --in-place mystery.bin` left a mystery.bin.bak sidecar behind
before exiting 2 — for a file the run never touches, and exactly what
clean_text.py avoids. The sniff now runs before backup_path(). The same hole
applied to `--as text` on a .docx, which bypasses classify() entirely.
The stdin path decoded before sniffing, which made detection depend on the
console codec. It was worse than codec drift: the text layer also translates
newlines, so PNG's `\x89PNG\r\n\x1a\n` arrived as `\x89PNG\n\x1a\n` and the
magic number never matched — the file was refused by the NUL-byte heuristic
instead, and would have sailed through had it lacked NULs. _read_stdin_capped
now reads sys.stdin.buffer and guards the raw octets, matching the file path,
with a text fallback for a replaced stdin.
guard_binary always advised "Use inspect_file.py / clean_file.py", which is
circular when the caller is one of them and classify() has already ruled out
every known container. The advice is now a parameter: the text-only scripts
keep the pointer to the routers, and the routers say the bytes match no
supported format and point at --force-text / --as.
Adds tests for the backup ordering (both --in-place paths), the advice split,
and stdin magic that is not ASCII, across default, cp1252 and latin-1 stdio
codecs — the previous stdin test piped a ZIP, whose "PK" header is ASCII and
survives any of them.
---------
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Wires a standalone clean_ctrlregen.py adapter plus setup_ctrlregen.sh bootstrap, Dockerfile, Makefile targets, and clean_image.py --remove-pixel ctrlregen. The backend is cloned at a pinned commit and never bundled (noai-watermark ships no LICENSE file). Includes mock-based tests and docs with research references.
The c2patool docs URL (opensource.contentauthenticity.org/docs/c2patool/)
returns 404. Update all four references in README and CONTRIBUTING to the
current repository location. Fixes#2.
Layer B removes statistical text marks by rewording, not restructuring,
and every rewrite pass taxes tone, voice, and precision. Document the
trade-off honestly, the full-circle economics of rewriting with a cheaper
model, and when to skip Layer B in favor of the lossless path.
Add Institute of AI PM C2PA/SynthID reference, soft-binding out-of-scope
notes, and residual-risk verification guidance so metadata strip is not
mistaken for full watermark clearance.
Add Makefile, pytest.ini, synthetic fixtures (MD/HTML/SVG), PDF degraded
clean test, and make smoke/test targets so the multi-vendor plan is fully closed.
Rename remove-claude-marks to remove-ai-marks, add container metadata
support (SVG/PDF/DOCX/ODT/HTML/MD), Layer B rewrite hook, unified file
CLI, and multi-vendor documentation for the first public release.
Publishable agent skill with Layer A Unicode scrub, agent-orchestrated
Layer B rewrite guidance, PNG/JPEG C2PA/metadata strip scripts, docs,
and tests.
