Implements the replay test of ARBI's keyed-Gumbel technical report
(u = PRF(Hash(key, window), token); exact Gamma-tail p-value; repeated-window
masking) as a stdlib-only detector (detect_gumbel.py) — no GPU, model, or
logits — wired into the TextDetector registry, /capabilities and the
iterative Layer B rewrite loop (--gumbel-key; priority gumbel > markllm >
lexical divergence) with a gumbel.before/after/cleared report.
Also cites the ARBI article in the README bibliography (renamed from
References) and names EXP/Gumbel in the open-LLM vendor row. Same-key-only:
valid against the same key, tokenizer, and PRF layout used at generation;
not a vendor oracle. Key is never logged.
Co-authored-by: guillaumemeyer <guillaumemeyer@users.noreply.github.com>
_clean_one() folded three different outcomes into one "skipped" bucket, and
main() returned 0 whenever changed_paths was empty. A clean_file.py that
raised, was killed, or died before writing its JSON therefore produced exit
0, the hook passed, and the uncleaned file went into the commit. pre-commit
only surfaces a passing hook's output in verbose mode, so the traceback on
stderr was invisible in a normal run: the hook silently failed open, against
the contract its own docstring states ("Exit 0 means every file was already
clean").
Judge the run by its report, not by its exit code
-------------------------------------------------
The discriminator cannot be "returncode != 0". clean_file.py deliberately
exits 1 for a *successful* clean that left residual signals behind, prints
valid JSON while doing so, and tests/test_json_exit_code.py pins that. So a
non-zero code is not by itself a failure. What actually separates the cases
is whether the child produced a parsable report: no report means it never
reached its write, and that file is still marked. rc == 2 keeps its meaning
(unrecognized format / oversized input) and still skips, unchanged.
Why exit 3 (EXIT_PARTIAL) and not 1 or 2
-----------------------------------------
1 is taken, and taking it again would be a lie: it means "I rewrote your
files, review the diff and re-stage", so reusing it for a crash sends the
developer looking for a diff that does not exist. 2 is usage/refusal — a bad
invocation — but the invocation was valid; it is also the code clean_file.py
returns for the skip we must keep reporting as 0, so overloading it invites
exactly the confusion this commit removes. 3 is EXIT_PARTIAL, already
defined in common.py as "partial scan, some files failed to scan; takes
precedence over actionable findings", and already used with that meaning by
audit_dir.py:162 and audit_website.py:555. A batch that could not process
some of its files is precisely a partial run, and the documented precedence
resolves the mixed case for free: a batch that cleaned one file and failed
on another exits 3, while still reporting both. pre-commit fails the hook on
any non-zero, so the code is for humans and direct/CI callers — which is the
reason to follow the convention the repo already has rather than invent one.
Reporting
---------
Failures get the summary block the sibling hooks use (headline, indented
path, indented reason) instead of a raw traceback behind a path prefix. Only
the last stderr line is quoted, which for an uncaught exception is the line
that names the cause; a child killed by a signal leaves no stderr, so the
exit status is reported instead.
The leftover .bak is named, not deleted
----------------------------------------
clean_file.py --in-place backs the original up (clean_file.py:100) before
the write that failed, so every failed run leaves a sidecar. This wrapper
does not remove it: it cannot tell its own leftover from a .bak the
developer already had staged, and deleting the wrong file from inside an
already-failing hook is the worse outcome. Nothing is lost by keeping it —
safe_write_bytes() replaces atomically or not at all, so the original is
intact and the sidecar is redundant rather than load-bearing. The report
names the path when it exists so the developer is not left wondering where
it came from. Not taking the backup until it is needed is a change to
clean_file.py's own --in-place ordering and belongs with that file.
Tests cover the crash, empty-stdout, malformed-JSON and killed-child paths,
the mixed cleaned-plus-failed precedence, an end-to-end symlink case that
needs no mocking (common.py:256 refuses to write through a symlink), and two
regressions: the intended rc == 2 skip still exits 0, and a residual-signal
clean is still reported as a clean, not a failure.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
c2patool exits non-zero both when an asset carries no manifest and when the
binary itself fails, and run_optional_tools only ever substring-matched the
output. A probe that died before main() produced has_manifest: False with no
other signal, so inspect reported has_c2pa: false -- the same answer it gives
for a genuinely clean asset.
Reproduced on Apple Silicon: service/Dockerfile pins a multi-arch base digest,
so an arm64 host builds an arm64 image while still installing the
x86_64-unknown-linux-gnu c2patool release (upstream publishes no linux-aarch64
build). c2patool dies with "rosetta error: failed to open elf at
/lib64/ld-linux-x86-64.so.2" and exit -5, and /capabilities kept reporting
c2patool: true because which() only finds the file.
Exposure is largest for PDF, where inspect_pdf documents exiftool and c2patool
as the more reliable detectors. For JPEG/PNG the native APP11 and PNG-chunk
scans still catch hard-bound C2PA, so there it costs corroboration rather than
the only signal.
- run_optional_tools marks a run conclusive only when it found a manifest or
said in so many words that there is none; anything else sets ok: False
- inspect_image and inspect_pdf surface that as an inconclusive note, worded
to land in the informational confidence bucket
- /capabilities probes each tool's version flag instead of trusting which()
Issues #1 and #3 fixed the opposite direction of this same function ("No claim
found" matching "claim"). This closes the false-negative side.
Co-authored-by: Italo Rodrigues <italo@pvwi.com.br>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* fix: strip three missed Default_Ignorable invisible carriers in Layer A
U+180F (Mongolian free variation selector-4, added in Unicode 14), U+3164
(Hangul filler), and U+FFA0 (halfwidth Hangul filler) are blank-rendering
Default_Ignorable code points, but their Mn/Lo categories meant the Cf
catch-all never saw them and they were absent from STRIP_CODEPOINTS. Both
inspect_text and clean_text therefore passed them through untouched, even
between plain ASCII.
Add them to the strip set and wire U+180F into the Mongolian-FVS handling
so it is stripped when floating but preserved after a Mongolian letter,
exactly like FVS1-3. Replace the 0x180B..0x180D range with the named
_MONGOLIAN_FVS set (clearer, and avoids the differently-handled Mongolian
vowel separator at U+180E). Applied to both the service engine and the
vendored lightweight-skill copy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HV9AE8QLqiXf7QnRe891EY
* feat: strip reserved Default_Ignorable code points in Layer A
U+2065, U+FFF0..U+FFF8, U+E0000, U+E0080..U+E00FF, and U+E01F0..U+E0FFF
are unassigned code points carrying Other_Default_Ignorable_Code_Point,
reserved for future default-ignorable characters. Conformant renderers
display them invisibly, normalisation preserves them, and the Cf
catch-all never sees them (they are category Cn), which made them ideal
covert carriers that both inspect_text and clean_text passed through
untouched.
Add them to the strip set as explicit ranges and report them under the
new "reserved_ignorable" inspect kind. Deliberately not a category-Cn
rule: unicodedata is pinned per Python build, so a Cn rule would destroy
characters assigned in newer Unicode versions. The table carries a
reminder to re-check the ranges on Unicode version bumps, since
assignment turns a strip entry into a potential preserve-in-context
case, exactly as happened when U+180F became Mongolian FVS4 in
Unicode 14.
Tests sweep all 3,739 code points through both the service engine and
the vendored lightweight-skill copy (clean and inspect CLIs), with a
boundary test pinning the assigned neighbours (U+2064, U+FFF9, U+E0001,
U+E0100) to their existing kinds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat: strip Unicode noncharacters in Layer A
The 66 noncharacters (U+FDD0..U+FDEF plus U+nFFFE/U+nFFFF at the end of
every plane) are permanently reserved for internal use and prohibited in
interchange text (TUS 23.7). They render as nothing or tofu, survive
normalisation and Python round-trips, and their category (Cn) meant the
Cf catch-all never saw them, so both inspect_text and clean_text passed
them through untouched: a ready-made covert channel.
Strip them and report them under the new "noncharacter" inspect kind.
Unlike the reserved Default_Ignorable ranges, noncharacters can never be
assigned, so this carries no future-Unicode risk. Applied to both the
service engine and the vendored lightweight-skill copy; tests sweep all
66 code points through both, and pin the assigned neighbours U+FDF0 and
U+FFFD (replacement character) as untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: preserve visible-layout format controls next to their own script
Egyptian hieroglyph quadrat controls (U+13430..U+1343F), Duployan
shorthand controls (U+1BCA0..U+1BCA3), and musical beam/tie/slur/phrase
controls (U+1D173..U+1D17A) are category Cf, so the catch-all stripped
them, yet they visibly govern how their script renders (quadrat
stacking, shorthand overlaps, beaming): removing them changes the
rendered text, contradicting the "cleaners preserve the document body"
invariant. Preserve them when adjacent to their own script, exactly
like the existing Mongolian/Khmer/Hangul handling; floating between
unrelated text they stay stripped and flagged, and --strip-emoji-glue
paranoid mode still strips them everywhere.
The vendored lightweight-skill engine needed next_input threaded
through its _decide to express "preserve a begin-control from the
character after it"; both engines now behave identically here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: preserve compatibility/halfwidth Hangul fillers in Hangul context
Review feedback on the Layer A cluster: U+3164 and U+FFA0 were stripped
unconditionally while the changelog claimed they were handled "like
U+115F/U+1160", which are preserved after a Hangul jamo. Make the
behaviour match the claim: both new fillers join _HANGUL_FILLERS and
_is_hangul_jamo now covers the compatibility (U+3131-U+318E) and
halfwidth (U+FFA1-U+FFDC) presentation forms, so each filler survives
after a letter of its own form and remains contraband when floating
between unrelated text. Changelog reworded to state the rule precisely.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* feat: add reproducible SynthID-text removal benchmark
bench_synthid_text.py orchestrates the existing Layer B machinery into a
controlled, shareable experiment: generate watermarked + unwatermarked
samples with the MarkLLM SynthID scheme, run removal variants (strength x
candidates) plus controls (no-removal, Layer-A-only, optional re-stamp),
and report clear rate, score suppression, quality, and cost (tokens,
wall time, USD) with a clears-per-MTok efficiency ratio.
Emits report.md / results.json / results.csv with the exact reproduction
command and pinned commits; optional Gemini official-detector tier when
WATERMARKS_GEMINI_API_KEY is set. Mock-based tests, no torch in CI.
* docs: add README section on running the SynthID-text benchmark
Explains what LLM performs the Layer B rewrite (an external model configured
via WATERMARKS_REWRITE_* env vars or --rewrite-* flags; MarkLLM's opt-1.3b is
only the watermark generator/detector) and how to run a benchmark with Ollama
or an OpenAI-compatible endpoint, plus the non-origin-model re-stamp caveat.
* fix: honor WATERMARKS_REWRITE_ALLOW_REMOTE in the SynthID-text benchmark
The --rewrite-allow-remote flag now defaults from the env var (matching
rewrite_text.py and the other WATERMARKS_REWRITE_* settings), so a
non-loopback rewrite endpoint works after sourcing .env without an extra
flag.
* fix: MarkLLM sparse checkout and deps for the SynthID harness
- setup_markllm.sh sparse-checkout omitted '/visualize/', which
watermark/base.py imports at module load — every scheme (incl. SynthID)
failed with 'No module named visualize' during generation/detection.
- requirements-markllm.txt omitted scikit-learn, imported by the SynthID
detector (watermark/synthid/detector_bayesian_torch.py).
Both broke the MarkLLM harness at runtime; the benchmark's sanity gate then
excluded every sample, producing empty per-variant results.
* fix: drop 4 GiB RLIMIT_AS on benchmark subprocesses
_run_cmd applied the common child RLIMIT_AS (default 4 GiB) via
subprocess_preexec_fn to every MarkLLM/rewrite child. torch needs a much
larger address space: CUDA init failed with 'out of memory' at
cudaGetDeviceCount and the 5.2 GB fp32 opt-1.3b could not load, so every
sample was excluded at generation. text_detectors.py already applies no
address-space cap to MarkLLM by default; the benchmark now matches.
* perf: keep MarkLLM resident via a serve worker (624 cold starts -> 1)
The benchmark spawned a fresh torch + opt-1.3b process per operation
(~60-90s each); a full run needs ~624 of them. detect_text_watermark.py
gains a 'serve' mode (JSON-lines over stdin/stdout, ready handshake) that
loads the model once; bench_synthid_text.py uses it via MarkLLMWorker with
automatic fallback to one-shot subprocesses (--no-worker to force).
Turns ~8h runs into ~40-60min.
* perf: skip per-candidate Gemini detections in rewrite subprocess
* feat: run the SynthID-text benchmark from the wr-markllm compose service
- Dockerfile.markllm: add '/visualize/' to the sparse checkout (same fix as
setup_markllm.sh) and COPY the benchmark + rewrite scripts (stdlib-only).
- compose.yaml: wr-markllm gets the WATERMARKS_REWRITE_* and
WATERMARKS_GEMINI_* env wiring, a bench-out volume for --out-dir, and a
read-only mount of the bundled corpus (build context is service/, so the
corpus cannot be COPY'd).
- docs: docker compose run example.
Note: the image ships CPU torch by design, so the container path is for
portability/CI; GPU runs use the host setup_markllm.sh venv.
* feat: per-sample progress logging in the benchmark
The persistent worker returns samples in-memory, so nothing is written
until the end of a run — runs looked stuck. eprint a [gen i/N] line per
generated sample and a [removal] summary per sample.
* chore: migrate Gemini config to gemini-3.6-flash; document SynthID-text retirement
Google retired SynthID text watermarking on the Generative Language API
(Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK
is rejected on current 3.x models (confirmed by Google AI staff). Migrate
the default detection model to gemini-3.6-flash, document the retirement
in vendor-notes.md and the benchmark report caveat, and keep the detector
seam fail-soft until a vendor endpoint (e.g. Vertex AI) returns.
* feat: remove gemini-synthid-text detector (Google retired text watermarking)
Google removed SynthID text watermarking from the Generative Language API
(Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK
is rejected on current 3.x models, so the vendor detector had nothing to
detect. Remove GeminiSynthIDTextDetector and its wiring:
- text_detectors.py: drop the Gemini class, HTTP helpers, and constants;
keep MarkLLM + Claude seams (registry now markllm + claude-text).
- server.py / rewrite_text.py: per-candidate detection now triggers on
--markllm-scheme only.
- bench_synthid_text.py: remove the Gemini tier (before/after, report
table, --no-gemini flag); report caveat notes the retirement.
- configs/docs: drop WATERMARKS_GEMINI_* from .env.example / compose /
README / SKILL.md / vendor-notes.md; keep the retirement note.
- tests: gemini tests removed or converted to MarkLLM (mocked subprocess).
- Dockerfile.markllm: parameterize BASE_IMAGE + TORCH_INDEX_URL so a GPU/
arm64 image can be built (used for the --gpus all benchmark run).
* feat: iterative detection-guided Layer B rewriting (default 3 attempts)
Layer B (rewrite_text.py) now rewrites iteratively and stops as soon as an
attempt passes watermark evaluation:
- --candidates defaults to 3 (WATERMARKS_REWRITE_CANDIDATES); each attempt is
one rewrite + one evaluation, and the loop exits on the first attempt the
evaluator reports as not watermarked.
- Evaluator priority: MarkLLM same-config detection (when --markllm-scheme is
passed) > bigram-Jaccard lexical divergence (fallback; no verdict, all
attempts generated, most diverged selected). A vendor-detector seam is
reserved ahead of MarkLLM for a future SynthID-text endpoint (Google retired
text watermarking on its API in Aug 2026).
- Best-effort fallback when the max is exhausted: the lowest-score attempt is
returned with a note; detector errors are fail-soft and never fail the
rewrite.
- --json-stats now reports evaluator / attempts_made / passed and per-attempt
candidate_scores records (passed, evaluation); markllm before/after/cleared
is unchanged and the selected attempt's verdict is reused (no duplicate
MarkLLM detection).
Benchmark (bench_synthid_text.py):
- --variants default becomes paraphrase:3 (candidates = max attempts).
- Rows/report/CSV carry attempts per document (mean_attempts, att column;
attempts / evaluator / passed columns).
Tests, README, docs/synthid-text-benchmark.md and .env.example updated; 490
tests pass, ruff clean.
* feat: split rewrite attempts into --candidates x --max-loops (defaults 1 x 1)
Follow-up to the iterative Layer B rewrite: separate "variants per round"
from "evaluation rounds", so the retry loop is explicit and defaults stay
conservative.
- rewrite_text.py: --candidates (WATERMARKS_REWRITE_CANDIDATES) is now the
number of variants generated per loop iteration (default 1); new
--max-loops (WATERMARKS_REWRITE_LOOPS) caps the evaluation rounds (default
1) -- each round generates --candidates variants and stops as soon as one
passes, so raising --max-loops retries new variants until an evaluation
passes. Stats now report max_loops and per-attempt records carry the loop
index.
- bench_synthid_text.py: new --rewrite-loops flag (default 1) passed through
to --max-loops.
- README / docs / .env.example updated; tests cover the 1x1 defaults, loop
retry until pass, and cross-loop exhaustion.
* feat: MarkLLM serve worker over loopback TCP (WATERMARKS_MARKLLM_PORT)
detect_text_watermark.py serve can now also listen on a loopback TCP port,
and MarkLLMTextDetector reuses a resident worker when
WATERMARKS_MARKLLM_PORT is set (falls back to a one-shot subprocess when the
worker is unreachable). This avoids a ~20s torch+model cold start per detect
for callers that run a worker out-of-band.
Tests: loopback worker protocol + detector worker-port routing (mock-based).
* ci: add macOS runner to the test matrix
* feat: add reproducible SynthID-text removal benchmark
bench_synthid_text.py orchestrates the existing Layer B machinery into a
controlled, shareable experiment: generate watermarked + unwatermarked
samples with the MarkLLM SynthID scheme, run removal variants (strength x
candidates) plus controls (no-removal, Layer-A-only, optional re-stamp),
and report clear rate, score suppression, quality, and cost (tokens,
wall time, USD) with a clears-per-MTok efficiency ratio.
Emits report.md / results.json / results.csv with the exact reproduction
command and pinned commits; optional Gemini official-detector tier when
WATERMARKS_GEMINI_API_KEY is set. Mock-based tests, no torch in CI.
* docs: add README section on running the SynthID-text benchmark
Explains what LLM performs the Layer B rewrite (an external model configured
via WATERMARKS_REWRITE_* env vars or --rewrite-* flags; MarkLLM's opt-1.3b is
only the watermark generator/detector) and how to run a benchmark with Ollama
or an OpenAI-compatible endpoint, plus the non-origin-model re-stamp caveat.
* fix: honor WATERMARKS_REWRITE_ALLOW_REMOTE in the SynthID-text benchmark
The --rewrite-allow-remote flag now defaults from the env var (matching
rewrite_text.py and the other WATERMARKS_REWRITE_* settings), so a
non-loopback rewrite endpoint works after sourcing .env without an extra
flag.
* fix: MarkLLM sparse checkout and deps for the SynthID harness
- setup_markllm.sh sparse-checkout omitted '/visualize/', which
watermark/base.py imports at module load — every scheme (incl. SynthID)
failed with 'No module named visualize' during generation/detection.
- requirements-markllm.txt omitted scikit-learn, imported by the SynthID
detector (watermark/synthid/detector_bayesian_torch.py).
Both broke the MarkLLM harness at runtime; the benchmark's sanity gate then
excluded every sample, producing empty per-variant results.
* fix: drop 4 GiB RLIMIT_AS on benchmark subprocesses
_run_cmd applied the common child RLIMIT_AS (default 4 GiB) via
subprocess_preexec_fn to every MarkLLM/rewrite child. torch needs a much
larger address space: CUDA init failed with 'out of memory' at
cudaGetDeviceCount and the 5.2 GB fp32 opt-1.3b could not load, so every
sample was excluded at generation. text_detectors.py already applies no
address-space cap to MarkLLM by default; the benchmark now matches.
* perf: keep MarkLLM resident via a serve worker (624 cold starts -> 1)
The benchmark spawned a fresh torch + opt-1.3b process per operation
(~60-90s each); a full run needs ~624 of them. detect_text_watermark.py
gains a 'serve' mode (JSON-lines over stdin/stdout, ready handshake) that
loads the model once; bench_synthid_text.py uses it via MarkLLMWorker with
automatic fallback to one-shot subprocesses (--no-worker to force).
Turns ~8h runs into ~40-60min.
* perf: skip per-candidate Gemini detections in rewrite subprocess
* feat: run the SynthID-text benchmark from the wr-markllm compose service
- Dockerfile.markllm: add '/visualize/' to the sparse checkout (same fix as
setup_markllm.sh) and COPY the benchmark + rewrite scripts (stdlib-only).
- compose.yaml: wr-markllm gets the WATERMARKS_REWRITE_* and
WATERMARKS_GEMINI_* env wiring, a bench-out volume for --out-dir, and a
read-only mount of the bundled corpus (build context is service/, so the
corpus cannot be COPY'd).
- docs: docker compose run example.
Note: the image ships CPU torch by design, so the container path is for
portability/CI; GPU runs use the host setup_markllm.sh venv.
* feat: per-sample progress logging in the benchmark
The persistent worker returns samples in-memory, so nothing is written
until the end of a run — runs looked stuck. eprint a [gen i/N] line per
generated sample and a [removal] summary per sample.
* chore: migrate Gemini config to gemini-3.6-flash; document SynthID-text retirement
Google retired SynthID text watermarking on the Generative Language API
(Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK
is rejected on current 3.x models (confirmed by Google AI staff). Migrate
the default detection model to gemini-3.6-flash, document the retirement
in vendor-notes.md and the benchmark report caveat, and keep the detector
seam fail-soft until a vendor endpoint (e.g. Vertex AI) returns.
* feat: remove gemini-synthid-text detector (Google retired text watermarking)
Google removed SynthID text watermarking from the Generative Language API
(Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK
is rejected on current 3.x models, so the vendor detector had nothing to
detect. Remove GeminiSynthIDTextDetector and its wiring:
- text_detectors.py: drop the Gemini class, HTTP helpers, and constants;
keep MarkLLM + Claude seams (registry now markllm + claude-text).
- server.py / rewrite_text.py: per-candidate detection now triggers on
--markllm-scheme only.
- bench_synthid_text.py: remove the Gemini tier (before/after, report
table, --no-gemini flag); report caveat notes the retirement.
- configs/docs: drop WATERMARKS_GEMINI_* from .env.example / compose /
README / SKILL.md / vendor-notes.md; keep the retirement note.
- tests: gemini tests removed or converted to MarkLLM (mocked subprocess).
- Dockerfile.markllm: parameterize BASE_IMAGE + TORCH_INDEX_URL so a GPU/
arm64 image can be built (used for the --gpus all benchmark run).
* fix: harden notes aggregation against non-string notes
A run completed all samples but crashed at the final aggregate step with
'cannot use list as a set element' when a row's notes contained a
non-string value. Filter notes to strings (aggregate + CSV) and add a
regression test.
* perf: let the rewrite subprocess reuse the resident MarkLLM worker
The rewrite subprocess (rewrite_text.py) ran its own before/after MarkLLM
detects, each a ~20s torch+model cold start (~12 per sample = ~5min of the
~6min/sample runtime). Now:
- detect_text_watermark.py serve gains --port N: a loopback TCP JSON-lines
listener (default -1 = off) sharing the resident model, with a lock so
stdin and socket requests never run the model concurrently.
- text_detectors.MarkLLMTextDetector checks WATERMARKS_MARKLLM_PORT and
does a fast loopback detect when a worker is up, falling back to the
one-shot subprocess otherwise.
- The benchmark worker publishes its port via that env var, so the rewrite
subprocess inherits it and its detects hit the resident model.
Turns ~6 min/sample into ~1-2 min; a full run drops from ~2h to ~40-50min.
Tests: loopback-client + fallback + env-publish coverage.
* chore: add benchmark-smoke.sh / benchmark-full.sh wrappers
Simple host wrappers: source .env, default MARKLLM_DIR to ~/MarkLLM, use a
repo-local HF cache by default, and run bench_synthid_text.py with a quick
(2 docs, 1 seed, paraphrase:1) or full (8 docs x 3 seeds, three variants,
re-stamp control) configuration. OUT_DIR overrides the output location.
The file-cleaners layer covered 15 formats -- all image, document, or
text -- and zero audio/video. That gap gets more expensive every month:
Sora, Veo, ElevenLabs, and Suno all embed provenance through the same
mechanisms image generators do, just in different containers.
New av_meta.py adds inspect/clean for:
- MP4/MOV/M4A/M4V: top-level C2PA (jumb/c2pa box) and XMP (uuid box)
detection/stripping reuse inspect_isobmff()/strip_isobmff() from
image_meta.py unchanged -- that's exactly the mechanism the C2PA spec
defines for ISOBMFF-family containers, already proven for AVIF/HEIC.
moov/udta (where generator/tool tags live) is handled separately since
it's MP4-specific.
- WAV: RIFF LIST INFO chunk + embedded id3 chunk.
- MP3: ID3v2 frames, per-frame for v2.3/v2.4, whole-tag fallback for
v2.2 (3-byte frame IDs are detected but not decomposed, so a partial
rewrite is never attempted there).
Every box/chunk/frame is either kept byte-identical or dropped whole --
nothing does a partial in-place rewrite of a payload, so a container can
never come out semantically mangled. Default strip_all_metadata=True
matches this project's existing default (privacy-first: drop everything,
--keep-non-ai-metadata narrows to only AI-flagged content), same as the
image cleaners.
Wired through the full dispatch stack so the feature isn't a half
integration: format_dispatch.py (new "av" Kind), inspect_file.py /
clean_file.py (--as av), audit_lib.py (so audit_dir.py's CI/SARIF path
and the pre-commit hooks from #135 both cover audio/video too), and
server.py (HTTP /inspect and /clean).
Closes#134
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
CI gating for AI provenance marks already exists (audit_dir.py's -j
concurrency + SARIF export from #101), but that only runs after a
marked file has already been committed and pushed. Catch it at commit
time instead, using git's own hook point.
Adds two hooks via .pre-commit-hooks.yaml:
- watermarks-remover-check: fails the commit and lists findings when
staged files carry AI/C2PA marks. Wraps audit_lib.scan_file() /
is_actionable() -- the exact per-file logic audit_dir.py already
uses for CI, so the pre-commit gate and the CI gate agree on what
counts as actionable.
- watermarks-remover-clean (opt-in): rewrites staged files in place by
shelling out to clean_file.py --in-place per file (no duplicated
cleaning logic), then exits 1 so the developer reviews the diff and
re-stages -- the same convention as auto-fixing hooks like ruff --fix.
.pre-commit-hooks.yaml needed an explicit allow-rule in the deny-by-
default .gitignore, same as every other root-level config file already
listed there.
Closes#135
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Directory-scale cleaning already exists in the CLI (audit_dir.py, -j
concurrency, SARIF export from #101), but the HTTP service handled one
file per request. Any web app or CI step talking to the service over
HTTP instead of the CLI paid N full round trips to clean N files.
Extract the single-file /inspect and /clean logic into _inspect_payload
and _clean_payload so both the existing single-file endpoints and the
new batch endpoints run the identical pipeline — no duplicated cleaning
logic. A malformed entry in a batch (bad base64, unknown option,
unrecognized format) surfaces as that entry's "ok": false with an
"error" string instead of aborting the rest of the batch.
Capped at WATERMARKS_MAX_BATCH_FILES per request (default 50) as
defense-in-depth against a request packing many tiny files into one
call; the existing MAX_BODY_BYTES envelope cap already bounds total
payload size the same as a single-file request.
/openapi.json picks up both routes automatically since the spec is
generated from the route table.
Closes#136
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* feat: add vendor text-watermark detection and SynthID image scorer sidecar
Adds Layer B watermark detection as a first-class service capability:
- text_detectors.py: a registry of text-watermark detectors behind one
interface — Google's official SynthID-text detector via the Gemini API
(taskType DETECT_TEXT_WATERMARK), a Claude placeholder ready for
Anthropic's announced detection API, and the MarkLLM research harness
(KGW / SynthID, same-config-only). Fail-soft: unconfigured or errored
detectors never block cleaning.
- server.py: new POST /detect endpoint, detect_before / detect_after options
on /clean (before/after scoring for text and images), an opt-in
/inspect "detect" flag, and /capabilities gains text_detectors and
scorers.synthid_http.
- synthid_score_server.py: a stdlib HTTP sidecar for the reverse-SynthID
scorer, so the published core image never bundles the non-commercial
upstream code; wired via WATERMARKS_SYNTHID_SCORER_URL.
- score_synthid.py: extract score_file() so the CLI and the sidecar share
one implementation.
- compose.yaml / Dockerfile.synthid / .env.example: wr-synthid-score sidecar
service and env wiring.
- README + skill docs, plus tests for the detectors, the /detect endpoint,
and the image sidecar.
* feat: per-candidate watermark detection for Layer B rewrite candidates
When --candidates N (N > 1) is combined with --markllm-scheme or
WATERMARKS_GEMINI_API_KEY, run every configured text detector from the
text_detectors.py registry on each candidate and report per-candidate
measurements in --json-stats as candidate_scores entries carrying
lexical_divergence, selection_score, selected, and per-detector reports
(is_watermarked, score, threshold where the detector provides one).
Candidate selection stays purely lexical; the detections are observability
for correlating lexical divergence with watermark removal (issue #106).
Converges rewrite_text.py onto the shared detector registry:
- MarkLLMTextDetector gains constructor overrides (scheme, upstream_dir,
model, timeout) plus the checkout-venv interpreter preference and the
WATERMARKS_MARKLLM_RLIMIT_AS preexec guard ported from rewrite_text.py;
the old _markllm_detect / _venv_python / _markllm_preexec helpers are gone.
- run_all_text_detectors() accepts an injected MarkLLM instance and an
include_markllm switch so CLI flag gating stays intact.
- before/after/cleared semantics unchanged; detection remains fail-soft.
* docs: pin Watermarks in the Sand reference to arXiv v5
* fix: mark only one rewrite candidate as selected (#110)
---------
Co-authored-by: Zhenxin Ai <142008897+ai-kunkun@users.noreply.github.com>
* fix: rewrite ODT/EPUB manifests and measure real zip bytes (#122)
Two container correctness/security fixes from issue #122:
- clean_odt dropped marker-bearing parts while leaving their entries in
META-INF/manifest.xml, so readers flagged the package as damaged. It is
now two-pass: compute the dropped set, then rewrite the manifest
attribute-order-independently, and write each part exactly once. The same
bug class in clean_epub (dropped parts left in the OPF manifest, plus
dangling spine itemrefs) gets the same two-pass treatment.
- The zip budget trusted ZipInfo.file_size from the archive's own central
directory, so a crafted DOCX/ODT could declare a tiny size and still
expand via zf.read. Budgets are now charged on actual decompressed bytes
via _read_zip_member (streaming, cap enforced mid-read), with the declared
size kept only as a fast-path pre-reject.
* fix: classify unrecognized bytes as "unknown", not text (#122)
Two classification defects from issue #122:
- format_dispatch.classify_bytes fell back to "text" for any unrecognized
file, so a binary with valid UTF-8 runs could be decoded and written back
mangled (corrupted with --in-place) in clean_file auto mode. Unrecognized
bytes now classify as "unknown"; clean_file refuses them in auto mode
(exit 2, no write, router advice) and --as text / --force-text are the
explicit opt-ins. inspect_file reports kind "unknown" (exit 0), audit_lib
records a non-actionable item, and the HTTP server answers /inspect with
kind "unknown" but rejects /clean of unknown formats (400).
- classify(path) read the whole file to sniff a header, and only a full read
could detect zip containers. It now routes known extensions without
reading, sniffs a 4096-byte header once for images and prefix-based
containers, and reads the whole file only when the header is a zip local
header (PK), where the container signature lives in the central directory.
* feat: distinct exit code for partial audits (#122)
audit_dir and audit_website reported success (0) even when some files or
URLs could not be scanned; the exit status was computed only over the items
that succeeded. A scan that is missing items is not a clean scan.
- common.EXIT_PARTIAL = 3, with precedence: partial (3) > actionable (1)
> clean (0) — an incomplete audit is the more important CI signal.
- audit_dir returns 3 when any file was skipped/failed; audit_website
returns 3 when any URL failed to fetch or inspect. Both are independent
of the output format (human/json/sarif already share one return).
* fix: verify the pinned upstream ref on existing checkouts (#122)
setup_ctrlregen.sh/setup_synthid.sh (and their .ps1 twins) only verified
the pinned commit in the fresh-clone branch; an existing checkout at an
unknown or drifted revision was silently reused, defeating the commit pin.
All four scripts now check HEAD against the pinned ref in the
existing-checkout branch too, and repair by fetch + detach checkout
(re-applying the sparse-checkout set), failing hard if the ref cannot be
reached or the re-pin does not land on it.
* docs: unknown-format behavior, audit exit codes, backend isolation (#122)
- README: clean_file no longer auto-cleans unrecognized formats (--as text
/ --force-text are the opt-ins), and the CtrlRegen bootstrap documents the
isolation expectation for its research-era dependency pins plus the new
re-pin check on existing checkouts.
- SKILL.md: audit exit codes (0/1/2/3, partial=3) and a note that /clean
requires a name with a known extension.
- audit_website: document why stdlib ElementTree is used (stdlib-first) and
that defusedxml is the fallback if that policy changes (DTD rejection stays).
- requirements-ctrlregen.txt: advisory/isolation note for the pinned research
dependencies.
* test: ODT manifest and EPUB OPF dangling-ref regressions (#122)
- clean_odt: dropped marker-bearing parts remove their META-INF/manifest.xml
file-entry (attribute-order-independent), exactly one manifest entry, root
and surviving entries kept, and the manifest is byte-identical when nothing
is dropped.
- clean_epub: dropped non-content parts lose their <item> entry in the OPF
manifest, so the book no longer references removed members.
Three independent failure modes from #117:
- $ErrorActionPreference 'Stop' + 2>$null on a native command aborts the
script on torch's harmless stderr warnings (e.g. "Failed to initialize
NumPy" when torch is installed before numpy). Run the probes through a
new Invoke-NativeQuiet helper that lowers EAP to 'Continue' for the
block and restores it afterwards.
- The wheel index tag was derived from the driver's CUDA version, e.g.
cu131 for a 13.1 driver, which does not exist (HTTP 403) and silently
fell back to the default index, i.e. the CPU build on Windows. Probe
the published indices and pick the highest one <= driver that answers
HTTP 200; cu126 is still forced below compute capability 7.5.
- Installing torch alone let requirements-ctrlregen.txt resolve torchvision
from PyPI, and torchvision pins an exact torch, so pip replaced the +cu
build with a +cpu one while the script still exited 0. Install torch AND
torchvision together from the chosen index, and verify after the
requirements install that torch.cuda.is_available() is true - if a GPU
was detected but torch ends up CPU-only, warn loudly and exit non-zero.
Also add a CI step (windows-latest, pwsh) that parses the setup .ps1
scripts and asserts the post-install CUDA verification survives.
Fixes#117
`base64 -w0 FILE` is a GNU coreutils form. On macOS/BSD it fails with
`base64: invalid argument FILE` and prints usage, so every documented
`/inspect` and `/clean` example is unrunnable on a Mac host.
This matters more than a normal docs typo because SKILL.md instructs the
agent to call the service via these exact commands and explicitly forbids
falling back to local cleaning when the service call fails. A macOS agent
therefore stops at its first request instead of degrading gracefully.
Replace all 7 occurrences (2 in README.md, 5 in SKILL.md) with the POSIX
form, which reads from stdin and unwraps any line breaks:
base64 < FILE | tr -d '\n'
Verified byte-identical output on both implementations:
- BSD/macOS base64 (macOS 15, Darwin 24.6.0)
- GNU coreutils base64 (`gbase64`)
Also verified the rewritten commands verbatim against a live
`server.py` on 127.0.0.1:8765 — `/inspect` and `/clean` both return
`ok: true` and correctly strip U+200B / U+00AD.
Docs-only; no code paths changed, so no unit tests added.
`python3 -m pytest -q` and `ruff check`/`format --check` pass.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* docs: add Windows auto-start guide via Task Scheduler
* docs: fix markdown escaping, remove admin requirement, use path placeholder, link from README
* docs: add watermarks-remover-web to Ecosystem (#77)
* docs: note AVIF and HEIC in the watermarks-remover-web entry
The client now ports the ISOBMFF paths from image_meta.py as well, so the
entry's format list was out of date.
* docs: rename the web UI entry to unmark-web and keep it higher-level
Renames the project per the maintainer's request, and drops the
references to internal module names, the client's proxy script and the
format list, which were the parts most likely to drift.
---------
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Add stdlib-only detection, inspection, and cleaning for four more formats.
- BMP: locate the pixel payload via the DIB header and strip trailing
non-image metadata (the only place non-standard BMP metadata can live),
rewriting the file-size field.
- GIF: drop comment and XMP application extensions while preserving
NETSCAPE2.0 looping, ICC, graphic-control, and image blocks.
- TIFF (classic + BigTIFF): walk IFD chains and drop XMP/EXIF/GPS/IPTC/
Photoshop/MakerNote tags, zeroing orphaned payloads while keeping
strip/tile offsets valid.
- EPUB: scrub OPF package metadata and XHTML meta/JSON-LD, clean embedded
raster/SVG media, apply Layer A to XHTML body text, and pass OCF-encrypted
parts through untouched.
All four route through format_dispatch, so the unified CLIs, the HTTP
service, and the audits pick them up automatically.
* fix: pin vendored Cursor-skill text engine to the service copy
The engine vendored into skills/clean-user-facing-text/ had silently
drifted behind service/scripts/text_unicode.py: it still blanket-
stripped legitimate RTL directional marks and isolates (corrupting
mixed RTL/LTR prose) and stripped emoji variation selectors after
arrow and symbol bases, because it never received the preservable-bidi
and emoji-base updates. Nothing in the suite compared the two copies.
Replace the vendored engine with an exact copy of the service one and
add a byte-equality test so any future engine change must land in both
files in the same commit, plus a behavioural regression test for the
RTL and arrow cases through the vendored CLI. The CLI wrappers remain
deliberately different (text-only skill: no stylometry, no
--strip-bidi).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: re-pin vendored engine after merging main and correct bidi comment
Merging main into the PR branch kept main's ruff-formatted service
engine (#103) but the PR's vendored copy pinned in 519c6db, so the two
drifted apart again and test_vendored_text_unicode_is_identical_to_service_engine
failed. Re-sync the vendored copy to the current service engine so the
byte-for-byte pin holds.
Also correct the _PRESERVABLE_BIDI_CPS comment: paired LRE/RLE embeddings
are preserved via _valid_bidi_embedding_indices, so only overrides and
unpaired embeddings remain destructive by default.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* fix(inspect): run Layer A scan on markdown/html containers
inspect_container() never scanned the text body, so a .md or .html file
carrying invisible Unicode was reported suspicious=false while
clean_container() went on to strip it. Identical bytes gave opposite
verdicts depending on the file extension.
Scan Layer A for exactly the formats clean_container() scrubs (markdown,
html) so inspect predicts clean. Decode with surrogateescape to match
clean's decoding. Expose the count as suspicious_total, the same key
TextInspectReport uses, so the HTTP server's suspicious flag and the
inspect_file CLI exit code pick it up without special-casing.
* docs: changelog entry for the container Layer A inspect fix
* fix(audit): drop duplicate Layer A scan for markdown/html containers
inspect_container() now scans the body for markdown/html, so
audit_lib.scan_file's own Layer A scan produced the same findings twice
(once as 'layer-a:' from the container report, once as 'layer-a [kind]')
and double-counted them in the aggregate. Keep the stylometry check,
which still needs the decoded body text.
---------
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* feat: add lightweight Cursor and Codex text skill
Package the text-only workflow with safe cross-platform installation, optional persistent instructions, and focused tests so users can adopt it without the media tooling.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: keep the lightweight skill independently reviewable
Avoid coupling the packaging PR to the separate Unicode safety change so either pull request can merge on its own.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: force UTF-8 for subprocess stdin in Windows CI
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* feat: split skill from service, add HTTP API and Docker distribution
The agent skill (skills/remove-ai-marks/) is now a code-free remote client:
all implementation moved to service/scripts/ and runs behind a stdlib HTTP
service (server.py) with /health, /capabilities, /inspect, /clean and a
dynamically generated OpenAPI 3.0.3 spec at /openapi.json.
- Move scripts/ and the backend Dockerfiles under service/
- server.py: JSON/base64 HTTP entrypoint with size caps, binary guard,
atomic writes, loopback default, optional bearer auth
- Core Dockerfile (exiftool/qpdf/c2patool preinstalled) and a GHCR publish
workflow for the core/markllm/markdiffusion images
- compose.yaml (wr-* services, harness/heavy profiles) + compose-check.sh
to validate the running stack (exit code only)
- Fix markllm image build (tokenizers 0.22.2, CPU-only torch) and ctrlregen
build (python:3.11 base for the 2023-era research pins)
- Fix markllm/markdiffusion harness images missing common.py at runtime
* docs: add .env.example and service configuration guide
* fix: disable chain-of-thought for openai-compatible Layer B rewrites
deepseek-v4-flash is a reasoning model: a one-line paraphrase burned 9,894
reasoning tokens (~100s) and hit the default timeout. Send
reasoning_effort=none by default for the openai-compatible backend
(--reasoning-effort / WATERMARKS_REWRITE_REASONING_EFFORT; 'off' omits the
parameter), cutting the same rewrite to ~1s / 12 tokens. Tested end-to-end
against api.deepseek.com.
* fix: sanitize client-supplied filename in HTTP service
CodeQL 'uncontrolled data in path expression' (server.py): a name like
'../../x' flowed into Path(tmpdir) / name, letting an upload escape the
request temp dir on write. Sanitize name to its basename in _decode_input
(_safe_name) and refuse any joined path whose parent is not the tmpdir at
the write sites (_tmp_path). Tests cover traversal names.
* chore: gitignore .env (contains local rewrite credentials)
* chore: deny-by-default gitignore and dockerignore; document compose env config
.gitignore and service/.dockerignore now exclude everything by default and
explicitly allow only what is publishable/needed: tracked source, docs,
tests, .github, and (for images) the service/scripts/ tree that every
Dockerfile COPYs. Root .dockerignore documents that all builds use service/
as context. README Configuration section now covers .env setup for docker
compose, host-side export for CLI runs, and the full variable table.
* feat: optional MarkLLM text-watermark verification harness
Add an optional external backend wrapping THU-BPM/MarkLLM (Apache-2.0)
so a specific statistical text-watermark scheme (KGW / SynthID-Text)
can be verified before/after a Layer B rewrite.
- detect_text_watermark.py: detect/watermark subcommands, external
checkout at a pinned commit, exit codes 0/1/2/3, --json
- rewrite_text.py --markllm-scheme: before/after detection around the
rewrite, reports a `cleared` flag; never fails the rewrite when the
backend is unavailable
- setup_markllm.sh + requirements-markllm.txt (pinned deps) +
Dockerfile.markllm + Makefile bootstrap/smoke/docker targets
- tests/test_markllm_detect.py: 16 mock-based cases (no torch in CI)
- Docs: verification-harness caveat (same-config-only, not a
vendor-detector oracle) in README, SKILL.md, removal-matrix, vendor-notes
* chore: tidy merged Unreleased changelog list
* security: harden the MarkLLM harness (offline, caps, supply-chain)
Addresses the PR security review:
- detect_text_watermark.py: --offline loads the scoring model from the HF
cache only (local_files_only + HF_HUB_OFFLINE, no remote code), and the
algorithm config is capped at 1 MiB so a crafted huge file is refused
before either this script or upstream reads it into memory
- rewrite_text.py: WATERMARKS_MARKLLM_RLIMIT_AS (env, POSIX) optionally
applies RLIMIT_AS to the MarkLLM subprocess; off by default because
torch/CUDA needs large address spaces
- Dockerfile.markllm: drop the unpinned torch install (it is pinned in
requirements-markllm.txt) and verify the cloned upstream commit SHA
- tests: offline flag, config-too-large, and preexec/rlimit cases
- docs: hardening knobs in README + SKILL.md; changelog updated
setup_ctrlregen.sh and setup_synthid.sh are POSIX-only: they invoke
"$DIR/.venv/bin/python", a layout `python -m venv` never produces on Windows,
so both abort right after the clone. image_meta.py already resolves
`.venv\Scripts\python.exe` under `os.name == "nt"`, so only the bootstrap was
missing.
setup_ctrlregen.ps1 and setup_synthid.ps1 mirror the shell scripts: same
pinned upstream commits and SHA verification, same sparse-checkout paths, same
`pip==26.2.1` pin, same flags (as -Dir / -Ref / -Full / -Python).
One deliberate difference in setup_ctrlregen.ps1. The shell script derives the
torch wheel tag from the CUDA version nvidia-smi prints, but that is the
maximum the *driver* supports, not what to install - drivers are backward
compatible, so a cu126 wheel runs fine on a 13.0 driver. Worse, the naive tag
breaks on older cards: cu128+ and CUDA 13 dropped Maxwell/Pascal/Volta, so a
GTX 1080 Ti (sm_61) would get a cu130 torch with no kernels for it and fail at
runtime with "no kernel image is available for execution". Checking that the
index URL exists does not catch this - every index returns 200. The port keys
off the GPU's compute capability instead, forces cu126 below 7.5, and verifies
the installed wheel with torch.cuda.get_arch_list().
Both scripts parse clean; the CUDA detection was checked against a real
compute-capability 6.1 GPU. The .sh scripts are left untouched.
Co-authored-by: uniendodimensionesgm <153029553+uniendodimensionesgm@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
`exiftool -all=` on a PDF appends a %BeginExifToolUpdate block that frees the
Info object and drops /Info from the trailer, but the original metadata bytes
stay in the file verbatim - exiftool itself reverts the edit with
`-PDF-update:all=`. clean_pdf reported "exiftool -all= (rc=0)" and nothing
else, so a PDF still carrying `/Producer (Claude Opus)` in plain bytes read as
cleaned. The output file gets *larger*, which is the tell.
Follow the exiftool pass with `qpdf --linearize`, which re-serializes the
document from its object graph and so drops the now-unreferenced objects.
Guarded on which("qpdf"): without qpdf the behaviour is unchanged except for
an explicit warning that the metadata bytes remain recoverable.
Tests cover the no-qpdf warning, the rebuild replacing the exiftool output,
qpdf's rc=3 "succeeded with warnings", a failed rewrite leaving the exiftool
output in place, and an end-to-end check (skipped without the real tools)
asserting no readable metadata survives in the output bytes. That last one
fails on the leak itself when the rewrite is reverted.
Co-authored-by: uniendodimensionesgm <153029553+uniendodimensionesgm@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* Refuse binary input in the text-only tools
inspect_text.py, clean_text.py and rewrite_text.py accept any path and decode
it with errors="surrogateescape". Pointed at a .docx - a zip - they walk
deflate-compressed bytes and report whatever codepoints fall out of them. The
counts look like findings but track the compression, not the content: in one
sample set a document with nothing hidden in its text reported 12 "suspicious"
characters, while another with 54 real no-break spaces reported 11, none of
which were the no-break spaces.
clean_text.py is worse than misleading. It writes the mangled decode back, so
`clean_text.py report.docx` reports "removed=1" and silently corrupts the
document - the output still passes zipfile.is_zipfile() because the end-of-
central-directory record survives, but reading a member raises.
common.looks_binary() now sniffs magic numbers plus a control-byte ratio, and
guard_binary() refuses with a message naming the tool that does handle the
format. The ratio test is deliberately conservative so text in encodings other
than UTF-8 keeps working, and every entry point takes --force-text to override.
clean_file.py gets the same check on the branch where classify() falls back to
"text" for unrecognised bytes.
Adds tests covering magic-number and heuristic detection, the override, refusal
without writing or backing up, and that clean_file.py still routes a .docx to
the container path.
* Address review: backup ordering, stdin sniff, router advice
Three fixes from the review on #24.
clean_file.py sniffed after --in-place had already taken the backup, so
`clean_file.py --in-place mystery.bin` left a mystery.bin.bak sidecar behind
before exiting 2 — for a file the run never touches, and exactly what
clean_text.py avoids. The sniff now runs before backup_path(). The same hole
applied to `--as text` on a .docx, which bypasses classify() entirely.
The stdin path decoded before sniffing, which made detection depend on the
console codec. It was worse than codec drift: the text layer also translates
newlines, so PNG's `\x89PNG\r\n\x1a\n` arrived as `\x89PNG\n\x1a\n` and the
magic number never matched — the file was refused by the NUL-byte heuristic
instead, and would have sailed through had it lacked NULs. _read_stdin_capped
now reads sys.stdin.buffer and guards the raw octets, matching the file path,
with a text fallback for a replaced stdin.
guard_binary always advised "Use inspect_file.py / clean_file.py", which is
circular when the caller is one of them and classify() has already ruled out
every known container. The advice is now a parameter: the text-only scripts
keep the pointer to the routers, and the routers say the bytes match no
supported format and point at --force-text / --as.
Adds tests for the backup ordering (both --in-place paths), the advice split,
and stdin magic that is not ASCII, across default, cp1252 and latin-1 stdio
codecs — the previous stdin test piped a ZIP, whose "PK" header is ASCII and
survives any of them.
---------
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Wires a standalone clean_ctrlregen.py adapter plus setup_ctrlregen.sh bootstrap, Dockerfile, Makefile targets, and clean_image.py --remove-pixel ctrlregen. The backend is cloned at a pinned commit and never bundled (noai-watermark ships no LICENSE file). Includes mock-based tests and docs with research references.
The c2patool docs URL (opensource.contentauthenticity.org/docs/c2patool/)
returns 404. Update all four references in README and CONTRIBUTING to the
current repository location. Fixes#2.
Layer B removes statistical text marks by rewording, not restructuring,
and every rewrite pass taxes tone, voice, and precision. Document the
trade-off honestly, the full-circle economics of rewriting with a cheaper
model, and when to skip Layer B in favor of the lossless path.
Add Institute of AI PM C2PA/SynthID reference, soft-binding out-of-scope
notes, and residual-risk verification guidance so metadata strip is not
mistaken for full watermark clearance.
