* fix: route website binary formats to their real scanners
guess_kind recognized html/png/jpeg/svg/pdf/docx/odt/markdown/text
and fell through to text for everything else: webp, avif, heic, gif,
bmp, tiff, xlsx, pptx, epub, mp4/mov/m4a, wav, and mp3 downloads were
written to asset.txt and handed to the Unicode scanner, so a WebP
carrying a C2PA RIFF chunk reported clean with no failure signal —
audit_website exited 0 with with_c2pa: 0 (#166). The drift between
this table and format_dispatch's IMAGE/CONTAINER/AV_EXTS was where
the bug lived.
Extend all three classification stages (content type, suffix, magic
bytes with ftyp brand dispatch) to cover every format the local audit
handles, and map the new kinds in _EXT_FOR_KIND so the temp file
keeps a true suffix for the local classify.
* fix: satisfy ruff lint and format checks
---------
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Co-authored-by: guillaumemeyer <guillaumemeyer@users.noreply.github.com>
* fix: truncated ISOBMFF containers still run the C2PA byte-scan fallback
inspect_isobmff returned early when no boxes parse, skipping the
whole-file C2PA byte scan that every sibling inspector (png, jpeg,
gif, tiff) reaches even after a truncation: a first box whose size
field overruns the file — the truncated AVIF/HEIC/MP4 download shape —
reported has_c2pa=False with only a not-a-valid finding while literal
b"c2pa" sat in the file (#167). Also reached for MP4/MOV/M4A through
av_meta._inspect_mp4.
Run the byte scan before reporting the parse failure: a truncated
container with markers byte-scans to has_c2pa=True (finding plus the
parse-failure note, so the truncation isn't hidden); a walkable
container keeps its stronger box-level findings; a markerless
truncated container keeps the plain not-a-valid shape.
* fix: satisfy ruff in isobmff truncated byte-scan tests
- drop the unused # noqa: E402 (already per-file-ignored for tests/**)
- bind the unused has_ai unpacked value to _ (RUF059)
---------
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Co-authored-by: guillaumemeyer <guillaumemeyer@users.noreply.github.com>
* fix: keep collected evidence when a later zip member fails to read
The OOXML, ODT, and EPUB inspectors accumulated has_c2pa/has_ai/
findings member by member, then discarded all of it if any later
member raised: the except returned hardcoded False, False, ["not a
valid X zip"] — markers already found in earlier members were thrown
away and a container that could not be fully read reported as one
that was read and found clean. The pre-commit gate then exited 0 on
exactly the file that carried evidence (#164).
Keep the accumulated evidence and append a partial-read note naming
the exception class; only a wholly-garbage container (nothing
accumulated) keeps the not-a-valid-zip shape. EPUB's except is
widened from BadZipFile to the shared _ZIP_PARSE_ERRORS tuple,
closing the asymmetry where a zlib error crashed the scan while a
CRC failure discarded everything.
* fix: silence RUF100/RUF059 lint errors in zip partial-evidence tests
---------
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Co-authored-by: guillaumemeyer <guillaumemeyer@users.noreply.github.com>
* fix: an unreadable text file is a failed scan, not a clean one
scan_file's text branch caught the OSError and returned an item with
an error key and no confidence key — is_actionable read that as
clean, so a file the scanner could not open was indistinguishable
from one it opened and found clean. All three audit callers inherited
the lie: the pre-commit hook returned 0, and audit_dir's _scan_worker
never produced the EXIT_PARTIAL signal its try/except exists for,
because scan_file swallowed the error before it could raise (#158;
the same probe-did-not-answer class as #155).
Let the OSError propagate: the audit_dir/audit_website wrappers
already convert a raised exception into a files_skipped entry with
EXIT_PARTIAL, which is exactly the contract common.py documents for
a partial scan.
* chore: satisfy ruff on the unreadable-text scan fix
- drop the unused OSError binding (F841) in scan_file's text branch
- sort the test's import block (I001): stdlib first, then audit_lib
---------
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Co-authored-by: guillaumemeyer <guillaumemeyer@users.noreply.github.com>
_clean_one() folded three different outcomes into one "skipped" bucket, and
main() returned 0 whenever changed_paths was empty. A clean_file.py that
raised, was killed, or died before writing its JSON therefore produced exit
0, the hook passed, and the uncleaned file went into the commit. pre-commit
only surfaces a passing hook's output in verbose mode, so the traceback on
stderr was invisible in a normal run: the hook silently failed open, against
the contract its own docstring states ("Exit 0 means every file was already
clean").
Judge the run by its report, not by its exit code
-------------------------------------------------
The discriminator cannot be "returncode != 0". clean_file.py deliberately
exits 1 for a *successful* clean that left residual signals behind, prints
valid JSON while doing so, and tests/test_json_exit_code.py pins that. So a
non-zero code is not by itself a failure. What actually separates the cases
is whether the child produced a parsable report: no report means it never
reached its write, and that file is still marked. rc == 2 keeps its meaning
(unrecognized format / oversized input) and still skips, unchanged.
Why exit 3 (EXIT_PARTIAL) and not 1 or 2
-----------------------------------------
1 is taken, and taking it again would be a lie: it means "I rewrote your
files, review the diff and re-stage", so reusing it for a crash sends the
developer looking for a diff that does not exist. 2 is usage/refusal — a bad
invocation — but the invocation was valid; it is also the code clean_file.py
returns for the skip we must keep reporting as 0, so overloading it invites
exactly the confusion this commit removes. 3 is EXIT_PARTIAL, already
defined in common.py as "partial scan, some files failed to scan; takes
precedence over actionable findings", and already used with that meaning by
audit_dir.py:162 and audit_website.py:555. A batch that could not process
some of its files is precisely a partial run, and the documented precedence
resolves the mixed case for free: a batch that cleaned one file and failed
on another exits 3, while still reporting both. pre-commit fails the hook on
any non-zero, so the code is for humans and direct/CI callers — which is the
reason to follow the convention the repo already has rather than invent one.
Reporting
---------
Failures get the summary block the sibling hooks use (headline, indented
path, indented reason) instead of a raw traceback behind a path prefix. Only
the last stderr line is quoted, which for an uncaught exception is the line
that names the cause; a child killed by a signal leaves no stderr, so the
exit status is reported instead.
The leftover .bak is named, not deleted
----------------------------------------
clean_file.py --in-place backs the original up (clean_file.py:100) before
the write that failed, so every failed run leaves a sidecar. This wrapper
does not remove it: it cannot tell its own leftover from a .bak the
developer already had staged, and deleting the wrong file from inside an
already-failing hook is the worse outcome. Nothing is lost by keeping it —
safe_write_bytes() replaces atomically or not at all, so the original is
intact and the sidecar is redundant rather than load-bearing. The report
names the path when it exists so the developer is not left wondering where
it came from. Not taking the backup until it is needed is a change to
clean_file.py's own --in-place ordering and belongs with that file.
Tests cover the crash, empty-stdout, malformed-JSON and killed-child paths,
the mixed cleaned-plus-failed precedence, an end-to-end symlink case that
needs no mocking (common.py:256 refuses to write through a symlink), and two
regressions: the intended rc == 2 skip still exits 0, and a residual-signal
clean is still reported as a clean, not a failure.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
c2patool exits non-zero both when an asset carries no manifest and when the
binary itself fails, and run_optional_tools only ever substring-matched the
output. A probe that died before main() produced has_manifest: False with no
other signal, so inspect reported has_c2pa: false -- the same answer it gives
for a genuinely clean asset.
Reproduced on Apple Silicon: service/Dockerfile pins a multi-arch base digest,
so an arm64 host builds an arm64 image while still installing the
x86_64-unknown-linux-gnu c2patool release (upstream publishes no linux-aarch64
build). c2patool dies with "rosetta error: failed to open elf at
/lib64/ld-linux-x86-64.so.2" and exit -5, and /capabilities kept reporting
c2patool: true because which() only finds the file.
Exposure is largest for PDF, where inspect_pdf documents exiftool and c2patool
as the more reliable detectors. For JPEG/PNG the native APP11 and PNG-chunk
scans still catch hard-bound C2PA, so there it costs corroboration rather than
the only signal.
- run_optional_tools marks a run conclusive only when it found a manifest or
said in so many words that there is none; anything else sets ok: False
- inspect_image and inspect_pdf surface that as an inconclusive note, worded
to land in the informational confidence bucket
- /capabilities probes each tool's version flag instead of trusting which()
Issues #1 and #3 fixed the opposite direction of this same function ("No claim
found" matching "claim"). This closes the false-negative side.
Co-authored-by: Italo Rodrigues <italo@pvwi.com.br>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* fix: strip three missed Default_Ignorable invisible carriers in Layer A
U+180F (Mongolian free variation selector-4, added in Unicode 14), U+3164
(Hangul filler), and U+FFA0 (halfwidth Hangul filler) are blank-rendering
Default_Ignorable code points, but their Mn/Lo categories meant the Cf
catch-all never saw them and they were absent from STRIP_CODEPOINTS. Both
inspect_text and clean_text therefore passed them through untouched, even
between plain ASCII.
Add them to the strip set and wire U+180F into the Mongolian-FVS handling
so it is stripped when floating but preserved after a Mongolian letter,
exactly like FVS1-3. Replace the 0x180B..0x180D range with the named
_MONGOLIAN_FVS set (clearer, and avoids the differently-handled Mongolian
vowel separator at U+180E). Applied to both the service engine and the
vendored lightweight-skill copy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HV9AE8QLqiXf7QnRe891EY
* feat: strip reserved Default_Ignorable code points in Layer A
U+2065, U+FFF0..U+FFF8, U+E0000, U+E0080..U+E00FF, and U+E01F0..U+E0FFF
are unassigned code points carrying Other_Default_Ignorable_Code_Point,
reserved for future default-ignorable characters. Conformant renderers
display them invisibly, normalisation preserves them, and the Cf
catch-all never sees them (they are category Cn), which made them ideal
covert carriers that both inspect_text and clean_text passed through
untouched.
Add them to the strip set as explicit ranges and report them under the
new "reserved_ignorable" inspect kind. Deliberately not a category-Cn
rule: unicodedata is pinned per Python build, so a Cn rule would destroy
characters assigned in newer Unicode versions. The table carries a
reminder to re-check the ranges on Unicode version bumps, since
assignment turns a strip entry into a potential preserve-in-context
case, exactly as happened when U+180F became Mongolian FVS4 in
Unicode 14.
Tests sweep all 3,739 code points through both the service engine and
the vendored lightweight-skill copy (clean and inspect CLIs), with a
boundary test pinning the assigned neighbours (U+2064, U+FFF9, U+E0001,
U+E0100) to their existing kinds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat: strip Unicode noncharacters in Layer A
The 66 noncharacters (U+FDD0..U+FDEF plus U+nFFFE/U+nFFFF at the end of
every plane) are permanently reserved for internal use and prohibited in
interchange text (TUS 23.7). They render as nothing or tofu, survive
normalisation and Python round-trips, and their category (Cn) meant the
Cf catch-all never saw them, so both inspect_text and clean_text passed
them through untouched: a ready-made covert channel.
Strip them and report them under the new "noncharacter" inspect kind.
Unlike the reserved Default_Ignorable ranges, noncharacters can never be
assigned, so this carries no future-Unicode risk. Applied to both the
service engine and the vendored lightweight-skill copy; tests sweep all
66 code points through both, and pin the assigned neighbours U+FDF0 and
U+FFFD (replacement character) as untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: preserve visible-layout format controls next to their own script
Egyptian hieroglyph quadrat controls (U+13430..U+1343F), Duployan
shorthand controls (U+1BCA0..U+1BCA3), and musical beam/tie/slur/phrase
controls (U+1D173..U+1D17A) are category Cf, so the catch-all stripped
them, yet they visibly govern how their script renders (quadrat
stacking, shorthand overlaps, beaming): removing them changes the
rendered text, contradicting the "cleaners preserve the document body"
invariant. Preserve them when adjacent to their own script, exactly
like the existing Mongolian/Khmer/Hangul handling; floating between
unrelated text they stay stripped and flagged, and --strip-emoji-glue
paranoid mode still strips them everywhere.
The vendored lightweight-skill engine needed next_input threaded
through its _decide to express "preserve a begin-control from the
character after it"; both engines now behave identically here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: preserve compatibility/halfwidth Hangul fillers in Hangul context
Review feedback on the Layer A cluster: U+3164 and U+FFA0 were stripped
unconditionally while the changelog claimed they were handled "like
U+115F/U+1160", which are preserved after a Hangul jamo. Make the
behaviour match the claim: both new fillers join _HANGUL_FILLERS and
_is_hangul_jamo now covers the compatibility (U+3131-U+318E) and
halfwidth (U+FFA1-U+FFDC) presentation forms, so each filler survives
after a letter of its own form and remains contraband when floating
between unrelated text. Changelog reworded to state the rule precisely.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
The core image installs libimage-exiftool-perl but not Archive::Zip;
without it exiftool warns "Install Archive::Zip to decode compressed
ZIP information" and misidentifies every Office/OpenDocument input
as a generic ZIP (File Type: ZIP, MIME application/zip), reading
none of the embedded metadata — the exiftool arm of /inspect is
blind for exactly the container formats the README advertises, and
its intended role as a second, independent check silently degrades
to nothing (#154).
Add libarchive-zip-perl (Debian's packaged Archive::Zip) to the apt
layer.
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* feat: add reproducible SynthID-text removal benchmark
bench_synthid_text.py orchestrates the existing Layer B machinery into a
controlled, shareable experiment: generate watermarked + unwatermarked
samples with the MarkLLM SynthID scheme, run removal variants (strength x
candidates) plus controls (no-removal, Layer-A-only, optional re-stamp),
and report clear rate, score suppression, quality, and cost (tokens,
wall time, USD) with a clears-per-MTok efficiency ratio.
Emits report.md / results.json / results.csv with the exact reproduction
command and pinned commits; optional Gemini official-detector tier when
WATERMARKS_GEMINI_API_KEY is set. Mock-based tests, no torch in CI.
* docs: add README section on running the SynthID-text benchmark
Explains what LLM performs the Layer B rewrite (an external model configured
via WATERMARKS_REWRITE_* env vars or --rewrite-* flags; MarkLLM's opt-1.3b is
only the watermark generator/detector) and how to run a benchmark with Ollama
or an OpenAI-compatible endpoint, plus the non-origin-model re-stamp caveat.
* fix: honor WATERMARKS_REWRITE_ALLOW_REMOTE in the SynthID-text benchmark
The --rewrite-allow-remote flag now defaults from the env var (matching
rewrite_text.py and the other WATERMARKS_REWRITE_* settings), so a
non-loopback rewrite endpoint works after sourcing .env without an extra
flag.
* fix: MarkLLM sparse checkout and deps for the SynthID harness
- setup_markllm.sh sparse-checkout omitted '/visualize/', which
watermark/base.py imports at module load — every scheme (incl. SynthID)
failed with 'No module named visualize' during generation/detection.
- requirements-markllm.txt omitted scikit-learn, imported by the SynthID
detector (watermark/synthid/detector_bayesian_torch.py).
Both broke the MarkLLM harness at runtime; the benchmark's sanity gate then
excluded every sample, producing empty per-variant results.
* fix: drop 4 GiB RLIMIT_AS on benchmark subprocesses
_run_cmd applied the common child RLIMIT_AS (default 4 GiB) via
subprocess_preexec_fn to every MarkLLM/rewrite child. torch needs a much
larger address space: CUDA init failed with 'out of memory' at
cudaGetDeviceCount and the 5.2 GB fp32 opt-1.3b could not load, so every
sample was excluded at generation. text_detectors.py already applies no
address-space cap to MarkLLM by default; the benchmark now matches.
* perf: keep MarkLLM resident via a serve worker (624 cold starts -> 1)
The benchmark spawned a fresh torch + opt-1.3b process per operation
(~60-90s each); a full run needs ~624 of them. detect_text_watermark.py
gains a 'serve' mode (JSON-lines over stdin/stdout, ready handshake) that
loads the model once; bench_synthid_text.py uses it via MarkLLMWorker with
automatic fallback to one-shot subprocesses (--no-worker to force).
Turns ~8h runs into ~40-60min.
* perf: skip per-candidate Gemini detections in rewrite subprocess
* feat: run the SynthID-text benchmark from the wr-markllm compose service
- Dockerfile.markllm: add '/visualize/' to the sparse checkout (same fix as
setup_markllm.sh) and COPY the benchmark + rewrite scripts (stdlib-only).
- compose.yaml: wr-markllm gets the WATERMARKS_REWRITE_* and
WATERMARKS_GEMINI_* env wiring, a bench-out volume for --out-dir, and a
read-only mount of the bundled corpus (build context is service/, so the
corpus cannot be COPY'd).
- docs: docker compose run example.
Note: the image ships CPU torch by design, so the container path is for
portability/CI; GPU runs use the host setup_markllm.sh venv.
* feat: per-sample progress logging in the benchmark
The persistent worker returns samples in-memory, so nothing is written
until the end of a run — runs looked stuck. eprint a [gen i/N] line per
generated sample and a [removal] summary per sample.
* chore: migrate Gemini config to gemini-3.6-flash; document SynthID-text retirement
Google retired SynthID text watermarking on the Generative Language API
(Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK
is rejected on current 3.x models (confirmed by Google AI staff). Migrate
the default detection model to gemini-3.6-flash, document the retirement
in vendor-notes.md and the benchmark report caveat, and keep the detector
seam fail-soft until a vendor endpoint (e.g. Vertex AI) returns.
* feat: remove gemini-synthid-text detector (Google retired text watermarking)
Google removed SynthID text watermarking from the Generative Language API
(Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK
is rejected on current 3.x models, so the vendor detector had nothing to
detect. Remove GeminiSynthIDTextDetector and its wiring:
- text_detectors.py: drop the Gemini class, HTTP helpers, and constants;
keep MarkLLM + Claude seams (registry now markllm + claude-text).
- server.py / rewrite_text.py: per-candidate detection now triggers on
--markllm-scheme only.
- bench_synthid_text.py: remove the Gemini tier (before/after, report
table, --no-gemini flag); report caveat notes the retirement.
- configs/docs: drop WATERMARKS_GEMINI_* from .env.example / compose /
README / SKILL.md / vendor-notes.md; keep the retirement note.
- tests: gemini tests removed or converted to MarkLLM (mocked subprocess).
- Dockerfile.markllm: parameterize BASE_IMAGE + TORCH_INDEX_URL so a GPU/
arm64 image can be built (used for the --gpus all benchmark run).
* feat: iterative detection-guided Layer B rewriting (default 3 attempts)
Layer B (rewrite_text.py) now rewrites iteratively and stops as soon as an
attempt passes watermark evaluation:
- --candidates defaults to 3 (WATERMARKS_REWRITE_CANDIDATES); each attempt is
one rewrite + one evaluation, and the loop exits on the first attempt the
evaluator reports as not watermarked.
- Evaluator priority: MarkLLM same-config detection (when --markllm-scheme is
passed) > bigram-Jaccard lexical divergence (fallback; no verdict, all
attempts generated, most diverged selected). A vendor-detector seam is
reserved ahead of MarkLLM for a future SynthID-text endpoint (Google retired
text watermarking on its API in Aug 2026).
- Best-effort fallback when the max is exhausted: the lowest-score attempt is
returned with a note; detector errors are fail-soft and never fail the
rewrite.
- --json-stats now reports evaluator / attempts_made / passed and per-attempt
candidate_scores records (passed, evaluation); markllm before/after/cleared
is unchanged and the selected attempt's verdict is reused (no duplicate
MarkLLM detection).
Benchmark (bench_synthid_text.py):
- --variants default becomes paraphrase:3 (candidates = max attempts).
- Rows/report/CSV carry attempts per document (mean_attempts, att column;
attempts / evaluator / passed columns).
Tests, README, docs/synthid-text-benchmark.md and .env.example updated; 490
tests pass, ruff clean.
* feat: split rewrite attempts into --candidates x --max-loops (defaults 1 x 1)
Follow-up to the iterative Layer B rewrite: separate "variants per round"
from "evaluation rounds", so the retry loop is explicit and defaults stay
conservative.
- rewrite_text.py: --candidates (WATERMARKS_REWRITE_CANDIDATES) is now the
number of variants generated per loop iteration (default 1); new
--max-loops (WATERMARKS_REWRITE_LOOPS) caps the evaluation rounds (default
1) -- each round generates --candidates variants and stops as soon as one
passes, so raising --max-loops retries new variants until an evaluation
passes. Stats now report max_loops and per-attempt records carry the loop
index.
- bench_synthid_text.py: new --rewrite-loops flag (default 1) passed through
to --max-loops.
- README / docs / .env.example updated; tests cover the 1x1 defaults, loop
retry until pass, and cross-loop exhaustion.
* feat: MarkLLM serve worker over loopback TCP (WATERMARKS_MARKLLM_PORT)
detect_text_watermark.py serve can now also listen on a loopback TCP port,
and MarkLLMTextDetector reuses a resident worker when
WATERMARKS_MARKLLM_PORT is set (falls back to a one-shot subprocess when the
worker is unreachable). This avoids a ~20s torch+model cold start per detect
for callers that run a worker out-of-band.
Tests: loopback worker protocol + detector worker-port routing (mock-based).
* ci: add macOS runner to the test matrix
* feat: add reproducible SynthID-text removal benchmark
bench_synthid_text.py orchestrates the existing Layer B machinery into a
controlled, shareable experiment: generate watermarked + unwatermarked
samples with the MarkLLM SynthID scheme, run removal variants (strength x
candidates) plus controls (no-removal, Layer-A-only, optional re-stamp),
and report clear rate, score suppression, quality, and cost (tokens,
wall time, USD) with a clears-per-MTok efficiency ratio.
Emits report.md / results.json / results.csv with the exact reproduction
command and pinned commits; optional Gemini official-detector tier when
WATERMARKS_GEMINI_API_KEY is set. Mock-based tests, no torch in CI.
* docs: add README section on running the SynthID-text benchmark
Explains what LLM performs the Layer B rewrite (an external model configured
via WATERMARKS_REWRITE_* env vars or --rewrite-* flags; MarkLLM's opt-1.3b is
only the watermark generator/detector) and how to run a benchmark with Ollama
or an OpenAI-compatible endpoint, plus the non-origin-model re-stamp caveat.
* fix: honor WATERMARKS_REWRITE_ALLOW_REMOTE in the SynthID-text benchmark
The --rewrite-allow-remote flag now defaults from the env var (matching
rewrite_text.py and the other WATERMARKS_REWRITE_* settings), so a
non-loopback rewrite endpoint works after sourcing .env without an extra
flag.
* fix: MarkLLM sparse checkout and deps for the SynthID harness
- setup_markllm.sh sparse-checkout omitted '/visualize/', which
watermark/base.py imports at module load — every scheme (incl. SynthID)
failed with 'No module named visualize' during generation/detection.
- requirements-markllm.txt omitted scikit-learn, imported by the SynthID
detector (watermark/synthid/detector_bayesian_torch.py).
Both broke the MarkLLM harness at runtime; the benchmark's sanity gate then
excluded every sample, producing empty per-variant results.
* fix: drop 4 GiB RLIMIT_AS on benchmark subprocesses
_run_cmd applied the common child RLIMIT_AS (default 4 GiB) via
subprocess_preexec_fn to every MarkLLM/rewrite child. torch needs a much
larger address space: CUDA init failed with 'out of memory' at
cudaGetDeviceCount and the 5.2 GB fp32 opt-1.3b could not load, so every
sample was excluded at generation. text_detectors.py already applies no
address-space cap to MarkLLM by default; the benchmark now matches.
* perf: keep MarkLLM resident via a serve worker (624 cold starts -> 1)
The benchmark spawned a fresh torch + opt-1.3b process per operation
(~60-90s each); a full run needs ~624 of them. detect_text_watermark.py
gains a 'serve' mode (JSON-lines over stdin/stdout, ready handshake) that
loads the model once; bench_synthid_text.py uses it via MarkLLMWorker with
automatic fallback to one-shot subprocesses (--no-worker to force).
Turns ~8h runs into ~40-60min.
* perf: skip per-candidate Gemini detections in rewrite subprocess
* feat: run the SynthID-text benchmark from the wr-markllm compose service
- Dockerfile.markllm: add '/visualize/' to the sparse checkout (same fix as
setup_markllm.sh) and COPY the benchmark + rewrite scripts (stdlib-only).
- compose.yaml: wr-markllm gets the WATERMARKS_REWRITE_* and
WATERMARKS_GEMINI_* env wiring, a bench-out volume for --out-dir, and a
read-only mount of the bundled corpus (build context is service/, so the
corpus cannot be COPY'd).
- docs: docker compose run example.
Note: the image ships CPU torch by design, so the container path is for
portability/CI; GPU runs use the host setup_markllm.sh venv.
* feat: per-sample progress logging in the benchmark
The persistent worker returns samples in-memory, so nothing is written
until the end of a run — runs looked stuck. eprint a [gen i/N] line per
generated sample and a [removal] summary per sample.
* chore: migrate Gemini config to gemini-3.6-flash; document SynthID-text retirement
Google retired SynthID text watermarking on the Generative Language API
(Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK
is rejected on current 3.x models (confirmed by Google AI staff). Migrate
the default detection model to gemini-3.6-flash, document the retirement
in vendor-notes.md and the benchmark report caveat, and keep the detector
seam fail-soft until a vendor endpoint (e.g. Vertex AI) returns.
* feat: remove gemini-synthid-text detector (Google retired text watermarking)
Google removed SynthID text watermarking from the Generative Language API
(Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK
is rejected on current 3.x models, so the vendor detector had nothing to
detect. Remove GeminiSynthIDTextDetector and its wiring:
- text_detectors.py: drop the Gemini class, HTTP helpers, and constants;
keep MarkLLM + Claude seams (registry now markllm + claude-text).
- server.py / rewrite_text.py: per-candidate detection now triggers on
--markllm-scheme only.
- bench_synthid_text.py: remove the Gemini tier (before/after, report
table, --no-gemini flag); report caveat notes the retirement.
- configs/docs: drop WATERMARKS_GEMINI_* from .env.example / compose /
README / SKILL.md / vendor-notes.md; keep the retirement note.
- tests: gemini tests removed or converted to MarkLLM (mocked subprocess).
- Dockerfile.markllm: parameterize BASE_IMAGE + TORCH_INDEX_URL so a GPU/
arm64 image can be built (used for the --gpus all benchmark run).
* fix: harden notes aggregation against non-string notes
A run completed all samples but crashed at the final aggregate step with
'cannot use list as a set element' when a row's notes contained a
non-string value. Filter notes to strings (aggregate + CSV) and add a
regression test.
* perf: let the rewrite subprocess reuse the resident MarkLLM worker
The rewrite subprocess (rewrite_text.py) ran its own before/after MarkLLM
detects, each a ~20s torch+model cold start (~12 per sample = ~5min of the
~6min/sample runtime). Now:
- detect_text_watermark.py serve gains --port N: a loopback TCP JSON-lines
listener (default -1 = off) sharing the resident model, with a lock so
stdin and socket requests never run the model concurrently.
- text_detectors.MarkLLMTextDetector checks WATERMARKS_MARKLLM_PORT and
does a fast loopback detect when a worker is up, falling back to the
one-shot subprocess otherwise.
- The benchmark worker publishes its port via that env var, so the rewrite
subprocess inherits it and its detects hit the resident model.
Turns ~6 min/sample into ~1-2 min; a full run drops from ~2h to ~40-50min.
Tests: loopback-client + fallback + env-publish coverage.
* chore: add benchmark-smoke.sh / benchmark-full.sh wrappers
Simple host wrappers: source .env, default MARKLLM_DIR to ~/MarkLLM, use a
repo-local HF cache by default, and run bench_synthid_text.py with a quick
(2 docs, 1 seed, paraphrase:1) or full (8 docs x 3 seeds, three variants,
re-stamp control) configuration. OUT_DIR overrides the output location.
* fix(server): preserve image format extensions in /clean and use safe writes in av_meta (#148)
* style: apply ruff formatting to server.py and test_http_server.py
---------
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* fix: decode XML entities before Layer A scrub in OOXML/ODF
Watermark carriers encoded as XML character references (e.g.
​) reached Layer A as nine ASCII characters and survived
cleaning untouched, because the XML parser in Word/Writer decodes
the entity back into the invisible carrier after the cleaner ran
(#129).
A shared _decode_xml_entities resolves exactly what a conforming
XML parser resolves (numeric references and the five predefined
entities) and leaves everything else literal. The DOCX/XLSX/PPTX
text-run scrubbers decode before clean_text and re-encode on the
way out; the ODT scrubber splits paragraph content into markup and
text segments and only round-trips the text segments, because a
whole-paragraph decode/re-encode would entity-escape the nested
text:span/text:tab markup. Untouched runs keep their original
bytes.
* fix: satisfy ruff lint and format checks
---------
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
COPY preserves source file and directory modes, so a build host with a
restrictive umask (e.g. 027) landed the scripts as 0640 root:root and
the /app/scripts directory as 0750 -- unreadable and untraversable for
the unprivileged runtime user (uid 10001). The image built fine and
failed only at container start, making it easy to miss (#131).
Normalize after the copy with chmod -R a+rX: world-readable everywhere,
execute bit for directories only, intentionally-executable scripts keep
their bit -- deterministic regardless of the builder's umask.
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* fix: report unmeasurable burstiness instead of scoring it as max LLM-like
compute_burstiness returned cv=0.0 for an empty (or single-sentence)
list, which the tiering read as perfectly uniform prose — the strongest
LLM-likeness signal — so any file whose body yielded no parseable
sentences (e.g. entirely wrapped in a code fence) scored falsely high
while word_count stayed high enough to defeat the small-sample dampener
(#132). On a real export this hit 159 of 162 files as pure artifact.
compute_burstiness now returns None for the CV when it cannot be
measured; the scoring site drops the burstiness component and
renormalizes the composite over AI-phrase density and lexical
diversity, with an explicit note. burstiness_cv is null in JSON for
unmeasurable cases, the CLI prints n/a instead of crashing, and the
uniform-cadence finding guards on cv not being None.
* style: satisfy ruff import sorting and formatting
---------
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Co-authored-by: Guillaume Meyer <guillaumemeyer@users.noreply.github.com>
* fix: parse single-quoted attributes in relationship/manifest pruning
XML allows attribute values in single or double quotes. The Target
extraction in _prune_dangling_relationships and the full-path
extraction in _prune_odt_manifest_entries matched only double quotes,
so a valid single-quoted Relationship parsed as an empty target,
resolved to the package base directory, and was deleted outright —
corrupting DOCX/XLSX/PPTX and ODT packages produced by tools that
emit single quotes (#130).
Both extractions now use a backreference pattern
((["'])(.*?)\1) so either quote style resolves the real target, and
pruning still only drops relationships whose parts were genuinely
removed. Regression tests cover the single-quoted kept relationship
(the reported corruption) and the double-quoted dropped part.
* fix: satisfy ruff lint and format checks
---------
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
clean_svg/clean_odt stripped metadata with lazy dot-matches-all regexes; on many unclosed opening tags the engine rescans to end-of-input from every candidate start (O(n^2)), and the GIL stalls the whole single-process service. A ~1.4 KB ODT request pinned a core for ~99 s. Replace every lazy .*?</close> block scan in container_meta.py (SVG metadata/xmpmeta/comments, ODT meta:generator/dc:creator, HTML JSON-LD, OOXML/ODF text-run scrubs, docProps fields, EPUB OPF meta/dc, PDF xpacket/stream) with a linear scan pairing opening tags with a forward pointer over closing tags - identical match semantics, O(n). Add regression tests asserting the advisory PoC completes in <5 s and that stripping behavior is unchanged.
The file-cleaners layer covered 15 formats -- all image, document, or
text -- and zero audio/video. That gap gets more expensive every month:
Sora, Veo, ElevenLabs, and Suno all embed provenance through the same
mechanisms image generators do, just in different containers.
New av_meta.py adds inspect/clean for:
- MP4/MOV/M4A/M4V: top-level C2PA (jumb/c2pa box) and XMP (uuid box)
detection/stripping reuse inspect_isobmff()/strip_isobmff() from
image_meta.py unchanged -- that's exactly the mechanism the C2PA spec
defines for ISOBMFF-family containers, already proven for AVIF/HEIC.
moov/udta (where generator/tool tags live) is handled separately since
it's MP4-specific.
- WAV: RIFF LIST INFO chunk + embedded id3 chunk.
- MP3: ID3v2 frames, per-frame for v2.3/v2.4, whole-tag fallback for
v2.2 (3-byte frame IDs are detected but not decomposed, so a partial
rewrite is never attempted there).
Every box/chunk/frame is either kept byte-identical or dropped whole --
nothing does a partial in-place rewrite of a payload, so a container can
never come out semantically mangled. Default strip_all_metadata=True
matches this project's existing default (privacy-first: drop everything,
--keep-non-ai-metadata narrows to only AI-flagged content), same as the
image cleaners.
Wired through the full dispatch stack so the feature isn't a half
integration: format_dispatch.py (new "av" Kind), inspect_file.py /
clean_file.py (--as av), audit_lib.py (so audit_dir.py's CI/SARIF path
and the pre-commit hooks from #135 both cover audio/video too), and
server.py (HTTP /inspect and /clean).
Closes#134
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
CI gating for AI provenance marks already exists (audit_dir.py's -j
concurrency + SARIF export from #101), but that only runs after a
marked file has already been committed and pushed. Catch it at commit
time instead, using git's own hook point.
Adds two hooks via .pre-commit-hooks.yaml:
- watermarks-remover-check: fails the commit and lists findings when
staged files carry AI/C2PA marks. Wraps audit_lib.scan_file() /
is_actionable() -- the exact per-file logic audit_dir.py already
uses for CI, so the pre-commit gate and the CI gate agree on what
counts as actionable.
- watermarks-remover-clean (opt-in): rewrites staged files in place by
shelling out to clean_file.py --in-place per file (no duplicated
cleaning logic), then exits 1 so the developer reviews the diff and
re-stages -- the same convention as auto-fixing hooks like ruff --fix.
.pre-commit-hooks.yaml needed an explicit allow-rule in the deny-by-
default .gitignore, same as every other root-level config file already
listed there.
Closes#135
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Directory-scale cleaning already exists in the CLI (audit_dir.py, -j
concurrency, SARIF export from #101), but the HTTP service handled one
file per request. Any web app or CI step talking to the service over
HTTP instead of the CLI paid N full round trips to clean N files.
Extract the single-file /inspect and /clean logic into _inspect_payload
and _clean_payload so both the existing single-file endpoints and the
new batch endpoints run the identical pipeline — no duplicated cleaning
logic. A malformed entry in a batch (bad base64, unknown option,
unrecognized format) surfaces as that entry's "ok": false with an
"error" string instead of aborting the rest of the batch.
Capped at WATERMARKS_MAX_BATCH_FILES per request (default 50) as
defense-in-depth against a request packing many tiny files into one
call; the existing MAX_BODY_BYTES envelope cap already bounds total
payload size the same as a single-file request.
/openapi.json picks up both routes automatically since the spec is
generated from the route table.
Closes#136
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* feat: add vendor text-watermark detection and SynthID image scorer sidecar
Adds Layer B watermark detection as a first-class service capability:
- text_detectors.py: a registry of text-watermark detectors behind one
interface — Google's official SynthID-text detector via the Gemini API
(taskType DETECT_TEXT_WATERMARK), a Claude placeholder ready for
Anthropic's announced detection API, and the MarkLLM research harness
(KGW / SynthID, same-config-only). Fail-soft: unconfigured or errored
detectors never block cleaning.
- server.py: new POST /detect endpoint, detect_before / detect_after options
on /clean (before/after scoring for text and images), an opt-in
/inspect "detect" flag, and /capabilities gains text_detectors and
scorers.synthid_http.
- synthid_score_server.py: a stdlib HTTP sidecar for the reverse-SynthID
scorer, so the published core image never bundles the non-commercial
upstream code; wired via WATERMARKS_SYNTHID_SCORER_URL.
- score_synthid.py: extract score_file() so the CLI and the sidecar share
one implementation.
- compose.yaml / Dockerfile.synthid / .env.example: wr-synthid-score sidecar
service and env wiring.
- README + skill docs, plus tests for the detectors, the /detect endpoint,
and the image sidecar.
* feat: per-candidate watermark detection for Layer B rewrite candidates
When --candidates N (N > 1) is combined with --markllm-scheme or
WATERMARKS_GEMINI_API_KEY, run every configured text detector from the
text_detectors.py registry on each candidate and report per-candidate
measurements in --json-stats as candidate_scores entries carrying
lexical_divergence, selection_score, selected, and per-detector reports
(is_watermarked, score, threshold where the detector provides one).
Candidate selection stays purely lexical; the detections are observability
for correlating lexical divergence with watermark removal (issue #106).
Converges rewrite_text.py onto the shared detector registry:
- MarkLLMTextDetector gains constructor overrides (scheme, upstream_dir,
model, timeout) plus the checkout-venv interpreter preference and the
WATERMARKS_MARKLLM_RLIMIT_AS preexec guard ported from rewrite_text.py;
the old _markllm_detect / _venv_python / _markllm_preexec helpers are gone.
- run_all_text_detectors() accepts an injected MarkLLM instance and an
include_markllm switch so CLI flag gating stays intact.
- before/after/cleared semantics unchanged; detection remains fail-soft.
* docs: pin Watermarks in the Sand reference to arXiv v5
* fix: mark only one rewrite candidate as selected (#110)
---------
Co-authored-by: Zhenxin Ai <142008897+ai-kunkun@users.noreply.github.com>
* fix: rewrite ODT/EPUB manifests and measure real zip bytes (#122)
Two container correctness/security fixes from issue #122:
- clean_odt dropped marker-bearing parts while leaving their entries in
META-INF/manifest.xml, so readers flagged the package as damaged. It is
now two-pass: compute the dropped set, then rewrite the manifest
attribute-order-independently, and write each part exactly once. The same
bug class in clean_epub (dropped parts left in the OPF manifest, plus
dangling spine itemrefs) gets the same two-pass treatment.
- The zip budget trusted ZipInfo.file_size from the archive's own central
directory, so a crafted DOCX/ODT could declare a tiny size and still
expand via zf.read. Budgets are now charged on actual decompressed bytes
via _read_zip_member (streaming, cap enforced mid-read), with the declared
size kept only as a fast-path pre-reject.
* fix: classify unrecognized bytes as "unknown", not text (#122)
Two classification defects from issue #122:
- format_dispatch.classify_bytes fell back to "text" for any unrecognized
file, so a binary with valid UTF-8 runs could be decoded and written back
mangled (corrupted with --in-place) in clean_file auto mode. Unrecognized
bytes now classify as "unknown"; clean_file refuses them in auto mode
(exit 2, no write, router advice) and --as text / --force-text are the
explicit opt-ins. inspect_file reports kind "unknown" (exit 0), audit_lib
records a non-actionable item, and the HTTP server answers /inspect with
kind "unknown" but rejects /clean of unknown formats (400).
- classify(path) read the whole file to sniff a header, and only a full read
could detect zip containers. It now routes known extensions without
reading, sniffs a 4096-byte header once for images and prefix-based
containers, and reads the whole file only when the header is a zip local
header (PK), where the container signature lives in the central directory.
* feat: distinct exit code for partial audits (#122)
audit_dir and audit_website reported success (0) even when some files or
URLs could not be scanned; the exit status was computed only over the items
that succeeded. A scan that is missing items is not a clean scan.
- common.EXIT_PARTIAL = 3, with precedence: partial (3) > actionable (1)
> clean (0) — an incomplete audit is the more important CI signal.
- audit_dir returns 3 when any file was skipped/failed; audit_website
returns 3 when any URL failed to fetch or inspect. Both are independent
of the output format (human/json/sarif already share one return).
* fix: verify the pinned upstream ref on existing checkouts (#122)
setup_ctrlregen.sh/setup_synthid.sh (and their .ps1 twins) only verified
the pinned commit in the fresh-clone branch; an existing checkout at an
unknown or drifted revision was silently reused, defeating the commit pin.
All four scripts now check HEAD against the pinned ref in the
existing-checkout branch too, and repair by fetch + detach checkout
(re-applying the sparse-checkout set), failing hard if the ref cannot be
reached or the re-pin does not land on it.
* docs: unknown-format behavior, audit exit codes, backend isolation (#122)
- README: clean_file no longer auto-cleans unrecognized formats (--as text
/ --force-text are the opt-ins), and the CtrlRegen bootstrap documents the
isolation expectation for its research-era dependency pins plus the new
re-pin check on existing checkouts.
- SKILL.md: audit exit codes (0/1/2/3, partial=3) and a note that /clean
requires a name with a known extension.
- audit_website: document why stdlib ElementTree is used (stdlib-first) and
that defusedxml is the fallback if that policy changes (DTD rejection stays).
- requirements-ctrlregen.txt: advisory/isolation note for the pinned research
dependencies.
* test: ODT manifest and EPUB OPF dangling-ref regressions (#122)
- clean_odt: dropped marker-bearing parts remove their META-INF/manifest.xml
file-entry (attribute-order-independent), exactly one manifest entry, root
and surviving entries kept, and the manifest is byte-identical when nothing
is dropped.
- clean_epub: dropped non-content parts lose their <item> entry in the OPF
manifest, so the book no longer references removed members.
Three independent failure modes from #117:
- $ErrorActionPreference 'Stop' + 2>$null on a native command aborts the
script on torch's harmless stderr warnings (e.g. "Failed to initialize
NumPy" when torch is installed before numpy). Run the probes through a
new Invoke-NativeQuiet helper that lowers EAP to 'Continue' for the
block and restores it afterwards.
- The wheel index tag was derived from the driver's CUDA version, e.g.
cu131 for a 13.1 driver, which does not exist (HTTP 403) and silently
fell back to the default index, i.e. the CPU build on Windows. Probe
the published indices and pick the highest one <= driver that answers
HTTP 200; cu126 is still forced below compute capability 7.5.
- Installing torch alone let requirements-ctrlregen.txt resolve torchvision
from PyPI, and torchvision pins an exact torch, so pip replaced the +cu
build with a +cpu one while the script still exited 0. Install torch AND
torchvision together from the chosen index, and verify after the
requirements install that torch.cuda.is_available() is true - if a GPU
was detected but torch ends up CPU-only, warn loudly and exit non-zero.
Also add a CI step (windows-latest, pwsh) that parses the setup .ps1
scripts and asserts the post-install CUDA verification survives.
Fixes#117
* Handle malformed DOCX/ODT beyond BadZipFile in container inspectors
A corrupt, truncated, encrypted, or unsupported-compression container
raises more than BadZipFile (NotImplementedError, RuntimeError, EOFError,
OSError, ValueError, zlib.error), so inspect_docx / inspect_odt and
detect_container_format now catch the whole family and degrade to a clear
finding.
The zip-bomb rejection in _check_zip_budget now raises a dedicated
ZipBudgetExceeded so it keeps propagating out of the inspectors, matching
clean_docx / clean_odt, instead of being reported as an unparseable zip.
* fix: satisfy ruff lint and format checks
---------
Co-authored-by: eeshsaxena <chinmaymd72@gmail.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
`base64 -w0 FILE` is a GNU coreutils form. On macOS/BSD it fails with
`base64: invalid argument FILE` and prints usage, so every documented
`/inspect` and `/clean` example is unrunnable on a Mac host.
This matters more than a normal docs typo because SKILL.md instructs the
agent to call the service via these exact commands and explicitly forbids
falling back to local cleaning when the service call fails. A macOS agent
therefore stops at its first request instead of degrading gracefully.
Replace all 7 occurrences (2 in README.md, 5 in SKILL.md) with the POSIX
form, which reads from stdin and unwraps any line breaks:
base64 < FILE | tr -d '\n'
Verified byte-identical output on both implementations:
- BSD/macOS base64 (macOS 15, Darwin 24.6.0)
- GNU coreutils base64 (`gbase64`)
Also verified the rewritten commands verbatim against a live
`server.py` on 127.0.0.1:8765 — `/inspect` and `/clean` both return
`ok: true` and correctly strip U+200B / U+00AD.
Docs-only; no code paths changed, so no unit tests added.
`python3 -m pytest -q` and `ruff check`/`format --check` pass.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* docs: add Windows auto-start guide via Task Scheduler
* docs: fix markdown escaping, remove admin requirement, use path placeholder, link from README
* docs: add watermarks-remover-web to Ecosystem (#77)
* docs: note AVIF and HEIC in the watermarks-remover-web entry
The client now ports the ISOBMFF paths from image_meta.py as well, so the
entry's format list was out of date.
* docs: rename the web UI entry to unmark-web and keep it higher-level
Renames the project per the maintainer's request, and drops the
references to internal module names, the client's proxy script and the
format list, which were the parts most likely to drift.
---------
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Add stdlib-only detection, inspection, and cleaning for four more formats.
- BMP: locate the pixel payload via the DIB header and strip trailing
non-image metadata (the only place non-standard BMP metadata can live),
rewriting the file-size field.
- GIF: drop comment and XMP application extensions while preserving
NETSCAPE2.0 looping, ICC, graphic-control, and image blocks.
- TIFF (classic + BigTIFF): walk IFD chains and drop XMP/EXIF/GPS/IPTC/
Photoshop/MakerNote tags, zeroing orphaned payloads while keeping
strip/tile offsets valid.
- EPUB: scrub OPF package metadata and XHTML meta/JSON-LD, clean embedded
raster/SVG media, apply Layer A to XHTML body text, and pass OCF-encrypted
parts through untouched.
All four route through format_dispatch, so the unified CLIs, the HTTP
service, and the audits pick them up automatically.
* fix: pin vendored Cursor-skill text engine to the service copy
The engine vendored into skills/clean-user-facing-text/ had silently
drifted behind service/scripts/text_unicode.py: it still blanket-
stripped legitimate RTL directional marks and isolates (corrupting
mixed RTL/LTR prose) and stripped emoji variation selectors after
arrow and symbol bases, because it never received the preservable-bidi
and emoji-base updates. Nothing in the suite compared the two copies.
Replace the vendored engine with an exact copy of the service one and
add a byte-equality test so any future engine change must land in both
files in the same commit, plus a behavioural regression test for the
RTL and arrow cases through the vendored CLI. The CLI wrappers remain
deliberately different (text-only skill: no stylometry, no
--strip-bidi).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: re-pin vendored engine after merging main and correct bidi comment
Merging main into the PR branch kept main's ruff-formatted service
engine (#103) but the PR's vendored copy pinned in 519c6db, so the two
drifted apart again and test_vendored_text_unicode_is_identical_to_service_engine
failed. Re-sync the vendored copy to the current service engine so the
byte-for-byte pin holds.
Also correct the _PRESERVABLE_BIDI_CPS comment: paired LRE/RLE embeddings
are preserved via _valid_bidi_embedding_indices, so only overrides and
unpaired embeddings remain destructive by default.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
The bullet 'Best-effort: prose was rewritten...' uses a full sentence with
passive verb while adjacent bullets use noun phrases. Removed 'was' to
match the parallel structure of the other bullets.
Audit metadata:
- Category: grammar
- Severity: minor
- File: skills/clean-user-facing-text/SKILL.md, line 61
- Fingerprint: a9c80f5f5afbfbe7
Co-authored-by: pi <pi@m2.local>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
resolve_device() auto-selects 'mps' whenever torch reports it available.
Both schemes this harness can load build a device-bound RNG in their
constructor:
KGW.__init__ -> torch.Generator(device=config.device) (kgw.py)
SynthID.__init__ -> torch.Generator(device=config.device) (synthid.py)
torch.Generator supports only the 'cpu' and 'cuda' device types, so on
Apple Silicon a resolved 'mps' device raises
RuntimeError: Device type MPS is not supported for torch.Generator() api
and both the `detect` and `watermark` subcommands crash out of the box on
any Mac where torch reports MPS available.
Drop the mps branch from the auto path so `--device auto` resolves to cpu
on CUDA-less machines. `--device mps` is still honored as an explicit
override (resolve_device returns any non-'auto' value unchanged), so this
changes only the automatic default, not the ability to opt in.
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Introduce Ruff (pinned at 0.16.3) as the project linter + formatter and
enforce it in CI:
- requirements-dev.txt: pin ruff==0.16.3 (exact pins, no drift)
- ruff.toml: line-length 100, target py312; rule set E/F/W/I/UP/B/SIM/RUF/PLW/S
with deliberate ignores (E501 for content strings, S603 for safe_arg
subprocess calls, S101 asserts in tests) and per-file test ignores
- Makefile: add lint / format / lint-fix targets
- .github/workflows/ci.yml: add lint job (ruff check + format --check)
- .gitignore: whitelist ruff.toml
Also fix every finding the new gate surfaced so CI is green:
- 109+ auto-fixes from ruff --fix (import sorting, simplifications,
unused vars, re.I aliases, etc.)
- explicit check=False on all subprocess.run calls (PLW1510)
- harden sitemap XML parsing: reject DTD/entity declarations (S314)
- replace hardcoded /tmp paths in tests with tmp_path (S108)
- narrow/annotate intentional bare excepts (S110/S112), bind loop vars
in closures (B023), raise ... from None (B904), strict= for zip (B905)
- ruff format applied across service/ and tests/
Verified: ruff check + ruff format --check pass; 287 tests pass, 1 skip.
