* fix: strip three missed Default_Ignorable invisible carriers in Layer A
U+180F (Mongolian free variation selector-4, added in Unicode 14), U+3164
(Hangul filler), and U+FFA0 (halfwidth Hangul filler) are blank-rendering
Default_Ignorable code points, but their Mn/Lo categories meant the Cf
catch-all never saw them and they were absent from STRIP_CODEPOINTS. Both
inspect_text and clean_text therefore passed them through untouched, even
between plain ASCII.
Add them to the strip set and wire U+180F into the Mongolian-FVS handling
so it is stripped when floating but preserved after a Mongolian letter,
exactly like FVS1-3. Replace the 0x180B..0x180D range with the named
_MONGOLIAN_FVS set (clearer, and avoids the differently-handled Mongolian
vowel separator at U+180E). Applied to both the service engine and the
vendored lightweight-skill copy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HV9AE8QLqiXf7QnRe891EY
* feat: strip reserved Default_Ignorable code points in Layer A
U+2065, U+FFF0..U+FFF8, U+E0000, U+E0080..U+E00FF, and U+E01F0..U+E0FFF
are unassigned code points carrying Other_Default_Ignorable_Code_Point,
reserved for future default-ignorable characters. Conformant renderers
display them invisibly, normalisation preserves them, and the Cf
catch-all never sees them (they are category Cn), which made them ideal
covert carriers that both inspect_text and clean_text passed through
untouched.
Add them to the strip set as explicit ranges and report them under the
new "reserved_ignorable" inspect kind. Deliberately not a category-Cn
rule: unicodedata is pinned per Python build, so a Cn rule would destroy
characters assigned in newer Unicode versions. The table carries a
reminder to re-check the ranges on Unicode version bumps, since
assignment turns a strip entry into a potential preserve-in-context
case, exactly as happened when U+180F became Mongolian FVS4 in
Unicode 14.
Tests sweep all 3,739 code points through both the service engine and
the vendored lightweight-skill copy (clean and inspect CLIs), with a
boundary test pinning the assigned neighbours (U+2064, U+FFF9, U+E0001,
U+E0100) to their existing kinds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat: strip Unicode noncharacters in Layer A
The 66 noncharacters (U+FDD0..U+FDEF plus U+nFFFE/U+nFFFF at the end of
every plane) are permanently reserved for internal use and prohibited in
interchange text (TUS 23.7). They render as nothing or tofu, survive
normalisation and Python round-trips, and their category (Cn) meant the
Cf catch-all never saw them, so both inspect_text and clean_text passed
them through untouched: a ready-made covert channel.
Strip them and report them under the new "noncharacter" inspect kind.
Unlike the reserved Default_Ignorable ranges, noncharacters can never be
assigned, so this carries no future-Unicode risk. Applied to both the
service engine and the vendored lightweight-skill copy; tests sweep all
66 code points through both, and pin the assigned neighbours U+FDF0 and
U+FFFD (replacement character) as untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: preserve visible-layout format controls next to their own script
Egyptian hieroglyph quadrat controls (U+13430..U+1343F), Duployan
shorthand controls (U+1BCA0..U+1BCA3), and musical beam/tie/slur/phrase
controls (U+1D173..U+1D17A) are category Cf, so the catch-all stripped
them, yet they visibly govern how their script renders (quadrat
stacking, shorthand overlaps, beaming): removing them changes the
rendered text, contradicting the "cleaners preserve the document body"
invariant. Preserve them when adjacent to their own script, exactly
like the existing Mongolian/Khmer/Hangul handling; floating between
unrelated text they stay stripped and flagged, and --strip-emoji-glue
paranoid mode still strips them everywhere.
The vendored lightweight-skill engine needed next_input threaded
through its _decide to express "preserve a begin-control from the
character after it"; both engines now behave identically here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: preserve compatibility/halfwidth Hangul fillers in Hangul context
Review feedback on the Layer A cluster: U+3164 and U+FFA0 were stripped
unconditionally while the changelog claimed they were handled "like
U+115F/U+1160", which are preserved after a Hangul jamo. Make the
behaviour match the claim: both new fillers join _HANGUL_FILLERS and
_is_hangul_jamo now covers the compatibility (U+3131-U+318E) and
halfwidth (U+FFA1-U+FFDC) presentation forms, so each filler survives
after a letter of its own form and remains contraband when floating
between unrelated text. Changelog reworded to state the rule precisely.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* feat: add reproducible SynthID-text removal benchmark
bench_synthid_text.py orchestrates the existing Layer B machinery into a
controlled, shareable experiment: generate watermarked + unwatermarked
samples with the MarkLLM SynthID scheme, run removal variants (strength x
candidates) plus controls (no-removal, Layer-A-only, optional re-stamp),
and report clear rate, score suppression, quality, and cost (tokens,
wall time, USD) with a clears-per-MTok efficiency ratio.
Emits report.md / results.json / results.csv with the exact reproduction
command and pinned commits; optional Gemini official-detector tier when
WATERMARKS_GEMINI_API_KEY is set. Mock-based tests, no torch in CI.
* docs: add README section on running the SynthID-text benchmark
Explains what LLM performs the Layer B rewrite (an external model configured
via WATERMARKS_REWRITE_* env vars or --rewrite-* flags; MarkLLM's opt-1.3b is
only the watermark generator/detector) and how to run a benchmark with Ollama
or an OpenAI-compatible endpoint, plus the non-origin-model re-stamp caveat.
* fix: honor WATERMARKS_REWRITE_ALLOW_REMOTE in the SynthID-text benchmark
The --rewrite-allow-remote flag now defaults from the env var (matching
rewrite_text.py and the other WATERMARKS_REWRITE_* settings), so a
non-loopback rewrite endpoint works after sourcing .env without an extra
flag.
* fix: MarkLLM sparse checkout and deps for the SynthID harness
- setup_markllm.sh sparse-checkout omitted '/visualize/', which
watermark/base.py imports at module load — every scheme (incl. SynthID)
failed with 'No module named visualize' during generation/detection.
- requirements-markllm.txt omitted scikit-learn, imported by the SynthID
detector (watermark/synthid/detector_bayesian_torch.py).
Both broke the MarkLLM harness at runtime; the benchmark's sanity gate then
excluded every sample, producing empty per-variant results.
* fix: drop 4 GiB RLIMIT_AS on benchmark subprocesses
_run_cmd applied the common child RLIMIT_AS (default 4 GiB) via
subprocess_preexec_fn to every MarkLLM/rewrite child. torch needs a much
larger address space: CUDA init failed with 'out of memory' at
cudaGetDeviceCount and the 5.2 GB fp32 opt-1.3b could not load, so every
sample was excluded at generation. text_detectors.py already applies no
address-space cap to MarkLLM by default; the benchmark now matches.
* perf: keep MarkLLM resident via a serve worker (624 cold starts -> 1)
The benchmark spawned a fresh torch + opt-1.3b process per operation
(~60-90s each); a full run needs ~624 of them. detect_text_watermark.py
gains a 'serve' mode (JSON-lines over stdin/stdout, ready handshake) that
loads the model once; bench_synthid_text.py uses it via MarkLLMWorker with
automatic fallback to one-shot subprocesses (--no-worker to force).
Turns ~8h runs into ~40-60min.
* perf: skip per-candidate Gemini detections in rewrite subprocess
* feat: run the SynthID-text benchmark from the wr-markllm compose service
- Dockerfile.markllm: add '/visualize/' to the sparse checkout (same fix as
setup_markllm.sh) and COPY the benchmark + rewrite scripts (stdlib-only).
- compose.yaml: wr-markllm gets the WATERMARKS_REWRITE_* and
WATERMARKS_GEMINI_* env wiring, a bench-out volume for --out-dir, and a
read-only mount of the bundled corpus (build context is service/, so the
corpus cannot be COPY'd).
- docs: docker compose run example.
Note: the image ships CPU torch by design, so the container path is for
portability/CI; GPU runs use the host setup_markllm.sh venv.
* feat: per-sample progress logging in the benchmark
The persistent worker returns samples in-memory, so nothing is written
until the end of a run — runs looked stuck. eprint a [gen i/N] line per
generated sample and a [removal] summary per sample.
* chore: migrate Gemini config to gemini-3.6-flash; document SynthID-text retirement
Google retired SynthID text watermarking on the Generative Language API
(Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK
is rejected on current 3.x models (confirmed by Google AI staff). Migrate
the default detection model to gemini-3.6-flash, document the retirement
in vendor-notes.md and the benchmark report caveat, and keep the detector
seam fail-soft until a vendor endpoint (e.g. Vertex AI) returns.
* feat: remove gemini-synthid-text detector (Google retired text watermarking)
Google removed SynthID text watermarking from the Generative Language API
(Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK
is rejected on current 3.x models, so the vendor detector had nothing to
detect. Remove GeminiSynthIDTextDetector and its wiring:
- text_detectors.py: drop the Gemini class, HTTP helpers, and constants;
keep MarkLLM + Claude seams (registry now markllm + claude-text).
- server.py / rewrite_text.py: per-candidate detection now triggers on
--markllm-scheme only.
- bench_synthid_text.py: remove the Gemini tier (before/after, report
table, --no-gemini flag); report caveat notes the retirement.
- configs/docs: drop WATERMARKS_GEMINI_* from .env.example / compose /
README / SKILL.md / vendor-notes.md; keep the retirement note.
- tests: gemini tests removed or converted to MarkLLM (mocked subprocess).
- Dockerfile.markllm: parameterize BASE_IMAGE + TORCH_INDEX_URL so a GPU/
arm64 image can be built (used for the --gpus all benchmark run).
* fix: harden notes aggregation against non-string notes
A run completed all samples but crashed at the final aggregate step with
'cannot use list as a set element' when a row's notes contained a
non-string value. Filter notes to strings (aggregate + CSV) and add a
regression test.
* perf: let the rewrite subprocess reuse the resident MarkLLM worker
The rewrite subprocess (rewrite_text.py) ran its own before/after MarkLLM
detects, each a ~20s torch+model cold start (~12 per sample = ~5min of the
~6min/sample runtime). Now:
- detect_text_watermark.py serve gains --port N: a loopback TCP JSON-lines
listener (default -1 = off) sharing the resident model, with a lock so
stdin and socket requests never run the model concurrently.
- text_detectors.MarkLLMTextDetector checks WATERMARKS_MARKLLM_PORT and
does a fast loopback detect when a worker is up, falling back to the
one-shot subprocess otherwise.
- The benchmark worker publishes its port via that env var, so the rewrite
subprocess inherits it and its detects hit the resident model.
Turns ~6 min/sample into ~1-2 min; a full run drops from ~2h to ~40-50min.
Tests: loopback-client + fallback + env-publish coverage.
* chore: add benchmark-smoke.sh / benchmark-full.sh wrappers
Simple host wrappers: source .env, default MARKLLM_DIR to ~/MarkLLM, use a
repo-local HF cache by default, and run bench_synthid_text.py with a quick
(2 docs, 1 seed, paraphrase:1) or full (8 docs x 3 seeds, three variants,
re-stamp control) configuration. OUT_DIR overrides the output location.
* feat: add vendor text-watermark detection and SynthID image scorer sidecar
Adds Layer B watermark detection as a first-class service capability:
- text_detectors.py: a registry of text-watermark detectors behind one
interface — Google's official SynthID-text detector via the Gemini API
(taskType DETECT_TEXT_WATERMARK), a Claude placeholder ready for
Anthropic's announced detection API, and the MarkLLM research harness
(KGW / SynthID, same-config-only). Fail-soft: unconfigured or errored
detectors never block cleaning.
- server.py: new POST /detect endpoint, detect_before / detect_after options
on /clean (before/after scoring for text and images), an opt-in
/inspect "detect" flag, and /capabilities gains text_detectors and
scorers.synthid_http.
- synthid_score_server.py: a stdlib HTTP sidecar for the reverse-SynthID
scorer, so the published core image never bundles the non-commercial
upstream code; wired via WATERMARKS_SYNTHID_SCORER_URL.
- score_synthid.py: extract score_file() so the CLI and the sidecar share
one implementation.
- compose.yaml / Dockerfile.synthid / .env.example: wr-synthid-score sidecar
service and env wiring.
- README + skill docs, plus tests for the detectors, the /detect endpoint,
and the image sidecar.
* feat: per-candidate watermark detection for Layer B rewrite candidates
When --candidates N (N > 1) is combined with --markllm-scheme or
WATERMARKS_GEMINI_API_KEY, run every configured text detector from the
text_detectors.py registry on each candidate and report per-candidate
measurements in --json-stats as candidate_scores entries carrying
lexical_divergence, selection_score, selected, and per-detector reports
(is_watermarked, score, threshold where the detector provides one).
Candidate selection stays purely lexical; the detections are observability
for correlating lexical divergence with watermark removal (issue #106).
Converges rewrite_text.py onto the shared detector registry:
- MarkLLMTextDetector gains constructor overrides (scheme, upstream_dir,
model, timeout) plus the checkout-venv interpreter preference and the
WATERMARKS_MARKLLM_RLIMIT_AS preexec guard ported from rewrite_text.py;
the old _markllm_detect / _venv_python / _markllm_preexec helpers are gone.
- run_all_text_detectors() accepts an injected MarkLLM instance and an
include_markllm switch so CLI flag gating stays intact.
- before/after/cleared semantics unchanged; detection remains fail-soft.
* docs: pin Watermarks in the Sand reference to arXiv v5
* fix: mark only one rewrite candidate as selected (#110)
---------
Co-authored-by: Zhenxin Ai <142008897+ai-kunkun@users.noreply.github.com>
* fix: rewrite ODT/EPUB manifests and measure real zip bytes (#122)
Two container correctness/security fixes from issue #122:
- clean_odt dropped marker-bearing parts while leaving their entries in
META-INF/manifest.xml, so readers flagged the package as damaged. It is
now two-pass: compute the dropped set, then rewrite the manifest
attribute-order-independently, and write each part exactly once. The same
bug class in clean_epub (dropped parts left in the OPF manifest, plus
dangling spine itemrefs) gets the same two-pass treatment.
- The zip budget trusted ZipInfo.file_size from the archive's own central
directory, so a crafted DOCX/ODT could declare a tiny size and still
expand via zf.read. Budgets are now charged on actual decompressed bytes
via _read_zip_member (streaming, cap enforced mid-read), with the declared
size kept only as a fast-path pre-reject.
* fix: classify unrecognized bytes as "unknown", not text (#122)
Two classification defects from issue #122:
- format_dispatch.classify_bytes fell back to "text" for any unrecognized
file, so a binary with valid UTF-8 runs could be decoded and written back
mangled (corrupted with --in-place) in clean_file auto mode. Unrecognized
bytes now classify as "unknown"; clean_file refuses them in auto mode
(exit 2, no write, router advice) and --as text / --force-text are the
explicit opt-ins. inspect_file reports kind "unknown" (exit 0), audit_lib
records a non-actionable item, and the HTTP server answers /inspect with
kind "unknown" but rejects /clean of unknown formats (400).
- classify(path) read the whole file to sniff a header, and only a full read
could detect zip containers. It now routes known extensions without
reading, sniffs a 4096-byte header once for images and prefix-based
containers, and reads the whole file only when the header is a zip local
header (PK), where the container signature lives in the central directory.
* feat: distinct exit code for partial audits (#122)
audit_dir and audit_website reported success (0) even when some files or
URLs could not be scanned; the exit status was computed only over the items
that succeeded. A scan that is missing items is not a clean scan.
- common.EXIT_PARTIAL = 3, with precedence: partial (3) > actionable (1)
> clean (0) — an incomplete audit is the more important CI signal.
- audit_dir returns 3 when any file was skipped/failed; audit_website
returns 3 when any URL failed to fetch or inspect. Both are independent
of the output format (human/json/sarif already share one return).
* fix: verify the pinned upstream ref on existing checkouts (#122)
setup_ctrlregen.sh/setup_synthid.sh (and their .ps1 twins) only verified
the pinned commit in the fresh-clone branch; an existing checkout at an
unknown or drifted revision was silently reused, defeating the commit pin.
All four scripts now check HEAD against the pinned ref in the
existing-checkout branch too, and repair by fetch + detach checkout
(re-applying the sparse-checkout set), failing hard if the ref cannot be
reached or the re-pin does not land on it.
* docs: unknown-format behavior, audit exit codes, backend isolation (#122)
- README: clean_file no longer auto-cleans unrecognized formats (--as text
/ --force-text are the opt-ins), and the CtrlRegen bootstrap documents the
isolation expectation for its research-era dependency pins plus the new
re-pin check on existing checkouts.
- SKILL.md: audit exit codes (0/1/2/3, partial=3) and a note that /clean
requires a name with a known extension.
- audit_website: document why stdlib ElementTree is used (stdlib-first) and
that defusedxml is the fallback if that policy changes (DTD rejection stays).
- requirements-ctrlregen.txt: advisory/isolation note for the pinned research
dependencies.
* test: ODT manifest and EPUB OPF dangling-ref regressions (#122)
- clean_odt: dropped marker-bearing parts remove their META-INF/manifest.xml
file-entry (attribute-order-independent), exactly one manifest entry, root
and surviving entries kept, and the manifest is byte-identical when nothing
is dropped.
- clean_epub: dropped non-content parts lose their <item> entry in the OPF
manifest, so the book no longer references removed members.
`base64 -w0 FILE` is a GNU coreutils form. On macOS/BSD it fails with
`base64: invalid argument FILE` and prints usage, so every documented
`/inspect` and `/clean` example is unrunnable on a Mac host.
This matters more than a normal docs typo because SKILL.md instructs the
agent to call the service via these exact commands and explicitly forbids
falling back to local cleaning when the service call fails. A macOS agent
therefore stops at its first request instead of degrading gracefully.
Replace all 7 occurrences (2 in README.md, 5 in SKILL.md) with the POSIX
form, which reads from stdin and unwraps any line breaks:
base64 < FILE | tr -d '\n'
Verified byte-identical output on both implementations:
- BSD/macOS base64 (macOS 15, Darwin 24.6.0)
- GNU coreutils base64 (`gbase64`)
Also verified the rewritten commands verbatim against a live
`server.py` on 127.0.0.1:8765 — `/inspect` and `/clean` both return
`ok: true` and correctly strip U+200B / U+00AD.
Docs-only; no code paths changed, so no unit tests added.
`python3 -m pytest -q` and `ruff check`/`format --check` pass.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Add stdlib-only detection, inspection, and cleaning for four more formats.
- BMP: locate the pixel payload via the DIB header and strip trailing
non-image metadata (the only place non-standard BMP metadata can live),
rewriting the file-size field.
- GIF: drop comment and XMP application extensions while preserving
NETSCAPE2.0 looping, ICC, graphic-control, and image blocks.
- TIFF (classic + BigTIFF): walk IFD chains and drop XMP/EXIF/GPS/IPTC/
Photoshop/MakerNote tags, zeroing orphaned payloads while keeping
strip/tile offsets valid.
- EPUB: scrub OPF package metadata and XHTML meta/JSON-LD, clean embedded
raster/SVG media, apply Layer A to XHTML body text, and pass OCF-encrypted
parts through untouched.
All four route through format_dispatch, so the unified CLIs, the HTTP
service, and the audits pick them up automatically.
* fix: pin vendored Cursor-skill text engine to the service copy
The engine vendored into skills/clean-user-facing-text/ had silently
drifted behind service/scripts/text_unicode.py: it still blanket-
stripped legitimate RTL directional marks and isolates (corrupting
mixed RTL/LTR prose) and stripped emoji variation selectors after
arrow and symbol bases, because it never received the preservable-bidi
and emoji-base updates. Nothing in the suite compared the two copies.
Replace the vendored engine with an exact copy of the service one and
add a byte-equality test so any future engine change must land in both
files in the same commit, plus a behavioural regression test for the
RTL and arrow cases through the vendored CLI. The CLI wrappers remain
deliberately different (text-only skill: no stylometry, no
--strip-bidi).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: re-pin vendored engine after merging main and correct bidi comment
Merging main into the PR branch kept main's ruff-formatted service
engine (#103) but the PR's vendored copy pinned in 519c6db, so the two
drifted apart again and test_vendored_text_unicode_is_identical_to_service_engine
failed. Re-sync the vendored copy to the current service engine so the
byte-for-byte pin holds.
Also correct the _PRESERVABLE_BIDI_CPS comment: paired LRE/RLE embeddings
are preserved via _valid_bidi_embedding_indices, so only overrides and
unpaired embeddings remain destructive by default.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
The bullet 'Best-effort: prose was rewritten...' uses a full sentence with
passive verb while adjacent bullets use noun phrases. Removed 'was' to
match the parallel structure of the other bullets.
Audit metadata:
- Category: grammar
- Severity: minor
- File: skills/clean-user-facing-text/SKILL.md, line 61
- Fingerprint: a9c80f5f5afbfbe7
Co-authored-by: pi <pi@m2.local>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* feat: add lightweight Cursor and Codex text skill
Package the text-only workflow with safe cross-platform installation, optional persistent instructions, and focused tests so users can adopt it without the media tooling.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: keep the lightweight skill independently reviewable
Avoid coupling the packaging PR to the separate Unicode safety change so either pull request can merge on its own.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: force UTF-8 for subprocess stdin in Windows CI
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* feat: split skill from service, add HTTP API and Docker distribution
The agent skill (skills/remove-ai-marks/) is now a code-free remote client:
all implementation moved to service/scripts/ and runs behind a stdlib HTTP
service (server.py) with /health, /capabilities, /inspect, /clean and a
dynamically generated OpenAPI 3.0.3 spec at /openapi.json.
- Move scripts/ and the backend Dockerfiles under service/
- server.py: JSON/base64 HTTP entrypoint with size caps, binary guard,
atomic writes, loopback default, optional bearer auth
- Core Dockerfile (exiftool/qpdf/c2patool preinstalled) and a GHCR publish
workflow for the core/markllm/markdiffusion images
- compose.yaml (wr-* services, harness/heavy profiles) + compose-check.sh
to validate the running stack (exit code only)
- Fix markllm image build (tokenizers 0.22.2, CPU-only torch) and ctrlregen
build (python:3.11 base for the 2023-era research pins)
- Fix markllm/markdiffusion harness images missing common.py at runtime
* docs: add .env.example and service configuration guide
* fix: disable chain-of-thought for openai-compatible Layer B rewrites
deepseek-v4-flash is a reasoning model: a one-line paraphrase burned 9,894
reasoning tokens (~100s) and hit the default timeout. Send
reasoning_effort=none by default for the openai-compatible backend
(--reasoning-effort / WATERMARKS_REWRITE_REASONING_EFFORT; 'off' omits the
parameter), cutting the same rewrite to ~1s / 12 tokens. Tested end-to-end
against api.deepseek.com.
* fix: sanitize client-supplied filename in HTTP service
CodeQL 'uncontrolled data in path expression' (server.py): a name like
'../../x' flowed into Path(tmpdir) / name, letting an upload escape the
request temp dir on write. Sanitize name to its basename in _decode_input
(_safe_name) and refuse any joined path whose parent is not the tmpdir at
the write sites (_tmp_path). Tests cover traversal names.
* chore: gitignore .env (contains local rewrite credentials)
* chore: deny-by-default gitignore and dockerignore; document compose env config
.gitignore and service/.dockerignore now exclude everything by default and
explicitly allow only what is publishable/needed: tracked source, docs,
tests, .github, and (for images) the service/scripts/ tree that every
Dockerfile COPYs. Root .dockerignore documents that all builds use service/
as context. README Configuration section now covers .env setup for docker
compose, host-side export for CLI runs, and the full variable table.
Unify the file-to-pipeline classification that was duplicated across
inspect_file.py, clean_file.py and audit_lib.py into one format_dispatch
module. Adding a format now means touching a single extension table and
sniff path; audit_lib no longer imports classify from a CLI entry point.
* feat: optional MarkLLM text-watermark verification harness
Add an optional external backend wrapping THU-BPM/MarkLLM (Apache-2.0)
so a specific statistical text-watermark scheme (KGW / SynthID-Text)
can be verified before/after a Layer B rewrite.
- detect_text_watermark.py: detect/watermark subcommands, external
checkout at a pinned commit, exit codes 0/1/2/3, --json
- rewrite_text.py --markllm-scheme: before/after detection around the
rewrite, reports a `cleared` flag; never fails the rewrite when the
backend is unavailable
- setup_markllm.sh + requirements-markllm.txt (pinned deps) +
Dockerfile.markllm + Makefile bootstrap/smoke/docker targets
- tests/test_markllm_detect.py: 16 mock-based cases (no torch in CI)
- Docs: verification-harness caveat (same-config-only, not a
vendor-detector oracle) in README, SKILL.md, removal-matrix, vendor-notes
* chore: tidy merged Unreleased changelog list
* security: harden the MarkLLM harness (offline, caps, supply-chain)
Addresses the PR security review:
- detect_text_watermark.py: --offline loads the scoring model from the HF
cache only (local_files_only + HF_HUB_OFFLINE, no remote code), and the
algorithm config is capped at 1 MiB so a crafted huge file is refused
before either this script or upstream reads it into memory
- rewrite_text.py: WATERMARKS_MARKLLM_RLIMIT_AS (env, POSIX) optionally
applies RLIMIT_AS to the MarkLLM subprocess; off by default because
torch/CUDA needs large address spaces
- Dockerfile.markllm: drop the unpinned torch install (it is pinned in
requirements-markllm.txt) and verify the cloned upstream commit SHA
- tests: offline flag, config-too-large, and preexec/rlimit cases
- docs: hardening knobs in README + SKILL.md; changelog updated
* fix: harden website audit against SSRF and gzip bombs
* fix: pin TLS minimum version to TLSv1.2 for pinned HTTPS connections
---------
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
setup_ctrlregen.sh and setup_synthid.sh are POSIX-only: they invoke
"$DIR/.venv/bin/python", a layout `python -m venv` never produces on Windows,
so both abort right after the clone. image_meta.py already resolves
`.venv\Scripts\python.exe` under `os.name == "nt"`, so only the bootstrap was
missing.
setup_ctrlregen.ps1 and setup_synthid.ps1 mirror the shell scripts: same
pinned upstream commits and SHA verification, same sparse-checkout paths, same
`pip==26.2.1` pin, same flags (as -Dir / -Ref / -Full / -Python).
One deliberate difference in setup_ctrlregen.ps1. The shell script derives the
torch wheel tag from the CUDA version nvidia-smi prints, but that is the
maximum the *driver* supports, not what to install - drivers are backward
compatible, so a cu126 wheel runs fine on a 13.0 driver. Worse, the naive tag
breaks on older cards: cu128+ and CUDA 13 dropped Maxwell/Pascal/Volta, so a
GTX 1080 Ti (sm_61) would get a cu130 torch with no kernels for it and fail at
runtime with "no kernel image is available for execution". Checking that the
index URL exists does not catch this - every index returns 200. The port keys
off the GPU's compute capability instead, forces cu126 below 7.5, and verifies
the installed wheel with torch.cuda.get_arch_list().
Both scripts parse clean; the CUDA detection was checked against a real
compute-capability 6.1 GPU. The .sh scripts are left untouched.
Co-authored-by: uniendodimensionesgm <153029553+uniendodimensionesgm@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
`exiftool -all=` on a PDF appends a %BeginExifToolUpdate block that frees the
Info object and drops /Info from the trailer, but the original metadata bytes
stay in the file verbatim - exiftool itself reverts the edit with
`-PDF-update:all=`. clean_pdf reported "exiftool -all= (rc=0)" and nothing
else, so a PDF still carrying `/Producer (Claude Opus)` in plain bytes read as
cleaned. The output file gets *larger*, which is the tell.
Follow the exiftool pass with `qpdf --linearize`, which re-serializes the
document from its object graph and so drops the now-unreferenced objects.
Guarded on which("qpdf"): without qpdf the behaviour is unchanged except for
an explicit warning that the metadata bytes remain recoverable.
Tests cover the no-qpdf warning, the rebuild replacing the exiftool output,
qpdf's rc=3 "succeeded with warnings", a failed rewrite leaving the exiftool
output in place, and an end-to-end check (skipped without the real tools)
asserting no readable metadata survives in the output bytes. That last one
fails on the leak itself when the rewrite is reverted.
Co-authored-by: uniendodimensionesgm <153029553+uniendodimensionesgm@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* fix: SECURITY.md only references the private advisories channel
* fix: preserve load-bearing script invisibles, strip PUA in Layer A (#38)
Mongolian FVS (U+180B-180D), Khmer inherent vowels (U+17B4/17B5) and
Hangul fillers (U+115F/1160) are orthography, not hidden marks, when
they follow a base from their own script. STRIP_CODEPOINTS removed them
unconditionally, corrupting real text (FVS glyph selection, Khmer
phonemic vowels, Hangul partial-syllable composition).
Mirror the existing #28 glue machinery: keep them after a same-script
base, still strip them when floating between unrelated characters, and
honor --strip-emoji-glue for paranoid mode. Add them to _is_glue so
chained FVS stay bound to their base.
Private-use codepoints (U+E000-F8FF and the supplementary PUA planes)
render as nothing portable and carry no agreed meaning, making them a
serviceable hiding place; they are Co, so they fell outside the Cf
strip path. Strip them by default with a new 'private_use' inspect kind.
Refs #38.
Add the resolved path at the top of every human report (File: line) and
include path in JSON output for all kinds, so batch inspection via find
can attribute hits to a file. Closes#31.
clean_markdown() had two bugs in the same loop:
1. It crashed with IndexError on any frontmatter containing a blank
line, because line[0] was evaluated on an empty string.
2. It kept the nested children of a dropped top-level key, so dropping
`model:` left `name: claude-opus` behind and produced invalid YAML,
while still reporting the key as removed.
Rewrite the loop with a `dropping` flag that tracks whether the current
top-level key was dropped, and guard blank/comment lines before
indexing. inspect_markdown() is unchanged (already guards line[0]).
Adds 4 regression tests: blank-line crash, nested-key leak, inspect
round-trip, and comment/list preservation.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Layer A already preserves emoji ZWJ/VS glue, but still stripped other
load-bearing invisibles, silently corrupting real text:
- ZWNJ/ZWJ inside complex scripts (Persian میروم, Devanagari क्ष)
- flag emoji tag sequences (🏴 -> 🏴)
- orthographic Arabic/Syriac Cf marks (U+0600, U+06DD, U+070F, ...)
Extend the existing _decide()/glue machinery: keep ZWNJ/ZWJ when a
neighbour is a complex-script letter, keep tag chars after an emoji
base, and allowlist the orthographic Cf codepoints. The same characters
between plain ASCII stay carriers and are still stripped.
--strip-emoji-glue continues to strip all of them (paranoid mode).
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
clean_file.py and clean_image.py computed the failure exit code inside
the human-output branch, so `--json` always exited 0 even when the clean
left C2PA/AI signals behind. A script gating on `clean_file --json` would
treat a still-marked file as clean.
Move the residual (and degraded-PDF) decision out of the output branch in
both entry points so the exit code is the same regardless of --json.
Human output is unchanged; degraded best-effort PDF copies stay
non-failures.
Adds tests asserting json and human modes return the same exit code for
residual, clean, and degraded cases.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* Refuse binary input in the text-only tools
inspect_text.py, clean_text.py and rewrite_text.py accept any path and decode
it with errors="surrogateescape". Pointed at a .docx - a zip - they walk
deflate-compressed bytes and report whatever codepoints fall out of them. The
counts look like findings but track the compression, not the content: in one
sample set a document with nothing hidden in its text reported 12 "suspicious"
characters, while another with 54 real no-break spaces reported 11, none of
which were the no-break spaces.
clean_text.py is worse than misleading. It writes the mangled decode back, so
`clean_text.py report.docx` reports "removed=1" and silently corrupts the
document - the output still passes zipfile.is_zipfile() because the end-of-
central-directory record survives, but reading a member raises.
common.looks_binary() now sniffs magic numbers plus a control-byte ratio, and
guard_binary() refuses with a message naming the tool that does handle the
format. The ratio test is deliberately conservative so text in encodings other
than UTF-8 keeps working, and every entry point takes --force-text to override.
clean_file.py gets the same check on the branch where classify() falls back to
"text" for unrecognised bytes.
Adds tests covering magic-number and heuristic detection, the override, refusal
without writing or backing up, and that clean_file.py still routes a .docx to
the container path.
* Address review: backup ordering, stdin sniff, router advice
Three fixes from the review on #24.
clean_file.py sniffed after --in-place had already taken the backup, so
`clean_file.py --in-place mystery.bin` left a mystery.bin.bak sidecar behind
before exiting 2 — for a file the run never touches, and exactly what
clean_text.py avoids. The sniff now runs before backup_path(). The same hole
applied to `--as text` on a .docx, which bypasses classify() entirely.
The stdin path decoded before sniffing, which made detection depend on the
console codec. It was worse than codec drift: the text layer also translates
newlines, so PNG's `\x89PNG\r\n\x1a\n` arrived as `\x89PNG\n\x1a\n` and the
magic number never matched — the file was refused by the NUL-byte heuristic
instead, and would have sailed through had it lacked NULs. _read_stdin_capped
now reads sys.stdin.buffer and guards the raw octets, matching the file path,
with a text fallback for a replaced stdin.
guard_binary always advised "Use inspect_file.py / clean_file.py", which is
circular when the caller is one of them and classify() has already ruled out
every known container. The advice is now a parameter: the text-only scripts
keep the pointer to the routers, and the routers say the bytes match no
supported format and point at --force-text / --as.
Adds tests for the backup ordering (both --in-place paths), the advice split,
and stdin magic that is not ASCII, across default, cp1252 and latin-1 stdio
codecs — the previous stdin test piped a ZIP, whose "PK" header is ASCII and
survives any of them.
---------
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Wires a standalone clean_ctrlregen.py adapter plus setup_ctrlregen.sh bootstrap, Dockerfile, Makefile targets, and clean_image.py --remove-pixel ctrlregen. The backend is cloned at a pinned commit and never bundled (noai-watermark ships no LICENSE file). Includes mock-based tests and docs with research references.
- classify findings as confirmed/probable/informational/likely_false_positive
- expose confidence in text/image/container JSON and human reports
- add audit_dir.py and audit_website.py for aggregate reports
- document the confidence taxonomy and audit commands in SKILL.md
Closes#13
- DOCX: scan only docProps/customXml parts, not the visible body (#14)
- Text Layer A: preserve emoji VS16/ZWJ after an emoji base; add paranoid flag (#22)
- HTML: treat CMS generator tags as informational, not AI metadata (#13)
- PDF: exclude stream payloads from AI-marker byte scan (#13)
- Add unsupported/best-effort notes to inspect reports
* fix: make subprocess and symlink tests Windows-compatible
- Gate preexec_fn (POSIX-only) so optional tools can run on Windows
- Skip symlink tests when the platform denies the privilege
- Add a Windows CI leg and a Windows CLI smoke run
* fix: use UTF-8 stdio and fail Windows smoke on first error
- Reconfigure stdin/stdout/stderr to UTF-8 so Windows redirected streams
stop raising UnicodeEncodeError on invisible Unicode characters
- Chain the Windows smoke commands so a failing script fails the step
safe_write_bytes() calls os.fchmod() to restore the umask-default mode
after mkstemp() creates the temp file as 0600. os.fchmod does not exist
on Windows, so every write path raised:
AttributeError: module 'os' has no attribute 'fchmod'
That broke clean_text.py, clean_file.py, clean_image.py and
rewrite_text.py whenever an output file was requested.
Windows has no POSIX mode bits to restore, so the call is skipped there.
The atomic-write guarantee is unaffected: the symlink refusal and
os.replace() are what protect the destination, not the chmod.
Adds a regression test that deletes os.fchmod via monkeypatch, so the
case is covered on POSIX CI too.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Security hardening for the cleaner/inspector entry points:
- common.safe_arg(): prefix './' to paths starting with '-', so a crafted
filename (e.g. '-@argfile') cannot be interpreted as an option by
exiftool/c2patool. Applied to every exiftool/c2patool invocation in
image_meta.py and container_meta.py (clean_pdf).
- container_meta.py: add MAX_ZIP_DECOMPRESSED_BYTES (512 MiB), checked via
ZipInfo.file_size before decompression in inspect_docx/clean_docx/
inspect_odt/clean_odt, rejecting zip bombs early.
- clean_file.py / inspect_file.py: refuse inputs larger than
WATERMARKS_MAX_INPUT_BYTES (default 1 GiB).
- Fix a latent NameError in inspect_docx: the finding used the undefined
variable `name` instead of info.filename, crashing on the very DOCX files
this tool exists for.
Tests: tests/test_security_hardening.py covers safe_arg, the zip budget
guard, and the DOCX AI-markers path.
Add Institute of AI PM C2PA/SynthID reference, soft-binding out-of-scope
notes, and residual-risk verification guidance so metadata strip is not
mistaken for full watermark clearance.
