COPY preserves source file and directory modes, so a build host with a
restrictive umask (e.g. 027) landed the scripts as 0640 root:root and
the /app/scripts directory as 0750 -- unreadable and untraversable for
the unprivileged runtime user (uid 10001). The image built fine and
failed only at container start, making it easy to miss (#131).
Normalize after the copy with chmod -R a+rX: world-readable everywhere,
execute bit for directories only, intentionally-executable scripts keep
their bit -- deterministic regardless of the builder's umask.
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* fix: report unmeasurable burstiness instead of scoring it as max LLM-like
compute_burstiness returned cv=0.0 for an empty (or single-sentence)
list, which the tiering read as perfectly uniform prose — the strongest
LLM-likeness signal — so any file whose body yielded no parseable
sentences (e.g. entirely wrapped in a code fence) scored falsely high
while word_count stayed high enough to defeat the small-sample dampener
(#132). On a real export this hit 159 of 162 files as pure artifact.
compute_burstiness now returns None for the CV when it cannot be
measured; the scoring site drops the burstiness component and
renormalizes the composite over AI-phrase density and lexical
diversity, with an explicit note. burstiness_cv is null in JSON for
unmeasurable cases, the CLI prints n/a instead of crashing, and the
uniform-cadence finding guards on cv not being None.
* style: satisfy ruff import sorting and formatting
---------
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Co-authored-by: Guillaume Meyer <guillaumemeyer@users.noreply.github.com>
* fix: parse single-quoted attributes in relationship/manifest pruning
XML allows attribute values in single or double quotes. The Target
extraction in _prune_dangling_relationships and the full-path
extraction in _prune_odt_manifest_entries matched only double quotes,
so a valid single-quoted Relationship parsed as an empty target,
resolved to the package base directory, and was deleted outright —
corrupting DOCX/XLSX/PPTX and ODT packages produced by tools that
emit single quotes (#130).
Both extractions now use a backreference pattern
((["'])(.*?)\1) so either quote style resolves the real target, and
pruning still only drops relationships whose parts were genuinely
removed. Regression tests cover the single-quoted kept relationship
(the reported corruption) and the double-quoted dropped part.
* fix: satisfy ruff lint and format checks
---------
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
clean_svg/clean_odt stripped metadata with lazy dot-matches-all regexes; on many unclosed opening tags the engine rescans to end-of-input from every candidate start (O(n^2)), and the GIL stalls the whole single-process service. A ~1.4 KB ODT request pinned a core for ~99 s. Replace every lazy .*?</close> block scan in container_meta.py (SVG metadata/xmpmeta/comments, ODT meta:generator/dc:creator, HTML JSON-LD, OOXML/ODF text-run scrubs, docProps fields, EPUB OPF meta/dc, PDF xpacket/stream) with a linear scan pairing opening tags with a forward pointer over closing tags - identical match semantics, O(n). Add regression tests asserting the advisory PoC completes in <5 s and that stripping behavior is unchanged.
The file-cleaners layer covered 15 formats -- all image, document, or
text -- and zero audio/video. That gap gets more expensive every month:
Sora, Veo, ElevenLabs, and Suno all embed provenance through the same
mechanisms image generators do, just in different containers.
New av_meta.py adds inspect/clean for:
- MP4/MOV/M4A/M4V: top-level C2PA (jumb/c2pa box) and XMP (uuid box)
detection/stripping reuse inspect_isobmff()/strip_isobmff() from
image_meta.py unchanged -- that's exactly the mechanism the C2PA spec
defines for ISOBMFF-family containers, already proven for AVIF/HEIC.
moov/udta (where generator/tool tags live) is handled separately since
it's MP4-specific.
- WAV: RIFF LIST INFO chunk + embedded id3 chunk.
- MP3: ID3v2 frames, per-frame for v2.3/v2.4, whole-tag fallback for
v2.2 (3-byte frame IDs are detected but not decomposed, so a partial
rewrite is never attempted there).
Every box/chunk/frame is either kept byte-identical or dropped whole --
nothing does a partial in-place rewrite of a payload, so a container can
never come out semantically mangled. Default strip_all_metadata=True
matches this project's existing default (privacy-first: drop everything,
--keep-non-ai-metadata narrows to only AI-flagged content), same as the
image cleaners.
Wired through the full dispatch stack so the feature isn't a half
integration: format_dispatch.py (new "av" Kind), inspect_file.py /
clean_file.py (--as av), audit_lib.py (so audit_dir.py's CI/SARIF path
and the pre-commit hooks from #135 both cover audio/video too), and
server.py (HTTP /inspect and /clean).
Closes#134
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
CI gating for AI provenance marks already exists (audit_dir.py's -j
concurrency + SARIF export from #101), but that only runs after a
marked file has already been committed and pushed. Catch it at commit
time instead, using git's own hook point.
Adds two hooks via .pre-commit-hooks.yaml:
- watermarks-remover-check: fails the commit and lists findings when
staged files carry AI/C2PA marks. Wraps audit_lib.scan_file() /
is_actionable() -- the exact per-file logic audit_dir.py already
uses for CI, so the pre-commit gate and the CI gate agree on what
counts as actionable.
- watermarks-remover-clean (opt-in): rewrites staged files in place by
shelling out to clean_file.py --in-place per file (no duplicated
cleaning logic), then exits 1 so the developer reviews the diff and
re-stages -- the same convention as auto-fixing hooks like ruff --fix.
.pre-commit-hooks.yaml needed an explicit allow-rule in the deny-by-
default .gitignore, same as every other root-level config file already
listed there.
Closes#135
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Directory-scale cleaning already exists in the CLI (audit_dir.py, -j
concurrency, SARIF export from #101), but the HTTP service handled one
file per request. Any web app or CI step talking to the service over
HTTP instead of the CLI paid N full round trips to clean N files.
Extract the single-file /inspect and /clean logic into _inspect_payload
and _clean_payload so both the existing single-file endpoints and the
new batch endpoints run the identical pipeline — no duplicated cleaning
logic. A malformed entry in a batch (bad base64, unknown option,
unrecognized format) surfaces as that entry's "ok": false with an
"error" string instead of aborting the rest of the batch.
Capped at WATERMARKS_MAX_BATCH_FILES per request (default 50) as
defense-in-depth against a request packing many tiny files into one
call; the existing MAX_BODY_BYTES envelope cap already bounds total
payload size the same as a single-file request.
/openapi.json picks up both routes automatically since the spec is
generated from the route table.
Closes#136
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* feat: add vendor text-watermark detection and SynthID image scorer sidecar
Adds Layer B watermark detection as a first-class service capability:
- text_detectors.py: a registry of text-watermark detectors behind one
interface — Google's official SynthID-text detector via the Gemini API
(taskType DETECT_TEXT_WATERMARK), a Claude placeholder ready for
Anthropic's announced detection API, and the MarkLLM research harness
(KGW / SynthID, same-config-only). Fail-soft: unconfigured or errored
detectors never block cleaning.
- server.py: new POST /detect endpoint, detect_before / detect_after options
on /clean (before/after scoring for text and images), an opt-in
/inspect "detect" flag, and /capabilities gains text_detectors and
scorers.synthid_http.
- synthid_score_server.py: a stdlib HTTP sidecar for the reverse-SynthID
scorer, so the published core image never bundles the non-commercial
upstream code; wired via WATERMARKS_SYNTHID_SCORER_URL.
- score_synthid.py: extract score_file() so the CLI and the sidecar share
one implementation.
- compose.yaml / Dockerfile.synthid / .env.example: wr-synthid-score sidecar
service and env wiring.
- README + skill docs, plus tests for the detectors, the /detect endpoint,
and the image sidecar.
* feat: per-candidate watermark detection for Layer B rewrite candidates
When --candidates N (N > 1) is combined with --markllm-scheme or
WATERMARKS_GEMINI_API_KEY, run every configured text detector from the
text_detectors.py registry on each candidate and report per-candidate
measurements in --json-stats as candidate_scores entries carrying
lexical_divergence, selection_score, selected, and per-detector reports
(is_watermarked, score, threshold where the detector provides one).
Candidate selection stays purely lexical; the detections are observability
for correlating lexical divergence with watermark removal (issue #106).
Converges rewrite_text.py onto the shared detector registry:
- MarkLLMTextDetector gains constructor overrides (scheme, upstream_dir,
model, timeout) plus the checkout-venv interpreter preference and the
WATERMARKS_MARKLLM_RLIMIT_AS preexec guard ported from rewrite_text.py;
the old _markllm_detect / _venv_python / _markllm_preexec helpers are gone.
- run_all_text_detectors() accepts an injected MarkLLM instance and an
include_markllm switch so CLI flag gating stays intact.
- before/after/cleared semantics unchanged; detection remains fail-soft.
* docs: pin Watermarks in the Sand reference to arXiv v5
* fix: mark only one rewrite candidate as selected (#110)
---------
Co-authored-by: Zhenxin Ai <142008897+ai-kunkun@users.noreply.github.com>
* fix: rewrite ODT/EPUB manifests and measure real zip bytes (#122)
Two container correctness/security fixes from issue #122:
- clean_odt dropped marker-bearing parts while leaving their entries in
META-INF/manifest.xml, so readers flagged the package as damaged. It is
now two-pass: compute the dropped set, then rewrite the manifest
attribute-order-independently, and write each part exactly once. The same
bug class in clean_epub (dropped parts left in the OPF manifest, plus
dangling spine itemrefs) gets the same two-pass treatment.
- The zip budget trusted ZipInfo.file_size from the archive's own central
directory, so a crafted DOCX/ODT could declare a tiny size and still
expand via zf.read. Budgets are now charged on actual decompressed bytes
via _read_zip_member (streaming, cap enforced mid-read), with the declared
size kept only as a fast-path pre-reject.
* fix: classify unrecognized bytes as "unknown", not text (#122)
Two classification defects from issue #122:
- format_dispatch.classify_bytes fell back to "text" for any unrecognized
file, so a binary with valid UTF-8 runs could be decoded and written back
mangled (corrupted with --in-place) in clean_file auto mode. Unrecognized
bytes now classify as "unknown"; clean_file refuses them in auto mode
(exit 2, no write, router advice) and --as text / --force-text are the
explicit opt-ins. inspect_file reports kind "unknown" (exit 0), audit_lib
records a non-actionable item, and the HTTP server answers /inspect with
kind "unknown" but rejects /clean of unknown formats (400).
- classify(path) read the whole file to sniff a header, and only a full read
could detect zip containers. It now routes known extensions without
reading, sniffs a 4096-byte header once for images and prefix-based
containers, and reads the whole file only when the header is a zip local
header (PK), where the container signature lives in the central directory.
* feat: distinct exit code for partial audits (#122)
audit_dir and audit_website reported success (0) even when some files or
URLs could not be scanned; the exit status was computed only over the items
that succeeded. A scan that is missing items is not a clean scan.
- common.EXIT_PARTIAL = 3, with precedence: partial (3) > actionable (1)
> clean (0) — an incomplete audit is the more important CI signal.
- audit_dir returns 3 when any file was skipped/failed; audit_website
returns 3 when any URL failed to fetch or inspect. Both are independent
of the output format (human/json/sarif already share one return).
* fix: verify the pinned upstream ref on existing checkouts (#122)
setup_ctrlregen.sh/setup_synthid.sh (and their .ps1 twins) only verified
the pinned commit in the fresh-clone branch; an existing checkout at an
unknown or drifted revision was silently reused, defeating the commit pin.
All four scripts now check HEAD against the pinned ref in the
existing-checkout branch too, and repair by fetch + detach checkout
(re-applying the sparse-checkout set), failing hard if the ref cannot be
reached or the re-pin does not land on it.
* docs: unknown-format behavior, audit exit codes, backend isolation (#122)
- README: clean_file no longer auto-cleans unrecognized formats (--as text
/ --force-text are the opt-ins), and the CtrlRegen bootstrap documents the
isolation expectation for its research-era dependency pins plus the new
re-pin check on existing checkouts.
- SKILL.md: audit exit codes (0/1/2/3, partial=3) and a note that /clean
requires a name with a known extension.
- audit_website: document why stdlib ElementTree is used (stdlib-first) and
that defusedxml is the fallback if that policy changes (DTD rejection stays).
- requirements-ctrlregen.txt: advisory/isolation note for the pinned research
dependencies.
* test: ODT manifest and EPUB OPF dangling-ref regressions (#122)
- clean_odt: dropped marker-bearing parts remove their META-INF/manifest.xml
file-entry (attribute-order-independent), exactly one manifest entry, root
and surviving entries kept, and the manifest is byte-identical when nothing
is dropped.
- clean_epub: dropped non-content parts lose their <item> entry in the OPF
manifest, so the book no longer references removed members.
Three independent failure modes from #117:
- $ErrorActionPreference 'Stop' + 2>$null on a native command aborts the
script on torch's harmless stderr warnings (e.g. "Failed to initialize
NumPy" when torch is installed before numpy). Run the probes through a
new Invoke-NativeQuiet helper that lowers EAP to 'Continue' for the
block and restores it afterwards.
- The wheel index tag was derived from the driver's CUDA version, e.g.
cu131 for a 13.1 driver, which does not exist (HTTP 403) and silently
fell back to the default index, i.e. the CPU build on Windows. Probe
the published indices and pick the highest one <= driver that answers
HTTP 200; cu126 is still forced below compute capability 7.5.
- Installing torch alone let requirements-ctrlregen.txt resolve torchvision
from PyPI, and torchvision pins an exact torch, so pip replaced the +cu
build with a +cpu one while the script still exited 0. Install torch AND
torchvision together from the chosen index, and verify after the
requirements install that torch.cuda.is_available() is true - if a GPU
was detected but torch ends up CPU-only, warn loudly and exit non-zero.
Also add a CI step (windows-latest, pwsh) that parses the setup .ps1
scripts and asserts the post-install CUDA verification survives.
Fixes#117
* Handle malformed DOCX/ODT beyond BadZipFile in container inspectors
A corrupt, truncated, encrypted, or unsupported-compression container
raises more than BadZipFile (NotImplementedError, RuntimeError, EOFError,
OSError, ValueError, zlib.error), so inspect_docx / inspect_odt and
detect_container_format now catch the whole family and degrade to a clear
finding.
The zip-bomb rejection in _check_zip_budget now raises a dedicated
ZipBudgetExceeded so it keeps propagating out of the inspectors, matching
clean_docx / clean_odt, instead of being reported as an unparseable zip.
* fix: satisfy ruff lint and format checks
---------
Co-authored-by: eeshsaxena <chinmaymd72@gmail.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Add stdlib-only detection, inspection, and cleaning for four more formats.
- BMP: locate the pixel payload via the DIB header and strip trailing
non-image metadata (the only place non-standard BMP metadata can live),
rewriting the file-size field.
- GIF: drop comment and XMP application extensions while preserving
NETSCAPE2.0 looping, ICC, graphic-control, and image blocks.
- TIFF (classic + BigTIFF): walk IFD chains and drop XMP/EXIF/GPS/IPTC/
Photoshop/MakerNote tags, zeroing orphaned payloads while keeping
strip/tile offsets valid.
- EPUB: scrub OPF package metadata and XHTML meta/JSON-LD, clean embedded
raster/SVG media, apply Layer A to XHTML body text, and pass OCF-encrypted
parts through untouched.
All four route through format_dispatch, so the unified CLIs, the HTTP
service, and the audits pick them up automatically.
* fix: pin vendored Cursor-skill text engine to the service copy
The engine vendored into skills/clean-user-facing-text/ had silently
drifted behind service/scripts/text_unicode.py: it still blanket-
stripped legitimate RTL directional marks and isolates (corrupting
mixed RTL/LTR prose) and stripped emoji variation selectors after
arrow and symbol bases, because it never received the preservable-bidi
and emoji-base updates. Nothing in the suite compared the two copies.
Replace the vendored engine with an exact copy of the service one and
add a byte-equality test so any future engine change must land in both
files in the same commit, plus a behavioural regression test for the
RTL and arrow cases through the vendored CLI. The CLI wrappers remain
deliberately different (text-only skill: no stylometry, no
--strip-bidi).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: re-pin vendored engine after merging main and correct bidi comment
Merging main into the PR branch kept main's ruff-formatted service
engine (#103) but the PR's vendored copy pinned in 519c6db, so the two
drifted apart again and test_vendored_text_unicode_is_identical_to_service_engine
failed. Re-sync the vendored copy to the current service engine so the
byte-for-byte pin holds.
Also correct the _PRESERVABLE_BIDI_CPS comment: paired LRE/RLE embeddings
are preserved via _valid_bidi_embedding_indices, so only overrides and
unpaired embeddings remain destructive by default.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Introduce Ruff (pinned at 0.16.3) as the project linter + formatter and
enforce it in CI:
- requirements-dev.txt: pin ruff==0.16.3 (exact pins, no drift)
- ruff.toml: line-length 100, target py312; rule set E/F/W/I/UP/B/SIM/RUF/PLW/S
with deliberate ignores (E501 for content strings, S603 for safe_arg
subprocess calls, S101 asserts in tests) and per-file test ignores
- Makefile: add lint / format / lint-fix targets
- .github/workflows/ci.yml: add lint job (ruff check + format --check)
- .gitignore: whitelist ruff.toml
Also fix every finding the new gate surfaced so CI is green:
- 109+ auto-fixes from ruff --fix (import sorting, simplifications,
unused vars, re.I aliases, etc.)
- explicit check=False on all subprocess.run calls (PLW1510)
- harden sitemap XML parsing: reject DTD/entity declarations (S314)
- replace hardcoded /tmp paths in tests with tmp_path (S108)
- narrow/annotate intentional bare excepts (S110/S112), bind loop vars
in closures (B023), raise ... from None (B904), strict= for zip (B905)
- ruff format applied across service/ and tests/
Verified: ruff check + ruff format --check pass; 287 tests pass, 1 skip.
* fix(inspect): run Layer A scan on markdown/html containers
inspect_container() never scanned the text body, so a .md or .html file
carrying invisible Unicode was reported suspicious=false while
clean_container() went on to strip it. Identical bytes gave opposite
verdicts depending on the file extension.
Scan Layer A for exactly the formats clean_container() scrubs (markdown,
html) so inspect predicts clean. Decode with surrogateescape to match
clean's decoding. Expose the count as suspicious_total, the same key
TextInspectReport uses, so the HTTP server's suspicious flag and the
inspect_file CLI exit code pick it up without special-casing.
* docs: changelog entry for the container Layer A inspect fix
* fix(audit): drop duplicate Layer A scan for markdown/html containers
inspect_container() now scans the body for markdown/html, so
audit_lib.scan_file's own Layer A scan produced the same findings twice
(once as 'layer-a:' from the container report, once as 'layer-a [kind]')
and double-counted them in the aggregate. Keep the stylometry check,
which still needs the decoded body text.
---------
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* fix: keep the SynthID scorer's --json stdout pure
The reverse-SynthID upstream prints progress ('CodebookV4 loaded: ...')
straight to stdout. image_meta.py parses the scorer's stdout with
json.loads, so the leak corrupts every score payload into
{'available': False, 'error': 'bad scorer JSON: ...'}.
Redirect stdout to stderr around the upstream calls so --json owns
stdout. Regression test drives the real script against a deliberately
noisy stub upstream (with a stub cv2, so it runs without OpenCV).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: probe realpath -m support instead of realpath presence
macOS ships BSD realpath, which exists but has no -m flag, so
'command -v realpath' takes the GNU branch and both setup bootstraps
abort on the first path they normalize. Probe the flag itself; BSD
systems fall through to the portable pwd fallback already in place.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Adriel <adriel@Adriels-MacBook-Pro-2026.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Keep valid RTL controls, script joiners, variation sequences, and emoji structure while still removing malformed carriers, with regression coverage for each case.
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Bump accelerate to 1.14.0, controlnet-aux to 0.0.10, and safetensors to
0.8.0 in requirements-ctrlregen.txt. Verified end-to-end against the pinned
noai-watermark commit b642ae45 (import + CUDA inference) and via Docker build.
Pin huggingface_hub to 0.24.0: 0.26.0+ removed the cached_download symbol
that diffusers 0.27.2 still imports, so unpinned installs resolve 0.36.2 and
the backend fails to import. Refresh the Dockerfile.ctrlregen base-image
comment to drop the now-stale safetensors 0.4.3 mention.
* feat: split skill from service, add HTTP API and Docker distribution
The agent skill (skills/remove-ai-marks/) is now a code-free remote client:
all implementation moved to service/scripts/ and runs behind a stdlib HTTP
service (server.py) with /health, /capabilities, /inspect, /clean and a
dynamically generated OpenAPI 3.0.3 spec at /openapi.json.
- Move scripts/ and the backend Dockerfiles under service/
- server.py: JSON/base64 HTTP entrypoint with size caps, binary guard,
atomic writes, loopback default, optional bearer auth
- Core Dockerfile (exiftool/qpdf/c2patool preinstalled) and a GHCR publish
workflow for the core/markllm/markdiffusion images
- compose.yaml (wr-* services, harness/heavy profiles) + compose-check.sh
to validate the running stack (exit code only)
- Fix markllm image build (tokenizers 0.22.2, CPU-only torch) and ctrlregen
build (python:3.11 base for the 2023-era research pins)
- Fix markllm/markdiffusion harness images missing common.py at runtime
* docs: add .env.example and service configuration guide
* fix: disable chain-of-thought for openai-compatible Layer B rewrites
deepseek-v4-flash is a reasoning model: a one-line paraphrase burned 9,894
reasoning tokens (~100s) and hit the default timeout. Send
reasoning_effort=none by default for the openai-compatible backend
(--reasoning-effort / WATERMARKS_REWRITE_REASONING_EFFORT; 'off' omits the
parameter), cutting the same rewrite to ~1s / 12 tokens. Tested end-to-end
against api.deepseek.com.
* fix: sanitize client-supplied filename in HTTP service
CodeQL 'uncontrolled data in path expression' (server.py): a name like
'../../x' flowed into Path(tmpdir) / name, letting an upload escape the
request temp dir on write. Sanitize name to its basename in _decode_input
(_safe_name) and refuse any joined path whose parent is not the tmpdir at
the write sites (_tmp_path). Tests cover traversal names.
* chore: gitignore .env (contains local rewrite credentials)
* chore: deny-by-default gitignore and dockerignore; document compose env config
.gitignore and service/.dockerignore now exclude everything by default and
explicitly allow only what is publishable/needed: tracked source, docs,
tests, .github, and (for images) the service/scripts/ tree that every
Dockerfile COPYs. Root .dockerignore documents that all builds use service/
as context. README Configuration section now covers .env setup for docker
compose, host-side export for CLI runs, and the full variable table.
