clean_svg/clean_odt stripped metadata with lazy dot-matches-all regexes; on many unclosed opening tags the engine rescans to end-of-input from every candidate start (O(n^2)), and the GIL stalls the whole single-process service. A ~1.4 KB ODT request pinned a core for ~99 s. Replace every lazy .*?</close> block scan in container_meta.py (SVG metadata/xmpmeta/comments, ODT meta:generator/dc:creator, HTML JSON-LD, OOXML/ODF text-run scrubs, docProps fields, EPUB OPF meta/dc, PDF xpacket/stream) with a linear scan pairing opening tags with a forward pointer over closing tags - identical match semantics, O(n). Add regression tests asserting the advisory PoC completes in <5 s and that stripping behavior is unchanged.
* feat: add vendor text-watermark detection and SynthID image scorer sidecar
Adds Layer B watermark detection as a first-class service capability:
- text_detectors.py: a registry of text-watermark detectors behind one
interface — Google's official SynthID-text detector via the Gemini API
(taskType DETECT_TEXT_WATERMARK), a Claude placeholder ready for
Anthropic's announced detection API, and the MarkLLM research harness
(KGW / SynthID, same-config-only). Fail-soft: unconfigured or errored
detectors never block cleaning.
- server.py: new POST /detect endpoint, detect_before / detect_after options
on /clean (before/after scoring for text and images), an opt-in
/inspect "detect" flag, and /capabilities gains text_detectors and
scorers.synthid_http.
- synthid_score_server.py: a stdlib HTTP sidecar for the reverse-SynthID
scorer, so the published core image never bundles the non-commercial
upstream code; wired via WATERMARKS_SYNTHID_SCORER_URL.
- score_synthid.py: extract score_file() so the CLI and the sidecar share
one implementation.
- compose.yaml / Dockerfile.synthid / .env.example: wr-synthid-score sidecar
service and env wiring.
- README + skill docs, plus tests for the detectors, the /detect endpoint,
and the image sidecar.
* feat: per-candidate watermark detection for Layer B rewrite candidates
When --candidates N (N > 1) is combined with --markllm-scheme or
WATERMARKS_GEMINI_API_KEY, run every configured text detector from the
text_detectors.py registry on each candidate and report per-candidate
measurements in --json-stats as candidate_scores entries carrying
lexical_divergence, selection_score, selected, and per-detector reports
(is_watermarked, score, threshold where the detector provides one).
Candidate selection stays purely lexical; the detections are observability
for correlating lexical divergence with watermark removal (issue #106).
Converges rewrite_text.py onto the shared detector registry:
- MarkLLMTextDetector gains constructor overrides (scheme, upstream_dir,
model, timeout) plus the checkout-venv interpreter preference and the
WATERMARKS_MARKLLM_RLIMIT_AS preexec guard ported from rewrite_text.py;
the old _markllm_detect / _venv_python / _markllm_preexec helpers are gone.
- run_all_text_detectors() accepts an injected MarkLLM instance and an
include_markllm switch so CLI flag gating stays intact.
- before/after/cleared semantics unchanged; detection remains fail-soft.
* docs: pin Watermarks in the Sand reference to arXiv v5
* fix: mark only one rewrite candidate as selected (#110)
---------
Co-authored-by: Zhenxin Ai <142008897+ai-kunkun@users.noreply.github.com>
* fix: rewrite ODT/EPUB manifests and measure real zip bytes (#122)
Two container correctness/security fixes from issue #122:
- clean_odt dropped marker-bearing parts while leaving their entries in
META-INF/manifest.xml, so readers flagged the package as damaged. It is
now two-pass: compute the dropped set, then rewrite the manifest
attribute-order-independently, and write each part exactly once. The same
bug class in clean_epub (dropped parts left in the OPF manifest, plus
dangling spine itemrefs) gets the same two-pass treatment.
- The zip budget trusted ZipInfo.file_size from the archive's own central
directory, so a crafted DOCX/ODT could declare a tiny size and still
expand via zf.read. Budgets are now charged on actual decompressed bytes
via _read_zip_member (streaming, cap enforced mid-read), with the declared
size kept only as a fast-path pre-reject.
* fix: classify unrecognized bytes as "unknown", not text (#122)
Two classification defects from issue #122:
- format_dispatch.classify_bytes fell back to "text" for any unrecognized
file, so a binary with valid UTF-8 runs could be decoded and written back
mangled (corrupted with --in-place) in clean_file auto mode. Unrecognized
bytes now classify as "unknown"; clean_file refuses them in auto mode
(exit 2, no write, router advice) and --as text / --force-text are the
explicit opt-ins. inspect_file reports kind "unknown" (exit 0), audit_lib
records a non-actionable item, and the HTTP server answers /inspect with
kind "unknown" but rejects /clean of unknown formats (400).
- classify(path) read the whole file to sniff a header, and only a full read
could detect zip containers. It now routes known extensions without
reading, sniffs a 4096-byte header once for images and prefix-based
containers, and reads the whole file only when the header is a zip local
header (PK), where the container signature lives in the central directory.
* feat: distinct exit code for partial audits (#122)
audit_dir and audit_website reported success (0) even when some files or
URLs could not be scanned; the exit status was computed only over the items
that succeeded. A scan that is missing items is not a clean scan.
- common.EXIT_PARTIAL = 3, with precedence: partial (3) > actionable (1)
> clean (0) — an incomplete audit is the more important CI signal.
- audit_dir returns 3 when any file was skipped/failed; audit_website
returns 3 when any URL failed to fetch or inspect. Both are independent
of the output format (human/json/sarif already share one return).
* fix: verify the pinned upstream ref on existing checkouts (#122)
setup_ctrlregen.sh/setup_synthid.sh (and their .ps1 twins) only verified
the pinned commit in the fresh-clone branch; an existing checkout at an
unknown or drifted revision was silently reused, defeating the commit pin.
All four scripts now check HEAD against the pinned ref in the
existing-checkout branch too, and repair by fetch + detach checkout
(re-applying the sparse-checkout set), failing hard if the ref cannot be
reached or the re-pin does not land on it.
* docs: unknown-format behavior, audit exit codes, backend isolation (#122)
- README: clean_file no longer auto-cleans unrecognized formats (--as text
/ --force-text are the opt-ins), and the CtrlRegen bootstrap documents the
isolation expectation for its research-era dependency pins plus the new
re-pin check on existing checkouts.
- SKILL.md: audit exit codes (0/1/2/3, partial=3) and a note that /clean
requires a name with a known extension.
- audit_website: document why stdlib ElementTree is used (stdlib-first) and
that defusedxml is the fallback if that policy changes (DTD rejection stays).
- requirements-ctrlregen.txt: advisory/isolation note for the pinned research
dependencies.
* test: ODT manifest and EPUB OPF dangling-ref regressions (#122)
- clean_odt: dropped marker-bearing parts remove their META-INF/manifest.xml
file-entry (attribute-order-independent), exactly one manifest entry, root
and surviving entries kept, and the manifest is byte-identical when nothing
is dropped.
- clean_epub: dropped non-content parts lose their <item> entry in the OPF
manifest, so the book no longer references removed members.
Three independent failure modes from #117:
- $ErrorActionPreference 'Stop' + 2>$null on a native command aborts the
script on torch's harmless stderr warnings (e.g. "Failed to initialize
NumPy" when torch is installed before numpy). Run the probes through a
new Invoke-NativeQuiet helper that lowers EAP to 'Continue' for the
block and restores it afterwards.
- The wheel index tag was derived from the driver's CUDA version, e.g.
cu131 for a 13.1 driver, which does not exist (HTTP 403) and silently
fell back to the default index, i.e. the CPU build on Windows. Probe
the published indices and pick the highest one <= driver that answers
HTTP 200; cu126 is still forced below compute capability 7.5.
- Installing torch alone let requirements-ctrlregen.txt resolve torchvision
from PyPI, and torchvision pins an exact torch, so pip replaced the +cu
build with a +cpu one while the script still exited 0. Install torch AND
torchvision together from the chosen index, and verify after the
requirements install that torch.cuda.is_available() is true - if a GPU
was detected but torch ends up CPU-only, warn loudly and exit non-zero.
Also add a CI step (windows-latest, pwsh) that parses the setup .ps1
scripts and asserts the post-install CUDA verification survives.
Fixes#117
Add stdlib-only detection, inspection, and cleaning for four more formats.
- BMP: locate the pixel payload via the DIB header and strip trailing
non-image metadata (the only place non-standard BMP metadata can live),
rewriting the file-size field.
- GIF: drop comment and XMP application extensions while preserving
NETSCAPE2.0 looping, ICC, graphic-control, and image blocks.
- TIFF (classic + BigTIFF): walk IFD chains and drop XMP/EXIF/GPS/IPTC/
Photoshop/MakerNote tags, zeroing orphaned payloads while keeping
strip/tile offsets valid.
- EPUB: scrub OPF package metadata and XHTML meta/JSON-LD, clean embedded
raster/SVG media, apply Layer A to XHTML body text, and pass OCF-encrypted
parts through untouched.
All four route through format_dispatch, so the unified CLIs, the HTTP
service, and the audits pick them up automatically.
Introduce Ruff (pinned at 0.16.3) as the project linter + formatter and
enforce it in CI:
- requirements-dev.txt: pin ruff==0.16.3 (exact pins, no drift)
- ruff.toml: line-length 100, target py312; rule set E/F/W/I/UP/B/SIM/RUF/PLW/S
with deliberate ignores (E501 for content strings, S603 for safe_arg
subprocess calls, S101 asserts in tests) and per-file test ignores
- Makefile: add lint / format / lint-fix targets
- .github/workflows/ci.yml: add lint job (ruff check + format --check)
- .gitignore: whitelist ruff.toml
Also fix every finding the new gate surfaced so CI is green:
- 109+ auto-fixes from ruff --fix (import sorting, simplifications,
unused vars, re.I aliases, etc.)
- explicit check=False on all subprocess.run calls (PLW1510)
- harden sitemap XML parsing: reject DTD/entity declarations (S314)
- replace hardcoded /tmp paths in tests with tmp_path (S108)
- narrow/annotate intentional bare excepts (S110/S112), bind loop vars
in closures (B023), raise ... from None (B904), strict= for zip (B905)
- ruff format applied across service/ and tests/
Verified: ruff check + ruff format --check pass; 287 tests pass, 1 skip.
Bump accelerate to 1.14.0, controlnet-aux to 0.0.10, and safetensors to
0.8.0 in requirements-ctrlregen.txt. Verified end-to-end against the pinned
noai-watermark commit b642ae45 (import + CUDA inference) and via Docker build.
Pin huggingface_hub to 0.24.0: 0.26.0+ removed the cached_download symbol
that diffusers 0.27.2 still imports, so unpinned installs resolve 0.36.2 and
the backend fails to import. Refresh the Dockerfile.ctrlregen base-image
comment to drop the now-stale safetensors 0.4.3 mention.
* feat: split skill from service, add HTTP API and Docker distribution
The agent skill (skills/remove-ai-marks/) is now a code-free remote client:
all implementation moved to service/scripts/ and runs behind a stdlib HTTP
service (server.py) with /health, /capabilities, /inspect, /clean and a
dynamically generated OpenAPI 3.0.3 spec at /openapi.json.
- Move scripts/ and the backend Dockerfiles under service/
- server.py: JSON/base64 HTTP entrypoint with size caps, binary guard,
atomic writes, loopback default, optional bearer auth
- Core Dockerfile (exiftool/qpdf/c2patool preinstalled) and a GHCR publish
workflow for the core/markllm/markdiffusion images
- compose.yaml (wr-* services, harness/heavy profiles) + compose-check.sh
to validate the running stack (exit code only)
- Fix markllm image build (tokenizers 0.22.2, CPU-only torch) and ctrlregen
build (python:3.11 base for the 2023-era research pins)
- Fix markllm/markdiffusion harness images missing common.py at runtime
* docs: add .env.example and service configuration guide
* fix: disable chain-of-thought for openai-compatible Layer B rewrites
deepseek-v4-flash is a reasoning model: a one-line paraphrase burned 9,894
reasoning tokens (~100s) and hit the default timeout. Send
reasoning_effort=none by default for the openai-compatible backend
(--reasoning-effort / WATERMARKS_REWRITE_REASONING_EFFORT; 'off' omits the
parameter), cutting the same rewrite to ~1s / 12 tokens. Tested end-to-end
against api.deepseek.com.
* fix: sanitize client-supplied filename in HTTP service
CodeQL 'uncontrolled data in path expression' (server.py): a name like
'../../x' flowed into Path(tmpdir) / name, letting an upload escape the
request temp dir on write. Sanitize name to its basename in _decode_input
(_safe_name) and refuse any joined path whose parent is not the tmpdir at
the write sites (_tmp_path). Tests cover traversal names.
* chore: gitignore .env (contains local rewrite credentials)
* chore: deny-by-default gitignore and dockerignore; document compose env config
.gitignore and service/.dockerignore now exclude everything by default and
explicitly allow only what is publishable/needed: tracked source, docs,
tests, .github, and (for images) the service/scripts/ tree that every
Dockerfile COPYs. Root .dockerignore documents that all builds use service/
as context. README Configuration section now covers .env setup for docker
compose, host-side export for CLI runs, and the full variable table.
Unify the file-to-pipeline classification that was duplicated across
inspect_file.py, clean_file.py and audit_lib.py into one format_dispatch
module. Adding a format now means touching a single extension table and
sniff path; audit_lib no longer imports classify from a CLI entry point.
* feat: optional MarkLLM text-watermark verification harness
Add an optional external backend wrapping THU-BPM/MarkLLM (Apache-2.0)
so a specific statistical text-watermark scheme (KGW / SynthID-Text)
can be verified before/after a Layer B rewrite.
- detect_text_watermark.py: detect/watermark subcommands, external
checkout at a pinned commit, exit codes 0/1/2/3, --json
- rewrite_text.py --markllm-scheme: before/after detection around the
rewrite, reports a `cleared` flag; never fails the rewrite when the
backend is unavailable
- setup_markllm.sh + requirements-markllm.txt (pinned deps) +
Dockerfile.markllm + Makefile bootstrap/smoke/docker targets
- tests/test_markllm_detect.py: 16 mock-based cases (no torch in CI)
- Docs: verification-harness caveat (same-config-only, not a
vendor-detector oracle) in README, SKILL.md, removal-matrix, vendor-notes
* chore: tidy merged Unreleased changelog list
* security: harden the MarkLLM harness (offline, caps, supply-chain)
Addresses the PR security review:
- detect_text_watermark.py: --offline loads the scoring model from the HF
cache only (local_files_only + HF_HUB_OFFLINE, no remote code), and the
algorithm config is capped at 1 MiB so a crafted huge file is refused
before either this script or upstream reads it into memory
- rewrite_text.py: WATERMARKS_MARKLLM_RLIMIT_AS (env, POSIX) optionally
applies RLIMIT_AS to the MarkLLM subprocess; off by default because
torch/CUDA needs large address spaces
- Dockerfile.markllm: drop the unpinned torch install (it is pinned in
requirements-markllm.txt) and verify the cloned upstream commit SHA
- tests: offline flag, config-too-large, and preexec/rlimit cases
- docs: hardening knobs in README + SKILL.md; changelog updated
* fix: SECURITY.md only references the private advisories channel
* fix: preserve load-bearing script invisibles, strip PUA in Layer A (#38)
Mongolian FVS (U+180B-180D), Khmer inherent vowels (U+17B4/17B5) and
Hangul fillers (U+115F/1160) are orthography, not hidden marks, when
they follow a base from their own script. STRIP_CODEPOINTS removed them
unconditionally, corrupting real text (FVS glyph selection, Khmer
phonemic vowels, Hangul partial-syllable composition).
Mirror the existing #28 glue machinery: keep them after a same-script
base, still strip them when floating between unrelated characters, and
honor --strip-emoji-glue for paranoid mode. Add them to _is_glue so
chained FVS stay bound to their base.
Private-use codepoints (U+E000-F8FF and the supplementary PUA planes)
render as nothing portable and carry no agreed meaning, making them a
serviceable hiding place; they are Co, so they fell outside the Cf
strip path. Strip them by default with a new 'private_use' inspect kind.
Refs #38.
Mongolian FVS (U+180B-180D), Khmer inherent vowels (U+17B4/17B5) and
Hangul fillers (U+115F/1160) are orthography, not hidden marks, when
they follow a base from their own script. STRIP_CODEPOINTS removed them
unconditionally, corrupting real text (FVS glyph selection, Khmer
phonemic vowels, Hangul partial-syllable composition).
Mirror the existing #28 glue machinery: keep them after a same-script
base, still strip them when floating between unrelated characters, and
honor --strip-emoji-glue for paranoid mode. Add them to _is_glue so
chained FVS stay bound to their base.
Private-use codepoints (U+E000-F8FF and the supplementary PUA planes)
render as nothing portable and carry no agreed meaning, making them a
serviceable hiding place; they are Co, so they fell outside the Cf
strip path. Strip them by default with a new 'private_use' inspect kind.
Refs #38.
Add the resolved path at the top of every human report (File: line) and
include path in JSON output for all kinds, so batch inspection via find
can attribute hits to a file. Closes#31.
The runner prefers .venv/Scripts/python.exe on Windows, but the test
hardcoded the POSIX .venv/bin/python layout, so it fell back to
sys.executable and failed the Windows CI leg.
Wires a standalone clean_ctrlregen.py adapter plus setup_ctrlregen.sh bootstrap, Dockerfile, Makefile targets, and clean_image.py --remove-pixel ctrlregen. The backend is cloned at a pinned commit and never bundled (noai-watermark ships no LICENSE file). Includes mock-based tests and docs with research references.
- classify findings as confirmed/probable/informational/likely_false_positive
- expose confidence in text/image/container JSON and human reports
- add audit_dir.py and audit_website.py for aggregate reports
- document the confidence taxonomy and audit commands in SKILL.md
Closes#13
- DOCX: scan only docProps/customXml parts, not the visible body (#14)
- Text Layer A: preserve emoji VS16/ZWJ after an emoji base; add paranoid flag (#22)
- HTML: treat CMS generator tags as informational, not AI metadata (#13)
- PDF: exclude stream payloads from AI-marker byte scan (#13)
- Add unsupported/best-effort notes to inspect reports
* fix: make subprocess and symlink tests Windows-compatible
- Gate preexec_fn (POSIX-only) so optional tools can run on Windows
- Skip symlink tests when the platform denies the privilege
- Add a Windows CI leg and a Windows CLI smoke run
* fix: use UTF-8 stdio and fail Windows smoke on first error
- Reconfigure stdin/stdout/stderr to UTF-8 so Windows redirected streams
stop raising UnicodeEncodeError on invisible Unicode characters
- Chain the Windows smoke commands so a failing script fails the step
- dependabot.yml: weekly updates for GitHub Actions (rotates the SHA pins),
root pip deps (requirements-dev.txt), and the scorer requirements
(compatibility-coupled to the pinned upstream commit — human re-verify
noted), monthly for the digest-pinned Docker base image.
- CODEOWNERS: keep the default maintainer ownership and make ownership of
/.github and SECURITY.md explicit.
