* docs: add watermarks-remover-web to Ecosystem (#77)
* docs: note AVIF and HEIC in the watermarks-remover-web entry
The client now ports the ISOBMFF paths from image_meta.py as well, so the
entry's format list was out of date.
* docs: rename the web UI entry to unmark-web and keep it higher-level
Renames the project per the maintainer's request, and drops the
references to internal module names, the client's proxy script and the
format list, which were the parts most likely to drift.
---------
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Add stdlib-only detection, inspection, and cleaning for four more formats.
- BMP: locate the pixel payload via the DIB header and strip trailing
non-image metadata (the only place non-standard BMP metadata can live),
rewriting the file-size field.
- GIF: drop comment and XMP application extensions while preserving
NETSCAPE2.0 looping, ICC, graphic-control, and image blocks.
- TIFF (classic + BigTIFF): walk IFD chains and drop XMP/EXIF/GPS/IPTC/
Photoshop/MakerNote tags, zeroing orphaned payloads while keeping
strip/tile offsets valid.
- EPUB: scrub OPF package metadata and XHTML meta/JSON-LD, clean embedded
raster/SVG media, apply Layer A to XHTML body text, and pass OCF-encrypted
parts through untouched.
All four route through format_dispatch, so the unified CLIs, the HTTP
service, and the audits pick them up automatically.
* fix: pin vendored Cursor-skill text engine to the service copy
The engine vendored into skills/clean-user-facing-text/ had silently
drifted behind service/scripts/text_unicode.py: it still blanket-
stripped legitimate RTL directional marks and isolates (corrupting
mixed RTL/LTR prose) and stripped emoji variation selectors after
arrow and symbol bases, because it never received the preservable-bidi
and emoji-base updates. Nothing in the suite compared the two copies.
Replace the vendored engine with an exact copy of the service one and
add a byte-equality test so any future engine change must land in both
files in the same commit, plus a behavioural regression test for the
RTL and arrow cases through the vendored CLI. The CLI wrappers remain
deliberately different (text-only skill: no stylometry, no
--strip-bidi).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: re-pin vendored engine after merging main and correct bidi comment
Merging main into the PR branch kept main's ruff-formatted service
engine (#103) but the PR's vendored copy pinned in 519c6db, so the two
drifted apart again and test_vendored_text_unicode_is_identical_to_service_engine
failed. Re-sync the vendored copy to the current service engine so the
byte-for-byte pin holds.
Also correct the _PRESERVABLE_BIDI_CPS comment: paired LRE/RLE embeddings
are preserved via _valid_bidi_embedding_indices, so only overrides and
unpaired embeddings remain destructive by default.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
The bullet 'Best-effort: prose was rewritten...' uses a full sentence with
passive verb while adjacent bullets use noun phrases. Removed 'was' to
match the parallel structure of the other bullets.
Audit metadata:
- Category: grammar
- Severity: minor
- File: skills/clean-user-facing-text/SKILL.md, line 61
- Fingerprint: a9c80f5f5afbfbe7
Co-authored-by: pi <pi@m2.local>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
resolve_device() auto-selects 'mps' whenever torch reports it available.
Both schemes this harness can load build a device-bound RNG in their
constructor:
KGW.__init__ -> torch.Generator(device=config.device) (kgw.py)
SynthID.__init__ -> torch.Generator(device=config.device) (synthid.py)
torch.Generator supports only the 'cpu' and 'cuda' device types, so on
Apple Silicon a resolved 'mps' device raises
RuntimeError: Device type MPS is not supported for torch.Generator() api
and both the `detect` and `watermark` subcommands crash out of the box on
any Mac where torch reports MPS available.
Drop the mps branch from the auto path so `--device auto` resolves to cpu
on CUDA-less machines. `--device mps` is still honored as an explicit
override (resolve_device returns any non-'auto' value unchanged), so this
changes only the automatic default, not the ability to opt in.
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Introduce Ruff (pinned at 0.16.3) as the project linter + formatter and
enforce it in CI:
- requirements-dev.txt: pin ruff==0.16.3 (exact pins, no drift)
- ruff.toml: line-length 100, target py312; rule set E/F/W/I/UP/B/SIM/RUF/PLW/S
with deliberate ignores (E501 for content strings, S603 for safe_arg
subprocess calls, S101 asserts in tests) and per-file test ignores
- Makefile: add lint / format / lint-fix targets
- .github/workflows/ci.yml: add lint job (ruff check + format --check)
- .gitignore: whitelist ruff.toml
Also fix every finding the new gate surfaced so CI is green:
- 109+ auto-fixes from ruff --fix (import sorting, simplifications,
unused vars, re.I aliases, etc.)
- explicit check=False on all subprocess.run calls (PLW1510)
- harden sitemap XML parsing: reject DTD/entity declarations (S314)
- replace hardcoded /tmp paths in tests with tmp_path (S108)
- narrow/annotate intentional bare excepts (S110/S112), bind loop vars
in closures (B023), raise ... from None (B904), strict= for zip (B905)
- ruff format applied across service/ and tests/
Verified: ruff check + ruff format --check pass; 287 tests pass, 1 skip.
* fix(inspect): run Layer A scan on markdown/html containers
inspect_container() never scanned the text body, so a .md or .html file
carrying invisible Unicode was reported suspicious=false while
clean_container() went on to strip it. Identical bytes gave opposite
verdicts depending on the file extension.
Scan Layer A for exactly the formats clean_container() scrubs (markdown,
html) so inspect predicts clean. Decode with surrogateescape to match
clean's decoding. Expose the count as suspicious_total, the same key
TextInspectReport uses, so the HTTP server's suspicious flag and the
inspect_file CLI exit code pick it up without special-casing.
* docs: changelog entry for the container Layer A inspect fix
* fix(audit): drop duplicate Layer A scan for markdown/html containers
inspect_container() now scans the body for markdown/html, so
audit_lib.scan_file's own Layer A scan produced the same findings twice
(once as 'layer-a:' from the container report, once as 'layer-a [kind]')
and double-counted them in the aggregate. Keep the stylometry check,
which still needs the decoded body text.
---------
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* fix: keep the SynthID scorer's --json stdout pure
The reverse-SynthID upstream prints progress ('CodebookV4 loaded: ...')
straight to stdout. image_meta.py parses the scorer's stdout with
json.loads, so the leak corrupts every score payload into
{'available': False, 'error': 'bad scorer JSON: ...'}.
Redirect stdout to stderr around the upstream calls so --json owns
stdout. Regression test drives the real script against a deliberately
noisy stub upstream (with a stub cv2, so it runs without OpenCV).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: probe realpath -m support instead of realpath presence
macOS ships BSD realpath, which exists but has no -m flag, so
'command -v realpath' takes the GNU branch and both setup bootstraps
abort on the first path they normalize. Probe the flag itself; BSD
systems fall through to the portable pwd fallback already in place.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Adriel <adriel@Adriels-MacBook-Pro-2026.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Keep valid RTL controls, script joiners, variation sequences, and emoji structure while still removing malformed carriers, with regression coverage for each case.
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* feat: add lightweight Cursor and Codex text skill
Package the text-only workflow with safe cross-platform installation, optional persistent instructions, and focused tests so users can adopt it without the media tooling.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: keep the lightweight skill independently reviewable
Avoid coupling the packaging PR to the separate Unicode safety change so either pull request can merge on its own.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: force UTF-8 for subprocess stdin in Windows CI
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Bump accelerate to 1.14.0, controlnet-aux to 0.0.10, and safetensors to
0.8.0 in requirements-ctrlregen.txt. Verified end-to-end against the pinned
noai-watermark commit b642ae45 (import + CUDA inference) and via Docker build.
Pin huggingface_hub to 0.24.0: 0.26.0+ removed the cached_download symbol
that diffusers 0.27.2 still imports, so unpinned installs resolve 0.36.2 and
the backend fails to import. Refresh the Dockerfile.ctrlregen base-image
comment to drop the now-stale safetensors 0.4.3 mention.
* feat: split skill from service, add HTTP API and Docker distribution
The agent skill (skills/remove-ai-marks/) is now a code-free remote client:
all implementation moved to service/scripts/ and runs behind a stdlib HTTP
service (server.py) with /health, /capabilities, /inspect, /clean and a
dynamically generated OpenAPI 3.0.3 spec at /openapi.json.
- Move scripts/ and the backend Dockerfiles under service/
- server.py: JSON/base64 HTTP entrypoint with size caps, binary guard,
atomic writes, loopback default, optional bearer auth
- Core Dockerfile (exiftool/qpdf/c2patool preinstalled) and a GHCR publish
workflow for the core/markllm/markdiffusion images
- compose.yaml (wr-* services, harness/heavy profiles) + compose-check.sh
to validate the running stack (exit code only)
- Fix markllm image build (tokenizers 0.22.2, CPU-only torch) and ctrlregen
build (python:3.11 base for the 2023-era research pins)
- Fix markllm/markdiffusion harness images missing common.py at runtime
* docs: add .env.example and service configuration guide
* fix: disable chain-of-thought for openai-compatible Layer B rewrites
deepseek-v4-flash is a reasoning model: a one-line paraphrase burned 9,894
reasoning tokens (~100s) and hit the default timeout. Send
reasoning_effort=none by default for the openai-compatible backend
(--reasoning-effort / WATERMARKS_REWRITE_REASONING_EFFORT; 'off' omits the
parameter), cutting the same rewrite to ~1s / 12 tokens. Tested end-to-end
against api.deepseek.com.
* fix: sanitize client-supplied filename in HTTP service
CodeQL 'uncontrolled data in path expression' (server.py): a name like
'../../x' flowed into Path(tmpdir) / name, letting an upload escape the
request temp dir on write. Sanitize name to its basename in _decode_input
(_safe_name) and refuse any joined path whose parent is not the tmpdir at
the write sites (_tmp_path). Tests cover traversal names.
* chore: gitignore .env (contains local rewrite credentials)
* chore: deny-by-default gitignore and dockerignore; document compose env config
.gitignore and service/.dockerignore now exclude everything by default and
explicitly allow only what is publishable/needed: tracked source, docs,
tests, .github, and (for images) the service/scripts/ tree that every
Dockerfile COPYs. Root .dockerignore documents that all builds use service/
as context. README Configuration section now covers .env setup for docker
compose, host-side export for CLI runs, and the full variable table.
Unify the file-to-pipeline classification that was duplicated across
inspect_file.py, clean_file.py and audit_lib.py into one format_dispatch
module. Adding a format now means touching a single extension table and
sniff path; audit_lib no longer imports classify from a CLI entry point.
* feat: optional MarkLLM text-watermark verification harness
Add an optional external backend wrapping THU-BPM/MarkLLM (Apache-2.0)
so a specific statistical text-watermark scheme (KGW / SynthID-Text)
can be verified before/after a Layer B rewrite.
- detect_text_watermark.py: detect/watermark subcommands, external
checkout at a pinned commit, exit codes 0/1/2/3, --json
- rewrite_text.py --markllm-scheme: before/after detection around the
rewrite, reports a `cleared` flag; never fails the rewrite when the
backend is unavailable
- setup_markllm.sh + requirements-markllm.txt (pinned deps) +
Dockerfile.markllm + Makefile bootstrap/smoke/docker targets
- tests/test_markllm_detect.py: 16 mock-based cases (no torch in CI)
- Docs: verification-harness caveat (same-config-only, not a
vendor-detector oracle) in README, SKILL.md, removal-matrix, vendor-notes
* chore: tidy merged Unreleased changelog list
* security: harden the MarkLLM harness (offline, caps, supply-chain)
Addresses the PR security review:
- detect_text_watermark.py: --offline loads the scoring model from the HF
cache only (local_files_only + HF_HUB_OFFLINE, no remote code), and the
algorithm config is capped at 1 MiB so a crafted huge file is refused
before either this script or upstream reads it into memory
- rewrite_text.py: WATERMARKS_MARKLLM_RLIMIT_AS (env, POSIX) optionally
applies RLIMIT_AS to the MarkLLM subprocess; off by default because
torch/CUDA needs large address spaces
- Dockerfile.markllm: drop the unpinned torch install (it is pinned in
requirements-markllm.txt) and verify the cloned upstream commit SHA
- tests: offline flag, config-too-large, and preexec/rlimit cases
- docs: hardening knobs in README + SKILL.md; changelog updated
* fix: harden website audit against SSRF and gzip bombs
* fix: pin TLS minimum version to TLSv1.2 for pinned HTTPS connections
---------
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
setup_ctrlregen.sh and setup_synthid.sh are POSIX-only: they invoke
"$DIR/.venv/bin/python", a layout `python -m venv` never produces on Windows,
so both abort right after the clone. image_meta.py already resolves
`.venv\Scripts\python.exe` under `os.name == "nt"`, so only the bootstrap was
missing.
setup_ctrlregen.ps1 and setup_synthid.ps1 mirror the shell scripts: same
pinned upstream commits and SHA verification, same sparse-checkout paths, same
`pip==26.2.1` pin, same flags (as -Dir / -Ref / -Full / -Python).
One deliberate difference in setup_ctrlregen.ps1. The shell script derives the
torch wheel tag from the CUDA version nvidia-smi prints, but that is the
maximum the *driver* supports, not what to install - drivers are backward
compatible, so a cu126 wheel runs fine on a 13.0 driver. Worse, the naive tag
breaks on older cards: cu128+ and CUDA 13 dropped Maxwell/Pascal/Volta, so a
GTX 1080 Ti (sm_61) would get a cu130 torch with no kernels for it and fail at
runtime with "no kernel image is available for execution". Checking that the
index URL exists does not catch this - every index returns 200. The port keys
off the GPU's compute capability instead, forces cu126 below 7.5, and verifies
the installed wheel with torch.cuda.get_arch_list().
Both scripts parse clean; the CUDA detection was checked against a real
compute-capability 6.1 GPU. The .sh scripts are left untouched.
Co-authored-by: uniendodimensionesgm <153029553+uniendodimensionesgm@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
`exiftool -all=` on a PDF appends a %BeginExifToolUpdate block that frees the
Info object and drops /Info from the trailer, but the original metadata bytes
stay in the file verbatim - exiftool itself reverts the edit with
`-PDF-update:all=`. clean_pdf reported "exiftool -all= (rc=0)" and nothing
else, so a PDF still carrying `/Producer (Claude Opus)` in plain bytes read as
cleaned. The output file gets *larger*, which is the tell.
Follow the exiftool pass with `qpdf --linearize`, which re-serializes the
document from its object graph and so drops the now-unreferenced objects.
Guarded on which("qpdf"): without qpdf the behaviour is unchanged except for
an explicit warning that the metadata bytes remain recoverable.
Tests cover the no-qpdf warning, the rebuild replacing the exiftool output,
qpdf's rc=3 "succeeded with warnings", a failed rewrite leaving the exiftool
output in place, and an end-to-end check (skipped without the real tools)
asserting no readable metadata survives in the output bytes. That last one
fails on the leak itself when the rewrite is reverted.
Co-authored-by: uniendodimensionesgm <153029553+uniendodimensionesgm@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* fix: SECURITY.md only references the private advisories channel
* fix: preserve load-bearing script invisibles, strip PUA in Layer A (#38)
Mongolian FVS (U+180B-180D), Khmer inherent vowels (U+17B4/17B5) and
Hangul fillers (U+115F/1160) are orthography, not hidden marks, when
they follow a base from their own script. STRIP_CODEPOINTS removed them
unconditionally, corrupting real text (FVS glyph selection, Khmer
phonemic vowels, Hangul partial-syllable composition).
Mirror the existing #28 glue machinery: keep them after a same-script
base, still strip them when floating between unrelated characters, and
honor --strip-emoji-glue for paranoid mode. Add them to _is_glue so
chained FVS stay bound to their base.
Private-use codepoints (U+E000-F8FF and the supplementary PUA planes)
render as nothing portable and carry no agreed meaning, making them a
serviceable hiding place; they are Co, so they fell outside the Cf
strip path. Strip them by default with a new 'private_use' inspect kind.
Refs #38.
Mongolian FVS (U+180B-180D), Khmer inherent vowels (U+17B4/17B5) and
Hangul fillers (U+115F/1160) are orthography, not hidden marks, when
they follow a base from their own script. STRIP_CODEPOINTS removed them
unconditionally, corrupting real text (FVS glyph selection, Khmer
phonemic vowels, Hangul partial-syllable composition).
Mirror the existing #28 glue machinery: keep them after a same-script
base, still strip them when floating between unrelated characters, and
honor --strip-emoji-glue for paranoid mode. Add them to _is_glue so
chained FVS stay bound to their base.
Private-use codepoints (U+E000-F8FF and the supplementary PUA planes)
render as nothing portable and carry no agreed meaning, making them a
serviceable hiding place; they are Co, so they fell outside the Cf
strip path. Strip them by default with a new 'private_use' inspect kind.
Refs #38.
Add the resolved path at the top of every human report (File: line) and
include path in JSON output for all kinds, so batch inspection via find
can attribute hits to a file. Closes#31.
clean_markdown() had two bugs in the same loop:
1. It crashed with IndexError on any frontmatter containing a blank
line, because line[0] was evaluated on an empty string.
2. It kept the nested children of a dropped top-level key, so dropping
`model:` left `name: claude-opus` behind and produced invalid YAML,
while still reporting the key as removed.
Rewrite the loop with a `dropping` flag that tracks whether the current
top-level key was dropped, and guard blank/comment lines before
indexing. inspect_markdown() is unchanged (already guards line[0]).
Adds 4 regression tests: blank-line crash, nested-key leak, inspect
round-trip, and comment/list preservation.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
