* fix: pin vendored Cursor-skill text engine to the service copy
The engine vendored into skills/clean-user-facing-text/ had silently
drifted behind service/scripts/text_unicode.py: it still blanket-
stripped legitimate RTL directional marks and isolates (corrupting
mixed RTL/LTR prose) and stripped emoji variation selectors after
arrow and symbol bases, because it never received the preservable-bidi
and emoji-base updates. Nothing in the suite compared the two copies.
Replace the vendored engine with an exact copy of the service one and
add a byte-equality test so any future engine change must land in both
files in the same commit, plus a behavioural regression test for the
RTL and arrow cases through the vendored CLI. The CLI wrappers remain
deliberately different (text-only skill: no stylometry, no
--strip-bidi).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: re-pin vendored engine after merging main and correct bidi comment
Merging main into the PR branch kept main's ruff-formatted service
engine (#103) but the PR's vendored copy pinned in 519c6db, so the two
drifted apart again and test_vendored_text_unicode_is_identical_to_service_engine
failed. Re-sync the vendored copy to the current service engine so the
byte-for-byte pin holds.
Also correct the _PRESERVABLE_BIDI_CPS comment: paired LRE/RLE embeddings
are preserved via _valid_bidi_embedding_indices, so only overrides and
unpaired embeddings remain destructive by default.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Introduce Ruff (pinned at 0.16.3) as the project linter + formatter and
enforce it in CI:
- requirements-dev.txt: pin ruff==0.16.3 (exact pins, no drift)
- ruff.toml: line-length 100, target py312; rule set E/F/W/I/UP/B/SIM/RUF/PLW/S
with deliberate ignores (E501 for content strings, S603 for safe_arg
subprocess calls, S101 asserts in tests) and per-file test ignores
- Makefile: add lint / format / lint-fix targets
- .github/workflows/ci.yml: add lint job (ruff check + format --check)
- .gitignore: whitelist ruff.toml
Also fix every finding the new gate surfaced so CI is green:
- 109+ auto-fixes from ruff --fix (import sorting, simplifications,
unused vars, re.I aliases, etc.)
- explicit check=False on all subprocess.run calls (PLW1510)
- harden sitemap XML parsing: reject DTD/entity declarations (S314)
- replace hardcoded /tmp paths in tests with tmp_path (S108)
- narrow/annotate intentional bare excepts (S110/S112), bind loop vars
in closures (B023), raise ... from None (B904), strict= for zip (B905)
- ruff format applied across service/ and tests/
Verified: ruff check + ruff format --check pass; 287 tests pass, 1 skip.
* fix(inspect): run Layer A scan on markdown/html containers
inspect_container() never scanned the text body, so a .md or .html file
carrying invisible Unicode was reported suspicious=false while
clean_container() went on to strip it. Identical bytes gave opposite
verdicts depending on the file extension.
Scan Layer A for exactly the formats clean_container() scrubs (markdown,
html) so inspect predicts clean. Decode with surrogateescape to match
clean's decoding. Expose the count as suspicious_total, the same key
TextInspectReport uses, so the HTTP server's suspicious flag and the
inspect_file CLI exit code pick it up without special-casing.
* docs: changelog entry for the container Layer A inspect fix
* fix(audit): drop duplicate Layer A scan for markdown/html containers
inspect_container() now scans the body for markdown/html, so
audit_lib.scan_file's own Layer A scan produced the same findings twice
(once as 'layer-a:' from the container report, once as 'layer-a [kind]')
and double-counted them in the aggregate. Keep the stylometry check,
which still needs the decoded body text.
---------
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* fix: keep the SynthID scorer's --json stdout pure
The reverse-SynthID upstream prints progress ('CodebookV4 loaded: ...')
straight to stdout. image_meta.py parses the scorer's stdout with
json.loads, so the leak corrupts every score payload into
{'available': False, 'error': 'bad scorer JSON: ...'}.
Redirect stdout to stderr around the upstream calls so --json owns
stdout. Regression test drives the real script against a deliberately
noisy stub upstream (with a stub cv2, so it runs without OpenCV).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: probe realpath -m support instead of realpath presence
macOS ships BSD realpath, which exists but has no -m flag, so
'command -v realpath' takes the GNU branch and both setup bootstraps
abort on the first path they normalize. Probe the flag itself; BSD
systems fall through to the portable pwd fallback already in place.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Adriel <adriel@Adriels-MacBook-Pro-2026.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Keep valid RTL controls, script joiners, variation sequences, and emoji structure while still removing malformed carriers, with regression coverage for each case.
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* feat: add lightweight Cursor and Codex text skill
Package the text-only workflow with safe cross-platform installation, optional persistent instructions, and focused tests so users can adopt it without the media tooling.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: keep the lightweight skill independently reviewable
Avoid coupling the packaging PR to the separate Unicode safety change so either pull request can merge on its own.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: force UTF-8 for subprocess stdin in Windows CI
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* feat: split skill from service, add HTTP API and Docker distribution
The agent skill (skills/remove-ai-marks/) is now a code-free remote client:
all implementation moved to service/scripts/ and runs behind a stdlib HTTP
service (server.py) with /health, /capabilities, /inspect, /clean and a
dynamically generated OpenAPI 3.0.3 spec at /openapi.json.
- Move scripts/ and the backend Dockerfiles under service/
- server.py: JSON/base64 HTTP entrypoint with size caps, binary guard,
atomic writes, loopback default, optional bearer auth
- Core Dockerfile (exiftool/qpdf/c2patool preinstalled) and a GHCR publish
workflow for the core/markllm/markdiffusion images
- compose.yaml (wr-* services, harness/heavy profiles) + compose-check.sh
to validate the running stack (exit code only)
- Fix markllm image build (tokenizers 0.22.2, CPU-only torch) and ctrlregen
build (python:3.11 base for the 2023-era research pins)
- Fix markllm/markdiffusion harness images missing common.py at runtime
* docs: add .env.example and service configuration guide
* fix: disable chain-of-thought for openai-compatible Layer B rewrites
deepseek-v4-flash is a reasoning model: a one-line paraphrase burned 9,894
reasoning tokens (~100s) and hit the default timeout. Send
reasoning_effort=none by default for the openai-compatible backend
(--reasoning-effort / WATERMARKS_REWRITE_REASONING_EFFORT; 'off' omits the
parameter), cutting the same rewrite to ~1s / 12 tokens. Tested end-to-end
against api.deepseek.com.
* fix: sanitize client-supplied filename in HTTP service
CodeQL 'uncontrolled data in path expression' (server.py): a name like
'../../x' flowed into Path(tmpdir) / name, letting an upload escape the
request temp dir on write. Sanitize name to its basename in _decode_input
(_safe_name) and refuse any joined path whose parent is not the tmpdir at
the write sites (_tmp_path). Tests cover traversal names.
* chore: gitignore .env (contains local rewrite credentials)
* chore: deny-by-default gitignore and dockerignore; document compose env config
.gitignore and service/.dockerignore now exclude everything by default and
explicitly allow only what is publishable/needed: tracked source, docs,
tests, .github, and (for images) the service/scripts/ tree that every
Dockerfile COPYs. Root .dockerignore documents that all builds use service/
as context. README Configuration section now covers .env setup for docker
compose, host-side export for CLI runs, and the full variable table.
* feat: optional MarkLLM text-watermark verification harness
Add an optional external backend wrapping THU-BPM/MarkLLM (Apache-2.0)
so a specific statistical text-watermark scheme (KGW / SynthID-Text)
can be verified before/after a Layer B rewrite.
- detect_text_watermark.py: detect/watermark subcommands, external
checkout at a pinned commit, exit codes 0/1/2/3, --json
- rewrite_text.py --markllm-scheme: before/after detection around the
rewrite, reports a `cleared` flag; never fails the rewrite when the
backend is unavailable
- setup_markllm.sh + requirements-markllm.txt (pinned deps) +
Dockerfile.markllm + Makefile bootstrap/smoke/docker targets
- tests/test_markllm_detect.py: 16 mock-based cases (no torch in CI)
- Docs: verification-harness caveat (same-config-only, not a
vendor-detector oracle) in README, SKILL.md, removal-matrix, vendor-notes
* chore: tidy merged Unreleased changelog list
* security: harden the MarkLLM harness (offline, caps, supply-chain)
Addresses the PR security review:
- detect_text_watermark.py: --offline loads the scoring model from the HF
cache only (local_files_only + HF_HUB_OFFLINE, no remote code), and the
algorithm config is capped at 1 MiB so a crafted huge file is refused
before either this script or upstream reads it into memory
- rewrite_text.py: WATERMARKS_MARKLLM_RLIMIT_AS (env, POSIX) optionally
applies RLIMIT_AS to the MarkLLM subprocess; off by default because
torch/CUDA needs large address spaces
- Dockerfile.markllm: drop the unpinned torch install (it is pinned in
requirements-markllm.txt) and verify the cloned upstream commit SHA
- tests: offline flag, config-too-large, and preexec/rlimit cases
- docs: hardening knobs in README + SKILL.md; changelog updated
* fix: harden website audit against SSRF and gzip bombs
* fix: pin TLS minimum version to TLSv1.2 for pinned HTTPS connections
---------
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
`exiftool -all=` on a PDF appends a %BeginExifToolUpdate block that frees the
Info object and drops /Info from the trailer, but the original metadata bytes
stay in the file verbatim - exiftool itself reverts the edit with
`-PDF-update:all=`. clean_pdf reported "exiftool -all= (rc=0)" and nothing
else, so a PDF still carrying `/Producer (Claude Opus)` in plain bytes read as
cleaned. The output file gets *larger*, which is the tell.
Follow the exiftool pass with `qpdf --linearize`, which re-serializes the
document from its object graph and so drops the now-unreferenced objects.
Guarded on which("qpdf"): without qpdf the behaviour is unchanged except for
an explicit warning that the metadata bytes remain recoverable.
Tests cover the no-qpdf warning, the rebuild replacing the exiftool output,
qpdf's rc=3 "succeeded with warnings", a failed rewrite leaving the exiftool
output in place, and an end-to-end check (skipped without the real tools)
asserting no readable metadata survives in the output bytes. That last one
fails on the leak itself when the rewrite is reverted.
Co-authored-by: uniendodimensionesgm <153029553+uniendodimensionesgm@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* fix: SECURITY.md only references the private advisories channel
* fix: preserve load-bearing script invisibles, strip PUA in Layer A (#38)
Mongolian FVS (U+180B-180D), Khmer inherent vowels (U+17B4/17B5) and
Hangul fillers (U+115F/1160) are orthography, not hidden marks, when
they follow a base from their own script. STRIP_CODEPOINTS removed them
unconditionally, corrupting real text (FVS glyph selection, Khmer
phonemic vowels, Hangul partial-syllable composition).
Mirror the existing #28 glue machinery: keep them after a same-script
base, still strip them when floating between unrelated characters, and
honor --strip-emoji-glue for paranoid mode. Add them to _is_glue so
chained FVS stay bound to their base.
Private-use codepoints (U+E000-F8FF and the supplementary PUA planes)
render as nothing portable and carry no agreed meaning, making them a
serviceable hiding place; they are Co, so they fell outside the Cf
strip path. Strip them by default with a new 'private_use' inspect kind.
Refs #38.
Add the resolved path at the top of every human report (File: line) and
include path in JSON output for all kinds, so batch inspection via find
can attribute hits to a file. Closes#31.
clean_markdown() had two bugs in the same loop:
1. It crashed with IndexError on any frontmatter containing a blank
line, because line[0] was evaluated on an empty string.
2. It kept the nested children of a dropped top-level key, so dropping
`model:` left `name: claude-opus` behind and produced invalid YAML,
while still reporting the key as removed.
Rewrite the loop with a `dropping` flag that tracks whether the current
top-level key was dropped, and guard blank/comment lines before
indexing. inspect_markdown() is unchanged (already guards line[0]).
Adds 4 regression tests: blank-line crash, nested-key leak, inspect
round-trip, and comment/list preservation.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Layer A already preserves emoji ZWJ/VS glue, but still stripped other
load-bearing invisibles, silently corrupting real text:
- ZWNJ/ZWJ inside complex scripts (Persian میروم, Devanagari क्ष)
- flag emoji tag sequences (🏴 -> 🏴)
- orthographic Arabic/Syriac Cf marks (U+0600, U+06DD, U+070F, ...)
Extend the existing _decide()/glue machinery: keep ZWNJ/ZWJ when a
neighbour is a complex-script letter, keep tag chars after an emoji
base, and allowlist the orthographic Cf codepoints. The same characters
between plain ASCII stay carriers and are still stripped.
--strip-emoji-glue continues to strip all of them (paranoid mode).
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
clean_file.py and clean_image.py computed the failure exit code inside
the human-output branch, so `--json` always exited 0 even when the clean
left C2PA/AI signals behind. A script gating on `clean_file --json` would
treat a still-marked file as clean.
Move the residual (and degraded-PDF) decision out of the output branch in
both entry points so the exit code is the same regardless of --json.
Human output is unchanged; degraded best-effort PDF copies stay
non-failures.
Adds tests asserting json and human modes return the same exit code for
residual, clean, and degraded cases.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* Refuse binary input in the text-only tools
inspect_text.py, clean_text.py and rewrite_text.py accept any path and decode
it with errors="surrogateescape". Pointed at a .docx - a zip - they walk
deflate-compressed bytes and report whatever codepoints fall out of them. The
counts look like findings but track the compression, not the content: in one
sample set a document with nothing hidden in its text reported 12 "suspicious"
characters, while another with 54 real no-break spaces reported 11, none of
which were the no-break spaces.
clean_text.py is worse than misleading. It writes the mangled decode back, so
`clean_text.py report.docx` reports "removed=1" and silently corrupts the
document - the output still passes zipfile.is_zipfile() because the end-of-
central-directory record survives, but reading a member raises.
common.looks_binary() now sniffs magic numbers plus a control-byte ratio, and
guard_binary() refuses with a message naming the tool that does handle the
format. The ratio test is deliberately conservative so text in encodings other
than UTF-8 keeps working, and every entry point takes --force-text to override.
clean_file.py gets the same check on the branch where classify() falls back to
"text" for unrecognised bytes.
Adds tests covering magic-number and heuristic detection, the override, refusal
without writing or backing up, and that clean_file.py still routes a .docx to
the container path.
* Address review: backup ordering, stdin sniff, router advice
Three fixes from the review on #24.
clean_file.py sniffed after --in-place had already taken the backup, so
`clean_file.py --in-place mystery.bin` left a mystery.bin.bak sidecar behind
before exiting 2 — for a file the run never touches, and exactly what
clean_text.py avoids. The sniff now runs before backup_path(). The same hole
applied to `--as text` on a .docx, which bypasses classify() entirely.
The stdin path decoded before sniffing, which made detection depend on the
console codec. It was worse than codec drift: the text layer also translates
newlines, so PNG's `\x89PNG\r\n\x1a\n` arrived as `\x89PNG\n\x1a\n` and the
magic number never matched — the file was refused by the NUL-byte heuristic
instead, and would have sailed through had it lacked NULs. _read_stdin_capped
now reads sys.stdin.buffer and guards the raw octets, matching the file path,
with a text fallback for a replaced stdin.
guard_binary always advised "Use inspect_file.py / clean_file.py", which is
circular when the caller is one of them and classify() has already ruled out
every known container. The advice is now a parameter: the text-only scripts
keep the pointer to the routers, and the routers say the bytes match no
supported format and point at --force-text / --as.
Adds tests for the backup ordering (both --in-place paths), the advice split,
and stdin magic that is not ASCII, across default, cp1252 and latin-1 stdio
codecs — the previous stdin test piped a ZIP, whose "PK" header is ASCII and
survives any of them.
---------
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
The runner prefers .venv/Scripts/python.exe on Windows, but the test
hardcoded the POSIX .venv/bin/python layout, so it fell back to
sys.executable and failed the Windows CI leg.
Wires a standalone clean_ctrlregen.py adapter plus setup_ctrlregen.sh bootstrap, Dockerfile, Makefile targets, and clean_image.py --remove-pixel ctrlregen. The backend is cloned at a pinned commit and never bundled (noai-watermark ships no LICENSE file). Includes mock-based tests and docs with research references.
- classify findings as confirmed/probable/informational/likely_false_positive
- expose confidence in text/image/container JSON and human reports
- add audit_dir.py and audit_website.py for aggregate reports
- document the confidence taxonomy and audit commands in SKILL.md
Closes#13
- DOCX: scan only docProps/customXml parts, not the visible body (#14)
- Text Layer A: preserve emoji VS16/ZWJ after an emoji base; add paranoid flag (#22)
- HTML: treat CMS generator tags as informational, not AI metadata (#13)
- PDF: exclude stream payloads from AI-marker byte scan (#13)
- Add unsupported/best-effort notes to inspect reports
* fix: make subprocess and symlink tests Windows-compatible
- Gate preexec_fn (POSIX-only) so optional tools can run on Windows
- Skip symlink tests when the platform denies the privilege
- Add a Windows CI leg and a Windows CLI smoke run
* fix: use UTF-8 stdio and fail Windows smoke on first error
- Reconfigure stdin/stdout/stderr to UTF-8 so Windows redirected streams
stop raising UnicodeEncodeError on invisible Unicode characters
- Chain the Windows smoke commands so a failing script fails the step
safe_write_bytes() calls os.fchmod() to restore the umask-default mode
after mkstemp() creates the temp file as 0600. os.fchmod does not exist
on Windows, so every write path raised:
AttributeError: module 'os' has no attribute 'fchmod'
That broke clean_text.py, clean_file.py, clean_image.py and
rewrite_text.py whenever an output file was requested.
Windows has no POSIX mode bits to restore, so the call is skipped there.
The atomic-write guarantee is unaffected: the symlink refusal and
os.replace() are what protect the destination, not the chmod.
Adds a regression test that deletes os.fchmod via monkeypatch, so the
case is covered on POSIX CI too.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Security hardening for the cleaner/inspector entry points:
- common.safe_arg(): prefix './' to paths starting with '-', so a crafted
filename (e.g. '-@argfile') cannot be interpreted as an option by
exiftool/c2patool. Applied to every exiftool/c2patool invocation in
image_meta.py and container_meta.py (clean_pdf).
- container_meta.py: add MAX_ZIP_DECOMPRESSED_BYTES (512 MiB), checked via
ZipInfo.file_size before decompression in inspect_docx/clean_docx/
inspect_odt/clean_odt, rejecting zip bombs early.
- clean_file.py / inspect_file.py: refuse inputs larger than
WATERMARKS_MAX_INPUT_BYTES (default 1 GiB).
- Fix a latent NameError in inspect_docx: the finding used the undefined
variable `name` instead of info.filename, crashing on the very DOCX files
this tool exists for.
Tests: tests/test_security_hardening.py covers safe_arg, the zip budget
guard, and the DOCX AI-markers path.
Add Makefile, pytest.ini, synthetic fixtures (MD/HTML/SVG), PDF degraded
clean test, and make smoke/test targets so the multi-vendor plan is fully closed.
Rename remove-claude-marks to remove-ai-marks, add container metadata
support (SVG/PDF/DOCX/ODT/HTML/MD), Layer B rewrite hook, unified file
CLI, and multi-vendor documentation for the first public release.
Publishable agent skill with Layer A Unicode scrub, agent-orchestrated
Layer B rewrite guidance, PNG/JPEG C2PA/metadata strip scripts, docs,
and tests.
