Commit Graph
111 Commits
Author SHA1 Message Date
Guillaume Meyer (The Opinionated Man)andGitHub e4003427f2 docs: add arXiv 2402.14904 reference to README (#144)
* docs: add arXiv 2402.14904 (Watermarking Makes Language Models Radioactive) to README references

* docs: add arXiv 2502.11598 (Can LLM Watermarks Robustly Prevent Unauthorized Knowledge Distillation?) to README references
2026-08-18 06:46:42 -07:00
a2e72ed019 feat: vendor text-watermark detection (Gemini SynthID, Claude seam, MarkLLM) + SynthID image scorer sidecar (#109)
* feat: add vendor text-watermark detection and SynthID image scorer sidecar

Adds Layer B watermark detection as a first-class service capability:

- text_detectors.py: a registry of text-watermark detectors behind one
  interface — Google's official SynthID-text detector via the Gemini API
  (taskType DETECT_TEXT_WATERMARK), a Claude placeholder ready for
  Anthropic's announced detection API, and the MarkLLM research harness
  (KGW / SynthID, same-config-only). Fail-soft: unconfigured or errored
  detectors never block cleaning.
- server.py: new POST /detect endpoint, detect_before / detect_after options
  on /clean (before/after scoring for text and images), an opt-in
  /inspect "detect" flag, and /capabilities gains text_detectors and
  scorers.synthid_http.
- synthid_score_server.py: a stdlib HTTP sidecar for the reverse-SynthID
  scorer, so the published core image never bundles the non-commercial
  upstream code; wired via WATERMARKS_SYNTHID_SCORER_URL.
- score_synthid.py: extract score_file() so the CLI and the sidecar share
  one implementation.
- compose.yaml / Dockerfile.synthid / .env.example: wr-synthid-score sidecar
  service and env wiring.
- README + skill docs, plus tests for the detectors, the /detect endpoint,
  and the image sidecar.

* feat: per-candidate watermark detection for Layer B rewrite candidates

When --candidates N (N > 1) is combined with --markllm-scheme or
WATERMARKS_GEMINI_API_KEY, run every configured text detector from the
text_detectors.py registry on each candidate and report per-candidate
measurements in --json-stats as candidate_scores entries carrying
lexical_divergence, selection_score, selected, and per-detector reports
(is_watermarked, score, threshold where the detector provides one).

Candidate selection stays purely lexical; the detections are observability
for correlating lexical divergence with watermark removal (issue #106).

Converges rewrite_text.py onto the shared detector registry:
- MarkLLMTextDetector gains constructor overrides (scheme, upstream_dir,
  model, timeout) plus the checkout-venv interpreter preference and the
  WATERMARKS_MARKLLM_RLIMIT_AS preexec guard ported from rewrite_text.py;
  the old _markllm_detect / _venv_python / _markllm_preexec helpers are gone.
- run_all_text_detectors() accepts an injected MarkLLM instance and an
  include_markllm switch so CLI flag gating stays intact.
- before/after/cleared semantics unchanged; detection remains fail-soft.

* docs: pin Watermarks in the Sand reference to arXiv v5

* fix: mark only one rewrite candidate as selected (#110)

---------

Co-authored-by: Zhenxin Ai <142008897+ai-kunkun@users.noreply.github.com>
2026-08-17 18:31:22 -07:00
Guillaume Meyer (The Opinionated Man)andGitHub 6df80e77a4 fix: correctness and security hardening of the cleaning scripts (#122) (#126)
* fix: rewrite ODT/EPUB manifests and measure real zip bytes (#122)

Two container correctness/security fixes from issue #122:

- clean_odt dropped marker-bearing parts while leaving their entries in
  META-INF/manifest.xml, so readers flagged the package as damaged. It is
  now two-pass: compute the dropped set, then rewrite the manifest
  attribute-order-independently, and write each part exactly once. The same
  bug class in clean_epub (dropped parts left in the OPF manifest, plus
  dangling spine itemrefs) gets the same two-pass treatment.
- The zip budget trusted ZipInfo.file_size from the archive's own central
  directory, so a crafted DOCX/ODT could declare a tiny size and still
  expand via zf.read. Budgets are now charged on actual decompressed bytes
  via _read_zip_member (streaming, cap enforced mid-read), with the declared
  size kept only as a fast-path pre-reject.

* fix: classify unrecognized bytes as "unknown", not text (#122)

Two classification defects from issue #122:

- format_dispatch.classify_bytes fell back to "text" for any unrecognized
  file, so a binary with valid UTF-8 runs could be decoded and written back
  mangled (corrupted with --in-place) in clean_file auto mode. Unrecognized
  bytes now classify as "unknown"; clean_file refuses them in auto mode
  (exit 2, no write, router advice) and --as text / --force-text are the
  explicit opt-ins. inspect_file reports kind "unknown" (exit 0), audit_lib
  records a non-actionable item, and the HTTP server answers /inspect with
  kind "unknown" but rejects /clean of unknown formats (400).
- classify(path) read the whole file to sniff a header, and only a full read
  could detect zip containers. It now routes known extensions without
  reading, sniffs a 4096-byte header once for images and prefix-based
  containers, and reads the whole file only when the header is a zip local
  header (PK), where the container signature lives in the central directory.

* feat: distinct exit code for partial audits (#122)

audit_dir and audit_website reported success (0) even when some files or
URLs could not be scanned; the exit status was computed only over the items
that succeeded. A scan that is missing items is not a clean scan.

- common.EXIT_PARTIAL = 3, with precedence: partial (3) > actionable (1)
  > clean (0) — an incomplete audit is the more important CI signal.
- audit_dir returns 3 when any file was skipped/failed; audit_website
  returns 3 when any URL failed to fetch or inspect. Both are independent
  of the output format (human/json/sarif already share one return).

* fix: verify the pinned upstream ref on existing checkouts (#122)

setup_ctrlregen.sh/setup_synthid.sh (and their .ps1 twins) only verified
the pinned commit in the fresh-clone branch; an existing checkout at an
unknown or drifted revision was silently reused, defeating the commit pin.

All four scripts now check HEAD against the pinned ref in the
existing-checkout branch too, and repair by fetch + detach checkout
(re-applying the sparse-checkout set), failing hard if the ref cannot be
reached or the re-pin does not land on it.

* docs: unknown-format behavior, audit exit codes, backend isolation (#122)

- README: clean_file no longer auto-cleans unrecognized formats (--as text
  / --force-text are the opt-ins), and the CtrlRegen bootstrap documents the
  isolation expectation for its research-era dependency pins plus the new
  re-pin check on existing checkouts.
- SKILL.md: audit exit codes (0/1/2/3, partial=3) and a note that /clean
  requires a name with a known extension.
- audit_website: document why stdlib ElementTree is used (stdlib-first) and
  that defusedxml is the fallback if that policy changes (DTD rejection stays).
- requirements-ctrlregen.txt: advisory/isolation note for the pinned research
  dependencies.

* test: ODT manifest and EPUB OPF dangling-ref regressions (#122)

- clean_odt: dropped marker-bearing parts remove their META-INF/manifest.xml
  file-entry (attribute-order-independent), exactly one manifest entry, root
  and surviving entries kept, and the manifest is byte-identical when nothing
  is dropped.
- clean_epub: dropped non-content parts lose their <item> entry in the OPF
  manifest, so the book no longer references removed members.
2026-08-17 17:15:35 -07:00
Guillaume Meyer (The Opinionated Man)andGitHub d4dd4735e9 fix: Windows setup_ctrlregen.ps1 torch install (probe published indices, keep CUDA torch) (#124)
Three independent failure modes from #117:

- $ErrorActionPreference 'Stop' + 2>$null on a native command aborts the
  script on torch's harmless stderr warnings (e.g. "Failed to initialize
  NumPy" when torch is installed before numpy). Run the probes through a
  new Invoke-NativeQuiet helper that lowers EAP to 'Continue' for the
  block and restores it afterwards.
- The wheel index tag was derived from the driver's CUDA version, e.g.
  cu131 for a 13.1 driver, which does not exist (HTTP 403) and silently
  fell back to the default index, i.e. the CPU build on Windows. Probe
  the published indices and pick the highest one <= driver that answers
  HTTP 200; cu126 is still forced below compute capability 7.5.
- Installing torch alone let requirements-ctrlregen.txt resolve torchvision
  from PyPI, and torchvision pins an exact torch, so pip replaced the +cu
  build with a +cpu one while the script still exited 0. Install torch AND
  torchvision together from the chosen index, and verify after the
  requirements install that torch.cuda.is_available() is true - if a GPU
  was detected but torch ends up CPU-only, warn loudly and exit non-zero.

Also add a CI step (windows-latest, pwsh) that parses the setup .ps1
scripts and asserts the post-install CUDA verification survives.

Fixes #117
2026-08-17 16:48:26 -07:00
Guillaume Meyer (The Opinionated Man)andGitHub b3de4c0085 fix: detect AI generator product names in PNG text metadata (#120) (#125) 2026-08-17 16:45:23 -07:00
Guillaume Meyer (The Opinionated Man)andGitHub a430827827 fix: prefer checkout venv for SynthID scorer subprocess (#115) (#123) 2026-08-17 16:34:51 -07:00
00596a7270 Handle malformed DOCX/ODT beyond BadZipFile in container inspectors (#92)
* Handle malformed DOCX/ODT beyond BadZipFile in container inspectors

A corrupt, truncated, encrypted, or unsupported-compression container
raises more than BadZipFile (NotImplementedError, RuntimeError, EOFError,
OSError, ValueError, zlib.error), so inspect_docx / inspect_odt and
detect_container_format now catch the whole family and degrade to a clear
finding.

The zip-bomb rejection in _check_zip_budget now raises a dedicated
ZipBudgetExceeded so it keeps propagating out of the inspectors, matching
clean_docx / clean_odt, instead of being reported as an unparseable zip.

* fix: satisfy ruff lint and format checks

---------

Co-authored-by: eeshsaxena <chinmaymd72@gmail.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-17 16:16:38 -07:00
cc7444eed2 fix(docs): use portable base64 encoding in curl examples (#119)
`base64 -w0 FILE` is a GNU coreutils form. On macOS/BSD it fails with
`base64: invalid argument FILE` and prints usage, so every documented
`/inspect` and `/clean` example is unrunnable on a Mac host.

This matters more than a normal docs typo because SKILL.md instructs the
agent to call the service via these exact commands and explicitly forbids
falling back to local cleaning when the service call fails. A macOS agent
therefore stops at its first request instead of degrading gracefully.

Replace all 7 occurrences (2 in README.md, 5 in SKILL.md) with the POSIX
form, which reads from stdin and unwraps any line breaks:

    base64 < FILE | tr -d '\n'

Verified byte-identical output on both implementations:
  - BSD/macOS base64 (macOS 15, Darwin 24.6.0)
  - GNU coreutils base64 (`gbase64`)

Also verified the rewritten commands verbatim against a live
`server.py` on 127.0.0.1:8765 — `/inspect` and `/clean` both return
`ok: true` and correctly strip U+200B / U+00AD.

Docs-only; no code paths changed, so no unit tests added.
`python3 -m pytest -q` and `ruff check`/`format --check` pass.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-17 16:11:06 -07:00
5e43d53fc3 fix: validate clean option types (#111)
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-17 16:08:31 -07:00
Johar Ali ShaikandGitHub 07be21e54a docs: add Windows auto-start guide via Task Scheduler (#113)
* docs: add Windows auto-start guide via Task Scheduler

* docs: fix markdown escaping, remove admin requirement, use path placeholder, link from README
2026-08-17 15:10:59 -07:00
Guillaume Meyer (The Opinionated Man)andGitHub c2ac8eeef3 docs: discourage watermarks-remover look-alike names in Ecosystem entries (#108) 2026-08-16 19:56:23 -07:00
84a1e9e7a9 docs: add unmark-web to Ecosystem (#77) (#86)
* docs: add watermarks-remover-web to Ecosystem (#77)

* docs: note AVIF and HEIC in the watermarks-remover-web entry

The client now ports the ISOBMFF paths from image_meta.py as well, so the
entry's format list was out of date.

* docs: rename the web UI entry to unmark-web and keep it higher-level

Renames the project per the maintainer's request, and drops the
references to internal module names, the client's proxy script and the
format list, which were the parts most likely to drift.

---------

Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-16 19:51:07 -07:00
Guillaume Meyer (The Opinionated Man)andGitHub ed8c8bc732 feat: add BMP, GIF, TIFF, and EPUB format support (#107)
Add stdlib-only detection, inspection, and cleaning for four more formats.

- BMP: locate the pixel payload via the DIB header and strip trailing
  non-image metadata (the only place non-standard BMP metadata can live),
  rewriting the file-size field.
- GIF: drop comment and XMP application extensions while preserving
  NETSCAPE2.0 looping, ICC, graphic-control, and image blocks.
- TIFF (classic + BigTIFF): walk IFD chains and drop XMP/EXIF/GPS/IPTC/
  Photoshop/MakerNote tags, zeroing orphaned payloads while keeping
  strip/tile offsets valid.
- EPUB: scrub OPF package metadata and XHTML meta/JSON-LD, clean embedded
  raster/SVG media, apply Layer A to XHTML body text, and pass OCF-encrypted
  parts through untouched.

All four route through format_dispatch, so the unified CLIs, the HTTP
service, and the audits pick them up automatically.
2026-08-16 19:47:11 -07:00
Guillaume Meyer (The Opinionated Man)andGitHub 1794036a71 docs: remove Unreleased section from README changelog (#105) 2026-08-16 18:49:29 -07:00
c1d1e5ac85 fix: pin vendored Cursor-skill text engine to the service copy (#96)
* fix: pin vendored Cursor-skill text engine to the service copy

The engine vendored into skills/clean-user-facing-text/ had silently
drifted behind service/scripts/text_unicode.py: it still blanket-
stripped legitimate RTL directional marks and isolates (corrupting
mixed RTL/LTR prose) and stripped emoji variation selectors after
arrow and symbol bases, because it never received the preservable-bidi
and emoji-base updates. Nothing in the suite compared the two copies.

Replace the vendored engine with an exact copy of the service one and
add a byte-equality test so any future engine change must land in both
files in the same commit, plus a behavioural regression test for the
RTL and arrow cases through the vendored CLI. The CLI wrappers remain
deliberately different (text-only skill: no stylometry, no
--strip-bidi).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: re-pin vendored engine after merging main and correct bidi comment

Merging main into the PR branch kept main's ruff-formatted service
engine (#103) but the PR's vendored copy pinned in 519c6db, so the two
drifted apart again and test_vendored_text_unicode_is_identical_to_service_engine
failed. Re-sync the vendored copy to the current service engine so the
byte-for-byte pin holds.

Also correct the _PRESERVABLE_BIDI_CPS comment: paired LRE/RLE embeddings
are preserved via _valid_bidi_embedding_indices, so only overrides and
unpaired embeddings remain destructive by default.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-16 18:44:18 -07:00
215715719d docs: fix grammar - parallel structure inconsistency in reporting list (#104)
The bullet 'Best-effort: prose was rewritten...' uses a full sentence with
passive verb while adjacent bullets use noun phrases. Removed 'was' to
match the parallel structure of the other bullets.

Audit metadata:
- Category: grammar
- Severity: minor
- File: skills/clean-user-facing-text/SKILL.md, line 61
- Fingerprint: a9c80f5f5afbfbe7

Co-authored-by: pi <pi@m2.local>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-16 18:25:21 -07:00
e3eb602128 fix: never auto-select MPS device for text watermark detection (#99)
resolve_device() auto-selects 'mps' whenever torch reports it available.
Both schemes this harness can load build a device-bound RNG in their
constructor:

  KGW.__init__      -> torch.Generator(device=config.device)   (kgw.py)
  SynthID.__init__  -> torch.Generator(device=config.device)   (synthid.py)

torch.Generator supports only the 'cpu' and 'cuda' device types, so on
Apple Silicon a resolved 'mps' device raises

  RuntimeError: Device type MPS is not supported for torch.Generator() api

and both the `detect` and `watermark` subcommands crash out of the box on
any Mac where torch reports MPS available.

Drop the mps branch from the auto path so `--device auto` resolves to cpu
on CUDA-less machines. `--device mps` is still honored as an explicit
override (resolve_device returns any non-'auto' value unchanged), so this
changes only the automatic default, not the ability to opt in.

Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-16 18:19:59 -07:00
27c1ce864e feat: add multi-worker concurrency and SARIF 2.1.0 export to audit_dir.py (#101) (#102)
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-16 18:14:41 -07:00
fd1a64b47e feat: add native stdlib XLSX and PPTX container metadata, text, and embedded media scrubbing (#91) (#100)
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-16 18:12:48 -07:00
Guillaume Meyer (The Opinionated Man) a65ecb67a6 fix: never auto-select MPS device for text watermark detection (#99) 2026-08-16 17:54:31 -07:00
Guillaume Meyer (The Opinionated Man) c14b5863f9 feat: add multi-worker concurrency and SARIF 2.1.0 export to audit_dir.py (#101) (#102) 2026-08-16 17:43:09 -07:00
Guillaume Meyer (The Opinionated Man) b49fe4e9fe feat: add native stdlib XLSX and PPTX container metadata, text, and embedded media scrubbing (#91) (#100) 2026-08-16 17:43:09 -07:00
Guillaume Meyer (The Opinionated Man)andGitHub 723627716c chore: add Ruff linting and formatting with CI enforcement (#103)
Introduce Ruff (pinned at 0.16.3) as the project linter + formatter and
enforce it in CI:

- requirements-dev.txt: pin ruff==0.16.3 (exact pins, no drift)
- ruff.toml: line-length 100, target py312; rule set E/F/W/I/UP/B/SIM/RUF/PLW/S
  with deliberate ignores (E501 for content strings, S603 for safe_arg
  subprocess calls, S101 asserts in tests) and per-file test ignores
- Makefile: add lint / format / lint-fix targets
- .github/workflows/ci.yml: add lint job (ruff check + format --check)
- .gitignore: whitelist ruff.toml

Also fix every finding the new gate surfaced so CI is green:
- 109+ auto-fixes from ruff --fix (import sorting, simplifications,
  unused vars, re.I aliases, etc.)
- explicit check=False on all subprocess.run calls (PLW1510)
- harden sitemap XML parsing: reject DTD/entity declarations (S314)
- replace hardcoded /tmp paths in tests with tmp_path (S108)
- narrow/annotate intentional bare excepts (S110/S112), bind loop vars
  in closures (B023), raise ... from None (B904), strict= for zip (B905)
- ruff format applied across service/ and tests/

Verified: ruff check + ruff format --check pass; 287 tests pass, 1 skip.
2026-08-16 17:32:40 -07:00
Poorvith M PandGitHub 47a44e6ca8 feat: recursively inspect and clean embedded raster data URIs in SVGs, HTML, and Markdown (#87) (#88) 2026-08-16 14:13:08 -07:00
Poorvith M PandGitHub fcebf53358 feat: add native stdlib AVIF and HEIC metadata and C2PA stripping (#84) (#85) 2026-08-15 17:23:21 -07:00
Guillaume Meyer (The Opinionated Man)andGitHub 737eaa327c fix: always empty DOCX docProps provenance fields (#76) (#83) 2026-08-15 15:32:12 -07:00
Guillaume Meyer (The Opinionated Man)andGitHub 6750f88d01 fix: keep --in-place and -o on the same format detection (#75) (#82) 2026-08-15 15:28:53 -07:00
Guillaume Meyer (The Opinionated Man)andGitHub e85656d996 fix: run Layer A over DOCX/ODT body text (#74) (#81) 2026-08-15 15:23:00 -07:00
Guillaume Meyer (The Opinionated Man)andGitHub e3ca353efb fix: prune dangling DOCX relationships after customXml removal (#73) (#80) 2026-08-15 15:01:21 -07:00
334b807a9f fix(inspect): run Layer A scan on markdown/HTML containers (#79)
* fix(inspect): run Layer A scan on markdown/html containers

inspect_container() never scanned the text body, so a .md or .html file
carrying invisible Unicode was reported suspicious=false while
clean_container() went on to strip it. Identical bytes gave opposite
verdicts depending on the file extension.

Scan Layer A for exactly the formats clean_container() scrubs (markdown,
html) so inspect predicts clean. Decode with surrogateescape to match
clean's decoding. Expose the count as suspicious_total, the same key
TextInspectReport uses, so the HTTP server's suspicious flag and the
inspect_file CLI exit code pick it up without special-casing.

* docs: changelog entry for the container Layer A inspect fix

* fix(audit): drop duplicate Layer A scan for markdown/html containers

inspect_container() now scans the body for markdown/html, so
audit_lib.scan_file's own Layer A scan produced the same findings twice
(once as 'layer-a:' from the container report, once as 'layer-a [kind]')
and double-counted them in the aggregate. Keep the stylometry check,
which still needs the decoded body text.

---------

Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-15 14:58:35 -07:00
d4ae4287c2 docs: add Ecosystem chapter for third-party projects (#71) (#71)
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-15 14:56:14 -07:00
97c32d58ff fix: macOS portability — pure --json stdout for the SynthID scorer, BSD realpath probe (#70)
* fix: keep the SynthID scorer's --json stdout pure

The reverse-SynthID upstream prints progress ('CodebookV4 loaded: ...')
straight to stdout. image_meta.py parses the scorer's stdout with
json.loads, so the leak corrupts every score payload into
{'available': False, 'error': 'bad scorer JSON: ...'}.

Redirect stdout to stderr around the upstream calls so --json owns
stdout. Regression test drives the real script against a deliberately
noisy stub upstream (with a stub cv2, so it runs without OpenCV).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: probe realpath -m support instead of realpath presence

macOS ships BSD realpath, which exists but has no -m flag, so
'command -v realpath' takes the GNU branch and both setup bootstraps
abort on the first path they normalize. Probe the flag itself; BSD
systems fall through to the portable pwd fallback already in place.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Adriel <adriel@Adriels-MacBook-Pro-2026.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-15 14:53:49 -07:00
Guillaume Meyer (The Opinionated Man) 1eeda9be89 docs: add Ecosystem chapter for third-party projects (#71) 2026-08-15 14:48:37 -07:00
1f95c548ce fix: preserve multilingual Unicode during text cleanup (#34)
Keep valid RTL controls, script joiners, variation sequences, and emoji structure while still removing malformed carriers, with regression coverage for each case.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-15 14:40:27 -07:00
8d79155ad2 feat: add a lightweight Cursor text skill (#35)
* feat: add lightweight Cursor and Codex text skill

Package the text-only workflow with safe cross-platform installation, optional persistent instructions, and focused tests so users can adopt it without the media tooling.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test: keep the lightweight skill independently reviewable

Avoid coupling the packaging PR to the separate Unicode safety change so either pull request can merge on its own.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test: force UTF-8 for subprocess stdin in Windows CI

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-15 14:37:19 -07:00
Poorvith M PandGitHub c7200d20c2 feat: add zero-LLM statistical and stylometric AI text detector for CI and audits (#68) (#69) 2026-08-15 14:20:50 -07:00
Guillaume Meyer (The Opinionated Man) 92f38b10b2 chore(deps): bump CtrlRegen backend deps and pin huggingface_hub (#62, #64, #65)
Bump accelerate to 1.14.0, controlnet-aux to 0.0.10, and safetensors to
0.8.0 in requirements-ctrlregen.txt. Verified end-to-end against the pinned
noai-watermark commit b642ae45 (import + CUDA inference) and via Docker build.

Pin huggingface_hub to 0.24.0: 0.26.0+ removed the cached_download symbol
that diffusers 0.27.2 still imports, so unpinned installs resolve 0.36.2 and
the backend fails to import. Refresh the Dockerfile.ctrlregen base-image
comment to drop the now-stale safetensors 0.4.3 mention.
2026-08-14 16:47:06 -07:00
Guillaume Meyer (The Opinionated Man) dc0ff78f39 docs: finalize v0.5.0 changelog and latest-release pointer v0.5.0 2026-08-14 15:45:46 -07:00
Guillaume Meyer (The Opinionated Man)andGitHub 55d4bdc9fc feat: split skill from service, add HTTP API and Docker distribution (#60)
* feat: split skill from service, add HTTP API and Docker distribution

The agent skill (skills/remove-ai-marks/) is now a code-free remote client:
all implementation moved to service/scripts/ and runs behind a stdlib HTTP
service (server.py) with /health, /capabilities, /inspect, /clean and a
dynamically generated OpenAPI 3.0.3 spec at /openapi.json.

- Move scripts/ and the backend Dockerfiles under service/
- server.py: JSON/base64 HTTP entrypoint with size caps, binary guard,
  atomic writes, loopback default, optional bearer auth
- Core Dockerfile (exiftool/qpdf/c2patool preinstalled) and a GHCR publish
  workflow for the core/markllm/markdiffusion images
- compose.yaml (wr-* services, harness/heavy profiles) + compose-check.sh
  to validate the running stack (exit code only)
- Fix markllm image build (tokenizers 0.22.2, CPU-only torch) and ctrlregen
  build (python:3.11 base for the 2023-era research pins)
- Fix markllm/markdiffusion harness images missing common.py at runtime

* docs: add .env.example and service configuration guide

* fix: disable chain-of-thought for openai-compatible Layer B rewrites

deepseek-v4-flash is a reasoning model: a one-line paraphrase burned 9,894
reasoning tokens (~100s) and hit the default timeout. Send
reasoning_effort=none by default for the openai-compatible backend
(--reasoning-effort / WATERMARKS_REWRITE_REASONING_EFFORT; 'off' omits the
parameter), cutting the same rewrite to ~1s / 12 tokens. Tested end-to-end
against api.deepseek.com.

* fix: sanitize client-supplied filename in HTTP service

CodeQL 'uncontrolled data in path expression' (server.py): a name like
'../../x' flowed into Path(tmpdir) / name, letting an upload escape the
request temp dir on write. Sanitize name to its basename in _decode_input
(_safe_name) and refuse any joined path whose parent is not the tmpdir at
the write sites (_tmp_path). Tests cover traversal names.

* chore: gitignore .env (contains local rewrite credentials)

* chore: deny-by-default gitignore and dockerignore; document compose env config

.gitignore and service/.dockerignore now exclude everything by default and
explicitly allow only what is publishable/needed: tracked source, docs,
tests, .github, and (for images) the service/scripts/ tree that every
Dockerfile COPYs. Root .dockerignore documents that all builds use service/
as context. README Configuration section now covers .env setup for docker
compose, host-side export for CLI runs, and the full variable table.
2026-08-14 15:42:48 -07:00
Guillaume Meyer (The Opinionated Man)andGitHub 61440192af feat: optional MarkDiffusion image-watermark harness and purification removal (#57)
Add an optional harness around THU-BPM/MarkDiffusion (Apache-2.0) for
controlled image-watermark experiments and an alternative pixel-removal engine:

- markdiffusion_harness.py with watermark / detect / purify subcommands for
  nine image schemes (Tree-Ring, Ring-ID, ROBIN, WIND, SFW, Gaussian-Shading,
  GaussMarker, PRC, SEAL); same-scheme/same-model detection only
- clean_image.py --remove-pixel diffusion runs the DiffusionPurification
  regeneration attack (blind regeneration; conservative strength 0.3 default)
- setup_markdiffusion.sh bootstrap (PyPI pin 1.0.2; --checkout editable clone
  at pinned commit), requirements-markdiffusion.txt, Dockerfile.markdiffusion,
  Makefile targets, and mock-based tests (no torch in CI)
- Docs: README section, SKILL.md, removal-matrix.md, vendor-notes.md,
  references/markdiffusion.md
2026-08-14 11:22:40 -07:00
Guillaume Meyer (The Opinionated Man) be1dc9a83d docs: add pull request template 2026-08-14 11:10:07 -07:00
Guillaume Meyer (The Opinionated Man) d64d915737 chore: drop tracked .figlet font from the repo 2026-08-14 11:03:23 -07:00
Guillaume Meyer (The Opinionated Man) f0c558bcfb docs: plan for Docker CLI + API deployment 2026-08-14 11:02:05 -07:00
Guillaume Meyer (The Opinionated Man)andGitHub ac96c6ce34 Merge branch 'main' into refactor/format-dispatch 2026-08-14 10:58:18 -07:00
Guillaume Meyer (The Opinionated Man) 203e57e508 refactor: extract shared format-dispatch module
Unify the file-to-pipeline classification that was duplicated across
inspect_file.py, clean_file.py and audit_lib.py into one format_dispatch
module. Adding a format now means touching a single extension table and
sniff path; audit_lib no longer imports classify from a CLI entry point.
2026-08-14 10:57:18 -07:00
Guillaume Meyer (The Opinionated Man)andGitHub 09e64c4ded feat: optional MarkLLM text-watermark verification harness (#53)
* feat: optional MarkLLM text-watermark verification harness

Add an optional external backend wrapping THU-BPM/MarkLLM (Apache-2.0)
so a specific statistical text-watermark scheme (KGW / SynthID-Text)
can be verified before/after a Layer B rewrite.

- detect_text_watermark.py: detect/watermark subcommands, external
  checkout at a pinned commit, exit codes 0/1/2/3, --json
- rewrite_text.py --markllm-scheme: before/after detection around the
  rewrite, reports a `cleared` flag; never fails the rewrite when the
  backend is unavailable
- setup_markllm.sh + requirements-markllm.txt (pinned deps) +
  Dockerfile.markllm + Makefile bootstrap/smoke/docker targets
- tests/test_markllm_detect.py: 16 mock-based cases (no torch in CI)
- Docs: verification-harness caveat (same-config-only, not a
  vendor-detector oracle) in README, SKILL.md, removal-matrix, vendor-notes

* chore: tidy merged Unreleased changelog list

* security: harden the MarkLLM harness (offline, caps, supply-chain)

Addresses the PR security review:

- detect_text_watermark.py: --offline loads the scoring model from the HF
  cache only (local_files_only + HF_HUB_OFFLINE, no remote code), and the
  algorithm config is capped at 1 MiB so a crafted huge file is refused
  before either this script or upstream reads it into memory
- rewrite_text.py: WATERMARKS_MARKLLM_RLIMIT_AS (env, POSIX) optionally
  applies RLIMIT_AS to the MarkLLM subprocess; off by default because
  torch/CUDA needs large address spaces
- Dockerfile.markllm: drop the unpinned torch install (it is pinned in
  requirements-markllm.txt) and verify the cloned upstream commit SHA
- tests: offline flag, config-too-large, and preexec/rlimit cases
- docs: hardening knobs in README + SKILL.md; changelog updated
2026-08-14 09:56:16 -07:00
1b9b94ac5c fix: harden website audit against SSRF and gzip bombs (#49)
* fix: harden website audit against SSRF and gzip bombs

* fix: pin TLS minimum version to TLSv1.2 for pinned HTTPS connections

---------

Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-14 09:25:37 -07:00
4b77c3f4ab fix: preserve mixed-case CMS generator meta tags (#42)
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-14 09:07:55 -07:00
7ef8e446e7 feat: PowerShell ports of the setup bootstraps for Windows (#40)
setup_ctrlregen.sh and setup_synthid.sh are POSIX-only: they invoke
"$DIR/.venv/bin/python", a layout `python -m venv` never produces on Windows,
so both abort right after the clone. image_meta.py already resolves
`.venv\Scripts\python.exe` under `os.name == "nt"`, so only the bootstrap was
missing.

setup_ctrlregen.ps1 and setup_synthid.ps1 mirror the shell scripts: same
pinned upstream commits and SHA verification, same sparse-checkout paths, same
`pip==26.2.1` pin, same flags (as -Dir / -Ref / -Full / -Python).

One deliberate difference in setup_ctrlregen.ps1. The shell script derives the
torch wheel tag from the CUDA version nvidia-smi prints, but that is the
maximum the *driver* supports, not what to install - drivers are backward
compatible, so a cu126 wheel runs fine on a 13.0 driver. Worse, the naive tag
breaks on older cards: cu128+ and CUDA 13 dropped Maxwell/Pascal/Volta, so a
GTX 1080 Ti (sm_61) would get a cu130 torch with no kernels for it and fail at
runtime with "no kernel image is available for execution". Checking that the
index URL exists does not catch this - every index returns 200. The port keys
off the GPU's compute capability instead, forces cu126 below 7.5, and verifies
the installed wheel with torch.cuda.get_arch_list().

Both scripts parse clean; the CUDA detection was checked against a real
compute-capability 6.1 GPU. The .sh scripts are left untouched.

Co-authored-by: uniendodimensionesgm <153029553+uniendodimensionesgm@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-14 09:04:33 -07:00
1878f37022 feat: support WebP provenance metadata (#37)
Agent: Codex

Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
2026-08-14 08:59:33 -07:00