mirror of https://github.com/guillaumemeyer/watermarks-remover.git synced 2026-08-22 13:11:57 +02:00

Files

T

Guillaume Meyer (The Opinionated Man)andGitHub d5f4f03f85 feat: add reproducible SynthID-text removal benchmark (#145 )

* feat: add reproducible SynthID-text removal benchmark

bench_synthid_text.py orchestrates the existing Layer B machinery into a
controlled, shareable experiment: generate watermarked + unwatermarked
samples with the MarkLLM SynthID scheme, run removal variants (strength x
candidates) plus controls (no-removal, Layer-A-only, optional re-stamp),
and report clear rate, score suppression, quality, and cost (tokens,
wall time, USD) with a clears-per-MTok efficiency ratio.

Emits report.md / results.json / results.csv with the exact reproduction
command and pinned commits; optional Gemini official-detector tier when
WATERMARKS_GEMINI_API_KEY is set. Mock-based tests, no torch in CI.

* docs: add README section on running the SynthID-text benchmark

Explains what LLM performs the Layer B rewrite (an external model configured
via WATERMARKS_REWRITE_* env vars or --rewrite-* flags; MarkLLM's opt-1.3b is
only the watermark generator/detector) and how to run a benchmark with Ollama
or an OpenAI-compatible endpoint, plus the non-origin-model re-stamp caveat.

* fix: honor WATERMARKS_REWRITE_ALLOW_REMOTE in the SynthID-text benchmark

The --rewrite-allow-remote flag now defaults from the env var (matching
rewrite_text.py and the other WATERMARKS_REWRITE_* settings), so a
non-loopback rewrite endpoint works after sourcing .env without an extra
flag.

* fix: MarkLLM sparse checkout and deps for the SynthID harness

- setup_markllm.sh sparse-checkout omitted '/visualize/', which
  watermark/base.py imports at module load — every scheme (incl. SynthID)
  failed with 'No module named visualize' during generation/detection.
- requirements-markllm.txt omitted scikit-learn, imported by the SynthID
  detector (watermark/synthid/detector_bayesian_torch.py).

Both broke the MarkLLM harness at runtime; the benchmark's sanity gate then
excluded every sample, producing empty per-variant results.

* fix: drop 4 GiB RLIMIT_AS on benchmark subprocesses

_run_cmd applied the common child RLIMIT_AS (default 4 GiB) via
subprocess_preexec_fn to every MarkLLM/rewrite child. torch needs a much
larger address space: CUDA init failed with 'out of memory' at
cudaGetDeviceCount and the 5.2 GB fp32 opt-1.3b could not load, so every
sample was excluded at generation. text_detectors.py already applies no
address-space cap to MarkLLM by default; the benchmark now matches.

* perf: keep MarkLLM resident via a serve worker (624 cold starts -> 1)

The benchmark spawned a fresh torch + opt-1.3b process per operation
(~60-90s each); a full run needs ~624 of them. detect_text_watermark.py
gains a 'serve' mode (JSON-lines over stdin/stdout, ready handshake) that
loads the model once; bench_synthid_text.py uses it via MarkLLMWorker with
automatic fallback to one-shot subprocesses (--no-worker to force).
Turns ~8h runs into ~40-60min.

* perf: skip per-candidate Gemini detections in rewrite subprocess

* feat: run the SynthID-text benchmark from the wr-markllm compose service

- Dockerfile.markllm: add '/visualize/' to the sparse checkout (same fix as
  setup_markllm.sh) and COPY the benchmark + rewrite scripts (stdlib-only).
- compose.yaml: wr-markllm gets the WATERMARKS_REWRITE_* and
  WATERMARKS_GEMINI_* env wiring, a bench-out volume for --out-dir, and a
  read-only mount of the bundled corpus (build context is service/, so the
  corpus cannot be COPY'd).
- docs: docker compose run example.

Note: the image ships CPU torch by design, so the container path is for
portability/CI; GPU runs use the host setup_markllm.sh venv.

* feat: per-sample progress logging in the benchmark

The persistent worker returns samples in-memory, so nothing is written
until the end of a run — runs looked stuck. eprint a [gen i/N] line per
generated sample and a [removal] summary per sample.

* chore: migrate Gemini config to gemini-3.6-flash; document SynthID-text retirement

Google retired SynthID text watermarking on the Generative Language API
(Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK
is rejected on current 3.x models (confirmed by Google AI staff). Migrate
the default detection model to gemini-3.6-flash, document the retirement
in vendor-notes.md and the benchmark report caveat, and keep the detector
seam fail-soft until a vendor endpoint (e.g. Vertex AI) returns.

* feat: remove gemini-synthid-text detector (Google retired text watermarking)

Google removed SynthID text watermarking from the Generative Language API
(Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK
is rejected on current 3.x models, so the vendor detector had nothing to
detect. Remove GeminiSynthIDTextDetector and its wiring:

- text_detectors.py: drop the Gemini class, HTTP helpers, and constants;
  keep MarkLLM + Claude seams (registry now markllm + claude-text).
- server.py / rewrite_text.py: per-candidate detection now triggers on
  --markllm-scheme only.
- bench_synthid_text.py: remove the Gemini tier (before/after, report
  table, --no-gemini flag); report caveat notes the retirement.
- configs/docs: drop WATERMARKS_GEMINI_* from .env.example / compose /
  README / SKILL.md / vendor-notes.md; keep the retirement note.
- tests: gemini tests removed or converted to MarkLLM (mocked subprocess).
- Dockerfile.markllm: parameterize BASE_IMAGE + TORCH_INDEX_URL so a GPU/
  arm64 image can be built (used for the --gpus all benchmark run).

* fix: harden notes aggregation against non-string notes

A run completed all samples but crashed at the final aggregate step with
'cannot use list as a set element' when a row's notes contained a
non-string value. Filter notes to strings (aggregate + CSV) and add a
regression test.

* perf: let the rewrite subprocess reuse the resident MarkLLM worker

The rewrite subprocess (rewrite_text.py) ran its own before/after MarkLLM
detects, each a ~20s torch+model cold start (~12 per sample = ~5min of the
~6min/sample runtime). Now:

- detect_text_watermark.py serve gains --port N: a loopback TCP JSON-lines
  listener (default -1 = off) sharing the resident model, with a lock so
  stdin and socket requests never run the model concurrently.
- text_detectors.MarkLLMTextDetector checks WATERMARKS_MARKLLM_PORT and
  does a fast loopback detect when a worker is up, falling back to the
  one-shot subprocess otherwise.
- The benchmark worker publishes its port via that env var, so the rewrite
  subprocess inherits it and its detects hit the resident model.

Turns ~6 min/sample into ~1-2 min; a full run drops from ~2h to ~40-50min.
Tests: loopback-client + fallback + env-publish coverage.

* chore: add benchmark-smoke.sh / benchmark-full.sh wrappers

Simple host wrappers: source .env, default MARKLLM_DIR to ~/MarkLLM, use a
repo-local HF cache by default, and run bench_synthid_text.py with a quick
(2 docs, 1 seed, paraphrase:1) or full (8 docs x 3 seeds, three variants,
re-stamp control) configuration. OUT_DIR overrides the output location.

2026-08-18 18:13:56 -07:00

5.7 KiB

Raw Blame History

Vendor notes (public / class-level)

This skill targets mark classes, not reverse-engineered private detectors. Details below are from public docs and the research literature. Algorithms may change.

Industry two-layer model (context)

Product and regulatory guidance often frames AI disclosure as:

C2PA Content Credentials — signed, hard-bound metadata (easy to strip; what this skill removes).
Imperceptible watermark (SynthID-class) — survives strip/re-upload; includes soft binding that can re-attach a remote C2PA manifest.

See: Institute of AI PM — C2PA and SynthID guide (SB 942 / EU AI Act Art. 50 framing). This project only implements the hard-bound / Unicode / rewrite side of that stack.

Anthropic / Claude

Embedded text watermarks at model level (imperceptible; survive copy-paste). Public description matches statistical token-sampling class, not only Unicode.
C2PA Content Credentials on supported files (e.g. PNG, JPEG, SVG).
Models launched on/after 2026-08-02: marking at launch; older models in transition; worldwide.
Detection APIs for third parties: described as forthcoming.
Caveats: mark ⇒ may have been processed by Claude; no mark ≠ human-only; proofreading can stamp human text.

Skill mapping: Layer A (Unicode hygiene) + Layer B (rewrite) + container/image C2PA strip.

Source: How Claude marks AI-generated content.

Google Gemini / SynthID-Text

Nature 2024 paper: Scalable watermarking for identifying large language model outputs (SynthID-Text).
Generative watermarking: modifies next-token sampling (Tournament sampling); detection uses a scoring function + key; no need for the LLM at detect time.
Paper also taxonomizes:
- Edit-based (Unicode / synonym rules) → our Layer A (+ Layer B for synonyms)
- Data-driven / backdoor (trigger phrases) → out of scope
- Generative (sampling) → Layer B best-effort
Productionized in Gemini-scale systems; open research code exists, but production keys are not public — this skill does not ship a SynthID detector.
Retired from the API (Aug 2026): Google confirmed the Generative Language API no longer watermarks text output and DETECT_TEXT_WATERMARK is rejected on current (3.x) models; "native text watermarking is not planned at the moment" (Google AI forum). The gemini-synthid-text detector was removed for this reason; a vendor detector can be re-added (e.g. via Vertex AI) if Google exposes detection again.
Optional external verification harness: THU-BPM/MarkLLM (Apache-2.0) reimplements SynthID-Text among other schemes with configurable keys; wired as detect_text_watermark.py / rewrite_text.py --markllm-scheme. Same-config-only — it verifies a mark you generated under a known config, not Google's production keying.
Current frontier production watermarks are token-by-token (streaming constraint); paragraph-level robust methods (SemStamp / PostMark) are not deployed yet, which keeps paraphrase-class attacks effective today.
Optional external reference: aloshdenny/reverse-SynthID provides a reverse-engineered pixel-domain scorer. It is not bundled here, is best-effort, and is under a non-commercial Research License; it is not the official Google detector.
Optional pixel-domain removal: mertizci/noai-watermark's CtrlRegen profile is wired through clean_image.py --remove-pixel ctrlregen / clean_ctrlregen.py. It is not bundled (no LICENSE file → all-rights-reserved), and no local detector certifies the result; the official Google check is the final authority. For Tree-Ring-class marks, the optional MarkDiffusion harness (markdiffusion_harness.py, Apache-2.0) adds a same-scheme detector and a blind-regeneration removal engine (--remove-pixel diffusion) — see references/markdiffusion.md.

Skill mapping: same Layer B rewrite attacks (paraphrase / back-translate / structural) used in the literature against sampling watermarks.

OpenAI / ChatGPT

Public provenance often surfaces as labels, C2PA / Content Credentials on some media exports, and product UI disclosure — not a fully public text-sampling watermark spec comparable to SynthID-Text.
Treat file metadata / C2PA as in-scope when present; treat any unpublished text watermark as the same statistical class → Layer B only, best-effort.
Do not invent algorithm claims.

Skill mapping: container/image metadata strip + Layer A/B on text.

Open-weight / open-LLM (Kirchenbauer-style)

Classic green-list / red-list sampling bias (Kirchenbauer et al.) and variants.
Detectable with the key and tokenizer; removal still relies on heavy paraphrase or regeneration.
Optional external harness: MarkLLM (detect_text_watermark.py --scheme kgw) reproduces KGW detection under a config you control, for controlled before/after experiments.

Skill mapping: Layer B multi-pass; prefer rewrite with a different model family when possible.

Cross-vendor hygiene rule

Suspected origin	Prefer rewrite backend
Claude	Non-Claude (local Ollama, other API)
Gemini	Non-Gemini
OpenAI	Non-OpenAI
Unknown	Local open-weight if available

Prefer local open-weight backends and avoid any known-watermarked vendor, not just the suspected origin. Then re-run Layer A on the rewritten text.

5.7 KiB Raw Blame History