External/watermarks-remover
1
0
Fork 0
mirror of https://github.com/guillaumemeyer/watermarks-remover.git synced 2026-08-22 13:11:57 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat: iterative detection-guided Layer B rewriting (--candidates x --max-loops) (#153)

Browse Source 
* feat: add reproducible SynthID-text removal benchmark

bench_synthid_text.py orchestrates the existing Layer B machinery into a
controlled, shareable experiment: generate watermarked + unwatermarked
samples with the MarkLLM SynthID scheme, run removal variants (strength x
candidates) plus controls (no-removal, Layer-A-only, optional re-stamp),
and report clear rate, score suppression, quality, and cost (tokens,
wall time, USD) with a clears-per-MTok efficiency ratio.

Emits report.md / results.json / results.csv with the exact reproduction
command and pinned commits; optional Gemini official-detector tier when
WATERMARKS_GEMINI_API_KEY is set. Mock-based tests, no torch in CI.

* docs: add README section on running the SynthID-text benchmark

Explains what LLM performs the Layer B rewrite (an external model configured
via WATERMARKS_REWRITE_* env vars or --rewrite-* flags; MarkLLM's opt-1.3b is
only the watermark generator/detector) and how to run a benchmark with Ollama
or an OpenAI-compatible endpoint, plus the non-origin-model re-stamp caveat.

* fix: honor WATERMARKS_REWRITE_ALLOW_REMOTE in the SynthID-text benchmark

The --rewrite-allow-remote flag now defaults from the env var (matching
rewrite_text.py and the other WATERMARKS_REWRITE_* settings), so a
non-loopback rewrite endpoint works after sourcing .env without an extra
flag.

* fix: MarkLLM sparse checkout and deps for the SynthID harness

- setup_markllm.sh sparse-checkout omitted '/visualize/', which
  watermark/base.py imports at module load — every scheme (incl. SynthID)
  failed with 'No module named visualize' during generation/detection.
- requirements-markllm.txt omitted scikit-learn, imported by the SynthID
  detector (watermark/synthid/detector_bayesian_torch.py).

Both broke the MarkLLM harness at runtime; the benchmark's sanity gate then
excluded every sample, producing empty per-variant results.

* fix: drop 4 GiB RLIMIT_AS on benchmark subprocesses

_run_cmd applied the common child RLIMIT_AS (default 4 GiB) via
subprocess_preexec_fn to every MarkLLM/rewrite child. torch needs a much
larger address space: CUDA init failed with 'out of memory' at
cudaGetDeviceCount and the 5.2 GB fp32 opt-1.3b could not load, so every
sample was excluded at generation. text_detectors.py already applies no
address-space cap to MarkLLM by default; the benchmark now matches.

* perf: keep MarkLLM resident via a serve worker (624 cold starts -> 1)

The benchmark spawned a fresh torch + opt-1.3b process per operation
(~60-90s each); a full run needs ~624 of them. detect_text_watermark.py
gains a 'serve' mode (JSON-lines over stdin/stdout, ready handshake) that
loads the model once; bench_synthid_text.py uses it via MarkLLMWorker with
automatic fallback to one-shot subprocesses (--no-worker to force).
Turns ~8h runs into ~40-60min.

* perf: skip per-candidate Gemini detections in rewrite subprocess

* feat: run the SynthID-text benchmark from the wr-markllm compose service

- Dockerfile.markllm: add '/visualize/' to the sparse checkout (same fix as
  setup_markllm.sh) and COPY the benchmark + rewrite scripts (stdlib-only).
- compose.yaml: wr-markllm gets the WATERMARKS_REWRITE_* and
  WATERMARKS_GEMINI_* env wiring, a bench-out volume for --out-dir, and a
  read-only mount of the bundled corpus (build context is service/, so the
  corpus cannot be COPY'd).
- docs: docker compose run example.

Note: the image ships CPU torch by design, so the container path is for
portability/CI; GPU runs use the host setup_markllm.sh venv.

* feat: per-sample progress logging in the benchmark

The persistent worker returns samples in-memory, so nothing is written
until the end of a run — runs looked stuck. eprint a [gen i/N] line per
generated sample and a [removal] summary per sample.

* chore: migrate Gemini config to gemini-3.6-flash; document SynthID-text retirement

Google retired SynthID text watermarking on the Generative Language API
(Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK
is rejected on current 3.x models (confirmed by Google AI staff). Migrate
the default detection model to gemini-3.6-flash, document the retirement
in vendor-notes.md and the benchmark report caveat, and keep the detector
seam fail-soft until a vendor endpoint (e.g. Vertex AI) returns.

* feat: remove gemini-synthid-text detector (Google retired text watermarking)

Google removed SynthID text watermarking from the Generative Language API
(Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK
is rejected on current 3.x models, so the vendor detector had nothing to
detect. Remove GeminiSynthIDTextDetector and its wiring:

- text_detectors.py: drop the Gemini class, HTTP helpers, and constants;
  keep MarkLLM + Claude seams (registry now markllm + claude-text).
- server.py / rewrite_text.py: per-candidate detection now triggers on
  --markllm-scheme only.
- bench_synthid_text.py: remove the Gemini tier (before/after, report
  table, --no-gemini flag); report caveat notes the retirement.
- configs/docs: drop WATERMARKS_GEMINI_* from .env.example / compose /
  README / SKILL.md / vendor-notes.md; keep the retirement note.
- tests: gemini tests removed or converted to MarkLLM (mocked subprocess).
- Dockerfile.markllm: parameterize BASE_IMAGE + TORCH_INDEX_URL so a GPU/
  arm64 image can be built (used for the --gpus all benchmark run).

* feat: iterative detection-guided Layer B rewriting (default 3 attempts)

Layer B (rewrite_text.py) now rewrites iteratively and stops as soon as an
attempt passes watermark evaluation:

- --candidates defaults to 3 (WATERMARKS_REWRITE_CANDIDATES); each attempt is
  one rewrite + one evaluation, and the loop exits on the first attempt the
  evaluator reports as not watermarked.
- Evaluator priority: MarkLLM same-config detection (when --markllm-scheme is
  passed) > bigram-Jaccard lexical divergence (fallback; no verdict, all
  attempts generated, most diverged selected). A vendor-detector seam is
  reserved ahead of MarkLLM for a future SynthID-text endpoint (Google retired
  text watermarking on its API in Aug 2026).
- Best-effort fallback when the max is exhausted: the lowest-score attempt is
  returned with a note; detector errors are fail-soft and never fail the
  rewrite.
- --json-stats now reports evaluator / attempts_made / passed and per-attempt
  candidate_scores records (passed, evaluation); markllm before/after/cleared
  is unchanged and the selected attempt's verdict is reused (no duplicate
  MarkLLM detection).

Benchmark (bench_synthid_text.py):

- --variants default becomes paraphrase:3 (candidates = max attempts).
- Rows/report/CSV carry attempts per document (mean_attempts, att column;
  attempts / evaluator / passed columns).

Tests, README, docs/synthid-text-benchmark.md and .env.example updated; 490
tests pass, ruff clean.

* feat: split rewrite attempts into --candidates x --max-loops (defaults 1 x 1)

Follow-up to the iterative Layer B rewrite: separate "variants per round"
from "evaluation rounds", so the retry loop is explicit and defaults stay
conservative.

- rewrite_text.py: --candidates (WATERMARKS_REWRITE_CANDIDATES) is now the
  number of variants generated per loop iteration (default 1); new
  --max-loops (WATERMARKS_REWRITE_LOOPS) caps the evaluation rounds (default
  1) -- each round generates --candidates variants and stops as soon as one
  passes, so raising --max-loops retries new variants until an evaluation
  passes. Stats now report max_loops and per-attempt records carry the loop
  index.
- bench_synthid_text.py: new --rewrite-loops flag (default 1) passed through
  to --max-loops.
- README / docs / .env.example updated; tests cover the 1x1 defaults, loop
  retry until pass, and cross-loop exhaustion.

* feat: MarkLLM serve worker over loopback TCP (WATERMARKS_MARKLLM_PORT)

detect_text_watermark.py serve can now also listen on a loopback TCP port,
and MarkLLMTextDetector reuses a resident worker when
WATERMARKS_MARKLLM_PORT is set (falls back to a one-shot subprocess when the
worker is unreachable). This avoids a ~20s torch+model cold start per detect
for callers that run a worker out-of-band.

Tests: loopback worker protocol + detector worker-port routing (mock-based).

* ci: add macOS runner to the test matrix
This commit is contained in:
Guillaume Meyer (The Opinionated Man)
2026-08-18 18:28:19 -07:00
committed by GitHub
parent d5f4f03f85
commit 8318d4df79
7 changed files with 601 additions and 203 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+73 -29
View File
@@ -511,38 +511,63 @@ MARKLLM_DIR=~/MarkLLM \
--markllm-scheme kgw --markllm-dir "$HOME/MarkLLM" --json-stats
```
**Per-candidate detection:** when `--candidates N` (`N > 1`) is combined with
`--markllm-scheme`, every generated
candidate is run through the configured text detectors and `--json-stats`
reports per-candidate measurements. Candidate selection stays purely lexical;
the detections exist so you can see whether divergence actually correlates with
watermark removal:
**Detection-guided iterative rewriting:** Layer B now rewrites iteratively and
stops as soon as an attempt passes evaluation. Each evaluation round generates
`--candidates` variants (default **1**, `WATERMARKS_REWRITE_CANDIDATES`)
and `--max-loops` caps how many rounds run before the best-effort variant is
returned (default **1**, `WATERMARKS_REWRITE_LOOPS`). Each variant is one
rewrite call plus one evaluation, and a round exits early on the first attempt
the evaluator reports as not watermarked — so raising `--max-loops` retries
new variants until an evaluation passes (a typical clean rewrite costs one
attempt). The evaluator is chosen by priority:
1. **MarkLLM** — same-config research detection, when `--markllm-scheme` is
passed (with `--markllm-dir`). A vendor-detector slot is reserved above
MarkLLM for Google's SynthID-text detector, which Google retired on its API
in Aug 2026 — a future vendor endpoint can plug in there.
2. **bigram-Jaccard lexical divergence** — when no detector is configured; no
pass/fail verdict, so every attempt is generated and the most lexically
diverged one is selected (the original behavior).
`--json-stats` reports the evaluator, attempts made, pass/fail, and per-attempt
records:
```json
"candidate_scores": [
{
"lexical_divergence": 0.91,
"selection_score": 0.91,
"selected": true,
"detections": [
{"detector": "markllm", "available": true, "scheme": "kgw",
"is_watermarked": true, "score": 4.3, "threshold": 3.0}
]
},
{
"lexical_divergence": 0.84,
"selection_score": 0.84,
"selected": false,
"detections": [
{"detector": "markllm", "available": true, "scheme": "kgw",
"is_watermarked": false, "score": 1.7, "threshold": 3.0}
]
}
]
{
"evaluator": "markllm",
"candidates": 1,
"max_loops": 2,
"attempts_made": 2,
"passed": true,
"candidate_scores": [
{
"lexical_divergence": 0.91,
"selection_score": 0.91,
"selected": false,
"passed": false,
"evaluation": {"detector": "markllm", "available": true, "scheme": "kgw",
"is_watermarked": true, "score": 4.3, "threshold": 3.0}
},
{
"lexical_divergence": 0.84,
"selection_score": 0.84,
"selected": true,
"passed": true,
"evaluation": {"detector": "markllm", "available": true, "scheme": "kgw",
"is_watermarked": false, "score": 1.7, "threshold": 3.0}
}
],
"markllm": {"scheme": "kgw", "before": {"...": "..."}, "after": {"...": "..."},
"cleared": true, "note": "same-config only"}
}
```
A detector that is unconfigured, times out, or errors yields an
`"available": false` entry with an `error` reason and never fails the rewrite.
`"available": false` entry with an `error` reason and never fails the
rewrite — that attempt simply cannot pass, and the loop falls back to
lexical-divergence selection. When the max is exhausted without a pass, the
least-watermarked (lowest score) attempt is returned as best-effort with a
note.
If the backend is unconfigured or its deps are missing, the rewrite proceeds
and the report notes verification was unavailable. A GPU is recommended; CPU
@@ -574,8 +599,9 @@ docker run --rm --user "$(id -u):$(id -g)" -v "$(pwd):/data" \
effectively a Layer B rewrite clears SynthID-text-class watermarks and at
what cost. It generates watermarked + unwatermarked samples with the MarkLLM
SynthID scheme (same-config detection, sanity-gated), runs your rewrite
variants (strength × candidates) plus controls (no-removal, Layer-A-only,
optional re-stamp check), and writes a shareable `report.md` /
variants (strength × max rewrite attempts; the loop stops early on pass) plus
controls (no-removal, Layer-A-only, optional re-stamp check), and writes a
shareable `report.md` /
`results.json` / `results.csv`. Full guide:
[`docs/synthid-text-benchmark.md`](docs/synthid-text-benchmark.md).
@@ -854,6 +880,24 @@ make smoke # quick CLI smoke on fixtures
## Changelog
### Unreleased — detection-guided iterative Layer B rewriting
- **Layer B rewriting is now iterative and evaluation-driven**: each round
generates `--candidates` variants (default 1,
`WATERMARKS_REWRITE_CANDIDATES`) and `--max-loops` (default 1,
`WATERMARKS_REWRITE_LOOPS`) caps the evaluation rounds, stopping as soon
as an attempt passes watermark detection. Evaluator priority: MarkLLM (when
`--markllm-scheme`) > bigram-Jaccard lexical divergence (fallback; a
vendor-detector seam is reserved for a future SynthID-text endpoint).
- `rewrite_text.py --json-stats` now reports `evaluator` /
`max_loops` / `attempts_made` / `passed` and per-attempt
`candidate_scores` records (`loop`, `passed`, `evaluation`);
`markllm.before/after/cleared` is unchanged.
- **SynthID-text benchmark**: default variants `paraphrase:3`; report and CSV
now carry attempts per document (`mean_attempts`, `att` column;
`attempts` / `evaluator` / `passed` columns); `--rewrite-loops`
mirrors `--max-loops`.
### [v0.5.0](https://github.com/guillaumemeyer/watermarks-remover/releases/tag/v0.5.0) — service & Docker distribution, HTTP API, and verification harnesses
**Service / Docker distribution**