External/watermarks-remover
1
0
Fork 0
mirror of https://github.com/guillaumemeyer/watermarks-remover.git synced 2026-08-22 13:11:57 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
watermarks-remover/Makefile
T
Guillaume Meyer (The Opinionated Man)andGitHub d5f4f03f85 feat: add reproducible SynthID-text removal benchmark (#145) 
* feat: add reproducible SynthID-text removal benchmark

bench_synthid_text.py orchestrates the existing Layer B machinery into a
controlled, shareable experiment: generate watermarked + unwatermarked
samples with the MarkLLM SynthID scheme, run removal variants (strength x
candidates) plus controls (no-removal, Layer-A-only, optional re-stamp),
and report clear rate, score suppression, quality, and cost (tokens,
wall time, USD) with a clears-per-MTok efficiency ratio.

Emits report.md / results.json / results.csv with the exact reproduction
command and pinned commits; optional Gemini official-detector tier when
WATERMARKS_GEMINI_API_KEY is set. Mock-based tests, no torch in CI.

* docs: add README section on running the SynthID-text benchmark

Explains what LLM performs the Layer B rewrite (an external model configured
via WATERMARKS_REWRITE_* env vars or --rewrite-* flags; MarkLLM's opt-1.3b is
only the watermark generator/detector) and how to run a benchmark with Ollama
or an OpenAI-compatible endpoint, plus the non-origin-model re-stamp caveat.

* fix: honor WATERMARKS_REWRITE_ALLOW_REMOTE in the SynthID-text benchmark

The --rewrite-allow-remote flag now defaults from the env var (matching
rewrite_text.py and the other WATERMARKS_REWRITE_* settings), so a
non-loopback rewrite endpoint works after sourcing .env without an extra
flag.

* fix: MarkLLM sparse checkout and deps for the SynthID harness

- setup_markllm.sh sparse-checkout omitted '/visualize/', which
  watermark/base.py imports at module load — every scheme (incl. SynthID)
  failed with 'No module named visualize' during generation/detection.
- requirements-markllm.txt omitted scikit-learn, imported by the SynthID
  detector (watermark/synthid/detector_bayesian_torch.py).

Both broke the MarkLLM harness at runtime; the benchmark's sanity gate then
excluded every sample, producing empty per-variant results.

* fix: drop 4 GiB RLIMIT_AS on benchmark subprocesses

_run_cmd applied the common child RLIMIT_AS (default 4 GiB) via
subprocess_preexec_fn to every MarkLLM/rewrite child. torch needs a much
larger address space: CUDA init failed with 'out of memory' at
cudaGetDeviceCount and the 5.2 GB fp32 opt-1.3b could not load, so every
sample was excluded at generation. text_detectors.py already applies no
address-space cap to MarkLLM by default; the benchmark now matches.

* perf: keep MarkLLM resident via a serve worker (624 cold starts -> 1)

The benchmark spawned a fresh torch + opt-1.3b process per operation
(~60-90s each); a full run needs ~624 of them. detect_text_watermark.py
gains a 'serve' mode (JSON-lines over stdin/stdout, ready handshake) that
loads the model once; bench_synthid_text.py uses it via MarkLLMWorker with
automatic fallback to one-shot subprocesses (--no-worker to force).
Turns ~8h runs into ~40-60min.

* perf: skip per-candidate Gemini detections in rewrite subprocess

* feat: run the SynthID-text benchmark from the wr-markllm compose service

- Dockerfile.markllm: add '/visualize/' to the sparse checkout (same fix as
  setup_markllm.sh) and COPY the benchmark + rewrite scripts (stdlib-only).
- compose.yaml: wr-markllm gets the WATERMARKS_REWRITE_* and
  WATERMARKS_GEMINI_* env wiring, a bench-out volume for --out-dir, and a
  read-only mount of the bundled corpus (build context is service/, so the
  corpus cannot be COPY'd).
- docs: docker compose run example.

Note: the image ships CPU torch by design, so the container path is for
portability/CI; GPU runs use the host setup_markllm.sh venv.

* feat: per-sample progress logging in the benchmark

The persistent worker returns samples in-memory, so nothing is written
until the end of a run — runs looked stuck. eprint a [gen i/N] line per
generated sample and a [removal] summary per sample.

* chore: migrate Gemini config to gemini-3.6-flash; document SynthID-text retirement

Google retired SynthID text watermarking on the Generative Language API
(Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK
is rejected on current 3.x models (confirmed by Google AI staff). Migrate
the default detection model to gemini-3.6-flash, document the retirement
in vendor-notes.md and the benchmark report caveat, and keep the detector
seam fail-soft until a vendor endpoint (e.g. Vertex AI) returns.

* feat: remove gemini-synthid-text detector (Google retired text watermarking)

Google removed SynthID text watermarking from the Generative Language API
(Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK
is rejected on current 3.x models, so the vendor detector had nothing to
detect. Remove GeminiSynthIDTextDetector and its wiring:

- text_detectors.py: drop the Gemini class, HTTP helpers, and constants;
  keep MarkLLM + Claude seams (registry now markllm + claude-text).
- server.py / rewrite_text.py: per-candidate detection now triggers on
  --markllm-scheme only.
- bench_synthid_text.py: remove the Gemini tier (before/after, report
  table, --no-gemini flag); report caveat notes the retirement.
- configs/docs: drop WATERMARKS_GEMINI_* from .env.example / compose /
  README / SKILL.md / vendor-notes.md; keep the retirement note.
- tests: gemini tests removed or converted to MarkLLM (mocked subprocess).
- Dockerfile.markllm: parameterize BASE_IMAGE + TORCH_INDEX_URL so a GPU/
  arm64 image can be built (used for the --gpus all benchmark run).

* fix: harden notes aggregation against non-string notes

A run completed all samples but crashed at the final aggregate step with
'cannot use list as a set element' when a row's notes contained a
non-string value. Filter notes to strings (aggregate + CSV) and add a
regression test.

* perf: let the rewrite subprocess reuse the resident MarkLLM worker

The rewrite subprocess (rewrite_text.py) ran its own before/after MarkLLM
detects, each a ~20s torch+model cold start (~12 per sample = ~5min of the
~6min/sample runtime). Now:

- detect_text_watermark.py serve gains --port N: a loopback TCP JSON-lines
  listener (default -1 = off) sharing the resident model, with a lock so
  stdin and socket requests never run the model concurrently.
- text_detectors.MarkLLMTextDetector checks WATERMARKS_MARKLLM_PORT and
  does a fast loopback detect when a worker is up, falling back to the
  one-shot subprocess otherwise.
- The benchmark worker publishes its port via that env var, so the rewrite
  subprocess inherits it and its detects hit the resident model.

Turns ~6 min/sample into ~1-2 min; a full run drops from ~2h to ~40-50min.
Tests: loopback-client + fallback + env-publish coverage.

* chore: add benchmark-smoke.sh / benchmark-full.sh wrappers

Simple host wrappers: source .env, default MARKLLM_DIR to ~/MarkLLM, use a
repo-local HF cache by default, and run bench_synthid_text.py with a quick
(2 docs, 1 seed, paraphrase:1) or full (8 docs x 3 seeds, three variants,
re-stamp control) configuration. OUT_DIR overrides the output location.
2026-08-18 18:13:56 -07:00

134 lines
4.5 KiB
Makefile
Raw Permalink Blame History

.PHONY: test lint format lint-fix smoke smoke-synthid bootstrap-synthid docker-synthid-build docker-synthid-help \
smoke-ctrlregen bootstrap-ctrlregen docker-ctrlregen-build docker-ctrlregen-help \
smoke-markllm bootstrap-markllm docker-markllm-build docker-markllm-help \
smoke-markdiffusion bootstrap-markdiffusion docker-markdiffusion-build docker-markdiffusion-help \
bench-synthid-text \
docker-core-build docker-core-help serve compose-up compose-up-heavy compose-check \
install-skill install-cursor-text-skill clean
SCRIPTS := service/scripts
PYTHON ?= $(shell if [ -x .venv/bin/python ]; then echo .venv/bin/python; else echo python3; fi)
test:
$(PYTHON) -m pytest
lint:
$(PYTHON) -m ruff check service tests
format:
$(PYTHON) -m ruff format --check service tests
lint-fix:
$(PYTHON) -m ruff check --fix service tests
smoke:
-python3 $(SCRIPTS)/inspect_text.py tests/fixtures/sample_watermarked.txt
python3 $(SCRIPTS)/clean_text.py tests/fixtures/sample_watermarked.txt -o /tmp/wm.cleaned.txt --stats
python3 $(SCRIPTS)/rewrite_text.py tests/fixtures/sample_watermarked.txt --backend print-prompt >/dev/null
-python3 $(SCRIPTS)/inspect_file.py tests/fixtures/sample_ai.md
python3 $(SCRIPTS)/clean_file.py tests/fixtures/sample_ai.md -o /tmp/sample_ai.cleaned.md
python3 $(SCRIPTS)/clean_file.py tests/fixtures/sample_ai.html -o /tmp/sample_ai.cleaned.html
python3 $(SCRIPTS)/clean_file.py tests/fixtures/sample_meta.svg -o /tmp/sample_meta.cleaned.svg
@echo "smoke ok"
smoke-synthid:
@if [ -z "$(REVERSE_SYNTHID_DIR)" ]; then \
echo "smoke-synthid skipped (set REVERSE_SYNTHID_DIR)"; \
else \
$(PYTHON) $(SCRIPTS)/score_synthid.py --help >/dev/null && echo "score_synthid adapter present"; \
fi
bootstrap-synthid:
./service/scripts/setup_synthid.sh
docker-synthid-build:
docker build -f service/Dockerfile.synthid -t watermarks-remover-synthid-scorer service/
docker-synthid-help:
docker run --rm watermarks-remover-synthid-scorer --help
smoke-ctrlregen:
@if [ -z "$(NOAI_WATERMARK_DIR)" ]; then \
echo "smoke-ctrlregen skipped (set NOAI_WATERMARK_DIR)"; \
else \
$(PYTHON) $(SCRIPTS)/clean_ctrlregen.py --help >/dev/null && echo "clean_ctrlregen adapter present"; \
fi
bootstrap-ctrlregen:
./service/scripts/setup_ctrlregen.sh
docker-ctrlregen-build:
docker build -f service/Dockerfile.ctrlregen -t watermarks-remover-ctrlregen service/
docker-ctrlregen-help:
docker run --rm watermarks-remover-ctrlregen --help
smoke-markllm:
@if [ -z "$(MARKLLM_DIR)" ]; then \
echo "smoke-markllm skipped (set MARKLLM_DIR)"; \
else \
$(PYTHON) $(SCRIPTS)/detect_text_watermark.py --help >/dev/null && echo "detect_text_watermark adapter present"; \
fi
bootstrap-markllm:
./service/scripts/setup_markllm.sh
docker-markllm-build:
docker build -f service/Dockerfile.markllm -t watermarks-remover-markllm service/
docker-markllm-help:
docker run --rm watermarks-remover-markllm --help
smoke-markdiffusion:
@if [ -z "$(MARKDIFFUSION_DIR)" ]; then \
echo "smoke-markdiffusion skipped (set MARKDIFFUSION_DIR)"; \
else \
$(PYTHON) $(SCRIPTS)/markdiffusion_harness.py --help >/dev/null && echo "markdiffusion_harness adapter present"; \
bench-synthid-text:
@if [ -z "$(MARKLLM_DIR)" ]; then \
echo "bench-synthid-text skipped (set MARKLLM_DIR; see docs/synthid-text-benchmark.md)"; \
else \
echo "run: $(PYTHON) $(SCRIPTS)/bench_synthid_text.py --markllm-dir $(MARKLLM_DIR) --rewrite-model <model> --rewrite-backend <backend>"; \
fi
bootstrap-markdiffusion:
./service/scripts/setup_markdiffusion.sh
docker-markdiffusion-build:
docker build -f service/Dockerfile.markdiffusion -t watermarks-remover-markdiffusion service/
docker-markdiffusion-help:
docker run --rm watermarks-remover-markdiffusion --help
# Core HTTP service (text + file/image metadata cleaning).
docker-core-build:
docker build -f service/Dockerfile -t watermarks-remover service/
docker-core-help:
docker run --rm watermarks-remover /app/scripts/server.py --help
# Run the HTTP service locally (stdlib only, no Docker).
serve:
$(PYTHON) $(SCRIPTS)/server.py --host 127.0.0.1 --port 8765
compose-up:
docker compose up --build -d
compose-up-heavy:
docker compose --profile harness --profile heavy up --build -d
compose-check:
./compose-check.sh
install-skill:
mkdir -p $(HOME)/.grok/skills
ln -sfn $(CURDIR)/skills/remove-ai-marks $(HOME)/.grok/skills/remove-ai-marks
@echo "linked -> $(HOME)/.grok/skills/remove-ai-marks"
install-cursor-text-skill:
$(PYTHON) install_skill.py
clean:
find . -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true
rm -rf .pytest_cache .venv
Reference in New Issue View Git Blame Copy Permalink