3 Commits

Include renames

Author	SHA1	Message	Date
Guillaume Meyer (The Opinionated Man)andGitHub	d5f4f03f85	feat: add reproducible SynthID-text removal benchmark (#145) ... * feat: add reproducible SynthID-text removal benchmark bench_synthid_text.py orchestrates the existing Layer B machinery into a controlled, shareable experiment: generate watermarked + unwatermarked samples with the MarkLLM SynthID scheme, run removal variants (strength x candidates) plus controls (no-removal, Layer-A-only, optional re-stamp), and report clear rate, score suppression, quality, and cost (tokens, wall time, USD) with a clears-per-MTok efficiency ratio. Emits report.md / results.json / results.csv with the exact reproduction command and pinned commits; optional Gemini official-detector tier when WATERMARKS_GEMINI_API_KEY is set. Mock-based tests, no torch in CI. * docs: add README section on running the SynthID-text benchmark Explains what LLM performs the Layer B rewrite (an external model configured via WATERMARKS_REWRITE_* env vars or --rewrite-* flags; MarkLLM's opt-1.3b is only the watermark generator/detector) and how to run a benchmark with Ollama or an OpenAI-compatible endpoint, plus the non-origin-model re-stamp caveat. * fix: honor WATERMARKS_REWRITE_ALLOW_REMOTE in the SynthID-text benchmark The --rewrite-allow-remote flag now defaults from the env var (matching rewrite_text.py and the other WATERMARKS_REWRITE_* settings), so a non-loopback rewrite endpoint works after sourcing .env without an extra flag. * fix: MarkLLM sparse checkout and deps for the SynthID harness - setup_markllm.sh sparse-checkout omitted '/visualize/', which watermark/base.py imports at module load — every scheme (incl. SynthID) failed with 'No module named visualize' during generation/detection. - requirements-markllm.txt omitted scikit-learn, imported by the SynthID detector (watermark/synthid/detector_bayesian_torch.py). Both broke the MarkLLM harness at runtime; the benchmark's sanity gate then excluded every sample, producing empty per-variant results. * fix: drop 4 GiB RLIMIT_AS on benchmark subprocesses _run_cmd applied the common child RLIMIT_AS (default 4 GiB) via subprocess_preexec_fn to every MarkLLM/rewrite child. torch needs a much larger address space: CUDA init failed with 'out of memory' at cudaGetDeviceCount and the 5.2 GB fp32 opt-1.3b could not load, so every sample was excluded at generation. text_detectors.py already applies no address-space cap to MarkLLM by default; the benchmark now matches. * perf: keep MarkLLM resident via a serve worker (624 cold starts -> 1) The benchmark spawned a fresh torch + opt-1.3b process per operation (~60-90s each); a full run needs ~624 of them. detect_text_watermark.py gains a 'serve' mode (JSON-lines over stdin/stdout, ready handshake) that loads the model once; bench_synthid_text.py uses it via MarkLLMWorker with automatic fallback to one-shot subprocesses (--no-worker to force). Turns ~8h runs into ~40-60min. * perf: skip per-candidate Gemini detections in rewrite subprocess * feat: run the SynthID-text benchmark from the wr-markllm compose service - Dockerfile.markllm: add '/visualize/' to the sparse checkout (same fix as setup_markllm.sh) and COPY the benchmark + rewrite scripts (stdlib-only). - compose.yaml: wr-markllm gets the WATERMARKS_REWRITE_* and WATERMARKS_GEMINI_* env wiring, a bench-out volume for --out-dir, and a read-only mount of the bundled corpus (build context is service/, so the corpus cannot be COPY'd). - docs: docker compose run example. Note: the image ships CPU torch by design, so the container path is for portability/CI; GPU runs use the host setup_markllm.sh venv. * feat: per-sample progress logging in the benchmark The persistent worker returns samples in-memory, so nothing is written until the end of a run — runs looked stuck. eprint a [gen i/N] line per generated sample and a [removal] summary per sample. * chore: migrate Gemini config to gemini-3.6-flash; document SynthID-text retirement Google retired SynthID text watermarking on the Generative Language API (Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK is rejected on current 3.x models (confirmed by Google AI staff). Migrate the default detection model to gemini-3.6-flash, document the retirement in vendor-notes.md and the benchmark report caveat, and keep the detector seam fail-soft until a vendor endpoint (e.g. Vertex AI) returns. * feat: remove gemini-synthid-text detector (Google retired text watermarking) Google removed SynthID text watermarking from the Generative Language API (Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK is rejected on current 3.x models, so the vendor detector had nothing to detect. Remove GeminiSynthIDTextDetector and its wiring: - text_detectors.py: drop the Gemini class, HTTP helpers, and constants; keep MarkLLM + Claude seams (registry now markllm + claude-text). - server.py / rewrite_text.py: per-candidate detection now triggers on --markllm-scheme only. - bench_synthid_text.py: remove the Gemini tier (before/after, report table, --no-gemini flag); report caveat notes the retirement. - configs/docs: drop WATERMARKS_GEMINI_* from .env.example / compose / README / SKILL.md / vendor-notes.md; keep the retirement note. - tests: gemini tests removed or converted to MarkLLM (mocked subprocess). - Dockerfile.markllm: parameterize BASE_IMAGE + TORCH_INDEX_URL so a GPU/ arm64 image can be built (used for the --gpus all benchmark run). * fix: harden notes aggregation against non-string notes A run completed all samples but crashed at the final aggregate step with 'cannot use list as a set element' when a row's notes contained a non-string value. Filter notes to strings (aggregate + CSV) and add a regression test. * perf: let the rewrite subprocess reuse the resident MarkLLM worker The rewrite subprocess (rewrite_text.py) ran its own before/after MarkLLM detects, each a ~20s torch+model cold start (~12 per sample = ~5min of the ~6min/sample runtime). Now: - detect_text_watermark.py serve gains --port N: a loopback TCP JSON-lines listener (default -1 = off) sharing the resident model, with a lock so stdin and socket requests never run the model concurrently. - text_detectors.MarkLLMTextDetector checks WATERMARKS_MARKLLM_PORT and does a fast loopback detect when a worker is up, falling back to the one-shot subprocess otherwise. - The benchmark worker publishes its port via that env var, so the rewrite subprocess inherits it and its detects hit the resident model. Turns ~6 min/sample into ~1-2 min; a full run drops from ~2h to ~40-50min. Tests: loopback-client + fallback + env-publish coverage. * chore: add benchmark-smoke.sh / benchmark-full.sh wrappers Simple host wrappers: source .env, default MARKLLM_DIR to ~/MarkLLM, use a repo-local HF cache by default, and run bench_synthid_text.py with a quick (2 docs, 1 seed, paraphrase:1) or full (8 docs x 3 seeds, three variants, re-stamp control) configuration. OUT_DIR overrides the output location.	2026-08-18 18:13:56 -07:00
3 people	a2e72ed019	feat: vendor text-watermark detection (Gemini SynthID, Claude seam, MarkLLM) + SynthID image scorer sidecar (#109) ... * feat: add vendor text-watermark detection and SynthID image scorer sidecar Adds Layer B watermark detection as a first-class service capability: - text_detectors.py: a registry of text-watermark detectors behind one interface — Google's official SynthID-text detector via the Gemini API (taskType DETECT_TEXT_WATERMARK), a Claude placeholder ready for Anthropic's announced detection API, and the MarkLLM research harness (KGW / SynthID, same-config-only). Fail-soft: unconfigured or errored detectors never block cleaning. - server.py: new POST /detect endpoint, detect_before / detect_after options on /clean (before/after scoring for text and images), an opt-in /inspect "detect" flag, and /capabilities gains text_detectors and scorers.synthid_http. - synthid_score_server.py: a stdlib HTTP sidecar for the reverse-SynthID scorer, so the published core image never bundles the non-commercial upstream code; wired via WATERMARKS_SYNTHID_SCORER_URL. - score_synthid.py: extract score_file() so the CLI and the sidecar share one implementation. - compose.yaml / Dockerfile.synthid / .env.example: wr-synthid-score sidecar service and env wiring. - README + skill docs, plus tests for the detectors, the /detect endpoint, and the image sidecar. * feat: per-candidate watermark detection for Layer B rewrite candidates When --candidates N (N > 1) is combined with --markllm-scheme or WATERMARKS_GEMINI_API_KEY, run every configured text detector from the text_detectors.py registry on each candidate and report per-candidate measurements in --json-stats as candidate_scores entries carrying lexical_divergence, selection_score, selected, and per-detector reports (is_watermarked, score, threshold where the detector provides one). Candidate selection stays purely lexical; the detections are observability for correlating lexical divergence with watermark removal (issue #106). Converges rewrite_text.py onto the shared detector registry: - MarkLLMTextDetector gains constructor overrides (scheme, upstream_dir, model, timeout) plus the checkout-venv interpreter preference and the WATERMARKS_MARKLLM_RLIMIT_AS preexec guard ported from rewrite_text.py; the old _markllm_detect / _venv_python / _markllm_preexec helpers are gone. - run_all_text_detectors() accepts an injected MarkLLM instance and an include_markllm switch so CLI flag gating stays intact. - before/after/cleared semantics unchanged; detection remains fail-soft. * docs: pin Watermarks in the Sand reference to arXiv v5 * fix: mark only one rewrite candidate as selected (#110) --------- Co-authored-by: Zhenxin Ai <142008897+ai-kunkun@users.noreply.github.com>	2026-08-17 18:31:22 -07:00
Guillaume Meyer (The Opinionated Man)andGitHub	55d4bdc9fc	feat: split skill from service, add HTTP API and Docker distribution (#60) ... * feat: split skill from service, add HTTP API and Docker distribution The agent skill (skills/remove-ai-marks/) is now a code-free remote client: all implementation moved to service/scripts/ and runs behind a stdlib HTTP service (server.py) with /health, /capabilities, /inspect, /clean and a dynamically generated OpenAPI 3.0.3 spec at /openapi.json. - Move scripts/ and the backend Dockerfiles under service/ - server.py: JSON/base64 HTTP entrypoint with size caps, binary guard, atomic writes, loopback default, optional bearer auth - Core Dockerfile (exiftool/qpdf/c2patool preinstalled) and a GHCR publish workflow for the core/markllm/markdiffusion images - compose.yaml (wr-* services, harness/heavy profiles) + compose-check.sh to validate the running stack (exit code only) - Fix markllm image build (tokenizers 0.22.2, CPU-only torch) and ctrlregen build (python:3.11 base for the 2023-era research pins) - Fix markllm/markdiffusion harness images missing common.py at runtime * docs: add .env.example and service configuration guide * fix: disable chain-of-thought for openai-compatible Layer B rewrites deepseek-v4-flash is a reasoning model: a one-line paraphrase burned 9,894 reasoning tokens (~100s) and hit the default timeout. Send reasoning_effort=none by default for the openai-compatible backend (--reasoning-effort / WATERMARKS_REWRITE_REASONING_EFFORT; 'off' omits the parameter), cutting the same rewrite to ~1s / 12 tokens. Tested end-to-end against api.deepseek.com. * fix: sanitize client-supplied filename in HTTP service CodeQL 'uncontrolled data in path expression' (server.py): a name like '../../x' flowed into Path(tmpdir) / name, letting an upload escape the request temp dir on write. Sanitize name to its basename in _decode_input (_safe_name) and refuse any joined path whose parent is not the tmpdir at the write sites (_tmp_path). Tests cover traversal names. * chore: gitignore .env (contains local rewrite credentials) * chore: deny-by-default gitignore and dockerignore; document compose env config .gitignore and service/.dockerignore now exclude everything by default and explicitly allow only what is publishable/needed: tracked source, docs, tests, .github, and (for images) the service/scripts/ tree that every Dockerfile COPYs. Root .dockerignore documents that all builds use service/ as context. README Configuration section now covers .env setup for docker compose, host-side export for CLI runs, and the full variable table.	2026-08-14 15:42:48 -07:00