mirror of
https://github.com/guillaumemeyer/watermarks-remover.git
synced 2026-08-22 13:11:57 +02:00
feat: add reproducible SynthID-text removal benchmark (#145)
* feat: add reproducible SynthID-text removal benchmark bench_synthid_text.py orchestrates the existing Layer B machinery into a controlled, shareable experiment: generate watermarked + unwatermarked samples with the MarkLLM SynthID scheme, run removal variants (strength x candidates) plus controls (no-removal, Layer-A-only, optional re-stamp), and report clear rate, score suppression, quality, and cost (tokens, wall time, USD) with a clears-per-MTok efficiency ratio. Emits report.md / results.json / results.csv with the exact reproduction command and pinned commits; optional Gemini official-detector tier when WATERMARKS_GEMINI_API_KEY is set. Mock-based tests, no torch in CI. * docs: add README section on running the SynthID-text benchmark Explains what LLM performs the Layer B rewrite (an external model configured via WATERMARKS_REWRITE_* env vars or --rewrite-* flags; MarkLLM's opt-1.3b is only the watermark generator/detector) and how to run a benchmark with Ollama or an OpenAI-compatible endpoint, plus the non-origin-model re-stamp caveat. * fix: honor WATERMARKS_REWRITE_ALLOW_REMOTE in the SynthID-text benchmark The --rewrite-allow-remote flag now defaults from the env var (matching rewrite_text.py and the other WATERMARKS_REWRITE_* settings), so a non-loopback rewrite endpoint works after sourcing .env without an extra flag. * fix: MarkLLM sparse checkout and deps for the SynthID harness - setup_markllm.sh sparse-checkout omitted '/visualize/', which watermark/base.py imports at module load — every scheme (incl. SynthID) failed with 'No module named visualize' during generation/detection. - requirements-markllm.txt omitted scikit-learn, imported by the SynthID detector (watermark/synthid/detector_bayesian_torch.py). Both broke the MarkLLM harness at runtime; the benchmark's sanity gate then excluded every sample, producing empty per-variant results. * fix: drop 4 GiB RLIMIT_AS on benchmark subprocesses _run_cmd applied the common child RLIMIT_AS (default 4 GiB) via subprocess_preexec_fn to every MarkLLM/rewrite child. torch needs a much larger address space: CUDA init failed with 'out of memory' at cudaGetDeviceCount and the 5.2 GB fp32 opt-1.3b could not load, so every sample was excluded at generation. text_detectors.py already applies no address-space cap to MarkLLM by default; the benchmark now matches. * perf: keep MarkLLM resident via a serve worker (624 cold starts -> 1) The benchmark spawned a fresh torch + opt-1.3b process per operation (~60-90s each); a full run needs ~624 of them. detect_text_watermark.py gains a 'serve' mode (JSON-lines over stdin/stdout, ready handshake) that loads the model once; bench_synthid_text.py uses it via MarkLLMWorker with automatic fallback to one-shot subprocesses (--no-worker to force). Turns ~8h runs into ~40-60min. * perf: skip per-candidate Gemini detections in rewrite subprocess * feat: run the SynthID-text benchmark from the wr-markllm compose service - Dockerfile.markllm: add '/visualize/' to the sparse checkout (same fix as setup_markllm.sh) and COPY the benchmark + rewrite scripts (stdlib-only). - compose.yaml: wr-markllm gets the WATERMARKS_REWRITE_* and WATERMARKS_GEMINI_* env wiring, a bench-out volume for --out-dir, and a read-only mount of the bundled corpus (build context is service/, so the corpus cannot be COPY'd). - docs: docker compose run example. Note: the image ships CPU torch by design, so the container path is for portability/CI; GPU runs use the host setup_markllm.sh venv. * feat: per-sample progress logging in the benchmark The persistent worker returns samples in-memory, so nothing is written until the end of a run — runs looked stuck. eprint a [gen i/N] line per generated sample and a [removal] summary per sample. * chore: migrate Gemini config to gemini-3.6-flash; document SynthID-text retirement Google retired SynthID text watermarking on the Generative Language API (Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK is rejected on current 3.x models (confirmed by Google AI staff). Migrate the default detection model to gemini-3.6-flash, document the retirement in vendor-notes.md and the benchmark report caveat, and keep the detector seam fail-soft until a vendor endpoint (e.g. Vertex AI) returns. * feat: remove gemini-synthid-text detector (Google retired text watermarking) Google removed SynthID text watermarking from the Generative Language API (Aug 2026): text output is no longer watermarked and DETECT_TEXT_WATERMARK is rejected on current 3.x models, so the vendor detector had nothing to detect. Remove GeminiSynthIDTextDetector and its wiring: - text_detectors.py: drop the Gemini class, HTTP helpers, and constants; keep MarkLLM + Claude seams (registry now markllm + claude-text). - server.py / rewrite_text.py: per-candidate detection now triggers on --markllm-scheme only. - bench_synthid_text.py: remove the Gemini tier (before/after, report table, --no-gemini flag); report caveat notes the retirement. - configs/docs: drop WATERMARKS_GEMINI_* from .env.example / compose / README / SKILL.md / vendor-notes.md; keep the retirement note. - tests: gemini tests removed or converted to MarkLLM (mocked subprocess). - Dockerfile.markllm: parameterize BASE_IMAGE + TORCH_INDEX_URL so a GPU/ arm64 image can be built (used for the --gpus all benchmark run). * fix: harden notes aggregation against non-string notes A run completed all samples but crashed at the final aggregate step with 'cannot use list as a set element' when a row's notes contained a non-string value. Filter notes to strings (aggregate + CSV) and add a regression test. * perf: let the rewrite subprocess reuse the resident MarkLLM worker The rewrite subprocess (rewrite_text.py) ran its own before/after MarkLLM detects, each a ~20s torch+model cold start (~12 per sample = ~5min of the ~6min/sample runtime). Now: - detect_text_watermark.py serve gains --port N: a loopback TCP JSON-lines listener (default -1 = off) sharing the resident model, with a lock so stdin and socket requests never run the model concurrently. - text_detectors.MarkLLMTextDetector checks WATERMARKS_MARKLLM_PORT and does a fast loopback detect when a worker is up, falling back to the one-shot subprocess otherwise. - The benchmark worker publishes its port via that env var, so the rewrite subprocess inherits it and its detects hit the resident model. Turns ~6 min/sample into ~1-2 min; a full run drops from ~2h to ~40-50min. Tests: loopback-client + fallback + env-publish coverage. * chore: add benchmark-smoke.sh / benchmark-full.sh wrappers Simple host wrappers: source .env, default MARKLLM_DIR to ~/MarkLLM, use a repo-local HF cache by default, and run bench_synthid_text.py with a quick (2 docs, 1 seed, paraphrase:1) or full (8 docs x 3 seeds, three variants, re-stamp control) configuration. OUT_DIR overrides the output location.
This commit is contained in:
Guillaume Meyer (The Opinionated Man)
parent
063119d7e5
commit
d5f4f03f85
File diff suppressed because it is too large
Load Diff
@@ -26,8 +26,12 @@ from __future__ import annotations
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import socketserver
|
||||
import sys
|
||||
import threading
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
SCRIPTS_DIR = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(SCRIPTS_DIR))
|
||||
@@ -143,6 +147,42 @@ def _resolve_config(upstream: Path, alg: str, config: str | None) -> Path:
|
||||
return path
|
||||
|
||||
|
||||
def _generate(
|
||||
wm: Any,
|
||||
prompt: str,
|
||||
seed: int | None,
|
||||
max_new_tokens: int,
|
||||
min_length: int = 0,
|
||||
need_unwatermarked: bool = True,
|
||||
) -> tuple[str, str | None]:
|
||||
"""Generate watermarked (and optionally unwatermarked) text for *prompt*."""
|
||||
if seed is not None:
|
||||
import torch
|
||||
|
||||
torch.manual_seed(seed)
|
||||
wm.config.gen_kwargs["max_new_tokens"] = max_new_tokens
|
||||
wm.config.gen_kwargs["min_length"] = min_length
|
||||
watermarked = wm.generate_watermarked_text(prompt)
|
||||
unwatermarked = wm.generate_unwatermarked_text(prompt) if need_unwatermarked else None
|
||||
return watermarked, unwatermarked
|
||||
|
||||
|
||||
def _detect_payload(wm: Any, text: str, threshold: float | None) -> dict[str, Any]:
|
||||
"""Same-config detection payload (is_watermarked/score/threshold)."""
|
||||
result = wm.detect_watermark(text, return_dict=True)
|
||||
is_watermarked = bool(result.get("is_watermarked", False))
|
||||
score = result.get("score")
|
||||
try:
|
||||
score = float(score)
|
||||
except (TypeError, ValueError):
|
||||
score = None
|
||||
return {
|
||||
"is_watermarked": is_watermarked,
|
||||
"score": score,
|
||||
"threshold": threshold,
|
||||
}
|
||||
|
||||
|
||||
def _cmd_detect(args: argparse.Namespace, upstream: Path, alg: str) -> int:
|
||||
if args.path != "-" and not Path(args.path).is_file():
|
||||
eprint(f"not a file: {args.path}")
|
||||
@@ -155,7 +195,7 @@ def _cmd_detect(args: argparse.Namespace, upstream: Path, alg: str) -> int:
|
||||
config = _resolve_config(upstream, alg, args.config)
|
||||
threshold = _threshold_from_config(config)
|
||||
wm = _load_algorithm(upstream, alg, config, args.model, device, offline=args.offline)
|
||||
result = wm.detect_watermark(text, return_dict=True)
|
||||
det = _detect_payload(wm, text, threshold)
|
||||
except _Unavailable as e:
|
||||
eprint(str(e))
|
||||
return 3
|
||||
@@ -163,12 +203,8 @@ def _cmd_detect(args: argparse.Namespace, upstream: Path, alg: str) -> int:
|
||||
eprint(f"detection error: {e}")
|
||||
return 1
|
||||
|
||||
is_watermarked = bool(result.get("is_watermarked", False))
|
||||
score = result.get("score")
|
||||
try:
|
||||
score = float(score)
|
||||
except (TypeError, ValueError):
|
||||
score = None
|
||||
is_watermarked = det["is_watermarked"]
|
||||
score = det["score"]
|
||||
|
||||
payload = {
|
||||
"available": True,
|
||||
@@ -201,16 +237,14 @@ def _cmd_watermark(args: argparse.Namespace, upstream: Path, alg: str) -> int:
|
||||
try:
|
||||
config = _resolve_config(upstream, alg, args.config)
|
||||
wm = _load_algorithm(upstream, alg, config, args.model, device, offline=args.offline)
|
||||
if args.seed is not None:
|
||||
import torch
|
||||
|
||||
torch.manual_seed(args.seed)
|
||||
wm.config.gen_kwargs["max_new_tokens"] = args.max_new_tokens
|
||||
wm.config.gen_kwargs["min_length"] = args.min_length
|
||||
watermarked = wm.generate_watermarked_text(prompt)
|
||||
unwatermarked = None
|
||||
if args.unwatermarked_output:
|
||||
unwatermarked = wm.generate_unwatermarked_text(prompt)
|
||||
watermarked, unwatermarked = _generate(
|
||||
wm,
|
||||
prompt,
|
||||
args.seed,
|
||||
args.max_new_tokens,
|
||||
args.min_length,
|
||||
need_unwatermarked=bool(args.unwatermarked_output),
|
||||
)
|
||||
except _Unavailable as e:
|
||||
eprint(str(e))
|
||||
return 3
|
||||
@@ -245,9 +279,150 @@ def _cmd_watermark(args: argparse.Namespace, upstream: Path, alg: str) -> int:
|
||||
f" unwatermarked sample ({payload['unwatermarked_chars']} chars) -> {args.unwatermarked_output}"
|
||||
)
|
||||
|
||||
|
||||
def _handle_serve_request(wm: Any, req: dict[str, Any], threshold: float | None) -> dict[str, Any]:
|
||||
"""Handle one JSON-lines request; never raises (responds with ok:false)."""
|
||||
rid = req.get("id")
|
||||
op = req.get("op")
|
||||
if op == "exit":
|
||||
return {"ok": True, "id": rid}
|
||||
try:
|
||||
if op == "watermark":
|
||||
prompt = req.get("prompt")
|
||||
if not isinstance(prompt, str) or not prompt:
|
||||
raise ValueError("'prompt' must be a non-empty string")
|
||||
watermarked, unwatermarked = _generate(
|
||||
wm,
|
||||
prompt,
|
||||
req.get("seed"),
|
||||
req.get("max_new_tokens", 200),
|
||||
req.get("min_length", 0),
|
||||
need_unwatermarked=True,
|
||||
)
|
||||
return {
|
||||
"ok": True,
|
||||
"id": rid,
|
||||
"watermarked": watermarked,
|
||||
"unwatermarked": unwatermarked,
|
||||
"watermarked_chars": len(watermarked),
|
||||
"unwatermarked_chars": len(unwatermarked),
|
||||
}
|
||||
if op == "detect":
|
||||
text = req.get("text")
|
||||
if not isinstance(text, str) or not text:
|
||||
raise ValueError("'text' must be a non-empty string")
|
||||
det = _detect_payload(wm, text, threshold)
|
||||
return {"ok": True, "id": rid, **det}
|
||||
return {"ok": False, "id": rid, "error": f"unknown op {op!r}"}
|
||||
except Exception as e: # a bad request must not kill the worker
|
||||
return {"ok": False, "id": rid, "error": str(e)}
|
||||
|
||||
|
||||
def _cmd_serve(args: argparse.Namespace, upstream: Path, alg: str) -> int:
|
||||
"""Serve watermark/detect requests over JSON-lines stdin/stdout.
|
||||
|
||||
Loads the MarkLLM model once and keeps it resident so callers (e.g. the
|
||||
SynthID-text benchmark) can run many operations without paying the
|
||||
torch + model load cost per call. Protocol:
|
||||
|
||||
request: {"op": "watermark", "id": N, "prompt": str, "seed": int|None,
|
||||
"max_new_tokens": int, "min_length": int}
|
||||
{"op": "detect", "id": N, "text": str}
|
||||
{"op": "exit", "id": N}
|
||||
response: {"ok": true, "id": N, ...} | {"ok": false, "id": N, "error": str}
|
||||
|
||||
The first stdout line is a {"ready": true, ...} handshake emitted after
|
||||
model load. Errors on one request never kill the worker.
|
||||
"""
|
||||
device = resolve_device(args.device)
|
||||
try:
|
||||
config = _resolve_config(upstream, alg, args.config)
|
||||
threshold = _threshold_from_config(config)
|
||||
wm = _load_algorithm(upstream, alg, config, args.model, device, offline=args.offline)
|
||||
except _Unavailable as e:
|
||||
eprint(str(e))
|
||||
return 3
|
||||
except Exception as e:
|
||||
eprint(f"serve load error: {e}")
|
||||
return 1
|
||||
|
||||
def respond(payload: dict[str, Any]) -> None:
|
||||
print(json.dumps(payload), flush=True)
|
||||
|
||||
ready: dict[str, Any] = {"ready": True, "scheme": alg, "model": args.model, "device": device}
|
||||
lock = threading.Lock()
|
||||
server: socketserver.ThreadingTCPServer | None = None
|
||||
if args.port >= 0:
|
||||
# Loopback TCP listener so other processes (e.g. the rewrite
|
||||
# subprocess's MarkLLM detector) can reuse this resident model
|
||||
# instead of cold-starting their own. Port 0 = ephemeral.
|
||||
server = _serve_socket_server(wm, threshold, args.port, lock)
|
||||
ready["port"] = server.server_address[1]
|
||||
respond(ready)
|
||||
|
||||
try:
|
||||
for raw_line in sys.stdin:
|
||||
line = raw_line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
req = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
respond({"ok": False, "error": "invalid JSON request"})
|
||||
continue
|
||||
if not isinstance(req, dict):
|
||||
respond({"ok": False, "error": "request must be a JSON object"})
|
||||
continue
|
||||
with lock:
|
||||
resp = _handle_serve_request(wm, req, threshold)
|
||||
respond(resp)
|
||||
if req.get("op") == "exit":
|
||||
return 0
|
||||
finally:
|
||||
if server is not None:
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
return 0
|
||||
|
||||
|
||||
def _serve_socket_server(
|
||||
wm: Any, threshold: float | None, port: int, lock: threading.Lock
|
||||
) -> socketserver.ThreadingTCPServer:
|
||||
"""A loopback JSON-lines TCP server sharing this process's model."""
|
||||
|
||||
class _Handler(socketserver.BaseRequestHandler):
|
||||
def handle(self) -> None:
|
||||
f = self.request.makefile("r", encoding="utf-8")
|
||||
for raw in f:
|
||||
line = raw.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
req = json.loads(line)
|
||||
if not isinstance(req, dict):
|
||||
raise ValueError("request must be a JSON object")
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
resp: dict[str, Any] = {"ok": False, "error": "invalid JSON request"}
|
||||
else:
|
||||
with lock:
|
||||
resp = _handle_serve_request(wm, req, threshold)
|
||||
try:
|
||||
self.request.sendall((json.dumps(resp) + "\n").encode("utf-8"))
|
||||
except OSError:
|
||||
return
|
||||
if isinstance(req, dict) and req.get("op") == "exit":
|
||||
return
|
||||
|
||||
class _Server(socketserver.ThreadingTCPServer):
|
||||
allow_reuse_address = True
|
||||
daemon_threads = True
|
||||
|
||||
_Server.address_family = socket.AF_INET
|
||||
srv = _Server(("127.0.0.1", port), _Handler)
|
||||
threading.Thread(target=srv.serve_forever, daemon=True).start()
|
||||
return srv
|
||||
|
||||
|
||||
def _add_common(p: argparse.ArgumentParser) -> None:
|
||||
p.add_argument(
|
||||
"--upstream-dir",
|
||||
@@ -320,6 +495,19 @@ def main() -> int:
|
||||
wm.add_argument("--json", action="store_true", help="Emit JSON on stdout")
|
||||
wm.set_defaults(handler=_cmd_watermark)
|
||||
|
||||
serve = sub.add_parser(
|
||||
"serve", help="Serve watermark/detect over JSON-lines stdin (persistent worker)"
|
||||
)
|
||||
_add_common(serve)
|
||||
serve.add_argument(
|
||||
"--port",
|
||||
type=int,
|
||||
default=-1,
|
||||
help="Also listen on 127.0.0.1:PORT (JSON-lines; 0 = ephemeral) for "
|
||||
"other processes to reuse this resident model (default: no listener)",
|
||||
)
|
||||
serve.set_defaults(handler=_cmd_serve)
|
||||
|
||||
args = p.parse_args()
|
||||
|
||||
if args.cmd == "detect" and args.path != "-" and not Path(args.path).is_file():
|
||||
|
||||
@@ -30,5 +30,6 @@ matplotlib==3.11.1
|
||||
Cython==3.2.9
|
||||
numpy==2.5.2
|
||||
scipy==1.18.0
|
||||
scikit-learn==1.9.0
|
||||
huggingface_hub==1.27.0
|
||||
Pillow==12.3.0
|
||||
|
||||
@@ -182,8 +182,7 @@ def _per_candidate_detections(
|
||||
Fail-soft: a detector that is unconfigured, times out, or errors yields
|
||||
an ``available: False`` entry and never fails the rewrite. The MarkLLM
|
||||
harness is only included when ``markllm_detector`` is given (i.e. the
|
||||
caller passed --markllm-scheme); other detectors (e.g.
|
||||
gemini-synthid-text) are key-gated by their own environment.
|
||||
caller passed --markllm-scheme).
|
||||
"""
|
||||
detections: list[list[dict]] = []
|
||||
for cand in candidates:
|
||||
@@ -379,9 +378,7 @@ def rewrite(
|
||||
info["candidates"] = n
|
||||
out, scores = _select_candidate(text, outs)
|
||||
selected_idx = max(range(len(outs)), key=lambda i: scores[i])
|
||||
trigger = markllm_scheme is not None or bool(
|
||||
os.environ.get("WATERMARKS_GEMINI_API_KEY", "").strip()
|
||||
)
|
||||
trigger = markllm_scheme is not None
|
||||
detections = _per_candidate_detections(outs, markllm_detector) if trigger else []
|
||||
info["candidate_scores"] = []
|
||||
for i, cand in enumerate(outs):
|
||||
|
||||
@@ -78,6 +78,7 @@ if [[ ! -d "$DIR/.git" ]]; then
|
||||
'/config/' \
|
||||
'/utils/' \
|
||||
'/exceptions/' \
|
||||
'/visualize/' \
|
||||
'/evaluation/dataset.py' \
|
||||
'/LICENSE' \
|
||||
'/README.md'
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Vendor and research text-watermark detectors behind one interface.
|
||||
"""Research text-watermark detectors behind one interface.
|
||||
|
||||
Detects statistical (Layer B) text watermarks using vendor-provided or
|
||||
research detectors. Every detector implements the same small protocol:
|
||||
Detects statistical (Layer B) text watermarks using research or vendor
|
||||
detectors. Every detector implements the same small protocol:
|
||||
|
||||
name: str stable identifier (surfaced in /capabilities)
|
||||
available() -> bool configured and usable right now
|
||||
@@ -14,16 +14,18 @@ never block cleaning.
|
||||
|
||||
Detectors:
|
||||
|
||||
- gemini-synthid-text — Google's official SynthID-text detector, called
|
||||
through the Gemini API (taskType DETECT_TEXT_WATERMARK). Activated by
|
||||
WATERMARKS_GEMINI_API_KEY. User text is sent to Google only when the
|
||||
operator sets that key.
|
||||
- markllm — optional research harness (KGW / SynthID schemes) via
|
||||
- markllm — research harness (KGW / SynthID schemes) via
|
||||
detect_text_watermark.py, activated by MARKLLM_DIR. Same-config-only
|
||||
detection; not a vendor oracle.
|
||||
- claude-text — placeholder for Anthropic's announced text-watermark
|
||||
detection API. Reports unavailable until a public endpoint exists; the
|
||||
interface it must implement is already defined here.
|
||||
|
||||
Vendor note (Aug 2026): Google retired SynthID text watermarking on the
|
||||
Generative Language API — API text output is no longer watermarked and
|
||||
DETECT_TEXT_WATERMARK is rejected on current (3.x) models. The former
|
||||
gemini-synthid-text detector was removed for this reason; a vendor seam can
|
||||
be re-added if Google exposes detection again (e.g. via Vertex AI).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -34,28 +36,14 @@ import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from typing import Any, Protocol
|
||||
from urllib.parse import urlparse
|
||||
|
||||
GEMINI_DETECT_URL = (
|
||||
"https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent"
|
||||
)
|
||||
DEFAULT_GEMINI_MODEL = "gemini-2.5-flash"
|
||||
DEFAULT_GEMINI_TIMEOUT = 30.0
|
||||
DEFAULT_GEMINI_MAX_CHARS = 1_000_000
|
||||
DEFAULT_MARKLLM_SCHEME = "kgw"
|
||||
DEFAULT_MARKLLM_TIMEOUT = 600.0
|
||||
|
||||
|
||||
class DetectorError(RuntimeError):
|
||||
"""A detector call failed (network, HTTP error, timeout)."""
|
||||
|
||||
|
||||
class TextDetector(Protocol):
|
||||
name: str
|
||||
|
||||
@@ -71,196 +59,35 @@ def _env_float(name: str, default: float) -> float:
|
||||
return default
|
||||
|
||||
|
||||
def _env_int(name: str, default: int) -> int:
|
||||
try:
|
||||
return int(os.environ.get(name, str(default)))
|
||||
except ValueError:
|
||||
return default
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Gemini (Google's official SynthID-text detector)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_WATERMARKED_VERDICTS = ("watermarked", "ai-generated", "ai generated", "likely ai")
|
||||
|
||||
|
||||
def _verdict_is_watermarked(verdict: str | None) -> bool | None:
|
||||
"""Map the detector model's free-text verdict to a boolean, or None."""
|
||||
if not verdict:
|
||||
def _worker_port() -> int | None:
|
||||
"""Loopback port of a resident MarkLLM serve worker (WATERMARKS_MARKLLM_PORT)."""
|
||||
raw = os.environ.get("WATERMARKS_MARKLLM_PORT", "").strip()
|
||||
if not raw:
|
||||
return None
|
||||
low = verdict.strip().lower()
|
||||
if low.startswith(("unlikely", "no", "not")):
|
||||
return False
|
||||
return any(marker in low for marker in _WATERMARKED_VERDICTS)
|
||||
try:
|
||||
port = int(raw)
|
||||
except ValueError:
|
||||
return None
|
||||
return port if 0 < port < 65536 else None
|
||||
|
||||
|
||||
def _extract_numeric_score(candidate: dict[str, Any], top: dict[str, Any]) -> float | None:
|
||||
"""Pull a numeric watermark score from any of the known response shapes."""
|
||||
for container in (candidate, top):
|
||||
for key in (
|
||||
"syntheticTextScore",
|
||||
"synthetic_text_score",
|
||||
"watermarkScore",
|
||||
"watermark_score",
|
||||
"score",
|
||||
):
|
||||
value = container.get(key)
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
attribution = candidate.get("attributionMetadata") or {}
|
||||
if isinstance(attribution, dict):
|
||||
for key in ("syntheticTextScore", "synthetic_text_score", "score"):
|
||||
value = attribution.get(key)
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
st = attribution.get("syntheticText")
|
||||
if isinstance(st, dict):
|
||||
for key in ("score", "confidence"):
|
||||
value = st.get(key)
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
return None
|
||||
def _detect_via_worker(port: int, text: str, timeout: float) -> dict[str, Any]:
|
||||
"""One detect request to a resident MarkLLM serve worker over loopback TCP."""
|
||||
import socket as _socket
|
||||
|
||||
|
||||
def parse_gemini_detect_response(data: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Parse a generateContent response from a DETECT_TEXT_WATERMARK call.
|
||||
|
||||
The endpoint can answer with either a free-text verdict
|
||||
("Likely AI-generated") or a structured score; both shapes are handled
|
||||
defensively so upstream schema changes degrade to an error report
|
||||
instead of a crash.
|
||||
"""
|
||||
candidates = data.get("candidates") or []
|
||||
candidate = candidates[0] if candidates else {}
|
||||
if not isinstance(candidate, dict):
|
||||
candidate = {}
|
||||
|
||||
if not candidate:
|
||||
feedback = data.get("promptFeedback") or {}
|
||||
block = feedback.get("blockReason")
|
||||
if block:
|
||||
raise DetectorError(f"Gemini blocked the request: {block}")
|
||||
raise DetectorError("Gemini returned no candidates")
|
||||
|
||||
verdict: str | None = None
|
||||
content = candidate.get("content") or {}
|
||||
parts = content.get("parts") or []
|
||||
if parts and isinstance(parts[0], dict):
|
||||
verdict = parts[0].get("text")
|
||||
|
||||
score = _extract_numeric_score(candidate, data)
|
||||
is_watermarked = _verdict_is_watermarked(verdict)
|
||||
if is_watermarked is None and score is not None:
|
||||
is_watermarked = score >= 0.5
|
||||
|
||||
if verdict is None and score is None:
|
||||
raise DetectorError(
|
||||
f"unexpected Gemini response (no verdict or score): {json.dumps(data)[:400]}"
|
||||
)
|
||||
|
||||
raw = {
|
||||
key: candidate[key]
|
||||
for key in ("attributionMetadata", "finishReason", "index")
|
||||
if candidate.get(key) is not None
|
||||
}
|
||||
return {
|
||||
"is_watermarked": is_watermarked,
|
||||
"score": score,
|
||||
"verdict": verdict,
|
||||
"raw": raw,
|
||||
}
|
||||
|
||||
|
||||
def _post_json(url: str, body: dict[str, Any], api_key: str, timeout: float) -> dict[str, Any]:
|
||||
"""POST *body* to *url*, retrying once on transient failures."""
|
||||
if urlparse(url).scheme not in ("http", "https"):
|
||||
raise DetectorError(f"refusing non-http(s) Gemini endpoint: {url}")
|
||||
# S310: URL scheme is restricted to http/https just above.
|
||||
req = urllib.request.Request( # noqa: S310
|
||||
url,
|
||||
data=json.dumps(body).encode("utf-8"),
|
||||
headers={"Content-Type": "application/json", "x-goog-api-key": api_key},
|
||||
method="POST",
|
||||
)
|
||||
last_err = "Gemini API call failed"
|
||||
for attempt in range(2):
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp: # noqa: S310
|
||||
payload = json.loads(resp.read().decode("utf-8"))
|
||||
if not isinstance(payload, dict):
|
||||
raise DetectorError("non-object Gemini response")
|
||||
return payload
|
||||
except urllib.error.HTTPError as e:
|
||||
last_err = f"Gemini API HTTP {e.code}: {e.read().decode('utf-8', 'replace')[:300]}"
|
||||
if e.code not in (429, 500, 502, 503, 504):
|
||||
raise DetectorError(last_err) from e
|
||||
except (urllib.error.URLError, TimeoutError, OSError) as e:
|
||||
last_err = f"Gemini API unreachable: {e}"
|
||||
if attempt == 0:
|
||||
time.sleep(1.0)
|
||||
raise DetectorError(last_err)
|
||||
|
||||
|
||||
class GeminiSynthIDTextDetector:
|
||||
"""Google's official SynthID-text detector via the Gemini API."""
|
||||
|
||||
name = "gemini-synthid-text"
|
||||
vendor = "google"
|
||||
|
||||
def available(self) -> bool:
|
||||
return bool(os.environ.get("WATERMARKS_GEMINI_API_KEY", "").strip())
|
||||
|
||||
def detect(self, text: str) -> dict[str, Any]:
|
||||
api_key = os.environ.get("WATERMARKS_GEMINI_API_KEY", "").strip()
|
||||
if not api_key:
|
||||
return {
|
||||
"detector": self.name,
|
||||
"vendor": self.vendor,
|
||||
"available": False,
|
||||
"error": "WATERMARKS_GEMINI_API_KEY not set",
|
||||
}
|
||||
|
||||
max_chars = _env_int("WATERMARKS_GEMINI_MAX_CHARS", DEFAULT_GEMINI_MAX_CHARS)
|
||||
if len(text) > max_chars:
|
||||
return {
|
||||
"detector": self.name,
|
||||
"vendor": self.vendor,
|
||||
"available": True,
|
||||
"skipped": True,
|
||||
"reason": f"text longer than {max_chars} chars",
|
||||
"is_watermarked": None,
|
||||
}
|
||||
|
||||
model = (
|
||||
os.environ.get("WATERMARKS_GEMINI_MODEL", DEFAULT_GEMINI_MODEL) or DEFAULT_GEMINI_MODEL
|
||||
)
|
||||
timeout = _env_float("WATERMARKS_GEMINI_TIMEOUT", DEFAULT_GEMINI_TIMEOUT)
|
||||
url = GEMINI_DETECT_URL.format(model=model)
|
||||
body = {
|
||||
"contents": [{"role": "user", "parts": [{"text": text}]}],
|
||||
"generationConfig": {"taskType": "DETECT_TEXT_WATERMARK"},
|
||||
}
|
||||
report: dict[str, Any] = {
|
||||
"detector": self.name,
|
||||
"vendor": self.vendor,
|
||||
"model": model,
|
||||
"available": True,
|
||||
}
|
||||
try:
|
||||
data = _post_json(url, body, api_key, timeout)
|
||||
except DetectorError as e:
|
||||
report["available"] = False
|
||||
report["error"] = str(e)
|
||||
return report
|
||||
try:
|
||||
parsed = parse_gemini_detect_response(data)
|
||||
except DetectorError as e:
|
||||
report["available"] = False
|
||||
report["error"] = str(e)
|
||||
return report
|
||||
report.update(parsed)
|
||||
return report
|
||||
with _socket.create_connection(("127.0.0.1", port), timeout=timeout) as conn:
|
||||
conn.sendall((json.dumps({"op": "detect", "text": text}) + "\n").encode("utf-8"))
|
||||
f = conn.makefile("r", encoding="utf-8")
|
||||
line = f.readline()
|
||||
if not line:
|
||||
raise RuntimeError("worker closed without a response")
|
||||
try:
|
||||
resp = json.loads(line)
|
||||
except json.JSONDecodeError as e:
|
||||
raise RuntimeError(f"worker emitted non-JSON: {line[:120]!r}") from e
|
||||
if not isinstance(resp, dict) or not resp.get("ok"):
|
||||
raise RuntimeError(resp.get("error") or "worker detect failed")
|
||||
return resp
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -346,12 +173,31 @@ class MarkLLMTextDetector:
|
||||
report["error"] = "MARKLLM_DIR not set"
|
||||
return report
|
||||
|
||||
script = Path(__file__).resolve().parent / "detect_text_watermark.py"
|
||||
timeout = (
|
||||
self._timeout
|
||||
if self._timeout is not None
|
||||
else _env_float("WATERMARKS_MARKLLM_TIMEOUT", DEFAULT_MARKLLM_TIMEOUT)
|
||||
)
|
||||
|
||||
# Reuse a resident serve worker (WATERMARKS_MARKLLM_PORT) when one is
|
||||
# up — avoids a ~20s torch+model cold start per detect. Falls back to
|
||||
# a one-shot subprocess if the worker is unreachable.
|
||||
port = _worker_port()
|
||||
if port is not None:
|
||||
try:
|
||||
resp = _detect_via_worker(port, text, timeout)
|
||||
return {
|
||||
**report,
|
||||
"available": True,
|
||||
"is_watermarked": bool(resp["is_watermarked"]),
|
||||
"score": resp.get("score"),
|
||||
"threshold": resp.get("threshold"),
|
||||
"note": "detected via resident MarkLLM serve worker",
|
||||
}
|
||||
except Exception as e:
|
||||
report["error"] = f"MarkLLM worker detect failed ({e}); falling back"
|
||||
|
||||
script = Path(__file__).resolve().parent / "detect_text_watermark.py"
|
||||
venv_python = _venv_python(Path(upstream).expanduser().resolve())
|
||||
python = str(venv_python) if venv_python is not None else sys.executable
|
||||
|
||||
@@ -446,7 +292,7 @@ class ClaudeTextDetector:
|
||||
def all_detectors(
|
||||
markllm: MarkLLMTextDetector | None = None, *, include_markllm: bool = True
|
||||
) -> list[TextDetector]:
|
||||
detectors: list[TextDetector] = [GeminiSynthIDTextDetector()]
|
||||
detectors: list[TextDetector] = []
|
||||
if include_markllm:
|
||||
detectors.append(markllm or MarkLLMTextDetector())
|
||||
detectors.append(ClaudeTextDetector())
|
||||
|
||||
Reference in New Issue
Block a user