feat: model-free keyed-Gumbel (EXP) same-key detector (#190)

Implements the replay test of ARBI's keyed-Gumbel technical report
(u = PRF(Hash(key, window), token); exact Gamma-tail p-value; repeated-window
masking) as a stdlib-only detector (detect_gumbel.py) — no GPU, model, or
logits — wired into the TextDetector registry, /capabilities and the
iterative Layer B rewrite loop (--gumbel-key; priority gumbel > markllm >
lexical divergence) with a gumbel.before/after/cleared report.

Also cites the ARBI article in the README bibliography (renamed from
References) and names EXP/Gumbel in the open-LLM vendor row. Same-key-only:
valid against the same key, tokenizer, and PRF layout used at generation;
not a vendor oracle. Key is never logged.

Co-authored-by: guillaumemeyer <guillaumemeyer@users.noreply.github.com>
This commit is contained in:
Guillaume Meyer (The Opinionated Man)
2026-08-19 12:49:11 -07:00
committed by GitHub
co-authored by guillaumemeyer
parent 8c9ff345ed
commit 9dda608a86
7 changed files with 940 additions and 24 deletions
+55 -2
View File
@@ -21,7 +21,7 @@ Agent skill + stdlib Python service to strip **multi-vendor AI provenance marks*
| **B** | Statistical (token-sampling) text watermarks | Agent rewrite + optional `rewrite_text.py` hook |
| **Files** | C2PA / EXIF / XMP / doc props | PNG, JPEG, WebP, AVIF, HEIC, BMP, GIF, TIFF, SVG, PDF, DOCX, XLSX, PPTX, EPUB, ODT, HTML, Markdown, MP4/MOV/M4A/M4V, WAV, MP3 |
Vendors / ecosystems (class-level): **Claude**, **Gemini / SynthID-Text**, **OpenAI** provenance surfaces, **open-LLM** Kirchenbauer-style marks.
Vendors / ecosystems (class-level): **Claude**, **Gemini / SynthID-Text**, **OpenAI** provenance surfaces, **open-LLM** Kirchenbauer-style (green-list) and keyed-Gumbel / EXP (Aaronson) marks.
**Latest release:** [v0.5.0](https://github.com/guillaumemeyer/watermarks-remover/releases/tag/v0.5.0)
@@ -203,6 +203,7 @@ Text detectors (see `/capabilities` → `text_detectors`):
| Detector | Activated by | Notes |
| --- | --- | --- |
| `markllm` | `MARKLLM_DIR` (host checkout) | Research harness (KGW / SynthID schemes), same-config-only — not a vendor oracle. |
| `gumbel` | `WATERMARKS_GUMBEL_KEY` | Model-free same-key replay of the keyed-Gumbel (Aaronson EXP) scheme (see `detect_gumbel.py`), stdlib-only — self-hosted engines such as arbi-serve; same-key-only, not a vendor oracle. |
| `claude-text` | — (placeholder) | Anthropic has announced a watermark detection API; this seam activates when it ships. |
Image scoring: when `WATERMARKS_SYNTHID_SCORER_URL` is set, the service
@@ -290,6 +291,7 @@ set -a; . ./.env; set +a; python3 service/scripts/rewrite_text.py /tmp/x.txt -o
| `WATERMARKS_REWRITE_API_KEY` | `rewrite_text.py` hook | API key — env only, never on argv |
| `WATERMARKS_REWRITE_ALLOW_REMOTE` | `rewrite_text.py` hook | `1` to allow non-loopback endpoints |
| `WATERMARKS_REWRITE_REASONING_EFFORT` | `rewrite_text.py` hook | `none` (default) / `low` / `medium` / `high` / `off` |
| `WATERMARKS_GUMBEL_KEY` | `detect_gumbel.py` / `text_detectors.py` | Secret key for keyed-Gumbel (EXP) same-key replay (e.g. `0x…`); preferred over argv — never logged |
Layer B is agent-orchestrated in the skill (it rewrites with its own model), so the `WATERMARKS_REWRITE_*` vars are only needed when driving `rewrite_text.py` directly.
@@ -593,6 +595,47 @@ docker run --rm --user "$(id -u):$(id -g)" -v "$(pwd):/data" \
watermarks-remover-markllm detect /data/wm.txt --scheme kgw --json
```
### Keyed-Gumbel (Aaronson EXP) same-key verification
[ARBI's technical report](https://arbicity.com/news/ai-text-watermarking-for-self-hosted-ai/) describes the
keyed-Gumbel ("exponential") text watermark — now shipping in the open-source
arbi-serve engine (`ARBI_WATERMARK_KEY`) — where the sampler's noise is derived
from a keyed hash of the last 4-token context window. Detection is a
**model-free replay**: recompute `u = PRF(Hash(key, window), token)` from the
text alone and test the Gamma tail, so it needs no GPU, model, or logits.
This repo ships that detector as `detect_gumbel.py` (stdlib-only; the p-value
is the exact Poisson-sum identity for an integer Gamma shape):
```bash
# Text mode (deterministic word/run tokenizer) — quick checks and rewrite-loop
# evaluation; exact replay against a real engine needs its tokenizer:
python3 service/scripts/detect_gumbel.py draft.txt --key 0x... --json
# Exact replay: pass the engine's token ids (JSON array or one per line).
python3 service/scripts/detect_gumbel.py ids.json --tokens --key 0x... --json
```
Same honesty caveat as MarkLLM: this is a **same-key replay** — valid only
against the same key, tokenizer, and PRF layout used at generation, and a
negative result establishes nothing. The HMAC-SHA256 layout here is an
auditable instantiation, not bit-compatible with any specific engine kernel
(see the module docstring for what to adapt for exact replay).
**Detection-guided rewriting:** pass `--gumbel-key` to `rewrite_text.py`
(env: `WATERMARKS_GUMBEL_KEY`, preferred) and the iterative rewrite loop is
driven by the same-key Gumbel replay — evaluator priority becomes gumbel >
MarkLLM > lexical divergence — with a `gumbel.before/after/cleared` report:
```bash
export WATERMARKS_REWRITE_BACKEND=ollama WATERMARKS_REWRITE_MODEL=llama3.2
export WATERMARKS_GUMBEL_KEY=0x...
python3 "$SCRIPTS/rewrite_text.py" wm.txt -o wm.rewritten.txt --json-stats
```
The key never appears in stats or logs. Self-hosted operators who hold their
engine's key can verify a rewrite cleared a Gumbel mark; everyone else treats
Layer B as best-effort only.
## Optional SynthID-text removal benchmark
[`bench_synthid_text.py`](service/scripts/bench_synthid_text.py) measures how
@@ -901,6 +944,15 @@ make smoke # quick CLI smoke on fixtures
now carry attempts per document (`mean_attempts`, `att` column;
`attempts` / `evaluator` / `passed` columns); `--rewrite-loops`
mirrors `--max-loops`.
- **Keyed-Gumbel (Aaronson EXP) same-key verification**: new stdlib-only
`detect_gumbel.py` implements the model-free replay test of ARBI's keyed-Gumbel
report (u = PRF(Hash(key, window), token); exact Gamma-tail p-value; repeated-
window masking) — no GPU, model, or logits. `rewrite_text.py --gumbel-key`
(env `WATERMARKS_GUMBEL_KEY`, preferred) makes it the iterative-loop evaluator
(priority: gumbel > markllm > lexical divergence) with a `gumbel.before/after/
cleared` report; the detector is also exposed as `gumbel` in `/capabilities`
and `/detect`. Same-key-only: valid against the same key, tokenizer, and PRF
layout used at generation — not a vendor oracle. The key is never logged.
### [v0.5.0](https://github.com/guillaumemeyer/watermarks-remover/releases/tag/v0.5.0) — service & Docker distribution, HTTP API, and verification harnesses
@@ -1045,13 +1097,14 @@ make smoke # quick CLI smoke on fixtures
MIT — see [LICENSE](LICENSE).
## References
## Bibliography
- [How Claude marks AI-generated content](https://support.claude.com/en/articles/16266773-how-claude-marks-ai-generated-content) (Anthropic)
- Dathathri et al., [*Scalable watermarking for identifying large language model outputs*](https://www.nature.com/articles/s41586-024-08025-4) (SynthID-Text, Nature 2024)
- Google AI for Developers, [*SynthID safeguards*](https://ai.google.dev/responsible/docs/safeguards/synthid) (Gemini API docs)
- [C2PA](https://c2pa.org/) / [c2patool](https://github.com/contentauth/c2pa-rs/tree/main/cli)
- Kirchenbauer et al., [*A Watermark for Large Language Models*](https://arxiv.org/abs/2301.10226)
- Evseev, D. (Arbitration City), [*Accurate, Costless, and Invisible AI Text Watermarking for Self-Hosted AI Inference*](https://arbicity.com/news/ai-text-watermarking-for-self-hosted-ai/) (technical report, August 2026) — keyed-Gumbel watermarking shipped in the open-source arbi-serve engine, with exact-test detection and speculative-decoding support — [PDF](https://arbicity.com/news/ai-text-watermarking-for-self-hosted-ai/ARBI-Watermark-Technical-Paper.pdf)
- [THU-BPM/MarkLLM](https://github.com/THU-BPM/MarkLLM) (unified toolkit for evaluating LLM watermarking algorithms)
- Pan et al., [*MarkDiffusion: An Open-Source Toolkit for Generative Watermarking of Latent Diffusion Models*](https://arxiv.org/abs/2509.10569) (JMLR) — the embedding toolkit this repo's optional image-watermark harness wraps — [code](https://github.com/THU-BPM/MarkDiffusion), [docs](https://markdiffusion.readthedocs.io)
- Zhang et al., [*Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models*](https://arxiv.org/abs/2311.04378v5) (ICML 2024)
+308
View File
@@ -0,0 +1,308 @@
#!/usr/bin/env python3
"""Model-free keyed-Gumbel (Aaronson EXP) text-watermark detector.
Implements the detection arithmetic of the ARBI keyed-Gumbel technical report
(Sections 2-3): replay the keyed sampler's noise from the text alone and test
whether the observed tokens look like winners of keyed draws.
seed = Hash(key, last H tokens) H = 4 by default
u_t = PRF(seed, token_t) -> replayable from text alone
S = sum_t -log(1 - u_t) ~ Gamma(counted, 1) under the null
p = P(Gamma(counted, 1) >= S) exact for integer shape
Repeated context windows are masked (the generator falls back to ordinary
randomness on recurrence; the detector applies the same skip rule), so reused
windows contribute no evidence. Positions with fewer than H preceding tokens
have no full window and are not counted either.
Stdlib-only: the p-value uses the exact Poisson-sum identity for an integer
Gamma shape (upper regularized incomplete gamma), so scipy is never required.
Honesty caveat (same as the MarkLLM harness): detection is a *same-key replay*
it is valid only against the same key, tokenizer, and PRF layout used at
generation (self-hosted engines such as arbi-serve). A negative result
establishes nothing: unwatermarked text, another provider's key, and human
text all sit at chance. This detector is not a vendor oracle.
The default PRF layout here is HMAC-SHA256 over packed token ids. It is a
clean-room, auditable instantiation of the scheme, not bit-compatible with any
specific engine's kernel: for exact replay against a real engine, pass its
token ids (--tokens) and, if the engine uses a different PRF, reimplement the
two functions in this module accordingly.
"""
from __future__ import annotations
import argparse
import hashlib
import hmac
import json
import math
import os
import re
import struct
import sys
from collections.abc import Sequence
from pathlib import Path
from typing import Any
sys.path.insert(0, str(Path(__file__).resolve().parent))
from common import emit_json, eprint, read_text_input
DEFAULT_WINDOW = 4
DEFAULT_THRESHOLD = 1e-6
_ID_PACK = struct.Struct(">Q")
_SIMPLE_TOKEN_RE = re.compile(r"[A-Za-z0-9]+")
def _token_id(tok: str) -> int:
"""Deterministic 64-bit id for a simple-tokenizer token string."""
return int.from_bytes(hashlib.sha256(tok.encode("utf-8")).digest()[:8], "big")
def tokenize_simple(text: str) -> list[int]:
"""Deterministic word/run tokenizer -> stable token ids.
Convenience path for testing and quick checks. Exact replay against a real
engine requires the engine's own tokenizer: use --tokens with its ids.
"""
return [_token_id(t) for t in _SIMPLE_TOKEN_RE.findall(text.lower())]
def load_token_ids(raw: str) -> list[int]:
"""Parse a token-id input: a JSON array, or one integer per line."""
stripped = raw.strip()
if stripped.startswith("["):
data = json.loads(stripped)
if not isinstance(data, list) or not all(isinstance(x, int) for x in data):
raise ValueError("token-id JSON must be an array of integers")
return data
ids: list[int] = []
for raw_line in stripped.splitlines():
line = raw_line.strip()
if not line:
continue
ids.append(int(line, 0))
return ids
def _normalize_key(raw: str) -> bytes:
"""Key -> bytes: 0x<hex> decodes to raw bytes, anything else is UTF-8."""
s = raw.strip()
if s.startswith("0x") or s.startswith("0X"):
hexpart = s[2:]
if (
not hexpart
or len(hexpart) % 2
or any(c not in "0123456789abcdefABCDEF" for c in hexpart)
):
raise ValueError("invalid hex key (expected 0x followed by even-length hex)")
return bytes.fromhex(hexpart)
return s.encode("utf-8")
def _seed(key: bytes, window: tuple[int, ...]) -> bytes:
"""Context-window seed: HMAC-SHA256(key, packed window)."""
packed = b"".join(_ID_PACK.pack(t) for t in window)
return hmac.new(key, packed, hashlib.sha256).digest()
def _uniform(seed: bytes, token_id: int) -> float:
"""Per-candidate uniform in (0, 1): HMAC-SHA256(seed, token id)."""
digest = hmac.new(seed, _ID_PACK.pack(token_id), hashlib.sha256).digest()
return (int.from_bytes(digest[:8], "big") + 0.5) / (1 << 64)
def _poisson_survival(s: float, n: int) -> float:
"""P(Gamma(n, 1) >= s) = e^{-s} * sum_{k=0}^{n-1} s^k / k! (exact for integer n).
Computed via logsumexp so large statistics never overflow or NaN.
"""
if n <= 0 or s <= 0.0:
return 1.0
lns = math.log(s)
log_term = 0.0 # k = 0 term is s^0 / 0! = 1
maxv = 0.0
for k in range(1, n):
log_term += lns - math.log(k)
if log_term > maxv:
maxv = log_term
log_term = 0.0
acc = math.exp(-maxv) # k = 0 term
for k in range(1, n):
log_term += lns - math.log(k)
acc += math.exp(log_term - maxv)
p = math.exp(-s + maxv + math.log(acc))
if p < 0.0:
return 0.0
if p > 1.0:
return 1.0
return p
def detect_token_ids(
token_ids: Sequence[int],
key: str | bytes,
*,
window: int = DEFAULT_WINDOW,
threshold: float = DEFAULT_THRESHOLD,
mask_repeated: bool = True,
) -> dict[str, Any]:
"""Run the keyed-Gumbel replay test over a sequence of token ids.
mask_repeated mirrors the generator's repeated-window masking: a context
window is counted only on its first occurrence (the generator fell back to
ordinary randomness on recurrence, so later occurrences carry no signal).
Positions with fewer than window preceding tokens are never counted.
"""
if window < 1:
raise ValueError("window must be >= 1")
if not 0.0 < threshold < 1.0:
raise ValueError("threshold must be in (0, 1)")
key_bytes = _normalize_key(key) if isinstance(key, str) else bytes(key)
ids = list(token_ids)
for t in ids:
if not isinstance(t, int) or isinstance(t, bool) or t < 0 or t > (1 << 64) - 1:
raise ValueError(f"token id out of range: {t!r}")
total = len(ids)
statistic = 0.0
counted = 0
seen_windows: set[tuple[int, ...]] = set()
skipped_repeated = 0
for t in range(window, total):
win = tuple(ids[t - window : t])
if mask_repeated:
if win in seen_windows:
skipped_repeated += 1
continue
seen_windows.add(win)
u = _uniform(_seed(key_bytes, win), ids[t])
statistic += -math.log1p(-u)
counted += 1
skipped_no_context = min(window, total)
report: dict[str, Any] = {
"detector": "gumbel",
"scheme": "exp",
"vendor": "self-hosted",
"available": True,
"window": window,
"threshold": threshold,
"tokens_total": total,
"skipped_no_context": skipped_no_context,
"skipped_repeated": skipped_repeated,
"counted": counted,
}
if counted == 0:
report["is_watermarked"] = False
report["p_value"] = 1.0
report["score"] = 0.0
report["note"] = "no verifiable token positions (text too short for a full context window)"
return report
p = _poisson_survival(statistic, counted)
report["statistic"] = round(statistic, 6)
report["p_value"] = p
report["score"] = round(-math.log10(p) if p > 0.0 else 300.0, 6)
report["is_watermarked"] = p < threshold
report["note"] = (
"same-key replay of the keyed-Gumbel (Aaronson EXP) watermark; valid only "
"against the same key, tokenizer, and PRF layout used at generation"
)
return report
def detect_text(
text: str,
key: str | bytes,
*,
window: int = DEFAULT_WINDOW,
threshold: float = DEFAULT_THRESHOLD,
) -> dict[str, Any]:
"""Run the replay test over plain text via the deterministic tokenizer."""
return detect_token_ids(tokenize_simple(text), key, window=window, threshold=threshold)
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(description=__doc__)
p.add_argument("path", nargs="?", default="-", help="Text file, or - for stdin")
(
p.add_argument(
"--tokens",
action="store_true",
help="Treat the input as token ids (JSON array or one integer per line) "
"instead of text — required for exact replay with an engine's tokenizer",
),
)
(
p.add_argument(
"--key",
default=os.environ.get("WATERMARKS_GUMBEL_KEY"),
help="Watermark key (0x<hex> or a string); default: $WATERMARKS_GUMBEL_KEY. "
"Preferred via env — keys on argv are visible in ps/history.",
),
)
(
p.add_argument(
"--window",
type=int,
default=DEFAULT_WINDOW,
help=f"Context window size H in tokens (default: {DEFAULT_WINDOW})",
),
)
(
p.add_argument(
"--threshold",
type=float,
default=DEFAULT_THRESHOLD,
help=f"p-value threshold for is_watermarked (default: {DEFAULT_THRESHOLD:g})",
),
)
p.add_argument("--json", action="store_true", help="Emit the report as JSON on stdout")
(
p.add_argument(
"--force-text",
action="store_true",
help="Process input even when it looks like a binary container",
),
)
return p
def main() -> int:
args = build_parser().parse_args()
if not args.key:
eprint("error: no watermark key (pass --key or set WATERMARKS_GUMBEL_KEY)")
return 2
raw = read_text_input(args.path, allow_binary=args.force_text)
try:
if args.tokens:
report = detect_token_ids(
load_token_ids(raw),
args.key,
window=args.window,
threshold=args.threshold,
)
else:
report = detect_text(raw, args.key, window=args.window, threshold=args.threshold)
except (ValueError, json.JSONDecodeError) as e:
eprint(f"error: {e}")
return 2
if args.json:
emit_json(report)
else:
print(
f"keyed-Gumbel (EXP) detection: watermarked={report['is_watermarked']} "
f"p={report['p_value']:.3g} (threshold {report['threshold']:g}) "
f"counted={report['counted']}/{report['tokens_total']} tokens, skipped "
f"{report['skipped_no_context'] + report['skipped_repeated']} "
f"(no-context {report['skipped_no_context']}, repeated-window "
f"{report['skipped_repeated']})"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
+59 -13
View File
@@ -19,12 +19,13 @@ Rewriting is iterative and evaluation-driven: each loop generates
--candidates (default 1) variants, evaluates each, and stops as soon as an
attempt passes watermark detection; --max-loops (default 1) caps how many
evaluation rounds run before the best-effort variant is returned
(WATERMARKS_REWRITE_LOOPS). The evaluator is chosen by priority: MarkLLM
same-config detection (when --markllm-scheme is passed) or, when no detector
is configured, bigram-Jaccard lexical divergence (no pass/fail verdict all
(WATERMARKS_REWRITE_LOOPS). The evaluator is chosen by priority: keyed-Gumbel
same-key replay (when --gumbel-key / WATERMARKS_GUMBEL_KEY is set), else
MarkLLM same-config detection (--markllm-scheme), else, when no detector is
configured, bigram-Jaccard lexical divergence (no pass/fail verdict all
attempts are generated and the most diverged one is selected). A vendor-detector
seam (Google's retired SynthID-text detector) is reserved ahead of MarkLLM
should a vendor endpoint return.
seam (Google's retired SynthID-text detector) is reserved ahead of the
same-config detectors should a vendor endpoint return.
Security notes:
- Only http(s) endpoints are accepted; redirects are refused outright so an
@@ -49,7 +50,7 @@ from urllib.parse import urlparse
sys.path.insert(0, str(Path(__file__).resolve().parent))
from common import cleaned_path, eprint, read_text_input, write_text_output
from text_detectors import MarkLLMTextDetector
from text_detectors import GumbelTextDetector, MarkLLMTextDetector
from text_unicode import clean_text
DEFAULT_MARKLLM_MODEL = "facebook/opt-1.3b"
@@ -205,16 +206,20 @@ def _safe_detect(detector: object, text: str) -> dict:
def _pick_evaluator(
markllm_detector: MarkLLMTextDetector | None,
gumbel_detector: GumbelTextDetector | None,
) -> tuple[str, object | None]:
"""Pick the evaluator that drives the iterative rewrite loop.
Priority: MarkLLM same-config detection (when the caller passed
--markllm-scheme) > bigram-Jaccard lexical divergence (fallback with no
pass/fail verdict). A vendor-detector seam (Google's SynthID-text detector,
retired Aug 2026) is reserved ahead of MarkLLM should a vendor endpoint
return; it only needs available()/detect()/name per the TextDetector
protocol in text_detectors.py.
Priority: keyed-Gumbel same-key replay (when the caller passed
--gumbel-key) > MarkLLM same-config detection (--markllm-scheme) >
bigram-Jaccard lexical divergence (fallback with no pass/fail verdict).
A vendor-detector seam (Google's SynthID-text detector, retired Aug 2026)
is reserved ahead of both should a vendor endpoint return; it only needs
available()/detect()/name per the TextDetector protocol in
text_detectors.py.
"""
if gumbel_detector is not None:
return "gumbel", gumbel_detector
if markllm_detector is not None:
return "markllm", markllm_detector
return "lexical-divergence", None
@@ -356,6 +361,7 @@ def rewrite(
markllm_dir: str | None = None,
markllm_model: str | None = None,
markllm_timeout: float = 180.0,
gumbel_key: str | None = None,
) -> tuple[str, dict]:
prompt = build_prompt(strength, text, lang=lang, original_lang=original_lang)
info: dict = {
@@ -387,6 +393,15 @@ def rewrite(
eprint(f"markllm verification unavailable: {markllm['before']['error']}")
info["markllm"] = markllm
gumbel: dict | None = None
gumbel_detector: GumbelTextDetector | None = None
if gumbel_key:
gumbel_detector = GumbelTextDetector(key=gumbel_key)
gumbel = {"before": _safe_detect(gumbel_detector, text)}
if not gumbel["before"]["available"]:
eprint(f"gumbel verification unavailable: {gumbel['before']['error']}")
info["gumbel"] = gumbel
if backend == "print-prompt":
info["mode"] = "print-prompt"
if candidates > 1:
@@ -404,7 +419,7 @@ def rewrite(
n_loops = max(1, max_loops)
info["candidates"] = n_cands
info["max_loops"] = n_loops
evaluator_name, evaluator = _pick_evaluator(markllm_detector)
evaluator_name, evaluator = _pick_evaluator(markllm_detector, gumbel_detector)
info["evaluator"] = evaluator_name
# Iterative rewrite: each loop generates --candidates variants and
@@ -522,6 +537,28 @@ def rewrite(
"keys used at generation; it does not certify a vendor detector."
)
if gumbel:
assert gumbel_detector is not None # set together with gumbel above
if evaluator_name == "gumbel":
# The loop already scored the selected attempt; reuse the verdict
# instead of paying another replay.
after = rec["evaluation"]
else:
after = _safe_detect(gumbel_detector, out)
gumbel["after"] = after
before = gumbel["before"]
if before.get("available") and after.get("available"):
gumbel["cleared"] = bool(
before.get("is_watermarked") and not after.get("is_watermarked")
)
else:
gumbel["cleared"] = None
gumbel["note"] = (
"Keyed-Gumbel detection is a same-key replay: valid only with the "
"same key, tokenizer, and PRF layout used at generation; it does "
"not certify a vendor detector."
)
eprint(
f"note: evaluator={evaluator_name} attempts={len(attempts)}/"
f"{n_cands * n_loops} loops={n_loops} passed={passed}"
@@ -623,6 +660,14 @@ def build_parser() -> argparse.ArgumentParser:
default=float(_env("WATERMARKS_MARKLLM_TIMEOUT", "180.0")),
help="Timeout per MarkLLM detection call (default: 180.0)",
)
p.add_argument(
"--gumbel-key",
default=_env("WATERMARKS_GUMBEL_KEY"),
help="Secret key for keyed-Gumbel (Aaronson EXP) same-key replay "
"detection; drives the iterative rewrite loop as the evaluator when "
"set (default: $WATERMARKS_GUMBEL_KEY). Preferred via env — keys on "
"argv are visible in ps/history; never logged.",
)
p.add_argument(
"--force-text",
action="store_true",
@@ -661,6 +706,7 @@ def main() -> int:
markllm_dir=args.markllm_dir,
markllm_model=args.markllm_model,
markllm_timeout=args.markllm_timeout,
gumbel_key=args.gumbel_key,
)
except (urllib.error.URLError, TimeoutError, RuntimeError) as e:
eprint(f"rewrite failed: {e}")
+102 -4
View File
@@ -17,6 +17,10 @@ Detectors:
- markllm research harness (KGW / SynthID schemes) via
detect_text_watermark.py, activated by MARKLLM_DIR. Same-config-only
detection; not a vendor oracle.
- gumbel model-free same-key replay of the keyed-Gumbel (Aaronson EXP)
scheme (detect_gumbel.py), activated by WATERMARKS_GUMBEL_KEY. Stdlib-only;
valid only against the same key, tokenizer, and PRF layout used at
generation (self-hosted engines such as arbi-serve); not a vendor oracle.
- claude-text placeholder for Anthropic's announced text-watermark
detection API. Reports unavailable until a public endpoint exists; the
interface it must implement is already defined here.
@@ -40,6 +44,8 @@ from collections.abc import Callable
from pathlib import Path
from typing import Any, Protocol
from detect_gumbel import DEFAULT_THRESHOLD, DEFAULT_WINDOW, detect_text
DEFAULT_MARKLLM_SCHEME = "kgw"
DEFAULT_MARKLLM_TIMEOUT = 600.0
@@ -251,6 +257,75 @@ class MarkLLMTextDetector:
return payload
# ---------------------------------------------------------------------------
# Keyed-Gumbel (Aaronson EXP) — model-free same-key replay
# ---------------------------------------------------------------------------
class GumbelTextDetector:
"""Same-key replay of the keyed-Gumbel (Aaronson EXP) text watermark.
Model-free: implements the detection arithmetic of the ARBI keyed-Gumbel
technical report (Sections 2-3) replay u = PRF(Hash(key, window), token)
from the text alone and test the Gamma tail so it needs no GPU, model,
or logits. Detection is valid only against the SAME key, tokenizer, and
PRF layout used at generation (self-hosted engines such as arbi-serve);
it is not a vendor oracle. Key from WATERMARKS_GUMBEL_KEY (env) or the
constructor override.
"""
name = "gumbel"
vendor = "self-hosted"
def __init__(
self,
*,
key: str | None = None,
window: int | None = None,
threshold: float | None = None,
) -> None:
self._key = key
self._window = window
self._threshold = threshold
def _key_env(self) -> str | None:
if self._key:
return self._key
return os.environ.get("WATERMARKS_GUMBEL_KEY", "").strip() or None
def available(self) -> bool:
return self._key_env() is not None
def detect(self, text: str) -> dict[str, Any]:
key = self._key_env()
report: dict[str, Any] = {
"detector": self.name,
"scheme": "exp",
"vendor": self.vendor,
"available": False,
}
if key is None:
report["error"] = "WATERMARKS_GUMBEL_KEY not set"
return report
try:
payload = detect_text(
text,
key,
window=self._window or DEFAULT_WINDOW,
threshold=self._threshold or DEFAULT_THRESHOLD,
)
except Exception as e: # fail-soft contract: never raise
report["error"] = f"keyed-Gumbel detection failed: {e}"
return report
payload["detector"] = self.name
payload["note"] = (
"same-key replay of the keyed-Gumbel (Aaronson EXP) watermark: valid "
"only with the same key, tokenizer, and PRF layout used at generation; "
"not a vendor detector."
)
return payload
# ---------------------------------------------------------------------------
# Claude (Anthropic) — announced detector API, not yet public
# ---------------------------------------------------------------------------
@@ -290,11 +365,17 @@ class ClaudeTextDetector:
def all_detectors(
markllm: MarkLLMTextDetector | None = None, *, include_markllm: bool = True
markllm: MarkLLMTextDetector | None = None,
*,
include_markllm: bool = True,
gumbel: GumbelTextDetector | None = None,
include_gumbel: bool = True,
) -> list[TextDetector]:
detectors: list[TextDetector] = []
if include_markllm:
detectors.append(markllm or MarkLLMTextDetector())
if include_gumbel:
detectors.append(gumbel or GumbelTextDetector())
detectors.append(ClaudeTextDetector())
return detectors
@@ -309,14 +390,24 @@ def run_all_text_detectors(
*,
markllm: MarkLLMTextDetector | None = None,
include_markllm: bool = True,
gumbel: GumbelTextDetector | None = None,
include_gumbel: bool = True,
) -> list[dict[str, Any]]:
"""Run every detector (including unavailable ones, with reasons).
markllm injects a caller-parameterized MarkLLM detector (e.g. one
driven by rewrite_text.py CLI flags); pass include_markllm=False to
exclude the MarkLLM harness entirely.
exclude the MarkLLM harness entirely. Same for gumbel.
"""
return [d.detect(text) for d in all_detectors(markllm, include_markllm=include_markllm)]
return [
d.detect(text)
for d in all_detectors(
markllm,
include_markllm=include_markllm,
gumbel=gumbel,
include_gumbel=include_gumbel,
)
]
def run_text_detectors(
@@ -324,10 +415,17 @@ def run_text_detectors(
*,
markllm: MarkLLMTextDetector | None = None,
include_markllm: bool = True,
gumbel: GumbelTextDetector | None = None,
include_gumbel: bool = True,
) -> list[dict[str, Any]]:
"""Run only the detectors that are configured and usable."""
return [
d.detect(text)
for d in all_detectors(markllm, include_markllm=include_markllm)
for d in all_detectors(
markllm,
include_markllm=include_markllm,
gumbel=gumbel,
include_gumbel=include_gumbel,
)
if d.available()
]
+1 -1
View File
@@ -107,7 +107,7 @@ def _clean_env(monkeypatch):
def test_capabilities_exposes_detectors(conn):
status, body = _get(conn, "/capabilities")
assert status == 200
assert set(body["text_detectors"]) == {"markllm", "claude-text"}
assert set(body["text_detectors"]) == {"markllm", "gumbel", "claude-text"}
assert "synthid_http" in body["scorers"]
+411
View File
@@ -0,0 +1,411 @@
"""Tests for the model-free keyed-Gumbel (Aaronson EXP) detector.
Exercises the replay arithmetic of detect_gumbel.py with a toy keyed
Gumbel-max generator, the TextDetector protocol in text_detectors.py, and
the gumbel evaluator wiring in rewrite_text.py.
"""
from __future__ import annotations
import json
import math
import os
import random
import subprocess
import sys
from pathlib import Path
import pytest
ROOT = Path(__file__).resolve().parents[1]
SCRIPTS = ROOT / "service" / "scripts"
sys.path.insert(0, str(SCRIPTS))
import detect_gumbel
import rewrite_text
import text_detectors
KEY_HEX = "0x" + "ab" * 16 # 16 raw bytes
KEY_HEX_OTHER = "0x" + "cd" * 16
KEY_BYTES = detect_gumbel._normalize_key(KEY_HEX)
# S311: deterministic toy RNG for the sampler tests — never used for secrets.
def _rng(seed: int) -> random.Random:
return random.Random(seed) # noqa: S311
_WORDS = [f"w{i}" for i in range(64)]
_WORD_IDS = [detect_gumbel._token_id(w) for w in _WORDS]
@pytest.fixture(autouse=True)
def _clean_env(monkeypatch):
monkeypatch.delenv("WATERMARKS_GUMBEL_KEY", raising=False)
# --- Toy keyed Gumbel-max generator (mirrors paper Section 2) ----------------
def _gumbel(u: list[float]) -> list[float]:
return [-math.log(-math.log(x)) for x in u]
def _keyed_uniforms(key_bytes: bytes, window_ids: tuple[int, ...], vocab: list[int]) -> list[float]:
seed = detect_gumbel._seed(key_bytes, window_ids)
return [detect_gumbel._uniform(seed, v) for v in vocab]
def _generate(
key_bytes: bytes | None,
*,
n: int,
vocab: int,
window: int,
rng: random.Random,
) -> list[int]:
"""Keyed (or RNG) Gumbel-max sampler with generator-side window masking."""
logits = [rng.uniform(-1.0, 1.0) for _ in range(vocab)]
ids: list[int] = []
seen: set[tuple[int, ...]] = set()
for _ in range(n):
win = tuple(ids[-window:])
use_keyed = key_bytes is not None and len(win) == window and win not in seen
if use_keyed:
seen.add(win)
if use_keyed:
u = _keyed_uniforms(key_bytes, win, list(range(vocab)))
else:
u = [rng.random() for _ in range(vocab)]
g = _gumbel(u)
ids.append(max(range(vocab), key=lambda v: logits[v] + g[v]))
return ids
def _marked_text(n: int, rng: random.Random, key_hex: str = KEY_HEX) -> str:
"""Marked *text* whose simple-tokenizer ids replay under the same key.
Generates over the word->id space detect_gumbel uses for plain text, so
the real detector re-derives identical seeds and uniforms.
"""
key_bytes = detect_gumbel._normalize_key(key_hex)
logits = [rng.uniform(-1.0, 1.0) for _ in _WORDS]
seq_ids: list[int] = []
seq_words: list[str] = []
seen: set[tuple[int, ...]] = set()
for _ in range(n):
win = tuple(seq_ids[-4:])
use_keyed = len(win) == 4 and win not in seen
if use_keyed:
seen.add(win)
if use_keyed:
u = _keyed_uniforms(key_bytes, win, _WORD_IDS)
else:
u = [rng.random() for _ in _WORD_IDS]
g = _gumbel(u)
chosen = max(range(len(_WORD_IDS)), key=lambda v: logits[v] + g[v])
seq_ids.append(_WORD_IDS[chosen])
seq_words.append(_WORDS[chosen])
return " ".join(seq_words)
# --- Replay arithmetic -------------------------------------------------------
def test_marked_sequence_is_detected():
rng = _rng(1)
ids = _generate(KEY_BYTES, n=700, vocab=32, window=4, rng=rng)
report = detect_gumbel.detect_token_ids(ids, KEY_HEX)
assert report["available"] is True
assert report["is_watermarked"] is True
assert report["p_value"] < 1e-12
def test_unmarked_sequence_at_chance():
rng = _rng(2)
ids = _generate(None, n=700, vocab=32, window=4, rng=rng)
report = detect_gumbel.detect_token_ids(ids, KEY_HEX)
assert report["is_watermarked"] is False
assert report["p_value"] > 0.01
def test_wrong_key_at_chance():
rng = _rng(3)
ids = _generate(KEY_BYTES, n=700, vocab=32, window=4, rng=rng)
report = detect_gumbel.detect_token_ids(ids, KEY_HEX_OTHER)
assert report["is_watermarked"] is False
assert report["p_value"] > 0.01
def test_marked_text_roundtrip_through_simple_tokenizer():
rng = _rng(4)
text = _marked_text(700, rng)
report = detect_gumbel.detect_text(text, KEY_HEX)
assert report["is_watermarked"] is True
assert report["p_value"] < 1e-9
# the same text under a different key sits at chance
other = detect_gumbel.detect_text(text, KEY_HEX_OTHER)
assert other["is_watermarked"] is False
assert other["p_value"] > 0.01
def test_detection_is_deterministic():
rng = _rng(5)
text = _marked_text(300, rng)
assert detect_gumbel.detect_text(text, KEY_HEX) == detect_gumbel.detect_text(text, KEY_HEX)
def test_repeated_window_masking_skip_rule():
ids = [0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1]
report = detect_gumbel.detect_token_ids(ids, KEY_HEX, window=2)
# positions 0-1 lack context; windows (0,1)/(1,0) are keyed once each and
# every later occurrence is a repeated window the detector must skip
assert report["skipped_no_context"] == 2
assert report["skipped_repeated"] == 16
assert report["counted"] == 2
assert report["is_watermarked"] is False # two tokens carry no evidence
# without masking every full-window position is counted
report2 = detect_gumbel.detect_token_ids(ids, KEY_HEX, window=2, mask_repeated=False)
assert report2["counted"] == 18
assert report2["skipped_repeated"] == 0
def test_short_text_counts_nothing():
report = detect_gumbel.detect_text("a b c", KEY_HEX)
assert report["counted"] == 0
assert report["is_watermarked"] is False
assert report["p_value"] == 1.0
def test_key_normalization():
assert detect_gumbel._normalize_key("0x" + "ab" * 8) == b"\xab" * 8
assert detect_gumbel._normalize_key("0xAB12") == b"\xab\x12"
assert detect_gumbel._normalize_key("s3cret key!") == b"s3cret key!"
with pytest.raises(ValueError):
detect_gumbel._normalize_key("0xzz")
with pytest.raises(ValueError):
detect_gumbel._normalize_key("0xabc")
def test_poisson_survival_sanity():
# Gamma(1,1) survival at s=1 is e^-1; at s=n the p-value is ~0.5
assert detect_gumbel._poisson_survival(1.0, 1) == pytest.approx(math.exp(-1.0))
assert detect_gumbel._poisson_survival(10.0, 10) == pytest.approx(0.5, abs=0.05)
assert detect_gumbel._poisson_survival(50.0, 10) < 1e-6
assert detect_gumbel._poisson_survival(0.0, 10) == 1.0
def test_poisson_survival_matches_scipy_gammaincc():
pytest.importorskip("scipy.special")
from scipy.special import gammaincc
for n in (1, 5, 50, 200):
for s in (n * 0.5, n, n * 1.5, n * 2.0):
ours = detect_gumbel._poisson_survival(s, n)
ref = float(gammaincc(n, s))
assert ours == pytest.approx(ref, rel=1e-6, abs=1e-300)
def test_load_token_ids_json_and_lines():
assert detect_gumbel.load_token_ids("[1, 2, 3]") == [1, 2, 3]
assert detect_gumbel.load_token_ids("1\n0x2\n3") == [1, 2, 3]
with pytest.raises(ValueError):
detect_gumbel.load_token_ids('[1, "x"]')
def test_token_id_out_of_range_rejected():
with pytest.raises(ValueError):
detect_gumbel.detect_token_ids([-1, 2, 3], KEY_HEX)
with pytest.raises(ValueError):
detect_gumbel.detect_token_ids([1 << 64, 2], KEY_HEX)
# --- CLI ---------------------------------------------------------------------
def _run_cli(
*args: str, env_extra: dict[str, str] | None = None
) -> subprocess.CompletedProcess[str]:
env = {k: v for k, v in os.environ.items() if k != "WATERMARKS_GUMBEL_KEY"}
env.update(env_extra or {})
return subprocess.run(
[sys.executable, str(SCRIPTS / "detect_gumbel.py"), *args],
capture_output=True,
text=True,
timeout=120,
env=env,
check=False,
)
def test_cli_detects_marked_text(tmp_path):
rng = _rng(6)
src = tmp_path / "marked.txt"
src.write_text(_marked_text(500, rng), encoding="utf-8")
r = _run_cli(str(src), "--key", KEY_HEX, "--json")
assert r.returncode == 0, r.stderr
report = json.loads(r.stdout)
assert report["is_watermarked"] is True
def test_cli_token_ids_mode(tmp_path):
rng = _rng(7)
ids = _generate(KEY_BYTES, n=500, vocab=32, window=4, rng=rng)
src = tmp_path / "ids.json"
src.write_text(json.dumps(ids), encoding="utf-8")
r = _run_cli(str(src), "--tokens", "--key", KEY_HEX, "--json")
assert r.returncode == 0, r.stderr
report = json.loads(r.stdout)
assert report["is_watermarked"] is True
def test_cli_requires_key(tmp_path):
src = tmp_path / "plain.txt"
src.write_text("hello world", encoding="utf-8")
r = _run_cli(str(src))
assert r.returncode == 2
assert "key" in (r.stderr + r.stdout).lower()
def test_cli_reads_key_from_env(tmp_path):
rng = _rng(8)
src = tmp_path / "marked.txt"
src.write_text(_marked_text(400, rng), encoding="utf-8")
r = _run_cli(str(src), "--json", env_extra={"WATERMARKS_GUMBEL_KEY": KEY_HEX})
assert r.returncode == 0, r.stderr
assert json.loads(r.stdout)["is_watermarked"] is True
def test_cli_bad_hex_key(tmp_path):
src = tmp_path / "plain.txt"
src.write_text("hello world", encoding="utf-8")
r = _run_cli(str(src), "--key", "0xzz", "--json")
assert r.returncode == 2
# --- GumbelTextDetector protocol ---------------------------------------------
def test_gumbel_detector_unconfigured():
assert text_detectors.GumbelTextDetector().available() is False
report = text_detectors.GumbelTextDetector().detect("hello")
assert report["available"] is False
assert "WATERMARKS_GUMBEL_KEY" in report["error"]
def test_gumbel_detector_env_key(monkeypatch):
monkeypatch.setenv("WATERMARKS_GUMBEL_KEY", KEY_HEX)
rng = _rng(9)
report = text_detectors.GumbelTextDetector().detect(_marked_text(400, rng))
assert report["available"] is True
assert report["is_watermarked"] is True
assert report["detector"] == "gumbel"
def test_gumbel_detector_constructor_key():
rng = _rng(10)
report = text_detectors.GumbelTextDetector(key=KEY_HEX).detect(_marked_text(400, rng))
assert report["available"] is True
assert report["is_watermarked"] is True
def test_gumbel_in_detector_registry(monkeypatch):
names = {d.name for d in text_detectors.all_detectors()}
assert "gumbel" in names
monkeypatch.setenv("WATERMARKS_GUMBEL_KEY", KEY_HEX)
assert text_detectors.detector_status()["gumbel"] is True
# --- rewrite_text.py gumbel evaluator ----------------------------------------
def _rewrite_kwargs(**overrides):
kwargs = dict(
backend="ollama",
model="m",
base_url="http://127.0.0.1:11434",
api_key=None,
strength="paraphrase",
lang="French",
original_lang="English",
timeout=10,
layer_a_after=False,
temperature=0.9,
candidates=1,
)
kwargs.update(overrides)
return kwargs
def test_rewrite_gumbel_evaluator_clears_mark(monkeypatch):
rng = _rng(11)
original = _marked_text(400, rng)
monkeypatch.setattr(
rewrite_text,
"call_ollama",
lambda *a, **k: "alpha beta gamma delta epsilon zeta eta theta",
)
_out, info = rewrite_text.rewrite(original, **_rewrite_kwargs(gumbel_key=KEY_HEX))
assert info["evaluator"] == "gumbel"
assert info["passed"] is True
assert info["attempts_made"] == 1
g = info["gumbel"]
assert g["before"]["is_watermarked"] is True
assert g["after"]["is_watermarked"] is False
assert g["cleared"] is True
assert info["candidate_scores"][0]["evaluation"]["detector"] == "gumbel"
assert _out == "alpha beta gamma delta epsilon zeta eta theta"
def test_gumbel_takes_priority_over_markllm(monkeypatch):
built = {}
class _FakeMarkLLM:
def __init__(self, **kwargs):
built["markllm"] = True
def available(self):
return True
def detect(self, text):
return {"detector": "markllm", "available": True, "is_watermarked": False, "score": 0.0}
monkeypatch.setattr(rewrite_text, "MarkLLMTextDetector", _FakeMarkLLM)
monkeypatch.setattr(rewrite_text, "call_ollama", lambda *a, **k: "alpha beta gamma delta")
_out, info = rewrite_text.rewrite(
"plain text here",
**_rewrite_kwargs(gumbel_key=KEY_HEX, markllm_scheme="kgw", markllm_dir="/x"),
)
assert built["markllm"] is True
assert info["evaluator"] == "gumbel"
assert "markllm" in info and "gumbel" in info
assert info["passed"] is True
def test_gumbel_key_does_not_build_markllm(monkeypatch):
monkeypatch.setattr(
rewrite_text,
"MarkLLMTextDetector",
lambda *a, **k: pytest.fail("markllm must not be built when only gumbel is set"),
)
monkeypatch.setattr(rewrite_text, "call_ollama", lambda *a, **k: "alpha beta gamma delta")
_out, info = rewrite_text.rewrite("plain text here", **_rewrite_kwargs(gumbel_key=KEY_HEX))
assert info["evaluator"] == "gumbel"
assert info["passed"] is True
assert "markllm" not in info
def test_no_gumbel_without_key(monkeypatch):
monkeypatch.setattr(rewrite_text, "call_ollama", lambda *a, **k: "alpha beta gamma delta")
_out, info = rewrite_text.rewrite("plain text here", **_rewrite_kwargs())
assert "gumbel" not in info
assert info["evaluator"] == "lexical-divergence"
def test_gumbel_key_flag_defaults_from_env(monkeypatch):
monkeypatch.setenv("WATERMARKS_GUMBEL_KEY", KEY_HEX)
args = rewrite_text.build_parser().parse_args(["x.txt"])
assert args.gumbel_key == KEY_HEX
monkeypatch.delenv("WATERMARKS_GUMBEL_KEY", raising=False)
args = rewrite_text.build_parser().parse_args(["x.txt"])
assert args.gumbel_key is None
+4 -4
View File
@@ -248,8 +248,8 @@ def test_run_all_text_detectors_can_exclude_markllm(monkeypatch):
lambda: pytest.fail("must not construct MarkLLM when excluded"),
)
reports = text_detectors.run_all_text_detectors("hello", include_markllm=False)
assert len(reports) == 1 # claude placeholder only
assert {r["detector"] for r in reports} == {"claude-text"}
assert len(reports) == 2 # gumbel + claude placeholder
assert {r["detector"] for r in reports} == {"gumbel", "claude-text"}
def test_run_all_text_detectors_injects_markllm_instance(monkeypatch, tmp_path):
@@ -284,12 +284,12 @@ def test_claude_placeholder():
def test_detector_status_keys():
status = text_detectors.detector_status()
assert set(status) == {"markllm", "claude-text"}
assert set(status) == {"markllm", "gumbel", "claude-text"}
def test_run_all_text_detectors_length():
reports = text_detectors.run_all_text_detectors("hello")
assert len(reports) == 2
assert len(reports) == 3 # markllm + gumbel + claude placeholder
assert all("detector" in r for r in reports)