mirror of
https://github.com/guillaumemeyer/watermarks-remover.git
synced 2026-08-22 13:11:57 +02:00
feat: model-free keyed-Gumbel (EXP) same-key detector (#190)
Implements the replay test of ARBI's keyed-Gumbel technical report (u = PRF(Hash(key, window), token); exact Gamma-tail p-value; repeated-window masking) as a stdlib-only detector (detect_gumbel.py) — no GPU, model, or logits — wired into the TextDetector registry, /capabilities and the iterative Layer B rewrite loop (--gumbel-key; priority gumbel > markllm > lexical divergence) with a gumbel.before/after/cleared report. Also cites the ARBI article in the README bibliography (renamed from References) and names EXP/Gumbel in the open-LLM vendor row. Same-key-only: valid against the same key, tokenizer, and PRF layout used at generation; not a vendor oracle. Key is never logged. Co-authored-by: guillaumemeyer <guillaumemeyer@users.noreply.github.com>
This commit is contained in:
co-authored by
guillaumemeyer
parent
8c9ff345ed
commit
9dda608a86
@@ -21,7 +21,7 @@ Agent skill + stdlib Python service to strip **multi-vendor AI provenance marks*
|
||||
| **B** | Statistical (token-sampling) text watermarks | Agent rewrite + optional `rewrite_text.py` hook |
|
||||
| **Files** | C2PA / EXIF / XMP / doc props | PNG, JPEG, WebP, AVIF, HEIC, BMP, GIF, TIFF, SVG, PDF, DOCX, XLSX, PPTX, EPUB, ODT, HTML, Markdown, MP4/MOV/M4A/M4V, WAV, MP3 |
|
||||
|
||||
Vendors / ecosystems (class-level): **Claude**, **Gemini / SynthID-Text**, **OpenAI** provenance surfaces, **open-LLM** Kirchenbauer-style marks.
|
||||
Vendors / ecosystems (class-level): **Claude**, **Gemini / SynthID-Text**, **OpenAI** provenance surfaces, **open-LLM** Kirchenbauer-style (green-list) and keyed-Gumbel / EXP (Aaronson) marks.
|
||||
|
||||
**Latest release:** [v0.5.0](https://github.com/guillaumemeyer/watermarks-remover/releases/tag/v0.5.0)
|
||||
|
||||
@@ -203,6 +203,7 @@ Text detectors (see `/capabilities` → `text_detectors`):
|
||||
| Detector | Activated by | Notes |
|
||||
| --- | --- | --- |
|
||||
| `markllm` | `MARKLLM_DIR` (host checkout) | Research harness (KGW / SynthID schemes), same-config-only — not a vendor oracle. |
|
||||
| `gumbel` | `WATERMARKS_GUMBEL_KEY` | Model-free same-key replay of the keyed-Gumbel (Aaronson EXP) scheme (see `detect_gumbel.py`), stdlib-only — self-hosted engines such as arbi-serve; same-key-only, not a vendor oracle. |
|
||||
| `claude-text` | — (placeholder) | Anthropic has announced a watermark detection API; this seam activates when it ships. |
|
||||
|
||||
Image scoring: when `WATERMARKS_SYNTHID_SCORER_URL` is set, the service
|
||||
@@ -290,6 +291,7 @@ set -a; . ./.env; set +a; python3 service/scripts/rewrite_text.py /tmp/x.txt -o
|
||||
| `WATERMARKS_REWRITE_API_KEY` | `rewrite_text.py` hook | API key — env only, never on argv |
|
||||
| `WATERMARKS_REWRITE_ALLOW_REMOTE` | `rewrite_text.py` hook | `1` to allow non-loopback endpoints |
|
||||
| `WATERMARKS_REWRITE_REASONING_EFFORT` | `rewrite_text.py` hook | `none` (default) / `low` / `medium` / `high` / `off` |
|
||||
| `WATERMARKS_GUMBEL_KEY` | `detect_gumbel.py` / `text_detectors.py` | Secret key for keyed-Gumbel (EXP) same-key replay (e.g. `0x…`); preferred over argv — never logged |
|
||||
|
||||
Layer B is agent-orchestrated in the skill (it rewrites with its own model), so the `WATERMARKS_REWRITE_*` vars are only needed when driving `rewrite_text.py` directly.
|
||||
|
||||
@@ -593,6 +595,47 @@ docker run --rm --user "$(id -u):$(id -g)" -v "$(pwd):/data" \
|
||||
watermarks-remover-markllm detect /data/wm.txt --scheme kgw --json
|
||||
```
|
||||
|
||||
### Keyed-Gumbel (Aaronson EXP) same-key verification
|
||||
|
||||
[ARBI's technical report](https://arbicity.com/news/ai-text-watermarking-for-self-hosted-ai/) describes the
|
||||
keyed-Gumbel ("exponential") text watermark — now shipping in the open-source
|
||||
arbi-serve engine (`ARBI_WATERMARK_KEY`) — where the sampler's noise is derived
|
||||
from a keyed hash of the last 4-token context window. Detection is a
|
||||
**model-free replay**: recompute `u = PRF(Hash(key, window), token)` from the
|
||||
text alone and test the Gamma tail, so it needs no GPU, model, or logits.
|
||||
This repo ships that detector as `detect_gumbel.py` (stdlib-only; the p-value
|
||||
is the exact Poisson-sum identity for an integer Gamma shape):
|
||||
|
||||
```bash
|
||||
# Text mode (deterministic word/run tokenizer) — quick checks and rewrite-loop
|
||||
# evaluation; exact replay against a real engine needs its tokenizer:
|
||||
python3 service/scripts/detect_gumbel.py draft.txt --key 0x... --json
|
||||
|
||||
# Exact replay: pass the engine's token ids (JSON array or one per line).
|
||||
python3 service/scripts/detect_gumbel.py ids.json --tokens --key 0x... --json
|
||||
```
|
||||
|
||||
Same honesty caveat as MarkLLM: this is a **same-key replay** — valid only
|
||||
against the same key, tokenizer, and PRF layout used at generation, and a
|
||||
negative result establishes nothing. The HMAC-SHA256 layout here is an
|
||||
auditable instantiation, not bit-compatible with any specific engine kernel
|
||||
(see the module docstring for what to adapt for exact replay).
|
||||
|
||||
**Detection-guided rewriting:** pass `--gumbel-key` to `rewrite_text.py`
|
||||
(env: `WATERMARKS_GUMBEL_KEY`, preferred) and the iterative rewrite loop is
|
||||
driven by the same-key Gumbel replay — evaluator priority becomes gumbel >
|
||||
MarkLLM > lexical divergence — with a `gumbel.before/after/cleared` report:
|
||||
|
||||
```bash
|
||||
export WATERMARKS_REWRITE_BACKEND=ollama WATERMARKS_REWRITE_MODEL=llama3.2
|
||||
export WATERMARKS_GUMBEL_KEY=0x...
|
||||
python3 "$SCRIPTS/rewrite_text.py" wm.txt -o wm.rewritten.txt --json-stats
|
||||
```
|
||||
|
||||
The key never appears in stats or logs. Self-hosted operators who hold their
|
||||
engine's key can verify a rewrite cleared a Gumbel mark; everyone else treats
|
||||
Layer B as best-effort only.
|
||||
|
||||
## Optional SynthID-text removal benchmark
|
||||
|
||||
[`bench_synthid_text.py`](service/scripts/bench_synthid_text.py) measures how
|
||||
@@ -901,6 +944,15 @@ make smoke # quick CLI smoke on fixtures
|
||||
now carry attempts per document (`mean_attempts`, `att` column;
|
||||
`attempts` / `evaluator` / `passed` columns); `--rewrite-loops`
|
||||
mirrors `--max-loops`.
|
||||
- **Keyed-Gumbel (Aaronson EXP) same-key verification**: new stdlib-only
|
||||
`detect_gumbel.py` implements the model-free replay test of ARBI's keyed-Gumbel
|
||||
report (u = PRF(Hash(key, window), token); exact Gamma-tail p-value; repeated-
|
||||
window masking) — no GPU, model, or logits. `rewrite_text.py --gumbel-key`
|
||||
(env `WATERMARKS_GUMBEL_KEY`, preferred) makes it the iterative-loop evaluator
|
||||
(priority: gumbel > markllm > lexical divergence) with a `gumbel.before/after/
|
||||
cleared` report; the detector is also exposed as `gumbel` in `/capabilities`
|
||||
and `/detect`. Same-key-only: valid against the same key, tokenizer, and PRF
|
||||
layout used at generation — not a vendor oracle. The key is never logged.
|
||||
|
||||
### [v0.5.0](https://github.com/guillaumemeyer/watermarks-remover/releases/tag/v0.5.0) — service & Docker distribution, HTTP API, and verification harnesses
|
||||
|
||||
@@ -1045,13 +1097,14 @@ make smoke # quick CLI smoke on fixtures
|
||||
|
||||
MIT — see [LICENSE](LICENSE).
|
||||
|
||||
## References
|
||||
## Bibliography
|
||||
|
||||
- [How Claude marks AI-generated content](https://support.claude.com/en/articles/16266773-how-claude-marks-ai-generated-content) (Anthropic)
|
||||
- Dathathri et al., [*Scalable watermarking for identifying large language model outputs*](https://www.nature.com/articles/s41586-024-08025-4) (SynthID-Text, Nature 2024)
|
||||
- Google AI for Developers, [*SynthID safeguards*](https://ai.google.dev/responsible/docs/safeguards/synthid) (Gemini API docs)
|
||||
- [C2PA](https://c2pa.org/) / [c2patool](https://github.com/contentauth/c2pa-rs/tree/main/cli)
|
||||
- Kirchenbauer et al., [*A Watermark for Large Language Models*](https://arxiv.org/abs/2301.10226)
|
||||
- Evseev, D. (Arbitration City), [*Accurate, Costless, and Invisible AI Text Watermarking for Self-Hosted AI Inference*](https://arbicity.com/news/ai-text-watermarking-for-self-hosted-ai/) (technical report, August 2026) — keyed-Gumbel watermarking shipped in the open-source arbi-serve engine, with exact-test detection and speculative-decoding support — [PDF](https://arbicity.com/news/ai-text-watermarking-for-self-hosted-ai/ARBI-Watermark-Technical-Paper.pdf)
|
||||
- [THU-BPM/MarkLLM](https://github.com/THU-BPM/MarkLLM) (unified toolkit for evaluating LLM watermarking algorithms)
|
||||
- Pan et al., [*MarkDiffusion: An Open-Source Toolkit for Generative Watermarking of Latent Diffusion Models*](https://arxiv.org/abs/2509.10569) (JMLR) — the embedding toolkit this repo's optional image-watermark harness wraps — [code](https://github.com/THU-BPM/MarkDiffusion), [docs](https://markdiffusion.readthedocs.io)
|
||||
- Zhang et al., [*Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models*](https://arxiv.org/abs/2311.04378v5) (ICML 2024)
|
||||
|
||||
@@ -0,0 +1,308 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Model-free keyed-Gumbel (Aaronson EXP) text-watermark detector.
|
||||
|
||||
Implements the detection arithmetic of the ARBI keyed-Gumbel technical report
|
||||
(Sections 2-3): replay the keyed sampler's noise from the text alone and test
|
||||
whether the observed tokens look like winners of keyed draws.
|
||||
|
||||
seed = Hash(key, last H tokens) H = 4 by default
|
||||
u_t = PRF(seed, token_t) -> replayable from text alone
|
||||
S = sum_t -log(1 - u_t) ~ Gamma(counted, 1) under the null
|
||||
p = P(Gamma(counted, 1) >= S) exact for integer shape
|
||||
|
||||
Repeated context windows are masked (the generator falls back to ordinary
|
||||
randomness on recurrence; the detector applies the same skip rule), so reused
|
||||
windows contribute no evidence. Positions with fewer than H preceding tokens
|
||||
have no full window and are not counted either.
|
||||
|
||||
Stdlib-only: the p-value uses the exact Poisson-sum identity for an integer
|
||||
Gamma shape (upper regularized incomplete gamma), so scipy is never required.
|
||||
|
||||
Honesty caveat (same as the MarkLLM harness): detection is a *same-key replay*
|
||||
— it is valid only against the same key, tokenizer, and PRF layout used at
|
||||
generation (self-hosted engines such as arbi-serve). A negative result
|
||||
establishes nothing: unwatermarked text, another provider's key, and human
|
||||
text all sit at chance. This detector is not a vendor oracle.
|
||||
|
||||
The default PRF layout here is HMAC-SHA256 over packed token ids. It is a
|
||||
clean-room, auditable instantiation of the scheme, not bit-compatible with any
|
||||
specific engine's kernel: for exact replay against a real engine, pass its
|
||||
token ids (--tokens) and, if the engine uses a different PRF, reimplement the
|
||||
two functions in this module accordingly.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import struct
|
||||
import sys
|
||||
from collections.abc import Sequence
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
from common import emit_json, eprint, read_text_input
|
||||
|
||||
DEFAULT_WINDOW = 4
|
||||
DEFAULT_THRESHOLD = 1e-6
|
||||
|
||||
_ID_PACK = struct.Struct(">Q")
|
||||
_SIMPLE_TOKEN_RE = re.compile(r"[A-Za-z0-9]+")
|
||||
|
||||
|
||||
def _token_id(tok: str) -> int:
|
||||
"""Deterministic 64-bit id for a simple-tokenizer token string."""
|
||||
return int.from_bytes(hashlib.sha256(tok.encode("utf-8")).digest()[:8], "big")
|
||||
|
||||
|
||||
def tokenize_simple(text: str) -> list[int]:
|
||||
"""Deterministic word/run tokenizer -> stable token ids.
|
||||
|
||||
Convenience path for testing and quick checks. Exact replay against a real
|
||||
engine requires the engine's own tokenizer: use --tokens with its ids.
|
||||
"""
|
||||
return [_token_id(t) for t in _SIMPLE_TOKEN_RE.findall(text.lower())]
|
||||
|
||||
|
||||
def load_token_ids(raw: str) -> list[int]:
|
||||
"""Parse a token-id input: a JSON array, or one integer per line."""
|
||||
stripped = raw.strip()
|
||||
if stripped.startswith("["):
|
||||
data = json.loads(stripped)
|
||||
if not isinstance(data, list) or not all(isinstance(x, int) for x in data):
|
||||
raise ValueError("token-id JSON must be an array of integers")
|
||||
return data
|
||||
ids: list[int] = []
|
||||
for raw_line in stripped.splitlines():
|
||||
line = raw_line.strip()
|
||||
if not line:
|
||||
continue
|
||||
ids.append(int(line, 0))
|
||||
return ids
|
||||
|
||||
|
||||
def _normalize_key(raw: str) -> bytes:
|
||||
"""Key -> bytes: 0x<hex> decodes to raw bytes, anything else is UTF-8."""
|
||||
s = raw.strip()
|
||||
if s.startswith("0x") or s.startswith("0X"):
|
||||
hexpart = s[2:]
|
||||
if (
|
||||
not hexpart
|
||||
or len(hexpart) % 2
|
||||
or any(c not in "0123456789abcdefABCDEF" for c in hexpart)
|
||||
):
|
||||
raise ValueError("invalid hex key (expected 0x followed by even-length hex)")
|
||||
return bytes.fromhex(hexpart)
|
||||
return s.encode("utf-8")
|
||||
|
||||
|
||||
def _seed(key: bytes, window: tuple[int, ...]) -> bytes:
|
||||
"""Context-window seed: HMAC-SHA256(key, packed window)."""
|
||||
packed = b"".join(_ID_PACK.pack(t) for t in window)
|
||||
return hmac.new(key, packed, hashlib.sha256).digest()
|
||||
|
||||
|
||||
def _uniform(seed: bytes, token_id: int) -> float:
|
||||
"""Per-candidate uniform in (0, 1): HMAC-SHA256(seed, token id)."""
|
||||
digest = hmac.new(seed, _ID_PACK.pack(token_id), hashlib.sha256).digest()
|
||||
return (int.from_bytes(digest[:8], "big") + 0.5) / (1 << 64)
|
||||
|
||||
|
||||
def _poisson_survival(s: float, n: int) -> float:
|
||||
"""P(Gamma(n, 1) >= s) = e^{-s} * sum_{k=0}^{n-1} s^k / k! (exact for integer n).
|
||||
|
||||
Computed via logsumexp so large statistics never overflow or NaN.
|
||||
"""
|
||||
if n <= 0 or s <= 0.0:
|
||||
return 1.0
|
||||
lns = math.log(s)
|
||||
log_term = 0.0 # k = 0 term is s^0 / 0! = 1
|
||||
maxv = 0.0
|
||||
for k in range(1, n):
|
||||
log_term += lns - math.log(k)
|
||||
if log_term > maxv:
|
||||
maxv = log_term
|
||||
log_term = 0.0
|
||||
acc = math.exp(-maxv) # k = 0 term
|
||||
for k in range(1, n):
|
||||
log_term += lns - math.log(k)
|
||||
acc += math.exp(log_term - maxv)
|
||||
p = math.exp(-s + maxv + math.log(acc))
|
||||
if p < 0.0:
|
||||
return 0.0
|
||||
if p > 1.0:
|
||||
return 1.0
|
||||
return p
|
||||
|
||||
|
||||
def detect_token_ids(
|
||||
token_ids: Sequence[int],
|
||||
key: str | bytes,
|
||||
*,
|
||||
window: int = DEFAULT_WINDOW,
|
||||
threshold: float = DEFAULT_THRESHOLD,
|
||||
mask_repeated: bool = True,
|
||||
) -> dict[str, Any]:
|
||||
"""Run the keyed-Gumbel replay test over a sequence of token ids.
|
||||
|
||||
mask_repeated mirrors the generator's repeated-window masking: a context
|
||||
window is counted only on its first occurrence (the generator fell back to
|
||||
ordinary randomness on recurrence, so later occurrences carry no signal).
|
||||
Positions with fewer than window preceding tokens are never counted.
|
||||
"""
|
||||
if window < 1:
|
||||
raise ValueError("window must be >= 1")
|
||||
if not 0.0 < threshold < 1.0:
|
||||
raise ValueError("threshold must be in (0, 1)")
|
||||
key_bytes = _normalize_key(key) if isinstance(key, str) else bytes(key)
|
||||
ids = list(token_ids)
|
||||
for t in ids:
|
||||
if not isinstance(t, int) or isinstance(t, bool) or t < 0 or t > (1 << 64) - 1:
|
||||
raise ValueError(f"token id out of range: {t!r}")
|
||||
|
||||
total = len(ids)
|
||||
statistic = 0.0
|
||||
counted = 0
|
||||
seen_windows: set[tuple[int, ...]] = set()
|
||||
skipped_repeated = 0
|
||||
for t in range(window, total):
|
||||
win = tuple(ids[t - window : t])
|
||||
if mask_repeated:
|
||||
if win in seen_windows:
|
||||
skipped_repeated += 1
|
||||
continue
|
||||
seen_windows.add(win)
|
||||
u = _uniform(_seed(key_bytes, win), ids[t])
|
||||
statistic += -math.log1p(-u)
|
||||
counted += 1
|
||||
|
||||
skipped_no_context = min(window, total)
|
||||
report: dict[str, Any] = {
|
||||
"detector": "gumbel",
|
||||
"scheme": "exp",
|
||||
"vendor": "self-hosted",
|
||||
"available": True,
|
||||
"window": window,
|
||||
"threshold": threshold,
|
||||
"tokens_total": total,
|
||||
"skipped_no_context": skipped_no_context,
|
||||
"skipped_repeated": skipped_repeated,
|
||||
"counted": counted,
|
||||
}
|
||||
if counted == 0:
|
||||
report["is_watermarked"] = False
|
||||
report["p_value"] = 1.0
|
||||
report["score"] = 0.0
|
||||
report["note"] = "no verifiable token positions (text too short for a full context window)"
|
||||
return report
|
||||
p = _poisson_survival(statistic, counted)
|
||||
report["statistic"] = round(statistic, 6)
|
||||
report["p_value"] = p
|
||||
report["score"] = round(-math.log10(p) if p > 0.0 else 300.0, 6)
|
||||
report["is_watermarked"] = p < threshold
|
||||
report["note"] = (
|
||||
"same-key replay of the keyed-Gumbel (Aaronson EXP) watermark; valid only "
|
||||
"against the same key, tokenizer, and PRF layout used at generation"
|
||||
)
|
||||
return report
|
||||
|
||||
|
||||
def detect_text(
|
||||
text: str,
|
||||
key: str | bytes,
|
||||
*,
|
||||
window: int = DEFAULT_WINDOW,
|
||||
threshold: float = DEFAULT_THRESHOLD,
|
||||
) -> dict[str, Any]:
|
||||
"""Run the replay test over plain text via the deterministic tokenizer."""
|
||||
return detect_token_ids(tokenize_simple(text), key, window=window, threshold=threshold)
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument("path", nargs="?", default="-", help="Text file, or - for stdin")
|
||||
(
|
||||
p.add_argument(
|
||||
"--tokens",
|
||||
action="store_true",
|
||||
help="Treat the input as token ids (JSON array or one integer per line) "
|
||||
"instead of text — required for exact replay with an engine's tokenizer",
|
||||
),
|
||||
)
|
||||
(
|
||||
p.add_argument(
|
||||
"--key",
|
||||
default=os.environ.get("WATERMARKS_GUMBEL_KEY"),
|
||||
help="Watermark key (0x<hex> or a string); default: $WATERMARKS_GUMBEL_KEY. "
|
||||
"Preferred via env — keys on argv are visible in ps/history.",
|
||||
),
|
||||
)
|
||||
(
|
||||
p.add_argument(
|
||||
"--window",
|
||||
type=int,
|
||||
default=DEFAULT_WINDOW,
|
||||
help=f"Context window size H in tokens (default: {DEFAULT_WINDOW})",
|
||||
),
|
||||
)
|
||||
(
|
||||
p.add_argument(
|
||||
"--threshold",
|
||||
type=float,
|
||||
default=DEFAULT_THRESHOLD,
|
||||
help=f"p-value threshold for is_watermarked (default: {DEFAULT_THRESHOLD:g})",
|
||||
),
|
||||
)
|
||||
p.add_argument("--json", action="store_true", help="Emit the report as JSON on stdout")
|
||||
(
|
||||
p.add_argument(
|
||||
"--force-text",
|
||||
action="store_true",
|
||||
help="Process input even when it looks like a binary container",
|
||||
),
|
||||
)
|
||||
return p
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = build_parser().parse_args()
|
||||
if not args.key:
|
||||
eprint("error: no watermark key (pass --key or set WATERMARKS_GUMBEL_KEY)")
|
||||
return 2
|
||||
raw = read_text_input(args.path, allow_binary=args.force_text)
|
||||
try:
|
||||
if args.tokens:
|
||||
report = detect_token_ids(
|
||||
load_token_ids(raw),
|
||||
args.key,
|
||||
window=args.window,
|
||||
threshold=args.threshold,
|
||||
)
|
||||
else:
|
||||
report = detect_text(raw, args.key, window=args.window, threshold=args.threshold)
|
||||
except (ValueError, json.JSONDecodeError) as e:
|
||||
eprint(f"error: {e}")
|
||||
return 2
|
||||
if args.json:
|
||||
emit_json(report)
|
||||
else:
|
||||
print(
|
||||
f"keyed-Gumbel (EXP) detection: watermarked={report['is_watermarked']} "
|
||||
f"p={report['p_value']:.3g} (threshold {report['threshold']:g}) "
|
||||
f"counted={report['counted']}/{report['tokens_total']} tokens, skipped "
|
||||
f"{report['skipped_no_context'] + report['skipped_repeated']} "
|
||||
f"(no-context {report['skipped_no_context']}, repeated-window "
|
||||
f"{report['skipped_repeated']})"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -19,12 +19,13 @@ Rewriting is iterative and evaluation-driven: each loop generates
|
||||
--candidates (default 1) variants, evaluates each, and stops as soon as an
|
||||
attempt passes watermark detection; --max-loops (default 1) caps how many
|
||||
evaluation rounds run before the best-effort variant is returned
|
||||
(WATERMARKS_REWRITE_LOOPS). The evaluator is chosen by priority: MarkLLM
|
||||
same-config detection (when --markllm-scheme is passed) or, when no detector
|
||||
is configured, bigram-Jaccard lexical divergence (no pass/fail verdict — all
|
||||
(WATERMARKS_REWRITE_LOOPS). The evaluator is chosen by priority: keyed-Gumbel
|
||||
same-key replay (when --gumbel-key / WATERMARKS_GUMBEL_KEY is set), else
|
||||
MarkLLM same-config detection (--markllm-scheme), else, when no detector is
|
||||
configured, bigram-Jaccard lexical divergence (no pass/fail verdict — all
|
||||
attempts are generated and the most diverged one is selected). A vendor-detector
|
||||
seam (Google's retired SynthID-text detector) is reserved ahead of MarkLLM
|
||||
should a vendor endpoint return.
|
||||
seam (Google's retired SynthID-text detector) is reserved ahead of the
|
||||
same-config detectors should a vendor endpoint return.
|
||||
|
||||
Security notes:
|
||||
- Only http(s) endpoints are accepted; redirects are refused outright so an
|
||||
@@ -49,7 +50,7 @@ from urllib.parse import urlparse
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
from common import cleaned_path, eprint, read_text_input, write_text_output
|
||||
from text_detectors import MarkLLMTextDetector
|
||||
from text_detectors import GumbelTextDetector, MarkLLMTextDetector
|
||||
from text_unicode import clean_text
|
||||
|
||||
DEFAULT_MARKLLM_MODEL = "facebook/opt-1.3b"
|
||||
@@ -205,16 +206,20 @@ def _safe_detect(detector: object, text: str) -> dict:
|
||||
|
||||
def _pick_evaluator(
|
||||
markllm_detector: MarkLLMTextDetector | None,
|
||||
gumbel_detector: GumbelTextDetector | None,
|
||||
) -> tuple[str, object | None]:
|
||||
"""Pick the evaluator that drives the iterative rewrite loop.
|
||||
|
||||
Priority: MarkLLM same-config detection (when the caller passed
|
||||
--markllm-scheme) > bigram-Jaccard lexical divergence (fallback with no
|
||||
pass/fail verdict). A vendor-detector seam (Google's SynthID-text detector,
|
||||
retired Aug 2026) is reserved ahead of MarkLLM should a vendor endpoint
|
||||
return; it only needs available()/detect()/name per the TextDetector
|
||||
protocol in text_detectors.py.
|
||||
Priority: keyed-Gumbel same-key replay (when the caller passed
|
||||
--gumbel-key) > MarkLLM same-config detection (--markllm-scheme) >
|
||||
bigram-Jaccard lexical divergence (fallback with no pass/fail verdict).
|
||||
A vendor-detector seam (Google's SynthID-text detector, retired Aug 2026)
|
||||
is reserved ahead of both should a vendor endpoint return; it only needs
|
||||
available()/detect()/name per the TextDetector protocol in
|
||||
text_detectors.py.
|
||||
"""
|
||||
if gumbel_detector is not None:
|
||||
return "gumbel", gumbel_detector
|
||||
if markllm_detector is not None:
|
||||
return "markllm", markllm_detector
|
||||
return "lexical-divergence", None
|
||||
@@ -356,6 +361,7 @@ def rewrite(
|
||||
markllm_dir: str | None = None,
|
||||
markllm_model: str | None = None,
|
||||
markllm_timeout: float = 180.0,
|
||||
gumbel_key: str | None = None,
|
||||
) -> tuple[str, dict]:
|
||||
prompt = build_prompt(strength, text, lang=lang, original_lang=original_lang)
|
||||
info: dict = {
|
||||
@@ -387,6 +393,15 @@ def rewrite(
|
||||
eprint(f"markllm verification unavailable: {markllm['before']['error']}")
|
||||
info["markllm"] = markllm
|
||||
|
||||
gumbel: dict | None = None
|
||||
gumbel_detector: GumbelTextDetector | None = None
|
||||
if gumbel_key:
|
||||
gumbel_detector = GumbelTextDetector(key=gumbel_key)
|
||||
gumbel = {"before": _safe_detect(gumbel_detector, text)}
|
||||
if not gumbel["before"]["available"]:
|
||||
eprint(f"gumbel verification unavailable: {gumbel['before']['error']}")
|
||||
info["gumbel"] = gumbel
|
||||
|
||||
if backend == "print-prompt":
|
||||
info["mode"] = "print-prompt"
|
||||
if candidates > 1:
|
||||
@@ -404,7 +419,7 @@ def rewrite(
|
||||
n_loops = max(1, max_loops)
|
||||
info["candidates"] = n_cands
|
||||
info["max_loops"] = n_loops
|
||||
evaluator_name, evaluator = _pick_evaluator(markllm_detector)
|
||||
evaluator_name, evaluator = _pick_evaluator(markllm_detector, gumbel_detector)
|
||||
info["evaluator"] = evaluator_name
|
||||
|
||||
# Iterative rewrite: each loop generates --candidates variants and
|
||||
@@ -522,6 +537,28 @@ def rewrite(
|
||||
"keys used at generation; it does not certify a vendor detector."
|
||||
)
|
||||
|
||||
if gumbel:
|
||||
assert gumbel_detector is not None # set together with gumbel above
|
||||
if evaluator_name == "gumbel":
|
||||
# The loop already scored the selected attempt; reuse the verdict
|
||||
# instead of paying another replay.
|
||||
after = rec["evaluation"]
|
||||
else:
|
||||
after = _safe_detect(gumbel_detector, out)
|
||||
gumbel["after"] = after
|
||||
before = gumbel["before"]
|
||||
if before.get("available") and after.get("available"):
|
||||
gumbel["cleared"] = bool(
|
||||
before.get("is_watermarked") and not after.get("is_watermarked")
|
||||
)
|
||||
else:
|
||||
gumbel["cleared"] = None
|
||||
gumbel["note"] = (
|
||||
"Keyed-Gumbel detection is a same-key replay: valid only with the "
|
||||
"same key, tokenizer, and PRF layout used at generation; it does "
|
||||
"not certify a vendor detector."
|
||||
)
|
||||
|
||||
eprint(
|
||||
f"note: evaluator={evaluator_name} attempts={len(attempts)}/"
|
||||
f"{n_cands * n_loops} loops={n_loops} passed={passed}"
|
||||
@@ -623,6 +660,14 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
default=float(_env("WATERMARKS_MARKLLM_TIMEOUT", "180.0")),
|
||||
help="Timeout per MarkLLM detection call (default: 180.0)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--gumbel-key",
|
||||
default=_env("WATERMARKS_GUMBEL_KEY"),
|
||||
help="Secret key for keyed-Gumbel (Aaronson EXP) same-key replay "
|
||||
"detection; drives the iterative rewrite loop as the evaluator when "
|
||||
"set (default: $WATERMARKS_GUMBEL_KEY). Preferred via env — keys on "
|
||||
"argv are visible in ps/history; never logged.",
|
||||
)
|
||||
p.add_argument(
|
||||
"--force-text",
|
||||
action="store_true",
|
||||
@@ -661,6 +706,7 @@ def main() -> int:
|
||||
markllm_dir=args.markllm_dir,
|
||||
markllm_model=args.markllm_model,
|
||||
markllm_timeout=args.markllm_timeout,
|
||||
gumbel_key=args.gumbel_key,
|
||||
)
|
||||
except (urllib.error.URLError, TimeoutError, RuntimeError) as e:
|
||||
eprint(f"rewrite failed: {e}")
|
||||
|
||||
@@ -17,6 +17,10 @@ Detectors:
|
||||
- markllm — research harness (KGW / SynthID schemes) via
|
||||
detect_text_watermark.py, activated by MARKLLM_DIR. Same-config-only
|
||||
detection; not a vendor oracle.
|
||||
- gumbel — model-free same-key replay of the keyed-Gumbel (Aaronson EXP)
|
||||
scheme (detect_gumbel.py), activated by WATERMARKS_GUMBEL_KEY. Stdlib-only;
|
||||
valid only against the same key, tokenizer, and PRF layout used at
|
||||
generation (self-hosted engines such as arbi-serve); not a vendor oracle.
|
||||
- claude-text — placeholder for Anthropic's announced text-watermark
|
||||
detection API. Reports unavailable until a public endpoint exists; the
|
||||
interface it must implement is already defined here.
|
||||
@@ -40,6 +44,8 @@ from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from typing import Any, Protocol
|
||||
|
||||
from detect_gumbel import DEFAULT_THRESHOLD, DEFAULT_WINDOW, detect_text
|
||||
|
||||
DEFAULT_MARKLLM_SCHEME = "kgw"
|
||||
DEFAULT_MARKLLM_TIMEOUT = 600.0
|
||||
|
||||
@@ -251,6 +257,75 @@ class MarkLLMTextDetector:
|
||||
return payload
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Keyed-Gumbel (Aaronson EXP) — model-free same-key replay
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class GumbelTextDetector:
|
||||
"""Same-key replay of the keyed-Gumbel (Aaronson EXP) text watermark.
|
||||
|
||||
Model-free: implements the detection arithmetic of the ARBI keyed-Gumbel
|
||||
technical report (Sections 2-3) — replay u = PRF(Hash(key, window), token)
|
||||
from the text alone and test the Gamma tail — so it needs no GPU, model,
|
||||
or logits. Detection is valid only against the SAME key, tokenizer, and
|
||||
PRF layout used at generation (self-hosted engines such as arbi-serve);
|
||||
it is not a vendor oracle. Key from WATERMARKS_GUMBEL_KEY (env) or the
|
||||
constructor override.
|
||||
"""
|
||||
|
||||
name = "gumbel"
|
||||
vendor = "self-hosted"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
key: str | None = None,
|
||||
window: int | None = None,
|
||||
threshold: float | None = None,
|
||||
) -> None:
|
||||
self._key = key
|
||||
self._window = window
|
||||
self._threshold = threshold
|
||||
|
||||
def _key_env(self) -> str | None:
|
||||
if self._key:
|
||||
return self._key
|
||||
return os.environ.get("WATERMARKS_GUMBEL_KEY", "").strip() or None
|
||||
|
||||
def available(self) -> bool:
|
||||
return self._key_env() is not None
|
||||
|
||||
def detect(self, text: str) -> dict[str, Any]:
|
||||
key = self._key_env()
|
||||
report: dict[str, Any] = {
|
||||
"detector": self.name,
|
||||
"scheme": "exp",
|
||||
"vendor": self.vendor,
|
||||
"available": False,
|
||||
}
|
||||
if key is None:
|
||||
report["error"] = "WATERMARKS_GUMBEL_KEY not set"
|
||||
return report
|
||||
try:
|
||||
payload = detect_text(
|
||||
text,
|
||||
key,
|
||||
window=self._window or DEFAULT_WINDOW,
|
||||
threshold=self._threshold or DEFAULT_THRESHOLD,
|
||||
)
|
||||
except Exception as e: # fail-soft contract: never raise
|
||||
report["error"] = f"keyed-Gumbel detection failed: {e}"
|
||||
return report
|
||||
payload["detector"] = self.name
|
||||
payload["note"] = (
|
||||
"same-key replay of the keyed-Gumbel (Aaronson EXP) watermark: valid "
|
||||
"only with the same key, tokenizer, and PRF layout used at generation; "
|
||||
"not a vendor detector."
|
||||
)
|
||||
return payload
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Claude (Anthropic) — announced detector API, not yet public
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -290,11 +365,17 @@ class ClaudeTextDetector:
|
||||
|
||||
|
||||
def all_detectors(
|
||||
markllm: MarkLLMTextDetector | None = None, *, include_markllm: bool = True
|
||||
markllm: MarkLLMTextDetector | None = None,
|
||||
*,
|
||||
include_markllm: bool = True,
|
||||
gumbel: GumbelTextDetector | None = None,
|
||||
include_gumbel: bool = True,
|
||||
) -> list[TextDetector]:
|
||||
detectors: list[TextDetector] = []
|
||||
if include_markllm:
|
||||
detectors.append(markllm or MarkLLMTextDetector())
|
||||
if include_gumbel:
|
||||
detectors.append(gumbel or GumbelTextDetector())
|
||||
detectors.append(ClaudeTextDetector())
|
||||
return detectors
|
||||
|
||||
@@ -309,14 +390,24 @@ def run_all_text_detectors(
|
||||
*,
|
||||
markllm: MarkLLMTextDetector | None = None,
|
||||
include_markllm: bool = True,
|
||||
gumbel: GumbelTextDetector | None = None,
|
||||
include_gumbel: bool = True,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Run every detector (including unavailable ones, with reasons).
|
||||
|
||||
markllm injects a caller-parameterized MarkLLM detector (e.g. one
|
||||
driven by rewrite_text.py CLI flags); pass include_markllm=False to
|
||||
exclude the MarkLLM harness entirely.
|
||||
exclude the MarkLLM harness entirely. Same for gumbel.
|
||||
"""
|
||||
return [d.detect(text) for d in all_detectors(markllm, include_markllm=include_markllm)]
|
||||
return [
|
||||
d.detect(text)
|
||||
for d in all_detectors(
|
||||
markllm,
|
||||
include_markllm=include_markllm,
|
||||
gumbel=gumbel,
|
||||
include_gumbel=include_gumbel,
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def run_text_detectors(
|
||||
@@ -324,10 +415,17 @@ def run_text_detectors(
|
||||
*,
|
||||
markllm: MarkLLMTextDetector | None = None,
|
||||
include_markllm: bool = True,
|
||||
gumbel: GumbelTextDetector | None = None,
|
||||
include_gumbel: bool = True,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Run only the detectors that are configured and usable."""
|
||||
return [
|
||||
d.detect(text)
|
||||
for d in all_detectors(markllm, include_markllm=include_markllm)
|
||||
for d in all_detectors(
|
||||
markllm,
|
||||
include_markllm=include_markllm,
|
||||
gumbel=gumbel,
|
||||
include_gumbel=include_gumbel,
|
||||
)
|
||||
if d.available()
|
||||
]
|
||||
|
||||
@@ -107,7 +107,7 @@ def _clean_env(monkeypatch):
|
||||
def test_capabilities_exposes_detectors(conn):
|
||||
status, body = _get(conn, "/capabilities")
|
||||
assert status == 200
|
||||
assert set(body["text_detectors"]) == {"markllm", "claude-text"}
|
||||
assert set(body["text_detectors"]) == {"markllm", "gumbel", "claude-text"}
|
||||
assert "synthid_http" in body["scorers"]
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,411 @@
|
||||
"""Tests for the model-free keyed-Gumbel (Aaronson EXP) detector.
|
||||
|
||||
Exercises the replay arithmetic of detect_gumbel.py with a toy keyed
|
||||
Gumbel-max generator, the TextDetector protocol in text_detectors.py, and
|
||||
the gumbel evaluator wiring in rewrite_text.py.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import random
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SCRIPTS = ROOT / "service" / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
import detect_gumbel
|
||||
import rewrite_text
|
||||
import text_detectors
|
||||
|
||||
KEY_HEX = "0x" + "ab" * 16 # 16 raw bytes
|
||||
KEY_HEX_OTHER = "0x" + "cd" * 16
|
||||
KEY_BYTES = detect_gumbel._normalize_key(KEY_HEX)
|
||||
|
||||
|
||||
# S311: deterministic toy RNG for the sampler tests — never used for secrets.
|
||||
def _rng(seed: int) -> random.Random:
|
||||
return random.Random(seed) # noqa: S311
|
||||
|
||||
|
||||
_WORDS = [f"w{i}" for i in range(64)]
|
||||
_WORD_IDS = [detect_gumbel._token_id(w) for w in _WORDS]
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_env(monkeypatch):
|
||||
monkeypatch.delenv("WATERMARKS_GUMBEL_KEY", raising=False)
|
||||
|
||||
|
||||
# --- Toy keyed Gumbel-max generator (mirrors paper Section 2) ----------------
|
||||
|
||||
|
||||
def _gumbel(u: list[float]) -> list[float]:
|
||||
return [-math.log(-math.log(x)) for x in u]
|
||||
|
||||
|
||||
def _keyed_uniforms(key_bytes: bytes, window_ids: tuple[int, ...], vocab: list[int]) -> list[float]:
|
||||
seed = detect_gumbel._seed(key_bytes, window_ids)
|
||||
return [detect_gumbel._uniform(seed, v) for v in vocab]
|
||||
|
||||
|
||||
def _generate(
|
||||
key_bytes: bytes | None,
|
||||
*,
|
||||
n: int,
|
||||
vocab: int,
|
||||
window: int,
|
||||
rng: random.Random,
|
||||
) -> list[int]:
|
||||
"""Keyed (or RNG) Gumbel-max sampler with generator-side window masking."""
|
||||
logits = [rng.uniform(-1.0, 1.0) for _ in range(vocab)]
|
||||
ids: list[int] = []
|
||||
seen: set[tuple[int, ...]] = set()
|
||||
for _ in range(n):
|
||||
win = tuple(ids[-window:])
|
||||
use_keyed = key_bytes is not None and len(win) == window and win not in seen
|
||||
if use_keyed:
|
||||
seen.add(win)
|
||||
if use_keyed:
|
||||
u = _keyed_uniforms(key_bytes, win, list(range(vocab)))
|
||||
else:
|
||||
u = [rng.random() for _ in range(vocab)]
|
||||
g = _gumbel(u)
|
||||
ids.append(max(range(vocab), key=lambda v: logits[v] + g[v]))
|
||||
return ids
|
||||
|
||||
|
||||
def _marked_text(n: int, rng: random.Random, key_hex: str = KEY_HEX) -> str:
|
||||
"""Marked *text* whose simple-tokenizer ids replay under the same key.
|
||||
|
||||
Generates over the word->id space detect_gumbel uses for plain text, so
|
||||
the real detector re-derives identical seeds and uniforms.
|
||||
"""
|
||||
key_bytes = detect_gumbel._normalize_key(key_hex)
|
||||
logits = [rng.uniform(-1.0, 1.0) for _ in _WORDS]
|
||||
seq_ids: list[int] = []
|
||||
seq_words: list[str] = []
|
||||
seen: set[tuple[int, ...]] = set()
|
||||
for _ in range(n):
|
||||
win = tuple(seq_ids[-4:])
|
||||
use_keyed = len(win) == 4 and win not in seen
|
||||
if use_keyed:
|
||||
seen.add(win)
|
||||
if use_keyed:
|
||||
u = _keyed_uniforms(key_bytes, win, _WORD_IDS)
|
||||
else:
|
||||
u = [rng.random() for _ in _WORD_IDS]
|
||||
g = _gumbel(u)
|
||||
chosen = max(range(len(_WORD_IDS)), key=lambda v: logits[v] + g[v])
|
||||
seq_ids.append(_WORD_IDS[chosen])
|
||||
seq_words.append(_WORDS[chosen])
|
||||
return " ".join(seq_words)
|
||||
|
||||
|
||||
# --- Replay arithmetic -------------------------------------------------------
|
||||
|
||||
|
||||
def test_marked_sequence_is_detected():
|
||||
rng = _rng(1)
|
||||
ids = _generate(KEY_BYTES, n=700, vocab=32, window=4, rng=rng)
|
||||
report = detect_gumbel.detect_token_ids(ids, KEY_HEX)
|
||||
assert report["available"] is True
|
||||
assert report["is_watermarked"] is True
|
||||
assert report["p_value"] < 1e-12
|
||||
|
||||
|
||||
def test_unmarked_sequence_at_chance():
|
||||
rng = _rng(2)
|
||||
ids = _generate(None, n=700, vocab=32, window=4, rng=rng)
|
||||
report = detect_gumbel.detect_token_ids(ids, KEY_HEX)
|
||||
assert report["is_watermarked"] is False
|
||||
assert report["p_value"] > 0.01
|
||||
|
||||
|
||||
def test_wrong_key_at_chance():
|
||||
rng = _rng(3)
|
||||
ids = _generate(KEY_BYTES, n=700, vocab=32, window=4, rng=rng)
|
||||
report = detect_gumbel.detect_token_ids(ids, KEY_HEX_OTHER)
|
||||
assert report["is_watermarked"] is False
|
||||
assert report["p_value"] > 0.01
|
||||
|
||||
|
||||
def test_marked_text_roundtrip_through_simple_tokenizer():
|
||||
rng = _rng(4)
|
||||
text = _marked_text(700, rng)
|
||||
report = detect_gumbel.detect_text(text, KEY_HEX)
|
||||
assert report["is_watermarked"] is True
|
||||
assert report["p_value"] < 1e-9
|
||||
# the same text under a different key sits at chance
|
||||
other = detect_gumbel.detect_text(text, KEY_HEX_OTHER)
|
||||
assert other["is_watermarked"] is False
|
||||
assert other["p_value"] > 0.01
|
||||
|
||||
|
||||
def test_detection_is_deterministic():
|
||||
rng = _rng(5)
|
||||
text = _marked_text(300, rng)
|
||||
assert detect_gumbel.detect_text(text, KEY_HEX) == detect_gumbel.detect_text(text, KEY_HEX)
|
||||
|
||||
|
||||
def test_repeated_window_masking_skip_rule():
|
||||
ids = [0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1]
|
||||
report = detect_gumbel.detect_token_ids(ids, KEY_HEX, window=2)
|
||||
# positions 0-1 lack context; windows (0,1)/(1,0) are keyed once each and
|
||||
# every later occurrence is a repeated window the detector must skip
|
||||
assert report["skipped_no_context"] == 2
|
||||
assert report["skipped_repeated"] == 16
|
||||
assert report["counted"] == 2
|
||||
assert report["is_watermarked"] is False # two tokens carry no evidence
|
||||
# without masking every full-window position is counted
|
||||
report2 = detect_gumbel.detect_token_ids(ids, KEY_HEX, window=2, mask_repeated=False)
|
||||
assert report2["counted"] == 18
|
||||
assert report2["skipped_repeated"] == 0
|
||||
|
||||
|
||||
def test_short_text_counts_nothing():
|
||||
report = detect_gumbel.detect_text("a b c", KEY_HEX)
|
||||
assert report["counted"] == 0
|
||||
assert report["is_watermarked"] is False
|
||||
assert report["p_value"] == 1.0
|
||||
|
||||
|
||||
def test_key_normalization():
|
||||
assert detect_gumbel._normalize_key("0x" + "ab" * 8) == b"\xab" * 8
|
||||
assert detect_gumbel._normalize_key("0xAB12") == b"\xab\x12"
|
||||
assert detect_gumbel._normalize_key("s3cret key!") == b"s3cret key!"
|
||||
with pytest.raises(ValueError):
|
||||
detect_gumbel._normalize_key("0xzz")
|
||||
with pytest.raises(ValueError):
|
||||
detect_gumbel._normalize_key("0xabc")
|
||||
|
||||
|
||||
def test_poisson_survival_sanity():
|
||||
# Gamma(1,1) survival at s=1 is e^-1; at s=n the p-value is ~0.5
|
||||
assert detect_gumbel._poisson_survival(1.0, 1) == pytest.approx(math.exp(-1.0))
|
||||
assert detect_gumbel._poisson_survival(10.0, 10) == pytest.approx(0.5, abs=0.05)
|
||||
assert detect_gumbel._poisson_survival(50.0, 10) < 1e-6
|
||||
assert detect_gumbel._poisson_survival(0.0, 10) == 1.0
|
||||
|
||||
|
||||
def test_poisson_survival_matches_scipy_gammaincc():
|
||||
pytest.importorskip("scipy.special")
|
||||
from scipy.special import gammaincc
|
||||
|
||||
for n in (1, 5, 50, 200):
|
||||
for s in (n * 0.5, n, n * 1.5, n * 2.0):
|
||||
ours = detect_gumbel._poisson_survival(s, n)
|
||||
ref = float(gammaincc(n, s))
|
||||
assert ours == pytest.approx(ref, rel=1e-6, abs=1e-300)
|
||||
|
||||
|
||||
def test_load_token_ids_json_and_lines():
|
||||
assert detect_gumbel.load_token_ids("[1, 2, 3]") == [1, 2, 3]
|
||||
assert detect_gumbel.load_token_ids("1\n0x2\n3") == [1, 2, 3]
|
||||
with pytest.raises(ValueError):
|
||||
detect_gumbel.load_token_ids('[1, "x"]')
|
||||
|
||||
|
||||
def test_token_id_out_of_range_rejected():
|
||||
with pytest.raises(ValueError):
|
||||
detect_gumbel.detect_token_ids([-1, 2, 3], KEY_HEX)
|
||||
with pytest.raises(ValueError):
|
||||
detect_gumbel.detect_token_ids([1 << 64, 2], KEY_HEX)
|
||||
|
||||
|
||||
# --- CLI ---------------------------------------------------------------------
|
||||
|
||||
|
||||
def _run_cli(
|
||||
*args: str, env_extra: dict[str, str] | None = None
|
||||
) -> subprocess.CompletedProcess[str]:
|
||||
env = {k: v for k, v in os.environ.items() if k != "WATERMARKS_GUMBEL_KEY"}
|
||||
env.update(env_extra or {})
|
||||
return subprocess.run(
|
||||
[sys.executable, str(SCRIPTS / "detect_gumbel.py"), *args],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=120,
|
||||
env=env,
|
||||
check=False,
|
||||
)
|
||||
|
||||
|
||||
def test_cli_detects_marked_text(tmp_path):
|
||||
rng = _rng(6)
|
||||
src = tmp_path / "marked.txt"
|
||||
src.write_text(_marked_text(500, rng), encoding="utf-8")
|
||||
r = _run_cli(str(src), "--key", KEY_HEX, "--json")
|
||||
assert r.returncode == 0, r.stderr
|
||||
report = json.loads(r.stdout)
|
||||
assert report["is_watermarked"] is True
|
||||
|
||||
|
||||
def test_cli_token_ids_mode(tmp_path):
|
||||
rng = _rng(7)
|
||||
ids = _generate(KEY_BYTES, n=500, vocab=32, window=4, rng=rng)
|
||||
src = tmp_path / "ids.json"
|
||||
src.write_text(json.dumps(ids), encoding="utf-8")
|
||||
r = _run_cli(str(src), "--tokens", "--key", KEY_HEX, "--json")
|
||||
assert r.returncode == 0, r.stderr
|
||||
report = json.loads(r.stdout)
|
||||
assert report["is_watermarked"] is True
|
||||
|
||||
|
||||
def test_cli_requires_key(tmp_path):
|
||||
src = tmp_path / "plain.txt"
|
||||
src.write_text("hello world", encoding="utf-8")
|
||||
r = _run_cli(str(src))
|
||||
assert r.returncode == 2
|
||||
assert "key" in (r.stderr + r.stdout).lower()
|
||||
|
||||
|
||||
def test_cli_reads_key_from_env(tmp_path):
|
||||
rng = _rng(8)
|
||||
src = tmp_path / "marked.txt"
|
||||
src.write_text(_marked_text(400, rng), encoding="utf-8")
|
||||
r = _run_cli(str(src), "--json", env_extra={"WATERMARKS_GUMBEL_KEY": KEY_HEX})
|
||||
assert r.returncode == 0, r.stderr
|
||||
assert json.loads(r.stdout)["is_watermarked"] is True
|
||||
|
||||
|
||||
def test_cli_bad_hex_key(tmp_path):
|
||||
src = tmp_path / "plain.txt"
|
||||
src.write_text("hello world", encoding="utf-8")
|
||||
r = _run_cli(str(src), "--key", "0xzz", "--json")
|
||||
assert r.returncode == 2
|
||||
|
||||
|
||||
# --- GumbelTextDetector protocol ---------------------------------------------
|
||||
|
||||
|
||||
def test_gumbel_detector_unconfigured():
|
||||
assert text_detectors.GumbelTextDetector().available() is False
|
||||
report = text_detectors.GumbelTextDetector().detect("hello")
|
||||
assert report["available"] is False
|
||||
assert "WATERMARKS_GUMBEL_KEY" in report["error"]
|
||||
|
||||
|
||||
def test_gumbel_detector_env_key(monkeypatch):
|
||||
monkeypatch.setenv("WATERMARKS_GUMBEL_KEY", KEY_HEX)
|
||||
rng = _rng(9)
|
||||
report = text_detectors.GumbelTextDetector().detect(_marked_text(400, rng))
|
||||
assert report["available"] is True
|
||||
assert report["is_watermarked"] is True
|
||||
assert report["detector"] == "gumbel"
|
||||
|
||||
|
||||
def test_gumbel_detector_constructor_key():
|
||||
rng = _rng(10)
|
||||
report = text_detectors.GumbelTextDetector(key=KEY_HEX).detect(_marked_text(400, rng))
|
||||
assert report["available"] is True
|
||||
assert report["is_watermarked"] is True
|
||||
|
||||
|
||||
def test_gumbel_in_detector_registry(monkeypatch):
|
||||
names = {d.name for d in text_detectors.all_detectors()}
|
||||
assert "gumbel" in names
|
||||
monkeypatch.setenv("WATERMARKS_GUMBEL_KEY", KEY_HEX)
|
||||
assert text_detectors.detector_status()["gumbel"] is True
|
||||
|
||||
|
||||
# --- rewrite_text.py gumbel evaluator ----------------------------------------
|
||||
|
||||
|
||||
def _rewrite_kwargs(**overrides):
|
||||
kwargs = dict(
|
||||
backend="ollama",
|
||||
model="m",
|
||||
base_url="http://127.0.0.1:11434",
|
||||
api_key=None,
|
||||
strength="paraphrase",
|
||||
lang="French",
|
||||
original_lang="English",
|
||||
timeout=10,
|
||||
layer_a_after=False,
|
||||
temperature=0.9,
|
||||
candidates=1,
|
||||
)
|
||||
kwargs.update(overrides)
|
||||
return kwargs
|
||||
|
||||
|
||||
def test_rewrite_gumbel_evaluator_clears_mark(monkeypatch):
|
||||
rng = _rng(11)
|
||||
original = _marked_text(400, rng)
|
||||
monkeypatch.setattr(
|
||||
rewrite_text,
|
||||
"call_ollama",
|
||||
lambda *a, **k: "alpha beta gamma delta epsilon zeta eta theta",
|
||||
)
|
||||
_out, info = rewrite_text.rewrite(original, **_rewrite_kwargs(gumbel_key=KEY_HEX))
|
||||
assert info["evaluator"] == "gumbel"
|
||||
assert info["passed"] is True
|
||||
assert info["attempts_made"] == 1
|
||||
g = info["gumbel"]
|
||||
assert g["before"]["is_watermarked"] is True
|
||||
assert g["after"]["is_watermarked"] is False
|
||||
assert g["cleared"] is True
|
||||
assert info["candidate_scores"][0]["evaluation"]["detector"] == "gumbel"
|
||||
assert _out == "alpha beta gamma delta epsilon zeta eta theta"
|
||||
|
||||
|
||||
def test_gumbel_takes_priority_over_markllm(monkeypatch):
|
||||
built = {}
|
||||
|
||||
class _FakeMarkLLM:
|
||||
def __init__(self, **kwargs):
|
||||
built["markllm"] = True
|
||||
|
||||
def available(self):
|
||||
return True
|
||||
|
||||
def detect(self, text):
|
||||
return {"detector": "markllm", "available": True, "is_watermarked": False, "score": 0.0}
|
||||
|
||||
monkeypatch.setattr(rewrite_text, "MarkLLMTextDetector", _FakeMarkLLM)
|
||||
monkeypatch.setattr(rewrite_text, "call_ollama", lambda *a, **k: "alpha beta gamma delta")
|
||||
_out, info = rewrite_text.rewrite(
|
||||
"plain text here",
|
||||
**_rewrite_kwargs(gumbel_key=KEY_HEX, markllm_scheme="kgw", markllm_dir="/x"),
|
||||
)
|
||||
assert built["markllm"] is True
|
||||
assert info["evaluator"] == "gumbel"
|
||||
assert "markllm" in info and "gumbel" in info
|
||||
assert info["passed"] is True
|
||||
|
||||
|
||||
def test_gumbel_key_does_not_build_markllm(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
rewrite_text,
|
||||
"MarkLLMTextDetector",
|
||||
lambda *a, **k: pytest.fail("markllm must not be built when only gumbel is set"),
|
||||
)
|
||||
monkeypatch.setattr(rewrite_text, "call_ollama", lambda *a, **k: "alpha beta gamma delta")
|
||||
_out, info = rewrite_text.rewrite("plain text here", **_rewrite_kwargs(gumbel_key=KEY_HEX))
|
||||
assert info["evaluator"] == "gumbel"
|
||||
assert info["passed"] is True
|
||||
assert "markllm" not in info
|
||||
|
||||
|
||||
def test_no_gumbel_without_key(monkeypatch):
|
||||
monkeypatch.setattr(rewrite_text, "call_ollama", lambda *a, **k: "alpha beta gamma delta")
|
||||
_out, info = rewrite_text.rewrite("plain text here", **_rewrite_kwargs())
|
||||
assert "gumbel" not in info
|
||||
assert info["evaluator"] == "lexical-divergence"
|
||||
|
||||
|
||||
def test_gumbel_key_flag_defaults_from_env(monkeypatch):
|
||||
monkeypatch.setenv("WATERMARKS_GUMBEL_KEY", KEY_HEX)
|
||||
args = rewrite_text.build_parser().parse_args(["x.txt"])
|
||||
assert args.gumbel_key == KEY_HEX
|
||||
monkeypatch.delenv("WATERMARKS_GUMBEL_KEY", raising=False)
|
||||
args = rewrite_text.build_parser().parse_args(["x.txt"])
|
||||
assert args.gumbel_key is None
|
||||
@@ -248,8 +248,8 @@ def test_run_all_text_detectors_can_exclude_markllm(monkeypatch):
|
||||
lambda: pytest.fail("must not construct MarkLLM when excluded"),
|
||||
)
|
||||
reports = text_detectors.run_all_text_detectors("hello", include_markllm=False)
|
||||
assert len(reports) == 1 # claude placeholder only
|
||||
assert {r["detector"] for r in reports} == {"claude-text"}
|
||||
assert len(reports) == 2 # gumbel + claude placeholder
|
||||
assert {r["detector"] for r in reports} == {"gumbel", "claude-text"}
|
||||
|
||||
|
||||
def test_run_all_text_detectors_injects_markllm_instance(monkeypatch, tmp_path):
|
||||
@@ -284,12 +284,12 @@ def test_claude_placeholder():
|
||||
|
||||
def test_detector_status_keys():
|
||||
status = text_detectors.detector_status()
|
||||
assert set(status) == {"markllm", "claude-text"}
|
||||
assert set(status) == {"markllm", "gumbel", "claude-text"}
|
||||
|
||||
|
||||
def test_run_all_text_detectors_length():
|
||||
reports = text_detectors.run_all_text_detectors("hello")
|
||||
assert len(reports) == 2
|
||||
assert len(reports) == 3 # markllm + gumbel + claude placeholder
|
||||
assert all("detector" in r for r in reports)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user