mirror of
https://github.com/guillaumemeyer/watermarks-remover.git
synced 2026-08-22 13:11:57 +02:00
Merge branch 'main' into fix/74-layer-a-docx-odt
This commit is contained in:
Guillaume Meyer (The Opinionated Man)
@@ -3,7 +3,7 @@
|
||||
smoke-markllm bootstrap-markllm docker-markllm-build docker-markllm-help \
|
||||
smoke-markdiffusion bootstrap-markdiffusion docker-markdiffusion-build docker-markdiffusion-help \
|
||||
docker-core-build docker-core-help serve compose-up compose-up-heavy compose-check \
|
||||
install-skill clean
|
||||
install-skill install-cursor-text-skill clean
|
||||
|
||||
SCRIPTS := service/scripts
|
||||
PYTHON ?= $(shell if [ -x .venv/bin/python ]; then echo .venv/bin/python; else echo python3; fi)
|
||||
@@ -110,6 +110,9 @@ install-skill:
|
||||
ln -sfn $(CURDIR)/skills/remove-ai-marks $(HOME)/.grok/skills/remove-ai-marks
|
||||
@echo "linked -> $(HOME)/.grok/skills/remove-ai-marks"
|
||||
|
||||
install-cursor-text-skill:
|
||||
$(PYTHON) install_skill.py
|
||||
|
||||
clean:
|
||||
find . -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true
|
||||
rm -rf .pytest_cache .venv
|
||||
|
||||
@@ -45,6 +45,36 @@ ln -sfn "$(pwd)/skills/remove-ai-marks" ~/.grok/skills/remove-ai-marks
|
||||
|
||||
Invoke with `/remove-ai-marks` or ask to “strip AI watermarks / C2PA / Claude marks / SynthID-class text.”
|
||||
|
||||
### Optional Cursor text-only skill
|
||||
|
||||
[`skills/clean-user-facing-text/`](skills/clean-user-facing-text/) is a
|
||||
self-contained Cursor skill for authorized manuscripts, documentation, and web
|
||||
copy. It excludes image, C2PA, service, and external-model tooling.
|
||||
|
||||
Install it into `~/.cursor/skills/clean-user-facing-text`:
|
||||
|
||||
```bash
|
||||
python3 install_skill.py
|
||||
```
|
||||
|
||||
On Windows, use `py install_skill.py`. The `install-skill.sh` wrapper is
|
||||
provided for macOS/Linux shells. Existing installations are preserved unless
|
||||
you pass `--force`; replacement is staged first and the previous install is
|
||||
kept as a uniquely named backup.
|
||||
|
||||
Skill invocation is model-selected. Projects that explicitly adopt this
|
||||
workflow can also copy the optional rule:
|
||||
|
||||
```bash
|
||||
mkdir -p /path/to/project/.cursor/rules
|
||||
cp integrations/cursor/clean-user-facing-text.mdc \
|
||||
/path/to/project/.cursor/rules/clean-user-facing-text.mdc
|
||||
```
|
||||
|
||||
For all projects, put the same instruction in Cursor **User Rules** instead.
|
||||
Rules improve consistency but remain model instructions; Cursor does not expose
|
||||
a deterministic pre-send filter for final chat responses.
|
||||
|
||||
### Start the service
|
||||
|
||||
The fastest path is a local HTTP server (Python 3.10+ stdlib only — no deps, no Docker):
|
||||
@@ -617,6 +647,18 @@ See [`skills/remove-ai-marks/references/ethics.md`](skills/remove-ai-marks/refer
|
||||
|
||||
**Responsible use:** This project is for content you own or are authorized to process. Users must adhere to local regulations and use it responsibly. The developers disclaim any liability for potential misuse by users.
|
||||
|
||||
## Ecosystem
|
||||
|
||||
Third-party projects that wrap or complement this repository, listed for discoverability only. **They are not maintained, endorsed, or supported by this project.** This project does not review their code, vouch for their behavior or guarantees, or take responsibility for anything you install or run from this list. Each project is governed by its own license, maintainers, and documentation — read those before using it.
|
||||
|
||||
### MetaClean — desktop GUI
|
||||
|
||||
[MetaClean](https://github.com/Moresyl/metaclean) is an independent MIT-licensed Rust/Tauri desktop application (Windows, macOS, Linux) providing a packaged native GUI for drag-and-drop metadata cleaning, with a system tray and Explorer integration. It is a separate codebase: it does not call this repository's Python service, and its supported formats and cleaning guarantees differ from this project's. See its README for details.
|
||||
|
||||
### Adding a project
|
||||
|
||||
To register a project here, open a PR adding a short entry — project name, what it wraps or adds, and a link to its own repository. Keep entries brief and factual; do not claim compatibility with, or endorsement by, this project.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
@@ -627,6 +669,10 @@ make smoke # quick CLI smoke on fixtures
|
||||
|
||||
## Changelog
|
||||
|
||||
### Unreleased
|
||||
|
||||
- **Fix `inspect` missing Layer A carriers in markdown/HTML**: `inspect_container` never scanned the document body, so a `.md` or `.html` file holding invisible Unicode came back `suspicious: false` while `clean_container` went on to strip it — the same bytes saved as `.txt` were correctly flagged. The scan now runs for exactly the formats `clean_container` scrubs, so inspect predicts clean. Container reports gain `suspicious_total` (the same key `TextInspectReport` uses, so the HTTP `suspicious` flag and the `inspect_file` exit code pick it up) and `layer_a_hits`
|
||||
|
||||
### [v0.5.0](https://github.com/guillaumemeyer/watermarks-remover/releases/tag/v0.5.0) — service & Docker distribution, HTTP API, and verification harnesses
|
||||
|
||||
**Service / Docker distribution**
|
||||
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
exec python3 "$ROOT/install_skill.py" "$@"
|
||||
@@ -0,0 +1,102 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Install the lightweight text skill for Cursor."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
SKILL_NAME = "clean-user-facing-text"
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
SOURCE = ROOT / "skills" / SKILL_NAME
|
||||
|
||||
|
||||
def _present(path: Path) -> bool:
|
||||
return path.exists() or path.is_symlink()
|
||||
|
||||
|
||||
def _remove_path(path: Path) -> None:
|
||||
if path.is_symlink() or path.is_file():
|
||||
path.unlink()
|
||||
elif path.exists():
|
||||
shutil.rmtree(path)
|
||||
|
||||
|
||||
def _stage(skills_dir: Path) -> tuple[Path, Path]:
|
||||
skills_dir.mkdir(parents=True, exist_ok=True)
|
||||
staging_root = Path(
|
||||
tempfile.mkdtemp(prefix=f".{SKILL_NAME}.staging.", dir=skills_dir)
|
||||
)
|
||||
staged_skill = staging_root / SKILL_NAME
|
||||
try:
|
||||
shutil.copytree(SOURCE, staged_skill)
|
||||
if not (staged_skill / "SKILL.md").is_file():
|
||||
raise RuntimeError("staged skill is missing SKILL.md")
|
||||
except BaseException:
|
||||
shutil.rmtree(staging_root, ignore_errors=True)
|
||||
raise
|
||||
return staging_root, staged_skill
|
||||
|
||||
|
||||
def _install(destination: Path, force: bool) -> tuple[bool, Path | None]:
|
||||
if _present(destination) and not force:
|
||||
print(f"already exists: {destination}", file=sys.stderr)
|
||||
print(
|
||||
"No changes made. Re-run with --force to back up and replace.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return False, None
|
||||
|
||||
staging_root, staged_skill = _stage(destination.parent)
|
||||
backup: Path | None = None
|
||||
try:
|
||||
if _present(destination):
|
||||
backup = destination.with_name(
|
||||
f"{destination.name}.backup.{uuid.uuid4().hex[:12]}"
|
||||
)
|
||||
os.replace(destination, backup)
|
||||
try:
|
||||
os.replace(staged_skill, destination)
|
||||
except BaseException:
|
||||
if backup is not None and not _present(destination):
|
||||
os.replace(backup, destination)
|
||||
raise
|
||||
finally:
|
||||
shutil.rmtree(staging_root, ignore_errors=True)
|
||||
|
||||
return True, backup
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"--force",
|
||||
action="store_true",
|
||||
help="Back up and replace an existing Cursor installation",
|
||||
)
|
||||
parser.add_argument("--cursor-home", help="Override Cursor home (default: ~/.cursor)")
|
||||
args = parser.parse_args()
|
||||
|
||||
cursor_home = Path(
|
||||
args.cursor_home
|
||||
or os.environ.get("CURSOR_HOME", Path.home() / ".cursor")
|
||||
).expanduser()
|
||||
destination = cursor_home / "skills" / SKILL_NAME
|
||||
|
||||
installed, backup = _install(destination, args.force)
|
||||
if not installed:
|
||||
return 1
|
||||
if backup is not None:
|
||||
print(f"Cursor: backed up existing skill to {backup}")
|
||||
print(f"Cursor: installed {destination}")
|
||||
print("Start a new Cursor session if the skill does not appear automatically.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
description: Apply text hygiene to natural-language content intended for readers
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
# Clean user-facing text
|
||||
|
||||
Before finalizing substantial natural-language content intended for readers, use the `clean-user-facing-text` skill.
|
||||
|
||||
- Apply this only to content the user owns or is authorized to process, and preserve required disclosures.
|
||||
- Apply it to articles, papers, reports, documentation, emails, product copy, UI text, Markdown prose, and HTML prose.
|
||||
- Preserve facts, numbers, names, citations, requirements, language, tone, and formatting.
|
||||
- Do not modify fenced or inline code, commands, paths, URLs, identifiers, APIs, formulas, or verbatim quotations.
|
||||
- Treat statistical-watermark reduction as best-effort; never claim the result is certified undetectable or proves human authorship.
|
||||
- Skip the skill for code-only tasks.
|
||||
@@ -51,6 +51,11 @@ def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument("path", type=Path, help="Directory to audit recursively")
|
||||
p.add_argument("--json", action="store_true", help="Emit a JSON report")
|
||||
p.add_argument(
|
||||
"--check-stylometry",
|
||||
action="store_true",
|
||||
help="Also evaluate text files for AI statistical & stylometric signals",
|
||||
)
|
||||
p.add_argument(
|
||||
"--skip",
|
||||
default="",
|
||||
@@ -76,7 +81,7 @@ def main() -> int:
|
||||
if path.stat().st_size > MAX_INPUT_BYTES:
|
||||
skipped.append({"path": str(path), "reason": "too large"})
|
||||
continue
|
||||
files.append(scan_file(path))
|
||||
files.append(scan_file(path, check_stylometry=args.check_stylometry))
|
||||
except Exception as e: # keep the audit going on one bad file
|
||||
skipped.append({"path": str(path), "reason": str(e)})
|
||||
|
||||
|
||||
@@ -13,6 +13,7 @@ from common import CONFIDENCE_LEVELS, classify_finding_confidence
|
||||
from container_meta import inspect_container
|
||||
from format_dispatch import classify
|
||||
from image_meta import inspect_image
|
||||
from score_stylometry import score_text_stylometry
|
||||
from text_unicode import inspect_text
|
||||
|
||||
|
||||
@@ -32,7 +33,11 @@ def text_findings(report: Any) -> tuple[list[str], list[str], int]:
|
||||
return findings, confidences, report.suspicious_total
|
||||
|
||||
|
||||
def scan_file(path: Path, display_name: str | None = None) -> dict[str, Any]:
|
||||
def scan_file(
|
||||
path: Path,
|
||||
display_name: str | None = None,
|
||||
check_stylometry: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
"""Inspect one local file and return a normalized audit item."""
|
||||
name = display_name or str(path)
|
||||
kind = classify(path)
|
||||
@@ -44,7 +49,7 @@ def scan_file(path: Path, display_name: str | None = None) -> dict[str, Any]:
|
||||
return {"path": name, "kind": "text", "error": str(e)}
|
||||
report = inspect_text(text)
|
||||
findings, confidences, suspicious = text_findings(report)
|
||||
return {
|
||||
item: dict[str, Any] = {
|
||||
"path": name,
|
||||
"kind": "text",
|
||||
"has_c2pa": False,
|
||||
@@ -54,6 +59,14 @@ def scan_file(path: Path, display_name: str | None = None) -> dict[str, Any]:
|
||||
"confidence": confidences,
|
||||
"notes": report.notes,
|
||||
}
|
||||
if check_stylometry and text:
|
||||
s_rep = score_text_stylometry(text, path=name)
|
||||
item["stylometry"] = s_rep.to_dict()
|
||||
if s_rep.score >= 0.65:
|
||||
item["findings"].append(f"stylometry [high_probability] score {s_rep.score:.2f} ({s_rep.confidence_level})")
|
||||
item["confidence"].append("probable")
|
||||
item["suspicious_total"] += 1
|
||||
return item
|
||||
|
||||
if kind == "image":
|
||||
report = inspect_image(path)
|
||||
@@ -71,23 +84,25 @@ def scan_file(path: Path, display_name: str | None = None) -> dict[str, Any]:
|
||||
report = inspect_container(path)
|
||||
findings = list(report.findings)
|
||||
confidences = [classify_finding_confidence(f) for f in report.findings]
|
||||
suspicious = 0
|
||||
# Layer A body-scan findings (and count) already come from
|
||||
# inspect_container() for markdown/html; it mirrors clean_container().
|
||||
suspicious = report.layer_a_total
|
||||
stylometry_dict = None
|
||||
|
||||
# Text-bearing containers also get a Layer A scan of their visible text,
|
||||
# mirroring the skill's "container + Layer A" workflow.
|
||||
if report.format in ("html", "markdown"):
|
||||
if check_stylometry and report.format in ("html", "markdown"):
|
||||
try:
|
||||
text = path.read_text(encoding="utf-8", errors="surrogateescape")
|
||||
except OSError:
|
||||
text = ""
|
||||
if text:
|
||||
t_report = inspect_text(text)
|
||||
t_findings, t_confidences, t_suspicious = text_findings(t_report)
|
||||
findings.extend(t_findings)
|
||||
confidences.extend(t_confidences)
|
||||
suspicious = t_suspicious
|
||||
s_rep = score_text_stylometry(text, path=name)
|
||||
stylometry_dict = s_rep.to_dict()
|
||||
if s_rep.score >= 0.65:
|
||||
findings.append(f"stylometry [high_probability] score {s_rep.score:.2f} ({s_rep.confidence_level})")
|
||||
confidences.append("probable")
|
||||
suspicious += 1
|
||||
|
||||
return {
|
||||
item = {
|
||||
"path": name,
|
||||
"kind": report.format,
|
||||
"has_c2pa": report.has_c2pa,
|
||||
@@ -97,6 +112,9 @@ def scan_file(path: Path, display_name: str | None = None) -> dict[str, Any]:
|
||||
"confidence": confidences,
|
||||
"notes": report.notes,
|
||||
}
|
||||
if stylometry_dict:
|
||||
item["stylometry"] = stylometry_dict
|
||||
return item
|
||||
|
||||
|
||||
def is_actionable(item: dict[str, Any]) -> bool:
|
||||
|
||||
@@ -34,6 +34,11 @@ def main() -> int:
|
||||
action="store_true",
|
||||
help="Paranoid: strip all load-bearing invisibles too (emoji glue, script joiners, flag tags, same-script fillers/selectors, orthographic Cf)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--strip-bidi",
|
||||
action="store_true",
|
||||
help="Also strip legitimate RTL/LTR directional marks and isolates",
|
||||
)
|
||||
p.add_argument("--stats", action="store_true", help="Print stats JSON to stderr")
|
||||
p.add_argument(
|
||||
"--force-text",
|
||||
@@ -55,6 +60,7 @@ def main() -> int:
|
||||
aggressive_homoglyphs=args.aggressive_homoglyphs,
|
||||
normalize_spaces=not args.no_normalize_spaces,
|
||||
strip_emoji_glue=args.strip_emoji_glue,
|
||||
strip_bidi=args.strip_bidi,
|
||||
)
|
||||
|
||||
out = args.output
|
||||
|
||||
@@ -67,6 +67,12 @@ class ContainerInspectReport:
|
||||
tools: dict[str, Any] = field(default_factory=dict)
|
||||
details: dict[str, Any] = field(default_factory=dict)
|
||||
notes: list[str] = field(default_factory=list)
|
||||
# Layer A (invisible/format Unicode) scan of the text body, populated only
|
||||
# for the formats clean_container() actually scrubs. Without it, inspect
|
||||
# reported a markdown/html file carrying invisible carriers as clean while
|
||||
# clean then went on to remove them.
|
||||
layer_a_total: int = 0
|
||||
layer_a_hits: list[dict] = field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
@@ -81,6 +87,10 @@ class ContainerInspectReport:
|
||||
"tools": self.tools,
|
||||
"details": self.details,
|
||||
"notes": self.notes,
|
||||
# Same key as TextInspectReport so every caller — including the
|
||||
# HTTP server's `suspicious` flag — reads both report kinds alike.
|
||||
"suspicious_total": self.layer_a_total,
|
||||
"layer_a_hits": self.layer_a_hits,
|
||||
}
|
||||
|
||||
|
||||
@@ -897,15 +907,32 @@ def inspect_container(path: Path) -> ContainerInspectReport:
|
||||
elif fmt == "odt":
|
||||
has_c2pa, has_ai, findings, details = inspect_odt(data)
|
||||
elif fmt == "html":
|
||||
text = data.decode("utf-8", errors="replace")
|
||||
has_c2pa, has_ai, findings, details = inspect_html(text)
|
||||
# surrogateescape, not replace: clean_container() decodes the same way,
|
||||
# and U+FFFD substitutions would make the two disagree on the counts.
|
||||
body = data.decode("utf-8", errors="surrogateescape")
|
||||
has_c2pa, has_ai, findings, details = inspect_html(body)
|
||||
elif fmt == "markdown":
|
||||
text = data.decode("utf-8", errors="replace")
|
||||
has_c2pa, has_ai, findings, details = inspect_markdown(text)
|
||||
body = data.decode("utf-8", errors="surrogateescape")
|
||||
has_c2pa, has_ai, findings, details = inspect_markdown(body)
|
||||
else:
|
||||
has_c2pa, has_ai, findings = False, False, [f"unsupported container: {fmt}"]
|
||||
details = {"unsupported": True}
|
||||
|
||||
# Layer A body scan for exactly the formats clean_container() scrubs, so
|
||||
# inspect predicts clean rather than contradicting it.
|
||||
layer_a_total = 0
|
||||
layer_a_hits: list[dict] = []
|
||||
if fmt in ("markdown", "html"):
|
||||
from text_unicode import inspect_text # local import to avoid cycles
|
||||
|
||||
ta = inspect_text(body).to_dict()
|
||||
layer_a_total = ta["suspicious_total"]
|
||||
layer_a_hits = ta["hits"]
|
||||
for h in layer_a_hits:
|
||||
findings.append(
|
||||
f"layer-a: {h['codepoint']} {h['label']} x{h['count']} ({h['kind']})"
|
||||
)
|
||||
|
||||
notes: list[str] = []
|
||||
if fmt == "pdf":
|
||||
notes.append("PDF inspection is best-effort; exiftool/c2patool give more reliable metadata detection")
|
||||
@@ -913,6 +940,11 @@ def inspect_container(path: Path) -> ContainerInspectReport:
|
||||
notes.append("DOCX: only metadata/provenance parts are scanned; visible body text is ignored")
|
||||
if "unsupported" in details:
|
||||
notes.append(f"format not fully inspected: {fmt}")
|
||||
if layer_a_total:
|
||||
notes.append(
|
||||
f"layer A: {layer_a_total} invisible/format codepoint(s) in body text; "
|
||||
"clean removes these"
|
||||
)
|
||||
|
||||
if fmt in ("svg", "pdf", "docx") and not tools:
|
||||
tools = run_optional_tools(path)
|
||||
@@ -926,6 +958,8 @@ def inspect_container(path: Path) -> ContainerInspectReport:
|
||||
tools=tools,
|
||||
details=details,
|
||||
notes=notes,
|
||||
layer_a_total=layer_a_total,
|
||||
layer_a_hits=layer_a_hits,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -94,7 +94,11 @@ def main() -> int:
|
||||
print(f"AI metadata: {report.has_ai_metadata}")
|
||||
for f in report.findings:
|
||||
print(f" - [{classify_finding_confidence(f)}] {f}")
|
||||
return 0 if not (report.has_c2pa or report.has_ai_metadata) else 1
|
||||
# layer_a_total counts body-text carriers that clean will strip; the text
|
||||
# branch above already exits non-zero for those, so containers match.
|
||||
if report.has_c2pa or report.has_ai_metadata or report.layer_a_total:
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Inspect text for invisible Unicode / space homoglyphs (Layer A)."""
|
||||
"""Inspect text for invisible Unicode / space homoglyphs (Layer A) and optional stylometry."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -11,6 +11,7 @@ from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
from common import emit_json, read_text_input # noqa: E402
|
||||
from score_stylometry import print_human_stylometry_report, score_text_stylometry # noqa: E402
|
||||
from text_unicode import human_report, inspect_text # noqa: E402
|
||||
|
||||
|
||||
@@ -28,6 +29,17 @@ def main() -> int:
|
||||
action="store_true",
|
||||
help="Paranoid: flag all load-bearing invisibles too (emoji glue, script joiners, flag tags, same-script fillers/selectors, orthographic Cf)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--stylometry",
|
||||
action="store_true",
|
||||
help="Also run zero-LLM statistical and stylometric AI cadence scoring",
|
||||
)
|
||||
p.add_argument(
|
||||
"--threshold",
|
||||
type=float,
|
||||
default=0.65,
|
||||
help="Score threshold for --stylometry exit code (default: 0.65)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--force-text",
|
||||
action="store_true",
|
||||
@@ -36,16 +48,38 @@ def main() -> int:
|
||||
args = p.parse_args()
|
||||
|
||||
text = read_text_input(args.path, allow_binary=args.force_text)
|
||||
if text is None:
|
||||
return 2
|
||||
|
||||
report = inspect_text(
|
||||
text,
|
||||
aggressive=args.aggressive,
|
||||
strip_emoji_glue=args.strip_emoji_glue,
|
||||
)
|
||||
|
||||
stylometry_report = None
|
||||
if args.stylometry:
|
||||
input_label = "<stdin>" if args.path == "-" else args.path
|
||||
stylometry_report = score_text_stylometry(text, path=input_label)
|
||||
|
||||
if args.json:
|
||||
emit_json(report.to_dict())
|
||||
data = report.to_dict()
|
||||
if stylometry_report:
|
||||
data["stylometry"] = stylometry_report.to_dict()
|
||||
emit_json(data)
|
||||
else:
|
||||
print(human_report(report))
|
||||
return 0 if report.suspicious_total == 0 else 1
|
||||
if stylometry_report:
|
||||
print("\n" + "=" * 50 + "\n")
|
||||
print_human_stylometry_report(stylometry_report, explain=True)
|
||||
|
||||
exit_code = 0
|
||||
if report.suspicious_total > 0:
|
||||
exit_code = 1
|
||||
if stylometry_report and stylometry_report.score >= args.threshold:
|
||||
exit_code = 1
|
||||
|
||||
return exit_code
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -0,0 +1,404 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Zero-LLM statistical & stylometric AI-text detector.
|
||||
|
||||
Evaluates text for high-frequency AI cadence markers, sentence-length
|
||||
burstiness variance, lexical diversity (MATTR), and structural uniformity.
|
||||
Stdlib-only, no PyTorch or external model dependencies.
|
||||
|
||||
Exit codes:
|
||||
0 clean (score < threshold)
|
||||
1 suspicious / AI stylometric signals detected (score >= threshold)
|
||||
2 bad input (missing file, binary data, bad args)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import unicodedata
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
SCRIPTS_DIR = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(SCRIPTS_DIR))
|
||||
|
||||
from common import ( # noqa: E402
|
||||
MAX_INPUT_BYTES,
|
||||
classify_finding_confidence,
|
||||
emit_json,
|
||||
eprint,
|
||||
looks_binary,
|
||||
read_text_input,
|
||||
)
|
||||
|
||||
DEFAULT_THRESHOLD = 0.65
|
||||
MIN_SAMPLE_WORDS = 30
|
||||
FULL_WEIGHT_WORDS = 100
|
||||
|
||||
# High-frequency formulaic transition markers, hedging verbs, and structural
|
||||
# boilerplate commonly overrepresented in AI-generated text across frontier LLMs.
|
||||
AI_PHRASE_PATTERNS: tuple[tuple[str, str, float], ...] = (
|
||||
# (regex_pattern, human_label, weight)
|
||||
(r"\bdelve(?:s|d)?\s+into\b", "delve into", 1.2),
|
||||
(r"\ba\s+testament\s+to\b", "a testament to", 1.1),
|
||||
(r"\brich\s+tapestry(?:\s+of)?\b", "rich tapestry", 1.3),
|
||||
(r"\bplays?\s+a\s+(?:pivotal|crucial|vital|key)\s+role\b", "plays a pivotal/crucial role", 1.0),
|
||||
(r"\bin\s+(?:today'?s|the)\s+(?:(?:fast-paced|ever-evolving|digital|rapidly\s+changing)\s+)*(?:world|landscape|era|environment)\b", "in today's fast-paced world/landscape", 1.4),
|
||||
(r"\bit\s+is\s+(?:important|essential|crucial|worth\s+noting)\s+to\s+(?:note|remember|consider|highlight)\b", "it is important/crucial to note", 0.9),
|
||||
(r"\bnot\s+only\b[\w\s,]+\bbut\s+(?:also\s+)?(?:serves\s+to|acts\s+as|highlights)\b", "not only ... but also serves to", 0.8),
|
||||
(r"\bserve(?:s|d)?\s+as\s+a\s+(?:beacon|reminder|catalyst|cornerstone)\b", "serves as a beacon/catalyst/cornerstone", 1.1),
|
||||
(r"\bunderscore(?:s|d)?\s+the\s+(?:importance|need|significance)\b", "underscores the importance/need", 0.9),
|
||||
(r"\bfoster(?:s|ing|ed)?\s+a\s+(?:sense|culture|deeper\s+understanding)\b", "fosters a sense/culture", 0.9),
|
||||
(r"\bseamlessly\s+(?:integrates?|integrated|blends?|combine[sd]?)\b", "seamlessly integrates/blends", 1.0),
|
||||
(r"\bnavigat(?:e|ing|es|ed)\s+the\s+(?:complexities|intricacies|nuances)\b", "navigating the complexities/nuances", 1.0),
|
||||
(r"\bmultifaceted\s+(?:nature|approach|landscape)\b", "multifaceted nature/approach", 1.0),
|
||||
(r"\bharness(?:ing|ed|es)?\s+the\s+power\s+of\b", "harnessing the power of", 1.0),
|
||||
(r"\ba\s+myriad\s+of\b", "a myriad of", 0.8),
|
||||
(r"\bparadigm\s+shift\b", "paradigm shift", 0.9),
|
||||
(r"\bholistic\s+(?:approach|view|perspective)\b", "holistic approach/perspective", 0.9),
|
||||
(r"\bin\s+conclusion\b[,\s]", "in conclusion", 0.8),
|
||||
(r"\bto\s+summarize\b[,\s]", "to summarize", 0.8),
|
||||
(r"\bultimately\b[,\s]", "ultimately,", 0.6),
|
||||
(r"\bfurthermore\b[,\s]", "furthermore,", 0.6),
|
||||
(r"\bmoreover\b[,\s]", "moreover,", 0.6),
|
||||
(r"\bas\s+an\s+ai\b", "as an AI", 1.5),
|
||||
(r"\bi\s+hope\s+this\s+helps\b", "I hope this helps", 1.2),
|
||||
)
|
||||
|
||||
RE_SENTENCE_SPLIT = re.compile(r"(?<=[.!?])\s+(?=[A-Z0-9\"'(\[])")
|
||||
RE_WORDS = re.compile(r"\b[\w'-]+\b", re.UNICODE)
|
||||
|
||||
|
||||
@dataclass
|
||||
class MarkerMatch:
|
||||
phrase: str
|
||||
count: int
|
||||
weight: float
|
||||
samples: list[str] = field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"phrase": self.phrase,
|
||||
"count": self.count,
|
||||
"weight": self.weight,
|
||||
"samples": self.samples,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class StylometryReport:
|
||||
path: str
|
||||
word_count: int
|
||||
sentence_count: int
|
||||
burstiness_cv: float
|
||||
lexical_diversity: float
|
||||
ai_ngram_density: float
|
||||
matched_markers: list[dict[str, Any]]
|
||||
score: float
|
||||
confidence_level: str # CLEAN | LOW | MEDIUM | HIGH
|
||||
status: str # ok | insufficient_length
|
||||
findings: list[str] = field(default_factory=list)
|
||||
notes: list[str] = field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"path": self.path,
|
||||
"word_count": self.word_count,
|
||||
"sentence_count": self.sentence_count,
|
||||
"burstiness_cv": round(self.burstiness_cv, 4),
|
||||
"lexical_diversity": round(self.lexical_diversity, 4),
|
||||
"ai_ngram_density": round(self.ai_ngram_density, 4),
|
||||
"matched_markers": self.matched_markers,
|
||||
"score": round(self.score, 4),
|
||||
"confidence_level": self.confidence_level,
|
||||
"status": self.status,
|
||||
"findings": self.findings,
|
||||
"findings_confidence": [
|
||||
classify_finding_confidence(f) for f in self.findings
|
||||
],
|
||||
"notes": self.notes,
|
||||
}
|
||||
|
||||
|
||||
def extract_sentences(text: str) -> list[str]:
|
||||
"""Split text into sentences while ignoring blank lines and code block markers."""
|
||||
clean_lines = []
|
||||
in_code_block = False
|
||||
for line in text.splitlines():
|
||||
trimmed = line.strip()
|
||||
if trimmed.startswith("```"):
|
||||
in_code_block = not in_code_block
|
||||
continue
|
||||
if in_code_block or not trimmed:
|
||||
continue
|
||||
clean_lines.append(trimmed)
|
||||
|
||||
raw_text = "\n".join(clean_lines)
|
||||
if not raw_text.strip():
|
||||
return []
|
||||
|
||||
chunks = re.split(r"(?<=[.!?])\s+|\n+", raw_text)
|
||||
sentences = [s.strip() for s in chunks if s.strip()]
|
||||
return sentences
|
||||
|
||||
|
||||
def extract_words(text: str) -> list[str]:
|
||||
"""Extract normalized alphanumeric word tokens."""
|
||||
return [w.lower() for w in RE_WORDS.findall(text)]
|
||||
|
||||
|
||||
def compute_burstiness(sentences: list[str]) -> tuple[float, float, float]:
|
||||
"""Compute mean sentence word length, standard deviation, and coefficient of variation (CV)."""
|
||||
if not sentences:
|
||||
return 0.0, 0.0, 0.0
|
||||
|
||||
lengths = [len(extract_words(s)) for s in sentences]
|
||||
lengths = [L for L in lengths if L > 0]
|
||||
if len(lengths) < 2:
|
||||
mean_len = float(lengths[0]) if lengths else 0.0
|
||||
return mean_len, 0.0, 0.0
|
||||
|
||||
mean_len = sum(lengths) / len(lengths)
|
||||
variance = sum((x - mean_len) ** 2 for x in lengths) / (len(lengths) - 1)
|
||||
std_dev = math.sqrt(variance)
|
||||
cv = (std_dev / mean_len) if mean_len > 0 else 0.0
|
||||
return mean_len, std_dev, cv
|
||||
|
||||
|
||||
def compute_mattr(words: list[str], window_size: int = 50) -> float:
|
||||
"""Compute Moving-Average Type-Token Ratio (MATTR) across sliding windows."""
|
||||
n = len(words)
|
||||
if n == 0:
|
||||
return 0.0
|
||||
if n <= window_size:
|
||||
return len(set(words)) / n
|
||||
|
||||
total_ttr = 0.0
|
||||
num_windows = n - window_size + 1
|
||||
# Sliding window counts
|
||||
current_window = Counter(words[:window_size])
|
||||
total_ttr += len(current_window) / window_size
|
||||
|
||||
for i in range(1, num_windows):
|
||||
leaving_word = words[i - 1]
|
||||
entering_word = words[i + window_size - 1]
|
||||
|
||||
current_window[leaving_word] -= 1
|
||||
if current_window[leaving_word] == 0:
|
||||
del current_window[leaving_word]
|
||||
|
||||
current_window[entering_word] += 1
|
||||
total_ttr += len(current_window) / window_size
|
||||
|
||||
return total_ttr / num_windows
|
||||
|
||||
|
||||
def scan_ai_phrases(text: str) -> list[MarkerMatch]:
|
||||
"""Find and tally high-frequency AI cadence phrases."""
|
||||
matches: list[MarkerMatch] = []
|
||||
for pattern, label, weight in AI_PHRASE_PATTERNS:
|
||||
found_spans = []
|
||||
for m in re.finditer(pattern, text, re.IGNORECASE):
|
||||
found_spans.append(m.group(0))
|
||||
if found_spans:
|
||||
matches.append(
|
||||
MarkerMatch(
|
||||
phrase=label,
|
||||
count=len(found_spans),
|
||||
weight=weight,
|
||||
samples=found_spans[:3],
|
||||
)
|
||||
)
|
||||
return matches
|
||||
|
||||
|
||||
def score_text_stylometry(text: str, path: str = "<text>") -> StylometryReport:
|
||||
"""Run full multi-dimensional stylometric analysis and return a structured report."""
|
||||
words = extract_words(text)
|
||||
word_count = len(words)
|
||||
sentences = extract_sentences(text)
|
||||
sentence_count = len(sentences)
|
||||
|
||||
findings: list[str] = []
|
||||
notes: list[str] = []
|
||||
|
||||
# 1. Length Guard
|
||||
if word_count < MIN_SAMPLE_WORDS:
|
||||
marker_matches = scan_ai_phrases(text)
|
||||
for m in marker_matches:
|
||||
findings.append(f"AI phrase marker '{m.phrase}' found ({m.count}x)")
|
||||
notes.append(
|
||||
f"Sample contains {word_count} words; statistical stylometry is uncalibrated below {MIN_SAMPLE_WORDS} words"
|
||||
)
|
||||
return StylometryReport(
|
||||
path=path,
|
||||
word_count=word_count,
|
||||
sentence_count=sentence_count,
|
||||
burstiness_cv=0.0,
|
||||
lexical_diversity=compute_mattr(words),
|
||||
ai_ngram_density=0.0,
|
||||
matched_markers=[m.to_dict() for m in marker_matches],
|
||||
score=0.0,
|
||||
confidence_level="CLEAN",
|
||||
status="insufficient_length",
|
||||
findings=findings,
|
||||
notes=notes,
|
||||
)
|
||||
|
||||
# 2. Metric Calculations
|
||||
_, _, cv = compute_burstiness(sentences)
|
||||
mattr = compute_mattr(words)
|
||||
marker_matches = scan_ai_phrases(text)
|
||||
|
||||
# N-gram density: weighted marker instances per 100 words
|
||||
total_marker_weight = sum(m.count * m.weight for m in marker_matches)
|
||||
ngram_density = (total_marker_weight / (word_count / 100.0)) if word_count > 0 else 0.0
|
||||
|
||||
# 3. Component Sub-scores (0.0 to 1.0)
|
||||
# Burstiness subscore: low CV (<0.35) is strongly characteristic of LLMs; high CV (>0.60) is human
|
||||
if cv < 0.25:
|
||||
burstiness_score = 0.95
|
||||
elif cv < 0.35:
|
||||
burstiness_score = 0.80
|
||||
elif cv < 0.45:
|
||||
burstiness_score = 0.50
|
||||
elif cv < 0.55:
|
||||
burstiness_score = 0.25
|
||||
else:
|
||||
burstiness_score = 0.05
|
||||
|
||||
# N-gram subscore: >1.5 weighted matches per 100 words is very high
|
||||
if ngram_density >= 2.0:
|
||||
ngram_score = 1.0
|
||||
elif ngram_density >= 1.0:
|
||||
ngram_score = 0.75
|
||||
elif ngram_density >= 0.5:
|
||||
ngram_score = 0.45
|
||||
elif ngram_density > 0:
|
||||
ngram_score = 0.20
|
||||
else:
|
||||
ngram_score = 0.0
|
||||
|
||||
# Lexical uniformity subscore: LLMs cluster tightly around MATTR 0.65-0.78 for 50-word windows
|
||||
if 0.68 <= mattr <= 0.76:
|
||||
diversity_score = 0.40
|
||||
else:
|
||||
diversity_score = 0.10
|
||||
|
||||
# 4. Composite Scoring & Small-Sample Dampening
|
||||
raw_composite = (burstiness_score * 0.45) + (ngram_score * 0.45) + (diversity_score * 0.10)
|
||||
|
||||
# Dampening factor: scales smoothly from 0.4 at MIN_SAMPLE_WORDS up to 1.0 at FULL_WEIGHT_WORDS
|
||||
if word_count < FULL_WEIGHT_WORDS:
|
||||
dampener = 0.4 + 0.6 * ((word_count - MIN_SAMPLE_WORDS) / (FULL_WEIGHT_WORDS - MIN_SAMPLE_WORDS))
|
||||
notes.append(
|
||||
f"Sample word count ({word_count}) is in calibration range ({MIN_SAMPLE_WORDS}–{FULL_WEIGHT_WORDS}); score dampened by factor {dampener:.2f}"
|
||||
)
|
||||
else:
|
||||
dampener = 1.0
|
||||
|
||||
final_score = min(1.0, max(0.0, raw_composite * dampener))
|
||||
|
||||
# 5. Classify Findings & Confidence Tiers
|
||||
if marker_matches:
|
||||
for m in marker_matches:
|
||||
findings.append(f"AI cadence phrase '{m.phrase}' ({m.count}x)")
|
||||
|
||||
if cv < 0.35 and sentence_count >= 3:
|
||||
findings.append(f"Unnaturally uniform sentence cadence (CV={cv:.2f} < 0.35)")
|
||||
|
||||
if ngram_density >= 1.0:
|
||||
findings.append(f"Elevated AI formulaic transition density ({ngram_density:.2f}/100w)")
|
||||
|
||||
if final_score >= 0.75:
|
||||
confidence = "HIGH"
|
||||
elif final_score >= 0.50:
|
||||
confidence = "MEDIUM"
|
||||
elif final_score >= 0.25:
|
||||
confidence = "LOW"
|
||||
else:
|
||||
confidence = "CLEAN"
|
||||
|
||||
return StylometryReport(
|
||||
path=path,
|
||||
word_count=word_count,
|
||||
sentence_count=sentence_count,
|
||||
burstiness_cv=cv,
|
||||
lexical_diversity=mattr,
|
||||
ai_ngram_density=ngram_density,
|
||||
matched_markers=[m.to_dict() for m in marker_matches],
|
||||
score=final_score,
|
||||
confidence_level=confidence,
|
||||
status="ok",
|
||||
findings=findings,
|
||||
notes=notes,
|
||||
)
|
||||
|
||||
|
||||
def print_human_stylometry_report(report: StylometryReport, explain: bool = False) -> None:
|
||||
"""Print clean human-readable output to stdout."""
|
||||
print(f"=== Stylometric AI-Text Report: {report.path} ===")
|
||||
print(f"Status: {report.status}")
|
||||
print(f"Confidence Level: {report.confidence_level}")
|
||||
print(f"AI Probability: {report.score * 100:.1f}% (score: {report.score:.3f})")
|
||||
print(f"Word Count: {report.word_count}")
|
||||
print(f"Sentence Count: {report.sentence_count}")
|
||||
print(f"Sentence CV: {report.burstiness_cv:.3f}")
|
||||
print(f"Lexical Diversity: {report.lexical_diversity:.3f} (MATTR)")
|
||||
print(f"AI Marker Density: {report.ai_ngram_density:.3f} / 100 words")
|
||||
|
||||
if report.findings:
|
||||
print("\nFindings:")
|
||||
for f in report.findings:
|
||||
print(f" - {f}")
|
||||
|
||||
if explain and report.matched_markers:
|
||||
print("\nMatched Phrases Detail:")
|
||||
for m in report.matched_markers:
|
||||
print(f" * {m['phrase']} (occurrences: {m['count']}, weight: {m['weight']})")
|
||||
if m.get("samples"):
|
||||
print(f" sample: \"{m['samples'][0]}\"")
|
||||
|
||||
if report.notes:
|
||||
print("\nNotes:")
|
||||
for n in report.notes:
|
||||
print(f" * {n}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(
|
||||
description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
p.add_argument("path", nargs="?", default="-", help="Text file to score ('-' for stdin)")
|
||||
p.add_argument("--threshold", type=float, default=DEFAULT_THRESHOLD, help=f"Score threshold to trigger exit code 1 (default: {DEFAULT_THRESHOLD})")
|
||||
p.add_argument("--json", action="store_true", help="Emit JSON output")
|
||||
p.add_argument("--explain", action="store_true", help="Include detailed matched phrase breakdown")
|
||||
|
||||
args = p.parse_args()
|
||||
|
||||
# Read input
|
||||
text = read_text_input(args.path)
|
||||
if text is None:
|
||||
return 2
|
||||
|
||||
input_label = "<stdin>" if args.path == "-" else args.path
|
||||
report = score_text_stylometry(text, path=input_label)
|
||||
|
||||
if args.json:
|
||||
emit_json(report.to_dict())
|
||||
else:
|
||||
print_human_stylometry_report(report, explain=args.explain)
|
||||
|
||||
if report.score >= args.threshold:
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -16,6 +16,7 @@ Exit codes:
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import contextlib
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
@@ -90,13 +91,17 @@ def main() -> int:
|
||||
return 2
|
||||
rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
|
||||
|
||||
codebook_v4 = SpectralCodebookV4()
|
||||
codebook_v4.load(str(codebook))
|
||||
# Upstream prints progress ("CodebookV4 loaded: ...") straight to
|
||||
# stdout, which corrupts --json for any caller that parses us
|
||||
# (image_meta.py json.loads our stdout). Keep stdout ours alone.
|
||||
with contextlib.redirect_stdout(sys.stderr):
|
||||
codebook_v4 = SpectralCodebookV4()
|
||||
codebook_v4.load(str(codebook))
|
||||
|
||||
extractor = RobustSynthIDExtractor()
|
||||
result = extractor.detect_from_v4_codebook(
|
||||
rgb, codebook_v4, model=args.model
|
||||
)
|
||||
extractor = RobustSynthIDExtractor()
|
||||
result = extractor.detect_from_v4_codebook(
|
||||
rgb, codebook_v4, model=args.model
|
||||
)
|
||||
except Exception as e:
|
||||
print(f"scorer error: {e}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
@@ -44,6 +44,7 @@ from common import ( # noqa: E402
|
||||
from container_meta import clean_container, inspect_container # noqa: E402
|
||||
from format_dispatch import classify_bytes # noqa: E402
|
||||
from image_meta import clean_image, inspect_image # noqa: E402
|
||||
from score_stylometry import score_text_stylometry # noqa: E402
|
||||
from text_unicode import clean_text, inspect_text # noqa: E402
|
||||
|
||||
VERSION = os.environ.get("WATERMARKS_SERVER_VERSION", "dev")
|
||||
@@ -84,6 +85,7 @@ def capabilities() -> dict[str, Any]:
|
||||
},
|
||||
"scorers": {
|
||||
"synthid": bool(os.environ.get("REVERSE_SYNTHID_DIR")),
|
||||
"stylometry": True,
|
||||
},
|
||||
"harnesses": {
|
||||
"markllm": bool(os.environ.get("MARKLLM_DIR")),
|
||||
@@ -165,7 +167,13 @@ _OPENAPI_PATHS: dict[str, dict[str, Any]] = {
|
||||
type="object",
|
||||
properties={k: _schema(type="boolean") for k in ("ctrlregen", "diffusion")},
|
||||
),
|
||||
"scorers": _schema(type="object", properties={"synthid": _schema(type="boolean")}),
|
||||
"scorers": _schema(
|
||||
type="object",
|
||||
properties={
|
||||
"synthid": _schema(type="boolean"),
|
||||
"stylometry": _schema(type="boolean"),
|
||||
},
|
||||
),
|
||||
"harnesses": _schema(type="object", properties={"markllm": _schema(type="boolean")}),
|
||||
},
|
||||
)
|
||||
@@ -403,14 +411,17 @@ class Handler(BaseHTTPRequestHandler):
|
||||
if kind == "text":
|
||||
if looks_binary(data):
|
||||
raise ValueError("refusing to inspect bytes that look like a binary container as text")
|
||||
report = inspect_text(data.decode("utf-8", errors="surrogateescape")).to_dict()
|
||||
raw_text = data.decode("utf-8", errors="surrogateescape")
|
||||
report = inspect_text(raw_text).to_dict()
|
||||
s_rep = score_text_stylometry(raw_text, path=name or "<text>")
|
||||
report["stylometry"] = s_rep.to_dict()
|
||||
elif kind == "image":
|
||||
report = inspect_image(path).to_dict()
|
||||
else:
|
||||
report = inspect_container(path).to_dict()
|
||||
suspicious = bool(report.get("suspicious_total")) or bool(
|
||||
report.get("has_c2pa") or report.get("has_ai_metadata")
|
||||
)
|
||||
) or bool(report.get("stylometry", {}).get("score", 0.0) >= 0.65)
|
||||
self._respond(HTTPStatus.OK, {"ok": True, "kind": kind, "report": report, "suspicious": suspicious})
|
||||
|
||||
def _handle_clean(self, data: bytes, name: str, body: dict[str, Any]) -> None:
|
||||
|
||||
@@ -58,7 +58,7 @@ done
|
||||
|
||||
DIR="${DIR:-$DEFAULT_DIR}"
|
||||
mkdir -p "$(dirname "$DIR")"
|
||||
if command -v realpath >/dev/null 2>&1; then
|
||||
if realpath -m . >/dev/null 2>&1; then # BSD/macOS realpath has no -m
|
||||
DIR="$(realpath -m "$DIR")"
|
||||
else
|
||||
DIR="$(cd "$(dirname "$DIR")" && pwd)/$(basename "$DIR")"
|
||||
|
||||
@@ -63,7 +63,7 @@ done
|
||||
|
||||
DIR="${DIR:-$DEFAULT_DIR}"
|
||||
mkdir -p "$(dirname "$DIR")"
|
||||
if command -v realpath >/dev/null 2>&1; then
|
||||
if realpath -m . >/dev/null 2>&1; then # BSD/macOS realpath has no -m
|
||||
DIR="$(realpath -m "$DIR")"
|
||||
else
|
||||
DIR="$(cd "$(dirname "$DIR")" && pwd)/$(basename "$DIR")"
|
||||
|
||||
+145
-11
@@ -5,6 +5,7 @@ from __future__ import annotations
|
||||
import unicodedata
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass, field
|
||||
from difflib import SequenceMatcher
|
||||
|
||||
# Format / invisible controls commonly used for steganography or broken pastes.
|
||||
STRIP_CODEPOINTS: frozenset[int] = frozenset(
|
||||
@@ -185,6 +186,21 @@ _BIDI_CPS: frozenset[int] = frozenset(
|
||||
}
|
||||
)
|
||||
|
||||
# Directional marks and isolates are legitimate in mixed RTL/LTR prose. Inspect
|
||||
# them, but preserve them during the default clean. Embeddings and overrides
|
||||
# remain destructive by default because they can reorder unrelated spans.
|
||||
_PRESERVABLE_BIDI_CPS: frozenset[int] = frozenset(
|
||||
{
|
||||
0x061C,
|
||||
0x200E,
|
||||
0x200F,
|
||||
0x2066,
|
||||
0x2067,
|
||||
0x2068,
|
||||
0x2069,
|
||||
}
|
||||
)
|
||||
|
||||
# Zero-width family (common edit-based carriers)
|
||||
_ZW_FAMILY: frozenset[int] = frozenset(
|
||||
{0x200B, 0x200C, 0x200D, 0x2060, 0xFEFF, 0x180E}
|
||||
@@ -239,6 +255,8 @@ def _is_emoji_base(cp: int) -> bool:
|
||||
"""Return True for characters that can start or continue an emoji sequence."""
|
||||
if 0x1F000 <= cp <= 0x1FAFF:
|
||||
return True
|
||||
if 0x2190 <= cp <= 0x25FF: # arrows, technical symbols, enclosed symbols
|
||||
return True
|
||||
if 0x2600 <= cp <= 0x27BF: # misc symbols / dingbats / arrows
|
||||
return True
|
||||
if 0x2B00 <= cp <= 0x2BFF: # misc symbols and arrows
|
||||
@@ -268,9 +286,71 @@ _HANGUL_FILLERS: frozenset[int] = frozenset({0x115F, 0x1160})
|
||||
_SCRIPT_GLUE: frozenset[int] = _MONGOLIAN_FVS | _KHMER_VOWELS | _HANGUL_FILLERS
|
||||
|
||||
|
||||
def _is_joining_letter(cp: int) -> bool:
|
||||
"""Non-ASCII letter/mark — the neighbour that makes a joiner orthographic."""
|
||||
return cp > 0x7F and unicodedata.category(chr(cp))[0] in ("L", "M")
|
||||
def _joining_script(cp: int) -> str | None:
|
||||
"""Return a broad script group where ZWJ/ZWNJ can be orthographic."""
|
||||
for start, end, name in (
|
||||
(0x0600, 0x08FF, "arabic"),
|
||||
(0x0900, 0x0DFF, "indic"),
|
||||
(0x0F00, 0x109F, "south-asian"),
|
||||
(0x1780, 0x17FF, "khmer"),
|
||||
(0x1800, 0x18AF, "mongolian"),
|
||||
):
|
||||
if start <= cp <= end and unicodedata.category(chr(cp))[0] in ("L", "M"):
|
||||
return name
|
||||
return None
|
||||
|
||||
|
||||
def _is_cjk_ideograph(cp: int) -> bool:
|
||||
return (
|
||||
0x3400 <= cp <= 0x4DBF
|
||||
or 0x4E00 <= cp <= 0x9FFF
|
||||
or 0xF900 <= cp <= 0xFAFF
|
||||
or 0x20000 <= cp <= 0x323AF
|
||||
)
|
||||
|
||||
|
||||
def _is_mongolian_base(cp: int) -> bool:
|
||||
return 0x1800 <= cp <= 0x18AF
|
||||
|
||||
|
||||
def _is_variation_selector(cp: int) -> bool:
|
||||
return cp in _VS_SUPPLEMENT or 0xFE00 <= cp <= 0xFE0F or 0x180B <= cp <= 0x180D
|
||||
|
||||
|
||||
def _valid_flag_tag_indices(text: str) -> set[int]:
|
||||
"""Indices in complete subdivision-flag tag sequences."""
|
||||
valid: set[int] = set()
|
||||
i = 0
|
||||
while i < len(text):
|
||||
if ord(text[i]) != 0x1F3F4: # waving black flag
|
||||
i += 1
|
||||
continue
|
||||
j = i + 1
|
||||
while j < len(text) and 0xE0020 <= ord(text[j]) <= 0xE007E:
|
||||
j += 1
|
||||
if j > i + 1 and j < len(text) and ord(text[j]) == 0xE007F:
|
||||
valid.update(range(i + 1, j + 1))
|
||||
i = j + 1
|
||||
else:
|
||||
i += 1
|
||||
return valid
|
||||
|
||||
|
||||
def _valid_bidi_embedding_indices(text: str) -> set[int]:
|
||||
"""Indices belonging to complete LRE/RLE ... PDF pairs, excluding overrides."""
|
||||
valid: set[int] = set()
|
||||
stack: list[tuple[int, int]] = []
|
||||
for index, char in enumerate(text):
|
||||
cp = ord(char)
|
||||
if cp in (0x202A, 0x202B, 0x202D, 0x202E):
|
||||
stack.append((cp, index))
|
||||
elif cp == 0x202C:
|
||||
if not stack:
|
||||
continue
|
||||
opener, opener_index = stack.pop()
|
||||
if opener in (0x202A, 0x202B):
|
||||
valid.update((opener_index, index))
|
||||
return valid
|
||||
|
||||
|
||||
def _is_mongolian_letter(cp: int) -> bool:
|
||||
@@ -294,6 +374,7 @@ def _is_glue(cp: int) -> bool:
|
||||
or same-script filler/selector (Mongolian FVS, Khmer vowel, Hangul filler)."""
|
||||
return (
|
||||
_is_emoji_glue(cp)
|
||||
or _is_variation_selector(cp)
|
||||
or cp in _SCRIPT_JOINERS
|
||||
or cp in _TAG_RANGE
|
||||
or cp in _SCRIPT_GLUE
|
||||
@@ -303,10 +384,15 @@ def _is_glue(cp: int) -> bool:
|
||||
def _decide(
|
||||
ch: str,
|
||||
prev_kept: str | None,
|
||||
prev_input: str | None,
|
||||
next_input: str | None,
|
||||
*,
|
||||
valid_flag_tag: bool,
|
||||
valid_bidi_embedding: bool,
|
||||
normalize_spaces: bool,
|
||||
treat_confusables: bool,
|
||||
strip_emoji_glue: bool,
|
||||
strip_bidi: bool,
|
||||
) -> tuple[str, str, str | None]:
|
||||
"""Classify one input char for both inspect and clean.
|
||||
|
||||
@@ -315,13 +401,36 @@ def _decide(
|
||||
kind is the inspect classification (None when not suspicious).
|
||||
"""
|
||||
cp = ord(ch)
|
||||
if valid_bidi_embedding and not strip_bidi:
|
||||
return ("keep", ch, None)
|
||||
if cp in _PRESERVABLE_BIDI_CPS and not strip_bidi:
|
||||
return ("keep", ch, None)
|
||||
if prev_input is not None and not strip_emoji_glue:
|
||||
prev_cp = ord(prev_input)
|
||||
if cp in _VS_SUPPLEMENT and _is_cjk_ideograph(prev_cp):
|
||||
return ("keep", ch, None)
|
||||
if 0x180B <= cp <= 0x180D and _is_mongolian_base(prev_cp):
|
||||
return ("keep", ch, None)
|
||||
if 0xFE00 <= cp <= 0xFE0D and _is_cjk_ideograph(prev_cp):
|
||||
return ("keep", ch, None)
|
||||
if _is_emoji_glue(cp) and not strip_emoji_glue:
|
||||
if prev_kept is not None and _is_emoji_base(ord(prev_kept)):
|
||||
if cp in (0xFE0E, 0xFE0F) and prev_input is not None and _is_emoji_base(ord(prev_input)):
|
||||
return ("keep", ch, None)
|
||||
if (
|
||||
cp == 0x200D
|
||||
and prev_kept is not None
|
||||
and next_input is not None
|
||||
and _is_emoji_base(ord(prev_kept))
|
||||
and _is_emoji_base(ord(next_input))
|
||||
):
|
||||
return ("keep", ch, None)
|
||||
if not strip_emoji_glue:
|
||||
if cp in _SCRIPT_JOINERS and prev_kept is not None and _is_joining_letter(ord(prev_kept)):
|
||||
return ("keep", ch, None)
|
||||
if cp in _TAG_RANGE and prev_kept is not None and _is_emoji_base(ord(prev_kept)):
|
||||
if cp in _SCRIPT_JOINERS and prev_input is not None and next_input is not None:
|
||||
prev_script = _joining_script(ord(prev_input))
|
||||
next_script = _joining_script(ord(next_input))
|
||||
if prev_script is not None and prev_script == next_script:
|
||||
return ("keep", ch, None)
|
||||
if cp in _TAG_RANGE and valid_flag_tag:
|
||||
return ("keep", ch, None)
|
||||
if cp in _MONGOLIAN_FVS and prev_kept is not None and _is_mongolian_letter(ord(prev_kept)):
|
||||
return ("keep", ch, None)
|
||||
@@ -398,13 +507,20 @@ def inspect_text(
|
||||
) -> TextInspectReport:
|
||||
buckets: dict[tuple[int, str], list[int]] = {}
|
||||
prev_kept: str | None = None
|
||||
valid_flag_tags = _valid_flag_tag_indices(text)
|
||||
valid_bidi_embeddings = _valid_bidi_embedding_indices(text)
|
||||
for i, ch in enumerate(text):
|
||||
action, out_char, kind = _decide(
|
||||
ch,
|
||||
prev_kept,
|
||||
text[i - 1] if i > 0 else None,
|
||||
text[i + 1] if i + 1 < len(text) else None,
|
||||
valid_flag_tag=i in valid_flag_tags,
|
||||
valid_bidi_embedding=i in valid_bidi_embeddings,
|
||||
normalize_spaces=True,
|
||||
treat_confusables=aggressive,
|
||||
strip_emoji_glue=strip_emoji_glue,
|
||||
strip_bidi=True,
|
||||
)
|
||||
if kind is None:
|
||||
# Kept; glue (emoji/script joiner/tag) does not advance the
|
||||
@@ -438,7 +554,7 @@ def inspect_text(
|
||||
"Layer A only: invisible/format Unicode and space homoglyphs (edit-based carriers).",
|
||||
"Statistical (token-sampling) watermarks are not detectable here; use Layer B rewrite.",
|
||||
"Inspect kinds: strip, bidi, tag_chars, variation_selector, zwj_family, private_use, space, confusable, other_cf.",
|
||||
"Load-bearing invisibles are preserved by default: emoji glue (ZWJ/VS after an emoji base), script joiners (ZWNJ/ZWJ inside complex scripts), flag tag chars, same-script fillers/selectors (Mongolian FVS, Khmer inherent vowels, Hangul jamo fillers), and orthographic Arabic/Syriac Cf marks. Use --strip-emoji-glue for paranoid mode (strips them all).",
|
||||
"Load-bearing invisibles are preserved by default during cleaning: emoji glue, CJK/Mongolian variation selectors, script joiners, complete flag tag sequences, same-script fillers/selectors (Mongolian FVS, Khmer inherent vowels, Hangul jamo fillers), RTL directional marks/paired embeddings, and orthographic Arabic/Syriac Cf marks. Inspection still reports bidi controls. Use explicit strip flags only after review.",
|
||||
]
|
||||
if not hits:
|
||||
notes.append(
|
||||
@@ -455,20 +571,28 @@ def clean_text(
|
||||
aggressive_homoglyphs: bool = False,
|
||||
normalize_spaces: bool = True,
|
||||
strip_emoji_glue: bool = False,
|
||||
strip_bidi: bool = False,
|
||||
) -> tuple[str, dict]:
|
||||
"""Return cleaned text and a stats dict."""
|
||||
removed: Counter[str] = Counter()
|
||||
replaced: Counter[str] = Counter()
|
||||
out_chars: list[str] = []
|
||||
prev_kept: str | None = None
|
||||
valid_flag_tags = _valid_flag_tag_indices(text)
|
||||
valid_bidi_embeddings = _valid_bidi_embedding_indices(text)
|
||||
|
||||
for ch in text:
|
||||
for i, ch in enumerate(text):
|
||||
action, out_char, _kind = _decide(
|
||||
ch,
|
||||
prev_kept,
|
||||
text[i - 1] if i > 0 else None,
|
||||
text[i + 1] if i + 1 < len(text) else None,
|
||||
valid_flag_tag=i in valid_flag_tags,
|
||||
valid_bidi_embedding=i in valid_bidi_embeddings,
|
||||
normalize_spaces=normalize_spaces,
|
||||
treat_confusables=aggressive_homoglyphs,
|
||||
strip_emoji_glue=strip_emoji_glue,
|
||||
strip_bidi=strip_bidi,
|
||||
)
|
||||
if action == "keep":
|
||||
out_chars.append(out_char)
|
||||
@@ -485,11 +609,20 @@ def clean_text(
|
||||
# prev_kept unchanged
|
||||
|
||||
result = "".join(out_chars)
|
||||
nfkc_changed = False
|
||||
if nfkc:
|
||||
before = result
|
||||
result = unicodedata.normalize("NFKC", result)
|
||||
if result != before:
|
||||
replaced["NFKC_normalize"] += abs(len(before) - len(result)) or 1
|
||||
nfkc_changed = True
|
||||
changed_inputs = sum(
|
||||
end - start
|
||||
for operation, start, end, _new_start, _new_end in SequenceMatcher(
|
||||
None, before, result, autojunk=False
|
||||
).get_opcodes()
|
||||
if operation != "equal"
|
||||
)
|
||||
replaced["NFKC_normalize"] += changed_inputs or 1
|
||||
|
||||
# Collapse runs of spaces only if we introduced space replacements? Keep conservative: no.
|
||||
|
||||
@@ -499,7 +632,8 @@ def clean_text(
|
||||
"removed": dict(removed),
|
||||
"replaced": dict(replaced),
|
||||
"removed_count": sum(removed.values()),
|
||||
"replaced_count": sum(v for k, v in replaced.items() if k != "NFKC_normalize"),
|
||||
"replaced_count": sum(replaced.values()),
|
||||
"nfkc_changed": nfkc_changed,
|
||||
}
|
||||
return result, stats
|
||||
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
name: clean-user-facing-text
|
||||
description: Clean and finalize authorized natural-language text intended for readers by auditing suspicious invisible Unicode and rewriting prose while preserving facts and meaning. Use in Cursor when the user asks to clean, humanize, polish, or finalize articles, manuscripts, reports, documentation, emails, product copy, UI text, Markdown, or HTML prose, or when an installed Cursor Rule explicitly requires this workflow. Do not use for code-only tasks or undisclosed authorship evasion; leave code, commands, identifiers, paths, APIs, formulas, citations, required disclosures, and verbatim quotations unchanged.
|
||||
---
|
||||
|
||||
# Clean user-facing text
|
||||
|
||||
Apply a final text-hygiene pass to prose the user owns or is authorized to process. Treat Unicode cleanup as deterministic and statistical-watermark reduction as best-effort; never claim that a rewrite proves human authorship or is undetectable. Preserve required academic, legal, platform, and regulatory disclosures.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Identify the prose that readers will see.
|
||||
2. Protect non-prose spans:
|
||||
- fenced and inline code
|
||||
- commands, paths, URLs, identifiers, API names, and exact values
|
||||
- formulas, citations, and text the user asks to quote verbatim
|
||||
3. Preserve every claim, fact, number, name, citation, and requirement.
|
||||
4. Rewrite the remaining prose once:
|
||||
- vary clause order, sentence boundaries, rhythm, connectors, and function words
|
||||
- replace formulaic transitions and filler with direct, natural wording
|
||||
- preserve the requested language, tone, structure, and formatting; never translate unless asked
|
||||
- for non-English text, use fluent constructions native to that language rather than English sentence patterns
|
||||
- do not add or remove claims merely to increase variation
|
||||
5. For text artifacts or supplied text files, run the deterministic Unicode pass after rewriting.
|
||||
6. Return only the polished result unless the user asks for an audit or explanation.
|
||||
|
||||
## Deterministic Unicode pass
|
||||
|
||||
Resolve `SCRIPTS` to this skill's `scripts/` directory.
|
||||
Use the available Python 3 launcher for the platform. Replace `PYTHON` below
|
||||
with `python3` on most macOS/Linux systems, `py` on Windows, or another verified
|
||||
Python 3 command.
|
||||
|
||||
Inspect first when editing an existing file:
|
||||
|
||||
```bash
|
||||
PYTHON "$SCRIPTS/inspect_text.py" --json INPUT
|
||||
PYTHON "$SCRIPTS/clean_text.py" INPUT -o OUTPUT --stats --no-normalize-spaces
|
||||
PYTHON "$SCRIPTS/inspect_text.py" --json OUTPUT
|
||||
```
|
||||
|
||||
Use `-` for stdin. Prefer a new `*.cleaned.*` output unless the user explicitly requests in-place editing.
|
||||
|
||||
Use `--no-normalize-spaces` by default so NBSP, narrow no-break spaces, figure spaces, and CJK ideographic spaces retain their layout semantics. Normalize spaces only when the user requests it.
|
||||
|
||||
Do not use `--aggressive-homoglyphs`, `--nfkc`, or `--strip-emoji-glue` unless the user requests aggressive normalization and accepts possible changes to multilingual text, emoji, directionality, or typography.
|
||||
|
||||
The scripts support plain text, source text, Markdown, and HTML source as text. For mixed Markdown or HTML, inspect hit positions first. If a hit falls inside protected code, attributes, or another non-prose span, do not run whole-file cleanup; clean only the prose segments or leave that hit unchanged. Do not pass binary containers such as PDF, DOCX, images, or archives.
|
||||
|
||||
For a chat-only response that is not written to a file, perform the rewrite workflow directly. Do not claim that the chat response received a deterministic post-send Unicode filter.
|
||||
|
||||
## Code boundary
|
||||
|
||||
When prose and code are mixed, rewrite prose only. Never rename variables, alter string literals, reformat code, or change executable output as part of this skill. If a Markdown or HTML file contains executable snippets, preserve those spans byte-for-byte whenever practical.
|
||||
|
||||
## Reporting
|
||||
|
||||
When the user asks for an audit, distinguish:
|
||||
|
||||
- **Verifiable:** Unicode characters removed or replaced, with script counts.
|
||||
- **Best-effort:** prose was rewritten to alter token and syntax patterns.
|
||||
- **Not established:** official detector evasion, human authorship, or removal of a vendor's secret-key watermark.
|
||||
|
||||
For technical background, read `references/watermark-notes.md`. For misuse or disclosure questions, read `references/responsible-use.md`.
|
||||
@@ -0,0 +1,7 @@
|
||||
# Responsible use
|
||||
|
||||
Use this skill for content the user owns or is authorized to process, including privacy, publishing hygiene, accessibility, and research.
|
||||
|
||||
Do not present cleaned text as proof that no AI assistance occurred. Do not help misrepresent authorship or bypass disclosure requirements in academic, legal, platform, or regulatory settings.
|
||||
|
||||
When intent is ambiguous, perform ordinary writing and Unicode hygiene without making detector-evasion claims. When disclosure is required, preserve or add the appropriate disclosure.
|
||||
@@ -0,0 +1,20 @@
|
||||
# Text watermark notes
|
||||
|
||||
## Deterministic marks
|
||||
|
||||
Invisible Unicode, bidirectional controls, tag characters, exotic spaces, and selected confusables can carry machine-readable signals or cause broken copy, search, and diffs.
|
||||
|
||||
`clean_text.py` removes or normalizes known carriers and reports exact counts. By default it preserves contextual characters used by emoji, joining scripts, Mongolian selectors, Khmer inherent vowels, Hangul fillers, and Arabic/Syriac orthography. The lightweight workflow also disables space normalization by default so multilingual typography remains intact. Aggressive flags can damage intentional text and should remain opt-in.
|
||||
|
||||
## Statistical marks
|
||||
|
||||
Token-sampling watermarks such as green-list or tournament-sampling schemes live in word choice and token sequences rather than metadata. A substantial rewrite can weaken such signals by changing syntax and vocabulary.
|
||||
|
||||
This is best-effort:
|
||||
|
||||
- no bundled detector has the vendor's secret key
|
||||
- a rewrite generated by the same provider may introduce a new signal
|
||||
- short or predictable text provides little statistical evidence either way
|
||||
- detector behavior can change independently of this skill
|
||||
|
||||
Never describe a successful rewrite as certified, undetectable, or proof of human authorship.
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Strip invisible Unicode / normalize space homoglyphs (Layer A)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
from common import backup_path, cleaned_path, eprint, read_text_input, write_text_output # noqa: E402
|
||||
from text_unicode import clean_text # noqa: E402
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument("path", nargs="?", default="-", help="Input text file, or - for stdin")
|
||||
p.add_argument("-o", "--output", help="Output path (default: stdout or *.cleaned.*)")
|
||||
p.add_argument("--nfkc", action="store_true", help="Apply Unicode NFKC after scrub")
|
||||
p.add_argument(
|
||||
"--aggressive-homoglyphs",
|
||||
action="store_true",
|
||||
help="Map Cyrillic/fullwidth Latin confusables to ASCII Latin",
|
||||
)
|
||||
p.add_argument(
|
||||
"--no-normalize-spaces",
|
||||
action="store_true",
|
||||
help="Do not rewrite exotic spaces to U+0020",
|
||||
)
|
||||
p.add_argument(
|
||||
"--strip-emoji-glue",
|
||||
action="store_true",
|
||||
help="Paranoid: strip all load-bearing invisibles too (emoji glue, script joiners, flag tags, same-script fillers/selectors, orthographic Cf)",
|
||||
)
|
||||
p.add_argument("--stats", action="store_true", help="Print stats JSON to stderr")
|
||||
p.add_argument(
|
||||
"--force-text",
|
||||
action="store_true",
|
||||
help="Clean even when the input looks like a binary container "
|
||||
"(this rewrites the bytes and will corrupt the file)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--in-place",
|
||||
action="store_true",
|
||||
help="Overwrite input file (creates .bak backup)",
|
||||
)
|
||||
args = p.parse_args()
|
||||
|
||||
text = read_text_input(args.path, allow_binary=args.force_text)
|
||||
cleaned, stats = clean_text(
|
||||
text,
|
||||
nfkc=args.nfkc,
|
||||
aggressive_homoglyphs=args.aggressive_homoglyphs,
|
||||
normalize_spaces=not args.no_normalize_spaces,
|
||||
strip_emoji_glue=args.strip_emoji_glue,
|
||||
)
|
||||
|
||||
out = args.output
|
||||
if args.in_place:
|
||||
if args.path in (None, "-"):
|
||||
eprint("--in-place requires a file path")
|
||||
return 2
|
||||
src = Path(args.path)
|
||||
bak = backup_path(src)
|
||||
out = str(src)
|
||||
elif out is None and args.path not in (None, "-"):
|
||||
out = str(cleaned_path(Path(args.path)))
|
||||
|
||||
write_text_output(cleaned, out)
|
||||
|
||||
if args.stats:
|
||||
eprint(json.dumps(stats, indent=2, ensure_ascii=False))
|
||||
else:
|
||||
eprint(
|
||||
f"removed={stats['removed_count']} replaced={stats['replaced_count']} "
|
||||
f"len {stats['input_length']}->{stats['output_length']}"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,178 @@
|
||||
"""Small, dependency-free helpers for the text-only skill."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
MAX_INPUT_BYTES = int(os.environ.get("WATERMARKS_MAX_INPUT_BYTES", str(256 << 20)))
|
||||
MAX_STDIN_BYTES = int(os.environ.get("WATERMARKS_MAX_STDIN_BYTES", str(64 << 20)))
|
||||
BINARY_SNIFF_BYTES = 8192
|
||||
|
||||
BINARY_MAGIC: tuple[tuple[bytes, str], ...] = (
|
||||
(b"PK\x03\x04", "a ZIP container such as DOCX or ODT"),
|
||||
(b"%PDF-", "a PDF"),
|
||||
(b"\x89PNG\r\n\x1a\n", "a PNG image"),
|
||||
(b"\xff\xd8\xff", "a JPEG image"),
|
||||
(b"GIF87a", "a GIF image"),
|
||||
(b"GIF89a", "a GIF image"),
|
||||
(b"RIFF", "a RIFF container"),
|
||||
(b"\x1f\x8b", "a gzip archive"),
|
||||
(b"7z\xbc\xaf\x27\x1c", "a 7-Zip archive"),
|
||||
(b"Rar!\x1a\x07", "a RAR archive"),
|
||||
(b"\x7fELF", "an ELF binary"),
|
||||
)
|
||||
|
||||
_ALLOWED_CONTROLS = frozenset({0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x1B})
|
||||
|
||||
|
||||
def eprint(*args: object) -> None:
|
||||
print(*args, file=sys.stderr)
|
||||
|
||||
|
||||
def _configure_stdio() -> None:
|
||||
for stream, errors in (
|
||||
(sys.stdin, "surrogateescape"),
|
||||
(sys.stdout, "backslashreplace"),
|
||||
(sys.stderr, "backslashreplace"),
|
||||
):
|
||||
reconfigure = getattr(stream, "reconfigure", None)
|
||||
if reconfigure is not None:
|
||||
try:
|
||||
reconfigure(encoding="utf-8", errors=errors)
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
|
||||
|
||||
_configure_stdio()
|
||||
|
||||
|
||||
def looks_binary(data: bytes) -> str | None:
|
||||
if not data:
|
||||
return None
|
||||
for magic, label in BINARY_MAGIC:
|
||||
if data.startswith(magic):
|
||||
return label
|
||||
head = data[:BINARY_SNIFF_BYTES]
|
||||
if b"\x00" in head:
|
||||
return "binary data containing NUL bytes"
|
||||
controls = sum(1 for value in head if value < 0x20 and value not in _ALLOWED_CONTROLS)
|
||||
if controls / len(head) > 0.05:
|
||||
return "binary data dense in control bytes"
|
||||
return None
|
||||
|
||||
|
||||
def guard_binary(data: bytes, origin: str, *, allow_binary: bool = False) -> None:
|
||||
if allow_binary:
|
||||
return
|
||||
kind = looks_binary(data)
|
||||
if kind is None:
|
||||
return
|
||||
eprint(f"refusing to treat {origin} as text: it looks like {kind}.")
|
||||
eprint("This lightweight skill accepts text files only.")
|
||||
raise SystemExit(2)
|
||||
|
||||
|
||||
def _read_stdin_capped(*, allow_binary: bool = False) -> str:
|
||||
stream = getattr(sys.stdin, "buffer", None)
|
||||
if stream is None:
|
||||
text = sys.stdin.read()
|
||||
data = text.encode("utf-8", errors="surrogateescape")
|
||||
if len(data) > MAX_STDIN_BYTES:
|
||||
raise SystemExit(f"refusing stdin input larger than {MAX_STDIN_BYTES} bytes")
|
||||
guard_binary(data[:BINARY_SNIFF_BYTES], "stdin", allow_binary=allow_binary)
|
||||
return text
|
||||
|
||||
chunks: list[bytes] = []
|
||||
total = 0
|
||||
while True:
|
||||
chunk = stream.read(1 << 20)
|
||||
if not chunk:
|
||||
break
|
||||
if not chunks:
|
||||
guard_binary(
|
||||
chunk[:BINARY_SNIFF_BYTES],
|
||||
"stdin",
|
||||
allow_binary=allow_binary,
|
||||
)
|
||||
total += len(chunk)
|
||||
if total > MAX_STDIN_BYTES:
|
||||
raise SystemExit(f"refusing stdin input larger than {MAX_STDIN_BYTES} bytes")
|
||||
chunks.append(chunk)
|
||||
return b"".join(chunks).decode("utf-8", errors="surrogateescape")
|
||||
|
||||
|
||||
def read_text_input(path: str | None, *, allow_binary: bool = False) -> str:
|
||||
if path is None or path == "-":
|
||||
return _read_stdin_capped(allow_binary=allow_binary)
|
||||
source = Path(path)
|
||||
if source.stat().st_size > MAX_INPUT_BYTES:
|
||||
raise SystemExit(f"refusing input larger than {MAX_INPUT_BYTES} bytes: {path}")
|
||||
data = source.read_bytes()
|
||||
guard_binary(data, str(source), allow_binary=allow_binary)
|
||||
return data.decode("utf-8", errors="surrogateescape")
|
||||
|
||||
|
||||
def _default_file_mode() -> int:
|
||||
mask = os.umask(0)
|
||||
os.umask(mask)
|
||||
return 0o666 & ~mask
|
||||
|
||||
|
||||
def safe_write_bytes(path: str | Path, data: bytes) -> None:
|
||||
destination = Path(path)
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
if destination.is_symlink():
|
||||
raise OSError(f"refusing to write through symlink: {destination}")
|
||||
|
||||
fd, temporary = tempfile.mkstemp(
|
||||
prefix=f".{destination.name}.",
|
||||
suffix=".tmp",
|
||||
dir=str(destination.parent),
|
||||
)
|
||||
try:
|
||||
if hasattr(os, "fchmod"):
|
||||
os.fchmod(fd, _default_file_mode())
|
||||
with os.fdopen(fd, "wb") as handle:
|
||||
handle.write(data)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
os.replace(temporary, destination)
|
||||
except BaseException:
|
||||
try:
|
||||
os.unlink(temporary)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
|
||||
|
||||
def write_text_output(text: str, path: str | None) -> None:
|
||||
if path is None or path == "-":
|
||||
sys.stdout.write(text)
|
||||
if text and not text.endswith("\n"):
|
||||
sys.stdout.write("\n")
|
||||
return
|
||||
safe_write_bytes(path, text.encode("utf-8", errors="surrogateescape"))
|
||||
|
||||
|
||||
def backup_path(source: Path) -> Path:
|
||||
backup = source.with_suffix(source.suffix + ".bak")
|
||||
try:
|
||||
safe_write_bytes(backup, source.read_bytes())
|
||||
except OSError as error:
|
||||
eprint(f"cannot create backup {backup}: {error}")
|
||||
raise SystemExit(2) from error
|
||||
return backup
|
||||
|
||||
|
||||
def emit_json(data: Any) -> None:
|
||||
json.dump(data, sys.stdout, indent=2, ensure_ascii=False)
|
||||
sys.stdout.write("\n")
|
||||
|
||||
|
||||
def cleaned_path(source: Path, suffix: str = ".cleaned") -> Path:
|
||||
return source.with_name(f"{source.stem}{suffix}{source.suffix}")
|
||||
+52
@@ -0,0 +1,52 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Inspect text for invisible Unicode / space homoglyphs (Layer A)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Allow running as script from any cwd
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
from common import emit_json, read_text_input # noqa: E402
|
||||
from text_unicode import human_report, inspect_text # noqa: E402
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument("path", nargs="?", default="-", help="Text file path, or - for stdin")
|
||||
p.add_argument("--json", action="store_true", help="JSON report")
|
||||
p.add_argument(
|
||||
"--aggressive",
|
||||
action="store_true",
|
||||
help="Also flag Latin confusable / fullwidth lookalikes",
|
||||
)
|
||||
p.add_argument(
|
||||
"--strip-emoji-glue",
|
||||
action="store_true",
|
||||
help="Paranoid: flag all load-bearing invisibles too (emoji glue, script joiners, flag tags, same-script fillers/selectors, orthographic Cf)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--force-text",
|
||||
action="store_true",
|
||||
help="Scan even when the input looks like a binary container",
|
||||
)
|
||||
args = p.parse_args()
|
||||
|
||||
text = read_text_input(args.path, allow_binary=args.force_text)
|
||||
report = inspect_text(
|
||||
text,
|
||||
aggressive=args.aggressive,
|
||||
strip_emoji_glue=args.strip_emoji_glue,
|
||||
)
|
||||
if args.json:
|
||||
emit_json(report.to_dict())
|
||||
else:
|
||||
print(human_report(report))
|
||||
return 0 if report.suspicious_total == 0 else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
+521
@@ -0,0 +1,521 @@
|
||||
"""Layer A: invisible Unicode / homoglyph space detection and cleaning."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import unicodedata
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
# Format / invisible controls commonly used for steganography or broken pastes.
|
||||
STRIP_CODEPOINTS: frozenset[int] = frozenset(
|
||||
{
|
||||
0x00AD, # soft hyphen
|
||||
0x034F, # combining grapheme joiner
|
||||
0x061C, # Arabic letter mark
|
||||
0x115F, # Hangul choseong filler
|
||||
0x1160, # Hangul jungseong filler
|
||||
0x17B4, # Khmer vowel inherent AQ
|
||||
0x17B5, # Khmer vowel inherent AA
|
||||
0x180B, # Mongolian free variation selector-1
|
||||
0x180C,
|
||||
0x180D,
|
||||
0x180E, # Mongolian vowel separator
|
||||
0x200B, # zero width space
|
||||
0x200C, # zero width non-joiner
|
||||
0x200D, # zero width joiner
|
||||
0x200E, # LRM
|
||||
0x200F, # RLM
|
||||
0x202A, # LRE
|
||||
0x202B, # RLE
|
||||
0x202C, # PDF
|
||||
0x202D, # LRO
|
||||
0x202E, # RLO
|
||||
0x2060, # word joiner
|
||||
0x2061, # function application
|
||||
0x2062, # invisible times
|
||||
0x2063, # invisible separator
|
||||
0x2064, # invisible plus
|
||||
0x2066, # LRI
|
||||
0x2067, # RLI
|
||||
0x2068, # FSI
|
||||
0x2069, # PDI
|
||||
0x206A, # inhibit symmetric swapping
|
||||
0x206B,
|
||||
0x206C,
|
||||
0x206D,
|
||||
0x206E,
|
||||
0x206F,
|
||||
0xFEFF, # BOM / ZWNBSP
|
||||
0xFE00, # variation selectors
|
||||
0xFE01,
|
||||
0xFE02,
|
||||
0xFE03,
|
||||
0xFE04,
|
||||
0xFE05,
|
||||
0xFE06,
|
||||
0xFE07,
|
||||
0xFE08,
|
||||
0xFE09,
|
||||
0xFE0A,
|
||||
0xFE0B,
|
||||
0xFE0C,
|
||||
0xFE0D,
|
||||
0xFE0E,
|
||||
0xFE0F,
|
||||
0xFFF9, # interlinear annotation
|
||||
0xFFFA,
|
||||
0xFFFB,
|
||||
}
|
||||
)
|
||||
|
||||
# Spaces that look like (or substitute for) U+0020.
|
||||
SPACE_HOMOGLYPHS: dict[int, str] = {
|
||||
0x00A0: " ", # no-break space
|
||||
0x1680: " ", # Ogham space mark
|
||||
0x2000: " ", # en quad
|
||||
0x2001: " ", # em quad
|
||||
0x2002: " ", # en space
|
||||
0x2003: " ", # em space
|
||||
0x2004: " ", # three-per-em space
|
||||
0x2005: " ", # four-per-em space
|
||||
0x2006: " ", # six-per-em space
|
||||
0x2007: " ", # figure space
|
||||
0x2008: " ", # punctuation space
|
||||
0x2009: " ", # thin space
|
||||
0x200A: " ", # hair space
|
||||
0x202F: " ", # narrow no-break space
|
||||
0x205F: " ", # medium mathematical space
|
||||
0x3000: " ", # ideographic space
|
||||
}
|
||||
|
||||
# Optional confusable Latin lookalikes (aggressive mode only).
|
||||
LATIN_CONFUSABLES: dict[int, str] = {
|
||||
0x0410: "A", # Cyrillic
|
||||
0x0412: "B",
|
||||
0x0415: "E",
|
||||
0x041A: "K",
|
||||
0x041C: "M",
|
||||
0x041D: "H",
|
||||
0x041E: "O",
|
||||
0x0420: "P",
|
||||
0x0421: "C",
|
||||
0x0422: "T",
|
||||
0x0425: "X",
|
||||
0x0430: "a",
|
||||
0x0435: "e",
|
||||
0x043E: "o",
|
||||
0x0440: "p",
|
||||
0x0441: "c",
|
||||
0x0443: "y",
|
||||
0x0445: "x",
|
||||
0x0456: "i",
|
||||
0xFF21: "A", # fullwidth
|
||||
0xFF22: "B",
|
||||
0xFF23: "C",
|
||||
0xFF24: "D",
|
||||
0xFF25: "E",
|
||||
0xFF26: "F",
|
||||
0xFF27: "G",
|
||||
0xFF28: "H",
|
||||
0xFF29: "I",
|
||||
0xFF2A: "J",
|
||||
0xFF2B: "K",
|
||||
0xFF2C: "L",
|
||||
0xFF2D: "M",
|
||||
0xFF2E: "N",
|
||||
0xFF2F: "O",
|
||||
0xFF30: "P",
|
||||
0xFF31: "Q",
|
||||
0xFF32: "R",
|
||||
0xFF33: "S",
|
||||
0xFF34: "T",
|
||||
0xFF35: "U",
|
||||
0xFF36: "V",
|
||||
0xFF37: "W",
|
||||
0xFF38: "X",
|
||||
0xFF39: "Y",
|
||||
0xFF3A: "Z",
|
||||
0xFF41: "a",
|
||||
0xFF42: "b",
|
||||
0xFF43: "c",
|
||||
0xFF44: "d",
|
||||
0xFF45: "e",
|
||||
0xFF46: "f",
|
||||
0xFF47: "g",
|
||||
0xFF48: "h",
|
||||
0xFF49: "i",
|
||||
0xFF4A: "j",
|
||||
0xFF4B: "k",
|
||||
0xFF4C: "l",
|
||||
0xFF4D: "m",
|
||||
0xFF4E: "n",
|
||||
0xFF4F: "o",
|
||||
0xFF50: "p",
|
||||
0xFF51: "q",
|
||||
0xFF52: "r",
|
||||
0xFF53: "s",
|
||||
0xFF54: "t",
|
||||
0xFF55: "u",
|
||||
0xFF56: "v",
|
||||
0xFF57: "w",
|
||||
0xFF58: "x",
|
||||
0xFF59: "y",
|
||||
0xFF5A: "z",
|
||||
}
|
||||
|
||||
# Variation selectors beyond FE0x (VS17–VS256 in Supplementary Special-purpose)
|
||||
_VS_SUPPLEMENT = range(0xE0100, 0xE01F0)
|
||||
|
||||
|
||||
# Bidi / directional format controls (subset of strip set, finer inspect labels)
|
||||
_BIDI_CPS: frozenset[int] = frozenset(
|
||||
{
|
||||
0x061C,
|
||||
0x200E,
|
||||
0x200F,
|
||||
0x202A,
|
||||
0x202B,
|
||||
0x202C,
|
||||
0x202D,
|
||||
0x202E,
|
||||
0x2066,
|
||||
0x2067,
|
||||
0x2068,
|
||||
0x2069,
|
||||
}
|
||||
)
|
||||
|
||||
# Zero-width family (common edit-based carriers)
|
||||
_ZW_FAMILY: frozenset[int] = frozenset(
|
||||
{0x200B, 0x200C, 0x200D, 0x2060, 0xFEFF, 0x180E}
|
||||
)
|
||||
|
||||
|
||||
def _is_private_use(cp: int) -> bool:
|
||||
"""BMP and supplementary private-use planes (Co: no portable meaning)."""
|
||||
return 0xE000 <= cp <= 0xF8FF or 0xF0000 <= cp <= 0xFFFFD or 0x100000 <= cp <= 0x10FFFD
|
||||
|
||||
|
||||
def _is_strip_cp(cp: int) -> bool:
|
||||
if cp in STRIP_CODEPOINTS:
|
||||
return True
|
||||
if cp in _VS_SUPPLEMENT:
|
||||
return True
|
||||
# Tag characters used in some stego schemes (U+E0001–U+E007F)
|
||||
if 0xE0001 <= cp <= 0xE007F:
|
||||
return True
|
||||
if _is_private_use(cp):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _strip_kind(cp: int) -> str:
|
||||
"""Finer-grained inspect kind for strip-class codepoints."""
|
||||
if 0xE0001 <= cp <= 0xE007F:
|
||||
return "tag_chars"
|
||||
if cp in _VS_SUPPLEMENT or 0xFE00 <= cp <= 0xFE0F or 0x180B <= cp <= 0x180D:
|
||||
return "variation_selector"
|
||||
if cp in _BIDI_CPS:
|
||||
return "bidi"
|
||||
if cp in _ZW_FAMILY:
|
||||
return "zwj_family"
|
||||
if _is_private_use(cp):
|
||||
return "private_use"
|
||||
return "strip"
|
||||
|
||||
|
||||
# Emoji presentation glue: zero-width joiner and text/emoji variation
|
||||
# selectors. These are invisible carriers when free-floating, but after an
|
||||
# emoji base they are part of the visible sequence (⚖️, 👨👩👧, ❤️🔥) and
|
||||
# stripping them visibly alters the text.
|
||||
EMOJI_GLUE_CODEPOINTS: frozenset[int] = frozenset({0x200D, 0xFE0E, 0xFE0F})
|
||||
|
||||
|
||||
def _is_emoji_glue(cp: int) -> bool:
|
||||
return cp in EMOJI_GLUE_CODEPOINTS
|
||||
|
||||
|
||||
def _is_emoji_base(cp: int) -> bool:
|
||||
"""Return True for characters that can start or continue an emoji sequence."""
|
||||
if 0x1F000 <= cp <= 0x1FAFF:
|
||||
return True
|
||||
if 0x2600 <= cp <= 0x27BF: # misc symbols / dingbats / arrows
|
||||
return True
|
||||
if 0x2B00 <= cp <= 0x2BFF: # misc symbols and arrows
|
||||
return True
|
||||
if cp in (0x00A9, 0x00AE, 0x2122, 0x3030, 0x303D, 0x3297, 0x3299):
|
||||
return True
|
||||
if cp in (0x0023, 0x002A) or 0x0030 <= cp <= 0x0039: # keycap bases
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
# ZWNJ/ZWJ are orthographic inside complex scripts (Persian میروم, Devanagari
|
||||
# क्ष); flag emoji are an emoji base followed by tag chars (🏴); and a
|
||||
# handful of Cf codepoints are normal Arabic/Syriac orthography, not carriers.
|
||||
# So are Mongolian free variation selectors (choose a glyph of the preceding
|
||||
# letter), Khmer inherent vowels (invisible but phonemic), and Hangul fillers
|
||||
# (hold a jamo slot in a partial syllable). Each is only meaningful directly
|
||||
# after a base from its own script; isolated instances are contraband.
|
||||
_SCRIPT_JOINERS: frozenset[int] = frozenset({0x200C, 0x200D})
|
||||
_TAG_RANGE = range(0xE0020, 0xE0080)
|
||||
_ORTHOGRAPHIC_CF: frozenset[int] = frozenset(
|
||||
{0x0600, 0x0601, 0x0602, 0x0603, 0x0604, 0x0605, 0x06DD, 0x070F, 0x08E2, 0x110BD, 0x110CD}
|
||||
)
|
||||
_MONGOLIAN_FVS: frozenset[int] = frozenset({0x180B, 0x180C, 0x180D})
|
||||
_KHMER_VOWELS: frozenset[int] = frozenset({0x17B4, 0x17B5})
|
||||
_HANGUL_FILLERS: frozenset[int] = frozenset({0x115F, 0x1160})
|
||||
_SCRIPT_GLUE: frozenset[int] = _MONGOLIAN_FVS | _KHMER_VOWELS | _HANGUL_FILLERS
|
||||
|
||||
|
||||
def _is_joining_letter(cp: int) -> bool:
|
||||
"""Non-ASCII letter/mark — the neighbour that makes a joiner orthographic."""
|
||||
return cp > 0x7F and unicodedata.category(chr(cp))[0] in ("L", "M")
|
||||
|
||||
|
||||
def _is_mongolian_letter(cp: int) -> bool:
|
||||
return 0x1800 <= cp <= 0x18AF and unicodedata.category(chr(cp))[0] == "L"
|
||||
|
||||
|
||||
def _is_khmer_letter(cp: int) -> bool:
|
||||
return 0x1780 <= cp <= 0x17FF and unicodedata.category(chr(cp))[0] == "L"
|
||||
|
||||
|
||||
def _is_hangul_jamo(cp: int) -> bool:
|
||||
return (
|
||||
0x1100 <= cp <= 0x11FF
|
||||
or 0xA960 <= cp <= 0xA97C # Hangul Jamo Extended-A
|
||||
or 0xD7B0 <= cp <= 0xD7C6 # Hangul Jamo Extended-B
|
||||
)
|
||||
|
||||
|
||||
def _is_glue(cp: int) -> bool:
|
||||
"""Load-bearing invisible char: emoji glue, script joiner, flag tag char,
|
||||
or same-script filler/selector (Mongolian FVS, Khmer vowel, Hangul filler)."""
|
||||
return (
|
||||
_is_emoji_glue(cp)
|
||||
or cp in _SCRIPT_JOINERS
|
||||
or cp in _TAG_RANGE
|
||||
or cp in _SCRIPT_GLUE
|
||||
)
|
||||
|
||||
|
||||
def _decide(
|
||||
ch: str,
|
||||
prev_kept: str | None,
|
||||
*,
|
||||
normalize_spaces: bool,
|
||||
treat_confusables: bool,
|
||||
strip_emoji_glue: bool,
|
||||
) -> tuple[str, str, str | None]:
|
||||
"""Classify one input char for both inspect and clean.
|
||||
|
||||
Returns ``(action, out_char, kind)`` where action is ``keep``, ``strip``
|
||||
or ``replace``; out_char is the surviving character for keep/replace; and
|
||||
kind is the inspect classification (None when not suspicious).
|
||||
"""
|
||||
cp = ord(ch)
|
||||
if _is_emoji_glue(cp) and not strip_emoji_glue:
|
||||
if prev_kept is not None and _is_emoji_base(ord(prev_kept)):
|
||||
return ("keep", ch, None)
|
||||
if not strip_emoji_glue:
|
||||
if cp in _SCRIPT_JOINERS and prev_kept is not None and _is_joining_letter(ord(prev_kept)):
|
||||
return ("keep", ch, None)
|
||||
if cp in _TAG_RANGE and prev_kept is not None and _is_emoji_base(ord(prev_kept)):
|
||||
return ("keep", ch, None)
|
||||
if cp in _MONGOLIAN_FVS and prev_kept is not None and _is_mongolian_letter(ord(prev_kept)):
|
||||
return ("keep", ch, None)
|
||||
if cp in _KHMER_VOWELS and prev_kept is not None and _is_khmer_letter(ord(prev_kept)):
|
||||
return ("keep", ch, None)
|
||||
if cp in _HANGUL_FILLERS and prev_kept is not None and _is_hangul_jamo(ord(prev_kept)):
|
||||
return ("keep", ch, None)
|
||||
if cp in _ORTHOGRAPHIC_CF:
|
||||
return ("keep", ch, None)
|
||||
if _is_strip_cp(cp):
|
||||
return ("strip", "", _strip_kind(cp))
|
||||
if normalize_spaces and cp in SPACE_HOMOGLYPHS:
|
||||
return ("replace", SPACE_HOMOGLYPHS[cp], "space")
|
||||
if treat_confusables and cp in LATIN_CONFUSABLES:
|
||||
return ("replace", LATIN_CONFUSABLES[cp], "confusable")
|
||||
if unicodedata.category(ch) == "Cf" and cp not in SPACE_HOMOGLYPHS:
|
||||
return ("strip", "", "other_cf")
|
||||
return ("keep", ch, None)
|
||||
|
||||
|
||||
def _char_label(ch: str) -> str:
|
||||
cp = ord(ch)
|
||||
name = unicodedata.name(ch, "UNKNOWN")
|
||||
cat = unicodedata.category(ch)
|
||||
return f"U+{cp:04X} {name} ({cat})"
|
||||
|
||||
|
||||
def _hit_confidence(kind: str) -> str:
|
||||
"""Layer A hits are edit-based carriers; space homoglyphs are weaker context."""
|
||||
return "informational" if kind == "space" else "probable"
|
||||
|
||||
|
||||
@dataclass
|
||||
class CharHit:
|
||||
codepoint: int
|
||||
char: str
|
||||
label: str
|
||||
count: int
|
||||
kind: str # strip | bidi | tag_chars | variation_selector | zwj_family | private_use | space | confusable | other_cf
|
||||
samples: list[int] = field(default_factory=list) # character offsets
|
||||
|
||||
|
||||
@dataclass
|
||||
class TextInspectReport:
|
||||
length: int
|
||||
suspicious_total: int
|
||||
hits: list[CharHit]
|
||||
notes: list[str] = field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"length": self.length,
|
||||
"suspicious_total": self.suspicious_total,
|
||||
"hits": [
|
||||
{
|
||||
"codepoint": f"U+{h.codepoint:04X}",
|
||||
"label": h.label,
|
||||
"count": h.count,
|
||||
"kind": h.kind,
|
||||
"confidence": _hit_confidence(h.kind),
|
||||
"sample_offsets": h.samples[:10],
|
||||
}
|
||||
for h in self.hits
|
||||
],
|
||||
"notes": self.notes,
|
||||
}
|
||||
|
||||
|
||||
def inspect_text(
|
||||
text: str,
|
||||
*,
|
||||
aggressive: bool = False,
|
||||
strip_emoji_glue: bool = False,
|
||||
) -> TextInspectReport:
|
||||
buckets: dict[tuple[int, str], list[int]] = {}
|
||||
prev_kept: str | None = None
|
||||
for i, ch in enumerate(text):
|
||||
action, out_char, kind = _decide(
|
||||
ch,
|
||||
prev_kept,
|
||||
normalize_spaces=True,
|
||||
treat_confusables=aggressive,
|
||||
strip_emoji_glue=strip_emoji_glue,
|
||||
)
|
||||
if kind is None:
|
||||
# Kept; glue (emoji/script joiner/tag) does not advance the
|
||||
# "previous kept" base so ZWJ chains and flag runs stay bound.
|
||||
if not _is_glue(ord(ch)):
|
||||
prev_kept = out_char
|
||||
continue
|
||||
key = (ord(ch), kind)
|
||||
buckets.setdefault(key, []).append(i)
|
||||
if action == "replace":
|
||||
prev_kept = out_char
|
||||
# strip: prev_kept unchanged
|
||||
|
||||
hits: list[CharHit] = []
|
||||
total = 0
|
||||
for (cp, kind), offsets in sorted(buckets.items(), key=lambda x: (-len(x[1]), x[0][0])):
|
||||
ch = chr(cp)
|
||||
hits.append(
|
||||
CharHit(
|
||||
codepoint=cp,
|
||||
char=ch,
|
||||
label=_char_label(ch),
|
||||
count=len(offsets),
|
||||
kind=kind,
|
||||
samples=offsets[:10],
|
||||
)
|
||||
)
|
||||
total += len(offsets)
|
||||
|
||||
notes = [
|
||||
"Layer A only: invisible/format Unicode and space homoglyphs (edit-based carriers).",
|
||||
"Statistical (token-sampling) watermarks are not detectable here; use Layer B rewrite.",
|
||||
"Inspect kinds: strip, bidi, tag_chars, variation_selector, zwj_family, private_use, space, confusable, other_cf.",
|
||||
"Load-bearing invisibles are preserved by default: emoji glue (ZWJ/VS after an emoji base), script joiners (ZWNJ/ZWJ inside complex scripts), flag tag chars, same-script fillers/selectors (Mongolian FVS, Khmer inherent vowels, Hangul jamo fillers), and orthographic Arabic/Syriac Cf marks. Use --strip-emoji-glue for paranoid mode (strips them all).",
|
||||
]
|
||||
if not hits:
|
||||
notes.append(
|
||||
"No deterministic Layer A (invisible Unicode/format) carriers detected; "
|
||||
"statistical and pixel-domain marks are out of scope here."
|
||||
)
|
||||
return TextInspectReport(length=len(text), suspicious_total=total, hits=hits, notes=notes)
|
||||
|
||||
|
||||
def clean_text(
|
||||
text: str,
|
||||
*,
|
||||
nfkc: bool = False,
|
||||
aggressive_homoglyphs: bool = False,
|
||||
normalize_spaces: bool = True,
|
||||
strip_emoji_glue: bool = False,
|
||||
) -> tuple[str, dict]:
|
||||
"""Return cleaned text and a stats dict."""
|
||||
removed: Counter[str] = Counter()
|
||||
replaced: Counter[str] = Counter()
|
||||
out_chars: list[str] = []
|
||||
prev_kept: str | None = None
|
||||
|
||||
for ch in text:
|
||||
action, out_char, _kind = _decide(
|
||||
ch,
|
||||
prev_kept,
|
||||
normalize_spaces=normalize_spaces,
|
||||
treat_confusables=aggressive_homoglyphs,
|
||||
strip_emoji_glue=strip_emoji_glue,
|
||||
)
|
||||
if action == "keep":
|
||||
out_chars.append(out_char)
|
||||
# Glue (emoji/script joiner/tag) does not advance the "previous
|
||||
# kept" base, so ZWJ chains (❤️🔥) and flag runs stay bound.
|
||||
if not _is_glue(ord(ch)):
|
||||
prev_kept = out_char
|
||||
elif action == "replace":
|
||||
out_chars.append(out_char)
|
||||
replaced[_char_label(ch)] += 1
|
||||
prev_kept = out_char
|
||||
else: # strip
|
||||
removed[_char_label(ch)] += 1
|
||||
# prev_kept unchanged
|
||||
|
||||
result = "".join(out_chars)
|
||||
if nfkc:
|
||||
before = result
|
||||
result = unicodedata.normalize("NFKC", result)
|
||||
if result != before:
|
||||
replaced["NFKC_normalize"] += abs(len(before) - len(result)) or 1
|
||||
|
||||
# Collapse runs of spaces only if we introduced space replacements? Keep conservative: no.
|
||||
|
||||
stats = {
|
||||
"input_length": len(text),
|
||||
"output_length": len(result),
|
||||
"removed": dict(removed),
|
||||
"replaced": dict(replaced),
|
||||
"removed_count": sum(removed.values()),
|
||||
"replaced_count": sum(v for k, v in replaced.items() if k != "NFKC_normalize"),
|
||||
}
|
||||
return result, stats
|
||||
|
||||
|
||||
def human_report(report: TextInspectReport) -> str:
|
||||
lines = [
|
||||
f"Length: {report.length} chars",
|
||||
f"Suspicious: {report.suspicious_total}",
|
||||
]
|
||||
if report.hits:
|
||||
lines.append("Hits:")
|
||||
for h in report.hits:
|
||||
lines.append(
|
||||
f" [{h.kind}/{_hit_confidence(h.kind)}] "
|
||||
f"{h.label} x{h.count} @ {h.samples[:5]}"
|
||||
)
|
||||
for n in report.notes:
|
||||
lines.append(f"Note: {n}")
|
||||
return "\n".join(lines)
|
||||
@@ -56,10 +56,10 @@ curl -s "$WM/capabilities"
|
||||
```
|
||||
|
||||
Reports which optional tools are available server-side (`c2patool`, `exiftool`,
|
||||
`qpdf`) and which heavy backends are configured (`pixel_backends.ctrlregen`,
|
||||
`pixel_backends.diffusion`, `scorers.synthid`, `harnesses.markllm`). **Drive
|
||||
your advice from this**: only recommend pixel removal / SynthID scoring when
|
||||
the service reports the backend present.
|
||||
`qpdf`), scorers present (`scorers.stylometry`, `scorers.synthid`), and which heavy
|
||||
backends are configured (`pixel_backends.ctrlregen`, `pixel_backends.diffusion`,
|
||||
`harnesses.markllm`). **Drive your advice from this**: only recommend pixel
|
||||
removal / SynthID scoring when the service reports the backend present.
|
||||
|
||||
## HTTP API (curl)
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
| Target | Method | Script / action | Side effects | Verifiable today? |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Invisible Unicode / exotic spaces / bidi / tags | Strip / normalize | `inspect_text.py`, `clean_text.py`, `clean_file.py` | Minimal | Yes (codepoint report) |
|
||||
| Stylometric AI cadence / burstiness / n-grams (zero-LLM) | Statistical variance & cadence scoring | `score_stylometry.py`, `inspect_text.py --stylometry`, `audit_dir.py --check-stylometry` | None (detection only) | Yes (calibrated score + phrase spans) |
|
||||
| Statistical text watermark (SynthID-class / Kirchenbauer) | Multi-pass paraphrase / humanize / back-translate / structural | Agent Layer B + optional `rewrite_text.py` | Meaning/style drift | No without vendor key/detector; **MarkLLM harness** (`detect_text_watermark.py`) verifies a specific scheme config before/after |
|
||||
| C2PA on PNG/JPEG/WebP | Drop APP11 / PNG `caBX` / RIFF `C2PA` / exiftool | `clean_image.py` | Loses provenance metadata | Yes |
|
||||
| SVG metadata / XMP | Drop `<metadata>`, xmpmeta | `clean_file.py` | Loses SVG metadata | Yes (re-inspect) |
|
||||
|
||||
+1
@@ -0,0 +1 @@
|
||||
In today's fast-paced digital world, artificial intelligence plays a crucial role in modern technological advancement. It is important to note that machine learning models delve into complex datasets to discover meaningful patterns. Furthermore, the architecture seamlessly integrates various algorithmic layers to optimize system performance. This technological breakthrough stands as a testament to human ingenuity and continuous innovation. Moreover, the multifaceted approach enables organizations to foster a culture of data-driven decision making. The rich tapestry of neural network topologies underscores the importance of scalable compute infrastructure. In conclusion, navigating the complexities of modern engineering requires a holistic perspective. I hope this helps!
|
||||
+1
@@ -0,0 +1 @@
|
||||
We wrote this tool on a Saturday night. The original python script was messy, full of hacky loops, and crashed on every third test run because of a missing null check. I fixed it with twenty lines of code and a cup of black coffee. Why does this matter? Because tools should just work when you run them in terminal, without breaking your workflow or failing silently. We tested the binary parser against five corrupt headers yesterday. Nothing broke. If you run into weird edge cases with weird unicode fonts, let us know on GitHub and we will patch it.
|
||||
@@ -108,6 +108,17 @@ def test_scan_file_text_and_html(tmp_path: Path):
|
||||
assert not is_actionable(item)
|
||||
|
||||
|
||||
def test_scan_file_container_layer_a_reported_once(tmp_path: Path):
|
||||
"""Layer A body findings come from inspect_container() exactly once."""
|
||||
md = tmp_path / "post.md"
|
||||
md.write_text("# Title\n\nHello\u200bWorld\n", encoding="utf-8")
|
||||
item = scan_file(md)
|
||||
layer_a = [f for f in item["findings"] if "layer-a" in f]
|
||||
assert len(layer_a) == 1
|
||||
assert item["suspicious_total"] == 1
|
||||
assert is_actionable(item)
|
||||
|
||||
|
||||
def test_aggregate_summary():
|
||||
files = [
|
||||
{
|
||||
|
||||
@@ -52,6 +52,32 @@ def test_inspect_bidi():
|
||||
assert "\u202e" not in cleaned
|
||||
|
||||
|
||||
def test_preserves_legitimate_bidi_marks_and_isolates_by_default():
|
||||
raw = "السعر \u2066123 USD\u2069\u200f"
|
||||
report = inspect_text(raw)
|
||||
assert any(h.kind == "bidi" for h in report.hits)
|
||||
assert clean_text(raw)[0] == raw
|
||||
assert clean_text(raw, strip_bidi=True)[0] == "السعر 123 USD"
|
||||
|
||||
|
||||
def test_preserves_legacy_bidi_embeddings_by_default():
|
||||
raw = "English \u202bالعربية\u202c end"
|
||||
assert clean_text(raw)[0] == raw
|
||||
assert clean_text(raw, strip_bidi=True)[0] == "English العربية end"
|
||||
|
||||
|
||||
def test_strips_override_and_its_pdf_terminator():
|
||||
raw = "abc\u202edef\u202c"
|
||||
cleaned, stats = clean_text(raw)
|
||||
assert cleaned == "abcdef"
|
||||
assert stats["removed_count"] == 2
|
||||
|
||||
|
||||
def test_strips_orphaned_bidi_embedding_controls():
|
||||
assert clean_text("abc\u202c")[0] == "abc"
|
||||
assert clean_text("abc\u202bdef")[0] == "abcdef"
|
||||
|
||||
|
||||
def test_clean_preserves_normal_text():
|
||||
raw = "Normal ASCII and café — fine."
|
||||
cleaned, stats = clean_text(raw)
|
||||
@@ -73,6 +99,34 @@ def test_clean_preserves_emoji_vs16():
|
||||
assert stats["removed_count"] == 0
|
||||
|
||||
|
||||
def test_clean_preserves_arrow_emoji_vs16():
|
||||
raw = "Move \u2194\ufe0f"
|
||||
cleaned, stats = clean_text(raw)
|
||||
assert cleaned == raw
|
||||
assert stats["removed_count"] == 0
|
||||
|
||||
|
||||
def test_clean_preserves_cjk_ideographic_variation_selector():
|
||||
raw = "\u845b\U000E0100" # CJK ideograph + VS17
|
||||
cleaned, stats = clean_text(raw)
|
||||
assert cleaned == raw
|
||||
assert stats["removed_count"] == 0
|
||||
|
||||
|
||||
def test_clean_strips_repeated_cjk_variation_selector():
|
||||
raw = "\u845b\U000E0100\U000E0101"
|
||||
cleaned, stats = clean_text(raw)
|
||||
assert cleaned == "\u845b\U000E0100"
|
||||
assert stats["removed_count"] == 1
|
||||
|
||||
|
||||
def test_clean_preserves_mongolian_variation_selector():
|
||||
raw = "\u1820\u180b"
|
||||
cleaned, stats = clean_text(raw)
|
||||
assert cleaned == raw
|
||||
assert stats["removed_count"] == 0
|
||||
|
||||
|
||||
def test_clean_preserves_zwj_family():
|
||||
raw = "Family time: \U0001F468\u200D\U0001F469\u200D\U0001F467" # 👨👩👧
|
||||
cleaned, stats = clean_text(raw)
|
||||
@@ -133,6 +187,20 @@ def test_clean_preserves_flag_tag_sequence():
|
||||
assert cleaned == raw
|
||||
|
||||
|
||||
def test_clean_strips_incomplete_flag_tag_sequence():
|
||||
raw = "\U0001F3F4\U000E0067\U000E0062"
|
||||
cleaned, stats = clean_text(raw)
|
||||
assert cleaned == "\U0001F3F4"
|
||||
assert stats["removed_count"] == 2
|
||||
|
||||
|
||||
def test_clean_strips_joiner_between_unrelated_scripts():
|
||||
raw = "\u845b\u200cA"
|
||||
cleaned, stats = clean_text(raw)
|
||||
assert cleaned == "\u845bA"
|
||||
assert stats["removed_count"] == 1
|
||||
|
||||
|
||||
def test_clean_preserves_orthographic_arabic_cf():
|
||||
raw = "x\u0600y\u06ddz" # ARABIC NUMBER SIGN, END OF AYAH
|
||||
cleaned, _ = clean_text(raw)
|
||||
@@ -152,6 +220,27 @@ def test_strip_emoji_glue_flag_restores_blanket_strip():
|
||||
assert clean_text("x\u0600y", strip_emoji_glue=True)[0] == "xy"
|
||||
|
||||
|
||||
def test_nfkc_change_is_in_replacement_count():
|
||||
cleaned, stats = clean_text("Ａ", nfkc=True)
|
||||
assert cleaned == "A"
|
||||
assert stats["nfkc_changed"] is True
|
||||
assert stats["replaced_count"] == 1
|
||||
|
||||
|
||||
def test_nfkc_counts_changed_input_codepoints():
|
||||
cleaned, stats = clean_text("ＡＢ ffi", nfkc=True)
|
||||
assert cleaned == "AB ffi"
|
||||
assert stats["replaced"]["NFKC_normalize"] == 3
|
||||
assert stats["replaced_count"] == 3
|
||||
|
||||
|
||||
def test_nfkc_counts_contextual_composition_input_codepoints():
|
||||
raw = "A\u030a A\u030a"
|
||||
cleaned, stats = clean_text(raw, nfkc=True)
|
||||
assert cleaned == "\u00c5 \u00c5"
|
||||
assert stats["replaced"]["NFKC_normalize"] == 4
|
||||
|
||||
|
||||
def test_clean_preserves_mongolian_fvs():
|
||||
# Mongolian letter + FVS1/2/3 selects a positional glyph variant.
|
||||
for raw in ("\u1820\u180b\u1821", "\u1820\u180c\u1821", "\u1820\u180d\u1821"):
|
||||
|
||||
@@ -383,6 +383,43 @@ def test_clean_container_markdown_file(tmp_path: Path):
|
||||
assert result["format"] == "markdown"
|
||||
|
||||
|
||||
def test_inspect_container_reports_layer_a_body_text(tmp_path: Path):
|
||||
"""inspect must flag invisible carriers that clean would strip.
|
||||
|
||||
Regression: markdown/html routed to the container inspector, which never
|
||||
ran the Layer A scan, so identical bytes were reported suspicious as .txt
|
||||
and clean as .md while clean_container() went on to remove them.
|
||||
"""
|
||||
body = "Helloworld testend.\n"
|
||||
for name in ("x.md", "x.html"):
|
||||
src = tmp_path / name
|
||||
src.write_text(body, encoding="utf-8")
|
||||
report = inspect_container(src)
|
||||
assert report.layer_a_total == 3, name
|
||||
assert report.to_dict()["suspicious_total"] == 3, name
|
||||
codepoints = {h["codepoint"] for h in report.layer_a_hits}
|
||||
assert {"U+200B", "U+200C", "U+2060"} <= codepoints, name
|
||||
assert any(f.startswith("layer-a:") for f in report.findings), name
|
||||
|
||||
|
||||
def test_inspect_container_clean_leaves_no_layer_a(tmp_path: Path):
|
||||
"""The post-clean re-inspect must come back with nothing left."""
|
||||
src = tmp_path / "x.md"
|
||||
src.write_text("Hithere\n", encoding="utf-8")
|
||||
dest = tmp_path / "x.cleaned.md"
|
||||
clean_container(src, dest)
|
||||
assert inspect_container(dest).layer_a_total == 0
|
||||
|
||||
|
||||
def test_inspect_container_clean_file_stays_clean(tmp_path: Path):
|
||||
"""No false positives on ordinary prose."""
|
||||
src = tmp_path / "x.md"
|
||||
src.write_text("# Title\n\nOrdinary prose, nothing hidden.\n", encoding="utf-8")
|
||||
report = inspect_container(src)
|
||||
assert report.layer_a_total == 0
|
||||
assert report.layer_a_hits == []
|
||||
|
||||
|
||||
def test_inspect_container_svg(tmp_path: Path):
|
||||
src = tmp_path / "a.svg"
|
||||
src.write_bytes(
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
"""Smoke tests for the standalone Cursor text skill."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SKILL = ROOT / "skills" / "clean-user-facing-text"
|
||||
|
||||
|
||||
def test_lightweight_clean_text_cli():
|
||||
result = subprocess.run(
|
||||
[sys.executable, str(SKILL / "scripts" / "clean_text.py"), "-", "--stats"],
|
||||
input="Hello\u200b world\u3000again",
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
capture_output=True,
|
||||
check=True,
|
||||
)
|
||||
|
||||
assert result.stdout.rstrip("\n") == "Hello world again"
|
||||
assert '"removed_count": 1' in result.stderr
|
||||
assert '"replaced_count": 1' in result.stderr
|
||||
|
||||
|
||||
def test_lightweight_skill_has_no_template_placeholders():
|
||||
skill_text = (SKILL / "SKILL.md").read_text(encoding="utf-8")
|
||||
|
||||
assert "TODO" not in skill_text
|
||||
assert (SKILL / "references" / "watermark-notes.md").is_file()
|
||||
|
||||
|
||||
def _run_installer(home: Path, *args: str, check: bool = True):
|
||||
env = os.environ.copy()
|
||||
env.pop("CURSOR_HOME", None)
|
||||
env.update({"HOME": str(home), "USERPROFILE": str(home)})
|
||||
return subprocess.run(
|
||||
[sys.executable, str(ROOT / "install_skill.py"), *args],
|
||||
env=env,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=check,
|
||||
)
|
||||
|
||||
|
||||
def test_installer_uses_cursor_skill_location(tmp_path):
|
||||
_run_installer(tmp_path)
|
||||
|
||||
assert (tmp_path / ".cursor" / "skills" / SKILL.name / "SKILL.md").is_file()
|
||||
|
||||
|
||||
def test_installer_preserves_existing_install_without_force(tmp_path):
|
||||
cursor_install = tmp_path / ".cursor" / "skills" / SKILL.name
|
||||
cursor_install.mkdir(parents=True)
|
||||
(cursor_install / "sentinel").write_text("keep", encoding="utf-8")
|
||||
|
||||
result = _run_installer(tmp_path, check=False)
|
||||
|
||||
assert result.returncode == 1
|
||||
assert (cursor_install / "sentinel").read_text(encoding="utf-8") == "keep"
|
||||
|
||||
|
||||
def test_installer_force_creates_backup_and_replaces(tmp_path):
|
||||
destination = tmp_path / ".cursor" / "skills" / SKILL.name
|
||||
destination.mkdir(parents=True)
|
||||
(destination / "old").write_text("old", encoding="utf-8")
|
||||
|
||||
_run_installer(tmp_path, "--force")
|
||||
|
||||
backups = list(destination.parent.glob(f"{SKILL.name}.backup.*"))
|
||||
assert len(backups) == 1
|
||||
assert (backups[0] / "old").read_text(encoding="utf-8") == "old"
|
||||
assert (destination / "SKILL.md").is_file()
|
||||
@@ -0,0 +1,177 @@
|
||||
"""Tests for zero-LLM statistical & stylometric AI-text detector."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
SCRIPTS_DIR = Path(__file__).resolve().parent.parent / "service" / "scripts"
|
||||
FIXTURES_DIR = Path(__file__).resolve().parent / "fixtures"
|
||||
sys.path.insert(0, str(SCRIPTS_DIR))
|
||||
|
||||
from audit_lib import scan_file # noqa: E402
|
||||
from score_stylometry import ( # noqa: E402
|
||||
compute_burstiness,
|
||||
compute_mattr,
|
||||
extract_sentences,
|
||||
extract_words,
|
||||
scan_ai_phrases,
|
||||
score_text_stylometry,
|
||||
)
|
||||
from server import capabilities # noqa: E402
|
||||
|
||||
|
||||
def test_sentence_and_word_extraction():
|
||||
sample = "Hello world! This is a test. Here is a code block:\n```python\nprint('hi')\n```\nDone."
|
||||
sentences = extract_sentences(sample)
|
||||
assert len(sentences) == 4
|
||||
assert sentences[0] == "Hello world!"
|
||||
assert sentences[1] == "This is a test."
|
||||
assert sentences[2] == "Here is a code block:"
|
||||
assert sentences[3] == "Done."
|
||||
|
||||
words = extract_words("Hello, World! Testing 1-2-3.")
|
||||
assert "hello" in words
|
||||
assert "world" in words
|
||||
assert "testing" in words
|
||||
|
||||
|
||||
def test_burstiness_variance():
|
||||
# Empty / single sentence edge cases
|
||||
assert compute_burstiness([]) == (0.0, 0.0, 0.0)
|
||||
assert compute_burstiness(["Only one sentence here."])[1] == 0.0
|
||||
|
||||
# Uniform sentences (every sentence is 5 words) -> CV should be 0.0
|
||||
uniform = ["One two three four five.", "Six seven eight nine ten.", "Alpha beta gamma delta epsilon."]
|
||||
mean, std, cv = compute_burstiness(uniform)
|
||||
assert mean == 5.0
|
||||
assert std == 0.0
|
||||
assert cv == 0.0
|
||||
|
||||
# Bursty sentences (highly varied lengths: 2 words vs 20 words) -> CV should be high (>0.7)
|
||||
bursty = [
|
||||
"Too short.",
|
||||
"This is an extraordinarily long and verbose sentence crafted intentionally to introduce high variance into the token distribution model.",
|
||||
"Short again.",
|
||||
]
|
||||
_, _, cv_bursty = compute_burstiness(bursty)
|
||||
assert cv_bursty > 0.70
|
||||
|
||||
|
||||
def test_mattr_lexical_diversity():
|
||||
words = ["apple"] * 100
|
||||
# Completely repetitive words -> MATTR should be 1/50 = 0.02
|
||||
mattr_rep = compute_mattr(words, window_size=50)
|
||||
assert mattr_rep < 0.05
|
||||
|
||||
# Distinct vocabulary
|
||||
distinct = [f"word_{i}" for i in range(100)]
|
||||
mattr_dist = compute_mattr(distinct, window_size=50)
|
||||
assert mattr_dist == 1.0
|
||||
|
||||
|
||||
def test_ai_phrase_scanner():
|
||||
text = "In today's fast-paced digital world, we must delve into this problem. It is crucial to note that this is a testament to progress."
|
||||
matches = scan_ai_phrases(text)
|
||||
matched_labels = [m.phrase for m in matches]
|
||||
assert "in today's fast-paced world/landscape" in matched_labels
|
||||
assert "delve into" in matched_labels
|
||||
assert "it is important/crucial to note" in matched_labels
|
||||
assert "a testament to" in matched_labels
|
||||
|
||||
|
||||
def test_short_text_floor():
|
||||
short_text = "This is a very short text with only nine words."
|
||||
report = score_text_stylometry(short_text)
|
||||
assert report.status == "insufficient_length"
|
||||
assert report.score == 0.0
|
||||
assert report.confidence_level == "CLEAN"
|
||||
assert any("uncalibrated" in n for n in report.notes)
|
||||
|
||||
|
||||
def test_ai_vs_human_discrimination():
|
||||
ai_sample = (FIXTURES_DIR / "stylometry_ai_sample.txt").read_text(encoding="utf-8")
|
||||
human_sample = (FIXTURES_DIR / "stylometry_human_sample.txt").read_text(encoding="utf-8")
|
||||
|
||||
ai_report = score_text_stylometry(ai_sample, path="ai_sample.txt")
|
||||
human_report = score_text_stylometry(human_sample, path="human_sample.txt")
|
||||
|
||||
assert ai_report.status == "ok"
|
||||
assert human_report.status == "ok"
|
||||
|
||||
# AI sample should have high score and HIGH/MEDIUM confidence
|
||||
assert ai_report.score >= 0.70
|
||||
assert ai_report.confidence_level in ("HIGH", "MEDIUM")
|
||||
assert len(ai_report.matched_markers) >= 3
|
||||
|
||||
# Human sample should have low score and CLEAN/LOW confidence
|
||||
assert human_report.score < 0.35
|
||||
assert human_report.confidence_level in ("CLEAN", "LOW")
|
||||
|
||||
|
||||
def test_score_stylometry_cli():
|
||||
ai_path = FIXTURES_DIR / "stylometry_ai_sample.txt"
|
||||
human_path = FIXTURES_DIR / "stylometry_human_sample.txt"
|
||||
|
||||
script = SCRIPTS_DIR / "score_stylometry.py"
|
||||
|
||||
# AI text should trigger exit code 1 (score >= 0.65)
|
||||
res_ai = subprocess.run(
|
||||
[sys.executable, str(script), str(ai_path), "--json"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
assert res_ai.returncode == 1
|
||||
data_ai = json.loads(res_ai.stdout)
|
||||
assert data_ai["score"] >= 0.65
|
||||
assert data_ai["status"] == "ok"
|
||||
|
||||
# Human text should exit 0 (score < 0.65)
|
||||
res_human = subprocess.run(
|
||||
[sys.executable, str(script), str(human_path), "--json"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
assert res_human.returncode == 0
|
||||
data_human = json.loads(res_human.stdout)
|
||||
assert data_human["score"] < 0.65
|
||||
|
||||
# Custom threshold override
|
||||
res_thresh = subprocess.run(
|
||||
[sys.executable, str(script), str(ai_path), "--threshold", "0.99"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
assert res_thresh.returncode == 0
|
||||
|
||||
|
||||
def test_inspect_text_stylometry_flag():
|
||||
ai_path = FIXTURES_DIR / "stylometry_ai_sample.txt"
|
||||
script = SCRIPTS_DIR / "inspect_text.py"
|
||||
|
||||
res = subprocess.run(
|
||||
[sys.executable, str(script), str(ai_path), "--stylometry", "--json"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
assert res.returncode == 1
|
||||
data = json.loads(res.stdout)
|
||||
assert "stylometry" in data
|
||||
assert data["stylometry"]["score"] >= 0.65
|
||||
|
||||
|
||||
def test_audit_lib_check_stylometry():
|
||||
ai_path = FIXTURES_DIR / "stylometry_ai_sample.txt"
|
||||
item = scan_file(ai_path, check_stylometry=True)
|
||||
assert "stylometry" in item
|
||||
assert item["stylometry"]["score"] >= 0.65
|
||||
assert any("stylometry" in f for f in item["findings"])
|
||||
|
||||
|
||||
def test_server_capabilities_includes_stylometry():
|
||||
caps = capabilities()
|
||||
assert caps["scorers"]["stylometry"] is True
|
||||
@@ -0,0 +1,72 @@
|
||||
"""score_synthid.py --json must emit pure JSON on stdout.
|
||||
|
||||
The reverse-SynthID upstream prints progress ("CodebookV4 loaded: ...")
|
||||
straight to stdout. image_meta.py json.loads the scorer's stdout, so any
|
||||
such leak corrupts the score payload. This test drives the real script as
|
||||
a subprocess against a stub upstream whose classes are deliberately noisy,
|
||||
and fails if a single non-JSON byte reaches stdout.
|
||||
|
||||
The stub extraction dir also ships a fake ``cv2`` module: the script does
|
||||
``sys.path.insert(0, extraction)`` before ``import cv2``, so the test runs
|
||||
without OpenCV installed.
|
||||
"""
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPT = Path(__file__).resolve().parents[1] / "service" / "scripts" / "score_synthid.py"
|
||||
|
||||
|
||||
def _write_stub_upstream(root: Path) -> Path:
|
||||
ext = root / "src" / "extraction"
|
||||
ext.mkdir(parents=True)
|
||||
(root / "artifacts").mkdir()
|
||||
(root / "artifacts" / "spectral_codebook_v4.npz").touch()
|
||||
|
||||
(ext / "cv2.py").write_text(
|
||||
"COLOR_BGR2RGB = 4\n"
|
||||
"def imread(path):\n"
|
||||
" return object()\n"
|
||||
"def cvtColor(img, code):\n"
|
||||
" return img\n"
|
||||
)
|
||||
(ext / "synthid_bypass_v4.py").write_text(
|
||||
"class SpectralCodebookV4:\n"
|
||||
" def load(self, path):\n"
|
||||
" print(f'CodebookV4 loaded: {path}')\n" # the noisy print
|
||||
)
|
||||
(ext / "robust_extractor.py").write_text(
|
||||
"from types import SimpleNamespace\n"
|
||||
"class RobustSynthIDExtractor:\n"
|
||||
" def detect_from_v4_codebook(self, rgb, codebook, model=None):\n"
|
||||
" print('extractor progress chatter')\n" # more stdout noise
|
||||
" return SimpleNamespace(\n"
|
||||
" details={'profile_key': 'stub', 'exact_match': False,\n"
|
||||
" 'per_channel_scores': [0.1], 'per_channel_n': [1]},\n"
|
||||
" is_watermarked=False, confidence=0.42,\n"
|
||||
" phase_match=0.0, multi_scale_consistency=0.0,\n"
|
||||
" )\n"
|
||||
)
|
||||
return root
|
||||
|
||||
|
||||
def test_json_stdout_survives_noisy_upstream(tmp_path):
|
||||
upstream = _write_stub_upstream(tmp_path / "upstream")
|
||||
img = tmp_path / "img.png"
|
||||
img.write_bytes(b"\x89PNG\r\n\x1a\nstub")
|
||||
|
||||
r = subprocess.run(
|
||||
[sys.executable, str(SCRIPT), str(img),
|
||||
"--upstream-dir", str(upstream), "--json"],
|
||||
capture_output=True, text=True, timeout=60,
|
||||
)
|
||||
|
||||
assert r.returncode == 0, r.stderr
|
||||
payload = json.loads(r.stdout) # corrupt stdout fails here
|
||||
assert payload["available"] is True
|
||||
assert payload["confidence"] == 0.42
|
||||
# the noise must land on stderr, not vanish
|
||||
assert "CodebookV4 loaded" in r.stderr
|
||||
assert "extractor progress chatter" in r.stderr
|
||||
Reference in New Issue
Block a user