47252419e4
* fix: strip three missed Default_Ignorable invisible carriers in Layer A U+180F (Mongolian free variation selector-4, added in Unicode 14), U+3164 (Hangul filler), and U+FFA0 (halfwidth Hangul filler) are blank-rendering Default_Ignorable code points, but their Mn/Lo categories meant the Cf catch-all never saw them and they were absent from STRIP_CODEPOINTS. Both inspect_text and clean_text therefore passed them through untouched, even between plain ASCII. Add them to the strip set and wire U+180F into the Mongolian-FVS handling so it is stripped when floating but preserved after a Mongolian letter, exactly like FVS1-3. Replace the 0x180B..0x180D range with the named _MONGOLIAN_FVS set (clearer, and avoids the differently-handled Mongolian vowel separator at U+180E). Applied to both the service engine and the vendored lightweight-skill copy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HV9AE8QLqiXf7QnRe891EY * feat: strip reserved Default_Ignorable code points in Layer A U+2065, U+FFF0..U+FFF8, U+E0000, U+E0080..U+E00FF, and U+E01F0..U+E0FFF are unassigned code points carrying Other_Default_Ignorable_Code_Point, reserved for future default-ignorable characters. Conformant renderers display them invisibly, normalisation preserves them, and the Cf catch-all never sees them (they are category Cn), which made them ideal covert carriers that both inspect_text and clean_text passed through untouched. Add them to the strip set as explicit ranges and report them under the new "reserved_ignorable" inspect kind. Deliberately not a category-Cn rule: unicodedata is pinned per Python build, so a Cn rule would destroy characters assigned in newer Unicode versions. The table carries a reminder to re-check the ranges on Unicode version bumps, since assignment turns a strip entry into a potential preserve-in-context case, exactly as happened when U+180F became Mongolian FVS4 in Unicode 14. Tests sweep all 3,739 code points through both the service engine and the vendored lightweight-skill copy (clean and inspect CLIs), with a boundary test pinning the assigned neighbours (U+2064, U+FFF9, U+E0001, U+E0100) to their existing kinds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: strip Unicode noncharacters in Layer A The 66 noncharacters (U+FDD0..U+FDEF plus U+nFFFE/U+nFFFF at the end of every plane) are permanently reserved for internal use and prohibited in interchange text (TUS 23.7). They render as nothing or tofu, survive normalisation and Python round-trips, and their category (Cn) meant the Cf catch-all never saw them, so both inspect_text and clean_text passed them through untouched: a ready-made covert channel. Strip them and report them under the new "noncharacter" inspect kind. Unlike the reserved Default_Ignorable ranges, noncharacters can never be assigned, so this carries no future-Unicode risk. Applied to both the service engine and the vendored lightweight-skill copy; tests sweep all 66 code points through both, and pin the assigned neighbours U+FDF0 and U+FFFD (replacement character) as untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: preserve visible-layout format controls next to their own script Egyptian hieroglyph quadrat controls (U+13430..U+1343F), Duployan shorthand controls (U+1BCA0..U+1BCA3), and musical beam/tie/slur/phrase controls (U+1D173..U+1D17A) are category Cf, so the catch-all stripped them, yet they visibly govern how their script renders (quadrat stacking, shorthand overlaps, beaming): removing them changes the rendered text, contradicting the "cleaners preserve the document body" invariant. Preserve them when adjacent to their own script, exactly like the existing Mongolian/Khmer/Hangul handling; floating between unrelated text they stay stripped and flagged, and --strip-emoji-glue paranoid mode still strips them everywhere. The vendored lightweight-skill engine needed next_input threaded through its _decide to express "preserve a begin-control from the character after it"; both engines now behave identically here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: preserve compatibility/halfwidth Hangul fillers in Hangul context Review feedback on the Layer A cluster: U+3164 and U+FFA0 were stripped unconditionally while the changelog claimed they were handled "like U+115F/U+1160", which are preserved after a Hangul jamo. Make the behaviour match the claim: both new fillers join _HANGUL_FILLERS and _is_hangul_jamo now covers the compatibility (U+3131-U+318E) and halfwidth (U+FFA1-U+FFDC) presentation forms, so each filler survives after a letter of its own form and remains contraband when floating between unrelated text. Changelog reworded to state the rule precisely. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
3.9 KiB
Mark classes
1. Edit-based text (Unicode / rules)
Invisible or near-invisible characters, exotic spaces, bidi controls, tag characters, synonym tables.
| Inspect kinds (Layer A) | Examples |
|---|---|
zwj_family |
ZWSP, ZWNJ, ZWJ, WJ, BOM |
bidi |
LRE/RLO/LRI/… |
tag_chars |
U+E0001–U+E007F |
variation_selector |
VS1–VS256 |
private_use |
U+E000–F8FF, U+F0000–FFFFD, U+100000–10FFFD |
noncharacter |
U+FDD0–FDEF, U+FFFE/U+FFFF at the end of every plane |
reserved_ignorable |
U+2065, U+FFF0–FFF8, U+E0000, U+E0080–E00FF, U+E01F0–E0FFF (unassigned Default_Ignorable) |
space |
NBSP, em space, ideographic space |
confusable |
Cyrillic/fullwidth Latin (aggressive) |
Removal: clean_text.py / Layer A — deterministic, verifiable.
Load-bearing invisibles are preserved by default so real text is not corrupted: emoji glue (ZWJ/VS after an emoji base), script joiners (ZWNJ/ZWJ inside complex scripts like Persian or Devanagari), flag tag-char sequences, same-script fillers/selectors (Mongolian free variation selectors after a Mongolian letter, Khmer inherent vowels after a Khmer consonant, Hangul jamo fillers in a partial syllable), orthographic Arabic/Syriac Cf marks, and visible-layout format controls next to their own script (Egyptian hieroglyph quadrat controls U+13430–U+1343F, Duployan shorthand controls U+1BCA0–U+1BCA3, musical beam/tie/slur/phrase controls U+1D173–U+1D17A). The same characters between plain ASCII stay carriers and are still stripped. Use --strip-emoji-glue for paranoid mode (strips all of them).
Maps to Nature paper “edit-based watermarking.”
2. Generative / statistical text (token sampling)
Bias next-token sampling toward a pseudo-random green list / score (Kirchenbauer, SynthID-Text / Tournament sampling, etc.). Signal lives in word choice, not metadata.
Removal: Layer B rewrite (paraphrase → back-translate → structural). Best-effort; no gold cert without vendor detector/key.
Maps to Nature paper primary method (SynthID-Text).
3. Data-driven / backdoor
Model trained or fine-tuned so trigger prompts produce marked or identifiable behavior.
Out of scope for this skill (model-side).
4. File provenance metadata (C2PA / EXIF / XMP / props)
Signed Content Credentials and AI generator tags in containers (hard-bound to the file: JUMBF/APP11, PNG chunks, XMP packets, OOXML props, etc.).
Industry framing (C2PA + SynthID two-layer model; see Institute of AI PM guide in README references):
| Layer | Mechanism | Survives metadata strip? | This project |
|---|---|---|---|
| Hard-bound C2PA | Signed manifest in the file | No — strip/re-encode drops it | In scope — clean_file / clean_image |
| Soft binding | Imperceptible watermark in content that can resolve to a remote manifest | Yes (by design) | Out of scope — pixel/audio/video signal |
| Standalone SynthID-class | Pixel / waveform / token watermark without needing C2PA | Yes for media; text is weaker | Media OOS; text → Layer B best-effort |
| Format | Support |
|---|---|
| PNG / JPEG / WebP | Full strip (stdlib + optional exiftool) |
| SVG | Drop metadata/XMP blocks |
| Prefer exiftool; degraded stdlib XMP strip | |
| DOCX / ODT | Scrub zip XML props / customXml |
| HTML | Meta generator / JSON-LD / data-ai* |
| Markdown | YAML frontmatter AI keys |
Removal: clean_file.py / clean_image.py — usually verifiable by re-inspect.
Honest report: after a successful C2PA strip, soft-bound / pixel SynthID (if the generator used them) may still be detectable by vendor tools (e.g. SynthID Detector, Content Credentials verify sites).
5. Pixel-domain image (and audio/video) watermarks
Invisible media marks (e.g. SynthID for images/audio/video) and C2PA soft binding that lives in the signal, not the metadata. Out of scope.