mirror of
https://github.com/guillaumemeyer/watermarks-remover.git
synced 2026-08-22 13:11:57 +02:00
47252419e4
* fix: strip three missed Default_Ignorable invisible carriers in Layer A U+180F (Mongolian free variation selector-4, added in Unicode 14), U+3164 (Hangul filler), and U+FFA0 (halfwidth Hangul filler) are blank-rendering Default_Ignorable code points, but their Mn/Lo categories meant the Cf catch-all never saw them and they were absent from STRIP_CODEPOINTS. Both inspect_text and clean_text therefore passed them through untouched, even between plain ASCII. Add them to the strip set and wire U+180F into the Mongolian-FVS handling so it is stripped when floating but preserved after a Mongolian letter, exactly like FVS1-3. Replace the 0x180B..0x180D range with the named _MONGOLIAN_FVS set (clearer, and avoids the differently-handled Mongolian vowel separator at U+180E). Applied to both the service engine and the vendored lightweight-skill copy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HV9AE8QLqiXf7QnRe891EY * feat: strip reserved Default_Ignorable code points in Layer A U+2065, U+FFF0..U+FFF8, U+E0000, U+E0080..U+E00FF, and U+E01F0..U+E0FFF are unassigned code points carrying Other_Default_Ignorable_Code_Point, reserved for future default-ignorable characters. Conformant renderers display them invisibly, normalisation preserves them, and the Cf catch-all never sees them (they are category Cn), which made them ideal covert carriers that both inspect_text and clean_text passed through untouched. Add them to the strip set as explicit ranges and report them under the new "reserved_ignorable" inspect kind. Deliberately not a category-Cn rule: unicodedata is pinned per Python build, so a Cn rule would destroy characters assigned in newer Unicode versions. The table carries a reminder to re-check the ranges on Unicode version bumps, since assignment turns a strip entry into a potential preserve-in-context case, exactly as happened when U+180F became Mongolian FVS4 in Unicode 14. Tests sweep all 3,739 code points through both the service engine and the vendored lightweight-skill copy (clean and inspect CLIs), with a boundary test pinning the assigned neighbours (U+2064, U+FFF9, U+E0001, U+E0100) to their existing kinds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: strip Unicode noncharacters in Layer A The 66 noncharacters (U+FDD0..U+FDEF plus U+nFFFE/U+nFFFF at the end of every plane) are permanently reserved for internal use and prohibited in interchange text (TUS 23.7). They render as nothing or tofu, survive normalisation and Python round-trips, and their category (Cn) meant the Cf catch-all never saw them, so both inspect_text and clean_text passed them through untouched: a ready-made covert channel. Strip them and report them under the new "noncharacter" inspect kind. Unlike the reserved Default_Ignorable ranges, noncharacters can never be assigned, so this carries no future-Unicode risk. Applied to both the service engine and the vendored lightweight-skill copy; tests sweep all 66 code points through both, and pin the assigned neighbours U+FDF0 and U+FFFD (replacement character) as untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: preserve visible-layout format controls next to their own script Egyptian hieroglyph quadrat controls (U+13430..U+1343F), Duployan shorthand controls (U+1BCA0..U+1BCA3), and musical beam/tie/slur/phrase controls (U+1D173..U+1D17A) are category Cf, so the catch-all stripped them, yet they visibly govern how their script renders (quadrat stacking, shorthand overlaps, beaming): removing them changes the rendered text, contradicting the "cleaners preserve the document body" invariant. Preserve them when adjacent to their own script, exactly like the existing Mongolian/Khmer/Hangul handling; floating between unrelated text they stay stripped and flagged, and --strip-emoji-glue paranoid mode still strips them everywhere. The vendored lightweight-skill engine needed next_input threaded through its _decide to express "preserve a begin-control from the character after it"; both engines now behave identically here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: preserve compatibility/halfwidth Hangul fillers in Hangul context Review feedback on the Layer A cluster: U+3164 and U+FFA0 were stripped unconditionally while the changelog claimed they were handled "like U+115F/U+1160", which are preserved after a Hangul jamo. Make the behaviour match the claim: both new fillers join _HANGUL_FILLERS and _is_hangul_jamo now covers the compatibility (U+3131-U+318E) and halfwidth (U+FFA1-U+FFDC) presentation forms, so each filler survives after a letter of its own form and remains contraband when floating between unrelated text. Changelog reworded to state the rule precisely. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>