mirror of
https://github.com/guillaumemeyer/watermarks-remover.git
synced 2026-08-22 13:11:57 +02:00
main
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
47252419e4 |
feat: consolidated Layer A hardening (missed carriers, reserved ignorables, noncharacters, layout format controls) (#133)
* fix: strip three missed Default_Ignorable invisible carriers in Layer A U+180F (Mongolian free variation selector-4, added in Unicode 14), U+3164 (Hangul filler), and U+FFA0 (halfwidth Hangul filler) are blank-rendering Default_Ignorable code points, but their Mn/Lo categories meant the Cf catch-all never saw them and they were absent from STRIP_CODEPOINTS. Both inspect_text and clean_text therefore passed them through untouched, even between plain ASCII. Add them to the strip set and wire U+180F into the Mongolian-FVS handling so it is stripped when floating but preserved after a Mongolian letter, exactly like FVS1-3. Replace the 0x180B..0x180D range with the named _MONGOLIAN_FVS set (clearer, and avoids the differently-handled Mongolian vowel separator at U+180E). Applied to both the service engine and the vendored lightweight-skill copy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HV9AE8QLqiXf7QnRe891EY * feat: strip reserved Default_Ignorable code points in Layer A U+2065, U+FFF0..U+FFF8, U+E0000, U+E0080..U+E00FF, and U+E01F0..U+E0FFF are unassigned code points carrying Other_Default_Ignorable_Code_Point, reserved for future default-ignorable characters. Conformant renderers display them invisibly, normalisation preserves them, and the Cf catch-all never sees them (they are category Cn), which made them ideal covert carriers that both inspect_text and clean_text passed through untouched. Add them to the strip set as explicit ranges and report them under the new "reserved_ignorable" inspect kind. Deliberately not a category-Cn rule: unicodedata is pinned per Python build, so a Cn rule would destroy characters assigned in newer Unicode versions. The table carries a reminder to re-check the ranges on Unicode version bumps, since assignment turns a strip entry into a potential preserve-in-context case, exactly as happened when U+180F became Mongolian FVS4 in Unicode 14. Tests sweep all 3,739 code points through both the service engine and the vendored lightweight-skill copy (clean and inspect CLIs), with a boundary test pinning the assigned neighbours (U+2064, U+FFF9, U+E0001, U+E0100) to their existing kinds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: strip Unicode noncharacters in Layer A The 66 noncharacters (U+FDD0..U+FDEF plus U+nFFFE/U+nFFFF at the end of every plane) are permanently reserved for internal use and prohibited in interchange text (TUS 23.7). They render as nothing or tofu, survive normalisation and Python round-trips, and their category (Cn) meant the Cf catch-all never saw them, so both inspect_text and clean_text passed them through untouched: a ready-made covert channel. Strip them and report them under the new "noncharacter" inspect kind. Unlike the reserved Default_Ignorable ranges, noncharacters can never be assigned, so this carries no future-Unicode risk. Applied to both the service engine and the vendored lightweight-skill copy; tests sweep all 66 code points through both, and pin the assigned neighbours U+FDF0 and U+FFFD (replacement character) as untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: preserve visible-layout format controls next to their own script Egyptian hieroglyph quadrat controls (U+13430..U+1343F), Duployan shorthand controls (U+1BCA0..U+1BCA3), and musical beam/tie/slur/phrase controls (U+1D173..U+1D17A) are category Cf, so the catch-all stripped them, yet they visibly govern how their script renders (quadrat stacking, shorthand overlaps, beaming): removing them changes the rendered text, contradicting the "cleaners preserve the document body" invariant. Preserve them when adjacent to their own script, exactly like the existing Mongolian/Khmer/Hangul handling; floating between unrelated text they stay stripped and flagged, and --strip-emoji-glue paranoid mode still strips them everywhere. The vendored lightweight-skill engine needed next_input threaded through its _decide to express "preserve a begin-control from the character after it"; both engines now behave identically here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: preserve compatibility/halfwidth Hangul fillers in Hangul context Review feedback on the Layer A cluster: U+3164 and U+FFA0 were stripped unconditionally while the changelog claimed they were handled "like U+115F/U+1160", which are preserved after a Hangul jamo. Make the behaviour match the claim: both new fillers join _HANGUL_FILLERS and _is_hangul_jamo now covers the compatibility (U+3131-U+318E) and halfwidth (U+FFA1-U+FFDC) presentation forms, so each filler survives after a letter of its own form and remains contraband when floating between unrelated text. Changelog reworded to state the rule precisely. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com> |
||
|
|
c1d1e5ac85 |
fix: pin vendored Cursor-skill text engine to the service copy (#96)
* fix: pin vendored Cursor-skill text engine to the service copy
The engine vendored into skills/clean-user-facing-text/ had silently
drifted behind service/scripts/text_unicode.py: it still blanket-
stripped legitimate RTL directional marks and isolates (corrupting
mixed RTL/LTR prose) and stripped emoji variation selectors after
arrow and symbol bases, because it never received the preservable-bidi
and emoji-base updates. Nothing in the suite compared the two copies.
Replace the vendored engine with an exact copy of the service one and
add a byte-equality test so any future engine change must land in both
files in the same commit, plus a behavioural regression test for the
RTL and arrow cases through the vendored CLI. The CLI wrappers remain
deliberately different (text-only skill: no stylometry, no
--strip-bidi).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: re-pin vendored engine after merging main and correct bidi comment
Merging main into the PR branch kept main's ruff-formatted service
engine (#103) but the PR's vendored copy pinned in
|
||
|
Guillaume Meyer (The Opinionated Man)andGitHub
|
723627716c |
chore: add Ruff linting and formatting with CI enforcement (#103)
Introduce Ruff (pinned at 0.16.3) as the project linter + formatter and enforce it in CI: - requirements-dev.txt: pin ruff==0.16.3 (exact pins, no drift) - ruff.toml: line-length 100, target py312; rule set E/F/W/I/UP/B/SIM/RUF/PLW/S with deliberate ignores (E501 for content strings, S603 for safe_arg subprocess calls, S101 asserts in tests) and per-file test ignores - Makefile: add lint / format / lint-fix targets - .github/workflows/ci.yml: add lint job (ruff check + format --check) - .gitignore: whitelist ruff.toml Also fix every finding the new gate surfaced so CI is green: - 109+ auto-fixes from ruff --fix (import sorting, simplifications, unused vars, re.I aliases, etc.) - explicit check=False on all subprocess.run calls (PLW1510) - harden sitemap XML parsing: reject DTD/entity declarations (S314) - replace hardcoded /tmp paths in tests with tmp_path (S108) - narrow/annotate intentional bare excepts (S110/S112), bind loop vars in closures (B023), raise ... from None (B904), strict= for zip (B905) - ruff format applied across service/ and tests/ Verified: ruff check + ruff format --check pass; 287 tests pass, 1 skip. |
||
|
|
1f95c548ce |
fix: preserve multilingual Unicode during text cleanup (#34)
Keep valid RTL controls, script joiners, variation sequences, and emoji structure while still removing malformed carriers, with regression coverage for each case. Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com> |
||
|
Guillaume Meyer (The Opinionated Man)andGitHub
|
55d4bdc9fc |
feat: split skill from service, add HTTP API and Docker distribution (#60)
* feat: split skill from service, add HTTP API and Docker distribution The agent skill (skills/remove-ai-marks/) is now a code-free remote client: all implementation moved to service/scripts/ and runs behind a stdlib HTTP service (server.py) with /health, /capabilities, /inspect, /clean and a dynamically generated OpenAPI 3.0.3 spec at /openapi.json. - Move scripts/ and the backend Dockerfiles under service/ - server.py: JSON/base64 HTTP entrypoint with size caps, binary guard, atomic writes, loopback default, optional bearer auth - Core Dockerfile (exiftool/qpdf/c2patool preinstalled) and a GHCR publish workflow for the core/markllm/markdiffusion images - compose.yaml (wr-* services, harness/heavy profiles) + compose-check.sh to validate the running stack (exit code only) - Fix markllm image build (tokenizers 0.22.2, CPU-only torch) and ctrlregen build (python:3.11 base for the 2023-era research pins) - Fix markllm/markdiffusion harness images missing common.py at runtime * docs: add .env.example and service configuration guide * fix: disable chain-of-thought for openai-compatible Layer B rewrites deepseek-v4-flash is a reasoning model: a one-line paraphrase burned 9,894 reasoning tokens (~100s) and hit the default timeout. Send reasoning_effort=none by default for the openai-compatible backend (--reasoning-effort / WATERMARKS_REWRITE_REASONING_EFFORT; 'off' omits the parameter), cutting the same rewrite to ~1s / 12 tokens. Tested end-to-end against api.deepseek.com. * fix: sanitize client-supplied filename in HTTP service CodeQL 'uncontrolled data in path expression' (server.py): a name like '../../x' flowed into Path(tmpdir) / name, letting an upload escape the request temp dir on write. Sanitize name to its basename in _decode_input (_safe_name) and refuse any joined path whose parent is not the tmpdir at the write sites (_tmp_path). Tests cover traversal names. * chore: gitignore .env (contains local rewrite credentials) * chore: deny-by-default gitignore and dockerignore; document compose env config .gitignore and service/.dockerignore now exclude everything by default and explicitly allow only what is publishable/needed: tracked source, docs, tests, .github, and (for images) the service/scripts/ tree that every Dockerfile COPYs. Root .dockerignore documents that all builds use service/ as context. README Configuration section now covers .env setup for docker compose, host-side export for CLI runs, and the full variable table. |