mirror of
https://github.com/guillaumemeyer/watermarks-remover.git
synced 2026-08-22 13:11:57 +02:00
Refuse binary input in the text-only tools (#24)
* Refuse binary input in the text-only tools inspect_text.py, clean_text.py and rewrite_text.py accept any path and decode it with errors="surrogateescape". Pointed at a .docx - a zip - they walk deflate-compressed bytes and report whatever codepoints fall out of them. The counts look like findings but track the compression, not the content: in one sample set a document with nothing hidden in its text reported 12 "suspicious" characters, while another with 54 real no-break spaces reported 11, none of which were the no-break spaces. clean_text.py is worse than misleading. It writes the mangled decode back, so `clean_text.py report.docx` reports "removed=1" and silently corrupts the document - the output still passes zipfile.is_zipfile() because the end-of- central-directory record survives, but reading a member raises. common.looks_binary() now sniffs magic numbers plus a control-byte ratio, and guard_binary() refuses with a message naming the tool that does handle the format. The ratio test is deliberately conservative so text in encodings other than UTF-8 keeps working, and every entry point takes --force-text to override. clean_file.py gets the same check on the branch where classify() falls back to "text" for unrecognised bytes. Adds tests covering magic-number and heuristic detection, the override, refusal without writing or backing up, and that clean_file.py still routes a .docx to the container path. * Address review: backup ordering, stdin sniff, router advice Three fixes from the review on #24. clean_file.py sniffed after --in-place had already taken the backup, so `clean_file.py --in-place mystery.bin` left a mystery.bin.bak sidecar behind before exiting 2 — for a file the run never touches, and exactly what clean_text.py avoids. The sniff now runs before backup_path(). The same hole applied to `--as text` on a .docx, which bypasses classify() entirely. The stdin path decoded before sniffing, which made detection depend on the console codec. It was worse than codec drift: the text layer also translates newlines, so PNG's `\x89PNG\r\n\x1a\n` arrived as `\x89PNG\n\x1a\n` and the magic number never matched — the file was refused by the NUL-byte heuristic instead, and would have sailed through had it lacked NULs. _read_stdin_capped now reads sys.stdin.buffer and guards the raw octets, matching the file path, with a text fallback for a replaced stdin. guard_binary always advised "Use inspect_file.py / clean_file.py", which is circular when the caller is one of them and classify() has already ruled out every known container. The advice is now a parameter: the text-only scripts keep the pointer to the routers, and the routers say the bytes match no supported format and point at --force-text / --as. Adds tests for the backup ordering (both --in-place paths), the advice split, and stdin magic that is not ASCII, across default, cp1252 and latin-1 stdio codecs — the previous stdin test piped a ZIP, whose "PK" header is ASCII and survives any of them. --------- Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
This commit is contained in:
@@ -79,6 +79,24 @@ python3 "$SCRIPTS/inspect_image.py" shot.png
|
||||
python3 "$SCRIPTS/clean_image.py" shot.png -o shot.cleaned.png
|
||||
```
|
||||
|
||||
### Text tools refuse binary input
|
||||
|
||||
`inspect_text.py`, `clean_text.py` and `rewrite_text.py` operate on text. Pointed
|
||||
at a `.docx`, `.pdf` or image they used to decode the compressed bytes and report
|
||||
whatever codepoints fell out — noise that tracks the compression, not the
|
||||
content — and `clean_text.py` then wrote those mangled bytes back, destroying the
|
||||
file. They now refuse binary input and name the tool that handles it:
|
||||
|
||||
```bash
|
||||
python3 "$SCRIPTS/inspect_text.py" report.docx
|
||||
# refusing to treat report.docx as text: it looks like a ZIP container (DOCX, ODT, …).
|
||||
# Use inspect_file.py / clean_file.py, which route by format,
|
||||
# or pass --force-text to scan the raw bytes anyway.
|
||||
```
|
||||
|
||||
Detection is by magic number plus a control-byte ratio, so text in encodings
|
||||
other than UTF-8 keeps working. `--force-text` overrides it everywhere.
|
||||
|
||||
## Optional SynthID pixel scoring
|
||||
|
||||
`inspect_image.py` and `clean_image.py` can report a pixel-domain SynthID
|
||||
|
||||
Reference in New Issue
Block a user