* Refuse binary input in the text-only tools
inspect_text.py, clean_text.py and rewrite_text.py accept any path and decode
it with errors="surrogateescape". Pointed at a .docx - a zip - they walk
deflate-compressed bytes and report whatever codepoints fall out of them. The
counts look like findings but track the compression, not the content: in one
sample set a document with nothing hidden in its text reported 12 "suspicious"
characters, while another with 54 real no-break spaces reported 11, none of
which were the no-break spaces.
clean_text.py is worse than misleading. It writes the mangled decode back, so
`clean_text.py report.docx` reports "removed=1" and silently corrupts the
document - the output still passes zipfile.is_zipfile() because the end-of-
central-directory record survives, but reading a member raises.
common.looks_binary() now sniffs magic numbers plus a control-byte ratio, and
guard_binary() refuses with a message naming the tool that does handle the
format. The ratio test is deliberately conservative so text in encodings other
than UTF-8 keeps working, and every entry point takes --force-text to override.
clean_file.py gets the same check on the branch where classify() falls back to
"text" for unrecognised bytes.
Adds tests covering magic-number and heuristic detection, the override, refusal
without writing or backing up, and that clean_file.py still routes a .docx to
the container path.
* Address review: backup ordering, stdin sniff, router advice
Three fixes from the review on #24.
clean_file.py sniffed after --in-place had already taken the backup, so
`clean_file.py --in-place mystery.bin` left a mystery.bin.bak sidecar behind
before exiting 2 — for a file the run never touches, and exactly what
clean_text.py avoids. The sniff now runs before backup_path(). The same hole
applied to `--as text` on a .docx, which bypasses classify() entirely.
The stdin path decoded before sniffing, which made detection depend on the
console codec. It was worse than codec drift: the text layer also translates
newlines, so PNG's `\x89PNG\r\n\x1a\n` arrived as `\x89PNG\n\x1a\n` and the
magic number never matched — the file was refused by the NUL-byte heuristic
instead, and would have sailed through had it lacked NULs. _read_stdin_capped
now reads sys.stdin.buffer and guards the raw octets, matching the file path,
with a text fallback for a replaced stdin.
guard_binary always advised "Use inspect_file.py / clean_file.py", which is
circular when the caller is one of them and classify() has already ruled out
every known container. The advice is now a parameter: the text-only scripts
keep the pointer to the routers, and the routers say the bytes match no
supported format and point at --force-text / --as.
Adds tests for the backup ordering (both --in-place paths), the advice split,
and stdin magic that is not ASCII, across default, cp1252 and latin-1 stdio
codecs — the previous stdin test piped a ZIP, whose "PK" header is ASCII and
survives any of them.
---------
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
