mirror of
https://github.com/guillaumemeyer/watermarks-remover.git
synced 2026-08-22 13:11:57 +02:00
main
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Guillaume Meyer (The Opinionated Man)andGitHub
|
6df80e77a4 |
fix: correctness and security hardening of the cleaning scripts (#122) (#126)
* fix: rewrite ODT/EPUB manifests and measure real zip bytes (#122) Two container correctness/security fixes from issue #122: - clean_odt dropped marker-bearing parts while leaving their entries in META-INF/manifest.xml, so readers flagged the package as damaged. It is now two-pass: compute the dropped set, then rewrite the manifest attribute-order-independently, and write each part exactly once. The same bug class in clean_epub (dropped parts left in the OPF manifest, plus dangling spine itemrefs) gets the same two-pass treatment. - The zip budget trusted ZipInfo.file_size from the archive's own central directory, so a crafted DOCX/ODT could declare a tiny size and still expand via zf.read. Budgets are now charged on actual decompressed bytes via _read_zip_member (streaming, cap enforced mid-read), with the declared size kept only as a fast-path pre-reject. * fix: classify unrecognized bytes as "unknown", not text (#122) Two classification defects from issue #122: - format_dispatch.classify_bytes fell back to "text" for any unrecognized file, so a binary with valid UTF-8 runs could be decoded and written back mangled (corrupted with --in-place) in clean_file auto mode. Unrecognized bytes now classify as "unknown"; clean_file refuses them in auto mode (exit 2, no write, router advice) and --as text / --force-text are the explicit opt-ins. inspect_file reports kind "unknown" (exit 0), audit_lib records a non-actionable item, and the HTTP server answers /inspect with kind "unknown" but rejects /clean of unknown formats (400). - classify(path) read the whole file to sniff a header, and only a full read could detect zip containers. It now routes known extensions without reading, sniffs a 4096-byte header once for images and prefix-based containers, and reads the whole file only when the header is a zip local header (PK), where the container signature lives in the central directory. * feat: distinct exit code for partial audits (#122) audit_dir and audit_website reported success (0) even when some files or URLs could not be scanned; the exit status was computed only over the items that succeeded. A scan that is missing items is not a clean scan. - common.EXIT_PARTIAL = 3, with precedence: partial (3) > actionable (1) > clean (0) — an incomplete audit is the more important CI signal. - audit_dir returns 3 when any file was skipped/failed; audit_website returns 3 when any URL failed to fetch or inspect. Both are independent of the output format (human/json/sarif already share one return). * fix: verify the pinned upstream ref on existing checkouts (#122) setup_ctrlregen.sh/setup_synthid.sh (and their .ps1 twins) only verified the pinned commit in the fresh-clone branch; an existing checkout at an unknown or drifted revision was silently reused, defeating the commit pin. All four scripts now check HEAD against the pinned ref in the existing-checkout branch too, and repair by fetch + detach checkout (re-applying the sparse-checkout set), failing hard if the ref cannot be reached or the re-pin does not land on it. * docs: unknown-format behavior, audit exit codes, backend isolation (#122) - README: clean_file no longer auto-cleans unrecognized formats (--as text / --force-text are the opt-ins), and the CtrlRegen bootstrap documents the isolation expectation for its research-era dependency pins plus the new re-pin check on existing checkouts. - SKILL.md: audit exit codes (0/1/2/3, partial=3) and a note that /clean requires a name with a known extension. - audit_website: document why stdlib ElementTree is used (stdlib-first) and that defusedxml is the fallback if that policy changes (DTD rejection stays). - requirements-ctrlregen.txt: advisory/isolation note for the pinned research dependencies. * test: ODT manifest and EPUB OPF dangling-ref regressions (#122) - clean_odt: dropped marker-bearing parts remove their META-INF/manifest.xml file-entry (attribute-order-independent), exactly one manifest entry, root and surviving entries kept, and the manifest is byte-identical when nothing is dropped. - clean_epub: dropped non-content parts lose their <item> entry in the OPF manifest, so the book no longer references removed members. |
||
|
Guillaume Meyer (The Opinionated Man)andGitHub
|
723627716c |
chore: add Ruff linting and formatting with CI enforcement (#103)
Introduce Ruff (pinned at 0.16.3) as the project linter + formatter and enforce it in CI: - requirements-dev.txt: pin ruff==0.16.3 (exact pins, no drift) - ruff.toml: line-length 100, target py312; rule set E/F/W/I/UP/B/SIM/RUF/PLW/S with deliberate ignores (E501 for content strings, S603 for safe_arg subprocess calls, S101 asserts in tests) and per-file test ignores - Makefile: add lint / format / lint-fix targets - .github/workflows/ci.yml: add lint job (ruff check + format --check) - .gitignore: whitelist ruff.toml Also fix every finding the new gate surfaced so CI is green: - 109+ auto-fixes from ruff --fix (import sorting, simplifications, unused vars, re.I aliases, etc.) - explicit check=False on all subprocess.run calls (PLW1510) - harden sitemap XML parsing: reject DTD/entity declarations (S314) - replace hardcoded /tmp paths in tests with tmp_path (S108) - narrow/annotate intentional bare excepts (S110/S112), bind loop vars in closures (B023), raise ... from None (B904), strict= for zip (B905) - ruff format applied across service/ and tests/ Verified: ruff check + ruff format --check pass; 287 tests pass, 1 skip. |
||
|
Guillaume Meyer (The Opinionated Man)andGitHub
|
55d4bdc9fc |
feat: split skill from service, add HTTP API and Docker distribution (#60)
* feat: split skill from service, add HTTP API and Docker distribution The agent skill (skills/remove-ai-marks/) is now a code-free remote client: all implementation moved to service/scripts/ and runs behind a stdlib HTTP service (server.py) with /health, /capabilities, /inspect, /clean and a dynamically generated OpenAPI 3.0.3 spec at /openapi.json. - Move scripts/ and the backend Dockerfiles under service/ - server.py: JSON/base64 HTTP entrypoint with size caps, binary guard, atomic writes, loopback default, optional bearer auth - Core Dockerfile (exiftool/qpdf/c2patool preinstalled) and a GHCR publish workflow for the core/markllm/markdiffusion images - compose.yaml (wr-* services, harness/heavy profiles) + compose-check.sh to validate the running stack (exit code only) - Fix markllm image build (tokenizers 0.22.2, CPU-only torch) and ctrlregen build (python:3.11 base for the 2023-era research pins) - Fix markllm/markdiffusion harness images missing common.py at runtime * docs: add .env.example and service configuration guide * fix: disable chain-of-thought for openai-compatible Layer B rewrites deepseek-v4-flash is a reasoning model: a one-line paraphrase burned 9,894 reasoning tokens (~100s) and hit the default timeout. Send reasoning_effort=none by default for the openai-compatible backend (--reasoning-effort / WATERMARKS_REWRITE_REASONING_EFFORT; 'off' omits the parameter), cutting the same rewrite to ~1s / 12 tokens. Tested end-to-end against api.deepseek.com. * fix: sanitize client-supplied filename in HTTP service CodeQL 'uncontrolled data in path expression' (server.py): a name like '../../x' flowed into Path(tmpdir) / name, letting an upload escape the request temp dir on write. Sanitize name to its basename in _decode_input (_safe_name) and refuse any joined path whose parent is not the tmpdir at the write sites (_tmp_path). Tests cover traversal names. * chore: gitignore .env (contains local rewrite credentials) * chore: deny-by-default gitignore and dockerignore; document compose env config .gitignore and service/.dockerignore now exclude everything by default and explicitly allow only what is publishable/needed: tracked source, docs, tests, .github, and (for images) the service/scripts/ tree that every Dockerfile COPYs. Root .dockerignore documents that all builds use service/ as context. README Configuration section now covers .env setup for docker compose, host-side export for CLI runs, and the full variable table. |
||
|
|
256d90d1b1 |
Refuse binary input in the text-only tools (#24)
* Refuse binary input in the text-only tools inspect_text.py, clean_text.py and rewrite_text.py accept any path and decode it with errors="surrogateescape". Pointed at a .docx - a zip - they walk deflate-compressed bytes and report whatever codepoints fall out of them. The counts look like findings but track the compression, not the content: in one sample set a document with nothing hidden in its text reported 12 "suspicious" characters, while another with 54 real no-break spaces reported 11, none of which were the no-break spaces. clean_text.py is worse than misleading. It writes the mangled decode back, so `clean_text.py report.docx` reports "removed=1" and silently corrupts the document - the output still passes zipfile.is_zipfile() because the end-of- central-directory record survives, but reading a member raises. common.looks_binary() now sniffs magic numbers plus a control-byte ratio, and guard_binary() refuses with a message naming the tool that does handle the format. The ratio test is deliberately conservative so text in encodings other than UTF-8 keeps working, and every entry point takes --force-text to override. clean_file.py gets the same check on the branch where classify() falls back to "text" for unrecognised bytes. Adds tests covering magic-number and heuristic detection, the override, refusal without writing or backing up, and that clean_file.py still routes a .docx to the container path. * Address review: backup ordering, stdin sniff, router advice Three fixes from the review on #24. clean_file.py sniffed after --in-place had already taken the backup, so `clean_file.py --in-place mystery.bin` left a mystery.bin.bak sidecar behind before exiting 2 — for a file the run never touches, and exactly what clean_text.py avoids. The sniff now runs before backup_path(). The same hole applied to `--as text` on a .docx, which bypasses classify() entirely. The stdin path decoded before sniffing, which made detection depend on the console codec. It was worse than codec drift: the text layer also translates newlines, so PNG's `\x89PNG\r\n\x1a\n` arrived as `\x89PNG\n\x1a\n` and the magic number never matched — the file was refused by the NUL-byte heuristic instead, and would have sailed through had it lacked NULs. _read_stdin_capped now reads sys.stdin.buffer and guards the raw octets, matching the file path, with a text fallback for a replaced stdin. guard_binary always advised "Use inspect_file.py / clean_file.py", which is circular when the caller is one of them and classify() has already ruled out every known container. The advice is now a parameter: the text-only scripts keep the pointer to the routers, and the routers say the bytes match no supported format and point at --force-text / --as. Adds tests for the backup ordering (both --in-place paths), the advice split, and stdin magic that is not ASCII, across default, cp1252 and latin-1 stdio codecs — the previous stdin test piped a ZIP, whose "PK" header is ASCII and survives any of them. --------- Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com> |