* fix(server): preserve image format extensions in /clean and use safe writes in av_meta (#148)
* style: apply ruff formatting to server.py and test_http_server.py
---------
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Directory-scale cleaning already exists in the CLI (audit_dir.py, -j
concurrency, SARIF export from #101), but the HTTP service handled one
file per request. Any web app or CI step talking to the service over
HTTP instead of the CLI paid N full round trips to clean N files.
Extract the single-file /inspect and /clean logic into _inspect_payload
and _clean_payload so both the existing single-file endpoints and the
new batch endpoints run the identical pipeline — no duplicated cleaning
logic. A malformed entry in a batch (bad base64, unknown option,
unrecognized format) surfaces as that entry's "ok": false with an
"error" string instead of aborting the rest of the batch.
Capped at WATERMARKS_MAX_BATCH_FILES per request (default 50) as
defense-in-depth against a request packing many tiny files into one
call; the existing MAX_BODY_BYTES envelope cap already bounds total
payload size the same as a single-file request.
/openapi.json picks up both routes automatically since the spec is
generated from the route table.
Closes#136
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* fix: rewrite ODT/EPUB manifests and measure real zip bytes (#122)
Two container correctness/security fixes from issue #122:
- clean_odt dropped marker-bearing parts while leaving their entries in
META-INF/manifest.xml, so readers flagged the package as damaged. It is
now two-pass: compute the dropped set, then rewrite the manifest
attribute-order-independently, and write each part exactly once. The same
bug class in clean_epub (dropped parts left in the OPF manifest, plus
dangling spine itemrefs) gets the same two-pass treatment.
- The zip budget trusted ZipInfo.file_size from the archive's own central
directory, so a crafted DOCX/ODT could declare a tiny size and still
expand via zf.read. Budgets are now charged on actual decompressed bytes
via _read_zip_member (streaming, cap enforced mid-read), with the declared
size kept only as a fast-path pre-reject.
* fix: classify unrecognized bytes as "unknown", not text (#122)
Two classification defects from issue #122:
- format_dispatch.classify_bytes fell back to "text" for any unrecognized
file, so a binary with valid UTF-8 runs could be decoded and written back
mangled (corrupted with --in-place) in clean_file auto mode. Unrecognized
bytes now classify as "unknown"; clean_file refuses them in auto mode
(exit 2, no write, router advice) and --as text / --force-text are the
explicit opt-ins. inspect_file reports kind "unknown" (exit 0), audit_lib
records a non-actionable item, and the HTTP server answers /inspect with
kind "unknown" but rejects /clean of unknown formats (400).
- classify(path) read the whole file to sniff a header, and only a full read
could detect zip containers. It now routes known extensions without
reading, sniffs a 4096-byte header once for images and prefix-based
containers, and reads the whole file only when the header is a zip local
header (PK), where the container signature lives in the central directory.
* feat: distinct exit code for partial audits (#122)
audit_dir and audit_website reported success (0) even when some files or
URLs could not be scanned; the exit status was computed only over the items
that succeeded. A scan that is missing items is not a clean scan.
- common.EXIT_PARTIAL = 3, with precedence: partial (3) > actionable (1)
> clean (0) — an incomplete audit is the more important CI signal.
- audit_dir returns 3 when any file was skipped/failed; audit_website
returns 3 when any URL failed to fetch or inspect. Both are independent
of the output format (human/json/sarif already share one return).
* fix: verify the pinned upstream ref on existing checkouts (#122)
setup_ctrlregen.sh/setup_synthid.sh (and their .ps1 twins) only verified
the pinned commit in the fresh-clone branch; an existing checkout at an
unknown or drifted revision was silently reused, defeating the commit pin.
All four scripts now check HEAD against the pinned ref in the
existing-checkout branch too, and repair by fetch + detach checkout
(re-applying the sparse-checkout set), failing hard if the ref cannot be
reached or the re-pin does not land on it.
* docs: unknown-format behavior, audit exit codes, backend isolation (#122)
- README: clean_file no longer auto-cleans unrecognized formats (--as text
/ --force-text are the opt-ins), and the CtrlRegen bootstrap documents the
isolation expectation for its research-era dependency pins plus the new
re-pin check on existing checkouts.
- SKILL.md: audit exit codes (0/1/2/3, partial=3) and a note that /clean
requires a name with a known extension.
- audit_website: document why stdlib ElementTree is used (stdlib-first) and
that defusedxml is the fallback if that policy changes (DTD rejection stays).
- requirements-ctrlregen.txt: advisory/isolation note for the pinned research
dependencies.
* test: ODT manifest and EPUB OPF dangling-ref regressions (#122)
- clean_odt: dropped marker-bearing parts remove their META-INF/manifest.xml
file-entry (attribute-order-independent), exactly one manifest entry, root
and surviving entries kept, and the manifest is byte-identical when nothing
is dropped.
- clean_epub: dropped non-content parts lose their <item> entry in the OPF
manifest, so the book no longer references removed members.
Introduce Ruff (pinned at 0.16.3) as the project linter + formatter and
enforce it in CI:
- requirements-dev.txt: pin ruff==0.16.3 (exact pins, no drift)
- ruff.toml: line-length 100, target py312; rule set E/F/W/I/UP/B/SIM/RUF/PLW/S
with deliberate ignores (E501 for content strings, S603 for safe_arg
subprocess calls, S101 asserts in tests) and per-file test ignores
- Makefile: add lint / format / lint-fix targets
- .github/workflows/ci.yml: add lint job (ruff check + format --check)
- .gitignore: whitelist ruff.toml
Also fix every finding the new gate surfaced so CI is green:
- 109+ auto-fixes from ruff --fix (import sorting, simplifications,
unused vars, re.I aliases, etc.)
- explicit check=False on all subprocess.run calls (PLW1510)
- harden sitemap XML parsing: reject DTD/entity declarations (S314)
- replace hardcoded /tmp paths in tests with tmp_path (S108)
- narrow/annotate intentional bare excepts (S110/S112), bind loop vars
in closures (B023), raise ... from None (B904), strict= for zip (B905)
- ruff format applied across service/ and tests/
Verified: ruff check + ruff format --check pass; 287 tests pass, 1 skip.
* feat: split skill from service, add HTTP API and Docker distribution
The agent skill (skills/remove-ai-marks/) is now a code-free remote client:
all implementation moved to service/scripts/ and runs behind a stdlib HTTP
service (server.py) with /health, /capabilities, /inspect, /clean and a
dynamically generated OpenAPI 3.0.3 spec at /openapi.json.
- Move scripts/ and the backend Dockerfiles under service/
- server.py: JSON/base64 HTTP entrypoint with size caps, binary guard,
atomic writes, loopback default, optional bearer auth
- Core Dockerfile (exiftool/qpdf/c2patool preinstalled) and a GHCR publish
workflow for the core/markllm/markdiffusion images
- compose.yaml (wr-* services, harness/heavy profiles) + compose-check.sh
to validate the running stack (exit code only)
- Fix markllm image build (tokenizers 0.22.2, CPU-only torch) and ctrlregen
build (python:3.11 base for the 2023-era research pins)
- Fix markllm/markdiffusion harness images missing common.py at runtime
* docs: add .env.example and service configuration guide
* fix: disable chain-of-thought for openai-compatible Layer B rewrites
deepseek-v4-flash is a reasoning model: a one-line paraphrase burned 9,894
reasoning tokens (~100s) and hit the default timeout. Send
reasoning_effort=none by default for the openai-compatible backend
(--reasoning-effort / WATERMARKS_REWRITE_REASONING_EFFORT; 'off' omits the
parameter), cutting the same rewrite to ~1s / 12 tokens. Tested end-to-end
against api.deepseek.com.
* fix: sanitize client-supplied filename in HTTP service
CodeQL 'uncontrolled data in path expression' (server.py): a name like
'../../x' flowed into Path(tmpdir) / name, letting an upload escape the
request temp dir on write. Sanitize name to its basename in _decode_input
(_safe_name) and refuse any joined path whose parent is not the tmpdir at
the write sites (_tmp_path). Tests cover traversal names.
* chore: gitignore .env (contains local rewrite credentials)
* chore: deny-by-default gitignore and dockerignore; document compose env config
.gitignore and service/.dockerignore now exclude everything by default and
explicitly allow only what is publishable/needed: tracked source, docs,
tests, .github, and (for images) the service/scripts/ tree that every
Dockerfile COPYs. Root .dockerignore documents that all builds use service/
as context. README Configuration section now covers .env setup for docker
compose, host-side export for CLI runs, and the full variable table.
