* fix: keep the truncated tail instead of dropping it in png/isobmff strips
strip_png stopped walking at the first chunk it could not parse and
never copied the remainder; strip_isobmff rebuilt only the boxes that
parsed. A truncated IDAT/mdat — the actual coded image — was dropped
from the output, the run reported "already clean", the exit code was
0: a user with a recoverable (viewable) image ended with an
unopenable husk and the tool saying it was fine (#170).
Both strippers now keep the unparseable tail verbatim and append a
"kept N bytes of truncated ... tail" action — a real report, so the
run is never mistaken for a no-op clean and every original byte
survives (truncated-image-capable readers can still open the output).
* refactor: surface parser walk end; stop false truncation reports
- _parse_isobmff_boxes returns (boxes, scanned_end), so strip_isobmff no longer re-walks parsed boxes to recover the stop offset.
- Fewer than 8 trailing bytes is trailing junk, not truncation: kept verbatim without a 'file truncated' action.
- Tests: drop unused noqa (RUF100) and unpack (RUF059), simplify the PNG tail assertion to byte-equality, cover the trailing-junk case.
---------
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
clean_staged folded three outcomes into one "skipped" bucket and
main() never let a skip affect the exit code: a clean_file.py that
raised, was killed, or died before writing JSON landed with the
deliberate unknown-format skip, exit 0 let the commit proceed
uncleaned, and the traceback stayed invisible outside pre-commit
verbose mode (#159) — breaking the hook docstring's own contract
that exit 0 means every file was already clean.
_clean_one now returns a distinct "failed" for nonzero non-2 cleaner
exits and unparseable/empty stdout (the unknown-format skip keeps its
own bucket), and main() exits 1 with a commit-blocked banner when any
path failed, printing the cleaner's stderr first so the cause is
visible.
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: guillaumemeyer <guillaumemeyer@users.noreply.github.com>
* fix: route website binary formats to their real scanners
guess_kind recognized html/png/jpeg/svg/pdf/docx/odt/markdown/text
and fell through to text for everything else: webp, avif, heic, gif,
bmp, tiff, xlsx, pptx, epub, mp4/mov/m4a, wav, and mp3 downloads were
written to asset.txt and handed to the Unicode scanner, so a WebP
carrying a C2PA RIFF chunk reported clean with no failure signal —
audit_website exited 0 with with_c2pa: 0 (#166). The drift between
this table and format_dispatch's IMAGE/CONTAINER/AV_EXTS was where
the bug lived.
Extend all three classification stages (content type, suffix, magic
bytes with ftyp brand dispatch) to cover every format the local audit
handles, and map the new kinds in _EXT_FOR_KIND so the temp file
keeps a true suffix for the local classify.
* fix: satisfy ruff lint and format checks
---------
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Co-authored-by: guillaumemeyer <guillaumemeyer@users.noreply.github.com>
* fix: truncated ISOBMFF containers still run the C2PA byte-scan fallback
inspect_isobmff returned early when no boxes parse, skipping the
whole-file C2PA byte scan that every sibling inspector (png, jpeg,
gif, tiff) reaches even after a truncation: a first box whose size
field overruns the file — the truncated AVIF/HEIC/MP4 download shape —
reported has_c2pa=False with only a not-a-valid finding while literal
b"c2pa" sat in the file (#167). Also reached for MP4/MOV/M4A through
av_meta._inspect_mp4.
Run the byte scan before reporting the parse failure: a truncated
container with markers byte-scans to has_c2pa=True (finding plus the
parse-failure note, so the truncation isn't hidden); a walkable
container keeps its stronger box-level findings; a markerless
truncated container keeps the plain not-a-valid shape.
* fix: satisfy ruff in isobmff truncated byte-scan tests
- drop the unused # noqa: E402 (already per-file-ignored for tests/**)
- bind the unused has_ai unpacked value to _ (RUF059)
---------
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Co-authored-by: guillaumemeyer <guillaumemeyer@users.noreply.github.com>
* fix: keep collected evidence when a later zip member fails to read
The OOXML, ODT, and EPUB inspectors accumulated has_c2pa/has_ai/
findings member by member, then discarded all of it if any later
member raised: the except returned hardcoded False, False, ["not a
valid X zip"] — markers already found in earlier members were thrown
away and a container that could not be fully read reported as one
that was read and found clean. The pre-commit gate then exited 0 on
exactly the file that carried evidence (#164).
Keep the accumulated evidence and append a partial-read note naming
the exception class; only a wholly-garbage container (nothing
accumulated) keeps the not-a-valid-zip shape. EPUB's except is
widened from BadZipFile to the shared _ZIP_PARSE_ERRORS tuple,
closing the asymmetry where a zlib error crashed the scan while a
CRC failure discarded everything.
* fix: silence RUF100/RUF059 lint errors in zip partial-evidence tests
---------
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Co-authored-by: guillaumemeyer <guillaumemeyer@users.noreply.github.com>
* fix: an unreadable text file is a failed scan, not a clean one
scan_file's text branch caught the OSError and returned an item with
an error key and no confidence key — is_actionable read that as
clean, so a file the scanner could not open was indistinguishable
from one it opened and found clean. All three audit callers inherited
the lie: the pre-commit hook returned 0, and audit_dir's _scan_worker
never produced the EXIT_PARTIAL signal its try/except exists for,
because scan_file swallowed the error before it could raise (#158;
the same probe-did-not-answer class as #155).
Let the OSError propagate: the audit_dir/audit_website wrappers
already convert a raised exception into a files_skipped entry with
EXIT_PARTIAL, which is exactly the contract common.py documents for
a partial scan.
* chore: satisfy ruff on the unreadable-text scan fix
- drop the unused OSError binding (F841) in scan_file's text branch
- sort the test's import block (I001): stdlib first, then audit_lib
---------
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Co-authored-by: guillaumemeyer <guillaumemeyer@users.noreply.github.com>
The core image installs libimage-exiftool-perl but not Archive::Zip;
without it exiftool warns "Install Archive::Zip to decode compressed
ZIP information" and misidentifies every Office/OpenDocument input
as a generic ZIP (File Type: ZIP, MIME application/zip), reading
none of the embedded metadata — the exiftool arm of /inspect is
blind for exactly the container formats the README advertises, and
its intended role as a second, independent check silently degrades
to nothing (#154).
Add libarchive-zip-perl (Debian's packaged Archive::Zip) to the apt
layer.
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* fix: decode XML entities before Layer A scrub in OOXML/ODF
Watermark carriers encoded as XML character references (e.g.
​) reached Layer A as nine ASCII characters and survived
cleaning untouched, because the XML parser in Word/Writer decodes
the entity back into the invisible carrier after the cleaner ran
(#129).
A shared _decode_xml_entities resolves exactly what a conforming
XML parser resolves (numeric references and the five predefined
entities) and leaves everything else literal. The DOCX/XLSX/PPTX
text-run scrubbers decode before clean_text and re-encode on the
way out; the ODT scrubber splits paragraph content into markup and
text segments and only round-trips the text segments, because a
whole-paragraph decode/re-encode would entity-escape the nested
text:span/text:tab markup. Untouched runs keep their original
bytes.
* fix: satisfy ruff lint and format checks
---------
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
COPY preserves source file and directory modes, so a build host with a
restrictive umask (e.g. 027) landed the scripts as 0640 root:root and
the /app/scripts directory as 0750 -- unreadable and untraversable for
the unprivileged runtime user (uid 10001). The image built fine and
failed only at container start, making it easy to miss (#131).
Normalize after the copy with chmod -R a+rX: world-readable everywhere,
execute bit for directories only, intentionally-executable scripts keep
their bit -- deterministic regardless of the builder's umask.
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
* fix: report unmeasurable burstiness instead of scoring it as max LLM-like
compute_burstiness returned cv=0.0 for an empty (or single-sentence)
list, which the tiering read as perfectly uniform prose — the strongest
LLM-likeness signal — so any file whose body yielded no parseable
sentences (e.g. entirely wrapped in a code fence) scored falsely high
while word_count stayed high enough to defeat the small-sample dampener
(#132). On a real export this hit 159 of 162 files as pure artifact.
compute_burstiness now returns None for the CV when it cannot be
measured; the scoring site drops the burstiness component and
renormalizes the composite over AI-phrase density and lexical
diversity, with an explicit note. burstiness_cv is null in JSON for
unmeasurable cases, the CLI prints n/a instead of crashing, and the
uniform-cadence finding guards on cv not being None.
* style: satisfy ruff import sorting and formatting
---------
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
Co-authored-by: Guillaume Meyer <guillaumemeyer@users.noreply.github.com>
* fix: parse single-quoted attributes in relationship/manifest pruning
XML allows attribute values in single or double quotes. The Target
extraction in _prune_dangling_relationships and the full-path
extraction in _prune_odt_manifest_entries matched only double quotes,
so a valid single-quoted Relationship parsed as an empty target,
resolved to the package base directory, and was deleted outright —
corrupting DOCX/XLSX/PPTX and ODT packages produced by tools that
emit single quotes (#130).
Both extractions now use a backreference pattern
((["'])(.*?)\1) so either quote style resolves the real target, and
pruning still only drops relationships whose parts were genuinely
removed. Regression tests cover the single-quoted kept relationship
(the reported corruption) and the double-quoted dropped part.
* fix: satisfy ruff lint and format checks
---------
Co-authored-by: yzxcj797 <yzxcj797@users.noreply.github.com>
Co-authored-by: Guillaume Meyer (The Opinionated Man) <1385518+guillaumemeyer@users.noreply.github.com>
