Files
Guillaume Meyer (The Opinionated Man)andGitHub 6df80e77a4 fix: correctness and security hardening of the cleaning scripts (#122) (#126)
* fix: rewrite ODT/EPUB manifests and measure real zip bytes (#122)

Two container correctness/security fixes from issue #122:

- clean_odt dropped marker-bearing parts while leaving their entries in
  META-INF/manifest.xml, so readers flagged the package as damaged. It is
  now two-pass: compute the dropped set, then rewrite the manifest
  attribute-order-independently, and write each part exactly once. The same
  bug class in clean_epub (dropped parts left in the OPF manifest, plus
  dangling spine itemrefs) gets the same two-pass treatment.
- The zip budget trusted ZipInfo.file_size from the archive's own central
  directory, so a crafted DOCX/ODT could declare a tiny size and still
  expand via zf.read. Budgets are now charged on actual decompressed bytes
  via _read_zip_member (streaming, cap enforced mid-read), with the declared
  size kept only as a fast-path pre-reject.

* fix: classify unrecognized bytes as "unknown", not text (#122)

Two classification defects from issue #122:

- format_dispatch.classify_bytes fell back to "text" for any unrecognized
  file, so a binary with valid UTF-8 runs could be decoded and written back
  mangled (corrupted with --in-place) in clean_file auto mode. Unrecognized
  bytes now classify as "unknown"; clean_file refuses them in auto mode
  (exit 2, no write, router advice) and --as text / --force-text are the
  explicit opt-ins. inspect_file reports kind "unknown" (exit 0), audit_lib
  records a non-actionable item, and the HTTP server answers /inspect with
  kind "unknown" but rejects /clean of unknown formats (400).
- classify(path) read the whole file to sniff a header, and only a full read
  could detect zip containers. It now routes known extensions without
  reading, sniffs a 4096-byte header once for images and prefix-based
  containers, and reads the whole file only when the header is a zip local
  header (PK), where the container signature lives in the central directory.

* feat: distinct exit code for partial audits (#122)

audit_dir and audit_website reported success (0) even when some files or
URLs could not be scanned; the exit status was computed only over the items
that succeeded. A scan that is missing items is not a clean scan.

- common.EXIT_PARTIAL = 3, with precedence: partial (3) > actionable (1)
  > clean (0) — an incomplete audit is the more important CI signal.
- audit_dir returns 3 when any file was skipped/failed; audit_website
  returns 3 when any URL failed to fetch or inspect. Both are independent
  of the output format (human/json/sarif already share one return).

* fix: verify the pinned upstream ref on existing checkouts (#122)

setup_ctrlregen.sh/setup_synthid.sh (and their .ps1 twins) only verified
the pinned commit in the fresh-clone branch; an existing checkout at an
unknown or drifted revision was silently reused, defeating the commit pin.

All four scripts now check HEAD against the pinned ref in the
existing-checkout branch too, and repair by fetch + detach checkout
(re-applying the sparse-checkout set), failing hard if the ref cannot be
reached or the re-pin does not land on it.

* docs: unknown-format behavior, audit exit codes, backend isolation (#122)

- README: clean_file no longer auto-cleans unrecognized formats (--as text
  / --force-text are the opt-ins), and the CtrlRegen bootstrap documents the
  isolation expectation for its research-era dependency pins plus the new
  re-pin check on existing checkouts.
- SKILL.md: audit exit codes (0/1/2/3, partial=3) and a note that /clean
  requires a name with a known extension.
- audit_website: document why stdlib ElementTree is used (stdlib-first) and
  that defusedxml is the fallback if that policy changes (DTD rejection stays).
- requirements-ctrlregen.txt: advisory/isolation note for the pinned research
  dependencies.

* test: ODT manifest and EPUB OPF dangling-ref regressions (#122)

- clean_odt: dropped marker-bearing parts remove their META-INF/manifest.xml
  file-entry (attribute-order-independent), exactly one manifest entry, root
  and surviving entries kept, and the manifest is byte-identical when nothing
  is dropped.
- clean_epub: dropped non-content parts lose their <item> entry in the OPF
  manifest, so the book no longer references removed members.
2026-08-17 17:15:35 -07:00

133 lines
4.2 KiB
Bash

#!/usr/bin/env bash
set -euo pipefail
# Bootstrap an external noai-watermark checkout for the optional CtrlRegen
# pixel-removal backend.
#
# The upstream project (https://github.com/mertizci/noai-watermark) does not
# ship a LICENSE file, so its code is treated as all-rights-reserved and is
# NOT bundled in this repository. This script clones it locally and installs
# only the dependencies needed by clean_ctrlregen.py.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DEFAULT_DIR="${NOAI_WATERMARK_DIR:-$HOME/noai-watermark}"
DIR=""
# Pinned upstream commit (2026-08-13). Do not point at a moving branch.
REF="b642ae45d20eded52c96d570985eb4e3e427aac8"
PYTHON="${PYTHON:-python3}"
usage() {
cat <<'EOF'
Usage: setup_ctrlregen.sh [--dir PATH] [--ref REF] [--python PYTHON]
Clones (if needed) mertizci/noai-watermark, creates a venv, and installs the
Python dependencies required by clean_ctrlregen.py (including torch).
Options:
--dir PATH checkout directory (default: $NOAI_WATERMARK_DIR or ~/noai-watermark)
--ref REF git ref to checkout (default: pinned commit SHA)
--python PY Python interpreter used to create the venv (default: python3)
EOF
}
while [[ $# -gt 0 ]]; do
case "$1" in
--dir)
DIR="${2:?--dir requires a value}"
shift 2
;;
--ref)
REF="${2:?--ref requires a value}"
shift 2
;;
--python)
PYTHON="${2:?--python requires a value}"
shift 2
;;
-h|--help)
usage
exit 0
;;
*)
echo "unknown option: $1" >&2
usage >&2
exit 2
;;
esac
done
DIR="${DIR:-$DEFAULT_DIR}"
mkdir -p "$(dirname "$DIR")"
if realpath -m . >/dev/null 2>&1; then # BSD/macOS realpath has no -m
DIR="$(realpath -m "$DIR")"
else
DIR="$(cd "$(dirname "$DIR")" && pwd)/$(basename "$DIR")"
fi
if [[ ! -d "$DIR/.git" ]]; then
echo "Cloning noai-watermark into $DIR (pinned ref: $REF)"
git clone --depth 1 --filter=blob:none --sparse \
https://github.com/mertizci/noai-watermark.git "$DIR"
git -C "$DIR" fetch --depth 1 origin "$REF"
git -C "$DIR" checkout --detach "$REF"
git -C "$DIR" sparse-checkout set --no-cone '/src/'
HEAD_SHA="$(git -C "$DIR" rev-parse HEAD)"
if [[ "$HEAD_SHA" != "$REF" ]]; then
echo "error: expected pinned ref $REF, got $HEAD_SHA" >&2
exit 1
fi
else
echo "Using existing checkout: $DIR"
HEAD_SHA="$(git -C "$DIR" rev-parse HEAD 2>/dev/null || true)"
if [[ "$HEAD_SHA" != "$REF" ]]; then
echo "existing checkout not at pinned ref $REF (HEAD: ${HEAD_SHA:-missing}); re-pinning"
git -C "$DIR" fetch --depth 1 origin "$REF" || {
echo "error: could not fetch pinned ref $REF" >&2
exit 1
}
git -C "$DIR" checkout --detach "$REF"
git -C "$DIR" sparse-checkout set --no-cone '/src/'
HEAD_SHA="$(git -C "$DIR" rev-parse HEAD)"
if [[ "$HEAD_SHA" != "$REF" ]]; then
echo "error: expected pinned ref $REF, got $HEAD_SHA" >&2
exit 1
fi
fi
fi
if [[ ! -x "$DIR/.venv/bin/python" ]]; then
echo "Creating venv at $DIR/.venv"
"$PYTHON" -m venv "$DIR/.venv"
fi
echo "Installing Python dependencies"
# Pin pip itself (unpinned --upgrade pip was a supply-chain drift point).
"$DIR/.venv/bin/python" -m pip install --upgrade "pip==26.2.1"
# Install torch with the right platform index before the other pinned deps.
if command -v nvidia-smi >/dev/null 2>&1; then
cuda="$(nvidia-smi 2>/dev/null | sed -n 's/.*CUDA Version: \([0-9]*\.[0-9]*\).*/\1/p' | head -1)"
if [[ -n "$cuda" ]]; then
tag="cu${cuda/./}"
index="https://download.pytorch.org/whl/$tag"
echo "NVIDIA GPU detected (CUDA $cuda); installing torch from $index"
"$DIR/.venv/bin/python" -m pip install torch --index-url "$index"
else
echo "nvidia-smi present but no CUDA version found; installing default torch"
"$DIR/.venv/bin/python" -m pip install torch
fi
else
echo "No NVIDIA GPU detected; installing default torch (CPU/MPS)"
"$DIR/.venv/bin/python" -m pip install torch
fi
"$DIR/.venv/bin/python" -m pip install -r "$SCRIPT_DIR/requirements-ctrlregen.txt"
cat <<EOF
Done. Remove a watermark with:
export NOAI_WATERMARK_DIR="$DIR"
"$DIR/.venv/bin/python" "$SCRIPT_DIR/clean_ctrlregen.py" IMAGE -o OUT
EOF