mirror of
https://github.com/guillaumemeyer/watermarks-remover.git
synced 2026-08-22 13:11:57 +02:00
* fix: rewrite ODT/EPUB manifests and measure real zip bytes (#122) Two container correctness/security fixes from issue #122: - clean_odt dropped marker-bearing parts while leaving their entries in META-INF/manifest.xml, so readers flagged the package as damaged. It is now two-pass: compute the dropped set, then rewrite the manifest attribute-order-independently, and write each part exactly once. The same bug class in clean_epub (dropped parts left in the OPF manifest, plus dangling spine itemrefs) gets the same two-pass treatment. - The zip budget trusted ZipInfo.file_size from the archive's own central directory, so a crafted DOCX/ODT could declare a tiny size and still expand via zf.read. Budgets are now charged on actual decompressed bytes via _read_zip_member (streaming, cap enforced mid-read), with the declared size kept only as a fast-path pre-reject. * fix: classify unrecognized bytes as "unknown", not text (#122) Two classification defects from issue #122: - format_dispatch.classify_bytes fell back to "text" for any unrecognized file, so a binary with valid UTF-8 runs could be decoded and written back mangled (corrupted with --in-place) in clean_file auto mode. Unrecognized bytes now classify as "unknown"; clean_file refuses them in auto mode (exit 2, no write, router advice) and --as text / --force-text are the explicit opt-ins. inspect_file reports kind "unknown" (exit 0), audit_lib records a non-actionable item, and the HTTP server answers /inspect with kind "unknown" but rejects /clean of unknown formats (400). - classify(path) read the whole file to sniff a header, and only a full read could detect zip containers. It now routes known extensions without reading, sniffs a 4096-byte header once for images and prefix-based containers, and reads the whole file only when the header is a zip local header (PK), where the container signature lives in the central directory. * feat: distinct exit code for partial audits (#122) audit_dir and audit_website reported success (0) even when some files or URLs could not be scanned; the exit status was computed only over the items that succeeded. A scan that is missing items is not a clean scan. - common.EXIT_PARTIAL = 3, with precedence: partial (3) > actionable (1) > clean (0) — an incomplete audit is the more important CI signal. - audit_dir returns 3 when any file was skipped/failed; audit_website returns 3 when any URL failed to fetch or inspect. Both are independent of the output format (human/json/sarif already share one return). * fix: verify the pinned upstream ref on existing checkouts (#122) setup_ctrlregen.sh/setup_synthid.sh (and their .ps1 twins) only verified the pinned commit in the fresh-clone branch; an existing checkout at an unknown or drifted revision was silently reused, defeating the commit pin. All four scripts now check HEAD against the pinned ref in the existing-checkout branch too, and repair by fetch + detach checkout (re-applying the sparse-checkout set), failing hard if the ref cannot be reached or the re-pin does not land on it. * docs: unknown-format behavior, audit exit codes, backend isolation (#122) - README: clean_file no longer auto-cleans unrecognized formats (--as text / --force-text are the opt-ins), and the CtrlRegen bootstrap documents the isolation expectation for its research-era dependency pins plus the new re-pin check on existing checkouts. - SKILL.md: audit exit codes (0/1/2/3, partial=3) and a note that /clean requires a name with a known extension. - audit_website: document why stdlib ElementTree is used (stdlib-first) and that defusedxml is the fallback if that policy changes (DTD rejection stays). - requirements-ctrlregen.txt: advisory/isolation note for the pinned research dependencies. * test: ODT manifest and EPUB OPF dangling-ref regressions (#122) - clean_odt: dropped marker-bearing parts remove their META-INF/manifest.xml file-entry (attribute-order-independent), exactly one manifest entry, root and surviving entries kept, and the manifest is byte-identical when nothing is dropped. - clean_epub: dropped non-content parts lose their <item> entry in the OPF manifest, so the book no longer references removed members.
133 lines
4.2 KiB
Bash
133 lines
4.2 KiB
Bash
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
|
|
# Bootstrap an external noai-watermark checkout for the optional CtrlRegen
|
|
# pixel-removal backend.
|
|
#
|
|
# The upstream project (https://github.com/mertizci/noai-watermark) does not
|
|
# ship a LICENSE file, so its code is treated as all-rights-reserved and is
|
|
# NOT bundled in this repository. This script clones it locally and installs
|
|
# only the dependencies needed by clean_ctrlregen.py.
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
DEFAULT_DIR="${NOAI_WATERMARK_DIR:-$HOME/noai-watermark}"
|
|
DIR=""
|
|
# Pinned upstream commit (2026-08-13). Do not point at a moving branch.
|
|
REF="b642ae45d20eded52c96d570985eb4e3e427aac8"
|
|
PYTHON="${PYTHON:-python3}"
|
|
|
|
usage() {
|
|
cat <<'EOF'
|
|
Usage: setup_ctrlregen.sh [--dir PATH] [--ref REF] [--python PYTHON]
|
|
|
|
Clones (if needed) mertizci/noai-watermark, creates a venv, and installs the
|
|
Python dependencies required by clean_ctrlregen.py (including torch).
|
|
|
|
Options:
|
|
--dir PATH checkout directory (default: $NOAI_WATERMARK_DIR or ~/noai-watermark)
|
|
--ref REF git ref to checkout (default: pinned commit SHA)
|
|
--python PY Python interpreter used to create the venv (default: python3)
|
|
EOF
|
|
}
|
|
|
|
while [[ $# -gt 0 ]]; do
|
|
case "$1" in
|
|
--dir)
|
|
DIR="${2:?--dir requires a value}"
|
|
shift 2
|
|
;;
|
|
--ref)
|
|
REF="${2:?--ref requires a value}"
|
|
shift 2
|
|
;;
|
|
--python)
|
|
PYTHON="${2:?--python requires a value}"
|
|
shift 2
|
|
;;
|
|
-h|--help)
|
|
usage
|
|
exit 0
|
|
;;
|
|
*)
|
|
echo "unknown option: $1" >&2
|
|
usage >&2
|
|
exit 2
|
|
;;
|
|
esac
|
|
done
|
|
|
|
DIR="${DIR:-$DEFAULT_DIR}"
|
|
mkdir -p "$(dirname "$DIR")"
|
|
if realpath -m . >/dev/null 2>&1; then # BSD/macOS realpath has no -m
|
|
DIR="$(realpath -m "$DIR")"
|
|
else
|
|
DIR="$(cd "$(dirname "$DIR")" && pwd)/$(basename "$DIR")"
|
|
fi
|
|
|
|
if [[ ! -d "$DIR/.git" ]]; then
|
|
echo "Cloning noai-watermark into $DIR (pinned ref: $REF)"
|
|
git clone --depth 1 --filter=blob:none --sparse \
|
|
https://github.com/mertizci/noai-watermark.git "$DIR"
|
|
git -C "$DIR" fetch --depth 1 origin "$REF"
|
|
git -C "$DIR" checkout --detach "$REF"
|
|
git -C "$DIR" sparse-checkout set --no-cone '/src/'
|
|
HEAD_SHA="$(git -C "$DIR" rev-parse HEAD)"
|
|
if [[ "$HEAD_SHA" != "$REF" ]]; then
|
|
echo "error: expected pinned ref $REF, got $HEAD_SHA" >&2
|
|
exit 1
|
|
fi
|
|
else
|
|
echo "Using existing checkout: $DIR"
|
|
HEAD_SHA="$(git -C "$DIR" rev-parse HEAD 2>/dev/null || true)"
|
|
if [[ "$HEAD_SHA" != "$REF" ]]; then
|
|
echo "existing checkout not at pinned ref $REF (HEAD: ${HEAD_SHA:-missing}); re-pinning"
|
|
git -C "$DIR" fetch --depth 1 origin "$REF" || {
|
|
echo "error: could not fetch pinned ref $REF" >&2
|
|
exit 1
|
|
}
|
|
git -C "$DIR" checkout --detach "$REF"
|
|
git -C "$DIR" sparse-checkout set --no-cone '/src/'
|
|
HEAD_SHA="$(git -C "$DIR" rev-parse HEAD)"
|
|
if [[ "$HEAD_SHA" != "$REF" ]]; then
|
|
echo "error: expected pinned ref $REF, got $HEAD_SHA" >&2
|
|
exit 1
|
|
fi
|
|
fi
|
|
fi
|
|
|
|
if [[ ! -x "$DIR/.venv/bin/python" ]]; then
|
|
echo "Creating venv at $DIR/.venv"
|
|
"$PYTHON" -m venv "$DIR/.venv"
|
|
fi
|
|
|
|
echo "Installing Python dependencies"
|
|
# Pin pip itself (unpinned --upgrade pip was a supply-chain drift point).
|
|
"$DIR/.venv/bin/python" -m pip install --upgrade "pip==26.2.1"
|
|
|
|
# Install torch with the right platform index before the other pinned deps.
|
|
if command -v nvidia-smi >/dev/null 2>&1; then
|
|
cuda="$(nvidia-smi 2>/dev/null | sed -n 's/.*CUDA Version: \([0-9]*\.[0-9]*\).*/\1/p' | head -1)"
|
|
if [[ -n "$cuda" ]]; then
|
|
tag="cu${cuda/./}"
|
|
index="https://download.pytorch.org/whl/$tag"
|
|
echo "NVIDIA GPU detected (CUDA $cuda); installing torch from $index"
|
|
"$DIR/.venv/bin/python" -m pip install torch --index-url "$index"
|
|
else
|
|
echo "nvidia-smi present but no CUDA version found; installing default torch"
|
|
"$DIR/.venv/bin/python" -m pip install torch
|
|
fi
|
|
else
|
|
echo "No NVIDIA GPU detected; installing default torch (CPU/MPS)"
|
|
"$DIR/.venv/bin/python" -m pip install torch
|
|
fi
|
|
|
|
"$DIR/.venv/bin/python" -m pip install -r "$SCRIPT_DIR/requirements-ctrlregen.txt"
|
|
|
|
cat <<EOF
|
|
|
|
Done. Remove a watermark with:
|
|
|
|
export NOAI_WATERMARK_DIR="$DIR"
|
|
"$DIR/.venv/bin/python" "$SCRIPT_DIR/clean_ctrlregen.py" IMAGE -o OUT
|
|
EOF
|