Files
Guillaume Meyer (The Opinionated Man)andGitHub 6df80e77a4 fix: correctness and security hardening of the cleaning scripts (#122) (#126)
* fix: rewrite ODT/EPUB manifests and measure real zip bytes (#122)

Two container correctness/security fixes from issue #122:

- clean_odt dropped marker-bearing parts while leaving their entries in
  META-INF/manifest.xml, so readers flagged the package as damaged. It is
  now two-pass: compute the dropped set, then rewrite the manifest
  attribute-order-independently, and write each part exactly once. The same
  bug class in clean_epub (dropped parts left in the OPF manifest, plus
  dangling spine itemrefs) gets the same two-pass treatment.
- The zip budget trusted ZipInfo.file_size from the archive's own central
  directory, so a crafted DOCX/ODT could declare a tiny size and still
  expand via zf.read. Budgets are now charged on actual decompressed bytes
  via _read_zip_member (streaming, cap enforced mid-read), with the declared
  size kept only as a fast-path pre-reject.

* fix: classify unrecognized bytes as "unknown", not text (#122)

Two classification defects from issue #122:

- format_dispatch.classify_bytes fell back to "text" for any unrecognized
  file, so a binary with valid UTF-8 runs could be decoded and written back
  mangled (corrupted with --in-place) in clean_file auto mode. Unrecognized
  bytes now classify as "unknown"; clean_file refuses them in auto mode
  (exit 2, no write, router advice) and --as text / --force-text are the
  explicit opt-ins. inspect_file reports kind "unknown" (exit 0), audit_lib
  records a non-actionable item, and the HTTP server answers /inspect with
  kind "unknown" but rejects /clean of unknown formats (400).
- classify(path) read the whole file to sniff a header, and only a full read
  could detect zip containers. It now routes known extensions without
  reading, sniffs a 4096-byte header once for images and prefix-based
  containers, and reads the whole file only when the header is a zip local
  header (PK), where the container signature lives in the central directory.

* feat: distinct exit code for partial audits (#122)

audit_dir and audit_website reported success (0) even when some files or
URLs could not be scanned; the exit status was computed only over the items
that succeeded. A scan that is missing items is not a clean scan.

- common.EXIT_PARTIAL = 3, with precedence: partial (3) > actionable (1)
  > clean (0) — an incomplete audit is the more important CI signal.
- audit_dir returns 3 when any file was skipped/failed; audit_website
  returns 3 when any URL failed to fetch or inspect. Both are independent
  of the output format (human/json/sarif already share one return).

* fix: verify the pinned upstream ref on existing checkouts (#122)

setup_ctrlregen.sh/setup_synthid.sh (and their .ps1 twins) only verified
the pinned commit in the fresh-clone branch; an existing checkout at an
unknown or drifted revision was silently reused, defeating the commit pin.

All four scripts now check HEAD against the pinned ref in the
existing-checkout branch too, and repair by fetch + detach checkout
(re-applying the sparse-checkout set), failing hard if the ref cannot be
reached or the re-pin does not land on it.

* docs: unknown-format behavior, audit exit codes, backend isolation (#122)

- README: clean_file no longer auto-cleans unrecognized formats (--as text
  / --force-text are the opt-ins), and the CtrlRegen bootstrap documents the
  isolation expectation for its research-era dependency pins plus the new
  re-pin check on existing checkouts.
- SKILL.md: audit exit codes (0/1/2/3, partial=3) and a note that /clean
  requires a name with a known extension.
- audit_website: document why stdlib ElementTree is used (stdlib-first) and
  that defusedxml is the fallback if that policy changes (DTD rejection stays).
- requirements-ctrlregen.txt: advisory/isolation note for the pinned research
  dependencies.

* test: ODT manifest and EPUB OPF dangling-ref regressions (#122)

- clean_odt: dropped marker-bearing parts remove their META-INF/manifest.xml
  file-entry (attribute-order-independent), exactly one manifest entry, root
  and surviving entries kept, and the manifest is byte-identical when nothing
  is dropped.
- clean_epub: dropped non-content parts lose their <item> entry in the OPF
  manifest, so the book no longer references removed members.
2026-08-17 17:15:35 -07:00

111 lines
4.5 KiB
PowerShell

<#
.SYNOPSIS
Port a Windows de setup_synthid.sh.
.DESCRIPTION
Bootstrap del checkout externo reverse-SynthID para el scorer opcional de
pixel. Es SOLO deteccion, no eliminacion.
El proyecto upstream (https://github.com/aloshdenny/reverse-SynthID) esta
bajo una Research License no comercial y NO se empaqueta en este
repositorio. Este script lo clona en local e instala solo las dependencias
que necesita score_synthid.py.
Diferencia con la version .sh: el venv de Windows vive en .venv\Scripts\
en vez de .venv/bin/.
.PARAMETER Dir
Directorio del checkout (por defecto: $env:REVERSE_SYNTHID_DIR o ~\reverse-SynthID)
.PARAMETER Ref
Commit a usar (por defecto: el SHA fijado; no apuntes a una rama movil)
.PARAMETER Full
Instala el requirements.txt completo del upstream (anade torch/diffusers
para el bypass del VAE) en vez de solo lo del scorer.
.PARAMETER Python
Interprete usado para crear el venv (por defecto: python)
#>
[CmdletBinding()]
param(
[string]$Dir,
[string]$Ref = 'b11083676fd3ee3ff97ce9d03c0e409e46905902',
[switch]$Full,
[string]$Python = 'python'
)
$ErrorActionPreference = 'Stop'
$ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
function Invoke-Checked {
param([string]$What, [scriptblock]$Block)
& $Block
if ($LASTEXITCODE -ne 0) { throw "$What fallo (codigo $LASTEXITCODE)" }
}
if (-not $Dir) {
if ($env:REVERSE_SYNTHID_DIR) { $Dir = $env:REVERSE_SYNTHID_DIR }
else { $Dir = Join-Path $HOME 'reverse-SynthID' }
}
$parent = Split-Path -Parent $Dir
if ($parent -and -not (Test-Path $parent)) { New-Item -ItemType Directory -Force -Path $parent | Out-Null }
if (-not (Test-Path $Dir)) { New-Item -ItemType Directory -Force -Path $Dir | Out-Null }
$Dir = (Resolve-Path $Dir).Path
if (-not (Test-Path (Join-Path $Dir '.git'))) {
Write-Host "Clonando reverse-SynthID en $Dir (ref fijado: $Ref)"
Invoke-Checked 'git clone' { git clone --depth 1 --filter=blob:none --sparse https://github.com/aloshdenny/reverse-SynthID.git $Dir }
Invoke-Checked 'git fetch' { git -C $Dir fetch --depth 1 origin $Ref }
Invoke-Checked 'git checkout' { git -C $Dir checkout --detach $Ref }
Invoke-Checked 'sparse-checkout' {
git -C $Dir sparse-checkout set --no-cone '/src/' '/artifacts/spectral_codebook_v4.npz' '/requirements.txt' '/LICENSE' '/README.md'
}
$head = (git -C $Dir rev-parse HEAD).Trim()
if ($head -ne $Ref) { throw "error: se esperaba el ref fijado $Ref, se obtuvo $head" }
} else {
Write-Host "Usando checkout existente: $Dir"
$head = ''
try { $head = "$(git -C $Dir rev-parse HEAD)".Trim() } catch { $head = '' }
if ($head -ne $Ref) {
Write-Host "checkout existente no esta en el ref fijado $Ref (HEAD: $head); re-fijando"
Invoke-Checked 'git fetch' { git -C $Dir fetch --depth 1 origin $Ref }
Invoke-Checked 'git checkout' { git -C $Dir checkout --detach $Ref }
Invoke-Checked 'sparse-checkout' {
git -C $Dir sparse-checkout set --no-cone '/src/' '/artifacts/spectral_codebook_v4.npz' '/requirements.txt' '/LICENSE' '/README.md'
}
$head = (git -C $Dir rev-parse HEAD).Trim()
if ($head -ne $Ref) { throw "error: se esperaba el ref fijado $Ref, se obtuvo $head" }
}
}
$venvPython = Join-Path $Dir '.venv\Scripts\python.exe'
if (-not (Test-Path $venvPython)) {
Write-Host "Creando venv en $Dir\.venv"
Invoke-Checked 'crear venv' { & $Python -m venv (Join-Path $Dir '.venv') }
}
Write-Host 'Instalando dependencias de Python'
# pip fijado (un --upgrade pip sin pin es un punto de deriva de cadena de suministro).
Invoke-Checked 'pip install pip' { & $venvPython -m pip install --upgrade 'pip==26.2.1' }
if ($Full) {
Write-Host 'Instalando el requirements.txt completo del upstream (incluye torch/diffusers)'
Invoke-Checked 'pip install upstream' { & $venvPython -m pip install -r (Join-Path $Dir 'requirements.txt') }
} else {
Write-Host 'Instalando solo las dependencias del scorer'
Invoke-Checked 'pip install scorer' { & $venvPython -m pip install -r (Join-Path $ScriptDir 'requirements-synthid-scorer.txt') }
}
$codebook = Join-Path $Dir 'artifacts\spectral_codebook_v4.npz'
if (-not (Test-Path $codebook)) {
Write-Warning "codebook no encontrado en $codebook"
Write-Warning "ejecuta: git -C '$Dir' sparse-checkout add '/artifacts/spectral_codebook_v4.npz'"
}
Write-Host ''
Write-Host 'Listo. Para puntuar una imagen:'
Write-Host ''
Write-Host " `$env:REVERSE_SYNTHID_DIR = '$Dir'"
Write-Host " python '$ScriptDir\inspect_image.py' IMAGEN"