mirror of
https://github.com/guillaumemeyer/watermarks-remover.git
synced 2026-08-22 13:11:57 +02:00
* fix: rewrite ODT/EPUB manifests and measure real zip bytes (#122) Two container correctness/security fixes from issue #122: - clean_odt dropped marker-bearing parts while leaving their entries in META-INF/manifest.xml, so readers flagged the package as damaged. It is now two-pass: compute the dropped set, then rewrite the manifest attribute-order-independently, and write each part exactly once. The same bug class in clean_epub (dropped parts left in the OPF manifest, plus dangling spine itemrefs) gets the same two-pass treatment. - The zip budget trusted ZipInfo.file_size from the archive's own central directory, so a crafted DOCX/ODT could declare a tiny size and still expand via zf.read. Budgets are now charged on actual decompressed bytes via _read_zip_member (streaming, cap enforced mid-read), with the declared size kept only as a fast-path pre-reject. * fix: classify unrecognized bytes as "unknown", not text (#122) Two classification defects from issue #122: - format_dispatch.classify_bytes fell back to "text" for any unrecognized file, so a binary with valid UTF-8 runs could be decoded and written back mangled (corrupted with --in-place) in clean_file auto mode. Unrecognized bytes now classify as "unknown"; clean_file refuses them in auto mode (exit 2, no write, router advice) and --as text / --force-text are the explicit opt-ins. inspect_file reports kind "unknown" (exit 0), audit_lib records a non-actionable item, and the HTTP server answers /inspect with kind "unknown" but rejects /clean of unknown formats (400). - classify(path) read the whole file to sniff a header, and only a full read could detect zip containers. It now routes known extensions without reading, sniffs a 4096-byte header once for images and prefix-based containers, and reads the whole file only when the header is a zip local header (PK), where the container signature lives in the central directory. * feat: distinct exit code for partial audits (#122) audit_dir and audit_website reported success (0) even when some files or URLs could not be scanned; the exit status was computed only over the items that succeeded. A scan that is missing items is not a clean scan. - common.EXIT_PARTIAL = 3, with precedence: partial (3) > actionable (1) > clean (0) — an incomplete audit is the more important CI signal. - audit_dir returns 3 when any file was skipped/failed; audit_website returns 3 when any URL failed to fetch or inspect. Both are independent of the output format (human/json/sarif already share one return). * fix: verify the pinned upstream ref on existing checkouts (#122) setup_ctrlregen.sh/setup_synthid.sh (and their .ps1 twins) only verified the pinned commit in the fresh-clone branch; an existing checkout at an unknown or drifted revision was silently reused, defeating the commit pin. All four scripts now check HEAD against the pinned ref in the existing-checkout branch too, and repair by fetch + detach checkout (re-applying the sparse-checkout set), failing hard if the ref cannot be reached or the re-pin does not land on it. * docs: unknown-format behavior, audit exit codes, backend isolation (#122) - README: clean_file no longer auto-cleans unrecognized formats (--as text / --force-text are the opt-ins), and the CtrlRegen bootstrap documents the isolation expectation for its research-era dependency pins plus the new re-pin check on existing checkouts. - SKILL.md: audit exit codes (0/1/2/3, partial=3) and a note that /clean requires a name with a known extension. - audit_website: document why stdlib ElementTree is used (stdlib-first) and that defusedxml is the fallback if that policy changes (DTD rejection stays). - requirements-ctrlregen.txt: advisory/isolation note for the pinned research dependencies. * test: ODT manifest and EPUB OPF dangling-ref regressions (#122) - clean_odt: dropped marker-bearing parts remove their META-INF/manifest.xml file-entry (attribute-order-independent), exactly one manifest entry, root and surviving entries kept, and the manifest is byte-identical when nothing is dropped. - clean_epub: dropped non-content parts lose their <item> entry in the OPF manifest, so the book no longer references removed members.
111 lines
4.5 KiB
PowerShell
111 lines
4.5 KiB
PowerShell
<#
|
|
.SYNOPSIS
|
|
Port a Windows de setup_synthid.sh.
|
|
|
|
.DESCRIPTION
|
|
Bootstrap del checkout externo reverse-SynthID para el scorer opcional de
|
|
pixel. Es SOLO deteccion, no eliminacion.
|
|
|
|
El proyecto upstream (https://github.com/aloshdenny/reverse-SynthID) esta
|
|
bajo una Research License no comercial y NO se empaqueta en este
|
|
repositorio. Este script lo clona en local e instala solo las dependencias
|
|
que necesita score_synthid.py.
|
|
|
|
Diferencia con la version .sh: el venv de Windows vive en .venv\Scripts\
|
|
en vez de .venv/bin/.
|
|
|
|
.PARAMETER Dir
|
|
Directorio del checkout (por defecto: $env:REVERSE_SYNTHID_DIR o ~\reverse-SynthID)
|
|
|
|
.PARAMETER Ref
|
|
Commit a usar (por defecto: el SHA fijado; no apuntes a una rama movil)
|
|
|
|
.PARAMETER Full
|
|
Instala el requirements.txt completo del upstream (anade torch/diffusers
|
|
para el bypass del VAE) en vez de solo lo del scorer.
|
|
|
|
.PARAMETER Python
|
|
Interprete usado para crear el venv (por defecto: python)
|
|
#>
|
|
[CmdletBinding()]
|
|
param(
|
|
[string]$Dir,
|
|
[string]$Ref = 'b11083676fd3ee3ff97ce9d03c0e409e46905902',
|
|
[switch]$Full,
|
|
[string]$Python = 'python'
|
|
)
|
|
|
|
$ErrorActionPreference = 'Stop'
|
|
$ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
|
|
|
|
function Invoke-Checked {
|
|
param([string]$What, [scriptblock]$Block)
|
|
& $Block
|
|
if ($LASTEXITCODE -ne 0) { throw "$What fallo (codigo $LASTEXITCODE)" }
|
|
}
|
|
|
|
if (-not $Dir) {
|
|
if ($env:REVERSE_SYNTHID_DIR) { $Dir = $env:REVERSE_SYNTHID_DIR }
|
|
else { $Dir = Join-Path $HOME 'reverse-SynthID' }
|
|
}
|
|
$parent = Split-Path -Parent $Dir
|
|
if ($parent -and -not (Test-Path $parent)) { New-Item -ItemType Directory -Force -Path $parent | Out-Null }
|
|
if (-not (Test-Path $Dir)) { New-Item -ItemType Directory -Force -Path $Dir | Out-Null }
|
|
$Dir = (Resolve-Path $Dir).Path
|
|
|
|
if (-not (Test-Path (Join-Path $Dir '.git'))) {
|
|
Write-Host "Clonando reverse-SynthID en $Dir (ref fijado: $Ref)"
|
|
Invoke-Checked 'git clone' { git clone --depth 1 --filter=blob:none --sparse https://github.com/aloshdenny/reverse-SynthID.git $Dir }
|
|
Invoke-Checked 'git fetch' { git -C $Dir fetch --depth 1 origin $Ref }
|
|
Invoke-Checked 'git checkout' { git -C $Dir checkout --detach $Ref }
|
|
Invoke-Checked 'sparse-checkout' {
|
|
git -C $Dir sparse-checkout set --no-cone '/src/' '/artifacts/spectral_codebook_v4.npz' '/requirements.txt' '/LICENSE' '/README.md'
|
|
}
|
|
$head = (git -C $Dir rev-parse HEAD).Trim()
|
|
if ($head -ne $Ref) { throw "error: se esperaba el ref fijado $Ref, se obtuvo $head" }
|
|
} else {
|
|
Write-Host "Usando checkout existente: $Dir"
|
|
$head = ''
|
|
try { $head = "$(git -C $Dir rev-parse HEAD)".Trim() } catch { $head = '' }
|
|
if ($head -ne $Ref) {
|
|
Write-Host "checkout existente no esta en el ref fijado $Ref (HEAD: $head); re-fijando"
|
|
Invoke-Checked 'git fetch' { git -C $Dir fetch --depth 1 origin $Ref }
|
|
Invoke-Checked 'git checkout' { git -C $Dir checkout --detach $Ref }
|
|
Invoke-Checked 'sparse-checkout' {
|
|
git -C $Dir sparse-checkout set --no-cone '/src/' '/artifacts/spectral_codebook_v4.npz' '/requirements.txt' '/LICENSE' '/README.md'
|
|
}
|
|
$head = (git -C $Dir rev-parse HEAD).Trim()
|
|
if ($head -ne $Ref) { throw "error: se esperaba el ref fijado $Ref, se obtuvo $head" }
|
|
}
|
|
}
|
|
|
|
$venvPython = Join-Path $Dir '.venv\Scripts\python.exe'
|
|
if (-not (Test-Path $venvPython)) {
|
|
Write-Host "Creando venv en $Dir\.venv"
|
|
Invoke-Checked 'crear venv' { & $Python -m venv (Join-Path $Dir '.venv') }
|
|
}
|
|
|
|
Write-Host 'Instalando dependencias de Python'
|
|
# pip fijado (un --upgrade pip sin pin es un punto de deriva de cadena de suministro).
|
|
Invoke-Checked 'pip install pip' { & $venvPython -m pip install --upgrade 'pip==26.2.1' }
|
|
|
|
if ($Full) {
|
|
Write-Host 'Instalando el requirements.txt completo del upstream (incluye torch/diffusers)'
|
|
Invoke-Checked 'pip install upstream' { & $venvPython -m pip install -r (Join-Path $Dir 'requirements.txt') }
|
|
} else {
|
|
Write-Host 'Instalando solo las dependencias del scorer'
|
|
Invoke-Checked 'pip install scorer' { & $venvPython -m pip install -r (Join-Path $ScriptDir 'requirements-synthid-scorer.txt') }
|
|
}
|
|
|
|
$codebook = Join-Path $Dir 'artifacts\spectral_codebook_v4.npz'
|
|
if (-not (Test-Path $codebook)) {
|
|
Write-Warning "codebook no encontrado en $codebook"
|
|
Write-Warning "ejecuta: git -C '$Dir' sparse-checkout add '/artifacts/spectral_codebook_v4.npz'"
|
|
}
|
|
|
|
Write-Host ''
|
|
Write-Host 'Listo. Para puntuar una imagen:'
|
|
Write-Host ''
|
|
Write-Host " `$env:REVERSE_SYNTHID_DIR = '$Dir'"
|
|
Write-Host " python '$ScriptDir\inspect_image.py' IMAGEN"
|