Two latent bugs the v2.2.0 release run surfaced, both added by #649 and never run
in a real release (v2.1.0 had no archive-security job).
archive-security asserted tsx at the workspace root, but tsx is a prod dependency
of apps/api, so pnpm places its bin at apps/api/node_modules/.bin/tsx, where the
Docker CMD runs it. The root path never existed and failed the extract step on
both arches. Fixed to the apps/api path, proven against the real prebuilt-amd64
artifact.
The blocking Trivy scans would then have failed on CVE-2026-56852,
golang.org/x/text v0.38.0 -> v0.39.0, the only fixed CRITICAL/HIGH in the image,
compiled into caire and pdfcpu. Pinned to v0.39.0 in both go-tools modules,
verified building in the golang:1.25.12 toolchain with -mod=readonly and linking
v0.39.0, with a clean Trivy rescan.
Guards added for both the tsx path and the x/text pin. Non-releasable type so a
re-dispatch re-runs 2.2.0.
Two release-pipeline defects found while pre-flighting 2.2.0, plus the image
hardening that the second one exposed.
The release job would have died immediately after pushing the v2.2.0 tag.
draftRelease was turned on in #649 and never executed, and GitHub's
/releases/tags/{tag} endpoint does not return draft releases, so all nine tag
lookups in release.yml would have 404'd against the draft semantic-release had
just created. Verified against this repo with a throwaway draft: the tag
endpoint 404s while gh release view reads it and /releases/{id} returns the same
REST shape. Every site now resolves the numeric id first, so existing jq
expressions are untouched.
The unfixed-vulnerability gate was measuring almost nothing. The blocking Trivy
steps run ignore-unfixed, and trivy-unfixed-gate.mjs was meant to cover the
remainder but defaults to CRITICAL with neither call site passing --severity. An
unfixed HIGH was gated by nothing, and the arm64 image carried 79 of them while
the summary read clean.
Rather than document 79 findings, the image lost what it did not need:
libde265 1.1.1 and libheif 1.23.1 are now built from source (the old libheif pin
was itself affected by CVE-2026-3950, and Debian's libde265 1.0.11 was the
decoder every .heic upload actually reached), and xvfb, wget and openssh-client
are purged. 15 CVEs left the image outright and the HIGH gap fell to 65, each
now carrying a rationale verified against the running container.
curl gets its own section: bookworm-backports has a fixed 8.14.1, so claiming no
fix was available would have been false. It is recorded as a declined fix.
Verified on both architectures: gate exits 0, the source-built libde265 is the
one libheif links, and HEIC, RAW, ImageMagick, Sharp AVIF and headless chromium
all still work after the purge.
A release-readiness QA pass over the whole product. The commits split into
defects a user would hit and gates that were reporting green while measuring
nothing.
## Fixes that change behaviour
Rate limiting was bypassable on every install: TRUST_PROXY defaulted to true, so
request.ip came from a client-set header and a forged X-Forwarded-For got past
the login limiter. The default is now a private-network trust list.
A transient Postgres outage stranded in-flight jobs, leaving finished output on
disk with no row pointing at it. A reconciler now resolves those rows and adopts
the bytes rather than dropping the work.
A Redis connection that moved to a new address wedged every read-blocked
consumer, so completions stopped signalling while health still answered 200.
Socket timeouts plus subscriber pings recover it.
Installing more than one AI bundle left the shared venv multi-versioned and
silently broke three tools. The installer now reconciles distributions to one
version each.
Converting an image to JXL at quality 1 through 4 returned a 500, because
libjxl 0.7 rejects the distance those values compute. The quality is floored at
what the encoder honours. A missing ffmpeg was also reported to the user as a
corrupt upload; it now says the engine is unavailable.
RAW uploads reached an unpatched LibRaw on arm64, so it is built from source at
0.22.2, and the release scan was split so it can fail on an unfixed critical
instead of hiding it behind ignore-unfixed.
## Gates that could not fail
Two mutation lanes ran zero mutants because Stryker crawled the gitignored docs
build; coverage discarded its whole report on any failing test; the lint gate
skipped root tests, scripts, and two workspaces; and several generated matrices
counted a host missing ffmpeg as a passing tool. Each now measures what it
claims.
Full evidence and the outstanding release items are tracked locally and are not
part of this branch.
Follow-up to a full re-audit of the 2.0 tree. Most prior findings were already
fixed; this closes the ones that were not:
- SAML assertion replay: validateInResponseTo ifPresent plus a Redis-backed
CacheProvider, so a captured signed assertion cannot be replayed. ifPresent
keeps IdP-initiated SSO working.
- MFA login challenge burned after 5 wrong TOTP codes.
- api_keys.key_prefix indexed; the per-request lookup was a full table scan.
- MAX_AI_JOBS_PER_USER caps a user's in-flight single-file AI jobs (the AI pool
runs at concurrency 1). Batch and pipeline AI stay uncapped.
- MAX_WORKSPACE_SIZE_GB enforced instead of being dead config.
- SUBPROCESS_MEMORY_LIMIT_MB (default off) for the native media and doc engines;
not applied to the AI sidecar.
- SVG sanitizer closes unquoted and whitespace-prefixed javascript: hrefs and
the animateTransform/animateMotion/handler/mpath elements.
- Windows-style paths stripped from error output to match the Sentry scrubber.
- Postgres and Redis compose services get cap_drop plus pids_limit and cpus.
- .env.example ships MAX_SVG_SIZE_MB=50 (0 disabled the cap).
Adds security-focused unit and integration tests. typecheck, biome, and the
full unit and integration suites pass.
Pin the compose Postgres healthchecks to POSTGRES_DB (pg_isready was
defaulting to the username, silently reporting healthy while spamming
FATAL logs when USER and DB differ), and make docker/wait-for-postgres.mjs
log the target host and error code instead of a silent retry loop. Adds a
change-me note next to the default password in README and Docker Hub.
Refs #592
* fix(api): prevent a crash when an over-limit upload stream has no consumer yet
busboy's "limit" handler destroyed the file stream with an error but never
attached its own error listener, relying entirely on whatever consumes
part.file downstream to do so. On a fast enough connection (or a fully
buffered body, e.g. Fastify inject()), busboy can process enough bytes to
hit the size limit before the route handler's receiveUpload() call has
attached its own stream listener, leaving the resulting "error" event with
zero listeners -- which crashes the whole process by default in Node.
Surfaced by tonight's FULL_MATRIX+FUZZ integration run (880 uncaught
exceptions, all the same root cause). Reproduces deterministically in
isolation; unrelated to this release's actual code delta (file untouched
since PR #413, well before the baseline QA pass).
Fix: attach a baseline no-op error listener the moment the stream is
created, guaranteeing at least one listener always exists. EventEmitter
delivers "error" to every registered listener, so the real consumer's own
error handling is unaffected.
* fix(ai-bundles): rebuild upscale-enhance and photo-restoration to reconcile scipy ABI
upscale-enhance and photo-restoration both depend on codeformer-pip, whose
transitive closure (basicsr -> realesrgan -> gfpgan) pulls in an unpinned
scipy. Both bundles were last built ~June 18-19, before PR #437 added the
manifest's `constraints` array (numpy==1.26.4, scipy==1.12.0, etc.) to pin
exactly this kind of dependency during bundle builds. Only the ocr bundle
was rebuilt after that fix landed.
install_feature.py has no pip install step -- it's a raw tarfile extraction
with no cross-bundle conflict resolution, so installing OCR alongside either
stale bundle left three incompatible scipy versions' files mixed in the same
site-packages directory (a compiled _rotation.*.so from one release next to
Python files expecting a different release's API), breaking the `upscale`
tool and OCR's higher-quality tiers with an ImportError.
Rebuilt both bundles for amd64-gpu and arm64-cpu from the current manifest,
verified scipy/scikit-learn/scikit-image/pandas all resolve to the pinned
versions in the tarballs themselves, then verified end-to-end on real
hardware (Mac arm64 CPU and ubuntu_gpu .248 RTX 4070): installing all
affected bundles together now yields exactly one version of each constrained
package, `upscale` produces correct output, and OCR's balanced/best tiers
correctly use PaddleOCR-GPU instead of erroring out.
Published the rebuilt tarballs to the public deepsafe/feature-bundles
HuggingFace repo and updated this manifest's sha256/sizes to match.
Also adds verify-bundle-compatibility.sh: verify-bundle.sh checks each
bundle in isolation (a fresh venv per bundle), which is exactly why this
shipped twice -- nothing ever checked that bundles built at different times
agree once layered into the one shared venv real installs use. The new
script installs every bundle for an arch into one venv and asserts each
constrained package has exactly one, correct version.
Known follow-up (not fixed here, needs separate discussion): uninstalling a
bundle only removes its downloaded model weights, never the site-packages
it added, so existing installations that already hit this bug have no clean
self-service fix via uninstall+reinstall -- they need a full AI-venv wipe.
* fix(docker): bake a real rate limit default for the all-in-one one-liner
The documented single-container `docker run` install had RATE_LIMIT_PER_MIN=0
(effectively unlimited, ~50k/min) baked in, since only docker-compose.yml
carried a hardened override. A self-hoster following the one-liner path got
no meaningful throttling anywhere, including auth-adjacent routes with no
dedicated per-route limit. Bakes a generous-but-real 1000/min default into
the Dockerfile, raises both compose files' fallback to match so the two
documented install paths converge on the same posture, and updates the Zod
schema default plus docs that quoted the old value.
* fix(api): boot log undercounted tool routes by the conversion-preset total
The "Tool routes: N active" line logged before registerConversionPresets(app)
ran, so it only ever reported the base 158 tools, 83 short of the real
241-tool total. Presets have to register after the base loop (they delegate
to each base tool's own processV2), so the fix moves the log line to after
that call and has registerConversionPresets return its count instead of
reordering the dependency.
* fix(ai): forward {info}/{warning} stderr JSON instead of dropping it
The dispatcher stderr parser only recognized {ready} and {progress,stage}
shaped JSON lines; anything else that parsed as valid JSON (like ocr.py's
GPU-to-tesseract downgrade notice, an {"info": ...} line) matched neither
branch and fell through silently, never reaching docker logs. Adds explicit
{info}/{warning} handling that forwards to console.log/console.warn, same as
the existing [prefix]-tagged non-JSON path.
* fix(api): fall back to a lower OCR tier when PaddleOCR itself is unusable
ocr.ts already retries lower quality tiers on a crashed dispatcher, but the
condition only matched crash-style messages (segfault, exited unexpectedly).
ocr.py's own ImportError/exception handlers already produce messages telling
the caller to use a lower tier (e.g. on the scipy ABI conflict class of bug),
but nothing ever acted on them, so a broken PaddleOCR hard-failed with 422
instead of degrading to Tesseract like ocr-pdf effectively does. Broadens the
retry condition to also catch PaddleOCR-engine-unusable messages.
Note: ocr-pdf's tesseract-only behavior turned out to be an unrelated,
pre-existing, deliberate design choice (PaddleOCR segfaults on rasterized PDF
pages on arm64), not a graceful-fallback mechanism to copy -- the two tools
weren't actually solving the same problem, so this fixes ocr.ts's own gap
rather than trying to mirror ocr-pdf.
AI feature installs now keep copied Python venv metadata (bin/pip shebang,
bin/activate, pyvenv.cfg) pointed at /data/ai/venv, so scripts no longer
silently fall back to the baked, read-only /opt/venv after the venv is
bootstrapped into /data. Fixes#127 (AI tools incompatible with PUID/PGID).
The entrypoint repairs both fresh bootstraps and already-stamped runtime
venvs (self-heals existing deployments on next restart, no reinstall
needed), with regression coverage for literal path replacement and binary
file safety.
Independently reviewed and verified: traced chown/gosu ordering in
entrypoint.sh to confirm no permission regression, reproduced the exact
issue #127 scenario (custom PUID + manual venv activation) in a live
container both before and after the fix, and ran the PR's own test suite
locally (16/16 passing).
Co-authored-by: SyntaxSawdust
Embedded Postgres 17 + Redis via s6-overlay when DATABASE_URL/REDIS_URL are unset; restores the one-command docker run for 2.0. EMBEDDED=0 disables; Compose stays the production path. Verified arm64 (14/14 lifecycle + Compose regression) and amd64 (build + embedded smoke).
The entrypoint only fixed volume permissions when started as root (chown +
gosu-drop to snapotter). Launched under a non-root/foreign UID (TrueNAS app
user, Kubernetes runAsUser, OpenShift) it did no permission setup, so /data and
/tmp/workspace -- owned by uid 999 from the image -- were not writable by the
running user. Uploads and processing then failed with a cryptic EACCES
("workspace folder is not writable") and AI bundle installs failed the same way,
while health checks still reported the container healthy.
- entrypoint: source new entrypoint-lib.sh; verify writability up front when
non-root, and as snapotter after chown when root (catches root-squashed
mounts), failing fast with an actionable message (which dir, uid/gid, how to
fix) instead of a late, cryptic EACCES
- Dockerfile: own /data and /tmp/workspace as snapotter:0, group-writable with
setgid, so an arbitrary UID with the root supplementary group (OpenShift /
Kubernetes fsGroup) can write; keep /opt/venv world-readable for the AI venv
bootstrap under arbitrary UIDs
- api: assert storage writability at boot (lib/storage-writable.ts), failing
fast with the same guidance even when the entrypoint is bypassed
- docs: add a Storage permissions section (named volumes, bind mounts, TrueNAS,
Kubernetes/OpenShift) and cross-link it from the security guide
Fixes#230
