fix: GPU deployment robustness (6 fixes from end-to-end testing on an RTX 4070) (#334)

* fix(docker): pin CUDA base to 12.6 so the GPU image starts on R560+ drivers

The amd64 base nvidia/cuda:12.9.2-cudnn-runtime bakes a cuda>=12.9 driver gate enforced by nvidia-container-toolkit at container start, so the image fails to launch on common production drivers (e.g. 570.x / CUDA 12.8). The AI bundles are all cu126 wheels and the image installs libcublas-12-6, so 12.9 was misaligned with the workload. Pin to nvidia/cuda:12.6.3-cudnn-runtime-ubuntu24.04 to match the wheels and lower the driver floor to R560+.

* fix(ai): broaden OOM detection so the rembg lighter-model fallback fires

onnxruntime/CUDA allocation failures surface as 'Failed to allocate memory for requested buffer', CUBLAS_STATUS_ALLOC_FAILED, or bad_alloc, not just 'out of memory'. The background-removal and transparency-fixer fallback-to-lighter-model paths only matched the literal 'out of memory', so the fallback was dead code and transparency-fixer (default birefnet-hr-matting) always failed with an allocation error. Add isMemoryAllocError() and use it in both checks.

* fix(ai): use bundled PaddleOCR models so OCR runs offline

ocr.py passed no model dirs to PaddleOCR, so PaddleX resolved models from ~/.paddlex and downloaded them from HuggingFace at runtime (slow first use, broken air-gapped), ignoring the models the OCR bundle ships in MODELS_PATH; it also pulled doc-orientation/unwarping models that are not bundled. Pin detection, recognition and textline models to the bundled dirs in MODELS_PATH (per language) and disable use_doc_orientation_classify / use_doc_unwarping, with per-component fallback when a model is absent. Verified: OCR runs with zero HuggingFace requests.

* fix(docker): add CAP_KILL so container shutdown is graceful

cap_drop: ALL without re-adding KILL meant tini (PID 1, root) could not forward SIGTERM to the gosu-dropped snapotter process (root minus CAP_KILL cannot signal a different UID). docker stop logged '[FATAL tini] forwarding signal: Operation not permitted', never delivered the signal, and fell back to SIGKILL after the 10s timeout. Add KILL to cap_add in both compose files. Verified: docker stop completes in 0s with SIGTERM delivered (exit 143) and no FATAL tini.

* fix(ai): serialize bundle installs against AI jobs to prevent sidecar segfault

A feature bundle install rewrites the shared Python venv (pip + copytree of site-packages/*.so) as a background subprocess, with no coordination against AI tool jobs that dlopen native libs (torch / onnxruntime CUDA) from the same venv; a job loading a shared object while it is overwritten segfaults the sidecar. Add a process-wide async mutex (venv-lock.ts): bridge.run() acquires it before every AI script and the install route holds it across the installer subprocess. Both run in the same Node process so a module-level lock suffices. Verified: concurrent install + AI job produces zero segfaults and the job serializes behind the install.

* fix(ai): make the venv lock read/write so concurrent AI jobs are not serialized

The first cut used an exclusive mutex, which (a) deferred the dispatcher spawn by a microtask and broke unit tests that synchronously drive the mocked spawn, and (b) serialized AI jobs against each other, removing the dispatcher's by-id request multiplexing. Make it a writer-preferring read/write lock: AI jobs are shared readers (with a synchronous fast path so spawn still happens in-tick) and a bundle install is the exclusive writer. Verified: all 764 AI unit tests pass.

* fix(ai): degrade OCR to Tesseract on CPU-only hosts instead of segfaulting

The amd64 AI bundle ships paddlepaddle-gpu, whose native libs dlopen
libcuda.so.1 at import and segfault on a host without a GPU (libcuda is the
driver lib, injected only by nvidia-container-toolkit on GPU hosts). The
segfault crashed the shared long-lived AI dispatcher and, after a few attempts,
tripped the bridge crash-recovery permanent-disable, wedging all AI until a
container restart. The standalone ocr tool defaults to quality=balanced
(PaddleOCR), so it hit this on every CPU-only deployment; ocr-pdf already
hardcoded Tesseract and was unaffected.

ocr.py now gates the PaddleOCR tiers on gpu_available(): balanced/best
transparently fall back to fast (Tesseract, CPU-capable) when no usable GPU is
present, and run_paddleocr_v5/run_paddleocr_vl refuse before importing paddle so
the GPU build is never dlopen'd on CPU. GPU hosts are unchanged.

Verified on a CPU-only Windows/WSL2 box: ocr returns Tesseract text across
repeated runs with the dispatcher staying healthy (no wedge).

apps/api/src/routes/features.ts

@@ -20,7 +20,7 @@ import {
   unlinkSync,
 } from "node:fs";
 import { join } from "node:path";
-import { shutdownDispatcher } from "@snapotter/ai";
+import { acquireVenvLock, shutdownDispatcher } from "@snapotter/ai";
 import { ANALYTICS_EVENTS, FEATURE_BUNDLES } from "@snapotter/shared";
 import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
 import { trackEvent } from "../lib/analytics.js";
@@ -167,6 +167,19 @@ export async function registerFeatureRoutes(app: FastifyInstance): Promise<void>
       const installStartTime = Date.now();
       const reqRef = request;
 
+      // Hold the venv lock across the whole install so no AI tool job loads
+      // native libs from the venv while pip is rewriting them (that segfaults
+      // the sidecar). This awaits any in-flight AI job before the installer
+      // starts; the lock is released when the installer process exits.
+      const releaseVenv = await acquireVenvLock();
+      let venvReleased = false;
+      const releaseVenvOnce = () => {
+        if (!venvReleased) {
+          venvReleased = true;
+          releaseVenv();
+        }
+      };
+
       const child = spawn(pythonPath, [scriptPath, bundleId, manifestPath, modelsDir], {
         stdio: ["ignore", "pipe", "pipe"],
         env: {
@@ -219,6 +232,7 @@ export async function registerFeatureRoutes(app: FastifyInstance): Promise<void>
       });
 
       child.on("close", (code) => {
+        releaseVenvOnce();
         releaseInstallLock();
 
         if (code === 0) {
@@ -275,6 +289,7 @@ export async function registerFeatureRoutes(app: FastifyInstance): Promise<void>
       });
 
       child.on("error", (err) => {
+        releaseVenvOnce();
         releaseInstallLock();
         const errorMsg = `Failed to spawn install process: ${err.message}`;
         setInstallProgress(bundleId, null, errorMsg);

apps/api/src/routes/tools/transparency-fixer.ts

@@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto";
 import { mkdir, rm } from "node:fs/promises";
 import { tmpdir } from "node:os";
 import { join } from "node:path";
-import { removeBackground } from "@snapotter/ai";
+import { isMemoryAllocError, removeBackground } from "@snapotter/ai";
 import { getBundleForTool, TOOL_BUNDLE_MAP } from "@snapotter/shared";
 import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
 import sharp from "sharp";
@@ -104,7 +104,7 @@ async function processTransparencyFix(
       onProgress,
     );
   } catch (err) {
-    const isOom = err instanceof Error && err.message.includes("out of memory");
+    const isOom = isMemoryAllocError(err);
     if (!isOom) throw err;
 
     onProgress?.(5, `Retrying with fallback model (${FALLBACK_MODEL})`);