3 Commits

Include renames

Author	SHA1	Message	Date
SnapOtterandGitHub	7cd514dd8c	fix(ai): detect a paddle-only GPU so OCR uses PaddleOCR-GPU not Tesseract (#439) ... gpu_available() probed torch, then ONNX Runtime, then nvidia-smi, but never paddle. The OCR bundle ships paddlepaddle-gpu with no torch or ONNX, so on an OCR-only GPU host every probe missed the GPU: nvidia-smi saw it but returned False by design, and OCR silently fell back to Tesseract (CPU, lower quality) with no signal why. Add a paddle probe as the last resort in gpu_available(). It runs only after nvidia-smi confirms a GPU is physically present, and in an isolated subprocess, because importing paddlepaddle-gpu on a GPU-less host segfaults and would wedge the shared AI dispatcher. It returns True only when paddle reports both a CUDA build and a visible device, signalling the result through the exit code so paddle's own import chatter on stdout cannot corrupt the reading. CPU-only and torch/ONNX GPU hosts are unaffected: the probe never runs on the former (nvidia-smi finds nothing) and is never reached on the latter (the torch step already returns True first). Claude-Session: https://claude.ai/code/session_01NfaRxjek8ex5nawvx3mVMf	2026-07-05 22:57:36 +08:00
SnapOtterandGitHub	35e18d8b79	fix: GPU deployment robustness (6 fixes from end-to-end testing on an RTX 4070) (#334) ... * fix(docker): pin CUDA base to 12.6 so the GPU image starts on R560+ drivers The amd64 base nvidia/cuda:12.9.2-cudnn-runtime bakes a cuda>=12.9 driver gate enforced by nvidia-container-toolkit at container start, so the image fails to launch on common production drivers (e.g. 570.x / CUDA 12.8). The AI bundles are all cu126 wheels and the image installs libcublas-12-6, so 12.9 was misaligned with the workload. Pin to nvidia/cuda:12.6.3-cudnn-runtime-ubuntu24.04 to match the wheels and lower the driver floor to R560+. * fix(ai): broaden OOM detection so the rembg lighter-model fallback fires onnxruntime/CUDA allocation failures surface as 'Failed to allocate memory for requested buffer', CUBLAS_STATUS_ALLOC_FAILED, or bad_alloc, not just 'out of memory'. The background-removal and transparency-fixer fallback-to-lighter-model paths only matched the literal 'out of memory', so the fallback was dead code and transparency-fixer (default birefnet-hr-matting) always failed with an allocation error. Add isMemoryAllocError() and use it in both checks. * fix(ai): use bundled PaddleOCR models so OCR runs offline ocr.py passed no model dirs to PaddleOCR, so PaddleX resolved models from ~/.paddlex and downloaded them from HuggingFace at runtime (slow first use, broken air-gapped), ignoring the models the OCR bundle ships in MODELS_PATH; it also pulled doc-orientation/unwarping models that are not bundled. Pin detection, recognition and textline models to the bundled dirs in MODELS_PATH (per language) and disable use_doc_orientation_classify / use_doc_unwarping, with per-component fallback when a model is absent. Verified: OCR runs with zero HuggingFace requests. * fix(docker): add CAP_KILL so container shutdown is graceful cap_drop: ALL without re-adding KILL meant tini (PID 1, root) could not forward SIGTERM to the gosu-dropped snapotter process (root minus CAP_KILL cannot signal a different UID). docker stop logged '[FATAL tini] forwarding signal: Operation not permitted', never delivered the signal, and fell back to SIGKILL after the 10s timeout. Add KILL to cap_add in both compose files. Verified: docker stop completes in 0s with SIGTERM delivered (exit 143) and no FATAL tini. * fix(ai): serialize bundle installs against AI jobs to prevent sidecar segfault A feature bundle install rewrites the shared Python venv (pip + copytree of site-packages/*.so) as a background subprocess, with no coordination against AI tool jobs that dlopen native libs (torch / onnxruntime CUDA) from the same venv; a job loading a shared object while it is overwritten segfaults the sidecar. Add a process-wide async mutex (venv-lock.ts): bridge.run() acquires it before every AI script and the install route holds it across the installer subprocess. Both run in the same Node process so a module-level lock suffices. Verified: concurrent install + AI job produces zero segfaults and the job serializes behind the install. * fix(ai): make the venv lock read/write so concurrent AI jobs are not serialized The first cut used an exclusive mutex, which (a) deferred the dispatcher spawn by a microtask and broke unit tests that synchronously drive the mocked spawn, and (b) serialized AI jobs against each other, removing the dispatcher's by-id request multiplexing. Make it a writer-preferring read/write lock: AI jobs are shared readers (with a synchronous fast path so spawn still happens in-tick) and a bundle install is the exclusive writer. Verified: all 764 AI unit tests pass. * fix(ai): degrade OCR to Tesseract on CPU-only hosts instead of segfaulting The amd64 AI bundle ships paddlepaddle-gpu, whose native libs dlopen libcuda.so.1 at import and segfault on a host without a GPU (libcuda is the driver lib, injected only by nvidia-container-toolkit on GPU hosts). The segfault crashed the shared long-lived AI dispatcher and, after a few attempts, tripped the bridge crash-recovery permanent-disable, wedging all AI until a container restart. The standalone ocr tool defaults to quality=balanced (PaddleOCR), so it hit this on every CPU-only deployment; ocr-pdf already hardcoded Tesseract and was unaffected. ocr.py now gates the PaddleOCR tiers on gpu_available(): balanced/best transparently fall back to fast (Tesseract, CPU-capable) when no usable GPU is present, and run_paddleocr_v5/run_paddleocr_vl refuse before importing paddle so the GPU build is never dlopen'd on CPU. GPU hosts are unchanged. Verified on a CPU-only Windows/WSL2 box: ocr returns Tesseract text across repeated runs with the dispatcher staying healthy (no wedge).	2026-06-23 18:39:51 +08:00
SnapOtterandGitHub	3fb8164fa5	feat: add OpenTelemetry distributed tracing (enterprise) (#232) ... * feat(tracing): add OpenTelemetry dependencies and --import preload flag * feat(enterprise): add distributed_tracing feature gate * feat(tracing): add SDK bootstrap with enterprise gating * fix(tracing): correct test coverage for enterprise-unavailable path and prevent double-init Test 2 now mocks @snapotter/enterprise to throw an import error, exercising the catch block in the preload. Test 3 imports with no endpoint so the preload is a no-op, avoiding leaked SDK from double-initialization. Added idempotency guard to initTracing() as a safety net. * feat(tracing): add Pino trace mixin and shared logger When OTel tracing is active, every Pino log line now includes traceId, spanId, and traceFlags fields for log-to-trace correlation. The mixin is a no-op when no SDK is registered (community users). * feat(tracing): add _otel to ToolJobData and inject trace context at enqueue Add optional _otel carrier field to ToolJobData for W3C trace context propagation across BullMQ job boundaries. When an active OTel span exists, propagation.inject() writes traceparent/tracestate into the job data before queue.add(). When no SDK is registered (community edition), the carrier stays empty and _otel remains undefined -- zero overhead. * feat(tracing): extract trace context and create spans in BullMQ worker * feat(tracing): inject trace context into Python sidecar calls * feat(tracing): add trace context extraction to Python sidecar * feat(tracing): add shutdownTracing to graceful shutdown sequence * feat(tracing): enrich HTTP spans with tool_id and user_id attributes * docs: add OpenTelemetry env var documentation to .env.example * test(tracing): add lifecycle integration tests for trace propagation * fix(tracing): inject trace context into pipeline and batch flow jobs * fix(tracing): add sidecar.execute Node-side span and remove unnecessary comment Wraps PythonDispatcher.run() with a sidecar.execute span on the Node side so traces show the full round-trip (Node span -> Python span). Also removes an obvious comment from logger.ts.	2026-06-15 12:53:06 +08:00