mirror of
https://github.com/snapotter-hq/SnapOtter.git
synced 2026-08-03 07:46:42 +02:00
SnapOtterandGitHub
17726ae59d
* docs: rebrand from image-only to multi-modality across docs and metadata SnapOtter expanded from image-only to 157 tools across 5 modalities (image, video, audio, document/PDF, data). Update all product-level copy, metadata, and i18n that still framed it as an image-only tool. - README, package.json, root llms.txt: multi-modality framing, 157 tools - OpenAPI info + tags, generated /llms.txt tagline (docs.ts) - VitePress docs site: hero, getting-started, architecture, security, deployment, configuration, developer, supported-formats - i18n: 10 product keys across all 21 locales (hero, app description, privacy notes, AI features, progress messages, getting-started) - web/demo/landing meta + privacy copy, COMMUNITY_GUIDE, .env.example Stale tool counts (53/50+/52/70+/35) corrected to 157 throughout. Database/container deployment claims left unchanged (out of scope). * docs: fix stale post-rebrand test assertions and README language list - tests/e2e-docs/homepage.spec.ts: assert the current docs homepage (file toolkit, 157 tools, 5 modalities) instead of the old image-only strings - tests/unit/api/docs-route.test.ts: sync the reproduced llms.txt tagline with docs.ts - README.md: 21 languages with the correct list (add Swedish and Chinese Traditional, drop Czech which is not supported) * docs: correct 2.0 architecture references (Postgres 17 + Redis 8, 3-container stack) The docs and metadata still described the 1.x stack (SQLite, single container, p-queue). Update them to the current 2.0 reality. - README: replace the broken single-container `docker run` quick-start with the real Docker Compose stack (app + Postgres 17 + Redis 8); fix the "no Redis, no Postgres" feature bullet - package.json: description no longer claims a single container - apps/docs: rewrite database.md for Postgres; configuration.md DB_PATH -> DATABASE_URL + REDIS_URL; architecture.md SQLite/p-queue/better-sqlite3 -> Postgres/BullMQ/pg and add media-engine + doc-engine; developer/security/deployment/docker-tags/getting-started/contributing compose examples now include postgres + redis; index.md + api/ai.md AI count 16 -> 19 - SECURITY.md: Drizzle (SQLite) -> (PostgreSQL) - landing: enterprise/FeatureHighlights single-container wording; TrustSignals/ToolGrid 150+ -> 157 (dynamic); Pricing/FAQ 15 -> 19 AI tools * docs(api): document all video, audio, document, and data tool endpoints in OpenAPI The spec covered only image tools; the Scalar UI and the generated /llms.txt and /llms-full.txt inherited that gap. Add the 104 missing tool endpoints so the API docs match the code. - Video: 29 endpoints (most long/async; auto-subtitles is AI) - Audio: 17 (transcribe-audio is AI) - Document/PDF: 36 (ocr-pdf is AI; conversions are long/async) - Data: 10 - Image: 12 newer tools (background-replace, blur-background AI; histogram/lqip-placeholder/sprite-sheet custom responses; barcode-generate uses a JSON body) Each schema is derived from the tool's Zod validator and executionHint (fast -> 200, long -> 202+SSE, AI adds 501 FeatureNotInstalledError, multi-file inputs as arrays), referencing the existing shared schemas. Tool path entries: 64 -> 168. Spec parses as valid YAML with no duplicate paths and only known $refs.
287 lines
12 KiB
Markdown
287 lines
12 KiB
Markdown
---
|
||
description: AI engine reference with all local ML tools. Background removal, upscaling, OCR, face detection, photo restoration, and more.
|
||
---
|
||
|
||
# AI Engine Reference
|
||
|
||
The `@snapotter/ai` package bridges Node.js to a **persistent Python sidecar** for all ML operations. The dispatcher process stays alive between requests for fast warm-start performance. GPU is auto-detected at startup and used when available.
|
||
|
||
19 AI tool routes. All models run locally - no internet required after initial model download.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
Node.js Tool Route
|
||
│
|
||
▼
|
||
@snapotter/ai bridge.ts
|
||
│ (stdin/stdout JSON + stderr progress events)
|
||
▼
|
||
Python dispatcher (persistent process)
|
||
│
|
||
├─ remove_bg.py (rembg / BiRefNet)
|
||
├─ upscale.py (RealESRGAN)
|
||
├─ inpaint.py (LaMa ONNX)
|
||
├─ ocr.py (PaddleOCR / Tesseract)
|
||
├─ detect_faces.py (MediaPipe)
|
||
├─ face_landmarks.py (MediaPipe landmarks)
|
||
├─ enhance_faces.py (GFPGAN / CodeFormer)
|
||
├─ colorize.py (DDColor)
|
||
├─ noise_removal.py (tiered denoising)
|
||
├─ red_eye_removal.py (landmark + color analysis)
|
||
├─ restore.py (scratch repair + enhancement + denoising)
|
||
├─ transparency_fix.py (BiRefNet HR-matting + defringe)
|
||
└─ seam_carving (Go caire binary - not Python)
|
||
```
|
||
|
||
**Timeouts:** 300 s default; OCR and BiRefNet background removal get 600 s.
|
||
|
||
## Background Removal
|
||
|
||
**Function:** `removeBackground`
|
||
**Tool route:** `remove-background`
|
||
**Model:** rembg with BiRefNet (default) or U2-Net variants
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `model` | string | `birefnet-general` | Model variant - see table below |
|
||
| `alphaMattingForeground` | number (1–255) | 240 | Foreground threshold for alpha matting |
|
||
| `alphaMattingBackground` | number (1–255) | 10 | Background threshold for alpha matting |
|
||
| `returnMask` | boolean | false | Return the mask instead of the cutout |
|
||
| `backgroundColor` | string | - | Fill removed area (hex color or "transparent") |
|
||
|
||
**Available models:**
|
||
|
||
| Model ID | Best for |
|
||
|----------|---------|
|
||
| `birefnet-general` | General purpose (default) |
|
||
| `birefnet-portrait` | People / portraits |
|
||
| `birefnet-dis` | Dichotomous Image Segmentation |
|
||
| `birefnet-hrsod` | High-resolution salient objects |
|
||
| `birefnet-cod` | Camouflaged objects |
|
||
| `u2net` | Fast general purpose |
|
||
| `u2net_human_seg` | Human segmentation |
|
||
| `isnet-general-use` | High quality general |
|
||
|
||
## Image Upscaling
|
||
|
||
**Function:** `upscale`
|
||
**Tool route:** `upscale`
|
||
**Model:** RealESRGAN (with Lanczos fallback on CPU-constrained systems)
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `scale` | 2 \| 4 | 4 | Upscale factor |
|
||
| `model` | string | `realesrgan-x4plus` | Model variant |
|
||
| `faceEnhance` | boolean | false | Apply GFPGAN face enhancement pass |
|
||
| `denoise` | number (0–1) | 0.5 | Denoising strength |
|
||
| `format` | string | - | Output format override |
|
||
| `quality` | number | 95 | Output quality (for JPEG/WebP) |
|
||
|
||
## OCR / Text Extraction
|
||
|
||
**Function:** `extractText`
|
||
**Tool route:** `ocr`
|
||
**Models:** Tesseract (fast), PaddleOCR PP-OCRv5 (balanced), PaddleOCR-VL 1.5 (best)
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `quality` | `fast` \| `balanced` \| `best` | `balanced` | Processing tier |
|
||
| `language` | string | `en` | Language code (ISO 639-1) |
|
||
| `enhance` | boolean | false | Pre-process image to improve OCR accuracy |
|
||
|
||
Returns structured results with bounding boxes, confidence scores, and extracted text blocks.
|
||
|
||
## Face / PII Blur
|
||
|
||
**Function:** `blurFaces`
|
||
**Tool route:** `blur-faces`
|
||
**Model:** MediaPipe face detection
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `blurRadius` | number | 30 | Gaussian blur radius |
|
||
| `sensitivity` | number (0–1) | 0.5 | Detection confidence threshold |
|
||
|
||
## Face Enhancement
|
||
|
||
**Function:** `enhanceFaces`
|
||
**Tool route:** `enhance-faces`
|
||
**Models:** GFPGAN, CodeFormer
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `model` | `gfpgan` \| `codeformer` | `gfpgan` | Enhancement model |
|
||
| `strength` | number (0–1) | 0.7 | Enhancement strength |
|
||
| `sensitivity` | number (0–1) | 0.5 | Face detection threshold |
|
||
| `centerFace` | boolean | false | Focus enhancement on center face only |
|
||
|
||
## AI Colorization
|
||
|
||
**Function:** `colorize`
|
||
**Tool route:** `colorize`
|
||
**Model:** DDColor (with OpenCV DNN fallback)
|
||
|
||
Converts black-and-white or grayscale photos to full color.
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `intensity` | number (0–1) | 0.85 | Color saturation strength |
|
||
| `model` | string | `ddcolor` | Model variant |
|
||
|
||
## Noise Removal
|
||
|
||
**Function:** `noiseRemoval`
|
||
**Tool route:** `noise-removal`
|
||
|
||
Three-tier denoising pipeline (fast: OpenCV bilateral filter; balanced: frequency-domain; best: deep learning model).
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `quality` | `fast` \| `balanced` \| `best` | `balanced` | Processing tier |
|
||
| `strength` | number (0–1) | 0.5 | Denoising strength |
|
||
| `preserveDetail` | boolean | true | Edge-preserving mode |
|
||
| `colorNoise` | boolean | false | Target color noise specifically |
|
||
|
||
## Red Eye Removal
|
||
|
||
**Function:** `removeRedEye`
|
||
**Tool route:** `red-eye-removal`
|
||
|
||
Detects face landmarks, locates eye regions, and corrects red-channel oversaturation.
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `sensitivity` | number (0–1) | 0.5 | Red pixel detection threshold |
|
||
| `strength` | number (0–1) | 0.9 | Correction strength |
|
||
|
||
## Photo Restoration
|
||
|
||
**Function:** `restorePhoto`
|
||
**Tool route:** `restore-photo`
|
||
|
||
Multi-step pipeline for old or damaged photos: scratch/tear detection and repair → face enhancement → denoising → optional colorization.
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `mode` | `auto` \| `light` \| `heavy` | `auto` | Restoration intensity |
|
||
| `scratchRemoval` | boolean | true | Detect and repair scratches, tears |
|
||
| `faceEnhancement` | boolean | true | Apply face enhancement pass |
|
||
| `fidelity` | number (0–1) | 0.7 | Face enhancement strength |
|
||
| `denoise` | boolean | true | Apply denoising pass |
|
||
| `denoiseStrength` | number (0–100) | 40 | Denoising strength |
|
||
| `colorize` | boolean | false | Colorize after restoration |
|
||
|
||
## Passport Photo
|
||
|
||
**Function:** Uses `detectFaceLandmarks` + `removeBackground`
|
||
**Tool route:** `passport-photo`
|
||
**Model:** MediaPipe face landmarks
|
||
|
||
Generates government-compliant ID photos. Supports **37 countries** across 6 regions (Americas, Europe, Asia, Africa, Oceania, Middle East). Each spec includes physical dimensions, DPI, head-height ratio, eye-line position, and background color requirements.
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `country` | string | `us` | ISO country code (see list in UI) |
|
||
| `printLayout` | `4x6` \| `A4` \| `none` | `none` | Output as print sheet or standalone |
|
||
| `backgroundColor` | string | country default | Background fill color |
|
||
|
||
## Object Erasing (Inpainting)
|
||
|
||
**Function:** `inpaint`
|
||
**Tool route:** `erase-object`
|
||
**Model:** LaMa via ONNX Runtime
|
||
|
||
| Parameter | Type | Required | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `maskData` | string | Yes | Base64-encoded PNG mask (white = erase) |
|
||
| `maskThreshold` | number (0–255) | No | Threshold for mask binarization |
|
||
|
||
GPU-accelerated when an NVIDIA GPU is available.
|
||
|
||
## Smart Crop
|
||
|
||
**Function:** Uses MediaPipe + Sharp attention/entropy
|
||
**Tool route:** `smart-crop`
|
||
**Model:** MediaPipe face detection
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `mode` | `subject` \| `face` \| `trim` | `subject` | Crop strategy |
|
||
| `width` | number | - | Output width |
|
||
| `height` | number | - | Output height |
|
||
| `facePreset` | string | - | Preset framing when `mode=face` |
|
||
|
||
**Face presets:**
|
||
|
||
| Preset | Head ratio | Best for |
|
||
|--------|-----------|---------|
|
||
| `close-up` | 1.8× face | Headshots |
|
||
| `head-and-shoulders` | 2.8× face | Profile photos |
|
||
| `upper-body` | 4.5× face | LinkedIn / formal |
|
||
| `half-body` | 7.0× face | Full upper body |
|
||
|
||
## Image Enhancement
|
||
|
||
**Function:** `analyzeImage` + `applyCorrections`
|
||
**Tool route:** `image-enhancement`
|
||
**Engine:** Analysis-based (Sharp histogram and statistics)
|
||
|
||
Analyzes the image and applies automatic corrections for exposure, contrast, white balance, saturation, sharpness, and noise. Supports scene-specific modes.
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `mode` | `auto` \| `portrait` \| `landscape` \| `low-light` \| `food` \| `document` | `auto` | Scene mode for tuning corrections |
|
||
| `intensity` | number (0-100) | 50 | Overall correction strength |
|
||
| `corrections.exposure` | boolean | true | Apply exposure correction |
|
||
| `corrections.contrast` | boolean | true | Apply contrast correction |
|
||
| `corrections.whiteBalance` | boolean | true | Apply white balance correction |
|
||
| `corrections.saturation` | boolean | true | Apply saturation correction |
|
||
| `corrections.sharpness` | boolean | true | Apply sharpness correction |
|
||
| `corrections.denoise` | boolean | true | Apply denoising |
|
||
|
||
An additional analysis endpoint is available at `POST /api/v1/tools/image-enhancement/analyze` which returns the detected corrections without applying them.
|
||
|
||
## Content-Aware Resize (Seam Carving)
|
||
|
||
**Function:** `seamCarve`
|
||
**Tool route:** `content-aware-resize`
|
||
**Engine:** Go `caire` binary (not Python - no GPU benefit)
|
||
|
||
Intelligently resizes images by removing or adding low-energy seams, preserving important content.
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `width` | number | - | Target width |
|
||
| `height` | number | - | Target height |
|
||
| `protectFaces` | boolean | true | Protect detected face regions from seam removal |
|
||
| `blurRadius` | number | 0 | Pre-blur to reduce noise sensitivity |
|
||
| `sobelThreshold` | number | 10 | Edge sensitivity threshold |
|
||
| `square` | boolean | false | Force square output |
|
||
|
||
Max input edge before auto-downscaling: **1200 px**.
|
||
|
||
## PNG Transparency Fixer
|
||
|
||
**Function:** `fixTransparency`
|
||
**Tool route:** `transparency-fixer`
|
||
**Model:** BiRefNet HR-matting (2048x2048 resolution)
|
||
|
||
Fixes "fake transparent" PNGs where the background was removed but left behind fringing, halos, or semi-transparent artifacts. Uses BiRefNet's high-resolution matting model to produce a clean alpha channel, then applies configurable defringe processing to remove color contamination along edges.
|
||
|
||
**OOM fallback chain:** If BiRefNet HR-matting exceeds available memory, the tool automatically falls back to `birefnet-general`, then to `u2net`.
|
||
|
||
**Feature bundle:** Background Removal (shared with Remove Background and Passport Photo).
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `defringe` | number (0-100) | 30 | Edge defringe strength to remove color contamination |
|
||
| `outputFormat` | `"png"` \| `"webp"` | `"png"` | Output image format |
|
||
|
||
```bash
|
||
curl -X POST http://localhost:1349/api/v1/tools/transparency-fixer \
|
||
-H "Authorization: Bearer <token>" \
|
||
-F "file=@fake-transparent.png" \
|
||
-F 'settings={"defringe":30,"outputFormat":"png"}'
|
||
```
|