diff --git a/apps/docs/api/ai.md b/apps/docs/api/ai.md index b9cb1a77..ee4f238d 100644 --- a/apps/docs/api/ai.md +++ b/apps/docs/api/ai.md @@ -6,119 +6,173 @@ description: AI engine reference with all local ML tools. Background removal, up The `@snapotter/ai` package bridges Node.js to a **persistent Python sidecar** for all ML operations. The dispatcher process stays alive between requests for fast warm-start performance. GPU is auto-detected at startup and used when available. -19 AI tool routes. All models run locally - no internet required after initial model download. +19 Python sidecar AI tools across four modalities (image, audio, video, document), plus 2 tools with optional AI capabilities. All models run locally -- no internet required after initial model download. ## Architecture ``` Node.js Tool Route - │ - ▼ + | + v @snapotter/ai bridge.ts - │ (stdin/stdout JSON + stderr progress events) - ▼ - Python dispatcher (persistent process) - │ - ├─ remove_bg.py (rembg / BiRefNet) - ├─ upscale.py (RealESRGAN) - ├─ inpaint.py (LaMa ONNX) - ├─ ocr.py (PaddleOCR / Tesseract) - ├─ detect_faces.py (MediaPipe) - ├─ face_landmarks.py (MediaPipe landmarks) - ├─ enhance_faces.py (GFPGAN / CodeFormer) - ├─ colorize.py (DDColor) - ├─ noise_removal.py (tiered denoising) - ├─ red_eye_removal.py (landmark + color analysis) - ├─ restore.py (scratch repair + enhancement + denoising) - ├─ transparency_fix.py (BiRefNet HR-matting + defringe) - └─ seam_carving (Go caire binary - not Python) + | (stdin/stdout JSON + stderr progress events) + v + Python dispatcher (persistent process, "ai" profile) + | + |-- remove_bg.py (rembg / BiRefNet) + |-- upscale.py (RealESRGAN) + |-- inpaint.py (LaMa ONNX) + |-- outpaint.py (LaMa canvas expansion) + |-- ocr.py (PaddleOCR / Tesseract) + |-- ocr_pdf.py (page-by-page document OCR) + |-- ocr_preprocess.py (image enhancement for OCR) + |-- detect_faces.py (MediaPipe) + |-- face_landmarks.py (MediaPipe landmarks) + |-- enhance_faces.py (GFPGAN / CodeFormer) + |-- colorize.py (DDColor) + |-- noise_removal.py (SCUNet / tiered denoising) + |-- red_eye_removal.py (landmark + color analysis) + |-- restore.py (scratch repair + enhancement + denoising) + |-- transcribe.py (faster-whisper speech-to-text) + +-- install_feature.py (on-demand bundle installer) ``` +A separate "docs" dispatcher profile replaces the AI allowlist with document-processing scripts (`doc_pagecount`, `doc_health`, `doc_flatten`, `doc_redact`, `doc_text`, `doc_to_word`, `doc_metadata`, `doc_html_pdf`) and skips heavy ML imports. + **Timeouts:** 300 s default; OCR and BiRefNet background removal get 600 s. +## Feature Bundles + +Each AI tool requires a model bundle to be installed before use. Bundles are installed on demand via the admin UI or `install_feature.py`. + +| Bundle | Size | Tools | +|--------|------|-------| +| `background-removal` | 4-5 GB | remove-background, passport-photo, transparency-fixer, background-replace, blur-background | +| `face-detection` | 200-300 MB | blur-faces, red-eye-removal, smart-crop | +| `object-eraser-colorize` | 1-2 GB | erase-object, colorize, ai-canvas-expand | +| `upscale-enhance` | 4-5 GB | upscale, enhance-faces, noise-removal | +| `photo-restoration` | 800 MB - 1 GB | restore-photo | +| `ocr` | 3-4 GB | ocr, ocr-pdf | +| `transcription` | ~600 MB | transcribe-audio, auto-subtitles | + +--- + ## Background Removal -**Function:** `removeBackground` **Tool route:** `remove-background` **Model:** rembg with BiRefNet (default) or U2-Net variants | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `model` | string | `birefnet-general` | Model variant - see table below | -| `alphaMattingForeground` | number (1–255) | 240 | Foreground threshold for alpha matting | -| `alphaMattingBackground` | number (1–255) | 10 | Background threshold for alpha matting | -| `returnMask` | boolean | false | Return the mask instead of the cutout | -| `backgroundColor` | string | - | Fill removed area (hex color or "transparent") | +| `model` | string | - | Model variant (optional override) | +| `backgroundType` | string | `"transparent"` | One of: `transparent`, `color`, `gradient`, `blur`, `image` | +| `backgroundColor` | string | - | Hex color for solid background | +| `gradientColor1` | string | - | First gradient color | +| `gradientColor2` | string | - | Second gradient color | +| `gradientAngle` | number | - | Gradient angle in degrees | +| `blurEnabled` | boolean | - | Enable background blur effect | +| `blurIntensity` | number (0-100) | - | Blur intensity | +| `shadowEnabled` | boolean | - | Enable drop shadow on subject | +| `shadowOpacity` | number (0-100) | - | Shadow opacity | +| `outputFormat` | string | - | Output format: `png`, `webp`, or `avif` | +| `edgeRefine` | integer (0-3) | - | Edge refinement level | +| `decontaminate` | boolean | - | Remove color bleed from edges | -**Available models:** +## Background Replace -| Model ID | Best for | -|----------|---------| -| `birefnet-general` | General purpose (default) | -| `birefnet-portrait` | People / portraits | -| `birefnet-dis` | Dichotomous Image Segmentation | -| `birefnet-hrsod` | High-resolution salient objects | -| `birefnet-cod` | Camouflaged objects | -| `u2net` | Fast general purpose | -| `u2net_human_seg` | Human segmentation | -| `isnet-general-use` | High quality general | +**Tool route:** `background-replace` +**Model:** rembg / BiRefNet (shared with remove-background) -## Image Upscaling - -**Function:** `upscale` -**Tool route:** `upscale` -**Model:** RealESRGAN (with Lanczos fallback on CPU-constrained systems) +Removes the background and replaces it with a solid color or gradient. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `scale` | 2 \| 4 | 4 | Upscale factor | -| `model` | string | `realesrgan-x4plus` | Model variant | -| `faceEnhance` | boolean | false | Apply GFPGAN face enhancement pass | -| `denoise` | number (0–1) | 0.5 | Denoising strength | -| `format` | string | - | Output format override | -| `quality` | number | 95 | Output quality (for JPEG/WebP) | +| `backgroundType` | `"color"` \| `"gradient"` | `"color"` | Background mode | +| `color` | string | `"#ffffff"` | Background hex color (when `backgroundType` is `color`) | +| `gradientColor1` | string | - | First gradient hex color | +| `gradientColor2` | string | - | Second gradient hex color | +| `gradientAngle` | integer (0-360) | `180` | Gradient angle in degrees | +| `feather` | integer (0-20) | `0` | Edge feathering radius | +| `format` | `"png"` \| `"webp"` | `"png"` | Output format | + +## Blur Background + +**Tool route:** `blur-background` +**Model:** rembg / BiRefNet (shared with remove-background) + +Blurs the background while keeping the subject sharp. + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `intensity` | integer (1-100) | `50` | Blur intensity | +| `feather` | integer (0-20) | `0` | Edge feathering radius | +| `format` | `"png"` \| `"webp"` | `"png"` | Output format | + +## Image Upscaling + +**Tool route:** `upscale` +**Model:** RealESRGAN (with Lanczos fallback when unavailable) + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `scale` | number | `2` | Upscale factor | +| `model` | string | `"auto"` | Model variant | +| `faceEnhance` | boolean | `false` | Apply GFPGAN face enhancement pass | +| `denoise` | number | `0` | Denoising strength | +| `format` | string | `"auto"` | Output format override | +| `quality` | number | `95` | Output quality (1-100) | ## OCR / Text Extraction -**Function:** `extractText` **Tool route:** `ocr` **Models:** Tesseract (fast), PaddleOCR PP-OCRv5 (balanced), PaddleOCR-VL 1.5 (best) | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `quality` | `fast` \| `balanced` \| `best` | `balanced` | Processing tier | -| `language` | string | `en` | Language code (ISO 639-1) | -| `enhance` | boolean | false | Pre-process image to improve OCR accuracy | +| `quality` | `"fast"` \| `"balanced"` \| `"best"` | `"balanced"` | Processing tier | +| `language` | string | `"auto"` | Language: `auto`, `en`, `de`, `fr`, `es`, `zh`, `ja`, `ko` | +| `enhance` | boolean | `true` | Pre-process image to improve OCR accuracy | +| `engine` | string | - | Deprecated. Maps `tesseract` to `fast`, `paddleocr` to `balanced` | Returns structured results with bounding boxes, confidence scores, and extracted text blocks. +## PDF OCR + +**Tool route:** `ocr-pdf` +**Models:** Same tier system as image OCR + +Extracts text from scanned PDF documents using AI-powered OCR, page by page. + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `quality` | `"fast"` \| `"balanced"` \| `"best"` | `"balanced"` | Processing tier | +| `language` | string | `"auto"` | Language: `auto`, `en`, `de`, `fr`, `es`, `zh`, `ja`, `ko` | +| `pages` | string | `"all"` | Page selection: `"all"`, `"1-3"`, `"1,3,5"` | + ## Face / PII Blur -**Function:** `blurFaces` **Tool route:** `blur-faces` **Model:** MediaPipe face detection | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `blurRadius` | number | 30 | Gaussian blur radius | -| `sensitivity` | number (0–1) | 0.5 | Detection confidence threshold | +| `blurRadius` | number (1-100) | `30` | Gaussian blur radius | +| `sensitivity` | number (0-1) | `0.5` | Detection confidence threshold | ## Face Enhancement -**Function:** `enhanceFaces` **Tool route:** `enhance-faces` **Models:** GFPGAN, CodeFormer | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `model` | `gfpgan` \| `codeformer` | `gfpgan` | Enhancement model | -| `strength` | number (0–1) | 0.7 | Enhancement strength | -| `sensitivity` | number (0–1) | 0.5 | Face detection threshold | -| `centerFace` | boolean | false | Focus enhancement on center face only | +| `model` | `"auto"` \| `"gfpgan"` \| `"codeformer"` | `"auto"` | Enhancement model | +| `strength` | number (0-1) | `0.8` | Enhancement strength | +| `sensitivity` | number (0-1) | `0.5` | Face detection threshold | +| `onlyCenterFace` | boolean | `false` | Enhance only the most central face | ## AI Colorization -**Function:** `colorize` **Tool route:** `colorize` **Model:** DDColor (with OpenCV DNN fallback) @@ -126,144 +180,182 @@ Converts black-and-white or grayscale photos to full color. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `intensity` | number (0–1) | 0.85 | Color saturation strength | -| `model` | string | `ddcolor` | Model variant | +| `intensity` | number (0-1) | `1.0` | Color saturation strength | +| `model` | `"auto"` \| `"ddcolor"` \| `"opencv"` | `"auto"` | Model variant | ## Noise Removal -**Function:** `noiseRemoval` -**Tool route:** `noise-removal` - -Three-tier denoising pipeline (fast: OpenCV bilateral filter; balanced: frequency-domain; best: deep learning model). +**Tool route:** `noise-removal` +**Model:** SCUNet (tiered denoising pipeline) | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `quality` | `fast` \| `balanced` \| `best` | `balanced` | Processing tier | -| `strength` | number (0–1) | 0.5 | Denoising strength | -| `preserveDetail` | boolean | true | Edge-preserving mode | -| `colorNoise` | boolean | false | Target color noise specifically | +| `tier` | `"quick"` \| `"balanced"` \| `"quality"` \| `"maximum"` | `"balanced"` | Processing tier | +| `strength` | number (0-100) | `50` | Denoising strength | +| `detailPreservation` | number (0-100) | `50` | How much detail to preserve; higher keeps more texture | +| `colorNoise` | number (0-100) | `30` | Color noise reduction strength | +| `format` | string | `"original"` | Output format: `original`, `png`, `jpeg`, `webp`, `avif`, `jxl` | +| `quality` | number (1-100) | `90` | Output encoding quality | ## Red Eye Removal -**Function:** `removeRedEye` **Tool route:** `red-eye-removal` Detects face landmarks, locates eye regions, and corrects red-channel oversaturation. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `sensitivity` | number (0–1) | 0.5 | Red pixel detection threshold | -| `strength` | number (0–1) | 0.9 | Correction strength | +| `sensitivity` | number (0-100) | `50` | Red pixel detection threshold | +| `strength` | number (0-100) | `70` | Correction strength | +| `format` | string | - | Output format override (optional) | +| `quality` | number (1-100) | `90` | Output quality | ## Photo Restoration -**Function:** `restorePhoto` **Tool route:** `restore-photo` -Multi-step pipeline for old or damaged photos: scratch/tear detection and repair → face enhancement → denoising → optional colorization. +Multi-step pipeline for old or damaged photos: scratch/tear detection and repair, face enhancement, denoising, and optional colorization. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `mode` | `auto` \| `light` \| `heavy` | `auto` | Restoration intensity | -| `scratchRemoval` | boolean | true | Detect and repair scratches, tears | -| `faceEnhancement` | boolean | true | Apply face enhancement pass | -| `fidelity` | number (0–1) | 0.7 | Face enhancement strength | -| `denoise` | boolean | true | Apply denoising pass | -| `denoiseStrength` | number (0–100) | 40 | Denoising strength | -| `colorize` | boolean | false | Colorize after restoration | +| `scratchRemoval` | boolean | `true` | Detect and repair scratches, tears | +| `faceEnhancement` | boolean | `true` | Apply face enhancement pass | +| `fidelity` | number (0-1) | `0.7` | Face enhancement strength (higher = more conservative) | +| `denoise` | boolean | `true` | Apply denoising pass | +| `denoiseStrength` | number (0-100) | `25` | Denoising strength | +| `colorize` | boolean | `false` | Colorize after restoration | +| `colorizeStrength` | number (0-100) | `85` | Colorization intensity | ## Passport Photo -**Function:** Uses `detectFaceLandmarks` + `removeBackground` **Tool route:** `passport-photo` -**Model:** MediaPipe face landmarks +**Models:** MediaPipe face landmarks + BiRefNet background removal -Generates government-compliant ID photos. Supports **37 countries** across 6 regions (Americas, Europe, Asia, Africa, Oceania, Middle East). Each spec includes physical dimensions, DPI, head-height ratio, eye-line position, and background color requirements. +Two-phase workflow: analyze (detect face + remove background) then generate (crop, resize, tile). Supports 37+ countries across 6 regions. + +### Phase 1: Analyze + +`POST /api/v1/tools/passport-photo/analyze` + +Accepts an image file (multipart). Returns face landmark data, a base64 preview, and image dimensions. + +### Phase 2: Generate + +`POST /api/v1/tools/passport-photo/generate` + +Accepts a JSON body with the Phase 1 results plus generation settings: | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `country` | string | `us` | ISO country code (see list in UI) | -| `printLayout` | `4x6` \| `A4` \| `none` | `none` | Output as print sheet or standalone | -| `backgroundColor` | string | country default | Background fill color | +| `jobId` | string | (required) | Job ID from Phase 1 | +| `filename` | string | (required) | Original filename from Phase 1 | +| `countryCode` | string | (required) | ISO country code (e.g., `US`, `GB`, `IN`) | +| `documentType` | string | `"passport"` | Document type | +| `bgColor` | string | `"#FFFFFF"` | Background color hex | +| `printLayout` | string | `"none"` | Print layout: `none`, `4x6`, `a4`, `letter` | +| `maxFileSizeKb` | number | `0` | Max file size in KB (0 = no limit) | +| `dpi` | number (72-1200) | `300` | Output DPI | +| `customWidthMm` | number | - | Custom width in mm (overrides country spec) | +| `customHeightMm` | number | - | Custom height in mm (overrides country spec) | +| `zoom` | number (0.5-3) | `1` | Zoom factor | +| `adjustX` | number | `0` | Horizontal position adjustment | +| `adjustY` | number | `0` | Vertical position adjustment | +| `landmarks` | object | (required) | Landmarks from Phase 1 | +| `imageWidth` | number | (required) | Image width from Phase 1 | +| `imageHeight` | number | (required) | Image height from Phase 1 | ## Object Erasing (Inpainting) -**Function:** `inpaint` **Tool route:** `erase-object` **Model:** LaMa via ONNX Runtime -| Parameter | Type | Required | Description | +The mask is sent as a **second file part** (fieldname `mask`), not as base64. White pixels in the mask indicate areas to erase. The `format` and `quality` settings are sent as top-level form fields. + +| Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `maskData` | string | Yes | Base64-encoded PNG mask (white = erase) | -| `maskThreshold` | number (0–255) | No | Threshold for mask binarization | +| `file` | file | (required) | Source image (multipart) | +| `mask` | file | (required) | Mask image (multipart, fieldname `mask`, white = erase) | +| `format` | string | `"auto"` | Output format: `auto`, `png`, `jpg`, `jpeg`, `webp`, `tiff`, `gif`, `avif`, `heic`, `heif`, `jxl` | +| `quality` | integer (1-100) | `95` | Output quality | GPU-accelerated when an NVIDIA GPU is available. -## Smart Crop +## AI Canvas Expand -**Function:** Uses MediaPipe + Sharp attention/entropy -**Tool route:** `smart-crop` -**Model:** MediaPipe face detection +**Tool route:** `ai-canvas-expand` +**Model:** LaMa-based outpainting + +Expands the canvas of an image in any direction and fills new areas with AI-generated content that matches the existing image. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `mode` | `subject` \| `face` \| `trim` | `subject` | Crop strategy | -| `width` | number | - | Output width | -| `height` | number | - | Output height | -| `facePreset` | string | - | Preset framing when `mode=face` | +| `extendTop` | integer | `0` | Pixels to extend at the top | +| `extendRight` | integer | `0` | Pixels to extend at the right | +| `extendBottom` | integer | `0` | Pixels to extend at the bottom | +| `extendLeft` | integer | `0` | Pixels to extend at the left | +| `tier` | `"fast"` \| `"balanced"` \| `"high"` | `"balanced"` | Quality tier | +| `format` | string | `"auto"` | Output format: `auto`, `png`, `jpg`, `jpeg`, `webp`, `tiff`, `gif`, `avif`, `heic`, `heif`, `jxl` | +| `quality` | integer (1-100) | `95` | Output quality | + +At least one extend direction must be greater than 0. + +## Smart Crop + +**Tool route:** `smart-crop` +**Model:** MediaPipe face detection (face mode only) + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `mode` | string | `"subject"` | Crop strategy: `subject`, `face`, `trim` | +| `strategy` | `"attention"` \| `"entropy"` | `"attention"` | Strategy for subject mode | +| `width` | integer | - | Output width | +| `height` | integer | - | Output height | +| `padding` | integer (0-50) | `0` | Padding percentage around subject | +| `facePreset` | string | `"head-shoulders"` | Preset framing when `mode=face` | +| `sensitivity` | number (0-1) | `0.5` | Face detection threshold | +| `threshold` | integer (0-255) | `30` | Background detection threshold (trim mode) | +| `padToSquare` | boolean | `false` | Pad trimmed result to a square | +| `padColor` | string | `"#ffffff"` | Background color for square padding | +| `targetSize` | integer | - | Target size for padded output (pixels) | +| `quality` | integer (1-100) | - | Output quality | + +Legacy `mode` values `attention` and `content` are accepted and mapped to `subject` and `trim` respectively. **Face presets:** -| Preset | Head ratio | Best for | -|--------|-----------|---------| -| `close-up` | 1.8× face | Headshots | -| `head-and-shoulders` | 2.8× face | Profile photos | -| `upper-body` | 4.5× face | LinkedIn / formal | -| `half-body` | 7.0× face | Full upper body | +| Preset | Best for | +|--------|---------| +| `closeup` | Headshots | +| `head-shoulders` | Profile photos | +| `upper-body` | LinkedIn / formal | +| `half-body` | Full upper body | -## Image Enhancement +## Transcribe Audio -**Function:** `analyzeImage` + `applyCorrections` -**Tool route:** `image-enhancement` -**Engine:** Analysis-based (Sharp histogram and statistics) +**Tool route:** `transcribe-audio` +**Model:** faster-whisper -Analyzes the image and applies automatic corrections for exposure, contrast, white balance, saturation, sharpness, and noise. Supports scene-specific modes. +Converts speech to text. Supports plain text, SRT, and VTT output formats. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `mode` | `auto` \| `portrait` \| `landscape` \| `low-light` \| `food` \| `document` | `auto` | Scene mode for tuning corrections | -| `intensity` | number (0-100) | 50 | Overall correction strength | -| `corrections.exposure` | boolean | true | Apply exposure correction | -| `corrections.contrast` | boolean | true | Apply contrast correction | -| `corrections.whiteBalance` | boolean | true | Apply white balance correction | -| `corrections.saturation` | boolean | true | Apply saturation correction | -| `corrections.sharpness` | boolean | true | Apply sharpness correction | -| `corrections.denoise` | boolean | true | Apply denoising | +| `language` | string | `"auto"` | Language: `auto`, `en`, `de`, `fr`, `es`, `zh`, `ja`, `ko`, `id`, `th`, `vi` | +| `outputFormat` | `"txt"` \| `"srt"` \| `"vtt"` | `"txt"` | Output format | -An additional analysis endpoint is available at `POST /api/v1/tools/image-enhancement/analyze` which returns the detected corrections without applying them. +## Auto Subtitles -## Content-Aware Resize (Seam Carving) +**Tool route:** `auto-subtitles` +**Model:** faster-whisper (extracts audio from video, then transcribes) -**Function:** `seamCarve` -**Tool route:** `content-aware-resize` -**Engine:** Go `caire` binary (not Python - no GPU benefit) - -Intelligently resizes images by removing or adding low-energy seams, preserving important content. +Generates subtitle files from a video's audio track. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `width` | number | - | Target width | -| `height` | number | - | Target height | -| `protectFaces` | boolean | true | Protect detected face regions from seam removal | -| `blurRadius` | number | 0 | Pre-blur to reduce noise sensitivity | -| `sobelThreshold` | number | 10 | Edge sensitivity threshold | -| `square` | boolean | false | Force square output | - -Max input edge before auto-downscaling: **1200 px**. +| `language` | string | `"auto"` | Language: `auto`, `en`, `de`, `fr`, `es`, `zh`, `ja`, `ko`, `id`, `th`, `vi` | +| `format` | `"srt"` \| `"vtt"` | `"srt"` | Output subtitle format | ## PNG Transparency Fixer -**Function:** `fixTransparency` **Tool route:** `transparency-fixer` **Model:** BiRefNet HR-matting (2048x2048 resolution) @@ -271,12 +363,11 @@ Fixes "fake transparent" PNGs where the background was removed but left behind f **OOM fallback chain:** If BiRefNet HR-matting exceeds available memory, the tool automatically falls back to `birefnet-general`, then to `u2net`. -**Feature bundle:** Background Removal (shared with Remove Background and Passport Photo). - | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `defringe` | number (0-100) | 30 | Edge defringe strength to remove color contamination | +| `defringe` | number (0-100) | `30` | Edge defringe strength to remove color contamination | | `outputFormat` | `"png"` \| `"webp"` | `"png"` | Output image format | +| `removeWatermark` | boolean | `false` | Apply watermark removal pre-processing (median filter) | ```bash curl -X POST http://localhost:1349/api/v1/tools/transparency-fixer \ @@ -284,3 +375,46 @@ curl -X POST http://localhost:1349/api/v1/tools/transparency-fixer \ -F "file=@fake-transparent.png" \ -F 'settings={"defringe":30,"outputFormat":"png"}' ``` + +--- + +## Tools with Optional AI Capabilities + +The following tools are not Python sidecar tools but use AI features when certain options are enabled. + +### Image Enhancement + +**Tool route:** `image-enhancement` +**Engine:** Analysis-based (Sharp histogram and statistics) + +Analyzes the image and applies automatic corrections for exposure, contrast, white balance, saturation, sharpness, and noise. Supports scene-specific modes. + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `mode` | `"auto"` \| `"portrait"` \| `"landscape"` \| `"low-light"` \| `"food"` \| `"document"` | `"auto"` | Scene mode for tuning corrections | +| `intensity` | number (0-100) | `50` | Overall correction strength | +| `corrections.exposure` | boolean | `true` | Apply exposure correction | +| `corrections.contrast` | boolean | `true` | Apply contrast correction | +| `corrections.whiteBalance` | boolean | `true` | Apply white balance correction | +| `corrections.saturation` | boolean | `true` | Apply saturation correction | +| `corrections.sharpness` | boolean | `true` | Apply sharpness correction | +| `corrections.denoise` | boolean | `true` | Apply denoising | +| `deepEnhance` | boolean | `false` | Enable AI noise removal via SCUNet (requires `upscale-enhance` bundle) | + +An additional analysis endpoint is available at `POST /api/v1/tools/image-enhancement/analyze` which returns the detected corrections without applying them. + +### Content-Aware Resize (Seam Carving) + +**Tool route:** `content-aware-resize` +**Engine:** Go `caire` binary (not Python -- no GPU benefit) + +Intelligently resizes images by removing low-energy seams, preserving important content. + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `width` | number | - | Target width | +| `height` | number | - | Target height | +| `protectFaces` | boolean | `false` | Protect detected face regions (requires `face-detection` bundle) | +| `blurRadius` | number (0-20) | `4` | Pre-blur for energy calculation | +| `sobelThreshold` | number (1-20) | `2` | Edge sensitivity threshold | +| `square` | boolean | `false` | Force square output | diff --git a/apps/docs/api/image-engine.md b/apps/docs/api/image-engine.md index dbcb6ef1..659b920f 100644 --- a/apps/docs/api/image-engine.md +++ b/apps/docs/api/image-engine.md @@ -32,6 +32,7 @@ Cut out a rectangular region from the image. | `top` | number | Y offset from the top edge | | `width` | number | Width of the crop area | | `height` | number | Height of the crop area | +| `unit` | string | `px` (default) or `percent` | ### rotate @@ -40,15 +41,16 @@ Rotate the image by a given angle. | Parameter | Type | Description | |---|---|---| | `angle` | number | Rotation angle in degrees (0-360) | -| `background` | string | Fill color for the exposed area (default: transparent or white) | +| `background` | string | Fill color for exposed area (default: `#000000`). Only applies to non-90-degree angles. | ### flip -Mirror the image horizontally or vertically. +Mirror the image horizontally, vertically, or both. At least one must be true. | Parameter | Type | Description | |---|---|---| -| `direction` | string | `horizontal` or `vertical` | +| `horizontal` | boolean | Mirror left to right | +| `vertical` | boolean | Mirror top to bottom | ### convert @@ -56,9 +58,11 @@ Change the image format. | Parameter | Type | Description | |---|---|---| -| `format` | string | Target format: `jpeg`, `png`, `webp`, `avif`, `tiff`, `gif`, `jxl`, `heic`, `heif`, `bmp`, `ico`, `jp2`, `qoi` | +| `format` | string | Target format: `jpg`, `png`, `webp`, `avif`, `tiff`, `gif`, `jxl`, `heic`, `heif`, `bmp`, `ico`, `jp2`, `qoi` | | `quality` | number | Compression quality (1-100, applies to lossy formats) | +The first seven formats (`jpg` through `jxl`) are encoded by Sharp in-process. The remaining formats use external encoders at the API layer: `heic`/`heif` via heif-enc, `bmp`/`ico` via ImageMagick, `jp2` via opj_compress, and `qoi` via an inline TypeScript codec. + ### compress Reduce file size while keeping the same format. @@ -66,11 +70,20 @@ Reduce file size while keeping the same format. | Parameter | Type | Description | |---|---|---| | `quality` | number | Target quality (1-100) | +| `targetSizeBytes` | number | Optional target file size in bytes | | `format` | string | Optional format override | ### strip-metadata -Remove EXIF, IPTC, and XMP metadata from the image. Useful for privacy before sharing photos publicly. Takes no parameters. +Remove EXIF, IPTC, XMP, and ICC metadata from the image. With no parameters (or `stripAll: true`), strips everything. Pass individual flags for selective stripping. + +| Parameter | Type | Description | +|---|---|---| +| `stripAll` | boolean | Strip all metadata (default when no flags are set) | +| `stripExif` | boolean | Strip EXIF data (including GPS if `stripGps` is not separately set) | +| `stripGps` | boolean | Strip GPS location data | +| `stripIcc` | boolean | Strip ICC color profile | +| `stripXmp` | boolean | Strip XMP metadata | ### Color adjustments @@ -94,35 +107,102 @@ These apply a fixed color transformation. They take no parameters. ### Color channels -Adjust individual RGB color channels. +Adjust individual RGB color channels. Values are multipliers where 100 = no change. | Parameter | Type | Description | |---|---|---| -| `red` | number | Red channel adjustment (-100 to 100) | -| `green` | number | Green channel adjustment (-100 to 100) | -| `blue` | number | Blue channel adjustment (-100 to 100) | +| `red` | number | Red channel multiplier (0 to 200, 100 = unchanged) | +| `green` | number | Green channel multiplier (0 to 200, 100 = unchanged) | +| `blue` | number | Blue channel multiplier (0 to 200, 100 = unchanged) | + +### sharpen + +Simple sharpening controlled by a single value. + +| Parameter | Type | Description | +|---|---|---| +| `value` | number | Sharpening intensity (0 to 100). Mapped to a Gaussian sigma of 0.5-10. | + +### sharpen-advanced + +Advanced sharpening with three selectable methods and an optional noise-reduction pre-pass. + +| Parameter | Type | Description | +|---|---|---| +| `method` | string | `adaptive`, `unsharp-mask`, or `high-pass` | +| `sigma` | number | Gaussian blur radius, 0.5-10 (adaptive) | +| `m1` | number | Flat-area sharpening, 0-10 (adaptive) | +| `m2` | number | Textured-area sharpening, 0-20 (adaptive) | +| `x1` | number | Flat/jagged threshold, 0-10 (adaptive) | +| `y2` | number | Max brightening (halo clamp), 0-50 (adaptive) | +| `y3` | number | Max darkening (halo clamp), 0-50 (adaptive) | +| `amount` | number | Intensity percentage, 0-500 (unsharp-mask) | +| `radius` | number | Blur radius, 0.1-5.0 (unsharp-mask) | +| `threshold` | number | Minimum edge brightness, 0-255 (unsharp-mask) | +| `strength` | number | Blend strength, 0-100 (high-pass) | +| `kernelSize` | number | `3` or `5` for 3x3 / 5x5 kernel (high-pass) | +| `denoise` | string | Noise reduction pre-pass: `off`, `light`, `medium`, or `strong` | + +Parameters are method-specific. Only supply the ones relevant to the chosen method. + +### color-blindness + +Simulate a color vision deficiency using a 3x3 color-recombination matrix. + +| Parameter | Type | Description | +|---|---|---| +| `type` | string | One of: `protanopia`, `deuteranopia`, `tritanopia`, `protanomaly`, `deuteranomaly`, `tritanomaly`, `achromatopsia`, `blueConeMonochromacy` | + +### edit-metadata + +Write or remove individual EXIF/IPTC metadata fields without stripping the entire block. + +| Parameter | Type | Description | +|---|---|---| +| `artist` | string | EXIF Artist tag | +| `copyright` | string | EXIF Copyright tag | +| `imageDescription` | string | EXIF ImageDescription tag | +| `software` | string | EXIF Software tag | +| `dateTime` | string | EXIF DateTime tag | +| `dateTimeOriginal` | string | EXIF DateTimeOriginal tag | +| `clearGps` | boolean | Remove all GPS tags | +| `fieldsToRemove` | string[] | List of EXIF field names to delete | + +All parameters are optional. Fields listed in `fieldsToRemove` are deleted from the existing EXIF block. Fields set via the named parameters are written (or overwritten). Binary/unsafe keys like MakerNote are silently ignored. ## Format detection The engine detects input formats automatically from file headers, not just file extensions. This means a `.jpg` file that is actually a PNG will be handled correctly. Detection uses a multi-layer approach: magic bytes first, then file extension as fallback. -SnapOtter supports **55+ input formats** and **14 output formats**, including 23 camera RAW formats from 20+ brands, professional formats (PSD, EPS, OpenEXR, HDR), modern codecs (JPEG XL, AVIF, HEIC, QOI, JPEG 2000), and scientific/gaming formats (FITS, DDS). Decoding is handled by Sharp natively where possible, with automatic fallback to ImageMagick, LibRaw, and specialized CLI decoders. +SnapOtter supports **55+ input formats** and **13 output formats**, including 23 camera RAW formats from 20+ brands, professional formats (PSD, EPS, OpenEXR, HDR), modern codecs (JPEG XL, AVIF, HEIC, QOI, JPEG 2000), and scientific/gaming formats (FITS, DDS). Decoding is handled by Sharp natively where possible, with automatic fallback to ImageMagick, LibRaw, and specialized CLI decoders. See the [Supported Formats](/guide/supported-formats) page for the complete list. ## Metadata extraction -The `info` tool returns image metadata: +The `info` tool returns image metadata. See [Image Info](/tools/image/info) for the full field reference. ```json { - "width": 1920, - "height": 1080, + "filename": "photo.jpg", + "fileSize": 2450000, + "width": 4032, + "height": 3024, "format": "jpeg", - "size": 245678, "channels": 3, "hasAlpha": false, - "dpi": 72, - "exif": { ... } + "colorSpace": "srgb", + "density": 72, + "isProgressive": false, + "hasExif": true, + "hasIcc": true, + "hasXmp": false, + "bitDepth": "8", + "pages": 1, + "histogram": [ + { "channel": "red", "min": 0, "max": 255, "mean": 128.45, "stdev": 52.31 }, + { "channel": "green", "min": 2, "max": 253, "mean": 115.22, "stdev": 48.76 }, + { "channel": "blue", "min": 0, "max": 250, "mean": 102.89, "stdev": 55.14 } + ] } ``` diff --git a/apps/docs/api/rest.md b/apps/docs/api/rest.md index a18ccbe5..a3d272f4 100644 --- a/apps/docs/api/rest.md +++ b/apps/docs/api/rest.md @@ -102,8 +102,9 @@ curl -X POST http://localhost:1349/api/v1/tools//batch \ - Upload is `multipart/form-data`. - `settings` is a JSON string with tool-specific options. -- Response is the processed file directly (or a ZIP for batch). -- Progress is tracked via SSE (see [Progress Tracking](#progress-tracking)). +- **Fast tools** (200) return JSON: `{"jobId":"...","downloadUrl":"/api/v1/download//","originalSize":1234,"processedSize":567}`. Fetch the processed file from `downloadUrl`. +- **Long-running tools** (202) return JSON: `{"jobId":"...","async":true}`. Connect to SSE for progress, then download when complete (see [Progress Tracking](#progress-tracking)). +- **Batch** returns a ZIP archive streamed directly (with `X-Job-Id` header). ## Tools Reference @@ -163,6 +164,7 @@ All AI tools run on your hardware (CPU or NVIDIA GPU). No internet required. | `transparency-fixer` | PNG Transparency Fixer | BiRefNet HR-matting | `defringe` (0-100), `outputFormat` (png/webp) | | `background-replace` | Background Replace | rembg (BiRefNet) | `backgroundType` (color/gradient), `color` (hex), `gradientColor1`, `gradientColor2`, `gradientAngle`, `feather` (0-20), `format` (png/webp) | | `blur-background` | Blur Background | rembg (BiRefNet) | `intensity` (1-100), `feather` (0-20), `format` (png/webp) | +| `ai-canvas-expand` | AI Canvas Expand | LaMa (outpainting) | `extendTop`, `extendRight`, `extendBottom`, `extendLeft` (px), `tier` (fast/balanced/high), `format`, `quality` | ### Watermark & Overlay @@ -435,7 +437,7 @@ Each step's output is the next step's input. Unlimited steps per pipeline by def Long-running jobs (AI tools, batch, pipelines) emit real-time progress via Server-Sent Events: ```bash -# Connect to the SSE stream (jobId returned in X-Job-Id response header) +# Connect to the SSE stream (jobId is in the JSON response body from the tool endpoint) curl -N http://localhost:1349/api/v1/jobs//progress \ -H "Authorization: Bearer " ``` @@ -505,7 +507,7 @@ Custom role management with granular permissions. | `PUT` | `/api/v1/roles/:id` | Admin (`users:manage`) | Update a custom role (cannot modify built-in roles) | | `DELETE` | `/api/v1/roles/:id` | Admin (`users:manage`) | Delete a custom role (cannot delete built-in roles; affected users revert to `user` role) | -Available permissions: `tools:use`, `files:own`, `files:all`, `apikeys:own`, `apikeys:all`, `pipelines:own`, `pipelines:all`, `settings:read`, `settings:write`, `users:manage`, `teams:manage`, `features:manage`, `system:health`, `audit:read`. +Available permissions (17): `tools:use`, `files:own`, `files:all`, `apikeys:own`, `apikeys:all`, `pipelines:own`, `pipelines:all`, `settings:read`, `settings:write`, `users:manage`, `teams:manage`, `features:manage`, `system:health`, `audit:read`, `compliance:manage`, `webhooks:manage`, `security:manage`. ## Audit Log diff --git a/apps/docs/guide/supported-formats.md b/apps/docs/guide/supported-formats.md index c17191b1..5dfeee04 100644 --- a/apps/docs/guide/supported-formats.md +++ b/apps/docs/guide/supported-formats.md @@ -8,7 +8,7 @@ SnapOtter processes files across five modalities: image, video, audio, document, ## Image Formats -SnapOtter supports 55+ image formats for input and 14 formats for output. +SnapOtter supports 55+ image formats for input and 13 formats for output. ## Input Formats @@ -101,7 +101,7 @@ SnapOtter supports 55+ image formats for input and 14 formats for output. | PAM | .pam | Sharp (native) | Arbitrary map | | PFM | .pfm | Sharp (native) | Float map | -## Output Formats (14) +## Output Formats (13) | Format | Encoder | Quality Control | Available In | |--------|---------|----------------|-------------|