External/SnapOtter
1
0
Fork 0
mirror of https://github.com/snapotter-hq/SnapOtter.git synced 2026-08-03 07:46:42 +02:00
Code Issues Packages Projects Releases Wiki Activity

docs: deep-audit the API reference pages against code

Browse Source 
rest.md: correct the tool response shape (JSON {jobId, downloadUrl, ...} for fast / 202 {jobId, async} for long, not the raw file), fix the SSE jobId source (body, with X-Job-Id only on batch/pipeline ZIP streams), complete the permissions list to all 17, and add ai-canvas-expand.

ai.md: reconcile ~14 AI tool parameter tables to the real Zod schemas (remove-background, upscale, ocr, enhance-faces, colorize, noise-removal, red-eye-removal, restore-photo, passport-photo, erase-object, smart-crop, content-aware-resize, transparency-fixer, image-enhancement), add the 6 missing AI tools (background-replace, blur-background, ocr-pdf, ai-canvas-expand, transcribe-audio, auto-subtitles), and fix the dispatcher script diagram and feature-bundle table.

image-engine.md: add 4 undocumented operations (sharpen, sharpen-advanced, color-blindness, edit-metadata) and fix flip/crop/rotate/convert/compress/strip-metadata/color-channels params, output-format count (13), and the info JSON example. supported-formats.md: output count 14 -> 13. Verified with a clean docs build.
This commit is contained in:
SnapOtter
2026-06-18 14:12:28 +08:00
parent 8532f3227b
commit 60e3008106
4 changed files with 377 additions and 161 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/docs/api/ai.md
+273 -139
View File
@@ -6,119 +6,173 @@ description: AI engine reference with all local ML tools. Background removal, up
The `@snapotter/ai` package bridges Node.js to a **persistent Python sidecar** for all ML operations. The dispatcher process stays alive between requests for fast warm-start performance. GPU is auto-detected at startup and used when available.
19 AI tool routes. All models run locally - no internet required after initial model download.
19 Python sidecar AI tools across four modalities (image, audio, video, document), plus 2 tools with optional AI capabilities. All models run locally -- no internet required after initial model download.
## Architecture
```
Node.js Tool Route
│
▼
|
v
@snapotter/ai bridge.ts
│ (stdin/stdout JSON + stderr progress events)
▼
Python dispatcher (persistent process)
│
├─ remove_bg.py (rembg / BiRefNet)
├─ upscale.py (RealESRGAN)
├─ inpaint.py (LaMa ONNX)
├─ ocr.py (PaddleOCR / Tesseract)
├─ detect_faces.py (MediaPipe)
├─ face_landmarks.py (MediaPipe landmarks)
├─ enhance_faces.py (GFPGAN / CodeFormer)
├─ colorize.py (DDColor)
├─ noise_removal.py (tiered denoising)
├─ red_eye_removal.py (landmark + color analysis)
├─ restore.py (scratch repair + enhancement + denoising)
├─ transparency_fix.py (BiRefNet HR-matting + defringe)
└─ seam_carving (Go caire binary - not Python)
| (stdin/stdout JSON + stderr progress events)
v
Python dispatcher (persistent process, "ai" profile)
|
|-- remove_bg.py (rembg / BiRefNet)
|-- upscale.py (RealESRGAN)
|-- inpaint.py (LaMa ONNX)
|-- outpaint.py (LaMa canvas expansion)
|-- ocr.py (PaddleOCR / Tesseract)
|-- ocr_pdf.py (page-by-page document OCR)
|-- ocr_preprocess.py (image enhancement for OCR)
|-- detect_faces.py (MediaPipe)
|-- face_landmarks.py (MediaPipe landmarks)
|-- enhance_faces.py (GFPGAN / CodeFormer)
|-- colorize.py (DDColor)
|-- noise_removal.py (SCUNet / tiered denoising)
|-- red_eye_removal.py (landmark + color analysis)
|-- restore.py (scratch repair + enhancement + denoising)
|-- transcribe.py (faster-whisper speech-to-text)
+-- install_feature.py (on-demand bundle installer)
```
A separate "docs" dispatcher profile replaces the AI allowlist with document-processing scripts (`doc_pagecount`, `doc_health`, `doc_flatten`, `doc_redact`, `doc_text`, `doc_to_word`, `doc_metadata`, `doc_html_pdf`) and skips heavy ML imports.
**Timeouts:** 300 s default; OCR and BiRefNet background removal get 600 s.
## Feature Bundles
Each AI tool requires a model bundle to be installed before use. Bundles are installed on demand via the admin UI or `install_feature.py`.
| Bundle | Size | Tools |
|--------|------|-------|
| `background-removal` | 4-5 GB | remove-background, passport-photo, transparency-fixer, background-replace, blur-background |
| `face-detection` | 200-300 MB | blur-faces, red-eye-removal, smart-crop |
| `object-eraser-colorize` | 1-2 GB | erase-object, colorize, ai-canvas-expand |
| `upscale-enhance` | 4-5 GB | upscale, enhance-faces, noise-removal |
| `photo-restoration` | 800 MB - 1 GB | restore-photo |
| `ocr` | 3-4 GB | ocr, ocr-pdf |
| `transcription` | ~600 MB | transcribe-audio, auto-subtitles |
---
## Background Removal
**Function:** `removeBackground`
**Tool route:** `remove-background`
**Model:** rembg with BiRefNet (default) or U2-Net variants
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `model` | string | `birefnet-general` | Model variant - see table below |
| `alphaMattingForeground` | number (1–255) | 240 | Foreground threshold for alpha matting |
| `alphaMattingBackground` | number (1–255) | 10 | Background threshold for alpha matting |
| `returnMask` | boolean | false | Return the mask instead of the cutout |
| `backgroundColor` | string | - | Fill removed area (hex color or "transparent") |
| `model` | string | - | Model variant (optional override) |
| `backgroundType` | string | `"transparent"` | One of: `transparent`, `color`, `gradient`, `blur`, `image` |
| `backgroundColor` | string | - | Hex color for solid background |
| `gradientColor1` | string | - | First gradient color |
| `gradientColor2` | string | - | Second gradient color |
| `gradientAngle` | number | - | Gradient angle in degrees |
| `blurEnabled` | boolean | - | Enable background blur effect |
| `blurIntensity` | number (0-100) | - | Blur intensity |
| `shadowEnabled` | boolean | - | Enable drop shadow on subject |
| `shadowOpacity` | number (0-100) | - | Shadow opacity |
| `outputFormat` | string | - | Output format: `png`, `webp`, or `avif` |
| `edgeRefine` | integer (0-3) | - | Edge refinement level |
| `decontaminate` | boolean | - | Remove color bleed from edges |
**Available models:**
## Background Replace
| Model ID | Best for |
|----------|---------|
| `birefnet-general` | General purpose (default) |
| `birefnet-portrait` | People / portraits |
| `birefnet-dis` | Dichotomous Image Segmentation |
| `birefnet-hrsod` | High-resolution salient objects |
| `birefnet-cod` | Camouflaged objects |
| `u2net` | Fast general purpose |
| `u2net_human_seg` | Human segmentation |
| `isnet-general-use` | High quality general |
**Tool route:** `background-replace`
**Model:** rembg / BiRefNet (shared with remove-background)
## Image Upscaling
**Function:** `upscale`
**Tool route:** `upscale`
**Model:** RealESRGAN (with Lanczos fallback on CPU-constrained systems)
Removes the background and replaces it with a solid color or gradient.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `scale` | 2 \| 4 | 4 | Upscale factor |
| `model` | string | `realesrgan-x4plus` | Model variant |
| `faceEnhance` | boolean | false | Apply GFPGAN face enhancement pass |
| `denoise` | number (0–1) | 0.5 | Denoising strength |
| `format` | string | - | Output format override |
| `quality` | number | 95 | Output quality (for JPEG/WebP) |
| `backgroundType` | `"color"` \| `"gradient"` | `"color"` | Background mode |
| `color` | string | `"#ffffff"` | Background hex color (when `backgroundType` is `color`) |
| `gradientColor1` | string | - | First gradient hex color |
| `gradientColor2` | string | - | Second gradient hex color |
| `gradientAngle` | integer (0-360) | `180` | Gradient angle in degrees |
| `feather` | integer (0-20) | `0` | Edge feathering radius |
| `format` | `"png"` \| `"webp"` | `"png"` | Output format |
## Blur Background
**Tool route:** `blur-background`
**Model:** rembg / BiRefNet (shared with remove-background)
Blurs the background while keeping the subject sharp.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `intensity` | integer (1-100) | `50` | Blur intensity |
| `feather` | integer (0-20) | `0` | Edge feathering radius |
| `format` | `"png"` \| `"webp"` | `"png"` | Output format |
## Image Upscaling
**Tool route:** `upscale`
**Model:** RealESRGAN (with Lanczos fallback when unavailable)
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `scale` | number | `2` | Upscale factor |
| `model` | string | `"auto"` | Model variant |
| `faceEnhance` | boolean | `false` | Apply GFPGAN face enhancement pass |
| `denoise` | number | `0` | Denoising strength |
| `format` | string | `"auto"` | Output format override |
| `quality` | number | `95` | Output quality (1-100) |
## OCR / Text Extraction
**Function:** `extractText`
**Tool route:** `ocr`
**Models:** Tesseract (fast), PaddleOCR PP-OCRv5 (balanced), PaddleOCR-VL 1.5 (best)
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `quality` | `fast` \| `balanced` \| `best` | `balanced` | Processing tier |
| `language` | string | `en` | Language code (ISO 639-1) |
| `enhance` | boolean | false | Pre-process image to improve OCR accuracy |
| `quality` | `"fast"` \| `"balanced"` \| `"best"` | `"balanced"` | Processing tier |
| `language` | string | `"auto"` | Language: `auto`, `en`, `de`, `fr`, `es`, `zh`, `ja`, `ko` |
| `enhance` | boolean | `true` | Pre-process image to improve OCR accuracy |
| `engine` | string | - | Deprecated. Maps `tesseract` to `fast`, `paddleocr` to `balanced` |
Returns structured results with bounding boxes, confidence scores, and extracted text blocks.
## PDF OCR
**Tool route:** `ocr-pdf`
**Models:** Same tier system as image OCR
Extracts text from scanned PDF documents using AI-powered OCR, page by page.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `quality` | `"fast"` \| `"balanced"` \| `"best"` | `"balanced"` | Processing tier |
| `language` | string | `"auto"` | Language: `auto`, `en`, `de`, `fr`, `es`, `zh`, `ja`, `ko` |
| `pages` | string | `"all"` | Page selection: `"all"`, `"1-3"`, `"1,3,5"` |
## Face / PII Blur
**Function:** `blurFaces`
**Tool route:** `blur-faces`
**Model:** MediaPipe face detection
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `blurRadius` | number | 30 | Gaussian blur radius |
| `sensitivity` | number (0–1) | 0.5 | Detection confidence threshold |
| `blurRadius` | number (1-100) | `30` | Gaussian blur radius |
| `sensitivity` | number (0-1) | `0.5` | Detection confidence threshold |
## Face Enhancement
**Function:** `enhanceFaces`
**Tool route:** `enhance-faces`
**Models:** GFPGAN, CodeFormer
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `model` | `gfpgan` \| `codeformer` | `gfpgan` | Enhancement model |
| `strength` | number (0–1) | 0.7 | Enhancement strength |
| `sensitivity` | number (0–1) | 0.5 | Face detection threshold |
| `centerFace` | boolean | false | Focus enhancement on center face only |
| `model` | `"auto"` \| `"gfpgan"` \| `"codeformer"` | `"auto"` | Enhancement model |
| `strength` | number (0-1) | `0.8` | Enhancement strength |
| `sensitivity` | number (0-1) | `0.5` | Face detection threshold |
| `onlyCenterFace` | boolean | `false` | Enhance only the most central face |
## AI Colorization
**Function:** `colorize`
**Tool route:** `colorize`
**Model:** DDColor (with OpenCV DNN fallback)
@@ -126,144 +180,182 @@ Converts black-and-white or grayscale photos to full color.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `intensity` | number (0–1) | 0.85 | Color saturation strength |
| `model` | string | `ddcolor` | Model variant |
| `intensity` | number (0-1) | `1.0` | Color saturation strength |
| `model` | `"auto"` \| `"ddcolor"` \| `"opencv"` | `"auto"` | Model variant |
## Noise Removal
**Function:** `noiseRemoval`
**Tool route:** `noise-removal`
Three-tier denoising pipeline (fast: OpenCV bilateral filter; balanced: frequency-domain; best: deep learning model).
**Tool route:** `noise-removal`
**Model:** SCUNet (tiered denoising pipeline)
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `quality` | `fast` \| `balanced` \| `best` | `balanced` | Processing tier |
| `strength` | number (0–1) | 0.5 | Denoising strength |
| `preserveDetail` | boolean | true | Edge-preserving mode |
| `colorNoise` | boolean | false | Target color noise specifically |
| `tier` | `"quick"` \| `"balanced"` \| `"quality"` \| `"maximum"` | `"balanced"` | Processing tier |
| `strength` | number (0-100) | `50` | Denoising strength |
| `detailPreservation` | number (0-100) | `50` | How much detail to preserve; higher keeps more texture |
| `colorNoise` | number (0-100) | `30` | Color noise reduction strength |
| `format` | string | `"original"` | Output format: `original`, `png`, `jpeg`, `webp`, `avif`, `jxl` |
| `quality` | number (1-100) | `90` | Output encoding quality |
## Red Eye Removal
**Function:** `removeRedEye`
**Tool route:** `red-eye-removal`
Detects face landmarks, locates eye regions, and corrects red-channel oversaturation.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `sensitivity` | number (0–1) | 0.5 | Red pixel detection threshold |
| `strength` | number (0–1) | 0.9 | Correction strength |
| `sensitivity` | number (0-100) | `50` | Red pixel detection threshold |
| `strength` | number (0-100) | `70` | Correction strength |
| `format` | string | - | Output format override (optional) |
| `quality` | number (1-100) | `90` | Output quality |
## Photo Restoration
**Function:** `restorePhoto`
**Tool route:** `restore-photo`
Multi-step pipeline for old or damaged photos: scratch/tear detection and repair → face enhancement → denoising → optional colorization.
Multi-step pipeline for old or damaged photos: scratch/tear detection and repair, face enhancement, denoising, and optional colorization.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `mode` | `auto` \| `light` \| `heavy` | `auto` | Restoration intensity |
| `scratchRemoval` | boolean | true | Detect and repair scratches, tears |
| `faceEnhancement` | boolean | true | Apply face enhancement pass |
| `fidelity` | number (0–1) | 0.7 | Face enhancement strength |
| `denoise` | boolean | true | Apply denoising pass |
| `denoiseStrength` | number (0–100) | 40 | Denoising strength |
| `colorize` | boolean | false | Colorize after restoration |
| `scratchRemoval` | boolean | `true` | Detect and repair scratches, tears |
| `faceEnhancement` | boolean | `true` | Apply face enhancement pass |
| `fidelity` | number (0-1) | `0.7` | Face enhancement strength (higher = more conservative) |
| `denoise` | boolean | `true` | Apply denoising pass |
| `denoiseStrength` | number (0-100) | `25` | Denoising strength |
| `colorize` | boolean | `false` | Colorize after restoration |
| `colorizeStrength` | number (0-100) | `85` | Colorization intensity |
## Passport Photo
**Function:** Uses `detectFaceLandmarks` + `removeBackground`
**Tool route:** `passport-photo`
**Model:** MediaPipe face landmarks
**Models:** MediaPipe face landmarks + BiRefNet background removal
Generates government-compliant ID photos. Supports **37 countries** across 6 regions (Americas, Europe, Asia, Africa, Oceania, Middle East). Each spec includes physical dimensions, DPI, head-height ratio, eye-line position, and background color requirements.
Two-phase workflow: analyze (detect face + remove background) then generate (crop, resize, tile). Supports 37+ countries across 6 regions.
### Phase 1: Analyze
`POST /api/v1/tools/passport-photo/analyze`
Accepts an image file (multipart). Returns face landmark data, a base64 preview, and image dimensions.
### Phase 2: Generate
`POST /api/v1/tools/passport-photo/generate`
Accepts a JSON body with the Phase 1 results plus generation settings:
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `country` | string | `us` | ISO country code (see list in UI) |
| `printLayout` | `4x6` \| `A4` \| `none` | `none` | Output as print sheet or standalone |
| `backgroundColor` | string | country default | Background fill color |
| `jobId` | string | (required) | Job ID from Phase 1 |
| `filename` | string | (required) | Original filename from Phase 1 |
| `countryCode` | string | (required) | ISO country code (e.g., `US`, `GB`, `IN`) |
| `documentType` | string | `"passport"` | Document type |
| `bgColor` | string | `"#FFFFFF"` | Background color hex |
| `printLayout` | string | `"none"` | Print layout: `none`, `4x6`, `a4`, `letter` |
| `maxFileSizeKb` | number | `0` | Max file size in KB (0 = no limit) |
| `dpi` | number (72-1200) | `300` | Output DPI |
| `customWidthMm` | number | - | Custom width in mm (overrides country spec) |
| `customHeightMm` | number | - | Custom height in mm (overrides country spec) |
| `zoom` | number (0.5-3) | `1` | Zoom factor |
| `adjustX` | number | `0` | Horizontal position adjustment |
| `adjustY` | number | `0` | Vertical position adjustment |
| `landmarks` | object | (required) | Landmarks from Phase 1 |
| `imageWidth` | number | (required) | Image width from Phase 1 |
| `imageHeight` | number | (required) | Image height from Phase 1 |
## Object Erasing (Inpainting)
**Function:** `inpaint`
**Tool route:** `erase-object`
**Model:** LaMa via ONNX Runtime
| Parameter | Type | Required | Description |
The mask is sent as a **second file part** (fieldname `mask`), not as base64. White pixels in the mask indicate areas to erase. The `format` and `quality` settings are sent as top-level form fields.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `maskData` | string | Yes | Base64-encoded PNG mask (white = erase) |
| `maskThreshold` | number (0–255) | No | Threshold for mask binarization |
| `file` | file | (required) | Source image (multipart) |
| `mask` | file | (required) | Mask image (multipart, fieldname `mask`, white = erase) |
| `format` | string | `"auto"` | Output format: `auto`, `png`, `jpg`, `jpeg`, `webp`, `tiff`, `gif`, `avif`, `heic`, `heif`, `jxl` |
| `quality` | integer (1-100) | `95` | Output quality |
GPU-accelerated when an NVIDIA GPU is available.
## Smart Crop
## AI Canvas Expand
**Function:** Uses MediaPipe + Sharp attention/entropy
**Tool route:** `smart-crop`
**Model:** MediaPipe face detection
**Tool route:** `ai-canvas-expand`
**Model:** LaMa-based outpainting
Expands the canvas of an image in any direction and fills new areas with AI-generated content that matches the existing image.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `mode` | `subject` \| `face` \| `trim` | `subject` | Crop strategy |
| `width` | number | - | Output width |
| `height` | number | - | Output height |
| `facePreset` | string | - | Preset framing when `mode=face` |
| `extendTop` | integer | `0` | Pixels to extend at the top |
| `extendRight` | integer | `0` | Pixels to extend at the right |
| `extendBottom` | integer | `0` | Pixels to extend at the bottom |
| `extendLeft` | integer | `0` | Pixels to extend at the left |
| `tier` | `"fast"` \| `"balanced"` \| `"high"` | `"balanced"` | Quality tier |
| `format` | string | `"auto"` | Output format: `auto`, `png`, `jpg`, `jpeg`, `webp`, `tiff`, `gif`, `avif`, `heic`, `heif`, `jxl` |
| `quality` | integer (1-100) | `95` | Output quality |
At least one extend direction must be greater than 0.
## Smart Crop
**Tool route:** `smart-crop`
**Model:** MediaPipe face detection (face mode only)
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `mode` | string | `"subject"` | Crop strategy: `subject`, `face`, `trim` |
| `strategy` | `"attention"` \| `"entropy"` | `"attention"` | Strategy for subject mode |
| `width` | integer | - | Output width |
| `height` | integer | - | Output height |
| `padding` | integer (0-50) | `0` | Padding percentage around subject |
| `facePreset` | string | `"head-shoulders"` | Preset framing when `mode=face` |
| `sensitivity` | number (0-1) | `0.5` | Face detection threshold |
| `threshold` | integer (0-255) | `30` | Background detection threshold (trim mode) |
| `padToSquare` | boolean | `false` | Pad trimmed result to a square |
| `padColor` | string | `"#ffffff"` | Background color for square padding |
| `targetSize` | integer | - | Target size for padded output (pixels) |
| `quality` | integer (1-100) | - | Output quality |
Legacy `mode` values `attention` and `content` are accepted and mapped to `subject` and `trim` respectively.
**Face presets:**
| Preset | Head ratio | Best for |
|--------|-----------|---------|
| `close-up` | 1.8× face | Headshots |
| `head-and-shoulders` | 2.8× face | Profile photos |
| `upper-body` | 4.5× face | LinkedIn / formal |
| `half-body` | 7.0× face | Full upper body |
| Preset | Best for |
|--------|---------|
| `closeup` | Headshots |
| `head-shoulders` | Profile photos |
| `upper-body` | LinkedIn / formal |
| `half-body` | Full upper body |
## Image Enhancement
## Transcribe Audio
**Function:** `analyzeImage` + `applyCorrections`
**Tool route:** `image-enhancement`
**Engine:** Analysis-based (Sharp histogram and statistics)
**Tool route:** `transcribe-audio`
**Model:** faster-whisper
Analyzes the image and applies automatic corrections for exposure, contrast, white balance, saturation, sharpness, and noise. Supports scene-specific modes.
Converts speech to text. Supports plain text, SRT, and VTT output formats.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `mode` | `auto` \| `portrait` \| `landscape` \| `low-light` \| `food` \| `document` | `auto` | Scene mode for tuning corrections |
| `intensity` | number (0-100) | 50 | Overall correction strength |
| `corrections.exposure` | boolean | true | Apply exposure correction |
| `corrections.contrast` | boolean | true | Apply contrast correction |
| `corrections.whiteBalance` | boolean | true | Apply white balance correction |
| `corrections.saturation` | boolean | true | Apply saturation correction |
| `corrections.sharpness` | boolean | true | Apply sharpness correction |
| `corrections.denoise` | boolean | true | Apply denoising |
| `language` | string | `"auto"` | Language: `auto`, `en`, `de`, `fr`, `es`, `zh`, `ja`, `ko`, `id`, `th`, `vi` |
| `outputFormat` | `"txt"` \| `"srt"` \| `"vtt"` | `"txt"` | Output format |
An additional analysis endpoint is available at `POST /api/v1/tools/image-enhancement/analyze` which returns the detected corrections without applying them.
## Auto Subtitles
## Content-Aware Resize (Seam Carving)
**Tool route:** `auto-subtitles`
**Model:** faster-whisper (extracts audio from video, then transcribes)
**Function:** `seamCarve`
**Tool route:** `content-aware-resize`
**Engine:** Go `caire` binary (not Python - no GPU benefit)
Intelligently resizes images by removing or adding low-energy seams, preserving important content.
Generates subtitle files from a video's audio track.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `width` | number | - | Target width |
| `height` | number | - | Target height |
| `protectFaces` | boolean | true | Protect detected face regions from seam removal |
| `blurRadius` | number | 0 | Pre-blur to reduce noise sensitivity |
| `sobelThreshold` | number | 10 | Edge sensitivity threshold |
| `square` | boolean | false | Force square output |
Max input edge before auto-downscaling: **1200 px**.
| `language` | string | `"auto"` | Language: `auto`, `en`, `de`, `fr`, `es`, `zh`, `ja`, `ko`, `id`, `th`, `vi` |
| `format` | `"srt"` \| `"vtt"` | `"srt"` | Output subtitle format |
## PNG Transparency Fixer
**Function:** `fixTransparency`
**Tool route:** `transparency-fixer`
**Model:** BiRefNet HR-matting (2048x2048 resolution)
@@ -271,12 +363,11 @@ Fixes "fake transparent" PNGs where the background was removed but left behind f
**OOM fallback chain:** If BiRefNet HR-matting exceeds available memory, the tool automatically falls back to `birefnet-general`, then to `u2net`.
**Feature bundle:** Background Removal (shared with Remove Background and Passport Photo).
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `defringe` | number (0-100) | 30 | Edge defringe strength to remove color contamination |
| `defringe` | number (0-100) | `30` | Edge defringe strength to remove color contamination |
| `outputFormat` | `"png"` \| `"webp"` | `"png"` | Output image format |
| `removeWatermark` | boolean | `false` | Apply watermark removal pre-processing (median filter) |
```bash
curl -X POST http://localhost:1349/api/v1/tools/transparency-fixer \
@@ -284,3 +375,46 @@ curl -X POST http://localhost:1349/api/v1/tools/transparency-fixer \
-F "file=@fake-transparent.png" \
-F 'settings={"defringe":30,"outputFormat":"png"}'
```
---
## Tools with Optional AI Capabilities
The following tools are not Python sidecar tools but use AI features when certain options are enabled.
### Image Enhancement
**Tool route:** `image-enhancement`
**Engine:** Analysis-based (Sharp histogram and statistics)
Analyzes the image and applies automatic corrections for exposure, contrast, white balance, saturation, sharpness, and noise. Supports scene-specific modes.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `mode` | `"auto"` \| `"portrait"` \| `"landscape"` \| `"low-light"` \| `"food"` \| `"document"` | `"auto"` | Scene mode for tuning corrections |
| `intensity` | number (0-100) | `50` | Overall correction strength |
| `corrections.exposure` | boolean | `true` | Apply exposure correction |
| `corrections.contrast` | boolean | `true` | Apply contrast correction |
| `corrections.whiteBalance` | boolean | `true` | Apply white balance correction |
| `corrections.saturation` | boolean | `true` | Apply saturation correction |
| `corrections.sharpness` | boolean | `true` | Apply sharpness correction |
| `corrections.denoise` | boolean | `true` | Apply denoising |
| `deepEnhance` | boolean | `false` | Enable AI noise removal via SCUNet (requires `upscale-enhance` bundle) |
An additional analysis endpoint is available at `POST /api/v1/tools/image-enhancement/analyze` which returns the detected corrections without applying them.
### Content-Aware Resize (Seam Carving)
**Tool route:** `content-aware-resize`
**Engine:** Go `caire` binary (not Python -- no GPU benefit)
Intelligently resizes images by removing low-energy seams, preserving important content.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `width` | number | - | Target width |
| `height` | number | - | Target height |
| `protectFaces` | boolean | `false` | Protect detected face regions (requires `face-detection` bundle) |
| `blurRadius` | number (0-20) | `4` | Pre-blur for energy calculation |
| `sobelThreshold` | number (1-20) | `2` | Edge sensitivity threshold |
| `square` | boolean | `false` | Force square output |
apps/docs/api/image-engine.md
+96 -16
View File
@@ -32,6 +32,7 @@ Cut out a rectangular region from the image.
| `top` | number | Y offset from the top edge |
| `width` | number | Width of the crop area |
| `height` | number | Height of the crop area |
| `unit` | string | `px` (default) or `percent` |
### rotate
@@ -40,15 +41,16 @@ Rotate the image by a given angle.
| Parameter | Type | Description |
|---|---|---|
| `angle` | number | Rotation angle in degrees (0-360) |
| `background` | string | Fill color for the exposed area (default: transparent or white) |
| `background` | string | Fill color for exposed area (default: `#000000`). Only applies to non-90-degree angles. |
### flip
Mirror the image horizontally or vertically.
Mirror the image horizontally, vertically, or both. At least one must be true.
| Parameter | Type | Description |
|---|---|---|
| `direction` | string | `horizontal` or `vertical` |
| `horizontal` | boolean | Mirror left to right |
| `vertical` | boolean | Mirror top to bottom |
### convert
@@ -56,9 +58,11 @@ Change the image format.
| Parameter | Type | Description |
|---|---|---|
| `format` | string | Target format: `jpeg`, `png`, `webp`, `avif`, `tiff`, `gif`, `jxl`, `heic`, `heif`, `bmp`, `ico`, `jp2`, `qoi` |
| `format` | string | Target format: `jpg`, `png`, `webp`, `avif`, `tiff`, `gif`, `jxl`, `heic`, `heif`, `bmp`, `ico`, `jp2`, `qoi` |
| `quality` | number | Compression quality (1-100, applies to lossy formats) |
The first seven formats (`jpg` through `jxl`) are encoded by Sharp in-process. The remaining formats use external encoders at the API layer: `heic`/`heif` via heif-enc, `bmp`/`ico` via ImageMagick, `jp2` via opj_compress, and `qoi` via an inline TypeScript codec.
### compress
Reduce file size while keeping the same format.
@@ -66,11 +70,20 @@ Reduce file size while keeping the same format.
| Parameter | Type | Description |
|---|---|---|
| `quality` | number | Target quality (1-100) |
| `targetSizeBytes` | number | Optional target file size in bytes |
| `format` | string | Optional format override |
### strip-metadata
Remove EXIF, IPTC, and XMP metadata from the image. Useful for privacy before sharing photos publicly. Takes no parameters.
Remove EXIF, IPTC, XMP, and ICC metadata from the image. With no parameters (or `stripAll: true`), strips everything. Pass individual flags for selective stripping.
| Parameter | Type | Description |
|---|---|---|
| `stripAll` | boolean | Strip all metadata (default when no flags are set) |
| `stripExif` | boolean | Strip EXIF data (including GPS if `stripGps` is not separately set) |
| `stripGps` | boolean | Strip GPS location data |
| `stripIcc` | boolean | Strip ICC color profile |
| `stripXmp` | boolean | Strip XMP metadata |
### Color adjustments
@@ -94,35 +107,102 @@ These apply a fixed color transformation. They take no parameters.
### Color channels
Adjust individual RGB color channels.
Adjust individual RGB color channels. Values are multipliers where 100 = no change.
| Parameter | Type | Description |
|---|---|---|
| `red` | number | Red channel adjustment (-100 to 100) |
| `green` | number | Green channel adjustment (-100 to 100) |
| `blue` | number | Blue channel adjustment (-100 to 100) |
| `red` | number | Red channel multiplier (0 to 200, 100 = unchanged) |
| `green` | number | Green channel multiplier (0 to 200, 100 = unchanged) |
| `blue` | number | Blue channel multiplier (0 to 200, 100 = unchanged) |
### sharpen
Simple sharpening controlled by a single value.
| Parameter | Type | Description |
|---|---|---|
| `value` | number | Sharpening intensity (0 to 100). Mapped to a Gaussian sigma of 0.5-10. |
### sharpen-advanced
Advanced sharpening with three selectable methods and an optional noise-reduction pre-pass.
| Parameter | Type | Description |
|---|---|---|
| `method` | string | `adaptive`, `unsharp-mask`, or `high-pass` |
| `sigma` | number | Gaussian blur radius, 0.5-10 (adaptive) |
| `m1` | number | Flat-area sharpening, 0-10 (adaptive) |
| `m2` | number | Textured-area sharpening, 0-20 (adaptive) |
| `x1` | number | Flat/jagged threshold, 0-10 (adaptive) |
| `y2` | number | Max brightening (halo clamp), 0-50 (adaptive) |
| `y3` | number | Max darkening (halo clamp), 0-50 (adaptive) |
| `amount` | number | Intensity percentage, 0-500 (unsharp-mask) |
| `radius` | number | Blur radius, 0.1-5.0 (unsharp-mask) |
| `threshold` | number | Minimum edge brightness, 0-255 (unsharp-mask) |
| `strength` | number | Blend strength, 0-100 (high-pass) |
| `kernelSize` | number | `3` or `5` for 3x3 / 5x5 kernel (high-pass) |
| `denoise` | string | Noise reduction pre-pass: `off`, `light`, `medium`, or `strong` |
Parameters are method-specific. Only supply the ones relevant to the chosen method.
### color-blindness
Simulate a color vision deficiency using a 3x3 color-recombination matrix.
| Parameter | Type | Description |
|---|---|---|
| `type` | string | One of: `protanopia`, `deuteranopia`, `tritanopia`, `protanomaly`, `deuteranomaly`, `tritanomaly`, `achromatopsia`, `blueConeMonochromacy` |
### edit-metadata
Write or remove individual EXIF/IPTC metadata fields without stripping the entire block.
| Parameter | Type | Description |
|---|---|---|
| `artist` | string | EXIF Artist tag |
| `copyright` | string | EXIF Copyright tag |
| `imageDescription` | string | EXIF ImageDescription tag |
| `software` | string | EXIF Software tag |
| `dateTime` | string | EXIF DateTime tag |
| `dateTimeOriginal` | string | EXIF DateTimeOriginal tag |
| `clearGps` | boolean | Remove all GPS tags |
| `fieldsToRemove` | string[] | List of EXIF field names to delete |
All parameters are optional. Fields listed in `fieldsToRemove` are deleted from the existing EXIF block. Fields set via the named parameters are written (or overwritten). Binary/unsafe keys like MakerNote are silently ignored.
## Format detection
The engine detects input formats automatically from file headers, not just file extensions. This means a `.jpg` file that is actually a PNG will be handled correctly. Detection uses a multi-layer approach: magic bytes first, then file extension as fallback.
SnapOtter supports **55+ input formats** and **14 output formats**, including 23 camera RAW formats from 20+ brands, professional formats (PSD, EPS, OpenEXR, HDR), modern codecs (JPEG XL, AVIF, HEIC, QOI, JPEG 2000), and scientific/gaming formats (FITS, DDS). Decoding is handled by Sharp natively where possible, with automatic fallback to ImageMagick, LibRaw, and specialized CLI decoders.
SnapOtter supports **55+ input formats** and **13 output formats**, including 23 camera RAW formats from 20+ brands, professional formats (PSD, EPS, OpenEXR, HDR), modern codecs (JPEG XL, AVIF, HEIC, QOI, JPEG 2000), and scientific/gaming formats (FITS, DDS). Decoding is handled by Sharp natively where possible, with automatic fallback to ImageMagick, LibRaw, and specialized CLI decoders.
See the [Supported Formats](/guide/supported-formats) page for the complete list.
## Metadata extraction
The `info` tool returns image metadata:
The `info` tool returns image metadata. See [Image Info](/tools/image/info) for the full field reference.
```json
{
"width": 1920,
"height": 1080,
"filename": "photo.jpg",
"fileSize": 2450000,
"width": 4032,
"height": 3024,
"format": "jpeg",
"size": 245678,
"channels": 3,
"hasAlpha": false,
"dpi": 72,
"exif": { ... }
"colorSpace": "srgb",
"density": 72,
"isProgressive": false,
"hasExif": true,
"hasIcc": true,
"hasXmp": false,
"bitDepth": "8",
"pages": 1,
"histogram": [
{ "channel": "red", "min": 0, "max": 255, "mean": 128.45, "stdev": 52.31 },
{ "channel": "green", "min": 2, "max": 253, "mean": 115.22, "stdev": 48.76 },
{ "channel": "blue", "min": 0, "max": 250, "mean": 102.89, "stdev": 55.14 }
]
}
```
apps/docs/api/rest.md
+6 -4
View File
@@ -102,8 +102,9 @@ curl -X POST http://localhost:1349/api/v1/tools/<toolId>/batch \
- Upload is `multipart/form-data`.
- `settings` is a JSON string with tool-specific options.
- Response is the processed file directly (or a ZIP for batch).
- Progress is tracked via SSE (see [Progress Tracking](#progress-tracking)).
- **Fast tools** (200) return JSON: `{"jobId":"...","downloadUrl":"/api/v1/download/<jobId>/<filename>","originalSize":1234,"processedSize":567}`. Fetch the processed file from `downloadUrl`.
- **Long-running tools** (202) return JSON: `{"jobId":"...","async":true}`. Connect to SSE for progress, then download when complete (see [Progress Tracking](#progress-tracking)).
- **Batch** returns a ZIP archive streamed directly (with `X-Job-Id` header).
## Tools Reference
@@ -163,6 +164,7 @@ All AI tools run on your hardware (CPU or NVIDIA GPU). No internet required.
| `transparency-fixer` | PNG Transparency Fixer | BiRefNet HR-matting | `defringe` (0-100), `outputFormat` (png/webp) |
| `background-replace` | Background Replace | rembg (BiRefNet) | `backgroundType` (color/gradient), `color` (hex), `gradientColor1`, `gradientColor2`, `gradientAngle`, `feather` (0-20), `format` (png/webp) |
| `blur-background` | Blur Background | rembg (BiRefNet) | `intensity` (1-100), `feather` (0-20), `format` (png/webp) |
| `ai-canvas-expand` | AI Canvas Expand | LaMa (outpainting) | `extendTop`, `extendRight`, `extendBottom`, `extendLeft` (px), `tier` (fast/balanced/high), `format`, `quality` |
### Watermark & Overlay
@@ -435,7 +437,7 @@ Each step's output is the next step's input. Unlimited steps per pipeline by def
Long-running jobs (AI tools, batch, pipelines) emit real-time progress via Server-Sent Events:
```bash
# Connect to the SSE stream (jobId returned in X-Job-Id response header)
# Connect to the SSE stream (jobId is in the JSON response body from the tool endpoint)
curl -N http://localhost:1349/api/v1/jobs/<jobId>/progress \
-H "Authorization: Bearer <token>"
```
@@ -505,7 +507,7 @@ Custom role management with granular permissions.
| `PUT` | `/api/v1/roles/:id` | Admin (`users:manage`) | Update a custom role (cannot modify built-in roles) |
| `DELETE` | `/api/v1/roles/:id` | Admin (`users:manage`) | Delete a custom role (cannot delete built-in roles; affected users revert to `user` role) |
Available permissions: `tools:use`, `files:own`, `files:all`, `apikeys:own`, `apikeys:all`, `pipelines:own`, `pipelines:all`, `settings:read`, `settings:write`, `users:manage`, `teams:manage`, `features:manage`, `system:health`, `audit:read`.
Available permissions (17): `tools:use`, `files:own`, `files:all`, `apikeys:own`, `apikeys:all`, `pipelines:own`, `pipelines:all`, `settings:read`, `settings:write`, `users:manage`, `teams:manage`, `features:manage`, `system:health`, `audit:read`, `compliance:manage`, `webhooks:manage`, `security:manage`.
## Audit Log