mirror of
https://github.com/snapotter-hq/SnapOtter.git
synced 2026-08-03 07:46:42 +02:00
docs: add restore-photo quality overhaul design spec
Evidence-based spec informed by diagnostic testing on 4 sample images. Addresses catastrophic scratch over-detection (up to 68.7% false coverage on small images), LaMa 512x512 resolution loss, CodeFormer over-smoothing on small faces, and excessive NLMeans defaults.
This commit is contained in:
SnapOtter
@@ -0,0 +1,182 @@
|
||||
# Restore Photo Quality Overhaul
|
||||
|
||||
**Date:** 2026-05-13
|
||||
**Status:** Draft
|
||||
**Scope:** Fix critical quality issues in the restore-photo pipeline, informed by diagnostic testing on real damaged photos
|
||||
**Supersedes:** `docs/specs/2026-04-15-restore-photo-design.md` (approved but unimplemented; this spec is scoped to the subset of changes backed by diagnostic evidence)
|
||||
|
||||
---
|
||||
|
||||
## Diagnostic Evidence
|
||||
|
||||
Full pipeline was run on 4 sample images with intermediate outputs saved at each step. Key findings:
|
||||
|
||||
| Image | Pixels | Mask Coverage | Actual Damage | Result |
|
||||
|---|---|---|---|---|
|
||||
| images2.jpg | 290x174 | **68.7%** | ~3% | Faces erased, gray blobs |
|
||||
| images.jpg | 188x267 | **30.6%** | ~1% | Face structure altered |
|
||||
| woman-baby1.webp | 400x277 | **23.2%** | ~6% | Baby face plasticky, scratches remain |
|
||||
| ai-old-photo...webp | 768x513 | **6.4%** | ~5% | Reasonable but scratches still visible |
|
||||
|
||||
Root causes identified (ranked by impact):
|
||||
|
||||
1. **Scratch detection massively over-detects** on small images (4-15x worse than reality)
|
||||
2. **LaMa at 512x512** can't fully remove scratches even when correctly detected
|
||||
3. **CodeFormer over-smooths small faces** (< 80px)
|
||||
4. **NLMeans default strength too high** (40 maps to h=7.8, over-smooths compressed images)
|
||||
|
||||
Diagnostic outputs saved at `/tmp/restore-diagnostic/` with per-step intermediate images.
|
||||
|
||||
---
|
||||
|
||||
## Changes
|
||||
|
||||
### 1. Scratch Detection Rewrite (`restore.py :: detect_scratches`)
|
||||
|
||||
**Current:** 4 angles (0/45/90/135), 3 kernel sizes, fixed threshold per mode, no filtering.
|
||||
|
||||
**New:**
|
||||
- **8 angles**: 0, 22.5, 45, 67.5, 90, 112.5, 135, 157.5 degrees
|
||||
- **Rotation-matrix kernels**: `cv2.getRotationMatrix2D` + `cv2.warpAffine` for accurate arbitrary-angle line kernels (replaces hard-coded pixel patterns)
|
||||
- **Image-size-adaptive kernel count**: for images with shortest side < 300px, use only 2 kernel sizes (max kernel capped at `base_dim // 15`). For >= 300px, use 3 sizes as before
|
||||
- **Pre-filtering**: light bilateral filter (`d=5, sigmaColor=50, sigmaSpace=50`) before CLAHE to suppress JPEG/WebP compression artifacts
|
||||
- **Adaptive threshold**: use `cv2.THRESH_BINARY + cv2.THRESH_OTSU` per combined morphological response map instead of a fixed threshold. This auto-adapts to each image's contrast characteristics. If Otsu selects a threshold < 60, skip detection entirely (image has no significant scratches; low Otsu = unimodal histogram = no damage signal)
|
||||
- **Connected component filtering**: after initial detection, run `cv2.connectedComponentsWithStats`. Keep a component if:
|
||||
- Area >= 20 pixels AND (elongation >= 2.5 OR area >= 200 pixels)
|
||||
- Elongation = `max(bbox_w, bbox_h) / max(min(bbox_w, bbox_h), 1)`
|
||||
- Reject components with area > 5% of total image area (too large to be a scratch)
|
||||
- **Coverage cap**: after filtering, compute total mask coverage. If it exceeds 15%, compute the morphological response intensity at the 85th percentile of masked pixels and re-threshold at that intensity, keeping only the strongest 15% of detections. Log a warning. This prevents the catastrophic "erase everything" failure mode
|
||||
- **Remove `sensitivity` parameter**: no more light/medium/heavy. The adaptive Otsu threshold + filtering handles all cases
|
||||
|
||||
**Removed:** `_make_line_kernel()` (replaced by `_make_line_kernel_rotated()`)
|
||||
|
||||
### 2. LaMa Inpainting Resolution Fix (`restore.py :: inpaint_damage`)
|
||||
|
||||
**Current:** Resize entire image to 512x512, process, resize back. Destroys detail and distorts aspect ratio.
|
||||
|
||||
**New two-path approach:**
|
||||
|
||||
**Small images (both dimensions <= 512px):**
|
||||
- Pad image to 512x512 using `cv2.copyMakeBorder` with `BORDER_REFLECT_101` (no resize, no aspect distortion)
|
||||
- Pad mask identically (padded area = mask 0, no damage)
|
||||
- Run single-pass LaMa at 512x512
|
||||
- Crop result back to original dimensions
|
||||
- Feathered composite with original
|
||||
|
||||
**Large images (any dimension > 512px):**
|
||||
- Process in overlapping 512x512 tiles (stride = 384, overlap = 128)
|
||||
- For edge tiles that extend past the image boundary: pad with `BORDER_REFLECT_101`
|
||||
- Only process tiles where mask has > 0 damaged pixels (skip clean tiles)
|
||||
- Blend tiles using raised-cosine window to eliminate seams
|
||||
- Feathered composite with original (only replace masked areas)
|
||||
|
||||
**Improved mask dilation:**
|
||||
- Current: `MORPH_ELLIPSE(3,3)`, 1 iteration = 3px dilation
|
||||
- New: `MORPH_ELLIPSE(5,5)`, 2 iterations = ~8px dilation
|
||||
- This ensures scratch edges are fully covered for cleaner inpainting
|
||||
|
||||
**Improved feathering:**
|
||||
- Current: `feather_r = max(3, min(w,h) // 200)` = 3px for most images
|
||||
- New: `feather_r = max(5, min(w,h) // 100)` = at least 5px, scales better
|
||||
|
||||
### 3. Face Enhancement Guard (`restore.py :: enhance_faces`)
|
||||
|
||||
**Current:** Minimum face size 24x24px. CodeFormer runs at full strength on all faces.
|
||||
|
||||
**New:**
|
||||
- Minimum face size raised to **48x48px** (skip smaller faces entirely)
|
||||
- For faces 48-120px: clamp fidelity to `max(fidelity, 0.85)` to prevent over-smoothing
|
||||
- For faces > 120px: use user's fidelity setting as-is
|
||||
|
||||
### 4. Lower Default Denoise Strength
|
||||
|
||||
**Current:** `denoiseStrength` default = 40, maps to NLMeans h = 7.8
|
||||
|
||||
**New:** `denoiseStrength` default = **25**, maps to NLMeans h = 6.0. Still removes grain but doesn't wash out small compressed images. Users can increase manually.
|
||||
|
||||
### 5. Remove Mode System
|
||||
|
||||
**API:** Remove `mode: "auto" | "light" | "heavy"` from Zod schema. Zod strips unknown fields by default, so old clients sending `mode` will work (it's silently ignored).
|
||||
|
||||
**Python:** Remove `mode` parameter from `main()`. The adaptive Otsu threshold replaces the fixed threshold that `mode` controlled.
|
||||
|
||||
**Frontend:** Remove 3-button mode selector from `RestorePhotoControls`.
|
||||
|
||||
### 6. Add Colorize Strength Control
|
||||
|
||||
**API:** Add `colorizeStrength: z.number().min(0).max(100).default(85)` to Zod schema.
|
||||
|
||||
**Python:** `colorize_bw()` already accepts `intensity` parameter. Pass `colorizeStrength / 100` from main.
|
||||
|
||||
**Frontend:** Add slider (0-100, step 5) below Auto-Colorize checkbox, visible when colorize is checked. Same nested styling as Face Fidelity slider.
|
||||
|
||||
**i18n:** Add `colorizeStrength: "Colorize Strength"` to `en.ts`.
|
||||
|
||||
### 7. TypeScript Bridge Update (`restoration.ts`)
|
||||
|
||||
- Remove `mode` from `RestorePhotoOptions`
|
||||
- Add `colorizeStrength?: number`
|
||||
|
||||
---
|
||||
|
||||
## Files Changed
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `packages/ai/python/restore.py` | Detection rewrite, tiled LaMa, face guard, remove mode, colorize strength passthrough |
|
||||
| `packages/ai/src/restoration.ts` | Remove `mode`, add `colorizeStrength` to `RestorePhotoOptions` |
|
||||
| `apps/api/src/routes/tools/restore-photo.ts` | Remove `mode` from both Zod schemas, add `colorizeStrength`, lower `denoiseStrength` default |
|
||||
| `apps/web/src/components/tools/restore-photo-settings.tsx` | Remove mode selector, add colorize strength slider, update default denoise to 25 |
|
||||
| `packages/shared/src/i18n/en.ts` | Add `colorizeStrength` key |
|
||||
| `tests/unit/ai/restoration.test.ts` | Remove mode tests, add colorizeStrength tests |
|
||||
| `tests/integration/restore-photo.test.ts` | Remove mode tests, add colorizeStrength validation |
|
||||
| `tests/e2e/restore-photo.spec.ts` | Remove mode UI test, add colorize strength slider test |
|
||||
|
||||
---
|
||||
|
||||
## What This Spec Does NOT Include
|
||||
|
||||
The following items from the earlier approved spec (`2026-04-15`) are deferred. They add model downloads and complexity without diagnostic evidence that they're needed:
|
||||
|
||||
- **Stable Diffusion ONNX inpainting** (~1.7 GB): LaMa with tiled inference at native resolution should be sufficient. If quality is still lacking after this fix, SD-inpainting can be added as a follow-up
|
||||
- **NAFNet neural denoising** (~68 MB): NLMeans at lower strength is adequate. NAFNet is a future upgrade
|
||||
- **CT2 colorization** (~850 MB): DDColor with adjustable strength is good enough
|
||||
- **GFPGAN fallback**: CodeFormer with the face size guard should handle all cases
|
||||
- **Real-ESRGAN super-resolution step**: Separate from restoration quality; can be added independently
|
||||
- **New `superResolution` setting**: Deferred with the SR step
|
||||
|
||||
These can each be added incrementally in future PRs if testing shows they're needed.
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit (Vitest)
|
||||
- API schema rejects `mode` field gracefully (stripped, not errored)
|
||||
- API schema accepts `colorizeStrength` with default 85
|
||||
- `denoiseStrength` default is 25
|
||||
- `colorizeStrength` out of range (101, -1) rejected
|
||||
|
||||
### Integration (Vitest)
|
||||
- POST with `colorizeStrength: 50` accepted
|
||||
- POST with old `mode: "auto"` doesn't error (backward compat)
|
||||
|
||||
### E2E (Playwright)
|
||||
- Mode selector buttons no longer present
|
||||
- Colorize strength slider appears when Auto-Colorize is checked
|
||||
- Default denoise strength shows 25
|
||||
|
||||
### Manual verification
|
||||
- Run pipeline on all 4 diagnostic sample images
|
||||
- Compare before/after at each step
|
||||
- Verify: mask coverage < 15% on all samples, scratches removed, faces natural, no blurriness
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
The fix is successful when all 4 diagnostic images produce visibly better results than current:
|
||||
1. `images2.jpg`: faces preserved (not erased), scratches reduced
|
||||
2. `images.jpg`: face unchanged (minimal actual damage detected)
|
||||
3. `woman-baby1.webp`: baby face natural (not plasticky), scratches removed
|
||||
4. `ai-old-photo...webp`: diagonal scratch fully removed, face natural
|
||||
Reference in New Issue
Block a user