diff --git a/docs/superpowers/specs/2026-05-13-restore-photo-quality-overhaul-design.md b/docs/superpowers/specs/2026-05-13-restore-photo-quality-overhaul-design.md new file mode 100644 index 00000000..f7b63393 --- /dev/null +++ b/docs/superpowers/specs/2026-05-13-restore-photo-quality-overhaul-design.md @@ -0,0 +1,182 @@ +# Restore Photo Quality Overhaul + +**Date:** 2026-05-13 +**Status:** Draft +**Scope:** Fix critical quality issues in the restore-photo pipeline, informed by diagnostic testing on real damaged photos +**Supersedes:** `docs/specs/2026-04-15-restore-photo-design.md` (approved but unimplemented; this spec is scoped to the subset of changes backed by diagnostic evidence) + +--- + +## Diagnostic Evidence + +Full pipeline was run on 4 sample images with intermediate outputs saved at each step. Key findings: + +| Image | Pixels | Mask Coverage | Actual Damage | Result | +|---|---|---|---|---| +| images2.jpg | 290x174 | **68.7%** | ~3% | Faces erased, gray blobs | +| images.jpg | 188x267 | **30.6%** | ~1% | Face structure altered | +| woman-baby1.webp | 400x277 | **23.2%** | ~6% | Baby face plasticky, scratches remain | +| ai-old-photo...webp | 768x513 | **6.4%** | ~5% | Reasonable but scratches still visible | + +Root causes identified (ranked by impact): + +1. **Scratch detection massively over-detects** on small images (4-15x worse than reality) +2. **LaMa at 512x512** can't fully remove scratches even when correctly detected +3. **CodeFormer over-smooths small faces** (< 80px) +4. **NLMeans default strength too high** (40 maps to h=7.8, over-smooths compressed images) + +Diagnostic outputs saved at `/tmp/restore-diagnostic/` with per-step intermediate images. + +--- + +## Changes + +### 1. Scratch Detection Rewrite (`restore.py :: detect_scratches`) + +**Current:** 4 angles (0/45/90/135), 3 kernel sizes, fixed threshold per mode, no filtering. + +**New:** +- **8 angles**: 0, 22.5, 45, 67.5, 90, 112.5, 135, 157.5 degrees +- **Rotation-matrix kernels**: `cv2.getRotationMatrix2D` + `cv2.warpAffine` for accurate arbitrary-angle line kernels (replaces hard-coded pixel patterns) +- **Image-size-adaptive kernel count**: for images with shortest side < 300px, use only 2 kernel sizes (max kernel capped at `base_dim // 15`). For >= 300px, use 3 sizes as before +- **Pre-filtering**: light bilateral filter (`d=5, sigmaColor=50, sigmaSpace=50`) before CLAHE to suppress JPEG/WebP compression artifacts +- **Adaptive threshold**: use `cv2.THRESH_BINARY + cv2.THRESH_OTSU` per combined morphological response map instead of a fixed threshold. This auto-adapts to each image's contrast characteristics. If Otsu selects a threshold < 60, skip detection entirely (image has no significant scratches; low Otsu = unimodal histogram = no damage signal) +- **Connected component filtering**: after initial detection, run `cv2.connectedComponentsWithStats`. Keep a component if: + - Area >= 20 pixels AND (elongation >= 2.5 OR area >= 200 pixels) + - Elongation = `max(bbox_w, bbox_h) / max(min(bbox_w, bbox_h), 1)` + - Reject components with area > 5% of total image area (too large to be a scratch) +- **Coverage cap**: after filtering, compute total mask coverage. If it exceeds 15%, compute the morphological response intensity at the 85th percentile of masked pixels and re-threshold at that intensity, keeping only the strongest 15% of detections. Log a warning. This prevents the catastrophic "erase everything" failure mode +- **Remove `sensitivity` parameter**: no more light/medium/heavy. The adaptive Otsu threshold + filtering handles all cases + +**Removed:** `_make_line_kernel()` (replaced by `_make_line_kernel_rotated()`) + +### 2. LaMa Inpainting Resolution Fix (`restore.py :: inpaint_damage`) + +**Current:** Resize entire image to 512x512, process, resize back. Destroys detail and distorts aspect ratio. + +**New two-path approach:** + +**Small images (both dimensions <= 512px):** +- Pad image to 512x512 using `cv2.copyMakeBorder` with `BORDER_REFLECT_101` (no resize, no aspect distortion) +- Pad mask identically (padded area = mask 0, no damage) +- Run single-pass LaMa at 512x512 +- Crop result back to original dimensions +- Feathered composite with original + +**Large images (any dimension > 512px):** +- Process in overlapping 512x512 tiles (stride = 384, overlap = 128) +- For edge tiles that extend past the image boundary: pad with `BORDER_REFLECT_101` +- Only process tiles where mask has > 0 damaged pixels (skip clean tiles) +- Blend tiles using raised-cosine window to eliminate seams +- Feathered composite with original (only replace masked areas) + +**Improved mask dilation:** +- Current: `MORPH_ELLIPSE(3,3)`, 1 iteration = 3px dilation +- New: `MORPH_ELLIPSE(5,5)`, 2 iterations = ~8px dilation +- This ensures scratch edges are fully covered for cleaner inpainting + +**Improved feathering:** +- Current: `feather_r = max(3, min(w,h) // 200)` = 3px for most images +- New: `feather_r = max(5, min(w,h) // 100)` = at least 5px, scales better + +### 3. Face Enhancement Guard (`restore.py :: enhance_faces`) + +**Current:** Minimum face size 24x24px. CodeFormer runs at full strength on all faces. + +**New:** +- Minimum face size raised to **48x48px** (skip smaller faces entirely) +- For faces 48-120px: clamp fidelity to `max(fidelity, 0.85)` to prevent over-smoothing +- For faces > 120px: use user's fidelity setting as-is + +### 4. Lower Default Denoise Strength + +**Current:** `denoiseStrength` default = 40, maps to NLMeans h = 7.8 + +**New:** `denoiseStrength` default = **25**, maps to NLMeans h = 6.0. Still removes grain but doesn't wash out small compressed images. Users can increase manually. + +### 5. Remove Mode System + +**API:** Remove `mode: "auto" | "light" | "heavy"` from Zod schema. Zod strips unknown fields by default, so old clients sending `mode` will work (it's silently ignored). + +**Python:** Remove `mode` parameter from `main()`. The adaptive Otsu threshold replaces the fixed threshold that `mode` controlled. + +**Frontend:** Remove 3-button mode selector from `RestorePhotoControls`. + +### 6. Add Colorize Strength Control + +**API:** Add `colorizeStrength: z.number().min(0).max(100).default(85)` to Zod schema. + +**Python:** `colorize_bw()` already accepts `intensity` parameter. Pass `colorizeStrength / 100` from main. + +**Frontend:** Add slider (0-100, step 5) below Auto-Colorize checkbox, visible when colorize is checked. Same nested styling as Face Fidelity slider. + +**i18n:** Add `colorizeStrength: "Colorize Strength"` to `en.ts`. + +### 7. TypeScript Bridge Update (`restoration.ts`) + +- Remove `mode` from `RestorePhotoOptions` +- Add `colorizeStrength?: number` + +--- + +## Files Changed + +| File | Change | +|---|---| +| `packages/ai/python/restore.py` | Detection rewrite, tiled LaMa, face guard, remove mode, colorize strength passthrough | +| `packages/ai/src/restoration.ts` | Remove `mode`, add `colorizeStrength` to `RestorePhotoOptions` | +| `apps/api/src/routes/tools/restore-photo.ts` | Remove `mode` from both Zod schemas, add `colorizeStrength`, lower `denoiseStrength` default | +| `apps/web/src/components/tools/restore-photo-settings.tsx` | Remove mode selector, add colorize strength slider, update default denoise to 25 | +| `packages/shared/src/i18n/en.ts` | Add `colorizeStrength` key | +| `tests/unit/ai/restoration.test.ts` | Remove mode tests, add colorizeStrength tests | +| `tests/integration/restore-photo.test.ts` | Remove mode tests, add colorizeStrength validation | +| `tests/e2e/restore-photo.spec.ts` | Remove mode UI test, add colorize strength slider test | + +--- + +## What This Spec Does NOT Include + +The following items from the earlier approved spec (`2026-04-15`) are deferred. They add model downloads and complexity without diagnostic evidence that they're needed: + +- **Stable Diffusion ONNX inpainting** (~1.7 GB): LaMa with tiled inference at native resolution should be sufficient. If quality is still lacking after this fix, SD-inpainting can be added as a follow-up +- **NAFNet neural denoising** (~68 MB): NLMeans at lower strength is adequate. NAFNet is a future upgrade +- **CT2 colorization** (~850 MB): DDColor with adjustable strength is good enough +- **GFPGAN fallback**: CodeFormer with the face size guard should handle all cases +- **Real-ESRGAN super-resolution step**: Separate from restoration quality; can be added independently +- **New `superResolution` setting**: Deferred with the SR step + +These can each be added incrementally in future PRs if testing shows they're needed. + +--- + +## Testing + +### Unit (Vitest) +- API schema rejects `mode` field gracefully (stripped, not errored) +- API schema accepts `colorizeStrength` with default 85 +- `denoiseStrength` default is 25 +- `colorizeStrength` out of range (101, -1) rejected + +### Integration (Vitest) +- POST with `colorizeStrength: 50` accepted +- POST with old `mode: "auto"` doesn't error (backward compat) + +### E2E (Playwright) +- Mode selector buttons no longer present +- Colorize strength slider appears when Auto-Colorize is checked +- Default denoise strength shows 25 + +### Manual verification +- Run pipeline on all 4 diagnostic sample images +- Compare before/after at each step +- Verify: mask coverage < 15% on all samples, scratches removed, faces natural, no blurriness + +--- + +## Success Criteria + +The fix is successful when all 4 diagnostic images produce visibly better results than current: +1. `images2.jpg`: faces preserved (not erased), scratches reduced +2. `images.jpg`: face unchanged (minimal actual damage detected) +3. `woman-baby1.webp`: baby face natural (not plasticky), scratches removed +4. `ai-old-photo...webp`: diagonal scratch fully removed, face natural