fix: QA sweep - tool routes, security, i18n, a11y, + AI bundle install hardening (#393)

* fix(api): correct format/filename/container handling across tool routes

Found during a comprehensive QA sweep exercising every tool against its
full accepted-format matrix:

- watermark-image, compose: preserve the requested output format and a
  matching download filename/extension instead of always emitting the
  source format
- compose: crop oversized overlays to the visible base area instead of
  crashing Sharp's composite, and reject only overlays fully outside the
  base image instead of any oversized one
- compare, vectorize: switch to the shared image input handler so
  filenames and formats like .svgz/.tga/RAW survive validation instead
  of being rejected pre-processing
- tool-factory, images-to-video: normalize frames through Sharp before
  handing them to FFmpeg, fixing GIF/AVIF/RAW image-to-video jobs that
  previously failed or hung
- media-tool, replace-audio, embed-subtitles: fix legacy container
  MIME/codec handling for MPEG sources and subtitle remux cases
- files: expand download MIME mapping for text/data/document/video/audio
  outputs that were falling back to a generic content type
- convert-document/presentation/spreadsheet: same-format conversions now
  return the original validated file instead of erroring or producing
  corrupt tiny output

Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG

* fix(web): dropzone a11y, stale localStorage getter, dead code

- dropzone: stop making the whole drop-zone section clickable/focusable.
  A section acting as an interactive element around a real upload button
  is a nested-interactive-element anti-pattern that confuses screen
  readers; drag-and-drop doesn't need focus semantics, only the button
  fallback does. Keeps that button semantic and keyboard-reachable.
  Updates the two e2e call sites that clicked the section directly.
- api, use-auth: read through window.localStorage via the existing API
  storage helper instead of the bare global, which resolves to Node's
  experimental localStorage getter under Vitest and threw
- find-duplicates-settings, info-settings, login-page: remove dead code
  (unused zip-download handler, a stale mount-only effect dependency
  that left cached info stuck at reused indices, an unused response
  variable)

Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG

* fix(i18n): pt-BR, zh-CN, zh-TW were silently falling back to English

The locale loader looked up dynamic-import exports by the raw locale
code (mod["pt-BR"], mod["zh-CN"], mod["zh-TW"]), but those three modules
export camelCased bindings (ptBR, zhCN, zhTW) since identifiers can't
contain hyphens. The lookup returned undefined and every consumer
silently fell back to English for these three locales. Replaces the
generic lookup with explicit per-locale loaders so the mapping can't
drift out of sync again.

Also updates the dropzone helper copy across all 21 locales to match
the drag-only dropzone wording from the previous commit.

Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG

* fix(docs): clear build warnings in the VitePress site

- config.mts: add an onwarn handler for the @vueuse INVALID_ANNOTATION
  warnings emitted during the docs build
- deployment.md: the caddyfile code fence language isn't a shiki grammar
  VitePress ships with, so it warned on every build; use txt instead

Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG

* test(qa): update QA harness for the drag-only dropzone and regen metadata

- api-sweep, qa-helpers, verify-ai: add JSON-body tools, multi-input
  secondary fixtures, async polling for slow valid jobs, 501
  FEATURE_NOT_INSTALLED skip handling, and safer per-tool settings
- input-preview, pipeline-ui specs: update upload flow for the
  drag-only dropzone surface
- add tests/fixtures/data/valid/chart.json, a valid chart fixture the
  updated helpers route to
- regenerate tools-meta.json against current TOOLS[]

Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG

* fix(security): close a login timing side-channel, harden zip-slip tests

Found during a black-box security sweep of the real auth-enabled
production container: a nonexistent username returned 401 in ~3-10ms,
while a wrong password for a real user took ~35-42ms, because scrypt
verification only ran when a user row existed. That timing gap lets an
attacker enumerate valid usernames without ever guessing a password.
Now runs verification against a cached dummy hash on the unknown-user
path too, so both cases cost the same regardless of outcome.

extract-zip already had a relative-traversal regression test
(../evil.txt), but its absolute-path rejection branches
(name.startsWith("/") / startsWith("\\")) had none. Added the three
missing cases: deep relative traversal, absolute Unix path, and
Windows-style absolute path.

Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG

* test(qa): add UI-driven AI bundle install scripts

QA_PROMPT.md's Phase 2 requires installing AI models the way a user
does -- through the UI, on demand from HuggingFace -- and treats the
curl-based admin install endpoint as fallback-only. Nothing in the
harness actually drove that flow; tests/qa/seed-ai-models.sh installs
via docker exec + pip, which is further from a real user than even the
API fallback.

install-ai-bundles-ui.mts logs in, opens Settings > AI Features,
screenshots the pre-install state, clicks Install All, and screenshots
progress -- then exits, since installs continue server-side once
triggered. verify-ai-install-complete.mts polls bundle status,
screenshots the completed state, and runs one real tool per installed
bundle to prove the freshly-downloaded model actually executes.

Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG

* fix(qa): correct the apiToolPath import in the AI verify script

Dynamic import of the package name failed under tsx's module resolution
from apps/api's node_modules context; use the same relative-path import
api-sweep.mts already uses successfully.

Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG

* fix(web): correct AI bundle size estimates shown before install

Measured real downloads during GPU-node QA verification: photo-restoration
pulls ~4.4GB (was advertised as 800MB-1GB, off by 4-5x) and ocr pulls
~5.5GB (was advertised as 3-4GB). Both estimates only accounted for model
weights, not the pip dependencies (torch/paddle) that come down with them.
Updated to reflect actual total download size, since that's what a user
deciding whether they have the disk/bandwidth actually needs to know.

Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG

* fix(web): make desktop Settings reachable when auth is disabled

AvatarDropdown (the only desktop entry point to Settings) was gated
behind `!isMobile && authEnabled`. With AUTH_ENABLED=false the synthetic
anonymous admin user should have full Settings access per how auth.ts
documents this mode -- and the mobile bottom nav already worked this way,
showing Settings unconditionally. Desktop just had a stray extra gate the
component doesn't need: AvatarDropdown already resolves its own username
internally (falling back to "admin") and reads authEnabled itself where
it actually matters (hiding the Logout button). Removed the outer gate;
verified end-to-end against a fresh AUTH_ENABLED=false instance -- avatar
now renders, Settings opens, shows the anonymous/Admin identity correctly.

Also documents (not changes) a related finding in install_feature.py:
detect_arch() always resolves amd64 hosts to the GPU-bundled archive
variant regardless of actual GPU presence, since no CPU-only amd64
archive is published to the bundle repo yet. Left as a code comment
rather than a behavior change, since requesting an unpublished archive
key would hard-fail installs entirely -- worse than the current
oversized-but-working download. Full detail in the QA report.

Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG

* fix(ai): stop logging expected dispatcher reloads as crashes

After each AI bundle install the Python dispatcher reloads because the
venv changed, and after every app shutdown it's SIGTERMed. Both took the
close handler's `code !== 0` branch (SIGTERM makes the exit code null),
so they were counted as crashes -- producing an alarming "crash" line in
the logs and a pointless ~1s recovery backoff after each of 7 installs.
A `stopping` flag set in shutdown() lets the close handler tell an
intentional stop apart from a real crash. The request-timeout kill path
deliberately does not set it, so a genuinely hung script still records a
crash and the 5-in-60s permanent-disable threshold is untouched.

Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG

* fix(api): return a clean message when content-aware resize times out

Carving a very high-resolution image down to a tiny target could exceed
the caire subprocess timeout, and the raw error forwarded to the user was
caire's terminal output -- ANSI color codes and progress-spinner control
characters -- instead of anything actionable. Now: the timeout path
throws a clear "timed out; try a smaller image or larger target" message
(keeping the raw stderr as `cause` for server logs); friendlyError()
strips ANSI/control chars centrally so any subprocess dump surfaced
through the shared sanitizer is plain text; and the content-aware-resize
route (a custom route that bypassed the sanitizer) now routes its error
paths through friendlyError like every other tool.

Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG

* fix(ai): stop bundle installs from exhausting host disk

Installing an AI bundle on a tight-disk host could push the root
filesystem to zero bytes free after the preflight check had already
passed. Two root causes:

- move_tree used copytree+rmtree, so during the move the extracted
  payload existed in both staging and the venv at once -- a full
  transient doubling on disk. Rewrote it to rename entries (a cheap
  metadata op on the same filesystem, no copy), falling back to a copy
  only across filesystems.
- the preflight budget used the manifest's extractedSize verbatim, which
  is 0 for several archives, collapsing the estimate to just the
  compressed size. Added a conservative fallback (3x compressed) so a
  missing value can't under-reserve.

Also added a real-on-disk re-check immediately before the first
destructive venv write (measuring the actual extracted payload and
whether the move needs extra space for a cross-filesystem copy), which
also now covers the offline-import path that previously skipped the disk
check entirely; wrapped the moves so an out-of-space failure returns a
clean actionable error instead of a traceback; and made the disk check
resolve the nearest existing ancestor so it never throws on a
not-yet-created venv path.

Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG

* feat(web): show the real per-arch AI bundle download size

The bundle cards and install prompt showed a hardcoded, architecture-blind
estimatedSize string. That's misleading: amd64 hosts always pull the
CUDA-inclusive archive (there's no CPU-only amd64 variant published), so a
bundle labelled "1-2 GB" can actually download several times that, while
arm64 pulls a much smaller archive for the same label. The manifest
already carries the real per-arch compressedSize (and extractedSize where
measured), so surface those: a new optional downloadBytes/installedBytes
on FeatureBundleState, populated in getFeatureStates() for this host's
arch (resolver mirrors install_feature.py detect_arch), shown by the UI
when present with estimatedSize kept as the fallback label. Also nudged
upscale-enhance's fallback string (4-5 -> 5-6 GB) to match its real
compressed size, consistent with the earlier photo-restoration/ocr fixes.

Fields are optional so demo/mock and existing tests stay compiling; the
manifest's extractedSize is 0 for a few archives, which now surfaces as
null rather than a bogus 0.

Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG

* fix(web): move the AI install queue to the server so it survives tab close

Installing multiple bundles could silently lose all but the first. The
server rejected a concurrent install with 409, so the client worked
around it by queueing the rest in browser-local state and only POSTing
each once it saw the previous finish. A single POSTed install is durable
(the installer child is detached from the request), but a queued one had
zero server footprint -- close the tab mid-queue and those installs
vanished with no error, while the UI still showed them "Queued". The
client "mutex" didn't even serialize: the queued bundles' local waits all
resolved at once and raced into concurrent POSTs that 409'd each other.

Now the queue lives on the server (a small in-memory FIFO leaf module).
The install endpoint enqueues instead of 409-ing and returns
202 {jobId, queued}; a pump starts the next bundle when the current one's
child exits (and after an offline import releases the lock), all behind
the existing venv + file locks, which are unchanged. The client just
POSTs every bundle immediately and reflects the server-reported
queued/installing status; Install All fires all POSTs and lets the server
serialize them, keeping the one-shot retry-on-failure. Adds "queued" to
FeatureStatus (the bundle card already rendered that state) and surfaces
it from getFeatureStates. In-memory is deliberate: it matches the
existing contract (survives a tab close, not a server restart, which
already clears the lock on boot).

Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG

* fix(qa): don't log env-derived credentials in the AI-install script

CodeQL flagged clear-text logging of sensitive information: the login
status line interpolated the QA base URL and username (both read from
the process environment) into a console.log. Replaced with a static
message. QA helper only, but it's a real hygiene issue and cleared the
high-severity code-scanning alert on the PR.

Claude-Session: https://claude.ai/code/session_019fpSXhLGLXWwfyZY2tWhLG
This commit is contained in:
SnapOtter
2026-07-03 09:54:02 +08:00
committed by GitHub
parent 4c32fee039
commit b37faed95f
81 changed files with 2466 additions and 758 deletions
+20 -1
View File
@@ -14,6 +14,25 @@ export function stripInternalPaths(message: string): string {
return message.replace(/\/(tmp|data|app|opt|home|workspace)\b[^\s'")}]*/g, "[internal]");
}
// Matching control characters is the entire point of these patterns (we strip
// them), so noControlCharactersInRegex's premise doesn't apply. ANSI_CSI =
// ESC [ ... final-byte (color/cursor/spinner); C0_CONTROLS = the C0 range + DEL,
// excluding tab (\x09) and newline (\x0A), which we keep.
// biome-ignore lint/suspicious/noControlCharactersInRegex: intentionally matches control chars to strip them
const ANSI_CSI = /\x1B\[[0-9;?]*[ -/]*[@-~]/g;
// biome-ignore lint/suspicious/noControlCharactersInRegex: intentionally matches control chars to strip them
const C0_CONTROLS = /[\x00-\x08\x0B-\x1F\x7F]/g;
/**
* Strip ANSI escape sequences and other terminal control characters that
* subprocess CLIs (e.g. caire's progress spinner) emit into their stderr, so a
* surfaced tool-failure message is plain text rather than raw terminal garbage.
* Preserves tab and newline.
*/
export function stripControlChars(message: string): string {
return message.replace(ANSI_CSI, "").replace(C0_CONTROLS, "");
}
/**
* Unambiguous markers of a raw external-tool failure dump: the
* `ffmpeg/ffprobe exited N:` prefix that media-engine throws, a Python
@@ -33,7 +52,7 @@ const RAW_TOOL_FAILURE =
* sanitized. Idempotent, so it is safe to apply at every error surface.
*/
export function friendlyError(message: string): string {
const cleaned = stripInternalPaths(message);
const cleaned = stripInternalPaths(stripControlChars(message));
if (RAW_TOOL_FAILURE.test(cleaned) || cleaned.length > 280 || cleaned.split("\n").length > 3) {
return "Processing failed. The file may be in an unsupported or corrupted format.";
}
+87
View File
@@ -0,0 +1,87 @@
/**
* Server-side feature-install queue (in-memory, FIFO).
*
* The queue lives on the server so a POST is always durable: the client POSTs
* immediately and the server serializes installs behind the existing file lock.
* A queued-but-not-yet-started install therefore has a real server footprint
* (this module), instead of only living in a browser tab that can close and
* silently drop it.
*
* This is a LEAF module: it imports NOTHING from feature-status.ts. The
* dependency runs the other way (feature-status.ts imports getQueuedBundleIds
* from here to surface "queued" status), so importing back would form a cycle.
*
* In-memory is deliberate, not a shortcut. The active install is a child of the
* API process and does not survive a server restart (recoverInterruptedInstalls
* clears the lock on boot because any previous install is dead). This queue
* matches that exact contract: it survives a browser tab close (the bug it
* fixes) but not a server restart.
*/
export interface QueuedInstall {
bundleId: string;
jobId: string;
}
let activeInstall: QueuedInstall | null = null;
const queue: QueuedInstall[] = [];
/** The bundle currently installing (holding the file lock), or null. */
export function getActiveBundleId(): string | null {
return activeInstall?.bundleId ?? null;
}
/** Bundle ids waiting in the queue, in FIFO order (excludes the active one). */
export function getQueuedBundleIds(): string[] {
return queue.map((entry) => entry.bundleId);
}
/** True when the bundle is currently installing or already waiting in the queue. */
export function isQueuedOrActive(bundleId: string): boolean {
return activeInstall?.bundleId === bundleId || queue.some((entry) => entry.bundleId === bundleId);
}
/**
* Add a bundle to the FIFO queue and return the effective jobId to track.
*
* Server-side dedup: if the bundle is already the active install or already in
* the queue, no second entry is added and the existing job's id is returned so
* the caller can attach to the in-flight progress stream.
*/
export function enqueue(entry: QueuedInstall): string {
if (activeInstall?.bundleId === entry.bundleId) {
return activeInstall.jobId;
}
const existing = queue.find((q) => q.bundleId === entry.bundleId);
if (existing) {
return existing.jobId;
}
queue.push(entry);
return entry.jobId;
}
/** Peek the head of the queue without removing it. */
export function peekQueue(): QueuedInstall | null {
return queue[0] ?? null;
}
/** Remove and return the head of the queue. */
export function dequeue(): QueuedInstall | null {
return queue.shift() ?? null;
}
/** Mark a bundle as the active install (called by the pump when it starts one). */
export function setActive(entry: QueuedInstall | null): void {
activeInstall = entry;
}
/** Clear the active install (called when the installer child exits). */
export function clearActive(): void {
activeInstall = null;
}
/** Test helper: wipe all queue state. */
export function resetQueueState(): void {
activeInstall = null;
queue.length = 0;
}
+31
View File
@@ -20,6 +20,7 @@ import { fileURLToPath } from "node:url";
import type { FeatureBundleState, FeatureStatus } from "@snapotter/shared";
import { FEATURE_BUNDLES, getRequiredBundlesForTool } from "@snapotter/shared";
import * as tar from "tar";
import { getQueuedBundleIds } from "./feature-install-queue.js";
// ── Paths ───────────────────────────────────────────────────────────────
@@ -281,14 +282,30 @@ interface ManifestModel {
minSize?: number;
}
interface ManifestArchive {
compressedSize?: number;
extractedSize?: number;
}
interface ManifestBundle {
models: ManifestModel[];
archives?: Record<string, ManifestArchive>;
}
interface Manifest {
bundles: Record<string, ManifestBundle>;
}
/**
* Bundle archive key for this host, mirroring detect_arch() in
* install_feature.py exactly so the size we surface matches what actually gets
* downloaded. Only "amd64-gpu" and "arm64-cpu" archives are published; amd64
* always resolves to the GPU variant (there is no CPU-only amd64 archive).
*/
function bundleArchKey(): string {
return process.arch === "arm64" ? "arm64-cpu" : "amd64-gpu";
}
function readManifest(): Manifest | null {
if (!existsSync(MANIFEST_PATH)) return null;
try {
@@ -492,6 +509,9 @@ export function verifyBundleModels(bundleId: string): string | null {
export function getFeatureStates(): FeatureBundleState[] {
const installed = readInstalled();
const lock = getInstallingBundle();
const manifest = readManifest();
const arch = bundleArchKey();
const queuedIds = new Set(getQueuedBundleIds());
return Object.values(FEATURE_BUNDLES).map((bundle) => {
const installedBundle = installed.bundles[bundle.id];
@@ -517,11 +537,20 @@ export function getFeatureStates(): FeatureBundleState[] {
} else {
status = "installed";
}
} else if (queuedIds.has(bundle.id)) {
// Waiting behind the active install in the server-side queue.
status = "queued";
} else if (currentProgress?.bundleId === bundle.id && currentProgress.error) {
status = "error";
error = currentProgress.error;
}
const archive = manifest?.bundles[bundle.id]?.archives?.[arch];
const downloadBytes =
archive?.compressedSize && archive.compressedSize > 0 ? archive.compressedSize : null;
const installedBytes =
archive?.extractedSize && archive.extractedSize > 0 ? archive.extractedSize : null;
return {
id: bundle.id,
name: bundle.name,
@@ -529,6 +558,8 @@ export function getFeatureStates(): FeatureBundleState[] {
status,
installedVersion: installedBundle?.version ?? null,
estimatedSize: bundle.estimatedSize,
downloadBytes,
installedBytes,
enablesTools: bundle.enablesTools,
progress,
error,
+15
View File
@@ -5,9 +5,20 @@ import type { ToolProcessCtxV2 } from "../routes/tool-factory.js";
const EXT_VIDEO_CONTENT_TYPES: Record<string, string> = {
".mp4": "video/mp4",
".m4v": "video/mp4",
".mov": "video/quicktime",
".webm": "video/webm",
".mkv": "video/x-matroska",
".avi": "video/x-msvideo",
".3gp": "video/3gpp",
".flv": "video/x-flv",
".wmv": "video/x-ms-wmv",
".mpg": "video/mpeg",
".mpeg": "video/mpeg",
".ts": "video/mp2t",
".mts": "video/mp2t",
".m2ts": "video/mp2t",
".ogv": "video/ogg",
};
/** Content type for a preserved-container video output; mp4 fallback. */
@@ -25,6 +36,9 @@ export function videoContentType(ext: string): string {
*/
export function videoEncodeArgsForContainer(ext: string): string[] {
const lower = ext.toLowerCase();
if (lower === ".mpg" || lower === ".mpeg") {
return ["-c:v", "mpeg2video", "-q:v", "4", "-pix_fmt", "yuv420p"];
}
if (lower === ".webm") {
return ["-c:v", resolveEncoder("vp9"), "-crf", "30", "-b:v", "0", "-row-mt", "1"];
}
@@ -44,6 +58,7 @@ export function videoEncodeArgsForContainer(ext: string): string[] {
*/
export function audioEncodeArgsForContainer(ext: string): string[] {
const lower = ext.toLowerCase();
if (lower === ".mpg" || lower === ".mpeg") return ["-c:a", "mp2", "-b:a", "192k"];
if (lower === ".webm") return ["-c:a", resolveEncoder("opus")];
if (lower === ".ogv" || lower === ".ogg") return ["-c:a", "libvorbis"];
return ["-c:a", resolveEncoder("aac")];
+13 -2
View File
@@ -6484,6 +6484,12 @@ paths:
Start async installation of an AI feature bundle. Downloads models and
configures the Python sidecar. Returns a jobId for progress tracking
via the SSE endpoint. Requires features:manage permission.
Installs are serialized server-side: a request made while another
install is running is queued (queued=true) rather than rejected, and
starts automatically when the running install finishes. A repeat
request for the same in-flight bundle is deduped and returns the same
jobId.
security:
- bearerAuth: []
parameters:
@@ -6495,7 +6501,7 @@ paths:
type: string
responses:
"202":
description: Installation started
description: Installation started or queued
content:
application/json:
schema:
@@ -6504,6 +6510,11 @@ paths:
jobId:
type: string
description: Job ID for tracking installation progress via SSE
queued:
type: boolean
description: >-
True when the install was queued behind another running
install instead of starting immediately.
"401":
description: Authentication required
content:
@@ -6523,7 +6534,7 @@ paths:
schema:
$ref: "#/components/schemas/Error"
"409":
description: Bundle already installed or another install in progress
description: Bundle already installed
content:
application/json:
schema:
+21
View File
@@ -49,6 +49,24 @@ export async function verifyPassword(password: string, stored: string): Promise<
return timingSafeEqual(derived, storedBuf);
}
/**
* Fixed-cost hash used to equalize login timing for usernames that don't
* exist (or have no local password). Computed once per process and cached --
* without this, a login attempt for an unknown username returns as soon as
* the user lookup misses, while a wrong password for a real user waits on a
* full scrypt run. That gap is a timing side-channel an attacker can use to
* enumerate valid usernames even though both cases return an identical 401
* body. Running verifyPassword against this dummy hash pays the same scrypt
* cost on the "unknown user" path so the two cases are timing-indistinguishable.
*/
let dummyHashPromise: Promise<string> | null = null;
function getDummyHash(): Promise<string> {
if (!dummyHashPromise) {
dummyHashPromise = hashPassword(randomBytes(SALT_LENGTH).toString("hex"));
}
return dummyHashPromise;
}
/**
* Compute a fast lookup prefix for an API key.
* Uses SHA-256 (not scrypt) so lookups are O(1) instead of O(n).
@@ -358,6 +376,9 @@ export async function authRoutes(app: FastifyInstance): Promise<void> {
}
if (!user?.passwordHash) {
// Pay the same scrypt cost a real password check would take, so
// response timing doesn't reveal whether the username exists.
await verifyPassword(body.password, await getDummyHash());
authAttempts.inc({ method: "password", result: "failure" });
await audit("LOGIN_FAILED", {
username: sanitizeAuditInput(body.username),
+196 -138
View File
@@ -24,6 +24,14 @@ import { acquireVenvLock, shutdownDispatcher } from "@snapotter/ai";
import { ANALYTICS_EVENTS, FEATURE_BUNDLES } from "@snapotter/shared";
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
import { trackEvent } from "../lib/analytics.js";
import {
clearActive,
dequeue,
enqueue,
getActiveBundleId,
peekQueue,
setActive,
} from "../lib/feature-install-queue.js";
import {
acquireInstallLock,
getAiDir,
@@ -49,6 +57,173 @@ import { updateSingleFileProgress } from "./progress.js";
const venvPath = process.env.PYTHON_VENV_PATH || "/opt/venv";
const pythonPath = `${venvPath}/bin/python3`;
/**
* Spawn the installer child for a bundle and wire up progress / analytics /
* error reporting. The child is detached from any HTTP request, so it finalizes
* via its own close/error handlers regardless of the connection. On exit it
* releases the venv + file locks, clears the active slot, and pumps the queue
* so the next waiting bundle starts automatically.
*
* Precondition: the file install lock is already held for `bundleId` and the
* queue's active slot is already set to it (pump() does both before calling).
*/
function startInstall(bundleId: string, jobId: string): void {
const scriptPath = getInstallScriptPath();
const manifestPath = getManifestPath();
const modelsDir = getModelsDir();
const installStartTime = Date.now();
void (async () => {
// Hold the venv lock across the whole install so no AI tool job loads
// native libs from the venv while pip is rewriting them (that segfaults
// the sidecar). This awaits any in-flight AI job before the installer
// starts; the lock is released when the installer process exits.
const releaseVenv = await acquireVenvLock();
let venvReleased = false;
const releaseVenvOnce = () => {
if (!venvReleased) {
venvReleased = true;
releaseVenv();
}
};
const child = spawn(pythonPath, [scriptPath, bundleId, manifestPath, modelsDir], {
stdio: ["ignore", "pipe", "pipe"],
env: {
...process.env,
BUNDLE_ID: bundleId,
PIP_CACHE_DIR: join(getAiDir(), "pip-cache"),
},
});
let stderrBuffer = "";
let stdoutBuffer = "";
const lastStderrLines: string[] = [];
child.stdout.on("data", (chunk: Buffer) => {
stdoutBuffer += chunk.toString();
});
child.stderr.on("data", (chunk: Buffer) => {
stderrBuffer += chunk.toString();
const lines = stderrBuffer.split("\n");
stderrBuffer = lines.pop() ?? "";
for (const line of lines) {
const trimmed = line.trim();
if (!trimmed) continue;
lastStderrLines.push(trimmed);
if (lastStderrLines.length > 20) lastStderrLines.shift();
try {
const parsed = JSON.parse(trimmed) as { progress?: number; stage?: string };
if (typeof parsed.progress === "number") {
setInstallProgress(
bundleId,
{ percent: parsed.progress, stage: parsed.stage ?? "" },
null,
);
updateSingleFileProgress({
jobId,
phase: "processing",
percent: parsed.progress,
stage: parsed.stage,
});
}
} catch {
// Not JSON progress - rembg/pip output noise, keep in lastStderrLines for error reporting
}
}
});
child.on("close", (code) => {
releaseVenvOnce();
releaseInstallLock();
clearActive();
pump();
if (code === 0) {
invalidateCache();
shutdownDispatcher();
setInstallProgress(null, null, null);
updateSingleFileProgress({ jobId, phase: "complete", percent: 100, stage: "Complete" });
trackEvent(ANALYTICS_EVENTS.AI_BUNDLE_ACTION, {
bundle_id: bundleId,
action: "installed",
duration_ms: Date.now() - installStartTime,
});
} else {
// Extract the structured error from Python's fail() function first.
// fail() writes {"error": "..."} to stderr - prefer this over raw lines.
let errorMsg: string | undefined;
for (let i = lastStderrLines.length - 1; i >= 0; i--) {
const line = lastStderrLines[i];
if (line.startsWith("{")) {
try {
const parsed = JSON.parse(line) as Record<string, unknown>;
if (typeof parsed.error === "string") {
errorMsg = parsed.error;
break;
}
} catch {
// Not valid JSON
}
}
}
if (!errorMsg) {
if (code === 137) {
errorMsg =
"Installation was killed due to insufficient memory. " +
"Try increasing the container's memory limit (e.g. mem_limit: 6g in docker-compose.yml) and retry.";
} else {
const meaningful = lastStderrLines.filter(
(l) =>
!l.startsWith("{") &&
!l.includes("pthread_setaffinity_np") &&
!l.includes("\x1b[") &&
!l.includes("━") &&
!/^\s*\d+%\|/.test(l),
);
errorMsg =
meaningful.join("\n") ||
stdoutBuffer.trim() ||
`Install failed with exit code ${code}`;
}
}
setInstallProgress(bundleId, null, errorMsg);
updateSingleFileProgress({ jobId, phase: "failed", percent: 0, error: errorMsg });
}
});
child.on("error", (err) => {
releaseVenvOnce();
releaseInstallLock();
clearActive();
pump();
const errorMsg = `Failed to spawn install process: ${err.message}`;
setInstallProgress(bundleId, null, errorMsg);
updateSingleFileProgress({ jobId, phase: "failed", percent: 0, error: errorMsg });
});
})();
}
/**
* Start the next queued install if nothing is running and the file lock is
* free. If the lock is held (an offline import is in progress) the head stays
* queued and gets pumped again when the import releases the lock.
*/
function pump(): void {
if (getActiveBundleId()) return;
const head = peekQueue();
if (!head) return;
if (!acquireInstallLock(head.bundleId)) return;
dequeue();
setActive(head);
startInstall(head.bundleId, head.jobId);
}
interface BundleIdParams {
bundleId: string;
}
@@ -122,6 +297,10 @@ export async function registerFeatureRoutes(app: FastifyInstance): Promise<void>
status: "installed" as const,
installedVersion: null,
estimatedSize: bundle.estimatedSize,
// Native (non-Docker) mode runs the models in-process with no bundle
// archive to download, so there are no per-arch download/on-disk sizes.
downloadBytes: null,
installedBytes: null,
enablesTools: bundle.enablesTools,
progress: null,
error: null,
@@ -155,147 +334,22 @@ export async function registerFeatureRoutes(app: FastifyInstance): Promise<void>
markUninstalled(bundleId);
}
if (!acquireInstallLock(bundleId)) {
return reply.status(409).send({ error: "Another install is already in progress" });
}
// Queue the install on the server so the POST is durable: enqueue()
// dedups an already-active/queued bundle and returns the effective jobId,
// then pump() starts it immediately if nothing else is installing. A
// bundle that lands behind another install stays queued server-side and
// starts automatically when the running install finishes.
const jobId = crypto.randomUUID();
const scriptPath = getInstallScriptPath();
const manifestPath = getManifestPath();
const modelsDir = getModelsDir();
const effectiveJobId = enqueue({ bundleId, jobId });
pump();
const installStartTime = Date.now();
// queued === true means it did NOT start right now (another install is
// active, or an offline import holds the lock). The client still opens
// the SSE stream for the returned jobId; the progress route tolerates a
// not-yet-started jobId and streams once the installer begins.
const queued = getActiveBundleId() !== bundleId;
// Hold the venv lock across the whole install so no AI tool job loads
// native libs from the venv while pip is rewriting them (that segfaults
// the sidecar). This awaits any in-flight AI job before the installer
// starts; the lock is released when the installer process exits.
const releaseVenv = await acquireVenvLock();
let venvReleased = false;
const releaseVenvOnce = () => {
if (!venvReleased) {
venvReleased = true;
releaseVenv();
}
};
const child = spawn(pythonPath, [scriptPath, bundleId, manifestPath, modelsDir], {
stdio: ["ignore", "pipe", "pipe"],
env: {
...process.env,
BUNDLE_ID: bundleId,
PIP_CACHE_DIR: join(getAiDir(), "pip-cache"),
},
});
let stderrBuffer = "";
let stdoutBuffer = "";
const lastStderrLines: string[] = [];
child.stdout.on("data", (chunk: Buffer) => {
stdoutBuffer += chunk.toString();
});
child.stderr.on("data", (chunk: Buffer) => {
stderrBuffer += chunk.toString();
const lines = stderrBuffer.split("\n");
stderrBuffer = lines.pop() ?? "";
for (const line of lines) {
const trimmed = line.trim();
if (!trimmed) continue;
lastStderrLines.push(trimmed);
if (lastStderrLines.length > 20) lastStderrLines.shift();
try {
const parsed = JSON.parse(trimmed) as { progress?: number; stage?: string };
if (typeof parsed.progress === "number") {
setInstallProgress(
bundleId,
{ percent: parsed.progress, stage: parsed.stage ?? "" },
null,
);
updateSingleFileProgress({
jobId,
phase: "processing",
percent: parsed.progress,
stage: parsed.stage,
});
}
} catch {
// Not JSON progress - rembg/pip output noise, keep in lastStderrLines for error reporting
}
}
});
child.on("close", (code) => {
releaseVenvOnce();
releaseInstallLock();
if (code === 0) {
invalidateCache();
shutdownDispatcher();
setInstallProgress(null, null, null);
updateSingleFileProgress({ jobId, phase: "complete", percent: 100, stage: "Complete" });
trackEvent(ANALYTICS_EVENTS.AI_BUNDLE_ACTION, {
bundle_id: bundleId,
action: "installed",
duration_ms: Date.now() - installStartTime,
});
} else {
// Extract the structured error from Python's fail() function first.
// fail() writes {"error": "..."} to stderr - prefer this over raw lines.
let errorMsg: string | undefined;
for (let i = lastStderrLines.length - 1; i >= 0; i--) {
const line = lastStderrLines[i];
if (line.startsWith("{")) {
try {
const parsed = JSON.parse(line) as Record<string, unknown>;
if (typeof parsed.error === "string") {
errorMsg = parsed.error;
break;
}
} catch {
// Not valid JSON
}
}
}
if (!errorMsg) {
if (code === 137) {
errorMsg =
"Installation was killed due to insufficient memory. " +
"Try increasing the container's memory limit (e.g. mem_limit: 6g in docker-compose.yml) and retry.";
} else {
const meaningful = lastStderrLines.filter(
(l) =>
!l.startsWith("{") &&
!l.includes("pthread_setaffinity_np") &&
!l.includes("\x1b[") &&
!l.includes("━") &&
!/^\s*\d+%\|/.test(l),
);
errorMsg =
meaningful.join("\n") ||
stdoutBuffer.trim() ||
`Install failed with exit code ${code}`;
}
}
setInstallProgress(bundleId, null, errorMsg);
updateSingleFileProgress({ jobId, phase: "failed", percent: 0, error: errorMsg });
}
});
child.on("error", (err) => {
releaseVenvOnce();
releaseInstallLock();
const errorMsg = `Failed to spawn install process: ${err.message}`;
setInstallProgress(bundleId, null, errorMsg);
updateSingleFileProgress({ jobId, phase: "failed", percent: 0, error: errorMsg });
});
return reply.status(202).send({ jobId });
return reply.status(202).send({ jobId: effectiveJobId, queued });
},
);
@@ -426,6 +480,10 @@ export async function registerFeatureRoutes(app: FastifyInstance): Promise<void>
return reply.status(400).send({ error: err.message });
}
throw err;
} finally {
// The import releases the file lock in its own finally; pump here so a
// bundle that was queued while the import held the lock starts now.
pump();
}
},
);
+46
View File
@@ -243,6 +243,52 @@ function getContentType(ext: string): string {
zip: "application/zip",
ico: "image/x-icon",
json: "application/json",
csv: "text/csv",
tsv: "text/tab-separated-values",
txt: "text/plain",
md: "text/markdown",
markdown: "text/markdown",
html: "text/html",
htm: "text/html",
xml: "application/xml",
yaml: "application/yaml",
yml: "application/yaml",
mp4: "video/mp4",
m4v: "video/mp4",
mov: "video/quicktime",
webm: "video/webm",
mkv: "video/x-matroska",
avi: "video/x-msvideo",
"3gp": "video/3gpp",
flv: "video/x-flv",
wmv: "video/x-ms-wmv",
mpg: "video/mpeg",
mpeg: "video/mpeg",
ts: "video/mp2t",
mts: "video/mp2t",
m2ts: "video/mp2t",
ogv: "video/ogg",
mp3: "audio/mpeg",
wav: "audio/wav",
flac: "audio/flac",
aac: "audio/aac",
m4a: "audio/mp4",
ogg: "audio/ogg",
opus: "audio/opus",
wma: "audio/x-ms-wma",
aiff: "audio/aiff",
amr: "audio/amr",
ac3: "audio/ac3",
docx: "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
odt: "application/vnd.oasis.opendocument.text",
rtf: "application/rtf",
pptx: "application/vnd.openxmlformats-officedocument.presentationml.presentation",
odp: "application/vnd.oasis.opendocument.presentation",
xlsx: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
ods: "application/vnd.oasis.opendocument.spreadsheet",
epub: "application/epub+zip",
srt: "application/x-subrip",
vtt: "text/vtt",
jxl: "image/jxl",
dng: "image/x-adobe-dng",
cr2: "image/x-canon-cr2",
+6 -7
View File
@@ -21,7 +21,7 @@ import { getObjectBuffer, putObject } from "../lib/object-storage.js";
import { resolveToolPool, shouldSkipSyncWindow } from "../lib/pool.js";
import { getSettingNumber } from "../lib/settings-helpers.js";
import { type ReceivedUpload, receiveUpload } from "../lib/upload-stream.js";
import { InputValidationError } from "../modality/contract.js";
import { type InputHandler, InputValidationError } from "../modality/contract.js";
import { inputHandlerFor } from "../modality/input-handler.js";
import { MediaInputHandler, type MediaInputKind } from "../modality/media-input.js";
import { requireToolAccess } from "../permissions.js";
@@ -384,16 +384,15 @@ export function createToolRoute<T>(app: FastifyInstance, config: ToolRouteConfig
}
}
// Build per-position input handlers: when inputKinds is present,
// each position gets a MediaInputHandler for its kind; otherwise
// the tool's modality drives a single shared handler as before.
const kindHandlers: Map<MediaInputKind, MediaInputHandler> = new Map();
function handlerForPosition(idx: number) {
// Build per-position input handlers. Mixed-input image slots still need
// the image pipeline so RAW/HEIC/SVG inputs are normalized before jobs.
const kindHandlers: Map<MediaInputKind, InputHandler> = new Map();
function handlerForPosition(idx: number): InputHandler {
if (config.inputKinds) {
const kind = config.inputKinds[Math.min(idx, config.inputKinds.length - 1)];
let h = kindHandlers.get(kind);
if (!h) {
h = new MediaInputHandler(kind);
h = kind === "image" ? inputHandlerFor("image") : new MediaInputHandler(kind);
kindHandlers.set(kind, h);
}
return h;
+22 -84
View File
@@ -1,12 +1,11 @@
import { randomUUID } from "node:crypto";
import { tmpdir } from "node:os";
import type { FastifyInstance } from "fastify";
import sharp from "sharp";
import { autoOrient } from "../../lib/auto-orient.js";
import { validateImageBuffer } from "../../lib/file-validation.js";
import { decodeToSharpCompat, needsCliDecode } from "../../lib/format-decoders.js";
import { decodeHeic } from "../../lib/heic-converter.js";
import { sanitizeFilename } from "../../lib/filename.js";
import { putObject } from "../../lib/object-storage.js";
import { decompressSvgz, sanitizeSvg } from "../../lib/svg-sanitize.js";
import { InputValidationError } from "../../modality/contract.js";
import { inputHandlerFor } from "../../modality/input-handler.js";
/**
* Compare two images: compute a pixel-level diff and similarity score.
@@ -15,6 +14,8 @@ export function registerCompare(app: FastifyInstance) {
app.post("/api/v1/tools/image/compare", async (request, reply) => {
let bufferA: Buffer | null = null;
let bufferB: Buffer | null = null;
let filenameA = "first.png";
let filenameB = "second.png";
try {
const parts = request.parts();
@@ -27,8 +28,10 @@ export function registerCompare(app: FastifyInstance) {
const buf = Buffer.concat(chunks);
if (!bufferA) {
bufferA = buf;
filenameA = sanitizeFilename(part.filename ?? filenameA);
} else {
bufferB = buf;
filenameB = sanitizeFilename(part.filename ?? filenameB);
}
}
}
@@ -44,85 +47,17 @@ export function registerCompare(app: FastifyInstance) {
}
try {
const valA = await validateImageBuffer(bufferA, "image");
if (!valA.valid) {
return reply.status(400).send({ error: `Invalid first image: ${valA.reason}` });
}
if (valA.format === "heif") {
try {
bufferA = await decodeHeic(bufferA);
} catch (err) {
return reply.status(422).send({
error: "Failed to decode first image (HEIC). Ensure libheif-examples is installed.",
details: err instanceof Error ? err.message : String(err),
});
}
}
if (needsCliDecode(valA.format)) {
try {
bufferA = await decodeToSharpCompat(bufferA, valA.format);
} catch {
try {
await sharp(bufferA).metadata();
} catch (err) {
return reply.status(422).send({
error: `Failed to decode first image (${valA.format.toUpperCase()})`,
details: err instanceof Error ? err.message : String(err),
});
}
}
}
if (valA.format === "svg") {
try {
bufferA = decompressSvgz(bufferA);
bufferA = sanitizeSvg(bufferA);
} catch (err) {
return reply.status(400).send({
error: err instanceof Error ? err.message : "Invalid SVG (first image)",
});
}
}
bufferA = await autoOrient(bufferA);
const valB = await validateImageBuffer(bufferB, "image");
if (!valB.valid) {
return reply.status(400).send({ error: `Invalid second image: ${valB.reason}` });
}
if (valB.format === "heif") {
try {
bufferB = await decodeHeic(bufferB);
} catch (err) {
return reply.status(422).send({
error: "Failed to decode second image (HEIC). Ensure libheif-examples is installed.",
details: err instanceof Error ? err.message : String(err),
});
}
}
if (needsCliDecode(valB.format)) {
try {
bufferB = await decodeToSharpCompat(bufferB, valB.format);
} catch {
try {
await sharp(bufferB).metadata();
} catch (err) {
return reply.status(422).send({
error: `Failed to decode second image (${valB.format.toUpperCase()})`,
details: err instanceof Error ? err.message : String(err),
});
}
}
}
if (valB.format === "svg") {
try {
bufferB = decompressSvgz(bufferB);
bufferB = sanitizeSvg(bufferB);
} catch (err) {
return reply.status(400).send({
error: err instanceof Error ? err.message : "Invalid SVG (second image)",
});
}
}
bufferB = await autoOrient(bufferB);
const imageHandler = inputHandlerFor("image");
bufferA = (
await imageHandler.prepare(bufferA, filenameA, {
scratchDir: tmpdir(),
})
).buffer;
bufferB = (
await imageHandler.prepare(bufferB, filenameB, {
scratchDir: tmpdir(),
})
).buffer;
// Normalize both to same size for comparison
const metaA = await sharp(bufferA).metadata();
@@ -189,6 +124,9 @@ export function registerCompare(app: FastifyInstance) {
processedSize: diffBuffer.length,
});
} catch (err) {
if (err instanceof InputValidationError) {
return reply.status(err.statusCode).send({ error: err.message, details: err.details });
}
return reply.status(422).send({
error: "Comparison failed",
details: err instanceof Error ? err.message : "Unknown error",
+49 -31
View File
@@ -1,35 +1,14 @@
import { randomUUID } from "node:crypto";
import { tmpdir } from "node:os";
import type { FastifyInstance } from "fastify";
import sharp, { type Blend } from "sharp";
import { z } from "zod";
import { autoOrient } from "../../lib/auto-orient.js";
import { formatZodErrors } from "../../lib/errors.js";
import { validateImageBuffer } from "../../lib/file-validation.js";
import { sanitizeFilename } from "../../lib/filename.js";
import { decodeToSharpCompat, needsCliDecode } from "../../lib/format-decoders.js";
import { decodeHeic } from "../../lib/heic-converter.js";
import { putObject } from "../../lib/object-storage.js";
import { decompressSvgz, sanitizeSvg } from "../../lib/svg-sanitize.js";
async function decodeBuffer(inputBuffer: Buffer, filename: string): Promise<Buffer> {
const validation = await validateImageBuffer(inputBuffer, filename);
if (!validation.valid) {
throw new Error(`Invalid image: ${validation.reason}`);
}
let decoded = inputBuffer;
if (validation.format === "heif") {
decoded = await decodeHeic(decoded);
} else if (needsCliDecode(validation.format)) {
const ext = filename.split(".").pop()?.toLowerCase();
decoded = await decodeToSharpCompat(decoded, validation.format, ext);
} else if (validation.format === "svg") {
decoded = decompressSvgz(decoded);
decoded = sanitizeSvg(decoded);
}
return autoOrient(decoded);
}
import { resolveOutputFormat } from "../../lib/output-format.js";
import { InputValidationError } from "../../modality/contract.js";
import { inputHandlerFor } from "../../modality/input-handler.js";
const settingsSchema = z.object({
x: z.number().min(0).default(0),
@@ -108,8 +87,27 @@ export function registerCompose(app: FastifyInstance) {
}
try {
baseBuffer = await decodeBuffer(baseBuffer, filename);
overlayBuffer = await decodeBuffer(overlayBuffer, overlayFilename);
const imageHandler = inputHandlerFor("image");
const base = await imageHandler.prepare(baseBuffer, filename, {
scratchDir: tmpdir(),
});
const overlay = await imageHandler.prepare(overlayBuffer, overlayFilename, {
scratchDir: tmpdir(),
});
baseBuffer = base.buffer;
overlayBuffer = overlay.buffer;
filename = base.filename;
const left = Math.floor(settings.x);
const top = Math.floor(settings.y);
const baseMeta = await sharp(baseBuffer).metadata();
const baseWidth = baseMeta.width ?? 0;
const baseHeight = baseMeta.height ?? 0;
const availableWidth = baseWidth - left;
const availableHeight = baseHeight - top;
if (availableWidth <= 0 || availableHeight <= 0) {
throw new InputValidationError("Overlay position is outside the base image");
}
// Apply opacity to overlay if needed
let processedOverlay = overlayBuffer;
@@ -136,27 +134,47 @@ export function registerCompose(app: FastifyInstance) {
.toBuffer();
}
const overlayMeta = await sharp(processedOverlay).metadata();
const overlayWidth = overlayMeta.width ?? 0;
const overlayHeight = overlayMeta.height ?? 0;
if (overlayWidth > availableWidth || overlayHeight > availableHeight) {
processedOverlay = await sharp(processedOverlay)
.extract({
left: 0,
top: 0,
width: Math.min(overlayWidth, availableWidth),
height: Math.min(overlayHeight, availableHeight),
})
.toBuffer();
}
const outputFormat = await resolveOutputFormat(baseBuffer, filename);
const result = await sharp(baseBuffer)
.composite([
{
input: processedOverlay,
top: settings.y,
left: settings.x,
top,
left,
blend: settings.blendMode as Blend,
},
])
.toFormat(outputFormat.format, { quality: outputFormat.quality })
.toBuffer();
const jobId = randomUUID();
await putObject(`outputs/${jobId}/${filename}`, result);
const outputFilename = `${filename.replace(/\.[^.]+$/, "")}_composed.${outputFormat.extension}`;
await putObject(`outputs/${jobId}/${outputFilename}`, result);
return reply.send({
jobId,
downloadUrl: `/api/v1/download/${jobId}/${encodeURIComponent(filename)}`,
downloadUrl: `/api/v1/download/${jobId}/${encodeURIComponent(outputFilename)}`,
originalSize: baseBuffer.length,
processedSize: result.length,
});
} catch (err) {
if (err instanceof InputValidationError) {
return reply.status(err.statusCode).send({ error: err.message, details: err.details });
}
return reply.status(422).send({
error: "Processing failed",
details: err instanceof Error ? err.message : "Image processing failed",
@@ -6,7 +6,7 @@ import { seamCarve } from "@snapotter/ai";
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
import { z } from "zod";
import { autoOrient } from "../../lib/auto-orient.js";
import { formatZodErrors } from "../../lib/errors.js";
import { formatZodErrors, friendlyError } from "../../lib/errors.js";
import { validateImageBuffer } from "../../lib/file-validation.js";
import { sanitizeFilename } from "../../lib/filename.js";
import { decodeToSharpCompat, needsCliDecode } from "../../lib/format-decoders.js";
@@ -73,7 +73,7 @@ export function registerContentAwareResize(app: FastifyInstance) {
} catch (err) {
return reply.status(422).send({
error: "Failed to decode HEIC/HEIF file",
details: err instanceof Error ? err.message : String(err),
details: friendlyError(err instanceof Error ? err.message : String(err)),
});
}
}
@@ -87,7 +87,7 @@ export function registerContentAwareResize(app: FastifyInstance) {
} catch (err) {
return reply.status(422).send({
error: `Failed to decode ${validation.format} file`,
details: err instanceof Error ? err.message : String(err),
details: friendlyError(err instanceof Error ? err.message : String(err)),
});
}
}
@@ -164,7 +164,7 @@ export function registerContentAwareResize(app: FastifyInstance) {
request.log.error({ err, toolId: "content-aware-resize" }, "Content-aware resize failed");
return reply.status(422).send({
error: "Content-aware resize failed",
details: err instanceof Error ? err.message : "Unknown error",
details: friendlyError(err instanceof Error ? err.message : "Unknown error"),
});
}
},
@@ -4,7 +4,6 @@ import { convertDocument } from "@snapotter/doc-engine";
import type { FastifyInstance } from "fastify";
import { z } from "zod";
import { env } from "../../config.js";
import { InputValidationError } from "../../modality/contract.js";
import { createToolRoute } from "../tool-factory.js";
const CONTENT_TYPES: Record<string, string> = {
@@ -28,14 +27,18 @@ export function registerConvertDocument(app: FastifyInstance) {
processV2: async (ctx) => {
const settings = ctx.settings as z.infer<typeof settingsSchema>;
const input = ctx.inputs[0];
const base = input.filename.replace(/\.[^.]+$/, "");
// Reject same-format no-ops
const inputExt = extname(input.filename).toLowerCase();
if (inputExt === `.${settings.format}`) {
throw new InputValidationError("The file is already in that format", 422);
ctx.report(90, "Done");
return {
buffer: input.buffer,
filename: `${base}.${settings.format}`,
contentType: CONTENT_TYPES[settings.format],
};
}
const base = input.filename.replace(/\.[^.]+$/, "");
// Preserve the real extension so LibreOffice can sniff the input format.
const sanitized = input.filename.replace(/[^A-Za-z0-9._-]/g, "_");
const inPath = join(ctx.scratchDir, `in-${sanitized}`);
@@ -4,7 +4,6 @@ import { convertDocument } from "@snapotter/doc-engine";
import type { FastifyInstance } from "fastify";
import { z } from "zod";
import { env } from "../../config.js";
import { InputValidationError } from "../../modality/contract.js";
import { createToolRoute } from "../tool-factory.js";
const CONTENT_TYPES: Record<string, string> = {
@@ -26,14 +25,18 @@ export function registerConvertPresentation(app: FastifyInstance) {
processV2: async (ctx) => {
const settings = ctx.settings as z.infer<typeof settingsSchema>;
const input = ctx.inputs[0];
const base = input.filename.replace(/\.[^.]+$/, "");
// Reject same-format no-ops
const inputExt = extname(input.filename).toLowerCase();
if (inputExt === `.${settings.format}`) {
throw new InputValidationError("The file is already in that format", 422);
ctx.report(90, "Done");
return {
buffer: input.buffer,
filename: `${base}.${settings.format}`,
contentType: CONTENT_TYPES[settings.format],
};
}
const base = input.filename.replace(/\.[^.]+$/, "");
// Preserve the real extension so LibreOffice can sniff the input format.
const sanitized = input.filename.replace(/[^A-Za-z0-9._-]/g, "_");
const inPath = join(ctx.scratchDir, `in-${sanitized}`);
@@ -4,7 +4,6 @@ import { convertDocument } from "@snapotter/doc-engine";
import type { FastifyInstance } from "fastify";
import { z } from "zod";
import { env } from "../../config.js";
import { InputValidationError } from "../../modality/contract.js";
import { createToolRoute } from "../tool-factory.js";
const CONTENT_TYPES: Record<string, string> = {
@@ -27,14 +26,18 @@ export function registerConvertSpreadsheet(app: FastifyInstance) {
processV2: async (ctx) => {
const settings = ctx.settings as z.infer<typeof settingsSchema>;
const input = ctx.inputs[0];
const base = input.filename.replace(/\.[^.]+$/, "");
// Reject same-format no-ops
const inputExt = extname(input.filename).toLowerCase();
if (inputExt === `.${settings.format}`) {
throw new InputValidationError("The file is already in that format", 422);
ctx.report(90, "Done");
return {
buffer: input.buffer,
filename: `${base}.${settings.format}`,
contentType: CONTENT_TYPES[settings.format],
};
}
const base = input.filename.replace(/\.[^.]+$/, "");
// Preserve the real extension so LibreOffice can sniff the input format.
const sanitized = input.filename.replace(/[^A-Za-z0-9._-]/g, "_");
const inPath = join(ctx.scratchDir, `in-${sanitized}`);
+16 -4
View File
@@ -2,7 +2,13 @@ import { basename, extname, join } from "node:path";
import { probeMedia } from "@snapotter/media-engine";
import type { FastifyInstance } from "fastify";
import { z } from "zod";
import { runFfmpegWithProgress, stageMediaInputs, videoContentType } from "../../lib/media-tool.js";
import {
audioEncodeArgsForContainer,
runFfmpegWithProgress,
stageMediaInputs,
videoContentType,
videoEncodeArgsForContainer,
} from "../../lib/media-tool.js";
import { InputValidationError } from "../../modality/contract.js";
import { createToolRoute } from "../tool-factory.js";
@@ -34,6 +40,7 @@ export function registerEmbedSubtitles(app: FastifyInstance) {
const toMp4 = [".mp4", ".mov", ".m4v"].includes(srcExt);
const outExt = toMp4 ? ".mp4" : ".mkv";
const scodec = toMp4 ? "mov_text" : "srt";
const reencodeInputStreams = outExt === ".mkv" && [".mpg", ".mpeg"].includes(srcExt);
const outName = `${base}_subs${outExt}`;
const contentType = videoContentType(outExt);
@@ -47,16 +54,21 @@ export function registerEmbedSubtitles(app: FastifyInstance) {
await runFfmpegWithProgress(
ctx,
[
"-fflags",
"+genpts",
"-i",
videoPath,
"-i",
subPath,
"-map",
"0",
"0:v:0",
"-map",
"0:a?",
"-map",
"1:0",
"-c",
"copy",
...(reencodeInputStreams
? [...videoEncodeArgsForContainer(outExt), ...audioEncodeArgsForContainer(outExt)]
: ["-c:v", "copy", "-c:a", "copy"]),
"-c:s",
scodec,
"-metadata:s:s:0",
+18 -3
View File
@@ -1,10 +1,12 @@
import { mkdir } from "node:fs/promises";
import { join } from "node:path";
import { resolveEncoder } from "@snapotter/media-engine";
import type { FastifyInstance } from "fastify";
import sharp from "sharp";
import { z } from "zod";
import { runFfmpegWithProgress, stageMediaInputs } from "../../lib/media-tool.js";
import { runFfmpegWithProgress } from "../../lib/media-tool.js";
import { InputValidationError } from "../../modality/contract.js";
import { createToolRoute } from "../tool-factory.js";
import { createToolRoute, type ToolProcessCtxV2 } from "../tool-factory.js";
const DIMS: Record<string, { w: number; h: number }> = {
"1080p": { w: 1920, h: 1080 },
@@ -35,7 +37,7 @@ export function registerImagesToVideo(app: FastifyInstance) {
const settings = settingsSchema.parse(ctx.settings);
const { w: W, h: H } = DIMS[settings.resolution];
const paths = await stageMediaInputs(ctx);
const paths = await stageImageFrames(ctx);
const parts: string[] = [];
const refs: string[] = [];
@@ -77,3 +79,16 @@ export function registerImagesToVideo(app: FastifyInstance) {
},
});
}
async function stageImageFrames(ctx: ToolProcessCtxV2): Promise<string[]> {
const dir = join(ctx.scratchDir, "media", "frames");
await mkdir(dir, { recursive: true });
return Promise.all(
ctx.inputs.map(async (input, index) => {
const framePath = join(dir, `frame-${String(index).padStart(4, "0")}.png`);
await sharp(input.buffer, { animated: false }).png().toFile(framePath);
return framePath;
}),
);
}
+17 -2
View File
@@ -41,9 +41,8 @@ export function registerReplaceAudio(app: FastifyInstance) {
throw new InputValidationError("Second file must be an audio file");
}
// vp8/vp9 in webm cannot be muxed into mp4; keep the source container
const vcodec = videoProbe.streams.find((s) => s.type === "video")?.codec ?? "";
const ext = ["vp8", "vp9"].includes(vcodec) ? ".webm" : ".mp4";
const ext = outputExtForCopiedVideo(vcodec);
const videoBase = basename(ctx.inputs[0].filename, extname(ctx.inputs[0].filename));
const outName = `${videoBase}_newaudio${ext}`;
const outPath = join(ctx.scratchDir, "media", outName);
@@ -75,3 +74,19 @@ export function registerReplaceAudio(app: FastifyInstance) {
},
});
}
function outputExtForCopiedVideo(codec: string): ".mp4" | ".webm" | ".mkv" | ".mpg" {
if (["mpeg1video", "mpeg2video"].includes(codec)) {
return ".mpg";
}
if (["vp8", "vp9"].includes(codec)) {
return ".webm";
}
if (["h264", "hevc", "mpeg4", "av1"].includes(codec)) {
return ".mp4";
}
return ".mkv";
}
+12 -46
View File
@@ -1,17 +1,15 @@
import { randomUUID } from "node:crypto";
import { tmpdir } from "node:os";
import { vectorize as vtrace } from "@neplex/vectorizer";
import type { FastifyInstance } from "fastify";
import potrace from "potrace";
import sharp from "sharp";
import { z } from "zod";
import { autoOrient } from "../../lib/auto-orient.js";
import { formatZodErrors } from "../../lib/errors.js";
import { validateImageBuffer } from "../../lib/file-validation.js";
import { sanitizeFilename } from "../../lib/filename.js";
import { decodeToSharpCompat, needsCliDecode } from "../../lib/format-decoders.js";
import { decodeHeic } from "../../lib/heic-converter.js";
import { putObject } from "../../lib/object-storage.js";
import { decompressSvgz, sanitizeSvg } from "../../lib/svg-sanitize.js";
import { InputValidationError } from "../../modality/contract.js";
import { inputHandlerFor } from "../../modality/input-handler.js";
import { registerToolProcessFn } from "../tool-factory.js";
const settingsSchema = z.object({
@@ -112,7 +110,7 @@ export function registerVectorize(app: FastifyInstance) {
chunks.push(chunk);
}
fileBuffer = Buffer.concat(chunks);
filename = sanitizeFilename(part.filename ?? "output").replace(/\.[^.]+$/, "");
filename = sanitizeFilename(part.filename ?? "output");
} else if (part.fieldname === "settings") {
settingsRaw = part.value as string;
}
@@ -143,46 +141,11 @@ export function registerVectorize(app: FastifyInstance) {
}
try {
const validation = await validateImageBuffer(fileBuffer, filename);
if (!validation.valid) {
return reply.status(400).send({ error: `Invalid image: ${validation.reason}` });
}
if (validation.format === "heif") {
try {
fileBuffer = await decodeHeic(fileBuffer);
} catch (err) {
return reply.status(422).send({
error: "Failed to decode HEIC file. Ensure libheif-examples is installed.",
details: err instanceof Error ? err.message : String(err),
});
}
}
if (needsCliDecode(validation.format)) {
try {
const fileExt = filename.split(".").pop()?.toLowerCase();
fileBuffer = await decodeToSharpCompat(fileBuffer, validation.format, fileExt);
} catch {
try {
await sharp(fileBuffer).metadata();
} catch (err) {
return reply.status(422).send({
error: `Failed to decode ${validation.format.toUpperCase()} file`,
details: err instanceof Error ? err.message : String(err),
});
}
}
}
if (validation.format === "svg") {
try {
fileBuffer = decompressSvgz(fileBuffer);
fileBuffer = sanitizeSvg(fileBuffer);
} catch (err) {
return reply.status(400).send({
error: err instanceof Error ? err.message : "Invalid SVG",
});
}
}
fileBuffer = await autoOrient(fileBuffer);
const prepared = await inputHandlerFor("image").prepare(fileBuffer, filename, {
scratchDir: tmpdir(),
});
fileBuffer = prepared.buffer;
filename = prepared.filename;
const result = await vectorizeBuffer(fileBuffer, settings, filename);
@@ -196,6 +159,9 @@ export function registerVectorize(app: FastifyInstance) {
processedSize: result.buffer.length,
});
} catch (err) {
if (err instanceof InputValidationError) {
return reply.status(err.statusCode).send({ error: err.message, details: err.details });
}
return reply.status(422).send({
error: "Vectorization failed",
details: err instanceof Error ? err.message : "Unknown error",
+6 -2
View File
@@ -9,6 +9,7 @@ import { sanitizeFilename } from "../../lib/filename.js";
import { decodeToSharpCompat, needsCliDecode } from "../../lib/format-decoders.js";
import { decodeHeic } from "../../lib/heic-converter.js";
import { putObject } from "../../lib/object-storage.js";
import { resolveOutputFormat } from "../../lib/output-format.js";
import { decompressSvgz, sanitizeSvg } from "../../lib/svg-sanitize.js";
const settingsSchema = z.object({
@@ -224,16 +225,19 @@ export function registerWatermarkImage(app: FastifyInstance) {
break;
}
const outputFormat = await resolveOutputFormat(mainBuffer, filename);
const result = await sharp(mainBuffer)
.composite([{ input: wmBuffer, top, left }])
.toFormat(outputFormat.format, { quality: outputFormat.quality })
.toBuffer();
const jobId = randomUUID();
await putObject(`outputs/${jobId}/${filename}`, result);
const outputFilename = `${filename.replace(/\.[^.]+$/, "")}_watermarked.${outputFormat.extension}`;
await putObject(`outputs/${jobId}/${outputFilename}`, result);
return reply.send({
jobId,
downloadUrl: `/api/v1/download/${jobId}/${encodeURIComponent(filename)}`,
downloadUrl: `/api/v1/download/${jobId}/${encodeURIComponent(outputFilename)}`,
originalSize: mainBuffer.length,
processedSize: result.length,
});