External/SnapOtter
1
0
Fork 0
mirror of https://github.com/snapotter-hq/SnapOtter.git synced 2026-08-03 07:46:42 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix: harden install queue/dispatcher lifecycle and repair review-sweep regressions (#395)

Browse Source 
Fixes 15 defects found by a max-effort multi-agent review of the last 6
merged PRs (#388, #390, #391, #392, #393, #394), all adversarially
verified before fixing.

Install queue + dispatcher (the serious cluster):

- features.ts: finalize the installer child exactly once. A failed spawn
  fires both "error" and "close", and the second event released the file
  lock and active slot that pump() had just handed to the next queued
  bundle, letting two pip processes write the same venv concurrently.
  Outcome recording now happens before pump() so the next bundle's first
  progress frame cannot race the previous install's bookkeeping.
- feature-status.ts: keep failed-install errors in a per-bundle map
  instead of the single progress slot. With the queue auto-starting the
  next install, the slot was overwritten within seconds and a failed
  install vanished without ever surfacing to GET /features.
- bridge.ts: scope child lifecycle per process (stopped-children set +
  request generation tags) instead of an instance-wide shuttingDown flag
  that the next spawn reset. A stale SIGTERMed child's late close event
  could record a phantom crash (5 of which permanently disable the
  dispatcher), null out the freshly spawned child, and reject the new
  child's pending requests. The request-timeout kill path still counts
  as a real crash.
- install_feature.py: the pre-write disk re-check measured ai_dir's
  filesystem even when budgeting the cross-filesystem copy that lands on
  the venv's disk; now each budget is checked against the filesystem the
  bytes actually land on, so ENOSPC cannot strike mid-write and leave
  site-packages half overwritten.

Behavior regressions:

- embed-subtitles: preserve pre-existing subtitle tracks (0:s?) and MKV
  attachments (0:t?) that the -map 0:v:0/0:a? rewrite silently dropped;
  data streams stay unmapped on purpose (the actual MPEG remux fix). The
  new subtitle maps first so the language tag hits the right stream.
- usage-survey-overlay: fail closed when the settings fetch fails; the
  fail-open path rendered the blocking survey against an unhealthy API
  and soft-locked admins, the lock-out class #392 fixed.
- features-store: queued bundles poll instead of each holding an SSE
  connection (Install All could pin 7 EventSources and exhaust the
  browser's 6-per-origin HTTP/1.1 limit, hanging the whole app);
  listenToProgress closes any prior stream and stops any poll before
  subscribing; installAll skips bundles already installing or queued.

Contracts, tests, i18n:

- openapi.yaml: add "queued" to the features status enum and document
  downloadBytes/installedBytes (Schemathesis conformance).
- feature-lifecycle e2e: queue transcription (~0.5 GB) instead of ocr
  (~6 GB) and give the test a budget that covers both install drains
  (the stacked waits exceeded the old 900s timeout).
- docker-compose.qa.yml: parameterize the host port (QA_APP_PORT) so
  QA_PROJECT_NAME concurrent stacks can actually bind.
- compare + watermark-image: restore per-input error attribution
  ("Invalid first/second image", "Invalid watermark image") lost in the
  shared-handler migration.
- ai-features-section: the "{size} on disk" suffix now goes through
  i18n; key added to all 21 locales.
- watermark-image + content-aware-resize: migrate to the shared
  inputHandlerFor("image") chain like compare/vectorize/compose, fixing
  drift in the inline copies (no SVG sanitize, no RAW extension hint,
  no AVIF probe).

Verified: typecheck across 9 workspaces, Biome clean on all changed
files, 584 targeted unit tests and 249 integration tests green
(including real-ffmpeg embed-subtitles runs). One unit test updated to
the new poll-while-queued contract with a single-EventSource assertion.

Claude-Session: https://claude.ai/code/session_017mR1HiHaf3a1BmUtrHX4j3
This commit is contained in:
SnapOtter
2026-07-03 13:47:15 +08:00
committed by GitHub
parent b37faed95f
commit b4375e558d
46 changed files with 383 additions and 255 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/ai/src/bridge.ts
+71 -42
View File
@@ -105,6 +105,8 @@ interface PendingRequest {
reject: (err: Error) => void;
onProgress?: ProgressCallback;
stderrLines: string[];
/** Which child generation this request was written to (see startChild). */
generation: number;
}
// Crash recovery constants
@@ -133,11 +135,24 @@ export class PythonDispatcher {
private childFailed = false;
private gpuAvail = false;
private pending = new Map<string, PendingRequest>();
private stdoutBuf = "";
private crashes = 0;
private lastCrashTs = 0;
private backoffEnd = 0;
private shuttingDown = false;
/**
* Monotonic child counter. Each spawned child and every request written to
* it carry the generation current at spawn time, so a stale child's late
* close/error events (SIGTERM delivery can lag a replacement spawn) only
* ever touch their own generation's pending requests.
*/
private generation = 0;
/**
* Children we SIGTERMed on purpose (shutdown/reload). Tracked per child
* rather than as an instance-wide flag: an instance flag reset by the next
* spawn would let the stale child's close event record a phantom crash and
* null out the fresh child. The request-timeout kill path deliberately does
* NOT add to this set, so a genuinely hung script still counts as a crash.
*/
private stoppedChildren = new WeakSet<ChildProcess>();
constructor(opts: { profile: "ai" | "docs" }) {
this.profile = opts.profile;
@@ -173,9 +188,19 @@ export class PythonDispatcher {
);
}
/** Reject and drop the pending requests written to one child generation. */
private rejectPendingForGeneration(generation: number, message: string): void {
for (const [id, req] of this.pending.entries()) {
if (req.generation !== generation) continue;
req.reject(new Error(message));
this.pending.delete(id);
}
}
private startChild(): ChildProcess | null {
if (this.childFailed) return null;
this.shuttingDown = false;
this.generation++;
const gen = this.generation;
try {
const proc = spawn(getPythonPath(), [resolve(PYTHON_DIR, "dispatcher.py")], {
@@ -188,23 +213,23 @@ export class PythonDispatcher {
console.error(
`[bridge] Dispatcher stdin pipe broken (${err.code}), rejecting pending requests`,
);
for (const [id, req] of this.pending.entries()) {
req.reject(new Error("Python dispatcher stdin closed unexpectedly"));
this.pending.delete(id);
this.rejectPendingForGeneration(gen, "Python dispatcher stdin closed unexpectedly");
// An intentional shutdown() ends stdin then SIGTERMs the child,
// which can surface here as an EPIPE/ERR_STREAM_DESTROYED. That is
// not a crash: counting it would let repeated legitimate restarts
// (shutdownDispatcher() runs after every AI bundle install) trip
// the crash limit and permanently disable the dispatcher. Guard
// mirrors the "close" handler below.
if (!this.stoppedChildren.has(proc)) this.recordCrash();
if (this.child === proc) {
this.child = null;
this.childReady = false;
}
// An intentional shutdown() ends stdin then SIGTERMs the child, which
// can surface here as an EPIPE/ERR_STREAM_DESTROYED. That is not a
// crash -- counting it would let repeated legitimate restarts (e.g.
// shutdownDispatcher() on every AI bundle install) trip the crash
// limit and permanently disable the dispatcher. Guard mirrors the
// "close" handler below.
if (!this.shuttingDown) this.recordCrash();
this.child = null;
this.childReady = false;
}
});
let stderrBuf = "";
let stdoutBuf = "";
proc.stderr?.on("data", (chunk: Buffer) => {
stderrBuf += chunk.toString();
@@ -218,21 +243,24 @@ export class PythonDispatcher {
try {
const parsed = JSON.parse(trimmed);
// Readiness signal
// Readiness signal. Ignore it from a superseded child so a stale
// process can't mark a not-yet-ready replacement as ready.
if (parsed.ready === true) {
this.childReady = true;
this.gpuAvail = parsed.gpu === true;
this.crashes = 0;
console.log(`[bridge] Python dispatcher ready (GPU: ${parsed.gpu === true})`);
if (this.child === proc) {
this.childReady = true;
this.gpuAvail = parsed.gpu === true;
this.crashes = 0;
console.log(`[bridge] Python dispatcher ready (GPU: ${parsed.gpu === true})`);
}
continue;
}
// Progress event - route to the currently active request
if (typeof parsed.progress === "number" && typeof parsed.stage === "string") {
// Progress goes to all pending requests (only one should be active at a time
// since Python processes synchronously)
// Progress goes to this child's pending requests (only one
// should be active at a time since Python processes synchronously)
for (const req of this.pending.values()) {
req.onProgress?.(parsed.progress, parsed.stage);
if (req.generation === gen) req.onProgress?.(parsed.progress, parsed.stage);
}
}
} catch {
@@ -242,16 +270,16 @@ export class PythonDispatcher {
console.log(`[python] ${trimmed}`);
}
for (const req of this.pending.values()) {
req.stderrLines.push(trimmed);
if (req.generation === gen) req.stderrLines.push(trimmed);
}
}
}
});
proc.stdout?.on("data", (chunk: Buffer) => {
this.stdoutBuf += chunk.toString();
const lines = this.stdoutBuf.split("\n");
this.stdoutBuf = lines.pop() ?? "";
stdoutBuf += chunk.toString();
const lines = stdoutBuf.split("\n");
stdoutBuf = lines.pop() ?? "";
for (const line of lines) {
const trimmed = line.trim();
@@ -292,29 +320,27 @@ export class PythonDispatcher {
console.error(`[bridge] Dispatcher error: ${err.message} (code: ${err.code})`);
if (err.code === "ENOENT") {
this.childFailed = true;
} else if (!this.shuttingDown) {
} else if (!this.stoppedChildren.has(proc)) {
// Skip crash accounting when we initiated the teardown (shutdown()
// sets shuttingDown before killing the child); mirrors "close".
// marks the child stopped before killing it); mirrors "close".
this.recordCrash();
}
for (const [id, req] of this.pending.entries()) {
req.reject(new Error(extractPythonError(err)));
this.pending.delete(id);
this.rejectPendingForGeneration(gen, extractPythonError(err));
if (this.child === proc) {
this.child = null;
this.childReady = false;
}
this.child = null;
this.childReady = false;
});
proc.on("close", (code) => {
for (const [id, req] of this.pending.entries()) {
req.reject(new Error("Python dispatcher exited unexpectedly"));
this.pending.delete(id);
}
if (code !== 0 && !this.shuttingDown) {
this.rejectPendingForGeneration(gen, "Python dispatcher exited unexpectedly");
if (code !== 0 && !this.stoppedChildren.has(proc)) {
this.recordCrash();
}
this.child = null;
this.childReady = false;
if (this.child === proc) {
this.child = null;
this.childReady = false;
}
});
return proc;
@@ -378,6 +404,9 @@ export class PythonDispatcher {
reject: wrappedReject,
onProgress: options.onProgress,
stderrLines: [],
// getChild() above either reused or just spawned the child this
// request is written to, so the current generation is its generation.
generation: this.generation,
});
const msg: Record<string, unknown> = { id, script: scriptName.replace(".py", ""), args };
@@ -541,7 +570,7 @@ export class PythonDispatcher {
*/
shutdown(): void {
if (this.child && !this.child.killed) {
this.shuttingDown = true;
this.stoppedChildren.add(this.child);
this.child.stdin?.end();
this.child.kill("SIGTERM");
this.child = null;