fix: reliable, self-healing AI feature-bundle installs (#472)

Make on-demand AI feature-bundle installs reliable and self-healing, closing
the failure modes behind most "some tool doesn't work" reports.

Multi-bundle installs: tools needing more than one bundle (Passport Photo,
Enhance Faces) install every required bundle from one action and stay
not-installed until all are present. Verified across all 19 AI tools.

Downloads: self-heal the accelerated Hugging Face (Xet) client so an upgraded
venv no longer silently falls back to slow urllib; restart instead of
corrupting a resumed partial when a proxy ignores Range and returns 200;
verify the completed size; fail fast on disk-full and HTTP 4xx; retry
transient errors five times; add hf_transfer fallback and document Xet egress.

Install integrity: crash-atomic venv writes so a killed or out-of-space
install can no longer tear the shared venv and break other tools; a boot
breadcrumb reseeds a torn venv to a clean state automatically; a post-install
smoke import test refuses to record a bundle whose libraries cannot load; an
install watchdog stops a wedged installer that would otherwise hold the venv
writer lock forever.

Adds unit and end-to-end tests for every failure mode above.
This commit is contained in:
SnapOtter
2026-07-10 07:32:48 +00:00
committed by GitHub
parent ffeacd4b3c
commit a731c3d1fe
33 changed files with 1913 additions and 125 deletions
@@ -175,6 +175,14 @@ describe("POST /api/v1/admin/features/:bundleId/install queue", () => {
});
}
async function postToolInstall(toolId: string) {
return app.inject({
method: "POST",
url: `/api/v1/admin/tools/${toolId}/features/install`,
headers: auth(),
});
}
async function getFeatures() {
const res = await app.inject({ method: "GET", url: "/api/v1/features", headers: auth() });
return JSON.parse(res.body).bundles as Array<{ id: string; status: string }>;
@@ -209,6 +217,29 @@ describe("POST /api/v1/admin/features/:bundleId/install queue", () => {
expect(bundles.find((b) => b.id === "face-detection")?.status).toBe("queued");
});
it("tool install enqueues every missing hard dependency in one request", async () => {
const res = await postToolInstall("passport-photo");
expect(res.statusCode).toBe(202);
const body = JSON.parse(res.body) as {
bundles: Array<{ bundleId: string; jobId: string; queued: boolean }>;
};
expect(body.bundles.map((b) => b.bundleId)).toEqual(["background-removal", "face-detection"]);
expect(body.bundles[0].queued).toBe(false);
expect(body.bundles[1].queued).toBe(true);
await waitFor(() => hoisted.spawnCalls.length === 1);
expect(hoisted.spawnCalls[0].bundleId).toBe("background-removal");
const queuedBundles = await getFeatures();
expect(queuedBundles.find((b) => b.id === "background-removal")?.status).toBe("installing");
expect(queuedBundles.find((b) => b.id === "face-detection")?.status).toBe("queued");
hoisted.spawnCalls[0].emit("close", 0);
await waitFor(() => hoisted.spawnCalls.length === 2);
expect(hoisted.spawnCalls[1].bundleId).toBe("face-detection");
});
it("dedups a duplicate install POST of the active bundle (no second entry, same job)", async () => {
const r1 = await postInstall("ocr");
const jobId1 = JSON.parse(r1.body).jobId;