External/SnapOtter
1
0
Fork 0
mirror of https://github.com/snapotter-hq/SnapOtter.git synced 2026-08-03 07:46:42 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix(jobs): pre-warm QueueEvents to kill first-sync-wait flake (#285)

Browse Source 
The csv-json integration test intermittently timed out at 30000ms on
the first worker-backed job in a fork. Root cause: waitForJob() creates
the BullMQ QueueEvents consumer lazily on first use, and a fresh consumer
reads the Redis events stream from "$" (the tail at the moment its run
loop starts). A trivial tool can publish its completed:<id> event before
the brand-new consumer positions itself, so waitUntilFinished() never
sees the event and blocks for the full sync-wait window. In tests
SYNC_WAIT_MS is floored at 30000ms, exactly the vitest per-test budget,
so the stall surfaces as an opaque timeout instead of a 202 fallback.
This is also a latent production latency bug: the first synchronous tool
request after each boot could hang up to the 8s prod window.

Fix: warmQueueEvents() eagerly constructs and connects every pool's
consumer at spine startup, before any job is enqueued, so each consumer
is positioned at the stream tail up front and never misses a completion.
Awaited in the test spine (deterministic for the first request) and fired
non-blocking at prod boot (a slow Redis must not stall startup).

Adds a regression guard in job-spine.test.ts that drops the cached
consumers, warms explicitly, and asserts a fast job's completion is
captured on the first sync-wait.

Verified: 3 parallel stress runs (276 file-runs across all pools), zero
timeouts; targeted job-spine + csv-json suites green; typecheck clean.
This commit is contained in:
SnapOtter
2026-06-21 23:22:59 +08:00
committed by GitHub
parent f75cc328ac
commit dba8a85a80
4 changed files with 80 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/integration/test-server.ts
+7 -1
View File
@@ -36,7 +36,7 @@ import {
stopCancelListener,
} from "../../apps/api/src/jobs/cancel.js";
import { pingRedis } from "../../apps/api/src/jobs/connection.js";
import { closeQueueEvents } from "../../apps/api/src/jobs/enqueue.js";
import { closeQueueEvents, warmQueueEvents } from "../../apps/api/src/jobs/enqueue.js";
import { closeWorkers, startWorkers } from "../../apps/api/src/jobs/worker.js";
import { requirePermission } from "../../apps/api/src/permissions.js";
import {
@@ -83,6 +83,12 @@ async function ensureSpine(): Promise<void> {
spineStarted = true;
await startCancelListener();
startWorkers();
// Position every pool's QueueEvents consumer at the stream tail *before* the
// first job is enqueued. Without this, the first sync-wait per fork lazily
// creates a consumer that can miss a fast job's completion event and block
// for the full SYNC_WAIT_MS window -- the root cause of the csv-json 30s
// timeout flake. Awaited here so it is deterministic for the first request.
await warmQueueEvents();
}
// Module-scope afterAll: vitest registers this into any importing file's