External/SnapOtter
1
0
Fork 0
mirror of https://github.com/snapotter-hq/SnapOtter.git synced 2026-08-03 07:46:42 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix(jobs): pre-warm QueueEvents to kill first-sync-wait flake (#285)

Browse Source 
The csv-json integration test intermittently timed out at 30000ms on
the first worker-backed job in a fork. Root cause: waitForJob() creates
the BullMQ QueueEvents consumer lazily on first use, and a fresh consumer
reads the Redis events stream from "$" (the tail at the moment its run
loop starts). A trivial tool can publish its completed:<id> event before
the brand-new consumer positions itself, so waitUntilFinished() never
sees the event and blocks for the full sync-wait window. In tests
SYNC_WAIT_MS is floored at 30000ms, exactly the vitest per-test budget,
so the stall surfaces as an opaque timeout instead of a 202 fallback.
This is also a latent production latency bug: the first synchronous tool
request after each boot could hang up to the 8s prod window.

Fix: warmQueueEvents() eagerly constructs and connects every pool's
consumer at spine startup, before any job is enqueued, so each consumer
is positioned at the stream tail up front and never misses a completion.
Awaited in the test spine (deterministic for the first request) and fired
non-blocking at prod boot (a slow Redis must not stall startup).

Adds a regression guard in job-spine.test.ts that drops the cached
consumers, warms explicitly, and asserts a fast job's completion is
captured on the first sync-wait.

Verified: 3 parallel stress runs (276 file-runs across all pools), zero
timeouts; targeted job-spine + csv-json suites green; typecheck clean.
This commit is contained in:
SnapOtter
2026-06-21 23:22:59 +08:00
committed by GitHub
parent f75cc328ac
commit dba8a85a80
4 changed files with 80 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/api/src/index.ts
+9 -1
View File
@@ -13,7 +13,7 @@ import { closeDb, db, schema } from "./db/index.js";
import { runMigrations } from "./db/migrate.js";
import { startCancelListener, stopCancelListener } from "./jobs/cancel.js";
import { closeRedis, pingRedis } from "./jobs/connection.js";
import { closeFlowProducer, closeQueueEvents } from "./jobs/enqueue.js";
import { closeFlowProducer, closeQueueEvents, warmQueueEvents } from "./jobs/enqueue.js";
import { closeQueues, perPoolHealth, queueCounts } from "./jobs/queues.js";
import { enqueueSystemJob, SYSTEM_JOBS, scheduleSystemJobs } from "./jobs/system-jobs.js";
import { closeWorkers, startWorkers } from "./jobs/worker.js";
@@ -596,6 +596,14 @@ if (await shouldRunStartupCleanup()) {
// Start BullMQ worker pools (after route registration so the tool registry is full)
startWorkers();
// Warm the per-pool QueueEvents consumers so the first synchronous tool request
// after boot does not pay the lazy-connect cost (and cannot miss a fast job's
// completion event). Non-blocking: a slow/unreachable Redis must not stall boot;
// the consumers fall back to lazy creation on first use if this has not finished.
void warmQueueEvents().catch((err) => {
app.log.warn({ err }, "QueueEvents warm-up failed; consumers will connect lazily");
});
// Start
try {
await app.listen({ port: env.PORT, host: "0.0.0.0" });