Files
open-design/apps/daemon/tests/vela-cli-collab-presence-timeout.test.ts
lefarcenandGitHub ad4f3609ca fix(collab): quell the packaged-client cloud 502 storm (heartbeat backoff, upstream error relay, proxy replay) (#6521)
* test(collab): red specs for the packaged-client presence/proxy 502 storm

Encode the three client-side halves of the 502 storm as failing specs on
main before any fix:

- web: a persistently failing presence upstream is heartbeated every 10s
  forever (62 POSTs in 10 minutes) with no backoff.
- daemon: every vela CLI presence failure is flattened into a 502 whose
  message embeds the spawned command line; upstream 403/404 rejections are
  indistinguishable from infra outages, every beat re-spawns the CLI, and
  presence spawns carry no timeout.
- web sidecar: an idempotent /api request that dies with a connection reset
  on a reused pooled daemon socket surfaces as a synthesized 502 instead of
  being replayed once on a fresh connection.

createDaemonProxyHandler is exported as the proxy spec's seam.

* fix(collab): stop the packaged-client presence/proxy 502 storm

Three client-side halves of one incident (hours of ~2050 DevTools errors:
POST /presence/heartbeat and PUT /workspace/billing/interests 502ing every
few seconds while the collab upstream rejected one project):

- web CollabClient: presence heartbeats move from a fixed 10s interval to a
  self-scheduling chain with exponential backoff on consecutive failures
  (10s -> 20s -> 40s -> ... capped at 5min, +/-10% jitter). One success or a
  new presence identity restores the base cadence, so recovery is untouched.

- daemon presence routes: vela CLI failures are classified via the CLI's
  'API request failed with status NNN' output. Upstream 401/403 relay as
  403, 404 as 404, timeouts as 503; only genuine infrastructure failures
  keep 502. Response bodies no longer embed the spawned command line or
  stderr (daemon log keeps the full detail). Repeated consecutive failures
  for one project open a 20s negative cache that answers without spawning
  another CLI process, and presence spawns now carry a 10s timeoutMs so a
  wedged CLI cannot pile up children.

- web sidecar proxy: proxied requests use an explicit keep-alive agent whose
  pooled idle sockets die well before the daemon's 120s keep-alive window,
  and an idempotent request with a small fully-buffered body that fails with
  ECONNRESET/EPIPE on a reused pooled socket before any response bytes is
  replayed once on a fresh connection instead of surfacing a synthesized 502
  the daemon never sent. Non-idempotent methods and large bodies keep the
  streaming path and are never replayed.
2026-08-06 18:47:07 +08:00

43 lines
1.7 KiB
TypeScript

// Red spec for the presence CLI spawn budget.
//
// Presence heartbeat/list/leave are high-frequency lease traffic: the web
// client beats every 10 seconds and every beat spawns a `vela collab presence
// …` child process. Without a wall-clock budget a wedged CLI (hung network,
// stalled auth refresh) piles up unbounded child processes while the client
// keeps beating. Presence spawns must carry an explicit `timeoutMs`; the
// lower-frequency member/comment commands keep their existing unbounded
// behavior.
import { describe, expect, it, vi } from 'vitest';
const { runVelaCommandMock } = vi.hoisted(() => ({
runVelaCommandMock: vi.fn(),
}));
vi.mock('../src/integrations/vela-command.js', async (importOriginal) => ({
...(await importOriginal<typeof import('../src/integrations/vela-command.js')>()),
runVelaCommand: runVelaCommandMock,
}));
import { createVelaCliCollabClient } from '../src/collab/vela-cli-collab-client.js';
describe('vela CLI presence command timeout', () => {
it('bounds every presence spawn with a positive timeoutMs', async () => {
runVelaCommandMock.mockReset();
runVelaCommandMock.mockResolvedValue('{"viewers":[]}');
const client = createVelaCliCollabClient();
await client.heartbeatPresence('p1', { member: { memberId: 'm1' } }, 'w1');
await client.listPresence('p1', 'w1');
await client.leavePresence('p1', { memberId: 'm1' }, 'w1');
expect(runVelaCommandMock).toHaveBeenCalledTimes(3);
for (const [args, options] of runVelaCommandMock.mock.calls) {
expect(args[0]).toBe('collab');
expect(args[1]).toBe('presence');
expect(options?.timeoutMs).toBeGreaterThan(0);
expect(options?.timeoutMs).toBeLessThanOrEqual(30_000);
}
});
});