mirror of
https://github.com/nexu-io/open-design.git
synced 2026-10-27 09:51:12 +01:00
* fix(daemon): reclassify AMR/vela upstream failures out of opaque execution_failed #3408 P1: the largest engineering-view failure bucket is the opaque `execution_failed` (~5,400/7d). Sampling real Langfuse error texts showed many are not engine bugs at all but product-view failures whose (often Chinese) upstream error text the English-only detectors miss, so they fall through to the catch-all. Pull the clearest, highest-confidence AMR/vela ones into their true categories: - vela Chinese pre-charge wallet failure ("预扣费额度失败" / "余额不足" / "额度不足") -> insufficient_balance (recharge), via classifyAmrAccountFailure. - Chinese / generic rate-limit text ("速率限制" / "控制请求频率" / "rate limit" / "too many requests") -> retryable rate_limit / rate_limit_429. - vela "model ... not in allowed list" rejection -> model_unavailable / model_not_found (switch_model). Verification: - Unit: vela-errors.test.ts covers the Chinese balance detection (the authoritative change); the classifier test covers rate-limit + model wiring (its vela-errors mock is updated to mirror the new Chinese signal). - End-to-end: a new run-failure-telemetry-smoke case drives a REAL daemon run whose agent emits each production text, then classifies the REAL recorded events.jsonl — proving the text flows through the run lifecycle into the classifier and lands in the correct category (not execution_failed), rather than only passing a hand-built unit input. Refs #3408 (P1 / execution_failed deep dive), #4502. * fix(daemon): also reclassify binary-not-found failures out of execution_failed Second P1 reclassification batch (same theme as the AMR/vela one): two install/env failure shapes that the agent emits and that currently leak into the opaque execution_failed bucket are an install/PATH problem, not an engine failure. Route them to the existing cli_not_installed detail: - Windows shell "'node' is not recognized as an internal or external command". - "Error: spawn <path> ENOENT" — the executable file does not exist (distinct from spawn EPERM/EBADF/ENOEXEC where the file exists but cannot run); real codex shape where its vendored sub-binary is missing. Red specs use the real production texts; the end-to-end smoke test now also drives these two through a real daemon run and classifies the recorded events.jsonl. (Note: a missing CLAUDE_BIN is a different scenario — it goes through the daemon's preflight path, not an agent-emitted ENOENT — so the e2e case emits the real codex spawn-ENOENT text via the agent, matching the production origin.) Refs #3408 (P1 / execution_failed deep dive), #4502. * fix(daemon): reclassify prompt-size / ACP-protocol / auth texts out of execution_failed Batch A of the P1 execution_failed reclassification — three more named causes sampled from Langfuse that currently leak into the opaque bucket, routed to existing categories (no new contract types): - "Prefill context too large" (local MLX runtime) -> prompt_too_large. - ACP "thread/start failed" -> agent_protocol_error. - vela "login fail: carry the API secret key" -> auth (auth_required). Red specs use the real production texts; the end-to-end smoke test drives all three through a real daemon run and classifies the recorded events.jsonl. Refs #3408 (P1 / execution_failed deep dive), #4502. * feat(analytics): classify local-model-not-loaded (LM Studio) out of execution_failed The single largest remaining slice of the opaque execution_failed bucket (#3408 P1, ~1/5 by Langfuse sampling) is opencode pointed at a local LM Studio provider that has no model loaded: "No models loaded. Please load a model … or use the 'lms load' command." Verified this is user-action, not a model-selection bug on our side: OD passes `-m <model>` to opencode only when the user picks one, and OD has no LM Studio integration (the user configures it as an opencode provider). The error is "No models loaded" (zero loaded), not "model not found" — independent of the model name we send, the local app returns this until the user loads a model. We cannot load it for them; the proportionate fix is to stop hiding it in the opaque bucket and give an actionable category. - New `TrackingRunFailureDetail` value `local_model_not_loaded`. - modelUnavailableDetail() maps "no models loaded" / "lms load" → it, so the existing model_unavailable block routes it (category model_unavailable, user_action switch_model). - Red spec + end-to-end smoke case (real daemon run, recorded events.jsonl). Refs #3408 (P1 / execution_failed deep dive), #4502. * feat(analytics): classify stale Claude session resume as session_resume_expired When `--resume <id>` targets a session that no longer resolves, the daemon already clears the stale id and tells the user to resend (a fresh session is re-seeded with the full transcript). This is a recoverable session-lifecycle failure, but it currently sits in the opaque execution_failed bucket (#3408 P1). Root cause verified (code + claude-code-guide): the stored id pointed at a real session (the daemon only persists it on success or a committed resumable failure), so "no conversation found" comes from Claude's own 30-day cleanupPeriodDays prune, a CLAUDE_CONFIG_DIR change, a cwd/worktree change, or a prior run killed before the session was flushed (`-p` mode has no fsync guarantee). All recoverable by a fresh session — which the daemon already does. - New `TrackingRunFailureDetail` value `session_resume_expired`. - Detector matches both the daemon's surfaced message and the raw Claude CLI shapes ("no conversation found with session id" / "no session found" / "session <id> not found"), tightened so it cannot bridge an unrelated "session …" and a far-away "404 Not Found". - Classified as process_exit / session_resume_expired, retryable, user_action retry (NOT resumable — isResumableFailure stays false, so we never retry by resuming a session that is already gone). - Red specs + end-to-end smoke case. This is the analytics/labelling half. Auto-recovery (transparently re-running with a fresh session so the user never sees a failure) is a separate, larger change tracked with the run-resume work. Refs #3408 (P1 / execution_failed deep dive), #4502. * refactor(daemon): narrow session_resume_expired daemon-message match Address review: '\bcould not be resumed\b' was broader than the rest of the helper's Claude session-miss shapes and runs before other process-exit fallbacks, so an unrelated future 'could not be resumed' stderr would be mislabeled. Anchor it to 'session could not be resumed' (matches the daemon's actual surfaced message 'The previous Claude session could not be resumed …').
This commit is contained in:
@@ -34,7 +34,12 @@ function containsInsufficientBalanceSignal(value: string): boolean {
|
||||
value.includes('not enough credits') ||
|
||||
value.includes('balance is empty') ||
|
||||
value.includes('balance too low') ||
|
||||
value.includes('billing balance')
|
||||
value.includes('billing balance') ||
|
||||
// vela returns the pre-charge (额度预扣) failure in Chinese when the wallet
|
||||
// cannot cover a model call; this currently leaks into execution_failed.
|
||||
value.includes('预扣费额度失败') ||
|
||||
value.includes('余额不足') ||
|
||||
value.includes('额度不足')
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
|
||||
@@ -121,6 +121,14 @@ function isHardQuotaText(text: string): boolean {
|
||||
.test(text);
|
||||
}
|
||||
|
||||
// A transient, retryable rate limit (distinct from a hard quota). vela/upstream
|
||||
// returns this in Chinese ("速率限制" / "请求频率"), which the English-only
|
||||
// quota check above misses, so it currently leaks into execution_failed.
|
||||
function isRateLimitText(text: string): boolean {
|
||||
return /(速率限制|控制请求频率|请求(?:过于)?频繁|rate[ _-]?limit|too many requests)/i
|
||||
.test(text);
|
||||
}
|
||||
|
||||
function isWorkspaceCreditsText(text: string): boolean {
|
||||
return /\b(?:your )?workspace is out of credits\b|\badd credits to continue\b|\bask your workspace owner to refill\b/i
|
||||
.test(text);
|
||||
@@ -156,7 +164,12 @@ function isAgentConfigInvalidText(text: string): boolean {
|
||||
}
|
||||
|
||||
function isCliNotInstalledText(text: string): boolean {
|
||||
return /\bnot installed|not on PATH|cannot find the path specified|system cannot find the path specified\b/i
|
||||
// Also covers the agent binary being absent at its resolved path:
|
||||
// - Windows shell "'node' is not recognized as an internal or external command"
|
||||
// - Node "Error: spawn <path> ENOENT" (the executable file does not exist —
|
||||
// distinct from spawn EPERM/EBADF/ENOEXEC where the file exists but can't run)
|
||||
// Both currently leak into the opaque execution_failed bucket (#3408 P1).
|
||||
return /\bnot installed|not on PATH|cannot find the path specified|system cannot find the path specified|is not recognized as an internal or external command|\bspawn\b[^\n]*\bENOENT\b/i
|
||||
.test(text);
|
||||
}
|
||||
|
||||
@@ -173,6 +186,7 @@ function isAgentProtocolErrorText(text: string): boolean {
|
||||
return /\bjson-rpc id \d+: Internal error\b/i.test(text) ||
|
||||
/\bACP session exited before completion\b/i.test(text) ||
|
||||
/\bQoder run failed: (?:stop_sequence|end_turn)\b/i.test(text) ||
|
||||
/\bthread\/start failed\b/i.test(text) ||
|
||||
/\bfailed to parse request\b/i.test(text);
|
||||
}
|
||||
|
||||
@@ -186,12 +200,26 @@ function isPermissionRequestNotFoundText(text: string): boolean {
|
||||
}
|
||||
|
||||
function isAuthDetailText(text: string): boolean {
|
||||
return /\b(refresh token|access token could not be refreshed|stale local profile|different or stale local profile|credentials from a different local environment|missing environment variable: `?[A-Z0-9_]*API_KEY`?|api key.*(?:missing|invalid)|invalid api key|credentials? (?:are )?missing|not logged in|Authentication required)\b/i
|
||||
return /\b(refresh token|access token could not be refreshed|stale local profile|different or stale local profile|credentials from a different local environment|missing environment variable: `?[A-Z0-9_]*API_KEY`?|api key.*(?:missing|invalid)|invalid api key|credentials? (?:are )?missing|not logged in|Authentication required|carry the api (?:secret )?key)\b/i
|
||||
.test(text);
|
||||
}
|
||||
|
||||
// A resume target that no longer exists. Matches both the daemon's own
|
||||
// surfaced message and the Claude CLI's raw "session not found" shapes.
|
||||
function isSessionResumeExpiredText(text: string): boolean {
|
||||
// Tightly anchored to Claude's actual resume-miss shapes. The session-id form
|
||||
// requires the id token immediately before "not found" so it cannot bridge an
|
||||
// unrelated "session …" and a far-away "404 Not Found" (e.g. opencode 4xx).
|
||||
return /\bsession could not be resumed\b/i.test(text) ||
|
||||
/\bno conversation found with session id\b/i.test(text) ||
|
||||
/\bno session found\b/i.test(text) ||
|
||||
/\bsession [\w-]+ not found\b/i.test(text);
|
||||
}
|
||||
|
||||
function isPromptTooLargeText(text: string): boolean {
|
||||
return /\b(context window|prompt too large|maximum context|too many tokens|input.*too large|exceeds the safe size|composed prompt exceeds|prompt token count .* exceeds|maximum context length|reduce the length of (?:the )?(?:messages|input prompt))\b/i
|
||||
// `prefill context too large` is the local-runtime (MLX) shape of the same
|
||||
// "the prompt does not fit" failure that currently leaks into execution_failed.
|
||||
return /\b(context window|prompt too large|maximum context|too many tokens|input.*too large|exceeds the safe size|composed prompt exceeds|prompt token count .* exceeds|maximum context length|context too large|prefill context too large|reduce the length of (?:the )?(?:messages|input prompt))\b/i
|
||||
.test(text);
|
||||
}
|
||||
|
||||
@@ -210,13 +238,19 @@ function modelUnavailableDetail(text: string): TrackingRunFailureDetail | null {
|
||||
return 'cli_version_incompatible';
|
||||
}
|
||||
if (/\bmodel is disabled\b/i.test(text)) return 'model_disabled';
|
||||
// A local model server (e.g. LM Studio, reached via opencode's own provider
|
||||
// config) is up but has no model loaded. Not a model we picked wrong — the
|
||||
// user must load a model in the local app first (`lms load`). User-action,
|
||||
// not an engine bug, so it should not sit in the opaque execution_failed
|
||||
// bucket. (#3408 P1)
|
||||
if (/\bno models loaded\b|\blms load\b/i.test(text)) return 'local_model_not_loaded';
|
||||
if (/\b(no endpoints found that support tool use|provider routing)\b/i.test(text)) {
|
||||
return 'provider_routing_error';
|
||||
}
|
||||
if (/\b(model .*not supported|requested model is not supported|supported api model names|not supported when using codex)\b/i.test(text)) {
|
||||
return 'model_not_supported';
|
||||
}
|
||||
if (/\b(model (?:is )?(?:unavailable|not available|unsupported|not found)|selected model is not available|not have access|no access|model .*not found|no healthy deployments)\b/i.test(text)) {
|
||||
if (/\b(model (?:is )?(?:unavailable|not available|unsupported|not found)|selected model is not available|not have access|no access|model .*not found|no healthy deployments|model .*not in (?:the )?allowed list)\b/i.test(text)) {
|
||||
return 'model_not_found';
|
||||
}
|
||||
return null;
|
||||
@@ -499,6 +533,22 @@ export function classifyRunFailure(
|
||||
);
|
||||
}
|
||||
|
||||
// A `--resume <id>` whose stored session no longer resolves (Claude's 30-day
|
||||
// cleanupPeriodDays prune, a CLAUDE_CONFIG_DIR change, a cwd/worktree change,
|
||||
// or a prior run killed before the session was flushed). The daemon already
|
||||
// clears the stale id so the next turn starts fresh — this is a recoverable
|
||||
// session-lifecycle failure, not an opaque engine crash, so name it and mark
|
||||
// it retryable instead of letting it sit in execution_failed. (#3408 P1)
|
||||
if (isSessionResumeExpiredText(text)) {
|
||||
return classification(
|
||||
'process_exit',
|
||||
'session_resume_expired',
|
||||
'session_init',
|
||||
true,
|
||||
'retry',
|
||||
);
|
||||
}
|
||||
|
||||
if (errorCode === 'AGENT_UNAVAILABLE') {
|
||||
return classification(
|
||||
'process_exit',
|
||||
@@ -570,7 +620,7 @@ export function classifyRunFailure(
|
||||
);
|
||||
}
|
||||
|
||||
if (errorCode === 'RATE_LIMITED' || serviceFailure === 'RATE_LIMITED' || isHardQuotaText(text)) {
|
||||
if (errorCode === 'RATE_LIMITED' || serviceFailure === 'RATE_LIMITED' || isHardQuotaText(text) || isRateLimitText(text)) {
|
||||
const retryable = retryableHint ?? !isHardQuotaText(text);
|
||||
return classification(
|
||||
'rate_limit',
|
||||
|
||||
@@ -52,6 +52,29 @@ describe('AMR account failure classification', () => {
|
||||
}
|
||||
});
|
||||
|
||||
it('classifies the Chinese vela pre-charge (额度预扣) failure as a rechargeable balance error', () => {
|
||||
// Real production text sampled from Langfuse (#3408 P1): vela reports the
|
||||
// wallet pre-charge failure in Chinese, which previously leaked into the
|
||||
// opaque execution_failed bucket instead of insufficient_balance.
|
||||
const failure = classifyAmrAccountFailure(
|
||||
'预扣费额度失败, 用户[141283]剩余额度: 💰0.040000, 需要预扣费额度: 💰0.060000 (request id: Babc)',
|
||||
);
|
||||
expect(failure).toMatchObject({
|
||||
code: 'AMR_INSUFFICIENT_BALANCE',
|
||||
action: 'recharge',
|
||||
actionUrl: DEFAULT_AMR_RECHARGE_URL,
|
||||
});
|
||||
});
|
||||
|
||||
it('classifies Chinese balance/quota shortfall variants as AMR balance errors', () => {
|
||||
for (const text of ['余额不足,请充值后重试', '账户额度不足']) {
|
||||
expect(classifyAmrAccountFailure(text)).toMatchObject({
|
||||
code: 'AMR_INSUFFICIENT_BALANCE',
|
||||
action: 'recharge',
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
it('does not classify non-billing throttling as AMR balance errors', () => {
|
||||
expect(classifyAmrAccountFailure('HTTP 429 rate limit reached')).toBeNull();
|
||||
expect(classifyAmrAccountFailure('quota exceeded')).toBeNull();
|
||||
|
||||
@@ -3,7 +3,14 @@ import { describe, expect, it, vi } from 'vitest';
|
||||
vi.mock('../src/integrations/vela-errors.js', () => ({
|
||||
classifyAmrAccountFailure(text: string) {
|
||||
const value = String(text || '').toLowerCase();
|
||||
if (value.includes('insufficient balance')) {
|
||||
// Mirror the real detector's signals exercised by these tests, including
|
||||
// the Chinese vela pre-charge text (see integrations/vela-errors.test.ts).
|
||||
if (
|
||||
value.includes('insufficient balance') ||
|
||||
value.includes('预扣费额度失败') ||
|
||||
value.includes('余额不足') ||
|
||||
value.includes('额度不足')
|
||||
) {
|
||||
return { code: 'AMR_INSUFFICIENT_BALANCE' as const };
|
||||
}
|
||||
if (value.includes('authentication required') || value.includes('not authenticated') || value.includes('unauthorized')) {
|
||||
@@ -919,3 +926,128 @@ describe('execution_failed close-reason refinement', () => {
|
||||
).toMatchObject({ failure_category: 'process_exit', failure_detail: 'exit_code' });
|
||||
});
|
||||
});
|
||||
|
||||
// Reclassify AMR/vela upstream failures that currently fall into the opaque
|
||||
// `execution_failed` bucket. These carry the generic `AGENT_EXECUTION_FAILED`
|
||||
// error code, and the real cause is only in the (often Chinese) upstream error
|
||||
// text, so the English-only detectors miss them. Real production texts were
|
||||
// sampled from Langfuse (#3408 P1). Each must land in its true product-view
|
||||
// category instead of the engineering-view opaque bucket.
|
||||
describe('classifyRunFailure — AMR/vela reclassification out of execution_failed', () => {
|
||||
it('classifies a vela Chinese pre-charge (insufficient balance) failure as insufficient_balance', () => {
|
||||
const result = classify(
|
||||
'AGENT_EXECUTION_FAILED',
|
||||
'预扣费额度失败, 用户[141283]剩余额度: 💰0.040000, 需要预扣费额度: 💰0.060000 (request id: B202606220543379765673248268d9d6vVKaiRPCMA)',
|
||||
);
|
||||
expect(result?.failure_category).toBe('insufficient_balance');
|
||||
expect(result?.failure_detail).toBe('amr_insufficient_balance');
|
||||
expect(result?.user_action).toBe('recharge');
|
||||
});
|
||||
|
||||
it('classifies a Chinese 429 rate-limit text as a retryable rate_limit_429', () => {
|
||||
const result = classify(
|
||||
'AGENT_EXECUTION_FAILED',
|
||||
'429 您的账户已达到速率限制,请您控制请求频率',
|
||||
);
|
||||
expect(result?.failure_category).toBe('rate_limit');
|
||||
expect(result?.failure_detail).toBe('rate_limit_429');
|
||||
expect(result?.retryable).toBe(true);
|
||||
});
|
||||
|
||||
it('classifies a vela "model not in allowed list" rejection as model_unavailable', () => {
|
||||
const result = classify(
|
||||
'AGENT_EXECUTION_FAILED',
|
||||
'API Error: 400 model deepseek-v4-pro-202606 not in allowed list',
|
||||
);
|
||||
expect(result?.failure_category).toBe('model_unavailable');
|
||||
expect(result?.failure_detail).toBe('model_not_found');
|
||||
expect(result?.user_action).toBe('switch_model');
|
||||
});
|
||||
});
|
||||
|
||||
// The agent binary being absent at its resolved path also leaks into the opaque
|
||||
// execution_failed bucket (#3408 P1). Real production texts sampled from
|
||||
// Langfuse. These are an install/PATH problem, not an opaque engine failure.
|
||||
describe('classifyRunFailure — binary-not-found reclassification out of execution_failed', () => {
|
||||
it('classifies a Windows "is not recognized as an internal or external command" as cli_not_installed', () => {
|
||||
const result = classify(
|
||||
'AGENT_EXECUTION_FAILED',
|
||||
"'node' is not recognized as an internal or external command, operable program or batch file.",
|
||||
);
|
||||
expect(result?.failure_category).toBe('process_exit');
|
||||
expect(result?.failure_detail).toBe('cli_not_installed');
|
||||
});
|
||||
|
||||
it('classifies a "spawn <path> ENOENT" (missing executable) as cli_not_installed', () => {
|
||||
const result = classify(
|
||||
'AGENT_EXECUTION_FAILED',
|
||||
'Error: spawn /opt/homebrew/lib/node_modules/@openai/codex/codex ENOENT',
|
||||
);
|
||||
expect(result?.failure_category).toBe('process_exit');
|
||||
expect(result?.failure_detail).toBe('cli_not_installed');
|
||||
});
|
||||
});
|
||||
|
||||
// Batch A: more named causes that currently leak into execution_failed, routed
|
||||
// to existing categories. Real production texts sampled from Langfuse (#3408 P1).
|
||||
describe('classifyRunFailure — batch A reclassification out of execution_failed', () => {
|
||||
it('classifies a local-runtime "Prefill context too large" as prompt_too_large', () => {
|
||||
const result = classify(
|
||||
'AGENT_EXECUTION_FAILED',
|
||||
'MLX prefill memory guard rejected this prompt: Prefill context too large for available memory (preflight safety margin)',
|
||||
);
|
||||
expect(result?.failure_category).toBe('prompt_too_large');
|
||||
expect(result?.failure_detail).toBe('prompt_too_large');
|
||||
});
|
||||
|
||||
it('classifies an ACP "thread/start failed" as agent_protocol_error', () => {
|
||||
const result = classify(
|
||||
'AGENT_EXECUTION_FAILED',
|
||||
'Reading prompt from stdin... Error: thread/start: thread/start failed: failed to start session',
|
||||
);
|
||||
expect(result?.failure_category).toBe('process_exit');
|
||||
expect(result?.failure_detail).toBe('agent_protocol_error');
|
||||
});
|
||||
|
||||
it('classifies a vela "login fail: carry the API secret key" as an auth failure', () => {
|
||||
const result = classify(
|
||||
'AGENT_EXECUTION_FAILED',
|
||||
"login fail: Please carry the API secret key in the 'Authorization' field of the request header (1004)",
|
||||
);
|
||||
expect(result?.failure_category).toBe('auth');
|
||||
});
|
||||
|
||||
it('classifies a local model server with no model loaded (LM Studio) as local_model_not_loaded', () => {
|
||||
// opencode pointed at a local LM Studio provider that has no model loaded.
|
||||
// Independent of the model name we pass: the user must load a model first.
|
||||
const result = classify(
|
||||
'AGENT_EXECUTION_FAILED',
|
||||
"No models loaded. Please load a model in the developer page or use the 'lms load' command.",
|
||||
);
|
||||
expect(result?.failure_category).toBe('model_unavailable');
|
||||
expect(result?.failure_detail).toBe('local_model_not_loaded');
|
||||
expect(result?.user_action).toBe('switch_model');
|
||||
});
|
||||
|
||||
it('classifies a stale Claude session resume as a retryable session_resume_expired', () => {
|
||||
// The daemon already cleared the stale session id; the next turn starts
|
||||
// fresh. This is recoverable, not an opaque engine crash.
|
||||
const result = classify(
|
||||
'AGENT_EXECUTION_FAILED',
|
||||
'The previous Claude session could not be resumed (it may have expired). Resend your message to continue with a fresh session.',
|
||||
);
|
||||
expect(result?.failure_category).toBe('process_exit');
|
||||
expect(result?.failure_detail).toBe('session_resume_expired');
|
||||
expect(result?.retryable).toBe(true);
|
||||
expect(result?.user_action).toBe('retry');
|
||||
});
|
||||
|
||||
it('classifies the raw Claude CLI "no conversation found with session id" as session_resume_expired', () => {
|
||||
const result = classify(
|
||||
'AGENT_EXECUTION_FAILED',
|
||||
'no conversation found with session id 1d2c3b4a-0000-0000-0000-000000000000',
|
||||
);
|
||||
expect(result?.failure_category).toBe('process_exit');
|
||||
expect(result?.failure_detail).toBe('session_resume_expired');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -189,6 +189,106 @@ describe('run failure telemetry smoke', () => {
|
||||
}
|
||||
});
|
||||
|
||||
it('reclassifies upstream + install/env failures end-to-end through a real daemon run (#3408 P1)', async () => {
|
||||
// End-to-end proof for the reclassification: a real agent process emits the
|
||||
// production error text (or fails to spawn), the daemon records it into the
|
||||
// run's events.jsonl, and classifyRunFailure (on the REAL recorded events,
|
||||
// not a hand-built input) must land it in the correct category instead of
|
||||
// the opaque execution_failed bucket. Generous inactivity timeout so the
|
||||
// 100ms exit always wins the race (this test is not about timeouts).
|
||||
binDir = await mkdtemp(path.join(os.tmpdir(), 'od-amr-reclassify-bin-'));
|
||||
await writeFakeClaude(
|
||||
binDir,
|
||||
'amr-balance',
|
||||
'预扣费额度失败, 用户[141283]剩余额度: 💰0.040000, 需要预扣费额度: 💰0.060000 (request id: Babc)',
|
||||
);
|
||||
await writeFakeClaude(binDir, 'amr-ratelimit', '429 您的账户已达到速率限制,请您控制请求频率');
|
||||
await writeFakeClaude(binDir, 'amr-model', 'API Error: 400 model deepseek-v4-pro-202606 not in allowed list');
|
||||
await writeFakeClaude(
|
||||
binDir,
|
||||
'env-node-path',
|
||||
"'node' is not recognized as an internal or external command, operable program or batch file.",
|
||||
);
|
||||
// The agent itself reports a missing vendored sub-binary (real codex shape).
|
||||
await writeFakeClaude(
|
||||
binDir,
|
||||
'env-spawn-enoent',
|
||||
'Error: spawn /opt/homebrew/lib/node_modules/@openai/codex/codex ENOENT',
|
||||
);
|
||||
await writeFakeClaude(
|
||||
binDir,
|
||||
'a-prefill',
|
||||
'MLX prefill memory guard rejected this prompt: Prefill context too large for available memory',
|
||||
);
|
||||
await writeFakeClaude(
|
||||
binDir,
|
||||
'a-thread-start',
|
||||
'Reading prompt from stdin... Error: thread/start: thread/start failed: failed to start session',
|
||||
);
|
||||
await writeFakeClaude(
|
||||
binDir,
|
||||
'a-auth',
|
||||
"login fail: Please carry the API secret key in the 'Authorization' field of the request header (1004)",
|
||||
);
|
||||
await writeFakeClaude(
|
||||
binDir,
|
||||
'a-lmstudio',
|
||||
"No models loaded. Please load a model in the developer page or use the 'lms load' command.",
|
||||
);
|
||||
await writeFakeClaude(
|
||||
binDir,
|
||||
'a-resume-expired',
|
||||
'no conversation found with session id 1d2c3b4a-0000-0000-0000-000000000000',
|
||||
);
|
||||
|
||||
process.env.OD_CHAT_RUN_INACTIVITY_TIMEOUT_MS = '5000';
|
||||
delete process.env.POSTHOG_KEY;
|
||||
started = await startServer({ port: 0, returnServer: true }) as StartedServer;
|
||||
await putConfig(started.url, {
|
||||
telemetry: { metrics: true, content: true, artifactManifest: false },
|
||||
privacyDecisionAt: Date.now(),
|
||||
});
|
||||
|
||||
const cases = [
|
||||
{ bin: 'amr-balance', category: 'insufficient_balance', detail: 'amr_insufficient_balance' },
|
||||
{ bin: 'amr-ratelimit', category: 'rate_limit', detail: 'rate_limit_429' },
|
||||
{ bin: 'amr-model', category: 'model_unavailable', detail: 'model_not_found' },
|
||||
{ bin: 'env-node-path', category: 'process_exit', detail: 'cli_not_installed' },
|
||||
{ bin: 'env-spawn-enoent', category: 'process_exit', detail: 'cli_not_installed' },
|
||||
{ bin: 'a-prefill', category: 'prompt_too_large', detail: 'prompt_too_large' },
|
||||
{ bin: 'a-thread-start', category: 'process_exit', detail: 'agent_protocol_error' },
|
||||
{ bin: 'a-auth', category: 'auth', detail: 'auth_required' },
|
||||
{ bin: 'a-lmstudio', category: 'model_unavailable', detail: 'local_model_not_loaded' },
|
||||
{ bin: 'a-resume-expired', category: 'process_exit', detail: 'session_resume_expired' },
|
||||
] as const;
|
||||
|
||||
for (const item of cases) {
|
||||
await putConfig(started.url, {
|
||||
agentId: 'claude',
|
||||
agentCliEnv: { claude: { CLAUDE_BIN: path.join(binDir, item.bin) } },
|
||||
});
|
||||
const run = await createAndWaitForRun(started.url, {
|
||||
caseId: item.bin,
|
||||
agentId: 'claude',
|
||||
message: `od-amr-reclassify-${item.bin}`,
|
||||
});
|
||||
const events = await readRunEvents(run.eventsLogPath);
|
||||
const errorCode = deriveRunErrorCode(run);
|
||||
const failure = classifyRunFailure({
|
||||
result: runResultFromStatus(run.status),
|
||||
status: run,
|
||||
...(errorCode ? { errorCode } : {}),
|
||||
agentId: run.agentId,
|
||||
events,
|
||||
});
|
||||
expect(run.status, item.bin).toBe('failed');
|
||||
// The reclassification must NOT leave it in the opaque bucket.
|
||||
expect(failure?.failure_detail, item.bin).not.toBe('execution_failed');
|
||||
expect(failure?.failure_category, item.bin).toBe(item.category);
|
||||
expect(failure?.failure_detail, item.bin).toBe(item.detail);
|
||||
}
|
||||
});
|
||||
|
||||
it('reports the terminal Langfuse fallback for headerless run requests', async () => {
|
||||
binDir = await mkdtemp(path.join(os.tmpdir(), 'od-run-failure-fallback-bin-'));
|
||||
await writeFakeClaude(binDir, 'claude-terminal-failure', 'terminal fallback smoke failure');
|
||||
|
||||
@@ -268,6 +268,7 @@ export type TrackingRunFailureDetail =
|
||||
| 'model_not_found'
|
||||
| 'model_not_supported'
|
||||
| 'model_disabled'
|
||||
| 'local_model_not_loaded'
|
||||
| 'cli_version_incompatible'
|
||||
| 'prompt_too_large'
|
||||
| 'upstream_5xx'
|
||||
@@ -290,6 +291,7 @@ export type TrackingRunFailureDetail =
|
||||
| 'spawn_eperm'
|
||||
| 'stdin_write_eof'
|
||||
| 'agent_protocol_error'
|
||||
| 'session_resume_expired'
|
||||
| 'fabricated_role_marker'
|
||||
| 'permission_request_not_found'
|
||||
| 'qoder_stop_sequence'
|
||||
|
||||
Reference in New Issue
Block a user