mirror of
https://github.com/nexu-io/open-design.git
synced 2026-10-27 09:51:12 +01:00
maoxin1234andGitHub
55b3762afe
* feat(daemon): capture per-request request_id + token usage for claude_code runs The claude-stream-json parser read every assistant message's `message.id` only to dedup streamed text/thinking and then dropped it, and never read `message.usage` at all — so a run with N model requests collapsed into a single run-level `result.usage` record with no request id. That left request-level cost/percentile analysis stuck at run-level (~0.9% coverage). Surface the per-request granularity the wire already carries: - claude-stream.ts emits a new `request_usage` event per assistant message, keyed by `message.id` (the provider `msg_…` request id), carrying the per-message input/output/cache tokens via a `perRequestUsageFrom` helper. - packages/contracts adds `request_usage` to both the live SSE `DaemonAgentPayload` and the `PersistedAgentEvent` union; server.ts maps the live event to its persisted form. - mocks/lib/format-claude.mjs now emits per-message `usage` summing to the run-level `result.usage`, so the capture path is verifiable by local replay without burning provider budget. The per-request token sum reconciles against the run-level `result.usage` aggregate — the correctness anchor. New daemon test plus regenerated claude golden cover both the synthetic stream and the mock-replay path. No change to run execution; only telemetry/contract surface is added. claude-stream- json only for now; other runtime families ship per-request shapes in follow-ups. Closes #4610 * feat(daemon): thread per-request usage through Langfuse + PostHog run telemetry Addresses review on #4620: the parser emitted the per-request records but they stopped at SSE/message persistence and never reached the telemetry sinks named in #4610's acceptance criteria, so PostHog/Langfuse still had no per-request request_id + token coverage. Wire the records through the existing run telemetry choke point: - run-analytics-observability.ts adds `scanRunEventsForPerRequestUsageAnalytics`, a pure scanner that collects the per-request records keyed by request_id and checks the reconciliation invariant (per-request token sum == run-level `result.usage`). Both sinks read this one source. - langfuse-bridge.ts `collectAgentEvents` exports one `agent-request-usage` Langfuse event per model request, keyed by the provider request id. - routes/runs.ts adds `run_finished` coverage properties (`request_usage_count`, per-request token sums, and `request_usage_reconciles_aggregate`) so the PostHog request-level surface lifts off the ~0.9% floor; contract fields added to RunFinishedProps. Test extended: a multi-request claude run produces telemetry records keyed by request_id whose token sum reconciles with `result.usage`, plus a non-reconciling guard. langfuse-bridge + run-analytics suites stay green. * fix(daemon): fold cache tokens into per-request usage reconciliation Addresses review on #4620: request_usage_reconciles_aggregate compared only input/output tokens, so a run whose prompt/completion tokens matched but whose cache creation/read tokens drifted still reported `true` — letting the new PostHog/Langfuse telemetry certify the #4610 invariant when part of `result.usage` had not reconciled. scanRunEventsForPerRequestUsageAnalytics now reads the aggregate `cache_creation_input_tokens` / `cache_read_input_tokens` and folds every field `result.usage` reported into the boolean (each compared only when the aggregate carries it, mirroring the input/output leniency). Added a fixture where cache tokens mismatch while input/output still match, so the flag stays false and this cannot regress. * feat(daemon): map request_usage to durable PersistedAgentEvent * feat(daemon): wire per-request usage rollup into run_finished analytics lifecycle * ci: trigger fresh validation run for approved changes * feat(daemon): preserve per-request usage in truncation-proof run side ledger
mocks/golden/ — daemon-event regression snapshots
Each <trace-id>.events.json is a committed snapshot of the event
sequence that flows out of the OD daemon's stream handler when fed the
matching mock CLI's output for that recording.
The shape is:
{
"agent": "claude",
"trace": "314d6833-...",
"events": [
{ "type": "status", ... },
{ "type": "tool_use", "name": "Read", ... },
{ "type": "tool_result", ... },
...
]
}
Why these exist
mocks/scripts/smoke-test.sh confirms the mock CLIs run and emit
events. It doesn't tell you whether the events have the right shape
— a parser change in apps/daemon/src/runtimes/claude-stream.ts or
runtimes/json-event-stream.ts could silently drop a field, rename
sessionID → sessionId, or stop emitting turn_end. The goldens
are diffed by apps/daemon/tests/mocks-golden.test.ts; a regression
makes that test fail loudly.
Updating after an intentional parser change
MOCKS_GOLDEN_UPDATE=1 pnpm --filter @open-design/daemon test mocks-golden
git diff mocks/golden/
git add mocks/golden/ && git commit -m "mocks: refresh goldens — parser X now emits Y"
The diff is the part the reviewer eyeballs. If the change matches what the parser refactor intended, ship. If it doesn't, the refactor broke something.
Coverage
| File | Agent | Trace | Tools | Why it's representative |
|---|---|---|---|---|
314d6833…events.json |
claude | 17-tool agent-browser session | 8 | Median complexity for claude; exercises Read / Write / Edit / Bash tool shapes + content_block streaming |
9a9522ec…events.json |
opencode | data-report session | 7 | Covers step_start / step_finish / tool_use / cost envelope |
dcdff3b3…events.json |
codex | 14-tool refactor | 14 | High tool density; exercises thread.started + thread message streaming |
Other agents are not golden-tested here for different reasons:
cursor-agentand plain streams do not emit structured tool events and are covered by simpler unit tests.- The retained Gemini parser emits structured
tool_use/tool_resultevents and is covered byapps/daemon/tests/runtimes/json-event-stream.test.ts, but has no committed golden. - ACP / vela use JSON-RPC over stdio rather than streamed stdout and need a
different harness; see
apps/daemon/tests/acp.test.ts.