## The bug
buzz-agent emitted its `usage_update` notification in exactly one place:
after `ctx.run()` returned. Until that moment a turn's token counters
lived only in the prompt task's stack frame. **A turn killed mid-flight
reported nothing at all** — the provider had already billed every round
it completed, and no consumer ever saw any of it.
That is not a corner case for anything that ends a turn on a clock. It
is the normal case for a long-horizon benchmark run that relaunches its
agent between phases.
## How big
Measured against a provider's own billing ledger over one run's window:
| | provider ledger | what we recorded |
|---|---|---|
| the relaunched lead seat | $485 / 348M tok | $98.99 / 90.3M tok |
| the two seats that were not relaunched | $29.90 / 856M | $25.81 / 765M
— reconciles |
97% of that run's usage rows came back all zeros, against 1–4% for
comparable runs that never relaunch. In one 450-phase trial exactly 7
phases recorded any usage — and each of those carries 177k–437k input
tokens, a whole session's worth landing in the one phase that happened
to end gracefully.
Worth being precise about what was *not* wrong, since both were
plausible and both were checked:
- **Not pricing.** The rates were verified against the provider's
endpoints API and match what we charge.
- **Not a truncation bug.** The usage files were intact and internally
consistent. The tokens were never captured in the first place.
## The fix
The run loop now emits a session-cumulative `usage_update` after every
usage-bearing provider response, so an interrupted turn has reported
everything but its single in-flight request.
- **Emitting more than once per turn is already part of the contract.**
buzz-acp's `UsageTracker` advances its committed baseline only at
publish time, and goose behaves the same way — which is why the tracker
was written to tolerate it.
- **The turn-start session baseline is snapshotted into `RunCtx`** so
the mid-turn figure stays *session*-cumulative. A turn-local number
would be discarded by a high-water-mark consumer and lose the turn
entirely; there is a test for exactly that.
- **Snapshot by value, not a session handle.** The loop reports once per
round, and taking the sessions lock on each would serialise concurrent
sessions behind one another's provider round-trips. Nothing else
advances those counters while the turn holds `busy`, so it cannot go
stale.
- **One shared `wire::usage_update_payload`** for both call sites, so
the mid-turn and end-of-turn shapes cannot drift. A drift there would
present as tokens silently vanishing, which is the failure this
reporting exists to prevent.
## Why not a SIGTERM handler
That was the obvious shape and it does not work. At signal time the
counters are not sitting anywhere a handler could reach — they are in
the turn's stack frame, and the value the handler would need has not
been folded into the session yet. Making usage durable *during* the turn
is what actually fixes it; once it is, a handler adds nothing beyond the
in-flight request, whose cost is unknown until its response lands.
## Tests
- `usage_is_reported_after_each_round_not_only_at_turn_end` — two
rounds; asserts the **first** notification carries round 1's counts
alone, proving it went out before round 2 returned.
- `mid_turn_usage_includes_earlier_turns` — a mid-turn report must be
session-cumulative, not turn-local.
buzz-agent 18/18 on the `fake_llm` suite, 382 unit. `cargo fmt` /
`clippy` / `cargo check --workspace --all-targets` clean.
## Scope
Agent-side only, against `main`. The matching harness change — settling
usage on the timeout path, which was skipped on the reasoning that an
incomplete turn has nothing to flush — is **#4553**, against the
benchmark branch, since that harness does not exist on `main`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Atish Patel <atish@squareup.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
## Why
A Buzz agent's assistant text and reasoning are never shown to anyone —
only what it posts through the CLI. A turn that runs fifteen tool calls
and never publishes is a silent failure: the requester waits on a result
that was produced and thrown away.
This adds an optional reminder at the end-of-turn gate, off by default.
Tyler asked for it in buzz-mesh; plan iterated to **9.5/10 with @Wren**
(Minimalness 9.7, Elegance 9.5, Correctness 9.3).
## What
`BUZZ_AGENT_REQUIRE_REPLY=1` (default off, per-agent opt-in). A turn
about to end with no recognized attempt to post gets a reminder and is
rerolled. **At most two, then the turn ends regardless** — the guard
catches accidental omission, it does not compel speech. The reminder
text explicitly licenses silence so it cannot fight the base prompt's
"silence is usually correct."
**This is not a new MCP hook.** `RunCtx::run` *is* the turn, so the two
per-turn locals need no plumbing, and every tool call already passes
through it with arguments visible. The objection is appended at the
existing `_Stop` gate and rides `push_hook_outputs_as_tool_results`, so
the model receives it as a lower-trust tool result with `{hook, server,
text}` attribution. No new trust path, no new lifecycle event, no
dev-mcp or CLI protocol change.
Earlier revisions of this plan needed four crates (a `_UserPromptSubmit`
hook, a marker file, a `buzz-cli` change, dev-mcp state). Tyler pointed
out the agent already knows both facts; that deleted all of it. Net
runtime change is ~35 lines in `agent.rs` + ~4 in `config.rs`.
### Recognition contract
A registered non-hook tool whose qualified name ends in `__shell`, whose
`command` argument contains `messages send` or `reactions add`.
- **The `__` separator is exact, not approximate.** Given `has()` +
`!is_hook()`, `ends_with("__shell")` is *provably equivalent* to a bare
name of `shell`: registration forbids `__` in server and bare names
(`mcp.rs:227,268`) and qnames are `{server}__{bare}`, so a trailing
`__shell` could only straddle the separator if the bare name began with
`_` — which `is_hook` excludes. Without the separator, `powershell` and
`noshell` would match.
- **Reads the structured `command` field**, not serialized arguments, so
a `description` that quotes a send cannot disarm the guard, and a
non-string `command` is rejected rather than coerced.
- **Detects an attempt, not a successful publish.** A failed send
already returns non-zero exit and error JSON — louder than this
reminder. The variable is named `buzz_reply_call_seen` so the code can't
pretend otherwise.
- **Checked after the per-turn tool-call cap**, since a discarded call
never ran.
- `messages send` also covers `messages send-diff`. Reactions count
because the base prompt directs agents to react rather than post a bare
acknowledgement.
**Known limits, both deliberate and documented:** a command assembled at
runtime (`$CMD`) or hidden in a wrapper script is missed; text that
merely quotes a send (`echo "buzz messages send"`) matches. Missing a
real post is the expensive direction and substring matching is the
forgiving one there. Neither edge is pinned by a test, so the matcher
stays free to improve.
### Budget
Reminders share `BUZZ_AGENT_STOP_MAX_REJECTIONS`, the existing outer cap
on every end-turn objection. Default 3 fits both; at 1 only one fits; at
0 the guard is off with the hooks. A round carrying both a hook
objection and a reminder costs one rejection and delivers both texts. An
independent budget would either violate that bound or need a second
arbitration rule.
## Prior art
- **#3467** (closed) built the same detector one layer up in `buzz-acp`
for a different remedy. None of its symbols are on main — this borrows
its permission to be coarse, but reads structured data that ACP didn't
have.
- **#3648** (open) detects turns with *no output at all*; a turn with
fifteen tool calls and no post counts as output there, so it does not
cover this case.
- **#3741** (merged) is mesh-only.
## Testing
**14 new tests.** 4 unit tests on the matcher; 10 integration tests
through the ACP wire harness: off by default, `=0` still off, opted-in
silent → exactly 2 reminders then `end_turn`, registered `fake__shell`
send → 0 reminders, hallucinated `fake__shell` → still reminded, publish
call truncated past the 64-call cap → still reminded, budget 1 → 1
reminder, budget 0 → off, combined `_Stop` hook objection + reminder →
one round both texts and after 2 reminders the hook objection continues
alone, unparseable `=true` → startup error naming the key.
**10 mutation checks, each breaking a specific named test** — neutralize
the nag cap, stop sharing the budget, neutralize `buzz_reply_call_seen`,
drop `has`/`is_hook`, ignore the flag, drop the `__`, drop `reactions
add`, read serialized args, move detection before truncation.
`tests/bin/fake_mcp.rs` gains `FAKE_MCP_SHELL_TOOL=1`: it previously
exposed no tool with a bare name of `shell`, so the satisfied-guard path
was untestable.
Full `cargo test -p buzz-agent` green at 9e0ae1f04; clippy `-D warnings`
and `cargo fmt --check` clean.
**Unrelated flake found:**
`cancelled_turn_with_usage_emits_notification_before_response`
(`tests/fake_llm.rs`) is timing-sensitive. Under 10 loaded cores it
fails **2/20 on this branch and 1/20 at unmodified
`origin/main@02be413b8`** — pre-existing, not caused by this change
(which is inert without the env var). Flagging so it isn't misattributed
to the next PR that's open when CI hits it.
## Docs
`crates/buzz-agent/README.md` is the primary home (env var, recognition
contract, limits, budget interaction). `docs/MCP_DRIVEN_HOOKS.md` gets a
short cross-reference explaining this is *not* a hook — otherwise
readers hunt for a `_ReplyGuard` tool that doesn't exist.
---------
Signed-off-by: tlongwell-block <109685178+tlongwell-block@users.noreply.github.com>
Signed-off-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz>
Co-authored-by: Dawn (sprout agent) <c6237ef84fa537c78dcee78efd2d4e59f728859c7f194da42ac51ededfa0be05@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz>
## What
Wires genuine provider-reported `total_tokens` through the full
buzz-agent → buzz-acp publish chain so kind-44200 events carry real
per-turn and cumulative totals for OpenAI-backed models, while
preserving all existing behaviour for Anthropic and external harnesses
(goose, claude-code).
## Why
Live prod data showed 0 of 1,934 archived reports carry `totalTokens`.
Both hardcoded `total_tokens: None` in `pool.rs` and the absent field in
`buzz-agent`'s parser are root causes. This is the backend half of a
two-track fix; the display-fallback half lands in
[#2035](https://github.com/block/buzz/pull/2035).
## Changes
**`crates/buzz-agent/src/types.rs`**
- Added `total_tokens: Option<u64>` to `LlmResponse` with an explicit
doc comment that NIP-AM forbids deriving it.
- Added `TurnTotalState` enum (`Unseen | Exact(u64) | Unknown`) with
`fold()` and `exact_value()` — the tri-state accumulator that
distinguishes not-yet-observed from permanently poisoned.
**`crates/buzz-agent/src/llm.rs`**
- `parse_responses` and `parse_openai`: read `usage.total_tokens` from
OpenAI Chat Completions (including Databricks routes) and the Responses
API via `sum_usage`.
- Anthropic: explicit `total_tokens: None` — no genuine total available;
NIP-AM forbids summing categories.
**`crates/buzz-agent/src/agent.rs`**
- Added `turn_total_state: &'a mut TurnTotalState` to `RunCtx`.
- Fold `response.total_tokens` into the accumulator after each
usage-bearing response; non-usage-bearing responses (keepalive/stream
frames) do not poison.
**`crates/buzz-agent/src/lib.rs`**
- Added `accumulated_total_state: TurnTotalState` to `Session` (default
`Unseen`).
- Per-turn state passed to `RunCtx`, folded into session cumulative
after each turn.
- Emits `accumulatedTotalTokens` in `usage_update` only when cumulative
is `Exact(n)`.
**`crates/buzz-acp/src/usage.rs`**
- Added `accumulated_total_tokens: Option<u64>` (serde default) to
`UsageUpdatePayload` — optional for goose compat.
- Added `last_total: Option<u64>` to `SessionState`.
- Added `turn_total_tokens` and `cumulative_total_tokens` to `TurnUsage`
(field-local — never affect `delta_reliable`).
- Derive turn-total delta only when prev and current are both `Some` and
monotonic; absence, decrease, or no baseline leaves only the total delta
null without touching input/output reliability.
**`crates/buzz-acp/src/pool.rs`**
- Replaced both hardcoded `total_tokens: None` in
`publish_agent_turn_metric` with `usage.turn_total_tokens` and
`usage.cumulative_total_tokens`.
## Tests
20 new tests across the four touched files:
| File | Tests |
|------|-------|
| `types.rs` | `TurnTotalState` fold, accumulation, exact_value, default
(7 tests) |
| `llm.rs` | Chat present/absent, Responses present/absent, Anthropic
always-None (5 tests) |
| `usage.rs` | First turn no baseline, second-turn delta, cumulative
decrease (field-local), current absent, goose-shaped deserialization,
baseline absent (6 tests) |
| `pool.rs` | Exact turn+cumulative mapping, null totals never derived
(2 tests) |
`cargo test -p buzz-acp -p buzz-agent` — all passing, 0 failures.
## Scope
Boundary: `crates/buzz-agent/**` + `crates/buzz-acp/**` only. Desktop
unchanged.
`costUsd` explicitly out of scope.
Related: [#2035](https://github.com/block/buzz/pull/2035)
---------
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Co-authored-by: npub1g8493u0xfsjrvflg4n08ezd7vec99mnwzlv0qgwpr9d7gvjwhuzqx59rhw <41ea58f1e64c243627e8acde7c89be667052ee6e17d8f021c1195be4324ebf04@buzz.block.builderlab.xyz>
