## What
Wires genuine provider-reported `total_tokens` through the full
buzz-agent → buzz-acp publish chain so kind-44200 events carry real
per-turn and cumulative totals for OpenAI-backed models, while
preserving all existing behaviour for Anthropic and external harnesses
(goose, claude-code).
## Why
Live prod data showed 0 of 1,934 archived reports carry `totalTokens`.
Both hardcoded `total_tokens: None` in `pool.rs` and the absent field in
`buzz-agent`'s parser are root causes. This is the backend half of a
two-track fix; the display-fallback half lands in
[#2035](https://github.com/block/buzz/pull/2035).
## Changes
**`crates/buzz-agent/src/types.rs`**
- Added `total_tokens: Option<u64>` to `LlmResponse` with an explicit
doc comment that NIP-AM forbids deriving it.
- Added `TurnTotalState` enum (`Unseen | Exact(u64) | Unknown`) with
`fold()` and `exact_value()` — the tri-state accumulator that
distinguishes not-yet-observed from permanently poisoned.
**`crates/buzz-agent/src/llm.rs`**
- `parse_responses` and `parse_openai`: read `usage.total_tokens` from
OpenAI Chat Completions (including Databricks routes) and the Responses
API via `sum_usage`.
- Anthropic: explicit `total_tokens: None` — no genuine total available;
NIP-AM forbids summing categories.
**`crates/buzz-agent/src/agent.rs`**
- Added `turn_total_state: &'a mut TurnTotalState` to `RunCtx`.
- Fold `response.total_tokens` into the accumulator after each
usage-bearing response; non-usage-bearing responses (keepalive/stream
frames) do not poison.
**`crates/buzz-agent/src/lib.rs`**
- Added `accumulated_total_state: TurnTotalState` to `Session` (default
`Unseen`).
- Per-turn state passed to `RunCtx`, folded into session cumulative
after each turn.
- Emits `accumulatedTotalTokens` in `usage_update` only when cumulative
is `Exact(n)`.
**`crates/buzz-acp/src/usage.rs`**
- Added `accumulated_total_tokens: Option<u64>` (serde default) to
`UsageUpdatePayload` — optional for goose compat.
- Added `last_total: Option<u64>` to `SessionState`.
- Added `turn_total_tokens` and `cumulative_total_tokens` to `TurnUsage`
(field-local — never affect `delta_reliable`).
- Derive turn-total delta only when prev and current are both `Some` and
monotonic; absence, decrease, or no baseline leaves only the total delta
null without touching input/output reliability.
**`crates/buzz-acp/src/pool.rs`**
- Replaced both hardcoded `total_tokens: None` in
`publish_agent_turn_metric` with `usage.turn_total_tokens` and
`usage.cumulative_total_tokens`.
## Tests
20 new tests across the four touched files:
| File | Tests |
|------|-------|
| `types.rs` | `TurnTotalState` fold, accumulation, exact_value, default
(7 tests) |
| `llm.rs` | Chat present/absent, Responses present/absent, Anthropic
always-None (5 tests) |
| `usage.rs` | First turn no baseline, second-turn delta, cumulative
decrease (field-local), current absent, goose-shaped deserialization,
baseline absent (6 tests) |
| `pool.rs` | Exact turn+cumulative mapping, null totals never derived
(2 tests) |
`cargo test -p buzz-acp -p buzz-agent` — all passing, 0 failures.
## Scope
Boundary: `crates/buzz-agent/**` + `crates/buzz-acp/**` only. Desktop
unchanged.
`costUsd` explicitly out of scope.
Related: [#2035](https://github.com/block/buzz/pull/2035)
---------
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Co-authored-by: npub1g8493u0xfsjrvflg4n08ezd7vec99mnwzlv0qgwpr9d7gvjwhuzqx59rhw <41ea58f1e64c243627e8acde7c89be667052ee6e17d8f021c1195be4324ebf04@buzz.block.builderlab.xyz>
> On 8 tasks matched by name across the two runs, cost fell $8.36 →
$1.77 (4.71×) and wall-clock 12,423 s → 1,085 s (11.45×).
## Summary
Two independent, self-contained fixes to `buzz-agent`/`buzz-acp`, split
out of the benchmark branch so they can land while the harness work
continues:
1. **Request and surface Anthropic prompt caching.** buzz never sent a
`cache_control` breakpoint, so on the Databricks Anthropic route
`cache_read_input_tokens` was **structurally always 0** and the ~10×
cache-read discount was never claimed. This teaches `anthropic_body()`
to mark the cacheable prefix, and plumbs the cache split end-to-end so
accounting can price it.
2. **Pass proxy + TLS-trust env into MCP tool subprocesses**, so agent
tools on a proxy-only host stop reporting a live network as offline.
## Why the caching gap matters
The Anthropic Messages API does **not** cache unless the request carries
a `cache_control` breakpoint, and the Databricks AI Gateway — a
third-party proxy in front of the model, in the same category as
Bedrock/Vertex — does **not** auto-cache (only the first-party Anthropic
API and Claude-on-AWS do zero-config caching). So every request was
billed cold.
Measured live against the Databricks gateway
(`databricks-claude-opus-5`, 2026-07-28), the same call with and without
a single `cache_control` marker:
| Run | `input_tokens` | `cache_creation` | `cache_read` | latency |
|---|---|---|---|---|
| No `cache_control`, two byte-identical calls | 121,625 | 0 | **0** |
~9.3 s |
| With one marker — cold (write) | 4 | 121,625 | 0 | 9.3 s |
| With one marker — warm (read) | 4 | 0 | **121,625** | **4.5 s** |
One marker moved 121,625 tokens from full-price input to a 0.1× cache
read and roughly halved latency (a clean, isolated ~2.07× prefill
speedup on this single-threaded microbenchmark). The gateway honours
`cache_control`; buzz simply never sent it.
At fleet scale this was a real budget item. Across matched
Terminal-Bench solo sweeps (89 tasks, `-n 20`, before the fix), the two
OpenAI-route models independently landed at ~86–87% cache reads — the
expected shape for an agentic loop, where system + tools + append-only
history repeat every turn — while the Anthropic route returned a hard 0%
on every receipt:
| Condition | Route | Input tokens | Cache reads | Cost | Cost if
uncached | Discount |
|---|---|---|---|---|---|---|
| luna (`gpt-5-6`) | OpenAI | 20,320,818 | **17.7M (87.0%)** | $6.96 |
$22.87 | **3.28×** |
| sol (`gpt-5-6`) | OpenAI | 22,312,290 | **19.2M (85.9%)** | $37.07 |
$123.35 | **3.33×** |
| opus (`claude-opus-5`) | Anthropic | 12,459,822 | **0 (0.0%)** |
$81.31 | $81.31 | **1.00×** |
Applying luna's measured 87% read rate to the opus token counts at list
prices (`input $5/M`, `cached_input $0.5/M`, `output $25/M`) puts the
opus run at **~$32.53 vs the $81.31 actually paid — a ~60% overspend on
those 49 trials (~$89 on a full sweep)**. That is an upper bound (it
prices every cached token at the 0.1× read rate and ignores the 1.25×
write premium), and the opus discount is structurally smaller than
luna/sol's because opus emits ~3.5× more uncacheable output per trial,
which sets a floor on what caching can recover.
There is also a plausible **second-order effect**: Databricks appears to
meter its per-minute rate limit on *uncached* input tokens, so the
missing cache also cost rate-limit headroom — the opus endpoint lost 63%
of its trials to fatal 429s while running alone at one-third of a GPT
endpoint's raw throughput. This is a hypothesis, not a proven mechanism
(the only zero-cache condition is also the only Anthropic endpoint), but
it is the reading that explains the throttling with one rule instead of
two.
## Post-fix results (provisional — first trials of an in-flight re-run)
On 8 tasks matched by name across the two runs, cost fell **$8.36 →
$1.77 (4.71×)** and wall-clock **12,423 s → 1,085 s (11.45×)**.
| Metric | before (`4a955a858`) | after (`3bef1f6a`) |
|---|---|---|
| Cache reads as % of input | **0.0%** | **78.7%** (still climbing
toward the ~86% steady state) |
| `cost_usd_no_cache_discount / cost_usd` | **1.00×** | **2.18×**
(tracking the projected ~2.5×) |
| Trials with a fatal 429 (same `-n 20`) | **63%** | **15–19%** |
To be clear about attribution: **~2× of that is the clean prefill saving
from caching itself**; the rest is second-order — cached requests burn
far less rate-limit budget, so they stall less and redo less destroyed
work. The 11.45× is a system-level result specific to this throttled
workspace, not a caching benchmark. Quality held (7/8 solved in each
run). A controlled low-`-n` A/B (neither arm hitting a 429), which the
`BUZZ_AGENT_PROMPT_CACHING` opt-out exists to enable, is still owed
before this becomes a published claim.
## What changed
### 1. Request caching (`llm.rs`, `config.rs`)
`anthropic_body()` emits ephemeral `cache_control` breakpoints, gated by
`BUZZ_AGENT_PROMPT_CACHING` (**default on**, `=0` to opt out):
- **Static prefix** — marker on the `system` block. Prefix order is
`tools → system → messages`, so this single marker caches **tools +
system** together. Byte-identical on every turn of a run, and survives a
context handoff (system/tools come from cfg/mcp, not `self.history`).
- **Rolling tail + leapfrog** — marker on the last block of the last
**two** messages. The append-only history re-reads the prior turn's
prefix from cache; marking two messages (not one) keeps consecutive
breakpoints inside Anthropic's **20-block lookback window** even as tool
parallelism rises, avoiding a silent full-price miss.
An empty system prompt stays a bare string (Anthropic rejects empty text
blocks), and below-threshold prefixes are silently not cached, so the
flag is safe on by default.
### 2. Surface the cache split end-to-end — the plumbing (`types.rs`,
`llm.rs`, `agent.rs`, `lib.rs`, `usage.rs`, `acp.rs`)
This is the part that makes gaps like the one above **visible** instead
of silent. A consumer that prices all of `input_tokens` at the full rate
can't tell a route that's caching from one that isn't — the total looks
right either way. So:
- `LlmResponse` gains `cached_input_tokens` (a **subset** of
`input_tokens`, never an addition); `parse_anthropic` / `parse_openai` /
`parse_responses` each populate it.
- A `usage_first()` helper reads the cache count wherever a provider
hides it — flat `cache_read_input_tokens` (Anthropic),
`prompt_tokens_details.cached_tokens` (OpenAI chat),
`input_tokens_details.cached_tokens` (Responses) — taking the **first
present value, never a sum**. Reading only flat keys is exactly why the
OpenAI route's nested `cached_tokens` had *also* been going unclaimed:
`prompt_tokens` is already inclusive, so the total looked correct while
the discount silently went unreported.
- The per-turn/per-session accumulators and the goose `usage_update`
payload now carry `accumulatedCachedInputTokens`; `buzz-acp`
deserializes it (`serde` default `0` for goose, which doesn't send it)
and logs `cached=<n>`.
### 3. Fix a Databricks MLflow-route double-count (`llm.rs`)
The Databricks MLflow route reports the flat Anthropic-spelled
`cache_read_input_tokens` *alongside* an already-inclusive
`prompt_tokens`, so the old code summed them and nearly doubled the
count — inflating both the context-budget gate and cost.
`openai_chat_input_tokens()` now reads `prompt_tokens` alone. Verified
on a live `databricks-glm-5-2` response where `prompt_tokens +
completion == total` proves inclusivity. (Anthropic's native route
genuinely *excludes* the cache fields and is still summed — the two
never collide, because `claude*` models route to the Anthropic path.)
### 4. Proxy + TLS-trust passthrough into MCP tools (`mcp.rs`) —
independent fix
`buzz-agent` `env_clear()`s each MCP child, and the allowlist carried no
proxy/TLS vars. On a proxy-only host that doesn't degrade the tools, it
**blinds** them: apt, curl, pip, git connect directly, the egress
firewall resets the socket, and the agent reports "Connection reset by
peer" — indistinguishable from a genuinely offline task. Adds both
spellings of `HTTP(S)_PROXY`/`NO_PROXY`/`ALL_PROXY` (curl/git read
lowercase; Go/Python read uppercase; libcurl ignores uppercase
`HTTP_PROXY`) plus `SSL_CERT_FILE`/`SSL_CERT_DIR` for TLS-terminating
proxies that present their own CA.
## Testing
- `cargo fmt --all -- --check`, `cargo clippy -p buzz-agent -p buzz-acp
--all-targets -- -D warnings` — clean.
- `cargo test -p buzz-agent -p buzz-acp` — **all green** (632 + 299 lib
tests plus integration suites, 0 failures). New tests cover: the three
breakpoints and the disabled/empty-system/single-message edge cases;
nested-vs-flat cache parsing for all three routes; the Databricks
inclusive-`prompt_tokens` fix; wire deserialization of
`accumulatedCachedInputTokens`; and the proxy/TLS passthrough allowlist.
- Pre-push lefthook suite green (branch-skew, rust-tests, test,
desktop-check/test/tauri).
## Relationship to the benchmark branch
These are the non-`benchmarks/` changes from
`benchmark/harness-accounting-and-solo`, lifted onto a clean base off
`main` so they can merge independently.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Atish Patel <atish@squareup.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Problem
The Databricks model dropdown offers a handful of stale models — and
there's no way to tell that list apart from the real one. The AI Gateway
exposes **66** chat/embedding endpoints on `block-lakehouse-production`,
but the picker was showing a short list that includes models the gateway
no longer serves and embedding endpoints that can't chat at all.
Three independent defects, all on the discovery path:
**1. Live discovery never ran for agents with no saved provider.**
`get_agent_models` gates every in-process discovery attempt on the
provider (`is_openai_compatible_provider` / `is_anthropic_provider` /
`is_databricks_provider`), reading it straight from `record.provider`.
That field is `null` for every agent record created before provider
persistence — and for any agent that inherits its provider from the
build. So all three gates saw `None`, no HTTP discovery ran, and the
request fell through to the `buzz-acp models` subprocess. On the
Databricks path that subprocess returns `discovery_failure_fallback` —
the small hardcoded `DATABRICKS_V2_KNOWN_MODELS` catalog — which the
frontend renders exactly like a live catalog. An internal DMG that bakes
`BUZZ_AGENT_PROVIDER=databricks_v2` and a `DATABRICKS_HOST` still got
the fallback.
**2. The fallback list couldn't represent the running model.**
When discovery genuinely fails, the picker should at minimum be able to
show what the agent is actually configured with. For `DatabricksV2` it
couldn't: the fallback returned only the hardcoded slate, so a model
like `databricks-gpt-5-5` wasn't selectable in its own picker.
**3. Embedding endpoints were offered as chat models.**
`databricks-bge-large-en` was selectable (visible in the dialog today).
The v2 endpoints payload carries no `task` or `state` field, so there is
nothing to filter on but the name.
## Changes
- **`effective_discovery_provider`** (new,
`desktop/src-tauri/src/commands/agent_models_env.rs`) — an explicit
provider (saved record value, or the create/edit dialog's current form
value) still always wins; when there is none, discovery falls back to
the runtime's own provider env var (`GOOSE_PROVIDER`,
`BUZZ_AGENT_PROVIDER`, …) read off the merged env, which by that point
already carries the baked build floor and the process env. Wired into
both `get_agent_models` and `discover_agent_models`.
`SavedAgentModelDiscoveryConfig` now carries `provider_env_var` from
`known_acp_runtime`, so each runtime reads *its own* key rather than a
shared guess.
- The relay-mesh branches in `discover_agent_models` deliberately keep
using `input.provider`: those key off a deliberate provider selection,
never a baked default.
- **Asserted vs inferred matters for missing credentials.** The OpenAI
and Anthropic gates error on a missing API key, while the Databricks
gate falls through; an inferred provider hitting the first two would
have replaced a working subprocess catalog with `config:
ANTHROPIC_API_KEY required` (`export GOOSE_PROVIDER=anthropic` is
goose's documented way to pick a provider, and it keeps the key in its
own keyring). So `effective_discovery_provider` returns a
`DiscoveryProvider` that remembers how the value was resolved, and
`required_env` only reports a missing credential for an asserted
provider. A wrong guess declines and lets the subprocess answer.
- **`is_chat_capable_endpoint`** (new,
`crates/buzz-agent/src/catalog.rs`) — applied in
`parse_v2_endpoints_page`. Drops `*embedding*` and segment-matched `bge`
/ `gte` endpoints; keeps everything unrecognised (fail-open, so a new
model family is never hidden). Segment matching is why it's `split('-')`
and not `contains`: a substring check would swallow legitimate names.
- **`discovery_failure_fallback`** for `Provider::DatabricksV2` now
leads with the configured model (deduped against the known slate,
blank-tolerant), so a failed discovery still yields a picker that can
show the running model. The configured model is trimmed once up front —
`resolve_model` doesn't trim, so a padded `DATABRICKS_MODEL` used to
slip past the dedupe and appear twice.
- **`sort_v2_endpoints_newest_first`** (new, second commit) — the
catalog is now ordered newest-first on each endpoint's
`created_timestamp`, ties broken by name. Previously Buzz sorted
nothing, so the gateway's own order reached the picker: it pages in two
phases (Databricks-managed, then workspace-created — the page token
decodes to `{"phase":"user"}`), each alphabetical, which buried
`databricks-claude-opus-5` 8th behind five older Claude endpoints and
`goose-claude-opus-5` — the newest endpoint in the catalog — 55th of 63.
Sorting in `fetch_v2_models` means both discovery paths inherit it with
no wire or type changes, and the combobox filter preserves incoming
order. Endpoints with an absent or unparseable timestamp sort last
rather than first, so a wire-shape change degrades to "unordered at the
bottom" instead of "shuffled to the top".
- The name tiebreak is load-bearing: eleven managed endpoints share one
placeholder timestamp (`1699610000000`), so without it their relative
order would vary between runs. That placeholder is also not always
accurate — a few genuinely recent endpoints
(`databricks-kimi-k2-7-code`, `databricks-llama-4-maverick`) land at the
bottom with the 2023 batch. The gateway offers nothing better to sort
on.
- Env/provider lookup helpers moved out of `agent_models.rs` into
`agent_models_env.rs`. This keeps the command module under the file-size
limit **without ratcheting the override up** — the existing 1079 entry
is untouched (file is now 1066 lines).
## Verification
Live against `block-lakehouse-production`, release build:
```
BUZZ_ACP_AGENT_COMMAND=$PWD/target/release/buzz-agent \
BUZZ_AGENT_PROVIDER=databricks_v2 \
DATABRICKS_HOST=https://block-lakehouse-production.cloud.databricks.com \
DATABRICKS_MODEL=databricks-gpt-5-5 \
./target/release/buzz-acp models --json
```
- before: 66 endpoints, including `databricks-bge-large-en`,
`databricks-gte-large-en`, `databricks-qwen3-embedding-0-6b`
- after: **63** endpoints, `[.models[] | select(.id |
test("embedding|-bge-|-gte-"))]` → `[]`
Top of the list after the sort commit:
```
goose-claude-opus-5 2026-07-24
databricks-claude-opus-5 2026-07-23
databricks-gemini-3-6-flash 2026-07-20
databricks-gemini-3-5-flash-lite 2026-07-20
databricks-inkling 2026-07-14
```
Tests: 15 new (8 in `catalog.rs` — including the two-wire-shape
timestamp parse, the sort's tiebreak/no-timestamp cases, and the
padded-model dedupe — and 7 plus one assertion in
`agent_models_tests.rs`, 3 of them covering the asserted/inferred
credential split), two existing tests updated. `just check`, `just
test-unit`, and `just desktop-tauri-test` all pass (1636 desktop-tauri
tests, 274 buzz-agent lib tests).
Not run locally: the Docker-backed integration suite (`just test`) —
this diff touches neither `buzz-relay`, `buzz-db`, nor `buzz-auth`.
## Follow-ups (deliberately out of scope)
Two inference-path defects found while investigating, both reproduced
live against the gateway and both independent of discovery:
1. **Gemini thought signatures are dropped.** The gateway returns a bare
`thoughtSignature` on tool calls; the external-model serving endpoints
return it nested as `extra_content.google.thought_signature`. Neither
shape is round-tripped, so multi-turn tool use on `databricks-gemini-*`
fails with a 400 on the second turn.
2. **Array-shaped `content` is silently discarded.** Some models return
OpenAI `content` as a block array rather than a string; `parse_openai`'s
`str_field` returns `None` and the text is dropped.
The legacy `serving-endpoints` path does not work around either one, and
costs reasoning support on the GPT-5 family.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
