mirror of
https://github.com/block/buzz.git
synced 2026-08-18 06:50:31 +02:00
8eb6e3eb60
## Problem
The Databricks model dropdown offers a handful of stale models — and
there's no way to tell that list apart from the real one. The AI Gateway
exposes **66** chat/embedding endpoints on `block-lakehouse-production`,
but the picker was showing a short list that includes models the gateway
no longer serves and embedding endpoints that can't chat at all.
Three independent defects, all on the discovery path:
**1. Live discovery never ran for agents with no saved provider.**
`get_agent_models` gates every in-process discovery attempt on the
provider (`is_openai_compatible_provider` / `is_anthropic_provider` /
`is_databricks_provider`), reading it straight from `record.provider`.
That field is `null` for every agent record created before provider
persistence — and for any agent that inherits its provider from the
build. So all three gates saw `None`, no HTTP discovery ran, and the
request fell through to the `buzz-acp models` subprocess. On the
Databricks path that subprocess returns `discovery_failure_fallback` —
the small hardcoded `DATABRICKS_V2_KNOWN_MODELS` catalog — which the
frontend renders exactly like a live catalog. An internal DMG that bakes
`BUZZ_AGENT_PROVIDER=databricks_v2` and a `DATABRICKS_HOST` still got
the fallback.
**2. The fallback list couldn't represent the running model.**
When discovery genuinely fails, the picker should at minimum be able to
show what the agent is actually configured with. For `DatabricksV2` it
couldn't: the fallback returned only the hardcoded slate, so a model
like `databricks-gpt-5-5` wasn't selectable in its own picker.
**3. Embedding endpoints were offered as chat models.**
`databricks-bge-large-en` was selectable (visible in the dialog today).
The v2 endpoints payload carries no `task` or `state` field, so there is
nothing to filter on but the name.
## Changes
- **`effective_discovery_provider`** (new,
`desktop/src-tauri/src/commands/agent_models_env.rs`) — an explicit
provider (saved record value, or the create/edit dialog's current form
value) still always wins; when there is none, discovery falls back to
the runtime's own provider env var (`GOOSE_PROVIDER`,
`BUZZ_AGENT_PROVIDER`, …) read off the merged env, which by that point
already carries the baked build floor and the process env. Wired into
both `get_agent_models` and `discover_agent_models`.
`SavedAgentModelDiscoveryConfig` now carries `provider_env_var` from
`known_acp_runtime`, so each runtime reads *its own* key rather than a
shared guess.
- The relay-mesh branches in `discover_agent_models` deliberately keep
using `input.provider`: those key off a deliberate provider selection,
never a baked default.
- **Asserted vs inferred matters for missing credentials.** The OpenAI
and Anthropic gates error on a missing API key, while the Databricks
gate falls through; an inferred provider hitting the first two would
have replaced a working subprocess catalog with `config:
ANTHROPIC_API_KEY required` (`export GOOSE_PROVIDER=anthropic` is
goose's documented way to pick a provider, and it keeps the key in its
own keyring). So `effective_discovery_provider` returns a
`DiscoveryProvider` that remembers how the value was resolved, and
`required_env` only reports a missing credential for an asserted
provider. A wrong guess declines and lets the subprocess answer.
- **`is_chat_capable_endpoint`** (new,
`crates/buzz-agent/src/catalog.rs`) — applied in
`parse_v2_endpoints_page`. Drops `*embedding*` and segment-matched `bge`
/ `gte` endpoints; keeps everything unrecognised (fail-open, so a new
model family is never hidden). Segment matching is why it's `split('-')`
and not `contains`: a substring check would swallow legitimate names.
- **`discovery_failure_fallback`** for `Provider::DatabricksV2` now
leads with the configured model (deduped against the known slate,
blank-tolerant), so a failed discovery still yields a picker that can
show the running model. The configured model is trimmed once up front —
`resolve_model` doesn't trim, so a padded `DATABRICKS_MODEL` used to
slip past the dedupe and appear twice.
- **`sort_v2_endpoints_newest_first`** (new, second commit) — the
catalog is now ordered newest-first on each endpoint's
`created_timestamp`, ties broken by name. Previously Buzz sorted
nothing, so the gateway's own order reached the picker: it pages in two
phases (Databricks-managed, then workspace-created — the page token
decodes to `{"phase":"user"}`), each alphabetical, which buried
`databricks-claude-opus-5` 8th behind five older Claude endpoints and
`goose-claude-opus-5` — the newest endpoint in the catalog — 55th of 63.
Sorting in `fetch_v2_models` means both discovery paths inherit it with
no wire or type changes, and the combobox filter preserves incoming
order. Endpoints with an absent or unparseable timestamp sort last
rather than first, so a wire-shape change degrades to "unordered at the
bottom" instead of "shuffled to the top".
- The name tiebreak is load-bearing: eleven managed endpoints share one
placeholder timestamp (`1699610000000`), so without it their relative
order would vary between runs. That placeholder is also not always
accurate — a few genuinely recent endpoints
(`databricks-kimi-k2-7-code`, `databricks-llama-4-maverick`) land at the
bottom with the 2023 batch. The gateway offers nothing better to sort
on.
- Env/provider lookup helpers moved out of `agent_models.rs` into
`agent_models_env.rs`. This keeps the command module under the file-size
limit **without ratcheting the override up** — the existing 1079 entry
is untouched (file is now 1066 lines).
## Verification
Live against `block-lakehouse-production`, release build:
```
BUZZ_ACP_AGENT_COMMAND=$PWD/target/release/buzz-agent \
BUZZ_AGENT_PROVIDER=databricks_v2 \
DATABRICKS_HOST=https://block-lakehouse-production.cloud.databricks.com \
DATABRICKS_MODEL=databricks-gpt-5-5 \
./target/release/buzz-acp models --json
```
- before: 66 endpoints, including `databricks-bge-large-en`,
`databricks-gte-large-en`, `databricks-qwen3-embedding-0-6b`
- after: **63** endpoints, `[.models[] | select(.id |
test("embedding|-bge-|-gte-"))]` → `[]`
Top of the list after the sort commit:
```
goose-claude-opus-5 2026-07-24
databricks-claude-opus-5 2026-07-23
databricks-gemini-3-6-flash 2026-07-20
databricks-gemini-3-5-flash-lite 2026-07-20
databricks-inkling 2026-07-14
```
Tests: 15 new (8 in `catalog.rs` — including the two-wire-shape
timestamp parse, the sort's tiebreak/no-timestamp cases, and the
padded-model dedupe — and 7 plus one assertion in
`agent_models_tests.rs`, 3 of them covering the asserted/inferred
credential split), two existing tests updated. `just check`, `just
test-unit`, and `just desktop-tauri-test` all pass (1636 desktop-tauri
tests, 274 buzz-agent lib tests).
Not run locally: the Docker-backed integration suite (`just test`) —
this diff touches neither `buzz-relay`, `buzz-db`, nor `buzz-auth`.
## Follow-ups (deliberately out of scope)
Two inference-path defects found while investigating, both reproduced
live against the gateway and both independent of discovery:
1. **Gemini thought signatures are dropped.** The gateway returns a bare
`thoughtSignature` on tool calls; the external-model serving endpoints
return it nested as `extra_content.google.thought_signature`. Neither
shape is round-tripped, so multi-turn tool use on `databricks-gemini-*`
fails with a 400 on the second turn.
2. **Array-shaped `content` is silently discarded.** Some models return
OpenAI `content` as a block array rather than a string; `parse_openai`'s
`str_field` returns `None` and the text is dropped.
The legacy `serving-endpoints` path does not work around either one, and
costs reasoning support on the GPT-5 family.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>