mirror of
https://github.com/block/buzz.git
synced 2026-08-18 06:50:31 +02:00
feat(agent): route Claude/GPT model families to their native gateway wire (#3538)
## Summary Databricks v2 chooses the gateway wire format — OpenAI Responses, Anthropic Messages, or MLflow chat — purely from substrings in the endpoint name. There is no family field on the endpoint to key off, so the substring set *is* the routing contract. The matcher only recognised `gpt-5`/`gpt5` and `claude`, which makes correct billing depend on every Claude endpoint happening to be named with the literal string "claude". ## Why this matters Getting a Claude model onto the Anthropic Messages route is exactly what lets buzz attach the `cache_control` breakpoint (the fix in #3463). If a Claude endpoint's catalog name omits "claude" — an alias, a bare `opus-5`, a `goose-opus-5` — it silently falls through to the MLflow (OpenAI-wire) path, where Anthropic prompt caching is **structurally impossible**. The result is the same failure #3463 fixed: 0% cache reads, the full ~10x read discount lost, and no error — a naming convention quietly holding up a billing-correctness invariant. ## What changed `databricks_v2_route_for_model` (`crates/buzz-agent/src/llm.rs`) now matches broader, case-insensitive marker sets: - **Claude → Anthropic Messages:** `claude`, `opus`, `sonnet`, `haiku`, `mythos`, `fable` — the Claude family names and release code names, so a Claude endpoint reaches the cache-capable route regardless of how it's named. - **GPT → OpenAI Responses:** the `gpt` family (now `gpt` on its own, not just `gpt-5`) plus the GPT-5 launch code names `sol`, `luna`, `terra`. OpenAI markers are evaluated first, preserving the prior `gpt-5`-first precedence for any name that could carry both. Names matching neither set still fall through to the MLflow chat route. ## Testing - `cargo fmt`, `cargo clippy -p buzz-agent --all-targets -- -D warnings` — clean. - `cargo test -p buzz-agent` — all green (299 lib + integration suites, 0 failures). The `databricks_v2_routes_by_model_family` test was expanded to cover each new marker, the GPT-5 code names, case-insensitivity, and the unchanged MLflow fallback (including `gemini`). ## Relationship to #3463 #3463 taught the Anthropic path to request caching; this makes sure Claude models actually land on that path. Follow-up still open: surfacing `cache_creation_input_tokens` end-to-end so a persistent `reads == 0 && writes == 0` reveals a disabled cache regardless of which wire a model takes — happy to do that next. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Atish Patel <atish@squareup.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Atish Patel
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
24d90d1280
commit
6438dedf83
@@ -1019,13 +1019,53 @@ fn is_responses_required_error(body: &str) -> bool {
|
||||
|| b.contains("use the responses api")
|
||||
}
|
||||
|
||||
/// OpenAI-family code names that appear as their own segment in a Databricks v2
|
||||
/// endpoint name (the GPT-5 launch aliases). The `gpt` family itself is matched
|
||||
/// separately by segment prefix so `gpt`, `gpt5`, and the `gpt` of a split
|
||||
/// `gpt-5` all qualify.
|
||||
const DATABRICKS_V2_OPENAI_CODE_NAMES: &[&str] = &["sol", "luna", "terra"];
|
||||
|
||||
/// Anthropic (Claude) family and release code names that appear as their own
|
||||
/// segment in a Databricks v2 endpoint name — the `claude` prefix, the family
|
||||
/// names (`opus`, `sonnet`, `haiku`), and the release code names (`mythos`,
|
||||
/// `fable`). Getting a Claude model onto the Anthropic Messages route is what
|
||||
/// lets it carry a `cache_control` breakpoint; an endpoint that matches none of
|
||||
/// these falls through to the MLflow (OpenAI-wire) path, where Anthropic prompt
|
||||
/// caching is structurally impossible and the discount is silently lost.
|
||||
const DATABRICKS_V2_CLAUDE_NAMES: &[&str] =
|
||||
&["claude", "opus", "sonnet", "haiku", "mythos", "fable"];
|
||||
|
||||
/// Split a Databricks v2 endpoint name into its lowercase alphanumeric segments,
|
||||
/// breaking on any non-alphanumeric delimiter (`-`, `_`, `.`, `/`, …). E.g.
|
||||
/// `Databricks-Claude-Opus-5` -> `["databricks", "claude", "opus", "5"]`.
|
||||
fn model_name_segments(model: &str) -> Vec<String> {
|
||||
model
|
||||
.split(|c: char| !c.is_ascii_alphanumeric())
|
||||
.filter(|s| !s.is_empty())
|
||||
.map(str::to_ascii_lowercase)
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn databricks_v2_route_for_model(model: &str) -> DatabricksV2Route {
|
||||
// Databricks v2 catalog names currently identify OpenAI-shaped GPT-5
|
||||
// models and Anthropic-shaped Claude models by these substrings.
|
||||
let lower = model.to_ascii_lowercase();
|
||||
if lower.contains("gpt-5") || lower.contains("gpt5") {
|
||||
// The v2 catalog exposes no family field, so the wire format is inferred
|
||||
// from the endpoint name. Discovery deliberately keeps arbitrary custom
|
||||
// aliases, so we match whole name *segments* rather than raw substrings: a
|
||||
// substring test would misroute unrelated names — `consolidated-llama`
|
||||
// (`sol`), `terraform-coder` (`terra`), `corpus-reranker`/`octopus-model`
|
||||
// (`opus`) — onto a wire whose request shape their backend can't parse,
|
||||
// turning a caching optimization into a hard request/parse failure. Segment
|
||||
// matching still accepts real prefixed names like `goose-opus-5`.
|
||||
let segments = model_name_segments(model);
|
||||
let has_named_segment =
|
||||
|names: &[&str]| segments.iter().any(|seg| names.contains(&seg.as_str()));
|
||||
// `gpt` family: any segment beginning with `gpt` — covers `gpt`, `gpt5`, and
|
||||
// the `gpt` segment of a split `gpt-5`, without matching mid-word.
|
||||
let is_gpt_family = segments.iter().any(|seg| seg.starts_with("gpt"));
|
||||
// OpenAI is checked before Claude so a name carrying both markers resolves
|
||||
// to the OpenAI wire (preserving the prior `gpt-5`-first precedence).
|
||||
if is_gpt_family || has_named_segment(DATABRICKS_V2_OPENAI_CODE_NAMES) {
|
||||
DatabricksV2Route::OpenAiResponses
|
||||
} else if lower.contains("claude") {
|
||||
} else if has_named_segment(DATABRICKS_V2_CLAUDE_NAMES) {
|
||||
DatabricksV2Route::AnthropicMessages
|
||||
} else {
|
||||
DatabricksV2Route::MlflowChatCompletions
|
||||
@@ -2579,20 +2619,105 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn databricks_v2_routes_by_model_family() {
|
||||
use DatabricksV2Route::{AnthropicMessages, MlflowChatCompletions, OpenAiResponses};
|
||||
for (model, route, path) in [
|
||||
// OpenAI-shaped: the gpt family plus the GPT-5 code names.
|
||||
(
|
||||
"databricks-gpt-5-5",
|
||||
DatabricksV2Route::OpenAiResponses,
|
||||
OpenAiResponses,
|
||||
"/ai-gateway/openai/v1/responses",
|
||||
),
|
||||
("gpt-4o", OpenAiResponses, "/ai-gateway/openai/v1/responses"),
|
||||
// The intentional dashless `gpt5` spelling still routes to OpenAI.
|
||||
("gpt5", OpenAiResponses, "/ai-gateway/openai/v1/responses"),
|
||||
(
|
||||
"databricks-gpt-5-6-luna",
|
||||
OpenAiResponses,
|
||||
"/ai-gateway/openai/v1/responses",
|
||||
),
|
||||
(
|
||||
"databricks-gpt-5-6-sol",
|
||||
OpenAiResponses,
|
||||
"/ai-gateway/openai/v1/responses",
|
||||
),
|
||||
(
|
||||
"databricks-terra",
|
||||
OpenAiResponses,
|
||||
"/ai-gateway/openai/v1/responses",
|
||||
),
|
||||
// Anthropic-shaped: the claude prefix, the family names, and the
|
||||
// release code names — each must reach the cache-capable route even
|
||||
// when the endpoint name omits the literal "claude".
|
||||
(
|
||||
"databricks-claude-opus-4-7",
|
||||
DatabricksV2Route::AnthropicMessages,
|
||||
AnthropicMessages,
|
||||
"/ai-gateway/anthropic/v1/messages",
|
||||
),
|
||||
(
|
||||
"goose-opus-5",
|
||||
AnthropicMessages,
|
||||
"/ai-gateway/anthropic/v1/messages",
|
||||
),
|
||||
(
|
||||
"databricks-sonnet-5",
|
||||
AnthropicMessages,
|
||||
"/ai-gateway/anthropic/v1/messages",
|
||||
),
|
||||
(
|
||||
"databricks-haiku-4-5",
|
||||
AnthropicMessages,
|
||||
"/ai-gateway/anthropic/v1/messages",
|
||||
),
|
||||
(
|
||||
"databricks-mythos-5",
|
||||
AnthropicMessages,
|
||||
"/ai-gateway/anthropic/v1/messages",
|
||||
),
|
||||
(
|
||||
"databricks-fable-5",
|
||||
AnthropicMessages,
|
||||
"/ai-gateway/anthropic/v1/messages",
|
||||
),
|
||||
// Case-insensitive.
|
||||
(
|
||||
"Databricks-Claude-Opus-5",
|
||||
AnthropicMessages,
|
||||
"/ai-gateway/anthropic/v1/messages",
|
||||
),
|
||||
// Unrecognised names still fall through to the MLflow chat route.
|
||||
(
|
||||
"custom-tool-model",
|
||||
DatabricksV2Route::MlflowChatCompletions,
|
||||
MlflowChatCompletions,
|
||||
"/ai-gateway/mlflow/v1/chat/completions",
|
||||
),
|
||||
(
|
||||
"databricks-gemini-3-pro",
|
||||
MlflowChatCompletions,
|
||||
"/ai-gateway/mlflow/v1/chat/completions",
|
||||
),
|
||||
// Collision guard: short code names must match only as whole
|
||||
// segments, never as substrings of an unrelated custom alias.
|
||||
// Each of these embeds a marker (`sol`, `terra`, `opus`) mid-word
|
||||
// and must stay on the MLflow fallback, not adopt a wire its
|
||||
// backend can't parse.
|
||||
(
|
||||
"consolidated-llama",
|
||||
MlflowChatCompletions,
|
||||
"/ai-gateway/mlflow/v1/chat/completions",
|
||||
),
|
||||
(
|
||||
"terraform-coder",
|
||||
MlflowChatCompletions,
|
||||
"/ai-gateway/mlflow/v1/chat/completions",
|
||||
),
|
||||
(
|
||||
"corpus-reranker",
|
||||
MlflowChatCompletions,
|
||||
"/ai-gateway/mlflow/v1/chat/completions",
|
||||
),
|
||||
(
|
||||
"octopus-model",
|
||||
MlflowChatCompletions,
|
||||
"/ai-gateway/mlflow/v1/chat/completions",
|
||||
),
|
||||
] {
|
||||
|
||||
Reference in New Issue
Block a user