mirror of
https://github.com/block/buzz.git
synced 2026-08-18 06:50:31 +02:00
4d47f48143
## What this does Introduces the model-capability manifest infrastructure (Phase 1 of the Model-Capability Manifest plan v4, Thufir-approved 9/9/9). No consumer cutover — `config.rs`, `llm.rs`, `catalog.rs`, and `buzzAgentConfig.ts` are unchanged. Phase 2 wires them. **Single source of truth** replaces hand-mirrored metadata across four files: ``` scripts/model-capabilities.json → hand-curated manifest scripts/generate-model-capabilities.mjs → emits Rust + TS artifacts crates/buzz-agent/src/generated_model_capabilities.rs desktop/src/features/agents/ui/modelCapabilities.ts ``` ## Resolver contract (plan v4 §Resolver contract) Total function `resolve(provider, raw_model_id) → CapabilityResult`. Three ordered steps: 1. Provider-qualified raw exact lookup — key is `(provider, raw_model_id)`, matched before any prefix stripping. A prefixed alias never inherits an exact record. 2. Provider-scoped ordered family rules — on normalized (prefix-stripped) alias, by `match_priority` desc. 3. Per-axis provider fallback — `blank` vs `concrete_unknown`, per provider. Result is complete — every axis populated, runtime consumers never compose fields. ## Boundaries the manifest does NOT own - Transport for pure OpenAI, legacy Databricks, OpenRouter: `OpenAiApi`/`openai_request()` remain authoritative. `databricks_v2_wire_route` is DBv2-only (all other providers emit `not-applicable`). - Final display labels: `resolveModelLabel()` three-tier precedence unchanged. `registry_label` feeds only the static registry tier. - `llm.rs` scope: only `databricks_v2_route_for_model` (Phase 2). ## Test oracle (three independent layers) 1. Generated full-table coverage — `scripts/generated-model-capabilities-coverage.json`: every manifest entry + provider fallbacks. 2. Hand-authored normative corpus — `scripts/normative-corpus.json` (44 vectors): Anthropic manual-budget/adaptive families, OpenAI gpt-5 adversarial boundary cases, DBv2 segment-routing collision tests, P2-A resolver-contract vectors, P2-B blank/concrete-unknown per provider. Runs against JS resolver (`run-corpus.mjs`) and mirrored in Rust (`generated_model_capabilities_tests.rs`). 3. Schema-negative tests — `scripts/test-manifest-validator.mjs` (17 tests): every validator rule has a failing-input test. ## Reconciliation table `scripts/MODELS_DEV_RECONCILIATION.md` — all models.dev divergences dispositioned. `databricks-gpt-5-4-mini` and `databricks-gpt-5-4-nano` adopt models.dev `[low,medium,high]` (family rule adds `none+xhigh` the endpoint doesn't advertise). ## CI `.github/workflows/model-capability-regen-diff.yml`: triggers on manifest/generator/artifact changes; regenerates and fails if stale; runs JS corpus + schema-negative tests. ## Acceptance criteria (plan v4 Phase 1) - Byte-clean regen: `node scripts/generate-model-capabilities.mjs --check` passes - Rust compiles: `cargo check -p buzz-agent` - TS typechecks: `pnpm tsc --noEmit --strict` - 44/44 normative corpus vectors pass (JS interpreter) - 41/41 Rust corpus tests pass - 17/17 schema-negative tests pass (every validator rule) - Reconciliation table complete with doc citations - No consumer changes (config.rs, llm.rs, catalog.rs, buzzAgentConfig.ts untouched) --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
135 lines
4.0 KiB
JSON
135 lines
4.0 KiB
JSON
{
|
|
"_comment": "Verbatim models.dev snapshot for differential harness (plan v4 §Oracle). Contains exact records captured from the live API for exact-override entries. Verbatim: name and reasoning_options are reproduced without transformation.",
|
|
"_source_url": "https://models.dev/api.json",
|
|
"_retrieval_date": "2026-07-31",
|
|
"_payload_sha256": "d5a4974cd69f19b0f67713acaa6bb3b16e920defdc07ecbdf6b0a936181bb0e0",
|
|
"_retrieval_note": "Full payload SHA-256 computed over the raw response body of GET https://models.dev/api.json (no transforms). Re-verify: curl -s https://models.dev/api.json | sha256sum",
|
|
"_models_dev_records": {
|
|
"databricks-gpt-5-5": {
|
|
"id": "databricks-gpt-5-5",
|
|
"name": "GPT-5.5",
|
|
"reasoning_options": [
|
|
{
|
|
"type": "effort",
|
|
"values": [
|
|
"low",
|
|
"medium",
|
|
"high"
|
|
]
|
|
}
|
|
]
|
|
},
|
|
"databricks-gpt-5-4-mini": {
|
|
"id": "databricks-gpt-5-4-mini",
|
|
"name": "GPT-5.4 mini",
|
|
"reasoning_options": [
|
|
{
|
|
"type": "effort",
|
|
"values": [
|
|
"low",
|
|
"medium",
|
|
"high"
|
|
]
|
|
}
|
|
]
|
|
},
|
|
"databricks-gpt-5-4-nano": {
|
|
"id": "databricks-gpt-5-4-nano",
|
|
"name": "GPT-5.4 nano",
|
|
"reasoning_options": [
|
|
{
|
|
"type": "effort",
|
|
"values": [
|
|
"low",
|
|
"medium",
|
|
"high"
|
|
]
|
|
}
|
|
]
|
|
},
|
|
"databricks-gpt-5-6-sol": {
|
|
"id": "databricks-gpt-5-6-sol",
|
|
"name": "GPT-5.6 Sol",
|
|
"reasoning_options": [
|
|
{
|
|
"type": "effort",
|
|
"values": [
|
|
"low",
|
|
"medium",
|
|
"high",
|
|
"max"
|
|
]
|
|
}
|
|
]
|
|
},
|
|
"databricks-claude-opus-4-7": {
|
|
"id": "databricks-claude-opus-4-7",
|
|
"name": "Claude Opus 4.7",
|
|
"reasoning_options": [
|
|
{
|
|
"type": "budget_tokens",
|
|
"min": 1024
|
|
}
|
|
]
|
|
}
|
|
},
|
|
"endpoints": [
|
|
{
|
|
"name": "databricks-gpt-5-5",
|
|
"note": "DATABRICKS_V2_KNOWN_MODELS entry; gpt5-5 family; openai-responses route"
|
|
},
|
|
{
|
|
"name": "databricks-gpt-5-4-mini",
|
|
"note": "exact record; models.dev override: low|medium|high (not family rule none+xhigh)"
|
|
},
|
|
{
|
|
"name": "databricks-gpt-5-4-nano",
|
|
"note": "exact record; models.dev override: low|medium|high"
|
|
},
|
|
{
|
|
"name": "databricks-gpt-5-6-sol",
|
|
"note": "exact record; models.dev source: low|medium|high|max (adopted as-is)"
|
|
},
|
|
{
|
|
"name": "databricks-claude-opus-4-7",
|
|
"note": "DATABRICKS_V2_KNOWN_MODELS entry; anthropic adaptive xhigh-capable; anthropic-messages route"
|
|
},
|
|
{
|
|
"name": "goose-claude-fable-5",
|
|
"note": "goose- prefix stripped; claude-fable-5 → anthropic adaptive xhigh-capable; anthropic-messages"
|
|
},
|
|
{
|
|
"name": "goose-claude-sonnet-5-20260101",
|
|
"note": "goose- prefix stripped; claude-sonnet-5 family; anthropic adaptive xhigh-capable"
|
|
},
|
|
{
|
|
"name": "goose-opus-5",
|
|
"note": "'opus' segment → anthropic-messages route; effort: fallback (prefix-stripped alias 'opus-5' not recognized Claude family)"
|
|
},
|
|
{
|
|
"name": "consolidated-llama",
|
|
"note": "segment test: 'sol' is substring of 'consolidated', NOT a segment → mlflow-chat"
|
|
},
|
|
{
|
|
"name": "terraform-coder",
|
|
"note": "segment test: 'terra' is prefix of 'terraform', NOT a segment → mlflow-chat"
|
|
},
|
|
{
|
|
"name": "corpus-reranker",
|
|
"note": "segment test: 'opus' is NOT a segment of 'corpus-reranker' → mlflow-chat"
|
|
},
|
|
{
|
|
"name": "octopus-model",
|
|
"note": "segment test: 'opus' is NOT a segment of 'octopus-model' → mlflow-chat"
|
|
},
|
|
{
|
|
"name": "llama-3-70b",
|
|
"note": "concrete non-Claude non-GPT → mlflow-chat; effort: all-except-max"
|
|
},
|
|
{
|
|
"name": "",
|
|
"note": "blank model → route-unknown; all 7 efforts; default medium"
|
|
}
|
|
]
|
|
}
|