mirror of
https://github.com/block/buzz.git
synced 2026-08-18 06:50:31 +02:00
hayt/usage-v2-pricing-axis
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 npub1g8493u0xfsjrvflg4n08ezd7vec99mnwzlv0qgwpr9d7gvjwhuzqx59rhwandWill Pfleger
|
4855d95252 |
feat(models): Phase 3 — pricing axis to D2' (authority, model) exact-lookup contract
Replaces lookup_pricing(raw_model_id) + normalize_for_pricing + 17 prefix rules
with an exact (billing-authority domain, billable-model-string) lookup. Billing
authority is a distinct namespace from the runtime transport Provider enum.
Match is exact: no case normalization, no prefix matching, no inference.
Manifest (scripts/model-capabilities.json):
- Removed pricing_rules (17 prefix-match rules), pricing_exact_records (3 records)
- Added flat pricing_records list of 20 exact records, each with authority + model +
usd_per_mtok (input/output/cache_read/cache_write) + _source provenance
- Authorities: api.anthropic.com, api.openai.com — registered bare-host tokens per
D2'/NIP-AM; not transport Provider enum values
- Semantics: current list prices (ccusage-aligned); cache_write scoped to ephemeral;
other classes unknown
Generator (scripts/generate-model-capabilities.mjs):
- Removed all old pricing machinery (pricingNormalize, normalizeVersionSeparators,
pricingRules, pricingExactRecords, resolvePricing, all TS rule-resolver emission)
- Strict pricing-record validation: registered authority (closed set: api.anthropic.com,
api.openai.com, openrouter.ai); nonempty verbatim model; no control/NUL/quote/backslash
in key fields; finite nonnegative input/output; both cache_read and cache_write required
(absent member is hard error; use null for unpublished); _source provenance required;
duplicate detection on exact emitted identities
- byAuthority grouping uses exact record values — no toLowerCase() anywhere in
pricing generation path
- REGISTERED_AUTHORITIES set; hasUnsafeKeyChars() guard
- TS PRICING_TABLE: key is ${authority}\0${model} (exact strings, no transforms)
- lookupModelPricing(authority, model) performs exact Map lookup (no normalization)
- modelPricing.ts is standalone (no import from modelCapabilities)
Generated artifacts:
- generated_model_capabilities.rs: lookup_pricing(authority: &str, model: &str)
performs exact match — no to_ascii_lowercase() calls, no fallback
- modelPricing.ts: exact PRICING_TABLE + lookupModelPricing(authority, model)
CI (.github/workflows/ci.yml):
- Replaced two-step "regenerate + git diff" check with a single
node scripts/generate-model-capabilities.mjs --check step, which validates
all three generated artifacts (Rust capabilities, TS capabilities, TS pricing)
and exits 1 if any are stale. Self-maintains if a fourth artifact is added.
Tests (generated_model_capabilities_tests.rs, 23 pricing tests):
- Anthropic models with correct rates (claude-fable-5, claude-opus-5, claude-sonnet-5,
claude-opus-4-8, claude-sonnet-4-6, claude-haiku-4-5)
- OpenAI models incl. gpt-5.6-luna different tier, gpt-5.5 null cache_write, gpt-5-pro
null cache fields
- Null guards: unknown authority, custom base URL, unknown model, empty authority,
empty model, both empty, wrong authority for known model
- Exact-match guards: uppercase authority -> None, mixed-case authority -> None,
mixed-case model -> None (no case folding)
- Databricks guards: workspace URL authority -> None, 'databricks' authority -> None
Manifest validator tests (scripts/test-manifest-validator.mjs, 45 tests total):
- 21 new pricing-record mutation tests covering: missing authority, unregistered/path-
bearing/scheme authority, empty model, unsafe chars (double-quote, backslash, NUL,
control), missing output, negative rates, invalid cache rate, deleted cache_read,
deleted cache_write, missing/empty provenance, duplicate records, uppercase authority
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
|
||
|
|
cc00060ea3 |
feat(agent): Phase 2 — wire Rust and TS consumers to generated model-capabilities module (#3958)
## What Phase 2 consumer cutover targeting the `duncan/databricks-model-label-registry` umbrella branch. Wires `crates/**` and `desktop/**` consumers to the generated capability module introduced in Phase 1 (#3821), while keeping old and new paths both live for differential testing. Phase 3 removes the old paths. ## Commits (boundary-separated) ### feat(agent): Phase 2a — wire Rust consumers to generated capability module (`crates/**`, `scripts/**`) - `catalog.rs`: `DATABRICKS_V2_KNOWN_MODELS` re-exported from the generated module — single source of truth. - `llm.rs`: `databricks_v2_route_for_model` delegates to `resolve_model_capabilities("databricks_v2", model)`. Old segment-based classifier preserved as `#[cfg(test)] _old_*` for the differential harness. New `databricks_v2_route_differential_old_vs_new` test confirms 100% agreement on all 20 route vectors. - `config.rs`: new `effort_table_fixture_differential_old_vs_new` test runs `resolve_model_capabilities` over the 36-entry `effortTable.fixture.json` and asserts old/new agree modulo a doc-cited allowlist (4 F1 corrections). - `scripts/run-differential.mjs`: JS differential harness over effortTable fixture + normative corpus + catalog-sample fixture. 85 checks, 0 unexpected divergences (5 allowlisted: 4 F1 corrections + goose-opus-5 anthropic route correction). - `scripts/MODELS_DEV_RECONCILIATION.md`: deferred MINOR from Phase 1 — 8 trailing-double-space line breaks replaced with `<br>`. ### feat(desktop): Phase 2b — cut TS consumers to generated model-capabilities module (`desktop/**`) - `buzzAgentConfig.ts`: adds `getProviderEffortConfigFromManifest(provider, model?)` — thin wrapper over `resolveModelCapabilities()` from `modelCapabilities.ts`. Maps `supportedEfforts → validValues` and `defaultEffort → defaultValue` (null preserved for manual-budget/Inherit). Old `getProviderEffortConfig()` and all hand-tables stay live for the differential harness; Phase 3 retires them. - `formatAgentModelLabel.ts`: registry-label lookup re-pointed from hand-maintained `databricksModelNames.ts` import to generated `DATABRICKS_MODEL_NAMES` exported from `modelCapabilities.ts`. Same Map shape, identical contents, behavior unchanged. ### fix(scripts): add ts-esm-loader and fix allowlist coverage in run-differential (`scripts/**`) - `scripts/ts-esm-loader.mjs`: minimal ESM custom loader that resolves extensionless relative TS imports. Required because Phase 2b's `buzzAgentConfig.ts` imports `modelCapabilities` without `.ts` extension — which Node's `--experimental-strip-types` runner cannot resolve without a hook. - `scripts/run-differential.mjs`: shebang updated to self-bootstrap with the loader; fixes the `totalAllowlisted` counter (was declared but never incremented — always printed `0 allowlisted`). Replaced with per-axis hit tracking: reports exercised slot count (`N/total`) in summary; fails with `STALE_ALLOWLIST` if any declared entry fires zero divergences, preventing stale entries from silently masking future regressions. ## Verification - `cargo test -p buzz-agent --lib`: 426/426 - Corpus: 45/45 · schema-negative: 24/24 · `--check` byte-clean - Differential: 85 checks, 0 unexpected divergences, 6/6 allowlist slots exercised - Desktop: 3847/3847 · typecheck clean · biome clean - Mobile: 1019 pass, 1 skipped — same 5 flaky tests in `mobile/test/features/channels/` that reproduce at the umbrella base; zero mobile files in this branch range - `git diff --check`: clean ## What Remains (Phase 3) Remove old hand-maintained paths: `_old_*` functions in `llm.rs`/`config.rs`, old `getProviderEffortConfig` tables in `buzzAgentConfig.ts`, old `databricksModelNames.ts` import in `formatAgentModelLabel.ts`, old `databricks_model_names.rs` module. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: npub1g8493u0xfsjrvflg4n08ezd7vec99mnwzlv0qgwpr9d7gvjwhuzqx59rhw <41ea58f1e64c243627e8acde7c89be667052ee6e17d8f021c1195be4324ebf04@buzz.block.builderlab.xyz> Signed-off-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: npub1g8493u0xfsjrvflg4n08ezd7vec99mnwzlv0qgwpr9d7gvjwhuzqx59rhw <41ea58f1e64c243627e8acde7c89be667052ee6e17d8f021c1195be4324ebf04@buzz.block.builderlab.xyz> |
||
|
|
4d47f48143 |
feat(agent): Phase 1 — model-capability manifest, generator, and test oracle (#3821)
## What this does Introduces the model-capability manifest infrastructure (Phase 1 of the Model-Capability Manifest plan v4, Thufir-approved 9/9/9). No consumer cutover — `config.rs`, `llm.rs`, `catalog.rs`, and `buzzAgentConfig.ts` are unchanged. Phase 2 wires them. **Single source of truth** replaces hand-mirrored metadata across four files: ``` scripts/model-capabilities.json → hand-curated manifest scripts/generate-model-capabilities.mjs → emits Rust + TS artifacts crates/buzz-agent/src/generated_model_capabilities.rs desktop/src/features/agents/ui/modelCapabilities.ts ``` ## Resolver contract (plan v4 §Resolver contract) Total function `resolve(provider, raw_model_id) → CapabilityResult`. Three ordered steps: 1. Provider-qualified raw exact lookup — key is `(provider, raw_model_id)`, matched before any prefix stripping. A prefixed alias never inherits an exact record. 2. Provider-scoped ordered family rules — on normalized (prefix-stripped) alias, by `match_priority` desc. 3. Per-axis provider fallback — `blank` vs `concrete_unknown`, per provider. Result is complete — every axis populated, runtime consumers never compose fields. ## Boundaries the manifest does NOT own - Transport for pure OpenAI, legacy Databricks, OpenRouter: `OpenAiApi`/`openai_request()` remain authoritative. `databricks_v2_wire_route` is DBv2-only (all other providers emit `not-applicable`). - Final display labels: `resolveModelLabel()` three-tier precedence unchanged. `registry_label` feeds only the static registry tier. - `llm.rs` scope: only `databricks_v2_route_for_model` (Phase 2). ## Test oracle (three independent layers) 1. Generated full-table coverage — `scripts/generated-model-capabilities-coverage.json`: every manifest entry + provider fallbacks. 2. Hand-authored normative corpus — `scripts/normative-corpus.json` (44 vectors): Anthropic manual-budget/adaptive families, OpenAI gpt-5 adversarial boundary cases, DBv2 segment-routing collision tests, P2-A resolver-contract vectors, P2-B blank/concrete-unknown per provider. Runs against JS resolver (`run-corpus.mjs`) and mirrored in Rust (`generated_model_capabilities_tests.rs`). 3. Schema-negative tests — `scripts/test-manifest-validator.mjs` (17 tests): every validator rule has a failing-input test. ## Reconciliation table `scripts/MODELS_DEV_RECONCILIATION.md` — all models.dev divergences dispositioned. `databricks-gpt-5-4-mini` and `databricks-gpt-5-4-nano` adopt models.dev `[low,medium,high]` (family rule adds `none+xhigh` the endpoint doesn't advertise). ## CI `.github/workflows/model-capability-regen-diff.yml`: triggers on manifest/generator/artifact changes; regenerates and fails if stale; runs JS corpus + schema-negative tests. ## Acceptance criteria (plan v4 Phase 1) - Byte-clean regen: `node scripts/generate-model-capabilities.mjs --check` passes - Rust compiles: `cargo check -p buzz-agent` - TS typechecks: `pnpm tsc --noEmit --strict` - 44/44 normative corpus vectors pass (JS interpreter) - 41/41 Rust corpus tests pass - 17/17 schema-negative tests pass (every validator rule) - Reconciliation table complete with doc citations - No consumer changes (config.rs, llm.rs, catalog.rs, buzzAgentConfig.ts untouched) --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> |