mirror of https://github.com/block/buzz.git synced 2026-08-18 06:50:31 +02:00

Files

T

Michael Neale ddce5e6694 feat(mesh): experimental tier for models served across machines

Advanced settings gains a nested Experimental section listing split-capable
models that exceed this machine, with the consequence stated before any choice:
sharing will not start until enough well-connected computers are sharing the
same model. It waits instead of loading, and that waiting is indistinguishable
from a slow download to anyone who was not told.

## Why "beyond this machine" is the partition, not "multi-machine"

The catalog's runtime field marks a model multi-machine whenever a layer package
exists, which is not the same as needing several computers -- Qwen3-8B-Q4_K_M is
5 GB and split-capable. mesh-llm agrees about precedence:
evaluate_model_target_capacity returns SingleNodeFit first and only falls back
to SplitCandidate when no single node can hold the model. So the tier is the
intersection: split-capable AND beyond local memory. Anything that fits belongs
in the ordinary picker, where it starts immediately.

## Catalog comes from mesh-llm's loader

remote_catalog is public in mesh-llm-host-runtime, so Buzz calls it rather than
fetching and parsing the dataset itself -- one implementation of the format.
Hand-parsing is a trap: the package discriminator is `type` on the wire and
`package_type` in Rust via serde rename, and reading the Rust name off the wire
produced a confident "zero models are split-capable" earlier in this work. A
test pins the wire spelling, and it turned out stronger than intended -- serde
rejects a wrong discriminator outright, so format drift surfaces as a parse
error rather than a silently empty tier.

Reads the on-disk cache first and downloads only when nothing is cached, in
spawn_blocking. No running node required: you choose what to share before you
start sharing it.

## Empty is four distinct sentences

Still loading, could not read the catalog, the catalog holds no packages, and
this machine is large enough that nothing needs splitting. The last is good news
and must not read as a failure. A test asserts all four differ.

Two upstream entries publish "30 layers" where bytes belong, which parses to
nothing -- and is why mesh-llm reports unknown_model_size for the model this
machine serves. Unknown size is never claimed to fit: offering the split path is
recoverable, promising a local start that then fails is not.

Also updates the Share compute description to say why it is worth doing rather
than restating the mechanism.

Signed-off-by: Michael Neale <michael.neale@gmail.com>

2026-08-05 15:22:24 +10:00