mirror of
https://github.com/block/buzz.git
synced 2026-08-18 06:50:31 +02:00
## Summary
CI now proves the full Buzz shared-compute join story end to end: a
member can discover another member's served model **through the Buzz
relay alone** and run inference over the mesh, while a non-member gets
nothing — the relay rejects its auth, and the mesh refuses to route for
it even holding a leaked endpoint address.
This is deliberately different from mesh-llm's own CI smokes (which
bootstrap two nodes with a hand-carried invite token / mdns): here the
**relay is the control plane**, exactly like the desktop app:
1. **Membership** — identities A and B are added via `buzz-admin`
(kind:13534 NIP-43 roster); C is not.
2. **Advertise** — each member publishes a client-signed kind:30003
discovery note carrying its MeshLLM owner binding and (for the serve
node) `serveTargets[].endpointAddr`, covered by an endpoint-binding
signature — the exact payload shape the desktop coordinator publishes.
3. **Trust** — the serve node derives its admission allowlist from the
relay (statuses ∩ roster) and requires the **exact expected {A, B}
owner-id set** before starting with `TrustPolicy::Allowlist`.
4. **Join** — the client verifies owner + endpoint bindings and
membership, then dials the relay-discovered endpoint (the desktop
join-watcher's `dial_endpoint_addr` step). No out-of-band token.
5. **Infer** — a chat completion against the client's local OpenAI
endpoint routes over QUIC to the serve node's model (CPU, SmolLM2-135M,
~105MB).
6. **Deny (differential)** — the stranger's NIP-42 auth must fail with
the relay's own membership rejection (`restricted: not a relay member` —
successful auth or any unrelated connect error fails the run), and
dialing the leaked endpoint must not produce a routed inference —
**while the trusted client re-proves inference immediately afterwards**,
so a dead serve node can't masquerade as an admission denial.
## What's in the PR
- `crates/buzz-relay/examples/mesh_relay_lifecycle_smoke.rs` — the
harness. One process per node (mesh-llm keeps process-global state under
`~/.mesh-llm`), orchestrator + serve/client/stranger roles,
byte-identical binding payloads to
`desktop/src-tauri/src/mesh_llm/identity.rs` (called out with
keep-in-sync comments). Child stdout is pumped through a reader thread
so every wait has a hard deadline; timed-out children are killed; exit
statuses are checked.
- `scripts/ci-mesh-lifecycle-smoke.sh` — provisions a membership-gated
relay (throwaway owner + signing identities via `buzz-admin
generate-key`), runs the harness, cleans up. Fails fast if :3000 is
already occupied (a stale open relay would mask gating).
- `scripts/start-relay-for-tests.sh` — gains opt-in NIP-43 membership
env passthrough (`BUZZ_REQUIRE_RELAY_MEMBERSHIP` + `RELAY_OWNER_PUBKEY`
+ `BUZZ_RELAY_PRIVATE_KEY`). Default behavior unchanged.
- `.github/workflows/mesh-lifecycle.yml` — separate, path-filtered,
non-required workflow (mesh paths, the harness's dependency crates,
`Cargo.lock`, dispatch), pinned to `ubuntu-24.04`. Caches the mesh
native runtime + HF model keyed on the lockfile hash, so a mesh pin bump
rolls the runtime cache. Uploads relay + harness logs on failure.
## Scope
This is an **independent protocol harness**: it speaks the same wire
protocol and payload shapes as the desktop but re-implements the
binding/verification logic (the desktop crate is outside the workspace).
Regressions inside the desktop's own discovery filtering are the desktop
unit tests' job; what this smoke proves is that the relay + mesh-llm SDK
+ admission stack support the lifecycle end to end.
## Relationship to mesh-llm's CI
Follows the shape mesh-llm's own CI proved stable (tiny CPU model, one
runner, multiple real mesh-llm processes over real QUIC — cf. their
`ci-two-node-client-serving-smoke.sh`), but swaps the token bootstrap
for the relay-driven lifecycle, which is the part only Buzz can test.
## Validation
Green on GitHub Actions (ubuntu-24.04) across three runs, including
after rebases onto the mesh v0.74 upgrade (#3467) and latest main:
```
PASS 1/6: relay-derived allowlist is exactly {A, B}
PASS 2/6: serve member ready + advertised model: jc-builds/SmolLM2-135M-Instruct-Q4_K_M-GGUF:Q4_K_M
PASS 3/6: client member discovered + joined via relay
PASS 4/6: inference routed over the mesh: "PONG"
PASS 5/6: relay rejected the stranger's NIP-42 auth (membership gate)
PASS 6/6: stranger denied (gossip visible, inference rejected: 503 all tunnels failed) while trusted inference still routes
PASS: full relay-driven mesh lifecycle verified
```
Also validated locally on macOS. `cargo fmt --all --check` and `cargo
clippy -p buzz-relay --all-targets -- -D warnings` pass.
## Notes
- The harness follows the repo's mesh `[dev-dependencies]` pin
automatically, so it doubles as a canary for future mesh upgrades (it
already caught the v0.73.1 → v0.74.0 bump during development).
- The stranger "deny" accepts either shape mesh-llm exhibits: no model
visibility at all, or gossip visibility with inference refused —
mesh-llm applies the receiving node's owner policy after the gossip
handshake, so admission gates *routing*, not gossip. The differential
trusted-inference re-check (PASS 6/6) is what makes that a real denial
rather than a dead server.
- Model-visibility windows are tunable via `MESH_CLIENT_WINDOW_SECS` /
`MESH_STRANGER_WINDOW_SECS` if shared runners prove slow — pin a longer
window in the workflow env rather than re-running the job.
---------
Signed-off-by: Michael Neale <michael.neale@gmail.com>
201 lines
8.0 KiB
Bash
Executable File
201 lines
8.0 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# =============================================================================
|
|
# start-relay-for-tests.sh — Start the Buzz relay and its backing services
|
|
# =============================================================================
|
|
# Shared script for CI jobs that need a running relay. Starts docker compose
|
|
# services, waits for health, applies the schema, builds the relay, starts it,
|
|
# and polls readiness.
|
|
#
|
|
# Usage:
|
|
# ./scripts/start-relay-for-tests.sh [--profile <cargo-profile>] [--no-build]
|
|
#
|
|
# Options:
|
|
# --profile <profile> Cargo build profile (default: ci)
|
|
# --no-build Use existing target/<profile>/ binaries (CI artifact reuse)
|
|
#
|
|
# Exports:
|
|
# RELAY_URL=ws://localhost:3000
|
|
# =============================================================================
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
|
|
|
# ── Defaults ──────────────────────────────────────────────────────────────────
|
|
|
|
CARGO_PROFILE="${CARGO_PROFILE:-ci}"
|
|
SKIP_BUILD=false
|
|
|
|
# ── Parse args ────────────────────────────────────────────────────────────────
|
|
|
|
while [[ $# -gt 0 ]]; do
|
|
case "$1" in
|
|
--profile)
|
|
CARGO_PROFILE="$2"
|
|
shift 2
|
|
;;
|
|
--no-build)
|
|
SKIP_BUILD=true
|
|
shift
|
|
;;
|
|
*)
|
|
echo "Unknown option: $1" >&2
|
|
exit 1
|
|
;;
|
|
esac
|
|
done
|
|
|
|
# ── Colors ────────────────────────────────────────────────────────────────────
|
|
|
|
BLUE='\033[0;34m'
|
|
GREEN='\033[0;32m'
|
|
RED='\033[0;31m'
|
|
NC='\033[0m'
|
|
|
|
log() { echo -e "${BLUE}[relay-test]${NC} $*"; }
|
|
ok() { echo -e "${GREEN}[relay-test]${NC} $*"; }
|
|
err() { echo -e "${RED}[relay-test]${NC} $*" >&2; }
|
|
|
|
# ── Start docker compose services ────────────────────────────────────────────
|
|
|
|
cd "${REPO_ROOT}"
|
|
|
|
log "Starting docker compose services..."
|
|
docker compose up -d postgres redis minio minio-init
|
|
|
|
# ── Wait for services to be healthy ──────────────────────────────────────────
|
|
|
|
wait_healthy() {
|
|
local service="$1"
|
|
local container="$2"
|
|
log "Waiting for ${service}..."
|
|
for attempt in $(seq 1 60); do
|
|
status=$(docker inspect --format='{{.State.Health.Status}}' "${container}" 2>/dev/null || echo "not_found")
|
|
if [ "${status}" = "healthy" ]; then
|
|
ok "${service} is healthy"
|
|
return 0
|
|
fi
|
|
sleep 2
|
|
done
|
|
err "${service} did not become healthy within 120s"
|
|
docker logs "${container}" || true
|
|
return 1
|
|
}
|
|
|
|
wait_healthy "Postgres" "buzz-postgres"
|
|
wait_healthy "Redis" "buzz-redis"
|
|
wait_healthy "MinIO" "buzz-minio"
|
|
|
|
# ── Apply database schema ────────────────────────────────────────────────────
|
|
|
|
log "Applying database schema..."
|
|
export PGHOST=localhost
|
|
export PGPORT=5432
|
|
export PGUSER=buzz
|
|
export PGPASSWORD=buzz_dev
|
|
export PGDATABASE=buzz
|
|
|
|
# Use the already-running docker postgres for desired-state planning instead of
|
|
# downloading an embedded Postgres from Maven Central (transient-fetch flake source).
|
|
export PGSCHEMA_PLAN_HOST=localhost
|
|
export PGSCHEMA_PLAN_PORT=5432
|
|
export PGSCHEMA_PLAN_DB=buzz
|
|
export PGSCHEMA_PLAN_USER=buzz
|
|
export PGSCHEMA_PLAN_PASSWORD=buzz_dev
|
|
|
|
./bin/pgschema apply --file schema/schema.sql --auto-approve
|
|
docker exec -i -e PGPASSWORD="${PGPASSWORD}" buzz-postgres \
|
|
psql -U "${PGUSER}" -d "${PGDATABASE}" -v ON_ERROR_STOP=1 < scripts/attach-schema-partitions.sql
|
|
ok "Schema applied"
|
|
|
|
# ── Seed the deployment community ────────────────────────────────────────────
|
|
# Multi-tenant: the relay resolves every connection's tenant from the durable
|
|
# communities host map (WHERE host = normalize_host($1)). normalize_host keeps
|
|
# non-default ports, so the host must be 'localhost:3000' verbatim to match
|
|
# RELAY_URL=ws://localhost:3000. The relay never auto-seeds a community
|
|
# (ensure_configured_community has no callers) and fails closed on an unmapped
|
|
# host, so without this row every e2e connection would 404 at host-binding.
|
|
# The unique index is on lower(host), so ON CONFLICT must target that expression.
|
|
# psql is not on PATH in the hermit env; postgres runs as the buzz-postgres
|
|
# docker container, so exec into it (same fallback as setup-desktop-test-data.sh).
|
|
log "Seeding deployment community (host=localhost:3000)..."
|
|
if command -v psql >/dev/null 2>&1; then
|
|
seed_psql() { PGPASSWORD="${PGPASSWORD}" psql -h "${PGHOST}" -p "${PGPORT}" -U "${PGUSER}" -d "${PGDATABASE}" -qtA "$@"; }
|
|
else
|
|
seed_psql() { docker exec -e PGPASSWORD="${PGPASSWORD}" buzz-postgres psql -U "${PGUSER}" -d "${PGDATABASE}" -qtA "$@"; }
|
|
fi
|
|
seed_psql -c "
|
|
INSERT INTO communities (id, host)
|
|
VALUES ('00000000-0000-4000-8000-00000000c0de', 'localhost:3000')
|
|
ON CONFLICT (lower(host)) DO NOTHING
|
|
;
|
|
"
|
|
ok "Community seeded"
|
|
|
|
# ── Build relay ──────────────────────────────────────────────────────────────
|
|
|
|
if [[ "${SKIP_BUILD}" == "true" ]]; then
|
|
for bin in buzz-relay git-credential-nostr; do
|
|
if [[ ! -x "./target/${CARGO_PROFILE}/${bin}" ]]; then
|
|
err "--no-build: ./target/${CARGO_PROFILE}/${bin} missing or not executable"
|
|
exit 1
|
|
fi
|
|
done
|
|
log "Skipping relay build (--no-build); using existing target/${CARGO_PROFILE}/ binaries"
|
|
else
|
|
log "Building relay (profile: ${CARGO_PROFILE})..."
|
|
cargo build --profile "${CARGO_PROFILE}" -p buzz-relay -p git-credential-nostr
|
|
ok "Relay built"
|
|
fi
|
|
|
|
# ── Start relay ──────────────────────────────────────────────────────────────
|
|
|
|
log "Starting relay..."
|
|
|
|
# Optional NIP-43 membership gating: exported by callers that need a
|
|
# membership-gated relay (e.g. the mesh lifecycle smoke). All three must be
|
|
# set together — the relay fails fast otherwise.
|
|
MEMBERSHIP_ENV=()
|
|
if [[ "${BUZZ_REQUIRE_RELAY_MEMBERSHIP:-}" == "true" ]]; then
|
|
MEMBERSHIP_ENV+=(
|
|
BUZZ_REQUIRE_RELAY_MEMBERSHIP=true
|
|
RELAY_OWNER_PUBKEY="${RELAY_OWNER_PUBKEY:?RELAY_OWNER_PUBKEY required with BUZZ_REQUIRE_RELAY_MEMBERSHIP=true}"
|
|
BUZZ_RELAY_PRIVATE_KEY="${BUZZ_RELAY_PRIVATE_KEY:?BUZZ_RELAY_PRIVATE_KEY required with BUZZ_REQUIRE_RELAY_MEMBERSHIP=true}"
|
|
)
|
|
log "Membership gating enabled (NIP-43)"
|
|
fi
|
|
|
|
nohup env \
|
|
DATABASE_URL=postgres://buzz:buzz_dev@localhost:5432/buzz \
|
|
REDIS_URL=redis://localhost:6379 \
|
|
RELAY_URL=ws://localhost:3000 \
|
|
BUZZ_BIND_ADDR=0.0.0.0:3000 \
|
|
BUZZ_REQUIRE_AUTH_TOKEN=false \
|
|
BUZZ_RECONCILE_CHANNELS=true \
|
|
BUZZ_GIT_PROBE_WRITERS=8 \
|
|
${MEMBERSHIP_ENV[@]+"${MEMBERSHIP_ENV[@]}"} \
|
|
"./target/${CARGO_PROFILE}/buzz-relay" > /tmp/buzz-relay.log 2>&1 &
|
|
echo $! > /tmp/buzz-relay.pid
|
|
|
|
# ── Poll readiness ───────────────────────────────────────────────────────────
|
|
|
|
log "Waiting for relay readiness..."
|
|
for attempt in $(seq 1 60); do
|
|
if ! kill -0 "$(cat /tmp/buzz-relay.pid)" 2>/dev/null; then
|
|
err "Relay process died"
|
|
cat /tmp/buzz-relay.log
|
|
exit 1
|
|
fi
|
|
status_code=$(curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:3000/_readiness || true)
|
|
if [ "${status_code}" = "200" ]; then
|
|
ok "Relay is ready at ws://localhost:3000"
|
|
export RELAY_URL=ws://localhost:3000
|
|
exit 0
|
|
fi
|
|
sleep 1
|
|
done
|
|
|
|
err "Relay did not become ready within 60s"
|
|
cat /tmp/buzz-relay.log
|
|
exit 1
|