ci: prove the relay-driven mesh lifecycle — discover, join, infer, deny — with real nodes (#3862)

## Summary CI now proves the full Buzz shared-compute join story end to end: a member can discover another member's served model **through the Buzz relay alone** and run inference over the mesh, while a non-member gets nothing — the relay rejects its auth, and the mesh refuses to route for it even holding a leaked endpoint address. This is deliberately different from mesh-llm's own CI smokes (which bootstrap two nodes with a hand-carried invite token / mdns): here the **relay is the control plane**, exactly like the desktop app: 1. **Membership** — identities A and B are added via `buzz-admin` (kind:13534 NIP-43 roster); C is not. 2. **Advertise** — each member publishes a client-signed kind:30003 discovery note carrying its MeshLLM owner binding and (for the serve node) `serveTargets[].endpointAddr`, covered by an endpoint-binding signature — the exact payload shape the desktop coordinator publishes. 3. **Trust** — the serve node derives its admission allowlist from the relay (statuses ∩ roster) and requires the **exact expected {A, B} owner-id set** before starting with `TrustPolicy::Allowlist`. 4. **Join** — the client verifies owner + endpoint bindings and membership, then dials the relay-discovered endpoint (the desktop join-watcher's `dial_endpoint_addr` step). No out-of-band token. 5. **Infer** — a chat completion against the client's local OpenAI endpoint routes over QUIC to the serve node's model (CPU, SmolLM2-135M, ~105MB). 6. **Deny (differential)** — the stranger's NIP-42 auth must fail with the relay's own membership rejection (`restricted: not a relay member` — successful auth or any unrelated connect error fails the run), and dialing the leaked endpoint must not produce a routed inference — **while the trusted client re-proves inference immediately afterwards**, so a dead serve node can't masquerade as an admission denial. ## What's in the PR - `crates/buzz-relay/examples/mesh_relay_lifecycle_smoke.rs` — the harness. One process per node (mesh-llm keeps process-global state under `~/.mesh-llm`), orchestrator + serve/client/stranger roles, byte-identical binding payloads to `desktop/src-tauri/src/mesh_llm/identity.rs` (called out with keep-in-sync comments). Child stdout is pumped through a reader thread so every wait has a hard deadline; timed-out children are killed; exit statuses are checked. - `scripts/ci-mesh-lifecycle-smoke.sh` — provisions a membership-gated relay (throwaway owner + signing identities via `buzz-admin generate-key`), runs the harness, cleans up. Fails fast if :3000 is already occupied (a stale open relay would mask gating). - `scripts/start-relay-for-tests.sh` — gains opt-in NIP-43 membership env passthrough (`BUZZ_REQUIRE_RELAY_MEMBERSHIP` + `RELAY_OWNER_PUBKEY` + `BUZZ_RELAY_PRIVATE_KEY`). Default behavior unchanged. - `.github/workflows/mesh-lifecycle.yml` — separate, path-filtered, non-required workflow (mesh paths, the harness's dependency crates, `Cargo.lock`, dispatch), pinned to `ubuntu-24.04`. Caches the mesh native runtime + HF model keyed on the lockfile hash, so a mesh pin bump rolls the runtime cache. Uploads relay + harness logs on failure. ## Scope This is an **independent protocol harness**: it speaks the same wire protocol and payload shapes as the desktop but re-implements the binding/verification logic (the desktop crate is outside the workspace). Regressions inside the desktop's own discovery filtering are the desktop unit tests' job; what this smoke proves is that the relay + mesh-llm SDK + admission stack support the lifecycle end to end. ## Relationship to mesh-llm's CI Follows the shape mesh-llm's own CI proved stable (tiny CPU model, one runner, multiple real mesh-llm processes over real QUIC — cf. their `ci-two-node-client-serving-smoke.sh`), but swaps the token bootstrap for the relay-driven lifecycle, which is the part only Buzz can test. ## Validation Green on GitHub Actions (ubuntu-24.04) across three runs, including after rebases onto the mesh v0.74 upgrade (#3467) and latest main: ``` PASS 1/6: relay-derived allowlist is exactly {A, B} PASS 2/6: serve member ready + advertised model: jc-builds/SmolLM2-135M-Instruct-Q4_K_M-GGUF:Q4_K_M PASS 3/6: client member discovered + joined via relay PASS 4/6: inference routed over the mesh: "PONG" PASS 5/6: relay rejected the stranger's NIP-42 auth (membership gate) PASS 6/6: stranger denied (gossip visible, inference rejected: 503 all tunnels failed) while trusted inference still routes PASS: full relay-driven mesh lifecycle verified ``` Also validated locally on macOS. `cargo fmt --all --check` and `cargo clippy -p buzz-relay --all-targets -- -D warnings` pass. ## Notes - The harness follows the repo's mesh `[dev-dependencies]` pin automatically, so it doubles as a canary for future mesh upgrades (it already caught the v0.73.1 → v0.74.0 bump during development). - The stranger "deny" accepts either shape mesh-llm exhibits: no model visibility at all, or gossip visibility with inference refused — mesh-llm applies the receiving node's owner policy after the gossip handshake, so admission gates *routing*, not gossip. The differential trusted-inference re-check (PASS 6/6) is what makes that a real denial rather than a dead server. - Model-visibility windows are tunable via `MESH_CLIENT_WINDOW_SECS` / `MESH_STRANGER_WINDOW_SECS` if shared runners prove slow — pin a longer window in the workflow env rather than re-running the job. --------- Signed-off-by: Michael Neale <michael.neale@gmail.com>
2026-08-18 06:50:31 +02:00 · 2026-08-05 23:13:16 -04:00
parent e2796d4a89
commit 38bf642fcf
6 changed files with 1311 additions and 0 deletions
@@ -0,0 +1,111 @@
+name: Mesh Lifecycle
+# Relay-driven mesh lifecycle smoke: membership → signed discovery notes →
+# relay-derived allowlist → join → CPU inference over QUIC → stranger denied
+# (relay membership rejection + no routed inference, with a differential
+# trusted-inference health proof so a dead serve node can't fake a denial).
+# Runs the full Buzz "shared compute" join story with three real mesh-llm
+# node processes on one runner, using the Buzz relay as the control plane
+# (no hand-carried invite tokens). Mirrors the shape mesh-llm's own CI uses
+# for its two-node smokes (tiny CPU model, one runner, real QUIC mesh).
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'crates/buzz-relay/examples/mesh_*.rs'
+      - 'crates/buzz-relay/Cargo.toml'
+      - 'crates/buzz-admin/**'
+      - 'crates/buzz-test-client/**'
+      - 'crates/buzz-ws-client/**'
+      - 'Cargo.lock'
+      - 'desktop/src-tauri/src/mesh_llm/**'
+      - 'scripts/ci-mesh-lifecycle-smoke.sh'
+      - 'scripts/start-relay-for-tests.sh'
+      - '.github/workflows/mesh-lifecycle.yml'
+  pull_request:
+    paths:
+      - 'crates/buzz-relay/examples/mesh_*.rs'
+      - 'crates/buzz-relay/Cargo.toml'
+      - 'crates/buzz-admin/**'
+      - 'crates/buzz-test-client/**'
+      - 'crates/buzz-ws-client/**'
+      - 'Cargo.lock'
+      - 'desktop/src-tauri/src/mesh_llm/**'
+      - 'scripts/ci-mesh-lifecycle-smoke.sh'
+      - 'scripts/start-relay-for-tests.sh'
+      - '.github/workflows/mesh-lifecycle.yml'
+  workflow_dispatch:
+
+concurrency:
+  group: mesh-lifecycle-${{ github.event_name == 'pull_request' && github.ref || github.sha }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+env:
+  CARGO_TERM_COLOR: always
+
+jobs:
+  lifecycle-smoke:
+    name: Relay-Driven Mesh Lifecycle Smoke
+    runs-on: ubuntu-24.04
+    timeout-minutes: 45
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+
+      - uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1
+
+      - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
+        with:
+          save-if: ${{ github.event_name != 'pull_request' }}
+
+      # The mesh-llm SDK downloads a signed native runtime (llama.cpp CPU
+      # build) on first init, and the serve node downloads the smoke model
+      # from HuggingFace on first run. Key on the lockfile so a mesh pin bump
+      # rolls the runtime cache; the model ref is stable.
+      - name: Restore mesh runtime + model caches
+        id: mesh-caches
+        uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5
+        with:
+          path: |
+            ~/.cache/mesh-llm/native-runtimes
+            ~/.cache/huggingface/hub
+          key: mesh-lifecycle-${{ runner.os }}-smollm2-135m-${{ hashFiles('Cargo.lock') }}
+          restore-keys: |
+            mesh-lifecycle-${{ runner.os }}-smollm2-135m-
+
+      - name: Start integration services
+        run: |
+          for attempt in 1 2 3; do
+            if docker compose up -d postgres redis minio minio-init; then
+              break
+            fi
+            if [ "$attempt" -eq 3 ]; then
+              echo "docker compose up failed after 3 attempts" >&2
+              exit 1
+            fi
+            echo "docker compose up failed (attempt $attempt), retrying in $((attempt * 5))s..." >&2
+            sleep $((attempt * 5))
+          done
+
+      - name: Run relay-driven mesh lifecycle smoke
+        run: ./scripts/ci-mesh-lifecycle-smoke.sh 2>&1 | tee /tmp/mesh-lifecycle-harness.log
+
+      - name: Save mesh runtime + model caches
+        if: github.ref == 'refs/heads/main' && steps.mesh-caches.outputs.cache-hit != 'true'
+        uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5
+        with:
+          path: |
+            ~/.cache/mesh-llm/native-runtimes
+            ~/.cache/huggingface/hub
+          key: mesh-lifecycle-${{ runner.os }}-smollm2-135m-${{ hashFiles('Cargo.lock') }}
+
+      - name: Upload relay + harness logs
+        if: failure()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
+        with:
+          name: mesh-lifecycle-logs
+          path: |
+            /tmp/buzz-relay.log
+            /tmp/mesh-lifecycle-harness.log
+          if-no-files-found: ignore
@@ -1196,11 +1196,13 @@ dependencies = [
 "buzz-relay-mesh",
 "buzz-sdk",
 "buzz-search",
+ "buzz-test-client",
 "buzz-workflow",
 "bytes",
 "chrono",
 "dashmap",
 "deadpool-redis",
+ "ed25519-dalek",
 "flate2",
 "futures",
 "futures-util",
@@ -86,6 +86,11 @@ dev = ["buzz-auth/dev"]
 [dev-dependencies]
 mesh-llm-sdk = { git = "https://github.com/Mesh-LLM/mesh-llm.git", tag = "v0.74.0", package = "mesh-llm-sdk", default-features = false, features = ["client", "serving"] }
 mesh-llm-host-runtime = { git = "https://github.com/Mesh-LLM/mesh-llm.git", tag = "v0.74.0", package = "mesh-llm-host-runtime", default-features = false, features = ["dynamic-native-runtime"] }
+# Relay-driven mesh lifecycle smoke (examples/mesh_relay_lifecycle_smoke.rs):
+# the relay client for discovery notes and the exact ed25519 the mesh owner
+# keys use for binding verification.
+buzz-test-client = { path = "../buzz-test-client" }
+ed25519-dalek = "=3.0.0-rc.0"
 buzz-core = { workspace = true, features = ["test-utils"] }
 buzz-auth = { workspace = true, features = ["dev"] }
 reqwest = { workspace = true }
@@ -0,0 +1,113 @@
+#!/usr/bin/env bash
+# =============================================================================
+# ci-mesh-lifecycle-smoke.sh — relay-driven mesh lifecycle smoke
+# =============================================================================
+# Provisions a membership-gated buzz-relay and runs the full relay-driven
+# mesh lifecycle harness (crates/buzz-relay/examples/mesh_relay_lifecycle_smoke.rs):
+# membership → signed discovery notes → relay-derived allowlist → join →
+# inference over QUIC → stranger denied.
+#
+# Mirrors the shape of mesh-llm's own CI smoke scripts (single runner, tiny
+# CPU model, real multi-process mesh), but with the Buzz relay as the control
+# plane instead of a hand-carried invite token.
+#
+# Usage:
+#   ./scripts/ci-mesh-lifecycle-smoke.sh [--profile <cargo-profile>] [--no-build]
+#
+# Env:
+#   MESH_SMOKE_MODEL   Override the served model ref (default: SmolLM2-135M).
+# =============================================================================
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
+cd "${REPO_ROOT}"
+
+CARGO_PROFILE="${CARGO_PROFILE:-ci}"
+SKIP_BUILD=false
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --profile) CARGO_PROFILE="$2"; shift 2 ;;
+    --no-build) SKIP_BUILD=true; shift ;;
+    *) echo "Unknown option: $1" >&2; exit 1 ;;
+  esac
+done
+
+BLUE='\033[0;34m'
+GREEN='\033[0;32m'
+RED='\033[0;31m'
+NC='\033[0m'
+log() { echo -e "${BLUE}[mesh-lifecycle]${NC} $*"; }
+ok()  { echo -e "${GREEN}[mesh-lifecycle]${NC} $*"; }
+err() { echo -e "${RED}[mesh-lifecycle]${NC} $*" >&2; }
+
+# ── Build ─────────────────────────────────────────────────────────────────────
+
+if [[ "${SKIP_BUILD}" == "true" ]]; then
+  log "Skipping build (--no-build)"
+else
+  log "Building relay, admin CLI, and lifecycle harness (profile: ${CARGO_PROFILE})..."
+  cargo build --profile "${CARGO_PROFILE}" -p buzz-relay -p buzz-admin -p git-credential-nostr
+  cargo build --profile "${CARGO_PROFILE}" -p buzz-relay --example mesh_relay_lifecycle_smoke
+fi
+
+ADMIN_BIN="target/${CARGO_PROFILE}/buzz-admin"
+HARNESS_BIN="target/${CARGO_PROFILE}/examples/mesh_relay_lifecycle_smoke"
+for bin in "${ADMIN_BIN}" "${HARNESS_BIN}" "target/${CARGO_PROFILE}/buzz-relay"; do
+  if [[ ! -x "${bin}" ]]; then
+    err "Missing binary: ${bin}"
+    exit 1
+  fi
+done
+
+# ── Relay identity + membership gating ───────────────────────────────────────
+# The relay owner and signing key are throwaway CI identities. RELAY_OWNER
+# never participates in the mesh; it only satisfies the NIP-43 requirement
+# that a membership-gated relay has an administrable owner.
+
+log "Generating relay owner + signing identities..."
+read_keys() { "${ADMIN_BIN}" generate-key 2>/dev/null; }
+OWNER_OUT="$(read_keys)"
+RELAY_OWNER_PUBKEY="$(echo "${OWNER_OUT}" | awk '/Public key:/ {print $3}')"
+SIGNER_OUT="$(read_keys)"
+BUZZ_RELAY_PRIVATE_KEY="$(echo "${SIGNER_OUT}" | awk '/Secret key:/ {print $3}')"
+if [[ -z "${RELAY_OWNER_PUBKEY}" || -z "${BUZZ_RELAY_PRIVATE_KEY}" ]]; then
+  err "Failed to generate relay identities via buzz-admin generate-key"
+  exit 1
+fi
+export BUZZ_REQUIRE_RELAY_MEMBERSHIP=true
+export RELAY_OWNER_PUBKEY
+export BUZZ_RELAY_PRIVATE_KEY
+
+# ── Start the membership-gated relay ─────────────────────────────────────────
+# A stale relay on :3000 would pass the readiness poll while silently running
+# WITHOUT membership gating — the stranger-denied assertion would then fail
+# (or worse, an open relay would mask a real gating regression). Fail fast.
+if lsof -nP -iTCP:3000 -sTCP:LISTEN >/dev/null 2>&1; then
+  err "Port 3000 is already in use — stop the existing relay first (its config would not be membership-gated)"
+  exit 1
+fi
+
+log "Starting membership-gated relay..."
+CARGO_PROFILE="${CARGO_PROFILE}" ./scripts/start-relay-for-tests.sh --no-build
+
+cleanup() {
+  log "Stopping relay..."
+  if [[ -f /tmp/buzz-relay.pid ]]; then
+    kill "$(cat /tmp/buzz-relay.pid)" 2>/dev/null || true
+  fi
+  # The harness kills its own children; sweep any stragglers from a hard fail.
+  pkill -f mesh_relay_lifecycle_smoke 2>/dev/null || true
+}
+trap cleanup EXIT
+
+# ── Run the lifecycle harness ────────────────────────────────────────────────
+
+log "Running relay-driven mesh lifecycle smoke..."
+RELAY_URL=ws://localhost:3000 \
+DATABASE_URL=postgres://buzz:buzz_dev@localhost:5432/buzz \
+REDIS_URL=redis://localhost:6379 \
+BUZZ_ADMIN_BIN="${ADMIN_BIN}" \
+  "${HARNESS_BIN}"
+
+ok "Relay-driven mesh lifecycle smoke passed"
@@ -151,6 +151,20 @@ fi
 # ── Start relay ──────────────────────────────────────────────────────────────

 log "Starting relay..."
+
+# Optional NIP-43 membership gating: exported by callers that need a
+# membership-gated relay (e.g. the mesh lifecycle smoke). All three must be
+# set together — the relay fails fast otherwise.
+MEMBERSHIP_ENV=()
+if [[ "${BUZZ_REQUIRE_RELAY_MEMBERSHIP:-}" == "true" ]]; then
+  MEMBERSHIP_ENV+=(
+    BUZZ_REQUIRE_RELAY_MEMBERSHIP=true
+    RELAY_OWNER_PUBKEY="${RELAY_OWNER_PUBKEY:?RELAY_OWNER_PUBKEY required with BUZZ_REQUIRE_RELAY_MEMBERSHIP=true}"
+    BUZZ_RELAY_PRIVATE_KEY="${BUZZ_RELAY_PRIVATE_KEY:?BUZZ_RELAY_PRIVATE_KEY required with BUZZ_REQUIRE_RELAY_MEMBERSHIP=true}"
+  )
+  log "Membership gating enabled (NIP-43)"
+fi
+
 nohup env \
  DATABASE_URL=postgres://buzz:buzz_dev@localhost:5432/buzz \
  REDIS_URL=redis://localhost:6379 \
@@ -159,6 +173,7 @@ nohup env \
  BUZZ_REQUIRE_AUTH_TOKEN=false \
  BUZZ_RECONCILE_CHANNELS=true \
  BUZZ_GIT_PROBE_WRITERS=8 \
+  ${MEMBERSHIP_ENV[@]+"${MEMBERSHIP_ENV[@]}"} \
  "./target/${CARGO_PROFILE}/buzz-relay" > /tmp/buzz-relay.log 2>&1 &
 echo $! > /tmp/buzz-relay.pid