buzz/docs/redis-cluster-mode.md

Redis/Valkey Cluster Mode for the Buzz Relay: A Formal Specification

draft

Abstract

This document specifies the migration of the Buzz relay's Redis/Valkey usage from a single-shard, cluster-mode-disabled ElastiCache replication group to a cluster-mode-enabled, N-shard topology, and gives a formal proof of the migration's safety-critical core. Valkey executes commands on one engine thread per node; the deployed primary's engine core is the relay's hard scaling ceiling, and it is filling on a measured doubling curve. Sharding the keyspace across N primaries is the durable fix — but only for keyed commands: classic pub/sub broadcasts every message to every node regardless of shard count, so converting the relay's event fan-out to sharded pub/sub (SPUBLISH/SSUBSCRIBE) is a prerequisite that feeds the sizing math, not a migration detail.

The safety-critical core is the fenced session directory: a per-session {lease, generation} key pair mutated atomically by Lua, whose generation counter is the relay mesh's fencing authority — non-expiring, never reset, never forked. Cluster mode forces these keys into one hash slot, which forces a key rename, which forces a live migration of fencing authority under a rolling deploy. We prove three safety theorems over that migration — single-authority (at no instant do two live leases exist for one session, across both old and new keyspaces, including under emergency rollback), generation-monotonicity (no generation is ever issued twice across the counter handoff or a rollback re-merge), and slot-rules safety (cluster protocol modes are entered only after old-keyspace authority is provably drained and the fleet runs only slot-clean scripts) — mechanized in TLA+ (docs/spec/RedisClusterFencingMigration.tla) with every invariant shown non-vacuous by mutation (9 mutants, all killed). The proof is scoped precisely: it covers acquisition authority, generation issuance, lease migration, and the mode doors, under a deployment-gate assumption (E1) that must be discharged operationally — it is not a property of our Kubernetes rollout machinery (§Deployment Gate).

Everything else in the migration is specified as ordered, gated engineering work with named failure modes: the split client architecture, the sharded pub/sub conversion, the silently-wrong SCAN in mesh peer discovery, and the staged ElastiCache mode change whose final step is irreversible.

Scope and Non-Goals

This specification proves safety of the fencing-key migration and states the rest of the migration as gated obligations. It deliberately does not prove:

• Liveness or performance. That N shards hold the growth curve, or that resharding completes within a window, is empirical — characterized by the profiling gate (G0) and load tests, not by theorem.
• Shard count. The shardable-keyspace vs. unshardable-broadcast ratio is an unmeasured input (§Open Decisions, D1). Committing a shard count before the profile exists would be fiction with units.
• redis-rs / deadpool-redis / ElastiCache internals. The cluster client's MOVED/ASK routing, RESP3 push delivery, and ElastiCache's online resharding are trusted vendor components, admitted per-deployment by conformance gates (§Conformance), not reproven.
• Valkey cluster correctness. Hash-slot assignment (CRC16 mod 16384, hash tags), single-slot Lua atomicity, and slot migration semantics are stated as axioms with documentation cites.

Stating this boundary is part of the claim, in the house style of docs/git-on-object-storage.md: "provably safe migration" without naming the trust boundary does not survive scrutiny; "the fencing handoff is machine-checked relative to four stated axioms, and every vendor behavior is gated by an explicit conformance test" does.

Problem Statement (measured, 2026-07-30)

ElastiCache replication group buzz-001 (acct 433851229429, us-west-2): Valkey 8.0.1, cache.r7g.8xlarge, 1 primary + 1 replica, cluster mode disabled, auto-failover enabled, Multi-AZ enabled (verified via describe-replication-groups / describe-cache-clusters, profile bb-public-operations-ro).

• EngineCPUUtilization (primary): 0.3% (Jul 21) → 8.4% (Jul 27) → 19.2% (Jul 30) — doubling every ~3.5–4 days. Saturation in ~8–10 days if the curve holds.
• Driver is command volume: get/set/eval families each ~150x in two weeks, pub/sub ~400x. Product growth (~80k users added), not a leak.
• Whole-instance CPU ~4%, memory 0.09%, replication lag <1ms. Vertical scaling is dead: same-family bigger instances have identical single-core speed; r8g (Graviton4) buys ~25–30% single-thread ≈ 1.5 days at the current doubling rate.

System Model

A relay pod is stateless compute (15 pods, HPA to 25). All pods share one Redis endpoint through two client surfaces (today): a deadpool-redis command pool built from one URL (crates/buzz-relay/src/main.rs), and three dedicated single-node pub/sub subscriber connections (crates/buzz-pubsub/src/{subscriber,cache_invalidation,conn_control}.rs).

The relay's Redis state divides into families with distinct consistency obligations. Making this distinction explicit is load-bearing for the whole design; the two Lua-fenced keys and the discovery index must not be held to the same standard:

Family	Keys	Obligation	Cluster-mode status
Fencing directory	buzz:<community>:tunnel:<session>:{lease,generation} (tunnel/directory.rs)	Strict: atomic two-key Lua; generation is non-expiring fencing authority — never fork, never repeat	Blocker (loud): keys share no hash tag; every script fails cross-slot. Migration is the mechanized core of this spec
Mesh ready registry	mesh:ready:<runtime_id> + SCAN discovery (relay-mesh/registry.rs)	Hint, by its own contract (registry.rs:3-6): entries are membership hints; the fenced directory arbitrates ownership. Bounded-staleness discovery is in-contract	Blocker (silent): SCAN has no key, routes to one random node, returns a partial keyspace with no error
Presence	buzz:<community>:presence:<pubkey>, TTL 90s (presence.rs)	Ephemeral, self-healing	Safe: single-key ops; bulk MGET is auto-split per node by redis-rs (MultipleNodeRoutingInfo::MultiSlot, cluster_handling/routing.rs)
Replay guard / rate limit	SET NX EX (nip98_replay.rs); single-key Lua (rate_limiter.rs)	Per-key atomic	Safe once routed
Event pub/sub	buzz:<community>:channel:<uuid>, buzz:<community>:global — exact topics (topic.rs)	At-most-once transport hint; relay re-checks access on fan-out	Unsharded by default (Axiom A4). Exact topics convert to SSUBSCRIBE/SPUBLISH; conversion is prerequisite P2
Broadcast pub/sub	PSUBSCRIBE buzz:*:cache-invalidate, buzz:*:conn-control	Must-not-miss control signals (cache coherence, live bans)	No sharded equivalent (SPSUBSCRIBE does not exist). Decision D2: stays classic — a missed transition is worse than a low-volume broadcast

Command inventory provenance: deployed image sha-22be8bb = Buzz commit 22be8bb35, scoped git grep over crates/buzz-pubsub, crates/buzz-relay, crates/buzz-relay-mesh; no MULTI/EXEC/pipeline found in that scope (RESEARCH/BB_PUBLIC_REDIS_CLUSTER_MODE_INVENTORY.md). The two hot event-path caches (channel_visibility_cached, is_member_cached) are in-process, not Redis — they are not the get/set driver.

Axioms

• (A1) Deterministic slot assignment. A key's slot is CRC16(key) mod 16384; if the key contains a {...} hash tag, only the tagged substring is hashed. Two keys sharing a tag always share a slot, on every topology. (Valkey cluster spec, key distribution model.)
• (A2) Single-slot Lua atomicity, declared-keys slot check. A Lua script whose declared KEYS all map to one slot executes atomically on that slot's primary, exactly as on a standalone node. A script whose declared keys span slots is rejected with CROSSSLOT (loud failure) on any cluster-enabled node — including a single-shard node that owns all 16384 slots: the check is on declared slot equality, not slot ownership (falsified live: single-shard cluster-enabled node, all slots assigned, cluster_state:ok, two-key EVAL on the un-tagged fencing pair → CROSSSLOT; raw RESP, no client library involved). Scripts must declare all keys via KEYS; ours do — and this is an enforced prohibition, not an observation: keys accessed via redis.call but not declared in KEYS bypass the slot check entirely and succeed on a single shard, then break silently the moment slots split. That is the deadline-pressure "fix" this spec explicitly forbids (verified live: numkeys=0 script reading a tagged + untagged key succeeds). The G2 validator (§Conformance) mechanically rejects undeclared key access.
• (A3) Staged mode change; slot rules start at compatible. ElastiCache supports cluster-mode: disabled → compatible → enabled. In compatible, one shard speaks both protocols and exposes a configuration endpoint; compatible → disabled is a supported revert. enabled is irreversible ("Reverting this configuration is not possible" — modify-cluster-mode.md). While compatible: scaling and engine-version changes are blocked. Working assumption: slot rules (A2's CROSSSLOT rejection) are enforced from compatible onward — compatible sets the cluster-enabled parameter, and the OSS behavior A2 documents attaches to cluster-enabled, not to shard count. We have not tested AWS compatible mode itself (it could conceivably relax slot checks for standalone-protocol clients); a one-command probe (EVAL on an un-tagged two-key pair) is a pre-G3 checklist item, and even if AWS turns out to permit it, the plan does not rely on that — vendor-tested leniency is not a protocol property, and it would silently vanish at enabled.
• (A4) Classic pub/sub is broadcast. PUBLISH to any node is propagated to every node in the cluster; each node's engine thread processes every message. Sharded pub/sub (SPUBLISH/SSUBSCRIBE) confines propagation to the channel's slot owner. (cluster-spec.md §pub/sub; pubsub.md §sharded.)

A3 and A4 are the two facts that shape the entire plan: A4 makes sharded pub/sub a sizing prerequisite; A3 makes compatible mode the last revert point.

Client Architecture (the split)

Verified constraints, from vendored redis 1.2.4 / deadpool-redis 0.23.0 source:

1. ssubscribe/sunsubscribe/cluster subscribe require RESP3 (check_resp3!, hard error on RESP2). We run RESP2 today.
2. The cluster async connection has no get_async_pubsub() and no split() — push messages arrive via an mpsc push_sender handed to ClusterClientBuilder at build time, and the client auto-resubscribes after disconnect, emitting duplicate subscription-confirmation pushes.
3. deadpool-redis's cluster Manager builds its client internally and exposes no push-sender hook — subscriptions need their own client regardless of protocol choice.
4. Protocol is per-connection, set by URL (?protocol=resp3) — the two clients can run different protocols with no global switch.

Therefore the target architecture is a split:

• Command pool: deadpool-redis cluster pool (feature cluster → redis/cluster-async) for all routed traffic — GET/SET/MGET/Lua/ SPUBLISH. Protocol (RESP2 vs RESP3) is a test-backed decision, not a forced rewrite: typed command APIs may remain source-compatible, but the choice is admitted only by the conformance gate (G2), never assumed.
• Subscription client: a separate raw ClusterClientBuilder with push_sender on RESP3, owning all three subscriber loops. The loops are rearchitected around push delivery (the select!-over-split() shape does not exist here); reconnect handling must tolerate auto-resubscribe and duplicate SSubscribe pushes.

Version floor: redis ≥ 1.4.1 (target 1.5.0), for: non-blocking replica connection repair (#2120, 1.4.0), cluster retry-backoff clamp (#2158, 1.3.0), and READONLY-path sleep removal (#2223, 1.4.1). Residual failure mode, not a fixed bug: on every released version, a dead primary halts dispatch pending topology refresh. Any degradation argument that assumes graceful behavior during primary loss is wrong; this belongs in shard-count risk math (more shards = smaller blast radius but more primaries that can die). The 1.2.4 → 1.5.0 bump is verified free in the current single-node world (Dawn, on origin/main 73589408d): cargo update -p redis --precise 1.5.0 resolves cleanly with no deadpool bump and no other crate moved; cargo build --workspace --all-targets is clean with zero source changes; all five redis-touching crates' tests pass (buzz-core, buzz-pubsub, buzz-relay-mesh, buzz-admin, buzz-relay — 0 failed), including buzz-pubsub's 11 live-Redis #[ignore] tests (pub/sub roundtrip, cache invalidation, presence, nip98 replay, cross- community isolation). The floor can therefore ship ahead of and independently from the cluster work — one less thing moving during the migration. That run proves nothing about cluster behavior (#2120/#2223 are cluster-only paths); the multi-shard suite (G2) still owns those. One additional known gap: async pub/sub reconnect fix redis-rs#2242 merged after 1.5.0's release — G2 must force-disconnect the subscription client and assert resubscription + delivery, never infer it from command tests.

The Fencing Migration (mechanized core)

Why it exists

The directory's four Lua scripts (acquire/renew/release/validate) each atomically touch …:lease and …:generation. Under cluster mode those keys must share a slot (A1, A2), so they must be renamed to carry a shared hash tag — e.g. buzz:tunnel:{<community>:<session>}:lease / …:generation. The generation counter is deliberately non-expiring and is the mesh's fencing authority (directory.rs:1-6): renaming it is a live handoff of authority under a rolling deploy, where old-script and new-script pods coexist. Done naively, two failure classes appear: fork (old-key lease and new-key lease alive simultaneously — two owners for one session) and generation reuse (new counter re-issues a generation the old counter already issued — a stale frame passes the fence).

The protocol

Script versions, deployed as a phased rollout. The fleet spanning at most two adjacent versions is NOT something Kubernetes gives us — it is environment assumption E1, discharged by the Deployment Gate below. All of O/A/B run in disabled mode only: A and B are cross-slot scripts, and cross-slot access is illegal on any cluster-enabled topology including a single shard (A2). There is no "compatible-mode grace period" for the migration — compatible comes after C.

• O (legacy, deployed today): old keys only.
• A: acquire checks both lease keys (union check); lease still written to the old key; generation issued as max(oldGen, newGen) + 1, written to the old counter. (The union read of both counters is the load-bearing part — mutation M5. An earlier draft also dual-wrote the generation to the new counter; mutation testing showed that write redundant, and it was removed: the smallest sufficient script wins.)
• B: union lease check; lease written to the new (hash-tagged) key; generation max(oldGen, newGen) + 1 written to the new counter. B's renewer migrates: when a B-pod's renew tick finds its own lease still on the old key (acquired before the pod upgraded), it moves the lease to the new key — same owner, same generation (no new issuance), one atomic cross-slot Lua script, legal because we are in disabled mode. This is what makes the drain reachable (see C-gate below).
• Backfill (once, after full-B): fold oldGen into newGen by max-merge. Idempotent.
• C: new keys only. No cross-slot access exists anywhere. C-gate (participation-based, not temporal): backfill has run and no live old-key lease remains, verified by direct observation (count old-keyspace leases = 0), never by waiting out a TTL. A fixed wait is wrong on its face: RENEW_SCRIPT is PEXPIRE (directory.rs) on a 10s cadence against the 30s TTL (reliable.rs), so a renewed lease never expires while its pod lives — "wait one TTL after full-B" drains nothing, and no session-lifetime bound exists in tunnel/ or audio/ to save it. What actually drains the old keyspace: B-pods migrate their own leases at the next renew tick (≤10s), and leases whose owners died expire by TTL because nobody renews them. Full-B therefore implies drain within bounded time — but the gate checks the state, not the clock. (The huddle lane, audio/join.rs, mirrors the reliable lane's 10s/30s renewer against the same directory and is expected to behave identically; that expectation is read, not traced, and the G2 drill covers both lanes explicitly.)
• Enter compatible (G3): only with the whole fleet on C. Slot rules turn on here (A3); C is the only slot-clean script version. Revert to disabled remains available.
• Enable (G4): compatible → enabled — the one-way door. Old keys become unreachable garbage; the fence never consults them again.

Per-phase operation table

"Renew/release operate on whichever key the caller's lease names" is not implementable with today's SessionLease — it carries no keyspace discriminator (directory.rs, SessionLease fields). The migration does not add one: instead each script version has a fixed, version-local key rule, and the table below is the normative statement per operation. A pod always addresses the keyspace its own script version dictates; the one cross-version case (a B-pod holding a lease acquired while it ran A-code on the old key) is handled by B's renew-migrate, not by lease-carried state. This version-local rule is an assumption the conformance suite tests across pod termination and rollback, not something the type system enforces.

Op	O	A	B	C
acquire	old key; INCR old ctr	union check; write old key; max(old,new)+1 → old ctr	union check; write new key; max(old,new)+1 → new ctr	new key; INCR new ctr
renew	old key PEXPIRE	old key PEXPIRE	migrate-then-renew: own lease on old key → move to new key (same gen, atomic); then new key PEXPIRE	new key PEXPIRE
release	old key	old key	try new key; fall back old key (owner+gen guarded, so wrong-key release is a no-op)	new key
lookup	old key	union (new wins)	union (new wins)	new key
known_generation	old ctr	max(old, new) ctrs	max(old, new) ctrs	new ctr
validate_fenced_header	old pair	union pair, max of counters	union pair, max of counters	new pair

validate_fenced_header is the hop-by-hop fence (directory.rs); its A/B variants read both pairs in one cross-slot script (legal: disabled mode) and fence against the max — a frame that passes validation under A/B would pass under whichever single keyspace currently holds authority, because T1 guarantees at most one does.

Scope of the mechanized proof

The TLA+ model covers acquisition authority, generation issuance, lease migration (B's renew-migrate), lease loss, phase machinery including emergency rollback, and the mode doors. It does not model renew, release, lookup, or validate as distinct transitions, and that is a scoping decision, not an omission:

• renew (TTL extension) is subsumed by the model's untimed nondeterminism — a renewed lease is exactly one where Expire has not yet fired. The renew-migrate step, which does change authority location, is modeled (MigrateB).
• release is state-identical to expiry (lease loss, generation preserved) — Expire covers both.
• lookup / known_generation / validate are read-only; they cannot violate T1/T2. Their per-phase read rules (table above) are specified and belong to the G2 test matrix.

So the honest claim is: T1/T2 are proved for every write to fencing state, under assumption E1; the read-side per-phase rules are specified and tested, not proved.

Deployment Gate (discharging E1)

The model's phase machinery assumes: the fleet chases one target phase, a phase advances only when every pod runs it, and rollback never skips a phase or overlaps a running downgrade. Kubernetes gives none of this. bb-public deploys a plain RollingUpdate (maxSurge: 1, maxUnavailable: 0) via GitOps; nothing prevents applying B while A is still rolling, rolling back two phases at once, or a new ReplicaSet coexisting with two older ones. E1 is therefore an operational gate, with these rules (each maps to a mutation that shows what its absence costs):

1. One image per phase. Each of A, B, C is a separately released, separately tagged image. No phase is a config flag on a shared image.
2. Advance gate: the next image change is blocked until (a) every live pod reports the expected phase and (b) all older ReplicaSets are at zero. Pods expose their script phase as a metric/readiness fact (e.g. a buzz_fencing_phase gauge), so the gate checks behavior, not an image tag. (Absence → mutants M1/M2-class forks: O+B coexist and fork authority.)
3. C-gate addition: backfill complete and observed old-keyspace lease count = 0 (direct observation, per §The protocol). (Absence → M4.)
4. Rollback matrix (emergency path, modeled as RollbackPhase):
   • Roll back one adjacent phase at a time; never start a second rollback while pods are still above the current target (absence → M8).
   • Any rollback invalidates the backfill; it must re-run before the C-gate can ever pass again.
   • Rolling back past A into O additionally requires the new keyspace lease-drained (observed zero; absence → M6) and an operator-run reverse max-merge of the new counter into the old one — O issues from the old counter alone and would otherwise re-issue generations B already issued (absence → M7).
   • No phase rollback while in compatible; revert the mode to disabled first (the mode doors and phase machinery never move in the same step).
5. Pre-G3 checklist: compute CLUSTER KEYSLOT on both members of a real hash-tagged pair and require equality before entering compatible — a mis-tagged pair is invisible in disabled mode (slot rules unenforced) and becomes a CROSSSLOT outage at G3. This is the only point the bug is catchable cheaply. Also run the one-command AWS compatible-mode probe from A3.

Safety theorems

T1 (Single Authority). At every instant of the migration — including under emergency rollback — at most one live lease exists per session across both keyspaces.

T2 (Generation Monotonicity). Generations issued to leases are strictly increasing per session across the old→new counter handoff, the renew-migrate, and any rollback re-merge; no generation is ever issued twice.

T3 (Slot-Rules Safety). A cluster-protocol mode (compatible or enabled) is entered only in states where old-keyspace authority is drained (backfill complete, no live old-key lease) and the whole fleet runs slot-clean C scripts — so no post-door execution ever attempts a cross-slot fencing script or consults a stale fencing key. That enabled is one-way is an environment axiom (A3), encoded as an action guard, not a theorem.

All three hold under environment assumption E1 (Deployment Gate).

Proof sketch. T1: every acquire version checks the union of both lease keys before creating authority; the renew-migrate moves a lease atomically (delete-old and write-new in one script) so no interleaving observes two; E1's phase adjacency means no pod that skips a keyspace coexists with a pod that writes it. T2: A and B issue max(oldGen,newGen)+1; the migrate carries an existing generation without issuing; the backfill max-merge makes C's counter dominate every generation ever issued; the rollback reverse max-merge restores that dominance to the old counter before O can issue again. T3: the mode door requires backfilled ∧ oldOwner = None ∧ fleet fully C, and phase rollback is disallowed while any cluster-protocol mode is active. ∎

The sketch is not the proof; the model is.

Mechanized verification

docs/spec/RedisClusterFencingMigration.tla models one session (sessions are independent — per-session keys), 3 pods, phased deployment with per-pod upgrade and downgrade interleaving, emergency rollback, TTL expiry, B's renew-migrate, backfill, and the three-position mode variable (disabled/compatible/enabled) with slot-rule enforcement from compatible onward. TLC checks five invariants; the history variables (lastIssued, monoOk) encode T2 as a single-run safety invariant.

$ java -cp tla2tools.jar tlc2.TLC RedisClusterFencingMigration.tla \
    -config RedisClusterFencingMigration.cfg -deadlock
Model checking completed. No error has been found.
12636 states generated, 3637 distinct states found.

(-deadlock disables deadlock reporting because the model has intended terminal states — migration complete, or the MaxGen finiteness bound reached — which TLC would otherwise report as errors. Pods = {p1,p2,p3}, MaxGen = 4. Three pods exercise every adjacent-version race — old/old, old/new, new/new writers; a fourth adds no qualitatively new interleaving. Bounded check, mutation-shown non-vacuous — the standard claim for a TLC-checked safety spec.)

Every invariant is non-vacuous by mutation (9 mutants, each run in isolation, all killed):

Mutation	Models the real bug	Trips
M1: AcquireB drops the old-lease check	new-script pod ignores legacy leases → two owners	Inv_SingleAuthority
M2: AcquireA drops the new-lease check	old-keyspace writer ignores new leases during rollback/mixed fleet	Inv_SingleAuthority
M3: AcquireB issues newGen+1 without max-merge	fresh counter re-issues generation 1 → stale frame passes fence	Inv_Monotonic
M4: C-gate dropped (no backfill/drain requirement)	C deployed while an old-key lease is live → C-pod acquires over it	Inv_SingleAuthority
M5: AcquireA drops the union max-merge read	A never sees B's issues → reuse	Inv_Monotonic
M6: rollback into O drops the new-keyspace drain check	O-pod acquires on the old key while a B-era lease lives on the new key → fork	Inv_SingleAuthority
M7: rollback into O drops the reverse max-merge	O re-issues generations B already issued	Inv_Monotonic
M8: rollback allowed past a running downgrade	skipped-phase rollback: O and B pods coexist	Inv_SingleAuthority
M9: compatible entered without fleet fully on C	A/B cross-slot scripts meet live slot rules	Inv_SlotRulesSafe

M4 is the operationally scary one: it is exactly "someone deploys the final script version early because everything looks green." M6–M8 are the price of admitting that rollbacks happen; they are what the Deployment Gate's rollback matrix discharges. Mutation testing also removed a mechanism: A's dual-write of the generation counter survived every invariant when deleted, so the protocol no longer carries it (§The protocol).

Sharded Pub/Sub Conversion (prerequisite P2)

By A4, shard count does not touch the ~400x pub/sub term; conversion does.

• Exact event topics (buzz:<community>:channel:<uuid>, buzz:<community>:global) convert to SSUBSCRIBE/SPUBLISH. The existing dynamic per-topic subscribe pattern maps directly (separate SSUBSCRIBE calls may span slots). Note: hash-tagging is deliberately not applied to topic names — each topic hashing to its own slot is what spreads propagation cost across shards; a {community} tag would recreate the hot node per community.
• The two pattern subscribers stay classic (Decision D2, argued in-thread and accepted): enumerating concrete channels would require a new subscribe-before-reachable / unsubscribe-after-last protocol with discovery races, and for cache invalidation and live-ban control a missed transition is strictly worse than a low-volume broadcast path. This accepts a permanently unsharded broadcast channel, gated by a volume measurement (G0 must confirm these two channels are noise; if they are not, D2 reopens).

Mesh Registry Discovery (the silent one)

scan_ready() (relay-mesh/registry.rs) discovers peers by SCAN MATCH mesh:ready:* COUNT 100. SCAN carries no key; redis-rs routes it to one random node (RouteBy::Undefined → SingleNodeRoutingInfo::Random) and the cursor legitimately reaches 0 on that node — so under cluster mode discovery silently returns a per-call-varying subset of peers. No error is ever raised. This is the only cluster-unsafe site that fails silently, and it must be tested by asserting completeness against a multi-shard cluster, not by checking for errors.

Fix: replace keyspace scanning with a scored-expiry index in one slot: heartbeat does ZADD <index> <expiry_ts> <runtime_id> alongside the existing SET EX record; discovery reads live-scored members, MGETs their records, filters, and opportunistically prunes dead scores. Staleness becomes representable (a score in the past) rather than an agreement property between two structures. The registry's own contract makes this sound: entries are membership hints — the fenced directory arbitrates ownership — and the read path already skips missing/undecodable/unattested records by design. The invariant is bounded discoverability (a live runtime is discoverable within one heartbeat interval; no unauthenticated record is ever returned), not set equality. The single-slot concentration clears on measured load: one ZADD per runtime per 15s across 15–25 pods ≈ 1–2 ops/s, three orders below the hot path — deployment-global, so re-check if fleet size explodes.

This spec is explicit that the registry's consistency obligations are weaker than the tunnel directory's by design. Holding both to the fencing standard would gold-plate a hint; holding the fence to the hint standard would corrupt sessions. (Table in §System Model.)

Migration Plan (gates, in order)

The ordering headline (changed from the first draft, per Dawn's live falsification): the entire fencing sequence A→B→backfill→drain→C happens in disabled mode. Compatible mode is not a grace period for cross-slot work — slot rules are assumed live from compatible onward (A3). This is strictly safer than the original ordering and costs nothing; it also means the vertical escape hatch (r8g) stays available for the whole of G1, since the scaling freeze only starts at G3.

G0 — Profile the hot path (informs everything; owner: option-3 owner). Attribute the 150x get/set and 400x pub/sub growth to code paths; measure the shardable-vs-broadcast ratio (this is the shard-count input) and the two pattern-subscriber volumes (gates D2). Known: in-process caches are not the driver; suspects are presence and per-event rate-limiter Lua — hypothesis, not finding. If the driver is the rate limiter, "batch it" is not available (atomic per-event counter); volume reduction takes a different shape.

G1 — Code prerequisites, all in disabled mode (can start now, independent of G0):

1. redis 1.2.4 → 1.5.0 (verified free; ship first, separately).
2. Fencing migration phases A→B→backfill→C per the mechanized protocol and the Deployment Gate (one image per phase, phase metric, advance gates, rollback matrix).
3. Registry scored-expiry index replacing SCAN.
4. Split client architecture + feature flags.
5. Subscriber loops rearchitected (push_sender, RESP3, auto-resubscribe).
6. Sharded pub/sub conversion for exact topics (D2 leaves patterns classic).

G2 — Conformance against a real multi-shard Valkey cluster (test matrix below). The A/B/backfill/C staged deploy gets its own disabled-mode integration/chaos gate (it can never legally run on a cluster-enabled topology); everything else tests C-scripts on multi-shard. No date is committed before this compiles and passes.

G3 — cluster-mode=compatible (revert available). Entered only with the fleet fully on C (T3). Pre-G3 checklist: CLUSTER KEYSLOT equality assertion on a real tagged pair; the AWS compatible-mode cross-slot probe (A3). Client cutover to the configuration endpoint; validate under real traffic. No scaling/engine changes while here (A3) — the r8g tourniquet and compatible-onward are mutually exclusive in flight; entering G3 forecloses the vertical escape hatch for the duration.

G4 — cluster-mode=enabled — the one-way door (A3). Requires: G2/G3 green and a key-size sanity pass (slots holding items >256MB silently refuse to migrate; our 0.09% memory makes this unlikely — "unlikely" is not "checked").

G5 — Add shards (online resharding). Do it early: AWS guidance says keep CPU <80% during resharding — resharding is compute-intensive, and doing it while the engine core saturates is the worst time. The countdown clock is the argument for starting G1 now, not for heroics at 90%.

Fallback (curve outruns the plan): replica read offload (buzz-001-002 idles at 7%) buys roughly one doubling. Note the coupling: in cluster mode, replica reads lean on exactly the replica-repair path fixed in redis-rs #2120/#2223, so the version floor is a prerequisite for the fallback too, not just hygiene.

Conformance (test matrix, gate G2)

Run against a real multi-shard Valkey 8 cluster (not a mock, not a single node in cluster mode) — except the fencing-migration row, which by construction runs on a standalone node (disabled mode is the only place A/B legally execute). Two verdict columns — errors and silently wrong — because three of our bugs would pass an error-only harness:

Surface	Must verify
Routing	MOVED/ASK redirects under slot migration; TLS/auth on the configuration endpoint
Locality contract	Atomic-looking work is never silently scattered: multi-key ops either share a slot or are explicitly per-key pipelines with partial-failure handling declared at the call site (the ioredis production resolution; Stripe's narrow-tag rule). A GitLab-style executable cross-slot validator in CI computes slots for every Lua KEYS declaration, pipeline, and multi-key command in the workspace, and rejects undeclared redis.call key access (A2's prohibition)
Fencing scripts	Hash-tagged pairs execute atomically; CROSSSLOT observed for un-tagged pairs (negative test); CLUSTER KEYSLOT equality asserted on real tagged pairs; per-node script cache / NOSCRIPT handling
Fencing migration	A/B/backfill/drain/C staged deploy against live traffic on a standalone (disabled-mode) node with chaos (pod kill mid-phase, rollback per the matrix, both tunnel lanes — reliable and huddle); generation strictly increases across the handoff and the renew-migrate (assert, don't assume); old-keyspace lease count observed to reach zero
Presence MGET	Nil/order preservation when split per node
Registry index	Completeness: every live runtime discoverable within one heartbeat; assert against known population, not absence of errors
Subscriptions	RESP3 push delivery; dynamic ssubscribe/sunsubscribe; forced disconnect of the subscription client with asserted resubscription + delivery (redis-rs#2242 merged after 1.5.0 — never infer this from command tests); duplicate subscription-confirmation pushes handled; message routing correct across shards
Failure drills	Replica connection repair under load (the #2120 path); primary failover — characterize the dispatch halt window; both clients (command pool + subscription client) have independent recovery behavior — drill both
Protocol choice	Command pool compiled and integration-tested under its chosen protocol (RESP2 or RESP3) — admitted by test, never assumed
Telemetry	Per-shard/slot/channel metrics exist before G5: aggregate CPU cannot attribute a hot slot, and the phase gauge (buzz_fencing_phase) feeds the Deployment Gate

Failure Modes (named, with dispositions)

Failure	Loud/Silent	Disposition
Cross-slot Lua (fencing keys, pre-fix)	Loud	Fixed by design (hash tag + mechanized migration)
Mis-tagged fencing pair (looks fine in disabled mode)	Silent until G3, then loud outage	Pre-G3 CLUSTER KEYSLOT equality assertion; G2 validator
Undeclared redis.call key access (bypasses slot check)	Silent until slots split	Prohibited by A2; G2 validator rejects mechanically
Deployment gate breach (phase skip, overlapping rollback, early phase apply)	Silent (authority fork / generation reuse)	E1 discharged operationally (§Deployment Gate); mutants M6–M8 show the cost
Old-keyspace drain assumed temporal ("wait one TTL")	Silent (renew = PEXPIRE; renewed leases never expire)	C-gate is participation-based + observed-zero; B renew-migrates
SCAN partial discovery	Silent	Fixed by design (scored-expiry index); completeness-asserting test
Dead primary halts client dispatch pending topology refresh	Loud-ish (latency wall)	Residual on all redis-rs versions; goes in shard-count risk math; drilled in G2
Missed cache-invalidate / conn-control message	Silent	Avoided by D2 (patterns stay classic broadcast)
Generation fork/reuse during migration	Silent (fence passes stale frame)	Excluded by T1/T2 (mechanized, mutation-tested)
Premature compatible/enabled	Loud (CROSSSLOT) / Irreversible	Excluded by T3 gate + A3 named as one-way door
Subscription client fails to resubscribe after disconnect	Silent (missed events)	Forced-disconnect drill in G2 (#2242 postdates 1.5.0)
>256MB items refuse slot migration	Silent (permanent imbalance)	Key-size pass at G4
Resharding under CPU pressure	Loud	G5 scheduled early, <80% CPU rule

Implementation Correspondence

Code pins are at deployed commit 22be8bb35 (verified against image tag sha-22be8bb); symbols, not line numbers, are normative after refactors.

Spec element	Code
Fencing scripts (acquire/renew/release/validate)	crates/buzz-relay/src/tunnel/directory.rs (ACQUIRE_SCRIPT etc.)
Fencing keys needing hash tag	SessionKeys::new (directory.rs)
Lease TTL (drain window for C-gate)	DEFAULT_LEASE_TTL = 30s (directory.rs)
Registry SCAN to replace	scan_ready (relay-mesh/registry.rs)
Registry heartbeat to extend with ZADD	publish_ready (registry.rs), spawn_registry_heartbeat (runtime.rs)
Exact topics for sharded pub/sub	EventTopicKey::redis_channel (buzz-pubsub/src/topic.rs)
Pattern subscribers staying classic	cache_invalidation.rs, conn_control.rs (psubscribe)
Subscriber loops to rearchitect	subscriber.rs (get_async_pubsub + split)
Huddle-lane lease renewer (mirrors reliable lane; drilled in G2)	audio/join.rs
Presence bulk read (auto-split, no change)	get_presence_bulk (presence.rs)
Command pool construction	deadpool_redis::Config::from_url (buzz-relay/src/main.rs)
Feature flags to add	workspace Cargo.toml redis/deadpool-redis entries
Phase gauge for the Deployment Gate	new metric (buzz_fencing_phase), exported per pod

Prior Art (what this spec adopts, and from where)

Three source-linked briefs inform this design (RESEARCH/ REDIS_CLUSTER_PRODUCTION_PRIOR_ART.md, …_OPEN_SOURCE_PRIOR_ART.md, …_OPERATIONAL_PRIOR_ART.md):

• Stripe (brandur.org/redis-cluster): a single hot engine core at scale, fixed by cluster mode with narrow, entity-scoped hash tags — adopted as the per-session fencing tag and the no-{community}-tag rule for topics. Their hardest part was hardening the client, not operating the cluster; this spec's weight distribution (client architecture + G2 over cluster ops) follows.
• GitLab (multi_store.rb, redis_cluster_validator): treats the move as an application migration with an old/dual/new state machine and an executable cross-slot validator — adopted as the O/A/B/C protocol shape (with the fencing-specific monotonic-generation handling GitLab doesn't need) and the G2 validator row.
• BullMQ: smallest-atomic-domain hash tags, spread across shards — corroborates per-session tagging; its years of cluster-fix changelog is the cautionary tale behind the version floor and G2's chaos rows.
• Sidekiq (negative prior art): refuses cluster configuration except a narrow cluster-safe subsystem — adopted as the per-family obligations table: cluster readiness is a capability boundary, not a client flag.
• Grafana Alertmanager HA: classic pub/sub coexisting with a cluster client — supports D2 (patterns stay classic) as viable, while proving nothing about fan-out capacity.
• ioredis #1842 (maintainer resolution): the locality contract — atomic work gets a narrow tag; unrelated multi-key work becomes per-slot pipelines with visible partial-failure semantics — adopted verbatim as the G2 locality row.
• No published production precedent exists for sharded pub/sub at Buzz-like fan-out scale (searched; Stripe validates command sharding only). The SPUBLISH/SSUBSCRIBE conversion therefore carries its own Buzz-specific load/reconnect gate rather than an appeal to prior art.
• AWS guidance (modify-cluster-mode.md, online-resharding best practices): compatible-as-revert-point (A3) and the <80% CPU resharding rule (G5).

Open Decisions

• D1 — Shard count. Blocked on G0's shardable/broadcast ratio. The spec deliberately refuses a number until the ratio exists.
• D2 — Pattern subscribers stay classic. Accepted in design; re-opens only if G0 shows their volume is non-trivial.
• D3 — Command-pool protocol (RESP2 vs RESP3). Test-backed decision at G2; the split architecture makes it independent of the subscription client's hard RESP3 requirement.
• D4 — G0 ownership. Profiling was kicked off as option 3; needs an owner and telemetry access. It gates D1, so it is on the critical path for shard count but not for starting G1.

Summary

Property	Status	Discharged by
Fencing single-authority (T1)	Proved (write-side, under E1)	TLA+ model, 9 mutants killed
Generation monotonicity (T2)	Proved (write-side, under E1)	TLA+ model, incl. rollback re-merge
Slot-rules safety (T3)	Proved (under E1)	Mode-door gate + A3, mechanized
Deployment gate E1	Operational obligation	§Deployment Gate (phase images, advance gates, rollback matrix, phase metric)
Read-side per-phase rules (lookup/validate/renew/release)	Specified + tested	§Per-phase operation table; G2
redis 1.2.4 → 1.5.0	Verified free (single-node)	Dawn's build+test run on 73589408d; cluster paths remain G2's
Vendor cluster behaviors	Empirical	Conformance matrix (G2)
Sharded pub/sub necessity	Documented fact	Axiom A4 (Valkey cluster spec)
Sharded pub/sub at scale	No prior art	Buzz-specific load/reconnect gate in G2
Shard count	Open	G0 profile (D1)
Timeline	~8–10 days of runway at current doubling	G1 starts now; fallback = replica offload