docs(relay): preserve fence watermarks on cutover

Co-authored-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@buzz.block.builderlab.xyz>
Signed-off-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@buzz.block.builderlab.xyz>
This commit is contained in:
npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr
2026-07-31 14:20:14 -04:00
parent ae4ce2e945
commit ccef5bc1ea
+26 -6
View File
@@ -14,12 +14,32 @@ rollout cannot activate new-format writers while old writers still run.
can acquire or renew a legacy lease.
2. **Enable deploy (non-rolling):** scale the relay deployment to zero and wait
until the maximum graceful-shutdown/lease window has elapsed. Confirm that no
relay process remains and no legacy lease can still be renewed. Set
`BUZZ_TUNNEL_FENCE_KEY_FORMAT=tagged` on the whole deployment, move to the
cluster-enforcing endpoint, and then scale up. Do **not** start the first
tagged pod while any legacy-mode pod is alive. The first tagged acquisition
seeds its non-expiring generation from the drained legacy watermark before
incrementing it. Tagged state is authoritative once initialized.
relay process remains and no legacy lease can still be renewed. While writers
remain stopped, build a manifest from the source endpoint containing every
legacy `buzz:*:tunnel:*:generation` key and its exact string value. Transfer
those non-expiring generation keys, under the same legacy key names, to the
cluster-enforcing target endpoint. Do **not** copy lease keys: all expiring
leases must remain drained and must not be resurrected on the target.
Before starting any relay, rescan both endpoints and verify that the complete
source and target legacy-generation key sets are identical and that every
corresponding string value is byte-for-byte equal to the manifest. Also
verify that the source manifest did not change during transfer and that the
target has no legacy or tagged tunnel `*:lease` keys. If a generation key is
missing, extra, changed, unreadable, or cannot be written, **abort the
cutover**, keep all relays scaled to zero, and repair/repeat the transfer;
never start tagged writers from a partial or unverified copy. If the target
already contains tagged generation keys, stop and perform an
operator-reviewed reconciliation rather than overwriting or assuming that
the legacy manifest is authoritative.
Only after that verification succeeds, set
`BUZZ_TUNNEL_FENCE_KEY_FORMAT=tagged` on the whole deployment, switch the
configured Redis endpoint to the verified target, and scale up. Do **not**
start the first tagged pod while any legacy-mode pod is alive. The first
tagged acquisition reads the copied legacy watermark from its currently
configured target pool, seeds its non-expiring tagged generation, and then
increments it. Tagged state is authoritative once initialized.
Do not mix `legacy` and `tagged` writers. The two legacy keys occupy different
cluster slots, so Redis cannot atomically prove that an old writer did not race