Cumulative PR layer #4789. Integrates the reviewed invalidation, resolver, restore, version, protected-publication, and migration status delta.
Neighbor disclosure: protected publication adapters overlap #4783 and preserve its canonical admission effects; status consumption remains in #4790, and relay production/operator activation remains in #4847.
Signed-off-by: Cea Stapleton Cordasco <261786559+cea@users.noreply.github.com>
Cumulative PR layer #4783. Integrates the reviewed protected admission effects, matching deployment schema, and single-flight publication regressions.
Neighbor disclosure: canonical lifecycle admission remains owned by #4772; invalidation and restore remain in #4789; live relay production composition remains in #4847.
Signed-off-by: Cea Stapleton Cordasco <261786559+cea@users.noreply.github.com>
Cumulative PR layer #1476. Integrates the reviewed authorization foundation, provider-neutral verification backend, loopback fixture repair, public dependency-source normalization, and the advisory-free Mesh revision.
Neighbor disclosure: bounded audit capacity remains in #3564; lifecycle and canonical admission remain in #4772; protected and status consumers remain in the later cumulative layers.
Signed-off-by: Cea Stapleton Cordasco <261786559+cea@users.noreply.github.com>
**Category:** fix
**User Impact:** Custom emoji with valid 64-character names can now be
used as reactions without errors.
**Problem:** Buzz accepted 64-character custom emoji names during
registration, but rejected them as reactions after the required
surrounding colons made the payload 66 characters. Validation also
differed between desktop, SDK, relay, and storage boundaries.
<img width="554" height="47" alt="image"
src="https://github.com/user-attachments/assets/4013452f-210e-4dd3-9003-f45ff3b28dc8"
/>
**Solution:** Keep the product limit at 64 ASCII characters for custom
emoji names, enforce it consistently when emoji sets are registered, and
allow only valid matching custom reaction payloads up to 66 characters.
Widen the reaction projection to preserve the wrapped payload while
retaining the existing 64-character limit for ordinary reactions.
<details>
<summary>File changes</summary>
**crates/buzz-sdk/src/builders.rs**
Defines the shared custom emoji boundaries and covers accepted
64-character and rejected 65-character shortcodes.
**crates/buzz-relay/src/handlers/ingest.rs**
Validates emoji-set shortcodes and permits 66-character reactions only
when they are valid colon-wrapped custom emoji with a matching tag.
**crates/buzz-db/src/event.rs**
Adds storage regression coverage for maximum-length custom emoji
reactions.
**crates/buzz-db/src/migration.rs**
Verifies the reaction column migration is applied correctly.
**desktop/src/shared/api/customEmoji.ts**
Enforces the existing 64-character shortcode maximum during desktop
normalization and registration/import.
**desktop/src/shared/api/customEmoji.test.mjs**
Covers the desktop shortcode boundary.
**migrations/0027_long_reaction_payloads.sql**
Widens stored reaction payloads to 66 characters for the two required
surrounding colons.
**schema/schema.sql**
Keeps the desired schema aligned with the migration.
</details>
## Reproduction Steps
1. Register or import a custom emoji whose ASCII shortcode is exactly 64
characters.
2. Select that emoji as a reaction to a message.
3. Confirm the reaction publishes, persists, and renders without an
error.
4. Attempt to register a 65-character shortcode and confirm it is
rejected.
5. Publish an ordinary or malformed reaction over 64 characters and
confirm the relay rejects it.
## Verification
- `cargo test -p buzz-sdk`: 243 passed
- `cargo test -p buzz-db`: 94 passed, 152 Postgres-required tests
ignored
- `pnpm test` in `desktop`: 3,859 passed
- `cargo test -p buzz-relay`: 795 passed, 9 existing
Postgres-unavailable failures, 35 ignored; new reaction boundary tests
pass directly
- `cargo fmt --all -- --check`
- `git diff --check`
Originating Buzz channel: `f2ec9671-d78e-4cde-894c-9f4c458c7f1f`
---------
Signed-off-by: Taylor Ho <taylorkmho@gmail.com>
Co-authored-by: npub1223z34hd7vtwc6qj4s7flsxkj644nlre2nthu7lrrmkumhu3xddsrx9r6w <52a228d6edf316ec6812ac3c9fc0d696ab59fc7954d77e7be31eedcddf91335b@buzz.block.builderlab.xyz>
## Problem
`SELECT id, community_id FROM channels WHERE id = ANY($1) AND deleted_at
IS NULL` is the top **Load by waits (AAS)** on the Buzz Postgres writer.
Two independent causes compound, and both are fixed here.
### 1. No index can serve it
`channels` is `PRIMARY KEY (community_id, id)`, and every secondary
index leads with `community_id`:
| Index | Columns |
|---|---|
| *(primary key)* | `(community_id, id)` |
| `idx_channels_nip29_group` | `(community_id, nip29_group_id)` |
| `idx_channels_dm_hash` | `(community_id, participant_hash)` |
| `idx_channels_community_type` | `(community_id, channel_type)` |
| `idx_channels_community_visibility` | `(community_id, visibility)` |
| `idx_channels_created_by` | `(community_id, created_by)` |
| `idx_channels_ttl_expiry` | `(ttl_deadline)` *(partial)* |
The two tenant-independent lookups carry **no `community_id` predicate**
— deliberately:
- `Db::communities_of_channels` — `WHERE id = ANY($1) AND deleted_at IS
NULL`
- `Db::community_of_channel` — `WHERE id = $1 AND deleted_at IS NULL`
That independence is load-bearing, not an oversight: projecting a row's
*true* owning community regardless of the fetch query's `WHERE` clause
is what makes `Inv_NonInterference` non-vacuous. If the fetch ever
dropped its tenant scoping, this lookup would still report the real
label and the checker would catch the mismatch.
But a composite btree is only usable when its leading column is
constrained, so neither query can use the primary key, and nothing else
leads with `id`. **Both sequentially scan `channels` on every call.**
### 2. In production the result is discarded
Both call sites feed `record_read_message_rows` /
`record_read_by_id_rows`, which call `tracer.record(...)`. Production
binds `NoopTracer` (`crates/buzz-relay/src/state.rs`), whose `record`
body is empty.
The existing guard tests `trace_state`, which is `Some` for every
well-formed request — it only goes `None` on malformed pubkey bytes. So
the scan ran on the hot read path and its output was dropped. This is
the classic eager-argument bug: `log.debug("..." + expensiveCall())`
with no `isDebugEnabled()` check.
### 3. Multiplied per filter
The non-search call site sits **inside the phase-3 per-filter loop**, so
a `REQ` carrying N filters performed N sequential scans of `channels`
before responding.
## Changes
**`Tracer::enabled()`** — a capability check on the trait (the
`isDebugEnabled()` of this seam), defaulting to `true`. `NoopTracer`
overrides it to `false`, and both emitters in `req.rs` now gate on it,
skipping the trace-only DB read entirely in production.
**`migrations/0027_channels_id_lookup_index.sql`**
```sql
CREATE INDEX IF NOT EXISTS idx_channels_id_live
ON channels (id) INCLUDE (community_id)
WHERE deleted_at IS NULL;
```
- `INCLUDE (community_id)` — both queries select exactly `(id,
community_id)`, so this is covering and can be served index-only.
- Partial on `deleted_at IS NULL` — matches both predicates exactly,
excludes soft-deleted history, and lets Postgres skip the recheck.
- **Not `UNIQUE`.** `id` alone is *not* unique in this table —
`command_executor.rs` documents that `community_of_channel(channel_id)`
is ambiguous because the same channel id can appear under more than one
community. A unique index would encode a false constraint and fail to
build on any database already holding such a pair.
Worth keeping the index even though fix#1 removes the production
caller: it still runs under conformance, and `community_of_channel` has
the same problem on its own paths.
**`schema/schema.sql`** — mirrored, since a test asserts desired-state
parity.
## Conformance is unchanged
This is the part worth reviewing closely. Under a real tracer
`enabled()` returns `true` and **every emit happens exactly as before**
— the gate only skips *building* emit inputs when nothing observes them,
never an emit that would otherwise have been made. The coverage-breach
guard stays non-vacuous.
`CountingTracer` forwards `enabled()` to its inner tracer rather than
inheriting the `true` default. Both directions matter and both fail
silently:
- inheriting `true` over a `NoopTracer` would keep the overhead this PR
removes;
- hardcoding `false` over a live tracer would suppress the emits whose
absence `EmitGuard` reports as `ImplBug` — masking real breaches behind
expected ones.
Covered by a new regression test,
`counting_tracer_delegates_enabled_to_inner`, which asserts delegation
in both directions.
## Verification
- `cargo check -p buzz-conformance -p buzz-relay` — clean
- `cargo clippy --all-targets` — clean, zero warnings
- `cargo test -p buzz-conformance` — 6/6
- `cargo test -p buzz-relay --lib conformance` — 11/11
- `cargo test -p buzz-db --lib migration` — 7/7
- `just test-unit` (pre-push) — green
Migration-count assertions in `crates/buzz-db/src/migration.rs` were
bumped 26 → 27, with content assertions for 0027 following the existing
per-migration pattern (including a guard that it never becomes
`UNIQUE`).
## Open questions for reviewers
1. **Lock strategy.** Built *without* `CONCURRENTLY`, following
migration 0004's precedent, because sqlx runs each migration inside a
transaction and `CREATE INDEX CONCURRENTLY` cannot run in one. This
takes a brief `SHARE` lock on `channels` (blocks writes, not reads) —
small relative to `events`, but an operator preferring zero
write-blocking can pre-build it by hand and `IF NOT EXISTS` makes the
migration a no-op. I could not confirm whether sqlx 0.9 supports a `--
no-transaction` directive; if it does, that may be preferable.
2. **Diagnosis is static.** This comes from reading the source, not from
`EXPLAIN` against the live database. Worth confirming with `EXPLAIN
(ANALYZE, BUFFERS)` on the writer before/after — that also sizes the win
by revealing the real table size and row counts.
3. **Expected impact** scales with average filters-per-`REQ`, which I
did not measure. `pg_stat_statements` ordered by `total_exec_time` would
confirm this query drops off the top and show whether anything else is
scanning the same way.
Signed-off-by: Jemiah Westerman <jemiah@squareup.com>
## Problem
The desktop deliberately shows the workspace icon editor on open relays
(#2640, gate: `canEditIcon` in
`desktop/src/features/communities/ui/EditCommunityDialog.tsx`) and
defers to the relay-side kind:9033 check — which required an admin/owner
row in `relay_members`. For a community with **no admin/owner row at
all** (the `ensure_configured_community` path, which never writes an
owner), every 9033 was refused and the icon was permanently unsettable.
**Correction from review (thanks @Dawn):** the original version of this
PR claimed nobody holds a role on an open relay. That's false —
`main.rs` bootstraps `RELAY_OWNER_PUBKEY` as owner regardless of
`BUZZ_REQUIRE_RELAY_MEMBERSHIP`, so a production open relay like
bb-block *does* have an owner row, and the old gate was refusing
everyone except that owner. The first revision of this diff would have
silently widened that owner-only control to any NIP-42-authenticated
sender.
## Fix — steward-wins
`may_set_workspace_profile(sender_role, membership_enforced,
community_has_steward)`:
| Relay mode | Community has admin/owner row? | Who may set the icon |
|---|---|---|
| Closed (`require_relay_membership=true`) | any | admin or owner
(unchanged) |
| Open | yes (e.g. bb-block) | admin or owner (unchanged posture) |
| Open | no (genuinely rosterless) | any NIP-42-authenticated sender |
- New DB helper `has_admin_or_owner(community)`
(`crates/buzz-db/src/relay_members.rs`); the call site only queries it
on open relays.
- The rosterless admit logs a `warn!` with the sender pubkey — 9033
writes no audit row and publishes no announcement event (unlike
9030/9031), so this is the only durable attribution.
- Kinds 9030–9032, NIP-42 auth, `AdminUsers` scope, ban gate, and icon
validation are all untouched.
- Doc comment fixed: cited nonexistent `canEditCommunityProfile`; real
symbol is `canEditIcon`.
## Test coverage — closing the mutation gap
Dawn's mutation testing showed the original unit tests pinned only the
helper's truth table: inverting the flag at the call site or deleting
the gate entirely survived the full suite.
- Unit tests now cover the 3-arg truth table (closed
steward-independent, open-with-steward stays steward-only,
rosterless-open admits).
- Two `#[ignore]`d Postgres integration tests drive
`handle_relay_admin_event` with a real `AppState` (open rosterless admit
→ steward appears → roleless refused again; closed relay member
refused). Wired into the Backend Integration CI job as a dedicated
nextest step.
- **Both of Dawn's mutants verified killed** at this head: flag
inversion fails 1 unit test; gate deletion fails both integration tests
(`Ok(())` where `Rejected` expected).
## CI wrinkle found and fixed: pre-existing schema drift
The first Backend Integration run of the new 9033 tests failed with
`column "icon" of relation "communities" does not exist` — migration
`0003_community_icon.sql` added the column, but `schema/schema.sql` (the
desired-state file that CI job applies via pgschema) was never updated.
Pre-existing drift, invisible until a test in that job actually wrote
the column. Fixed in `297148f62` (3-line addition to
`schema/schema.sql`).
## Receipts (at `1b4b52db8` code / `297148f62` head)
- `cargo test -p buzz-relay`: 835 pass, 1 fail —
`api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo`,
pre-existing (fails identically at the old base and on clean main);
`telemetry::trace_context_lookup_does_not_enable_callsites` is a known
order-dependent flake, passes in isolation.
- `cargo test -p buzz-db`: 94 pass.
- Both ignored integration tests pass live against local Postgres.
- `cargo fmt --all -- --check`: clean.
- Live-local pass per TESTING.md at this head (release build, relay on
:3199, real WS + NIP-42 via nak):
- open rosterless: roleless key sets icon → NIP-11 serves it; `warn!`
with sender pubkey in the relay log
- open + owner row inserted: fresh roleless key refused ("must be admin
or owner"); owner sets icon
- closed relay (owner bootstrapped, `BUZZ_RELAY_PRIVATE_KEY` set): plain
member refused, owner sets icon, `javascript:` URL rejected, empty icon
clears (NIP-11 → null)
---------
Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
## Summary
Expands read-replica usage on the relay per the Rev 2 design (thread
39d1e174 in #buzz-read-only-replica-usage): replaces the
Aurora-incompatible WAL-LSN read-side fence observation with a portable
**heartbeat token**, makes the freshness proof **snapshot-local to the
serving reader session**, and adds a default-off bounded-staleness gate
for head fetches.
### Probe (writer side, unchanged ordering)
The ordered writer scan is kept verbatim — `S = clock_timestamp()` →
masked-visibility `pg_stat_activity` oldest-xact scan — and now ends by
committing a heartbeat token **last** on the same pinned connection
(single-row `UPDATE replica_heartbeat ... RETURNING token, epoch`,
migration `0026`). The single-row update serializes all pods' probes, so
tokens are globally commit-ordered; the three-bucket completeness
argument carries over unchanged. Ring retains `(token, committed_at,
fence_wall)`; epoch change resets the ring; a same-epoch token
regression (restore adversary) clears the ring **and rotates the epoch
on the writer** so pre-rewind readers fail the epoch check. Cadence 1s.
### Routing (snapshot-local proof)
Every routed read opens `BEGIN ISOLATION LEVEL REPEATABLE READ, READ
ONLY` on a reader session and observes the heartbeat as the
transaction's **first statement** — the proof's snapshot is exactly the
snapshot the page, participants batch, and bridge aux closure read from
(`ReadSession` carries the open transaction; drop = rollback).
Fail-closed everywhere: begin failure, missing heartbeat row, epoch
mismatch, token below the ring, over-budget entry → writer.
- **Predicate B (cursor pages, default-on):** same completeness math as
the existing fence — cursor timestamp must be ≤ the proved wall; thread
candidate-terminal and above-wall pages re-run on the writer.
- **Predicate A (head fetches, default-OFF):** gated by
`BUZZ_REPLICA_HEAD_MAX_AGE_SECS` (0 = off, clamped to fence staleness).
Bounded-stale head semantics are an explicit product decision — **do not
enable anywhere without Tyler's backdated-event-semantics acceptance.**
### Observability
`buzz_db_route_decision{path, decision, reason}` across all five paths,
`buzz_db_replica_heartbeat_age_seconds` gauge, and per-decision debug
logs carrying the proved token plus backend identity: `addr:port pid=N`,
prefixed with `aurora_db_instance_identifier()` when the endpoint
supports it (probed once per process on an autocommit checkout; SQLSTATE
42883 caches a definitive false; identity is evidence, never a routing
gate).
## Review & verification
- **Wren:** full review 9/9/9 at `5f81b10e5`; identity delta re-review
approved at exact head `fedb46368` (90/90 unit, clippy `-D warnings`,
fmt independently reproduced).
- **Max:** local E2E **PASS** at `5f81b10e5` — isolated PG17 writer +
two streaming standbys behind HAProxy; paused-reader legs split exactly
as designed (6 replica/fresh + 6 writer/stale), recovery clean, RR
snapshot hardening observed in runtime logs. Evidence:
`WORK_LOGS/2026-07-28_REPLICA_HEARTBEAT_REPLACEMENT_SHA_E2E.md`.
- **My gates at head (same shell):** buzz-db 90 unit + 143
Postgres-gated green (scratch DBs); buzz-relay 767/768 — lone red is the
pre-existing `mesh_demo` flake, red on base; clippy `-D warnings` + fmt
clean.
- Key regression tests:
`routed_request_holds_one_snapshot_across_page_and_aux`
(mutation-verified both directions), same-epoch rotation,
capability-probe negative, head-gate truth table, divergent-fixture
routing suite.
## Post-merge plan
Merge publishes the immutable `sha-*` main image → bb-public PR pins it
→ ArgoCD sync → Max runs the production cursor-routing canary on Aurora
(positive identity branch proven live). Head gate stays off. Client
`since`-widening ships separately.
---
## Update — full read routing (Rev 6, commit `9fa3c9c0b`)
Extends routing to the remaining read seams per
`PLANS/REPLICA_FULL_READ_ROUTING_DESIGN.md` Rev 6 (thread cf5deba4 in
#buzz-read-only-replica-usage).
### New routed seams — ALL deploy-default dark
The new seams (`query_events_routed`, `query_events_routed_bounded`,
`count_events_routed`, `get_events_by_ids_routed`,
`query_feed_{mentions,needs_action,activity}_routed`) are
**Bounded-only** and gated on `BUZZ_REPLICA_READ_MAX_AGE_MS`: unset ⇒
every new seam records `writer/disabled` and merging is a no-op. COUNT
and feed/by-ids never take the covered arm (deletion visibility: covered
bounds insert-completeness only). **Note:** the pre-existing cursor
paths (channel windows, thread pages) are *not* gated by this env var —
they route at B=0 today and that status quo is unchanged.
### Reader pool (D4/D5)
- Lazy pool (`connect_lazy`, `min_connections(0)`): reader-down at boot
can't crash the relay; a warn-only boot ping is the only boot-time
visibility, and it primes the Aurora identity capability cache. Priming
is an optimization only — the routed path itself spends a single acquire
budget regardless (see the single-checkout fix below), because a boot
ping that *fails* is correlated with exactly the reader-unavailable case
the budget bound exists for.
- `READER_ACQUIRE_TIMEOUT` = 150ms; a miss fails closed to the writer
with reason **`reader_acquire_timeout`** — named for the mechanism, not
a diagnosis. The budget includes cold connects and sqlx's `size` counts
in-flight dials, so this metric alone does not distinguish contention
from slow connection establishment (see `proved_reader` doc-comment for
the runbook guidance; the pool gauges are 10s samples and are for
capacity planning).
- `BUZZ_DB_READ_POOL_SIZE` sizes the reader independently (invalid/0
inherits writer sizing); `read_pool_stats().max` reports the reader's
own ceiling.
### Community isolation (formal-model question)
Isolation is structural — an explicit `community_id = $n` predicate
compiled into every query builder; no RLS, no session-GUC tenant state
(zero `CREATE POLICY` across migrations). The `_on` variants reuse the
exact same builders with only the executor swapped. Proven by a
seven-seam two-community divergent-fixture test
(`routed_reads_are_confined_to_the_requested_community`),
mutation-tested by Dawn: all four single-predicate stubs killed; the
mention-join feeds are defended in depth (three independent predicates)
so only complete removal leaks there.
### Accepted limitation
D6: client-side staleness on bounded reads (up to `_MS`) is accepted
product behavior when the gate is enabled; gate stays off at merge.
## Update — single acquire budget per routed read (commit `dd26caa9f`)
Max found (and Dawn independently reproduced, 302–330ms measured) that
the boot-unavailable cold path spent **two** stacked
`READER_ACQUIRE_TIMEOUT` budgets: the Aurora capability probe did its
own `pool.acquire()` before `begin_with` acquired again. Fixed by
acquiring **once** per routed read — the capability probe runs on the
held connection (`reader_aurora_capability_on`) and the read-only
`REPEATABLE READ` transaction begins on that same connection
(`Transaction::begin` accepts a `PoolConnection` at `'static` via sqlx's
`MaybePoolConnection`). Reason codes unchanged; capability still never
negatively cached. Measured routed fallback: ~150ms (one budget).
Ships with a PG-gated regression fixture
(`routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold`,
authored by Dawn): size-1 reader saturated, capability cache asserted
cold, routes through `count_events_routed`, asserts writer answer +
one-budget elapsed + `writer/reader_acquire_timeout` label
(mutation-tested: `pool_busy` and `reader_validation_error` mutants both
killed). It fails at 330ms on the previous commit and passes on this
one.
### Verification at `dd26caa9f` (pinned toolchain rustc 1.95.0 via repo
`bin/`; ambient rustc 1.89 fails sqlx resolution — always prepend
`bin/`)
- buzz-db: 94 unit + **150/151** PG-gated serial (`--test-threads=1`;
suite is not parallel-safe against one Postgres). The 1 red is
`test_usage_metrics_lock_has_single_owner_and_releases_on_drop` — a
**pre-existing** same-key-same-database advisory-lock collision with any
live relay pointed at the shared `buzz` DB (the relay re-arms
`USAGE_METRICS_LOCK_KEY` on a 300s tick). It passes on a private scratch
DB with the relay still running — verified this run. Remedy is a scratch
`TEST_DATABASE_URL`, **not** terminating lock holders: inspect
`pg_locks`/`pg_stat_activity` and resolve the owning process first; an
idle advisory holder may be a live dev relay. Pre-existing, not a #3268
blocker — filed as #3619.
- buzz-relay: 773/773; clippy `-D warnings` + fmt clean. (`mesh_demo`
echo test is a known pre-existing Redis-dependent flake — Dawn confirmed
it fails identically on the base commit, untouched by this PR.)
- Dawn: independent re-verification at `dd26caa9f` — byte-identical diff
application confirmed, suite reproduced, and fixture proven to still
discriminate (fails at 320ms with only the production fix reverted).
Gate clear. Max: independent rig pass at `dd26caa9f` — 94/94 unit,
**15/15 routing/fallback matrix** (cursor behavior, default-dark seams,
seven-seam confinement, one-snapshot, dead-reader/no-URL fallback,
hard-delete fail-closed, reader max), 773/773 relay, private-DB lock
control. Gate clear; recommends merge + staged bb-block rollout
(no-reader-URL no-op → cursor-only → observe `buzz_db_route_decision` →
conservative nonzero `BUZZ_REPLICA_READ_MAX_AGE_MS`; rollback is
config-only).
## Known gap — CI does not run the PG-backed fixtures (#3622)
Dawn found post-sign-off (confirmed by Max and Eva at `dd26caa9f`):
**every Postgres-backed fixture in this PR is `#[ignore]`d and no CI job
selects it.** `Unit Tests` runs `cargo nextest run -p buzz-db --lib`
(`Justfile:279-285`, ignored tests excluded); the only `--run-ignored
ignored-only` invocations (`ci.yml:690`, `:702`) filter to
`relay_invite::tests`. So the one-budget regression, the seven-seam
isolation fixture, and the fence/floor-guard/fallback tests have zero
automated execution — the verification record above is exhaustive but
manual, at this exact SHA.
Not introduced by this PR (the `#[ignore]` + narrow-filter pattern
predates it; 34 ignored tests total) and not a merge blocker: all new
seams are dark until `BUZZ_REPLICA_READ_MAX_AGE_MS` is set. But it
changes the rollout gate — **do not set a nonzero
`BUZZ_REPLICA_READ_MAX_AGE_MS` in bb-block until CI enforces these
fixtures.** Filed as #3622 (explicit CI selection against the existing
backend-integration Postgres archive; ordering note: #3619 must land
first or be excluded, since widening the filter would select the
colliding usage-metrics lock test).
## Live redteam results (Max, real Helm + PG17 streaming replication at
`dd26caa9f`) — #3643, #3644
Max deployed this exact SHA via the repo chart against a physical
writer/standby pair and attacked it. **The PR's fenced routing passed
every fault**: healthy baseline routes proven on the real standby;
paused WAL replay moved head/cursor reads to `writer/stale`; B=0 moved
routed reads to `writer/disabled`; reader-URL removal rolled back
cleanly.
The redteam also found a **pre-existing** production risk this PR does
not create and does not fix: NIP-50 search builds its own eager,
unfenced pool straight from `READ_DATABASE_URL` (`main.rs:389-402`,
introduced by #2084, present in the deployed bb-block `sha-dd222a5` and
bb-public `sha-22be8bb` images, and both production ESOs already
template `READ_DATABASE_URL`). Measured: ~30s search stall on reader
loss, acknowledged-but-unsearchable writes under replica lag (unaffected
by `BUZZ_REPLICA_READ_MAX_AGE_MS`), fatal eager connect blocking pod
startup when the reader is down, and `replica=true` FTS even at B=0.
Filed as **#3643** (fix: pin FTS to the writer, or fence it like the
routed seams) with **#3644** for the writer-pointing-reader-URL
telemetry gap. Aurora's cluster-ro writer-fallback DNS softens the
outage modes in prod but not the lag mode.
**Revised ladder consequence:** step 2 ("add `READ_DATABASE_URL`, budget
unset") is NOT a no-op with today's binary — it hands FTS an unfenced
replica pool. Rollout order is now: merge (still a true no-op —
bb-block/bb-public already run the unfenced-FTS code and this PR only
adds dark seams) → fix#3643 → CI enforcement #3622 → then reader URL +
budget per the original ladder, re-running Max's three deployment tests
(reader-down latency, reader-down cold boot, paused-replay
read-your-own-write search).
## Activation gate (consolidated) — merge is clear; ALL of the below
precede any nonzero `BUZZ_REPLICA_READ_MAX_AGE_MS`
Six independent verification passes at `dd26caa9f` (Dawn ×2 incl.
byte-level + defang check, Max matrix + live Helm redteam, Wren full
CRUD regression + greenfield 0026 + 151/151 PG, Eva). Every induced
fault was either handled correctly by this PR's machinery or traced to
pre-existing main code. Remaining work gates **activation, not merge**:
1. **#3643** — unfenced NIP-50 FTS pool (pre-existing #2084; live in
prod today; also explains the blackhole search hang Wren observed — the
routed path was measured bounded at 151-152ms against a silent endpoint,
sqlx `inner.rs:252-255`).
2. **#3651** — reader `statement_timeout`: statements after acquire are
unbounded; mid-transaction blackhole reproduced hanging ≥15s (Dawn).
Small fix via `.after_connect` on the reader pool.
3. **#3622** — CI enforcement of the PG fixtures (with #3619 ordering).
4. **Recovery-conflict cancellation live test** — standby cancels a
routed read mid-snapshot → complete writer page, never partial (#3651's
fix also bounds the cancellation-never-arrives shape).
5. **DDL replication lag test** — migration on writer + paused replay +
budget on → fallback, not client-visible SQL errors.
6. **Reader-tx soak** — 30-60min mixed load; no idle-in-tx accumulation
on the standby.
Then Max's three deployment tests (reader-down latency, reader-down cold
boot, paused-replay read-your-own-write search) re-run on the fixed
binary before the first nonzero budget on bb-block.
---------
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Cea Stapleton Cordasco