mirror of
https://github.com/block/buzz.git
synced 2026-08-18 06:50:31 +02:00
Jemiah WestermanandGitHub
bc9e6528a7
## Problem
`SELECT id, community_id FROM channels WHERE id = ANY($1) AND deleted_at
IS NULL` is the top **Load by waits (AAS)** on the Buzz Postgres writer.
Two independent causes compound, and both are fixed here.
### 1. No index can serve it
`channels` is `PRIMARY KEY (community_id, id)`, and every secondary
index leads with `community_id`:
| Index | Columns |
|---|---|
| *(primary key)* | `(community_id, id)` |
| `idx_channels_nip29_group` | `(community_id, nip29_group_id)` |
| `idx_channels_dm_hash` | `(community_id, participant_hash)` |
| `idx_channels_community_type` | `(community_id, channel_type)` |
| `idx_channels_community_visibility` | `(community_id, visibility)` |
| `idx_channels_created_by` | `(community_id, created_by)` |
| `idx_channels_ttl_expiry` | `(ttl_deadline)` *(partial)* |
The two tenant-independent lookups carry **no `community_id` predicate**
— deliberately:
- `Db::communities_of_channels` — `WHERE id = ANY($1) AND deleted_at IS
NULL`
- `Db::community_of_channel` — `WHERE id = $1 AND deleted_at IS NULL`
That independence is load-bearing, not an oversight: projecting a row's
*true* owning community regardless of the fetch query's `WHERE` clause
is what makes `Inv_NonInterference` non-vacuous. If the fetch ever
dropped its tenant scoping, this lookup would still report the real
label and the checker would catch the mismatch.
But a composite btree is only usable when its leading column is
constrained, so neither query can use the primary key, and nothing else
leads with `id`. **Both sequentially scan `channels` on every call.**
### 2. In production the result is discarded
Both call sites feed `record_read_message_rows` /
`record_read_by_id_rows`, which call `tracer.record(...)`. Production
binds `NoopTracer` (`crates/buzz-relay/src/state.rs`), whose `record`
body is empty.
The existing guard tests `trace_state`, which is `Some` for every
well-formed request — it only goes `None` on malformed pubkey bytes. So
the scan ran on the hot read path and its output was dropped. This is
the classic eager-argument bug: `log.debug("..." + expensiveCall())`
with no `isDebugEnabled()` check.
### 3. Multiplied per filter
The non-search call site sits **inside the phase-3 per-filter loop**, so
a `REQ` carrying N filters performed N sequential scans of `channels`
before responding.
## Changes
**`Tracer::enabled()`** — a capability check on the trait (the
`isDebugEnabled()` of this seam), defaulting to `true`. `NoopTracer`
overrides it to `false`, and both emitters in `req.rs` now gate on it,
skipping the trace-only DB read entirely in production.
**`migrations/0027_channels_id_lookup_index.sql`**
```sql
CREATE INDEX IF NOT EXISTS idx_channels_id_live
ON channels (id) INCLUDE (community_id)
WHERE deleted_at IS NULL;
```
- `INCLUDE (community_id)` — both queries select exactly `(id,
community_id)`, so this is covering and can be served index-only.
- Partial on `deleted_at IS NULL` — matches both predicates exactly,
excludes soft-deleted history, and lets Postgres skip the recheck.
- **Not `UNIQUE`.** `id` alone is *not* unique in this table —
`command_executor.rs` documents that `community_of_channel(channel_id)`
is ambiguous because the same channel id can appear under more than one
community. A unique index would encode a false constraint and fail to
build on any database already holding such a pair.
Worth keeping the index even though fix #1 removes the production
caller: it still runs under conformance, and `community_of_channel` has
the same problem on its own paths.
**`schema/schema.sql`** — mirrored, since a test asserts desired-state
parity.
## Conformance is unchanged
This is the part worth reviewing closely. Under a real tracer
`enabled()` returns `true` and **every emit happens exactly as before**
— the gate only skips *building* emit inputs when nothing observes them,
never an emit that would otherwise have been made. The coverage-breach
guard stays non-vacuous.
`CountingTracer` forwards `enabled()` to its inner tracer rather than
inheriting the `true` default. Both directions matter and both fail
silently:
- inheriting `true` over a `NoopTracer` would keep the overhead this PR
removes;
- hardcoding `false` over a live tracer would suppress the emits whose
absence `EmitGuard` reports as `ImplBug` — masking real breaches behind
expected ones.
Covered by a new regression test,
`counting_tracer_delegates_enabled_to_inner`, which asserts delegation
in both directions.
## Verification
- `cargo check -p buzz-conformance -p buzz-relay` — clean
- `cargo clippy --all-targets` — clean, zero warnings
- `cargo test -p buzz-conformance` — 6/6
- `cargo test -p buzz-relay --lib conformance` — 11/11
- `cargo test -p buzz-db --lib migration` — 7/7
- `just test-unit` (pre-push) — green
Migration-count assertions in `crates/buzz-db/src/migration.rs` were
bumped 26 → 27, with content assertions for 0027 following the existing
per-migration pattern (including a guard that it never becomes
`UNIQUE`).
## Open questions for reviewers
1. **Lock strategy.** Built *without* `CONCURRENTLY`, following
migration 0004's precedent, because sqlx runs each migration inside a
transaction and `CREATE INDEX CONCURRENTLY` cannot run in one. This
takes a brief `SHARE` lock on `channels` (blocks writes, not reads) —
small relative to `events`, but an operator preferring zero
write-blocking can pre-build it by hand and `IF NOT EXISTS` makes the
migration a no-op. I could not confirm whether sqlx 0.9 supports a `--
no-transaction` directive; if it does, that may be preferable.
2. **Diagnosis is static.** This comes from reading the source, not from
`EXPLAIN` against the live database. Worth confirming with `EXPLAIN
(ANALYZE, BUFFERS)` on the writer before/after — that also sizes the win
by revealing the real table size and row counts.
3. **Expected impact** scales with average filters-per-`REQ`, which I
did not measure. `pg_stat_statements` ordered by `total_exec_time` would
confirm this query drops off the top and show whether anything else is
scanning the same way.
Signed-off-by: Jemiah Westerman <jemiah@squareup.com>