🤖
## Summary
Workflow messages still wake the managed agent named by their
owner-signed definition, but the relay is now only a doorbell. It
carries pointers to the exact workflow revision and cause; ACP
independently verifies those signed artifacts, reconstructs trigger
input, and renders the prompt locally. Relay-authored prose is never
shown to the agent or treated as owner authority.
Before this change, a compromised relay signing key could attach
`workflow-owner` to chosen message content. After this change, ACP
grants the workflow owner principal only when the doorbell, owner-signed
kind `30620` definition, referenced step, channel, and cause all verify.
Invalid workflow-shaped deliveries are dropped rather than falling
through as ordinary relay messages.
### Doorbell behavior
- Workflow rows retain the exact kind `30620` event ID that materialized
them. Existing rows remain nullable and fail closed until the definition
is saved again.
- Relay events contain the definition event/step and one typed cause:
signed event, signed manual command, deterministic schedule slot, or
webhook cargo.
- ACP refetches and verifies the owner-signed definition. It also
refetches signed event/command causes, reconstructs their trigger
context, rechecks that the signed trigger matches the definition,
validates exact cron/interval slots, and renders `send_message` text
itself.
- Manual commands from an agent's cryptographically verified human owner
remain authorized, but command JSON deliberately no longer populates
arbitrary webhook fields. Unrelated channel members remain denied. If
parameterized manual runs are added later, those parameters must return
as explicitly untrusted cargo under the same boundary as webhooks rather
than inheriting owner authority.
- Prior workflow-step outputs cannot be independently reconstructed, so
`send_message` templates and conditions that depend on them fail closed
rather than treating unresolved placeholders as owner-authorized text.
- Mention wake targets are resolved only from the owner-signed template.
`@Name` text arriving through a signed source event or webhook field
cannot steer a different agent.
### Webhooks and replay controls
Webhooks have no signed source event. Their JSON therefore travels as
explicitly untrusted external cargo, can fill only `{{trigger.*}}` slots
declared by the owner-signed template, and is labelled as untrusted in
the prompt. ACP applies a per-definition token bucket (five immediate
turns, refilling at five per minute) and caps serialized cargo at 61,440
bytes.
Signed event, command, and schedule causes are deduplicated by
definition revision × semantic cause. Dedupe is intentionally
process-lifetime/in-memory: an ACP restart can admit a replay again.
Schedule causes must be exact authorized cron occurrences or interval
boundaries; wall-clock freshness/skew policy is deferred.
### Rollout
This is a coordinated protocol change. New ACP drops old relay workflow
messages because they lack the `doorbell-v1` pointer shape. Old ACP does
not recognize new doorbells as delegated owner messages, so owner-only
policy rejects them. Existing database rows without a definition event
ID do not fire a doorbell until the workflow is re-saved.
### Related issue
Security correction stacked directly on #2737. The owner-trigger
authorization in #5854 remains separate and will be restacked after this
correction merges into #2737.
### Testing
- `cargo test -p buzz-workflow` — 156 passed, 2 Postgres-gated ignored
- `cargo test -p buzz-acp` — 783 unit + 9 lifecycle passed
- `cargo test -p buzz-db migration --lib` — 10 passed, 6 Postgres-gated
ignored
- `cargo test -p buzz-relay --no-run`
- `cargo test -p buzz-test-client --test e2e_workflow_agent_owner
--no-run`
- `cargo clippy -p buzz-acp -p buzz-workflow -p buzz-relay -p
buzz-test-client --all-targets`
- `cargo fmt --check` and `git diff --check`
---------
Signed-off-by: Larry <8cf5a83f590ec0955b11647d1c88f796a98e088c30a492c58e0e46c3026ae7a4@buzz.block.builderlab.xyz>
Co-authored-by: Larry <8cf5a83f590ec0955b11647d1c88f796a98e088c30a492c58e0e46c3026ae7a4@buzz.block.builderlab.xyz>
## Summary
- persist stable workflow run `error_code` values separately from human
diagnostics
- expose NIP-98 authenticated, channel-authorized run history and
approval reads with stable keyset pagination
- connect Desktop to those authoritative reads and return the
relay-created run ID on trigger
- show truthful loading, failure, and pending-trace states, and do not
render approval actions from non-actionable stored hashes
## Validation
- pre-push `branch-skew`, `desktop-typecheck`, `desktop-test`,
`rust-tests`, `desktop-tauri-checks`, and `desktop-check` all passed on
`a097dbe5f`
- Desktop tests: 4,761 passed, 0 failed
- `cargo check -p buzz-relay`
- `git diff --check`
## Remaining gate
This does not claim a relay-backed Playwright workflow journey. The
browser relay bridge still routes workflow invokes through in-memory
handlers; that production-shaped acceptance gate remains follow-up work
before Workflows can leave preview.
---------
Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Co-authored-by: Carl <c7ebe626f000404285d3686e1dc74cc07cc60a9754a150041ba132e14bd3e2ec@buzz.block.builderlab.xyz>
Co-authored-by: Mongo <5c25403eab7271f9f94ddd4f2b270e8cac2c92e2c830c51877cca6ec974ffb3f@buzz.block.builderlab.xyz>
Co-authored-by: Princess Donut <68157ebd23b3897c1991015c3038658ea916200c67d3a54620b0754d1b92f6e0@buzz.block.builderlab.xyz>
## Summary
Adds a durable, operator-controlled V1 for deleting an entire Buzz
community without deleting another tenant's data.
The workflow is exposed through `buzz-admin deletions`:
- `sweep` records independent fleet storage-taxonomy observations
- `submit`, `list`, `inspect`, and `approve` manage a deletion request
- `unblock` resumes a fail-closed request after an operator records
remediation identity and reason
- `run` and `drain` execute bounded work
Requests advance through a PostgreSQL-backed state machine and stop at
`retention_pending` after logical deletion has been independently
verified across PostgreSQL, object storage, and Redis.
This PR ships the engine and CLI, not a continuously running worker or
Kubernetes packaging. For V1, a cluster/VM administrator invokes
`/usr/local/bin/buzz-admin` from the existing relay image, for example
with `kubectl exec` or an equivalent container/VM exec path.
## What whole-community V1 removes
For the target community, V1 removes:
- rows from the allowlisted community-scoped PostgreSQL catalog,
including members, profiles, authored events and bodies, DMs, reactions,
mentions, memberships, tokens, workflows, moderation, audit, feedback,
and rate-limit state
- media sidecars and upload-attribution records under
`_meta/<community>/` and `_uploads/<community>/`
- Git repository pointers under `repos/<community>/`
- Redis keys under `buzz:<community>:*`
The community row survives as a permanent tombstone, and deletion
control-plane records remain as evidence of the request, approval,
execution, and result.
## Safety model
Deletion is not a broad `DELETE CASCADE` followed by optimistic cleanup.
The destructive boundaries are durable and fail closed.
### 1. Inventory and approval
- `submit` resolves the target and freezes the schema plus summary-only
storage inventory.
- Approval is bound to the exact request, community, and frozen
inventory digest.
- Unsupported manifest versions, malformed keys inside the target's
owned prefixes, live scoped-table/write-fence coverage drift,
frozen-inventory mismatch, and approval mismatch block execution rather
than guessing. Migration and catalog revision numbers are not
authorization gates; the executor validates the live safety shape
instead.
- Storage inventory is server-side prefix scoped to exactly:
- `_meta/<community>/`
- `_uploads/<community>/`
- `repos/<community>/`
- The deletion path never lists the whole shared bucket and has no
arbitrary per-community object cap. Its listing work is proportional to
the target community's bindings, not total fleet storage.
- Fleet-wide taxonomy sweeps remain independent observability. They
report unknown writer shapes but do not gate deletion submission,
fencing, or destructive progress. Maintainers must add deletion taxonomy
coverage whenever a new community-owned object-key class is introduced;
writer-coverage tests bind the current media and Git writers to that
contract.
### 2. Quiesce, fence, and destructive freeze
- Writes continue through submission, inventory, and approval. They stop
when execution moves the target into `quiescing` and then establishes
the durable fence.
- Already-admitted external effects finish under heartbeated
serving-write leases; the exact admitted lease may renew while the
community is quiescing, but new lease acquisition is rejected. The
executor drains admitted leases before destructive work.
- Invite minting after quiescing begins fails as typed `AccessDenied`
(HTTP 503 at the relay boundary) before an invite can be persisted.
- Database triggers enforce the community write fence across the
complete catalog of community-scoped tables. Startup/readiness and
destructive execution validate that catalog so a newly added but
unfenced table cannot silently escape.
- **Named isolation assumption — fresh write snapshot.** Every writer
transaction that can reach a community-fenced relation must use
PostgreSQL `READ COMMITTED`; each guarded write therefore observes a
statement snapshot no older than acquisition of the community deletion
lock. `REPEATABLE READ` and `SERIALIZABLE` can retain a pre-fence
snapshot and are unsupported for writers. The writer pool refuses
non-`READ COMMITTED` sessions at connection setup, and both SQL fence
functions reject an explicit per-transaction isolation override with
SQLSTATE `25000`. Configuration-delivered bad isolation can surface
through SQLx as a pool-acquire timeout because every `after_connect`
attempt is rejected; the precise `community writes require READ
COMMITTED isolation` reason remains observable when the SQL guard is
reached. Read-only replica transactions are outside this assumption.
- Holding the shared advisory lock until the guarded write executes is a
separate liveness condition: under `READ COMMITTED`, releasing it early
does not permit resurrection because the trigger rechecks the fence, but
it can turn a fleet sweep into a statement-wide SQLSTATE `55000` abort.
- After the fence closes writers, storage is re-enumerated into chunked
side-table rows. Per-prefix counts and digests bind those concrete keys
to the destructive manifest.
- Manifest chunk insertion, update, and deletion are protected after
freeze. This closes the race where an unbound key could otherwise appear
after the manifest was committed.
### 3. Checkpointed destruction
- Target-owned object bindings are deleted from the frozen destructive
manifest in bounded batches with durable progress.
- The concrete key list lives in chunked side-table rows rather than one
request-row JSON value. It supports large communities, resumable
execution, and terminal cleanup.
- Missing objects are accepted as idempotent crash-window outcomes;
malformed ownership, changed evidence, and unexplained target-prefix
drift fail closed.
- PostgreSQL purging remains scoped by `community_id`, including the
guarded NIP-RS hard-delete path discovered with real Desktop kind
`30078` read-state data.
- Redis cleanup explicitly scans and `UNLINK`s only
`buzz:<community_id>:*`. Natural expiry is insufficient because some
keys, including tunnel generation counters used as fencing state, are
deliberately persistent.
### 4. Independent verification
- PostgreSQL logical absence is checked after purge.
- The three target-owned storage prefixes are freshly inventoried again
and must be empty.
- Redis requires two complete empty namespace scans.
- Only after all three stores pass does the request advance through
`logically_verified` to `retention_pending`.
## What V1 deliberately does not erase
### Shared content-addressed storage
Per-community deletion removes bindings, metadata, attribution records,
and Git pointers. It does **not** physically delete fleet-shared CAS
bytes that another community may still reference:
- media blobs and thumbnails
- Git manifests, packs, and indexes (`manifests/`, `packs/`, and `idx/`)
Safe reclamation requires a separate fleet-wide reachability and
retention GC. Unknown keys elsewhere in the shared bucket do not block
one community's deletion; malformed or unrecognized keys inside that
community's three owned prefixes still fail closed.
### External retained copies
The online logical-deletion proof does not erase object
versions/replicas, database backups/WAL, CDN copies, provider retention
copies, or observability exports. Those require their own retention and
purge controls.
### Member-only erasure
This PR erases a whole community. It does not implement the different
operation "erase one npub while preserving the community."
Removing membership or accepting NIP-09 is not member erasure. A
member-only workflow would need to find and selectively remove or redact
authored event content and pubkeys, profile data, DMs, reactions,
mentions, memberships/roles, tokens, workflows/subscriptions, upload
attribution, moderation/audit history, repository attribution, and
identity embedded in tags or JSON. It would also need explicit rules for
ownership transfer, surviving replies and thread metadata, audit-chain
integrity, immutable Git history, and shared-CAS reachability. That
requires a pubkey-level fence and selective graph rewrite; it is a
separate deletion product, not a safe extension of this whole-tenant
worker.
## In scope
- migration `0029_community_deletion.sql`: requests, approvals, leases,
manifest chunks, checkpoints, tombstones, and the universal write-fence
catalog
- durable executor leases, generations, heartbeats, retry/block state,
and resumable stage transitions
- operator-driven `sweep`, `submit`, `list`, `inspect`, `approve`,
`unblock`, `run`, and `drain` commands
- serving-path fences for database writes and external effects across
event ingest, media, Git, workflow, push, invites, mesh/tunnel, and
related paths
- target-prefix-only storage inventory, summary manifests, post-fence
destructive chunks, and bounded batch deletion
- exact community Redis namespace purge and two-pass absence
verification
- cross-community isolation, crash/resume, manifest-integrity,
writer-taxonomy, and schema/migration regressions
- desired-state `schema/schema.sql` support without requiring a SQLx
migration ledger
## Deferred / not covered
- dedicated Helm/chart worker Deployment, service account, secrets,
probes, resources, and network policy
- autonomous `buzz-admin deletions worker` poll loop and worker-only
health server
- least-privilege separation among migration, relay-serving, and
destructive execution roles
- fleet-wide shared-CAS physical GC
- backup/provider/CDN/observability retention completion
- member-only erasure
- provider-native conditional-delete improvements
- a general force-continue escape hatch; permanent safety failures
remain fail closed unless an operator remediates the cause and records
an audited `unblock`
The removed continuous-worker implementation remains deferred; no remote
follow-up branch is claimed by this PR.
## Validation
### Current PR head and repository state
Current pushed head: `359d8402ee15f049768f54156f67b953c7a7e2ed`, rebased
onto `cc9a2f783375e51a6e8d1f2f9d01d5f7e22813d1` (`origin/main` at push
time). The complete PR diff is now 47 files, 9,834 additions, and 517
deletions.
The bespoke source-scanner stack was removed to keep this PR scoped to
community deletion. Tyler/team requested the underlying fenced-write
safety behavior, not `ast-grep`,
`crates/buzz-db/tests/community_fenced_writes.rs`, its 27 fixtures, or
the new `scripts/lints/community_*.yml` rules. Those scanner-specific
files, dependencies, Hermit links, and runner wiring are absent from the
current tree. The production database write fence, startup/destructive
live-catalog validation, and deletion behavior remain.
Source validation on this exact SHA passed:
- `cargo fmt --all -- --check`
- `bash -n scripts/run-tests.sh`
- `cargo nextest run -p buzz-db --all-targets`: 102 passed, 173 skipped,
0 failed
- `cargo nextest run -p buzz-deletion --all-targets`: 10 passed, 9
skipped, 0 failed
- `cargo nextest run -p buzz-admin --all-targets`: 1 passed, 0 failed
- affected-package/all-target Clippy with warnings denied
- lockfile consistency
- Helm 3.16.4 lint and all 44 chart unit tests
- Helm region controls using that fixture: default
`BUZZ_S3_REGION=us-east-1`, explicit `eu-west-2` override, and
blank-region schema rejection
The prior Kubernetes battery below was run against
`928992237358a3294621ac0280830b77155abc04`. It remains useful evidence
for the patch-equivalent production deletion implementation, but it is
**not** claimed as exact-SHA evidence for current head
`359d8402ee15f049768f54156f67b953c7a7e2ed`; the current cleanup removes
only scanner/test/tooling infrastructure. CI restarted for the new head
after the rebase and is pending. Human review remains
`CHANGES_REQUESTED`.
### Prior-head live Kubernetes deletion and safety gates
The full program used one immutable image, real PostgreSQL, Redis,
MinIO, and a three-relay Kubernetes release:
- source: `928992237358a3294621ac0280830b77155abc04` (**prior head**)
- image: `buzz-e2e:sha-928992237358`
- immutable image digest:
`sha256:a1a204f4618ac22d9e210be5e5290645a15d79831ae30b0e44379357c8e4a895`
- evidence root:
`/tmp/buzz-e2e/20260807T033025Z-928992237358-full-gates/`
- evidence-manifest digest:
`82875c5bc9bea7370b796a7aef3457b3a1c8306c84c59e0f7388bbb5ad30e865`
Passed gates at that prior head:
- **Chart/operator region:** default `us-east-1`, explicit nondefault
propagation, blank-region schema rejection, live in-pod environment, and
an in-pod taxonomy sweep over 18 objects with zero unknown.
- **Fenced writers and lifecycle:** open-write/fence ordering;
100-attempt anti-starvation; invite, push matcher, and exhausted-reaper
bystander isolation; non-`READ-COMMITTED` rejection; manifest/tombstone
contracts; eight-failure stage block and audited `unblock`.
- **Destructive lifecycle:** submit → approve → run →
`retention_pending`; PostgreSQL tombstone and Redis/S3 verification
true; zero retries/errors; terminal reruns rejected with exit 5.
- **Fresh 10,001-object crash boundary:** exactly two chunks (10,000 +
1). The executor deleted chunk 0 from MinIO while its PostgreSQL stamp
was row-lock-blocked, was killed with `SIGKILL`, left one object and
both stamps absent, then resumed the same request under generation 2 to
zero objects and terminal state.
- **Independent dead-owner recovery:** a dedicated executor claimed
generation 1, blocked before effects, and was killed through containerd
with `SIGKILL` (no TERM cleanup). The request remained owned and
unreclaimable before lease expiry; a successor claimed generation 2
after 60 seconds and completed with two attempts and zero retries.
- **Three-pod socket isolation:** ordinary NIP-42 and joined
huddle-audio target witnesses on every replica received exact `1008 /
community deleted`; healthy-tenant witnesses on those pods remained
live; deleted-host reconnect returned HTTP 404.
- **Health/provenance:** all replicas independently returned ready and
retained the exact image digest before/after destructive runs and an
audio-enabled rolling restart; PostgreSQL, Redis, and MinIO were healthy
at close.
Instrument corrections were retained as evidence rather than counted as
product failures: a foreground PostgreSQL forward caused an initial
`PoolTimedOut`; Kubernetes pod deletion exercised graceful TERM rather
than dead-owner recovery; shell-background socket witnesses died with
their parent; and the first image build hit the corporate TLS proxy.
Detached forwarding/witnesses, containerd `SIGKILL`, and the configured
internal CA/Artifactory mirror produced the discriminating runs without
weakening product security.
### Prior-head cleanup
For the prior-head Kubernetes run, the Helm release was removed,
namespace absence was verified, run-owned Screen sessions were absent,
and that source worktree remained clean. The evidence manifest was
independently recomputed and every indexed artifact passed `shasum -a
256 -c`. The current `359d8402` source worktree is also clean after the
scanner-only cleanup and push.
---------
Signed-off-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz>
Signed-off-by: npub1dccv64krpcpse5cmkzfeh998cftungyatw3djt8jwdw6g43f7fyqzzmrf7 <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
Signed-off-by: Kalvin Chau <kalvin@block.xyz>
Signed-off-by: am <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
Signed-off-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz>
Co-authored-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz>
Co-authored-by: npub1dccv64krpcpse5cmkzfeh998cftungyatw3djt8jwdw6g43f7fyqzzmrf7 <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
Co-authored-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz>
**Category:** fix
**User Impact:** Custom emoji with valid 64-character names can now be
used as reactions without errors.
**Problem:** Buzz accepted 64-character custom emoji names during
registration, but rejected them as reactions after the required
surrounding colons made the payload 66 characters. Validation also
differed between desktop, SDK, relay, and storage boundaries.
<img width="554" height="47" alt="image"
src="https://github.com/user-attachments/assets/4013452f-210e-4dd3-9003-f45ff3b28dc8"
/>
**Solution:** Keep the product limit at 64 ASCII characters for custom
emoji names, enforce it consistently when emoji sets are registered, and
allow only valid matching custom reaction payloads up to 66 characters.
Widen the reaction projection to preserve the wrapped payload while
retaining the existing 64-character limit for ordinary reactions.
<details>
<summary>File changes</summary>
**crates/buzz-sdk/src/builders.rs**
Defines the shared custom emoji boundaries and covers accepted
64-character and rejected 65-character shortcodes.
**crates/buzz-relay/src/handlers/ingest.rs**
Validates emoji-set shortcodes and permits 66-character reactions only
when they are valid colon-wrapped custom emoji with a matching tag.
**crates/buzz-db/src/event.rs**
Adds storage regression coverage for maximum-length custom emoji
reactions.
**crates/buzz-db/src/migration.rs**
Verifies the reaction column migration is applied correctly.
**desktop/src/shared/api/customEmoji.ts**
Enforces the existing 64-character shortcode maximum during desktop
normalization and registration/import.
**desktop/src/shared/api/customEmoji.test.mjs**
Covers the desktop shortcode boundary.
**migrations/0027_long_reaction_payloads.sql**
Widens stored reaction payloads to 66 characters for the two required
surrounding colons.
**schema/schema.sql**
Keeps the desired schema aligned with the migration.
</details>
## Reproduction Steps
1. Register or import a custom emoji whose ASCII shortcode is exactly 64
characters.
2. Select that emoji as a reaction to a message.
3. Confirm the reaction publishes, persists, and renders without an
error.
4. Attempt to register a 65-character shortcode and confirm it is
rejected.
5. Publish an ordinary or malformed reaction over 64 characters and
confirm the relay rejects it.
## Verification
- `cargo test -p buzz-sdk`: 243 passed
- `cargo test -p buzz-db`: 94 passed, 152 Postgres-required tests
ignored
- `pnpm test` in `desktop`: 3,859 passed
- `cargo test -p buzz-relay`: 795 passed, 9 existing
Postgres-unavailable failures, 35 ignored; new reaction boundary tests
pass directly
- `cargo fmt --all -- --check`
- `git diff --check`
Originating Buzz channel: `f2ec9671-d78e-4cde-894c-9f4c458c7f1f`
---------
Signed-off-by: Taylor Ho <taylorkmho@gmail.com>
Co-authored-by: npub1223z34hd7vtwc6qj4s7flsxkj644nlre2nthu7lrrmkumhu3xddsrx9r6w <52a228d6edf316ec6812ac3c9fc0d696ab59fc7954d77e7be31eedcddf91335b@buzz.block.builderlab.xyz>
## Problem
`SELECT id, community_id FROM channels WHERE id = ANY($1) AND deleted_at
IS NULL` is the top **Load by waits (AAS)** on the Buzz Postgres writer.
Two independent causes compound, and both are fixed here.
### 1. No index can serve it
`channels` is `PRIMARY KEY (community_id, id)`, and every secondary
index leads with `community_id`:
| Index | Columns |
|---|---|
| *(primary key)* | `(community_id, id)` |
| `idx_channels_nip29_group` | `(community_id, nip29_group_id)` |
| `idx_channels_dm_hash` | `(community_id, participant_hash)` |
| `idx_channels_community_type` | `(community_id, channel_type)` |
| `idx_channels_community_visibility` | `(community_id, visibility)` |
| `idx_channels_created_by` | `(community_id, created_by)` |
| `idx_channels_ttl_expiry` | `(ttl_deadline)` *(partial)* |
The two tenant-independent lookups carry **no `community_id` predicate**
— deliberately:
- `Db::communities_of_channels` — `WHERE id = ANY($1) AND deleted_at IS
NULL`
- `Db::community_of_channel` — `WHERE id = $1 AND deleted_at IS NULL`
That independence is load-bearing, not an oversight: projecting a row's
*true* owning community regardless of the fetch query's `WHERE` clause
is what makes `Inv_NonInterference` non-vacuous. If the fetch ever
dropped its tenant scoping, this lookup would still report the real
label and the checker would catch the mismatch.
But a composite btree is only usable when its leading column is
constrained, so neither query can use the primary key, and nothing else
leads with `id`. **Both sequentially scan `channels` on every call.**
### 2. In production the result is discarded
Both call sites feed `record_read_message_rows` /
`record_read_by_id_rows`, which call `tracer.record(...)`. Production
binds `NoopTracer` (`crates/buzz-relay/src/state.rs`), whose `record`
body is empty.
The existing guard tests `trace_state`, which is `Some` for every
well-formed request — it only goes `None` on malformed pubkey bytes. So
the scan ran on the hot read path and its output was dropped. This is
the classic eager-argument bug: `log.debug("..." + expensiveCall())`
with no `isDebugEnabled()` check.
### 3. Multiplied per filter
The non-search call site sits **inside the phase-3 per-filter loop**, so
a `REQ` carrying N filters performed N sequential scans of `channels`
before responding.
## Changes
**`Tracer::enabled()`** — a capability check on the trait (the
`isDebugEnabled()` of this seam), defaulting to `true`. `NoopTracer`
overrides it to `false`, and both emitters in `req.rs` now gate on it,
skipping the trace-only DB read entirely in production.
**`migrations/0027_channels_id_lookup_index.sql`**
```sql
CREATE INDEX IF NOT EXISTS idx_channels_id_live
ON channels (id) INCLUDE (community_id)
WHERE deleted_at IS NULL;
```
- `INCLUDE (community_id)` — both queries select exactly `(id,
community_id)`, so this is covering and can be served index-only.
- Partial on `deleted_at IS NULL` — matches both predicates exactly,
excludes soft-deleted history, and lets Postgres skip the recheck.
- **Not `UNIQUE`.** `id` alone is *not* unique in this table —
`command_executor.rs` documents that `community_of_channel(channel_id)`
is ambiguous because the same channel id can appear under more than one
community. A unique index would encode a false constraint and fail to
build on any database already holding such a pair.
Worth keeping the index even though fix#1 removes the production
caller: it still runs under conformance, and `community_of_channel` has
the same problem on its own paths.
**`schema/schema.sql`** — mirrored, since a test asserts desired-state
parity.
## Conformance is unchanged
This is the part worth reviewing closely. Under a real tracer
`enabled()` returns `true` and **every emit happens exactly as before**
— the gate only skips *building* emit inputs when nothing observes them,
never an emit that would otherwise have been made. The coverage-breach
guard stays non-vacuous.
`CountingTracer` forwards `enabled()` to its inner tracer rather than
inheriting the `true` default. Both directions matter and both fail
silently:
- inheriting `true` over a `NoopTracer` would keep the overhead this PR
removes;
- hardcoding `false` over a live tracer would suppress the emits whose
absence `EmitGuard` reports as `ImplBug` — masking real breaches behind
expected ones.
Covered by a new regression test,
`counting_tracer_delegates_enabled_to_inner`, which asserts delegation
in both directions.
## Verification
- `cargo check -p buzz-conformance -p buzz-relay` — clean
- `cargo clippy --all-targets` — clean, zero warnings
- `cargo test -p buzz-conformance` — 6/6
- `cargo test -p buzz-relay --lib conformance` — 11/11
- `cargo test -p buzz-db --lib migration` — 7/7
- `just test-unit` (pre-push) — green
Migration-count assertions in `crates/buzz-db/src/migration.rs` were
bumped 26 → 27, with content assertions for 0027 following the existing
per-migration pattern (including a guard that it never becomes
`UNIQUE`).
## Open questions for reviewers
1. **Lock strategy.** Built *without* `CONCURRENTLY`, following
migration 0004's precedent, because sqlx runs each migration inside a
transaction and `CREATE INDEX CONCURRENTLY` cannot run in one. This
takes a brief `SHARE` lock on `channels` (blocks writes, not reads) —
small relative to `events`, but an operator preferring zero
write-blocking can pre-build it by hand and `IF NOT EXISTS` makes the
migration a no-op. I could not confirm whether sqlx 0.9 supports a `--
no-transaction` directive; if it does, that may be preferable.
2. **Diagnosis is static.** This comes from reading the source, not from
`EXPLAIN` against the live database. Worth confirming with `EXPLAIN
(ANALYZE, BUFFERS)` on the writer before/after — that also sizes the win
by revealing the real table size and row counts.
3. **Expected impact** scales with average filters-per-`REQ`, which I
did not measure. `pg_stat_statements` ordered by `total_exec_time` would
confirm this query drops off the top and show whether anything else is
scanning the same way.
Signed-off-by: Jemiah Westerman <jemiah@squareup.com>
## Summary
Expands read-replica usage on the relay per the Rev 2 design (thread
39d1e174 in #buzz-read-only-replica-usage): replaces the
Aurora-incompatible WAL-LSN read-side fence observation with a portable
**heartbeat token**, makes the freshness proof **snapshot-local to the
serving reader session**, and adds a default-off bounded-staleness gate
for head fetches.
### Probe (writer side, unchanged ordering)
The ordered writer scan is kept verbatim — `S = clock_timestamp()` →
masked-visibility `pg_stat_activity` oldest-xact scan — and now ends by
committing a heartbeat token **last** on the same pinned connection
(single-row `UPDATE replica_heartbeat ... RETURNING token, epoch`,
migration `0026`). The single-row update serializes all pods' probes, so
tokens are globally commit-ordered; the three-bucket completeness
argument carries over unchanged. Ring retains `(token, committed_at,
fence_wall)`; epoch change resets the ring; a same-epoch token
regression (restore adversary) clears the ring **and rotates the epoch
on the writer** so pre-rewind readers fail the epoch check. Cadence 1s.
### Routing (snapshot-local proof)
Every routed read opens `BEGIN ISOLATION LEVEL REPEATABLE READ, READ
ONLY` on a reader session and observes the heartbeat as the
transaction's **first statement** — the proof's snapshot is exactly the
snapshot the page, participants batch, and bridge aux closure read from
(`ReadSession` carries the open transaction; drop = rollback).
Fail-closed everywhere: begin failure, missing heartbeat row, epoch
mismatch, token below the ring, over-budget entry → writer.
- **Predicate B (cursor pages, default-on):** same completeness math as
the existing fence — cursor timestamp must be ≤ the proved wall; thread
candidate-terminal and above-wall pages re-run on the writer.
- **Predicate A (head fetches, default-OFF):** gated by
`BUZZ_REPLICA_HEAD_MAX_AGE_SECS` (0 = off, clamped to fence staleness).
Bounded-stale head semantics are an explicit product decision — **do not
enable anywhere without Tyler's backdated-event-semantics acceptance.**
### Observability
`buzz_db_route_decision{path, decision, reason}` across all five paths,
`buzz_db_replica_heartbeat_age_seconds` gauge, and per-decision debug
logs carrying the proved token plus backend identity: `addr:port pid=N`,
prefixed with `aurora_db_instance_identifier()` when the endpoint
supports it (probed once per process on an autocommit checkout; SQLSTATE
42883 caches a definitive false; identity is evidence, never a routing
gate).
## Review & verification
- **Wren:** full review 9/9/9 at `5f81b10e5`; identity delta re-review
approved at exact head `fedb46368` (90/90 unit, clippy `-D warnings`,
fmt independently reproduced).
- **Max:** local E2E **PASS** at `5f81b10e5` — isolated PG17 writer +
two streaming standbys behind HAProxy; paused-reader legs split exactly
as designed (6 replica/fresh + 6 writer/stale), recovery clean, RR
snapshot hardening observed in runtime logs. Evidence:
`WORK_LOGS/2026-07-28_REPLICA_HEARTBEAT_REPLACEMENT_SHA_E2E.md`.
- **My gates at head (same shell):** buzz-db 90 unit + 143
Postgres-gated green (scratch DBs); buzz-relay 767/768 — lone red is the
pre-existing `mesh_demo` flake, red on base; clippy `-D warnings` + fmt
clean.
- Key regression tests:
`routed_request_holds_one_snapshot_across_page_and_aux`
(mutation-verified both directions), same-epoch rotation,
capability-probe negative, head-gate truth table, divergent-fixture
routing suite.
## Post-merge plan
Merge publishes the immutable `sha-*` main image → bb-public PR pins it
→ ArgoCD sync → Max runs the production cursor-routing canary on Aurora
(positive identity branch proven live). Head gate stays off. Client
`since`-widening ships separately.
---
## Update — full read routing (Rev 6, commit `9fa3c9c0b`)
Extends routing to the remaining read seams per
`PLANS/REPLICA_FULL_READ_ROUTING_DESIGN.md` Rev 6 (thread cf5deba4 in
#buzz-read-only-replica-usage).
### New routed seams — ALL deploy-default dark
The new seams (`query_events_routed`, `query_events_routed_bounded`,
`count_events_routed`, `get_events_by_ids_routed`,
`query_feed_{mentions,needs_action,activity}_routed`) are
**Bounded-only** and gated on `BUZZ_REPLICA_READ_MAX_AGE_MS`: unset ⇒
every new seam records `writer/disabled` and merging is a no-op. COUNT
and feed/by-ids never take the covered arm (deletion visibility: covered
bounds insert-completeness only). **Note:** the pre-existing cursor
paths (channel windows, thread pages) are *not* gated by this env var —
they route at B=0 today and that status quo is unchanged.
### Reader pool (D4/D5)
- Lazy pool (`connect_lazy`, `min_connections(0)`): reader-down at boot
can't crash the relay; a warn-only boot ping is the only boot-time
visibility, and it primes the Aurora identity capability cache. Priming
is an optimization only — the routed path itself spends a single acquire
budget regardless (see the single-checkout fix below), because a boot
ping that *fails* is correlated with exactly the reader-unavailable case
the budget bound exists for.
- `READER_ACQUIRE_TIMEOUT` = 150ms; a miss fails closed to the writer
with reason **`reader_acquire_timeout`** — named for the mechanism, not
a diagnosis. The budget includes cold connects and sqlx's `size` counts
in-flight dials, so this metric alone does not distinguish contention
from slow connection establishment (see `proved_reader` doc-comment for
the runbook guidance; the pool gauges are 10s samples and are for
capacity planning).
- `BUZZ_DB_READ_POOL_SIZE` sizes the reader independently (invalid/0
inherits writer sizing); `read_pool_stats().max` reports the reader's
own ceiling.
### Community isolation (formal-model question)
Isolation is structural — an explicit `community_id = $n` predicate
compiled into every query builder; no RLS, no session-GUC tenant state
(zero `CREATE POLICY` across migrations). The `_on` variants reuse the
exact same builders with only the executor swapped. Proven by a
seven-seam two-community divergent-fixture test
(`routed_reads_are_confined_to_the_requested_community`),
mutation-tested by Dawn: all four single-predicate stubs killed; the
mention-join feeds are defended in depth (three independent predicates)
so only complete removal leaks there.
### Accepted limitation
D6: client-side staleness on bounded reads (up to `_MS`) is accepted
product behavior when the gate is enabled; gate stays off at merge.
## Update — single acquire budget per routed read (commit `dd26caa9f`)
Max found (and Dawn independently reproduced, 302–330ms measured) that
the boot-unavailable cold path spent **two** stacked
`READER_ACQUIRE_TIMEOUT` budgets: the Aurora capability probe did its
own `pool.acquire()` before `begin_with` acquired again. Fixed by
acquiring **once** per routed read — the capability probe runs on the
held connection (`reader_aurora_capability_on`) and the read-only
`REPEATABLE READ` transaction begins on that same connection
(`Transaction::begin` accepts a `PoolConnection` at `'static` via sqlx's
`MaybePoolConnection`). Reason codes unchanged; capability still never
negatively cached. Measured routed fallback: ~150ms (one budget).
Ships with a PG-gated regression fixture
(`routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold`,
authored by Dawn): size-1 reader saturated, capability cache asserted
cold, routes through `count_events_routed`, asserts writer answer +
one-budget elapsed + `writer/reader_acquire_timeout` label
(mutation-tested: `pool_busy` and `reader_validation_error` mutants both
killed). It fails at 330ms on the previous commit and passes on this
one.
### Verification at `dd26caa9f` (pinned toolchain rustc 1.95.0 via repo
`bin/`; ambient rustc 1.89 fails sqlx resolution — always prepend
`bin/`)
- buzz-db: 94 unit + **150/151** PG-gated serial (`--test-threads=1`;
suite is not parallel-safe against one Postgres). The 1 red is
`test_usage_metrics_lock_has_single_owner_and_releases_on_drop` — a
**pre-existing** same-key-same-database advisory-lock collision with any
live relay pointed at the shared `buzz` DB (the relay re-arms
`USAGE_METRICS_LOCK_KEY` on a 300s tick). It passes on a private scratch
DB with the relay still running — verified this run. Remedy is a scratch
`TEST_DATABASE_URL`, **not** terminating lock holders: inspect
`pg_locks`/`pg_stat_activity` and resolve the owning process first; an
idle advisory holder may be a live dev relay. Pre-existing, not a #3268
blocker — filed as #3619.
- buzz-relay: 773/773; clippy `-D warnings` + fmt clean. (`mesh_demo`
echo test is a known pre-existing Redis-dependent flake — Dawn confirmed
it fails identically on the base commit, untouched by this PR.)
- Dawn: independent re-verification at `dd26caa9f` — byte-identical diff
application confirmed, suite reproduced, and fixture proven to still
discriminate (fails at 320ms with only the production fix reverted).
Gate clear. Max: independent rig pass at `dd26caa9f` — 94/94 unit,
**15/15 routing/fallback matrix** (cursor behavior, default-dark seams,
seven-seam confinement, one-snapshot, dead-reader/no-URL fallback,
hard-delete fail-closed, reader max), 773/773 relay, private-DB lock
control. Gate clear; recommends merge + staged bb-block rollout
(no-reader-URL no-op → cursor-only → observe `buzz_db_route_decision` →
conservative nonzero `BUZZ_REPLICA_READ_MAX_AGE_MS`; rollback is
config-only).
## Known gap — CI does not run the PG-backed fixtures (#3622)
Dawn found post-sign-off (confirmed by Max and Eva at `dd26caa9f`):
**every Postgres-backed fixture in this PR is `#[ignore]`d and no CI job
selects it.** `Unit Tests` runs `cargo nextest run -p buzz-db --lib`
(`Justfile:279-285`, ignored tests excluded); the only `--run-ignored
ignored-only` invocations (`ci.yml:690`, `:702`) filter to
`relay_invite::tests`. So the one-budget regression, the seven-seam
isolation fixture, and the fence/floor-guard/fallback tests have zero
automated execution — the verification record above is exhaustive but
manual, at this exact SHA.
Not introduced by this PR (the `#[ignore]` + narrow-filter pattern
predates it; 34 ignored tests total) and not a merge blocker: all new
seams are dark until `BUZZ_REPLICA_READ_MAX_AGE_MS` is set. But it
changes the rollout gate — **do not set a nonzero
`BUZZ_REPLICA_READ_MAX_AGE_MS` in bb-block until CI enforces these
fixtures.** Filed as #3622 (explicit CI selection against the existing
backend-integration Postgres archive; ordering note: #3619 must land
first or be excluded, since widening the filter would select the
colliding usage-metrics lock test).
## Live redteam results (Max, real Helm + PG17 streaming replication at
`dd26caa9f`) — #3643, #3644
Max deployed this exact SHA via the repo chart against a physical
writer/standby pair and attacked it. **The PR's fenced routing passed
every fault**: healthy baseline routes proven on the real standby;
paused WAL replay moved head/cursor reads to `writer/stale`; B=0 moved
routed reads to `writer/disabled`; reader-URL removal rolled back
cleanly.
The redteam also found a **pre-existing** production risk this PR does
not create and does not fix: NIP-50 search builds its own eager,
unfenced pool straight from `READ_DATABASE_URL` (`main.rs:389-402`,
introduced by #2084, present in the deployed bb-block `sha-dd222a5` and
bb-public `sha-22be8bb` images, and both production ESOs already
template `READ_DATABASE_URL`). Measured: ~30s search stall on reader
loss, acknowledged-but-unsearchable writes under replica lag (unaffected
by `BUZZ_REPLICA_READ_MAX_AGE_MS`), fatal eager connect blocking pod
startup when the reader is down, and `replica=true` FTS even at B=0.
Filed as **#3643** (fix: pin FTS to the writer, or fence it like the
routed seams) with **#3644** for the writer-pointing-reader-URL
telemetry gap. Aurora's cluster-ro writer-fallback DNS softens the
outage modes in prod but not the lag mode.
**Revised ladder consequence:** step 2 ("add `READ_DATABASE_URL`, budget
unset") is NOT a no-op with today's binary — it hands FTS an unfenced
replica pool. Rollout order is now: merge (still a true no-op —
bb-block/bb-public already run the unfenced-FTS code and this PR only
adds dark seams) → fix#3643 → CI enforcement #3622 → then reader URL +
budget per the original ladder, re-running Max's three deployment tests
(reader-down latency, reader-down cold boot, paused-replay
read-your-own-write search).
## Activation gate (consolidated) — merge is clear; ALL of the below
precede any nonzero `BUZZ_REPLICA_READ_MAX_AGE_MS`
Six independent verification passes at `dd26caa9f` (Dawn ×2 incl.
byte-level + defang check, Max matrix + live Helm redteam, Wren full
CRUD regression + greenfield 0026 + 151/151 PG, Eva). Every induced
fault was either handled correctly by this PR's machinery or traced to
pre-existing main code. Remaining work gates **activation, not merge**:
1. **#3643** — unfenced NIP-50 FTS pool (pre-existing #2084; live in
prod today; also explains the blackhole search hang Wren observed — the
routed path was measured bounded at 151-152ms against a silent endpoint,
sqlx `inner.rs:252-255`).
2. **#3651** — reader `statement_timeout`: statements after acquire are
unbounded; mid-transaction blackhole reproduced hanging ≥15s (Dawn).
Small fix via `.after_connect` on the reader pool.
3. **#3622** — CI enforcement of the PG fixtures (with #3619 ordering).
4. **Recovery-conflict cancellation live test** — standby cancels a
routed read mid-snapshot → complete writer page, never partial (#3651's
fix also bounds the cancellation-never-arrives shape).
5. **DDL replication lag test** — migration on writer + paused replay +
budget on → fallback, not client-visible SQL errors.
6. **Reader-tx soak** — 30-60min mixed load; no idle-in-tx accumulation
on the standby.
Then Max's three deployment tests (reader-down latency, reader-down cold
boot, paused-replay read-your-own-write search) re-run on the fixed
binary before the first nonzero budget on bb-block.
---------
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
## Context
`buzz users get --name Honey` searches relay-wide profiles and returns
up to 100 identically named results without verified ownership metadata.
An agent resolving “my Honey” cannot distinguish the requesting human's
agent from another owner's agent, and the owned match can be excluded by
the result limit. This caused the wrong Honey and Bumble pubkeys to be
added to a channel.
## Summary
This bug fix makes personal-agent resolution owner-aware. Callers can
filter profiles by a verified owner identity before result limits are
applied, and all profile results expose enough ownership context to
diagnose duplicate names.
## Changes
- Adds `buzz users get --owner me|<hex>|<npub>` for name and pubkey
lookups.
- Resolves `me` to the NIP-OA owner identity when the CLI runs as an
agent.
- Filters profiles by the relay's verified `agent_owner_pubkey`
relationship before applying the result limit.
- Returns `owner_pubkey`, `owner_display_name`, and client-relative
`owned_by_me` in compact and JSON output.
- Returns an empty result when no owned profile matches instead of
removing the ownership constraint.
- Rejects malformed owner values instead of silently running an unscoped
query; explicit `null` remains equivalent to no owner filter for
ordinary CLI lookups.
- Rejects owner constraints on specialized channel-window, feed, and
thread filters that cannot enforce author filtering.
- Scopes owner filtering and enrichment to the active community.
- Adds a partial `(community_id, agent_owner_pubkey)` index for owner
lookups.
- Documents the safe `users get --name Honey --owner me` lookup.
## Reviewer-reproducible examples
The relay-backed test creates two same-name agents with different
verified owners, queries through the HTTP `/query` route, verifies only
the selected owner's agent is returned with verified owner metadata, and
verifies a missing owner returns `[]`.
```bash
cargo test -p buzz-relay query_agent_owner_returns_only_verified_owner_matches --lib -- --ignored
```
The owner/author intersection and unsupported-specialized-filter
contracts also have infrastructure-free relay tests:
```bash
cargo test -p buzz-relay agent_owner --lib
```
The CLI surface is visible in command help:
```bash
cargo run -q -p buzz-cli -- users get --help | grep -- --owner
```
```text
--owner <OWNER> Filter agents by verified owner (`me`, 64-char hex, or npub)
```
## Validation
- `cargo test -p buzz-cli` (252 passed)
- `cargo test -p buzz-db` (84 passed, 122 infrastructure tests ignored)
- `cargo test -p buzz-relay --lib` (owner-filter tests pass; the full
local suite is blocked by unrelated Postgres pool timeouts in
media/admin tests)
- `cargo test -p buzz-relay
query_agent_owner_returns_only_verified_owner_matches --lib --
--ignored` (passed)
- `cargo check --workspace --all-targets`
- `cargo fmt --all -- --check`
- Pre-push Rust, Desktop, and Desktop Tauri suites passed
- Pre-push mobile suite could not start because `flutter` is not
installed
- `pnpm check:file-sizes` (passed after rebasing onto current `main`)
---------
Signed-off-by: npub1qye6rec0htgg3np8yt6plpyyg8cyffaq66emt3kmk05eylckkzhq0hnf2k <0133a1e70fbad088cc2722f41f848441f044a7a0d6b3b5c6dbb3e9927f16b0ae@buzz.block.builderlab.xyz>
Co-authored-by: npub1qye6rec0htgg3np8yt6plpyyg8cyffaq66emt3kmk05eylckkzhq0hnf2k <0133a1e70fbad088cc2722f41f848441f044a7a0d6b3b5c6dbb3e9927f16b0ae@buzz.block.builderlab.xyz>
