Files
buzz/crates/buzz-db/src/feed.rs
T
63496cc1d4 feat(replica): portable heartbeat-token fence with snapshot-local reader routing (#3268)
## Summary

Expands read-replica usage on the relay per the Rev 2 design (thread
39d1e174 in #buzz-read-only-replica-usage): replaces the
Aurora-incompatible WAL-LSN read-side fence observation with a portable
**heartbeat token**, makes the freshness proof **snapshot-local to the
serving reader session**, and adds a default-off bounded-staleness gate
for head fetches.

### Probe (writer side, unchanged ordering)
The ordered writer scan is kept verbatim — `S = clock_timestamp()` →
masked-visibility `pg_stat_activity` oldest-xact scan — and now ends by
committing a heartbeat token **last** on the same pinned connection
(single-row `UPDATE replica_heartbeat ... RETURNING token, epoch`,
migration `0026`). The single-row update serializes all pods' probes, so
tokens are globally commit-ordered; the three-bucket completeness
argument carries over unchanged. Ring retains `(token, committed_at,
fence_wall)`; epoch change resets the ring; a same-epoch token
regression (restore adversary) clears the ring **and rotates the epoch
on the writer** so pre-rewind readers fail the epoch check. Cadence 1s.

### Routing (snapshot-local proof)
Every routed read opens `BEGIN ISOLATION LEVEL REPEATABLE READ, READ
ONLY` on a reader session and observes the heartbeat as the
transaction's **first statement** — the proof's snapshot is exactly the
snapshot the page, participants batch, and bridge aux closure read from
(`ReadSession` carries the open transaction; drop = rollback).
Fail-closed everywhere: begin failure, missing heartbeat row, epoch
mismatch, token below the ring, over-budget entry → writer.

- **Predicate B (cursor pages, default-on):** same completeness math as
the existing fence — cursor timestamp must be ≤ the proved wall; thread
candidate-terminal and above-wall pages re-run on the writer.
- **Predicate A (head fetches, default-OFF):** gated by
`BUZZ_REPLICA_HEAD_MAX_AGE_SECS` (0 = off, clamped to fence staleness).
Bounded-stale head semantics are an explicit product decision — **do not
enable anywhere without Tyler's backdated-event-semantics acceptance.**

### Observability
`buzz_db_route_decision{path, decision, reason}` across all five paths,
`buzz_db_replica_heartbeat_age_seconds` gauge, and per-decision debug
logs carrying the proved token plus backend identity: `addr:port pid=N`,
prefixed with `aurora_db_instance_identifier()` when the endpoint
supports it (probed once per process on an autocommit checkout; SQLSTATE
42883 caches a definitive false; identity is evidence, never a routing
gate).

## Review & verification

- **Wren:** full review 9/9/9 at `5f81b10e5`; identity delta re-review
approved at exact head `fedb46368` (90/90 unit, clippy `-D warnings`,
fmt independently reproduced).
- **Max:** local E2E **PASS** at `5f81b10e5` — isolated PG17 writer +
two streaming standbys behind HAProxy; paused-reader legs split exactly
as designed (6 replica/fresh + 6 writer/stale), recovery clean, RR
snapshot hardening observed in runtime logs. Evidence:
`WORK_LOGS/2026-07-28_REPLICA_HEARTBEAT_REPLACEMENT_SHA_E2E.md`.
- **My gates at head (same shell):** buzz-db 90 unit + 143
Postgres-gated green (scratch DBs); buzz-relay 767/768 — lone red is the
pre-existing `mesh_demo` flake, red on base; clippy `-D warnings` + fmt
clean.
- Key regression tests:
`routed_request_holds_one_snapshot_across_page_and_aux`
(mutation-verified both directions), same-epoch rotation,
capability-probe negative, head-gate truth table, divergent-fixture
routing suite.

## Post-merge plan

Merge publishes the immutable `sha-*` main image → bb-public PR pins it
→ ArgoCD sync → Max runs the production cursor-routing canary on Aurora
(positive identity branch proven live). Head gate stays off. Client
`since`-widening ships separately.

---

## Update — full read routing (Rev 6, commit `9fa3c9c0b`)

Extends routing to the remaining read seams per
`PLANS/REPLICA_FULL_READ_ROUTING_DESIGN.md` Rev 6 (thread cf5deba4 in
#buzz-read-only-replica-usage).

### New routed seams — ALL deploy-default dark
The new seams (`query_events_routed`, `query_events_routed_bounded`,
`count_events_routed`, `get_events_by_ids_routed`,
`query_feed_{mentions,needs_action,activity}_routed`) are
**Bounded-only** and gated on `BUZZ_REPLICA_READ_MAX_AGE_MS`: unset ⇒
every new seam records `writer/disabled` and merging is a no-op. COUNT
and feed/by-ids never take the covered arm (deletion visibility: covered
bounds insert-completeness only). **Note:** the pre-existing cursor
paths (channel windows, thread pages) are *not* gated by this env var —
they route at B=0 today and that status quo is unchanged.

### Reader pool (D4/D5)
- Lazy pool (`connect_lazy`, `min_connections(0)`): reader-down at boot
can't crash the relay; a warn-only boot ping is the only boot-time
visibility, and it primes the Aurora identity capability cache. Priming
is an optimization only — the routed path itself spends a single acquire
budget regardless (see the single-checkout fix below), because a boot
ping that *fails* is correlated with exactly the reader-unavailable case
the budget bound exists for.
- `READER_ACQUIRE_TIMEOUT` = 150ms; a miss fails closed to the writer
with reason **`reader_acquire_timeout`** — named for the mechanism, not
a diagnosis. The budget includes cold connects and sqlx's `size` counts
in-flight dials, so this metric alone does not distinguish contention
from slow connection establishment (see `proved_reader` doc-comment for
the runbook guidance; the pool gauges are 10s samples and are for
capacity planning).
- `BUZZ_DB_READ_POOL_SIZE` sizes the reader independently (invalid/0
inherits writer sizing); `read_pool_stats().max` reports the reader's
own ceiling.

### Community isolation (formal-model question)
Isolation is structural — an explicit `community_id = $n` predicate
compiled into every query builder; no RLS, no session-GUC tenant state
(zero `CREATE POLICY` across migrations). The `_on` variants reuse the
exact same builders with only the executor swapped. Proven by a
seven-seam two-community divergent-fixture test
(`routed_reads_are_confined_to_the_requested_community`),
mutation-tested by Dawn: all four single-predicate stubs killed; the
mention-join feeds are defended in depth (three independent predicates)
so only complete removal leaks there.

### Accepted limitation
D6: client-side staleness on bounded reads (up to `_MS`) is accepted
product behavior when the gate is enabled; gate stays off at merge.

## Update — single acquire budget per routed read (commit `dd26caa9f`)

Max found (and Dawn independently reproduced, 302–330ms measured) that
the boot-unavailable cold path spent **two** stacked
`READER_ACQUIRE_TIMEOUT` budgets: the Aurora capability probe did its
own `pool.acquire()` before `begin_with` acquired again. Fixed by
acquiring **once** per routed read — the capability probe runs on the
held connection (`reader_aurora_capability_on`) and the read-only
`REPEATABLE READ` transaction begins on that same connection
(`Transaction::begin` accepts a `PoolConnection` at `'static` via sqlx's
`MaybePoolConnection`). Reason codes unchanged; capability still never
negatively cached. Measured routed fallback: ~150ms (one budget).

Ships with a PG-gated regression fixture
(`routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold`,
authored by Dawn): size-1 reader saturated, capability cache asserted
cold, routes through `count_events_routed`, asserts writer answer +
one-budget elapsed + `writer/reader_acquire_timeout` label
(mutation-tested: `pool_busy` and `reader_validation_error` mutants both
killed). It fails at 330ms on the previous commit and passes on this
one.

### Verification at `dd26caa9f` (pinned toolchain rustc 1.95.0 via repo
`bin/`; ambient rustc 1.89 fails sqlx resolution — always prepend
`bin/`)
- buzz-db: 94 unit + **150/151** PG-gated serial (`--test-threads=1`;
suite is not parallel-safe against one Postgres). The 1 red is
`test_usage_metrics_lock_has_single_owner_and_releases_on_drop` — a
**pre-existing** same-key-same-database advisory-lock collision with any
live relay pointed at the shared `buzz` DB (the relay re-arms
`USAGE_METRICS_LOCK_KEY` on a 300s tick). It passes on a private scratch
DB with the relay still running — verified this run. Remedy is a scratch
`TEST_DATABASE_URL`, **not** terminating lock holders: inspect
`pg_locks`/`pg_stat_activity` and resolve the owning process first; an
idle advisory holder may be a live dev relay. Pre-existing, not a #3268
blocker — filed as #3619.
- buzz-relay: 773/773; clippy `-D warnings` + fmt clean. (`mesh_demo`
echo test is a known pre-existing Redis-dependent flake — Dawn confirmed
it fails identically on the base commit, untouched by this PR.)
- Dawn: independent re-verification at `dd26caa9f` — byte-identical diff
application confirmed, suite reproduced, and fixture proven to still
discriminate (fails at 320ms with only the production fix reverted).
Gate clear. Max: independent rig pass at `dd26caa9f` — 94/94 unit,
**15/15 routing/fallback matrix** (cursor behavior, default-dark seams,
seven-seam confinement, one-snapshot, dead-reader/no-URL fallback,
hard-delete fail-closed, reader max), 773/773 relay, private-DB lock
control. Gate clear; recommends merge + staged bb-block rollout
(no-reader-URL no-op → cursor-only → observe `buzz_db_route_decision` →
conservative nonzero `BUZZ_REPLICA_READ_MAX_AGE_MS`; rollback is
config-only).

## Known gap — CI does not run the PG-backed fixtures (#3622)

Dawn found post-sign-off (confirmed by Max and Eva at `dd26caa9f`):
**every Postgres-backed fixture in this PR is `#[ignore]`d and no CI job
selects it.** `Unit Tests` runs `cargo nextest run -p buzz-db --lib`
(`Justfile:279-285`, ignored tests excluded); the only `--run-ignored
ignored-only` invocations (`ci.yml:690`, `:702`) filter to
`relay_invite::tests`. So the one-budget regression, the seven-seam
isolation fixture, and the fence/floor-guard/fallback tests have zero
automated execution — the verification record above is exhaustive but
manual, at this exact SHA.

Not introduced by this PR (the `#[ignore]` + narrow-filter pattern
predates it; 34 ignored tests total) and not a merge blocker: all new
seams are dark until `BUZZ_REPLICA_READ_MAX_AGE_MS` is set. But it
changes the rollout gate — **do not set a nonzero
`BUZZ_REPLICA_READ_MAX_AGE_MS` in bb-block until CI enforces these
fixtures.** Filed as #3622 (explicit CI selection against the existing
backend-integration Postgres archive; ordering note: #3619 must land
first or be excluded, since widening the filter would select the
colliding usage-metrics lock test).
## Live redteam results (Max, real Helm + PG17 streaming replication at
`dd26caa9f`) — #3643, #3644

Max deployed this exact SHA via the repo chart against a physical
writer/standby pair and attacked it. **The PR's fenced routing passed
every fault**: healthy baseline routes proven on the real standby;
paused WAL replay moved head/cursor reads to `writer/stale`; B=0 moved
routed reads to `writer/disabled`; reader-URL removal rolled back
cleanly.

The redteam also found a **pre-existing** production risk this PR does
not create and does not fix: NIP-50 search builds its own eager,
unfenced pool straight from `READ_DATABASE_URL` (`main.rs:389-402`,
introduced by #2084, present in the deployed bb-block `sha-dd222a5` and
bb-public `sha-22be8bb` images, and both production ESOs already
template `READ_DATABASE_URL`). Measured: ~30s search stall on reader
loss, acknowledged-but-unsearchable writes under replica lag (unaffected
by `BUZZ_REPLICA_READ_MAX_AGE_MS`), fatal eager connect blocking pod
startup when the reader is down, and `replica=true` FTS even at B=0.
Filed as **#3643** (fix: pin FTS to the writer, or fence it like the
routed seams) with **#3644** for the writer-pointing-reader-URL
telemetry gap. Aurora's cluster-ro writer-fallback DNS softens the
outage modes in prod but not the lag mode.

**Revised ladder consequence:** step 2 ("add `READ_DATABASE_URL`, budget
unset") is NOT a no-op with today's binary — it hands FTS an unfenced
replica pool. Rollout order is now: merge (still a true no-op —
bb-block/bb-public already run the unfenced-FTS code and this PR only
adds dark seams) → fix #3643 → CI enforcement #3622 → then reader URL +
budget per the original ladder, re-running Max's three deployment tests
(reader-down latency, reader-down cold boot, paused-replay
read-your-own-write search).
## Activation gate (consolidated) — merge is clear; ALL of the below
precede any nonzero `BUZZ_REPLICA_READ_MAX_AGE_MS`

Six independent verification passes at `dd26caa9f` (Dawn ×2 incl.
byte-level + defang check, Max matrix + live Helm redteam, Wren full
CRUD regression + greenfield 0026 + 151/151 PG, Eva). Every induced
fault was either handled correctly by this PR's machinery or traced to
pre-existing main code. Remaining work gates **activation, not merge**:

1. **#3643** — unfenced NIP-50 FTS pool (pre-existing #2084; live in
prod today; also explains the blackhole search hang Wren observed — the
routed path was measured bounded at 151-152ms against a silent endpoint,
sqlx `inner.rs:252-255`).
2. **#3651** — reader `statement_timeout`: statements after acquire are
unbounded; mid-transaction blackhole reproduced hanging ≥15s (Dawn).
Small fix via `.after_connect` on the reader pool.
3. **#3622** — CI enforcement of the PG fixtures (with #3619 ordering).
4. **Recovery-conflict cancellation live test** — standby cancels a
routed read mid-snapshot → complete writer page, never partial (#3651's
fix also bounds the cancellation-never-arrives shape).
5. **DDL replication lag test** — migration on writer + paused replay +
budget on → fallback, not client-visible SQL errors.
6. **Reader-tx soak** — 30-60min mixed load; no idle-in-tx accumulation
on the standby.

Then Max's three deployment tests (reader-down latency, reader-down cold
boot, paused-replay read-your-own-write search) re-run on the fixed
binary before the first nonzero budget on bb-block.

---------

Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
2026-07-29 20:35:15 -04:00

890 lines
31 KiB
Rust

//! Feed-specific DB queries for the Home Feed feature.
//!
//! Aggregates three categories of data:
//! - **Mentions**: Events where the user's pubkey appears in a `p` tag.
//! - **Needs Action**: Approval requests (kind 46010) and reminders (kind 40007) tagged to the user.
//! - **Activity**: Recent events from channels the user can access.
//!
//! ## Performance characteristics
//!
//! `query_mentions` and `query_needs_action` join against the `event_mentions` table,
//! which carries community-leading composite indexes on
//! `(community_id, pubkey_hex, event_created_at DESC)` and
//! `(community_id, pubkey_hex, event_kind, event_created_at DESC)`. This replaces the Phase 1
//! full-table scan with an indexed lookup, keeping feed queries
//! sub-millisecond at scale (>100k events).
//!
//! **Phase 2 implemented**: the `event_mentions` table is populated by
//! [`crate::insert_mentions`] on every event insert. `query_mentions` and
//! `query_needs_action` now use `INNER JOIN event_mentions` instead of
//! scanning tags JSON.
//!
//! All feed queries enforce a hard `LIMIT` cap of `FEED_MAX_LIMIT` rows to bound
//! the result-set size and prevent runaway memory usage.
/// Hard upper bound on rows returned by any feed query.
///
/// Callers may request fewer rows, but never more. Enforced in every feed function
/// before the query is issued so the SQL `LIMIT` clause always reflects this cap.
pub const FEED_MAX_LIMIT: i64 = 100;
use chrono::{DateTime, Utc};
use sqlx::postgres::PgRow;
use sqlx::{PgPool, QueryBuilder};
use uuid::Uuid;
use buzz_core::kind::{
KIND_FORUM_COMMENT, KIND_FORUM_POST, KIND_GIT_ISSUE, KIND_GIT_PR_UPDATE, KIND_GIT_PULL_REQUEST,
KIND_GIT_STATUS_CLOSED, KIND_GIT_STATUS_DRAFT, KIND_GIT_STATUS_MERGED, KIND_GIT_STATUS_OPEN,
KIND_JOB_PROGRESS, KIND_JOB_REQUEST, KIND_JOB_RESULT, KIND_STREAM_MESSAGE,
KIND_STREAM_MESSAGE_V2, KIND_STREAM_REMINDER, KIND_TEXT_NOTE, KIND_WORKFLOW_APPROVAL_REQUESTED,
};
use buzz_core::{CommunityId, StoredEvent};
use crate::error::Result;
use crate::event::row_to_stored_event;
/// Column list shared by every feed subquery that aliases the `events` table as `e`.
const EVENT_COLS: &str =
"e.id, e.pubkey, e.created_at, e.kind, e.tags, e.content, e.sig, e.received_at, e.channel_id";
/// Column list for queries that select directly from `events` (no table alias).
const EVENT_COLS_UNALIASED: &str =
"id, pubkey, created_at, kind, tags, content, sig, received_at, channel_id";
/// Append channel visibility filtering for feed queries.
///
/// Feed reads may include channel-less community-global events, plus events in
/// channels the caller can access. An empty accessible-channel list therefore
/// means "global only", never "all channels".
fn push_visible_channel_filter(qb: &mut QueryBuilder<sqlx::Postgres>, col: &str, ids: &[Uuid]) {
if ids.is_empty() {
qb.push(format!(" AND {col} IS NULL"));
return;
}
qb.push(format!(" AND ({col} IS NULL OR {col} IN ("));
let mut sep = qb.separated(", ");
for id in ids {
sep.push_bind(*id);
}
qb.push("))");
}
/// Convert fetched rows into `Vec<StoredEvent>`, skipping any that fail conversion.
fn collect_stored_events(rows: Vec<PgRow>) -> Result<Vec<StoredEvent>> {
let mut out = Vec::with_capacity(rows.len());
for row in rows {
if let Some(ev) = row_to_stored_event(row)? {
out.push(ev);
}
}
Ok(out)
}
fn build_mentions_query(
community: CommunityId,
pubkey_bytes: &[u8],
accessible_channel_ids: &[Uuid],
since: Option<DateTime<Utc>>,
limit: i64,
) -> QueryBuilder<sqlx::Postgres> {
let limit = limit.min(FEED_MAX_LIMIT);
let pubkey_hex = hex::encode(pubkey_bytes);
let mut qb: QueryBuilder<sqlx::Postgres> = QueryBuilder::new(format!(
"SELECT {EVENT_COLS} FROM events e \
INNER JOIN event_mentions m ON e.community_id = m.community_id AND e.id = m.event_id \
WHERE e.community_id = "
));
qb.push_bind(*community.as_uuid());
qb.push(" AND m.community_id = ")
.push_bind(*community.as_uuid());
qb.push(" AND m.pubkey_hex = ").push_bind(pubkey_hex);
qb.push(" AND e.deleted_at IS NULL");
qb.push(format!(
" AND e.kind IN ({KIND_STREAM_MESSAGE}, {KIND_STREAM_MESSAGE_V2}, \
{KIND_TEXT_NOTE}, {KIND_FORUM_POST}, {KIND_FORUM_COMMENT}, {KIND_GIT_PULL_REQUEST}, \
{KIND_GIT_PR_UPDATE}, {KIND_GIT_ISSUE}, {KIND_GIT_STATUS_OPEN}, \
{KIND_GIT_STATUS_MERGED}, {KIND_GIT_STATUS_CLOSED}, {KIND_GIT_STATUS_DRAFT})"
));
push_visible_channel_filter(&mut qb, "e.channel_id", accessible_channel_ids);
if let Some(s) = since {
qb.push(" AND m.event_created_at >= ").push_bind(s);
}
qb.push(" ORDER BY m.event_created_at DESC LIMIT ")
.push_bind(limit);
qb
}
/// Find events that @mention the given pubkey (have `["p", pubkey_hex]` in tags).
///
/// Joins against the `event_mentions` table -- Phase 2 implementation.
/// **Performance**: community-leading indexed lookup on
/// `(community_id, pubkey_hex, event_created_at DESC)`.
///
/// Only returns community-global events and events from `accessible_channel_ids`.
/// `limit` is capped at [`FEED_MAX_LIMIT`] regardless of the value passed by the caller.
pub async fn query_mentions(
pool: &PgPool,
community: CommunityId,
pubkey_bytes: &[u8],
accessible_channel_ids: &[Uuid],
since: Option<DateTime<Utc>>,
limit: i64,
) -> Result<Vec<StoredEvent>> {
let mut conn = pool.acquire().await?;
query_mentions_on(
&mut conn,
community,
pubkey_bytes,
accessible_channel_ids,
since,
limit,
)
.await
}
/// [`query_mentions`] on a specific session — the replica-routing path runs
/// the query on the exact reader connection whose heartbeat observation
/// proved its predicate.
pub(crate) async fn query_mentions_on(
conn: &mut sqlx::PgConnection,
community: CommunityId,
pubkey_bytes: &[u8],
accessible_channel_ids: &[Uuid],
since: Option<DateTime<Utc>>,
limit: i64,
) -> Result<Vec<StoredEvent>> {
let mut qb = build_mentions_query(
community,
pubkey_bytes,
accessible_channel_ids,
since,
limit,
);
let rows = qb.build().fetch_all(&mut *conn).await?;
collect_stored_events(rows)
}
fn build_needs_action_query(
community: CommunityId,
pubkey_bytes: &[u8],
accessible_channel_ids: &[Uuid],
since: Option<DateTime<Utc>>,
limit: i64,
) -> QueryBuilder<sqlx::Postgres> {
let limit = limit.min(FEED_MAX_LIMIT);
let pubkey_hex = hex::encode(pubkey_bytes);
let mut qb: QueryBuilder<sqlx::Postgres> = QueryBuilder::new(format!(
"SELECT {EVENT_COLS} FROM events e \
INNER JOIN event_mentions m ON e.community_id = m.community_id AND e.id = m.event_id \
WHERE e.community_id = "
));
qb.push_bind(*community.as_uuid());
qb.push(" AND m.community_id = ")
.push_bind(*community.as_uuid());
qb.push(" AND m.pubkey_hex = ").push_bind(pubkey_hex);
qb.push(" AND e.deleted_at IS NULL");
qb.push(format!(
" AND e.kind IN ({KIND_WORKFLOW_APPROVAL_REQUESTED}, {KIND_STREAM_REMINDER})"
));
push_visible_channel_filter(&mut qb, "e.channel_id", accessible_channel_ids);
if let Some(s) = since {
qb.push(" AND m.event_created_at >= ").push_bind(s);
}
qb.push(" ORDER BY m.event_created_at DESC LIMIT ")
.push_bind(limit);
qb
}
/// Find events that require action from the given pubkey:
/// - [`KIND_WORKFLOW_APPROVAL_REQUESTED`] (workflow approval requested, tagged with user pubkey)
/// - [`KIND_STREAM_REMINDER`] (reminder, tagged with user pubkey)
///
/// Only returns community-global events and events from channels the user has access to
/// (`accessible_channel_ids`). This prevents surfacing approval requests from channels
/// the user was removed from.
/// **Performance**: community-leading indexed lookup via `event_mentions` join on
/// `(community_id, pubkey_hex, event_kind, event_created_at DESC)`.
/// `limit` is capped at [`FEED_MAX_LIMIT`] regardless of the value passed by the caller.
pub async fn query_needs_action(
pool: &PgPool,
community: CommunityId,
pubkey_bytes: &[u8],
accessible_channel_ids: &[Uuid],
since: Option<DateTime<Utc>>,
limit: i64,
) -> Result<Vec<StoredEvent>> {
let mut conn = pool.acquire().await?;
query_needs_action_on(
&mut conn,
community,
pubkey_bytes,
accessible_channel_ids,
since,
limit,
)
.await
}
/// [`query_needs_action`] on a specific session — see [`query_mentions_on`].
pub(crate) async fn query_needs_action_on(
conn: &mut sqlx::PgConnection,
community: CommunityId,
pubkey_bytes: &[u8],
accessible_channel_ids: &[Uuid],
since: Option<DateTime<Utc>>,
limit: i64,
) -> Result<Vec<StoredEvent>> {
let mut qb = build_needs_action_query(
community,
pubkey_bytes,
accessible_channel_ids,
since,
limit,
);
let rows = qb.build().fetch_all(&mut *conn).await?;
collect_stored_events(rows)
}
fn build_activity_query(
community: CommunityId,
accessible_channel_ids: &[Uuid],
since: Option<DateTime<Utc>>,
limit: i64,
) -> QueryBuilder<sqlx::Postgres> {
let limit = limit.min(FEED_MAX_LIMIT);
let mut qb: QueryBuilder<sqlx::Postgres> = QueryBuilder::new(format!(
"SELECT {EVENT_COLS_UNALIASED} FROM events WHERE community_id = "
));
qb.push_bind(*community.as_uuid());
qb.push(" AND deleted_at IS NULL");
qb.push(format!(
" AND kind IN ({KIND_STREAM_MESSAGE}, {KIND_STREAM_MESSAGE_V2}, {KIND_FORUM_POST}, \
{KIND_JOB_REQUEST}, {KIND_JOB_PROGRESS}, {KIND_JOB_RESULT})"
));
push_visible_channel_filter(&mut qb, "channel_id", accessible_channel_ids);
if let Some(s) = since {
qb.push(" AND created_at >= ").push_bind(s);
}
qb.push(" ORDER BY created_at DESC LIMIT ").push_bind(limit);
qb
}
/// Find recent activity across accessible channels (for watched topics / agent activity).
///
/// Returns stream messages, forum posts, and agent job events.
/// Workflow execution kinds (46001-46012) are intentionally excluded to avoid noise.
/// **Performance**: uses indexed `kind` + `channel_id` columns -- no JSON scan.
/// `limit` is capped at [`FEED_MAX_LIMIT`] regardless of the value passed by the caller.
pub async fn query_activity(
pool: &PgPool,
community: CommunityId,
accessible_channel_ids: &[Uuid],
since: Option<DateTime<Utc>>,
limit: i64,
) -> Result<Vec<StoredEvent>> {
let mut conn = pool.acquire().await?;
query_activity_on(&mut conn, community, accessible_channel_ids, since, limit).await
}
/// [`query_activity`] on a specific session — see [`query_mentions_on`].
pub(crate) async fn query_activity_on(
conn: &mut sqlx::PgConnection,
community: CommunityId,
accessible_channel_ids: &[Uuid],
since: Option<DateTime<Utc>>,
limit: i64,
) -> Result<Vec<StoredEvent>> {
let mut qb = build_activity_query(community, accessible_channel_ids, since, limit);
let rows = qb.build().fetch_all(&mut *conn).await?;
collect_stored_events(rows)
}
// -- Tests --------------------------------------------------------------------
#[cfg(test)]
mod tests {
use super::*;
use nostr::{EventBuilder, Keys, Kind, Tag};
use uuid::Uuid;
const TEST_DB_URL: &str = "postgres://buzz:buzz_dev@localhost:5432/buzz"; // sadscan:disable np.postgres.1 -- local test-only credentials
async fn setup_pool() -> PgPool {
let database_url = std::env::var("BUZZ_TEST_DATABASE_URL")
.or_else(|_| std::env::var("DATABASE_URL"))
.unwrap_or_else(|_| TEST_DB_URL.to_owned());
PgPool::connect(&database_url)
.await
.expect("connect to test DB")
}
async fn make_test_community(pool: &PgPool) -> Uuid {
let id = Uuid::new_v4();
let host = format!("feed-test-{}.example", id.simple());
sqlx::query("INSERT INTO communities (id, host) VALUES ($1, $2)")
.bind(id)
.bind(host)
.execute(pool)
.await
.expect("insert test community");
id
}
async fn insert_test_channel(pool: &PgPool, community: CommunityId) -> Uuid {
let id = Uuid::new_v4();
let creator = [0x11u8; 32];
sqlx::query(
"INSERT INTO channels (id, community_id, name, channel_type, visibility, created_by) \
VALUES ($1, $2, $3, 'stream'::channel_type, 'open'::channel_visibility, $4)",
)
.bind(id)
.bind(community.as_uuid())
.bind(format!("feed-test-channel-{}", id.simple()))
.bind(creator.as_slice())
.execute(pool)
.await
.expect("insert test channel");
id
}
async fn store_feed_event(
pool: &PgPool,
community: CommunityId,
kind: u32,
content: &str,
channel_id: Option<Uuid>,
tags: Vec<Tag>,
) -> nostr::Event {
let keys = Keys::generate();
let event = EventBuilder::new(Kind::Custom(kind as u16), content)
.tags(tags)
.sign_with_keys(&keys)
.expect("sign event");
crate::event::insert_event(pool, community, &event, channel_id)
.await
.expect("insert feed event");
crate::insert_mentions(pool, community, &event, channel_id)
.await
.expect("insert mentions");
event
}
// -- Postgres tenant-scope regressions ------------------------------------
#[tokio::test]
#[ignore = "requires Postgres"]
async fn query_mentions_is_scoped_across_communities() {
let pool = setup_pool().await;
let community_a = CommunityId::from_uuid(make_test_community(&pool).await);
let community_b = CommunityId::from_uuid(make_test_community(&pool).await);
let channel_a = insert_test_channel(&pool, community_a).await;
let channel_b = insert_test_channel(&pool, community_b).await;
let mentioned_pubkey = "02".repeat(32);
let mentioned_bytes = hex::decode(&mentioned_pubkey).expect("hex pubkey");
let event_a = store_feed_event(
&pool,
community_a,
KIND_STREAM_MESSAGE,
"community-a mention",
Some(channel_a),
vec![Tag::parse(["p", mentioned_pubkey.as_str()]).unwrap()],
)
.await;
let event_b = store_feed_event(
&pool,
community_b,
KIND_STREAM_MESSAGE,
"community-b mention",
Some(channel_b),
vec![Tag::parse(["p", mentioned_pubkey.as_str()]).unwrap()],
)
.await;
let rows = query_mentions(
&pool,
community_a,
&mentioned_bytes,
&[channel_a, channel_b],
None,
10,
)
.await
.expect("query mentions");
assert!(rows.iter().any(|row| row.event.id == event_a.id));
assert!(
rows.iter().all(|row| row.event.id != event_b.id),
"community B mention must not appear in community A feed"
);
}
#[tokio::test]
#[ignore = "requires Postgres"]
async fn query_needs_action_is_scoped_across_communities() {
let pool = setup_pool().await;
let community_a = CommunityId::from_uuid(make_test_community(&pool).await);
let community_b = CommunityId::from_uuid(make_test_community(&pool).await);
let channel_a = insert_test_channel(&pool, community_a).await;
let channel_b = insert_test_channel(&pool, community_b).await;
let actor_pubkey = "03".repeat(32);
let actor_bytes = hex::decode(&actor_pubkey).expect("hex pubkey");
let event_a = store_feed_event(
&pool,
community_a,
KIND_WORKFLOW_APPROVAL_REQUESTED,
"community-a approval",
Some(channel_a),
vec![Tag::parse(["p", actor_pubkey.as_str()]).unwrap()],
)
.await;
let event_b = store_feed_event(
&pool,
community_b,
KIND_WORKFLOW_APPROVAL_REQUESTED,
"community-b approval",
Some(channel_b),
vec![Tag::parse(["p", actor_pubkey.as_str()]).unwrap()],
)
.await;
let rows = query_needs_action(
&pool,
community_a,
&actor_bytes,
&[channel_a, channel_b],
None,
10,
)
.await
.expect("query needs_action");
assert!(rows.iter().any(|row| row.event.id == event_a.id));
assert!(
rows.iter().all(|row| row.event.id != event_b.id),
"community B needs_action item must not appear in community A feed"
);
}
#[tokio::test]
#[ignore = "requires Postgres"]
async fn query_activity_is_scoped_and_empty_channels_are_global_only() {
let pool = setup_pool().await;
let community_a = CommunityId::from_uuid(make_test_community(&pool).await);
let community_b = CommunityId::from_uuid(make_test_community(&pool).await);
let channel_a = insert_test_channel(&pool, community_a).await;
let channel_b = insert_test_channel(&pool, community_b).await;
let a_global = store_feed_event(
&pool,
community_a,
KIND_STREAM_MESSAGE,
"community-a global",
None,
vec![],
)
.await;
let a_channel = store_feed_event(
&pool,
community_a,
KIND_STREAM_MESSAGE,
"community-a channel",
Some(channel_a),
vec![],
)
.await;
let b_global = store_feed_event(
&pool,
community_b,
KIND_STREAM_MESSAGE,
"community-b global",
None,
vec![],
)
.await;
let b_channel = store_feed_event(
&pool,
community_b,
KIND_STREAM_MESSAGE,
"community-b channel",
Some(channel_b),
vec![],
)
.await;
let global_only = query_activity(&pool, community_a, &[], None, 10)
.await
.expect("query activity global only");
assert!(global_only.iter().any(|row| row.event.id == a_global.id));
assert!(
global_only.iter().all(|row| row.event.id != a_channel.id),
"empty accessible channels must not mean all tenant channels"
);
assert!(global_only.iter().all(|row| row.event.id != b_global.id));
assert!(global_only.iter().all(|row| row.event.id != b_channel.id));
let visible = query_activity(&pool, community_a, &[channel_a, channel_b], None, 10)
.await
.expect("query visible activity");
assert!(visible.iter().any(|row| row.event.id == a_global.id));
assert!(visible.iter().any(|row| row.event.id == a_channel.id));
assert!(
visible
.iter()
.all(|row| row.event.id != b_global.id && row.event.id != b_channel.id),
"community B activity must not appear in community A feed"
);
}
// -- Hex encoding of pubkey -----------------------------------------------
#[test]
fn pubkey_hex_encoding_is_lowercase() {
let pubkey_bytes = vec![0xAB, 0xCD, 0xEF, 0x01, 0x23, 0x45];
let hex = hex::encode(&pubkey_bytes);
assert_eq!(hex, "abcdef012345");
assert_eq!(hex, hex.to_lowercase());
}
#[test]
fn pubkey_hex_encoding_32_byte_key() {
let pubkey_bytes: Vec<u8> = (0u8..32).collect();
let hex = hex::encode(&pubkey_bytes);
assert_eq!(hex.len(), 64);
assert!(hex.chars().all(|c| c.is_ascii_hexdigit()));
assert_eq!(hex, hex.to_lowercase());
}
#[test]
fn pubkey_hex_encoding_all_zeros() {
let pubkey_bytes = vec![0u8; 32];
let hex = hex::encode(&pubkey_bytes);
assert_eq!(hex, "0".repeat(64));
}
#[test]
fn pubkey_hex_encoding_all_ff() {
let pubkey_bytes = vec![0xFFu8; 32];
let hex = hex::encode(&pubkey_bytes);
assert_eq!(hex, "f".repeat(64));
}
// -- JSON tag format for tag matching -------------------------------------
#[test]
fn json_tag_format_for_p_tag_mention() {
let pubkey_hex = "abc123def456".to_owned();
let tag_json = serde_json::json!([["p", pubkey_hex]]).to_string();
assert_eq!(tag_json, r#"[["p","abc123def456"]]"#);
}
#[test]
fn json_tag_format_is_compact_not_pretty() {
let pubkey_hex = "deadbeef".to_owned();
let tag_json = serde_json::json!([["p", pubkey_hex]]).to_string();
assert!(
!tag_json.contains(' '),
"tag JSON must be compact, got: {tag_json}"
);
}
#[test]
fn json_tag_format_p_tag_is_first_element() {
let pubkey_hex = "aabbccdd".to_owned();
let tag_json = serde_json::json!([["p", pubkey_hex]]).to_string();
assert!(tag_json.starts_with(r#"[["p","#), "got: {tag_json}");
}
#[test]
fn json_tag_format_round_trips_through_serde() {
let pubkey_hex = "cafebabe00112233".to_owned();
let tag_json = serde_json::json!([["p", pubkey_hex.clone()]]).to_string();
let parsed: serde_json::Value = serde_json::from_str(&tag_json).unwrap();
let outer = parsed.as_array().unwrap();
assert_eq!(outer.len(), 1, "outer array must have exactly one element");
let inner = outer[0].as_array().unwrap();
assert_eq!(inner.len(), 2);
assert_eq!(inner[0].as_str().unwrap(), "p");
assert_eq!(inner[1].as_str().unwrap(), pubkey_hex);
}
// -- Kind number sets -----------------------------------------------------
#[test]
fn mentions_query_includes_stream_message_kind() {
use buzz_core::kind::{
KIND_FORUM_COMMENT, KIND_FORUM_POST, KIND_STREAM_MESSAGE, KIND_STREAM_MESSAGE_V2,
};
let mention_kinds: &[u32] = &[
KIND_STREAM_MESSAGE,
KIND_STREAM_MESSAGE_V2,
KIND_FORUM_POST,
KIND_FORUM_COMMENT,
];
assert!(
mention_kinds.contains(&KIND_STREAM_MESSAGE),
"stream message kind must be in mentions"
);
assert!(
mention_kinds.contains(&KIND_STREAM_MESSAGE_V2),
"stream message v2 kind must be in mentions"
);
assert!(
mention_kinds.contains(&KIND_FORUM_POST),
"forum post kind must be in mentions"
);
assert!(
mention_kinds.contains(&KIND_FORUM_COMMENT),
"forum comment kind must be in mentions"
);
}
#[test]
fn needs_action_query_includes_approval_and_reminder_kinds() {
use buzz_core::kind::{KIND_STREAM_REMINDER, KIND_WORKFLOW_APPROVAL_REQUESTED};
let needs_action_kinds: &[u32] = &[KIND_WORKFLOW_APPROVAL_REQUESTED, KIND_STREAM_REMINDER];
assert!(
needs_action_kinds.contains(&KIND_WORKFLOW_APPROVAL_REQUESTED),
"approval request kind must be in needs_action"
);
assert!(
needs_action_kinds.contains(&KIND_STREAM_REMINDER),
"reminder kind must be in needs_action"
);
}
#[test]
fn activity_query_includes_agent_job_kinds() {
use buzz_core::kind::{
KIND_FORUM_POST, KIND_JOB_PROGRESS, KIND_JOB_REQUEST, KIND_JOB_RESULT,
KIND_STREAM_MESSAGE, KIND_STREAM_MESSAGE_V2,
};
let activity_kinds: &[u32] = &[
KIND_STREAM_MESSAGE,
KIND_STREAM_MESSAGE_V2,
KIND_FORUM_POST,
KIND_JOB_REQUEST,
KIND_JOB_PROGRESS,
KIND_JOB_RESULT,
];
assert!(
activity_kinds.contains(&KIND_JOB_REQUEST),
"job request kind must be in activity"
);
assert!(
activity_kinds.contains(&KIND_JOB_PROGRESS),
"job progress kind must be in activity"
);
assert!(
activity_kinds.contains(&KIND_JOB_RESULT),
"job result kind must be in activity"
);
assert!(
activity_kinds.contains(&KIND_STREAM_MESSAGE),
"stream message kind must be in activity"
);
assert!(
activity_kinds.contains(&KIND_FORUM_POST),
"forum post kind must be in activity"
);
}
#[test]
fn activity_query_excludes_workflow_execution_kinds() {
use buzz_core::kind::{
KIND_FORUM_POST, KIND_JOB_PROGRESS, KIND_JOB_REQUEST, KIND_JOB_RESULT,
KIND_STREAM_MESSAGE, KIND_STREAM_MESSAGE_V2,
};
let activity_kinds: &[u32] = &[
KIND_STREAM_MESSAGE,
KIND_STREAM_MESSAGE_V2,
KIND_FORUM_POST,
KIND_JOB_REQUEST,
KIND_JOB_PROGRESS,
KIND_JOB_RESULT,
];
use buzz_core::kind::{KIND_WORKFLOW_APPROVAL_DENIED, KIND_WORKFLOW_TRIGGERED};
for kind in KIND_WORKFLOW_TRIGGERED..=KIND_WORKFLOW_APPROVAL_DENIED {
assert!(
!activity_kinds.contains(&kind),
"workflow execution kind {kind} must NOT be in activity"
);
}
}
#[test]
fn needs_action_kinds_do_not_overlap_with_activity_kinds() {
use buzz_core::kind::{
KIND_FORUM_POST, KIND_JOB_PROGRESS, KIND_JOB_REQUEST, KIND_JOB_RESULT,
KIND_STREAM_MESSAGE, KIND_STREAM_MESSAGE_V2, KIND_STREAM_REMINDER,
KIND_WORKFLOW_APPROVAL_REQUESTED,
};
let needs_action_kinds: &[u32] = &[KIND_WORKFLOW_APPROVAL_REQUESTED, KIND_STREAM_REMINDER];
let activity_kinds: &[u32] = &[
KIND_STREAM_MESSAGE,
KIND_STREAM_MESSAGE_V2,
KIND_FORUM_POST,
KIND_JOB_REQUEST,
KIND_JOB_PROGRESS,
KIND_JOB_RESULT,
];
for kind in needs_action_kinds {
assert!(
!activity_kinds.contains(kind),
"kind {kind} appears in both needs_action and activity -- check intent"
);
}
}
// -- Channel ID filtering logic -------------------------------------------
#[test]
fn channel_id_bytes_encoding_is_correct() {
let channel_id = Uuid::parse_str("9a1657ac-f7aa-5db0-b632-d8bbeb6dfb50").unwrap();
let bytes = channel_id.as_bytes().to_vec();
assert_eq!(bytes.len(), 16);
let recovered = Uuid::from_slice(&bytes).unwrap();
assert_eq!(channel_id, recovered);
}
#[test]
fn multiple_channel_ids_produce_distinct_byte_sequences() {
let id1 = Uuid::new_v4();
let id2 = Uuid::new_v4();
let bytes1 = id1.as_bytes().to_vec();
let bytes2 = id2.as_bytes().to_vec();
assert_ne!(bytes1, bytes2);
}
#[test]
fn nil_uuid_channel_id_bytes_are_all_zeros() {
let nil_id = Uuid::nil();
let bytes = nil_id.as_bytes().to_vec();
assert_eq!(bytes, vec![0u8; 16]);
}
#[test]
fn empty_channel_list_means_global_only() {
let community = buzz_core::CommunityId::from_uuid(Uuid::new_v4());
let mut qb = build_activity_query(community, &[], None, 10);
let query = qb.build();
let sql_str = sqlx::Execute::sql(query);
let sql = sql_str.as_str();
assert!(
sql.contains("WHERE community_id = "),
"activity feed must bind the tenant community: {sql}"
);
assert!(
sql.contains("AND channel_id IS NULL"),
"empty accessible-channel list must mean global-only, not all tenant channels: {sql}"
);
assert!(
!sql.contains("channel_id IN"),
"empty accessible-channel list must not emit an IN filter: {sql}"
);
}
#[test]
fn non_empty_channel_list_includes_global_and_accessible_channels() {
let community = buzz_core::CommunityId::from_uuid(Uuid::new_v4());
let channel_id = Uuid::new_v4();
let mut qb = build_activity_query(community, &[channel_id], None, 10);
let query = qb.build();
let sql_str = sqlx::Execute::sql(query);
let sql = sql_str.as_str();
assert!(
sql.contains("AND (channel_id IS NULL OR channel_id IN ("),
"feed should include community-global events plus accessible channels: {sql}"
);
}
#[test]
fn mentions_query_is_tenant_scoped_and_joins_mentions_by_composite_key() {
let community = buzz_core::CommunityId::from_uuid(Uuid::new_v4());
let pubkey = vec![0x42; 32];
let channel_id = Uuid::new_v4();
let mut qb = build_mentions_query(community, &pubkey, &[channel_id], None, 10);
let query = qb.build();
let sql_str = sqlx::Execute::sql(query);
let sql = sql_str.as_str();
assert!(
sql.contains("INNER JOIN event_mentions m ON e.community_id = m.community_id AND e.id = m.event_id"),
"mentions must join event_mentions on the composite tenant/event key: {sql}"
);
assert!(
sql.contains("WHERE e.community_id = "),
"mentions feed must scope events to the tenant community: {sql}"
);
assert!(
sql.contains("AND m.community_id = "),
"mentions feed must also bind event_mentions.community_id: {sql}"
);
assert!(
sql.contains(&KIND_GIT_PULL_REQUEST.to_string())
&& sql.contains(&KIND_GIT_ISSUE.to_string())
&& sql.contains(&KIND_TEXT_NOTE.to_string()),
"mentions feed must include Buzz Git roots and comments: {sql}"
);
}
#[test]
fn needs_action_query_is_tenant_scoped_and_joins_mentions_by_composite_key() {
let community = buzz_core::CommunityId::from_uuid(Uuid::new_v4());
let pubkey = vec![0x42; 32];
let channel_id = Uuid::new_v4();
let mut qb = build_needs_action_query(community, &pubkey, &[channel_id], None, 10);
let query = qb.build();
let sql_str = sqlx::Execute::sql(query);
let sql = sql_str.as_str();
assert!(
sql.contains("INNER JOIN event_mentions m ON e.community_id = m.community_id AND e.id = m.event_id"),
"needs_action must join event_mentions on the composite tenant/event key: {sql}"
);
assert!(
sql.contains("WHERE e.community_id = "),
"needs_action feed must scope events to the tenant community: {sql}"
);
assert!(
sql.contains("AND m.community_id = "),
"needs_action feed must also bind event_mentions.community_id: {sql}"
);
}
#[test]
fn channel_id_list_with_single_entry() {
let channel_id = Uuid::new_v4();
let accessible = [channel_id];
assert_eq!(accessible.len(), 1);
let bytes = accessible[0].as_bytes().to_vec();
assert_eq!(bytes.len(), 16);
}
#[test]
fn channel_id_list_with_multiple_entries_are_distinct() {
let ids: Vec<Uuid> = (0..5).map(|_| Uuid::new_v4()).collect();
assert_eq!(ids.len(), 5);
let byte_seqs: Vec<Vec<u8>> = ids.iter().map(|id| id.as_bytes().to_vec()).collect();
let unique: std::collections::HashSet<Vec<u8>> = byte_seqs.into_iter().collect();
assert_eq!(unique.len(), 5, "all channel IDs must be distinct");
}
}