External/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-08-18 06:50:31 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix(search): restore privacy kind exclusions at the FTS storage layer

Browse Source 
The Typesense→Postgres FTS rewrite replaced out-of-band indexing with
`search_tsv TSVECTOR GENERATED ALWAYS AS (to_tsvector('simple', content))
STORED` over every row. The old relay (handlers/event.rs:287 on main)
deliberately skipped search-indexing for three kind classes, and the new
search query layer has no kind exclusion — so gift wraps, DM-visibility
snapshots, and event reminders were all in the FTS index.

Fix at the storage layer (option A — single source of truth, zero
app-layer drift across multiple search call sites): make the generated
column yield `NULL::tsvector` for excluded kinds via a CASE expression.
A NULL tsvector never matches `@@`, so excluded rows are structurally
unsearchable.

Excluded set, parity with main's `handlers/event.rs:287-290`:
- 1059  KIND_GIFT_WRAP        (NIP-17 ciphertext)
- 30300 KIND_EVENT_REMINDER   (AUTHOR_ONLY_KINDS — defense in depth)
- 30622 KIND_DM_VISIBILITY    (per-viewer private hide state)

Constants are inlined in the migration with a comment naming the
`buzz_core::kind` names: sqlx migrations are frozen SQL and can't
`use buzz_core::kind`; importing core into a migration would be worse
drift than the inline-with-comment shape.

Three coupled layers:

1. Schema CASE in migrations/0001_initial_schema.sql. The 0001 schema
   was consolidated by Max in 4b7654a1c (Lane-0 contract) and is
   pre-deploy; editing in place rather than adding a new migration
   matches the agreed shape and is covered by
   `run_migrations_applies_consolidated_initial_schema_on_fresh_database`.

2. buzz-search/tests/fts_integration.rs — new test
   `excluded_kinds_are_storage_level_unsearchable` inserts kind:1059 +
   kind:30300 + kind:30622 + kind:9 with the same unique token; asserts
   only the kind:9 control surfaces. Each excluded kind has its own
   load-bearing negative assertion with a diagnostic message naming the
   regression, plus a tight exactly-one-hit bound.

   Mutate-bite verified: dropping the CASE's NULL branch (revert to
   `to_tsvector('simple', content)`) makes excluded kinds searchable
   and the test fails RED with the designed message
   "kind:1059 MUST NOT be searchable — privacy regression in search_tsv
   generated column". Restored → 10/10 green.

3. crates/buzz-test-client/tests/e2e_nostr_interop.rs — rewrote
   `test_nip17_gift_wrap_not_searchable` to use the relay's actual
   NIP-50 search seam (`Filter::new().search(token)`,
   `collect_until_eose`) instead of the now-removed Typesense
   `/multi_search`. Pattern cribbed from
   `test_nip50_search_returns_results_and_eose`. Pointer comment to
   the underlying mutate-bite in fts_integration.rs.

Verification on `quinn/search-kind-exclusions` off `34ffb8ab3`:
- `cargo test -p buzz-search --tests -- --include-ignored`: 10/10
- `cargo test -p buzz-db -- --include-ignored`: 99/99 (incl. migration
  lints + consolidated-schema fresh-DB)
- `cargo check -p buzz-test-client --tests`: clean
- `cargo clippy -p buzz-search -p buzz-db --all-targets -- -D warnings`:
  clean
- `cargo fmt --all -- --check`: clean

Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
This commit is contained in:
npub1jmc9dt2lyvzu3h0kxlwxt5zg4fxp9476awyxw6gwxn72g6cw7exqs64whm
2026-06-27 01:09:33 -04:00
co-authored by Tyler Longwell
parent 34ffb8ab37
commit 3fd16cc1f7
3 changed files with 180 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/buzz-search/tests/fts_integration.rs
+122
View File
@@ -675,3 +675,125 @@ async fn channel_less_only_excludes_per_channel_events() {
teardown(pool, &schema).await;
}
/// Privacy regression gate: the storage layer MUST NOT make these kinds
/// searchable. The migration's `search_tsv` generated column emits NULL
/// for excluded kinds, so a `search_tsv @@ query` probe never matches.
///
/// Set kept in sync with the pre-rewrite skip in `handlers/event.rs:287-290`
/// on `main`:
/// - 1059 = `KIND_GIFT_WRAP` (NIP-17 ciphertext)
/// - 30300 = `KIND_EVENT_REMINDER` (in `AUTHOR_ONLY_KINDS`)
/// - 30622 = `KIND_DM_VISIBILITY` (per-viewer private hide state)
///
/// All four events are inserted with the same unique token in their content
/// so a single search query exercises every kind in one round-trip. Only
/// the kind:9 control must surface — the three excluded kinds must not.
///
/// Mutate-bite: drop the `CASE WHEN kind IN (…)` from the generated column
/// (revert to `to_tsvector('simple', content)`) → all four events surface →
/// restore.
#[tokio::test]
#[ignore = "requires Postgres"]
async fn excluded_kinds_are_storage_level_unsearchable() {
let (pool, schema) = setup().await;
let c = mk_community(&pool, "privacy.example").await;
let token = "privacykinds_unique_marker_xyzzy";
// kind:9 control — MUST be searchable.
insert_event(
&pool,
c,
rand_bytes32(),
rand_bytes32(),
9,
&format!("public chat — {token}"),
None,
1_700_000_000,
)
.await;
// kind:1059 gift wrap (NIP-17 ciphertext) — MUST NOT be searchable.
insert_event(
&pool,
c,
rand_bytes32(),
rand_bytes32(),
1059,
&format!("gift wrap — {token}"),
None,
1_700_000_001,
)
.await;
// kind:30300 event reminder (AUTHOR_ONLY_KINDS) — MUST NOT be searchable.
insert_event(
&pool,
c,
rand_bytes32(),
rand_bytes32(),
30300,
&format!("reminder — {token}"),
None,
1_700_000_002,
)
.await;
// kind:30622 DM visibility snapshot — MUST NOT be searchable.
insert_event(
&pool,
c,
rand_bytes32(),
rand_bytes32(),
30622,
&format!("dm visibility — {token}"),
None,
1_700_000_003,
)
.await;
let svc = SearchService::new(pool.clone());
let result = svc
.search(&SearchQuery {
community: c,
q: token.into(),
channel_scope: ChannelScope::Any,
kinds: None,
authors: None,
since: None,
until: None,
page: 1,
per_page: 10,
})
.await
.expect("search ok");
let kinds: Vec<i32> = result.hits.iter().map(|h| h.kind).collect();
// Positive: kind:9 surfaces (control — proves the search index works at all).
assert!(
kinds.contains(&9),
"kind:9 control row MUST be searchable, got kinds={kinds:?}",
);
// Negative (load-bearing): each excluded kind MUST NOT surface.
for forbidden in [1059, 30300, 30622] {
assert!(
!kinds.contains(&forbidden),
"kind:{forbidden} MUST NOT be searchable — \
privacy regression in `search_tsv` generated column. kinds={kinds:?}",
);
}
// Tight bound: exactly one hit (the control). Catches any future
// weakening where some-but-not-all excluded kinds surface.
assert_eq!(
result.hits.len(),
1,
"expected exactly 1 hit (the kind:9 control), got {} (kinds={kinds:?})",
result.hits.len(),
);
teardown(pool, &schema).await;
}
crates/buzz-test-client/tests/e2e_nostr_interop.rs
+41 -44
View File
@@ -971,6 +971,19 @@ async fn test_nip10_thread_reply_not_in_top_level() {
#[tokio::test]
#[ignore]
async fn test_nip17_gift_wrap_not_searchable() {
// Privacy regression gate at the relay's actual NIP-50 search seam.
//
// Storage-level exclusion lives in the `events.search_tsv` generated
// column (migrations/0001_initial_schema.sql): a row whose `kind` is
// in the privacy skip-set yields a NULL tsvector and never matches
// `@@`. This test proves the property from the wire: REQ with a
// NIP-50 `search` filter returns the kind:9 control and does NOT
// return the kind:1059 gift wrap.
//
// The mutate-bite for the underlying property lives in
// `crates/buzz-search/tests/fts_integration.rs::
// excluded_kinds_are_storage_level_unsearchable`
// (drops the CASE → all excluded kinds surface).
let url = relay_url();
let keys_a = Keys::generate();
let keys_b = Keys::generate();
@@ -992,65 +1005,49 @@ async fn test_nip17_gift_wrap_not_searchable() {
let ok = client.send_event(gift_wrap).await.expect("send gift wrap");
assert!(ok.accepted, "relay rejected gift wrap: {}", ok.message);
// 2. Send kind:9 control message with the same content.
// 2. Send kind:9 control message containing the same unique token.
let ok2 = client
.send_text_message(&keys_a, &channel, &unique_token, 9)
.await
.expect("send kind:9");
assert!(ok2.accepted, "relay rejected kind:9: {}", ok2.message);
client.disconnect().await.expect("disconnect");
// Small delay so the FTS column (generated, in-row) is visible to readers.
tokio::time::sleep(Duration::from_millis(500)).await;
// Wait for async Typesense indexing.
tokio::time::sleep(Duration::from_secs(3)).await;
// 3. NIP-50 search via the relay (no kind filter — we want to see what
// surfaces). `h` tag scopes to our channel to keep the result set tight.
let sid = sub_id("nip17-not-searchable");
let filter = Filter::new()
.search(&unique_token)
.custom_tags(SingleLetterTag::lowercase(Alphabet::H), [channel.as_str()]);
// 3. Query Typesense DIRECTLY — bypasses all relay-level filtering.
let ts_url =
std::env::var("TYPESENSE_URL").unwrap_or_else(|_| "http://localhost:8108".to_string());
let ts_key = std::env::var("TYPESENSE_API_KEY").unwrap_or_else(|_| "buzz_dev_key".to_string());
let http = reqwest::Client::new();
let resp = http
.post(format!("{ts_url}/multi_search"))
.header("X-TYPESENSE-API-KEY", &ts_key)
.json(&serde_json::json!({
"searches": [{
"collection": "events",
"q": unique_token,
"query_by": "content",
"per_page": 10
}]
}))
.send()
client
.subscribe(&sid, vec![filter])
.await
.expect("Typesense multi_search request");
.expect("subscribe");
let events = client
.collect_until_eose(&sid, Duration::from_secs(10))
.await
.expect("collect until EOSE");
let kinds: Vec<u16> = events.iter().map(|e| e.kind.as_u16()).collect();
// Control: kind:9 IS searchable — proves the search path works at all.
assert!(
resp.status().is_success(),
"Typesense returned {}",
resp.status()
);
let body: serde_json::Value = resp.json().await.expect("parse Typesense response");
let hits = body["results"][0]["hits"].as_array().expect("hits array");
// Control: kind:9 IS indexed.
let has_kind9 = hits
.iter()
.any(|h| h["document"]["kind"].as_i64() == Some(9));
assert!(
has_kind9,
"kind:9 control message not found in Typesense — indexing broken"
kinds.contains(&9),
"kind:9 control not returned by NIP-50 search — indexing broken. kinds={kinds:?}",
);
// Assertion: kind:1059 is NOT indexed.
let has_kind1059 = hits
.iter()
.any(|h| h["document"]["kind"].as_i64() == Some(1059));
// Load-bearing: kind:1059 MUST NOT surface.
assert!(
!has_kind1059,
"kind:1059 found in Typesense — gift wraps must NOT be indexed. hits: {hits:?}"
!kinds.contains(&1059),
"kind:1059 gift wrap surfaced via NIP-50 search — privacy regression in \
the `search_tsv` generated column. kinds={kinds:?}",
);
client.disconnect().await.expect("disconnect");
}
/// Send 3 messages with varying relevance to a query, wait for indexing, then search.
migrations/0001_initial_schema.sql
+17 -1
View File
@@ -203,7 +203,23 @@ CREATE TABLE events (
-- community-leading btree filters BitmapAnd-ed with the GIN probe, so the
-- GIN index itself stays the minimal `GIN (search_tsv)` (Max's caveat:
-- avoid btree_gin unless EXPLAIN proves it buys something).
search_tsv TSVECTOR GENERATED ALWAYS AS (to_tsvector('simple', content)) STORED,
--
-- Privacy kind exclusions (parity with the pre-rewrite Typesense feed —
-- old relay's `handlers/event.rs:287-290` skip set):
-- 1059 = KIND_GIFT_WRAP (NIP-17 ciphertext)
-- 30300 = KIND_EVENT_REMINDER (AUTHOR_ONLY_KINDS — defense in depth)
-- 30622 = KIND_DM_VISIBILITY (per-viewer private hide state)
-- NULL tsvector never matches `@@`, so excluded rows are storage-level
-- unsearchable. Constants kept in `buzz_core::kind` (KIND_GIFT_WRAP,
-- KIND_EVENT_REMINDER, KIND_DM_VISIBILITY); inlined here because a sqlx
-- migration is frozen SQL and cannot import the Rust constant. If a new
-- privacy-sensitive kind is added there, update this list and add a
-- regression test in `buzz-search/tests/fts_integration.rs`.
search_tsv TSVECTOR GENERATED ALWAYS AS (
CASE WHEN kind IN (1059, 30300, 30622) THEN NULL::tsvector
ELSE to_tsvector('simple', content)
END
) STORED,
sig BYTEA NOT NULL,
received_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
channel_id UUID,