## Why
Expose PostgreSQL datastore latency within existing request traces so
slow logical database operations can be identified without recording
tenant data or query arguments.
## What
- Add client spans around logical PostgreSQL operations across the
database facade, search, audit, replica fencing, and command persistence
- Use a dedicated `buzz_datastore` target and `db.system.name =
"postgresql"` for filtering and backend classification
- Exclude health-check database calls and scrub raw identifiers and
errors from newly traced paths
## Risk Assessment
Medium — this instruments frequently used datastore paths and increases
trace volume when enabled, but does not change SQL execution or
datastore behavior. Existing OpenTelemetry filtering controls export.
## References
- Pre-push clippy and fast unit-test hooks passed
Generated with Amp
---------
Signed-off-by: David Grochowski <dgrochowski@squareup.com>
Co-authored-by: Amp <amp@ampcode.com>
Fixes#2637 — full analysis and reproduction there.
## Problem
Audit entries are stamped and hashed with `Utc::now()` (nanoseconds),
then stored in a `TIMESTAMPTZ` column (microseconds). `compute_hash`
covers `created_at.to_rfc3339()`, and chrono emits 0/3/6/**9**
fractional digits depending on the value — so the digest written at
`service.rs:103` is computed over `…T12:00:00.123456789+00:00` while
`verify_chain` recomputes over the `…T12:00:00.123456+00:00` that
Postgres hands back.
Every hash chain backed by a real database therefore fails verification
at its first entry, on untampered data. That is not just a broken
feature — it means a genuinely forged row is indistinguishable from the
permanent baseline failure, so `HashMismatch` carries no signal.
It is invisible in CI because all six chain tests are `#[ignore =
"requires Postgres"]`, and the in-process `hash.rs` tests use a fixture
timestamp of `2026-01-01T00:00:00Z` — zero sub-seconds, the one value
where the bug cannot appear.
## Solution
Reduce `created_at` to the stored precision *before* hashing, so the
in-memory entry and the row are byte-identical:
```rust
pub fn to_storage_precision(created_at: DateTime<Utc>) -> DateTime<Utc> {
created_at.trunc_subsecs(6)
}
```
`log_inner` is the only place that assigns `created_at` — every caller
goes through `NewAuditEntry`, which carries no timestamp — so this is a
single choke point. It is wrapped in a `log_timestamp()` helper purely
so the invariant is assertable without a database.
I chose truncation at the write path over the alternative (hashing a
precision-independent encoding such as
`timestamp_micros().to_be_bytes()`). Both fix the mismatch, but
truncating keeps the existing hash preimage format and gives the
stronger invariant: the `AuditEntry` returned from `log()` is now
exactly what a later read returns.
Truncation matches what actually happens on the wire — sqlx encodes
`DateTime<Utc>` as microseconds since the Postgres epoch, truncating —
so the value hashed is the value stored.
## Validation
Toolchain note: built on Windows with the `x86_64-pc-windows-gnu`
toolchain (no MSVC linker locally).
**Before**, against Postgres 17 with `migrations/*` applied:
```
$ cargo test -p buzz-audit --lib -- --ignored --test-threads=1
test service::tests::chain_links_within_one_community ... FAILED
test service::tests::chains_are_independent_per_community ... FAILED
test service::tests::community_chain_starts_at_seq_1_with_null_prev ... ok
test service::tests::cross_community_row_does_not_verify ... ok
test service::tests::verify_detects_tampering_within_a_community ... FAILED
test service::tests::verify_empty_range_is_false ... ok
test result: FAILED. 3 passed; 3 failed
```
with `HashMismatch { seq: 2 }` / `HashMismatch { seq: 1 }` on untampered
chains.
**After**, same database:
```
test result: ok. 6 passed; 0 failed
```
`verify_detects_tampering_within_a_community` is the one to look at: it
asserts `HashMismatch` lands on the *tampered* entry's `seq`. It was
failing because verification already blew up on an earlier untampered
row — so the assertion proving tamper detection works had never actually
been exercised. It passes now.
Also:
- `cargo test -p buzz-audit --lib` (no Postgres) — 12 passed, 0 failed.
- `cargo clippy -p buzz-audit --all-targets -- -D warnings` — clean.
- `cargo fmt -p buzz-audit -- --check` — clean.
## New tests
Three in `hash.rs`, none needing Postgres:
- `storage_precision_drops_sub_microsecond_digits` — the helper's
contract, and that it is idempotent so a re-read value is unchanged.
- `nanosecond_timestamps_cannot_survive_a_database_round_trip` — asserts
the digests **differ**. This is the trap itself, written down so the
next person changing the hash preimage sees why the precision reduction
is load-bearing.
- `storage_precision_timestamps_survive_a_database_round_trip` — the
invariant the write path must hold.
Plus `log_timestamp_carries_no_sub_microsecond_digits` in `service.rs`,
deliberately **not** `#[ignore]`d, so a regression on the write path is
caught by `just test-unit` instead of only by Postgres-gated tests that
normally never run.
## Compatibility
Rows written before this stay unverifiable — they always were — so there
is no migration. An operator relying on an existing chain has to
re-anchor.
## Relationship to #2620#2620 proposes a shared `verify_entries` walk (anchoring, seq
contiguity, tail-truncation detection) plus a `buzz-admin audit verify`
command. Its Postgres-free unit tests build entries in memory and would
pass regardless, but its `#[ignore]` Postgres tests and the operator
command itself would fail on every real chain until this lands. Worth
taking this first so that work has a verifiable baseline — the two
changes don't overlap in code.
---------
Signed-off-by: Shani Singh <teamdeveloperworld@gmail.com>
