Files
buzz/scripts/run-desktop-release-smoke.sh
76f114a252 test: add deterministic desktop release smoke (#5699)
## Summary

- add `just desktop-release-smoke`, a deterministic desktop
correctness/reachability smoke against an ephemeral real local relay
- preserve existing DM history when the first live DM enters a pageless
query window, the desktop-v0.5.10 disappearing-DM regression
- enforce foreground JS ordering: a frame and actionable sidebar input
must dispatch before mounted stale queries begin resume refetches, while
separately requiring the navigation to commit promptly
- seed a 10,000-event dense-second fixture and verify exact event-ID
reachability, SHA-256 identity, ordering, duplicate absence, bounded
mounted rows, and drained render work
- isolate Postgres per run, serialize the shared Redis DB, retain
phase/relay/Playwright diagnostics, and gate desktop release manifest
assembly on the smoke

This is deliberately **not a performance-regression gate**. CDP and
action timing fields are informational only. There is no
candidate/baseline comparison or threshold. A future performance lane
needs repeated equivalent fixtures, discrete interaction samples, and an
explicit comparator/noise policy.

The diagnostics record the fixture version, row count, wall-clock base
timestamp (`fixtureSecond`), expected event-ID hash, observed state, and
measurements. Because the created-at floor requires a current timestamp,
paired comparison remains disabled.

The release job runs on an isolated GitHub-hosted runner. The script
also guards automatic local runs with a Redis allocation lock. Its
remaining direct-PID cleanup and free-port selection race mean it should
not be repurposed onto a persistent concurrent shared runner without
first hardening process-group cleanup and port reservation.

### Related issue

N/A

### Testing

- `pnpm --dir desktop typecheck`
- focused real-local-relay release smoke passed after adversarial review
fixes
- identical DM witness passed current and failed `desktop-v0.5.10` with
the history-loss signature
- identical foreground witness bytes
(`2c1e97df04c9b8ca0304b66bbbe9bdb4d08924ad8ce0f68a9c490458fcc3aca8`)
failed `desktop-v0.5.10` structurally: the first resume fetch was marker
1, before first frame/sidebar dispatch at marker 8
- with PR #5696 (`59f613c40`) merged, the witness showed focus at 951.3
ms, first frame at 951.6 ms, click dispatch at 952.1 ms, first resume
fetch at 968.9 ms, and route commit at 992.4 ms
- the gate therefore protects first paint and actionable input dispatch;
route commit is a bounded responsiveness witness, not a prerequisite for
resume work
- the corrected focused foreground scenario passed at
`6d9b5be40da58bbee92a856b04c3558946d0a950`; the prior merged-tree full
run passed DM retention and 10k reachability before exposing this
contract mismatch
- pre-push passed on exact pushed head
`6d9b5be40da58bbee92a856b04c3558946d0a950`, including desktop checks,
typecheck, desktop tests, Rust tests, mobile tests, and Tauri checks
- full 10,000-event scenario reached 10,000/10,000 exact IDs with
matching SHA-256, 199 continuation requests, and 95 mounted rows in
about 4.4 minutes
- reduced-row review run passed in 18.4 seconds

### Foreground witness boundary

The Chromium test is a deterministic JS policy gate. Headless Chromium
does not expose an honest blur/focus transition in this fixture, so the
test drives the production focus listener and `document.hasFocus()`
predicate together and records that simulation explicitly. It proves
refetch fan-out ordering, not AppKit activation, WKWebView paint, or an
activating physical click. A packaged macOS native lane is still
required before claiming the actual desktop activation experience is
certified.

---------

Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Co-authored-by: Carl <c7ebe626f000404285d3686e1dc74cc07cc60a9754a150041ba132e14bd3e2ec@buzz.block.builderlab.xyz>
2026-08-13 14:49:40 -07:00

149 lines
5.2 KiB
Bash
Executable File

#!/usr/bin/env bash
# Run deterministic desktop correctness smoke; timing metrics are informational.
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
ARTIFACT_DIR="${BUZZ_RELEASE_SMOKE_ARTIFACT_DIR:-${ROOT}/desktop/test-results/release-smoke}"
DB_NAME="${BUZZ_RELEASE_SMOKE_DB:-buzz_release_smoke_${$}}"
REDIS_DB="${BUZZ_RELEASE_SMOKE_REDIS_DB:-}"
LOCK_DIR="${TMPDIR:-/tmp}/buzz-desktop-release-smoke.lock"
RELAY_PID=""
LOCK_HELD=false
free_port() {
python3 - <<'PY'
import socket
with socket.socket() as sock:
sock.bind(("127.0.0.1", 0))
print(sock.getsockname()[1])
PY
}
RELAY_PORT="${BUZZ_RELEASE_SMOKE_RELAY_PORT:-$(free_port)}"
HEALTH_PORT="${BUZZ_RELEASE_SMOKE_HEALTH_PORT:-$(free_port)}"
METRICS_PORT="${BUZZ_RELEASE_SMOKE_METRICS_PORT:-$(free_port)}"
COMMUNITY_HOST="localhost:${RELAY_PORT}"
RELAY_HTTP_URL="http://${COMMUNITY_HOST}"
STARTED_AT="$(date +%s)"
log() { printf '[desktop-release-smoke] %s\n' "$*"; }
phase() {
local name="$1" start="$2"
printf '{"phase":"%s","duration_ms":%d}\n' "$name" "$(( ($(date +%s) - start) * 1000 ))" >> "${ARTIFACT_DIR}/phases.jsonl"
}
cleanup() {
local status=$?
trap - EXIT INT TERM
if [[ -n "${RELAY_PID}" ]]; then
kill "${RELAY_PID}" 2>/dev/null || true
for _ in $(seq 1 50); do
kill -0 "${RELAY_PID}" 2>/dev/null || break
sleep 0.1
done
kill -9 "${RELAY_PID}" 2>/dev/null || true
fi
docker exec buzz-redis redis-cli -n "${REDIS_DB}" FLUSHDB >/dev/null 2>&1 || true
docker exec buzz-postgres dropdb -U buzz --if-exists "${DB_NAME}" >/dev/null 2>&1 || true
if [[ "${LOCK_HELD}" == true ]]; then rmdir "${LOCK_DIR}" 2>/dev/null || true; fi
exit "${status}"
}
trap cleanup EXIT INT TERM
# Shared Docker services expose one Redis instance and a finite database index
# space. Serialize automatic allocation; callers that deliberately own an
# isolated Redis DB may opt out by setting BUZZ_RELEASE_SMOKE_REDIS_DB.
if [[ -z "${REDIS_DB}" ]]; then
mkdir "${LOCK_DIR}" 2>/dev/null || {
log "another release-smoke run owns ${LOCK_DIR}; set BUZZ_RELEASE_SMOKE_REDIS_DB only for an isolated runner"
exit 1
}
LOCK_HELD=true
REDIS_DB=15
fi
mkdir -p "${ARTIFACT_DIR}"
: > "${ARTIFACT_DIR}/phases.jsonl"
cd "${ROOT}"
phase_start="$(date +%s)"
log "starting backing services"
docker compose up -d postgres redis minio minio-init
for container in buzz-postgres buzz-redis buzz-minio; do
for _ in $(seq 1 60); do
[[ "$(docker inspect --format='{{.State.Health.Status}}' "${container}" 2>/dev/null || true)" == "healthy" ]] && break
sleep 1
done
[[ "$(docker inspect --format='{{.State.Health.Status}}' "${container}" 2>/dev/null || true)" == "healthy" ]] || {
docker logs "${container}" || true
exit 1
}
done
phase services "${phase_start}"
phase_start="$(date +%s)"
log "creating isolated database ${DB_NAME}"
docker exec buzz-postgres createdb -U buzz "${DB_NAME}"
export PGHOST=localhost PGPORT=5432 PGUSER=buzz PGPASSWORD=buzz_dev PGDATABASE="${DB_NAME}"
export PGSCHEMA_PLAN_HOST=localhost PGSCHEMA_PLAN_PORT=5432 PGSCHEMA_PLAN_DB="${DB_NAME}"
export PGSCHEMA_PLAN_USER=buzz PGSCHEMA_PLAN_PASSWORD=buzz_dev
./bin/pgschema apply --file schema/schema.sql --auto-approve
docker exec -i -e PGPASSWORD=buzz_dev buzz-postgres \
psql -U buzz -d "${DB_NAME}" -v ON_ERROR_STOP=1 < scripts/attach-schema-partitions.sql
BUZZ_DB_NAME="${DB_NAME}" BUZZ_COMMUNITY_HOST="${COMMUNITY_HOST}" ./scripts/setup-desktop-test-data.sh
docker exec buzz-redis redis-cli -n "${REDIS_DB}" FLUSHDB >/dev/null
phase database "${phase_start}"
phase_start="$(date +%s)"
if [[ -n "${BUZZ_E2E_RELAY_BIN:-}" ]]; then
RELAY_BIN="${BUZZ_E2E_RELAY_BIN}"
else
log "building relay"
cargo build --profile ci -p buzz-relay
RELAY_BIN="${ROOT}/target/ci/buzz-relay"
fi
log "starting relay at ${RELAY_HTTP_URL}"
env \
DATABASE_URL="postgres://buzz:buzz_dev@localhost:5432/${DB_NAME}" \
REDIS_URL="redis://localhost:6379/${REDIS_DB}" \
RELAY_URL="ws://${COMMUNITY_HOST}" \
BUZZ_BIND_ADDR="127.0.0.1:${RELAY_PORT}" \
BUZZ_HEALTH_PORT="${HEALTH_PORT}" \
BUZZ_METRICS_PORT="${METRICS_PORT}" \
BUZZ_REQUIRE_AUTH_TOKEN=false \
BUZZ_RECONCILE_CHANNELS=true \
BUZZ_RATE_LIMIT_HUMAN_MESSAGES_PER_MIN=1000000 \
BUZZ_RATE_LIMIT_HUMAN_API_CALLS_PER_MIN=1000000 \
BUZZ_RATE_LIMIT_HUMAN_WS_EVENTS_PER_SEC=100000 \
"${RELAY_BIN}" > "${ARTIFACT_DIR}/relay.log" 2>&1 &
RELAY_PID=$!
ready=false
for _ in $(seq 1 300); do
kill -0 "${RELAY_PID}" 2>/dev/null || { cat "${ARTIFACT_DIR}/relay.log"; exit 1; }
if curl --silent --fail --max-time 1 "http://127.0.0.1:${HEALTH_PORT}/_readiness" >/dev/null; then
ready=true
break
fi
sleep 0.1
done
[[ "${ready}" == true ]] || { cat "${ARTIFACT_DIR}/relay.log"; exit 1; }
phase relay "${phase_start}"
phase_start="$(date +%s)"
if [[ "${BUZZ_RELEASE_SMOKE_NO_BUILD:-0}" == "1" ]]; then
log "reusing existing desktop E2E bundle"
else
log "building desktop E2E bundle"
pnpm -C desktop build:e2e
fi
phase build "${phase_start}"
phase_start="$(date +%s)"
log "running release smoke"
BUZZ_E2E_RELAY_URL="${RELAY_HTTP_URL}" \
BUZZ_RELEASE_SMOKE_ARTIFACT_DIR="${ARTIFACT_DIR}" \
pnpm -C desktop exec playwright test --config=playwright.release-smoke.config.ts
phase smoke "${phase_start}"
phase total "${STARTED_AT}"
log "artifacts: ${ARTIFACT_DIR}"