test: add deterministic desktop release smoke (#5699)

## Summary

- add `just desktop-release-smoke`, a deterministic desktop
correctness/reachability smoke against an ephemeral real local relay
- preserve existing DM history when the first live DM enters a pageless
query window, the desktop-v0.5.10 disappearing-DM regression
- enforce foreground JS ordering: a frame and actionable sidebar input
must dispatch before mounted stale queries begin resume refetches, while
separately requiring the navigation to commit promptly
- seed a 10,000-event dense-second fixture and verify exact event-ID
reachability, SHA-256 identity, ordering, duplicate absence, bounded
mounted rows, and drained render work
- isolate Postgres per run, serialize the shared Redis DB, retain
phase/relay/Playwright diagnostics, and gate desktop release manifest
assembly on the smoke

This is deliberately **not a performance-regression gate**. CDP and
action timing fields are informational only. There is no
candidate/baseline comparison or threshold. A future performance lane
needs repeated equivalent fixtures, discrete interaction samples, and an
explicit comparator/noise policy.

The diagnostics record the fixture version, row count, wall-clock base
timestamp (`fixtureSecond`), expected event-ID hash, observed state, and
measurements. Because the created-at floor requires a current timestamp,
paired comparison remains disabled.

The release job runs on an isolated GitHub-hosted runner. The script
also guards automatic local runs with a Redis allocation lock. Its
remaining direct-PID cleanup and free-port selection race mean it should
not be repurposed onto a persistent concurrent shared runner without
first hardening process-group cleanup and port reservation.

### Related issue

N/A

### Testing

- `pnpm --dir desktop typecheck`
- focused real-local-relay release smoke passed after adversarial review
fixes
- identical DM witness passed current and failed `desktop-v0.5.10` with
the history-loss signature
- identical foreground witness bytes
(`2c1e97df04c9b8ca0304b66bbbe9bdb4d08924ad8ce0f68a9c490458fcc3aca8`)
failed `desktop-v0.5.10` structurally: the first resume fetch was marker
1, before first frame/sidebar dispatch at marker 8
- with PR #5696 (`59f613c40`) merged, the witness showed focus at 951.3
ms, first frame at 951.6 ms, click dispatch at 952.1 ms, first resume
fetch at 968.9 ms, and route commit at 992.4 ms
- the gate therefore protects first paint and actionable input dispatch;
route commit is a bounded responsiveness witness, not a prerequisite for
resume work
- the corrected focused foreground scenario passed at
`6d9b5be40da58bbee92a856b04c3558946d0a950`; the prior merged-tree full
run passed DM retention and 10k reachability before exposing this
contract mismatch
- pre-push passed on exact pushed head
`6d9b5be40da58bbee92a856b04c3558946d0a950`, including desktop checks,
typecheck, desktop tests, Rust tests, mobile tests, and Tauri checks
- full 10,000-event scenario reached 10,000/10,000 exact IDs with
matching SHA-256, 199 continuation requests, and 95 mounted rows in
about 4.4 minutes
- reduced-row review run passed in 18.4 seconds

### Foreground witness boundary

The Chromium test is a deterministic JS policy gate. Headless Chromium
does not expose an honest blur/focus transition in this fixture, so the
test drives the production focus listener and `document.hasFocus()`
predicate together and records that simulation explicitly. It proves
refetch fan-out ordering, not AppKit activation, WKWebView paint, or an
activating physical click. A packaged macOS native lane is still
required before claiming the actual desktop activation experience is
certified.

---------

Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Co-authored-by: Carl <c7ebe626f000404285d3686e1dc74cc07cc60a9754a150041ba132e14bd3e2ec@buzz.block.builderlab.xyz>
This commit is contained in:
Wes
2026-08-13 14:49:40 -07:00
committed by GitHub
co-authored by Carl
parent e0940927ff
commit 76f114a252
10 changed files with 1008 additions and 1 deletions
+59 -1
View File
@@ -775,6 +775,63 @@ jobs:
${{ steps.artifacts.outputs.exe }}
${{ steps.artifacts.outputs.sig }}
desktop-release-smoke:
name: Desktop release smoke
if: github.repository == 'block/buzz'
runs-on: ubuntu-latest
needs: setup
timeout-minutes: 20
permissions:
contents: read
env:
PLAYWRIGHT_BROWSERS_PATH: ${{ github.workspace }}/.cache/ms-playwright
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
ref: ${{ needs.setup.outputs.source_sha }}
persist-credentials: false
- uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1
- name: Install desktop dependencies
run: just desktop-install-ci
- name: Get Playwright version
id: pw-version
run: echo "version=$(cd desktop && node -e \"console.log(require('@playwright/test/package.json').version)\")" >> "$GITHUB_OUTPUT"
- name: Restore Playwright browser cache
id: playwright-cache
uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5
with:
path: ${{ env.PLAYWRIGHT_BROWSERS_PATH }}
key: playwright-${{ runner.os }}-${{ steps.pw-version.outputs.version }}
- name: Install Playwright Chromium
if: steps.playwright-cache.outputs.cache-hit != 'true'
run: cd desktop && pnpm exec playwright install chromium
- name: Install Playwright system dependencies
run: cd desktop && pnpm exec playwright install-deps chromium
- name: Save Playwright browser cache
if: steps.playwright-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5
with:
path: ${{ env.PLAYWRIGHT_BROWSERS_PATH }}
key: playwright-${{ runner.os }}-${{ steps.pw-version.outputs.version }}
- name: Build test relay
run: cargo build --profile ci -p buzz-relay
- name: Run deterministic correctness smoke
env:
BUZZ_E2E_RELAY_BIN: ${{ github.workspace }}/target/ci/buzz-relay
BUZZ_RELEASE_SMOKE_ARTIFACT_DIR: ${{ github.workspace }}/release-smoke-artifacts
run: just desktop-release-smoke
- name: Upload release-smoke diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: desktop-release-smoke
path: |
release-smoke-artifacts
desktop/test-results
desktop/playwright-release-smoke-report
if-no-files-found: warn
retention-days: 14
assemble-manifest:
name: Assemble multi-platform latest.json
# Only the tag-bound setup path can reach this job.
@@ -785,9 +842,10 @@ jobs:
needs.release-macos-x64.result == 'success' &&
needs.release-linux.result == 'success' &&
needs.release-windows.result == 'success' &&
needs.desktop-release-smoke.result == 'success' &&
github.ref == format('refs/tags/desktop-v{0}', needs.setup.outputs.version)
runs-on: ubuntu-latest
needs: [setup, release, release-macos-x64, release-linux, release-windows]
needs: [setup, release, release-macos-x64, release-linux, release-windows, desktop-release-smoke]
timeout-minutes: 10
permissions:
contents: write
+4
View File
@@ -276,6 +276,10 @@ desktop-e2e-smoke:
desktop-e2e-integration: _ensure-migrations
cd {{desktop_dir}} && pnpm test:e2e:integration
# Run the deterministic desktop correctness smoke against an isolated local relay
desktop-release-smoke:
./scripts/run-desktop-release-smoke.sh
# Run only the e2e specs changed vs origin/main (both projects) before pushing
desktop-e2e-pre-push: _ensure-migrations
git fetch origin main
+1
View File
@@ -14,6 +14,7 @@ dist-ssr
playwright-report
playwright-report.json
test-results
playwright-release-smoke-report
*.local
playwright-report
test-results
+1
View File
@@ -20,6 +20,7 @@
"test:e2e": "pnpm build:e2e && playwright test",
"test:e2e:smoke": "pnpm build:e2e && playwright test --project=smoke",
"test:e2e:integration": "pnpm build:e2e && playwright test --project=integration",
"test:e2e:release-smoke": "pnpm build:e2e && playwright test --config=playwright.release-smoke.config.ts",
"test:e2e:report": "playwright show-report",
"tauri:build": "tauri build"
},
@@ -0,0 +1,36 @@
import { defineConfig, devices } from "@playwright/test";
const webPort = process.env.BUZZ_RELEASE_SMOKE_WEB_PORT ?? "4173";
const webUrl = `http://127.0.0.1:${webPort}`;
export default defineConfig({
testDir: "./tests/e2e",
testMatch: [
"**/release-smoke.spec.ts",
"**/dm-history-live-regression.spec.ts",
"**/foreground-responsiveness-regression.spec.ts",
],
timeout: 10 * 60_000,
retries: 0,
workers: 1,
reporter: [
["list"],
["json", { outputFile: "test-results/release-smoke/playwright.json" }],
[
"html",
{ open: "never", outputFolder: "playwright-release-smoke-report" },
],
],
use: {
...devices["Desktop Chrome"],
baseURL: webUrl,
screenshot: "only-on-failure",
trace: "retain-on-failure",
},
webServer: {
command: `python3 -m http.server ${webPort} -d dist`,
cwd: ".",
reuseExistingServer: false,
url: webUrl,
},
});
@@ -0,0 +1,203 @@
import { expect, test, type Page, type Route } from "@playwright/test";
import { hexToBytes } from "@noble/hashes/utils.js";
import { finalizeEvent, type VerifiedEvent } from "nostr-tools/pure";
import { installRelayBridge, TEST_IDENTITIES } from "../helpers/bridge";
import { assertRelaySeeded } from "../helpers/seed";
const RELAY_HTTP = process.env.BUZZ_E2E_RELAY_URL ?? "http://localhost:3000";
const DM_ID = "5a9c064e-0411-5242-ae6b-0363ba99b8e6";
async function publishAliceDm(
content: string,
createdAt: number,
): Promise<VerifiedEvent> {
const event = finalizeEvent(
{
kind: 9,
content,
created_at: createdAt,
tags: [
["h", DM_ID],
["p", TEST_IDENTITIES.tyler.pubkey],
],
},
hexToBytes(TEST_IDENTITIES.alice.privateKey),
);
const response = await fetch(`${RELAY_HTTP}/events`, {
method: "POST",
headers: { "Content-Type": "application/json", "X-Pubkey": event.pubkey },
body: JSON.stringify(event),
});
if (!response.ok) {
throw new Error(
`POST /events failed (${response.status}): ${await response.text()}`,
);
}
return event;
}
async function timelineIds(page: Page) {
return page.getByTestId("message-timeline").evaluate((element) =>
Array.from(element.querySelectorAll<HTMLElement>("[data-message-id]"))
.map((row) => row.dataset.messageId)
.filter((id): id is string => Boolean(id)),
);
}
async function cachedMessageIds(page: Page) {
return page.evaluate((channelId) => {
const client = window.__BUZZ_E2E_QUERY_CLIENT__ as unknown as {
getQueryData: (
key: readonly unknown[],
) => Array<{ id: string }> | undefined;
};
if (!client) throw new Error("E2E query client is unavailable.");
return (client.getQueryData(["channel-messages", channelId]) ?? []).map(
(event) => event.id,
);
}, DM_ID);
}
function isDmWindowQuery(route: Route) {
const request = route.request();
if (request.method() !== "POST" || !request.url().endsWith("/query")) {
return false;
}
try {
const body = request.postDataJSON() as Array<Record<string, unknown>>;
return body.some(
(filter) =>
Array.isArray(filter["#h"]) &&
filter["#h"].includes(DM_ID) &&
filter.top_level === true &&
filter.include_summaries === true &&
filter.include_aux === true,
);
} catch {
return false;
}
}
async function dmLiveRequestCount(page: Page) {
return page.evaluate((channelId) => {
let count = 0;
for (const entry of window.__BUZZ_E2E_COMMAND_LOG__ ?? []) {
if (entry.command !== "plugin:websocket|send") continue;
const data = (
entry.payload as { message?: { data?: string } } | undefined
)?.message?.data;
if (!data) continue;
try {
const frame = JSON.parse(data) as [
string,
string,
...Record<string, unknown>[],
];
if (
frame[0] === "REQ" &&
frame
.slice(2)
.some(
(filter) =>
Array.isArray(filter["#h"]) &&
filter["#h"].includes(channelId) &&
typeof filter.since === "number",
)
) {
count += 1;
}
} catch {
// Ignore non-JSON websocket payloads.
}
}
return count;
}, DM_ID);
}
test.beforeAll(async () => {
test.setTimeout(90_000);
await assertRelaySeeded();
});
test("existing DM history remains when the first live DM reaches a pageless window", async ({
page,
}) => {
const baseSecond = Math.floor(Date.now() / 1000) - 10;
const history = await Promise.all(
[0, 1, 2].map((index) =>
publishAliceDm(`dm history ${index}`, baseSecond + index),
),
);
const historyIds = history.map((event) => event.id);
await installRelayBridge(page, "tyler");
await page.goto("/");
await page.getByTestId("channel-alice-tyler").click();
await expect(page.getByTestId("chat-title")).toHaveText("alice-tyler");
await expect.poll(() => timelineIds(page)).toEqual(historyIds);
await page.getByTestId("channel-general").click();
await expect(page.getByTestId("chat-title")).toHaveText("general");
await expect.poll(() => cachedMessageIds(page)).toEqual(historyIds);
await page.evaluate((channelId) => {
const client = window.__BUZZ_E2E_QUERY_CLIENT__ as unknown as {
removeQueries: (filters: {
queryKey: readonly unknown[];
exact: boolean;
}) => void;
};
if (!client) throw new Error("E2E query client is unavailable.");
client.removeQueries({
queryKey: ["channel-window", channelId],
exact: true,
});
}, DM_ID);
let heldDmWindowQuery = false;
let releaseHeldQuery: (() => void) | undefined;
let finishHeldQuery: (() => void) | undefined;
const heldQueryReleased = new Promise<void>((resolve) => {
releaseHeldQuery = resolve;
});
const heldQueryFinished = new Promise<void>((resolve) => {
finishHeldQuery = resolve;
});
const holdDmWindowQuery = async (route: Route) => {
if (!isDmWindowQuery(route)) {
await route.continue();
return;
}
heldDmWindowQuery = true;
await heldQueryReleased;
await route.continue();
finishHeldQuery?.();
};
await page.route("**/query", holdDmWindowQuery);
try {
const priorLiveRequests = await dmLiveRequestCount(page);
await page.getByTestId("channel-alice-tyler").click();
await expect(page.getByTestId("chat-title")).toHaveText("alice-tyler");
await expect
.poll(() => dmLiveRequestCount(page))
.toBeGreaterThan(priorLiveRequests);
const live = await publishAliceDm(
"dm live arrival",
Math.floor(Date.now() / 1000),
);
const expectedIds = [...historyIds, live.id];
await expect.poll(() => timelineIds(page)).toEqual(expectedIds);
await expect.poll(() => cachedMessageIds(page)).toEqual(expectedIds);
expect(new Set(expectedIds).size).toBe(expectedIds.length);
await expect(page.getByTestId("channel-alice-tyler")).toBeVisible();
await expect(page.getByTestId("message-input")).toBeVisible();
await expect(page.getByTestId("message-input")).toBeEnabled();
} finally {
releaseHeldQuery?.();
if (heldDmWindowQuery) await heldQueryFinished;
await page.unroute("**/query", holdDmWindowQuery);
}
});
@@ -0,0 +1,242 @@
import { expect, test } from "@playwright/test";
import { installRelayBridge } from "../helpers/bridge";
import { assertRelaySeeded } from "../helpers/seed";
const REPRESENTATIVE_FAMILIES = new Set([
"channels",
"home-feed",
"relay-agents",
"channel-templates",
"custom-emoji",
"user-status",
]);
type ForegroundProbe = {
marker: string;
at: number;
detail?: unknown;
};
test.beforeAll(async () => {
test.setTimeout(90_000);
await assertRelaySeeded();
});
test("foreground paints and handles a sidebar action before resume fetches", async ({
page,
}) => {
await page.addInitScript(() => {
type Probe = { marker: string; at: number; detail?: unknown };
const probes: Probe[] = [];
const record = (marker: string, detail?: unknown) =>
probes.push({ marker, at: performance.now(), detail });
(
window as typeof window & { __FOREGROUND_PROBES__?: Probe[] }
).__FOREGROUND_PROBES__ = probes;
let captured = false;
window.addEventListener(
"focus",
() => {
const armed = (
window as typeof window & { __FOREGROUND_PROBE_ARMED__?: boolean }
).__FOREGROUND_PROBE_ARMED__;
if (!armed || captured) return;
captured = true;
record("focus", {
hasFocus: document.hasFocus(),
visibilityState: document.visibilityState,
});
window.requestAnimationFrame(() => {
record("first-frame");
const target = document.querySelector<HTMLElement>(
'[data-testid="channel-alice-tyler"]',
);
if (!target)
throw new Error("Sidebar sentinel channel is unavailable.");
target.click();
record("interaction-dispatched");
const recordCommitWhenSelected = () => {
const selectedTitle = document
.querySelector<HTMLElement>('[data-testid="chat-title"]')
?.textContent?.trim();
if (selectedTitle === "alice-tyler") {
record("interaction-committed");
return;
}
window.requestAnimationFrame(recordCommitWhenSelected);
};
recordCommitWhenSelected();
});
},
{ capture: true },
);
});
await installRelayBridge(page, "tyler");
await page.goto("/");
await expect(page.getByTestId("app-sidebar")).toBeVisible();
await page.getByTestId("channel-general").click();
await expect(page.getByTestId("chat-title")).toHaveText("general");
const seasonedKeys = await page.evaluate(
(representativeFamilies) => {
type QueryLike = {
queryKey: readonly unknown[];
options: { refetchOnWindowFocus?: unknown };
state: Record<string, unknown>;
getObserversCount: () => number;
setState: (state: Record<string, unknown>) => void;
};
type QueryCacheLike = {
findAll: () => QueryLike[];
subscribe: (
listener: (event: { query?: QueryLike }) => void,
) => () => void;
};
type QueryClientLike = { getQueryCache: () => QueryCacheLike };
const client =
window.__BUZZ_E2E_QUERY_CLIENT__ as unknown as QueryClientLike;
if (!client) throw new Error("E2E query client is unavailable.");
const families = new Set(representativeFamilies);
const active = client
.getQueryCache()
.findAll()
.filter(
(query) =>
query.getObserversCount() > 0 &&
families.has(String(query.queryKey[0] ?? "")),
);
for (const query of active) {
query.setState({
...query.state,
dataUpdatedAt: 1,
isInvalidated: true,
});
}
const probes = (
window as typeof window & { __FOREGROUND_PROBES__?: ForegroundProbe[] }
).__FOREGROUND_PROBES__;
if (!probes) throw new Error("Foreground probe was not installed.");
const record = (marker: string, detail?: unknown) =>
probes.push({ marker, at: performance.now(), detail });
const activeHashes = new Set(
active.map((query) => JSON.stringify(query.queryKey)),
);
const fetchingHashes = new Set<string>();
client.getQueryCache().subscribe((event) => {
const query = event.query;
if (!query) return;
const hash = JSON.stringify(query.queryKey);
if (
activeHashes.has(hash) &&
query.state.fetchStatus === "fetching" &&
!fetchingHashes.has(hash)
) {
fetchingHashes.add(hash);
record("resume-fetch-start", query.queryKey);
}
});
return active.map((query) => ({
key: query.queryKey,
refetchOnWindowFocus: query.options.refetchOnWindowFocus ?? null,
}));
},
[...REPRESENTATIVE_FAMILIES],
);
expect(seasonedKeys.length).toBeGreaterThanOrEqual(3);
const originalHasFocus = await page.evaluateHandle(() => document.hasFocus);
await page.evaluate(() => {
(
window as typeof window & { __FOREGROUND_PROBE_ARMED__?: boolean }
).__FOREGROUND_PROBE_ARMED__ = true;
Object.defineProperty(document, "hasFocus", {
configurable: true,
value: () => false,
});
window.dispatchEvent(new Event("blur"));
});
await expect.poll(() => page.evaluate(() => document.hasFocus())).toBe(false);
// Headless Chromium reports every page as focused, even after bringToFront on
// another target. Drive the production listener and isAppFocused predicate
// together rather than shipping a false-green no-op focus transition.
await page.evaluate(() => {
Object.defineProperty(document, "hasFocus", {
configurable: true,
value: () => true,
});
window.dispatchEvent(new Event("focus"));
});
await expect.poll(() => page.evaluate(() => document.hasFocus())).toBe(true);
await expect(page.getByTestId("chat-title")).toHaveText("alice-tyler", {
timeout: 5_000,
});
const probes = await expect
.poll(
() =>
page.evaluate(
() =>
(
window as typeof window & {
__FOREGROUND_PROBES__?: ForegroundProbe[];
}
).__FOREGROUND_PROBES__ ?? [],
),
{ timeout: 5_000 },
)
.toEqual(
expect.arrayContaining([
expect.objectContaining({ marker: "focus" }),
expect.objectContaining({ marker: "first-frame" }),
expect.objectContaining({ marker: "interaction-dispatched" }),
expect.objectContaining({ marker: "interaction-committed" }),
]),
);
// expect.poll returns void; read the settled marker stream for ordering.
void probes;
const settled = await page.evaluate(
() =>
(window as typeof window & { __FOREGROUND_PROBES__?: ForegroundProbe[] })
.__FOREGROUND_PROBES__ ?? [],
);
await test.info().attach("foreground-ordering", {
body: Buffer.from(
`${JSON.stringify({ seasonedKeys, probes: settled }, null, 2)}\n`,
),
contentType: "application/json",
});
const index = (marker: string) =>
settled.findIndex((probe) => probe.marker === marker);
const focus = settled.find((probe) => probe.marker === "focus");
expect(focus?.detail).toEqual({
hasFocus: true,
visibilityState: "visible",
});
expect(index("first-frame")).toBeGreaterThan(index("focus"));
expect(index("interaction-dispatched")).toBeGreaterThan(index("first-frame"));
expect(index("interaction-committed")).toBeGreaterThan(
index("interaction-dispatched"),
);
// The production scheduler promises a painted interaction boundary before
// resume work, not completion of an asynchronous route transition. Dispatch
// proves the sidebar target was actionable after the first frame; the commit
// marker remains a responsiveness witness and must still settle promptly.
const firstResumeFetch = index("resume-fetch-start");
if (firstResumeFetch !== -1) {
expect(firstResumeFetch).toBeGreaterThan(index("interaction-dispatched"));
}
await originalHasFocus.dispose();
});
+68
View File
@@ -0,0 +1,68 @@
import type { CDPSession, Page } from "@playwright/test";
export type BrowserMetrics = {
layoutMs: number;
recalcMs: number;
layoutCount: number;
scriptMs: number;
taskMs: number;
};
export type ActionMeasurement<T> = {
metrics: BrowserMetrics;
result: T;
wallMs: number;
};
async function readBrowserMetrics(client: CDPSession): Promise<BrowserMetrics> {
const { metrics } = (await client.send("Performance.getMetrics")) as {
metrics: Array<{ name: string; value: number }>;
};
const metric = (name: string) =>
metrics.find((candidate) => candidate.name === name)?.value ?? 0;
return {
layoutMs: metric("LayoutDuration") * 1000,
recalcMs: metric("RecalcStyleDuration") * 1000,
layoutCount: metric("LayoutCount"),
scriptMs: metric("ScriptDuration") * 1000,
taskMs: metric("TaskDuration") * 1000,
};
}
function delta(after: BrowserMetrics, before: BrowserMetrics): BrowserMetrics {
return {
layoutMs: after.layoutMs - before.layoutMs,
recalcMs: after.recalcMs - before.recalcMs,
layoutCount: after.layoutCount - before.layoutCount,
scriptMs: after.scriptMs - before.scriptMs,
taskMs: after.taskMs - before.taskMs,
};
}
export async function measureAction<T>(
page: Page,
action: () => Promise<T>,
): Promise<ActionMeasurement<T>> {
const client = await page.context().newCDPSession(page);
await client.send("Performance.enable");
try {
const before = await readBrowserMetrics(client);
const startedAt = performance.now();
const result = await action();
await page.evaluate(
() =>
new Promise<void>((resolve) =>
requestAnimationFrame(() => requestAnimationFrame(() => resolve())),
),
);
const wallMs = performance.now() - startedAt;
return {
metrics: delta(await readBrowserMetrics(client), before),
result,
wallMs,
};
} finally {
await client.send("Performance.disable");
await client.detach();
}
}
+246
View File
@@ -0,0 +1,246 @@
import { mkdir, writeFile } from "node:fs/promises";
import { resolve } from "node:path";
import { expect, test, type Page, type TestInfo } from "@playwright/test";
import { installRelayBridge } from "../helpers/bridge";
import { assertRelaySeeded } from "../helpers/seed";
import { denseSecondWall, seedScenario } from "../helpers/seedRelay";
import { measureAction, type ActionMeasurement } from "./perf/metrics";
const RELAY_HTTP = process.env.BUZZ_E2E_RELAY_URL ?? "http://localhost:3000";
const GENERAL_CHANNEL_ID = "9f28288a-d724-587a-9709-92dc7f967110";
const DEEP_ROW_COUNT = Number.parseInt(
process.env.BUZZ_RELEASE_SMOKE_ROWS ?? "10000",
10,
);
const FIXTURE_VERSION = 1;
type ScenarioArtifact = {
schemaVersion: 1;
fixtureVersion: number;
fixtureSecond: number;
scenario: string;
relayUrl: string;
rowCount: number;
expectedIdHash: string;
observed: Record<string, unknown>;
measurement: Omit<ActionMeasurement<unknown>, "result">;
};
async function sha256(values: readonly string[]) {
const bytes = new TextEncoder().encode([...values].sort().join("\n"));
const digest = await crypto.subtle.digest("SHA-256", bytes);
return Array.from(new Uint8Array(digest), (byte) =>
byte.toString(16).padStart(2, "0"),
).join("");
}
async function writeArtifact(testInfo: TestInfo, artifact: ScenarioArtifact) {
const body = `${JSON.stringify(artifact, null, 2)}\n`;
const outputPath = testInfo.outputPath("metrics.json");
await writeFile(outputPath, body);
await testInfo.attach("release-smoke-metrics", {
body: Buffer.from(body),
contentType: "application/json",
});
const requestedDirectory = process.env.BUZZ_RELEASE_SMOKE_ARTIFACT_DIR;
if (requestedDirectory) {
const directory = resolve(requestedDirectory);
await mkdir(directory, { recursive: true });
await writeFile(resolve(directory, `${artifact.scenario}.json`), body);
}
}
async function openGeneral(page: Page) {
await installRelayBridge(page, "tyler");
await page.goto("/");
await page.getByTestId("channel-general").click();
await expect(page.getByTestId("chat-title")).toHaveText("general");
await expect(page.locator('[data-render-pending="true"]')).toHaveCount(0);
const timeline = page.getByTestId("message-timeline");
await expect(timeline.locator("[data-message-id]").first()).toBeVisible();
return timeline;
}
async function mountedRowCount(page: Page) {
return page.getByTestId("message-row").count();
}
test.beforeAll(async () => {
test.setTimeout(90_000);
await assertRelaySeeded();
});
test("deep local-relay timeline remains bounded and every row is reachable", async ({
page,
}, testInfo) => {
testInfo.setTimeout(10 * 60_000);
const fixtureSecond = Math.floor(Date.now() / 1000) - 1;
const fixture = denseSecondWall({
channelId: GENERAL_CHANNEL_ID,
count: DEEP_ROW_COUNT,
second: fixtureSecond,
});
const expected = await seedScenario(fixture, {
relayHttpUrl: RELAY_HTTP,
concurrency: 64,
});
expect(expected).toHaveLength(DEEP_ROW_COUNT);
const expectedIdHash = await sha256(expected.map(({ id }) => id));
const expectedOrder = expected.map(({ id }) => id).sort();
const expectedRank = new Map(expectedOrder.map((id, index) => [id, index]));
const measurement = await measureAction(page, async () => {
const timeline = await openGeneral(page);
const initialMountedRows = await mountedRowCount(page);
let duplicateSnapshots = 0;
let orderingViolations = 0;
let orderDirection: "ascending" | "descending" | null = null;
let maxMountedRows = initialMountedRows;
let renderPendingTimeouts = 0;
const seen = new Set<string>();
const collect = async () => {
const ids = await timeline.evaluate((element) =>
Array.from(element.querySelectorAll<HTMLElement>("[data-message-id]"))
.map((row) => row.dataset.messageId)
.filter((id): id is string => Boolean(id)),
);
if (new Set(ids).size !== ids.length) duplicateSnapshots += 1;
const ranks = ids.map((id) => expectedRank.get(id) ?? -1);
if (ranks.includes(-1)) orderingViolations += 1;
for (let index = 1; index < ranks.length; index += 1) {
if (ranks[index] === ranks[index - 1]) {
orderingViolations += 1;
continue;
}
const direction =
ranks[index] > ranks[index - 1] ? "ascending" : "descending";
orderDirection ??= direction;
if (direction !== orderDirection) orderingViolations += 1;
}
for (const id of ids) seen.add(id);
};
await collect();
await timeline.hover();
const passes: Array<{
pass: number;
before: number;
after: number;
continuationRequests: number;
scrollTop: number;
scrollHeight: number;
}> = [];
let stalled = 0;
let stallReason = "row-target-reached";
const maxPasses = Math.ceil(DEEP_ROW_COUNT / 25) + 32;
for (
let pass = 0;
pass < maxPasses && seen.size < DEEP_ROW_COUNT;
pass += 1
) {
const before = seen.size;
// Keep wheel increments below a typical virtualized page height. Large
// jumps can legitimately skip transiently mounted rows and turn a DOM
// reachability assertion into a collector bug.
for (let step = 0; step < 128; step += 1) {
await page.mouse.wheel(0, -400);
await page.waitForTimeout(20);
await collect();
const atTop = await timeline.evaluate(
(element) => (element as HTMLDivElement).scrollTop <= 1,
);
if (atTop) break;
}
await expect
.poll(
async () => {
await collect();
return seen.size;
},
{ timeout: 4_000 },
)
.toBeGreaterThan(before)
.catch(() => {});
try {
await expect(page.locator('[data-render-pending="true"]')).toHaveCount(
0,
{ timeout: 5_000 },
);
} catch {
renderPendingTimeouts += 1;
}
await collect();
const mounted = await mountedRowCount(page);
maxMountedRows = Math.max(maxMountedRows, mounted);
const state = await timeline.evaluate((element) => ({
continuationRequests:
(
window as typeof window & {
__CHANNEL_WINDOW_FETCH_COUNT__?: number;
}
).__CHANNEL_WINDOW_FETCH_COUNT__ ?? 0,
scrollTop: (element as HTMLDivElement).scrollTop,
scrollHeight: (element as HTMLDivElement).scrollHeight,
}));
passes.push({ pass, before, after: seen.size, ...state });
stalled = seen.size === before ? stalled + 1 : 0;
if (stalled >= 8) {
stallReason = "eight-passes-without-new-rows";
break;
}
stallReason = "pass-limit-reached";
}
if (seen.size >= DEEP_ROW_COUNT) stallReason = "row-target-reached";
return {
initialMountedRows,
finalMountedRows: await mountedRowCount(page),
maxMountedRows,
duplicateSnapshots,
orderingViolations,
orderDirection,
renderPendingTimeouts,
reachableRows: seen.size,
reachableIdHash: await sha256([...seen]),
continuationRequests: await page.evaluate(
() =>
(
window as typeof window & {
__CHANNEL_WINDOW_FETCH_COUNT__?: number;
}
).__CHANNEL_WINDOW_FETCH_COUNT__ ?? 0,
),
passCount: passes.length,
stallReason,
passes,
};
});
await writeArtifact(testInfo, {
schemaVersion: 1,
fixtureVersion: FIXTURE_VERSION,
fixtureSecond,
scenario: "deep-local-relay-timeline",
relayUrl: RELAY_HTTP,
rowCount: DEEP_ROW_COUNT,
expectedIdHash,
observed: measurement.result,
measurement: {
wallMs: measurement.wallMs,
metrics: measurement.metrics,
},
});
expect(measurement.result.initialMountedRows).toBeGreaterThan(0);
expect(measurement.result.maxMountedRows).toBeLessThanOrEqual(300);
expect(measurement.result.duplicateSnapshots).toBe(0);
expect(measurement.result.orderingViolations).toBe(0);
expect(measurement.result.renderPendingTimeouts).toBe(0);
expect(measurement.result.reachableRows).toBe(DEEP_ROW_COUNT);
expect(measurement.result.reachableIdHash).toBe(expectedIdHash);
});
+148
View File
@@ -0,0 +1,148 @@
#!/usr/bin/env bash
# Run deterministic desktop correctness smoke; timing metrics are informational.
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
ARTIFACT_DIR="${BUZZ_RELEASE_SMOKE_ARTIFACT_DIR:-${ROOT}/desktop/test-results/release-smoke}"
DB_NAME="${BUZZ_RELEASE_SMOKE_DB:-buzz_release_smoke_${$}}"
REDIS_DB="${BUZZ_RELEASE_SMOKE_REDIS_DB:-}"
LOCK_DIR="${TMPDIR:-/tmp}/buzz-desktop-release-smoke.lock"
RELAY_PID=""
LOCK_HELD=false
free_port() {
python3 - <<'PY'
import socket
with socket.socket() as sock:
sock.bind(("127.0.0.1", 0))
print(sock.getsockname()[1])
PY
}
RELAY_PORT="${BUZZ_RELEASE_SMOKE_RELAY_PORT:-$(free_port)}"
HEALTH_PORT="${BUZZ_RELEASE_SMOKE_HEALTH_PORT:-$(free_port)}"
METRICS_PORT="${BUZZ_RELEASE_SMOKE_METRICS_PORT:-$(free_port)}"
COMMUNITY_HOST="localhost:${RELAY_PORT}"
RELAY_HTTP_URL="http://${COMMUNITY_HOST}"
STARTED_AT="$(date +%s)"
log() { printf '[desktop-release-smoke] %s\n' "$*"; }
phase() {
local name="$1" start="$2"
printf '{"phase":"%s","duration_ms":%d}\n' "$name" "$(( ($(date +%s) - start) * 1000 ))" >> "${ARTIFACT_DIR}/phases.jsonl"
}
cleanup() {
local status=$?
trap - EXIT INT TERM
if [[ -n "${RELAY_PID}" ]]; then
kill "${RELAY_PID}" 2>/dev/null || true
for _ in $(seq 1 50); do
kill -0 "${RELAY_PID}" 2>/dev/null || break
sleep 0.1
done
kill -9 "${RELAY_PID}" 2>/dev/null || true
fi
docker exec buzz-redis redis-cli -n "${REDIS_DB}" FLUSHDB >/dev/null 2>&1 || true
docker exec buzz-postgres dropdb -U buzz --if-exists "${DB_NAME}" >/dev/null 2>&1 || true
if [[ "${LOCK_HELD}" == true ]]; then rmdir "${LOCK_DIR}" 2>/dev/null || true; fi
exit "${status}"
}
trap cleanup EXIT INT TERM
# Shared Docker services expose one Redis instance and a finite database index
# space. Serialize automatic allocation; callers that deliberately own an
# isolated Redis DB may opt out by setting BUZZ_RELEASE_SMOKE_REDIS_DB.
if [[ -z "${REDIS_DB}" ]]; then
mkdir "${LOCK_DIR}" 2>/dev/null || {
log "another release-smoke run owns ${LOCK_DIR}; set BUZZ_RELEASE_SMOKE_REDIS_DB only for an isolated runner"
exit 1
}
LOCK_HELD=true
REDIS_DB=15
fi
mkdir -p "${ARTIFACT_DIR}"
: > "${ARTIFACT_DIR}/phases.jsonl"
cd "${ROOT}"
phase_start="$(date +%s)"
log "starting backing services"
docker compose up -d postgres redis minio minio-init
for container in buzz-postgres buzz-redis buzz-minio; do
for _ in $(seq 1 60); do
[[ "$(docker inspect --format='{{.State.Health.Status}}' "${container}" 2>/dev/null || true)" == "healthy" ]] && break
sleep 1
done
[[ "$(docker inspect --format='{{.State.Health.Status}}' "${container}" 2>/dev/null || true)" == "healthy" ]] || {
docker logs "${container}" || true
exit 1
}
done
phase services "${phase_start}"
phase_start="$(date +%s)"
log "creating isolated database ${DB_NAME}"
docker exec buzz-postgres createdb -U buzz "${DB_NAME}"
export PGHOST=localhost PGPORT=5432 PGUSER=buzz PGPASSWORD=buzz_dev PGDATABASE="${DB_NAME}"
export PGSCHEMA_PLAN_HOST=localhost PGSCHEMA_PLAN_PORT=5432 PGSCHEMA_PLAN_DB="${DB_NAME}"
export PGSCHEMA_PLAN_USER=buzz PGSCHEMA_PLAN_PASSWORD=buzz_dev
./bin/pgschema apply --file schema/schema.sql --auto-approve
docker exec -i -e PGPASSWORD=buzz_dev buzz-postgres \
psql -U buzz -d "${DB_NAME}" -v ON_ERROR_STOP=1 < scripts/attach-schema-partitions.sql
BUZZ_DB_NAME="${DB_NAME}" BUZZ_COMMUNITY_HOST="${COMMUNITY_HOST}" ./scripts/setup-desktop-test-data.sh
docker exec buzz-redis redis-cli -n "${REDIS_DB}" FLUSHDB >/dev/null
phase database "${phase_start}"
phase_start="$(date +%s)"
if [[ -n "${BUZZ_E2E_RELAY_BIN:-}" ]]; then
RELAY_BIN="${BUZZ_E2E_RELAY_BIN}"
else
log "building relay"
cargo build --profile ci -p buzz-relay
RELAY_BIN="${ROOT}/target/ci/buzz-relay"
fi
log "starting relay at ${RELAY_HTTP_URL}"
env \
DATABASE_URL="postgres://buzz:buzz_dev@localhost:5432/${DB_NAME}" \
REDIS_URL="redis://localhost:6379/${REDIS_DB}" \
RELAY_URL="ws://${COMMUNITY_HOST}" \
BUZZ_BIND_ADDR="127.0.0.1:${RELAY_PORT}" \
BUZZ_HEALTH_PORT="${HEALTH_PORT}" \
BUZZ_METRICS_PORT="${METRICS_PORT}" \
BUZZ_REQUIRE_AUTH_TOKEN=false \
BUZZ_RECONCILE_CHANNELS=true \
BUZZ_RATE_LIMIT_HUMAN_MESSAGES_PER_MIN=1000000 \
BUZZ_RATE_LIMIT_HUMAN_API_CALLS_PER_MIN=1000000 \
BUZZ_RATE_LIMIT_HUMAN_WS_EVENTS_PER_SEC=100000 \
"${RELAY_BIN}" > "${ARTIFACT_DIR}/relay.log" 2>&1 &
RELAY_PID=$!
ready=false
for _ in $(seq 1 300); do
kill -0 "${RELAY_PID}" 2>/dev/null || { cat "${ARTIFACT_DIR}/relay.log"; exit 1; }
if curl --silent --fail --max-time 1 "http://127.0.0.1:${HEALTH_PORT}/_readiness" >/dev/null; then
ready=true
break
fi
sleep 0.1
done
[[ "${ready}" == true ]] || { cat "${ARTIFACT_DIR}/relay.log"; exit 1; }
phase relay "${phase_start}"
phase_start="$(date +%s)"
if [[ "${BUZZ_RELEASE_SMOKE_NO_BUILD:-0}" == "1" ]]; then
log "reusing existing desktop E2E bundle"
else
log "building desktop E2E bundle"
pnpm -C desktop build:e2e
fi
phase build "${phase_start}"
phase_start="$(date +%s)"
log "running release smoke"
BUZZ_E2E_RELAY_URL="${RELAY_HTTP_URL}" \
BUZZ_RELEASE_SMOKE_ARTIFACT_DIR="${ARTIFACT_DIR}" \
pnpm -C desktop exec playwright test --config=playwright.release-smoke.config.ts
phase smoke "${phase_start}"
phase total "${STARTED_AT}"
log "artifacts: ${ARTIFACT_DIR}"