fix(telemetry): data-quality pass (opt-in noise, onboarding split, file_count, OIDC) (#614)

Five fixes to the PostHog event stream, from an audit of what we actually collect versus what's flowing in. Each one is test-first.

## What changed

**Silenced the `$opt_in` noise.** `initAnalytics` called `opt_in_capturing()` on every page load to clear a stale opt-out flag, and posthog-js emits an `$opt_in` event on every call. That was 10k+ events a month (up to 55 per user) carrying no signal: analytics is on by default with an admin opt-out, so there is no per-user consent to record. Both call sites now pass `captureEventName: false`.

**Split the onboarding survey out of `feedback_submitted`.** The onboarding usage survey rode the same event as real feedback, so about 93% of "feedback" was actually onboarding profiling. It now emits `onboarding_survey_submitted`, so feedback metrics mean feedback again.

**Set `pipeline_executed.file_count`.** It was declared in the properties interface but never populated. A pure `pipelineExecutedProps` helper now derives it (batch size for a batch run, else 1) and is shared by the success and failure paths, which also drops a duplicated payload.

**Tracked OIDC login failures.** All six OIDC callback failure branches bumped the Prometheus counter and wrote an audit log but never emitted `auth_login_failed`. A `recordOidcFailure` helper mirrors the password path.

**Added `TELEMETRY.md`.** A contributor-facing event dictionary: every event, its properties, where it fires, and the privacy invariants, with the allowlists as source of truth. A drift test fails if any `ANALYTICS_EVENTS` value goes undocumented.

I left the published telemetry guide (`apps/docs/guide/telemetry.md`) alone. It is high-level and still accurate, and editing it would pull in the 21-locale stale-gate for no gain.

## Verification

- Unit (63 tests): `analytics-events`, `telemetry-doc-drift`, `api/analytics`, `web/analytics`, `worker.behavior`
- Integration (41 tests): `oidc-auth`, `feedback`
- Full typecheck across all 9 workspaces
- Biome clean on the changed files

All green locally.
This commit is contained in:
SnapOtter
2026-07-21 23:36:02 +08:00
committed by GitHub
parent 6a0768b39d
commit b20bca3c3c
12 changed files with 285 additions and 27 deletions
+7 -2
View File
@@ -2,8 +2,8 @@ import { ANALYTICS_EVENTS } from "@snapotter/shared";
import { describe, expect, it } from "vitest";
describe("ANALYTICS_EVENTS", () => {
it("has exactly 24 event keys", () => {
expect(Object.keys(ANALYTICS_EVENTS)).toHaveLength(24);
it("has exactly 25 event keys", () => {
expect(Object.keys(ANALYTICS_EVENTS)).toHaveLength(25);
});
it("contains the expected keys", () => {
@@ -20,6 +20,7 @@ describe("ANALYTICS_EVENTS", () => {
expect(ANALYTICS_EVENTS).toHaveProperty("AI_BUNDLE_PROMPTED");
expect(ANALYTICS_EVENTS).toHaveProperty("BATCH_PROCESSED");
expect(ANALYTICS_EVENTS).toHaveProperty("FEEDBACK_SUBMITTED");
expect(ANALYTICS_EVENTS).toHaveProperty("ONBOARDING_SURVEY_SUBMITTED");
expect(ANALYTICS_EVENTS).toHaveProperty("SPONSOR_CLICKED");
expect(ANALYTICS_EVENTS).toHaveProperty("INSTANCE_STARTED");
expect(ANALYTICS_EVENTS).toHaveProperty("EDITOR_OPENED");
@@ -59,6 +60,10 @@ describe("ANALYTICS_EVENTS", () => {
expect(ANALYTICS_EVENTS.FEEDBACK_SUBMITTED).toBe("feedback_submitted");
});
it("ONBOARDING_SURVEY_SUBMITTED has the correct snake_case value", () => {
expect(ANALYTICS_EVENTS.ONBOARDING_SURVEY_SUBMITTED).toBe("onboarding_survey_submitted");
});
it("INSTANCE_STARTED has the correct snake_case value", () => {
expect(ANALYTICS_EVENTS.INSTANCE_STARTED).toBe("instance_started");
});