mirror of
https://github.com/snapotter-hq/SnapOtter.git
synced 2026-08-03 07:46:42 +02:00
fix(telemetry): data-quality pass (opt-in noise, onboarding split, file_count, OIDC) (#614)
Five fixes to the PostHog event stream, from an audit of what we actually collect versus what's flowing in. Each one is test-first. ## What changed **Silenced the `$opt_in` noise.** `initAnalytics` called `opt_in_capturing()` on every page load to clear a stale opt-out flag, and posthog-js emits an `$opt_in` event on every call. That was 10k+ events a month (up to 55 per user) carrying no signal: analytics is on by default with an admin opt-out, so there is no per-user consent to record. Both call sites now pass `captureEventName: false`. **Split the onboarding survey out of `feedback_submitted`.** The onboarding usage survey rode the same event as real feedback, so about 93% of "feedback" was actually onboarding profiling. It now emits `onboarding_survey_submitted`, so feedback metrics mean feedback again. **Set `pipeline_executed.file_count`.** It was declared in the properties interface but never populated. A pure `pipelineExecutedProps` helper now derives it (batch size for a batch run, else 1) and is shared by the success and failure paths, which also drops a duplicated payload. **Tracked OIDC login failures.** All six OIDC callback failure branches bumped the Prometheus counter and wrote an audit log but never emitted `auth_login_failed`. A `recordOidcFailure` helper mirrors the password path. **Added `TELEMETRY.md`.** A contributor-facing event dictionary: every event, its properties, where it fires, and the privacy invariants, with the allowlists as source of truth. A drift test fails if any `ANALYTICS_EVENTS` value goes undocumented. I left the published telemetry guide (`apps/docs/guide/telemetry.md`) alone. It is high-level and still accurate, and editing it would pull in the 21-locale stale-gate for no gain. ## Verification - Unit (63 tests): `analytics-events`, `telemetry-doc-drift`, `api/analytics`, `web/analytics`, `worker.behavior` - Integration (41 tests): `oidc-auth`, `feedback` - Full typecheck across all 9 workspaces - Biome clean on the changed files All green locally.
This commit is contained in:
@@ -295,6 +295,41 @@ describe("captureFeedback", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("routes an onboarding survey to its own event, not feedback_submitted", async () => {
|
||||
bakedConfig.enabled = true;
|
||||
bakedConfig.posthogApiKey = "phc_test_key";
|
||||
await mod.initAnalytics();
|
||||
|
||||
await mod.captureFeedback(
|
||||
{
|
||||
source: "onboarding",
|
||||
survey_id: "onboarding-usage-v1",
|
||||
contact_ok: false,
|
||||
usage_type: "personal",
|
||||
},
|
||||
"distinct-onboarding",
|
||||
);
|
||||
|
||||
expect(mockCapture).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
distinctId: "distinct-onboarding",
|
||||
event: "onboarding_survey_submitted",
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps genuine feedback sources on the feedback_submitted event", async () => {
|
||||
bakedConfig.enabled = true;
|
||||
bakedConfig.posthogApiKey = "phc_test_key";
|
||||
await mod.initAnalytics();
|
||||
|
||||
await mod.captureFeedback({ source: "global", contact_ok: false, message: "A message" });
|
||||
|
||||
expect(mockCapture).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ event: "feedback_submitted" }),
|
||||
);
|
||||
});
|
||||
|
||||
it("drops an empty important_areas array instead of forwarding it", async () => {
|
||||
bakedConfig.enabled = true;
|
||||
bakedConfig.posthogApiKey = "phc_test_key";
|
||||
|
||||
@@ -252,3 +252,40 @@ describe("worker result payload behavior", () => {
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("pipelineExecutedProps", () => {
|
||||
it("reports the batch file count for a batch-finalize pipeline", async () => {
|
||||
const { pipelineExecutedProps } = await loadWorker();
|
||||
|
||||
expect(
|
||||
pipelineExecutedProps(
|
||||
{ kind: "batch-finalize", totalFiles: 5 },
|
||||
3,
|
||||
["resize", "compress", "watermark"],
|
||||
1200,
|
||||
"completed",
|
||||
),
|
||||
).toEqual({
|
||||
step_count: 3,
|
||||
tool_ids: ["resize", "compress", "watermark"],
|
||||
is_batch: true,
|
||||
file_count: 5,
|
||||
duration_ms: 1200,
|
||||
status: "completed",
|
||||
});
|
||||
});
|
||||
|
||||
it("defaults file_count to 1 for a single-file pipeline-finalize", async () => {
|
||||
const { pipelineExecutedProps } = await loadWorker();
|
||||
|
||||
expect(
|
||||
pipelineExecutedProps(
|
||||
{ kind: "pipeline-finalize" },
|
||||
2,
|
||||
["grayscale", "resize"],
|
||||
800,
|
||||
"failed",
|
||||
),
|
||||
).toMatchObject({ is_batch: false, file_count: 1, status: "failed" });
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user