fix(enterprise): ship enterprise package in prod image + S3, analytics, tracing, queue fixes (#342)

* fix(enterprise): ship enterprise pkg in prod image, full license features, tracing key fallback

docker/Dockerfile: COPY packages/enterprise manifest+src into the production stage.
Without it, apps/api's workspace link to @snapotter/enterprise dangles and every
import() throws (silently caught), so all 19 enterprise features failed closed
(enterprise.active=false) regardless of a valid license.

scripts/generate-license.mjs: sync PLAN_FEATURES with packages/enterprise/src/license.ts
so a --plan enterprise license unlocks all 19 features (was 8) and team unlocks 8.

apps/api/src/tracing.ts: accept SNAPOTTER_LICENSE_KEY as a fallback to LICENSE_KEY so
distributed_tracing activates with the same key as the rest of the app.

* fix(docker): keep scripts/bake-analytics.mjs in build context

.dockerignore excluded the whole scripts/ dir (PR #82, V1 hardening), but
docker/Dockerfile later added 'COPY scripts/bake-analytics.mjs' for the analytics
bake step. A clean production image build therefore fails with
'scripts/bake-analytics.mjs: not found'. The published image build is gated off in
CI so this latent break went unnoticed. Exclude scripts/* but re-include the one
file the Dockerfile needs.

* fix: S3 upload stream, analytics bake reaches API, dedupe retention field, reconcile orphan jobs

storage-s3.ts: wrap the upload AsyncIterable in Readable.from() so @aws-sdk/lib-storage
accepts it. STORAGE_MODE=s3 file uploads failed with 'Body Data is unsupported format'
for every tool because a bare async generator is not a Readable.

docker/Dockerfile: COPY the builder-baked analytics baked.ts into the API runtime stage.
The API re-copied the committed (off) baked.ts from the build context, so the
SNAPOTTER_ANALYTICS build arg had no effect on the API -- and since the SPA reads
/api/v1/config/analytics, analytics was off everywhere regardless of the arg.

settings-dialog.tsx: remove the duplicate tempFileMaxAgeHours control under Data
Retention; it bound the same setting key as the File Management control with a different
default, so editing either silently overwrote the other.

apps/api/src/index.ts: reconcile orphaned job rows (empty tool_id, never enqueued to
BullMQ) at boot so they don't sit in processing/queued forever and inflate the per-user
concurrent-job count and the upgrade-check in-flight gate.

* fix(web): style the SSO login buttons (they referenced undefined theme tokens)

The OIDC/SAML 'Sign in with <provider>' buttons used bg-secondary /
text-secondary-foreground, which the web theme never defines (it has primary,
background, foreground, muted, border, card, primary-subtle). Those classes resolved
to nothing, so the buttons rendered as bare unstyled text on the login page.

Restyle: the optional (non-enforced) buttons become white-card outline buttons with a
key icon and an orange hover tint, secondary to the primary Login button; the
SSO-enforced buttons become solid primary with the icon.

* fix: gate S3 behind license, custom-role enterprise perms, wire retention UI, cleanup

S3 is a licensed feature, but shipping packages/enterprise in every image removed the
implicit gate, so STORAGE_MODE=s3 worked without a license. Enforce
isFeatureEnabled('s3_storage') at boot and fail fast if unlicensed.

Custom roles can now be granted security:manage / compliance:manage / webhooks:manage
(roles.ts ALL_PERMISSIONS + the Roles UI) so admins can build least-privilege
compliance/security roles instead of only the built-in admin role.

retentionSweep now reads the jobsRetentionDays / auditRetentionDays DB settings the
System Settings UI writes (env vars become the fallback default), mirroring how the
temp-file sweep reads tempFileMaxAgeHours. Previously those two UI controls were no-ops.

Cleanup: drop the never-set snapotter_storage_bytes gauge and the unused
MAX_WORKSPACE_SIZE_GB env var; emit tool_client_error to PostHog from the web
ErrorBoundary (client crashes were not reaching analytics); add the Python
OpenTelemetry packages so the innermost sidecar.<script> span exports; fix the stale
'only local storage' line in the docs; delete two e2e-analytics specs that tested the
removed consent UI.

* fix(env): restore MAX_WORKSPACE_SIZE_GB default

security-auth-hardening.test.ts asserts env.MAX_WORKSPACE_SIZE_GB defaults to 10, so
the var is an intentional (tested) default, not dead code. Removing it in the cleanup
commit broke that unit test. Keep the declaration.

apps/api/src/index.ts

@@ -188,6 +188,27 @@ try {
   // Enterprise package not available
 }
 
+// S3 storage is a licensed feature (s3_storage). packages/enterprise now ships in
+// every image, so STORAGE_MODE=s3 would otherwise function without any license check.
+// Enforce the gate at boot so an unlicensed deploy fails fast rather than silently
+// writing data to S3 it isn't entitled to use.
+if (env.STORAGE_MODE === "s3") {
+  let s3Licensed = false;
+  try {
+    const { isFeatureEnabled } = await import("@snapotter/enterprise");
+    s3Licensed = isFeatureEnabled("s3_storage");
+  } catch {
+    s3Licensed = false;
+  }
+  if (!s3Licensed) {
+    console.error(
+      "[FATAL] STORAGE_MODE=s3 requires a license that includes the s3_storage feature. " +
+        "Set a valid SNAPOTTER_LICENSE_KEY (team or enterprise plan) or use STORAGE_MODE=local.",
+    );
+    process.exit(1);
+  }
+}
+
 // Start the cooperative cancellation listener (Redis pub/sub)
 await startCancelListener();
 
@@ -625,6 +646,22 @@ if (await shouldRunStartupCleanup()) {
 // Start BullMQ worker pools (after route registration so the tool registry is full)
 startWorkers();
 
+// Reconcile orphaned job rows. A jobs row created without a tool_id/pool (e.g. an
+// SSE-progress placeholder for a clientJobId whose client then disconnected) is
+// never enqueued to BullMQ, so unlike a genuinely interrupted job (which BullMQ's
+// stalled-detection requeues) it would sit in 'processing'/'queued' forever --
+// inflating the per-user concurrent-job count and the upgrade-check in-flight gate.
+// Only rows with an empty tool_id are touched, so real jobs are never affected.
+void db
+  .execute(
+    sql`UPDATE jobs SET status = 'failed', error = '{"message":"Orphaned job reconciled at startup"}'::jsonb, completed_at = now() WHERE status IN ('processing','queued') AND (tool_id IS NULL OR tool_id = '')`,
+  )
+  .then((r) => {
+    const n = (r as { rowCount?: number }).rowCount ?? 0;
+    if (n > 0) app.log.info({ count: n }, "Reconciled orphaned job rows at startup");
+  })
+  .catch((err) => app.log.warn({ err }, "Orphaned-job reconciliation failed"));
+
 // Warm the per-pool QueueEvents consumers so the first synchronous tool request
 // after boot does not pay the lazy-connect cost (and cannot miss a fast job's
 // completion event). Non-blocking: a slow/unreachable Redis must not stall boot;

apps/api/src/jobs/system-jobs.ts

@@ -15,6 +15,7 @@ import { env } from "../config.js";
 import { db, schema } from "../db/index.js";
 import { getMaxAgeMs } from "../lib/cleanup.js";
 import { deletePrefix, listJobDirs, type ObjectInfo } from "../lib/object-storage.js";
+import { getSettingNumber } from "../lib/settings-helpers.js";
 import { runAuditArchive } from "./audit-archive.js";
 import { getQueue } from "./queues.js";
 import { runSiemForward } from "./siem-forward.js";
@@ -275,15 +276,20 @@ async function retentionSweep(): Promise<void> {
     WHERE u.legal_hold = true OR t.legal_hold = true
   )`;
 
-  if (env.JOBS_RETENTION_DAYS > 0) {
+  // The Data Retention settings UI writes jobsRetentionDays / auditRetentionDays to
+  // the DB. Honor those (the env vars are the fallback default), mirroring how the
+  // temp-file sweep reads tempFileMaxAgeHours; otherwise the UI controls are no-ops.
+  const jobsRetentionDays = await getSettingNumber("jobsRetentionDays", env.JOBS_RETENTION_DAYS);
+  if (jobsRetentionDays > 0) {
     await db.execute(
       sql`DELETE FROM jobs
-        WHERE created_at < now() - ${env.JOBS_RETENTION_DAYS} * interval '1 day'
+        WHERE created_at < now() - ${jobsRetentionDays} * interval '1 day'
         AND status IN ('completed', 'failed', 'canceled')
         AND (user_id IS NULL OR user_id NOT IN ${heldUsersSubquery})`,
     );
   }
-  if (env.AUDIT_RETENTION_DAYS > 0) {
+  const auditRetentionDays = await getSettingNumber("auditRetentionDays", env.AUDIT_RETENTION_DAYS);
+  if (auditRetentionDays > 0) {
     const tamperResult = await db
       .select({ value: schema.settings.value })
       .from(schema.settings)
@@ -296,7 +302,7 @@ async function retentionSweep(): Promise<void> {
     if (!isTamperResistant) {
       await db.execute(
         sql`DELETE FROM audit_log
-          WHERE created_at < now() - ${env.AUDIT_RETENTION_DAYS} * interval '1 day'
+          WHERE created_at < now() - ${auditRetentionDays} * interval '1 day'
           AND (actor_id IS NULL OR actor_id NOT IN ${heldUsersSubquery})`,
       );
     }

apps/api/src/lib/metrics.ts

@@ -5,7 +5,7 @@
  * and a metricsText() function that appends live queue-depth gauges
  * from BullMQ before returning the scrape payload.
  */
-import { Counter, collectDefaultMetrics, Gauge, Histogram, Registry } from "prom-client";
+import { Counter, collectDefaultMetrics, Histogram, Registry } from "prom-client";
 import { perPoolCounts } from "../jobs/queues.js";
 
 export const registry = new Registry();
@@ -34,13 +34,6 @@ export const requestDuration = new Histogram({
   registers: [registry],
 });
 
-export const storageUsage = new Gauge({
-  name: "snapotter_storage_bytes",
-  help: "Storage usage in bytes",
-  labelNames: ["category"] as const,
-  registers: [registry],
-});
-
 export const authAttempts = new Counter({
   name: "snapotter_auth_attempts_total",
   help: "Authentication attempts",

apps/api/src/routes/roles.ts

@@ -22,6 +22,9 @@ const ALL_PERMISSIONS: Permission[] = [
   "features:manage",
   "system:health",
   "audit:read",
+  "compliance:manage",
+  "webhooks:manage",
+  "security:manage",
 ];
 
 const roleNameField = z

apps/api/src/tracing.ts

@@ -79,7 +79,10 @@ const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT;
 if (endpoint) {
   try {
     const enterprise = await import("@snapotter/enterprise");
-    const licenseKey = process.env.LICENSE_KEY ?? "";
+    // The rest of the app reads SNAPOTTER_LICENSE_KEY (env.ts / index.ts). Accept
+    // either here so distributed_tracing activates with the same key as every other
+    // enterprise feature; LICENSE_KEY kept as an override for preload-only setups.
+    const licenseKey = process.env.LICENSE_KEY ?? process.env.SNAPOTTER_LICENSE_KEY ?? "";
     if (licenseKey) {
       enterprise.initEnterprise(licenseKey);
     }