Commit Graph
112 Commits
Author SHA1 Message Date
coo1estguy 7b4d7f8efd fix: restore scheduled queues from postgres and recover consumer groups after redis data loss 2026-09-29 14:33:41 +05:30
coo1estguy 35a34df2fe refactor: extract worker bootstrap runtime and remove unused code 2026-09-29 13:10:15 +05:30
coo1estguy 880fe32b03 perf: rebuild profiling harness and reduce per-message pipeline cost
- Rebuild profiling suite with compose CLI, in-network k6 load generator, and production-shaped tiers
- Add fused pipeline mode for in-memory single-user stage hand-offs
- Route directly by contact ID and defer template rendering to delivery
- Add Redis connection sharing, auto-pipelining, stream read coalescing, and chunked provider throttling
- Switch hot message_logs and delivery_outbox inserts to unnest bulk SQL queries
- Drop redundant indexes on message_logs
- Optimize worker batching, flush concurrency, and dataloader windowing
2026-09-29 00:35:01 +05:30
coo1estguy c58b5e9d7b removed scaling doc [skip ci] 2026-09-28 14:14:53 +05:30
coo1estguy 88a13de2c2 fix(ci): build root package before typechecking workspace packages
Workspace packages import 'notifkit' which resolves to root dist/index.d.mts. Running build first ensures declaration files exist before workspace typecheck runs.
2026-09-28 13:14:53 +05:30
coo1estguy 516ea88bb1 fix(ci): resolve TypeScript typecheck OOM and add workspace typecheck step
- Scope root tsconfig to exclude workspace packages and restore declaration resolution

- Align MCP package zod dependency to ^4.6.5 and update record schemas to explicit key/value syntax

- Upgrade ai dependency to ^7.0.118 to restore bundled dist/index.d.ts declarations

- Fix rate limiter inFlightLease promise initialization in TS strict mode

- Add typecheck:workspaces script and wire into CI workflow
2026-09-28 13:11:07 +05:30
coo1estguy 6b35029577 remove tsc checks from pre commit 2026-09-27 19:02:18 +05:30
coo1estguy 8af3f36833 refactor(profiling): simplify profiling suite and document scaling bottlenecks 2026-09-27 18:00:37 +05:30
coo1estguy ad89e9c66d fix(api): fix project ID typing in auth cache and undefined url reference 2026-09-27 16:16:08 +05:30
coo1estguy 33825bb5f0 feat(profiling): migrate load generator to containerized k6 with warm-up phase
- Run k6 inside Docker bridge network to eliminate host networking and Node loop overhead
- Add dual-scenario execution for 45s warm-up and 180s steady-state isolation
- Track custom k6 Trend metrics and parse steady-state latency percentiles
- Support --warmup and --vus CLI flags in test runner
2026-09-27 15:47:48 +05:30
coo1estguy c82d424b60 perf(queue): add event loop write coalescing for stream publish
- Buffer concurrent publish and publishBatch calls into a single pipeline flush per event loop tick
- Apply MAXLEN ~ trimming only to the last command in each batch to reduce scan overhead
- Throttle periodic stream length monitoring to once every 5 seconds
2026-09-27 15:47:28 +05:30
coo1estguy d33edac03d perf(api): avoid sha256 calculation on auth cache hits
- Key in-memory authCache by raw token instead of tokenHash

- Compute SHA-256 only on cache misses when querying database

- Store structured auth record to avoid string splitting and parseInt

- Maintain tokenHash-to-token reverse map for targeted cache invalidations
2026-09-26 23:30:27 +05:30
coo1estguy d3b249539d perf(profiling): expose node inspector port on api-server-1 in distributed compose 2026-09-26 23:24:01 +05:30
coo1estguy c8e443fd0f chore: ignore profiling benchmark results 2026-09-26 23:23:44 +05:30
coo1estguy 39da6c23d7 perf(api/notify): drop per-request log from info to debug
- Demote logger.info to logger.debug in notify handler

- Eliminate worker-thread pipe overhead from pino-pretty on the hot path
2026-09-26 23:22:42 +05:30
coo1estguy 7680570bee perf(api): stream utf-8 strings in readJsonBody to avoid buffer copy
- Accumulate UTF-8 strings directly instead of Buffer.concat and toString copies

- Add fallback handling for mock streams and non-string chunks
2026-09-26 23:21:31 +05:30
coo1estguy 77cd28c4ad perf(api/router): index routes by method and segment count
- Key route lookup table by METHOD:segmentCount to avoid scanning all routes

- Drop comparisons on /v1/notify from 35 routes to 4 candidates
2026-09-26 23:20:49 +05:30
coo1estguy a1670bfeec perf(api): wire in-memory rate limiter and use lazy url parsing on ingress
- Replace per-request synchronous Redis check with ApiRateLimiter in request pipeline

- Replace eager new URL() allocation with indexOf query pathname slice and lazy getters

- Invalidate in-memory rate limiter leases on API key broadcast
2026-09-26 23:20:15 +05:30
coo1estguy ca95c8b917 perf(rate-limiter): add sliding-window token leasing to cut redis roundtrips
- Add LUA_LEASE_API_RATE_LIMIT script and register leaseApiRateLimit custom command

- Implement ApiRateLimiter with in-memory token leasing and 5s lease target

- Coalesce concurrent requests on depleted lease to avoid thundering-herd refills

- Prefill tokens immediately on minute bucket transition

- Add unit tests covering memory caching, pre-fetching, concurrency, and fail-open
2026-09-26 23:16:19 +05:30
coo1estguy ceeecf3937 refactor(profiling): name service containers and update dependencies
- Upgrade zod to ^4.6.5 along with compatible dependency updates across ai, dotenv, drizzle-orm, ioredis, pino, postgres, and dev tools.

- Adjust tests/workers-ai.test.ts mock UUIDs to adhere to RFC 4122 v4 validation in Zod 4.

- Update tests/http.test.ts validation message expectations for Zod 4.

- Replace generic server container names with role-based definitions (api-server-1, api-server-2, enricher-worker, engine-worker, delivery-worker).

- Add serverNodes option in distributed profiling harness to route and health-check designated worker and API nodes.

- Add multi-run statistical metrics (mean, median, min, max, stdDev) reporting to money-scaling benchmark runner.
2026-09-26 16:49:23 +05:30
coo1estguy 9e6a9631db refactor: remove fast-path bypasses and restore engine decision stage 2026-09-25 23:29:07 +05:30
coo1estguy 85f92d8bda perf(profiling): adjust scaling concurrency and tier flag parsing
- Support budget tier arguments with or without leading dollar signs in test:money
- Tune worker concurrency and provider mock latency defaults for scaling tests
2026-09-25 22:47:13 +05:30
coo1estguy 4642308fe8 fix(delivery): enforce two-phase idempotency leases and stream retry tracking
- Add two-phase idempotency with acquireLease and markProcessed using a single-key SET NX fast path
- Return { ack: false } on lock collisions to leave messages pending in the stream instead of acknowledging them
- Bound provider dispatch timeout within the lock lease window to prevent duplicate delivery under slow responses
- Replace per-message Redis MULTI/INCR retry keys with Redis Stream PEL delivery counts
- Handle LockHeldError in BaseWorker to skip message acknowledgement when another worker holds the lock
2026-09-25 22:46:06 +05:30
coo1estguy 905485c0c7 perf: replace dynamic imports with static top-level imports
Eliminate dynamic imports on execution hot paths and module startup:
- Import drizzle-orm sql statically in delivery worker
- Import node:crypto and transport registry statically in API handlers
- Import provider SDKs statically in Twilio and FCM packages
- Replace root barrel imports in config and db with direct submodules to prevent circular evaluation
- Update workflow tests to preserve redis mock exports via importOriginal
2026-09-25 14:43:05 +05:30
coo1estguy 3be502943f perf(redis): pre-compile Lua scripts via defineCommand across services
- Register custom Lua scripts during RedisClient initialization via defineCommand
- Pre-compile checkApiRateLimit, throttleProvider, throttleUser, releaseLock, renewLock, and schedulerPoll
- Augment ioredis Redis and ChainableCommander type interfaces for custom commands
- Maintain eval fallbacks across callers for compatibility with test mocks
- Add test suite in tests/redis.test.ts verifying registration, execution, and fallbacks
2026-09-25 14:20:46 +05:30
coo1estguy 6c4315d9e7 perf(api): replace sorted sets with sliding-window counter in rate limiting
- Replace O(N) ZSET with O(1) sliding-window counter across two buckets
- Define LUA_SLIDING_WINDOW_COUNTER at module level to avoid per-request string allocations
- Eliminate crypto randomBytes generation on hot path
- Use Redis hash tags for cluster-safe multi-key evaluation
- Add tests for window weighting, TTL, and limit rejection
2026-09-25 13:52:01 +05:30
coo1estguy d439f051a6 fix(metrics): compute true stream backlog via consumer group lag and set retention to 100k
- Cap Redis stream retention at 100,000 messages (down from 10,000,000) in
  src/config/index.ts and .env.example to prevent unbounded memory growth.
- In src/services/api/handlers.ts, update getSystemMetrics to resolve stream
  queue depths using consumer group lag and in-flight pending entries (via
  XINFO GROUPS) instead of raw XLEN. This prevents acknowledged and delivered
  notifications from persisting as ghost backlog in the dashboard.
- Retain XLEN measurement for DEAD_LETTER queue where messages persist until
  replayed or deleted.
- Adjust dashboard capacity scaling to 100,000 items in AnalyticsPage.tsx and
  update backpressure warning threshold to 80,000 in DashboardHeader.tsx.
- Ensure 0-depth queues render empty (0% width) in the backlog gauge meter.
- Update queue test expectations in tests/queue.test.ts and add unit tests
  verifying consumer group lag resolution in tests/api-admin.test.ts.
2026-09-25 12:57:02 +05:30
coo1estguy c72718a69d feat(profiling): add containerized load, CPU scaling, and distributed budget test suite
- Provide Docker Compose test harness with CPU and memory resource limits\n- Add single-node CPU scaling benchmark suite (0.5 to 4.0 vCPU)\n- Add distributed multi-node scaling and budget benchmarks (, , /mo)\n- Support partitioned service topologies (dedicated API, pipeline worker, and delivery worker instances)\n- Implement configurable provider latency simulation with randomized network jitter\n- Support dynamic client and worker concurrency tuning
2026-09-25 00:41:02 +05:30
coo1estguy 4173537aa4 feat(provider-console): add mock latency support with jitter to ConsoleTransport
- Support latencyMs in ConsoleTransportOptions as a fixed number or range string (e.g. '60-150')\n- Apply randomized jitter between bounds to simulate real-world vendor network response times
2026-09-25 00:35:51 +05:30
coo1estguy 7624d5d3c2 perf(pipeline): collapse enricher-engine streams and add direct critical fast-path
- Collapse Enricher and Decision Engine into direct outbound dispatch: Enricher evaluates quiet hours, topic opt-outs, and renders templates in-memory before publishing directly to STREAMS.OUTBOUND_*, eliminating intermediate STREAMS.ENRICHED_* Redis stream hops
- Add transactional direct fast-path in /v1/notify: Critical priority notifications with inline user targets bypass inbound and decision queues, rendering templates from cache and dispatching directly to STREAMS.OUTBOUND_CRITICAL
- Reduce first-delivery latency from 6.2s to 0.38s and increase wall drain throughput by up to 58% on budget infrastructure
- Add unit and integration tests covering Enricher direct outbound dispatch and API transactional fast-path routing
2026-09-24 22:58:13 +05:30
coo1estguy c327a28933 perf(pipeline): batch stream ACKs, consolidate outbox writes, and remove blocking Redis round-trips
- Implement stream-keyed ACK buffering in StreamConsumer with periodic and capacity-triggered flushes
- Consolidate delivery outbox persistence to a single post-dispatch write, cutting PostgreSQL outbox writes by 50%
- Make retry key deletion and idempotency TTL extension non-blocking on message success
- Add host-unique consumer identifiers across distributed delivery, engine, and enricher workers
- Add unit tests verifying buffered stream ACK flushing behavior
2026-09-24 22:31:42 +05:30
coo1estguy 57e6197080 perf: optimize delivery pipeline batching, timers, and caching 2026-09-24 16:28:51 +05:30
coo1estguy dd48a16f5a chore: add funding.json and github funding config [skip ci] 2026-09-23 13:10:53 +05:30
coo1estguy fed2b27211 chore(release): bump version to 0.1.10 v0.1.10 2026-09-19 23:18:20 +05:30
coo1estguy 2c106f1a3f chore(release): bump version to 0.1.9 v0.1.9 2026-09-19 22:21:49 +05:30
coo1estguy 46a08915e7 fix(security): reject DLQ replay/delete when message projectId is absent for project-scoped callers 2026-09-19 15:30:32 +05:30
coo1estguy 020c71bd9f fix(security): require UNSUBSCRIBE_SECRET in production to prevent silent opt-out drops 2026-09-19 15:27:34 +05:30
coo1estguy b305d0ea03 fix(security): restrict CORS to configured origin for session-auth routes 2026-09-19 15:25:46 +05:30
coo1estguy 7db1046582 chore: ignore skills-lock.json 2026-09-19 15:02:09 +05:30
coo1estguy ace80fe5a1 fix(api): remove redundant decodeURIComponent in deleteSuppression 2026-09-19 15:00:59 +05:30
coo1estguy 1a6ba84b3a fix(api): handle malformed URI parameters gracefully in router 2026-09-19 14:58:53 +05:30
coo1estguy d876be851f fix(api): scope getUserDetails message logs to user contacts 2026-09-19 14:57:24 +05:30
coo1estguy d0ca566d6c fix(api): prevent path prefix traversal in admin static handler 2026-09-19 14:54:28 +05:30
coo1estguy 1625cdc8f8 fix(api): protect metrics endpoint and prevent proxy header spoofing 2026-09-19 14:27:48 +05:30
coo1estguy 8c907fa6bc fix(api): isolate tenant metrics and scope global queue depths to admin 2026-09-19 14:23:50 +05:30
coo1estguy 151db527e1 fix(auth): add login rate limiting and eliminate timing side-channel 2026-09-19 14:18:35 +05:30
coo1estguy 6f64922151 fix(api): enforce project isolation on dead letter queue endpoints 2026-09-19 14:12:09 +05:30
coo1estguy f14478d7c1 fix(provider-discord): validate webhook url to prevent ssrf 2026-09-19 14:08:20 +05:30
coo1estguy d87b88d2eb chore(release): bump version to 0.1.8 v0.1.8 2026-09-15 23:11:53 +05:30
coo1estguy afbbaae72e refactor(dashboard): migrate to vite spa and standard shadcn ui
- migrate dashboard from Next.js to Vite SPA with React Router
- install standard shadcn/ui components using shadcn CLI
- use standard shadcn theme with default CSS variables
- handle optional project scoping in system metrics and health handlers
- update admin-static service to serve Vite dist and proxy dev server
2026-09-15 23:07:59 +05:30