mirror of
https://github.com/snapotter-hq/SnapOtter.git
synced 2026-08-03 07:46:42 +02:00
* feat(infra): add dev compose stack with postgres and redis
* fix(infra): comment dev env defaults until wired; harden dev compose restart and start_period
* chore(deps): add pg driver and testcontainers for postgres migration
* feat(db): translate schema to drizzle pg-core (timestamptz, boolean, pgEnum, jsonb)
Schema translation (apps/api/src/db/schema.ts):
- sqlite-core -> pg-core, all 10 tables preserved 1:1
- integer(mode:'timestamp') -> timestamp({ withTimezone: true })
- integer(mode:'boolean') -> boolean
- jobs.status text enum -> pgEnum('job_status') with same 4 values
- 7 columns changed from text to jsonb: jobs.inputFiles, jobs.settings,
pipelines.steps, apiKeys.permissions, roles.permissions,
auditLog.details, userFiles.toolChain
- settings.value stays text, jobs.error stays text, jobs.progress stays real
jsonb call-site sweep (removed JSON.stringify on writes, JSON.parse on reads):
- apps/api/src/routes/roles.ts: permissions read/write (3 sites)
- apps/api/src/routes/api-keys.ts: permissions write + read (2 sites)
- apps/api/src/routes/audit-log.ts: details read (1 site)
- apps/api/src/routes/pipeline.ts: steps write + read (2 sites)
- apps/api/src/routes/progress.ts: inputFiles write (2 sites)
- apps/api/src/routes/tool-factory.ts: toolChain read + write (2 sites)
- apps/api/src/routes/user-files.ts: toolChain read + write (4 sites)
- apps/api/src/permissions.ts: roles.permissions read (1 site)
- apps/api/src/lib/audit.ts: details write (1 site)
- apps/api/src/plugins/auth.ts: apiKeys.permissions read (1 site)
* refactor(db): type jsonb columns via $type and note raw CTE conversion requirements
* feat(db): archive sqlite migrations and generate postgres baseline
* chore(db): dockerignore legacy migrations, add archive breadcrumb, fix trailing newline
* feat(db): pg pool connection, advisory-locked boot migrations, DATABASE_URL config
* fix(db): friendly fatal on unreachable postgres, idempotent closeDb, lock-key convention note
* refactor(db): async drizzle calls in plugins, lib, permissions
* fix(api): analytics never throws, typed permission guard, single-query session invalidation
* refactor(db): async drizzle calls across all routes and bootstrap
Convert every route file and index.ts from sync SQLite drizzle
patterns to async node-postgres drizzle:
- .all() removed (bare await on select)
- .get() converted to destructured [row] = await ...
- .run() removed (bare await on insert/update/delete)
- .changes replaced with .rowCount (null-guarded) in progress.ts
- sqlite import removed from user-files.ts; raw CTEs converted to
await db.execute(sql`...`) with postgres-dialect recursive CTEs
- ChainRow types updated: tool_chain is parsed jsonb (string[] | null),
created_at is Date (timestamptz) with no * 1000 conversion
- All requirePermission() guard calls awaited (security: unawaited
async guard returns truthy Promise, bypassing permission check)
- All hasEffectivePermission() and getPermissions() calls awaited
- All auditLog() calls awaited (preserves write-before-response order)
- trackEvent() and captureException() left un-awaited (fire-and-forget
by design, guaranteed never-throw)
- ensureAnonymousUser(), startCleanupCron(), recoverStaleJobs() awaited
in bootstrap sequence
- ensureInstanceId() and ensureDefaultSettings() made async
Files converted: 14 (index.ts + 12 route files + tools/index.ts)
* fix(db): await async checkStorageQuota in user-files upload/save routes
* fix(db): await checkStorageQuota in save-result route (missed second call site)
* feat(db): sqlite-to-postgres migrator with CLI and first-boot import
* fix(db): migrator error context, honest force semantics, boot-hook fatal, null-variance tests
* test: run suite against per-file postgres databases via testcontainers
- Add tests/global-setup.ts: spins up a Postgres testcontainer,
creates a migrated template database once per vitest run.
- Rewrite tests/setup/per-fork-env.ts: each test file (forks pool)
clones the template into its own database via CREATE DATABASE ...
TEMPLATE, preserving the same per-file isolation granularity.
- Update vitest.config.ts: add globalSetup, pg alias, update comment.
- Fix tests/integration/test-server.ts: remove DB_PATH mkdir, async
runMigrations, async db operations, remove SQLite WAL checkpoint.
- Fix 21 unit test db/index mocks: add pool and closeDb exports.
- Fix 8 unit test files: add async/await for now-async permission,
audit, and analytics functions.
- Fix 18 integration test files: convert sync .run()/.all()/.get()
to async drizzle patterns, add async to callbacks.
- Production change: apps/api/src/routes/teams.ts: cast COUNT(*)
to ::int so Postgres returns a number instead of bigint string.
* fix(db): seed built-in roles, reject NUL bytes, cast COUNT, serialize job persists
- Seed built-in roles (admin, editor, user) at boot via ensureBuiltinRoles()
with onConflictDoNothing, restoring data that legacy SQLite migration 0007
provided via INSERT statements (the pg baseline is DDL-only).
- Reject NUL bytes in login credentials with 401 (postgres rejects \x00 in
text columns; valid usernames never contain NUL, matching 1.x behavior).
- Cast COUNT(*)::int in user-files, audit-log, and roles listing queries so
postgres returns a JS number instead of bigint-as-string.
- Serialize fire-and-forget job progress DB writes per jobId so the final
"completed" status is never overwritten by a late-arriving "processing"
write (race condition exposed by async postgres round-trips).
* test: fix teams race, seed roles in test server, poll for job status
- Add missing await to resetTeams() in teams PUT beforeEach (the async
delete raced with the subsequent insert under postgres).
- Call ensureBuiltinRoles() in test server bootstrap so integration tests
have the same built-in roles as production.
- Replace fixed 100ms flushPersist delay with a polling helper that waits
for terminal job status, eliminating timing-dependent failures caused by
postgres network round-trip latency.
* test: make heic temp-file cleanup assertion resilient to concurrent workers
Use a set-based diff instead of raw file count when checking that
decodeHeic cleans up temp files. Other concurrent test workers can
create heic-in-*/heic-out-* files in the shared tmpdir, inflating the
"after" count and causing spurious failures under full-suite load.
* fix(db): align builtin-role seed to post-0010 legacy state; test polish
* feat(docker): three-container compose (app, postgres, redis) with boot wait and migrations
* fix(docker): set TEST_DATABASE_URL so containerized tests skip testcontainers
* chore(docker): test compose project name, clearer 1.x upgrade comment, unref probe timer
* feat(enterprise): enforce D15 license boundary; move s3 storage into packages/enterprise
* fix(enterprise): restore lazy aws-sdk loading; community installs load no s3 code at boot
* fix(enterprise): boundary check catches dynamic imports; document getS3 concurrency
* feat(db)!: SnapOtter 2.0 phase 1 foundation: postgres, migrator, compose stack
BREAKING CHANGE: SQLite is no longer the runtime database. Deployments now
require Postgres (and Redis, used from phase 2). Existing installs migrate
with SQLITE_MIGRATE_PATH or 'pnpm --filter @snapotter/api migrate:sqlite'.
* fix(ci): postgres service + fresh e2e database per run; ignore unfixable torch CVE-2025-3000
199 lines
6.9 KiB
YAML
199 lines
6.9 KiB
YAML
name: CI
|
|
|
|
on:
|
|
workflow_dispatch:
|
|
push:
|
|
branches: [main]
|
|
paths-ignore:
|
|
- "README.md"
|
|
- "CONTRIBUTING.md"
|
|
- "branding/**"
|
|
- "apps/docs/**"
|
|
- "apps/landing/**"
|
|
- "docs/**"
|
|
pull_request:
|
|
branches: [main]
|
|
paths-ignore:
|
|
- "README.md"
|
|
- "CONTRIBUTING.md"
|
|
- "branding/**"
|
|
- "apps/docs/**"
|
|
- "apps/landing/**"
|
|
- "docs/**"
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
lint:
|
|
name: Lint
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
- uses: ./.github/actions/setup
|
|
- run: pnpm lint
|
|
- name: License boundary check (D15)
|
|
run: pnpm check:license-boundary
|
|
|
|
typecheck:
|
|
name: Typecheck
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
- uses: ./.github/actions/setup
|
|
- run: pnpm typecheck
|
|
|
|
test-unit:
|
|
name: Unit Tests
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 5
|
|
steps:
|
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
|
|
- name: Install system dependencies (HEIC + ExifTool + ImageMagick + exotic format tools)
|
|
run: sudo apt-get update -qq && sudo apt-get install -y --no-install-recommends libheif-examples libheif-plugin-x265 libheif-plugin-libde265 libimage-exiftool-perl imagemagick ghostscript libjxl-tools libopenjp2-tools
|
|
|
|
- uses: ./.github/actions/setup
|
|
- run: pnpm vitest run tests/unit/ --reporter=verbose
|
|
|
|
test-integration:
|
|
name: Integration (${{ matrix.shard }}/4)
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 30
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
shard: [1, 2, 3, 4]
|
|
steps:
|
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
|
|
- name: Install system dependencies (HEIC + ExifTool + ImageMagick + exotic format tools)
|
|
run: sudo apt-get update -qq && sudo apt-get install -y --no-install-recommends libheif-examples libheif-plugin-x265 libheif-plugin-libde265 libimage-exiftool-perl imagemagick ghostscript libjxl-tools libopenjp2-tools
|
|
|
|
- name: Allow ImageMagick to read EPS/PS via Ghostscript delegate
|
|
run: |
|
|
POLICY_FILE=$(find /etc/ImageMagick* -name policy.xml 2>/dev/null | head -1)
|
|
if [ -n "$POLICY_FILE" ]; then
|
|
sudo sed -i 's/<policy domain="coder" rights="none" pattern="EPS"/<policy domain="coder" rights="read" pattern="EPS"/' "$POLICY_FILE"
|
|
sudo sed -i 's/<policy domain="coder" rights="none" pattern="PS"/<policy domain="coder" rights="read" pattern="PS"/' "$POLICY_FILE"
|
|
fi
|
|
|
|
- uses: ./.github/actions/setup
|
|
- run: pnpm vitest run tests/integration/ --reporter=verbose --shard=${{ matrix.shard }}/4
|
|
|
|
test-e2e-smoke:
|
|
name: E2E Smoke (Chromium)
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 15
|
|
services:
|
|
postgres:
|
|
image: postgres:17-alpine
|
|
env:
|
|
POSTGRES_USER: snapotter
|
|
POSTGRES_PASSWORD: snapotter
|
|
POSTGRES_DB: snapotter
|
|
ports:
|
|
- 5432:5432
|
|
options: >-
|
|
--health-cmd "pg_isready -U snapotter"
|
|
--health-interval 5s
|
|
--health-timeout 3s
|
|
--health-retries 10
|
|
steps:
|
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
- uses: ./.github/actions/setup
|
|
- name: Get Playwright version
|
|
id: pw-version
|
|
run: echo "version=$(node -p "require('@playwright/test/package.json').version")" >> "$GITHUB_OUTPUT"
|
|
- name: Cache Playwright browsers
|
|
uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
|
|
with:
|
|
path: ~/.cache/ms-playwright
|
|
key: playwright-${{ runner.os }}-${{ steps.pw-version.outputs.version }}
|
|
- name: Install Playwright Chromium
|
|
run: pnpm playwright install --with-deps chromium
|
|
- name: Run smoke specs
|
|
run: pnpm playwright test tests/e2e/smoke.spec.ts tests/e2e/tools-all.spec.ts tests/e2e/navigation.spec.ts tests/e2e/home-page.spec.ts --project=chromium
|
|
env:
|
|
PW_WORKERS: "1"
|
|
- name: Upload report on failure
|
|
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
|
|
if: failure()
|
|
with:
|
|
name: e2e-smoke-report
|
|
path: playwright-report/
|
|
retention-days: 7
|
|
|
|
pip-audit:
|
|
name: Python Dependency Audit
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
|
|
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
|
|
with:
|
|
python-version: "3.11"
|
|
|
|
- run: pip install "pip-audit==2.10.0"
|
|
- name: Run pip-audit (ignoring CVEs blocked by dependency constraints)
|
|
# CVE-2025-3000: torch 2.12.0, no fixed release available as of 2026-06-11
|
|
run: >-
|
|
pip-audit -r packages/ai/python/requirements.txt
|
|
--ignore-vuln CVE-2024-27763
|
|
--ignore-vuln CVE-2025-3000
|
|
--ignore-vuln CVE-2026-40086
|
|
--ignore-vuln CVE-2026-25990
|
|
--ignore-vuln CVE-2026-40192
|
|
--ignore-vuln GHSA-55v6-g8pm-pw4c
|
|
--ignore-vuln CVE-2026-42308
|
|
--ignore-vuln CVE-2026-42310
|
|
--ignore-vuln CVE-2026-42311
|
|
--ignore-vuln PYSEC-2025-189
|
|
--ignore-vuln PYSEC-2025-190
|
|
--ignore-vuln PYSEC-2025-191
|
|
--ignore-vuln PYSEC-2025-192
|
|
--ignore-vuln PYSEC-2025-193
|
|
--ignore-vuln PYSEC-2025-194
|
|
--ignore-vuln PYSEC-2025-195
|
|
--ignore-vuln PYSEC-2025-196
|
|
--ignore-vuln PYSEC-2025-197
|
|
--ignore-vuln PYSEC-2025-210
|
|
--ignore-vuln PYSEC-2026-139
|
|
--ignore-vuln PYSEC-2024-277
|
|
--ignore-vuln PYSEC-2026-89
|
|
|
|
build:
|
|
name: Build
|
|
runs-on: ubuntu-latest
|
|
needs: [lint, typecheck, test-unit, test-integration, test-e2e-smoke]
|
|
steps:
|
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
- uses: ./.github/actions/setup
|
|
- run: pnpm build
|
|
|
|
# docker:
|
|
# name: Docker Build Test
|
|
# runs-on: ubuntu-latest
|
|
# steps:
|
|
# - uses: actions/checkout@v4
|
|
# - uses: docker/setup-buildx-action@v3
|
|
# - name: Log in to GHCR (for registry cache)
|
|
# uses: docker/login-action@v3
|
|
# with:
|
|
# registry: ghcr.io
|
|
# username: ${{ github.repository_owner }}
|
|
# password: ${{ secrets.GHCR_TOKEN }}
|
|
# - uses: docker/build-push-action@v6
|
|
# with:
|
|
# context: .
|
|
# file: docker/Dockerfile
|
|
# push: false
|
|
# tags: snapotter:ci
|
|
# build-args: SKIP_MODEL_DOWNLOADS=true
|
|
# cache-from: type=registry,ref=ghcr.io/snapotter-hq/snapotter:cache-linux-amd64
|
|
# cache-to: type=registry,ref=ghcr.io/snapotter-hq/snapotter:cache-ci,mode=max
|