# E2E Test Refactoring Spec

Status: Proposed
Owner: PMG maintainers
Target branch: `claude/e2e-tests-github-actions-hkxij2`

## Goal

Make PMG's end-to-end (E2E) tests **easy to write, maintain, and extend**, and
let us run them **across Linux, macOS, and Windows** without copy-pasting YAML.
Reliability of PMG is the driver: we expect to add many more E2E cases, so the
cost of adding one must be low and the structure must scale to a 3-OS matrix.

Chosen direction: **Go test harness (testify) for the package-manager matrix +
a shared composite setup action + a reusable sandbox workflow** (Option D from
the analysis). The package-manager matrix must run on **all three OS** in the
near term.

## Current state (what we are replacing)

E2E logic is spread across five places with heavy duplication:

| File | Jobs / content |
|---|---|
| `.github/workflows/pmg-e2e.yml` (808 lines) | One giant `e2e-test` job (~18 inline bash steps) + 3 near-identical sandbox jobs (macOS Seatbelt, Linux Bubblewrap, Linux Landlock) |
| `.github/workflows/action-e2e.yml` | 4 jobs testing the composite `action.yml` |
| `.github/workflows/ci.yml` | `e2e-test` job running `test/e2e.sh` (trivial dry-run smoke) |
| `test/sandbox-e2e.js`, `test/pm-e2e.js` | Node assertions invoked from sandbox jobs |
| `test/e2e.sh` | 3-line dry-run smoke test |
| `sandbox/platform/landlock_e2e_linux_test.go` | Go-based Landlock helper E2E (the pattern we extend) |

### Concrete duplication to eliminate

1. **Environment setup copy-pasted 7+ times.** The
   `checkout → setup-go → corepack → setup-node → pnpm → build → add-to-PATH →
   pmg setup install` block repeats in all 4 jobs of `pmg-e2e.yml`, in
   `action-e2e.yml`, and in `ci.yml`. GitHub Actions has no YAML anchors, so it
   is literal duplication.
2. **The 3 sandbox jobs are ~90% identical.** Same "Create Test
   Directories/Files", same "Disable AppArmor", same 12-line canary-secret
   `env:` block (also duplicated in the `Makefile`), same two run steps. Deltas:
   `runs-on`, driver env var, two extra files on Linux, one Landlock-only step.
3. **Every PM install step has the same shape**
   (`mkdir → init → install A → install B → assert node_modules + manifest →
   rm lockfile → reinstall → re-assert`), hand-written ~10 times for npm, pnpm,
   bun, yarn (proxy/non-proxy) and the pip/pip3/uv/poetry analog.
4. **Hardcoded fixtures repeated everywhere:** `express@5.2.1`/`lodash@4.17.21`,
   `requests==2.32.4`/`numpy==2.3.5`, and malicious markers
   `safedep-test-pkg@0.1.3` / `nyc-config@10.0.0`.
5. **Weak/ad-hoc assertions.** `pmg-e2e.yml:521`
   (`! pmg ... npm install nyc-config@10.0.0 || echo "blocked"`) passes
   regardless of outcome — a real reliability gap. Temp-dir handling is
   inconsistent (`mkdir x && cd x` vs `mktemp -d`).
6. **Multi-OS goal structurally unmet.** The PM matrix is `os: [ubuntu-latest]`
   only; adding an OS today means duplicating a ~500-line job.
7. **No granularity/parallelism.** ~18 PM tests run serially in one 20-minute
   job; one failure masks the rest.

## Target architecture

```
.github/
  actions/
    setup-pmg/action.yml         # composite: toolchains + build pmg + PATH + pmg setup install
  workflows/
    pmg-e2e.yml                  # slim: PM matrix (Go harness) over OS + calls sandbox workflow
    e2e-sandbox.yml              # reusable (workflow_call): sandbox over {os, driver}
    action-e2e.yml               # uses setup-pmg where applicable
    ci.yml                       # e2e-test job replaced by harness smoke (or removed)
test/
  e2e/                           # NEW Go harness (testify, mirrors landlock_e2e pattern)
    main_test.go                 # TestMain: build pmg once, locate repo root
    harness.go                   # RunPmg, project dirs, assert helpers, env seeding
    fixtures.go                  # benign/malicious coordinates, canary env, PM descriptors
    npm_test.go pnpm_test.go bun_test.go yarn_test.go
    pip_test.go uv_test.go poetry_test.go npx_test.go
    modes_test.go                # --dry-run, --silent, --verbose, --debug/--log, --paranoid
    regression_test.go           # httpx NO_PROXY #339, yarn-berry corepack pinning
  sandbox-e2e.js  pm-e2e.js      # kept, still invoked by the sandbox workflow
```

### The Go harness (`test/e2e/`)

Reuse the proven shape from `sandbox/platform/landlock_e2e_linux_test.go`
(walk up to repo root, build/locate `bin/pmg`).

- **Build once:** `TestMain` builds `bin/pmg` a single time per `go test`
  invocation (guarded so a prebuilt binary from `make` is reused in CI).
- **PM descriptor table** drives the repeated pattern instead of bespoke bash:

  ```go
  type packageManager struct {
      name        string   // "npm", "pnpm", "bun", "yarn", ...
      bin         string   // executable to probe on PATH
      initArgs    []string // e.g. {"init", "-y"}
      addArgs     func(pkg string) []string
      installArgs []string // manifest install
      manifest    string   // "package.json" / "pyproject.toml"
      lockfile    string   // "package-lock.json" / "pnpm-lock.yaml" / ...
      proxyModes  []bool   // which proxy modes to exercise
  }
  ```

- **Helpers** (testify `require`/`assert`, per CLAUDE.md):
  - `RunPmg(t, opts, args...) result` — runs `pmg`, captures stdout/stderr/exit.
  - `NewProject(t)` — `t.TempDir()`-based working dir with cleanup for free.
  - `RequireInstalled(t, dir, pkg)`, `RequireManifestContains(t, manifest, pkg)`,
    `RequireLockfile(t, dir, name)`.
  - `RequireBlocked(t, result, marker)` — asserts **non-zero exit AND** the
    "Malicious package blocked" signal AND absence in `node_modules`. This
    replaces every weak `|| echo` block (fixes #5) in one place.
  - `RequireTool(t, name)` — `t.Skip` when a PM is not installed on the runner.
  - `SeedCanaryEnv()` — single source of truth for the scrub/keep canaries
    shared with the sandbox JS scripts and `Makefile`.
- **Fixtures centralized** in `fixtures.go`: benign packages, malicious markers,
  and Python/Node version pins. Changing `express@5.2.1` happens in one place.
- **Granularity & parallelism:** each PM is a table-driven test with subtests
  per `{proxyMode}`; independent cases use `t.Parallel()`.
- **Skips over failures:** tools absent on a given OS (e.g. poetry, bun) are
  skipped via `RequireTool`, not failed — so the same suite is portable.

### Composite setup action (`.github/actions/setup-pmg`)

Encapsulates the repeated environment block with inputs for what each job
needs (`go`, `node`, `pnpm`, `bun`, `python`, `uv`, `poetry`, `build`,
`pmg-setup`). All workflows consume it, killing duplication #1. Pin action SHAs
exactly as today.

### Reusable sandbox workflow (`.github/workflows/e2e-sandbox.yml`)

`on: workflow_call` with inputs `os` and `driver`. Defines the canary `env:`
once, the test-dir/file seeding once, and the AppArmor relaxation + Landlock
probe behind `if:` guards. `pmg-e2e.yml` calls it three times via matrix
(`{macos, seatbelt}`, `{ubuntu, bubblewrap}`, `{ubuntu, landlock}`), collapsing
3 jobs into one definition (fixes #2).

### Slimmed `pmg-e2e.yml`

```yaml
jobs:
  pm-matrix:
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: ./.github/actions/setup-pmg
        with: { node: true, pnpm: true, bun: true, python: true, uv: true, poetry: true }
      - run: go test ./test/e2e/... -count=1 -v
  sandbox:
    strategy:
      fail-fast: false
      matrix:
        include:
          - { os: macos-latest,  driver: seatbelt }
          - { os: ubuntu-latest, driver: bubblewrap }
          - { os: ubuntu-latest, driver: landlock }
    uses: ./.github/workflows/e2e-sandbox.yml
    with: { os: '${{ matrix.os }}', driver: '${{ matrix.driver }}' }
```

Adding a PM = a new descriptor + `_test.go`. Adding an OS = a matrix entry.

## Multi-OS plan (all three soon)

- **Go is the portability lever:** the harness compiles and runs on Linux,
  macOS, and Windows, so the PM matrix fans out via `runs-on` alone.
- **Per-OS tool availability** handled by `RequireTool` skips and by toggling
  setup-action inputs (e.g. don't request poetry where we don't test it).
- **Sandbox is inherently OS-specific** (Seatbelt=macOS, Bubblewrap/Landlock=
  Linux, none on Windows) and stays in the reusable sandbox workflow, not the
  cross-OS PM matrix.
- **Windows risk to resolve in Phase 3 (see Open Questions):** today
  `ci.yml`'s Windows job only runs `./internal/flows/...`, implying limited
  Windows coverage. We must confirm `pmg setup install`, proxy mode, and PATH
  shims behave on Windows before enabling the Windows leg; gate with
  `runtime.GOOS` skips where features are unsupported rather than failing.

## Migration phases

1. **Setup action.** Add `.github/actions/setup-pmg`; switch every existing job
   to it. No behavior change — pure dedupe. Verify CI stays green.
2. **Reusable sandbox workflow.** Extract `e2e-sandbox.yml`; collapse the 3
   sandbox jobs into a matrix. Centralize canaries (workflow + `Makefile` +
   harness reference the same list).
3. **Harness bootstrap.** Stand up `test/e2e/` with the harness, fixtures, and
   **npm + pnpm** ported as proof. Run Linux only. Validate parity with the old
   bash (same packages, same assertions) and fix the `nyc-config` weak check.
4. **Port the rest.** bun, yarn (classic + berry), npx/pnpx, pip, pip3, uv,
   poetry, modes, and the two regression cases (#339, yarn-berry corepack).
   Delete the corresponding inline bash from `pmg-e2e.yml`.
5. **Enable macOS + Windows** legs with `RequireTool`/`runtime.GOOS` skips;
   resolve the Windows support questions; tune the OS-specific toolchain inputs.
6. **Consolidate leftovers.** Fold `action-e2e.yml` block tests and
   `ci.yml`/`test/e2e.sh` smoke into the harness where it reduces surface;
   remove dead scripts.

Each phase is independently shippable and leaves CI green.

## Cross-cutting cleanups (apply during the relevant phase)

- Single source of truth for fixtures and canary env.
- Standardized assertions and `t.TempDir()` lifecycle (no manual `cd`/cleanup).
- `fail-fast: false` + per-PM subtests for at-a-glance failure isolation.
- Fix `pmg-e2e.yml:521` weak malicious-block assertion via `RequireBlocked`.

## Non-goals

- Rewriting the sandbox policy `.js` assertion scripts (kept; only their
  invocation is deduped).
- Changing what PMG itself does; this is test-infrastructure only.
- Eliminating external package-manager processes (PMs are still real
  subprocesses, wrapped by the harness).

## Risks & trade-offs

- **Upfront port cost** is the largest of the options; mitigated by phasing
  (npm/pnpm proof first) and by reusing the existing Landlock E2E pattern.
- **Runner cost:** macOS/Windows minutes are pricier than Linux. Keep
  `fail-fast: false` but consider running the full 3-OS matrix on `main`/nightly
  and a Linux-only fast leg on PRs if minutes become a concern.
- **Windows feature gaps** (above) may force `runtime.GOOS` skips initially;
  acceptable as long as skips are explicit and visible.

## Open questions

1. Does `pmg setup install` + proxy mode + PATH shims work on Windows today? If
   not, which PM cases can run there in Phase 5, and which are skipped?
2. Should the full 3-OS matrix run on every PR, or PR=Linux + nightly/`main`=all
   three to control runner minutes?
3. Keep the sandbox assertions in Node (`test/*.js`) long-term, or eventually
   port them into the Go harness for one language? (Spec assumes: keep Node.)