Files
pmg/docs/persistent-proxy.md
T
c47776db27 feat(proxy): persistent proxy server mode (#351)
* refactor(flows): extract SetupCACertificate for reuse

Move the CA load/generate/merge logic out of proxyFlow into an exported
flows.SetupCACertificate so the persistent proxy server can reuse it.

* feat(proxy): add persistent proxy server with start/stop/env/status

Introduces 'pmg proxy' commands backed by internal/proxyserver: a long-lived
MITM proxy that intercepts package managers via env vars (no shims). Supports
--daemon (Unix), --state, --port; generic 'env' output that skips cert vars
when the CA is OS-trusted; opt-in 'stop --fail-on-violation' (fail-closed on
crash) with a synchronous cloud event flush; and the malysis analysis cache.

* feat(action): add server-mode for persistent proxy

When server-mode=true the action starts the proxy daemon and injects proxy
env vars into the job instead of installing shims.

* test(proxy): add persistent proxy server E2E workflow

* docs(readme): document persistent proxy server mode

* fix(proxy): create cache dir before writing state file and daemon log

On a fresh CI runner the cache directory does not exist yet; os.OpenFile and
os.WriteFile do not create parent dirs, so 'pmg proxy start --daemon' failed
with 'no such file or directory'. MkdirAll the parent before writing.

* docs: add persistent proxy server architecture doc

* refactor proxyserver

* fix(proxy): always emit cert env vars instead of skipping on OS-trust status

npm/pip/yarn/requests trust the MITM CA inconsistently across tools, versions,
and configs; many still use bundled CA stores. Always emitting the cert-path
env vars is the conservative choice that works regardless, and is harmless for
tools that read the OS store (they ignore the vars). Skipping them when a
system CA exists would silently break any tool still on a bundled store.

* refactor(proxy): drop redundant audit init in daemon; rely on main.go

main.go's PersistentPreRun already initializes the audit pipeline for every
command (including the daemon's re-exec'd child) and closes it at process exit.
Re-initializing in proxyserver.Run created a second auditor and a second
cloud-sync WAL connection, orphaning the first. Removing it makes the daemon
consistent with the normal proxy flow, which never self-initializes audit.

* fix(proxy): bypass proxy env when flushing events to cloud on stop

pmg proxy stop inherits HTTP(S)_PROXY (injected by 'pmg proxy env') pointing at
the PMG proxy it just shut down. The cloud sync gRPC client honored those vars
and routed api.safedep.io through the dead proxy, failing with 'connection
refused' so no events were delivered. Clear the proxy env vars before the sync
so PMG's own cloud traffic goes direct.

* chore(proxy): address review feedback

- configurable bind host via proxy.server.listen_host (default loopback)
- proxy commands use ui.ErrorExit instead of returning errors to cobra
- rename errcode to ProxyPolicyViolation (covers malware + cooldown)
- share cloud sync via audit.DrainToCloud (de-dup with cmd/cloud/sync)
- centralize proxy CA bundle path in certmanager
- docs: persistent proxy cert trust + bind address

* fix(proxy): show real message on fail-on-violation error

stopExitError set only WithMsg, but ui.ErrorExit renders HumanError, so the
framed error showed 'no human-readable message available'. Set both from one
string, and emit the framed error before the stdout summary so the blocked
count is stated once.

* fix(proxy): flush cloud events from the daemon, not stop

The stop process inherits HTTP_PROXY (from 'pmg proxy env'), so its cloud
client routed api.safedep.io through the already-stopped proxy and failed with
connection refused. Move the flush into the daemon's shutdown, which has no
proxy env (it started before env injection) and dials SafeDep directly.

- daemon flushes on shutdown via audit.DrainToCloud and records the result in
  the state file; stop surfaces it (on both success and fail-on-violation
  paths) since the daemon's own logs aren't visible to stop
- coordinate stop's wait with the daemon shutdown budget; on timeout, error
  out without reading stale state or deleting the file (fail-closed)
- persist blocked count before the flush so the gate stays correct if the
  flush hangs or the daemon is killed mid-flush
- remove now-redundant cloud_flush.go

* disable auto-sync for proxy cmds

* feat(proxy): periodic cloud sync + move proxy env vars to packagemanager

- daemon runs a periodic cloud-sync ticker so the shutdown flush stays small;
  the run total is reported by stop, and shutdown timeouts are coordinated
- move EnvVarForProxy from config to packagemanager (it is package-manager
  knowledge); the shared function now builds the proxy URL and NO_PROXY itself,
  removing the duplicated construction in the per-command and persistent paths
- relocate the #319 yarn and #339 IPv6 regression tests alongside the function
- enable cloud sync in the persistent-proxy E2E workflow and fix the stale
  internal/proxystate path filter

* refactor(proxy): rename cloudFlushLockTimeout to cloudFlushLockWait

Consistent timeout naming: *LockWait is the lock-acquire bound, *Timeout is
the sync-RPC bound. Previously the final-flush pair was cloudFlushLockTimeout
vs cloudFlushTimeout — two lookalike names for different operations.

* refactor(proxy): extract cloudFlush and trim duplicate shutdown comments

The shutdown's final-flush block is now a cloudFlush helper, symmetric with
startCloudSyncLoop (one-shot vs loop). Removed the triplicated ticker/lock
contention comments, keeping the contract on the function doc and one-line
pointers at the call sites.

* docs: update persistent proxy cloud sync to daemon-owned model

The daemon now owns cloud delivery (periodic sync while serving + final flush
on shutdown); stop signals it, waits, and reports the result. Rewrite the Cloud
event sync section, fix stop attributions, add the cloud_sync state field, and
update the sequence diagram.

* docs: move Usage section up below How it works

Put the copy-paste recipes near the top so users find them before the internals.

* refactor(proxy): address PR review feedback

- configurable bind host/port via --host/--port flags + config (listen_host,
  listen_port), bound directly to config fields per PMG's flag pattern
- daemon log path via --log-file and readiness timeout in ProxyDaemonConfig;
  Daemonize no longer owns path policy (caller validates, fails fast)
- gate periodic cloud sync on auto_sync; suppress detached background sync for
  proxy commands instead of flipping the flag
- pmg proxy env --export emits shell-quoted lines for eval (spaces survive)
- extract shared flows.BuildCachedMalysisAnalyzer, dropping the analyzer+cache
  duplication between proxy flow and proxy server
- add internal/proxyserver/doc.go documenting the package + boundary vs flows
- E2E: assert malicious installs are blocked (drop continue-on-error)
- docs: trim Commands/State-file to user contracts; refresh bind address

* refactor(proxy): proactive alignment fixes from whole-PR review

- gate the shutdown cloud flush on auto_sync too, matching the periodic ticker
  (auto_sync consistently controls all daemon-driven cloud delivery)
- ResolveStatePath takes cacheDir instead of *RuntimeConfig, keeping state.go
  free of config dependency
- drop the empty-host comment in listenAddr; keep the loopback guard so a blank
  host never silently binds all interfaces

* fix: Decouple localdb with malysis analyser construction

* fix: Persist global args before proxy server daemon exec

* fix: GitHub Action for cloud auto-sync in server mode

---------

Co-authored-by: Abhisek Datta <abhisek.datta@gmail.com>
2026-06-26 11:19:28 +05:30

8.0 KiB

Persistent Proxy Server

The persistent proxy server runs PMG's MITM proxy as a long-lived process that intercepts every supported package manager invocation in an environment via standard proxy environment variables, without shims, aliases, or wrapping each command with pmg. It is built for non-interactive environments, primarily CI/CD pipelines (e.g. GitHub Actions), where the environment can be configured once for the whole job.

It builds on the generic MITM proxy described in proxy.md, reusing the same interceptor chain, malware analyzer, and certificate manager. The difference is the lifecycle: instead of PMG starting an ephemeral proxy around a single subprocess, the proxy is started once, advertises itself to the other pmg proxy commands, and serves many package manager processes until it is stopped.

Default proxy mode vs. persistent proxy server

PMG's default proxy mode (see proxy.md) wraps a single command. pmg npm install starts an ephemeral proxy, runs npm as a child with proxy env vars injected, then tears the proxy down. The persistent server decouples these steps.

Default proxy mode Persistent proxy server
Invocation pmg npm install (wrapped) bare npm install (no wrapper)
Proxy lifetime One subprocess Until pmg proxy stop
Who runs the PM PMG (as a child) The user / CI directly
Ecosystems served The one being run All supported (npm + PyPI)
Confirmation on malware Interactive prompt (TTY) Auto-block (non-interactive)
Reporting At subprocess exit At pmg proxy stop
Target Local dev CI/CD pipelines

How it works

The diagram below shows the order of events in a CI job. The pmg proxy commands run in separate workflow steps and coordinate through the running daemon.

sequenceDiagram
    participant CI as CI Job
    participant Proxy as Proxy Daemon
    participant PM as Package Manager
    participant Cloud as SafeDep Cloud

    CI->>Proxy: pmg proxy start --daemon
    Proxy-->>CI: ready (addr, ca path)
    CI->>CI: pmg proxy env  (set HTTP_PROXY + CA vars)
    PM->>Proxy: package download (via HTTP_PROXY)
    Proxy->>Proxy: analyze package
    Proxy-->>PM: allow, or 403 block + record event
    Proxy->>Cloud: periodic sync of events (while serving)
    CI->>Proxy: pmg proxy stop --fail-on-violation
    Proxy->>Cloud: final flush of remaining events
    Proxy-->>CI: exit non-zero if anything was blocked

Usage

The persistent server targets non-interactive CI/CD. For local development use the default proxy mode (pmg npm install), which keeps the interactive malware confirmation prompt. The persistent server auto-blocks without prompting.

GitHub Actions (raw commands):

- run: pmg proxy start --daemon
- run: pmg proxy env >> "$GITHUB_ENV"
- run: npm ci
- run: pmg proxy stop --fail-on-violation
  if: always()

GitHub Actions (via the safedep/pmg action server-mode):

- uses: safedep/pmg@v1
  with:
    server-mode: true
    api-key: ${{ secrets.SAFEDEP_API_KEY }}
    tenant-id: ${{ secrets.SAFEDEP_TENANT_ID }}

- run: npm ci          # intercepted automatically

- name: Enforce PMG policy
  if: always()
  run: pmg proxy stop --fail-on-violation

In server-mode, the action starts the daemon and injects env vars instead of installing shims. Because composite actions cannot run an automatic cleanup step, the final pmg proxy stop --fail-on-violation step is required. It stops the proxy (the daemon flushes events to the cloud during shutdown) and fails the job on a block.

Commands

pmg proxy start    # start the proxy (foreground, or detached with --daemon)
pmg proxy stop     # stop the proxy and report the outcome
pmg proxy env      # print env vars that route package managers through it
pmg proxy status   # report whether a proxy is running

Run pmg proxy <command> --help for flags. --daemon is Unix only: on Windows it returns a clear "not supported" error, and the foreground pmg proxy start still works. To run multiple independent proxies on one host, give each a distinct --state path and --port.

Bind address

The proxy binds 127.0.0.1 on a random port by default, reachable only from the host (the right choice for CI and local use). Override with --host/--port, or the proxy.server.listen_host/listen_port config (flags take precedence).

Bind a non-loopback address (e.g. --host 0.0.0.0) only for a deliberately hosted deployment: it exposes the MITM proxy to the network, and every client routed through it has its HTTPS intercepted and must trust the PMG CA.

Certificate trust

The proxy performs TLS MITM, so clients must trust its CA. Trust is delivered through environment variables, not the OS trust store. pmg proxy env always emits the cert-path variables pointing at the proxy's CA bundle: NODE_EXTRA_CA_CERTS, SSL_CERT_FILE, REQUESTS_CA_BUNDLE, PIP_CERT, YARN_HTTPS_CA_FILE_PATH. Package managers pick these up from the job environment and trust the proxy's CA, with no OS trust-store install required.

This is deliberate: whether a tool consults the OS trust store varies by tool, version, and config (npm/Node ignore it by default; modern pip can read it; requests/certifi ship their own bundle). The cert-path vars work across all of them, and are harmlessly ignored by tools that do read the OS store.

As a result pmg setup cert install is not needed for the persistent proxy. If a persisted CA from pmg setup cert install exists the proxy reuses it, otherwise it generates an ephemeral one. Either way pmg proxy env carries the trust. OS trust-store install (pmg setup cert install --system) is intentionally not used: it needs root (breaking container and locked-down runners), persistently installs a MITM-capable CA into the machine trust store, and still does not remove the need for the env vars.

Loopback addresses are always excluded from proxying via NO_PROXY (localhost,127.0.0.1,::1).

Cloud event sync

When SafeDep Cloud is enabled, malware-block events must reach the cloud even on ephemeral CI runners that are destroyed immediately after the job. The daemon owns delivery: it records each blocked package to a durable local event log as it happens, syncs pending events to SafeDep Cloud periodically while serving, and flushes whatever remains on shutdown.

pmg proxy stop reports the recorded result (Synced N event(s) to SafeDep Cloud, or a Cloud sync failed line). A flush failure is surfaced but does not mask the fail-on-violation exit code.

Fail on violation

By default pmg proxy stop just stops the proxy and exits 0. Failing the CI job on a policy violation is opt-in via --fail-on-violation.

  • It exits non-zero when any package was blocked.
  • It fails closed. If the daemon shut down without writing a verifiable final state (e.g. it crashed), --fail-on-violation also fails, because a security gate must not pass on an unverifiable run.

The package manager's own non-zero exit (from the 403 on a blocked download) is a separate signal. --fail-on-violation gives an authoritative gate from the proxy regardless of how the package manager reported the failure.

Limitations

  • Unix-only daemon. --daemon is not supported on Windows (foreground mode works).
  • Non-interactive only. There is no interactive confirmation; flagged packages are always auto-blocked. This is intentional for CI.
  • Single proxy per state file. Starting a second proxy that points at the same state file is refused while one is running.
  • System-level trust enforcement is out of scope. The server relies on env var propagation. Enforcing interception for sudo-scrubbed environments (e.g. via iptables) and system-wide install (pmg setup install --system) are tracked separately.

References

  • proxy.md is the underlying generic MITM proxy server
  • config.md is the configuration schema (cloud, proxy, cache dir)
  • action.yml is the PMG GitHub Action (server-mode)