External/pmg
1
0
Fork 0
mirror of https://github.com/safedep/pmg.git synced 2026-08-03 07:24:09 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat(proxy): persistent proxy server mode (#351)

Browse Source 
* refactor(flows): extract SetupCACertificate for reuse

Move the CA load/generate/merge logic out of proxyFlow into an exported
flows.SetupCACertificate so the persistent proxy server can reuse it.

* feat(proxy): add persistent proxy server with start/stop/env/status

Introduces 'pmg proxy' commands backed by internal/proxyserver: a long-lived
MITM proxy that intercepts package managers via env vars (no shims). Supports
--daemon (Unix), --state, --port; generic 'env' output that skips cert vars
when the CA is OS-trusted; opt-in 'stop --fail-on-violation' (fail-closed on
crash) with a synchronous cloud event flush; and the malysis analysis cache.

* feat(action): add server-mode for persistent proxy

When server-mode=true the action starts the proxy daemon and injects proxy
env vars into the job instead of installing shims.

* test(proxy): add persistent proxy server E2E workflow

* docs(readme): document persistent proxy server mode

* fix(proxy): create cache dir before writing state file and daemon log

On a fresh CI runner the cache directory does not exist yet; os.OpenFile and
os.WriteFile do not create parent dirs, so 'pmg proxy start --daemon' failed
with 'no such file or directory'. MkdirAll the parent before writing.

* docs: add persistent proxy server architecture doc

* refactor proxyserver

* fix(proxy): always emit cert env vars instead of skipping on OS-trust status

npm/pip/yarn/requests trust the MITM CA inconsistently across tools, versions,
and configs; many still use bundled CA stores. Always emitting the cert-path
env vars is the conservative choice that works regardless, and is harmless for
tools that read the OS store (they ignore the vars). Skipping them when a
system CA exists would silently break any tool still on a bundled store.

* refactor(proxy): drop redundant audit init in daemon; rely on main.go

main.go's PersistentPreRun already initializes the audit pipeline for every
command (including the daemon's re-exec'd child) and closes it at process exit.
Re-initializing in proxyserver.Run created a second auditor and a second
cloud-sync WAL connection, orphaning the first. Removing it makes the daemon
consistent with the normal proxy flow, which never self-initializes audit.

* fix(proxy): bypass proxy env when flushing events to cloud on stop

pmg proxy stop inherits HTTP(S)_PROXY (injected by 'pmg proxy env') pointing at
the PMG proxy it just shut down. The cloud sync gRPC client honored those vars
and routed api.safedep.io through the dead proxy, failing with 'connection
refused' so no events were delivered. Clear the proxy env vars before the sync
so PMG's own cloud traffic goes direct.

* chore(proxy): address review feedback

- configurable bind host via proxy.server.listen_host (default loopback)
- proxy commands use ui.ErrorExit instead of returning errors to cobra
- rename errcode to ProxyPolicyViolation (covers malware + cooldown)
- share cloud sync via audit.DrainToCloud (de-dup with cmd/cloud/sync)
- centralize proxy CA bundle path in certmanager
- docs: persistent proxy cert trust + bind address

* fix(proxy): show real message on fail-on-violation error

stopExitError set only WithMsg, but ui.ErrorExit renders HumanError, so the
framed error showed 'no human-readable message available'. Set both from one
string, and emit the framed error before the stdout summary so the blocked
count is stated once.

* fix(proxy): flush cloud events from the daemon, not stop

The stop process inherits HTTP_PROXY (from 'pmg proxy env'), so its cloud
client routed api.safedep.io through the already-stopped proxy and failed with
connection refused. Move the flush into the daemon's shutdown, which has no
proxy env (it started before env injection) and dials SafeDep directly.

- daemon flushes on shutdown via audit.DrainToCloud and records the result in
  the state file; stop surfaces it (on both success and fail-on-violation
  paths) since the daemon's own logs aren't visible to stop
- coordinate stop's wait with the daemon shutdown budget; on timeout, error
  out without reading stale state or deleting the file (fail-closed)
- persist blocked count before the flush so the gate stays correct if the
  flush hangs or the daemon is killed mid-flush
- remove now-redundant cloud_flush.go

* disable auto-sync for proxy cmds

* feat(proxy): periodic cloud sync + move proxy env vars to packagemanager

- daemon runs a periodic cloud-sync ticker so the shutdown flush stays small;
  the run total is reported by stop, and shutdown timeouts are coordinated
- move EnvVarForProxy from config to packagemanager (it is package-manager
  knowledge); the shared function now builds the proxy URL and NO_PROXY itself,
  removing the duplicated construction in the per-command and persistent paths
- relocate the #319 yarn and #339 IPv6 regression tests alongside the function
- enable cloud sync in the persistent-proxy E2E workflow and fix the stale
  internal/proxystate path filter

* refactor(proxy): rename cloudFlushLockTimeout to cloudFlushLockWait

Consistent timeout naming: *LockWait is the lock-acquire bound, *Timeout is
the sync-RPC bound. Previously the final-flush pair was cloudFlushLockTimeout
vs cloudFlushTimeout — two lookalike names for different operations.

* refactor(proxy): extract cloudFlush and trim duplicate shutdown comments

The shutdown's final-flush block is now a cloudFlush helper, symmetric with
startCloudSyncLoop (one-shot vs loop). Removed the triplicated ticker/lock
contention comments, keeping the contract on the function doc and one-line
pointers at the call sites.

* docs: update persistent proxy cloud sync to daemon-owned model

The daemon now owns cloud delivery (periodic sync while serving + final flush
on shutdown); stop signals it, waits, and reports the result. Rewrite the Cloud
event sync section, fix stop attributions, add the cloud_sync state field, and
update the sequence diagram.

* docs: move Usage section up below How it works

Put the copy-paste recipes near the top so users find them before the internals.

* refactor(proxy): address PR review feedback

- configurable bind host/port via --host/--port flags + config (listen_host,
  listen_port), bound directly to config fields per PMG's flag pattern
- daemon log path via --log-file and readiness timeout in ProxyDaemonConfig;
  Daemonize no longer owns path policy (caller validates, fails fast)
- gate periodic cloud sync on auto_sync; suppress detached background sync for
  proxy commands instead of flipping the flag
- pmg proxy env --export emits shell-quoted lines for eval (spaces survive)
- extract shared flows.BuildCachedMalysisAnalyzer, dropping the analyzer+cache
  duplication between proxy flow and proxy server
- add internal/proxyserver/doc.go documenting the package + boundary vs flows
- E2E: assert malicious installs are blocked (drop continue-on-error)
- docs: trim Commands/State-file to user contracts; refresh bind address

* refactor(proxy): proactive alignment fixes from whole-PR review

- gate the shutdown cloud flush on auto_sync too, matching the periodic ticker
  (auto_sync consistently controls all daemon-driven cloud delivery)
- ResolveStatePath takes cacheDir instead of *RuntimeConfig, keeping state.go
  free of config dependency
- drop the empty-host comment in listenAddr; keep the loopback guard so a blank
  host never silently binds all interfaces

* fix: Decouple localdb with malysis analyser construction

* fix: Persist global args before proxy server daemon exec

* fix: GitHub Action for cloud auto-sync in server mode

---------

Co-authored-by: Abhisek Datta <abhisek.datta@gmail.com>
This commit is contained in:
Sahil Bansal
2026-06-26 11:19:28 +05:30
committed by GitHub
co-authored by Abhisek Datta
parent 7ee4187d50
commit c47776db27
41 changed files with 1948 additions and 235 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/persistent-proxy.md
+189
View File
@@ -0,0 +1,189 @@
# Persistent Proxy Server
The persistent proxy server runs PMG's MITM proxy as a long-lived process that
intercepts **every** supported package manager invocation in an environment via
standard proxy environment variables, without shims, aliases, or wrapping each
command with `pmg`. It is built for non-interactive environments, primarily
CI/CD pipelines (e.g. GitHub Actions), where the environment can be configured
once for the whole job.
It builds on the generic MITM proxy described in [proxy.md](./proxy.md), reusing
the same interceptor chain, malware analyzer, and certificate manager. The
difference is the **lifecycle**: instead of PMG starting an ephemeral proxy
around a single subprocess, the proxy is started once, advertises itself to the
other `pmg proxy` commands, and serves many package manager processes until it
is stopped.
## Default proxy mode vs. persistent proxy server
PMG's default proxy mode (see [proxy.md](./proxy.md)) wraps a single command.
`pmg npm install` starts an ephemeral proxy, runs `npm` as a child with proxy
env vars injected, then tears the proxy down. The persistent server decouples
these steps.
| | Default proxy mode | Persistent proxy server |
| --- | --- | --- |
| Invocation | `pmg npm install` (wrapped) | bare `npm install` (no wrapper) |
| Proxy lifetime | One subprocess | Until `pmg proxy stop` |
| Who runs the PM | PMG (as a child) | The user / CI directly |
| Ecosystems served | The one being run | All supported (npm + PyPI) |
| Confirmation on malware | Interactive prompt (TTY) | Auto-block (non-interactive) |
| Reporting | At subprocess exit | At `pmg proxy stop` |
| Target | Local dev | CI/CD pipelines |
## How it works
The diagram below shows the order of events in a CI job. The `pmg proxy`
commands run in separate workflow steps and coordinate through the running
daemon.
```mermaid
sequenceDiagram
participant CI as CI Job
participant Proxy as Proxy Daemon
participant PM as Package Manager
participant Cloud as SafeDep Cloud
CI->>Proxy: pmg proxy start --daemon
Proxy-->>CI: ready (addr, ca path)
CI->>CI: pmg proxy env (set HTTP_PROXY + CA vars)
PM->>Proxy: package download (via HTTP_PROXY)
Proxy->>Proxy: analyze package
Proxy-->>PM: allow, or 403 block + record event
Proxy->>Cloud: periodic sync of events (while serving)
CI->>Proxy: pmg proxy stop --fail-on-violation
Proxy->>Cloud: final flush of remaining events
Proxy-->>CI: exit non-zero if anything was blocked
```
## Usage
The persistent server targets non-interactive CI/CD. For local development use
the default proxy mode (`pmg npm install`), which keeps the interactive malware
confirmation prompt. The persistent server auto-blocks without prompting.
GitHub Actions (raw commands):
```yaml
- run: pmg proxy start --daemon
- run: pmg proxy env >> "$GITHUB_ENV"
- run: npm ci
- run: pmg proxy stop --fail-on-violation
if: always()
```
GitHub Actions (via the [safedep/pmg action](../action.yml) `server-mode`):
```yaml
- uses: safedep/pmg@v1
with:
server-mode: true
api-key: ${{ secrets.SAFEDEP_API_KEY }}
tenant-id: ${{ secrets.SAFEDEP_TENANT_ID }}
- run: npm ci # intercepted automatically
- name: Enforce PMG policy
if: always()
run: pmg proxy stop --fail-on-violation
```
In `server-mode`, the action starts the daemon and injects env vars instead of
installing shims. Because composite actions cannot run an automatic cleanup
step, the final `pmg proxy stop --fail-on-violation` step is required. It stops
the proxy (the daemon flushes events to the cloud during shutdown) and fails the
job on a block.
## Commands
```bash
pmg proxy start # start the proxy (foreground, or detached with --daemon)
pmg proxy stop # stop the proxy and report the outcome
pmg proxy env # print env vars that route package managers through it
pmg proxy status # report whether a proxy is running
```
Run `pmg proxy <command> --help` for flags. `--daemon` is **Unix only**: on
Windows it returns a clear "not supported" error, and the foreground
`pmg proxy start` still works. To run multiple independent proxies on one host,
give each a distinct `--state` path and `--port`.
## Bind address
The proxy binds `127.0.0.1` on a random port by default, reachable only from the
host (the right choice for CI and local use). Override with `--host`/`--port`, or
the `proxy.server.listen_host`/`listen_port` config (flags take precedence).
Bind a non-loopback address (e.g. `--host 0.0.0.0`) **only** for a deliberately
hosted deployment: it exposes the MITM proxy to the network, and every client
routed through it has its HTTPS intercepted and must trust the PMG CA.
## Certificate trust
The proxy performs TLS MITM, so clients must trust its CA. Trust is delivered
through **environment variables, not the OS trust store**. `pmg proxy env`
always emits the cert-path variables pointing at the proxy's CA bundle:
`NODE_EXTRA_CA_CERTS`, `SSL_CERT_FILE`, `REQUESTS_CA_BUNDLE`, `PIP_CERT`,
`YARN_HTTPS_CA_FILE_PATH`. Package managers pick these up from the job
environment and trust the proxy's CA, with no OS trust-store install required.
This is deliberate: whether a tool consults the OS trust store varies by tool,
version, and config (npm/Node ignore it by default; modern pip can read it;
`requests`/`certifi` ship their own bundle). The cert-path vars work across all
of them, and are harmlessly ignored by tools that do read the OS store.
As a result `pmg setup cert install` is **not** needed for the persistent proxy.
If a persisted CA from `pmg setup cert install` exists the proxy reuses it,
otherwise it generates an ephemeral one. Either way `pmg proxy env` carries the
trust. OS trust-store install (`pmg setup cert install --system`) is
intentionally not used: it needs root (breaking container and locked-down
runners), persistently installs a MITM-capable CA into the machine trust store,
and still does not remove the need for the env vars.
Loopback addresses are always excluded from proxying via `NO_PROXY`
(`localhost,127.0.0.1,::1`).
## Cloud event sync
When SafeDep Cloud is enabled, malware-block events must reach the cloud even on
ephemeral CI runners that are destroyed immediately after the job. The daemon
owns delivery: it records each blocked package to a durable local event log as
it happens, syncs pending events to SafeDep Cloud periodically while serving, and
flushes whatever remains on shutdown.
`pmg proxy stop` reports the recorded result (`Synced N event(s) to SafeDep
Cloud`, or a `Cloud sync failed` line). A flush failure is surfaced but does not
mask the fail-on-violation exit code.
## Fail on violation
By default `pmg proxy stop` just stops the proxy and exits `0`. Failing the CI
job on a policy violation is opt-in via `--fail-on-violation`.
- It exits non-zero when any package was blocked.
- It **fails closed**. If the daemon shut down without writing a verifiable
final state (e.g. it crashed), `--fail-on-violation` also fails, because a
security gate must not pass on an unverifiable run.
The package manager's own non-zero exit (from the `403` on a blocked download)
is a separate signal. `--fail-on-violation` gives an authoritative gate from the
proxy regardless of how the package manager reported the failure.
## Limitations
- **Unix-only daemon.** `--daemon` is not supported on Windows (foreground mode
works).
- **Non-interactive only.** There is no interactive confirmation; flagged
packages are always auto-blocked. This is intentional for CI.
- **Single proxy per state file.** Starting a second proxy that points at the
same state file is refused while one is running.
- **System-level trust enforcement is out of scope.** The server relies on env
var propagation. Enforcing interception for `sudo`-scrubbed environments (e.g.
via `iptables`) and system-wide install (`pmg setup install --system`) are
tracked separately.
## References
- [proxy.md](./proxy.md) is the underlying generic MITM proxy server
- [config.md](./config.md) is the configuration schema (cloud, proxy, cache dir)
- [action.yml](../action.yml) is the PMG GitHub Action (`server-mode`)
docs/proxy-mode.md
+4
View File
@@ -68,3 +68,7 @@ Legacy variables `PMG_PROXY_MODE` and `PMG_PROXY_INSTALL_ONLY` (for the old flat
| `pip` | ✅ |
| `uv` | ✅ |
| `poetry` | ✅ |
## References
- [Persistent Proxy Mode](./persistent-proxy.md)