feat(proxy): persistent proxy server mode (#351)

* refactor(flows): extract SetupCACertificate for reuse

Move the CA load/generate/merge logic out of proxyFlow into an exported
flows.SetupCACertificate so the persistent proxy server can reuse it.

* feat(proxy): add persistent proxy server with start/stop/env/status

Introduces 'pmg proxy' commands backed by internal/proxyserver: a long-lived
MITM proxy that intercepts package managers via env vars (no shims). Supports
--daemon (Unix), --state, --port; generic 'env' output that skips cert vars
when the CA is OS-trusted; opt-in 'stop --fail-on-violation' (fail-closed on
crash) with a synchronous cloud event flush; and the malysis analysis cache.

* feat(action): add server-mode for persistent proxy

When server-mode=true the action starts the proxy daemon and injects proxy
env vars into the job instead of installing shims.

* test(proxy): add persistent proxy server E2E workflow

* docs(readme): document persistent proxy server mode

* fix(proxy): create cache dir before writing state file and daemon log

On a fresh CI runner the cache directory does not exist yet; os.OpenFile and
os.WriteFile do not create parent dirs, so 'pmg proxy start --daemon' failed
with 'no such file or directory'. MkdirAll the parent before writing.

* docs: add persistent proxy server architecture doc

* refactor proxyserver

* fix(proxy): always emit cert env vars instead of skipping on OS-trust status

npm/pip/yarn/requests trust the MITM CA inconsistently across tools, versions,
and configs; many still use bundled CA stores. Always emitting the cert-path
env vars is the conservative choice that works regardless, and is harmless for
tools that read the OS store (they ignore the vars). Skipping them when a
system CA exists would silently break any tool still on a bundled store.

* refactor(proxy): drop redundant audit init in daemon; rely on main.go

main.go's PersistentPreRun already initializes the audit pipeline for every
command (including the daemon's re-exec'd child) and closes it at process exit.
Re-initializing in proxyserver.Run created a second auditor and a second
cloud-sync WAL connection, orphaning the first. Removing it makes the daemon
consistent with the normal proxy flow, which never self-initializes audit.

* fix(proxy): bypass proxy env when flushing events to cloud on stop

pmg proxy stop inherits HTTP(S)_PROXY (injected by 'pmg proxy env') pointing at
the PMG proxy it just shut down. The cloud sync gRPC client honored those vars
and routed api.safedep.io through the dead proxy, failing with 'connection
refused' so no events were delivered. Clear the proxy env vars before the sync
so PMG's own cloud traffic goes direct.

* chore(proxy): address review feedback

- configurable bind host via proxy.server.listen_host (default loopback)
- proxy commands use ui.ErrorExit instead of returning errors to cobra
- rename errcode to ProxyPolicyViolation (covers malware + cooldown)
- share cloud sync via audit.DrainToCloud (de-dup with cmd/cloud/sync)
- centralize proxy CA bundle path in certmanager
- docs: persistent proxy cert trust + bind address

* fix(proxy): show real message on fail-on-violation error

stopExitError set only WithMsg, but ui.ErrorExit renders HumanError, so the
framed error showed 'no human-readable message available'. Set both from one
string, and emit the framed error before the stdout summary so the blocked
count is stated once.

* fix(proxy): flush cloud events from the daemon, not stop

The stop process inherits HTTP_PROXY (from 'pmg proxy env'), so its cloud
client routed api.safedep.io through the already-stopped proxy and failed with
connection refused. Move the flush into the daemon's shutdown, which has no
proxy env (it started before env injection) and dials SafeDep directly.

- daemon flushes on shutdown via audit.DrainToCloud and records the result in
  the state file; stop surfaces it (on both success and fail-on-violation
  paths) since the daemon's own logs aren't visible to stop
- coordinate stop's wait with the daemon shutdown budget; on timeout, error
  out without reading stale state or deleting the file (fail-closed)
- persist blocked count before the flush so the gate stays correct if the
  flush hangs or the daemon is killed mid-flush
- remove now-redundant cloud_flush.go

* disable auto-sync for proxy cmds

* feat(proxy): periodic cloud sync + move proxy env vars to packagemanager

- daemon runs a periodic cloud-sync ticker so the shutdown flush stays small;
  the run total is reported by stop, and shutdown timeouts are coordinated
- move EnvVarForProxy from config to packagemanager (it is package-manager
  knowledge); the shared function now builds the proxy URL and NO_PROXY itself,
  removing the duplicated construction in the per-command and persistent paths
- relocate the #319 yarn and #339 IPv6 regression tests alongside the function
- enable cloud sync in the persistent-proxy E2E workflow and fix the stale
  internal/proxystate path filter

* refactor(proxy): rename cloudFlushLockTimeout to cloudFlushLockWait

Consistent timeout naming: *LockWait is the lock-acquire bound, *Timeout is
the sync-RPC bound. Previously the final-flush pair was cloudFlushLockTimeout
vs cloudFlushTimeout — two lookalike names for different operations.

* refactor(proxy): extract cloudFlush and trim duplicate shutdown comments

The shutdown's final-flush block is now a cloudFlush helper, symmetric with
startCloudSyncLoop (one-shot vs loop). Removed the triplicated ticker/lock
contention comments, keeping the contract on the function doc and one-line
pointers at the call sites.

* docs: update persistent proxy cloud sync to daemon-owned model

The daemon now owns cloud delivery (periodic sync while serving + final flush
on shutdown); stop signals it, waits, and reports the result. Rewrite the Cloud
event sync section, fix stop attributions, add the cloud_sync state field, and
update the sequence diagram.

* docs: move Usage section up below How it works

Put the copy-paste recipes near the top so users find them before the internals.

* refactor(proxy): address PR review feedback

- configurable bind host/port via --host/--port flags + config (listen_host,
  listen_port), bound directly to config fields per PMG's flag pattern
- daemon log path via --log-file and readiness timeout in ProxyDaemonConfig;
  Daemonize no longer owns path policy (caller validates, fails fast)
- gate periodic cloud sync on auto_sync; suppress detached background sync for
  proxy commands instead of flipping the flag
- pmg proxy env --export emits shell-quoted lines for eval (spaces survive)
- extract shared flows.BuildCachedMalysisAnalyzer, dropping the analyzer+cache
  duplication between proxy flow and proxy server
- add internal/proxyserver/doc.go documenting the package + boundary vs flows
- E2E: assert malicious installs are blocked (drop continue-on-error)
- docs: trim Commands/State-file to user contracts; refresh bind address

* refactor(proxy): proactive alignment fixes from whole-PR review

- gate the shutdown cloud flush on auto_sync too, matching the periodic ticker
  (auto_sync consistently controls all daemon-driven cloud delivery)
- ResolveStatePath takes cacheDir instead of *RuntimeConfig, keeping state.go
  free of config dependency
- drop the empty-host comment in listenAddr; keep the loopback guard so a blank
  host never silently binds all interfaces

* fix: Decouple localdb with malysis analyser construction

* fix: Persist global args before proxy server daemon exec

* fix: GitHub Action for cloud auto-sync in server mode

---------

Co-authored-by: Abhisek Datta <abhisek.datta@gmail.com>
This commit is contained in:
Sahil Bansal
2026-06-26 11:19:28 +05:30
committed by GitHub
co-authored by Abhisek Datta
parent 7ee4187d50
commit c47776db27
41 changed files with 1948 additions and 235 deletions
+110
View File
@@ -0,0 +1,110 @@
# .github/workflows/persistent-proxy-e2e.yml
name: Persistent Proxy E2E
on:
workflow_dispatch:
pull_request:
branches:
- main
paths:
- "cmd/proxy/**"
- "internal/proxyserver/**"
- "internal/flows/cert.go"
- "action.yml"
permissions:
contents: read
jobs:
persistent-proxy-e2e:
name: Persistent Proxy E2E
runs-on: ubuntu-latest
timeout-minutes: 10
env:
SAFEDEP_API_KEY: ${{ secrets.SAFEDEP_CLOUD_API_KEY }}
SAFEDEP_TENANT_ID: ${{ secrets.SAFEDEP_CLOUD_TENANT_DOMAIN}}
PMG_CLOUD_ENABLED: "true"
PMG_CLOUD_ENDPOINT_ID: github-actions/${{ github.repository }}
defaults:
run:
shell: bash
steps:
- name: Checkout Source
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Setup Go
uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6
with:
go-version-file: go.mod
cache: true
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: "24"
- name: Setup Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.11"
- name: Build PMG
run: make
- name: Add pmg to PATH
run: echo "$GITHUB_WORKSPACE/bin" >> "$GITHUB_PATH"
- name: Start proxy daemon
run: pmg proxy start --daemon
- name: Inject proxy env
run: pmg proxy env >> "$GITHUB_ENV"
- name: Verify proxy env is set
run: |
echo "HTTP_PROXY=$HTTP_PROXY"
test -n "$HTTP_PROXY"
- name: Benign npm install succeeds
run: |
mkdir benign-test && cd benign-test
npm init -y
npm install lodash@4.17.21
test -d node_modules/lodash
cd .. && rm -rf benign-test
- name: Malicious npm install is blocked
run: |
mkdir mal-npm && cd mal-npm
npm init -y
# The install MUST fail (proxy blocks it). Assert that, instead of
# continue-on-error which would hide a regression where it succeeds.
if npm --no-cache --prefer-online install safedep-test-pkg@0.1.3; then
echo "ERROR: malicious npm install succeeded but should have been blocked"
exit 1
fi
echo "OK: malicious npm install was blocked"
cd .. && rm -rf mal-npm
- name: Malicious pip install is blocked (python3 -m pip)
run: |
python3 -m venv venv && source venv/bin/activate
if python3 -m pip install safedep-test-pkg; then
echo "ERROR: malicious pip install succeeded but should have been blocked"
deactivate
exit 1
fi
echo "OK: malicious pip install was blocked"
deactivate && rm -rf venv
- name: Stop proxy and verify it fails on the blocks
if: always()
run: |
# We blocked packages above, so --fail-on-violation must exit non-zero.
# Assert that (a green test means the gate works), and always run so the
# daemon is stopped even if an earlier step failed.
if pmg proxy stop --fail-on-violation; then
echo "ERROR: stop should have exited non-zero (packages were blocked)"
exit 1
fi
echo "OK: stop correctly failed on the policy violation"
+9 -5
View File
@@ -209,12 +209,15 @@ Protect CI workflows with one step. PMG analyzes every `npm install`,
`pip install`, etc. in the job.
```yaml
# Consider pinning third-party Actions to a full commit SHA
- uses: actions/setup-node@v6
with:
node-version: 24
- uses: safedep/pmg@v1
- run: npm ci
with:
server-mode: true
- run: npm ci # intercepted via HTTP_PROXY automatically
- name: Enforce PMG policy
if: always()
run: pmg proxy stop --fail-on-violation # stops the daemon, fails the job on a block
```
By default you get malware blocking and dependency cooldown. Sandbox isolation
@@ -262,6 +265,7 @@ PMG builds are reproducible and signed.
- [Dependency Cooldown](docs/dependency-cooldown.md)
- [Caching](docs/caching.md)
- [Proxy Mode Architecture](docs/proxy-mode.md)
- [Persistent Proxy Server](docs/persistent-proxy.md)
- [Certificate Authority](docs/cert.md)
- [Sandboxing](docs/sandbox.md)
+33 -8
View File
@@ -73,6 +73,11 @@ inputs:
required: false
default: "false"
server-mode:
description: Run PMG as a persistent proxy server instead of shims. The proxy is started as a daemon and HTTP_PROXY/cert env vars are exported to the job. Requires a job-end "pmg proxy stop --fail-on-violation" step.
required: false
default: "false"
runs:
using: composite
steps:
@@ -235,6 +240,7 @@ runs:
IN_VERBOSITY: ${{ inputs.verbosity }}
IN_DISABLE_TELEMETRY: ${{ inputs.disable-telemetry }}
IN_SKIP_EVENT_LOGGING: ${{ inputs.skip-event-logging }}
IN_SERVER_MODE: ${{ inputs.server-mode }}
GH_REPOSITORY: ${{ github.repository }}
run: |
set -euo pipefail
@@ -258,12 +264,18 @@ runs:
export_var SAFEDEP_TENANT_ID "$IN_TENANT_ID"
export_var PMG_CLOUD_ENABLED "true"
# PMG's opportunistic background auto-sync is designed for long-lived
# workstations. On an ephemeral runner the detached child can be
# torn down before it drains the WAL, and the WAL goes with it. Turn
# it off and rely on an explicit `pmg cloud sync` at job-end if the
# user wants events delivered.
export_var PMG_CLOUD_AUTO_SYNC_ENABLED "false"
if [ "$IN_SERVER_MODE" = "true" ]; then
# In server mode the proxy daemon owns delivery and uses auto_sync
# to gate both periodic sync and the shutdown flush.
export_var PMG_CLOUD_AUTO_SYNC_ENABLED "true"
else
# PMG's opportunistic background auto-sync is designed for
# long-lived workstations. On an ephemeral runner the detached child
# can be torn down before it drains the WAL, and the WAL goes with
# it. Turn it off and rely on an explicit `pmg cloud sync` at
# job-end if the user wants events delivered.
export_var PMG_CLOUD_AUTO_SYNC_ENABLED "false"
fi
if [ -n "$IN_ENDPOINT_ID" ]; then
export_var PMG_CLOUD_ENDPOINT_ID "$IN_ENDPOINT_ID"
@@ -292,14 +304,27 @@ runs:
export_var PMG_DISABLE_TELEMETRY "$IN_DISABLE_TELEMETRY"
export_var PMG_SKIP_EVENT_LOGGING "$IN_SKIP_EVENT_LOGGING"
- name: Run pmg setup install
- name: Run pmg setup install (shim mode)
if: inputs.server-mode != 'true'
shell: bash
run: pmg setup install
- name: Add PMG shims to PATH
- name: Add PMG shims to PATH (shim mode)
if: inputs.server-mode != 'true'
shell: bash
run: echo "$HOME/.pmg/bin" >> "$GITHUB_PATH"
- name: Start PMG proxy server (server mode)
if: inputs.server-mode == 'true'
shell: bash
run: |
set -euo pipefail
# No cert install needed: `pmg proxy env` exports the CA path via
# NODE_EXTRA_CA_CERTS/SSL_CERT_FILE/PIP_CERT, so package managers trust
# the proxy without touching the OS trust store.
pmg proxy start --daemon
pmg proxy env >> "$GITHUB_ENV"
- name: Verify PMG
shell: bash
run: |
+7 -44
View File
@@ -1,10 +1,9 @@
package cloud
import (
"context"
"errors"
"time"
"github.com/safedep/dry/log"
"github.com/safedep/dry/usefulerror"
"github.com/safedep/pmg/config"
"github.com/safedep/pmg/errcodes"
@@ -43,50 +42,14 @@ func runSync(cmd *cobra.Command, args []string) error {
WithHelp("Set 'cloud.enabled: true' in PMG config to enable cloud sync"))
}
lock := audit.NewSyncLock(cfg.CloudSyncLockPath())
lockCtx, lockCancel := context.WithTimeout(cmd.Context(), manualSyncLockTimeout)
defer lockCancel()
locked, err := lock.TryLockContext(lockCtx, 250*time.Millisecond)
synced, err := audit.DrainToCloud(cmd.Context(), cfg, manualSyncLockTimeout, syncTimeout)
if err != nil {
ui.ErrorExit(usefulerror.NewUsefulError().
Wrap(err).
WithCode(errcodes.Lifecycle).
WithHumanError("Failed to acquire cloud sync lock").
WithHelp("Another sync may be in progress; try again shortly"))
}
if !locked {
ui.ErrorExit(usefulerror.NewUsefulError().
WithCode(errcodes.Lifecycle).
WithHumanError("Another cloud sync is already in progress").
WithHelp("Wait for the in-progress sync to finish, then try again"))
}
defer func() {
if err := lock.Unlock(); err != nil {
log.Warnf("failed to release cloud sync lock: %v", err)
if errors.Is(err, audit.ErrSyncInProgress) {
ui.ErrorExit(usefulerror.NewUsefulError().
WithCode(errcodes.Lifecycle).
WithHumanError("Another cloud sync is already in progress").
WithHelp("Wait for the in-progress sync to finish, then try again"))
}
}()
ctx, cancel := context.WithTimeout(cmd.Context(), syncTimeout)
defer cancel()
bundle, err := audit.NewSyncClientBundle(cfg)
if err != nil {
ui.ErrorExit(usefulerror.NewUsefulError().
Wrap(err).
WithCode(errcodes.Lifecycle).
WithHumanError("Failed to initialize cloud sync client").
WithHelp("Run 'pmg cloud login' to store credentials, or set SAFEDEP_API_KEY and SAFEDEP_TENANT_ID environment variables"))
}
defer func() {
if err := bundle.Close(); err != nil {
log.Warnf("failed to close sync client: %v", err)
}
}()
synced, err := bundle.Sync(ctx)
recordLastSyncAttempt(cfg)
if err != nil {
ui.ErrorExit(usefulerror.NewUsefulError().
Wrap(err).
WithCode(errcodes.Network).
+74
View File
@@ -0,0 +1,74 @@
package proxy
import (
"bufio"
"fmt"
"os"
"strings"
"github.com/safedep/pmg/config"
"github.com/safedep/pmg/internal/proxyserver"
"github.com/safedep/pmg/internal/ui"
"github.com/spf13/cobra"
)
var envExportFlag bool
func newEnvCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "env",
Short: "Print proxy environment variables (KEY=VALUE per line)",
Long: "Print proxy environment variables as KEY=VALUE lines.\n\n" +
"GitHub Actions: pmg proxy env >> \"$GITHUB_ENV\"\n" +
"Shell: eval \"$(pmg proxy env --export)\"\n\n" +
"Use --export for shell `eval`: it emits quoted `export KEY='VALUE'`\n" +
"lines that survive values containing spaces. The\n" +
"default raw KEY=VALUE form is for $GITHUB_ENV, which must not be quoted.",
RunE: runEnv,
}
cmd.Flags().BoolVar(&envExportFlag, "export", false, "Emit shell `export KEY='VALUE'` lines for `eval`")
return cmd
}
func runEnv(_ *cobra.Command, _ []string) error {
cfg := config.Get()
statePath := proxyserver.ResolveStatePath(stateFlag, cfg.CacheDir())
vars, err := proxyserver.EnvVars(statePath)
if err != nil {
ui.ErrorExit(err)
}
w := bufio.NewWriter(os.Stdout)
for _, v := range vars {
line := v
if envExportFlag {
line = exportLine(v)
}
if _, werr := fmt.Fprintln(w, line); werr != nil {
ui.ErrorExit(fmt.Errorf("write env var: %w", werr))
}
}
if err := w.Flush(); err != nil {
ui.ErrorExit(err)
}
return nil
}
// exportLine turns "KEY=VALUE" into a shell-safe `export KEY='VALUE'`, so values
// with spaces (e.g. the macOS "Application Support" path) survive `eval`.
func exportLine(kv string) string {
k, v, ok := strings.Cut(kv, "=")
if !ok {
return kv
}
return fmt.Sprintf("export %s=%s", k, shellSingleQuote(v))
}
// shellSingleQuote wraps s in single quotes, escaping any embedded single quote
// as '\” (close, escaped quote, reopen).
func shellSingleQuote(s string) string {
return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'"
}
+20
View File
@@ -0,0 +1,20 @@
package proxy
import "github.com/spf13/cobra"
// stateFlag binds the persistent --state flag shared by all proxy subcommands.
var stateFlag string
func NewProxyCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "proxy",
Short: "Manage the persistent PMG proxy server",
}
cmd.AddCommand(newStartCommand())
cmd.AddCommand(newStopCommand())
cmd.AddCommand(newEnvCommand())
cmd.AddCommand(newStatusCommand())
cmd.PersistentFlags().StringVar(&stateFlag, "state", "",
"Path to the proxy state file (default: <cache-dir>/proxy-state.json)")
return cmd
}
+103
View File
@@ -0,0 +1,103 @@
package proxy
import (
"fmt"
"os"
"path/filepath"
"strconv"
"github.com/safedep/pmg/config"
"github.com/safedep/pmg/internal/proxyserver"
"github.com/safedep/pmg/internal/ui"
"github.com/spf13/cobra"
)
var (
daemonFlag bool
logFileFlag string
foregroundInternalFlag bool
)
func newStartCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "start",
Short: "Start the persistent PMG proxy server",
RunE: runStart,
}
// Bind --host/--port directly onto the config fields with the loaded config
// values as defaults, matching PMG's flag pattern (see config/cobra.go): a
// supplied flag overwrites the field, otherwise the config value stands.
// Precedence: flag > env > config file > default.
srv := &config.Get().Config.Proxy.Server
cmd.Flags().BoolVarP(&daemonFlag, "daemon", "D", false, "Run the proxy as a detached background process")
cmd.Flags().StringVar(&srv.ListenHost, "host", srv.ListenHost, "Host to bind")
cmd.Flags().IntVar(&srv.ListenPort, "port", srv.ListenPort, "Port to bind (0 = a random free port)")
cmd.Flags().StringVar(&logFileFlag, "log-file", "", "File for the daemon's output (default: <cache-dir>/proxy.log)")
cmd.Flags().BoolVar(&foregroundInternalFlag, "foreground-internal", false, "Internal: run the foreground server (used by --daemon)")
if err := cmd.Flags().MarkHidden("foreground-internal"); err != nil {
panic(err)
}
return cmd
}
func runStart(cmd *cobra.Command, _ []string) error {
cfg := config.Get()
statePath := proxyserver.ResolveStatePath(stateFlag, cfg.CacheDir())
host := cfg.Config.Proxy.Server.ListenHost
port := cfg.Config.Proxy.Server.ListenPort
if daemonFlag && !foregroundInternalFlag {
if err := startDaemon(cmd, cfg, statePath, host, port); err != nil {
ui.ErrorExit(err)
}
return nil
}
if err := proxyserver.Run(cmd.Context(), cfg, statePath, host, port); err != nil {
ui.ErrorExit(err)
}
return nil
}
func startDaemon(cmd *cobra.Command, cfg *config.RuntimeConfig, statePath, host string, port int) error {
exe, err := os.Executable()
if err != nil {
return fmt.Errorf("resolve executable: %w", err)
}
// Daemon log: the --log-file flag if set, else <cache-dir>/proxy.log. The
// caller owns this path, so ensure its parent directory exists here.
logPath := logFileFlag
if logPath == "" {
logPath = filepath.Join(cfg.CacheDir(), "proxy.log")
}
if err := os.MkdirAll(filepath.Dir(logPath), 0o700); err != nil {
return fmt.Errorf("create daemon log dir: %w", err)
}
args := daemonArgs(cmd, statePath, host, port)
daemonCfg := proxyserver.ProxyDaemonConfig{
LogPath: logPath,
ReadyTimeout: proxyserver.DefaultDaemonReadyTimeout,
}
state, err := proxyserver.Daemonize(daemonCfg, statePath, exe, args)
if err != nil {
return err
}
_, werr := fmt.Fprintf(os.Stdout, "PMG proxy daemon started on %s (pid %d)\n", state.Addr, state.PID)
return werr
}
func daemonArgs(cmd *cobra.Command, statePath, host string, port int) []string {
args := append([]string{}, config.ChangedConfigFlagArgs(cmd)...)
return append(args,
"proxy", "start", "--foreground-internal",
"--state", statePath,
"--host", host,
"--port", strconv.Itoa(port),
)
}
+41
View File
@@ -0,0 +1,41 @@
package proxy
import (
"testing"
"github.com/safedep/pmg/config"
"github.com/spf13/cobra"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestDaemonArgsPrependsChangedConfigFlags(t *testing.T) {
root := &cobra.Command{Use: "pmg"}
config.ApplyCobraFlags(root)
var got []string
start := &cobra.Command{
Use: "start",
Run: func(cmd *cobra.Command, _ []string) {
got = daemonArgs(cmd, "/tmp/proxy-state.json", "127.0.0.1", 9000)
},
}
proxyCmd := &cobra.Command{Use: "proxy"}
proxyCmd.AddCommand(start)
root.AddCommand(proxyCmd)
root.SetArgs([]string{
"--paranoid",
"--skip-dependency-cooldown",
"proxy", "start",
})
require.NoError(t, root.Execute())
assert.Equal(t, []string{
"--paranoid=true",
"--skip-dependency-cooldown=true",
"proxy", "start", "--foreground-internal",
"--state", "/tmp/proxy-state.json",
"--host", "127.0.0.1",
"--port", "9000",
}, got)
}
+42
View File
@@ -0,0 +1,42 @@
package proxy
import (
"fmt"
"os"
"github.com/safedep/pmg/config"
"github.com/safedep/pmg/internal/proxyserver"
"github.com/safedep/pmg/internal/ui"
"github.com/spf13/cobra"
)
func newStatusCommand() *cobra.Command {
return &cobra.Command{
Use: "status",
Short: "Show the status of the persistent PMG proxy server",
RunE: runStatus,
}
}
func runStatus(_ *cobra.Command, _ []string) error {
cfg := config.Get()
statePath := proxyserver.ResolveStatePath(stateFlag, cfg.CacheDir())
st := proxyserver.GetStatus(statePath)
var line string
switch {
case !st.Found:
line = "PMG proxy: not running (no state file)\n"
case st.Running:
line = fmt.Sprintf("PMG proxy: running (pid %d, addr %s, ca %s)\n", st.PID, st.Addr, st.CACert)
default:
line = fmt.Sprintf("PMG proxy: stopped (stale state for pid %d — run 'pmg proxy stop' to clean up)\n", st.PID)
}
if _, err := fmt.Fprint(os.Stdout, line); err != nil {
ui.ErrorExit(err)
}
return nil
}
+100
View File
@@ -0,0 +1,100 @@
package proxy
import (
"fmt"
"os"
"github.com/safedep/dry/usefulerror"
"github.com/safedep/pmg/config"
"github.com/safedep/pmg/errcodes"
"github.com/safedep/pmg/internal/proxyserver"
"github.com/safedep/pmg/internal/ui"
"github.com/spf13/cobra"
)
var failOnViolation bool
func newStopCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "stop",
Short: "Stop the running persistent PMG proxy server",
// Failures here are user-facing policy outcomes, not usage errors.
SilenceUsage: true,
RunE: runStop,
}
cmd.Flags().BoolVar(&failOnViolation, "fail-on-violation", false,
"Exit non-zero if any package was blocked during the proxy session")
return cmd
}
func runStop(_ *cobra.Command, _ []string) error {
cfg := config.Get()
statePath := proxyserver.ResolveStatePath(stateFlag, cfg.CacheDir())
res, err := proxyserver.Stop(statePath)
if err != nil {
ui.ErrorExit(err)
}
// Surface the daemon's cloud flush outcome before the violation gate, so it
// shows on both the success and the failure path (the daemon's own logs are
// not visible to this process). nil means cloud sync is disabled.
if res.CloudSync != nil {
line := fmt.Sprintf("Synced %d event(s) to SafeDep Cloud\n", res.CloudSync.Synced)
if res.CloudSync.Error != "" {
line = fmt.Sprintf("Cloud sync failed: %s\n", res.CloudSync.Error)
}
if _, werr := fmt.Fprint(os.Stdout, line); werr != nil {
ui.ErrorExit(werr)
}
}
// On a policy violation the framed error states the blocked count, so exit
// here before the plain summary to avoid stating the count twice.
if verr := stopExitError(res, failOnViolation); verr != nil {
ui.ErrorExit(verr)
}
summary := fmt.Sprintf("PMG proxy stopped — %d package(s) blocked\n", res.BlockedCount)
if !res.StateVerified {
summary = fmt.Sprintf("PMG proxy (pid %d) stopped (final state unavailable)\n", res.PID)
}
if _, werr := fmt.Fprint(os.Stdout, summary); werr != nil {
ui.ErrorExit(werr)
}
return nil
}
// stopExitError maps a stop result to the command's exit status. With
// --fail-on-violation: any blocked package fails, and an unverifiable final
// state (e.g. a crashed proxy) fails closed — a security gate must not pass on
// an unverifiable run. Without the flag, stop always succeeds.
func stopExitError(res proxyserver.StopResult, failOnViolation bool) error {
if !failOnViolation {
return nil
}
// HumanError drives the displayed message; Msg drives Error()/logs and the
// --verbose tail. Set both from one string so verbose shows the real
// message instead of usefulerror's "unknown error" fallback.
if !res.StateVerified {
msg := "Proxy shut down but the blocked-package count could not be verified"
return usefulerror.NewUsefulError().
WithCode(errcodes.ProxyPolicyViolation).
WithHumanError(msg).
WithMsg(msg).
WithHelp("The proxy may have crashed; treat this run as failed and re-run")
}
if res.BlockedCount > 0 {
msg := fmt.Sprintf("%d package(s) were blocked by the proxy", res.BlockedCount)
return usefulerror.NewUsefulError().
WithCode(errcodes.ProxyPolicyViolation).
WithHumanError(msg).
WithMsg(msg).
WithHelp("Review the proxy logs for details on blocked packages")
}
return nil
}
+38
View File
@@ -0,0 +1,38 @@
package proxy
import (
"testing"
"github.com/safedep/pmg/internal/proxyserver"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestStopExitError(t *testing.T) {
t.Run("no flag, blocks present -> nil", func(t *testing.T) {
err := stopExitError(proxyserver.StopResult{BlockedCount: 3, StateVerified: true}, false)
assert.NoError(t, err)
})
t.Run("flag, no blocks -> nil", func(t *testing.T) {
err := stopExitError(proxyserver.StopResult{BlockedCount: 0, StateVerified: true}, true)
assert.NoError(t, err)
})
t.Run("flag, blocks present -> error", func(t *testing.T) {
err := stopExitError(proxyserver.StopResult{BlockedCount: 2, StateVerified: true}, true)
require.Error(t, err)
assert.Contains(t, err.Error(), "2 package")
})
t.Run("flag, crash (unverified state) -> fail closed", func(t *testing.T) {
err := stopExitError(proxyserver.StopResult{StateVerified: false}, true)
require.Error(t, err)
assert.Contains(t, err.Error(), "could not be verified")
})
t.Run("no flag, crash -> nil", func(t *testing.T) {
err := stopExitError(proxyserver.StopResult{StateVerified: false}, false)
assert.NoError(t, err)
})
}
+2 -2
View File
@@ -8,10 +8,10 @@ import (
"os"
"path/filepath"
"github.com/safedep/dry/localdb"
"github.com/safedep/dry/log"
"github.com/safedep/pmg/analyzer/malysiscache"
"github.com/safedep/pmg/config"
"github.com/safedep/pmg/internal/localstore"
"github.com/spf13/cobra"
)
@@ -61,7 +61,7 @@ func openCache(ctx context.Context, cfg *config.RuntimeConfig) (cache *malysisca
return nil, func() {}, false, statErr
}
mgr := localdb.New(localdb.Config{Dir: cfg.LocalDBDir(), FileName: cfg.LocalDBFileName()})
mgr := localstore.NewManager(cfg)
store, serr := mgr.Store(ctx, malysiscache.Descriptor())
if serr != nil {
if cerr := mgr.Close(); cerr != nil {
+30
View File
@@ -113,6 +113,36 @@ func ApplyCobraFlags(cmd *cobra.Command) {
}
}
// ChangedConfigFlagArgs returns the explicitly supplied root config flags in a
// form that can be passed to a re-execed PMG process.
func ChangedConfigFlagArgs(cmd *cobra.Command) []string {
var args []string
for _, spec := range configFlagSpecs {
flag := cmd.Flags().Lookup(spec.name)
if flag == nil || !flag.Changed {
continue
}
name := "--" + spec.name
if flag.Value.Type() == "bool" {
args = append(args, name+"="+flag.Value.String())
continue
}
for _, value := range changedFlagValues(flag) {
args = append(args, name, value)
}
}
return args
}
func changedFlagValues(flag *pflag.Flag) []string {
if value, ok := flag.Value.(pflag.SliceValue); ok {
return value.GetSlice()
}
return []string{flag.Value.String()}
}
// RejectManagedFlagOverrides fails when the active config is a locked global
// config and the user explicitly set a flag whose value that config governs.
// Operational flags (managed == false) are unaffected, and an unlocked managed
+35
View File
@@ -83,6 +83,41 @@ func TestRejectManagedFlagOverridesDetectsInheritedFlag(t *testing.T) {
assert.Contains(t, err.Error(), "globally managed")
}
func TestChangedConfigFlagArgs(t *testing.T) {
withLockedState(t, false)
root := &cobra.Command{Use: "pmg"}
ApplyCobraFlags(root)
var got []string
child := &cobra.Command{
Use: "proxy",
Run: func(cmd *cobra.Command, _ []string) {
got = ChangedConfigFlagArgs(cmd)
},
}
root.AddCommand(child)
root.SetArgs([]string{
"--paranoid=false",
"--transitive-depth", "7",
"--sandbox-profile", "strict",
"--sandbox-allow", "read=/tmp",
"--sandbox-allow", "net-connect=registry.npmjs.org:443",
"--skip-dependency-cooldown",
"proxy",
})
require.NoError(t, root.Execute())
assert.Equal(t, []string{
"--transitive-depth", "7",
"--paranoid=false",
"--sandbox-profile", "strict",
"--sandbox-allow", "read=/tmp",
"--sandbox-allow", "net-connect=registry.npmjs.org:443",
"--skip-dependency-cooldown=true",
}, got)
}
// Proves the SSOT table is internally consistent: every spec actually binds a
// flag, every registered flag traces back to a spec (no out-of-band flags), and
// the managed classification matches intent. Catches accidental managed flips
+17
View File
@@ -165,6 +165,20 @@ type ProxyConfig struct {
Enabled bool `mapstructure:"enabled"`
InstallOnly bool `mapstructure:"install_only"`
SkipCommands map[string][]string `mapstructure:"skip_commands"`
Server ProxyServerConfig `mapstructure:"server"`
}
// ProxyServerConfig configures the persistent proxy server (`pmg proxy start`).
type ProxyServerConfig struct {
// ListenHost is the host the persistent proxy binds to. Defaults to
// 127.0.0.1 (loopback). Set to 0.0.0.0 or a specific interface only for a
// deliberately hosted deployment: a non-loopback bind exposes the MITM
// proxy to the network. The --host flag overrides this.
ListenHost string `mapstructure:"listen_host"`
// ListenPort is the port the persistent proxy binds to. 0 (default) means a
// random free port. The --port flag overrides this.
ListenPort int `mapstructure:"listen_port"`
}
// SandboxConfig configures the sandbox system for isolating package manager processes.
@@ -481,6 +495,9 @@ func DefaultConfig() RuntimeConfig {
Enabled: true,
InstallOnly: false,
SkipCommands: map[string][]string{},
Server: ProxyServerConfig{
ListenHost: "127.0.0.1",
},
},
},
DryRun: false,
+13
View File
@@ -51,6 +51,19 @@ proxy:
skip_commands:
npm: []
# Persistent proxy server (`pmg proxy start`) settings.
server:
# Host the persistent proxy binds to. Defaults to 127.0.0.1 (loopback),
# which keeps the MITM proxy private to the host (the right choice for CI
# and local use). Set to 0.0.0.0 or a specific interface ONLY for a
# deliberately hosted deployment: a non-loopback bind exposes the proxy,
# and every client must trust the PMG CA. The --host flag overrides this.
listen_host: 127.0.0.1
# Port the persistent proxy binds to. 0 means a random free port. The
# --port flag overrides this.
listen_port: 0
# Trusted packages are packages that are trusted by the user and will be ignored by the security guardrails.
# This is useful for packages that are known to be safe and are used in the application.
# Example:
+189
View File
@@ -0,0 +1,189 @@
# Persistent Proxy Server
The persistent proxy server runs PMG's MITM proxy as a long-lived process that
intercepts **every** supported package manager invocation in an environment via
standard proxy environment variables, without shims, aliases, or wrapping each
command with `pmg`. It is built for non-interactive environments, primarily
CI/CD pipelines (e.g. GitHub Actions), where the environment can be configured
once for the whole job.
It builds on the generic MITM proxy described in [proxy.md](./proxy.md), reusing
the same interceptor chain, malware analyzer, and certificate manager. The
difference is the **lifecycle**: instead of PMG starting an ephemeral proxy
around a single subprocess, the proxy is started once, advertises itself to the
other `pmg proxy` commands, and serves many package manager processes until it
is stopped.
## Default proxy mode vs. persistent proxy server
PMG's default proxy mode (see [proxy.md](./proxy.md)) wraps a single command.
`pmg npm install` starts an ephemeral proxy, runs `npm` as a child with proxy
env vars injected, then tears the proxy down. The persistent server decouples
these steps.
| | Default proxy mode | Persistent proxy server |
| --- | --- | --- |
| Invocation | `pmg npm install` (wrapped) | bare `npm install` (no wrapper) |
| Proxy lifetime | One subprocess | Until `pmg proxy stop` |
| Who runs the PM | PMG (as a child) | The user / CI directly |
| Ecosystems served | The one being run | All supported (npm + PyPI) |
| Confirmation on malware | Interactive prompt (TTY) | Auto-block (non-interactive) |
| Reporting | At subprocess exit | At `pmg proxy stop` |
| Target | Local dev | CI/CD pipelines |
## How it works
The diagram below shows the order of events in a CI job. The `pmg proxy`
commands run in separate workflow steps and coordinate through the running
daemon.
```mermaid
sequenceDiagram
participant CI as CI Job
participant Proxy as Proxy Daemon
participant PM as Package Manager
participant Cloud as SafeDep Cloud
CI->>Proxy: pmg proxy start --daemon
Proxy-->>CI: ready (addr, ca path)
CI->>CI: pmg proxy env (set HTTP_PROXY + CA vars)
PM->>Proxy: package download (via HTTP_PROXY)
Proxy->>Proxy: analyze package
Proxy-->>PM: allow, or 403 block + record event
Proxy->>Cloud: periodic sync of events (while serving)
CI->>Proxy: pmg proxy stop --fail-on-violation
Proxy->>Cloud: final flush of remaining events
Proxy-->>CI: exit non-zero if anything was blocked
```
## Usage
The persistent server targets non-interactive CI/CD. For local development use
the default proxy mode (`pmg npm install`), which keeps the interactive malware
confirmation prompt. The persistent server auto-blocks without prompting.
GitHub Actions (raw commands):
```yaml
- run: pmg proxy start --daemon
- run: pmg proxy env >> "$GITHUB_ENV"
- run: npm ci
- run: pmg proxy stop --fail-on-violation
if: always()
```
GitHub Actions (via the [safedep/pmg action](../action.yml) `server-mode`):
```yaml
- uses: safedep/pmg@v1
with:
server-mode: true
api-key: ${{ secrets.SAFEDEP_API_KEY }}
tenant-id: ${{ secrets.SAFEDEP_TENANT_ID }}
- run: npm ci # intercepted automatically
- name: Enforce PMG policy
if: always()
run: pmg proxy stop --fail-on-violation
```
In `server-mode`, the action starts the daemon and injects env vars instead of
installing shims. Because composite actions cannot run an automatic cleanup
step, the final `pmg proxy stop --fail-on-violation` step is required. It stops
the proxy (the daemon flushes events to the cloud during shutdown) and fails the
job on a block.
## Commands
```bash
pmg proxy start # start the proxy (foreground, or detached with --daemon)
pmg proxy stop # stop the proxy and report the outcome
pmg proxy env # print env vars that route package managers through it
pmg proxy status # report whether a proxy is running
```
Run `pmg proxy <command> --help` for flags. `--daemon` is **Unix only**: on
Windows it returns a clear "not supported" error, and the foreground
`pmg proxy start` still works. To run multiple independent proxies on one host,
give each a distinct `--state` path and `--port`.
## Bind address
The proxy binds `127.0.0.1` on a random port by default, reachable only from the
host (the right choice for CI and local use). Override with `--host`/`--port`, or
the `proxy.server.listen_host`/`listen_port` config (flags take precedence).
Bind a non-loopback address (e.g. `--host 0.0.0.0`) **only** for a deliberately
hosted deployment: it exposes the MITM proxy to the network, and every client
routed through it has its HTTPS intercepted and must trust the PMG CA.
## Certificate trust
The proxy performs TLS MITM, so clients must trust its CA. Trust is delivered
through **environment variables, not the OS trust store**. `pmg proxy env`
always emits the cert-path variables pointing at the proxy's CA bundle:
`NODE_EXTRA_CA_CERTS`, `SSL_CERT_FILE`, `REQUESTS_CA_BUNDLE`, `PIP_CERT`,
`YARN_HTTPS_CA_FILE_PATH`. Package managers pick these up from the job
environment and trust the proxy's CA, with no OS trust-store install required.
This is deliberate: whether a tool consults the OS trust store varies by tool,
version, and config (npm/Node ignore it by default; modern pip can read it;
`requests`/`certifi` ship their own bundle). The cert-path vars work across all
of them, and are harmlessly ignored by tools that do read the OS store.
As a result `pmg setup cert install` is **not** needed for the persistent proxy.
If a persisted CA from `pmg setup cert install` exists the proxy reuses it,
otherwise it generates an ephemeral one. Either way `pmg proxy env` carries the
trust. OS trust-store install (`pmg setup cert install --system`) is
intentionally not used: it needs root (breaking container and locked-down
runners), persistently installs a MITM-capable CA into the machine trust store,
and still does not remove the need for the env vars.
Loopback addresses are always excluded from proxying via `NO_PROXY`
(`localhost,127.0.0.1,::1`).
## Cloud event sync
When SafeDep Cloud is enabled, malware-block events must reach the cloud even on
ephemeral CI runners that are destroyed immediately after the job. The daemon
owns delivery: it records each blocked package to a durable local event log as
it happens, syncs pending events to SafeDep Cloud periodically while serving, and
flushes whatever remains on shutdown.
`pmg proxy stop` reports the recorded result (`Synced N event(s) to SafeDep
Cloud`, or a `Cloud sync failed` line). A flush failure is surfaced but does not
mask the fail-on-violation exit code.
## Fail on violation
By default `pmg proxy stop` just stops the proxy and exits `0`. Failing the CI
job on a policy violation is opt-in via `--fail-on-violation`.
- It exits non-zero when any package was blocked.
- It **fails closed**. If the daemon shut down without writing a verifiable
final state (e.g. it crashed), `--fail-on-violation` also fails, because a
security gate must not pass on an unverifiable run.
The package manager's own non-zero exit (from the `403` on a blocked download)
is a separate signal. `--fail-on-violation` gives an authoritative gate from the
proxy regardless of how the package manager reported the failure.
## Limitations
- **Unix-only daemon.** `--daemon` is not supported on Windows (foreground mode
works).
- **Non-interactive only.** There is no interactive confirmation; flagged
packages are always auto-blocked. This is intentional for CI.
- **Single proxy per state file.** Starting a second proxy that points at the
same state file is refused while one is running.
- **System-level trust enforcement is out of scope.** The server relies on env
var propagation. Enforcing interception for `sudo`-scrubbed environments (e.g.
via `iptables`) and system-wide install (`pmg setup install --system`) are
tracked separately.
## References
- [proxy.md](./proxy.md) is the underlying generic MITM proxy server
- [config.md](./config.md) is the configuration schema (cloud, proxy, cache dir)
- [action.yml](../action.yml) is the PMG GitHub Action (`server-mode`)
+4
View File
@@ -68,3 +68,7 @@ Legacy variables `PMG_PROXY_MODE` and `PMG_PROXY_INSTALL_ONLY` (for the old flat
| `pip` | ✅ |
| `uv` | ✅ |
| `poetry` | ✅ |
## References
- [Persistent Proxy Mode](./persistent-proxy.md)
+5
View File
@@ -24,6 +24,11 @@ const (
CertTrustStore = "CertTrustStore"
UnsupportedPlatform = "UnsupportedPlatform"
// Proxy error codes. ProxyPolicyViolation is returned when the proxy blocked
// one or more packages by policy (malware, dependency cooldown, or a denied
// suspicious package) and the run was gated with --fail-on-violation.
ProxyPolicyViolation = "ProxyPolicyViolation"
// Unknown mirrors the default code that dry/usefulerror returns for errors
// created without an explicit code, so unset and explicitly-unknown errors
// classify identically (e.g. the bug-report hint in ui.ErrorExit).
+14 -1
View File
@@ -22,6 +22,19 @@ func MarkBackgroundSyncChild() {
isBackgroundSyncChild = true
}
// backgroundSyncSuppressed lets a command opt out of the detached auto-sync
// spawn for its process. The proxy daemon uses this: it delivers events itself
// (periodic sync + shutdown flush), so the detached child would be redundant
// and, from `pmg proxy stop`, would route cloud traffic through the now-stopped
// proxy.
var backgroundSyncSuppressed bool
// SuppressBackgroundSync disables MaybeSpawnBackgroundSync for the current
// process.
func SuppressBackgroundSync() {
backgroundSyncSuppressed = true
}
// detachedSpawner forks a detached child running `name` with `args`. Pulled
// behind a package var so tests can intercept without actually forking the
// test binary into the background.
@@ -41,7 +54,7 @@ func MaybeSpawnBackgroundSync(cfg *config.RuntimeConfig) {
if cfg == nil {
return
}
if isBackgroundSyncChild {
if isBackgroundSyncChild || backgroundSyncSuppressed {
return
}
if !cfg.Config.Cloud.Enabled || !cfg.Config.Cloud.AutoSync.Enabled {
+55
View File
@@ -4,6 +4,7 @@ import (
"context"
"errors"
"fmt"
"time"
"github.com/safedep/dry/cloud"
"github.com/safedep/dry/cloud/endpointsync"
@@ -13,6 +14,60 @@ import (
appVersion "github.com/safedep/pmg/internal/version"
)
// ErrSyncInProgress is returned by DrainToCloud when another process already
// holds the cloud sync lock.
var ErrSyncInProgress = errors.New("another cloud sync is already in progress")
// DrainToCloud acquires the cross-process sync lock and drains pending audit
// events from the WAL to SafeDep Cloud, returning the number synced. It records
// the attempt (so a failing endpoint does not make every run retry). Callers
// are responsible for gating on cloud.enabled and for surfacing errors;
// ErrSyncInProgress is returned when the lock is held elsewhere.
//
// lockTimeout bounds acquiring the shared lock; syncTimeout bounds the drain.
func DrainToCloud(ctx context.Context, cfg *config.RuntimeConfig, lockTimeout, syncTimeout time.Duration) (int, error) {
lock := NewSyncLock(cfg.CloudSyncLockPath())
lockCtx, lockCancel := context.WithTimeout(ctx, lockTimeout)
defer lockCancel()
locked, err := lock.TryLockContext(lockCtx, 250*time.Millisecond)
if err != nil {
return 0, fmt.Errorf("acquire cloud sync lock: %w", err)
}
if !locked {
return 0, ErrSyncInProgress
}
defer func() {
if uerr := lock.Unlock(); uerr != nil {
log.Warnf("failed to release cloud sync lock: %v", uerr)
}
}()
bundle, err := NewSyncClientBundle(cfg)
if err != nil {
return 0, fmt.Errorf("init cloud sync client: %w", err)
}
defer func() {
if cerr := bundle.Close(); cerr != nil {
log.Warnf("failed to close cloud sync client: %v", cerr)
}
}()
syncCtx, cancel := context.WithTimeout(ctx, syncTimeout)
defer cancel()
synced, syncErr := bundle.Sync(syncCtx)
// Record the attempt on every outcome so a stuck endpoint does not make the
// background auto-sync refire on the next invocation.
if werr := WriteLastSyncAttempt(cfg.CloudSyncLastRunPath()); werr != nil {
log.Warnf("failed to update cloud sync lastrun: %v", werr)
}
return synced, syncErr
}
// SyncClientBundle holds a SyncClient and its underlying cloud client.
// Callers must call Close() when done.
type SyncClientBundle struct {
+34
View File
@@ -0,0 +1,34 @@
package flows
import (
"context"
"github.com/safedep/dry/localdb"
"github.com/safedep/dry/log"
"github.com/safedep/pmg/analyzer"
"github.com/safedep/pmg/analyzer/malysiscache"
"github.com/safedep/pmg/config"
)
// BuildMalysisAnalyzer constructs the malysis analyzer with its optional
// analyzer-specific persistent cache. The caller owns the shared localdb manager
// lifecycle. Cache failures degrade to an uncached analyzer and never abort.
func BuildMalysisAnalyzer(ctx context.Context, cfg *config.RuntimeConfig, db localdb.Manager) (analyzer.PackageVersionAnalyzer, error) {
return analyzer.NewMalysisAnalyzer(analyzer.MalysisQueryAnalyzerConfig{
Cache: buildMalysisCache(ctx, db, cfg.Config.AnalysisCache.Malysis),
})
}
func buildMalysisCache(ctx context.Context, db localdb.Manager, cacheCfg config.MalysisCacheConfig) analyzer.MalysisCache {
if db == nil || !cacheCfg.Enabled || cacheCfg.TTL <= 0 {
return nil
}
store, err := db.Store(ctx, malysiscache.Descriptor())
if err != nil {
log.Warnf("analysis cache unavailable, continuing without it: %v", err)
return nil
}
return malysiscache.New(store, cacheCfg)
}
+50
View File
@@ -0,0 +1,50 @@
package flows
import (
"context"
"errors"
"testing"
"time"
"github.com/safedep/dry/localdb"
"github.com/safedep/pmg/config"
"github.com/stretchr/testify/assert"
)
type fakeLocalDBManager struct {
storeCalls int
storeErr error
}
func (f *fakeLocalDBManager) Store(context.Context, localdb.Descriptor) (*localdb.Store, error) {
f.storeCalls++
return nil, f.storeErr
}
func (f *fakeLocalDBManager) Close() error {
return nil
}
func TestBuildMalysisCacheDisabledDoesNotOpenLocalDB(t *testing.T) {
db := &fakeLocalDBManager{}
cache := buildMalysisCache(context.Background(), db, config.MalysisCacheConfig{
Enabled: false,
TTL: time.Hour,
})
assert.Nil(t, cache)
assert.Equal(t, 0, db.storeCalls)
}
func TestBuildMalysisCacheStoreErrorDegradesToNil(t *testing.T) {
db := &fakeLocalDBManager{storeErr: errors.New("db unavailable")}
cache := buildMalysisCache(context.Background(), db, config.MalysisCacheConfig{
Enabled: true,
TTL: time.Hour,
})
assert.Nil(t, cache)
assert.Equal(t, 1, db.storeCalls)
}
+73
View File
@@ -0,0 +1,73 @@
package flows
import (
"fmt"
"os"
"path/filepath"
"time"
"github.com/safedep/dry/log"
"github.com/safedep/pmg/proxy/certmanager"
"github.com/safedep/pmg/truststore"
)
// SetupCACertificate loads the persisted CA from configDir or generates an
// ephemeral one, merges it with the system CA bundle, and writes the result
// to outputPath. Returns the certificate, whether it was ephemeral, and any
// error. The caller is responsible for cleaning up outputPath when ephemeral.
func SetupCACertificate(configDir, outputPath string) (*certmanager.Certificate, bool, error) {
caCert, persisted := loadPersistedCA(configDir)
ephemeral := !persisted
if persisted {
log.Debugf("Using persisted CA certificate from %s", configDir)
warnIfCANotTrusted()
} else {
log.Debugf("Generating ephemeral CA certificate for proxy MITM")
generated, err := certmanager.GenerateCA(certmanager.DefaultCertManagerConfig())
if err != nil {
return nil, false, fmt.Errorf("generate CA certificate: %w", err)
}
caCert = generated
}
merged := certmanager.MergeWithSystemCA(caCert.Certificate)
if err := os.MkdirAll(filepath.Dir(outputPath), 0o700); err != nil {
return nil, false, fmt.Errorf("create CA certificate directory: %w", err)
}
if err := os.WriteFile(outputPath, merged, 0o600); err != nil {
return nil, false, fmt.Errorf("write CA certificate to %s: %w", outputPath, err)
}
log.Debugf("CA certificate written to %s", outputPath)
return caCert, ephemeral, nil
}
func loadPersistedCA(dir string) (*certmanager.Certificate, bool) {
caCert, err := certmanager.LoadCA(dir)
if err != nil {
if !os.IsNotExist(err) {
log.Warnf("Failed to load persisted CA, using ephemeral: %v", err)
}
return nil, false
}
if caCert.IsExpired(time.Hour) {
log.Warnf("Persisted CA is expired; using ephemeral. Re-run `pmg setup cert install`")
return nil, false
}
return caCert, true
}
func warnIfCANotTrusted() {
user, system, err := truststore.Status(certmanager.CACommonName)
if err != nil {
log.Debugf("Could not determine CA trust status: %v", err)
return
}
if !user && !system {
log.Warnf("Persisted CA is not trusted in the OS store; native tools may reject TLS. Run `pmg setup cert install`.")
}
}
+13 -126
View File
@@ -2,26 +2,22 @@ package flows
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"time"
"github.com/safedep/dry/localdb"
"github.com/safedep/dry/log"
"github.com/safedep/pmg/analyzer"
"github.com/safedep/pmg/analyzer/malysiscache"
"github.com/safedep/pmg/config"
"github.com/safedep/pmg/guard"
"github.com/safedep/pmg/internal/audit"
"github.com/safedep/pmg/internal/localstore"
"github.com/safedep/pmg/internal/runner"
"github.com/safedep/pmg/internal/ui"
"github.com/safedep/pmg/packagemanager"
"github.com/safedep/pmg/proxy"
"github.com/safedep/pmg/proxy/certmanager"
"github.com/safedep/pmg/proxy/interceptors"
"github.com/safedep/pmg/truststore"
)
type proxyFlow struct {
@@ -137,31 +133,16 @@ func (f *proxyFlow) Run(ctx context.Context, args []string, parsedCmd *packagema
return fmt.Errorf("failed to create certificate manager: %w", err)
}
// Optional persistent analysis cache. Disposable: any failure degrades to
// running uncached, never blocks the install.
var malysisCache analyzer.MalysisCache
cacheCfg := cfg.Config.AnalysisCache.Malysis
if cacheCfg.Enabled && cacheCfg.TTL > 0 {
mgr := localdb.New(localdb.Config{
Dir: cfg.LocalDBDir(),
FileName: cfg.LocalDBFileName(),
})
defer func() {
if cerr := mgr.Close(); cerr != nil {
log.Warnf("failed to close localdb: %v", cerr)
}
}()
store, serr := mgr.Store(ctx, malysiscache.Descriptor())
if serr != nil {
log.Warnf("analysis cache unavailable, continuing without it: %v", serr)
} else {
malysisCache = malysiscache.New(store, cacheCfg)
localDB := localstore.NewManager(cfg)
defer func() {
if cerr := localDB.Close(); cerr != nil {
log.Warnf("failed to close localdb: %v", cerr)
}
}
}()
// Create analyzer
malysisAnalyzer, err := f.createAnalyzer(malysisCache)
// Analyzer with an optional persistent cache. Cache failures degrade to
// running uncached and never block the install.
malysisAnalyzer, err := BuildMalysisAnalyzer(ctx, cfg, localDB)
if err != nil {
return fmt.Errorf("failed to create analyzer: %w", err)
}
@@ -231,7 +212,7 @@ func (f *proxyFlow) Run(ctx context.Context, args []string, parsedCmd *packagema
PackageManagerName: f.pm.Name(),
DryRun: cfg.DryRun,
Mode: runner.ExecutionModeAuto,
EnvOverrides: f.setupEnvForProxy(proxyAddr, caCertPath),
EnvOverrides: packagemanager.EnvVarForProxy(proxyAddr, caCertPath),
DirectEnvOverrides: ciEnvOverride(),
BeforeDirectRun: func() error {
log.Debugf("Executing proxy for non interactive TTY")
@@ -329,69 +310,11 @@ func handleExecutionResultError(err error) error {
return fmt.Errorf("failed to execute command: %w", err)
}
// setupCACertificate prefers the persisted CA (created by `pmg setup cert install`)
// so the proxy signs leaves with the same CA that is in the OS trust store. When no
// persisted CA exists it falls back to an ephemeral per-run CA, preserving original
// behavior. The temp file always carries the pure CA merged with the system bundle so
// env-var trust injection works on every platform.
func (f *proxyFlow) setupCACertificate() (*certmanager.Certificate, string, error) {
dir := config.Get().ConfigDir()
caCert, persisted := loadPersistedCA(dir)
if persisted {
log.Debugf("Using persisted CA certificate from %s", dir)
warnIfCANotTrusted()
} else {
log.Debugf("Generating ephemeral CA certificate for proxy MITM")
generated, err := certmanager.GenerateCA(certmanager.DefaultCertManagerConfig())
if err != nil {
return nil, "", fmt.Errorf("failed to generate CA certificate: %w", err)
}
caCert = generated
}
mergedPEM := certmanager.MergeWithSystemCA(caCert.Certificate)
tempDir := os.TempDir()
caCertPath := filepath.Join(tempDir, fmt.Sprintf("pmg-ca-cert-%d.pem", os.Getpid()))
if err := os.WriteFile(caCertPath, mergedPEM, 0o600); err != nil {
return nil, "", fmt.Errorf("failed to write CA certificate to %s: %w", caCertPath, err)
}
log.Debugf("CA certificate written to %s", caCertPath)
return caCert, caCertPath, nil
}
// loadPersistedCA returns the on-disk CA when present and not expired.
func loadPersistedCA(dir string) (*certmanager.Certificate, bool) {
caCert, err := certmanager.LoadCA(dir)
if err != nil {
if !errors.Is(err, os.ErrNotExist) {
log.Warnf("Failed to load persisted CA, using ephemeral: %v", err)
}
return nil, false
}
if caCert.IsExpired(time.Hour) {
log.Warnf("Persisted CA is expired; using ephemeral. Re-run `pmg setup cert install`")
return nil, false
}
return caCert, true
}
// warnIfCANotTrusted logs a hint when the persisted CA is not in any OS store,
// which matters for native tools (e.g. Go on macOS/Windows) that ignore the
// injected env vars. Best-effort; never blocks the run.
func warnIfCANotTrusted() {
user, system, err := truststore.Status(certmanager.CACommonName)
if err != nil {
log.Debugf("Could not determine CA trust status: %v", err)
return
}
if !user && !system {
log.Warnf("Persisted CA is not trusted in the OS store; native tools may reject TLS. Run `pmg setup cert install`.")
}
outputPath := certmanager.EphemeralProxyCABundlePath()
cert, _, err := SetupCACertificate(dir, outputPath)
return cert, outputPath, err
}
// createCertificateManager creates a certificate manager with the given CA certificate
@@ -405,12 +328,6 @@ func (f *proxyFlow) createCertificateManager(caCert *certmanager.Certificate) (c
return certMgr, nil
}
// createAnalyzer creates the malysis query analyzer
func (f *proxyFlow) createAnalyzer(cache analyzer.MalysisCache) (analyzer.PackageVersionAnalyzer, error) {
log.Debugf("Creating malysis query analyzer")
return analyzer.NewMalysisAnalyzer(analyzer.MalysisQueryAnalyzerConfig{Cache: cache})
}
// createAndStartProxyServer creates and starts the proxy server with the given interceptor
func (f *proxyFlow) createAndStartProxyServer(
certMgr certmanager.CertificateManager,
@@ -446,33 +363,3 @@ func ciEnvOverride() []string {
}
return []string{"CI=true"}
}
func (f *proxyFlow) setupEnvForProxy(proxyAddr, caCertPath string) []string {
proxyURL := fmt.Sprintf("http://%s", proxyAddr)
// IPv6 loopback uses the bare ::1: the bracketed [::1] is URL syntax that
// crashes Python's urllib/httpx (#339). Trade-off: Node's NODE_USE_ENV_PROXY
// (undici) only bypasses the bracketed form, so a literal http://[::1] from
// Node still gets proxied. localhost/127.0.0.1 cover the common cases; the
// IPv6 literal is a rare edge we accept since NO_PROXY can't be set per-client.
noProxyList := "localhost,127.0.0.1,::1"
return []string{
"NODE_USE_ENV_PROXY=1",
fmt.Sprintf("HTTP_PROXY=%s", proxyURL),
fmt.Sprintf("HTTPS_PROXY=%s", proxyURL),
fmt.Sprintf("NO_PROXY=%s", noProxyList),
fmt.Sprintf("NODE_EXTRA_CA_CERTS=%s", caCertPath),
fmt.Sprintf("YARN_HTTP_PROXY=%s", proxyURL),
fmt.Sprintf("YARN_HTTPS_PROXY=%s", proxyURL),
fmt.Sprintf("YARN_HTTPS_CA_FILE_PATH=%s", caCertPath),
fmt.Sprintf("http_proxy=%s", proxyURL),
fmt.Sprintf("https_proxy=%s", proxyURL),
fmt.Sprintf("no_proxy=%s", noProxyList),
fmt.Sprintf("SSL_CERT_FILE=%s", caCertPath),
fmt.Sprintf("REQUESTS_CA_BUNDLE=%s", caCertPath),
fmt.Sprintf("PIP_CERT=%s", caCertPath),
fmt.Sprintf("PIP_PROXY=%s", proxyURL),
"PIP_RETRIES=0",
}
}
+14
View File
@@ -2,6 +2,7 @@ package flows
import (
"os"
"path/filepath"
"testing"
"github.com/safedep/pmg/proxy/certmanager"
@@ -41,3 +42,16 @@ func TestSetupCACertificateWritesMergedTempFile(t *testing.T) {
require.NoError(t, err)
assert.Equal(t, caCert.Certificate, merged[:len(caCert.Certificate)])
}
func TestSetupCACertificateCreatesMissingOutputDir(t *testing.T) {
root := t.TempDir()
configDir := filepath.Join(root, "config")
outputPath := filepath.Join(configDir, "proxy-ca.pem")
caCert, ephemeral, err := SetupCACertificate(configDir, outputPath)
require.NoError(t, err)
require.NotNil(t, caCert)
assert.True(t, ephemeral)
assert.FileExists(t, outputPath)
}
-49
View File
@@ -2,61 +2,12 @@ package flows
import (
"os"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func envToMap(env []string) map[string]string {
m := make(map[string]string, len(env))
for _, e := range env {
k, v, ok := strings.Cut(e, "=")
if ok {
m[k] = v
}
}
return m
}
// TestSetupEnvForProxyConfiguresYarn proves the root cause of #319: yarn Berry
// (yarn 2+) ignores the standard HTTP_PROXY/HTTPS_PROXY env vars and only honors
// its own config, which can be set via YARN_* env overrides. Without these,
// yarn bypasses the MITM proxy entirely and no packages are analyzed.
func TestSetupEnvForProxyConfiguresYarn(t *testing.T) {
f := &proxyFlow{}
const proxyAddr = "127.0.0.1:54321"
const caCertPath = "/tmp/pmg-ca-cert.pem"
env := envToMap(f.setupEnvForProxy(proxyAddr, caCertPath))
proxyURL := "http://" + proxyAddr
assert.Equal(t, proxyURL, env["YARN_HTTP_PROXY"],
"yarn ignores HTTP_PROXY; YARN_HTTP_PROXY is required to route yarn through the proxy")
assert.Equal(t, proxyURL, env["YARN_HTTPS_PROXY"],
"yarn ignores HTTPS_PROXY; YARN_HTTPS_PROXY is required to route yarn through the proxy")
assert.Equal(t, caCertPath, env["YARN_HTTPS_CA_FILE_PATH"],
"yarn ignores NODE_EXTRA_CA_CERTS; YARN_HTTPS_CA_FILE_PATH is required to trust the MITM CA")
}
// TestSetupEnvForProxyNoProxyIPv6 proves the fix for #339: the IPv6 loopback in
// NO_PROXY must be bare (::1), not bracketed ([::1]). Brackets are URL syntax,
// not NO_PROXY syntax, and Python's urllib/httpx crashes parsing them with
// "Invalid port: ':1]'".
func TestSetupEnvForProxyNoProxyIPv6(t *testing.T) {
f := &proxyFlow{}
env := envToMap(f.setupEnvForProxy("127.0.0.1:54321", "/tmp/pmg-ca-cert.pem"))
for _, key := range []string{"NO_PROXY", "no_proxy"} {
assert.Equal(t, "localhost,127.0.0.1,::1", env[key],
"%s must use bare ::1; bracketed [::1] is invalid NO_PROXY syntax and crashes httpx", key)
assert.NotContains(t, env[key], "[::1]",
"%s must not contain bracketed IPv6 loopback", key)
}
}
// TestCIEnvOverride proves the fix for #335: pmg forces CI=true for
// non-interactive runs but must not clobber a CI value the user set
// explicitly (e.g. CI=false on a build server).
+13
View File
@@ -0,0 +1,13 @@
package localstore
import (
"github.com/safedep/dry/localdb"
"github.com/safedep/pmg/config"
)
func NewManager(cfg *config.RuntimeConfig) localdb.Manager {
return localdb.New(localdb.Config{
Dir: cfg.LocalDBDir(),
FileName: cfg.LocalDBFileName(),
})
}
+49
View File
@@ -0,0 +1,49 @@
//go:build !windows
package proxyserver
import (
"fmt"
"os"
"os/exec"
"syscall"
"time"
)
// Daemonize re-execs the current binary with args (which must run the proxy in
// foreground mode), detached into its own session (Setsid), with child stdio
// redirected to cfg.LogPath. It waits up to cfg.ReadyTimeout for the child to
// write the state file and returns the running state. The caller owns the log
// path (its parent directory must exist); Daemonize fails if it cannot be
// opened.
func Daemonize(cfg ProxyDaemonConfig, statePath, exe string, args []string) (State, error) {
logFile, err := os.OpenFile(cfg.LogPath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o600)
if err != nil {
return State{}, fmt.Errorf("open daemon log %s: %w", cfg.LogPath, err)
}
defer func() { _ = logFile.Close() }()
cmd := exec.Command(exe, args...)
cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
cmd.Stdin = nil
cmd.Stdout = logFile
cmd.Stderr = logFile
if err := cmd.Start(); err != nil {
return State{}, fmt.Errorf("start daemon: %w", err)
}
if err := cmd.Process.Release(); err != nil {
return State{}, fmt.Errorf("release daemon process: %w", err)
}
deadline := time.Now().Add(cfg.ReadyTimeout)
for time.Now().Before(deadline) {
if state, rerr := readState(statePath); rerr == nil && state.IsRunning() {
return state, nil
}
time.Sleep(200 * time.Millisecond)
}
return State{}, fmt.Errorf("daemon did not become ready within %s; see %s", cfg.ReadyTimeout, cfg.LogPath)
}
+15
View File
@@ -0,0 +1,15 @@
//go:build windows
package proxyserver
import (
"github.com/safedep/dry/usefulerror"
"github.com/safedep/pmg/errcodes"
)
func Daemonize(_ ProxyDaemonConfig, _, _ string, _ []string) (State, error) {
return State{}, usefulerror.NewUsefulError().
WithCode(errcodes.UnsupportedPlatform).
WithMsg("pmg proxy start --daemon is not supported on Windows").
WithHelp("Run 'pmg proxy start' in the foreground instead")
}
+22
View File
@@ -0,0 +1,22 @@
// Package proxyserver implements the persistent proxy flow: running PMG as a
// long-lived MITM proxy server that many package-manager processes route through
// via environment variables, for non-interactive CI/CD use.
//
// It is a flow, not a proxy implementation. The actual MITM proxy lives in the
// `proxy` package; proxyserver wires that proxy together with the analyzer,
// interceptors, audit pipeline, and cloud sync, and manages the daemon lifecycle
// (start/stop/env/status) coordinated through an on-disk state file.
//
// # Boundary with internal/flows
//
// internal/flows owns the per-command flow, where PMG wraps a single package
// manager invocation (spawns it as a child, injects env, prompts on confirmation,
// reports at exit). proxyserver owns the persistent flow, where the proxy
// outlives any single invocation and is driven by separate `pmg proxy` commands
// across processes. The two share lower-level building blocks (CA setup in
// flows.SetupCACertificate, proxy env vars in packagemanager.EnvVarForProxy,
// cloud sync in audit.DrainToCloud) but differ in lifecycle and process model,
// so they are kept as separate flows rather than one. Some assembly (analyzer +
// cache wiring) is intentionally duplicated for the MVP rather than prematurely
// unified.
package proxyserver
+18
View File
@@ -0,0 +1,18 @@
package proxyserver
import (
"fmt"
"github.com/safedep/pmg/packagemanager"
)
// EnvVars returns the proxy environment variables (KEY=VALUE lines) for the
// running proxy described by the state file at statePath.
func EnvVars(statePath string) ([]string, error) {
state, err := readState(statePath)
if err != nil {
return nil, fmt.Errorf("proxy not running, start with 'pmg proxy start' first: %w", err)
}
return packagemanager.EnvVarForProxy(state.Addr, state.CACertPath), nil
}
+276
View File
@@ -0,0 +1,276 @@
package proxyserver
import (
"context"
"errors"
"fmt"
"net"
"os"
"os/signal"
"strconv"
"syscall"
"time"
"github.com/safedep/dry/log"
"github.com/safedep/pmg/config"
"github.com/safedep/pmg/internal/audit"
"github.com/safedep/pmg/internal/flows"
"github.com/safedep/pmg/internal/localstore"
pmgproxy "github.com/safedep/pmg/proxy"
"github.com/safedep/pmg/proxy/certmanager"
"github.com/safedep/pmg/proxy/interceptors"
)
const (
serverStopTimeout = 5 * time.Second
// Periodic cloud sync runs while the daemon is alive so most audit events are
// delivered during the run and the shutdown flush stays small. A tick that
// cannot get the sync lock quickly is skipped (the next tick retries).
cloudSyncInterval = 15 * time.Second
cloudSyncTickLockWait = 5 * time.Second
cloudSyncTickTimeout = 30 * time.Second
// Final flush at shutdown, when the daemon drains whatever the ticker left.
cloudFlushLockWait = 30 * time.Second
cloudFlushTimeout = 2 * time.Minute
// daemonShutdownBudget is the worst-case time the daemon needs to shut down:
// drain in-flight requests, wait for an in-flight periodic tick to finish,
// then the final flush. `pmg proxy stop` waits at least this long for the
// daemon to exit; see stopWaitTimeout.
daemonShutdownBudget = serverStopTimeout +
cloudSyncTickLockWait + cloudSyncTickTimeout +
cloudFlushLockWait + cloudFlushTimeout
)
// DefaultDaemonReadyTimeout is how long the parent waits for the daemon to
// become ready before giving up, when ProxyDaemonConfig.ReadyTimeout is unset.
const DefaultDaemonReadyTimeout = 10 * time.Second
// ProxyDaemonConfig carries the daemon-launch parameters the caller decides, so
// daemonization stays free of config and path-policy concerns.
type ProxyDaemonConfig struct {
// LogPath is the file the detached daemon's stdout/stderr is redirected to.
// The caller owns this path (its parent directory must exist).
LogPath string
// ReadyTimeout bounds how long to wait for the daemon to write its state
// file and become live.
ReadyTimeout time.Duration
}
// Run starts the persistent proxy server in the foreground and blocks until it
// receives SIGINT/SIGTERM. It writes the state file on startup, auto-blocks
// suspicious packages, and records the final blocked count on shutdown.
func Run(ctx context.Context, cfg *config.RuntimeConfig, statePath, host string, port int) error {
if existing, err := readState(statePath); err == nil && existing.IsRunning() {
return fmt.Errorf("proxy already running (pid %d, addr %s) — run 'pmg proxy stop' first", existing.PID, existing.Addr)
}
caCertPath := certmanager.ProxyCABundlePath(cfg.ConfigDir())
caCert, _, err := flows.SetupCACertificate(cfg.ConfigDir(), caCertPath)
if err != nil {
return fmt.Errorf("setup CA certificate: %w", err)
}
certMgr, err := certmanager.NewCertificateManagerWithCA(caCert, certmanager.DefaultCertManagerConfig())
if err != nil {
return fmt.Errorf("create certificate manager: %w", err)
}
localDB := localstore.NewManager(cfg)
defer func() {
if cerr := localDB.Close(); cerr != nil {
log.Warnf("failed to close localdb: %v", cerr)
}
}()
malysisAnalyzer, err := flows.BuildMalysisAnalyzer(ctx, cfg, localDB)
if err != nil {
return fmt.Errorf("create analyzer: %w", err)
}
cache := interceptors.NewInMemoryAnalysisCache()
stats := interceptors.NewAnalysisStatsCollector()
confirmationChan := make(chan *interceptors.ConfirmationRequest, 100)
go autoBlockConfirmations(confirmationChan)
factory := interceptors.NewInterceptorFactory(
malysisAnalyzer, cache, stats, confirmationChan, interceptors.InterceptorContext{},
)
var interceptorList []pmgproxy.Interceptor
for _, eco := range interceptors.SupportedEcosystems() {
i, ferr := factory.CreateInterceptor(eco)
if ferr != nil {
return fmt.Errorf("create interceptor for %s: %w", eco.String(), ferr)
}
interceptorList = append(interceptorList, i)
}
interceptorList = append(interceptorList, interceptors.NewAuditLoggerInterceptor())
proxyConfig := pmgproxy.DefaultProxyConfig()
proxyConfig.ListenAddr = listenAddr(host, port)
proxyConfig.CertManager = certMgr
proxyConfig.Interceptors = interceptorList
server, err := pmgproxy.NewProxyServer(proxyConfig)
if err != nil {
return fmt.Errorf("create proxy server: %w", err)
}
if err := server.Start(); err != nil {
return fmt.Errorf("start proxy server: %w", err)
}
state := State{
PID: os.Getpid(),
Addr: server.Address(),
CACertPath: caCertPath,
}
if err := writeState(statePath, state); err != nil {
stopCtx, cancel := context.WithTimeout(context.Background(), serverStopTimeout)
defer cancel()
if serr := server.Stop(stopCtx); serr != nil {
log.Warnf("failed to stop proxy after state write failure: %v", serr)
}
return fmt.Errorf("write proxy state: %w", err)
}
log.Infof("PMG persistent proxy running on %s (pid %d)", state.Addr, state.PID)
if _, err := fmt.Fprintf(os.Stderr, "PMG proxy running on %s\nRun: export $(pmg proxy env | xargs) # or: pmg proxy env >> \"$GITHUB_ENV\"\n", state.Addr); err != nil {
log.Warnf("failed to write startup message: %v", err)
}
// Periodically flush events to the cloud while serving so the shutdown flush
// stays small. See startCloudSyncLoop for the stop-function contract.
stopSyncLoop := startCloudSyncLoop(cfg)
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, os.Interrupt, syscall.SIGTERM)
<-sigCh
// Drain in-flight requests before closing the confirmation channel, so no
// request handler can send on a closed channel (panic) during shutdown.
stopCtx, cancel := context.WithTimeout(context.Background(), serverStopTimeout)
defer cancel()
stopErr := server.Stop(stopCtx)
close(confirmationChan)
// Count is read after drain so a package analyzed at shutdown is not missed.
// Persist it BEFORE the (possibly slow) cloud flush so the blocked count
// survives even if the flush hangs or the daemon is killed mid-flush, which
// keeps `stop --fail-on-violation` correct in those cases.
state.BlockedCount = stats.GetStats().BlockedCount
if werr := writeState(statePath, state); werr != nil {
log.Warnf("failed to write final proxy state: %v", werr)
}
// Halt the periodic sync (waits for any in-flight drain) before the final
// flush, so the two never hold the sync lock at once.
periodicSynced := stopSyncLoop()
if cs := cloudFlush(cfg, periodicSynced); cs != nil {
state.CloudSync = cs
if werr := writeState(statePath, state); werr != nil {
log.Warnf("failed to write final proxy state: %v", werr)
}
}
return stopErr
}
// cloudFlush drains whatever the periodic sync left and returns the outcome
// (total delivered, including periodicSynced). Returns nil when automatic cloud
// delivery is off (cloud or auto-sync disabled) — same gate as the periodic
// ticker, so auto_sync consistently controls all daemon-driven cloud delivery.
// The daemon does this itself rather than `pmg proxy stop` because, unlike stop,
// it has no proxy env vars and so dials SafeDep directly instead of routing
// through the proxy that is now shutting down. Uses a fresh context since the
// caller's may already be cancelled at shutdown.
func cloudFlush(cfg *config.RuntimeConfig, periodicSynced int) *CloudSyncResult {
if !cfg.Config.Cloud.Enabled || !cfg.Config.Cloud.AutoSync.Enabled {
return nil
}
synced, err := audit.DrainToCloud(context.Background(), cfg, cloudFlushLockWait, cloudFlushTimeout)
res := &CloudSyncResult{Synced: periodicSynced + synced}
if err != nil {
res.Error = err.Error()
log.Warnf("cloud event flush failed: %v", err)
} else {
log.Infof("Flushed %d events to SafeDep Cloud (%d during the run)", res.Synced, periodicSynced)
}
return res
}
// startCloudSyncLoop periodically drains pending audit events to SafeDep Cloud
// while the daemon runs. It returns a stop function that halts the ticker, waits
// for any in-flight drain to finish, and returns the running total of events
// delivered. The stop function must be called before the daemon's final flush so
// the two never hold the sync lock at once. A no-op when cloud sync or auto-sync
// is disabled; the daemon's automatic cloud delivery (periodic ticker and
// shutdown flush alike) honors the auto_sync flag.
func startCloudSyncLoop(cfg *config.RuntimeConfig) func() int {
if !cfg.Config.Cloud.Enabled || !cfg.Config.Cloud.AutoSync.Enabled {
return func() int { return 0 }
}
stop := make(chan struct{})
done := make(chan struct{})
var total int // written only by the goroutine; read after <-done (happens-before)
go func() {
defer close(done)
ticker := time.NewTicker(cloudSyncInterval)
defer ticker.Stop()
for {
select {
case <-stop:
return
case <-ticker.C:
synced, err := audit.DrainToCloud(context.Background(), cfg, cloudSyncTickLockWait, cloudSyncTickTimeout)
if err != nil {
if errors.Is(err, audit.ErrSyncInProgress) {
log.Debugf("periodic cloud sync skipped: another sync in progress")
} else {
log.Warnf("periodic cloud sync failed: %v", err)
}
continue
}
total += synced
if synced > 0 {
log.Infof("Periodic cloud sync: flushed %d events", synced)
}
}
}
}()
return func() int {
close(stop)
<-done
return total
}
}
// listenAddr resolves the proxy's bind address from config (host) and the
// --port flag. Host defaults to loopback.
func listenAddr(host string, port int) string {
if host == "" {
host = "127.0.0.1"
}
return net.JoinHostPort(host, strconv.Itoa(port))
}
// autoBlockConfirmations drains the confirmation channel and always denies,
// appropriate for non-interactive CI/CD environments.
func autoBlockConfirmations(ch chan *interceptors.ConfirmationRequest) {
for req := range ch {
log.Warnf("Persistent proxy: auto-blocking suspicious package %s", req.PackageVersion.GetPackage().GetName())
req.ResponseChan <- false
close(req.ResponseChan)
}
}
+84
View File
@@ -0,0 +1,84 @@
package proxyserver
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"syscall"
)
const stateFileName = "proxy-state.json"
// State is the on-disk record of a running persistent proxy. It is written by
// the daemon and read by the stop/env/status commands.
type State struct {
PID int `json:"pid"`
Addr string `json:"addr"`
CACertPath string `json:"ca_cert_path"`
BlockedCount int `json:"blocked_count"`
// CloudSync records the daemon's shutdown cloud flush so `pmg proxy stop`
// can report the outcome. The daemon's own logs go to proxy.log (and are
// suppressed without --debug), so the state file is how the result reaches
// the stop process. nil when cloud sync is disabled.
CloudSync *CloudSyncResult `json:"cloud_sync,omitempty"`
}
// CloudSyncResult is the outcome of the daemon's shutdown flush to SafeDep Cloud.
type CloudSyncResult struct {
Synced int `json:"synced"`
Error string `json:"error,omitempty"`
}
func stateFilePath(dir string) string {
return filepath.Join(dir, stateFileName)
}
func writeState(path string, s State) error {
data, err := json.Marshal(s)
if err != nil {
return fmt.Errorf("marshal proxy state: %w", err)
}
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
return fmt.Errorf("create proxy state dir: %w", err)
}
return os.WriteFile(path, data, 0o600)
}
func readState(path string) (State, error) {
data, err := os.ReadFile(path)
if err != nil {
return State{}, fmt.Errorf("read proxy state: %w", err)
}
var s State
if err := json.Unmarshal(data, &s); err != nil {
return State{}, fmt.Errorf("unmarshal proxy state: %w", err)
}
return s, nil
}
func removeState(path string) error {
return os.Remove(path)
}
// IsRunning reports whether the recorded PID is a live process.
func (s State) IsRunning() bool {
if s.PID <= 0 {
return false
}
proc, err := os.FindProcess(s.PID)
if err != nil {
return false
}
return proc.Signal(syscall.Signal(0)) == nil
}
// ResolveStatePath returns the effective state file path: the flag override
// when set, otherwise <cacheDir>/proxy-state.json.
func ResolveStatePath(flag, cacheDir string) string {
if flag != "" {
return flag
}
return stateFilePath(cacheDir)
}
+73
View File
@@ -0,0 +1,73 @@
package proxyserver
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestWriteAndReadState(t *testing.T) {
dir := t.TempDir()
path := stateFilePath(dir)
s := State{PID: 12345, Addr: "127.0.0.1:9999", CACertPath: "/tmp/ca.pem"}
require.NoError(t, writeState(path, s))
got, err := readState(path)
require.NoError(t, err)
assert.Equal(t, s.PID, got.PID)
assert.Equal(t, s.Addr, got.Addr)
assert.Equal(t, s.CACertPath, got.CACertPath)
}
func TestWriteStateCreatesMissingDir(t *testing.T) {
path := filepath.Join(t.TempDir(), "nested", "dir", "proxy-state.json")
require.NoError(t, writeState(path, State{PID: 1, Addr: "127.0.0.1:1"}))
got, err := readState(path)
require.NoError(t, err)
assert.Equal(t, 1, got.PID)
}
func TestReadStateMissingFile(t *testing.T) {
_, err := readState(filepath.Join(t.TempDir(), "nonexistent.json"))
assert.Error(t, err)
}
func TestRemoveState(t *testing.T) {
dir := t.TempDir()
path := stateFilePath(dir)
require.NoError(t, writeState(path, State{PID: 1, Addr: "127.0.0.1:1"}))
require.NoError(t, removeState(path))
_, err := os.Stat(path)
assert.True(t, os.IsNotExist(err))
}
func TestIsRunningCurrentProcess(t *testing.T) {
s := State{PID: os.Getpid(), Addr: "127.0.0.1:1"}
assert.True(t, s.IsRunning())
}
func TestIsRunningDeadPID(t *testing.T) {
s := State{PID: 999999999}
assert.False(t, s.IsRunning())
}
func TestStateFilePath(t *testing.T) {
assert.Equal(t, "/some/dir/proxy-state.json", stateFilePath("/some/dir"))
}
func TestResolveStatePath(t *testing.T) {
t.Run("flag override wins", func(t *testing.T) {
assert.Equal(t, "/custom/proxy.json", ResolveStatePath("/custom/proxy.json", "/cache"))
})
t.Run("defaults to cacheDir", func(t *testing.T) {
assert.Equal(t, filepath.Join("/cache", "proxy-state.json"), ResolveStatePath("", "/cache"))
})
}
+26
View File
@@ -0,0 +1,26 @@
package proxyserver
// StatusInfo describes the proxy's current state for rendering by the caller.
type StatusInfo struct {
Found bool
Running bool
PID int
Addr string
CACert string
}
// GetStatus reports the proxy status from the state file at statePath.
func GetStatus(statePath string) StatusInfo {
state, err := readState(statePath)
if err != nil {
return StatusInfo{Found: false}
}
return StatusInfo{
Found: true,
Running: state.IsRunning(),
PID: state.PID,
Addr: state.Addr,
CACert: state.CACertPath,
}
}
+90
View File
@@ -0,0 +1,90 @@
package proxyserver
import (
"fmt"
"os"
"syscall"
"time"
"github.com/safedep/dry/log"
)
const (
stopPollInterval = 200 * time.Millisecond
// stopWaitTimeout must exceed the daemon's worst-case shutdown (drain + cloud
// flush), so we never read stale state or delete the state file while the
// daemon is still flushing. Derived from the daemon's budget so the two can't
// drift apart.
stopWaitTimeout = daemonShutdownBudget + 15*time.Second
)
// StopResult carries the outcome of stopping the proxy so the caller can render
// a summary and decide whether to fail (e.g. on policy violations).
type StopResult struct {
PID int
BlockedCount int
// StateVerified is false when the final state could not be read after
// shutdown (e.g. the proxy crashed), which callers may treat as fail-closed.
StateVerified bool
// CloudSync is the daemon's shutdown flush outcome (nil when cloud sync is
// disabled), surfaced so the caller can report it.
CloudSync *CloudSyncResult
}
// Stop signals the running proxy to terminate, waits for it to exit, and
// removes the state file. It returns a StopResult describing the run. The
// daemon flushes audit events to the cloud itself during shutdown (it, unlike
// this process, has no proxy env vars). Operational failures (no proxy running,
// signal errors) are returned as errors.
func Stop(statePath string) (StopResult, error) {
state, err := readState(statePath)
if err != nil {
return StopResult{}, fmt.Errorf("no proxy state found — is the proxy running? (%w)", err)
}
if !state.IsRunning() {
if rerr := removeState(statePath); rerr != nil {
log.Warnf("failed to remove proxy state file: %v", rerr)
}
return StopResult{}, fmt.Errorf("proxy process (pid %d) is not running; state file cleaned up", state.PID)
}
proc, err := os.FindProcess(state.PID)
if err != nil {
return StopResult{}, fmt.Errorf("find proxy process (pid %d): %w", state.PID, err)
}
if err := proc.Signal(syscall.SIGTERM); err != nil {
return StopResult{}, fmt.Errorf("send SIGTERM to proxy (pid %d): %w", state.PID, err)
}
deadline := time.Now().Add(stopWaitTimeout)
exited := false
for time.Now().Before(deadline) {
if !state.IsRunning() {
exited = true
break
}
time.Sleep(stopPollInterval)
}
// If the daemon is still alive it is likely mid-flush. Do not read the state
// (it is not final yet) or remove the file (the daemon still owns it).
// Return an error so --fail-on-violation fails closed rather than reporting
// a stale "0 blocked".
if !exited {
return StopResult{}, fmt.Errorf("proxy (pid %d) did not shut down within %s; leaving state file in place", state.PID, stopWaitTimeout)
}
final, readErr := readState(statePath)
if rerr := removeState(statePath); rerr != nil {
log.Warnf("failed to remove proxy state file: %v", rerr)
}
return StopResult{
PID: state.PID,
BlockedCount: final.BlockedCount,
StateVerified: readErr == nil,
CloudSync: final.CloudSync,
}, nil
}
+23
View File
@@ -12,6 +12,7 @@ import (
"github.com/safedep/pmg/cmd/executors"
landlockCmd "github.com/safedep/pmg/cmd/landlock"
"github.com/safedep/pmg/cmd/npm"
proxyCmd "github.com/safedep/pmg/cmd/proxy"
"github.com/safedep/pmg/cmd/pypi"
sandboxCmd "github.com/safedep/pmg/cmd/sandbox"
"github.com/safedep/pmg/cmd/setup"
@@ -108,6 +109,17 @@ func main() {
ui.ErrorExit(err)
}
// The proxy daemon delivers events itself (periodic sync + shutdown
// flush), so suppress the detached background auto-sync for proxy
// commands. It would otherwise spawn a redundant child that contends
// for the sync lock and, from `pmg proxy stop`, routes through the
// now-stopped proxy. We suppress the spawn directly rather than
// flipping AutoSync.Enabled, because the daemon's periodic sync
// honors that flag.
if isProxyCommand(cmd) {
audit.SuppressBackgroundSync()
}
config.FinalizeDependencyCooldownOverride()
// Parse and validate --sandbox-allow flags after all flags are resolved
@@ -148,6 +160,7 @@ func main() {
cmd.AddCommand(pypi.NewUvCommand())
cmd.AddCommand(pypi.NewPoetryCommand())
cmd.AddCommand(executors.NewPipxCommand())
cmd.AddCommand(proxyCmd.NewProxyCommand())
cmd.AddCommand(version.NewVersionCommand())
cmd.AddCommand(setup.NewSetupCommand())
cmd.AddCommand(setup.NewRemoveCommand())
@@ -214,3 +227,13 @@ func logDebugContext() {
log.Debugf("Dry run: %t, insecure installation: %t, trusted packages: %d",
cfg.DryRun, cfg.InsecureInstallation, len(cfg.Config.TrustedPackages))
}
// isProxyCommand reports whether cmd is `pmg proxy` or one of its subcommands.
func isProxyCommand(cmd *cobra.Command) bool {
for c := cmd; c != nil; c = c.Parent() {
if c.Name() == "proxy" {
return true
}
}
return false
}
+49
View File
@@ -0,0 +1,49 @@
package packagemanager
import "fmt"
// proxyNoProxyList is the NO_PROXY value for proxied package-manager runs.
//
// The IPv6 loopback uses the bare ::1: the bracketed [::1] is URL syntax that
// crashes Python's urllib/httpx (#339). Trade-off: Node's NODE_USE_ENV_PROXY
// (undici) only bypasses the bracketed form, so a literal http://[::1] from
// Node still gets proxied. localhost/127.0.0.1 cover the common cases; the
// IPv6 literal is a rare edge we accept since NO_PROXY can't be set per-client.
const proxyNoProxyList = "localhost,127.0.0.1,::1"
// EnvVarForProxy returns the environment variables (KEY=VALUE lines) that route
// the supported package managers through the proxy at proxyAddr and make them
// trust its MITM CA at certPath. It encodes per-package-manager quirks: yarn
// Berry ignores HTTP_PROXY and needs YARN_* (#319); pip/requests and Node each
// read their own CA-bundle var.
//
// The cert-path variables are always emitted, never skipped based on OS
// trust-store status. Whether a tool trusts the OS store varies by tool,
// version and config: modern pip (>=24.2) and recent Node (--use-system-ca) can
// read it, but older versions, requests/certifi, and default configs still rely
// on bundled CA lists. Emitting these vars is the conservative choice that works
// across that matrix, and is harmless for tools that do read the OS store (they
// ignore the vars). Skipping them when a system CA exists would silently break
// any tool still on a bundled store.
func EnvVarForProxy(proxyAddr, certPath string) []string {
proxyURL := fmt.Sprintf("http://%s", proxyAddr)
return []string{
"PIP_RETRIES=0",
"NODE_USE_ENV_PROXY=1",
fmt.Sprintf("HTTP_PROXY=%s", proxyURL),
fmt.Sprintf("HTTPS_PROXY=%s", proxyURL),
fmt.Sprintf("http_proxy=%s", proxyURL),
fmt.Sprintf("https_proxy=%s", proxyURL),
fmt.Sprintf("NO_PROXY=%s", proxyNoProxyList),
fmt.Sprintf("no_proxy=%s", proxyNoProxyList),
fmt.Sprintf("PIP_PROXY=%s", proxyURL),
fmt.Sprintf("YARN_HTTP_PROXY=%s", proxyURL),
fmt.Sprintf("YARN_HTTPS_PROXY=%s", proxyURL),
fmt.Sprintf("NODE_EXTRA_CA_CERTS=%s", certPath),
fmt.Sprintf("SSL_CERT_FILE=%s", certPath),
fmt.Sprintf("REQUESTS_CA_BUNDLE=%s", certPath),
fmt.Sprintf("PIP_CERT=%s", certPath),
fmt.Sprintf("YARN_HTTPS_CA_FILE_PATH=%s", certPath),
}
}
+67
View File
@@ -0,0 +1,67 @@
package packagemanager
import (
"strings"
"testing"
"github.com/stretchr/testify/assert"
)
func envToMap(env []string) map[string]string {
m := make(map[string]string, len(env))
for _, e := range env {
if k, v, ok := strings.Cut(e, "="); ok {
m[k] = v
}
}
return m
}
// TestEnvVarForProxyConfiguresYarn proves the root cause of #319: yarn Berry
// (yarn 2+) ignores the standard HTTP_PROXY/HTTPS_PROXY env vars and only honors
// its own config, which can be set via YARN_* env overrides. Without these,
// yarn bypasses the MITM proxy entirely and no packages are analyzed.
func TestEnvVarForProxyConfiguresYarn(t *testing.T) {
const proxyAddr = "127.0.0.1:54321"
const caCertPath = "/tmp/pmg-ca-cert.pem"
env := envToMap(EnvVarForProxy(proxyAddr, caCertPath))
proxyURL := "http://" + proxyAddr
assert.Equal(t, proxyURL, env["YARN_HTTP_PROXY"],
"yarn ignores HTTP_PROXY; YARN_HTTP_PROXY is required to route yarn through the proxy")
assert.Equal(t, proxyURL, env["YARN_HTTPS_PROXY"],
"yarn ignores HTTPS_PROXY; YARN_HTTPS_PROXY is required to route yarn through the proxy")
assert.Equal(t, caCertPath, env["YARN_HTTPS_CA_FILE_PATH"],
"yarn ignores NODE_EXTRA_CA_CERTS; YARN_HTTPS_CA_FILE_PATH is required to trust the MITM CA")
}
// TestEnvVarForProxyNoProxyIPv6 proves the fix for #339: the IPv6 loopback in
// NO_PROXY must be bare (::1), not bracketed ([::1]). Brackets are URL syntax,
// not NO_PROXY syntax, and Python's urllib/httpx crashes parsing them with
// "Invalid port: ':1]'".
func TestEnvVarForProxyNoProxyIPv6(t *testing.T) {
env := envToMap(EnvVarForProxy("127.0.0.1:54321", "/tmp/pmg-ca-cert.pem"))
for _, key := range []string{"NO_PROXY", "no_proxy"} {
assert.Equal(t, "localhost,127.0.0.1,::1", env[key],
"%s must use bare ::1; bracketed [::1] is invalid NO_PROXY syntax and crashes httpx", key)
assert.NotContains(t, env[key], "[::1]",
"%s must not contain bracketed IPv6 loopback", key)
}
}
// TestEnvVarForProxyAlwaysEmitsCertVars proves the cert-path vars are always
// present (never skipped on OS trust-store status), since trust behavior varies
// by tool/version/config and many tools still rely on bundled CA stores.
func TestEnvVarForProxyAlwaysEmitsCertVars(t *testing.T) {
env := envToMap(EnvVarForProxy("127.0.0.1:9000", "/tmp/ca.pem"))
assert.Equal(t, "http://127.0.0.1:9000", env["HTTP_PROXY"])
for _, key := range []string{
"NODE_EXTRA_CA_CERTS", "SSL_CERT_FILE", "REQUESTS_CA_BUNDLE",
"PIP_CERT", "YARN_HTTPS_CA_FILE_PATH",
} {
assert.Equal(t, "/tmp/ca.pem", env[key], "%s must point at the CA bundle", key)
}
}
+18
View File
@@ -9,11 +9,29 @@ import (
const (
caCertFileName = "ca-cert.pem"
caKeyFileName = "ca-key.pem"
// proxyCABundleFileName is the merged CA bundle (PMG CA + system roots) that
// proxy clients trust via SSL_CERT_FILE/NODE_EXTRA_CA_CERTS. It is distinct
// from caCertFileName, which is just the PMG CA. Owned here so callers don't
// invent their own names.
proxyCABundleFileName = "proxy-ca.pem"
)
func CACertPath(dir string) string { return filepath.Join(dir, caCertFileName) }
func CAKeyPath(dir string) string { return filepath.Join(dir, caKeyFileName) }
// ProxyCABundlePath returns the stable, machine-local path of the merged CA
// bundle. Used by the persistent proxy, whose daemon/env/stop processes all
// reference the same file for the proxy's lifetime.
func ProxyCABundlePath(dir string) string { return filepath.Join(dir, proxyCABundleFileName) }
// EphemeralProxyCABundlePath returns a per-process temp path for the merged CA
// bundle. Used by the per-command flow, which writes a throwaway bundle and
// removes it on exit; the pid keeps concurrent runs from colliding.
func EphemeralProxyCABundlePath() string {
return filepath.Join(os.TempDir(), fmt.Sprintf("pmg-%d-%s", os.Getpid(), proxyCABundleFileName))
}
// PersistentCACertManagerConfig returns the config for the on-disk,
// system-trusted CA. The root is long-lived (10 years) because rotating an
// installed, trusted root is expensive; leaf certs remain short (1 day).