docs: document pf NAT interface-macro silent-no-op bug on shared-IP-alias hosts

Real incident on the staging host: nat on $ext_if from <subnet> to any ->
($ext_if) silently failed to rewrite the source IP when that IP was
already aliased on the same interface, breaking all outbound jail traffic
with no error anywhere - just clean timeouts. Fixed by using an explicit
target IP instead of the interface macro. Cost real debugging time to
trace from "WordPress admin hangs after a PHP upgrade" back to this.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-07 21:04:24 +02:00
co-authored by Claude Sonnet 5
parent 106e61cfbb
commit be36bcc802
@@ -169,6 +169,59 @@ If a host doesn't show this pattern on ANY existing jail, it likely
doesn't use per-jail dedicated ports — don't assume the convention
applies fleet-wide without checking the specific host first.
## pf `nat on $ext_if from <subnet> to any -> ($ext_if)` silently fails to translate the source IP when that IP is already aliased on the same interface
**Confirmed real incident, 2026-08-07**, on a host using the shared-IP-
aliasing jail pattern (jail IPs are `/32` aliases directly on the host's
own external interface, e.g. `192.168.0.185` aliased onto `vtnet0`
alongside the interface's real address `192.168.168.64` -- not the VNET/
bridge pattern where jails get a genuinely separate subnet). A `nat on
$ext_if from <jail-subnet> to any -> ($ext_if)` rule -- the standard,
seemingly-correct pf idiom for masquerading outbound jail traffic --
**did not rewrite the source address at all** for these jails. Outbound
packets left with the jail's own 192.168.0.x source IP untouched (only
the source *port* got NAT'd), which the upstream network silently
dropped (source IP outside the VM's assigned subnet, likely anti-spoof
filtering) -- symptom was a clean connection timeout, not a pf/rule
error, with a real `SYN_SENT` state visible in `pfctl -s states` showing
the untranslated source IP in parens. This broke every outbound call any
jail on that subnet tried to make (external API calls, WordPress core's
own update-check pings, etc.) -- surfaced as WordPress admin pages
hanging/timing out with no PHP error logged anywhere, since the PHP-FPM
worker was just sleeping on a TCP connect that would never complete.
**Root cause**: the `($ext_if)` interface-macro NAT target apparently
skips/no-ops translation when the connection's source address is *already*
one of the interface's own configured addresses -- plausible as "this
traffic looks locally-originated already, no need to rewrite" logic in
pf's NAT implementation, but wrong for this topology where those aliased
addresses are jail IPs that specifically need translating to look like
the host's *primary* address before leaving.
**Fix: use an explicit IP as the NAT target, not the interface macro.**
```
ext_if = "vtnet0"
ext_ip = "192.168.168.64" # the interface's real, non-aliased address
lan_net = "192.168.0.0/24" # the jail-alias subnet
nat on $ext_if from $lan_net to any -> $ext_ip # works
# nat on $ext_if from $lan_net to any -> ($ext_if) # looked identical, silently no-ops
```
**Verify a NAT fix actually rewrites the source**, don't just check
`pfctl -s nat` shows the rule (a no-op-translating rule still shows up
there looking correct) -- watch `pfctl -s states` during a live outbound
attempt from the jail and confirm the parenthetical translated
address/port shown is genuinely different from the jail's own IP, not
just a port change on the same source IP.
This is a distinct topology from the VNET/bridge pattern documented
elsewhere in this skill and in `jail-dedicated-subnet-migration` (where
jails get a real separate subnet on a bridge interface, e.g.
`10.20.0.0/24`) -- that pattern's NAT rules were unaffected by this bug
in the same test. Check which pattern a given host actually uses
(`ifconfig` -- are jail IPs aliased directly on the external interface,
or on a separate bridge?) before assuming either NAT idiom is safe.
## Stock stack includes Valkey, not just nginx/php/MariaDB
The fleet's standard per-host stack (see `docs/server-funky.md`) is