From be36bcc802c9df3ee8c07d1699e40ecbfae0fc88 Mon Sep 17 00:00:00 2001 From: Malin Date: Fri, 7 Aug 2026 21:04:24 +0200 Subject: [PATCH] docs: document pf NAT interface-macro silent-no-op bug on shared-IP-alias hosts Real incident on the staging host: nat on $ext_if from to any -> ($ext_if) silently failed to rewrite the source IP when that IP was already aliased on the same interface, breaking all outbound jail traffic with no error anywhere - just clean timeouts. Fixed by using an explicit target IP instead of the interface macro. Cost real debugging time to trace from "WordPress admin hangs after a PHP upgrade" back to this. Co-Authored-By: Claude Sonnet 5 --- skills/bastille-jail-provisioning/SKILL.md | 53 ++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/skills/bastille-jail-provisioning/SKILL.md b/skills/bastille-jail-provisioning/SKILL.md index 3abd73c..79c2561 100644 --- a/skills/bastille-jail-provisioning/SKILL.md +++ b/skills/bastille-jail-provisioning/SKILL.md @@ -169,6 +169,59 @@ If a host doesn't show this pattern on ANY existing jail, it likely doesn't use per-jail dedicated ports — don't assume the convention applies fleet-wide without checking the specific host first. +## pf `nat on $ext_if from to any -> ($ext_if)` silently fails to translate the source IP when that IP is already aliased on the same interface + +**Confirmed real incident, 2026-08-07**, on a host using the shared-IP- +aliasing jail pattern (jail IPs are `/32` aliases directly on the host's +own external interface, e.g. `192.168.0.185` aliased onto `vtnet0` +alongside the interface's real address `192.168.168.64` -- not the VNET/ +bridge pattern where jails get a genuinely separate subnet). A `nat on +$ext_if from to any -> ($ext_if)` rule -- the standard, +seemingly-correct pf idiom for masquerading outbound jail traffic -- +**did not rewrite the source address at all** for these jails. Outbound +packets left with the jail's own 192.168.0.x source IP untouched (only +the source *port* got NAT'd), which the upstream network silently +dropped (source IP outside the VM's assigned subnet, likely anti-spoof +filtering) -- symptom was a clean connection timeout, not a pf/rule +error, with a real `SYN_SENT` state visible in `pfctl -s states` showing +the untranslated source IP in parens. This broke every outbound call any +jail on that subnet tried to make (external API calls, WordPress core's +own update-check pings, etc.) -- surfaced as WordPress admin pages +hanging/timing out with no PHP error logged anywhere, since the PHP-FPM +worker was just sleeping on a TCP connect that would never complete. + +**Root cause**: the `($ext_if)` interface-macro NAT target apparently +skips/no-ops translation when the connection's source address is *already* +one of the interface's own configured addresses -- plausible as "this +traffic looks locally-originated already, no need to rewrite" logic in +pf's NAT implementation, but wrong for this topology where those aliased +addresses are jail IPs that specifically need translating to look like +the host's *primary* address before leaving. + +**Fix: use an explicit IP as the NAT target, not the interface macro.** +``` +ext_if = "vtnet0" +ext_ip = "192.168.168.64" # the interface's real, non-aliased address +lan_net = "192.168.0.0/24" # the jail-alias subnet + +nat on $ext_if from $lan_net to any -> $ext_ip # works +# nat on $ext_if from $lan_net to any -> ($ext_if) # looked identical, silently no-ops +``` +**Verify a NAT fix actually rewrites the source**, don't just check +`pfctl -s nat` shows the rule (a no-op-translating rule still shows up +there looking correct) -- watch `pfctl -s states` during a live outbound +attempt from the jail and confirm the parenthetical translated +address/port shown is genuinely different from the jail's own IP, not +just a port change on the same source IP. + +This is a distinct topology from the VNET/bridge pattern documented +elsewhere in this skill and in `jail-dedicated-subnet-migration` (where +jails get a real separate subnet on a bridge interface, e.g. +`10.20.0.0/24`) -- that pattern's NAT rules were unaffected by this bug +in the same test. Check which pattern a given host actually uses +(`ifconfig` -- are jail IPs aliased directly on the external interface, +or on a separate bridge?) before assuming either NAT idiom is safe. + ## Stock stack includes Valkey, not just nginx/php/MariaDB The fleet's standard per-host stack (see `docs/server-funky.md`) is