Files

T

Malin 8dbe1eb9c1 docs: document the proxy_pass cutover outage and its fix

2026-08-03 14:29:27 +02:00

12 KiB

Raw Blame History

name, description, license, source

name	description	license	source
jail-dedicated-subnet-migration	Move Bastille jails from shared-IP-aliasing on the LAN interface to a dedicated private VNET subnet, eliminating ARP collisions with other physical LAN devices. Piloted successfully on staging-1 -> staging1, 2026-08-03.	MIT	original, from the staging host subnet migration pilot

Jail Dedicated Subnet Migration

Why

Jails using shared-IP aliasing directly on the host's physical LAN interface (vtnet0) share that interface's broadcast domain with every other device on the LAN. Confirmed real incidents (see bastille-jail-provisioning's ARP-collision section): a jail IP that looked free by every host-local check turned out to already belong to a different physical device or another host's jail on the same shared subnet, causing silent networking failures that look exactly like a config bug. Moving jails to their own private VNET subnet eliminates this class of collision entirely — confirmed via arp-scan on the physical LAN interface after migration: the jail's new IP produces zero response, proving it's genuinely invisible outside the host.

Prerequisites

if_bridge and pf kernel modules must load (kldload if_bridge, kldload pf — both loaded cleanly on a stock FreeBSD 15.1 box with no prior config).
A private subnet not overlapping the LAN (192.168.0.0/24 here) or the Tailscale CGNAT range (100.64.0.0/10) — used 10.20.0.0/24.
Root on the jail host.

Setup (host-level, once per host)

Create the bridge, give the host itself an IP on it:

ifconfig bridge create name bastille0
ifconfig bastille0 inet 10.20.0.1/24 up

Persist across reboot:

sysrc cloned_interfaces+=bastille0
sysrc ifconfig_bastille0='inet 10.20.0.1/24'
sysrc gateway_enable=YES

NAT for jail egress (jails need this to reach the internet — the private subnet isn't routable from the LAN's own gateway):
```
cat > /etc/pf.conf <<'EOF'
ext_if = "vtnet0"
jail_net = "10.20.0.0/24"

nat on $ext_if from $jail_net to any -> ($ext_if)
pass in
pass out
EOF
```
sysrc gateway_enable=YES alone is not enough — it only persists net.inet.ip.forwarding=1 for the next boot, it doesn't apply it to the live running kernel. Confirmed on venus: NAT looked correctly configured (pfctl -s nat showed the rule) but every jail's outbound traffic still timed out until sysctl net.inet.ip.forwarding=1 was run live. Always run both, not just the sysrc persistence step. The pass in/pass out rules are deliberately permissive — this is a NAT-only ruleset, not a firewall. Don't add restrictive rules without separately verifying they don't break existing services (SSH/Tailscale) on the host.
Enabling pf is a genuine remote-lockout risk — even a permissive ruleset syntax error can default-deny. Test with service pf onestart first (bypasses the pf_enable rc.conf check, doesn't persist) and confirm the SSH session survives before persisting with sysrc pf_enable=YES + service pf start. This got flagged by this session's own permission classifier as needing explicit confirmation before proceeding — treat that as correct caution, not an obstacle to route around.

Creating a VNET jail on the new subnet

bastille create -B -g 10.20.0.1 <name> <release> 10.20.0.X/24 bastille0

-B/--bridge enables VNET with a bridge interface (not -V, which is for a physical interface).
-g sets the jail's default gateway — must be the bridge's own IP (10.20.0.1), or the jail has no route out.
VNET jail names cannot contain - or _ ([ERROR]: VNET jail names may not contain (-|_) characters.) — bastille derives epair interface names from the jail name. A jail called staging-1 under shared-IP mode had to become staging1 under VNET.

Verify before trusting it:

bastille cmd <name> fetch -o /dev/null http://1.1.1.1   # raw IP, outbound NAT
bastille cmd <name> fetch -o /dev/null https://pkg.freebsd.org/  # DNS + outbound
ping -c2 10.20.0.X                                       # host -> jail, no NAT needed (same bridge)
arp-scan --interface=vtnet0 10.20.0.X                    # must show ZERO responses -- proves LAN isolation

Bridge port showing LEARNING in ifconfig bastille0 output is a red herring, not a real blocker — host-to-jail ping worked fine on venus even while the member port's flags still showed LEARNING well past the RSTP forward-delay window. Don't chase STP state as the cause of a connectivity problem without first confirming with a direct ping from the host that L2 forwarding is actually broken (it usually isn't).

A fresh jail may inherit a broken resolv.conf from the host. On a host running local_unbound_enable="YES" (venus does, staging didn't), new jails got nameserver 127.0.0.1 copied into their resolv.conf -- which inside the jail points at the jail's own loopback, where nothing is listening, not the host's unbound instance. Symptom: raw-IP fetch still fails with "Transient resolver failure" even after outbound NAT/forwarding is confirmed working, because fetch calls the resolver even for IP literals. Fix: overwrite the jail's resolv.conf with public DNS directly (1.1.1.1 / 1.0.0.1), don't assume the host's own /etc/resolv.conf is safe to inherit as-is.

Migrating an existing jail's content (not just spinning up empty)

Don't destroy-and-rebuild-from-scratch if the jail has real installed state worth keeping (plugins, test data, DB content). Instead:

Create the new VNET jail per above, install the same stock stack (nginx, php-fpm, mariadb-client, etc.) matching the old jail's.
Copy the web root: cp -a <old-jail-root>/usr/local/www/site/. <new-jail-root>/usr/local/www/site/, then chown -R www:www on the destination.
The database doesn't move if it lives on the host's own local MariaDB (the common pattern on these hosts) — only the jail's IP changes, so wp-config.php's DB_HOST (the host's own LAN IP) stays the same. What changes is the MySQL grant, which is scoped by the connecting IP: CREATE USER IF NOT EXISTS '<dbuser>'@'<new-jail-ip>' IDENTIFIED BY '<password>'; GRANT ALL PRIVILEGES ON <dbname>.* TO '<dbuser>'@'<new-jail-ip>';. Extract the existing DB password from the copied wp-config.php programmatically (awk -F"'" '{print $4}' on the DB_PASSWORD line, or similar) — don't manually retype/copy-paste a password across terminal output, a single mistyped character produces a misleading "Access denied (using password: YES)" error that looks like a grant/network problem but is actually just a wrong password. If that error shows up despite a seemingly-correct CREATE USER, the fastest fix is ALTER USER '<user>'@'<ip>' IDENTIFIED BY '<password-read-fresh-from-file>'; rather than debugging network connectivity first.
Test host→jail DB connectivity is not NAT-hairpinned: since the host's MariaDB and the new jail are both local to the same host, and the bridge gives the jail a direct route to the host's own bridge IP (10.20.0.1) without going through the NAT'd vtnet0 path at all, this generally works cleanly — confirmed connections to the host's LAN IP from a jail on the private subnet arrive with the jail's real source IP intact (not NAT-masqueraded), because the kernel routes host-destined traffic locally rather than sending it out and back through the NAT rule. Don't assume a hairpin NAT problem without testing first — it added an unnecessary detour.
Verify the site fully — a page-cache HIT can mislead you into thinking DB connectivity works when it doesn't, since a cached page never touches the database. Purge any cache first, then confirm both a fresh MISS render (real DB read) and a subsequent HIT (served from cache) both work, plus a wp-cli command that requires DB access (wp option get siteurl) as an unambiguous connectivity check.

Cutover

Real outage, 2026-08-03: verifying the new jail by curling its IP directly with a Host: header does NOT prove the live site works, because that bypasses the actual production path. The real path is NPM → the jail host's own nginx (a per-domain vhost with proxy_pass hardcoded to the jail's IP) → the jail. Stopping the old jail after only an IP-direct check leaves that vhost still pointing at a now-dead IP — instant, silent outage for every domain migrated this way (hit 5 domains across 2 hosts, funky and venus, before being caught by the user noticing timeouts, not by this process). migrate-jail-to-subnet.sh now rewires the host-nginx vhost automatically as its own step (finds the .conf referencing the old jail's IP via bastille list, seds it to the new IP, nginx -t then -s reload, aborts and reverts from .bak if the syntax check fails) — but always verify through curl -H 'Host: <domain>' -H 'X-Forwarded-Proto: https' http://127.0.0.1/ on the jail host itself (exercises the real vhost path) before trusting a migration, not a direct-by-IP request.

Verify the new jail through the real path (see above), not by IP alone.
Stop (don't destroy) the old jail — keep it as a rollback safety net until the new one is fully proven in real use. Safe to do now that the vhost itself has already been repointed.
Destroy the old jail only once genuinely confident — this session left it stopped rather than destroyed on the first migration, as the controlled/cautious default.
Update any skill/doc/memory that references the old jail name or IP (grep for both across agent-skills and this repo's docs/) — easy to miss and leaves stale instructions for the next delegate.
After any batch of migrations, sweep for stragglers: grep -rl 'proxy_pass.*192\.168\.' /usr/local/etc/nginx/sites-*/ on the host catches any vhost still pointing at a shared-subnet IP whose jail has since been stopped — the fastest way to confirm nothing was missed.

Check disk space BEFORE creating any jail, every time

Do not assume "blank slate, plenty of room" holds on every host just because it held on the pilot host. Confirmed real incident on the second host attempted (venus, 2026-08-03): creating 3 new full-stack jails (~200MB retained footprint each after packages) pushed an already-96%-full ZFS pool over the edge and crashed the shared MariaDB instance, taking down every site on the host (a real, if brief, production outage) -- this exact failure mode had already been documented once before on this same host from an unrelated earlier incident, and should have been checked proactively rather than triggered again.

Before creating even one new jail, run zfs list zroot/ROOT/default -o avail (or df -h if not ZFS) and treat anything under a few GB as a hard stop -- clean up first (bastille pkg <jail> clean -ay across existing jails reclaims real space safely, it's just downloaded package cache) or get the capacity issue resolved before proceeding. Re-check after every 1-2 jail creations during a bulk rollout, don't just check once at the start and assume it holds for the whole batch.

If MariaDB (or any host-shared service) does go down from disk pressure: the rc.d script name may not match the obvious guess (mysql-server, not mysql or mariadb, was the actual name on venus) -- check ls /usr/local/etc/rc.d/ rather than guessing. After restarting, verify with actual curl tests against multiple real sites (not just "is the process running") before considering it resolved.

Rollout scope discipline

This was explicitly piloted on one jail on one host before being considered for the rest of the fleet — treat "prove it on a single low-stakes target first" as the right default for any jail-networking architecture change, not just this one. Don't extend to production jails or other hosts without a separate, explicit go-ahead per host.

12 KiB Raw Blame History