--- name: jail-dedicated-subnet-migration description: Move Bastille jails from shared-IP-aliasing on the LAN interface to a dedicated private VNET subnet, eliminating ARP collisions with other physical LAN devices. Piloted successfully on staging-1 -> staging1, 2026-08-03. license: MIT source: original, from the staging host subnet migration pilot --- # Jail Dedicated Subnet Migration ## Why Jails using shared-IP aliasing directly on the host's physical LAN interface (`vtnet0`) share that interface's broadcast domain with every other device on the LAN. Confirmed real incidents (see `bastille-jail-provisioning`'s ARP-collision section): a jail IP that looked free by every host-local check turned out to already belong to a different physical device or another host's jail on the same shared subnet, causing silent networking failures that look exactly like a config bug. Moving jails to their own private VNET subnet eliminates this class of collision entirely — confirmed via `arp-scan` on the physical LAN interface after migration: the jail's new IP produces zero response, proving it's genuinely invisible outside the host. ## Prerequisites - `if_bridge` and `pf` kernel modules must load (`kldload if_bridge`, `kldload pf` — both loaded cleanly on a stock FreeBSD 15.1 box with no prior config). - A private subnet not overlapping the LAN (`192.168.0.0/24` here) or the Tailscale CGNAT range (`100.64.0.0/10`) — used `10.20.0.0/24`. - Root on the jail host. ## Setup (host-level, once per host) 1. **Create the bridge, give the host itself an IP on it:** ```sh ifconfig bridge create name bastille0 ifconfig bastille0 inet 10.20.0.1/24 up ``` 2. **Persist across reboot:** ```sh sysrc cloned_interfaces+=bastille0 sysrc ifconfig_bastille0='inet 10.20.0.1/24' sysrc gateway_enable=YES ``` 3. **NAT for jail egress** (jails need this to reach the internet — the private subnet isn't routable from the LAN's own gateway): ```sh cat > /etc/pf.conf <<'EOF' ext_if = "vtnet0" jail_net = "10.20.0.0/24" nat on $ext_if from $jail_net to any -> ($ext_if) pass in pass out EOF ``` **`sysrc gateway_enable=YES` alone is not enough** — it only persists `net.inet.ip.forwarding=1` for the *next* boot, it doesn't apply it to the live running kernel. Confirmed on venus: NAT looked correctly configured (`pfctl -s nat` showed the rule) but every jail's outbound traffic still timed out until `sysctl net.inet.ip.forwarding=1` was run live. Always run both, not just the `sysrc` persistence step. **The `pass in`/`pass out` rules are deliberately permissive** — this is a NAT-only ruleset, not a firewall. Don't add restrictive rules without separately verifying they don't break existing services (SSH/Tailscale) on the host. 4. **Enabling pf is a genuine remote-lockout risk** — even a permissive ruleset syntax error can default-deny. Test with `service pf onestart` first (bypasses the `pf_enable` rc.conf check, doesn't persist) and confirm the SSH session survives before persisting with `sysrc pf_enable=YES` + `service pf start`. This got flagged by this session's own permission classifier as needing explicit confirmation before proceeding — treat that as correct caution, not an obstacle to route around. ## Creating a VNET jail on the new subnet ```sh bastille create -B -g 10.20.0.1 10.20.0.X/24 bastille0 ``` - `-B`/`--bridge` enables VNET with a bridge interface (not `-V`, which is for a *physical* interface). - `-g` sets the jail's default gateway — must be the bridge's own IP (`10.20.0.1`), or the jail has no route out. - **VNET jail names cannot contain `-` or `_`** (`[ERROR]: VNET jail names may not contain (-|_) characters.`) — bastille derives epair interface names from the jail name. A jail called `staging-1` under shared-IP mode had to become `staging1` under VNET. Verify before trusting it: ```sh bastille cmd fetch -o /dev/null http://1.1.1.1 # raw IP, outbound NAT bastille cmd fetch -o /dev/null https://pkg.freebsd.org/ # DNS + outbound ping -c2 10.20.0.X # host -> jail, no NAT needed (same bridge) arp-scan --interface=vtnet0 10.20.0.X # must show ZERO responses -- proves LAN isolation ``` **Bridge port showing `LEARNING` in `ifconfig bastille0` output is a red herring, not a real blocker** — host-to-jail `ping` worked fine on venus even while the member port's flags still showed `LEARNING` well past the RSTP forward-delay window. Don't chase STP state as the cause of a connectivity problem without first confirming with a direct `ping` from the host that L2 forwarding is actually broken (it usually isn't). **A fresh jail may inherit a broken `resolv.conf` from the host.** On a host running `local_unbound_enable="YES"` (venus does, staging didn't), new jails got `nameserver 127.0.0.1` copied into their `resolv.conf` -- which inside the jail points at the jail's *own* loopback, where nothing is listening, not the host's unbound instance. Symptom: raw-IP `fetch` still fails with "Transient resolver failure" even after outbound NAT/forwarding is confirmed working, because `fetch` calls the resolver even for IP literals. Fix: overwrite the jail's `resolv.conf` with public DNS directly (`1.1.1.1` / `1.0.0.1`), don't assume the host's own `/etc/resolv.conf` is safe to inherit as-is. ## Migrating an existing jail's content (not just spinning up empty) Don't destroy-and-rebuild-from-scratch if the jail has real installed state worth keeping (plugins, test data, DB content). Instead: 1. Create the new VNET jail per above, install the same stock stack (nginx, php-fpm, mariadb-client, etc.) matching the old jail's. 2. Copy the web root: `cp -a /usr/local/www/site/. /usr/local/www/site/`, then `chown -R www:www` on the destination. 3. **The database doesn't move** if it lives on the host's own local MariaDB (the common pattern on these hosts) — only the jail's IP changes, so `wp-config.php`'s `DB_HOST` (the host's own LAN IP) stays the same. What changes is the MySQL **grant**, which is scoped by the connecting IP: `CREATE USER IF NOT EXISTS ''@'' IDENTIFIED BY ''; GRANT ALL PRIVILEGES ON .* TO ''@'';`. Extract the existing DB password from the copied `wp-config.php` programmatically (`awk -F"'" '{print $4}'` on the `DB_PASSWORD` line, or similar) — **don't manually retype/copy-paste a password across terminal output**, a single mistyped character produces a misleading "Access denied (using password: YES)" error that looks like a grant/network problem but is actually just a wrong password. If that error shows up despite a seemingly-correct `CREATE USER`, the fastest fix is `ALTER USER ''@'' IDENTIFIED BY '';` rather than debugging network connectivity first. 4. Test host→jail DB connectivity is **not** NAT-hairpinned: since the host's MariaDB and the new jail are both local to the same host, and the bridge gives the jail a direct route to the host's own bridge IP (`10.20.0.1`) without going through the NAT'd `vtnet0` path at all, this generally works cleanly — confirmed connections to the host's LAN IP from a jail on the private subnet arrive with the jail's real source IP intact (not NAT-masqueraded), because the kernel routes host-destined traffic locally rather than sending it out and back through the NAT rule. Don't assume a hairpin NAT problem without testing first — it added an unnecessary detour. 5. Verify the site fully — **a page-cache HIT can mislead you into thinking DB connectivity works when it doesn't**, since a cached page never touches the database. Purge any cache first, then confirm both a fresh MISS render (real DB read) and a subsequent HIT (served from cache) both work, plus a `wp-cli` command that requires DB access (`wp option get siteurl`) as an unambiguous connectivity check. ## Cutover **Real outage, 2026-08-03: verifying the new jail by curling its IP directly with a `Host:` header does NOT prove the live site works**, because that bypasses the actual production path. The real path is NPM → the jail host's own nginx (a per-domain vhost with `proxy_pass` hardcoded to the jail's IP) → the jail. Stopping the old jail after only an IP-direct check leaves that vhost still pointing at a now-dead IP — instant, silent outage for every domain migrated this way (hit 5 domains across 2 hosts, funky and venus, before being caught by the user noticing timeouts, not by this process). `migrate-jail-to-subnet.sh` now rewires the host-nginx vhost automatically as its own step (finds the `.conf` referencing the old jail's IP via `bastille list`, `sed`s it to the new IP, `nginx -t` then `-s reload`, aborts and reverts from `.bak` if the syntax check fails) — **but always verify through `curl -H 'Host: ' -H 'X-Forwarded-Proto: https' http://127.0.0.1/` on the jail host itself** (exercises the real vhost path) before trusting a migration, not a direct-by-IP request. 1. Verify the new jail through the real path (see above), not by IP alone. 2. Stop (don't destroy) the old jail — keep it as a rollback safety net until the new one is fully proven in real use. Safe to do now that the vhost itself has already been repointed. 3. Destroy the old jail only once genuinely confident — this session left it stopped rather than destroyed on the first migration, as the controlled/cautious default. 4. Update any skill/doc/memory that references the old jail name or IP (grep for both across `agent-skills` and this repo's `docs/`) — easy to miss and leaves stale instructions for the next delegate. 5. **After any batch of migrations, sweep for stragglers**: `grep -rl 'proxy_pass.*192\.168\.' /usr/local/etc/nginx/sites-*/` on the host catches any vhost still pointing at a shared-subnet IP whose jail has since been stopped — the fastest way to confirm nothing was missed. ## Check disk space BEFORE creating any jail, every time **Do not assume "blank slate, plenty of room" holds on every host** just because it held on the pilot host. Confirmed real incident on the second host attempted (venus, 2026-08-03): creating 3 new full-stack jails (~200MB retained footprint each after packages) pushed an already-96%-full ZFS pool over the edge and **crashed the shared MariaDB instance**, taking down every site on the host (a real, if brief, production outage) -- this exact failure mode had already been documented once before on this same host from an unrelated earlier incident, and should have been checked proactively rather than triggered again. **Before creating even one new jail**, run `zfs list zroot/ROOT/default -o avail` (or `df -h` if not ZFS) and treat anything under a few GB as a hard stop -- clean up first (`bastille pkg clean -ay` across existing jails reclaims real space safely, it's just downloaded package cache) or get the capacity issue resolved before proceeding. Re-check after every 1-2 jail creations during a bulk rollout, don't just check once at the start and assume it holds for the whole batch. If MariaDB (or any host-shared service) does go down from disk pressure: the rc.d script name may not match the obvious guess (`mysql-server`, not `mysql` or `mariadb`, was the actual name on venus) -- check `ls /usr/local/etc/rc.d/` rather than guessing. After restarting, verify with actual `curl` tests against multiple real sites (not just "is the process running") before considering it resolved. ## Rollout scope discipline This was explicitly piloted on **one jail on one host** before being considered for the rest of the fleet — treat "prove it on a single low-stakes target first" as the right default for any jail-networking architecture change, not just this one. Don't extend to production jails or other hosts without a separate, explicit go-ahead per host.