agent-skills/skills/jail-dedicated-subnet-migration/SKILL.md

name, description, license, source

name	description	license	source
jail-dedicated-subnet-migration	Move Bastille jails from shared-IP-aliasing on the LAN interface to a dedicated private VNET subnet, eliminating ARP collisions with other physical LAN devices. Piloted successfully on staging-1 -> staging1, 2026-08-03.	MIT	original, from the staging host subnet migration pilot

Jail Dedicated Subnet Migration

Why

Jails using shared-IP aliasing directly on the host's physical LAN interface (vtnet0) share that interface's broadcast domain with every other device on the LAN. Confirmed real incidents (see bastille-jail-provisioning's ARP-collision section): a jail IP that looked free by every host-local check turned out to already belong to a different physical device or another host's jail on the same shared subnet, causing silent networking failures that look exactly like a config bug. Moving jails to their own private VNET subnet eliminates this class of collision entirely — confirmed via arp-scan on the physical LAN interface after migration: the jail's new IP produces zero response, proving it's genuinely invisible outside the host.

Prerequisites

• if_bridge and pf kernel modules must load (kldload if_bridge, kldload pf — both loaded cleanly on a stock FreeBSD 15.1 box with no prior config).
• A private subnet not overlapping the LAN (192.168.0.0/24 here) or the Tailscale CGNAT range (100.64.0.0/10) — used 10.20.0.0/24.
• Root on the jail host.

Setup (host-level, once per host)

1. Create the bridge, give the host itself an IP on it:

   ifconfig bridge create name bastille0
   ifconfig bastille0 inet 10.20.0.1/24 up
   

2. Persist across reboot:

   sysrc cloned_interfaces+=bastille0
   sysrc ifconfig_bastille0='inet 10.20.0.1/24'
   sysrc gateway_enable=YES
   

3. NAT for jail egress (jails need this to reach the internet — the private subnet isn't routable from the LAN's own gateway):

   cat > /etc/pf.conf <<'EOF'
   ext_if = "vtnet0"
   jail_net = "10.20.0.0/24"
   
   nat on $ext_if from $jail_net to any -> ($ext_if)
   pass in
   pass out
   EOF
   

   The pass in/pass out rules are deliberately permissive — this is a NAT-only ruleset, not a firewall. Don't add restrictive rules without separately verifying they don't break existing services (SSH/Tailscale) on the host.
4. Enabling pf is a genuine remote-lockout risk — even a permissive ruleset syntax error can default-deny. Test with service pf onestart first (bypasses the pf_enable rc.conf check, doesn't persist) and confirm the SSH session survives before persisting with sysrc pf_enable=YES + service pf start. This got flagged by this session's own permission classifier as needing explicit confirmation before proceeding — treat that as correct caution, not an obstacle to route around.

Creating a VNET jail on the new subnet

bastille create -B -g 10.20.0.1 <name> <release> 10.20.0.X/24 bastille0

• -B/--bridge enables VNET with a bridge interface (not -V, which is for a physical interface).
• -g sets the jail's default gateway — must be the bridge's own IP (10.20.0.1), or the jail has no route out.
• VNET jail names cannot contain - or _ ([ERROR]: VNET jail names may not contain (-|_) characters.) — bastille derives epair interface names from the jail name. A jail called staging-1 under shared-IP mode had to become staging1 under VNET.

Verify before trusting it:

bastille cmd <name> fetch -o /dev/null http://1.1.1.1   # raw IP, outbound NAT
bastille cmd <name> fetch -o /dev/null https://pkg.freebsd.org/  # DNS + outbound
ping -c2 10.20.0.X                                       # host -> jail, no NAT needed (same bridge)
arp-scan --interface=vtnet0 10.20.0.X                    # must show ZERO responses -- proves LAN isolation

Migrating an existing jail's content (not just spinning up empty)

Don't destroy-and-rebuild-from-scratch if the jail has real installed state worth keeping (plugins, test data, DB content). Instead:

1. Create the new VNET jail per above, install the same stock stack (nginx, php-fpm, mariadb-client, etc.) matching the old jail's.
2. Copy the web root: cp -a <old-jail-root>/usr/local/www/site/. <new-jail-root>/usr/local/www/site/, then chown -R www:www on the destination.
3. The database doesn't move if it lives on the host's own local MariaDB (the common pattern on these hosts) — only the jail's IP changes, so wp-config.php's DB_HOST (the host's own LAN IP) stays the same. What changes is the MySQL grant, which is scoped by the connecting IP: CREATE USER IF NOT EXISTS '<dbuser>'@'<new-jail-ip>' IDENTIFIED BY '<password>'; GRANT ALL PRIVILEGES ON <dbname>.* TO '<dbuser>'@'<new-jail-ip>';. Extract the existing DB password from the copied wp-config.php programmatically (awk -F"'" '{print $4}' on the DB_PASSWORD line, or similar) — don't manually retype/copy-paste a password across terminal output, a single mistyped character produces a misleading "Access denied (using password: YES)" error that looks like a grant/network problem but is actually just a wrong password. If that error shows up despite a seemingly-correct CREATE USER, the fastest fix is ALTER USER '<user>'@'<ip>' IDENTIFIED BY '<password-read-fresh-from-file>'; rather than debugging network connectivity first.
4. Test host→jail DB connectivity is not NAT-hairpinned: since the host's MariaDB and the new jail are both local to the same host, and the bridge gives the jail a direct route to the host's own bridge IP (10.20.0.1) without going through the NAT'd vtnet0 path at all, this generally works cleanly — confirmed connections to the host's LAN IP from a jail on the private subnet arrive with the jail's real source IP intact (not NAT-masqueraded), because the kernel routes host-destined traffic locally rather than sending it out and back through the NAT rule. Don't assume a hairpin NAT problem without testing first — it added an unnecessary detour.
5. Verify the site fully — a page-cache HIT can mislead you into thinking DB connectivity works when it doesn't, since a cached page never touches the database. Purge any cache first, then confirm both a fresh MISS render (real DB read) and a subsequent HIT (served from cache) both work, plus a wp-cli command that requires DB access (wp option get siteurl) as an unambiguous connectivity check.

Cutover

1. Stop (don't destroy) the old jail — keep it as a rollback safety net until the new one is fully proven in real use.
2. If a host-level reverse-proxy vhost points at the old jail's IP, update it. (Not needed if the domain was never wired to a public vhost yet — check first with grep -rl <old-ip> /usr/local/etc/nginx/.)
3. Destroy the old jail only once genuinely confident — this session left it stopped rather than destroyed on the first migration, as the controlled/cautious default.
4. Update any skill/doc/memory that references the old jail name or IP (grep for both across agent-skills and this repo's docs/) — easy to miss and leaves stale instructions for the next delegate.

Rollout scope discipline

This was explicitly piloted on one jail on one host before being considered for the rest of the fleet — treat "prove it on a single low-stakes target first" as the right default for any jail-networking architecture change, not just this one. Don't extend to production jails or other hosts without a separate, explicit go-ahead per host.