agent-skills/skills/jail-dedicated-subnet-migration/SKILL.md

---
name: jail-dedicated-subnet-migration
description: Move Bastille jails from shared-IP-aliasing on the LAN interface to a dedicated private VNET subnet, eliminating ARP collisions with other physical LAN devices. Piloted successfully on staging-1 -> staging1, 2026-08-03.
license: MIT
source: original, from the staging host subnet migration pilot
---

# Jail Dedicated Subnet Migration

## Why

Jails using shared-IP aliasing directly on the host's physical LAN
interface (`vtnet0`) share that interface's broadcast domain with every
other device on the LAN. Confirmed real incidents (see
`bastille-jail-provisioning`'s ARP-collision section): a jail IP that
looked free by every host-local check turned out to already belong to a
different physical device or another host's jail on the same shared
subnet, causing silent networking failures that look exactly like a
config bug. Moving jails to their own private VNET subnet eliminates this
class of collision entirely — confirmed via `arp-scan` on the physical
LAN interface after migration: the jail's new IP produces zero response,
proving it's genuinely invisible outside the host.

## Prerequisites

- `if_bridge` and `pf` kernel modules must load (`kldload if_bridge`,
  `kldload pf` — both loaded cleanly on a stock FreeBSD 15.1 box with no
  prior config).
- A private subnet not overlapping the LAN (`192.168.0.0/24` here) or the
  Tailscale CGNAT range (`100.64.0.0/10`) — used `10.20.0.0/24`.
- Root on the jail host.

## Setup (host-level, once per host)

1. **Create the bridge, give the host itself an IP on it:**
   ```sh
   ifconfig bridge create name bastille0
   ifconfig bastille0 inet 10.20.0.1/24 up
   ```
2. **Persist across reboot:**
   ```sh
   sysrc cloned_interfaces+=bastille0
   sysrc ifconfig_bastille0='inet 10.20.0.1/24'
   sysrc gateway_enable=YES
   ```
3. **NAT for jail egress** (jails need this to reach the internet — the
   private subnet isn't routable from the LAN's own gateway):
   ```sh
   cat > /etc/pf.conf <<'EOF'
   ext_if = "vtnet0"
   jail_net = "10.20.0.0/24"

   nat on $ext_if from $jail_net to any -> ($ext_if)
   pass in
   pass out
   EOF
   ```
   **`sysrc gateway_enable=YES` alone is not enough** — it only persists
   `net.inet.ip.forwarding=1` for the *next* boot, it doesn't apply it to
   the live running kernel. Confirmed on venus: NAT looked correctly
   configured (`pfctl -s nat` showed the rule) but every jail's outbound
   traffic still timed out until `sysctl net.inet.ip.forwarding=1` was
   run live. Always run both, not just the `sysrc` persistence step.
   **The `pass in`/`pass out` rules are deliberately permissive** — this
   is a NAT-only ruleset, not a firewall. Don't add restrictive rules
   without separately verifying they don't break existing services
   (SSH/Tailscale) on the host.
4. **Enabling pf is a genuine remote-lockout risk** — even a permissive
   ruleset syntax error can default-deny. Test with `service pf onestart`
   first (bypasses the `pf_enable` rc.conf check, doesn't persist) and
   confirm the SSH session survives before persisting with `sysrc
   pf_enable=YES` + `service pf start`. This got flagged by this
   session's own permission classifier as needing explicit confirmation
   before proceeding — treat that as correct caution, not an obstacle to
   route around.

## Creating a VNET jail on the new subnet

```sh
bastille create -B -g 10.20.0.1 <name> <release> 10.20.0.X/24 bastille0
```
- `-B`/`--bridge` enables VNET with a bridge interface (not `-V`, which
  is for a *physical* interface).
- `-g` sets the jail's default gateway — must be the bridge's own IP
  (`10.20.0.1`), or the jail has no route out.
- **VNET jail names cannot contain `-` or `_`** (`[ERROR]: VNET jail
  names may not contain (-|_) characters.`) — bastille derives epair
  interface names from the jail name. A jail called `staging-1` under
  shared-IP mode had to become `staging1` under VNET.

Verify before trusting it:
```sh
bastille cmd <name> fetch -o /dev/null http://1.1.1.1   # raw IP, outbound NAT
bastille cmd <name> fetch -o /dev/null https://pkg.freebsd.org/  # DNS + outbound
ping -c2 10.20.0.X                                       # host -> jail, no NAT needed (same bridge)
arp-scan --interface=vtnet0 10.20.0.X                    # must show ZERO responses -- proves LAN isolation
```

**Bridge port showing `LEARNING` in `ifconfig bastille0` output is a red
herring, not a real blocker** — host-to-jail `ping` worked fine on venus
even while the member port's flags still showed `LEARNING` well past the
RSTP forward-delay window. Don't chase STP state as the cause of a
connectivity problem without first confirming with a direct `ping` from
the host that L2 forwarding is actually broken (it usually isn't).

**A fresh jail may inherit a broken `resolv.conf` from the host.** On a
host running `local_unbound_enable="YES"` (venus does, staging didn't),
new jails got `nameserver 127.0.0.1` copied into their `resolv.conf` --
which inside the jail points at the jail's *own* loopback, where nothing
is listening, not the host's unbound instance. Symptom: raw-IP `fetch`
still fails with "Transient resolver failure" even after outbound
NAT/forwarding is confirmed working, because `fetch` calls the resolver
even for IP literals. Fix: overwrite the jail's `resolv.conf` with public
DNS directly (`1.1.1.1` / `1.0.0.1`), don't assume the host's own
`/etc/resolv.conf` is safe to inherit as-is.

## Migrating an existing jail's content (not just spinning up empty)

Don't destroy-and-rebuild-from-scratch if the jail has real installed
state worth keeping (plugins, test data, DB content). Instead:

1. Create the new VNET jail per above, install the same stock stack
   (nginx, php-fpm, mariadb-client, etc.) matching the old jail's.
2. Copy the web root: `cp -a <old-jail-root>/usr/local/www/site/. <new-jail-root>/usr/local/www/site/`, then `chown -R www:www` on the destination.
3. **The database doesn't move** if it lives on the host's own local
   MariaDB (the common pattern on these hosts) — only the jail's IP
   changes, so `wp-config.php`'s `DB_HOST` (the host's own LAN IP)
   stays the same. What changes is the MySQL **grant**, which is scoped
   by the connecting IP: `CREATE USER IF NOT EXISTS
   '<dbuser>'@'<new-jail-ip>' IDENTIFIED BY '<password>'; GRANT ALL
   PRIVILEGES ON <dbname>.* TO '<dbuser>'@'<new-jail-ip>';`. Extract the
   existing DB password from the copied `wp-config.php` programmatically
   (`awk -F"'" '{print $4}'` on the `DB_PASSWORD` line, or similar) —
   **don't manually retype/copy-paste a password across terminal
   output**, a single mistyped character produces a misleading "Access
   denied (using password: YES)" error that looks like a grant/network
   problem but is actually just a wrong password. If that error shows up
   despite a seemingly-correct `CREATE USER`, the fastest fix is
   `ALTER USER '<user>'@'<ip>' IDENTIFIED BY '<password-read-fresh-from-file>';`
   rather than debugging network connectivity first.
4. Test host→jail DB connectivity is **not** NAT-hairpinned: since the
   host's MariaDB and the new jail are both local to the same host, and
   the bridge gives the jail a direct route to the host's own bridge IP
   (`10.20.0.1`) without going through the NAT'd `vtnet0` path at all,
   this generally works cleanly — confirmed connections to the host's
   LAN IP from a jail on the private subnet arrive with the jail's real
   source IP intact (not NAT-masqueraded), because the kernel routes
   host-destined traffic locally rather than sending it out and back
   through the NAT rule. Don't assume a hairpin NAT problem without
   testing first — it added an unnecessary detour.
5. Verify the site fully — **a page-cache HIT can mislead you into
   thinking DB connectivity works when it doesn't**, since a cached page
   never touches the database. Purge any cache first, then confirm both
   a fresh MISS render (real DB read) and a subsequent HIT (served from
   cache) both work, plus a `wp-cli` command that requires DB access
   (`wp option get siteurl`) as an unambiguous connectivity check.

## Cutover

**Real outage, 2026-08-03: verifying the new jail by curling its IP directly
with a `Host:` header does NOT prove the live site works**, because that
bypasses the actual production path. The real path is NPM → the jail host's
own nginx (a per-domain vhost with `proxy_pass` hardcoded to the jail's IP)
→ the jail. Stopping the old jail after only an IP-direct check leaves that
vhost still pointing at a now-dead IP — instant, silent outage for every
domain migrated this way (hit 5 domains across 2 hosts, funky and venus,
before being caught by the user noticing timeouts, not by this process).
`migrate-jail-to-subnet.sh` now rewires the host-nginx vhost automatically
as its own step (finds the `.conf` referencing the old jail's IP via
`bastille list`, `sed`s it to the new IP, `nginx -t` then `-s reload`,
aborts and reverts from `.bak` if the syntax check fails) — **but always
verify through `curl -H 'Host: <domain>' -H 'X-Forwarded-Proto: https'
http://127.0.0.1/` on the jail host itself** (exercises the real vhost path)
before trusting a migration, not a direct-by-IP request.

1. Verify the new jail through the real path (see above), not by IP alone.
2. Stop (don't destroy) the old jail — keep it as a rollback safety net
   until the new one is fully proven in real use. Safe to do now that the
   vhost itself has already been repointed.
3. Destroy the old jail only once genuinely confident — this session left
   it stopped rather than destroyed on the first migration, as the
   controlled/cautious default.
4. Update any skill/doc/memory that references the old jail name or IP
   (grep for both across `agent-skills` and this repo's `docs/`) — easy
   to miss and leaves stale instructions for the next delegate.
5. **After any batch of migrations, sweep for stragglers**: `grep -rl
   'proxy_pass.*192\.168\.' /usr/local/etc/nginx/sites-*/` on the host
   catches any vhost still pointing at a shared-subnet IP whose jail has
   since been stopped — the fastest way to confirm nothing was missed.

## Check disk space BEFORE creating any jail, every time

**Do not assume "blank slate, plenty of room" holds on every host** just
because it held on the pilot host. Confirmed real incident on the second
host attempted (venus, 2026-08-03): creating 3 new full-stack jails
(~200MB retained footprint each after packages) pushed an already-96%-full
ZFS pool over the edge and **crashed the shared MariaDB instance**, taking
down every site on the host (a real, if brief, production outage) --
this exact failure mode had already been documented once before on this
same host from an unrelated earlier incident, and should have been
checked proactively rather than triggered again.

**Before creating even one new jail**, run `zfs list zroot/ROOT/default -o
avail` (or `df -h` if not ZFS) and treat anything under a few GB as a hard
stop -- clean up first (`bastille pkg <jail> clean -ay` across existing
jails reclaims real space safely, it's just downloaded package cache) or
get the capacity issue resolved before proceeding. Re-check after every
1-2 jail creations during a bulk rollout, don't just check once at the
start and assume it holds for the whole batch.

If MariaDB (or any host-shared service) does go down from disk pressure:
the rc.d script name may not match the obvious guess (`mysql-server`, not
`mysql` or `mariadb`, was the actual name on venus) -- check `ls
/usr/local/etc/rc.d/` rather than guessing. After restarting, verify with
actual `curl` tests against multiple real sites (not just "is the process
running") before considering it resolved.

## Rollout scope discipline

This was explicitly piloted on **one jail on one host** before being
considered for the rest of the fleet — treat "prove it on a single
low-stakes target first" as the right default for any jail-networking
architecture change, not just this one. Don't extend to production jails
or other hosts without a separate, explicit go-ahead per host.