feat: document the dedicated-subnet jail migration pattern, update staging jail identity

This commit is contained in:
2026-08-03 13:17:57 +02:00
parent c868eae35f
commit eb7c63d1f9
3 changed files with 169 additions and 6 deletions
+6 -1
View File
@@ -55,9 +55,14 @@ than assuming the delegate can fetch it itself.
the same bootstrap/wiring file; use self-registration instead. Real the same bootstrap/wiring file; use self-registration instead. Real
incident from the iWP Cache build (lost updates, twice). incident from the iWP Cache build (lost updates, twice).
- `wordpress-plugin-staging-verification` -- use the persistent - `wordpress-plugin-staging-verification` -- use the persistent
`staging-1` jail (real-world plugin set left active) for verifying new `staging1` jail (real-world plugin set left active) for verifying new
plugins, not a disposable clean-room jail; includes the full plugins, not a disposable clean-room jail; includes the full
cache-plugin verification checklist and cosmetic-vs-real error gotchas. cache-plugin verification checklist and cosmetic-vs-real error gotchas.
- `jail-dedicated-subnet-migration` -- move jails from shared-IP-on-LAN
aliasing to a dedicated private VNET subnet, eliminating ARP collisions
with other physical LAN devices. Full bridge/NAT/pf setup, VNET jail
creation syntax, and a content-preserving migration procedure (DB grant
re-scoping, hairpin-NAT non-issue, cache-HIT-hides-DB-failures gotcha).
## Provenance ## Provenance
@@ -0,0 +1,154 @@
---
name: jail-dedicated-subnet-migration
description: Move Bastille jails from shared-IP-aliasing on the LAN interface to a dedicated private VNET subnet, eliminating ARP collisions with other physical LAN devices. Piloted successfully on staging-1 -> staging1, 2026-08-03.
license: MIT
source: original, from the staging host subnet migration pilot
---
# Jail Dedicated Subnet Migration
## Why
Jails using shared-IP aliasing directly on the host's physical LAN
interface (`vtnet0`) share that interface's broadcast domain with every
other device on the LAN. Confirmed real incidents (see
`bastille-jail-provisioning`'s ARP-collision section): a jail IP that
looked free by every host-local check turned out to already belong to a
different physical device or another host's jail on the same shared
subnet, causing silent networking failures that look exactly like a
config bug. Moving jails to their own private VNET subnet eliminates this
class of collision entirely — confirmed via `arp-scan` on the physical
LAN interface after migration: the jail's new IP produces zero response,
proving it's genuinely invisible outside the host.
## Prerequisites
- `if_bridge` and `pf` kernel modules must load (`kldload if_bridge`,
`kldload pf` — both loaded cleanly on a stock FreeBSD 15.1 box with no
prior config).
- A private subnet not overlapping the LAN (`192.168.0.0/24` here) or the
Tailscale CGNAT range (`100.64.0.0/10`) — used `10.20.0.0/24`.
- Root on the jail host.
## Setup (host-level, once per host)
1. **Create the bridge, give the host itself an IP on it:**
```sh
ifconfig bridge create name bastille0
ifconfig bastille0 inet 10.20.0.1/24 up
```
2. **Persist across reboot:**
```sh
sysrc cloned_interfaces+=bastille0
sysrc ifconfig_bastille0='inet 10.20.0.1/24'
sysrc gateway_enable=YES
```
3. **NAT for jail egress** (jails need this to reach the internet — the
private subnet isn't routable from the LAN's own gateway):
```sh
cat > /etc/pf.conf <<'EOF'
ext_if = "vtnet0"
jail_net = "10.20.0.0/24"
nat on $ext_if from $jail_net to any -> ($ext_if)
pass in
pass out
EOF
```
**The `pass in`/`pass out` rules are deliberately permissive** — this
is a NAT-only ruleset, not a firewall. Don't add restrictive rules
without separately verifying they don't break existing services
(SSH/Tailscale) on the host.
4. **Enabling pf is a genuine remote-lockout risk** — even a permissive
ruleset syntax error can default-deny. Test with `service pf onestart`
first (bypasses the `pf_enable` rc.conf check, doesn't persist) and
confirm the SSH session survives before persisting with `sysrc
pf_enable=YES` + `service pf start`. This got flagged by this
session's own permission classifier as needing explicit confirmation
before proceeding — treat that as correct caution, not an obstacle to
route around.
## Creating a VNET jail on the new subnet
```sh
bastille create -B -g 10.20.0.1 <name> <release> 10.20.0.X/24 bastille0
```
- `-B`/`--bridge` enables VNET with a bridge interface (not `-V`, which
is for a *physical* interface).
- `-g` sets the jail's default gateway — must be the bridge's own IP
(`10.20.0.1`), or the jail has no route out.
- **VNET jail names cannot contain `-` or `_`** (`[ERROR]: VNET jail
names may not contain (-|_) characters.`) — bastille derives epair
interface names from the jail name. A jail called `staging-1` under
shared-IP mode had to become `staging1` under VNET.
Verify before trusting it:
```sh
bastille cmd <name> fetch -o /dev/null http://1.1.1.1 # raw IP, outbound NAT
bastille cmd <name> fetch -o /dev/null https://pkg.freebsd.org/ # DNS + outbound
ping -c2 10.20.0.X # host -> jail, no NAT needed (same bridge)
arp-scan --interface=vtnet0 10.20.0.X # must show ZERO responses -- proves LAN isolation
```
## Migrating an existing jail's content (not just spinning up empty)
Don't destroy-and-rebuild-from-scratch if the jail has real installed
state worth keeping (plugins, test data, DB content). Instead:
1. Create the new VNET jail per above, install the same stock stack
(nginx, php-fpm, mariadb-client, etc.) matching the old jail's.
2. Copy the web root: `cp -a <old-jail-root>/usr/local/www/site/. <new-jail-root>/usr/local/www/site/`, then `chown -R www:www` on the destination.
3. **The database doesn't move** if it lives on the host's own local
MariaDB (the common pattern on these hosts) — only the jail's IP
changes, so `wp-config.php`'s `DB_HOST` (the host's own LAN IP)
stays the same. What changes is the MySQL **grant**, which is scoped
by the connecting IP: `CREATE USER IF NOT EXISTS
'<dbuser>'@'<new-jail-ip>' IDENTIFIED BY '<password>'; GRANT ALL
PRIVILEGES ON <dbname>.* TO '<dbuser>'@'<new-jail-ip>';`. Extract the
existing DB password from the copied `wp-config.php` programmatically
(`awk -F"'" '{print $4}'` on the `DB_PASSWORD` line, or similar) —
**don't manually retype/copy-paste a password across terminal
output**, a single mistyped character produces a misleading "Access
denied (using password: YES)" error that looks like a grant/network
problem but is actually just a wrong password. If that error shows up
despite a seemingly-correct `CREATE USER`, the fastest fix is
`ALTER USER '<user>'@'<ip>' IDENTIFIED BY '<password-read-fresh-from-file>';`
rather than debugging network connectivity first.
4. Test host→jail DB connectivity is **not** NAT-hairpinned: since the
host's MariaDB and the new jail are both local to the same host, and
the bridge gives the jail a direct route to the host's own bridge IP
(`10.20.0.1`) without going through the NAT'd `vtnet0` path at all,
this generally works cleanly — confirmed connections to the host's
LAN IP from a jail on the private subnet arrive with the jail's real
source IP intact (not NAT-masqueraded), because the kernel routes
host-destined traffic locally rather than sending it out and back
through the NAT rule. Don't assume a hairpin NAT problem without
testing first — it added an unnecessary detour.
5. Verify the site fully — **a page-cache HIT can mislead you into
thinking DB connectivity works when it doesn't**, since a cached page
never touches the database. Purge any cache first, then confirm both
a fresh MISS render (real DB read) and a subsequent HIT (served from
cache) both work, plus a `wp-cli` command that requires DB access
(`wp option get siteurl`) as an unambiguous connectivity check.
## Cutover
1. Stop (don't destroy) the old jail — keep it as a rollback safety net
until the new one is fully proven in real use.
2. If a host-level reverse-proxy vhost points at the old jail's IP,
update it. (Not needed if the domain was never wired to a public vhost
yet — check first with `grep -rl <old-ip> /usr/local/etc/nginx/`.)
3. Destroy the old jail only once genuinely confident — this session left
it stopped rather than destroyed on the first migration, as the
controlled/cautious default.
4. Update any skill/doc/memory that references the old jail name or IP
(grep for both across `agent-skills` and this repo's `docs/`) — easy
to miss and leaves stale instructions for the next delegate.
## Rollout scope discipline
This was explicitly piloted on **one jail on one host** before being
considered for the rest of the fleet — treat "prove it on a single
low-stakes target first" as the right default for any jail-networking
architecture change, not just this one. Don't extend to production jails
or other hosts without a separate, explicit go-ahead per host.
@@ -9,15 +9,19 @@ source: original, from the iWP Cache staging verification (2026-08-02)
## Use the persistent staging jail, not a throwaway one ## Use the persistent staging jail, not a throwaway one
`staging-1` on the `staging` host (Tailscale/SSH: `100.104.61.54`, jail IP `staging1` on the `staging` host (Tailscale/SSH: `100.104.61.54`, jail IP
`192.168.0.187`, intended domain `staging.palmasolutions.net`) is a `10.20.0.11` — migrated 2026-08-03 to a dedicated private VNET subnet, see
[[jail_dedicated_subnet_migration]], was `staging-1` at `192.168.0.187`
before that — intended domain `staging.palmasolutions.net`) is a
standing verification environment — deploy new/updated plugins there and standing verification environment — deploy new/updated plugins there and
**leave them installed and active** rather than spinning up a fresh **leave them installed and active** rather than spinning up a fresh
disposable jail per plugin and tearing it down after. disposable jail per plugin and tearing it down after. Note the name has
no hyphen (`staging1`, not `staging-1`) — VNET jail names on this host
can't contain `-`/`_`.
**Why:** a clean-room jail with only the plugin under test installed **Why:** a clean-room jail with only the plugin under test installed
proves the plugin works in isolation, but says nothing about how it proves the plugin works in isolation, but says nothing about how it
behaves alongside the plugins real sites actually run. `staging-1` ships behaves alongside the plugins real sites actually run. `staging1` ships
with a representative baseline already active: WooCommerce, Classic with a representative baseline already active: WooCommerce, Classic
Editor, Contact Form 7, Yoast SEO (`wordpress-seo` is the correct wp.org Editor, Contact Form 7, Yoast SEO (`wordpress-seo` is the correct wp.org
slug — `yoast-seo` doesn't exist), Wordfence, LiteSpeed Cache (a slug — `yoast-seo` doesn't exist), Wordfence, LiteSpeed Cache (a
@@ -27,7 +31,7 @@ Each new plugin verified here adds to that baseline instead of replacing
it, so the set of "things this plugin has been proven compatible with" it, so the set of "things this plugin has been proven compatible with"
only grows over time. only grows over time.
## Setup, once per jail (already done for `staging-1`) ## Setup, once per jail (already done for `staging1`)
Standard stock stack per [[bastille_jail_provisioning]] (nginx, php-fpm, Standard stock stack per [[bastille_jail_provisioning]] (nginx, php-fpm,
MariaDB-client pointed at the host's local MariaDB, Valkey + MariaDB-client pointed at the host's local MariaDB, Valkey +