feat: document the dedicated-subnet jail migration pattern, update staging jail identity
This commit is contained in:
@@ -55,9 +55,14 @@ than assuming the delegate can fetch it itself.
|
||||
the same bootstrap/wiring file; use self-registration instead. Real
|
||||
incident from the iWP Cache build (lost updates, twice).
|
||||
- `wordpress-plugin-staging-verification` -- use the persistent
|
||||
`staging-1` jail (real-world plugin set left active) for verifying new
|
||||
`staging1` jail (real-world plugin set left active) for verifying new
|
||||
plugins, not a disposable clean-room jail; includes the full
|
||||
cache-plugin verification checklist and cosmetic-vs-real error gotchas.
|
||||
- `jail-dedicated-subnet-migration` -- move jails from shared-IP-on-LAN
|
||||
aliasing to a dedicated private VNET subnet, eliminating ARP collisions
|
||||
with other physical LAN devices. Full bridge/NAT/pf setup, VNET jail
|
||||
creation syntax, and a content-preserving migration procedure (DB grant
|
||||
re-scoping, hairpin-NAT non-issue, cache-HIT-hides-DB-failures gotcha).
|
||||
|
||||
## Provenance
|
||||
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
---
|
||||
name: jail-dedicated-subnet-migration
|
||||
description: Move Bastille jails from shared-IP-aliasing on the LAN interface to a dedicated private VNET subnet, eliminating ARP collisions with other physical LAN devices. Piloted successfully on staging-1 -> staging1, 2026-08-03.
|
||||
license: MIT
|
||||
source: original, from the staging host subnet migration pilot
|
||||
---
|
||||
|
||||
# Jail Dedicated Subnet Migration
|
||||
|
||||
## Why
|
||||
|
||||
Jails using shared-IP aliasing directly on the host's physical LAN
|
||||
interface (`vtnet0`) share that interface's broadcast domain with every
|
||||
other device on the LAN. Confirmed real incidents (see
|
||||
`bastille-jail-provisioning`'s ARP-collision section): a jail IP that
|
||||
looked free by every host-local check turned out to already belong to a
|
||||
different physical device or another host's jail on the same shared
|
||||
subnet, causing silent networking failures that look exactly like a
|
||||
config bug. Moving jails to their own private VNET subnet eliminates this
|
||||
class of collision entirely — confirmed via `arp-scan` on the physical
|
||||
LAN interface after migration: the jail's new IP produces zero response,
|
||||
proving it's genuinely invisible outside the host.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- `if_bridge` and `pf` kernel modules must load (`kldload if_bridge`,
|
||||
`kldload pf` — both loaded cleanly on a stock FreeBSD 15.1 box with no
|
||||
prior config).
|
||||
- A private subnet not overlapping the LAN (`192.168.0.0/24` here) or the
|
||||
Tailscale CGNAT range (`100.64.0.0/10`) — used `10.20.0.0/24`.
|
||||
- Root on the jail host.
|
||||
|
||||
## Setup (host-level, once per host)
|
||||
|
||||
1. **Create the bridge, give the host itself an IP on it:**
|
||||
```sh
|
||||
ifconfig bridge create name bastille0
|
||||
ifconfig bastille0 inet 10.20.0.1/24 up
|
||||
```
|
||||
2. **Persist across reboot:**
|
||||
```sh
|
||||
sysrc cloned_interfaces+=bastille0
|
||||
sysrc ifconfig_bastille0='inet 10.20.0.1/24'
|
||||
sysrc gateway_enable=YES
|
||||
```
|
||||
3. **NAT for jail egress** (jails need this to reach the internet — the
|
||||
private subnet isn't routable from the LAN's own gateway):
|
||||
```sh
|
||||
cat > /etc/pf.conf <<'EOF'
|
||||
ext_if = "vtnet0"
|
||||
jail_net = "10.20.0.0/24"
|
||||
|
||||
nat on $ext_if from $jail_net to any -> ($ext_if)
|
||||
pass in
|
||||
pass out
|
||||
EOF
|
||||
```
|
||||
**The `pass in`/`pass out` rules are deliberately permissive** — this
|
||||
is a NAT-only ruleset, not a firewall. Don't add restrictive rules
|
||||
without separately verifying they don't break existing services
|
||||
(SSH/Tailscale) on the host.
|
||||
4. **Enabling pf is a genuine remote-lockout risk** — even a permissive
|
||||
ruleset syntax error can default-deny. Test with `service pf onestart`
|
||||
first (bypasses the `pf_enable` rc.conf check, doesn't persist) and
|
||||
confirm the SSH session survives before persisting with `sysrc
|
||||
pf_enable=YES` + `service pf start`. This got flagged by this
|
||||
session's own permission classifier as needing explicit confirmation
|
||||
before proceeding — treat that as correct caution, not an obstacle to
|
||||
route around.
|
||||
|
||||
## Creating a VNET jail on the new subnet
|
||||
|
||||
```sh
|
||||
bastille create -B -g 10.20.0.1 <name> <release> 10.20.0.X/24 bastille0
|
||||
```
|
||||
- `-B`/`--bridge` enables VNET with a bridge interface (not `-V`, which
|
||||
is for a *physical* interface).
|
||||
- `-g` sets the jail's default gateway — must be the bridge's own IP
|
||||
(`10.20.0.1`), or the jail has no route out.
|
||||
- **VNET jail names cannot contain `-` or `_`** (`[ERROR]: VNET jail
|
||||
names may not contain (-|_) characters.`) — bastille derives epair
|
||||
interface names from the jail name. A jail called `staging-1` under
|
||||
shared-IP mode had to become `staging1` under VNET.
|
||||
|
||||
Verify before trusting it:
|
||||
```sh
|
||||
bastille cmd <name> fetch -o /dev/null http://1.1.1.1 # raw IP, outbound NAT
|
||||
bastille cmd <name> fetch -o /dev/null https://pkg.freebsd.org/ # DNS + outbound
|
||||
ping -c2 10.20.0.X # host -> jail, no NAT needed (same bridge)
|
||||
arp-scan --interface=vtnet0 10.20.0.X # must show ZERO responses -- proves LAN isolation
|
||||
```
|
||||
|
||||
## Migrating an existing jail's content (not just spinning up empty)
|
||||
|
||||
Don't destroy-and-rebuild-from-scratch if the jail has real installed
|
||||
state worth keeping (plugins, test data, DB content). Instead:
|
||||
|
||||
1. Create the new VNET jail per above, install the same stock stack
|
||||
(nginx, php-fpm, mariadb-client, etc.) matching the old jail's.
|
||||
2. Copy the web root: `cp -a <old-jail-root>/usr/local/www/site/. <new-jail-root>/usr/local/www/site/`, then `chown -R www:www` on the destination.
|
||||
3. **The database doesn't move** if it lives on the host's own local
|
||||
MariaDB (the common pattern on these hosts) — only the jail's IP
|
||||
changes, so `wp-config.php`'s `DB_HOST` (the host's own LAN IP)
|
||||
stays the same. What changes is the MySQL **grant**, which is scoped
|
||||
by the connecting IP: `CREATE USER IF NOT EXISTS
|
||||
'<dbuser>'@'<new-jail-ip>' IDENTIFIED BY '<password>'; GRANT ALL
|
||||
PRIVILEGES ON <dbname>.* TO '<dbuser>'@'<new-jail-ip>';`. Extract the
|
||||
existing DB password from the copied `wp-config.php` programmatically
|
||||
(`awk -F"'" '{print $4}'` on the `DB_PASSWORD` line, or similar) —
|
||||
**don't manually retype/copy-paste a password across terminal
|
||||
output**, a single mistyped character produces a misleading "Access
|
||||
denied (using password: YES)" error that looks like a grant/network
|
||||
problem but is actually just a wrong password. If that error shows up
|
||||
despite a seemingly-correct `CREATE USER`, the fastest fix is
|
||||
`ALTER USER '<user>'@'<ip>' IDENTIFIED BY '<password-read-fresh-from-file>';`
|
||||
rather than debugging network connectivity first.
|
||||
4. Test host→jail DB connectivity is **not** NAT-hairpinned: since the
|
||||
host's MariaDB and the new jail are both local to the same host, and
|
||||
the bridge gives the jail a direct route to the host's own bridge IP
|
||||
(`10.20.0.1`) without going through the NAT'd `vtnet0` path at all,
|
||||
this generally works cleanly — confirmed connections to the host's
|
||||
LAN IP from a jail on the private subnet arrive with the jail's real
|
||||
source IP intact (not NAT-masqueraded), because the kernel routes
|
||||
host-destined traffic locally rather than sending it out and back
|
||||
through the NAT rule. Don't assume a hairpin NAT problem without
|
||||
testing first — it added an unnecessary detour.
|
||||
5. Verify the site fully — **a page-cache HIT can mislead you into
|
||||
thinking DB connectivity works when it doesn't**, since a cached page
|
||||
never touches the database. Purge any cache first, then confirm both
|
||||
a fresh MISS render (real DB read) and a subsequent HIT (served from
|
||||
cache) both work, plus a `wp-cli` command that requires DB access
|
||||
(`wp option get siteurl`) as an unambiguous connectivity check.
|
||||
|
||||
## Cutover
|
||||
|
||||
1. Stop (don't destroy) the old jail — keep it as a rollback safety net
|
||||
until the new one is fully proven in real use.
|
||||
2. If a host-level reverse-proxy vhost points at the old jail's IP,
|
||||
update it. (Not needed if the domain was never wired to a public vhost
|
||||
yet — check first with `grep -rl <old-ip> /usr/local/etc/nginx/`.)
|
||||
3. Destroy the old jail only once genuinely confident — this session left
|
||||
it stopped rather than destroyed on the first migration, as the
|
||||
controlled/cautious default.
|
||||
4. Update any skill/doc/memory that references the old jail name or IP
|
||||
(grep for both across `agent-skills` and this repo's `docs/`) — easy
|
||||
to miss and leaves stale instructions for the next delegate.
|
||||
|
||||
## Rollout scope discipline
|
||||
|
||||
This was explicitly piloted on **one jail on one host** before being
|
||||
considered for the rest of the fleet — treat "prove it on a single
|
||||
low-stakes target first" as the right default for any jail-networking
|
||||
architecture change, not just this one. Don't extend to production jails
|
||||
or other hosts without a separate, explicit go-ahead per host.
|
||||
@@ -9,15 +9,19 @@ source: original, from the iWP Cache staging verification (2026-08-02)
|
||||
|
||||
## Use the persistent staging jail, not a throwaway one
|
||||
|
||||
`staging-1` on the `staging` host (Tailscale/SSH: `100.104.61.54`, jail IP
|
||||
`192.168.0.187`, intended domain `staging.palmasolutions.net`) is a
|
||||
`staging1` on the `staging` host (Tailscale/SSH: `100.104.61.54`, jail IP
|
||||
`10.20.0.11` — migrated 2026-08-03 to a dedicated private VNET subnet, see
|
||||
[[jail_dedicated_subnet_migration]], was `staging-1` at `192.168.0.187`
|
||||
before that — intended domain `staging.palmasolutions.net`) is a
|
||||
standing verification environment — deploy new/updated plugins there and
|
||||
**leave them installed and active** rather than spinning up a fresh
|
||||
disposable jail per plugin and tearing it down after.
|
||||
disposable jail per plugin and tearing it down after. Note the name has
|
||||
no hyphen (`staging1`, not `staging-1`) — VNET jail names on this host
|
||||
can't contain `-`/`_`.
|
||||
|
||||
**Why:** a clean-room jail with only the plugin under test installed
|
||||
proves the plugin works in isolation, but says nothing about how it
|
||||
behaves alongside the plugins real sites actually run. `staging-1` ships
|
||||
behaves alongside the plugins real sites actually run. `staging1` ships
|
||||
with a representative baseline already active: WooCommerce, Classic
|
||||
Editor, Contact Form 7, Yoast SEO (`wordpress-seo` is the correct wp.org
|
||||
slug — `yoast-seo` doesn't exist), Wordfence, LiteSpeed Cache (a
|
||||
@@ -27,7 +31,7 @@ Each new plugin verified here adds to that baseline instead of replacing
|
||||
it, so the set of "things this plugin has been proven compatible with"
|
||||
only grows over time.
|
||||
|
||||
## Setup, once per jail (already done for `staging-1`)
|
||||
## Setup, once per jail (already done for `staging1`)
|
||||
|
||||
Standard stock stack per [[bastille_jail_provisioning]] (nginx, php-fpm,
|
||||
MariaDB-client pointed at the host's local MariaDB, Valkey +
|
||||
|
||||
Reference in New Issue
Block a user