feat: document the dedicated-subnet jail migration pattern, update staging jail identity
This commit is contained in:
@@ -55,9 +55,14 @@ than assuming the delegate can fetch it itself.
|
|||||||
the same bootstrap/wiring file; use self-registration instead. Real
|
the same bootstrap/wiring file; use self-registration instead. Real
|
||||||
incident from the iWP Cache build (lost updates, twice).
|
incident from the iWP Cache build (lost updates, twice).
|
||||||
- `wordpress-plugin-staging-verification` -- use the persistent
|
- `wordpress-plugin-staging-verification` -- use the persistent
|
||||||
`staging-1` jail (real-world plugin set left active) for verifying new
|
`staging1` jail (real-world plugin set left active) for verifying new
|
||||||
plugins, not a disposable clean-room jail; includes the full
|
plugins, not a disposable clean-room jail; includes the full
|
||||||
cache-plugin verification checklist and cosmetic-vs-real error gotchas.
|
cache-plugin verification checklist and cosmetic-vs-real error gotchas.
|
||||||
|
- `jail-dedicated-subnet-migration` -- move jails from shared-IP-on-LAN
|
||||||
|
aliasing to a dedicated private VNET subnet, eliminating ARP collisions
|
||||||
|
with other physical LAN devices. Full bridge/NAT/pf setup, VNET jail
|
||||||
|
creation syntax, and a content-preserving migration procedure (DB grant
|
||||||
|
re-scoping, hairpin-NAT non-issue, cache-HIT-hides-DB-failures gotcha).
|
||||||
|
|
||||||
## Provenance
|
## Provenance
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,154 @@
|
|||||||
|
---
|
||||||
|
name: jail-dedicated-subnet-migration
|
||||||
|
description: Move Bastille jails from shared-IP-aliasing on the LAN interface to a dedicated private VNET subnet, eliminating ARP collisions with other physical LAN devices. Piloted successfully on staging-1 -> staging1, 2026-08-03.
|
||||||
|
license: MIT
|
||||||
|
source: original, from the staging host subnet migration pilot
|
||||||
|
---
|
||||||
|
|
||||||
|
# Jail Dedicated Subnet Migration
|
||||||
|
|
||||||
|
## Why
|
||||||
|
|
||||||
|
Jails using shared-IP aliasing directly on the host's physical LAN
|
||||||
|
interface (`vtnet0`) share that interface's broadcast domain with every
|
||||||
|
other device on the LAN. Confirmed real incidents (see
|
||||||
|
`bastille-jail-provisioning`'s ARP-collision section): a jail IP that
|
||||||
|
looked free by every host-local check turned out to already belong to a
|
||||||
|
different physical device or another host's jail on the same shared
|
||||||
|
subnet, causing silent networking failures that look exactly like a
|
||||||
|
config bug. Moving jails to their own private VNET subnet eliminates this
|
||||||
|
class of collision entirely — confirmed via `arp-scan` on the physical
|
||||||
|
LAN interface after migration: the jail's new IP produces zero response,
|
||||||
|
proving it's genuinely invisible outside the host.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- `if_bridge` and `pf` kernel modules must load (`kldload if_bridge`,
|
||||||
|
`kldload pf` — both loaded cleanly on a stock FreeBSD 15.1 box with no
|
||||||
|
prior config).
|
||||||
|
- A private subnet not overlapping the LAN (`192.168.0.0/24` here) or the
|
||||||
|
Tailscale CGNAT range (`100.64.0.0/10`) — used `10.20.0.0/24`.
|
||||||
|
- Root on the jail host.
|
||||||
|
|
||||||
|
## Setup (host-level, once per host)
|
||||||
|
|
||||||
|
1. **Create the bridge, give the host itself an IP on it:**
|
||||||
|
```sh
|
||||||
|
ifconfig bridge create name bastille0
|
||||||
|
ifconfig bastille0 inet 10.20.0.1/24 up
|
||||||
|
```
|
||||||
|
2. **Persist across reboot:**
|
||||||
|
```sh
|
||||||
|
sysrc cloned_interfaces+=bastille0
|
||||||
|
sysrc ifconfig_bastille0='inet 10.20.0.1/24'
|
||||||
|
sysrc gateway_enable=YES
|
||||||
|
```
|
||||||
|
3. **NAT for jail egress** (jails need this to reach the internet — the
|
||||||
|
private subnet isn't routable from the LAN's own gateway):
|
||||||
|
```sh
|
||||||
|
cat > /etc/pf.conf <<'EOF'
|
||||||
|
ext_if = "vtnet0"
|
||||||
|
jail_net = "10.20.0.0/24"
|
||||||
|
|
||||||
|
nat on $ext_if from $jail_net to any -> ($ext_if)
|
||||||
|
pass in
|
||||||
|
pass out
|
||||||
|
EOF
|
||||||
|
```
|
||||||
|
**The `pass in`/`pass out` rules are deliberately permissive** — this
|
||||||
|
is a NAT-only ruleset, not a firewall. Don't add restrictive rules
|
||||||
|
without separately verifying they don't break existing services
|
||||||
|
(SSH/Tailscale) on the host.
|
||||||
|
4. **Enabling pf is a genuine remote-lockout risk** — even a permissive
|
||||||
|
ruleset syntax error can default-deny. Test with `service pf onestart`
|
||||||
|
first (bypasses the `pf_enable` rc.conf check, doesn't persist) and
|
||||||
|
confirm the SSH session survives before persisting with `sysrc
|
||||||
|
pf_enable=YES` + `service pf start`. This got flagged by this
|
||||||
|
session's own permission classifier as needing explicit confirmation
|
||||||
|
before proceeding — treat that as correct caution, not an obstacle to
|
||||||
|
route around.
|
||||||
|
|
||||||
|
## Creating a VNET jail on the new subnet
|
||||||
|
|
||||||
|
```sh
|
||||||
|
bastille create -B -g 10.20.0.1 <name> <release> 10.20.0.X/24 bastille0
|
||||||
|
```
|
||||||
|
- `-B`/`--bridge` enables VNET with a bridge interface (not `-V`, which
|
||||||
|
is for a *physical* interface).
|
||||||
|
- `-g` sets the jail's default gateway — must be the bridge's own IP
|
||||||
|
(`10.20.0.1`), or the jail has no route out.
|
||||||
|
- **VNET jail names cannot contain `-` or `_`** (`[ERROR]: VNET jail
|
||||||
|
names may not contain (-|_) characters.`) — bastille derives epair
|
||||||
|
interface names from the jail name. A jail called `staging-1` under
|
||||||
|
shared-IP mode had to become `staging1` under VNET.
|
||||||
|
|
||||||
|
Verify before trusting it:
|
||||||
|
```sh
|
||||||
|
bastille cmd <name> fetch -o /dev/null http://1.1.1.1 # raw IP, outbound NAT
|
||||||
|
bastille cmd <name> fetch -o /dev/null https://pkg.freebsd.org/ # DNS + outbound
|
||||||
|
ping -c2 10.20.0.X # host -> jail, no NAT needed (same bridge)
|
||||||
|
arp-scan --interface=vtnet0 10.20.0.X # must show ZERO responses -- proves LAN isolation
|
||||||
|
```
|
||||||
|
|
||||||
|
## Migrating an existing jail's content (not just spinning up empty)
|
||||||
|
|
||||||
|
Don't destroy-and-rebuild-from-scratch if the jail has real installed
|
||||||
|
state worth keeping (plugins, test data, DB content). Instead:
|
||||||
|
|
||||||
|
1. Create the new VNET jail per above, install the same stock stack
|
||||||
|
(nginx, php-fpm, mariadb-client, etc.) matching the old jail's.
|
||||||
|
2. Copy the web root: `cp -a <old-jail-root>/usr/local/www/site/. <new-jail-root>/usr/local/www/site/`, then `chown -R www:www` on the destination.
|
||||||
|
3. **The database doesn't move** if it lives on the host's own local
|
||||||
|
MariaDB (the common pattern on these hosts) — only the jail's IP
|
||||||
|
changes, so `wp-config.php`'s `DB_HOST` (the host's own LAN IP)
|
||||||
|
stays the same. What changes is the MySQL **grant**, which is scoped
|
||||||
|
by the connecting IP: `CREATE USER IF NOT EXISTS
|
||||||
|
'<dbuser>'@'<new-jail-ip>' IDENTIFIED BY '<password>'; GRANT ALL
|
||||||
|
PRIVILEGES ON <dbname>.* TO '<dbuser>'@'<new-jail-ip>';`. Extract the
|
||||||
|
existing DB password from the copied `wp-config.php` programmatically
|
||||||
|
(`awk -F"'" '{print $4}'` on the `DB_PASSWORD` line, or similar) —
|
||||||
|
**don't manually retype/copy-paste a password across terminal
|
||||||
|
output**, a single mistyped character produces a misleading "Access
|
||||||
|
denied (using password: YES)" error that looks like a grant/network
|
||||||
|
problem but is actually just a wrong password. If that error shows up
|
||||||
|
despite a seemingly-correct `CREATE USER`, the fastest fix is
|
||||||
|
`ALTER USER '<user>'@'<ip>' IDENTIFIED BY '<password-read-fresh-from-file>';`
|
||||||
|
rather than debugging network connectivity first.
|
||||||
|
4. Test host→jail DB connectivity is **not** NAT-hairpinned: since the
|
||||||
|
host's MariaDB and the new jail are both local to the same host, and
|
||||||
|
the bridge gives the jail a direct route to the host's own bridge IP
|
||||||
|
(`10.20.0.1`) without going through the NAT'd `vtnet0` path at all,
|
||||||
|
this generally works cleanly — confirmed connections to the host's
|
||||||
|
LAN IP from a jail on the private subnet arrive with the jail's real
|
||||||
|
source IP intact (not NAT-masqueraded), because the kernel routes
|
||||||
|
host-destined traffic locally rather than sending it out and back
|
||||||
|
through the NAT rule. Don't assume a hairpin NAT problem without
|
||||||
|
testing first — it added an unnecessary detour.
|
||||||
|
5. Verify the site fully — **a page-cache HIT can mislead you into
|
||||||
|
thinking DB connectivity works when it doesn't**, since a cached page
|
||||||
|
never touches the database. Purge any cache first, then confirm both
|
||||||
|
a fresh MISS render (real DB read) and a subsequent HIT (served from
|
||||||
|
cache) both work, plus a `wp-cli` command that requires DB access
|
||||||
|
(`wp option get siteurl`) as an unambiguous connectivity check.
|
||||||
|
|
||||||
|
## Cutover
|
||||||
|
|
||||||
|
1. Stop (don't destroy) the old jail — keep it as a rollback safety net
|
||||||
|
until the new one is fully proven in real use.
|
||||||
|
2. If a host-level reverse-proxy vhost points at the old jail's IP,
|
||||||
|
update it. (Not needed if the domain was never wired to a public vhost
|
||||||
|
yet — check first with `grep -rl <old-ip> /usr/local/etc/nginx/`.)
|
||||||
|
3. Destroy the old jail only once genuinely confident — this session left
|
||||||
|
it stopped rather than destroyed on the first migration, as the
|
||||||
|
controlled/cautious default.
|
||||||
|
4. Update any skill/doc/memory that references the old jail name or IP
|
||||||
|
(grep for both across `agent-skills` and this repo's `docs/`) — easy
|
||||||
|
to miss and leaves stale instructions for the next delegate.
|
||||||
|
|
||||||
|
## Rollout scope discipline
|
||||||
|
|
||||||
|
This was explicitly piloted on **one jail on one host** before being
|
||||||
|
considered for the rest of the fleet — treat "prove it on a single
|
||||||
|
low-stakes target first" as the right default for any jail-networking
|
||||||
|
architecture change, not just this one. Don't extend to production jails
|
||||||
|
or other hosts without a separate, explicit go-ahead per host.
|
||||||
@@ -9,15 +9,19 @@ source: original, from the iWP Cache staging verification (2026-08-02)
|
|||||||
|
|
||||||
## Use the persistent staging jail, not a throwaway one
|
## Use the persistent staging jail, not a throwaway one
|
||||||
|
|
||||||
`staging-1` on the `staging` host (Tailscale/SSH: `100.104.61.54`, jail IP
|
`staging1` on the `staging` host (Tailscale/SSH: `100.104.61.54`, jail IP
|
||||||
`192.168.0.187`, intended domain `staging.palmasolutions.net`) is a
|
`10.20.0.11` — migrated 2026-08-03 to a dedicated private VNET subnet, see
|
||||||
|
[[jail_dedicated_subnet_migration]], was `staging-1` at `192.168.0.187`
|
||||||
|
before that — intended domain `staging.palmasolutions.net`) is a
|
||||||
standing verification environment — deploy new/updated plugins there and
|
standing verification environment — deploy new/updated plugins there and
|
||||||
**leave them installed and active** rather than spinning up a fresh
|
**leave them installed and active** rather than spinning up a fresh
|
||||||
disposable jail per plugin and tearing it down after.
|
disposable jail per plugin and tearing it down after. Note the name has
|
||||||
|
no hyphen (`staging1`, not `staging-1`) — VNET jail names on this host
|
||||||
|
can't contain `-`/`_`.
|
||||||
|
|
||||||
**Why:** a clean-room jail with only the plugin under test installed
|
**Why:** a clean-room jail with only the plugin under test installed
|
||||||
proves the plugin works in isolation, but says nothing about how it
|
proves the plugin works in isolation, but says nothing about how it
|
||||||
behaves alongside the plugins real sites actually run. `staging-1` ships
|
behaves alongside the plugins real sites actually run. `staging1` ships
|
||||||
with a representative baseline already active: WooCommerce, Classic
|
with a representative baseline already active: WooCommerce, Classic
|
||||||
Editor, Contact Form 7, Yoast SEO (`wordpress-seo` is the correct wp.org
|
Editor, Contact Form 7, Yoast SEO (`wordpress-seo` is the correct wp.org
|
||||||
slug — `yoast-seo` doesn't exist), Wordfence, LiteSpeed Cache (a
|
slug — `yoast-seo` doesn't exist), Wordfence, LiteSpeed Cache (a
|
||||||
@@ -27,7 +31,7 @@ Each new plugin verified here adds to that baseline instead of replacing
|
|||||||
it, so the set of "things this plugin has been proven compatible with"
|
it, so the set of "things this plugin has been proven compatible with"
|
||||||
only grows over time.
|
only grows over time.
|
||||||
|
|
||||||
## Setup, once per jail (already done for `staging-1`)
|
## Setup, once per jail (already done for `staging1`)
|
||||||
|
|
||||||
Standard stock stack per [[bastille_jail_provisioning]] (nginx, php-fpm,
|
Standard stock stack per [[bastille_jail_provisioning]] (nginx, php-fpm,
|
||||||
MariaDB-client pointed at the host's local MariaDB, Valkey +
|
MariaDB-client pointed at the host's local MariaDB, Valkey +
|
||||||
|
|||||||
Reference in New Issue
Block a user