diff --git a/README.md b/README.md index c300ca7..33dc447 100644 --- a/README.md +++ b/README.md @@ -55,9 +55,14 @@ than assuming the delegate can fetch it itself. the same bootstrap/wiring file; use self-registration instead. Real incident from the iWP Cache build (lost updates, twice). - `wordpress-plugin-staging-verification` -- use the persistent - `staging-1` jail (real-world plugin set left active) for verifying new + `staging1` jail (real-world plugin set left active) for verifying new plugins, not a disposable clean-room jail; includes the full cache-plugin verification checklist and cosmetic-vs-real error gotchas. +- `jail-dedicated-subnet-migration` -- move jails from shared-IP-on-LAN + aliasing to a dedicated private VNET subnet, eliminating ARP collisions + with other physical LAN devices. Full bridge/NAT/pf setup, VNET jail + creation syntax, and a content-preserving migration procedure (DB grant + re-scoping, hairpin-NAT non-issue, cache-HIT-hides-DB-failures gotcha). ## Provenance diff --git a/skills/jail-dedicated-subnet-migration/SKILL.md b/skills/jail-dedicated-subnet-migration/SKILL.md new file mode 100644 index 0000000..d9cb4d6 --- /dev/null +++ b/skills/jail-dedicated-subnet-migration/SKILL.md @@ -0,0 +1,154 @@ +--- +name: jail-dedicated-subnet-migration +description: Move Bastille jails from shared-IP-aliasing on the LAN interface to a dedicated private VNET subnet, eliminating ARP collisions with other physical LAN devices. Piloted successfully on staging-1 -> staging1, 2026-08-03. +license: MIT +source: original, from the staging host subnet migration pilot +--- + +# Jail Dedicated Subnet Migration + +## Why + +Jails using shared-IP aliasing directly on the host's physical LAN +interface (`vtnet0`) share that interface's broadcast domain with every +other device on the LAN. Confirmed real incidents (see +`bastille-jail-provisioning`'s ARP-collision section): a jail IP that +looked free by every host-local check turned out to already belong to a +different physical device or another host's jail on the same shared +subnet, causing silent networking failures that look exactly like a +config bug. Moving jails to their own private VNET subnet eliminates this +class of collision entirely — confirmed via `arp-scan` on the physical +LAN interface after migration: the jail's new IP produces zero response, +proving it's genuinely invisible outside the host. + +## Prerequisites + +- `if_bridge` and `pf` kernel modules must load (`kldload if_bridge`, + `kldload pf` — both loaded cleanly on a stock FreeBSD 15.1 box with no + prior config). +- A private subnet not overlapping the LAN (`192.168.0.0/24` here) or the + Tailscale CGNAT range (`100.64.0.0/10`) — used `10.20.0.0/24`. +- Root on the jail host. + +## Setup (host-level, once per host) + +1. **Create the bridge, give the host itself an IP on it:** + ```sh + ifconfig bridge create name bastille0 + ifconfig bastille0 inet 10.20.0.1/24 up + ``` +2. **Persist across reboot:** + ```sh + sysrc cloned_interfaces+=bastille0 + sysrc ifconfig_bastille0='inet 10.20.0.1/24' + sysrc gateway_enable=YES + ``` +3. **NAT for jail egress** (jails need this to reach the internet — the + private subnet isn't routable from the LAN's own gateway): + ```sh + cat > /etc/pf.conf <<'EOF' + ext_if = "vtnet0" + jail_net = "10.20.0.0/24" + + nat on $ext_if from $jail_net to any -> ($ext_if) + pass in + pass out + EOF + ``` + **The `pass in`/`pass out` rules are deliberately permissive** — this + is a NAT-only ruleset, not a firewall. Don't add restrictive rules + without separately verifying they don't break existing services + (SSH/Tailscale) on the host. +4. **Enabling pf is a genuine remote-lockout risk** — even a permissive + ruleset syntax error can default-deny. Test with `service pf onestart` + first (bypasses the `pf_enable` rc.conf check, doesn't persist) and + confirm the SSH session survives before persisting with `sysrc + pf_enable=YES` + `service pf start`. This got flagged by this + session's own permission classifier as needing explicit confirmation + before proceeding — treat that as correct caution, not an obstacle to + route around. + +## Creating a VNET jail on the new subnet + +```sh +bastille create -B -g 10.20.0.1 10.20.0.X/24 bastille0 +``` +- `-B`/`--bridge` enables VNET with a bridge interface (not `-V`, which + is for a *physical* interface). +- `-g` sets the jail's default gateway — must be the bridge's own IP + (`10.20.0.1`), or the jail has no route out. +- **VNET jail names cannot contain `-` or `_`** (`[ERROR]: VNET jail + names may not contain (-|_) characters.`) — bastille derives epair + interface names from the jail name. A jail called `staging-1` under + shared-IP mode had to become `staging1` under VNET. + +Verify before trusting it: +```sh +bastille cmd fetch -o /dev/null http://1.1.1.1 # raw IP, outbound NAT +bastille cmd fetch -o /dev/null https://pkg.freebsd.org/ # DNS + outbound +ping -c2 10.20.0.X # host -> jail, no NAT needed (same bridge) +arp-scan --interface=vtnet0 10.20.0.X # must show ZERO responses -- proves LAN isolation +``` + +## Migrating an existing jail's content (not just spinning up empty) + +Don't destroy-and-rebuild-from-scratch if the jail has real installed +state worth keeping (plugins, test data, DB content). Instead: + +1. Create the new VNET jail per above, install the same stock stack + (nginx, php-fpm, mariadb-client, etc.) matching the old jail's. +2. Copy the web root: `cp -a /usr/local/www/site/. /usr/local/www/site/`, then `chown -R www:www` on the destination. +3. **The database doesn't move** if it lives on the host's own local + MariaDB (the common pattern on these hosts) — only the jail's IP + changes, so `wp-config.php`'s `DB_HOST` (the host's own LAN IP) + stays the same. What changes is the MySQL **grant**, which is scoped + by the connecting IP: `CREATE USER IF NOT EXISTS + ''@'' IDENTIFIED BY ''; GRANT ALL + PRIVILEGES ON .* TO ''@'';`. Extract the + existing DB password from the copied `wp-config.php` programmatically + (`awk -F"'" '{print $4}'` on the `DB_PASSWORD` line, or similar) — + **don't manually retype/copy-paste a password across terminal + output**, a single mistyped character produces a misleading "Access + denied (using password: YES)" error that looks like a grant/network + problem but is actually just a wrong password. If that error shows up + despite a seemingly-correct `CREATE USER`, the fastest fix is + `ALTER USER ''@'' IDENTIFIED BY '';` + rather than debugging network connectivity first. +4. Test host→jail DB connectivity is **not** NAT-hairpinned: since the + host's MariaDB and the new jail are both local to the same host, and + the bridge gives the jail a direct route to the host's own bridge IP + (`10.20.0.1`) without going through the NAT'd `vtnet0` path at all, + this generally works cleanly — confirmed connections to the host's + LAN IP from a jail on the private subnet arrive with the jail's real + source IP intact (not NAT-masqueraded), because the kernel routes + host-destined traffic locally rather than sending it out and back + through the NAT rule. Don't assume a hairpin NAT problem without + testing first — it added an unnecessary detour. +5. Verify the site fully — **a page-cache HIT can mislead you into + thinking DB connectivity works when it doesn't**, since a cached page + never touches the database. Purge any cache first, then confirm both + a fresh MISS render (real DB read) and a subsequent HIT (served from + cache) both work, plus a `wp-cli` command that requires DB access + (`wp option get siteurl`) as an unambiguous connectivity check. + +## Cutover + +1. Stop (don't destroy) the old jail — keep it as a rollback safety net + until the new one is fully proven in real use. +2. If a host-level reverse-proxy vhost points at the old jail's IP, + update it. (Not needed if the domain was never wired to a public vhost + yet — check first with `grep -rl /usr/local/etc/nginx/`.) +3. Destroy the old jail only once genuinely confident — this session left + it stopped rather than destroyed on the first migration, as the + controlled/cautious default. +4. Update any skill/doc/memory that references the old jail name or IP + (grep for both across `agent-skills` and this repo's `docs/`) — easy + to miss and leaves stale instructions for the next delegate. + +## Rollout scope discipline + +This was explicitly piloted on **one jail on one host** before being +considered for the rest of the fleet — treat "prove it on a single +low-stakes target first" as the right default for any jail-networking +architecture change, not just this one. Don't extend to production jails +or other hosts without a separate, explicit go-ahead per host. diff --git a/skills/wordpress-plugin-staging-verification/SKILL.md b/skills/wordpress-plugin-staging-verification/SKILL.md index 11048e8..fe220ad 100644 --- a/skills/wordpress-plugin-staging-verification/SKILL.md +++ b/skills/wordpress-plugin-staging-verification/SKILL.md @@ -9,15 +9,19 @@ source: original, from the iWP Cache staging verification (2026-08-02) ## Use the persistent staging jail, not a throwaway one -`staging-1` on the `staging` host (Tailscale/SSH: `100.104.61.54`, jail IP -`192.168.0.187`, intended domain `staging.palmasolutions.net`) is a +`staging1` on the `staging` host (Tailscale/SSH: `100.104.61.54`, jail IP +`10.20.0.11` — migrated 2026-08-03 to a dedicated private VNET subnet, see +[[jail_dedicated_subnet_migration]], was `staging-1` at `192.168.0.187` +before that — intended domain `staging.palmasolutions.net`) is a standing verification environment — deploy new/updated plugins there and **leave them installed and active** rather than spinning up a fresh -disposable jail per plugin and tearing it down after. +disposable jail per plugin and tearing it down after. Note the name has +no hyphen (`staging1`, not `staging-1`) — VNET jail names on this host +can't contain `-`/`_`. **Why:** a clean-room jail with only the plugin under test installed proves the plugin works in isolation, but says nothing about how it -behaves alongside the plugins real sites actually run. `staging-1` ships +behaves alongside the plugins real sites actually run. `staging1` ships with a representative baseline already active: WooCommerce, Classic Editor, Contact Form 7, Yoast SEO (`wordpress-seo` is the correct wp.org slug — `yoast-seo` doesn't exist), Wordfence, LiteSpeed Cache (a @@ -27,7 +31,7 @@ Each new plugin verified here adds to that baseline instead of replacing it, so the set of "things this plugin has been proven compatible with" only grows over time. -## Setup, once per jail (already done for `staging-1`) +## Setup, once per jail (already done for `staging1`) Standard stock stack per [[bastille_jail_provisioning]] (nginx, php-fpm, MariaDB-client pointed at the host's local MariaDB, Valkey +