--- name: bastille-jail-provisioning description: Use when creating a new FreeBSD bastille jail (a new site/service on a shared-LAN jail host) via clone-from-known-good-base. Covers the IP-conflict gotcha that has caused real, time-costly incidents twice. --- # Bastille Jail Provisioning Pattern for standing up a new jail on a FreeBSD host running Bastille, by cloning a known-working base jail (already has nginx+php-fpm+packages configured) rather than building from scratch. ## CRITICAL FIRST STEP: arp-scan before picking an IP **Confirmed real incident, twice, on two different hosts on the same shared LAN**: an IP address that looked free by every host-local check (`bastille list all`, `ifconfig` aliases, a manually-tracked "already used" list) turned out to already belong to a **different physical device, or a jail on a different host sharing the same LAN segment**. Symptom: the new jail's DNS resolution and outbound TCP connections silently fail or time out with no useful error — this looks exactly like a jail networking/pf config bug and can cost real time chasing the wrong theory before the actual cause (an IP conflict) is found. **Before assigning ANY candidate IP, always:** ```bash pkg install -y arp-scan # if not already present arp-scan --interface=vtnet0 ``` Zero responses = genuinely free. **Any** response (even from a device with no recognizable vendor match) means it's taken — pick a different IP and re-check. Do not rely on ping (misses devices with ICMP disabled/filtered — a real false-negative that happened once already) or on a per-host "what's already in use" list (each host only knows about its own jails, not other physical devices or other hosts' jails on the same shared subnet). If you inherit a jail already provisioned on a conflicting IP: stop the jail, remove the bad alias, update `jail.conf`'s `ip4.addr`, start the jail again (this re-adds the alias on the new IP), then propagate the IP change everywhere else it's referenced (DB user grants scoped by IP, host-level nginx `proxy_pass` target, etc.) — an IP conflict fixed in `jail.conf` alone but not in the DB grant/nginx config leaves the site broken even though the jail itself is now healthy. ## Provisioning steps (once the IP is confirmed free) 1. **Clone from a known-good base jail** rather than building from scratch — inherits working nginx/php-fpm/package configuration: ```bash bastille clone -a ``` 2. **Wipe cloned content, do a fresh install** if this is a brand-new site (not a migration) — the cloned base's site files are a *template* for the stack config, not content you want to keep. 3. **Create the database** on the host's local MariaDB, with the grant scoped to the **jail's own IP**, not the host's IP: ```sql CREATE DATABASE `dbname` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; CREATE USER 'dbuser'@'' IDENTIFIED BY ''; GRANT ALL PRIVILEGES ON `dbname`.* TO 'dbuser'@''; ``` The application's `DB_HOST` config value is the **host's** own LAN IP (where MariaDB actually listens), NOT the jail's IP — the jail IP is only used for the GRANT's scope. Confirmed gotcha across multiple real migrations: mixing these two up is an easy, non-obvious mistake. 4. **Wire the host-level nginx reverse-proxy vhost**, `proxy_pass`ing to the jail's IP. 5. **Do NOT touch DNS or the reverse-proxy manager's public routing** as part of automated provisioning — that's a deliberate, manual cutover step for a human to do once the site is verified working over a direct IP/Host-header test. Build and verify fully "dark" first. 6. **Smoke test** both directly to the jail IP and via the host nginx proxy (with the right `Host` header and, if the site checks for HTTPS via `X-Forwarded-Proto`, that header too) before considering provisioning done. ## Persist credentials immediately Write the generated DB password (and any other generated secrets) to a durable location the moment they're generated — if a later provisioning step fails, an unrecorded generated password is otherwise lost with no way to recover it short of resetting it. ## `bastille create` interface gotcha: don't assume `bastille0` exists Some hosts' `bastille.conf` still has the default `bastille_network_loopback="bastille0"` / `bastille_network_vnet_type="if_bridge"` values even though the host actually uses plain shared-IP aliasing on the physical interface (`vtnet0`) for every real jail on it — confirmed on the `staging` host (100.104.61.54) 2026-08-02. A bare `bastille create ` fails with `[ERROR]: bastille0 interface does not exist`. **Check an existing working jail's `jail.conf`** (`ip4.addr = vtnet0|;`) to see which interface this host actually uses, then pass it explicitly: ```bash bastille create vtnet0 ``` ## After create/destroy churn, verify the IP alias actually attached A `bastille create` immediately following a `bastille destroy` of a jail that held the *same* IP can silently fail to (re-)add the alias — the jail comes up "running" but has zero network connectivity (DNS and raw-IP fetch both time out, no error surfaced anywhere). Confirmed live 2026-08-02: `pkg install` inside the new jail failed with "Non-recoverable resolver failure" on the first attempt, which looks exactly like a DNS/resolv.conf problem but wasn't (resolv.conf was correct and identical to a working jail). **Verify directly** before assuming jail networking is up: ```bash ifconfig | grep # must show the alias bastille cmd fetch -o /dev/null http://1.1.1.1 # raw IP, bypasses DNS entirely ``` If the alias is missing, `bastille restart ` (not just `start`) or a manual `ifconfig inet netmask 255.255.255.255 alias` reapplies it. ## Testing a jail before its public vhost exists: hit the jail IP, not the host If the site's domain isn't wired into the host's reverse-proxy nginx yet (no `sites-enabled/.conf`), curling the **host's** own `127.0.0.1` with a `Host:` header does NOT reach the new jail — it falls through to whichever `server_name` on the host's nginx matches first (often a different site's `default_server`), returning a normal-looking `200 OK` for entirely the wrong site's content. This produced a long, wrong-track debugging session on 2026-08-02 (chasing a "cache never writes" theory against a site that was never actually being hit). **Always curl the jail's own IP directly** until the real vhost is wired: ```bash curl -H 'Host: ' http:/// ``` Only switch to testing via the host's `127.0.0.1` (or the public domain) once the `sites-enabled` vhost for this specific site actually exists. ## Default-deny outbound is the standard for every new jail **This is no longer an incident workaround — it is the default posture for any jail created on any Bastille host in this fleet.** Most jails are reverse-proxied inbound-only services and do not need outbound internet access. Denying it by default limits blast radius if a jail is compromised (fewer callback/exfil paths, no unexpected update pings). When you finish provisioning a new jail, it must land in the `block out` ruleset, not the `pass out` ruleset. Only add it to the allowed list if its actual function genuinely requires external reach, and document the reason in `pf.conf` next to the IP: ``` # Jails that need real outbound internet access (function confirmed): table { 10.20.0.14, 10.20.0.22, ... } nat on $ext_if from to any -> ($ext_if) pass out quick on $ext_if from to any keep state block out quick on $ext_if from 10.20.0.0/24 to any ``` The live reference implementation is gringo's `/etc/pf.conf` (tracked in `infra/gringo/pf/pf.conf`). Before considering any new jail complete, verify its IP is either in the deny block or explicitly listed with a comment explaining why it needs outbound access. "Might need it later" is not a reason. ## Every jail needs a dedicated NPM-facing port, not just `listen 80` **Confirmed real gap, 2026-08-07**: on jail hosts fronted by a shared Nginx Proxy Manager (NPM) instance, every existing jail's host-level nginx vhost has **two** `listen` directives — the shared port 80 (for same-host/Host-header testing) **and a dedicated port unique to that jail**, following the convention `8000 + ` (e.g. a jail at `10.20.0.43` gets `listen 8043;`, one at `.50` gets `listen 8050;`). This dedicated port is what NPM's own proxy-host config actually targets when routing the public domain to this jail host — **NPM does not rely on Host-header-based routing through the shared port 80 for this fleet**, it connects to a specific port per site. This was missed migrating 5 Laravel apps from a Linux host to a FreeBSD jail host (gringo) in one session: all 5 new vhosts were created with only `listen 80;`, matching the *shared* port but missing the dedicated one entirely. Every functional test the delegate ran (raw jail IP, and via the shared port 80 with the right `Host:` header) passed cleanly — the gap was invisible until specifically checked against sibling jails' configs, at which point every other jail on the host turned out to follow the two-port pattern without exception. **Before considering any new jail's nginx vhost complete, diff its `listen` directives against at least one working sibling jail's config on the same host** — don't just confirm the site loads via a manual test, confirm the *routing surface* matches the established convention, since NPM (external, not scriptable from this sandbox) is what will actually determine reachability once cut over, and a working manual test doesn't prove NPM's real path works. ```bash # find the convention on any host that uses it for f in /usr/local/etc/nginx/sites-available/*.conf; do echo "=== $(basename $f) ==="; grep -E '^\s*listen' "$f" done ``` If a host doesn't show this pattern on ANY existing jail, it likely doesn't use per-jail dedicated ports — don't assume the convention applies fleet-wide without checking the specific host first. ## pf `nat on $ext_if from to any -> ($ext_if)` silently fails to translate the source IP when that IP is already aliased on the same interface **Confirmed real incident, 2026-08-07**, on a host using the shared-IP- aliasing jail pattern (jail IPs are `/32` aliases directly on the host's own external interface, e.g. `192.168.0.185` aliased onto `vtnet0` alongside the interface's real address `192.168.168.64` -- not the VNET/ bridge pattern where jails get a genuinely separate subnet). A `nat on $ext_if from to any -> ($ext_if)` rule -- the standard, seemingly-correct pf idiom for masquerading outbound jail traffic -- **did not rewrite the source address at all** for these jails. Outbound packets left with the jail's own 192.168.0.x source IP untouched (only the source *port* got NAT'd), which the upstream network silently dropped (source IP outside the VM's assigned subnet, likely anti-spoof filtering) -- symptom was a clean connection timeout, not a pf/rule error, with a real `SYN_SENT` state visible in `pfctl -s states` showing the untranslated source IP in parens. This broke every outbound call any jail on that subnet tried to make (external API calls, WordPress core's own update-check pings, etc.) -- surfaced as WordPress admin pages hanging/timing out with no PHP error logged anywhere, since the PHP-FPM worker was just sleeping on a TCP connect that would never complete. **Root cause**: the `($ext_if)` interface-macro NAT target apparently skips/no-ops translation when the connection's source address is *already* one of the interface's own configured addresses -- plausible as "this traffic looks locally-originated already, no need to rewrite" logic in pf's NAT implementation, but wrong for this topology where those aliased addresses are jail IPs that specifically need translating to look like the host's *primary* address before leaving. **Fix: use an explicit IP as the NAT target, not the interface macro.** ``` ext_if = "vtnet0" ext_ip = "192.168.168.64" # the interface's real, non-aliased address lan_net = "192.168.0.0/24" # the jail-alias subnet nat on $ext_if from $lan_net to any -> $ext_ip # works # nat on $ext_if from $lan_net to any -> ($ext_if) # looked identical, silently no-ops ``` **Verify a NAT fix actually rewrites the source**, don't just check `pfctl -s nat` shows the rule (a no-op-translating rule still shows up there looking correct) -- watch `pfctl -s states` during a live outbound attempt from the jail and confirm the parenthetical translated address/port shown is genuinely different from the jail's own IP, not just a port change on the same source IP. This is a distinct topology from the VNET/bridge pattern documented elsewhere in this skill and in `jail-dedicated-subnet-migration` (where jails get a real separate subnet on a bridge interface, e.g. `10.20.0.0/24`) -- that pattern's NAT rules were unaffected by this bug in the same test. Check which pattern a given host actually uses (`ifconfig` -- are jail IPs aliased directly on the external interface, or on a separate bridge?) before assuming either NAT idiom is safe. ## Stock stack includes Valkey, not just nginx/php/MariaDB The fleet's standard per-host stack (see `docs/server-funky.md`) is nginx + php-fpm + MariaDB (host-local) + **Valkey** (host-local, ACL-auth'd, `aclfile`-based, admin password at `/root/.valkey_admin_pw`). A host that's never hosted a Redis/object-cache-dependent site before may be missing Valkey entirely — install and configure it the same way (`bind 127.0.0.1 `, `aclfile`, admin password file) rather than treating it as optional, so any plugin/site expecting a persistent object cache gets a real one to test against, not silently falls back to a weaker default. Inside a jail that needs to reach it, install the PHP Redis extension — **package name is `php85-pecl-redis`** (not `php85-redis`, which doesn't exist under that name in the FreeBSD ports tree) — and remember Valkey's ACL means any code connecting to it needs an explicit `AUTH` step, not just `connect()`; a `connect()`-only implementation will silently fail every subsequent command with `NOAUTH` against this fleet's standard Valkey setup.