--- name: bastille-jail-provisioning description: Use when creating a new FreeBSD bastille jail (a new site/service on a shared-LAN jail host) via clone-from-known-good-base. Covers the IP-conflict gotcha that has caused real, time-costly incidents twice. --- # Bastille Jail Provisioning Pattern for standing up a new jail on a FreeBSD host running Bastille, by cloning a known-working base jail (already has nginx+php-fpm+packages configured) rather than building from scratch. ## CRITICAL FIRST STEP: arp-scan before picking an IP **Confirmed real incident, twice, on two different hosts on the same shared LAN**: an IP address that looked free by every host-local check (`bastille list all`, `ifconfig` aliases, a manually-tracked "already used" list) turned out to already belong to a **different physical device, or a jail on a different host sharing the same LAN segment**. Symptom: the new jail's DNS resolution and outbound TCP connections silently fail or time out with no useful error — this looks exactly like a jail networking/pf config bug and can cost real time chasing the wrong theory before the actual cause (an IP conflict) is found. **Before assigning ANY candidate IP, always:** ```bash pkg install -y arp-scan # if not already present arp-scan --interface=vtnet0 ``` Zero responses = genuinely free. **Any** response (even from a device with no recognizable vendor match) means it's taken — pick a different IP and re-check. Do not rely on ping (misses devices with ICMP disabled/filtered — a real false-negative that happened once already) or on a per-host "what's already in use" list (each host only knows about its own jails, not other physical devices or other hosts' jails on the same shared subnet). If you inherit a jail already provisioned on a conflicting IP: stop the jail, remove the bad alias, update `jail.conf`'s `ip4.addr`, start the jail again (this re-adds the alias on the new IP), then propagate the IP change everywhere else it's referenced (DB user grants scoped by IP, host-level nginx `proxy_pass` target, etc.) — an IP conflict fixed in `jail.conf` alone but not in the DB grant/nginx config leaves the site broken even though the jail itself is now healthy. ## Provisioning steps (once the IP is confirmed free) 1. **Clone from a known-good base jail** rather than building from scratch — inherits working nginx/php-fpm/package configuration: ```bash bastille clone -a ``` 2. **Wipe cloned content, do a fresh install** if this is a brand-new site (not a migration) — the cloned base's site files are a *template* for the stack config, not content you want to keep. 3. **Create the database** on the host's local MariaDB, with the grant scoped to the **jail's own IP**, not the host's IP: ```sql CREATE DATABASE `dbname` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; CREATE USER 'dbuser'@'' IDENTIFIED BY ''; GRANT ALL PRIVILEGES ON `dbname`.* TO 'dbuser'@''; ``` The application's `DB_HOST` config value is the **host's** own LAN IP (where MariaDB actually listens), NOT the jail's IP — the jail IP is only used for the GRANT's scope. Confirmed gotcha across multiple real migrations: mixing these two up is an easy, non-obvious mistake. 4. **Wire the host-level nginx reverse-proxy vhost**, `proxy_pass`ing to the jail's IP. 5. **Do NOT touch DNS or the reverse-proxy manager's public routing** as part of automated provisioning — that's a deliberate, manual cutover step for a human to do once the site is verified working over a direct IP/Host-header test. Build and verify fully "dark" first. 6. **Smoke test** both directly to the jail IP and via the host nginx proxy (with the right `Host` header and, if the site checks for HTTPS via `X-Forwarded-Proto`, that header too) before considering provisioning done. ## Persist credentials immediately Write the generated DB password (and any other generated secrets) to a durable location the moment they're generated — if a later provisioning step fails, an unrecorded generated password is otherwise lost with no way to recover it short of resetting it. ## `bastille create` interface gotcha: don't assume `bastille0` exists Some hosts' `bastille.conf` still has the default `bastille_network_loopback="bastille0"` / `bastille_network_vnet_type="if_bridge"` values even though the host actually uses plain shared-IP aliasing on the physical interface (`vtnet0`) for every real jail on it — confirmed on the `staging` host (100.104.61.54) 2026-08-02. A bare `bastille create ` fails with `[ERROR]: bastille0 interface does not exist`. **Check an existing working jail's `jail.conf`** (`ip4.addr = vtnet0|;`) to see which interface this host actually uses, then pass it explicitly: ```bash bastille create vtnet0 ``` ## After create/destroy churn, verify the IP alias actually attached A `bastille create` immediately following a `bastille destroy` of a jail that held the *same* IP can silently fail to (re-)add the alias — the jail comes up "running" but has zero network connectivity (DNS and raw-IP fetch both time out, no error surfaced anywhere). Confirmed live 2026-08-02: `pkg install` inside the new jail failed with "Non-recoverable resolver failure" on the first attempt, which looks exactly like a DNS/resolv.conf problem but wasn't (resolv.conf was correct and identical to a working jail). **Verify directly** before assuming jail networking is up: ```bash ifconfig | grep # must show the alias bastille cmd fetch -o /dev/null http://1.1.1.1 # raw IP, bypasses DNS entirely ``` If the alias is missing, `bastille restart ` (not just `start`) or a manual `ifconfig inet netmask 255.255.255.255 alias` reapplies it. ## Testing a jail before its public vhost exists: hit the jail IP, not the host If the site's domain isn't wired into the host's reverse-proxy nginx yet (no `sites-enabled/.conf`), curling the **host's** own `127.0.0.1` with a `Host:` header does NOT reach the new jail — it falls through to whichever `server_name` on the host's nginx matches first (often a different site's `default_server`), returning a normal-looking `200 OK` for entirely the wrong site's content. This produced a long, wrong-track debugging session on 2026-08-02 (chasing a "cache never writes" theory against a site that was never actually being hit). **Always curl the jail's own IP directly** until the real vhost is wired: ```bash curl -H 'Host: ' http:/// ``` Only switch to testing via the host's `127.0.0.1` (or the public domain) once the `sites-enabled` vhost for this specific site actually exists. ## Stock stack includes Valkey, not just nginx/php/MariaDB The fleet's standard per-host stack (see `docs/server-funky.md`) is nginx + php-fpm + MariaDB (host-local) + **Valkey** (host-local, ACL-auth'd, `aclfile`-based, admin password at `/root/.valkey_admin_pw`). A host that's never hosted a Redis/object-cache-dependent site before may be missing Valkey entirely — install and configure it the same way (`bind 127.0.0.1 `, `aclfile`, admin password file) rather than treating it as optional, so any plugin/site expecting a persistent object cache gets a real one to test against, not silently falls back to a weaker default. Inside a jail that needs to reach it, install the PHP Redis extension — **package name is `php85-pecl-redis`** (not `php85-redis`, which doesn't exist under that name in the FreeBSD ports tree) — and remember Valkey's ACL means any code connecting to it needs an explicit `AUTH` step, not just `connect()`; a `connect()`-only implementation will silently fail every subsequent command with `NOAUTH` against this fleet's standard Valkey setup.