agent-skills/skills/bastille-jail-provisioning/SKILL.md

name, description

name	description
bastille-jail-provisioning	Use when creating a new FreeBSD bastille jail (a new site/service on a shared-LAN jail host) via clone-from-known-good-base. Covers the IP-conflict gotcha that has caused real, time-costly incidents twice.

Bastille Jail Provisioning

Pattern for standing up a new jail on a FreeBSD host running Bastille, by cloning a known-working base jail (already has nginx+php-fpm+packages configured) rather than building from scratch.

CRITICAL FIRST STEP: arp-scan before picking an IP

Confirmed real incident, twice, on two different hosts on the same shared LAN: an IP address that looked free by every host-local check (bastille list all, ifconfig aliases, a manually-tracked "already used" list) turned out to already belong to a different physical device, or a jail on a different host sharing the same LAN segment. Symptom: the new jail's DNS resolution and outbound TCP connections silently fail or time out with no useful error — this looks exactly like a jail networking/pf config bug and can cost real time chasing the wrong theory before the actual cause (an IP conflict) is found.

Before assigning ANY candidate IP, always:

pkg install -y arp-scan   # if not already present
arp-scan --interface=vtnet0 <candidate-ip>

Zero responses = genuinely free. Any response (even from a device with no recognizable vendor match) means it's taken — pick a different IP and re-check. Do not rely on ping (misses devices with ICMP disabled/filtered — a real false-negative that happened once already) or on a per-host "what's already in use" list (each host only knows about its own jails, not other physical devices or other hosts' jails on the same shared subnet).

If you inherit a jail already provisioned on a conflicting IP: stop the jail, remove the bad alias, update jail.conf's ip4.addr, start the jail again (this re-adds the alias on the new IP), then propagate the IP change everywhere else it's referenced (DB user grants scoped by IP, host-level nginx proxy_pass target, etc.) — an IP conflict fixed in jail.conf alone but not in the DB grant/nginx config leaves the site broken even though the jail itself is now healthy.

Provisioning steps (once the IP is confirmed free)

1. Clone from a known-good base jail rather than building from scratch — inherits working nginx/php-fpm/package configuration:

   bastille clone -a <known-good-base-jail> <new-jail-name> <new-ip>
   

2. Wipe cloned content, do a fresh install if this is a brand-new site (not a migration) — the cloned base's site files are a template for the stack config, not content you want to keep.
3. Create the database on the host's local MariaDB, with the grant scoped to the jail's own IP, not the host's IP:

   CREATE DATABASE `dbname` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
   CREATE USER 'dbuser'@'<jail-ip>' IDENTIFIED BY '<random-password>';
   GRANT ALL PRIVILEGES ON `dbname`.* TO 'dbuser'@'<jail-ip>';
   

   The application's DB_HOST config value is the host's own LAN IP (where MariaDB actually listens), NOT the jail's IP — the jail IP is only used for the GRANT's scope. Confirmed gotcha across multiple real migrations: mixing these two up is an easy, non-obvious mistake.
4. Wire the host-level nginx reverse-proxy vhost, proxy_passing to the jail's IP.
5. Do NOT touch DNS or the reverse-proxy manager's public routing as part of automated provisioning — that's a deliberate, manual cutover step for a human to do once the site is verified working over a direct IP/Host-header test. Build and verify fully "dark" first.
6. Smoke test both directly to the jail IP and via the host nginx proxy (with the right Host header and, if the site checks for HTTPS via X-Forwarded-Proto, that header too) before considering provisioning done.

Persist credentials immediately

Write the generated DB password (and any other generated secrets) to a durable location the moment they're generated — if a later provisioning step fails, an unrecorded generated password is otherwise lost with no way to recover it short of resetting it.