agent-skills/skills/bastille-jail-provisioning/SKILL.md

name, description

name	description
bastille-jail-provisioning	Use when creating a new FreeBSD bastille jail (a new site/service on a shared-LAN jail host) via clone-from-known-good-base. Covers the IP-conflict gotcha that has caused real, time-costly incidents twice.

Bastille Jail Provisioning

Pattern for standing up a new jail on a FreeBSD host running Bastille, by cloning a known-working base jail (already has nginx+php-fpm+packages configured) rather than building from scratch.

CRITICAL FIRST STEP: arp-scan before picking an IP

Confirmed real incident, twice, on two different hosts on the same shared LAN: an IP address that looked free by every host-local check (bastille list all, ifconfig aliases, a manually-tracked "already used" list) turned out to already belong to a different physical device, or a jail on a different host sharing the same LAN segment. Symptom: the new jail's DNS resolution and outbound TCP connections silently fail or time out with no useful error — this looks exactly like a jail networking/pf config bug and can cost real time chasing the wrong theory before the actual cause (an IP conflict) is found.

Before assigning ANY candidate IP, always:

pkg install -y arp-scan   # if not already present
arp-scan --interface=vtnet0 <candidate-ip>

Zero responses = genuinely free. Any response (even from a device with no recognizable vendor match) means it's taken — pick a different IP and re-check. Do not rely on ping (misses devices with ICMP disabled/filtered — a real false-negative that happened once already) or on a per-host "what's already in use" list (each host only knows about its own jails, not other physical devices or other hosts' jails on the same shared subnet).

If you inherit a jail already provisioned on a conflicting IP: stop the jail, remove the bad alias, update jail.conf's ip4.addr, start the jail again (this re-adds the alias on the new IP), then propagate the IP change everywhere else it's referenced (DB user grants scoped by IP, host-level nginx proxy_pass target, etc.) — an IP conflict fixed in jail.conf alone but not in the DB grant/nginx config leaves the site broken even though the jail itself is now healthy.

Provisioning steps (once the IP is confirmed free)

1. Clone from a known-good base jail rather than building from scratch — inherits working nginx/php-fpm/package configuration:

   bastille clone -a <known-good-base-jail> <new-jail-name> <new-ip>
   

2. Wipe cloned content, do a fresh install if this is a brand-new site (not a migration) — the cloned base's site files are a template for the stack config, not content you want to keep.
3. Create the database on the host's local MariaDB, with the grant scoped to the jail's own IP, not the host's IP:

   CREATE DATABASE `dbname` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
   CREATE USER 'dbuser'@'<jail-ip>' IDENTIFIED BY '<random-password>';
   GRANT ALL PRIVILEGES ON `dbname`.* TO 'dbuser'@'<jail-ip>';
   

   The application's DB_HOST config value is the host's own LAN IP (where MariaDB actually listens), NOT the jail's IP — the jail IP is only used for the GRANT's scope. Confirmed gotcha across multiple real migrations: mixing these two up is an easy, non-obvious mistake.
4. Wire the host-level nginx reverse-proxy vhost, proxy_passing to the jail's IP.
5. Do NOT touch DNS or the reverse-proxy manager's public routing as part of automated provisioning — that's a deliberate, manual cutover step for a human to do once the site is verified working over a direct IP/Host-header test. Build and verify fully "dark" first.
6. Smoke test both directly to the jail IP and via the host nginx proxy (with the right Host header and, if the site checks for HTTPS via X-Forwarded-Proto, that header too) before considering provisioning done.

Persist credentials immediately

Write the generated DB password (and any other generated secrets) to a durable location the moment they're generated — if a later provisioning step fails, an unrecorded generated password is otherwise lost with no way to recover it short of resetting it.

bastille create interface gotcha: don't assume bastille0 exists

Some hosts' bastille.conf still has the default bastille_network_loopback="bastille0" / bastille_network_vnet_type="if_bridge" values even though the host actually uses plain shared-IP aliasing on the physical interface (vtnet0) for every real jail on it — confirmed on the staging host (100.104.61.54) 2026-08-02. A bare bastille create <name> <release> <ip> fails with [ERROR]: bastille0 interface does not exist. Check an existing working jail's jail.conf (ip4.addr = vtnet0|<ip>;) to see which interface this host actually uses, then pass it explicitly:

bastille create <name> <release> <ip> vtnet0

After create/destroy churn, verify the IP alias actually attached

A bastille create immediately following a bastille destroy of a jail that held the same IP can silently fail to (re-)add the alias — the jail comes up "running" but has zero network connectivity (DNS and raw-IP fetch both time out, no error surfaced anywhere). Confirmed live 2026-08-02: pkg install inside the new jail failed with "Non-recoverable resolver failure" on the first attempt, which looks exactly like a DNS/resolv.conf problem but wasn't (resolv.conf was correct and identical to a working jail). Verify directly before assuming jail networking is up:

ifconfig <interface> | grep <jail-ip>   # must show the alias
bastille cmd <jail> fetch -o /dev/null http://1.1.1.1  # raw IP, bypasses DNS entirely

If the alias is missing, bastille restart <jail> (not just start) or a manual ifconfig <interface> inet <ip> netmask 255.255.255.255 alias reapplies it.

Testing a jail before its public vhost exists: hit the jail IP, not the host

If the site's domain isn't wired into the host's reverse-proxy nginx yet (no sites-enabled/<jail>.conf), curling the host's own 127.0.0.1 with a Host: header does NOT reach the new jail — it falls through to whichever server_name on the host's nginx matches first (often a different site's default_server), returning a normal-looking 200 OK for entirely the wrong site's content. This produced a long, wrong-track debugging session on 2026-08-02 (chasing a "cache never writes" theory against a site that was never actually being hit). Always curl the jail's own IP directly until the real vhost is wired:

curl -H 'Host: <domain>' http://<jail-ip>/

Only switch to testing via the host's 127.0.0.1 (or the public domain) once the sites-enabled vhost for this specific site actually exists.

Default-deny outbound is the standard for every new jail

This is no longer an incident workaround — it is the default posture for any jail created on any Bastille host in this fleet. Most jails are reverse-proxied inbound-only services and do not need outbound internet access. Denying it by default limits blast radius if a jail is compromised (fewer callback/exfil paths, no unexpected update pings).

When you finish provisioning a new jail, it must land in the block out ruleset, not the pass out ruleset. Only add it to the allowed list if its actual function genuinely requires external reach, and document the reason in pf.conf next to the IP:

# Jails that need real outbound internet access (function confirmed):
table <outbound_allowed> { 10.20.0.14, 10.20.0.22, ... }
nat on $ext_if from <outbound_allowed> to any -> ($ext_if)
pass out quick on $ext_if from <outbound_allowed> to any keep state
block out quick on $ext_if from 10.20.0.0/24 to any

The live reference implementation is gringo's /etc/pf.conf (tracked in infra/gringo/pf/pf.conf). Before considering any new jail complete, verify its IP is either in the deny block or explicitly listed with a comment explaining why it needs outbound access. "Might need it later" is not a reason.

Every jail needs a dedicated NPM-facing port, not just listen 80

Confirmed real gap, 2026-08-07: on jail hosts fronted by a shared Nginx Proxy Manager (NPM) instance, every existing jail's host-level nginx vhost has two listen directives — the shared port 80 (for same-host/Host-header testing) and a dedicated port unique to that jail, following the convention 8000 + <jail's last IP octet> (e.g. a jail at 10.20.0.43 gets listen 8043;, one at .50 gets listen 8050;). This dedicated port is what NPM's own proxy-host config actually targets when routing the public domain to this jail host — NPM does not rely on Host-header-based routing through the shared port 80 for this fleet, it connects to a specific port per site.

This was missed migrating 5 Laravel apps from a Linux host to a FreeBSD jail host (gringo) in one session: all 5 new vhosts were created with only listen 80;, matching the shared port but missing the dedicated one entirely. Every functional test the delegate ran (raw jail IP, and via the shared port 80 with the right Host: header) passed cleanly — the gap was invisible until specifically checked against sibling jails' configs, at which point every other jail on the host turned out to follow the two-port pattern without exception. Before considering any new jail's nginx vhost complete, diff its listen directives against at least one working sibling jail's config on the same host — don't just confirm the site loads via a manual test, confirm the routing surface matches the established convention, since NPM (external, not scriptable from this sandbox) is what will actually determine reachability once cut over, and a working manual test doesn't prove NPM's real path works.

# find the convention on any host that uses it
for f in /usr/local/etc/nginx/sites-available/*.conf; do
  echo "=== $(basename $f) ==="; grep -E '^\s*listen' "$f"
done

If a host doesn't show this pattern on ANY existing jail, it likely doesn't use per-jail dedicated ports — don't assume the convention applies fleet-wide without checking the specific host first.

pf nat on $ext_if from <subnet> to any -> ($ext_if) silently fails to translate the source IP when that IP is already aliased on the same interface

Confirmed real incident, 2026-08-07, on a host using the shared-IP- aliasing jail pattern (jail IPs are /32 aliases directly on the host's own external interface, e.g. 192.168.0.185 aliased onto vtnet0 alongside the interface's real address 192.168.168.64 -- not the VNET/ bridge pattern where jails get a genuinely separate subnet). A nat on $ext_if from <jail-subnet> to any -> ($ext_if) rule -- the standard, seemingly-correct pf idiom for masquerading outbound jail traffic -- did not rewrite the source address at all for these jails. Outbound packets left with the jail's own 192.168.0.x source IP untouched (only the source port got NAT'd), which the upstream network silently dropped (source IP outside the VM's assigned subnet, likely anti-spoof filtering) -- symptom was a clean connection timeout, not a pf/rule error, with a real SYN_SENT state visible in pfctl -s states showing the untranslated source IP in parens. This broke every outbound call any jail on that subnet tried to make (external API calls, WordPress core's own update-check pings, etc.) -- surfaced as WordPress admin pages hanging/timing out with no PHP error logged anywhere, since the PHP-FPM worker was just sleeping on a TCP connect that would never complete.

Root cause: the ($ext_if) interface-macro NAT target apparently skips/no-ops translation when the connection's source address is already one of the interface's own configured addresses -- plausible as "this traffic looks locally-originated already, no need to rewrite" logic in pf's NAT implementation, but wrong for this topology where those aliased addresses are jail IPs that specifically need translating to look like the host's primary address before leaving.

Fix: use an explicit IP as the NAT target, not the interface macro.

ext_if = "vtnet0"
ext_ip = "192.168.168.64"        # the interface's real, non-aliased address
lan_net = "192.168.0.0/24"       # the jail-alias subnet

nat on $ext_if from $lan_net to any -> $ext_ip     # works
# nat on $ext_if from $lan_net to any -> ($ext_if) # looked identical, silently no-ops

Verify a NAT fix actually rewrites the source, don't just check pfctl -s nat shows the rule (a no-op-translating rule still shows up there looking correct) -- watch pfctl -s states during a live outbound attempt from the jail and confirm the parenthetical translated address/port shown is genuinely different from the jail's own IP, not just a port change on the same source IP.

This is a distinct topology from the VNET/bridge pattern documented elsewhere in this skill and in jail-dedicated-subnet-migration (where jails get a real separate subnet on a bridge interface, e.g. 10.20.0.0/24) -- that pattern's NAT rules were unaffected by this bug in the same test. Check which pattern a given host actually uses (ifconfig -- are jail IPs aliased directly on the external interface, or on a separate bridge?) before assuming either NAT idiom is safe.

Stock stack includes Valkey, not just nginx/php/MariaDB

The fleet's standard per-host stack (see docs/server-funky.md) is nginx + php-fpm + MariaDB (host-local) + Valkey (host-local, ACL-auth'd, aclfile-based, admin password at /root/.valkey_admin_pw). A host that's never hosted a Redis/object-cache-dependent site before may be missing Valkey entirely — install and configure it the same way (bind 127.0.0.1 <host-lan-ip>, aclfile, admin password file) rather than treating it as optional, so any plugin/site expecting a persistent object cache gets a real one to test against, not silently falls back to a weaker default. Inside a jail that needs to reach it, install the PHP Redis extension — package name is php85-pecl-redis (not php85-redis, which doesn't exist under that name in the FreeBSD ports tree) — and remember Valkey's ACL means any code connecting to it needs an explicit AUTH step, not just connect(); a connect()-only implementation will silently fail every subsequent command with NOAUTH against this fleet's standard Valkey setup.