agent-skills/skills/bastille-jail-provisioning/SKILL.md

---
name: bastille-jail-provisioning
description: Use when creating a new FreeBSD bastille jail (a new site/service on a shared-LAN jail host) via clone-from-known-good-base. Covers the IP-conflict gotcha that has caused real, time-costly incidents twice.
---

# Bastille Jail Provisioning

Pattern for standing up a new jail on a FreeBSD host running Bastille,
by cloning a known-working base jail (already has nginx+php-fpm+packages
configured) rather than building from scratch.

## CRITICAL FIRST STEP: arp-scan before picking an IP

**Confirmed real incident, twice, on two different hosts on the same
shared LAN**: an IP address that looked free by every host-local check
(`bastille list all`, `ifconfig` aliases, a manually-tracked "already
used" list) turned out to already belong to a **different physical
device, or a jail on a different host sharing the same LAN segment**.
Symptom: the new jail's DNS resolution and outbound TCP connections
silently fail or time out with no useful error — this looks exactly like
a jail networking/pf config bug and can cost real time chasing the wrong
theory before the actual cause (an IP conflict) is found.

**Before assigning ANY candidate IP, always:**
```bash
pkg install -y arp-scan   # if not already present
arp-scan --interface=vtnet0 <candidate-ip>
```
Zero responses = genuinely free. **Any** response (even from a device
with no recognizable vendor match) means it's taken — pick a different
IP and re-check. Do not rely on ping (misses devices with ICMP
disabled/filtered — a real false-negative that happened once already) or
on a per-host "what's already in use" list (each host only knows about
its own jails, not other physical devices or other hosts' jails on the
same shared subnet).

If you inherit a jail already provisioned on a conflicting IP: stop the
jail, remove the bad alias, update `jail.conf`'s `ip4.addr`, start the
jail again (this re-adds the alias on the new IP), then propagate the IP
change everywhere else it's referenced (DB user grants scoped by IP,
host-level nginx `proxy_pass` target, etc.) — an IP conflict fixed in
`jail.conf` alone but not in the DB grant/nginx config leaves the site
broken even though the jail itself is now healthy.

## Provisioning steps (once the IP is confirmed free)

1. **Clone from a known-good base jail** rather than building from
   scratch — inherits working nginx/php-fpm/package configuration:
   ```bash
   bastille clone -a <known-good-base-jail> <new-jail-name> <new-ip>
   ```
2. **Wipe cloned content, do a fresh install** if this is a brand-new
   site (not a migration) — the cloned base's site files are a *template*
   for the stack config, not content you want to keep.
3. **Create the database** on the host's local MariaDB, with the grant
   scoped to the **jail's own IP**, not the host's IP:
   ```sql
   CREATE DATABASE `dbname` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
   CREATE USER 'dbuser'@'<jail-ip>' IDENTIFIED BY '<random-password>';
   GRANT ALL PRIVILEGES ON `dbname`.* TO 'dbuser'@'<jail-ip>';
   ```
   The application's `DB_HOST` config value is the **host's** own LAN IP
   (where MariaDB actually listens), NOT the jail's IP — the jail IP is
   only used for the GRANT's scope. Confirmed gotcha across multiple
   real migrations: mixing these two up is an easy, non-obvious mistake.
4. **Wire the host-level nginx reverse-proxy vhost**, `proxy_pass`ing to
   the jail's IP.
5. **Do NOT touch DNS or the reverse-proxy manager's public routing** as
   part of automated provisioning — that's a deliberate, manual cutover
   step for a human to do once the site is verified working over a
   direct IP/Host-header test. Build and verify fully "dark" first.
6. **Smoke test** both directly to the jail IP and via the host nginx
   proxy (with the right `Host` header and, if the site checks for HTTPS
   via `X-Forwarded-Proto`, that header too) before considering
   provisioning done.

## Persist credentials immediately

Write the generated DB password (and any other generated secrets) to a
durable location the moment they're generated — if a later provisioning
step fails, an unrecorded generated password is otherwise lost with no
way to recover it short of resetting it.

## `bastille create` interface gotcha: don't assume `bastille0` exists

Some hosts' `bastille.conf` still has the default
`bastille_network_loopback="bastille0"` / `bastille_network_vnet_type="if_bridge"`
values even though the host actually uses plain shared-IP aliasing on the
physical interface (`vtnet0`) for every real jail on it — confirmed on the
`staging` host (100.104.61.54) 2026-08-02. A bare `bastille create <name>
<release> <ip>` fails with `[ERROR]: bastille0 interface does not exist`.
**Check an existing working jail's `jail.conf`** (`ip4.addr = vtnet0|<ip>;`)
to see which interface this host actually uses, then pass it explicitly:
```bash
bastille create <name> <release> <ip> vtnet0
```

## After create/destroy churn, verify the IP alias actually attached

A `bastille create` immediately following a `bastille destroy` of a jail
that held the *same* IP can silently fail to (re-)add the alias — the jail
comes up "running" but has zero network connectivity (DNS and raw-IP fetch
both time out, no error surfaced anywhere). Confirmed live 2026-08-02: `pkg
install` inside the new jail failed with "Non-recoverable resolver failure"
on the first attempt, which looks exactly like a DNS/resolv.conf problem
but wasn't (resolv.conf was correct and identical to a working jail).
**Verify directly** before assuming jail networking is up:
```bash
ifconfig <interface> | grep <jail-ip>   # must show the alias
bastille cmd <jail> fetch -o /dev/null http://1.1.1.1  # raw IP, bypasses DNS entirely
```
If the alias is missing, `bastille restart <jail>` (not just `start`) or a
manual `ifconfig <interface> inet <ip> netmask 255.255.255.255 alias`
reapplies it.

## Testing a jail before its public vhost exists: hit the jail IP, not the host

If the site's domain isn't wired into the host's reverse-proxy nginx yet
(no `sites-enabled/<jail>.conf`), curling the **host's** own `127.0.0.1`
with a `Host:` header does NOT reach the new jail — it falls through to
whichever `server_name` on the host's nginx matches first (often a
different site's `default_server`), returning a normal-looking `200 OK`
for entirely the wrong site's content. This produced a long, wrong-track
debugging session on 2026-08-02 (chasing a "cache never writes" theory
against a site that was never actually being hit). **Always curl the
jail's own IP directly** until the real vhost is wired:
```bash
curl -H 'Host: <domain>' http://<jail-ip>/
```
Only switch to testing via the host's `127.0.0.1` (or the public domain)
once the `sites-enabled` vhost for this specific site actually exists.

## Stock stack includes Valkey, not just nginx/php/MariaDB

The fleet's standard per-host stack (see `docs/server-funky.md`) is
nginx + php-fpm + MariaDB (host-local) + **Valkey** (host-local, ACL-auth'd,
`aclfile`-based, admin password at `/root/.valkey_admin_pw`). A host that's
never hosted a Redis/object-cache-dependent site before may be missing
Valkey entirely — install and configure it the same way (`bind 127.0.0.1
<host-lan-ip>`, `aclfile`, admin password file) rather than treating it as
optional, so any plugin/site expecting a persistent object cache gets a
real one to test against, not silently falls back to a weaker default.
Inside a jail that needs to reach it, install the PHP Redis extension —
**package name is `php85-pecl-redis`** (not `php85-redis`, which doesn't
exist under that name in the FreeBSD ports tree) — and remember Valkey's
ACL means any code connecting to it needs an explicit `AUTH` step, not just
`connect()`; a `connect()`-only implementation will silently fail every
subsequent command with `NOAUTH` against this fleet's standard Valkey setup.