docs: add disk-space-check discipline after a real venus MariaDB outage caused by bulk jail creation

2026-08-03 13:47:46 +02:00
parent d837c6dc1d
commit 4eff29939e
1 changed files with 27 additions and 0 deletions
@@ -169,6 +169,33 @@ state worth keeping (plugins, test data, DB content). Instead:
   (grep for both across `agent-skills` and this repo's `docs/`) — easy
   to miss and leaves stale instructions for the next delegate.
 ## Check disk space BEFORE creating any jail, every time
 **Do not assume "blank slate, plenty of room" holds on every host** just
 because it held on the pilot host. Confirmed real incident on the second
 host attempted (venus, 2026-08-03): creating 3 new full-stack jails
 (~200MB retained footprint each after packages) pushed an already-96%-full
 ZFS pool over the edge and **crashed the shared MariaDB instance**, taking
 down every site on the host (a real, if brief, production outage) --
 this exact failure mode had already been documented once before on this
 same host from an unrelated earlier incident, and should have been
 checked proactively rather than triggered again.
 **Before creating even one new jail**, run `zfs list zroot/ROOT/default -o
 avail` (or `df -h` if not ZFS) and treat anything under a few GB as a hard
 stop -- clean up first (`bastille pkg <jail> clean -ay` across existing
 jails reclaims real space safely, it's just downloaded package cache) or
 get the capacity issue resolved before proceeding. Re-check after every
 1-2 jail creations during a bulk rollout, don't just check once at the
 start and assume it holds for the whole batch.
 If MariaDB (or any host-shared service) does go down from disk pressure:
 the rc.d script name may not match the obvious guess (`mysql-server`, not
 `mysql` or `mariadb`, was the actual name on venus) -- check `ls
 /usr/local/etc/rc.d/` rather than guessing. After restarting, verify with
 actual `curl` tests against multiple real sites (not just "is the process
 running") before considering it resolved.
 ## Rollout scope discipline
 This was explicitly piloted on **one jail on one host** before being