docs: add disk-space-check discipline after a real venus MariaDB outage caused by bulk jail creation

2026-08-03 13:47:46 +02:00
parent d837c6dc1d
commit 4eff29939e
1 changed files with 27 additions and 0 deletions
@@ -169,6 +169,33 @@ state worth keeping (plugins, test data, DB content). Instead:
   (grep for both across `agent-skills` and this repo's `docs/`) — easy
   to miss and leaves stale instructions for the next delegate.

+## Check disk space BEFORE creating any jail, every time
+
+**Do not assume "blank slate, plenty of room" holds on every host** just
+because it held on the pilot host. Confirmed real incident on the second
+host attempted (venus, 2026-08-03): creating 3 new full-stack jails
+(~200MB retained footprint each after packages) pushed an already-96%-full
+ZFS pool over the edge and **crashed the shared MariaDB instance**, taking
+down every site on the host (a real, if brief, production outage) --
+this exact failure mode had already been documented once before on this
+same host from an unrelated earlier incident, and should have been
+checked proactively rather than triggered again.
+
+**Before creating even one new jail**, run `zfs list zroot/ROOT/default -o
+avail` (or `df -h` if not ZFS) and treat anything under a few GB as a hard
+stop -- clean up first (`bastille pkg <jail> clean -ay` across existing
+jails reclaims real space safely, it's just downloaded package cache) or
+get the capacity issue resolved before proceeding. Re-check after every
+1-2 jail creations during a bulk rollout, don't just check once at the
+start and assume it holds for the whole batch.
+
+If MariaDB (or any host-shared service) does go down from disk pressure:
+the rc.d script name may not match the obvious guess (`mysql-server`, not
+`mysql` or `mariadb`, was the actual name on venus) -- check `ls
+/usr/local/etc/rc.d/` rather than guessing. After restarting, verify with
+actual `curl` tests against multiple real sites (not just "is the process
+running") before considering it resolved.
+
 ## Rollout scope discipline

 This was explicitly piloted on **one jail on one host** before being