From 4eff29939edfdc164a6e692b4747bc410acb415d Mon Sep 17 00:00:00 2001 From: Malin Date: Mon, 3 Aug 2026 13:47:46 +0200 Subject: [PATCH] docs: add disk-space-check discipline after a real venus MariaDB outage caused by bulk jail creation --- .../jail-dedicated-subnet-migration/SKILL.md | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/skills/jail-dedicated-subnet-migration/SKILL.md b/skills/jail-dedicated-subnet-migration/SKILL.md index 051d6f8..6aad9d8 100644 --- a/skills/jail-dedicated-subnet-migration/SKILL.md +++ b/skills/jail-dedicated-subnet-migration/SKILL.md @@ -169,6 +169,33 @@ state worth keeping (plugins, test data, DB content). Instead: (grep for both across `agent-skills` and this repo's `docs/`) — easy to miss and leaves stale instructions for the next delegate. +## Check disk space BEFORE creating any jail, every time + +**Do not assume "blank slate, plenty of room" holds on every host** just +because it held on the pilot host. Confirmed real incident on the second +host attempted (venus, 2026-08-03): creating 3 new full-stack jails +(~200MB retained footprint each after packages) pushed an already-96%-full +ZFS pool over the edge and **crashed the shared MariaDB instance**, taking +down every site on the host (a real, if brief, production outage) -- +this exact failure mode had already been documented once before on this +same host from an unrelated earlier incident, and should have been +checked proactively rather than triggered again. + +**Before creating even one new jail**, run `zfs list zroot/ROOT/default -o +avail` (or `df -h` if not ZFS) and treat anything under a few GB as a hard +stop -- clean up first (`bastille pkg clean -ay` across existing +jails reclaims real space safely, it's just downloaded package cache) or +get the capacity issue resolved before proceeding. Re-check after every +1-2 jail creations during a bulk rollout, don't just check once at the +start and assume it holds for the whole batch. + +If MariaDB (or any host-shared service) does go down from disk pressure: +the rc.d script name may not match the obvious guess (`mysql-server`, not +`mysql` or `mariadb`, was the actual name on venus) -- check `ls +/usr/local/etc/rc.d/` rather than guessing. After restarting, verify with +actual `curl` tests against multiple real sites (not just "is the process +running") before considering it resolved. + ## Rollout scope discipline This was explicitly piloted on **one jail on one host** before being