docs: document the proxy_pass cutover outage and its fix

This commit is contained in:
2026-08-03 14:29:27 +02:00
parent 4eff29939e
commit 8dbe1eb9c1
@@ -157,17 +157,36 @@ state worth keeping (plugins, test data, DB content). Instead:
## Cutover
1. Stop (don't destroy) the old jail — keep it as a rollback safety net
until the new one is fully proven in real use.
2. If a host-level reverse-proxy vhost points at the old jail's IP,
update it. (Not needed if the domain was never wired to a public vhost
yet — check first with `grep -rl <old-ip> /usr/local/etc/nginx/`.)
**Real outage, 2026-08-03: verifying the new jail by curling its IP directly
with a `Host:` header does NOT prove the live site works**, because that
bypasses the actual production path. The real path is NPM → the jail host's
own nginx (a per-domain vhost with `proxy_pass` hardcoded to the jail's IP)
→ the jail. Stopping the old jail after only an IP-direct check leaves that
vhost still pointing at a now-dead IP — instant, silent outage for every
domain migrated this way (hit 5 domains across 2 hosts, funky and venus,
before being caught by the user noticing timeouts, not by this process).
`migrate-jail-to-subnet.sh` now rewires the host-nginx vhost automatically
as its own step (finds the `.conf` referencing the old jail's IP via
`bastille list`, `sed`s it to the new IP, `nginx -t` then `-s reload`,
aborts and reverts from `.bak` if the syntax check fails) — **but always
verify through `curl -H 'Host: <domain>' -H 'X-Forwarded-Proto: https'
http://127.0.0.1/` on the jail host itself** (exercises the real vhost path)
before trusting a migration, not a direct-by-IP request.
1. Verify the new jail through the real path (see above), not by IP alone.
2. Stop (don't destroy) the old jail — keep it as a rollback safety net
until the new one is fully proven in real use. Safe to do now that the
vhost itself has already been repointed.
3. Destroy the old jail only once genuinely confident — this session left
it stopped rather than destroyed on the first migration, as the
controlled/cautious default.
4. Update any skill/doc/memory that references the old jail name or IP
(grep for both across `agent-skills` and this repo's `docs/`) — easy
to miss and leaves stale instructions for the next delegate.
5. **After any batch of migrations, sweep for stragglers**: `grep -rl
'proxy_pass.*192\.168\.' /usr/local/etc/nginx/sites-*/` on the host
catches any vhost still pointing at a shared-subnet IP whose jail has
since been stopped — the fastest way to confirm nothing was missed.
## Check disk space BEFORE creating any jail, every time