docs: document the proxy_pass cutover outage and its fix
This commit is contained in:
@@ -157,17 +157,36 @@ state worth keeping (plugins, test data, DB content). Instead:
|
||||
|
||||
## Cutover
|
||||
|
||||
1. Stop (don't destroy) the old jail — keep it as a rollback safety net
|
||||
until the new one is fully proven in real use.
|
||||
2. If a host-level reverse-proxy vhost points at the old jail's IP,
|
||||
update it. (Not needed if the domain was never wired to a public vhost
|
||||
yet — check first with `grep -rl <old-ip> /usr/local/etc/nginx/`.)
|
||||
**Real outage, 2026-08-03: verifying the new jail by curling its IP directly
|
||||
with a `Host:` header does NOT prove the live site works**, because that
|
||||
bypasses the actual production path. The real path is NPM → the jail host's
|
||||
own nginx (a per-domain vhost with `proxy_pass` hardcoded to the jail's IP)
|
||||
→ the jail. Stopping the old jail after only an IP-direct check leaves that
|
||||
vhost still pointing at a now-dead IP — instant, silent outage for every
|
||||
domain migrated this way (hit 5 domains across 2 hosts, funky and venus,
|
||||
before being caught by the user noticing timeouts, not by this process).
|
||||
`migrate-jail-to-subnet.sh` now rewires the host-nginx vhost automatically
|
||||
as its own step (finds the `.conf` referencing the old jail's IP via
|
||||
`bastille list`, `sed`s it to the new IP, `nginx -t` then `-s reload`,
|
||||
aborts and reverts from `.bak` if the syntax check fails) — **but always
|
||||
verify through `curl -H 'Host: <domain>' -H 'X-Forwarded-Proto: https'
|
||||
http://127.0.0.1/` on the jail host itself** (exercises the real vhost path)
|
||||
before trusting a migration, not a direct-by-IP request.
|
||||
|
||||
1. Verify the new jail through the real path (see above), not by IP alone.
|
||||
2. Stop (don't destroy) the old jail — keep it as a rollback safety net
|
||||
until the new one is fully proven in real use. Safe to do now that the
|
||||
vhost itself has already been repointed.
|
||||
3. Destroy the old jail only once genuinely confident — this session left
|
||||
it stopped rather than destroyed on the first migration, as the
|
||||
controlled/cautious default.
|
||||
4. Update any skill/doc/memory that references the old jail name or IP
|
||||
(grep for both across `agent-skills` and this repo's `docs/`) — easy
|
||||
to miss and leaves stale instructions for the next delegate.
|
||||
5. **After any batch of migrations, sweep for stragglers**: `grep -rl
|
||||
'proxy_pass.*192\.168\.' /usr/local/etc/nginx/sites-*/` on the host
|
||||
catches any vhost still pointing at a shared-subnet IP whose jail has
|
||||
since been stopped — the fastest way to confirm nothing was missed.
|
||||
|
||||
## Check disk space BEFORE creating any jail, every time
|
||||
|
||||
|
||||
Reference in New Issue
Block a user