Malin/Migrations
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: phase 2 complete — zamolxis ready for cutover

Browse Source 
- rsync: 7.1GB transferred from dracula (exit 0), all years 2012-2026
- wp-config.php: DB_HOST=100.67.166.29, new DB password, Redis constants
- W3TC: removed (plugin + all drop-ins: db.php, advanced-cache.php)
- W3TC db.php drop-in was W3TC module, not index-wp-mysql-for-speed
- Redis Object Cache: plugin installed, object-cache.php drop-in active
- .user.ini: open_basedir updated from AApanel path to /var/www/...
- WP-CLI: installed (required php85-phar, php85-ctype, php85-filter,
  php85-tokenizer which were not in the base php85 package)
- Kernel tuning: sysctl.conf on both nodes for 1000+ concurrent users
  - zamolxis: kern.maxfiles=200k, somaxconn=4096, tcp buffers 256KB
  - decebal: vfs.zfs.arc_max=2GB (critical — prevents ZFS evicting InnoDB)
- PHP-FPM: pm.max_children 40→80, start_servers 8→16, spare 16→32
- nginx: worker_connections 4096→8192, worker_rlimit_nofile=65536
- MySQL: max_connections 150→300 (live SET GLOBAL + my.cnf updated)
- Pre-cutover checklist items 1-8: all passed
- Cutover sequence updated with correct rsync excludes and post-rsync-fixup.sh

Pending at cutover: DB import, checklist steps 9-11, NPM upstream switch

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Malin
2026-05-29 17:10:28 +02:00
parent 288b79d8eb
commit e04ac02fa2
1 changed files with 117 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files

165
README.md
View File

@@ -942,61 +942,113 @@ curl -s https://connect.facebook.net/ro_RO/sdk.js > ${WEBROOT}/fb-sdk-ro.js
### Pre-cutover checklist (run on zamolxis before touching NPM)
Steps 1–8 below are already verified. Steps 9–10 require the DB import (done at cutover).
```sh
# 1. Site responds via new stack directly
# 1. PHP extensions — VERIFIED OK
php -m | grep -E "imagick|redis|pdo_mysql|intl|soap|sodium|gd|mbstring"
# opcache: verified active via FPM web request
# 2. Services running — VERIFIED OK
service nginx status && service php_fpm status && service redis status
# 3. Redis socket responding — VERIFIED OK
redis-cli -s /var/run/redis/redis.sock ping # → PONG
# 4. PHP-FPM socket present — VERIFIED OK
ls /var/run/php-fpm-si.sock
# 5. DB connectivity zamolxis->decebal — VERIFIED OK
# Note: use credentials file to avoid ! shell expansion issue
cat > /tmp/.chk.cnf << EOF
[client]
user=sql_sibiuindepen
password=W0rdPr3ss@SI2026!
host=100.67.166.29
port=3306
EOF
mysql --defaults-file=/tmp/.chk.cnf --get-server-public-key \
-e "SELECT 1 AS db_ok;"
rm -f /tmp/.chk.cnf
# 6. WordPress core files present — VERIFIED OK
ls /var/www/sibiuindependent.ro/index.php /var/www/sibiuindependent.ro/wp-login.php
# 7. Redis Object Cache drop-in installed, W3TC absent — VERIFIED OK
ls /var/www/sibiuindependent.ro/wp-content/object-cache.php
ls /var/www/sibiuindependent.ro/wp-content/plugins/w3-total-cache 2>/dev/null || echo "W3TC absent (good)"
# 8. nginx responds (MISS expected without DB) — VERIFIED OK
curl -sI http://127.0.0.1/ -H "Host: sibiuindependent.ro" | grep X-Cache-Status
# --- Steps 9-10: run AFTER DB import at cutover ---
# 9. WordPress returns 200 (post DB import)
curl -sk -o /dev/null -w "%{http_code}" \
http://100.115.128.41/ \
-H "Host: sibiuindependent.ro"
http://127.0.0.1/ -H "Host: sibiuindependent.ro"
# Expected: 200
# 2. DB connection live
mysql -h 100.67.166.29 \
-u sql_sibiuindepen -p \
--get-server-public-key \
-e "SELECT COUNT(*) FROM wp_posts WHERE post_status='publish';"
# 10. Cache HIT on second request (post DB import)
curl -sI http://127.0.0.1/ -H "Host: sibiuindependent.ro" | grep X-Cache-Status # MISS
curl -sI http://127.0.0.1/ -H "Host: sibiuindependent.ro" | grep X-Cache-Status # HIT
# 3. Redis object cache connected
wp --path=/var/www/sibiuindependent.ro --allow-root redis-cache status
# 4. PHP extensions present
php85 -m | grep -E "imagick|redis|pdo_mysql|opcache|intl|soap|sodium"
# 5. nginx fastcgi_cache working — second request should be HIT
curl -sI http://100.115.128.41/ -H "Host: sibiuindependent.ro" \
| grep X-Cache-Status
# First request: MISS, second request: HIT
# 11. Redis object cache connected (post DB import)
wp --path=/var/www/sibiuindependent.ro --allow-root redis enable
wp --path=/var/www/sibiuindependent.ro --allow-root redis status
```
### Cutover sequence
```
T-0 Disable cache warmer crons on OLD dracula (comment out both lines)
T-0 Disable cache warmer crons on OLD dracula (comment out both lines in crontab)
T-1 Final rsync of uploads (incremental delta only):
rsync -avz --progress \
-e "ssh -i /root/.ssh/id_ed25519 -p 79" \
T-1 Final rsync — incremental delta only, excludes W3TC and cache:
# Run on zamolxis:
rsync -avz \
-e "ssh -i /root/.ssh/migration_key -p 79 -o StrictHostKeyChecking=no" \
--exclude="wp-content/cache/" \
root@dracula:/www/wwwroot/sibiuindependent.ro/wp-content/uploads/ \
/var/www/sibiuindependent.ro/wp-content/uploads/
--exclude="wp-content/w3tc-config/" \
--exclude="wp-content/plugins/w3-total-cache/" \
--exclude="wp-content/uploads/wpo_wcml_cache/" \
root@100.99.157.56:/www/wwwroot/sibiuindependent.ro/ \
/var/www/sibiuindependent.ro/
# Then immediately run fixup (rewrites wp-config, re-installs Redis drop-in):
/root/post-rsync-fixup.sh
T-2 Final DB dump and import:
ssh root@transilvan \
# Run from any host with access to both (or from this local machine):
ssh -p 79 root@100.99.157.56 \
"mysqldump -S /tmp/mysqld.sock --single-transaction \
--routines --triggers --events sql_sibiuindepen" \
| ssh root@decebal "mysql -u root sql_sibiuindepen"
| ssh root@100.67.166.29 \
"mysql --socket=/var/db/mysql/mysql.sock sql_sibiuindepen"
T-3 Switch NPM upstream:
# ZFS snapshot immediately after import:
ssh root@100.67.166.29 \
"zfs snapshot data/mysql@post-migration-$(date +%Y%m%d)"
T-2.5 Run WP-CLI to enable Redis and activate Redis Object Cache plugin:
# Run on zamolxis after DB is imported:
wp --path=/var/www/sibiuindependent.ro --allow-root redis enable
wp --path=/var/www/sibiuindependent.ro --allow-root plugin activate redis-cache
T-3 Checklist steps 9–11 (see above): verify 200, cache HIT, Redis status
T-4 Switch NPM upstream:
old: 100.99.157.56 (dracula Tailscale IP)
new: 100.115.128.41 (zamolxis Tailscale IP)
(single upstream change in NPM — no DNS TTL involved, instant)
(single upstream change in NPM UI — instant, no DNS TTL)
T-4 Smoke test:
- Homepage loads
T-5 Smoke test:
- Homepage loads (public, non-logged-in)
- An article page loads
- wp-admin accessible
- Second page load shows X-Cache-Status: HIT
- wp-admin accessible and working
- Second page load for any article: X-Cache-Status: HIT in response headers
T-5 Enable cache warmer on new dracula
T-6 Enable cache warmer cron on zamolxis (already installed, just runs at 03:00/15:00)
# Optionally run immediately:
/root/cache-warmer.sh &
```
**Rollback:** Revert NPM upstream to `100.99.157.56`. Old stack is untouched throughout. No DNS change required. Instant.
@@ -1023,27 +1075,44 @@ zfs snapshot data/mysql@post-migration-$(date +%Y%m%d)
- [x] Tailscale IPs confirmed — zamolxis: 100.115.128.41 / decebal: 100.67.166.29
- [ ] NPM node Tailscale IP — tighten zamolxis pf HTTP rule to specific NPM IP once confirmed
- [x] PHP 8.5 — binary package on FreeBSD 15.0, all extensions installed. PECL packages use `php85-pecl-*` prefix (imagick, redis). opcache bundled in base php85.
- [ ] Redis Object Cache plugin — confirm if already in wp-content/plugins from old dracula or install fresh
- [ ] W3TC minify in use? — if yes, decide before cutover: drop minify or use separate plugin
- [x] ZFS pool — data pool on /dev/da1 (100GB), ONLINE. data/mysql dataset ONLINE at /var/db/mysql
- [x] MySQL 8.4.9 LTS — deployed and running on decebal
- [x] Beszel and Telegraf — dropped. Proxmox native monitoring sufficient.
- [x] Redis placement — local to zamolxis (unix socket). No 3rd host needed.
- [ ] Phase 2 — rsync WP files from dracula (port 79) to zamolxis /var/www/sibiuindependent.ro
- [ ] Phase 2 — update wp-config.php with DB_HOST=100.67.166.29 and Redis constants
- [ ] Phase 2 — remove W3TC, install Redis Object Cache plugin
- [ ] Phase 3 — pre-cutover checklist verification
- [x] PHP 8.5 — all extensions installed; PECL packages use `php85-pecl-*` prefix; opcache bundled in base php85; phar installed for WP-CLI
- [x] Redis Object Cache plugin — downloaded and drop-in installed at wp-content/object-cache.php
- [ ] W3TC minify in use? — confirm before cutover; if yes: drop (nginx serves pre-minified assets) or use Autoptimize as replacement
- [x] ZFS pool — data pool on /dev/da1 (100GB) ONLINE, data/mysql at /var/db/mysql
- [x] MySQL 8.4.9 LTS — deployed on decebal
- [x] Beszel and Telegraf — dropped
- [x] Redis — local to zamolxis, unix socket
- [x] Phase 2 — rsync complete (7.1GB, exit 0), wp-config updated, W3TC removed
- [x] Phase 2 — pre-cutover checklist steps 1–8 all passed
- [ ] Phase 3 — DB import at cutover (checklist steps 9–11 post-import)
- [ ] Phase 3 — NPM upstream switch 100.99.157.56 → 100.115.128.41
## Deployment Status (as of 2026-05-29)
| Node | Status | Notes |
|---|---|---|
| zamolxis | **Deployed** | nginx 1.30.2, PHP 8.5.4, Redis 8.6.2, pf active, crons installed |
| decebal | **Deployed** | MySQL 8.4.9, ZFS pool online, pf active, app user created, backup cron installed |
| zamolxis | **Ready for cutover** | All services running, 6.8GB webroot synced, checklist items 1–8 passed |
| decebal | **Ready for cutover** | MySQL 8.4.9 running, ZFS ONLINE, app user verified, backup cron installed |
### What was tuned for 1000 concurrent users (5× baseline)
| Setting | zamolxis | decebal |
|---|---|---|
| `kern.maxfiles` | 200,000 | 200,000 |
| `kern.ipc.somaxconn` | 4096 | 1024 |
| `net.inet.tcp.sendspace` | 262,144 | 131,072 |
| `vfs.zfs.arc_max` | — | 2GB (leaves RAM for InnoDB) |
| PHP-FPM `pm.max_children` | 80 (was 40) | — |
| nginx `worker_connections` | 8192 (was 4096) | — |
| nginx `worker_rlimit_nofile` | 65536 | — |
| MySQL `max_connections` | — | 300 (was 150) |
| WP-CLI | Installed at /usr/local/bin/wp | — |
### Connectivity verified
- zamolxis → decebal MySQL (100.67.166.29:3306) as `sql_sibiuindepen`: OK
- pf on decebal allows only 100.115.128.41 to reach port 3306: OK
- Backup test run: /var/db/backups/mysql/sql_sibiuindepen_*.sql.gz created OK
- pf on decebal: only 100.115.128.41 reaches port 3306: OK
- Backup cron: /var/db/backups/mysql/sql_sibiuindepen_*.sql.gz runs daily at 02:00
- nginx fastcgi_cache + PHP-FPM + Redis all responding on zamolxis
### Known issue to fix at cutover
- W3TC plugin will be re-added by the final rsync (it's on the source). The cutover rsync command above uses `--exclude="wp-content/plugins/w3-total-cache/"` to prevent this. `/root/post-rsync-fixup.sh` must be run after every rsync.