From e04ac02fa2c429e0b8b362802163c291e2364cbc Mon Sep 17 00:00:00 2001 From: Malin Date: Fri, 29 May 2026 17:10:28 +0200 Subject: [PATCH] =?UTF-8?q?feat:=20phase=202=20complete=20=E2=80=94=20zamo?= =?UTF-8?q?lxis=20ready=20for=20cutover?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - rsync: 7.1GB transferred from dracula (exit 0), all years 2012-2026 - wp-config.php: DB_HOST=100.67.166.29, new DB password, Redis constants - W3TC: removed (plugin + all drop-ins: db.php, advanced-cache.php) - W3TC db.php drop-in was W3TC module, not index-wp-mysql-for-speed - Redis Object Cache: plugin installed, object-cache.php drop-in active - .user.ini: open_basedir updated from AApanel path to /var/www/... - WP-CLI: installed (required php85-phar, php85-ctype, php85-filter, php85-tokenizer which were not in the base php85 package) - Kernel tuning: sysctl.conf on both nodes for 1000+ concurrent users - zamolxis: kern.maxfiles=200k, somaxconn=4096, tcp buffers 256KB - decebal: vfs.zfs.arc_max=2GB (critical — prevents ZFS evicting InnoDB) - PHP-FPM: pm.max_children 40→80, start_servers 8→16, spare 16→32 - nginx: worker_connections 4096→8192, worker_rlimit_nofile=65536 - MySQL: max_connections 150→300 (live SET GLOBAL + my.cnf updated) - Pre-cutover checklist items 1-8: all passed - Cutover sequence updated with correct rsync excludes and post-rsync-fixup.sh Pending at cutover: DB import, checklist steps 9-11, NPM upstream switch Co-Authored-By: Claude Sonnet 4.6 --- README.md | 165 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 117 insertions(+), 48 deletions(-) diff --git a/README.md b/README.md index 8c38811..fed1f8e 100644 --- a/README.md +++ b/README.md @@ -942,61 +942,113 @@ curl -s https://connect.facebook.net/ro_RO/sdk.js > ${WEBROOT}/fb-sdk-ro.js ### Pre-cutover checklist (run on zamolxis before touching NPM) +Steps 1–8 below are already verified. Steps 9–10 require the DB import (done at cutover). + ```sh -# 1. Site responds via new stack directly +# 1. PHP extensions — VERIFIED OK +php -m | grep -E "imagick|redis|pdo_mysql|intl|soap|sodium|gd|mbstring" +# opcache: verified active via FPM web request + +# 2. Services running — VERIFIED OK +service nginx status && service php_fpm status && service redis status + +# 3. Redis socket responding — VERIFIED OK +redis-cli -s /var/run/redis/redis.sock ping # → PONG + +# 4. PHP-FPM socket present — VERIFIED OK +ls /var/run/php-fpm-si.sock + +# 5. DB connectivity zamolxis->decebal — VERIFIED OK +# Note: use credentials file to avoid ! shell expansion issue +cat > /tmp/.chk.cnf << EOF +[client] +user=sql_sibiuindepen +password=W0rdPr3ss@SI2026! +host=100.67.166.29 +port=3306 +EOF +mysql --defaults-file=/tmp/.chk.cnf --get-server-public-key \ + -e "SELECT 1 AS db_ok;" +rm -f /tmp/.chk.cnf + +# 6. WordPress core files present — VERIFIED OK +ls /var/www/sibiuindependent.ro/index.php /var/www/sibiuindependent.ro/wp-login.php + +# 7. Redis Object Cache drop-in installed, W3TC absent — VERIFIED OK +ls /var/www/sibiuindependent.ro/wp-content/object-cache.php +ls /var/www/sibiuindependent.ro/wp-content/plugins/w3-total-cache 2>/dev/null || echo "W3TC absent (good)" + +# 8. nginx responds (MISS expected without DB) — VERIFIED OK +curl -sI http://127.0.0.1/ -H "Host: sibiuindependent.ro" | grep X-Cache-Status + +# --- Steps 9-10: run AFTER DB import at cutover --- + +# 9. WordPress returns 200 (post DB import) curl -sk -o /dev/null -w "%{http_code}" \ - http://100.115.128.41/ \ - -H "Host: sibiuindependent.ro" + http://127.0.0.1/ -H "Host: sibiuindependent.ro" # Expected: 200 -# 2. DB connection live -mysql -h 100.67.166.29 \ - -u sql_sibiuindepen -p \ - --get-server-public-key \ - -e "SELECT COUNT(*) FROM wp_posts WHERE post_status='publish';" +# 10. Cache HIT on second request (post DB import) +curl -sI http://127.0.0.1/ -H "Host: sibiuindependent.ro" | grep X-Cache-Status # MISS +curl -sI http://127.0.0.1/ -H "Host: sibiuindependent.ro" | grep X-Cache-Status # HIT -# 3. Redis object cache connected -wp --path=/var/www/sibiuindependent.ro --allow-root redis-cache status - -# 4. PHP extensions present -php85 -m | grep -E "imagick|redis|pdo_mysql|opcache|intl|soap|sodium" - -# 5. nginx fastcgi_cache working — second request should be HIT -curl -sI http://100.115.128.41/ -H "Host: sibiuindependent.ro" \ - | grep X-Cache-Status -# First request: MISS, second request: HIT +# 11. Redis object cache connected (post DB import) +wp --path=/var/www/sibiuindependent.ro --allow-root redis enable +wp --path=/var/www/sibiuindependent.ro --allow-root redis status ``` ### Cutover sequence ``` -T-0 Disable cache warmer crons on OLD dracula (comment out both lines) +T-0 Disable cache warmer crons on OLD dracula (comment out both lines in crontab) -T-1 Final rsync of uploads (incremental delta only): - rsync -avz --progress \ - -e "ssh -i /root/.ssh/id_ed25519 -p 79" \ +T-1 Final rsync — incremental delta only, excludes W3TC and cache: + # Run on zamolxis: + rsync -avz \ + -e "ssh -i /root/.ssh/migration_key -p 79 -o StrictHostKeyChecking=no" \ --exclude="wp-content/cache/" \ - root@dracula:/www/wwwroot/sibiuindependent.ro/wp-content/uploads/ \ - /var/www/sibiuindependent.ro/wp-content/uploads/ + --exclude="wp-content/w3tc-config/" \ + --exclude="wp-content/plugins/w3-total-cache/" \ + --exclude="wp-content/uploads/wpo_wcml_cache/" \ + root@100.99.157.56:/www/wwwroot/sibiuindependent.ro/ \ + /var/www/sibiuindependent.ro/ + + # Then immediately run fixup (rewrites wp-config, re-installs Redis drop-in): + /root/post-rsync-fixup.sh T-2 Final DB dump and import: - ssh root@transilvan \ + # Run from any host with access to both (or from this local machine): + ssh -p 79 root@100.99.157.56 \ "mysqldump -S /tmp/mysqld.sock --single-transaction \ --routines --triggers --events sql_sibiuindepen" \ - | ssh root@decebal "mysql -u root sql_sibiuindepen" + | ssh root@100.67.166.29 \ + "mysql --socket=/var/db/mysql/mysql.sock sql_sibiuindepen" -T-3 Switch NPM upstream: + # ZFS snapshot immediately after import: + ssh root@100.67.166.29 \ + "zfs snapshot data/mysql@post-migration-$(date +%Y%m%d)" + +T-2.5 Run WP-CLI to enable Redis and activate Redis Object Cache plugin: + # Run on zamolxis after DB is imported: + wp --path=/var/www/sibiuindependent.ro --allow-root redis enable + wp --path=/var/www/sibiuindependent.ro --allow-root plugin activate redis-cache + +T-3 Checklist steps 9–11 (see above): verify 200, cache HIT, Redis status + +T-4 Switch NPM upstream: old: 100.99.157.56 (dracula Tailscale IP) new: 100.115.128.41 (zamolxis Tailscale IP) - (single upstream change in NPM — no DNS TTL involved, instant) + (single upstream change in NPM UI — instant, no DNS TTL) -T-4 Smoke test: - - Homepage loads +T-5 Smoke test: + - Homepage loads (public, non-logged-in) - An article page loads - - wp-admin accessible - - Second page load shows X-Cache-Status: HIT + - wp-admin accessible and working + - Second page load for any article: X-Cache-Status: HIT in response headers -T-5 Enable cache warmer on new dracula +T-6 Enable cache warmer cron on zamolxis (already installed, just runs at 03:00/15:00) + # Optionally run immediately: + /root/cache-warmer.sh & ``` **Rollback:** Revert NPM upstream to `100.99.157.56`. Old stack is untouched throughout. No DNS change required. Instant. @@ -1023,27 +1075,44 @@ zfs snapshot data/mysql@post-migration-$(date +%Y%m%d) - [x] Tailscale IPs confirmed — zamolxis: 100.115.128.41 / decebal: 100.67.166.29 - [ ] NPM node Tailscale IP — tighten zamolxis pf HTTP rule to specific NPM IP once confirmed -- [x] PHP 8.5 — binary package on FreeBSD 15.0, all extensions installed. PECL packages use `php85-pecl-*` prefix (imagick, redis). opcache bundled in base php85. -- [ ] Redis Object Cache plugin — confirm if already in wp-content/plugins from old dracula or install fresh -- [ ] W3TC minify in use? — if yes, decide before cutover: drop minify or use separate plugin -- [x] ZFS pool — data pool on /dev/da1 (100GB), ONLINE. data/mysql dataset ONLINE at /var/db/mysql -- [x] MySQL 8.4.9 LTS — deployed and running on decebal -- [x] Beszel and Telegraf — dropped. Proxmox native monitoring sufficient. -- [x] Redis placement — local to zamolxis (unix socket). No 3rd host needed. -- [ ] Phase 2 — rsync WP files from dracula (port 79) to zamolxis /var/www/sibiuindependent.ro -- [ ] Phase 2 — update wp-config.php with DB_HOST=100.67.166.29 and Redis constants -- [ ] Phase 2 — remove W3TC, install Redis Object Cache plugin -- [ ] Phase 3 — pre-cutover checklist verification +- [x] PHP 8.5 — all extensions installed; PECL packages use `php85-pecl-*` prefix; opcache bundled in base php85; phar installed for WP-CLI +- [x] Redis Object Cache plugin — downloaded and drop-in installed at wp-content/object-cache.php +- [ ] W3TC minify in use? — confirm before cutover; if yes: drop (nginx serves pre-minified assets) or use Autoptimize as replacement +- [x] ZFS pool — data pool on /dev/da1 (100GB) ONLINE, data/mysql at /var/db/mysql +- [x] MySQL 8.4.9 LTS — deployed on decebal +- [x] Beszel and Telegraf — dropped +- [x] Redis — local to zamolxis, unix socket +- [x] Phase 2 — rsync complete (7.1GB, exit 0), wp-config updated, W3TC removed +- [x] Phase 2 — pre-cutover checklist steps 1–8 all passed +- [ ] Phase 3 — DB import at cutover (checklist steps 9–11 post-import) - [ ] Phase 3 — NPM upstream switch 100.99.157.56 → 100.115.128.41 ## Deployment Status (as of 2026-05-29) | Node | Status | Notes | |---|---|---| -| zamolxis | **Deployed** | nginx 1.30.2, PHP 8.5.4, Redis 8.6.2, pf active, crons installed | -| decebal | **Deployed** | MySQL 8.4.9, ZFS pool online, pf active, app user created, backup cron installed | +| zamolxis | **Ready for cutover** | All services running, 6.8GB webroot synced, checklist items 1–8 passed | +| decebal | **Ready for cutover** | MySQL 8.4.9 running, ZFS ONLINE, app user verified, backup cron installed | + +### What was tuned for 1000 concurrent users (5× baseline) + +| Setting | zamolxis | decebal | +|---|---|---| +| `kern.maxfiles` | 200,000 | 200,000 | +| `kern.ipc.somaxconn` | 4096 | 1024 | +| `net.inet.tcp.sendspace` | 262,144 | 131,072 | +| `vfs.zfs.arc_max` | — | 2GB (leaves RAM for InnoDB) | +| PHP-FPM `pm.max_children` | 80 (was 40) | — | +| nginx `worker_connections` | 8192 (was 4096) | — | +| nginx `worker_rlimit_nofile` | 65536 | — | +| MySQL `max_connections` | — | 300 (was 150) | +| WP-CLI | Installed at /usr/local/bin/wp | — | ### Connectivity verified - zamolxis → decebal MySQL (100.67.166.29:3306) as `sql_sibiuindepen`: OK -- pf on decebal allows only 100.115.128.41 to reach port 3306: OK -- Backup test run: /var/db/backups/mysql/sql_sibiuindepen_*.sql.gz created OK +- pf on decebal: only 100.115.128.41 reaches port 3306: OK +- Backup cron: /var/db/backups/mysql/sql_sibiuindepen_*.sql.gz runs daily at 02:00 +- nginx fastcgi_cache + PHP-FPM + Redis all responding on zamolxis + +### Known issue to fix at cutover +- W3TC plugin will be re-added by the final rsync (it's on the source). The cutover rsync command above uses `--exclude="wp-content/plugins/w3-total-cache/"` to prevent this. `/root/post-rsync-fixup.sh` must be run after every rsync.