Malin/Migrations
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
Migrations/sibiuindependent.ro.MD
Malin 1a07c11c4b fix: nginx real IP, pf open HTTP, homepage cache purge on publish 
- nginx real_ip_header X-Real-IP: access log and PHP now see true client IPs
- pf: HTTP open to all (WAF on NPM handles filtering)
- sibiu-independent-tweaks: si_purge_nginx_homepage_cache() hook deletes
  homepage fastcgi cache file on post publish/unpublish — fixes stale
  content for anonymous (mobile) users
- SSH config: updated to direct Tailscale IPs, removed old migration proxy
- sibiuindependent.ro.MD: documented issues 9-11 and fixes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-01 16:21:29 +02:00

740 lines
25 KiB
Markdown
Raw Permalink Blame History

# sibiuindependent.ro — Stack Migration
## May 2026
Migration of `sibiuindependent.ro` from Debian 12 + AApanel to FreeBSD 15.0 minimal stack.
**Status: LIVE as of 2026-05-31. Migration complete.**
---
## Live Architecture
```
[internet] → [NPM node, 100.89.238.4 Tailscale] → [zamolxis 100.115.128.41] → [decebal 100.67.166.29]
```
| Node | Role | OS | vCPU | RAM | Disk | Tailscale IP |
|---|---|---|---|---|---|---|
| zamolxis | Web | FreeBSD 15.0-p9 | 6 | 24GB | 80GB HDD | 100.115.128.41 |
| decebal | Database | FreeBSD 15.0-p9 | 6 | 30GB | 20GB OS + 100GB ZFS data (SSD) | 100.67.166.29 |
Both VMs are on separate bare-metal Proxmox hosts. All inter-service traffic runs over Tailscale. NPM handles TLS termination. No public-facing ports on zamolxis or decebal.
---
## Stack
### zamolxis (web)
| Component | Version / Detail |
|---|---|
| OS | FreeBSD 15.0-RELEASE-p9 |
| Web server | nginx 1.30.2 |
| PHP | 8.5.4 + FPM (unix socket `/var/run/php-fpm-si.sock`) |
| Page cache | nginx fastcgi_cache (`/var/cache/nginx/si`, 4GB max) |
| Object cache | Redis Object Cache plugin v2.8.0 (Till Krüss) via unix socket |
| Bytecode cache | OPcache 256MB, 20000 files, JIT **disabled** (caused FPM crashes with WordPress plugin stacks) |
| Redis | 8.6.2, unix socket `/var/run/redis/redis.sock`, maxmemory 3GB allkeys-lru |
| Sessions | Redis via `session.save_path = unix:///var/run/redis/redis.sock` |
| Media | wp-content/uploads local (6.5GB), S3 offload via WPS3Media to `s3.palmasolutions.net/sibiuindependent` (101,145 files in `ddf44d_as3cf_items`) |
| Login URL | `/intrarepresa` (Perfmatters custom login — `/wp-login.php` and `/wp-admin` return 403) |
| WP prefix | `ddf44d_` |
| Firewall | pf — HTTP open to all (WAF on NPM), SSH only from mgmt + Tailscale |
| Real IP | nginx `real_ip_header X-Real-IP` from NPM — access log and plugins see true client IPs |
**Active PHP extensions:** bcmath, curl, exif, fileinfo, gd, imagick, intl, mbstring, mysqli, opcache, pcntl, pdo_mysql, pecl-redis, posix, simplexml, soap, sockets, sodium, xml, xmlreader, xmlwriter, xsl, zip
Note: `php85-simplexml` is required by WPS3Media (AWS SDK XML parsing). Without it, the entire WPS3Media plugin fails to instantiate silently.
### decebal (db)
| Component | Version / Detail |
|---|---|
| OS | FreeBSD 15.0-RELEASE-p9 |
| Database | MySQL 8.4.9 LTS |
| Database name | `sql_sibiuindepen` |
| Socket | `/var/db/mysql/mysql.sock` |
| Data directory | `/var/db/mysql` (ZFS pool `data/mysql`) |
| ZFS settings | recordsize=16K, compression=lz4, primarycache=metadata |
| InnoDB buffer pool | 26GB (dataset ~1.85GB — fits entirely in RAM) |
| Firewall | pf — MySQL only from zamolxis (100.115.128.41), SSH only from mgmt + Tailscale |
| Backup | `/root/db-backup.sh` daily 02:00, 7-day retention in `/var/db/backups/mysql/` |
| ZFS snapshots | Weekly (Sunday 00:30), last 4 kept |
---
## Migration Source (decommissioned)
| Node | Role | OS | Tailscale IP |
|---|---|---|---|
| dracula | Web | Debian 12 + AApanel | 100.99.157.56 |
| transilvan | Database | Debian 12 + AApanel | — |
| transilvan MySQL socket | — | — | `/tmp/mysqld.sock` |
---
## OS Selection
### Candidates: OpenBSD, FreeBSD, Gentoo, Alpine Linux
**OpenBSD** — best security (pledge, unveil, W^X), but PHP 8.5 not in packages, MySQL 8.x absent, imagick friction. Viable only with PHP 8.4 + MariaDB.
**FreeBSD** — pf, Capsicum, ZFS native, PHP 8.5 + all extensions as binary packages, MySQL 8.4 LTS, Redis 8.x, official Tailscale. Strongest all-round candidate.
**Gentoo** — genuine minimalism via USE flags, but every update is a compile run. Production liability.
**Alpine Linux** — easiest migration (still Linux), musl + busybox, apk. No ZFS, no jails. Best minimal Linux option.
**Decision: FreeBSD 15.0** — ZFS native on the DB node is the tiebreaker (`zfs snapshot` before any migration step). FreeBSD 15.0 over 14.x specifically because PHP 8.5 is a binary package on 15.0.
---
## What Was Dropped
| Component | Reason |
|---|---|
| AApanel / btpanel | Replaced by direct config management |
| W3 Total Cache | Replaced by nginx fastcgi_cache + Redis Object Cache |
| Memcached | Replaced by Redis (unix socket, faster, simpler) |
| Fail2ban / SSHGuard | No public SSH port, Tailscale-only |
| Postfix / sendmail | SMTP plugin in use, no local MTA needed |
| Wazuh / Five Nines / Beszel / Telegraf | Not required — Proxmox native monitoring covers VM metrics |
| memory-cleanup.sh | Linux `/proc` hack, was on wrong server anyway |
| ngxblocker | Not applicable |
| AApanel cron scripts | Panel-specific: acme renewal, site_total, backup.py |
## Caching Architecture
| Layer | Old | New |
|---|---|---|
| Page cache | W3TC on-disk (4.8GB, PHP-generated) | nginx fastcgi_cache (PHP bypassed entirely on HIT) |
| Object cache | W3TC + Memcached (TCP) | Redis Object Cache + Redis (unix socket) |
| Bytecode cache | OPcache | OPcache (JIT disabled) |
| Sessions | PHP file sessions | Redis via unix socket |
---
## Host Hardware
| Host | CPU | RAM | Storage | VM |
|---|---|---|---|---|
| zamolxis host | Xeon E5-1620 v2 @ 3.70GHz (8 logical) | 64GB | Datacenter HDD | zamolxis (ID 900) |
| decebal host | Xeon E5-1620 v2 @ 3.70GHz (8 logical) | 32GB | SSD 160GB | decebal (ID 901) — sole VM |
---
## VM Provisioning
### 1. FreeBSD 15.0 ISO
```sh
wget -P /var/lib/vz/template/iso/ \
https://download.freebsd.org/releases/amd64/amd64/ISO-IMAGES/15.0/FreeBSD-15.0-RELEASE-amd64-disc1.iso
```
### 2. zamolxis (ID 900, HDD host)
```sh
qm create 900 \
--name zamolxis \
--memory 24576 --balloon 24576 \
--cores 6 --cpu host \
--machine q35 --bios ovmf \
--efidisk0 local:1,format=raw \
--net0 virtio,bridge=vmbr0 \
--ostype other \
--scsihw virtio-scsi-single \
--scsi0 local:80,format=raw \
--cdrom local:iso/FreeBSD-15.0-RELEASE-amd64-disc1.iso \
--boot order=ide2;scsi0 \
--agent enabled=1
```
### 3. decebal (ID 901, SSD host — sole VM)
```sh
qm create 901 \
--name decebal \
--memory 30720 --balloon 0 \
--cores 6 --cpu host \
--machine q35 --bios ovmf \
--efidisk0 local:1,format=raw \
--net0 virtio,bridge=vmbr0 \
--ostype other \
--scsihw virtio-scsi-single \
--scsi0 local:20,format=raw,discard=on,ssd=1 \
--cdrom local:iso/FreeBSD-15.0-RELEASE-amd64-disc1.iso \
--boot order=ide2;scsi0 \
--agent enabled=1
qm set 901 --scsi1 local:100,format=raw,discard=on,ssd=1
```
`--balloon 0` is required — memory ballooning must not reclaim InnoDB buffer pool pages.
### 4. FreeBSD installer settings (both VMs)
```
Welcome → Install
Hostname → zamolxis (or decebal)
Distribution → base, kernel only
Partitioning → Auto (ZFS), stripe, da0, swap 2GB, lz4
Network → vtnet0, temporary IP for pkg bootstrap
SSHD → enable
```
### 5. First boot (both VMs)
```sh
freebsd-update fetch install
pkg update && pkg upgrade -y
pkg install -y qemu-guest-agent tailscale sudo curl wget
sysrc qemu_guest_agent_enable=YES
service qemu-guest-agent start
sysrc tailscaled_enable=YES
service tailscaled start
tailscale up --hostname=zamolxis # or decebal
tailscale ip -4
```
---
## Phase 1 — DB Node (decebal)
### ZFS
```sh
zpool create -o ashift=12 data /dev/da1
zfs create -o mountpoint=/var/db/mysql data/mysql
zfs set recordsize=16K data/mysql
zfs set primarycache=metadata data/mysql
zfs set compression=lz4 data/mysql
```
### MySQL 8.4 LTS
MySQL 8.4.9 is the current LTS branch. MySQL 9.0/9.1 were Oracle's "innovation" track — both EOL and removed from FreeBSD ports.
```sh
pkg install mysql84-server
sysrc mysql_enable=YES
sysrc mysql_dbdir=/var/db/mysql
```
`/usr/local/etc/mysql/my.cnf`:
```ini
[mysqld]
bind-address = 100.67.166.29
port = 3306
mysqlx = 0
datadir = /var/db/mysql
socket = /var/db/mysql/mysql.sock
log_error = /var/db/mysql/decebal.err
pid-file = /var/db/mysql/decebal.pid
innodb_buffer_pool_size = 26G
innodb_buffer_pool_instances = 8
innodb_redo_log_capacity = 2G
innodb_flush_log_at_trx_commit = 1
innodb_flush_method = O_DIRECT
innodb_io_capacity = 2000
innodb_io_capacity_max = 4000
max_connections = 300
thread_cache_size = 16
table_open_cache = 4000
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
slow_query_log = 1
slow_query_log_file = /var/db/mysql/slow.log
long_query_time = 2
binlog_expire_logs_seconds = 259200
skip-name-resolve
sql_mode = STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
[client]
socket = /var/db/mysql/mysql.sock
```
Note: `innodb_redo_log_capacity` replaces deprecated `innodb_log_file_size` in MySQL 8.4.
### pf (decebal)
`/etc/pf.conf`:
```
ext_if = "vtnet0"
ts_if = "tailscale0"
mgmt = "{ 127.0.0.0/8, 192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12, 100.64.0.0/10, 194.56.239.153, 109.69.48.0/24 }"
zamolxis_ts = "100.115.128.41"
set skip on lo0
set block-policy drop
scrub in all
block in all
block out quick inet6 all
pass out all keep state
pass in on $ext_if proto tcp from $mgmt to any port 22 keep state
pass in on $ts_if proto tcp to any port 22 keep state
pass in on $ts_if proto tcp from $zamolxis_ts to any port 3306 keep state
```
### DB Backup (decebal)
`/root/db-backup.sh`:
```sh
#!/bin/sh
BACKUP_DIR="/var/db/backups/mysql"
DB="sql_sibiuindepen"
DATE=$(date +%Y%m%d_%H%M%S)
LOG="/var/log/db-backup.log"
mkdir -p "$BACKUP_DIR"
echo "[$(date)] Starting backup of $DB" >> "$LOG"
/usr/local/bin/mysqldump \
--socket=/var/db/mysql/mysql.sock \
--single-transaction \
--quick \
--routines \
--triggers \
--set-gtid-purged=OFF \
"$DB" | gzip > "${BACKUP_DIR}/${DB}_${DATE}.sql.gz"
if [ $? -eq 0 ]; then
SIZE=$(du -sh "${BACKUP_DIR}/${DB}_${DATE}.sql.gz" | cut -f1)
echo "[$(date)] Backup OK: ${DB}_${DATE}.sql.gz ($SIZE)" >> "$LOG"
else
echo "[$(date)] BACKUP FAILED for $DB" >> "$LOG"
exit 1
fi
find "$BACKUP_DIR" -name "${DB}_*.sql.gz" -mtime +7 -delete
echo "[$(date)] Pruned backups older than 7 days" >> "$LOG"
```
Root crontab on decebal (`crontab -l`):
```
# Daily MySQL dump at 02:00
0 2 * * * /root/db-backup.sh
# Weekly ZFS snapshot (Sunday 00:30)
30 0 * * 0 zfs snapshot data/mysql@weekly-$(date +%Y%m%d)
# Prune ZFS snapshots older than 4 weeks (Sunday 01:00)
0 1 * * 0 zfs list -H -t snapshot -o name data/mysql | sort | head -n -4 | xargs -r zfs destroy
```
Note: user crontab format — no `user` field between time spec and command (that is `/etc/crontab` format).
---
## Phase 2 — Web Node (zamolxis)
### Packages
```sh
pkg install -y nginx redis git curl wget
sysrc nginx_enable=YES redis_enable=YES
service redis start
pkg install -y php85 \
php85-bcmath php85-curl php85-exif php85-fileinfo \
php85-gd php85-pecl-imagick php85-intl php85-mbstring \
php85-mysqli php85-pcntl php85-pdo_mysql php85-posix \
php85-pecl-redis php85-simplexml php85-soap php85-sockets \
php85-sodium php85-xml php85-xmlreader php85-xmlwriter \
php85-xsl php85-zip ImageMagick7
sysrc php_fpm_enable=YES
```
`php85-simplexml` is mandatory — WPS3Media uses the AWS SDK which requires SimpleXML. Without it the plugin silently fails to instantiate.
### PHP-FPM pool
`/usr/local/etc/php-fpm.d/sibiuindependent.conf`:
```ini
[sibiuindependent]
user = www
group = www
listen = /var/run/php-fpm-si.sock
listen.owner = www
listen.group = www
pm = dynamic
pm.max_children = 80
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_spare_servers = 16
pm.max_requests = 500
```
`/usr/local/etc/php/php.ini` (relevant):
```ini
opcache.enable = 1
opcache.memory_consumption = 256
opcache.interned_strings_buffer = 32
opcache.max_accelerated_files = 20000
opcache.validate_timestamps = 0
opcache.jit = 0
opcache.jit_buffer_size = 0
session.save_handler = redis
session.save_path = "unix:///var/run/redis/redis.sock"
```
JIT is explicitly disabled. WordPress plugin stacks (especially Yoast + TagDiv composer) caused PHP-FPM crashes under JIT on this install.
### nginx
`/usr/local/etc/nginx/nginx.conf`:
```nginx
worker_processes auto;
worker_rlimit_nofile 65536;
events { worker_connections 8192; }
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
tcp_nopush on;
server_tokens off;
fastcgi_cache_path /var/cache/nginx/si
levels=1:2
keys_zone=si_cache:64m
max_size=4g
inactive=24h
use_temp_path=off;
fastcgi_cache_key "$scheme$request_method$host$request_uri";
server {
listen 80;
server_name sibiuindependent.ro www.sibiuindependent.ro;
root /var/www/sibiuindependent.ro;
index index.php;
set $skip_cache 0;
if ($request_method = POST) { set $skip_cache 1; }
if ($query_string) { set $skip_cache 1; }
if ($request_uri ~* "/wp-admin/|/wp-login|/xmlrpc|/feed|sitemap") {
set $skip_cache 1; }
if ($http_cookie ~* "comment_author|wordpress_[a-f0-9]+|wp-postpass|wordpress_logged_in") {
set $skip_cache 1; }
location / {
try_files $uri $uri/ /index.php?$args;
}
location ~ \.php$ {
fastcgi_pass unix:/var/run/php-fpm-si.sock;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_cache si_cache;
fastcgi_cache_valid 200 301 302 24h;
fastcgi_cache_bypass $skip_cache;
fastcgi_no_cache $skip_cache;
add_header X-Cache-Status $upstream_cache_status;
}
location ~* \.(js|css|png|jpg|jpeg|gif|webp|svg|ico|woff|woff2|ttf|eot|mp4|webm|ogv)$ {
expires 1y;
add_header Cache-Control "public, immutable";
access_log off;
log_not_found off;
}
}
}
```
`webp` must be in the static location regex. WPS3Media serves WebP variants from S3; without it, `.webp` requests fall through to PHP-FPM and trigger a full WordPress load for a file that doesn't exist locally.
### Redis
`/usr/local/etc/redis.conf` (relevant):
```
maxmemory 3gb
maxmemory-policy allkeys-lru
unixsocket /var/run/redis/redis.sock
unixsocketperm 777
```
### pf (zamolxis)
`/etc/pf.conf`:
```
ext_if = "vtnet0"
ts_if = "tailscale0"
mgmt = "{ 127.0.0.0/8, 192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12, 100.64.0.0/10, 194.56.239.153, 109.69.48.0/24 }"
npm_ts = "100.89.238.4"
set skip on lo0
set block-policy drop
scrub in all
block in all
block out quick inet6 all
pass out all keep state
pass in on $ext_if proto tcp from $mgmt to any port 22 keep state
pass in on $ts_if proto tcp to any port 22 keep state
pass in on $ts_if proto tcp from $npm_ts to any port 80 keep state
```
### wp-config.php changes
```php
define( 'DB_HOST', '100.67.166.29' );
define( 'DB_PASSWORD', 'W0rdPr3ss@SI2026!' );
define( 'WP_REDIS_SCHEME', 'unix' );
define( 'WP_REDIS_PATH', '/var/run/redis/redis.sock' );
define( 'WP_DEBUG', false );
define( 'WP_DEBUG_LOG', false );
define( 'WP_DEBUG_DISPLAY', false );
```
### post-rsync-fixup.sh
`/tmp/post-rsync-fixup.sh` — run after every rsync:
- Rewrites `DB_HOST` and `DB_PASSWORD` in wp-config.php
- Removes W3TC plugin and its drop-ins (advanced-cache.php, db.php)
- Installs Redis Object Cache drop-in if missing
- Fixes `.user.ini` open_basedir
- Fixes ownership (`www:www`) and wp-config permissions (640)
- Reloads PHP-FPM
### Kernel tuning (sysctl.conf, both VMs)
```
kern.maxfiles=200000
kern.ipc.somaxconn=4096
net.inet.tcp.sendspace=262144
net.inet.tcp.recvspace=262144
```
Additional on zamolxis:
```
vfs.zfs.arc_max=2147483648 # cap ZFS ARC at 2GB, leave RAM for PHP-FPM + nginx UBC
```
### Crons (zamolxis root crontab)
```
*/5 * * * * curl -s -o /dev/null https://sibiuindependent.ro/wp-cron.php?doing_wp_cron
0 3 * * * /root/cache-warmer.sh > /dev/null 2>&1
0 3 * * 1 /root/js-refresh.sh >> /root/js-refresh.log 2>&1
@reboot sleep 60 && /root/cache-warmer.sh > /dev/null 2>&1
```
---
## Cache Warmer
`/root/cache-warmer.sh` — reads Yoast sitemap index (89 sub-sitemaps), warms all URLs via HTTP directly to nginx on port 80 (nginx does not serve HTTPS; TLS is terminated at NPM).
```sh
#!/bin/sh
DOMAIN="sibiuindependent.ro"
NGINX="127.0.0.1"
CONCURRENCY=6
SITEMAP_INDEX=$(curl -s --max-time 30 \
--resolve "${DOMAIN}:80:${NGINX}" \
"http://${DOMAIN}/sitemap_index.xml")
SITEMAPS=$(echo "$SITEMAP_INDEX" | grep -oE 'https?://[^<]+\.xml' \
| sed "s|https://${DOMAIN}|http://${DOMAIN}|")
URLS=$(for SM in $SITEMAPS; do
curl -s --max-time 30 --resolve "${DOMAIN}:80:${NGINX}" "$SM" \
| grep -oE 'https?://[^<]+' | grep -v '\.xml' \
| sed "s|https://${DOMAIN}|http://${DOMAIN}|"
done)
echo "$URLS" | grep 'http' | xargs -P $CONCURRENCY -I{} sh -c \
'curl -s -o /dev/null --max-time 20 --resolve "'"${DOMAIN}"':80:'"${NGINX}"'" "{}" 2>/dev/null'
```
---
## JS Asset Refresh
`/root/js-refresh.sh` — weekly, Monday 03:00:
```sh
#!/bin/sh
WEBROOT=/var/www/sibiuindependent.ro
curl -s https://cdn.onesignal.com/sdks/OneSignalSDK.js > ${WEBROOT}/onesignal.js
curl -s https://cdn.onesignal.com/sdks/OneSignalPageSDKES6.js > ${WEBROOT}/onesignal-es6.js
curl -s https://connect.facebook.net/en_US/sdk.js > ${WEBROOT}/fb-sdk.js
curl -s https://connect.facebook.net/ro_RO/sdk.js > ${WEBROOT}/fb-sdk-ro.js
```
---
## DB Migration
Source: transilvan (Debian 12 + MariaDB), socket `/tmp/mysqld.sock`, SSH port 79.
Destination: decebal (FreeBSD 15 + MySQL 8.4.9), socket `/var/db/mysql/mysql.sock`.
```sh
# Run from zamolxis (has access to both via Tailscale/direct)
ssh -p 79 root@100.99.157.56 \
"mysqldump -S /tmp/mysqld.sock --single-transaction --no-tablespaces \
--routines --triggers --events sql_sibiuindepen" \
| mysql --defaults-file=/tmp/.imp.cnf -f --get-server-public-key sql_sibiuindepen
```
Notes:
- `--no-tablespaces` instead of `--set-gtid-purged=OFF` on MariaDB source (MariaDB does not support that flag)
- `--defaults-file` must be the **first** argument to `mysql`
- `-f` (force) was needed due to a duplicate key in `ddf44d_yoast_indexable_hierarchy` — Yoast's indexable table had integrity issues, non-critical (regenerates on save)
- Final import: 143,266 posts, 121 tables, 1,850MB
---
## Post-Cutover Issues & Fixes
### 1. S3 images not loading — WPS3Media class not instantiating
**Root cause:** `php85-simplexml` not installed. WPS3Media's AWS SDK compat check (`AS3CF_Pro_Installer`) detected missing SimpleXML and blocked the entire plugin from loading with "no SimpleXML PHP module." No PHP error was emitted — plugin simply never registered its `wp_get_attachment_url` filter.
**Fix:** `pkg install -y php85-simplexml php85-xmlwriter && service php_fpm restart`
**Verification:** `Amazon_S3_And_CloudFront_Pro` class now instantiates; URLs rewrite to `https://s3.palmasolutions.net/...`. 101,145 files tracked in `ddf44d_as3cf_items`.
### 2. wp-admin "This has been disabled." — by design
Perfmatters plugin custom login URL feature: `login_url: intrarepresa`. Both `/wp-login.php` and `/wp-admin` are blocked with a 403. Login is at `/intrarepresa`.
### 3. Articles redirecting to example.com
Two concurrent causes:
**a) Empty `permalink_structure`** — the DB import came from a server that had the structure set, but something reset it on the new DB. Pretty URLs were broken entirely.
Fix: `wp --allow-root rewrite structure '/%year%/%monthnum%/%day%/%postname%/' --hard && wp rewrite flush`
**b) InformatiQ-Toolkit security plugin** — was configured with `redirect_url: https://example.com/blocked` and `response_code: 301_custom`. Any request the plugin classified as a bot or bad actor got a 301 to example.com, which was then cached by nginx (`fastcgi_cache_valid 200 301 302 24h`) for 24 hours.
Fix: `response_code` changed to `403`, `redirect_url` cleared.
Note: nginx's `fastcgi_cache_valid` should not cache 301/302. Consider changing to `fastcgi_cache_valid 200 24h` only.
### 4. Redis Object Cache plugin inactive
The `object-cache.php` drop-in was installed and functioning (Redis was receiving cache hits). However the plugin itself was marked inactive in WordPress — WP-CLI `wp redis` commands and the admin UI status panel were non-functional.
Fix: `wp --allow-root plugin activate redis-cache`
### 5. DB backup cron never ran
The root crontab had `0 2 * * * root /root/db-backup.sh` — with `root` as a spurious user field (valid in `/etc/crontab` system format, not in user crontab format). Cron was trying to execute `root /root/db-backup.sh` as the command, which failed silently. No backups had run since before the migration.
Fix: removed the `root` field. First successful backup: 137MB, completed in 19 seconds.
### 6. Redis maxmemory unbounded
Redis `maxmemory` was 0 (no limit) with `noeviction` policy. On a server with competing workloads, unbounded Redis with noeviction risks OOM if the object cache grows unexpectedly.
Fix: `maxmemory 3gb`, `maxmemory-policy allkeys-lru`, persisted to `/usr/local/etc/redis.conf`.
### 7. WP_DEBUG left on after migration
`WP_DEBUG = true` was generating PHP notices on every uncached request, logged to nginx error log. Overhead on both disk I/O and PHP execution.
Fix: `WP_DEBUG`, `WP_DEBUG_LOG`, `WP_DEBUG_DISPLAY` all set to `false`.
### 8. nginx static location missing webp
The static file location regex did not include `webp`. WPS3Media serves WebP variants; without the static location match, `.webp` requests fell through to PHP-FPM, which loaded WordPress, checked the DB, found nothing local, and generated errors.
Fix: added `webp|mp4|webm|ogv` to the static location regex.
### 9. Login blocked for all users — ASE Limit Login Attempts locked NPM's IP
All traffic passes through NPM (Tailscale 100.89.238.4). nginx passed `REMOTE_ADDR = 100.89.238.4` to PHP. The Admin Site Enhancements plugin's Limit Login Attempts feature tracked all failed logins against this single IP. After multiple users tried and failed, the 24h lockout triggered — blocking everyone from `/intrarepresa`.
Fix (immediate): ASE deactivated.
Fix (permanent): nginx `real_ip_header X-Real-IP` + `set_real_ip_from 100.89.238.4` configured. `$remote_addr` now reflects true client IP from NPM's `X-Real-IP` header. Access log and all PHP plugins see real per-user IPs. If ASE is re-enabled, Limit Login Attempts will work correctly per-user.
### 10. pf HTTP restriction opened — WAF on NPM handles filtering
HTTP was restricted to NPM's Tailscale IP only. Opened to all: `pass in proto tcp from any to any port 80 keep state`. WAF rules on NPM handle filtering upstream.
### 11. Stale nginx cache on homepage for anonymous (mobile) users
No cache invalidation on post publish. Anonymous users (typically mobile) got the nginx-cached homepage for up to 24h after new posts were published. Logged-in users (desktop) bypassed cache via `wordpress_logged_in` cookie.
Fix: `si_purge_nginx_homepage_cache()` hook added to `sibiu-independent-tweaks/sb.php`. On `transition_post_status` (publish/unpublish), it calculates the homepage cache file path (MD5 of `httpGETsibiuindependent.ro/`, nginx `levels=1:2`) and deletes it. Next organic request regenerates the cache. PHP `www` user has write access to nginx cache files (same owner).
---
## Current Performance (2026-05-31)
### zamolxis
| Metric | Value |
|---|---|
| Load average | 1.76 (6 vCPU) |
| RAM | 24GB, no swap |
| Disk | 10GB / 75GB (14%) |
| nginx cache | 8,288 files, 1.0GB |
| Redis hit rate | 86.9% |
| Redis memory | 1.03GB / 3GB cap |
| OPcache | 99.6%+ hit rate, 3,343 cached scripts |
### decebal
| Metric | Value |
|---|---|
| Load average | 0.06 (6 vCPU) |
| RAM | No swap, ~18GB free |
| MySQL DB size | 1,850MB |
| Posts | 143,266 |
| Threads connected | 4 |
| Slow queries | 73 in 2 days |
| Disk (MySQL data) | 3.4GB / 94GB (4%) |
| Backup | 137MB daily, verified |
---
## Deployment Status
| Item | Status |
|---|---|
| zamolxis provisioned and configured | Done |
| decebal provisioned and configured | Done |
| rsync (7.1GB webroot) | Done |
| DB import (143,266 posts, 121 tables) | Done — 2026-05-31 |
| NPM cutover (dracula → zamolxis) | Done — 2026-05-31 |
| Tailscale IPs confirmed | zamolxis: 100.115.128.41 / decebal: 100.67.166.29 |
| NPM Tailscale IP confirmed | 100.89.238.4 |
| pf HTTP rule locked to NPM | Done |
| php85-simplexml installed (WPS3Media) | Done |
| WPS3Media S3 rewrite working | Done — 101,145 files |
| Redis Object Cache plugin active | Done |
| Redis maxmemory capped | Done — 3GB allkeys-lru |
| WP_DEBUG disabled | Done |
| nginx static location includes webp | Done |
| InformatiQ-Toolkit redirect fixed | Done — 403 instead of 301→example.com |
| Permalink structure restored | Done |
| DB backup cron fixed and verified | Done — 137MB/run |
| Temp credential files removed | Done |
| Migration key removed | Done |
| Cache warmer running | Done — @reboot + daily 03:00 |
| Old VMs (dracula, transilvan) | Keep powered, no traffic — decommission after 7 days stable |
Reference in New Issue View Git Blame Copy Permalink