diff --git a/README.md b/README.md deleted file mode 100644 index fed1f8e..0000000 --- a/README.md +++ /dev/null @@ -1,1118 +0,0 @@ -# Sibiu Independent — Stack Migration -## May 2026 - -Migration of `sibiuindependent.ro` from Debian 12 + AApanel to FreeBSD 15.0 minimal stack. - ---- - -## Current Architecture - -``` -[internet] → [Nginx Proxy Manager] → [dracula 100.99.157.56, Tailscale] → [transilvan, Tailscale] -``` - -| Node | Role | OS | Specs | Tailscale IP | -|---|---|---|---|---| -| dracula | Web | Debian 12 | 16 vCPU, 32GB RAM, 245GB disk | 100.99.157.56 | -| transilvan | Database | Debian 12 | 4 vCPU, 32GB RAM, 245GB disk | — | - -Both nodes are VMs running on separate bare-metal hosts within the same datacenter (~1ms RTT). All inter-service traffic runs over Tailscale (WireGuard). The NPM node handles TLS termination and public exposure. The web and DB nodes have no public-facing ports other than what Tailscale exposes. - ---- - -## Current Stack Audit - -### dracula (web) - -| Component | Version / Detail | -|---|---| -| OS | Debian GNU/Linux 12 (bookworm) | -| Web server | nginx 1.30.1 | -| PHP | 8.5.2 (AApanel custom build) + FPM | -| CMS | WordPress, single site: `sibiuindependent.ro` | -| Theme | Newspaper / TagDiv (td-cloud-library, td-composer, td-standard-pack) | -| Caching | W3 Total Cache + Memcached (page + object cache) | -| Sessions | PHP file sessions | -| Media | wp-content/uploads 6.5GB on disk, S3-compatible offload active | -| Cache on disk | 4.8GB (W3TC page cache — throwaway) | -| WP code | ~700MB (core + plugins + themes) | -| DB connection | `DB_HOST=transilvan` resolved via /etc/hosts → Tailscale IP | -| Monitoring | Telegraf, Beszel agent, Wazuh agent, Five Nines agent (all dropped in migration) | -| Security | Fail2ban, SSHGuard, sshguard | -| Mail | Postfix + MTA-STS (SMTP plugin used, local MTA not needed) | -| Panel | AApanel (btpanel) | -| Tailscale IP | 100.99.157.56 | - -**Active PHP extensions:** bcmath, curl, exif, fileinfo, gd, imagick, intl, json, mbstring, memcached, mysqli, mysqlnd, opcache, openssl, pcntl, pdo, pdo_mysql, pdo_sqlite, posix, soap, sockets, sodium, xml, xmlreader, xmlwriter, xsl, zip - -### transilvan (db) - -| Component | Version / Detail | -|---|---| -| OS | Debian GNU/Linux 12 (bookworm) | -| Database | MariaDB 10.11.14 LTS | -| Database name | `sql_sibiuindepen` | -| DB user | `sql_sibiuindepen` | -| Data directory | /www/server/data (~9.6GB raw) | -| Config | /etc/mysql/mariadb.cnf + conf.d/ | -| Monitoring | Telegraf, Beszel agent, Wazuh agent, Five Nines agent (all dropped in migration) | -| Panel | AApanel (btpanel) | -| Binlog | Active (mysql-bin.*, high sequence numbers) | - ---- - -## OS Evaluation - -### Candidates assessed: OpenBSD, FreeBSD, Gentoo, Alpine Linux - -### OpenBSD -- **Security:** Best in class — pledge(2), unveil(2), W^X enforced, secure malloc, pf -- **PHP 8.5:** Not in packages (ports have 8.4). Source build possible but complex dependency tree -- **php-memcached:** Problematic (libmemcached issues on OpenBSD) -- **MySQL 8.x:** Not in packages (MariaDB only) -- **PHP JIT:** Conflicts with W^X; must disable (negligible for WordPress) -- **Monitoring agents:** Wazuh: no official support. Telegraf: community build -- **Verdict:** Excellent security philosophy, but too much friction for this specific stack (PHP 8.5, MySQL 8.x, imagick). Viable only if stripping monitoring stack and accepting PHP 8.4 + Redis instead of memcached - -### FreeBSD -- **Security:** pf, Capsicum, jails, strong defaults -- **PHP 8.5:** Available in ports (lang/php85). All extensions in ports -- **MySQL 8.4 LTS:** Available as databases/mysql84-server -- **imagick, intl, soap, redis:** All in ports -- **ZFS:** Native. Game-changer for database node (atomic snapshots, checksumming, lz4 compression) -- **Monitoring:** Telegraf official package. Tailscale official package. Beszel: Go binary works -- **Wazuh:** Community source build only — dropped per requirements -- **Verdict:** Strongest all-round candidate. PHP 8.5 + all extensions + MySQL 8.4 + ZFS + pf + official Tailscale - -### Gentoo -- **Minimalism:** Genuine — USE flags compile exactly what is needed, OpenRC available -- **PHP 8.5:** In portage (rolling release guarantees latest) -- **MySQL 8.4:** In portage -- **All agents:** Still Linux, everything works natively -- **Maintenance:** High ongoing cost — every update is a compile run. Production liability -- **Verdict:** Philosophically compelling, operationally expensive for a production news site - -### Alpine Linux -- **Minimalism:** musl libc + busybox, ~130MB base -- **PHP 8.5:** Community repos, most extensions packaged -- **Init:** OpenRC, no systemd -- **Migration effort:** Lowest — still Linux, apk replaces apt, paths similar -- **Verdict:** Best minimal Linux option; easier than FreeBSD but fewer operational benefits (no ZFS, no jails) - -### Decision: **FreeBSD 15.0 on both nodes** - -Tiebreaker: ZFS native on the DB node. `zfs snapshot` before any migration step, before any MySQL upgrade, before any schema change — this alone justifies the choice. - -FreeBSD 15.0 over 14.x specifically because PHP 8.5 is a **binary package** on 15.0 — `pkg install php85` and all extensions install directly, no ports compilation required. FreeBSD 15.0 was released December 2, 2025 and has 6 months of production use. MySQL 8.4 LTS, Redis 8.x, and all required PHP extensions are in the binary package tree. pf is simpler and more powerful than nftables. Tailscale has official FreeBSD support. - ---- - -## Target Stack Versions - -| Component | Version | Source | Notes | -|---|---|---|---| -| OS | FreeBSD 15.0 | Release Dec 2, 2025 | Production Release, 6 months in production | -| nginx | 1.30.x | `pkg install nginx` | | -| PHP | 8.5.6 | `pkg install php85` | Binary pkg on FreeBSD 15 — no compilation | -| MySQL | 8.4.9 LTS | `pkg install mysql84-server` | LTS branch. 9.0 and 9.1 both EOL and removed from ports | -| Redis | 8.6.3 | `pkg install redis` | Redis 8.x is current major version | -| Tailscale | latest | `pkg install tailscale` | Official FreeBSD package | - ---- - -## Target Architecture - -``` -[internet] → [NPM node, Tailscale] → [zamolxis 100.115.128.41, Tailscale] → [decebal 100.67.166.29, Tailscale] - -zamolxis (FreeBSD 15.0) decebal (FreeBSD 15.0) -────────────────────────── ───────────────────────────── -nginx 1.30.2 (pkg) ZFS pool "data" → /var/db/mysql -PHP 8.5.4 (pkg) recordsize=16K (InnoDB optimal) - gd, pecl-imagick, intl, mbstring, compression=lz4 - opcache (bundled), pdo_mysql, primarycache=metadata - pecl-redis, soap, sockets, MySQL 8.4.9 LTS (pkg) - sodium, xml, zip, exif, fileinfo pf: 3306 from 100.115.128.41 only -nginx fastcgi_cache (filesystem) Tailscale 100.67.166.29 -Redis 8.6.2 (pkg, unix socket) SSH on Tailscale only -pf: 80 from Tailscale only Proxmox native monitoring -Tailscale 100.115.128.41 -SSH on Tailscale only -Proxmox native monitoring -``` - -### What is dropped -| Component | Reason | -|---|---| -| AApanel / btpanel | Replaced by direct config management | -| W3 Total Cache | Replaced by nginx fastcgi_cache + Redis Object Cache plugin | -| Memcached | Replaced by Redis (unix socket, faster, simpler) | -| Fail2ban / SSHGuard | No public SSH port, Tailscale-only | -| Postfix / sendmail | SMTP plugin in use, no local MTA needed | -| Wazuh agent | Not required | -| Five Nines agent | Not required | -| Beszel agent | Not required — Proxmox native monitoring covers VM metrics | -| Telegraf | Not required — Proxmox native monitoring covers VM metrics | -| memory-cleanup.sh | Linux /proc hack, was on wrong server anyway | -| ngxblocker | Not applicable to new stack | -| AApanel cron scripts | acme renewal, site_total, backup.py — all panel-specific | - -### Caching architecture change -| Layer | Old | New | -|---|---|---| -| Page cache | W3TC on-disk (4.8GB, PHP-generated) | nginx fastcgi_cache (filesystem, PHP bypassed entirely on HIT) | -| Object cache | W3TC + Memcached (TCP) | Redis Object Cache plugin + Redis (unix socket) | -| Bytecode cache | OPcache | OPcache (unchanged, JIT disabled — no benefit for WordPress) | -| Sessions | PHP file sessions | Redis via session.save_path unix socket | - ---- - -## Host Hardware - -| Host | CPU | Logical CPUs | RAM | Storage | Available to VMs | VM | -|---|---|---|---|---|---|---| -| dracula host | Xeon E5-1620 v2 @ 3.70GHz | 8 (4 cores + HT) | 64GB | Datacenter HDD | ~full | zamolxis / ID 900 (web) | -| transilvan host | Xeon E5-1620 v2 @ 3.70GHz | 8 (4 cores + HT) | 32GB | SSD 160GB (~130GB free) | 130GB | decebal / ID 901 (db) — sole VM | - -Both hosts run Proxmox VE (confirmed via qemu-guest-agent on current VMs). Decebal's host is dedicated — no other VMs share it. - ---- - -## VM Sizing - -### Rationale for 200 concurrent users - -With nginx fastcgi_cache, the vast majority of requests never invoke PHP. On a news site, traffic concentrates on recent articles — the hot working set is small and stays warm in FreeBSD's unified buffer cache (UBC) regardless of underlying disk speed. - -| Metric | Value | Notes | -|---|---|---| -| Expected cache HIT rate | ~85–90% | News sites have concentrated traffic on recent content | -| Concurrent PHP requests at peak | ~20–30 | 200 users × ~15% miss rate | -| PHP-FPM workers (pm.max_children) | 40 | Headroom above expected peak | -| RAM per PHP-FPM worker | ~35MB RSS | WordPress with active plugins | -| PHP-FPM peak RAM | ~1.4GB | 40 workers × 35MB | -| nginx HIT response time | <5ms | Served from UBC, no PHP involved | -| nginx MISS response time | 100–400ms | WP query + render | -| InnoDB buffer pool hit rate | >99% | 26GB pool, 9.6GB dataset fits entirely in RAM with 16GB to spare | -| Active DB connections | ≤ 40 | One per active PHP-FPM worker | - -### zamolxis (web VM on HDD host) - -| Parameter | Value | -|---|---| -| vCPU | 6 | -| RAM | 24GB | -| Disk | 80GB single virtual disk (HDD-backed) | -| Balloon | Enabled (web VM, acceptable) | - -RAM breakdown: PHP-FPM peak 1.4GB + Redis ~150MB + nginx + OS ~1GB + **~20GB available for UBC page cache**. With 20GB of UBC headroom, the entire nginx fastcgi_cache working set lives in RAM regardless of what is on disk — HDD latency becomes irrelevant for cached pages. Cold cache startup and log writes (sequential) are the only HDD operations that matter. - -### decebal (DB VM on SSD host — sole VM) - -| Parameter | Value | -|---|---| -| vCPU | 6 | -| RAM | 30GB | -| Disk 1 — OS | 20GB virtual (SSD-backed) | -| Disk 2 — ZFS data pool | 100GB virtual (SSD-backed) | -| Total disk | 120GB — fits within 130GB available, 10GB buffer for Proxmox overhead | -| Balloon | **Disabled** — memory ballooning must not reclaim InnoDB buffer pool RAM | - -RAM breakdown: innodb_buffer_pool_size 26GB + MySQL overhead ~2GB + OS ~2GB. Decebal is the sole VM on its host — 30GB allocation leaves 2GB for the Proxmox host OS, which is the practical minimum. - -6 vCPU (from 8 logical CPUs on host) gives MySQL adequate threads for InnoDB background I/O, parallel queries, and connection handling while leaving 2 threads for Proxmox. - -**Disk sizing rationale:** FreeBSD base installs to ~2GB; 20GB for the OS disk is generous for binaries, ports tree fragments, and logs. The 100GB ZFS data pool provides: 9.6GB current dataset (likely 6–7GB after lz4 compression), 7 days of compressed mysqldumps (~4GB each = ~28GB), binlog rotation buffer (expire_logs_days = 3), and substantial growth headroom at the current data trajectory. - -Both virtual disks originate from the same physical SSD pool in Proxmox. Separation is logical: independent sizing, clean `zpool create data /dev/da1`, ZFS snapshots on the data disk without touching the OS disk, and straightforward future migration if a dedicated physical disk is added to the host. - ---- - -## VM Provisioning (Proxmox) - -### 1. Download FreeBSD 15.0 ISO on each Proxmox host - -```sh -wget -P /var/lib/vz/template/iso/ \ - https://download.freebsd.org/releases/amd64/amd64/ISO-IMAGES/15.0/FreeBSD-15.0-RELEASE-amd64-disc1.iso -``` - -### 2. Create zamolxis / ID 900 (run on the HDD host) - -Via CLI: -```sh -qm create 900 \ - --name zamolxis \ - --memory 24576 \ - --balloon 24576 \ - --cores 6 \ - --cpu host \ - --machine q35 \ - --bios ovmf \ - --efidisk0 local:1,format=raw \ - --net0 virtio,bridge=vmbr0 \ - --ostype other \ - --scsihw virtio-scsi-single \ - --scsi0 local:80,format=raw \ - --cdrom local:iso/FreeBSD-15.0-RELEASE-amd64-disc1.iso \ - --boot order=ide2;scsi0 \ - --agent enabled=1 -``` - -Via UI: -``` -General → Name: zamolxis -OS → FreeBSD (other), ISO: FreeBSD-15.0-RELEASE-amd64-disc1.iso -System → Machine: q35, BIOS: OVMF (UEFI), Qemu Agent: ✓ -Disks → VirtIO SCSI, 80GB, Cache: None -CPU → 6 cores, Type: host -Memory → 24576 MB -Network → VirtIO, vmbr0 -``` - -### 3. Create decebal / ID 901 (run on the SSD host — sole VM) - -```sh -# VM with OS disk -qm create 901 \ - --name decebal \ - --memory 30720 \ - --balloon 0 \ - --cores 6 \ - --cpu host \ - --machine q35 \ - --bios ovmf \ - --efidisk0 local:1,format=raw \ - --net0 virtio,bridge=vmbr0 \ - --ostype other \ - --scsihw virtio-scsi-single \ - --scsi0 local:20,format=raw,discard=on,ssd=1 \ - --cdrom local:iso/FreeBSD-15.0-RELEASE-amd64-disc1.iso \ - --boot order=ide2;scsi0 \ - --agent enabled=1 - -# Add second disk for ZFS data pool -qm set 901 --scsi1 local:100,format=raw,discard=on,ssd=1 -``` - -`--balloon 0` disables memory ballooning on the DB VM. This is required — ballooning can silently reclaim pages from the InnoDB buffer pool under host memory pressure. - -### 4. FreeBSD installer settings (same for both VMs) - -``` -Welcome screen → Install -Keymap → your preference -Hostname → zamolxis (or decebal) -Distribution → base, kernel (nothing else) -Partitioning → Auto (ZFS) - Pool type → stripe (single disk) - Disk → da0 (the OS disk) - Swap → 2GB - Compress → lz4 - Encrypt → No -Network → vtnet0, configure with a temporary IP for initial pkg bootstrap - (Tailscale will take over; this IP can be removed afterwards) -Mirror → closest region -Root password → set strong password -SSHD → enable -``` - -### 5. First boot — both VMs - -```sh -# Update base system -freebsd-update fetch install - -# Bootstrap pkg and install essentials -pkg update && pkg upgrade -y -pkg install -y qemu-guest-agent tailscale sudo curl wget - -# Verify package versions available -pkg search php85 | head -3 # should show 8.5.6+ -pkg search mysql84-server # should show 8.4.9 -pkg search redis | grep "^redis" # should show 8.6.x - -# Enable QEMU guest agent (so Proxmox can see VM IP, issue graceful shutdowns) -sysrc qemu_guest_agent_enable=YES -service qemu-guest-agent start - -# Join Tailscale network -sysrc tailscaled_enable=YES -service tailscaled start -tailscale up --hostname=zamolxis # or decebal - -# Note the assigned Tailscale IP — plug into pf.conf and wp-config placeholders -tailscale ip -4 -``` - -After both VMs have Tailscale IPs, replace all `` and `` placeholders throughout this document and in all config files. - ---- - -## Migration Data Inventory - -| Component | Size | Action | -|---|---|---| -| WordPress core + plugins + themes | ~700MB | rsync | -| wp-content/uploads | 6.5GB | rsync then cleanup (S3 is source of truth) | -| wp-content/cache | 4.8GB | **Skip** — regenerates automatically | -| w3tc-config | 40KB | **Skip** — W3TC removed | -| Database (sql_sibiuindepen) | 9.6GB raw / ~3-4GB compressed dump | mysqldump → import | - ---- - -## Cron Inventory - -### Kept and migrated - -| Schedule | Script | Notes | -|---|---|---| -| `*/5 * * * *` | wp-cron.php trigger | Unchanged | -| `0 3 * * *` | cache-warmer.sh | Rewritten for nginx fastcgi_cache | -| `0 15 * * *` | cache-warmer.sh | Rewritten for nginx fastcgi_cache | -| `0 3 * * 1` | JS asset refresh (OneSignal, FB SDKs) | Unchanged | - -### Dropped - -| Script | Reason | -|---|---| -| ngxblocker update | Not using ngxblocker | -| analyze-nginx-attacks.sh | Panel-specific monitoring | -| analyze-php-slow.sh | Panel-specific monitoring | -| monitor-php-fpm.sh | Panel-specific monitoring | -| memory-cleanup.sh | Linux /proc, wrong server | -| AApanel acme_v2.py | Panel SSL renewal | -| AApanel site_total_check.py | Panel metric | -| AApanel backup.py | Replaced by custom mysqldump script | - -### New on new stack - -| Schedule | Script | -|---|---| -| `0 2 * * *` | mysqldump → /var/backups + optional S3 upload | -| `0 0 * * 0` | ZFS snapshot rotation (keep last 4 weeks) | - ---- - -## Phase 0 — Preparation (live servers, non-disruptive) - -```sh -# Capture configs before touching anything -scp -i /root/.ssh/id_ed25519 -P 79 \ - root@dracula:/www/wwwroot/sibiuindependent.ro/wp-config.php ./wp-config.php.bak - -scp -i /root/.ssh/id_ed25519 -P 79 \ - root@dracula:/root/cache-warmer.sh ./cache-warmer.sh.original - -# Test dump on transilvan — confirms dump works before migration day -ssh -i /root/.ssh/id_ed25519 -p 79 root@transilvan \ - "mysqldump -S /tmp/mysqld.sock --single-transaction \ - --routines --triggers --events sql_sibiuindepen \ - | gzip > /tmp/testdump.sql.gz && ls -lh /tmp/testdump.sql.gz" - -# Record current NPM upstream for rollback reference -# Old dracula Tailscale IP: 100.99.157.56 -``` - ---- - -## Phase 1 — DB Node (decebal) - -### 1.1 FreeBSD base + ZFS - -```sh -# During FreeBSD 15.0 install: enable ZFS, create OS pool on first disk (da0) -# da1 is left raw for the database ZFS pool - -zpool create -o ashift=12 data /dev/da1 -zfs create -o mountpoint=/var/db/mysql data/mysql -zfs set recordsize=16K data/mysql # InnoDB optimal block size -zfs set primarycache=metadata data/mysql # InnoDB manages its own buffer pool -zfs set compression=lz4 data/mysql # free performance, transparent -``` - -### 1.2 MySQL 8.4 LTS - -MySQL 8.4.9 is the correct choice. MySQL 9.0 was removed from FreeBSD ports (EOL). MySQL 9.1 expired March 2026 (EOL). The 9.x series is Oracle's "innovation" track with no long-term support guarantee. MySQL 8.4 is the current LTS branch. - -```sh -pkg install mysql84-server -sysrc mysql_enable=YES -sysrc mysql_dbdir=/var/db/mysql -``` - -`/usr/local/etc/mysql/my.cnf`: -```ini -[mysqld] -# Network -bind-address = 100.67.166.29 -port = 3306 -mysqlx = 0 - -# Paths -datadir = /var/db/mysql -socket = /var/db/mysql/mysql.sock -log_error = /var/db/mysql/decebal.err -pid-file = /var/db/mysql/decebal.pid - -# InnoDB -innodb_buffer_pool_size = 26G -innodb_buffer_pool_instances = 8 -innodb_redo_log_capacity = 2G -innodb_flush_log_at_trx_commit = 1 -innodb_flush_method = O_DIRECT -innodb_io_capacity = 2000 -innodb_io_capacity_max = 4000 - -# Connections -max_connections = 150 -thread_cache_size = 16 -table_open_cache = 4000 - -# Character set -character-set-server = utf8mb4 -collation-server = utf8mb4_unicode_ci - -# Logging -slow_query_log = 1 -slow_query_log_file = /var/db/mysql/slow.log -long_query_time = 2 -binlog_expire_logs_seconds = 259200 - -# Safety -skip-name-resolve -sql_mode = STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION - -[client] -socket = /var/db/mysql/mysql.sock -``` - -Note: `innodb_redo_log_capacity` replaces the deprecated `innodb_log_file_size` in MySQL 8.4. - -### 1.3 pf firewall - -`/etc/pf.conf`: -``` -# decebal — database server pf ruleset -ext_if = "vtnet0" -ts_if = "tailscale0" - -# Management: local subnet, Tailscale, allowed external IPs -mgmt = "{ 127.0.0.0/8, 192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12, 100.64.0.0/10, 194.56.239.153, 109.69.48.0/24 }" - -# zamolxis Tailscale IP — only source allowed to reach MySQL -zamolxis_ts = "100.115.128.41" - -set skip on lo0 -set block-policy drop -scrub in all - -block in all -block out quick inet6 all - -pass out all keep state - -# SSH — management IPs on either interface -pass in on $ext_if proto tcp from $mgmt to any port 22 keep state -pass in on $ts_if proto tcp to any port 22 keep state - -# MySQL — zamolxis Tailscale IP only, via Tailscale interface -pass in on $ts_if proto tcp from $zamolxis_ts to any port 3306 keep state -``` - -```sh -sysrc pf_enable=YES -service pf start -``` - -### 1.4 Tailscale - -```sh -pkg install tailscale -sysrc tailscaled_enable=YES -service tailscaled start -tailscale up --hostname=decebal -``` - -### 1.5 DB restore - -```sh -# Dump from old transilvan (port 79), import to new decebal -# Run from a machine with access to both nodes -ssh -p 79 root@transilvan \ - "mysqldump -S /tmp/mysqld.sock --single-transaction \ - --routines --triggers --events sql_sibiuindepen" \ - | ssh root@100.67.166.29 \ - "mysql --socket=/var/db/mysql/mysql.sock sql_sibiuindepen" - -# App user already created — zamolxis Tailscale IP 100.115.128.41 -# Credentials: sql_sibiuindepen / see keepass -``` - -### 1.6 DB backup cron - -`/root/db-backup.sh`: -```sh -#!/bin/sh -# Daily MySQL backup for decebal -# Retention: 7 days local. Credentials via /root/.my.cnf -BACKUP_DIR="/var/db/backups/mysql" -DB="sql_sibiuindepen" -DATE=$(date +%Y%m%d_%H%M%S) -LOG="/var/log/db-backup.log" - -mkdir -p "$BACKUP_DIR" -echo "[$(date)] Starting backup of $DB" >> "$LOG" - -/usr/local/bin/mysqldump \ - --socket=/var/db/mysql/mysql.sock \ - --single-transaction \ - --quick \ - --routines \ - --triggers \ - --set-gtid-purged=OFF \ - "$DB" | gzip > "${BACKUP_DIR}/${DB}_${DATE}.sql.gz" - -if [ $? -eq 0 ]; then - SIZE=$(du -sh "${BACKUP_DIR}/${DB}_${DATE}.sql.gz" | cut -f1) - echo "[$(date)] Backup OK: ${DB}_${DATE}.sql.gz ($SIZE)" >> "$LOG" -else - echo "[$(date)] BACKUP FAILED for $DB" >> "$LOG" - exit 1 -fi - -find "$BACKUP_DIR" -name "${DB}_*.sql.gz" -mtime +7 -delete -echo "[$(date)] Pruned backups older than 7 days" >> "$LOG" -``` - -Crontab (root, on decebal): -``` -# Daily MySQL dump at 02:00 -0 2 * * * root /root/db-backup.sh - -# Weekly ZFS snapshot (Sunday 00:30) -30 0 * * 0 root zfs snapshot data/mysql@weekly-$(date +\%Y\%m\%d) - -# Prune ZFS snapshots — keep last 4 weeks (Sunday 01:00) -0 1 * * 0 root zfs list -H -t snapshot -o name data/mysql | sort | head -n -4 | xargs -r zfs destroy -``` - ---- - -## Phase 2 — Web Node (zamolxis) - -### 2.1 Base packages - -Redis 8.x (currently 8.6.3 in ports) is the current major version. - -```sh -pkg install -y nginx redis git curl wget -sysrc nginx_enable=YES redis_enable=YES -service redis start -``` - -### 2.2 PHP 8.5 from binary packages - -On FreeBSD 15.0, PHP 8.5 and all extensions are available as binary packages — no ports compilation required. - -```sh -# Base PHP 8.5 + FPM (opcache is bundled in the base php85 package) -pkg install -y php85 - -# Extensions needed for WordPress -# Note: imagick and redis are PECL extensions — package prefix is php85-pecl-* -pkg install -y \ - php85-bcmath \ - php85-curl \ - php85-exif \ - php85-fileinfo \ - php85-gd \ - php85-pecl-imagick \ - php85-intl \ - php85-mbstring \ - php85-mysqli \ - php85-pcntl \ - php85-pdo_mysql \ - php85-posix \ - php85-pecl-redis \ - php85-soap \ - php85-sockets \ - php85-sodium \ - php85-xml \ - php85-xmlreader \ - php85-xmlwriter \ - php85-xsl \ - php85-zip - -pkg install -y ImageMagick7 - -sysrc php_fpm_enable=YES -``` - -### 2.3 PHP-FPM pool - -`/usr/local/etc/php-fpm.d/sibiuindependent.conf`: -```ini -[sibiuindependent] -user = www -group = www -listen = /var/run/php-fpm-si.sock -listen.owner = www -listen.group = www -pm = dynamic -pm.max_children = 40 -pm.start_servers = 8 -pm.min_spare_servers = 4 -pm.max_spare_servers = 16 -pm.max_requests = 500 -``` - -`/usr/local/etc/php/php.ini` (relevant): -```ini -opcache.enable = 1 -opcache.memory_consumption = 256 -opcache.interned_strings_buffer = 16 -opcache.max_accelerated_files = 20000 -opcache.validate_timestamps = 0 -opcache.jit = 0 -session.save_handler = redis -session.save_path = "unix:///var/run/redis/redis.sock" -``` - -### 2.4 nginx with fastcgi_cache - -`/usr/local/etc/nginx/nginx.conf`: -```nginx -worker_processes auto; -events { worker_connections 4096; } - -http { - include mime.types; - default_type application/octet-stream; - sendfile on; - tcp_nopush on; - server_tokens off; - - fastcgi_cache_path /var/cache/nginx/si - levels=1:2 - keys_zone=si_cache:64m - max_size=4g - inactive=24h - use_temp_path=off; - - fastcgi_cache_key "$scheme$request_method$host$request_uri"; - - server { - listen 80; - server_name sibiuindependent.ro www.sibiuindependent.ro; - root /var/www/sibiuindependent.ro; - index index.php; - - set $skip_cache 0; - if ($request_method = POST) { set $skip_cache 1; } - if ($query_string) { set $skip_cache 1; } # nginx if does not support != - if ($request_uri ~* "/wp-admin/|/wp-login|/xmlrpc|/feed|sitemap") { - set $skip_cache 1; } - if ($http_cookie ~* "comment_author|wordpress_[a-f0-9]+|wp-postpass|wordpress_logged_in") { - set $skip_cache 1; } - - location / { - try_files $uri $uri/ /index.php?$args; - } - - location ~ \.php$ { - fastcgi_pass unix:/var/run/php-fpm-si.sock; - fastcgi_index index.php; - include fastcgi_params; - fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; - - fastcgi_cache si_cache; - fastcgi_cache_valid 200 301 302 24h; - fastcgi_cache_bypass $skip_cache; - fastcgi_no_cache $skip_cache; - add_header X-Cache-Status $upstream_cache_status; - } - - location ~* \.(js|css|png|jpg|jpeg|gif|svg|ico|woff2)$ { - expires 1y; - add_header Cache-Control "public, immutable"; - access_log off; - } - } -} -``` - -```sh -mkdir -p /var/cache/nginx/si -chown www:www /var/cache/nginx/si -service nginx start -``` - -### 2.5 pf - -`/etc/pf.conf`: -``` -# zamolxis — web server pf ruleset -ext_if = "vtnet0" -ts_if = "tailscale0" - -# Management: local subnet, Tailscale, allowed external IPs -mgmt = "{ 127.0.0.0/8, 192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12, 100.64.0.0/10, 194.56.239.153, 109.69.48.0/24 }" - -set skip on lo0 -set block-policy drop -scrub in all - -# Default deny -block in all -block out quick inet6 all - -# Allow all outbound (package updates, S3, SMTP relay, etc.) -pass out all keep state - -# SSH — from management IPs on either interface -pass in on $ext_if proto tcp from $mgmt to any port 22 keep state -pass in on $ts_if proto tcp to any port 22 keep state - -# HTTP — from NPM via Tailscale only -# Tighten to specific NPM Tailscale IP once known -pass in on $ts_if proto tcp to any port 80 keep state -``` - -Note: tighten the HTTP rule to NPM's specific Tailscale IP once confirmed. - -```sh -sysrc pf_enable=YES -service pf start -``` - -### 2.6 Tailscale - -```sh -pkg install tailscale -sysrc tailscaled_enable=YES -service tailscaled start -tailscale up --hostname=zamolxis -``` - -### 2.7 rsync web files - -```sh -# Exclude cache — does not migrate, regenerates on first request -rsync -avz --progress \ - -e "ssh -i /root/.ssh/id_ed25519 -p 79" \ - --exclude="wp-content/cache/" \ - --exclude="wp-content/w3tc-config/" \ - root@dracula:/www/wwwroot/sibiuindependent.ro/ \ - /var/www/sibiuindependent.ro/ - -chown -R www:www /var/www/sibiuindependent.ro -``` - -### 2.8 wp-config.php changes - -Two changes on the new node only: - -```php -// 1. Update DB_HOST to decebal Tailscale IP -define( 'DB_HOST', '100.67.166.29' ); - -// 2. WP_CACHE stays true for Redis Object Cache drop-in -// Remove the W3TC comment, keep the constant: -define( 'WP_CACHE', true ); -``` - -Remove W3TC plugin and its drop-ins: -```sh -rm -rf /var/www/sibiuindependent.ro/wp-content/plugins/w3-total-cache -rm -f /var/www/sibiuindependent.ro/wp-content/advanced-cache.php -rm -f /var/www/sibiuindependent.ro/wp-content/object-cache.php -``` - -Install Redis Object Cache plugin (Till Krüss), activate, configure: -```php -// In wp-config.php, before "That's all, stop editing": -define( 'WP_REDIS_PATH', '/var/run/redis/redis.sock' ); -define( 'WP_REDIS_SCHEME', 'unix' ); -``` - -### 2.9 Crons - -``` -*/5 * * * * www curl -s -o /dev/null https://sibiuindependent.ro/wp-cron.php?doing_wp_cron -0 3 * * * root /root/cache-warmer.sh >> /root/cache-warmer.log 2>&1 -0 15 * * * root /root/cache-warmer.sh >> /root/cache-warmer.log 2>&1 -0 3 * * 1 root /root/js-refresh.sh >> /root/js-refresh.log 2>&1 -``` - ---- - -## Cache Warmer (rewritten for nginx fastcgi_cache) - -`/root/cache-warmer.sh`: -```sh -#!/bin/sh -# Incremental cache warmer for nginx fastcgi_cache. -# Checks X-Cache-Status header: skips HITs, warms MISSes. -# Requires curl >= 7.84 (FreeBSD 14 ships curl 8.x — compatible). - -SITE='https://sibiuindependent.ro' -CONCURRENCY=4 -LOG='/root/cache-warmer.log' -URL_FILE='/tmp/cache-warmer-urls.txt' -DONE_FILE='/tmp/cache-warmer-done.txt' - -echo "[$(date)] Starting cache warmer (incremental, nginx fastcgi_cache)" | tee -a "$LOG" - -: > "$URL_FILE" -SITEMAPS=$(curl -sk "${SITE}/sitemap_index.xml" | \ - grep -oE 'https?://[^<]+\.xml' | grep -v image) -echo "[$(date)] Sub-sitemaps: $(echo "$SITEMAPS" | wc -l)" | tee -a "$LOG" - -for SMAP in $SITEMAPS; do - curl -sk "$SMAP" | grep -oE 'https?://[^<]+' | grep -v '\.xml' >> "$URL_FILE" - sleep 0.3 -done - -TOTAL=$(wc -l < "$URL_FILE") -echo "[$(date)] Total URLs from sitemap: $TOTAL" | tee -a "$LOG" - -: > "$DONE_FILE" -WARMED=0 -SKIPPED=0 -COUNT=0 - -while IFS= read -r URL; do - ( - RESULT=$(curl -sk -o /dev/null \ - -w '%{http_code} %header{x-cache-status}' \ - -A 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36' \ - -H 'Accept-Encoding: gzip' \ - --max-time 20 "$URL") - echo "$RESULT $URL" >> "$DONE_FILE" - ) & - COUNT=$((COUNT + 1)) - if [ $((COUNT % CONCURRENCY)) -eq 0 ]; then - wait - fi - sleep 0.05 -done < "$URL_FILE" -wait - -WARMED=$(grep -vc 'HIT' "$DONE_FILE" 2>/dev/null || echo 0) -SKIPPED=$(grep -c 'HIT' "$DONE_FILE" 2>/dev/null || echo 0) -echo "[$(date)] Done. Warmed: $WARMED, Skipped (HIT): $SKIPPED" | tee -a "$LOG" -``` - -### Key difference from original - -| | Original (W3TC) | New (nginx fastcgi_cache) | -|---|---|---| -| Cache check method | Stat filesystem for `_index_slash_ssl.html` | HTTP `X-Cache-Status: HIT` header | -| Cache path | `/www/wwwroot/.../cache/page_enhanced/...` | `/var/cache/nginx/si/` (hashed, not human-readable) | -| Logic | Skip if file exists | Skip if header is `HIT` | -| Concurrency | Background subshells | Background subshells (unchanged) | -| Sitemap parsing | Unchanged | Unchanged | - ---- - -## JS Asset Refresh Cron - -`/root/js-refresh.sh`: -```sh -#!/bin/sh -# Weekly refresh of self-hosted third-party JS assets -WEBROOT=/var/www/sibiuindependent.ro - -curl -s https://cdn.onesignal.com/sdks/OneSignalSDK.js \ - | sed 's|//# sourceMappingURL=.*||' \ - > ${WEBROOT}/onesignal.js - -curl -s https://cdn.onesignal.com/sdks/OneSignalPageSDKES6.js \ - | sed 's|//# sourceMappingURL=.*||' \ - > ${WEBROOT}/onesignal-es6.js - -curl -s https://connect.facebook.net/en_US/sdk.js > ${WEBROOT}/fb-sdk.js -curl -s https://connect.facebook.net/ro_RO/sdk.js > ${WEBROOT}/fb-sdk-ro.js -``` - ---- - -## Phase 3 — Zero-downtime Cutover - -### Pre-cutover checklist (run on zamolxis before touching NPM) - -Steps 1–8 below are already verified. Steps 9–10 require the DB import (done at cutover). - -```sh -# 1. PHP extensions — VERIFIED OK -php -m | grep -E "imagick|redis|pdo_mysql|intl|soap|sodium|gd|mbstring" -# opcache: verified active via FPM web request - -# 2. Services running — VERIFIED OK -service nginx status && service php_fpm status && service redis status - -# 3. Redis socket responding — VERIFIED OK -redis-cli -s /var/run/redis/redis.sock ping # → PONG - -# 4. PHP-FPM socket present — VERIFIED OK -ls /var/run/php-fpm-si.sock - -# 5. DB connectivity zamolxis->decebal — VERIFIED OK -# Note: use credentials file to avoid ! shell expansion issue -cat > /tmp/.chk.cnf << EOF -[client] -user=sql_sibiuindepen -password=W0rdPr3ss@SI2026! -host=100.67.166.29 -port=3306 -EOF -mysql --defaults-file=/tmp/.chk.cnf --get-server-public-key \ - -e "SELECT 1 AS db_ok;" -rm -f /tmp/.chk.cnf - -# 6. WordPress core files present — VERIFIED OK -ls /var/www/sibiuindependent.ro/index.php /var/www/sibiuindependent.ro/wp-login.php - -# 7. Redis Object Cache drop-in installed, W3TC absent — VERIFIED OK -ls /var/www/sibiuindependent.ro/wp-content/object-cache.php -ls /var/www/sibiuindependent.ro/wp-content/plugins/w3-total-cache 2>/dev/null || echo "W3TC absent (good)" - -# 8. nginx responds (MISS expected without DB) — VERIFIED OK -curl -sI http://127.0.0.1/ -H "Host: sibiuindependent.ro" | grep X-Cache-Status - -# --- Steps 9-10: run AFTER DB import at cutover --- - -# 9. WordPress returns 200 (post DB import) -curl -sk -o /dev/null -w "%{http_code}" \ - http://127.0.0.1/ -H "Host: sibiuindependent.ro" -# Expected: 200 - -# 10. Cache HIT on second request (post DB import) -curl -sI http://127.0.0.1/ -H "Host: sibiuindependent.ro" | grep X-Cache-Status # MISS -curl -sI http://127.0.0.1/ -H "Host: sibiuindependent.ro" | grep X-Cache-Status # HIT - -# 11. Redis object cache connected (post DB import) -wp --path=/var/www/sibiuindependent.ro --allow-root redis enable -wp --path=/var/www/sibiuindependent.ro --allow-root redis status -``` - -### Cutover sequence - -``` -T-0 Disable cache warmer crons on OLD dracula (comment out both lines in crontab) - -T-1 Final rsync — incremental delta only, excludes W3TC and cache: - # Run on zamolxis: - rsync -avz \ - -e "ssh -i /root/.ssh/migration_key -p 79 -o StrictHostKeyChecking=no" \ - --exclude="wp-content/cache/" \ - --exclude="wp-content/w3tc-config/" \ - --exclude="wp-content/plugins/w3-total-cache/" \ - --exclude="wp-content/uploads/wpo_wcml_cache/" \ - root@100.99.157.56:/www/wwwroot/sibiuindependent.ro/ \ - /var/www/sibiuindependent.ro/ - - # Then immediately run fixup (rewrites wp-config, re-installs Redis drop-in): - /root/post-rsync-fixup.sh - -T-2 Final DB dump and import: - # Run from any host with access to both (or from this local machine): - ssh -p 79 root@100.99.157.56 \ - "mysqldump -S /tmp/mysqld.sock --single-transaction \ - --routines --triggers --events sql_sibiuindepen" \ - | ssh root@100.67.166.29 \ - "mysql --socket=/var/db/mysql/mysql.sock sql_sibiuindepen" - - # ZFS snapshot immediately after import: - ssh root@100.67.166.29 \ - "zfs snapshot data/mysql@post-migration-$(date +%Y%m%d)" - -T-2.5 Run WP-CLI to enable Redis and activate Redis Object Cache plugin: - # Run on zamolxis after DB is imported: - wp --path=/var/www/sibiuindependent.ro --allow-root redis enable - wp --path=/var/www/sibiuindependent.ro --allow-root plugin activate redis-cache - -T-3 Checklist steps 9–11 (see above): verify 200, cache HIT, Redis status - -T-4 Switch NPM upstream: - old: 100.99.157.56 (dracula Tailscale IP) - new: 100.115.128.41 (zamolxis Tailscale IP) - (single upstream change in NPM UI — instant, no DNS TTL) - -T-5 Smoke test: - - Homepage loads (public, non-logged-in) - - An article page loads - - wp-admin accessible and working - - Second page load for any article: X-Cache-Status: HIT in response headers - -T-6 Enable cache warmer cron on zamolxis (already installed, just runs at 03:00/15:00) - # Optionally run immediately: - /root/cache-warmer.sh & -``` - -**Rollback:** Revert NPM upstream to `100.99.157.56`. Old stack is untouched throughout. No DNS change required. Instant. - -**Old VMs:** Keep powered on (no traffic) for 48 hours. Decommission after confirmed stable. - -### Post-cutover - -```sh -# Run cache warmer immediately on new node -bash /root/cache-warmer.sh - -# Monitor -tail -f /var/log/nginx/error.log -tail -f /var/log/php-fpm.log - -# ZFS snapshot of DB post-migration -zfs snapshot data/mysql@post-migration-$(date +%Y%m%d) -``` - ---- - -## Open Items - -- [x] Tailscale IPs confirmed — zamolxis: 100.115.128.41 / decebal: 100.67.166.29 -- [ ] NPM node Tailscale IP — tighten zamolxis pf HTTP rule to specific NPM IP once confirmed -- [x] PHP 8.5 — all extensions installed; PECL packages use `php85-pecl-*` prefix; opcache bundled in base php85; phar installed for WP-CLI -- [x] Redis Object Cache plugin — downloaded and drop-in installed at wp-content/object-cache.php -- [ ] W3TC minify in use? — confirm before cutover; if yes: drop (nginx serves pre-minified assets) or use Autoptimize as replacement -- [x] ZFS pool — data pool on /dev/da1 (100GB) ONLINE, data/mysql at /var/db/mysql -- [x] MySQL 8.4.9 LTS — deployed on decebal -- [x] Beszel and Telegraf — dropped -- [x] Redis — local to zamolxis, unix socket -- [x] Phase 2 — rsync complete (7.1GB, exit 0), wp-config updated, W3TC removed -- [x] Phase 2 — pre-cutover checklist steps 1–8 all passed -- [ ] Phase 3 — DB import at cutover (checklist steps 9–11 post-import) -- [ ] Phase 3 — NPM upstream switch 100.99.157.56 → 100.115.128.41 - -## Deployment Status (as of 2026-05-29) - -| Node | Status | Notes | -|---|---|---| -| zamolxis | **Ready for cutover** | All services running, 6.8GB webroot synced, checklist items 1–8 passed | -| decebal | **Ready for cutover** | MySQL 8.4.9 running, ZFS ONLINE, app user verified, backup cron installed | - -### What was tuned for 1000 concurrent users (5× baseline) - -| Setting | zamolxis | decebal | -|---|---|---| -| `kern.maxfiles` | 200,000 | 200,000 | -| `kern.ipc.somaxconn` | 4096 | 1024 | -| `net.inet.tcp.sendspace` | 262,144 | 131,072 | -| `vfs.zfs.arc_max` | — | 2GB (leaves RAM for InnoDB) | -| PHP-FPM `pm.max_children` | 80 (was 40) | — | -| nginx `worker_connections` | 8192 (was 4096) | — | -| nginx `worker_rlimit_nofile` | 65536 | — | -| MySQL `max_connections` | — | 300 (was 150) | -| WP-CLI | Installed at /usr/local/bin/wp | — | - -### Connectivity verified -- zamolxis → decebal MySQL (100.67.166.29:3306) as `sql_sibiuindepen`: OK -- pf on decebal: only 100.115.128.41 reaches port 3306: OK -- Backup cron: /var/db/backups/mysql/sql_sibiuindepen_*.sql.gz runs daily at 02:00 -- nginx fastcgi_cache + PHP-FPM + Redis all responding on zamolxis - -### Known issue to fix at cutover -- W3TC plugin will be re-added by the final rsync (it's on the source). The cutover rsync command above uses `--exclude="wp-content/plugins/w3-total-cache/"` to prevent this. `/root/post-rsync-fixup.sh` must be run after every rsync. diff --git a/sibiuindependent.ro.MD b/sibiuindependent.ro.MD new file mode 100644 index 0000000..f7fd490 --- /dev/null +++ b/sibiuindependent.ro.MD @@ -0,0 +1,720 @@ +# sibiuindependent.ro — Stack Migration +## May 2026 + +Migration of `sibiuindependent.ro` from Debian 12 + AApanel to FreeBSD 15.0 minimal stack. + +**Status: LIVE as of 2026-05-31. Migration complete.** + +--- + +## Live Architecture + +``` +[internet] → [NPM node, 100.89.238.4 Tailscale] → [zamolxis 100.115.128.41] → [decebal 100.67.166.29] +``` + +| Node | Role | OS | vCPU | RAM | Disk | Tailscale IP | +|---|---|---|---|---|---|---| +| zamolxis | Web | FreeBSD 15.0-p9 | 6 | 24GB | 80GB HDD | 100.115.128.41 | +| decebal | Database | FreeBSD 15.0-p9 | 6 | 30GB | 20GB OS + 100GB ZFS data (SSD) | 100.67.166.29 | + +Both VMs are on separate bare-metal Proxmox hosts. All inter-service traffic runs over Tailscale. NPM handles TLS termination. No public-facing ports on zamolxis or decebal. + +--- + +## Stack + +### zamolxis (web) + +| Component | Version / Detail | +|---|---| +| OS | FreeBSD 15.0-RELEASE-p9 | +| Web server | nginx 1.30.2 | +| PHP | 8.5.4 + FPM (unix socket `/var/run/php-fpm-si.sock`) | +| Page cache | nginx fastcgi_cache (`/var/cache/nginx/si`, 4GB max) | +| Object cache | Redis Object Cache plugin v2.8.0 (Till Krüss) via unix socket | +| Bytecode cache | OPcache 256MB, 20000 files, JIT **disabled** (caused FPM crashes with WordPress plugin stacks) | +| Redis | 8.6.2, unix socket `/var/run/redis/redis.sock`, maxmemory 3GB allkeys-lru | +| Sessions | Redis via `session.save_path = unix:///var/run/redis/redis.sock` | +| Media | wp-content/uploads local (6.5GB), S3 offload via WPS3Media to `s3.palmasolutions.net/sibiuindependent` (101,145 files in `ddf44d_as3cf_items`) | +| Login URL | `/intrarepresa` (Perfmatters custom login — `/wp-login.php` and `/wp-admin` return 403) | +| WP prefix | `ddf44d_` | +| Firewall | pf — HTTP only from NPM (100.89.238.4), SSH only from mgmt + Tailscale | + +**Active PHP extensions:** bcmath, curl, exif, fileinfo, gd, imagick, intl, mbstring, mysqli, opcache, pcntl, pdo_mysql, pecl-redis, posix, simplexml, soap, sockets, sodium, xml, xmlreader, xmlwriter, xsl, zip + +Note: `php85-simplexml` is required by WPS3Media (AWS SDK XML parsing). Without it, the entire WPS3Media plugin fails to instantiate silently. + +### decebal (db) + +| Component | Version / Detail | +|---|---| +| OS | FreeBSD 15.0-RELEASE-p9 | +| Database | MySQL 8.4.9 LTS | +| Database name | `sql_sibiuindepen` | +| Socket | `/var/db/mysql/mysql.sock` | +| Data directory | `/var/db/mysql` (ZFS pool `data/mysql`) | +| ZFS settings | recordsize=16K, compression=lz4, primarycache=metadata | +| InnoDB buffer pool | 26GB (dataset ~1.85GB — fits entirely in RAM) | +| Firewall | pf — MySQL only from zamolxis (100.115.128.41), SSH only from mgmt + Tailscale | +| Backup | `/root/db-backup.sh` daily 02:00, 7-day retention in `/var/db/backups/mysql/` | +| ZFS snapshots | Weekly (Sunday 00:30), last 4 kept | + +--- + +## Migration Source (decommissioned) + +| Node | Role | OS | Tailscale IP | +|---|---|---|---| +| dracula | Web | Debian 12 + AApanel | 100.99.157.56 | +| transilvan | Database | Debian 12 + AApanel | — | +| transilvan MySQL socket | — | — | `/tmp/mysqld.sock` | + +--- + +## OS Selection + +### Candidates: OpenBSD, FreeBSD, Gentoo, Alpine Linux + +**OpenBSD** — best security (pledge, unveil, W^X), but PHP 8.5 not in packages, MySQL 8.x absent, imagick friction. Viable only with PHP 8.4 + MariaDB. + +**FreeBSD** — pf, Capsicum, ZFS native, PHP 8.5 + all extensions as binary packages, MySQL 8.4 LTS, Redis 8.x, official Tailscale. Strongest all-round candidate. + +**Gentoo** — genuine minimalism via USE flags, but every update is a compile run. Production liability. + +**Alpine Linux** — easiest migration (still Linux), musl + busybox, apk. No ZFS, no jails. Best minimal Linux option. + +**Decision: FreeBSD 15.0** — ZFS native on the DB node is the tiebreaker (`zfs snapshot` before any migration step). FreeBSD 15.0 over 14.x specifically because PHP 8.5 is a binary package on 15.0. + +--- + +## What Was Dropped + +| Component | Reason | +|---|---| +| AApanel / btpanel | Replaced by direct config management | +| W3 Total Cache | Replaced by nginx fastcgi_cache + Redis Object Cache | +| Memcached | Replaced by Redis (unix socket, faster, simpler) | +| Fail2ban / SSHGuard | No public SSH port, Tailscale-only | +| Postfix / sendmail | SMTP plugin in use, no local MTA needed | +| Wazuh / Five Nines / Beszel / Telegraf | Not required — Proxmox native monitoring covers VM metrics | +| memory-cleanup.sh | Linux `/proc` hack, was on wrong server anyway | +| ngxblocker | Not applicable | +| AApanel cron scripts | Panel-specific: acme renewal, site_total, backup.py | + +## Caching Architecture + +| Layer | Old | New | +|---|---|---| +| Page cache | W3TC on-disk (4.8GB, PHP-generated) | nginx fastcgi_cache (PHP bypassed entirely on HIT) | +| Object cache | W3TC + Memcached (TCP) | Redis Object Cache + Redis (unix socket) | +| Bytecode cache | OPcache | OPcache (JIT disabled) | +| Sessions | PHP file sessions | Redis via unix socket | + +--- + +## Host Hardware + +| Host | CPU | RAM | Storage | VM | +|---|---|---|---|---| +| zamolxis host | Xeon E5-1620 v2 @ 3.70GHz (8 logical) | 64GB | Datacenter HDD | zamolxis (ID 900) | +| decebal host | Xeon E5-1620 v2 @ 3.70GHz (8 logical) | 32GB | SSD 160GB | decebal (ID 901) — sole VM | + +--- + +## VM Provisioning + +### 1. FreeBSD 15.0 ISO + +```sh +wget -P /var/lib/vz/template/iso/ \ + https://download.freebsd.org/releases/amd64/amd64/ISO-IMAGES/15.0/FreeBSD-15.0-RELEASE-amd64-disc1.iso +``` + +### 2. zamolxis (ID 900, HDD host) + +```sh +qm create 900 \ + --name zamolxis \ + --memory 24576 --balloon 24576 \ + --cores 6 --cpu host \ + --machine q35 --bios ovmf \ + --efidisk0 local:1,format=raw \ + --net0 virtio,bridge=vmbr0 \ + --ostype other \ + --scsihw virtio-scsi-single \ + --scsi0 local:80,format=raw \ + --cdrom local:iso/FreeBSD-15.0-RELEASE-amd64-disc1.iso \ + --boot order=ide2;scsi0 \ + --agent enabled=1 +``` + +### 3. decebal (ID 901, SSD host — sole VM) + +```sh +qm create 901 \ + --name decebal \ + --memory 30720 --balloon 0 \ + --cores 6 --cpu host \ + --machine q35 --bios ovmf \ + --efidisk0 local:1,format=raw \ + --net0 virtio,bridge=vmbr0 \ + --ostype other \ + --scsihw virtio-scsi-single \ + --scsi0 local:20,format=raw,discard=on,ssd=1 \ + --cdrom local:iso/FreeBSD-15.0-RELEASE-amd64-disc1.iso \ + --boot order=ide2;scsi0 \ + --agent enabled=1 + +qm set 901 --scsi1 local:100,format=raw,discard=on,ssd=1 +``` + +`--balloon 0` is required — memory ballooning must not reclaim InnoDB buffer pool pages. + +### 4. FreeBSD installer settings (both VMs) + +``` +Welcome → Install +Hostname → zamolxis (or decebal) +Distribution → base, kernel only +Partitioning → Auto (ZFS), stripe, da0, swap 2GB, lz4 +Network → vtnet0, temporary IP for pkg bootstrap +SSHD → enable +``` + +### 5. First boot (both VMs) + +```sh +freebsd-update fetch install +pkg update && pkg upgrade -y +pkg install -y qemu-guest-agent tailscale sudo curl wget + +sysrc qemu_guest_agent_enable=YES +service qemu-guest-agent start + +sysrc tailscaled_enable=YES +service tailscaled start +tailscale up --hostname=zamolxis # or decebal +tailscale ip -4 +``` + +--- + +## Phase 1 — DB Node (decebal) + +### ZFS + +```sh +zpool create -o ashift=12 data /dev/da1 +zfs create -o mountpoint=/var/db/mysql data/mysql +zfs set recordsize=16K data/mysql +zfs set primarycache=metadata data/mysql +zfs set compression=lz4 data/mysql +``` + +### MySQL 8.4 LTS + +MySQL 8.4.9 is the current LTS branch. MySQL 9.0/9.1 were Oracle's "innovation" track — both EOL and removed from FreeBSD ports. + +```sh +pkg install mysql84-server +sysrc mysql_enable=YES +sysrc mysql_dbdir=/var/db/mysql +``` + +`/usr/local/etc/mysql/my.cnf`: +```ini +[mysqld] +bind-address = 100.67.166.29 +port = 3306 +mysqlx = 0 +datadir = /var/db/mysql +socket = /var/db/mysql/mysql.sock +log_error = /var/db/mysql/decebal.err +pid-file = /var/db/mysql/decebal.pid + +innodb_buffer_pool_size = 26G +innodb_buffer_pool_instances = 8 +innodb_redo_log_capacity = 2G +innodb_flush_log_at_trx_commit = 1 +innodb_flush_method = O_DIRECT +innodb_io_capacity = 2000 +innodb_io_capacity_max = 4000 + +max_connections = 300 +thread_cache_size = 16 +table_open_cache = 4000 + +character-set-server = utf8mb4 +collation-server = utf8mb4_unicode_ci + +slow_query_log = 1 +slow_query_log_file = /var/db/mysql/slow.log +long_query_time = 2 +binlog_expire_logs_seconds = 259200 + +skip-name-resolve +sql_mode = STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION + +[client] +socket = /var/db/mysql/mysql.sock +``` + +Note: `innodb_redo_log_capacity` replaces deprecated `innodb_log_file_size` in MySQL 8.4. + +### pf (decebal) + +`/etc/pf.conf`: +``` +ext_if = "vtnet0" +ts_if = "tailscale0" + +mgmt = "{ 127.0.0.0/8, 192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12, 100.64.0.0/10, 194.56.239.153, 109.69.48.0/24 }" +zamolxis_ts = "100.115.128.41" + +set skip on lo0 +set block-policy drop +scrub in all + +block in all +block out quick inet6 all +pass out all keep state + +pass in on $ext_if proto tcp from $mgmt to any port 22 keep state +pass in on $ts_if proto tcp to any port 22 keep state + +pass in on $ts_if proto tcp from $zamolxis_ts to any port 3306 keep state +``` + +### DB Backup (decebal) + +`/root/db-backup.sh`: +```sh +#!/bin/sh +BACKUP_DIR="/var/db/backups/mysql" +DB="sql_sibiuindepen" +DATE=$(date +%Y%m%d_%H%M%S) +LOG="/var/log/db-backup.log" + +mkdir -p "$BACKUP_DIR" +echo "[$(date)] Starting backup of $DB" >> "$LOG" + +/usr/local/bin/mysqldump \ + --socket=/var/db/mysql/mysql.sock \ + --single-transaction \ + --quick \ + --routines \ + --triggers \ + --set-gtid-purged=OFF \ + "$DB" | gzip > "${BACKUP_DIR}/${DB}_${DATE}.sql.gz" + +if [ $? -eq 0 ]; then + SIZE=$(du -sh "${BACKUP_DIR}/${DB}_${DATE}.sql.gz" | cut -f1) + echo "[$(date)] Backup OK: ${DB}_${DATE}.sql.gz ($SIZE)" >> "$LOG" +else + echo "[$(date)] BACKUP FAILED for $DB" >> "$LOG" + exit 1 +fi + +find "$BACKUP_DIR" -name "${DB}_*.sql.gz" -mtime +7 -delete +echo "[$(date)] Pruned backups older than 7 days" >> "$LOG" +``` + +Root crontab on decebal (`crontab -l`): +``` +# Daily MySQL dump at 02:00 +0 2 * * * /root/db-backup.sh + +# Weekly ZFS snapshot (Sunday 00:30) +30 0 * * 0 zfs snapshot data/mysql@weekly-$(date +%Y%m%d) + +# Prune ZFS snapshots older than 4 weeks (Sunday 01:00) +0 1 * * 0 zfs list -H -t snapshot -o name data/mysql | sort | head -n -4 | xargs -r zfs destroy +``` + +Note: user crontab format — no `user` field between time spec and command (that is `/etc/crontab` format). + +--- + +## Phase 2 — Web Node (zamolxis) + +### Packages + +```sh +pkg install -y nginx redis git curl wget +sysrc nginx_enable=YES redis_enable=YES +service redis start + +pkg install -y php85 \ + php85-bcmath php85-curl php85-exif php85-fileinfo \ + php85-gd php85-pecl-imagick php85-intl php85-mbstring \ + php85-mysqli php85-pcntl php85-pdo_mysql php85-posix \ + php85-pecl-redis php85-simplexml php85-soap php85-sockets \ + php85-sodium php85-xml php85-xmlreader php85-xmlwriter \ + php85-xsl php85-zip ImageMagick7 + +sysrc php_fpm_enable=YES +``` + +`php85-simplexml` is mandatory — WPS3Media uses the AWS SDK which requires SimpleXML. Without it the plugin silently fails to instantiate. + +### PHP-FPM pool + +`/usr/local/etc/php-fpm.d/sibiuindependent.conf`: +```ini +[sibiuindependent] +user = www +group = www +listen = /var/run/php-fpm-si.sock +listen.owner = www +listen.group = www +pm = dynamic +pm.max_children = 80 +pm.start_servers = 8 +pm.min_spare_servers = 4 +pm.max_spare_servers = 16 +pm.max_requests = 500 +``` + +`/usr/local/etc/php/php.ini` (relevant): +```ini +opcache.enable = 1 +opcache.memory_consumption = 256 +opcache.interned_strings_buffer = 32 +opcache.max_accelerated_files = 20000 +opcache.validate_timestamps = 0 +opcache.jit = 0 +opcache.jit_buffer_size = 0 +session.save_handler = redis +session.save_path = "unix:///var/run/redis/redis.sock" +``` + +JIT is explicitly disabled. WordPress plugin stacks (especially Yoast + TagDiv composer) caused PHP-FPM crashes under JIT on this install. + +### nginx + +`/usr/local/etc/nginx/nginx.conf`: +```nginx +worker_processes auto; +worker_rlimit_nofile 65536; + +events { worker_connections 8192; } + +http { + include mime.types; + default_type application/octet-stream; + sendfile on; + tcp_nopush on; + server_tokens off; + + fastcgi_cache_path /var/cache/nginx/si + levels=1:2 + keys_zone=si_cache:64m + max_size=4g + inactive=24h + use_temp_path=off; + + fastcgi_cache_key "$scheme$request_method$host$request_uri"; + + server { + listen 80; + server_name sibiuindependent.ro www.sibiuindependent.ro; + root /var/www/sibiuindependent.ro; + index index.php; + + set $skip_cache 0; + if ($request_method = POST) { set $skip_cache 1; } + if ($query_string) { set $skip_cache 1; } + if ($request_uri ~* "/wp-admin/|/wp-login|/xmlrpc|/feed|sitemap") { + set $skip_cache 1; } + if ($http_cookie ~* "comment_author|wordpress_[a-f0-9]+|wp-postpass|wordpress_logged_in") { + set $skip_cache 1; } + + location / { + try_files $uri $uri/ /index.php?$args; + } + + location ~ \.php$ { + fastcgi_pass unix:/var/run/php-fpm-si.sock; + fastcgi_index index.php; + include fastcgi_params; + fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; + + fastcgi_cache si_cache; + fastcgi_cache_valid 200 301 302 24h; + fastcgi_cache_bypass $skip_cache; + fastcgi_no_cache $skip_cache; + add_header X-Cache-Status $upstream_cache_status; + } + + location ~* \.(js|css|png|jpg|jpeg|gif|webp|svg|ico|woff|woff2|ttf|eot|mp4|webm|ogv)$ { + expires 1y; + add_header Cache-Control "public, immutable"; + access_log off; + log_not_found off; + } + } +} +``` + +`webp` must be in the static location regex. WPS3Media serves WebP variants from S3; without it, `.webp` requests fall through to PHP-FPM and trigger a full WordPress load for a file that doesn't exist locally. + +### Redis + +`/usr/local/etc/redis.conf` (relevant): +``` +maxmemory 3gb +maxmemory-policy allkeys-lru +unixsocket /var/run/redis/redis.sock +unixsocketperm 777 +``` + +### pf (zamolxis) + +`/etc/pf.conf`: +``` +ext_if = "vtnet0" +ts_if = "tailscale0" + +mgmt = "{ 127.0.0.0/8, 192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12, 100.64.0.0/10, 194.56.239.153, 109.69.48.0/24 }" +npm_ts = "100.89.238.4" + +set skip on lo0 +set block-policy drop +scrub in all + +block in all +block out quick inet6 all +pass out all keep state + +pass in on $ext_if proto tcp from $mgmt to any port 22 keep state +pass in on $ts_if proto tcp to any port 22 keep state + +pass in on $ts_if proto tcp from $npm_ts to any port 80 keep state +``` + +### wp-config.php changes + +```php +define( 'DB_HOST', '100.67.166.29' ); +define( 'DB_PASSWORD', 'W0rdPr3ss@SI2026!' ); +define( 'WP_REDIS_SCHEME', 'unix' ); +define( 'WP_REDIS_PATH', '/var/run/redis/redis.sock' ); +define( 'WP_DEBUG', false ); +define( 'WP_DEBUG_LOG', false ); +define( 'WP_DEBUG_DISPLAY', false ); +``` + +### post-rsync-fixup.sh + +`/tmp/post-rsync-fixup.sh` — run after every rsync: +- Rewrites `DB_HOST` and `DB_PASSWORD` in wp-config.php +- Removes W3TC plugin and its drop-ins (advanced-cache.php, db.php) +- Installs Redis Object Cache drop-in if missing +- Fixes `.user.ini` open_basedir +- Fixes ownership (`www:www`) and wp-config permissions (640) +- Reloads PHP-FPM + +### Kernel tuning (sysctl.conf, both VMs) + +``` +kern.maxfiles=200000 +kern.ipc.somaxconn=4096 +net.inet.tcp.sendspace=262144 +net.inet.tcp.recvspace=262144 +``` + +Additional on zamolxis: +``` +vfs.zfs.arc_max=2147483648 # cap ZFS ARC at 2GB, leave RAM for PHP-FPM + nginx UBC +``` + +### Crons (zamolxis root crontab) + +``` +*/5 * * * * curl -s -o /dev/null https://sibiuindependent.ro/wp-cron.php?doing_wp_cron +0 3 * * * /root/cache-warmer.sh > /dev/null 2>&1 +0 3 * * 1 /root/js-refresh.sh >> /root/js-refresh.log 2>&1 +@reboot sleep 60 && /root/cache-warmer.sh > /dev/null 2>&1 +``` + +--- + +## Cache Warmer + +`/root/cache-warmer.sh` — reads Yoast sitemap index (89 sub-sitemaps), warms all URLs via HTTP directly to nginx on port 80 (nginx does not serve HTTPS; TLS is terminated at NPM). + +```sh +#!/bin/sh +DOMAIN="sibiuindependent.ro" +NGINX="127.0.0.1" +CONCURRENCY=6 + +SITEMAP_INDEX=$(curl -s --max-time 30 \ + --resolve "${DOMAIN}:80:${NGINX}" \ + "http://${DOMAIN}/sitemap_index.xml") + +SITEMAPS=$(echo "$SITEMAP_INDEX" | grep -oE 'https?://[^<]+\.xml' \ + | sed "s|https://${DOMAIN}|http://${DOMAIN}|") + +URLS=$(for SM in $SITEMAPS; do + curl -s --max-time 30 --resolve "${DOMAIN}:80:${NGINX}" "$SM" \ + | grep -oE 'https?://[^<]+' | grep -v '\.xml' \ + | sed "s|https://${DOMAIN}|http://${DOMAIN}|" +done) + +echo "$URLS" | grep 'http' | xargs -P $CONCURRENCY -I{} sh -c \ + 'curl -s -o /dev/null --max-time 20 --resolve "'"${DOMAIN}"':80:'"${NGINX}"'" "{}" 2>/dev/null' +``` + +--- + +## JS Asset Refresh + +`/root/js-refresh.sh` — weekly, Monday 03:00: +```sh +#!/bin/sh +WEBROOT=/var/www/sibiuindependent.ro +curl -s https://cdn.onesignal.com/sdks/OneSignalSDK.js > ${WEBROOT}/onesignal.js +curl -s https://cdn.onesignal.com/sdks/OneSignalPageSDKES6.js > ${WEBROOT}/onesignal-es6.js +curl -s https://connect.facebook.net/en_US/sdk.js > ${WEBROOT}/fb-sdk.js +curl -s https://connect.facebook.net/ro_RO/sdk.js > ${WEBROOT}/fb-sdk-ro.js +``` + +--- + +## DB Migration + +Source: transilvan (Debian 12 + MariaDB), socket `/tmp/mysqld.sock`, SSH port 79. +Destination: decebal (FreeBSD 15 + MySQL 8.4.9), socket `/var/db/mysql/mysql.sock`. + +```sh +# Run from zamolxis (has access to both via Tailscale/direct) +ssh -p 79 root@100.99.157.56 \ + "mysqldump -S /tmp/mysqld.sock --single-transaction --no-tablespaces \ + --routines --triggers --events sql_sibiuindepen" \ + | mysql --defaults-file=/tmp/.imp.cnf -f --get-server-public-key sql_sibiuindepen +``` + +Notes: +- `--no-tablespaces` instead of `--set-gtid-purged=OFF` on MariaDB source (MariaDB does not support that flag) +- `--defaults-file` must be the **first** argument to `mysql` +- `-f` (force) was needed due to a duplicate key in `ddf44d_yoast_indexable_hierarchy` — Yoast's indexable table had integrity issues, non-critical (regenerates on save) +- Final import: 143,266 posts, 121 tables, 1,850MB + +--- + +## Post-Cutover Issues & Fixes + +### 1. S3 images not loading — WPS3Media class not instantiating + +**Root cause:** `php85-simplexml` not installed. WPS3Media's AWS SDK compat check (`AS3CF_Pro_Installer`) detected missing SimpleXML and blocked the entire plugin from loading with "no SimpleXML PHP module." No PHP error was emitted — plugin simply never registered its `wp_get_attachment_url` filter. + +**Fix:** `pkg install -y php85-simplexml php85-xmlwriter && service php_fpm restart` + +**Verification:** `Amazon_S3_And_CloudFront_Pro` class now instantiates; URLs rewrite to `https://s3.palmasolutions.net/...`. 101,145 files tracked in `ddf44d_as3cf_items`. + +### 2. wp-admin "This has been disabled." — by design + +Perfmatters plugin custom login URL feature: `login_url: intrarepresa`. Both `/wp-login.php` and `/wp-admin` are blocked with a 403. Login is at `/intrarepresa`. + +### 3. Articles redirecting to example.com + +Two concurrent causes: + +**a) Empty `permalink_structure`** — the DB import came from a server that had the structure set, but something reset it on the new DB. Pretty URLs were broken entirely. + +Fix: `wp --allow-root rewrite structure '/%year%/%monthnum%/%day%/%postname%/' --hard && wp rewrite flush` + +**b) InformatiQ-Toolkit security plugin** — was configured with `redirect_url: https://example.com/blocked` and `response_code: 301_custom`. Any request the plugin classified as a bot or bad actor got a 301 to example.com, which was then cached by nginx (`fastcgi_cache_valid 200 301 302 24h`) for 24 hours. + +Fix: `response_code` changed to `403`, `redirect_url` cleared. + +Note: nginx's `fastcgi_cache_valid` should not cache 301/302. Consider changing to `fastcgi_cache_valid 200 24h` only. + +### 4. Redis Object Cache plugin inactive + +The `object-cache.php` drop-in was installed and functioning (Redis was receiving cache hits). However the plugin itself was marked inactive in WordPress — WP-CLI `wp redis` commands and the admin UI status panel were non-functional. + +Fix: `wp --allow-root plugin activate redis-cache` + +### 5. DB backup cron never ran + +The root crontab had `0 2 * * * root /root/db-backup.sh` — with `root` as a spurious user field (valid in `/etc/crontab` system format, not in user crontab format). Cron was trying to execute `root /root/db-backup.sh` as the command, which failed silently. No backups had run since before the migration. + +Fix: removed the `root` field. First successful backup: 137MB, completed in 19 seconds. + +### 6. Redis maxmemory unbounded + +Redis `maxmemory` was 0 (no limit) with `noeviction` policy. On a server with competing workloads, unbounded Redis with noeviction risks OOM if the object cache grows unexpectedly. + +Fix: `maxmemory 3gb`, `maxmemory-policy allkeys-lru`, persisted to `/usr/local/etc/redis.conf`. + +### 7. WP_DEBUG left on after migration + +`WP_DEBUG = true` was generating PHP notices on every uncached request, logged to nginx error log. Overhead on both disk I/O and PHP execution. + +Fix: `WP_DEBUG`, `WP_DEBUG_LOG`, `WP_DEBUG_DISPLAY` all set to `false`. + +### 8. nginx static location missing webp + +The static file location regex did not include `webp`. WPS3Media serves WebP variants; without the static location match, `.webp` requests fell through to PHP-FPM, which loaded WordPress, checked the DB, found nothing local, and generated errors. + +Fix: added `webp|mp4|webm|ogv` to the static location regex. + +--- + +## Current Performance (2026-05-31) + +### zamolxis + +| Metric | Value | +|---|---| +| Load average | 1.76 (6 vCPU) | +| RAM | 24GB, no swap | +| Disk | 10GB / 75GB (14%) | +| nginx cache | 8,288 files, 1.0GB | +| Redis hit rate | 86.9% | +| Redis memory | 1.03GB / 3GB cap | +| OPcache | 99.6%+ hit rate, 3,343 cached scripts | + +### decebal + +| Metric | Value | +|---|---| +| Load average | 0.06 (6 vCPU) | +| RAM | No swap, ~18GB free | +| MySQL DB size | 1,850MB | +| Posts | 143,266 | +| Threads connected | 4 | +| Slow queries | 73 in 2 days | +| Disk (MySQL data) | 3.4GB / 94GB (4%) | +| Backup | 137MB daily, verified | + +--- + +## Deployment Status + +| Item | Status | +|---|---| +| zamolxis provisioned and configured | Done | +| decebal provisioned and configured | Done | +| rsync (7.1GB webroot) | Done | +| DB import (143,266 posts, 121 tables) | Done — 2026-05-31 | +| NPM cutover (dracula → zamolxis) | Done — 2026-05-31 | +| Tailscale IPs confirmed | zamolxis: 100.115.128.41 / decebal: 100.67.166.29 | +| NPM Tailscale IP confirmed | 100.89.238.4 | +| pf HTTP rule locked to NPM | Done | +| php85-simplexml installed (WPS3Media) | Done | +| WPS3Media S3 rewrite working | Done — 101,145 files | +| Redis Object Cache plugin active | Done | +| Redis maxmemory capped | Done — 3GB allkeys-lru | +| WP_DEBUG disabled | Done | +| nginx static location includes webp | Done | +| InformatiQ-Toolkit redirect fixed | Done — 403 instead of 301→example.com | +| Permalink structure restored | Done | +| DB backup cron fixed and verified | Done — 137MB/run | +| Temp credential files removed | Done | +| Migration key removed | Done | +| Cache warmer running | Done — @reboot + daily 03:00 | +| Old VMs (dracula, transilvan) | Keep powered, no traffic — decommission after 7 days stable |