From 3b8ef056cbcc1e1a19b7f404481bc9e95d21cc03 Mon Sep 17 00:00:00 2001 From: Malin Date: Sat, 15 Aug 2026 21:25:05 +0200 Subject: [PATCH] Add mailcow-diagnostics skill for multi-tenant mailcow-dockerized ops External DNS/SPF/DKIM/DMARC/MTA-STS/TLS/DNSBL diagnostics adapted from arova-ai/agent-skills' email-server-diagnostics, combined with this session's own direct mailcow migration experience (geeky/posta/postman -> spoof consolidation): real container naming pattern (mailcowdockerized--mailcow-1), mailcow's own official helper-scripts/backup_and_restore.sh and update.sh, the acme-mailcow cert container, and an explicit multi-tenant blast-radius guardrail since one instance serves many unrelated businesses' domains. Co-Authored-By: Claude Sonnet 5 --- skills/mailcow-diagnostics/SKILL.md | 424 ++++++++++++++++++++++++++++ 1 file changed, 424 insertions(+) create mode 100644 skills/mailcow-diagnostics/SKILL.md diff --git a/skills/mailcow-diagnostics/SKILL.md b/skills/mailcow-diagnostics/SKILL.md new file mode 100644 index 0000000..04ea9d2 --- /dev/null +++ b/skills/mailcow-diagnostics/SKILL.md @@ -0,0 +1,424 @@ +--- +name: mailcow-diagnostics +description: Use when diagnosing, auditing, backing up, updating, or triaging a mailcow-dockerized instance in this fleet (geeky, posta, postman, spoof, or any future mailcow host) — deliverability issues, spam/rejection complaints, container health, mail queue backlog, certificate problems, or a scheduled multi-tenant mail audit. Covers external DNS/TLS/blacklist checks plus mailcow's real container names, official backup/restore script, and update flow. +--- + +# Mailcow Diagnostics + +Diagnostic and operational reference for **mailcow-dockerized** instances in +this fleet. Provenance: the external (DNS/TLS/reputation) checks are adapted +from [`arova-ai/agent-skills` — `email-server-diagnostics`](https://github.com/arova-ai/agent-skills/tree/main/email-server-diagnostics) +(generic self-hosted mail diagnostics, not mailcow-specific). The internal +(container/service) commands, backup/restore/update procedures, and +multi-tenant guidance below are grounded in this fleet's own direct +migration experience consolidating three mailcow instances — `geeky`, +`posta`, and `postman` — onto a new host (`spoof`), where real container +names, config paths, and the official helper scripts were confirmed live via +`docker ps` and by reading mailcow's own `docker-compose.yml` and +`helper-scripts/`. + +## CRITICAL: this is multi-tenant mail hosting, not one domain per instance + +**Every mailcow instance in this fleet hosts many unrelated businesses on +one shared stack** — one real instance observed in this fleet's migration +served 26+ distinct domains for entirely separate customers. A report of +"mail is broken" almost always means *one domain or one mailbox* is +affected, not the whole instance. + +Before taking any remediation action: +- **Scope every diagnostic and every fix to the specific domain/mailbox + reported broken.** Never restart a shared container, flush a shared + queue, or touch instance-wide config (`mailcow.conf`, `docker-compose.yml`) + to fix one customer's problem unless you have first confirmed the issue is + genuinely instance-wide (e.g. all domains show the same DNS/cert failure). +- Postfix's mail queue (`postqueue -p`), Rspamd, and the shared IP's + reputation/blacklist status are genuinely shared resources — a queue + backlog or blacklisting caused by one domain's outbound spam can degrade + deliverability for every other domain on the same instance. Identify the + actual source domain/sender in the queue or logs before concluding "the + instance is blacklisted" is the customer's problem to fix vs. everyone's. +- When in doubt about blast radius, prefer read-only diagnosis + (`postqueue -p`, log greps scoped to the domain, `SELECT ... WHERE + domain=...`) over any command that acts on `all` domains/mailboxes. + +## Identify the target before anything else + +```bash +# Confirm which host/instance and get real container names — don't guess +docker ps --format "table {{.Names}}\t{{.Status}}" | grep mailcow +``` + +Real container naming pattern (confirmed via `docker ps` on live hosts in +this fleet): `mailcowdockerized--mailcow-1`, e.g. +`mailcowdockerized-postfix-mailcow-1`, `mailcowdockerized-dovecot-mailcow-1`, +`mailcowdockerized-rspamd-mailcow-1`, `mailcowdockerized-mysql-mailcow-1`. +The exact prefix depends on the install directory's basename (Compose +derives the project name from it), so **confirm with `docker ps` on the +actual host rather than assuming the prefix** — it may differ from +`mailcowdockerized-` if the directory was cloned under a different name. + +The install itself normally lives at `/opt/mailcow-dockerized`, containing +`mailcow.conf` (`MAILCOW_HOSTNAME`, `HTTP_PORT`/`HTTPS_PORT`, `SKIP_*` +flags, DB/Redis credentials) and `docker-compose.yml`. Two of the three +hosts touched in this fleet's migration used a non-standard SSH port +(**79**, not 22) — check `~/.ssh/config` or ask rather than assuming 22. + +When compose project context is available (you're `cd`'d into +`/opt/mailcow-dockerized`), prefer: +```bash +cd /opt/mailcow-dockerized +docker compose exec -T -mailcow +``` +Otherwise target the container directly by its confirmed full name: +```bash +docker exec +``` + +## Diagnostic sequence + +Run external checks first — no server access needed, and they establish +whether the problem is DNS/reputation (affects one domain) vs. shared +infrastructure (affects the whole instance). Internal checks require SSH +into the mailcow host. + +### 1. External: DNS records (per affected domain) + +| Record | Lookup | What to verify | +|---|---|---| +| **MX** | `dig MX +short` | Points to correct hostname; priority reasonable; no stale entries | +| **A / AAAA** | `dig A ` + `dig AAAA ` | Resolves to expected IP; if AAAA exists, IPv6 must be fully configured | +| **SPF** | `dig TXT +short` | Has `v=spf1`; includes server IP or MX; ends `-all` or `~all`; exactly ONE SPF record (multiple = error) | +| **DKIM** | `dig TXT ._domainkey.` | Valid public key; **get the real selector from mailcow itself** (see below) rather than guessing `dkim`/`default`/`mail` | +| **DMARC** | `dig TXT _dmarc.` | `v=DMARC1`; policy `quarantine` or `reject` for production; has `rua=` for reporting | +| **rDNS (PTR)** | `dig -x +short` | Matches mail hostname exactly | +| **Forward-confirmed rDNS** | PTR → A lookup | PTR hostname must resolve back to the original IP | +| **MTA-STS** | `dig TXT _mta-sts.` + `curl https://mta-sts./.well-known/mta-sts.txt` | TXT has `v=STSv1; id=...`; policy file accessible over HTTPS | +| **TLSRPT** | `dig TXT _smtp._tls.` | Has `v=TLSRPTv1; rua=mailto:...` | +| **DANE/TLSA** | `dig TLSA _25._tcp.` | If DNSSEC enabled, TLSA record should exist (usage 3, selector 1, type 1 recommended) | +| **BIMI** | `dig TXT default._bimi.` | Optional; requires DMARC p=reject/quarantine with pct=100 | +| **Autodiscover** | `dig A autodiscover.` + `dig A autoconfig.` | Records exist for mail client auto-configuration | +| **SRV** | `dig SRV _imaps._tcp.`, `_submission._tcp.`, etc. | Service records for IMAP, SMTP, CalDAV, CardDAV | + +Also check for stale records: old provider DKIM CNAMEs, conflicting SPF +records, orphaned MX entries from a domain's pre-migration mail provider — +common when onboarding a domain onto a consolidated instance. + +**SPF deep validation** — these limits are almost never checked manually +and cause silent failures: +- **DNS lookup count (max 10)**: count all `include`/`a`/`mx`/`redirect`/ + `exists` mechanisms recursively across the SPF chain. Exceeding 10 = + PermError = SPF fails outright. +- **Void lookup limit (max 2)**: DNS queries returning NXDOMAIN or empty + count as void lookups; more than 2 = PermError. Often caused by stale + `include:` references to a decommissioned prior mail provider. +- Flag: `+all`/`?all` (too permissive), deprecated `ptr` mechanism, + multiple SPF TXT records on one domain. + +**DKIM validation**: key length (1024-bit = WARNING, should rotate to +2048-bit; 2048-bit = OK), `t=y` (testing mode) still set in production, +selector matches what mailcow actually generated for that domain (see +below — don't assume a selector name). + +### 2. External: TLS certificates + +```bash +# HTTPS (web UI / autodiscover) +echo | openssl s_client -connect :443 -servername 2>/dev/null | \ + openssl x509 -noout -subject -issuer -dates -ext subjectAltName + +# IMAPS / SMTPS / SMTP STARTTLS / Submission STARTTLS +echo | openssl s_client -connect :993 -servername 2>/dev/null | openssl x509 -noout -subject -issuer -dates +echo | openssl s_client -connect :465 -servername 2>/dev/null | openssl x509 -noout -subject -issuer -dates +echo | openssl s_client -connect :25 -starttls smtp -servername 2>/dev/null | openssl x509 -noout -subject -issuer -dates +echo | openssl s_client -connect :587 -starttls smtp -servername 2>/dev/null | openssl x509 -noout -subject -issuer -dates + +# Protocol version — TLS 1.0/1.1 should NOT be accepted +openssl s_client -connect :25 -starttls smtp -tls1 2>&1 | grep "Protocol" +openssl s_client -connect :25 -starttls smtp -tls1_1 2>&1 | grep "Protocol" +``` + +Since a single mailcow instance serves many domains behind one cert (mailcow +issues one Let's Encrypt cert covering `MAILCOW_HOSTNAME` plus every active +mail domain as SANs), a cert problem here is almost always **instance-wide**, +not domain-specific — treat expiry/self-signed findings as higher urgency +and check the ACME container (below) rather than looking for a per-domain +cause. + +### 3. External: port connectivity & SMTP banner + +| Port | Protocol | Purpose | +|---|---|---| +| 25 | SMTP | Incoming mail | +| 80 / 443 | HTTP/HTTPS | Web UI, ACME challenges, autodiscover | +| 110 / 995 | POP3 / POP3S | Legacy retrieval | +| 143 / 993 | IMAP / IMAPS | Mail retrieval | +| 465 | SMTPS | Submission (implicit TLS) | +| 587 | Submission | Submission (STARTTLS) | +| 4190 | Sieve | Mail filtering | + +```bash +nc -z -w5 && echo open || echo closed +echo "EHLO test" | nc -w5 25 # banner hostname, STARTTLS advertised, 250 codes +``` + +**HELO hostname consistency**: the SMTP banner hostname should match the +PTR record, which should match the A record — this three-way match +(forward-confirmed reverse DNS) is critical for deliverability and is +instance-wide (all domains share the same MX/banner hostname). + +### 4. External: IP/domain reputation (DNSBLs) + +```bash +IP="" +REV=$(echo $IP | awk -F. '{print $4"."$3"."$2"."$1}') +for bl in zen.spamhaus.org b.barracudacentral.org bl.spamcop.net dnsbl.sorbs.net; do + result=$(dig +short ${REV}.${bl} 2>/dev/null) + [ -z "$result" ] && echo "$bl: CLEAN" || echo "$bl: LISTED ($result)" +done +for bl in dbl.spamhaus.org multi.surbl.org multi.uribl.com; do + result=$(dig +short .${bl} 2>/dev/null) + [ -z "$result" ] && echo "$bl: CLEAN" || echo "$bl: LISTED ($result)" +done +``` + +Since the IP/hostname is shared across every domain on the instance, an +IP-level DNSBL listing affects **all tenants at once** — this is exactly +the shared-resource case from the multi-tenant note above: find which +domain's outbound traffic actually got the IP listed (queue/log +investigation, section 6) rather than treating every affected domain as +independently broken. + +### 5. Do NOT run active open-relay/spam tests without explicit authorization + +The source material this skill is adapted from includes a live open-relay +probe (connect from an external host, attempt `RCPT TO` an outside address +without auth, confirm `554`/`550` reject). **Keep the caution, not the +habit of running it casually**: an unauthorized relay/spam-injection test +against a shared multi-tenant mail instance can itself look like abuse +traffic to the very DNSBLs being checked, and risks a false-positive +listing that then affects every other domain on the instance. Only run an +active relay test with the instance owner's explicit go-ahead, and prefer +reading `smtpd_relay_restrictions` from the live Postfix config (section 6) +to confirm `reject_unauth_destination` is present — that's sufficient +evidence in almost every case without an active probe. + +### 6. Internal: mailcow container/service checks (requires SSH) + +```bash +cd /opt/mailcow-dockerized # confirm this path on the actual host first + +# Key config values — confirm names against the live mailcow.conf, don't +# assume every key below still matches every mailcow version +grep -E "^(MAILCOW_HOSTNAME|SKIP_LETS_ENCRYPT|ENABLE_IPV6|ACL_ANYONE|HTTP_PORT|HTTPS_PORT)" mailcow.conf + +DBUSER=$(grep ^DBUSER mailcow.conf | cut -d= -f2) +DBPASS=$(grep ^DBPASS mailcow.conf | cut -d= -f2) +REDISPASS=$(grep ^REDISPASS mailcow.conf | cut -d= -f2) + +# Domains on this instance — establishes the actual tenant list +docker compose exec -T mysql-mailcow mysql -u"$DBUSER" -p"$DBPASS" mailcow \ + -e "SELECT domain,active,backupmx FROM domain;" + +# One domain's mailboxes/aliases — scope to the reported domain, not SELECT * +docker compose exec -T mysql-mailcow mysql -u"$DBUSER" -p"$DBPASS" mailcow \ + -e "SELECT username,domain,active,quota FROM mailbox WHERE domain='';" +docker compose exec -T mysql-mailcow mysql -u"$DBUSER" -p"$DBPASS" mailcow \ + -e "SELECT address,goto,active FROM alias WHERE domain='' AND address NOT LIKE '@%';" + +# DKIM selectors/keys actually in use (Redis) — get the real selector, +# don't guess it for the DNS DKIM lookup in section 1 +docker compose exec -T redis-mailcow redis-cli -a "$REDISPASS" HGETALL DKIM_SELECTORS +docker compose exec -T redis-mailcow redis-cli -a "$REDISPASS" HKEYS DKIM_PRIV_KEYS + +# Container health — check the whole stack, a customer-facing symptom can +# trace to an unrelated dependency container being down +docker compose ps --format "table {{.Name}}\t{{.Status}}" + +# Postfix config highlights, incl. open-relay protection +docker compose exec -T postfix-mailcow postconf -n | \ + grep -E "(myhostname|relayhost|smtpd_tls|smtpd_relay_restrictions)" + +# Mail queue — inspect before assuming instance-wide backlog; grep for the +# specific domain to see whose mail is actually stuck +docker compose exec -T postfix-mailcow postqueue -p +docker compose exec -T postfix-mailcow postqueue -p | grep -c '^[A-F0-9]' # rough depth + +# Rspamd status +docker compose exec -T rspamd-mailcow rspamc stat 2>/dev/null | head -20 + +# Dovecot — mailbox/log inspection for one user +docker compose exec -T dovecot-mailcow doveconf -n | grep -E "(ssl|auth_mechanisms|mail_location)" +docker compose logs dovecot-mailcow --tail 100 | grep '@' +``` + +Internal checks to verify: +- All containers/services running and healthy (`docker compose ps`) +- DKIM private keys exist and the DNS public key selector actually matches + what's in Redis for that domain (don't guess the selector) +- Let's Encrypt working (not a self-signed fallback — see certificates below) +- Postfix `myhostname` matches DNS and rDNS +- `smtpd_relay_restrictions` contains `reject_unauth_destination` (open + relay protection) +- Mail queue not backed up (>100 = investigate, >1000 = problem) — and + identify whose mail before declaring it "the instance's" problem +- Rspamd running and processing (not bypassed) + +### 7. Certificate renewal — mailcow's own ACME container + +mailcow runs Let's Encrypt itself through a dedicated container, +**`acme-mailcow`** (service name in `docker-compose.yml`, so the live +container is `-acme-mailcow-1` following the same naming +pattern as every other service). Don't invent a separate certbot/ACME +mechanism — use mailcow's own: + +```bash +docker compose logs acme-mailcow --tail 50 # renewal attempts/errors +docker compose restart acme-mailcow # nudge a retry if genuinely stuck +``` + +Relevant `mailcow.conf` flags: `SKIP_LETS_ENCRYPT` (if set to `y`, mailcow +is not managing certs at all — an external reverse proxy/cert is expected +instead, which changes where to look for the real cause of a cert +problem) and `ADDITIONAL_SAN`. Since the cert covers every active domain on +the instance as a SAN, a domain that was just added/re-enabled may need an +ACME container restart before its SAN is picked up — **verify against the +live `mailcow.conf` and current mailcow docs/changelog for the exact +mechanism on the version actually deployed**, this area has changed across +mailcow releases and is not safe to assert further without checking the +live instance. + +## Backup, restore, and update — use mailcow's own official tooling + +**Never invent a backup mechanism.** mailcow-dockerized ships its own +sanctioned script at `helper-scripts/backup_and_restore.sh`, confirmed +directly against the upstream script: + +```bash +cd /opt/mailcow-dockerized + +# Backup one or more components (repeat/select as needed — not always "all") +./helper-scripts/backup_and_restore.sh backup {vmail|crypt|redis|rspamd|postfix|mysql|all} + +# Restore +./helper-scripts/backup_and_restore.sh restore {vmail|crypt|redis|rspamd|postfix|mysql|all} +``` + +The script prompts for (or reads from `$MAILCOW_BACKUP_LOCATION`) an +absolute backup directory path, and supports `--delete-days N` for backup +retention pruning. `THREADS=N` can be set in the environment to parallelize. +Because this is a shared multi-tenant instance, `mysql`/`vmail`/`crypt` +backups are inherently **all-domains-at-once** — there is no +official per-domain backup mode. A restore of `mysql` or `vmail` is +instance-wide and reversible only from a prior full backup; treat it as +R8-class (destructive, needs explicit confirmation) even when the +motivating complaint was a single domain, since the restore itself affects +every tenant. + +**Updates**: run the official `./update.sh` from inside +`/opt/mailcow-dockerized` (must be run from that directory — it checks for +`mailcow.conf` in the cwd first). Do not update via a raw `docker compose +pull && docker compose up -d` — `update.sh` handles config migration, +`_modules` updates, and version-specific migration steps that a bare pull +skips. + +```bash +cd /opt/mailcow-dockerized +./update.sh # interactive +./update.sh --dev # skip _modules update, developer mode — don't use in production without knowing why +``` + +Take a backup (above) before any update, same multi-tenant caution: an +update problem is instance-wide, not scoped to one domain. + +## Google/Yahoo/Microsoft bulk-sender compliance (2024-2025 requirements) + +These are enforced with permanent rejections. Check per-domain, since +compliance posture (DMARC policy, sending volume) can legitimately differ +between tenants on the same instance. + +**All senders:** +- [ ] SPF or DKIM set up (at minimum) +- [ ] Valid PTR (forward-confirmed reverse DNS) — instance-wide, shared +- [ ] TLS for mail transmission +- [ ] RFC 5322 compliant message formatting +- [ ] Spam rate < 0.3% (Google Postmaster Tools, per sending domain) + +**Bulk senders (>5,000 emails/day) additionally:** +- [ ] Both SPF AND DKIM +- [ ] DMARC with at least `p=none` and alignment pass +- [ ] One-click unsubscribe headers (`List-Unsubscribe` + + `List-Unsubscribe-Post`) +- [ ] ARC headers when forwarding +- [ ] From: domain aligned with SPF or DKIM domain + +**Microsoft Outlook (May 2025+)**: SPF, DKIM, and DMARC all required for +>5,000 emails/day to Outlook/Hotmail/Live. + +These external-provider policies change; treat the checklist above as a +snapshot and verify against current Google/Microsoft postmaster +documentation before treating an item as authoritative, especially for a +compliance-critical audit. + +## Report format + +```markdown +## Mailcow Diagnostic Report (instance: ) + +### Instance overview +| Item | Value | +|---|---| +| Mailcow host | ... | +| Install path | /opt/mailcow-dockerized | +| MAILCOW_HOSTNAME | ... | +| Total domains on instance | N (multi-tenant — see scope note) | +| Container health | X/Y running | + +### Scope of this report +Affected domain(s): ... | Instance-wide finding: yes/no + +### DNS Records (this domain) +| Record | Value | Status | +|---|---|---| + +### SPF / DKIM Analysis +... + +### TLS Certificates (instance-wide unless noted) +| Protocol | Port | Issuer | Expires | TLS Version | Status | +|---|---|---|---|---|---| + +### Blacklist Status +| Blacklist | Status | +|---|---| + +### Mail Queue (scoped to affected domain where possible) +... + +### Issues Found +1. **CRITICAL** — blocks delivery/rejection +2. **WARNING** — degrades deliverability/security +3. **INFO** — best-practice + +### Recommended Actions +(numbered, priority order, explicit about blast radius — "affects only +" vs "affects all N domains on this instance") +``` + +## Severity guidelines + +**CRITICAL**: MX missing/wrong; SPF/DKIM/DMARC completely absent; SPF +exceeds 10 DNS lookups; TLS cert expired/invalid on mail ports; port 25 +unreachable; open relay detected; IP listed on Spamhaus ZEN (instance-wide +impact); IPv6 AAAA exists but no IPv6 PTR; HELO hostname doesn't match rDNS. + +**WARNING**: rDNS doesn't match mail hostname; SPF `~all` with DMARC +`p=reject` (inconsistent); DMARC still `p=none` in production; DKIM +1024-bit key; self-signed cert on any port; MTA-STS missing; stale DNS +records from a domain's pre-migration provider; ACME container failing; +mail queue >100; TLS 1.0/1.1 still accepted; non-compliant with bulk sender +requirements. + +**INFO**: TLSRPT not configured; BIMI not set up; DANE/TLSA not configured +(needs DNSSEC); ARC not configured for forwarding; SRV autodiscover records +missing; POP3 open but unused; no evident DKIM key rotation; void lookup +count close to the SPF limit.