agent-skills/skills/mailcow-diagnostics/SKILL.md

name, description

name

description

mailcow-diagnostics

Use when diagnosing, auditing, backing up, updating, or triaging a mailcow-dockerized instance in this fleet (geeky, posta, postman, spoof, or any future mailcow host) — deliverability issues, spam/rejection complaints, container health, mail queue backlog, certificate problems, or a scheduled multi-tenant mail audit. Covers external DNS/TLS/blacklist checks plus mailcow's real container names, official backup/restore script, and update flow.

Mailcow Diagnostics

Diagnostic and operational reference for mailcow-dockerized instances in this fleet. Provenance: the external (DNS/TLS/reputation) checks are adapted from arova-ai/agent-skills — email-server-diagnostics (generic self-hosted mail diagnostics, not mailcow-specific). The internal (container/service) commands, backup/restore/update procedures, and multi-tenant guidance below are grounded in this fleet's own direct migration experience consolidating three mailcow instances — geeky, posta, and postman — onto a new host (spoof), where real container names, config paths, and the official helper scripts were confirmed live via docker ps and by reading mailcow's own docker-compose.yml and helper-scripts/.

CRITICAL: this is multi-tenant mail hosting, not one domain per instance

Every mailcow instance in this fleet hosts many unrelated businesses on one shared stack — one real instance observed in this fleet's migration served 26+ distinct domains for entirely separate customers. A report of "mail is broken" almost always means one domain or one mailbox is affected, not the whole instance.

Before taking any remediation action:

• Scope every diagnostic and every fix to the specific domain/mailbox reported broken. Never restart a shared container, flush a shared queue, or touch instance-wide config (mailcow.conf, docker-compose.yml) to fix one customer's problem unless you have first confirmed the issue is genuinely instance-wide (e.g. all domains show the same DNS/cert failure).
• Postfix's mail queue (postqueue -p), Rspamd, and the shared IP's reputation/blacklist status are genuinely shared resources — a queue backlog or blacklisting caused by one domain's outbound spam can degrade deliverability for every other domain on the same instance. Identify the actual source domain/sender in the queue or logs before concluding "the instance is blacklisted" is the customer's problem to fix vs. everyone's.
• When in doubt about blast radius, prefer read-only diagnosis (postqueue -p, log greps scoped to the domain, SELECT ... WHERE domain=...) over any command that acts on all domains/mailboxes.

Identify the target before anything else

# Confirm which host/instance and get real container names — don't guess
docker ps --format "table {{.Names}}\t{{.Status}}" | grep mailcow

Real container naming pattern (confirmed via docker ps on live hosts in this fleet): mailcowdockerized-<service>-mailcow-1, e.g. mailcowdockerized-postfix-mailcow-1, mailcowdockerized-dovecot-mailcow-1, mailcowdockerized-rspamd-mailcow-1, mailcowdockerized-mysql-mailcow-1. The exact prefix depends on the install directory's basename (Compose derives the project name from it), so confirm with docker ps on the actual host rather than assuming the prefix — it may differ from mailcowdockerized- if the directory was cloned under a different name.

The install itself normally lives at /opt/mailcow-dockerized, containing mailcow.conf (MAILCOW_HOSTNAME, HTTP_PORT/HTTPS_PORT, SKIP_* flags, DB/Redis credentials) and docker-compose.yml. Two of the three hosts touched in this fleet's migration used a non-standard SSH port (79, not 22) — check ~/.ssh/config or ask rather than assuming 22.

When compose project context is available (you're cd'd into /opt/mailcow-dockerized), prefer:

cd /opt/mailcow-dockerized
docker compose exec -T <service>-mailcow <command>

Otherwise target the container directly by its confirmed full name:

docker exec <confirmed-full-container-name> <command>

Diagnostic sequence

Run external checks first — no server access needed, and they establish whether the problem is DNS/reputation (affects one domain) vs. shared infrastructure (affects the whole instance). Internal checks require SSH into the mailcow host.

1. External: DNS records (per affected domain)

Record	Lookup	What to verify
MX	dig MX <domain> +short	Points to correct hostname; priority reasonable; no stale entries
A / AAAA	dig A <hostname> + dig AAAA <hostname>	Resolves to expected IP; if AAAA exists, IPv6 must be fully configured
SPF	dig TXT <domain> +short	Has v=spf1; includes server IP or MX; ends -all or ~all; exactly ONE SPF record (multiple = error)
DKIM	dig TXT <selector>._domainkey.<domain>	Valid public key; get the real selector from mailcow itself (see below) rather than guessing dkim/default/mail
DMARC	dig TXT _dmarc.<domain>	v=DMARC1; policy quarantine or reject for production; has rua= for reporting
rDNS (PTR)	dig -x <ip> +short	Matches mail hostname exactly
Forward-confirmed rDNS	PTR → A lookup	PTR hostname must resolve back to the original IP
MTA-STS	dig TXT _mta-sts.<domain> + curl https://mta-sts.<domain>/.well-known/mta-sts.txt	TXT has v=STSv1; id=...; policy file accessible over HTTPS
TLSRPT	dig TXT _smtp._tls.<domain>	Has v=TLSRPTv1; rua=mailto:...
DANE/TLSA	dig TLSA _25._tcp.<hostname>	If DNSSEC enabled, TLSA record should exist (usage 3, selector 1, type 1 recommended)
BIMI	dig TXT default._bimi.<domain>	Optional; requires DMARC p=reject/quarantine with pct=100
Autodiscover	dig A autodiscover.<domain> + dig A autoconfig.<domain>	Records exist for mail client auto-configuration
SRV	dig SRV _imaps._tcp.<domain>, _submission._tcp.<domain>, etc.	Service records for IMAP, SMTP, CalDAV, CardDAV

Also check for stale records: old provider DKIM CNAMEs, conflicting SPF records, orphaned MX entries from a domain's pre-migration mail provider — common when onboarding a domain onto a consolidated instance.

SPF deep validation — these limits are almost never checked manually and cause silent failures:

• DNS lookup count (max 10): count all include/a/mx/redirect/ exists mechanisms recursively across the SPF chain. Exceeding 10 = PermError = SPF fails outright.
• Void lookup limit (max 2): DNS queries returning NXDOMAIN or empty count as void lookups; more than 2 = PermError. Often caused by stale include: references to a decommissioned prior mail provider.
• Flag: +all/?all (too permissive), deprecated ptr mechanism, multiple SPF TXT records on one domain.

DKIM validation: key length (1024-bit = WARNING, should rotate to 2048-bit; 2048-bit = OK), t=y (testing mode) still set in production, selector matches what mailcow actually generated for that domain (see below — don't assume a selector name).

2. External: TLS certificates

# HTTPS (web UI / autodiscover)
echo | openssl s_client -connect <ip>:443 -servername <hostname> 2>/dev/null | \
  openssl x509 -noout -subject -issuer -dates -ext subjectAltName

# IMAPS / SMTPS / SMTP STARTTLS / Submission STARTTLS
echo | openssl s_client -connect <ip>:993 -servername <hostname> 2>/dev/null | openssl x509 -noout -subject -issuer -dates
echo | openssl s_client -connect <ip>:465 -servername <hostname> 2>/dev/null | openssl x509 -noout -subject -issuer -dates
echo | openssl s_client -connect <ip>:25  -starttls smtp -servername <hostname> 2>/dev/null | openssl x509 -noout -subject -issuer -dates
echo | openssl s_client -connect <ip>:587 -starttls smtp -servername <hostname> 2>/dev/null | openssl x509 -noout -subject -issuer -dates

# Protocol version — TLS 1.0/1.1 should NOT be accepted
openssl s_client -connect <ip>:25 -starttls smtp -tls1   2>&1 | grep "Protocol"
openssl s_client -connect <ip>:25 -starttls smtp -tls1_1 2>&1 | grep "Protocol"

Since a single mailcow instance serves many domains behind one cert (mailcow issues one Let's Encrypt cert covering MAILCOW_HOSTNAME plus every active mail domain as SANs), a cert problem here is almost always instance-wide, not domain-specific — treat expiry/self-signed findings as higher urgency and check the ACME container (below) rather than looking for a per-domain cause.

3. External: port connectivity & SMTP banner

Port	Protocol	Purpose
25	SMTP	Incoming mail
80 / 443	HTTP/HTTPS	Web UI, ACME challenges, autodiscover
110 / 995	POP3 / POP3S	Legacy retrieval
143 / 993	IMAP / IMAPS	Mail retrieval
465	SMTPS	Submission (implicit TLS)
587	Submission	Submission (STARTTLS)
4190	Sieve	Mail filtering

nc -z -w5 <ip> <port> && echo open || echo closed
echo "EHLO test" | nc -w5 <ip> 25    # banner hostname, STARTTLS advertised, 250 codes

HELO hostname consistency: the SMTP banner hostname should match the PTR record, which should match the A record — this three-way match (forward-confirmed reverse DNS) is critical for deliverability and is instance-wide (all domains share the same MX/banner hostname).

4. External: IP/domain reputation (DNSBLs)

IP="<server-ip>"
REV=$(echo $IP | awk -F. '{print $4"."$3"."$2"."$1}')
for bl in zen.spamhaus.org b.barracudacentral.org bl.spamcop.net dnsbl.sorbs.net; do
  result=$(dig +short ${REV}.${bl} 2>/dev/null)
  [ -z "$result" ] && echo "$bl: CLEAN" || echo "$bl: LISTED ($result)"
done
for bl in dbl.spamhaus.org multi.surbl.org multi.uribl.com; do
  result=$(dig +short <domain>.${bl} 2>/dev/null)
  [ -z "$result" ] && echo "$bl: CLEAN" || echo "$bl: LISTED ($result)"
done

Since the IP/hostname is shared across every domain on the instance, an IP-level DNSBL listing affects all tenants at once — this is exactly the shared-resource case from the multi-tenant note above: find which domain's outbound traffic actually got the IP listed (queue/log investigation, section 6) rather than treating every affected domain as independently broken.

5. Do NOT run active open-relay/spam tests without explicit authorization

The source material this skill is adapted from includes a live open-relay probe (connect from an external host, attempt RCPT TO an outside address without auth, confirm 554/550 reject). Keep the caution, not the habit of running it casually: an unauthorized relay/spam-injection test against a shared multi-tenant mail instance can itself look like abuse traffic to the very DNSBLs being checked, and risks a false-positive listing that then affects every other domain on the instance. Only run an active relay test with the instance owner's explicit go-ahead, and prefer reading smtpd_relay_restrictions from the live Postfix config (section 6) to confirm reject_unauth_destination is present — that's sufficient evidence in almost every case without an active probe.

6. Internal: mailcow container/service checks (requires SSH)

cd /opt/mailcow-dockerized   # confirm this path on the actual host first

# Key config values — confirm names against the live mailcow.conf, don't
# assume every key below still matches every mailcow version
grep -E "^(MAILCOW_HOSTNAME|SKIP_LETS_ENCRYPT|ENABLE_IPV6|ACL_ANYONE|HTTP_PORT|HTTPS_PORT)" mailcow.conf

DBUSER=$(grep ^DBUSER mailcow.conf | cut -d= -f2)
DBPASS=$(grep ^DBPASS mailcow.conf | cut -d= -f2)
REDISPASS=$(grep ^REDISPASS mailcow.conf | cut -d= -f2)

# Domains on this instance — establishes the actual tenant list
docker compose exec -T mysql-mailcow mysql -u"$DBUSER" -p"$DBPASS" mailcow \
  -e "SELECT domain,active,backupmx FROM domain;"

# One domain's mailboxes/aliases — scope to the reported domain, not SELECT *
docker compose exec -T mysql-mailcow mysql -u"$DBUSER" -p"$DBPASS" mailcow \
  -e "SELECT username,domain,active,quota FROM mailbox WHERE domain='<affected-domain>';"
docker compose exec -T mysql-mailcow mysql -u"$DBUSER" -p"$DBPASS" mailcow \
  -e "SELECT address,goto,active FROM alias WHERE domain='<affected-domain>' AND address NOT LIKE '@%';"

# DKIM selectors/keys actually in use (Redis) — get the real selector,
# don't guess it for the DNS DKIM lookup in section 1
docker compose exec -T redis-mailcow redis-cli -a "$REDISPASS" HGETALL DKIM_SELECTORS
docker compose exec -T redis-mailcow redis-cli -a "$REDISPASS" HKEYS DKIM_PRIV_KEYS

# Container health — check the whole stack, a customer-facing symptom can
# trace to an unrelated dependency container being down
docker compose ps --format "table {{.Name}}\t{{.Status}}"

# Postfix config highlights, incl. open-relay protection
docker compose exec -T postfix-mailcow postconf -n | \
  grep -E "(myhostname|relayhost|smtpd_tls|smtpd_relay_restrictions)"

# Mail queue — inspect before assuming instance-wide backlog; grep for the
# specific domain to see whose mail is actually stuck
docker compose exec -T postfix-mailcow postqueue -p
docker compose exec -T postfix-mailcow postqueue -p | grep -c '^[A-F0-9]'   # rough depth

# Rspamd status
docker compose exec -T rspamd-mailcow rspamc stat 2>/dev/null | head -20

# Dovecot — mailbox/log inspection for one user
docker compose exec -T dovecot-mailcow doveconf -n | grep -E "(ssl|auth_mechanisms|mail_location)"
docker compose logs dovecot-mailcow --tail 100 | grep '<affected-user>@<affected-domain>'

Internal checks to verify:

• All containers/services running and healthy (docker compose ps)
• DKIM private keys exist and the DNS public key selector actually matches what's in Redis for that domain (don't guess the selector)
• Let's Encrypt working (not a self-signed fallback — see certificates below)
• Postfix myhostname matches DNS and rDNS
• smtpd_relay_restrictions contains reject_unauth_destination (open relay protection)
• Mail queue not backed up (>100 = investigate, >1000 = problem) — and identify whose mail before declaring it "the instance's" problem
• Rspamd running and processing (not bypassed)

7. Certificate renewal — mailcow's own ACME container

mailcow runs Let's Encrypt itself through a dedicated container, acme-mailcow (service name in docker-compose.yml, so the live container is <project-prefix>-acme-mailcow-1 following the same naming pattern as every other service). Don't invent a separate certbot/ACME mechanism — use mailcow's own:

docker compose logs acme-mailcow --tail 50    # renewal attempts/errors
docker compose restart acme-mailcow           # nudge a retry if genuinely stuck

Relevant mailcow.conf flags: SKIP_LETS_ENCRYPT (if set to y, mailcow is not managing certs at all — an external reverse proxy/cert is expected instead, which changes where to look for the real cause of a cert problem) and ADDITIONAL_SAN. Since the cert covers every active domain on the instance as a SAN, a domain that was just added/re-enabled may need an ACME container restart before its SAN is picked up — verify against the live mailcow.conf and current mailcow docs/changelog for the exact mechanism on the version actually deployed, this area has changed across mailcow releases and is not safe to assert further without checking the live instance.

Backup, restore, and update — use mailcow's own official tooling

Never invent a backup mechanism. mailcow-dockerized ships its own sanctioned script at helper-scripts/backup_and_restore.sh, confirmed directly against the upstream script:

cd /opt/mailcow-dockerized

# Backup one or more components (repeat/select as needed — not always "all")
./helper-scripts/backup_and_restore.sh backup {vmail|crypt|redis|rspamd|postfix|mysql|all}

# Restore
./helper-scripts/backup_and_restore.sh restore {vmail|crypt|redis|rspamd|postfix|mysql|all}

The script prompts for (or reads from $MAILCOW_BACKUP_LOCATION) an absolute backup directory path, and supports --delete-days N for backup retention pruning. THREADS=N can be set in the environment to parallelize. Because this is a shared multi-tenant instance, mysql/vmail/crypt backups are inherently all-domains-at-once — there is no official per-domain backup mode. A restore of mysql or vmail is instance-wide and reversible only from a prior full backup; treat it as R8-class (destructive, needs explicit confirmation) even when the motivating complaint was a single domain, since the restore itself affects every tenant.

Updates: run the official ./update.sh from inside /opt/mailcow-dockerized (must be run from that directory — it checks for mailcow.conf in the cwd first). Do not update via a raw docker compose pull && docker compose up -d — update.sh handles config migration, _modules updates, and version-specific migration steps that a bare pull skips.

cd /opt/mailcow-dockerized
./update.sh          # interactive
./update.sh --dev    # skip _modules update, developer mode — don't use in production without knowing why

Take a backup (above) before any update, same multi-tenant caution: an update problem is instance-wide, not scoped to one domain.

Google/Yahoo/Microsoft bulk-sender compliance (2024-2025 requirements)

These are enforced with permanent rejections. Check per-domain, since compliance posture (DMARC policy, sending volume) can legitimately differ between tenants on the same instance.

All senders:

• SPF or DKIM set up (at minimum)
• Valid PTR (forward-confirmed reverse DNS) — instance-wide, shared
• TLS for mail transmission
• RFC 5322 compliant message formatting
• Spam rate < 0.3% (Google Postmaster Tools, per sending domain)

Bulk senders (>5,000 emails/day) additionally:

• Both SPF AND DKIM
• DMARC with at least p=none and alignment pass
• One-click unsubscribe headers (List-Unsubscribe + List-Unsubscribe-Post)
• ARC headers when forwarding
• From: domain aligned with SPF or DKIM domain

Microsoft Outlook (May 2025+): SPF, DKIM, and DMARC all required for

5,000 emails/day to Outlook/Hotmail/Live.

These external-provider policies change; treat the checklist above as a snapshot and verify against current Google/Microsoft postmaster documentation before treating an item as authoritative, especially for a compliance-critical audit.

Report format

## <domain> Mailcow Diagnostic Report (instance: <host>)

### Instance overview
| Item | Value |
|---|---|
| Mailcow host | ... |
| Install path | /opt/mailcow-dockerized |
| MAILCOW_HOSTNAME | ... |
| Total domains on instance | N (multi-tenant — see scope note) |
| Container health | X/Y running |

### Scope of this report
Affected domain(s): ... | Instance-wide finding: yes/no

### DNS Records (this domain)
| Record | Value | Status |
|---|---|---|

### SPF / DKIM Analysis
...

### TLS Certificates (instance-wide unless noted)
| Protocol | Port | Issuer | Expires | TLS Version | Status |
|---|---|---|---|---|---|

### Blacklist Status
| Blacklist | Status |
|---|---|

### Mail Queue (scoped to affected domain where possible)
...

### Issues Found
1. **CRITICAL** — blocks delivery/rejection
2. **WARNING** — degrades deliverability/security
3. **INFO** — best-practice

### Recommended Actions
(numbered, priority order, explicit about blast radius — "affects only
<domain>" vs "affects all N domains on this instance")

Severity guidelines

CRITICAL: MX missing/wrong; SPF/DKIM/DMARC completely absent; SPF exceeds 10 DNS lookups; TLS cert expired/invalid on mail ports; port 25 unreachable; open relay detected; IP listed on Spamhaus ZEN (instance-wide impact); IPv6 AAAA exists but no IPv6 PTR; HELO hostname doesn't match rDNS.

WARNING: rDNS doesn't match mail hostname; SPF ~all with DMARC p=reject (inconsistent); DMARC still p=none in production; DKIM 1024-bit key; self-signed cert on any port; MTA-STS missing; stale DNS records from a domain's pre-migration provider; ACME container failing; mail queue >100; TLS 1.0/1.1 still accepted; non-compliant with bulk sender requirements.

INFO: TLSRPT not configured; BIMI not set up; DANE/TLSA not configured (needs DNSSEC); ARC not configured for forwarding; SRV autodiscover records missing; POP3 open but unused; no evident DKIM key rotation; void lookup count close to the SPF limit.