CloudHost/agent-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add mailcow-diagnostics skill for multi-tenant mailcow-dockerized ops

Browse Source 
External DNS/SPF/DKIM/DMARC/MTA-STS/TLS/DNSBL diagnostics adapted from
arova-ai/agent-skills' email-server-diagnostics, combined with this
session's own direct mailcow migration experience (geeky/posta/postman
-> spoof consolidation): real container naming pattern
(mailcowdockerized-<service>-mailcow-1), mailcow's own official
helper-scripts/backup_and_restore.sh and update.sh, the acme-mailcow
cert container, and an explicit multi-tenant blast-radius guardrail
since one instance serves many unrelated businesses' domains.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Malin
2026-08-15 21:25:05 +02:00
co-authored by Claude Sonnet 5
parent 65c01ddb43
commit 3b8ef056cb
1 changed files with 424 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/mailcow-diagnostics/SKILL.md
+424
View File
@@ -0,0 +1,424 @@
---
name: mailcow-diagnostics
description: Use when diagnosing, auditing, backing up, updating, or triaging a mailcow-dockerized instance in this fleet (geeky, posta, postman, spoof, or any future mailcow host) — deliverability issues, spam/rejection complaints, container health, mail queue backlog, certificate problems, or a scheduled multi-tenant mail audit. Covers external DNS/TLS/blacklist checks plus mailcow's real container names, official backup/restore script, and update flow.
---
# Mailcow Diagnostics
Diagnostic and operational reference for **mailcow-dockerized** instances in
this fleet. Provenance: the external (DNS/TLS/reputation) checks are adapted
from [`arova-ai/agent-skills` — `email-server-diagnostics`](https://github.com/arova-ai/agent-skills/tree/main/email-server-diagnostics)
(generic self-hosted mail diagnostics, not mailcow-specific). The internal
(container/service) commands, backup/restore/update procedures, and
multi-tenant guidance below are grounded in this fleet's own direct
migration experience consolidating three mailcow instances — `geeky`,
`posta`, and `postman` — onto a new host (`spoof`), where real container
names, config paths, and the official helper scripts were confirmed live via
`docker ps` and by reading mailcow's own `docker-compose.yml` and
`helper-scripts/`.
## CRITICAL: this is multi-tenant mail hosting, not one domain per instance
**Every mailcow instance in this fleet hosts many unrelated businesses on
one shared stack** — one real instance observed in this fleet's migration
served 26+ distinct domains for entirely separate customers. A report of
"mail is broken" almost always means *one domain or one mailbox* is
affected, not the whole instance.
Before taking any remediation action:
- **Scope every diagnostic and every fix to the specific domain/mailbox
reported broken.** Never restart a shared container, flush a shared
queue, or touch instance-wide config (`mailcow.conf`, `docker-compose.yml`)
to fix one customer's problem unless you have first confirmed the issue is
genuinely instance-wide (e.g. all domains show the same DNS/cert failure).
- Postfix's mail queue (`postqueue -p`), Rspamd, and the shared IP's
reputation/blacklist status are genuinely shared resources — a queue
backlog or blacklisting caused by one domain's outbound spam can degrade
deliverability for every other domain on the same instance. Identify the
actual source domain/sender in the queue or logs before concluding "the
instance is blacklisted" is the customer's problem to fix vs. everyone's.
- When in doubt about blast radius, prefer read-only diagnosis
(`postqueue -p`, log greps scoped to the domain, `SELECT ... WHERE
domain=...`) over any command that acts on `all` domains/mailboxes.
## Identify the target before anything else
```bash
# Confirm which host/instance and get real container names — don't guess
docker ps --format "table {{.Names}}\t{{.Status}}" | grep mailcow
```
Real container naming pattern (confirmed via `docker ps` on live hosts in
this fleet): `mailcowdockerized-<service>-mailcow-1`, e.g.
`mailcowdockerized-postfix-mailcow-1`, `mailcowdockerized-dovecot-mailcow-1`,
`mailcowdockerized-rspamd-mailcow-1`, `mailcowdockerized-mysql-mailcow-1`.
The exact prefix depends on the install directory's basename (Compose
derives the project name from it), so **confirm with `docker ps` on the
actual host rather than assuming the prefix** — it may differ from
`mailcowdockerized-` if the directory was cloned under a different name.
The install itself normally lives at `/opt/mailcow-dockerized`, containing
`mailcow.conf` (`MAILCOW_HOSTNAME`, `HTTP_PORT`/`HTTPS_PORT`, `SKIP_*`
flags, DB/Redis credentials) and `docker-compose.yml`. Two of the three
hosts touched in this fleet's migration used a non-standard SSH port
(**79**, not 22) — check `~/.ssh/config` or ask rather than assuming 22.
When compose project context is available (you're `cd`'d into
`/opt/mailcow-dockerized`), prefer:
```bash
cd /opt/mailcow-dockerized
docker compose exec -T <service>-mailcow <command>
```
Otherwise target the container directly by its confirmed full name:
```bash
docker exec <confirmed-full-container-name> <command>
```
## Diagnostic sequence
Run external checks first — no server access needed, and they establish
whether the problem is DNS/reputation (affects one domain) vs. shared
infrastructure (affects the whole instance). Internal checks require SSH
into the mailcow host.
### 1. External: DNS records (per affected domain)
| Record | Lookup | What to verify |
|---|---|---|
| **MX** | `dig MX <domain> +short` | Points to correct hostname; priority reasonable; no stale entries |
| **A / AAAA** | `dig A <hostname>` + `dig AAAA <hostname>` | Resolves to expected IP; if AAAA exists, IPv6 must be fully configured |
| **SPF** | `dig TXT <domain> +short` | Has `v=spf1`; includes server IP or MX; ends `-all` or `~all`; exactly ONE SPF record (multiple = error) |
| **DKIM** | `dig TXT <selector>._domainkey.<domain>` | Valid public key; **get the real selector from mailcow itself** (see below) rather than guessing `dkim`/`default`/`mail` |
| **DMARC** | `dig TXT _dmarc.<domain>` | `v=DMARC1`; policy `quarantine` or `reject` for production; has `rua=` for reporting |
| **rDNS (PTR)** | `dig -x <ip> +short` | Matches mail hostname exactly |
| **Forward-confirmed rDNS** | PTR → A lookup | PTR hostname must resolve back to the original IP |
| **MTA-STS** | `dig TXT _mta-sts.<domain>` + `curl https://mta-sts.<domain>/.well-known/mta-sts.txt` | TXT has `v=STSv1; id=...`; policy file accessible over HTTPS |
| **TLSRPT** | `dig TXT _smtp._tls.<domain>` | Has `v=TLSRPTv1; rua=mailto:...` |
| **DANE/TLSA** | `dig TLSA _25._tcp.<hostname>` | If DNSSEC enabled, TLSA record should exist (usage 3, selector 1, type 1 recommended) |
| **BIMI** | `dig TXT default._bimi.<domain>` | Optional; requires DMARC p=reject/quarantine with pct=100 |
| **Autodiscover** | `dig A autodiscover.<domain>` + `dig A autoconfig.<domain>` | Records exist for mail client auto-configuration |
| **SRV** | `dig SRV _imaps._tcp.<domain>`, `_submission._tcp.<domain>`, etc. | Service records for IMAP, SMTP, CalDAV, CardDAV |
Also check for stale records: old provider DKIM CNAMEs, conflicting SPF
records, orphaned MX entries from a domain's pre-migration mail provider —
common when onboarding a domain onto a consolidated instance.
**SPF deep validation** — these limits are almost never checked manually
and cause silent failures:
- **DNS lookup count (max 10)**: count all `include`/`a`/`mx`/`redirect`/
`exists` mechanisms recursively across the SPF chain. Exceeding 10 =
PermError = SPF fails outright.
- **Void lookup limit (max 2)**: DNS queries returning NXDOMAIN or empty
count as void lookups; more than 2 = PermError. Often caused by stale
`include:` references to a decommissioned prior mail provider.
- Flag: `+all`/`?all` (too permissive), deprecated `ptr` mechanism,
multiple SPF TXT records on one domain.
**DKIM validation**: key length (1024-bit = WARNING, should rotate to
2048-bit; 2048-bit = OK), `t=y` (testing mode) still set in production,
selector matches what mailcow actually generated for that domain (see
below — don't assume a selector name).
### 2. External: TLS certificates
```bash
# HTTPS (web UI / autodiscover)
echo | openssl s_client -connect <ip>:443 -servername <hostname> 2>/dev/null | \
openssl x509 -noout -subject -issuer -dates -ext subjectAltName
# IMAPS / SMTPS / SMTP STARTTLS / Submission STARTTLS
echo | openssl s_client -connect <ip>:993 -servername <hostname> 2>/dev/null | openssl x509 -noout -subject -issuer -dates
echo | openssl s_client -connect <ip>:465 -servername <hostname> 2>/dev/null | openssl x509 -noout -subject -issuer -dates
echo | openssl s_client -connect <ip>:25 -starttls smtp -servername <hostname> 2>/dev/null | openssl x509 -noout -subject -issuer -dates
echo | openssl s_client -connect <ip>:587 -starttls smtp -servername <hostname> 2>/dev/null | openssl x509 -noout -subject -issuer -dates
# Protocol version — TLS 1.0/1.1 should NOT be accepted
openssl s_client -connect <ip>:25 -starttls smtp -tls1 2>&1 | grep "Protocol"
openssl s_client -connect <ip>:25 -starttls smtp -tls1_1 2>&1 | grep "Protocol"
```
Since a single mailcow instance serves many domains behind one cert (mailcow
issues one Let's Encrypt cert covering `MAILCOW_HOSTNAME` plus every active
mail domain as SANs), a cert problem here is almost always **instance-wide**,
not domain-specific — treat expiry/self-signed findings as higher urgency
and check the ACME container (below) rather than looking for a per-domain
cause.
### 3. External: port connectivity & SMTP banner
| Port | Protocol | Purpose |
|---|---|---|
| 25 | SMTP | Incoming mail |
| 80 / 443 | HTTP/HTTPS | Web UI, ACME challenges, autodiscover |
| 110 / 995 | POP3 / POP3S | Legacy retrieval |
| 143 / 993 | IMAP / IMAPS | Mail retrieval |
| 465 | SMTPS | Submission (implicit TLS) |
| 587 | Submission | Submission (STARTTLS) |
| 4190 | Sieve | Mail filtering |
```bash
nc -z -w5 <ip> <port> && echo open || echo closed
echo "EHLO test" | nc -w5 <ip> 25 # banner hostname, STARTTLS advertised, 250 codes
```
**HELO hostname consistency**: the SMTP banner hostname should match the
PTR record, which should match the A record — this three-way match
(forward-confirmed reverse DNS) is critical for deliverability and is
instance-wide (all domains share the same MX/banner hostname).
### 4. External: IP/domain reputation (DNSBLs)
```bash
IP="<server-ip>"
REV=$(echo $IP | awk -F. '{print $4"."$3"."$2"."$1}')
for bl in zen.spamhaus.org b.barracudacentral.org bl.spamcop.net dnsbl.sorbs.net; do
result=$(dig +short ${REV}.${bl} 2>/dev/null)
[ -z "$result" ] && echo "$bl: CLEAN" || echo "$bl: LISTED ($result)"
done
for bl in dbl.spamhaus.org multi.surbl.org multi.uribl.com; do
result=$(dig +short <domain>.${bl} 2>/dev/null)
[ -z "$result" ] && echo "$bl: CLEAN" || echo "$bl: LISTED ($result)"
done
```
Since the IP/hostname is shared across every domain on the instance, an
IP-level DNSBL listing affects **all tenants at once** — this is exactly
the shared-resource case from the multi-tenant note above: find which
domain's outbound traffic actually got the IP listed (queue/log
investigation, section 6) rather than treating every affected domain as
independently broken.
### 5. Do NOT run active open-relay/spam tests without explicit authorization
The source material this skill is adapted from includes a live open-relay
probe (connect from an external host, attempt `RCPT TO` an outside address
without auth, confirm `554`/`550` reject). **Keep the caution, not the
habit of running it casually**: an unauthorized relay/spam-injection test
against a shared multi-tenant mail instance can itself look like abuse
traffic to the very DNSBLs being checked, and risks a false-positive
listing that then affects every other domain on the instance. Only run an
active relay test with the instance owner's explicit go-ahead, and prefer
reading `smtpd_relay_restrictions` from the live Postfix config (section 6)
to confirm `reject_unauth_destination` is present — that's sufficient
evidence in almost every case without an active probe.
### 6. Internal: mailcow container/service checks (requires SSH)
```bash
cd /opt/mailcow-dockerized # confirm this path on the actual host first
# Key config values — confirm names against the live mailcow.conf, don't
# assume every key below still matches every mailcow version
grep -E "^(MAILCOW_HOSTNAME|SKIP_LETS_ENCRYPT|ENABLE_IPV6|ACL_ANYONE|HTTP_PORT|HTTPS_PORT)" mailcow.conf
DBUSER=$(grep ^DBUSER mailcow.conf | cut -d= -f2)
DBPASS=$(grep ^DBPASS mailcow.conf | cut -d= -f2)
REDISPASS=$(grep ^REDISPASS mailcow.conf | cut -d= -f2)
# Domains on this instance — establishes the actual tenant list
docker compose exec -T mysql-mailcow mysql -u"$DBUSER" -p"$DBPASS" mailcow \
-e "SELECT domain,active,backupmx FROM domain;"
# One domain's mailboxes/aliases — scope to the reported domain, not SELECT *
docker compose exec -T mysql-mailcow mysql -u"$DBUSER" -p"$DBPASS" mailcow \
-e "SELECT username,domain,active,quota FROM mailbox WHERE domain='<affected-domain>';"
docker compose exec -T mysql-mailcow mysql -u"$DBUSER" -p"$DBPASS" mailcow \
-e "SELECT address,goto,active FROM alias WHERE domain='<affected-domain>' AND address NOT LIKE '@%';"
# DKIM selectors/keys actually in use (Redis) — get the real selector,
# don't guess it for the DNS DKIM lookup in section 1
docker compose exec -T redis-mailcow redis-cli -a "$REDISPASS" HGETALL DKIM_SELECTORS
docker compose exec -T redis-mailcow redis-cli -a "$REDISPASS" HKEYS DKIM_PRIV_KEYS
# Container health — check the whole stack, a customer-facing symptom can
# trace to an unrelated dependency container being down
docker compose ps --format "table {{.Name}}\t{{.Status}}"
# Postfix config highlights, incl. open-relay protection
docker compose exec -T postfix-mailcow postconf -n | \
grep -E "(myhostname|relayhost|smtpd_tls|smtpd_relay_restrictions)"
# Mail queue — inspect before assuming instance-wide backlog; grep for the
# specific domain to see whose mail is actually stuck
docker compose exec -T postfix-mailcow postqueue -p
docker compose exec -T postfix-mailcow postqueue -p | grep -c '^[A-F0-9]' # rough depth
# Rspamd status
docker compose exec -T rspamd-mailcow rspamc stat 2>/dev/null | head -20
# Dovecot — mailbox/log inspection for one user
docker compose exec -T dovecot-mailcow doveconf -n | grep -E "(ssl|auth_mechanisms|mail_location)"
docker compose logs dovecot-mailcow --tail 100 | grep '<affected-user>@<affected-domain>'
```
Internal checks to verify:
- All containers/services running and healthy (`docker compose ps`)
- DKIM private keys exist and the DNS public key selector actually matches
what's in Redis for that domain (don't guess the selector)
- Let's Encrypt working (not a self-signed fallback — see certificates below)
- Postfix `myhostname` matches DNS and rDNS
- `smtpd_relay_restrictions` contains `reject_unauth_destination` (open
relay protection)
- Mail queue not backed up (>100 = investigate, >1000 = problem) — and
identify whose mail before declaring it "the instance's" problem
- Rspamd running and processing (not bypassed)
### 7. Certificate renewal — mailcow's own ACME container
mailcow runs Let's Encrypt itself through a dedicated container,
**`acme-mailcow`** (service name in `docker-compose.yml`, so the live
container is `<project-prefix>-acme-mailcow-1` following the same naming
pattern as every other service). Don't invent a separate certbot/ACME
mechanism — use mailcow's own:
```bash
docker compose logs acme-mailcow --tail 50 # renewal attempts/errors
docker compose restart acme-mailcow # nudge a retry if genuinely stuck
```
Relevant `mailcow.conf` flags: `SKIP_LETS_ENCRYPT` (if set to `y`, mailcow
is not managing certs at all — an external reverse proxy/cert is expected
instead, which changes where to look for the real cause of a cert
problem) and `ADDITIONAL_SAN`. Since the cert covers every active domain on
the instance as a SAN, a domain that was just added/re-enabled may need an
ACME container restart before its SAN is picked up — **verify against the
live `mailcow.conf` and current mailcow docs/changelog for the exact
mechanism on the version actually deployed**, this area has changed across
mailcow releases and is not safe to assert further without checking the
live instance.
## Backup, restore, and update — use mailcow's own official tooling
**Never invent a backup mechanism.** mailcow-dockerized ships its own
sanctioned script at `helper-scripts/backup_and_restore.sh`, confirmed
directly against the upstream script:
```bash
cd /opt/mailcow-dockerized
# Backup one or more components (repeat/select as needed — not always "all")
./helper-scripts/backup_and_restore.sh backup {vmail|crypt|redis|rspamd|postfix|mysql|all}
# Restore
./helper-scripts/backup_and_restore.sh restore {vmail|crypt|redis|rspamd|postfix|mysql|all}
```
The script prompts for (or reads from `$MAILCOW_BACKUP_LOCATION`) an
absolute backup directory path, and supports `--delete-days N` for backup
retention pruning. `THREADS=N` can be set in the environment to parallelize.
Because this is a shared multi-tenant instance, `mysql`/`vmail`/`crypt`
backups are inherently **all-domains-at-once** — there is no
official per-domain backup mode. A restore of `mysql` or `vmail` is
instance-wide and reversible only from a prior full backup; treat it as
R8-class (destructive, needs explicit confirmation) even when the
motivating complaint was a single domain, since the restore itself affects
every tenant.
**Updates**: run the official `./update.sh` from inside
`/opt/mailcow-dockerized` (must be run from that directory — it checks for
`mailcow.conf` in the cwd first). Do not update via a raw `docker compose
pull && docker compose up -d` — `update.sh` handles config migration,
`_modules` updates, and version-specific migration steps that a bare pull
skips.
```bash
cd /opt/mailcow-dockerized
./update.sh # interactive
./update.sh --dev # skip _modules update, developer mode — don't use in production without knowing why
```
Take a backup (above) before any update, same multi-tenant caution: an
update problem is instance-wide, not scoped to one domain.
## Google/Yahoo/Microsoft bulk-sender compliance (2024-2025 requirements)
These are enforced with permanent rejections. Check per-domain, since
compliance posture (DMARC policy, sending volume) can legitimately differ
between tenants on the same instance.
**All senders:**
- [ ] SPF or DKIM set up (at minimum)
- [ ] Valid PTR (forward-confirmed reverse DNS) — instance-wide, shared
- [ ] TLS for mail transmission
- [ ] RFC 5322 compliant message formatting
- [ ] Spam rate < 0.3% (Google Postmaster Tools, per sending domain)
**Bulk senders (>5,000 emails/day) additionally:**
- [ ] Both SPF AND DKIM
- [ ] DMARC with at least `p=none` and alignment pass
- [ ] One-click unsubscribe headers (`List-Unsubscribe` +
`List-Unsubscribe-Post`)
- [ ] ARC headers when forwarding
- [ ] From: domain aligned with SPF or DKIM domain
**Microsoft Outlook (May 2025+)**: SPF, DKIM, and DMARC all required for
>5,000 emails/day to Outlook/Hotmail/Live.
These external-provider policies change; treat the checklist above as a
snapshot and verify against current Google/Microsoft postmaster
documentation before treating an item as authoritative, especially for a
compliance-critical audit.
## Report format
```markdown
## <domain> Mailcow Diagnostic Report (instance: <host>)
### Instance overview
| Item | Value |
|---|---|
| Mailcow host | ... |
| Install path | /opt/mailcow-dockerized |
| MAILCOW_HOSTNAME | ... |
| Total domains on instance | N (multi-tenant — see scope note) |
| Container health | X/Y running |
### Scope of this report
Affected domain(s): ... | Instance-wide finding: yes/no
### DNS Records (this domain)
| Record | Value | Status |
|---|---|---|
### SPF / DKIM Analysis
...
### TLS Certificates (instance-wide unless noted)
| Protocol | Port | Issuer | Expires | TLS Version | Status |
|---|---|---|---|---|---|
### Blacklist Status
| Blacklist | Status |
|---|---|
### Mail Queue (scoped to affected domain where possible)
...
### Issues Found
1. **CRITICAL** — blocks delivery/rejection
2. **WARNING** — degrades deliverability/security
3. **INFO** — best-practice
### Recommended Actions
(numbered, priority order, explicit about blast radius — "affects only
<domain>" vs "affects all N domains on this instance")
```
## Severity guidelines
**CRITICAL**: MX missing/wrong; SPF/DKIM/DMARC completely absent; SPF
exceeds 10 DNS lookups; TLS cert expired/invalid on mail ports; port 25
unreachable; open relay detected; IP listed on Spamhaus ZEN (instance-wide
impact); IPv6 AAAA exists but no IPv6 PTR; HELO hostname doesn't match rDNS.
**WARNING**: rDNS doesn't match mail hostname; SPF `~all` with DMARC
`p=reject` (inconsistent); DMARC still `p=none` in production; DKIM
1024-bit key; self-signed cert on any port; MTA-STS missing; stale DNS
records from a domain's pre-migration provider; ACME container failing;
mail queue >100; TLS 1.0/1.1 still accepted; non-compliant with bulk sender
requirements.
**INFO**: TLSRPT not configured; BIMI not set up; DANE/TLSA not configured
(needs DNSSEC); ARC not configured for forwarding; SRV autodiscover records
missing; POP3 open but unused; no evident DKIM key rotation; void lookup
count close to the SPF limit.