MTA-STS and TLS-RPT: Enforcing Encrypted Email Delivery for Your Domain

The Problem with Opportunistic Encryption

Most email sent between mail servers today is encrypted in transit using STARTTLS. When a sending server connects to a receiving server and STARTTLS is available, the connection upgrades to TLS before any message content is transmitted. This is called opportunistic encryption — it happens automatically when both parties support it, which most do.

The word "opportunistic" carries a critical limitation: if STARTTLS negotiation fails, the connection typically falls back to plain text rather than failing entirely. This design was intentional — it kept email flowing when encryption wasn't universally supported — but it creates an opening for downgrade attacks. An attacker positioned between two mail servers can strip the STARTTLS advertisement from the receiving server's greeting, causing the sender to proceed in plain text. The sender believes encryption happened. It didn't.

MTA-STS was created to close this gap. It lets you publish a policy declaring that your domain requires TLS for inbound mail delivery — and that senders must refuse delivery rather than fall back to plain text.

How MTA-STS Works

MTA-STS (Mail Transfer Agent Strict Transport Security) uses two components working together: a DNS TXT record that announces a policy exists, and a policy file hosted at a specific HTTPS URL on your domain.

The DNS record:

_mta-sts.yourdomain.com  TXT  "v=STSv1; id=20260618T000000"

The policy file, hosted at https://mta-sts.yourdomain.com/.well-known/mta-sts.txt:

version: STSv1
mode: enforce
mx: mail.yourdomain.com
max_age: 86400

The mode field controls what sending servers do when TLS validation fails:

• none: Policy exists but is not enforced. Delivery proceeds regardless. Used during initial deployment for observation only.
• testing: Delivery proceeds even if TLS fails, but failures are reported via TLS-RPT. Good for catching problems before full enforcement.
• enforce: Delivery is refused if TLS cannot be established to the listed MX host with a valid certificate. This is the target state.

When a compliant sending server discovers your policy, it caches it for the max_age duration and applies it to all subsequent delivery attempts to your domain. Senders that support MTA-STS — including all major providers — will honour the policy.

Deploying MTA-STS Safely

Moving directly to enforce mode without testing first can cause legitimate email to bounce if your MX certificates have any problems — expired certificates, hostname mismatches, or missing intermediate certificates. The safe deployment sequence takes a few weeks but prevents surprises:

1. Start with mode: none. Publish the DNS record and policy file. Enable TLS-RPT (below) to start collecting data. Let this run for 7–14 days.
2. Review TLS-RPT reports. Look for certificate validation failures, MX hostname mismatches, or STARTTLS negotiation errors. Fix any problems before continuing.
3. Switch to mode: testing. Delivery continues on failure, but you get full reporting on what would have been blocked in enforce mode. Run for another 7–14 days.
4. Switch to mode: enforce. Update the id value in your DNS record whenever the policy changes — this signals to senders that the cached policy is stale and should be refreshed.

TLS-RPT: Visibility Into Encryption Failures

TLS-RPT (TLS Reporting) is the companion standard to MTA-STS. It lets you specify a reporting address where sending servers send daily reports on TLS failures they encountered when delivering to your domain.

The DNS record:

_smtp._tls.yourdomain.com  TXT  "v=TLSRPTv1; rua=mailto:tls-reports@yourdomain.com"

Sending servers that support TLS-RPT generate daily JSON reports that include:

• Total successful and failed TLS connection attempts
• Specific failure types — certificate expired, hostname mismatch, STARTTLS not offered, policy validation failure
• Which sending servers encountered each failure type

These reports are conceptually similar to DMARC aggregate reports — both give you visibility into what's happening in production mail flows. Even before enabling MTA-STS enforcement, TLS-RPT reports can surface unexpected TLS issues with your mail infrastructure that you wouldn't otherwise know about.

Certificate Requirements

For MTA-STS to work in enforce mode, your MX hosts need valid TLS certificates from a trusted certificate authority. The certificate must cover the exact MX hostname (in the Common Name or Subject Alternative Name). Let's Encrypt certificates work perfectly for this purpose.

This is the most common reason for the staged deployment approach: mail servers running self-signed certificates or certificates with hostname mismatches would cause delivery failures in enforce mode. The testing phase surfaces these before they affect actual mail flow.

MTA-STS and the Broader Security Stack

MTA-STS builds directly on STARTTLS — it doesn't replace it, it enforces it. If you haven't yet reviewed how STARTTLS works and why it has limitations, that's the foundational context for understanding what MTA-STS adds.

MTA-STS fits alongside the rest of your email authentication stack. The DMARC setup guide and MailDog's DNS security tools cover the broader authentication picture — SPF, DKIM, and DMARC — that MTA-STS complements.

An alternative approach to the same problem is DANE (DNS-based Authentication of Named Entities), which uses DNSSEC to publish certificate fingerprints directly in DNS. DANE is technically elegant but requires DNSSEC support from your DNS provider and registrar, which isn't universally available. MTA-STS uses standard HTTPS for policy delivery and works with any domain that has a valid TLS certificate — making it the more practical starting point for most organisations today.

MTA-STS is a narrow but meaningful security control. It doesn't affect spam filtering or inbox placement — it protects the delivery path itself from downgrade attacks. For security-conscious organisations, it's increasingly standard practice.