MTA-STS Explained: How to Enforce TLS for Email Delivered to Your Domain

TLS encryption for email in transit is supposed to be standard, but there's a problem with how it's typically implemented: it's opportunistic. SMTP's STARTTLS command asks the receiving server if it supports encryption — and if the answer is no, or if an attacker intercepts the conversation and says no on the server's behalf, the sending server falls back to transmitting the message unencrypted. MTA-STS is the mechanism that closes this gap.

The Problem With Opportunistic TLS

STARTTLS works like this: the sending mail server connects, asks "do you support TLS?", and if yes, upgrades to an encrypted connection before transmitting the message. It's better than nothing, but it has a critical weakness. A network attacker positioned between the two servers can intercept that initial exchange, strip out the TLS advertisement, and cause the sending server to believe encryption isn't available — so the message goes out in plaintext. This is called a STARTTLS downgrade attack.

It's not theoretical. It's been demonstrated in real network analysis and used in targeted attacks against email in transit. STARTTLS's limitations have been documented for years, and MTA-STS is the widely-supported, practical solution that doesn't require DNSSEC.

What MTA-STS Does

MTA-STS (Mail Transfer Agent Strict Transport Security) lets a domain publish a policy telling sending mail servers: "Always use TLS when delivering to this domain, and validate the certificate. If you can't establish a valid TLS connection, don't deliver at all — hold and retry or bounce." It transforms opportunistic TLS into enforced TLS.

The policy lives in two places:

1. A DNS TXT record at _mta-sts.yourdomain.com that signals the policy exists and carries a version ID
2. A policy file served over HTTPS at https://mta-sts.yourdomain.com/.well-known/mta-sts.txt

Sending mail servers that support MTA-STS fetch both. If the DNS record and policy file are present and consistent, the sender applies the policy for all future mail to your domain and caches it for the duration you specify.

The Policy File Format

The MTA-STS policy file is a plain text document served with Content-Type: text/plain. A typical example:

version: STSv1
mode: enforce
mx: mail.yourdomain.com
mx: mail2.yourdomain.com
max_age: 604800

The mode field controls enforcement:

• none: Policy is published but not enforced — useful for signaling intent without impact
• testing: TLS failures are reported but delivery still proceeds — lets you see issues without blocking mail
• enforce: Senders must deliver over valid TLS; if they can't, they must not deliver

Always start with testing mode and collect data before switching to enforce. Moving directly to enforce on a domain with certificate issues or misconfigured MX records can cause real mail to bounce.

TLS-RPT: Visibility Into What's Happening

MTA-STS is typically deployed alongside TLS-RPT (TLS Reporting), which tells sending servers where to send reports about TLS connection failures. This is your visibility mechanism — even in testing mode, you won't know about TLS negotiation failures unless you're receiving TLS-RPT reports.

The DNS record looks like this:

_smtp._tls.yourdomain.com IN TXT "v=TLSRPTv1;rua=mailto:tlsrpt@yourdomain.com"

Reports arrive in JSON format and tell you which sending servers attempted delivery, whether TLS succeeded, and if not, exactly why it failed. The experience is similar to reading DMARC aggregate reports — technical but highly actionable once you know what to look for.

Deploying MTA-STS Step by Step

1. Confirm your MX servers support TLS and have valid certificates with names matching your MX hostnames
2. Create the mta-sts subdomain and configure HTTPS with a valid certificate
3. Publish the policy file at https://mta-sts.yourdomain.com/.well-known/mta-sts.txt in testing mode
4. Add the DNS TXT record at _mta-sts.yourdomain.com with a unique id value
5. Add the TLS-RPT DNS record pointing to your reporting address
6. Monitor TLS-RPT reports for 2–4 weeks to identify any TLS failures
7. Resolve any certificate or MX configuration issues surfaced by reports
8. Switch the policy mode to enforce and update the id value in DNS

The id field in the DNS record must change any time you update the policy file. Sending servers cache the policy by its ID — if the ID doesn't change, cached senders won't know to fetch the updated policy file.

MTA-STS vs DANE

DANE (DNS-Based Authentication of Named Entities) solves a similar problem using DNSSEC and TLSA records to bind certificate expectations directly to DNS names. DANE provides strong guarantees but requires DNSSEC to be deployed on your domain, which adds significant complexity. MTA-STS doesn't require DNSSEC, making it more broadly deployable for organizations that haven't implemented DNSSEC yet.

The two aren't mutually exclusive. If DNSSEC is already in place, adding DANE provides an additional layer. If it isn't, MTA-STS is the practical path to enforced TLS for inbound delivery. Review MailDog's DNS security tooling for configuration support, check the documentation for MX and TLS certificate guidance, and see the PTR record guide to ensure your mail server hostnames are correctly configured before enabling enforce mode.