STARTTLS and DANE Explained: How Email Is Protected in Transit (and Where It Falls Short)

When an email leaves your server and travels to its destination, what actually protects it in transit? The short answer is TLS encryption — but the way TLS is negotiated for email is more nuanced than most people realize. STARTTLS and DANE are the two mechanisms at the center of this, and understanding the difference between them explains a lot about where email security is strong and where it still has gaps.

How Email Gets Encrypted in Transit

Email delivery between servers happens over SMTP. When two servers connect, they negotiate whether to use TLS through a mechanism called STARTTLS. The process works like this: the sending server opens a connection to the receiving server, the receiving server announces its capabilities including STARTTLS if it supports it, and the sending server issues a STARTTLS command to upgrade the connection to TLS. A TLS handshake establishes an encrypted channel, and delivery proceeds over that encrypted connection.

This is called opportunistic TLS. The encryption happens when both servers support it, but it is not enforced. If the receiving server does not advertise STARTTLS, or if something strips that advertisement from the response, the sending server typically falls back to delivering in the clear.

The Limitation of Opportunistic TLS

Opportunistic TLS is vulnerable to downgrade attacks. An attacker positioned between the two mail servers can strip the STARTTLS advertisement, causing the sending server to believe TLS is not available. The connection falls back to unencrypted delivery, and the attacker can read or modify the email in transit.

This is why simply using STARTTLS is not the same as email being secure in transit. The encryption is only as reliable as the negotiation process that sets it up.

MTA-STS: Enforced TLS for Incoming Mail

MTA-STS (Mail Transfer Agent Strict Transport Security) addresses the downgrade problem. When a domain publishes an MTA-STS policy, it tells sending servers they must use TLS to deliver mail to that domain — and if TLS negotiation fails, they must not deliver at all.

MTA-STS is configured via a DNS TXT record and a policy file served over HTTPS. When a sending server checks MX records before delivery, it also checks for an MTA-STS policy. If found in enforce mode, the sending server must establish a valid TLS connection with a trusted certificate matching the MX hostname. Delivery without TLS is not permitted.

DANE: Certificate Validation Via DNS

DANE (DNS-based Authentication of Named Entities) takes a different approach. Instead of relying on the traditional certificate authority system to validate TLS certificates, DANE lets domain owners publish their expected TLS certificate information directly in DNS using TLSA records.

A TLSA record at the SMTP port for your MX host specifies the certificate or certificate hash that sending servers should expect when connecting. This creates a direct link between DNS ownership and certificate validation. If the certificate the receiving server presents does not match what is published in DNS, the connection is rejected — even if the certificate is otherwise valid from a certificate authority perspective.

DANE requires DNSSEC on your domain. Without DNSSEC, the TLSA records themselves can be spoofed, which defeats the purpose. This dependency on DNSSEC is the main reason DANE adoption has been slower than MTA-STS, which works without it.

What Each Standard Protects Against

• STARTTLS: Protects against passive eavesdropping when both sides support it. Does not protect against active downgrade attacks.
• MTA-STS: Prevents downgrade attacks and requires a valid CA-issued certificate. Does not require DNSSEC.
• DANE: Prevents downgrade attacks and man-in-the-middle attacks using fraudulent certificates. Requires DNSSEC.

The two enforced standards are complementary. A domain can implement both MTA-STS and DANE for layered protection. DANE is more technically rigorous. MTA-STS is more widely supported by current sending infrastructure.

TLS-RPT: Reporting What Is Happening

TLS Reporting works alongside both MTA-STS and DANE. It lets domain owners receive daily reports from sending servers about TLS failures when delivering to their domain — similar to how DMARC reports work for authentication. Configuring TLS-RPT gives you visibility into failed TLS negotiations you would otherwise never know about. The DNS record looks like this:

_smtp._tls.example.com. IN TXT "v=TLSRPTv1; rua=mailto:tls-reports@example.com"

What This Means for Your Setup

For most organizations, the immediate priority is confirming that STARTTLS is working correctly on both inbound and outbound connections. From there, implementing MTA-STS for your receiving infrastructure is a meaningful step toward enforced encryption that does not require DNSSEC.

For teams using managed email infrastructure like MailDog's SMTP relay, TLS configuration for outbound connections is handled at the infrastructure level. For your receiving domain, configuration guidance is available in the MailDog documentation. Understanding how transit security works helps when reviewing your full email security posture — alongside authentication standards covered in the DNS security overview. For more on email security topics, visit the MailDog blog.

Email encryption in transit is not binary. The combination of STARTTLS, MTA-STS, DANE, and TLS-RPT creates defense in depth — each layer closing a gap the previous one leaves open.