SMTP Server Hardening: A Practical Checklist for Securing Your Mail Server

A mail server connected to the internet is a high-value target. It handles authentication credentials, sensitive business communications, and significant sending volume tied to your domain's reputation. Misconfigured mail servers get exploited — as open relays, as sources of spam, as starting points for phishing attacks using your domain. This checklist covers the practical steps to harden an SMTP server against these outcomes.

Close the Open Relay First

An open relay is a mail server that will forward email for any sender to any destination without authentication. In the early internet this was normal. Today it gets you blocklisted within hours of being discovered by spam scanners that continuously probe for open relays.

On Postfix, verify your relay restrictions explicitly reject unauthenticated relay:

smtpd_relay_restrictions =
    permit_mynetworks,
    permit_sasl_authenticated,
    reject_unauth_destination

After applying the configuration, test it by telnet-connecting to port 25 and attempting to relay a message between two external addresses without authenticating. You should receive a relay access denied response. If you do not, you have an open relay and this is the first thing to fix.

Separate Ports by Function

Port 25 is for server-to-server delivery. Port 587 (and port 465 for implicit TLS) is for authenticated client submission. Keep them separate in your configuration:

• Port 25: Accept inbound delivery from other mail servers. Do not require SASL authentication — legitimate sending servers do not authenticate with SASL. Apply IP reputation checks and content filtering.
• Port 587: Require SASL authentication for all mail. This is what email clients and internal applications use.
• Port 465: Same policy as 587 — require authentication, use implicit TLS.

Never allow unauthenticated submission on port 587.

Enforce TLS and Disable Old Protocols

Configure your server to use TLS for both inbound and outbound connections wherever possible. Disable old protocol versions that are no longer considered secure:

smtp_tls_security_level = may
smtp_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_security_level = may
smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

Use a valid CA-issued TLS certificate for your mail server hostname — not a self-signed certificate. Self-signed certificates cause TLS failures with strict receiving servers and warnings in email client connections. Let's Encrypt provides free certificates with automated renewal. Modern mail servers should communicate using TLS 1.2 or TLS 1.3 only.

Configure Authentication Records in DNS

Hardening is not limited to server configuration — your DNS authentication stack matters too:

• SPF: Publish an SPF record that explicitly lists your sending IPs and ends with -all to hard-reject anything not on your list.
• DKIM: Sign all outgoing messages with a 2048-bit RSA key. Keep the private key file permissions restricted to the mail server process only.
• DMARC: Start with p=none and advance to p=reject once reports confirm all legitimate mail is authenticating correctly.

These records, combined, mean that even if someone uses your domain in a From address, receiving servers can verify the mail did not originate from your authorized infrastructure. Full DNS and authentication setup is covered in the MailDog DNS security overview.

Rate Limiting and Connection Controls

Brute force attacks against submission ports and spam injection attempts are common. Configure rate limiting to reduce exposure:

smtpd_client_connection_count_limit = 10
smtpd_client_connection_rate_limit = 30
smtpd_client_message_rate_limit = 100

Also deploy fail2ban or an equivalent tool to automatically block IPs that generate repeated authentication failures. A single IP that fails authentication multiple times in a short window should not be allowed to continue trying indefinitely.

Restrict Administrative Interfaces

If your mail server has a web-based admin panel, restrict access to it:

• Limit access to specific trusted IP addresses via firewall rules or a VPN
• Require multi-factor authentication for admin logins
• Keep admin panels off standard ports and not exposed through your public MX records

Logging and Monitoring

A hardened server you are not monitoring is still a liability. At minimum, log all SMTP connections, authentication attempts, and delivery outcomes. Alert on unusual patterns: sudden spikes in outbound volume, authentication failures from unknown IPs, or connections to destinations you have never sent to before. Monitor your sending IP against major blocklists daily.

A Hardening Checklist

• Open relay test passes — server rejects unauthenticated relay
• Port 587 and 465 require SASL authentication
• TLS enabled on all connections with a valid CA certificate
• Old TLS versions (1.0 and 1.1) and weak cipher suites disabled
• SPF, DKIM, and DMARC all correctly configured
• Connection rate limiting configured on submission ports
• Fail2ban or equivalent protecting authentication endpoints
• Admin interfaces restricted to trusted IPs
• Logging active and behavioral alerts configured
• Blocklist monitoring in place for your sending IPs

Teams that would rather focus on their core work than ongoing server maintenance often move to managed infrastructure like MailDog's SMTP relay, where the security layer is maintained at the platform level. The MailDog documentation provides more on what is included. For more on securing your email infrastructure, visit the MailDog blog or explore available plans.