SMTP Server Hardening: Locking Down Your Mail Transfer Agent

A misconfigured SMTP server is a liability. Left insecure, it can be exploited as an open relay for spam campaigns, used to send phishing messages from your domain, or abused to cause delivery failures that follow your IP for months. The consequence isn't just getting blocklisted — it's a damaged sending reputation that takes time to repair and potential inclusion in threat intelligence databases that persist long after the original problem is resolved.

SMTP server hardening is the practice of configuring your mail transfer agent and surrounding infrastructure to eliminate known attack surfaces. Here's a practical checklist for locking down an SMTP server, whether you're running Postfix, Exim, or another MTA.

Close the Open Relay Immediately

An open relay is an SMTP server that forwards email for anyone, to anyone, without authentication. In the early internet, open relays were normal. Today they're discovered by scanners within hours of going live, listed on blocklists, and used to send spam at scale.

Ensure your server only relays email for authenticated users and for your own domains. In Postfix, this is controlled by smtpd_relay_restrictions. A safe minimal configuration looks like:

smtpd_relay_restrictions =
  permit_mynetworks,
  permit_sasl_authenticated,
  reject_unauth_destination

Test your relay status after making configuration changes using an online open relay test. If mail gets through without authentication, something in your configuration is wrong.

Require Authentication on Submission

All email submission on port 587 should require authentication. Never allow unauthenticated email submission from external sources. Use SASL authentication and require valid credentials before accepting a message for relay.

Password security for SMTP accounts also deserves attention. If your users authenticate with weak or reused passwords, an attacker who brute-forces their way in gets full relay access. Enforce strong passwords and implement rate limiting on authentication attempts to slow down automated attacks.

Enforce TLS on All Connections

All SMTP connections — both inbound and outbound — should use TLS. For inbound mail on port 25, advertise STARTTLS and accept encrypted connections. For submission on port 587, require STARTTLS; never allow credentials to be sent over an unencrypted connection. For port 465 (SMTPS), TLS is mandatory by design.

Your TLS certificate should be valid and signed by a trusted certificate authority. Using a self-signed certificate causes some receiving servers to reject connections or log warnings. Services like Let's Encrypt make valid certificates free and straightforward to automate with renewal.

For inter-server encryption, implementing MTA-STS ensures that servers sending mail to your domain know to require TLS and cannot be downgraded to a plain-text connection by an intermediary.

Disable Weak Protocols and Ciphers

Disable any protocol version or cipher suite that's known to be insecure:

• Disable SSLv2, SSLv3, TLSv1.0, and TLSv1.1 — allow only TLSv1.2 and TLSv1.3
• Disable weak cipher suites including RC4, export-grade ciphers, and NULL ciphers
• Disable the SMTP VRFY and EXPN commands if not needed — they can be used to enumerate valid addresses
• Disable plain-text authentication mechanisms (PLAIN, LOGIN) on any unencrypted connection path

Implement Rate Limiting

Rate limiting protects your server from two different problems: external actors attempting to abuse it as a relay, and compromised internal accounts that suddenly start sending large volumes of email.

Apply limits at two levels:

• Connection rate limiting: Restrict how many SMTP connections a single IP can establish per minute from the internet.
• Message rate limiting: Restrict how many messages a single authenticated account can send per hour or per day.

When an account that normally sends 50 emails a day suddenly generates 5,000, that's a signal worth acting on immediately — either the account is compromised, or a developer accidentally pushed production credentials into a test environment that's now running without restraint.

Use DNS-Based Blocklist Filtering on Port 25

Inbound connections on port 25 come from other mail servers delivering messages to your users. You can add an early filtering layer by checking the connecting IP against reputation blocklists before accepting the connection. Most MTAs support this via DNS-based blocklists (DNSBLs).

Be conservative about which blocklists you apply at the connection level — overly aggressive blocking can reject legitimate mail. Spamhaus ZEN is widely respected and has a well-maintained low false positive rate. Review any blocklist's policies and delisting procedures before applying it.

Keep MTA Software Updated

SMTP server software — Postfix, Exim, Sendmail, Dovecot, and others — receives regular security updates. Running outdated versions means running with known vulnerabilities. Subscribe to the security announcement list for your MTA and apply patches promptly. This applies to supporting software too: OpenSSL vulnerabilities affect your TLS implementation directly, and a compromised web application on the same server can expose SMTP configuration files or credentials stored on disk.

Restrict Server Administrative Access

The server running your MTA should follow standard server hardening practices:

• SSH access via key authentication only; disable password-based SSH login
• Firewall rules that allow only the ports your mail server legitimately needs (25, 587, 465, 993, 995 as appropriate)
• No root login over SSH; use a non-root account and sudo for administrative tasks
• Regular review of which users and processes can modify MTA configuration files

Monitor and Alert Continuously

Hardening is a point-in-time action; ongoing monitoring is what catches problems between hardening reviews. At minimum, alert on:

• Authentication failures above your expected daily baseline — suggests a brute force or spraying attempt
• Sudden spikes in outbound message volume from any single authenticated account
• Delivery rejections from major receiving providers — may indicate a reputation problem developing
• Appearance of your sending IPs on any major blocklist

If you're looking for a more managed approach where infrastructure security is handled for you, a hosted SMTP relay service like MailDog takes care of server hardening, TLS configuration, IP reputation management, and blocklist monitoring as part of the service. The MailDog documentation covers available security configuration options. For questions about hardening your specific setup, the support team can help, and the blog has additional resources on SMTP security and relay best practices.