SMTP Server Hardening: A Practical Security Checklist for Mail Administrators

Why SMTP Servers Are High-Value Targets

An exposed SMTP server is attractive to attackers for one clear reason: it sends email. Compromised mail servers are used to relay spam at scale, deliver phishing campaigns, and spread malware to recipient lists that trust the sending domain. The moment your server starts forwarding someone else's spam, you'll face IP blocklisting, potential hosting provider account suspension, and reputational damage that can take weeks to undo.

Hardening an SMTP server is methodical work, not dramatic work. It's a series of specific configuration decisions. But skipping any of them is how otherwise competent infrastructure setups end up as someone else's spam relay within days of going live — and it happens far more often than most people expect.

Eliminate Open Relay First

An open relay is an SMTP server that accepts and forwards mail for any sender to any recipient, without authentication. Open relays were common in the early internet; today they're exploited within hours of discovery. Spam bots continuously scan the internet for them — yours will be found.

In Postfix, relay access is controlled by mynetworks, relay_domains, and smtpd_relay_restrictions. In Exim, relay_to_domains and relay_from_hosts control the same. In any MTA, the principle is identical: only accept relay requests from authenticated users or explicitly defined trusted internal IP ranges. Never relay for the open internet without authentication.

Before putting a server into production, verify it with an open relay test tool. MXToolbox offers one at no cost. If it shows your server as an open relay, don't move on until that's fixed — everything else on this list is secondary to this one.

Require SMTP Authentication for All Client Submissions

Any user or application submitting mail through your server should authenticate with credentials before being permitted to relay. Configure SMTP AUTH (SASL) and require it for all relay connections from external clients. Specific requirements:

• Disable AUTH on port 25 — that port is reserved for server-to-server SMTP delivery, not client submission. Clients should use port 587 (with STARTTLS required) or 465 (implicit TLS).
• Require TLS before accepting credentials. If a client can send AUTH in plaintext, credentials will be sent in plaintext. This is not acceptable.
• Issue separate credentials to each application and user. Shared credentials make it impossible to determine which application is responsible for a problem, and a single compromised credential affects everyone sharing it.

TLS Configuration: Don't Leave Weak Protocols Enabled

Your SMTP server should enforce TLS for client connections and opportunistically encrypt server-to-server connections. Specific steps:

• Install a valid TLS certificate from a trusted CA. Let's Encrypt provides free certificates and automated renewal — there's no valid reason to run self-signed certificates on a public-facing mail server.
• Disable SSLv2, SSLv3, and TLS 1.0. These have known vulnerabilities and are no longer acceptable in production deployments.
• Configure strong cipher suites. The Mozilla SSL Configuration Generator (modern profile) provides sensible defaults for common mail servers.
• Enable STARTTLS for outbound connections to other mail servers.
• Consider implementing MTA-STS for your domain to enforce TLS for inbound delivery. The MTA-STS guide explains what the policy does and how to publish it.

Rate Limiting and Connection Controls

Without rate limiting, a single compromised credential or misbehaving application can use your server to send thousands of messages per minute before anyone notices. Implement controls at multiple levels:

• Connection rate limits: Restrict the number of simultaneous connections and connection attempts per minute from any single IP address. In Postfix, the anvil service handles this. In Exim, RATE_LIMIT ACL conditions apply.
• Per-account sending limits: Set hourly or daily message quotas per authenticated account. This limits the blast radius when a single account is compromised.
• Recipient count limits per message: Restrict the number of recipients per message submission. This prevents a single submission from becoming a mass blast.

Rate limiting doesn't prevent all abuse, but it converts a catastrophic incident into a manageable one by limiting what any single bad actor can do before alerts fire.

Brute-Force Protection

SMTP AUTH on port 587 is a constant target for credential stuffing attacks. Fail2ban — or an equivalent rate-based ban tool — should watch your SMTP authentication logs and block IPs that exceed a threshold of failed attempts. A reasonable configuration bans after 5–10 failures within a short window, with a ban duration of at least 24 hours.

Complement fail2ban with firewall rules. Port 25 must be open to the internet for inbound server-to-server delivery. Ports 587 and 465 can be open for client submission, but consider restricting them by country or ASN if your user base is geographically concentrated — this dramatically reduces the attack surface at no cost to legitimate users.

Outbound Content Filtering

Even with authentication enforced, a compromised account can send spam. Outbound content filtering — via Rspamd, SpamAssassin, or a commercial equivalent — catches abusive sends before they leave your server. Configure it to score outbound messages, reject or quarantine high-scoring messages, and alert on sudden volume spikes from individual accounts. An outbound content filter is what prevents a single compromised credential from becoming a full relay abuse incident before the account owner is even aware their password was stolen.

DNS Configuration: The Authentication Foundation

A hardened SMTP server needs correct DNS to be useful. At minimum:

• A properly configured PTR (reverse DNS) record that matches your server's hostname. Without it, many receiving servers will refuse or defer your mail regardless of content quality. The PTR records guide explains how to set this up correctly.
• Valid SPF, DKIM, and DMARC records for every domain you send mail from. Authentication failures hurt deliverability and leave your domains exposed to spoofing.

For teams that want a hardened, monitored sending infrastructure without the maintenance overhead, MailDog's SMTP relay provides established IP reputation, built-in authentication, and active abuse monitoring. Compare it to your self-managed setup using the pricing page. Questions about migrating from a self-hosted setup are handled through the support team.