SMTP Relay Security Checklist: 10 Things Every Sender Should Lock Down

An SMTP relay is one of the most abused components in email infrastructure. When properly configured it's an efficient, reliable message routing service. When misconfigured or left unsecured, it becomes an open invitation for spammers, a direct path to blacklisting, and a significant legal liability. SMTP relay security isn't a set-it-once task — it's an ongoing operational discipline, and the checklist below covers the configurations that matter most.

1. Close Your Open Relay Immediately

An open relay accepts and forwards email from any sender to any destination without authentication. This was how early email worked, and it's completely unacceptable today. If your SMTP server will relay mail for any host that connects to it, you're running an open relay — and it will be found, exploited, and blacklisted faster than you can imagine.

Test your server right now: several free tools will attempt to relay a test message through your server. If the test succeeds, your relay is open. Locking down the open relay is the first priority — everything else on this list is secondary.

2. Enforce SMTP Authentication

Every legitimate client sending through your SMTP relay should authenticate with a username and password, a client certificate, or a trusted IP allowlist. Never allow unauthenticated relaying from anything other than explicitly trusted internal systems.

For outbound SMTP relay, use SASL (Simple Authentication and Security Layer) authentication. Require it for any connection that isn't from a known, static internal IP address.

3. Require Encrypted Connections

SMTP relay connections should always use transport encryption — either STARTTLS (which upgrades a plain connection to TLS) or a dedicated TLS port. Accepting email credentials over unencrypted connections is not a configuration choice; it's a vulnerability.

Configure your relay to:

• Advertise STARTTLS support on port 587
• Require TLS for authenticated client connections — never allow AUTH over plain connections
• Use a valid, up-to-date TLS certificate from a trusted certificate authority

4. Restrict Relay by IP Where Possible

If your sending applications have static IP addresses, allowlist them explicitly in your relay configuration and reject connections from any source not on the allowlist. IP allowlisting adds a layer of defense even when authentication is also in place — a compromised credential can't be used from an unexpected IP.

5. Set Outbound Rate Limits

Rate limiting on outbound SMTP relay serves two purposes: it prevents a compromised account from sending unlimited spam before you notice, and it smooths out sending volume in ways that receiving ISPs prefer. Configure per-account and per-domain sending limits appropriate to your expected volume, and set alerts for accounts approaching those limits unexpectedly.

6. Implement SPF, DKIM, and DMARC on Every Sending Domain

SMTP relay security isn't just about the relay itself — it includes the authentication posture of every domain you relay mail for. Every domain using your relay should have a properly configured SPF record listing your relay's IP, a DKIM key signed by your relay, and a DMARC policy set to at least p=none with reporting enabled.

Relaying mail for domains without DMARC reporting means you have no visibility into whether someone is spoofing those domains through other paths.

7. Log Everything

SMTP relay logs are your primary forensic and diagnostic resource. Log every connection, every authentication attempt (successful and failed), every message accepted, and every rejection with reason codes. Retain logs for at least 90 days. When a blacklisting or abuse complaint occurs — and if you run a relay long enough, one eventually will — logs are how you reconstruct what happened and prove what you didn't send.

8. Set Up Bounce and Complaint Handling

Unhandled bounce and complaint traffic is a common source of reputation damage. Configure your relay to process bounce messages (the NDRs that come back when messages fail) and complaints (FBL data from ISPs) and suppress those addresses from future sending. Relaying indefinitely to known-bad addresses inflates bounce rates and signals poor list hygiene to receiving systems.

9. Monitor Blocklist Status Continuously

SMTP relay security includes knowing your current standing with major blocklist operators. Set up automated monitoring for your sending IP addresses and domains against the major blocklists — Spamhaus, Barracuda, SURBL at minimum. Automated alerts give you a chance to investigate and remediate before a listing compounds into a full deliverability crisis.

10. Patch and Update Your MTA Regularly

Mail transfer agents — Postfix, Exim, Sendmail, and commercial alternatives — receive regular security updates. Unpatched MTAs have known vulnerabilities that are actively exploited in the wild. Establish a patching schedule for your relay infrastructure and test updates in a staging environment before production deployment.

When a Managed Relay Is the Better Answer

Running your own SMTP relay and maintaining all of the above is a real operational commitment. For teams that don't have dedicated infrastructure staff, a managed SMTP relay service handles authentication, TLS, rate limiting, bounce processing, and reputation monitoring as part of the service itself.

MailDog's SMTP relay is built with all of these security controls in place — authenticated connections, enforced TLS, bounce and complaint processing, and real-time delivery monitoring. You get the deliverability of a hardened relay without the operational overhead of maintaining one yourself. Review setup requirements in the MailDog documentation, or contact the team if you're migrating from a self-managed relay setup.

Related reading: the post on SMTP ports explained covers port selection as part of a secure relay configuration. The guide on SMTP error codes helps diagnose the issues you'll encounter when tightening relay restrictions. And if you're concerned about your domain's authentication posture, start with SPF's 10-lookup limit to make sure your SPF record is valid before locking down the relay.