How to Stop Email Spam Bots Using SMTP on Your VPS

The Problem

The email said that I had reached my “daily SMTP relay limit”. While this can be caused by things other than bulk email (such as server verifications and stuff, which I won’t go into here), the main cause is that a whole pile of email was sent. In my case, over 1,000 in a 24-hour period.

Finding the Source

One of the more common sources of the problem is if you have a responder setting in your default email address. So, off you go to check cPanel (or your memory). Look for the “Default Address” option in the “Mail” section.

If your default is set to produce an error message, you may be sending more emails than you thought. Every spam message coming in to the server which does not match a real account will generate an outgoing error report. In my case, this would be a lot of email, since my domains are constantly bombarded by messages to “information@domain.com”, which doesn’t exist.

This is actually a double-hit, as it not only generates an outgoing email to the spammer – thus knocking one more send off your available limit – but also informs them that the address doesn’t exist. They can update their list and save time and money. So it’s best not to auto-respond, in my opinion.

I set my default to forward the message to Spamcop instead. It costs me a send, but it does the spammer harm!

Check who’s sending

In my case, this was unlikely. So I went to check the mail queue and see what was going on. This is in WHM, rather than cPanel.

There are two ways to figure out who’s sending: check the mail queue or check the relays. They’re both in the “Email” section of WHM and are, respectively, “Mail Queue Manager” and “View Relayers”. In my case, as someone with a lot of domains on the server, I went for the relayers option first, shown below:

As you can see, the top account is obviously the one sending. The others all look OK, but there’s something bad going on with that one!

Fixing it

The first thing to do is to have a look in the domain mailboxes and see if any are stupidly big. If they are, there’s a chance that they’re the one sending the emails, especially if they’re bloated with bounce messages. You can also look at the WHM mail queue and see where the messages originate. Changing the password on a compromised account should lock out simpler spambots.

Unfortunately, most spammers use clever little apps which hit the mail servers directly, finding open relays, sending as “nobody” and other stuff. Open relays are extremely rare these days and you’re unlikely to find yourself with one unless your provider really sucks or you messed with the SMTP setup and screwed everything. You can still check by going toMXToolbox and typing in your mail server address. Their report looks like this (my server’s fine!):

So, now we know it’s not an open relay or simple compromised account problem, let’s look at the steps to lock things down.

1. Change the default email address password

First off, change your cPanel password. This will change the default email account’s password at the same time (the one that’s set to cpanel-login@domain.com).

2. Enable WHM’s restrictions

WHM allows you to restrict email sends to root, exim and mailman – the three real sources of email – so that anyone trying to work around the system gets booted out.

This setting is in two places, either of which will do the job:

• Security Center -> SMTP Restrictions (press the button)
• Server Configuration -> Tweak Settings (look in the Mail tab for the option called “Restrict outgoing SMTP to root, exim, and mailman”)

3. Set the maximum sends per hour

In the Tweak Settings options, on the Mail page, there’s also a setting for the maximum number of emails per hour, per domain. Be careful with this one, especially if you use mailing lists for subscribers. Setting it to something low should do the trick if you just want to block everything.

4. Stop “nobody” from sending email

Automated scripts often use the “nobody” user when they send emails, so you need to shut off access. The option is in the Mail tab of Tweak Settings, the same as 2 and 3 above. It’s called “Prevent “nobody” from sending mail”

5. Enable suPHP and suExec

This one’s really techie. Go to the Service (not Server) Configuration -> Configure PHP and suExec option. Switch on suExec and make sure the php handler (for your version) is suPHP.

There’s another option in lieu of switching on suExec, which is to include mod_ruid2 in your Apache configuration (it’s listed in the standard modules in the Easy Apache option). However, as the warning on the module says, it switches off loads of stuff and you shouldn’t install it unless you know how to administer it. Since you’re here on a VPS for noobs page, that’s unlikely. I certainly don’t know how to administer it, so I’m sticking with suExec!

6. Switch off Mailman

If you never use email lists or, more specifically, never use server-based Mailman lists, switch them off. You can do this in the Service (not Server) Configuration section of WHM, using the Service Manager. Unmark the Mailman box and save the new configuration.

Note that clever users will still be able to get around this basic “off” setting by using cron jobs, but it’ll close another hole that is often abused.

7. One last Tweak

One final tool for slapping spambots is a tweak (in the Server Configuration menu as before) called “Maximum percentage of failed or deferred messages a domain may send per hour”. I set this one to 20%, so that anyone who gets more than 2 bounces in 10 sent emails in an hour is shut down temporarily.

Check the results

To see if your changes have protected your VPS from the assault, you’ll need to restart some services, wait a bit, then check the email queue and relays.

First off, go to Restart Services -> Mail Server (Exim). Give the service a kick. It only takes a couple of seconds to restart.

Leave things alone for a few minutes, while the bots try to log in again, then hit the Email -> Mail Queue Manager option. Run the report and you’ll see the list of queued emails. There should only be old ones, from before the changes. You can refresh the list by clicking Mail Queue Manager on the left (or changing the times and running the report again).

If all else fails

No matter how many options you set and how many barriers you put in place, there is always the possibility that some asshole has sufficiently compromised your system to screw with everything beyond your control. If that’s the case, contact tech support.