Intro

This is the main wiki for finding and removing outgoing spam on a Cpanel server as well as the best ways to clean up the server and delist from RBLs. If you have incoming spam, you'll want to head over to SpamFu#Inbound_Spam

Don't discredit a server if there's very few emails in the queue. You can even track down a spam problem even if there's only one or two in the queue. Remember, mail doesn't get stuck in the queue until they're getting blocked and getting bouncebacks, so if it's a new problem, it could be sending out successfully and not show up in the queue

Types of spam

Scripts

Mail that is being sent from nobody@host.domain.com or user@host.domain.com is generally spam being generated from a script. The actual folder that it's coming from varies.

Password compromises

Some spam will be sent from user@domain.com, but there can also be spoofed addresses. A lot of different email addresses that are sending to a large number of recipients is a common symptom of this

Darkmailers

These are scripts that are bypassing Exim in order to send the mail. If there's spam in the queue, then it's not a Darkmailer (though that doesn't mean there isn't a darkmailer present)

Tools of the trade

If these aren't installed on the server already, go ahead and install them. Otherwise, skip to the next section.

Spamfu

wget -O /root/bin/spamfu.sh  https://watters.ws/scripts/spamfu.sh 
chmod +x /root/bin/spamfu.sh
/root/bin/spamfu.sh

Nukespam

cd /scripts
rm -f nukespam*
wget https://layer3.example.com/scripts/nukespam
chmod 700 nukespam

PHP Spam

This has been used on live spamming servers and using the mail.log does identify the correct script being used to spam.

Please note the variables below are not enabled by default in php.ini.

If you suspect there is a PHP script sending out email (and it is still doing so) try adding these two lines:

mail.add_x_header = On
mail.log = /var/log/php_maillog

to the [mail function] section of:

/usr/local/lib/php.ini

Also make sure to create the log file manually otherwise you may get permissions errors and it won't work:

touch /var/log/php_maillog
chmod 666 /var/log/php_maillog

Or if you prefer the easy oneliner method you can use:

if [[ "$(php -v | grep -oP 'PHP 5.[^12]')" != '' ]]; then if [[ -z $(egrep '(^mail.add_x_header|^mail.log)' /usr/local/lib/php.ini) ]]; then cp -a /usr/local/lib/php.ini{,.pre_php_mail_log_addition}; perl -n -i -e 'print; print "mail.add_x_header = On\nmail.log = /var/log/php_maillog\n" if /(\[mail function\])/' /usr/local/lib/php.ini; echo -e "\nVariables Added to [mail function]:\n"; touch /var/log/php_maillog; chmod 666 /var/log/php_maillog; /etc/init.d/httpd restart 2>/dev/null; egrep --color=never '(^mail.add_x_header = On|^mail.log = /var/log/php_maillog)' /usr/local/lib/php.ini; echo -e "\nLog File Created:";ls -l /var/log | grep php_maillog | awk '{print $1"  /var/log/"$9}'; else echo -e "\nNothing Done.\nCheck /usr/local/lib/php.ini:\n"; egrep -n --color=never '(mail.add_x_header|mail.log)' /usr/local/lib/php.ini; fi; else echo -e "\nNothing Done.\nThis only works with 5.3 or higher"; fi

This will check if PHP 5.3 or higher installed and if so it checks if the mail.add_x_header or mail.log variables exists. if they do it stops and outputs what it found and the line numbers it found them on. If it does not see them (or they are commented out) it will make a backup of /usr/local/lib/php.ini, add the variables, create the log file in /var/log, chmod it, restart Apache and output what it did.

Example of failure:

Nothing Done.                                                                                                                                                                                                                                
Check /usr/local/lib/php.ini:                                                                                                                                                                                                                
                                                                                                                                                                                                                                             
601:mail.add_x_header = On                                                                                                                                                                                                                   
602:mail.log = /var/log/php_maillog

Example of success:

Variables Added to [mail function]:                                                                                                                                                                                                          
                                                                                                                                                                                                                                             
mail.add_x_header = On                                                                                                                                                                                                                       
mail.log = /var/log/php_maillog

Log File Created:
-rw-rw-rw-  /var/log/php_maillog

The first variable adds:

This checks if PHP 5.3 or higher is installed and if it is it will then check if either of the above variables are present. If they are

X-PHP-Originating-Script:

to the exim email header (the header variable should only show up when PHP does the sending). So, for example if you had a a PHP script sending from bad_script.php the header would look something like:

X-PHP-Originating-Script: 500:bad_script.php

You can actually search the queue for this header (doesn't show up in the regular exim_mainlog) by using:

exiqgrep 'X-PHP-Originating-Script'

This should give you a list of emails that have been sent using PHP.

Or, if you know the script name, you could also use:

exiqgrep 'bad_script.php'

This should give you a list of emails that have that have that script name in it.

And if you're REALLY sure that only spam emails are listed you could use:

exiqgrep -i 'bad_script.php' | xargs exim -Mrm

which filters out all the emails with bad_script.php in it, then only displays the message ids and then delete them.

The above header stuff is really useful, but when combined with the mail.log variable can be useful. This causes PHP to record whenever:

mail()

is used. A simple output would be similar to:

mail() on [/home/user/public_html/bad_script.php:11]: To: alice@domain.com -- Headers: From: eve@other.net  Reply-To: bob@next.org  Content-type: text/html; charset=iso-8859-1

If you were to compare the cwd output of Spamfu to this log you could get a pretty good idea of where the spam is coming from.

Starting off

First get an idea of how much spam you're dealing with. However, if it's taking a really long time to complete, you may want to skip this step

exim -bpc

Afterwords, you'll start the investigation by running Spamfu on the log file. Be patient and wait for the entire scan to finish.

/scripts/spamfu.sh
Choose option 1 (Check for spammers via email logs)
Choose option 1 again (Proceed with check)

Template:Notice

Getting the Spamfu results of a log file is more useful than running it on the queue because it will both tell you if it's a script or password compromise, and it'll tell you where the script is located so you're already a step ahead. Plus, if there's no mail in the queue, then you have nothing to go on.

A server that's spamming via a script can be determined via Spamfu by a large number of emails sent under the Emails sent from scripts section and Emails sent from cpanel/system accounts. Here's an example:

Select option: 1
Checking for scripts...
Emails sent from scripts:
 171640  cwd=/home/wwsubs/public_html/wp-content/themes/twentyeleven
 164119  cwd=/home/wwsubs/public_html/wp-includes/SimplePie/Cache
  59353  cwd=/usr/local/cpanel/scripts
   2730  cwd=/home/wwsubs/public_html/wp-includes/js/tinymce/plugins/inlinepopups/skins
    103  cwd=/
     38  cwd=/home/gailkim/public_html/forums
     32  cwd=/home/workedsh/public_html
     30  cwd=/home/gailkim/public_html
     25  cwd=/root
     19  cwd=/home/reachusa/public_html

Checking for cpanel/system accounts...
Emails sent from cpanel/system accounts:
 338350 wwsubs
    100 root
     59 gailkim
     33 workedsh
     19 reachusa
      2 wslive
      2 vow
      1 drtom

A server that's spamming via a password compromise can be determined via Spamfu by a large number of emails sent under the Most emails sent by authenticated userssection and/or a bunch of emails sent to the same number of recipients, regardless of sender. Here's an example:

Checking for auth users...
Most emails sent by authenticated users:
   6452 krishnapnt@emkayauto.com
    287 noreply1@emkayauto.com
    250 support@ykassociates.co.in
    237 erp@infotrack.co.in
    178 reservations@pkbengaluru.com
    164 anil@surbhiindia.com
    135 vjajoo@emkayauto.com
    129 namit@shivalic.com
    129 dm@pkbengaluru.com
    128 catchall@emkayauto.com

Checking for recipients...
Most recipients by Mail and Sender ID's:
1Wh7Gs-0002s0-Qt with 50 recipients was sent by kevin.kho@msa.hinet.net
1Wh7Gt-0002s2-CC with 50 recipients was sent by kevin.kho@msa.hinet.net
1Wh7Gt-0002s1-FS with 50 recipients was sent by kevin.kho@msa.hinet.net
1Wh7Gt-0002ry-Ig with 50 recipients was sent by kevin.kho@msa.hinet.net
1Wh7Gt-0002rz-Ig with 50 recipients was sent by kevin.kho@msa.hinet.net
1Wh7H1-0002sM-0g with 50 recipients was sent by kevin.kho@msa.hinet.net
1Wh7H1-0002sN-EV with 50 recipients was sent by kevin.kho@msa.hinet.net
1Wh7H1-0002sO-Jb with 50 recipients was sent by kevin.kho@msa.hinet.net
1Wh7H1-0002sQ-LE with 50 recipients was sent by kevin.kho@msa.hinet.net
1Wh7H1-0002sP-LE with 50 recipients was sent by kevin.kho@msa.hinet.net
1Wh7H8-0002sp-0f with 50 recipients was sent by kevin.kho@msa.hinet.net
1Wh7H8-0002sq-V1 with 50 recipients was sent by kevin.kho@msa.hinet.net
1Wh7H9-0002ss-8X with 50 recipients was sent by kevin.kho@msa.hinet.net
1Wh7H9-0002sr-8J with 50 recipients was sent by kevin.kho@msa.hinet.net
1Wh7H9-0002st-Bj with 50 recipients was sent by kevin.kho@msa.hinet.net

Drive full?

A good thing to start with is truncating eximstats - eximstats basically records what is sent, what fails, and what is delayed so that you can get this information in WHM's "Mail Delivery Report". Since we won't be needing this information to clean up the spam (that's what the /var/log/exim_mainlog is for!) and the database tends to get rather large, it's a good first step. Truncate eximstats with

mysql eximstats -e "TRUNCATE defers;TRUNCATE failures;TRUNCATE sends;TRUNCATE smtp;"; mysqlcheck -r eximstats

Inodes full?

Check to see if inodes are full on a drive with

df -ih

You should be able to run Spamfu and clean up the queue if inodes are full. However, if you can't do either of those, check out Disk_cleanup#Finding_Inode_Utilization

Malicious scripts

So you've got a script that's spamming? First thing you want to do is head over to the homedir of the account as reported by Spamfu

 171640  cwd=/home/cpuser/public_html/wp-content/themes/twentyeleven

Next, grep the access logs for posts:

grep POST /home/cpuser/access-logs/domain.com | awk '{print $6,$7}' | sort | uniq -c | sort -rn | head

You can also pipe it to a file if you prefer to review it that way. This is very useful as spam coming from an account will have to go through a POST. If there's multiple domains under one account, looking at the Spamfu results may help you track down which domain to grep out. Once you have the list of posts, look to see which file(s) are getting posted to most frequently. They can be called anything, but you'll want to specifically take a look at POSTs to scripts that reside in the folders Spamfu reported.

 11765 "POST /wp-content/themes/twentyeleven/xmptvacmz.php
 109 "POST /wp-login.php
  7 "POST /wp-content/themes/twentyeleven/content.php
  2 "POST /wp-content/themes/twentyeleven/search.php
  1 "POST /wp-content/themes/twentyeleven/comments.php
  1 "POST /wp-content/themes/twentyeleven/header.php
  1 "POST /wp-content/themes/twentyeleven/sidebar.php
  1 "POST /wp-content/themes/twentyeleven/style.php

The above grep reveals that the file "xmptvacmz.php" is likely the source of the spam. Now cd to that folder, stat the file, vim it to make sure it really does look malicious, then chmod it 000. MAKE SURE TO STAT AND NOTE THE RESULTS OF THE SCRIPT BEFORE DISABLING IT

One last thing before considering it stopped is to run a killall on PHP/httpd and restart exim. Then start apache again. This ensures that there's no active process that will continue to spam from the server. Please check for any sticky notes on the account about not restarting services first!

killall -9 php
killall -9 httpd
/etc/init.d/exim restart
/etc/init.d/httpd start

Template:Notice

So you stopped the spam? Great! However, there's still the question of how it got on the server in the first place. For that, you'll need to head over to Infected_Website. A good time to investigate how the account was compromised is while you're waiting for your spam cleanup to finish.

cwd is /tmp

Investigation is similar to searching for a malicious script. The difference being, though, is that even after disabling the script on the account, spam is still being generated from /tmp. You can even completely empty /tmp on the server and it'll still be spamming.

 304945  cwd=/tmp

In situations like these, I've found the solution to be Suspend and then Unsuspend the account through WHM. Make sure to check the "Prevent Reseller from Unsuspending" box when you suspend it, or it sometimes doesn't suspend the mail functionality of the account.

Exploited forms

Contact forms and Tell-A-Friend forms are oftentimes exploited if they lack a Captcha. In a case where a form on a site is being exploited, you might not be able to stop the spam. The access logs may or may not show multiple POSTs to a file, so you'll need to rely on the subject of the email to try and track down where it's coming from. Visiting the site and having a look or even grepping for a common part of the subject in public_html may be the only way to go.

Thankfully, Tell-A-Friend scripts are frequently separate php files in public_html. Just stat and chmod 000 the file and let the customer know what was going on so they can either disable them or add a Captcha.

Spam from Cron Jobs

There's two different types of cron spam: legit and non-legit email. If there's an error in a cron job such as referencing a file that doesn't exist or a syntax error, they'll frequently send an email either to root or (if set up this way) an email address specified in the crontab for that user. Before just removing the task from the crontab, ask the customer about it.

You can oftentimes find legit cronjob spam by a cwd of /home/user:

2310  cwd=/home/mjtouchc

If you grep out POSTs from the access logs, there will be none that appear to be malicious. However, if you take a look at one of the emails it sent out, it'd have a subject similar to:

Subject: Cron <mjtouchc@host> php /home/mjtouchc/public_html/cron.php

You'll want to take a look at the file it references to find out why it's failing. Common reasons are either a) doesn't exist or b) it's been disabled. At this point, you'll want to remove the job from the user's crontab (note the syntax of the job in case it needs to go back) and notify the customer.

Additionally, malicious scripts can set up cron jobs to send out spam, too (which was the situation above). The easiest way to determine this is to grep out the cpanel user from the cron log:

grep cpuser /var/log/cron

If it returns any suspicious-looking results, examine the script that it's running to determine if it's malicious then note the command in the ticket and commend out the cron. Do not delete the cron job until you verify with the customer whether it is legitimate or not.

Stopping password compromises

Basically you just reset the password on the account and send the customer something like this:

I changed the password on that account but please have that user scan any computers they use to access that email account for malware and viruses. After any infections have been cleaned, please change the password through cpanel to a strong password with a mixture of upper and lowercase letters, numbers, and at least one symbol with no dictionary words.

User that exists on server

You don't need to do anything special if Spamfu returns only one result for a compromised account. Move on to the cleanup section at this point. However, if you're seeing email sent to multiple recipients or email sent from a user that doesn't exist, keep following this section.

Multiple recipients

Sending to a lot of people at once is an easy way to hide a compromised account as the account won't appear to be sending gobs of email so it might be missed in Spamfu'sMost emails sent by authenticated users but will show up in the results for Most recipients by Mail and Sender ID's

Template:Notice

Running Spamfu on the queue for 30 seconds is a good thing at this point as it'll provide a few examples of emails being sent from a user that you can then vim to determine if it's spam or not. However, if an account is sending to the same number of recipients over and over, that's usually spam.

User that does NOT exist on server

So Spamfu says there's an account sending email but it doesn't even exist on the server? Don't worry, it's easy to track down!

1Wh7Gs-0002s0-Qt with 50 recipients was sent by kevin.kho@msa.hinet.net

Take the ID of the message and grep it out of the exim_mainlog:

2014-05-05 06:00:47 [11036] 1Wh7Gs-0002s0-Qt <= kevin.kho@msa.hinet.net H=23-24-119-90-static.hfc.comcastbusiness.net (User) [23.24.119.90]:51058 I=[69.167.168.100]:25 P=esmtpa A=courier_login:krishnapnt@emkayauto.com S=400990 T="Invoice Satander Bank" from <kevin.kho@msa.hinet.net> for 395169549@qq.com 397610569@qq.com 402737294@qq.com 407153699@qq.com 40977@ctci.com.tw 414466165@qq.com 416569866@qq.com 419913758@qq.com 420616@fedex.com 42388428@qq.com 444154@egat.co.th 458677472@qq.com 462306419@qq.com 46573@ctcim.com.tw 47680@ctci.com.tw 48822@ctci.com.tw 491464@fedex.com 496928@fedex.com 50088@ctci.com.tw 510902382285782@star.c10r.facebook.com 511941466@qq.com 52531@ctci.com.tw 564427184@qq.com 578321282@qq.com 578498375@qq.com 578762@cpc.com.tw 581607@cpc.com.tw 589096@fedex.com 592276@fedex.com 616556599@qq.com 620006399@qq.com 665210@metro.ax 665322@metro.ax 666@xmas-house.com.tw 669308812@qq.com 702378094@qq.com 70237894@88.com 70237894@qq.com 756413254@qq.com 774144300@qq.com 798624384@qq.com 8100@mail.ttl.com.tw 824611721@qq.com 826448@fedex.com 846590840@qq.com 868394@fedex.com 884701@metro.ax 88536779@hc360.com.cn 89933125@pchome.com.tw 921336583@163.com

What you're looking for here is the A=courier_login or A=dovecot_login depending on what mailserver they're using. This shows the actual email address that logged in. In the result above, you can see that krishnapnt@emkayauto.com is compromised. You'll want to change the password for that account and then remove the spam with:

find /var/spool/exim/input -name '*-H' | xargs grep 'auth_id' | grep $EMAILADDRESS | cut -d: -f1 | cut -d/ -f7 | cut -d- -f1-3 | xargs exim -Mrm

Replace $EMAILADDRESS with the full email address and then run it.

Template:Notice

If you have a small queue (<1000), it may be easier to manually go through the queue in "Mail Queue Manager".

1. Grep the ID of an obvious spam message
2. Change the password on the compromised account
3. Delete the spam from that account using the one-liner
4. Refresh queue to see if there's any more
5. Rinse and repeat until no more spoofed addresses remain

Here's a good example of a common spoofed address as reported by WHM:

File:Spoofedaddress.JPG

Gmail isn't hosted on the server, obviously, and it's sending to a bunch of remote address. When you grep out the Email ID From the exim_mainlog you see the actual problem:

2014-05-11 12:09:40 [31121] 1WjWJl-00085x-Mk <= loanmoreinfo@gmail.com H=8ta-229-133-103.telkomadsl.co.za [197.229.133.103]:54035 I=[67.225.216.115]:25 P=esmtpsa X=TLSv1:DHE-RSA-AES256-SHA:256 CV=no A=courier_login:admin@apritcun.com S=1046 M8S=0 T="APPLY" from <loanmoreinfo@gmail.com> for...

It shows that the password for admin@apritcun.com is compromised.

Darkmailers

Quickest way to see if there's an active one is to run:

lsof -i :25

Template:Note

COMMAND   PID     USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
exim    19958 mailnull    3u  IPv6 54506176      0t0  TCP *:smtp (LISTEN)
exim    19958 mailnull    4u  IPv4 54506177      0t0  TCP *:smtp (LISTEN)

If 'nobody,' a normal cPanel account, or a perl process shows up in this check, the server is probably actively darkmailing. There's a nice wiki available for Darkmailers here:Darkmailer. If you have any questions about Darkmailers, please feel free to consult with SecTeam or a senior technician.

Cleaning up spam

Set AUTHID to the user you want.

authid='username'
exim -bpr | grep -w $authid | cut -c 10-26 | xargs exim -Mrm | nl

Remove spam from scripts:

find /var/spool/exim/input -name '*-H' | xargs grep 'X-PHP-Originating-Script: 512:gallery.php' | cut -c 25-40 | xargs exim -Mrm

Script removal

For SuPHP servers, use the cpanel username. For DSO servers, use "nobody" even though the Auth ID may be @host.domain.com.

Password Compromise removal

Use the whole email address to remove the spam since there may be multiples of the same username across different accounts (IE info@domain.com)

Clean up bounce-backs with this command

find /var/spool/exim/input -name '*-H' | xargs grep 'ident mailnull' | cut -c 25-40 | while read id; do exim -Mrm $id; done

Password compromises

Replace $EMAILADDRESS with the full email address from your spamfu results and then run it in screen. It may take several minutes before you actually see that it's removing spam, so just be patient.

If there's only one email account that's compromised and you know which account it is, you can run both one-liners above to remove the spam and bouncebacks. However, if an email address is being spoofed, it takes a bit more work than just tossing in the Auth ID and letting it run. There can be more than one compromised account on a single server so follow the instructions at Cpanel_Spam_Procedure#User_that_does_NOT_exist_on_server to find and remove the spam.

Wrapping up

If the load on the server has been elevated for a long time due to the spam, legitimate mail may be stuck in the queue. After you've cleaned up the spam and confirmed the server isn't generating any new spam, it might be a good idea to start a queue run to deliver the rest of the email in the queue. Make sure to manually check what messages are left in the queue to make sure you didn't miss anything such as spoofed addresses or a lot of email from an account that didn't show up in spamfu.

Nukespam can help you determine if there's some other type of compromise or problem on the server at this point.

...The spam is stopped and the queue is clean. Maldet is only useful if it was a malicious script that was uploaded to the server. If a password was compromised or a form was being exploited, Maldet probably won't do anything other than use up resources. Please don't run a maldet scan on the entire /home folder as it can take hours and you want to disable potential Cmd Shells on the compromised account as soon as possible.

Delisting the server

Visit Blacklist_Removal_Options for instructions for delisting from the most common places.

Only blocked by private RBLs

This usually indicates a spoofed email address or some type of form that's getting exploited so make sure that you're checking for those specifically.

The other less-common reason for this is forwards that are set up to forward email from the server to AOL/Yahoo/Gmail/Hotmail. When the account receives spam, the spam is then forwarded on and can cause temporary or permanent blacklists. Oftentimes there's a couple specific email addresses that are particularly active whose forwards are causing a problem.

Once the problem is resolved, then follow the procedures in the previous section to get them delisted.