I’ve been trying to get a new client on board and as part of that I’ve made a list of things that need to get fixed for that business to operate reliably.
One of the dangerous things they are doing is running Mail Server on a Mac. This is about to become an unsupported config, so there is some urgency in getting a replacement service running. And of course the server was compromised, blacklisted and I’ve just spent nearly 3 days running around with my hair on fire.
With most mail servers it’s a simple task to check the mail queues or mailboxes to figure out if the threat is internal or external. On a Mac it could be even easier if you could just check the spool folder sizes. But you can’t- Postfix is immune to admin based attacks because it runs pretty much everything as the postfix user.
You could run Roundcube on the server, log into each account via webmail and perhaps do some good there. Unfortunately Apple has removed it, and while this server did have an install, it wasn’t working. And even an update didn’t fix it. That quickly became a rabbit hole I wished I’d stayed out of.
What other options do we have?
- Built in tools. I did eventually use the command line tools but finding the correct (and working) commands was hellacious. As you probably know, Apple have a habit of adding only the bits of open source that they like, so loads of potentially useful commends just didn’t work.
- Postfixadmin – this web based admin tool can be installed on most Unix based systems but there is very little info about Mac OS X, and it’s also geared towards basic admin, so I wasn’t sure if it would give me the info I needed, and it required installing MySQL or Postgres, which was a bit of a sledgehammer approach
- WebMin. I’ll admit it- I loved WebMin- in 2001 when Mac OS X first came out it was awesome for a guy who couldn’t use a CLI to save his life. But this one was scary too- the install instructions on the manufacturers website are from 2013. Really.
When you get a problem like this you really want to know
A. Where is the issue?
B. How do I fix it?
c. How do I prevent it from happening again?
But we were quite stuck with A because we couldn’t see any of the spam emails! Acting quickly is critical in these situations, so I changed all of the passwords for all of the Users, and also looked to see if the attacker had somehow managed to set up a hidden user on the machine to continue their evil plans. None found, it was time to look at the spam.
I eventually was able to export a list of mail in the queue to a text file-
mailq > myfile.txt
Which saves it in your ~/Home folder
and from there, I got the ID from one of them- (say) 95D4B100C24ACE. From there, I issued
sudo postcat -q 95D4B100C24ACE
and was able to view the email- finally!
Sadly, there wasn’t much in the way of clues. The spammer had set the original and return address to ‘manager@company.com’ and the only good info I had from this is the originating IP which was ‘70.39.115.201’ (may you rot in hell). This IP appears to be registered to ‘Sharktech Inc’ who are a host in the USA, so I’m guessing one of their VMs was compromised, or a spammer had set up shop.
The only other place I could go to was the Logs. I browsed the logs and saw a bunch of logins from local accounts, but there were some interesting IP addresses attached to logins for one account- hr@spamvictim.com
I found 2 IP addresses, one from Indonesia and one from Singapore that looked suspicious, so I figured that was the compromised account.
Unfortunately, when I told the customer, he said that his wife (who owns the hr@… account) has been overseas with her laptop and phone- and she has been in Indonesia and Singapore!
So we are no wiser as to which account has been hacked. Back to wondering if we could get some Postfix management tools on to the server to help, however I posted in several places looking for hints and no one had suggested these tools. I’m ok with breaking a server if I have good backups, but this seemed to be a complete shot in the dark with no guarantee of success so I held off.
Back to using the built in tools. I had a lot of trouble here- there’s posts all over the internet that apparently show you how to find and delete a bunch of emails, here are a few that I tried (all commands that mention ‘post super’ require sudo). None of these did the job I wanted, perhaps they are too complicated and the syntax is pretty impenetrable for me.
mailq | tail +2 | awk 'BEGIN { RS = "" }
# $7=sender, $8=recipient1, $9=recipient2
{ if ($7 == “manager@company.com" && $9 == "")
print $1 }
' | tr -d '*!' | postsuper -d -
or this
mailq| grep '^[A-Z0-9]'|grep @company.com|cut -f1 -d' ' |tr -d \*|postsuper -d -
or this
mailq | tail +2 | awk 'BEGIN { RS = "" } \ / manager@company\.com$/ { print $1 } \ ' | tr -d '*!' | postsuper -d -
or this
/usr/sbin/postqueue -p | grep '^[A-Z0-9]' | grep *company.com* | cut -f1 -d' ' |tr -d \*|postsuper -d -
I was able to figure out the extent of the problem by using
mailq | grep company.com | awk ‘{ print $1′} | postsuper -d -
to find out how many emails were queued up. Nearly 12,000 in all. So while it was obviously a big problem, not as bad as it could have been. Faint praise?
I did finally get some action with a hint from this page
sudo postsuper -d ALL deferred
This tells Postfix to delete all messages in the deferred queue. These should all be spam, because normal mail would not be deferred- the spam ends up there because the receiving MTA won’t accept it, and this is where we get the rejection messages from.
That’s it. I monitored the queue to see if more spam was being born by issuing
mailq
occasionally- this shows you what is happening in the queue. When I was satisfied that the problem was fixed, I got on to the blacklists and asked to be delisted. Note- you have to do this AFTER cleaning up the server because if you do this and the server is still spewing spam, you’ll find it VERY difficult to get off those lists….. the owners may decide to not talk to you, and then you’re screwed.
Best guess of how the infection occurred is a password compromise on the hr@… account- there are loads of public wifi hotspots in Asia that will sniff your passwords…….
By the way- Apple are deprecating Mac OS Server, and have published a guide (here) that shows you how to move from their services to open source. One of the services Apple are abandoning is Mail, and yet there is NO mention of Mail migration in this new migration guide. Oops