Postfix: Deal with a cracked email account
This is a tipical friday story on sysadmin world, starting with a common Munin alert email and ending fighting against a cracked email account due a weak password. The alert referred to a weird problem with inodes outside range in / partition:
servers :: our.server :: Inode usage in percent CRITICALs: / is 100.00 (outside range [:98]).
Inode usage in percent was outside normal range, but df
gave me no clue about that strange situation. So after a couple of searches about how to check a directories' list order by inodes usage, I found an interesting resource (stackoverflow, of course):
# find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n 9 tmp 23 lib32 31 root 91 bin 129 sbin 212 boot 888 etc 2929 lib 27393 usr 2327393 var # cd /var # find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n 245 log 912 cache 2325393 spool
Round one, MTA (Postfix in this case) was the main suspect. After playing a bit with mailq
I confirmed that something was rotting there, so before restart the service and clear the queue, should find from where the disaster came: external IP or local script.
Dropping port 25 vía iptables should resolve the question (iptables -A INPUT -p tcp --dport 25 -j DROP
) and there it was, attack stopped. So we configure iptables again to accept pettions -iptables -A INPUT -p tcp --dport 25 -j DROP
- and let's see what netstat can show:
# netstat -nap | grep ":25" | more tcp 0 0 192.168.1.25:25 0.0.0.0:* LISTEN 2239/master tcp 0 14 192.168.1.25:25 94.251.111.121:56253 ESTABLISHED 2310/smtpd
After a docen of tries, the fact of ever seeing that IP (94.251.111.121) was really annoying, even further in case of a .ru IP so we proceeded with a ban to really confirm the source of the attack.
# iptables -A INPUT -s 94.251.111.121 -p tcp --dport 25 -j DROP
Attack stopped again, in fact we have found the real offender but... how that machine have obtained that right?. That's the other unknown we must resolve. As we still have the mailq
information (remember, we didn't stop the service nor delete the queue), we could proceed with some commands that help us in searching. With mailq
we can focus the mail ID, and withpostcat
we can show the mail details:
# mailq 840A4C0FF0 1145 Fri May 23 13:56:34 non-existent@ourdomain.com (host alt1.gmail-smtp-in.l.googl... # postcat -q 840A4C0FF0 # postcat -q 840A4C0FF0 | grep sasl_username named_attribute: sasl_username=email@ourdomain.com
We can figure that someone is sending spam with the email@ourdomain.com local account, but the best approach is to see more information than an unique random ID/email:
# for i in $(mailq | cut -f1 -d" " | grep -v "(" | grep -v "-" | sort | uniq); do postcat -q $i | grep sasl_username; done named_attribute: sasl_username=email@ourdomain.com named_attribute: sasl_username=email@ourdomain.com ...
Suspicion confirmed, a weak password was the cause of all that weird situation. So let's change the password, clear the queue and restart the servers. Last task, but not least, is to have a serious talk with the proper user.