One semi-common issue I get to see, is when a server we send mail to doesn’t timeout. This ties up an outgoing mail slot. Over a period of time, this can lead to issues where the whole outgoing or incoming queue is sitting doing nothing, as every connection is tied up by ‘tarpitted’ connections.

Ideally Qmail should be able to cope with these.  There are settings in qmail to control how long a connection takes, and how long it should wait for.  These settings are covered in the following files (usually set in /var/qmail/control)

timeoutconnect – how long for qmail to wait on initial outgoing connection before trying another mail server.
timeoutremote – how long to wait before timing out a connected outgoing server.
timeoutsmtpd – how long for qmail to wait before dropping an incoming connection.

In our system, we set these values to:
30 seconds for timeoutconnect
600 seconds for timeoutremote
360 seconds for timeoutsmtpd

In theory timeoutremote should see qmail drop a connection after 10 minutes (600 seconds).
In practice, qmail doesn’t.

Why?

timeoutremote only applies if the connection hasn’t received any data for the timeout period.
It doesn’t apply to the connection time as a whole.
If the remote end sends some data, the timeout is reset again, and it will wait again for the timeoutremote period. If the remote server dribbles back an ACK or similar once every few minutes, then it can keep a connection alive for as long as it wants.

This may not happen very often, but it can happen enough to tie up our connection queue over a period of time. I’ve seen connections go on for as long as days or weeks in practice.

Ideally one should be able to set a proper timeout period in qmail which it adheres to, so that any connection over a certain time period gets killed, or at least set something up in ucspi-tcp, however thats something for another time.

Here is a real world example.

I’ve run my kill zombie script in test mode (see bottom of page for the script)

/var/qmail/bin/kill-qmail-smtpd-zombies --test
**Running in TEST mode**
Running: ps ax -o etime,pid,comm --no-heading | grep qmail-remote | grep ':[0-9][0-9]:' | awk '{print }'
-=-=-=-=-=-=-=-=-=-=-
Found zombies, setting up shotgun.
Killing qmail-remote zombies
kill -9 26707
-=-=-=-=-=-=-=-=-=-=-

Its come up with a connection thats been running longer than an hour. – 26707

I’ll double check to see that its correct

ps ax -o etime,pid,comm | grep 26707
01:39:07 26707 qmail-remote


Yup, qmail-remote has been running for 1hr39minutes on that connection.

Lets check what the connection is

ps -ef | grep 26707
root 2964 17112 0 13:01 pts/2 00:00:00 grep 26707
qmailr 26707 21959 0 11:23 ? 00:00:00 qmail-remote bamboo.sz.js.cn zhangbin@bamboo.sz.js.cn


Hmm, its a known troublesome server bamboo.sz.js.cn.
In fact, its the one that caused me to write this article!

Lets watch whats actually happening in real time.

strace -p 26707
Process 26707 attached - interrupt to quit
read(3,

[wait for a minute or two...]

Still nothing.

Hmm, sitting there waiting for a response to a read. Guess what happens before the timeout period?
Yup, we receive some more characters just in time to keep the connection up and running…

We could set the timeoutremote to a lower number, but we do actually have cases where servers genuinely are slow on responses for various spam testing reasons (although they usually pickup speed once they pass those tests), so I prefer another method.

Whats my current (lazy in lieu of patching qmail or ucspi-tcp) solution for this?

A culling the zombies script!

To install in your qmail/bin folder, do the following:


cd /var/qmail/bin
wget http://www.computersolutions.cn/blog/wp-content/uploads/2010/02/kill-qmail-zombies.txt
mv kill-qmail-zombies.txt kill-qmail-zombies.sh
chmod 0700 kill-qmail-zombies.sh


The script has a help file built in, parameters are:
./kill-qmail-zombies.sh
--test - Run in test mode (zombie friendly)
--help - Show the help
--force - Kill some zombies!

eg

./kill-qmail-zombies.sh --test

You could set this to run every few hours in a cron script, but I strongly suggest you test first to see if it works correctly. See the help file for more info on that.

Script below for those who want to take a look. Its one of my first shell scripts, so feel free to laugh, and comment accordingly!

#!/bin/sh

# ===========================
# qmail zombie killer script
# Version: 1.0
# Author: L. Sheed
# Company: Computer Solutions
# URL: http://www.computersolutions.cn
# ===========================

PATH=/usr/bin:/bin

function short_usage
{
cat <<- _EOF_
$0: missing parameter
Try '$0 --help' for more information.

_EOF_
}

function usage
{
cat <<- _EOF_
Parameters:
--force  kill qmail-smtpd and qmail-send processes (aka zombies) older than 1 hour
--test 	 do a test run (no zombie processes will be harmed)
--help   show this help page

Notes:
Strongly suggest test first to see if the ps line works correct on your system before killing any processes!
eg -  Run the ps below on your system, and see if the output looks similar

ps ax -o etime,pid,comm --no-heading | grep qmail-smtp
      04:40  6468 qmail-smtpd
      01:47  7473 qmail-smtpd
      01:00  8142 qmail-smtpd
      01:00  8143 qmail-smtpd
      00:46  8235 qmail-smtpd
      00:36  8283 qmail-smtpd
      00:19  8391 qmail-smtpd
      00:11  8445 qmail-smtpd
      00:07  8494 qmail-smtpd

_EOF_
}

function zap_the_bastards
{
PLIST=`ps ax -o etime,pid,comm --no-heading | grep $WHAT | grep ':[0-9][0-9]:' | awk '{print $2}'`

#In test mode, show what would be called also
if [ "$test" = "1" ]; then
	echo "Running:  ps ax -o etime,pid,comm --no-heading | grep $WHAT | grep ':[0-9][0-9]:' | awk '{print $2}'"
fi

if [ -n "${PLIST:-}" ]
then
	echo "-=-=-=-=-=-=-=-=-=-=-"
	echo "Found zombies, setting up shotgun."
	echo "Killing $WHAT zombies"
	for p in $PLIST
	do
		if [ "$force" = "1" ]; then
			echo "Kabooom:"
			kill -9 $p
		fi
		echo "kill -9 $p"
	done
	echo "-=-=-=-=-=-=-=-=-=-=-"
else
	echo "Good news everybody.  No $WHAT zombies found."
fi
}

## Main

#parse our parameters
if [ ! $# == 1 ]; then
	short_usage
	exit
fi

while [ "$1" != "" ]; do
 case $1 in
        --force )
        echo "**Running in FORCE mode**"
        force=1
        ;;
        --help )
        usage
        exit
        ;;
	--test )
	echo "**Running in TEST mode**"
	test=1
	;;
 esac
shift
done

#do the deed
targets=( "qmail-remote" "qmail-smtpd" )

for target in ${targets[@]}
do
	WHAT=$target
	zap_the_bastards
done

Have been seeing sample dictionary attacks on some servers for a while now from random ip addresses – eg

Sep 28 13:01:03 www vpopmail[20410]: vchkpw-pop3: vpopmail user not found www@:24.153.205.71
Sep 28 13:01:03 www vpopmail[20411]: vchkpw-pop3: vpopmail user not found web@:24.153.205.71
Sep 28 13:01:09 www vpopmail[20417]: vchkpw-pop3: vpopmail user not found web@:24.153.205.71
Sep 28 13:01:11 www vpopmail[20420]: vchkpw-pop3: vpopmail user not found web@:24.153.205.71

Annoying, but not realistically going to provide much of a security issue – most of the user names are the generic ones which aren’t actually in use on the servers.

As we already use fail2ban to perform basic service blocks against naughty script kiddie wannabee’s, why not have it block vpopmail attacks also.

Our mail error logs are located in /var/log/mail.log

As you saw above, the logs show the same common text for each failed login –

vchkpw-pop3: vpopmail user not found web@:24.153.205.71

A simple regex to identify that in the logs would look like this (as per the fail2ban wiki)

failregex = vchkpw-pop3: vpopmail user not found .*@:<HOST>$

First step is to create a filter for fail2ban.

Create /etc/fail2ban/filter.d/vpopmail.conf as below:

# Fail2Ban configuration file for vpopmail
#
# Author: Lawrence Sheed
#
# $Revision: 1.0 $
#

[Definition]

# Option: failregex
# Notes.: regex to match the password failures messages in the logfile.
# Values: TEXT
#
failregex = vchkpw-pop3: vpopmail user not found .*@:<HOST>$

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#
ignoreregex =

Second step is to add our filter to the fail2ban setup

Add this to the bottom of /etc/fail2ban/jail.conf

[vpopmail]
enabled = true
port    = pop3
filter  = vpopmail
logpath = /var/log/mail.log
maxretry = 3

logpath should be amended to whatever your mail logs for vpopmail appear.
maxretry should be set to a value that you agree with.

Restart fail2ban with a: /etc/init.d/fail2ban restart
and check that it has added the filter.

tail /var/log/fail2ban.log

You should see a line like this:

2009-10-01 12:36:09,590 fail2ban.jail   : INFO   Jail 'vpopmail' started

If so, you’re all set!

Some additional tips, as I have found some issues subsequently in Fail2ban on some systems:

If you find that fail2ban gives error 200 or 400 on occasion, this is due to a timing issue bug in fail2ban.
There are 2 possible solutions:

Solution 1 – Edit fail2ban

Open /usr/bin/fail2ban-client

Look for

def __processCmd(self, cmd, showRet = True):
beautifier = Beautifier() for c in cmd:

After for c in cmd: add a delay
time.sleep(0.5)

This should look similar to this now –

def __processCmd(self, cmd, showRet = True):
beautifier = Beautifier() for c in cmd:
time.sleep(0.5)

Save, and restart fail2ban. If you still see 200 or 400 issues, increase the delay higher e.g. time.sleep(0.8)

Solution 2 – Use a different block method

Instead of iptables, we can configure fail2ban to use route

Add a config file for this:

pico /etc/fail2ban/action.d/route.conf

Add this into the file and save it.

# Fail2Ban configuration file
[Definition]
actionban = ip route add unreachable <ip>
actionunban = ip route del unreachable <ip>

Open /etc/fail2ban/jail.conf

Look for ban action = … in the [DEFAULT] section, and comment it out with a # at the start of the line
then add
eg

#banaction = iptables
banaction = route

Save the file.
Restart fail2ban

It will now use route to block bad ip’s.

A check of the logs showed that we’ve received at least 100,000 of these spam mails over the last month that have gotten through to our users.
This is something I’d obviously like to remedy.  Not receiving, processing, or storing that much spam free’s up the servers for other things.

As the number of valid addresses using @live.com accounts appears to be minimal (I could only see a handful of legitimate users sending from that domain), I have taken the decision to block any email from the @live.com domain until Microsoft can resolve their spam issues.

If you do have clients using @live.com addresses, you will be able to send email to them, but not receive from them.
We apologize for the inconvenience, but unfortunately there is no other solution that easily mitigates the issue, other than completely blocking them.

For a more technical explanation of whats happening, read below:

This is a header from a sample spam email from a live.com address.
As you can see below, the header shows that it passes an SPF check – meaning that the sending email server was verified to be a microsoft one.
That means that the sender also passes our greylist and SPF checks, as Hotmail is a valid sender (for most of the time!).

Return-Path: <lourdesuxanbirr1980@live.com>
Delivered-To: XXXX
Received: (qmail 3070 invoked from network); 20 Apr 2009 11:53:37 +0800
DomainKey-Status: no signature
Received: from blu0-omc2-s16.blu0.hotmail.com (65.55.111.91)
by mail.computersolutions.cn with SMTP; 20 Apr 2009 11:53:37 +0800
Received-SPF: pass (mail.computersolutions.cn: SPF record at spf-a.hotmail.com designates 65.55.111.91 as permitted sender)
Received: from BLU128-W5 ([65.55.111.72]) by blu0-omc2-s16.blu0.hotmail.com with Microsoft SMTPSVC(6.0.3790.3959);
Sun, 19 Apr 2009 20:54:11 -0700
Message-ID: <BLU128-W51B2E2DECCF46C5980E74DC760@phx.gbl>
Return-Path: lourdesuxanbirr1980@live.com
X-Originating-IP: [201.150.66.6]
From: Lourdes Browne <Lourdesuxanbirr1980@live.com>
Sender: <lourdesuxanbirr1980@live.com>
To: XXXXX
Subject: Hi! This is Muriel. Young girls in action with animals.
Date: Mon, 20 Apr 2009 03:54:11 +0000
Importance: Normal
Content-Type: text/plain; charset=”iso-8859-1″
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 20 Apr 2009 03:54:11.0766 (UTC) FILETIME=[AA2F5560:01C9C16B]

As the sender is a legitimate hotmail / live account “lourdesuxanbirr1980@live.com” (albeit a garbage generated name), its probable that the sender is generated from a script.

A check on google reveals that the live.com captcha system has been cracked, and is being abused by botnets to send spam.

http://arstechnica.com/security/news/2008/04/gone-in-60-seconds-spambot-cracks-livehotmail-captcha.ars

This probably explains the sudden flood of spam coming from @live.com addresses, although its a bit strange that we didn’t see this sooner!
Hopefully they’ll resolve it soon, so we can unblock them.

12.149.35.75 does not like recipient.
Remote host said: 554 Service unavailable; Client host [usa.computersolutions.cn] blocked using Barracuda Reputation; http://bbl.barracudacentral.com/q.cgi?ip=72.51.39.20
Giving up on 12.149.35.75.

Simple enough – we’re getting blocked by Barracuda Reputation, so off I go to the link to see why.

Sorry, your email was blocked

We are sorry you have reached this page because an email was blocked based on its originating IP address having a “poor” reputation. The “poor” reputation may have been caused by one of the following reasons:

* Your email server contains a virus and has been sending out spam.
* Your email server may be misconfigured.
* Your PC may be infected with a virus or botnet software program.
* Someone in your organization may have a PC infected with a virus or botnet program.
* You may be utilizing a dynamic IP address which was previously utilized by a known spammer.
* Your marketing department may be sending out bulk emails that do not comply with the CAN-SPAM Act.
* You may have an insecure wireless network which is allowing unknown users to use your network to send spam.
* In some rare cases, your recipient’s Barracuda Spam Firewall may be misconfigured.

A quick check of our ip space over at a more legitimate place shows we’re fine – http://www.senderbase.org/senderbase_queries/detailip?search_string=72.51.39.20

I double check with a rbl lookup over here – http://www.mxtoolbox.com/blacklists.aspx, nope, we’re clean as a whistle.

However, on the same page, they have an big button helpfully letting us know that:

Many Barracuda Spam & Virus Firewalls are configured, as a policy, to automatically deliver email that comes from sources that are properly registered at EmailReg.org.

Ok, so follow the link through to EmailReg.org, and sign up.
Looks good until we get to the – a $20 fee will be charged per domain per year.

Hmm, so email will possibly be blocked by Barracuda unless I pay them $20 a year.
Sounds like Blackmail to me.

I also note that although EmailReg.org appears to be a separate entity, it is in fact owned by Barracuda. So a neutral third party blocking service just so happens to be owned by the people doing the blocking. If thats not a conflict of interest, I don’t know what is!
This is actually illegal in some countries, although apparently, not the USA.
It also doesn’t stop actual spammers coughing up money, and getting greenlisted.

Seems the rest of the net agrees with us on this one.

Quote from Mike E. that pretty much sums it up: I feel compelled to add this. If I’m paying Barracuda for a appliance to filter out spam and they in turn are being paid by spammers to allow their messages through my spam firewall, how is that different than an antivirus company taking money from somone that write viruses to have their product not detect a virus? None. It’s slimy.

So, in future when clients are unable to send mail to people using Barracuda firewall devices, I’ll be able to point them to this post, and let them know the situation.

We don’t like spam either, and work hard to avoid clients misusing our services.
However, we don’t blackmail senders into paying us money to accept their mail.

For a rundown on legitimate practices, read this:

http://en.wikipedia.org/wiki/Anti-spam_techniques_(e-mail)

Further References/Complaints:
http://www.nabble.com/zen.spamhaus.org-td22805806.html
http://www.debian-administration.org/users/simonw/weblog/295
http://zacharyozer.blogspot.com/2008/10/worst-engineers-ever.html
http://andrew.triumf.ca/barracuda-problems.html
http://community.spiceworks.com/topic/32502
http://steve.heyvan.com/2008/11/06/technology-reviews/barracudacentral-another-blacklist-black-hole/
http://ithelp.ithome.com.tw/question/10013491?tab=opinion (Trad Chinese)
http://www.linux.com/feature/155880

http://www.emailreg.org/