Occasionally even in a well maintained system, qmail has issues.<\/p>\n
One semi-common issue I get to see, is when a server we send mail to doesn’t timeout. This ties up an outgoing mail slot. Over a period of time, this can lead to issues where the whole outgoing or incoming queue is sitting doing nothing, as every connection is tied up by ‘tarpitted’ connections.<\/p>\n
Ideally Qmail should be able to cope with these.\u00a0 There are settings in qmail to control how long a connection takes, and how long it should wait for.\u00a0 These settings are covered in the following files (usually set in \/var\/qmail\/control)<\/p>\n
<\/p>\n
timeoutconnect – how long for qmail to wait on initial outgoing connection before trying another mail server.
\ntimeoutremote – how long to wait before timing out a connected outgoing server.
\ntimeoutsmtpd – how long for qmail to wait before dropping an incoming connection.<\/p><\/blockquote>\nIn our system, we set these values to:
\n30 seconds for timeoutconnect
\n600 seconds for timeoutremote
\n360 seconds for timeoutsmtpd<\/p>\nIn theory timeoutremote should see qmail drop a connection after 10 minutes (600 seconds).
\nIn practice, qmail doesn’t<\/strong>.<\/p>\nWhy?<\/p>\n
timeoutremote only<\/strong> applies if the connection hasn’t received any data for the timeout period.
\nIt doesn’t apply to the connection time as a whole<\/em>.
\nIf the remote end sends some data, the timeout is reset again, and it will wait again for the timeoutremote period. If the remote server dribbles back an ACK or similar once every few minutes, then it can keep a connection alive for as long as it wants.<\/p>\nThis may not happen very often, but it can happen enough to tie up our connection queue over a period of time. I’ve seen connections go on for as long as days or weeks in practice.<\/p>\n
Ideally one should be able to set a proper timeout period in qmail which it adheres to, so that any connection over a certain time period gets killed, or at least set something up in ucspi-tcp, however thats something for another time.<\/p>\n
Here is a real world example. <\/p>\n
I’ve run my kill zombie script in test mode (see bottom of page for the script)<\/p>\n
\/var\/qmail\/bin\/kill-qmail-smtpd-zombies --test
\n**Running in TEST mode**
\nRunning: ps ax -o etime,pid,comm --no-heading | grep qmail-remote | grep ':[0-9][0-9]:' | awk '{print }'
\n-=-=-=-=-=-=-=-=-=-=-
\nFound zombies, setting up shotgun.
\nKilling qmail-remote zombies
\nkill -9 26707
\n-=-=-=-=-=-=-=-=-=-=-<\/code><\/p>\nIts come up with a connection thats been running longer than an hour. – 26707<\/p>\n
I’ll double check to see that its correct<\/p>\n
ps ax -o etime,pid,comm | grep 26707
\n 01:39:07 26707 qmail-remote
\n<\/code><\/p>\nYup, qmail-remote has been running for 1hr39minutes on that connection.<\/p>\n
Lets check what the connection is<\/p>\n
ps -ef | grep 26707
\nroot 2964 17112 0 13:01 pts\/2 00:00:00 grep 26707
\nqmailr 26707 21959 0 11:23 ? 00:00:00 qmail-remote bamboo.sz.js.cn zhangbin@bamboo.sz.js.cn
\n<\/code><\/p>\nHmm, its a known troublesome server bamboo.sz.js.cn<\/strong>.
\nIn fact, its the one that caused me to write this article!<\/p>\nLets watch whats actually happening in real time.<\/p>\n
strace -p 26707
\nProcess 26707 attached - interrupt to quit
\nread(3, <\/code><\/p>\n[wait for a minute or two…]<\/p>\n
Still nothing.<\/p>\n
Hmm, sitting there waiting for a response to a read. Guess what happens before the timeout period?
\nYup, we receive some more characters just in time to keep the connection up and running…<\/p>\nWe could set the timeoutremote to a lower number, but we do actually have cases where servers genuinely are slow on responses for various spam testing reasons (although they usually pickup speed once they pass those tests), so I prefer another method.<\/p>\n
Whats my current (lazy in lieu of patching qmail or ucspi-tcp) solution for this?<\/p>\n
A culling the zombies script!<\/p>\n
To install in your qmail\/bin folder, do the following:<\/p>\n
\ncd \/var\/qmail\/bin
\nwget http:\/\/www.computersolutions.cn\/blog\/wp-content\/uploads\/2010\/02\/kill-qmail-zombies.txt
\nmv kill-qmail-zombies.txt kill-qmail-zombies.sh
\nchmod 0700 kill-qmail-zombies.sh
\n<\/code><\/p>\nThe script has a help file built in, parameters are:
\n.\/kill-qmail-zombies.sh
\n--test - Run in test mode (zombie friendly)
\n--help - Show the help
\n--force - Kill some zombies!<\/code><\/p>\neg<\/p>\n
.\/kill-qmail-zombies.sh --test<\/code><\/p>\nYou could set this to run every few hours in a cron script, but I strongly<\/strong> suggest you test first to see if it works correctly. See the help file for more info on that.<\/p>\n
Script below for those who want to take a look. Its one of my first shell scripts, so feel free to laugh, and comment accordingly!<\/p>\n
\r\n#!\/bin\/sh\r\n\r\n# ===========================\r\n# qmail zombie killer script\r\n# Version: 1.0\r\n# Author: L. Sheed\r\n# Company: Computer Solutions\r\n# URL: http:\/\/www.computersolutions.cn\r\n# ===========================\r\n\r\nPATH=\/usr\/bin:\/bin\r\n\r\nfunction short_usage\r\n{\r\ncat <<- _EOF_\r\n$0: missing parameter\r\nTry '$0 --help' for more information.\r\n\r\n_EOF_\r\n}\r\n\r\nfunction usage\r\n{\r\ncat <<- _EOF_\r\nParameters:\r\n--force kill qmail-smtpd and qmail-send processes (aka zombies) older than 1 hour\r\n--test \t do a test run (no zombie processes will be harmed)\r\n--help show this help page\r\n\r\nNotes:\r\nStrongly suggest test first to see if the ps line works correct on your system before killing any processes!\r\neg - Run the ps below on your system, and see if the output looks similar\r\n\r\nps ax -o etime,pid,comm --no-heading | grep qmail-smtp\r\n 04:40 6468 qmail-smtpd\r\n 01:47 7473 qmail-smtpd\r\n 01:00 8142 qmail-smtpd\r\n 01:00 8143 qmail-smtpd\r\n 00:46 8235 qmail-smtpd\r\n 00:36 8283 qmail-smtpd\r\n 00:19 8391 qmail-smtpd\r\n 00:11 8445 qmail-smtpd\r\n 00:07 8494 qmail-smtpd\r\n\r\n_EOF_\r\n}\r\n\r\nfunction zap_the_bastards\r\n{\r\nPLIST=`ps ax -o etime,pid,comm --no-heading | grep $WHAT | grep ':[0-9][0-9]:' | awk '{print $2}'`\r\n\r\n#In test mode, show what would be called also\r\nif [ \"$test\" = \"1\" ]; then\r\n\techo \"Running: ps ax -o etime,pid,comm --no-heading | grep $WHAT | grep ':[0-9][0-9]:' | awk '{print $2}'\"\r\nfi\r\n\r\nif [ -n \"${PLIST:-}\" ]\r\nthen\r\n\techo \"-=-=-=-=-=-=-=-=-=-=-\"\r\n\techo \"Found zombies, setting up shotgun.\"\r\n\techo \"Killing $WHAT zombies\"\r\n\tfor p in $PLIST\r\n\tdo\r\n\t\tif [ \"$force\" = \"1\" ]; then\r\n\t\t\techo \"Kabooom:\"\r\n\t\t\tkill -9 $p\r\n\t\tfi\r\n\t\techo \"kill -9 $p\"\r\n\tdone\r\n\techo \"-=-=-=-=-=-=-=-=-=-=-\"\r\nelse\r\n\techo \"Good news everybody. No $WHAT zombies found.\"\r\nfi\r\n}\r\n\r\n## Main\r\n\r\n#parse our parameters\r\nif [ ! $# == 1 ]; then\r\n\tshort_usage\r\n\texit\r\nfi\r\n\r\nwhile [ \"$1\" != \"\" ]; do\r\n case $1 in\r\n --force )\r\n echo \"**Running in FORCE mode**\"\r\n force=1\r\n ;;\r\n --help )\r\n usage\r\n exit\r\n ;;\r\n\t--test )\r\n\techo \"**Running in TEST mode**\"\r\n\ttest=1\r\n\t;;\r\n esac\r\nshift\r\ndone\r\n\r\n#do the deed\r\ntargets=( \"qmail-remote\" \"qmail-smtpd\" )\r\n\r\nfor target in ${targets[@]}\r\ndo\r\n\tWHAT=$target\r\n\tzap_the_bastards\r\ndone<\/pre>\n","protected":false},"excerpt":{"rendered":"Occasionally even in a well maintained system, qmail has issues. One semi-common issue I get to see, is when a server we send mail to doesn’t timeout. This ties up an outgoing mail slot. Over a period of time, this can lead to issues where the whole outgoing or incoming queue is sitting doing nothing, […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[73,25],"tags":[193,192,533,190,191,188],"class_list":["post-355","post","type-post","status-publish","format-standard","hentry","category-email","category-technical-mumbo-jumbo","tag-connection","tag-long","tag-qmail","tag-qmail-remote","tag-timeout","tag-zombies"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/posts\/355","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/comments?post=355"}],"version-history":[{"count":13,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/posts\/355\/revisions"}],"predecessor-version":[{"id":364,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/posts\/355\/revisions\/364"}],"wp:attachment":[{"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/media?parent=355"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/categories?post=355"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/tags?post=355"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}