{"id":181,"date":"2009-04-11T22:49:22","date_gmt":"2009-04-11T14:49:22","guid":{"rendered":"http:\/\/www.computersolutions.cn\/blog\/?p=181"},"modified":"2009-04-11T22:49:22","modified_gmt":"2009-04-11T14:49:22","slug":"bandwidth-and-avoiding-the-slashdotdigg-effect","status":"publish","type":"post","link":"https:\/\/www.computersolutions.cn\/blog\/2009\/04\/bandwidth-and-avoiding-the-slashdotdigg-effect\/","title":{"rendered":"Bandwidth and avoiding the slashdot\/digg effect"},"content":{"rendered":"

As I’ve spent the day doing some pre-emptive maintenance type stuff on our servers, I noticed that one of our servers appeared to have way slower ping times than normal from our other servers.<\/p>\n

Taking a closer look I also saw that the logs said it having timeout problems talking to our database server.<\/p>\n

A quick look at the server logs showed that the server in question was using 100% of its bandwidth for the last hour or so according to Monit.<\/p>\n

A quick check with ntop (excellent web based network analysis statistics) and iftop (console based network traffic analysis) confirmed that 99.5% of the bandwidth was going to HTTP requests.
\nStrangely enough, the server wasn’t even stressed at all (I guess I’ve overspecced that one!)<\/p>\n

I installed the apache2 module server-status ( a2enmod status, apache2ctl restart), and took a closer look at where the requests were going to.<\/p>\n

In this case, all the requests were going to a single domain, to the detriment of other clients on the same server.<\/p>\n

I decided to take a look at limiting the maximum bandwidth that that domain could use. I’ve used mod_bandwidth in the past for limiting apache virtual hosts, but sadly, it seems to have stopped development, so I took a look at whats currently available for apache2.<\/p>\n

A few minutes googling later, I decided on installing mod-cband.<\/p>\n

In debian, its fairly simple to install: (i’m assuming apache2 here)<\/p>\n


\napt-get install libapache2-mod-cband
\napache2ctl restart
\n<\/code><\/p>\n

Now I had it installed, I had to configure it.
\nA quick read of the documentation shows that it needs some additions to the apache configuration.<\/p>\n

I added this to my \/etc\/apache2\/httpd.conf<\/p>\n


\nCBandScoreFlushPeriod 1
\nCBandRandomPulse On
\n<\/code><\/p>\n

and then went to the virtual host in question, and added this:<\/p>\n


\nCBandSpeed 1Mbps 10 200
\n<\/code><\/p>\n

The first number 1Mbps is how much total bandwidth the domain can use.
\nThe second number is how many requests a second the domain can use.
\nThe third number sets the maximum number of client connections to that domain.<\/p>\n

I restarted apache2 – apache2ctl restart, and took a look at the excellent statistics provided by mod_cband<\/p>\n

http:\/\/[theserverinquestion]\/cband-status?refresh=15<\/p>\n

…and then tweaked the CBandSpeed numbers based on what i was seeing (the values in the example above represent the optimum for the traffic i was seeing).<\/p>\n

Any excess visitors over the (quite generously high) limits for that specific domain will see a 503 error, which can be customized to a page letting them know to come back later, as the server has exceeded its bandwidth limits.<\/p>\n

The official documentation for cband is here – http:\/\/cband.linux.pl\/<\/a>, although their website doesn’t appear to be working currently (due to DNS timeouts).<\/p>\n

A check of the logs showed that mod_cband was working nicely.<\/p>\n

\"localhostlocaldomain-if_eth0-day\"<\/p>\n

At around 8pm we started seeing heavy traffic (the far right), which started saturating the link about an hour later.
\nOnce I’d installed mod_cband, and limited that domain, the traffic load dropped nicely.<\/p>\n

Note that this server also acts as a backup server, so the increase in receive traffic at midnight is normal (the dip in the left side).<\/p>\n

Conclusions:<\/strong><\/p>\n

Checking logs is a necessary evil \ud83d\ude42<\/p>\n

I also need to add some additional warning emails to the servers to notify me when things like this happen.
\nI already do this for some things like load, security scans, and failed logins, so its one more thing to monitor!<\/p>\n

I’ve taken a look at this script here – http:\/\/sws.dett.de\/mini\/VHost_Traffic\/<\/a> and will probably implement that at some point so I can see the biggest bandwidth usage in terms of client sites on a daily \/ weekly basis, with a view to moving them to a different server, and charging appropriately (yay, money!)<\/p>\n

The good news is that it looks like the server in question can easily sustain slashdotting.<\/p>\n","protected":false},"excerpt":{"rendered":"

As I’ve spent the day doing some pre-emptive maintenance type stuff on our servers, I noticed that one of our servers appeared to have way slower ping times than normal from our other servers. Taking a closer look I also saw that the logs said it having timeout problems talking to our database server. A […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[25],"tags":[57,56,55,58],"class_list":["post-181","post","type-post","status-publish","format-standard","hentry","category-technical-mumbo-jumbo","tag-apache2","tag-bandwidth","tag-slashdot-effect","tag-usage"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/posts\/181","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/comments?post=181"}],"version-history":[{"count":1,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/posts\/181\/revisions"}],"predecessor-version":[{"id":183,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/posts\/181\/revisions\/183"}],"wp:attachment":[{"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/media?parent=181"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/categories?post=181"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/tags?post=181"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}