Computer Solutions Blog » Service Issues http://www.computersolutions.cn/blog Whats happening at Computer Solutions Fri, 13 Jan 2012 02:32:34 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 Outgoing TLS port Denial of Service – Fixed http://www.computersolutions.cn/blog/2010/01/outgoing-tls-port-denial-of-service-fixed/?utm_source=rss&utm_medium=rss&utm_campaign=outgoing-tls-port-denial-of-service-fixed http://www.computersolutions.cn/blog/2010/01/outgoing-tls-port-denial-of-service-fixed/#comments Tue, 05 Jan 2010 08:24:17 +0000 Lawrence Sheed http://www.computersolutions.cn/blog/?p=330 Noticed that our incoming TLS connection queue was a little high – running at 60 concurrent connections for an hour or so.

A check of the queue revealed that all the connections were coming from a single IP – and were tying up the queue, making it a denial of service attack. This one ip address was connecting and reconnecting multiple times, hogging up all the connections.

I’ve blocked that ip address, and restarted the TLS service, its back at normal levels now.

For the interested –

Connections:

mail:/var/log/qmail/qmail-tls# netstat -an –numeric-ports | grep 587 | grep 58.246.24.242
tcp 0 0 61.129.49.190:587 58.246.24.242:24384 ESTABLISHED
tcp 1 0 61.129.49.190:587 58.246.24.242:39581 CLOSE_WAIT
tcp 0 0 61.129.49.190:587 58.246.24.242:11625 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:42614 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:46630 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:20802 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:51956 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:4463 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:39878 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:43678 ESTABLISHED
tcp 1 0 61.129.49.190:587 58.246.24.242:39181 CLOSE_WAIT
tcp 0 0 61.129.49.190:587 58.246.24.242:62054 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:64421 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:24326 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:58740 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:25779 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:50209 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:41358 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:32383 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:27925 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:46540 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:7049 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:13999 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:62962 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:19771 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:43604 ESTABLISHED
tcp 1 0 61.129.49.190:587 58.246.24.242:38757 CLOSE_WAIT
tcp 0 0 61.129.49.190:587 58.246.24.242:39909 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:30470 ESTABLISHED
tcp 1 0 61.129.49.190:587 58.246.24.242:39754 CLOSE_WAIT
tcp 0 0 61.129.49.190:587 58.246.24.242:61393 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:52072 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:22294 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:60398 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:60530 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:36049 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:1426 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:40190 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:15402 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:23457 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:65187 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:39910 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:23181 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:3286 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:40540 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:12957 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:39829 ESTABLISHED
tcp 0 0 61.129.49.190:587 58.246.24.242:19748 ESTABLISHED

IP Owner:

mail:/var/log/qmail/qmail-tls# whois 58.246.24.242

% [whois.apnic.net node-2]
% Whois data copyright terms http://www.apnic.net/db/dbcopyright.html

inetnum: 58.246.24.240 – 58.246.24.247
netname: SH-Ribeira
country: cn
descr: Ribeira (Shanghai) Business Consulting Co., Ltd.
admin-c: YR194-AP
tech-c: YR194-AP
status: ASSIGNED NON-PORTABLE
changed: sh-ipmaster@chinaunicom.cn 20081125
mnt-by: MAINT-CNCGROUP-SH
source: APNIC

I’ve sent a note to China Unicom, but don’t expect any reply. A google of Ribiera Shanghai doesn’t reveal any obvious people to complain to either.

Yet another instance where I should put some active logging into the server to notify me when queue / connection sizes stay at high levels…

]]> http://www.computersolutions.cn/blog/2010/01/outgoing-tls-port-denial-of-service-fixed/feed/ 0 Another outage! http://www.computersolutions.cn/blog/2009/12/another-outage/?utm_source=rss&utm_medium=rss&utm_campaign=another-outage http://www.computersolutions.cn/blog/2009/12/another-outage/#comments Mon, 07 Dec 2009 15:23:10 +0000 Lawrence Sheed http://www.computersolutions.cn/blog/?p=290 Seems that when it rains, it pours.

The gods were not content to give us only one issue today from an external provider, but two!

At approximately 7pm the network that includes our mail server was on got hit by a massive denial of service attack.
The nice people at Shanghai Telecom decided that they would simply shut off routing for the entire subnet as their optimal solution.

We have a nice graph of that happening here:

net01.day

Note the sudden precipitous drop in network traffic starting at approximately 7pm, which lasted until approximately 8pm.

We also have images of the DoS attack [although not completely, as our network was null routed (shut off) for the brunt of the attack]

You can see the sudden increase in incoming traffic in this image below (which occurred before they killed the network completely).
The green line which indicates incoming packets suddenly goes sky high before the network people shut off the network.

net02z.day

Some of the other servers also got hit by this – notable our web servers, although they didn’t cut those off thankfully.
See below for a view of that traffic.

net02z.88.213.day

As the old curse goes – may you live in interesting times.
Some days are more interesting than others!

]]> http://www.computersolutions.cn/blog/2009/12/another-outage/feed/ 0 System Outage Issues Resolved (April 16th) http://www.computersolutions.cn/blog/2009/04/system-outage-issues-resolved-april-16th/?utm_source=rss&utm_medium=rss&utm_campaign=system-outage-issues-resolved-april-16th http://www.computersolutions.cn/blog/2009/04/system-outage-issues-resolved-april-16th/#comments Thu, 16 Apr 2009 09:37:31 +0000 Lawrence Sheed http://www.computersolutions.cn/blog/?p=218 From 12:00 – 2:40pm today Shanghai Telecom was experiencing router problems for servers in the 61.129.88.xx address space in the data centre at WuSheng Lu (the main Shanghai Telecom building).
This affected 3 of our servers, and one of our clients managed servers.

Shanghai Telecoms official response below:

武胜机房托管了服务器61.129.88段在4月16日12:00-14:00出现无法访问连接，经检查该段均出现该种情况。我们公司技术向电信反映该情况后，经电信查看是由于该段中有主机发送广播包导致路由中毒（环路）而造成的，经过紧急的抢修最终恢复正常。此次给贵公司的日常运作带来很多不便，在此深表歉意。

Unfortunately once they had resolved their router issues, at around 3pm,  Shanghai Telecom decided to create some new ones, by arbitrarily rebooting all the servers in that address space.
Due to their actions, on reboot, our database server could not fully mount the data partition, and so a number of our client websites were unaccessible, as was our webmail service.

Repairing the damage caused by Shanghai Telecoms actions took around 2 1/2 hours.

Full services resumed at approximately 5:20pm

All services are currently running smoothly, although we do have some reports of connectivity issues from some clients.
If you are still unable to connect to the mail server, please turn off your ADSL modem or Router, and log onto the internet again.
(This will clear any route issues  in your router, and you should be able to connect successfully.)

Apologies for the inconvenience.

Lawrence.

]]> http://www.computersolutions.cn/blog/2009/04/system-outage-issues-resolved-april-16th/feed/ 0