Noticed that our incoming TLS connection queue was a little high – running at 60 concurrent connections for an hour or so.

A check of the queue revealed that all the connections were coming from a single IP – and were tying up the queue, making it a denial of service attack. This one ip address was connecting and reconnecting multiple times, hogging up all the connections.
Read more »

Seems that when it rains, it pours.

The gods were not content to give us only one issue today from an external provider, but two!

At approximately 7pm the network that includes our mail server was on got hit by a massive denial of service attack.
The nice people at Shanghai Telecom decided that they would simply shut off routing for the entire subnet as their optimal solution.

We have a nice graph of that happening here:

Note the sudden precipitous drop in network traffic starting at approximately 7pm, which lasted until approximately 8pm.

We also have images of the DoS attack [although not completely, as our network was null routed (shut off) for the brunt of the attack]

You can see the sudden increase in incoming traffic in this image below (which occurred before they killed the network completely).
The green line which indicates incoming packets suddenly goes sky high before the network people shut off the network.

Some of the other servers also got hit by this – notable our web servers, although they didn’t cut those off thankfully.
See below for a view of that traffic.

As the old curse goes – may you live in interesting times.
Some days are more interesting than others!

From 12:00 – 2:40pm today Shanghai Telecom was experiencing router problems for servers in the 61.129.88.xx address space in the data centre at WuSheng Lu (the main Shanghai Telecom building).
This affected 3 of our servers, and one of our clients managed servers.

Shanghai Telecoms official response below:

武胜机房托管了服务器61.129.88段在4月16日12:00-14:00出现无法访问连接，经检查该段均出现该种情况。我们公司技术向电信反映该情况后，经电信查看是由于该段中有主机发送广播包导致路由中毒（环路）而造成的，经过紧急的抢修最终恢复正常。此次给贵公司的日常运作带来很多不便，在此深表歉意。

Unfortunately once they had resolved their router issues, at around 3pm,  Shanghai Telecom decided to create some new ones, by arbitrarily rebooting all the servers in that address space.
Due to their actions, on reboot, our database server could not fully mount the data partition, and so a number of our client websites were unaccessible, as was our webmail service.

Repairing the damage caused by Shanghai Telecoms actions took around 2 1/2 hours.

Full services resumed at approximately 5:20pm

All services are currently running smoothly, although we do have some reports of connectivity issues from some clients.
If you are still unable to connect to the mail server, please turn off your ADSL modem or Router, and log onto the internet again.
(This will clear any route issues  in your router, and you should be able to connect successfully.)

Apologies for the inconvenience.

Lawrence.