Denial of service attack February 5, 2013 (resolved)

Beginning at 3:00 PM Pacific time February 5, a server on our network was the target of an extremely high volume DNS amplification denial of service attack. The inbound network data exceeded 11.6 Gbps, which is an extremely large amount — large enough to exceed the 10 Gpbs capacity of our upstream Ethernet switches and cause our entire network to slow down dramatically.

This affected all servers for about 19 minutes, until we and our network partners began discarding (“null routing”) all traffic targeted at that server. This fixed the problem for the rest of our network, but still left sites on the “web11” server unavailable.

To solve that, the IP addresses of all sites on the web11 server have been changed to new IP addresses that are working correctly and are not under attack. This was completed by 3:44 PM, and all sites on all servers are now working properly.

If the attackers target another IP address, we’re ready to immediately block that one, too. If that does happen, the way we’ve redistributed the IP addresses, in combination with previous analysis we’ve done on this attack, will allow us to immediately know which site is under attack. (It’s otherwise hard to determine which IP address is involved, because the type of attack we’re seeing targets only an IP address and not a specific Web site name.) That site will then be moved off our main network to prevent a recurrence.

We sincerely apologize for the inconvenience this caused our customers; we know you count on us for reliable service, and we’re committed to doing everything possible to avoid problems.