Between 10:00 PM and 10:59 PM Pacific time Wednesday May 22, 2013, the “web05” and “web07” servers will be restarted. This will cause an eight minute interruption of service for each server at some point during this hour.
The “web04” server experienced extremely high load for several minutes beginning at 8:00 AM Pacific time on May 9. Sites on this server were slow or unavailable as a result.
1:31 PM Pacific time: Our technicians are investigating high load and slow page load times on the “web04” server.
2:09 PM Pacific time: This is being caused by a distributed denial of service attack on WordPress sites that is causing outages for many companies. We’re working to block it.
Between 12:50 and 1:23 PM Pacific time, service was intermittently unavailable or slow for sites and e-mail on the web12 server. In addition, customers on other servers may have seen brief delays or high load for about two minutes during this period.
Between 11:04 PM and 11:44 PM March 23, our network was either slow to respond due to high packet loss or completely unavailable to some customers.
There was a brief but severe performance problem on the web12 server today between 9:59 and 10:07 AM Pacific time. During this time, many Web server requests were very slow to load or even “timed out” completely. All services are now operating normally again. Other servers were not affected.
Between 11:00 PM and 11:59 PM Pacific time February 26, 2013, each of our servers will be restarted for a “kernel upgrade”. This will cause an approximately four minute interruption of service for each customer at some point during this hour.
There was a brief outage on the web12 server today starting at about 6:22 PM Pacific time. This was caused by a “SYN flood” attack, which effectively blocked all other connections with the server.
We took steps to work around the attack, which we completed by 7:08 PM Pacific time (46 minutes after the start of the attack). Furthermore, the attack itself seems to have stopped; the steps we took should help in case in starts again.
We sincerely apologize for the interruption in service for those affected customers; we know that reliable service is a primary concern for all of our customers.
At 9:45 PM Pacific time February 6 2013, our “web03” server experienced a “kernel panic” and needed to be restarted. This led to an 11 minute outage of Web sites and e-mail hosted on that server.
All services are now working normally, and other servers were not affected. We apologize for the trouble this caused customers on the web03 server.
Beginning at 3:00 PM Pacific time February 5, a server on our network was the target of an extremely high volume DNS amplification denial of service attack. The inbound network data exceeded 11.6 Gbps, which is an extremely large amount — large enough to exceed the 10 Gpbs capacity of our upstream Ethernet switches and cause our entire network to slow down dramatically.
This affected all servers for about 19 minutes, until we and our network partners began discarding (“null routing”) all traffic targeted at that server. This fixed the problem for the rest of our network, but still left sites on the “web11″ server unavailable.
To solve that, the IP addresses of all sites on the web11 server have been changed to new IP addresses that are working correctly and are not under attack. This was completed by 3:44 PM, and all sites on all servers are now working properly.
If the attackers target another IP address, we’re ready to immediately block that one, too. If that does happen, the way we’ve redistributed the IP addresses, in combination with previous analysis we’ve done on this attack, will allow us to immediately know which site is under attack. (It’s otherwise hard to determine which IP address is involved, because the type of attack we’re seeing targets only an IP address and not a specific Web site name.) That site will then be moved off our main network to prevent a recurrence.
We sincerely apologize for the inconvenience this caused our customers; we know you count on us for reliable service, and we’re committed to doing everything possible to avoid problems.
