At 12:50 AM Pacific time, the web08 server experienced extremely high disk load and needed to be restarted as a result, resulting in approximately 5 minutes downtime for sites on that server. Other servers were not affected, and the server is now working normally.
Today at about 12:26 PM and 12:34 PM (Pacific time), our network was briefly attacked by an extremely high volume of data — a “distributed denial of service” (DDoS) attack using forged (“spoofed”) source addresses. The volume of the attack was more than 50 times greater than the usual peak inbound data rate to all our servers combined. This caused Web sites and e-mail we host to be very slow or timeout completely for a few minutes. (All services are working normally now.)
The same attack happened a week ago. Based on what we learned previously, we were able to trace the attack in more detail, and we have identified a specific controversial site that the attackers are targeting. We have moved that site to a different section of our network that can fail without affecting other sites, and we will work with the site owner to move it to a dedicated DDoS protection service.
We apologize for the problems caused by this incident. We know that achieving maximum uptime and availability is important for all of our customers.
Within the next few weeks, we’ll be making PHP version 5.3.15 available to customers in our account management control panel (and making the 5.3 series the default for all customers several months after that).
We’ve been testing PHP 5.3 ourselves for some time (among other things, it’s been running our Webmail system for several weeks, handling millions of page views without any problems), but it makes sense to test it on a wider variety of sites before deploying it for everyone.
If you would like to help us test PHP 5.3, just contact us and let us know what site(s) you’d like to enable it for. We’ll do that for you (it needs to be done manually by our staff for now).
Update 1:41 PM Pacific time: A contact at Level 3 Communications confirms that their San Francisco Bay Area network was disrupted by a “configuration error”, causing problems for a great deal of Internet traffic that passes through Level 3 (not related to us in particular). Level 3 has corrected the problem, so we’re marking this issue as “closed”.
We had a couple of brief network interruptions today (at about 8:50am and 12:49pm Pacific time). We are investigating them, and will update this post with more details later.
Updated 2012-07-19 2:23PM Pacific: One of our upstream network providers has traced this to what appears to be some very brief, very high-packet-per-second attacks. The attacks have not recurred, and we are continuing to monitor all systems.
