Today at about 12:26 PM and 12:34 PM (Pacific time), our network was briefly attacked by an extremely high volume of data — a “distributed denial of service” (DDoS) attack using forged (“spoofed”) source addresses. The volume of the attack was more than 50 times greater than the usual peak inbound data rate to all our servers combined. This caused Web sites and e-mail we host to be very slow or timeout completely for a few minutes. (All services are working normally now.)
The same attack happened a week ago. Based on what we learned previously, we were able to trace the attack in more detail, and we have identified a specific controversial site that the attackers are targeting. We have moved that site to a different section of our network that can fail without affecting other sites, and we will work with the site owner to move it to a dedicated DDoS protection service.
We apologize for the problems caused by this incident. We know that achieving maximum uptime and availability is important for all of our customers.
Within the next few weeks, we’ll be making PHP version 5.3.15 available to customers in our account management control panel (and making the 5.3 series the default for all customers several months after that).
We’ve been testing PHP 5.3 ourselves for some time (among other things, it’s been running our Webmail system for several weeks, handling millions of page views without any problems), but it makes sense to test it on a wider variety of sites before deploying it for everyone.
If you would like to help us test PHP 5.3, just contact us and let us know what site(s) you’d like to enable it for. We’ll do that for you (it needs to be done manually by our staff for now).
Read the rest of this entry »
Update 1:41 PM Pacific time: A contact at Level 3 Communications confirms that their San Francisco Bay Area network was disrupted by a “configuration error”, causing problems for a great deal of Internet traffic that passes through Level 3 (not related to us in particular). Level 3 has corrected the problem, so we’re marking this issue as “closed”.
Read the rest of this entry »
We had a couple of brief network interruptions today (at about 8:50am and 12:49pm Pacific time). We are investigating them, and will update this post with more details later.
Updated 2012-07-19 2:23PM Pacific: One of our upstream network providers has traced this to what appears to be some very brief, very high-packet-per-second attacks. The attacks have not recurred, and we are continuing to monitor all systems.
Our business offices will be closed on Wednesday, July 4 to observe the US legal holiday. As always, we’ll provide same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until the next business day, and telephone support (via callbacks) will be available only for urgent problems.
The “web11” server became very slow and needed to be restarted at approximately 2:00 a.m. Pacific time this morning (June 25). This caused a brief outage for Web sites on that server. Other servers were not affected.
Between 11:00 PM and 11:59 PM Pacific time on Saturday June 23 2012, the MySQL database software on each of our servers will be upgraded to version 5.1.63 and restarted. This will cause an approximately 30 second interruption of service on each customer Web site at some point during this hour.
This upgrade is necessary for security reasons. We apologize for the inconvenience this causes.
Update 11:12 PM June 23: The maintenance was completed as planned.
Between 5:10 and 5:22 A.M. Pacific time this morning (June 15), one of our upstream network providers experienced a large distributed denial of service attack (DDoS) targeted at one of their other customers, overwhelming their core network routers. This resulted in many people being unable to connect to our network during this period.
The problem has been resolved (the provider has blocked the attack), and they tell us it should not recur. We sincerely apologize for the inconvenience this caused.
WordPress 3.4 was released yesterday, with some nice new features. Our WordPress one-click installer automatically installs the latest version for new sites. If you’ve previously installed WordPress, you should upgrade it from within your WordPress Dashboard.
Our business offices will be closed on Monday, May 28 to observe the US legal holiday. As always, we’ll provide same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until the next day, and telephone support (via callbacks) will be available only for urgent problems.
