We’ve installed MySQL and PHP 5 security updates. Customers should not notice any changes; the updates just fix several security issues in PHP 5 and MySQL.
The updates were performed in such a way that new Web server connections were delayed during the 30 seconds or so that PHP and MySQL were unavailable on each server. That should mean that as far as scripts on your Web site were concerned, there was zero downtime.
This is a followup to last night’s post about a network outage.
The root cause of the problem was the failure of an Ethernet switch at our data center. The switch was the one that our network cables actually plug into to connect to the Internet. Unfortunately, it’s one of the few pieces of the network infrastructure that’s not automatically redundant: although the “other side” of the switch is connected to multiple fully redundant upstream paths to the Internet, the side of it that goes to our server cabinets effectively has a single connection for each a group of servers.
When the switch failed, the data center staff replaced it with a new spare one. Because the faulty hardware was completely replaced, the problem is properly solved, and this won’t be something that’s an ongoing problem.
Between 9:52 and 11:06 PM Pacific time on January 10, a complete network failure at our primary data center caused an unscheduled outage that resulted in all services (all Web sites and e-mail) being unreachable from the Internet.
This problem has been resolved and all services are now available. We are waiting for a full report from the data center personnel so that we can determine the cause and ensure that it won’t recur.
We sincerely apologize to our customers who were affected by this. This kind of outage is not normal (it’s the longest outage we’ve experienced in more than four years), and we know it’s not acceptable to our customers who rely on our services. We’ll post a followup message with more details when they become available.
Update Friday 10 AM: As a clarification, we should also have originally mentioned that no e-mail is lost during this kind of outage: it’s delivered after the issue is resolved. While some messages were certainly delayed, they were all properly delivered afterward.
A customer pointed out that our servers didn’t have many “locales” installed. A “locale” is a set of rules that apply to a language, region or culture — things like the language’s words for “January” and “Monday”, the way that dates are displayed, and the currency symbol used.
On the morning of December 25, a technical problem with our spam filters allowed more spam than usual for several hours. Customers may have seen a spike in spam arriving during that period.
We found and fixed the root cause of the problem, and it won’t occur again.
We apologize for any inconvenience this may have caused. We know that no one wanted more spam for Christmas!
Our business offices will be closed on Tuesday, January 1 to observe the US legal holiday. As always, our support staff will be providing same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until Wednesday January 2, and telephone support (via callbacks) will be available only for urgent problems.
Our business offices will be closed on Tuesday, December 25 to observe the US legal holiday. As always, our support staff will be providing same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until Wednesday, and telephone support (via callbacks) will be available only for urgent problems.
Here’s an obscure thing we spent some time on today that doesn’t affect our customers, but might be useful to someone else searching the Internet. The problem was a MySQL database table that, as far as we can tell, worked perfectly… except that it shows this message when we run CHECK TABLE:
Table upgrade required. Please do "REPAIR TABLE `users`" to fix it!
We applied a MySQL security update tonight. The version number remains 5.0.32, and customers should not notice any changes; the update just fixes several security issues.
The update was performed in such a way that new Web server connections were delayed during the 30 seconds or so that MySQL was unavailable on each server. That should mean that as far as scripts on your Web site were concerned, there was zero MySQL downtime.
We’ve updated several things on our servers today:
