We’ve installed MySQL and PHP 5 security updates. Customers should not notice any changes; the updates just fix several security issues in PHP 5 and MySQL.
The updates were performed in such a way that new Web server connections were delayed during the 30 seconds or so that PHP and MySQL were unavailable on each server. That should mean that as far as scripts on your Web site were concerned, there was zero downtime.
This is a followup to last night’s post about a network outage.
The root cause of the problem was the failure of an Ethernet switch at our data center. The switch was the one that our network cables actually plug into to connect to the Internet. Unfortunately, it’s one of the few pieces of the network infrastructure that’s not automatically redundant: although the “other side” of the switch is connected to multiple fully redundant upstream paths to the Internet, the side of it that goes to our server cabinets effectively has a single connection for each a group of servers.
When the switch failed, the data center staff replaced it with a new spare one. Because the faulty hardware was completely replaced, the problem is properly solved, and this won’t be something that’s an ongoing problem.
Between 9:52 and 11:06 PM Pacific time on January 10, a complete network failure at our primary data center caused an unscheduled outage that resulted in all services (all Web sites and e-mail) being unreachable from the Internet.
This problem has been resolved and all services are now available. We are waiting for a full report from the data center personnel so that we can determine the cause and ensure that it won’t recur.
We sincerely apologize to our customers who were affected by this. This kind of outage is not normal (it’s the longest outage we’ve experienced in more than four years), and we know it’s not acceptable to our customers who rely on our services. We’ll post a followup message with more details when they become available.
Update Friday 10 AM: As a clarification, we should also have originally mentioned that no e-mail is lost during this kind of outage: it’s delivered after the issue is resolved. While some messages were certainly delayed, they were all properly delivered afterward.
A customer pointed out that our servers didn’t have many “locales” installed. A “locale” is a set of rules that apply to a language, region or culture — things like the language’s words for “January” and “Monday”, the way that dates are displayed, and the currency symbol used.
Our business offices will be closed on Tuesday, December 25 to observe the US legal holiday. As always, our support staff will be providing same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until Wednesday, and telephone support (via callbacks) will be available only for urgent problems.
Here’s an obscure thing we spent some time on today that doesn’t affect our customers, but might be useful to someone else searching the Internet. The problem was a MySQL database table that, as far as we can tell, worked perfectly… except that it shows this message when we run CHECK TABLE:
Table upgrade required. Please do "REPAIR TABLE `users`" to fix it!
We applied a MySQL security update tonight. The version number remains 5.0.32, and customers should not notice any changes; the update just fixes several security issues.
The update was performed in such a way that new Web server connections were delayed during the 30 seconds or so that MySQL was unavailable on each server. That should mean that as far as scripts on your Web site were concerned, there was zero MySQL downtime.
We’ve updated several things on our servers today:
We’ve installed several software upgrades on our servers. First of all, updates to our new Webmail system over the last few days fix:
These fixes solve all the bugs that we know of in the new Webmail system (although we have plenty of feature requests that we’re working on). If you’re still using the old system, now’s the time to switch, or to let us know why you’re using the old system so we can address that.
In addition, we also upgraded the following software on our servers Monday night:
As always, let us know if you have any questions or concerns.
We’re pleased to announce that our new Webmail system is available. You’ll find many improved features:
If you’re already a Webmail user, you’ll see the new pages the next time you login. If you haven’t tried Webmail before, this page explains how to get started.
By the way, we’ll be keeping our old Webmail system around for a couple of weeks at this special address, just in case anyone experiences any problems and needs to temporarily “downgrade”. If you have any trouble with the new system, do let us know right away so we can address it before the old pages are permanently removed.
We’d also like to thank the more-than-200 people who helped us test the new system over the last few weeks. The new system is better because of your efforts.
