A heads-up if you use Ruby on Rails: We’re going to be upgrading the default version on our servers to 2.0.2 soon. We want to give you plenty of notice, because when we tried upgrading some older test applications, they didn’t work without changes.
As you probably know, we back up all of our servers every night. Our goal is to keep at least seven days of backups available, with additional older backups available where possible.
We’ve added a new page to our control panel showing the backups available for each account. To see it, just login to the control panel, then click Backups.
We make backups so that we can recover from unexpected occurrences such as data erasure or server failures. Of course, we also make the backups available to customers, because they can be a real life-saver when you need one. However, we want to remind customers that they should also make their own backups, especially if you need a different backup frequency or retention policy. Our Web page describing our backup system and policies has more details.
We’ve installed MySQL and PHP 5 security updates. Customers should not notice any changes; the updates just fix several security issues in PHP 5 and MySQL.
The updates were performed in such a way that new Web server connections were delayed during the 30 seconds or so that PHP and MySQL were unavailable on each server. That should mean that as far as scripts on your Web site were concerned, there was zero downtime.
Between 9:52 and 11:06 PM Pacific time on January 10, a complete network failure at our primary data center caused an unscheduled outage that resulted in all services (all Web sites and e-mail) being unreachable from the Internet.
This problem has been resolved and all services are now available. We are waiting for a full report from the data center personnel so that we can determine the cause and ensure that it won’t recur.
We sincerely apologize to our customers who were affected by this. This kind of outage is not normal (it’s the longest outage we’ve experienced in more than four years), and we know it’s not acceptable to our customers who rely on our services. We’ll post a followup message with more details when they become available.
Update Friday 10 AM: As a clarification, we should also have originally mentioned that no e-mail is lost during this kind of outage: it’s delivered after the issue is resolved. While some messages were certainly delayed, they were all properly delivered afterward.
A customer pointed out that our servers didn’t have many “locales” installed. A “locale” is a set of rules that apply to a language, region or culture — things like the language’s words for “January” and “Monday”, the way that dates are displayed, and the currency symbol used.
On the morning of December 25, a technical problem with our spam filters allowed more spam than usual for several hours. Customers may have seen a spike in spam arriving during that period.
We found and fixed the root cause of the problem, and it won’t occur again.
We apologize for any inconvenience this may have caused. We know that no one wanted more spam for Christmas!
Here’s an obscure thing we spent some time on today that doesn’t affect our customers, but might be useful to someone else searching the Internet. The problem was a MySQL database table that, as far as we can tell, worked perfectly… except that it shows this message when we run CHECK TABLE:
Table upgrade required. Please do "REPAIR TABLE `users`" to fix it!
We applied a MySQL security update tonight. The version number remains 5.0.32, and customers should not notice any changes; the update just fixes several security issues.
The update was performed in such a way that new Web server connections were delayed during the 30 seconds or so that MySQL was unavailable on each server. That should mean that as far as scripts on your Web site were concerned, there was zero MySQL downtime.
We’ve updated several things on our servers today:
We’ve installed several software upgrades on our servers. First of all, updates to our new Webmail system over the last few days fix:
These fixes solve all the bugs that we know of in the new Webmail system (although we have plenty of feature requests that we’re working on). If you’re still using the old system, now’s the time to switch, or to let us know why you’re using the old system so we can address that.
In addition, we also upgraded the following software on our servers Monday night:
As always, let us know if you have any questions or concerns.
