Our business offices will be closed on Monday, May 26 to observe the US legal holiday. As always, we’ll provide same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until the next day, and telephone support (via callbacks) will be available only for urgent problems.
This afternoon at 3:49 PM (Pacific time), one of the cabinets at our data center tripped a circuit breaker, causing all of the servers in that cabinet to lose power. Power was restored nine minutes later.
Customer Web sites on the calculon, lrrr, and zapp Web servers were unavailable during this time. The ability to send and receive e-mail was also interrupted (no mail was lost, of course). Other servers were not affected.
We pay close attention to the power load in each cabinet to avoid this sort of problem. The previously measured peak load of that cabinet had been 12 amps. Since the circuit allows 15 amps, this issue surprised us (we’ve been using the same setup in the same data center for seven years and this has never happened before). It appears that a combination of several servers experiencing unusually high CPU loads led to power usage beyond what we previously considered possible.
We will take immediate steps to make sure the problem doesn’t happen again, and we sincerely apologize to customers who were affected by this incident.
Update 7:26 PM: We have removed a server from the cabinet in question, lowering the power use.
Update 10:38 PM: We have removed a second server from the cabinet, ensuring that power use is well below any level that could cause further trouble. The problem will not recur.
If you use the WordPress 2.5 blog software on your site, be sure to upgrade to WordPress 2.5.1 as soon as possible. The upgrade contains an important security fix. (We’ve updated our own blog, and it was painless.)
Although all WordPress users should upgrade right away, we’ve also added a security rule to our servers to try and protect our customers who haven’t yet upgraded. Other people may also find the security rule useful if they use mod_security on Apache Web servers. The rest of this post contains more technical details.
A heads-up if you use Ruby on Rails: We’re going to be upgrading the default version on our servers to 2.0.2 soon. We want to give you plenty of notice, because when we tried upgrading some older test applications, they didn’t work without changes.
Starting at 10:14 AM this morning, our elzar server experienced an unexpectedly high server load that effectively made some processes on the server unusable for about 10 minutes.
Web sites using scripts or databases on the elzar server may have seemed unresponsive during that time. Also, any customer hosted on elzar who was reading their e-mail during this time may have felt the system was slow or unresponsive (no e-mail was lost, of course).
Customers on other servers were not affected.
The “farnsworth” Apache Web server had an outage lasting approximately five minutes at 11:36 AM Pacific time today, resulting in an interruption of service for Web sites and e-mail on that server. Other servers were not affected.
The problem occurred when the Apache web server process failed to gracefully restart when a new SSL certificate was added. We have discovered why this happened and will take steps to prevent it in the future.
We sincerely apologize to anyone affected by this.
Update: We have added a new automated step to our certificate installation process that checks for problems with SSL certificates, guaranteeing that this problem will not recur.
Between 4:58 and 5:39 AM Pacific time today (March 23), our server which runs the Mailman mailing list software encountered an internal problem. During most of this time, all Mailman-related functionality was unavailable.
Since Mailman most works via e-mail, no data was lost. Some messages might have been slightly delayed, but not for any longer than might normally be noticed with mail delivery via the Internet.
We apologize for any inconvenience that this might have caused!
As you probably know, we back up all of our servers every night. Our goal is to keep at least seven days of backups available, with additional older backups available where possible.
We’ve added a new page to our control panel showing the backups available for each account. To see it, just login to the control panel, then click Backups.
We make backups so that we can recover from unexpected occurrences such as data erasure or server failures. Of course, we also make the backups available to customers, because they can be a real life-saver when you need one. However, we want to remind customers that they should also make their own backups, especially if you need a different backup frequency or retention policy. Our Web page describing our backup system and policies has more details.
The “calculon” Web server needed to be restarted at 10:14 AM Pacific time, resulting in a five-minute interruption of service for Web sites and e-mail on that server.
One of the features of our new(ish) Webmail system is “thread view”. This groups similar messages together based on their “Subject” and other headers, which can occasionally be useful if you’re trying to see all the replies to a particular message and you want them grouped together.
However, thread view has a potential downside: it you have several active threads going with several messages each, new messages can sometimes appear on the second page of the incoming mail screens, instead of the first page.
That’s not a problem if you’re expecting it. However, since we introduced the new Webmail system, we’ve had several complaints from customers who accidentally clicked “Switch to Thread View” without realizing what it does, then thought some of their incoming mail was missing because they aren’t used to looking for new mail on other pages. Since thread view is “remembered” even after you logout and login again, this caused some people a great deal of heartache.
From our logs, we’ve found that very few people actually use thread view. Because it seems to cause frequent problems and few people use it, we’ve made it an optional feature instead of being always enabled.
If (like most people) you don’t use thread view, you don’t need to do anything. If do you want to use thread view, it’s still available: just click “Preferences”, then click “Display Preferences”, then change “Show ‘Thread View’ Link” to “Yes”.
