Customer Web sites on the calculon, lrrr, and zapp Web servers were unavailable during this time. The ability to send and receive e-mail was also interrupted (no mail was lost, of course). Other servers were not affected.

We pay close attention to the power load in each cabinet to avoid this sort of problem. The previously measured peak load of that cabinet had been 12 amps. Since the circuit allows 15 amps, this issue surprised us (we’ve been using the same setup in the same data center for seven years and this has never happened before). It appears that a combination of several servers experiencing unusually high CPU loads led to power usage beyond what we previously considered possible.

We will take immediate steps to make sure the problem doesn’t happen again, and we sincerely apologize to customers who were affected by this incident.

Update 7:26 PM: We have removed a server from the cabinet in question, lowering the power use.

Update 10:38 PM: We have removed a second server from the cabinet, ensuring that power use is well below any level that could cause further trouble. The problem will not recur.

Although all WordPress users should upgrade right away, we’ve also added a security rule to our servers to try and protect our customers who haven’t yet upgraded. Other people may also find the security rule useful if they use mod_security on Apache Web servers. The rest of this post contains more technical details.

So what’s the security problem with WordPress 2.5? Well, if your blog allows strangers to register, the vulnerability allows evildoers to create a new unprivileged username like “admin0″, then use the cookie from that username to login as the privileged username “admin”.

This works because WordPress creates a login cookie with a value like this for “admin0″:

admin0|1209331453|2a771ff005c67b2aaa0a872aaa213f39

The first part is the username, the second part is the cookie’s expiration time, and the third part is an MD5 hash of the concatenated username, expiration, and some secret text.

The trouble is that this cookie is also valid:

admin|01209331453|2a771ff005c67b2aaa0a872aaa213f39

We simply moved the “0″ from the end of “admin” to the beginning of the expiration. Although this changes the username, it doesn’t change the MD5 hash! That’s because the text that the hash is based on doesn’t contain a separator such as a pipe character, as it should. In either case, it’s just hashing “admin01209331453″ (plus the same secret text), and the identical hash is valid for both the privileged and unprivileged users. A nasty bug indeed.

You don’t necessarily need to use “0″ on the end of the username, either; other digits can be made to work. They increase the expiration time of the cookie, but that doesn’t make it invalid.

So this vulnerability allows people to create a WordPress username ending in one or more digits, then use the cookie from that username to login as another username without the digits on the end.

Here’s where mod_security can be useful. It’s a great defensive tool that we’ve been using more and more to protect our customers from this kind of attack. If it’s possible to identify something that’s different about an “evil” page request (compared to a “good” page request), you can probably block the evil request using mod_security.

In this case, one thing immediately stands out: a “good” request always has a 10 digit expiration time in the cookie, but a “bad” request always has 11 or more digits. If we can block WordPress cookies that contain 11 or more digit expirations, that should block the evil request.

This rule for mod_security 1.x does the trick:

SecFilterSelective HTTP_Cookie "wordpress_[a-f0-9]{32}=[^\|]+\|[0-9]{11,}\|[a-f0-9]{32}"

And here’s the same rule for mod_security 2.x:

SecRule REQUEST_HEADERS:Cookie "wordpress_[a-f0-9]{32}=[^\|]+\|[0-9]{11,}\|[a-f0-9]{32}"

This looks for a cookie value containing 11 or more digits between the pipe symbols. This rule successfully blocks a test attack we created, and does not appear to give false positives (we’ve had it in place on servers that have served several million pages over the last few hours, with no matches beyond our tests).

Just so it’s clear, our customers don’t need to use these rules, because they’re already on all our servers — but we hope they’re useful to someone else.

Of course, if you know of any other attack vectors, please post a comment so we can improve the rules.

We’ll get to the problems below, but first, let’s consider how Rails versions work. One nice Rails feature is that there can be multiple versions of Rails on the server, and you can choose to “bind” your application to a particular Rails version. In fact, new Rails apps are “bound” to the default Rails version that’s on the server when you create the app, which is good for stability. Even when we update Rails, your app continues using the same old version until you change it.

You actually get even more control than that: you can “freeze” your Rails application to any version you like — even versions newer than the ones on the server. That’s the main reason we haven’t been in a hurry to update to Rails 2 systemwide; you can easily update your Rails app to 2.0.2 now if you want.

So if you’ve created your own app, and you haven’t removed the configuration line that binds Rails to a particular server version, nothing we do will affect you. Until you change your Rails version, you’ll continue using 1.2.6 (or whatever version you’ve chosen). However, some people may have installed third-party Rails apps that aren’t bound to a particular version and don’t have a frozen copy of Rails, and those would be affected by an upgrade.

Which brings us to the problems we saw when upgrading some old test applications. After changing the Rails version, they started displaying a “500 Internal Server Error” message. The Rails log file shows:

Status: 500 Internal Server Error
A secret is required to generate an integrity hash for cookie session data. Use config.action_controller.session = { :session_key => "_myapp_session", :secret => "some secret phrase of at least 30 characters" } in config/environment.rb


Hmmm, all right. So we need to add a line like this to the “Rails::Initializer.run” section of the “config.rb” file, with some random text like so:

config.action_controller.session = { :session_key => "_myapp_session", :secret => "b836b5a8578d3c74a5dbd0956102784f864e7b67" }


That fixed the 500 error. (By the way, if you’re looking for a good way to generate random strings, try “head /dev/urandom | sha1sum” on a command line.)

We then had a couple of other minor issues. The first was that Rails 2 really doesn’t want to work without a valid database. If your Rails app doesn’t use a database, that’s apparently “not a supported configuration” and you’ll need to either create a fake database or make some code changes. The second issue was that some things that have been deprecated for some time (such as “render_text”) have been completely removed; if you’ve been putting off making those changes, now’s the time.

Anyway, if you’re using Rails, you probably want to test your application for compatibility now. You can do that at any time by “freezing” it to version 2.0.2. If it doesn’t work, just freeze it back to 1.2.6 while you fix the problems.

(We’ll post another blog entry when we actually upgrade Rails, of course.)

Web sites using scripts or databases on the elzar server may have seemed unresponsive during that time. Also, any customer hosted on elzar who was reading their e-mail during this time may have felt the system was slow or unresponsive (no e-mail was lost, of course).

Customers on other servers were not affected.

The problem was related to an extremely high number of Apache Web server processes running. We’ve changed some Apache configuration settings (lowering the maximum number of Apache processes and reducing some timeout settings so that fewer processes are required in the first place) on all our servers. We expect those change will prevent this symptom from happening again in the future.

We apologize for any inconvenience that this may have caused.

The problem occurred when the Apache web server process failed to gracefully restart when a new SSL certificate was added. We have discovered why this happened and will take steps to prevent it in the future.

We sincerely apologize to anyone affected by this.

Since Mailman most works via e-mail, no data was lost. Some messages might have been slightly delayed, but not for any longer than might normally be noticed with mail delivery via the Internet.

We apologize for any inconvenience that this might have caused!

We’ve added a new page to our control panel showing the backups available for each account. To see it, just login to the control panel, then click Backups.

We make backups so that we can recover from unexpected occurrences such as data erasure or server failures. Of course, we also make the backups available to customers, because they can be a real life-saver when you need one. However, we want to remind customers that they should also make their own backups, especially if you need a different backup frequency or retention policy. Our Web page describing our backup system and policies has more details.

The problem was caused while debugging high bandwidth usage on that server; running the Linux “iptraf” command caused the server to instantly lose network connectivity for some reason. We will not use that command again, obviously.

We apologize to anyone affected by this problem.

However, thread view has a potential downside: it you have several active threads going with several messages each, new messages can sometimes appear on the second page of the incoming mail screens, instead of the first page.

That’s not a problem if you’re expecting it. However, since we introduced the new Webmail system, we’ve had several complaints from customers who accidentally clicked “Switch to Thread View” without realizing what it does, then thought some of their incoming mail was missing because they aren’t used to looking for new mail on other pages. Since thread view is “remembered” even after you logout and login again, this caused some people a great deal of heartache.

From our logs, we’ve found that very few people actually use thread view. Because it seems to cause frequent problems and few people use it, we’ve made it an optional feature instead of being always enabled.

If (like most people) you don’t use thread view, you don’t need to do anything. If do you want to use thread view, it’s still available: just click “Preferences”, then click “Display Preferences”, then change “Show ‘Thread View’ Link” to “Yes”.

The problem was caused by a faulty hard disk in the RAID array (which theoretically shouldn’t cause a server to stop responding, but did). The hard disk has been removed from the array and will be replaced tonight at 10 PM. The server will be restarted at that time, resulting in about 4 minutes additional downtime.

We sincerely apologize for this problem. We will be investigating the root cause: it’s normal for hard drives to fail — we expect that occasionally — but it shouldn’t cause such negative effects (normally the RAID array would prevent the failure of any single drive from causing the entire machine to fail).