Although all WordPress users should upgrade right away, we’ve also added a security rule to our servers to try and protect our customers who haven’t yet upgraded. Other people may also find the security rule useful if they use mod_security on Apache Web servers. The rest of this post contains more technical details.

So what’s the security problem with WordPress 2.5? Well, if your blog allows strangers to register, the vulnerability allows evildoers to create a new unprivileged username like “admin0″, then use the cookie from that username to login as the privileged username “admin”.

This works because WordPress creates a login cookie with a value like this for “admin0″:

admin0|1209331453|2a771ff005c67b2aaa0a872aaa213f39

The first part is the username, the second part is the cookie’s expiration time, and the third part is an MD5 hash of the concatenated username, expiration, and some secret text.

The trouble is that this cookie is also valid:

admin|01209331453|2a771ff005c67b2aaa0a872aaa213f39

We simply moved the “0″ from the end of “admin” to the beginning of the expiration. Although this changes the username, it doesn’t change the MD5 hash! That’s because the text that the hash is based on doesn’t contain a separator such as a pipe character, as it should. In either case, it’s just hashing “admin01209331453″ (plus the same secret text), and the identical hash is valid for both the privileged and unprivileged users. A nasty bug indeed.

You don’t necessarily need to use “0″ on the end of the username, either; other digits can be made to work. They increase the expiration time of the cookie, but that doesn’t make it invalid.

So this vulnerability allows people to create a WordPress username ending in one or more digits, then use the cookie from that username to login as another username without the digits on the end.

Here’s where mod_security can be useful. It’s a great defensive tool that we’ve been using more and more to protect our customers from this kind of attack. If it’s possible to identify something that’s different about an “evil” page request (compared to a “good” page request), you can probably block the evil request using mod_security.

In this case, one thing immediately stands out: a “good” request always has a 10 digit expiration time in the cookie, but a “bad” request always has 11 or more digits. If we can block WordPress cookies that contain 11 or more digit expirations, that should block the evil request.

This rule for mod_security 1.x does the trick:

SecFilterSelective HTTP_Cookie "wordpress_[a-f0-9]{32}=[^\|]+\|[0-9]{11,}\|[a-f0-9]{32}"

And here’s the same rule for mod_security 2.x:

SecRule REQUEST_HEADERS:Cookie "wordpress_[a-f0-9]{32}=[^\|]+\|[0-9]{11,}\|[a-f0-9]{32}"

This looks for a cookie value containing 11 or more digits between the pipe symbols. This rule successfully blocks a test attack we created, and does not appear to give false positives (we’ve had it in place on servers that have served several million pages over the last few hours, with no matches beyond our tests).

Just so it’s clear, our customers don’t need to use these rules, because they’re already on all our servers — but we hope they’re useful to someone else.

Of course, if you know of any other attack vectors, please post a comment so we can improve the rules.

We’ll get to the problems below, but first, let’s consider how Rails versions work. One nice Rails feature is that there can be multiple versions of Rails on the server, and you can choose to “bind” your application to a particular Rails version. In fact, new Rails apps are “bound” to the default Rails version that’s on the server when you create the app, which is good for stability. Even when we update Rails, your app continues using the same old version until you change it.

You actually get even more control than that: you can “freeze” your Rails application to any version you like — even versions newer than the ones on the server. That’s the main reason we haven’t been in a hurry to update to Rails 2 systemwide; you can easily update your Rails app to 2.0.2 now if you want.

So if you’ve created your own app, and you haven’t removed the configuration line that binds Rails to a particular server version, nothing we do will affect you. Until you change your Rails version, you’ll continue using 1.2.6 (or whatever version you’ve chosen). However, some people may have installed third-party Rails apps that aren’t bound to a particular version and don’t have a frozen copy of Rails, and those would be affected by an upgrade.

Which brings us to the problems we saw when upgrading some old test applications. After changing the Rails version, they started displaying a “500 Internal Server Error” message. The Rails log file shows:

Status: 500 Internal Server Error
A secret is required to generate an integrity hash for cookie session data. Use config.action_controller.session = { :session_key => "_myapp_session", :secret => "some secret phrase of at least 30 characters" } in config/environment.rb


Hmmm, all right. So we need to add a line like this to the “Rails::Initializer.run” section of the “config.rb” file, with some random text like so:

config.action_controller.session = { :session_key => "_myapp_session", :secret => "b836b5a8578d3c74a5dbd0956102784f864e7b67" }


That fixed the 500 error. (By the way, if you’re looking for a good way to generate random strings, try “head /dev/urandom | sha1sum” on a command line.)

We then had a couple of other minor issues. The first was that Rails 2 really doesn’t want to work without a valid database. If your Rails app doesn’t use a database, that’s apparently “not a supported configuration” and you’ll need to either create a fake database or make some code changes. The second issue was that some things that have been deprecated for some time (such as “render_text”) have been completely removed; if you’ve been putting off making those changes, now’s the time.

Anyway, if you’re using Rails, you probably want to test your application for compatibility now. You can do that at any time by “freezing” it to version 2.0.2. If it doesn’t work, just freeze it back to 1.2.6 while you fix the problems.

(We’ll post another blog entry when we actually upgrade Rails, of course.)

We’ve added a new page to our control panel showing the backups available for each account. To see it, just login to the control panel, then click Backups.

We make backups so that we can recover from unexpected occurrences such as data erasure or server failures. Of course, we also make the backups available to customers, because they can be a real life-saver when you need one. However, we want to remind customers that they should also make their own backups, especially if you need a different backup frequency or retention policy. Our Web page describing our backup system and policies has more details.

The updates were performed in such a way that new Web server connections were delayed during the 30 seconds or so that PHP and MySQL were unavailable on each server. That should mean that as far as scripts on your Web site were concerned, there was zero downtime.

Since this is “Tech Corner”… If you’re curious how to do “zero downtime” upgrades on Linux, you can use iptables. Something like this works on Debian for MySQL:

iptables -I INPUT -p tcp -m multiport --syn --dports 80,443 -j DROP
apt-get install mysql-server-5.0
iptables -D INPUT -p tcp -m multiport --syn --dports 80,443 -j DROP

The dropping of SYN packets blocks new Web connections (and therefore new MySQL queries) without interrupting existing ones. Since we use DROP instead of REJECT, the client’s Web browser doesn’t show an immediate error; it keeps retrying.

The upgrade process then takes about 15 seconds to actually stop MySQL (assuming you’ve already downloaded the package), which is enough time for existing connections to finish up. It then takes another 15 seconds or so to install the upgrade and restart MySQL.

Finally, removing the iptables rule allows the pending Web connections to succeed. Since the whole thing took only about 30 seconds, it shouldn’t cause a browser error — to the visitor, it just seems like a page took longer than normal to display. A slow page load is obviously an annoyance (and we normally only do this kind of upgrade in the wee hours of weekends to minimize that annoyance), but it’s much better than a visitor receiving a “database connection failed” error when they try to place an order on your online store, etc.

By the way, the above example doesn’t cover the possibility that MySQL connections could be opened by non-Web processes like cron jobs, and it doesn’t handle connections that take longer than 15 seconds to finish. You could automate this with a little shell scripting that queries MySQL to see if all connections are closed, but we just watch it from a different terminal window to make sure.

This problem has been resolved and all services are now available. We are waiting for a full report from the data center personnel so that we can determine the cause and ensure that it won’t recur.

We sincerely apologize to our customers who were affected by this. This kind of outage is not normal (it’s the longest outage we’ve experienced in more than four years), and we know it’s not acceptable to our customers who rely on our services. We’ll post a followup message with more details when they become available.

Update Friday 10 AM: As a clarification, we should also have originally mentioned that no e-mail is lost during this kind of outage: it’s delivered after the issue is resolved. While some messages were certainly delayed, they were all properly delivered afterward.

Locales are useful, because when they’re available, a script can easily generate things like dates in different languages. For example, this script:

<?php
  echo strftime("%A %E %B %Y");
?>

… will normally print “Tuesday 8 January 2008″. But if the French locale is available, adding one line:

<?php
  setlocale(LC_ALL, 'fr_FR');
  echo strftime("%A %E %B %Y");
?>

… will print “mardi 8 janvier 2008″ instead.

So having many locales available on a server makes it easy for scripts to show dates and other information in ways beyond the standard “US English” display.

As of today, we’ve installed many extra locales on our servers for English, French, German, Italian and Spanish languages. (All these languages have more than one actual locale, because the English language, for example, uses different currency symbols and date formats depending on which country you’re in.) We hope some customers find this useful.

For more information about changing locales in a script, see the “setlocale()” command in your favorite programming language. Advanced users can get a full list of the available locales on the server by making a shell connection and typing “locale -a”.

We found and fixed the root cause of the problem, and it won’t occur again.

We apologize for any inconvenience this may have caused. We know that no one wanted more spam for Christmas!

Table upgrade required. Please do "REPAIR TABLE `users`" to fix it!

Hmm. Well, that error message seems odd, but what the heck. After making sure there’s a current backup, let’s give it a shot:

mysql> REPAIR TABLE users;

+---------------------+--------+----------+-----------------------------+
| Table               | Op     | Msg_type | Msg_text                    |
+---------------------+--------+----------+-----------------------------+
| database.users      | repair | status   | Table is already up to date |
+---------------------+--------+----------+-----------------------------+

And yet, CHECK TABLE still says there’s a problem:

mysql> CHECK TABLE users;

+---------------------+-------+----------+-------------------------------------+
| Table               | Op    | Msg_type | Msg_text                            |
+---------------------+-------+----------+-------------------------------------+
| database.users      | check | error    | Table upgrade required. Please do   |
|                     |       |          |  "REPAIR TABLE `users`" to fix it!  |
+---------------------+-------+----------+-------------------------------------+

Hmmm. We tried every reasonable combination of REPAIR TABLE and myisamchk options we could think of, but nothing helped. Then we tried unreasonable options, including the frightening REPAIR TABLE table USE_FRM option. In our experience USE_FRM usually ends up doing the same thing as DELETE FROM table, and this time was no exception. (And that’s why you make a backup first.)

Well, let’s take a look at the table status (with the uninteresting bits removed):

mysql> show table status;

+-------+--------+---------+-----+---------------------+
| Name  | Engine | Version | ... | Create_time         |
+-------+--------+---------+-----+---------------------+
| good  | MyISAM |      10 | ... | 2007-06-08 09:44:01 |
| users | MyISAM |       7 | ... | 2005-03-15 12:03:11 |
+-------+--------+---------+-----+---------------------+

Sure enough, the bad “users” table is a different version than a good table that doesn’t have this problem. And the bad table was created when we were still using MySQL 4.1.x. This table should have been changed when we upgraded to MySQL 5, and it should be possible to fix now. Why on earth won’t REPAIR TABLE fix it?

We puzzled over this for some time, and eventually just decided to reload it from a dump. But here’s the strange thing: it wouldn’t reload from the dump, either!

ERROR 1060 (42S21) at line 20: Duplicate column name 'user_id'

What the heck? So we looked closely at the dump file, and it somehow contains this line:

PRIMARY KEY (user_id,user_id)

MySQL thinks this table has a composite (two-column) primary key, but on the same column twice, which doesn’t make any sense. So we fix the dump to look like this:

PRIMARY KEY (user_id)

And everything works perfectly when the table is reloaded.

So if CHECK TABLE tells you a table upgrade is required, but REPAIR TABLE won’t fix it, check whether you can actually reload a dump of that table. You might find something unusual and easily fixable in the dump.

The update was performed in such a way that new Web server connections were delayed during the 30 seconds or so that MySQL was unavailable on each server. That should mean that as far as scripts on your Web site were concerned, there was zero MySQL downtime.

• Ruby on Rails was updated from version 1.2.3 to 1.2.6. (If you use Rails on your site, our page explaining how to freeze Rails explains how you can get total control of Rails updates.)
• phpMyAdmin was updated from version 2.11.2.1 to 2.11.2.2.
• The WordPress software that runs this blog was updated to version 2.3.1. That doesn’t directly affect our customers — but if you’ve installed your own version of WordPress on your own site, this is a good reminder to update it: some older versions have security vulnerabilities. (We found that the update from 2.2.X to 2.3.1 was painless.)