Our automated monitoring systems didn’t detect any general problems, so the majority of customers were certainly unaffected — but we suspect that one of the “Internet backbones” between the affected customers and our data center had high packet loss during that period.

Both customers reported that the problem resolved itself by 7:45, and we haven’t received similar reports since, so there does not appear to be be an ongoing problem. We’ll continue to monitor it closely.

Those runaway queries can slow down the MySQL server for many minutes on end, causing performance problems.

To prevent the worst of that, we’ve set the max_join_size setting to 1,000,000 on our MySQL servers.

This shouldn’t affect any normal queries, but queries that attempt to sort more than a million rows will now return this error:

ERROR 1104 (42000): The SELECT would examine more than MAX_JOIN_SIZE rows; check your WHERE and use SET SQL_BIG_SELECTS=1 or SET SQL_MAX_JOIN_SIZE=# if the SELECT is okay

If you have any questions, please contact our support staff.

If you care about the performance of your site, please stick with WP Super Cache unless you have a very good reason to switch. It works, and it works well.

Some people tell us that W3 Total Cache works just as well if it’s properly configured, and they may well be right — but it seems like it’s difficult to configure properly. Our experience is showing that it’s easy to get wrong, and performance ends up suffering. WP Super Cache makes it easy to get great performance.

Other servers will not be affected, and incoming mail will only be delayed, not lost.

We apologize for the problem and for the short notice: the restart is necessary to replace a disk in the RAID array.

Update 11:03 PM Pacific time: The restart was completed with less than 3 minutes “downtime”.

This server had similar high load symptoms (but much more briefly) earlier this week. We took some steps to reduce the load then, but it appears those weren’t sufficient. We’re now taking much stronger action to ensure that this does not happen again.

We sincerely apologize to customers affected by this problem. We don’t consider it normal or acceptable, and we will make sure this isn’t a recurring issue.

We’ve investigated this, and it’s caused by Joomla assuming that PHP has a bug that makes it work incorrectly, when in fact it’s supposed to work differently (and is clearly documented to work differently). Older versions of PHP had this bug, but the new version doesn’t.

To help our customers work around this, we’ve “patched” PHP to intentionally reintroduce the old bug for now, thus keeping it “compatible” with Joomla. If you were having trouble with Joomla’s “Search Engine Friendly URLs”, it should be fixed.

We’ll provide more technical details (and a more robust long-term solution) in the near future.

Update: We’ve also reported this problem to the Joomla developers and suggested a solution.

• Thursday, February 4 (servers beginning with the letter “L”, such as “lrrr”)
• Friday, February 5 (all other servers beginning with letters “F-Z”, such as “farnsworth”)
• Saturday, February 6 (servers beginning with letters “A-E”, such as “amy”)

What’s changing?

Our servers use the Debian GNU/Linux operating system, which is updated with new versions every couple of years (just like Windows or Mac OS X). Last year, Debian was updated to version 5.0 (aka “lenny”), and since then, we’ve been running a “mixed” system. Some programs on each server have already been updated, but other programs still need updating.

We’re now finalizing the upgrade. This will make sure we can continue to use Debian security updates (they’ll soon be provided only for version 5.0) to keep our systems secure, and will give customers access to newer versions of some programs. Among the many updates:

• The Apache Web server will be upgraded from version 2.2.3 to 2.2.9
• The MySQL database server will be upgraded from 5.0.32 to 5.0.51a
• The PHP scripting language will be upgraded from 5.2.0 to 5.2.6
• The Perl scripting language will be upgraded from 5.8.8 to 5.10.0
• The default version of the Python scripting language will change from 2.4.4 to 2.5.2

Sharp-eyed readers will notice that these are not the newest, “cutting-edge” versions available. This is intentional; the philosophy of Debian Linux (and of our hosting platform) is to only use extremely well-tested, tried-and-true versions. We strongly value stability over new features, within reason, because that improves reliability for the majority of our customers.

However, we also want to emphasize that these versions also include security updates from later versions — for example, PHP includes the features from 5.2.6, but also includes security fixes from versions 5.2.7 to 5.2.12.

Why the downtime?

Unfortunately, a side-effect of the upgrade is about 7 total minutes of downtime.

We always strive to avoid any downtime, but in this case, it’s unavoidable: There’s a four-minute outage while the server is restarted for the new kernel version, and three outages of about a minute each while Apache, PHP, and MySQL are upgraded. During those three periods, we disable access to the Web server to prevent strange technical error messages appearing on your visitor’s screens.

Incoming mail may be delayed for a few minutes during the maintenance, but as always, no mail will be lost: it will just be queued and delivered a few minutes later.

Questions?

Please contact our support team if you have any questions. We appreciate your business!

Update: The maintenance was completed as scheduled.

But recently we’ve seen a number of our customers getting hammered by a ton of requests from FeedBurner. Usually the request is of this form:

/somepost?utm_source=feedburner&utm_medium=feed&utm_campaign=SomeCampaignString

We’ve also seen FeedBurner going crazy and making thousands of duplicate requests. One of the sites we host has gotten 45,000 simple status requests (HTTP “HEAD” requests) from FeedBurner today, for no good reason that we can see.

Unfortunately, the default rules of WP Super Cache prevent it from caching any request with a query that contains an equal sign. So all of these requests are being unnecessarily run freshly each time, rather than being served from the cache.

There’s an easy fix to this. Open your Web site’s .htaccess file. Look for the section of lines for WP Super Cache, and find the line which tests %{QUERY_STRING}. Insert this new line of text immediately above the existing line:

RewriteCond %{QUERY_STRING} ^utm_source=(feedburner|twitterfeed) [OR]

The new line (ending with [OR]) must come before the existing %{QUERY_STRING} line. After inserting, the two lines should look exactly like this:

RewriteCond %{QUERY_STRING} ^utm_source=(feedburner|twitterfeed) [OR]
RewriteCond %{QUERY_STRING} !.*=.*

There are two very similar blocks right next to each other in the .htaccess file. Be sure to add the new line to the same place in each block.

That’s the only change you’ll have to make! WP Super Cache will now be able to cache requests for normal pages that come from FeedBurner or Twitterfeed. If your Web site was being abused by FeedBurner (or Twitterfeed), you should see a definite improvement.

In some cases, you might be storing data that’s not quite so important. And if it means your application can run much faster, you might be willing to risk a very small chance of data loss. That’s where MySQL’s “INSERT DELAYED” statement, which works with MyISAM table types (but not InnoDB tables), can be useful. (Tables are created as type MyISAM by default, so most tables are eligible to benefit from this tip.)

Adding the word “DELAYED” to your statement tells MySQL to remember the data to be added and return immediately to your application. MySQL will then write the data as soon as the database isn’t busy. This lets your insertion happen (effectively) immediately, and reduces the load on the database (and server). Using this technique can give your application a huge performance gain.

Since the record is not written immediately, there is a very small chance that the data will be lost before it’s written to the disk. However, the odds of this happening are very small. It would only happen if MySQL crashed before it had a moment of idle time to write out the record, if the server lost power, or if some similar unexpected event happened.

Using “INSERT DELAYED” is recommended for applications that do not absolutely depend upon records being immediately written. Any data which is only referred to at a later time (rather than on-demand) or in a summary fashion is a good candidate for using “INSERT DELAYED”. Some examples are logging the IP address of each visitor, or tracking ad impressions for each page viewed on your Web site. Be sure to read the official documentation for full details and additional considerations.

If you want to use “INSERT DELAYED”, first check the documentation to see if your application already supports it. If it does not, you’d need to modify the application. If you have any questions about whether a particular database table is a good candidate for using “INSERT DELAYED”, just ask us and we’ll take a look.