If you care about the performance of your site, please stick with WP Super Cache unless you have a very good reason to switch. It works, and it works well.

Some people tell us that W3 Total Cache works just as well if it’s properly configured, and they may well be right — but it seems like it’s difficult to configure properly. Our experience is showing that it’s easy to get wrong, and performance ends up suffering. WP Super Cache makes it easy to get great performance.

We’ve investigated this, and it’s caused by Joomla assuming that PHP has a bug that makes it work incorrectly, when in fact it’s supposed to work differently (and is clearly documented to work differently). Older versions of PHP had this bug, but the new version doesn’t.

To help our customers work around this, we’ve “patched” PHP to intentionally reintroduce the old bug for now, thus keeping it “compatible” with Joomla. If you were having trouble with Joomla’s “Search Engine Friendly URLs”, it should be fixed.

We’ll provide more technical details (and a more robust long-term solution) in the near future.

Update: We’ve also reported this problem to the Joomla developers and suggested a solution.

• Thursday, February 4 (servers beginning with the letter “L”, such as “lrrr”)
• Friday, February 5 (all other servers beginning with letters “F-Z”, such as “farnsworth”)
• Saturday, February 6 (servers beginning with letters “A-E”, such as “amy”)

What’s changing?

Our servers use the Debian GNU/Linux operating system, which is updated with new versions every couple of years (just like Windows or Mac OS X). Last year, Debian was updated to version 5.0 (aka “lenny”), and since then, we’ve been running a “mixed” system. Some programs on each server have already been updated, but other programs still need updating.

We’re now finalizing the upgrade. This will make sure we can continue to use Debian security updates (they’ll soon be provided only for version 5.0) to keep our systems secure, and will give customers access to newer versions of some programs. Among the many updates:

• The Apache Web server will be upgraded from version 2.2.3 to 2.2.9
• The MySQL database server will be upgraded from 5.0.32 to 5.0.51a
• The PHP scripting language will be upgraded from 5.2.0 to 5.2.6
• The Perl scripting language will be upgraded from 5.8.8 to 5.10.0
• The default version of the Python scripting language will change from 2.4.4 to 2.5.2

Sharp-eyed readers will notice that these are not the newest, “cutting-edge” versions available. This is intentional; the philosophy of Debian Linux (and of our hosting platform) is to only use extremely well-tested, tried-and-true versions. We strongly value stability over new features, within reason, because that improves reliability for the majority of our customers.

However, we also want to emphasize that these versions also include security updates from later versions — for example, PHP includes the features from 5.2.6, but also includes security fixes from versions 5.2.7 to 5.2.12.

Why the downtime?

Unfortunately, a side-effect of the upgrade is about 7 total minutes of downtime.

We always strive to avoid any downtime, but in this case, it’s unavoidable: There’s a four-minute outage while the server is restarted for the new kernel version, and three outages of about a minute each while Apache, PHP, and MySQL are upgraded. During those three periods, we disable access to the Web server to prevent strange technical error messages appearing on your visitor’s screens.

Incoming mail may be delayed for a few minutes during the maintenance, but as always, no mail will be lost: it will just be queued and delivered a few minutes later.

Questions?

Please contact our support team if you have any questions. We appreciate your business!

Update: The maintenance was completed as scheduled.

In some cases, you might be storing data that’s not quite so important. And if it means your application can run much faster, you might be willing to risk a very small chance of data loss. That’s where MySQL’s “INSERT DELAYED” statement, which works with MyISAM table types (but not InnoDB tables), can be useful. (Tables are created as type MyISAM by default, so most tables are eligible to benefit from this tip.)

Adding the word “DELAYED” to your statement tells MySQL to remember the data to be added and return immediately to your application. MySQL will then write the data as soon as the database isn’t busy. This lets your insertion happen (effectively) immediately, and reduces the load on the database (and server). Using this technique can give your application a huge performance gain.

Since the record is not written immediately, there is a very small chance that the data will be lost before it’s written to the disk. However, the odds of this happening are very small. It would only happen if MySQL crashed before it had a moment of idle time to write out the record, if the server lost power, or if some similar unexpected event happened.

Using “INSERT DELAYED” is recommended for applications that do not absolutely depend upon records being immediately written. Any data which is only referred to at a later time (rather than on-demand) or in a summary fashion is a good candidate for using “INSERT DELAYED”. Some examples are logging the IP address of each visitor, or tracking ad impressions for each page viewed on your Web site. Be sure to read the official documentation for full details and additional considerations.

If you want to use “INSERT DELAYED”, first check the documentation to see if your application already supports it. If it does not, you’d need to modify the application. If you have any questions about whether a particular database table is a good candidate for using “INSERT DELAYED”, just ask us and we’ll take a look.

Many other companies still run their servers on slower 100 Mbps Ethernet connections, leading to slow speeds for all sites on a busy server that experiences a sudden burst of traffic. We don’t have that limitation, which is why we can now allow individual sites to “burst” up to 100 Mbps without affecting other sites on the same server.

Of course, good network performance means more than just raw throughput — low latency (”ping time”) is also important.

If you think of a network as being like a road, “throughput” is [sort of] the measure of how many cars the road can handle per second, and “latency” is [sort of] the time it takes an individual car to get from one end of the road to another. It’s no good having a wide, gridlocked road that can handle lots of cars at 5 miles an hour, and it’s no good having a road that allows only a small handful of fast-moving cars.

Our network has both high throughput and low latency. For example, I just transferred a large file from kernel.org to one of our hosting servers at over 180 Mbps, and a quick ping test shows our latency to www.yahoo.com and www.google.com is under 3 ms:

$ ping -n -c 3 www.yahoo.com
PING www-real.wa1.b.yahoo.com (209.131.36.158) 56(84) bytes of data.
64 bytes from 209.131.36.158: icmp_seq=1 ttl=59 time=1.71 ms
64 bytes from 209.131.36.158: icmp_seq=2 ttl=59 time=1.69 ms
64 bytes from 209.131.36.158: icmp_seq=3 ttl=59 time=1.80 ms

$ ping -n -c 3 www.google.com
PING www.l.google.com (74.125.19.105) 56(84) bytes of data.
64 bytes from 74.125.19.105: icmp_seq=1 ttl=59 time=2.22 ms
64 bytes from 74.125.19.105: icmp_seq=2 ttl=59 time=1.80 ms
64 bytes from 74.125.19.105: icmp_seq=3 ttl=59 time=1.91 ms


By the way, if you try your own speed tests (or try ours), be sure to notice if they’re measuring the speed in megabits per second (Mbps), megabytes per second (MBps), kilobits per second (Kbps), or kilobytes per second (KBps).

Those are easy to mix up, and we sometimes hear complaints like “My Internet connection is supposed to be 1500 Kbps (1.5 Mbps), but my FTP program says I’m only getting 187 KBps.” Those are all the same speeds, just measured in different units.

That advice makes sense if you have a sudden surge of traffic to a single page. However, if your site is generally very busy across all pages (for example, if you have an archive of hundreds or thousands of posts that are constantly being indexed by search engines), we’ve found that you should do the opposite to improve performance: set it much higher. We recommend setting it to 172800 seconds (which is 48 hours). This can cut your CPU usage in half, which will speed up your site.

The reason for this is that when WP Super Cache creates a cached page, it wants to make sure that those pages don’t build up forever. Every ten minutes or so, it looks through them all and deletes any that are older than the “expire time”.

On some servers that use a network file system called “NFS”, looking through a large number of files causes performance problems. That’s why the WP Super Cache author recommends making them expire quickly: it reduces the number of files it has to examine each time.

On our servers, we don’t use NFS and looking through lots of files does not cause a performance problem. Leaving the files for a longer time is safe and increases the chance that a page will already be cached when it’s needed.

If you’re a Tiger Technologies customer who makes this change and you want to see how it affects the CPU usage, just let us know and we can provide you with details.

First of all, the problem: Our primary internal PostgreSQL database server (which we use to manage all customer data) completely failed due to a nasty problem: a program on the server tried to delete every file on the disk.

The way this happened was surprising. A long time ago, our database server had a certain Debian GNU/Linux software “package” installed that — somehow — erroneously created a user with a “home directory” set to the top level of the entire disk (”/”). This caused no harm for almost two years… until we removed the obsolete, unused package while cleaning up odds and ends. The automatic package uninstallation script then began to delete all the files in the “/” directory. The script deleted several important files, including some necessary to “boot” that server, before we stopped it.

When the worst happens, we have a plan to recover from database failures:

• If the server hardware somehow fails (despite having redundant hard drives, power supplies, etc.), we can move the data disks to a spare server. This can be done in a few minutes.
• If the database files are corrupted or unavailable, we can restore from the correct onsite or offsite hourly backup, re-apply changes made since that backup (either manually or through database “journal logs”), and make it available again. This takes longer and is more tedious and error-prone.

In this case, we knew that some of the files on the server had been deleted, but it wasn’t clear which ones. That caused a significant delay while we contemplated whether it was safe to use the files of the existing database, or whether we needed to restore the database from backups. We eventually determined that it was safe to use the existing data, and restored things to a working state on a spare server.

Unfortunately, it took longer than we’d like to make sure all our systems that rely on databases were fully functional, and there were several things that we’d like to have avoided even if “My Account” was temporarily not working:

• The server problem also made our own www.tigertech.net Web site completely stop working for a short time.
• Our blog.tigertech.net pages relied on the same database server, so we couldn’t communicate the status well.
• Some people saw error messages when using the Webmail system to read certain messages. This shouldn’t happen; the Webmail system is designed to be independent of our database system.
• Our “Support” pages don’t work properly when the database is unavailable. This is actually a known issue; the pages are entirely database-driven from what amounts to a custom wiki on our end. On balance, this drawback is probably worth it; it allows our staff to make changes very easily (the ease of changes keeps it up-to-date and comprehensive), and we also customize pages for each viewer depending on what accounts they have with us.

“Learning is fun!”

That’s a sarcastic comment (from Futurama’s Bender character), of course. This kind of learning is actually not fun at all. On the other hand, one of the things we’re committed to is the idea of continuous improvement (fans of management systems probably know about “Kaizen”, or 改善). Each problem is an opportunity to make things better.

Any sufficiently advanced computer system is a web of interdependent parts that humans have difficulty understanding. Occasional failures are probably inevitable. But when we have a problem, we want to learn from it.

To that end, we’re making the following changes:

• We’ve added a check to ensure that no software package ever uses the top level of the disk as its home directory.
• We’ve made sure that blog.tigertech.net does not rely on our primary database system at all (it’s self-contained on servers in a completely separate data center), so we can use it to communicate with customers no matter what happens. If you normally reach our blog via a link on our main Web site, you should take a moment now to create a bookmark for blog.tigertech.net that you can use later (in case you somehow can’t reach our main site).
• We now have a simple way for any employee here to add a status update banner to every page on our site.
• We’re documenting improvements to our procedure for restoring database backups, avoiding several little annoyances that cost us valuable minutes.
• We’re fixing the software problem that causes some Webmail error messages during database failures.

And long-term, we’re looking into these:

• Adding the ability for our “Support” pages to work in a degraded, non-customized mode when the database is not available.
• Using the “streaming replication” and/or “hot standby” features that will be available in a forthcoming version of PostgreSQL to provide better database redundancy.
• Using a snapshot-able file system like Btrfs to more easily recover from processes that destroy data on a disk.
• Better segregation of changing data (databases, etc.) and unchanging data (program files and the root level of disks) on our servers, putting the unchanging data on partitions that are read-only and undeletable. The runaway file deletion process would not have caused problems if the root file system on this server was mounted in “read-only” mode.

We’re always trying to improve, and we regret that our best side wasn’t on display last Friday. We apologize for the inconvenience caused to our customers.

Although all WordPress users should upgrade, we’ve added security rules to our servers to protect our Web hosting customers who haven’t yet upgraded. Other people may find the rules useful if they use mod_security on Apache Web servers. The rest of this post contains more technical details.

Surprising Apache script behavior

One of the fixes in WordPress 2.8.6 compensates for a peculiar and surprising feature (or bug, if you prefer) in the Apache Web server: in many cases, it will run a file with a name like “image.php.jpeg” as a PHP script.

Some software (like WordPress 2.8.5 and earlier) can be configured to allow strangers to upload JPEG and other files to your site — but it checks only the “.jpeg” file extension to see if the uploaded file is safe. So a “hacker” could upload a malicious PHP script named “image.php.jpeg”, then “view the image” in a Web browser… but the server would actually run the PHP script.

Because of that, we’ve added a mod_security rule that prevents site visitors from requesting certain file extensions that include “.php.” in their name. (There are other possible solutions, but testing has shown this is the least intrusive to our existing customers.)

Here’s a mod_security rule that accomplishes that (adjust the extensions to suit your taste; these are the WordPress 2.8.5 allowed extensions):

SecRule REQUEST_FILENAME "\.php[456]?\.(asf|asx|avi|bmp|gif|ico|jpe|jpeg|jpg|png|tif|tiff|wax|wmv|wmx)$" "deny,status:412,auditlog"

WordPress brute force attacks

Another recently reported attack against WordPress (which is unrelated to version 2.8.6 in particular) involves “brute force” attempts to guess the administrator password. The proper solution is to choose a good password for your blog (not a word from the dictionary!), but some people don’t do that.

To reduce the risk of successful attacks against our customers, we limit each IP address to 25 “wp-login.php” attempts within a five minute period. Here’s how you can do that with mod_security:

SecAction phase:1,initcol:ip=%{REMOTE_ADDR},nolog
SecRule REQUEST_LINE "post .*/wp-login" "nolog,phase:1,setvar:ip.wordpress_login=+1,deprecatevar:ip.wordpress_login=5/60"
SecRule IP:WORDPRESS_LOGIN "@gt 25" "deny,status:412,auditlog,chain"
SecRule REQUEST_LINE "post .*/wp-login"

This unfortunately doesn’t prevent “distributed” attacks in which many different IP addresses submit different password guesses, but it will help in many cases.

Summary

So: if you use WordPress yourself, be sure to update. And if you provide blog hosting services for others, consider adding the mod_security rules to your Apache server.

If you use Rails and you haven’t explicitly chosen to stick with an existing version, it’s a good idea to check your applications and make sure there are no problems. But if you’re concerned about compatibility, it’s probably best to freeze Rails anyway so that server upgrades like this can’t affect your application.

Note that this version still isn’t the latest version of Rails, the 2.3 series. We’ll be updating to that shortly (for technical reasons, we needed to make sure we were using the last version of the 2.2 series first — of course, you can always install your own version, which is why we don’t necessarily make the latest version the default). The 2.3 series will introduce even bigger changes, so again, make sure you’ve frozen your version if you don’t want your applications to break.

By the way, if you haven’t frozen your application and you have problems with “No route matches” errors in 2.2.3, make sure you’ve followed the “3. Edit config/environment.rb” instructions on our Rails example page. That should fix it.

The good news is that this is just a logging bug caused by a recent Apache Web server update. Visitors to your site are seeing exactly what they always saw, and there isn’t any problem besides the incorrect logging.

The bug happens when a computer connects to your site and successfully requests a page, but closes the connection before the entire page is received.

That situation happens fairly often with Mozilla browser “prefetching”, for instance. It’s especially common on WordPress blogs because they contain prefetching hints.

Previous version of Apache would log these broken connections as a successful “200″ status code, albeit with a smaller-than-normal size. That makes sense: there’s no problem on the server end. But this Apache bug fix causes an interrupted connection to send a much quicker notice to the FastCGI Apache module. That module logs the unfinished page as a 500 error, even though there’s nothing happening that hasn’t aways happened.

We plan to patch the FastCGI module to prevent it from changing the logging, which will fix this annoyance. We’ll post an update as soon as this change has been made.

Update September 18: This issue has been fixed on all our servers as of 1:10 PM Pacific time.