This was caused by an upgrade to the eAccelerator PHP caching system that removed all the cached files at once, which doesn’t normally happen.

The problem has been permanently resolved and will not recur.

A technical explanation for why this caused trouble is that the sudden large number of disk writes caused by new eAccelerator files made the Linux kernel decide that disk “buffer” memory was so full that all disk writes needed to happen “synchronously”.

That caused the MySQL database to start writing temporary “filesort” data to the actual RAID array on the server, instead of just storing those files in memory (as Linux usually does for files that exist for less than a few seconds before being deleted). Some of our servers handle hundreds of MySQL queries a second, and the extra disk writing load overwhelmed the “/tmp” filesystem, slowing down MySQL dramatically.

We’ve made three changes to prevent this from happening again:

• We’ve modified our Debian eAccelerator package to not remove all the cached files at once during a future upgrade.
• We’ve changed where MySQL stores temporary files. It now uses “/dev/shm” shared memory instead of “/tmp”. (Ironically, “/tmp” used to be shared memory on our servers, but we had to change it to a real disk-based filesystem because it would fill up with large amounts of data if the server wasn’t restarted for months. That past experience gives us some assurance that this MySQL change won’t cause problems, though — and in fact, we’ve been testing this change on a small number of servers for some time anyway as a general performance improvement.)
• On our servers that support it, we’re now using “AMD64/Intel 64″ kernels that allow much larger disk memory buffers before the kernel switches to synchronous disk writes, avoiding the problem a different way. Some servers are already using the improved kernel (sadly, not these three servers), and all of our 64-bit-capable servers will be using it after the scheduled maintenance this coming Saturday.

We sincerely apologize for this incident. Don’t hesitate to let us know if you have any questions.

There are many sources of missing image file references. Themes and plugins can generate them. Your site may be missing a favicon.ico file. Some RSS news readers make a lot of requests for files named apple-touch-icon.png and apple-touch-icon-precomposed.png, which most Web sites do not have. Check your Web site’s analytics report (such as AWStats) or raw log files to see what image files are missing from your site.

If you’re a Tiger Technologies customer running WordPress, you already benefit from a global rule we implemented for missing favicon.ico files (as discussed in a previous blog post). We’ve also recently updated our WordPress Performance page with extra tips on returning faster “404″ results for other missing files such as robots.txt and for all missing images.

Well, “Good news, everyone!” If you use Tiger Technologies to host your WordPress blog, you’re all set: we already use later versions of PHP and MySQL than that.

WordPress also now provides a “health check” plugin that shows you whether your hosting company is ready. If you run it on one of our servers, you’ll see:

(By the way, our versions of PHP and MySQL also include security updates from later versions.)

Here’s the relevant log entries from a site that FeedBurner hit 5,836 times in one hour this morning (up to 8 times a second). There’s nothing unusual about the site: it’s on a single IP address with a single hostname, and the feed doesn’t change often.

Some sites run a PHP script for every request, so this FeedBurner problem generates high load for no useful purpose at all.

Google: Please fix this. Thanks!

It turns out that wget has a security bug that needs to be avoided. As a result, the behavior of wget has changed in some situations. If you use wget (most of our customers don’t), you should be aware of this change.

The problem

If wget requests a file with an innocuous name from another server, but that server replies with a “redirect” to a file with a malicious name, wget will save the file with the malicious name instead.

As an example, imagine you run this command and expect wget to save a file named “feed.rss” at the top level of your site:

$ wget http://example.com/feed.rss

However, the remote server redirects the original request to a malicious file at:

$ wget http://example.com/index.html

In that situation, wget will actually save the file as “index.html” on your site, potentially changing your home page to something awful. More complex attacks that actually run malicious code are also possible.

The solution

A wget security update that we installed today changes how it handles a redirect. By default, it now saves the file under the original name (“feed.rss” in the example above) instead of a new name the server provides.

This change will not affect most people who use wget. However, if you do use wget to request URLs that redirect to a new name, and you rely on wget saving the file using that new name despite the security risks, you’ll need to start using wget’s “--use-server-file-name” option to keep it working the way you expect.

Here’s another tip that can help dramatically: Remove “bot”, “ia_archive”, “slurp”, “crawl”, “spider” and “Yandex” from the Rejected User Agents box in the WP Super Cache plugin settings. (In most cases, this will leave the box completely empty.)

Those “Rejected User Agents” prevent cached pages from being created when a search engine “visits” your site. The FAQ says this is because there’s no point creating a cached file for a page that isn’t popular, which may be true for sites that don’t have many posts and aren’t that busy. The author of the plugin is wisely being conservative to avoid problems on hosting companies that use “NFS” disks where saving a cached file can be slow.

We don’t use NFS like that, though. And on a busy site with lots of archived posts, there’s a very good chance that a page that’s indexed by a search engine will be reindexed or viewed several times within the next couple of days. Allowing these pages to be cached can make a big difference. Besides, there’s no good reason not to cache a copy of the page, since you should be sending the same page to search engines as you do to actual users.

We removed those “Rejected User Agents” on a busy site with several thousand archived posts, with the “Expire time” set to 48 hours (172800 seconds) as we recommend, and here’s what happened to that site’s CPU usage:

As you can see, the average load dropped by almost half as more and more pages were cached. That’s another way of saying that the average page loading speed almost doubled. Those are pretty good results for a change that took just a few seconds to make.

By default, each WordPress blog is configured to send the login username and password as plain (unencrypted) text. If a hacker can see what you are sending during your login, they can easily steal your username and password. This can happen if you have a virus installed on your computer. It can also happen if your computer is virus-free but connects via WiFi. If your main computer uses a wireless connection, or if you or other users of your blog ever login with their laptops — blogging from a coffee shop, anyone? — remember that these connections can be insecure, and could be susceptible to revealing your password.

You can protect your blog by installing an “SSL certificate” and configuring WordPress to require secure logins. Your browser will then encrypt your username and password so that no one can intercept them.

Traditionally, only online stores used SSL certificates because they were very expensive. But SSL certificate prices have dropped quite a bit recently, and they’re now low enough that we think SSL certificates should be widely used to protect all logins and other sensitive data.

If you are a Tiger Technologies customer, you can get an SSL certificate for a great price. (One type of certificate, a “self-signed certificate”, is even free if you’re already on our Gold or Platinum hosting plans.) If you’re not a Tiger Technologies customer, you can search for companies selling SSL certificates or search for free self-signed certificates.

Configuring WordPress

Once you have an SSL certificate installed on your site, it’s easy to configure WordPress to use secure logins. Simply add this line anywhere to your wp-config.php file (after the opening “<?php” line):

define('FORCE_SSL_ADMIN', true);

This will ensure that your username and password are submitted to WordPress securely; all of your subsequent work (creating posts, etc) will be secure as well. You’ll see your Web browser’s “padlock” icon when you are using a secure connection. The WordPress “Administration Over SSL” page has more details.

Several shared hosting companies apparently allow customers to view the text of other customer’s files by default, and that allows malicious customers to discover the database password of another site (from the “wp-config.php” file) and alter the site.

Some security researchers think this is a WordPress flaw, but we agree with the WordPress folks that that’s nonsense. The real problem is that these hosting companies are allowing customers to view each other’s files, which shows a reckless disregard for security.

For the record, this kind of attack is impossible on our servers. We prevent customers from viewing each other’s files. In particular, we use suEXEC to run each user’s scripts under a separate Unix user ID, and we ensure that your files are protected from other users at the file system level.

In fact, we go a step further than even most reasonably secure companies: we protect your files even if you change every possible file and directory to be world-writable (mode 777), using an additional layer of Linux “access control list” rules on your top-level directories. We also automatically reset the bad directory permissions for you, just to be sure there are always two levels of protection.

So you don’t need to worry about this kind of attack; “we’ve got your back”.

A different attack on wp-config.php

Separately and coincidentally, our security systems detected a different type of attack on a “wp-config.php” file today. Some “hackers” on the Internet are trying to load files named “wp-config.php~” with a tilde character at the end. In most cases, that file won’t exist, but if you’ve edited your “wp-config.php” file with some text editors, it creates a “wp-config.php~” version as a backup copy.

This is an interesting attack. Of course, it’s not possible to load this file in a Web browser and see the source:

http://example.com/wp-config.php

However, it is possible to see the source of this file if it exists:

http://example.com/wp-config.php~

That’s because the server doesn’t recognize that the “.php~” file contains PHP code, so it shows the visitor the contents (source) of the file, with disastrous results.

We’ve fixed this potential attack by blocking all requests for filenames ending in “.php~” on our servers, since it appears that nobody is using this legitimately.

If you run your own Web servers, this simple mod_security rule will do the trick:

SecRule REQUEST_FILENAME "\.php~$" "msg:'PHP file backup exploit',deny,status:412,auditlog"

(Just so it’s clear, we’ve already done this for our customers; that rule is mentioned in the hope that it might be useful to someone else. We also mentioned this issue on the WordPress support forums.)

The Domain Block List is an extremely reliable list of domain names that are used only in spam. Blocking most mail that advertises these domain names improves our spam filtering: we’re now blocking about 1% more spam as a result.

That may not sound like much, but it represents about 150 more blocked spam messages per year for each customer. (We block an average of over 15,000 spam messages per year per customer.)

Those runaway queries can slow down the MySQL server for many minutes on end, causing performance problems.

To prevent the worst of that, we’ve set the max_join_size setting to 1,000,000 on our MySQL servers.

This shouldn’t affect any normal queries, but queries that attempt to sort more than a million rows will now return this error:

ERROR 1104 (42000): The SELECT would examine more than MAX_JOIN_SIZE rows; check your WHERE and use SET SQL_BIG_SELECTS=1 or SET SQL_MAX_JOIN_SIZE=# if the SELECT is okay

If you have any questions, please contact our support staff.