When search engines swarm new posts

We saw an interesting problem today. One of our customers’ Web sites uses WordPress with WP Super Cache to (dramatically) improve its performance. Every time the customer posts new content, though, the site is immediately swarmed by search engines, feeds, robots, and other non-humans retrieving the new post. There are a lot of unnecessary duplicate requests, but even excluding the duplicates there are hundreds of requests arriving almost simultaneously.

Unfortunately, WP Super Cache is configured by default not to serve cached results to any request that contains an “equals sign” in the query string — and the plugin that notifies the other sites of new content is including an equals sign.

So rather than being immediately served from the cache, all of the new requests were run through WordPress PHP scripts, driving up the script usage and causing “503 Service Unavailable” errors for up to two minutes on that Web site (not for other Web sites on the same Web server, though; we have protection against that).

Read the rest of this entry »

Even better performance from WP Super Cache

In a previous post, we talked about how increasing the WP Super Cache “Expire time” from 1 hour to 48 hours can help the performance of WordPress blogs.

Here’s another tip that can help dramatically: Remove “bot”, “ia_archive”, “slurp”, “crawl”, “spider” and “Yandex” from the Rejected User Agents box in the WP Super Cache plugin settings. (In most cases, this will leave the box completely empty.)

Read the rest of this entry »

WP Super Cache and FeedBurner

We’ve got a lot of customers running WordPress, and we definitely recommend running WP Super Cache to improve performance. It can help dramatically!

But recently we’ve seen a number of our customers getting hammered by a ton of requests from FeedBurner. Usually the request is of this form:

/somepost?utm_source=feedburner&utm_medium=feed&utm_campaign=SomeCampaignString

We’ve also seen FeedBurner going crazy and making thousands of duplicate requests. One of the sites we host has gotten 45,000 simple status requests (HTTP “HEAD” requests) from FeedBurner today, for no good reason that we can see.

Read the rest of this entry »

Better performance from WP Super Cache

If you use the WP Super Cache WordPress plugin (and you should, if you use WordPress), it has a settings page section titled “Expiry Time & Garbage Collection”. It sets the “Cache Timeout” to 3600 seconds by default, and warns that you should set it lower on a busy site.

That advice makes sense if you have a sudden surge of traffic to a single page. However, if your site is generally very busy across all pages (for example, if you have an archive of hundreds or thousands of posts that are constantly being indexed by search engines), we’ve found that you should do the opposite to improve performance: set it much higher. We recommend setting it to 172800 seconds (which is 48 hours). This can cut your CPU usage in half, which will speed up your site.

Read the rest of this entry »