We’ve also updated the ionCube Loader PHP extension (which most of our customers don’t use) from version 4.2.2 to version 4.4.0.

These changes should be transparent to customers. In the unlikely event you experience any issues, don’t hesitate to let us know.

This was caused by a single site making “runaway” database queries that left almost no MySQL “cache” memory available for other queries. The problem has been resolved by suspending the site involved, and we are analyzing how to prevent anything similar from happening in the future.

We apologize for this incident; we take reliability seriously and strive to avoid this kind of problem.

Followup: We have made a technical change that will prevent this from recurring. Our MySQL servers are configured to write temporary tables to /dev/shm, which defaults to 12 GB in size on our 24 GB RAM servers. This effectively allowed runaway queries to use up to 12 GB of RAM, emptying much of server’s general file cache. We have lowered the size of /dev/shm to a maximum of 6 GB, ensuring that the file cache doesn’t empty out and cause load spikes.

For the most part, sites hosted on our servers aren’t vulnerable to this because we block comments that contain the malicious code.

However, some customers asking about this haven’t yet updated their old copies of the plugins to the latest secure versions. You should always update plugins (and themes, and WordPress itself) as soon as the WordPress dashboard suggests it. If you do, your site will usually be secure long before you read about a vulnerability elsewhere. You should also delete any inactive plugins or themes, because it’s sometimes possible for “hackers” to take advantage of security bugs in them even if they’re deactivated.

By the way, although our customers are already protected against the most common forms of this, here’s a sample mod_security rule to block comments that include these malicious snippets if you’re not one of our customers but you’re running your own server with mod_security enabled:

SecRule REQUEST_LINE "^post .*/wp-comments-post\.php" "deny,status:412,auditlog,chain"
SecRule ARGS_POST:comment "< !-+\s*(mclude|mfunc|dynamic-cached-content)"


The average WordPress site we host has received tens of thousands of malicious login attempts this month, with hundreds of thousands of different IP addresses being used in the attacks. We try to block the IP addresses that are responsible, but the ever increasing number of addresses means we can’t block all of them — an individual address often attempts a login only once a day for a given site. We need to adopt other tactics.

So we now track other information about visitors to see if they’re legitimate when they attempt to login. In particular, we make sure that visitors have visited the WordPress login page on a site before they send a login “POST” request with a username and password. If they haven’t, we redirect them back to the login page.

This rule blocks the vast majority of fake login attempts, and it should cause no problems for legitimate logins: in the worst case, the login screen might be redisplayed to a human visitor. Let us know if you have any problems with that happening.

You can do your part, too. Never use a password that appears in any dictionary, and never choose an “obvious” password related to your site. In the last month, we’ve seen two WordPress sites hacked because the owners used their real name (visible as the “author” of each post on their site), or the site name, as their password. The automated software that hackers use can “scrape” words like that from pages and submit them as an attempted password. Always use something unrelated.

2:09 PM Pacific time: This is being caused by a distributed denial of service attack on WordPress sites that is causing outages for many companies. We’re working to block it.

2:36 PM Pacific time: The attack has been successfully blocked and all services are now working normally.

We sincerely apologize for the inconvenience this problem caused our customers.

This problem was caused by a brief period of high network latency to some destinations. That caused a larger-than-usual number of PHP processes to start, leading to reduced memory available for file system caching. This in turn made the server respond more slowly than usual, which caused even more PHP processes to start to handle the incoming requests. This made the problem worse in a “vicious circle” until we could manually limit the number of PHP processes being started.

The web12 server appears to be more vulnerable to this problem than other servers because of its PHP script usage pattern. While the number of PHP processes on all our servers increased, the problem was just bad enough on web12 that it couldn’t recover from it gracefully. We haven’t seen this particular issue happen before.

We are making immediate changes to the way PHP processes are started and limited to ensure this problem does not recur.

We sincerely apologize for this. We know you count on us for reliable service, and we are constantly striving to avoid this kind of problem.

This was due to a hardware failure at one of our upstream network partner companies. To work around the problem, our technicians had to manually close down the network session with that company to re-route all network traffic.

The problem has now been completely resolved. The upstream provider located and has replaced the faulty equipment. We sincerely apologize for the trouble this caused our customers.

In addition, we now offer PHP version 5.4.12 as an optional choice in our control panel. For now, the PHP 5.4 series is recommended only for customers who need to test “cutting edge” features. Most customers should stick with the PHP 5.3 series, which is compatible with a wider variety of scripts.

This problem was caused by high database load due to a customer making “runaway” database queries on that server. We are investigating the details to avoid future problems, and we apologize to affected customers.

During that four minute period, customers will not be able to use their Web sites or read e-mail. E-mail that arrives during this period will be queued and redelivered after the maintenance, not lost.

This maintenance and restart of all servers is necessary for security reasons. We apologize for the inconvenience this causes, together with the short notice (this issue is important enough that the patch should be applied as soon as it’s available, which is today).

Update 11:30 PM Pacific time: The maintenance was completed with less than 3 minutes downtime for all servers except web01, which took a few minutes longer because the Apache software on that server needed manually starting due to a technical problem that we have permanently resolved.