Our customers are not vulnerable to this problem because of the way PHP is set up on our servers. You don’t need to worry about it.

Since this is a big deal, we’ve checked all the possibilities very carefully:

• PHP scripts that run without FastCGI;
• PHP scripts using FastCGI;
• Custom compiled PHP versions that use our “compile-and-install-php” shortcuts.

None of these are susceptible to the bug. The first and last aren’t vulnerable for the reason listed in comment 38 of that page (we use “AddHandler” instead of “Action”), and the FastCGI case isn’t vulnerable because the “wrapper script” we suggest doesn’t pass any user-supplied parameters through to the PHP binary.

So unless you’ve compiled your own version of PHP and installed it in a way that the PHP documentation recommends against for security reasons (and you’d certainly know if you’d done so), you’re safe.

As an extra measure, we’ve also added “mod_security” rules to block the most common “in the wild” attempted attacks that try to exploit this bug (based on seeing a very large number of them in our logs). These attackers can’t even start PHP running.

We’re investigating the underlying cause of this, and we sincerely apologize for the trouble if you were affected.

Most customers will not notice any service interruption because we use redundant network providers, but in the worst case it can take up to about 90 seconds for certain parts of the Internet to see the changed “routes”. That means a brief interruption is theoretically possible for some connections. We’re announcing this just so you know that if you do see any problem, it will be resolved quickly.

Update 4:33 PM Pacific time: The maintenance has been completed.

This maintenance is necessary to install a mandatory MySQL security update that will upgrade the MySQL version to 5.1.61. We apologize for any inconvenience this causes.

Update 10:13 PM: The maintenance was completed with less than 30 seconds downtime on each server. Customers should not notice any changes, but as always, don’t hesitate to contact us with any questions or problems.

This was caused by a “hung” process that prevented a routine Apache Web server reload from completing. Other servers were not affected. Our staff restarted the server to stop the “hung” process, and the problem was resolved.

We sincerely apologize to customers affected by this incident. We’re considering possible underlying causes to prevent a recurrence.

Most customers will not notice anything, but the upgrade will cause approximately 30 seconds of slow Web page loading at some point during that hour as we delay incoming connections at the network level.

This maintenance is necessary to apply security and reliability fixes released by the Apache developers. (We’ve been using the upgraded version on our Webmail servers for several days, so it’s well tested.)

Update: The maintenance was completed at 10:03 PM Pacific time.

We’re investigating the underlying cause of this and will take appropriate action to avoid a recurrence. We sincerely apologize for the trouble if this affected you.

What we found was that the owner of one of the sites on that server made a mistake that allowed attackers to run their own scripts. That’s all too common, unfortunately, but usually only the single site is affected by this kind of thing. What was surprising in this case was that the script used a previously unknown method of causing problems for other sites running on the server.

As a result of this investigation, we’ve made several changes to our systems to ensure the problem won’t recur. The rest of this post has a detailed technical description of the problem in case it’s useful for others.

Technical details

Over a year ago, one of our customers installed a script called “vBSEO” on his site. Unfortunately, that script had a known programming error that allowed strangers to run any software as if they were the site owner, and the customer didn’t update the script promptly. An attacker took advantage of that bug to run some malicious software.

Note that this isn’t due to any kind of security flaw in our systems, and the attacker didn’t gain any access to any other sites, passwords, or anything else. All that happened is that the attacker was able to run some software that the real site owner could have run (but wouldn’t have done). Finding flaws in vBSEO is, in one sense, a waste of time, because the attacker could have simply opened an account with us and been able to run the same software. Any of our customers could run it.

Because of that, one of the important features of our service is that we isolate each site from other sites. It should be impossible for one site owner to do something that affects others. In this case, though, the malicious software used up general server memory in a way that bypassed our normal per-site memory restrictions. That had the effect of making a small number of PHP scripts on the server fail to start, generating “Internal Server Error” messages instead.

What the malicious software actually did was create over 4000 small “shared memory segments“, filling up the entire 4,096 available shared memory slots on the server. It appears that unlike some other operating systems, Linux has no way to prevent one process from doing this. (Solaris, for example, allows you to set this by altering SHMSEG, and it defaults to allowing just 6 segments per process.) In fact, this single, trivial line of PHP code allows any user to fill up all the slots on a Linux server:

while (shmop_open(NULL, "c", 0644, 100)) {}

Once the shared memory slots are all taken, any other process on the server that tries to use shared memory will fail until a new slot opens up. This means that PHP with eAccelerator wasn’t able to successfully run some scripts (which set off immediate alarms in our monitoring systems).

It’s interesting that such a trivial attack from a non-privileged user breaks the widely used eAccelerator (and other) software. We can’t find any other discussion of this on the Internet, and it’s likely that many other companies are vulnerable, too.

Changes to solve the problem

After discovering the exact cause of this problem, we took three steps to make sure it can’t happen again:

1. We’ve added a mod_security rule to block this attack on vBSEO, even if our customers haven’t updated their vBSEO software. (This rule has prevented attacks on six other sites we host since we added it — it’s a widespread attack.)
2. Our monitoring systems now check for the “out of shared memory slots” situation and automatically fix it by deleting rogue segments.
3. Most importantly, we’ve changed how eAccelerator works on our systems so that it doesn’t require a shared memory segment. Even if someone does fill up the shared memory slots, eAccelerator will no longer fail, so it won’t cause noticeable problems for customers.

We’ll also notify the eAccelerator folks of this incident in case they want to change the default way it works, avoiding this problem for others.

The restart resolved the immediate problem, and a followup post explains what happened and the changes we made to prevent it from happening again.

We’ve received scattered reports of high “packet loss” to a few Comcast locations (but not most). Packet loss can cause pages to load slowly in some cases.

Debugging of this with our network providers shows that the problem is within the Comcast network: one of their “backbone” routes between San Jose and Denver has high packet loss. Comcast is aware of the problem and working on it.

Since Comcast uses many different “routes”, this doesn’t affect most customers. If your connection does happen to use that route for some connections, though, we know this is frustrating.

For networking experts: We have made an effort to “route around” the problem by sending Comcast traffic to different network peers, but unfortunately Comcast ends up sending it through the same backbone route regardless of how they receive it. This is affecting network traffic for Comcast customers to many different locations, and we expect they’ll have it resolved quickly.