This was caused by an upgrade to the eAccelerator PHP caching system that removed all the cached files at once, which doesn’t normally happen.

The problem has been permanently resolved and will not recur.

A technical explanation for why this caused trouble is that the sudden large number of disk writes caused by new eAccelerator files made the Linux kernel decide that disk “buffer” memory was so full that all disk writes needed to happen “synchronously”.

That caused the MySQL database to start writing temporary “filesort” data to the actual RAID array on the server, instead of just storing those files in memory (as Linux usually does for files that exist for less than a few seconds before being deleted). Some of our servers handle hundreds of MySQL queries a second, and the extra disk writing load overwhelmed the “/tmp” filesystem, slowing down MySQL dramatically.

We’ve made three changes to prevent this from happening again:

• We’ve modified our Debian eAccelerator package to not remove all the cached files at once during a future upgrade.
• We’ve changed where MySQL stores temporary files. It now uses “/dev/shm” shared memory instead of “/tmp”. (Ironically, “/tmp” used to be shared memory on our servers, but we had to change it to a real disk-based filesystem because it would fill up with large amounts of data if the server wasn’t restarted for months. That past experience gives us some assurance that this MySQL change won’t cause problems, though — and in fact, we’ve been testing this change on a small number of servers for some time anyway as a general performance improvement.)
• On our servers that support it, we’re now using “AMD64/Intel 64″ kernels that allow much larger disk memory buffers before the kernel switches to synchronous disk writes, avoiding the problem a different way. Some servers are already using the improved kernel (sadly, not these three servers), and all of our 64-bit-capable servers will be using it after the scheduled maintenance this coming Saturday.

We sincerely apologize for this incident. Don’t hesitate to let us know if you have any questions.

No e-mail will be lost, of course; incoming mail on those servers will just be slightly delayed.

We apologize for the inconvenience this causes. This maintenance is necessary to install an updated “kernel” on all of our servers for security reasons.

Update: The maintenance was completed by 11:10 PM with no more than three minutes of “downtime” per server.

This isn’t an outage on our end; these visitors are also unable to reach other sites that Comcast routes through Global Crossing (and not related to us), such as www.globalcrossing.com. It’s something Comcast and Global Crossing need to address.

We’ll continue to monitor this issue closely and post an update when we’re confident that it’s been resolved.

By the way, if you ever find that you’re unable to connect to our servers (or anyone else’s), a very useful site is CheckSite.us. It shows you whether the destination servers are down, or whether the problem is just a local routing problem that isn’t affecting most other people.

Update 9 AM PDT August 13: According to our monitoring systems, Comcast resolved this shortly after our post, and the problem has not recurred in the ten hours since then.

As a result, Web site service and the ability to read incoming e-mail for some customers will be unavailable for approximately five minutes at some point during this maintenance “window”.

No e-mail will be lost, of course; incoming mail on those servers will just be slightly delayed. Customers using other servers will not be affected.

We apologize for the inconvenience this causes and for the short notice. Restarting these servers now is necessary to prevent a potential issue that could cause problems with these servers if a RAID disk needs replacing in the future. (We hope to provide technical details in a future post. The problem — an issue with the GRUB bootloader — is interesting and the solution would be useful to others.)

Update: The maintenance was completed with less than 5 minutes “downtime” per server.

Other servers will not be affected, and incoming mail will only be delayed, not lost.

We apologize for the problem and for the short notice: the restart is necessary to replace a potentially failing disk in the RAID array.

(This is the second disk on that server that has needed replacing this year. We track these things closely, and we are monitoring it to make sure there isn’t a problem beyond unfortunate coincidence.)

No e-mail will be lost, of course; incoming mail on those servers will just be slightly delayed.

We apologize for the inconvenience this causes. This maintenance is necessary to install an updated “kernel” on all of our servers for security reasons, and we’re taking that opportunity to install additional memory on some servers, too.

Update: All server updates were completed as expected.

Update 12:35 PM: Our upstream provider reports that an 8 minute network interruption for some connections, beginning at 11:11 AM Pacific time, was caused by a router failure at Global Crossing. The problem has been resolved.

Our automated monitoring systems didn’t detect any general problems, so the majority of customers were certainly unaffected — but we suspect that one of the “Internet backbones” between the affected customers and our data center had high packet loss during that period.

Both customers reported that the problem resolved itself by 7:45, and we haven’t received similar reports since, so there does not appear to be be an ongoing problem. We’ll continue to monitor it closely.

Other servers will not be affected, and incoming mail will only be delayed, not lost.

We apologize for the problem and for the short notice: the restart is necessary to replace a disk in the RAID array.

Update 11:03 PM Pacific time: The restart was completed with less than 3 minutes “downtime”.

This server had similar high load symptoms (but much more briefly) earlier this week. We took some steps to reduce the load then, but it appears those weren’t sufficient. We’re now taking much stronger action to ensure that this does not happen again.

We sincerely apologize to customers affected by this problem. We don’t consider it normal or acceptable, and we will make sure this isn’t a recurring issue.