Other servers will not be affected. Mail for customers on these servers will be queued and delivered after a short delay.

We apologize for the inconvenience this causes. This is necessary to allow our technicians to perform hardware maintenance on these servers, which includes replacement of RAID array disks and the addition of RAM memory.

Update 11:15 PM: Both servers were upgraded. The upgrade of web05 took longer than expected, for which we apologize.

This was caused by a single site making “runaway” database queries that left almost no MySQL “cache” memory available for other queries. The problem has been resolved by suspending the site involved, and we are analyzing how to prevent anything similar from happening in the future.

We apologize for this incident; we take reliability seriously and strive to avoid this kind of problem.

Followup: We have made a technical change that will prevent this from recurring. Our MySQL servers are configured to write temporary tables to /dev/shm, which defaults to 12 GB in size on our 24 GB RAM servers. This effectively allowed runaway queries to use up to 12 GB of RAM, emptying much of server’s general file cache. We have lowered the size of /dev/shm to a maximum of 6 GB, ensuring that the file cache doesn’t empty out and cause load spikes.

2:09 PM Pacific time: This is being caused by a distributed denial of service attack on WordPress sites that is causing outages for many companies. We’re working to block it.

2:36 PM Pacific time: The attack has been successfully blocked and all services are now working normally.

We sincerely apologize for the inconvenience this problem caused our customers.

This problem was caused by a brief period of high network latency to some destinations. That caused a larger-than-usual number of PHP processes to start, leading to reduced memory available for file system caching. This in turn made the server respond more slowly than usual, which caused even more PHP processes to start to handle the incoming requests. This made the problem worse in a “vicious circle” until we could manually limit the number of PHP processes being started.

The web12 server appears to be more vulnerable to this problem than other servers because of its PHP script usage pattern. While the number of PHP processes on all our servers increased, the problem was just bad enough on web12 that it couldn’t recover from it gracefully. We haven’t seen this particular issue happen before.

We are making immediate changes to the way PHP processes are started and limited to ensure this problem does not recur.

We sincerely apologize for this. We know you count on us for reliable service, and we are constantly striving to avoid this kind of problem.

This was due to a hardware failure at one of our upstream network partner companies. To work around the problem, our technicians had to manually close down the network session with that company to re-route all network traffic.

The problem has now been completely resolved. The upstream provider located and has replaced the faulty equipment. We sincerely apologize for the trouble this caused our customers.

This problem was caused by high database load due to a customer making “runaway” database queries on that server. We are investigating the details to avoid future problems, and we apologize to affected customers.

During that four minute period, customers will not be able to use their Web sites or read e-mail. E-mail that arrives during this period will be queued and redelivered after the maintenance, not lost.

This maintenance and restart of all servers is necessary for security reasons. We apologize for the inconvenience this causes, together with the short notice (this issue is important enough that the patch should be applied as soon as it’s available, which is today).

Update 11:30 PM Pacific time: The maintenance was completed with less than 3 minutes downtime for all servers except web01, which took a few minutes longer because the Apache software on that server needed manually starting due to a technical problem that we have permanently resolved.

We took steps to work around the attack, which we completed by 7:08 PM Pacific time (46 minutes after the start of the attack). Furthermore, the attack itself seems to have stopped; the steps we took should help in case in starts again.

We sincerely apologize for the interruption in service for those affected customers; we know that reliable service is a primary concern for all of our customers.

All services are now working normally, and other servers were not affected. We apologize for the trouble this caused customers on the web03 server.

This affected all servers for about 19 minutes, until we and our network partners began discarding (“null routing”) all traffic targeted at that server. This fixed the problem for the rest of our network, but still left sites on the “web11″ server unavailable.

To solve that, the IP addresses of all sites on the web11 server have been changed to new IP addresses that are working correctly and are not under attack. This was completed by 3:44 PM, and all sites on all servers are now working properly.

If the attackers target another IP address, we’re ready to immediately block that one, too. If that does happen, the way we’ve redistributed the IP addresses, in combination with previous analysis we’ve done on this attack, will allow us to immediately know which site is under attack. (It’s otherwise hard to determine which IP address is involved, because the type of attack we’re seeing targets only an IP address and not a specific Web site name.) That site will then be moved off our main network to prevent a recurrence.

We sincerely apologize for the inconvenience this caused our customers; we know you count on us for reliable service, and we’re committed to doing everything possible to avoid problems.