High load on bender (2009-11-11) (resolved)

Our “bender” server encountered had an extremely high load starting at about 1:36 PM (Pacific time) today, and lasting until about 2:00 PM. Due to unusual circumstances, a runaway customer script caused the RAID disk array to become overloaded with writes, causing scripts that write to the disk to run slowly. The server may have seemed slow or unresponsive to customers at some point. We don’t consider this type of event to be acceptable, and apologize for the inconvenience.

We fixed the problem, and modified the server’s settings to hopefully prevent a recurrence. The changes were propagated to all similar hosting servers to protect them as well.

Today’s problem was unrelated to the one that happened on 2009-11-02.