Our business offices will be closed on Thursday, November 26 to observe the US legal holiday. As always, we’ll provide same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until Friday, and telephone support (via callbacks) will be available only for urgent problems.
Customers on the “amy” server experienced a nine minute interruption in Web site and e-mail service between 5:58 and 6:07 PM Pacific time today (November 16).
Customers on other servers were not affected.
Our “bender” server encountered had an extremely high load starting at about 1:36 PM (Pacific time) today, and lasting until about 2:00 PM. Due to unusual circumstances, a runaway customer script caused the RAID disk array to become overloaded with writes, causing scripts that write to the disk to run slowly. The server may have seemed slow or unresponsive to customers at some point. We don’t consider this type of event to be acceptable, and apologize for the inconvenience.
We fixed the problem, and modified the server’s settings to hopefully prevent a recurrence. The changes were propagated to all similar hosting servers to protect them as well.
Starting just after 9AM (Pacific time) today, the “bender” server experienced some very high loads (for about 40 minutes). It seemed to be coming from a combination of severe database, e-mail, and Web server access. Sort of a “perfect storm” of unusual load.
We work very hard to run all of our servers at a reasonable level, with excess capacity to spare. Even though the load was unusual, we don’t consider this type of limitation acceptable. We are reviewing the server’s configuration files to see if we can make changes to avoid this sort of problem in the future.
