The “bender” Web server experienced intermittently high load between about 7:40 and 10:15 AM Pacific time this morning, February 18. This resulted in slow or even inaccessible Web sites on that server. (Some e-mail was also delayed before being properly delivered.) Other servers were not affected.
This server had similar high load symptoms (but much more briefly) earlier this week. We took some steps to reduce the load then, but it appears those weren’t sufficient. We’re now taking much stronger action to ensure that this does not happen again.
We sincerely apologize to customers affected by this problem. We don’t consider it normal or acceptable, and we will make sure this isn’t a recurring issue.
Our “bender” server encountered had an extremely high load starting at about 1:36 PM (Pacific time) today, and lasting until about 2:00 PM. Due to unusual circumstances, a runaway customer script caused the RAID disk array to become overloaded with writes, causing scripts that write to the disk to run slowly. The server may have seemed slow or unresponsive to customers at some point. We don’t consider this type of event to be acceptable, and apologize for the inconvenience.
We fixed the problem, and modified the server’s settings to hopefully prevent a recurrence. The changes were propagated to all similar hosting servers to protect them as well.
Read the rest of this entry »
Starting just after 9AM (Pacific time) today, the “bender” server experienced some very high loads (for about 40 minutes). It seemed to be coming from a combination of severe database, e-mail, and Web server access. Sort of a “perfect storm” of unusual load.
We work very hard to run all of our servers at a reasonable level, with excess capacity to spare. Even though the load was unusual, we don’t consider this type of limitation acceptable. We are reviewing the server’s configuration files to see if we can make changes to avoid this sort of problem in the future.
At approximately 11:00 PM Pacific time this Saturday, May 2, the “bender”, “calculon”, “lrrr” and “hypnotoad” servers will be restarted. As a result, Web site and e-mail service for customers on those servers will be unavailable for approximately five minutes.
Read the rest of this entry »
At approximately 11:00 PM Pacific time on Saturday, January 31, all of our Web hosting servers (except the “hypnotoad” and “mom” servers) will be restarted. As a result, Web site and e-mail service for some customers will be unavailable for approximately five minutes.
No e-mail will be lost, of course; incoming mail will just be delayed for a few minutes.
We apologize for any inconvenience this may cause. This maintenance is necessary to install an updated “kernel” on our servers, as described in an earlier maintenance announcement.
Update: the maintenance was successfully completed on all servers with less than 5 minutes of “downtime”.
This morning at 12:11 AM (Pacific time), one of the cabinets at our data center tripped a circuit breaker, causing all of the servers in that cabinet to lose power. Power was restored at 12:18 AM.
Customer Web sites and e-mail on the bender, calculon, lrrr, and zapp Web servers were unavailable during this 7 minute period. The ability to send and receive e-mail was also interrupted (no mail was lost, of course).
We are investigating the root cause of this problem to prevent it from happening again.
