The “fry” and “bender” Web servers will be restarted between 11:00 and 11:15 PM Pacific time tonight (Friday, April 29, 2011). This will cause a five-minute interruption of Web and e-mail service for customers on those servers.
Other servers will not be affected, and incoming mail will only be delayed, not lost.
We had a couple of instances of MySQL queries overloading the bender server today. The first one happened at about 3:41 AM (Pacific time) and the second one happened at about 7:48 AM. Each occurrence lasted about 20 minutes. The problem each time was that a database was running extremely inefficient queries. Each time we fixed the problem by creating indexes so that the queries could then run in a fraction of the time previously required.
We apologize for any inconvenience caused by this problem. Visitors to your Web site (on the bender server) might have seen reduced performance (or, in rare cases, 503 errors). E-mail was not affected. We don’t consider this type of problem to be acceptable. These problems should not recur since the indexes have been created.
Between 11:00 PM and 11:59 PM Pacific time tonight (Monday August 2), several of our hosting servers will be restarted: bender, elzar, farnsworth, lrrr, mom, and seymour.
As a result, Web site service and the ability to read incoming e-mail for some customers will be unavailable for approximately five minutes at some point during this maintenance “window”.
The “bender” Web server experienced intermittently high load between about 7:40 and 10:15 AM Pacific time this morning, February 18. This resulted in slow or even inaccessible Web sites on that server. (Some e-mail was also delayed before being properly delivered.) Other servers were not affected.
This server had similar high load symptoms (but much more briefly) earlier this week. We took some steps to reduce the load then, but it appears those weren’t sufficient. We’re now taking much stronger action to ensure that this does not happen again.
We sincerely apologize to customers affected by this problem. We don’t consider it normal or acceptable, and we will make sure this isn’t a recurring issue.
Our “bender” server encountered had an extremely high load starting at about 1:36 PM (Pacific time) today, and lasting until about 2:00 PM. Due to unusual circumstances, a runaway customer script caused the RAID disk array to become overloaded with writes, causing scripts that write to the disk to run slowly. The server may have seemed slow or unresponsive to customers at some point. We don’t consider this type of event to be acceptable, and apologize for the inconvenience.
We fixed the problem, and modified the server’s settings to hopefully prevent a recurrence. The changes were propagated to all similar hosting servers to protect them as well.
Starting just after 9AM (Pacific time) today, the “bender” server experienced some very high loads (for about 40 minutes). It seemed to be coming from a combination of severe database, e-mail, and Web server access. Sort of a “perfect storm” of unusual load.
We work very hard to run all of our servers at a reasonable level, with excess capacity to spare. Even though the load was unusual, we don’t consider this type of limitation acceptable. We are reviewing the server’s configuration files to see if we can make changes to avoid this sort of problem in the future.
At approximately 11:00 PM Pacific time this Saturday, May 2, the “bender”, “calculon”, “lrrr” and “hypnotoad” servers will be restarted. As a result, Web site and e-mail service for customers on those servers will be unavailable for approximately five minutes.
At approximately 11:00 PM Pacific time on Saturday, January 31, all of our Web hosting servers (except the “hypnotoad” and “mom” servers) will be restarted. As a result, Web site and e-mail service for some customers will be unavailable for approximately five minutes.
No e-mail will be lost, of course; incoming mail will just be delayed for a few minutes.
We apologize for any inconvenience this may cause. This maintenance is necessary to install an updated “kernel” on our servers, as described in an earlier maintenance announcement.
Update: the maintenance was successfully completed on all servers with less than 5 minutes of “downtime”.
This morning at 12:11 AM (Pacific time), one of the cabinets at our data center tripped a circuit breaker, causing all of the servers in that cabinet to lose power. Power was restored at 12:18 AM.
Customer Web sites and e-mail on the bender, calculon, lrrr, and zapp Web servers were unavailable during this 7 minute period. The ability to send and receive e-mail was also interrupted (no mail was lost, of course).
We are investigating the root cause of this problem to prevent it from happening again.
