The “calculon” Web server needed to be restarted at 12:40 AM Pacific time this morning due to extremely high load.
However, the server did not restart immediately, because it performed a time-consuming disk file system check (“fsck”) after the restart, causing an interruption in Web service and a delay in mail delivery for customers on that server (other servers were not affected).
The server finished its fsck check at 3:45 AM and is now working normally.
This is by far the longest outage we’ve experienced on a server in several years. I want to personally apologize to every affected customer: we don’t consider this kind of problem acceptable at all, and we deeply regret the downtime. We’ll be carefully reviewing this incident to see what we can learn from it in the future.
Our business offices will be closed on Monday, September 1 to observe the US legal holiday. As always, we’ll provide same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until the next day, and telephone support (via callbacks) will be available only for urgent problems.
Between 5:11 and 5:46 PM Pacific time today, some people who reach our servers via an “Internet backbone” called Global Crossing (including some Comcast cable customers) were unable to connect to our data center. Other users weren’t affected.
Global Crossing has apparently corrected the problem, and everything is now operating normally. We’ll continue to monitor this issue closely.
Read the rest of this entry »
Due to software updates on our servers, most Web hosting customers will experience about ten minutes of scheduled maintenance downtime between 11 PM and 1 AM Pacific time starting on one of the following nights, depending on which server your site is on:
- Friday, August 22 (servers beginning with letter “l-z”)
- Saturday, August 23 (servers beginning with letter “a-k”)
(The servers named “bender” and “lrrr” have already been upgraded, and those customers are not affected.)
Read the rest of this entry »
