If you have a Ruby on Rails application that originally used Rails 1.1.6 or earlier, you might have trouble after yesterday’s Rails update (which also updated several other Ruby “gems”, including the “RubyGems” gem itself).
The network interface on the “farnsworth” Web server stopped responding at 6:38 AM Pacific time today, and the server needed to be manually restarted by our data center staff. The server was unavailable for 17 minutes, causing an interruption of service for Web sites on that server. It also prevented users of that server from reading incoming e-mail (such e-mail was delayed and delivered after the outage).
Other servers were not affected.
We sincerely apologize to anyone affected by this problem.
We’ve updated the default version of Ruby on Rails on our servers to version 2.1.1.
The “calculon” Web server needed to be restarted at 12:40 AM Pacific time this morning due to extremely high load.
However, the server did not restart immediately, because it performed a time-consuming disk file system check (“fsck”) after the restart, causing an interruption in Web service and a delay in mail delivery for customers on that server (other servers were not affected).
The server finished its fsck check at 3:45 AM and is now working normally.
This is by far the longest outage we’ve experienced on a server in several years. I want to personally apologize to every affected customer: we don’t consider this kind of problem acceptable at all, and we deeply regret the downtime. We’ll be carefully reviewing this incident to see what we can learn from it in the future.
Our business offices will be closed on Monday, September 1 to observe the US legal holiday. As always, we’ll provide same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until the next day, and telephone support (via callbacks) will be available only for urgent problems.
Between 5:11 and 5:46 PM Pacific time today, some people who reach our servers via an “Internet backbone” called Global Crossing (including some Comcast cable customers) were unable to connect to our data center. Other users weren’t affected.
Global Crossing has apparently corrected the problem, and everything is now operating normally. We’ll continue to monitor this issue closely.
Due to software updates on our servers, most Web hosting customers will experience about ten minutes of scheduled maintenance downtime between 11 PM and 1 AM Pacific time starting on one of the following nights, depending on which server your site is on:
(The servers named “bender” and “lrrr” have already been upgraded, and those customers are not affected.)
As you may know, we perform nightly backups of each Web hosting customer’s data, including your Web site, databases, log files, and any e-mail stored on our servers. Until recently, however, you usually needed to write to our support team if you wanted to get something restored.
We’re pleased to announce that we have recently made it easier for you to access your backups. Now you can easily browse through all the backups available for your site and retrieve any files using a simple FTP or SSH connection. In addition, our control panel lets you quickly see exactly which backup sets are available.
See our support page for full details.
We’ve updated the default version of Ruby on Rails on our servers to version 2.1.0.
Around 11:26 AM (Pacific time) this morning, one of our mail servers encountered an unusual load, became unresponsive, and needed to be restarted. This affected our users’ ability to read e-mail and to use our Webmail system for several minutes.
