As regular readers know, our blog is usually about the (often boring) technical details of Web hosting service. This post is different. We want to publicly state that our company is opposed to Proposition 8, the California ballot initiative that would take away marriage rights for some Californians.
This morning at 12:11 AM (Pacific time), one of the cabinets at our data center tripped a circuit breaker, causing all of the servers in that cabinet to lose power. Power was restored at 12:18 AM.
Customer Web sites and e-mail on the bender, calculon, lrrr, and zapp Web servers were unavailable during this 7 minute period. The ability to send and receive e-mail was also interrupted (no mail was lost, of course).
We are investigating the root cause of this problem to prevent it from happening again.
At approximately 11:00 PM Pacific time tonight (October 18), the “calculon” Web server will be restarted. As a result, Web sites and e-mail service for customers using that server will be unavailable for approximately five minutes.
We’ve installed a PHP 5 security update. Customers should not notice any changes; the updates just fix several security issues in PHP 5.
Due to a problem with our spam filtering system, some customers received blank incoming messages between 11:35 and 11:53 AM (Pacific time) this morning.
A non-blank copy of these messages was also properly delivered (although with a delay), so no mail is missing.
We have permanently fixed the underlying cause of the problem, and we apologize for the concern and annoyance this caused.
We’ve updated rsync to version 3 on our servers. (We’re using Debian version 3.0.3-2, which includes patches from rsync 3.0.4.)
Rsync 3 is significantly faster than previous versions for recursive file transfers (which we use in our backup system). However, the new version is backward compatible with rsync 2.x, so users shouldn’t notice any changes or problems, even if you haven’t updated your own copy of rsync.
At approximately 11:00 PM Pacific time this Saturday night (September 20), all Tiger Technologies servers will be restarted. As a result, customer Web sites and e-mail service will be unavailable for three to five minutes.
No e-mail will be lost, of course; incoming mail will just be delayed for a few minutes.
This brief maintenance is necessary to upgrade the operating system “Linux kernel” to a newer version for security reasons. We apologize for the inconvenience this causes.
Update: the maintenance was completed with less than three minutes “downtime” per server.
If you have a Ruby on Rails application that originally used Rails 1.1.6 or earlier, you might have trouble after yesterday’s Rails update (which also updated several other Ruby “gems”, including the “RubyGems” gem itself).
We’ve updated the default version of Ruby on Rails on our servers to version 2.1.1.
The “calculon” Web server needed to be restarted at 12:40 AM Pacific time this morning due to extremely high load.
However, the server did not restart immediately, because it performed a time-consuming disk file system check (“fsck”) after the restart, causing an interruption in Web service and a delay in mail delivery for customers on that server (other servers were not affected).
The server finished its fsck check at 3:45 AM and is now working normally.
This is by far the longest outage we’ve experienced on a server in several years. I want to personally apologize to every affected customer: we don’t consider this kind of problem acceptable at all, and we deeply regret the downtime. We’ll be carefully reviewing this incident to see what we can learn from it in the future.
