PHP 5 updated

We’ve installed a PHP 5 security update. Customers should not notice any changes; the updates just fix several security issues in PHP 5.

Mail with blank contents for some customers (resolved)

Due to a problem with our spam filtering system, some customers received blank incoming messages between 11:35 and 11:53 AM (Pacific time) this morning.

A non-blank copy of these messages was also properly delivered (although with a delay), so no mail is missing.

We have permanently fixed the underlying cause of the problem, and we apologize for the concern and annoyance this caused.

Rsync 3 available

We’ve updated rsync to version 3 on our servers. (We’re using Debian version 3.0.3-2, which includes patches from rsync 3.0.4.)

Rsync 3 is significantly faster than previous versions for recursive file transfers (which we use in our backup system). However, the new version is backward compatible with rsync 2.x, so users shouldn’t notice any changes or problems, even if you haven’t updated your own copy of rsync.

Brief scheduled maintenance on Saturday, September 20 (completed)

At approximately 11:00 PM Pacific time this Saturday night (September 20), all Tiger Technologies servers will be restarted. As a result, customer Web sites and e-mail service will be unavailable for three to five minutes.

No e-mail will be lost, of course; incoming mail will just be delayed for a few minutes.

This brief maintenance is necessary to upgrade the operating system “Linux kernel” to a newer version for security reasons. We apologize for the inconvenience this causes.

Update: the maintenance was completed with less than three minutes “downtime” per server.

Tip for users of Rails 1.1.6 and earlier

If you have a Ruby on Rails application that originally used Rails 1.1.6 or earlier, you might have trouble after yesterday’s Rails update (which also updated several other Ruby “gems”, including the “RubyGems” gem itself).

Read the rest of this entry »

Farnsworth server restarted (resolved)

The network interface on the “farnsworth” Web server stopped responding at 6:38 AM Pacific time today, and the server needed to be manually restarted by our data center staff. The server was unavailable for 17 minutes, causing an interruption of service for Web sites on that server. It also prevented users of that server from reading incoming e-mail (such e-mail was delayed and delivered after the outage).

Other servers were not affected.

We sincerely apologize to anyone affected by this problem.

Ruby on Rails updated to version 2.1.1

We’ve updated the default version of Ruby on Rails on our servers to version 2.1.1.

Read the rest of this entry »

Calculon server problem (resolved)

The “calculon” Web server needed to be restarted at 12:40 AM Pacific time this morning due to extremely high load.

However, the server did not restart immediately, because it performed a time-consuming disk file system check (“fsck”) after the restart, causing an interruption in Web service and a delay in mail delivery for customers on that server (other servers were not affected).

The server finished its fsck check at 3:45 AM and is now working normally.

This is by far the longest outage we’ve experienced on a server in several years. I want to personally apologize to every affected customer: we don’t consider this kind of problem acceptable at all, and we deeply regret the downtime. We’ll be carefully reviewing this incident to see what we can learn from it in the future.

Labor Day 2008 holiday hours

Our business offices will be closed on Monday, September 1 to observe the US legal holiday. As always, we’ll provide same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until the next day, and telephone support (via callbacks) will be available only for urgent problems.

Routing problem for some Comcast customers (resolved)

Between 5:11 and 5:46 PM Pacific time today, some people who reach our servers via an “Internet backbone” called Global Crossing (including some Comcast cable customers) were unable to connect to our data center. Other users weren’t affected.

Global Crossing has apparently corrected the problem, and everything is now operating normally. We’ll continue to monitor this issue closely.

Read the rest of this entry »