Our automated monitoring systems didn’t detect any general problems, so the majority of customers were certainly unaffected — but we suspect that one of the “Internet backbones” between the affected customers and our data center had high packet loss during that period.

Both customers reported that the problem resolved itself by 7:45, and we haven’t received similar reports since, so there does not appear to be be an ongoing problem. We’ll continue to monitor it closely.

Other servers will not be affected, and incoming mail will only be delayed, not lost.

We apologize for the problem and for the short notice: the restart is necessary to replace a disk in the RAID array.

Update 11:03 PM Pacific time: The restart was completed with less than 3 minutes “downtime”.

This server had similar high load symptoms (but much more briefly) earlier this week. We took some steps to reduce the load then, but it appears those weren’t sufficient. We’re now taking much stronger action to ensure that this does not happen again.

We sincerely apologize to customers affected by this problem. We don’t consider it normal or acceptable, and we will make sure this isn’t a recurring issue.

We’ve investigated this, and it’s caused by Joomla assuming that PHP has a bug that makes it work incorrectly, when in fact it’s supposed to work differently (and is clearly documented to work differently). Older versions of PHP had this bug, but the new version doesn’t.

To help our customers work around this, we’ve “patched” PHP to intentionally reintroduce the old bug for now, thus keeping it “compatible” with Joomla. If you were having trouble with Joomla’s “Search Engine Friendly URLs”, it should be fixed.

We’ll provide more technical details (and a more robust long-term solution) in the near future.

Update: We’ve also reported this problem to the Joomla developers and suggested a solution.

• Thursday, February 4 (servers beginning with the letter “L”, such as “lrrr”)
• Friday, February 5 (all other servers beginning with letters “F-Z”, such as “farnsworth”)
• Saturday, February 6 (servers beginning with letters “A-E”, such as “amy”)

What’s changing?

Our servers use the Debian GNU/Linux operating system, which is updated with new versions every couple of years (just like Windows or Mac OS X). Last year, Debian was updated to version 5.0 (aka “lenny”), and since then, we’ve been running a “mixed” system. Some programs on each server have already been updated, but other programs still need updating.

We’re now finalizing the upgrade. This will make sure we can continue to use Debian security updates (they’ll soon be provided only for version 5.0) to keep our systems secure, and will give customers access to newer versions of some programs. Among the many updates:

• The Apache Web server will be upgraded from version 2.2.3 to 2.2.9
• The MySQL database server will be upgraded from 5.0.32 to 5.0.51a
• The PHP scripting language will be upgraded from 5.2.0 to 5.2.6
• The Perl scripting language will be upgraded from 5.8.8 to 5.10.0
• The default version of the Python scripting language will change from 2.4.4 to 2.5.2

Sharp-eyed readers will notice that these are not the newest, “cutting-edge” versions available. This is intentional; the philosophy of Debian Linux (and of our hosting platform) is to only use extremely well-tested, tried-and-true versions. We strongly value stability over new features, within reason, because that improves reliability for the majority of our customers.

However, we also want to emphasize that these versions also include security updates from later versions — for example, PHP includes the features from 5.2.6, but also includes security fixes from versions 5.2.7 to 5.2.12.

Why the downtime?

Unfortunately, a side-effect of the upgrade is about 7 total minutes of downtime.

We always strive to avoid any downtime, but in this case, it’s unavoidable: There’s a four-minute outage while the server is restarted for the new kernel version, and three outages of about a minute each while Apache, PHP, and MySQL are upgraded. During those three periods, we disable access to the Web server to prevent strange technical error messages appearing on your visitor’s screens.

Incoming mail may be delayed for a few minutes during the maintenance, but as always, no mail will be lost: it will just be queued and delivered a few minutes later.

Questions?

Please contact our support team if you have any questions. We appreciate your business!

Update: The maintenance was completed as scheduled.

First of all, the problem: Our primary internal PostgreSQL database server (which we use to manage all customer data) completely failed due to a nasty problem: a program on the server tried to delete every file on the disk.

The way this happened was surprising. A long time ago, our database server had a certain Debian GNU/Linux software “package” installed that — somehow — erroneously created a user with a “home directory” set to the top level of the entire disk (”/”). This caused no harm for almost two years… until we removed the obsolete, unused package while cleaning up odds and ends. The automatic package uninstallation script then began to delete all the files in the “/” directory. The script deleted several important files, including some necessary to “boot” that server, before we stopped it.

When the worst happens, we have a plan to recover from database failures:

• If the server hardware somehow fails (despite having redundant hard drives, power supplies, etc.), we can move the data disks to a spare server. This can be done in a few minutes.
• If the database files are corrupted or unavailable, we can restore from the correct onsite or offsite hourly backup, re-apply changes made since that backup (either manually or through database “journal logs”), and make it available again. This takes longer and is more tedious and error-prone.

In this case, we knew that some of the files on the server had been deleted, but it wasn’t clear which ones. That caused a significant delay while we contemplated whether it was safe to use the files of the existing database, or whether we needed to restore the database from backups. We eventually determined that it was safe to use the existing data, and restored things to a working state on a spare server.

Unfortunately, it took longer than we’d like to make sure all our systems that rely on databases were fully functional, and there were several things that we’d like to have avoided even if “My Account” was temporarily not working:

• The server problem also made our own www.tigertech.net Web site completely stop working for a short time.
• Our blog.tigertech.net pages relied on the same database server, so we couldn’t communicate the status well.
• Some people saw error messages when using the Webmail system to read certain messages. This shouldn’t happen; the Webmail system is designed to be independent of our database system.
• Our “Support” pages don’t work properly when the database is unavailable. This is actually a known issue; the pages are entirely database-driven from what amounts to a custom wiki on our end. On balance, this drawback is probably worth it; it allows our staff to make changes very easily (the ease of changes keeps it up-to-date and comprehensive), and we also customize pages for each viewer depending on what accounts they have with us.

“Learning is fun!”

That’s a sarcastic comment (from Futurama’s Bender character), of course. This kind of learning is actually not fun at all. On the other hand, one of the things we’re committed to is the idea of continuous improvement (fans of management systems probably know about “Kaizen”, or 改善). Each problem is an opportunity to make things better.

Any sufficiently advanced computer system is a web of interdependent parts that humans have difficulty understanding. Occasional failures are probably inevitable. But when we have a problem, we want to learn from it.

To that end, we’re making the following changes:

• We’ve added a check to ensure that no software package ever uses the top level of the disk as its home directory.
• We’ve made sure that blog.tigertech.net does not rely on our primary database system at all (it’s self-contained on servers in a completely separate data center), so we can use it to communicate with customers no matter what happens. If you normally reach our blog via a link on our main Web site, you should take a moment now to create a bookmark for blog.tigertech.net that you can use later (in case you somehow can’t reach our main site).
• We now have a simple way for any employee here to add a status update banner to every page on our site.
• We’re documenting improvements to our procedure for restoring database backups, avoiding several little annoyances that cost us valuable minutes.
• We’re fixing the software problem that causes some Webmail error messages during database failures.

And long-term, we’re looking into these:

• Adding the ability for our “Support” pages to work in a degraded, non-customized mode when the database is not available.
• Using the “streaming replication” and/or “hot standby” features that will be available in a forthcoming version of PostgreSQL to provide better database redundancy.
• Using a snapshot-able file system like Btrfs to more easily recover from processes that destroy data on a disk.
• Better segregation of changing data (databases, etc.) and unchanging data (program files and the root level of disks) on our servers, putting the unchanging data on partitions that are read-only and undeletable. The runaway file deletion process would not have caused problems if the root file system on this server was mounted in “read-only” mode.

We’re always trying to improve, and we regret that our best side wasn’t on display last Friday. We apologize for the inconvenience caused to our customers.

As a result, for customers on the “amy” server (only), Web site service and the ability to read incoming e-mail will be unavailable for approximately five minutes. Customers on other servers will not be affected.

No e-mail will be lost, of course; incoming mail will just be slightly delayed.

We apologize for the inconvenience this causes. This maintenance is necessary to replace a possibly faulty hard disk in the RAID array (we’re being cautious and taking action before it becomes a problem).

Update: the server was restarted and there were no problems. Customers only saw an impact for less than five minutes, as expected.

No customer data or Web sites were affected, and no e-mail was lost.

All systems are running again, so no one should see any problems — please let us know if you do! Some things are running on backup systems, so we’re working on finishing up the fixes and restoring everything to its normal status.

This was an unplanned and fairly horrible (and embarrassing) problem. This is the first time our own account management database has completely failed in the 10 years we’ve been providing Web hosting service. Obviously, we consider this sort of thing to be unacceptable, and we sincerely apologize for any inconvenience caused.

We will update this blog in the near future with more details.

Update: We have a new post which explains the problem (and solutions) in more detail.

A bug in the SpamAssassin software caused SpamAssassin scores to be incorrectly calculated for the first few days of this year: the scores were higher than they should have been.

We don’t use SpamAssassin scores as part of our spam filtering system, so this doesn’t affect most of our customers at all. However, some customers may have added custom rules to their mail programs that examine the SpamAssassin headers. If you do that, and you’ve directed high-scoring messages into a spam folder in your mail program that you don’t usually look at, you should look at all messages received between January 1 and the morning of January 6 to verify that they are actually spam.

Just so it’s clear, this bug affected everyone using SpamAssassin with any ISP or hosting company, not just our customers. That said, this bug unfortunately persisted on some of our servers for longer than it should have done, due to a technical issue with the way Debian Linux distributes SpamAssassin updates. We apologize for any problems this caused our customers; the problem was resolved on all servers early today.

No e-mail will be lost, of course; incoming mail on those servers will just be slightly delayed.

We apologize for the inconvenience this causes. This maintenance is necessary to install an updated “kernel” on all of our servers for security reasons.