2011 server upgrades

Over the next four weeks, we’ll be migrating customer Web sites to upgraded servers. The servers have updated software (and upgraded hardware in some cases), and are also located in a data center with increased power reliability.

For most customers, these changes will be completely unnoticeable. However, a very small number of customers might notice software differences or experience up to five minutes total of “downtime” at some point. We recommend reading through this entire post for details.

• Linux software upgrades
• Hardware upgrades
• Data center reliability improvements
• Will there be any downtime?
• Will anything else change?
• Have questions?

Linux software upgrades

First of all, we’re updating the software on all our Web servers, which we do every couple of years. Our servers use the Debian GNU/Linux operating system, which releases new versions from time to time (just like Windows or Mac OS X).

Earlier this year, Debian released version 6.0 (aka “squeeze”), so it’s time for us to upgrade. This will make sure we can continue to use Debian security updates to keep our systems safe, and will give customers access to newer versions of some programs. Among the many updates:

• The Apache Web server will be upgraded from version 2.2.9 to 2.2.16
• The MySQL database server will be upgraded from 5.0.51a to 5.1.49
• The Perl scripting language will be upgraded from 5.10.0 to 5.10.1
• The default version of the Python scripting language will change from 2.5.2 to 2.6.6

Sharp-eyed readers will notice that these are not the newest, “cutting-edge” versions available. This is intentional; the philosophy of Debian Linux (and of our hosting platform) is to only use extremely well-tested, tried-and-true versions. We strongly value stability over new features, within reason, because that improves reliability for the majority of our customers.

However, we want to emphasize that these versions also include security updates from later versions — for example, Apache includes the features from 2.2.16, but also includes security fixes from versions 2.2.17 through 2.2.21.

By the way, we’re intentionally not changing the version of the PHP scripting language with this upgrade. We’ll be introducing the newer PHP 5.3 series as a separate future update to minimize the number of changes that occur at once.

All modern software scripts you may have installed on your Web site should be compatible with the update. As always, using the latest version of any software you’ve installed is the way to ensure that your site is secure and doesn’t experience any compatibility problems. If you’re using an outdated version of a script you’ve installed on your site and you’ve been putting off an upgrade, now is a great time to take care of that.

Hardware upgrades

We’re also finishing a years-long project to replace older servers with newer, more powerful hardware. Most sites are already hosted on newer servers with dual quad-core (8 CPU) Xeon processors, 16 GB of RAM, and a solid state drive in the RAID array providing MySQL database storage. After this upgrade, those are the only hosting servers we’ll have, so all our customers will see the fastest possible site performance.

Data center reliability improvements

We’ve been using the same primary data center (the building that stores most of our physical servers) for more than ten years, with generally good results.

Earlier this year, though, that data center had several electrical power problems that caused unacceptable interruptions in service. While it’s been stable for a couple of months, the data center engineers failed to convince us that problems can’t recur.

We have business relationships with several data center companies, and as a result of these problems, we’re now using a different building as our primary data center. Our own site and recently opened customer sites are already located there; older sites are being “migrated” as part of the changes mentioned above.

The data center is managed by CoreSite, a large company with an excellent track record. The facility was originally designed for telephone company service, which has the highest power availability standards in the industry. In addition, the data center is the home of MAE-West, one of the largest Internet traffic exchange points in the world. This gives us a wider choice of network providers and peering connections to make sure we can continue to meet the growing bandwidth demands of modern Web sites.

We’re confident that this will allow us to provide the reliability you expect and need for your Web site and e-mail service.

Will there be any downtime?

With these changes, we’re “migrating” Web sites to upgraded servers. We do this using the same well-tested procedure we’ve always used to move sites when we need to balance server loads or retire obsolete hardware; it allows us to move most sites “live” with no service interruption at all.

If you have a very large, busy MySQL database, though, your site may be unavailable for a few seconds — or even a few minutes in the worst case. That’s because the migration process works like this:

1. It first copies all the files and databases for a site to a new server.
2. Next, it queues incoming connections so that they don’t reach the old server. This is done in such a way that Web browsers won’t see any error messages if the following steps finish quickly.
3. It then re-copies any files or databases that changed since step 1.
4. Finally, it routes the queued connections to the new server.

For most sites, the amount of data changed in step 3 is small: it completes in a second or two, so any visitors to the site at that moment would see nothing but a short delay in loading a page.

But if you have a large database that’s constantly changing, the migration may need to copy over an entire database again in step 3. On average, this takes about 10 seconds per 70 MB of database tables, so if you have a very large database, the delay can be long enough that some visitors might see “timeout on server” errors before step 4. The largest database we recommend is 2 GB (2000 MB), so the worst case is about 5 minutes of unavoidable downtime. (This is still less than the downtime that would result if we upgraded the Linux software “in place” on the existing servers, as we used to.)

We automatically move sites with large databases outside of normal business hours (usually between 9 PM and midnight Pacific time) to minimize the impact. However, if your site has databases over 100 MB in size (most don’t) and you’d like us to migrate your site at a time when you feel the site traffic is slower, don’t hesitate to contact us. We can also display a custom “this site is undergoing maintenance” type of message during the migration if you wish.

Will anything else change?

One other thing we should mention for technically advanced users is that the IP address of your site will change. However, this won’t cause any problems, because we update the DNS entries for your site as it’s migrated. We also “proxy” any connections that arrive at the old IP address so that the DNS “time to live” doesn’t cause any issues.

(This is why we insist on handling the DNS nameservers for sites we host, even though we have to turn away some business as a result. When we manage the DNS entries, we can make sure you won’t experience downtime if your site’s IP address changes.)

Have questions?

Please contact our support team if you have any questions. We appreciate your business!