Stability improvements for a server memory problem

A couple of days ago, one of our Web servers became unstable for an unknown reason and needed to be restarted. This is rare: on average, this happens less than once every five years of uptime per server, so we took it very seriously and launched an investigation.

What we found was that the owner of one of the sites on that server made a mistake that allowed attackers to run their own scripts. That’s all too common, unfortunately, but usually only the single site is affected by this kind of thing. What was surprising in this case was that the script used a previously unknown method of causing problems for other sites running on the server.

As a result of this investigation, we’ve made several changes to our systems to ensure the problem won’t recur. The rest of this post has a detailed technical description of the problem in case it’s useful for others.

Read the rest of this entry »

web07 server restart on February 1, 1012 (resolved)

Our “web07” server needed restarting at 11:36 AM Pacific time on February 1, 2012, because it had been intermittently unable to run some PHP scripts for 22 minutes.

The restart resolved the immediate problem, and a followup post explains what happened and the changes we made to prevent it from happening again.

Comcast routing problems December 16 2011 (resolved)

Update: The problems described below were resolved by Comcast around 11:00 AM Pacific time and have not recurred since. We’re cautiously marking this issue closed, but continuing to monitor it.

We’ve received scattered reports of high “packet loss” to a few Comcast locations (but not most). Packet loss can cause pages to load slowly in some cases.

Read the rest of this entry »

Outage December 12, 2011 (resolved)

Between 5:34 PM and 6:10 PM Pacific time December 12, many customers experienced a complete outage of their sites (and of our own www.tigertech.net and mail.tigertech.net sites).

This was caused by the failure of a hardware Ethernet switch in one of our server cabinets, cutting off all access to the servers that plug into it. The Ethernet switch began working after being physically unplugged and plugged in again, but since we do not know why it failed, it will be completely replaced tonight as a result of this incident.

This is the same model of Ethernet switch that we’ve been using in all our cabinets for years, so we don’t believe it is a general problem with the hardware in question.

We sincerely apologize for this incident. We take reliability seriously, and we don’t consider it acceptable.

Update 1:20 AM: The failed Ethernet switch was replaced with no further downtime.

Brief scheduled maintenance on web06 server (completed)

At approximately 8:00 PM Pacific time on November 12, 2011, the “web06” Web server will be restarted.

As a result, for customers on the “web06” server (only), Web site service and the ability to read incoming e-mail will be unavailable for approximately five minutes. Customers on other servers will not be affected.

Read the rest of this entry »

Brief network maintenance November 5, 2011 (completed)

We’ve been notified by an upstream network provider that they will be performing router firmware upgrades on Saturday, November 5, 2011 between 5:00 and 6:00 PM Pacific time. In the worst case, some customers may see a network interruption of up to five minutes during this period.

Please accept our apologies for any interruption this causes; we’re told that the upgrades are necessary to prevent possible network problems.

(This is the rescheduled time for an original announcement that was postponed.)

Update: The maintenance was completed with approximately three minutes downtime for affected connections.

Brief network maintenance November 3 (postponed)

We’ve been notified that the maintenance previously scheduled for tonight (November 3, 2011) has been canceled and will be rescheduled for a future time.

Read the rest of this entry »

Data center move complete

As a followup to our previous posts about the move to a new primary data center, we want to confirm to our customers that the change was successfully completed.

Due to unrelated network outages at the old data center, we accelerated the original schedule mentioned in that post. Almost all customer sites were moved by October 7, and the remainder (a small handful of customer sites that needed manual intervention due to old software that was incompatible with the Debian Linux software update) were moved as of October 18. Everything is, and has been, working normally.

I want to again take the time to apologize to our customers for the service interruptions that occurred because of the original power problem and the later network problem. They weren’t acceptable. We know you count on us for your success, and we’re constantly working to improve reliability.

New data center status 2011-10-03

Our previously-announced move to the new data center is continuing. The old data center was hit with a lot of network attacks throughout the day today. We are very glad to be moving to the new center!

E-mail should be fully working for everyone. Many customers have been moved to the new data center, and we’ll be continuing to move the remaining sites as quickly as possible (while of course being very careful about data integrity and site uptime). Sites which have not yet been moved are still working at the old data center. The old data center has been up for several hours now with only very occasional interruptions (due to attacks). They might still experience occasional bouts of downtime if the attacks continue.

If you want to know if your Web site has been moved to a server at the new data center, please see the update at the bottom of this page.

Old data center problems continuing (resolved)

We are continuing to experience intermittent problems at Hurricane Electric, the data center that we’ve been using until now. Fortunately, as previously mentioned, we are in the process of moving to a new data center.

Some customers have already been moved to the new data center, and they have not seen any of these problems. We are moving the remaining sites as quickly as possible.

We are extremely upset with the attacks that have been hitting Hurricane Electric, and consider these outages to be unacceptable. It definitely appears that Hurricane Electric is being specifically targeted. You can watch their Twitter feed to see information about the latest attacks and the steps they’re taking to fix the attacks, as well as what they’re doing to try to protect themselves against further ones. While it’s undoubtedly of little comfort, it’s worth noting that these attacks are not directed at our customers or at us, but at Hurricane Electric in general. You can see that other large companies hosted at Hurricane Electric are also being affected, such as Linode.

We apologize for the inconvenience these outages have caused, and hope to have all Web sites moved to the new data center very soon.