The disk load on the “web05” server was extremely high between 2:30 and 2:42 AM Pacific time Saturday February 4, causing some downtime during that period for sites using that server. Other servers were not affected.
A couple of days ago, one of our Web servers became unstable for an unknown reason and needed to be restarted. This is rare: on average, this happens less than once every five years of uptime per server, so we took it very seriously and launched an investigation.
What we found was that the owner of one of the sites on that server made a mistake that allowed attackers to run their own scripts. That’s all too common, unfortunately, but usually only the single site is affected by this kind of thing. What was surprising in this case was that the script used a previously unknown method of causing problems for other sites running on the server.
As a result of this investigation, we’ve made several changes to our systems to ensure the problem won’t recur. The rest of this post has a detailed technical description of the problem in case it’s useful for others.
Our “web07” server needed restarting at 11:36 AM Pacific time on February 1, 2012, because it had been intermittently unable to run some PHP scripts for 22 minutes.
The restart resolved the immediate problem, and a followup post explains what happened and the changes we made to prevent it from happening again.
Our business offices will be closed on Monday, January 2 to observe the US legal holiday. As always, our support staff will be providing same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until Tuesday, and telephone support (via callbacks) will be available only for urgent problems.
Our business offices will be closed on Monday, December 26 to observe the US legal holiday. As always, our support staff will be providing same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until Tuesday, and telephone support (via callbacks) will be available only for urgent problems.
Update: The problems described below were resolved by Comcast around 11:00 AM Pacific time and have not recurred since. We’re cautiously marking this issue closed, but continuing to monitor it.
We’ve received scattered reports of high “packet loss” to a few Comcast locations (but not most). Packet loss can cause pages to load slowly in some cases.
Between 5:34 PM and 6:10 PM Pacific time December 12, many customers experienced a complete outage of their sites (and of our own www.tigertech.net and mail.tigertech.net sites).
This was caused by the failure of a hardware Ethernet switch in one of our server cabinets, cutting off all access to the servers that plug into it. The Ethernet switch began working after being physically unplugged and plugged in again, but since we do not know why it failed, it will be completely replaced tonight as a result of this incident.
This is the same model of Ethernet switch that we’ve been using in all our cabinets for years, so we don’t believe it is a general problem with the hardware in question.
We sincerely apologize for this incident. We take reliability seriously, and we don’t consider it acceptable.
Update 1:20 AM: The failed Ethernet switch was replaced with no further downtime.
Our business offices will be closed on Thursday, November 24 to observe the US legal holiday for Thanksgiving.
As always, we’ll provide same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until Friday, and telephone support (via callbacks) will be available only for urgent problems.
At approximately 8:00 PM Pacific time on November 12, 2011, the “web06” Web server will be restarted.
As a result, for customers on the “web06” server (only), Web site service and the ability to read incoming e-mail will be unavailable for approximately five minutes. Customers on other servers will not be affected.
We’ve been notified by an upstream network provider that they will be performing router firmware upgrades on Saturday, November 5, 2011 between 5:00 and 6:00 PM Pacific time. In the worst case, some customers may see a network interruption of up to five minutes during this period.
Please accept our apologies for any interruption this causes; we’re told that the upgrades are necessary to prevent possible network problems.
(This is the rescheduled time for an original announcement that was postponed.)
Update: The maintenance was completed with approximately three minutes downtime for affected connections.
