Update: The problems described below were resolved by Comcast around 11:00 AM Pacific time and have not recurred since. We’re cautiously marking this issue closed, but continuing to monitor it.
We’ve received scattered reports of high “packet loss” to a few Comcast locations (but not most). Packet loss can cause pages to load slowly in some cases.
Between 5:34 PM and 6:10 PM Pacific time December 12, many customers experienced a complete outage of their sites (and of our own www.tigertech.net and mail.tigertech.net sites).
This was caused by the failure of a hardware Ethernet switch in one of our server cabinets, cutting off all access to the servers that plug into it. The Ethernet switch began working after being physically unplugged and plugged in again, but since we do not know why it failed, it will be completely replaced tonight as a result of this incident.
This is the same model of Ethernet switch that we’ve been using in all our cabinets for years, so we don’t believe it is a general problem with the hardware in question.
We sincerely apologize for this incident. We take reliability seriously, and we don’t consider it acceptable.
Update 1:20 AM: The failed Ethernet switch was replaced with no further downtime.
At approximately 8:00 PM Pacific time on November 12, 2011, the “web06” Web server will be restarted.
As a result, for customers on the “web06” server (only), Web site service and the ability to read incoming e-mail will be unavailable for approximately five minutes. Customers on other servers will not be affected.
We’ve been notified by an upstream network provider that they will be performing router firmware upgrades on Saturday, November 5, 2011 between 5:00 and 6:00 PM Pacific time. In the worst case, some customers may see a network interruption of up to five minutes during this period.
Please accept our apologies for any interruption this causes; we’re told that the upgrades are necessary to prevent possible network problems.
(This is the rescheduled time for an original announcement that was postponed.)
Update: The maintenance was completed with approximately three minutes downtime for affected connections.
We’ve been notified that the maintenance previously scheduled for tonight (November 3, 2011) has been canceled and will be rescheduled for a future time.
As a followup to our previous posts about the move to a new primary data center, we want to confirm to our customers that the change was successfully completed.
Due to unrelated network outages at the old data center, we accelerated the original schedule mentioned in that post. Almost all customer sites were moved by October 7, and the remainder (a small handful of customer sites that needed manual intervention due to old software that was incompatible with the Debian Linux software update) were moved as of October 18. Everything is, and has been, working normally.
I want to again take the time to apologize to our customers for the service interruptions that occurred because of the original power problem and the later network problem. They weren’t acceptable. We know you count on us for your success, and we’re constantly working to improve reliability.
Between 10:32 AM and 10:47 AM Pacific time this morning (October 3), our monitoring systems detected high “packet loss” from one “network backbone”, which may have caused slow connections or timeouts for some customers. The monitoring systems show that this issue is resolved.
The data center that experienced network problems earlier today has just informed us that they’ll be performing emergency maintenance on all their network routers tonight (Thursday, September 29, 2011) between 6:00 and 7:00 PM Pacific time.
During that hour, there may be up to five minutes total of network connectivity problems that makes some sites load slowly or fail to load.
In an apparent continuation of last night’s incident, many sites we host were intermittently unavailable between 12:01 PM and 1:20 PM Pacific time today (September 29, 2011). This also caused slow mail delivery and reduced spam filtering effectiveness until around 2:00 PM (no mail was lost, of course).
All systems are operating normally as of 2:15 PM.
The “elzar” Web hosting server experienced very high load between 9:07 and 9:14 AM Pacific time this morning (September 27, 2011), causing sites on that server to load slowly during those seven minutes. Other servers were not affected.
This was caused by a distributed denial of service (“DDOS”) attack against a site on that server. We manually blocked the attackers to resolve it, and we’re continuing to monitor it closely to make sure it doesn’t recur.
