Between 5:34 PM and 6:10 PM Pacific time December 12, many customers experienced a complete outage of their sites (and of our own www.tigertech.net and mail.tigertech.net sites).
This was caused by the failure of a hardware Ethernet switch in one of our server cabinets, cutting off all access to the servers that plug into it. The Ethernet switch began working after being physically unplugged and plugged in again, but since we do not know why it failed, it will be completely replaced tonight as a result of this incident.
This is the same model of Ethernet switch that we’ve been using in all our cabinets for years, so we don’t believe it is a general problem with the hardware in question.
We sincerely apologize for this incident. We take reliability seriously, and we don’t consider it acceptable.
Update 1:20 AM: The failed Ethernet switch was replaced with no further downtime.
Our previously-announced move to the new data center is continuing. The old data center was hit with a lot of network attacks throughout the day today. We are very glad to be moving to the new center!
E-mail should be fully working for everyone. Many customers have been moved to the new data center, and we’ll be continuing to move the remaining sites as quickly as possible (while of course being very careful about data integrity and site uptime). Sites which have not yet been moved are still working at the old data center. The old data center has been up for several hours now with only very occasional interruptions (due to attacks). They might still experience occasional bouts of downtime if the attacks continue.
If you want to know if your Web site has been moved to a server at the new data center, please see the update at the bottom of this page.
Between 10:32 AM and 10:47 AM Pacific time this morning (October 3), our monitoring systems detected high “packet loss” from one “network backbone”, which may have caused slow connections or timeouts for some customers. The monitoring systems show that this issue is resolved.
The data center that experienced network problems earlier today has just informed us that they’ll be performing emergency maintenance on all their network routers tonight (Thursday, September 29, 2011) between 6:00 and 7:00 PM Pacific time.
During that hour, there may be up to five minutes total of network connectivity problems that makes some sites load slowly or fail to load.
Read the rest of this entry »
In an apparent continuation of last night’s incident, many sites we host were intermittently unavailable between 12:01 PM and 1:20 PM Pacific time today (September 29, 2011). This also caused slow mail delivery and reduced spam filtering effectiveness until around 2:00 PM (no mail was lost, of course).
All systems are operating normally as of 2:15 PM.
Read the rest of this entry »
A problem at our old data center (the one we’re moving sites from this month) caused some sites to be intermittently unavailable between 10:22 and 10:47 PM Pacific time.
Read the rest of this entry »
Over the next four weeks, we’ll be migrating customer Web sites to upgraded servers. The servers have updated software (and upgraded hardware in some cases), and are also located in a data center with increased power reliability.
For most customers, these changes will be completely unnoticeable. However, a very small number of customers might notice software differences or experience up to five minutes total of “downtime” at some point. We recommend reading through this entire post for details.
Read the rest of this entry »
Between 6:00 AM and 6:29 AM Pacific time August 7, 2011, all services were unavailable due to a power failure at our primary data center.
The problem was resolved for most servers by 6:29 AM, and for all servers except the “amy” server by 6:53 AM. The “amy” server needed extra manual intervention, and was working by 7:55 AM. All services are now operating normally.
Any e-mail that arrived during the outage was queued at our secondary data center and delivered as soon as the outage ended.
We sincerely apologize for this problem. We know you count on us for reliability, and we don’t consider this acceptable, especially since the data center has had previous power problems this year. However, this incident had a different root cause. It wasn’t a utility power failure that the redundant UPS systems didn’t handle, but was instead caused by a circuit breaker incorrectly “tripping” to prevent the power output of the UPS systems from reaching the server cabinets.
Update 4:15 PM: We have received an incident report from the data center indicating that they are working to replace the affected part of the UPS system to prevent further problems.
We’ve installed a PHP 5 security update. Customers should not notice any changes; the update just fixes several security issues in PHP 5.
Some Comcast users in California had trouble connecting to some of our servers beginning at around 7 AM Pacific May 21, 2011. Non-Comcast users were not affected at all, and even people who were affected were able to reach some of our servers with no trouble.
This was caused by a technical problem at Comcast, and not related to us specifically. It appears Comcast was incorrectly filtering some combinations of IP addresses and ports in one of their California network routers, preventing their customers from reaching some sites.
The issue was apparently resolved by Comcast at 10:07 AM Pacific time, and we are not aware of any ongoing problems. As always, don’t hesitate to contact us if you have any trouble.
