Service outage May 6, 2011 (resolved)
May 6, 4:43 AM Pacific time: An outage at our primary data center caused a complete service interruption for all customers.
Update 5:08 AM: All services have been restored and are working normally.
May 6, 4:43 AM Pacific time: An outage at our primary data center caused a complete service interruption for all customers.
Update 5:08 AM: All services have been restored and are working normally.
The “fry” and “bender” Web servers will be restarted between 11:00 and 11:15 PM Pacific time tonight (Friday, April 29, 2011). This will cause a five-minute interruption of Web and e-mail service for customers on those servers.
Other servers will not be affected, and incoming mail will only be delayed, not lost.
8:52 PM Pacific time: We’re investigating a problem with the “fry” hosting server that’s requiring us to restart it; further details in a few minutes.
Update 9:42 PM Pacific time: The “fry” server was restarted, but a technician will be doing some maintenance on the server for approximately an hour. This will require a reboot, meaning the server will be unavailable for approximately 5 – 10 minutes. Web service will be unavailable during that time. E-mail service on that server also will be unavailable; delivery of new incoming mail will suspend during that time and then resume when the server comes back; no e-mail will be lost.
All others servers are unaffected.
Update 10:50 PM Pacific time: The “fry” web server will be rebooted in about 10 minutes, at approximately 11:00 PM Pacific time.
Update 11:10 PM Pacific time: The “fry” web server was successfully rebooted as planned. There may be more maintenance on the server this weekend; watch our blog or follow us on Twitter for updates.
Our FTP servers now support TLS/SSL encryption of FTP passwords, adding more security to FTP.
Confusingly, there are a variety of different SSL/TLS encryption schemes for FTP offered by various FTP clients. The one we support is the most widespread, known as “explicit TLS encryption” of the FTP command channel. It’s defined in RFC 4217.
Encryption is supported by many popular FTP clients, including the FileZilla FTP client. (The quickest way to use it in FileZilla is to put ftpes://ftp.tigertech.net in the QuickConnect “Host” box, then accept the “Unknown certificate”.)
Our primary data center experienced network routing problems between 2:06 PM and 2:49 PM Pacific time today (April 10, 2011).
During this time, packets from some (but not all) places on on the Internet were unreliable, causing connection problems. The data center technicians have resolved the issue, and all services are now working normally.
We don’t consider this normal or acceptable, and we sincerely apologize for the inconvenience this caused. (We do not yet have a full explanation from the data center about the root cause, but have requested one so that we can be sure it won’t recur.)
Beginning mid-day yesterday (Sunday February 6th), we received reports from a small number of customers that connections to our data center were slow. This was traced to a problem with a combination router/Ethernet switch in a cabinet in our data center corrupting some packets of data. The router has been replaced and the problem resolved.
One of the positive developments on the Internet over the last few years has been increased encryption of e-mail. The Internet is a hostile environment; sometimes your data goes through the servers and routers of companies you’ve never even heard of, or of governments you’ve heard of but don’t like. It makes sense to encrypt e-mail whenever possible.
We’ve supported encryption between our customers and our e-mail servers for a long time, protecting you from eavesdropping “hackers” when you use a WiFi connection at an Internet cafe, for example. But like most companies, we didn’t try encrypting outgoing e-mail after it left our servers or encrypting incoming e-mail from other servers. Although technical standards for doing that exist, they’re relatively new in Internet terms, and our original testing indicated it could cause problems with mail delivery due to many misconfigured servers on the Internet.
That’s changed: More recent testing indicates that it’s much more reliable, and other large companies like Gmail are starting to use it. Because of that, we now use strong TLS (SSL) encryption for inbound and outbound SMTP mail connections (“MX” mail delivery) wherever possible.
If you use Microsoft Outlook 2007 to read mail and you installed the December 2010 Outlook update, you might find that Outlook is slow to respond when you click between folders. Sometimes it can take several seconds.
This is caused by a bug in the Outlook update, not by a problem on our servers. To fix this, Microsoft recommends uninstalling the update for now.
Our business offices will be closed on Friday, December 31 to observe the US legal holiday. As always, our support staff will be providing same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until Monday, and telephone support (via callbacks) will be available only for urgent problems.
AOL.com had an outage lasting about 3 hours last night (from 11:24 PM Pacific time December 20 to 2:28 AM Pacific time December 21). This problem — a failure of AOL’s DNS servers — affected many people sending e-mail to AOL, and wasn’t related to our service (see this report and this one).
However, if you sent mail to an aol.com address during this time, your messages probably “bounced” with an error saying “Host or domain name not found. Name service error for name=aol.com”. If so, you should try sending the message again, and it will work normally. As always, we’ll continue to monitor AOL deliveries closely.