Beginning mid-day yesterday (Sunday February 6th), we received reports from a small number of customers that connections to our data center were slow. This was traced to a problem with a combination router/Ethernet switch in a cabinet in our data center corrupting some packets of data. The router has been replaced and the problem resolved.
Between 3:29 PM Pacific time and 3:33 PM Pacific time, our monitoring systems detected that most Internet users could not connect to our primary data center. E-mail delivery was properly queued up and delayed during this period.
We will follow up with the data center team, but the problem appears to have been resolved, and all services are operating normally. We’re continuing to monitor it closely, and we sincerely apologize for the inconvenience this caused our customers.
Updated: connectivity was lost for four minutes because the data center was fighting off a severe DoS attack.
AOL.com had an outage lasting about 3 hours last night (from 11:24 PM Pacific time December 20 to 2:28 AM Pacific time December 21). This problem — a failure of AOL’s DNS servers — affected many people sending e-mail to AOL, and wasn’t related to our service (see this report and this one).
However, if you sent mail to an aol.com address during this time, your messages probably “bounced” with an error saying “Host or domain name not found. Name service error for name=aol.com”. If so, you should try sending the message again, and it will work normally. As always, we’ll continue to monitor AOL deliveries closely.
Between 2:35 PM Pacific time and 3:03 PM Pacific time, our monitoring systems detected that connections to our primary data center from some locations on the Internet were slow or failing due to problems at an Internet “backbone”. Connections from other locations were unaffected.
We’re waiting for a full report from the data center team, but the problem appears to have been resolved, and all services are operating normally. We’re continuing to monitor it closely, and we sincerely apologize for the inconvenience this caused our customers.
Our primary data center had another power interruption this morning at 7:28 am (Pacific time). All of our servers lost power and then had it restored, thus rebooting them. All customer web sites were unavailable during this time. Incoming email would have simply been delayed during the downtime, not lost. When the servers came back online e-mail may have seemed sluggish to some customers for a while but this should also be fixed now.
This incident follows another power incident the previous Saturday night. We are working with the data center to get more details, including an estimate of when they will have replaced any faulty equipment. We will update this post as more information becomes available.
Update Nov. 29: The final data center report is that on the night of November 20, lightning strikes damaged both of the redundant UPS systems, interrupting data center power for a few seconds. The UPS manufacturer scheduled replacements for November 23, but another PG&E utility power interruption lasting a few seconds occurred that morning before it was finished. The UPS manufacturer has since replaced all damaged parts, restoring full redundancy. In addition, the UPS manufacturer has overhauled each unit, replacing and upgrading other parts to increase robustness. We take this very seriously — it’s at the core of what we do — and we will continue to work with the data center to ensure that their infrastructure meets our high standards.
A major power failure at our primary data center in Fremont, California, caused a complete outage for nearly all services beginning at 8:32 PM Pacific time Saturday night. It lasted between six and 13 minutes, depending on the server. Only our blog and redundant DNS infrastructure was unaffected.
All services are now fully operational; please don’t hesitate to contact us if you have any questions. We sincerely apologize for the inconvenience this caused our customers.
The aol.com mail servers have been having problems for the last 24 hours, according to the AOL blog and the AOL Twitter feed.
At approximately 10:00 PM Pacific time tonight, October 23, the “flexo” Web server will be restarted.
As a result, for customers on the “flexo” server (only), Web site service and the ability to read incoming e-mail will be unavailable for approximately five minutes. Customers on other servers will not be affected.
We’ve fixed a bug in our Webmail system that could, in rare cases, make Japanese language symbols display incorrectly. This change shouldn’t affect anything else, but as always, feel free to contact us if you have any trouble.
Between 11:00 PM and 11:30 PM Pacific time this Friday, September 24, some of our hosting servers will be restarted. As a result, some customers will find that Web site service and the ability to read incoming e-mail will be unavailable for approximately five minutes at some point during this maintenance “window”.
