AOL mail delivery delays (resolved)
Update 5:00 PM December 27: AOL has resolved the problem described below. All delayed mail has been delivered, and all services are operating normally.
Update 5:00 PM December 27: AOL has resolved the problem described below. All delayed mail has been delivered, and all services are operating normally.
Between 11:32 and 11:36 AM Pacific time on November 11, 2013, the “web04” server experienced very high load that led to an outage for sites on that server. (Other servers were not affected.)
The high load was caused by a RAID array hard disk that suddenly became very slow to respond to requests. The disk was automatically reset and the server resumed working normally. We’re keeping a close eye on this and doing further testing, and will replace any defective hardware if necessary.
We sincerely apologize to our customers affected by this problem.
We’ve updated MySQL from version 5.1.66 to version 5.1.72. This upgrade was necessary for security reasons.
We’ve also updated the PHP 5.4 series from 5.4.20 to 5.4.21, and the PHP 5.5 series from 5.5.4 to 5.5.5, to fix several bugs.
These updates should be invisible to customers, but as always, don’t hesitate to contact us if you have any questions or concerns.
Between 11:59 AM and 1:13 PM Pacific time on October 23, 2013, there was an outage on the “web11” server due to a hardware problem. Other servers were not affected.
The hardware has been replaced and the server is running normally again. During the outage, incoming email was queued for delivery. All incoming email has now been delivered to the appropriate mailboxes. No email was lost.
At approximately 11:00 PM Saturday September 28, 2013, the “web06” server will be restarted. This will cause a short interruption of service for Web sites on that server lasting about 8 minutes.
Other servers will not be affected. Mail for customers on this server will be queued and delivered after a short delay.
We’ve received reports from some of our customers about errors or delays sending mail to or from Gmail on September 23, 2013.
This was caused by a general problem at Gmail that they’ve since resolved, and wasn’t related to our servers.
At 7:59 AM Pacific time on September 13, we tweeted this:
7:59 AM Pacific time: We're investigating a possible network outage affecting some customers. http://t.co/TySP4KGNY1
— Tiger Technologies (@Tigertech) September 13, 2013
However, this was a false alarm. The problem was in our independent external monitoring system, not a real problem with any of our servers or network.
At approximately 11:00 PM Saturday August 23, 2013, the “web06” server will be restarted. This will cause a short interruption of service for Web sites that server lasting about 8 minutes.
Other servers will not be affected. Mail for customers on this server will be queued and delivered after a short delay.
There’s been a lot of discussion recently about a critical Joomla security bug that allows “hackers” to upload malicious PHP script files to Joomla sites, then run them. This would allow hackers to use your site to send spam, or to replace any file on your Web site.
Although our customers running Joomla should always upgrade to the latest versions when available, we’ve also put rules in place to protect against this vulnerability.
Between 7:41 PM and 8:04 PM Pacific time August 4, 2013, one of our network providers experienced high packet loss, causing intermittently slow or unavailable connections for some customers.
All services are now running normally, and we’re working with that network provider to determine the root cause and prevent a recurrence. We apologize for the inconvenience this caused customers who were affected.
Update at 9:33 PM Pacific time: Our provider reports that their network in San Jose, California was overwhelmed by a large denial of service attack directed at another one of their customers (unrelated to our company). They believe they have resolved the problem.