At approximately 9:30 PM Pacific time, all of our servers began to experience a large “distributed denial of service” (DDoS) attack via attempts to login to blogs using the standard WordPress wp-login.php script. This attack was very broad: it attacked thousands of sites across all of our servers, and it came from a huge number of IP addresses.
Processing these requests caused the overall load on all servers to increase. On “web04” the increase was enough to cause the server to start returning “503” errors for Web page requests.
Our servers already have a set of rules to protect against attacks on wp-login.php, but the rules were not quite sufficient to block tonight’s attack. We added a new rule to match tonight’s attack, and it fixed the problem.
We apologize for the time that the “web04” server returned 503 errors. As you can see by reviewing our blog posts we try to be very proactive to protect our customers’ WordPress sites, and hope that the new security rule will prevent future attacks with the same characteristics.
Update 5:00 PM December 27: AOL has resolved the problem described below. All delayed mail has been delivered, and all services are operating normally.
Read the rest of this entry »
Between 11:32 and 11:36 AM Pacific time on November 11, 2013, the “web04” server experienced very high load that led to an outage for sites on that server. (Other servers were not affected.)
The high load was caused by a RAID array hard disk that suddenly became very slow to respond to requests. The disk was automatically reset and the server resumed working normally. We’re keeping a close eye on this and doing further testing, and will replace any defective hardware if necessary.
We sincerely apologize to our customers affected by this problem.
We’ve updated MySQL from version 5.1.66 to version 5.1.72. This upgrade was necessary for security reasons.
We’ve also updated the PHP 5.4 series from 5.4.20 to 5.4.21, and the PHP 5.5 series from 5.5.4 to 5.5.5, to fix several bugs.
These updates should be invisible to customers, but as always, don’t hesitate to contact us if you have any questions or concerns.
Between 11:59 AM and 1:13 PM Pacific time on October 23, 2013, there was an outage on the “web11” server due to a hardware problem. Other servers were not affected.
The hardware has been replaced and the server is running normally again. During the outage, incoming email was queued for delivery. All incoming email has now been delivered to the appropriate mailboxes. No email was lost.
Read the rest of this entry »
At approximately 11:00 PM Saturday September 28, 2013, the “web06” server will be restarted. This will cause a short interruption of service for Web sites on that server lasting about 8 minutes.
Other servers will not be affected. Mail for customers on this server will be queued and delivered after a short delay.
Read the rest of this entry »
At 7:59 AM Pacific time on September 13, we tweeted this:
However, this was a false alarm. The problem was in our independent external monitoring system, not a real problem with any of our servers or network.
Read the rest of this entry »
At approximately 11:00 PM Saturday August 23, 2013, the “web06” server will be restarted. This will cause a short interruption of service for Web sites that server lasting about 8 minutes.
Other servers will not be affected. Mail for customers on this server will be queued and delivered after a short delay.
Read the rest of this entry »
Between 7:41 PM and 8:04 PM Pacific time August 4, 2013, one of our network providers experienced high packet loss, causing intermittently slow or unavailable connections for some customers.
All services are now running normally, and we’re working with that network provider to determine the root cause and prevent a recurrence. We apologize for the inconvenience this caused customers who were affected.
Update at 9:33 PM Pacific time: Our provider reports that their network in San Jose, California was overwhelmed by a large denial of service attack directed at another one of their customers (unrelated to our company). They believe they have resolved the problem.
Between 9:50 AM and 9:59 AM Pacific time August 1, 2013, the “web08” server experienced a distributed denial of service (DDoS) attack, causing Web pages to load slowly or not at all.
The attacks are continuing, but we are blocking the IP addresses involved and all services are operating normally. We’re monitoring things closely to avoid problems, and we apologize for the inconvenience this caused customers.
