Beginning at 3:00 PM Pacific time February 5, a server on our network was the target of an extremely high volume DNS amplification denial of service attack. The inbound network data exceeded 11.6 Gbps, which is an extremely large amount — large enough to exceed the 10 Gpbs capacity of our upstream Ethernet switches and cause our entire network to slow down dramatically.
This affected all servers for about 19 minutes, until we and our network partners began discarding (“null routing”) all traffic targeted at that server. This fixed the problem for the rest of our network, but still left sites on the “web11” server unavailable.
To solve that, the IP addresses of all sites on the web11 server have been changed to new IP addresses that are working correctly and are not under attack. This was completed by 3:44 PM, and all sites on all servers are now working properly.
If the attackers target another IP address, we’re ready to immediately block that one, too. If that does happen, the way we’ve redistributed the IP addresses, in combination with previous analysis we’ve done on this attack, will allow us to immediately know which site is under attack. (It’s otherwise hard to determine which IP address is involved, because the type of attack we’re seeing targets only an IP address and not a specific Web site name.) That site will then be moved off our main network to prevent a recurrence.
We sincerely apologize for the inconvenience this caused our customers; we know you count on us for reliable service, and we’re committed to doing everything possible to avoid problems.
There was a brief outage on the web11 server today at about 2:42 PM Pacific time.
This was caused by a “denial of service” attack that increased the incoming network traffic to that server from the usual 5 Mbps or so to over 350 Mbps. Servers other than web11 were not affected.
This appears to be very similar to the attacks that occurred last Monday morning.
We are closely monitoring all systems so that we can see exactly how to block future attacks.
There were two brief outages on the web11 server on January 28, 2013, at about 8:09 AM and 8:46 AM Pacific time.
Read the rest of this entry »
Update: The maintenance described below was completed with less than 5 minutes downtime.
At 11:00 PM Pacific time January 18 2013, the “web04” server will be restarted.
Read the rest of this entry »
Between 11:00 PM and 11:59 PM Pacific time on Saturday December 22 2012, the MySQL database software on each of our servers will be upgraded to version 5.1.66 and restarted. This will cause an approximately 30 second interruption of service on each customer Web site at some point during this hour.
This upgrade is necessary for security reasons. We apologize for the inconvenience this causes.
Update December 22 11:17 PM: The maintenance was completed with less than 30 seconds downtime per server.
At 9:45 PM Pacific time November 15 2012, our “web10” server became unstable and we eventually decided to restart it to resolve the problem. This caused a period of about 20 minutes where the server was intermittently not working reliably, then a four minute outage while it restarted.
Read the rest of this entry »
At 11:00 PM Pacific time October 26 2012, our “web12” server experienced a “kernel panic” and needed to be restarted. This led to an 8 minute outage of Web sites and e-mail hosted on that server.
All services are now working normally, and other servers were not affected.
Read the rest of this entry »
Update 6:20 PM October 11: AOL has resolved the problem described below. All delayed mail has been delivered, and all services are operating normally.
Read the rest of this entry »
At 3:33 PM Pacific time September 12 2012, our “web04” server became unstable and needed to be restarted. This led to an 8 minute outage of Web sites and e-mail hosted on that server.
All services are now working normally, and other servers were not affected.
This isn’t normal or acceptable. We take server reliability seriously, and we’re investigating the underlying cause to avoid a recurrence of this problem.
This post was updated November 30, 2012 to reflect the additional availability of PHP 5.2.17.
We currently offer PHP versions 5.2.6, 5.2.17, and the 5.3 series. You can choose which version your account uses in the “PHP Settings” section of our “My Account” control panel.
PHP 5.2 has been obsolete for many years. Because of that, we’re beginning the process of removing PHP 5.2.6 from our servers and encouraging customers to switch to PHP 5.3. (PHP 5.2.17 is still available for now, but discouraged.)
Read the rest of this entry »