At 9:45 PM Pacific time February 6 2013, our “web03” server experienced a “kernel panic” and needed to be restarted. This led to an 11 minute outage of Web sites and e-mail hosted on that server.
All services are now working normally, and other servers were not affected. We apologize for the trouble this caused customers on the web03 server.
Beginning at 3:00 PM Pacific time February 5, a server on our network was the target of an extremely high volume DNS amplification denial of service attack. The inbound network data exceeded 11.6 Gbps, which is an extremely large amount — large enough to exceed the 10 Gpbs capacity of our upstream Ethernet switches and cause our entire network to slow down dramatically.
This affected all servers for about 19 minutes, until we and our network partners began discarding (“null routing”) all traffic targeted at that server. This fixed the problem for the rest of our network, but still left sites on the “web11” server unavailable.
To solve that, the IP addresses of all sites on the web11 server have been changed to new IP addresses that are working correctly and are not under attack. This was completed by 3:44 PM, and all sites on all servers are now working properly.
If the attackers target another IP address, we’re ready to immediately block that one, too. If that does happen, the way we’ve redistributed the IP addresses, in combination with previous analysis we’ve done on this attack, will allow us to immediately know which site is under attack. (It’s otherwise hard to determine which IP address is involved, because the type of attack we’re seeing targets only an IP address and not a specific Web site name.) That site will then be moved off our main network to prevent a recurrence.
We sincerely apologize for the inconvenience this caused our customers; we know you count on us for reliable service, and we’re committed to doing everything possible to avoid problems.
WordPress 3.5.1 was recently released, and as always, we’ve updated our WordPress one-click installer to automatically install the latest version for new WordPress sites.
If you’ve previously installed WordPress, you can upgrade it from within your WordPress Dashboard.
As a reminder, you should always update immediately when WordPress tells you there’s a new version available in the Dashboard. Don’t let yourself get behind, because it gets more difficult to update smoothly if you’re several versions out-of-date.
In addition, don’t avoid upgrading just because the upgrade screen says you should make a backup of your WordPress files and database first: we already make backups for you, automatically, every day.
The PHP developers have announced the release of version 5.3.21 that fixes several bugs.
We’ve upgraded PHP 5.3.20 to PHP 5.3.21 on our servers as a result.
Update: The maintenance described below was completed with less than 5 minutes downtime.
At 11:00 PM Pacific time January 18 2013, the “web04” server will be restarted.
One of the options we offer to speed up busy PHP sites is called eAccelerator. However, eAccelerator doesn’t usually help for low volume sites, so we’ve disabled it by default in those cases. The rest of this post explains more.
WordPress 3.5 was recently released, and as always, we’ve updated our WordPress one-click installer to automatically install the latest version for new WordPress sites.
If you’ve previously installed WordPress, you can upgrade it from within your WordPress Dashboard.
As a reminder, you should always update immediately when WordPress tells you there’s a new version available in the Dashboard. Don’t let yourself get behind, because it gets more difficult to update smoothly if you’re several versions out-of-date.
The PHP developers have announced the release of version 5.3.20 that fixes several bugs.
We’ve upgraded PHP 5.3.19 to PHP 5.3.20 on our servers as a result.
Between 11:00 PM and 11:59 PM Pacific time on Saturday December 22 2012, the MySQL database software on each of our servers will be upgraded to version 5.1.66 and restarted. This will cause an approximately 30 second interruption of service on each customer Web site at some point during this hour.
This upgrade is necessary for security reasons. We apologize for the inconvenience this causes.
Update December 22 11:17 PM: The maintenance was completed with less than 30 seconds downtime per server.
In an earlier post, we described how we’re phasing out PHP 5.2.6 in favor of the newer PHP 5.3 series.
Our original intention was to remove the long-obsolete 5.2 series entirely. However, a small number of customers have told us they’re unable to update their scripts to work with PHP 5.3, usually because the script is more than five years old and no newer version is available.
To help these customers, we’re making an extra version of PHP available: version 5.2.17. That’s still an outdated version, but for technical reasons we’re able to support it for a while longer than PHP 5.2.6, giving customers at least six extra months — perhaps even longer — before PHP 5.3 is the oldest version available on our servers.
We still recommend that all customers use the PHP 5.3 series. However, if you’re unable to do that, you can now use our “My Account” control panel to choose PHP 5.2.17 instead:
Our previously announced schedule to remove PHP 5.2.6 still applies, except that PHP 5.2.6 customers will be switched to version 5.2.17 (instead of all the way to the 5.3 series) if they don’t upgrade to 5.3 themselves.
