High load on web04 server May 9 2013 (resolved)
The “web04” server experienced extremely high load for several minutes beginning at 8:00 AM Pacific time on May 9. Sites on this server were slow or unavailable as a result.
The “web04” server experienced extremely high load for several minutes beginning at 8:00 AM Pacific time on May 9. Sites on this server were slow or unavailable as a result.
Several people have asked us about the recent WordPress WP Super Cache and W3 Total Cache plugin security vulnerability.
For the most part, sites hosted on our servers aren’t vulnerable to this because we block comments that contain the malicious code.
We’ve talked before about WordPress login rate limiting. Attempts to guess WordPress administrator passwords are an ongoing problem, getting worse all the time.
The average WordPress site we host has received tens of thousands of malicious login attempts this month, with hundreds of thousands of different IP addresses being used in the attacks. We try to block the IP addresses that are responsible, but the ever increasing number of addresses means we can’t block all of them — an individual address often attempts a login only once a day for a given site. We need to adopt other tactics.
1:31 PM Pacific time: Our technicians are investigating high load and slow page load times on the “web04” server.
2:09 PM Pacific time: This is being caused by a distributed denial of service attack on WordPress sites that is causing outages for many companies. We’re working to block it.
Between 12:50 and 1:23 PM Pacific time, service was intermittently unavailable or slow for sites and e-mail on the web12 server. In addition, customers on other servers may have seen brief delays or high load for about two minutes during this period.
Between 11:04 PM and 11:44 PM March 23, our network was either slow to respond due to high packet loss or completely unavailable to some customers.
The PHP developers have announced the release of version 5.3.22 that fixes several bugs. We’ve upgraded PHP 5.3.21 to version 5.3.22 on our servers as a result.
In addition, we now offer PHP version 5.4.12 as an optional choice in our control panel. For now, the PHP 5.4 series is recommended only for customers who need to test “cutting edge” features. Most customers should stick with the PHP 5.3 series, which is compatible with a wider variety of scripts.
There was a brief but severe performance problem on the web12 server today between 9:59 and 10:07 AM Pacific time. During this time, many Web server requests were very slow to load or even “timed out” completely. All services are now operating normally again. Other servers were not affected.
Between 11:00 PM and 11:59 PM Pacific time February 26, 2013, each of our servers will be restarted for a “kernel upgrade”. This will cause an approximately four minute interruption of service for each customer at some point during this hour.
Earlier today, Twitter user @adam_baldwin mentioned finding a security flaw on our site. He reported this to us (thanks!) and we fixed it, then another Twitter user @mattmcgee asked what it was. It helps everyone on the Internet be transparent about security, so here’s an attempt at an explanation.