There’s been a lot of talk in the last few days about a nasty PHP security bug that allows “hackers” to compromise some Web sites that use the PHP scripting language.
Our customers are not vulnerable to this problem because of the way PHP is set up on our servers. You don’t need to worry about it.
Between 1:53 AM Pacific time and 2:09 AM on May 1, the disk load on the “web11” server became very slow, requiring that server to be restarted. We did so, and normal service was resumed at 2:10 AM. Other servers were not affected.
WordPress 3.3.2 was released today, and it contains an important security update to keep your site safe.
Our WordPress one-click installer automatically installs the latest version for new sites. If you’ve previously installed WordPress, you should upgrade it right away from within your WordPress Dashboard. (You should always do that when WordPress tells you there’s a new version available.)
Lots of people (and lots of our customers) use WordPress to run their Web sites. This unfortunately means that lots of “hackers” also try to guess the passwords of those sites.
That’s a problem, so we’ve had WordPress login “rate limiting” in place for a long time. When a single IP address tries loading the WordPress “wp-login.php” script many more times than a human would, we temporarily block that IP address from accessing the “wp-login.php” page until the requests stop for a while.
This works pretty well: we’ve blocked literally millions of password attempts this way. However, last week one of our customers had his site hijacked by someone who did indeed simply guess his WordPress password.
We’ve been notified by an upstream network provider that they will be performing router firmware upgrades on Saturday, March 24 2012 between 4:00 and 4:30 PM Pacific time.
Most customers will not notice any service interruption because we use redundant network providers, but in the worst case it can take up to about 90 seconds for certain parts of the Internet to see the changed “routes”. That means a brief interruption is theoretically possible for some connections. We’re announcing this just so you know that if you do see any problem, it will be resolved quickly.
Update 4:33 PM Pacific time: The maintenance has been completed.
Between 10:00 PM and 11:00 PM Pacific time on Friday March 9, 2012, we’ll be updating the MySQL database software on all our hosting servers. This will cause a Web site service interruption of about 30 seconds for some customers at some time during this period. E-mail will not be affected.
This maintenance is necessary to install a mandatory MySQL security update that will upgrade the MySQL version to 5.1.61. We apologize for any inconvenience this causes.
Update 10:13 PM: The maintenance was completed with less than 30 seconds downtime on each server. Customers should not notice any changes, but as always, don’t hesitate to contact us with any questions or problems.
Web sites on the web03 server suffered an interruption in service between 7:32 AM and 7:45 AM this morning (Tuesday, February 21).
This was caused by a “hung” process that prevented a routine Apache Web server reload from completing. Other servers were not affected. Our staff restarted the server to stop the “hung” process, and the problem was resolved.
We sincerely apologize to customers affected by this incident. We’re considering possible underlying causes to prevent a recurrence.
On Saturday, February 18, 2012 between 10:00 and 11:00 PM Pacific time, we’ll be upgrading the Apache Web server software on each of our Web servers.
Most customers will not notice anything, but the upgrade will cause approximately 30 seconds of slow Web page loading at some point during that hour as we delay incoming connections at the network level.
This maintenance is necessary to apply security and reliability fixes released by the Apache developers. (We’ve been using the upgraded version on our Webmail servers for several days, so it’s well tested.)
Update: The maintenance was completed at 10:03 PM Pacific time.
The disk load on the “web05” server was extremely high between 2:30 and 2:42 AM Pacific time Saturday February 4, causing some downtime during that period for sites using that server. Other servers were not affected.
A couple of days ago, one of our Web servers became unstable for an unknown reason and needed to be restarted. This is rare: on average, this happens less than once every five years of uptime per server, so we took it very seriously and launched an investigation.
What we found was that the owner of one of the sites on that server made a mistake that allowed attackers to run their own scripts. That’s all too common, unfortunately, but usually only the single site is affected by this kind of thing. What was surprising in this case was that the script used a previously unknown method of causing problems for other sites running on the server.
As a result of this investigation, we’ve made several changes to our systems to ensure the problem won’t recur. The rest of this post has a detailed technical description of the problem in case it’s useful for others.
