Between 6:00 AM and 6:29 AM Pacific time August 7, 2011, all services were unavailable due to a power failure at our primary data center.
The problem was resolved for most servers by 6:29 AM, and for all servers except the “amy” server by 6:53 AM. The “amy” server needed extra manual intervention, and was working by 7:55 AM. All services are now operating normally.
Any e-mail that arrived during the outage was queued at our secondary data center and delivered as soon as the outage ended.
We sincerely apologize for this problem. We know you count on us for reliability, and we don’t consider this acceptable, especially since the data center has had previous power problems this year. However, this incident had a different root cause. It wasn’t a utility power failure that the redundant UPS systems didn’t handle, but was instead caused by a circuit breaker incorrectly “tripping” to prevent the power output of the UPS systems from reaching the server cabinets.
Update 4:15 PM: We have received an incident report from the data center indicating that they are working to replace the affected part of the UPS system to prevent further problems.
At approximately 11:00 PM Pacific time July 26 2011, the “pazuzu” Web server will be restarted.
As a result, for customers on the “pazuzu” server (only), Web site service and the ability to read incoming e-mail will be unavailable for approximately five minutes. Customers on other servers will not be affected.
Read the rest of this entry »
We’ve updated phpMyAdmin to the latest version, 3.4.3.1.
Read the rest of this entry »
We’ve installed a PHP 5 security update. Customers should not notice any changes; the update just fixes several security issues in PHP 5.
WordPress 3.2 was released a couple of days ago, and it looks like a great update. (We even contributed a little bit of performance-improving code to it ourselves.)
Our WordPress one-click installer automatically installs the latest version for new installs.
If you’ve previously installed WordPress, you can upgrade it from within your WordPress Dashboard. You should always do that when WordPress tells you there’s a new version available.
Our business offices will be closed on Monday, July 4 to observe the US legal holiday. As always, we’ll provide same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until the next day, and telephone support (via callbacks) will be available only for urgent problems.
Many, many years ago, some e-mail programs didn’t use a password when sending outgoing mail. That meant they didn’t work with many mail servers, including ours. To help customers with that problem, we used to allow a horrible alternate method called “POP before SMTP”, although it was never recommended or officially supported (it was unreliable and made it harder for us to prevent spam).
Well, here we are in a new millennium (“welcome!”). No popular mail program has needed “POP before SMTP” for more than a decade, and only a small handful of our customers are still using it. But spammers are continually trying to take advantage of the security problems it creates for all e-mail addresses, making it just as much of a nuisance on our end as it ever was.
Because of that, we no longer allow e-mail addresses to send mail using “POP before SMTP” unless they were previously doing so. In other words, if an address wasn’t using “POP before SMTP” before now, it won’t be able to start using it in the future.
Read the rest of this entry »
We’ve updated our servers with a Perl security bug fix. This won’t affect most customers, but read on if you know you use Perl scripts on your site.
Read the rest of this entry »
The “fry” server was the victim of a high-bandwidth Distributed Denial of Service (DDoS) attack beginning at 3:11 AM Pacific time this morning. On that server, Web pages were intermittently slow to load or generated timeout errors. (Other servers were not affected.)
We’ve blocked the large number of IP addresses from the “botnet” attacking the server, and the issue was completely resolved by 4:19 AM Pacific time. Please accept our apologies if you noticed any problems with your site loading slowly during this period.
We’ll be performing brief maintenance on the Web server that runs the Mailman list interface and archives tonight (June 9, 2011) between 10:00 PM and 11:00 PM Pacific time.
Read the rest of this entry »
