Between 4:58 and 5:39 AM Pacific time today (March 23), our server which runs the Mailman mailing list software encountered an internal problem. During most of this time, all Mailman-related functionality was unavailable.
Since Mailman most works via e-mail, no data was lost. Some messages might have been slightly delayed, but not for any longer than might normally be noticed with mail delivery via the Internet.
We apologize for any inconvenience that this might have caused!
As you probably know, we back up all of our servers every night. Our goal is to keep at least seven days of backups available, with additional older backups available where possible.
We’ve added a new page to our control panel showing the backups available for each account. To see it, just login to the control panel, then click Backups.
We make backups so that we can recover from unexpected occurrences such as data erasure or server failures. Of course, we also make the backups available to customers, because they can be a real life-saver when you need one. However, we want to remind customers that they should also make their own backups, especially if you need a different backup frequency or retention policy. Our Web page describing our backup system and policies has more details.
The “calculon” Web server needed to be restarted at 10:14 AM Pacific time, resulting in a five-minute interruption of service for Web sites and e-mail on that server.
Read the rest of this entry »
One of the features of our new(ish) Webmail system is “thread view”. This groups similar messages together based on their “Subject” and other headers, which can occasionally be useful if you’re trying to see all the replies to a particular message and you want them grouped together.
However, thread view has a potential downside: it you have several active threads going with several messages each, new messages can sometimes appear on the second page of the incoming mail screens, instead of the first page.
That’s not a problem if you’re expecting it. However, since we introduced the new Webmail system, we’ve had several complaints from customers who accidentally clicked “Switch to Thread View” without realizing what it does, then thought some of their incoming mail was missing because they aren’t used to looking for new mail on other pages. Since thread view is “remembered” even after you logout and login again, this caused some people a great deal of heartache.
From our logs, we’ve found that very few people actually use thread view. Because it seems to cause frequent problems and few people use it, we’ve made it an optional feature instead of being always enabled.
If (like most people) you don’t use thread view, you don’t need to do anything. If do you want to use thread view, it’s still available: just click “Preferences”, then click “Display Preferences”, then change “Show ‘Thread View’ Link” to “Yes”.
The “zapp” Web server was unavailable between 8:20 and 8:40 Pacific time this morning. This resulted in an interruption of service for Web sites and e-mail on that server.
The problem was caused by a faulty hard disk in the RAID array (which theoretically shouldn’t cause a server to stop responding, but did). The hard disk has been removed from the array and will be replaced tonight at 10 PM. The server will be restarted at that time, resulting in about 4 minutes additional downtime.
We sincerely apologize for this problem. We will be investigating the root cause: it’s normal for hard drives to fail — we expect that occasionally — but it shouldn’t cause such negative effects (normally the RAID array would prevent the failure of any single drive from causing the entire machine to fail).
Between 5:58 and 6:26 AM Pacific time today (March 12), a network problem on one of our mail servers prevented some customers from being able to read and send e-mail.
The issue has been resolved and everything is working normally. Although incoming mail was delayed, no mail was lost. Web site service was not affected.
The cause of the problem was that a debugging tool used by one of our technicians (“tcpdump”), when used with certain options, can apparently cause network interface failures. This was not an issue we were previously aware of. We will avoid using the tool in that manner in the future, so the problem should not recur.
We regret the problem and sincerely apologize to our customers who were affected by this issue.
Let’s say that you want to register a domain name, so you go to the Network Solutions Web site to see if the name is available. If the domain name has not yet been taken, Network Solutions will register it for themselves behind your back, forcing you either to buy the name from them immediately (at their high price of $34.99), or wait four days until they release it again (at which point someone else may be able to get it before you).
If this happens to you, we imagine that you might feel something like this:
We’re introducing a new “We’ll Pay the Ransom” promotion, wherein we will pay Network Solutions the $34.99 necessary to register the domain name for you. Or, if you’ve already paid it, we’ll credit your account $34.99. Either way, just sign up to transfer the domain name to us and a year of our Web hosting.
Read the rest of this entry »
