Customer Web sites on the calculon, lrrr, and zapp Web servers were unavailable during this time. The ability to send and receive e-mail was also interrupted (no mail was lost, of course). Other servers were not affected.

We pay close attention to the power load in each cabinet to avoid this sort of problem. The previously measured peak load of that cabinet had been 12 amps. Since the circuit allows 15 amps, this issue surprised us (we’ve been using the same setup in the same data center for seven years and this has never happened before). It appears that a combination of several servers experiencing unusually high CPU loads led to power usage beyond what we previously considered possible.

We will take immediate steps to make sure the problem doesn’t happen again, and we sincerely apologize to customers who were affected by this incident.

Update 7:26 PM: We have removed a server from the cabinet in question, lowering the power use.

Update 10:38 PM: We have removed a second server from the cabinet, ensuring that power use is well below any level that could cause further trouble. The problem will not recur.

Web sites using scripts or databases on the elzar server may have seemed unresponsive during that time. Also, any customer hosted on elzar who was reading their e-mail during this time may have felt the system was slow or unresponsive (no e-mail was lost, of course).

Customers on other servers were not affected.

The problem was related to an extremely high number of Apache Web server processes running. We’ve changed some Apache configuration settings (lowering the maximum number of Apache processes and reducing some timeout settings so that fewer processes are required in the first place) on all our servers. We expect those change will prevent this symptom from happening again in the future.

We apologize for any inconvenience that this may have caused.

The problem occurred when the Apache web server process failed to gracefully restart when a new SSL certificate was added. We have discovered why this happened and will take steps to prevent it in the future.

We sincerely apologize to anyone affected by this.

Since Mailman most works via e-mail, no data was lost. Some messages might have been slightly delayed, but not for any longer than might normally be noticed with mail delivery via the Internet.

We apologize for any inconvenience that this might have caused!

The problem was caused while debugging high bandwidth usage on that server; running the Linux “iptraf” command caused the server to instantly lose network connectivity for some reason. We will not use that command again, obviously.

We apologize to anyone affected by this problem.

The problem was caused by a faulty hard disk in the RAID array (which theoretically shouldn’t cause a server to stop responding, but did). The hard disk has been removed from the array and will be replaced tonight at 10 PM. The server will be restarted at that time, resulting in about 4 minutes additional downtime.

We sincerely apologize for this problem. We will be investigating the root cause: it’s normal for hard drives to fail — we expect that occasionally — but it shouldn’t cause such negative effects (normally the RAID array would prevent the failure of any single drive from causing the entire machine to fail).

The issue has been resolved and everything is working normally. Although incoming mail was delayed, no mail was lost. Web site service was not affected.

The cause of the problem was that a debugging tool used by one of our technicians (”tcpdump”), when used with certain options, can apparently cause network interface failures. This was not an issue we were previously aware of. We will avoid using the tool in that manner in the future, so the problem should not recur.

We regret the problem and sincerely apologize to our customers who were affected by this issue.

No e-mail will be lost, of course; incoming mail will just be delayed for a few minutes.

This brief maintenance is necessary to upgrade the operating system “Linux kernel” to a newer version for security reasons. This was also done two weeks ago; unfortunately our operating system vendor has released an even newer kernel since then — it doesn’t usually happen this often.

We apologize for the inconvenience this causes.

(This  maintenance was also successfully completed with less than four minutes of downtime per server.)

No e-mail will be lost, of course; incoming mail will just be delayed for a few minutes.

This brief maintenance is necessary to upgrade the operating system “Linux kernel” to a newer version for security reasons. We apologize for the inconvenience this causes.

 (This  maintenance was successfully completed with less than four minutes of downtime per server.)

The root cause of the problem was the failure of an Ethernet switch at our data center. The switch was the one that our network cables actually plug into to connect to the Internet. Unfortunately, it’s one of the few pieces of the network infrastructure that’s not automatically redundant: although the “other side” of the switch is connected to multiple fully redundant upstream paths to the Internet, the side of it that goes to our server cabinets effectively has a single connection for each a group of servers.

When the switch failed, the data center staff replaced it with a new spare one. Because the faulty hardware was completely replaced, the problem is properly solved, and this won’t be something that’s an ongoing problem.

Just so it’s clear, we own and operate all our own servers, but we house these servers in a professional data center that provides provide uninterruptible electrical power, cooling, and extremely fast network connectivity. The data center has engineers on site 24 hours a day, 365 days a year to handle this kind of issue, and they started working on it the minute the outage started. That said, we’re disappointed that the problem wasn’t resolved sooner.

We’ve used the same data center for many years, and the small number of problems we’ve experienced have usually been taken care of very quickly. We expect that generally good performance to continue, but we will take appropriate remedial action if it does not.

Again, we apologize to our customers for the inconvenience caused by this outage.