Flexo server temporarily unavailable (resolved)

Customers on the “flexo” server experienced a four-minute interruption in Web site service between 9:48 and 9:52 AM Pacific time this morning (August 12).

E-mail was not affected, and customers on other servers were not affected.

The problem happened when the Apache Web server did not respond to a “graceful reload” command when we installed a “mod_security” update to block certain attacks against the WordPress blog software.

We are looking into the root cause of this incident and will take steps to prevent it from recurring. We don’t consider any kind of service interruption acceptable, and we sincerely apologize for the problem.

PHP 4 being phased out

For the last several years, we’ve offered PHP versions 4 and 5 on our servers. This made sense when PHP 5 was new: Even though PHP 5 is faster and more secure than PHP 4, a small handful of scripts were originally incompatible with version 5, and we wanted to give customers a choice.

However, PHP 5 is now more than five years old, and the PHP developers declared version 4 obsolete in 2007. All our new customers have been using PHP 5 by default for more than a year, and we’ve received no complaints about incompatibilities.

No PHP script should require the obsolete PHP version 4 any more. Because of that, we’re beginning the process of removing it from our servers.

Read the rest of this entry »

Denial of service attack update

As we mentioned in an earlier post, someone attacked our network earlier this morning. Although we blocked the attack, we’ve also been working to identify who attacked our network and why. We now know the answer, and we are almost positive that the problem won’t recur.

Read the rest of this entry »

Denial of service attack (resolved)

Beginning at 2:16 AM Pacific time this morning, we began experiencing a “distributed denial of service” attack aimed at our “flexo” Web server.

The attack used more than 2 Gbps of network bandwidth from several thousand different IP addresses. This is an extremely high amount of traffic, saturating even our network connections.

The problem caused most of our servers to become unreachable (or very slow) from the Internet.

We restored service to all servers except the flexo Web server at 2:59 AM (by getting our network providers to block all packets for certain IP addresses). We restored service to the flexo server at 3:29 AM (by getting them to identify and block specific characteristics of the attack).

All services are now operating normally, and all delayed incoming mail has been delivered.

We take reliability seriously. Unfortunately, this is by far the largest attack we’ve seen on our network in ten years. We sincerely regret and apologize for the impact this had on our customers.

Brief scheduled maintenance Saturday, May 2 (completed)

At approximately 11:00 PM Pacific time this Saturday, May 2, the “bender”, “calculon”, “lrrr” and “hypnotoad” servers will be restarted. As a result, Web site and e-mail service for customers on those servers will be unavailable for approximately five minutes.

Read the rest of this entry »

farnsworth hosting server restarted

The “farnsworth” server was restarted at 11:45 PM Pacific time tonight, causing a brief 2 minute interruption in Web and e-mail service for customers on that server. Incoming mail was queued and delivered after the interruption.

Read the rest of this entry »

Problem affecting two servers (resolved)

We posted earlier about a problem affecting the elzar Web server. While we were investigating the cause of that, the same thing happened on another Web server, “calculon”, causing a separate outage for customers on that server from 2:34 PM to 2:43 PM Pacific time this afternoon.

During this period, Web sites on that server were unavailable and incoming e-mail was delayed. (The Web server was slow for about six minutes after it was restarted, too.)

On both servers, high disk and memory usage caused the load to skyrocket to the point where they effectively stopped responding.

The good news is that we have narrowed down the cause, so it shouldn’t happen again. A bug in one of our maintenance programs that runs on each server was almost certainly responsible. The bug has been fixed.

We sincerely apologize for this issue, and regret the inconvenience it caused for customers hosted on these servers. Other servers were not affected.

Elzar server temporarily unavailable (resolved)

The “elzar” Web server experienced high load between 5.40 and 6.00 AM Pacific time this morning, April 15. This resulted in slow Web sites and some interruption of service. (Some e-mail activity was delayed, but no e-mail was lost.)

We sincerely apologize for this problem. We consider this type of failure to be unacceptable, and are looking into the cause of the problem so that we can take the appropriate steps to prevent it from happening again.

Zapp server added to brief scheduled maintenance (completed)

As we’ve already posted, some of our Web servers will be restarted tonight at 11 PM Pacific time.

We’re adding the “zapp” Web server to that list so we can replace a RAID array disk that caused a problem on that server earlier today.

Update: The maintenance was completed with less than five minutes of “downtime”.

Zapp server temporarily unavailable (resolved)

The “zapp” Web server was unavailable between 3:43 and 3.53 AM Pacific time this morning, April 4. This resulted in an interruption of service for Web sites on that server. (Some e-mail activity was delayed, but no e-mail was lost.)

We sincerely apologize for this problem. We consider this type of failure to be unacceptable, and are looking into the cause of the problem so that we can take the appropriate steps to prevent it from happening again.