At approximately 8:00 PM Pacific time on November 12, 2011, the “web06” Web server will be restarted.
As a result, for customers on the “web06” server (only), Web site service and the ability to read incoming e-mail will be unavailable for approximately five minutes. Customers on other servers will not be affected.
We’ve been notified by an upstream network provider that they will be performing router firmware upgrades on Saturday, November 5, 2011 between 5:00 and 6:00 PM Pacific time. In the worst case, some customers may see a network interruption of up to five minutes during this period.
Please accept our apologies for any interruption this causes; we’re told that the upgrades are necessary to prevent possible network problems.
(This is the rescheduled time for an original announcement that was postponed.)
Update: The maintenance was completed with approximately three minutes downtime for affected connections.
We’ve been notified that the maintenance previously scheduled for tonight (November 3, 2011) has been canceled and will be rescheduled for a future time.
As a followup to our previous posts about the move to a new primary data center, we want to confirm to our customers that the change was successfully completed.
Due to unrelated network outages at the old data center, we accelerated the original schedule mentioned in that post. Almost all customer sites were moved by October 7, and the remainder (a small handful of customer sites that needed manual intervention due to old software that was incompatible with the Debian Linux software update) were moved as of October 18. Everything is, and has been, working normally.
I want to again take the time to apologize to our customers for the service interruptions that occurred because of the original power problem and the later network problem. They weren’t acceptable. We know you count on us for your success, and we’re constantly working to improve reliability.
Our previously-announced move to the new data center is continuing. The old data center was hit with a lot of network attacks throughout the day today. We are very glad to be moving to the new center!
E-mail should be fully working for everyone. Many customers have been moved to the new data center, and we’ll be continuing to move the remaining sites as quickly as possible (while of course being very careful about data integrity and site uptime). Sites which have not yet been moved are still working at the old data center. The old data center has been up for several hours now with only very occasional interruptions (due to attacks). They might still experience occasional bouts of downtime if the attacks continue.
If you want to know if your Web site has been moved to a server at the new data center, please see the update at the bottom of this page.
We are continuing to experience intermittent problems at Hurricane Electric, the data center that we’ve been using until now. Fortunately, as previously mentioned, we are in the process of moving to a new data center.
Some customers have already been moved to the new data center, and they have not seen any of these problems. We are moving the remaining sites as quickly as possible.
We are extremely upset with the attacks that have been hitting Hurricane Electric, and consider these outages to be unacceptable. It definitely appears that Hurricane Electric is being specifically targeted. You can watch their Twitter feed to see information about the latest attacks and the steps they’re taking to fix the attacks, as well as what they’re doing to try to protect themselves against further ones. While it’s undoubtedly of little comfort, it’s worth noting that these attacks are not directed at our customers or at us, but at Hurricane Electric in general. You can see that other large companies hosted at Hurricane Electric are also being affected, such as Linode.
We apologize for the inconvenience these outages have caused, and hope to have all Web sites moved to the new data center very soon.
Between 10:32 AM and 10:47 AM Pacific time this morning (October 3), our monitoring systems detected high “packet loss” from one “network backbone”, which may have caused slow connections or timeouts for some customers. The monitoring systems show that this issue is resolved.
The data center that experienced network problems earlier today has just informed us that they’ll be performing emergency maintenance on all their network routers tonight (Thursday, September 29, 2011) between 6:00 and 7:00 PM Pacific time.
During that hour, there may be up to five minutes total of network connectivity problems that makes some sites load slowly or fail to load.
In an apparent continuation of last night’s incident, many sites we host were intermittently unavailable between 12:01 PM and 1:20 PM Pacific time today (September 29, 2011). This also caused slow mail delivery and reduced spam filtering effectiveness until around 2:00 PM (no mail was lost, of course).
All systems are operating normally as of 2:15 PM.
A problem at our old data center (the one we’re moving sites from this month) caused some sites to be intermittently unavailable between 10:22 and 10:47 PM Pacific time.
