Brief network maintenance November 3 (postponed)

We’ve been notified that the maintenance previously scheduled for tonight (November 3, 2011) has been canceled and will be rescheduled for a future time.

Read the rest of this entry »

Data center move complete

As a followup to our previous posts about the move to a new primary data center, we want to confirm to our customers that the change was successfully completed.

Due to unrelated network outages at the old data center, we accelerated the original schedule mentioned in that post. Almost all customer sites were moved by October 7, and the remainder (a small handful of customer sites that needed manual intervention due to old software that was incompatible with the Debian Linux software update) were moved as of October 18. Everything is, and has been, working normally.

I want to again take the time to apologize to our customers for the service interruptions that occurred because of the original power problem and the later network problem. They weren’t acceptable. We know you count on us for your success, and we’re constantly working to improve reliability.

Network problems for some connections (resolved)

Between 10:32 AM and 10:47 AM Pacific time this morning (October 3), our monitoring systems detected high “packet loss” from one “network backbone”, which may have caused slow connections or timeouts for some customers. The monitoring systems show that this issue is resolved.

Short scheduled maintenance (completed)

The data center that experienced network problems earlier today has just informed us that they’ll be performing emergency maintenance on all their network routers tonight (Thursday, September 29, 2011) between 6:00 and 7:00 PM Pacific time.

During that hour, there may be up to five minutes total of network connectivity problems that makes some sites load slowly or fail to load.

Read the rest of this entry »

Network problem September 29 (resolved)

In an apparent continuation of last night’s incident, many sites we host were intermittently unavailable between 12:01 PM and 1:20 PM Pacific time today (September 29, 2011). This also caused slow mail delivery and reduced spam filtering effectiveness until around 2:00 PM (no mail was lost, of course).

All systems are operating normally as of 2:15 PM.

Read the rest of this entry »

High load on the “elzar” server (resolved)

The “elzar” Web hosting server experienced very high load between 9:07 and 9:14 AM Pacific time this morning (September 27, 2011), causing sites on that server to load slowly during those seven minutes. Other servers were not affected.

This was caused by a distributed denial of service (“DDOS”) attack against a site on that server. We manually blocked the attackers to resolve it, and we’re continuing to monitor it closely to make sure it doesn’t recur.

2011 server upgrades

Over the next four weeks, we’ll be migrating customer Web sites to upgraded servers. The servers have updated software (and upgraded hardware in some cases), and are also located in a data center with increased power reliability.

For most customers, these changes will be completely unnoticeable. However, a very small number of customers might notice software differences or experience up to five minutes total of “downtime” at some point. We recommend reading through this entire post for details.

Read the rest of this entry »

September 5, 2011 Labor Day holiday hours

Our business offices will be closed on Monday, September 5 to observe the US Labor Day legal holiday. As always, we’ll provide same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until Tuesday, and telephone support (via callbacks) will be available only for urgent issues.

The perils of quick tweeting

We’re making a determined effort to post Twitter status updates very quickly if our monitoring systems detect any kind of problem.

Earlier, we tweeted “We’re investigating a possible outage on the mail.tigertech.net server” because one of the multiple external monitoring systems we use alerted us that it was unable to connect to our mail server cluster.

Upon investigation, it turns out to have just been a false alarm. There was a problem with the monitoring system; there was nothing wrong with the mail servers at all. Unfortunately, there’s probably no way to prevent occasional false alarms like this; we’d rather get the information out quickly, and by definition that means posting preliminary information before we’ve had a chance to fully investigate what’s happening.

Behind-the-scenes POP and IMAP mail upgrades

Over the next month or so, we’ll be upgrading the POP and IMAP software we use for e-mail mailboxes. We don’t expect customers to notice any change (except possibly improved speed) or experience any service interruption at all; we’re mentioning it just for completeness.

Read the rest of this entry »