From 12:51 to 12:54 PM Pacific time today, one of our inbound mail servers (mx2.tigertech.net) incorrectly rejected some incoming mail that wouldn’t normally have been rejected, due to a configuration problem. This resulted in a small handful of messages being returned to the sender instead of properly delivered.
In an earlier post, we talked about how we use a monitoring system that forwards test e-mail to other large ISPs, then checks to make sure the message was promptly delivered.
We already check delivery to AOL, Comcast, GMail and Verizon, and we’ve now added AT&T/SBCGlobal and Yahoo mail. We’ll continue to expand it in the future.
An extensive monitoring and alert system is at the heart of our reliability, really; it’s what allows us to know that things are working properly. We can guarantee that if our customers send mail to those ISPs right now, it’s being delivered. That’s something few other companies even bother to check.
Due to a failure of the power distribution unit (essentially a fancy power strip) in one of the cabinets at our data center, the following services became unavailable at 05:52 AM Pacific time:
(Other Web servers are not affected.) A data center technician is replacing the power unit in that cabinet and all systems should be be back online within 15 minutes; we’ll update this post when that happens.
Update: The faulty hardware has been completely replaced. All servers are back online and functioning normally, and all queued e-mail has been delivered and is available for retrieval. The total outage for these servers was from 05:52 AM to 06:15 AM (Pacific time).
In addition, the FTP service on the “zapp” server was not fully working after it was restarted, so FTP publishing on that server was unavailable until shortly after 7:00 AM. This has been corrected (and the underlying problem that could cause incorrect startup was fixed).
We sincerely apologize to customers affected by this outage. This kind of issue has happened to us only once before in the last seven years (and that was with a different brand of power unit). Since the replacement power unit is brand new, we don’t expect the problem to recur.
Tonight at 11 PM Pacific time (2 AM Eastern time May 10) we’ll be performing brief scheduled maintenance on our mail servers. (We’ll be adding more RAM and adding more disk space to make sure that our mail servers continue to keep up with the growth in our service.) This requires restarting, which takes about five minutes, so you will see a brief period of about five minutes where you are unable to connect to our mail servers. No mail will be lost, of course; it will be queued and available after the maintenance.
We apologize for the inconvenience this causes. We schedule this kind of maintenance for late Saturday night/early Sunday morning (the least busy time of the week) to minimize the impact.
One of the features of our e-mail service is the ability to create e-mail forwarding aliases to forward messages from an address at your new Web site to existing e-mail account (AOL, Hotmail etc…). This is a useful feature if you need to receive e-mail from your new Web site and need to get it going quickly.
However, in the long term it’s better to use mailboxes on our servers (referred to as “POP mailboxes” on our setup screens, although they can also be accessed by IMAP or Webmail). In fact, one of the biggest advantages of having your own Web site and domain name is that you own it and all of its e-mail addresses. From our experience this is much better than being at the whim of a company that’s almost impossible to contact if you have trouble.
No matter how hard we try to make sure that other ISPs never block mail from our servers, it happens occasionally. All it takes is someone at another ISP clicking “this is spam” on a few legitimate messages sent by one of our customers, and some automated system at the other ISP thinks “hey, one of these tigertech.net servers is sending spam; let’s block it for a while without bothering to notify them, ‘for your convenience'”.
Now, we should emphasize that this is actually quite rare.
Anyone who works with computers has been asked to do this at some point in time and probably felt it was a ploy used to put off investigating the problem in detail. However, it really does work, and can fix program and software glitches.
For instance, we often have customers who cannot send and receive mail using the mail programs on their personal computers. After checking the mail servers and port numbers, everything seems to be set up correctly, but the mail program refuses to work. In these situations we often ask customers to restart their computer. Surprisingly they often tell us that everything works again.
We had an interesting support phone call today from a customer who was having trouble sending and receiving e-mail. She was using a wireless connection today. She had also been using a wireless connection yesterday, and it had been working fine then. Before calling us she had tried restarting her computer (as we recommend for most e-mail problems), but that didn’t seem to fix the problem.
