Outgoing e-mail monitoring
No matter how hard we try to make sure that other ISPs never block mail from our servers, it happens occasionally. All it takes is someone at another ISP clicking “this is spam” on a few legitimate messages sent by one of our customers, and some automated system at the other ISP thinks “hey, one of these tigertech.net servers is sending spam; let’s block it for a while without bothering to notify them, ‘for your convenience'”.
Now, we should emphasize that this is actually quite rare.
We have a strict anti-spam policy, and we use several methods (including outgoing filtering, rate limiting, and proper feedback loop/complaint monitoring) to try to make sure that real spam doesn’t leave our network. Spam is online pollution, and we spend a great deal of time and effort making sure our servers pollute as little as possible. As a result, mail from our servers tends to be blocked less often than mail sent from most ISPs or hosting companies (and we take it extremely seriously when it happens).
Still — it’s not unheard of for someone to block our servers, because other ISPs trust their users when they click “this is spam” , which you can’t really blame them for, although if you saw our AOL feedback loop mail, you wouldn’t believe the kind of innocent messages people sometimes report as spam: birthday greetings to grandma, order receipts, and other messages that aren’t even remotely close to spam. (People also report messages that are spam, of course; despite the noise, we do value the AOL feedback loop, because it lets us see when one of our customers is sending the kind of mail that could lead to blocking.)
Anyway: on Monday night, att.net started refusing to deliver mail from our servers because of an extremely vague “abuse issue” (I note their slogan this year is “Your world, delivered”, which apparently should be taken with a grain of salt when it comes to e-mail). A customer reported it to us and we immediately changed the IP addresses of our mail servers to work around the problem, and AT&T removed the blocking fairly promptly after we complained (albeit without any explanation), so it wasn’t a major incident. Again, though, we take anything like this very seriously; any undeliverable mail is extremely annoying, and I myself ended up working until 2 AM making sure it was fixed and wasn’t an ongoing problem.
This gave us the impetus to finish a long-planned project, though: adding other ISP’s mail systems to our own existing e-mail monitoring system. That system (based on the excellent Nagios software) checks our mail infrastructure every minute by sending messages from an external source to several test addresses, using both SMTP and the sendmail command-line interface, then reading them using a POP connection. This goes beyond naive checks that just make sure that the SMTP and POP servers are “up”: because it’s an end-to-end test, it can detect problems with SMTP, virus scanning, spam filtering, mail queue delivery delays, mail disk storage, POP servers, and more.
Well, we’re now expanding that to other ISPs: If we have (or can get) access to a mailbox at another ISP, our system sends messages from each of our outgoing mail servers to that mailbox, then checks to make sure the messages were delivered. That will alert us if the ISP starts blocking any mail, or if routing problems, etc. are preventing delivery to the ISP. It’s also going to sometimes give false positives, of course: a general problem at the other ISP is going to trigger an alarm and we’ll have to check the true cause manually, but hey, you’re paying us to look after the boring details.
We’re already monitoring outgoing mail sent to AOL, Comcast, GMail and Verizon, and we plan to add more ISPs in the future. If another ISP blocks mail from our servers, our goal is to know about it (and fix it) before you’re affected.
