A couple of times in the last week, we’ve seen one of our MySQL database servers have an unusually high number of connections. That’s a serious issue: If there are too many connections to a MySQL server, customer scripts won’t be able to connect to a database, so we’ve spent some time looking at the cause and fixing it.
One of the features of our service is the industrial-strength Mailman mailing list manager. Mailman is a very good program in some ways (it’s built like a tank and reliably handles very large volumes of list mail, and it removes much of the drudgery of managing large lists), but it has a couple of undesirable “features”.
The most obvious is that the interface is terribly ugly (the Mailman developers are working on a big improvement to this, thankfully; just so it’s clear, we didn’t create the program, and we’re as horrified by the circa-1996 appearance as everyone else). Another problem with the program, though, is the option for “monthly password reminders”. This is a design flaw that’s being removed from Mailman, and although most of the lists on our servers don’t use password reminders, customers who do should probably turn them off now in preparation for that change.
The “farnsworth” Web server locked up and needed restarting again last night at about 9:04 PM Pacific time, causing another short outage for some customers. (A similar problem happened Monday night.) To make sure this doesn’t happen again, we’ll be replacing the entire server (switching it with a spare server) at about 11 PM (Pacific) tonight, which will result in about 5 minutes of downtime.
We’re also taking this opportunity to upgrade the hardware on one of our mail servers to allow for future growth; customers (even those with accounts on other Web servers besides the farnsworth server) may see a short (approximately 5 minute) interruption in their ability to retrieve e-mail between 11 PM and midnight.
We apologize for any inconvenience this causes — as always, we’re committed to the highest possible levels of reliability.
Some of the posts on our blog mention specific servers. You’ll occasionally see things like “The web14 server will be rebooted at 11 PM”, “mail sent from the web01 server was delayed”, or “more memory has been added to the web10 server”. Your question, quite naturally, is “How do I know if they’re talking about the server that has my account?”
One of our Web servers (the “farnsworth” server) stopped responding at 7:07 PM Pacific time today, and needed to be forcibly restarted. This resulted in a Web server and FTP server outage of about 15 minutes for some customers, although most sites were unaffected.
After being restarted, the server is responding properly, but still showing a problem with one of the disks in its RAID array. Because of that, we plan to replace the disk to prevent future problems, meaning we’ll restart that server again later tonight (after 11 PM Pacific time).
We apologize to all customers affected; we strive hard to avoid this kind of problem.
We’ve updated PHP 5 on our servers to cover sixteen recently identified security issues. This only affects customers who have chosen to use PHP 5 — but since this upgrade only fixes security bugs, even those customers shouldn’t notice any changes.
We’re currently seeing about 15% “packet loss” from our data center to a handful of locations on the Internet (notably connections that go through the above.net backbone). Most people aren’t affected by this, but for those that are, this can cause connections to be slower than normal. We have a ticket open with the data center for this issue, and we’ll update this page when it’s resolved.
Update May 20: The packet loss problem was effectively resolved on Friday, although we’ve been monitoring the above.net backbone connection closely to ensure that there is no ongoing problem. Although we’ve seen a couple of short latency issues that we’re still following up with the data center about, customers are not experiencing any problems.
No matter how hard we try to make sure that other ISPs never block mail from our servers, it happens occasionally. All it takes is someone at another ISP clicking “this is spam” on a few legitimate messages sent by one of our customers, and some automated system at the other ISP thinks “hey, one of these tigertech.net servers is sending spam; let’s block it for a while without bothering to notify them, ‘for your convenience'”.
Now, we should emphasize that this is actually quite rare.
We’ll be talking a lot about “security updates” on the blog, so a word about what these are and how we handle them is probably in order.
There are literally thousands of software programs on our servers, most of which are written by other people and used by many companies. From time to time, “security vulnerabilities” with these kinds of programs are discovered. A security vulnerability is something that could allow a “hacker” (or “cracker”, for purists, although that battle has been lost) to take advantage of a programming bug to do something unauthorized with the program, such as send spam or delete files.
The following stable PEAR packages were updated on our hosting servers today:
The full list of PEAR modules we have available (and more details about PEAR) is on this page.
