Our business offices will be closed on Monday, May 25 to observe the US legal holiday. As always, we’ll provide same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until the next day, and telephone support (via callbacks) will be available only for urgent problems.
An earlier blog post described how several of our customers got their personal computers infected by a new virus that has been spreading across the Internet. Initial versions of the virus spread themselves by reading a Web site’s FTP username and password stored on the PC, then downloading Web pages, inserting an “iframe” tag, and re-uploading the Web pages back to the server. As a proactive measure, we started scanning all uploaded files and stripping out any malicious “iframe” tags.
We are now seeing newer versions (commonly called “Gumblar”) which spread by inserting “script” tags with encoded JavaScript code. Because there are several variations of this approach, and because some legitimate commercial scripts use the same technique to hide their source code, we cannot perfectly identify and strip out these infections. Therefore, we will not automatically strip out the “script” tags from any upload file that looks suspicious.
As we mentioned in an earlier post, someone attacked our network earlier this morning. Although we blocked the attack, we’ve also been working to identify who attacked our network and why. We now know the answer, and we are almost positive that the problem won’t recur.
Beginning at 2:16 AM Pacific time this morning, we began experiencing a “distributed denial of service” attack aimed at our “flexo” Web server.
The attack used more than 2 Gbps of network bandwidth from several thousand different IP addresses. This is an extremely high amount of traffic, saturating even our network connections.
The problem caused most of our servers to become unreachable (or very slow) from the Internet.
We restored service to all servers except the flexo Web server at 2:59 AM (by getting our network providers to block all packets for certain IP addresses). We restored service to the flexo server at 3:29 AM (by getting them to identify and block specific characteristics of the attack).
All services are now operating normally, and all delayed incoming mail has been delivered.
We take reliability seriously. Unfortunately, this is by far the largest attack we’ve seen on our network in ten years. We sincerely regret and apologize for the impact this had on our customers.
At approximately 11:00 PM Pacific time this Saturday, May 2, the “bender”, “calculon”, “lrrr” and “hypnotoad” servers will be restarted. As a result, Web site and e-mail service for customers on those servers will be unavailable for approximately five minutes.
