An earlier blog post described how several of our customers got their personal computers infected by a new virus that has been spreading across the Internet. Initial versions of the virus spread themselves by reading a Web site’s FTP username and password stored on the PC, then downloading Web pages, inserting an “iframe” tag, and re-uploading the Web pages back to the server. As a proactive measure, we started scanning all uploaded files and stripping out any malicious “iframe” tags.
We are now seeing newer versions (commonly called “Gumblar”) which spread by inserting “script” tags with encoded JavaScript code. Because there are several variations of this approach, and because some legitimate commercial scripts use the same technique to hide their source code, we cannot perfectly identify and strip out these infections. Therefore, we will not automatically strip out the “script” tags from any upload file that looks suspicious.
Read the rest of this entry »
As we mentioned in an earlier post, someone attacked our network earlier this morning. Although we blocked the attack, we’ve also been working to identify who attacked our network and why. We now know the answer, and we are almost positive that the problem won’t recur.
Read the rest of this entry »
Beginning at 2:16 AM Pacific time this morning, we began experiencing a “distributed denial of service” attack aimed at our “flexo” Web server.
The attack used more than 2 Gbps of network bandwidth from several thousand different IP addresses. This is an extremely high amount of traffic, saturating even our network connections.
The problem caused most of our servers to become unreachable (or very slow) from the Internet.
We restored service to all servers except the flexo Web server at 2:59 AM (by getting our network providers to block all packets for certain IP addresses). We restored service to the flexo server at 3:29 AM (by getting them to identify and block specific characteristics of the attack).
All services are now operating normally, and all delayed incoming mail has been delivered.
We take reliability seriously. Unfortunately, this is by far the largest attack we’ve seen on our network in ten years. We sincerely regret and apologize for the impact this had on our customers.
At approximately 11:00 PM Pacific time this Saturday, May 2, the “bender”, “calculon”, “lrrr” and “hypnotoad” servers will be restarted. As a result, Web site and e-mail service for customers on those servers will be unavailable for approximately five minutes.
Read the rest of this entry »
The “farnsworth” server was restarted at 11:45 PM Pacific time tonight, causing a brief 2 minute interruption in Web and e-mail service for customers on that server. Incoming mail was queued and delivered after the interruption.
Read the rest of this entry »
We’ve added a new feature to hosting accounts: Live, realtime access to the Apache Web server “error log”, both in the “My Account” control panel and as raw files you can access through FTP/ssh/etc.
To view the most recent 200 lines of the error log, login to the control panel (having trouble?), click “Statistics and Logs”, and look at the new “Web site error logs” section.
To download the full raw error log files, see this page.
We hope you find this useful!
Recently, several customers have told us that pages on their Web sites have been modified without their knowledge. Upon investigation, the customers found their computers had been infected with a virus that steals saved FTP passwords, such as the “Gumblar” or Trojan.PWS.Tupai.A virus.
We’ve taken a step to protect you against this problem (described below), but it’s wise to protect yourself, too.
Read the rest of this entry »
We posted earlier about a problem affecting the elzar Web server. While we were investigating the cause of that, the same thing happened on another Web server, “calculon”, causing a separate outage for customers on that server from 2:34 PM to 2:43 PM Pacific time this afternoon.
During this period, Web sites on that server were unavailable and incoming e-mail was delayed. (The Web server was slow for about six minutes after it was restarted, too.)
On both servers, high disk and memory usage caused the load to skyrocket to the point where they effectively stopped responding.
The good news is that we have narrowed down the cause, so it shouldn’t happen again. A bug in one of our maintenance programs that runs on each server was almost certainly responsible. The bug has been fixed.
We sincerely apologize for this issue, and regret the inconvenience it caused for customers hosted on these servers. Other servers were not affected.
The “elzar” Web server experienced high load between 5.40 and 6.00 AM Pacific time this morning, April 15. This resulted in slow Web sites and some interruption of service. (Some e-mail activity was delayed, but no e-mail was lost.)
We sincerely apologize for this problem. We consider this type of failure to be unacceptable, and are looking into the cause of the problem so that we can take the appropriate steps to prevent it from happening again.
As we’ve already posted, some of our Web servers will be restarted tonight at 11 PM Pacific time.
We’re adding the “zapp” Web server to that list so we can replace a RAID array disk that caused a problem on that server earlier today.
Update: The maintenance was completed with less than five minutes of “downtime”.
