There was a brief outage on the web12 server today starting at about 6:22 PM Pacific time. This was caused by a “SYN flood” attack, which effectively blocked all other connections with the server.
We took steps to work around the attack, which we completed by 7:08 PM Pacific time (46 minutes after the start of the attack). Furthermore, the attack itself seems to have stopped; the steps we took should help in case in starts again.
We sincerely apologize for the interruption in service for those affected customers; we know that reliable service is a primary concern for all of our customers.
There were two brief outages on the web11 server on January 28, 2013, at about 8:09 AM and 8:46 AM Pacific time.
Read the rest of this entry »
At 3:33 PM Pacific time September 12 2012, our “web04” server became unstable and needed to be restarted. This led to an 8 minute outage of Web sites and e-mail hosted on that server.
All services are now working normally, and other servers were not affected.
This isn’t normal or acceptable. We take server reliability seriously, and we’re investigating the underlying cause to avoid a recurrence of this problem.
At 12:50 AM Pacific time, the web08 server experienced extremely high disk load and needed to be restarted as a result, resulting in approximately 5 minutes downtime for sites on that server. Other servers were not affected, and the server is now working normally.
We had a couple of brief network interruptions today (at about 8:50am and 12:49pm Pacific time). We are investigating them, and will update this post with more details later.
Updated 2012-07-19 2:23PM Pacific: One of our upstream network providers has traced this to what appears to be some very brief, very high-packet-per-second attacks. The attacks have not recurred, and we are continuing to monitor all systems.
Our “web07” server needed restarting at 11:36 AM Pacific time on February 1, 2012, because it had been intermittently unable to run some PHP scripts for 22 minutes.
The restart resolved the immediate problem, and a followup post explains what happened and the changes we made to prevent it from happening again.
A problem at our old data center (the one we’re moving sites from this month) caused some sites to be intermittently unavailable between 10:22 and 10:47 PM Pacific time.
Read the rest of this entry »
Between 6:00 AM and 6:29 AM Pacific time August 7, 2011, all services were unavailable due to a power failure at our primary data center.
The problem was resolved for most servers by 6:29 AM, and for all servers except the “amy” server by 6:53 AM. The “amy” server needed extra manual intervention, and was working by 7:55 AM. All services are now operating normally.
Any e-mail that arrived during the outage was queued at our secondary data center and delivered as soon as the outage ended.
We sincerely apologize for this problem. We know you count on us for reliability, and we don’t consider this acceptable, especially since the data center has had previous power problems this year. However, this incident had a different root cause. It wasn’t a utility power failure that the redundant UPS systems didn’t handle, but was instead caused by a circuit breaker incorrectly “tripping” to prevent the power output of the UPS systems from reaching the server cabinets.
Update 4:15 PM: We have received an incident report from the data center indicating that they are working to replace the affected part of the UPS system to prevent further problems.
The “fry” server was the victim of a high-bandwidth Distributed Denial of Service (DDoS) attack beginning at 3:11 AM Pacific time this morning. On that server, Web pages were intermittently slow to load or generated timeout errors. (Other servers were not affected.)
We’ve blocked the large number of IP addresses from the “botnet” attacking the server, and the issue was completely resolved by 4:19 AM Pacific time. Please accept our apologies if you noticed any problems with your site loading slowly during this period.
Some Comcast users in California had trouble connecting to some of our servers beginning at around 7 AM Pacific May 21, 2011. Non-Comcast users were not affected at all, and even people who were affected were able to reach some of our servers with no trouble.
This was caused by a technical problem at Comcast, and not related to us specifically. It appears Comcast was incorrectly filtering some combinations of IP addresses and ports in one of their California network routers, preventing their customers from reaching some sites.
The issue was apparently resolved by Comcast at 10:07 AM Pacific time, and we are not aware of any ongoing problems. As always, don’t hesitate to contact us if you have any trouble.
