Network problem September 29 (resolved)
In an apparent continuation of last night’s incident, many sites we host were intermittently unavailable between 12:01 PM and 1:20 PM Pacific time today (September 29, 2011). This also caused slow mail delivery and reduced spam filtering effectiveness until around 2:00 PM (no mail was lost, of course).
All systems are operating normally as of 2:15 PM.
This problem was caused by an extremely large-scale network attack against the data center we’re moving from for unrelated reasons.
We sincerely apologize for this incident. We consider it unacceptable, as we know you do.
on Thursday, September 29, 2011 at 1:09 pm (Pacific) Ki wrote:
Hope it gets fixed soon…
Having a really hard time now…
on Thursday, September 29, 2011 at 1:13 pm (Pacific) Volodymyr Frytskyy wrote:
Same here…
on Thursday, September 29, 2011 at 1:17 pm (Pacific) Tom Hauber wrote:
Can you be more specific about the type of attack? Is it directed at a particular domain/ip address?
on Thursday, September 29, 2011 at 1:17 pm (Pacific) Jodi wrote:
Still down – any updates?
on Thursday, September 29, 2011 at 1:18 pm (Pacific) joan wrote:
Any ETA on when this will be repaired? It’s affecting my business. Thanks.
on Thursday, September 29, 2011 at 1:21 pm (Pacific) Jenn wrote:
Not able to access at all. Was trying to update site today.
ETA on fix? đŸ™‚
I know you guys are working hard on this. I’ve never had this happen in the last 10 years that I’ve been with you.
on Thursday, September 29, 2011 at 1:21 pm (Pacific) KT wrote:
I was able to re-establish contact for a minute and it just went down again.
on Thursday, September 29, 2011 at 1:41 pm (Pacific) Jodi wrote:
Any progress?
on Thursday, September 29, 2011 at 1:45 pm (Pacific) Volodymyr Frytskyy wrote:
Looks like up and running during 5 minutes already…
on Thursday, September 29, 2011 at 1:46 pm (Pacific) Noah wrote:
Looks like we’re up and running again. Thanks for your hard work.
on Thursday, September 29, 2011 at 2:06 pm (Pacific) KT wrote:
I am also stable now – thank you for keeping on top of things.
on Thursday, September 29, 2011 at 2:14 pm (Pacific) Ki wrote:
looks good now
Thanks for the hard work!
hope this never happens again though đŸ™‚
on Thursday, September 29, 2011 at 2:48 pm (Pacific) Robert Mathews wrote:
Tom Hauber wrote:
>Can you be more specific about the type of attack? Is it directed at a particular domain/ip address?
It wasn’t directed at our company or any of our customers. We’re still waiting for a final report, but it took out the entire data center (hundreds of times our size), overwhelming multiple ten-gigabit fiber connections.
on Thursday, September 29, 2011 at 3:46 pm (Pacific) lknobel wrote:
The New York Times site was down at that point. Same data center?
on Thursday, September 29, 2011 at 4:46 pm (Pacific) Robert Mathews wrote:
Lance,
Don’t know if they use the same data center (they probably use many), but apparently many data centers are under successful attack today because of a problem discovered in all Cisco Internet routers, leading to lots of engineers scrambling to block it all. Ugh.
on Thursday, September 29, 2011 at 7:57 pm (Pacific) Geff wrote:
Not a huge deal, but as of 8 pm pdt I seem to still be getting a bit more spam than usual (you mentioned that in the op). Everything else is good.
on Monday, October 3, 2011 at 10:47 am (Pacific) Ki wrote:
Is it just me?
It is happening again…