Service outage Nov. 20, 2010 (resolved)

A major power failure at our primary data center in Fremont, California, caused a complete outage for nearly all services beginning at 8:32 PM Pacific time Saturday night. It lasted between six and 13 minutes, depending on the server. Only our blog and redundant DNS infrastructure was unaffected.

All services are now fully operational; please don’t hesitate to contact us if you have any questions. We sincerely apologize for the inconvenience this caused our customers.

We’re waiting for a detailed report from the data center staff. We know that the PG&E utility power to the data center was interrupted by a storm, but it’s not yet clear why the backup generators failed to provide power immediately. This is clearly not normal or acceptable, and we’ll update this post as we find out more.

Update November 22: According to the data center power engineers, a nearby lightning strike from a thunderstorm damaged both of the redundant industrial UPS systems that were online at the time, requiring a manual switch to a backup system. Lightning strikes should not cause this to happen, though, so they’re working with the UPS manufacturer to make sure it does not recur.