Unexpected internal systems problem 2010-01-15

Starting at about 1:30 PM (Pacific time) today, some of our internal database systems had an unexpected failure. This lead to problems with our control panel (“My Account”), support site, and blog. Some customers may have also had problems with some aspects of Webmail (in particular, with the address book).

No customer data or Web sites were affected, and no e-mail was lost.

All systems are running again, so no one should see any problems — please let us know if you do! Some things are running on backup systems, so we’re working on finishing up the fixes and restoring everything to its normal status.

This was an unplanned and fairly horrible (and embarrassing) problem. This is the first time our own account management database has completely failed in the 10 years we’ve been providing Web hosting service. Obviously, we consider this sort of thing to be unacceptable, and we sincerely apologize for any inconvenience caused.

We will update this blog in the near future with more details.

Update: We have a new post which explains the problem (and solutions) in more detail.