Our primary data center experienced network routing problems between 2:06 PM and 2:49 PM Pacific time today (April 10, 2011).
During this time, packets from some (but not all) places on on the Internet were unreliable, causing connection problems. The data center technicians have resolved the issue, and all services are now working normally.
We don’t consider this normal or acceptable, and we sincerely apologize for the inconvenience this caused. (We do not yet have a full explanation from the data center about the root cause, but have requested one so that we can be sure it won’t recur.)
Our primary data center had another power interruption this morning at 7:28 am (Pacific time). All of our servers lost power and then had it restored, thus rebooting them. All customer web sites were unavailable during this time. Incoming email would have simply been delayed during the downtime, not lost. When the servers came back online e-mail may have seemed sluggish to some customers for a while but this should also be fixed now.
This incident follows another power incident the previous Saturday night. We are working with the data center to get more details, including an estimate of when they will have replaced any faulty equipment. We will update this post as more information becomes available.
Update Nov. 29: The final data center report is that on the night of November 20, lightning strikes damaged both of the redundant UPS systems, interrupting data center power for a few seconds. The UPS manufacturer scheduled replacements for November 23, but another PG&E utility power interruption lasting a few seconds occurred that morning before it was finished. The UPS manufacturer has since replaced all damaged parts, restoring full redundancy. In addition, the UPS manufacturer has overhauled each unit, replacing and upgrading other parts to increase robustness. We take this very seriously — it’s at the core of what we do — and we will continue to work with the data center to ensure that their infrastructure meets our high standards.
A major power failure at our primary data center in Fremont, California, caused a complete outage for nearly all services beginning at 8:32 PM Pacific time Saturday night. It lasted between six and 13 minutes, depending on the server. Only our blog and redundant DNS infrastructure was unaffected.
All services are now fully operational; please don’t hesitate to contact us if you have any questions. We sincerely apologize for the inconvenience this caused our customers.
Read the rest of this entry »
Between 10:00 PM and 11:59 PM Pacific time this Saturday, August 28, all our hosting servers will be restarted. As a result, Web site service and the ability to read incoming e-mail will be unavailable for approximately five minutes at some point during this maintenance “window”.
Read the rest of this entry »
Between 10:00 PM and 11:59 PM Pacific time this Saturday, May 22, all our hosting servers will be restarted. As a result, Web site service and the ability to read incoming e-mail will be unavailable for approximately five minutes at some point during this maintenance “window”.
Read the rest of this entry »
Between 7:00 and 7:45 PM Pacific time Thursday night (March 11), we received two reports of slow or nonexistent network connections to sites on our servers.
Our automated monitoring systems didn’t detect any general problems, so the majority of customers were certainly unaffected — but we suspect that one of the “Internet backbones” between the affected customers and our data center had high packet loss during that period.
Both customers reported that the problem resolved itself by 7:45, and we haven’t received similar reports since, so there does not appear to be be an ongoing problem. We’ll continue to monitor it closely.
Due to software updates on our servers, Web hosting customers will experience about seven minutes of scheduled maintenance downtime between 11 PM and 1 AM Pacific time starting on one of the following nights, depending on which server your site is on:
- Thursday, February 4 (servers beginning with the letter “L”, such as “lrrr”)
- Friday, February 5 (all other servers beginning with letters “F-Z”, such as “farnsworth”)
- Saturday, February 6 (servers beginning with letters “A-E”, such as “amy”)
Read the rest of this entry »
Between 10:00 PM and 11:59 PM Pacific time this Sunday January 3, all our hosting servers will be restarted. As a result, Web site service and the ability to read incoming e-mail will be unavailable for approximately five minutes at some point during this maintenance “window”.
Read the rest of this entry »
Between 10:00 PM and 11:59 PM Pacific time this Friday September 11, all our servers will be restarted. As a result, Web site service and the ability to read incoming e-mail will be unavailable for approximately five minutes at some point during this maintenance “window”.
Read the rest of this entry »
Some of our customers may have noticed “high packet loss” today from about noon to 12:25 PM (Pacific time). This could make it seem like Web sites hosted on our servers were loading slowly, or even timing out.
The problem has been resolved by our upstream provider, but we are working with them to make sure it doesn’t recur.
