2:16 PM Pacific time: We’re receiving scattered reports that some Comcast users, particularly in the San Francisco Bay Area, are unable to reach some sites on our servers (but others are fine).
This appears to be a general Comcast problem; there are many similar reports on Twitter unrelated to us.
Update: The problems were resolved by Comcast at 4:23 PM Pacific time, according to several Internet reports and our own testing.
Between 9:00 PM and 11:59 PM Pacific time on Friday January 30 2015, the MySQL database software on each of our servers will be upgraded from version 5.5.40 to 5.5.41. This will cause an approximately 60 second interruption of service on each MySQL-using customer Web site at some point during this period.
This upgrade is necessary for security reasons. We apologize for the inconvenience this causes.
Update 9:57 PM Pacific time: The maintenance was completed and all services are running normally.
Between 10:00 PM and 11:59 PM Pacific time on Saturday, January 24, each of our hosting servers will be restarted. This will cause a brief interruption of service (less than 10 minutes) for each site at some point during this 2 hour period.
Between 9:00 PM and 11:59 PM Pacific time on Friday October 24 2014, the MySQL database software on each of our servers will be upgraded from version 5.5.38 to 5.5.40. This will cause an approximately 60 second interruption of service on each MySQL-using customer Web site at some point during this period.
This upgrade is necessary for security reasons. We apologize for the inconvenience this causes.
Update 9:23 PM Pacific time: The maintenance was completed and all services are running normally.
Update October 14: This process described below is complete. All the updates were installed, and we’re now using only Debian wheezy on all servers.
Over the last year, we’ve been slowly upgrading our servers from Debian Linux version 6 (codename “squeeze”) to version 7 (codename “wheezy”).
All the “prominent” software (such as the Apache Web server, MySQL, PHP, the Linux kernel, and so on) was updated months ago, one piece at a time, usually with individual announcements here on our blog. Any software with security or compatibility issues has also already been upgraded.
What’s left at the end of that process are many “minor” packages, each probably used by less than 1% of our customers. We’ll be upgrading the rest of these over the next 30 days.
Between 5:29 and 5:42 PM Pacific time, one of our upstream network providers had connectivity problems, causing many people to be unable to reach any of our servers.
We have resolved this by removing that provider from our network while we investigate the issue with them, and all services are now working normally.
Earlier today (August 29, 2014), some people in the US Southeast who use certain ISPs were unable to reach our servers. This was caused by a problem at a third party that has been resolved.
Between 5:18 and 5:23 PM Pacific time today (Wednesday August 13, 2014), the
“web06” server experienced very high “load” due to a runaway MySQL database process. Other servers were not affected.
This caused an outage lasting approximately 5 minutes for all sites hosted on web06.
Between 9:00 PM and 11:59 PM Pacific time on Saturday August 9 2014, the MySQL database software on each of our servers will be upgraded from version 5.5.35 to 5.5.38. This will cause an approximately 30 second interruption of service on each MySQL-using customer Web site at some point during this period.
This upgrade is necessary for security reasons. We apologize for the inconvenience this causes.
Update 9:43 PM Pacific time: The maintenance was completed and all services are running normally.
7:11 AM Pacific time: Our technicians are investigating a possible outage on the “web14” server.
Final update: We found the source of the problem — there were a couple of short load spikes over a period of 10 minutes. We have taken steps so that it doesn’t happen again.
We know that uptime is critical for our customers, and we apologize for the inconvenience caused by this event.
