Scheduled maintenance May 10 & 11, 2019 for some servers (completed)

Update May 11 10:20 PM Pacific time: The maintenance described below has been completed and all services are running normally.

Over the last year, we’ve been slowly upgrading our servers from Debian Linux version 8 (codename “jessie”) to version 9 (codename “stretch”). We’ll be finishing that process over the next few weeks, with brief scheduled maintenance on each server.

The upgrade requires that each hosting server be taken offline for a few minutes over a three hour maintenance window and then restarted, causing brief scheduled “downtime” for websites and email on that server. The total downtime for any site should not exceed ten minutes during this three hour period.

Mail arriving while a server is being restarted will be queued and delivered after a short delay. No mail will be lost.

This coming weekend, we’ll be updating some (not all) web servers:

  • Friday, May 10, 9:00-11:59 PM Pacific: servers ending in digits “0” and “9” (completed)
  • Saturday, May 11, 9:00-11:59 PM Pacific: servers ending in digits “8” and “7” (completed)

So, for example, the “web10” server will be updated on May 10, and the “web07” server will be updated on May 11. This page explains how to find which server a site is on.

(Additional servers will be updated the following weekends; we’ll post separate announcements about that.)

Read the rest of this entry »

Brief outage on web11 server March 20, 2019 (resolved)

There was a brief interruption of service for customer websites on the “web11” server this morning (March 20, 2019) between 7:24 AM and 7:34 AM Pacific time.  (Other servers were not affected.)

This problem was caused by a bug in a script that resulted in a configuration error. The bug has been fixed so that it will not recur.

We apologize for the trouble this caused customers who were affected.

Brief MySQL scheduled maintenance February 8, 2019 (completed)

Update 9:32 PM Pacific time: the maintenance described below has been completed, and all services are running normally.

Between 9:00 PM and 11:59 PM Pacific time on Friday, February 8, 2019, the MySQL database software on each of our servers will be upgraded from MariaDB version 10.0.37 to 10.0.38 (equivalent to MySQL 5.6.43). This will cause an approximately 60 second interruption of service on each MySQL-using customer website at some point during this period.

This upgrade is necessary for security reasons and to fix bugs in MySQL. We apologize for the inconvenience this causes.

Amazon Web Services (AWS) network problems January 16, 2019 (resolved)

Update 9:30 PM Pacific time: the problem described below is resolved, as Amazon is no longer sending data through the problematic route to our servers.

Original post: Our monitoring systems are showing that this evening, there have been short periods of network failures between our data center and some Amazon Web Services (AWS) data centers on the US East Coast. This appears to be due to a problem Amazon is having connecting to an intermediate “Internet backbone” connection in Virginia run by a third party.

This isn’t affecting other connections, so most of our customers are unaffected, and we see no overall drop in traffic.

Read the rest of this entry »

Brief MySQL scheduled maintenance December 1, 2018 for some servers (completed)

Update 11:22 PM Pacific time Dec. 1: The maintenance described below has been completed and all services are running normally.

Between 10:00 PM and 11:59 PM Pacific time on Saturday, December 1, 2018, the MySQL database software on some hosting servers will be updated. This will cause an approximately two minute interruption of service on each MySQL-using customer Web site (including WordPress sites) at some point during this period.

Specifically, this will affect servers ending in digits 0, 1, 2 and 3, such as “web01” and “web10”. (All other servers were previously updated on November 17 or November 24.)

Read the rest of this entry »

Brief MySQL scheduled maintenance November 24, 2018 for some servers (completed)

Update 10:39 PM Pacific time Nov. 24: The maintenance described below has been completed and all services are running normally.

Between 10:00 PM and 11:59 PM Pacific time on Saturday, November 24, 2018, the MySQL database software on some hosting servers will be updated. This will cause an approximately two minute interruption of service on each MySQL-using customer Web site (including WordPress sites) at some point during this period.

Specifically, this will affect servers ending in digits 4, 5 and 6, such as “web04” and “web14”. Servers ending in digits 0-3 are not affected now and will be updated later; servers ending with digits 7-9 were previously updated.

Read the rest of this entry »

Brief MySQL scheduled maintenance November 17, 2018 for some servers (completed)

Update 10:46 PM Pacific time Nov. 17: The maintenance described below has been completed and all services are running normally.

Between 10:00 PM and 11:59 PM Pacific time on Saturday, November 17, 2018, the MySQL database software on some hosting servers will be updated. This will cause an approximately two minute interruption of service on each MySQL-using customer Web site (including WordPress sites) at some point during this period.

Specifically, this will affect servers ending in digits 7, 8 and 9, such as “web07” and “web08”. Servers ending in digits 0-6 are not affected now and will be updated later this month.

Read the rest of this entry »

Brief scheduled maintenance October 27, 2018 (completed)

Between 10:00 PM and 11:59 PM Pacific time on Saturday, October 27, 2018, each of our hosting servers will be restarted. This will cause a brief interruption of service (less than 5 minutes) for each site at some point during this period.

Read the rest of this entry »

Brief outage on web03 server October 19, 2018 (resolved)

Between 10:58 and 11:09 AM Pacific time today (October 19, 2018), the “web03” server experienced an outage that led to an interruption in service for some customer websites and email.

The problem has now been resolved, and other servers were not affected. We apologize for the inconvenience this caused our customers.

We’re investigating the root cause (a Linux “kernel panic”) to ensure it does not recur.

Brief scheduled maintenance on web11 server July 21, 2018 (completed)

At approximately 10:30 PM Pacific time on Saturday, July 21, 2018, the web11 server will be restarted.

This will cause a brief interruption of service (less than 5 minutes) for sites and email on this server. Sites on other servers will not be affected.

Read the rest of this entry »