Between 7:56 and 8:19 AM Pacific time today (August 26, 2022), a network outage at our data center caused most sites to be unavailable.
The underlying cause of the problem has been identified and resolved, and should not recur.
We apologize to our customers affected by this incident: we take reliability seriously, and don’t consider any outage normal or acceptable.
Between 5:18 PM and 5:41 PM Pacific time today, our “My Account” control panel was intermittently unavailable. This problem has been fixed and services are operating normally; we apologize for any inconvenience this caused.
Over the next three weeks, we’ll be updating the MariaDB (MySQL) database server software on all our servers from the MariaDB 10.3 series to the MariaDB 10.5 series (equivalent to the Oracle MySQL 8.0 series).
The MariaDB/MySQL database stores pages for WordPress and other sites that are run by scripts. Customers should not notice any difference after this change; we’re upgrading it to a more recent version simply to make sure it’s as fast, reliable and secure as possible. We’ve been using the new version on internal and test servers for some time.
At the moment the software is updated on a server, WordPress and other database-backed sites on that server will have 30-60 seconds of unavoidable “downtime”. To minimize the impact of that, we do these upgrades only on Friday, Saturday and Sunday nights between 9 PM and midnight Pacific time (midnight-3 AM Eastern time). We expect this process to be complete on all servers by September 6.
Beyond that one-time brief interruption in service, customers should not notice any difference to how their site works, as we said. But as always, don’t hesitate to contact us if you have any trouble or questions.
Update 9:24 PM Pacific time September 5: This upgrade has been completed on all servers.
Between 10:00 PM and 11:59 PM Pacific time on Friday, July 16, each of our hosting servers will be restarted. This will cause a brief interruption of service (less than 5 minutes) for each site at some point during this 2 hour period.
Read the rest of this entry »We’re upgrading our servers from Debian Linux version 9 (codename “stretch”) to version 10 (codename “buster”). We’ll be finishing that process over the next couple of weeks.
Your website and email should continue working as they always have, and we don’t expect any downtime. However, if you do have any trouble, don’t hesitate to contact us.
Although the final steps of the upgrade modify hundreds of software packages on the server, we’ve tested it extensively and don’t expect most customers to notice any change. (We’ve already been using the newer version of Debian for our own servers.)
Read the rest of this entry »Update 9:16 PM June 27: the maintenance described below has been completed on all servers, so all servers are now running the MariaDB 10.3 series.
Over the next three weeks, we’ll be updating the MariaDB (MySQL) database server software on all our servers from the MariaDB 10.1 series to the MariaDB 10.3 series (equivalent to the Oracle MySQL 5.7 series).
The MariaDB/MySQL database software is what stores pages for WordPress and other sites that are run by scripts. Customers should not notice any difference after this change; we’re upgrading it to a more recent version simply to make sure it’s as fast, reliable and secure as possible. We’ve been using the new version on internal and test servers for some time.
At the moment the software is updated on a server, WordPress and other database-backed sites on that server will have 30-60 seconds of unavoidable “downtime”. To minimize the impact of that, we do these upgrades only on Friday, Saturday and Sunday nights between 9 PM and midnight Pacific time (midnight-3 AM Eastern time). This process will be complete on all servers by June 28.
Beyond that one-time brief interruption in service, customers should not notice any difference to how their site works, as we said. But as always, don’t hesitate to contact us if you have any trouble or questions.
Update 10:58 PM Pacific time: the maintenance described below has been completed, and all services are running normally.
Between 9:00 PM and 11:59 PM Pacific time on Saturday, February 15, 2020, the MySQL database software on each of our servers will be upgraded from MariaDB version 10.0.41 to 10.0.44 (roughly equivalent to MySQL 5.6.47). This will cause an approximately 60 second interruption of service on each MySQL-using customer website at some point during this period.
This upgrade is necessary for security reasons and to fix bugs in MySQL.
In addition, the web14 server will be restarted during this period for a hardware upgrade, causing an approximately 3-minute additional outage for sites and email on that server only.
We apologize for the inconvenience this causes.
We’ve updated the MariaDB / MySQL database software on all our servers to version 10.1.41. This was part of a general update of Debian Linux to version 9.10 and then 9.11.
This upgrade should not be noticeable to our customers in any way, but of course, don’t hesitate to contact us if you have any questions or difficulties.
Between 4:06 PM and 5:03 PM Pacific time today (Aug 22, 2019), sites hosted on the “web08” server were intermittently unavailable due to a technical problem.
The problem was caused by a flood of connections that our systems failed to automatically block as they should. We’ve modified the software that handles this, and we do not expect the problem to recur.
We sincerely apologize to customers affected by this problem; we don’t consider it normal or acceptable, and strive to avoid incidents like this.
There was an intermittent interruption of service for certain customers this morning (July 23rd, 2019) from about 9:15 AM – 10:00 AM Pacific Time.
During this time, a hardware failure in a router at one of our upstream data providers would have dropped incoming traffic for sites hosted in the 74.114.88.0/22 IP address range.
Once aware of the issue, we were able to reroute all traffic for that range through our own routers directly and avoid the issue. We’ve confirmed with the upstream provider that the faulty hardware has been identified and replaced.
We apologize for the trouble this caused customers who were affected.
