Brief outage on web04 server June 1 2018

Between 5:58 and 6:04 AM Pacific time today (June 1, 2018), the “web04” server experienced a partial outage that led to some sites being unavailable or slow to respond.

The problem has now been resolved, and other servers were not affected. We apologize for the inconvenience this caused our customers.

Apache 2.4.25 web server upgrade (completed)

Update May 22, 2018: The change described below is complete on all servers.

Over the next few days, we’ll be updating the software used on each web server from Apache version 2.4.10 to 2.4.25.

Customers should not notice any changes or downtime. We’re mentioning it here just so that customers who do notice a change for some reason will know about it.

This is part of a series of software upgrades to allow us to add support for the newer HTTP/2 protocol on all sites in the near future.

As always, don’t hesitate to contact us if you have any trouble.

Apache 2.4 web server upgrade (complete)

Update May 3, 2018: The change described below is complete on all servers.

Over the next few days, we’ll be updating the software used on each web server from the Apache 2.2 series to the 2.4 series.

Customers should not notice any changes or downtime. We’re mentioning it here just so that customers who do notice a change for some reason will know about it.

While the Apache 2.4 series has some changes and new features, we’ve intentionally kept things compatible with older versions. In particular, we’re using mod_access_compat to ensure that existing “Allow / Deny” authorization directives work, and we’re using SSILegacyExprParser on to ensure that existing Server Side Includes work.

The Apache 2.4 series brings changes we’re using to improve our customers’ websites. For example, we’re already using OCSP stapling to speed up the initial connection to SSL sites in many browsers, and in the future we’ll be adding support for the newer HTTP/2 protocol. Neither of these were possible with older versions of Apache.

As always, don’t hesitate to contact us if you have any trouble.

Brief MySQL scheduled maintenance April 20, 2018 (completed)

Between 9:00 PM and 11:59 PM Pacific time on Friday, April 20, 2018, the MySQL database software on each of our servers will be upgraded from version 5.6.39 to 5.6.40. This will cause an approximately 60 second interruption of service on each MySQL-using customer website at some point during this period.

This upgrade is necessary for security reasons and to fix bugs in MySQL. We apologize for the inconvenience this causes.

Update 9:44 PM Pacific time: The maintenance was completed as planned and all services are running normally.

Wildcard Let’s Encrypt certificates now available

Let’s Encrypt recently started offering wildcard SSL certificates that work with any subdomain, without forcing you to get a new SSL certificate every time you change the hostnames you use.

If we host your site’s DNS nameservers (which is true for almost all of our hosting customers), we can now automatically provide you with a wildcard certificate, for free. We’ve already updated every existing Let’s Encrypt certificate to be a wildcard wherever possible.

If you’re still paying GoDaddy $349.99 a year for a wildcard SSL certificate, or paying Network Solutions $579 a year for it, now might be a good time to switch to our service. 😉 (In the last week, we’ve provided several million dollars worth of wildcard certificates to our customers even at GoDaddy’s introductory prices. You’re welcome!)

We’re using Let’s Encrypt wildcard certificates ourselves, too

We’re now also using these certificates on everything related to our own services, too, including our website, blog, FTP servers, and mail servers.

Almost all customers shouldn’t notice any change, but if you use secure connections with old or unusual programs that don’t handle SSL connections properly, you might be asked to “accept” the new certificate.

Read the rest of this entry »

Brief MySQL scheduled maintenance March 30, 2018 (completed)

Between 9:00 PM and 11:59 PM Pacific time on Friday, March 30, 2018, the MySQL database software on each of our servers will be upgraded from version 5.6.38 to 5.6.39. This will cause an approximately 60 second interruption of service on each MySQL-using customer website at some point during this period.

This upgrade is necessary for security reasons and to fix bugs in MySQL. We apologize for the inconvenience this causes.

Update 9:36 PM Pacific time: The maintenance was completed as planned and all services are running normally.

Scheduled maintenance March 9 & 10, 2018 for some servers (completed)

Update 11:55 PM Pacific time: The maintenance described below was completed successfully and all services are running normally.

Over the last year, we’ve been slowly upgrading our servers from Debian Linux version 7 (codename “wheezy”) to version 8 (codename “jessie”). We’ll be finishing that process over the next few weeks, with brief scheduled maintenance on each server.

The upgrade requires that each hosting server be taken offline for a few minutes over a three hour maintenance window and then restarted, causing brief scheduled “downtime” for websites and email on that server. The total downtime for any server should not exceed ten minutes during this three hour period.

Mail arriving while a server is being restarted will be queued and delivered after a short delay. No mail will be lost.

This coming weekend, we’ll be updating some (not all) web servers:

  • Friday, March 9, 9:00-11:59 PM Pacific: servers ending in digit “2” (completed)
  • Saturday, March 10, 9:00-11:59 PM Pacific: servers ending in digit “1” (completed)

So, for example, the “web12” server will be updated on March 9, and the “web01” server will be updated on March 10. This page explains how to find which server a site is on.

(Servers ending with other digits have already been updated in maintenance in previous weeks. This will complete the upgrades.)

Although the final step of the upgrade modifies hundreds of software packages on the server, we have tested it extensively don’t expect most customers to notice any change. Your website and email should continue working as they always have. However, if you do have any trouble, don’t hesitate to contact us.

Scheduled maintenance March 2 & 3, 2018 for some servers (completed)

Update 11:18 PM Pacific time: The maintenance described below was completed successfully and all services are running normally.

Over the last year, we’ve been slowly upgrading our servers from Debian Linux version 7 (codename “wheezy”) to version 8 (codename “jessie”). We’ll be finishing that process over the next few weeks, with brief scheduled maintenance on each server.

The upgrade requires that each hosting server be taken offline for a few minutes over a three hour maintenance window and then restarted, causing brief scheduled “downtime” for websites and email on that server. The total downtime for any server should not exceed ten minutes during this three hour period.

Mail arriving while a server is being restarted will be queued and delivered after a short delay. No mail will be lost.

This coming weekend, we’ll be updating some (not all) web servers:

  • Friday, March 2, 9:00-11:59 PM Pacific: servers ending in digits “6” and “5” (completed)
  • Saturday, March 3, 9:00-11:59 PM Pacific: servers ending in digits “4” and “3” (completed)

So, for example, the “web06” server will be updated on March 2, and the “web13” server will be updated on March 4. This page explains how to find which server a site is on.

(Servers ending in “0”, “9”, “8” and “7” have already been updated. Servers ending in “2” and “1” will be updated the following weekend; we’ll post a separate announcement about that.)

Although the final step of the upgrade modifies hundreds of software packages on the server, we have tested it extensively don’t expect most customers to notice any change. Your website and email should continue working as they always have. However, if you do have any trouble, don’t hesitate to contact us.

Scheduled maintenance February 23 & 24, 2018 for some servers (completed)

Update 11:18 PM Pacific time: The maintenance described below was completed successfully.

Over the last year, we’ve been slowly upgrading our servers from Debian Linux version 7 (codename “wheezy”) to version 8 (codename “jessie”). We’ll be finishing that process over the next few weeks, with brief scheduled maintenance on each server.

The upgrade requires that each hosting server be taken offline for a few minutes over a three hour maintenance window and then restarted, causing brief scheduled “downtime” for websites and email on that server. The total downtime for any server should not exceed ten minutes during this three hour period.

Mail arriving while a server is being restarted will be queued and delivered after a short delay. No mail will be lost.

This coming weekend, we’ll be updating some (not all) web servers:

  • Friday, February 23, 9:00-11:59 PM Pacific: servers ending in digits “0” and “9” (completed)
  • Saturday, February 24, 9:00-11:59 PM Pacific: servers ending in digits “8” and “7” (completed)

So, for example, the “web10” server will be updated on February 23, and the “web07” server will be updated on February 24. This page explains how to find which server a site is on.

(Additional servers will be updated the following weekends; we’ll post separate announcements about that.)

Although the final step of the upgrade modifies hundreds of software packages on the server, we have tested it extensively don’t expect most customers to notice any change. Your website and email should continue working as they always have. However, if you do have any trouble, don’t hesitate to contact us.

Brief outage on web07 server Feb 7 2018 (resolved)

Between 04:23 AM Pacific time (February 7 2018) and 04:33 AM Pacific time (February 7 2018), websites on our “web07” server were inaccessible while we restarted the server due to an unexpected kernel software issue.

All services are now running normally, and we are investigating the initial cause of the problem. We apologize for the inconvenience this caused.