Brief MySQL scheduled maintenance April 20, 2018

Between 9:00 PM and 11:59 PM Pacific time on Friday, April 20, 2018, the MySQL database software on each of our servers will be upgraded from version 5.6.39 to 5.6.40. This will cause an approximately 60 second interruption of service on each MySQL-using customer website at some point during this period.

This upgrade is necessary for security reasons and to fix bugs in MySQL. We apologize for the inconvenience this causes.

Wildcard Let’s Encrypt certificates now available

Let’s Encrypt recently started offering wildcard SSL certificates that work with any subdomain, without forcing you to get a new SSL certificate every time you change the hostnames you use.

If we host your site’s DNS nameservers (which is true for almost all of our hosting customers), we can now automatically provide you with a wildcard certificate, for free. We’ve already updated every existing Let’s Encrypt certificate to be a wildcard wherever possible.

If you’re still paying GoDaddy $349.99 a year for a wildcard SSL certificate, or paying Network Solutions $579 a year for it, now might be a good time to switch to our service. 😉 (In the last week, we’ve provided several million dollars worth of wildcard certificates to our customers even at GoDaddy’s introductory prices. You’re welcome!)

We’re using Let’s Encrypt wildcard certificates ourselves, too

We’re now also using these certificates on everything related to our own services, too, including our website, blog, FTP servers, and mail servers.

Almost all customers shouldn’t notice any change, but if you use secure connections with old or unusual programs that don’t handle SSL connections properly, you might be asked to “accept” the new certificate.

Read the rest of this entry »

Brief MySQL scheduled maintenance March 30, 2018 (completed)

Between 9:00 PM and 11:59 PM Pacific time on Friday, March 30, 2018, the MySQL database software on each of our servers will be upgraded from version 5.6.38 to 5.6.39. This will cause an approximately 60 second interruption of service on each MySQL-using customer website at some point during this period.

This upgrade is necessary for security reasons and to fix bugs in MySQL. We apologize for the inconvenience this causes.

Update 9:36 PM Pacific time: The maintenance was completed as planned and all services are running normally.

Scheduled maintenance March 9 & 10, 2018 for some servers (completed)

Update 11:55 PM Pacific time: The maintenance described below was completed successfully and all services are running normally.

Over the last year, we’ve been slowly upgrading our servers from Debian Linux version 7 (codename “wheezy”) to version 8 (codename “jessie”). We’ll be finishing that process over the next few weeks, with brief scheduled maintenance on each server.

The upgrade requires that each hosting server be taken offline for a few minutes over a three hour maintenance window and then restarted, causing brief scheduled “downtime” for websites and email on that server. The total downtime for any server should not exceed ten minutes during this three hour period.

Mail arriving while a server is being restarted will be queued and delivered after a short delay. No mail will be lost.

This coming weekend, we’ll be updating some (not all) web servers:

  • Friday, March 9, 9:00-11:59 PM Pacific: servers ending in digit “2” (completed)
  • Saturday, March 10, 9:00-11:59 PM Pacific: servers ending in digit “1” (completed)

So, for example, the “web12” server will be updated on March 9, and the “web01” server will be updated on March 10. This page explains how to find which server a site is on.

(Servers ending with other digits have already been updated in maintenance in previous weeks. This will complete the upgrades.)

Although the final step of the upgrade modifies hundreds of software packages on the server, we have tested it extensively don’t expect most customers to notice any change. Your website and email should continue working as they always have. However, if you do have any trouble, don’t hesitate to contact us.

Scheduled maintenance March 2 & 3, 2018 for some servers (completed)

Update 11:18 PM Pacific time: The maintenance described below was completed successfully and all services are running normally.

Over the last year, we’ve been slowly upgrading our servers from Debian Linux version 7 (codename “wheezy”) to version 8 (codename “jessie”). We’ll be finishing that process over the next few weeks, with brief scheduled maintenance on each server.

The upgrade requires that each hosting server be taken offline for a few minutes over a three hour maintenance window and then restarted, causing brief scheduled “downtime” for websites and email on that server. The total downtime for any server should not exceed ten minutes during this three hour period.

Mail arriving while a server is being restarted will be queued and delivered after a short delay. No mail will be lost.

This coming weekend, we’ll be updating some (not all) web servers:

  • Friday, March 2, 9:00-11:59 PM Pacific: servers ending in digits “6” and “5” (completed)
  • Saturday, March 3, 9:00-11:59 PM Pacific: servers ending in digits “4” and “3” (completed)

So, for example, the “web06” server will be updated on March 2, and the “web13” server will be updated on March 4. This page explains how to find which server a site is on.

(Servers ending in “0”, “9”, “8” and “7” have already been updated. Servers ending in “2” and “1” will be updated the following weekend; we’ll post a separate announcement about that.)

Although the final step of the upgrade modifies hundreds of software packages on the server, we have tested it extensively don’t expect most customers to notice any change. Your website and email should continue working as they always have. However, if you do have any trouble, don’t hesitate to contact us.

Scheduled maintenance February 23 & 24, 2018 for some servers (completed)

Update 11:18 PM Pacific time: The maintenance described below was completed successfully.

Over the last year, we’ve been slowly upgrading our servers from Debian Linux version 7 (codename “wheezy”) to version 8 (codename “jessie”). We’ll be finishing that process over the next few weeks, with brief scheduled maintenance on each server.

The upgrade requires that each hosting server be taken offline for a few minutes over a three hour maintenance window and then restarted, causing brief scheduled “downtime” for websites and email on that server. The total downtime for any server should not exceed ten minutes during this three hour period.

Mail arriving while a server is being restarted will be queued and delivered after a short delay. No mail will be lost.

This coming weekend, we’ll be updating some (not all) web servers:

  • Friday, February 23, 9:00-11:59 PM Pacific: servers ending in digits “0” and “9” (completed)
  • Saturday, February 24, 9:00-11:59 PM Pacific: servers ending in digits “8” and “7” (completed)

So, for example, the “web10” server will be updated on February 23, and the “web07” server will be updated on February 24. This page explains how to find which server a site is on.

(Additional servers will be updated the following weekends; we’ll post separate announcements about that.)

Although the final step of the upgrade modifies hundreds of software packages on the server, we have tested it extensively don’t expect most customers to notice any change. Your website and email should continue working as they always have. However, if you do have any trouble, don’t hesitate to contact us.

Brief outage on web07 server Feb 7 2018 (resolved)

Between 04:23 AM Pacific time (February 7 2018) and 04:33 AM Pacific time (February 7 2018), websites on our “web07” server were inaccessible while we restarted the server due to an unexpected kernel software issue.

All services are now running normally, and we are investigating the initial cause of the problem. We apologize for the inconvenience this caused.

Mailman mailing list software upgraded to version 2.1.26

The authors of the Mailman mailing list software we provide for customers have recently released version 2.1.26 to fix several bugs.

We’ve upgraded the Mailman software on our servers as a result.

Users of Mailman lists shouldn’t notice any changes, but as always, don’t hesitate to contact us if you have any questions or see any problems.

Brief scheduled maintenance January 8, 2018 (completed)

Between 9:00 PM and 11:59 PM Pacific time on Monday, January 8, 2018, each of our hosting servers will be restarted. This will cause a brief interruption of service (less than 5 minutes) for each site at some point during this period.

Read the rest of this entry »

Web02 RAID and MySQL problems (resolved)

Between 05:46 AM Pacific time and 06:58 AM Pacific time (December 29 2017) some Web sites on our web02 server were inaccessible. This was caused by the failure of a disk in the RAID array. When the disk failed MySQL continued to read corrupted data from it’s file cache rather than using another disk in the RAID array. The cache has now been cleared and the server restarted. This was unrelated to the problems on our web13 server earlier in the day.