Brief server upgrades August 2019

Over the next few weeks, we’ll be retiring some old servers and moving sites on those servers to new (often faster) ones. Migrating a site takes just a few seconds in most cases, so it’s likely that neither you nor your visitors will notice this happening.

If someone does view your site while it’s being migrated, they’ll see a maintenance screen with a link to this blog post, like this:

If your site is in the small minority that has a large database (more than a few hundred MB), the migration could take a little longer — perhaps a few minutes. We try to do migrations during slow periods for each site, minimizing the impact on visitors.

Read the rest of this entry »

Network Outage for Subset of Customers July 23, 2019 (resolved)

There was an intermittent interruption of service for certain customers this morning (July 23rd, 2019) from about 9:15 AM – 10:00 AM Pacific Time.

During this time, a hardware failure in a router at one of our upstream data providers would have dropped incoming traffic for sites hosted in the 74.114.88.0/22 IP address range.

Once aware of the issue, we were able to reroute all traffic for that range through our own routers directly and avoid the issue. We’ve confirmed with the upstream provider that the faulty hardware has been identified and replaced.

We apologize for the trouble this caused customers who were affected.

Scheduled maintenance May 24, 2019 for some servers (complete)

Update May 24 11:00 PM Pacific time: The maintenance described below has been completed and all services are running normally.

Over the last year, we’ve been slowly upgrading our servers from Debian Linux version 8 (codename “jessie”) to version 9 (codename “stretch”). We’ll be finishing that process soon, with brief scheduled maintenance on each server.

The upgrade requires that each hosting server be taken offline for a few minutes over a three hour maintenance window and then restarted, causing brief scheduled “downtime” for websites and email on that server. The total downtime for any site should not exceed ten minutes during this three hour period.

Mail arriving while a server is being restarted will be queued and delivered after a short delay. No mail will be lost.

This coming weekend, we’ll be updating some (not all) web servers:

  • Friday, May 24, 9:00-11:59 PM Pacific: servers ending in digits “4”, “5” and “6”

So, for example, the “web05” and “web14” servers will be updated. This page explains how to find which server a site is on. (Servers ending in digits “0” – “3” and “7” – “9” have already been updated.)

Read the rest of this entry »

Scheduled maintenance May 17 & 18, 2019 for some servers (completed)

Update May 18 10:46 PM Pacific time: The maintenance described below has been completed and all services are running normally.

Over the last year, we’ve been slowly upgrading our servers from Debian Linux version 8 (codename “jessie”) to version 9 (codename “stretch”). We’ll be finishing that process over the next few weeks, with brief scheduled maintenance on each server.

The upgrade requires that each hosting server be taken offline for a few minutes over a three hour maintenance window and then restarted, causing brief scheduled “downtime” for websites and email on that server. The total downtime for any site should not exceed ten minutes during this three hour period.

Mail arriving while a server is being restarted will be queued and delivered after a short delay. No mail will be lost.

This coming weekend, we’ll be updating some (not all) web servers:

  • Friday, May 17, 9:00-11:59 PM Pacific: servers ending in digits “2” and “3” (completed)
  • Saturday, May 18, 9:00-11:59 PM Pacific: servers ending in digit “1” (completed)

So, for example, the “web03” server will be updated on May 17, and the “web11” server will be updated on May 18. This page explains how to find which server a site is on.

(Servers ending in digits “0”, “9”, “8” and “7” were updated last weekend, and servers ending in digits “4”, “5” and “6” will be updated the following weekend; we’ll post a separate announcement about that.)

Read the rest of this entry »

Scheduled maintenance May 10 & 11, 2019 for some servers (completed)

Update May 11 10:20 PM Pacific time: The maintenance described below has been completed and all services are running normally.

Over the last year, we’ve been slowly upgrading our servers from Debian Linux version 8 (codename “jessie”) to version 9 (codename “stretch”). We’ll be finishing that process over the next few weeks, with brief scheduled maintenance on each server.

The upgrade requires that each hosting server be taken offline for a few minutes over a three hour maintenance window and then restarted, causing brief scheduled “downtime” for websites and email on that server. The total downtime for any site should not exceed ten minutes during this three hour period.

Mail arriving while a server is being restarted will be queued and delivered after a short delay. No mail will be lost.

This coming weekend, we’ll be updating some (not all) web servers:

  • Friday, May 10, 9:00-11:59 PM Pacific: servers ending in digits “0” and “9” (completed)
  • Saturday, May 11, 9:00-11:59 PM Pacific: servers ending in digits “8” and “7” (completed)

So, for example, the “web10” server will be updated on May 10, and the “web07” server will be updated on May 11. This page explains how to find which server a site is on.

(Additional servers will be updated the following weekends; we’ll post separate announcements about that.)

Read the rest of this entry »

Brief outage on web11 server March 20, 2019 (resolved)

There was a brief interruption of service for customer websites on the “web11” server this morning (March 20, 2019) between 7:24 AM and 7:34 AM Pacific time.  (Other servers were not affected.)

This problem was caused by a bug in a script that resulted in a configuration error. The bug has been fixed so that it will not recur.

We apologize for the trouble this caused customers who were affected.

IPv6 enabled for new sites by default (and eventually for older sites, too)

We’ve supported IPv6 on customer websites for many years, but it didn’t default to “on”: customers had to explicitly enable it in our account management control panel.

Starting today, IPv6 is on by default for all new accounts signed up with us (although you can turn it off if you want).

In addition, we’re beginning a gradual process of slowly enabling IPv6 for existing sites if they haven’t chosen to disable it. If you don’t want IPv6 to be enabled for your site in the future, you should use our control panel to disable it.

Read the rest of this entry »

Brief MySQL scheduled maintenance February 8, 2019 (completed)

Update 9:32 PM Pacific time: the maintenance described below has been completed, and all services are running normally.

Between 9:00 PM and 11:59 PM Pacific time on Friday, February 8, 2019, the MySQL database software on each of our servers will be upgraded from MariaDB version 10.0.37 to 10.0.38 (equivalent to MySQL 5.6.43). This will cause an approximately 60 second interruption of service on each MySQL-using customer website at some point during this period.

This upgrade is necessary for security reasons and to fix bugs in MySQL. We apologize for the inconvenience this causes.

Amazon Web Services (AWS) network problems January 16, 2019 (resolved)

Update 9:30 PM Pacific time: the problem described below is resolved, as Amazon is no longer sending data through the problematic route to our servers.

Original post: Our monitoring systems are showing that this evening, there have been short periods of network failures between our data center and some Amazon Web Services (AWS) data centers on the US East Coast. This appears to be due to a problem Amazon is having connecting to an intermediate “Internet backbone” connection in Virginia run by a third party.

This isn’t affecting other connections, so most of our customers are unaffected, and we see no overall drop in traffic.

Read the rest of this entry »

SSLv3 disabled on all servers

We’ve updated the SSL/TLS security settings on our servers to match current “best practices” for security, disabling the long-obsolete, insecure “SSLv3” in all cases.

Our customers shouldn’t notice any changes. We made this change on our own websites a long time ago with no reports of problems, and nearly all of the largest sites on the Internet have done the same. We’re just mentioning this so that people know to contact us in the unlikely event they do have any trouble.

That said, if you do have any trouble, it’s probably because you’re using a long-outdated, insecure web browser that you should update. You can check your browser by visiting www.howsmyssl.com. If you can’t update it, using a different browser on your computer will probably help.