Service outage May 6, 2011 (resolved)

May 6, 4:43 AM Pacific time: An outage at our primary data center caused a complete service interruption for all customers.

Update 5:08 AM: All services have been restored and are working normally.

Read the rest of this entry »

Brief scheduled maintenance on “fry” and “bender” servers (completed)

The “fry” and “bender” Web servers will be restarted between 11:00 and 11:15 PM Pacific time tonight (Friday, April 29, 2011). This will cause a five-minute interruption of Web and e-mail service for customers on those servers.

Other servers will not be affected, and incoming mail will only be delayed, not lost.

Read the rest of this entry »

Problem with “fry” server (resolved)

8:52 PM Pacific time: We’re investigating a problem with the “fry” hosting server that’s requiring us to restart it; further details in a few minutes.

Update 9:42 PM Pacific time: The “fry” server was restarted, but a technician will be doing some maintenance on the server for approximately an hour. This will require a reboot, meaning the server will be unavailable for approximately 5 – 10 minutes. Web service will be unavailable during that time. E-mail service on that server also will be unavailable; delivery of new incoming mail will suspend during that time and then resume when the server comes back; no e-mail will be lost.

All others servers are unaffected.

Update 10:50 PM Pacific time: The “fry” web server will be rebooted in about 10 minutes, at approximately 11:00 PM Pacific time.

Update 11:10 PM Pacific time: The “fry” web server was successfully rebooted as planned. There may be more maintenance on the server this weekend; watch our blog or follow us on Twitter for updates.

TLS now supported with FTP

Our FTP servers now support TLS/SSL encryption of FTP passwords, adding more security to FTP.

Confusingly, there are a variety of different SSL/TLS encryption schemes for FTP offered by various FTP clients. The one we support is the most widespread, known as “explicit TLS encryption” of the FTP command channel. It’s defined in RFC 4217.

Encryption is supported by many popular FTP clients, including the FileZilla FTP client. (The quickest way to use it in FileZilla is to put ftpes://ftp.tigertech.net in the QuickConnect “Host” box, then accept the “Unknown certificate”.)

Read the rest of this entry »

Network issues April 10, 2011

Our primary data center experienced network routing problems between 2:06 PM and 2:49 PM Pacific time today (April 10, 2011).

During this time, packets from some (but not all) places on on the Internet were unreliable, causing connection problems. The data center technicians have resolved the issue, and all services are now working normally.

We don’t consider this normal or acceptable, and we sincerely apologize for the inconvenience this caused. (We do not yet have a full explanation from the data center about the root cause, but have requested one so that we can be sure it won’t recur.)

Brief MySQL load problems (resolved)

We had a couple of instances of MySQL queries overloading the bender server today. The first one happened at about 3:41 AM (Pacific time) and the second one happened at about 7:48 AM. Each occurrence lasted about 20 minutes. The problem each time was that a database was running extremely inefficient queries. Each time we fixed the problem by creating indexes so that the queries could then run in a fraction of the time previously required.

We apologize for any inconvenience caused by this problem. Visitors to your Web site (on the bender server) might have seen reduced performance (or, in rare cases, 503 errors). E-mail was not affected. We don’t consider this type of problem to be acceptable. These problems should not recur since the indexes have been created.

When search engines swarm new posts

We saw an interesting problem today. One of our customers’ Web sites uses WordPress with WP Super Cache to (dramatically) improve its performance. Every time the customer posts new content, though, the site is immediately swarmed by search engines, feeds, robots, and other non-humans retrieving the new post. There are a lot of unnecessary duplicate requests, but even excluding the duplicates there are hundreds of requests arriving almost simultaneously.

Unfortunately, WP Super Cache is configured by default not to serve cached results to any request that contains an “equals sign” in the query string — and the plugin that notifies the other sites of new content is including an equals sign.

So rather than being immediately served from the cache, all of the new requests were run through WordPress PHP scripts, driving up the script usage and causing “503 Service Unavailable” errors for up to two minutes on that Web site (not for other Web sites on the same Web server, though; we have protection against that).

Read the rest of this entry »

Brief scheduled maintenance on elzar server (completed)

The “elzar” Web server will be restarted at 10 PM Pacific time tonight (February 25). This will cause a five-minute interruption of Web and e-mail service for customers on that server.

Other servers will not be affected, and incoming mail will only be delayed, not lost.

This restart is necessary to fix a memory problem. We apologize for the inconvenience.

Update 10:03 PM: The maintenance was completed with less than 3 minutes downtime.

President’s Day 2011 holiday hours

Our business offices will be closed on Monday, February 21 to observe the US legal holiday. As always, we’ll provide same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren’t time-sensitive (including most billing matters) may not be answered until the next day, and telephone support (via callbacks) will be available only for urgent problems.

Slow connections for some customers (resolved)

Beginning mid-day yesterday (Sunday February 6th), we received reports from a small number of customers that connections to our data center were slow. This was traced to a problem with a combination router/Ethernet switch in a cabinet in our data center corrupting some packets of data. The router has been replaced and the problem resolved.

Read the rest of this entry »