Amazon Web Services (AWS) network problems January 16, 2019 (resolved)

Update 9:30 PM Pacific time: the problem described below is resolved, as Amazon is no longer sending data through the problematic route to our servers.

Original post: Our monitoring systems are showing that this evening, there have been short periods of network failures between our data center and some Amazon Web Services (AWS) data centers on the US East Coast. This appears to be due to a problem Amazon is having connecting to an intermediate “Internet backbone” connection in Virginia run by a third party.

This isn’t affecting other connections, so most of our customers are unaffected, and we see no overall drop in traffic.

However, we’ve received a couple of reports from people who subscribe to monitoring services that run from AWS, saying they detected sites as “down” even though humans outside AWS could still visit them. This problem could also cause sites to seem slow if they use WordPress plugins (etc.) that connect to external services running on AWS, because each page would be delayed by the time it takes that plug-in to “time out”.

This problem has so far happened between 6:16 and 6:21 PM Pacific time, again between 6:37 and 6:49 PM, and again between 7:20 and 7:28 PM.

We’ll continue to monitor this issue closely and post an update when we’re confident that it’s been resolved.

By the way, if you ever wonder if your site is working for other people, try Uptrends and/or GeoPeeker. These check your site from many different locations around the world, so you can quickly see whether the site’s servers are down, or whether the problem is just a routing problem that isn’t affecting most other people.