Scattered network problem reports July 19, 2012 (resolved)

Update 1:41 PM Pacific time: A contact at Level 3 Communications confirms that their San Francisco Bay Area network was disrupted by a “configuration error”, causing problems for a great deal of Internet traffic that passes through Level 3 (not related to us in particular). Level 3 has corrected the problem, so we’re marking this issue as “closed”.

Original post (resolved as described above)

For the last couple of hours, we’ve been receiving scattered reports of people using smaller ISPs who are unable to connect to our servers, although most people are unaffected.

All our services are running normally. This problem appears caused by a network routing problem at a major backbone unrelated to us, “Level 3 Communications”, that is affecting connections to many different companies (for example, this Mozilla tweet describes the same thing, as does this thread on the Internet network operator’s outages list).

Most larger ISPs are handling this by routing the data through alternate “paths” that don’t go through Level 3. But it appears that some smaller ISPs completely rely on Level 3 for at least part of their connectivity and are not able to “route around” this problem. If you’re seeing it happen (for example, if you can’t access your own site), rest assured that most people are still able to reach your site, and that Level 3 is aware of the issue and is working to resolve it.