MySQL updated to version 5.5.37

We’ve updated the MySQL database software on our servers from version 5.5.35 to 5.5.37 for security reasons.

Customers should not notice any changes, as the update merely fixes bugs and doesn’t introduce new features. But as always, don’t hesitate to contact us if you have any questions.

Some Mailman list subscriptions delayed (resolved)

We recently upgraded Mailman to fix a problem for Yahoo and AOL users, which has worked well.

Today we found a problem where since the initial upgrade on April 24, people who tried subscribing to a Mailman list by sending an e-mail message (instead of using the more common Web interface) weren’t properly added to the list.

This was caused by an incompatibility between our mail system and one of the Mailman software changes. We didn’t receive a report of this problem until today because few people try to subscribe this way.

We’ve resolved this: Any delayed subscriptions have been correctly handled and the incompatibility has been fixed. We apologize to anyone affected by it.

Mailman updated to avoid DMARC problem

In recent days, Yahoo and AOL made technical changes that prevented their own users from sending messages to most mailing lists. (That’s a simplified summary; your favorite search engine will show the gory details if you search for “Yahoo DMARC mailing lists”.)

If you use the Mailman software to run discussion lists, like many of our customers, this is a problem. To fix it, the authors of the Mailman software are making a new version to work around the Yahoo and AOL change.

That new version hasn’t been officially released yet, but we think this is so important that we’ve upgraded our installed copy of Mailman to a “pre-release” version of the fixed software.

Yahoo and AOL users should now be able to send to our customers’ Mailman lists without trouble.

Read the rest of this entry »

Seeing more bounces for yahoo.com addresses? It’s not just you.

Over the last couple of days, mail from yahoo.com addresses has been “bouncing” a lot more than usual, often with error messages about “policy” or “DMARC” problems. This is especially true for messages involving mailing lists or forwarding from other addresses.

If you see this happening, it isn’t because of anything wrong on your end, or our end. It’s happening all over the Internet.

Read the rest of this entry »

Brief scheduled maintenance on web07 server April 5, 2014 (completed)

At approximately 10:00 PM Pacific time Saturday April 5, 2014, the “web07” server will be restarted. This will cause a short interruption of service for Web sites on that server lasting less than 8 minutes.

Other servers will not be affected. Mail for customers on this server will be queued and delivered after a short delay.

Read the rest of this entry »

Brief MySQL problem on web12 server (resolved)

Between 10:41 and 10:49 AM Pacific time today (March 25, 2014), some MySQL database queries on the web12 server ran very slowly and caused “timeouts” for scripts. (No other servers were affected.)

The problem is fixed, and we’ve added an automated check to prevent it from recurring.

Read the rest of this entry »

Brief scheduled maintenance on web05, web07 & web08 servers March 1, 2014 (completed)

Update: The maintenance described below was completed with less than 10 minutes downtime per server.

Between 10:00 PM and 11:59 PM Pacific time on Saturday March 1 2014, the “web05“, “web07” and “web08” servers will be restarted. This will cause a short interruption of service on each of these three servers, lasting less than 10 minutes, at some point during this two hour period.

Other servers will not be affected. Incoming mail for customers on these servers will not be lost, but will be queued and delivered after the maintenance.

Read the rest of this entry »

Brief Apache problem on web13 server (resolved)

At approximately 2:45 PM Pacific time February 6, 2014, the Apache Web server running on the web13 server hung and needed to be manually restarted. We fixed the problem at 2:56 PM.

Obviously this sort of problem should not happen. We are monitoring it closely, and investigating it in more detail to try to find the root cause, so that we can take corrective action as necessary to prevent it from happening again. We apologize for the inconvenience caused by this problem.

Upcoming scheduled maintenance for MySQL 5.5 upgrades (completed)

Over the next ten days (February 7-15, 2014), we’ll be upgrading the MySQL database on all of our servers from version 5.1.72 to 5.5.35. The upgrade requires that each Web server be taken offline for two minutes, causing brief scheduled “downtime”.

Read the rest of this entry »

Brief denial of service on web04 server on January 15, 2014 (resolved)

At approximately 9:30 PM Pacific time, all of our servers began to experience a large “distributed denial of service” (DDoS) attack via attempts to login to blogs using the standard WordPress wp-login.php script. This attack was very broad: it attacked thousands of sites across all of our servers, and it came from a huge number of IP addresses.

Processing these requests caused the overall load on all servers to increase. On “web04” the increase was enough to cause the server to start returning “503” errors for Web page requests.

Our servers already have a set of rules to protect against attacks on wp-login.php, but the rules were not quite sufficient to block tonight’s attack. We added a new rule to match tonight’s attack, and it fixed the problem.

We apologize for the time that the “web04” server returned 503 errors. As you can see by reviewing our blog posts we try to be very proactive to protect our customers’ WordPress sites, and hope that the new security rule will prevent future attacks with the same characteristics.