In recent days, Yahoo and AOL made technical changes that prevented their own users from sending messages to most mailing lists. (That’s a simplified summary; your favorite search engine will show the gory details if you search for “Yahoo DMARC mailing lists”.)
If you use the Mailman software to run discussion lists, like many of our customers, this is a problem. To fix it, the authors of the Mailman software are making a new version to work around the Yahoo and AOL change.
That new version hasn’t been officially released yet, but we think this is so important that we’ve upgraded our installed copy of Mailman to a “pre-release” version of the fixed software.
Yahoo and AOL users should now be able to send to our customers’ Mailman lists without trouble.
Read the rest of this entry »
Over the last couple of days, mail from yahoo.com addresses has been “bouncing” a lot more than usual, often with error messages about “policy” or “DMARC” problems. This is especially true for messages involving mailing lists or forwarding from other addresses.
If you see this happening, it isn’t because of anything wrong on your end, or our end. It’s happening all over the Internet.
Read the rest of this entry »
At approximately 10:00 PM Pacific time Saturday April 5, 2014, the “web07” server will be restarted. This will cause a short interruption of service for Web sites on that server lasting less than 8 minutes.
Other servers will not be affected. Mail for customers on this server will be queued and delivered after a short delay.
Read the rest of this entry »
Between 10:41 and 10:49 AM Pacific time today (March 25, 2014), some MySQL database queries on the web12 server ran very slowly and caused “timeouts” for scripts. (No other servers were affected.)
The problem is fixed, and we’ve added an automated check to prevent it from recurring.
Read the rest of this entry »
Update: The maintenance described below was completed with less than 10 minutes downtime per server.
Between 10:00 PM and 11:59 PM Pacific time on Saturday March 1 2014, the “web05“, “web07” and “web08” servers will be restarted. This will cause a short interruption of service on each of these three servers, lasting less than 10 minutes, at some point during this two hour period.
Other servers will not be affected. Incoming mail for customers on these servers will not be lost, but will be queued and delivered after the maintenance.
Read the rest of this entry »
At approximately 2:45 PM Pacific time February 6, 2014, the Apache Web server running on the web13 server hung and needed to be manually restarted. We fixed the problem at 2:56 PM.
Obviously this sort of problem should not happen. We are monitoring it closely, and investigating it in more detail to try to find the root cause, so that we can take corrective action as necessary to prevent it from happening again. We apologize for the inconvenience caused by this problem.
Over the next ten days (February 7-15, 2014), we’ll be upgrading the MySQL database on all of our servers from version 5.1.72 to 5.5.35. The upgrade requires that each Web server be taken offline for two minutes, causing brief scheduled “downtime”.
Read the rest of this entry »
At approximately 9:30 PM Pacific time, all of our servers began to experience a large “distributed denial of service” (DDoS) attack via attempts to login to blogs using the standard WordPress wp-login.php script. This attack was very broad: it attacked thousands of sites across all of our servers, and it came from a huge number of IP addresses.
Processing these requests caused the overall load on all servers to increase. On “web04” the increase was enough to cause the server to start returning “503” errors for Web page requests.
Our servers already have a set of rules to protect against attacks on wp-login.php, but the rules were not quite sufficient to block tonight’s attack. We added a new rule to match tonight’s attack, and it fixed the problem.
We apologize for the time that the “web04” server returned 503 errors. As you can see by reviewing our blog posts we try to be very proactive to protect our customers’ WordPress sites, and hope that the new security rule will prevent future attacks with the same characteristics.
Update 5:00 PM December 27: AOL has resolved the problem described below. All delayed mail has been delivered, and all services are operating normally.
Read the rest of this entry »
Between 11:32 and 11:36 AM Pacific time on November 11, 2013, the “web04” server experienced very high load that led to an outage for sites on that server. (Other servers were not affected.)
The high load was caused by a RAID array hard disk that suddenly became very slow to respond to requests. The disk was automatically reset and the server resumed working normally. We’re keeping a close eye on this and doing further testing, and will replace any defective hardware if necessary.
We sincerely apologize to our customers affected by this problem.
