There was a brief but severe performance problem on the web12 server today between 9:59 and 10:07 AM Pacific time. During this time, many Web server requests were very slow to load or even “timed out” completely. All services are now operating normally again. Other servers were not affected.
Between 11:00 PM and 11:59 PM Pacific time on Saturday December 22 2012, the MySQL database software on each of our servers will be upgraded to version 5.1.66 and restarted. This will cause an approximately 30 second interruption of service on each customer Web site at some point during this hour.
This upgrade is necessary for security reasons. We apologize for the inconvenience this causes.
Update December 22 11:17 PM: The maintenance was completed with less than 30 seconds downtime per server.
Between 11:00 PM and 11:59 PM Pacific time on Saturday June 23 2012, the MySQL database software on each of our servers will be upgraded to version 5.1.63 and restarted. This will cause an approximately 30 second interruption of service on each customer Web site at some point during this hour.
This upgrade is necessary for security reasons. We apologize for the inconvenience this causes.
Update 11:12 PM June 23: The maintenance was completed as planned.
Between 10:00 PM and 11:00 PM Pacific time on Friday March 9, 2012, we’ll be updating the MySQL database software on all our hosting servers. This will cause a Web site service interruption of about 30 seconds for some customers at some time during this period. E-mail will not be affected.
This maintenance is necessary to install a mandatory MySQL security update that will upgrade the MySQL version to 5.1.61. We apologize for any inconvenience this causes.
Update 10:13 PM: The maintenance was completed with less than 30 seconds downtime on each server. Customers should not notice any changes, but as always, don’t hesitate to contact us with any questions or problems.
We had a couple of instances of MySQL queries overloading the bender server today. The first one happened at about 3:41 AM (Pacific time) and the second one happened at about 7:48 AM. Each occurrence lasted about 20 minutes. The problem each time was that a database was running extremely inefficient queries. Each time we fixed the problem by creating indexes so that the queries could then run in a fraction of the time previously required.
We apologize for any inconvenience caused by this problem. Visitors to your Web site (on the bender server) might have seen reduced performance (or, in rare cases, 503 errors). E-mail was not affected. We don’t consider this type of problem to be acceptable. These problems should not recur since the indexes have been created.
If you use a MySQL database with large tables, it’s possible to accidentally run queries that try to sort millions of rows (usually through some kind of programming error, such as an “unconstrained table join”).
Those runaway queries can slow down the MySQL server for many minutes on end, causing performance problems.
To prevent the worst of that, we’ve set the max_join_size setting to 1,000,000 on our MySQL servers.
Many Web sites write data to a database. Usually, the data absolutely must be properly saved, so the default way of adding records (using an SQL “INSERT” statement) ensures that the data is permanently stored on the server’s disks. Doing that takes a relatively long time in computer terms — it’s much slower than most things computers do.
In some cases, you might be storing data that’s not quite so important. And if it means your application can run much faster, you might be willing to risk a very small chance of data loss. That’s where MySQL’s “INSERT DELAYED” statement, which works with MyISAM table types (but not InnoDB tables), can be useful. (Tables are created as type MyISAM by default, so most tables are eligible to benefit from this tip.)
We’ve installed MySQL and PHP 5 security updates. Customers should not notice any changes; the updates just fix several security issues in PHP 5 and MySQL.
The updates were performed in such a way that new Web server connections were delayed during the 30 seconds or so that PHP and MySQL were unavailable on each server. That should mean that as far as scripts on your Web site were concerned, there was zero downtime.
Here’s an obscure thing we spent some time on today that doesn’t affect our customers, but might be useful to someone else searching the Internet. The problem was a MySQL database table that, as far as we can tell, worked perfectly… except that it shows this message when we run CHECK TABLE:
Table upgrade required. Please do "REPAIR TABLE `users`" to fix it!
We applied a MySQL security update tonight. The version number remains 5.0.32, and customers should not notice any changes; the update just fixes several security issues.
The update was performed in such a way that new Web server connections were delayed during the 30 seconds or so that MySQL was unavailable on each server. That should mean that as far as scripts on your Web site were concerned, there was zero MySQL downtime.
