High load on web04 server May 9 2013 (resolved)

Posted May 9th, 2013 in System Status. Tags: .

The “web04” server experienced extremely high load for several minutes beginning at 8:00 AM Pacific time on May 9. Sites on this server were slow or unavailable as a result.

This was caused by a single site making “runaway” database queries that left almost no MySQL “cache” memory available for other queries. The problem has been resolved by suspending the site involved, and we are analyzing how to prevent anything similar from happening in the future.

We apologize for this incident; we take reliability seriously and strive to avoid this kind of problem.

Followup: We have made a technical change that will prevent this from recurring. Our MySQL servers are configured to write temporary tables to /dev/shm, which defaults to 12 GB in size on our 24 GB RAM servers. This effectively allowed runaway queries to use up to 12 GB of RAM, emptying much of server’s general file cache. We have lowered the size of /dev/shm to a maximum of 6 GB, ensuring that the file cache doesn’t empty out and cause load spikes.

  1. Recent Posts

    1. Thanksgiving 2024 Holiday Hours
    2. PHP versions 8.1.31, 8.2.26, and 8.3.14
    3. PHP versions 8.1.30, 8.2.24, and 8.3.12
    4. MariaDB (MySQL) database server update from 10.5 to 10.11 (completed)
    5. Labor Day 2024 holiday hours
    6. PHP versions 8.2.22 and 8.3.10
    7. WordPress 6.6
    8. PHP versions 8.1.29, 8.2.21, and 8.3.9
    9. 4th of July 2024 holiday hours
    10. Memorial Day 2024 holiday hours

  2. Categories

  3. Tags

    all servers email holiday hours linux mailman maintenance mod_security mysql network php phpmyadmin security server updates spam ssl status updates webmail wordpress zapp

  4. Feeds