Upgrade MySQL in production and staging (Carrenza only)
This process is only applicable to Carrenza environments.
Upgrading MySQL will cause 500 errors on sites as the master and slave machines are restarted, so this procedure needs to be done at a suitable time on production - i.e. outside of core business hours (9:00 - 18:00 Monday to Friday).
It is safe to reboot the
_slave-2 machines on production
as they are not actively used by sites and will not cause 500 errors.
The following Puppet classes will need upgrading in
To upgrade the version of MySQL, use this fabric command from the
fab $environment -H <hostname> sdo:"apt-get install mysql-server-5.5"
Complete the upgrade for the backup machines first, followed by second slave, first slave, and finally the master.
Before upgrading the master, stop the slow query logs on the slave machines. Note: only do this on staging – slow query logs are not enabled in production.
Stopping the slow query logs can be done with the following Fabric command (or a similar one replacing the classes for Whitehall):
fab $environment class:mysql_backup class:mysql_slave mysql.stop_slow_query_log
The upgrade task can then be run on the master. Stopping the slow query log should allow replication to continue - if alerts occur about replication not running, then run the following Fabric task:
fab $environment class:mysql_backup class:mysql_slave mysql.fix_replication_from_slow_query_log_after_upgrade
Once the master has safely upgraded and replication is running, then turn the slow query log back on. Note: only do this on staging - slow query logs are not enabled in production:
fab $environment class:mysql_backup class:mysql_slave mysql.start_slow_query_log