Skip to main content
Table of contents

Databases

Upgrade MySQL in production and staging (Carrenza only)

Note

This process is only applicable to Carrenza environments.

Upgrading MySQL will cause 500 errors on sites as the master and slave machines are restarted, so this procedure needs to be done at a suitable time on production - i.e. outside of core business hours (9:00 - 18:00 Monday to Friday).

It is safe to reboot the _backup and _slave-2 machines on production as they are not actively used by sites and will not cause 500 errors.

The following Puppet classes will need upgrading in govuk-puppet:

MySQL cluster:

mysql_backup

mysql_slave

mysql_master

Whitehall cluster:

whitehall_mysql_backup

whitehall_mysql_slave

whitehall_mysql_master

To upgrade the version of MySQL, use this fabric command from the fabric-scripts repo:

fab $environment -H <hostname> sdo:"apt-get install mysql-server-5.5"

Complete the upgrade for the backup machines first, followed by second slave, first slave, and finally the master.

Before upgrading the master, stop the slow query logs on the slave machines. Note: only do this on staging – slow query logs are not enabled in production.

Stopping the slow query logs can be done with the following Fabric command (or a similar one replacing the classes for Whitehall):

fab $environment class:mysql_backup class:mysql_slave mysql.stop_slow_query_log

The upgrade task can then be run on the master. Stopping the slow query log should allow replication to continue - if alerts occur about replication not running, then run the following Fabric task:

fab $environment class:mysql_backup class:mysql_slave mysql.fix_replication_from_slow_query_log_after_upgrade

Once the master has safely upgraded and replication is running, then turn the slow query log back on. Note: only do this on staging - slow query logs are not enabled in production:

fab $environment class:mysql_backup class:mysql_slave mysql.start_slow_query_log

This page was last reviewed on 2 March 2020. It needs to be reviewed again on 2 September 2020 by the page owner #re-govuk .
This page was set to be reviewed before 2 September 2020 by the page owner #re-govuk. This might mean the content is out of date.