Table of contents
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert

MySQL Xtrabackups to S3

We send backups to an Amazon S3 bucket every 15 minutes. This is performed using a nightly “base” backup and “incremental” backups which run every 15 minutes, which are linked to the base backup.

Further detail on the implementation can be found here

MySQL Xtrabackup daily base push

If this alert triggers it means the nightly base backup has failed to run. Check the logs in Kibana using this query:

@source_host:mysql-backup* AND @fields.syslog_program:xtrabackup_s3_base

As this runs on a slave, it is safe to rerun at anytime. Log into the machine and run:

sudo -i /usr/bin/timeout 1h /usr/bin/setlock -N /var/run/mysql_xtrabackup /usr/local/bin/xtrabackup_s3_base

If it errors you should be able to find out what the issue is.

MySQL Xtrabackup incremental-push

When this alert triggers it means the incremental backups, which run every 15 minutes, have had a problem. Check Kibana with the following query:

@source_host:mysql-backup* AND @fields.syslog_program:xtrabackup_s3_incremental

It is safe to run manually, though this will by nature run regularly so you should be able to view what the problem is in the logs. If the incremental backup is failing, it may be worth rerunning the base backup job, and then running the incremental job. Other possible problems could point to connectivity to Amazon S3.

This page is owned by #2ndline and needs to be reviewed