Table of contents
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert
This page was set to be reviewed before 2018-08-13 by the page owner: #govuk-2ndline. This might mean the content is out of date. Read how to review a page.

PagerDuty drill

Every week we test PagerDuty to make sure it can phone to alert us to any issues.

Cron creates a file on disk. Icinga raises a critical alert when that file exists with an Icinga contact group set so that PagerDuty is notified. After a short amount of time, cron will remove the file to resolve the alert. The code that does this is in Puppet.

You don’t need to take any action for this alert. The primary in-office 2nd line should escalate the call to the secondary who should escalate it to “escalations” to ensure that phone redirection is working. The person on “escalations” will resolve the PagerDuty alert to prevent anyone else being phoned.

More about Icinga alerts

This page was set to be reviewed before 2018-08-13 by the page owner: #govuk-2ndline. This might mean the content is out of date. Read how to review a page.