Skip to main content
Table of contents
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert

Icinga alerts

Grafana Dashboards

This alert means there are dashboards that exist only in the database of a single Grafana instance.

Sometimes a temporary dashboard is necessary to deal with an incident. In this case, you can acknowledge the alert temporarily until the incident concludes and we’re confident we won’t need it again.

First you need to find out who ‘owns’ the dashboard. The best place to do this is Slack. Once you’ve found an owner, you can point them to this documentation and acknowledge the alert temporarily.

Keeping dashboards temporarily

If the dashboard was temporary or experimental, then you should delete it to resolve the alert. Having too many temporary dashboards makes it harder to find the ones we need.

You can export your dashboard to avoid deleting it completely. This means you can import it again in the future and continue editing, ideally in a non-production environment.

Keeping dashboards permanently

Dashboards that have longterm value should be moved into version control. We don’t store backups of the Grafana database, so we risk losing them if the Grafana instance is destroyed.

Before adding a new dashboard, think about whether you could improve an existing one.

We use Puppet to provision dashboards consistently across all environments. Export your dashboard and make a PR in govuk-puppet to add it alongside the existing ones.

Use the 'View JSON’ method to export your dashboard. Exporting the dashboard as a JSON file introduces placeholders that are not compatible with Puppet provisioning it automatically.

This page was last reviewed on 6 December 2019. It needs to be reviewed again on 6 June 2020 by the page owner #govuk-developers .
This page was set to be reviewed before 6 June 2020 by the page owner #govuk-developers. This might mean the content is out of date.