Skip to main content
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert
Warning This document has not been updated for a while now. It may be out of date.
Last updated: 17 Aug 2020

Grafana Dashboards

This alert means there are dashboards that exist only in the database of a single Grafana instance.

Sometimes a temporary dashboard is necessary to deal with an incident. In this case, you can acknowledge the alert temporarily until the incident concludes and we’re confident we won’t need it again.

First you need to find out who ‘owns’ the dashboard. The best place to do this is Slack. Once you’ve found an owner, you can point them to this documentation and acknowledge the alert temporarily.

Keeping dashboards temporarily

Having too many temporary dashboards makes it harder to find the ones we need. If the dashboard was temporary or experimental, you should include the word TEMPORARY in the title.

Alternatively, you can export your dashboard and delete the original. This means you can import it again in the future and continue editing, ideally in a non-production environment.

Keeping dashboards permanently

Dashboards that have longterm value should be moved into version control. We don’t store backups of the Grafana database, so we risk losing them if the Grafana instance is destroyed.

Before adding a new dashboard, think about whether you could improve an existing one.

We use Puppet to provision dashboards consistently across all environments. Export your dashboard and make a PR in govuk-puppet to add it alongside the existing ones.

Use the ‘View JSON’ method to export your dashboard. Exporting the dashboard as a JSON file introduces placeholders that are not compatible with Puppet provisioning it automatically.