Prometheus, grafana and alertmanager
How we use Prometheus on GOV.UK
Prometheus is a systems monitoring and alerting toolkit. Its features include:
- A time-series database, specifically optimised for graphing data over time.
- A DSL for querying the data, called PromQL.
- An HTTP pull model, consisting of Rack middleware that configures Prometheus to scrape data from an HTTP endpoint. Many GOV.UK applications leverage this functionality via the govuk-app-config gem: For example Finder Frontend’s monitoring of search metrics.
- A push gateway model, which can be used when an individual job (eg cronjob) creates data that requires graphing or monitoring: Example.
It is possible to query the Prometheus database via its UI, with PromQL but some people find it easier to use Grafana.
Grafana
Grafana provides a frontend for visualizing the Prometheus data. A common workflow when exploring the logs and or creating a new dashboard would be:
- Visit https://grafana.eks.production.govuk.digital/dashboards
- Create a new dashboard, and save it in the Experiments folder
- Click “Add visualization”, and select “Prometheus” as the datasoure
- Enter your PromQL query into the Metrics browser field (See Grafana tips) for pointers.
- If you would like to create a permanent dashboard, select “Export” and then “Export as JSON”.
- Copy the output, and open a PR in govuk-helm-charts. Prior art
Grafana tips
- When creating a new visualisation, select “Builder” instead of “Code” which provides a nicer UI with drop down menus so you can see, for example, the names of all the labels that are available.
- The label filter “job” refers to the name of the application.
- The “Explain” toggle is very helpful, it gives recommendations and syntax.
- If you open up an existing dashboard and find a panel that is similar to your use case, you can obtain the PromQL expression to use as a starting point by selecting the three dots in the top right hand corner, and then “Explore”.
Alertmanager
Alertmanager reads data directly data from Prometheus, and sends notifications when metrics cross predefined thresholds. These thresholds are defined as rules in the monitoring-config folder of govuk-helm-charts.
It is also possible to see all of the configured alerts via the Prometheus UI itself, by clicking the Alerts tab.
Somewhat confusingly, the Alertmanager UI does not show all configured alerts, only those which are currently firing.