Skip to main content
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert
Warning This document has not been updated for a while now. It may be out of date.
Last updated: 17 Sep 2020

Redis alerts

We have a few monitoring checks for Redis:

  • memory usage
  • number of connected clients

Redis is configured to use 75% of the system memory. If Redis memory usage reaches this limit it will stop accepting data input. Redis is also unable to write its RDB data dumps when it’s out of memory, so data loss is possible.

Since version 2.6, Redis dynamically sets the maximum allowed number of clients to 32 less than the number of file descriptors that can be opened at a time.

General Redis investigation tips

If Redis is using large amounts of memory you can use redis-cli --bigkeys to find out what those things might be. If any of the returned keys are new, try and speak to the team responsible.

If there is an associated graph, it may also be worth checking as the alert could be a temporary spike, for example: due to running a rake task.

Uniquejobs hash

We use a custom fork of a gem called sidekiq-unique-jobs in some of our applications. This gem creates a key in Redis called uniquejobs

We have had an incident where the uniquejobs key has grown to the point that it caused high memory alerts for Redis. You can delete this key safely by runnning the following command on the associated Redis box:

$ redis-cli DEL uniquejobs

We have noticed that when this key grows on the redis box used by the Publishing Api (currently redis-1), it can be accompanied by high Sidekiq queue latency for Whitehall’s queues. You can check the queue latency on the Grafana dashboard.

Redis in AWS

In AWS we are using a hosted Redis cluster blue-backend-redis , provided by AWS ElastiCache. Monitoring information on the nodes can be obtained via the AWS Console.

In order to access the Redis command line interface in AWS, it is required to log in to the db-admin machine via the primary endpoint url of the cluster.

redis-cli -h <primary_endpoint url>