Table of contents
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert

Redis alerts

We have a few monitoring checks for Redis:

  • memory usage
  • number of connected clients

Redis is configured to use 75% of the system memory. If Redis memory usage reaches this limit it will stop accepting data input. Redis is also unable to write its RDB data dumps when it’s out of memory, so data loss is possible.

Since version 2.6, Redis dynamically sets the maximum allowed number of clients to 32 less than the number of file descriptors that can be opened at a time.

General Redis investigation tips

If Redis is using large amounts of memory you can use redis-cli --bigkeys to find out what those things might be. If any of the returned keys are new, try and speak to the team responsible.

If there is an associated graph, it may also be worth checking as the alert could be a temporary spike, for example: due to running a rake task.

Uniquejobs hash

We use a custom fork of a gem called sidekiq-unique-jobs in some of our applications. This gem creates a key in Redis called uniquejobs

We have had an incident where the uniquejobs key has grown to the point that it caused high memory alerts for Redis. You can delete this key safely by runnning the following command on the associated Redis box:

$ redis-cli DEL uniquejobs

We have noticed that when this key grows on the redis box used by the Publishing Api (currently redis-1), it can be accompanied by high Sidekiq queue latency for Whitehall’s queues. You can check the queue latency on the Grafana dashboard.

More about Icinga alerts

This page was last reviewed . It needs to be reviewed again by the page owner #govuk-2ndline.