This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert

Warning This document has not been updated for a while now. It may be out of date.

Last updated: 8 Sep 2020

RabbitMQ: High number of unprocessed messages

For information about how we use RabbitMQ, see here.

We check that there is not a significant build up of messages compared to the normal amounts on certain queues. The queues this alert applies to are: email_alert_service, email_unpublishing and cache_clearing_service-* (low, medium and high). The plugin which implements this alert is here.

The Icinga check is performed by connecting to RabbitMQ’s admin API and triggering if the number of messages on the queues listed above crosses certain thresholds.

For email_alert_service and email_unpublishing queues the message thresholds are: 25 for critical and 5 for warning.

For cache_clearing_service-* queues the message thresholds are: 100,000 for critical and 80,000 for warning.

Note

You may see the no consumers listening alert too, as issues with consumers processing messages could then lead to a high backlog of messages.

Troubleshooting

For troubleshooting steps, see here.