Table of contents
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert

RabbitMQ: Consumers not processing messages in queue

Read more about how we use RabbitMQ

For some named RabitMQ queues, we run a check that messages are being consumed. This is currently only the case for the email_alert_service and email_unpublishing queues. The queue name should indicate the app responsible for consuming the queue.

This is different to the check that there are consumers. This alert catches the case where a consumer is connected to the queue but failing to process messages in a timely fashion.

The check is performed by connecting to RabbitMQ’s admin API, so the information given is from Rabbit’s point of view. It looks at the number of messages still to be delivered.

If the check succeeds, it will return the number of unprocessed messages in the queue.

If the check fails due to a build up of messages, it will report how many messages there are, and what the threshold is.

Consequences of message build-up

If messages are building up, it may indicate that the service which consumes them has hit a problem, or there is an unusually high amount of activity. For example, email alerts about updated content will not be sent if the email_alert_sevice queue isn’t being processed.

Unless there is a wider RabbitMQ failure, messages will not be lost - they will be processed once the problem is resolved.

Investigation

The same approach as “RabbitMQ: No consumers listening to queue”.

More about Icinga alerts

This page was last reviewed . It needs to be reviewed again by the page owner #govuk-2ndline.