This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert

Last updated: 14 Feb 2023

AmazonMQ: High number of unprocessed messages

For information about how we use RabbitMQ on AmazonMQ, see here.

We check that there is not a significant build up of messages compared to the normal amounts on certain queues. The queues this alert applies to are:

• email_alert_service
• email_unpublishing
• subscriber_list_details_update_major
• subscriber_list_details_update_minor
• cache_clearing_service-* (low, medium and high)

The plugin which implements this alert is here.

The Icinga check is performed by connecting to RabbitMQ’s admin API and triggering if the number of messages on the queues listed above crosses certain thresholds.

For the email_alert_service, email_unpublishing, subscriber_list_details_update_major and subscriber_list_details_update_minor queues the message thresholds are: 25 for critical and 5 for warning.

For cache_clearing_service-* queues the message thresholds are: 100,000 for critical and 80,000 for warning.

Note

You may see the no consumers listening alert too, as issues with consumers processing messages could then lead to a high backlog of messages.

Troubleshooting

For troubleshooting steps, see here.