Skip to main content
Table of contents
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert

Icinga alerts

Email Alert API: Unprocessed messages

Messages are similar to content changes, introduced in September 2019 to support the GOV.UK Get ready for Brexit checker. It is intended to provide a means for applications to alert subscribers to ad hoc events that may not be represented by a content change. See the ADR for Messages for more information.

This alert is triggered when these messages are not processed within the time we would expect. This may be fine and the emails will eventually go out, but it’s worth some investigation.

  • warning - unprocessed messages created more than 5 minutes ago
  • critical - unprocessed messages created more than 10 minutes ago

See the ProcessMessageWorker for more information.

Useful queries

Check which messages this affects

Message.where("created_at < ?", 5.minutes.ago).where(processed_at: nil)

Check the count, then run the above query again to see if the count has decreased. If it’s decreasing, then it means that emails are going out and there’s probably a lot being processed. If it’s not decreasing the Sidekiq worker might be stuck, see the sidekiq section on how to view the Sidekiq queues.

Check number of subscription contents built for a message (you would expect this number to keep going up)

SubscriptionContent.where(message: message).count

Resend the emails for a message (ignore ones that have already gone out)

ProcessMessageWorker.new.perform(message.id)

Resend the emails for a message in bulk (ignore ones that have already gone out)

Message.where("created_at < ?", 10.minutes.ago).where(processed_at: nil).map { |message| ProcessMessageWorker.new.perform(message.id)  }

You can also check the Email Alert API Metrics dashboard to monitor if emails are going out and see the general troubleshooting tips section for more information.

This page was last reviewed on 7 January 2020. It needs to be reviewed again on 7 July 2020 by the page owner #govuk-2ndline .
This page was set to be reviewed before 7 July 2020 by the page owner #govuk-2ndline. This might mean the content is out of date.