Travel Advice or Drug and Medical Device email alerts not sent
GOV.UK’s email-alert-api sends two kinds of time-sensitive email notifications:
- travel advice alerts
- medical safety alerts, for example product recalls of medical devices
These email alerts are supposed to be delivered to subscribers within a few hours.
email-alert-api-worker monitors deliverability by comparing records of what should have been sent against callbacks from GOV.UK Notify, which is the system that sends the bulk email to subscribers.
Email alerts have nothing to do with Prometheus/Alertmanager/PagerDuty alerts. They’re just ambiguously named.
Find out which mailshot was affected
Search for recent WARN-level logs from email-alert-api-worker. Workers log a
Couldn't find any delivered emails for ... warning periodically when there
are potentially undelivered medical/travel alerts.
The log message should contain the Content ID and path of the affected mailshot. Visit the path on GOV.UK to find the title of the alert.
Verify whether the email was received
The email-alert-api-alert-listener mailing list is subscribed to all travel advice and medical safety alerts. Check whether the message appears there.
Troubleshooting
Check whether an email matching the details from the logs was delivered to the email-alert-api-alert-listener mailing list.
Check Notify’s service status.
Using the Content ID from the logs, use the email statistics rake task to determine whether the app failed to send messages to Notify, or whether Notify failed to deliver them.
Check the size of email-alert-api’s work queue using the Sidekiq dashboard and email-alert-api dashboard. Is there a backlog? If so, is it diminishing? Are the workers making sufficient progress?
Check logs in Kibana and events in Sentry for errors or clues.
Previously, the introduction of a new content sub-type of medical safety alerts caused an outage. See “all” checkbox is misleading for finder_email_signups.
If all else fails, investigate the Email Alert API database to determine whether the content change was received and what state it is in.
Resend a medical safety alert
Find the content ID of the affected content item. In Specialist Publisher, go to Medical Alerts and choose the affected document. The content ID is the UUID in the URL. For example, in
https://specialist-publisher.publishing.service.gov.uk/medical-safety-alerts/de8a10f1-eac7-49a8-a8ef-1769a6f6cef9:enthe content ID isde8a10f1-eac7-49a8-a8ef-1769a6f6cef9.Run the
republish:one[<CONTENT_ID>]Rake task in specialist-publisher, replacing<CONTENT_ID>with the one from the previous step.
Resend a travel advice alert
Find the edition ID of the affected content item. In Travel Advice Publisher, go to the URL of the country’s Edit page. The edition ID is a 24-digit hex number, for example
fedc13e231ccd7d63e1abf65.Run the
email_alerts:trigger[<EDITION_ID>]Rake task in travel-advice-publisher, replacing<EDITION_ID>with the one from the previous step.