Skip to main content
Table of contents

Emails

Load Test Email Alert API

You may wish to load test email-alert-api to get a realistic idea of how the system performs under high load, or if your’re making changes to how emails are processed to ensure your changes have the desired effect.

Before you begin, you should post in the #2nd-line channel that you’re about to do some load testing and that as part of that you’ll be disabling puppet runs on Staging for email-alert-api.

Rake tasks for load testing can be found here.

Testing without Notify

email-alert-api has a DelayProvider to simulate the delay of actual requests to Notify, so we can focus on internal load testing, without Notify being a factor. You will need to:

  • Deploy this PR or similar to the Staging environment.

  • Run Puppet on each of the machines. fab -P staging_aws class:email_alert_api puppet

  • Disable Puppet to persist the temporary change. fab -P staging_aws class:email_alert_api 'puppet.disable:"<date> load testing"'

  • Manually restart the workers to pick up the change. fab -P staging_aws class:email_alert_api app.restart:email-alert-api-procfile-worker

Testing with Notify

Let the Notify team know if you’re planning on doing a large scale test (anything more than a few thousand emails) and they can scale it up as necessary.

Normally emails from the Staging environment are severely rate limited. You can repoint the NotifyProvider to a Staging version of Notify, which is more realistic. You will need to:

  • Update the Puppet Deploy job to use a branch of govuk-secrets with the required API key. The API key is stored in the infra pass store in govuk-secrets under govuk-notify/test-api-key.

  • Deploy this PR or similar to the Staging environment.

  • Run Puppet on each of the machines. fab -P staging_aws class:email_alert_api puppet

  • Disable Puppet to persist the temporary change. fab -P staging_aws class:email_alert_api 'puppet.disable:"<date> load testing"'

  • Manually restart the workers to pick up the change. fab -P staging_aws class:email_alert_api app.restart:email-alert-api-procfile-worker

After the test

Once you’ve finished the test and the queues have been cleared, re-enable puppet runs, run puppet in order to overwrite the environment variables you’ve set and restart the workers again.

You can manually clear a large backlog of content changes by running the clear_emails rake task, and then wiping all active jobs with fab -P class:email_alert_api sdo:"pkill -9 --full sidekiq".

Let 2nd line and Notify know you’ve finished, and if you were using Notify, let them know too.

This page was last reviewed on 10 June 2020. It needs to be reviewed again on 10 June 2021 by the page owner #govuk-platform-health .
This page was set to be reviewed before 10 June 2021 by the page owner #govuk-platform-health. This might mean the content is out of date.