Load Test Email Alert API
You may wish to load test email-alert-api to get a realistic idea of how the system performs under high load, or if your’re making changes to how emails are processed to ensure your changes have the desired effect.
You can load test email-alert-api with the staging version of Notify so that it won’t put unnecessary load on their production version. Let them know if you’re planning on doing a large scale test (anything more than a few thousand emails) and they can scale it up as necessary.
Setting up for the test
- Post on #2nd-line channel that you’re about to do some load testing and that as part of that you’ll be disabling puppet runs on staging for email-alert-api.
- Use a fabric script to disable puppet runs for email-alert-api so it won’t overwrite or remove the environment variables you need to set to run the test.
- If you’re running the test against Notify’s staging environment, you need to set a few environment variables. The API key is stored in the pass store in govuk-secrets under
|GOVUK_NOTIFY_BASE_URL||https://api.staging-notify.works/||We need to send requests to the correct endpoint|
|GOVUK_NOTIFY_API_KEY||Key from govuk-secrets||The production key will not be correct for staging|
|GOVUK_NOTIFY_TEMPLATE_ID||76d21ce7-54c3-4fb7-8830-ba3b79287985||Requests will fail if it cannot find the template, this exists only on staging|
- You need to set the following environment variable key value pairs whether or not you’re sending emails to Notify staging:
|EMAIL_ADDRESS_OVERRIDEemail@example.com||Sends all emails to Amazon SES, preventing it going to real users|
- Use fabric to restart the workers so they’ll pick up the new environment variables by running
fab aws_staging node_type:email_alert_api app.restart:email-alert-api-procfile-worker
- Now you’re ready to start, open a rails console on an email-alert-api machine on staging. How you perform the test may depend on what you want to test, in the past we’ve duplicated content changes from a few hours or a day.
During the test
You can keep an eye on the Grafana dashboard to see how it’s performing. If it’s a large test, a lot of logs can be produced in a short amount of time so use the machine dashboard filtered by email-alert-api machines to check disk space usage.
After the test
Once you’ve finished the test and the queues have been cleared, re-enable puppet runs, run puppet in order to overwrite the environment variables you’ve set, restart the workers again and let 2nd line and Notify know you’ve finished.