Run an A/B or multivariate test
Before you run an A/B or multivariate test, you should read the guidance on how A/B testing works.
1. Prepare the A/B test
Choose a test name
Choose a short test name, and prefix that test name with
ABTest-. For example,
Consider the load on GOV.UK
You must make sure that GOV.UK can handle the extra load
on origin generated by an A/B test. Extra load occurs because A/B tests add
to responses. If you add A/B testing headers to all content, then requests for any
content will tend to miss the cache more often. This adds load to origin servers.
You can minimise the impact of this by configuring your test to only return A/B testing headers on pages that you are measuring. See this example testing navigation on mainstream guides for more information.
You may want to deploy your test at a quieter time of day. Check when a quiet time for your test case is.
Make sure you monitor your test after deployment.
2. Set up an A/B test
- Write the A/B test. Use the information in the govuk_ab_testing gem to understand how to serve different versions to your users.
- Add your test to the A/B test register.
- If you want to use Google Analytics to monitor the A/B test, talk to a performance analyst and pick a GA dimension to use for your test. If you wish to run an A/B test in integration or staging you may need to ensure that analytics tracking exists in those environments.
- Add your test to the ab_tests configuration file in the govuk-fastly repo and remove the relevant dictionary config. See an example pull request updating this config.
3. Deploy and activate an A/B test
- Deploy the dictionary changes to each environment using Terraform Cloud (Note that Terraform Cloud’s VCL diff display is poor. To see a better diff, check the fastly-vcl-diff post-plan task on your run page)
- To activate or deactivate the test, or to change the percentages, update your test in the govuk-fastly repo and deploy the terraform.
4. Remove the A/B test
- Remove your test from the ab_tests configuration file in the govuk-fastly repo and remove the relevant dictionary config.
- Deploy the Fastly configuration using Terraform Cloud (Note that Terraform Cloud’s VCL diff display is poor. To see a better diff, check the fastly-vcl-diff post-plan task on your run page)
- Mark the end date in the A/B test register.