Run an A/B or multivariate test
Before you run an A/B or multivariate test, you should read the guidance on how A/B testing works.
1. Prepare the A/B test
Choose a test name
Choose a short test name, and prefix that test name with ABTest-
. For example, ABTest-dvlacheck
You do not need to use the ABTest-
prefix in your code because this prefix is already configured in Fastly.
Consider the load on GOV.UK
You must make sure that GOV.UK can handle the extra load
on origin generated by an A/B test. Extra load occurs because A/B tests add Vary
headers
to responses. If you add A/B testing headers to all content, then requests for any
content will tend to miss the cache more often. This adds load to origin servers.
You can minimise the impact of this by configuring your test to only return A/B testing headers on pages that you are measuring. See this example testing navigation on mainstream guides for more information.
You may want to deploy your test at a quieter time of day. Check when a quiet time for your test case is.
Make sure you monitor your test after deployment.
2. Set up an A/B test
- Write the A/B test. Use the information in the govuk_ab_testing gem to understand how to serve different versions to your users.
- Add your test to the A/B test register.
- If you want to use Google Analytics to monitor the A/B test, talk to a performance analyst and pick a GA dimension to use for your test.
- Create dictionary and A/B test files in the govuk-cdn-config repo. See an example for the dictionaries and an example for the A/B configuration. For more information, see the dictionaries README.
3. Deploy and activate an A/B test
To deploy and activate an A/B test, you must set up a personal API token in your Fastly account.
- Deploy the dictionary changes to each environment using the Update_CDN_Dictionaries Jenkins job. The API key is in the [govuk-secrets repo][govuk-secrets-fastly]. You must deploy the dictionaries to the
www
vhost. - Deploy the Fastly configuration to each environment using the Deploy_CDN Jenkins job. Use the same parameters as the previous step. You can test the deployment on staging by visiting https://www.staging.publishing.service.gov.uk. Changes should appear almost immediately as there is no caching of the CDN config.
- To activate or deactivate the test, or to change the B percentage, update your test in the govuk-cdn-config repo and deploy the dictionaries.
4. Remove the A/B test
- Remove your test from the ab_tests configuration file in the govuk-cdn-config repo and remove the dictionary files.
- Deploy the Fastly configuration to each environment using the Deploy_CDN Jenkins job. Set the
vhost
towww
. The credentials are in thegovuk-secrets
repo. - Deploy the
govuk-cdn-config
changes to each environment using the Update_CDN_Dictionaries Jenkins job. Use the same parameters as the previous step. - Mark the end date in the A/B test register.