Skip to main content
  1. Home
  2. A/B testing
Last updated: 5 Jul 2024

Run an A/B or multivariate test

Before you run an A/B or multivariate test, you should read the guidance on how A/B testing works.

1. Prepare the A/B test

Choose a test name

Choose a short test name, and prefix that test name with ABTest-. For example, ABTest-dvlacheck

You do not need to use the ABTest- prefix in your code because this prefix is already configured in Fastly.

Consider the load on GOV.UK

You must make sure that GOV.UK can handle the extra load on origin generated by an A/B test. Extra load occurs because A/B tests add Vary headers to responses. If you add A/B testing headers to all content, then requests for any content will tend to miss the cache more often. This adds load to origin servers.

You can minimise the impact of this by configuring your test to only return A/B testing headers on pages that you are measuring. See this example testing navigation on mainstream guides for more information.

You may want to deploy your test at a quieter time of day. Check when a quiet time for your test case is.

Make sure you monitor your test after deployment.

2. Set up an A/B test

  1. Write the A/B test. Use the information in the govuk_ab_testing gem to understand how to serve different versions to your users.
  2. Add your test to the A/B test register.
  3. If you want to use Google Analytics to monitor the A/B test, talk to a performance analyst.
  4. Add your test to the ab_tests configuration file in the govuk-fastly repo and remove the relevant dictionary config. See an example pull request updating this config.

3. Deploy and activate an A/B test

  1. Deploy the dictionary changes to each environment using Terraform Cloud (Note that Terraform Cloud’s VCL diff display is poor. To see a better diff, check the fastly-vcl-diff post-plan task on your run page)
  2. To activate or deactivate the test, or to change the percentages, update your test in the govuk-fastly repo and deploy the terraform.

4. Remove the A/B test

  1. Remove your test from the ab_tests configuration file in the govuk-fastly repo and remove the relevant dictionary config.
  2. Deploy the Fastly configuration using Terraform Cloud (Note that Terraform Cloud’s VCL diff display is poor. To see a better diff, check the fastly-vcl-diff post-plan task on your run page)
  3. Mark the end date in the A/B test register.

More in the A/B testing section

Learn