Skip to main content
  1. Home
  2. A/B testing
Warning This document has not been updated for a while now. It may be out of date.
Last updated: 28 Apr 2022

Run an A/B or multivariate test

Before you run an A/B or multivariate test, you should read the guidance on how A/B testing works.

1. Prepare the A/B test

Choose a test name

Choose a short test name, and prefix that test name with ABTest-. For example, ABTest-dvlacheck

You do not need to use the ABTest- prefix in your code because this prefix is already configured in Fastly.

Consider the load on GOV.UK

You must make sure that GOV.UK can handle the extra load on origin generated by an A/B test. Extra load occurs because A/B tests add Vary headers to responses. If you add A/B testing headers to all content, then requests for any content will tend to miss the cache more often. This adds load to origin servers.

You can minimise the impact of this by configuring your test to only return A/B testing headers on pages that you are measuring. See this example testing navigation on mainstream guides for more information.

You may want to deploy your test at a quieter time of day. Check when a quiet time for your test case is.

Make sure you monitor your test after deployment.

2. Set up an A/B test

  1. Write the A/B test. Use the information in the govuk_ab_testing gem to understand how to serve different versions to your users.
  2. Add your test to the A/B test register.
  3. If you want to use Google Analytics to monitor the A/B test, talk to a performance analyst and pick a GA dimension to use for your test.
  4. Create dictionary and A/B test files in the govuk-cdn-config repo. See an example for the dictionaries and an example for the A/B configuration. For more information, see the dictionaries README.

3. Deploy and activate an A/B test

To deploy and activate an A/B test, you must set up a personal API token in your Fastly account.

  1. Deploy the dictionary changes to each environment using the Update_CDN_Dictionaries Jenkins job. The API key is in the [govuk-secrets repo][govuk-secrets-fastly]. You must deploy the dictionaries to the www vhost.
  2. Deploy the Fastly configuration to each environment using the Deploy_CDN Jenkins job. Use the same parameters as the previous step. You can test the deployment on staging by visiting https://www.staging.publishing.service.gov.uk. Changes should appear almost immediately as there is no caching of the CDN config.
  3. To activate or deactivate the test, or to change the B percentage, update your test in the govuk-cdn-config repo and deploy the dictionaries.

4. Remove the A/B test

  1. Remove your test from the ab_tests configuration file in the govuk-cdn-config repo and remove the dictionary files.
  2. Deploy the Fastly configuration to each environment using the Deploy_CDN Jenkins job. Set the vhost to www. The credentials are in the govuk-secrets repo.
  3. Deploy the govuk-cdn-config changes to each environment using the Update_CDN_Dictionaries Jenkins job. Use the same parameters as the previous step.
  4. Mark the end date in the A/B test register.

More in the A/B testing section

Learn