Table of contents

A/B testing

Run an A/B or multivariate test

1. Preparation

Choose your test name

The name should begin with ABTest-. Try to keep it short.

You don’t need to use the ABTest- prefix in your code though as it’s already configured in Fastly.

Decide how the variants should be split

A higher percentage on your B variant will reduce the time that you need to run the test. Your performance analyst can help here.

You may want to start with a small percentage and ramp up gradually though, particularly if there’s something a bit controversial in your test.

The cookie expiry time should be short until you have established traffic across the desired split between your variants.

Once your variants are at the desired proportions, the cookie expiry time should be longer than the expected test duration. This ensures that users are kept in the same variant for the duration of the test which allows them to get used to whatever changes you have made.

It’s good to have a margin for error, to (for example) correct faulty tracking at the start, or perhaps run the test for a few more days at the end if you need more data.

2. How to set up an A/B test

  1. Add your test to the A/B test register.
  2. If you want to use Google Analytics to monitor the A/B test, talk to a performance analyst and pick a GA dimension to use for your test.
  3. Create dictionary and A/B test files in the govuk-cdn-config repo. See an example for the dictionaries and an example for the A/B configuration (these used to be in a different repo). For more details, see the dictionaries README.
  4. Deploy the dictionary changes to each environment using the Update_CDN_Dictionaries Jenkins job. The vhost must be set to www, and the credentials are in the govuk-secrets repo.
  5. Deploy the Fastly configuration to each environment using the Deploy_CDN Jenkins job. Use the same parameters as in step 4. You can test it on staging by visiting https://www.staging.publishing.service.gov.uk. Changes should appear almost immediately - there is no caching of the CDN config.
  6. Use the govuk_ab_testing gem to serve different versions to your users. It can be configured with the analytics dimension selected in step 2.
  7. To activate or deactivate the test, or to change the B percentage, update your test in the govuk-cdn-config repo and deploy the dictionaries.

3. How to remove your A/B test

Follow these steps:

  1. Remove your test from the ab_tests configuration file in the govuk-cdn-config repo and remove the dictionary files.
  2. Deploy the Fastly configuration to each environment using the Deploy_CDN Jenkins job. The vhost must be set to www, and the credentials are in the govuk-secrets repo.
  3. Deploy the govuk-cdn-config changes to each environment using the Update_CDN_Dictionaries Jenkins job. Use the same parameters as in step 2.
  4. Mark the end date in A/B test register.
This page was last reviewed on 22 March 2019. It needs to be reviewed again on 22 September 2019 by the page owner #govuk-2ndline .
This page was set to be reviewed before 22 September 2019 by the page owner #govuk-2ndline. This might mean the content is out of date.