Skip to main content
Last updated: 25 Jun 2021

Run an A/B or multivariate test

1. Preparation

Choose your test name

The name should begin with ABTest-. Try to keep it short.

You don’t need to use the ABTest- prefix in your code though as it’s already configured in Fastly.

Consider load

You will need to take steps to ensure that GOV.UK can cope with the extra load on origin generated by an AB test. Extra load occurs because AB tests add Vary headers to responses. If you add AB testing headers to all content, then requests for any content will tend to miss the cache more often. This adds load to origin servers.

You can minimise the impact of this by configuring your test to only return AB testing headers on pages that you are measuring. Example testing navigation on mainstream guides.

You may want to deploy your test at a quieter time of day (such as 9:30am) rather than the middle of the day. This allows CDN caches to warm up in a slightly more graceful way. Check when a quiet time for your test case is.

Make sure you monitor your test for some time after deployment to ensure that everything is working OK.

Decide how the variants should be split

Bear in mind that users who have not opted in to analytics cookies will not be assigned to a variant by Fastly and will not have a cookie set.

A higher percentage on your B variant will reduce the time that you need to run the test. Your performance analyst can help here.

You may want to start with a small B percentage and ramp up gradually, either for load reasons, or if there’s something a bit controversial in your test that may perform particularly poorly with users.

The cookie expiry time should be short until you have established traffic across the desired split between your variants.

Once your variants are at the desired proportions, the cookie expiry time should be longer than the expected test duration. This ensures that users are kept in the same variant for the duration of the test which allows them to get used to whatever changes you have made.

It’s good to have a margin for error, to (for example) correct faulty tracking at the start, or perhaps run the test for a few more days at the end if you need more data.

2. How to set up an A/B test

  1. Add your test to the A/B test register.
  2. If you want to use Google Analytics to monitor the A/B test, talk to a performance analyst and pick a GA dimension to use for your test.
  3. Create dictionary and A/B test files in the govuk-cdn-config repo. See an example for the dictionaries and an example for the A/B configuration (these used to be in a different repo). For more details, see the dictionaries README.
  4. Deploy the dictionary changes to each environment using the Update_CDN_Dictionaries Jenkins job. The dictionaries must be deployed to the www vhost.
  5. Deploy the Fastly configuration to each environment using the Deploy_CDN Jenkins job. Use the same parameters as in step 4. You can test it on staging by visiting Changes should appear almost immediately - there is no caching of the CDN config.
  6. Use the govuk_ab_testing gem to serve different versions to your users. It can be configured with the analytics dimension selected in step 2.
  7. To activate or deactivate the test, or to change the B percentage, update your test in the govuk-cdn-config repo and deploy the dictionaries.

If you’re making a change to Search API, you may also want to test using the search performance explorer. In which case, you’ll need to add your AB test to the application, and manually deploy it to Heroku.

3. How to remove your A/B test

Follow these steps:

  1. Remove your test from the ab_tests configuration file in the govuk-cdn-config repo and remove the dictionary files.
  2. Deploy the Fastly configuration to each environment using the Deploy_CDN Jenkins job. The vhost must be set to www, and the credentials are in the govuk-secrets repo.
  3. Deploy the govuk-cdn-config changes to each environment using the Update_CDN_Dictionaries Jenkins job. Use the same parameters as in step 2.
  4. Mark the end date in A/B test register.