Run an A/B or multivariate test
Choose your test name
The name should begin with
ABTest-. Try to keep it short.
You will need to take steps to ensure that GOV.UK can cope with the extra load
on origin generated by an AB test. Extra load occurs because AB tests add
to responses. If you add AB testing headers to all content, then requests for any
content will tend to miss the cache more often. This adds load to origin servers.
You can minimise the impact of this by configuring your test to only return AB testing headers on pages that you are measuring. Example testing navigation on mainstream guides.
You may want to deploy your test at a quieter time of day (such as 9:30am) rather than the middle of the day. This allows CDN caches to warm up in a slightly more graceful way. Check when a quiet time for your test case is.
Make sure you monitor your test for some time after deployment to ensure that everything is working OK.
Decide how the variants should be split
Bear in mind that users who have not opted in to analytics cookies will not be assigned to a variant by Fastly and will not have a cookie set.
A higher percentage on your B variant will reduce the time that you need to run the test. Your performance analyst can help here.
You may want to start with a small B percentage and ramp up gradually, either for load reasons, or if there’s something a bit controversial in your test that may perform particularly poorly with users.
Choose your cookie expiry
The cookie expiry time should be short until you have established traffic across the desired split between your variants.
Once your variants are at the desired proportions, the cookie expiry time should be longer than the expected test duration. This ensures that users are kept in the same variant for the duration of the test which allows them to get used to whatever changes you have made.
It’s good to have a margin for error, to (for example) correct faulty tracking at the start, or perhaps run the test for a few more days at the end if you need more data.
2. How to set up an A/B test
- Add your test to the A/B test register.
- If you want to use Google Analytics to monitor the A/B test, talk to a performance analyst and pick a GA dimension to use for your test.
- Create dictionary and A/B test files in the govuk-cdn-config repo. See an example for the dictionaries and an example for the A/B configuration (these used to be in a different repo). For more details, see the dictionaries README.
- Deploy the dictionary changes to each environment using the Update_CDN_Dictionaries Jenkins job. The dictionaries must be deployed to the
- Deploy the Fastly configuration to each environment using the Deploy_CDN Jenkins job. Use the same parameters as in step 4. You can test it on staging by visiting https://www.staging.publishing.service.gov.uk. Changes should appear almost immediately - there is no caching of the CDN config.
- Use the govuk_ab_testing gem to serve different versions to your users. It can be configured with the analytics dimension selected in step 2.
- To activate or deactivate the test, or to change the B percentage, update your test in the govuk-cdn-config repo and deploy the dictionaries.
If you’re making a change to Search API, you may also want to test using the search performance explorer. In which case, you’ll need to add your AB test to the application, and manually deploy it to Heroku.
3. How to remove your A/B test
Follow these steps:
- Remove your test from the ab_tests configuration file in the govuk-cdn-config repo and remove the dictionary files.
- Deploy the Fastly configuration to each environment using the Deploy_CDN Jenkins job. The
vhostmust be set to
www, and the credentials are in the govuk-secrets repo.
- Deploy the govuk-cdn-config changes to each environment using the Update_CDN_Dictionaries Jenkins job. Use the same parameters as in step 2.
- Mark the end date in A/B test register.