Table of contents

Run an A/B test

1. Overview

GOV.UK uses our Content Delivery Network (Fastly) to run A/B tests.

For a general introduction to A/B testing from a content design perspective, see the Confluence Wiki.

2. How it works

Pub?w=1330&h=517

Source: GOV.UK Architecture Google Drive

Fastly receives the request

When the user requests a GOV.UK page that has A/B testing enabled, they will reach Fastly first.

Fastly appends the GOVUK-ABTest-Example header to the request for downstream apps:

  • If the request already has a cookie, use the value of that
  • If not, randomly assign the user to a test variant and set it value based on that

Fastly looks up the weighting for the random assignment in a Fastly dictionary configured for our A/B tests.

Fastly will then try to get a response from its cache. The Vary: GOVUK-ABTest-Example header on previously cached responses will ensure that the right version is returned to the user.

If there’s nothing in Fastly’s cache, it’ll send the request down the stack to GOV.UK.

GOV.UK (Varnish caching layer)

Varnish tries to get a response from the cache. Because Fastly has sent the GOVUK-ABTest-Example header, it knows whether to return the A or B version from the cache. If there’s nothing in the cache, Varnish forwards the request to the application server.

Application layer

The application (for example, government-frontend or collections) inspects the GOVUK-ABTest-Example header to determine which version of the content to return.

It also adds an extra response header: Vary: GOVUK-ABTest-Example. This instructs Fastly and Varnish to cache both versions of the page separately.

Varnish (response)

Varnish saves the response in the cache. The Vary: GOVUK-ABTest-Example response header will ensure that A and B are cached separately.

Fastly responds

Fastly also saves the response in the cache. The Vary: GOVUK-ABTest-Example response header will ensure that A and B are cached separately.

If the original request did not have the ABTest-Example cookie, Fastly will set a Set-Cookie header to the response based on the value of the GOVUK-ABTest-Example header.

3. How to set up an A/B test

Follow these steps:

  1. Get your cookie listed on the cookies page. Raise a ticket on GOV.UK Zendesk and assign it to the content team’s 2nd line GOV.UK content triage. They need to know the name of the cookie that you will be using, a description and the expiry time.
  2. If you want to use Google Analytics to monitor the A/B test, talk to a performance analyst and pick a GA dimension to use for your test.
  3. Configure the A/B test in the cdn-configs repo (see an example). For more details, see the dictionaries README.
  4. Deploy the cdn-configs changes to staging and production using the Update_CDN_Dictionaries Jenkins job. The vhost must be set to www, and the credentials are in the govuk-secrets repo.
  5. Add your test to the ab_tests configuration file in the fastly-configure repo (see an example). The test name must match the name configured in the cdn-configs repo in step 3.
  6. Deploy the Fastly configuration to staging and production using the Deploy_CDN Jenkins job. Use the same parameters as in step 4. You can test it on staging by visiting https://www.staging.publishing.service.gov.uk. Changes should appear almost immediately - there is no caching of the CDN config.
  7. Use the govuk_ab_testing gem to serve different versions to your users. It can be configured with the analytics dimension selected in step 2.
  8. To activate or deactivate the test, or to change the B percentage, update your test in the cdn-configs repo and deploy the change.

4. How to tear down an A/B test

Follow these steps:

  1. Remove your test from the ab_tests configuration file in the fastly-configure repo.
  2. Deploy the Fastly configuration to staging and production using the Deploy_CDN Jenkins job. The vhost must be set to www, and the credentials are in the govuk-secrets repo.
  3. Remove your test from the cdn-configs repo. If you are removing the very last A/B test, you should replace the last test name with [] (See an example of having no A/B tests).
  4. Deploy the cdn-configs changes to staging and production using the Update_CDN_Dictionaries Jenkins job. Use the same parameters as in step 2.
  5. Get your cookie removed from the cookies page. Raise a ticket on GOV.UK Zendesk and assign it to the content team’s 2nd line GOV.UK content triage.

5. Further reading

This page is owned by @tijmen and needs to be reviewed