Table of contents

A/B testing

How A/B testing works

GOV.UK uses our Content Delivery Network (Fastly) to run A/B and multivariate tests.

For a general introduction to A/B testing from a content design perspective, see the Confluence Wiki.

Pub?w=1330&h=517

Source: GOV.UK Architecture Google Drive

How A/B testing works

Fastly receives the request

When the user requests a GOV.UK page that has A/B testing enabled, they will reach Fastly first.

Fastly appends the GOVUK-ABTest-Example header to the request for downstream apps:

  • If the request already has a cookie, use the value of that
  • If not, randomly assign the user to a test variant and set it value based on that

Fastly looks up the weighting for the random assignment in a Fastly dictionary configured for our A/B tests.

Fastly will then try to get a response from its cache. The Vary: GOVUK-ABTest-Example header on previously cached responses will ensure that the right version is returned to the user.

If there’s nothing in Fastly’s cache, it’ll send the request down the stack to GOV.UK.

GOV.UK (Varnish caching layer)

Varnish tries to get a response from the cache. Because Fastly has sent the GOVUK-ABTest-Example header, it knows whether to return the A or B version from the cache. If there’s nothing in the cache, Varnish forwards the request to the application server.

Application layer

The application (for example, government-frontend or collections) inspects the GOVUK-ABTest-Example header to determine which version of the content to return.

It also adds an extra response header: Vary: GOVUK-ABTest-Example. This instructs Fastly and Varnish to cache both versions of the page separately.

Varnish (response)

Varnish saves the response in the cache. The Vary: GOVUK-ABTest-Example response header will ensure that A and B are cached separately.

Fastly responds

Fastly also saves the response in the cache. The Vary: GOVUK-ABTest-Example response header will ensure that A and B are cached separately.

If the original request did not have the ABTest-Example cookie, Fastly will set a Set-Cookie header to the response based on the value of the GOVUK-ABTest-Example header.

Further reading

This page was last reviewed on 31 January 2019. It needs to be reviewed again on 31 July 2019 by the page owner #govuk-2ndline .
This page was set to be reviewed before 31 July 2019 by the page owner #govuk-2ndline. This might mean the content is out of date.