Skip to main content
Warning This document has not been updated for a while now. It may be out of date.
Last updated: 10 Jan 2022

How A/B testing works

GOV.UK uses our Content Delivery Network (Fastly) to run A/B and multivariate tests.

For a general introduction to A/B testing from a content design perspective, see the Confluence Wiki.

Source: GOV.UK Architecture Google Drive

How A/B testing works

A/B tests are only enabled for users who have opted in to analytics cookies.

Fastly receives the request

When a user requests a GOV.UK page, the request reaches Fastly’s CDN first. If the page has A/B testing enabled, Fastly will add a GOVUK-ABTest-Example header to downstream requests. The header’s value is set depending on whether a cookie called ABTest-Example is present in the original request:

  • If the original request has the cookie, then the header’s value is set to that cookie’s value. For instance, the downstream request will have GOVUK-ABTest-Example: B if the ABTest-Example cookie of the original request has value B.
  • If the original request doesn’t have the cookie, then Fastly chooses a variant (e.g. A) and sets the header’s value accordingly: GOVUK-ABTest-Example: A

In order to choose a variant, Fastly looks up the weighting of each variant in a Fastly dictionary configured for all A/B tests.

Fastly will then try to get a response from its cache.

  • if the requested page is cached, the vary: GOVUK-ABTest-Example response header on previously cached responses will ensure that the right version is returned to the user.

  • if not, Fastly will send the request downstream to GOV.UK (including the ABTest-Example header)

GOV.UK (Varnish caching layer)

Varnish tries to get a response from the cache. Because Fastly has sent the GOVUK-ABTest-Example header, it knows whether to return the A or B version from the cache. If there’s nothing in the cache, Varnish forwards the request to the application server.

Application layer

The application (for example, government-frontend or collections) inspects the GOVUK-ABTest-Example header to determine which version of the content to return.

It also adds an extra response header: vary: GOVUK-ABTest-Example. This instructs Fastly and Varnish to cache both versions of the page separately.

Varnish (response)

Varnish saves the response in the cache. The vary: GOVUK-ABTest-Example response header will ensure that A and B are cached separately.

Fastly responds

Fastly also saves the response in the cache. The vary: GOVUK-ABTest-Example response header will ensure that A and B are cached separately.

If the original request did not have the ABTest-Example cookie, Fastly will set a Set-Cookie header to the response based on the value of the GOVUK-ABTest-Example header.

Further reading