How A/B testing works
GOV.UK uses our Content Delivery Network (Fastly) to run A/B and multivariate tests. A list of previous and currently running A/B tests is available internally.
For a general introduction to A/B testing from a content design perspective, see the GOV.UK internal wiki.
A/B tests are only enabled for users who have opted in to analytics cookies.
If the user is opted-in to usage cookies, but has not got an ABTest cookie yet:
If the user is opted in to usage cookies, and has already got an ABTest cookie set:
req.http.GOVUK-ABTest-Example is set as soon as the user is assigned a bucket, which happens for users who have not got cookies yet (as well as for those who already have). vcl_recv where this happens cannot set the cookie itself, because it has not got a response object to mutate yet.
So if we say “Only set the cookie if GOVUK-ABTest-Example is not already set” then we’ll never set the cookie.
We need a way of detecting the situation where the user already had a cookie when they arrived, if we want to avoid sending a Set-Cookie header in every response.
Fastly receives the request
When a user requests a GOV.UK page, the request reaches Fastly’s CDN first. If the page has A/B testing enabled, Fastly will add a GOVUK-ABTest-Example header to downstream requests. The header’s value is set depending on whether a cookie called ABTest-Example is present in the original request:
- If the original request has the cookie, then the header’s value is set to that cookie’s value. For instance, the downstream request will have
GOVUK-ABTest-Example: Bif theABTest-Examplecookie of the original request has valueB. - If the original request does not have the cookie, then Fastly chooses a variant (e.g.
A) and sets the header’s value accordingly:GOVUK-ABTest-Example: A
In order to choose a variant, Fastly looks up the weighting of each variant in a Fastly dictionary configured for all A/B tests.
Fastly will then try to get a response from its cache.
if the requested page is cached, the
vary: GOVUK-ABTest-Exampleresponse header on previously cached responses will ensure that the right version is returned to the user.if not, Fastly will send the request downstream to GOV.UK (including the
ABTest-Exampleheader)
Application layer
The application (for example, government-frontend or collections) inspects the GOVUK-ABTest-Example header to determine which version of the content to return.
It also adds an extra response header: vary: GOVUK-ABTest-Example. This instructs Fastly and Varnish to cache both versions of the page separately.
Fastly responds
Fastly also saves the response in the cache. The vary: GOVUK-ABTest-Example response header will ensure that A and B are cached separately.
If the original request did not have the ABTest-Example cookie, Fastly will set a Set-Cookie header to the response based on the value of the GOVUK-ABTest-Example header.