Table of contents

Run an A/B test

1. Overview

There are two ways of doing A/B testing on GOV.UK. The first uses Javascript to change the contents on a page. The second uses our CDN to allow frontend applications to render different pages.

For a general introduction to A/B testing from a content design perspective, see the Confluence Wiki

Which implementation you choose depends on your A/B testing needs:

JavaScript

  • Pro: much simpler to configure
  • Con: content flashes on page load: users see one version first, which is then replaced with the other version
  • Con: every HTTP response contains both versions, which could significantly increase the page size if the change is large
  • Choose if: you want to A/B test a small, simple change to a page, such as the wording of a link

Fastly

  • Pro: No content flashing
  • Con: Requires changes to the CDN configuration
  • Con: Because it depends on Fastly this can only be fully tested on staging and production. You can still test your A and B versions in other environments, it’s just not a completely realistic test without the full stack.
  • Choose if: you are testing wide-ranging changes, perhaps with redirects to the new version, or if you really need to avoid content flashing

Both options let you report the results to Google Analytics.

2. Javascript testing

Javascript testing uses the Multivariate test framework from the GOV.UK frontend toolkit. It works by running a piece of Javascript after the page is loaded.

A step-by-step walkthrough is provided on the Wiki.

3. Using Fastly

This method uses Fastly, our Content Delivery Network (CDN).

3.1 How it works

Fastly receives the request

When the user requests a GOV.UK page that has A/B testing enabled, they will reach Fastly first.

Fastly appends the GOVUK-ABTest-Example header to the request for downstream apps:

  • If the request already has a cookie, use the value of that
  • If not, randomly assign the user to a test variant and set it value based on that

Fastly will then try to get a response from its cache. The Vary: GOVUK-ABTest-Example header on previously cached responses will ensure that the right version is returned to the user.

If there’s nothing in Fastly’s cache, it’ll send the request down the stack to GOV.UK.

GOV.UK (Varnish caching layer)

Varnish tries to get a response from the cache. Because Fastly has sent the GOVUK-ABTest-Example header, it knows whether to return the A or B version from the cache. If there’s nothing in the cache, Varnish forwards the request to the application server.

Application layer

The application (for example, government-frontend or collections) inspects the GOVUK-ABTest-Example header to determine which version of the content to return.

It also adds an extra response header: Vary: GOVUK-ABTest-Example. This instructs Fastly and Varnish to cache both versions of the page separately.

Varnish (response)

Varnish saves the response in the cache. The Vary: GOVUK-ABTest-Example response header will ensure that A and B are cached separately.

Fastly responds

Fastly also saves the response in the cache. The Vary: GOVUK-ABTest-Example response header will ensure that A and B are cached separately.

If the original request did not have the ABTest-Example cookie, Fastly will set a Set-Cookie header to the response based on the value of the GOVUK-ABTest-Example header.

3.2 Checklist for enabling A/B testing

Follow these steps:

  1. Get your cookie listed on the cookies page. The content team can help.
  2. You will have to make appropriate changes in govuk-cdn-config.
  3. Use the govuk_ab_testing gem to serve different versions to your users.

Resources

This page is owned by @tijmen and needs to be reviewed