Pact Testing
Pact is a tool we use for contract testing. Contract testing involves creating a set of tests that are shared between an API (the “provider”) and its users (“consumers”). Contract tests allow consumers to define a contract that it expects its provider to uphold. This means that a consumer can specify what response (both status code and body) they expect from a provider given a certain http call. This is useful to ensure that changes to the provider do not break the interaction between it and its consumers.
For example, a consumer might expect a certain field to be returned in a json response from a provider, and use that field in its application code. Without pact tests, we might be able to remove that field from the provider’s response without knowing that it is a breaking change. Good pact tests will prevent such breaking changes from being made.
Since in tests we do not want to spin up both apps to test the interactions, pact provides us with a “broker” which deals with the interactions between the two apps in test.
For example, the Places Manager API has a “pact” or “contract” with GDS API Adapters:
- the expected interactions are defined in places_manager_api_pact_test.rb in GDS API Adapters
- when these tests are run they output a JSON pactfile which is published to our pact broker (live site)
- the build of Places Manager will setup a test environment and use this pactfile to test the real API
GDS API Adapters is really a proxy for real “consumer” apps, like Whitehall. The gem includes a set of shared stubs for use in each app’s own tests (example). Using the stubs ensures each app stays in sync with GDS API Adapters, which can then do contract testing on their behalf.
Running Pact tests locally
For a consumer (GDS API Adapters)
Pact tests are run as part of the regular test suite:
bundle exec rake test
Or they can be run on their own:
bundle exec rake pact_test
For a provider
Example for Frontend (provider of /bank-holidays.json):
bundle exec rake pact:verify
You can also test against a specific branch of GDS API Adapters:
env PACT_CONSUMER_VERSION=branch-<branch-of-gds-api-adapters> bundle exec rake pact:verify
Note: when a build runs for GDS API Adapters, the generated JSON pact is pushed to the broker as branch-<branch-name> using the pact:publish:branch rake task; this is why PACT_CONSUMER_VERSION needs to start with branch-.
Alternatively, you can test against local changes to GDS API Adapters:
env PACT_URI="../gds-api-adapters/spec/pacts/gds_api_adapters-bank_holidays_api.json" bundle exec rake pact:verify
Adding tests for a new app
If the app already had some Pact tests, follow the steps for changing existing Pact tests.
Write the consumer and provider tests.
- Consumer example (Note: since this example was written we now store pact tests in the
tests/pactsdirectory). - Provider example.
- Consumer example (Note: since this example was written we now store pact tests in the
Check the tests pass locally for both provider and consumer
- CI won’t be able to test them yet as they won’t be pushed to the pact broker.
Merge the consumer tests for the new app.
- This will cause the pact tests to be published to the pact broker.
Merge a GitHub Action (example) into the provider app to verify the pact
- This will verify the provider app fulfills the contract published by the consumer.
Update the consumer to utilise the provider GitHub action as part of the build process (example)
- This will verify the provider contract when the consumer builds.
Changing existing Pact tests
Follow these steps in order to change the provider and consumer in tandem.
Make the change to the API (your provider, e.g. Places Manager), in a branch.
- Its build should fail at the Pact test stage, because it is testing against the default branch of the consumer.
Make the change to the pactfile in the consumer (see ‘Where to define the consumer tests’ below) in a branch.
- Its build should fail at the Pact test stage, because it is testing against the default branch of the provider.
Confirm locally that the provider can verify against the published consumer pact
env PACT_CONSUMER_VERSION=branch-<branch-of-gds-api-adapters> bundle exec rake pact:verify
Merge the provider change
- We’ve configured failing Pact tests to not block merging, so you should still be able to merge.
Re-run the consumer tests
- These should pass now that the provider has been updated.
Merge the consumer change
Where to define the consumer tests
As is noted above, GDS API Adapters is often used as the consumer for pact tests. This is because many api calls made within our system are made via GDS API Adapters. Therefore treating GDS API Adapters as the consumer saves us from needing to write pact tests for the same api call in multiple different applications.
When an application calls an API directly, not via GDS API Adapters, this application will be the consumer. For example, Publishing API is the consumer in the pact tests between it and Content Store, since it makes direct calls to Content store.