Deployments for data.gov.uk

Warning This document has not been updated for a while now. It may be out of date.

Last updated: 5 Apr 2024

This document describes the release and rollout automation (CI/CD) for data.gov.uk and how to deploy a release to production.

Overview

The data.gov.uk applications run in the datagovuk namespace on the GOV.UK Kubernetes clusters. Builds and tests (CI) run in GitHub Actions.

Rollout automation is in Argo CD under the datagovuk, ckan and dgu-shared Argo applications. These are configured via the dgu-app-of-apps Argo application.

There are some differences between data.gov.uk and other GOV.UK applications in how rollouts work. In data.gov.uk, you need to:

push a Git tag to create a release, such as v1.2.3
approve and merge an automated PR to roll out a release to integration, staging or production

data.gov.uk’s releases and rollouts tooling is otherwise similar to the rest of GOV.UK.

List existing releases of a data.gov.uk application

You can see a list of all the tagged releases for each application:

Publish
Find
CKAN (ckanext-datagovuk)
PyCSW
Solr

Find and Publish

Create a release

Create a release of datagovuk_find or datagovuk_publish by pushing a Git tag to the application’s GitHub repository.

You must name the tag using the format v<major>.<minor>.<patch>, where major, minor and patch are integers. For example v1.0.23 is a valid release tag whereas 1.0.23 is not.

You should determine the version number for your release by looking up the current version and applying the principles of semantic versioning (semver). For example if your release includes a new feature but no breaking changes, you should increment the minor version number.

Find the latest existing release tag.

git tag --list --sort=-version:refname v\* | head

Tag your new release.
```
git tag v1.0.23
```
Push your new release tag to GitHub.
```
git push origin v1.0.23
```
The create-pr-on-tags workflow in GitHub Actions will raise a PR in govuk-dgu-charts with a title like Update <app> tags for integration (<commit_sha>).
Merge the PR to roll the release out to the integration environment.

Builds can take up to 10 minutes. You can view progress in GitHub Actions under the build-and-push-images-on-tags workflow in the app’s repo.

Deployments typically take 5 minutes. You can view progress in Argo CD.

Promote a release to staging or production

Once you are happy that your change works well in the integration environment, you can promote it to staging and then to production.

The create-charts-pr workflow in GitHub Actions will have automatically raised a pair of PRs after you merged the Update <app> tags for integration PR. See the previous section if your release is not yet in integration.

Merge the Update <app> tags for staging PR to roll your release out to staging.
Test that your changes are working in Staging.
Merge the Update <app> tags for production PR.

CKAN

You can create a CKAN release by pushing a tag to the ckanext-datagovuk repo. CKAN releases and rollouts work the same way as Find and Publish.

Update a CKAN extension

To update a CKAN extension such as spatial, harvest or dcat, you must update the extension’s pinned version SHA in the Dockerfile for the appropriate version of CKAN under docker/ckan.

Manually test the CSW service

When deploying changes that affect the CSW service, such as OWSLib or PyCSW updates, you should test that the /csw endpoint still works properly.

curl -v https://ckan.staging.publishing.service.gov.uk/csw

Alternatively you can visit https://ckan.staging.publishing.service.gov.uk/csw in Firefox. Chrome and Safari don’t display the XML very nicely.

You can also test the daily sync between PyCSW and CKAN:

ckan ckan-pycsw load -p /var/ckan/pycsw.cfg -u http://ckan-ckan:5000

PyCSW should automatically update as part of the CKAN rollout. If changes are not appearing, you can:

check the rollout status in Kubernetes

kubectl -n datagovuk rollout status deploy/ckan-pycsw

check the status of the ckan-pycsw pods in Argo CD, under the ckan Argo application