Skip to main content
Table of contents

data.gov.uk

Deployments for data.gov.uk

Find and Publish (Rails Apps)

Continuous Integration

Travis is configured for both Publish (CI) and Find (CI) according to .travis.yml file in each repo. Each PR request should automatically build a Heroku Review App, which can be accessed from the PR page on GitHub.

Heroku Integration Environment

Heroku has a pipeline for each of Publish and Find, with each app set to run in its ‘integration’ environment. Each pipeline has a permanent instance of the app, providing a common instance of Elasticsearch for us by the Find PR apps.

Each repo has a Procfile and an app.json file, which help to specify how the app is deployed. The environment variables ('Config Vars’) are then set via the website, both for the permanent app instance, and the review app template.

Note: At the moment the Heroku deployed apps do not have much data in them as they are not hooked up with the Integration environment so should be purely used to test frontend changes and not as an end to end test of the Integration stack.

PaaS Staging and Production Environments

Publish and Find are provisioned on GOV.UK PaaS. Each app repo contains a set of manifests that specify the container settings for when it’s deployed.

The deployment for both apps can be triggered automatically via GitHub flow, or manually via command line tools.

The deployment to staging is triggered when a PR gets merged into master. You can check the Travis logs of the master build to see progress. The deployment to production is triggered when a new release with an appropriate version number is created in GitHub. This behaviour is defined in Publish’s travis.yml and in Find’s travis.yml.

The process to manually deploy is as follows.

## run once to install the plugin
cf add-plugin-repo CF-Community https://plugins.cloudfoundry.org
cf install-plugin autopilot -r CF-Community -f

## manually deploy the publish app
cf zero-downtime-push publish-data-beta-staging -f staging-app-manifest.yml
cf zero-downtime-push publish-data-beta-staging-worker -f staging-worker-manifest.yml

There are some scripts available for datagovuk_publish which can run a deploy for you, for example:

$ cd datagovuk_publish
$ git pull
$ ./scripts/deploy-staging.sh
$ ./scripts/deploy-production.sh

For more advanced uses of the PaaS that are not covered in the PaaS internal documentation (e.g. provisioning an app using a buildpack that is not Ruby or Java), refer to the Cloud Foundry documentation.

Set up interactive SSH session

You may need this to run things such as rake tasks:

/tmp/lifecycle/shell

You need to run this every time you start an SSH session.

For more information, refer to the Cloud Foundry documentation on SSH Session Environment.

CKAN

CKAN uses GOV.UK infrastructure, which includes Jenkins for CI.

Integration, staging and production environments behave like all other GOV.UK applications on AWS.

Deployments for ckan are initiated via updates to ckanext-datagovuk, whenever a ckan extension is updated the install-dependencies file will need an update with the relevant extensions (eg. spatial, harvest, dcat) commit or version number.

CSW

When deploying changes that affect the CSW service, (OWSLib or PyCSW updates), provided at the /csw endpoint for the CKAN publisher you should make sure that the endpoint is still running correctly by curling it, curl "https://ckan.publishing.service.gov.uk/csw", or view it in Firefox. Chrome and Safari do not show the XML on the page correctly.

The daily sync between pycsw and ckan should also be tested:

sudo su deploy
cd /var/apps/ckan && ./venv/bin/paster --plugin=ckanext-spatial ckan-pycsw load -p /var/ckan/pycsw.cfg -u http://localhost:3220

The pycsw web process should be automatically restarted after each ckan deployment.

If changes are not appearing, you can try these commands.

# show running csw workers
sudo initctl list | grep csw

# restart the pycsw web worker
sudo service pycsw_web-procfile-worker restart
This page was last reviewed on 3 October 2019. It needs to be reviewed again on 3 April 2020 by the page owner #govuk-platform-health .
This page was set to be reviewed before 3 April 2020 by the page owner #govuk-platform-health. This might mean the content is out of date.