Table of contents

Replicate application data locally for development

Dumps are generated from production data in the early hours each day, and can then be downloaded. You’ll use the replicate-data-local.sh script script in the govuk-puppet repository.

The Licensify and Signon databases aren’t synced from production because of security concerns. Mapit’s database is downloaded in the Mapit repo, so won’t be in the backups folder.

Pre-requisites to importing data

👉 First, set up your AWS account 👉 Then, set up your CLI access for AWS

Replication

When you have AWS access, you can download and import the latest data by running:

mac$ cd ~/govuk/govuk-puppet/development-vm/replication
mac$ ./replicate-data-local.sh -n

The data will download to a folder named with today’s date in ./backups, for example ./backups/2018-01-01.

then

dev$ cd /var/govuk/govuk-puppet/development-vm/replication
dev$ ./replicate-data-local.sh -d backups/YYYY-MM-DD/ -s

Databases take a long time to download and use a lot of disk space (up to ~30GB uncompressed). The process also uses a lot of compute resource as you import the data.

The downloaded backups will automatically be deleted after import (whether successful or not) unless the -k flag is specified.

If you don’t have AWS access

If you don’t have AWS access, ask someone to give you a copy of their dump. Then, from govuk-puppet/development-vm/replication run:

dev$ ./replicate-data-local.sh -d path/to/dir -s

If you’re running out of disk space

See running out of disk space in development.

If you get a curl error when restoring Elasticsearch data

Check the service is running:

dev$ sudo service elasticsearch-development.development start

If you get an error saying Elasticsearch is not installed, you may need to reprovision the VM from your host machine:

mac$ vagrant provision

Can’t take a write lock while out of disk space (in MongoDB)

You may see such an error message which will prevent you from creating or even dropping collections. So you won’t be able to replicate the latest data.

You will need to delete large Mongo collections to free up space before they can be re-imported. Follow this guide on how to delete them, and ensure that Mongo honours their removal.

Find your biggest Mongo collections by running:

dev$ sudo ncdu /var/lib/mongodb

You can re-run the replication but skip non-Mongo imports like MySQL if it’s already succesfully imported. Use replicate-data-local.sh --helpto see the options.

For example, to run an import but skip MySQL and Elasticsearch:

dev$ replicate-data-local.sh -q -e -d backups/2017-06-08 -s
This page was last reviewed on 10 October 2018. It needs to be reviewed again on 10 April 2019 by the page owner #govuk-2ndline .
This page was set to be reviewed before 10 April 2019 by the page owner #govuk-2ndline. This might mean the content is out of date.