It’s possible this page is out of date, but it’s is still relevant if we ever need to rebuild a VMWare vCloud environment somewhere.
Create a new environment for GOV.UK
Manual steps to ask of the IaaS provider
Ask the IaaS provider to:
a. Create a vCloud Director organisation. b. Provision these virtual data centres (‘vDCs’):
- GOV.UK API
- GOV.UK Backend
- GOV.UK Frontend
- GOV.UK Licensify
- GOV.UK Management
- GOV.UK Redirector
- GOV.UK Router
c. Provision a vShield Edge Gateway and associate it to the vDCs. The edge gateway needs to be put into 'Full’ mode; the default is compact mode. d. Allocate external IP addresses and configure a 10.0.0.0/8 subnet on the vShield Edge Gateway. e. Ensure that storage policies have a sufficient quota for the amount of disk space required in for each vDC. f. Ensure ingress access to the organisation on these TCP ports:
- 22 and 1022 (for jumpbox-2): for the nat_a IP address
- 80 and 443 for all IP addresses
Further restrictions on where traffic on these ports can go is handled later in the edge gateway firewall.
g. Ensure egress access from the vCloud organisation on these TCP ports:
- 9418 (git)
- 11371 (hkp)
Creating vApp templates
“vApp templates” are machine images that are used to create virtual machines. They are equivalent to an Amazon Machine Image (AMI).
The process for creating a vApp template is documented in govuk-provisioning.
Configuring the vDC networks
The input to vcloud-net-launch is a yaml file describing the networks to launch.
- Start by copying the configuration from another environment
Correct the following values:
- Edge Gateway name
- vDC names
vcloud-net-launchwith the yaml file as an argument
Find your Skyscape API username using the link at the top-right of the Portal.
Configuring edge gateways - firewalls, NAT and load balancing
- Firewall traffic
- Load balance at the transport layer (ie TCP and UDP)
- Translate addresses as part of Network Address Translation
Edge gateway configuration is automated; use the vcloud-configure-edge tool with the configuration checked in to govuk-provisioning/vcloud-edge_gateway. For example, to configure firewalls in the Carrenza Preview environment:
# get an authentication token eval $(FOG_CREDENTIAL=foo bundle exec vcloud-login) FOG_CREDENTIAL=foo bundle exec vcloud-edge-configure \ rules/firewall.yaml.mustache --template-vars vars/preview_carrenza_vars.yaml FOG_CREDENTIAL=foo bundle exec vcloud-logout
You will have to create a new template vars file for the new environment but you should not have to change the firewall.yaml.mustache file unless you are changing how firewalls are configured in all environments.
Launching and configuring virtual machines
Note: When provisioning VMs, make sure your machine doesn’t suspend partway though. If you’re using a Mac, you might need to use caffeinate
In order to make sure we are billed for what we use, it is important to align the size of the machines being created with the sizes available from the IaaS supplier. vCloud allows unrestricted sizes, but in practise the vendor bills based on discrete sizes of machine.
Machine sizes should be aligned to the ones in use by your vendor.
Troubleshooting VM provisioning
Should you encounter problems provisioning a VM and need to reprovision
it, see our guide <reprovision>. If you need to delete a VM that
you won’t be recreating, you can remove it from Puppet and Icinga by
running the following command on
that has already been provisioned):
Set up the Puppet Master
Launch the Puppet Master VM:
cd ~/govuk/govuk-provisioning # get an authentication token eval $(FOG_CREDENTIAL=foo bundle exec vcloud-login) FOG_CREDENTIAL=foo tools/launch_vapp.rb vcloud-launcher/environment_provider/management.yaml puppetmaster-1 FOG_CREDENTIAL=foo bundle exec vcloud-logout
Make sure you have the puppet repository checked out locally and have pulled the latest from the master branch.
Make sure you have the deployment repository checked out locally and have pulled the latest from the master branch.
Install ssh-copy-id if you do not have it already:
brew install ssh-copy-id
Run this script to push the Puppet code to the Puppet Master (you will need to pass the appropriate options to push-puppet.sh):
cd ~/govuk/govuk-provisioning/tools ./push-puppet.sh -h
Set up a deployment Jenkins
- Create any DNS records (deploy.preview.etc)
Create a Jenkins machine:
cd ~/govuk/govuk-provisioning # get an authentication token eval $(FOG_CREDENTIAL=foo bundle exec vcloud-login) FOG_CREDENTIAL=foo tools/launch_vapp.rb vcloud-launcher/environment_provider/management.yaml jenkins-1 FOG_CREDENTIAL=foo bundle exec vcloud-logout
- On Github Enterprise, create a new configuration repository for your environment. Make sure that the Bots team has access to to your new repository
Add public half of the Jenkins user’s SSH key to github.gds:
ssh jenkins-1.management.staging 'sudo cat /var/lib/jenkins/.ssh/id_rsa.pub'
You will need to “Fake GitHub Sign In” from the staff tools user page.
Also put that public half into the deploy user on puppetmaster-1
Configure scm with details for git repo
Clone the configuration repository for Jenkins:
sudo rm -rf /var/lib/jenkins/scm-sync-configuration/checkoutConfiguration/ git clone firstname.lastname@example.org:gds/jenkins-config-p1production.git sudo mkdir /var/lib/jenkins/scm-sync-configuration/checkoutConfiguration sudo rsync -avPh ~bob/jenkins-config-p1production/ /var/lib/jenkins/scm-sync-configuration/checkoutConfiguration/ sudo chown -R jenkins:jenkins /var/lib/jenkins/scm-sync-configuration/checkoutConfiguration/
Tell scm-sync to reload from git. (in the UI “Manage Jenkins” -> “Config System” -> “Reload Config from SCM”)
Restart Jenkins by accessing
Run the Jenkins job “Deploy: Puppet”
- you will need to
chown -Rv deploy: /usr/share/puppet/production/releases/*first
- you will need to
Launch other virtual machines
Run the “autosign loop” on the puppetmaster and keep an eye on the output:
while true; do sudo puppet cert sign --all; sleep 10; done
Create a vcloud-launch job on Jenkins that will run vcloud-launch over all of the yaml files in the
You can now use the deployment Jenkins to deploy applications. The order of deployment matters and might have changed since this was written. Here is the order that was used for building the Carrenza preview environment:
- licensify (it has separete Jenkins jobs)
Other environment setup
Allow access to the new environment from CI. This might require firewall changes.
While you’re still setting things up, it’s a good idea to create separate, new, deployment jobs and have the old deployment jobs trigger the new jobs after they successfully complete. This way, every deployment goes to both environments during the switchover, but there is no chance of breaking the build pipelines for anyone else.
There are two MySQL clusters:
- “Normal” mysql
- Whitehall mysql
Both clusters need to be set up so that
slave from the
To set up slaving, follow the documentation on setting up mysql replication , except that you might need to take the initial dump from a different environment. This is the case if you are rebuilding preview.
There are two PostgreSQL clusters:
- “Normal” PostgreSQL
- Transition PostgreSQL
To set up slaving, follow the documentation on setting up PostgreSQL replication.
You will need to copy them from production using the
env-sync-and-backup tools. There are existing Jenkins jobs for doing
Assuming Elasticsearch has been installed by Puppet, and is contactable on :9200, then you can run the elasticsearch job in Jenkins to copy the relevant data and indexes over.
If it turns out that this isn’t the case, you can attempt to fix this manually by using the HEAD plugin to check the indexes and aliases match that of a known-working environment.
- download the existing Elasticsearch indexes from a known working environment, and copy the files to /var/es_dump on the new environment’s custer
- install the es_dump_restore Gem, at version 0.0.6 or above
- import the indexes using
es_dump_restore restore "<http://localhost:9200/>" INDEX_NAME INDEX_FILE
Then, ask Rummager to re-index:
This page is owned by #infrastructure and needs to be reviewed
RUMMAGER_INDEX=all bundle exec rake rummager:migrate_from_unaliased_index