Table of contents

Move apps between servers

Most frontend and backend apps on GOV.UK share a small number of servers. In some circumstances, apps may use more than their share of resources and may affect other apps on the same server. In these cases, apps can be moved to their own servers using the appropriate steps for either Carrenza or AWS.

Carrenza

Create the new servers (if required)

If you’re moving an app to new servers, start by creating those servers.

  1. Add vCloud configuration for the new servers for production and staging (you can choose any IP addresses that are not currently used as long as they use the same prefix as others in the same file).
  2. Add the IP addresses for the new servers and a new Puppet node class in govuk-puppet.
  3. Deploy puppet in staging.
  4. SSH to the puppetmaster in staging and run a loop to sign SSL certificates for the new servers as they’re created: bash $ ssh puppetmaster-1.management.staging $ while true; do sudo puppet cert sign --all; sleep 10; done
  5. Run the Launch VMs Jenkins job, using the Carrenza staging username and password, to create the new servers.
  6. Once the job has completed, terminate the loop on the puppetmaster.
  7. Re-run everything in production once you’ve checked everything works.

Add the relevant app(s) to the new servers

Once the servers are created, they will run puppet to apply relevant configuration, but there will be no apps on them.

  1. Change the hieradata to add the relevant app(s) to the new servers.
  2. Deploy puppet in staging.
  3. To speed up the process, you can run puppet manually on the new servers to deploy the apps.
  4. Change the relevant app deployment scripts to deploy to both the old and new servers.
  5. Deploy the app(s) (they will be deployed to both the old and new server).
  6. Re-run everything in production once you’ve checked everything works.

Add the new servers to the load balancers

Note It is important that all servers are running the same version of the app at this point.

Once you’ve verified that the app(s) have been deployed to all the new servers, you’ll need to change the load balancers to start using the new servers as part of a managed migration from the old to the new servers.

  1. Change the hieradata for the load balancers to balance between both the old and new servers.
  2. Deploy puppet in staging.
  3. Re-run everything in production once you’ve checked everything works.

The app(s) will now be running on both old and new servers and the load balancers will use both sets of servers as their configuration is updated by puppet.

Remove the old servers from the load balancers

Once all the load balancers have been updated, requests to the app(s) will be using both sets of servers, which will allow you to remove the old servers from serving the app(s).

  1. Change the hieradata once more to remove the old servers from the load balancers for the relevant app(s), and also to remove the app(s) from the old servers since they will no longer be serving the app(s).
  2. Deploy puppet in staging.
  3. Change the relevant app deployment scripts to deploy to only the new servers.
  4. Re-run everything in production once you’ve checked everything works.

Warning Bundling up the removal of the old servers from the load balancers and the removal of the app(s) from the old servers may result in a period where the old servers are still part of the load balancer group but don’t have the app(s) running. This can be mitigated by either splitting up these changes, or running puppet manually on the load balancers after deployment to ensure no further traffic is routed to the old servers.

Clean up

Once everything is done, make some final changes to the puppet configuration and hieradata to clean up the temporary changes you made above.

AWS

Note You need to be at least a Power User in AWS to be able to run the following procedure. You can check by looking in the govuk-aws-data repository. Some IAM changes may require Administrator access, so you’ll need to ask someone in the Reliability Engineering team to run these for you.

  1. Add Terraform configuration (1, 2, 3) to create the new servers, load balancers, security groups, DNS entries etc.
  2. Add data to complement the configuration above (1, 2).
  3. Deploy the Terraform configuration. You need to do this three times for each environment:
    1. infra-security-groups project in the govuk stack.
    2. app-name-of-your-app project in the blue stack (replace app-name-of-your-app with the name you configured above).
    3. infra-public-services project in the govuk stack (only if your app is accessible directly from the public internet).

For each deployment, set the environment to one of integration, staging or production and run the plan command first to double-check the changes before running the apply command to make the changes.

👉 Deploy AWS infrastructure with Terraform

This page was last reviewed . It needs to be reviewed again by the page owner #govuk-2ndline.