Skip to main content
Table of contents

Deployment

Deploy when GitHub is unavailable

Public GitHub (application code)

Many of the git repositories which make up GOV.UK are hosted on public GitHub. We may need to deploy changes at any time, and GitHub is a Software as a Service (SaaS) product which is not guaranteed to be available.

If GitHub is unavailable, we lose:

  • Access to our primary code repository
  • The ability to authenticate with Jenkins, as it makes use of GitHub groups

We mirror all GitHub repositories tagged with govuk to AWS CodeCommit every 2 hours. In the event of GitHub being down, we can deploy from AWS CodeCommit repos. This requires help from a GOV.UK AWS admin.

Deploying from AWS CodeCommit

Use the normal deployment Jenkins job but check the box to deploy from AWS CodeCommit.

Making changes to code in AWS CodeCommit before deployment

GOV.UK AWS admin users can give access to developers who need to make changes to the code before deployment.

  1. In the root of the local repo, run the following commands to install the AWS credential helper and add CodeCommit as a remote:
   git config credential.helper '!aws codecommit credential-helper $@'
   git config credential.UseHttpPath true
   git remote add aws https://git-codecommit.eu-west-2.amazonaws.com/v1/repos/<app>
  1. Get some AWS credentials for the govuk-tools AWS account

  2. Export the access key ID, secret access key and session token from the last step, for example:

   export AWS_ACCESS_KEY_ID=...
   export AWS_SECRET_ACCESS_KEY=...
   export AWS_SESSION_TOKEN=...
  1. Fetch the AWS upstream by running git fetch aws

  2. Checkout a new branch on the upstream by running git checkout -b aws/my-super-secret-fix

  3. Make and commit your changes to this branch, and make sure all tests run successfully locally (since CodeCommit does not run tests)

  4. Push your changes to CodeCommit by running git push

  5. Tag your changes by running git tag release_XYZ, where XYZ is one more that the latest release tag for the application you’re working on, as reported by the Release app

  6. Push your new tag to CodeCommit by running git push aws release_XYZ

Deploying the code change

  1. Review the pull request on AWS CodeCommit through the AWS Console (access to GOV.UK repos must be granted by a GDS AWS administrator).

  2. Create a release tag manually in git. This should follow the standard format release_X. Tag the branch directly instead of merging it.

  3. Don’t use the release app. Go directly to the Deploy_App Jenkins job, and check DEPLOY_FROM_AWS_CODECOMMIT.

After deploying the change

  1. Push the branch and tag to GitHub.

  2. Merge the branch into master.

  3. Record the missing deployment in the Release app.

Troubleshooting

If running any git commands against CodeCommit returns you a 403, you probably have expired credentials stored in your MacOS keychain from a previous attempt. Apparently MacOS stores these the first time you use it and subsequently tries to use them again despite you setting new AWS credentials.

To fix this:

  1. Open Keychain Access (use cmd-space to search for it).

  2. Select “Passwords” from the “Category” on the left.

  3. Search for git-codecommit.

  4. Right click on the item and select “Get Info”.

  5. Click “Access Control” on the modal that pops up.

  6. Select “git-credential-osxkeychain” from the list.

  7. Hit the minus key.

  8. Try your terminal commands again.

  9. If you are prompted to add the item to keychain, deny.

There is more information about setting up your access key in the AWS guide

Authenticating with Jenkins

If GitHub.com is down, we will not be able to log in to Jenkins.

In this scenario, Jenkins security should be disabled to enable deployment:

  1. SSH to the Jenkins deploy instance:
ssh jenkins-1.<environment>

in Carrenza, and

govukcli set-context <environment>
govukcli ssh jenkins

in AWS.

  1. Disable Puppet: govuk_puppet -r "Emergency Jenkins deploy" --disable
  2. Edit the Jenkins configuration file: sudo vim /var/lib/jenkins/config.xml
  3. Replace <useSecurity>true</useSecurity> with <useSecurity>false</useSecurity> and save
  4. Restart Jenkins: sudo service jenkins restart
  5. Browse to the Jenkins UI and begin the deployment process
  6. When completed, enable and run Puppet on the instance: govuk_puppet --enable && govuk_puppet --test

Note that once security is disabled, anyone on GDS trusted IPs will be able to deploy to that environment. This will bypass protection for Production - do not leave Production without security for any longer than necessary.

See the Jenkins documentation for further details.

Simulating a GitHub outage on 2nd line

You can simulate an outage of GitHub.com by modifying your local hosts file.

  1. sudo vi /etc/hosts
  2. Add 127.0.0.1 github.com

Don’t forget to remove it afterwards!

This page was last reviewed on 4 November 2019. It needs to be reviewed again on 4 May 2020 by the page owner #govuk-developers .
This page was set to be reviewed before 4 May 2020 by the page owner #govuk-developers. This might mean the content is out of date.