Table of contents

Deployment

Deploy fixes for a security vulnerability

When responding to a security incident, you should review any code changes in private before deploying them, so that you don’t accidentally disclose the vulnerability.

To do this, push branches to the AWS CodeCommit backup of the repository, rather than the normal repository on GitHub.

This repository should be up to date as of the previous release, but will be missing any unreleased commits that are on GitHub master. This is fine as you don’t want to deploy these.

Applying the fix

  1. In the root of the local repo, run the following commands to install the AWS credential helper and add CodeCommit as a remote:
   git config credential.helper '!aws codecommit credential-helper $@'
   git config credential.UseHttpPath true
   git remote add aws https://git-codecommit.eu-west-2.amazonaws.com/v1/repos/<app>
  1. Get some AWS credentials for the govuk-tools AWS account

  2. Export the access key ID, secret access key and session token from the last step, for example:

   export AWS_ACCESS_KEY_ID=...
   export AWS_SECRET_ACCESS_KEY=...
   export AWS_SESSION_TOKEN=...
  1. Fetch the AWS upstream by running git fetch aws

  2. Checkout a new branch on the upstream by running git checkout -b aws/my-super-secret-fix

  3. Make and commit your changes to this branch, and make sure all tests run successfully locally (since CodeCommit does not run tests)

  4. Push your changes to CodeCommit by running git push

  5. Tag your changes by running git tag release_XYZ, where XYZ is one more that the latest release tag for the application you’re working on, as reported by the Release app

  6. Push your new tag to CodeCommit by running git push aws release_XYZ

Deploying the fix

  1. Ensure nobody else deploys the app until you’ve confirmed the vulnerability is fixed.

  2. Review the pull request on AWS CodeCommit.

  3. Create a release tag manually in git. This should follow the standard format release_X. Tag the branch directly instead of merging it.

  4. Don’t use the release app. Go directly to the Deploy_App Jenkins job, and check DEPLOY_FROM_AWS_CODECOMMIT.

After deploying the fix

  1. Ensure the vulnerability is fixed.

  2. Push the branch and tag to GitHub.

  3. Merge the branch into master.

  4. Record the missing deployment in the Release app.

Troubleshooting

If running any git commands against CodeCommit returns you a 403, you probably have expired credentials stored in your MacOS keychain from a previous attempt. Apparently MacOS stores these the first time you use it and subsequently tries to use them again despite you setting new AWS credentials.

To fix this:

  1. Open Keychain Access (use cmd-space to search for it).

  2. Select “Passwords” from the “Category” on the left.

  3. Search for git-codecommit.

  4. Right click on the item and select “Get Info”.

  5. Click “Access Control” on the modal that pops up.

  6. Select “git-credential-osxkeychain” from the list.

  7. Hit the minus key.

  8. Try your terminal commands again.

  9. If you are prompted to add the item to keychain, deny.

There is more information in Step 3 in the AWS guide

This page was last reviewed on 15 March 2019. It needs to be reviewed again on 15 September 2019 by the page owner #govuk-2ndline .
This page was set to be reviewed before 15 September 2019 by the page owner #govuk-2ndline. This might mean the content is out of date.