Deploy fixes for a security vulnerability
When responding to a security incident, you should review any code changes in private before deploying them, so that you don’t accidentally disclose the vulnerability.
To do this, push branches to the AWS CodeCommit backup of the repository, rather than the normal repository on GitHub.
This repository should be up to date as of the previous release, but will be missing any unreleased commits that are on GitHub master. This is fine as you don’t want to deploy these.
Applying the fix
- In the root of the local repo, run the following commands to install the AWS credential helper and add CodeCommit as a remote:
git config credential.helper '!aws codecommit credential-helper $@' git config credential.UseHttpPath true git remote add aws https://git-codecommit.eu-west-2.amazonaws.com/v1/repos/<app>
Get some AWS credentials for the
Export the access key ID, secret access key and session token from the last step, for example:
export AWS_ACCESS_KEY_ID=... export AWS_SECRET_ACCESS_KEY=... export AWS_SESSION_TOKEN=...
Fetch the AWS upstream by running
git fetch aws
Checkout a new branch on the upstream by running
git checkout -b aws/my-super-secret-fix
Make and commit your changes to this branch, and make sure all tests run successfully locally (since CodeCommit does not run tests)
Push your changes to CodeCommit by running
Tag your changes by running
git tag release_XYZ, where XYZ is one more that the latest release tag for the application you’re working on, as reported by the Release app
Push your new tag to CodeCommit by running
git push aws release_XYZ
Deploying the fix
Ensure nobody else deploys the app until you’ve confirmed the vulnerability is fixed.
Review the pull request on AWS CodeCommit.
Create a release tag manually in git. This should follow the standard format
release_X. Tag the branch directly instead of merging it.
Don’t use the release app. Go directly to the
Deploy_AppJenkins job, and check
After deploying the fix
Ensure the vulnerability is fixed.
Push the branch and tag to GitHub.
Merge the branch into master.
Record the missing deployment in the Release app.
If running any
git commands against CodeCommit returns you a 403, you probably
have expired credentials stored in your MacOS keychain from a previous attempt.
Apparently MacOS stores these the first time you use it and subsequently tries
to use them again despite you setting new AWS credentials.
To fix this:
Open Keychain Access (use cmd-space to search for it).
Select “Passwords” from the “Category” on the left.
Right click on the item and select “Get Info”.
Click “Access Control” on the modal that pops up.
Select “git-credential-osxkeychain” from the list.
Hit the minus key.
Try your terminal commands again.
If you are prompted to add the item to keychain, deny.
There is more information in Step 3 in the AWS guide