Skip to main content
  1. Home
  2. Deployment
Last updated: 16 Apr 2024

Deploy when GitHub is unavailable

AWS CodeCommit

We mirror all non-archived GitHub repositories tagged with govuk to AWS CodeCommit via the “Mirror repositories” GitHub Actions workflow. The workflow uses the github_action_mirror_repos_role IAM role.

Build and deploy an app when GitHub is unavailable

If GitHub.com is down, you can still clone a GOV.UK repo from CodeCommit and build and deploy a container image from your workstation.

Run the following commands from the root directory of the repository.

  1. Set up CodeCommit on your machine if you haven’t already.
  2. Follow the CodeCommit guide to clone the repo and commit/push changes if needed.
  3. Set the image tag, image registry and image repository names that you will use by running the following commands. You don’t need to modify any of the values in the commands.
LOCAL_HEAD_SHA=$(git rev-parse HEAD)
IMAGE_TAG="release-${LOCAL_HEAD_SHA}"
REGISTRY="172025368201.dkr.ecr.eu-west-1.amazonaws.com"
REPO=$(basename "$PWD")
  1. Build the container image and tag it appropriately.
docker build --platform linux/amd64 -t $REGISTRY/$REPO:$IMAGE_TAG .
  1. Log into ECR and push the image:
gds aws govuk-production-poweruser aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin $REGISTRY
docker push $REGISTRY/$REPO:$IMAGE_TAG

Deploy the image to Kubernetes using Argo CD

⚠️ This procedure will disable automatic deployments for all applications, not only in the environment you are working on but also in higher environments. For example, performing this procedure on integration will prevent all automatic deployments in integration, staging and production.

  1. Fetch the password for the Argo CD admin user. This varies by environment.

    kubectl -n cluster-services get secret argocd-initial-admin-secret -oyaml |yq .data.password |base64 -d

  2. Log into Argo CD (for example in integration).

  3. Disable auto-sync for the app-config application:

    1. From the Applications page (the Argo CD homepage), choose the app-config application.
    2. Press the App Details button near the top of the page.
    3. Scroll down to the bottom of the page and press Disable auto-sync. Argo will prompt you before actually disabling auto-sync.

  4. Repeat the steps above to turn off auto-sync for the application you wish to deploy.

  5. Close the App details sidebar, then select the Deploy object for the component of the application you’d like to redeploy. For example, to update the Sidekiq workers for Account API, you would open up the account-api-worker Deploy object.

  6. Go to Live manifest and select Edit.

  7. Find the image: field for the app container. It should look something like 172025368201.dkr.ecr.eu-west-1.amazonaws.com/<app-name>:release-2e902e3df274a00bbabba7ccf84cbef96ccc9b9e.

  8. Update the tag part of the image: value to the new image tag that you pushed to ECR. The part you are changing should look something like release-2e902e3df274a00bbabba7ccf84cbef96ccc9b9e.

  9. Click Save. Argo CD will start the deployment, which should complete in under ten minutes.

When GitHub is available again (or when you’ve completed the above as part of a Technical 2nd Line drill), return things to normal by re-enabling the auto-sync:

  1. From the Applications page (the Argo CD homepage), choose the app-config application.
  2. Press the App Details button near the top of the page.
  3. Scroll down to the bottom of the page and press Enable auto-sync.

Troubleshooting 403 errors from AWS

If running any git commands against CodeCommit returns a 403 response, you probably have expired credentials stored in your macOS keychain from a previous attempt. Apparently macOS stores these the first time you use it and subsequently tries to use them again despite you setting new AWS credentials.

To fix this:

  1. Open Keychain Access (use cmd-space to search for it).
  2. Select “Passwords” from the “Category” on the left.
  3. Search for git-codecommit.
  4. Right click on the item and select “Get Info”.
  5. Click “Access Control” on the modal that pops up.
  6. Select “git-credential-osxkeychain” from the list.
  7. Hit the minus key.
  8. Try your terminal commands again.
  9. If you are prompted to add the item to keychain, deny.

There is more information about setting up your access key in the AWS guide

More in the Deployment section

Learn

How to...