Skip to main content
  1. Home
  2. AWS
Warning This document has not been updated for a while now. It may be out of date.
Last updated: 19 Apr 2021

AWS IAM Key Rotation

Overview

GOV.UK uses AWS IAM accounts to manage applications access to our AWS infrastructure. The Access Keys associated with these accounts need to be rotated every 3 months.

Rotating AWS IAM keys requires permissions which are available with the admin, internal-admin, platformhealth-poweruser or poweruser roles. You’ll be able to rotate keys only if you can assume one of these roles.

How to rotate Access Keys

Log in to the IAM Management Console in the environment you want to rotate keys for and you’ll see a list of users and the age of their access keys.

1. Find users access key

  • Click on the user whose key you want to rotate.
  • Click on the Security Credentials tab.

Users should always only have one access key. The only time a user should have two keys is when we are in the process of rotating the key that is in use. For this reason, AWS allow users to have a maximum of two Access Keys at the same time.

Note

If a user has two keys, the first task is to remove one of them.

If one key is not in use, you can remove it and rotate the other key (keep reading to see how).

If both keys are in use, you need to pick one and replace all uses of it with the other key. Then, once it’s no longer in use, you can remove it and rotate the other key (keep reading to see how).

2. Find out if the key is used

3. Find where the key is used

The first place you should look is the GOV.UK AWS User Keys spreadsheet. Someone may already have done the work of logging where the key is used. If they haven’t, you’ll need to follow the manual steps below (and should update the spreadsheet with your findings when you’re done).

When searching for keys, keep in consideration that they may be present in govuk-secrets or in the control panel of 3rd party services, such as Fastly or Logit.

Firstly, search for the key in govuk-secrets:

  • Enter govuk-secrets/puppet in your machine.
  • Run the appropriate Puppet AWS common actions considering the following:
    • The command to run is related to the environment you need to target. For example, to edit global staging credentials in the govuk::apps namespace, you need to run bundle exec rake 'eyaml:edit[staging,apps]'.
    • You may need to run more than one command. For example, to edit the global integration credentials both in and out of the govuk::apps namespace, you need run both $ bundle exec rake 'eyaml:edit[integration]' and $ bundle exec rake 'eyaml:edit[integration,apps]'.
  • Search for the Access Key ID in the result.

Repeat the previous steps for govuk-secrets/puppet_aws:

Keys may also be defined in more then one place. For example, the hieradata values govuk::apps::support::aws_access_key_id and govuk::apps::support_api::aws_access_key_id contain the same key.

Secondly, search for the key in the control panel of 3rd party services we use, for example, Fastly or Logit.

If it isn’t present in govuk-secrets hieradata, nor in any 3rd party services, you can safely remove the key.

4. Create and deploy new key

  • Click Create access key in Security credentials.
  • If the key to change was in a 3rd party service:
    • Replace the old Access Key ID and Secret access key with the new ones.
  • If the key to change was in govuk-secrets:
    • Replace the old Access Key ID and Secret access key with the new ones.
    • Redeploy govuk-puppet so it picks up the new key.
    • Either wait up to 30 minutes for this to take effect or run govuk_puppet --verbose in the machine that uses that key.

Rotating SES SMTP credentials

The SMTP credentials for SES are connected to a user in IAM, but they are separate to the security credentials. As of August 2020, to rotate SMTP credentials, the user needs to be re-created and the new credentials will be displayed. They cannot be rotated after that point.

When creating the new user, it’s sensible to append the current date (such as .YYYYMMDD) to the username so it’s easy to see when the user was created and the username won’t conflict with the existing user. This allows both credentials to be available at the same time while switching over. The old user can be deleted once the app is using the new credentials.

See this PR for an example and the relevant AWS documentation on obtaining SES credentials.

5. Ensure new key is being used

Ensure that the Last used values change to show that the new key is being used and the old key is not used. Keys that are used infrequently may take a while to update and updates can take a few minutes to show in the control panel.

If the Last used time of the old key continues to change, the key has not been completely removed so you’ll need to find where the key is used again.

If the Last used time of the old key doesn’t change anymore, you can remove the key.

6. Remove the old key

Old keys can either be deleted or made inactive. Making the key inactive will prevent the key from being used, but it can quickly be made active again. This is useful if you are not confident that all uses of the key have been updated. In production systems, this could prevent minor outages scaling into incidents.

  • Notify 2ndline that you are about to delete a key or make a key inactive.
  • Click Make inactive to inactivate the key or click the X to delete it.

Command Line Interface

It’s also possible to view and update access keys with the AWS CLI.

More in the AWS section

Learn

How to...