Skip to main content
Warning This document has not been updated for a while now. It may be out of date.
Last updated: 1 Sep 2021

govuk-infrastructure: Kubernetes external secrets

This project implements external-secrets as described in ADR 8.

AWS SecretsManager is the source of truth for most secrets.

How are SecretsManager Secrets accessed by applications in Kubernetes?

The external-secrets operator (ESO) creates CRDs such as ClusterSecretStore and ExternalSecret. The ClusterSecretStore is not namespaced, and is used by all namespaces.

The ClusterSecretStore is responsible for authenticating with AWS when we need to manage a secret in SecretsManager. It holds references to secrets which hold credentials to access the SecretsManager API. The ClusterSecretStore 'aws-secretsmanager' uses a short-lived Service Account token to authenticate with AWS.

The ESO reconciles each ExternalSecret with secrets in AWS SecretsManager (using creds provided by ClusterSecretStore), and creates a Secret (and keeps the Kubernetes Secret in sync with the SecretsManager Secret).

external-secrets syncing AWS SM secrets to Kubernetes secrets

Application developers create an ExternalSecret when required for their apps, and reference the associated Kubernetes Secret in the app’s Helm deployment.

The platform team is responsible for providing the ClusterSecretStore. Service teams are responsible for defining ExternalSecrets.

Not discussed here is how secrets are added to SecretsManager. It would be nice if we could avoid having Terraform config for every SecretsManager secret, and have this config live entirely in Kubernetes. But the lifecycle of secrets in SecretsManager is not the responsibility of ESO, so we need to figure out how to manage SecretsManager secrets. Also not discussed is how we’ll rotate secrets, this is also out of scope of the ESO.