govuk-infrastructure: 0004-use-aws-load-balancer-controller-for-edge-traffic-services
Date: 2021-08-12
Status
Accepted
Context
We require a method of managing and directing external internet traffic into the cluster. Kubernetes provides several options for handling inbound traffic.
We do not want to expose the cluster directly to the internet, so we require an AWS load balancer in front of the cluster. The load balancer must support TLS termination and integration with our DNS provider (AWS Route 53).
Historically, Kubernetes has supported provisioning of Application Load Balancers (ALBs) and Network Load Balancers (NLBs)
for Service resources of type=LoadBalancer using the in-tree (built-in) AWS cloud provider.Ingress resources
have always been supported by out-of-tree controllers required.
Built-in cloud providers are now considered deprecated overall, in favour of out-of-tree providers,
so users are now required to use an Ingress Controller
with support for Service resources.
The primary, and recommended, ingress controller for AWS' Elastic Kubernetes Service (EKS) is
the AWS Load Balancer Controller, which can provision and manage ALBs for Ingress resources and NLBs for
Service resources.
We must also consider how Kubernetes edge services and AWS load balancers will interact with the existing
GOV.UK Router service, because there is significant overlap in their functionality and responsibilities.
This will require further investigation, so we should be able to use both Ingress and Service
Kubernetes resources. This will ensure we have the flexibility to support a wide range of use cases in the immediate
term: layer 4 and layer 7 traffic, name-based routing, HTTP-to-HTTPS redirection.
Decision
Use the AWS Load Balancer Controller.
Consequences
The AWS Load Balancer Controller supports TLS certificates with AWS Certificate Manager only. A future ADR will cover the way in which we use AWS Certificate Manager.
The load balancer controller does not handle DNS for declared ingress host names. A future ADR will cover a solution to this.
We will need to establish an appropriate Application and Network Load Balancer topography: how many load balancers routing
to where. By default the controller will provision one load balancer per Ingress resource, which might not be what we want.
Ingresses can be grouped, however.
The load balancer controller supports AWS Web Application Firewall and Shield, both of which are currently in use on GOV.UK.
We must also investigate access control for Ingress rules, likely in conjunction with Kubernetes namespace usage. If all
of GOV.UK is deployed into a single namespace, and many users or service accounts have the same level of access to
Ingress objects, then user or process for component A could modify or destroy ingress rules for component B.