govuk-infrastructure: 4. Use AWS Load Balancer Controller for edge traffic services
Date: 2021-08-12
Status
Accepted
Context
We require a method of managing and directing external internet traffic into the cluster. Kubernetes provides several options for handling inbound traffic.
We do not want to expose the cluster directly to the internet, and so require an AWS load balancer in front of the cluster. The load balancer must support TLS termination and integration with our DNS provider (AWS Route 53).
Historically Kubernetes has supported provisioning of ALBs and NLBs for Service resources of type=LoadBalancer via the in-tree (built-in) AWS cloud provider, with out-of-tree controllers required for Ingress resources. Built-in cloud providers are now considered deprecated overall, in favour of out-of-tree providers, so an Ingress Controller with support for Service resources is required.
The primary and recommended ingress controller for AWS/EKS is the AWS Load Balancer Controller, which can provision and manage ALBs for Ingress resources and NLBs for Service resources.
We must also consider how Kubernetes edge services and AWS load balancers will interact with the existing GOV.UK Router service, as there is significant overlap in their functionality and responsibilities. This will require further investigation and likely experimentation, and so that end we should ensure that we're able to use both Ingress and Service Kubernetes resources so that we have the flexibility to support a wide range of use cases in the immediate term - L4 & L7 traffic, name-based routing, HTTP->HTTPS redirection, etc.
Decision
Use the AWS Load Balancer Controller.
Consequences
The AWS Load Balancer Controller supports TLS certificates via AWS Certificate Manager only, so certificates must be managed there (to be covered in a future ADR).
The load balancer controller does not handle DNS for declared ingress hostnames - a solution to this will be covered in a future ADR.
An appropriate ALB/NLB topography (how many LBs routing to where) will need to be established - by default the controller will provision one ALB per Ingress resource, which may not be what we want. Ingresses can be grouped however.
The load balancer controller supports AWS WAF and Shield, both of which are currently in use on GOV.UK.
Access control for Ingress rules must be investigated, likely in conjunction with Kubernetes namespace usage - if all of GOV.UK is deployed into a single namespace, and multiple users or service accounts have the same level of access to Ingress objects, then user or process for component A could modify or destroy ingress rules for component B.