Skip to main content
Last updated: 3 May 2023

EKS/Kubernetes infrastructure

This is a temporary page to tide us over as we get used to the new EKS platform. It will eventually be consolidated when the GOV.UK Kubernetes docs are absorbed into the Developer Docs.

Quick reference

You need to have completed the set up instructions and tested your access.

The following commands assume you have alias k=kubectl as per the kubectl cheatsheet, and also set the apps namespace as your default (k config set-context --current --namespace=apps).

You’ll need to set your region and context as below (swapping out the environment as appropriate. Note that the poweruser role is required as a minimum when opening shell or rails console. readonly can be used for accessing logs):

export AWS_REGION=eu-west-1
eval $(gds aws govuk-integration-poweruser -e --art 8h)
  • To access logs for an app:
    • k logs deploy/account-api
  • To open a rails console:
    • k -n apps exec -it deploy/router-api -- rails c
  • To open a shell:
    • k -n apps exec -it deploy/government-frontend -- bash >> curl # (for example)
  • To open a shell on Router:
    • k -n apps exec -it deploy/router -c nginx

Read more about each command further on in the document.

What’s staying and going?

All front-end traffic and back-end publisher traffic is now on Kubernetes (EKS) instead of EC2.

Some things haven’t yet been replatformed, and some things will remain unchanged - see what stays/goes/changes with Platform Engineering.

App deployments

  • To see what’s deployed where, use the Release app.
  • Continuously deployed apps will continue to be deployed post-merge, using ArgoCD. You can see the progress of a workflow in Argo Workflows
  • To deploy a branch (or for non-continuously deployed apps):
    • Go to the app’s repo in GitHub
    • Choose Actions
    • Choose Deploy (from the left-hand column).
    • Click “Run workflow”
    • Enter the name of the branch to deploy under “Commit, tag or branch name to deploy”.
    • (Always leave the “Use workflow from” option set as main, unless you’re actually trying to change the workflow itself)

CDN and DNS deployments

  • DNS rollouts are via Terraform Cloud
  • Fastly CDN config rollouts are still via Jenkins for now.


Smokey runs in Argo Workflows persist for longer only if they have failed (because of limitations of Argo Workflows that we can’t easily get around). The Smokey cronjob keeps the last 3 successes and the last 3 failures.

Dashboards, alerting and Icinga

  • The Technical 2nd Line Dashboard will be kept up to date as we improve our dashboards. It’s worth bookmarking
  • The detailed Sidekiq dashboard is not yet available for EKS (the old one still looks at EC2) but you can see queue lengths and age-of-oldest-job on the Sidekiq Grafana dashboard.
  • Dashboards are on and no longer require VPN. Don’t get confused by the old Grafana, which will show mostly empty graphs about the old EC2 setup.
  • Most Icinga alerts are irrelevant now, except for:
    • “scheduled publications in Whitehall not queued” / “overdue publications in Whitehall”
    • “Travel Advice email alert check” / “Medical Safety Email alert check”
    • Anything to do with crawler/mirrors
    • Anything to do with Licensify

Fetch logs

You can fetch the list of pods:

k get pods

This will return a list, including output like:

NAME                                                              READY   STATUS             RESTARTS           AGE
account-api-568d46d6f7-4pnld                                      2/2     Running            0                  34m
account-api-568d46d6f7-txwsx                                      2/2     Running            0                  34m
account-api-dbmigrate-6n5mp                                       0/1     Completed          0                  35m
account-api-worker-54597b666c-jlcfn                               1/1     Running            0                  34m
asset-manager-7bd745d655-tzlgz                                    2/2     Running            0                  15h

As you can see above, each pod either runs the application (we can see two pods running Account API, for example) or runs a worker or cron job for the application (we can see one pod running the Account API application worker).

You can also fetch just a subset list of pods by passing the lapp argument:

$ k get pods -lapp=account-api

NAME                           READY   STATUS    RESTARTS   AGE
account-api-75f95899f6-gjdmg   2/2     Running   0          21h
account-api-75f95899f6-nn56t   2/2     Running   0          21h

You can fetch the logs for all running containers in a pod. Take one of the pods from the output above:

k logs account-api-568d46d6f7-4pnld

Or you can fetch the logs using its ‘app deployment’ name. This will choose one of the Account API pods at random, much like the old jumpbox methods:

k logs deploy/account-api

Open a rails console

Open a rails console with:

k exec -it deploy/whitehall-admin -- rails c

If you try doing this having assumed a readonly role when setting context, you’ll see a permissions error:

Defaulted container "app" out of: app, nginx
Error from server (Forbidden): pods "whitehall-admin-6bb47c48d-wkkxk" is forbidden: User "christopher.ashton-user" cannot create resource "pods/exec" in API group "" in the namespace "apps"

You need write permissions to be able to open a console (poweruser or above).

Open a shell

Open a shell (requires poweruser or above):

k exec -it deploy/whitehall-admin -- bash

Note that the prompt will have prefix that may feel a little unfamiliar: I have no name!@whitehall-admin-6bb47c48d-wkkxk:/app$. This only happens for asset-manager and whitehall-admin, because those have to run as a specific user ID that matches what’s on NFS (due to how NFS was set up on the old system).

To open a shell on Router, you’ll need to exec into the nginx container:

k exec -it deploy/router -c nginx -- bash

This is because Router’s image is FROM scratch, so bash isn’t available.

Run a rake task

See Run a Rake task on EKS

Here’s an idempotent rake task you can run to try one out:

k exec deploy/email-alert-api -- rake 'support:view_emails[]'

Note that we haven’t specified the -it flags in the command above because we don’t need to interact with the task. To find out what they do, run k exec --help.