Skip to main content
Last updated: 9 Oct 2023

EKS/Kubernetes GOV.UK cheatsheet

This is a quick summary of the most useful commands and guidance for interacting with GOV.UK’s EKS platform.

This page assumes you have completed the set up instructions, tested your access and set up the recommended aliases.

Common tasks

  • To access logs for an app:
    • k logs deploy/account-api
  • To open a read-only rails console:
    • k exec -it deploy/router-api -- rails c --sandbox
  • To open a normal rails console:
    • k exec -it deploy/router-api -- rails c
  • To open a shell:
    • k exec -it deploy/government-frontend -- bash
  • To open a shell on Router:
    • k exec -it deploy/router -c nginx

Read more about each command further on in the document.

Set your AWS environment and role

# set the correct kubectl context (`k config get-contexts` to see available ones)
k config use-context integration
# set the appropriate AWS environment (integration/staging/production) and role (see below)
eval $(gds aws govuk-integration-poweruser -e --art 8h)

Use an AWS role with sufficient permissions:

  • readonly can view logs
  • poweruser can run Rake tasks or open a shell
  • administrator gives full access (you’ll rarely need this)

See full instructions for more detail.

Prior to using kubectl, you’ll need to set the recommended aliases (see the kubectl cheatsheet), default contexts and globals:

alias k=kubectl
k config set-context --current --namespace=apps
export AWS_REGION=eu-west-1

What’s staying and going?

All front-end traffic and back-end publisher traffic is now on Kubernetes (EKS) instead of EC2.

Some things haven’t yet been replatformed, and some things will remain unchanged - see what stays/goes/changes with Platform Engineering.

App deployments

  • To see what’s deployed where, use the Release app.
  • Continuously deployed apps will continue to be deployed post-merge, using ArgoCD. You can see the progress of a workflow in Argo Workflows
  • To deploy a branch (or for non-continuously deployed apps):
    • Go to the app’s repo in GitHub
    • Choose Actions
    • Choose Deploy (from the left-hand column).
    • Click “Run workflow”
    • Enter the name of the branch to deploy under “Commit, tag or branch name to deploy”.
    • (Always leave the “Use workflow from” option set as main, unless you’re actually trying to change the workflow itself)

CDN and DNS deployments

Smokey

Smokey runs in Argo Workflows persist for longer only if they have failed (because of limitations of Argo Workflows that we can’t easily get around). The Smokey cronjob keeps the last 3 successes and the last 3 failures.

Dashboards, alerting and Icinga

Fetch logs

You can fetch the list of pods:

k get pods

This will return a list, including output like:

NAME                                                              READY   STATUS             RESTARTS           AGE
account-api-568d46d6f7-4pnld                                      2/2     Running            0                  34m
account-api-568d46d6f7-txwsx                                      2/2     Running            0                  34m
account-api-dbmigrate-6n5mp                                       0/1     Completed          0                  35m
account-api-worker-54597b666c-jlcfn                               1/1     Running            0                  34m
asset-manager-7bd745d655-tzlgz                                    2/2     Running            0                  15h
...

As you can see above, each pod either runs the application (we can see two pods running Account API, for example) or runs a worker or cron job for the application (we can see one pod running the Account API application worker).

You can also fetch just a subset list of pods by passing the lapp argument:

$ k get pods -lapp=account-api

NAME                           READY   STATUS    RESTARTS   AGE
account-api-75f95899f6-gjdmg   2/2     Running   0          21h
account-api-75f95899f6-nn56t   2/2     Running   0          21h

You can fetch the logs for all running containers in a pod. Take one of the pods from the output above:

k logs account-api-568d46d6f7-4pnld

Or you can fetch the logs using its ‘app deployment’ name. This will choose one of the Account API pods at random, much like the old jumpbox methods:

k logs deploy/account-api

Open a rails console

Open a rails console with:

k exec -it deploy/whitehall-admin -- rails c

If you try doing this having assumed a readonly role when setting context, you’ll see a permissions error:

Defaulted container "app" out of: app, nginx
Error from server (Forbidden): pods "whitehall-admin-6bb47c48d-wkkxk" is forbidden: User "christopher.ashton-user" cannot create resource "pods/exec" in API group "" in the namespace "apps"

You need write permissions to be able to open a console (poweruser or above).

Open a shell

Open a shell (requires poweruser or above):

k exec -it deploy/whitehall-admin -- bash

Note that the prompt will have prefix that may feel a little unfamiliar: I have no name!@whitehall-admin-6bb47c48d-wkkxk:/app$. This only happens for asset-manager and whitehall-admin, because those have to run as a specific user ID that matches what’s on NFS (due to how NFS was set up on the old system).

To open a shell on Router, you’ll need to exec into the nginx container:

k exec -it deploy/router -c nginx -- bash

This is because Router’s image is FROM scratch, so bash isn’t available.

Run a rake task

See Run a Rake task on EKS

Here’s an idempotent rake task you can run to try one out:

k exec deploy/email-alert-api -- rake 'support:view_emails[your.email@digital.cabinet-office.gov.uk]'

Note that we haven’t specified the -it flags in the command above because we don’t need to interact with the task. To find out what they do, run k exec --help.