Skip to main content
  1. Home
  2. Infrastructure
Last updated: 5 Sep 2023

EKS/Kubernetes infrastructure

This is a temporary page to tide us over as we get used to the new EKS platform. It will eventually be removed and everything consolidated into the Kubernetes section of the Developer Docs.

Quick reference

This section assumes you satisfy all the prerequisites.

Common tasks

  • To access logs for an app:
    • k logs deploy/account-api
  • To open a read-only rails console:
    • k exec -it deploy/router-api -- rails c --sandbox
  • To open a normal rails console:
    • k exec -it deploy/router-api -- rails c
  • To open a shell:
    • k exec -it deploy/government-frontend -- bash
  • To open a shell on Router:
    • k exec -it deploy/router -c nginx

Read more about each command further on in the document.

Set your AWS environment and role

# set the correct kubectl context (`k config get-contexts` to see available ones)
k config use-context integration
# set the appropriate AWS environment (integration/staging/production) and role (see below)
eval $(gds aws govuk-integration-poweruser -e --art 8h)

Use an AWS role with sufficient permissions:

  • readonly can view logs
  • poweruser can run Rake tasks or open a shell
  • administrator gives full access (you’ll rarely need this)

See full instructions for more detail.

Prerequisites

You need to have:

The following commands also assume you have set the recommended aliases (see the kubectl cheatsheet), default contexts and globals:

alias k=kubectl
k config set-context --current --namespace=apps
export AWS_REGION=eu-west-1

What’s staying and going?

All front-end traffic and back-end publisher traffic is now on Kubernetes (EKS) instead of EC2.

Some things haven’t yet been replatformed, and some things will remain unchanged - see what stays/goes/changes with Platform Engineering.

App deployments

  • To see what’s deployed where, use the Release app.
  • Continuously deployed apps will continue to be deployed post-merge, using ArgoCD. You can see the progress of a workflow in Argo Workflows
  • To deploy a branch (or for non-continuously deployed apps):
    • Go to the app’s repo in GitHub
    • Choose Actions
    • Choose Deploy (from the left-hand column).
    • Click “Run workflow”
    • Enter the name of the branch to deploy under “Commit, tag or branch name to deploy”.
    • (Always leave the “Use workflow from” option set as main, unless you’re actually trying to change the workflow itself)

CDN and DNS deployments

Smokey

Smokey runs in Argo Workflows persist for longer only if they have failed (because of limitations of Argo Workflows that we can’t easily get around). The Smokey cronjob keeps the last 3 successes and the last 3 failures.

Dashboards, alerting and Icinga

  • The Technical 2nd Line Dashboard will be kept up to date as we improve our dashboards. It’s worth bookmarking https://govuk-2ndline-dashboard.herokuapp.com/
  • The detailed Sidekiq dashboard is not yet available for EKS (the old one still looks at EC2) but you can see queue lengths and age-of-oldest-job on the Sidekiq Grafana dashboard.
  • Dashboards are on https://grafana.eks.production.govuk.digital/ and no longer require VPN. Don’t get confused by the old Grafana, which will show mostly empty graphs about the old EC2 setup.
  • Most Icinga alerts are irrelevant now, except for:
    • “scheduled publications in Whitehall not queued” / “overdue publications in Whitehall”
    • “Travel Advice email alert check” / “Medical Safety Email alert check”
    • Anything to do with crawler/mirrors
    • Anything to do with Licensify

Fetch logs

You can fetch the list of pods:

k get pods

This will return a list, including output like:

NAME                                                              READY   STATUS             RESTARTS           AGE
account-api-568d46d6f7-4pnld                                      2/2     Running            0                  34m
account-api-568d46d6f7-txwsx                                      2/2     Running            0                  34m
account-api-dbmigrate-6n5mp                                       0/1     Completed          0                  35m
account-api-worker-54597b666c-jlcfn                               1/1     Running            0                  34m
asset-manager-7bd745d655-tzlgz                                    2/2     Running            0                  15h
...

As you can see above, each pod either runs the application (we can see two pods running Account API, for example) or runs a worker or cron job for the application (we can see one pod running the Account API application worker).

You can also fetch just a subset list of pods by passing the lapp argument:

$ k get pods -lapp=account-api

NAME                           READY   STATUS    RESTARTS   AGE
account-api-75f95899f6-gjdmg   2/2     Running   0          21h
account-api-75f95899f6-nn56t   2/2     Running   0          21h

You can fetch the logs for all running containers in a pod. Take one of the pods from the output above:

k logs account-api-568d46d6f7-4pnld

Or you can fetch the logs using its ‘app deployment’ name. This will choose one of the Account API pods at random, much like the old jumpbox methods:

k logs deploy/account-api

Open a rails console

Open a rails console with:

k exec -it deploy/whitehall-admin -- rails c

If you try doing this having assumed a readonly role when setting context, you’ll see a permissions error:

Defaulted container "app" out of: app, nginx
Error from server (Forbidden): pods "whitehall-admin-6bb47c48d-wkkxk" is forbidden: User "christopher.ashton-user" cannot create resource "pods/exec" in API group "" in the namespace "apps"

You need write permissions to be able to open a console (poweruser or above).

Open a shell

Open a shell (requires poweruser or above):

k exec -it deploy/whitehall-admin -- bash

Note that the prompt will have prefix that may feel a little unfamiliar: I have no name!@whitehall-admin-6bb47c48d-wkkxk:/app$. This only happens for asset-manager and whitehall-admin, because those have to run as a specific user ID that matches what’s on NFS (due to how NFS was set up on the old system).

To open a shell on Router, you’ll need to exec into the nginx container:

k exec -it deploy/router -c nginx -- bash

This is because Router’s image is FROM scratch, so bash isn’t available.

Run a rake task

See Run a Rake task on EKS

Here’s an idempotent rake task you can run to try one out:

k exec deploy/email-alert-api -- rake 'support:view_emails[your.email@digital.cabinet-office.gov.uk]'

Note that we haven’t specified the -it flags in the command above because we don’t need to interact with the task. To find out what they do, run k exec --help.

More in the Infrastructure section

Learn

How to...