This is a temporary page to tide us over as we get used to the new EKS platform. It will eventually be consolidated when the GOV.UK Kubernetes docs are absorbed into the Developer Docs.
You need to have completed the set up instructions and tested your access.
The following commands assume you have
alias k=kubectl as per the kubectl cheatsheet, and also set the
apps namespace as your default (
k config set-context --current --namespace=apps).
You’ll need to set your region and context as below (swapping out the environment as appropriate. Note that the
poweruser role is required as a minimum when opening shell or rails console.
readonly can be used for accessing logs):
export AWS_REGION=eu-west-1 eval $(gds aws govuk-integration-poweruser -e --art 8h)
- To access logs for an app:
k logs deploy/account-api
- To open a rails console:
k -n apps exec -it deploy/router-api -- rails c
- To open a shell:
k -n apps exec -it deploy/government-frontend -- bash>>
curl# (for example)
- To open a shell on Router:
k -n apps exec -it deploy/router -c nginx
Read more about each command further on in the document.
What’s staying and going?
All front-end traffic and back-end publisher traffic is now on Kubernetes (EKS) instead of EC2.
Some things haven’t yet been replatformed, and some things will remain unchanged - see what stays/goes/changes with Platform Engineering.
- To see what’s deployed where, use the Release app.
- Continuously deployed apps will continue to be deployed post-merge, using ArgoCD. You can see the progress of a workflow in Argo Workflows
- To deploy a branch (or for non-continuously deployed apps):
- Go to the app’s repo in GitHub
- Choose Actions
- Choose Deploy (from the left-hand column).
- Click “Run workflow”
- Enter the name of the branch to deploy under “Commit, tag or branch name to deploy”.
- (Always leave the “Use workflow from” option set as
main, unless you’re actually trying to change the workflow itself)
CDN and DNS deployments
- DNS rollouts are via Terraform Cloud
- Fastly CDN config rollouts are still via Jenkins for now.
Smokey runs in Argo Workflows persist for longer only if they have failed (because of limitations of Argo Workflows that we can’t easily get around). The Smokey cronjob keeps the last 3 successes and the last 3 failures.
Dashboards, alerting and Icinga
- The Technical 2nd Line Dashboard will be kept up to date as we improve our dashboards. It’s worth bookmarking https://govuk-2ndline-dashboard.herokuapp.com/
- The detailed Sidekiq dashboard is not yet available for EKS (the old one still looks at EC2) but you can see queue lengths and age-of-oldest-job on the Sidekiq Grafana dashboard.
- Dashboards are on https://grafana.eks.production.govuk.digital/ and no longer require VPN. Don’t get confused by the old Grafana, which will show mostly empty graphs about the old EC2 setup.
- Most Icinga alerts are irrelevant now, except for:
- “scheduled publications in Whitehall not queued” / “overdue publications in Whitehall”
- “Travel Advice email alert check” / “Medical Safety Email alert check”
- Anything to do with crawler/mirrors
- Anything to do with Licensify
You can fetch the list of pods:
k get pods
This will return a list, including output like:
NAME READY STATUS RESTARTS AGE account-api-568d46d6f7-4pnld 2/2 Running 0 34m account-api-568d46d6f7-txwsx 2/2 Running 0 34m account-api-dbmigrate-6n5mp 0/1 Completed 0 35m account-api-worker-54597b666c-jlcfn 1/1 Running 0 34m asset-manager-7bd745d655-tzlgz 2/2 Running 0 15h ...
As you can see above, each pod either runs the application (we can see two pods running Account API, for example) or runs a worker or cron job for the application (we can see one pod running the Account API application worker).
You can also fetch just a subset list of pods by passing the
$ k get pods -lapp=account-api NAME READY STATUS RESTARTS AGE account-api-75f95899f6-gjdmg 2/2 Running 0 21h account-api-75f95899f6-nn56t 2/2 Running 0 21h
You can fetch the logs for all running containers in a pod. Take one of the pods from the output above:
k logs account-api-568d46d6f7-4pnld
Or you can fetch the logs using its ‘app deployment’ name. This will choose one of the Account API pods at random, much like the old jumpbox methods:
k logs deploy/account-api
Open a rails console
Open a rails console with:
k exec -it deploy/whitehall-admin -- rails c
If you try doing this having assumed a
readonly role when setting context, you’ll see a permissions error:
Defaulted container "app" out of: app, nginx Error from server (Forbidden): pods "whitehall-admin-6bb47c48d-wkkxk" is forbidden: User "christopher.ashton-user" cannot create resource "pods/exec" in API group "" in the namespace "apps"
You need write permissions to be able to open a console (
poweruser or above).
Open a shell
Open a shell (requires
poweruser or above):
k exec -it deploy/whitehall-admin -- bash
Note that the prompt will have prefix that may feel a little unfamiliar:
I have no name!@whitehall-admin-6bb47c48d-wkkxk:/app$.
This only happens for asset-manager and whitehall-admin, because those have to run as a specific user ID that matches what’s on NFS (due to how NFS was set up on the old system).
To open a shell on Router, you’ll need to exec into the nginx container:
k exec -it deploy/router -c nginx -- bash
This is because Router’s image is
FROM scratch, so bash isn’t available.
Run a rake task
Here’s an idempotent rake task you can run to try one out:
k exec deploy/email-alert-api -- rake 'support:view_emails[email@example.com]'
Note that we haven’t specified the
-it flags in the command above because we don’t need to interact with the task.
To find out what they do, run
k exec --help.