Common AWS tasks for Technical 2nd Line
This document details some of the tasks that GOV.UK Technical 2nd Line may carry out regarding AWS.
Logging into the AWS web console
Once you’ve set up AWS access, you can log into the AWS console for the relevant environment by running:
gds aws govuk-<environment>-<role> -l
See these notes about the different AWS IAM roles e.g.
gds aws govuk-integration-readonly -l
It will then ask you to supply your aws-vault password, followed by your 2FA code.
Getting help with AWS
Read the docs
The AWS docs are comprehensive, and the GOV.UK developer docs explain how to do common tasks.
Raise an AWS support request
We pay AWS for Premium Support. You are strongly encouraged to contact AWS to help you solve problems when using AWS products. Contacting AWS Support is a very common procedure.
See our documentation on how to escalate to AWS support
Internal support
Usually, 2nd-line Tech Support should be able to investigate issues related to AWS.
During working hours, one of the Site Reliability Engineers (SREs) on the Platform Engineering or Platform Security and Reliability teams may be able to provide advice or expert knowledge. Outside office hours, you should escalate to AWS Support if the engineers on call can’t resolve an issue themselves.
If you are experiencing an incident, refer to the So, you’re having an incident documentation.
Troubleshooting
How to view ALB metrics
You can see metrics for load balancers in CloudWatch. See the AWS documentation on load balancer CloudWatch metrics for more detail.
How to query Athena logs
How to identify AWS managed DB performance issues
If you find an AWS managed DB is experiencing performance issues, it’s often worth having a look at the CloudWatch metrics for the service. This might tell you which resource is the limiting factor impacting performance.
It is also worth looking at AWS’s troubleshooting documentation, such as the DocumentDB documentation.
How to restore an AWS managed DB from a backup
View the documentation on how to backup and restore in AWS RDS.
How to resize a persistent disk
If you’re not sure how to do this, ask an Site Reliability Engineer to give you a walk through.
See the docs: /manual/manually-resize-ebs.html.
How to replace an instance with a persistent disk
If you’re not sure how to do this, ask an Site Reliability Engineer to give you a walk through.
Assuming we want to destroy and recreate a VM, but have the new VM attach to the old persistent disk
- Find the VM and note down the details of the persistent volume you want to keep
- Shutdown the VM
- Detach the EBS volume from the VM.
- Delete the VM
- Run terraform apply which will now recreate the VM
- Manually re-attach the EBS volume to the new VM
- Reboot the machine to make sure the persistent disk reattaches between reboots
- Initiate a puppet run to make sure that everything works
Learn
How do we do DNS?
GOV.UK is effectively a DNS registrar for some third-level domain names, for example service.gov.uk.
See how GOV.UK does DNS.
How are EC2 instances (legacy infrastructure) provisioned?
As of Mar 2023, only Crawler, CKAN (for data.gov.uk) and Licensing still use the legacy EC2 infrastructure.
The legacy EC2 infrastructure is configured via Terraform code in the govuk-aws repo.
There are a few exceptions to this, such as ad-hoc instances started from Concourse via the AWS CLI - these are mainly for data science projects.
We use userdata scripts to run commands on our instances at launch. These scripts install various core bits of software needed by a particular instance and then typically use govuk-puppet to provision our instances.
Finally, new instances send Jenkins their Fully Qualified Domain Name (FQDN) and puppet class. Jenkins automatically deploys apps to newly provisioned instances.