Skip to main content
Last updated: 16 Jan 2023

2nd line drills

There are a number of areas that are important to drill on 2nd line and include some tasks you may not necessarily encounter in your mission team. We want to ensure developers have the opportunity to practise these tasks ahead of the real thing and in preparation of going on call if you are part of the out of hours rota.

Drill detaching an instance

Follow the Detaching an instance from an Auto Scaling Group guidance.

Drill publishing emergency banner

Follow the Deploy an emergency banner on Staging.

You’ll need to choose a non-serious and clearly fake news headline. For example:

  • CAMPAIGN_CLASS: Death of a notable person
  • HEADING: Henry Fielding dies
  • SHORT_DESCRIPTION: English novelist and dramatist known for his earthy humour and satire dies, age 47
  • LINK: https://en.wikipedia.org/wiki/Henry_Fielding
  • LINK_TEXT: More information

Drill an end to end incident

Decide on a hypothetical incident scenario, e.g. “GOV.UK is down”. Walk through the incident management guidance. Use common sense when following the steps (i.e. don’t actually publish an incident to Statuspage or email stakeholders).

Deploy from AWS CodeCommit when Github is unavailable

Follow the Deploy when GitHub is unavailable instructions.

Run a Terraform plan

Follow the Deploy Terraform instructions, picking a project at random. You can run this in any environment, as you’re only running plan - not apply - so shouldn’t be making any changes.

Update homepage promotion slots

Follow the Update homepage promotion slots instructions, using an appropriate image and text. Open a draft PR, and deploy your branch to integration. Once deployed, check your change and redeploy the previous branch to integration.

Use a restored database in an app

On Integration or Staging, follow the Restore an RDS instance via the AWS CLI instructions for an app of your choice.

Force failover to GOV.UK mirror and Emergency publishing using the GOV.UK mirror

  1. Warn in #govuk-2ndline-tech that you’re about to do this, as it will lead to a spike in alerts and will also break continuous deployment for a while (due to Smokey failures).
  2. Follow the Forcing failover to the GOV.UK mirrors instructions on Integration or Staging.
  3. To verify that it worked, visit a page at random and purge the page from cache. Reload the page, to see the ‘mirrored’ version of the content. NB: you wouldn’t do this in a real incident, as we’d want to serve Fastly’s cached version for as long as possible.
  4. Undo your changes to have Nginx handling requests again.

Drill logging into accounts

Make sure you can log into the following accounts:

  1. The AWS Console
  2. Your individual Fastly account
  3. Your individual Statuspage account
  4. Your individual Logit account
  5. Shared Heroku account
  6. Shared CKAN account
  7. Shared Rubygems account
  8. Shared NPM account
  9. Shared data.gov.uk publisher account

Drill how to communicate when Slack is down

Ensure you know how to communicate with your 2nd line colleagues if Slack is unavailable. See “If Slack is unavailable” for details.

Drill scaling up number of workers

In preparation for a spike in traffic, you can increase the number of unicorn workers for an app.

Pick an application (e.g. smartanswers) and drill scaling up the number of workers, by following scale unicorn workers.

Note that you’ll need to edit the integration.yaml file, not the production.yaml file as described in the docs above. And instead of merging to main, just build your branch directly to Integration. When you’re done drilling, re-deploy the previous release and delete your branch.

Drill enabling a code freeze

Choose a continuously-deployed app where you can make a meaningful change to the default branch, e.g. fixing a typo, or merging a Dependabot PR.

Either before merging the change, or part way through the continuous deployment process, follow the instructions for implementing a deploy freeze for that app.

Follow the deployment pipeline in Jenkins. Confirm that no further environment deployments are triggered. For example, if you implemented the deploy freeze just after the app was deployed to Staging, confirm that the app was then not automatically deployed to Production.

Remove the code freeze, then manually push the changes to all remaining environments so that they’re in sync.

Drill getting a password out of govuk-secrets

Most of our shared accounts are stored in this repo. Follow the steps to get a password, a useful one for 2nd line is the Technical 2nd line Dashboard which shows a high level overview of the state of GOV.UK environments. Upon loading this dashboard it will ask for the govuk-zendesk-display-screen credentials.

Drill making changes to user accounts

Assign a user to their publisher in data.gov.uk

Log into our shared data.gov.uk publisher account. Pick a publisher to do a hypothetical walk though of Assign users to publishers.

Change a user’s permissions in Signon

Carry out a hypothetical walk through of unsuspending a user and resetting a user’s 2FA.

Drill creating and changing redirects

Redirect a route

On Integration or Staging, follow the Removing a route created in the Short URL Manager and Removing a route completely so it can be replaced with another route instructions.

Change a slug and create a redirect

On Integration or Staging, follow the Change a slug and create redirect in Whitehall, picking something at random in Whitehall from one of the group of entities listed (people, role, organisation, etc).

Drill modifying a document’s change note

Modify and remove a document’s change note in Whitehall

On Integration or Staging, follow Modify a change note in Whitehall using this document or one of your choice. Once you have successfully updated the change note you can drill removing a change note in Whitehall.

Modify and remove a document’s change note in Content Publisher

On Integration or Staging, follow Modify a change note in Content Publisher using this document. The 30th November 2021 shows a bespoke change note which you could try changing - click “show all updates” at the bottom of the page.

You can also try deleting the change note. Again, ensure you do this on Staging or Integration.

Modify a document’s change note in Publishing API

On Integration or Staging, follow Modify a change note in Publishing API using this document or one of your choice. Once you have successfully updated the change note you can drill removing a change note in Publishing API.

Change the homepage popular links following Update popular links. Open a draft PR, and deploy your branch to integration. Once deployed, check your change and redeploy the previous branch to integration.