Deploy an application to GOV.UK
2nd line is responsible for:
- ensuring that software is released to GOV.UK responsibly
- providing access to deploy software for teams who can’t deploy it themselves
As far as possible, teams are responsible for deploying their own work. We believe that regular releases minimise the risk of major problems and improve recovery time.
If possible, you should avoid coupling multiple applications so that they all have to be deployed at once. Changes should be backwards compatible except in exceptional circumstances.
We have a staging environment that must always be deployed to immediately before production. Access to staging and production is restricted to certain people.
Only one release takes place at a time. One product team owns the release - if multiple teams are involved in the release, pick one. Releases are tracked by the release app.
You need to have a Signon account with appropriate permissions to access the release app.
As a response to RFC-70 starting 15 May 2017 we are using a process which allows us to deploy without the need for a booked deployment slot.
Deployment communications are in the
#govuk-deploy Slack channel. If you are on 2ndline you should add yourself to that channel. As before, releases can start from 9:30am and must be finished by 5pm, or 4pm on Fridays.
- Check with anyone whose changes you will release during your deploy (check the release app)
#govuk-deployrecent history and the channel topic
- Announce your deployment if it’s potentially problematic
- Deploy to staging
- Check Smokey passes; remember most apps take a couple of minutes to restart
- Check the new functionality works as you would expect
- Deploy to production
- Check Smokey passes in production; once again, apps take a couple of minutes to restart
- Check the new functionality works in production
- Take a look at any alerts and metrics, just to check you haven’t broken something
- Stay around for a while just in case something goes wrong
An alert for the start and end of your deployment will appear in the channel. Jenkins will still enforce sequential deployments per environment across all applications, so you may end up in a queue.
Holding deployment of other applications
If you need to hold deployments of applications during your deploy say so in your announcement post and add it to the channel topic (along with your name). Post again and remove from the topic when you release your hold.
2nd line support
If you need support from 2nd line during your deploy, contact them in advance and agree a time.
If you are responding to a security incident, follow the steps to deploy fixes for a security vulnerability.
Make sure you have a rollback plan if things go wrong. When you’re just changing code, this is relatively easy; when you’re doing data migrations, less so. As far as is possible, all data migrations should be reversible. Don’t rely on backups unless you absolutely have to.
If a release fails in staging and it can be easily and quickly fixed, you don’t need to rollback the change. Just post a message in the
#govuk-deploy Slack channel to let people know what’s going. Then test your fix in integration and redeploy and test it in staging before continuing the deploy to production.
If a release fails in staging and it is not a quick-fix, you should rollback the change and try again later.
We depend on GitHub for deploying software to GOV.UK. We have processes in place to deploy if GitHub is unavailable.
On the blog
- Releasing applications to GOV.UK (older post, using the badger)
More about Deployment
- Block apps from being deployed
- Deploy fixes for a security vulnerability
- Deploy Puppet
- Deploy when GitHub is unavailable
- Fall back to the static mirrors
- Handle encrypted hieradata
- Monitor your app during deployment
- Restart an application
- Run a rake task
- Set up Heroku review apps for pull requests
- Switch an app off