The development and deployment pipeline
When responding to a security incident, you should make any code changes in private, so that you don’t accidentally disclose the vulnerability. To do this, follow the instructions for making a repo private.
Our development and deployment pipeline looks like this:
- Open a Pull Request (PR)
- Wait for Continuous Integration to pass
- Review your own changes
- Get someone to review your Pull Request
- Check for or implement a deploy freeze
- Merge your own Pull Request
- Deploy through each of the environments
Wait for Continuous Integration to pass
When a Pull Request (PR) is opened, it often triggers an automated job, which typically lints the code and runs the tests.
Review your own changes
As well as double-checking your code and commits, you may refer to the following:
Heroku App Review
Sometimes you may need to demo a running version of your change. All frontend applications and some publishing apps have Heroku Review Apps enabled, with a link near the bottom of each PR.
Branch Deploy Review
Sometimes you may need to deploy your change in Integration in
order to test it works on real infrastructure. Go to
Deploy_App job in Jenkins
and click ‘Build with Parameters’:
TARGET_APPLICATION- the name of the repository you want to deploy
DEPLOY_TASK- usually ‘deploy’ is most appropriate
TAG- put the name of your branch
- Typically you can leave the checkboxes as they are
Get someone to review your Pull Request
The GDS Way has guidelines on how to review code.
Only when the tests pass and the code has been approved the Pull Request can be merged, since we’ve configured GitHub to prevent merges otherwise.
Check for or implement a deploy freeze
In exceptional circumstances, we may wish to block or freeze deployments for a short period of time. This should be done by checking “Freeze deployments?” and adding an explanatory note to the application in the Release app.
Checking “Freeze deployments?” will turn off all automatic deployments for the application. You can still deploy urgent changes manually if necessary.
It is important to ensure people are aware of a code freeze.
Send a message to #govuk-developers on slack with the @channel prefix (to ensure people who are offline are notified) and email email@example.com. Your message should include the repo you are freezing, the reason why, and the expected duration. Follow up to let people know when the freeze is lifted.
When a deploy freeze is in effect, you should avoid merging any PRs. This is because your changes may block other, urgent changes related to the deploy freeze. Your changes will also remain undeployed for a long time.
Merge your own Pull Request
We’ve got guidelines on merging of Pull Requests.
Code that is merged to
main is tested again on CI. This is because
main branch may have changed since the tests last ran on the PR.
If the tests on
main pass, Jenkins pushes a
release_123 git tag to
WARNING: some applications have Continuous Deployment enabled, which means the deployment process is fully automated. You should do any manual testing with a temporary, branch deployment before you merge.
Teams are responsible for deploying their own work. We believe that regular releases minimise the risk of major problems and improve recovery time. The Technical 2nd Line team is responsible for providing access to deploy software for teams who can’t deploy it themselves.
- Check the notes in the Release app to see if Continuous Deployment is enabled.
- If so, after merging, you should check the Release app to see if the deployment succeeds.
- If the latest release is not on Production within about 15 minutes, something went wrong:
Wait for the release to deploy to Integration
When a new release is created, CI sends a message to Integration Deploy Jenkins to deploy the tag. You should verify your changes work in Integration before deploying downstream:
- Run a build of smoke tests.
- Look for any Icinga alerts related to your application.
Our apps should always be in a state where
main is deployable. You
should raise a PR to revert your changes if they cause a problem and
you’re unable to resolve that problem straight away.
Manually deploy to Staging, then Production
Deployments to these environments are manual and require production access. Go to the Release application and find the application you want to deploy, then select the release tag you want to deploy. Follow these rules:
- Deployments should generally take place between 9.30am and 5pm (4pm on Fridays), the core hours when most people are in the office.
- If there’s other people’s code to deploy, ask them whether they’re okay for the changes to go out.
- Announce in
#govuk-2ndline-techif you anticipate your release causing any issues. Stay around for a while just in case something goes wrong.
- Check the Release app for a deploy note for the application, to see if there are any special instructions or reasons not to deploy. Individual app deploy freezes are usually announced here.
After a deployment: