Skip to main content
  1. Home
  2. Deployment
Last updated: 24 May 2023

Deploy Puppet

Building a release

Jenkins should normally build govuk-puppet automatically on push to any branch, but unfortunately this is currently broken (May 2023).

To work around this, go to govuk-puppet in CI Jenkins and choose Scan Repository Now from the left-hand column.

You will usually need to do this twice: once to make the pre-merge checks run against your branch and again to build the release after you have merged your PR.

Deploying a branch

Before merging your PR, it’s recommended that you deploy the change to Integration in order to test it works on real infrastructure. Go to the Deploy_Puppet job in Jenkins and click ‘Build with Parameters’. The TAG field should contain the name of the branch you wish to deploy.

Deploying a release

You can deploy Puppet to Staging and Production using the following steps:

  1. In the Release app, select the release tag and review the changes listed.
  2. If your change on integration appears as expected, deploy a newer version of govuk-puppet to staging by using the Deploy to Staging button.
  3. Wait 30 minutes or read about convergence. You should monitor Staging Icinga and Staging Smokey, and test anything you’re concerned about.
  4. Repeat the above steps for Production.

Puppet is automatically deployed to integration by a combination of the integration-puppet-deploy job on Jenkins CI and Deploy Puppet job on Jenkins Deploy jobs.

WARNING

If you’re deploying a change to hiera.yml or hiera_aws.yml, you will need to restart the Puppet server on the Puppet Master machine, otherwise these changes will not be picked up.

To restart the server, run sudo service puppetserver restart.

Convergence

The deployment only pushes the new code to the Puppet Master. Each node runs a Puppet agent every 30 minutes (via cron), so it may be some time before the release takes effect. This has an implication on how quickly you can go from staging to production.

If you’d rather not wait and you’re able to safely determine from the diff what classes of machines the change will affect, you can force a run of puppet on the relevant machine classes.

This will run in serial across the nodes so there is a reduced chance of downtime caused by a service restarting on all nodes of a given class at the same time. You should still be careful though, because some services take longer to restart than others.

Preventing service restarts

It may occassionally be neccessary to trick Puppet into not restarting a service, if it is a single point of failure and doing so would cause a brief outage, e.g. MySQL.

WARNING

This is not a “normal” procedure. You should only do this if you need to and you MUST have some plan for restarting the service in the near future so that it’s not inconsistent with its configuration.

  1. Find out which machine class the service is running on

  2. Disable puppet on all machines of that class

  3. On each machine, change the file content to match what Puppet wants it to be. If it’s a plain file you can probably apply the diff from git using sudo patch source.diff dest. If it’s a template then you may need to refer to an existing environment or figure it out yourself.

  4. Verify that Puppet won’t change the file or notify the service by running it in noop mode. ou will need to provide a different lock path to bypass the disable:

govuk_puppet --noop --test --agent_disabled_lockfile=/tmp/puppet.noop

  1. If you’re happy with the results then re-enable Puppet on all of the machines

  2. Schedule a time to actually restart the service if necessary.

Instructions on how to disable and enable puppet can be found here.

More in the Deployment section

Learn

How to...