Remove a machine
Before you start, make sure that the machines you are removing are no longer needed for anything.
Removing a class of machine
If you are removing a class of machines, you will need to remove the definitions from Puppet.
1. Remove from Terraform
Firstly, deploy Terraform for the relevant project using the destroy action. This will remove all the EC2 instances.
Then, remove the project itself from govuk-aws.
If you’re removing a single machine, change the asg_size for the
relevant project in govuk-aws-data
(example) and deploy Terraform.
It is not possible to disable alerts for the machine before
it is removed, as the ASG will terminate arbitrary instances to shrink
to the desired size.
If there are particular machines in the ASG that you do not want terminated you can enable “Scale in Protection” on them before you reduce the size of the ASG. You can find more information on this in the AWS Docs.
2. Remove the node from the puppetmaster
Icinga will forget about the machine after Puppet runs on the puppetmaster and then on the monitoring machine, which could be up to an hour. You can force this by running Puppet on the puppetmaster:
$ gds govuk connect -e <environment> ssh puppetmaster
$ govuk_puppet --verbose
And then on the monitoring machine:
$ gds govuk connect -e <environment> ssh monitoring
$ govuk_puppet --verbose