Table of contents
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert

Re-create an AWS mongo instance

Re-create an AWS mongo instance


Mongo log file error similar to the below text. can't get local.system.replset config from self or any seed (EMPTYCONFIG) When you access the mongo shell and execute rs.status(), you can only see your instance and no other members of the cluster.


There is a fault with the clustering and it might take a considerable amount of time to identify the root cause. It might be easier to re-create the instance.


In this example we are looking at the integration environment.

  1. You will need the following repositories.

  2. It is very useful to have acces to the AWS console, to observer the changes as they happen.

  3. You can now execute a terraform plan to view the actual module names.

    • $ cd govuk-aws
    • tools/ -c plan -p app-mongo -s blue -d data -e integration Please check the script to identify the keys
  4. The out-put, will show you the components and how they are named by terraform. Initializing modules... - module.mongo-1 Getting source "../../modules/aws/node_group" ....

  5. The instance of intrest to us at this point is mongo-1.

  6. We can now execute a targetted plan to see details related to that specific component and make sure that we are working on the correct instance.

    • tools/ -c plan -p app-mongo -s blue -d data -e integration – -target=module.mongo-1 data.terraform_remote_state.infra_monitoring: Refreshing state... data.terraform_remote_state.infra_security_groups: Refreshing state... data.terraform_remote_state.infra_networking: Refreshing state... null_resource.user_data[0]: Refreshing state... (ID: 8732671785102119409) null_resource.user_data[2]: Refreshing state... (ID: 2147797089831867692) ...
  7. Once you are really sure that you have the correct instance, you can destroy the instance as below.

    • tools/ -c destroy -p app-mongo -s blue -d data -e integration – -target=module.mongo-1
  8. Once the command completes execution, we can execute a terraform apply. This will recreate the deleted instance.

    • tools/ -c apply -p app-mongo -s blue -d data -e integration you can do a targetted apply but it is not necessary

More about Icinga alerts

This page was last reviewed . It needs to be reviewed again by the page owner #govuk-2ndline.