Table of contents
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert

Icinga alerts

Re-create an AWS mongo instance

Re-create an AWS mongo instance


Mongo log file error similar to the below text. can't get local.system.replset config from self or any seed (EMPTYCONFIG) When you access the mongo shell and execute rs.status(), you can only see your instance and no other members of the cluster.


There is a fault with the clustering and it might take a considerable amount of time to identify the root cause. It might be easier to re-create the instance.


In this example we are looking at the integration environment.

  1. You will need the following repositories:
  1. It is very useful to have access to the AWS console, to observer the changes as they happen.

  2. You can now execute a terraform plan to view the actual module names.

   $ cd govuk-aws
   $ tools/ -c plan  -p app-mongo -s blue -d data -e integration # Please check the script to identify the keys
  1. The out-put, will show you the components and how they are named by terraform.
  Initializing modules...
  - module.mongo-1
  Getting source "../../modules/aws/node_group"
  1. The instance of interest to us at this point is mongo-1.

  2. We can now execute a targeted plan to see details related to that specific component and make sure that we are working on the correct instance.

   $ tools/ -c plan -p app-mongo -s blue -d data -e integration -- -target=module.mongo-1

   data.terraform_remote_state.infra_monitoring: Refreshing state...
   data.terraform_remote_state.infra_security_groups: Refreshing state...
   data.terraform_remote_state.infra_networking: Refreshing state...
   null_resource.user_data[0]: Refreshing state... (ID: 8732671785102119409)
   null_resource.user_data[2]: Refreshing state... (ID: 2147797089831867692)
  1. Once you are really sure that you have the correct instance, you can destroy the instance as below.
   $ tools/ -c destroy -p app-mongo -s blue -d data -e integration -- -target=module.mongo-1
  1. Once the command completes execution, we can execute a terraform apply. This will recreate the deleted instance.
   tools/ -c apply  -p app-mongo -s blue -d data -e integration # you can do a targeted apply but it is not necessary
This page was last reviewed on 22 March 2019. It needs to be reviewed again on 22 September 2019 by the page owner #govuk-2ndline .
This page was set to be reviewed before 22 September 2019 by the page owner #govuk-2ndline. This might mean the content is out of date.