Skip to main content
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert
Warning This document has not been updated for a while now. It may be out of date.
Last updated: 5 Apr 2022

Re-create an AWS mongo instance

Symptom

Mongo log file error similar to the below text:

can't get local.system.replset config from self or any seed (EMPTYCONFIG)

When you access the mongo shell and execute rs.status(), you can only see your instance and no other members of the cluster.

Prognosis

There is a fault with the clustering and it might take a considerable amount of time to identify the root cause. It might be easier to re-create the instance.

Solution

In this example we are looking at the integration environment.

  1. You will need the following repositories:

  1. It is very useful to have access to the AWS console, to observe the changes as they happen.

  2. You can now execute a terraform plan to view the actual module names.

   $ cd govuk-aws
   $ tools/build-terraform-project.sh -c plan  -p app-mongo -s blue -d data -e integration # Please check the script to identify the keys
  1. The output, will show you the components and how they are named by terraform.
  Initializing modules...
  - module.mongo-1
  Getting source "../../modules/aws/node_group"
  ....

  1. The instance of interest to us at this point is mongo-1.

  2. We can now execute a targeted plan to see details related to that specific component and make sure that we are working on the correct instance.

   $ tools/build-terraform-project.sh -c plan -p app-mongo -s blue -d data -e integration -- -target=module.mongo-1

   data.terraform_remote_state.infra_monitoring: Refreshing state...
   data.terraform_remote_state.infra_security_groups: Refreshing state...
   data.terraform_remote_state.infra_networking: Refreshing state...
   null_resource.user_data[0]: Refreshing state... (ID: 8732671785102119409)
   null_resource.user_data[2]: Refreshing state... (ID: 2147797089831867692)
   ...
  1. Once you are really sure that you have the correct instance, you can destroy the instance as below.
   $ tools/build-terraform-project.sh -c destroy -p app-mongo -s blue -d data -e integration -- -target=module.mongo-1
  1. Once the command completes execution, we can execute a terraform apply. This will recreate the deleted instance.
   $ tools/build-terraform-project.sh -c apply  -p app-mongo -s blue -d data -e integration # you can do a targeted apply but it is not necessary