Re-create an AWS mongo instance
Mongo log file error similar to the below text:
can't get local.system.replset config from self or any seed (EMPTYCONFIG)
When you access the mongo shell and execute
rs.status(), you can only see
your instance and no other members of the cluster.
There is a fault with the clustering and it might take a considerable amount of time to identify the root cause. It might be easier to re-create the instance.
In this example we are looking at the
- You will need the following repositories:
It is very useful to have access to the AWS console, to observe the changes as they happen.
You can now execute a
terraform planto view the actual module names.
$ cd govuk-aws $ tools/build-terraform-project.sh -c plan -p app-mongo -s blue -d data -e integration # Please check the script to identify the keys
- The output, will show you the components and how they are named by
Initializing modules... - module.mongo-1 Getting source "../../modules/aws/node_group" ....
The instance of interest to us at this point is mongo-1.
We can now execute a targeted plan to see details related to that specific component and make sure that we are working on the correct instance.
$ tools/build-terraform-project.sh -c plan -p app-mongo -s blue -d data -e integration -- -target=module.mongo-1 data.terraform_remote_state.infra_monitoring: Refreshing state... data.terraform_remote_state.infra_security_groups: Refreshing state... data.terraform_remote_state.infra_networking: Refreshing state... null_resource.user_data: Refreshing state... (ID: 8732671785102119409) null_resource.user_data: Refreshing state... (ID: 2147797089831867692) ...
- Once you are really sure that you have the correct instance, you can destroy the instance as below.
$ tools/build-terraform-project.sh -c destroy -p app-mongo -s blue -d data -e integration -- -target=module.mongo-1
- Once the command completes execution, we can execute a
terraform apply. This will recreate the deleted instance.
$ tools/build-terraform-project.sh -c apply -p app-mongo -s blue -d data -e integration # you can do a targeted apply but it is not necessary