Re-create an AWS mongo instance
Symptom
Mongo log file error similar to the below text:
can't get local.system.replset config from self or any seed (EMPTYCONFIG)
When you access the mongo shell and execute rs.status()
, you can only see
your instance and no other members of the cluster.
Prognosis
There is a fault with the clustering and it might take a considerable amount of time to identify the root cause. It might be easier to re-create the instance.
Solution
In this example we are looking at the integration
environment.
- You will need the following repositories:
- https://github.com/alphagov/govuk-aws
- https://github.com/alphagov/govuk-aws-data
- https://github.com/alphagov/govuk-secrets
It is very useful to have access to the AWS console, to observe the changes as they happen.
You can now execute a
terraform plan
to view the actual module names.
$ cd govuk-aws
$ tools/build-terraform-project.sh -c plan -p app-mongo -s blue -d data -e integration # Please check the script to identify the keys
- The output, will show you the components and how they are named by
terraform
.
Initializing modules...
- module.mongo-1
Getting source "../../modules/aws/node_group"
....
The instance of interest to us at this point is mongo-1.
We can now execute a targeted plan to see details related to that specific component and make sure that we are working on the correct instance.
$ tools/build-terraform-project.sh -c plan -p app-mongo -s blue -d data -e integration -- -target=module.mongo-1
data.terraform_remote_state.infra_monitoring: Refreshing state...
data.terraform_remote_state.infra_security_groups: Refreshing state...
data.terraform_remote_state.infra_networking: Refreshing state...
null_resource.user_data[0]: Refreshing state... (ID: 8732671785102119409)
null_resource.user_data[2]: Refreshing state... (ID: 2147797089831867692)
...
- Once you are really sure that you have the correct instance, you can destroy the instance as below.
$ tools/build-terraform-project.sh -c destroy -p app-mongo -s blue -d data -e integration -- -target=module.mongo-1
- Once the command completes execution, we can execute a
terraform apply
. This will recreate the deleted instance.
$ tools/build-terraform-project.sh -c apply -p app-mongo -s blue -d data -e integration # you can do a targeted apply but it is not necessary