Fix Mapit database not available
Each Mapit machine has its own PostgreSQL database which powers the Mapit application. When the machine is first created by AWS, the latest data is supposed to be imported into the database.
This feature is currently not working which means that new Mapit machines are not added to the load balancer until the data has been imported manually. There will also be numerous errors in Icinga related to 5xx errors and events in Sentry:
OperationalError: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432?
RE are aware that the automatic import process doesn't work.
SSH onto the machine which is experiencing the problem and run the following comments:
Stop any running PostgreSQL instance
$ ps aux | grep postgresql $ sudo kill -9 <pid>
Recreate the PostgreSQL cluster
$ sudo pg_dropcluster --stop 9.6 main; sudo pg_createcluster 9.6 main
$ govuk_puppet --test
Check that PostgreSQL is up and running correctly
$ sudo service postgresql status
Make the import script available
$ wget https://raw.githubusercontent.com/alphagov/govuk-puppet/master/modules/govuk/files/etc/govuk/import_mapit_data.sh $ chmod +x import_mapit_data.sh
Run the import script
$ sudo ./import_mapit_data.sh
The import script will take about 10 minutes to run. Sometimes your SSH connection might time out while the script runs, it's worth checking on the terminal every few minutes to check that hasn't happened. Once Mapit is healthy, the AWS load balancer will automatically detect this and add the machine into the rotation.