Replaying traffic to correct an out-of-sync search index
If the data in the search index is out-of-sync with the Publishing API,
(for example, after restoring a backup),
unpublish messages that have not been processed need to be resent.
Content in the
govuk index is populated from the Publishing API message queue.
Missing documents can be recovered by resending the content to the message
queue, for example by running
represent_downstream:document_type[:document_type] rake task in Publishing
This will not be neccessary after whitehall content has been moved to the
These indexes are populated by whitehall calling an HTTP API in Rummager.
We have also setup GOR logging for
GET requests so that we can replay the traffic.
The logs are stored on the rummager servers (1 file per server) location at:
A copy of the file should be taken for the restore, as the restore requests will be logged to the file. The following command can be used to run the restore:
$ sudo gor --input-file "20171031.log|6000%" --stats --output-http-stats --output-http "http://localhost:3009/|6000%" -verbose
This runs the restore at 60x the speed it was saved so each hour of logs takes 1 minute to process.
This process failed when running on the rummager server, but was successful when run locally with port forwarding. This may be an issue with the
GORversion on the server.