Replay traffic to correct an out-of-sync search index
If the data in the search index is out-of-sync with the Publishing API,
(for example, after restoring a backup), then any
unpublish messages that have not been processed need to be resent.
Content in the
govuk index is populated from the Publishing API message queue.
Missing documents can be recovered by resending the content to the message queue. In the
Publishing API, run the following rake task (including the quotes) to replay traffic between
bundle exec rake 'represent_downstream:published_between[2018-12-17T01:02:30, 2018-12-18T10:20:30]'
Other replay options are available, for example replaying all traffic for a single publishing app or doctype. Be aware that these options will replay the entire Publisher API history for that app or doctype, and may take some time.
This will not be neccessary after whitehall content has been moved to the
These indexes are populated by whitehall calling an HTTP API in Search API.
We have also setup Gor logging for
GET requests so that we
can replay the traffic.
The logs are stored on the search-api servers. You will need to run the replay on each server.
The location of the logs is:
You must copy the file for the restore, as the restore requests will be logged to the file.
The following command can be used to run the restore:
$ sudo goreplay -input-file "20171031.log|1000%" -stats -output-http-stats -output-http "http://localhost:3233/|6000%" -verbose
This runs the restore at 10x the speed it was saved so each hour of logs takes 6 minutes to process.