This gem retrieves a list of seed URLs from the GOV.UK sitemap and adds them to RabbitMQ
so that the crawler can consume them.
Add this line to your application’s Gemfile:
And then execute:
Or install it yourself as:
$ gem install govuk_seed_crawler
To run with the RabbitMQ connection defaults:
bundle exec seed-crawler https://www.gov.uk/
Run with --help to see a list of options:
bundle exec seed-crawler --help
git checkout -b my-new-feature
git commit -am 'Add some feature'
git push origin my-new-feature