Application: search-api

Search API for GOV.UK

• GitHub: search-api
• Ownership: #govuk-platform-health
• Hosting: AWS, on the search machine class
• Continuously deployed?: No
• Rake tasks: Run a Rake task on Integration, Staging or ⚠️ Production ⚠️
• Links:
  • Sentry (error tracking)
  • Puppet configuration
  • Deploy scripts
  • Deployment dashboard

Relevant manual pages

• Content that doesn't show up correctly in search or list pages
• Make a new document type available to search
• Reindex an Elasticsearch index

Imported documents

• Adding new fields to a document type
• Content API
• Content in the search index and where it comes from
• Documents API (to be deprecated)
• How Search Works
• Indexing
• learning-to-rank
• Moving a format to the new indexing process
• Popularity information
• Publishing finders
• Relevancy
• Schema definitions
• Using the search API
• Search Quality Metrics

Example pages published by search-api

• Search
• News and communications
• Services
• Research and statistics
• Guidance and regulation
• Policy papers and consultations
• Transparency and freedom of information releases
• Official documents

README

Search API (née "rummager") indexes content into Elasticsearch and serves the GOV.UK Search API.

GOV.UK applications use the API to search and filter GOV.UK content. For example, alphagov/finder-frontend uses the search API to render site search and finder pages (such as gov.uk/aaib-reports).

Search API also provides a public API: https://www.gov.uk/api/search.json?q=taxes.

API documentation

If you would like to use the Search API, please see the Search API documentation.

You can also find some examples in the blog post: "Use the search API to get useful information about GOV.UK content".

Getting started

The instructions will help you to get Search API running locally on your machine.

Prequisites

Install govuk-docker!

govuk-docker, a wrapper around docker-compose, is the supported way to run Search API and its dependencies locally.

Once you have installed govuk-docker, run

cd ~/govuk/govuk-docker && make search-api

Running the application

Once you have completed the prerequisites you'll be able to run Search API locally by running

cd ~/govuk/search-api && govuk-docker up search-api-app

This starts the Search API application and its dependencies.

The Search API will be running locally at search-api.dev.gov.uk.

If you run docker ps this will tell you that there are containers running for Search API, Nginx, Redis, Publishing API, and Elasticsearch.

Note: If you're not using docker, you can run ./startup.sh to start the application. However, this is not officially supported, and you will need to run dependencies such as Elasticsearch and Redis yourself.

Replicating data locally

If you've started running Search API for the first time you probably aren't seeing any search results for your queries. That's likely because your local search indexes will be empty.

Once you have got everything running locally, another step most people take is to get a copy of the search indexes running locally. This will let you search for real documents on your local machine.

See the govuk-docker documentation on replicating data, or run

gds aws govuk-integration-poweruser ./bin/replicate-elasticsearch.sh

Refer to the govuk-docker documentation for more details about the replication scripts.

Running the test suite

Complete the prerequisites, then run

govuk-docker run search-api-lite bundle exec rake

Note: You can also run the tests without using docker, by running bundle exec rake. This is not officially supported, but can be a quick way to run unit tests.

Technical documentation

Search API is a Sinatra application that interfaces with Elasticsearch.

Search API puts documents into Elasticsearch indexes (index time), and serves documents in search results (query time).

It does some clever stuff at both parts, but that's the meat of it.

Read the documentation to find out how documents are indexed or how documents are retrieved.

Dependencies

Search API depends on other services in order to index documents and provide relevant search results:

• Elasticsearch - "You Know, for Search...".
• Redis - used by indexing workers.
• AWS Sagemaker (optional) - used for search relevancy

If you use govuk-docker locally, the required dependencies will be started automatically when you start Search API. You don't need to set these up yourself.

See the learning to rank documentation for guidance on how to run the ranking model locally.

Additional Docs

• New indexing process: how to update a format to use the new indexing process
• Schemas: how to work with schemas and the document types
• Popularity information: Search API uses Google Analytics data to improve search results.
• Publishing document finders: Information about publishing finders using rake tasks

Licence

MIT License