Skip to main content
Warning This document has not been updated for a while now. It may be out of date.
Last updated: 9 Jun 2020

search-api: Popularity information

The search uses page popularity information extracted from Google Analytics as one of the factors in weighting search results. This is extracted from Google Analytics by the search-analytics project, but for dev machines, you should be able to obtain a copy of the page traffic index from preview when you run the standard replication of search indexes from preview to dev.

If you do need to fetch the analytics data directly yourself, the search-analytics project README describes how to set up and run the extraction of page traffic information from Google Analytics. It will produce a dump file suitable for loading into an elasticsearch index using the page_traffic_load tool.

Loading popularity data

Once you have the popularity data in a file named, say, page-traffic.dump, load it into elasticsearch using:

bundle exec bin/page_traffic_load page-traffic < page-traffic.dump

The popularity information won't affect search results until the search:update_popularity rake task is run:

PROCESS_ALL_DATA=true SEARCH_INDEX=all bundle exec rake search:update_popularity

This creates a lot of Redis jobs, so the script runs the task for one index at a time, with a delay between each.

The popularity information will also be applied when an index migration is run:

SEARCH_INDEX=all bundle exec rake search:migrate_schema

How popularity is computed

Popularity is computed by Indexer::PopularityLookup, and is determined by this formula:

if ranks[link] == 0
  popularity_score = 0
  popularity_score = 1.0 / (ranks[link] + SearchConfig.popularity_rank_offset)

Where ranks[link] gives the ranking of the link in order of number of page views: the most viewed page is rank 1, the second most vewed page is rank 2, and so on.

The popularity_rank_offset is determined by the Elasticsearch configuration file.

How popularity affects results

Popularity is applied as a multiplier to a document's score. This is added by QueryComponents::Popularity.

The POPULARITY_OFFSET value is added to the popularity score before multiplying, to ensure that a document with no popularity won't have a score of zero. Without this, all such documents would have the same score, and so would be ranked the same by Elasticsearch, regardless of how well they match the query.