Skip to main content
Last updated: 9 Nov 2026

Add Missing User Events to Site Search

We capture interaction data from opted-in users in Google Analytics. This data is processed and ingested into Discovery Engine each day, and feeds into autocomplete suggestions and click-based search result ranking.

Sometimes the GA4 data is delayed. If it’s not received before our dataform pipelines run at 12:00 then it will not be imported into Discovery Engine.

A Sentry alert will notify us if the count of events is below expected thresholds. We should manually import the missing user events.

If user events data is missing

Check which data is missing

  1. Run a query in BigQuery Studio to check which tables are affected.

    1. You will need to run this for the search-event, view-item-event & view-item-external-link-event tables, and should check all environments in case they are not in the same state as each other.
    -- Example query to check number of rows of data in BigQuery

SELECT COUNT(*)
FROM `search-api-v2-production.analytics_events_vertex.search-event`
WHERE TIMESTAMP_TRUNC(_PARTITIONTIME, DAY) = TIMESTAMP("2026-08-24")

  2. Compare the results to the minimum_events_thresholds defined in Search API v2.

Confirm that the GA4 data has now been received

Check in #govuk-ga4 slack channel to confirm that GA4 data has been received.

Note: There is no point in re-running the dataform pipelines until the GA4 data has been received.

Re-run the dataform pipelines for the tables that are missing data

You will need to re-run the dataform pipelines in all environments for the tables that are missing data.

  1. Navigate to the search-daily- pipeline in GCP (e.g. search-daily-production)
  2. Click on the date of the run that is missing data
    1. (Note: The query is fairly stable, so it won’t usually matter which date you choose. Picking the date of the failed run will prevent mishaps, though.)
  3. For each table that is missing data
    1. Click ‘view details’ for the row whose action matches the table name
    2. Click on the ‘Job ID’ (to open an editable version of the query in the console)
    3. Adjust the query to include the appropriate date (replace DATE_SUB(CURRENT_DATE(),INTERVAL 1 DAY) with date in format '2026-01-31'
    4. Run query

Re-run the rake task to import user events to Discovery Engine

You will need to re-run the rake task in all environments.

rake "user_events:import_events_for_date['<date>']"

Background information

Google recommends uploading user events at least daily to maintain good data quality.

The process by which the data is ingested is as follows:

  1. GA4 user interaction data is processed overnight by the GOV.UK analytics pipeline and ingested into GCP BigQuery.

  2. The interaction data is transformed, via dataform pipelines, into a format that can be ingested by Discovery Engine.

  3. The transformed data is imported into Discovery Engine as “user events”, via automated Rake tasks.

How Google Agent Search uses user events

Google recommends recording user events for custom search and recommendations apps.

Improve quality of search results

By importing user events, you can improve the quality of your search results. For example, user events affect the ranking of search results. Results with higher click-through rates are boosted, while results with lower click-through rates are buried.

Generate autocomplete suggestions

The user events model generates suggestions from user-imported events of type search.

Related repositories

More in the Search on GOV.UK section

Learn

How to...