Skip to main content
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert
Warning This document has not been updated for a while now. It may be out of date.
Last updated: 4 May 2022

Search API index size has significantly increased/decreased

This alert indicates that a large number of documents have been added to or removed from the search-api search index.

It compares the current size of the index with the size from one week ago, and triggers the alert if the difference is more than the thresholds configured in puppet.

Investigation

The Icinga alert should say which index is affected.

Check the index size dashboard to see how many documents have been affected.

If the dashboard shows that the index is currently empty:

  • Check the cluster health to make sure that the Elasticsearch cluster is available and has the expected number of documents (around 300,000 in total across the indices).
  • Check that the search_index_checks Jenkins job has been run recently. It should run regularly to populate the Graphite data which powers the check, but if Jenkins is busy then sometimes it cause missing Graphite data which incorrectly triggers the alert.

If the index size has been growing steadily over the course of several hours, the change may be due to a large number of documents being published. This means that the index size has genuinely grown, so the alert can be ignored.

If the number of documents has increased very suddenly, this may be because content has been accidentally duplicated.

If there does appear to be a real problem with search, see the cluster health investigation guide for help.