SearchAPI and Opensearch alerts and monitoring
As described in GOV.UK Search: How it works there are two search stacks on GOV.UK. This page includes information on the monitoring and alerting for Search API and its Opensearch cluster.
Monitoring and alerting tools
Sentry
Sentry is set up to track application errors from Search API.
All environments are configured for Slack notifications. Production alerts are routed to #govuk-search-alerts, while alerts for other environments are sent to #govuk-search-alerts-nonprod.
Kibana
Kibana can be used to query application logs as per all other GOV.UK apps.
Grafana
The GOV.UK Search API v1 Grafana dashboard visualizes core metrics for Search API and Finder Frontend.
Alert types and how to handle them
Search API success rate and latency alerts
We monitor and alert on Search API success rates. There are currently two Alertmanager rules configured in govuk-helm-charts to send notifications on Slack:
SearchApiSearchSuccessRateDegradedAcute (Critical) 5 minute rolling success rate for search requests to search-api has dropped below 99%.
SearchApiSearchSuccessRateDegradedMid (Critical) 1 hour rolling success rate for search requests to search-api has dropped below 99.9%.
Refinement of these alerts (to make sure they are comprehensively covering relevant errors) and further alerting has been prioritised by the Search Team for Q3 26/27.
Possible causes and steps to take in the event of a critical alert being triggered
- If search-api has recently been deployed and is returning errors, it may require a rollback.
- Search-api is heavily dependent on OpenSearch. If search-api is showing a high number of errors, it could indicate an issue with the OpenSearch
cluster health, or a disk space issue. Both cluster health and disk space are shown on the GOV.UK Search API v1 Grafana dashboard,
and we have separate alerting for these scenarios.
- If there is a problem with cluster health, see Opensearch Cluster Health for more information on how to investigate and fix an issue.
- If disk space is below 30%, consider increasing storage capacity.
- For further diagnosis of issues, check the issue reports in Sentry and/or the logs in Kibana.
Opensearch Cluster health alerts
Opensearch cluster health monitoring is available, and is described at Opensearch Cluster Health
We have the following alerts configured:
SearchApiClusterStatus triggers if the status is unavailable, yellow or red. See Fixing issues with the cluster
SearchApiElasticsearchDiskSpace triggers if disk space drops below 30% (warning level) or 10% (critical level). Consider increasing storage capacity if this alert fires.