Table of contents
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert

Fastly error rate for GOV.UK

We get response code reporting from Fastly (with a 15 minute delay). It averages out the last 15 minutes worth of 5xx errors. This is a useful supplementary metric to highlight low-level errors that occur over a longer period of time.

The alert appears on monitoring-1.management.

It is possible to query the CDN logs using AWS Athena. The following query gets a count of URLs where a 5xx error has been served between the given timestamps:

SELECT url, status, COUNT(*) AS count
FROM fastly_logs.govuk_www
WHERE status >= 500 AND status <= 599
AND request_received >= TIMESTAMP '2018-11-26 11:00'
AND request_received < TIMESTAMP '2018-11-26 11:30'
AND date = 26 AND month = 11 AND year = 2018
GROUP BY url, status
ORDER BY count DESC

It is also possible to examine the raw Fastly CDN logs:

  • ssh monitoring-1.management.production
  • cd /var/log/cdn to access log files

Unknown alert

The alert appears on monitoring-1.management. Collectd uses the Fastly API to get statistics which it pushes to Graphite. If the alert is unknown, collectd likely cannot talk to Fastly so restart collectd.

$ sudo service collectd restart

To prove collectd is the problem, use this query in Kibana:

syslog_hostname:monitoring-1.management AND syslog_program:collectd

You will see many reports similar to:

cdn_fastly plugin: Failed to query service
This page was last reviewed on 31 August 2018. It needs to be reviewed again on 28 February 2019 by the page owner #govuk-2ndline .
This page was set to be reviewed before 28 February 2019 by the page owner #govuk-2ndline. This might mean the content is out of date.