Skip to main content
Table of contents

Logging

Query Kibana (includes useful queries)

All logs for GOV.UK on all environments are collected in Kibana, which you can access through Logit.

Kibana can be searched using the Lucene search syntax or full JSON-based Elasticsearch queries.

Examples

You can save and load queries using the buttons in the top right. You may want to use one of the existing queries as a starting point instead of writing a query from scratch.

Kibana saved searches

5xx errors returned from cache layer

host:cache* AND (@fields.status:[500 TO 504] OR status:[500 TO 504])

Puppet runs

# both agent and master
syslog_program:puppet*

# agent only
syslog_program:"puppet-agent"

# master only
syslog_program:"puppet-master"

Syslog logs

application:"syslog"

Syslog logs filtered by program

application:"syslog" AND syslog_program:"rsync"

Nginx logs

tags:"nginx"

Nginx logs for frontend:

tags:"nginx" AND application:frontend*

Note

The @timestamp field records the request END time. To calculate request start time subtract request_time.

Application upstart logs

tags:"upstart"

tags:"upstart" AND tags:"stdout"

tags:"upstart" AND tags:"stderr"

tags:"upstart" AND application:"licensify"

Application production.log files

tags:"application"

tags:"application" AND application:"smartanswers"

MongoDB slow queries

application:"mongodb" AND message:"command"

Audit/access logs

application:"syslog" AND syslog_program:"audispd"

Mirrorer logs

syslog_program:"govuk_sync_mirror"

Publishing API timeouts

message:"TimedOutException" AND (application:"specialist-publisher" OR application:"whitehall" OR application:"content-tagger")

Syslog program names

If you’re looking for specific program outputs, use syslog_program:FOO:

  • audispd: This is used to see all audit logs from various servers. You can refer to README for searching particular types of audit logs. The program name with combination of source_host and message can be helped for looking at various specific audit log lines on a server.
  • clamd
  • cron
  • govuk_sync_mirror: Records information from govuk_sync_mirror script
  • puppet-agent: Records output for govuk_puppet script on various servers
  • puppet-master
  • smokey

Gotchas

This page was last reviewed on 27 June 2019. It needs to be reviewed again on 27 December 2019 by the page owner #govuk-2ndline .
This page was set to be reviewed before 27 December 2019 by the page owner #govuk-2ndline. This might mean the content is out of date.