Table of contents

Useful Kibana queries

All logs for GOV.UK are collected in Kibana:

Kibana can be searched using the Lucene search syntax.

Examples

5xx errors returned from cache layer

host:cache* AND @fields.status:[500 TO 504]

Puppet runs

# both agent and master
syslog_program:puppet

# agent only
syslog_program:"puppet-agent"

# master only
syslog_program:"puppet-master"

Syslog logs

application:"syslog"

Syslog logs filtered by program

application:"syslog" AND syslog_program:"rsync"

Nginx logs

tags:"nginx"

Nginx logs for frontend:

tags:"nginx" AND application:frontend*

Note: the @timestamp field records the request END time. To calculate request start time subtract @fields.request_time.

CDN logs

application:"govuk-cdn-logs-monitor"

Application upstart logs

tags:"upstart"

tags:"upstart" AND tags:"stdout"

tags:"upstart" AND tags:"stderr"

tags:"upstart" AND application:"licensify"

Application production.log files

tags:"application"

tags:"application" AND application:"smartanswers"

MongoDB slow queries

application:"mongodb" AND message:"command"

Audit/access logs

application:"syslog" AND syslog_program:"audispd"

Mirrrorer logs

syslog_program:"mirrorer"

Publishing API timeouts

@fields.error:"TimedOutException" AND (application:"specialist-publisher" OR application:"whitehall" OR application:"content-tagger")

Syslog program names

If you’re looking for specific program outputs, use syslog_program:FOO:

  • audispd: This is used to see all audit logs from various servers. You can refer to README for searching particular types of audit logs. The program name with combination of source_host and message can be helped for looking at various specific audit log lines on a server.
  • clamd
  • cron
  • mirrorer: Records information from govuk_mirrorer script. It contains INFO, WARN and ERROR information
  • puppet-agent: Records output for govuk_puppet script on various servers
  • puppet-master
  • smokey

Gotchas

This page is owned by #2ndline and needs to be reviewed