Table of contents

Useful Kibana queries

All logs for GOV.UK are collected in Kibana:

Kibana can be searched using the Lucene search syntax.

Examples

5xx errors returned from cache layer

@source_host:cache* AND @fields.status:[500 TO 504]

Puppet runs

# both agent and master
@fields.syslog_program:puppet

# agent only
@fields.syslog_program:"puppet-agent"

# master only
@fields.syslog_program:"puppet-master"

Syslog logs

@type:"syslog"

Syslog logs filtered by program

@type:"syslog" AND @fields.syslog_program:"rsync"

Nginx logs

nginx @tags:"nginx"

Nginx logs for frontend:

@tags:"nginx" AND @fields.application:frontend*

Note: the @timestamp field records the request END time. To calculate request start time subtract @fields.request_time.

CDN logs

@fields.application:"govuk-cdn-logs-monitor"

Application upstart logs

@tags:"upstart" 

@tags:"upstart" AND @tags:"stdout"

@tags:"upstart" AND @tags:"stderr"

@tags:"upstart" AND @fields.application:"licensify"

Application production.log files

@tags:"application"

@tags:"application" AND @fields.application:"smartanswers"

MongoDB slow queries

@fields.application:"mongodb" AND @message:"command"

Audit/access logs

@type:"syslog" AND @fields.syslog_program:"audispd"

Mirrrorer logs

@fields.syslog_program:"mirrorer"

Publishing API timeouts

@fields.error:"TimedOutException" AND (@fields.application:"specialist-publisher" OR @fields.application:"whitehall" OR @fields.application:"content-tagger")

Syslog program names

If you’re looking for specific program outputs, use @fields.syslog_program:FOO:

  • audispd: This is used to see all audit logs from various servers. You can refer to README for searching particular types of audit logs. The program name with combination of source_host and message can be helped for looking at various specific audit log lines on a server.
  • clamd
  • cron
  • mirrorer: Records information from govuk_mirrorer script. It contains INFO, WARN and ERROR information
  • puppet-agent: Records output for govuk_puppet script on various servers
  • puppet-master
  • smokey

Gotchas

This page is owned by #2ndline and needs to be reviewed