Useful Kibana queries
All logs for GOV.UK are collected in Kibana:
Kibana can be searched using the Lucene search syntax.
5xx errors returned from cache layer
host:cache* AND @fields.status:[500 TO 504]
# both agent and master syslog_program:puppet # agent only syslog_program:"puppet-agent" # master only syslog_program:"puppet-master"
Syslog logs filtered by program
application:"syslog" AND syslog_program:"rsync"
Nginx logs for frontend:
tags:"nginx" AND application:frontend*
@timestamp field records the request END time. To calculate request start time subtract
Application upstart logs
tags:"upstart" tags:"upstart" AND tags:"stdout" tags:"upstart" AND tags:"stderr" tags:"upstart" AND application:"licensify"
Application production.log files
tags:"application" tags:"application" AND application:"smartanswers"
MongoDB slow queries
application:"mongodb" AND message:"command"
application:"syslog" AND syslog_program:"audispd"
Publishing API timeouts
@fields.error:"TimedOutException" AND (application:"specialist-publisher" OR application:"whitehall" OR application:"content-tagger")
Syslog program names
If you’re looking for specific program outputs, use
audispd: This is used to see all audit logs from various servers. You can refer to README for searching particular types of audit logs. The program name with combination of source_host and message can be helped for looking at various specific audit log lines on a server.
mirrorer: Records information from govuk_mirrorer script. It contains INFO, WARN and ERROR information
puppet-agent: Records output for govuk_puppet script on various servers
- Score: does a aggregation of field on last 2000 results
- Terms is not an aggregation of field, it is an aggregation of terms in the field across 1 recent indices
- For more elaborate searching, read about the Lucene syntax
@timestampof nginx log entries records request end time is sometimes confusing