Useful Kibana queries
All logs for GOV.UK on all environments are collected in Kibana, which you can access through Logit.
You can save and load queries using the buttons in the top right. You may want to use one of the existing queries as a starting point instead of writing a query from scratch.
5xx errors returned from cache layer
host:cache* AND (@fields.status:[500 TO 504] OR status:[500 TO 504])
# both agent and master syslog_program:puppet* # agent only syslog_program:"puppet-agent" # master only syslog_program:"puppet-master"
Syslog logs filtered by program
application:"syslog" AND syslog_program:"rsync"
Nginx logs for frontend:
tags:"nginx" AND application:frontend*
@timestamp field records the request END time. To calculate request start time subtract
Application upstart logs
tags:"upstart" tags:"upstart" AND tags:"stdout" tags:"upstart" AND tags:"stderr" tags:"upstart" AND application:"licensify"
Application production.log files
tags:"application" tags:"application" AND application:"smartanswers"
MongoDB slow queries
application:"mongodb" AND message:"command"
application:"syslog" AND syslog_program:"audispd"
Publishing API timeouts
@fields.error:"TimedOutException" AND (application:"specialist-publisher" OR application:"whitehall" OR application:"content-tagger")
Syslog program names
If you’re looking for specific program outputs, use
audispd: This is used to see all audit logs from various servers. You can refer to README for searching particular types of audit logs. The program name with combination of source_host and message can be helped for looking at various specific audit log lines on a server.
govuk_sync_mirror: Records information from govuk_sync_mirror script
puppet-agent: Records output for govuk_puppet script on various servers
- Score: does a aggregation of field on last 2000 results
- Terms is not an aggregation of field, it is an aggregation of terms in the field across 1 recent indices
- For more elaborate searching, read about the Lucene and Elasticsearch syntax
@timestampof nginx log entries records request end time is sometimes confusing