How logging works on GOV.UK
Logit
GOV.UK is following The GDS Way guidance on logging by using the approved vendor Logit.
For information on how to log in and view stacks, please see the GOV.UK Logit documentation.
Note: although Logit is where we currently store logs, the GDS Way suggests we are migrating towards Splunk and therefore some logs are currently available in Splunk.
Kibana
Kibana is the interface for viewing logs in Elasticsearch. Use the Logit interface to log into Kibana.
There’s some documentation on useful Kibana queries for Technical 2nd Line.
Filebeat
Each machine runs Elastic Filebeat, and independently ships logs to the Logit-provided logstash endpoint.
Filebeat tails logs every 10 seconds and can output to a variety of sources. It is fully incorporated into the Elastic ecosystem.
We use the filebeat::prospector defined type to create the filebeat configuration on each instance.
Logstream and Logship
We have a defined type in our Puppet code which uses logship to tail logfiles. We only use Logstream to send nginx metrics, via statsd, to Graphite.
In the future this will be replaced.
Fastly
Fastly sends logs to S3 for the www, assets and bouncer services. These can be queried through Athena.
Logs are also available in Splunk in the govuk_cdn index. Here’s an example query for POST requests
SSH logs
We ship /var/log/auth.log and /var/log/secure.log to Splunk via CloudWatch and CDIO Cyber’s centralised security logging service.
There are different indices for each environment:
You can see the logs if your account has access to GOV.UK’s Splunk. If you do not have access to Splunk, you can request access by raising a support ticket with IT and asking them to enable Splunk for your Google account with a note that you work on GOV.UK.