Logs are not being received from the CDN
Sometimes, we stop receiving either GOV.UK or Bouncer logs from Fastly
logs-cdn-1.management. We run Rsyslog on this machine, and Fastly
send us logs using Syslog on their side to a port we have previously
Firstly, look at how old and how large
/mnt/logs_cdn/cdn-bouncer.log is. In GOV.UK’s case, the file should be
created early in the morning each day, and the file size should be
positive and greater than 0 bytes. At the end of each day, a file is
created for the following day, the previous day’s file is gzipped up.
Bouncer’s logs rotate once per hour.
To fix, try (in order):
if you have access to Fastly, check if Fastly have recorded an error when writing to our rsyslog endpoint by logging in to the Fastly control panel and open the following URL in your browser:
You can find the service ID in the deployment repository.
In particular, notice the
BrokenNowboolean which indicates if Fastly are currently unable to log to our endpoint.
restarting Rsyslog on the box:
sudo service rsyslog restart, then
sudo tail -f /mnt/logs_cdn/cdn-govuk.log, or
/mnt/logs_cdn/cdn-bouncer.log, to check that logs are being streamed in to us. You’ll be able to see when they are, as the file size increases and human-readable log lines are entered in to that file. It can take as long as 45 minutes for logs to start coming through, so acknowledge the alert and come back to it later.
redeploying the existing Fastly service configuration using this Jenkins job. Authenticate with the 2nd Line user, details of which can be found in the
infrapassword store. If you don’t have access, ask one of GOV.UK’s senior technologists, or Reliability Engineering, to deploy it for you.
tailing Syslog -
sudo tail -f /var/log/syslogfor the current file, and
sudo zless syslog.x.gzfor any gzipped files replacing x with the number from the file itself. It’s pretty verbose, and might not be much use, but it’s worth a look.