Table of contents
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert

Icinga alerts

Outstanding security updates

Most security updates should be automagically applied overnight by unattended-upgrades and our Nagios check accounts for that by delaying alerts for up to 24hrs.

Please note: At the moment, an issue with load balancer health checks in AWS prevents automatic reboots. RE are working on this. In the mean time, manual intervention is required. Please refer to RE docs and contact #re-govuk on Slack if you require assistance.

To see which packages are outstanding you can use the following fabric task:

fab $environment -H frontend-lb-1.frontend apt.security_updates

Before running an unattended upgrade manually it’s worth checking why it failed to run. Logs of the previous runs can be found in /var/log/unattended-upgrades. The most recent run log can be found at unattended-upgrades.log, whereas older logs can be found tarballed by date in the same directory.

Perform an unattended upgrade manually with the apt.unattended_upgrade Fabric task.

It has been known to alert about packages that were not currently installed and thus not picked up by unattended-upgrades, such as libunwind7 and dh-apparmor. Such examples of new packages can be taken care of by manually installing them with apt-get install.

We explicitly exclude or ‘blacklist’ some security updates from being applied by unattended-upgrades. This currently includes MySQL Server, which would represent a single point of failure if upgraded automatically; in such cases a manual upgrade during office hours is preferable.

Please see the apt::unattended_upgrades::blacklist: node in Hiera for details of the exact packages that are blacklisted.

You should only consider running apt-get dist-upgrade as a last resort, because it will aggressively upgrade, downgrade, and even remove packages as it deems necessary. Run it with --dry-run in Integration first and review the output very carefully. If in doubt about any of this output ask the Reliability Engineering team for assistance.

You may see an alert telling you that a connection to NRPE could not be established. See Nagios NRPE connection failures for more information.

'Problem with MergeList’ error

apt-get is known to fail occasionally on machines running Ubuntu Trusty with an error similar to:

Exception: invalid literal for int() with base 10:
'E: Error: Opening the cache (E:Encountered a section with no Package: header,
E:Problem with MergeList /var/lib/apt/lists/mirrors.ubuntu.com_mirrors.txt_dists_trusty_restricted_i18n_Translation-en%5fU'

As a temporary work-around, run these commands on the machine to fix:

sudo rm -r /var/lib/apt/lists/*
sudo apt-get update
This page was last reviewed on 28 May 2019. It needs to be reviewed again on 28 June 2019 by the page owner #govuk-2ndline .
This page was set to be reviewed before 28 June 2019 by the page owner #govuk-2ndline. This might mean the content is out of date.