Table of contents
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert

Outstanding security updates

Most security updates should be automagically applied overnight by unattended-upgrades and our Nagios check accounts for that by delaying alerts for up to 24hrs.

To see which packages are outstanding you can use the apt.security_updates Fabric task.

Before running an unattended upgrade manually it’s worth checking why it failed to run. Logs of the previous runs can be found in /var/log/unattended-upgrades. The most recent run log can be found at unattended-upgrades.log, whereas older logs can be found tarballed by date in the same directory.

Perform an unattended upgrade manually with the apt.unattended_upgrade Fabric task.

It has been known to alert about packages that were not currently installed and thus not picked up by unattended-upgrades, such as libunwind7 and dh-apparmor. Such examples of new packages can be taken care of by manually installing them with apt-get install.

We explicitly exclude or ‘blacklist’ some security updates from being applied by unattended-upgrades. This currently includes MySQL Server, which would represent a single point of failure if upgraded automatically; in such cases a manual upgrade during office hours is preferable.

Please see the apt::unattended_upgrades::blacklist: node in Hiera for details of the exact packages that are blacklisted.

You should only consider running apt-get dist-upgrade as a last resort, because it will aggressively upgrade and even downgrade packages as it deems necessary. Run it with --dry-run in Integration first and review the output very carefully. If in doubt ask someone in the Infrastructure team.

You may see an alert telling you that a connection to NRPE could not be established. See Nagios NRPE connection failures for more information.

'Problem with MergeList’ error

apt-get is known to fail occasionally on machines running Ubuntu Trusty with an error similar to:

Exception: invalid literal for int() with base 10:
'E: Error: Opening the cache (E:Encountered a section with no Package: header,
E:Problem with MergeList /var/lib/apt/lists/mirrors.ubuntu.com_mirrors.txt_dists_trusty_restricted_i18n_Translation-en%5fU'

As a temporary work-around, run these commands on the machine to fix:

sudo rm -r /var/lib/apt/lists/*
sudo apt-get update

More about Icinga alerts

This page was last reviewed . It needs to be reviewed again by the page owner #2ndline.