Table of contents
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert

Icinga alerts

Low available disk inodes

The inode usage on a machine can be checked using:

df -i

which will give output like:

Filesystem           Inodes  IUsed   IFree IUse% Mounted on
/dev/mapper/os-root 3121152 151646 2969506    5% /

with a row for each filesystem currently mounted.

Low available disk inodes (Jenkins)

If Jenkins is running out of inodes (rather than disk space) then it may be possible to free some by clearing out old workspaces:

find /var/lib/jenkins/workspace/ -maxdepth 1 -type d -mtime +1 -exec rm -rf {} \;

This will find any directories that are older than 1 day and delete them.

Low available disk space on Jenkins

One possible cause of this is that the /var/lib/docker directory is consuming a large amount of disk space. This has been found to happen on the ci-agent machines.

Verify this:

cd /var/lib
sudo ncdu

If /var/lib/docker is consuming a large amount of disk space run the following as root:

docker rmi $(docker images --filter "dangling=true" -q --no-trunc)
docker volume rm $(docker volume ls -qf dangling=true)

This will remove ‘dangling’ images and volumes. A comprehensive set of instructions can be found in this Docker resource cleanup gist.

This page was last reviewed on 21 March 2019. It needs to be reviewed again on 21 September 2019 by the page owner #govuk-2ndline .
This page was set to be reviewed before 21 September 2019 by the page owner #govuk-2ndline. This might mean the content is out of date.