Skip to main content
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert
Warning This document has not been updated for a while now. It may be out of date.
Last updated: 4 Aug 2022

Low available disk inodes

See also: low available disk space

This alerts means that a machine’s filesystem has too many files or directories.

An inode is a data structure used on unix filesystems to store metadata about files and dirs. On a given filesystem the number of inodes is limited, so a machine may run out if it creates too many files – even if there’s plenty of disk space left. The solution is therefore to remove unused files or dirs.

The inode usage on a machine can be checked using:

$ df -i

which will give output like:

Filesystem           Inodes  IUsed   IFree IUse% Mounted on
/dev/mapper/os-root 3121152 151646 2969506    5% /

with a row for each filesystem currently mounted.

Locating high inode usage

The following command will print out a list of directories prefixed with the number of files they contain, sorted low to high. The largest are therefore most likely candidates for removal.

{ find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; } 2>/dev/null

Change the find / to find /some-other-path if the df -i command identified some other filesystem as being the full one.

Note that if the /tmp directory is on the full filesystem sort may be unable to create a tempfile. In that case, add the -T /some-other-path parameter to both of the sort commands in the above one-liner.

Low available disk inodes (Jenkins)

If Jenkins is running out of inodes (rather than disk space) then it may be possible to free some by clearing out old workspaces (on the ci-agent machines), or failed jobs (on the ci-master machine):

Clearing old workspaces

We have a process which deletes old workspaces automatically. The threshold is 90 days by default, and 30 days for CI agents.

If the number of workspaces is still too high on a particular machine, you can delete workspaces manually with the command below. However, you should also consider lowering the thresholds above (and updating this documentation) so that manual intervention is not required again.

# Delete any workspaces older than 1 day
$ sudo find /var/lib/jenkins/workspace/ -maxdepth 1 -type d -mtime +1 -exec rm -rf {} \;

Clearing failed jobs

$ sudo find /var/lib/jenkins/jobs/ -mindepth 3 -maxdepth 3 -type d -mtime +30 | grep -v -P "main|master" | xargs -I {} sudo rm -rf {}

This will find any job branches that are older than 30 days (except for main and master, which it might be prudent to keep) and delete them. If you want to know how many folders will be affected, you can see with:

$ sudo find /var/lib/jenkins/jobs/ -mindepth 3 -maxdepth 3 -type d -mtime +30 | grep -v -P "main|master" | wc -l