Skip to main content
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert
Last updated: 12 Apr 2021

Free memory warning on backend

This alert is often caused by an application slowly leaking memory, which isn’t usually an issue for apps that are deployed/restarted frequently. Some less frequently deployed apps will continue to grow over time.

For more info, you can also read the page for the alert that triggers when a application uses too much memory.

Investigation of the problem

Check Graphite to see which apps are consuming a lot of memory, and are growing over time.

You could also try using htop and sorting by memory to locate high memory usage, though this won’t identify creeping memory leaks in the same way Graphite will.

Identify if the service is being run under Unicorn or not. Unicorn worker processes can be killed and automatically respawned if they consume too much memory.

Kill memory leaking Unicorn workers

Once you’ve identified a Unicorn application with a memory leak, you can gracefully kill a Unicorn worker to reclaim some memory. Sending a SIGQUIT to the worker causes it to finish serving its current request then exit, the master will notice this and spawn a fresh worker.

sudo kill --signal QUIT <worker process>

Restart other memory leaking apps

For leaky applications not managed by Unicorn (for example, Whitehall’s Sidekiq worker), restart it to reset the memory usage, and increase free memory on the host:

sudo service whitehall-admin-procfile-worker restart

Fix memory leaks

A common cause of the alerts are memory leaks. For Rails applications you can try diagnosing the error with the derailed gem.