Free memory warning on backend
This alert is often caused by an application slowly leaking memory, which isn’t usually an issue for apps that are deployed/restarted frequently. Some less frequently deployed apps will continue to grow over time.
For more info, you can also read the page for the alert that triggers when a application uses too much memory.
Investigation of the problem
Check Graphite to see which apps are consuming a lot of memory, and are growing over time.
You could also try using htop
and sorting by memory to locate high memory
usage, though this won’t identify creeping memory leaks in the same way
Graphite will.
Identify if the service is being run under Unicorn or not. Unicorn worker processes can be killed and automatically respawned if they consume too much memory.
Kill memory leaking Unicorn workers
Once you’ve identified a Unicorn application with a memory leak, you can
gracefully kill a Unicorn worker to reclaim some memory. Sending a SIGQUIT
to
the worker causes it to finish serving its current request then exit, the
master will notice this and spawn a fresh worker.
sudo kill --signal QUIT <worker process>
Restart other memory leaking apps
For leaky applications not managed by Unicorn (for example, Whitehall’s Sidekiq worker), restart it to reset the memory usage, and increase free memory on the host:
sudo service whitehall-admin-procfile-worker restart
Fix memory leaks
A common cause of the alerts are memory leaks. For Rails applications you can try diagnosing the error with the derailed gem.