Free memory warning on backend
This is often caused by an application slowly leaking memory, which isn’t usually an issue for apps that are deployed/restarted frequently. Some less frequently deployed apps will continue to grow over time.
Investigation of the problem
Check Graphite to see which apps are consuming a lot of memory, and are growing over time.
You could also try using htop and sorting by memory to locate high memory usage, though this won’t identify creeping memory leaks in the same way Graphite will.
Identify if the service is being run under Unicorn or not. Unicorn worker processes can be killed and automatically respawned if they consume too much memory.
Kill memory leaking Unicorn workers
Once you’ve identified a Unicorn application with a memory leak, you can gracefully kill a Unicorn worker to reclaim some memory. Sending a SIGQUIT to the worker causes it to finish serving its current request then exit, the master will notice this and spawn a fresh worker. This fabric task will kill the worker using the most memory:
fab $environment -H backend-1.backend app.respawn_large_unicorns:APP-NAME
This will only kill the most bloated Unicorn worker, so you may need to run it multiple times if the alert persists.
Restart other memory leaking apps
For leaky applications not managed by Unicorn (for example, Whitehall’s Sidekiq worker), restart it to reset the memory usage, and increase free memory on the host:
This page is owned by #2ndline and needs to be reviewed
fab $environment -H whitehall-backend-1.backend app.restart:whitehall-admin-procfile-worker