Table of contents
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert

Run high priority tests

The high priority tests come from Smokey and the Icinga check is defined in Puppet.

Tests failing

If many of the tests are failing in an AWS environment, it may be because the Nginx services haven’t registered new boxes coming online or old ones going offline. You can try to restart the following services:

$ fab $environment class:cache app.reload:nginx
$ fab $environment class:draft_cache app.reload:nginx
$ fab $environment class:monitoring app.reload:nginx
$ fab $environment class:monitoring app.restart:smokey-loop

Traceback (most recent call last):

If you see this error in Icinga, it may mean that the smokey-loop process has died. You can try looking through the logs or restarting the process.

$ ssh monitoring-1.production
> sudo less /var/log/upstart/smokey-loop.log
$ ssh monitoring-1.production
> sudo service smokey-loop restart

Integration with Signon

These tests rely on a user in GOV.UK Signon. All Signon users have their passphrase expire periodically. This will cause the tests to fail.

You can either change the passphrase of the account and rotate it in encrypted hieradata, or you can fake a passphrase change in the Signon Rails console:

$ govuk_app_console signon
irb(main):001:0> smokey = User.find_by(name: "Smokey (test user)")
irb(main):002:0> smokey.update_attribute(:password_changed_at, Time.now)

More about Icinga alerts

This page was last reviewed . It needs to be reviewed again by the page owner #govuk-2ndline.