Table of contents
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert

Icinga alerts

Run high priority tests

The high priority tests come from Smokey and the Icinga check is defined in Puppet.

Tests failing

If many of the tests are failing in an AWS environment, it may be because the Nginx services haven’t registered new boxes coming online or old ones going offline. You can try to restart the following services:

$ fab $environment class:cache app.reload:nginx
$ fab $environment class:draft_cache app.reload:nginx
$ fab $environment class:monitoring app.reload:nginx
$ fab $environment class:monitoring app.restart:smokey-loop

Traceback (most recent call last):

If you see this error in Icinga, it may mean that the smokey-loop process has died. You can try looking through the logs or restarting the process.

$ ssh monitoring-1.production
> sudo less /var/log/upstart/smokey-loop.log

If you see recent log entries like HTTP status code 550 (RestClient::RequestFailed) this usually means that the BrowserMob Proxy java process is running as part of a previously aborted smokey-loop and the new smoke tests cannot start a new proxy. It’s necessary to kill the existing java process.

$ ps -ef | grep java
> smokey    6385  6380 26 14:58 ?        00:00:54 java -Dbasedir=/opt/smokey -jar /opt/smokey/lib/browsermob-dist-2.1.4.jar --port 3222
$ sudo kill -9 6385

(replacing process numbers as appropriate).

It’s also useful to ensure no java proxy processes are left running when restarting smokey-loop.

$ ssh monitoring-1.production
> sudo service smokey-loop stop
$ ps -ef | grep java
> smokey    6385  6380 26 14:58 ?        00:00:54 java -Dbasedir=/opt/smokey -jar /opt/smokey/lib/browsermob-dist-2.1.4.jar --port 3222
$ sudo kill -9 6385
$ sudo service smokey-loop start

Integration with Signon

These tests rely on a user in GOV.UK Signon. All Signon users have their passphrase expire periodically. This will cause the tests to fail.

You can either change the passphrase of the account and rotate it in encrypted hieradata, or you can fake a passphrase change in the Signon Rails console:

$ govuk_app_console signon
irb(main):001:0> smokey = User.find_by(name: "Smokey (test user)")
irb(main):002:0> smokey.update_attribute(:password_changed_at,
This page was last reviewed on 17 April 2019. It needs to be reviewed again on 17 October 2019 by the page owner #govuk-2ndline .
This page was set to be reviewed before 17 October 2019 by the page owner #govuk-2ndline. This might mean the content is out of date.