Skip to main content
Table of contents
This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert

smokey.json older than 30m

There can be the rare case wherein Smokey has failed to correctly run. While the causes can be various and ultimately unknown, the symptoms include a large number of Unknown errors in Icinga on the monitoring box.

The easiest way to clear these issues is ensure all relevant Smokey tasks are killed:

$ sudo killall chromedriver
$ sudo killall chrome
$ sudo killall java

Alternatively if killall doesn’t work, attempt to kill the processes directly either through use of htop or with the kill command:

# Repeat until all chromedriver processes that do not mention 'grep' are gone
# Then repeat for chrome and java

$ ps -ef | grep chromedriver # To find the process ID to kill
$ sudo kill -15 $processID # Use the first number from the top line of grep

With ps -ef | grep smokey you should only see a running series of Postgres services. These are not actually Smokey tasks, but Docker tasks that are being misreported in the process list and it is advised to leave them running.

After the above, restart the service with sudo service smokey-loop start and you should see it clear up.

If the error is still hanging around

In the very rare occurrence that the smokey.json 30m error returns half an hour after the above steps are followed, retry the steps and then run govuk_puppet --test after starting the service. This is ultimately a solution with very little testing, as we only saw this happen once, but it worked in that case and may work again.

This page was last reviewed on 7 April 2020. It needs to be reviewed again on 7 October 2020 by the page owner #govuk-2ndline .
This page was set to be reviewed before 7 October 2020 by the page owner #govuk-2ndline. This might mean the content is out of date.