Skip to main content
  1. Home
  2. Monitoring
Warning This document has not been updated for a while now. It may be out of date.
Last updated: 16 Sep 2021

Add an Icinga passive check to a Jenkins job

If you would like Icinga to raise an alert when a Jenkins job has not completed successfully in a while, you can add an Icinga passive check to the Jenkins job by configuring Puppet.

  1. Add a parameterized build trigger to the Jenkins job you wish to monitor in Puppet, so that the success or failure of the job builds will be reported to Icinga.

  2. If the alert should be a warning rather than a critical alert, configure the alert level by setting the status code parameter in the build trigger configured in the previous step (example):

     NSCA_CODE=1

    Valid values are 0 = OK, 1 = warning, 2 = critical, 3 = unknown. The default is 2.

  3. Configure an icinga::passive_check exported resource in the Puppet manifest file for the Jenkins job you wish to monitor, making sure that the service description matches the one used to configure the build trigger above.

    You will need to determine an appropriate ‘freshness threshold’, which determines how much time can pass, in seconds, before Icinga will raise a ‘freshness threshold alert’. You should allow enough time for the interval between scheduled builds, plus the time it usually takes the job to run, plus a little extra in case the job takes longer than usual.

  4. Strongly consider creating a page in this manual which explains the alert and suggests how to resolve it.

    In the page metadata, configure the section to be Icinga alerts.

    Update the icinga::passive_check puppet settings added in the previous step: configure the notes_url so that it links to your new page.

  5. Deploy Puppet.

Troubleshooting “Freshness threshold exceeded” warnings

You can manually refresh a ‘passive’ Icinga check by running the Success_Passive_Check job. You’ll need to provide values for NSCA_CHECK_DESCRIPTION and NSCA_OUTPUT, which you can derive from the configuration for the alert.

For example, the “Whitehall run broken link checker” configuration refers to a @service_description variable that is populated by the corresponding Puppet code. So, in this example:

  • NSCA_CHECK_DESCRIPTION = “Runs a rake task on Whitehall that generates a broken link report”
  • NSCA_OUTPUT = “Runs a rake task on Whitehall that generates a broken link report success”

More in the Monitoring section

Learn

How to...