Whitehall scheduled publishing

This page describes what to do in case of an Icinga alert. For more information you could search the govuk-puppet repo for the source of the alert

Warning This document has not been updated for a while now. It may be out of date.

Last updated: 14 Jul 2022

Overdue publications in Whitehall

This alert means that there are scheduled editions which have passed their publication due date but haven’t been published by the scheduled publishing workers. Scheduled publishing is performed by Sidekiq workers picking up jobs from a scheduled queue.

You can see whether Sidekiq picked up the job for processing, and the current state of the job using Sidekiq monitoring apps.

You can verify that there are overdue published documents using:

You’ll see output like this:

ID  Scheduled date             Title

If there are overdue publications you can publish by running the following:

After a database restore

If the above rake tasks aren’t working, it could be because the database was recently restored, perhaps due to the data sync. In that case, you can try running the following Rake task on a whitehall_backend machine:

Due to the overnight data anonymisation process you may notice that some of the pending documents have one or more edition that is in a scheduled state, is access_limited, and may have one or more attachments with the filename redacted.pdf.

These documents cannot be published and will cause a RestClient::UnprocessableEntity: 422 Unprocessable Entity error.

The situation is likely to be re-created the next day by the above process on the same documents/editions.

Scheduled publications in Whitehall not queued

This alert means that the number of editions in the database which are scheduled to be published in the future is different from the number currently in the queue.

Run the publishing:scheduled:requeue_all_jobs Rake task to requeue all scheduled editions:

Historically, this error has happened in the staging and integration environments, after the nightly data sync. We no longer spawn scheduling workers for Whitehall on Staging and Integration, so this will no longer affect those environments. The alert and rake task fix could still prove useful on Production.