Skip to main content
Table of contents

data.gov.uk

Monitor data.gov.uk

Prometheus

There are two dashboards: DGU for 2nd line displays a summary overview of key stats (including queue length), and DGU Apps Dashboard which shows the health of each app deployment (a drop-down menu on the dashboard allows the PaaS app to be selected).

Metrics are exposed to Prometheus through a /metrics endpoint from the PaaS-team maintained paas-metric-exporter app. This provides separate stats for each app running in the DGU space on the PaaS.

Two additional apps expose /metrics endpoints which summarise the state of various parts of data.gov.uk. These are datagovuk_publish_queue_monitor for the state of the Sidekiq queues used to sync data between CKAN and Publish, and datagovuk-publish-elasticsearch-monitor for monitoring the Elasticsearch indices.

Pingdom

Pingdom monitors https://data.gov.uk uptime and alerts PagerDuty when downtime is detected. Maintenace of this service forms part of #govuk-2nd-line.

Sentry

Sentry monitors application errors. The Sentry pages for each app can be found on the Find and Publish app pages.

Log.it

Each application sends logs to Logit. Publish and Find use the corresponding PaaS Service. Example query: source_host: "gds-data-gov-uk.data-gov-uk.find-data-beta" && access.response_code: 500.

Sidekiq (Publish)

You can monitor the number of jobs in each queue using the following.

First, follow the instructions on logging into the paas

cf ssh publish-data-beta-production-worker
/tmp/lifecycle/launcher /home/vcap/app 'rails console' ''
>>> Sidekiq::Queue.new.each_with_object(Hash.new(0)) {|j, h| h[j.klass] += 1 }

Sidekiq UI (Publish)

Sidekiq UI is only accessible to the localhost domain, so you’ll need an SSH tunnel to see it on staging/production.

First, follow the instructions on logging into the paas

cf ssh -L 9000:localhost:8080 publish-data-beta-staging

Then go to localhost:9000/sidekiq/recurring-jobs in your browser to see active jobs, retries and to manually modify the schedule.

Analytics

Google Analytics records traffic for Find. Ask for ‘read’ access to all 'properties’ in your request.

This page was last reviewed on 15 August 2019. It needs to be reviewed again on 15 February 2020 by the page owner #govuk-platform-health .
This page was set to be reviewed before 15 February 2020 by the page owner #govuk-platform-health. This might mean the content is out of date.