Monitor data.gov.uk
Prometheus
There are two dashboards: DGU for 2nd line displays a summary overview of key stats (including queue length), and DGU Apps Dashboard which shows the health of each app deployment (a drop-down menu on the dashboard allows the PaaS app to be selected).
Metrics are exposed to Prometheus through a /metrics endpoint from the PaaS-team maintained paas-metric-exporter app. This provides separate stats for each app running in the DGU space on the PaaS.
Two additional apps expose /metrics endpoints which summarise the state of various parts of data.gov.uk. These are datagovuk_publish_queue_monitor for the state of the Sidekiq queues used to sync data between CKAN and Publish, and datagovuk-publish-elasticsearch-monitor for monitoring the Elasticsearch indices.
Pingdom
Pingdom monitors https://data.gov.uk uptime and alerts PagerDuty when downtime is detected. Maintenace of this service forms part of #govuk-2nd-line.
Sentry
Sentry monitors application errors. The Sentry pages for each app can be found on the Find and Publish app pages.
Log.it
Each application sends logs to Logit. Publish and Find use the corresponding PaaS Service. Example query: source_host: "gds-data-gov-uk.data-gov-uk.find-data-beta" && access.response_code: 500.
Data.gov.uk production dashboard
You can view the DGU dashboard to get an overview of DGU that includes Find 4xx and 5xx, Publish elasticsearch index size and Sidekiq metrics.
Sidekiq (Publish)
You can monitor the number of jobs in each queue using the following.
First, follow the instructions on logging into the paas
cf ssh publish-data-beta-production-worker
/tmp/lifecycle/launcher /home/vcap/app 'rails console' ''
>>> Sidekiq::Queue.new.each_with_object(Hash.new(0)) {|j, h| h[j.klass] += 1 }
Analytics
Google Analytics records traffic for Find. Ask for ‘read’ access to all ‘properties’ in your request.