Most teams in GOV.UK have screens set up to show data about pull requests and releases.
The search screen displays live data from GOV.UK. It includes the number of people on GOV.UK, latest searches, trending and recent content. It’s not publicly accessible because there’s sometimes personal data in the latest searches.
2nd line screen
There is a screen by the 2ndline desks.
The screen is a webpage running David Singleton’s Frame Splits with 3 splits: production health, icinga alert summary per environment, recent deployments.
This screen contains a dashboard giving an overview of health for the platform, a list of upcoming releases, and a dashboard showing the alerts for each environment.
This dashboard contains 2 graphs, one of origin 4xx and 5xx, and one of edge 4xx and 5xx. It’s worth keeping an eye on this and looking for any anomalies, as this may indicate issues on production. It’s likely due to our caching behaviour that the top graph of origin errors will indicate issues before they are visible in the second graph, and to end users.
Sometimes the ‘EDGE’ graphs may disappear. These are obtained by the
collectd-cdn plugin on
monitoring-1.management.production. If the graphs disappear, they
should write errors to
/var/log/syslog. They may look something like
Nov 10 11:37:17 monitoring-1.management collectd: cdn_fastly plugin: Failed to query service: govuk Nov 10 11:37:17 monitoring-1.management collectd: cdn_fastly plugin: Failed to query service: tldredirect Nov 10 11:37:17 monitoring-1.management collectd: cdn_fastly plugin: Failed to query service: assets Nov 10 11:37:17 monitoring-1.management collectd: cdn_fastly plugin: Failed to query service: redirector
If this happens, restarting collectd on the monitoring server may kick things into life.
sudo service collectd restart
Icinga alert summary per environment
This screen shows a summary of the critical and warning alerts for our three environments (production, staging, integration) in a colour coded box (red for criticals, yellow for warnings, purple for unknowns, green for no issues).
This shows a list of recent app deployments to staging and production.
More about Tools
- Access apps on the shared Heroku account
- Add a new document type
- Add a new Ruby version
- Configure a GitHub repo
- GitHub Enterprise
- Graphite and deployment dashboards
- How we use GitHub
- Pact Broker
- Run an A/B or multivariate test
- SSH Configuration
- Tools: Icinga, Grafana and Graphite, Kibana and Fabric
- Update Hubot (Slack bot)