GOV.UK
This page describes what to do in case of an
Icinga alert.
For more information you could
search the govuk-puppet repo for the source of the alert
App isn't running the expected Ruby version
We have checks to ensure that the version of Ruby the application is using is
the same as the version specified in its .ruby-version file.
The standard reload that happens when an application is deployed is not enough to move between Ruby versions, so the application will continue to run under the old version.
A hard restart of the application is required before the application will run
under the new version. You can do this by running the app:hard_restart deploy
task within the ‘Deploy App’ job in Jenkins.
More about Icinga alerts
- App healthcheck failed
- Asset master and slave disk space comparison
- Asset master attachment processing
- AWS SES quota usage higher than expected
- Backup passive checks
- Benchmark search queries failed
- Check for spelling suggestions failed
- Check that correct users have access
- ClamAV definitions out of date
- Data sync
- Defined CPU type does not match
- duplicate SSH host keys
- Elasticsearch cluster health
- Elasticsearch: not receiving Syslog from Logstash
- Email alerts not sent
- Enhanced ecommerce data export
- es-rotate
- Fastly error rate for GOV.UK
- Fetch analytics data for search failed
- Fix stuck virus scanning
- Free memory warning on backend
- GitHub Enterprise connectivity
- Gor
- high disk time
- High memory for application
- high zombie procs
- Logs are not being received from the CDN
- logspout running
- Low available disk inodes
- Low available disk space
- mongod replication lag
- MongoDB rollback
- MySQL Xtrabackups to S3
- MySQL: replication lag
- MySQL: replication running
- Nginx 429 too many requests
- Nginx 5xx rate too high for many apps/boxes
- Nginx high conn writing -- upstream indicator Check
- Nginx requests too low
- ntp drift too high
- Offsite backups
- Onsite backups
- Outstanding security updates
- PagerDuty drill
- Pingdom check for homepage failing
- Pingdom search check
- PostgreSQL: replication too far behind
- PostgreSQL: S3 Backups
- Prolonged GC collection times
- publisher app healthcheck not ok
- Puppet last run errors
- RabbitMQ: high watermark has been exceeded
- RabbitMQ: No consumers listening to queue
- reboot required by apt
- Redis alerts
- rkhunter warnings
- root filesystem is readonly
- Unicorn Herder
- Varnish port not responding
- VPN down
- Whitehall app healthcheck not ok
- Whitehall scheduled publishing
This page is owned by #2ndline
and needs to be reviewed