GOV.UK operates three standalone services for COVID-19 response:
- Offer coronavirus support from your business
- Get coronavirus support as an extremely vulnerable person
- Find support if you're affected by coronavirus
Offer coronavirus support from your business
This service allows businesses to tell us how they might be able to help with the response to coronavirus. This may include goods or services such as medical equipment, hotel rooms or childcare.
Get coronavirus support as an extremely vulnerable person
This service allows people identified as vulnerable by the NHS to tell us if they need help accessing essential supplies and support. Users will have received a link to the service in a letter or a text message from the NHS, or been advised by their GP to fill in the form. They can fill the form in themselves, or someone else can fill it in for them.
There is also an interactive voice response (IVR) automated phone service provided by AWS. This is for users who do not have access to the internet.
The service is also known as the Shielded Vulnerable service, or SV.
Find support if you're affected by coronavirus
This service allows the public - who may not have been eligible for the extremely vulnerable service - to find information about what help is available if they're struggling with unemployment, an inability to get food, having somewhere to live, or their general wellbeing as a result of coronavirus.
The service is also known as the Non-Shielded Vulnerable service, or NSV.
All applications have a similar architecture of:
- Framework: Ruby on Rails
- Database: AWS RDS (business volunteering, find support) and DynamoDB (vulnerable people)
- Hosting: PaaS
- CDN: AWS CloudFront
- CI: GitHub Actions (branches and PRs)
- CD: Concourse (master builds and production deploys)
- Logging: Sentry and Logit
- Email: GOV.UK Notify
- Queuing: Sidekiq (business volunteering only)
Each application is a standard Rails application with:
- question pages, each has their own controller, view and route
- a check your answers page (except find support form)
- a confirmation ("Thank you") page
- ineligible pages (vulnerable people form only)
- a privacy page
To run one of the applications locally, see the README in the GitHub repo.
Session and data storage
While the user is filling out the form, we use session storage to store the user's data.
For the vulnerable people form and find support form, we store session data for 4 hours in an encrypted cookie, persisted in the browser and sent back to the server for every question. Cookies have a limit of 4KB, so this approach could cause errors if the user submits large inputs.
For the business volunteering form, we store the following for 4 hours:
- the session id in an encrypted cookie
- the session data in Redis
For all but the find support form, when the user submits the form on the "Check your answers" page, the application writes user data to the database.
For the find support form, user responses are written to the database on submission of the final question. There is no "Check your answers" page.
The vulnerable people form application only has permission to write items to the database. For security and privacy reasons, there's no way to read or change already-submitted user data.
Developers should not have access to the production database for the vulnerable people form application.
The business volunteering form and find support form applications can both read and write to their databases as the security and privacy requirements are lower.
Developers may treat data in the business volunteering form and find support form to the same standards as any other GOV.UK personal data store, and are able to access it for legitimate development duties.
All applications are hosted on GOV.UK Platform as a Service (PaaS) in the Ireland region. PaaS provides a scalable hosting platform based on CloudFoundry managed by GDS. You can read more in the PaaS technical documentation.
To get access, email
firstname.lastname@example.org and ask for an
invite to the
govuk_development organisation. They need to give you
a role that lets you access both
production spaces and
the applications inside.
Before you start, install the CloudFoundry CLI.
- Log in with
cf login -a api.cloud.service.gov.uk --sso
- Select staging or production with
cf target -s <staging or production>
Manage backing services
The business volunteer application uses PaaS's Redis backing
service. The backing service is attached to the
application by a setting in the
manifest.yml file in the repository
The JSON data in the
VCAP_SERVICES environment variable exposes
settings and credentials for the backing service automatically.
We have two AWS accounts (staging and production) for CDN and data storage. They contain DynamoDB, IAM and CloudFront resources, which were provisioned by Terraform in the alphagov/covid-engineering repository. Members of the GOV.UK Coronavirus Services team have access.
To log in to AWS:
- Install and set up the gds-cli, then log in to the AWS Console with either:
gds aws govuk-corona-data-staging-poweruser -lfor staging
gds aws govuk-corona-data-prod-poweruser -lfor production
Deployment is managed via TechOps shared Concourse.
CI/CD pipelines are configured under the
govuk-tools team. You can
read about what Concourse is, its access controls, configuring secrets,
and how to administer it, on the Concourse
developer docs page.
In terms of these applications, any changes to application code on the master branch are continuously deployed to staging. Smoke tests are set up to run on staging, if these pass the applications are automatically deployed to production. The smoke tests are the feature tests within the application repository.
The Concourse deployment pipeline and task configuration for each
application is stored in the "concourse" directory at the root of each
repository. Changes to these pipelines apply automatically when pushed
Links to Concourse Pipelines:
The deployment tasks use the CloudFoundry v3 API commands in order to support zero-downtime deployment and various new features.
When working with the cf cli tool, use the
v3 prefixed commands.
Cancelling a failed deployment
Occasionally a zero-downtime deployment may fail. This can occur if
the application fails to start, or the required number of app
instances cannot be created in time. To cancel and roll back to the
previous deployment, run:
cf v3-cancel-zdt-push <app-name>.
Links to Staging
These are protected by HTTP basic auth. You can find the credentials in govuk-secrets and get them by running:
cd ~/govuk/govuk-secrets/pass ./edit.sh 2ndline coronavirus-forms/creds
There's a Splunk dashboard for all of these services. To
access Splunk, you need to have the
GDS-006-GOVUK permission on your
Google account. To get this permission, raise an IT Helpdesk ticket
and post in
#cyber-security-help to get them to confirm the request
Application errors are sent to the GOV.UK hosted Sentry:
These are under the GOV.UK Sentry account and you can sign in with your Google Account.
Logs for all applications are streamed to Logit and can be found in following dashboards:
The business volunteering service sends confirmation emails on form submission, using GOV.UK Notify.
Sidekiq is used to manage the queuing of email jobs. The application README details how to manage the Sidekiq queue.
To login to the GOV.UK Notify Dashboard obtain the credentials using govuk-secrets:
cd ~/govuk/govuk-secrets/pass pass govuk-notify/govuk-coronavirus-services
In staging only, emails are sent to
rather than the form submitter.
Extracting form responses (business volunteering only)
Cabinet Office require data exports at regular intervals. A
recurring Concourse job
exports the data to a S3 bucket each day between midnight and 1am.
Details of the bucket can be obtained using
cf services and
Should further exports be required, there is a rake task to export the data for a single day in JSON format.
cf v3-ssh govuk-coronavirus-business-volunteer-form $ /tmp/lifecycle/shell $ rake export:form_responses["<date>"]
Date to be included in the format 2020-03-26.
GOV.UK on-call operators will be paged for two reasons:
- if Pingdom thinks a service is down
- if a service has a high rate of 5xx errors
You should find the Monitoring and Useful commands sections in this document helpful when investigating this issue.
Useful Slack channels:
#govuk-corona-services-tech(builds the forms)
#covid-engineering-team(runs the data pipeline)
#re-prometheus-support(runs the Prometheus infra)
The GOV.UK Pingdom account is configured to checks if the first of each form service page is up. If it can't reach a page for a configured period of time, it will call you.
The login credentials for the service are in govuk-secrets pass.
High 5xx rate**
Note: Connect to the VPN to view Prometheus and AlertManager UIs.
These alerts are triggered by Reliability Engineering's Prometheus when the applications are serving a high number of 5xx errors.
The configuration for the alert is stored in the prometheus-aws-configuration-beta repository.
You can also view the alert in AlertManager, which sends
the Prometheus alerts to PagerDuty. You can silence the alert through
the user interface, or use the
amtool cli to end the alert.
This alert indicates a problem with the application origin servers.
Steps you could take:
- See the monitoring section, to look at some graphs
- Check the logs to see which requests are failing
- Check the commit log to see if there was a recent change
- Check the concourse pipeline to see if there was a recent deploy, or if the smoke test is failing
- Check the
/metricsendpoints on the apps, to see if they are serving metrics to prometheus correctly
Deactivate Prometheus alerts for these services
Note: you should only do this if you have confirmed that there is a problem with the alert.
You can deactivate the alert in the PagerDuty UI.
- Go to the PagerDuty ruleset
- Disable the Event Rule that is triggering alerts on the GOV.UK COVID-19 Forms service
What things will call you
The GOV.UK PagerDuty will page on-call/2ndline if these services experience issues. See the Alerting section above.
Note: Make sure to use the
v3-commands for things like restarting or deploying apps, to do so in a zero-downtime way.
See the status of all the apps and how many instances are running:
Get the logs for an app:
cf logs <app-name> --recent
View detailed information about an app:
cf app <app-name>
Restart an app:
cf v3-restart <app-name>
Access a Postgres console (for business form only):
cf conduit govuk-coronavirus-business-volunteer-form-db -- psql
Escalating to PaaS support
PaaS support contact details are in the legacy opsmanual doc under the heading "PaaS Support (COVID-19 forms)".