Fall back to the static mirrors
We maintain a static copy of most of the site, which gets used by the content delivery network (CDN) whenever origin (the application server) times out or serves an error response.
This process is handled by our CDN config and is entirely transparent to us and our users. It happens multiple times a day, for lots of different reasons.
This is why we refer to switching off Nginx on the origin cache machines as “falling back to the mirrors”.
GOV.UK is mirrored to two cloud providers:
Amazon S3: the static mirror is hosted in a bucket
govuk-<environment>-mirrorand the content is retrieved by the CDN using an API. This bucket is replicated to
govuk-<environment>-mirror-replicain another AWS region.
Google GCS: the static mirror is hosted in a bucket
govuk-<environment>-mirrorand the content is retrieved by the CDN using an API.
Access to the:
Amazon S3 buckets are restricted to Fastly, Office and Pingdom IP addresses for read-only access and authenticated users in AWS web console.
Google GCS buckets are restricted by secret keys in
govuk-secretsand authenticated users in Google GCP web console.
Updates to the mirror
Every day at 20:00, the govuk_seed_crawler on the Mirrorer machine adds all GOV.UK URLs listed within the sitemaps (in the magnitude of hundreds of thousands) to a message queue. The govuk_crawler_worker on the Mirrorer machine consumes these URLs, retrieves the HTML returned by these URLs, saves the content to disk and adds any new URLs found on those pages to the back of the queue.
Every hour, the static copy of the site is copied from the Mirrorer machine to the primary AWS S3 bucket
govuk-<environment>-mirror. This primary bucket is automatically replicated to another S3 bucket
govuk-<environment>-mirror-replica) in another region by AWS.
In addition, the primary AWS S3 bucket
govuk-<environment>-mirror is synced to the Google GCS bucket
of the same name daily at 12:00.
The crawler is entirely independent of the mirrors. Stopping the crawler means no new updates are made to the mirrors, but it will not stop the mirrors from working.
To inspect the contents of the mirror:
ssh <mirrorer_machine_name> cd /mnt/crawler_worker/www.gov.uk
<mirrorer_machine_name> can be obtained from the
govuk_node_list -c mirrorer command
on the jumpbox.
Forcing failover to the static mirrors
Because the CDN will retry every request against the mirrors automatically if origin is unavailable, stopping Nginx on the cache machines with Fabric will result in falling back to mirrors:
$ fab $environment class:cache incident.fail_to_mirror
To disable the fallback:
$ fab $environment class:cache incident.recover_origin
Emergency publishing using the static mirror
If you need to make changes to the site while origin is unavailable, you’ll have to modify content on the static mirrors. Bear in mind that because the mirror is static HTML, it’s hard to make broad changes to the site (like putting a banner on every page).
You’ll be notified by the escalation on-call contact that you need to edit the site.
If you’re at home, connect to the VPN.
ssh to the Mirrorer machine by running:
<mirrorer_machine_name>can be obtained from the
govuk_node_list -c mirrorercommand on the jumpbox.
Disable puppet on the machine by running:
govuk_puppet --disable "stopping crawling to avoid mirror changes"
Stop the govuk_crawler_worker by running:
initctl stop govuk_crawler_worker
Modify the relevant file in the directory
Upload the file to the AWS S3 bucket via the AWS console.
Upload the file to Google Cloud Storage using the GCP console. Credentials are located in the govuk-secrets password store, under
If you’re notified that the edit you’ve made can be reverted, do that the same way.
Once origin becomes available again, somebody (maybe you) will have to ensure that origin has been updated to serve the change that you made.