Manage AmazonMQ
RabbitMQ is a message broker based on the Advanced Message Queuing Protocol (AMQP). Publishing’s RabbitMQ cluster is provided by AWS’ AmazonMQ service.
Connecting to the AmazonMQ web control panel
Run gds govuk connect amazonmq -e integration and point your
browser at the URL it gives you - it will look like http://127.0.0.1:45612, but will have a random port number. You can connect to staging and production the same way, just replace integration above with the environment of your choice.
The username for connecting to the RabbitMQ web control panel is root and the password
can be decrypted from the govuk-secrets repo via bundle exec rake 'eyaml:decrypt_value[integration,govuk_amazonmq::root_password]' (from the puppet_aws directory).
AmazonMQ metrics
A generic AmazonMQ dashboard shows metrics for queues and exchanges.
How we run RabbitMQ
Overview
Producer: an application that publishes messages to RabbitMQ. On GOV.UK this could be publishing-api.
Exchanges are AMQP entities where messages are sent. They take a message
and route it into zero or more queues. The routing algorithm used depends on
the exchange type and rules called bindings. When a content change is made
in a publishing application (e.g. Travel Advice Publisher), Publishing API
publishes a message to our main exchange,
publishing_documents.
Queues are very similar to queues in other message and task-queueing
systems: they store messages that are consumed by applications. An example of a
queue is email_alert_service which is used by
email-alert-service to forward publishing activity to
email-alert-api. Queues are created by consumer applications.
Bindings are rules that exchanges use (among other things) to route
messages to queues. To instruct an exchange E to route messages to a queue Q, Q
has to be bound to E. Bindings may have an optional routing key attribute. An
example of a binding is the cache_clearing_service-high queue is
bound to the published_documents exchange with a routing
key matching of *.major. E.g messages sent to the exchange with a routing key
of guide.major will be routed to that queue.
Messages consist of a JSON payload and publish options (we predominantly use content type, routing key and persistant).
Options:
- content_type (string) - tells the consumer the type of message. E.g
"application/json" - routing_key (string) - matches against bindings to filter messages to certain
queues. E.g
"guide.major" - persistant (boolean) - tells RabbitMQ whether to save the message to disk.
Message options are set when a message is published. In our use case, the message’s payload is the content item in JSON format. The code in the publishing-api to publish a message is here.
Consumer applications are applications which consume messages from one or more queues. For email-alert-service this is done by running this rake task and using the major change processor to do the processing of the consumed messages. All our consumer applications use the govuk_message_queue_consumer gem to consume messages from RabbitMQ in a standardised way.
You can see live examples of things like queues, exchanges, bindings etc by connecting to the RabbitMQ control panel (see above).
Further reading
- RabbitMQ Tutorials
- AmazonMQ documentation
- Bunny is the RabbitMQ client we use.
- The Bunny Guides explain all AMQP concepts really well.