THE
OPTIMAL
RABBITMQ
GUIDE
FROM BEGINNER
TO ADVANCED
BY LOVISA JOHANSSON
OVER 12,000+
READERS & COUNTING
Acknowledgments
I want to say a big thank you to everyone who has helped me, from the earliest draft of this
book, up to this first edition. A special thanks go out to my lovely colleagues at 84codes
and to my computer nerd (and IRL) friends; Emil, Filip, Lisa, and Thomas.
Finally: A big thank you, to all [Link] users for your feedback and continued
support. It has been a great motivator to see our customers' projects succeed!
I'd love to hear from you
I encourage you to e-mail me any comments that you might have about the book. Tell us
what you think should or shouldn't be included in the next edition. If you have an
application which is using RabbitMQ, and a user story that you would like to share; please
send me an e-mail!
Book version: 1.0
ISBN: 978-91-639-9585-9
Author: Lovisa Johansson
Email: lovisa@[Link]
Published: 2018-10-25
Graphics: Elin Vinka, Daniel Marklund
•
This book is dedicated to all the frequent and future users ofRabbitMQ, and to the Swedish
mentality that made us queue lovers by heart.
Table of Contents
Part 1: Introduction to RabbitMQ
Microservices and RabbitMQ..........................................................................................11
What is RabbitMQ?.........................................................................................................13
Exchanges, routing keys and bindings ...........................................................................22
Headers Exchange ..........................................................................................................28
RabbitMQ and client libraries........................................................................................30
Ruby with Bunny.............................................................................................................31
[Link] with Amqplib ..................................................................................................... 36
Python with Pika .............................................................................................................41
The Management Interface ............................................................................................46
Part 2: Advanced Message Queuing
RabbitMQ Best Practice .................................................................................................61
Best Practices For High Performance ............................................................................73
Best Practices for High Availability................................................................................75
Queue Federation............................................................................................................77
RabbitMQ Protocols .......................................................................................................81
Part 3: RabbitMQ User Stories
Breaking Down a Monolithic System into Microservices .............................................87
A Microservice Architecture built upon RabbitMQ .....................................................90
Introduction
Times are changing! Reliability and scalability are more important than ever. Because of that com
panies are rethinking their architecture. Monoliths are evolving into microservices and servers are
moving into the cloud. Message Queues and RabbitMQ, one of the most widely deployed open
source message brokers, has come to play a significant role in the growing world of microservices.
This book is divided into three parts:
The first part focuses on giving a gentle introduction to microservices and RabbitMQ, and how to
use RabbitMQ on the CloudAMQP online platform. You will learn the most important RabbitMQ
concepts and how RabbitMQ allows users to create products that match all of the demands made
by the industry of both today and tomorrow.
The second part of the book is for the more advanced users. You will, after this part, be able to take
full advantage of RabbitMQ. We talk about Best Practice recommendations for High Performance
and High Availability and common RabbitMQ errors and mistakes. It's also a deep dive into some
RabbitMQ concepts and features.
The third part gives some real-world user stories from own experiences and a CloudAMQP cus
tomer's point of view.
We hope that this book will help to take you even further on your message queuing journey.
9
Microservices and RabbitMQ
Managing a complex, unified enterprise application can be a lot more difficult
than it seems. Before making a small change, you might need to spend hours
of brainstorming and analyze possible impacts your change could have on the
overall system. Also, new developers have to spend days getting familiar with
the system before they are considered ready to write a non-breaking line of
code. The situation could be less complicated, through the use of microser
vices. This chapter talks about the benefits of a microservice architecture.
In a microservice architecture (also called modular architecture) different modules are de-coupled
from each other. These modules often offer various functionalities, forming a complete software
application, via properly articulated operations and interfaces. Unlike a monolithic application,
where all chunks of functionality are present within a single module, a microservice application
divides different functionalities across different modules, to enhance maintainability and ease-of
use. More and more organizations are going down the microservices road because it allows them to
have a more agile approach to software development and maintenance. Microservices also make it
easy for businesses to scale and deliver updated versions of software weekly, instead of yearly.
Benefits of a microservice architecture
♦ Development and maintenance made easier
Imagine that you have to build a huge, bulky billing application which will involve authentica
tion, authorization, charging and reporting of transactions. If you divide your application across
multiple services (one for each functionality), you will be able to separate the responsibilities
and also give developers the freedom to write code for a specific service in any chosen language;
something that a monolithic application can't provide. Additionally, it will also be easier to
maintain written code and make changes to the system. For example, if you are asked to update
your authentication scheme, you will only have to add code to the authentication module and
test it, without having to worry about disrupting any other functionalities.
♦ Fault isolation
Another obvious advantage offered by a microservice architecture is its ability to isolate the fault
to a single module. For example, if your reporting service is down, your authentication and
11
billing services will still be up, ensuring that your customers can perform important transactions
even when they aren't able to view reports.
♦ Enhanced levels of speed and productivity
Microservices is all about fragmenting applications into manageable modules and functionali
ties that are easy to maintain. Different developers can work on different modules at the same
time. In addition to the development cycle, the testing phase is also sped up by the use of mi
croservices, as you can test each microservice on its own to determine the readiness of the over
all system.
♦ Improved scalability
Microservices also allow you to scale effortlessly and at will. If you want to add new components
to just one service, you can do that without changing any of the other services. You can also de
ploy the more resource-intensive services across multiple servers.
♦ Easy to understand
Another advantage offered by a microservice application is the ease of understandability. Since
each module represents a single functionality, getting to know the relevant details becomes easi
er and faster. For example, if you have hired a consultant for your charging services, they don't
have to understand the whole system in order to offer you insight; they only have to familiarize
themselves with one module to do so.
The role of a message queue in a microservice architecture
Typically, in a microservice architecture, there are cross-dependencies, which entail that no single
service can perform its functionalities without getting help from other services. This is where it's
crucial for your system to have a mechanism in place which allows services to keep in touch with
each other without getting blocked by responses. Message queuing fulfills this purpose by provid
ing a means for services to push messages to a queue asynchronously and ensure that they get de
livered to the correct destination. To implement message queuing, you need a message broker;
think of it as a mailman, who takes mail from a sender and delivers it to the correct destination.
Get started for free with CloudAMQP
Perfectly configured and optimized RabbitMQ clusters ready in 2 minutes.
[Link]
Custom Alarms• Free Plan Available• Easy Monitoring • 24/7 support• 99.95% SLA
12
What is RabbitMQ?
Message queuing is an important facet of any microservice architecture, and
it's a way of exchanging data between processes, applications and servers.
RabbitMQ is one of the most widely used message brokers, with over 35,000
production deployments worldwide, across startups and big enterprises alike.
This chapter gives a brief understanding of messaging and defines essential
RabbitMQ concepts. The chapter also explains the steps to go through when
setting up a connection and how to publish or consume messages from a
queue.
MESSAGE
RabbitMQ is a message-queuing software called a mes
sage broker or queue manager. Simply put; it's a software �
where queues can be defined, and applications may con Enqueue
nect to the queue and transfer a message onto it. Message
queues enable asynchronous communication, which MESSAGE QUEUE
means that other applications (endpoints) that are pro
ducing and consuming messages interact with the queue,
instead of communicating directly with each other. I MESSACE#2
A message can include any information. It could, for ex
I MESSACE#l
ample, contain information about a process/job that Dequeue
should start on another application (that could be on an
other server), or it might as well just be a simple text
message.
Figure 1 - Messages sentfrom a
The message broker stores the messages until a receiving sender to a receiver.
application connects and takes a message off the queue.
The receiving application then appropriately processes the
message. A producer can add messages to a queue without having to wait for them to be processed.
13
3. In this case, we see two bindings to two different queues from the exchange. The exchange
routes the message to the correct queue, depending on its attributes.
4. The messages stay in the queue until a consumer handles them.
5. The consumer handles the message, thus removing it from the queue.
PROOUCER
CD
BROKER
EXCHANCE
@ ®
®
□
Blndlng Blndlng
QUEUES
W 0
RabbltMQ
CONSUMER
®
Figure 4-Illustration ofthe messagejlow in RabbitMQ, withfive steps highlighted.
Types of exchanges
Only direct exchanges are used within our sample code in the upcoming chapters. More in-depth
understanding of the different exchange types, binding keys, routing keys and how/when they
should be used can be found in the chapter: Exchanges, routing keys and bindings.
♦ Direct: A direct exchange delivers messages to queues based on a message routing key. In a di
rect exchange, the message is routed to the queue whose binding key exactly matches the routing
key of the message. If the queue is bound to the exchange with the binding key pdfprocess, a
message published to the exchange with a routing key pdfprocess is routed to that queue.
♦ Topic: The topic exchange performs a wildcard match between the routing key and the routing
pattern specified in the binding.
♦ Fanout: A fanout exchange routes messages to ali of the queues that are bound to it.
♦ Headers: Headers exchanges use the message header attributes to do their routing.
16
PRODUCER �
BROKER
EXCHANCE Dlrect
Binding Key
BINDINCS
PDFprocess us.#
QUEUES [Queuel ) [Queue2 )
@=) @=) @=) �
a RabbitMQ
CONSUMER
Figure 5 - Three dijferent exchanges: direct, tapie, andfanout.
RabbitMQ and server concepts
Here are sorne important concepts that are helpful to know before we dig deeper into RabbitMQ.
The default virtual host, the default user, and the default permissions are used in the examples that
follow.
♦ Producer: Application that sends the messages.
♦ Consumer: Application that receives the messages.
♦ Queue: Buffer that stores messages.
♦ Message: Data that is sent from the producer to a consumer through RabbitMQ.
♦ Connection: A connection is a TCP connection between your application and the RabbitMQ
broker.
♦ Channel: A channel is a virtual connection inside a connection. When you are publishing or
consuming messages or subscribing to a queue, it's ali done over a channel.
♦ Exchange: Receives messages from producers and pushes them to queues depending on rules
defined by the exchange type. A queue needs to be bound to at least one exchange to be able to
receive messages.
♦ Binding: A binding is a link between a queue and an exchange.
17
♦ Routing key: The routing key is a key that the exchange looks at to decide how to route the
message to queues. You can think of the routing key as the destination address of a message.
♦ AMQP: The Advanced Message Queuing Protocol is the primary protocol used by RabbitMQ for
messaging.
♦ Users: It's possible to connect to RabbitMQ with a given username and password. Every user
can be assigned permissions such as rights to read, write and configure privileges within the in
stance. Users can also be assigned permissions to specific virtual hosts.
♦ Vhost, virtual host: A virtual host provides a way to segregate applications that are using the
same RabbitMQ instance. Different users can have different access privileges to different vhosts
and queues, and exchanges can be created so that they only exist in one vhost.
♦ Acknowledgments and Confirms: Acknowledgments and confirms indicate that messages
have been received or acted upon. Acknowledgments can be used in both directions; A consumer
can indicate to the server that it has received/processed a message, and the server could report
the same thing to the producer.
Setting up a RabbitMQ instance
To be able to follow this guide you need to set up a CloudAMQP instance or set up RabbitMQ on
your local machine. CloudAMQP is a hosted RabbitMQ solution (RabbitMQ as a Service), meaning
that all you need to do is sign up for an account and create an instance. You do not need to set up
and install RabbitMQ or care about cluster handling. CloudAMQP will do all of that for you. It is
also available for free with the plan Little Lemur. Go to the plan page ([Link]/
[Link]) and sign up for a plan of your choice. Click on "details" for your cloud-hosted Rabbit
MQ instance to find your username, password and connection URL (figure 7).
Cto dAMQP ".
lnstances
.. -
-
- ...., • q
- . .. � ... ,_
► �• Ctülé t#_e..i l!'lWI�' ,
Figure 6 - Instances in the CloudAMQP web interface.
18
RabbitMQ y conceptos de servidor
It's possible to send messages across languages, platforms, and operating systems once you have
the RabbitMQ instance up and running. You can now start by opening the management interlace to
get an overview ofyour RabbitMQ server.
CloudAMQP 1 •
Details
Figure 7 - Detailed information of an instance in the CloudAMQP Console.
The Management Interface - Management and Monitoring
RabbitMQ provides an easy to use web user interlace (UI) for management and monitoring ofyour
RabbitMQ server. A link to the management interlace can be found on the details page for your
CloudAMQP instance.
The management interlace allows you to handle, create, delete and list queues. It also gives you the
possibility to monitor queue length, check message rate, change and add users permissions, and
much more.
Detailed information about the management interlace will be given in the chapter: The Manage
ment lnterlace.
19
3. Declare/create a queue. Declaring a queue will cause it to be created if it does not already
exist. All queues need to be declared before they can be used.
4. Set up exchanges and bind a queue to an exchange. Messages are only routed to a queue if
the queue is bound to an exchange.
5. Publish a message to an exchange, and consume a message from the queue.
6. Close the channel and the connection.
Sample code
Sample code for Ruby, [Link], and Python can be found in upcoming chapters. It's possible to use
different programming languages in different parts of your system. The subscriber could, for ex
ample, be written in [Link] while the subscriber is written in Python.
Hint: Separate your Projects and Environments Using
Vhosts
You might create different databases within a single PostgreSQL host for different projects. In the
same way, vhosts makes it possible to separate applications on one broker. You can isolate users,
exchanges, queues etc to one specific vhost. You can separate environments, e.g., production to one
vhost and staging to another vhost, within the same broker, instead of setting up multiple brokers.
The downside of using a single instance is that there's no resource isolation between vhosts.
Shared plans on CloudAMQP are located on isolated vhosts.
Get started for free with CloudAMQP
Perfectly configured and optimized RabbitMQ clusters ready in 2 minutes.
[Link]
Custom Alarms • Free Plan Available • Easy Monitoring • 24/7 support• 99.95% SLA
21
Open the management interface for the downstream server (EU-West-1) and go to the Admin ->
Federation Upstreams screen and press Add a new upstream (figure 42). Fill in all information
needed, the URI should be the URI of the upstream server.
Leave expiry time and TIL blank. Leaving this blank means that it will stay forever.
4. Start to move messages
It's time to connect or move the publisher or the consumer to the new cluster. If you are migrating
the cluster, you can move the publisher and/or the consumer in any order. The federated queue
will only retrieve messages when it has run out of messages locally when it has consumers that
need messages, or when the upstream queue has "spare" messages that are not being consumed.
5. Verify that messages are federated
Verify that the downstream server consumes the messages published to the queue at the upstream
server when there are consumers available at the downstream server. If it does, then we are done.
Get started for free with CloudAMQP
Perfectly configured and optimized RabbitMQ clusters ready in 2 minutes.
[Link]
Custom Alarms• Free Plan Available• Easy Monitoring • 24/7 support• 99.95% SLA
Bo
}, msg);
function onError(e) {
[Link]("STOMP ERROR", e);
function onDebug(m) {
[Link]("STOMP DEBUG", m);
Get started for free with CloudAMQP
Perfectly configured and optimized RabbitMQ clusters ready in 2 minutes.
[Link]
Custom Alarms• Free Plan Available• Easy Monitoring • 24/7 support• 99.95% SLA
85
A dedicated RabbitMQ instance can be created via the CloudAMQP control panel, or by adding the
CloudAMQP add-on from any of our integrated platforms, like Heroku, IBM Cloud Catalogue,
AWS marketplace, Azure marketplace, Manifold just to mention a few.
Integrated Application Platforms
[ill heroku CLOUD Flf!UNDRY
IBM Bluemix'" �� AppHarbor
Q OPENSHIFT 'ttcloudAMQP
,-- -G) - CloudAMQP - - - - - -,
Cloud Platforms
API
Database
®
0 Google Cloud Platform
RobbitMQ [ mgmt-shovel I
:I Windows·Azure· [ server-metrics]
0 DigitalOcean ssh-monitor
(3 rockspace.
SOFTLAY:R'
�[
'--------------------
machine ]
Figure 45 - The automated process behind [Link]
When a client creates a new dedicated instance, an HTIP request is sent from the reseller to a ser
vice called CloudAMQP-API (1). The HTIP request includes all information specified by the client:
plan, server name, data center, region, number of nodes etc., as you see in the picture above.
CloudAMQP-API handles the request, save information into a database (2), and finally, send a ac
[Link]-message to one of our RabbitMQ-clusters (3).
Another service, called CloudAMQP-machine, is subscribing to [Link]. CloudAMQP-ma
chine takes the [Link]-message of the queue and performs actions for the new account (4).
CloudAMQP-machine trigger a whole lot of scripts and processes. First, it creates the new server(s)
in the chosen datacenter via an HTIP request (5). We use different underlying instance types de
pending on data center, plan and number of nodes. CloudAMQP-machine is responsible for all
configuration of the server, setting up RabbitMQ, mirror nodes, handle clustering for RabbitMQ
etc., all depending on the number of nodes and chosen datacenter.
CloudAMQP-machine sends a [Link]-message back to the RabbitMQ cluster once the
cluster is created and configured. The message is sent on the topic exchange (6). The great thing
about the topic exchange is that a whole lot of services can subscribe to the event. We have a few
91
This section described a small part of the CloudAMQP service. We do in total have around 100 mi
croservices, all communicating via CloudAMQP and RabbitMQ.
I hope you enjoyed the reading and that you at least learned someting new. Never hesitate to send
me an email!
Get started for free with CloudAMQP
Perfectly configured and optimized RabbitMQ clusters ready in 2 minutes.
[Link]
Custom Alarms• Free Plan Available• Easy Monitoring • 24/7 support• 99.95% SLA
93
TIMES ARE CHANGING!
Reliability and scalability are more important than ever.
Therefore companies are rethinking their architecture.
Monoliths are evolving into microservices and servers are
moving into the cloud. Message Queues and especially
RabbitMQ has come to play a significant role in the
growing world of microservices. After all, it’s one of the
most widely deployed open source message brokers.
This book will help you along your RabbitMQ journey. It
consists of three main parts:
- Introduction to Message Queuing and RabbitMQ
- Advanced Message Queuing with RabbitMQ
- User Stories
Essentially, this book is about RabbitMQ;
and who knows about queuing better than a bunch of
Swedes*?
In case you haven’t heard: queuing is an integral part of
the Swedish everyday life.
This book is provided by the Swedish tech company, ISBN 978-91-639-9585-9
84codes AB, which provides the service CloudAMQP -
RabbitMQ as a service. 9 789163 995859
