An introduction to RabbitMQ, a broker that deals in messages
In distributed systems, communication between various applications plays an important role. Effectively passing messages between applications was always a crucial decision in system design. One of the elegant solutions to pass messages around your distributed system is a message broker.
They bring in decoupling between the applications and provide an effective way to communicate. With message brokers, an application needs no prior knowledge of their recipients to communicate.
However, what is RabbtiMQ? How does RabbitMQ fit in this picture? Also, what is AMQP?
By the end of this article, we will be able to answer these questions. I also added a few animations so that you can visualize RabbitMQ concepts.
So are you excited? I am! If you ever had difficulties understanding message brokers, as I did, then this article is the right place to start your journey. Stay with me 😊
In general, a broker is a person who facilitates trades between a buyer and a seller. An example could be a real estate agent or a stockbroker.
Similarly, if we want to trade messages between two distributed software components, we need a mediator. This mediator is known as the message broker. It receives incoming messages from a sender and sends them to a recipient. This way the sender and receiver can be totally isolated.
Another analogy for a message broker can be a Post Office (see Figure 1). Let’s take a scenario where you are going to send a letter to your cousin living in another city. Then as per this analogy, you are a producer, your cousin is a consumer, and the post office is a message broker.
RabbitMQ as message broker
Now we know that the purpose of a message broker is to route messages from a producer to a consumer. Let’s examine one such message broker — RabbitMQ. It’s one of the most extensively used message brokers these days.
The way RabbitMQ routes messages depends upon the messaging protocol it implements. RabbitMQ supports multiple messaging protocols. However, the one we are interested in is AMQP. It is an acronym for Advanced Message Queuing Protocol.
So without any further ado let’s have a closer look at the AMQP protocol model.
Advanced Message Queuing Protocol
The conceptual model of AMQP is quite simple and straightforward. It has three entities:
When a publisher pushes a message to RabbitMQ, it first arrives at an exchange. The exchange then distributes copies of these messages to variously connected queues. Finally, consumers receive these messages.
Consider a message as a piece of data. It is necessarily a package with a payload and some meta-data. The payload contains full data whereas meta-data are properties used by RabbitMQ.
Figure 2 depicts a graphical representation of the AMQP model.
AMQP is a programmable protocol. Programmers have the liberty to use libraries to configure entities (exchange, binding, and queue) as per their own needs. A RabbitMQ admin has no role in setting up these entities.
These queues are somehow similar to the queues from our data structure classes. RabbitMQ queues also follow FIFO — First-In-First-Out methodology. A queue is a place where RabbitMQ stores messages/data.
Programmers can configure queues through available programming libraries. You can make a queue durable ( with the
Durability property) to safeguard your data in case the broker crashes. You can also provide a name(with the
Name property) to a queue. Other than
Durability, a queue has a few other properties like auto-delete, exclusive, and arguments.
Before moving any further, it’s important to understand who is a direct consumer of these queues. Moreover, how many ways can a user consume messages from a queue?
Consumers are the ones who are going to use messages stored in a queue. It is possible to connect more than one consumer to a queue at a time. Consumers can either pull the message from the queue by pooling it or queues can even push the message to various connect consumers.
Bindings are the rules that a queue defines while establishing a connection with an exchange. You can have a queue connected to multiple exchanges. Every queue is also connected to a default exchange. An exchange will use these bindings to route messages to queues.
Exchanges and their types
An Exchange is a gateway to RabbitMQ for your messages. The distance the message has to travel inside RabbitMQ depends on the type of exchange. Primarily there are four types.
The name explains it all! — A direct exchange delivers a message directly to the queues that satisfy the below condition:
Routing key == Binding key
A routing key is an attribute of the message. On the other hand, a binding key is something you specify while creating a binding between a queue and an exchange.
Figure 3 is a visual explanation of how messages flow while using a direct exchange.
A message originates from a producer (green circle) with a routing key —
img.resize. Once it reaches the exchange (Orange circle), the exchange will try to find all queues with binding key —
img.resize. In case of a match, the message is pushed to all matched queues (resize in our case). If there is no match found, the message can be sent back to the producer or can even be discarded. We are lucky that we found a match in our example 😊
Once the message reaches the desired queue (resize in our case), they are distributed in round-robin fashion to all the connected consumers (resizer.1/resizer.2 in our case).
By distributing messages in a round-robin fashion, RabbitMQ makes sure that the messages are load balanced.
You must have noticed that the queue named crop is not receiving any messages. Because the routing key in this example is
img.resize. To send messages to this queue, we need to send messages with a routing key that would match the binding key (say
img.crop for instance).
A Fanout exchange ignores routing keys and distributes a message to all the connected queues. No wonder it is called Fanout (blowing messages to all connected queues! 😊).
One of the use cases for this type of exchange is message broadcast.
Please note that RabbitMQ will still do round robin if there is more than one consumer of the queue.
A topic exchange routes a message by matching routing key with a pattern in the binding key.
Routing key == Pattern in binding key.
RabbitMQ uses two wild card characters for pattern matching
#. Use a
* to match 1 word and a
# to match 0 or more words.
Figure 5 is a visual depiction of a topic exchange. Messages with routing key —
logs.error will match patterns —
logs.*. Hence these messages will end up in the queues —
only error and
Whereas for the producer at the bottom-left, messages with routing key—
logs.success will match patterns of binding key
logs.* . Hence these messages will end up in the queues —
all logs and
only success .
This type of exchange has a vast range of use cases. It can be used in the publish-subscribe pattern, distributing relevant data to desiring workers processes and many more.
A header is a particular type of exchange that routes messages based on keys present in the message header. It overlooks routing key attribute of the message.
When creating bindings for a header exchange, it is possible to bind a queue to match more than one header. In such a case, RabbitMQ should know from the producer if it should match all or any of these keys.
A producer/application can do this by providing an extra flag called ‘x-match’. ‘x-match’ can have
all values. The first one mandates that only one value should match while the latter mandates that all must match.
Once a message reaches its destination, the broker should delete the message from the queue. It is necessary because a queue overflow can occur if it keeps accumulating messages.
Before deleting any message, the broker must have a delivery acknowledgment. There are two possible ways to acknowledge message delivery.
- Automatic acknowledgment: Once a consumer receives the message
- Explicit acknowledgment: When a consumer sends back an acknowledgment
In most cases, explicit acknowledgment is used as it makes sure that the consumer has consumed the message without any failover.
RabbitMQ is a very mature and useful product. This article is only a high-level introduction to RabbitMQ. I simplified the concepts to provide a reference point for you to move further. Visit the RabbitMQ website for more complex topics.
Hope you like the article. Don’t forget to clap(or applaud 😊). Follow to read my upcoming stories. Until next time, keep Queuing.