Hi, friends, and welcome to another tutorial. We'll be talking about RabbitMQ, a tool that is widely used as a message broker in distributed and Pub-Sub systems.

While multiple applications can share a single RabbitMQ instance, it's a great approach to configure it to have different virtual hosts for each application.

One practical usage in a Python application is using it as a Celery broker URL or simply as a message queue.

One benefit of this is that it saves the costs that comes with spinning up multiple instances of RabbitMQ. Creating RabbitMQ virtual hosts is similar both locally and when running in a self-managed solution such as AWS RabbitMQ

By the end of this guide, you will be able to:

1. Create multiple virtual hosts for a single RabbitMQ instance.
2. Test the connections using the correct credentials, that is user, password, and virtual host.

Prerequisites to follow along:

1. Ensure you have Docker and Docker Compose installed, or
2. Have a self-managed RabbitMQ instance from AWS, CloudAMQP, or any other service provider

Without any delay, let’s get started

How to Run a RabbitMQ Instance Locally

To keep things simple, we are going to run the RabbitMQ instance in a Docker container.

Create a file named docker-compose.yml and update it with the following content:

docker-compose.yml

version: "3.8"
services:
  rabbitmq:
    image: rabbitmq:3.8-management-alpine
    container_name: 'rabbitmq_test'
    environment:
      - RABBITMQ_DEFAULT_USER=sampleuser
      - RABBITMQ_DEFAULT_PASS=samplepassword
    ports:
        - 5672:5672
        - 15672:15672
    volumes:
      - ~/.docker-conf/rabbitmq/data/:/var/lib/rabbitmq/
      - ~/.docker-conf/rabbitmq/log/:/var/log/rabbitmq

To start the container, run this command in the terminal:

docker-compose up

Once the container is up and running, access the GUI by visiting this link: http://localhost:15672/

RabbitMQ GUI login page

Use the credentials specified in the docker-compose file to gain access to the dashboard – “sampleuser” and “samplepassword” in my case.

Note: If you are using a self-managed RabbitMQ instance from AWS or CloudAMQP, the RabbitMQ connection string will look similar to this:

amqps://ausername:apassword@32wewjijiokkoo.mq.eu-west-3.amazonaws.com:5671

To access the RabbitMQ admin dashboard, follow these steps:

1. Visit the URL using the HTTPS protocol: https://32wewjijiokkoo.mq.eu-west-3.amazonaws.com
2. Remove the username, password, and port from the URL.
3. After accessing the URL, the system will prompt you to enter the username and password specified in the RabbitMQ connection string (“ausername” and “apassword” in this case). Provide these credentials to log in to the admin dashboard.

How to Create a Virtual Host

Once you are logged in, click on the ‘Admin’ tab, and then navigate to the ‘Virtual Hosts’ tab as shown in the image below.

RabbitMQ Virtualhost tab

Click on ‘Add a new virtual host’, and provide it with an appropriate name. Finally, click on the ‘Add virtual host’ button to create the new virtual host.

New Virtual Host created

How to Create a New User

On the Users tab, click on the ‘Add a user’ toggle. Specify a Username and Password for the new user. If you wish, you may grant the user access to the Admin (GUI) interface. Finally, click on the ‘Add user’ button to create the new user.

Create a new user

How to Assign the New User to the Created Virtual Host (Vhost)

On the Users tab, locate and click on the previously created user, which was named ‘user1’ in my case.

New user in users list

For the new user, choose the virtual host as ‘new-virtual-host’, and then click the ‘Set permission’ button. This grants the user access to the specified virtual host. To confirm this, go back to the Users tab and verify that the user now appears with access to the ‘new-virtual-host'

Assign new user to the created virtual host

By now, the new user 'user1' has been configured to access the vhost 'new-virtual-host'.

New user assigned to the created virtual host

How to Test the Connection for the New User and Vhost

To test if the new user can access the new vhost, you can use the following sample Python script. Once everything is working correctly, you can use the URL string to connect to the RabbitMQ broker in any application.

_testrabbit.py

import pika

def test_rabbitmq_url(url):

    try:
        params = pika.URLParameters(url)
        connection = pika.BlockingConnection(params)
        connection.close()
        print("Connection successful!")

    except pika.exceptions.AMQPError as e:
        print("Connection failed:", e)

test_rabbitmq_url('amqp://user1:password@localhost:5672/new-virtual-host')

Note how we have specified the RabbitMQ URL: it has the username as “user1” and password as “password9,” while the virtual host is named ‘new-virtual-host.’

Remember to install pika by running pip install pika on the command line.

To run this script on the terminal, enter the following command:

python test_rabbit.py

Successfully connected to RabbitMQ

This indicates a successful connection to the RabbitMQ instance.

Change the password in the utility function to an incorrect one, and you should see an outcome similar to this:

Connection to RabbitMQ failed

Wrapping Up

In conclusion, creating multiple virtual hosts on a single RabbitMQ instance can significantly enhance your message queue management capabilities.

By effectively segregating applications and resources, you ensure better organization, security, and scalability. With the step-by-step guide provided in this article, you now have the knowledge to harness the full potential of RabbitMQ

Thanks for reading, and I hope you enjoyed the article.

For more interesting content and tips, please feel free to check out my video collection on YouTube and consider hitting the subscribe button.

You can also find me on LinkedIn.

An introduction to RabbitMQ, message broker, AMQP model and more.

In distributed systems, communication between various applications plays an important role. Effectively passing messages between applications was always a crucial decision in system design. One of the elegant solutions to pass messages around your distributed system is a message broker.

In distributed systems, communication between various applications plays an important role. Effectively passing messages between applications was always a crucial decision in system design. One of the elegant solutions to pass messages around your distributed system is a message broker.

They bring in decoupling between the applications and provide an effective way to communicate. With message brokers, an application needs no prior knowledge of their recipients to communicate.

However, what is RabbtiMQ? How does RabbitMQ fit in this picture? Also, what is AMQP?

By the end of this article, we will be able to answer these questions. I also added a few animations so that you can visualize RabbitMQ concepts.

So are you excited? I am! If you ever had difficulties understanding message brokers, as I did, then this article is the right place to start your journey. Stay with me ?

Message Broker

In general, a broker is a person who facilitates trades between a buyer and a seller. An example could be a real estate agent or a stockbroker.

Similarly, if we want to trade messages between two distributed software components, we need a mediator. This mediator is known as the message broker. It receives incoming messages from a sender and sends them to a recipient. This way the sender and receiver can be totally isolated.

Another analogy for a message broker can be a Post Office (see Figure 1). Let’s take a scenario where you are going to send a letter to your cousin living in another city. Then as per this analogy, you are a producer, your cousin is a consumer, and the post office is a message broker.

Figure 1: Analogy for message broker

RabbitMQ as message broker

Now we know that the purpose of a message broker is to route messages from a producer to a consumer. Let’s examine one such message broker — RabbitMQ. It’s one of the most extensively used message brokers these days.

The way RabbitMQ routes messages depends upon the messaging protocol it implements. RabbitMQ supports multiple messaging protocols. However, the one we are interested in is AMQP. It is an acronym for Advanced Message Queuing Protocol.

So without any further ado let’s have a closer look at the AMQP protocol model.

Advanced Message Queuing Protocol

The conceptual model of AMQP is quite simple and straightforward. It has three entities:

1. Queue
2. Binding
3. Exchange

When a publisher pushes a message to RabbitMQ, it first arrives at an exchange. The exchange then distributes copies of these messages to variously connected queues. Finally, consumers receive these messages.

Consider a message as a piece of data. It is necessarily a package with a payload and some meta-data. The payload contains full data whereas meta-data are properties used by RabbitMQ.

Figure 2 depicts a graphical representation of the AMQP model.

Figure 2: AMQP model

AMQP is a programmable protocol. Programmers have the liberty to use libraries to configure entities (exchange, binding, and queue) as per their own needs. A RabbitMQ admin has no role in setting up these entities.

There are plenty of libraries available to work with RabbitMQ. You can choose from Nodejs, Python, .Net, Java, and many more.

Queues

These queues are somehow similar to the queues from our data structure classes. RabbitMQ queues also follow FIFO — First-In-First-Out methodology. A queue is a place where RabbitMQ stores messages/data.

Programmers can configure queues through available programming libraries. You can make a queue durable ( with the Durability property) to safeguard your data in case the broker crashes. You can also provide a name(with the Name property) to a queue. Other than Name and Durability, a queue has a few other properties like auto-delete, exclusive, and arguments.

Before moving any further, it’s important to understand who is a direct consumer of these queues. Moreover, how many ways can a user consume messages from a queue?

Consumers

Consumers are the ones who are going to use messages stored in a queue. It is possible to connect more than one consumer to a queue at a time. Consumers can either pull the message from the queue by pooling it or queues can even push the message to various connect consumers.

Bindings

Bindings are the rules that a queue defines while establishing a connection with an exchange. You can have a queue connected to multiple exchanges. Every queue is also connected to a default exchange. An exchange will use these bindings to route messages to queues.

Exchanges and their types

An Exchange is a gateway to RabbitMQ for your messages. The distance the message has to travel inside RabbitMQ depends on the type of exchange. Primarily there are four types.

• Direct
• Fanout
• Topic
• Header

Direct

The name explains it all! — A direct exchange delivers a message directly to the queues that satisfy the below condition:

Routing key == Binding key

A routing key is an attribute of the message. On the other hand, a binding key is something you specify while creating a binding between a queue and an exchange.

Figure 3 is a visual explanation of how messages flow while using a direct exchange.

A message originates from a producer (green circle) with a routing key — img.resize. Once it reaches the exchange (Orange circle), the exchange will try to find all queues with binding key — img.resize. In case of a match, the message is pushed to all matched queues (resize in our case). If there is no match found, the message can be sent back to the producer or can even be discarded. We are lucky that we found a match in our example ?

[gif image]

Once the message reaches the desired queue (resize in our case), they are distributed in round-robin fashion to all the connected consumers (resizer.1/resizer.2 in our case).

By distributing messages in a round-robin fashion, RabbitMQ makes sure that the messages are load balanced.

You must have noticed that the queue named crop is not receiving any messages. Because the routing key in this example is img.resize. To send messages to this queue, we need to send messages with a routing key that would match the binding key (say img.crop for instance).

Fanout

A Fanout exchange ignores routing keys and distributes a message to all the connected queues. No wonder it is called Fanout (blowing messages to all connected queues! ?).

One of the use cases for this type of exchange is message broadcast.

Please note that RabbitMQ will still do round robin if there is more than one consumer of the queue.

Topic

A topic exchange routes a message by matching routing key with a pattern in the binding key.

Routing key == Pattern in binding key.

RabbitMQ uses two wild card characters for pattern matching * and #. Use a * to match 1 word and a # to match 0 or more words.

Figure 5 is a visual depiction of a topic exchange. Messages with routing key — logs.error will match patterns — logs.error and logs.*. Hence these messages will end up in the queues — only error and all logs.

Whereas for the producer at the bottom-left, messages with routing key— logs.success will match patterns of binding key #success and logs.* . Hence these messages will end up in the queues — all logsand only success .

[gif]

This type of exchange has a vast range of use cases. It can be used in the publish-subscribe pattern, distributing relevant data to desiring workers processes and many more.

Header

A header is a particular type of exchange that routes messages based on keys present in the message header. It overlooks routing key attribute of the message.

When creating bindings for a header exchange, it is possible to bind a queue to match more than one header. In such a case, RabbitMQ should know from the producer if it should match all or any of these keys.

A producer/application can do this by providing an extra flag called ‘x-match’. ‘x-match’ can have any or all values. The first one mandates that only one value should match while the latter mandates that all must match.

Message Acknowledgement

Once a message reaches its destination, the broker should delete the message from the queue. It is necessary because a queue overflow can occur if it keeps accumulating messages.

Before deleting any message, the broker must have a delivery acknowledgment. There are two possible ways to acknowledge message delivery.

1. Automatic acknowledgment: Once a consumer receives the message
2. Explicit acknowledgment: When a consumer sends back an acknowledgment

In most cases, explicit acknowledgment is used as it makes sure that the consumer has consumed the message without any failover.

What’s next

RabbitMQ is a very mature and useful product. This article is only a high-level introduction to RabbitMQ. I simplified the concepts to provide a reference point for you to move further. Visit the RabbitMQ website for more complex topics.

Hope you like the article. Don’t forget to clap(or applaud ?). Follow to read my upcoming stories. Until next time, keep Queuing.