Processing millions of messages with Queues and Event Hubs

Which, when and why

📝 tldr;

Here’s a short breakdown between event hubs and queues:

Event Hub: A cloud computing architecture designed to collect, store, and process large volumes of event data.

Azure Queue: a cloud computing service aimed at providing scalable messaging solutions.

• Queues are generally used when there is no need to preserve the order of the events/messages

• Event hubs are used when there is a need to preserve the relative order of events/messages.

🙋‍♂️ Credit to our Contributing Author

Krutam wrote this article with experience as a software engineer to share with our 19,500+ readers! If you’re also interested in being a byte-sized design writer, consider applying here!

In his own words:

I have been with Microsoft since 2021. My journey with software engineering has been a roller coaster and thanks to this journey now I proudly call myself a curious and an acing engineer.

In my leisure time I like to read, write and play outdoor sports. I am a firm believer of do things with 100% commitment or don't do it. And here I am, in the world of writing newsletters to help others ace in the software industry.”

📁Message Storage Order

Let’s take another example, say John has 50 bucks in his bank account and then he takes the following steps:

• Withdraws 150 bucks

• Deposits 100 bucks

What happens to John’s bank account depends on the order of execution and they’re not guaranteed.

This is a quality of Queues.

In queues, the message order can be jumbled and changed depending on the computation time of the message.

If you look at the steps, it is clear that John shouldn’t be allowed to withdraw 150 bucks first because he won’t have enough funds.

To keep messages in order, use Event Hubs. Event hubs are used when the ordering of the messages matter and must be guaranteed.

Fig 1.1: Messaging Order

🌐Message Passing and Receiving

The message passing/listening mechanism that queues use is called as polling. In polling the listener function keeps on asking the queue whether there is a new message or not.

This increases the latency and also is costly. An advantage of using queue is that a message is only removed from the queue when the listener function asks for it which ensures the availability of the listener function.

Fig 1.2: Queue Message Transfer Mechanism

In case of an event hub, a pub/sub model is what it uses to pass the messages. In this type of communication, the event hub sends the newly entered message to all it’s subscribers. This is done using a unidirectional channel.

It is relatively cheaper as compared to polling and also it ensures real time data.

Fig 1.3: Event hub Message Transfer Mechanism

Fig 1.2 and 1.3 shows the message transfer mechanism in case of queues and event hubs respectively. How the relative positioning of is and there makes such a huge difference.

🏭Message Processing

Both queues and event hubs follow a batch processing mechanism with the batch size depending on the computing capacity of the listener function.

But for queues a task starts processing as soon as it finds a thread.

For event hubs a task won’t be started until the task before it is completed because event hubs keep the order of messages consistent.

Official Articles

A link to the official article and source! (Subscribing is the best way to support the newsletter!)