Message Queue

In this section, we are going to learn about communication between two services. Let's take an example: In a company, people are creating build requests for source code. Let's call these people who create build requests as producers. We also have build systems in place that take these build requests and build the source code in the appropriate environment. Let's call them consumers. Now, how do we manage this system ? The answer is message queues.

What is a Message Queue ?

A message queue is a system that maintains a queue of messages between applications. It acts as a temporary storage buffer, and redirects the messages from a producer to an available consumer. A message queue can also be distributed in nature, that is operating on multiple machines in a cluster, forming one logical unit.

A simplified overview of how message queues work is that a producer sends a message to a queue and the consumer consumes message from the queue. Each message is processed only once, and that's why message queues are also known as one-to-one communication.

How Message Queues work ?

A Message Queue system has two components: Message exchange and queue.

  • A Publisher sends a message to the message exchange. The message exchange is responsible for receiving the message and letting the producer know about it.

  • The exchange then redirects the message to the appropriate queue. Redirection happens based on bindings or rules between exchange and queue. These bindings can be based on type of message, like message for a user service, ticket service, etc. or based on type of computation by consumer, like fast or slow process, etc. or could even be based on priority of messages.

  • The consumers maintains connections to their respective queues, from which the messages can be processed. The consumers sends message back regarding successful completion or failure of the message processing.

  • The messages are only removed from queue once the message has been successfully processed.

A consumer group can connect to a single queue, thereby a cluster of workers can feed off messages from a single queue. This is a common architecture pattern to scale the throughput of jobs for intensive jobs.

A message queue system can also have multiple queues, for different purposes, having their own consumer or consumer groups. This allows a single message queue system to serve as a single system for multiple kinds of messages.

Properties

  • Message queue only dequeues the message once it has been successfully processed.

  • Message queues are horizontally scalable by increasing the number of workers, queues or exchanges.

  • Message queues are fault tolerant, that is the messages once received are not lost in case of failure.

  • Messages are delivered in the order of their arrival to the queue.

  • Push vs Pull: Message queues push messages to the consumers instead of consumers pulling them from queues.

Pull mechanism increases a round trip cycle of request-acknowledgement to the queue. Push mechanism can overwhelm a consumer by sending more messages than the consumer is able to process. This can be controlled by a setting a limit on the number of messages a consumer can retain. Push also allows message balancing between workers.

  • There exists multiple ways of binding between exchange and queue: could be either fanout, one-to-one based on a topic or a wild card or message header, or could be based on consistent hashing by dividing a queue into multiple smaller queues.

Quick Q&A

What is an alternative of a Message queue ?

A simple SQL database can serve as an alternate, where every-time a publisher sends a message, the message is stored in the database, and the consumer can change the status of message whenever it starts to process it. This system will however be only useful if the number of messages per second are less (in terms of 1000s per sec). SQL database won't be able to handle larger loads and would increase the overall latency of the system.

Do message queues provide data retention ?

Typical message queues don't provide message retention, that is once a message has been successfully processed, the message is no longer available to be processed (an additional storage medium can be attached to keep backup of messages). However, in the next section we will read about Kafka, which allows messages to be reprocessed, and acts as a storage medium of messages with some retention.

For More Reading

Last updated