I've written this article for explaining what's the main difference between Kafka and RabbitMQ. I know there are a lot of articles try to compare them on their capabilities like the message routing, performance, persistence, etc. I like this post the most, which gives a precise and fair summary to both of them.

Nevertheless, those articles all stand on the server side to consider how to choose Kafka or RabbitMQ. They rarely consider from the client side, i.e the story on workers. In my opinion, both Kafka and RabbitMQ itself work very well on every use case. The most common thing we encounter is not that the server cannot handle messages, but the workers are too slow to process messages. If the workers indeed cannot afford the workload, we usually adopt the horizontal scaling to increase the throughput.

RabbitMQ

I demonstrate this scenario as follows:

The MQ represents either RabbitMQ or other message queue systems. In this case, we can extend the amount of workers to handle more messages at the same time. However, the order of messages cannot be preserved. RabbitMQ treats every message as the standalone entity and can be dispatched to any one of unified workers. In fact, you can setup a complex exchange and routing rule to ensure the message ordering manually. If you do so, you will introduce the complexity in a system and significantly make a tightly coupling between producers and consumers.

Kafka

How to deal with this problem? I mean how does Kafka preserve the message order. The illustration is:

The magic is a producer still publishes a message to a topic, but Kafka pushes the message into the corresponding partition based on the key. Consumers can be a consumer group, and the consumers in the same group will be dispatched to the certain partition(s). In other words, the same kind of messages are processed on the same consumer to make sure the correctness of orderings. Furthermore, the participants in a consumer groups can be dynamically adjusted to sort out the run-time workload.

Conclusion

From my point of view, if the user scenario is the producer generates lots of messages, and you have to add more consumers to digest them, you should use Kafka. Otherwise, Kafka and RabbitMQ both are great tools, you can choose according to the technology stack in your organization.