Kafka RabbitMQ Features
RabbitMQ and Kafka – An Analysis
According to the recent analysis of Nokia Bell Labs had analyzed the relative features of Apache Kafka and RabbitMQ. Both are open-source, commercially supported systems that publish and subscribe messages.
Philippe Dobbelaere and Kyumars sheikh Esmaili are researchers and in a publication on the Cornell University Library’s Computer Science section, they proposed a common comparison framework, designed to cover the core functionalities of systems that publish and subscribe. They considered the advantages of both programs and designed the appropriate criteria. They concluded that both have very different histories, design goals and quite distinct features.
Kafka producers are now able to publish batches of messages to a topic-specific disk-based append log, and any amount of consumers can pull the stored messages with the use of an indexing mechanism. RabbbitMQ publishes batches of messages using a routing key. The messages go to a network of exchanges where routing decisions are made. Consumers can obtain messages through a push preferred or pull mechanism.
Kafka RabbitMQ Messaging Systems
RabbitMQ is more similar to classic messaging systems, with the AMQP protocol, offering a mechanism for flexible routing, which uses the exchanges/binding notions. It is now able to manage most of the consumption bookkeeping, but its main goal is to handle messages in memory. RabibbitMQs queue logic is designed primarily for empty-or-nearly-empty queues. Kafka has moved away from the classic messaging systems. It uses partitioning extensively, at the expense of data order. To allow for replayability, the queues are logical views on persisted logs but still allow manual retention policies. It is aimed at high-throughput and for consumers using varying speeds. In addition to this Kafka applies several effective techniques for optimization including aggressive batching and their reliance on persistent data structures and a cache for the OS page.
Kafka RabbitMQ – Latency Difference
The researchers found that both systems are able to deliver low-latency results. The difference between at most once and at least once delivery modes is not significant in the case of RabbitMQ. However, with Kafka, latency is about twice as large for the ‘at least once’ mode, and if it is reading from a disk, the latency can increase substantially.
Kafka RabbitMQ – Performance Difference
For the most basic setup (a single node, producer/channel, partition, and no replication) RabbitMQ performed better than Kafka. However, the researchers found that increasing the partition count on the same node, significantly improved Kafka’s performance. This further demonstrates Kafka’s exceptional scalability. Doing the same in RabbitMQ only moderately improved its performance.
Both systems are able to scale further by partitioning rows on multiple nodes. RabbitMQ requires additional special logic, but with Kafka this is free. The performance of both RabbitMQ and Kafka is drastically reduced when replicating. RabbitMQ by 50% and Kafka by 75%.
See detail Kafka vs RabbitMQ comparison
Some people prefer Kafka. RabbitMQ on the other hand is preferred by those who consider it faster.
The full paper is available from Cornell University’s website.