Confused between RabbitMQ, Apache Kafka, and Redis? Trying to pick your microservices message broker? In this article, we’re diving into microservices and giving you all the information you need to pick the right microservices message broker for your business.
What are microservices?
Individual microservices are services that exist only for a single purpose, are self-contained, and are independent of other instances and services. When you’re building an application in a microservice architectural style, you would be developing a single application made up of two or more small services (microservices). Every microservice is built separately and the finished application is the sum of all your microservices.
Microservices or modules are decoupled from each other but they still have the ability to communicate with each other. A microservice architecture will typically have cross-dependencies, which means that no single service would work without receiving help from other services. There are several ways to interconnect microservices, these include:
- Brokers (eg. RabbitMQ, Kafka, and Redis)
- Remote Prodecure Calls (RPC)
- REST APIS
The biggest advantage of the microservices architecture is that it gets easier for businesses to scale and maintain their application. Development, testing, and updating of individual parts can be carried out continuously and separately. Various types of industries and organizations lare embracing the microservices architecture because it offers a more agile approach to software development and maintenance.
What is Microservices Communication?
Microservices communication can take place in a synchronous or asynchronous manner. When synchronous communication takes place, the caller will wait for a response before it sends the next message, and it operates as a REST protocol on top of HTTP. But when asynchronous communication occurs, the messages are sent without waiting for a response. This is great for distributed systems and generally needs a message broker to manage the messages.
When you’re picking the type of microservices communication you want to use, you’ll have to consider a range of parameters like the way you structure your microservices, the infrastructure you have in place, latency, scale, dependencies and the purpose of the communication.
It might be more complicated for you to establish asynchronous communication and you’d have to stack more components for it, but the benefits of using asynchronous communication for microservices far outweigh the drawbacks.
By definition, asynchronous communication is non-blocking. It even supports better scaling than synchronous operations do. Another major benefit of asynchronous is that if the microservice crashes, asynchronous communication mechanisms offer you a range of recovery techniques and tend to be more effective at dealing with errors pertaining to the crash.
There’s also the fact that when you use brokers instead of a REST protocol, the services receiving communication don’t really need to know each other. You can introduce a new service even after an old one has been running for a long time (which means that it’s better than decoupling services).
Going for asynchronous operations allows you to increase your capability to create a central discovery, monitoring, load balancing, or even policy enforcer in the future. This enables you to have greater flexibility and scalibility, and more capabilities in your code and system building.
How to pick the right message broker
Asychronous communication is generally managed by a broker. Here are a few factors that you should take into account when you’re picking a broker for your asynchronous operations:
- Broker Scale – The number of messages sent every second in the system.
- Data Persistency – The capability of recovering messages.
- Consumer Capability – The broker’s capability to manage one-to-one and/or one-to-many consumers.
Comparing the 3 best microservices message brokers - Redis vs Rabbitmq vs Kafka
Let's compare 3 of the best message brokers microservices i.e. Redis vs Rabbitmq vs Kafka
RabbitMQ (AMQP)
RabbitMQ was one of the first common and best message queue for microservices message brokers to be built. It was released in 2007 and is an open source message broker that delivers messages via both point-to-point as well pub-sub methods by making use of Advanced Message Queuing Protocols (AMQP). RabbitMQ is designed to support complex routing logic.
Some managed services make it possible for you to use RabbitMQ as a SaaS, but you won’t find it as a part of the native major cloud provider stack. This broker supports all the popular languages like Python, Java, .NET, PHP, Ruby, JavaScript, Go, Swift, and others.
You might end up experiencing a few performance issues when you use it in persistent mode.
Scale: This depends on configuration and resources, but you can expect it to be around 50,000 messages per second.
Persistency: RabbitMQ supports both, persistent as well as transient messages.
Consumer Capability: It can manage both, one-to-one and one-to-many consumers.
Apache Kafka
In 2011, LinkedIn built Kafka to deal with high throughput, low latency processing. It is a distributed streaming platform and replicates a publish-subscribe service. data persistency and stores streams of records that enable it to exchange quality messages. Apache Kafka is optimized for ingesting and processing streaming data in real-time. Streaming data is data that is continuously generated by thousands of data sources, which generally send the data records in simultaneously.
Kafka has managed SaaS on Azure, AWS, and Confluent. These are the the creators of and the primary contributors to the Kafka project. Kafka also supports all the widely used languages, such as Python, Java, C/C++, Clojure, .NET, PHP, Ruby, JavaScript, Go, Swift and others.
It is mainly used to create real-time streaming data pipelines and applications that adapt to the data streams.
Scale: Apache Kafka can scale up to send a million messages every second.
Persistency: It supports data persistency.
Consumer Capability: Kafka only supports one-to-many consumers
Redis
Redis is another one of the best message broker for microservices, is somewhat different from other message brokers. It is essentially an in-memory data store that you can use either as a high-performance key-value store or as a message broker. This message broker doesn’t have any data persistency, instead, it dumps its memory into a disk or a database. It is a great choice for real-time data processing.
Earlier, Redis was only a one-to-one message broker. But after Redis 5.0 introduced the pub-sub, it’s capabilities increased and Redis became a one-to-many message broker as well.
Scale: Redis can send up to 1 million messages per second.
Persistency: It does not support data persistency, it is an in-memory data store.
Consumer Capability: It can handle both, one-to-one and one-to-many consumers.
Situational choices
Each of these message brokers are great for specific use cases.
Complex Routing
RabbitMQ is a mature message broker that has a lot of features and capabilities, making it a great option for complex routing. It can even carry out complex routing communications when the required rate isn’t high.
Vast quantities of data
If you’re working with vast volumes of data, Apache Kafka is a great choice. It is a high throughput distributed queue that’s developed to store large volumes of data for extended periods and is brilliant for one-to-many use cases where persistency is needed.
Short-lived messages
Since Redis has an in-memory database, it’s a brilliant fit for use-cases with short-lived messages for which persistence isn’t needed. Due to its extremely fast service and in-memory database, it is good for short retention messages for which persistence isn’t of importance and some level of loss can be handled.
