Brokers
Definition
Intermediary platform when it comes to processing communication between two applications
Functions:
Decouple message publisher and consumer
Store the messages
Routing of messages
Monitoring and management of messages
Messaging
A way of exchanging messages from point A to point B or many points C.
Enables distributed communication that is loosely coupled.
A message sender sends a message to a destination, and the message recipient can retrieve the message from the destination. However, the sender and the receiver do not have to be available at the same time in order to communicate. (Example: email).
Example scenarios:
Send data to many applications without calling their API directly into your application
Want to do things in certain order like a transactional system
Monitor data feeds like the number of registrations in any application
The component which receives the message from the sender and recipient retrieves the message from a message broker or messaging middleware.
RabbitMQ
Messaging system modeled on the AMQP (Advanced Message Queuing Protocol) standard
Language-neutral and while using it you can write and read to them in any language just like you would while using TCP or HTTP
Redis
In-memory data structure store.
Used as a database, cache and message broker.
Supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes with radius queries and streams.
Has built-in replication, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster.
Works with an in-memory dataset. Depending on your use case, you can persist it either by dumping the dataset to disk every once in a while, or by appending each command to a log.
You can use Redis from most programming languages out there.
RabbitMQ vs Redis
RabbitMQ
Pros
Most developers agree that RabbitMQ is the best message broker on the planet today. It can be configured to use SSL thereby providing additional layer of security.
The Interface, much more versatile when compared to Redis as a message broker.
Clustering and is very good at it.
Scales to around 500,000+ messages a second.
Cons
Needs Erlang
Minimal configuration that can be done through code. (Configuring RabbitMQ is something that must be done first before even implementing your task queue.)
Since our comparison is with Redis one can also say that for non extensive message brokers Redis obtains a slight edge due to its ability to multi-task.
Redis
Pros
Since it has dealings with a system's main memory Redis is fast and also scalable.
Extremely easy to set up, use and deploy.
Provides in-memory and advanced key-value cache.
Cons
Redis was created with a different intentions and not for being a message broker. It does support basic operations as a message broker however for powerful message routing Redis is not the preferred option.
High latency in dealing with large messages. Redis is better suited for small messages.
Personally I use Redis to implement very basic straight forward message queues because it doubles up as key value datastore but for huge and scalable applications with huge messages RabbitMQ is the way to go. As in most situations in life you can totally use a different message broker perhaps something like Kafka, ActiveMQ, ZeroMQ etc. A side by side comparison of them all is an article for another day
Kafka
Distributed streaming platform
A streaming platform has 3 key capabilities:
Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system.
Store streams of records in a fault-tolerant durable way.
Process streams of records as they occur.
Generally used for 2 broad classes of applications:
Building real-time streaming data pipelines that reliably get data between systems or applications
Building real-time streaming applications that transform or react to the streams of data
To understand how Kafka does these things, let's dive in and explore Kafka's capabilities from the bottom up.
First a few concepts:
Kafka is run as a cluster on one or more servers that can span multiple datacenters.
The Kafka cluster stores streams of records in categories called topics.
Each record consists of a key, a value, and a timestamp.
Kafka has 4 core APIs:
Producer API allows an application to publish a stream of records to one or more Kafka topics.
Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them.
Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams.
Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table.
Last updated
