> For the complete documentation index, see [llms.txt](https://ricardomol.gitbook.io/notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ricardomol.gitbook.io/notes/assorted/celery/brokers.md).

# Brokers

## Definition

**Intermediary platform when it comes to processing communication between two applications**

Functions:

* **Decouple** message publisher and consumer
* **Store** the messages
* **Routing** of messages
* **Monitoring** and management of messages

## Messaging

A way of exchanging messages from point A to point B or many points C.

Enables distributed communication that is loosely coupled.

A message sender sends a message to a destination, and the message recipient can retrieve the message from the destination. However, the **sender and the receiver do not have to be available at the same time** in order to communicate. (Example: email).

Example scenarios:

* Send data to many applications without calling their API directly into your application
* Want to do things in certain order like a transactional system
* Monitor data feeds like the number of registrations in any application

The component which receives the message from the sender and recipient retrieves the message from a message broker or messaging middleware.

## RabbitMQ

Messaging system modeled on the **AMQP** (Advanced Message Queuing Protocol) standard

**Language-neutral** and while using it you can write and read to them in any language just like you would while using TCP or HTTP

## Redis

In-memory **data structure store.**

Used as a database, cache and **message broker**.

Supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes with radius queries and streams.

Has **built-in replication**,  transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster.

Works with an **in-memory dataset**. Depending on your use case, you can persist it either by dumping the dataset to disk every once in a while, or by appending each command to a log.

You can use Redis from most programming languages out there.

## RabbitMQ vs Redis

### RabbitMQ

#### **Pros**

* Most developers agree that RabbitMQ is the best message broker on the planet today. It can be configured to use SSL thereby providing additional layer of security.
* The **Interface**, much more versatile when compared to Redis as a message broker.
* **Clustering** and is very good at it.
* **Scales** to around 500,000+ messages a second.

#### **Cons**

* Needs **Erlang**
* Minimal configuration that can be done through code. (Configuring RabbitMQ is something that must be done first before even implementing your task queue.)
* Since our comparison is with Redis one can also say that for non extensive message brokers Redis obtains a slight edge due to its ability to multi-task.

### Redis

**Pros**

* Since it has dealings with a system's main memory Redis is **fast** and also scalable.
* Extremely **easy to set up, use and deploy**.
* Provides in-memory and advanced key-value cache.

#### **Cons**

* Redis was created with a different intentions and not for being a message broker. It **does support basic operations as a message broker** however for powerful message routing Redis is not the preferred option.
* **High latency in dealing with large messages**. Redis is better suited for small messages.

Personally I use Redis to implement very basic straight forward message queues because it doubles up as key value datastore but for huge and scalable applications with huge messages RabbitMQ is the way to go. As in most situations in life you can totally use a different message broker perhaps something like Kafka, ActiveMQ, ZeroMQ etc. A side by side comparison of them all is an article for another day

## Kafka 

#### Distributed streaming platform

A streaming platform has **3 key capabilities**:

* P**ublish and subscribe to streams of records**, similar to a message queue or enterprise messaging system.
* **Store streams of records** in a fault-tolerant durable way.
* **Process streams of records** as they occur.

Generally used for 2 broad classes of applications:

* **Building real-time streaming data pipelines** that reliably get data between systems or applications
* **Building real-time streaming applications that transform or react to the streams of data**

To understand how Kafka does these things, let's dive in and explore Kafka's capabilities from the bottom up.

First a few concepts:

* Kafka is run as a **cluster** on one or more servers that can span multiple datacenters.
* The Kafka cluster stores streams of records in categories called **topics**.
* Each **record** consists of a key, a value, and a timestamp.

Kafka has 4 core APIs:

* **Producer API** allows an application to publish a stream of records to one or more Kafka topics.
* **Consumer AP**I allows an application to subscribe to one or more topics and process the stream of records produced to them.
* **Streams API** allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams.
* **Connector API** allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table.

![](https://kafka.apache.org/22/images/kafka-apis.png)

The communication between the clients and the servers is done with a simple, high-performance, language agnostic [TCP protocol](https://kafka.apache.org/protocol.html). This protocol is versioned and maintains backwards compatibility with older version. We provide a Java client for Kafka, but clients are available in [many languages](https://cwiki.apache.org/confluence/display/KAFKA/Clients).