How to use RabbitMQ in service integration

I concure. I like RMQ because I know how to configure and administer it but I would never trust anyone with it. My first exposure to it was on a project where the lead architect failed to read the documentation let alone understand any of it. Several years later I was able to fix it and then left it in the hands of some other incompetents.

Not OP, but that is my experience. It worked like a rock on a single server. Clustering brought us issues rooting from its complexity. Split brain scenario, corruption of the Mnesia database and such. We went back to single server mode.

Well, they’re two different things. AMQP 1.0 is everyone to everyone messaging where every party can be a consumer and a server. RabbitMQ, traditionally, is a queue. You add a message, you take it and lock it, if no processing confirmation, it is released back for someone else to process. Kafka is an append only log. You put a message in and consumers just roll over them. Rabbit/amqp is random access, Kafka sucks at random access. With amqp, you’ll have hundreds/thousands of queues, this may be difficult with Kafka.

You’d use Kafka more as an unbounded buffer and build different paradigms on top of it. It not unusual to ingest 100s if mbits of data into kafka, potentially saturating the network while also reading that much out. Amqp is better for large number of queues where each queue has less messages in. Think mqtt, websockets - many, many consumers.

It would be reasonable to use both next to each other.

But I’d never go for rabbitmq. I’d go for azure servicebus or artemis with qpid.

> I'd love to hear what exact scenarios it does actually make sense to use Kafka for instead of Rabbit or AMQP1.0 :)

If order of message processing matters, then Kafka is better suited then AMQP. For example, In a distributed application for money transfers, if AMQP used, message order will be lost and some problems will occur in the following scenario:

User A with an accound of $1000 makes order for two transfers T1 ($600) and T2 ($500)

  - Rabbit delivers T1 to server1, before processing message, server1 enters a full GC.
  - Rabbit delivers T2 to server2 and server2 processes message immediately, now User A's account have $500  
  - Server1 resumes its life after the end of GC, but fails to process T1 since account's balance is less than required amount.

However, it is T2 that should have failed because User A ordered T1 first and T2 after.

In Kafka, when user account identifier is used for partitioning key, all User A's messages will be processed by same consumer (i.e server1), so even if server1 enters a full GC, that is OK, since T2 will be processed after T1.

Kafka was designed for scale and developed at LinkedIn to meet their extremely high throughput. It's a distributed log, basically writing to an append-only file that's partitioned by a hash of a key (that can be set on every message).

It makes the brokers as dumb as possible to optimize for performance and the logic sits in the client. You can ask to read back from the log at any point or at the current tail. Acknowledging messages is just writing a bookmark to another topic saying where you last read up to, or you can keep track of it yourself somewhere else.

You can always build more complex logic on top which Confluent has done with things like ksqldb.

Kafka is distinctly different from enterprise messaging systems like AMQP.

I generally think of messaging systems falling into 4 distinct categores: PubSub, Streaming, Queues and Enterprise Messaging Systems.

PubSub sytems are focused on non-durable (usually), low latency messaging generally without acknowledgements and generally at-most-once. i.e things like Redis PUBSUB, NATS, etc

Queues are generally focused on fanout to multiple consumers with at-least-once processing of durable messages with acknowledgements. i.e Celery/Sidekiq, Que, AWS SQS.

Streaming systems are designed for throughput and usually are based on some form of a distributed log concept. Generally offload offset management to consumers. i.e Kafka, Kinesis

Enterpise Messaging Systems favor flexibility above all else and usually have some mechanism of encoding the flow of data separately from the applications themselves. i.e exchange routing topologies in AMQP as an example. They can generally implement pubsub, queues and direct messaging paradigms. Tradeoffs being poorer availability, complexity and poor performance vs specialised systems. i.e RabbitMQ, HornetMQ, etc.

So you end up using Kafka when it's limitations aren't a problem and you need the throughput. It works best when each and every message in a stream is homogenous as such failure to process a message is unlikely to be independent of failure to process a following message. This alleviates the main drawback of streaming systems which is head of line blocking.

Some cases where it works very well is event streams, data replication/CDC, etc.

DLQ is the way to do this. Nothing silly about it.

One thing to note is the pricing. Clustering is not supported with the micro instance, so if you need it, you must go with at least mq.m5.large [1] and you need three of them, so the total cost is around $620 per month (or closer to $700 for example in eu-central-1).

[1] https://docs.aws.amazon.com/amazon-mq/latest/developer-guide...

I can sympathize with your colleagues. Developing the distributed messaging system is not where the bulk of the cost of running such a system goes - it's the maintenance and debugging that soak up time, money, and tears.

While my past experiences with RabbitMQ in production have been stellar, I can see why a team would be hesitant to add this complexity to their infrastructure.

Not SQS, but Google PubSub: everything was working allright, until one day messages started arriving with horrible latency and vital parts of our service went down. We spent close to two weeks debugging our infrastructure, because _sure it could not be PubSub_. Well, _it was_. I guess nothing too big, a small bit or tweak that changed in their service that didn't propagate to the client library we were using, or god knows what. Now we run RabbitMQ for everything; at least we control our versions and we can confidently blame ourselves when something happens. But RabbitMQ, standalone and clustered, has served us well across different systems for about fifteen years.