Hacker News Reader

Comment by dividedbyzero | Hacker News Reader

NATS (https://nats.io/) is another option, though I'm not sure if it's still considered a viable Kafka replacement.

It’s true FOSS, and the server is standalone Go binary that’s so small it can even be embedded. Lots of language bindings for clients. Has persistence, durability, and nicely aligns into a raft-like cluster in a DC without a separate orchestrator.

I’m a big fan – never understood why it’s not at the top of the list in these tech reviews.

Rabbitmq is FOSS, has lots of language bindings. It has persistence, durability, and doesn't require a separate orchestrator.

I was mostly comparing against Kafka but yes I should def take a look at RabbitMQ again. I remember there was some reason it wasn’t a good fit for me but can’t recall what it was.

Are the horizontal scaling issues solved now?

Yes, if you use quorum queues

NATS is something else, but it's awesome. It has awesome throughput and latency out of the box (without Jetstream), while using little resources.

I'd recommend considering it, especially as an alternative to RabbitMQ.

I only tested NATS using JetStream and I struggled with the throughput in Python. I probably used it wrong. But your comment may imply that jetstream is slow.

we used it for some low latency stuff in python. it was about 10ms to enqueue at worst. However we were using raw NATS, and had a clear SLA that meant that the queue was allowed to be offline or messages lost, so long as we notified the right services.

I think sometimes the client bindings are/were in need of improvement.

As an example, the C# API was originally very 'go-like' and written to .NET Framework, didn't take advantage of a lot of newer features... to the point a 3rd party client was able to get somewhere between 3-4x the throughput. This is now finally being rectified with a new C# client, however it wouldn't surprise me if other languages have similar pains.

I haven't tested JetStream but my general understanding is that you do have to be mindful of the different options for publishing; especially in the case of JetStream, a synchronous publish call can be relatively time consuming; it's better to async publish and (ab)use the future for flow control as needed.

I’m not familiar with the Python lib but it could be waiting for streams to acknowledge each message reception/persistence before sending the next one. Some clients allow transactions to run in parallel e.g. with futures.

I didn’t mean to imply that Jetstream is slow. It’s just that I did my benchmarks without it. On a local PC, with 10 KiB messages sent (synchronously) in a loop, I could transfer 3.2 GiB over 5 seconds with 0.2 nanoseconds latency. Performing the same test with RabbitMQ, I got even better throughput out of the box, but way worse latency.

Those numbers are for server 2.9.6 and .NET client 1.0.8.

If I'm not buying a message bus in as a service, then NATS is great for pub/sub and or message passing system

it is simple to configure, has good documentation, and excellent integration into most languages. It guarantees uptime, and thats about it. It clusters really well, so you can swap out instances, or scale in/out as you need.

Different set of promises. NATS is great but has a different tradeoff bargain from Rabbit or Kafka.

Could you expand on this a bit more? I am curious.

NATs has a decent-ish guide here: https://docs.nats.io/nats-concepts/overview/compare-nats

A few things they get wrong mostly about Rabbit:

+ RabbitMQ does support replay, and also has a memory only mode which will support persistance in a cluster

+ RabbitMQ doesn't have that sensitive of a latency between cluster members (no more sensitive than NATS in some setups).

+ RabbitMQ also supports Prometheus

A good (but incomplete) rule of thumb:

+ Kafka is a distributed Append-Only-Log that looks like a message bus. It allows time travel but has very simple server semantics.

+ RabbitMQ is a true message broker with all the abilities and patterns therein. It comes with more complexity.

+ NATs is primarily a straight forward streaming messaging platform.

Also consider Redis' message queue mode, zeromq, mqtt brokers (Eclipse Mosquitto) and the option of just not using a message broker/queue system. Even as someone who really likes the pubsub pattern, there's a good chance you don't need it and you may be heading to a distributed monolith antipattern.

> distributed monolith antipattern

Beauty is in the eye of the beholder, I guess. (I would definitely not consider a distributed monolith an antipattern)

Also wanted to add NSQ to your very good list.

Thanks for this link -- really interesting.

I like the suggestion to rethink whether you actually need to be doing asynchronous computing with a message broker/queue/stream or whether you can represent your work another way.

Also see https://www.youtube.com/watch?v=C4BnJ5QLeTY

Reply on news.ycombinator.com