Hacker News Reader

Comment by FridgeSeal | Hacker News Reader

I’m personally a fan of Kafka. I think the design of persisting the messages, and tracking offsets for progress instead of message acknowledgments is a much cleaner and more versatile design.

You can get all the same advantages of message acknowledgments, but now you can also replay queues, let different applications use the messages (handy for cross cutting event/notification systems) and you get better scaling properties-which doesn’t hurt at the small scale, and provides further scaling when you need it.

View on HN

> You can get all the same advantages of message acknowledgments, but now you can also replay queues

with rmq you can reject/nack a message and have it put back on the queue. rmq is not well suited for long term historical retention inside queues a-la kafka's logs but it is possible to do.

> let different applications use the messages (handy for cross cutting event/notification systems)

rmq also does a publish once and fanout to multiple queues to support this. data is replicated so that could be a deal breaker, but it is possible.

how often have you had to diagnose a stuck consumer or some other kind of offset glitch where a consumer is unable to resume where it left off?

not knocking kafka here but I do think it is a tool you should reach for when you need to solve a very hyper focused problem, while rabbit is a tool more suited to most cases where queuing is required. kafka is a code smell in a lot of organizations from my experience - most do not need it.

> afka is a code smell in a lot of organizations from my experience - most do not need it.

Kafka is really nice if you don't care that much about latency during peak load and you don't have absurd processing times for messages.

Kafka can sustain sub 20ms at millions or even billions per second scale. Processing time delays is bad consumer code and partition design smell. Aka , your consumer shouldnt depend on a slower resource within an ordering domain. This can also be mitigated with an async consumer

These sound like consumer issues to me.

Kafka had been extremely reliable with latency, even under load in my experience.

If you’ve got badly lagging consumers that are trying to read from very old points in the topic while everyone else is at the head, you’ll definitely see some increased resource usage, but again, that’s mostly a consumer issue, and I’ve need seen performance degrades that much.

If you're concerned about latency you might want to consider zeromq. Stream processing doesn't really have a time expectation to it.

> with rmq you can reject/nack a message and have it put back on the queue

I know other systems have semi-similar mechanisms, however most of them retain the “someone is the sole owner of this message” style design, which I think is fundamentally limiting. Owning application dies, is it acked or not? Acks but never gets around to putting it back on the queue? Who takes priority if 2 separate applications wish to watch the same stream of events?

I think Kafka’s “nobody owns it, acks are consumer group level” give you the same advantages for the application itself, without a number of the more difficult complications.

> rmq also does a publish once and fanout to multiple queues to support this

Which is probably fine for small volume or velocity topics, but is going to cause all sorts of load issues at higher scale.

> Who takes priority if 2 separate applications wish to watch the same stream of events?

each app would get its own queue, the messages would hit a fanout exchange that would route the same message to both queues.

> now you can also replay queues

yeahnah, that leads to people treating queues like databases (I'm looking at you new york times, you know what you did wrong)

its either a queue, or a pubsub, either way its ephemeral. Once its gone, it should stay gone. thats what database, object stores or filesystems are for.

Kafka is a beast, has lots of bells and whistles and grinds to a halt when you look at it funny. Yes, it can scale, but also it can just sulk.

rabbit has it's own set of problems, and frankly it's probably not choose either anymore.

> (I'm looking at you new york times, you know what you did wrong)

You're going to have to be a tiny bit more specific here. NYT is THE factory of wrongness for sure. In every dimension. Are we talking "yellow cake" wrong, or somewhere else on the severity of f'up scale...

They wrote a post on how they disabled the deletion and compaction of the data in Kafka and used it as the source of truth.

https://www.confluent.io/blog/publishing-apache-kafka-new-yo...

^ this.

All they needed was a database, or possibly a DB that supports row signing. I mean actually they could have done it with git. They don't publish that many stories an hour.

Everything about this setup is just plain wrong, and to then boast about it, absolute madness.

What would you choose today ?

It depends on the context.

Currently I'm using DDS, specifically from eprosma. I would avoid that implementation unless you're using java.

I really like NATS. However I would probably use what every is bundled with the cloud system I'm using, unless its super critical.

MQTT is quite nice for things, as is rabbit.

> You can get all the same advantages of message acknowledgments.

Maybe 95% of cases, but not all.

Long message processing time really kills kafka in a way it doesn't kill Rabbit Mq. Combine it with inherent read paralelism being limited to the number of partitions. Add in high variability of message rates and bingo, that's like 90% of the issues I've had with kafka over the years.

Reply on news.ycombinator.com