Hacker News Reader

Comment by agacera | Hacker News Reader

I can think in some circumtances: 1 - You already have a Redis infrastructure and don't wanna or don't have resources to deploy a full Kafka infrastructure (3 kafka brokers + 3 zookeeper nodes)

2 - Kafka clients are not available (or are poorly available) for every programming language. Redis has a simpler protocol, so it has more/and better clients available and even if you use an exotic language, it is easy to write a client to it (well... easier than Kafka)

3 - Kafka AFAIK does not have any internal cache implementation, so every read is served from disk (+ page cache). This means that Redis Streams will (probably) perform much better for use cases when the consumers need to fetch data from old offsets.

edit: added reason number 3.

View on HN

Regarding point 3, unless your system is under massive memory pressure, no caught-up Kafka consumer should be serviced from disk. Old offsets that are flushed out of memory because you do not have it obviously are served from disk, with essentially linear reads of disk blocks (of consecutive logical addresses if flush sizes are large enough that then can end up on disk in any number of ways, depending on how much the firmware lies, I know) of the requested file.

I really can not see how Redis is going to perform "much better" reading from disk once the entries are no longer in RAM. At that point both Kafka and Redis have to read from disk, and you either have the IOPS to serve all the lagging consumers or you don't. Maybe you have enough of them to service 1 or 2 concurrent reads, maybe 10-12. But for the same messages counts, sizes and concurrent consumers, your workload will become IOPS bound rather fast.

Note: "much better" to me implies 10x+ better, not "my C library read() is 2.3% better than your Java".

re: client support - I dunno, this seems like a pretty comprehensive list to me? I mean, there's even a rust client: https://cwiki.apache.org/confluence/display/KAFKA/Clients

Confluent only officially supports the Java client (and now has a Python, Go and .Net clients as well that I didn't know) and it is really recommended to use the client with the same version of broker due to protocol incompatibilities.

Most Kafka client implementations are open-source projects of their own, this is also true for most redis clients implementations, but again: Kafka protocol is much more complicated than Redis.

I haven't used Kafka with other languages besides Java or Scala, so I can't really say how mature are the other clients.

But my point about how easy is to implement a client for Redis if needed is still valid. =)

They also support a c reference implementation (which is how they get others).

As someone who has been burned by non-Java Kafka drivers, beware the perception of ecosystem support here. The Kafka design pushes a huge amount of complexity onto the client, and in our experience only the Java client deals with this complexity well. We started out using Python clients but eventually moved to Confluent's REST API (wrapping the Java driver) because we had so many problems with it.

Reply on news.ycombinator.com