Hacker News Reader

Comment by josephg | Hacker News Reader

Can you make your point without the appeal to a (missing) authority? There is no standards document I've ever seen which names kafka explicitly.

Kafka has some benefits (lovely API) and some tradeoffs (hard to maintain, zookeeper is a bit of a mess). But there are obviously a lot of other ways to build successful, scalable software. WhatsApp was implemented on top of Erlang, and scaled to over 500m users when they had only 35 employees.

I want to disagree with your claim that kafka should be considered the default option - but you haven't dignified your (controversial) claim with any actual arguments. There's nothing here to even disagree with.

View on HN

It should be considered the default option because everything works with it out of the box and for pretty much all Big Data folk it's considered table stakes.

All major streaming event processors are designed around it or work seamlessly with it (Spark, Beam, Flink, Storm, etc).

There are pre-existing integrations for all long term storage options (S3, GCS, Swift, etc).

Maybe it could have been elaborated in the original comment but it shouldn't need to be - all of this is about 10 minutes research away from anyone that wants to understand why Kafka is considered the core of modern data architectures.

WhatsApp isn't really a relevant counter-example and if anything reveals the misconception yourself and many others hold about Kafka which is that it's a messaging solution. It's not. You can bastardize one on top of it but really, don't. WhatsApp was built around RabbitMQ which -is- a messaging solution (one of the more complex ones, crazy complicated compared to Kafka at least). Kafka -doesn't- compete with RabbitMQ any more than MongoDB competes with PostgreSQL. Apples and oranges if you are trying to do anything but the most basic stuff.

Why would you want to disagree with Kafka being the default option? What could you reasonably advocate for in its place?

>Kafka is considered the core of modern data architectures.

Ah, yes, I like synergyc technobullshit, too.

If Erlang itself could manage a damn load of >500M concurrent users (if was made for that), then everything else is sucks hard. A lot.

Seriously, half of the "devops" today know shit about the underlying systems.

> Why would you want to disagree with Kafka being the default option? What could you reasonably advocate for in its place?

Very few people roll out of bed and think "my application needs a high performance distributed event stream". For every problem that can be solved with kafka, there are other solutions that don't involve kafka. Many of the alternate solutions don't involve event streams at all. Are they better? I don't know. That question is unknowable without looking into the problems in particular, and looking at what other technology is out there. And I say this with a lot of fondness for kafka. I really like it. I just think there are always technical tradeoffs to make.

> any more than MongoDB competes with PostgreSQL

MongoDB and Postgres do compete with each other though! They work differently - so part of the competition isn't around features; its around how your people think about your data. "Think of your data in a way that makes sense for mongodb!" say mongodb sales reps. "Think of your problem as a message queue" say fans of rabbitmq. They aren't wrong in the general case. But there are lots of problems where I could use kafka, or I could use redis, or I could use a naked postgres instance. You need a much more complex heuristic to decide what technology to adopt than "Should I use kafka?" -> "Yes."[1]

And I say all that as a big fan of kafka. I think its API is delightful and just today I was plagerizing its API design to sort out backpressure in a system I'm building.

> What could you reasonably advocate for in its place?

I haven't used it but by all reports RedPanda is lovely: https://vectorized.io/

[1] Though I'm convinced the answer to "Should I use mongodb" is always "No."

I think we are agreeing on a different point than I was making.

Kafka isn't the solution to all problems, like I noted with WhatsApp not being the best fit. Though you can shoehorn it into the same problem space, this is what LINE Messenger did which is of equal scale to WhatsApp and runs entirely on Kafka. So I completely agree Kafka isn't the solution to all A needs to talk to B problems, even when Kafka might be able to do the job.

I think what was perhaps lost is that I am not talking about adding Kafka to an arbitrary application but rather operating a Big Data pipeline - i.e data from applications, devices, sensors, whatever and transporting it to data lake, warehouse and online stores like Druid.

While you can use Kafka in other contexts this is it's original purpose and main use case and where I don't think any other piece of software replaces it right now. Pulsar is on the way there but not quite ready yet. Redpanda is just Kafka without being the real thing - maybe you save some CPU, maybe you hit an incompatibility, really it's not worth the effort because Kafka in 99% of use cases is network bound as it's already efficient enough. (Not always true, there are some CPU bound cases with tons of producers but they are rare)

If you chose anything else you are signing up for writing all of these integrations yourself and that just doesn't make sense. If you pick Kafka you get all the hookups for free and you only need to write custom code for your existing custom applications.

If say you were to want something more "lightweight" sure you could use Redis PUBSUB but now you are writing code that dumps from Redis PUBSUB into S3 for archival instead of just pulling something like Secor or Kafka Connect and being done with it.

Perhaps I should have been more clear that I think it's only in this specific domain/use case that I think it's completely dominant.

redpanda is not just a performance (latency) gain. The main thing people pay us money for is not performance, it is operational simplicity - no zk, no jvm, nothing to tune (https://vectorized.io/docs/autotune) . one binary you can `scp` around a few servers is really quite easy to work with.

A different example of a system that can start small is Joe Hellertein's research (https://dsf.berkeley.edu/jmh/papers/anna_ieee18.pdf)

the point is that operational simplicity, often trumps performance for most ppl. In fact, that's what the parent argument suggests redpanda for, the parent comment was not about performance.

I get that but often times you need or want ZK anyway. If you are running Druid, Pinot, Hadoop or anything else Big Data they generally have a ZK dependency. If you need service discovery, consensus or config storage ZK is generally great for all of those usecases too.

I think if you want a completely standalone distributed log for an application then it could be a good fit but it's not going to replace Kafka as the goto for Big Data pipelines because it's advantages aren't worth it in this context. (i.e you are already running ZK, you are throughput and not latency bound, rest of the ecosystem is JVM anyway, etc)

There is work in progress to remove the Zookeeper dependency:

https://www.confluent.io/blog/removing-zookeeper-dependency-...

Reply on news.ycombinator.com