Hacker News Reader

Comment by agentgt | Hacker News Reader

I think in large part why people dislike Kafka is that they don't really need Kafka (and the complexity that comes with it).

Don't get me wrong Kafka is good tech if really need that level of throughput but I honestly think most companies don't have that much data and/or just putting too much in the pipe. But they go with Kafka anyway I guess to "CYA" for future scaling only to find out Kafka is complicated.

I mentioned this earlier (a couple days ago https://news.ycombinator.com/item?id=12520159) for someone using Kafka for a logging aggregation system only to drop it for ZMQ.

My other point is if your endpoint isn't fast enough it doesn't really matter what your pipe is. Pick an easy to use pipe first (like RMQ) and worry about scaling the endpoints.

View on HN

I agree! We've had success using redis as our pipe... with fast consumers we regularly push MB/s of data... no problem. And redis is super easy to manage.

Yes Redis is a solid choice! And it like RabbitMQ gets the constant "single point of failure" critique (we get this crap from tech partners when we tell them how heavily we use RMQ)... oh so I have should have chosen something even more complicated so that we have multiple points of failure (I'm sort of joking and being sarcastic)?

Kafka and ZMQ though are solid choices for intensive stuff... just be prepared to build extra stuff and don't complain when it gets hard :)

Serious question, what exactly is the "complexity that comes with it"?

ZooKeeper. AIUI, creating your first set of ZooKeeper nodes is a serious PITA. My employer gave up before even started and decided to pay Cloudera for CDH even though we have _zero_ need for Hadoop.

this isn't my experience at all. setting up zookeeper is super easy and ZK is rock solid. there really is no need to pay anyone to setup ZK, it takes maybe an afternoon to automate setting up a cluster

This. I setup ZK and Kafka over a year ago. It took about a day, and it's been running ever since.

Another serious question (and I mean this as un-snarky as I can). Are people just terrible at devops? I'm wondering if the people who think it is "too complex" grew up in a heroku world?

Honest question, how is ZooKeeper more painful to manage than Kafka itself? ZooKeeper is one of the components from the Hadoop ecosystem that I had never trouble with.

Reply on news.ycombinator.com