Starting with Kafka | Hacker News Reader

Starting with Kafka

blog.tinybird.co

105 pointscarlosap14 comments

Using confluent platform community (free) gives you a working env with Kafka/Zookeeper/SchemaRegistry/Connect immediately without having to bother with setting up clusters, etc. Beyond that, yes, it's a pain to properly setup and I think most of the time you're looking at managed options unless you want to invest in a SRE team.

In most cases, a single process app sending data to a file and a small HTTP server serving it will perform much better with less downtime.

As I understand it, this is where Redpanda enters the picture. Maybe others here disagree?

https://softwareengineeringdaily.com/2021/01/22/redpanda-kaf...

https://github.com/vectorizedio/redpanda

https://vectorized.io/

Isn‘t redpanda a reimplementation of kafka without the jvm and zookeeper? I mean how should they api compatible if they don‘t implement the same concepts.

At least it's what their homepage says and the above article suggests. Kafka 2.8.0 preview removes the need to run Zookeeper, as well, though.

This [1] could be a solution to most of author’s problem. Simple to use / few moving parts, very high perf, native HTTP produce and consume, clear and simple processing model with strong guarantees, among other things.

[1] https://github.com/dataptive/styx

Kafka Streams are also amazing, and would have solved many problems I've encountered in the past better than our microservices moving data around themselves.

Would you mind expanding the use cases you would have used Kafka streams for? And why no other solutions could have helped? (Performance or complexity etc)

> And why no other solutions could have helped?

Where did that idea come from? He literally wrote how they did it without streams by moving data between their Microservices, likely with synchronous http calls. He would've just preferred to just write a stream instead of the services.

Right! Except we used Thrift, and had around 100 services.

Awesome work! How long did it take to finish this project?

There have been like three iterations until we've been able to actually release an stable integration that deals well with high traffic. All together, almost 6 months.

i thought it was about kafka's work.

I guess that it is part of the curse of knowledge.

Even the article starts by

"I just want to share my thoughts on Kafka after using it for a few months, always from a practical point of view. I don’t know anything more than the basics about Kafka internals. "

It is just confusing for anyone that does not know that Apache Kafka is a software application to process queues.

The article is named "Starting with Kafka" but assumes that you have knowledge of what it is, its use cases, and other not-so-basic information.

I think if someone were to write an article on the writer Kafka they would not phrase the headline "Starting with ... " but something like "Reading Kafka" or "How to read Kafka". I am not a native speaker, though.

Reply on news.ycombinator.com