Kafka in a Nutshell | Hacker News Reader

Kafka in a Nutshell

sookocheff.com

78 pointssoofaloofa20 comments

Are there any alternatives to kafka that are also modeled after the "message queue as a log" concept? In particular, I'd like to be able to reconsume arbitrary ranges of 'already processed' log/event/message data as well as trust the log as the ultimate deterministic source of truth for the state of the system.

The reason I don't just use kafka is because the "quorum" style scaling is overkill for our needs since the app will be for internal use only and will likely never exceed more than 1000 simultaneous users. Also, I've heard that zookeeper (required to use kafka) has its own technical overhead that I'd like to avoid dealing with if possible.

I'd give Elasticsearch a go. Easy to install, easy to run, easy to use, easy to scale into a cluster if you need it.

Try Eventstore

Give a try to kafka ! It worth the value and in practice zookeeper is not a big burden.

Google Cloud Pub/Sub perhaps https://cloud.google.com/pubsub/

Conceptually, how do these types of pub/sub messaging systems work at scale? How does the number of subscribers impact the efficiency of updates being distributed to the subscribers? Is the server pushing these messages to them all simultaneously, or is there some logic that might result in one subscriber receiving an update faster than another? Is the publishing server opening up a ton of ports to handle the communication, or from a networking/ports perspective how is this handled?

With Kafka specifically the consumers of messages use standard TCP to receive messages by pulling them from Kafka. Kafka is not a push system. Consumers can come and go in and ad-hoc fashion.

So I guess to address user volume, you just scale horizontally increasing the number of server instances in your cluster?

Yep. Also, you increase partition counts (shards) in your topics, which then lets you place replicas and replica leaders across more server instances.

Linkedin blogged about how kafka is their data backbone.

https://engineering.linkedin.com/kafka/running-kafka-scale

When combined, the Kafka ecosystem at LinkedIn is sent over 800 billion messages per day which amounts to over 175 terabytes of data. Over 650 terabytes of messages are then consumed daily, which is why the ability of Kafka to handle multiple producers and multiple consumers for each topic is important. At the busiest times of day, we are receiving over 13 million messages per second, or 2.75 gigabytes of data per second. To handle all these messages, LinkedIn runs over 1100 Kafka brokers organized into more than 60 clusters.

Publishers are decoupled from brokers which are decoupled from consumers. If a publisher is publishing on multiple topics they can be talking to a whole fleet of brokers and similarly consumers can be reading from many servers.

I'd also be careful when saying "these types" of pub/sub messaging systems as Kafka, while not unique, is pretty lonely in the space it targets. Most pub/sub systems don't make nearly the data garauntees that Kafka does (and therefore Kafka isn't appropriate for some pub/sub patterns).

Consumers request blocks of messages starting at a given offset. If the consumers are keeping up then memory pages of message can be served directly from the RAM buffer they were produced into, since the broker is constantly appending new messages a memory-mapped file. Producers can request acknowledgements, which they get after the buffer has been sync'd to disk.

In practice a broker uses very little CPU even with compression and compaction happening, and is bound by the disk or network speed--our brokers on AWS run about 80MByte/sec with room on top for bursts to ~110. Interestingly and in addition, blocks of messages can pass through Kafka without being decompressed because of this design.

Everything happens from one port on the brokers, who are backed with Zookeeper for cluster management; consumers may also connect to Zookeeper or similar to track the latest offset that they have processed.

Great post, but one thing to note is that the number of partitions DOES NOT need to be equivalent to the number of consumers. A consumer can reasonably consume multiple partitions, as most do. On the other hand, it does provide an upper bound to the number of consumers in one group, as a consumer can only reasonably consume as few as one of the partitions.

so do you drop the idea of partitions completely because you have shown its structure proves a soft, reasonable, upperbound?

implying overworking

what do you replace it with?

what problem was 'use the same number of partitions as consumers' trying to fix?

or if you keep it, how do you measure the optimal partition count?

Is AWS SNS/SQS equivalent to Kafka? And what are the differences, advantages/disadvantages of either one?

Kinesis is the AWS alternative to Kafka, minus some features.

Kinesis is for streaming computations. It's closer to Apache Storm (which typically uses Kafka) than Kafka.

You might be thinking of Simple Workflow Service, which is closer to Storm. Grandparent had it right.

Agree with Cieplak. Kinesis is very Kafka-esque, with less flexibility (which makes sense for a managed service).

Producer/Consumer semantics are pretty similar. Partitions in Kafka are Shards in Kinesis terminology.

One big difference is retention period in Kinesis has a hard limit of 24 hours (no way to request increase on this limit).

Kinesis IMO is easier to use being a managed service. I have performed a Kafka to Kinesis migration & have found Kinesis easier to use. Plus, AWS Lambda makes consuming Kinesis a breeze (if your usecase suits it).

Reply on news.ycombinator.com