Hacker News Reader

Comment by uptown | Hacker News Reader

Conceptually, how do these types of pub/sub messaging systems work at scale? How does the number of subscribers impact the efficiency of updates being distributed to the subscribers? Is the server pushing these messages to them all simultaneously, or is there some logic that might result in one subscriber receiving an update faster than another? Is the publishing server opening up a ton of ports to handle the communication, or from a networking/ports perspective how is this handled?

View on HN

Consumers request blocks of messages starting at a given offset. If the consumers are keeping up then memory pages of message can be served directly from the RAM buffer they were produced into, since the broker is constantly appending new messages a memory-mapped file. Producers can request acknowledgements, which they get after the buffer has been sync'd to disk.

In practice a broker uses very little CPU even with compression and compaction happening, and is bound by the disk or network speed--our brokers on AWS run about 80MByte/sec with room on top for bursts to ~110. Interestingly and in addition, blocks of messages can pass through Kafka without being decompressed because of this design.

Everything happens from one port on the brokers, who are backed with Zookeeper for cluster management; consumers may also connect to Zookeeper or similar to track the latest offset that they have processed.

Publishers are decoupled from brokers which are decoupled from consumers. If a publisher is publishing on multiple topics they can be talking to a whole fleet of brokers and similarly consumers can be reading from many servers.

I'd also be careful when saying "these types" of pub/sub messaging systems as Kafka, while not unique, is pretty lonely in the space it targets. Most pub/sub systems don't make nearly the data garauntees that Kafka does (and therefore Kafka isn't appropriate for some pub/sub patterns).

With Kafka specifically the consumers of messages use standard TCP to receive messages by pulling them from Kafka. Kafka is not a push system. Consumers can come and go in and ad-hoc fashion.

So I guess to address user volume, you just scale horizontally increasing the number of server instances in your cluster?

Linkedin blogged about how kafka is their data backbone.

https://engineering.linkedin.com/kafka/running-kafka-scale

When combined, the Kafka ecosystem at LinkedIn is sent over 800 billion messages per day which amounts to over 175 terabytes of data. Over 650 terabytes of messages are then consumed daily, which is why the ability of Kafka to handle multiple producers and multiple consumers for each topic is important. At the busiest times of day, we are receiving over 13 million messages per second, or 2.75 gigabytes of data per second. To handle all these messages, LinkedIn runs over 1100 Kafka brokers organized into more than 60 clusters.

Yep. Also, you increase partition counts (shards) in your topics, which then lets you place replicas and replica leaders across more server instances.

Reply on news.ycombinator.com