Handling 100k consumers with one pulsar topic
streamnative.io
streamnative.io
https://github.com/jgaa/pulsar-test-consumer-scale
I was a bit disappointed by the result. It's known that Pulsar shards it's topics so that one topic is handled by one broker - for subscribers and publishers. However, at least at that time, it appeared as the broker looped trough all it's subscribers in a single thread, effectively enforcing a linear limit on the number of subscribers it could handle on a topic.
I'm just mentioning it, because when I tested, I could definitely not handle 100k shared consumers.
(Eventually, I ended up writing a consumer-proxy that could scale the number of subscribers indefinitely, using just one or a few subscriber-connections to the broker. Unfortunately, it's not open source.)
"Pulsar uses two individual tasks to complete the message-sending process - one for sending the message and the other for the ACK. For each message sent by the consumer, Pulsar generates these two tasks and pushes them to OrderedExecutor. To guarantee the order of messages, Pulsar always adds them to the same thread...As shown in Figure 12, Pulsar generates 200K tasks, all of which are inserted into a single thread."
It does, at the beginning, talk about other ways to address this, like "shadow topics". But it goes on to solve it instead with a combination of virtual cursors, and this:
"Therefore, to solve this problem, we used a random thread for ACK tasks instead of the same thread."