- Decoupling of Broker & Storage Layer
- Tierered Storage (SSD, HDD, S3,...)
We use both Kafka and Pulsar in our systems.
- Kafka is used for microservices communication and operational data sharing
- Pulsar is used for streaming large customer data in thousands of topics
The only downside with Pulsar (which is constantly improving) was that community support for teething issues was a bit slower than what we saw with Kafka.
- Even empty Kafka topic consumes 20MB of on-disk storage (that's 20GB for 1000 topics)
- Inevitable coupling of non-partitioned topic to a particular Kafka broker limiting the storage scale
- Tiered storage was not available previously in Kafka (is it now available in open source version?)
- Native Multi Tenant support with Authentication / Authorization support not available in Kafka (essential for customer based thousands of namespaces and topics)
It also uses num_partitions*2 open file descriptors per topic, which can quickly surpass the default ulimit on a host. Always remember to raise the ulimit before you near 1000 topics, otherwise Kafka crashes.
If we are gonna move from Kafka to Pulsar (operational data),
- Publish new messages only to Pulsar
- Wait for Kafka Subscriber to process all pending messages
- Start Pulsar Subscriber (step 2 and 3 can done in parallel or sequential based on message ordering needs)
We don't need to worry about pulsar subscriber offset since existing messages in kafka topic won't be copied to pulsar topic preventing duplicate messages. The whole migration would be completed in 1-2 min (depending on Step 2 wait time).
This approach is carefully taken such that there can be little lag in subscriber but the publisher should not be affected by any means.
We have it in our roadmap to migrate the former to Pulsar, but not the utmost priority right now. It also simplifies Infrastructure management by keeping one event streaming stack instead of two.