How we saved $100k a month with Kafka networking optimization

segment.com

11 pointsfullung3 comments

Given the way you were able to logically group data so that producers and their consumers are in the same zone, would it not have been simpler to just create three completely independent 3-node clusters, rather than worry about all this at the partition and keying level?

Yes this seems confusing to me as well. AZs are usually used for reliability. I'm looking at their new topology and wondering why AZs were used at all?

Maybe if one AZ fails only 1/3 of the producer, topics and consumers fail. Seems like a lot of overhead for a relatively minor advantage.

The idea is that we are running few multi-tenant Kafka clusters to amortize the operation cost. These clusters host many topics with different topology depending on different requirements, cost profile and scale.

Some clusters are hosting hundreds of topics, thousands of partitions and used (consumers/producers) by hundreds of services.

Reply on news.ycombinator.com