The blog post is quite clear in stating that their pain points had nothing to do with scaling or throughput. The author explicitly mentions idempotency, custom headers, and authentication.
I think you're ranting about a strawman you put up.
The blog post is quite clear in stating that their pain points had nothing to do with scaling or throughput. The author explicitly mentions idempotency, custom headers, and authentication.
I think you're ranting about a strawman you put up.
"We couldn’t create a Kafka source connector from the database holding the messages" – Why couldn't they?
"we had extreme difficulty troubleshooting when their system refused our best understanding of said custom headers" – What made this so extremely difficult for them?
"Authentication with Kafka, and authentication to encrypt confidential data in Kafka, and getting the Avro schema from the schema registry were all separate and painful steps" – These are 3 totally unrelated features, so of course they each have settings. What made the steps so painful for them?
"We would never tolerate this level of complexity with a database" the author says of configuring kafka features, apparently missing that a database setup with all those features (external authentication, encryption, custom serialization) also requires "separate" configuration for each one.
> While my client was in a situation where we had no choice but to use this monstrosity of Kafka, Avro, and custom message headers, I would never recommend this usage of Kafka if I had the option.
The antipattern isn't Kafka, it's the client situation.
Still, I think talking about Kafka without load feels like Christmas without tree. I thought that was the main point of it. Handling massive loads (by distributing them).