I've been working on a pure-Racket Kafka client[1] off and on (for fun) since February and it's a good amount of work. When the official protocol docs and Wireshark fail me, I usually look at librdkafka and kafka-python[2] to figure out how things are supposed to fit together. Kudos to the authors of both libraries for writing code that's easy to follow!
I wonder what the license implications are there, considering that Kafka python is Apache 2 where as your project is BSD
I think it depends whether you consider me making a model in my head of how things are supposed to fit together a "derivative work," which I don't. The codebases are vastly different, and it's clear in certain places that kafka-python has taken inspiration from librdkafka (2-clause BSD license), and that both have similarly taken inspiration from the Java client (Apache 2), which makes complete sense to me since the docs are insufficient for building a client, as I mentioned in my OP.
If you're looking for a pure C++ implementation of Kafka check RedPanda, it seems a viable option since the lack of interest Confluent showed to their open source deployment.
A few colleagues if mine made a pure rust client https://github.com/influxdata/rskafka (we had some problems with the rust bindings of the rdkafka library)
What problems did you have with the rust bindings?
Is there anything new about this? It's been out for ages.
This is what the official confluent python library binds to.
Honestly it’s not perfect, but Eden seems to be a maintainer that cares, so thanks to him.
Eden is also the maintainer of kcat, a user friendly non-JVM consumer/producer cli for Kafka. It also uses librdkafka https://github.com/edenhill/kcat