hackmanytrades

Hi, I'm the author of the blog post. I'll be around to answer questions.

99% of Pony Kafka is written by me (for better or worse). I've been working on it since about May 2017 off and on with it being my primary focus for the majority of that time. However, due to working arrangements and other commitments, I've only spent about 12 or so weeks of time on it (where 1 week is equal to 5 days and 1 day is equal to 8 hours).

There have been a few iterations on the abstractions and API of the library but the majority of the architecture has been the same from the initial design sketch. I started by envisioning the features the library needed for end users and also internally in order to fully take advantage of Pony's actor concurrency model. From there I worked out the data and functionality ownership of the various bits (i.e. which actor does what and why). Lastly, I ran it by a couple of folks here at Wallaroo Labs to make sure I wasn't making any obvious mistakes.

The biggest change so far has been caused by building in the leader failover handling in relation to the data/responsibility ownership transferring from one actor to another. That's not entirely completed yet but it has mostly been an internal change. The end user API has also changed, but that has mostly been about fixing abstractions and/or data ownership issues.

I'm sure there will be additional changes as I have time to go through to fix abstractions and add in other features. Dynamic configuration changes, exactly once semantics, and group consumer functionality are all likely to impact the end user API along with requiring internal changes.

Great questions!

Re: FFI. In general, using the C FFI and creating C wrappers is not a major issue (aside from them possibly not being idiomatic) and, as you mentioned, required for a language as young as Pony. In fact, Pony Kafka internally relies on a number of C libraries via FFI. However, for the Pony Kafka use case, performance was a key driver. This meant that the risk of contention between the librdkafka thread pool and the Pony thread pool was one that had to be taken into account for long term performance goals. Same regarding the polling nature of getting data from librdkafka. Neither of these would have been a major issue had performance not been a high priority for Wallaroo. In regards to the idiomatic C wrappers, I think it's possible to wrap C libraries in an idiomatic way. It's not easy though because it's hard to figure out the right abstractions in Pony and map them to the C functionality. I don't think that's a problem though because not everything has to be idiomatic from day one. Over time, the C wrappers can be improved to be more idiomatic and/or phased out via native implementations.

Re: Ponylang direction. My personal experience on this is that the Pony community has been very receptive to our ideas. I've mainly focused one the Pony runtime side of things and less on the compiler/language design side of things though.

In regards to the C and Scala/Java client libraries. They are great for what they do and how they do it. However, that doesn't mean they're ideal in every scenario. For example, the Scala/Java client is the most feature rich client and is actively developed in sync with the Kafka brokers. This, however, doesn't make it suitable for embedding in other languages. As a result, the C client was created by the community and is now officially supported by Confluent. That doesn't in any way take away from the quality of the Scala/Java client though.

Also, while the C client is more featureful and better tested, there is still the concern regarding the thread pools internal to Pony and librdkafka. We've seen first hand how CPU cache invalidation can impact performance so we are very aware of the potential negatives if the Pony and librdkafka threads ever end up fighting with each other over the same CPU resources and would prefer to avoid that.

Yes, Pony Kafka is currently slower than the C client. But it is also almost completely untuned as of right now. We expect there is a lot of low hanging fruit on that front that will give us significant gains. Yes, we mention in the blog post that we would be happy at being parity with the C client but our goal has always been to exceed it, eventually. Both in terms of performance and features.

From what I understand of librdkafka, there's no easy way to disable the internal thread pool it uses.

I'd imagine that the internal thread pool for sending/receiving data from Kafka is as core to librdkafka as the internal thread pool for running actors is to Pony and trying to remove or disable either of them would be a large undertaking.

Thanks! Blog post corrected.

Yes, that's correct on both counts.

Pony Kafka is almost completely untuned as of right now. We expect there is a lot of low hanging fruit on that front that will give us significant gains.

And yes, we're concerned about the potential thread pool contention between Pony and librdkafka.

That's a great question.

Yes, Pony Kafka is currently slower than the C client. But it is also almost completely untuned as of right now. We expect there is a lot of low hanging fruit on that front that will give us significant gains.

There is also the secondary concern regarding the thread pools internal to Pony and librdkafka. We've seen first hand how CPU cache invalidation can impact performance so we are very aware of the potential negatives if the Pony and librdkafka threads ever end up fighting with each other over the same CPU resources.

Just one of the many perks of working here at Wallaroo Labs. 8*P

Hi, I'm the author of the blog post and the primary author of Pony Kafka. I'll be around to answer questions.