Because it's more fun to learn a new programming language than deliver features to users. We're VC financed so no need to get money, also spend it before people talk about profitability, then the good times are over (see Etsy). We also can add Pony to our CV and move on in one year to the next company where we will introduce the next big thing to add to our CVs. Plus 10% more salary! Kaching! #LivingTheLife
When I read their blog, I get the feeling that they are thinking things through. They seem to target a specific market and focus on what can make their product desirable. And yes, they are definitely experimenting on a few things but they are well aware of the trade off: it appears to be a calculated risks, and might very well pay off in the near future.
[EDIT]: typo
And it is a risk - big whoop, you've written production code in Pony. Good luck doing that again elsewhere, or hiring someone competent in it. I don't think it's quite hip or popular enough to be very useful on your resume (but I could be wrong).
If keep doing stuff because its done elsewhere we stop innovating.
> it reminds me of WhatsApp's decision to write their backend in Erlang.
WhatsApp's decision to use erlang is based upon totally valid points. In fact Erlangs concurrency model made scaling so easy for them, they only needed 50 engineers overall for handling 50B messages a day. [0]
You might not know Pony, that doesn't mean it couldn't make sense for someone else.
[0]: http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...
Is the "dr" in "tl;dr" literal?
If it was anything beyond the strategic interests of the company it's that developers love a) clean code and possibilities of a fresh new project and b) playing with new modern toys.
We put a lot of thought and experimentation into our decision to use Pony. It wasn't a decision we made lightly.
I wrote a post a few months back that covers that decision:
https://blog.wallaroolabs.com/2017/10/why-we-used-pony-to-wr...
Chosing newer languages gets maligned far more often than pigeonholing the wrong old languages onto problems.
Besides someone's got to take risks with (potentially) better technology, as long as they know the risks and fully considered them going in, then by all means. Plus as long as we continue to use C for all systems development the longer we'll have preventable security issues.
I've read a few blog posts now and the end result always seems to be Pony.
You even talk up how great the C and Java client libraries are. Well they can't be as great as you say or you would have used them.
The C library seems to perform better, be more featureful, and tested better. So once again, why? It certainly can't be because you library is going to top the C library in any way. The article even seems to imply you'd be happy being at parity with the C lib.
There's a large performance improvement we get by having a single scheduler thread for each CPU. The performance impact of that is very large. Adding another threadpool that competes for CPU usage would be problematic.
Our client is for those high-performance use cases where if we can get parity or close to parity with the C client then we should get much better performance due to those architectural concerns.
That said, we plan on providing a way for folks who are less concerned with performance to use the C client library.
In the end, it was less about "use Pony" and more about "do this in a way that matches with Wallaroo's architecture".
This could be accomplished without using Pony, unless you view "Wallaroo's architecture" as being effectively synonymous with "use Pony."
Sure, but why should it be accomplished without Pony? Languages are optimized for use-cases. This means that some languages are good and some are worse at handling particular use-cases. If Pony is the best choice for their use-case, why would you not choose it? Taking all the risks of a new tech into account, of course.
Also, while the C client is more featureful and better tested, there is still the concern regarding the thread pools internal to Pony and librdkafka. We've seen first hand how CPU cache invalidation can impact performance so we are very aware of the potential negatives if the Pony and librdkafka threads ever end up fighting with each other over the same CPU resources and would prefer to avoid that.
Yes, Pony Kafka is currently slower than the C client. But it is also almost completely untuned as of right now. We expect there is a lot of low hanging fruit on that front that will give us significant gains. Yes, we mention in the blog post that we would be happy at being parity with the C client but our goal has always been to exceed it, eventually. Both in terms of performance and features.
Now, this is what I'm hearing from what you are saying:
> 1- We can't use a JVM implementation because we aren't using a JVM language.
Makes sense.
> 2- The C library is okay, but hides its thread pool with no way to access it.
Ugh. Hate that. Its like these people writing these have never had to use them in a real project. The sign of a mediocre library.
Pony's actor model might have to rewrite almost any library used by it when concurrency is involved.
But now, I think you answered your own question in the titles now:
> Why we wrote our Kafka Client in Pony
1- Because the C library is mediocre and hides its threads from users making it not very useful for high-performance applications.
2- Because the rest of the system is in Pony. Really, you could write it in C/C++ or even Rust as long as you wrote it in a way that played well with Pony's concurrency model, but why bother with that extra effort, especially if you believe - as you seem to - that Pony's concurrency story is superior.
Saying that JVMs are 'stop the world' and Pony is 'concurrent' feels like it's ignoring modern JVM GC strategies.
It might be true that you can avoid stop the world collection for an actor-system, but by logical extension would that not be possible on the JVM as well for that particular workload, given a suitably designed actor-system?
My intent was not be unfair. There's a lot of nuance in the topic that can be hard to cover in a more general blog post. Garbage collection is a fascinating topic, there's a great amount of detail that is left out in that post. I was going to a broad overview of general thinking.
[edited for typo]
What it sounds like instead was very minimal benchmarks or science was performed ahead of time, then a lot of justification written afterward. I know that's a reach as well, but this is a well-trodden path, so the answer "Pony is the best possible solution for the interest of our business" just seems like a very strange conclusion.
Curious if you did any benchmarks for Pony against standard JVM Concurrent Mark Sweep GC, which aims to reduce GC induced pauses?
We looked at Erlang. Several of us are friends with folks who worked at Basho on Riak and we talked with them about our performance goals. They were very skeptical that we could meet them using Erlang. Based on that, we moved on from Erlang.
So either:
a) Erlang is not a good fit, and I'm wrong. Then I'd really like to know why I'm wrong!
b) Your friends at Basho led you astray. Would also be interesting to know what happened in this case!
Either way, without knowing more details, the short version you just posted is inconsistent with the claim that you guys did serious research into existing language ecosystems before going your own way.
I think this is a rather in-depth conversation where HN comments aren't the most productive mechanism. If either or both of you are interested in chatting more on this, my email is sean@wallaroolabs.com. Drop me an email and we can arrange a time to chat.
Now Erlang often feels fast, because of the architectures it allows, but when you get down to shuffling bytes around or doing low level math it is currently slow, slow, slow.
Given Wallaroo's speed goals, I would have been really surprised had they used Erlang:
If you're doing an n-body simulation, then this benchmark is a good benchmark for deciding whether or not to use erlang/elixir. If you're doing a server which is mostly parsing JSON inputs over HTTP and spitting out more JSON with HTTP, and needs to handle thousands or millions of parallel connections without hiccuping, is an nbody simulation the right thing to use as your benchmark reference?
So that's not a good example of Erlang/Elixirs performance which is hardly known to be 'slow'. The language and process/actor model is far faster than many other languages particularly in the web space.
The author's also mentioned heavy dependency on the actor model as a performance optimizing strategy and optimal code structure which is why it likely is worth fully exploring for the OPs problem.
There is definitely a predilection in certain parts of the coder community to prefer newness and difference over tried and true. There isn't anything wrong with that necessarily: it's part of how progress is made. However I think it's often taken to extremes in the coder community.
Also, picking a right tool for the job can be a real advantage, which makes it easier to deliver features.
Getting so close to C implementation (in terms of speed) with Pony is actually insane if you look at the number of guarantees Pony gives you. Next time you dereference a NULL pointer please remember that it's impossible in Pony. Oh, and next time you spend a week debugging some hairy locking issue, consider that issue wouldn't happen in Pony at all. EDIT3: removed EDIT1 from here.
Currently, Pony is in direct competition with Go (but uses the other concurrency model) and Erlang/Elixir (but is natively compiled). People and companies frequently choose Go or Erlang, so I don't really understand why they shouldn't choose Pony if their use-case fits.
EDIT2: And here I am getting downvoted... I wonder, is anything I wrote not true?
I don't understand? Why are they irrelevant? They're basically Pony's reason for existing... Cheap, efficient and safe concurrency, coupled with very high level of type-safety, is the main selling point on Pony. It is much better on these counts than many other languages. Maybe I shouldn't have mentioned Python and Ruby, I'll edit the post.
> It's the aggressive tone
I see. Compared with the charming politeness of the OP comment, I must have sounded really rude. I apologize.
I'm somewhat skeptical of GC pauses being a problem in anything that's not actual hard real time like avionics or manufacturing equipment or similar. What difference will a few hundred millisecond pause even make in an distributed async data pipeline? And that's on the higher end of pauses these days.
The difference between Pony and Go or Erlang (or even Elixir) is that the Pony team still are making breaking changes to the language. That means that your dev team may need to spend time to update features due to breaking changes in the language. Also, the ecosystem isn't there like it is for Go or Erlang.
In any case, if you are aware of the risks and plan to mitigate them - by, for example, employing people capable of debugging and fixing the language's implementation - you're left with some risk and a lot of advantage (if you're lucky and your domain is indeed the one your language is best suited for). It's a gamble, of course, but then nearly every decision (other than buying IBM) is one.
> add Pony to our CV and move on in one year to the next company where we will introduce the next big thing to add to our CVs. Plus 10% more salary!
It just sounds like venting to me, without any rational or fact-based argument for why using Pony is bad in this specific case. Doesn't it?
These read/write differences will compound to make the rest of the data processing pipeline slower.
> Yes, Pony Kafka is currently slower than the C client. But it is also almost completely untuned as of right now. We expect there is a lot of low hanging fruit on that front that will give us significant gains.
>There is also the secondary concern regarding the thread pools internal to Pony and librdkafka. We've seen first hand how CPU cache invalidation can impact performance so we are very aware of the potential negatives if the Pony and librdkafka threads ever end up fighting with each other over the same CPU resources.
Maybe, just maybe, we're starting to demand more of career software engineers. Maybe, just maybe, it's time for folks to stop grousing about how value-driven they are and realize that bad engineering is more expensive than good engineering over even medium term timescales.
Rust has some flaws. Kotlin has some flaws. Pony appears to have even less flaws than either.
Of course I have merely read the tutorial. In practice it could be worse than either.
You know HN is written in ARC, right? I'll just jump to the end: they did it for the features, and having fun just happens.