Take explorer, it’s a mess trying to implement dplyr verbs in elixir. Anyone trying to use it is going to hit its limitations sooner or later. I tried migrating to it from polars but it is too frustrating.. gave up after some time.
Why will people use half baked libraries instead of python ? I will stick to Keres/pytorch, polars, etc
I have been dealing with this type of feedback for almost 20 years at this point: "This library doesn't work for my needs, so the conclusion is that it is hobby/not useful/etc for everyone". In your experience, we are not there yet (and we may never be), but the generalization does not hold.
Take Explorer, it was the first library of the bunch to be used in production because it was created by Chris Grainger to write data pipelines at his company. And the blog post does include the link to a talk on production usage.
Other projects, like Nx, started as an experiment, but nowadays it is used in production quite a bit and we spend a lot of the time refining it based on feedback we receive.
No one is claiming those libraries are complete. It is still pre-1.0. But we claim to be evolving and improving based on feedback. So if anyone wants to try something different than Python, and you can afford to engage, give feedback to maintainers, and even contribute, then you will hopefully have a good time and be closer to running it in production too.
From my perspective, just because Chris Grainger does data pipelines in production does not prove the project is for everyone. It means it worked out for him.
May be seeing more updates, papers, interviews, podcasts about their usage in production will change things.
For now, my assessment is everything is wonderful till one hits a limitation of these libraries. If it works out, its wonderful, if not its like neither here nor there.
It’s not an all or nothing game, and I rue the day that I decided to do everything in python just because I needed a couple LoC to call a python module.
Exactly. These projects are still during early adoption stage.
While it is production-ready and many are doing so, getting there requires improving documentation, discussing roadmaps with maintainers about a feature you may need (perhaps even contributing it), giving presentations (at your company or at events), and so forth. This can be exciting to some, but a distraction for others. There is certainly plenty of work ahead.
As for your issues with Explorer, perhaps it is not the library, but your knowledge of Elixir that was a little too half baked? ;)
The biggest barrier to adoption of these libraries I can see is that there is not a wealth of resources to solve the actual problem the person is experiencing.
Googling “dyplr verbs in python” there’s a towards data science post that’s the 6th result. It has step by step code examples to get you started with what I guess the person is looking for.
Googling “dyplr verbs in elixir”, you get the explorer library’s GitHub page and docs as results. I see there’s an example on the first page, but docs present as a high barrier to entry. For better or worse, hand-holding helps get people using the thing, and there’s a SEO-fueled ton of python hand-holding resources out there.
I’m sure you’re aware of these things as the author of the book mentioned, but buying a book is a step for someone who’s already sold on learning the language/framework. Having google-able content is much more useful to get people in the door and in the day-to-day work looking something up.
Until something “cool” and well-known is made with Nx/Axon and friends, they’re gonna be seen as second-class to most people who aren’t elixir diehards. Somehow everyone in elixir-land recs these, but never has anything they actually use them for.
There’s also the switching cost. Most Python ML compute is actually done in C/off-loaded to gpu, so its speed is ok and not greatly constrained by the language choice. Python does have multiprocessing stuff which, while annoying in my experience, makes parallelizing stuff possible. Again, thinking from an outsider’s perspective, there doesn’t seem to be a “must have” reason to use elixir when they’ve already been working in python.
Sometimes though it's nice to have most of your stuff in one language.
- In current explorer one cannot write user defined functions, plugins. - They don’t even have it in roadmap. - Also the exposed functionality is always lagging when compared to polars.
- elixir, nx, explorer,distributed, etc nice keywords - reality is it is still early for them. Better to stay away for production apps in case of ai/ml unless people are connecting to OpenAI and writing chat apps for which they won’t need any of the above except elixir.