Hacker News Reader

Comment by paddw | Hacker News Reader

People will go through any number of hoops to not give, "I like the feel of the language more", as a reason for using it. Clearly, the main reason you would consider using Elixir for this is just that. There may be benefits to Elixir as a language, but there is no way they are going to overcome the ecosystem deficiencies to make it worth while in any unbiased cost benefit calculation. Just say you want to work in Elixir! It's okay to have a preference on the tools you use, even if it's for aesthetic or experiential reasons.

View on HN

> but there is no way they are going to overcome the ecosystem deficiencies to make it worth while in any unbiased cost benefit calculation

Is there a concrete reason for that? A lot of the approach in Elixir is to not go the route of other languages and merely slap an adapter interface on Python libraries. The approach is to approach things from a very Elixir and BEAM specific point of view. This is how Livebook was developed and how the machine learning libraries are being developed.

In fact, one could view it as Python being the one that's going to struggle to overcome the limitations set forth by its own language design. Elixir comes with very tightly integrated tooling (projects, testing, type analysis, static analysis, notebooks, scripting, etc.) that basically has 100% adoption rate without a plethora of third-party competitors and lack of integration with the language such as Python has. With immutability and concurrency built into the language, Elixir has a leg up, and there is active research into bringing static typing to Elixir and also binding Elixir to faster runtime environments, such as Rust.

Python simply cannot make the jump to Elixir, Erlang, and BEAM's language and VM capabilities like they can to a machine learning ecosystem akin to Python's.

> type analysis, static analysis

These two are so terribly, laughably bad in Elixir that you listing them as advantages makes me doubt the integrity of your entire post.

Worse than Python? It's a question; I don't know. I do know that the two most terrible (for me; I like types and a working IDE that helps me, not fight me) languages (js/py) are also the most popular. I don't know what that means, but as I did build large systems with Erlang in the past (in banking) which are still running production with massive load (and don't fall down in 2 weeks because the ENTIRE ecosystem had a minor version update, invalidating all the ABI's), I cannot imagine it's worse than Python or JS.

I hope you can tell me.

Python has at least a couple of functional type checking systems. Elixir only has Dialyxer, which has never actually worked for me any time I’ve attempted to use it. (It’s not just me. At work we have a medium-sized Phoenix codebase and about every fifth person who joins is determined to add Dialyxer to our CI pipeline - so far without success.)

I use Dialyzer on all side projects and it always works. However, I enforce via Credo that everything gets a typespec, and I employ type-driven design.

Dialyzer and Credo for Phoenix does require some workarounds, as that project isn't interested in providing typespecs or base compliance with Credo and is very heavy on macros.

I believe you. However, the current Dialyzer/Dialyxir experience bad enough that the creator of Elixir is working on an effort to specify an entirely different† gradual typing system for the language:

https://news.ycombinator.com/item?id=36282419

https://elixir-lang.org/blog/2022/10/05/my-future-with-elixi...

This strongly suggests that Dialyxir is not working as effectively for most users as e.g. Typescript or Mypy.

† I can imagine people quibbling with 'entirely different', but the proposed type system does have a new(ish) theoretical basis that's still the subject of active research.

My last weekend Project in Elixir is roughly 3 yrs ago now, but from what I recall the system was leagues better then pythons optional typing. It's just bad if you're used to languages that have been built around types from the start.

I think you’re just saying that you prefer dynamically typed languages. The point here is that Python now has much better options than Elixir if you want to check types statically.

I'm not sure how you could've gotten that idea from my comment. If I put it on a line, I'd say (no typing) < (python optional types) < (elixirs structs and pattern matching) < (typescripts types|interfaces) < (real static types from Java etc)

Hmm, ok, but structs and pattern matching aren’t static types. They certainly don’t offer anything like the kind of type safety you can obtain (with some effort) with mypy or other Python type systems. Mypy is quite similar to Typescript.

Those are two distinct capabilities.

Static analysis in Elixir is quite more accessible because you rely way less frequently on dynamic dispatch than Python (or Ruby or JS). You typically know which module you are calling to, structs and pattern matching tell you about fields (which we verify at compile-time) and primitive type information. Things like deprecated and undefined functions are part of the compiler, while most other dynamic languages typically require type systems or linters (or do it exclusively at runtime).

I agree on type analysis though. It is better than nothing but that's not much to talk about. We are working on it. :)

Ok fair, compared to other dynamically typed languages Elixir’s static analysis is better, by virtue of existing at all :-) Something is better than nothing, but getting “function has no local return” when you make a particular kind of typo three modules down the stack is a long way from eg Java or TypeScript just telling you about the typo, instantly, right where you made it. That’s the norm IMO and that’s what I was referring to.

I’m excited to hear that Elixir may be getting a bit closer to that, and I agree with your observation that because typical Elixir code uses dynamic dispatch very little, there’s a lot of room for improvement in ways that eg Ruby will never be able to offer.

I’m sorry about the “terribly, laughably” bit of my comment.

No worries!

For what is worth, I was not considering Dialyzer as part of my reply. You should be getting many warnings related to typos from the compiler!

Anyway I agree with the sentiment: we have a high ceiling but we are currently far away from it. If we had a Dialyzer that runs all the time, is fast (Erlang/OTP 26 already improved here), and has good error messages, it would already be great.

We are currently researching a proper type system into the language. Fully integrating and improving the Dialyzer experience would be our plan B.

Wow that would be great! I recall reading that an effort some years ago to do a type system on top of Erlang failed because typechecking inter process communication got too messy (or slow). Personally, I feel like a type system that leaves message passing completely out of scope would solve so many problems already. And then people can invent typechecked GenServers and the likes in userland.

I mean when 2 NodeJS processes communicate via JSON over HTTP then those aren’t typechecked either, even if their codebases are all TypeScript. Except, of course, when using the same TS rpc lib on both sides of an interface, with some code sharing between them so they use the same types.

I feel like even when all typechecking happens only on regular static synchronous function calls, and message passing is left out of scope, it’ll be so extremely useful. And like you said, the low amounts of dynamic dispatch makes it tractable. Still a huge amount of work I bet though :-)

By type analysis, I mean Dialyzer and built-in type errors, and by static analysis, I mean Credo and its ability to define custom checks and Elixir's ability to access and walk its own AST plus the compiler features.

Yes, Dialyzer could be better, but it's pretty good if you write typespecs. But Credo is quite good and one of the best such tools I have ever used. And Elixir g enerally has excellent error messages. Why do you say it's laughably bad?

Also, the point was primarily that all these tools are effectively built-in to the language without several different competing libraries.

And I neglected to mention the official formatter.

People are willing to put in thousands if not millions of working hours into shoehorning things into the language they already know, unwilling to learn a language that already has a good set of basic principles and concepts, that would enable the thing they envision.

People will build huge C and C++ libraries for realizing distributed systems that they can then use from Python, instead of getting to know a language on the Beam VM, because they only want to stay in the mainstream ecosystems or are unwilling to deal with new concepts like in Elixir or Erlang (for starters lets say lightweight processes, distributed system, pattern matching, proper recursion (TCO), the functional programming view of things. Some of those are very scary for many Python developers.).

Fortunately there are those on the other side as well, stubborn enough to try to implement things in those more niche languages that they prefer, based on technological merit of basic principles and concepts of a language, rather than just doing what everyone else does. So sometimes something really great and interesting comes out of that.

Often however, the shoehorning masses are simply too many, the people knowing the other ecosystems too few, to keep up with them or the newest developments, further enabling the "but this is what everyone else is doing!" kind of mindset. Something something "beating the averages" here.

> because they only want to stay in the mainstream ecosystems or are unwilling to deal with new concepts like in Elixir or Erlang

Or want to have a large easily accessible pool of developers to pull from. There are real, practical considerations here more than just feelz.

Are (some of) those developers not those who "are unwilling to deal with new concepts like in Elixir or Erlang"?

Maybe, but you make it sound like it's always some Blub paradox inspired reason. That someone doesn't want to learn thing <X> might be they're totally fine with <Y>; they can get work with it, they like working with it, whatever.

Not everyone is always looking for the next thing, better or not.

I think BEAM languages are kind of one of the few exceptions to this argument. Its tradeoffs and runtime semantics really are just very different from other mainstream languages. Other languages & communities are adapting to modern trends towards high concurrency, distributed systems, horizontal scaling, tight fault detection & recovery with varying degrees of struggle and success. Meanwhile in erlang these have been priority concerns for decades and they have highly sophisticated refined solutions to them built directly into the system.

Not that this article makes a good argument for the benefit of any of those things for ML. But at least generally while I am a pretty strong believer of "eh language doesn't really matter" erlang-based languages I do entertain a possible exception for.

Agreed, You might find Handbook of Neuroevolution Through Erlang by Gene Sher interesting.

> I think BEAM languages are kind of one of the few exceptions to this argument. Its tradeoffs and runtime semantics really are just very different from other mainstream languages.

Agree wholeheartedly. Elixir/BEAM/Erlang are a league of their own.

Never really played with the Erlang VM other than a little more than the Hello World in Elixir, but when I'm in the debugging hell of Apache Spark I really wish it wasn't based on the JVM.

I think debugging in Elixir is easier precisely because of it functional immutable nature. To contrast it, take Ruby (Ruby on Rails) where objects will acquire functionality at runtime. That makes debugging particularly difficult especially if you have jumped into a new codebase with little context.

Java has breakpoints though.

Side effects free is a good thing? Sure. Easier to debug? Cmon, elixir MO is based on, if not copied from, ruby. If anything, elixir is as easy to debug as ruby, if you can claim such a thing (I don't have substantial XP)

> elixir MO is based on, if not copied from, ruby

That's flat out false. Only the syntax is taken from Ruby, that's it. Even the added metaprogramming isn't anything like Ruby's. It otherwise just like Erlang.

Elixir's most popular debugger tool is called pry.

Not really. Elixir tries to use an approach to its "syntax design" that is similar to Ruby. But ruby goes all out in its approach to look like English. Elixir doesn't go that far.

Other than that if you look at how the language works under the surface it is completely different from Ruby which uses Runtime Reflection to add methods to objects in memory.

I was mentioning mostly debugging XP. And even if my comment applies to other facets of elixir, it does not describe it as a whole. That being said, I don't find it easier to debug than ruby, IME.

Oh, author here. I absolutely do simply prefer it but I also think it has distinct upside compared to Python in this context. I see a number of companies orchestrating their Python ML with Elixir. For them the advantage is even more obvious. But overall Elixir has certain strengths. It also has drawbacks and trade-offs.

I would absokutely write the post "Why I prefer Elixir" and I think I have. But this was not intended to be that.

The reality is that neither Python nor Elixir is performant enough to do the actual work of ML, that is all going to be wrapped libraries in C, C++ ... Rust?

And while Elixir may be as well suited as Python to being the interface/scripting language for ML tasks, Python has a pretty unassailable lead here with an overwhelmingly dominant ecosystem.

Elixir could be used to build and manage pipelines (like a roll-your-own Jenkins) in this area, and in this regard (concurrency, reliability, async) it is probably a significantly better language than Python, but the actual tasks on the pipeline will need to be defined in Python, just because it's use is so entrenched.

Also, the "reinvent something another language can already do" pitch is going to become a much harder sell in today's funding market as the silly money dries up.

A lot of comments here seem to be framed around the idea that Elixir is aiming to some day dethrone Python. Knowing the Elixir ecosystem, I wouldn't even say it's the case that it's trying to gain equal market share. The Explorer library even calls out: "The aim here isn't to have the fastest dataframe library around". There seems to be a significant interest within the Elixir community around this and, as a non-data scientist who needs to do some analytics sometimes, I certainly appreciate having powerful tools like this without having to step outside Elixir. I've never had to learn about Python tooling and I'm happy to keep it that way!

What hoops? Your comment makes it sound like the author is hacking together some libraries that happen to exist just to do ML in Elixir. Numerical Elixir is a very real thing being actively worked on by the creator of the language.

https://github.com/elixir-nx

The only reason I discovered Elixir was through an unbiased cost benefit calculation while working on my last startup.

This was in early 2017, and my search essentially went from a variety of JS frameworks to RoR to Scala + Play and then eventually to Elixir + Phoenix. The ecosystem was much smaller then but of the options that were viable, it was by far the most productive.

> but there is no way they are going to overcome the ecosystem deficiencies to make it worth while in any unbiased cost benefit calculation.

Another way to say this is: "no other language will ever be better than X", which is clearly not true.

Eventually something better will arrive. Maybe that's Elixir, maybe not, but is sure won't be Python forever.

> Maybe that's Elixir, maybe not, but is sure won't be Python forever.

It will be Python for all of our lifetimes here.

I am just hoping, that when that better thing appears, it wont only be better because it makes use of what is available in Python via some FFI thingy, adding another layer of indirection and abstraction and overhead, but actually better from first principles, from the ground up, or at least not going via Python, to get a similarly big ecosystem.

I unironically think this is an absolutely legitimate reason even in a purely results-oriented, commercial context... but saying that is not politically expedient :(

It's not the language specification that makes the languages special, its the VM they run on.

(Elixir is a little more special with its hygienic macros, whi allow for for libs like NX to exist.)

The BEAM/erlangVM is probably closer to a OS than a traditional-ish VM like the JVM.

The interesting thing about BEAM languages is that they naturally treat the gpu as an asynchronously communicating resource. No other language gets that abstraction correct.

Just to inform the reader, the experimental WebGPU API does embrace async communications with rendering devices: https://developer.mozilla.org/en-US/docs/Web/API/GPUDevice

It's still not the same, because of the way js async works

Curious why that is. How does V8 handle it differently?

V8 is asynchronous but single threaded, elixir gets m:n threaded. Job cancellation is trivial, safe, and automatically recovers the resource in elixir (the gpu devs have hooked cleanup in to the BEAM GC, because that is a thing). Moreover, if the caller of the agent asynchronously controlling the gpu is cancelled, it will also release the gpu automatically (in most cases, if you use the stdlib otp features instead of trying to build with primitives)

AFAIK V8 does make use of threads internally via libuv in order to make truly concurrent system calls. GPU device comms should fit perfectly in that design.

Reply on news.ycombinator.com