Why Elixir (2014) | Hacker News Reader

Why Elixir (2014)

theerlangelist.com

124 pointsarrowsmith69 comments

From the author of this post, here's one of the best demos of Elixir, Erlang and modern distributed computing in general:

"The Soul of Erlang and Elixir • Saša Jurić • GOTO 2019" - https://www.youtube.com/watch?v=JvBT4XBdoUE

When you feel sufficiently amazed by this demo, I also recommend the Elixir in Action book, by this same author, to get started with this incredible ecosystem and paradigm.

---

It's hard to imagine a better platform than Erlang's for writing distributed and networked systems of any kind. I fell in love with Elixir in 2016 and have been using it full time since, first adopting as CTO in my previous company, now powering my new solo business. It's such a good language, with great community and stewardship, running on a rock solid platform that's so advanced compared to most popular languages today.

Erlang was created 38 years ago, and it's good to see the ideas of Armstrong, Virding and Williams to be vindicated today. They simply were too far ahead of their time.

Soul of erlang got me hooked to Elixir recently, trying to get my hands dirty as well.

Other than distributed/concurrent system use-cases, could you share what kind of products are best when built with elixir/erlang compared to easier to write languages like Go, for example.

Anything that spends most of its time waiting on IO, and doesn't do a great deal of number crunching: i.e. any kind of server talking over a network socket.

Go, and any other language, lowers the barrier to running concurrent code, but there is much more to concurrent servers than concurrency: fault tolerance, isolation, shared state management, instrumentation, introspection, clustering, process migration. The BEAM and its ecosystem gives you all of that out of the box.

Also, the BEAM offers an immutable, functional environment. Data races are impossible, which are the biggest pain and source of heisenbugs in any kind of system with > 1 concurrent thread. This is huge. You can model your entire system as concurrent processes without ever having to deal with concurrency issues. You only ever have to think in "single-threaded" mode.

What language and programming style did you use prior to using Elixir the most? Have you been able to build systems of similar or even larger size as previous projects in other languages, and what was that experience like (i.e. how did Elixir make things better)?

I've been doing this job for 20 years. I am fluent in C, Go, Rust, Python, as well as Elixir, and for my clients I have written concurrent systems running in production in all of these languages. I know how painful and complex the problem space is.

My opinion is not that rare, so instead of listing my credentials, you can search and find many others that have found this platform to be a great fit, simply because it has been designed to solve this very problem since 1986 when everybody else was focusing on single core, isolated systems.

Sounds good, you make it sound like that Elixir is now your de-facto, even for non-distributed systems projects?

Servers are 99% of the code I write, so yes I choose Elixir, though for very conservative clients and small projects I use Go.

For everything else, which is not a lot, there's Rust, Scheme, Lisp and many other fun languages to explore. My focus these days is on my business rather than consulting, so I have a lot of freedom.

How do you manage to convince employers of such language choices? (Do you need to convince?) And what kind of jobs or positions are that? I would love to use my skills like that, instead of building CRUD in Python, not really being able to apply my skillset, but employers are not ready to make the smallest leap it seems.

I've convinced the CEO that Elixir/BEAM/OTP is a good choice for a fairly large, multi-application project. It wasn't very hard, factors like high availability, the same programming language and runtime in the entire system, extremely fast prototyping, battle tested in absurdly demanding settings, sounds very nice to a business strategist able to understand at least some of the implications.

The drawbacks are basically in recruiting and a few other areas and quite manageable.

Edit: There's a book for this particular purpose, convincing the suits, https://pragprog.com/titles/tvmelixir/adopting-elixir/ .

Recruiting was exactly the one argument that I got to hear. That hiring people would be difficult and people demanding high wages and so on. I could not convince an employer to not weigh that heavily. At least that is the brought forward argument. There might also be an element of "don't know it myself, don't want it in my company".

I worked at WhatsApp, one of the big Erlang users, almost none of our server team knew Erlang before joining, including me. Until we hired someone who actually used it before, I was the most knowledgeable pre-hire, because I remember seeing a post about it when Erlang open sourced it.

Yes, we probably could have done some things better if we had a bit more Erlang experience on our team; at least while I was there, none of our applications were properly packaged as OTP applications, and maybe that could have been useful. But overall, we were smart, experienced server people who were willing to learn Erlang and we were handed a tool that fit our needs very well, so we all got Erlang books and figured it out. If you can recruit smart, experienced server people who are willing to learn a new language, you don't have a recruiting problem. I haven't personally worked with Elixir, but I feel like most of the unfamiliarity is going to come from the underlying BEAM and OTP, so same difference; Elixir just has different syntax and macros are more heavily used, IMHO both syntaxes are going to be unfamiliar to most.

That'll be my approach, together with knowing some people that know some people who would probably enjoy finally getting to use the BEAM in prod we're betting that clever candidates that find our business interesting will also have an easy time learning this specific tooling.

It takes a bit of getting used to pattern matching and the overall functional/declarative approach when coming from a more imperative background, but I believe learning the ins and outs of the business niche will generally be harder and take more time.

In some companies handling recruitment issues isn't enough, investors or shareholders might be worried about 'exit' and how to maximise it for themselves, and refuse tooling they perceive as obscure and expect to lower bids to buy them out.

That is exactly what I think. If you got decent and educated developers, they should be up to speed fairly quickly. But management layers often have no trust at all, even if it would take maybe merely 2 weeks to be able to do basic implementations in a new language and ecosystem. Basically it means, that we cannot possibly spend 2 weeks becoming better engineers, but we can spend infinite time on wrangling with lesser tools.

I would love the chance to learn more Erlang (looked at the beginning of "Learn you some Erlang for great Good") or Elixir (used in last year's AoC) on a job and get to use OTP, watch it run my function calls on multiple machines and all that. I know a lot about functional programming, as I do it in my free time (big Scheme fan).

As it is currently, I cannot apply my skills at the job. For example when I think that some code should not mutate some state, but rather use pure functions and the tests should be simply function calls and checking the output, then I don't get the time to do that, nor the time to show how this would look like and how it would make things simpler. No one aside me on the job seems to be interested in purely functional data structures/persistent data structures either, which sooner or later are necessary, if one wants to make things purely functional. So basically I am the only person with that knowledge and cannot apply it. It is so dull.

Why specifically Elixir? Have you tried other languages that run on BEAM?

> Data races are impossible

As far as the application code goes, but most systems have databases which opens you up to all kinds of race conditions. Does Elixir help in that case?

Modern relation databases are not a concern at all. You get race conditions if the developer has only a passing knowledge of SQL, because otherwise that's a basically solved problem, with the existence of transactions, isolation modes, etc. If all else fails, your query throws an error, but it does not corrupt your database.

Much more common instead are race conditions on local storage, external APIs and traditional memory races when using unsophisticated languages.

There’s a lot to learn about transaction isolation levels. I’d bet 90% of developers don’t know anything about them.

Also using serializable isn’t free.

Races within an elixir application are still possible it is just data races that are prevented. For example if you have a bank account in an elixir process nothing will stop you from implementing a withdrawal as 2 external operations: read balance, write balance which is inherently racy but not a data race.

My experience with databases in Erlang was using the pattern where one process recieves a high level message and does the database work was best. If you can split the queue in a sensible way, that gives you concurrency; for example if all the requests address a single user, you can hash users into as many buckets as you need for concurrency; if the individual requests take a long time, it might make more sense to keep a scoreboard of if a user has a process working for it, and send further requests there, otherwise to an idle process (or spawn).

You do need to manage this somehow, but the building blocks are there. Of course, the building blocks for transactionless database access are there too.

Races and deadlocks are certainly possible in pure Erlang/Elixir - no databases or global state required. There is no magic bullet.

For example, if you use blocking RPCs between two processes, they can deadlock waiting for each other's responses. Try implementing Dining Philosophers, you will learn a lot.

The answer is not to use blocking RPC, but that will challenge your brain topology. It takes some time to be able to lower yourself into the hot bath of full asynchrony, but when you do, it feels very very good.

I characterize it as the co-problem (in the sense of the dual of a problem). Distributing a problem, starting processes and pushing messages is easy. But it may be hard to know when you're finished. Everything is easy, but termination is difficult. It's the Erlang/Elixir Halting Problem.

It is not often mentioned as a strong feature for concurrent programming, but Erlang/Elixir have timeouts built into the language. And when you add OTP supervisor restarts, you can avoid some common programming mistakes through random evasion - don't do this by design, but it does help resilience.

In other words, Erlang and Elixir push the capabilities of Kubernetes down into the language itself.

Its somewhat painful to watch modern languages and ecosystem reinvent (mostly in an inferior manner) things that Erlang (and Lisp) had since so long ago.

Talk about under-appreciated.

Virding's First Rule of Programming (2008):

  Any sufficiently complicated concurrent program in another language 
  contains an ad hoc informally-specified bug-ridden slow implementation 
  of half of Erlang.

https://rvirding.blogspot.com/2008/01/virdings-first-rule-of...

Not sure how the 'best products' constraint is supposed to be interpreted, but Elixir is a more flexible tool for developers than Golang. It has a decent REPL, can be used for scripting, and so on.

I haven't used Golang for things like binary protocols so I can't really compare, but Elixir or Erlang would be a good fit since they're very good for expressing grammars and fundamentally treat strings as byte sequences.

If pattern matching helps you express your problem domain succinctly they're also a good fit. Same goes for macros. My impression is that Golang commonly requires quite verbose or complex code compared to Elixir.

I expect raw number crunching performance to be better in Golang, but BEAM processes are very lightweight so it might win on either performance or developer ergonomics if the task can be solved in parallel.

Easier to write by whom? I ask because to me, Go has one of the ugliest syntaxes I've seen. On the other hand Elixir is very easy to write.

I use both on a regular basis for many years now (and some others), and I'd say it depends.

Elixir matches the way my brain thinks somehow, with (kind of) pure functions transforming data step by step, and when you can break down some task in such a way its only a simple set of pipelines (|>) its really great and feels great. It is also exceptionally cool if you spawn Agents to hold some global state where multiple processes talk with it, plus the integrated stateful introspection/debugging is just chefs kiss. In the context of pg's "Blub Paradox" Elixir is an acceptable Lisp (not homoiconic, but with modern tooling). You can solve very complicated problems in very clean ways. Also often underrated: Phoenix/LiveView is probably the best escape hatch out of JS frontend hell alltogether, and leads to better outcomes (performance, scalability, sanity, maintainability, ...) compared to JS frameworks.

On the other hand, sometimes I have a very dirty real life thing I want to achieve. Like doing some Unix stuff, interacting with some ugly APIs, implementing an given imperative algorithm to brute force a problem quickly within reasonable constraints, manipulate some image file, automate some adhoc outlook365 process, ... you name it. The weakness of Go (extremely simple/plain, verbosity, boilerplate, ...) here is actually the strength of it in these cases, but took me a while to realize. In Go, I don't even care anymore to make anything "elegant" (which is very tempting in Elixir!), but write the absolute straightforward series of steps in brutal directness, including nested loops and very long functions. This leads to rapid dirty work solving, and has the added benefit of trivial distribution (cross-compile to a single self contained binary) to other folks that don't have dev dependencies installed. Also I rely solely and the compiler/linter for this type of code and have zero tests for it (I am not going to mock the filesystem interactions and all that for basically a better ad hoc shell script).

So for the big/complex/scalable projects, I think Elixir/Phoenix is just perfect in terms of a "web stack", only a few rough edges left in iE the docs for beginners looking into LiveView. But for the small ad-hoc stuff or in situations where I can "brute force" my way through an ad hoc problem, Go is it.

Just my 2 cents.

> Easier to write by whom?

Most people? I have no love lost for Go, but most (if not all) programmers have experience with imperative code, making it very easy to learn, even if the syntax is ugly. Elix is from a different paradigm, so you have to learn that in addition to the syntax.

“Easier to write” has to be bait, right? That’s entirely subjective. Someone out there might think Assembly is easiest to write.

I’m an elixir noob, but I’m actually surprised that the distributed story is not _better_. For example, a big thing about OTP is that it allows running many processes that communicate by message passing: but all these processes are _colocated_ to a given node, i would have expected that OTP (or genserver or whatever) handles clustering of machines, scheduling of workloads and routing of messages for me so that I can start processes and call them without caring about where they run.

What about “distributed systems” is elixir actually shining at? From my reading, it seems rather like a good option to _avoid_ distributing (thanks to the light processes and concurrency handling)

I think you need to go deeper, because it does exactly all the things you’re describing out of the box.

A process can be anywhere in the cluster, and you can interact with it as if it was on your local node.

I use all of these facilities in production on my product with ease. I’m happy to answer your specific questions.

Erlang/Elixir nodes can actually be rather trivially clustered, you can spawn processes on other nodes and transparently pass messages to processes on other nodes, including things like closures. It does not magically schedule workloads across the cluster, but there are libraries that can do so.

You can message pass across a cluster. In fact, it's identical code to message a remote vs local process.

Erlang provides "process groups" that allow you to structure groups of processes across a cluster.

It doesn't try to be magic. It's not a workload scheduler or "automatic distributed machine".

You structure things the way you want, on top of it's primitives. It's just that all of the primitives you need are built in and work really well.

It does all of that.

The structure of the system is such that you can call any function on any node in the cluster.

In order to do it well, you need to understand your own abstractions. If we have a cluster are we better off passing every function around the cluster or running some locally? Is the network overhead worth it in every case or only in some?

It's up to your code to make those determinations about your application, but everything you describe is fairly simple to implement on the BEAM.

As others have commented it can do all this. What it doesn't do is automatically balance/place GenServers and processes across your cluster. This is up to you. The ways in which you distribute work across a cluster is full of trade-offs.

Need raft concensus? Paxos? Something else? Want to place work with consistent hashing? Want to hydrate/dehydrate with persistence vs memory only?

Those are all important decisions and the languages do not force you to one option.

First, if you write process-concurrent code in Elixir/Erlang, it will run on all the cores in your machine. It's automatically locally scalable. The number of processes should range from some small multiple of the no. of cores (say, 10x cores), up to 100k - 1m on a big memory multicore machine.

Second, as the other comments say, Elixir/Erlang have very transparent built-in ways to address other nodes, and the programs will scale as expected: more cpus v. network data distribution cost. So good for small tasks that require lots of compute. The native clustering will run to a few dozen nodes over a trusted LAN.

But soon, you will want some layer managing the cluster, try Swarm:

https://github.com/bitwalker/swarm

I'm working through Elixir in Action now - it's fantastic. I could have saved a lot of time and effort if I'd read it years ago.

A great talk, but that cameraperson got a workout!

I read Elixir in Action which was a great intro, but it was Concurrent Data Processing in Elixir [1] that really made the penny drop for me.

I have only finished one Elixir app. It was a lot fun and the app has proven rock solid. For me it felt a bit like using microservices but with lots of the footguns and ops overhead removed. I would love to use it for more than a hobby project.

[1] https://pragprog.com/titles/sgdpelixir/concurrent-data-proce...

Thanks for the talk link, it completely kept me engrossed and was a pleasure to watch! Addendum question: what is the state of database drivers for Elixir? Does it have mature libraries for Postgres for eg? I imagine if it's targeting web development, it will need those.

Elixir has Ecto, which is not-quite an ORM but is the de-facto standard for database access in Elixir/Phoenix.

https://hexdocs.pm/ecto/Ecto.html

Thank you!

Elixir is amazing. Had a stab at it back in 2018, then shifted to work with other traditional langs/stacks, but Elixir has always stayed in the back of my head teasing me. It's an elegant language, that you can see that the authors have put effort in keeping it lean, powerful, expressive. Phoenix (which is the goto web framework) may seem magical in the beginning, but once you dive in you realize that there is no magic, it's all there explicitly in modules (well maybe for macros which do feel magical). What I was missing back in 2018 is IDE support which was iffy. I wonder how that changed recently, if anyone who knows cares to share?

Not sure what you're looking for in IDE support...

If you're a Jetbrains addict, there's this IDE extension which has worked well for me: https://github.com/KronicDeth/intellij-elixir

VSCode support for the language is better than ever, outside of VSCode too! NextLS and ElixirLS provide the editor integration you’re probably looking for :)

I like a lot of things about Elixir but I struggled with what felt like clumsy IDE capabilities. Stuff like 'go to definition' that I use all the time in other languages didn't work right. Lack of strong types makes autocomplete problematic. I've started to mess around with Gleam hoping that it spans this gap because I want to like this ecosystem.

I see so much effusive praise for Elixir that I feel like something must have gone really wrong with our codebase, because build times from scratch are horrible (10 minutes or so), and I frequently have problems with the incremental build when switching between branches which force me to do a clean build.

Coming from a Node and PHP background makes me really appreciate the lack of compile step.

Here's a nice series of articles if you haven't seen it before, which can help you to reduce your compilation times: https://medium.com/multiverse-tech/how-to-speed-up-your-elix...

How many .ex files in code base?

I'm fascinated by Elixir, but more so for using to do neural network experiments with more unique topologies, training methods, and activation functions. Being able to spin every neuron out into a process just seems too damn elegant for doing work with spiking neural networks. Will it be as fast as the python stacks? Probably not, but not being constrained to the implicit assumptions in a lot of those libraries will be worth a lot. Plus, there's already good neural network and genetic algorithm libraries for Elixir. I really think this langauge is gonna go places.

Not a lot of technology I have work with every day “sparks joy”.

But the Elixir language is one that’s usually an exception!

I really encourage people to take a look at it even if you don’t end up using it right away- the OTP actor pattern is elegant yet powerful and extremely fun to prototype with.

I recently used Elixir for Advent of Code 2023. It is elegant in many places, at least from what I have explored during puzzle solving. However, to me it does not reach the elegance of Scheme or Lisp and I switched back to Guile while solving more puzzles. Due to its not as simple syntax, naming choices are more limited than in many Lispy languages. Also the way anonymous functions are written did not appeal to me especially that much and how they must be written when being passed to other function and how to reference functions in the same module sometimes requiring to still write the name of the module and all that is not that intuitive.

> how to reference functions in the same module sometimes requiring to still write the name of the module

Not sure what you mean here. If the function is in the same module then you should be able to refer to it without including the module name.

Can you give some example code to show what you mean?

I don't know exactly where it happened, but here [1] is a typical AoC code using Elixir. At some point I got so annoyed by Elixir telling me it does not find the functions of the very same module, that I simply defaulted to always writing the module name.

[1]: https://codeberg.org/ZelphirKaltstahl/advent-of-code-2023/sr...

Edit: Ah yes, I remember it happening in |> pipes a lot.

I still don't get it. I downloaded your code and stripped `Part01.` from all the function calls and it worked fine with no warnings. What's the problem?

By the way, these two lines are redundant:

    alias FileHandling, as: FileHandling
    alias ListUtils

The first line is redundant because `alias X, as: Y` makes module X available with the name Y - so if X and Y are the same then it doesn't achieve anything. And the second line is redundant because `alias X.Y.Z` with no `as` makes module `X.Y.Z` available with the abbreviated name `Z`. But if the original module name contains no dots then then there's nothing to abbreviate.

Not a function but when a module has a struct that you want to use inside the module you need to use the full module name or alias itself.

You can write __MODULE__ to reference the current module. Or if the current module is a struct you can write %__MODULE__{}

That's not very pretty though ;)

IIRC you can also write:

alias __MODULE__

Do you have a repo for your Guile solutions to AoC? Would love to see it.

“Why not Elixir” is a more interesting question and I suggest you go ask it to engineering managers of polyglot organizations. They will usually bring you the super low nps from not-elixir-only devs and the resignation letters from elixir “talents” that are asked to do non elixir stuff.

Is this a problem with Elixir? Have you seen a Java dev using a different technology? They usually haven't even heard of anything besides Java.

> super low nps

Net Promoter Score?

> resignation letters from elixir “talents” that are asked to do non elixir stuff.

I mean, I would prefer to be working in Erlang, but I took a Rusty job recently. OTOH, if I was working in Erlang for you, and you made me switch to something else, I would most likely not be happy and if my job is changing, I may as well change jobs, or at least consider it. Leaving behind simple concurrency and hot loading means it takes a lot longer for me to get things done, and it's one thing to work at a job where things take forever, but it's even worse to go from being able to get things done quickly to slowly.

yes, Net Promoter Score

about the second part of your answer; my (probably very rare) opinion is that our job is not to "work in erlang" or "work in rust", is "solve problems/automate stuff". If I ask you to work in Foo instead of Erlang, it's the same job. I highly doubt that your job is slow because Go and fast because Rust, it's slow because process/idiots in other teams/idiots in your team/idiots as your "agile coach" etc.

I understand wanting to have a good career, but language is never the obstacle to a successful career. Also, this implicit bias that people who know exotic languages are better is completely false.

> I highly doubt that your job is slow because Go and fast because Rust, it's slow because process/idiots in other teams/idiots in your team/idiots as your "agile coach" etc.

Team dynamics are of course, very important. But my job is slow absolutely because I don't have hot loading. Sure, yes, I could build hotloading in Rust, but it would be difficult and the resulting code would look be a lot less idiomatic than my Rust code is already; which makes it harder to do.

Because I don't have hotloading and I work with a system where long lived tasks stay on a single server, I can't change the code for existing tasks in progress. Instead, I have to set up a new cluster and let new tasks start on the new cluster, and then I have to later come in and cleanup the old cluster. This is a lot more work than pushing and loading new code on the existing cluster. And it means I need to wait longer to see the results. And tasks have a limited lifetime. Because of the operational cost of doing an update, it's not worthwhile to do small incremental updates, instead I have to batch things into substantial updates and keep track of more things.

At my previous job, we would regularly have tcp connected clients connected for weeks or months with the latest server code via hotloading. I can't do that now. Either I let clients stay connected to old code servers, or I force them to reconnect and lose their state. (Yeah, maybe there's a way to force them to reconnect and transfer their state; that's not easy though)

It's like making a mechanic work without powertools. Yes, you can do everything, but without the proper tools to do my job, everything takes longer. A mechanic trained without access to or knowledge of power tools might not notice the difference, of course. And power tools offer the ability to do both good work and bad work much faster.

Edit to add:

Re the job is to solve problems regardless of language; I don't necessarily disagree, and I'm definitely willing to pitch in and debug nearby systems that are outside of my preferred language, or even write code in a language I despise if it's the best way to solve a problem. But if the majority of my job is in languages that make it harder for me to work, and that's not something that was clear before I took the job, and especially if it's a new thing, it's going to make me look for something else. I'd expect that just as much for someone who feels Erlang makes them more productive as for someone who feels Rust or Java or C or Erlang or Fortran or Lisp or Python or whatever language they love. There's a lot to like about a lot of languages, even if they don't appeal to me.

hey, maybe you have one of the use cases erl/el excels at; I'm not going to go against that - I also like the language and the runtime and everything.

my point is just that at some point and at a certain org size, the technical prowess of the platform is not the dominant term in the equation; social merits of your platform become it.

organizations also don't see the value of retraining everyone, risking bugs, customer and dev dissatisfactions, and a myriad other correlated problems; and as elixir orgs are not running laps around non elixir orgs (you know, executives do talk with other executives in other companies - and whatsapp is an once in a decade) and given that most of us build web cruds, internal LoB apps, other small automations, they can tolerate the eventual delays of having worser tools.

closing our eyes and thinking these things do not exist is ingenuous imho; I wish I was writing rust or el, I'm stuck in python trying to convince people using immutable dataclasses.

I still feel langs like el/erl and in general pleasant, powerful but niche things like clojure, are better placed as secret weapons for teams of highly skilled, motivated individuals with homogeneous culture about code. They are not industry standard and they should not be. Touting them as magical solutions just hurts them in the long run.

Funny, this happened to me specifically.

We were an all Elixir (and legacy Ruby) shop with some wildly smart engineers.

Got acquired and jammed into another larger org. Got pressured to switch to Java and Go without considering that maybe our system was designed around the way OTP apps work.

Fast forward a few years and lots of us left, and found successful Elixir jobs where we were much happier elsewhere, making much more interesting technology.

Sounds more like an org problem to me.

You've successfully been dealing with a sand problem with shovels. Then the new boss tells you about these amazing forks that they've used in the past to address any and all issues and wants you to start using them too.

Quitting is not you being difficult. It's just opting out of a bad developer experience. Contrary to popular belief, you're entitled to also be happy as a professional programmer.

Which is why I would quit if someone tried to pry high throughput .NET-based microservices from my hands to replace them with something like Elixir.

(A similar story has happened at Walmart when their architects did not understand F# and instituted a Java rewrite of an acquired company)

This often happens regardless of tech. The old school Unix guy that does everything in C... The C# person who hates Java... The Ruby guy that won't do Python... The backend developer that despises Javascript... on and on. Some people compromise, some don't.

Elixir is nice, only if there was a robust job market for it

Reply on news.ycombinator.com