Erlang/OTP 26 Highlights
erlang.org
erlang.org
OTP and Erlang make it so easy to build distributed systems end to end. It makes my past professional efforts look like a shoddy amateurish joke. If you find yourself needing to build a distributed stateful service, Erlang/OTP should be the top choice.
TLDR: GC and runtime behaviour generally are built from the ground up in OTP with this kind of concurrency and programming model in mind. Its advantages can be approximated but not fully replicated in "general purpose" runtimes like the JVM.
Consider that the JVM model is for long-running shared memory systems. So its GC in particular is built around that model. Erlang/OTP as I understand is a lot of short lived processes that communicate through messages, sharing little state. Designing a GC for a system like that has entirely different trade-offs. Isolated short lived actors can throw-away their entire memory state at process end, without waiting for a generational collector to discard the garbage, etc.
And that's just the GC. In general in a system bolting "actors" onto another shared memory language it's always going to be possible to "escape" that framework and do things in another paradigm. That might be fine for the real world, but it negates some of the advantages that OTP brings to the table.
— Joe Armstrong, on Elixir (2013)
The thing that Erlang and Elixir have different is the BEAM VM and its support of concurrent processes at the VM level. So there are some things that they can do that Akka can't. For example, the scheduler is preemptive and keeps processes alive even though there may be rogue processes pegging the CPU.
Whereas Erlang/OTP is a very pragmatic, simple language and runtime geared solely towards building distributed services. The gen_* behaviours are awesome and I’ve never seen anything like it (at least in terms of simplicity and practicality) in other languages or runtimes.
what might explain this? I mean hypes and infatuations are common but tangible advantages usually win the day because in the end people will pick the right tool for the right job.
is it that distributed systems are not actually as big a domain so it does not get much press?
No, they don't. They pick the familiar and the easy-to-hire-for. I've worked on systems (written in C++, Java, others) that would have been perfect for Erlang, but at most I was able to present a prototype. No one is interested in a better solution that they can't hire for or have to retrain their staff for. (The familiar and the easy-to-hire-for may, by some metric, be the right tool for the job, but it isn't necessarily the right tool by technical metrics.)
Not too long ago I interviewed with a company that only uses Erlang, not Elixir, and they rejected me because I am not an Erlang expert. There wasn't anything I could say that would change their minds. I even volunteered to do a coding challenge of some type & they still wouldn't consider me further.
Sometimes a talent shortage is a state of mind.
Wow... was this an actual hiring manager or someone, or like an HR person with a checklist?
I've interviewed for one or two Elixir positions where the opposite thing happened: I talked with someone who had a checklist. I haven't done Elixir professionally even though I have extensive Erlang experience and that was that.
There is a pretty stark difference between "has architected an application with OTP components" and "can write linear code in Erlang or Elixir", but even then you'd probably be able to fit someone in unless you're hiring for that kind of senior/architect position.
> No, they don't.
I'm in strong agreement. People pick the option that has a good story associated with it. Having a good story protects you in the event of a failure. And it protects your non-technical boss (or your boss' non-technical boss if your boss is technical).
I assert the following:
Scenario A: I have a literal mathematical proof that erlang is the best choice. I implement the solution, but due to the market changing (or an earthquake or ninjas literally destroy the server room) the overall project fails.
* Me: But, I had math proving everything out. Ultimate Boss: I don't believe in math, you're fired.
* Boss: Well, it failed, but look at this incomprehensible math that I don't understand. Ultimate Boss: You've obviously been hoodwinked, you're fired.
Scenario B: I have a really good story that includes phrases like: "best practices" and "industry standard", and is doing exactly the same thing as everyone else. The project fails because the tool isn't great.
* Me: It failed, but here's a good story. Ultimate Boss: Well, it looks exactly like everything else, so I'll look for a different nail that's standing out from the crowd to blame for this so that the muggles don't get nervous about taking non-chances in the future.
* Boss: Here's this great story that I totally understand and can embellish upon because it operates on storytelling and not category theory. Ultimate Boss: That's a really good story. I'll have to steal it myself to give to the board to handle some unrelated problems of my own. Better luck next time.
Even the scenario of master craftsman using erlang because it's the right tool for the job is more about a story about an uber expert who is off doing wizard stuff than it is about an actual merited set of reasoning.
The "nobody ever got fired for buying IBM" defense. Which has since been updated to Oracle, Microsoft, Java, C# (for a large portion of the enterprise world), Python, and Javascript.
Best-in-class that isn't mainstream is a hard sell to employers if they are strongly risk averse (which many people, if not most, are).
I understand why Elixir was born.
I think Typescript would be "easier" for a larger group of developers to hook on
But the hard part of Erlang is OTP / understanding the concepts or rather unlearning the concepts from other programming paradigms.
Then again, languages lose adoption pitches based on whitespace and tabs significance.
If one can get past the initial impression though, they'll find it's actually a very simple, nice language. Much simpler than Scala/C++ and probably even Java these days. Also simpler than other functional languages like Haskell. I picked it up in about 1-2 weeks. There's very few gotchas in the language or hidden complexity. It flows well.
I'll also add the language suits the usage pattern. Pattern matching works very well with the gen_server approach. And the binary syntax is so nice if you need to deal with over the wire protocols, as is common in distributed systems.
A lot of it is unfamiliarity with the tool and unfamiliarity with the paradigm. And of course, you can do distributed systems with other tools that people tend to be more familiar with, so go with what you know. Erlang is adaptable, but was built for a Telcom setting with a lot more permanence than today's popular cloud model with nodes coming and going.
On the other hand, an awful lot of large scale chat systems start as ejabberd and adjust from there; sometimes rewritten in another language, sometimes rebuilt in company specific Erlang while the service is running, sometimes without muvh adjustment.
For example, you can effectively architect with microservices without actually creating more deployable artifacts. This is because each GenServer is already a microservice. You build it from the ground up reasoning through failure modes and coordination. Those are easier to separate out into its own deployable artifact when the time comes, if it ever does.
People hear about code hot reloainding (where you deploy new code without shutting down the production environment) and think that’s the killer feature of the BEAM vm. Code hot reloading is a narrow use—case of a more general capability — you can directly write or patch new code in a running system in a live environment through the REPL. That has been very useful for improving the system, to stop the bleeding or identifying bottlenecks and trying out fixes, even when deployed as ephemeral pods in Kubernetes.
Elixir’s runtime, unlike many others, uses preemptive lightweight processes. Unlike async-only systems, a runaway process won’t block out other processes. That’s just the start of the kind of advantages in writing with Elixir/Erlang/OTP
It’s been battle tested for years. I think many of the hype cycles for distributed systems reexplore or reinvent what Erlang and Elixir has to offer.
So, Elixir/Erlang combo seems like a perfect fit. Instead, we are having all remaining Elixir and Erlang systems ripped out and re-written in Java.
The reason is simple - readability and hire-ability. The cherries on the cake - Java's "good enough" for what we do, the tooling, the quality and availability of reference resources and subject matter experts - are just that, cherries. But the ability to hire and onboard new engineers is THE thing - it took us 3 years to fully hire out the headcount we were allocated for just our little team and that is in Java. Hiring for Elixir and Erlang has been such a pain that we are having to pay contractors while we re-write the current Elixir and Erlang systems into Java. Our domain is somewhat complicated - most people who have not worked in food retail generally would not imagine it to be so - so onboarding people to the domain knowledge takes 3-6 months and that is without "This is what? Erlang?" barrier. That is where readability of Java and non-FP vs Elixir/Erlang and FP comes in as well.
I guess what I am trying to say is that, yes, the right tool for the job and all that but good enough tool with a significantly higher hiring pool is proving so much better.
That said, it is a fantastic system for some problems and I really miss working with it.
We used it as the command and control language for these: https://www.icare-world.com/us/product/icare-eidon/
* Semi-embedded system that's used as an appliance, not a computer.
* Runs in an environment where people mostly don't want to mess with it. Might be days between reboots; might get turned off every night.
* Has to be pretty deterministic in terms of memory usage, response times, and so on.
* Recovers from something going wrong by going to a known-good state.
We didn't actually use any of the distributed aspects of it, but it worked out great for the other stuff.
Scripting, string munging, any sort of small task, anything which runs locally (unless it’s changed significantly erlang’s distribution story is the worst).
It does have good FFI, like rustler for the elixir is great. It makes sure rust panics unroll into elixir exceptions and that takes some of the scariness of writing them.
That plus dirty schedulers means you don't have to be a good citizen and can write cpu intensive code in a faster language.
the main reference that shows up if you search for erlang + hpc is from 2008
https://www.researchgate.net/publication/221211398_High-perf...
I'm a Elixir and Erlang developer who is just starting to explore ML, but these libraries seem solid and well designed.
There is also the NX project (numerical elixir) which compiles tensors to GPU code making it much faster for those particular usecases. https://github.com/elixir-nx This has things like explorer which is elixir bindings to polars, livebook which is like jupyter notebooks (with enforced execution order thang god), aswell as some machine learning libraries in axon and scholar.
There has also been some work with libraries like flow and broadway which allow pipelines to be constructed with the usual syntaxt `step1 |> step2 |> step3` but to be executed across cores (or cluster).
I can imagine orgs using a combination of the above tools to do some interested distrubuted data pipelining and ML.
I am not at all saying this is you, to be clear, but one problem I have had in the past looking for Erlang jobs is that occasionally you find people who view it as some kind of magic sauce, or "I'm curious to see what it's like".
In the former case, they think just writing their code in Erlang is going to give it magic scaling powers. This is probably less common these days, but I recall a while back when Erlang first got a bit of hype, there were people who weren't thinking very clearly about it.
In the latter case, sometimes it's not the best tool for the job but someone wanted to learn the intriguing new thing.
Edit: and of course there are great use cases for it. See my other comment elsewhere on a system where it was a great fit. I'm always curious to see where others are using it successfully.
but also
Choose the right tool for the job.
Plenty of people build highly tolerant, robust distributed system of top of regular languages when Erlang exists, and no one bats an eye. Erlang is no magic sauce, but for its niche it is unrivalled and yet routinely ignored in favour of subpar platforms because of dubious concerns such as "there are no Erlang/Elixir developers around", which is nonsense.
The true reason is that there are plenty of engineers and architects that have no exposure to the BEAM, so have no idea how productive they could be developing their distributed system on it. They simply do not know what Erlang & co. could do for them.
Certainly, but for each and every small task you do, you don't evaluate a new language for it. If you have a web application that's done in Rails and have a bunch of Ruby programmers, if you need to, say, add a simple web scraping system to it in house, you're likely going to reach for a Ruby tool at first.
On one side, you have shops that are like "we only use Java". They're maybe missing out on some good tools for various tasks. The other extreme would be a big list of languages, and that's difficult to maintain as people come and go.
Are you building a distributed system? Are you building a highly concurrent IO-bound server? Are you building a system that needs to be fault-tolerant? Going for anything other than Erlang/Elixir is a misinformed choice.
The thing is, most people don't start out wanting a distributed system, it's something you get when you've scaled to a certain size.
If you know you need it, and Erlang's version of it is a good fit, yes, it's a good way to start.
I find that is often a lot more relatable to average-joe-tasks than bigger picture distributed ideas, even if they're actually the same thing at their heart.
I just don't want one request to blow out the system, or some sporadically turned remote API to hose everything, Elixir gives me that (with the simplicity other languages often need other stuff stuck onto to handle).
Your options are
* Keep trying to restart it forever if it fails (by setting the timeout so low that it'll keep trying)
* Try a set number of restarts, then pass the failure higher up, potentially taking down the whole system if there are enough failures.
* Just let it fail and walk away.
https://www.erlang.org/doc/design_principles/sup_princ.html#...
If you have something like a web site with a database, where 1) the database is critical to the normal functioning of the system, but 2) you still want the site up to let users know that "it's broken and we're fixing it", and maybe also send some alerts out, you might want something more like an exponential backoff or some other more involved strategy, which isn't shipped with Erlang.
In fact, I am writing this week a dynamic process pool in Elixir which is so peculiar in its requirements I don't think I've ever seen anywhere, and it's in total 500 lines of code spread over 2 GenServers (+ Registry and DynamicSupervisor). I reckon it would easily be 25x as much in C++ or Rust.
If you want exponential backoff, I suppose it should be pretty easy to implement with a custom Supervisor (though the best practice would be to have a sidecar GenServer to deal with the backoff/alerting business rules, and using a regular Supervisor.)
Back in the day, I found this to be a pretty good 'circuit breaker' type of thing: https://github.com/jlouis/fuse
I wrote something similar for the (OTP) application level: https://github.com/davidw/hardcore
If your BEAM app goes down because your GenServer crashes, something is terribly wrong with your code.
We've been able to easily scale with minimal effort and have never had downtime.
The dev responded that 3 of their primary third party services were built on Elixir, so they're worried about using it in house they should probably drop those too.
AFAIK that worked.
__
0. HN post: https://news.ycombinator.com/item?id=35914216
1. Article: https://gist.github.com/evadne/c9aeca424d30f024ba048158bcad3...
https://www.erlang-solutions.com/blog/how-hca-healthcare-use...
Yes, it is. It's not a statically typed language. strong typing is not synonymous with static typing.
What's not solved by erlang is IMHO the hard part : how does your system ensures data persistence, availability and consistency ?
Rolling my own cassandra or kafka or distributed db isn't really something that enchants me, and i don't see how the beam can makes this any easier ( they are fundamental problems, not technological ones).
We need a larger ecosystem, because most of the times when people like me look for what language to use to solve a problem, we try to select the best / easiest tool for the job, and many times if the ecosystem has easy to use tool, we go with that.
Not all developers build distributed systems and even tho I'd love to, many times I don't have the capacity to build things I need from scratch so that leads me more towards java/js/python sadly.
Finally, Joe can stop rolling in his grave, as newbies will no longer have to ask why Erlang forms are not possible to express in the shell.
See the "Funs and defs are not the same" section from https://joearms.github.io/#2013-05-31%20A%20week%20with%20El... for details
My only gripes would be that I've had to switch to Elixir as it's more developed for web development. I actually like the syntax of Erlang more (I don't come from Ruby) and it's a shame that they're split. Also, in Elixir, I attempted to write the fibonacci sequence using memoization and the solutions I saw online were horrendously ugly. Maybe it's because I'm still a beginner but I did see an example in Haskell and it looked so much cleaner.
I love how simple the actor model is and how it just works in Erlang. No ifs, ands, or buts.
I know a lot of the comments are about the awesomeness, or how hard it is to build teams, but honestly, the language is great.
When i learned about :ets and could move data around at lightening speed without having to use a database, i was in love!
I really hope the focus on performance improvements hasn't waned.
HN comments from 2-weeks ago on this topic.