The Development of Erlang (1997)
fermatslibrary.com
fermatslibrary.com
The thing is however, with Erlang you get all of that in a pretty easy to learn language and it will scale to practically anything. Erlang is also an incredibly productive language. You can get a whole lot of things done in a short amount of time.
While I understand there are some path-dependent reasons for why we ended up where we are at, but setting up a new service as part of an Erlang release is just an order of magnitude simpler. It's not even close.
Maybe this is something to consider: by the use of gen_servers erlang/elixir allows mere developers to build a service, which in the end is just a (lightweight) process managed by OTP (the erlang runtime, so to speak). No ops involved.
It's like sort of Dr Seuss riff for an experienced programmer.
So you've done a lot of program?
Can you, can you, without a reference?
Could you, could you, with nary a pointer?
Will you, will you, use no return?
No for loop? And less control flow goop?
I had to switch back to do some Python and Swift maintenance/development last week after 2 solid weeks of climbing the Elixir newbie staircase, and I was amazed at how much code I write in "normal" languages whose sole purpose is to decide where the code goes next. For the first hour or two my brain kept screaming "why do you keep putting all of these if blocks everywhere?!"
After that I just read some codebases. I find elixir codebases extraordinarily easy to read. The language does a great job of making ‘the good way’ also ‘the easy way’, and the linear nature of functional code is much clearer than the usual indirect dependency injected hell. Papercups is a fairly good starting point I think. I liked the livebook codebases too.
As others have mentioned on some other threads, learning about OTP, Genserver is really worthwhile. Apart from language, the book teaches a bit about that too. But maybe just enough.
It's not a quick start though: I really dig into how Elixir works and move fairly slowly. I've finished covering the language itself, and I'm about to move into the OTP functionality that's inherited from Erlang.
Also, the official docs and guides are fantastic once you have your bearings.
it's more like the = sign in math mathematics.
2x = 10 we can deduce x = 5
Here's an elixir example where a bunch of "if/else" blocks are refactored to use case statements, and then from case statements to pattern matching.
https://nickjanetakis.com/blog/refactoring-elixir-code-if-co...
After using mostly Erlang for several years, 'if' feels like ugly atavism.
If-expressions is a specialized version that only works on booleans, pattern matching is a more general concept (that of course can be used to implement if-expressions, but so can pure lambda calculus)
And you still have to decide where your code goes at some level. The pattern of having two function variants (one which matches an empty list, and one the head/tail) is a different idiom to looping. So while for loops go away, you still traverse sequences. And more generally, a lot of the if(else) clauses I would normally write, now get solved with pattern matching.
With control flow languages, the where the code goes and what the code does feel very intertwined, whereas with the pattern matching idioms, I find that where it goes, and what it does, are more orthogonally declared.
I wouldn't spawn and kill thousands of processes with Kubernetes. That's a bit heavy-handed. A couple of Erlang pods that do the same? Sure.
- light "threads" and supervision of them
- OTP which is largely a library other languages with message passing and light threads could implement
- garbage collection per light "thread" rather than stop-the-world
Akka was on the right track in supporting the same approach to highly concurrent and fault tolerant software. The JVM might have gotten better at garbage collection without paid plugins since 2015 when I last used Erlang?
But I'm curious what stack in particular is seen as superceding erlang.
PAAS level deployment and failover is sufficient. (Edit: for my needs at the moment)
If you keep your async tasks in memory in any system they'll disappear when you deploy.
So they weren't really reliant on the things that set erlang apart. So the cost of the small support base in tooling and libraries is more significant.
But I should learn Erlang eventually.
Quickly going through https://learnyousomeerlang.com/ to discover OTP, processes and supervisor and you're ready to code.
After learning Erlang, and writing some basic stuff with it, I discovered Elixir and had nothing to learn except:
- "defmodule" instead of "package"
- "@callback" / "@behaviour" instead of "callback" / "behaviour"
- ":an_atom" instead of "an_atom"The pinning operator x = :x; {^x, y} = {x, "y"}
Exceptions are slightly different.
There are a few others that made me realize "oh shit I need to learn Erlang first."
Therefore saying "you need to learn 2 languages" isn't correct. It's like saying Javascript/Typescript are 2 different languages. Well yes, but if you're fluent with Javascript, you'll be able to read typescript, and if you're fluent with Typescript, chances are that you are fluent with Javascript too.
But I agree that one should learn Erlang first, because most of the documentation about Elixir seems to assume that you already know Erlang/OTP.
“Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.”
That is, while other languages and systems barely scratched the surface in terms of distributed programming, Erlang was perfecting all the models needed for it, because that's what it was built for. So everyone else is now playing catch-up with Erlang.
I have learned Erlang, and like its Prolog inspired syntax, but that is all about it, there is hardly a place for it on the Java/.NET/C++ shops I work on.
Akka, Orleans, HPX, OpenMP have much more uptake.
As for industry adoption, it is used by some game studios, for example,
http://www.odbms.org/blog/2016/02/orleans-the-technology-beh...
https://www.gamasutra.com/blogs/AshkanSaeediMazdeh/20170209/...
http://rvirding.blogspot.com/2008/01/virdings-first-rule-of-...
K8s is like an additional level of supervisor that applies to an entire node, and provides really easy scaling and clustering. It's a great tool to run an Erlang app, especially if you already have a managed k8s cluster anyway.
This article describes it well: http://blog.plataformatec.com.br/2019/10/kubernetes-and-the-...
But as I said in that thread, that's actually one of the traps.
While Erlang/Elixir as languages and runtimes will scale to "practically anything", a lot of the technologies in the Erlang stack don't. Mnesia is almost useless, if not totally useless. Erlang's not the slowest language, but it's not very fast, and using something 5-10x faster can "scale" nicely too by needing that many fewer systems in the first place. The message system is not what I'd call best-of-breed anymore; very ahead of its time, but most messaging APIs have more functionality in them now for a reason. Erlang is 1-or-none and 1-or-many seems to be the choice winning out in general.
As long as you stay in the ecosystem you're OK, but if you're using Erlang to speak to a non-Erlang DB, to interact with a non-Erlang message bus like Kafka or something, to call APIs from non-Erlang systems and provide APIs to non-Erlang systems, all this integration becomes more a trap than an advantage. Plus Erlang's type system starts to become a bear in those contexts; Erlang's types integrate well with Erlang, but are just quirky enough that it's a pain to integrate with anything else. There isn't even a clean embedding of JSON into Erlang types. (Erlang, like many functional languages of the era, made the mistake of defining a "string" as a "linked list of numbers representing characters", which makes lists ambiguous as to whether they are lists or strings.) Interaction with non-Erlang systems is always kind of a pain because of the lack of clean embeddings into the Erlang type system.
In 2005, scaling to "practically anything" was a pretty decent advantage for Erlang. Scaling to "practically anything" is table stakes for a language/runtime now. Someone else says languages are "catching up" to Erlang, but to a large degree, they have, and exceeded it in many places. To be honest I think it's Erlang trying to catch up now. And it can't. It's too integrated, too opinionated. What was absolutely an advantage in 1999 and 2010 is now a significant disadvantage.
People try to scale Erlang systems by trying to get one distribution set to have thousands of nodes; and then taking that thousands-of-nodes cluster, calling that the “master” cluster, and then having another thousands-of-nodes cluster and calling that the “hot standby” cluster. But that’s precisely backwards.
Think like Ericsson. You’re designing a switch. You have two independent problems:
1. How do you make your switch Highly Available / fault-tolerant?
2. How do you handle higher load than one switch could possibly handle (for either hardware or software reasons)?
To solve problem 1, you create nodes, with roles (e.g. a master and hot standby), where there are an exact number of pre-defined nodes, each with exact static roles. These nodes have defined fault-tolerance relationships. Databases like Mnesia that replicate from one node to another. Applications that have defined chain-of-command such that it’s clear which node becomes the leader if the first leader fails. Etc. You design your system so that some of these nodes can go down, or netsplit, without the system itself going down/crashing. And then you deploy these nodes to separate machines, so that hardware failures will end up being treated the same as node failures.
To solve problem 2, you stamp out copies of this entire system of nodes, and add more infrastructure external to it to route between them.
In this model, a system of nodes — a distribution-set — is the same type of thing as a Kubernetes Pod. It’s several static processes, one of each role, that inter-depend, and are “wired” to one-another; and then get managed together / treated as one workload. The big difference is that, in contrast to K8s pods that all must get scheduled to one host together, Erlang distribution-sets usually get “scheduled” across several machines. But they still have one identity. They’re a single abstract machine that happens to consist of workloads running on multiple physical machines; where those workloads going down is a designed-for and tolerated aspect of the abstract machine they compose.
Everything that happens inside that single abstract machine, is the purview of Erlang and Erlang’s stdlib. Application failover, Mnesia, the thin-client code_server: they’re all designed to make a single abstract machine out of a defined set of Erlang nodes where each node has a static role.
Everything that happens outside of the abstract machine, is the purview of user code. Want to horizontally scale the abstract machine? Don’t use the distribution protocol; it’s not tuned for that. Use a userspace library. Have an application that does abstract-machine-to-abstract-machine peering. Make it the role of one or more of your nodes to run that application. Make it fault-tolerant, too.
Under this model, I hope you can see that statements like this are mostly incoherent:
> Erlang's not the slowest language, but it's not very fast, and using something 5-10x faster can "scale" nicely too by needing that many fewer systems in the first place.
...because an Erlang system architecture that builds one Highly Available abstract machine out of five-to-ten nodes, cannot be swapped out for a system architecture composed of one node, no matter the language it’s written in.
The other system, to implement the same fault-tolerance strategies, will need exactly the same number of nodes, because each node performs one or more roles, and serves as a VM-level failure-kernel / bulkhead for HA within those roles.
The non-Erlang-implemented system will just have a harder time implementing such strategies, because it’ll mostly have to use userspace libraries/frameworks — rather than VM-level abstractions — to do so. (Though you can get close with languages that have basic syntax that compiles down to framework code in other languages, e.g. if there were a hypothetical “Erleans” that compiled down to C# + .NET Orleans framework calls.)
——————
Of course, none of this matters if you don’t have the problem that Erlang solves. If you don’t need a stateful abstract machine that keeps itself online and computing over that state at all costs with fancy multi-node footwork, then you really don’t need any of Erlang’s distribution stuff.
But there are still, today, problems that do put you in that position. Stateful packet switching is still a thing. Game servers are another good example. App-layer cache servers (Redis, memcached, etc.) where your architecture couldn’t survive the thundering-herd that would happen if you needed to re-warm the cache from a cold start. Etc.
In those contexts, do you really want to try to jury-rig a solution using e.g. Hazelcast + IPVS failover? Better to just use Erlang. Because it’s still ahead of its time, for this problem domain.
It's amazing at what it does - If you had lots of physical nodes and needed to build a robust, fixed-size system at any moment in time, it's an incredible option.
But with container orchestrators, you get the ability to design a polyglot with more fluidity in your design. The admission that different languages are best suited to different tasks throws a monkey wrench into the Erlang ecosystem, because now I all of a sudden can't use Erlang for everything.
It does sound like an excellent choice for a control plane in many cases, however. I think it's phenomenal at what it does, but I think you have provided excellent context regarding its ideal use case.
The other commenter said it best: "But at the end of the day, Erlang is one technology that is good enough for most and does not require large operational experience to run. Your entire stack is in your own code, understandable by your entire team."
So the difference is doing your scaling and distribution natively in code, versus hiring and maintaining devops expertise to manage your containers. On top of that you now have one more "node" in your team (whether a single devops engineer or a group of them) that needs to collaborate with everyone.
And anecdotally, for me who never had the need to use distributed erlang in practice, the benefits lay completely elsewhere. Just looking at programs written out there in the wild, some ubiquitous, and knowing the mess that is software development makes one wonder.
This a great way to make things:
- You have N Cores on machine
- You create N Threads to match, you don't "create/destroy/create/destroy", no, you already have the size of the problem set!
- You assign roles, maybe just split between "heavy" and "light"
- Your context switching is for computing small things on light
- You do sequential code, tight loops and heavy I/O on heavy
I truly wish I could do something like this on most languages (ie: is part of the paradigm/semantics). I use Rust now, and async is great but the mental model is HARD.
- You don't have a easy way to split between workloads, anything could suddenly stall
- When is easy to launch async/goroutines you launch them like is no tomorrow.
- So, this mean I don't know what are the size of my threads or task are(could) be running without manually inspecting or assume.
- I don't know, after look at async function, to know which workload is for (without inspect the code and the flow of it) (ok, in rust you can maybe infer if you are clever in the use of types to be markers... but still is disconnected to the run of the program)
"container orchestrators" is like put a hammer to handle another hammer ("heavy OS + VM/DbEngines") under the pretense of being "light" and efficient, when actually this is how most software stack are:
- A "heavy" DB engine - A "light" logic server that connect to it - MAYBE out-of-band processing for send emails and stuff
So, the thing is, having Kubernetes or anything like put the solution too far away. I need, as developer, to MODEL that relationships and KNOW in what workloads this or that must operate. And the most fixed/predictable is it, the better.
You are then adding some redis/memcached, kafka or something similar, etcd perhaps and kubernetes.
A bunch of technologies your developers need to use properly and be running to develop on.
As opposed to having an easy to use single technology named Erlang which is almost certainly good enough for your needs. And your developers can easily be running a distributed system of erlang nodes locally without any issues or complex setups.
I would note that Erlang is pretty good at absorbing other things into the Erlang abstraction. Depending on your needs, you can use:
• NIFs (library code loaded into the Erlang VM — good for performance, but not good for fault-tolerance)
• Port programs: arbitrary POSIX processes (as long as they speak a specified protocol on their stdio) spawned by the Erlang VM and communicated with + managed through a fancy socket abstraction (Erlang ports).
• Erlang "C nodes": arbitrary software that links in Erlang's erl_interface distribution-protocol library (https://erlang.org/doc/apps/erl_interface/ei_users_guide.htm...) as C code, and so can then join an Erlang distribution-set as a full peer.
That last one is especially interesting if you're building a polyglot system where Erlang is just one component: it allows you to build an Erlang "node" entirely in C or Go or Rust or Java or Haskell or whatever, that — from Erlang's perspective — can occupy one of the node-roles in your Erlang system architecture; while from the other software's side, can treat the Erlang abstract-machine as a guest within some other abstracted system. Through the erl_interface library, the C node gets access to all the interesting distribution and fault-tolerance tools of Erlang (e.g. it can interact with other nodes' ports; it can hold monitor refs; etc.) So you still can "use Erlang for everything" (related to distribution and fault-tolerance, at least) even in your other software.
> As long as you stay in the ecosystem you're OK, but if you're using Erlang to speak to a non-Erlang DB, to interact with a non-Erlang message bus like Kafka or something, to call APIs from non-Erlang systems and provide APIs to non-Erlang systems, all this integration becomes more a trap than an advantage.
I fail to see why. You can write an API and communicate with an API just like any other language.
> Erlang's types integrate well with Erlang, but are just quirky enough that it's a pain to integrate with anything else.
You integrate through APIs. I do not see how it has anything to do with Erlang's internal types. Any service designed to be used by a diverse set of clients requires a clean API.
> (Erlang, like many functional languages of the era, made the mistake of defining a "string" as a "linked list of numbers representing characters", which makes lists ambiguous as to whether they are lists or strings.)
Where does this matter in practice? I've been building Erlang software for 10+ years and people keep criticizing it for this and I still do not understand why this is such a big deal to everyone.
> Scaling to "practically anything" is table stakes for a language/runtime now.
Python/PHP/Ruby/Node.js are pretty terrible at taking advantage of many core systems.
> Someone else says languages are "catching up" to Erlang, but to a large degree, they have, and exceeded it in many places.
Erlang is still unmatched when it comes to ease of use building reliable distributed systems and it is not even close. Yeah you could run Kafka, Kubernetes and a bunch of other technologies that do their job better than Erlang does it natively. But at the end of the day, Erlang is one technology that is good enough for most and does not require large operational experience to run. Your entire stack is in your own code, understandable by your entire team.
Mmmm it's not quite that simple. Erlang requires thought and effort to scale, same as anything.
Erlang is a programming language that allows message passing between concurrent actors. Are you saying that, containers are comparable to actors, and that using IPC between them is equivalent to message passing?
Maybe - but that requires the internet to work. That’s a pretty large dependency to bring into what Erlang does in a single process.
- "failure happens"
- "think asynchronous"
- "processes, not objects"
and Joe had a very nice email on it [2].
K8s is "the way" but also very complex (it does not shield you from the low-level network complexity, like Erlang does).
[1] https://gioorgi.com/2015/erlang-lesson1/
[2] http://erlang.org/pipermail/erlang-questions/2015-April/0841...
The funniest part of the talk was when Joe presented a diagram showing the stock market value of Ericsson being up always when he was working at Ericsson. :-D
I even had a chance to have a beer with him afterwards and talk about some programming ideas related to audio programming and also some related to typography!
RIP Joe Armstrong, you were an amazing person and programmer.
From the talks I have seen (youtube) I wonder how would it be if Joe had Rich Hickey's presentation/ communication skills.
Huh, I always assumed it was a portmanteau of Erricsson and Language.
I’m sure it’s both.
https://en.wikipedia.org/wiki/Erlang_(unit)
So, yes, a clever name all around.
Regarding your recommendation of Actix: according to its website it seems to be mainly a web framework?
Ditto for something like the actor model.
(Just to be clear: "We" does not refer just to devs/engineers, companies may also be wary of using an alternate technology, even though it could bring benefits down the road)
Every place wants to be the best. And yet, most places, when confronted with new technology, ask the same questions. "Who else is using it?", "How easy is it to hire for?", and if they're a bit sharper, "How much library/community support is there?" And they ask these without realizing that all of those questions will ensure they adhere to the mean. Positioning themselves that way prevents the risk of dropping below the mean due to technology choices, but it also prevents rising above it. But that's okay with most corporations, which are all about risk management, and who don't equate a tech win -> faster/better software -> better user experience -> a business win.