Ericsson to WhatsApp: The Story of Erlang
thechipletter.substack.com
thechipletter.substack.com
He kept asking "why use this over Java at my work X". He seemed to have the idea I wanted to join his company and rewrite their web administrative tools used by 5 people in Erlang. Not sure why he got so annoyed, and didn't ask me about anything other than a language he thought was silly!
The interviewer kept mocking it and asking what did I mean by prose and whether I can also write in rhymes if needed, and ended up not making an offer. Twenty years later, good writing is still the most important skill in my career and the one that I rely on most often.
About 10 years ago a friend of mine asked me to apply for a job in his team (at this big famous company starting with A). I had exactly the right qualifications and we had worked before so it was basically a done deal. I send my application through the normal channels and their HR team rejected me before reaching my friend.
Among my qualifications I had jokingly added that I could fake a Pink Floyd riff or two on the guitar...
I was asked to apply again. I declined.
The argument here might be less about Erlang and more about Maslow's hammer - i.e., if all you've got is a hammer then all you can see are nails.
At least for us, CS is just algorithms and maths, we had only one single course for a specific language in my five years (C) but you where required to just pick up whatever the professor chose. If you check the list of computer science awards they all focus on algos & maths so it feels weird if you could get a degree with a "language is what matters" mindset.
Yes, there are courses on Java and C/C++ (and also LATEX, MATLAB and Prolog!), but these are a few out of 60 courses in total, which include things like algorithms, information theory, distributed systems and optimizing compilers.
...although maybe Cambridge isn't a "standard" CS degree?
Both Oxford and Cambridge currently begin with an ML as first language (Cambridge chooses Ocaml), for the reasons I was taught an ML first too. These languages have sound fundamentals, their type system is good, they can handle recursion and other natural ways to express software. Also, almost none of the entrants know an ML [this will have started to change somewhat because of Rust] and so the introductory material holds everybody's attention and you don't need to spend as much time breaking bad habits.
Places which begin with just Java need to wrestle with the fact that Java has some very weird assumptions which you need to either mention but move past or else ignore and then they leave a weird hole in the knowledge of your students. (For now) All Java's user defined types live on the heap - that's very weird, do you call it out?. Some of your cohort know this language well, and so they can do your exercises without paying attention and they lose interest right at the start of your course which isn't good. But others are new to programming entirely as well as Java, for them every concept you introduce is a general idea - this isn't "How variables work in Java" for them it's "How variables work". Beware the resulting prejudices you have created.
Reminds about me I was talking to a company at a booth. They were using Java and up comes this guy who started professing that that was all wrong and they should rewrite their whole stack in Lisp.
I guess you’ll become allergic to minority programming languages if that happens often enough.
Same story except it wasn't Lisp, but Rust
Theres zealots on both sides of the fence.
The vast majority of programmers don't read HN, reddit or really follow any industry news. They work in Java or C# which are entirely self sufficient with very comprehensive SDKs.
It's a real problem. So after few bad encounters businesses might prefer to hire boring people.
Having to hide your interests to look more bland might be the saddest thing I read today. I Just imagine that those recruiters have some sort of list of subversive activities.
1. Was an active member of the Red Army Faction OR
2. Read a book about Erlang or Lisp, is now too dangerous to be reintroduced into the general developer population
A link to a github account is far more valuable IMO. If you have nothing on github (or another code sharing service), then you probably don't have a passion for software.
This is a communication problem, and not something you can infer from a CV.
I can't see why having experience with any particular language would be a bad thing—quite the contrary. There are concepts to learn from each language, and the more languages under an engineer's belt (particularly "weird" languages), the more open they are to novel solutions. The tool analogy works well: you would want someone who's familiar with an axe and a chisel, and not only a hammer.
Someone thinking this is a bad thing shouldn't be making hiring decisions. It speaks more of the company, so OP probably dodged a bullet.
However it might be one of the only professions where this tool-obsession is so valorized and where there's a whole subculture of it, and where it proliferates at such a high rate. There are people who've made half their career out of chasing the technology of the month or advising that a rewrite in language/framework/technique will solve all problems.
It's of course natural because we talk about the tech industry, and so people become focused on the technology rather than its application to human problems, or see all problems as a technology choice issue rather than a issue of implementing business logic with technology choices... So it depends on the discipline of the engineer how deep down that rabbit hole people go, and the quality of management about how well they can keep their staff focused on the problem domain first, and tools second.
All this to say: I've been that engineer, I've also suffered from those engineers, and because of that I'm now careful. I think Erlang is great. And my career now is Rust. But I hope I'm wise enough to recognize that how harmful rewrite in X and if only this was built in X can be to customers' projects and budgets.
Not to say the original-comment was that person, or that the person sneering about the Erlang was right, but I do understand where the wariness comes from.
All that said, I have more fear of the proliferation of novel JS frameworks than I do of stuff like Erlang or Lisp. And Java is another place where esp in the early days the proliferation of frameworks-as-solutions was a real problem.
To be fair, this is exactly how I would react if Haskell is the main language on someones CV.
Disclaimer: Im okay with Erlang but hate Haskell and ML derivatives
That's what people do with rust! Not erlang!
Oops - wrong pun - or maybe - er - lang one.
There was a huge push to move on from the in-house PLEX to industry standard software engineering. Object orientation, component architecture, UML, design patterns, etc.; think of a software engineering kool-aid from that time period, they were drinking it. And C++ was the industrial strength vehicle to deliver all of this.
AXE-N wasn't just a project that flopped, it was the crown jewel of this giant effort to switch to the new "industry standard" way to write telephony software. So the failure was a giant cultural shock.
But even after the AXE-N failure, the phalanx behind it weren't done for, and apparently it was they who managed to push through a ban on Erlang, that embarrassing thorn in their side.
As for what happened later, I don't know. Anybody who knows what language Ericsson is currently using to program their exchanges?
Open sourcing Erlang and it finding some success outside, meant it's no longer an in-house only language.
I left Ericsson many years ago, but I bet that they are still using ClearCase for version control, FrameMaker for documentation, and that there are still five managers for every developer. Pretty sure they are still drinking the kool-aid, trying to put AI in their exchanges glued together with TypeScript.
Even back then you probably could use something as simple as xmlrpc and get the job done at 1/100 of cost of an CORBA solution.
It all feels very CS-theory elegant and then you get to OTP, the most grimy roughneck Système D ass standard library I've ever seen. But that has no actual problems anywhere in my experience other than that you'll go years not knowing something is in there bc you didn't guess what five letter abbreviation it would be under.
Plus the syntax is like what you would get if you described a programming language to someone who had never seen one. Code from another planet. Flawless.
The key components of OTP include:
OTP Behaviors: Behaviors are generic programming frameworks that define the structure and behavior of a particular type of process. OTP provides several predefined behaviors, such as gen_server, gen_fsm, and gen_event, which encapsulate common patterns of communication and state handling.
Supervision Trees: OTP introduces a hierarchical structure called a supervision tree, which allows for the supervision and management of processes. Supervision trees help handle process failures and enable the system to automatically restart failed processes or take appropriate actions in response to failures.
Application Framework: OTP provides an application framework that facilitates the organization and configuration of Erlang applications. The framework allows developers to define dependencies between applications, manage application startup and shutdown, and handle system-wide configuration.
OTP Design Principles: OTP promotes the use of certain design principles, such as "Let It Crash" and "Fail Fast," which encourage building fault-tolerant systems that can recover from failures quickly and gracefully.I don't think it stands for anything anymore, officially.
that is, facebook messenger was also to a significant extent written in erlang, though i don't have any insight into how much of the internal facebook messenger system was erlang-based (i imagine probably not that much)
also why does the article say
> In 2022, it received a notable update that speeded up the execution of Erlang programs significantly through the introduction of a ‘just in time’ compiler.
and not mention that the hipe jit compiler was added in 02000, 22 years earlier, with x86 support added the next year? there have literally been jit compilers for erlang since last millennium. https://www.researchgate.net/publication/225129672_The_HiPEx...
It’s notable because previous JITs were not very successful at increasing performance:
http://cufp.org/2009/functional-programming-facebook.html
The abstract covers the pros and cons behind using Erlang back then pretty well.
The article does say that that Facebook Messenger used Erlang.
I've amended the comment on the jit compiler to say 'updated jit compiler' - my mistake.
Sounds like some manager had drunk the FOSS/bazaar kool-aid at that point, but wanted to keep the costs down by just taking from FOSS efforts in other languages (as opposing to continue to use more resources to maintain Erlang at the previous degree, and giving to FOSS).
AFAIU, nothing to do with FOSS, which wasn't really on the radar for a somewhat conservative telecom company at the time. It was about custom in-house stuff (PLEX, Erlang, etc.) vs. "industry standard" (C++, OO, UML, etc.).
See also my other comment https://news.ycombinator.com/item?id=36570765
Since WhatsApp handled much more users with just 35 engineers, I would have thought it wouldn't be so much of a deal to recruit some Erlang engineers. They didn't need to recruit thousands of engineers.
What platform have Facebook moved Messenger to?
The story I heard was they were looking for people who already knew Erlang and were having trouble finding them, so they rewrote the thing in a language that they could hire for.
From my recollection, when I was at WhatsApp, we only hired two server engineers who had used Erlang before. I was considered knowledgable about Erlang, because I vaguely remembered seeing the Slashdot post when Ericsson open sourced it. Pretty much we hired smart people and gave them an Erlang book. As a result, I don't think we did OTP applications properly, and we might have missed out on a couple other things, and our code might not be very idiomatic always, but I think we did ok.
But, beyond hiring, Erlang is a weird fit inside Facebook's datacenter infrastructure. The Messenger server team was (and I believe still is) very small, and rewriting into something with less impedence mismatch for a small team to deal with probably made sense. For WhatsApp, we spent quite a lot of effort making our software fit at Facebook, and we had to give up FreeBSD and from what I've heard, hotloading, too.
The Erlang/Elixir OTP has design constraints like any other solution, but has survived 4+ business cycles. One can only hope Julia ends up with something similar, but more user friendly.
Distributed systems is hard, and training obstinate people is like trying to teach a Raccoon TCL. If a design really needs Erlang/Elixir, than there are few alternatives that won't degenerate into a polyglot dumpster-fire.
Happy computing =)
> OTP or the ‘Open Telecoms Platform’ provided capabilities such as ‘hot code reloading’, databases (he ‘Mnesia’ package) and many other facilities.
And yet most erlangvm users use it with phoenix as a better rails, deploying with git and docker, attaching a postgres database.
Consider hot code reloading to be like changing the tyres on a moving car, the pieces and tools are there to make it possible, but in almost every situation it's more sensible to stop and change. Backwards compatibility for all API changes, code correctness and deployment methodology need to be fixed before you can start a hot rollout. That's not even to consider a flexible application architecture that will be compatible with such changes in functionality (OTP does indeed ensure this is compatible on the basic process level, but there is no magic bullet around high level architecture).
For given hard requirements, e.g. upgrading the application without dropping TCP connections - this can be a killer feature that removes considerable development effort, but usually it's a very cool thing that you don't need.
That being said, hot code reloading on the individual module level I have found to be a useful and dirty trick to change the behaviour of running applications (add log statements / telemetry where they weren't before to diagnose production issues). This still has all the caveats and danger of the above, but is more limited in scope.
This was an interesting discussion I had with a few of our B2B customers at a customer event.
Even with redundancy and everything, something like a major postgres upgrade is a tricky thing. You either have to do some amount of work of setting up several clusters, setup logical replication, decide a cutover point (most of them things pg_easy_replicate (if I remember the name right) does), while juggling the backup solution on the side and such. It's overall understandable, but a decent number of steps to do right, and if you don't you might even lose transactions to the upgrade. And losing transactions is way worse than being down.
The alternative though is somewhat brute: After testing everything, shut the app down, shut the cluster down, upgrade binaries, run pg_upgrade. Restart cluster, check cluster, start application. This requires downtime, but as a procedure, it is very straightforward, very testable and overall predictable. Running a readonly version of the application sadly wouldn't be possible because of choices in the application.
Interestingly, many of our larger customers obviously preferred if vendors don't go down, naturally. But for a complex migration or update, most of them have been burnt by horror stories of upgrades going entirely sideways. If an outage is of predictable time, well-planned and communicated in advance, they can prepare for it and deal with it. One just shrugged and said "So what, then we just tell our customers we don't provide that service for half a day. No big deal with a month in advance or so"
This gave me a very valuable perspective of how much energy to put into avoiding downtimes for rare procedures and interactions. Talk with your customers, and sometimes the correct answer is indeed "don't".
Especially if you have state in your service. Shutting down servers with active clients is disruptive, and moving clients immediately is difficult. Simple HTTP request/response is easier, but a lot depends on your load balancing.
Mnesia is the only major piece that I agree is left out, for good reasons (it's fine software but really not built for cloud deployments).