Inside Erlang – Joe Armstrong tells his story (2014) | Hacker News Reader

Inside Erlang – Joe Armstrong tells his story (2014)

wcm.ericsson.net

209 pointslelf26 comments

Joe mentions he used to be a physicist and how Erlang contributed to the discovery of the Higgs Boson.

I heard him tell this joke more than once (we were colleagues): "Higgs boson walks into a church, and the priest says, 'I'm sorry we don't allow Higgs bosons to come to churches.'And [the Higgs] says, 'But without me, you can't have mass.'"

I tweeted this about Joe:

> I wish I'd gotten to meet @joeerl. So bright and enthusiastic. Erlang embodies so many insights that seem obvious in hindsight. Eg, you can't prevent all programmer mistakes. You can't ensure computers never die or get disconnected. All you can do is make a system that copes.

> Many programming languages and tools exist to deal with the basic fact that humans make mistakes. Type checking, encapsulation, TDD, etc - all because without help, we will do dumb things. Erlang taps us on the shoulder and says "and ALSO your computer could melt. Start there."

> In Joe's words, "To guard against the failure of an entire computer, we need two computers." Obvious. Brilliant.

> The only place you can handle failure of a process on Machine A is in a process on Machine B, which can't share any memory with the first process and can only pass messages to/from it. And for consistency, Erlang handles all failures that way.

> Then you notice that these shared-nothing processes are easy to run concurrently because neither can mutate the other's state and cause race conditions. And that this scales beautifully when you have multiple CPUs.

> These are some of the insights I got from reading Joe's thesis. I'm sure I missed a lot of others and hope to re-read sometime. I wrote up some of what I learned at https://dockyard.com/blog/2018/07/18/all-for-reliability-ref...

The thesis itself is at http://erlang.org/download/armstrong_thesis_2003.pdf

I'd never heard of the man before his passing but I don't recall another person being remembered here with as much affection.

I had heard of him but because no one uses Erlang where I live it was kind of easy to ignore. I wish I hadn’t though.

Watching some of the talks he’s given on the problems of software in the wake of his death has been a real eye opener. The physics approach to building a language which tries to break the laws of physics as little as possible is truly brilliant.

I don’t know how much time and effort we could have saved ourselves from wasting if we’d gone with Erlang’s take on concurrency and messaging, but it’s a lot. Just today I scheduled a meeting where we need to talk about how to handle messages not making it all the way through a few REST-based sevices.

I doubt we’ll start using Erlang though, it’s just too risky when you can’t hire anyone, but you do feel a little stupid fighting problems Joe solved before your youngest hires were born.

There's always Elixir where you can get the best of both. There's a lot of developers out there.

Not in my country.

Which country?

> I doubt we’ll start using Erlang though, it’s just too risky when you can’t hire anyone

Can you train them?

> I don’t know how much time and effort we could have saved ourselves from wasting if we’d gone with Erlang’s take on concurrency and messaging, but it’s a lot. [...]

> I doubt we’ll start using Erlang though, it’s just too risky when you can’t hire anyone [...]

It's a risk to get started, but if you're saving time and effort, that time can be used to train. I work at an Erlang shop, and none of our early employees knew Erlang before we were hired -- we took an existing Erlang application and adjusted it to suit our needs better; over time we've rewritten the whole thing (while it was running). If you're trying to hire experienced Erlang developers, it's hard because there's not that many.

Instead, hire smart developers that are willing to try something different, and are good at interfacing with other systems, and willing to look under the hood from time to time.

If you watch some of his videos and read his work, you just get this feeling that he's this happy, hard-working soul that has created a masterpiece and loves it with all his heart. His positivity is infectious.

His book on Erlang was one of my first programming books I ever read. I didn't know of his passing until I saw this. :(

What sometimes puzzels me is that ERLANG is described as quirky and difficult to read.

I find that once you learned a bit about it for just one day or so it is a very simple to write language that just flows on the screen from your thoughts.

So it might look quirky, but really not for long and then it's quite fun to use. And that's before looking at the ease how you use concurrency, process restart, hot swap..

I think the application of the language is quirkier than the language. There is a fairly high barrier to doing something ideal. In languages like python or c you can pretty much just write a few lines and you got something representative of what you can do. In this thread for example people are arguing whether you should make command line programs even in elixir: https://elixirforum.com/t/elixir-static-app-binaries/15335/4...

Let them argue.

https://erlang.org/doc/man/escript.html

> There is a fairly high barrier to doing something ideal.

Can you elaborate on what you mean here? I don't quite understand your message.

I mean that if you want to do something with e.g. python it is pretty easy to access what is good about python. You can make a script, make it "pythonic", use some modules and package it. And that sort represents a good, useful, python program.

With Erlang it is generally a lot harder to access what it is good at. Not only do you have to learn the language, which is a bit different, but things like how to configure and distribute the virtual machine, how the concepts of otp works and maybe even things like dialyzer.

I realize that I don't have to learn all that, but still the distance from writing a simple program to what is the unique features is still a lot longer. At least that is my impressions.

So my follow up question then:

How would you do distributed programming in Python? Could you do it quickly and easily in a Pythonic way without needing to learn those same kinds of things as Erlang?

I feel that you may be close to an apples-and-oranges comparison there. Erlang is several things: language, distributed system, and almost-OS.

If you want to do distributed systems in any language, it'll be a steep learning curve. I cut my teeth on that in college with OpenMPI (paired with OpenMP for local parallelism) in C and Fortran. It wasn't too hard, but someoene else had set up the server to actually distribute the MPI jobs across all the nodes, I had no idea how to do that setup. With Erlang, if someone else has already set up the nodes for you it's not really any harder than MPI to handle distribution of work across those nodes. Once you start trying to configure and administer the distributed system, yes it'll be somewhat complex. Same thing with the various Java application servers I used a decade or so ago.

The language itself, however, permits programming in an Erlangic way without needing to involve OTP or multiple nodes or dialyzer. Processes and message passing are very accessible and easy to use, permitting a way of structuring programs that's very Erlangic. OTP permits a more consistent and uniform way that's in line with what other systems do and might expect (if you were to pass your modules off to them to be put into a larger system), but it's not needed for writing an Erlangic program. Pattern matching and recursion are other quintessential features of Erlang and encourage a particular program design, also accessible without reaching for those other tools.

It is an apples and oranges comparison because they are. Python exposes it's feature fairly readily in a relatively similar context. If I can open "Erlang Studio", plot my nodes, write my recursive functions and run them to some effect on some "Erlang Application Server" then Erlang would be as well. But as far as I know I can't. And I think that makes the application of Erlang somewhat inaccessible.

One could of course "just" use Erlang the language, but that use case has a lot more competition.

I started moving my team to elixir mix tasks and it really feels like a sensible shell script. 1) our scripts must live in our repo, so changes are tracked with the evolution of our software, 2) many of these scripts are deployment and resource management, so they share elixir code anyways. 3) debugging and testing are easier than shell, or even python. 4) one of the things I'm doing is PXE booting, and we have functional DHCP, tftp, and http services that are very easily instrumentable can run the lifetime of the script instead of turning then on and off using some fragile config file twiddling and sudo systemctl dhcpcd incantation or whatever.

Do you mind elaborating a bit on your use case for PXE booting?

sorry, can't give too many details on the specific, but the PXE booting spec requires that you start up a:

- DHCP server that responds to broadcasts on 0.0.0.0

- TFTP server that responds to requests for the bootstrap image

- HTTP server (for most linux distros, anyways) that fetches more boot resources that are big enough that TFTP is probably not the best protocol (UDP doesn't have packet transmission guarantees).

I use a dead-simple, transient erlang HTTP server that my buddy wrote: https://github.com/StoneCypher/htstub

And hand-rolled custom DHCP and TFTP servers (existing options are not exactly what I need) that basically operate like htstub.

This is all tied into a mix task, so you start the mix task, then walk over to your target machine (or IPMI), reboot it, and then it's provisioned from scratch, reproducibly, with two-touches. Since everything is done in a single language, if something is wrong in your provisioning scheme, it's really easy to figure out what went wrong and deploy a fix.

That does seem interesting. I could definitely see that as something I would want to use for the functionality rather than the "underlying feature". The story around this is pretty weak in most language ecosystems from what I have seen.

Yeah, this phenomenon definitely exists. Probably just the unfamiliar syntax, — somehow enough to make people see Erlang as “complicated”. While in reality Erlang is one of the most simple and consistent languages out there.

While functional programming is imo easier to understand in general, you can really get screwed with spaghetti when passing modular lambdas around. Take a look at the OTP tftp server code. It's a nightmare.

Such incredibly sad news that he has passed away, I met Robert Virding a few years ago (small group of us went out for a meal). I had hardly heard of Erlang before, followed much more since, plus many of their talks. It's fascinating how physicists design a programming language/ system, especially the treading/ distribution model. I think it was WhatsApp the catapulted Erlang to the limelight (outside of telecoms), it's doubtful that they would have scaled so quickly had WhatsApp been written in anything else.

Rest in Peace Joe, and my condolences to his friends and family.

Reply on news.ycombinator.com