My favorite Erlang program (2013)
joearms.github.io
joearms.github.io
Completely unrelated, but I remember talking to Joe at one of the Erlang conferences. He was always excited about technology and always happy to chat with anyone. He was dismayed how Windows had gotten worse and less usable over the years, and how one day we won't be able to browse our own files on it until we sit and watch advertisements for a while to unlock them. Joe was nearly right! Sure enough, years later I hear there are ads in Windows 11 and you have to go out of your way to remove them. Not quite the same yet, but by Windows 13 I am sure we'll get there.
1. My Windows machine has a lot more of the weird issues I used to see on Linux like 6-7 years ago. Like nothing showstopping. More like quality of life reducing stuff, like the volume in one headphone will be far louder than in the other, despite the mixer showing them the same. Or weird crashes in applications (specifically with Microsoft software). Or Windows documentation pages not resolving in any browser, forcing me to use an archive site to view them. Or my laptop occasionally not detecting my external monitor after detecting it fine for weeks. Or the monitor will flash occasionally
2. Less surprising, more dismaying: Office software by default tries to get you to save files to the cloud instead of locally. Yes, having your files everywhere across all devices is nice, I know, but I would rather have my files local first, with cloud backup, rather than have my files in the cloud first. I'm old fashioned, I guess
3. WSL is pretty jank, but it is probably the only reason Windows is usable at all for me
More topically: I've always heard that about Joe being a wonderful human being, and it makes a lot of sense. When I was first teaching myself to code, I came across his book on Erlang, and it had a huge impact on me. I loved the playfulness, humility, and imagination that he brought to his work. Even though I hardly ended up writing any Erlang, reading his writing changed the way I think about so many things related to code. It's clear he cared a lot about pedagogy. I have a math background, and wish that both math and software engineering and CS culture embraced playfulness and humility more
A while back I installed a Windows update that permanently prevented my existing Bose headphones from being able to connect to my laptop over bluetooth.
They also fired their entire technical documentation team.
They now treat Windows like freeware/shareware, and monetise it the same way -- selling ads, collecting telemetry, bundling, and being pushy.
Windows Server is abandonware, Microsoft only cares about Azure now. As far as they are concerned, hosting software yourself on-premises is a strange aberration and will eventually just go away.
Entire categories of their products are no longer available for on-prem installation, such as: CosmosDB, Data Factory, Purview, Logic Apps, etc...
> They also fired their entire technical documentation team.
Explains why I, when I tried to use the new Clipchamp app, I first had to look for the docs because what should have been obvious wasn't obvious
... and when I find the docs
- they are translated (which is bad because it makes it harder to search in online resources when you have to guess what the concepts are called in English)
- and it is impossible to change language
- and navigation is so badly broken that the only way to navigate reliably between pages is to scroll to the top of the one you are currently on, find the link tree and choose another
- oh, and the translations are hilariously bad: "to share" (as in sharing, which in my language is "å dele" or "deling") was consistently translated as "a share" (as in stocks, which in my language is "aksje"). As someone who knows English reasonably well and will even sometimes translate it is easily enough to understand, but really really jarring for me and probably not understandable for someone who doesn't understand English and needs translations.
I tried to use the built-in video editor in Windows 10, and it told me to start using Clipchamp instead. A few clicks later the new program was installed and running, and before I could do anything it asked for my name. This is normal in many office programs that want to embed author info in metadata, so I entered it without any hesitation. A few seconds later a new email pops up in my inbox - I apparently had signed up to an online video editor service using my Microsoft account without knowledge.
Honest question: Is this a scurrilous rumor or the actual truth?
If you're just being hyperbolic, I'm down with that, no complaint, I'm just curious how literally true this is.
(I have to admit it'd explain a lot of what I see lately; I've observed for well over a decade now that if people held Windows to the same standards they do for "the Year of the Linux Desktop!" claims that Windows doesn't necessarily pass the test either. But it does seem to be getting worse lately, and with things like ads it's downright broken by design. Windows now survives in my house by the skin of its teeth, only because I haven't found a Linux Cricut solution and we use that every few months.)
The book How We Test Software At Microsoft describes in detail what the SDET role was about and how Microsoft used to approach quality.
I think it’s a tragedy that they eliminated the discipline entirely.
[0] https://techcommunity.microsoft.com/t5/exchange-team-blog/wh...
The intrusive ads are not new to Windows 11. They're already there in Windows 10.
universal_server() ->
receive
{become, F} ->
F()
end.
But with Erlang, you can have a distributed network of Erlang servers where the server is a generic computing resource that can do anything the client wants.
The code actually comes from the client. No need to get your system administrators to install some binary on all the machines. You simply pass the function along and the remote machine calls it.
I often wonder what would have happened if the "BEAM renaissance" (driven largely by the birth of Elixir and associated tools) had happened a decade earlier, before Kubernetes became the de-facto standard for ad-hoc distributed computing in web software.
It’s possible people are showing the capability of BEAM thou
Is pushing new code for your server to run "arbitrary code execution"? I guess we can call it that. Is it an exploit?
Depends if the code comes from some random person on the internet from mechanisms that you don't intend for pushing new code to run (e.g. through a buffer overflow on your server or XSS), or if it comes from yourself through your official mechanisms.
[1]: https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31...
Erlang’s architecture is unusual; both the virtual machine and the language are built around the idea of tiny processes operating concurrently, each process running in an infinite loop waiting for incoming messages to interpret.
This allows a process to become whatever code you send it. If you need a process to control a microwave, and then run some quantum computations, and then predict the winner of tomorrow’s football game, you just send it the code it needs for each operation and it happily does so.
f(10).
Pid ! 10.
receive
N -> ... % do something with this value
end.
And in his case he had access to some 9000 computers. If each was running at least one Erlang node and each node was running a universal server, then with a very simple program he could write a function, serialize the function, and distribute the function to his 9k+ running universal servers and turn them into 9k+ specialized servers.
What he is saying is that you can swap out the logic of this waiting loop with whatever protocol logic you want. His example was that he had a fleet of machines that were running this loop, and he sent all of them a function that implemented a gossip protocol. But he could easily send them all another message that turns them all into BitTorrent clients.
Joe was an absolute genius and an extremely kind person. I had the honor of meeting him once. Erlang is still one of the most beautiful technical creations I've ever encountered. It really does make you see concurrency in a whole new way.
Is there a concise way to explain how Erlang achieves this property?
Completely agree. Joe was always willing to reply to my emails and answer my questions in detail. He was an exceptionally talented explainer, and his messages were always interesting and entertaining, while also being informative.
I never met him in person but I will always treasure the correspondence I had with him.
“Make it work, then make it beautiful, then if you really, really have to, make it fast. 90 percent of the time, if you make it beautiful, it will already be fast. So really, just make it beautiful!”
When you open a file, a socket, allocate memory, borrow a database connection from a pool, etc. you don't need to write try/catch/finally, or defer() statements, or logging, or any kind of error handling like that - if your code crashes, the VM will take care of the cleanup (because it knows what resources are owned by your process) and the supervisor above your process will log the problem and restart the process if necessary.
This makes Erlang applications much safer by default, and the business logic is clearer and simpler because it's not mixed with error recovery code (the "if err != nil" every other line that Go is famous for). The error kernel (the part of the app that must be carefully written to ensure reliability) can be kept really small[1].
This comes at a cost in performance, because data must generally be copied between processes, whereas goroutines can send pointers directly to each other. But if you can afford it, it's _very_ nice. Besides the error handling this also enables crazy observability (you can connect to a running app and inspect processes[2], kill them, send them messages, jump between nodes in a cluster, etc.), live code reload, and a bunch of nice things like that.
[Process isolation also enables clustering, where processes running on different machines can talk to each other as if they were in the same OS process. The copying means it doesn't matter if the other process is remote or local.]
[1] https://medium.com/@jlouis666/error-kernels-9ad991200abd
It's not an accident that large distributed chat programs or MQ systems are often written in erlang. From inside the code, writing a distributed system feels the same as working on one node.
With go, you can't do channels to a go process running on a different server without significantly changing the language. In erlang, you don't notice because that's just how it works.
Write an app that runs on two nodes and sends updates to all clients via websockets. In erlang, you just write it. In EVERY other language, you're loading a non-idiomatic library and probably running an MQ cluster or using an MQ service... probably written in erlang.
Here they made a new server that takes an return process (From) and a number (N). The exclamation point sends the result back to the return process.
factorial_server() ->
receive
{From, N} ->
From ! factorial(N),
factorial_server()
end.
factorial(0) -> 1;
factorial(N) -> N * factorial(N-1).
test() ->
Pid = spawn(fun universal_server/0),
Pid ! {become, fun factorial_server/0},
Pid ! {self(), 50},
receive
X -> X
end.
[0] https://gist.github.com/mndvns/80b00cf67d418e8359fb5566b80ae...
As exotic as this sounds, this is very similar to what web-browsers do with script src tags, especially from 3rd parties. The page is saying "Hey let me eval a function that can do whatever it wants in this context. I trust you!" Most webdevs don't consider this a threat vector!
From now on I'll just link them to to Joe's favorite Erlang program and this HN thread.
Covers/Explains all the steps in a High-Availability, Fault-Tolerant, Distributed Erlang-based system architecture beautifully.
Thank you very much for the pointer.
universal_server() ->
receive
{become, F} ->
F()
end.
(Perhaps that would need to have the code for F already on those remote vms, whereas this eg. also serializes and forwards F's code?)
I would understand the utility if this also handled some common boilerplate, but it seems to just wait for become F message and then doing F().
The way it works is that F is a closure. It contains some code but could have captured variables as well as it’s environment. Calling F() then executes the code and it will have access to the whole surrounding environment when F was created.
In Erlang we can send a closure across the network to another Erlang node. That can be used and is cool but you still need the module code referenced in the closure available on all nodes.
Long story short, in production code you’re right, we would just call the F module directly and pass its the arguments explicitly. This is more of a cool demonstration of possibilities.
For modules, BEAM can have two versions loaded: the current version, and an old version. When you make a fully qualified function call module:function(...), it will call the current version, but calls within the module stay within the same version.
So if you load a module's new beam file, all processes that make fully qualified calls into that module get the new version. In case you're wondering, if you load a third version, any processes that still running in the first version get killed (you can check before you load a third version if you don't want that).
It's idiomatic to write receive loops as
loop(State) ->
NewState = receive ...
end,
?MODULE:loop(NewState).
yes.
but you can also send the code to the remote VM
for example in Elixir you can `Code.compile_string` which returns the name of the module and the code as binary (beam bytecode)
You can then send that bytecode to another VM and load it in the remote VM using
:rpc.call('node@remote_host', :code, :load_binary, [the_module_name, "filename_doesnt_really_matter", binary_code])
spawn('node@remote_host', my_module, my_function, [args])https://gioorgi.com/2015/erlang-lesson1/
I transcribed and explained a bit more: it shows the power of an async language like Erlang/Elixir compared to other ones. Sadly, it is little used nowadays
RIP Joe.
I absolutely love Erlang, and I wish I had more of an opportunity to use it, but it's nice to see that some of its concepts are bleeding into other platforms.
But you already had an empty network that in a few seconds would become anything you wanted it to, that's how you installed empty universal Erlang servers on all the computers.
Of course security guys are going "Wait, every machine has an open RPC channel that will blindly execute code to every other machine???"
It's a bit concerning if it's only "security guys" seeing issues with this approach.
If that's an issue, and I can see why it might be for some, you'll need to use something other than Erlang distribution to connect your cluster.
You can patch out the explicit rpc server, but I'm not sure that you can patch out receiving functions (maybe removing the code that deserializes them from external term format would work?), and you'd need to audit the whole thing to ensure nothing ever calls a function it receives.
I think my concern would be more in the realm of, you're inherently relying on whatever peer authentication system BEAM uses (and it being bug/exploit-free), to determine whether an RPC call can suddenly inject new code into a running system.
In my experience good security is a lot like an onion - there's lots of layers and generally some crying.
If we consider a "non-BEAM" system that runs on say JVM, or even a "scripting" runtime (e.g. PHP, Python, Ruby, NodeJS) - you typically have a number of elements that (can) contribute to making the code that runs, relatively immutable outside of a deployment event (e.g., restricting access to ssh/sftp/etc protocols; varying types of filesystem permissions; readonly volumes; container filesystems; etc)
Do other systems have similar "run arbitrary code" vulnerabilities? Sure. Allowing uploaded content to be executed by the PHP runtime is a classic example. But there's almost no legitimate purpose for this, it's almost certainly a configuration error/mistake, and it's relatively trivial to prevent it completely.
You don't need security people for basic sysadmin know how.
If you have someone malicious in your internal network that can connect to your Erlang nodes, you have bigger issues that them connecting to your Erlang nodes.
I don't see the point of your original comment. Having an out-of-the-box RPC mechanism means you gotta secure it as you secure any other internal service. That's sysadmin 101.
Most RPC systems can't inject code into the running system dynamically. That's the point of my original post.
Granted they are probably both great at highly parallel and distributed computing applications, I am just wondering if there is an interesting differentiation between these two in this aspect that I should be aware of.
Goroutines are a pure concurrency primitive, they don't contain errors. A goroutine crashing will take down the entire program.
Elixir would be absolutely beautiful for a trading program - you could run each strategy (trading algorithm instance) in a separate process, so that a bug in one of them can't take down the whole system, and use monitors to guarantee orders are cancelled if a strategy crashes. I actually started learning it because I worked on such a system, and it was hard to sleep at night when any small coding error in the giant C++ codebase could send the whole thing crashing.
It would definitely be too slow for HFT though :)
Elixir has GenServer and BEAM niceties which is something you'll have to replicate, likely poorly, in another environment.
As for learnability, meh. Have a senior on staff and new employees will be able to learn either.
Elixir has Phoenix, liveview, and Ecto. You'll just have to look into them, but they are good. The game changer is iex, you can terminal into any running process in the cluster and interact with it.
For me, I'm much more experienced in Go and would leverage managed kubernetes if I needed to cluster. I think it scales better for larger orgs. If it is a core group of one small team, Elixir is pretty rad.
I've done a bit of Go in the past and it definitely has its place, too. It feels more mainstream. We converted a Python app to Go a few jobs back and the performance increase was incredible.
On scalability, given that high-frequency financial applications are generally super performance sensitive, or at least have a reputation of such, I would probably consider the BEAM VM disqualified (not the slowest language but definitely not the fastest) and even still have some serious questions around Go. I like to characterize it as the one of the slowest members of the fastest class of languages; yes, it's compiled, and that means it generally beats interpreted and VM-based languages, but it's definitely on the slower end of compiled languages due to its prioritization of compile speed over optimization time.
In the case where I ported a system straight out of Erlang into Go, without a rearchitecture, pretty much a straight port, the Go system was 5-10x faster than the Erlang system, and I'd expect that to hold. BEAM made some decisions back in the 90s that give it some significant performance headaches nowadays... not fatal for all uses by any means, it's completely possible to build a reasonable system on it, but if raw computational performance is high on your list of needs it's got some nontrivial disadvantages.
That's what Rustler, Zigler, and Numerical Elixir (Nx) solve. It doesn't mitigate issues around deep copies when passing messages between processes, but the absence of shared memory means that race conditions aren't a problem to worry about, and the memory model is such that garbage collection is never going to be a performance issue even if you scale up to millions of processes per node. It also means you can trivially scale horizontally across multiple nodes because your code doesn't (more accurately can't) depend on sharing memory.
Also, I think the BEAM GC is widely misunderstood. Per process it's kinda neat that it can GC without having to consider the others. But globally it's actually kinda slow. Like I said, I've straight-up ported Erlang programs into Go, network clustering and all, and Go was quite significantly faster, even with the not-so-great GC Go had in the ~1.6 era. (I suspect if I ran the same code now it would just work with no GC problems.) BEAM GC is more "hey, look at this neat trick!" than "better than anything around". I actually wouldn't consider it very good at all in modern terms. The Erlang community has a huge problem with thinking it's still 2003 and comparing Erlang to all the other competition in 2003, but unless you're still living in 2003, you need to be comparing to what's around in 2023.
I’m not sure if that’s measurably true in terms of ergonomics (e.g., there’s no cognitive load difference between using a borrow checker, types, programming defensively, thinking about async at all). But I have no actual experience with go to offer a compelling counterpoint
> Like I said, I've straight-up ported Erlang programs into Go, network clustering and all, and Go was quite significantly faster
Was the JIT any help?
In your view, what technical decisions do you think prevent performance in Erlang VM? Why do you think good performance either impossible, difficult to achieve, or unlikely due to these reasons?
The differentiation used to be HUGE. In the 200xs, especially earlier in that era, static languages were a NIGHTMARE. I personally believe this is a large part of the reason that the dynamic scripting languages won as much of the market as they did, despite their manifold disadvantages. They were going up against language environments that were even worse, by a large margin. 1990s C++ was just a nightmare.
Today the difference is much more muted. I've had the pleasure of being assigned a greenfield research project by myself for the last 6 weeks, and I'm not only making plenty of progress in such a task in a statically typed language (Go), I am now well into the point where I am actually better off in a static language, because I can make very significant infrastructure changes safely, without having to bodge adaptors everywhere.
But it's still an opinion thing, sure.
"performance issues"
The super weak typing story is a big problem. We've seen a whole bunch of attempts over the years to take a weakly typed language and speed it up with JITs. In a nutshell, the answer is, you can do better, but we didn't get anywhere near C speed performance in general like there was so much hope in the early 2000s... and the programming community loves imbibing old propaganda and still has not really grappled with the fact that JITs do not generally "work" in that sense.
Linked lists as a primary data structure for so many things is not good in 2023. While you avoid some of their issues in that the other end of the linked list is much more likely to be in cache because of the smallness of the local OS process, they're still linked lists.
The BEAM VM does a lot of bookkeeping. When this bookkeeping is useful, it is invaluable, but when you have a system just chugging along, working, and not needing any intervention, it's doing a lot of unnecessary bookkeeping that most other languages aren't. Note I'm not saying this is all bad; I'm just saying, it isn't free.
Pattern matching is probably generally slower than polymorphic dispatch, let alone static type-based dispatch. Especially when you start writing deep patterns. I'm a little heterodox on this question in general, probably. I'm actually not entirely convinced that pattern matching is somehow generally "superior" than the alternatives. There are several negatives I'd associate with it that advocates often overlook. It's not disastrously broken or anything, but I find its cost/benefit over alternatives to much more complicated than often supposed.
Immutability slows down a language. Compilers can try to optimize around it in the common case, but while it may have software engineering disadvantages it is certainly generally faster. While it's easy to show immutability has a maximum "log n" slowdown factor over mutable code, it's still easy to accidentally do worse than that, and a "log n" factor is still something you'll feel.
It's a lot of little things. And like I said, the totality still adds up to a usable language. It's still generally faster than Python, for what that's worth. But it's also not a fast language. The BEAM VM itself performs well but the language it implements is slow, and I think in the intervening years there's been a lot of languages taking a lot of different approaches that achieves the same goals without the same slowdowns.
https://www.youtube.com/watch?v=pO4_Wlq8JeI
If you need to break out into SIMD, you also have NX at your disposal. And Rustler/Zigler provide you an escape hatch to low-level languages, with the latter you can code in Zig inline
https://github.com/elixir-nx https://docs.rs/rustler/latest/rustler/ https://hexdocs.pm/zigler/Zig.html
For performance, drop down to C/C++ (carefully);
1) https://stackoverflow.com/questions/1811516/integrating-erla...
Genuine question. It's a fantastic attribute of Erlang to be able to do this.
I think you can assume that they are sending something like bytecode, but might optimize that, if both are eg on the same physical machine.
However if this code refers to any functions from modules via syntax "foo:bar/1" then these references will be resolved on the target node, and both nodes better have the same versions of modules loaded.
My favorite Erlang Program (2013) - https://news.ycombinator.com/item?id=31639382 - June 2022 (19 comments)
My favorite Erlang program (2013) - https://news.ycombinator.com/item?id=22413029 - Feb 2020 (54 comments)
My favorite Erlang Program (2013) - https://news.ycombinator.com/item?id=12396420 - Aug 2016 (38 comments)
My favorite Erlang program (2013) - https://news.ycombinator.com/item?id=8807660 - Dec 2014 (2 comments)
I wish such a canonical example was easy to find for every programming language, particularly those on the boundary between mainstream and exotic languages.
Most languages present themselves by waxing philosophical about HoTT, zero-cost abstractions, or parametric types. Show me an example where your language is clearly better than Python, Ruby, TypeScript, C#, or Rust, because those are the languages to beat, and they already have the entire infrastructure set up so unless you can demonstrably outperform them in some way, it's probably not worth my time to take a deeper look.