Hacker News Reader

Comment by hopia | Hacker News Reader

A new Erlang VM with just replicated functionality is a fairly hard sell to the Erlang/Elixir community, who brag with the industrial track record of the BEAM.

I believe you'd get much more interest if there was some ambitious new promise for this new VM, such as 10x sequential performance etc.

View on HN

Someday this is going to need to happen though. IMO, "the right way" to do this is via the strangler pattern:

https://www.michielrook.nl/2016/11/strangler-pattern-practic...

Probably the language that is most poised to achieve this is Zig; it would be feasible to start by wrapping the entire BEAM in a zig compilation unit; which at the very least potentially offers an easier path to maintaining the codebase across multiple platforms. Followed by hodgepodge doing bits and pieces in zig, which could be achieved via straightforward transliteration at first.

The very different mindset of the rust PL lends itself to total rewrites, which I don't think will sit well in the BEAM community. On the other hand erlang has tons of strange rewrites happening over its own internal ecosystem all the time (gen_fsm -> gen_statem, pg -> pg2 -> pg), etc.

Do you know of any examples of this being done with Zig? I can think of a couple with Rust:

- https://gitlab.gnome.org/GNOME/librsvg completed a migration to Rust.

- https://github.com/RazrFalcon/rustybuzz and https://github.com/immunant/rexpat are making decent progress.

No, because zig is still in 0.6.0, and the BDFL says "don't use this in prod yet"? Yeesh.

Perhaps then, for the sake of setting those expectations, the wording "the language that may be most poised to achieve this in the future is Zig" would be preferred?

Last I checked "poised to achieve this" implies the future. Moreover I have specific reasons to say that, namely that zig now ships with C compilation with transparent cross-compilation support so you can use it as a direct replacement for C and (I believe make), so the statement is based on current zig features whose importance I alluded to in the comment.

Fwiw this is also a real problem as I am often hearing complaints about windows ABI support in elixir, anyways. Moreover, having written nifs, I have to say the c header for nifs is basically unintelligible without manually parsing the DEFINES because of windows cross-support needs.

Please let me know if I'm using "poised" incorrectly.

To my mind, "poised" in this context is roughly synonymous with "ready". If the official position of the project is "don't use it yet" then it's not ready yet.

The future implied is the completion of the action one is ready for, not further work of preparation for that action.

Your take may, of course, differ.

Yeah I definitely didn't see it being production ready any time soon, but I thought it was an interesting project for people that wanted to learn BEAM internals. That's how I got started with it at least, I had problems trying to contribute BEAM just because of the sheer size of the codebase and lacking the domain specific knowledge.

I do think that having alternative implementations is good for experimentation though, similar to how Ruby was improved upon ideas from JRuby and Rubinius, even if most users never used those two directly.

> That's how I got started with it at least, I had problems trying to contribute BEAM just because of the sheer size of the codebase and lacking the domain specific knowledge.

And the fact that a LOT of BEAM was terribly undocumented. I'm impressed you got anywhere, personally. The last time I looked at BEAM to understand some weird behavior (that eventually did turn out to be a problem in Erlang/BEAM), it was completely impenetrable.

Your implementation is probably useful as documentation, if nothing else.

100%, when I started building Lumen, I spent an enormous amount of time working out how various parts of the BEAM runtime were implemented, and it was (and still is sometimes) a grind. That macro-heavy C code is just such a bear to read. I understand why it was written that way, but working with it is just unpleasant.

Having Enigma, and other implementations like it, provides a huge value in terms of understanding how it all fits together - understanding the Enigma implementation and then going and trying to make sense of the BEAM would probably be a way better path than trying to dive into the BEAM straight away.

We do a lot of Erlang work here. The BEAM is so reliable that I wouldn't spend 1 minute looking at an experimental alternative.

Yeah, this. I'm just getting started with Erlang and I already feel like an idiot for not using it sooner. When I think of some of the things I've done to try to achieve what the BEAM does out-of-the-box...

Can you give some examples? I've been getting more and more interested in Erlang.

RPC is basically built in, so you'll probably never use REST internally. There's an in memory database built in (ETS) that will replace redis for most key value storage cases. There's easy recovery from crashes via supervision trees and associated features. You can do hot upgrades while your system is fully operational.

A few protips: hot reloads are outside of a few corner cases generally unnecessary these days. Be careful about erlang rpc if you expose it in any fashion over the network and follow the eef security working group guidelines re: erlang term format. I use Elixir's plug.crypto.non_executable_binary_to_term/2

Thanks, I knew both of those bits of info, but I didn't know about the guidelines and how to deal with the RPC issue, very helpful.

to be fair, the guidelines are hot off the presses. I literally went to CodeBEAM, ran into to Bram Verburg (whose fantastic X509 library i had just found a week prior) and he told me about those guidelines, which had just hit hours before we spoke. Bram is great, he's very modest, and was amused that I thought he was some sort of superstar.

Well I'm "hot off the press" when it comes to elixir, so it's a good time for me to learn this stuff, thanks for the link below, too.

You wouldn't happen to have a link handy would you? It would be much appreciated.

Erlef security group guidelines: https://erlef.github.io/security-wg/secure_coding_and_deploy...

X509 library: https://hex.pm/packages/x509

Bram verburg talk: https://youtu.be/0jzcPnsE4nQ

Bless you! "You're a scholar and a gentleman." :-D

Actually I'm a bit of an asshole if you really get to know me, but I'm usually nice on the internet. Thank you!

When I used Erlang in 2005, there was a problem with message queues growing and OOM-ing the process. The standard solution to that is RPC flow control and backpressure. Does this problem exist with modern Erlang?

Google's gRPC/Stubby has the same problem. It's one reason why gRPC servers are never deployed on the public Internet and always use hefty reverse proxies.

I always feel like Erlang and more generally BEAM is the language for a lot of today’s problems but will never see any kind of traction because of “worse is better”.

Can you share something on this reliability? Is it more reliable than the JVM? Or is it more predictable in terms of performance? I’ve found the jvm itself to be rock solid as well.

Erlang VM is consistent/predictable in terms of latency, it's engineered that way. For pure computational performance you won't find it optimal.

Erlang/OTP and BEAM have been around a long time and is probably just as mature as the JVM and Java, but they have slightly different goals: very low tail latency, easy error recovery, and easy distributed compute in erlang vs. better customization and median performance for the JVM. I'm sure there's more differences and more nuance than that, but I'd be very happy to use either language/VM in production.

It's also harder to write abjectly wrong code in the BEAM. I found out that my code was effectively self-forkbombing, because I had hooked up an error reporter (I adapted myself because the version I wanted had bitrotted) to rollbar which (very rarely, when the service was in outage) cause a crash, which would try to report, crash, and exponentially grow. Amazingly, this caused no service degradation for our customers.

There is some go projects like [1] that can connect to Erlang nodes and claim to be speedier.

[1] https://github.com/halturin/ergo

If the VM is in Rust could it be compiled to WASM?

You wouldn't want to, for various reasons. See this blog post about Lumen and the decision decisions:

https://tylerscript.dev/bringing-the-beam-to-webassembly-wit...

You wouldn't want to _right now_. However for almost all points there is a solution underway/in planning. In a year or two it might be feasible.

There would however be other limitations, like filesystem APIs etc. not being available in the browser that a lot of frameworks in BEAM languages expect, that would severely limit the usefulness, though I guess that applies to either implementation strategy.

I can't imagine a situation in which it would be acceptable to download an entire (WASM compiled) copy of the BEAM from every new website I visit.

Full AOT compilation with dead code elimination seems like a much better fit for minimising that bundle size.

WASM isn't just for the web, though. It's beginning to look a lot like the next "portable ISA" (successor to Pascal's pcode, applet-era JVM bytecode, Google's PNaCl, etc.)

Add a progressive web-app to your mobile's home screen—get a cached WASM bundle, for near-native performance of the resulting "app." Download an Electron app—get a WASM bundle wrapped in a projector.

Even then, there are still browser use-cases where "full fat" apps are no problem. The "apps" in the Chrome Web Store—Adobe has a version of Photoshop in there, I think—make a lot of sense to be WASM. You'll launch them once and keep them open for hours.

I think the point though is that architecturally there are performance hits if you don't respect the fact that WASM has a different architecture than what the BEAM expects (harvard vs von neumann IIRC), so you may NEVER want to if you get it right in the first place.

The Harvard-Von Neumann dichotomy is wrong. C also works perfectly fine on Harvard architectures--it's why function pointers in C are special, aren't guaranteed to be convertible to a void pointer, and why uintptr_t is optional. POSIX adds these additional guarantees to support dlsym, which returns function addresses as void pointers.

The problems with compiling other languages to Web Assembly are primarily 1) lack of goto and 2) inability to instantiate and jump between different stacks. These limitations are especially problematic for languages like Erlang/BEAM and Go because Web Assembly-based VM implementations require an extra level of indirection in order to implement some of their core language semantics, resulting in quite slow performance compared to even a pure, strictly compliant C implementation (and presuming the WASM VM itself adds no overheard, which is not actually the case).

WASM excluded goto support because it was argued that the relooper algorithm required to translate goto constructs to structured WASM statements was sufficiently capable to cover the vast majority of existing code. And they provided evidence to back up that claim. The flaw in that reasoning is that language implementations and similar niche cases have special needs that application code rarely requires, and in that space constructs like goto are crucial to both simplicity of implementation and performance; the inadequacies of relooper become the norm rather than the exception.

thanks for the clarification!

Just being able to amend job requirements with "or rust experience" is most probably worth it.

Reply on news.ycombinator.com