Firefly – A new compiler and runtime for BEAM languages
github.com
github.com
A lot has changed since then.
The name change came about because we found too many projects named "Lumen" also were launched around the same time or shortly thereafter.
Originally I envisioned a path towards creating a new WASM framework to build web applications using Elixir. Since then LiveView has come out and has made this less of a concern for us. While WASM compilation will be possible and we hope to backfill the runtime needs for web development the real focus and interest in this project for me is...
WASI compilation. With Firefly we will be able build self-contained binaries that will have all (many) of the benefits building applications with the BEAM and distributed systems. If WASI can run on something you should be able to compile your Elixir applications with Firefly and target your desired chipset.
We recently reached a milestone that we are hoping to release a blog post about it in the near future.
Have you considered Cranelift as an alternative backend like rustc is doing?
I’ve got bad news…
https://news.adobe.com/news/news-details/2023/Adobe-Unveils-...
It supports many programming languages.
This new compiler/runtime targets WebAssembly.
However, there is also GRiSP: https://www.grisp.org/
I imagine we would be able to write elixir on the frontend with this.
I feel like the dockyard folks just got rust crazy and decided to do this -- for reasons[1], rust is honestly not the best language to do a BEAM implementation in, which is I think why they pivoted to a compiler
[0] except for the fact that a lot of libraries unnecessarily spawn processes.
... use LiveView :)
Well yeah, that's kinda the whole point of Erlang in the first place :)
> The primary motivator for Firefly's development was the ability to compile Elixir applications that could target WebAssembly, enabling use of Elixir as a language for frontend development
> The primary motivator for Firefly's development was the ability to compile Elixir applications that could target WebAssembly, enabling use of Elixir as a language for frontend development. It is also possible to use Firefly to target other platforms as well, by producing self-contained executables on platforms such as x86.
I'd be very very interested to hear follow up, on how Firefly does actors. I feel like there's so many potential ways to target wasm, but the high concurrency spirit of Beam has such unique flavor. I'd love to read in & hear that spirit is well preserved.
Generally it should be assumed that actors and their concurrency model is fully supported as that is a part of the core semantics for BEAM languages.
It's a part of the semantics of the runtime:
- an actor is all but guaranteed to not bring down the runtime
- an actor is all but guaranteed to never affect other actors
- runtime knows how put processes to sleep until the message they listen to arrives. This means all functions are re-entrant. Well, any process is put to sleep after a cretain number of reductions so that no process takes away time from other processes.
- runtime all but guarantees that process errors are a) isolated and b) propagated. That is when a process dies all other processes that monitor it are guaranteed to receive a notification. That's why supervision hierarchies in Erlang are possible
This is implied. It would be absurd to have a different static and runtime semantics. in fact, the core goal for formal methods is a statically be able to reason about runtime dynamics.
Just like it would be absurd to build a compiler for C where "+" in fact is treated as "-", it would be absurd to build a compiler and runtime system for, eg., Elixir that is not able to execute GenServers.
You'd be surprised how many people miss that runtime drives this. I've seen many discussions where people claimed "you can implement all this is a library" :)
> it would be absurd to build a compiler and runtime system for, eg., Elixir that is not able to execute GenServers.
Well, Akka did it on top of JVM: https://doc.akka.io/docs/akka/current/typed/fault-tolerance.... Can't say about its limitations though.
I undrstand, everything concurrency is definitely not easy to implement in "User land" and is something you want good primitives for – why I am also amaxed over this project as they must have embedded that functionality (the scheduler) in the executable (Which they also say they did).
> Well, Akka did it on top of JVM ...
> Akka is a toolkit for building highly concurrent, distributed, and resilient message-driven applications for Java and Scala.
Akka does not seem to claim that they build a new runtime for BEAM languages?
This paragraph seems confused. Re-entrancy doesn't have a lot to do with sleeping. Do you perhaps mean to say something about preemption? BEAM/ERTS is not really preemptive, a process can only be suspended at specific places, but one of those places is function calls (aka reductions) and BEAM languages don't offer looping constructs other than recursion, so it's hard to go for very long without calling a function, so it's effectively/semanticaly preemptive, unless there's naughty things in NIFs you brought, or the native code provided by ERTS.
Processes being descheduled after a while doesn't mean they didn't take away time from other processes: if you have one cpu and one process running an infinite loop, adding a second one takes cpu time away from the first, but they both will get some time (if they're both set to the same priority)
An AOT compiler with better optimizations will run circles around BEAM on benchmarks.
EDIT: Apparently not LLVM JIT but that's beside the point.
Edit: It actually uses part of AsmJit, not quite from scratch, my mistake.
In BEAM Asm, the design is for the whole VM to either be interpretted (status quo) or native (JIT). In JIT mode, all the loaded modules are translated to native code as they're loaded; this needs to be fast or startup times are delayed, IIRC, there is an optimization path, but it's simple. There's no reoptimization of hot code paths later either; just the one time process.
The main benefit of this process is to remove a specific part of interpretation overhead, the instruction unpacking and dispatch overhead is eliminated. This can be significant for some applications and not for others, but it's really the main target, any other optimizations that happen are a bonus.
Can you provide any citations for the BEAM runtime being unoptimised? In my experience it has been very carefully optimised over many years, generally prioritising latency over throughput.
I don't see careful optimizations, it's rather sloppy. More like perl and python, unlike lua, php or v8.
More like zero downtime maintenance.
UI components operate in separate threads and communicate with the back end by sending and receiving async messages.
Back-end components communicate with servers using async messages, then publish results to front-end components.
This is similar to what Joe Armstrong was working on for X-Windows GUIs: https://erlang.org/workshop/2004/ex11.pdf