bitwalker

I worked on F-16 avionics (in aircraft maintenance, on the flightline), there are a lot of little nooks/holes/slots/gaps where small bits of FOD can fall and be incredibly difficult to extract, and the fear of that FOD causing a jam, flying around the cockpit, or getting wedged and causing unexpected wear on wiring harnesses and then shorting out (or worse, arcing) during flight was a _big_ deal. FOD in the cockpit was basically the worst thing that could happen during routine maintenance, because if you couldn't see it, and either couldn't get to it with a magnet (or the FOD wasn't metal), it might require pulling _a lot_ of stuff out of the cockpit before you could reach into the area where it fell. The worst case that could easily happen was having to have Egress come out and pull the ejection seat so you could get under it.

I always figured that all of those little gaps/etc. were due to a couple factors:

1.) the aircraft are constantly being upgraded/modified, so even if you designed the aircraft to be gap-free initially, there will inevitably be changes that introduce them. The cockpit itself is basically a frame with racks that hold all of the avionics, seat, etc.

2.) in conjunction with the above, ease of maintenance was somewhat important, so they tried to leave at least a little room to maneuver in the cockpit where possible (though there were plenty of places which were a nightmare to work regardless), but that comes at the cost of introducing areas where things can fall.

3.) some components have to be regularly removed and worked on outside the aircraft, or must be free of obstruction during flight, e.g. the ejection seat. So you end up with plenty of gaps where things can fall.

> If we pretend the Supreme Court is partisan, which it is not but let's entertain that thought here.. I mean, that’s laughable, I don’t know how anyone can call the current Court non-partisan with a straight face. That doesn’t mean every decision is necessarily partisan, but multiple decisions over the past few years have been blatantly partisan, with Thomas and Alito especially making very little effort, if any, to hide it. I used to feel that the Supreme Court was the one of the few non-partisan institutions in our government, but the way this Court has run roughshod over stare decisis to effect change in the interests of its majority was more than enough to disabuse me of that notion. Everyone loves when decisions are going their way, consequences be damned, but the way those decisions get made matters, and nobody should be happy with the way the current court is choosing to act. It has set a really bad precedent for future courts, and has already done immense damage to its reputation in the eyes of many.

What are you talking about? I’ve been on the same dosage for over 12 years, and it is just as effective for me today as it was on day 1. Still waiting for those consequences I guess..

The BEAM is so much more than just a green thread runtime - and it is not straightforward at all to just "bring it over" to another virtual machine, the entire BEAM VM is designed around the scheduler, and core features such as signals (used most notably for links/monitors, but also for a number of other system features) and messaging, are deeply integrated with the scheduler; as are system tasks, execution of NIFs (natively-implemented functions), and more. Furthermore, the schedulers are adaptive to the amount of work in the system, and the behavior of the other schedulers, i.e. they aren't just simple work-stealing queues for green threads. Over 20 years of engineering effort have been invested in this system, and it is highly optimized for the types of workloads that Erlang is used for.

In my opinion, trying to replicate ERTS on top of another virtual machine either requires making that virtual machine more like the BEAM, or will end up always playing catch up with the BEAM itself in some aspect. That's just my two cents though.

I mean, the BEAM doesn't have global GC pauses either, as each process has its own heap - but I would expect Pony can take things a step further as a result of its strong type system, which IIRC is why it can support zero-copy messaging.

I really strongly doubt that GC is a bottleneck for Erlang programs on either the BEAM or the JVM - the sophistication of the scheduler, and the way various language primitives interact with it, is where the BEAM is almost certainly gaining an edge over the JVM. That said, I'm sure there are a subset of programs that _would_ be faster on the JVM, just depends on what metrics are being compared.

Erlang only uses reference counting for binaries larger than 64 bytes, everything else is allocated on the process heap (or in heap fragments) and copied. Just that is enough to have a beneficial effect though, since large binaries are relatively common in practice, and are frequently passed around from process-to-process.

I think that's largely an implementation detail, so not a requirement as far as I know. That said, delimited continuations certainly make them easier to implement, as I understand it.

If you are handwriting the function in assembly, you'll know what registers hold the function parameters, what types of values they are supposed to be, and with care, you can produce debug information and CFI directives to allow for stack unwinding, it's just annoying to do - but that's just the tradeoff you make for the performance improvement I suppose.

Erlang checks for overflow after arithmetic ops are performed internally (IIRC), so you are generally only penalized when overflow actually occurs and requires promotion, which probably isn’t much different in performance from a language that would raise an error on overflow, though obviously not as efficient as a language that just wraps around on overflow.

Erlang binaries on the other hand are awesome, and the use of lists in what Erlang refers to as chardata/iodata is really nice when generating data that will be written to some kind of output device, since no preprocessing is needed to convert the data to a flat array of bytes. Overall though, yeah, it plays pretty loose with types, though it is generally strict about not using integers where floats are expected and vice versa.

This is one of the things I'm personally excited about using Firefly for, since I first started working professionally with Erlang/Elixir, I wanted the ability to use it for CLIs. You can be sure it will be a use case well supported :)

The goal is to support all libraries, but because there are some out there that dynamically compile/load code at runtime, despite it being a bad idea, there will necessarily be some libraries that are not supported, at least in the near term. Likewise, you might also want to compile a library for say, Wasm, that uses a NIF which lacks support for that target, which wouldn't work; but this is already a restriction with the BEAM today with NIFs (i.e. if they don't support a particular target, then the library won't work).

But as long as a library only uses code loading APIs at compile-time, and not runtime, it should be supported just fine. To be clear, we only plan to raise runtime errors in those cases, as if the call failed normally, and not prevent compilation just because a call to one of those APIs exists in the code, though we may choose to emit diagnostic warnings for it, that remains to be seen.

> How do you plan on ensuring your compiler tracks updates introduced by major Erlang releases? Will there be features introduced in the Erlang runtime (I'm thinking things like Erlang 21's `atomics`) that won't be immediately available for FireFly-compiled programs, where FireFly will just choke on these?

We would plan to track Firefly releases against specific mainline Erlang/OTP releases, so for example, we would explicitly state that Firefly v1 tracks Erlang/OTP 25, v2 tracks 26, and so on.

Most changes implemented in each release are implemented in the Erlang-based standard library, so we would get most of that for free (we aren't planning to maintain a completely separate fork of OTP, only the parts that we need to implement ourselves, primarily stuff that is already separate and provided in the preloaded modules shipped with ERTS). What remains is typically small enough that we should be able to maintain parity pretty closely, though naturally there may be delays depending on the scope/effort involved. Obviously we hope to grow enough community around the project that this is a non-issue, but the project has enough financial backing at this point to handle this in the near term. I also hope to make alternative runtimes/implementations of Erlang/OTP something that the core team takes into account via the EEF, ideally resulting in some kind of technical spec around semantics of the language; in the near term we have to rely on the OTP test suite, various papers that have been produced over the years, and a whole lot of digging through the ERTS code itself.

> Hot code loading happens in places other than just relups, though, no?

Yes, of course, and obviously that means some libraries won't be compilable with Firefly. In my experience though, the majority of production apps are not and should not be doing dynamic code generation/loading at runtime; that's certainly something I would raise in a code review if I saw it. Naturally there are going to be teams out there that _are_ doing so, or use tools/libraries that do so, but I'm fine with this being a reason why you would choose the BEAM instead.

> Even if you have no plans to support hot code loading in the "remote calls can jump into the new version of a module" sense, do you think it would make sense for FireFly to ever support "module compilation+loading at runtime"? Maybe with the resulting modules being native DLLs?

We do support dynamic loading of libraries containing new modules/functions, and I think it is certainly possible for us to provide a JIT on supported platforms in the future to do the kind of unbounded code generation/loading that the BEAM supports, but it is not a priority by any means. As I've stated previously, it is rare that I've seen that kind of thing abused in production applications, and I don't see it as a major selling point of the platform (or a blocker for Firefly). It is a tradeoff though, and that does mean there will be things you just can't do with Firefly - but I think that's an important property of alternative implementations; ideally you want them to be tailored towards different use cases, otherwise there is little reason to have multiple implementations in the first place.

> Tangent to that — and I know that this was probably nowhere on your mind when you were working on the project, but something interesting to consider: how hard would it be to convince the FireFly compiler to emit a C-ABI library that could be loaded into the BEAM as a NIF?

You'd be surprised what things I've considered while working on this project ;). I think it is certainly possible, and I'd like to support it, especially since I think it might be a great way to use Firefly in a traditional BEAM deployment for things that, as you mentioned, one would have previously considered using HiPE for. I have yet to investigate just how tight that integration could be though.

Smaller code size was the goal; and BEAM bytecode adds up quick, even compressed. For example, the BEAM files for _just_ Ecto come out to about 1.2M in an uncompressed tarball, but in a gzipped tarball at maximum compression is still 799K. In my experience, about the smallest release for the average application in terms of BEAM bytecode is about 30M (uncompressed), across the standard library, dependencies, and your own code. Virtually none of that can be dead code eliminated, because the BEAM has to treat it all as potentially reachable. Shipping that much to a browser (even compressed) is just not viable.

Your point about instructions remaining hot in the cache might very well be true more often than not, but is highly sensitive to the application in question. The core interpreter opcode impls might all fit in cache at the same time (though I doubt even that with the BEAM due to how many there are), but any call to BIFs/NIFs is likely to cause evictions. It still might do better than natively-compiled code overall in that specific sense, but I would be hesitant to state any generalities about it when considered as a whole with all of the many other factors that play in to overall performance.

In any case, Firefly wasn't about building a faster BEAM, but about bringing BEAM languages to the browser (or really any Wasm host). While targeting standard server/desktop architectures was something we also wanted to support, particularly for writing CLI tools and such, we expect the BEAM will always be the first choice for people deploying to those systems. If we can build something that is faster than the BEAM in some cases due to the tradeoffs we make, that's great, but it isn't an explicit goal.

When I say hot code loading, I’m really specifically referring to how that works in the BEAM, which is a more sophisticated mechanism than simply compiling/generating code on the fly. The biggest problem though is that it prevents most forms of dead code elimination. It also means that you can never assume anything about how a function will be called, because at any point new code could be loaded that calls it differently. You can still optimize such code with a tracing JIT, but in Wasm (at least currently) that’s not even an option.

Without knowing too much detail about SBCL specifically, I suspect that they use a combination of clever static analysis and specialization to unlock a lot of that speed. That way functions can be specialized/inlined where beneficial, but new code can safely call the unoptimized versions when loaded at runtime, but even hot loaded code could be specialized with a JIT on hand. The big reason we made the trade off with hot code loading though is due to the restrictions that Wasm imposes - there’s no particular reason we couldn’t support it otherwise. In general it is rarely used in production BEAM apps in my experience, so from my perspective it seemed like an opportunity to stop paying for a feature unused and gain something in return.

Thanks! I’ll put some together in the next couple days just to make sure it gets done, so even though running those apps may not work (depending on what’s used), at least it’ll be easy to play with it.

Speaking as the lead on the project, this is partially due to this weekend being ElixirConf, so things are hectic, but you can also blame me, as I probably should have written this post, but didn’t make the time as I was pretty heads down on the lead up to the conference.

I’ll make sure to do a follow up post in the near future that is more in depth

It's not quite ready for that level of experimentation yet, we only recently got the compiler implemented, and there is much remaining in the runtime to finish up. There will be more announcements in the future related to the project, and I'll make a note to ensure we provide the kind of instructions you're looking for when the time comes.

> Only AOT compiled native code, except targeting WASI.

More precisely, a number of targets are supported, WASI would be just one.

> Generally BEAM is understood to be slower of the big runtimes (compared against Java, CLR, Go and often V8/Javascript). FireFly claims to be faster, and smaller in compiled form than current interation compiled against BEAM.

The point is that by placing some restrictions on what is possible at runtime (specifically by removing the possibility of hot code loading), we can do whole program analysis and thereby do much more aggressive forms of optimization and dead code elimination across compiled applications _and_ the runtime they link to. It isn't guaranteed that such programs would be faster (though I suspect in some cases they would be), but they almost certainly should be smaller, which is important for Wasm, and other constrained targets.

> How that plays with actor model and preemptively switched lightweight processes is a mystery to me too.

It makes no difference, you can implement all of that with identical semantics from the perspective of the developer, the strategy used for compilation is orthogonal to those features, though naturally the implementation details are tightly integrated.

> What does this mean? Erlang/Elixir are AOT-compiled languages..

With the BEAM compiler today (and by BEAM compiler, they are referring to the compiler provided as part of the standard library shipped with the BEAM, it is just a way to clarify which Erlang compiler is referred to), Erlang sources are partially AOT-compiled to BEAM bytecode, then the code loader does additional compilation steps at runtime. Firefly AOT compiles Erlang to native code directly, and does not use a virtual machine at runtime.

> So is the key benefit here that the compilation is faster because it's not running on the BEAM (good for e.g. CI); or is the key benefit that the resulting executable is faster because it's not running on the BEAM?

There are a few benefits we hope to provide using this approach:

1. Compilation can be faster because the compiler is implemented in Rust, rather than implemented in Erlang and running on the BEAM. 2. We impose a restriction that hot code loading is not permitted, so we are able to do forms of optimization that the BEAM cannot do, as a result of having to pessimize in the presence of hot code loading. 3. Related to above, we can do whole program analysis, including optimizations that take advantage of unboxing terms and working with more machine-friendly types. 4. We can produce a single statically-linked executable that is as small as possible, in part due to being able to do more aggressive and precise dead-code elimination across both the application being compiled and the runtime it links to, because we know what must be available at runtime. On the other hand, if one were to ship the BEAM itself to run in browsers via Wasm (if it was modified in such a way as to be possible), you'd also need to ship large amounts of bytecode for most applications, regardless of whether much of that bytecode is actually needed at runtime. Deployment has always been a pain point of the BEAM (and I'm speaking as someone who built the primary release tools for the Elixir ecosystem), Firefly aims to make deployment as simple as Go/Rust/etc. 5. I'd like to be able to take advantage of the fact that we use MLIR behind the scenes to explore using Firefly as a natural pairing with Elixir applications using Nx, by being able to more seamlessly integrate regular Elixir code with Nx-managed functions.

> How does a non-bytecode version of Erlang/Elixir abstract-machine semantics, achieve these same guarantees? Is there an explicit reduction-counter being carried around in the emitted native code?

In short, yes, we implement preemptive-scheduling the same way the BEAM does, using compiler-injected yield points based on a few criteria (a reduction counter is just one, some others are garbage collection and blocking I/O).

> And also, there is no mention of disadvantages/constraints of using this system. It's pretty clear that you wouldn't be able to do hot reloading or dynamic trace-point insertion without the BEAM there to intermediate it. That's fine for some use-cases, but they should explicitly mention the trade-offs and target audience.

The readme of the project certainly does this, but I agree it would have been good to include in the blog post. In any case, Firefly is still in early stages, so this isn't something anyone is using today. When we reach the point where we feel it is production ready, I can assure you I will be writing up a very detailed analysis of what it is ideally suited for, and what it is not - like you, I feel it is critically important to be clear about that.

It doesn't currently, but you are correct that the goal is to maintain feature parity with the BEAM (with explicit caveats to that, namely hot code loading). There actually is support for NIFs, just not via the erl_nif interface that NIFs use today, support for that will arrive eventually.

Correct, the BEAM is just one implementation of the runtime, there can (and have) been others. Firefly aims to be as close to the BEAM semantics as possible, but there will naturally be some differences as we're taking a different approach with compilation, with some benefits as a result, but naturally there are also tradeoffs.

You don't lose OTP, because OTP is a library, written almost entirely in Erlang (not counting the set of NIFs/BIFs which provide intrinsic functionality), which we absolutely aim to compile with Firefly just like any other Erlang (or other BEAM language) sources.

The BEAM also provides a runtime, but that runtime can be implemented using other strategies. It essentially provides a M:N green threading abstraction (processes), with a specific set of semantics around how those communicate (messages) and how failure is handled (links/monitors/etc). Firefly provides a runtime that aims to be equivalent to that of the BEAM from the perspective of the developer, the only difference is in how that is done behind the scenes, what is produced by the compiler, and what restrictions we impose that the BEAM doesn't (namely no hot code loading, at least for the forseeable future).

I'm not sure where you got the idea that Firefly throws away OTP, or tries to implement Erlang with different semantics, because that is explicitly _not_ the goal.

One example would be in a parser, as various parsing functions would be invoked recursively until end of input, or until some non-matching input is encountered. In pretty much any scenario where you would use recursive functions, it is quite common to have mutual recursion between multiple functions. Not always of course, which is where the tail call elimination optimization is typically applied, but not being able to freely use mutual recursion to arbitrary depth turns out to be quite an annoying limitation.

Fred's blog, and Learn You Some Erlang for Great Good are invaluable, but on the topic of production systems, his ebook Erlang In Anger (https://www.erlang-in-anger.com/) is excellent as well - honestly it's hard to overstate just how much good he's done for the community in terms of documenting and philosphizing about Erlang, architecture and operating production systems. He's solid gold!

The Erlang Manual does a pretty good job of describing the system limits and how to configure them, as well as the failure modes of various components.

By default, a mailbox will continue to fill up until process reaches its configured max heap size (which by default is unlimited, i.e. the process heap will grow until the system runs out of memory, eventually crashing the node its running on). However, you can configure this on a process-by-process basis, by specifying a max heap size and what to do when that limit is reached. This is described in the docs, but as you mentioned, it's not necessarily apparent to newcomers.

But aside from that scenario, I think a lot of the interesting failure scenarios are really sensitive to what the system is doing. For example, network partitioning can either be a non-issue, or critical, depending on how you are managing state in the system. As a result, I don't think there is too much that really digs deep into those problems because it turns out to be really hard to document that kind of knowledge in a generic fashion - or at least that's how it feels to me. Everyone I've worked with has built up a toolbox of techniques they use for the task at hand, and do their best to share them when they can. It's unfortunate there isn't really a one-stop shop of such information out there though.

I think it's probably also good advice for newcomers to remember that you don't have to use something just because its there (like mnesia) versus something you are already running or are more familiar with which solves the same problem (e.g. redis).

I mean, if it doesn't count as an implementation of an actor system, then I'm not sure what does, regardless of whether it was incidental to the design goals of Erlang as a language. It certainly walks and quacks like an actor system in my opinion, and I don't think you have to squint very hard at all to see it.

Perhaps we have different ideas of what an "actor system" is though.

This isn't really an issue with threads though, the exact same issue is present in green thread/fiber implementations; it just so happens that in Async Ruby the GIL saves you from this specific problem due to making variable accesses atomic (as I understand it anyway, I'm not super familiar with the Ruby VM).

In general, green threads/fibers are vulnerable to the exact same shared memory issues as threads, the only benefit to them is that they are, for a certain class of problems, a more efficient concurrency primitive than threads by avoiding context switches out of userspace, and in many cases provide you the ability to plug in your own scheduler if you so desire allowing you to optimize scheduling for your own workload.

Just like in Erlang, you would handle that at the transport level, and it appears Swift supports providing your own transports. They didn't mention (or I didn't see) whether the transport they are shipping has any specific auth wrapped around it though.