What Erlang Taught Me About Distributed Systems

You can cast bytes from a memory-mapped file just fine, if you trust the file/other writers, or are willing to sacrifice some safety. There are also a lot of serialization systems that are simple/fast enough as to basically be structs, e.g. flatbuffers: https://google.github.io/flatbuffers/. Even those still don't avoid that pesky copy, though; for that you'll have to interact with raw memory and hope it's in the right layout for your purposes.

Refcounted binaries are (mostly) effectively copy on write though; if you create a term that's a modified form, a new value will be formed.

Sure, you could pass a pointer around. But then you'd have all the usual headaches about concurrent modification (some of this stuff is mutable when you get down to the implementation/pointer level) and garbage collection (who has it?).

It's optimized well enough that the perf penalty is minor. Copy-on-write semantics, "tail-send" optimizations (I don't know what the technical term for this is, but: data or pointers are moved if a message send is in particular positions in code, without getting the GC involved at all until later) and more are all used where they make sense.

Could it be faster if you built it around a zero-copy runtime? Sure. So could Perl. But then you'd be building your own runtime, with different priorities and tradeoffs than the one that already exists--one whose biggest benefit is that you don't have to build it yourself.

Erlang doesn't promise zero copies locally, but nor does it take the most wasteful possible path; I am not a deep BEAM expert, but am consistently surprised at how well it works. Seldom have I wondered "hmm, the runtime could do something fast/cool with this particular type of code; I wonder if it does..." and, upon checking the available sources/debugger output, been disappointed.

The runtime isn't built around hyper-efficient zero copy semantics (it has a garbage collector, an M:N scheduler, a bloody instruction counter routine that runs every time you invoke any function at all for heaven's sake). For all that, it's more than fast enough for anything besides high-performance scientific computing/number crunching locally. If you need to do that, use a language that lets you sent pointers between threads exactly how you want. If you still want to use that in Erlang, add some reduction-count markers in your $fast_language code and hook it up to Erlang via an NIF.

The BEAM doesn't waste time/complexity more than necessary on hyper-efficient local optimizations so it can focus on its main strengths: really good concurrency and parallelism, and making a near-bulletproof, super-high-performance set of primitives for distributing your code with minimal changes. Of all the "you call this function as if it were local, but it actually runs remotely in nearly the same exact way!" tools out there, even ones invented recently with the benefit of history to learn from, Erlang/BEAM's implementation comes the closest to fulfilling that promise IMO.

> Why would reference counting be worse than copying?

Refcounting is a valid strategy in some cases, but has drawbacks as well. It doesn't handle cycles well, and requires some stop-the-world events (or lots of locking complexity) for garbage collection in concurrent environments. More details on Google, this thread goes into the differences reasonably well: https://www.quora.com/How-do-reference-counting-and-garbage-...

Nobody will complain if it's needed, but chances are, it's not needed. (It depends on what you're doing though).

Plus refcounted binaries scare people. Until recent versions, references didn't play nice with GC targets, so it was easy for a very efficient proxy process to not trigger GC but still keep references to a lot of garbage. If you're not careful with referencing pieces of binaries you can do something similar too -- maybe you got a huge binary, but only need to save a few bytes, it's easy to keep a reference to the huge thing (hint: use binary:copy/1 to get a clean copy of the small piece that you want to keep in your state or ets)

One of the interesting advantage of erlang is that its processes are small and mostly fit in L2 memory. So you can load a whole process at once, then compute for your whole slice, all in cache.

The result may surprise you.