Why Erlang Matters
sameroom.io
sameroom.io
All other major implementations (including Akka) have cooperative scheduling, i.e. forbidding blocking code in actors. Erlang allows it. This is huge.
And actor supervision is the best way to write reliable systems. I have wrote some code in Akka without much effort and testing (streaming market data aggregation), and it is still running with a few years uptime.
This isn't to say that one can't create a process that eventually has harmful consequences to the entire system, it just means it's much less likely. These little points of isolation add up. Scheduling, memory management, lifecycle, crashes, &c... It's why it's a bit funny when I hear Akka compared. Akka is quite impressive but I'd never consider it a true alternative to an Erlang or BEAM style runtime.
This allows skipping most of error handling in Akka code. If something's wrong, your actor will be restarted. Works great!
I think a preemptive scheduling model is really the only way to implement that model effectively. Any other scheduling method eventually causes the consistency of the programming model to break down as it's now pushing the cognitive burden about what to run, where, and when onto me as the developer again.
Numerical and statistical stuff, for example, is never going to be performant--but that's okay, because I'd much rather have a whole bunch of slow actors making incremental progress safely than a couple of blazing fireballs that could take down my system using NIFs or whatever.
I implore anyone interested in Erlang or its surrounding languages, to read its source code. [2] More specifically, the BEAM. I'll warn you that it's very 80's hackeresque, but in a good way. Incredibly pragmatic. The way they achieve their world-class scheduling? Tip: not via a perfect theoretical design cooked up in some comp-sci lab. [3] If you've been following game engine design within the last 9 years or so, you'll probably have a good idea of how they do it. [4][5]
[1]: https://medium.com/@jlouis666/erlang-dirty-scheduler-overhea...
[2]: https://github.com/erlang/otp/tree/maint/erts/emulator/beam
[3]: https://hamidreza-s.github.io/erlang/scheduling/real-time/pr...
[4]: http://blog.molecular-matters.com/2012/04/05/building-a-load...
[5]: There's a hell of a lot of depth to this problem domain. Charles Bloom's guiding principles for Oodle are spot, and well worth a look.
Where possible, I think the Erlangy thing to do would be to just have a separate executable and communicate with that. Also ensures that things keep running if there's a segfault or something.
You can always throw the NIFs onto other nodes. You can also write ports that are glorified external processes.
Other times, you just need low-latency and high throughput. In that case, you expect failure and design accordingly.
I consider the BEAM VM as one of the marvels of software engineering.
You know it is good, when you explain to other programmers that you can have something like an isolated memory process just like an OS process, with preemption and only a few Ks of memory, with a low latency GC, with distribution across machines built in -- and they don't believe me.
Even experienced senior developers are skeptical saying stuff like "well yeah but those are green threads then and they have to yield explicitly -- nope, they don't", "so you have callbacks then somehow, no, not callbacks", "what do you mean separate memory spaces, it is a single OS process right?" and so on. It sounds like magic -- this stuff shouldn't exist, right, but the awesome thing is, it does.
Moreover, it is not a hacked up version from a lab some place or a PhD disertation proof of concept -- this is what powers banking, databases, messaging, and probably more than 50% of smartphone access to the internet today.
$ erl
Erlang/OTP 18 ... [smp:4:4] [async-threads:10] ...
The smp:4:4 says there are 4 schedulers configured and enabled. Usually there is a scheduler (as an OS thread) which runs on each core (also highly configurable, with custom topoligies, affinities etc). Those schedulers will pick and run Erlang processes.
Typically to ensure fairness, each process is allowed to run a set number of reductions (think of them like equivalent bytecode, but say an internal C driver like a regex parser should also periodically yield and report it consumed a given number of reductions, so it can be descheduled).
Funny enough all that sounds a bit like an operating system, and that's a good way to conceptualize it. Erlang/OTP is like an OS for your code. A modern OS is expected to be resilient against bad processes messing with other processes' memory, multiple applications should run and be descheduled preemptively etc.
https://github.com/mbrock/HBEAM
It's a Haskell executor of .beam programs, in very early prototype stage, and I abandoned working on it 5 years ago (apparently).
Of course it's not meant to be competitive in any way, it's basically just for fun, and because I wanted to learn more about how Erlang works.
The function `interpret1` in the middle of this file has the main opcode switch.
https://github.com/mbrock/HBEAM/blob/master/src/Language/Erl...
I think it's about the smallest subset needed to run a factorial program, but also to implement the very basics of mailboxes with send/receive/timeout.
It uses GHC's shared transactional memory for the mailboxes:
https://github.com/mbrock/HBEAM/blob/master/src/Language/Erl...
Someone, fork it and finish it! :-)
(1) How does it preempt threads (my guess: it doesn't actually preempt threads, the interpreter yields after a certain number of instructors, or (if compiled) the compiler inserts conditional yields in each loop/function call/return)?
(2) How are memory spaces isolated (my guess: they aren't really, it's just that the memory allocator doesn't mix memory allocated by different threads)?
Yep. The interpreter lets each thread do 2000 reductions (roughly == function calls), or until it waits for new messages if that's sooner.
(2) Erlang VM instance (it is called a node) is an OS process (plus some helper processes, but they are not important in this case) is of course one heap from kernel's point of view. But internal allocator keep spaces separated for BEAM's data. It is not always as basic as in some cases for binary blocks and sub-blocks it can actually share and ref-count them.
In the new release I like that it could have mailbox outside the main heap of each Erlang process. That could be an interesting parameter to play with.
(bad acting) NIFs. Thankfully you can mark 'em as the dirty evil little things that they are (with negligible overhead):
I feel like I'm missing something important.
If you only have two or four or sixteen threads you write overtly threaded code. But when you have millions you don't. Your program looks single-threaded and yet operates better and more safely.
You don't have to intentionally write a work queue and balance the number of readers vs writers, etc. You just let the runtime make things go as fast as possible.
They do get balanced across cores in almost all cases. Erlang is a functional language which really lends itself to this.
For example: on a network service, you have 1 thread listening for new connections, when a new connection is made. It starts a new thread, which calls the handler. The listener thread then goes back to listening for new connections.
Now the advantages can depend on your green threading implementation. If a listener thread blocks on reading from Disk or a DB. Then the listener thread can still wait for new connections and other connection handlers can still operate. Making you network application responsive, without increasing latency on clone syscalls.
Of course you can achieve this in other ways.
Second, they are much lighter on memory (well, comes with the first point, but still).
Third, the supervising VM/environment has more fine-grained control over them than with native threads.
And in any decent implementation, they are absolutely load balanced across cores, why wouldn't they be?
Linux switches between threads at some frequency, I think it used to be 100 Hz. It involves swapping out the process registers, doing some kernel bookkeeping, etc—this is called a "context switch" and it's quite costly. Also, Linux threads allocate at least one memory page (4 KB) for the stack. [If I'm wrong about these details, please correct me!]
Basically, the cost associated with an operating system thread comes from the fact that it has to be isolated from other system threads on a low level... whereas language runtimes that offer green threads impose their own safety via language construction, e.g., Erlang processes can't reach in and mess with other processes memory (without C hacks).
So green threads can be much more efficient, but they require some care in the implementation, especially to support I/O, and to have fair and efficient load balancing, etc. Then you run N operating system threads to get balancing across cores, and distribute green thread work.
As others have indicated, they're extremely lightweight, cheap to create and throw away, and are load balanced across cores.
But also, each has isolated memory, with share nothing semantics (with a couple caveats) which means that an exception in one won't affect others -unless you want it to-. That's huge.
But as has also been mentioned, you can create many of them. Someone else threw out 50k; nevermind that, try a million of them on a single box. That kind of concurrency opens up an entirely new paradigm of coding. One that is actually very useful, because it turns out, a lot of problems are naturally concurrent problems, that we've been trained to think about in sequential patterns because of how hard concurrency is.
An example I like to give is from the real world - task scheduling. We had to write some simple task scheduling for an application. Each task was multiple steps, many of which were time based (i.e., "execute this command, wait X amount of time, execute another command, wait Y amount of time, execute a third command, once that is successful execute a fourth command"). The traditional way of doing this would be some sort of priority queue, with tasks weighted by how long from now until they were to be done. You check how long until the next event, sleep until then, fire it, then repeat. Simple, right?
Except...each event leads to more events. And event timings can change. And events can happen simultaneously, so you actually need a pool of threads to actually execute the events on. Locks everywhere. Task logic is very hard to isolate from the execution logic (i.e., the bit that says "do X, and create an event to execute at time Y" is hard to keep entirely separate from the "pull event from priority queue, throw to a new thread to execute on, sleep until next event", since there are so many interactions between the two that can affect one another, changing when events happen, and when the queue puller needs to wake up).
In Erlang though? Trivial. Write your entire task as a single job. I.e., do x, wait, do y, wait, do z, wait for a message that z has completed, do a, etc. Then spin up one of those for each task that you need and let the VM handle the concurrency aspects of it. Even additional complexity, like "in the event of a message, change the amount of time until the next event to be half of what it was" is trivial; it's all contained in the same module, it all describes the same lifecycle of a single task. All the concurrency, the running of many of those tasks, and their interactions, and ensuring none is blocked, etc, is -free-.
This sounds like an obvious, ideal example once explained, and yet, every person where I worked who was unfamiliar with Erlang (and even some of those who had coded a little in it, but hadn't come to grasp the paradigm as well), who was explained what we were trying to do, described it as "easy, we just need to use a priority queue and pull from it!"
i = 0
for{
i += 1
}
Erlang on the other hand assigns a certain amount of time units to each process, and each pure-Erlang operation consumes these, so an Erlang process can always be interrupted as it will eventually use up this time allocation and yield (unless it's actually calling a C function etc).
When your CPU intensive code does not allocate, it will not switch to another green thread.
So point in the example above is absence of memory allocation. And, having decent codegenerator, Haskell could produce the code above where registers are reused for new values (effectively, mutation).
package main
import "fmt"
func main() {
go hog()
for i := 0; ; i++ {
fmt.Println(i)
}
}
func hog() {
for {
}
}
Here's the issue on github:
package main
import "fmt"
import "runtime"
func main() {
go hog()
for i := 0; ; i++ {
fmt.Println(i)
}
}
func hog() {
for {
runtime.Gosched()
}
}Need some citation there, because we've been writing extremely reliable systems (multiple nines) for decades in various languages (mostly C, C++ and Java).
All these languages (including Erlang) come with pros and cons to write such systems, but so far, Erlang has failed to deliver on most of the promises that its advocates repeatedly make.
Speaking of citations...
We're doing multithreaded programming just fine in a lot of very varied languages.
Need some citation there,
According to Wikipedia, shortly after Armstrong was let go of Ericsson, the company quickly ripped out Erlang from all its products and replaced it with C and C++.
> In 1998 Ericsson announced the AXD301 switch, containing over a million lines of Erlang[...].[8] Shortly thereafter, Ericsson Radio Systems banned the in-house use of Erlang for new products, citing a preference for non-proprietary languages. The ban caused Armstrong and others to leave Ericsson.[9] The implementation was open-sourced at the end of the year.[5] Ericsson eventually lifted the ban; it re-hired Armstrong in 2004.[9][...]
> Erlang has now been adopted by companies worldwide, including Nortel and T-Mobile. Erlang is used in Ericsson’s support nodes, and in GPRS, 3G and LTE mobile networks worldwide.[10]
https://en.wikipedia.org/wiki/Erlang_%28programming_language...
"In 1998 Ericsson announced the AXD301 switch, containing over a million lines of Erlang and reported to achieve a high availability of nine "9"s"
That was the first commercial use of the language. The first commercial use of a 'niche' language, with over a million lines of code, achieved a downtime of just over half a second over 20 years. That's total downtime, too, not just 'unplanned'. And even if you take into account the numbers touted by critics of that quote, of 5 nines...that's still considered world class. For the first damn commercial product.
That's delivering.
Nonetheless, it powers WhatsApp and Facebook Messenger, with their billions of users. Quite a proof, I'd say.
Not that the others are unsuccessful from an engineering or financial standpoint, but the last one, Riot Games' League of Legends recently hit 1 billion in revenue. It's a fact that it's the most popular video game on Earth today.
If it didn't work or "failed to deliver on its promises"- none of these world-class, top-tier services would be where they are. To repeat myself so it sinks in. Yes. It is a fact that every phone call you make hits Erlang code somewhere.
Because people actually stand in the face of Erlang and spout this nonsense to this day... I'll take the claim one step further and will back it up if challenged as if it's not already self-evident.
Erlang has proven itself better, for a longer period of time than Java. Many have tried to replace Java. No one, and I mean no one has tried nor has anyone succeeded in replacing Erlang.
Only C rivals it for carrying the world on its shoulders.
for(;;){
}
The IO you are talking about - that is I assume IO methods/libraries that are specifically written/adapted for use in Quasar, yes? If you just use an off-the-shelf JDBC provider, you would block the entire thread when one fiber calls out to the database right?
If so, that is not preemption, that is cooperative multi-tasking.
As to the choice of libraries, that is an artificial distinction. Runtimes like Erlang and Go also require libraries specifically written/adapted for the runtime; calling an off-the-shelf IO library from Erlang/Go would also block the entire thread. As integrating a library with Quasar does not require changing its interface, the question of whether to trap calls to specific implementations (which is trivial on the JVM) or require the use of fiber-friendly implementation (wrappers) is a question of design. So far, we've opted for the latter simply for the sake of "least surprise", but we may choose to do the former, too. Also, unlike Erlang/Go (I believe), any accidental blocking of the kernel thread is automatically detected by Quasar at run time, and reported with a stack-trace.
Quasar operates in the exact same manner as Erlang/Go, only that we've disabled time-slice preemption once we realized it neither helps nor is it required on the JVM. The only real difference is one the nature of the ecosystem: while all pure-Erlang libraries are designed to work with fibers, most JVM libraries aren't, and so require a thin integration layer that is provided separately. OTOH, thanks to the size of the JVM ecosystem and the standard interface approach, I believe it is the case that today there are more IO libraries that support Quasar fibers (e.g. all servlet servers) than those supporting Erlang processes.
It is one of the open issues with Golang.
https://github.com/golang/go/issues/11462
User aclements notes:
> @GoranP, you can think of the Go scheduler as being partially preemptive. It's by no means fully cooperative, since user code generally has no control over scheduling points, but it's also not able to preempt at arbitrary points. Currently, the general rule is that it can preempt at function calls, since at that point there's very little state to save or restore, which makes preemption very fast and dramatically simplifies some aspects of garbage collection. Your loop happens to contain none of these controlled preemption points, which is why the scheduler can't preempt it, but as @davecheney mentioned, this is fairly uncommon in real code (though by no means unheard of). We would like to detect loops like this and insert preemption points, but that work just hasn't happened yet.
Now, each actor per actor model convention has to survive restarts. In distributed environment, the state can be persisted e.g. in distributed memory cache / storage (Hazelcast, Cassandra).
> All other major implementations (including Akka) have cooperative scheduling, i.e. forbidding blocking code in actors. Erlang allows it.
Curious as to what people's thoughts on Celluloid:IO are with respect to solving the problem of blocking code in actors. Celluloid:IO provides an event driven reactor that works in parallel with the actors. I haven't used it myself, but it seems an interesting approach.
Every single state change requires a function call, every function call involves a pattern matching algorithm that has to account for all of your program arguments (effectively almost the entire state of your thread!) and isn't even strongly typed. And then there is synchronization and data sharing between threads, which seems a bit handwavy and effectively requires a database running in your program that the threads can poll.
If your problem set is lots and lots of small independent tasks that don't have to finish overly quickly, then Erlang is fantastic. Stuff like switching voice circuits for example. But I'm trying to imagine manipulating a 1TB dataset with thousands of worker threads if you have to make a copy on the stack for every change every worker makes.
I know people do some big data stuff with Erlang, so these have to be solved problems somehow, but I can't help but to suspect that they have to compromise some of the ideals espoused in this article to make it work.
It would be interest to re-engineer a language today that tries to fit into Erlang's niche but has a stronger performance focus. Rust, Cloud Haskell, and Go all sort of cluster in the area I'm thinking about, but none are quite what I'm thinking of. Cloud Haskell is probably closest but writing high-performing Haskell can be harder than you'd like. Rust shows you don't have to clone Erlang's immutability and all the associated issues it brings with it for inter-process safety, but Rust of course is "just" a standard programming language next to what Erlang brings for multi-node communication.
In the aughts Ruby and Python were really slow so if you had computational-heavy problems you had to drop into C.
It worked but C isn't great. The thing is - we have a lot of languages that can do computational problems easily now - Rust, Go, Nim, and the list goes on....
It becomes relatively trivial to create libraries which could wrap these languages for Erlang/Elixir. In the few cases where Elixir isn't fast enough, just drop down to something else. Write a small piece of code in Rust (you may as well call Elixir/Rust peanut butter and jelly). Optionally, you could create some kind of DSL which compiles down to another language in Elixir. I did the same with xjs (Elixir syntax, Javascript semantics) [0], and I must say, for a 200-line hack it works really well.
This isn't even considering the fact that a lot more work could be invested in Erlang's VM. Yes, Ericsson is its corporate sponsor but imagine if you had companies trying to make it fast the way they try with Ruby, Python, JS, or another of other more complicated languages.
I think this is relatively low-hanging fruit. You can't tell me that Erlang and Elixir are harder to make fast than other dynamic languages (in most contexts).
One language that I think is woefully underappreciated for how ubiquitous it is is GLSL. With OpenGL 4 Compute Shader you get surprisingly close to general-purpose use for the type of tasks that benefit from massive parallelism. And GLSL is really quite a nice language; driver bugs are the main things holding it back.
Maybe for highly SIMD stuff but it seems like pull in a bunch of baggage around GPUs, etc.
On the other hand it would force you to do your data partitioning right up-front(much like the SPUs on the PS3).
Then again I'm used to mobile GPUs where any conditional statement used to cause the shader to be evaluated 2^n for each and the gathered at the end(aka forget about any branching).
For my 2c I'm a fan of Elixir + Rust, Rust has a nice C ABI that should make it easy to embed.
And yes, I am doing this, and the personal reason is for fault-tolerance and the professional reason (or how I get time to do it) is based on security criteria.
Speaks the Erlang network protocol and impersonates
an Erlang node on the network. Fully capable of
bi-directional communication with Erlang.
You can also use c nodes (or the JVM interface, which is pretty similar to c nodes I think).
... You can also use ports which define an interface for communicating with external processes.
The world is your oyster!
A language that allows mutable state aside from at process scope is also a no-go. In Erlang you're not supposed to think about what code will be running on which machines as you write the business logic, so you have to assume that every process is running on a seperate machine with no shared memory, so no mutable state above the process level. And mutable state at local scope makes hot swapping code messier, although that's easier to work with than static typing.
Nothing about Erlang is inherently slow. Someone could hire a bunch of developers from v8 or Spidermonkey or maybe just Mike Pall to write a better Erlang runtime.
"A language that allows mutable state aside from at process scope is also a no-go."
Well, I did say the language I'm spec'ing probably doesn't exist. Rust is an interesting example of what can be done to make it so that not actually copying memory is safe, but you'd still have to do some work to make it do the copying more transparently across nodes, which is why I said it's "just" a regular language from Erlang's point of view.
"Nothing about Erlang is inherently slow."
I now believe that dynamically-typed languages that are not built from speed from the beginning (LuaJIT being pretty much the only reason I even have to add that parenthetical) are inherently slow. I've been hearing people claim for 20 years that "languages aren't slow, only implementations are", I've even echoed this myself in my younger days, yet (almost) none of the dynamic languages go faster than "an order of magnitude slower than C with a huge memory penalty" even today, after a ton of effort has been poured into them. Some of them still clock in in the 15-20x slower range. Erlang is a great deal simpler than most of them, and I don't know whether that would net an advantage (fewer things to have to constantly dynamically check, although "code reloading" mitigates against some of these) or disadvantage (less information for the JIT to work with). Still, at this point, if someone's going to claim that Erlang could go C speed or even close to it in the general case, I'm very firmly in the "show me and I'll believe it" camp.
At some point it's time to just accept the inevitable and admit that, yes, languages can be slow. If there is a hypothetical JS or Python or PHP interpreter that could run on a modern computer and be "C-fast", humans do not seem to be capable of producing it on a useful time scale.
---
There are a number of other dynamically typed languages that have fast implementations. Javascript has a few; Common Lisp, Self, Julia, to name some others. They'll never be as fast as C, certainly not when comparing highly optimized programs, but they're fast. It looks like most dynamically typed languages can be made to run 10x slower than C. Compare that to CPython and HiPE which are more like 100x slower.
I don't think code reloading would hurt JIT performance too much. The prerequisites for runtime specialization of procedures basically accomodate eveything hot swapping would need. I also think the way people use Erlang's type system is probably more ammenable to conservative type inference than the existing fast dynamic languages, and that's one of the more important metrics.
Heh it is funny. I might have an idea why. It is hard to use regularly to perform day to day releases. Simply because building correct apups and so on take a lot of time. Most systems are already decomposed into separate nodes and cand handle single node maintenance, so that is what we do at least. Take a node down, upgrade, bring it back up. Care has to to taken to have mixed version in a cluster but that is easier than proper 100% clean hot upgrade.
But having said that, I have used hotpatching by hand probably 5-6 times in the last couple of months. Once on a 12 node live cluster. That was to fix an one-off bug, for that customer before having to wait for a full release, another time was to catch an function_clause error that was crashing gen_server and so on. It was is very valuable having that ability.
> till, at this point, if someone's going to claim that Erlang could go C speed or even close to it in the general case, I'm very firmly in the "show me and I'll believe it" camp.
It doesn't matter if it goes C speed, it has the fault tolerance, expressive language, it is battle tested, it has good runtime inspection and monitoring capability, if someone came one day and said you lose all those but you gain C speed, I wouldn't make that trade.
Interesting. What are the exceptions you have in mind to warrant the (almost) hypothetical? You mention LuaJIT. I've also heard that Q/KDB are quite fast. Anything else?
One is where you live upgrade a full running release, including all applications and version, where you mutate state that had its format changed. All this in production, without any downtime. This is incredibly hard to get right. Erlang gives you a lot of tools (OTP & friends) to achieve this, but it is still very complex.
The other is reloading Erlang code in a runtime system. I.e. recompiling and reloading one or several modules in a runtime system. This is usually done during development (see Phoenix for Elixir for example) or perhaps even in production when you know what you're doing. This is relatively easy, with some risks of course if you are doing it in production.
An example would be changing log level settings. Normally Elixir log blocks can be compiled entirely out when running in production mode. But it should be possible to fairly safely recompile with debug logs enabled and reload without missing a beat.
A nit: Erlang is strongly typed. You cannot (ok, excluding numbers) almost transparently convert a string into a number into a tuple like you could in C, to which everything is merely memory so casting allows trivial but potentially erroneous conversions. Erlang is dynamically typed, not statically typed. Dialyzer + type annotations (and some inferred) allows for static analysis, but it's not directly part of the compiler so it can't be strictly enforced (automatically).
EDIT:
Also, consider that erlang processes are a dual of objects in the OO sense. They can receive various messages and respond by changing state and/or transmitting messages. This interaction is similar to the way objects (instances of classes) behave in OO languages. The difference being that they're all running concurrently. So you could put a massive amount of state into one process. OR you could have a handful of processes that act together like you have a handful of classes in an OO language.
If I write a server for playing a game, I don't include all of a player's state and their connection in one process. I have a process that handles the network connection. It sends messages to a process (or processes) that handle player state. Which connect to some game state processes, and on until the network handler sends messages back to the player or gets more messages from the player.
Huge parallelism is easy if your data and processes are largely independent, but the real world is rarely so kind.
Your tradeoff is between “tightly bound by shared data structures” vs. “tightly bound by process synchronization”. I don't see how either is better than the other.
> the real world is rarely so kind
In the real world, from what I've seen, while everything is interconnected to everything else, not all the connections are equally strong or important. If you want to compute exact results, without possibility of failure, no matter what the computational cost, then sure, you need to take all the connections into account. If you can trade some accuracy for performance gains in the average case, you'll probably want to find ways to prevent minor failures from bringing down the entire system.
But you've also increased your reliabily as well. Who cares if the tight single threaded application with a shared heap cand handle 100K connections, if as soon as one of those connection leads to a segfault, all the other 99999 crash as well.
The dual of object types (records of methods) are sum types: Object types are defined by how you can eliminate them (calling a method on an object), whereas sum types are defined by how you can introduce them (applying a constructor to suitable arguments).
In my experience with Elixir, objects, method calls and mutable data are easier to write than processes, messages and immutable data. It's not the mutable and immutable part, it's more about the boiler plate of spawning processes, receiving messages and matching them to dispatch them to the appropriate functions. Ruby's and Python's class and method definitions are much more compact and less error prone. Unfortunately they are not as good at parallelism. I wonder if it could be possible to have an OO language with an automatic Erlang process per object, automatic immutable data (just don't let reassign values to variables, like Erlang), and a way to define supervisor trees.
Is it possible to implement it using the metaprogramming features in Elixir?
That's a pretty big exclusion. There are times one really doesn't want type promotion from a particular number representation for performance or accuracy reasons, and if your dynamic typing system does not allow contracts to reject numerical types that will implicitly promote, you can run into huge performance problems.
An example: suppose you want to divide a list of numerators by a single denominator producing a new list. Each division is not guaranteed to produce an whole number. What's the type of the resulting list? What is the type of each member of the resulting list? What is the type of a sum of the resulting list? Is the per-division cost roughly constant or does it depend on the particular numerator, denominator pair?
Bear in mind not just differences between integer and real numbers, but that real numbers may be exact or inexact, and that there may be various representations of inexact numbers (from native floating point to various types of bigfloat).
Really strict typing -- whether in a dynamically typed language with contracts or similar mostly runtime mechanisms, or in a statically typed language with type checking at compile time -- forces you to deal with these questions explicitly. In this case, you might specify that the source list of divisors is a list of positive integers, and that each division produces a single floating point result (and thus you wind up with a list of floats; and you'd specify that the sum of the resulting list should be a float too). That sacrifices precision for performance, which will tend to matter depending on the content and length of the source list.
There are plenty of dynamically typed languages that get fiddly trying to avoid turning some or all of the operations into much more precise types than single float (or even doing exact representations, rational number style), and the performance impact can be dramatic.
Conversely, you may not want to lose precision as you move away from +-0.0f, so you may want to specify that exact arithmetic will be used in the operations in this example instead.
Erlang doesn't have contracts, but it has pattern matching. You can write a function in Erlang that is guaranteed to return only a list of integers let's say, by either converting them, or not accepting lists of floats in later part of your code. On top of this, you have Dialyzer which can warn you when your code doesn't do this (if you typed your functions correctly).
So it's not new. It doesn't make Lisp any less dynamically typed as a language, and it is wholly optional. Newer dynamic languages also let one do some partial or "gradual" typing where a programmer wants to use it.
There is a cost to this at function calling time, but then there is also a cost if one manually programs in a check on the type of an argument, for instance. Good compilers, however, can prove that functions that aren't of (or exported to) global scope will never be called with anything other than the specified types, and will omit the type-checking code.
Additionally, there is plenty of research into interoperation between code in statically typed languages and dynamically typed ones.
Naturally you can always convert back after your arithmetic operator produces the wrong type. But that can be expensive in itself, and it hurts more when the arithmetic operator could have performed a much cheaper operation.
A way around this of course is to eventually de-polymorphize the potentially-expensive operator and program in a hopefully cheap type-check by hand, in write-generically-first/optimize-(or even make correct)-after fashion.
I am also an Erlang noob, but I don't think pattern matching has to account for all parameters. To my understanding erlang pattern matching is very efficient, and anything you would pattern match on function parameters, you would have to do some type of logic in the function if you had no pattern matching, so I don't see how pattern matching would negatively effect performance.
ETS tables to share data between threads actually make a lot of sense if you think of your program with the idea that no matter what "x" is, this process should properly handle it, because you can't guarantee when the message you sent will be processed.
An ETS table is a DB, but you can replicate it by having a single genserver hold that data, and use calls to retrieve and manipulate it. It gives you one spot for your data to be. If you had that same data in messages or function call stacks, by the time the process executes it, it could be stale.
Its useful if you need one "true source" of individual pieces of data that needs to be synchronized. Its about decoupling data from processes.
What makes scaling up hard is moving data around. Once you have more than a single computer, there is no way you can easily share memory between them, so you have to impose some kind of copying for the system to work. If you want to demux a stream for multiple workers, you have to distribute work to the workers. If you have massive amounts of data in a cluster, you have to move the computation to the nodes in the cluster on which the data resides.
Moving data around requires you to have good orchestration of "mostly stateless" computations, with a couple of pinches of persistence strewn in as well. You can do this well in any language, but what makes Erlang well suited for it is that it provides some decent primitives for you with a lot of time sunk into the architecture. Beating this architecture in any other system requires you to spend some time doing that. And chances are it isn't as general, so when the world around you change, the framework you used is left behind.
Before Erlang, Tandem systems built hardware/software with many of the same ideas in them. They built these systems primarily for fault tolerance and robustness, but they found, somewhat to their surprise, the same architecture is good at scaling. The reason I believe, after 10 years of Erlang programming, is mostly that the computation model of isolated services forces you to think distribution into the system from day one. Your solution naturally gravitates toward the distributed model, and this in turn means it is easier to scale out later. The model also makes it hard to accidentally build a part of the system which can slow down everything. I think this should be given more credit than it is normally given.
And once you have your problem distributed, you call into that CUDA GPU code on the node to obtain the high computation speed. Or you call into your FPGA or DSP ASIC. Any problem on the CPU is slow because of its general purpose behavior (the exception: You are Fabrice Bellard)
Indeed, Jim-gray's (from tandem) paper 'why computers stop and what we can do about it' is an quite good. It contains a detailed report of machine failure including s/w and h/w and details techniques for reducing the mtbf by these.
Erlang's language and runtime seems to have picked seminal ideas from here...
For your convenience: http://www.hpl.hp.com/techreports/tandem/TR-85.7.pdf
When I learned a bit more about Erlang, I realized that the point is that there are a lot of applications that are IO bound rather than computation bound, and for those apps Erlang performs very well and is easy to program in. Crunching numbers the fastest isn't always the most important thing.
The other advantage is power of abstraction. With Erlang it is easy to describe and program distributed system with lots concurrent components. Sure you can do it with C++, Java, and Node.js etc, but those things are awkward there -- either threads share memory and have to deal mutexes, or you in are in callback/promise hell.
Because they are different problems. Concurrency, meet parallelism.
Concurrency: many smaller tasks that can be multiplexed over one core. The core doesn't need to be particularly fast; it just needs to be able to handle multiple tasks in-flight at once. Web serving, etc.
Parallelism: one big, honking task that can be split over multiple cores (or nodes), and each core needs all the horsepower you can eke out, so it can finish its part of the task quicker. Big Data, etc.
Erlang is not particularly fast (bad for parallelism), but it excels at concurrency.
> I know people do some big data stuff with Erlang
Not sure who is doing big data with Erlang, or why anyone would want to -- unless they have some very fundamental misconceptions about the problem at hand and the tools available.
> compromise some of the ideals espoused in this article to make it work
The article was really nonsense. Most of the terms it throws around have nothing to do with Erlang specifically, and proclamations like Erlang being used for a hypothetical "data center on a chip" are... what? The author's gist basically boils down to this: Use supervision trees with unikernels. I have had the same fantasy, for what it's worth.
Parallelism and Concurrency are not mutually exclusive terms. All parallelism is concurrent.
Concurrency is whenever more than one thing is happening at the same time conceptually. Everything from iterators to threads.
Parallelism is whenever more than one thing is happening at the same time physically. From a user-space perspective, this means threads or a coprocessor (like a GPU).
There are many, many ways to think about this difference, and it's fun (and beneficial!) to do so every once in a while.
> From a user-space perspective, this means threads
Interestingly enough, before SMP and multicore, thread-level "parallelism" was actually disguised by a time-sharing concurrent implementation. That remains true for most threading libraries in languages with a GIL, and any time you have threads exceeding the number of physical cores. In fact, the primary purpose of threads was (and still mostly is) to get a semblance of concurrency... Even the Erlang implementation was single-threaded 2008 or so.
It's the fact that they might run at the same time. Also from the perspective of the programmer, there's no difference between two threads running at the same time or by time sharing, since preemptive multitasking is non-deterministic and has the same implications as true parallelism.
> from the perspective of the programmer, there's no difference between two threads running at the same time or by time sharing
is incorrect when you're talking about computational parallelism (as OP was), because you're not going to realize any speedups with time sharing. In that case, you're using threads as a concurrency mechanism -- not for parallelism.
Riak?
This assumes a pedantic qualification of "big data" handling -- does storage count, as opposed to the actual processing of the data? But I think that's an important distinction in this context, as it strikes the heart of the dichotomy between concurrency and parallelism.
(Moreover, I've never really been sure how much Riak is being used for actual 'big data', as opposed to being a master-less, highly available repository for 'regular data' (distinction between 'big' and 'regular' deliberately left vague). More so since Riak is key-value as opposed to columnar, but I suppose that depends on your workload. But that's all besides the point here.)
Nokia created an open-source Hadoop-replacement called Disco [0] that used Erlang for coordination/orchestration -- an underappreciated strength of the language -- of map-reduce jobs, where the jobs were written in Python (and later OCaml, etc.). They've shown that it can handily outperform Hadoop (at least in the canonical wordcount example shown in this talk[1] -- there may be other examples, I haven't actually watched the talk yet). They've used it to mine terabytes of logs, daily, as described in this talk[2] and others apparently have used it as well.
From the abstract[3] describing the first talk, about the project:
We will describe our experiences using Erlang within Nokia to build Disco, a lean and flexible MapReduce framework for large-scale data analysis that scales to large clusters and is used in production at Nokia. Disco is an open-source project that was started in 2008 when attempts to use Hadoop to analyze data proved to be a painful experience. The MapReduce step formed only a portion of the analytics stack, and it was felt that it would be faster to write a custom implementation that would integrate well, than adapt Hadoop with the amount of internal Hadoop expertise available. Among the crucial tasks of such an implementation would be to deal with cluster monitoring, fault- tolerance, and the management and scheduling of a large number of concurrent and distributed jobs. To keep the implementation simple, the use of a platform that provided first-class support for distribution and concurrency was imperative. This motivated the choice of Erlang/OTP to implement the core control plane of Disco. It bears stressing that this choice was driven primarily by pragmatic concerns, as opposed to any beliefs about the superiority of functional programming languages in general or Erlang in particular.
The project's homepage [0] has information, a link to its Github, etc.
[1] https://youtu.be/IjOGUC-iR_Q
[3] http://cufp.org/2011/disco-using-erlang-implement-mapreduce-...
But Erlang isn't really for doing computation, it's for communication. If you have an existing erlang system that has a lot of data and you want to perform some computations on all of it, you'd probably use erlang to get the data onto the appropriate servers and then spawn another process, that can compute efficiently, with numpy or just C or similar.
Distributed systems aren't just a way of scaling performance beyond the number of CPUs you can have using the same memory. It's about reliability. Erlang's design enables you to create network services that have decades of uptime. It achieves this through its concurrency model, through its error handling approach, by allowing hotswapping code (an operation that's relatively easy to reason about without mutable state), and probably more ways that I as an outsider am not aware of.
[1] http://benchmarksgame.alioth.debian.org/u64q/measurements.ph...
[2] http://benchmarksgame.alioth.debian.org/u64q/measurements.ph...
[3] http://benchmarksgame.alioth.debian.org/u64q/measurements.ph...
Is Erlang too esoteric?
Also, the Phoenix Framework (if you're building a web app) is really, really nice.
Here in New York, it's not too hard to find Erlang enthusiasts, particularly ones that work on Wall Street.
That said, no one seemed to have even heard of the language in Dallas when I lived there.
So the long-story-short of this is that if you don't mind doing some remote-hiring, it's not too hard to find workers, and things like Ejabberd have a ton of tutorials on writing modules.
Do you hire people who know how to code in language X and are really good at coding in language X but nothing else?
Or, do you hire people who are go-getters, want to use the best tool for the job, and want to learn?
Because if the latter, Erlang/Elixir might be esoteric, but Elixir specifically is a simpler language than currently more popular languages like Python or Ruby. It's also more elegant. I like to think of it as Scheme + Ruby syntax + pattern matching + concurrency. If that excites you, carry on. If that scares you, go for something more traditional (and no hard feelings) :).
Erlang:
loop_through([H|T]) ->
io:format('~p~n', [H]),
loop_through(T);
ok.
Elixir:
def loop_through([h|t]) do
IO.inspect h
loop_through t
def loop_through([]) do
:ok
Erlang's syntax may be more efficient, but a lot of people won't touch it becuase the syntax is unfamiliar. Elixir has a huge advantage in this. It might not matter to you specifically, but in the realm of adoption that is huge.
And honestly I really enjoy the syntax...
Also, all the different import-related keywords are pretty confusing. Maybe that's improved though
I don't find either syntax to be off-putting but that's just me.
Enum.each(<the list>, &IO.inspect(&1))
<series of data transformations resulting in list> |> Enum.each(&IO.inspect(&1))
Plus there is an up-to-date package manager/global repo (https://hex.pm/) which is used by both Erlang and Elixir. Plus Rebar3, the primary build tool for Erlang projects, is actively developed.
But Elixir packages are very active in general since it's relatively new and these packages can be used within Erlang apps relatively easily as well. Plus the Erlang packages from 2013 that I've used have all been pretty stable. The quality of libraries available on Github have all been very high from my experience. I believe Erlang attracts experienced developers, which is reflected in the typical code quality and documentation.
My only wish is for the Erlang website (http://www.erlang.org/) to get a redesign. It gives the language an esoteric apperance to newbies. Which is a shame because I absolutely love the language and think it's far superior for many of the types of projects for which people have been using Node.
* Color scheme and fonts are easy to read.
* One can switch between API reference for the current module and Users Guide for its containing application with one click.
* On the left-hand-side one has a scrollable tree of modules in the current application, each of which is expandable to reveal the API calls within any given module.
* Sensible URL scheme: doc/man/$MODULE.html for a module's API docs and doc/apps/$APP/users_guide.html for an application's Users Guide.
One thing I've found I love are the PDF documentation downloads; I wasn't expecting much, but (for example, the xmerl one) they're great, step by step useful examples that I can shove on an ereader to go through. Really solid.
Erlang 18 brought maps, people complained about that for years. Erlang 18 also brought (via a feature flag) dirty schedulers so can have long running C function embedded in without messing up process scheduling also an often requested features. Probably the best thought-out handling of time in any language I've seen so far ( synchronization, warping, moving backwards: http://erlang.org/doc/apps/erts/time_correction.html )
Erlang 19 is exciting as well -- 10x faster tracing, dirty schedulers turned on by default, a new state machine OTP module, an external plugin (with LevelDB as one example) for Mnesia storage also something people complained, 2x-3x faster spawning external processes + many others.
The impressive part is that these changes are done to a 30 year old language.
I can't tell you how many times I've cursed at the overfilled-hours-ago-but-mnesia-didnt-care DETS table shard. Getting a on-disk backend that can store more than 2GB at a time will be great!
It'll be even better if the writer code actually notices failures to write to the backing store and aborts transactions when they happen! [0] :)
[0] Seriously, who thought it was a good idea to ignore the return value from dets:insert/2? :(
erlang isn't great because of a single feature or library, it's great because it's designed from the ground up for one very particular role. you can't just port the otp api to go or the jvm. you need the whole foundation
For example, I've never had a single HTTP request bring down an entire website -- that's already isolated. Same with message-queue listening processes. For general batch applications, I've always had them short-lived and running periodically, e.g. every minute, so even a complete crash there is isolated between runs.
One powerful aspect is how it strongly encourages you to design loosely-coupled message-passing systems that should be easier to scale out. But I'm not convinced that's enough to warrant a switch.
There's also some more serious stuff like your workers getting killed by the OS for whatever reason and you might need to go in and restart it.
You can do this through most queue-based systems in other languages but having everything be built-in is useful.
Once you start doing it, it becomes more apparent what the difference is... And that's not to say you can't design fault tolerant systems, it's just "easier" to do so with Erlang/BEAM.
That means that if you planning to provide proper availability you want to work with system of applications, not just one. That means you should look into creating networking architecture that can handle all of that.
Most of the time it means that people just use tools that solve that for you, like load balancers. But it doesn't mean that somehow all modern applications are immune to failures.
In a traditional architecture, you would use one thread for each connection (let's ignore the issue of the memory used by each thread), but when one thread fails, it would bring down all connections instead of just the failing one.
You can have continuously running processes without relying on something outside of the language. You can more easily distribute such code as an Elixir package. The code can work without relying on e.g. cron or redis being available and configured.
The fact that you program expecting failures, and forced isolation of everything allows for incredibly "sturdy" code. The other features are fantastic, but they're just gravy as far as I'm concerned.
http://www.phoenixframework.org/blog/the-road-to-2-million-w...
I think of it like big corps vs startups. Startups may have more efficient engineers and development cycles and produce higher quality work because of the selectivity of their employees, but the sheer amount of work a big Corp can do is significantly more.
(Just an example, not saying big corps don't have talent - far from it)
When designing your system you need to decide which model it needs to follow. There are places for both.
And Go/Java implementation can be trounced by hand written assembly and ASIC accelerators probably.
> All this stuff about multicore scaling is baloney
You say baloney I say money in the pocket. I've seen it scale, I've seen it work reliably in large clusters, I've been able to inspect, debug and hotpatch running systems while they are still running. I have seen systems which had non-critical components crash and auto-restart for days without impacting customers and needed teams of "devops" to babysit it.
Moreover I've see single and multi-threaded C++ and Java applications with threads and and data races which take weeks or months to find. Or they are screaming fast until they take a nosedive and segfault (also in some minor stupid new feature which nobody uses). You know what the transaction processing rate of a segfaulted process is? - 0 tps.
That's why teams like Whatsapp could get by with only 10 or so back-end engineers handling billions of messages / day from various devices new and old, while other companies need 10x or even 20x more than that.
It is not just being able to run fast. Assembly runs very fast. It is also about being able to have the right tools and abstraction to define a problem. Erlang has those and they come built-in (the OTP library, the distribution protocol etc), C++ doesn't, so have to start from STL and boost and so on, then get serialization, monitoring, supervision, etc bootstrapped.
You can run multiple apps on the same VM and they will run concurrently and communicate with each other using protocols that are core of the language.
I agree, erlang seems weird at first, simply because there is not anything like it. It also solved todays problems with software a decade ago.
If you're on the JVM, I'm not sure what Erlang buys you in practice. It's slower and more obscure. It has a much smaller ecosystem. While process-safety is frequently touted, in the real world this is a non-issue among non-issues. It's just not an actual thing. It's not like the JVM goes around Segfaulting all the time.
I'm totally sold on Actor Systems and think it's something more programmers should expose themselves to. I'm just not sure there's much of an argument for Erlang vs the JVM unless you're completely sold on the notion of process isolation for some reason.
http://erlang.org/doc/apps/jinterface/jinterface_users_guide...
Plus I can imagine accidental mutability really being a problem. In my Akka course much effort was spent on explaining how to avoid side effect of it (no pun intended).
You do learn not to close over local scopes in futures pretty quickly though.
A 3GB heap is pretty typical, but that's mostly just room for cache.
If you're writing web-apps, there's a ridiculous amount of opportunity for using Actors to good effect.
On the other hand I try to keep an eye on resources. I try not to throw millions of messages at an Actor. If I'm batch processing, I'm probably work-pulling, mapping and prefetching my Sources to attempt to minimize latency. Even if I could tune mailbox sizes to avoid all that I prefer not to code in upper bounds on input.
I would be surprised to hear this is a widespread problem in people's work. You don't often hear the same complaints from Rubyists for example and they're much worse off in that regard. So I feel like this case is massively overstated.
But if you have that problem I can certainly see how that might figure into your equation.
Erlang's GC model is just incredibly elegant. Almost like pooled allocators that we used to use in GameDev but in a completely transparent and intuitive way.
OTOH the JVM will have much higher throughput on the other 99% of apps.
Don't get me wrong. I like Erlang. It's definitely in my top.. Uhm... 3 languages of interest. Being the only other language I know with Pattern Matching is a huge huge part of that.
For most developers I'd just be careful in making that jump. Because most of what this article is about isn't Erlang specific. It's more like a pretty small club of platforms/languages. IMO.
FWIW a lot of game engines use message passing between components/actors due to the low coupling and architectural benefits so it's not quite as small of a domain as you think.
It's about the share-nothing architecture of Erlang: it means your processes are isolated and can be stopped/restarted independently, crash/hang without writing into the memory of another process, have their own garbage collection, and be moved to a different physical computer (or even data center) with no impact at the logical level. Erlang forces you to architecture your program as a collection of nano-services. You can of course emulate some of it on the JVM, but you can't go lazy and cut corners with Erlang.
Speaking of which, I just realized the AtomicLong I'm using in my IdGenerationActor (performs an atomic increment of a processId counter in the database, then uses that with the AtomicLong as the input to Hashids; fast, in-process, cluster-safe short-Id generation that will leave popular Redis based solutions in the dust) is completely unnecessary. Copied and pasted from non-Actor code without consideration.
I guess Actors still can't keep you from doing dumb things yet. ;-)
I was referring to "cheating", with the idea that you might pollute your Actors with... I dunno. Programmatic connection pooling for your database driver? Passing mutable messages around?
Both of those things would be very unusual in Actors written in Scala since pretty much everything is immutable by default. Seeing that sort of thing should at least raise some eyebrows.
As far as non-responsive, I've never seen that. But I do take care to use Futures where appropriate, and pipeTo. Maybe it's good habits, or maybe I'm just very lucky.
One of my first Akka projects is still running, still processing content it's notified of by Postgres through LISTEN/NOTIFY, still posting that content into Cloudant (basically a managed Lucene deployment in this case).
And it's been running since September 2014 without a restart AFAIK. Which would have never happened with a previous non-Actor solution we might have used if for no other reason that a network blip might detach the listener, whereas here the Actor just restarts.
My experience has been overwhelmingly positive. Despite doing the wrong thing occasionally.
If you have Actors that are hung, I guess the first thing to try is figuring out which one(s). After that, you could just schedule the supervisor to routinely PoisonPill them, and forcefully stop them after a grace period.
I'm not sure that OTP is going to help with this sort of issue either. You have an apparently blocking process that makes an Actor non-responsive. The fact that it's single threaded within the Actor and stalls it's mailbox is kind of the point of Actor systems AFAIK.
I guess one thing I feel like helps me is to keep my Actors small and doing one thing. It's hard to do too much damage when you only have a dozen lines of actual message handling. "One thing" is sometimes coordination/aggregation BTW. Which means I might have a GetRequestActor, PutRequestActor, DeleteRequestActor, etc. Which do the one thing. But then I also have a DatabaseActor that basically just coordinates/forwards for all those so you don't actually have to deal with them yourself.
And from there if I decided that no GetRequestActor should take longer than 10 seconds to do it's thing, I can easily schedule the DatabaseActor to forcefully stop any task exceeding it (without the need for an actual watch if you choose). Which you can then log an ERROR for and work out why. And maybe some requests just take longer and that's OK. So maybe you then write a LongRunningRequestPathExtractor and put that pattern before the normal one. And now you can have multiple timeouts for different paths.
Or maybe you just record the epoch for different requests, and if you've had 1,000 updates since the last View request, you know the next one is going to trigger a reindex. So you give it extra time. Or you set it to allow stale results and reschedule the same call to occur again in 1 minute to minimize the disruption of indexing the changes. Just some thoughts.
Well, the idea is with Erlang you can't.
> Erlang matters today because it demonstrates how these semantics can be elegantly packaged in one language, execution model, and virtual machine.
Why is it so important to demonstrate that these semantics can be packaged into such a homogeneous environment ? Is it even a good idea ? Is this way of doing things superior to having micro-services that talk to each, all managed by a supervisor system ? Wouldn't that allow us to take advantage of the unique upsides of multiple programming languages, virtual machines and execution models ?
Erlang happens to nail that part with OTP.
Text is here (CTRL-F erlang):
https://zwischenzugs.wordpress.com/2015/11/17/dockerconeu-20...
Video here:
This is a controversial statement, and an opinion that is not shared by many respected computer science researchers: http://research.cs.wisc.edu/multifacet/papers/tr2011-1_coher...
Does anyone working link to this article?
Go is more of a systems-ish language that has an emphasis on concurrency, but also on being relatively C-like. It's much more general-purpose than Erlang.
Erlang is more about fault-tolerant, concurrent applications, and more specifically servers. It's not designed to be especially fast, it's designed to make it very simple to write systems with a ton of throughput, that can handle outages, and perform passably in the process.
They're different things, and the only thing they have in common is that they both advertise concurrency, really.
You still don't have to guarantee of thread-safety, you still don't have the immutability, you still don't have tail-call optimization, and perhaps most importantly, you don't have a REPL for hot-code updates.
Erlangs primary difference over Go is that it can gracefully handle the case where a programmer has made an error in the program by accident. A typical Go program cannot gracefully recover from an error which were unforseen and never considered by the programmer. The Erlang equivalent Erlang program, however, has mechanisms to safely clean up resources for the faulty part and then get on solving work. This is the reason Erlang has a nice robustness story.
Supervisors for golang. It looks excellent. I personally have fallen for Elixir and think the syntax, immutability, community and class functionality like Phoenix.Presence make me bet that it'll be bigger than Python for jobs in 5 years.
In Go, your whole program is in consistency trouble and you have to write code to handle the case.
In Erlang, your monitor on B means you get told it is dead via an async exception delivered into your mailbox.
I know what kind of system I want to work with here :)
Is there a language that compiles to both Erlang and Javascript?
The foundation principles is not only in selecting a functional language and enforce immutability, but explicitly rejecting all sharing (threads in the first place) and providing a theoretical basis of why JVM is an unacceptable target for reliable, soft-realtime systems, (no matter what Scala guys might tell you) - a crashed process would affect none other, no data corruption, no locks, no messed up stack. This is what is behind the "let it crash" meme.
Erlang is not "matters", it is a masterpiece of software engineering to study and learn insights from. Especially, how to make ones own decisions based on right principles and rejecting sectarian dogmas (run everywhere!) of wast majority.