Hacker News Reader

Comment by akkartik | Hacker News Reader

This article got me to go figure out precisely what Erlang processes are. Tl;dr - they aren't OS processes. So it is still conceivable that an error in Erlang can bring down all your web servers.

http://stackoverflow.com/questions/2708033/technically-why-a...

View on HN

An error in the beam VM (or a nif) could lead to crash of the OS process, and related termination of all of the erlang processes within. However, the beam VM is pretty robust, because of the design, it turns out to be quite small and relatively simple[1], so you don't tend to end up with things were weird edge cases cause terrible crashes that are hard to find. For nifs you write yourself, it's up to you :)

Process isolation isn't complete though, one erlang process can use all of many of the limited resources and end up with the whole vm shutting down. A single process can't use all the CPU, but it can use all the memory, all the atoms, all the filedescriptors, all the ets tables, etc; basically everything in the system limits[2]

[1] Calling it simple is a compliment -- a normal person can look at the vm code and understand it, if they need to.

[2] http://erlang.org/doc/efficiency_guide/advanced.html#id70200

Thanks, yours was finally the comment I was hoping for.

Could you point me at the right place to read (or read about) the beam sources? I'm looking at https://github.com/erlang/otp/wiki/Routemap-source-tree -- is there something better?

You're in the right place; unfortunately there's not a lot of good documentation on the internals (although do check out erts/emulator/internal_doc ). It's much easier to dive into the source if you have something specific on your mind (like debugging a performance problem, or if you find an beam crash...). erts/emulator/beam has the most basic parts of the virtual machine -- processes, memory allocation, process switching, message passing, interfaces to the operating system (although a lot of that is in erts/emulator/drivers), etc.

> So it is still conceivable that an error in Erlang can bring down all your web servers.

No it won't. Not if you mean an error in an Erlang process. It almost sounds impossible, but that is the beauty of the BEAM virtual machine. It is really a marvel of engineering.

Now if you mean an error in Erlang VM itself, then yeah, if that crashes it will bring down the all the connections. But even that there is a answer -- distributed Erlang. Erlang VM can talk to processes on other Erlang VMs, even if those are running on another server on other side of the planet.

> ... So it is still conceivable that an error in Erlang...

are you talking about erlang-runtime here ? it is not very clear what you are implying. can you please elaborate ?

if i take your statement to mean 'erlang runtime' then this is no different than saying, if there is a bug in glibc which gets tickled when i do something, then my process will crash, or if there is a bug in jvm, then the application hosted on the jvm will crash etc. etc.

From OP: "In Erlang we create very lightweight processes, one per connection and within that process spin up a web server. So we might end up with a few million web-servers with one user each."

I don't see the difference between spawning a thread for each connection in other languages and spawning a process for each connection in Erlang, so I was curious if the word 'process' implied some extra level of insulation. Reading all the comments here it's still not clear to me what benefit is being obtained by having "2 million web servers" rather than 2 million connections.

Shared memory space is one thing. A process has isolated memory. An error in a spawned thread could mess up shared memory, crashing multiple threads. An error in a process can't mess with the memory space of other processes.

Since there's no shared memory, performance characteristics change. Garbage collection is done per process and so "stop the world" doesn't happen, meaning each individual "web server" isn't at the mercy of other requests for performance.

It's lots of little things like that.

> I don't see the difference between spawning a thread for each connection in other languages and spawning a process for each connection in Erlang,

That is a huge difference. Otherwise by now, why would Erlang not just run pthread calls and just spawn a thread. Why bother doing all that work?

The reason is because Erlang processes do not share heap memory. That is one of the most important features of Erlang runtime. A crashing process can just crash on its own without leaving all the other 1999999 processes in an undetermined state.

It also simplifies garbage collection. Rather than having to stop the world for garbage collection, it happens on a per-process basis.

Additionally, if a process terminates before a garbage collection cycle is run the VM just reclaims all of its memory without any additional computation needed.

I don't follow, how does Erlang processes not being OS processes relate to what could happen if an Erlang error occurred?

Well, not if you distribute your erlang processes across several physical nodes.

You mean just like you can with Apache or Nginx?

No. Not at all like those.

Can you please expand on why?

The processes can communicate with processes on other servers in exactly the same way that they can communicate with processes on the same server.

If you have one traditional application server and want to do some form of cross-user interaction - let's say chat - you can do that trivially, put it in a queue for that user in a global map. Now when you outgrow that server, you need to rewrite all your code to understand the concept of users being on other servers or use an external message queueing system.

In Erlang, all of this is built in by default - if you write code for the OTP framework (the standard library for dealing with messaging, process supervision, etc), all you need to do is connect the two servers together and point them at the same shared user->process mapping process (which you have to build whether you're dealing with one server or 20, as there's no global data otherwise).

Of course, if you have absolutely no direct interaction between users, it's trivial to scale anything - fire up a new server and direct some portion of your traffic at it. Erlang's trick is to make it that easy even when you do have direct interaction between users. And of course that works for even backend workloads - if you have a backend server that your frontend servers talk to and need to scale it, if you've coded in Erlang and put as much logic as possible in per-connection processes, you're probably a significant chunk of the way there.

What does any of these have to do with the equivalence between Apache/Ngix and Erlang when running multiple servers of them to avoid crashing all?

You can't use traditional servers to scale in lots of cases without a lot of additional development work. You can use Erlang servers to do so. Therefore, they're not the same - Erlang covers a broader range of use cases.

A constructive comment would be appreciated.

Yes - it's exactly like that. The Erlang VM itself has to be rock solid just like you expect Apache and Nginx to be rock solid.

Fortunately this more or less seems to be the case.

I would say a better comparison is the Linux kernel. If the kernel crashes, everything on that box goes.

The Erlang error in code could be replicated to every single physical node and all of them would crash.

That is not what happens.

Presumably one would catch such a blatant error early in testing. The problem is the odd case, the code that only crashes in a rare state, never seen in test. When the odd state does occur the effect is mild, as only one actor dies as opposed to the entire OS process.

I was responding to the assertion that distributing code to multiple "processes" would avoid crashing all of them, which is simply not true. Even the rare state you mentioned would crash all the processes. The rare state would be an input that testing failed to catch. The code in any of the processes encountering the rare input could crash.

Reply on news.ycombinator.com