Comment by dhab | Hacker News Reader

Comment by dhab | Hacker News Reader

Elixir, which runs on the Beam VM, had piqued my interest for a while, so finally set out to learn about it this weekend. On searching for "flaws" of the language and the platform, I (accidentally) ran into: https://www.youtube.com/watch?v=42k70Y-yTYY . (year 2017)

Video description:

  ...What many developers don’t understand is that Erlang is 
  built on an architecture and within ecosystem 
  that contains many subtle security flaws. 
  One such set of flaws allows anyone with the ability to 
  interact with a remote Erlang node to compromise 
  that node by abusing the underlying BEAM Virtual Machine 
  and the services required to run Erlang...

My notes:

* Looks like a deep issue in VM architecture

* It's not detectable at all

* Ericsson was informed 1 year prior to the talk, and their recommendation is to not expose nodes publicly

Speakers argument: Yes, but the threat still exists for another internal project to exploit it

* Speaker's belief: It doesn't look like it is going to be fixed (any time soon)

To me to me this sounds like a very serious issue - to the point that I have crossed off anything on Beam that - I wouldn't build/learn to build on, wouldn't trust (note: didn't say wouldn't use) another software that was built on it.

Overall, it seemed like a great platform that scaled upto a certain point with good guarantees on latency and throughput and resource utlitization. Not to mention, seems like (probably) only one platform that does pre-emptive scheduling. Heartbroken that after being marketed as "battle tested", this aspect of the VM/lang has gone under radar for so long.

It seems to lack any context for the claims. It is true that the distributed Erlang protocol is not something you'd want to expose to other parts of your system with different privilege levels, but it's entirely possible to restrict what connects and how (if at all, it is an optional part of the system).

The main idea behind clustering is resilience of homogeneous components, not between varied applications or security domains. I do find it a bit disingenuous from that perspective. However the video is at least right on a few points:

* if you have control of one node, you can effectively control the connected nodes (if any) as well

* you can dynamically update and load code on all nodes (by design)

* you can connect to other nodes as a hidden node, though it's still technically visible, but you need to specifically look for it and it only is visible to the node you connect to (so be sure to monitor this on all nodes)

* the protocol itself is not designed to be public facing as timing attacks and cookie guessing are problems. (Cookies in this context are meant to prevent mistaken cluster connections and aren't really a security feature.)

* the clustering is a fully connected mesh so a malicious user could DOS a network by rapidly rebooting a node to clog existing communication and burn through ephemeral ports for connections to use (really only applies to very large clusters but it's good to know how well you can handle spurious node membership changes)

Having said that though, undetectable is a lazy claim (there are many ways to address this depending on how you setup your nodes and which epmd implementation you use) and the requirements are to keep the interface/ports open to things not part of the cluster. It's also far from a hard requirement to use the distribution feature at all if you don't trust your ability to setup an isolated network. It'd be the same issue if you had some external untrusted etcd or consul node connect to the rest and start screwing with consensus with similar devastation.

Lastly, there are other ways to cluster Erlang nodes which don't necessarily inherit disterl's problems (disterl = what is built-in but it can be replaced or augmented). Libraries like Partisan look very promising.

(EDIT: formatting)

Erlang distribution is just not meant to be used over the Internet, both for security and network reliability reasons. It’s something you would use on a restricted LAN. It’s also entirely optional.

There’s an FAQ entry explaining the reasoning behind this - in short, making a distribution protocol that can work securely with untrusted nodes is very difficult and risky, so the more pragmatic choice is to just not support that use case.

https://github.com/erlang/otp/wiki/FAQ:-What-kind-of-patches...

This is an issue that’s overblown, simply have a firewall that exposes used ports and you’re fine.

There’s also an option within the release tool distillery to limit it to the local network.

Not to mention even if an attacker found your unsecured setup, they’d also need to know your “cookie”/key to do anything. It’s no different than leaving SSH login with password enabled on any server.

The speaker claims that the cookie/key can be known by causing VM crash, which is done by exploiting the monotonic nature of cookie values which are stored as atoms which has a fixed cap on how many they can be, which when exceeded causes VM to crash. He estimates the time to do that be 10 mins (I forgot the exact memory limit)

That’s definitely an issue, but again can be easily mitigated with a simple firewall that doesn’t allow every port to be open to the world.

Reply on news.ycombinator.com