How And Why We Switched from Erlang to Python (2011)
engineering.mixpanel.com
engineering.mixpanel.com
My recollection of events (of 8 years ago): The Erlang endpoint was written a year or two earlier as someone's first Erlang project, in a tiny org with no Erlang experience. The endpoint worked well enough and people moved onto other projects in the mostly Python codebase. Eventually, it became painful to debug or add features to this endpoint because no one was particularly proficient in Erlang. Ankur (author of the post, intern) then rewrote it in Python.
I wouldn't read this as a negative about Erlang or a positive article about Python. I'd instead read it as "use a language that makes sense for your organization and already existing codebase".
Source: https://web.archive.org/web/20121224094022/http://cufp.galoi...
So you let them rewrite something that already works. If what they build is good, you‘ve just knocked out a good chunk of technical debt and found someone you probably want to hire. If not, they don’t get an offer. Either way you probably didn’t pay them much to begin with so there’s a lot of upside and not much downside.
And, of course, a company that puts this kind of work on the back of a temporary worker is not treating its own codebase with the respect it deserves. This is rolling the dice with your expertise store -- you have to hope they did it well, because they aren't going to be around to answer questions if they didn't.
The point is this was a well-specified, independent component with an existing functional implementation. This makes an ideal intern/temp/first project, because of those factors - it doesn't require deep knowledge of the organization or other services in the environment or business requirements. You build this, and if we like it, we use it and may give you a job offer.
Unless given other information, you can assume the company complied with the law with respect to paying the intern and the intern accepted the internship offer, and it sounds like he got a great experience out of it.
Yeah, and that logic is almost inherently a violation of labor law. The whole idea behind allowing "internships" at all is that the intern is deriving value (education) from the relationship that isn't captured by wages alone. The test for whether it's legal involves how much they are supervised by the people who are supposed to be teaching them.
Handing out throwaway projects like you posit is an easy trip to a class action suit.
> Handing out throwaway projects like you posit is an easy trip to a class action suit.
So if an intern shouldn't do work on the core product but also shouldn't do throwaway projects, what do you suggest an intern should do? What do you consider an "underpaid" intern? Would you consider 75% salary of campus new hire as underpaid?
You ask the intern to do work that you want them to succeed at and you think they will succeed at: like the example given.
If you’re recruiting on campus, you’re competing with other companies for the most promising candidates. If you come in with an unpaid internship, you’re going to get the leftovers.
I live with an HR exec and read some of this thread to her while she was cutting rhubarb for something she's baking.
She said (paraphrased), "Are you kidding? This is a perfect project for a summer intern. They won't have to take three months to get up to speed with all your internal systems, and of course there is educational value - they will get to see how some talented developers tackled the problem in one language while they translate it to another."
Most kids right out of school are honestly pretty decent coders. The internship is more about the professional skills than the technical ones.
I was an intern back in 1995 and they paid us $10/hour ($16.88 inflation adjusted) and they provided housing on a college dorm. To put that in perspective, the $650 I made after taxes every two weeks was the cost of tuition every quarter at the state school I attended. I made enough that summer to cover my entire senior year including books.
Yes the project was impactful, but there already was a reference implementation with a reference performance target. Worst case scenario, they would have been back to using the old implementation.
You're also presupposing Python is inferior to Erlang, which I'm not really going to argue about here, but I think we can all agree that Python isn't so bad that the decision to continue to use Python substantially and negatively impacted the org's ability to succeed.
If it's a component that nobody owns, and nobody cares about, then I'm not very surprised.
>As someone who has started learning it, this seems like the obvious choice.
The other obvious choice is to rewrite it.
To me, the fact that a novice was able to deploy production code that went on unmaintained for months, running a crucial part of the operation, speaks positively about Erlang.
I didn't see anywhere in the article blaming the language - can you find a quote to support your assertion?
From the article: "After two years of iteration, the code has become difficult to maintain. No one on our team is an Erlang expert, and we have had trouble debugging downtime and performance problems. So, we decided to rewrite it in Python, the de-facto language at Mixpanel."
If you have some developer that wants to write something in a new (for your company) language, or even worse goes off and does so without asking, because he wants to try it out, you probably want to fire that guy. Either he doesn't understand the downsides or he does, but doesn't care.
It’s always worth trying a hypothesis in a small spike and reliably proving whether it will scale to the rest of the org and the impact will be worth the effort.
But you’re right, there should be one official supported way of doing some kind of thing at a company.
As an example, my current team's codebase is Python and C++. When we need to do some basic Linux scripting (check if this file exists, if not send an error email), the two prime candidates are Bash and Python. Bash might be exactly designed for this type of thing, but a lot of the team would need to Google stuff like "bash logical and of two booleans" for Bash where they already know the Python syntax. My current rule of thumb here is "<10 lines of code: use Bash. else use Python". More generally, I often try to err on the side of using the tool I know than the unknown tool that might be perfect for the problem.
It seems madness is pretty common.
For most projects, it's good enough.
Also, I worked for many shops that had a two language policy, one for development speed and one for performance.
For example, PHP and C or something.
They had many PHP devs and a few C devs and would write anything new in PHP and then later replace the few performance critical parts with C implementations.
I thought about doing this with JavaScript and Rust in my projects, but somehow I never got to the point that JavaScript became the bottle-neck.
If it’s out of ignorance, then sure by all means educate.
Now that's
> a cost with no benefit.
We teach, not react.
Of course C# you get all of the advantages of statically typed languages, so for major programs where you wil have multiple developers, it’s the go to language. But for smaller scripts like event based lambdas and scripting type scenarios, C# is overkill. Besides, each piece of AWS functionality is in its own NuGet package as opposed to one package for everything for JavaScript (Node) and Python.
JavaScript - our product is web based. Every developer is expected to know JS. So if someone decides to do a simple Microservice or lambda in C# or JS, it’s okay. Whichever they feel like doing, it’s okay.
We have another team that does a lot of ETL type processing. I doubt that many people would argue that JS is a better language for it than Python. If you are on that team and need to write something, you can choose whichever of the three languages you wish.
lol yeah that guy should totally be fired.
Writing initial version of the code is only a small part of it. Maintaining and enhancing it is much harder.
What is the company going to do if the new endpoint went down and the author is on vacation?
A dev ran up against the limitations of one platform and solved the problem by choosing a platform which didn't have those limitations. And management responded by banning the new platform.
How long do you think that dev is going to stick around in an environment like that?
Ultimately this comes down to what sort of environment you want to foster. One approach leads to an environment where devs have lots of freedom, and that creates certain problems. The other approach eventually leads to an environment where even the choice of editor / IDE is dictated, and that creates a different set of problems.
I'd favor attracting a bunch of smart folks and give them as much freedom as possible. I'd much rather deal with the kind of problems which that sort of environment creates.
Right now, I am responsible for maintaining one of our internal services. If someone tries to rewrite part of it in Clojure without extensive internal discussion first, I'll do all I can to stop it (and I am not "management" at all). And most people on the team will agree with me.
If the original dev would not want to stick around after that, good for them. If they do not care about teammates at all, they would likely be happier working alone anyway.
After all, single-handedly moving to a new technology and then forcing everyone else on the team to support it is impolite and annoying. Clojure is nice, but I don't want to be forced to learn it because the server went down on my shift.
(this is different if the service does not need to be up all the time, or the company is very small. But if you are talking about API endpoint being overloaded, the chances are, neither of this is true)
“single-handedly moving to a new technology and then forcing everyone else on the team to support it is impolite and annoying”
The dev you’re describing has indeed made some bad choices and I wouldn’t want to work with that guy either.
What I had in mind was more along the lines of “check out this thing I made, what do you think?”
Totally ok responses would include: “let’s just bypass the ORM here instead”, “let’s divert 5% of traffic to it and see what happens”, “this stays a prototype until you reach a bus factor of 2”, etc.
I see this a lot and I just don't get it. Is a mechanic who uses air tools trying to "prove how clever he is"?
Knowing more than one tool allows you to choose the right tool for the job. Assuming arrogance and a motive of "proving cleverness" sounds like a great way to pick a fight.
a mechanics' tools would be IDE, build tools etc- I'm not interested how every dev uses personal level tools.
But just writing a program with a new dev stack without thinking of the possible consequences is just plain unprofessional.
But if I give my car to mechanic for carb cleaning and tune-up, and get it back with fuel injection system and explanation "it is better anyway", I would be very unhappy.
Unfortunately, politically, the kind of shop which fires a dev for choosing another platform is unlikely to approve a PR which bypasses the ORM (maybe also a fireable offense!)
Yeah, introducing a new ecosystem to mask bad DB design or bad queries because she is lacking basic SQL and relational foundations is how I envision the typical 10x programmer.
What if it has the potential to be a much better tool for the job than the mainstream options, and your developer thinks it's worth trying it to find out?
I've seen this situation with adopting a relatively unusual language for a task that needed particularly high reliability. Yes, it was a concern that only one person currently working in the organisation knew much about it and that it might be harder to hire help if more developers were required later. On the other hand, that software was developed faster than the similar system it was due to supersede, and it had a much lower rate of reported bugs once deployed.
Of course you don't want to hire people who intend to pad their resumes with extra buzzwords on your dollar when it has no business benefit, but equally, if you never use tools that aren't chosen from among a small set of mainstream options, you'll never outperform what you can do with mainstream tools.
They'll still achieve their best possible results and performance with high quality tools of their own choosing, though.
My remark about firing a person merely wanting to write something in a different language was off base and end up derailing the discussion. But the point I should have made, and stick by, is that this decision is properly made at either the company wide level, or for large companies at the level several rungs up the ladder. It isn't at all, like someone suggested above, equivalent to picking a personal IDE.
Maybe, particularly if it's a smaller organisation.
In any case, it's obviously a management decision at some point, as all things are. However, it seems quite reasonable for me for a developer to advocate using a more suitable tool even if it's not an existing choice/standard within the organisation, and if they make a good case, it seems quite reasonable to me that management might then back them despite the downsides if the upsides appear greater.
I agree. It really increases the difficulty of understanding the whole system. But it's a negative that has to be considered alongside potential benefits.
> If you have some developer that wants to write something in a new (for your company) language, or even worse goes off and does so without asking, because he wants to try it out, you probably want to fire that guy.
I get what you're saying; resume-driven development can ruin your systems. But at some point between "we wrote this new banking system in COBOL" and "every developer who knows COBOL and could maintain our old system is now dead", there needs to be an accepted way to experiment.
We implemented its replacement in Elixir, a solution we are still running. It is written by a developer we still have, is well organized, follows proper functional paradigms, and fully leverages OTP/Genserver. It is resilient, and so far as the one two developers we have working in it at present, easy to maintain. We love it.
But we also are switching back to Python. The reason is simple: We only have two Elixir developers (really only 1.5), and we have far more work than they can complete. And when something breaks (usually because something changed in the various systems being integrated), we have one guy who can fix it, and get our production queues flowing again.
We have looked for more Elixir developers. They are rare beasts. But we have a lot of Python developers. And now that we have asyncio/eventlet/etc., the landscape has drastically changed. In Python, the various teams can create their own worker modules, and do so more efficiently because they know their own workloads far better.
Language diversity IS a problem when it impedes development and puts production at risk.
Was there any consideration of training up the Python developers on Elixir?
As the CIO, where the buck stops, I learned Elixir well enough to write my own Genserver-like implementation from scratch, and thoroughly enjoyed it. I wanted to be able to understand the code base. But I am still a far better Python developer than I will ever be an Elixir developer. It's all a question of time utilization.
Due to the modular nature of our integration, it's relatively easy to move from one platform to the other, one module-at-a-time. And in the end, the cost of getting a new Python-based platform up-and-running far outweighed the cost of trying to retrain Python devs.
But even if I didn't need the clarification, the additional data was awesome, so my thank you remain sincere :)
I'm a big fan of the Erlang platform, and I even kind of begrudgingly love the language, but Erlang didn't even have first-class immutable maps until 2015, and while I actually think the semi-prolog pattern-matching syntax is elegant and beautiful once you get a handle on it, I actually really dislike the whole comma-semicolon-period thing. (Personally, nowadays when I say I'm writing "Erlang", I'm actually writing LFE).
Also, the best webserver for Erlang is Cowboy (at least in my opinion), and the initial commit for that wasn't until 2011, and it wasn't really an awesome server until around ~2014. I'm not overly familiar with the Python ecosystem, but if I were to guess, the server frameworks probably were more mature than a lot of the Erlang ones.
I don't fully agree that it's difficult to benchmark Erlang, since it has a lot of awesome diagnostic tools available out of the box and moreover it even has stuff like caching and whatnot built-in with ETS, so I think that argument comes down to "we didn't really know what we're doing", but, you know, that's a valid enough reason to not use a platform if your job is to get something done. I wouldn't recommend most people write their new apps in Idris or Coq either, for no other reason than it is difficult to fully pick up and use those platforms easily.
That said, I do use LFE for a lot of personal projects. I don't mind it being niche-ish largely because Virding wrote it to be a very thin wrapper around vanilla Erlang, so as a result there's not a lot to mess up. With the exception of macros, there's really not many changes to the semantics from regular Erlang, meaning that the transformation to and from LFE is trivial. LFE mostly exists to keep the syntax consistent and cleaner, and while the language is kind of toy-ish, it almost doesn't matter.
Honestly, I'd be a lot more worried for something like Joxa or Clojerl, not because they aren't awesome projects (they are), but they don't have as direct of mappings to the BEAM semantics.
Also, Virding is very active on the #lfe Erlangers Slack, so typically when I have a question, it is resolved pretty quickly.
It's fair to switch away from a technology your team doesn't know well and doesn't want to learn. Maybe that should have been the title.
It's a favorite pet peeve of mine too that people often try to rationalize either their lack of experience with a platform or a broken architecture as a problem with the language in question. Though sometimes it's done because it helps paper over internal politics (e.g. lack of clout to be perceived to criticize past decisions) by blaming something external like a language or framework to justify architectural changes, so it's not always that these teams don't know what the actual problem is.
(Twitter's transition from Rails back in the day, always springs to mind as my goto example of this, because as much as the transition might well have been appropriate, as much as they claimed otherwise, the real problem was never Rails, but the fact they'd written a monolith instead of building a system where the message delivery was designed to be sharded; no language or framework choice could have saved them from a rewrite)
Do some languages handle this differently?
Isnt this based on how the code was written rather than a language natively sharding?
There are however, frameworks that enforce some level of "shardability". The Datastore (now Firestore) in AppEngine/Firebase comes to mind, in that your data is shared all over the place (the details are hidden from you), and you can't write un-indexed queries that don't scale well. The Datastore has some limitations that definitely seem awkward if you're coming from a relational DB world (e.g. no joins, limits on the kinds of inequality queries you can do, etc.), but these limitations are there specifically so Datastore can guarantee performant distributed queries.
Erlang encourages shardable solutions from OTP concepts being so strongly integrated into Erlang.
I used "system" and "program" there specifically. One of the key things erlang "forces".
Languages do matter.
When they were rewriting anyways, it's very much possibly that moving off Rails was the right choice anyway, but that was incidental to the far bigger problem of their broken architecture.
Well yeah, because you avoid me like I am an obsessive ex-girlfriend. :D
You're the last person I expected to reply! Very pleasant surprise.
> It's a favorite pet peeve of mine too that people often try to rationalize either their lack of experience with a platform or a broken architecture as a problem with the language in question.
I am the same. I get it, I understand the mechanisms leading to that but I still get irritated sometimes that people pretend their motivation is something more. I am quite OK with admitting: "No I don't want to learn Rust right now; I use Go and OCaml when I need fast(ish) native code and I am quite OK with Elixir doing everything else". I don't pretend that Rust is bad; I know it's very good but I am openly admitting that currently I am working on deepening my skills and not widening them.
> Though sometimes it's done because it helps paper over internal politics
Yep, doesn't help when the CEO and the CTO are old friends back from their Perl and bash and Slackware days and they don't want any newer technologies. I also get that motivation and maybe I'll become that conservative one day myself but in the meantime it's really weird to pretend that we had everything we ever needed in the older languages / OSes / frameworks. We definitely didn't. And things definitely improved hugely in the last several years. (We might need a new OS though, Linux feels kind of stuck.)
As for monoliths, it's a 50/50, we know it. There are many projects (I'd say most) that are served quite well as a monolith. And most projects never grow that big that they need the horizontal scaling anyway. But yeah, let's all pretend we're Google, it's kind of how it goes these days.
Hah. I'm equally slow to reply to most e-mails :-P
> and they don't want any newer technologies.
I don't think it's necessarily that often that it is that they don't want any newer technologies per se, but that it's easy to avoid technologies you don't personally see the benefits of, even if that is because you don't know them.
But I also see the reverse relatively regularly: People who push the new hotness because they want to play with them rather than for any real business reason, but who try to pretend otherwise. It's fine, if you want to try to rewrite something in a new language because you want to see if it brings benefits, cool, justify it as research.
I'd focus on the endless pretence of many people in HN and Reddit; they really love to downplay the clearly documented and unique advantages various pieces of tech have: Erlang's OTP and BEAM's preemptive scheduling and soft-real-time guarantees, Go's goroutines and channel primitives (which I dislike but I admit they are often useful), OCaml's amazing typing system that can and does prevent a ton of bugs by the mere virtue of your program compiling, Racket's ability to make a fully functioning DSL for practically every business need you might have, etc. ad infinitum.
What often rubs me the wrong way is the really strong maintenance of the illusion that "we can do what Erlang OTP does in C++" (yes you can, but do you have 20 years to reinvent that particular wheel?), or "Go can have as strong a typing as OCaml and Haskell" (not when most Go devs reach for the interface{} escape hatch every day, and not when you don't have algebraic data types with variants and maybes), or "C can be an extremely safe language if you know what you're doing" (lol don't even get me started on that one!).
People in the tech communities should know better -- that's my argument. People cannot just pretend Python is God's gospel while we all know they just don't want to learn anything else but Python.
This is a serious problem in most technical discussions which very quickly get hijacked away from the objective technical merits and into the murky territory of beliefs and prejudice.
Pretty sad. We the techies should be better than that.
Back in the day not many organizations survived on Ruby/Rails and many of them moved over to Erlang or Java just to deal with resource issues.
https://blog.chef.io/2013/02/15/the-making-of-erchef-the-che...
Not sure what is up with Ruby nowadays. JRuby was always an interesting option though.
It's really not productive when even minor version bumps of Rails back in the days of v3.2 all the way to v4.2 were a nightmare to execute, often impossible. You were stuck with a ton of quirks for years, likely all the way to your next job.
Rails was productive, but only compared to a ton of other lame half-frameworks at the time. Nowadays it isn't special by any measure.
---
So TL;DR: it doesn't really matter what Ruby or Rails are up to these days. Many people have moved on, the free lunch is over and the hype is gone. Which is a good thing: hype more often than not kills technology and doesn't enable it.
I didn't see anywhere in the article blaming Erlang as a language - it just said Python was better for mixpanel.
From the article: "After two years of iteration, the code has become difficult to maintain. No one on our team is an Erlang expert, and we have had trouble debugging downtime and performance problems. So, we decided to rewrite it in Python, the de-facto language at Mixpanel."
The impression I got from the article was that they only wanted similar performance (hence intern), and that they got a suitable result.
I think that recently a similar service GetSentry changed their ingress code from Python to rust presumably for performance and security (although I might be utterly incorrect, I can't remember why I think they are using rust).
2. Even for 2011, 1k rps with an average latency of 100ms seems laughably bad. I have to be missing something here.
Perhaps that has something to do with it
Don't ever choose python for performance. Its definitely fast enough for most things, but its just not the tool to leverage if you actually need performance.
Its wonderful if you just want to get shit done though, there are just so many good libraries around that you can use to get your project finished without committing several times the effort it takes in other languages
Also, note ratio of clients to server is 100:1. So 100ms latency almost has to be the maximally experienced: that single core node cannot spend more than 1ms per request on average, or it will fall behind.
The classic problem with these benchmarks are that they fall to what Gil Tene calls coordinated omission. I can park some of the clients for a while and process the other clients far quicker. It improves the req/s and if I'm just coming back to the parked requests within, say, 100ms, everything will look fine. But there is a lot of variance on response times.
We have a monorepo and I imported our entire commit history into Mixpanel - https://twitter.com/i0exception/status/1010663994435067904
As you can see, over the last 3 years, we've rewritten large parts of our infrastructure in golang. While we still use python for a lot of things, we felt that the type safety and concurrency primitives in go were a much better fit for writing some of our core services.
Adding 1k lines of code per day to a monorepo while keeping the project manageable is no easy feat.
Is it split into microservices? Any tips to tame the beast?
OTOH, if there is strong reason to think the tool is otherwise correct, finding resources to. enable the team to gain and/or borrow the knowledge they are currently lacking should be practical.
It's incredible companies don't invest in training enough, and this goes both for the companies who provide training (causing companies to avoid wasting money on it to begin with) and the ones who need training for their teams.
A senior I worked with mentioned how he learned OO when his company he worked at had everybody trained on it, and hasn't forgotten it since.
But now it's all the job of the degrees right. They'll totally cover everything your company needs to use every obscure tool you want to be used for pennies on the dollar.
Sometimes I have the impression that functional programmers dislike Go especially for having such an inelegant language design ;-)
Personally, I don't have hard feelings for either side: I love Go and like the functional style in general (I know a bit of Scheme). Erlang is one of my top 2 languages I want to learn (together with Rust).
How? We had a intern build it.
Why? Intern didn't know erlang.
... and neither did we.
So they have changed the language to suit the programmers instead of changing programmers to suit the language.
OR
>Go with a mainstream language that they teach every CS kid and has every googable question imaginable in SO, hire 1 experienced programmer to manage a bunch of post-college-kids.
There are lots of good reasons to pick a language with a strong community. Once in a while, I hear people recommend obscure languages(or up-and-coming), and I think- that is going to be expensive to maintain.
of course we can have someone learn and then teach to the rest of the team but why bother doing that when you can select another language where it's a lot easier to hire
I take this to mean that the new server implementation in Python was written in at most two weeks by a developer who was an intern at the time. I don't think I'd be easily convinced that best results could be obtained this way.
The original Erlang implementation could be flawed and buggy, but wouldn't a much better result be achieved by fixing its problems, rather than rewriting it from scratch to another language, that is not known for its efficiency or for being very good for networking, unlike Erlang?
In any case- is there ever a case where an intern can write the best possible system in two weeks? And why would you hand off a critical part of your system to the most junior member of the team? I mean, unless your team are all elite hackers from the higher echelons of computer science masters and the "most junior member" is one of the best programmers in the world, or so?
Apart from a particular cpu intensive task that would make just a single request perform badly, it seems ( from just reading about the language) that just following the guidelines ensures top performance.
Video description:
...What many developers don’t understand is that Erlang is
built on an architecture and within ecosystem
that contains many subtle security flaws.
One such set of flaws allows anyone with the ability to
interact with a remote Erlang node to compromise
that node by abusing the underlying BEAM Virtual Machine
and the services required to run Erlang...
* Looks like a deep issue in VM architecture
* It's not detectable at all
* Ericsson was informed 1 year prior to the talk, and their recommendation is to not expose nodes publicly
Speakers argument: Yes, but the threat still exists for another internal project to exploit it
* Speaker's belief: It doesn't look like it is going to be fixed (any time soon)
To me to me this sounds like a very serious issue - to the point that I have crossed off anything on Beam that - I wouldn't build/learn to build on, wouldn't trust (note: didn't say wouldn't use) another software that was built on it.
Overall, it seemed like a great platform that scaled upto a certain point with good guarantees on latency and throughput and resource utlitization. Not to mention, seems like (probably) only one platform that does pre-emptive scheduling. Heartbroken that after being marketed as "battle tested", this aspect of the VM/lang has gone under radar for so long.
The main idea behind clustering is resilience of homogeneous components, not between varied applications or security domains. I do find it a bit disingenuous from that perspective. However the video is at least right on a few points:
* if you have control of one node, you can effectively control the connected nodes (if any) as well
* you can dynamically update and load code on all nodes (by design)
* you can connect to other nodes as a hidden node, though it's still technically visible, but you need to specifically look for it and it only is visible to the node you connect to (so be sure to monitor this on all nodes)
* the protocol itself is not designed to be public facing as timing attacks and cookie guessing are problems. (Cookies in this context are meant to prevent mistaken cluster connections and aren't really a security feature.)
* the clustering is a fully connected mesh so a malicious user could DOS a network by rapidly rebooting a node to clog existing communication and burn through ephemeral ports for connections to use (really only applies to very large clusters but it's good to know how well you can handle spurious node membership changes)
Having said that though, undetectable is a lazy claim (there are many ways to address this depending on how you setup your nodes and which epmd implementation you use) and the requirements are to keep the interface/ports open to things not part of the cluster. It's also far from a hard requirement to use the distribution feature at all if you don't trust your ability to setup an isolated network. It'd be the same issue if you had some external untrusted etcd or consul node connect to the rest and start screwing with consensus with similar devastation.
Lastly, there are other ways to cluster Erlang nodes which don't necessarily inherit disterl's problems (disterl = what is built-in but it can be replaced or augmented). Libraries like Partisan look very promising.
(EDIT: formatting)
There’s an FAQ entry explaining the reasoning behind this - in short, making a distribution protocol that can work securely with untrusted nodes is very difficult and risky, so the more pragmatic choice is to just not support that use case.
https://github.com/erlang/otp/wiki/FAQ:-What-kind-of-patches...
There’s also an option within the release tool distillery to limit it to the local network.
Not to mention even if an attacker found your unsecured setup, they’d also need to know your “cookie”/key to do anything. It’s no different than leaving SSH login with password enabled on any server.