Erlang/OTP – The fastest, cheapest way to build reliable clusters of computers | Hacker News Reader

Erlang/OTP – The fastest, cheapest way to build reliable clusters of computers

erlangotp.com

104 pointschops45 comments

Just linking processes together is not enough to build a cluster. How do you allocate the physical resources, deal with network splits, package and deploy new code, monitor... There's a whole layer that's missing there that would surely be nice to have. The OTP framework could even be used to sandbox in LXC and start other type of applications.

Lots of that is a scoosh in Erlang, deploy, monitor, manage clusters. You have monitoring, cross-machine process linking, remote code loading all based on network transparency....

That is what LING (formerly erlang-on-xen) is here for: http://erlangonxen.org

That doesnt really solve all those problems in itself does it?

It's pretty cool but I don't think that it's open-source.

It seems to have an open-source, copyleft, non-GPL license [1].

[1] https://github.com/cloudozer/ling/blob/master/LICENSE

why, you can grab the source on github: https://github.com/cloudozer/ling

Network splits and many other multi machine distributed computing problems aren't addressed well by Erlang because it was designed mostly for distributed computing within a single machine (that might have several CPUs and/or blades).

Erlang will work spectacularly well in those scenarios and is explicitly aimed at them, definitely not just for 'distributed computing within a single machine' (which is nonsense anyway). The reason why it can't deal well with network splits is better explained by internalizing this:

http://en.wikipedia.org/wiki/CAP_theorem

If I remember correctly, Erlang was designed for fault-tolerant telephone switches and there were always two of them (the switches) in one box, because the system had to cope with even hardware failures.

So Erlang is pretty much designed to deal with systems distributed on different machines, it's nice behaviour on modern multicore machines is just a byproduct of that.

For an Erlang OTP page it doesn't cover much OTP :-) It doesn't even mention gen_server.

Gripe: OTP should be renamed. The name serves no purpose these days. Rename it "Erlang Design Patterns" and be done with it.

Ever try to tell someone about "Erlang/OTP?" They ask "What's OTP?" Then you begin a 30 minute speech about the history of Erlang and phone switches (agile web ninjas ain't got time for 80s telecom stories) and how all modern fad problems were solved 20 years ago.

Erlang is great, but the core Erlang devs don't understand "web things." See their arguments against why Erlang shouldn't support JSON: http://erlang.org/pipermail/erlang-questions/2014-March/0782...

The renaming question has been debated at length multiple times, more recently in: http://erlang.org/pipermail/erlang-questions/2014-February/t...

Here's what I tell people when I mention Erlang and then OTP: "OTP stands for 'Open Telecom Platform', but it's just badly named due to old corporate history at Ericsson. It's more of a general development framework that includes generic versions of design patterns to build pretty much any Erlang application."

I have never received questions after that about the name -- people get more interested by what it contains than what the initials stand for.

For the argument about 'core Erlang devs', the post you mention is:

1. not from a core Erlang dev

2. actually from the main Erlang developer of Cowboy, one of Erlang's most used web server/framework

3. Right that JSON is a pretty inefficient serialization format, full of ambiguities and with many undesirable limitations.

4. It's also right that JSON contains nothing regarding hypertext within its data type (ignoring JSON-schema) and is therefore a weird choice for the web and REST, where hypertext is fundamental.

> It's also right that JSON contains nothing regarding hypertext within its data type (ignoring JSON-schema) and is therefore a weird choice for the web and REST, where hypertext is fundamental.

It's easy to consume with Javascript, which is where most JSON probably ends up.

Ultimately though, something I've learned over the years is that you can either "go with the flow" and do things the way most people are, or you can try and do things your own way. Unless you're Microsoft or Google or something though, mostly, you don't have enough resources to do everything your own way, so you have to strike out on your own only where it really counts. Erlang does that in a number of places already, for sensible reasons - it's part of what makes the language so good at some things. However, fighting JSON is likely one of those Don Quixote battles that's not really going to get you much outside of some very specific contexts.

I don't disagree, but none of the comments given are otherwise false.

I've worked on real-time bidding software where the incoming protocol got changed from JSON to protobuffs. Just doing that made bandwidth costs go down majorly, CPU usage more than halved, and response times got cut.

When the question becomes "easy" vs. "good", always picking "easy" isn't very nice.

You were definitely working in a "very specific context"!

Also, I think that you have to consider how a company grows: JSON is simple to get started with. It's a good default - pretty much anyone can consume it very easily. As you explore your problem space and iterate and improve, you may realize it's not good enough. Fair enough - that's a good time to switch. But many people never reach that point, and will happily continue with JSON for a long time.

That's not invalid and I don't debate that. JSON is obviously ubiquitous. It's just a bad format. A typical case of worse is better.

https://github.com/johnezang/JSONKit has a few interesting corner cases, for example.

2600hz uses Erlang/OTP to build scalable, restful phone switches. Company was founded 3 years ago or so.

>Then you begin a 30 minute speech about the history of Erlang and phone switches

I don't do that. I'd want to do that like I'd want to give a 30 minute speech about what EMACS, UNIX, or GNU stands for. Instead, I'd just answer the question about what it was.

Yea, why understand your problem space and learn from history.

If only Erlangers spent more time on marketing and less time on building things!

Marketing has always been as important as making things.

These days, someone launches a new open source product with an insanely well designed web presence. This open source project has a better visual design than I could make in two months: http://www.serfdom.io (along with the related http://www.consul.io).

Erlang is still stuck in "my first bootstrap page" mode of marketing (which is an improvement over the old version which was My 12 Year Old Neighbor's First Webpage mode).

Erlang is designed to appeal to people who already like Erlang. It doesn't get all the new programmer hyperfad attention of things like Go, but people who know how to use Erlang properly do amazing things more reliably in shorter timeframes than other programmers even know how to think about.

Erlang/OTP is not a general-purpose programming environment. It solves a specific problem in a very specific way, that's not even necessarily the best way for everyone who has the problem.

Generally, for people who actually have problems that they need Erlang to solve, they know that Erlang exists. I don't think marketing it as the solution to end all solutions to agile web ninjas is a very good idea.

>Marketing has always been as important as making things.

This is post-modernism gone mad.

If you make something and nobody knows about it, did you actually make it?

Richard Hamming, May 1986: you should spend at least as much time in the polish and presentation as you did in the original research. Now at least 50% of the time must go for the presentation. It's a big, big number.

>If you make something and nobody knows about it, did you actually make it?

Of course I did. You seem to be confusing things that you use for things that you are trying to sell (e.g. if you are an an academic, you want to sell your research as valid and worthwhile.) I use tools that I built to do work, and I've never spent a moment trying to sell the tools, but I actually made them. Some things in this world aren't even open source - people use them to make money, and never reveal them to anyone. Erlang started as an internal project, and people had to bargain to get it released externally.

You're drawing a false equivalence between marketing and substance. Erlang is doing billions of dollars of business. I love to talk about it, but the fewer people that know it, the better for me professionally. People have no idea how easy it makes everything.

I wish they would spend more time on marketing (author of erlangotp.com here!)

I think it is a great name. It says this is reliable, old school and deep -- not questionable fad marketing fluff. But maybe it would help to do something like:

* eOTP

* iOTP

* openOTP

* OTPly

* bitOTP

* OTP2000

* Margay

* Marvelous Margay

* OTP Classic

Or Outlaw Techno Psychobitch http://www.gar1t.com/blog/erlang-the-movie-ii-the-sequel.htm...

It is a bit like SQL. Structured Query Language. I bet many don't even know the meaning. Or are there many unstructured query languages to compare it to...

I think OTP should mean a library that helps you build reliable software. Telecom is old and crusty but! that also means reliable. Phone system is surprisingly reliable compared to computer systems. You can play it off that legacy.

"See their arguments against why Erlang shouldn't support JSON"

Erlang supports JSON if you use a library[1], which is how pretty much every language other than javascript does it. Are you expecting native syntax or something?

[1] https://github.com/davisp/jiffy

The fact that OTP sounds like something less general than Erlang itself makes sense to me. If anyone with their own Big Architecture (e.g. Facebook, Amazon) they went all-in on Erlang, they would likely end up writing their own equivalent to OTP to work with that architecture. But the language itself would stay.

Correct (erlangotp site author here...)

The extent to which this person, Loïc Hoguin, is out of touch is frightening.

Hopefully he doesn't speak for the rest of the Erlang community.

Did it in response to asking this on HN https://news.ycombinator.com/item?id=7277797

I wanted to try an experiment in presenting Erlang/OTP in a different way from lang Vs lang. Wanted a "Hello World" that was building a cluster, cos you can't do that with Node, lol...

I appreciate the effort, but focusing on the multi-machine single-image clustering part would probably make more sense. Runing many processes on one machine is pretty easy and mainstream these days. Show me how to easily run code securely across a handful of dedicated machines (perhaps connected via ip6 and ipsec) - and you'll have my attention. ..

I'm still waiting on an update on dragonfly's ssi story now that there are no free and open ssi solutions for linux... i fondly remember having fun with open mosix...

> focusing on the multi-machine single-image clustering

Yeah, but I am trying to get people to build a "Hello World" cluster in 5 minutes - so single machine, multi-node is easier...

Wow, I haven't heard mosix in forever. Or beowulf cluster, for that matter...

The amount of times OTP is written out is unusual in and of itself. Dude was probably just excited to have snagged that domain.

Nope, that domain was in the bag for years (not by me though...)

I wrote the site, fire thoughts at me @gordonguthrie or gordon@vixo.com

Oh, yeah, in response to asking this question here https://news.ycombinator.com/item?id=7277797

How do you justify the claim of it being the fastest when at least some benchmarks put Akka at over twice as fast in terms of messages per second? Do you have your own comparisons that you're basing that on?

http://uberblo.gs/2011/12/scala-akka-and-erlang-actor-benchm...

It's almost like frameworks can't ever solve all of our problems.

Isnt Docker + etcd a higher level abstraction of erlang/OTP ?

Not even remotely.

Reply on news.ycombinator.com