Comment by josh2600 | Hacker News Reader

Comment by josh2600 | Hacker News Reader

Parent Full thread

This happened on Facebook's chat system too. For some reason, and I know no other language where this is true to quite this degree (except maybe lisp), but Erlang has a tendency for people to believe they understand how to build an app using it when in reality they can do nothing of the sort (at least in Erlang. It's common for folks to 'do it in Erlang' because its seen as different and somewhat elitist. People think of Erlang engineers as 'hardcore'; I shudder to attribute skill to a language rather than a person, but I digress...).

In the case of FB, as told to me, a bunch of folks in the valley said "of course we know Erlang!" and built the app but when push came to shove the only hardcore Erlang guys were in Poland and that made working on chat difficult. I'm not sure how much of that is truth, but its a recurring theme in Erlang projects, one that I've personally heard time and again.

For some reason, Erlang inspires confidence, perhaps even foolhardiness.

Side note: I work at an Erlang shop and our lead engineers speak at Erlang Factory every year. It's a great language and quite powerful but it can be difficult to understand.

So I heard a bit differently about the FB Chat thing. Facebook likes to move its engineers around to different teams from time to time (if only temporarily) to spread institutional knowledge around and prevent bus factors from being too low. This also is a way to alleviate burnout. There simply weren't enough engineers at FB who knew Erlang to make this practical for chat, so they rewrote it in C++. The Erlang bits were reasonably written and working fine, but it didn't fit into this policy. Any non-mainstream languages wouldn't fit, this isn't something about Erlang in particular.

Erlang is a great fit for a number of problem domains, but if you silo it into a small part of your organization, you're going to run into issues such as these. You have to be willing to train people in it. It isn't that hard to pick up if it's someone's full time job to do so, as long as they're smart. And you want to hire smart people anyway, right?

FWIW, the Erlang bits (a rather small chunk of code considering its scale) took over a year to effectively replace and led to numerous chat SEVs as people learned the hard way that their c++ replacement was missing some little bit of "seldom used but very nice to have when things get dicey" component or behavior in OTP. Given how people move around projects the difficulty of maintaining an Erlang team for just chat is hard to justify, but the move to c++ for chat back-end service says more about FB internal structure than it does about Erlang.

I think the truth is that whenever you migrate a code base, it's trivial to get 90% of the functionality quickly, but it turns out that things you thought of as unimportant, or edge use cases, are actually not rare events at all. I agree that maintaining an internal Erlang team for one piece of code isn't smart, but I tend to think that in light of the move to broader media services (chat, video and voice streaming) perhaps it might make sense to look at the language once again.

Maybe this time they'll do it in Go.

Erlang has gotten some hype about how "amazingly scalable" it is. The reality is that, as with anything, it takes work to make it scale, and once things get Really Big, it gets difficult.

My sneaking suspicion is that, in some cases, people pick it because they read that it's "web scale" or something like that without really knowing the details, and when things fall apart, they revert to something that they know better.

BTW, reference for the FB chat thing being rewritten?

JHeiliger was talking about it at an interview I went to discussing his years at Facebook.

On the note of Erlang's complexity, it's not so much complex as it is different. I think all languages get hard at scale, Erlang just has some nice native concurrent processing capabilities that are making more sense with each passing day as we move towards more cores in processors. It's easier to Parallelize Erlang than say C++ for example.

Reply on news.ycombinator.com