Scaling to millions of simultaneous connections (2012) [pdf] | Hacker News Reader

Scaling to millions of simultaneous connections (2012) [pdf]

erlang-factory.com

147 pointspitchups81 comments

Thanks for sharing this.

Given the same problem, I'd look at a reactor pattern MQTT mosquitto like pubsub protocol in Go, Erlang or something JVM/CLR. The beauty of publish subscribe of many low bandwidth clients is that they need tiny bits low latency work in a sizable bandwidth app. Could run BINC/CBOR RFC 7409 over zeromq on lower-end infiniband between boxen. Millions of IM users are just similar enough to tons of smart meters that the solution overlap is enough, but obviously people are more bursty with jagged sinewave daily traffic loads.

Let the clients handle end-to-end encryption with something zk DH/OTR/SMP.

I'm curious how they later monitored this and did other common SRE stuff. Also deployments.

You could use also Apollo, ActiveMQ's successor, which comes with an MQTT adapter and runs on the JVM. Has the added benefit of supporting other messaging protocols as well. I'm not sure how well mosquitto would hold up, do you have any experience with it?

As an aside, MQTT has a constrained device optimized sibling MQTT-SN, which works over UDP and supports sleepy nodes, etc. Unfortunately that one's stuck in licensing hell and does not yet have good tooling, at least last time I checked.

My company selling Telecom-grade distributed MQTT broker and MQTT-SN gateway. Both written in Erlang/OTP and come with 1M concurrent clients SLA (was tested with even more).

I gave a talk about MQTT-SN Gateway at Erlang User Conference 2013:

http://www.erlang-factory.com/conference/ErlangUserConferenc...

http://www.erlang-factory.com/upload/presentations/807/ZviMQ...

>Let the clients handle end-to-end encryption with something zk DH/OTR/SMP.

ZeroMQ also has its own experimental elliptical-curve encryption protocol now: http://curvezmq.org/ They call it experimental, but I've haven't heard anything negative about the implementation from the cryptographers, and elliptical curve crypto is well-established (it's the basis for Bitcoin's key signing, for example). Also, it's closely based on Daniel J. Bernstein's CurveCP.

Truly impressive scaling and efficiency. This presentation is two years old. I'd love to hear how far they've come since. My limited experience dealing with many many orders of magnitude fewer users has found the same thing this presentation stresses, though: today's hardware is outrageously fast and powerful. The slowdowns are all contention contention contention.

An interesting historical note is that Facebook's chat system was originally written in Erlang (https://www.facebook.com/note.php?note_id=14218138919), before being replaced with a C++/Java system.

I am more interested in why they switched away from Erlang, that is the opposite of the usual story.

They had trouble finding/training enough Erlang engineers.

Tell me about it!

I used to work for a company that built it's main business product in Erlang. It worked well as product, but business wise hiring engineers was the hardest part. It wasn't helped that they are based in London, where a certain consultancy snatches pretty much anyone who lists Erlang in their CV.

It's hard to believe Facebook can't find Erlang engineers. May be their interview process got in the way. If you use standard algorithm/datastructures interview questions to hire Erlang devs you will have a hard time. All normal data structures in Erlang are immutable and algorithm complexity is a controversial topic there. Using Mnesia/ETS is a bit of a taboo for interviews (like using SQL on your coding task in a C/Java interview). Many Erlang devs would dismiss such questions as irrelevant and try to move on asap which is a big no-no in interviewing.

ETS/DETS are mutable hashtables FYI.

> ETS/DETS are mutable hashtables FYI.

An immutable disk persistence mechanism would be a strange thing.

Sorry, but any good programmer will know the conventional data structures and algorithms, regardless of whether they write programs in Erlang, C or some other language.

It's not about knowing, it's about relevance. These data structures do not exist in Erlang in the same way. So when they ask you to code a snippet to implement a HashMap in Erlang, that will be a pretty insane task.

That is a bad bias. "Data structures & algorithms" could be a footnote in the history of a developer, plus, which "data structures and algorithms" will you use to hire one? The ones relevant to a erlang dev are not the same to a C/embeded one.

I've never seen the point of these questions. Yes I learned half a dozen sorts in college. But in my professional career I have done sorting exactly one way: ORDER BY. Ask me something halfway relevant...

i was headhunted and interviewed for an erlang position (working on chat). i 'failed' a technical interview when my interviewer insisted i provide sample code in php, python or c++. i offered erlang, c, ruby or js and he dismissed me

Check these guys for example http://yuilop.com/intl/lead-server-core-developer-javaerlang... . Look at the challenges they have. These are well known classic problems that generally require mutable lookup tables among other things. Doing it in erlang "to be executed as fast as possible" is crazy.

So many companies and hiring managers get hiring wrong. This is a classic example.

Bob Ippolito answers: http://erlang.org/pipermail/erlang-questions/2014-February/0...

From http://sequoiacapital.tumblr.com/post/77211282835/four-numbe...

32. Even by the standards of the world’s best technology companies, WhatsApp runs lean. With only 32 engineers, one WhatsApp developer supports 14 million active users, a ratio unheard of in the industry. (WhatsApp’s support team is even smaller.) This L E G E N D A R Y crew has built a reliable, low-latency service that processes 50 billion messages every day across seven platforms using Erlang, an unusual but particularly well-suited choice. All that, while maintaining greater than 99.9% uptime, so users can rely on WhatsApp the way they depend on a dial-tone.

Is it really that unheard of ratio for the backend? Some of those developers must be client engineers, so their ratio is probably a wee bit better than 14 million to 1, but really the backend service is rather simple and shouldn't change much whether you have 5 million or 50 million users. Further when you control the client and the server, hiding downtime is stupidly easy.

The same is true of client engineers too I suppose. One developer putting out an app to 1 person is identical to putting one out for 100 million. The work is related to the number of clients, not the number of installs.

When the guy from Sequoia, who has the inside numbers on the top companies, says it is unheard of, its probably unheard of.

The backend service is rather simple for 50 billion messages a day? hmmm not so much, I'm thinking.

With the right architecture, it is largely a horizontal scaling problem.

I ran and scaled the Napster server to 80 million users. We pushed about 70 billion "index file" commands an hour alone at peak. We had a chat and messaging system that also pushed millions of messages an hour.

There were some pieces of data that had to live on all the servers (what user was connected to what server) which would have caused some problems eventually (though with how much memory you scan stick in today's machines, not for a very long time), but from experience, 10k really was the same as 80 million.

The code to do this was a relatively small amount of C++ code. Most of our problems with scaling had to do with bugs in Linux when it was younger and "line-speed" Cisco switches that would blow up just because you pushed a few million packets per second through them.

And Erlang is great and building and managing a horizontally scalable architecture - about building repeatable work units and co-ordinating them. So how many engineers did you have? What were your ratios?

The Napster production server engineering team was just me for most of it. Early on there were two other engineers, but one left for management and the other to start a company.

I'm personally not familiar with Erlang to know if it would have worked for us. A lot of our scaling was about being very efficient with memory (the whole thing ran on little Pentium 2 machines with 2 GB of RAM each and part of the protocol unfortunately relied on MD5 hashes of files for file transfer which took up most of the memory).

Wow, you worked on Napster? You mention the server was written in C++. Out of curiosity, do you know what the client was written in? I found a random forum post by an anonymous user that said it was written in Delphi.

The Windows client was written in C++ (MFC iirc).

The backend service is rather simple for 50 billion messages a day?

Yes and no. It's a solved problem (Tibco RV installations handle this volume without breaking a sweat and have done so for years). On the other hand, this is some impressive price:performance for doing it. Or was; $19Bn would buy you a hell of a lot of Tibco licenses...

Would something like golang even need lock-counting like BEAM ? I think the idiomatic way to work with golang is goroutines rather than mutex, etc. However, I'm not sure if any goroutine mailbox profiling tools are available.

> Would something like golang even need lock-counting like BEAM ?

BEAM locks are within the VM's implementation[0]: http://www.erlang.org/doc/apps/tools/lcnt_chapter.html. BEAM lock counting is about introspecting runtime state and capabilities.

I doubt Go's runtime is lockfree. The answer would thus be "yes, most likely".

> I think the idiomatic way to work with golang is goroutines rather than mutex

1. depends: http://stackoverflow.com/questions/10728863/how-to-lock-sync... https://code.google.com/p/go-wiki/wiki/MutexOrChannel

2. I doubt channels implementation is lock-free, though I could well be wrong

[0] and maybe in NIF[1] as well

[1] http://www.erlang.org/doc/tutorial/nif.html

You're right and they're not currently a goal:

"Non-goals: make channels completely lock-free (this would significantly complicate implementation and make it slower for common cases)"

http://talks.golang.org/2014/go1.3.slide#8

571k pkts/s is pretty respectable, probably nearing or at the bandwidth limits of their network interface. but i'd like to know how many new connections per second can they do (avg), and how long does it take to reach 2.8M connections? (if it takes a long time to open a connection, it could take a while to establish all those open connections, and a reset of all conns could create a long wait for new users to connect)

Could similar scaling be achieved with Scala & Akka or did I just make Erlang devs snicker.

If you want evidence that HN has a fetish for money, it's that the technical details of Whatsapp's success are only closely inspected now that gratuitous cash has been thrown around.

You may view this comment as too cynical for this particular topic, but the ridiculous surge in Whatsapp stories doesn't lie. And of course we should inspect the action of successful companies.

I am merely pointing at the crude vulgarity of the Valley's barometer for success.

It is quite well known that WhatsApp uses Erlang. I bet it must been discussed here already.

Perhaps among the Erlang community but I only found out as a result of digging into the company.

Would be great to have someone from WhatsApp speak at CUFP this year [1]. If anyone reading this can encourage them to do so (or put me in touch with someone one there), that would be great.

[1] http://cufp.org/

The Snabb Switch* people have been talking about it a lot on HN, and there was a link to their github the other day.

*My understanding is that it's a similarly architected system, but built on a Linux/luajit stack

Just because you haven't seen this many articles about WhatsApp before doesn't mean they haven't been closely inspected before.

This article is from March 2012. It has been posted twice before to HN to little impact.

http://hn.algolia.com/#!/story/forever/prefix/0/Whatsapp%20s...

(Correction: the presentation is from 2012; I don't know when it was posted online.)

https://news.ycombinator.com/item?id=3028547

I think the app may not be used much here so people were not expecting it to be bought for so much. Anecdotally I know people who use Line, WeChat, and GroupMe. I don't know anyone who uses Whatsapp.

If "here" means the US, then yeah. WhatsApp is replacing SMS with I guess pretty much the rest of the world.

WhatsApp is also not associated with the titillating "share nude pics" culture of SnapChat.

Wasn't the US also slow in adopting SMS? http://www.ft.com/cms/s/2/618c4f5c-37ac-11db-bc01-0000779e23...

Yup - I lived in NYC and Chicago from 2000 - 2006 - and as far as I can remember, I didn't see much usage of SMS at that time.

Is it really vulgar to desire money and take it as indicator of success, or are you holding people here to rather high standards?

>Is it really vulgar to desire money and take it as indicator of success

Yes? It is not only vulgar, it is utterly clueless.

Money is certainly not an indicator of true success, other than, of course, success of acquiring money. However, why is money vulgar. Sure, there are many vulgar things one can purchase with money. But money in and of it self isn't vulgar.

Many people here want to understand how WhatsApp managed to convince someone to pay so much money for it, perhaps so they can replicate this success and also become rich. What's so wrong with that?

> perhaps so they can replicate this success and also become rich. What's so wrong with that?

My theory: Perhaps because an ever growing population of tech professionals are disgusted with the race for more wealth?

I will be the first to admit, I'm an Elon Musk fanboy, although I think its justified. Instagram. Whatsapp. Messaging apps! Billions of dollars! For messaging! With that same amount of cash, Musk has delivered goods to an orbiting space station and has succeeded in changing the direction of transportation, something an entire auto industry wouldn't (not couldn't, wouldn't) do.

I could be wrong. Perhaps money is still the end goal for everyone. If its just to have money for money's sake, I feel sad for you. If its to pursue your passion, I think we can all empathize with that.

Musk had to make his fortune first...

I get making your fortune. My point was: Use the opportunity you get from it wisely.

Money for money sake is silly. You can't take it to your grave, and you can't enjoy it unless you spend it. But there are a lot of cool toys money can buy, like space rockets or cool cars and motorcycles. In my case money would buy me time for tinkering with unprofitable projects and building things with my hands, which I enjoy doing

That totally depends on what your goals in life are and is highly subjective.

Forgive me if I'm contemptuous of those whose subjective goal is the acquisition of money.

I want the freedom to do what I want, and in my case that involves a certain quantity of money. Not tons of it, but more than I have now.

There's paying the bills, and there's lust for percentages of 16 billion dollars.

And who are you to say what is the "correct" amount of money for people to desire?

This really wasn't the appropriate thread to hijack for this discussion anyway; I came here to talk about Erlang.

I'm not specifying an amount or a correctness, and I do not disagree that there is a line between means and end.

I'm saying that HN's attention span is dominated by dollar signs.

What do you expect? It's a startup-oriented hacker bulletin board run by a guy that suggests that hiring is obsolete and most nerds should be able to start a new venture and make a killing.

Even if it's overly optimistic, it's a pretty reasonable aspiration compared to the wage slavery most have to endure, so I fail to see why this is wrong.

There are plenty more of stories of failure studied at HN than successes like WhatsApp. That's actually more correlated with reality than the mainstream business media that focuses mostly on success.

I don't expect different, but pointing out that the technology sector is no less rotten than the financial sector is a pastime of mine.

Still waiting for the explanation of why it's rotten to want to create value through technology and seek reward for it.

One might argue WhatsApp isn't worth what was paid for, or that the money would be better spent elsewhere. But that's a rather subjective judgement. Spending $16 billion on, say, philanthropic causes, doesn't necessarily lead to desirable or sustainable effects. As with any scarce resource, it depends on how the money is managed.

I respect your position, though please also consider that not everyone views money as the end, but rather a means.

I don't care about money; I care about living an enjoyable life.

I do work hard and desire success; which means to have the liquidity to support the activities and lifestyle that I prefer. This requires money.

In other words, you don't want money, but you are forced to deal with it.

There's nothing vulgar or clueless with loving money and wanting more of it. Most startups got started thanks to people who have insane amounts of money and enjoy making more, this includes investors in startups that changed the world such as Google, Apple and Facebook. The world needs people who love money so they can fund people who love making stuff that put this money to good use. Simple as that.

So yeah, get off your high horse and please stop lecturing people with cheap and meaningless moral judgments.

>please stop lecturing people with cheap and meaningless moral judgments.

Ha. I'm not lecturing; I simply have no respect or consideration for your position.

You are not being lectured: you are encountering an attitude with which you disagree, and one that has no regard for your disagreement.

I didn't say I personally enjoy making money more than coding/building cool stuff. I just said it was ok if some people do. And yes, when you call something "vulgar and utterly clueless", you're definitely making moral judgments.

> consideration for your position

You have enough consideration to feel like posting a comment about how much you feel about it.

>And yes, when you call something "vulgar and utterly clueless", you're definitely making moral judgments.

I know! My point is merely that I wasn't lecturing.

>You have enough consideration to feel like posting a comment about how much you feel about it.

I suppose I enjoy throwing the moral imperative into sharp relief; making it clear just how much I disdain your vulgar acceptance of the love of wanting money.

That moral imperative alone is enough to carry weight in this thread. It has wandered in as mere sentiment without ever presenting an argument.

...Now, of course, I have strayed into lecturing you about the mechanics of this whole thing--but note I still have not lectured you with the moral judgment itself. I don't care about convincing you that money is vulgar. Sanctimony is enough for me.

You are quite right, though; I have spent enough time here.

> I suppose I enjoy throwing the moral imperative into sharp relief; making it clear just how much I disdain your vulgar acceptance of the love of wanting money.

Really? You disdain the love of wanting money? Looks like you have some personal issues, you're taking this way too personal. Get over yourself, some people love making money and there's nothing wrong with that. I tend to worry more about people who hate making money and have to rely on others, even if that means giving away their freedom or stealing.

Oh, I can't resist.

>You disdain the love of wanting money?

At what point has my absolute contempt for "loving money and wanting more of it" been unclear to you? Because you really should have clued in around when I said "Yes?".

>Looks like you have some personal issues, you're taking this way too personal.

This is banter while I fold my laundry.

>some people love making money and there's nothing wrong with that.

See, you just keep on repeating your disagreement with me as if it's going to change anything. Moral imperatives are, well, imperative, you know? I'm just going to reiterate: there is something wrong with that.

I'll rephrase you: some people hate the love of money and there's nothing wrong with that, or at the very least, you can't do anything about it.

You're wasting your own time and giving me more opportunity to bash greed, which is, as I said, making the folding of my laundry more enjoyable.

But I guess it might bother you that someone might think your moral compass is off center because of your position on money.

I'm fine with people hating money, but people hating the love of money? Sounds like personal issues to me. I don't hate hating money as I don't have a personal grudge against people who hate money but it sounds like you do with people who love money. Oh, and I used to think greed was bad too, like when I was 12 but since then I've figured every single one of us act out of greed all the time and waiting for a greedless messiah is naive at best https://www.youtube.com/watch?v=RWsx1X8PV_A

> Yes? It is not only vulgar, it is utterly clueless.

I don't have the knowledge or experience to say otherwise, so if someone were to ask me if WhatsApp was a success, $16B seems like a good reason to say yes. Are you implying that WhatsApp's sale was a failure case for their owners, and success for them would have looked differently? (reject the sale and continue growth of the platform?)

Please. Money is a measure of value, plain and simple. If a lot of smart people thought WhatsApp was worth 14B then it's probably worth taking a second look at what WhatsApp did to get to this point.

This seems to me to be an excellent example of the kind of poor thinking money promotes; do prove me wrong.

>If a lot of smart people thought WhatsApp was worth 14B

Why are they smart? Because they have billions of dollars? Because Whatsapp is valuable? Because Whatsapp was less useful before the smart people bought it?

Are you incapable of viewing the transaction of 16 billion as a possible utter mistake? As financial hand-waving assigning a liquid value to shares whose value might well be inflated?

Money is a poor measure of value. Plain and simple. Are you drawn to the volume of the transaction because you think it signifies true success?

> it's probably worth taking a second look at what WhatsApp did to get to this point.

Something I never contested and agreed with explicitly.

http://en.wikipedia.org/wiki/Time_Warner

> Money is a poor measure of value. Plain and simple. Are you drawn to the volume of the transaction because you think it signifies true success?

Money is just a number. I can't quite tell if you're suggesting that value can't be quantified, or if you believe we could quantify value some other way that would be more functional than capital markets. Do Tell.

Value cannot be adequately quantified. At its best, money is an inadequate substitute for such a quantification that is often useful; at its worst, the failure to be mindful of that inadequacy exacerbates its inaccuracy.

I disagree. If a food charity enhanced their distribution system in a way that allowed them to feed an additional million people internationally I think we'd be just as interested in what they changed in order to meet their goals. We're interested in any significant success. As it stands though, for any for-profit company, money is the primary measure of success.

I agree that money is not the only indicator, but how else do you propose that we direct our collective attention?

Reply on news.ycombinator.com