Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD users [video] | Hacker News Reader

Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD users [video]

youtube.com

60 pointsadamnemecek47 comments

To me, the more fascinating fact is that Erlang enabled their massive ability to scale.

http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...

The "miracle" of Erlang/OTP is that it is well-researched and based on proper concepts, so there is absolutely no usual bullshitting here.

The crucial mantra by Joe Armstrong is "to model a behavior in a "parallel" real world one have to choice a proper abstraction (which is about granularity, not classes or objects), then everything follows".

So they have selected a connection as a fundamental abstractions and a lightweight process as a unit of granularity. Everything else follows, indeed.

Processes share nothing, so there is no locks, semaphores, race-conditions. There is even no loops - only recursive procedures, which could be restarted if crashed, or even its code could be replaced without stopping the service (why, a new copy of the code would be called "in a next recursive call", because it is stateless/pure-functional).

Each lightweight Erlang process serves its own connection, so it is perfectly natural and efficient to block/sleep on it (the OS kernel will do the job). It is not a "busy waiting", and there are no callbacks either. No "event buses", no nonsense.

There is no threads, so no shared state or even shared stack. If a process crashes (why should it? - it communicates with outside world only via message-passing) it cannot corrupt or lock any shared data, etc. The whole class of problems which comes form a flawed concept of pthreads does not arise.

There is no "reactive asynchronous event processing (written in Java or Scala)", there is no non-blocking I/O (btw, the only real non-blocking I/O could be done by an OS - aio_read/write and friends). There is no meaningless over-abstraction. That is why it works so well. Mo magic.

Not just Erlang/BEAM optimization, but the team's background in high performance, C-based systems on FreeBSD enabled them to achieve massive scale.

What's more fascinating is that erlang underpins many more staples of the internet than most geeks even know. WhatsApp is just one 19B example.

Yep. Especially when you consider that things like RabbitMQ, Riak, ejabberd, etc. are written in Erlang and take advantage of OTP.

Not only the Internet.

A few German game studios are using it on their backends.

Physics in Erlang? Game scripting engine implemented in Erlang? What types of games are we talking about here?

I wrote backend. Physics and game scripting belong on the client side, usually.

http://de.slideshare.net/wooga/erlang-the-big-switch-in-soci...

http://sdtimes.com/enterprises-can-learn-from-game-developme...

This feels like mixing of levels to me. The Internet traffic of nearly every user will run through a Cisco or Juniper router. Does that make us all IOS or JunOS users?

It will and if Junos or ios stop working so will the internet to a large extent. It speaks to reliability and scalability aspects.

You could say the same about erlang for example, it used by WhatsApp and in many internet-cell network gateways.

Netflix is the other FreeBSD based service that many people dont know about. While the control plane is the much hyped Linux on AWS stuff, the actual streams are FreeBSD boxes in ISPs[1].

[1] https://openconnect.itp.netflix.com/software/index.html

There's also Juniper, EMC2, Apple, Yahoo (back in the day, at least), Sony PlayStation and many, many others.

I would only really count Juniper in that list, building on and with FreeBSD. Apple take some bits but their product is not really FreeBSD (alas) and they dont seem to give back. AFAIK EMC and Yahoo are ex-users now, or largely.

EMC Isilon still uses FreeBSD. There are employees who are also FreeBSD developers. They're working hard on giving back changes so they can jump to a much newer FreeBSD release.

Yahoo's FreeBSD isn't dead; YBSD! still exists too.

Also HN :-)

Since Whatsapp doesn't have client facing web page, but only api's used by the client apps, are their scalability requirements different from those that have a web interface as well?

messages are probably much smaller, but more numerous and latency has to be 'realtime', but also client-server-client juggling. so, to me this means they need to respond faster, more often, and deal with more sharing than a website.

I found the pragmatism and sense of deferring until the least responsible moment—especially the speaker's view of what that moment is—very interesting.

In the Q&A "What do you use for configuration management?" "<slight laugh> `make`", going on to say they don't have enough node types or a large enough cluster to do automation yet.

Not sure the title is correct. After all, the people use WhatsApp through Linux (Android), Symbian or whatever and not FreeBSD.

The servers were (are?) FreeBSD.

The servers are FreeBSD 9.x. After Facebook acquisition they've tried migrating it all to Linux but quickly discovered that Linux cant cope with that amount of traffic and they decided the FreeBSD is the tool to be used here. That lead to 'famous' Facebook job ad looking for Linux kernel engineers to help them bring the Linux TCP/IP capabilities in par with FreeBSD. Currently they're testing FreeBSD 10.1-RELEASE for their next platform OS upgrade.

Do you have any idea why Facebook would rather develop a TCP/IP Stack for Linux to match FreeBSD's, instead of further evolving FreeBSD's stack? Licensing issues?

While I have no actual knowledge why they made that decision, its not very hard to guess, and the possible answers are around their own staff familiarity with FreeBSD, the pros of having the platform simplified by using single OS (Linux in their case, since for sure they've more servers with Linux already than they've acquired with WhatsApp) and all possible security and financial implications following these factors. However, it seems like at least for know they might be staying with FreeBSD a bit longer, what's a great PR/recognition win for the FreeBSD itself. I really doubt its for 'licencing issues' because (religious wars aside) the BSD license is way more attractive to businesses than GPL one.

Said all that - I might be terribly wrong about everything I said, because its not an insider knowledge ;)

How could licensing issues be a problem?

I think people prefer a monoculture, even though the security idea of having multiple software stacks is appealing, thoroughly testing issues on two OS stacks is a lot more work, especially at Facebook scale.

Presumably they are tied into Linux for a lot of their internal infrastructure.

The main security threat these days isn't viruses or worms so a monoculture makes no real difference.

In fact running multiple operating systems is more likely to lead to a misconfiguration issue which allows an attacker into your network.

If someone getting into your network is your threat model then, well, you have already been owned. If you want to have resilient services where you can swap them out if there are security issues, a monoculture is not a win.

Facebook is almost certainly further evolving Linux's existing TCP/IP stack, not developing a new one.

I would rather guess it is for consistency reasons. People are more familiar with Linux than FreeBSD.

Similarly one of the reasons Facebook relies on PHP is because a lot of developers are familiar with it.

I know lots of people who are more familiar with FreeBSD who are working on Linux systems. The Linux monoculture is just massively dominant.

Pretty telling technical story and also flattering for the FreeBSD team.

This is not true. Facebook has never attempted to migrate them to Linux. If WhatsApp wants to convert to Linux, it is their decision. More than likely they will stay on FreeBSD as that is what they know how to run and a good way to kill a product is to force them to do things instead of focusing on building their product.

http://bsd.slashdot.org/story/14/08/06/1731218/facebook-seek...

A job posting in August indicates a failed migration attempt that couldn't have started until October, because that's when the transaction closed?

> Linux cant cope with that amount of traffic

I like FreeBSD but I don't think we can say that Linux can't cope with that amount of traffic while Google and Facebook use Linux. There is a missing piece here that's not about network traffic.

I think it is amount of traffic per server. WhatsApp didnt have a lot of servers and maybe reengineering for massive scale out on Linux doesnt make sense versus staying with what works.

Are there more details available about this? What specifically about the Linux TCP/IP stack is weaker than BSD's stack? Where does the weakness lie?

It doesn't have to be a weaker stack per se. It is also a lot about knowledge. WhatsApp vested much time into understanding FreeBSD. It may be if you vested the equal amount of time on Linux with knowledgable people, then you would see roughly the same performance.

If I should take a guess, the advantage of the FreeBSD stack is that it is simpler and easier to understand. Hence, it is easier to tune for performance.

Yes, of course, and Whatsapp users care very much about what's going on behind the scenes ;-) I'm not a mainframe user because I book flights, or a Windows user because i tried Bing.

It takes very little sense of humor to detect the fact that the title is a nerdy joke ;)

Correct me if I'm wrong, but one of the biggest scaleups in FreeBSD seemed to be zero-copy sockets. This should be solved in Linux when kdbus is turned on (replacing the current IPC : D-Bus).

just throwing myself in here: Starve [1] also runs on FreeBSD.

[1] https://www.starve.io

if one HAD to choose a non-erlang stack (for whatever reason), what would you choose ?

Go, Nodejs or Java - I'm mainly concerned about clustering, because I think that is what gave the Whatsapp devs their ability to scale(without having to do too much re-engineering).

Or can the clustering be moved to something like the queuing layer ?

Java, of course.

There are lots of JVM implementations to choose from, many with high quality code generation (both AOT and JIT) and GC implementations.

It has VM and cluster monitoring tools only comparable by .NET offerings.

The majority of big data solutions are built on top of it.

There are lots of libraries to choose from.

If Java 8, the language, isn't your thing, there are lots of modern languages to choose from.

I wsa hoping you wouldnt say that :(

Java high performance clustering is... intimidating. It is too enterprisey. From my (little) research, it seems that one would need to build this ground up on top of Netty or Undertow.

Care to elaborate? Load balancing to a bunch of http servers, sticky sessions, or store the session to a db, or the best option is no session data, memcached, its all pretty straight forward?

Showing FreeBSD/Erlang stuff that just works and cope with substantial amounts of traffic is not politically correct, because it will cause a severe pain of cognitive dissonance to the Docker/JVM/Node/Cloud crowds..

Lets chant together: JVM is the best VM in the world with millions of man-hours invested in optimizations. Servlet/JSP are mature, battle-tested, enterprise-grade, type-safe solutions. NodeJS is the only way to develop high-scalable internet services. Everything must be a containerized micro-service, orchestrated by Docker/CoreOS/systemd.

It is worse than that, where you would offshore development like on the other cases? :)

Reply on news.ycombinator.com