whatsapp being a closed system I don't understand how this disparity is possible.
Having extremely talented engineers and a rather small scope is what allowed them to grow til 1B users, with a minimum staff. Yet people seem to focus on the technology, because that's more easily replicated, and easier to accept, in my opinion.
I think you are correct. It also works as some sort of advertisement. Others feel like they made the right tech choices, if they choose the same tech as WhatsApp or Spotify. Completely forgetting that they probably need to be somewhat proficient in Erlang to get the benefits from the language etc.
I guess that "hire extremely good developers" is less sexy than just choose Rust/Erlang etc...
In that way you only attract people who do sit down and learn new languages by themselves in their spare time, which filters out people with less interest in programming at least.
But if you choose technology for its intrinsic merits you attract engineers with taste, who have the drive and luxury to care about their work on a different level.
There are plenty more people who can't be arsed to or don't have the time or simply don't find it valuable enough to adopt "niche" technology, despite technical merits.
I don't think it has anything to do with smarts, but rather with priorities and human nature. Most people follow the mainstream because they favor stability and convention, those who break out in different directions favor autonomy and freedom.
The choice of Erlang was crucial: it's basically built for communications and has all pesky issues like version updating, resilience to failure and scalability already taken care of by nature of its architecture and framework.
Of course, these 50 engineers were smart, most in that specialised space are, but I strongly doubt that they could have achieved the reliability and scalability of what made the success of WhatsApp with a PHP or Ruby back-end without more complexity, more ressouces and more hardware (like what FB and Twitter had to go through).
You're right that Erlang was/is the right choice for WhatsApp, and it was most likely picked as the language of choice, because of the smart people working there. It's the same with FreeBSD, a failing startup isn't going to be able to layoff 50% of its engineers just by switching from Linux.
I’m pretty sure they hired Erlang developers to dig into ejabberd internals and optimize certain things. They didn’t just decide to become an Erlang shop out of the blue.
It wasn’t Erlang that was the initial right choice, it was using xmpp and ejabberd that was the root reason. Erlang just happened to be a consequence of that.
I will contest your attribution of ‘smart’ with Erlang. These types of correlations are generally bias fitting. It justifies ‘smart’ being correlated with any and all niche languages, eg ‘so and so likes Haskell, so they must be smart’. No good.
It’s better to attribute ‘smart’ with the pragmatic decision they made to simply use a pre-existing chat server solution that already has the capability to scale. Harder to assess this as smart since there’s no ‘signaling’ here, you have to objectively assess if it was the right tech (which it seems like it was). Way less vanity in this assessment as opposed to what I already pointed out, how your Haskell or Rust devs must be particularly smarter, as opposed to say PHP or JavaScript devs who are considered dumber. I don’t buy it, I need to see more than just your affiliations.
So, I reject your initial post contending ‘The technology stack isn't even that relevant.’ It was precisely the tech decisions that mattered, and the right people to make such decisions. Chicken and egg scenario, I’ll concede that.
In any case, one does not simply pick a old chat server written in Erlang out of the blue - this decision was critical. How many over-funded tech teams would try to do this from scratch in Go? Plenty, and that whole team would easily be full of ‘smart’ people.
Eh, I was there since October 2011, and we didn't hire any people who knew Erlang until much later. All of the early server engineers (including me) learned it on the job. By the time I left in 2019, I think we hired two people to the server team with previous Erlang experience; it's hard to find people with it, and while it might have been nice, it's not important.
It's possible we had some consulting possibly before I was hired, but I don't remember seeing any evidence of that; OTOH, I do remember setting up and working with a FreeBSD consultant and Moxie when he was consulting on end to end. From my understanding, when things started bottlenecking, Rick Reed was hired to fix bottlenecks; which he had been doing at Yahoo! for many years and had worked with Jan and Brian there.
FWIW; WhatsApp the service started as just a text status, built on PHP and MySQL, but people were using it to chat, so the founders went looking for a chat server to use rather than building one from scratch. I don't know the decision process, but ejabberd was then powering Facebook chat at the time. (Of course, Facebook abandoned Erlang, they said because they couldn't find people with Erlang experience to hire)
Anyway, by the time I got there, I was told that the chat server had been mostly refactored over time and while a lot of names remained the same, and some of the basic architecture was the same, it wasn't ejabberd anymore. Mostly I worked on things that weren't chatd, and I don't think I've seen ejabberd code, so I can't verify, but it seems likely, as we customized the protocol, auth, offline messaging, contacts, session handling, etc.
Erlang is a tremendously right fit for a chat server, and hot code loading is almost necessary when you have hundreds of thousands or millions of connections per chat machine and want to push small changes. Of course, changes to BEAM itself, or the OS kernel take restarts, so you still need to do those from time to time.
Author of the article here.
Totally agreed. Apologies for that!
The article isn't really "an article" but it's an email newsletter I send out. I'm quite surprised to see it on the frontpage of HN lol.
So what you're reading is one of my emails.
It's meant to be more of a summary with additional links to sources where people can read more if they're interested (emailing people long-form articles doesn't work well as a content format from what I've seen - even ignoring the length limit that email has).
I'll add in some more links for additional reading (like the high scalability post) to give some more detail for people interested.
Thanks a lot for the feedback.
It's something I think we have all seen a lot of times - that by the time a company is serving 1 billion users it has quickly expanded out it's engineering team hugely, and because of that additional abstraction is required and the complexity/LOC skyrockets.
But makes little sense (as a developer/engineer myself) to think that growth in users, requires a ton of new developers/engineers. We are not tattoo artists, our job can scale indefinitely if set up properly, i.e. all you should need to scale further (to 1BN or 7BN) should be enough money to buy more hardware; which WhatsApp/Facebook clearly has.
True in this instance, but far from a universal truth.
"this team of 4 engineers is responsible for formatting the date of a message"
"this team of 7 engineers is responsible for the overall formatting of a chat"
"this team of 5 engineers is responsible for the formatting of the non-chat pars of the application, settings, profile page"
"this team of 4 ux persons is responsible for aligning the non-technical parts of formatting and the user experience over all parts of the application"
"this crossfunctional team of 5 is responsible for creating a framework to let the configuration of formatting be disconnected from the actual implementation of the formatting"
"this team of 7 QA engineers will aid in manual verification of changes and bugfixes but will also automate test cases for formatting in the entire application"
"this supporting team of 4 will develop automation tools and enable the formatting teams to collaborate in a high speed agile context"