WhatsApp scaled to 1B users with only 50 engineers

Amen to that!

Off loading the founders vision to someone else is a failure of both communication and org structure (at the seed-SB stage). People may not think hiring their first PdM is offloading their vision, but it is. That person is going to come up with their own ideas, validate their own assumptions, and push for product updates that directly serve their own self interests. To think that this won’t happen is naive.

What I’ve found works the best is hiring competent engineers, giving them adequate business context, a decent design spec, and letting them figure it out.

For example Google had thousands of engineers back when they had only 3 products: search, auction, and AdWords. I remember Travelocity having over a thousand developers and not being able to redesign a webpage.

Its deeper than a product problem. Its an execution problem. It’s the ability to do what works with great criticality opposed to what’s expected, standard, normal, or comforting.

They had a year fee of $1. Which I never paid yet I used their service anyways. The reason they were able to scale so large with so few devs is simple, their product was super simple. 50 tech people to build a messaging system is actually a lot.

Doing one thing good is a thing in the past. Venture capital has pushed basically all start-ups into 100-feature shops

WA apps are not better than the competition. Basic stuff like a way to disable automatic downloading of media for groups you are in is still missing. No desktop app. The web frontend is still crap. Outsourcing account registration to mobile phone companies is ridiculous (same as Signal) -- in most of the world, it makes it pretty much impossible to preserve your privacy while using WA. Endless nagging about plain text cloud backups makes WA E2E encryption completely worthless, since even if you opt-out, 99% of your contacts will have them enabled.

The impressive part, considering the size of their team, was their backend stuff and scale.

Discord, with similar team size, was able to build something that succeeded in displacing IRC. (Their Android app was laughably bad for the longest time as well.)

I tried to convince the CEO of Research In Motion, who I worked for, that WhatsApp would end up destroying his business. He thought I was insane for thinking something so small and simple could have such a big effect on his big, important company. Yeah, well, whatever you say, man.

Then, at that new upstart, someone asks "What else can we bolt on to this thing to make it really grow??"

I think of it in terms of divergence and convergence. Divergent thinking is great. If you think of an army on the march, leaders will be thinking about possible paths and sending out scouts. But when the scouts (experiments) come back with reports (results) they don't just send some soldiers along every path that looks promising. They winnow the options down, converging their thinking.

Bad leaders don't have the discipline for that. And I think a lot of them pick up that habit from sales and/or seeking investment. "What will it do? What won't it do!" That's great for manipulating the feelings of the person you're currently talking to so they'll sign the deal. But it's terrible for running an effective, focused product development organization.

I know I've seen companies die like that. And it's probably way more than we know, because as WhatsApp demonstrates, focusing hard on what's working is a great way to succeed well enough that people study and talk about you later.

This is the key mistake. Having teams of 2-3 with a manager each is way too many managers. If you have a clear focus, small teams are great because they can work autonomously, with little management and good results. You can't hire your way out of coordination hell.

I was at WhatsApp between 2011 and 2019; for the most part early on, we had essentially the CEO who does a mix of product management and engineering management and engineering, an actively engineering server manager (who also did some client work), and a client engineering manager. Android team grew enough to have an engineering manager and we also ended up with one product manager. Post aquisition, staff grew and so did management.

Part 1: By definition: more engineers = more ideas.

Part 2: Not so much.

On the one hand, mediocre engineers are sometimes sequestered into small teams where they can't interfere with the important projects that all the smart people and other resources are being thrown at. They just sit there and churn out crappy ideas and crappy non-vital tools. Like daily indicators.

On the other hand, I can see that as more of a tautology: ideas come from one person and are grown by a team. So be definition a quality idea comes from one person, and turns into a team.

Much better are the folks with a solid vision who can drive folks to execute on it, successfully.

I wrote something similar [1] about its success not long ago on HN,

>WhatApp grew because it was the only IM that worked on Smartphone. Thanks to Erlang. Yes. ICQ didn't bother. Microsoft refuse to accept Smartphone is the future of computing hence there never was MSN on Smartphone. Other contender tried but none of them managed to Scale. I still remember all my colleagues and friends used to download instant messenger on our phone to test them out, creating a group and basically DDoS each other until the system or network crash. No IM was able to handle the traffic load, not only WhatsApp did it many times better than their competitors, they also had almost all platform support, from Symbian to Blackberry.

There is another part of the story that dont get much appreciation, is how they managed 2 million connection per box on a what is now a relatively slow server. The latest M1 Max would have been a lot faster. I often wonder if they could have pushed to 10 Million connection per box with same latency under current CPU and Memory. ( Not saying it is a good idea. )

[1] https://news.ycombinator.com/item?id=25984924

1,000 times this.

I get scared every time that senior leadership pushes a new member on the team, because I know that overall quality will drop; their view is completely related to how much things they want to done on every cycle, and they don't care about quality (until it blows up in their face)

The best way I can call it is the impedence mismatch between what business people imagine and envision, and what the real products, their code, their systems are actually doing and are actually designed to do.

It creates a kind of cost explosion in the velocity needed and it is often solved by throwing more developers at it instead of rethinking the actual design of the products so that they can better grow to support additional use cases and scale.

You can't totally call it a hack, but it kind of is. A system was designed in a way that doesn't scale to the user demand and to the additional features or the new abstractions that it wants to grow into. Instead of rethinking holistically about how that system should be re-designed to now encompass all of this new stuff, and then put the work into changing it from its current model to that new one. What people do is they break up the current system in two, so now you have two service where you had one before. And now they hire a new team of engineers, manager, PM, etc. to start working on this new half. This repeats again where each of those two half are split in half again, hiring a new team, now there are four teams and four service, etc.

As that happens, you can no longer redesign the way the systems fundamentally work, so each team accrues more and more complicated workarounds on their individual pieces and all new feature becomes cross team projects where you now need someone to coordinate the work between all teams for each little project.

Once you start down this path, there is no way out, the only solution becomes throwing even more teams at it, breaking things down again and again into smaller pieces with a whole team behind it as each piece becomes more and more complex to all work around the inherent holistic limitations that were never reconsidered.

It's a vicious cycle, the piece cannot grow in it's current state because it is too complex, break it up, have two teams tackle each half, now they can take the complexity until they themselves add more stuff to it which makes them too complex once more, break it again, add another team, rinse and repeat.

The issue is often that complexity is accidental, due to the design in place and the tech dept accrued till now, as opposed to being inherent to the domain problem. But instead of addressing the accidental complexity, it is simply broken down in two so we can throw more people behind taming it.

Some companies spend a lot of time & energy trying to implement operating models that try to overcome this but actually end up putting in place more decision processes & oversight. Sometimes the real solution is to reduce control and let the efficient teams lead the way. But then what would all that management do to fill their days...

From my experience in a deadline-constrained environment, things end up being slapped together until they "just work". Adding new features gets more and more difficult and if you're still constrained by deadlines, then the solution is to hire more, which is effectively a brute-force solution to meet the deadlines. Output per employee gets smaller, but total output should grow by some marginal amount.

I think that's one of the conclusion of the "mythical man-month" book.

Often inversely correlated...

I quit after a few years, and about 3-4 years later they went on a hiring spree. Tripled the number of developers. Refreshed some products, discontinued some, and some went into maintenance mode. They still had around the same 10 websites, but 12 developers.

The products didn't get better. The products didn't get made any faster. They didn't have any less bugs. My wife actually works there now, and when she tells me about the issues they have I laugh a little bit.

It's all the same problems we had back then. Invites don't work. Teams doesn't work. Uploading profile pictures has issues. Entering data doesn't work. Registration is broken.

In my experience this is the hardest part. Can't count how many times my team started documenting and then slowly, for one or the other reason, the documentation became outdated.

The cycle is: Meetings for communication -> becomes too cumbersome and time consuming -> implement policy to document things -> everyone start documenting (for a month or two) -> docs are not up-to-date because people forget/leave etc.. -> go back to meetings

This is ~our approach, but it seemed to stop after ~150 employees or so.

(You also seem to have the time to write the docs…)

The only way I've seen documentation get the priority it needs is if management mandates it as part of the release package on which developers' performance evaluations are based.

Huh? They're all million/billionaires now. That part of the plan seems to have worked.

Sending messages to a known (small) number of people is a solved problem for a long time. Email for example.

With WhatsApp, it is even easier than email because WhatsApp is a closed system, and you can optimize.

They used an uncommon setup? Okay cool. How does this help me? or any other company?

I'm not sure what I should learn from this story?

The only thing is: Feature creep is bad.

That's not the case. Consider the "one tick" messages - they must be on a server somewhere. One tick means they've left your phone but aren't yet on (all) the recipients' phones

Sure. And so do I. Specifically, I don't want to rebuild systems that have already been built for me which I can use as dependencies. And I don't want to spend all my time troubleshooting bugs, either - I'll take a well-maintained repository on GitHub with thousands of users bug-testing it for me over some janky thing some guy on another team threw together a few quarters ago before he quit.

You answered your own question. Because it's "exotic" and "hyped".

But the sort of people choosing a JS stack are quite quick to move on. Make a mess, slap a new achievement on CV and dash.

That doesn't mean it can't have issues like anything else. And because of the massive scale of the language you'll see a lot of shit. It's a language and ecosystem like anything else - it's what you do with it that matters.

Regardless which language you pick, there will always be these kinds of issues.

Docker itself can provide a stable backbone that allows you to run repeatable and predictable builds.

So the same cannot be said for "newer" languages like Rust, Go, and other's whose ecosystems and paradigms aren't completely fleshed out yet?

This comment reeks of someone who is looking from the outside in when it comes to building stuff in JS. Most comments I see criticizing JS stacks are so superficial and demeaning that's its obvious the commenters have little to no experience working in JS.

See whilst obviously the JS ecosystem is quite volatile and can be trend-based, I think there are benefits - the language breeds innovation. Innovation doesn't always make the correct choices, and some tech-conservativism is important in many types of companies. But if our ancestors had just been lazy and just stuck with what was known and good, we'd never even had computers in the first place.

One is that the future is unknown. Your goal might be to keep the product simple and scale to 1bn users, but you don't really know if it will play out like this.

Maybe you plateau at 1m users with the initial idea and pivot the product somehow. Now the app does dating, social media or specialised communications for highway construction teams.

Tradeoffs are easier to make in hindsight. Whatsapp is a good demonstration of what you can gain with good trade offs. Do less but well. Whatsapp did trade some things off though. Their web version came late, is feature poor and a little clunky. Same for a lot of features, compared to similar apps ATT. IDK what exactly can be traced to the stack, but their decisions around user identity are similar. Phone numbers only. One device at a time. They gained simplicity, but lost options.

It's easy to see the benefits in hindsight. Real time is harder. Play that game again and it might turn out different. Most apps that set out to have 1bn never get close. At this point, scalability doesn't matter and tradeoffs made for maximum scalability don't seem as wise.

True, but one can do simpler things with JavaScript, too.

And a word on exotic tech: Well, some of it is really good in helping small dev shops achieve scale of development, deployment, and maintenance.

Don't we all, I guess we all crave that snap, click, build -feeling Lego gave us. But in my experience building a system means debugging all kinds of weird problems. At least, when I start something, I usually for a long time feel like I'm just hopping from weird problem to weird problem. But, TBH, those problems don't seem weird anymore as experience grows.

You know that JavaScript developers don't see JavaScript as weird or exotic, right?

It's something I think we have all seen a lot of times - that by the time a company is serving 1 billion users it has quickly expanded out it's engineering team hugely, and because of that additional abstraction is required and the complexity/LOC skyrockets.

Having extremely talented engineers and a rather small scope is what allowed them to grow til 1B users, with a minimum staff. Yet people seem to focus on the technology, because that's more easily replicated, and easier to accept, in my opinion.

Author of the article here.

Totally agreed. Apologies for that!

The article isn't really "an article" but it's an email newsletter I send out. I'm quite surprised to see it on the frontpage of HN lol.

So what you're reading is one of my emails.

It's meant to be more of a summary with additional links to sources where people can read more if they're interested (emailing people long-form articles doesn't work well as a content format from what I've seen - even ignoring the length limit that email has).

I'll add in some more links for additional reading (like the high scalability post) to give some more detail for people interested.

Thanks a lot for the feedback.

I don't really believe that. I worked at Google and the frontend teams were usually just a couple of engineers per target client, I doubt any projects had 200 entire engineers just for a single target client. Not even big projects like gmail has nearly that many.

Edit: If I were to guess Google maps would have 200 engineers in total, or at least 200 figure would be the entire team of frontend engineers for every target client, and then many more working in other roles. 1 engineer per million users is a pretty good rule of thumb for Google products, and a large majority of those will work on backend systems.

I think this XKCD applies: https://xkcd.com/1205/

What's different about Whatsapp compared to a traditional XMPP experience is:

- it uses a single centralized server, without federating

- it uses phone numbers as identifiers instead of nicknames

- it has a tightly integrated (closed-source) client

So you can not only "build your own WhatsApp", but also federate with others doing the same, which is pretty cool and prevents a single actor from abusing the whole network.

[0] https://www.infoq.com/presentations/whatsapp-scalability/

Also some features like language support and such get more complicated when you want to widen userbase from just English speaking ones.

Some things just can't really make money unless they do it in scummy ways (or we haven't found a viable alternative or a way to protect them).

Public instant messengers, public forums, public file hosting, public image hosting.

What I'm saying is that for the general case of "scaling with few engineers", it seems to me that the tools available matter a lot more than the specifics of what hardware is available at the time. Launching an international service "from scratch" with a small team and a modest budget just wouldn't have been possible 20 years ago, but not because CPUs were too slow.

> Ten years or longer ago I suspect the story would be different

Whatsapp was built 12 years ago.

I was at WhatsApp from 2011 to 2019.

Hardware was outsourced to SoftLayer, of course. I think they had a sizable operation, but if you move that in house, you've got a team of 4ish for 24x7 colo work (we were only in one location for an embarassing amount of time), plus a networking person, and maybe a manager. We certainly got better service with SoftLayer's larger team including ease of getting more servers, but we would have had better visibility in house, so it's a tradeoff. Softlayer's staff was probably smaller than the combined staff if all of their major customers in-sourced, anyway.

Other than that, we used multiple providers for SMS and voice verification; some of whom have a lot of staff. You need more staff for this, IMHO, because in addition to 24x7 coverage, you also need to arrange connectivity with carriers in all timezones and all languages. But some of the companies we used seemed pretty small, so I dunno.

We were using Zendesk in our customer service system after growing out of Gmail and before in-sourcing it after aquisition, and Dyn for DNS (which doesn't take much staff to insource).

What other SaaS do you think we used? Do you want to count the OS app stores and push services? Maybe payments while that lasted? G Suite for corporate email when forwarding to personal mail became too silly?

We self-hosted our code repository and bug tracker.

If builds need to be signed by the OS maker and they won't sign any more builds, user count doesn't matter (sorry Nokia S40 and iPhone 3G).

We did make some longer lasting builds for platforms we were ending support for, but there needs to be an end. Older clients don't support newer features and we didn't want to have a network where that was the status quo. I don't know what the current expirations are, but mandatory updates a couple times a year helps keep the feature support similar.

There is a webshop in my country that copied Amazon in the early 2000s and today they are beating Amazon.

1)mobile operators charged an arm and leg for text messages, especially in developing countries

2)whatsapp did not require registration with user name and password, as their competitors (Skype, ICQ and others).

My understanding is that vulns in Whatsapp so far were client-side, but i'm interested if i missed some on the server side.

People living someplace don't own it, and people working some field don't own it. Some "owner" owns the land and means of production and extracts value from people doing the actual work.

Yes, it's a deeply broken system. A slightly better system revolves around self-organized workers coop where everyone gets equal pay and there are no shares to hold (or everyone owns the same amount). An even better system abolishes money and private property so that people can live meaningful/useful lives without worrying about imaginary numbers ruining their entire existence.