Hacker News Reader

Comment by metadat | Hacker News Reader

An unfortunate title, completely devoid of any hint about the Very Interesting content in this post:

This is an insider story chronicling the evolution of Twitter 1.0's approach to build infrastructure and developer efficiency strategies at scale on teams of hundreds to thousands of SWEs.

View on HN

Please excuse my ignorance. Why are 1,000+ SWEs needed (let alone custom tooling to support them)?

Twitter features I know about:

1. Timeline recommendations (with personalized/relevant ads sprinkled in)

2. Notifications (so you get pushed an iOS/Android alert when somebody you subscribe to tweets)

3. Being able to post/view media (image/video/polls)

4. Direct private messaging

5. Concept of following/followers with recommendations

6. Threads/retweets/replies

7. Advertising dashboard/analytics for advertisers to create + track campaigns

8. "Explore" / "What's happening" / automatically serve "trending" content / topics

9. Lists / bookmarks

10. Anti-spam / anti-fraud

11. User profiles / settings

If I missed noteworthy/sizable/heavily used by masses features, please let me know.

Why can't that be managed + developed on new features wise with 1,000 SWEs? or 500? Genuinely asking. Would love to learn what I'm missing.

You're not missing anything. WhatsApp had 14 million active users with 32 engineers. Instagram had 30 million active users with 13 employees. It's just delusional to think you need four-figure engineers to run Twitter—a product that basically hasn't changed for over 15 years.

I can't find it now, but someone in another post on Twitter pointed out that these comparisons aren't very meaningful: Twitter's social topology is any-any between users. A major (and unique) aspect of Twitter is that everybody on Twitter can talk to just about anybody else, and is encouraged to. This is also a critical value proposition for advertising on Twitter, and probably explains the size of their ads team.

This is very different from WhatsApp's topology, which is a relatively traditional chat system (small groups of shared state). Instagram's is closer, but still favors smaller social groups over a global feed.

Beyond that, Twitter is well known for running its own DCs, etc. It doesn't seem inconceivable that all of this, plus internationalization, mature platforms teams, etc. all adds up to four figures.

It boggles my mind how much people think "active users" is like some kind of fungible unit for engineering work required.

Sometimes it is, until you start comparing between wildly different companies and business models.

It's absolutely an OKR which implies higher workloads, are you kidding me‽ I'm sure you can build some architectural rat's nest where you need 100 engineers to maintain systems that barely serve a dozen requests a second (kind of like the blockchain) but that doesn't mean you couldn't do a much better job.

Everything else being equal, sure, more traffic is harder. But everything else is generally not equal. And despite your sleight of hand in this comment, your original claim about what's "delusional" was based on asserting an equivalence between the WhatsApp and Twitter workloads.

I don't think there's a inviolable ratio between active users and engineers, but you do absolutely need more engineers as your user base increases: you need more people on call, more people maintaining the servers you're running (especially if, like Twitter, you run your own DCs), more people doing QA and reliability engineering because of increased load, etc.

While the product looks like it hasn't changed much from the outside, it's been completely rewritten once or twice, at least. When Twitter launched it was a Ruby on Rails app. They scaled it to the absolute limit, then started breaking services away using Scala, then they kept at it adding several other languages to the mix.

While all this happened, there was almost zero product guidance. Instead, they watched for usage patterns, and started to incorporate those into the product. The original Twitter didn't have mentions, replies, or hashtags. Adding support for all that on a product not designed for it while managing exponential growth is an extremely difficult challenge.

On top of that, Twitter added divisions that are known to be engineer-heavy, such as ads, search, and infrastructure.

---

The comparison with Whatsapp and Instagram is interesting. Whatsapp's case is special: they famously opted to use Erlang and it turned out to be a genius decision, as that allowed them to scale beyond expectations. Whatsapp never changed much from its initial design, which means no need for deep rewrites. Instagram was a Django app running on AWS servers, and that setup was enough until they got bought by Facebook and eventually moved from AWS to FB's infrastructure.

That does sound like a task for few dozen very skilled people per bigger system, not thousand+

Well, since you only have like 2 hours of productive work per week, thousands might still not enough.

At my company there are like 20-30 engineers involved in search (using ElasticSearch). The code has grown massively and they are all bogged down fixing small stuff, improvements, doing A/B-tests and the like. They also have a whole ETL setup to gather and transform data from a lot of different sources.

In my spare-time I build a small search function with ElasticSearch for a big dataset. The use case was almost identical; and I feel my small setup works better, faster and has more options than where I work.

I don't know, things just grow over time, old developers leave and new ones join who will never be as productive as the original devs were (it's basically rewriting someone else's book without breaking it).

But I'm puzzled about it too sometimes.

> with ads sprinkled in

They aren't just splatting a tag from a third party ad platform into the tweets, they're building on your own data and handling the entire auction/bidding process internally along with all the tools for advertiser to manage that, click fraud prevention, etc. There are many 100-500 engineer companies built on doing this, along with lots of 50-100 engineer companies doing just one part of this.

> Why can't that be managed + developed on new features wise with 1,000 SWEs? or 500?

Because the bigger the company the more valuation it gets (although, with the latest layoffs and all, I think this won't be the case much more). Why would Twitter (or any tech company) would remain static in size (let's say at 500 engineers) when it can just grow and grow? As long as money comes in (funding) they company has to do something with it (hiring and invent new areas to explore that requires more hiring).

It's not a question of technical requirements, it's just all about money.

Each bullet point can be a massive undertaking, especially at scale. "Being able to post/view media (image/video/polls)" think about how much media that is. How much infrastructure you need to support that. How much it takes to make tools to manage that infrastructure.

Google has thousands of engineers just on point 1 of your list - search results.

And way more on the “ads sprinkled in” bit that wasn’t even its own line.

See how long it took them to develop their own shorted links and image uploads.

“ads sprinkled in” could be 1,000 engineers without surprising me.

Anything could be 1k engineers if you mismanage it enough

Almost all of the things in your list are harder to do at scale, especially anti-spam/anti-fraud. Plus, all of this needs to be done in compliance with the legal regimes of many countries where Twitter operates -- especially regarding advertising and privacy. Having seen how some other large web services work from the inside, I'm not surprised Twitter needs a lot of people.

It would be possible to pull this off with far fewer engineers, but not sustainably. People aren't robots, they can't work 80-hour weeks forever. People also need to be able to take vacations, and sometimes they will leave the company -- so it's not a good idea to run super lean. Plus, when companies run lean, the security and privacy aspects of the engineering tend not to be done very well (if at all), and for a social network that is a serious issue.

Twitter was definitely bloated, but I've seen and heard of worse bloat at other tech companies.

You could probably build it in a weekend.

twitter runs its own datacenters and has teams to support that. Since its not in a cloud the company runs all their own internal platform stuff you would get on something like aws. that is a large engineering effort. theres also the andorid and ios app, more teams. elon tweeted out a picture for tweet or timeline reads i think. each square is basically a team with an oncall rotation. this is off the top of my head, there's plenty more not visible stuff happening

Reply on news.ycombinator.com