Comment by meddlepal | Hacker News Reader

Comment by meddlepal | Hacker News Reader

Parent Full thread

My problem with Twitter has always been what the fuck does it do that requires 1000 employees to run? I don't use Twitter so I may be missing something huge here, but my understanding of the service:

1. User A posts messages either to themselves or by addressing another user.

2. Users B, C, D .. Z subscribe to messages from A and see them on their own feeds.

There are no concerns about message QoS and you don't need a consistency so you can basically blast out messages and if B views it before C then no biggie.

Twitter's basically a pub-sub messaging platform without any kind of need QoS... what am I missing? Why does this require 1000 employees or even 500? it seems like something that could absolutely be run by 50 people with software (feature, optimization) engineers concentrated in one location and operations (test, deploy) teams spread out across the globe at major hot spots. Obviously you also have some stuff like sales and marketing, but I still don't see how that requires 1000 employees.

Large scale distributed systems aren't that simple. I don't know any internal details of Twitter, but from what I've gathered they are trying to hit the kind of QoS you are dismissing. Imagine some celebrity with millions of followers says something controversial: it would be confusing to a lot of people if they start getting retweets or discussions of that tweet significantly before they see that tweet themselves. And that's just one scenario.

I don't know about the 500 vs. 1000 employee question, but Twitter is a big engineering challenge.

>I don't know any internal details of Twitter

There were quite a couple of relevant posts on Twitter Engineering Blog if you are interested:

https://blog.twitter.com/2011/spiderduck-twitters-real-time-...

https://blog.twitter.com/2012/blobstore-twitter-s-in-house-p...

https://blog.twitter.com/2014/manhattan-our-real-time-multi-...

https://blog.twitter.com/2014/push-our-limits-reliability-te...

https://blog.twitter.com/2015/building-a-new-trends-experien...

> Large scale distributed systems aren't that simple.

Nobody's debating that. If it weren't a "large scale distributed system" then the debate would be about 3 employees vs. 10. The core experience without scale could be made by a single engineer.

But running a service like Twitter doesn't require 4,000 employees, even at scale. Facebook has "only" twice as many employees, despite a 10x more complicated product and 12x as much revenue.

> Large scale distributed systems aren't that simple.

It is not simple but it also doesn't have to be 1000 employee-type of complexity.

As a counter-example I'd point to WhatsApp was handling 1B users with 50 employees. That includes custom apps for fringe phones used in 3rd world countries. It included media sharing, it including managing and ops.

I know they are not Twitter but still 50 vs 1000 is a huge difference.

They've traditionally credited Erlang and FreeBSD as one the techologies that was responsible for their success. Not sure if Facebook will strong-arm them into switching to whatever Facebook is using.

One can then ask, well, why don't companies look at that and learn and do the same? And I think the answer is once they headed in one particular direction with their choices, they are more susceptable to the sunk cost fallacy. Coming in one morning and telling everyone, screw it, we picked the wrong direction 2 years ago. We'll lay off 50% of the people and re-write the whole thing in Swift using this other stack. We'll cut half the products and work on new ones. Etc. That is almost impossible to do. So they are sort of bound to heading in the direction they've been heading, even if individual people will all see it is probably the wrong direction.

WhatsApp chose engineering cost first. It's a singular device only service, they do not retain any messages and a slew of other decisions. Look at their iOS client for more examples of engineering cost first. A new client for an obscure phone costs about 1 engineer for WhatsApp too.

Twitter is not an engineering cost first company.

> they do not retain any messages

Right, though they retain messages until their recipient comes online.

How many employees did it take to run IG? Architecturally they are very similar.

Now, I do think there's more to it, once you start adding in monetization and business features, but it doesn't take that many people to run a highly scalable distributed pub/sub.

I've always assumed most of Twitter's employees are either customer support, or "community managers" for various high-profile accounts. That's the sort of thing that has no economy of scale whatsoever.

My guess is embedding "monetization" platform within existing. So yea it's a simple service, but under the surface they probably got all sorts of hooks to increase value of their 'social graph.' My guess also is, by the looks of their revenue, etc., it's a mess

This is true, but as others stated an over-simplification of Twitter's features. Even without the factor of scale, building Twitter for 1000 users would be hard when you consider things like private messages, e-mail notifications, on-site notifications, hash tags, retweets, favoriting, follower-lists, url shorteners, images / videos, ads, analytics, search, security, new features like their 'moments' tab, - and considering how all of these features work together on mobile as well as web. THEN start thinking about scale and how these features work together - not just the C and R of CRUD, but the U and D too.

A company needs more than engineers to run.

Reply on news.ycombinator.com