How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.
Please tell us in detail about the Twitter stack.
Because I always find it fascinating how people think they can estimate the effort to maintain it whilst having next to no understanding what so ever of the tech stack.
1. A single person can run a mastodon instance in their spare time. Spinning up some containers for the app, a background worker and a database is quite simple.
2. Modern devops tooling makes it fairly trivial to spin up 10k instances of a container instead of 1, by just altering a number in a k8s manifest somewhere.
3. Ergo, a single person equipped with modern tooling (and sufficient funding) could spin up any number of mastodon instances.
4. Twitter is just a big mastodon instance.
5. Now that keeping everything up is sorted, add another 99 devs for feature development and you are done.
Now this is obviously faulty logic because points 3 and 4 are very false, but they look reasonable enough at first glance.
15 database admins
10 linux sys admins
5 kubernetes specialists
10 windows tech support
25 front end developers
15 back end developers w/ Scala
10 machine learning experts
Whether that makeup could or couldn't do it is a different question, or whether it would be a different mix; all of that is up for debate, but the 1/99 ratio is just one very specific, extreme, and laughable mix for anyone who has supported a system of any real size.
Creating a husk of an app that looks vaguely like Twitter and supports hundreds of users is a weekend project for anyone with a modicum of talent; building a platform that supports billions of users and is monetized well enough to support itself is an entirely different beast.
These can make for a massive hairball of complexity that can swell the number of people needed to support it.
This reminds me of a talk I once saw by a Netflix SRE, who showed a crazy convoluted mess of a diagram with thousands of crisscrossing lines going everywhere, and him screaming "No one understands Netflix!!!"
Yes, using open source code built and maintained by 732 contributors: https://github.com/mastodon/mastodon/graphs/contributors
Source: Wikipedia pages for both
Internal sources only, but topics like regionalization, localization, bespoke caching implementations, hardware-level optimizations, content moderation, policy adherence (e.g. GDPR, CCPA), long-term monetization (especially if supporting advertisers as a direct customer), 3P support, public APIs, documentation, SRE (a fledgling product doesn't need 5 9s), analytics (internal and for advertisers and 3P partners) and for the most part, security, are non-exhaustive examples of things you can mostly ignore when your product supports 10M users that are unavoidable when your product supports half of the US, a good chunk of the rest of the world, and other large businesses that consume you at large business scale.
I'd definitely like to hear more about the scale differences. So far at best, you've accounted for one order of magnitude. How do you explain the second?
So increasing your user base by one order of magnitude requires increasing the number of employees by more than two orders of magnitude? Rate of employee acquisition should probably never outpace rate of user acquisition, so I think that's a pretty clear sign that something was off.
As for sales staff being good at selling themselves, agreed, so maybe Musk's ruthless firing spree will end up as a good thing. Maybe.
No, that just results in those businesses and brands leaving you, unless you can provide them a LARGE revenue stream that is impossible to get anywhere. A large brand will absolutely give up a little money just to spite you and your company for not treating them like god.
It was also pathetically unprofitable, and had serious problems with inappropiate child photos, gore etc.
Some of those problems require man power, there is no 10 man team who are very good at devops who can solve that.
And moderation requieres humans but also a big tech team regarding bots, known offenders, etc.
it's not exclusively a tech problem but in a tech company tons of those responsabilities will be handled, addressed and solved by product and tech teams. And on 10 people, no matter how smart, they got no chance
I have no reason to distrust her and can't see any reason for her to lie about it.
0: https://www.businessinsider.com/elon-musk-says-child-sexual-... 1: https://twitter.com/elizableu/status/1595109108608544768
Obviously it's "top priority"! Is there any other acceptable answer? This answer means nothing except that he saw the question.
The reason for distrust is that the Q people have been insisting that Trump was secretly engaged in an enormous battle with pedophile rings, and using the typical, constant, and normal arrests of pedophile rings as proof. Now, another claim of a right-wing hero bravely picking up the sword and vanquishing the forces of evil, but again, no evidence! No proof at all. We just have to take her word for it, eh?
Of course, she can't give out the hashtags and allow independent verification, yet somehow hordes of pedophiles already know these hashtags? (How are they publishing those to each other, and if they have such a channel, why aren't they using that instead of Twitter?) So who is she keeping the hashtags from then?
So I remain super skeptical.
the answer means nothing, yes, the supposed good action that triggered the question is the important thing. The only relevant part of the article is them claiming to have verified it, so it's Business Insider's word added to hers.
I'm not even going to touch the conspiracy theory madness, it's irrelevant, and the person in question has zero signs of being afflicted by it, her entire existence in the platform seem to have been focused on actually working against the issue.
Its also been pretty widely publicized twitter's issue with CP[0][1][2][3], and given how hastags are the way you find things in the platform it's natural that's the way they'd do it.
> Of course, she can't give out the hashtags and allow independent verification, yet somehow hordes of pedophiles already know these hashtags?
She is giving it out to journalists and apparently they are confirming it, it's not a tragedy that someone does not want to amplify possible child abuse, actually teaching even more pedophiles how to find it.
> How are they publishing those to each other, and if they have such a channel, why aren't they using that instead of Twitter?
Twitter is protected by its sheer scale, is huge making detection harder than dedicated sites, and is far far more stable, safe and accessible than some darknet website they'd have to run themselves on average.
> So who is she keeping the hashtags from then?
From the general public to not further humilliate the victims, from other pedophiles because its not some hivemind, the list goes on.
Not everything is about Trump, and you're falling for the same level of conspiracy if behind all this out of all this you see pizzagate Qanon.
[0]: https://www.reuters.com/technology/exclusive-brands-blast-tw... [1]:https://nypost.com/2021/01/21/twitter-sued-for-allegedly-ref... [2]:https://www.theverge.com/23327809/twitter-onlyfans-child-sex... [3]:https://www.theblaze.com/op-ed/child-sex-abuser-twitter
For me, such a story is totally meaningless. It conveys no information about reality either way. The chances of truth or falsehood are exactly equal.
We have different epistemologies.
> from other pedophiles because its not some hivemind
Did you think critically about this? If so, for how long?
How are the pedophiles teaching the sooper-seekrit hashtags to each other? Is in the manual "So You've Decided to Become a Pedophile" that they send to new members of the vast conspiracy?
The problem with these conspiracy theories is that any inspection of how they might actually operate, day to day in the real world, is always neglected and handwaved away.
The whole point of a hashtag is that they spread virally, or are obvious terms. The very idea of a secret hashtag is an oxymoron.
THEY could probably do it with 100 people, YOU cannot.
100 people is most likely within the ballpark for a group of people whose sole purpose is to write and maintain twitter's tech stack. Unfortunately, that is not NEARLY the sole purpose of most people in businesses and that adds all kinds of productivity hits.
What happens is that people like yourself become convinced that's the only way to operate.
Likewise, bringing in Ad money would be a few more hundreds, because you need to chase leads in all countries.
Getting the Ads to work? That's tech and I'd be surprised if it was less than 100 people, too.
I subpoena telcos all the time. My sense is that the number is closer to 2 to 3 dozen.
Maybe the user facing site, but that's just the tip of the iceberg.
There are plenty of internal/backend/restricted systems to support and/or monetize this part.
And that's not counting the huge number of support people & moderators needed.
And a website is easy. You could do it with 1 person.
But Elon is such a machine, he could keep it running by himself.
I mean, a mediocre complexity app doesn't need more than one developer for a "slow improvement mode".
How many bug reports do you think the Twitter site and apps get each day? How many employees do you think it takes just to triage them?
You still have to read them to work out that user X saying they can't log in is the same as user Y saying authentication is failing at A stage, issue is present on B and C platforms version D.E
Further the issue of 'site using the wrong font' is not equally as important as 'site is exposing private user data'. And you're likely to get many people reporting the former, and maybe only one reporting the latter, so a sample isn't likely to catch the really important stuff.
The point I’m trying to make is that it takes some effort (beyond just the plumbing) to create an experience that folks actually want to use on an ongoing basis
Rode map is just: keep it working, or whatever EM twitted about hour ago :)
But I agree it's way to optimistic number. You need at least a few for each platform, just because of bus factor, you also need people to keep in touch with Apple/Google reps etc., making sure bills are paid etc.
I imagine it will mostly be just minor tweaks and no major features, you can easily do both mobile targets by cca 10 people, not working crazy hours. There are plenty of successful apps with smaller teams, that make it work.
Eh, tossing the direct comparison to Twitter out here because they seem to have gone too far the other way, this is much like saying "I can make 1,000,000 screws a day, so assembling at least 100 cars a day should be easy".
I am not saying that writing twitter like app is easy, but writing it and keeping it running are two different things.
If you are not adding new features, you are just mostly keeping up with Android/Apple platforms, which can be annoying, but not that difficult.
> also need people to keep in touch with Apple/Google reps etc.
You certainly don't need people specifically for this task.
Mainly the project manager.
> What about user testing and customer development?
Testing could be done by everyone, mainly the project manager, although you could perhaps add a tester to the team. Don't know what customer development is.
> By “graphic design”, do you mean User Interface design? If so, who is responsible for the user experience beyond just UI?
Yes, UI design. The graphic designer would be primarily responsible for UX, but also the project manager and developers.
This is based on my experience with mobile app development. Right now, it's a fairly complex product, users often say that it's much better for many use cases than the alternative by Google, which I guess may be developed by a 10x larger team.
It is so incomparable in scope I don't know why people bring it up.
Is there a reason to integrating ads into WhatsApp would require more than another 50 people? Twitter ads are certainly do not appear very complicated. The most complicated thing about Twitter is scale, which is why the comparison is made with WhatsApp.
> recommendations,
Does Twitter have recommendations? From what I understand, the front page was actively curated - that is, a human chose stories to put there. I guess you could count the god-awful default feed ordering as "recommendations", but there is nothing advanced about it.
> bots
If WhatsApp doesn't have bots, it's the only social media/chat app I've ever heard of that doesn't. What is needed for this other an an API?
> had to provide tooling for governments, regulators, content moderators etc.
I'm sure at least some of this exists for WhatsApp. Nevertheless, how many additional employees does this have take?
I am not sure why there is so pushback against the idea most companies are overstaffed. For the most part, yes, everyone has "work" to do. But most of the work is fundamentally unproductive. It's this way throughout the economy, but a few tech companies probably do represent extreme cases. I think the best argument for their case is that most of them are very profitable anyway (not Twitter, somehow), and they might as well throw money at thousands of people to do stuff in case one of them accidentally does something that ends up being wildly profitable. I am fairly neutral on the whole thing; I strongly dislike Elon, but I also think Twitter was horrifically mismanaged. While I doubt Twitter will come out better than it is, the idea that firing most of such a large organization would necessarily result in the immediate collapse of a mature product does not say much about the people that were fired.
I'm more sympathetic to the idea that it would get even worse over time, but I don't think there's anything necessary about this. You could focus on resolving longstanding issues while pausing most new work and probably come out perfectly fine.
You see a couple of ads mixed in your feed; behind that there's a big machine selling that space to advertisers and mixing it into the timeline of every user based on whatever profile Twitter has created for you. Then the advertisers want to know how their ads are doing, or they'll stop buying them…and you'll probably need to have salespeople to get them to put money into your ad system in the first place.
> I guess you could count the god-awful default feed ordering as "recommendations", but there is nothing advanced about it.
Just because you don't like the ordering doesn't mean it's not advanced.
> I am not sure why there is so pushback against the idea most companies are overstaffed.
Twitter could be overstaffed. In fact it probably was overstaffed. But it's not overstaffed in the tune of of "it should be 10 people working out of a garage".
This is not crazily complex, bleeding-edge tech. This is something fairly well-understood and at any rate done by a lot of teams in a lot of places. (Twitter's ad profiling also seems awful. Maybe I am hard to pin down.) Probably the most complicated part is coming up with data to make advertisers think their campaign is working. (I am extremely skeptical most ad spend is actually worthwhile.)
> Twitter could be overstaffed. In fact it probably was overstaffed. But it's not overstaffed in the tune of of "it should be 10 people working out of a garage".
I agree 10 is too low for anything but bare-bones keep-the-lights-on-this-month maintenance, but it seems likely you could have a great and functional Twitter run by ~200 employees. I've seen more done with less.
There are very few teams doing advertising at the scale of Twitter, saying "done by a lot of teams in a lot of places" is accurate just like "programming is done at a lot of places so why is programming hard".
looks like actual numbers don't agree with you
(hint: for TikTok it's Douyin + TikTok)
The claim was very few companies do what Twitter does at that scale, truth is Twitter is not a big fish in that space.
Since you are at it, do GDP of countries where they operate too.
I reckon a $ million in ad revenues in India is harder to come by that the same amount in USA.
That doesn’t mean your AI system performs better than a simpler one. Or that the system is useful in the first place (recommendations.) I’m not saying they were sitting around twiddling their thumbs. I’m saying the vast majority of Twitter staff were not actually improving the Twitter product noticeably to users. They were doing highly complex, cutting-edge engineering that was make-work.
If Twitter tech was so advanced, why were they losing so much money?
Advertising is a hard problem that not many companies have solved at the scale of Twitter, that is what I am trying to get at. There are not too many social media networks out there which have hundreds of millions of users and billions of data points, and it's very misleading to say that work done in such a scenario is "something fairly well-understood and at any rate done by a lot of teams in a lot of places", when literally they're the only ones with Twitter type data outside of a couple of other Chinese social networks.
Yes, this is my point. All this incredible AI engineering did not actually make Twitter a better product. They could have just as well not spent the money. The work was ultimately futile for Twitter, even though it might have advanced our understanding of AI and have incredibly practical applications elsewhere. Conventional measures worked fine.
>provide tooling for governments
What tooling do they provide for governments?
No, they had 30M:
https://www.vox.com/2017/4/9/15235940/facebook-instagram-acq...
This right here is why you can discount most replies on HN right off the bat. The "I can make software X in a day" posts are 99% bullshit because the posters making them have idea what business reality look like. If their program gained any popularity they'd be in a panic the first time the FBI dumped a warrant in their lap and their full stack developer is now spending a week with the lawyer trying to figure out how to untangle their data while the customers that paid for ads are yelling the metrics API went down 2 days ago.
IG 13 employees at 30m users. Couldn't find # of servers.
FB had 10k servers in 2008 and 100m users, 850 employees.
I believe Doubleclick had ~500-1000 servers for ~10b daily impressions in mid-2000s.
Those numbers are all on circa 2010 hardware, so.. divide by a decade of performance doubling every 2 years (conservatively), or ~5x fewer servers in 2020.
The government takedown stuff, from personal experience, is tiny on the systems side; much more about moderators and expensive legal staffing.
These are very rough estimates, but I've heard 250k servers for Twitter.. that's much more on par with Goog/Amzn/Msft serving clouds at ~1m+ machines. That's a mystery to me.
so you mean that 50 people could run Twitter, if only they removed the bloat and focused on their core business?
> or had to provide tooling for governments, regulators,
nobody did back in 2014.
Since then WhatsApp has grown a lot, doubling its size to around a hundred employees.
What's their core business? Losing money on a platform where people can post racist comments?
Or do they have to earn money without getting sued for being used to spread CSAM and being a platform for harassment?
Because the first is very easy with 50 people. Elon can keep sinking money into it and never earn a dime (see Truth Social, they seem to be doing well!). The last is a lot more complicated and requires an ad platform, ad sales, content moderation, documentation writers, support agents, management, scrum masters, SREs, purchasing, et cetera.
IT people are really good at ignoring everything but the tech stack. Like tech is the only thing needed to run a profitable business. (It is... But not to run a 40B valued profitable business... And Twitter wasn't even profitable at all.)
they've been running the same exact business for years now, if what you say it's true, maybe the answer is yes?
I never heard of Twitter making profit, so maybe their core business was "Losing money on a platform where people can post racist comments"
There was abundance of both, AFAIK, long before Musk
Remember when Dorsey tweeted nazi propaganda and then said his account was compromised? (which if true it means at Twitter they don't know what they are doing, if false, well...)
Remember when journalists wrote articles titled "Twitter is a Nazi haven for the same reason its CEO claims no bias" because Dorsey never actually distanced himself from the worst of the worst the platform hosted? Fearing he would be labeled as "too liberal"?
Remember when he started spamming crypto-bro propaganda?
Remember when a spy from Suadi crown worked at Twitter helping to uncover activists using the so called "free" platform and after he was discovered and reported to authorities, Saudi prince Alwaleed bin Talal bought 4.61% of Twitter shares?
Did the situation got better with more and more employees or it just revolved around banning prominent accounts? (not that I necessarily disagree with the reasons behind it, but if that is the best solution they've found, after years of fine tuning, they could have done it before with much less people involved)
> Or do they have to earn money without getting sued for being used to spread CSAM and being a platform for harassment?
it's easily provable that the 7,500 employees did not improve things on that front.
> IT people are really good at ignoring everything but the tech stack
that's a really odd proposition.
Looks to me that Twitter was in bad shape already, despite thousands of non tech employees, Facebook it's in no good shape either, despite tens of thousands of them, basically the only thing still working as intended in those companies is the tech stack.
I guess the real question you're asking is "why non IT people are so bad at doing their job"?
Not my opinion though, I never said IT over other departments, I simply said nobody is able to explain what makes WhatsApp so special that a hundred people can run it while Twitter requires 75 times that and still doesn't work as well.
They are entirely different technological challenges.
ok, so this must be the hardest problem in the World, given that
- WhatsApp had 55 employees at the time of acquisition
- Twitter had 7,500 full-time employees at the end of October
But still, you can't compare apples to oranges.
right, because WhatsApp is a small company that makes very little money and has virtually no users and it mostly does not work nor scale...
We could all write that in a single weekend, if only we had no family to spend time with.
It's interesting how the prospective shifts when people are told "yeah, that's impressive per se, not very impressive compared to what the others are doing"
It's like discounting Sputnik 1 because the Russians did not employ an army of people selling ads, but just the people necessary to launch a satellite in orbit, which is actually the real achievement.
Anyway, from the news: Nearly 1200 software engineers left Twitter last week
Suddenly the Twitter engineering team sounds not so capable, which clearly is not the truth, the truth is that if you have hundreds of managers, you'll end up with hundreds of small teams competing to boost the ego of the manager, usually wasting thousands of man hours on miniscule returns (if not losses) while those power point slides will help someone else to get promoted for the new project that nobody uses.
Been there, done that, I don't know why a demographic so well versed in the dichotomies of the tech industry such us the users of HN is so baffled by the claim that 2,000 engineers for a single company that does what Twitter does is a complete waste of human potential.
Elon Musk is a person I would never work for and I think he's not even a good entrepreneur, but one thing he does right: he calls the shots and then executes them.
He said he would fire people and he did, many helped him by leaving on their own, which left Musk with the responsibility of proving he was right.
If Twitter will still be up and running in a year time, we can be sure that there were 1,200 engineer too many working there.
because, honestly, who really believes that the "influencers" will actually leave for the fediverse, where they'll have to work hard and compete with mere mortals, while they could keep cashing from advertisers to promote shit to their already established audience?
nobody believes that.
Also because the fact that Twitter will sell less ads in the next future doesn't mean that advertisers won't spend that money on Twitter, they will simply not pay Twitter, but the Twitter users. For them it's exactly the same thing, for Twitter celebrities it's a giant opportunity.
So twitter can afford to deliver those tweets with higher maximum latency than WhatsApp.
And it's scaling when you need to keep low latencies, that really kills you, at least in my experience.
Twitter’s latency stems from calculating what tweets should show on a given request. Even if you try to show tweets from 1 minute ago, it’s hard to cache that stuff using traditional systems because of the fan out. If an account with 50 million followers tweets, you need to update 50 million timelines. How do you do that quickly?
And you would have to define maximum latency, is it seconds, minutes, hours? because you can’t have the timelines be inconsistent for too long as that leads to some people getting news faster than others.
now you have to deliver them, exactly one time, to each recipient or groups of recipients, through different network topologies, with different challenges and vastly different bandwidth and latency guarantees, in exact order, while also keeping track of who is online e who's not, and distributing that information in real time, only to the edge nodes that should know about it, all of that fully E2E encrypted but stored (indefinitely?) in case the recipient is currently offline and unless that recipient blocked the sender.
let's agree that both companies solve hard problems and that it's not the technical difficulties that make the two companies sizes so different.
Making it globally available and legally compliant, that's where the next few thousand folks come in.
The people shouting loudly about how Twitter must have been so bloated are really just shouting their obvious inexperience working at global scales or their localized ambitions.
Could there be too many employees at Twitter? Sure. Most companies have dead weight. The number who were "extra" is probably not 9/10ths the employees though.
This is because you don't see the complexity. What you see as a Twitter user is a fraction of what's actually there.
You have to build a platform for ads. Not just serving ads, but allowing advertisers to prepare their collateral, preview them, get their results, and be billed. So that's an entire content and invoicing platform separate from your main feed.
And since your platform is all user generated content, you've got to build a moderation pipeline. A place for users to make reports, but also an interface for your content moderators to view content and make decisions. Oh, and while you're there you'd better build a portal for law enforcement to make data requests, along with your DMCA takedowns. Oh yeah, DMCA - that's another whole thing you've got to worry about.
Then the EU comes along and needs you to build something to support your GDPR obligations. Then India wants something similar, but only for its citizens. Your users also want verification, so better build that platform for securely verifying accounts and awarding checkmarks.
It snowballs. Was Twitter's engineering group bloated? Probably. Most large companies are. Could you run the whole Twitter tech stack as it exists today with a hundred people? Absolutely not.
Separately, some commenters here are flatly delusional about the effort to ship a site, android and ios apps, internal mod tools, help docs, support, and legal docs in 34 supported languages. Not to mention obeying laws in all the countries that implies.
Or image and video hosting! With recoding of videos, resizing of images, and the management of what is surely petabytes of images and videos with very high reliability! That is not a 1, 2, or 3 person job to do well.
Anyone who has used Twitter, have you seen any evidence they do this beyond extremely basic geographical targeting.
Like people keep listing off all this stuff when we’ve all used the site and can see if it does have a team working on it then they’re not doing it to the levels of their competitors.
https://twitter.com/settings/your_twitter_data/twitter_inter...
>> "These are some of the interests matched to you based on your profile, activity, and the Topics you follow. These are used to personalize your experience across Twitter, including the ads you see. You can adjust your interests if something doesn’t look right. Any changes you make may take a little while to go into effect."
They're not good at identifying interests and making them targetable, but they try.
It takes an army of engineers to build a resilient architecture at Twitter's scale.
And why are we even talking about "keeping the lights on"? Elon is claiming he's going to build a better video platform than YouTube, complete with better tools and for creators, for crying out loud.
SpaceX is an interesting result, but it is starting to look more like a fluke than what we'd expect from him. Perhaps rocket scientists "get him" and understand the stakes better, after all, if you screw up a rocket you could die in the aftermath.
Tesla is certainly similar stakes -- except one mistake can kill every passenger on the road instead of your fellow employees. Twitter is basically 0-stakes, and is unlikely to take itself that seriously. If they screw up, maybe a bad government won't get overthrown somewhere, or trolls will take over the internet. But mostly, people will survive -- or rather, the outcomes are far more abstract.
He doesn't seem like a good fit for this kind of environment.
"20 with cloud, 40 without. So much overlap between iOS, Android, and the web, three people can do all three. More for the backend." https://twitter.com/realGeorgeHotz/status/159371372367535718...
>Their moderation
Above I assumed their moderation team was probably larger than their engineering team, and mostly contractors. Thus I kept my estimate to the size of their engineering team.