Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usually have higher expectations for established companies. As always, the last 2% is a hard problem and business doesn't really want to deal with a such unreliable platform. You wanna onboard big advertisers which potentially spend $100M ARR? Then you need to assign a dedicated account manager to handle all customer escalations. PMs then triage and plan their feature requests and later engineers implement it. Which all adds up.
And they also uses your competitor's product, like Google, FB, TikTok etc etc... Twitter is a severely underdog here, so you need to support at least a minimal, essential subset of features in those products to convince them to spend their money on Twitter. That alone takes hundreds of engineers, data scientists and PM thanks to modern ad serving stacks with massive complexity.
Yeah, it ultimately boils down into a simple fact that it's really hard to take other folk's money. You need to first earn trust from them. They want to see if your product is capable of following a modern standard of digital ad serving for now and foreseeable futures. Twitter has spent lots of time for earning trusts and the original post is one evidence of such efforts. And this usually needs more man power. You might be able to do that in a more efficient manner, but I don't think that's as simple as firing 75% of your entire headcount.
This exactly. During the recent Whatsapp outage, many threads popped up on HN about how big of an issue this is in Europe, since Whatsapp is the main messaging platform in Europe. Thankfully, these outages are short and far between, so they never actually cause real issues. This is obviously costing Meta/Facebook a lot of money, but allows them to be an essential service. So essential in fact, that every major news outlet in my country sends a push message as soon as Whatsapp is down.
If Twitter wants to be a comparably important platform, they need that same stability. And Twitter, for me, is very much the best place to stay up-to-date on any current event (in near real-time). Reddit used to be pretty good with Live, but that's pretty much died (and was mostly a summary of tweets anyway). I really hope Twitter survives Elon, because I don't know of an alternative right now that has the same value in this use case.
What Twitter is doing is to scale down first, focus on the product, and once it gains traction, it definitely can scale up again. I don't think it will hurt the product very much.
I don't think you have a good understanding on how those companies are growing and scaling out. Don't take growth for the granted. "Right product" or "Right technology" won't give you that. It only comes from solving thousands of very specific, never-ending customer problems. If you do B2B, you need to spend most of your time on very specific requests from priority customers. And they are not one, but hundreds of them if you targets $xB business. It's just physically impossible to keep up with a small team even with a very aggressive prioritization.
Still not convinced? Google has a notoriously bad reputation for their customer supports and it's primarily because of their tendency of keeping "inessential headcounts" low as possible. And think about how many cloud customers they lost to AWS and Azure. TK came to GCP and his first work was adding an army of sales and account managers. This almost immediately yielded a rapid acceleration of the platform, although it's too late to catch up.
The freeze worked: we got rid of some products that weren't getting traction and were able to improve the products that did have traction. But the cost of the freeze lingered for at least a year; it reset the hiring pipeline, we couldn't grow fast when we needed to because the limited number of recruiters we had were already overworked, and the limited number of engineers had to balance interviewing needs with their real work. This all happened when my employer was <10% of its current size and pre-IPO, and we didn't even take a headcount reduction.
Twitter is simply at a different scale. 7500 -> 2500 employees is a 66% reduction. Going 2500 -> 7500 is a 200% increase. Recruiting is likely totally gutted, and the current 2500 employees have to support systems previously maintained by a 7500-person company. If they decide they need to grow, it'll have to restart at a snail's pace, and they'll have to make sacrifices on feature development or stability along the way.
Edit: for what it's worth, the fastest way to regrow back 200% is to rehire the people laid off. But, given that I happened to interview earlier today an ex-Twitter candidate who didn't make it through the Elon snap, that route is rapidly closing up.
I mean this is just wrong. Companies are always under pressure to cut costs (employees) and it is always talked about when quarterly results are posted. Look at how the market reacted to Facebook's latest results and then again what happened when they laid off thousands of staff.
This is not a trivial task. With such a heavy reduction and how entire teams have been completely decimated, there will be a lot of lost knowledge. I'm sure there will even be cases where the people who stay don't even know what knowledge was lost.
All that puts Twitter in a very risky position, specially in a product of such complexity that does a lot of things in-house. It shouldn't be underestimated.
Coincidentally, in 2013 SpaceX was just starting to provide commerical launch capacity, at which point I think they too had < 100 software engineers. A few short years later and they were re-using rockets, a feat many people had thought unlikely/impossible and requires some hardcore software eng.
Not surprised Elon Musk thinks he can run twitter with a skeleton crew.
2) WhatsApp does not support the type of public broadcasts done at Twitter, and due to its e2ee doesn't require much human moderation.
2. WhatsApp didn't have e2ee back then. The broadcasting is important, yes, but it is very heavily biased towards reads over writes, so something like Cloudflare would solve 99% of the load.
Your point 2 sounds like there were additional factors that could have influence the reliability besides the load (they didn't simply migrate to FB infra but also switched to Signal).
> The broadcasting is important, yes, but it is very heavily biased towards reads over writes, so something like Cloudflare would solve 99% of the load.
There's push-notifying millions of devices within seconds after a celeb or a major news source tweets. There's tracking view and engagement stats on that in realtime. There's making sure a tweet is not available to any of those within seconds after it's been deleted or moderator. There're separate back-office apps for moderating that firehose of content. And that's just what I can see from the outside. An e2ee instant messenger with size-limited chat groups doesn't even come close.
Please don't say "just stick a CDN on top of it and you are 99% there", it's embarrassing (and not to twitter). This will maybe get you 80% there if your goal is "a microblogging platform" but not even 20% if your goal is being both the go-to news source and shitpost forum for people worldwide reliably working even in sensitive times and emergencies. Twitter used to be a microblogging platform back when it had much fewer employees and you'd see a fail whale regularly even as it had much fewer active users, in recent yeas it's a completely different beast and saying increased headcount is unrelated is amusing.
The push notification argument is also overstated. Sharding and fan-out solves the burstiness. And people overall receive a similar number of messages (and thus push notifications) from WhatsApp as Twitter. Besides, these days the push notifications go through Google/Apple servers anyways to reduce the number of open connections needed on the phone side.
Then there are DMs. They are per person so CDNs don't help much (just static assets), but also they shard basically perfectly. So, shard them.
Which in the end leaves the user feeds. Designed correctly, sharding would work extremely well, and what doesn't work could be handled by caching closer to users for those 1k most popular accounts.
Honestly, with the correct architecture, languages and tooling, it could be handled by an experienced 50 person dev team plus another hundred in ops. Obviously Twitter doesn't have the perfect setup, so maybe an order of magnitude more? And if you throw a bunch of subpar engineers and tooling at the problem, nothing can dig you out of inefficiencies at this scale anyways.
And no, I'm not wildly optimistic here. StackOverflow still runs off of 9 on-prem servers [0]. I've seen message queues that can give 200M notifications per second on a single machine (written in C++, for HFT). This stuff is hard yes, but throwing more bodies at it doesn't help past the point your fundamentals are solved.
0. https://www.datacenterdynamics.com/en/news/stack-overflow-st...
They switched to a new protocol and grew from 200 million to I guess about a billion users since 2013. If you believe a team of 50 developers could deal with this and not cause extensive downtime and service disruption along the way I pray you never ever manage software engineers.
> Sharding and fan-out solves the burstiness.
Great, at least it's no longer "just add CDN to solve 99%" here;)
> And people overall receive a similar number of messages (and thus push notifications) from WhatsApp as Twitter.
Yeah again WhatsApp has many users but as an engineer you just don't ever have to worry about delivering a message instantly to more than 32 people (512 as of this year), and you never have to account for moderating any of that because it's e2ee and there are no adverts next to the messages. It's basically dumb pipes terminated by one native client. Twitter has to maintain a mix of automated and human review of all UGC and is accessible via extensive APIs and search engine indexed web app in addition to native client.
> Then there are DMs
Let's ignore Twitter's DMs, even without them it's far more complex and demanding than an IM app.
> StackOverflow still runs off of 9 on-prem servers [0].
Yeah, and SO maintenance page or read-only mode is up about once a month and lasts dozens of minutes. What are you even talking about now bringing up a niche programmer-oriented help forum for comparison here?
You may be stuck in the times where Twitter was a RoR-based microblogging platform. It's not been that for years.
This isn't 2005 anymore, we have multiple parallel 40gb LAN, 64 cores per socket and 2MB of L2(!!!) cache per core, and a full terabyte of RAM (!!!) per server. If you program with anything that makes cache-aware data structures and can avoid pointer chasing, your throughput will be astounding and latency will be sub-millisecond. How else do you think WhatsApp managed 300M clients connected per server without having just the in-flight messages overflowing memory, on top of all the TCP connection state?
Things only get slow when scripting languages, serialisation, network calls and neural networks get involved. (AKA "I don't care if you want docker, a function call is 10000x faster than getting a response over gRPC and putting that in the hot loop will increase our hardware requirements by 20x.")
The more distributed your architecture the more network overhead you introduce and the more machines you need. Running the WhatsApp way with less, higher performance servers simply scales better. Just from the hardware improvements since 2013 there was no reason for WhatsApp to change their architecture as they grew.
And if you think rolling out a new protocol while maintaining backwards compatibility is hard and somehow adding more people will help, I have a team of engineers from Accenture to sell you. I did this straight out of university, to thousands of remote devices, over 2G networks, with many of the devices being offline for months in between connections. You just need a solid architecture, competent people and (I can't stress this enough) excellent testing, both automated and manual. And the team that did this was 6 engineers, and this wasn't their only responsibility.
2) Public broadcasts make a lot of things easier because that means more of your workload is relatively straightforward caching (as evidenced by this blog post).
So, you don't have the numbers.
> easier
Have you even thought about moderation and all the other concerns that go with this? How does instantly notifying millions of devices helped with caching, for example?
Now the systems are stable but human workers either be sick, leave, or die eventually.
Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them.
The article wrote about achieving stability by the distributed system so an unexpected death of one rack doesn't affect the service availability. The same can be done for the human workers unexpectedly not working anymore. Have a multiple workers doing the same things improve stability.
Sure, it's inefficient in terms of money. But alternative is one sick important employee catch a COVID-19 and die lost the knowledge of the system. Documents doesn't solve it because you want the manual operation available right now rather than a few months later when replaced workers learned from the documents.
People would absolutely be more engaged and more excited about their work if they were paid more. The only reason people work is literally for money…
Well, yes, but also that's super reductive.
The primary reason people work is literally for money. The specific _place_ people work is for things that interest them, engage them, challenge them, provide a path for continued growth in their career, or contribute to a common good they believe in.
And I would argue that it's not necessarily a given that people will be "more engaged and more excited about their work if they were paid more," but instead, may be willing to stick around in an unengaging, unexciting environment _longer_ when they're paid more.
Theories like Hertzberg exist for a reason and show that money isn't even close to the most important thing for people to work for. If the money is good enough, there are many other factors that are way more important. Not enough money is a reason for people to leave, getting more money is hardly ever a reason to work harder. The possibility of getting more money if they work harder is a good motivator though, once they have it that stops.
In most software engineering jobs, people have enough money to live an enjoyable life. At that point, having more money is not really such a strong motivator.
It's largely focussed on the event stream behind the core service and data analytics. There's maybe one entry on the main data store and one on search over the last few years.
This is more true when the layoffs happen because the company’s situation deteriorates. If the company cuts jobs because revenues fall and products fail, better employees are indeed more likely to move to greener pastures before mediocre ones do. If, however, the company prospects improve, rather than worsen, this is no longer the case.
There also isn't a measure of employee quality, and layoffs are often not about the individual (e.g. Amazon cutting robotics). I've had no clue how likely I was to be laid off during most points in my career.
I was given a 33% raise, moved to a salaried position, and got a $5000 cash bonus 10 minutes later.
I had mastered the new system we were implementing and converted and supported the remainder of the customers from the legacy system. Shitty support of the product my team of now one actually drove revenue as the customers would pay higher support prices because of the shitty support I offered.
The service I offered was shitty because my job was to meet minimal contractual obligations for multi-decade contracts to persuade the customers to renegotiate the contracts for a product that hadn't been EOL'd. All the while offering onsite conversion services from the legacy product for most of my day job.
What I'm saying is that company is still alive today, they had a pivot point and literally cut 55% of their employees on a single day.
Google wants to beat facebook still, there is one of a billionity pivot points that twitter could take.
It also seems like a lot of the layoffs were achieved by offering everyone the severence of they left.
Assuming only people on visad stayed, I wonder if that skews more or less competent.
The short amount of time they took and methods they reportedly used don't instill much confidence.
They had wine on tap.
https://www.tiktok.com/@realpankhilpatel/video/7159187292631...
Normal people don't have vacations like that.
There have been several front-page posts on HN in the past few weeks about big name company CEOs apologizing for massively over hiring during the pandemic not realizing that it wouldn't pay off once the lockdowns ended and people decided to go back to living their normal lives again.
He's also extremely divorced, keeps having off the record children with his executives, and is having a midlife crisis.
Why do people do this? I mean in any industry. Just why? Is there something I don't understand about executives or C-levels or something?
There may also be some game of thrones shit going on where if a boss and a mini-boss have a baby it solidifies rank and power.
[1] https://www.sec.gov/news/press-release/2018-219
[2] https://www.washingtonpost.com/technology/2022/05/27/elon-se...
I have no doubt he had excellent reasons to buy Twitter, but what he says they are in public probably isn't the whole story.
He uses Twitter non-stop, and has an incredibly curated public image.
- He purchased it because he’s in a right-wing filter bubble and bought into the idea that Twitter is controlled by a secret left-wing cabal and genuinely believes he’s doing the world a favor because he thinks right wing propaganda is actually moderate
- He has the same brain damage as Trump and requires adulation by sycophants and he thinks this is the best way to guarantee his fix and give him control over his critics
- He wanted the brand name and doesn’t care about what services are currently offered under it
- He’s from the southern hemisphere and wants to destroy the social fabric of the United States
- Some permutation of all of the above
Not sure which of these are the truth, but I’m guessing a combination of the first two.
Twitter was poorly managed. It would have been a $100B company in 2020-21 if it had shown any semblance of consistent profitability.
The opportunity here is to:
- Take the company private
- Cut costs by pruning staff and reducing bloat
- Get high-engagement accounts back, which, in turn, will bring back advertisers (perhaps sweeten the deal by promising something with your other ventures)
- Eke out a few quarters of consistent profits
- Go public again, this time with $500M/quarter of profits
- Enjoy the standard "Musk-effect" 40x PE multiple and slow keep cashing out
Cut costs by pruning staff and reducing bloat
Get high-engagement accounts back
There were multiple executives making $10m/yr+
There were board members
There were shareholders
Why did all of them not stop this headcount increase if it's as easily reduced as "too much headcount bad, smaller headcount good"? These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc.
How can us commenters on HackerNews sit from our armchair and say "ah, goofballs should've just not let headcount get so high!"
These qualified people thought at the time it was a good idea to get up to 7.5k people. How were they all wrong?
The cliche HN comment on sites like Twitter (and many, many others, any time headcount comes up) has always been "why do they need so many people?" I've mostly dismissed it the same way I dismiss "I could build Uber in a weekend," but with every other tech giant laying people off, maybe I shouldn't. Maybe the effect of all that extra money sloshing around in the system was to incentivize hiring everyone to make sure you didn't accidentally get a false negative, and not all of those hires were good ones.
OTOH, at non tech companies, all the research interviews and surveys I have done recently continue to show a talent shortage and fears of losing tech staff, so maybe it is just that Tech companies are saturated and have too much capital for their creativity.
I wonder if any science fiction writer could have predicted silliness like Snapchat having fewer outages than serious government websites because Snapchat is better at hiring engineers. Tech hiring is so bizarre.
Our total headcount was under 10 000. Most of that was logistics and call centres (taking retail orders). Our IT dept was literally about 200 ish. All our office staff fit within a few floors of a medium sized building. IT was less than a single floor.
I’d say our software, whilst serving fewer users, was completing tasks orders of magnitude more complex than Twitter. We didn’t have much down time, in fact almost zero beyond physical damage to our networks and the once every few years disaster. Our critical support team was a handful of greybeards who spent much of the time playing solitaire. Code reviews were tight and access to production strictly controlled.
What we did have was a very lean culture, carefully managed over a long time. The company was over 100 years old.
I do not understand how, with far more sophisticated tech, companies like Twitter need so many staff. We could have run Twitter with a thousand folks, including sales. From an engineering standpoint a few hundred would do. And most of us weren’t great engineers. There were a few very experienced freelancers, though. We all worked 0900-1730, but we worked 7 full hours every single day, every minute of those 7 hours. No beanbags. Most of that time was sat quietly coding in a very cleverly designed open plan office (no cubicles) that was like a library.
Looking back, I think the key insight was that we were very cost driven and absolutely kept everything as simple as possible. We had all the usual project overrun problems, usually because things turned out to be more complex than previously imagined. But it baffles me when I hear engineering head count at places like Twitter. If you had given us 1000 engineers, we wouldn’t have known what to do with them.
Cue a team of engineers working for a year+ to build a complicated data pipeline to bring all that data back together into a graph DB (not a clue why a graph DB) and build a DSL to write the code to make these decisions.
That's where the engineers go. It's complexity that is introduced setting off a chain reaction of increased complexity to account for the complexity introduced. Then you need complexity to account for that complexity and so on.
Not that it's all self inflicted wounds. Running a site that counts 1/10 of humanity as active users is a hard thing to do
If you watch interviews with famous developers like John Carmack, they'll mention that working alone scales to about the equivalent of 5x developers. That is, adding 1-3 extra people might slow you down because of the overheads of communication and coordination. It's only around 5+ in a team that there is a definite advantage.
But what are the chances of putting together a team of 5 rockstar developers that all agree on language, style, and vision? Basically zero. So you have to settle for mediocrity. Popular languages, simple approaches, established design patterns.
If you're an experienced "rockstar" developer coding by yourself and use a fancy language like F#, you can outperform a team of 10+ people. If you're replicating a system you've seen already, 20-50 might not be out of the question, especially if you're smart enough to avoid "tarpits" and instead rely on good quality libraries and CotS components like databases, PaaS, and the like.
Organizations tend to bloat.
Random, rapid cuts might not be the fix here, but headcount was too high.
Say I'm an engineer. And I write code all day. And I get paid by a company to do it. And it blows up in production. And I tell my boss "code tends to blow up".
Or I tell my boss "code tends to take longer than estimated to deliver".
I wouldn't be given lots of promotions/bonuses with that outlook.
We're talking about a Twitter CEO with $30m/yr+ in total compensation.
There's no way his/her view was "shrug, organizations tend to bloat" while racking in extremely competitive + good pay as somebody whose main job is to drive a company towards maximum growth/profitability.
There's no way the CEO was able to convince a majority of people responsible for paying him (the board), to pay award him $30m/yr in stock options, while he was also "bad at his job enough" to let the organization bloat without as much as an afterthought to it.
We, the people on the outside looking in, have to be missing something.
> There's no way the CEO was able to convince a majority of people responsible for paying him (the board), to pay award him $30m/yr in stock options, while he was also "bad at his job enough" to let the organization bloat without as much as an afterthought to it.
Why not? Why would those board members have ever called him out? The more they're paying him, the more they can pay themselves too.
One way to fix this is to make corporate takeovers easier. This sort of cronyism ends up being a con on the shareholders (and the con is especially easy to pull off if the "shareholders" are passive index funds that don't pay much attention to their holdings).
By removing legal protection on corporate raiding, the board+CEO have to worry about activist investors who ask inconvenient questions like "why are you paying yourself $30M a year instead of giving shareholders a dividend?"
Elon himself is a CEO with a better track record than Parag, why is he not more legitimate if we are pretending leaders can't make mistakes
There's a clear correlation between org size and CEO pay, even when not in the company's interests. Some CEOs and managers also love the power. Politics is also at work.
I think you’re confusing takeovers with mergers/acquisitions
> Or I tell my boss "code tends to take longer than estimated to deliver".
Jack, who was once the CEO of Twitter, has said that they hired too many people too fast.
Elon, who is "paid" way more than $30 million/year to run several companies, also apparently thinks it was very bloated.
I'm not necessarily saying the billionaires are right and the millions are wrong. I'm just pointing out the clear counterexamples to your appeal to authority. I honestly don't understand why you would be so surprised to find out that millionaires also make plenty of mistakes. The Twitter stock price to a degree is one indicator that the Twitter executives were making mistakes.
For the same reason that colleges and universities have seen their administrative bloat skyrocket at 10x the rate of student enrollment. Administrative bloat inevitably creeps into all large organizations. Many of the people in the trenches making hiring decisions weren't considering the overall financial performance of Twitter as a company. They were making hiring decisions based on what was happening in their own department, or how that decision would help advance their own agenda, or increase their budget, or increase manpower on a favored project. When you further consider that many at Twitter openly conceded (and in many cases, bragged about) that they viewed their role at Twitter as moral arbiters of society, crucial to policing the discourse of the public, it is not hard to see how enlisting as many true believers as possible to the cause would be seen as desirable, regardless of the larger financial implications.
And the leads installed by the acquiring company seem to be highly intelligent - the company continues helicoptering in their brightest minds to fix this mess. It just so happens they are forced out every 12-24 months because it’s almost impossible to deliver anything there.
For example, last year they introduced SAFe. That caused an exodus which basically equaled a brain drain (still ongoing). Most people I valued for their straightforwardness and ability to cut through BS there have left or are leaving. The person pushing for SAFe introduction has resigned already.
The founders of that company have all left ofc, and are basically trying to restart the same company (or variations) with their own money. And their products are already picking up speed, becoming threats to the original one.
Come on, just look at the tech industry. When rates were low and stock prices kept going up, "headcount" was used as an indicator of future growth. Grow headcount, investors are happy. After all, the promise of tech stocks was "growth". Usually you're not looking to cut costs until you think growth is over. Of course, Twitter was a dog and did nothing useful for years, no innovation, no new products, nothing. But tech investors definitely saw rising headcount as a good thing...
> These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc.
Wealth is not a valid indicator of ability.
I'm not judging the execs and board members individually but rather questioning your assumption. I have read you mention "supposedly", yet it can be read as a rhetorical term.
Hiring often isn't done because of current requirements. Senior execs come and go and with them so do strategic objectives. You accumulate people and they're often not laid off when the thing they work on becomes redundant. Large scale layoffs are awful for morale and usually only come after a 'crisis' occurs.
https://nitter.lacontrevoie.fr/libsoftiktok/status/158539526...