2 Years at Twitter
eed3si9n.com
eed3si9n.com
Two things to consider:
1. It is sometimes hard to buy things that works well: https://danluu.com/nothing-works/
2. The productivity gains of something like Bazel and Pants are pretty amazing if you are working within a monorepo.
It is very difficult to understand how big these two things play into these decisions. Having ~20 people manage build tooling, build caching, source control, CI (build+test), and building release artifacts (which it sounds like Pants did) is not a bad deal.
Consider:
1. GitLab will charge you $20/month/user.
2. If you have 2000 SWEs you'll pay 40k/month just to have access to the website.
3. You'll still need to hire a person or two to negotiate the purchasing of the GitLab.
Now remember that GitLab might not be prefect for your use case. GitLab doesn't increase the performance of builds on your laptop by sharing a cache with all engineers. There's also a limit to the mount of data you can store for source + build artifacts (50GB). The CI runners they host for you are very expensive for how slow the machines are ($10/1000 minutes).
Sure, you can do things like run your own GitLab runners but now you're back to running your own infrastructure which is an engineer's job. Sure, you could save money by using Gitea or something that if free but again hosting your own infra. As soon as you have someone managing that infrastructure they're bound to say "how else can I save time and money" and at a certain scale these things become reasonable or even required.
That doesn't sound like a bad deal if you have to hire such a big build team; let's say you could fire 10 people, 10 * 150.000 (rough estimate) / 12 = 125k/month. That's a 85k/month saving.
I highly doubt GitLab fits the need of such massive monorepos; my only point is that the financial argument is not very strong IF GitLab were a viable alternative to such a big build team.
> 2. If you have 2000 SWEs you'll pay 40k/month just to have access to the website.
So the cost of just 1 more engineer... 2001 instead of the 20 extra that this build team creates.
In that year, Twitter had 100 million users and already functioning build system, that they opensourced soon later.
Do you happen to have some examples? Very curious to hear about this.
Other companies had similar problems to solve, and they also had to build it themselves for the most part.
Kubernetes was launched in 2014, and similarly, was mostly a curiosity for a few years. Twitter at that time made an (probably?) unlucky bet on Mesos, a competing but not that similar technology, that is now pretty much dead.
Google Cloud launched it's VM service in beta in 2012. A lot of base AWS services did exist, but was similarly a curiosity: AWS 2012 revenue was 2 billion. This year they are on track to 80 billion.
In theory, maybe. But can anyone external honestly look at what Twitter has shipped for the last few years and justify that it was at all likely their productivity problems in engineering had anything to do with the build system.
I've worked as an FTE in a platform team for a major public tech company before, and I don't know many smart ones that came out of that thinking it was a good trade off to have teams whose very survival necessitated potentially unnecessary platform tooling / changes -- unless you're literally Google or FB and can see measurable results from optimizing that tooling.
Twitter never got close to that point.
You say you've worked at a "major public tech company before", so why would you assume Twitter's paralysis was the fault of engineering and not management/product?
There'd be exceptions, but you'd save a lot on build cost/time and dependency and artifact management.
We already hire the people who negotiate the purchase- it's the same people who manage the infrastructure for our onsite instances, etc. IE, that cost is already priced in.
We run our own runners, on a nodegroup in k8s, which downsizes after the runners go idle for a while.
In enterprise, many attempts to save or squeeze a last penny out of something just ends up moving the costs elsewhere in the company, or reducing productivity, or driving away good employees.
Monorepo for the entire front end and back end of Google services and products (excluding the Android OS code, and some other projects) made it really easy to get started doing interesting things without having to do a lot of work. LexicographicRangeSharder, which shards data streams using reservoir sampling, is a great example that I wouldn't have been able to build myself, but could trivially link into my code.
Blaze was an excellent build tool most of the time, and the team that build it/supported it was excellent.
The build/test platform (Forge and TAP) were both absolutely genius, and made large-scale testing easy. I was the owner of numpy and scipy and frequently had to do megabuilds to ensure that upgrades didn't break important things (IE, we forward-fixed all the dependencies that would break when the upgrade was committed).
A highly scalable version control system with serial commit numbers, making all sorts of projects involving bisection and problem hunting far easier than git.
A wide collection of very smart people who could answer nearly any technical question that came up
After returning to more conventional company, as well as a startup I really miss a lot of that. but blaze and TAP were partly necessary because of the design decisions made by early (pre-google3) engineers.
For example my company is reinventing Next.js in house because the politics in the company won't allow us ditch this half-baked Next.js clone and just use the real thing. Our deployment pipeline will be much simpler if we simply bought Vercel. I'm sure we can negotiate prices with them that is better than hiring 12 engineers
He links the pants-devel announcement and here is what it says:
> Twitter has made the decision to migrate from Pants to Bazel as our primary build tool, as we believe that we have unique needs that Bazel can better address
Pants was your own fucking build tool that you invented, controlled and maintained! How on earth can it not meet your unique needs.
It’s much cheaper in the long run to migrate to a well supported tool that you can extend than have to continuously staff some of the best engineers to keep building on your homegrown mess.
This comment is reminiscent of others i have seen about “building uber with 20 engineers” what could be so hard?
They build it years before Bazel was opensourced.
I find it weird that everyone acts like Twitter is a new startup that created all of it's infra 1-2 years ago, while they had 100 millions of users in 2012 - a lot of this stuff was already years in the making then.
This is an insider story chronicling the evolution of Twitter 1.0's approach to build infrastructure and developer efficiency strategies at scale on teams of hundreds to thousands of SWEs.
Twitter features I know about:
1. Timeline recommendations (with personalized/relevant ads sprinkled in)
2. Notifications (so you get pushed an iOS/Android alert when somebody you subscribe to tweets)
3. Being able to post/view media (image/video/polls)
4. Direct private messaging
5. Concept of following/followers with recommendations
6. Threads/retweets/replies
7. Advertising dashboard/analytics for advertisers to create + track campaigns
8. "Explore" / "What's happening" / automatically serve "trending" content / topics
9. Lists / bookmarks
10. Anti-spam / anti-fraud
11. User profiles / settings
If I missed noteworthy/sizable/heavily used by masses features, please let me know.
Why can't that be managed + developed on new features wise with 1,000 SWEs? or 500? Genuinely asking. Would love to learn what I'm missing.
This is very different from WhatsApp's topology, which is a relatively traditional chat system (small groups of shared state). Instagram's is closer, but still favors smaller social groups over a global feed.
Beyond that, Twitter is well known for running its own DCs, etc. It doesn't seem inconceivable that all of this, plus internationalization, mature platforms teams, etc. all adds up to four figures.
While all this happened, there was almost zero product guidance. Instead, they watched for usage patterns, and started to incorporate those into the product. The original Twitter didn't have mentions, replies, or hashtags. Adding support for all that on a product not designed for it while managing exponential growth is an extremely difficult challenge.
On top of that, Twitter added divisions that are known to be engineer-heavy, such as ads, search, and infrastructure.
---
The comparison with Whatsapp and Instagram is interesting. Whatsapp's case is special: they famously opted to use Erlang and it turned out to be a genius decision, as that allowed them to scale beyond expectations. Whatsapp never changed much from its initial design, which means no need for deep rewrites. Instagram was a Django app running on AWS servers, and that setup was enough until they got bought by Facebook and eventually moved from AWS to FB's infrastructure.
In my spare-time I build a small search function with ElasticSearch for a big dataset. The use case was almost identical; and I feel my small setup works better, faster and has more options than where I work.
I don't know, things just grow over time, old developers leave and new ones join who will never be as productive as the original devs were (it's basically rewriting someone else's book without breaking it).
But I'm puzzled about it too sometimes.
They aren't just splatting a tag from a third party ad platform into the tweets, they're building on your own data and handling the entire auction/bidding process internally along with all the tools for advertiser to manage that, click fraud prevention, etc. There are many 100-500 engineer companies built on doing this, along with lots of 50-100 engineer companies doing just one part of this.
Because the bigger the company the more valuation it gets (although, with the latest layoffs and all, I think this won't be the case much more). Why would Twitter (or any tech company) would remain static in size (let's say at 500 engineers) when it can just grow and grow? As long as money comes in (funding) they company has to do something with it (hiring and invent new areas to explore that requires more hiring).
It's not a question of technical requirements, it's just all about money.
It would be possible to pull this off with far fewer engineers, but not sustainably. People aren't robots, they can't work 80-hour weeks forever. People also need to be able to take vacations, and sometimes they will leave the company -- so it's not a good idea to run super lean. Plus, when companies run lean, the security and privacy aspects of the engineering tend not to be done very well (if at all), and for a social network that is a serious issue.
Twitter was definitely bloated, but I've seen and heard of worse bloat at other tech companies.
I feel like the most productive I’ve been is
1. Psychological safety and trust in engineering to lead
2. Coworkers that intrinsically care about their work
3. Coworkers that are fantastic people
4. Meaningful work
The top down, hard ass culture of Elon etc, basically flips the work to being extrinsically motivated based on fear. If the goal is engineering excellence, it just doesn’t work. A lot of people shut down and won’t take any risks or innovate in that situation. Or they leave.
There may be other goals of course, but there’s a valuable reason tech companies have been selective in hiring and generous in trust.
An engineering culture that’s awesome at a company like that will only last until the money runs out.
I’m not supporting what Elon is doing. I’m just saying that what Twitter was doing before wasn’t sustainable or on a path to being sustainable. The culture was part of that
Not very positive! They weren’t a great business!
Then they had to pay a huge fine, and lost a bunch of money because of it.
But they weren’t operating at a loss. Certainly not “millions a day.” Where do people get that figure?
2020, they had a $1B net loss for the year. 2019, they had a $1B net gain.
I dunno, it’s clear they were a weird and not great business. But like, they weren’t JUST lighting cash on fire.
But yes, this notion that Twitter was just hemorrhaging money seems flatly false and easily disprovable. I haven't read the financial statements in a while, but when I last did I indeed saw one of the few tech companies that was actually profitable.
Is there a such thing as a person who reaches that level of wealth who isn't? Perhaps people who were born with it might not be, but it seems to be a prerequisite for people who achieve that level of wealth themselves.
It seems like cherry-picking excuses to support Musk's leadership style rather than any justified basis. If the parent comment is correct, they just aren't as profitable as Wall Street would like (therefore they are not "profitable"). Saying that the firings are justified is then just moving the goalposts to support an a-priori conviction.
Additionally, it seems ironic that this is coming from proponents of the Silicon-Valley ethos of "screw profitability, we'll figure it out later"... an ethos that created Amazon and so many other tech titans. Yes, profitability matters, but only when your other forms of capital (e.g., social, political) have been depleted.
It's tragic that Musk is depleting whatever capital twitter had remaining on its current death march. Financially "profitable" or not, twitter did undeniably have social and political capital. Soon, it will have neither.
There’s room for sincere curiosity about what’s going on and what’ll happen, but a lot of what I see online is people just relishing in the sadism of the whole ordeal.
Not sure how we escape the gravity of it.
But I read the GP to mean they've seen people crossing a thin line.
I doubt any of us are above a bit of schadenfreude when the fates deal a fair dose of karma to someone we think deserves it.
But actively cheering (and trying to multiply, in the case of aggressive trolling) the sadism in public is a kind of dark prayer.
Also, it’s very obviously not true that all the people who lost their jobs are highly paid. It’s curious how this self-deception keeps appearing among people who are cheering it on.
It’s really not a game show. There are important ideas being tested and norms being created (or destroyed).
The biggest self-own in the industry will be if Elon is successful and then the whole software industry suddenly is inspired not by (yes, perhaps excessive) kindness/comfort/generosity of a good work life but by cut throat PE-style management with an added layer of emotional manchild tantrums.
There’s a reason VCs are cheering this on and it’s because lower standards in the software industry will save them all money.
And I also see musk's compulsive lying and arrogance as harmful, althought in more limited scope.
So from my perspective it's one bad thing destroying other bad thing. What's not to be happy about ?
Sure, Twitter transforming into not a vile piece of shit it is today into something decent would also be nice outcome but I don't even know how that would work so it burning is next best thing.
* [1] https://www.forbes.com/sites/olliebarder/2022/11/14/japanese...
It is absolutely possible that Twitter turns into a 4chan-esque cesspool of vitriol and hatred but at massive scale due to a gradual frog boil of 160MM+. Not to say Twitter had none of this, but again, it takes little imagination to worry about it getting much, much worse.
Also more broadly, people quitting en-masse with new leadership attempting to make the place profitable should tell you all you need to know about the culture.
Trying to make the place profitable by firing large swaths of their coworkers. You make it sound like people started jumping overboard right when the rescue crew was on the way, rather than when a new captain took over and started trying to stop the ship from sinking by forcibly tossing people himself.
Twitter has a very long runway where > 90% of its existence it was making a loss.
Where competing social networks have exponentially grown in size (FB, Insta, Tiktok, even Reddit), Twitter has stagnated.
Twitter is widely known to never really ship anything. So I have no idea which "velocity" the author is talking about.
Before the musk take-over, a plan was already in the works to cut 800M in expenses.
May I remind you that Twitter almost went bankrupt in 2016? They tried to sell but nobody wanted it.
He isn't firing those people as a component of "leadership", he's clearly cleaning house. He doesn't _want_ the vast majority of twitter staff to remain for obvious reasons.
Now of course that was tiny minority of workers there, but there seemed to be zero that's happy about it before acquisition so the "obvious reason" is "aside from cost cutting he's kicking people that would give resistance to whatever he wants to do with twitter"
Of course if employees don't believe in leaderships vision they might not be a good fit. Of course a company looking to cut costs is going to look at trimming their workforce. This all makes sense, it just has absolutely nothing to do with free speech.
> It's tragic that Musk is depleting whatever capital twitter had remaining on its current death march. Financially "profitable" or not, twitter did undeniably have social and political capital. Soon, it will have neither.
Is it tho ? The social capital Twitter had was just used to push ideas of whoever was in charge there at the time. It was slowly becoming a propaganda machine that was disguised as social network.
Just like a week ago we had HN discussion about how after firing a bunch of people Japanese twitter suddenly stopped being so political [1], coz those topics were being forced.
Twitter dying in fiery musky fireball might be the best thing to happen in long term.
* [1] https://www.forbes.com/sites/olliebarder/2022/11/14/japanese...
Is there a difference? From what we've seen over the past couple of decades, it seems like social networks are doomed to devolve into propaganda machines.
>Twitter dying in fiery musky fireball might be the best thing to happen in long term.
That's what I feel like too.
First one is de facto popularity contest and each side is free to call on other's bullshit, if control comes from company running it the detractors from "right" vision gets silenced,either explicitly, or by just modifying algorithm to deprioritize their content
I suspect that $4m a day number Musk has been tossing around is rooted in his terrible financial investment. He purchased the company for $44b, which has something like $1.4b in interest every year? Which would be $3.8m a day.
So Twitter wasn't losing $4m a day--Musk was.
https://www.statista.com/statistics/274563/annual-net-income...
https://www.businessofapps.com/data/twitter-statistics/
and see that revenue is growing but it's still in the red, and obvious conclusion to that being "they hired a lot of people that don't bring that much profit"
In 2021, Twitter would have been profitable were it not for a ~$800 million legal settlement payout.
They posted a net profit in Q4 2021 and Q1 2022.
This is false. Twitter was profitable in 2017, 2018, 2019 and 2020 and its R&D budget was half that of 2022, or less, for those years and by all accounts it had a fantastic culture.
In 2021, the last year available and its very worst year since 2015, Twitter made $5 billion in revenue and lost $492 million. So your "millions of dollars a day" is not wrong-- $1.4 million per day.
Twitter spend $1.2 billion on R&D-- less than 25% of its revenue. In 2020, it spend $873 million on R&D and made a profit. In 2019, it spent a mere $682 million on R&D and made a $366 million dollar profit. Similarly in 2018.
Additionally, the incentive from wall street is to spend all your money to grow users. Whether that is right or wrong it’s the path they chose which led to users and revenue roughly doubling over 5 years.
https://s22.q4cdn.com/826641620/files/doc_financials/2021/ar...
https://s22.q4cdn.com/826641620/files/doc_financials/2019/Fi...
https://www.wsj.com/articles/elon-musks-buyout-will-load-twi...
Allowing so long for the deal to close may have been a mistake due to what happened in the market between when they agreed to terms and when the deal closed.
https://www.bloomberg.com/opinion/articles/2022-04-21/elon-g...
- (mine, software engineer) city life, large middle class, high taxes, some luxury districts
- (environmental city planner) Almost entirely renewable energy, vast public transit, living and jobs spread evenly on the map
- ([company redacted] clerk) giant coal mine in the center of the city, mass low-end housing, unchecked health issues, ruled by iron fist
[0] https://medium.com/age-of-awareness/why-simcity-died-and-how...
Edit: redacted company name due to people really liking their samples.
I worked at a company (well known in tech circles) where the CEO went through a phase of ruling by fear. He was firing and replacing people constantly, ever present in Slack to criticize work and threaten people if he thought delivery wasn’t fast enough, and even fired a couple people on the spot in front of a large group of people just to make a point.
It created a wave of productivity at first. We all thought we could just get ahead of it by doing good work. If we work hard, we won’t end up embarrassed and disgraced like our fired ex-colleagues.
But after a while everyone realizes that there is no winning in a fear-based environment. It doesn’t matter how hard you work or how great your results, the fear-based boss will always find some other excuse to apply more fear push everyone harder. People must be sacrificed constantly to maintain the image of a ruthless leader.
Eventually everyone rotates out of the company into better jobs. The company became a revolving door. Product quality declined as knowledge was constantly lost with departing employees.
Twitter "engineering" is a complete joke, the website is somehow less reliable than _reddit_ and that is saying something. There has been little to no innovation at Twitter for how long? Pretty much since it blew up.
Meanwhile the culture you're criticizing is responsible for Tesla and SpaceX, both extreme disruptors in their spaces. You don't have to like them, but you can't deny they are extremely innovative companies.
Why not both? You can have a hard-ass culture of relentless insistence on highest standards and a good ambiance within the team.
What musk is doing is obviously not sustainable and can ruin technical competency but it is not exactly example of "ruling by performance" as much as him just cleaning house/cost cutting company that was not profitable for more years than profitable.
Also people's motivation are different. Allowing people to self govern and do what they want to makes for happy employees but shit output, example being Valve with it famously flat structure and barely any release. You do need some pressure to make stuff useful for the company and you do need some pressure to actually finish it instead of playing around.
Rewarding innovation sounds good on paper but ends to Googler complex where nobody wants to do maintenance as you won't get pay rise out of it compared to working on some cash cow or something new, if there are no incentives for maintenance.
Cutting low performers no matter what gets you stack ranking and old Microsoft, and team that happened to hire only good people are hurt for no good reason.
Do you really believe that? If you look at videos from SpaceX launches for example, do you see scared people motivated by their fear of Elon…?
I’m not saying everything Elon does is golden. But there seems to be a major gap in understanding here between different personality types. I find that interesting. If you could elaborate a bit on your point of view it would be much appreciated.
Also, there are more large companies running on way more "shoddy" technologies like Rails, which Twitter also used up to some point.
Yet, such a pain in the ass…
and
> For example, I think it was Adam who pointed out that the Mac laptops were unable to fully utilize the remote cache because action cache was platform specific. This remains to be an open challenge of Bazel to date.
It's always macs that are suffering from problems in my experience.
Reading that line it's evident that the issue isn't Mac-specific, just that you can't reuse build artifacts across different platforms (which is expected for pretty much any build system).
That does not mean that all of those people had a single manager and were all responsible for the same chunks of code.
Most likely there were 3-4 sub-teams responsible for different parts of the system.