Elon Musk moving servers himself shows his ‘maniacal sense of urgency’ at X
cnbc.com
cnbc.com
Except they had many people tell him it was a bad idea.
In the end, it was indeed a bad idea. Unfortunately, his loyal lieutenants get rewarded for “kicking ass” while the person who knew it was a bad idea probably started looking for opportunities elsewhere.
The only people who would be left in the end are people who worship him unquestionably.
For the next two months, X was destabilized. The lack of servers caused meltdowns, including when Musk hosted a Twitter Spaces for presidential candidate Ron DeSantis. “In retrospect, the whole Sacramento shutdown was a mistake,” Musk would admit in March 2023. “I was told we had redundancy across our data centers. What I wasn’t told was that we had 70,000 hard-coded references to Sacramento. And there’s still shit that’s broken because of it.”You can't play the "I wasn't told" card, this is something that I would hope 99% of engineers would have warned about, and if Elon was the "coder" he claims to be he should have known himself
https://twitter.com/esthercrawford/status/168429104868268441...
also Gavin Belson was spot on
https://youtu.be/XAeEpbtHDPw?t=15
The recounted conversation involved multiple individuals, many of whom have an adversarial relationship with the subject and would be inclined to sue. Misrepresentations of facts could have meaningful reputational damage.
If you're the author you do not risk going to publication without verification that the gist of the conversation is accurate. Your editor literally will not let you. The publisher's reputation is on the line too.
Elon is the kind of deep-pocketed person that lawyers take cases on pro-bono against.
If you're doing your job properly you don't build things this way.
What it does show is how much dead weight there was at Twitter and how little was lost by letting it go.
Even that scale of the servers and monthly infrastructure cost is _absurd_ for the product Twitter is serving.
This is just tech debt. Every company has it, except apparently the ones you've worked for. I'll bet half of these issues are from the fail whale days when every day brought a new existential threat to the company's survival.
70,000 instances of the same hack speaks to a systemic failure of process and a company culture not focused on continuous improvement.
This is a public company used to being analyzed for these types of risks.
I also work at a public company. That's why I know that this shit is not okay.
You have to prioritise these things. At a certain point, you often do have to say “well, it’s not ideal, but we’ll know well in advance of it becoming a problem, deal with it then”. This approach is, of course, not compatible with the company being bought by an insane billionaire who thinks he’s an expert, but that’s a bit of a black swan event, and not something that _should_ form part of your prioritisation process.
Amazon also gets away with this because of following the Bezos API mandate and these issues each being in isolation. As long as everything keeps functioning at the "api calls between services" level the code can be sick with rot without killing the organism.
Some of the best people I've ever worked with come from Amazon though.
You could make the argument that Amazon has baked into their operating model the failings of human beings and the failings of the processes of teams.
Being behind the curtain might be a nightmare but being in front of the curtain might just be the best ticket in town.
> “This is making my brain hurt,” he said.
> “I’m sorry, that was not my intention,” she replied in a measured monotone.
> “Do you know the head-explosion emoji?” he asked her. “That’s what my head feels like right now. What a pile of f---ing bulls---. Jesus H f---ing Christ. Portland obviously has tons of room. It’s trivial to move servers one place to another.”
Have we read the same article?
Ah, yes, the “confident insistence on something you know nothing about” style of management.
This is The Office-esque.
If he wasn't told something vitally important, then the odds are the engineers he let go were ones that knew about that vital knowledge.
And if you think the scale and infrastructure cost is absurd for Twitter I'd like to see the services you've worked on. Right now.
This is the same company that took a nuclear-fucking-orbit dump on Rails because they didn't understand that your standard relational databases cannot handle their architecture pattern of very high write rate and very high read rate.
This is the same company where the same lead engineer's NIH-syndrome led to them writing their own bespoke message queue twice. And they threw it out twice.
And I do run 10^5 scale infrastructure -- globally distributed and not with any single building at some eye-watering $100M/yr run rate.
It may be a highly visited site, but if you tell me they actually need that much physical stuff to _host tweets_ then I'm calling bullshit.
Their quarterly operating costs _not including staff_ were over half a billion dollars. Twice what they were paying in salaries. That's a completely inverted cost relationship to how companies should operate.
Second of all, this isn't just '140 character tweets'. It's images, video, data being sent at staggering speeds because people tweet and they tweet frequently. And people want instant response, they want to see tweets instantly and respond to them instantly.
I highly doubt you do actually run infrastructure given that you're both misrepresenting what's actually going on here and taking the word of a person who in the article alone shows the various different ways of how he distorts reality. This isn't to say Twitter's infrastructure was great, because all services have heavy amounts of tech debt and cruft associated with years of fixes and updates. But rather that data being sent at high speeds with high reliability and at high frequency leads to high costs.
Pre-Musk, Twitter's quarterly infrastructure bill dwarfed both Cloudflare and Akamai's combined. All without even having paying custoemrs.
That's utterly ridiculous.
I'm not trusting Elon Musk to believe this either, it's all available to look up on EDGAR.
In TFA, people tell Musk that this is hard to do and will take time. Musk ignores this, demands that it be done at rapid speed, and doesn't even consider that there is user data on the machines.
He was warned.
The platform stumbled for a few months. Some people got bigmad. 80% of their most active users have stayed their most active users since the purchase.
I'm so reminded of Carl Icahn's story of firing an entire 9 story building full of people and not hearing a single complaint about it from anyone.
Everyone loves to tell you how critical they are. They even feel it to be true, but companies lay important people off all the time and the world goes on. People shouldn't be so invested in their jobs to define their value.
Musk himself blames the DeSantis Spaces debacle on this poorly planned server move. If that's not catastrophic, I don't know what is.
There's almost certainly entire teams of on-call engineers pulling heroic shifts because of this. I'm sure Musk DGAF, but those poor souls losing their holidays, weekends and time with their friends and family probably care.
I can't tell exactly where you lie, because you blast Twitter for having incompetent engineers, yet defend Twitter for also having incompetent engineers but this time under Musk. He 'supposedly' cleaned up Twitter (well, the employees, never mind the moderation policies when he personally unbans someone that posts CSAM), but runs it worse than anyone pre-acquisition did.
The problem is people like yourself (nothing personal) acting like there's sides to be on and that it matters. Twitter was a stupid company before and is still a stupid company after. Bystanders are stupid for being waaaaaaaaay too invested in the story and the outcome. Tribalism is fucking cancer.
The era of risk free money created a world-scale reality distortion field where people believed that a company like Twitter -- with no novel product, no paying customers, no growth trajectory and a cash burn rate that could put men on the fucking moon -- is a viable, thriving business.
Brand equity is only powerful for as long as you have money in your pocket and the rent isn't fucking due. And advertisers are fickle.
I don't know how anyone who read "The Emperor's New Clothes" as a child could give the slightest shit about Twitter, but here we are.
Gallant humbly admit his lack of knowledge, seeks to learn by asking people he employs, and doesn't take rash action without considering the consequences
Goofus thinks he knows it all, doesn't heed good advice, fires all those people he employs, makes ill-advised, unplanned changes, and blames the resulting problems on someone else
It's probably just the location itself that's valuable. Knocking down walls here and there doesn't actually make a functional bit of difference.
I'm not saying Musk isn't a stupid gorilla. I'm saying that anyone with a functioning brain could see from orbit that an organization at that size, with that product, with no customers, with that revenue and those expenses is deeply dysfunctional.
The fact that Musk stepped in with real dollars to assume responsibility for that is somewhere between charitable and nonsensical.
Gallant humbly admit his lack of knowledge, seeks to learn by asking people he employs, doesn't take rash action without considering the consequences, and honors the work put in by all the people who work there
Goofus immediately says everything and everyone is rotten without taking time to learn about them, fires most people without taking time to figure out what they do, singlehandedly causes site-wide infrastructure fires pretending he was as capable as the people he fired, never admits he was wrong to do so, continues blaming everyone but himself
these are personal qualities, or results of personal qualities. It doesn't matter if elmu's thoughts on the size of the company are wrong or right. He did all the things goofus did, and that's bad.
Like, your entire first paragraph is goofus: elmu had no idea of the state of things, he just kept causing problems, blaming them on others, and crying "look at the state of things!”
he fired the people who knew the state of things without asking them what it was, and he's not an engineer (hence all the fires he causes), so he certainly can't discern it on his own
and the fact that elmu bought twitter after desperately trying to renege on his commitment to do so was purely a result of court action (which he fought against) making the purchase inevitable
Life is full of people who make bad decisions and disclaim responsibility afterwards.
It’s a company in serious financial trouble sinking and the CEO making terrible micromanaged decisions and railroading and bullying his experts.
Straight NO to working for Musk.
I loved the Walter Isaacson Steve Jobs biography but zero interest in reading this book.
Really, Musk sounds like an amusing guy to work with. "Fuck, let's do it" is an attitude more execs should take.
Twitter has been one of the most expensive human undertakings in history and it's just so mediocre. If it isn't actually harmful to us.
In fact since Musk bought it, people have been tripping over each other rushing to proclaim just how mediocre and unnecessary it is. So much so that they don't even need it anymore.
And yet when anyone points this out, these same people expend an enormous amount of energy pillorying that person and shouting incoherences about Musk.
However less i like Musk, i have to agree that he upset a lot of people by clasifying propaganda outlets as propaganda. /s
That's why, from a mostly positive person, he became a negative one, also on HN.