The Day Elon Musk Decided to Rip Servers Out of a Data Center | Hacker News Reader

The Day Elon Musk Decided to Rip Servers Out of a Data Center

techdirt.com

32 pointsghewgill23 comments

I know the tone of the article was to be critical, but honestly, I found myself identifying with opposite emotions reading it. There's a lot of Musk's arrogance that really puts me off, but I do appreciate his willingness to push back on the BS and waste in our industry.

I actually was also inspired by it.

The reality is you can take months to plan it , have it be really expensive to try to get 100% accurate, but that may not be the most efficient way.

It may be better from a trade off perspective to take this approach way more frequently than we realize.

> tone of the article was to be critical

I mean... it's Musk. The shit-on-Musk phenomenon has transcended normality. Reddit has about six subreddits that do nothing apart from constantly shit on him. Twitter is infested with users/bots that do nothing apart from post the 'this mf paid for twitter' meme. Articles like this come out week after week after week. And yet, the site stays up and the products, by and large, work.

It's just one of those things that presumably makes sense to somebody but doesn't to me.

> It's just one of those things that presumably makes sense to somebody but doesn't to me.

Normal journalism at work: one people pays, the other one writes.

I mean, that's the usual answer when it comes to yellow journalism but it still leaves the question of who's paying and why in particular him, because the Musk ecosystem is not one that's particularly hard to avoid; if you don't own a Tesla and don't use Starlink you're 99% of the way there.

It's just odd the amount of online hate the guy garners considering how narrow his reach really is.

Have you worked in datacenters? This isn't pushing back on BS and waste, it's ignoring user privacy, security, server reliability (and lifespan), site reliability, vendor relationships and in a lot of cases actual laws.

The shit I get when I bring a small piece of cardboard into the DC here. The strict rules are there for a reason.

This, I really saw nothing there that I would not have done, if I had the money to pay for damages. The floor caving, losing a rack, causing a surge, etc, are small risks comparing to that he was trying to save, and to what he was trying to prove to his team. The only thing that would not be tolerable was having someone die in the process, but then he went under the floor himself. I respect that and really I think the article seems to be written by someone in infra that wants to protect some form of status quo?

The fact is that there are "rules" because they are needed to cover all the possible ground. Yeah, one lone operation can skip on almost all the rules and still be successful, but that doesn't prove anything. Rules are needed at scale, to try to cover for unlucky coincidences, people not that smart, unforeseen (by you) consequences etc etc

I guess the point is to find opportunities to do better than the average, not that the average should not exist. That’s often where competitive advantages arise. Most people can’t identify or execute on them because rules.

>What I wasn’t told was that we had 70,000 hard-coded references to Sacramento

There's plenty of real world scenarios that could have taken out that data center and the Twitter engineering team would have been scrambling to fix, just the same.

I read somewhere once that v1 of chaos monkey was the Netflix CEO randomly pulling cables in a data center to prove the point that systems needed rugged designs. Not all too different.

I regularly show off my work as an SRE by hard-terminating a machine in my client's AWS infrastructure. If everything is right, there's no impact - I pull up their products and test them as I'm doing so.

It's a great test and if you do it with enough confidence it really sells your skills.

The author of this article is a moron. He's constantly saying how bad Musk's actions were while describing how everything worked out! Look, guy, there are some big lessons here that just went right past you...

Yeah I thought it was ironic when the author was talking about the lesson Musk somehow missed.. while missing the lesson that it actually worked.

Wow, I would never hire you with such a reckless attitude. “Turned out ok” is not an acceptable excuse for high risk behavior. I think your insult is icing on the cake. GTFO

> Wow, I would never hire you with such a reckless attitude. “Turned out ok” is not an acceptable excuse for high risk behavior. I think your insult is icing on the cake. GTFO

What would be an "acceptable excuse for high risk behavior" ?

I thought the idea was to learn from past mistakes, not to find scapegoats.

I'm assuming you missed the quote about how Musk said it was a bad decision and it lead to outages? You must have also missed the blantant privacy violations?

From the article:

> The article does note that Musk himself eventually said he shouldn’t have done this and it did cause a fair bit of problems for the site, including the disastrous “Twitter Spaces” where Ron DeSantis tried to launch his Presidential campaign.

I mean, the DeSantis thing was _extremely funny_, and very much a gift to the internet, but I doubt Musk would think of it as 'working out'.

However, even if it _had_ worked out without incident (it did not), it was _still a very bad idea_. If you jump out a third floor window, well, your odds of surviving that are actually _not bad_, but it's generally advisable to take the stairs anyway.

Question: if one day truckloads of servers suddenly arrive at your workplace from another data center (they were not gracefully shutdown) and you're told to bring them online asap, what are the steps you'll do to bring them online again and make sure all applications running on those servers start again? Not sure how many servers in those trucks, but the old data center contract was $100MM/year. How long it'll take you to do it? Your team is also recently hit with a terrible round of layoff, leaving only a skeleton crew.

Plug them in, turn them on, see what happens. :')

Comments from when this was posted yesterday.

https://news.ycombinator.com/item?id=37486822

'Still up tho'

Elon musk is an idiot, and deliberately so. Most fan noise on the net is paid, I reckon

Reply on news.ycombinator.com