Hacker News Reader

Comment by slg | Hacker News Reader

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

View on HN

Not necessarily directed at roblox but honestly I’m surprised this doesn’t happen more often given how many teams I see run software they don't understand or don’t even have access to its source code.

Edit: but yeah must’ve been tough 72+hrs i hope their version of reliability team can use this to bash the support they need out of the management and not get scapegoated instead

I think it happens all the time. There just aren't that many services, relatively speaking, that you'd hear about when this happens. The more-popular services (that you'd hear about) probably already have a bias toward having their stuff better in order, since it's often also really just a matter of making the right investments.

> really just a matter of making the right investments

Right. But the person deciding on those investments is a product owner who's looking to release new features because that's what management wants. At least that's my experience more often than not.

My guess is that the bias might be the opposite. Maybe these companies become more popular because they are reliable and have the mindset to focus on that.

By the time a company gets large enough to be a household name, they have usually been burned a few times and have someone technical who can push back against short-termism from POs.

Either you listen to and trust your people or your don't.

At my company there was a service outage that lasted 2-3 weeks for a specific feature we have. This was caused by everyone quitting and no one having any experience with this service. The rest of the application remained working so it wasn't so noticable to the outside world. But internally and for customers it was massive since it was the billing system that went down.

There was another incident that took down everything that was caused by a Spring cloud config being changed that no one in the company had access to.

It’s been known for a while that human communication is the real impediment to technical development.

Companies I’ve worked at that sucked had awful internal communication. It was all very friendly, but it was all platitudes and euphemisms, dumpster fire technology implementation.

Phone and web apps are basically librarian work these days. If a businesses tech stack is having issues it’s human communication that’s the real problem.

> Companies I’ve worked at that sucked had awful internal communication. It was all very friendly, but it was all platitudes and euphemisms, dumpster fire technology implementation.

Honestly, I think part of the problem is people going around talking so much about soft skills and how people don't like to be treated harshly that they've forgotten you need hard skills and nearly every profession where they need real leadership you get told where you screwed up and how to fix it. Literally, at the company I work for, people screw up and all you hear about is positive things

So true. And also at the systems level. In most places I've seen, the #1 priority is hitting arbitrary executive feature/date goals, not maintaining robust systems. At some point, the shit will hit the fan, causing a "Why didn't you do perfectly the thing that wasn't a real priority?!?" reaction and a temporary lurch toward robustness. Although often the lurch will be less about actual robustness and more toward performative addition of control mechanisms like more layers of review.

I remember one large company I consulted for in the mid-aughts. Before I got there, they had an outage so severe it was on CNN and caused a short but notable dip in their stock price. By the time I got there, the ops people were absolutely dominant. All changes had to go through their review board, and woe be unto any project that they raised an eyebrow at.

Of course, the real problem was that developers were scheduled out 18 months in advance to work on a list of projects they hadn't been involved in estimating and where often they hadn't seen the code before being swapped onto the project. Dates were absolutely not allowed to slip, so it came out in frantic overwork, which meant a lot of bad/confusing code. Meaning more likelihood of failure and no time to think about systemic issues and impacts.

I worked at a big fintech company about to go public in a SPAC worth $4 billion. They were a unicorn when I was there. Literally everything was in a GCP MySQL Database......and they refused to pay for a hot spare. They had backups but nothing for redundancy. We had downtime almost every week.

Guessing its SoFi.

SoFi's devops function is extremely robust, if overcomplicated, and they've got redundancy for all their application servers and production databases. They're also an AWS shop, not GCP.

Lol, I can just hear the politicians now.

"Tough on bugs, and tough on the causes of bugs!"

As they add even more time-sapping process to every release.

This is a benefit of working remote: I work at Automattic and have worked on teams where the members are literally around the world. When shit hits the fan, people are able to hand-off this kind of work to people who are fresh after a good night’s sleep and no one needs (or should need to) work crazy hours.

Automattic always sounds awesome to work for - I know a couple of people who work there as well.

In your experience, when something like this happens, what can leaders do to help in the moment? I'm an engineering manager, and watching people getting over-stressed is much easier to do when you're physically in the office. Working remotely, it's easy to slip in the trap of either assuming no news is good news, or the opposite approach of asking "are you OK?" every 2 minutes. The only solution I've found is to all sit on a call, and pay attention to people's tone of voice, but I don't know whether that actually contributes to people's stress levels.

There's avid communication and coordination is all done via text (Slack/IRC). As long as you're following along, I think you can spot it pretty well. People are also more used to managing their own stress and time. They will let the people know if they need to step out for a few minutes so the workload could be shifted/paused to give them time to step away.

The honest truth - if you're an engineering manager and you're no longer anywhere near the code then just at the heat of the moment there really isn't much you can do, aside from what you've been doing: listening on the call and doing a minimal check-in.

The main thing you can do is after all is over to make people feel awesome, give them recognition in all the ways possible: bonus, extra vacation days to compensate for the crunch time, etc

But if you still have your technical chops - jump in the trenches and debug!

Best thing managers can do in these situations is deal with internal politics - be the gateway for information going higher up the chair. The last thing the actual engineering team needs is direct calls from C-level execs.

One of the greatest values of good technical manager is to run interference/shield for your team. Establish a chain of communication, or if there isn't one in place BE the active reporter, to ensure your hands-on-keyboard people aren't interrupted every 5 minutes (and this is a realistic timeline) for updates from various managers, execs, C-levels, and so on.

Otherwise, it's mostly things you can do before and after an incident like this.

Depending on your role and place in hierarchy, yes DO sit on calls, even if you are quiet; to understand what's going on, to help prioritize if needed, to see if your experienced "spidey sense" catches some risk others may not even though they're closer to the problem (or precisely because of it), encourage, guide, and focus; but all of that without being overbearing or making them feel watched. You need to feel like an ally, part of the team, a friend as opposed to political officer. This again is best achieved ahead of time by demonstrating your value to the team by being supportive and effective always.

This is a benefit of a global company, not working remote. You could do all that stuff in multiple offices around the world.

In a global but not-remote company the team responsible for the particular service that failed is probably still concentrated in one office. Bringing in people from other offices who aren't familiar with the problem service probably isn't that helpful.

You can have knowledge silos in either kind of company. We didn't magically get better at spreading work around or writing documentation just because we went remote.

This has nothing to do with remote.

Remote teams are usually concentrated in the same timezone too. In-office global companies make a point of having at least a few teams owning a service for this very reason.

From my experience there are a few engineers that are usually key to fixing such issues (not always the same people, depending on the failure). Handing-off work is more for day to day work like handeling a support ticket. So the key engineers need to work crazy hours when things like that happen. But that's OK, ideally these things are rare and people need to work crazy hours only in those rare situations. I'm super protective about my evening/weekend time but I wouldn't have any problem working the entire weekend like crazy when something like this happens. If it happens too often I will consider finding a better place to work.

At Automattic, we handle this by offering a 3-month sabbatical every 5 years and a “minimum” 20 day a year vacation policy. The idea being that no one person should hold knowledge to key systems for too much long.

I wonder if Engineers from Roblox are now worth a little more just because they have this experience.

I also think it is time that people should take a look again at Chaos Engineering [1] from Netflix. It is sometimes ironic that the best technology often comes from companies that aren't a technology company at all.

[1] https://principlesofchaos.org

And I wonder if their tech management is worth less for this clusterf** to happen during their oversight?

This is how I felt reading his comment. Were Volkswagen emission engineers and the captain of the Exxon Valdez worth more after they screwed up?

Are you implying Netflix is not a tech company?

I definitely see Netflix as a Media Company, much like Disney. Disney and Pixar have great tech too, but they are not tech company in any shape or form.

How Netflix got lumped into "FAANG" aka Big Tech is still a mystery to me.

By that logic Facebook certainly wouldn't be a tech company either.

Why not? Didn't Facebook create tools like GraphQL and React?

Netflix has created tools and open source libraries too.

RxJava was opensourced by Netflix, the defacto standard for reactive programming in Java. They also open sourced a lot of their work for working with avif files, which has managed to find its way into Chrome, among other projects.

Among other things.

they do occasionally use BSD licensed software so they can choose what/when/if to open source.

Facebook dont consider themselves as tech companies. But there aren't any other category that are clearly defined in which they fit in. Social Media isn't one.

I wonder how long it will take until people adopt their new name, and MAANG

might as well go for MAMAG at that point, swapping Netflix out and Microsoft in. Or MAMANG. Or even AMMANG. Possibilities are endless. :)

MAGMA. You heard it here first

The new offical acronym on the street is MAMAA by the way. Microsoft, Apple, Meta, Amazon and Alphabet.

Jim Cramer announced this on Friday.

Now probably, but at the time the FAANG acrnonym was conceived, Netflix Originals and other productions ran internally was a drop in the ocean compared to their licensed content, similar to how Apple has controlled music (before streaming), so it made sense.

Because FAANG was coined as a term for super high growth stocks, not for anything to do with technology.

NFLX was founded in 1997. They didn't have original content until 2013.

Before their original content I seriously considered shorting their stock. Their business model without original content was structurally flawed. They effectively had a maximum profit they could earn. Literally anything they did to increase their profit would be taken by the content producers they were forced to buy the licenses from.

Creating original content was a necessary step to survive just like pivoting from their original DVD mailing business to streaming. They're still a technology company though.

It's hard to define any company as a 'tech company' if you're willing to define Netflix as a Media Company.

In some sense, I agree - Netflix, Disney, and Pixar are media companies that use technology to distribute, to recommend, and to produce their content.

But in the same sense, Google is an advertising company that uses technology to sell and to show their ads (and they have a few loss-leading tech ventures like search, mail, a phone OS, and a browser - all of which they give away to more effectively show those ads). Facebook is another advertising company, they happen to show and profile their ads using their social network tech. Amazon is a retail sales, warehousing, and product distribution company that happens to use a lot of technology. They also have a "tech company" division that sells web services, but that's only 13% of their revenue.

Similarly, Walmart is a retail sales and warehousing company (as well as product distribution now, but most recognizable by their brick-and-mortar stores) that uses technology in every aspect of buying product, shipping it to stores, deciding where and how much of it to stock, how to price it, and when to restock it, just like Amazon. There was a time when Sam Walton walked around, squinted at the shelves, and made notes about what and how much to order on a clipboard, but that's long past - it's all tech now. Automotive manufacturers are other very big tech companies, they build and sell automobiles but every step of that design, fabricate, market, and sell process uses technology.

Apple, at least, sells technology hardware. Microsoft also sells technology hardware and software, but they did so before FAANG were big so they're not in the list either.

The list is arbitrary. Software and technology is eating - has eaten - the world, and management of it is critical to every business.

> How Netflix got lumped into

Otherwise it would be a slur.

Two ways I would argue for Netflix being a tech company.

A critical question is, how and how much do they pay their developers (and other tech folks)? I believe Netflix compensates their tech workers more richly than the average media company. I am certain of it if you take an average over the past 10 years.

The other argument is, when we say tech, we really mean qualitative innovation. Did Netflix disrupt an existing 800 pound gorilla using technology? The answer is yes, they killed Blockbuster by embracing tech.

Since the idea of a "tech company" comes up so often, I'll just leave this here:

https://news.ycombinator.com/item?id=27693634

Not a software outage, but the Texas ice storm of February resulted in me not receiving mail for two weeks, not being able to use the streets for at least a week (they never plowed and it just eventually melted), rolling blackout where I had power for about four hours a day, and no Internet service at all for the first two days. No water for most of the first day.

I'd argue keeping basic city services and roads operational is more important than a gaming platform, but city and state governments in the US often seem to not agree and are more than happy to save money skimping on maintenance and resilience.

That behavior is a reflection of their citizenry who have been convinced by propaganda that government can never be effective at anything and therefore should not be funded properly.

>”citizenry who have been convinced by propaganda that government can never be effective at anything and therefore should not be funded properly.”

People are treating the Texas ice storm like it was some comeuppance event but the truth of the matter is that Texas rarely experiences this kind of winter weather and that’s the reason why it didn’t handle the big freeze the same way as a midwestern or northeastern state would.

The reason why cities like DFW do not maintain a fleet of snowplows isn't because of anti-government propaganda, but because it is completely uneconomical to do so. Even during the big freeze there wasn’t a foot of snow on the ground. The major roadways were salted, though, and you could drive on them.

Snow and icing events tend to only happen for a day or two each year in my part of Texas. And it is often the case that when temperatures do dip below zero there isn’t preceding rain or snow. But when that does happen, most people just ride it out for a day or two and wait for things to melt.

NERC created a report after the Texas 2011 ice storm power outage with recommendations for ERCOT generators to improve their weatherization standards [0]. In that report they referenced not only the 2011 incident but previous incidents where the grid generators failed to adequately prepare for potential problems due to weather related issues.

In the 2021 storm, hundreds of people died. Comeuppance is not the right word. I don't know the right one, but I believe it's closer to willful blindness or predatory delay.

[0] https://www.ferc.gov/sites/default/files/2020-04/08-16-11-re...

It's a joke at this point: https://www.texastribune.org/2021/09/28/texas-power-grid-loo...

So. Do you think Texas should winterize or not?

Skimping on resilience has been done for decades as part of "cost cutting," in large part because even when we had staffing for resilience, we found that people would collect their check and not actually do the maintenance.

In the game industry, working yourself to death for 4 days, that's called a hot fix ;)

And that is after you've been working 6 days a week for the last 9 months. As an EA spouse survivor - fuck that industry.

“Spouse survivor”, I like that! I worked on the tech/distribution/data reporting side of the Covid vaccine and consider my wife, and all the sacrifices she made and extra things she handled up on while I was on the keyboard and phone, a critical component of my team’s success.

"EA Spouse" is the name is a specific period during the early 90's when Electronic Arts burned out an entire generation of game developers.

Here is the source of the term:

https://ea-spouse.livejournal.com/274.html

People working on the cloud infra are not the people working on games.

My favourite was the Skype outage in 2007 where a peer to peer algorithm didn’t scale - they had to work this out, and change the algorithm to fix it. It took ages!

https://www.wired.com/2007/08/update-skype-ou/

The last outage of this kind I can even remember is the summer PSN was down for like, a month?

Although that PSN downtime probably fails into a different category as it was a response to being hacked. Sony intentionally took the service down and kept it down to investigate and fix their security holes. Basically the equivalent of temporarily closing a brick and mortar store after a robbery compared to being involuntarily closed because no one could find the right set of keys to unlock the store's front door.

looks familiar to me also. was roblox also hacked?

From the blog post it sounds like no. They say a service got overloaded due to an increase in the number of datacenters and triggered a bug.

We have hospitals being knocked out for weeks after ransomware attacks. If we complain about roblox I think we should reflect on our priorities.

Why not complain about both?

You can, wasn't the point I tried to make. I tried to point out the arbitrary expectations here. Depending on the fault such downtime can last a few hours to a few weeks. I don't think you can make conclusions on architecture in this case.

Perhaps it would be possible to roll back everything in just a few minutes but some users might lose a lot of work. If such special cases are to be resolved, it may take a lot of time. You just cannot really infer quality of architecture by downtime alone.

Not of this size or scale but ransomeware attacks typically take days to resolve. The pipeline attack and the large multinational vet company come to mind.

It won't be a doozy: it will be a few paragraphs of general statements without saying anything specific, just like this communication (the first of consequence in days, another big failure on their part).

Roblox has to start taking infrastructure a lot more seriously than they have so far.

> What was the last service of this size that was down for 4 days

Was "only" two days, but Fortnite just shutdown for 2 days as a part of an end-of-season event back in 2019 https://www.polygon.com/fortnite/2019/10/13/20911691/fortnit...

https://www.polygon.com/fortnite/2019/10/13/20911691/fortnit...

The immediate technical root cause and the true root cause (often down to poor decision making among the senior technical staff and management) are two different things, and I suspect the public and internal post-mortems both will focus on the former rather than the latter, sadly.

I am sure they got a lot of stock.

I mean I'd gladly pull an all-nighter if I were a millionaire thanks to the thing I'm fixing. I think. Then I'd quit and do something leisurely.

Over what time frame would you tolerate poor working conditions to make a million bucks? Four year equity vesting? A million dollars in 2025 too, not a million dollars in 2010.

Just one million? About 6 months, as I currently earn over $2MM per year from my options alone at the company I work for thanks to their IPO. Once those dry up it's going to be a hard sell for me to stay. Maybe I'll coast for a year on cash + RSU salary (which is much less than salary plus pre-IPO options) to top off my $5MM investment portfolio and ensure enough of it is invested to produce an income, but after that? No thanks.

Reply on news.ycombinator.com