An Update on Our Outage

By the time a company gets large enough to be a household name, they have usually been burned a few times and have someone technical who can push back against short-termism from POs.

Either you listen to and trust your people or your don't.

> Companies I’ve worked at that sucked had awful internal communication. It was all very friendly, but it was all platitudes and euphemisms, dumpster fire technology implementation.

Honestly, I think part of the problem is people going around talking so much about soft skills and how people don't like to be treated harshly that they've forgotten you need hard skills and nearly every profession where they need real leadership you get told where you screwed up and how to fix it. Literally, at the company I work for, people screw up and all you hear about is positive things

Guessing its SoFi.

"EA Spouse" is the name is a specific period during the early 90's when Electronic Arts burned out an entire generation of game developers.

You can have knowledge silos in either kind of company. We didn't magically get better at spreading work around or writing documentation just because we went remote.

This has nothing to do with remote.

Remote teams are usually concentrated in the same timezone too. In-office global companies make a point of having at least a few teams owning a service for this very reason.

Just one million? About 6 months, as I currently earn over $2MM per year from my options alone at the company I work for thanks to their IPO. Once those dry up it's going to be a hard sell for me to stay. Maybe I'll coast for a year on cash + RSU salary (which is much less than salary plus pre-IPO options) to top off my $5MM investment portfolio and ensure enough of it is invested to produce an income, but after that? No thanks.

By that logic Facebook certainly wouldn't be a tech company either.

Now probably, but at the time the FAANG acrnonym was conceived, Netflix Originals and other productions ran internally was a drop in the ocean compared to their licensed content, similar to how Apple has controlled music (before streaming), so it made sense.

Because FAANG was coined as a term for super high growth stocks, not for anything to do with technology.

NFLX was founded in 1997. They didn't have original content until 2013.

Before their original content I seriously considered shorting their stock. Their business model without original content was structurally flawed. They effectively had a maximum profit they could earn. Literally anything they did to increase their profit would be taken by the content producers they were forced to buy the licenses from.

Creating original content was a necessary step to survive just like pivoting from their original DVD mailing business to streaming. They're still a technology company though.

It's hard to define any company as a 'tech company' if you're willing to define Netflix as a Media Company.

In some sense, I agree - Netflix, Disney, and Pixar are media companies that use technology to distribute, to recommend, and to produce their content.

But in the same sense, Google is an advertising company that uses technology to sell and to show their ads (and they have a few loss-leading tech ventures like search, mail, a phone OS, and a browser - all of which they give away to more effectively show those ads). Facebook is another advertising company, they happen to show and profile their ads using their social network tech. Amazon is a retail sales, warehousing, and product distribution company that happens to use a lot of technology. They also have a "tech company" division that sells web services, but that's only 13% of their revenue.

Similarly, Walmart is a retail sales and warehousing company (as well as product distribution now, but most recognizable by their brick-and-mortar stores) that uses technology in every aspect of buying product, shipping it to stores, deciding where and how much of it to stock, how to price it, and when to restock it, just like Amazon. There was a time when Sam Walton walked around, squinted at the shelves, and made notes about what and how much to order on a clipboard, but that's long past - it's all tech now. Automotive manufacturers are other very big tech companies, they build and sell automobiles but every step of that design, fabricate, market, and sell process uses technology.

Apple, at least, sells technology hardware. Microsoft also sells technology hardware and software, but they did so before FAANG were big so they're not in the list either.

The list is arbitrary. Software and technology is eating - has eaten - the world, and management of it is critical to every business.

> How Netflix got lumped into

Otherwise it would be a slur.

Two ways I would argue for Netflix being a tech company.

A critical question is, how and how much do they pay their developers (and other tech folks)? I believe Netflix compensates their tech workers more richly than the average media company. I am certain of it if you take an average over the past 10 years.

The other argument is, when we say tech, we really mean qualitative innovation. Did Netflix disrupt an existing 800 pound gorilla using technology? The answer is yes, they killed Blockbuster by embracing tech.

NERC created a report after the Texas 2011 ice storm power outage with recommendations for ERCOT generators to improve their weatherization standards [0]. In that report they referenced not only the 2011 incident but previous incidents where the grid generators failed to adequately prepare for potential problems due to weather related issues.

In the 2021 storm, hundreds of people died. Comeuppance is not the right word. I don't know the right one, but I believe it's closer to willful blindness or predatory delay.

[0] https://www.ferc.gov/sites/default/files/2020-04/08-16-11-re...

So. Do you think Texas should winterize or not?

But most people use them because they're required by something else, like Vault, or Solr, or some other tool built to require a particular distributed key value store.

Kafka still requires it, at least for all production clusters. In a few years the recommendation might be the new internal Raft based consensus though.

So relating to things I do know about, consul is like cloud infra, vault is a service for secret storage and nomad is like Kubernetes?

Thanks for the explanation. I see how that would cause a cascading failure.

> Consul is a service mesh. It’s your dynamic service discovery and routing layer.

How does this differ from DNS?

This dynamic service terminology sounds very familiar for someone with a few years in the feature film VFX/Animation industries - however, it would not surprise me if Consol/Vault/Nomad were created without a review of similar situations and solutions in other industries.

For those unaware, the feature film VFX/Animation industry has been global and performing large scale technologically ambitious projects requiring multi-company tech collaboration for decades. And not just with code, with assets too: visual assets of all kinds, audio in every possible form, and in some cases legal documents with new approaches to industry issues. Plus these media studios tend to have proprietary workflows, so there is very sophisticated file formats to contain all these information in agnostic manners. All this deep collaboration across creative technical organizations that do not trust one another has developed scalable solutions which the web and formal software development is completely unaware.

You don’t need deep etcd experience to run k8s. Why is that bad?

Abstracting things away is literally the entire point of programming.

yep.

This totally happens, and is a thing, but tracing a "unexpected change in metrics on our DB server to "increase in metric showing call volume to DB server" to "increase in metric showing increase in fleet size of app servers that call the DB server" takes minutes to identify, not 24 hours.

And the remediation procedure is quite simple too - scale down. If the fleet can't handle it, increase your throttling. If even with increased throttling you can't handle the load and everybody's timing out (but you still can't scale out), you start spinning up new load balancers, and applying weights in DNS where at least some percentage of your callers are getting through while others see full failure.

The key point is in anything related to such an outage, even if it takes a long time to recover, your customers shouldn't see a full outage.

So it was either an entirely different class of problems or Roblex has a huge ops skills gap. I'm betting the former.

edit: never mind, maybe it's the latter after all: https://twitter.com/NIDeveloper/status/1454773313792880640

I think there are two big things that can be really hard with scalability -

1) There are a lot of dimensions

I think it is fairly common to focus on a throughput-related dimension like requests per second and test that very thoroughly, while not paying enough attention to other aspects. We have services where, if we were told requests per second were going to go up 100x tomorrow, I wouldn't be too concerned, but if we were told request size were going up 2x, I'd be freaking out.

Even when you think you've identified all of the dimensions, there are usually ones you missed and/or they interact in weird and unexpected ways.

2) Performance can appear to be linear when it really isn't

So many outages I've seen have resulted from everything being fine until some threshold is hit, at which point it doesn't start to slowly degrade, but instead immediately explodes. Often times due to feedback loops (GC activity in garbage-collected languages often can behave this way) or because some cache overflowed (data that is queried at a high rate no longer fitting in memory on a DBMS comes to mind).

I worked at a place with a few thousand servers.

Because the business had been around for a while, there was two separate ways to deploy and manage apps: old and new.

We reconfigured how code was deployed within and between racks for increase resistance to racks or even whole colos dying. This change was made in the new deployment system. And all was well for most of a year. Software and services gradually migrated from old to new as people had the time and inclination.

The new deployment also packed services more tightly, so most computers kept copies of most binaries on them.

Then we crossed one of those tipping points. On a code update, several thousand servers across the world demanded a small set of binaries from a distributed data node via the newer deployment method. Which would be fine, except we had degraded service from a couple TOR routers due to a separate bug. And so the distributed data nodes hosting this particular set of binaries were less "distributed" and more "just one poor about to be overloaded machine and network."

So a few thousand machines demanded a couple hundred megs nearly simultaneously. And when they didn't get it in a timely fashion, all triggered a rollback. Simultaneously. Which demanded another couple hundred mb of code. Simultaneously. And then the freakouts started.

And because other deploys were in flight, there was more than one service deploy ongoing. And they all started fighting each other for bandwidth.

Or a compiler bug...

http://r6.ca/blog/20200929T023701Z.html

And Roblox is already well beyond the luck threshold

Apple is the worst of them all. They are much more restrictive and arbitrary with their app store policies (see previous HN articles over the last 5 years), they have done much more to lock down their devices to prevent circumvention of things like banning emulators, and they don't license the apple device model to other vendors, so they are the true definition of a walled garden. Google has plenty of problems too, but at least random device companies can make android devices, and third party app stores are (sort of) allowed to exist.

I assume they have a deal with Apple and Google for less than 30% on in-app transactions. That probably means Roblox is getting between 10-20% of the transaction.

Right, that's why they get an extra 2%.

Any scenario where a single company controls an entire marketplace I do believe that company's markup should be limited to 5% to prevent abuse, yes. If there are multiple marketplaces serving the exact same devices, and with equal status, then that's another story, and price-fixing is already illegal, so that situation is covered. Vendor-specific app stores on platforms where third party app stores are not allowed or are centrally discouraged in any way are monopolies and should be treated as such. It isn't enough to say that android apps compete with ios apps -- we aren't talking about selling iphones and android phones, we're talking about selling vendor-specific apps, and these are different ecosystems. Apple is sitting there gatekeeping everything allowed in their ecosystem and charging an extremely hefty toll, in exactly the way a monopoly prefers to, because they have made sure competition is not allowed. On that note, I also think this level of gatekeeping should not be allowed, but that's a whole other rant.

There is nothing Apple (or Google, or Robolox) does to justify more than a 5% cut. 5% is generous. In many cases they don't even take the time to have a human review every submission. You can say they built the platform, but aren't we already paying for that with our 5%?

And to your point, yes, when Zoomers reach 50, we'll see awesome things like companies losing the rights to their IP if they submit 3 bogus DMCA claims, extreme regulation of things like content-ID that threaten the existence of fair use in practice, outright banning of all lobbying and corporate political donations, things like this, etc etc.

> Do you think any markup of more than 5% is unreasonable? What about Apple's physical products? What about ... every retail store in existence?

Retail stores maintain a physical presence and have actual overhead, as do online retailers that have to maintain warehouses, subsidize free shipping, etc. What is the overhead on an app submission, or an app purchase? Charging 30% overhead for $0.0012 cents of bandwidth, an algorithmic review of your app that only might be supplemented by an actual human review, and a literally free file copy operation comes nowhere close to that. This is a lot closer to payment processor fee territory in terms of the actual work being done by the provider.

You would think they would do more for their app developers with such a hefty fee, but they don't, so they need to be regulated. These fees are just theft.

In the case of Roblox, you can argue "but what about the platform?" but 5% is still a metric-crap-ton of revenue and is more than enough to keep their platform afloat (especially if they don't have to give 30% upstream to Apple/Google). We've just become accustomed to ridiculous % cuts because monopolies have normalized this practice. In the early 00s, the idea of an app store even taking a cut was viewed as laughable. Then it became normal.