Defcon: Preventing overload with graceful feature degradation (2023)
micahlerner.com
micahlerner.com
1. It auto restarts all workers in the cluster in "maintenance mode".
2. A "maintenance mode" message shows on the homepage.
3. The top 100 pages by comment volume will still render their comment threads, as a job on each edge node recalculates and stores this on disk periodically.
4. Logging in is disabled.
5. All db calls to the driver are stubbed out with mocks to prevent crashes.
6. Comments can still be posted and are added into an on-disk queue on each edge node.
7. When the system is back online the queue is processed (and stuff checked for spam etc like normal).
It's not perfect but it means in a lot of cases I can completely turn off the DB for a few minutes without panic. I haven't had to use it in over a year, though, and the DB doesn't really go down. But useful for upgrades.
built it on my couch during a Jurassic park marathon :P
Previously I worked on a big web app at a growing startup, and it's probably the sort of thing I'd start adding in small ways from the early days. Being able to turn off unnecessary writes, turn down the rate of more expensive computation, turn down rates of traffic amplification, these would all have been useful levers in some of our outages.
I've run into many situations where something was deemed costly, is found out later, and the team ultimately has implement it all while hoping no one groks that is was predicted. "Nobody ever gets credit for fixing problems that never happened" (https://news.ycombinator.com/item?id=39472693) is related.
Those hours are directly expensive when your developers cost hundreds of dollars a day; and have a material opportunity cost in that their commitment to one particular project delays the delivery of other features.
Moreover, any new features would have to be made compatible with the graceful degradation pattern, creating an ongoing cost.
My point isn't that we all need to make dams that can hold up for a century. The point is that you hire an engineer because you want someone with the judgement and expertise to apply the correct amount of engineering to any given solution. Over-engineering is on the pathway to correct-sized engineering. It's the experience, discovery, and exploration required to arrive at choosing what things actually do not need to be done.
When your manager asks you, "do we really need to do that?" It's the expert that can explain why it really is necessary, and the professional who accepts "we're not going to do that" as an answer. And if they still feel it would be harmful not to do it, then that's where professional duty kicks in.
Just spending a few moments to consider whether queues should grow, block, or spill when adding them makes a big difference, along with choices in error handling. You can get a lot of things to gracefully degrade for free if that's a part of your decision-making process.
Consumer apps that scale to hundreds of thousands of users with five 9s+ uptime requirements are very rare.
The AdMixer was an "optional" response for the search page. If the ads didn't return before the search results did, the search would just not show ads, and Google wouldn't get any revenue for it. Showed the premium that Google of the day put on latency and user experience. I think we lost a few million per year to timeouts, but it was worth it for generating user loyalty, and it put a very big incentive on the ads team to keep the serving stack fast.
No idea if it's still architected like that, I kinda doubt it given recent search experiences, but I thought it was brilliant just for the sake of aligning incentives between different parts of the organization.
Applying rate limiting, selective dropping of traffic, even just monitoring things by how much they affect the user experience, all require knowing what each one is doing. Figuring that out for one takes very little time. Figuring it out for 700 made it a project we'd never do.
The way I'd start with this is just by tagging things as I go. I'd build a lightweight way to attach a small amount of metadata to URL handlers/RPC handlers/GraphQL resolvers/whatever, and I'd decide a few facts to start with about each one – is it customer facing, is it authenticated, is it read or write, is it critical or nice to have, a few things like that. Then I'd do nothing else. That's probably a few hours of work, and would add almost no overhead
Now when it comes to needing something like this, you've got a base of understanding of the system to start from. You can incrementally use these, you can incrementally enforce that they are correct through other analysis, but the point is that I think it's low effort as a starting point with a potentially very high payoff.
Facebook claims to have 2 billion accounts but no where near 2 billion unique accounts. I don't know what facebook calls an active user but it use to mean logged in once in the past 30 days.
1: https://www.statista.com/statistics/346167/facebook-global-d...
2: https://www.innertrends.com/blog/active-users-measuring-busi...
"For example, an active user can be measured as a user that has logged back into her account to interact with the product in the last 30 days."
Even the marketing material is designed to confuse.
And yes, that's a claim that if each user is a separate person, >20% of the world's population interacts with Facebook at least minimally each day. You can add your own interpretation about how many of the accounts are bots or otherwise duplicates, but it's a staggering amount either way.
https://www.theatlantic.com/technology/archive/2011/04/quote...
The only exception is people who’ve managed to sneak their way into positions where they don’t contribute anything to the company. Those people are doing society a favor by wasting Facebook’s money.
The whole Metaverse bet at Meta is also plausibly a 10s-of-$b bet that isn’t going to drive the ad flywheel, though I’m sure they hope it is. My impression is Zuck would be ok with just having a VR platform even if it’s generating revenue from not-ads.
That all being said, as humans have free will, imagining what we "could have done" if we just _forced_ everyone to do something different is flirting with fascism.
I’m oversimplifying, but that’s the root of it. If people instead of buying things from ads were looking for the best ways to purchase offsets for CO2 emissions, then the best minds would be working on that problem.
Governmental incentives and carbon taxes can make more jobs in CO2, but expecting individual citizens to solve a collective action problem is doomed.
At the same time, I think it is fair to ask people to look at what they're doing for/to the world and decide if that's something that fits their values. 300k salary is nice, but almost certainly not something you need
I’m on board with solving this at the governmental level FWIW. I support stronger anti-trust enforcement for example (and not just targeting tech).
But yea govt level would solve it much better
(* allegedly, didn't verify it myself)
Here is the thing, hypermedia is cacheable. React/Graphql not so much.
Facebook is now just an application that runs in the browser.
As a poor, small developer who doesn't want to hemorrhage money, I tend to want things to be more hypermedia and less app. It saves on complexity and bandwidth and costs.
Similarly, after getting to the bottom of their "infinite" scroll, my friend feed (which is annoyingly hidden away) gives me... 15 items. Just send me all 15. It's like 1-2 kB worth of data. If you're going to end the scroll after a dozen items, why is it using infinite scroll?
??
I would check myself, but I haven't logged into Facebook in many years. :)
That sounds totally plausible to me.
Also keep in mind they didn't say what system this is. It's often true that 1 request to a frontend system becomes 1 each to 10 different backend services owned by different teams and then 20+ total to some database/storage layer many of them depend on. The qps at the bottom of the stack is in general a lot higher than the qps at the top, though with caching and static file requests and such this isn't a universal truth.
- 10^n is more compact than million/billion/etc, more consistent, easier to translate, and doesn't suffer from regional differences (e.g. "one billion" is a different number in Britain than in the US).
I'm not saying it's clearly better than "million" in this specific case, but it's definitely not clearly worse.
Defcon: Preventing Overload with Graceful Feature Degradation - https://news.ycombinator.com/item?id=36923049 - July 2023 (1 comment)
We built a similar tool at Netflix but the degradations could be both manual and automatic.
The manual part of Defcon is more "holy crap, we lost a datacenter and the whole site is melting, turn stuff off to bring the load down ASAP"
* Perhaps "graceful feature degradation" as a choice of words is a way of noting there's immediate user impact (but less than ungracefully running out of capacity). "Load shedding" could also mean something less impactful, for example some cron job that updates some internal dashboard skipping a run.
* "feature degradation" might focus on how this works at the granularity of features, where load shedding might mean something like dropping request hedges / retries, or individual servers saying they're overloaded and the request should go elsewhere.
The situation is that A depends on B, but B is overloaded; if we allow B to do load shedding, we must also write A to gracefully degrade when B is not available.
For example, as soon as I log in I see a bell icon in the upper right with a bright red circle containing an exact positive integer number of notifications. It practically screams “click here, you have urgent business”.
I can then leave the web page sitting there for any number of minutes, and no matter how long I wait, if I click on that notification icon it will take a good 20 seconds to load the list of new notifications. (This is on gigabit fiber in a major metro area, so not a plumbing issue.)
Same thing here. Thought it was my (relatively much) slower Internet connection, or maybe that I had something "wrong" (what exactly that might have been, I don't know).
To give a super simple example, suppose you have a database that can transparently fail over to a backup, but it's so "transparent" that nobody even gets notified. Suppose the team even tests it and it proves to work well. The team will then believe that they are very well protected and tell all their customers and management all about how bulletproof their setup is, but if they don't notice that the primary database corrupted and permanently went down in month six because their systems just handle it so well, they'll actually be operating on a single database after all and just be one hiccup from failure.
One of the jobs of an ethical engineer is to make sure management doesn't just say "it's OK, the site is working, forget about it and work on something else" without some appropriate amount of pushback, which you can ground on the fact that sure, they're saying to ignore it now, but when the second DB goes down and the site goes down they sure won't be defending you with "oh, but I told the engineering team to ignore the alerts and keep delivering features so it's really my fault and not theirs the site went down".
At Facebook's scale, something will always be in a state of degradation. It's just a fact of life.
This could use some light-touch code reviewing