Using load shedding to avoid overload
aws.amazon.com
aws.amazon.com
Like, for an ecommerce site, being able to prioritize users in the checkout path first, not-in-checkout but non-empty cart second, etc.
Or prioritizing features in a similar way. Turning off, for example, "people that bought this, bought that", under load.
The (fantastic) article suggests precisely this:
> let's say a service has two APIs: start() and end(). In order to finish their work, clients need to be able to call both APIs. In this case, the service should prioritize end() requests over start() requests
There’s also someone I was talking to after writing the article who said they can fall back to statically rendered versions of certain pages on Amazon.com during overload. The trick is to have a page that is still useful!
And for the “turning off features” idea - this happens today on Amazon.com. If a feature on the site fails to render successfully or on time, it’s left off of the page. Critical functionality can be left off, so it’s a judgement call on what’s allowed to fail the page render.
"If a feature on the site fails to render successfully or on time, it’s left off of the page. Critical functionality can be left off, so it’s a judgement call on what’s allowed to fail the page render."
Oh, that's useful also, but I meant a step farther where the page doesn't ask for those widgets if (load > X). Which avoids calling it at all.
For example, say a service is backed by a partitioned cache cluster, where the data is hashed to a particular cache node. Now let’s say one node has a problem, causing requests to data that lives on that node to fail, but others to succeed. If a client is making requests for data that happens to live on all nodes (the client doesn’t know about these nodes by the way, it’s just an implementation detail of the service) and sees an increased error rate, should it start failing some requests? It could take a single partition outage and increase the scope of impact into a full outage.
Anyway I’ve been meaning to write an Amazon Builders’ Library article on this topic, or to convince someone else to do it (looking at you, Marc Brooker!)
p.s. - btw, no cloud has that much elastic capacity.
What is reco?
Another pattern from the networking world, one which Facebook wrote about, is CoDel aka Controlled Delay (switches tasks to lower timeouts when the work-queue begins to build-up) combined with adaptive LIFO (processes last-in tasks first when under load, but FIFO otherwise): https://queue.acm.org/detail.cfm?id=2839461
Same with deprioritizing queries that are older in the queue. You’d end up deprioritizing people who’ve already been waiting, so they’d likely try again instead of waiting.
Thus you end up with the feedback loop this article talks about, where you’re amplifying the problem instead of addressing it.
Another simple approach is to serve queues (of like requests) in LIFO order.
This means that if you catch up to the incoming new requests, a queue keeps every request running slower due to the time spent in the queue.
The steady state stack on the other hand, gives the same couple items worse and worse latency, while all the new items go back to normal.
Chances are the long latency requests will be retried, and there's a roughly fixed number of items to be retried, so you don't have to worry too much about the ones stuck in the stack. For a queue, the retries lengthen the queue, and that waiting time is added to every item, making them more likely to retry too, lengthening the queue further etc
Having the stack leak items at the bottom gets the same benefit - you don't have to fulfil them, but if you can get the items back out of the stack quickly, it's still worth working on and completing them before the client needs to send a retry. The more of them you get in, the more 9s you get to look more like your p50 than your p100
You are keeping up and the queue is mostly empty, the order does not matter
You are not keeping up, the queue is growing. If nothing changes, the age of items removed from the front will grow and grow, and eventually all be timeouts or abandoned.
"Back in my day..." we used HPS (hits per second) and CPU load as the "goodput". A 'hit' was only recorded via an access log at the end of a successful request<->response. Of course this doesn't measure whether the latency of the responses is unacceptable, just that there was a completed response... so that's where CPU Load came in. If you're doing a lot of HPS and load gets too high, you know latency is just going to get worse to the point of unavailability, so you start load shedding.
In our less-advanced old-school practice, load shedding was merely lowering the maximum requests per second option of the HTTP servers, waiting, and lowering again, until the load recovered. Our tools could reconfigure the server's settings without restarting, but required issuing an admin command to the server and "waiting in line" while the server processed all the other CPU requests under load, which meant the problem might continue for longer than we'd like.
To get around "waiting in line", you would use a transparent network filter (either a proxy, or IPTables) to load-shed. This can be called a "middlebox" solution. But sometimes termination of connections isn't feasible without "dirty" terminations that might cause bugs in clients or servers... and that's [one of the reasons] why people hate middleboxes. But they're great when they work!
You would also probably use Apdex today instead of CPU Load. If your API has contractual minimum latency guarantees, you'd skip Apdex and just use latency itself.
And good callout on middle boxes. Even high level abstraction ones like Amazon API Gateway. In fact this is my favorite feature of it. API Gateway can reject a very high rate of excess traffic for a small overloaded service behind it.
"Keeping Netflix Reliable Using Prioritized Load Shedding" https://netflixtechblog.com/keeping-netflix-reliable-using-p...
At (work) our services achieve high availability (99.99%) simply by being composed of reliable parts. Most effort is focused here and it seems to have a good payoff ratio.
When we do try to add anything like graceful degradation it seems to strongly hurt code readability/ease of understanding because it requires additional code branches and either a fair bit of boilerplate or extra abstractions.
This could be a result of the scale we are at/the maturity of our tools but I'm interested in if anyone else has some hands on experience.
That pretty much means you can't have hard coded '100 requests per second per instance'.
Instead, I suspect the future of load shedding is automatic maximum "goodput" tracking. For example, the load balancer can alternate between 90 and 100 parallel requests, and if more requests are completed with fewer parallel requests then start alternating between 80 and 90 to figure which of those is better...
A key sort of “continuation” to this article is the one on fairness: https://aws.amazon.com/builders-library/fairness-in-multi-te... . This gets into the topic of utilization a bit more.
But you’re right - good load shedding gives a business a tool to make an easier trade off when it comes to capacity management. A slight error rate until autoscaling kicks in is an easier pill to swallow than a worse outage.
Not really. My service may depend on other services that I have no control over. Perhaps I have extra money to scale up my own service, but those other services may be owned by different teams or organizations entirely.
That could at least postpone the load shedding, or handle localised surges in service demand due to user behaviour more effectively.
The request dropping (less jargon-y and more descriptive than "load shedding", which refers to consuming more rather than less) only kicks in when that fails or isn't fast enough.