Exponential backoff is the wrong answer in a highly available system in the typical case where (a) failure is expected and (b) you have nodes you are supposed to fail over to.
While your premise holds for happy cases, when you do have a cascading series of outages, not using exponential backoff is just adding a self-inflicted DoS to when you do go down.
I don’t really follow your premise and can’t really articulate many cases for when you shouldn’t use exponential backoff. Maybe if you’re working at Jane St or something; or other circumstances where you can deploy immediate changes to the client; and you’re willing to trade ‘better p50 for worse outages’.
But in the case of shipped code that’s run on clients, I’ll continue exponentially backing off all the way, all the time, for everything.
To handle this correctly you need your RPC framework to accurately communicate retryable vs non-retryable failures to clients. Then service A knows service B is dead, does not retry, and proapgates the failure to clients. This is hard to do perfectly, but there's no alternative that works.
> To handle this correctly you need your RPC framework to accurately communicate retryable vs non-retryable failures to clients.
Even this is not enough, since you cannot always reliably know whether service B is dead or suffers an intermittent issue that can be safely retried just from looking at a single failure.
The classic solution, in the monolith/few-services world would be a circuit breaker. High failure rates on any service trigger a circuit breaker in the services calling it, and they'll wait for a cooldown period before trying again.
When you move to a massive microservice architecture with hundreds or thousands of microservices, setting up circuit breakers manually becomes very hard to track and do reliably. Service meshes like Istio make this slightly easier, but they still don't let you verify that all possible paths have circuit breakers and that retries are not excessive etc.
When one end user request fans out to hundreds of backend requests (typical for microservices), you can't have each of those backend requests do its own exponential backoff. If they do it in parallel, they're a thundering herd, and if they do it in serial, the end user request will time out before you finish all the work, at which point you're doing a bunch of slow expensive work for no gain (and the enqueued slow expensive work will make your outage worse).
There are client-side retries (which have broken GitHub in this case) and server-side initiated API calls between microservices that don't pass through any of your ingresses (e.g. triggered by an ETL pipeline, or a scheduled job).
With a complex architecture you can't just slap a circuit breaker on a couple of ingresses and call it a day. Don't get me wrong, putting them there does go a long way, but you won't be covering all your bases.
At least it should be required for critical level services in production.
“Circuit breakers, where calls to a downstream service are stopped entirely when an error threshold is exceeded, are widely promoted to solve this problem. Unfortunately, circuit breakers introduce modal behavior into systems that can be difficult to test, and can introduce significant additional time to recovery. We have found that we can mitigate this risk by limiting retries locally using a token bucket. This allows all calls to retry as long as there are tokens, and then retry at a fixed rate when the tokens are exhausted.”
I’m particularly fond of the token-bucket mechanism for pacing recovery.
Yes, exponential backoffs alone are insufficient. Yes, adding jitter helps randomize the calls across a fleet and should be the default with exponential backoffs. Yes, both of these may be sufficient for most systems. Yes, you can dive more into circuit breakers and adaptive retries to limit thundering herd.
https://aws.amazon.com/blogs/architecture/exponential-backof...
I've seen this happen multiple times. Someone designs some clever backoff strategy without considering how it fits in the context of the rest of the system. Hilarity ensues.
If you have something taking an entire minute on a computer, please ensure you implement it in such a way that no connections are actually held open for that entire minute.
I'm not proposing doing any exponential backoff retries, or even retries at all for internal services.
In my mind, the retries with exponential backoff and jitter belong only on the end-client(VSCode in this case). Everything else -> fail fast.
On that note, one of the more memorable incidents of my career was when a 10M+ node client decided to retry as hard as possible on 4xx. That was fun x_x.
[edit] that is to say, for this mechanism to be robust your retry-after enforcement mechanism needs to be capable of withstanding almost every single one of your users attempting to illegally retry as fast as they physically can without negatively impacting that one user requesting legitimate traffic. https://media.tenor.com/p3mss3YI6TcAAAAM/wat.gif