There is no point in having your service try to speculate what happened when that isn't its job.
As long as your service doesn't need to ensure things happen only once or you build in that recovery mechanism (identifying unique events and throwing them out if your system has already seen them), being able to toss work to another instance is great in my mind.
Like philsnow says in a different comment (https://news.ycombinator.com/item?id=16832566)
> If your backend is Redis, how is starting more speculative hits to Redis going to help, since it's single threaded?
I agree; really understanding this problem is a lot better than just blindly retrying because it seems to work. Is there something wrong with the network that'll affect every service?
Retrying could be the solution, but it should be so with the acknowledgement that it's incurring technical debt.
Maybe. But again, this condition can occur with no flaws. It may just be you're on the edge of an autoscale event or a box with a bad xen peer. There may not a be a problem. It doesn't matter.
So sure, folks should keep to their SLA, but it doesn't have a lot of bearing on this. It's just best practice.
You can do none, one of them, or both. You may have team skills to tackle none, one, or both. You may or may not control / have access to one of the sides. You may have a timeline for solving this problem where you decide which task is faster to finish. Finally, you may have a recurring problem which makes you lose a specific amount of money every time and a speculative request stops that now, while putting people on a performance debugging mission may only potentially have a positive return in the future.
Sure, it's a technical debt. It's not illegal - you just need a really good reason to commit to it.
Oh also, because so many people use languages where even a basic production-quality binary tree is not easy to write. in.