How Distributed Systems Avoid Race Conditions Using Pessimistic Locking?
newsletter.scalablethread.com
newsletter.scalablethread.com
Distributed locking, like all pessimistic concurrency, has a place but it comes with serious scaling concerns depending on how granular the scope of the locks are. Lock lived locks suffer from blocking other writers from large chunks of time, destroying latency. Very fine-grained locks eventually become dominated by networking overhead since the lock requires two to three trips to the database for each logical write/update.
Conversely, optimistic concurrency under high contention can lead to excessive waste from clients needing to retry operations because of conflicts.
The only way to win is reduce the scope of your writes/updates to avoid the contention as much as possible.
The former is much less likely to break (less maintenance overhead) but harder to set up and cash in on, because essentially you need to move all of the deterministic work to the end of the request, do all of the nondeterministic parts first, calculate the eTag from that data, and then if it matches don't do the rest.
And the problem is that pure functional part at the end often just isn't that expensive compared to all the fanout that comes before. It's data transforms and the cost of writing to cache. So you have to do that pattern in your services too in order to get some oomph out of this solution, except perhaps in some narrow problem domains.