Comment by jedberg | Hacker News Reader

Comment by jedberg | Hacker News Reader

If you need a global lock in your distributed system, then you're already in trouble. In most cases, the best solution is to rewrite your software so that you don't need the lock. I don't want to say all, because I'm sure there are use cases where it would be impossible to do so. However, for my own research, I'm having trouble finding examples where a global lock is absolutely necessary. If anyone has any examples, please send them to me.

For example, a lot of people say, "updating a bank balance". But even that can easily be eventually consistent. You send increments and decrements and settle up at the end. Yes, your balance might go negative, but that's what ATM limits are for -- the most you could go negative is that limit. Most other systems can be written the same way. And for non-numeric values, there are always vector clocks or timestamps or quorum to determine which update came next, or you could send a client a list of all recent updates and let the client "do the right thing".

While I agree with the general sentiment, the matter is, there were a ton of people using a single Redis instance for distributed locks, since people find this model useful sometimes, even when best alternatives exist, or maybe because of limitations of systems they interact with. So I tried to provide something better compared to what people were using, but comparatively easy.

I think what you have provided is a far better solution than how most people were using redis. :) I was just hoping to convince people that they should think a little deeper before going straight for the global lock.

Hope you're doing well BTW!

Have definitely thought through this process and wound up using Redlock for the time being.

I didn't expect it to be a perfect system but I don't want to deal with setting up a queuing system right now when there simply isn't enough activity taking place.

Will it fail at some point? Possibly, but it's a great intermediary solution as you state. I for one would be curious to see other solutions to this problem. Are there any resources you'd recommend?

That may be the case, but all other protocols that guarantee consistency suffer from similar subtle issues (which can be resolved through careful algorithm design). Eventual consistency, however, is never the answer, unless your question is "how do I get eventual consistency?" The consistency required by an application is entirely imposed by the domain, and it is that requirement that causes algorithmic difficulties. Using less-consistent protocols to achieve a consistent view simply pushes the problem to a different layer.

Also, eventual consistency is a weak requirement that, when sufficient, enables a certain level of fault tolerance and low latency not achievable with other consistency levels (see [1]), although other consistency levels may offer good-enough fault-tolerance and latency (rivaling EC in many cases). EC is not in general the solution to the problem of "how do I best distribute my data?"

[1]: http://www.bailis.org/blog/when-does-consistency-require-coo...

> Using less-consistent protocols to achieve a consistent view simply pushes the problem to a different layer.

Absolutely, but if you can avoid using global locks, which are error prone and reduce reliability, isn't that a good thing?

It's not a question of good or bad. Providing a higher level of consistency than what your application needs will only hurt you, but many applications just cannot do with weak consistency.

I think the point being made here is that "what your application needs" is often something you chose your way into, and can choose your way back out of by backing out some other constraints—and that it's often more sensible to back out these constraints than to dive head-first into a problem domain that requires distributed locking.

In other words: if you set out to create Bitcoin in 1998, you might end up running into distributed consensus problems that are difficult enough that you decide to "solve an easier problem"—such that you end up creating Paypal instead.

> I think the point being made here is that "what your application needs" is often something you chose your way into, and can choose your way back out of

I think that while this may be true for many requirements, consistency is among the least flexible ones. Besides, losing consistency usually makes your system more complicated, not less. Weak consistency has some real benefits (latency in some cases, tolerance for more failure modes), but simplicity is not usually one of them.

> In other words: if you set out to create Bitcoin in 1998, you might end up running into distributed consensus problems that are difficult enough that you decide to "solve an easier problem"—such that you end up creating Paypal instead.

I don't think that the design problems behind Bitcoin are any easier than distributed consensus, which, BTW, is not necessary for quite a few consistency levels.

What happens when you sent the increment and then failed to send the decrement? The problem isn't typically the eventual consistency aspect, though that can also be a problem since you want the balance to be accurate, but rather the transactional nature.

How would a global lock fix this? Presumably you would have a process that is trying to decrement get a global lock and then crash, and eventually release the lock anyway, right? So you get the same outcome.

You can build things like 2 phase locking on top of that. You're right though that simply taking a lock before updating two values doesn't solve the atomicity of that update problem. However if you're updating a single value then taking a lock can make your life easier vs. systems with no locks and eventual consistency.

You can make your life as a programmer easier in exchange for making your life as an operator harder. :) Locks are brittle and hurt reliability.

I'd imagine it being pretty unsettling to watch a bank balance fluctuate. Also, what if there are downstream operations that depend on the value being accurate I.e. Trigger overdraft condition?

> I'd imagine it being pretty unsettling to watch a bank balance fluctuate.

Banks have generally solved this by batching updates, but yes, it could be unsettling for the customer in a similar situation. There are ways around that though, for example you update a cache that stores the data instantly and then let the eventually consistent source of truth update the cache again if it is not in sync, or perhaps have a short timeout on the cache, so the customer gets a consistent and smooth experience but the source of truth is still accurate, eventually.

Is leader election somehow a usage of a global lock? I do agree it should be avoided actively though.

I don't think so, there are distributed leader election schemes that don't require a lock. The simplest is using HRW hashing[0] so that each node independently determines the same leader.

[0] https://en.wikipedia.org/wiki/Rendezvous_hashing

It's related.

Leader election generally[1] requires consensus among distributed processes, and global locks generally[1] require consensus as well. The benefit of this is that both problems can be solved on top of a common consensus implementation which is what http://atomix.io does.

1: I say generally because you can do fancy things with fancy clocks to avoid running operations through a quorum under certain circumstances, but these carry caveats that preclude them from being reliable enough to use in many use cases.

Reply on news.ycombinator.com