Hacker News Reader

Comment by helper | Hacker News Reader

As a long time Cassandra user its easy to forget that some of Cassandra's semantics will be surprising to new users. That being said, if you are considering adopting an AP database it really is important for you to know the details about how write conflicts get resolved. This is perhaps the biggest difference between Cassandra other databases like Riak and ought to be part of your decision making process instead of a surprise you run into later.

That being said, using Cassandra for distributed locks is a terrible idea. I can't think of any way in which Cassandra would be better than using {Zookeeper,etcd,consul}. Trying to force a database to do something it really isn't designed for will almost always lead to disappointment (and often resentment) of said database.

View on HN

Seconded. Don't use an AP system for distributed lock management. As with most things, use the right tool for the right job.

When I speak to developers about Riak I tell them the biggest difference between systems like Riak (including Cassandra) and traditional relational systems is not the data model, ie. relational vs non-relations (or structured vs unstructured) but rather the architecture, ie. distributed vs not distributed. As in a "C" system vs a "non-C" system, where "C" is consistency.

Disclaimer, I work for Basho, makers of Riak.

I think you can make Cassandra "C" by using quorum reads and writes. Also they support some SERIAL consistency level, which uses PAXOS underneath, and supposedly achieves consistency as well.

Quorum won't solve consistency problems like this unless you can also guarantee a serialized reader/writer for any given piece of data. The best you can do by pinning queries to primary replicas and enforcing R + W > N is RYOW consistency, and that depends on the aforementioned serialization point.

Quorum reads + writes and applying non-destructive updates only are enough to get linearizabile consistency in Cassandra, even if clocks are not perfectly synchronized.

If you're applying non-destructive updates then you don't even need quorum to achieve consistency. You're just writing immutable data.

If you're applying non-destructive updates without quorum, you would not get monotonic read consistency, because stale reads would be possible. You could write a row, immediately read it back from another replica and find out the row is not there, because that replica didn't get it yet. You'd have only eventual consistency.

The IF statement uses Paxos as underlying implementation and still it may result into an inconsistency if you are using two separate CQL statements. Cassandra does not provide begin ... end construction where everything is atomically committed or rolled back.

You can use Cassandra LWT as a building block for multi-partition transactions, in a similar way like CAS atomic operations are used to build locks, mutexes and monitors and then monitors are used to implement ACID transactions in RDBMS. However, noone says it would be easy or performant, therefore this is probably not a good idea.

Right. Cassandra crawled at 20 TPS when we sent IF cql statements. One thing to note is that Cassandra txns can never be rolled back - they can only be committed/ retrried and so on.

Minor nitpick: distributed systems can be consistent too. Your comment implies all distributed systems choose availability over consistency.

While they don't all pick one over another, there are limitations on how available and consistent you can make systems, for the same level of partition tolerance - https://en.wikipedia.org/wiki/CAP_theorem

While technically you're right, the CAP theorem uses a very strict definition of availability, which is often not needed to consider the system available in real-life. Also the consistency in CAP theorem is defined as linearizability, and this is just one of very many kinds of consistency. Often the systems are ok with weaker types of consistency.

Therefore, I'd really like that everyone stopped labeling distributed databases as AP or CP, because this is an oversimplification and most of the time, just plain wrong.

Datastax seems to think that it is a good idea: http://www.datastax.com/dev/blog/consensus-on-cassandra

That piece was written in 2014, which was a different time ;)

Plus, if we were to take the vendor words at face value, we'd all be using Docker and MongoDB in production.

I work for a large company that used docker in production. A lot of companies are. There are a lot of big names using Kubernetes, Mesos or some other container orchestration system.

Just like many companies use mongodb and nanoservices.

:) My last client (very, very big client) for the past year ran nearly all their production systems with Docker and MongoDb...

Never had any problems with Docker. MongoDB on the other hand... Suffice to say it is really fragile when spread across multiple datacentres, which is probably not a surprise to HN.

Details matter. All updates in the Datastax post are protected by LWT. That code is correct. The OP's code was wrong, because he was mixing non-LWT updates.

cassandra never claimed to be a consistent distributed database. its really quite sad that someone had to find that out the hard way.

It is optionally consistent if you know how to use it and you know its limitations. Their use of Cassandra was obviously wrong. Mixing LWT and non-LWT is like having only half of your shared data protected by mutexes - this isn't going to work correctly.

In such a situation though, shouldn't it raise an error? At least that way, the user could have a chance to recover.

Why should it make an arbitrary decision that unknowingly corrupts data for some users?

The only way for C* to know that it should raise an error is if it would implicitly protect all writes with LWT and this is not what most users want.

Following the parent's example, if you don't protect memory access with a lock, you can't know that somebody else locked it.

"Must be consistent" is a property of the data, not of a particular transaction. Data on which inconsistent transactions are ever allowed should be declared, preferably with big hazard signs.

The data are not either conistent or inconsistent. The data are either correct or incorrect, and the definition of "correct" is determined by business requirements. Most of the time weak eventual consistency model provided by Cassandra is sufficient to keep data correct if you are using it right. E.g. you don't need ACID and serializable consistency to do financial stuff correctly - banks and accountants figured out how to do this a long time before computers were invented. That's why Cassandra has serializable consistency as an opt-in only (LWTs), but this comes at a price of latency and availability. By using strong consistency all the time, you'd lose most of the benefits of Cassandra, and you could probably just replace it with a single RDBMS node (and suffer scalability and availability problems then).

Right does Java throw an exception if you forget a lock around a shared variable?

> I can't think of any way in which Cassandra would be better than using {Zookeeper,etcd,consul}

This is the correct answer. Zookeeper & Curator make it nearly foolproof to implement a distributed lock correctly, whereas with Cassandra, its the other way round.

> Zookeeper & Curator make it nearly foolproof to implement a distributed lock correctly, whereas with Cassandra, its the other way round.

Cassandra makes it nearly distributed to implement a foolproof lock correctly?

Yes, nearly! But not quite.

what about using redis for distributed locks? (assuming you are not using redis cluster)

I personally wouldn't use redis for anything besides a cache.

Why not? It has a WAL persisted to disk at a desired interval. It provides the same durability levels of most databases.

http://redis.io/topics/persistence

Let me be clear, I think redis is a well engineered piece of software and I have been constantly impressed with the way antirez runs the project.

Specifically for the question of a locking service, I need more availability than what a single node can offer. I'm not sure if you are advocating for using redis cluster as a locking service, but since it intentionally doesn't offer write safety during a partition that does not seem advisable.

https://aphyr.com/posts/283-jepsen-redis

You'll want to use a library that implements Redlock: http://redis.io/topics/distlock

You really don't want to use redlock: http://martin.kleppmann.com/2016/02/08/how-to-do-distributed...

Antirez's rebuttle didn't really rebuke martin's overall theory that redlock is a very bad locking algorithm: http://antirez.com/news/101

Did anyone else ever publish a meaningful response? I thought Antirez's response was reasonable but I also recognize the inherent bias. I'd love to see responses from other experts, ideally written such that non-experts can reasonably understand.

I recall several people on twitter (I follow a lot of distributed systems people / cs professors as it is part of my job to build these things) who agreed with Martin's analysis.

I'll have to see if I can dig some of the responses up. It bugs me that I still don't know if this algorithm is actually safe.

Honestly given Antirez's response I'm genuinely surprised no one has written a TLA+ or similar formal verification proof. Then the outcome is binary; either redlock is sound and works as expected, or it is not.

Amazon's AWS Architect, James Hamilton is a big fan of this approach, as are most of the heavyweights in distributed systems:

http://perspectives.mvdirona.com/2014/07/challenges-in-desig...

EDIT: prefaced URL with http

Thanks, I learned the fencing token lock algorithm from this.

Reply on news.ycombinator.com