Cassandra is not row level consistent
datanerds.io
datanerds.io
Lots of comments here about how Cassandra is AP so of course you get inconsistent (non-serializable) results.
This is true, to a point. I'm firmly convinced that AP is a better way to build distributed systems for fault tolerance, performance, and simplicity. But it's incredibly useful to be able to "opt in" to CP for pieces of the application as needed. That's what Cassandra's lightweight transactions (LWT) are for, and that's what the authors of this piece used.
However! Fundamentally, mixing serializable (LWT) and non-serializable (plain UPDATE) ops will produce unpredictable results and that's what bit them here.
Basically the same as if you marked half the accesses to a concurrently-updated Java variable with "synchronized" and left it off of the other half as an "optimization."
Don't take shortcuts and you won't get burned.
DELETE ... IF <condition>
UPDATE ... IF <condition>
I'm sorry that the user did something unexpected and then posted about it in a way that made your application look bad. I know that must be frustrating. However...
Calling the user out as doing something wrong when your application is failing because of a use case you can't handle properly just looks bad. You serve your users, not the other way around. Don't forget that.
If it were me and there were a case that my application couldn't handle properly, if I couldn't fix it, I'd raise an error, and then document clearly that they should not do this, such that when they search for that error, they'd find the answer. Then, I'd work to see if there were a way I could avoid the error altogether by not allowing that use case.
Cassandra makes no claims to be such a holy grail. Read their documentation, and you can see the use cases it is good for and those it is not.
The author of this blog post chose one it is not good for.
Put another way, "I'm sorry that the Lamborghini you bought broke when you attempted to go off-roading with it. Perhaps you should have bought a Jeep instead?"
Cassandra is AP with opt-in CP. This is an explicit tradeoff. You're giving up the assumption (which enables error checking) that everything is CP in order to get AP performance. This tradeoff is one of the main use cases for which Cassandra exists.
The vast majority of the time, error checking is way more valuable than AP performance, so your approach to handling the error makes sense, but if that's your situation you shouldn't be using Cassandra. There are a wide variety of ACID-compliant relational databases that do what you want.
TL;DR: Using Cassandra and expecting CP error checking is like using a hammer and expecting screwdriver behavior.
When using a DB like C*, you always have to ask yourself, "does this update happen before or after this one - have I done anything to ensure that's the case?"
In this example, the second query (the UPDATE) is being partially applied before the first query (the INSERT) - and that's OK, because there's no ordering or dependency in the second query that forces it to run second. So the reordering of the queries that he's observing is legal, and can be simply avoided with "UPDATE ... IF revision = :last-rev".
"Partially applied" is ok with a database?
The description of Cassandra on it's site is "Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data."
If it's mission-critical data, I wouldn't do arbitrary things with it for conflict resolution that can corrupt data.
As Johnathon pointed out, it's like properly using synchronized or volatile half the time. I don't call it sometimes not working as arbitrary. I call it expected for not following the rules of the system.
If I can't see the child, should I continue to give them direction, or tell them to stop?
In this use case, the database makes changes to data without knowing what is correct and what is harmful. That is not the user's fault. It's a code choice.
Like it was said a couple of time already, all of your complaints about C* would work just as well for locking mechanisms in most popular languages.
Which may be a technically valid argument against the compiler/database system, but it's not a valid argument for defending the system as a whole: if a standard allows arbitrary execution instead of bailing out on non-standard (ambiguous) input, it is unreliable.
Getting access to things like concurrent locks is HARD to get right. That is why there are so many simple languages that don't let you touch concurrency.
Doesn't mean there is no need for it in the world, and no one should be able to use it.
If you are used to relational databases and SQL, it's a real struggle to get your head in the right place. Some of the gotchas:
Cell level vs row level consistency/locking (as stated in this article) * Grouping / counting isn't really a thing. * WHERE statements are actually used for identifying the key/value pair you want and not for filtering the rows. * Indexes aren't really indexes. They are definitions of partitions and sort order.
It makes getting standard, and understanding what's going on a breeze.
The problem is even with using LWTs for both updates you are running into the same problem, the only thing you gain is that one of the statements (either lock or release) wins and in this case it then only works because the lock uses a TTL and is removed after some time.
Also, two conflicting statements from the same thread going to the same node should be easily serializable for Cassandra - or at least be logged.
> Don't take shortcuts and you won't get burned
Since you are the Datastax CTO, maybe some alignment with the marketing team on how features are communicated to users might help users not getting burned? :D
.NET v4.5 introduced a new approach (Task-based Asynchronous Pattern)² with changes to framework and language (C#5) to "simplify" implementing things correctly.
.NET took the developer-friendly path; Cassandra, not yet.
¹ https://web.archive.org/web/20060414185346/http://msdn.micro...
That being said, using Cassandra for distributed locks is a terrible idea. I can't think of any way in which Cassandra would be better than using {Zookeeper,etcd,consul}. Trying to force a database to do something it really isn't designed for will almost always lead to disappointment (and often resentment) of said database.
Plus, if we were to take the vendor words at face value, we'd all be using Docker and MongoDB in production.
Never had any problems with Docker. MongoDB on the other hand... Suffice to say it is really fragile when spread across multiple datacentres, which is probably not a surprise to HN.
Why should it make an arbitrary decision that unknowingly corrupts data for some users?
Following the parent's example, if you don't protect memory access with a lock, you can't know that somebody else locked it.
Specifically for the question of a locking service, I need more availability than what a single node can offer. I'm not sure if you are advocating for using redis cluster as a locking service, but since it intentionally doesn't offer write safety during a partition that does not seem advisable.
Antirez's rebuttle didn't really rebuke martin's overall theory that redlock is a very bad locking algorithm: http://antirez.com/news/101
Amazon's AWS Architect, James Hamilton is a big fan of this approach, as are most of the heavyweights in distributed systems:
http://perspectives.mvdirona.com/2014/07/challenges-in-desig...
EDIT: prefaced URL with http
When I speak to developers about Riak I tell them the biggest difference between systems like Riak (including Cassandra) and traditional relational systems is not the data model, ie. relational vs non-relations (or structured vs unstructured) but rather the architecture, ie. distributed vs not distributed. As in a "C" system vs a "non-C" system, where "C" is consistency.
Disclaimer, I work for Basho, makers of Riak.
Therefore, I'd really like that everyone stopped labeling distributed databases as AP or CP, because this is an oversimplification and most of the time, just plain wrong.
This is the correct answer. Zookeeper & Curator make it nearly foolproof to implement a distributed lock correctly, whereas with Cassandra, its the other way round.
Cassandra makes it nearly distributed to implement a foolproof lock correctly?
INSERT INTO locks (id, lock, revision)
VALUES ('Tom', true, 1)
IF NOT EXISTS USING TTL 20;
BEGIN TRANSACTION
SELECT FROM locks WHERE id = "Tom" AND lock = true AND revision = 1;
-- if no records returned
INSERT INTO LOCKS locks (id, lock, revision) VALUES ('Tom', true, 1)
COMMIT
The usual solution in SQL is to use UNIQUE indices which will cause an INSERT to fail if the record about to be inserted already exists.
I ran into this reassembling SMS message fragments, where I wanted to detect that all the parts had come in. The right answer was to do an INSERT for each new fragment, then COMMIT, then do a SELECT to see if all the fragments of a message were in. Doing the SELECT first produced a race condition.
IF NOT EXISTS causes replicas to agree on the result using PAXOS, and only responds with success if a certain number of replicas concur + write the transaction to disk.
i believe it is a quorum for SERIAL consistency level, and local quorum (quorum of replicas in the local DC) for LOCAL_SERIAL.
They are wrong though. As HN submission illustrates, people want some order and eventual consistency, not a rule to select a single field during concurrent changes. And this is where a Lamport timestamp could help.
Vector clocks are still out, though.
Edit: Actually, you can't do Lamport timestamps because you can't query the current value of a timestamp.
Re-edit: That's wrong. I shouldn't believe any old blog I find. (I'm leaving it in the comment because it was quoted in a reply.)
you can get the timestamp of each column back as part of a SELECT.
is that not enough?
Great discussion around the CAP theorem but it misses the point. AP vs CP / Cassandra being AP is not relevant to this particular problem:
1) This is not a distributed systems corner case. You will run into this if you are running Cassandra on a single node. A node should be able to guarantee consistency internally during normal operation. If it is not able to do that, there is something wrong with the system.
2) This is a case where queries are being send from the same process/thread and go to exactly the same nodes. Attach a simple, monotonically increasing query counter to each call and you can easily serialize it on the other side.
Even better, stop trying to emulate the old and tired distributed lock methods that have been proven over and over again to be insufficient.
Even now I'm still not sure but I presume WAT = what!
HBase is strictly CP (except for its geo-replication, and optional timeline-consistent region replicas).
It uses MVCC for row "transactions" to always keep rows consistent. HBase also has checkAndPut and checkAndDelete primitives, which are atomic, as well as Increment and Append, which are atomic and serializable.
http://hadoop-hbase.blogspot.com/2012/03/acid-in-hbase.html explains it fairly well.
Together with Apache Phoenix you have full multi-row transactions, but they come with a price obviously.