Hacker News Reader

Comment by jbellis | Hacker News Reader

Cassandra developer here.

Lots of comments here about how Cassandra is AP so of course you get inconsistent (non-serializable) results.

This is true, to a point. I'm firmly convinced that AP is a better way to build distributed systems for fault tolerance, performance, and simplicity. But it's incredibly useful to be able to "opt in" to CP for pieces of the application as needed. That's what Cassandra's lightweight transactions (LWT) are for, and that's what the authors of this piece used.

However! Fundamentally, mixing serializable (LWT) and non-serializable (plain UPDATE) ops will produce unpredictable results and that's what bit them here.

Basically the same as if you marked half the accesses to a concurrently-updated Java variable with "synchronized" and left it off of the other half as an "optimization."

Don't take shortcuts and you won't get burned.

View on HN

I agree, Cassandra is doing here exactly what I (as a user) would expect.

When using a DB like C*, you always have to ask yourself, "does this update happen before or after this one - have I done anything to ensure that's the case?"

In this example, the second query (the UPDATE) is being partially applied before the first query (the INSERT) - and that's OK, because there's no ordering or dependency in the second query that forces it to run second. So the reordering of the queries that he's observing is legal, and can be simply avoided with "UPDATE ... IF revision = :last-rev".

> the second query (the UPDATE) is being partially applied before the first query (the INSERT) - and that's OK

"Partially applied" is ok with a database?

The description of Cassandra on it's site is "Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data."

If it's mission-critical data, I wouldn't do arbitrary things with it for conflict resolution that can corrupt data.

This is not arbitrary. It may be not intuitive coming from a rdbms background, but it isn't arbitrary.

As Johnathon pointed out, it's like properly using synchronized or volatile half the time. I don't call it sometimes not working as arbitrary. I call it expected for not following the rules of the system.

Let's pretend I'm leading a blind child by telling them which direction they should go, and if they don't step carefully, they could be hurt.

If I can't see the child, should I continue to give them direction, or tell them to stop?

In this use case, the database makes changes to data without knowing what is correct and what is harmful. That is not the user's fault. It's a code choice.

No, it's a users choice to use a DB that has this tradeoff.

Like it was said a couple of time already, all of your complaints about C* would work just as well for locking mechanisms in most popular languages.

If I follow your argument correctly, it is basically the same argument as "your C compiler is correct, what you've written is invalid and the standard allows undefined behaviour here".

Which may be a technically valid argument against the compiler/database system, but it's not a valid argument for defending the system as a whole: if a standard allows arbitrary execution instead of bailing out on non-standard (ambiguous) input, it is unreliable.

Is variable assignment in c/java/... unreliable? It behaves very similar to what C* does. Concurrent access and modification will produce undefined behaviour if you don't explicitly protect it.

Exactly.

Getting access to things like concurrent locks is HARD to get right. That is why there are so many simple languages that don't let you touch concurrency.

Doesn't mean there is no need for it in the world, and no one should be able to use it.

RDBMSes can cause similar inconsistencies if you don't know what you're doing. It is like setting read uncommitted and then complaining about dirty reads.

Cassandra is perfectly fine. If you want your writes to be consistent, learn to use LWTs properly. The same way, if you want your data to be fully consistent in RDBMS, learn to use SERIALIZABLE isolation level (which is not default in most RDBMSes for performance reasons). If, in an RDBMS, you use SERIALIZABLE for half of the updates and READ UNCOMMITTED for another half, guess what consistency guarantees do you really get?

My biggest complaint about C: SQL-like interface. These seems like a terrible decision to me.
If you are used to relational databases and SQL, it's a real struggle to get your head in the right place. Some of the gotchas:

Cell level vs row level consistency/locking (as stated in this article) * Grouping / counting isn't really a thing. * WHERE statements are actually used for identifying the key/value pair you want and not for filtering the rows. * Indexes aren't really indexes. They are definitions of partitions and sort order.

That's actually my favourite part of C*. I think the CQL interface makes programming against it extremely easy, and I dare might say enjoyable.

It makes getting standard, and understanding what's going on a breeze.

This would be a more sympathetic response if Cassandra did less to tempt devs not intimately familiar with it into doing things which will hose them later on. I've worked with it very successfully in the past and contributed to one of its client libraries, but I hesitate to recommend it because that trait makes it unnecessarily dangerous to use.

So, since their use of UPDATE is problematic, what is the correct way to release the locks?

DELETE ... IF EXISTS

DELETE ... IF <condition>

UPDATE ... IF <condition>

Disclaimer: I am working with the author

The problem is even with using LWTs for both updates you are running into the same problem, the only thing you gain is that one of the statements (either lock or release) wins and in this case it then only works because the lock uses a TTL and is removed after some time.

Also, two conflicting statements from the same thread going to the same node should be easily serializable for Cassandra - or at least be logged.

> Don't take shortcuts and you won't get burned

Since you are the Datastax CTO, maybe some alignment with the marketing team on how features are communicated to users might help users not getting burned? :D

This is too subtle. Incompatible operations should be rejected. Reliance on the programmer to correctly use systems that don't enforce consistency to write consistent transactions is a bad strategy.

This kind of nannying can be really expensive to do correctly when trying to build high performance systems -- to the point where it doesn't make sense to punish everyone just because someone who didn't read the manual MIGHT misuse the product. It's not at all unreasonable to mix transactions with different guarantees if they never touch the same data, and tracking that accurately enough without pissing off your customers with performance needs seems like a fool's errand.

Read the CAP theorem, or stick to your single node.

The situation could be considered similar to .NET adding the IllegalCrossThreadException exception to Windows Forms in v2¹, defaulting to detecting invalid behavior with the option to disable checking (Control.CheckForIllegalCrossThreadCalls).

.NET v4.5 introduced a new approach (Task-based Asynchronous Pattern)² with changes to framework and language (C#5) to "simplify" implementing things correctly.

.NET took the developer-friendly path; Cassandra, not yet.

¹ https://web.archive.org/web/20060414185346/http://msdn.micro...

² http://stackoverflow.com/a/18033198

haha, I was just going to ask you what you thought of this, and I'm glad to see you responded already! Yes! :)

> Don't take shortcuts and you won't get burned.

I'm sorry that the user did something unexpected and then posted about it in a way that made your application look bad. I know that must be frustrating. However...

Calling the user out as doing something wrong when your application is failing because of a use case you can't handle properly just looks bad. You serve your users, not the other way around. Don't forget that.

If it were me and there were a case that my application couldn't handle properly, if I couldn't fix it, I'd raise an error, and then document clearly that they should not do this, such that when they search for that error, they'd find the answer. Then, I'd work to see if there were a way I could avoid the error altogether by not allowing that use case.

There is no "one size" fits all database or distributed system in existence. To say that there is is to say that MySQL is equally as well suited to all use cases as Cassandra or Redis. They all store and serve data, right?

Cassandra makes no claims to be such a holy grail. Read their documentation, and you can see the use cases it is good for and those it is not.

The author of this blog post chose one it is not good for.

Put another way, "I'm sorry that the Lamborghini you bought broke when you attempted to go off-roading with it. Perhaps you should have bought a Jeep instead?"

The high-handedness doesn't improve your argument. Consider forgoing it next time.

It's not possible to detect whether the user wants CP. You can assume they want CP, but the entire point of Cassandra is that it doesn't make that assumption.

Cassandra is AP with opt-in CP. This is an explicit tradeoff. You're giving up the assumption (which enables error checking) that everything is CP in order to get AP performance. This tradeoff is one of the main use cases for which Cassandra exists.

The vast majority of the time, error checking is way more valuable than AP performance, so your approach to handling the error makes sense, but if that's your situation you shouldn't be using Cassandra. There are a wide variety of ACID-compliant relational databases that do what you want.

TL;DR: Using Cassandra and expecting CP error checking is like using a hammer and expecting screwdriver behavior.

Reply on news.ycombinator.com