Jepsen: Scylla 4.2-rc3
jepsen.io
jepsen.io
Every time I see people talking about cassandra or scylla I am reminded of the huge pains and pitfalls of what feels like hacked-together solutions that were built on a very shaky theory.
I worked only 2 years on systems that used cassandra and scylla (both on dev and ops side). About 7TB total.
I admit they are very fast, provided you do proper continuous maintenance and that your application goes out of the way to conform to the database (absolutely not the other way around)
The only safe use of these 2 software is basically: 1) add new unique rows (no updates) 2) much later (granted only after repairs, which take a real lot) you can read, but only up to a certain time in the past. 3) basically almost avoid deleting
Anything else, and you are in the realm of cases and special cases.
You can't trust errors, your operation might have been applied anyway. LWT terribly slow, limited and you should not run more than a very limited number of them.
It's full of features that are added, then the cassandra guys realize that it really does not work, in both theory and practice, and than they deprecate those a few releases later.
Compared to SQL, it feels worse than navigating blind on a minefield of special cases.
The CQL language does not help. Only use the very basics of the language, otherwise it's a constant game of: This is the rule / With This exception / Unless you also have This / Except That / but it works again with X
and on and on.
It's like they had a very basic, specific use case, and went to build a ton of features that don't stack on top of it.
Like adding full SQL on top of redis, but worse.
The right way to use NoSQL is when you know your scaling model. LWT helps you to serialize important tables.
Scylla consistently makes improvements in operations, consistency and functionality, so more to come, see what we'll announce at the summit next month
There's something to be said about CQL specifically here, because its transactional model is an odd duck, and not always for scalability reasons. Scylla and Cassandra could offer richer transactions without significant performance penalties--indeed, which would significantly speed up some types of transactional workloads, but instead there are, as the grandparent poster notes, strange edge cases.
For instance, you can't select multiple CQL rows where the select would cover a column backed by a CQL collection. That's been fixed in Cassandra, but is still present in Scylla, and limited the kinds of tests we could write for Scylla.
You also can't perform an LWT write without a guard clause: linearizable upserts in the Jepsen tests require the presence of an always-null field in every row whose only purpose is to convince Scylla that yes, we really would like to use Paxos for this write.
Likewise, there's no logical notion of a batch combining reads and writes, or, for that matter, a batch of reads. If you want those sorts of things in C* or Scylla, I imagine one winds up stringing together reads and CaS statements in retry loops, or something to that effect. We were trying to hack together a batch read via a no-op write, except that it's actually impossible to do that robustly--you can only "read" values involving guard clauses, and there's no guard which will pass on all values. It's just... awkward.
This isn't a scalability thing--all three of these cases could be executed in a single Paxos round, scoped to a single partition. It's just... weird API design.
It's not 'fair' to compare CQL to SQL, if you compare CQL to DynamoDB's http api, which scylla implements as well, you'll see that cql is better: https://www.scylladb.com/2020/05/12/comparing-cql-and-the-dy...
He didn't do that. He pointed out issues with the cql syntax and specifically scylla's implementation of it. he is also uniquely qualified to make such a statement, as his Jepsen Reports are high quality analyses looking at various databases.
But to address your other claim: they solve the same problem of providing data to your application. They should be compared if that's the decision you have to make.
1000x this
That is exactly what Cassandra was originally designed to do. Cassandra doesn't implement complex distributed mechanisms that would make it hard to operate and implement like most distributed databases or impose a cost on application developers to understand the tradeoffs they will have to grapple with and make the. Cassandra is not AP with less than a quorum read/write or CP with a quorum read/write. It only fulfills those guarantees in the way you might expect if your cluster topology never changes and your timestamps are carefully synchronized.
Absent these guarantees modifying existing data is essentially an undefined operation. This isn't a shot at cassandra developers. Cassandra isn't magic. There is no possible way it can magically use a simple last write wins strategy to guarantee that data updates won't be lost in the absence of outside guarantees.
Cassandra is essentially a prototype of a class of NoSQL databases try to be incredibly easy to manage for write heavy workloads. Look, I get it, it was a collection of appealing technologies, I found the original white paper on Cassandra interesting too. columnar read-repair eventually consistent merkle trees NoSQL column family super columns cluster gossip phi accrual failure detector
Of course, the 'columnar' attribute was originally intended so that applications designed around cassandra could reconcile multiple values associated with a single key:attribute path without having to utilize a complex api. Whether this is actually viable is another story entirely.
CQL's only advantage is that it vaguely resembles SQL. CQL is not a query language. It is a collection of Cassandra operators mapped onto a vaguely SQLish syntax that bears little relation to the underlying semantics they are derived from. Cassandra just shoehorns a bunch of its idiosyncrasies onto an SQL look alike. INSERT and UPDATE do the same thing(which is not what SQL does for either and are probably not what you expect). The entire behavior of how a 'query' is actually executed can change drastically. In order to use Cassandra correctly you have to understand how the database physically implements every operation. So the query language doesn't provide a meaningful abstraction at all.
CQL would have never made sense in the original implementation of cassandra because cassandra was never designed to do the things that people expect from a database that implements a query language.
I agree that scylla shows a lot more promise than C, but we hit (different) instabilities and gotchas on both
All has its place. I am not saying C or scylla should not be used. I was just pointing out that the basic ideas and the CQL language itself is designed to trick you in in doing a lot of things that turn out to be really wrong on the data model used by these software.
It's not an implementation thing. Should I consider only that, scylla'd be really good (provided you have the correct hardware) and much better than C*. Still, the issue (IMHO) is with the theory and the fact that the actual correct applications are much, much narrower than the initial impact lets you believe (especially due to CQL)
--edit: Kudos on scylla btw, but basically I am doubtful how much it's possible to overcome that legacy without going really incompatible even on the API
I guess I'm nostalgic for it. Your posts were during a time when I was still on a steep trajectory in my career growth and discovery. Just soaking everything up, and it was FUN. And others in my "generation" seemed to also be having FUN, and having FUN with it. Things are just a bit more bland these days :|
These range tombstones are also evil. If you keep doing it to the same partition, even at a rather low QPS, you can easily bring down some nodes by keeping their CPU at 100%. Only tested this with Cassandra though, not Scylla.
Of course distributed filesystems are even harder but we can see that even writing to a local file is surprisingly complex and in many (most?) apps probably not really guaranteeing proper data consistency.
Imho the whole Posix filesystem API is flawed to begin with and it would be great if a modern replacement would emerge.
Not as elaborate as Jepsen, but there has been some work:
- Kirk McKusick's papers and work on BSD fs log semantics
https://www.researchgate.net/scientific-contributions/Marsha...
I believe the term to look for is "soft updates."
- the BSD/NeXT file test program "fstest.c", used on local and NFS (Samba), which found many bugs in popular fs using simple operations. The ZFS team also has a version.
You can Bing versions of that by using quotes "fstest.c".
- the Luster/Gluster maintainers/consulting team used to just untar emacs on their distributed fs buildouts and see how many nodes left the cluster. (They lived off DARPA funding basically, and were paid to configure and install distributed fs for US govt/military supercomputer installations, and fix the underlying bugs as found.)
- Ironically, the Ceph team did not own any commercial storage devices, so just tested on regular linux machines.
- Reiserfs 3 was the first GA log fs on linux (default on SUSE), so I was one of the earliest US users in production.
SUSE's rep called me a liar at trade show, saying "nobody uses our distro in the US. :) It worked well on email server loads, and could delete 1 million files in a directory in under 1 second. I followed the development of v4, but the "wandering logs" and "dancing trees", etc. kind of wigged me out.
https://en.wikipedia.org/wiki/Dancing_tree
Source: DBA and storage engineer.
EIO: Error Handling is Occasionally Correct https://www.usenix.org/legacy/events/fast08/tech/gunawi.html
Evaluating File System Reliability on Solid State Drives https://www.usenix.org/conference/atc19/presentation/jaffer
> I/O errors are simulated in both the TCL and TH3 test harnesses by inserting a new Virtual File System object that is specially rigged to simulate an I/O error after a set number of I/O operations. As with OOM error testing, the I/O error simulators can be set to fail just once, or to fail continuously after the first failure. Tests are run in a loop, slowly increasing the point of failure until the test case runs to completion without error. The loop is run twice, once with the I/O error simulator set to simulate only a single failure and a second time with it set to fail all I/O operations after the first failure.
Tangentially, I would love to have a FUSE filesystem with a.) minimal build dependencies, b.) some sort of CLI interface, and c.) the ability to, say, forget to flush un-fsynced data to "disk", allowing us to simulate a power failure. There have been a bunch of research projects on this front, and they find bugs spectacularly. I bet this approach would also find errors in distributed systems, but I've yet to find one that really has the right shape for use with Jepsen.
All popular databases shall routinely undergo Jepsen reviews; And none shall prohibit publication of test or benchmark results.
Isaiah 11:6-7 (New Database Translation)
> Just a few examples should be enough to show you the problem. In the New International Version (NIV) it says in 2 Samuel 21:19 that someone other than David killed the giant Goliath. Anyone knows this is not true, except the 'scholars' who translated it. According to Hebrews 3:16 in the NIV all the children of Israel rebelled in the wilderness and none made it into the promised land; however, we know that Joshua and Caleb did not rebel, and God promised them each a place in the promised land (just read Numbers 13 & 14). Another mistake is found in Mark 1:2, 3 where according to the NIV the writer of Malachi was Isaiah! Just check out Isaiah 40:3 and Malachi 3:1 - the KJV has it right and the NIV has it wrong.
For comparison of the verse this comment parodies, https://biblehub.com/isaiah/11-6.htm