Jepsen: Crate 0.54.9 version divergence | Hacker News Reader

Jepsen: Crate 0.54.9 version divergence

aphyr.com

127 pointsgmcabrita27 comments

I have a general question about database reliability that perhaps someone could help me look into.

I love these Jepsen posts, and Kyle Kingsbury's work is amazing.

It all shows that distributed systems are really complicated and difficult to get right. These posts on the open source ones are really great. But how do you go about evaluating a proprietary one?

Let's say my company is looking for a columnar store. We look at the evaluation of Cassandra and think, "Nope."

But then someone says, "Hey! Let's go with Redshift!" How do you know that's any better?

We all know that DB makers claim certain things. Why would Amazon be better at this tricky subject than the people behind ElasticSearch?

My question is that I would really like to know what people think about this?

Is the decision that it's better to go with the unreliable demon that you know? Or is it better to pass the responsibility off to a company like Amazon, hope for the best, and lawyer up if unexpected things happen?

I try and find engineers whose job is to run that distributed datastore in production with a scale similar to my case. Chances are really high that even in a related IRC one can find someone for any database. And just ask for their experiences.

I think he traditionally did OSS datastores because he didn't want to buy licenses for a side-project, but Jepsen uses a black box method--it creates special clients and introduces network partitions. So for a proprietary database that you host, you could try to figure out how to write Jepsen tests. That won't help for services, though.

It wouldn't work for services because you can't introduce network partitions but shouldn't they occur anyway, if you wait long enough? How much time would one have to invest until there is a reasonable chance that such a partition has occurred (often enough) to get a reasonably accurate test result?

Could be a while. I think one of the benefits of Jepsen is controlling this otherwise erratic element and introduce controlled failures within minutes rather than months.

It's not even a given that you'll know that's the problem. Some of your data is wrong, some updates are lost...are you guaranteed to see the issue and be able to rule out that it was your code?

Why no to Cassandra? I don't have the reference on hand, but I believe the issues identified by aphyr have since been addressed. It is important to do your own due diligence, especially when referencing work that is more than a year or two old.

Not using a vector clock and relying on system time is a big red flag to me.

http://www.datastax.com/dev/blog/why-cassandra-doesnt-need-v...

This is why every time I see a new Jepsen article come out my immediate reaction is actually of mild disappointment that he isn't going back and retrying Cassandra to see if it works now: by only evaluating the product enough to do an article on how bad things are, and then only evaluating fixes when he is pretty sure the authors are being dumb (redis), it turns the entire experience one into a trail of destruction rather than one of helping anyone (whether it be the database vendor or the audience) build better things. If nothing else, at least linking the article to Cassandra's own testing with Jepsen would make it live up to the feel of the database quality review site people are looking for: it also would serve to reward communities like Cassandra's that have embraced the kind of analysis and testing that Aphyr wanted.

http://www.datastax.com/dev/blog/testing-apache-cassandra-wi...

So he should work for free on your pet request?

So pay him to do a Cassandra re-test

My money is on: He'll happily retest Cassandra if paid to do so. He may even accept a reduced rate to test if the specific concerns mentioned before are gone without testing everything else again.

To clarify, I wasn't intending to speak about any particular OSS datastore or any specific proprietary one.

I meant a general hypothetical situation where I'm the decision maker and also the DBA.

I have my own experiences with different products. Likely a lot of other people do too. I don't really think that the weaknesses exposed in the Jepsen tests would be covered by your everyday person on IRC.

I mean, I'm one of those everyday people on IRC talking about the things I do and don't know about. I could tell you from personal experience that I've had a bad run with MySQL and MongoDB in production. But I couldn't tell you exactly what went wrong or where or why.

I could talk to you about the symptoms, but I couldn't address the root causes off the top of my head.

And that's exactly the point that I'm trying to ask about here.

Let's say that I know OSS database is weak in certain situations. Then I know exactly what those are and what to do to protect myself or at the very least what use cases not to deploy them for.

The same can't be said for proprietary products like Redshift. I might know, anecdotally that it does or doesn't fail, but I don't know exactly how or why or what to do to protect myself.

I was asking how other people in this situation make their decisions. Obviously, a lot of people use a lot of things that don't get CAP theorem right. And perhaps a lot of people never experience any problems. But if I'm going to be extremely thorough about this, I wonder what other people think about when making these same decisions.

If it's proprietary tech than that risk is part of the game. This is one reason FOSS zealots are so, well, zealous. Without access to the source code for auditing, and especially without access to proprietary programs for running at all, your chances of properly evaluating the software is effectively nil.

The choice between proprietary data systems seems more likely to be driven by the reputation of the company than by the reputation of the tech. The whole point of paying when OSS alternatives are available is that when something breaks or goes wrong you actually have a proverbial "neck to choke" (one that isn't your own ;) ).

There's nothing wrong with Cassandra but its reliability depends on settings you use. For example you should prefer using CRDTs. The article is a good guide to understand what's going on.

RedShift technically is a service for which you pay not a database (I did not use it but I believe it is Postgres - don't quote me on that), so if it doesn't provide what's promised you take it with Amazon.

Anyway the summarize databases. Currently we have two types:

- relational databases - old and proven, always consistent, but typically not distributed

- so called NoSQL which I think only worth attention are AP (from the CAP). These databases scale, but are eventually consistent. This means they are consistent most of the time, but not always. It's good for storing things that while important, it's acceptable when some individual entries are wrong or lost (user sessions, shopping carts, tracked information about users etc)

There are also databases like Mongo which are snake oil.

Redshift is based on tech from ParAccel.

Redshift is definitely "based" on Postgres, but its distributed systems components are probably proprietary enough that the concern is valid.

You need to look at the data model of the db and see if that works for your app or workload. You also need to look at the consistency guarantees a particular datastore provides and see if that would be acceptable.

Generally these two narrow it down quite a bit. Consider how your app(s) would deal these possibly relaxed consistency guarantees.

Hoping for the best and thinking about lawyers shouldn't be part of any evaluation strategy.

Not to be confused with Rust Crates (which I mention because I was confused).

Was my first thought as well.

It should be referred to as crate.io for this reason

Meh. I think a better root solution is picking more unique names. Like, instead of Create, why don't they call it: Rustpkg? Or maybe, Rusputin? You know, a mix of (Grigori) Rasputin and Rust. It avoids all the (many) drawbacks of giving projects too generic of a name, while also making it more memorable and fun!

"rustpkg" was the name of the thing Cargo replaced.

I think something went wrong here:

  ; For each [version, reads] pair, discard those with one value
  multis (remove (fn <a href="/data/posts/332/k vs">k vs</a>
    (= 1 (count (set (map :value vs)))))

Is this bug in Crate, or Elasticsearch? I believe Crate bypasses Elasticsearch for some things and accesses shards directly, at least to perform query pushdown operations, but are they also handling the optimistic version control?

Crate.io does skip elasticsearch and go straight to lucene in many cases, see https://crate.io/a/sql-for-elasticsearch/ for some examples. It's still so new that I don't trust it with master data though. They do have good backup/restore functionality

Reply on news.ycombinator.com