I'm not an expert with either system, but I think the answer is pretty clear: that's not at all an efficient solution. Spark's approach is: do the data however you want, then use a ridiculous amount of resources getting that data into the format you want it in. You also get a ton of guarantees that you don't get with that Spark system you've proposed. Here's their comparison page[1]. If you're familiar with the technology you're suggesting it should become fairly obvious why Cockroach's nuanced approach to SQL is superior to the Spark SQL solution. For one, Spark SQL doesn't support indices. You can build one, but I'd be surprised if you could get the same general performance out of that system vs. a dedicated system like the one that's built into CockroachDB.

I know the hype is to "use NoSQL" but the reality is that there's a limited set of cases where the existing solutions truly make sense. Generally speaking, SQL is the best choice for many companies. That CockroachDB is bringing some new ideas to the SQL space is something that should be lauded. And I think they're doing it in a way that can't really be replicated easily or efficiently by current systems (which is perhaps why they're receiving so much attention.)

[1] https://www.cockroachlabs.com/docs/stable/cockroachdb-in-com...