Hacker News Reader

For the record, he's literally listed off most of my languages: I have years of experience in SAS, R, Python, and SQL, and I also have no idea what he's saying.

Also, for the record, that's largely how I think of databricks ideally. A big distributed database/storage layer into which I can write queries in my chosen language...

So i...admit equal confusion...

View on HN

As someone who does understand the distinction, for most end-users, I think the distinction doesn’t matter. It’s like the culinary vs botanical categorization of a tomato.

The performance of different queries is different across a typical RDBMS and something like Spark/BigQuery.

A traditional database is efficient (i.e. cost effective) at doing a lot of the same query repeatedly, e.g. looking up a customer's account balance, whereas these query engines are good at doing infrequent queries with lots of complicated, expensive logic on very large datasets.

You could run simple account lookup queries in a CRUD app with Spark, but you'd be setting a lot of compute/money on fire, and your latency would probably be terrible.

There are a lot of optimizations in query engines as well - results cache, local caches of files stored in cloud, data layout optimization and skipping (to avoid reading not necessary files), etc.

Reply on news.ycombinator.com