Hacker News Reader

I've seen and have compared Databricks clusters to a 10-15yo Teradata cluster and no way in hell I would use Databricks.

Teradata is a lot faster for interactive workloads than Databricks.

PS: I agree there's no comparing on Databricks vs Snowflake/BigQuery.

Teradata is shared-nothing architecture. Of course it will outperform Databricks and Snowflake shared-disk model as the data is colocated on the compute nodes so it doesn't have to travel anywhere.

Good luck on your budget trying to scale up shared-nothing database and making it scale up and down based on workload without downtime.

You can achieve significant speedup resembling shared-nothing databases by pushing the data close to the query using caching. Snowflake does it out of the box as it maintains table metadata. Databricks can do it too, but you have to be careful and it sucks.

Wait what? I thought spark is designed to work on top of a shared nothing cluster? Have you any resources describing the architecture of databricks specifically?

In Databricks you only get cloud storage (s3) as a persistent storage for data. So the data has to move from s3 to compute nodes.

Self hosted HDFS will work more like shared nothing.

Ah,thanks, makes sense.

If Teradata is faster for workload X or Y, it isn’t because of a shared nothing architecture. Databricks and Snowflake both cache the working set on local disks of the worker nodes, so in practice they are shared nothing systems from a performance perspective.

Only after using the data beforehand it gets cached or by actively pushing it into cache. Otherwise everything sits in cold storage far away.

And having control over the cache sucks as hell. You can't pin down a table to reside in compute node disk cache when you know it will be used often.

Fair enough, I'm just talking from the perspective of a SQL analyst who could use both Teradata and Databricks at work.

Teradata was faster with no user-facing tuning vs tuned Databricks. And if you can pay for Teradata you may as well use it.

Do you have any resources or examples of challenges with scaling up a shared-nothing architecture? At the crossroads between a shared-nothing and shared disk decision at the moment.

With shared-disk the data sits in cold storage (eg. s3) for which you pay cheaply. Only when you have to do some big computation you spin up a temporary compute cluster, do the computation, store resukt back in s3 and kill the cluster. You can also have long lived compute clusters and shut them or scale them down at night when nobody uses them. Since the data doesn't "live" in the compute nodes, you can add or remove as many compute nodes as you wish.

In shared-nothing the data "lives" on the compute nodes, so you can't willy nilly add or remove nodes. The data would either get lost if you removed more nodes than what is necessary for triple replicated redundancy. Or if you added nodes you'd have to wait before the data gets rebalanced to those new nodes, resulting in massive reshuffling of everything.

Keep in mind that shared-nothing clusters are most likely long lived and have multiple large datasets sitting on them, so by adding nodes you will start to shuffle everything around.

With shared-disk your compute cluster is only a single use for one dataset and you don't lose data as you scale cluster up and down.

There is a hybrid architecture which will give you both, but doesn't exist out there: Shared-disk with active caching. Ie. giving you control over which tables will be pinned down on your temporary compute-cluster. That will give you performance of shared-nothing but with convenience of shared-disk temporary compute clusters.

I got excellent performance in Databricks with well partitioned Parquet and Spark 2.4. What is making the queries slow? Data scanning?

They use DeltaLake + Spark 3.0, and are mostly careful to partition well.

Their datasets are small. Most tables are ~50GB, the odd table up to ~2TB. The clusters typically are nothing shabby for this size, defaults to ~[4-12]x32GB.

The queries that I have seen are typically not written well. Think view-on-view-on-view (there's a BigCo policy against them materialising data..), and where the filter is applied in the last step. The stuff of horrors, but something I've seen in more-than-one-BigCo.

But we have compared some of those same queries on BigQuery vs. Databricks, and, I don't know if BigQuery's execution optimiser is better? Or if the BigQuery storage is better organising the data? Or if BigQuery is simply throwing more resource their way?

> Think view-on-view-on-view (there's a BigCo policy against them materialising data..), and where the filter is applied in the last step. The stuff of horrors, but something I've seen in more-than-one-BigCo.

That sounds wonderful (really). I was contracting for a BigCo where they materialised things all the time, and they would regularly end up running queries over multiple materialisations from different points in time, which invariably means that you always get wrong answers. I very much wished to put a stop to use of any materilised views, but didn't have the buy-in to make the policy.

> That sounds wonderful (really)

Was going to say. Most of the times all it takes is to have a proper data model.

For analytics I favour de-nomarlized schemas and, if necessary, nested fields. Queries are much easier to write (fewer joins), much faster, no need to incrementally materialize (sigh), fewer backfills and no messy field definitions.

What you often see instead is highly-normalized data models with an un-trackable amount of materialized views (usually on top each others) and some complicated tools/solutions to try to deal with all that mess. The cost of a bad design.

In my experience Athena on AWS beats Spark by an order of magnitude in terms of performance and price. Without the annoyance of having to start/stop/run a cluster. Snowflake is even faster than that since its storage is more optimized.

And Athena is an old fork of Presto, provided as a service; modern Presto e.g. Starburst is much faster than Athena, and cheaper if you use it much as Athena has usage fees not hosting fees.

> You cannot browse the data structure without an active cluster

That's one of the reason's I'm interested in delta-rs [1], which has delta lake bindings for Python. Would love to read a delta lake table into a native python object without the need for spark.

[1] https://github.com/delta-io/delta-rs

Yes, and I would add to this the (nearly) complete lack of IDE support makes working with Spark SQL quite painful.

And on the top of that if i need transactions and SQL Data Lake wouldn’t be my first choice. There are other options.

> Calling UDFs from SQL means you can create great features for SQL analysts

You can do that in Spark, no?

You sure can :)

I see it as why the article supports Databricks as an RDBMS; it offers something others do not.

You can't currently* do the same extensive UDFs in Snowflake or BQ and, sometimes, they are important. But with SnowPark coming, hopefully you won't have to make such a large sacrifice to SQL users' experience for it.

* Currently you can do JavaScript UDFs and external functions in Snowflake, and BigQuery ML is worth mentioning here too. Those cover some, but not all, of what you might use a Spark UDF for in SQL.

but they have $400MM of 2020 revenue. How can they do that with such a disappointing product?

Databricks is good as a managed Spark platform.

They have thought about how they can improve the DS experience. Inconsistent storage? DeltaLake. Slow Spark queries? Databricks Delta. Model management? MLFlow (I haven't adopted this, but can't pin down why -- on face value it seems great). Development environment? Databricks Connect. Cluster management? Core.

But the same is not true for SQL analysts. Today's offering does not empathise with them. I'm unsure integrating Redash is a genuine reply to their needs.

The upside here is that (1) Databricks (or at least, Databricks' marketing) appears to be prioritising this need, and (2) A lot of people are betting a lot money that they can do this well.

Tomorrow looks sunny.

>Model management? MLFlow (I haven't adopted this, but can't pin down why -- on face value it seems great).

Probably because you want your code to be about the problem you're trying to solve, not about tracking experiments. Similar to Anti-lock braking system or Electronic stability control systems in a car: you want them to be "on" by default, not to activate them every five minutes while driving.

I would guess high cost, market saturation, and not enough research and development. It seemed like an ok product when we evaluated them, but the cost was prohibitive. After the hortonworks and cloudera merger, the future of the hadoop/spark open source ecosystem seemed extremely dull and we came really close to using either databricks or one of their competitors. We found that spark on kubernetes appears to have a bright future. So, we didn't have to double our spark cluster costs by going to databricks.

Sigma Computing (https://www.sigmacomputing.com) might be a good fit here too.

(disclaimer: plug)