Databricks is an RDBMS
fivetran.com
fivetran.com
As a DS/DE, there's a lot to love (not all, but a lot). The easy provision of Spark clusters. The jobs API. DeltaLake (mostly). Easy notebooks (please don't create a prod system from these..). And Spark itself continues to improve, albeit in an increasingly crowded field.
But I've worked closely with BigCo SQL analysts on Azure Databricks, and their experience was terrible. For example:
- You cannot browse the data structure without an active cluster
- Starting a cluster can take ~5 minutes and, since you missed that moment, you may not submit your first query until 10-15 minutes.
- The SQL error messages are often (perhaps usually?) nonsense, so you have to operate without them.
- An unfortunate amount of downtime, followed by bizarre excuses.
- It's so darn slow, relative to equivalent queries on BigQuery or Snowflake.
- Even submitting a query can take a weird amount of time.
If Databricks-as-an-RDBMS were competing against Teradata, sure, let's have a chat.But we're in 2021, and there's just no comparing the experience of the SQL analyst on Databricks-as-an-RDBMS vs. Snowflake/BigQuery.
I'm excited for the potential of Snowflake's SnowPark (though know little about it). Calling UDFs from SQL means you can create great features for SQL analysts, provided that they can build the momentum to need it.
Teradata is a lot faster for interactive workloads than Databricks.
PS: I agree there's no comparing on Databricks vs Snowflake/BigQuery.
Good luck on your budget trying to scale up shared-nothing database and making it scale up and down based on workload without downtime.
You can achieve significant speedup resembling shared-nothing databases by pushing the data close to the query using caching. Snowflake does it out of the box as it maintains table metadata. Databricks can do it too, but you have to be careful and it sucks.
Self hosted HDFS will work more like shared nothing.
And having control over the cache sucks as hell. You can't pin down a table to reside in compute node disk cache when you know it will be used often.
Teradata was faster with no user-facing tuning vs tuned Databricks. And if you can pay for Teradata you may as well use it.
In shared-nothing the data "lives" on the compute nodes, so you can't willy nilly add or remove nodes. The data would either get lost if you removed more nodes than what is necessary for triple replicated redundancy. Or if you added nodes you'd have to wait before the data gets rebalanced to those new nodes, resulting in massive reshuffling of everything.
Keep in mind that shared-nothing clusters are most likely long lived and have multiple large datasets sitting on them, so by adding nodes you will start to shuffle everything around.
With shared-disk your compute cluster is only a single use for one dataset and you don't lose data as you scale cluster up and down.
There is a hybrid architecture which will give you both, but doesn't exist out there: Shared-disk with active caching. Ie. giving you control over which tables will be pinned down on your temporary compute-cluster. That will give you performance of shared-nothing but with convenience of shared-disk temporary compute clusters.
Their datasets are small. Most tables are ~50GB, the odd table up to ~2TB. The clusters typically are nothing shabby for this size, defaults to ~[4-12]x32GB.
The queries that I have seen are typically not written well. Think view-on-view-on-view (there's a BigCo policy against them materialising data..), and where the filter is applied in the last step. The stuff of horrors, but something I've seen in more-than-one-BigCo.
But we have compared some of those same queries on BigQuery vs. Databricks, and, I don't know if BigQuery's execution optimiser is better? Or if the BigQuery storage is better organising the data? Or if BigQuery is simply throwing more resource their way?
That sounds wonderful (really). I was contracting for a BigCo where they materialised things all the time, and they would regularly end up running queries over multiple materialisations from different points in time, which invariably means that you always get wrong answers. I very much wished to put a stop to use of any materilised views, but didn't have the buy-in to make the policy.
Was going to say. Most of the times all it takes is to have a proper data model.
For analytics I favour de-nomarlized schemas and, if necessary, nested fields. Queries are much easier to write (fewer joins), much faster, no need to incrementally materialize (sigh), fewer backfills and no messy field definitions.
What you often see instead is highly-normalized data models with an un-trackable amount of materialized views (usually on top each others) and some complicated tools/solutions to try to deal with all that mess. The cost of a bad design.
That's one of the reason's I'm interested in delta-rs [1], which has delta lake bindings for Python. Would love to read a delta lake table into a native python object without the need for spark.
You can do that in Spark, no?
I see it as why the article supports Databricks as an RDBMS; it offers something others do not.
You can't currently* do the same extensive UDFs in Snowflake or BQ and, sometimes, they are important. But with SnowPark coming, hopefully you won't have to make such a large sacrifice to SQL users' experience for it.
* Currently you can do JavaScript UDFs and external functions in Snowflake, and BigQuery ML is worth mentioning here too. Those cover some, but not all, of what you might use a Spark UDF for in SQL.
They have thought about how they can improve the DS experience. Inconsistent storage? DeltaLake. Slow Spark queries? Databricks Delta. Model management? MLFlow (I haven't adopted this, but can't pin down why -- on face value it seems great). Development environment? Databricks Connect. Cluster management? Core.
But the same is not true for SQL analysts. Today's offering does not empathise with them. I'm unsure integrating Redash is a genuine reply to their needs.
The upside here is that (1) Databricks (or at least, Databricks' marketing) appears to be prioritising this need, and (2) A lot of people are betting a lot money that they can do this well.
Tomorrow looks sunny.
Probably because you want your code to be about the problem you're trying to solve, not about tracking experiments. Similar to Anti-lock braking system or Electronic stability control systems in a car: you want them to be "on" by default, not to activate them every five minutes while driving.
(disclaimer: plug)
Yes, it's very good if you don't like setting up clusters (few do/can) and the UI is rather useful for getting up and running (not so much for writing code though). But you need to really understand the platform before adopting it. Please, please don't just adopt it because it's popular.
Because it's unnecessary to those 99% of companies. If company data fits in an Excel spreadsheet, Access database, or a single MySQL/Postgres instance, then introducing all this new tech with all of the associated costs and little return gain is a net loss.
Often time the query perf per machine goes down as you move from a single node, indexed, postgresql To a multi node unstructured data.
If you don’t have so much data or unstructured data. Why use the more complex and generally slower per FLOP solution.
What would "this new tech" bring us and our customers, and would it be enough to justify the cost of porting?
1. It is not necessary. For smaller datasets excel or sql servers are sufficient.
2. The small dataset has grown over the years and a other solution would be beneficial. But people like the things they are familiar with. They do not want to have the additional work, of learning new tools. Who ever wants to change the existing structures, will usually face high resistance.
In the second case, companies often accept the issues of the old solution and they tell themself it is the right way.
I enjoyed the pun.
I'm still coming to terms with the fact that there's actually a technology called Kafka...
/sorry, I'm just particularly bitter after having to do some databricks training yesterday... it was basically 80% trying to rote the sales literature about why they're so great/enterprisy and parroting company and platform specific jargon. I don't like seeing my profession turn into an obsession with tech and platforms when 99.9% of companies and people can't reason properly or operate their current tools/resources efficiently. obviously just my own opinion.
I will try to look up the link and obviously 'he' can be 'she' as well.
I don't know about this, I would be a bit hesitant if the carpenter to build my kitchen cabinets only knows how to use hammer and a chisel.
I hope you don't mind, it was, to my eye and ear, an instant classic.
It’s all buzz words, no content and please signup now.
How so this link so we’ll upvoted?
Did people just see “databricks” and think ooo... I have an opinion on that?
Read the article before you upvote folks; this one doesn’t deserve your time.
Everyone I know who uses Databricks (and they all like it) use it as hosted Spark with S3 integrations, or write... directly to Snowflake. I'm a little skeptical they're going to get traction as a true data lake model
You will have to develop some kind of data lake to store unstructured data anyway. You will end up with a Snowflake data warehouse and a data lake. Why not just go with data lake first then.
Databricks/Spark are just good platforms to help you do something with structured data in your lake. With the recent additions to its execution engine and Delta (strange naming tbh) it will be pretty much the same as Snowflake for you.
You need something more complicated than what can be done using BigQuery/Snowflake (that remaining 20%, though I would say 10%)? Export the dataset to CSV/Parquet/Avro/ORC/whatever and process it with anything, including Dataproc/HDInsight/EMR or even Databricks. That's actually a common pattern.
Can't do a simple
select * from s3://file.parquet
which you can do in Spark. Having to load it into the data warehouse means that you duplicate your data two times and it is stupidly annoying.Many times the data doesn't even resemble anything tabular before I structure it in python scripts. Why would I load it inside a warehouse only to then pull it down to do some python processing and loading it back. Which makes the data travel from DWH storage to a data lake and then to my compute cluster and then the same cumbersome roadtrip back. Pretty wasteful. Spark at least allows me to schedule a python function across a cluster while copying only from lake to my compute node and back.
Data warehouses like BQ and Snowflake are great for data scientists after a bunch of engineers slice and dice raw data into clean tables. For anyone working with not yet structured data, data lake wins hands down.
[1] https://docs.snowflake.com/en/user-guide/querying-stage.html [2] https://docs.snowflake.com/en/user-guide/tables-external-int...
External table in Snowflake only allows you to ingest data from s3 to their storage (which is also s3 behind the scenes).
Perhaps something has changed since the last time I tried, but when I tried my conclusion was that "external tables" in Snowflake are not what you think they are.
Also I have not seen examples of "select * from s3://file.json" in the links you provided.
I would like to be corrected if I'm mistaken.
No. External table doesn't do any "ingesting" whatsoever. You can if you want to but you thats not the primary usecase.
https://docs.snowflake.com/en/user-guide/tables-external-int...
" enables querying data stored in files in an external stage as if it were inside a database "
> I have not seen examples of "select * from s3://file.json" in the links you provided.
you cannot directly query a random file off of s3( not sure why someone would want to do that) . Snowflake is a database , not a python script.
You have to build a an external table
" create external table abc location=s3://dir-to-file "
then
" select * from abc "
> "external tables" in Snowflake are not what you think they are.
They are exactly what you think they are. Tables stored externally to snowflake.
That's not really true; Snowflake can query directly from S3 just fine, even from other clouds. You just need to set up the credentials (or supply them in the query, but that's not usually a good idea).
Disclaimer: I work for Snowflake
This is not true. Snowflake allows you to create an external stage (pointer to s3) and then query any prefix you want as long as you provide the correct file format arguments or types.
select * from @somestage/someprefix file_format(....)
edit: hadn't refreshed the page and i can see others have already responded to this point.
Pretty similar as defining a Hive Table and then using any other engine to process it.
PS: BigQuery Omni (now beta) will support object storage solutions from other cloud providers.
Snowflake has spark connector too. So I don't know what the difference would be writing a spark job against deltalake vs snowflake.
> 80% of the things you will need - which is great for some newbie stuff or for sales presentations.
This is obviously wrong.
> You will have to develop some kind of data lake to store unstructured data anyway. You will end up with a Snowflake data warehouse and a data lake. Why not just go with data lake first then.
We store unstructured data in snowflake. I don't understand why you need a datalake on top of it.
I could do the same in s3 for much cheaper.
Snowflake supports enough SQL constructs to allow for very complicated queries. If that doesn't suit your needs then there's stored procedures and custom javascript UDFs you can write. That covers probably 99+% of the use cases at most companies and usually the rest can be done somewhere else on pre-aggregated data.
2. People usually load complex Python libraries for data processing. I wonder if Snowflake UDF would support that or just allow you to use standard library.
Delta lake is something you can run on data you feel you still have some semblance of control over.
You don't own the underlying storage, true, but there's a defined method of getting data in and a defined method of getting that data out. The API is different to S3, sure, but it's an API all the same.
1. from snowflake storage to s3
2. from s3 to my compute node
3. do the processing
4. store result in s3
5. copy result from s3 to snowflake storage
If I only use s3, my data gies like this 1. s3 to compute node
2. run my script
3. result to s3
Perhaps I could use bunch if insert statement and load data into Snowflake through sql straight feom compute node, but then at a great cost.I don't really see much benefit of using Snowflake for not-yet structured data. For structured data it works wonders. Only downsides there:
1. Expensive as f when you scale up
2. can't control cost in pay-per-query model.Is making money by tricking customers into product lock in really a good business strategy. Wouldn't the word get out sooner or later?
dont you have to pay aws for s3,emr /data engineers monthly and forever too.
My company loves them, I think they only do a few things good of which marketing the best, and are not worth the money for data science teams with devops skills. Happy to hear from others if I am wrong.
I have a feeling legacy databricks installations will be much derided in 5-10 years time
Felt like the CEO wanted to make a strong point, but has a relationship with Databricks that he didn't want to undermine, so did some reclassification gymnastics to make it all fit. But I am probably too cynical.
Finally, I beg for mercy from the 'lakehouse'. It is a bridge too far ;-)
[1] https://www.forbes.com/sites/forbestechcouncil/2021/02/01/th...
(Why no vector clocks in the manifest files or something?)
disclaimer: an author of Nessie
link seems to be broken.