As someone new to the data ops platform world.. what does Databricks even do? They get so much hype as a great employer, but why are they attracting people if their secret sauce is irrelevant?
While Databricks definitely started with Spark, and it’s still a significant foundation, there’s done much more on top and around it. For instance:
- MLFlow for ML lifecycle management from experiments to real-time ML serving;
- Delta Storage format, extending parquet to leverage cloud storage and enable efficient updates and very fast access;
- SQL Warehouses, which expose Databricks as a SQL engine for Analytics;
- Jobs/Workflows, which is one of the most used orchestration engines in the world;
- Unity Catalog, which will replace (at least in Databricks) Hive Metastore for metadata, access control, lineage and data governance tooling in general.
And now LLMs, on top of the data and ML capabilities mentioned above, much extended by the Mosaic deal (still to be approved).
The interesting thing is that yes, Databricks can do the whole “Data Warehousing” thing, but it can also do very large scale streaming, machine learning, process unstructured data like text, audio or images, support BI applications, etc - all accessing the same data with compatible tooling. So, it’s a full blown multi-workload data platform for any kind of use case and company size.
One can argue that most components are open-source and can be deployed independently - and Databricks has open-sourced Spark, MLFlow and Delta. It’s just that most companies simply don’t have enough (if any) staff with skills to deploy and operate all these things, let alone as one integrated platform. With Databricks, I’m used to deliver a demo where it takes me about 20 minutes from having a new cloud account to be running data workloads against a cluster or SQL, with all the functionality above.
Snowflake stores data in S3 in Snowflake's various AWS accounts. Egress fees are necessary when performing anything outside of Snowflake.
Databricks operates on data stored in your S3 on your AWS account(s). Databricks also runs on your compute contained within your AWS account(s).
Both approaches have valid use cases.
Egress fees are only if you are unloading into a different region/cloud. If your data is in SF AWS us-east-1, and you unload to your S3 in us-east-1, there is no egress fee.
> Snowflake charges a per-byte fee for data egress when users transfer data from a Snowflake account into a different region on the same cloud platform or into a completely different cloud platform. Data transfers within the same region are free.
https://docs.snowflake.com/en/user-guide/cost-understanding-...
Usually, no. But in some cases, yes.
If you * need * it, you find a human to talk to (sales or connect from your network).