I’m a solution architect at Databricks, so feel free to skip if you think I’m too biased - just know that I worked with Databricks before working for them, and I liked the platform then as well.
While Databricks definitely started with Spark, and it’s still a significant foundation, there’s done much more on top and around it. For instance:
- MLFlow for ML lifecycle management from experiments to real-time ML serving;
- Delta Storage format, extending parquet to leverage cloud storage and enable efficient updates and very fast access;
- SQL Warehouses, which expose Databricks as a SQL engine for Analytics;
- Jobs/Workflows, which is one of the most used orchestration engines in the world;
- Unity Catalog, which will replace (at least in Databricks) Hive Metastore for metadata, access control, lineage and data governance tooling in general.
And now LLMs, on top of the data and ML capabilities mentioned above, much extended by the Mosaic deal (still to be approved).
The interesting thing is that yes, Databricks can do the whole “Data Warehousing” thing, but it can also do very large scale streaming, machine learning, process unstructured data like text, audio or images, support BI applications, etc - all accessing the same data with compatible tooling. So, it’s a full blown multi-workload data platform for any kind of use case and company size.
One can argue that most components are open-source and can be deployed independently - and Databricks has open-sourced Spark, MLFlow and Delta. It’s just that most companies simply don’t have enough (if any) staff with skills to deploy and operate all these things, let alone as one integrated platform. With Databricks, I’m used to deliver a demo where it takes me about 20 minutes from having a new cloud account to be running data workloads against a cluster or SQL, with all the functionality above.