Maybe unrelated but Databricks is the most annoying garbage I have ever had to use. It fascinates me how anyone uses it by choice.
The product is still centered Spark, but most companies don't want or need Spark and a combination of Iceberg and DuckDB will work for 95% of companies. It's cheaper, just as fast or faster and way easier to reason about.
We're building a data platform around that premise at Definite[0]. It includes everything you need to get started with data (ETL, BI, datalake).
There are some examples[0] of enabling DuckDB to manage distributed workloads, but these are pretty experimental.
What are some bad UX choices you generally dislike in data products?
The biggest gripe in have is how crazy expensive it is.
But these days just use trino or whatever. There are lots of new ways to work on data that are all bigger steps up - ergonomically, performance and price - over spark as spark was over hadoop.
What an impressive feat of engineering.
Today it suffers from feature creep and too many pivots & acquisitions. That they are insanely bad at naming features doesn't help either.
I really don't understand the valuation for this company. Why is it so high.
But if you're inclined to use it, databricks' setup of spark just saves you an incredible amount of time that you'd normally waste on configuration and wiring infrastructure (storage, compute, pipelines, unified access, VPNs etc). It's expensive and opinionated, but the data engineers you need to deal with spark OOM errors constantly is greater. Also databricks' default configs give you MUCH better performance out of the box than anything DIY and you don't have to fiddle with partitions and super niche config options to get even medium workloads stable
* No persist(). Not being able to cache dataframes is a nightmare when it's a workflow that involves taking a massive source of data, doing some rough filtering on it that gets it down to a tiny subset, and then doing more complex stuff with that.
* No good way to get usage info programmatically that I've found. For things like monitoring for periodic queries that get out of hand.
* Can't set Spark config. There are often ways to get around this, like when I recently had to set S3A credentials and needed a way that wasn't OS environment variables (this doesn't work for worker nodes). Eventually, through much documentation browsing and finally an exasperated hail mary question to ChatGPT that solved it (told me the things to pass into options() ) I got it working. But all the documentation and online QA resources just say to use Spark config.
* This is more of a Unity Catalog problem, but kind of applies because Serverless and UC often go very hand in hand (particularly when dealing with things that used to be stored in a cluster like credentials), but it drives me insane that I can only mount external volumes with the same block storage as my workspace provider. So I can't mount an external volume to an AWS bucket on an Azure UC. That means if I want to write stuff that can run the same regardless of what my customer is running their Databricks workspace with, I need to use less sophisticated approaches.
It's still nowhere near the pain that Databrick's attempt at copying Snowflake's VARIANT data type has caused me, but there are many times when I find myself having to work around serverless limitations. Especially when these limitations aren't really mentioned much upfront when Databricks pushes serverless aggressively.
Databricks' proprietary notebook format that introduced subtle incompatibilities with Jupyter was infuriating embrace-extend-extinguish style bullshit, but on-prem cluster instability causing jobs to crash on a daily basis was way more infuriating, and at that time, enterprises were more than happy to pay a premium to accelerate analytics teams.
In the 2010s, Databricks had a solid billion-dollar business. But Spark-as-a-Service by itself was never going to be a unicorn idea. AWS EMR was the giant tortoise lurking in the background, slowly but surely closing the gap. The status quo couldn't hold, and who doesn't want to be a unicorn? So, they bloated the hell out of the product, drank that off-brand growth-hacker Kool-Aid, and started spewing some of the most incoherent buzz-word salad to ever come out of the Left Coast. Just slapping data, lake, and house onto the ends of everything, like it was baby oil at a Diddy Party.
Now, here we are in 2025, deep into the terminal decline of enshittification, and they're just rotting away, waiting for One Real Asshole Called Larry Ellison to scoop them up and take them straight to Hell. The State of Florida, but for Big Data companies.
It would be a mystery to me too, why anyone would pick Databricks today for a greenfield project, but those enterprises from 5+ years ago are locked in hard now. They'll squeeze those whales and they'll shit money like a golden goose for a few more years, but their market share will steadily decrease over the next few years.
It's the cycle of life. Entropy always wins. Eventually the Grim Reaper Larry comes for us all. I wouldn't hate on them too hard. They had a pretty solid run.
Can't imagine someone incapable of building a website would deliver a good (digital) product.