We use Databricks on a data processing pipeline. I don't know of anyone in love with it. It is by far the number 1 source of problems on that pipeline. Just in the last week:

* It deleted hundreds of log files without warning

* We had a failure starting a cluster; the web UI listed the cluster, but in fact it no longer existed - we had to recreate it.

* Log files that it didn't delete show that it is having problems pulling some internal metadata from an AWS IP address (we are on Azure).

If the directive from on high came down that we are to rip it out and replace it with something else, nobody would be surprised, or care.