Considering the makers of Databricks are also the maintainers of Spark it makes you wonder whether this is deliberate