It takes around 20 minutes to spin up HDInsight clusters. So it is not instant but I wonder if databricks on Azure can do any better given they are running in customer environments so wont have a big pool of preallocated multitenant machines available.

agree, if you have batch jobs for spark, Azure Batch Service may not be a bad solution either. HDInsight includes too much other complexity if all you wanted was spark. Databricks will be interesting, as they can take away even they mysticism of touching azure beyond initially provisioning them some rights.