Microsoft makes Databricks a first-party service on Azure

techcrunch.com

101 points·manigandham··7 comments

This is great to see. Beyond databricks itself, it has one of the slickest Notebook interfaces around - from collaboration to one-click dashboards.

Really a new day for Apache Spark now accessible to Microsoft developers to use along with all Azure cloud-native services.

Azure has already had a great spark experience available with HDInsight.

The startup time of a HDInsight cluster is quite long; it's not really suited for ad hoc clusters (which are really easy to spin up quickly in Databricks).

It takes around 20 minutes to spin up HDInsight clusters. So it is not instant but I wonder if databricks on Azure can do any better given they are running in customer environments so wont have a big pool of preallocated multitenant machines available.

AFAIK it is not multitenant, you get your own machines.

They spin up the VMs with either open source Spark or the Databricks runtime - you get to choose the distribution and version before spinning up the cluster.

If you have enough workloads you can run your own pool of VMs to provide a 'serverless' experience to your Spark users: https://databricks.com/blog/2017/06/07/databricks-serverless...

agree, if you have batch jobs for spark, Azure Batch Service may not be a bad solution either. HDInsight includes too much other complexity if all you wanted was spark. Databricks will be interesting, as they can take away even they mysticism of touching azure beyond initially provisioning them some rights.

Reply on news.ycombinator.com