Congrats to them. What's the value prop for databricks for orgs that already have data/infra engineers? I ask because in our org of ~40 engineers, our entire data warehouse (Spark on Mesos, Jupyter notebooks for analysis) is managed by 2 people, but it did take a year to build. Is it cutting down the time to getting a data warehouse MVP? Have users found that it significantly reduces ops overhead down the line?
May I ask why you have chosen Spark? I get that it it more convenient than Hadoop. But why not use an analytics DB like Redshift or Impala. What's the use case for Spark? Do you store your data in flat files and how do you access it with standard tools ala Power BI, Tableau, Looker?