If you're doing anything ML related your best bet is to niche down and solve a very narrow problem for a specific industry / market segment. Building generic AI tooling sounds great but rarely solves anyones problems.
Even Sagemaker and Fargate are mostly not useful to me as my company can easily operate its own k8s platform.
Open source tools for model building are great and a small team of engineers can write whatever odds & ends that aren’t covered, or make scalable implementations.
Notebook platforms like Databricks are similarly useless. A notebook is not a frontend to a cluster and notebooks are universally poor development environments even for the very tasks they are advertised.
The main thing that matters for ML development is how quickly & easily you can put an arbitrary container or VM into a deployment environment. You’re always going to need to rapidly change your container or VM, have common ancestors that get specialized to support GPU training or some web server application layer. Making the shortest possible distance between the definition of this environment and deployment of it is the whole game.
The type of tools I’m willing to pay for are specialized database engines, rapid data annotation tools, and specialized hardware environments.
I have no time for some “platform” tool for data provenance, notebook environments, model tracking, model-as-a-service APIs like Rekognition, or anything where I upload data and get back a model.
Any platform that advertises something like you don’t have to spend time defining your training environment, whether it’s Databricks or an out of the box deep learning VM on GCP, is a liability waiting to happen. You always need to define your own training environment, especially because you’ll almost always need specific (likely pinned) versions of all your dependencies, including system dependencies, to manage model training as part of a production life cycle. Very often you also need e.g. custom compiled TensorFlow, custom GPU settings or drivers, etc. It’s very foolish to base that environment on whatever comes out of the box.
Spark also is not universally useful. For training small models many times, like a workload that trains hundreds of small models all day (this was a production use case I had before that my company pilot tested with Databricks), the overhead of Py4J connector is insanely bad. It’s a really terrible paradigm for Python software, meanwhile Scala is a miserable ecosystem for production machine learning models. On top of all this, Spark MLib has huge gaps in functionality and whole classes of problems (e.g. large scale MCMC inference) are not solvable in a way in Spark that is seriously comparable to other tools like STAN & pymc running on not-Spark with simple multiprocessing.
Spark only serves a few special use cases that usually don’t justify its cost, similar to map reduce on Hadoop.
Databricks though (distinct from Spark) serves no use cases.
unnecessarily dismissive, grandstanding.. Bravado of the rejection, not supported at all
Rich-content Notebooks tell a story; well-worn Notebooks are a familiar context for experiment and viz.
They're incredibly non-portable; they don't play nice with version control; I know how they're supposed to be used, but I'm yet to see them actually used in a way that isn't a breeding ground for the worst software engineering I've seen.
Personally I would be happy with just proper version control support.
A notebook frontend however is less than useless, and is actively harmful by propagating poor notebook environments even further into aspects of computing where they cause harm and hurt reproducability and code factoring.
Given this, even if Databricks offered perfectly complete features for all aspects of cluster computing, it would still be inferior to just my own managed EC2 or EMR clusters or equivalent with other providers, where there is no “notebook as control plane” garbage.
But when you add to that the fact that Databricks lacks full features for me to totally own every customized detail of my cluster environment (e.g. how can I run plain Python multiprocessing tasks with zero Spark in Databricks? How can I bring my own custom defined GPU container with custom compiled Tensorflow?) it makes the deal even worse.
Databricks is just another Spark / Hadoop style snake oil seller banking on capturing a bunch of data science teams before people realize that it’s a conceptually junk way to work.
As for the other tooling you mention, I’d almost always say to build it in house. For example, I don’t know of a single A/B testing provider that actually uses frequentist sequential testing to correctly avoid early stopping bias. You actually need real statisticians hired in-house to solve these problems, and the engineering work to set up an extensible A/B test as a service internal tool is not bad. (I’ve built Bayesian A/B test frameworks with teams of 2-3 engineers in 3 different companies). It’s just not cost effective to outsource it on the false hopes of not needing to hire your own in-house statistics experts. Just pony up the dough and hire them.
Actually, Azure already has ML.NET, but it is about classical machine learning, and their own deep learning product, CNTK, lost the marketing fight with TensorFlow and PyTorch. Which is where I come in :)