Then what is "Auto ML"? It sounds like just another cloud service.
Correct, it's a cloud service, based on the research Google published for model exploration[1][2]. There are research examples today where this service provided better models than humans were able to achieve by hand or with genetic algorithms (models trained faster and/or with better error rates)[3]
[1] https://static.googleusercontent.com/media/research.google.c...
[2] https://blog.acolyer.org/2017/10/02/google-vizier-a-service-...
Note I left it off of this one. :)
Eventually useful for building a very basic model to serve as a benchmark for a real model.
There are a few different reasons:
1) Most companies have very limited skill in building advanced models. While many folks are trying to achieve 5% or better accuracy, MOST folks are trying to achieve ANY accuracy (since they have nothing)
2) Many problems are not very complicated and do not require a custom model. From the blog post, the "cloud" example only requires a small amount of changes to classify for a specific domain - to have to build an entirely new model for that, or train on MM of images seems like overkill
3) AutoML (often) is better than humans already[1]. So if you want to achieve that 5%, you MIGHT need to use a machine anyway.
And second, your comment that most folks achieve any accuracy is strange. These are not real businesses, they are mostly developers and hobbyists trying to learn. These folks sign up for kaggle and poke a few scripts and view half a class on coursera on ML. They are not real businesses and they have no money. Most of the real businesses are hiring startups or large companies that hire data scientists with domain expertise like in oil, manufacturing etc. (the IBM model). ML as an API is a disaster as a business model.
Also, AutoML is not even close to being better than humans even for a specific problem (across datasets). These click-bait titles don't fly outside of AI conferences.
Put another way, the average customer has ~zero ML usage today. I'd guess that 95%+ of all businesses have ML usage today. Further guessing would say that <1% of ML usage actually care about levels of accuracy beyond "it's better than the hacked together set of rules/filters I use today." These are very large businesses with lots of money to spend on a solution.
There are many ways to measure "better", and AutoML does apply here. This includes "better == faster to train or develop" [1], "better == you need less data" [2], "better == lower error rates"[3]. While I agree that many of measures do not apply across datasets, most customers only have one dataset per problem.
[1] Predictive accuracy and run-time - https://repositorio-aberto.up.pt/bitstream/10216/104210/2/19...
[2] Less data - https://arxiv.org/abs/1703.03400
[3] https://link.springer.com/article/10.1007/s10994-017-5687-8
Disclosure: I work at Google on Kubeflow
- Average customer has zero ML
- Nearly no customers are using any ML (difference between median and mode)
- Of those that are using, very few care about better than human perf
Is the argument that it is easy to implement stock models but hard to tune the models for specific types of image inputs? Inst that pretty easily solved with some parameter grid searches? How much specialized skill does it take to re-do networks from traditional inception architecture or what not into something specific for hot-dogs or satellite imagery or medical images?
* half the student interns I interview from top-20 comp sci programs do this on weekends for hackathons*
It's trivially easy to take what someone else has built and modify it slightly for a similar problem, especially in a hackerthon environment where you can ignore edge cases etc.
See if they can build a new model from scratch for a new type of problem. I'm not saying that AutoML can do this either, but I interview large numbers of PhDs who don't know where to start on doing something new.
"It's very basic but you can use the model as a benchmark" is actually the pitch you get from AWS.