Is ML model deployment going to be commoditized?
With all this ML-ops tools I wonder, if someday (sooner rather than later) deploying a deep learning model will be as easy as creating a API rest end-point for a web application.
The real limiting factor with complex ML models is developing training sets. My experience is that you need 500-1000 left-swipe/right-swipe judgements on texts to build a classification model that is starting to look effective but it might be more like 5000 to get a model that really seems smart. Most people wish they could get results with 5 judgements and seem to think 50 is an awful lot, with the backdrop that one person can make about 2000 judgements a day if they take it seriously. (I've done 5000+ judgements a day for visual recognition tasks for a week and found it's not sustainable because it causes my visual cortex to malfunction.)
Models that do information extraction at the sentence level take more like 20,000 training examples.
So I think the kind of tool you're talking about is only going to have a limited amount of success except for people who have the grit to invest heavily in developing training sets for problems that are really solvable with the models they're working on. I think tools for curating and developing good training sets are the 'missing link'.
You can sample classes with lower confidence detections (and at specified IOU thresholds) and bring them back in as new training data. https://blog.roboflow.com/upload-api/
If you were to combine the "smart-sampling" with things like the dataset health check for more insight into the class balance, object count histograms for the labeled data, etc. would that be a good first step in curating/developing the better training sets?
My understanding is that sampling highly uncertain examples doesn't work as well as you might think it would.
As I see it the problem is that there are "easy" examples and there are "hard" examples and whether or not the algorithm is certain about it isn't what makes it hard. For instance in text analysis I'd say "easy" problems are ones that bag-of-words works well on and "hard" ones are ones where the exact ordering and relationship between the words matter.
For some kinds of algorithms you might say that's a matter of feature engineering but for deep algorithms you've got the potential that the system might be able to learn the right features in the middle layers if it gets the right stimulus. There's still some possibility however that the structure of the model is wrong and there are some things it is never really going to learn right (like the XOR problem for single layer perceptrons)
I'm imagining for text extraction that it would be necessary to train on multiple tasks (some of which are data-rich but not directly relevant to the task, say inserting punctuation after it has been removed and others of which are of tasks you really want done) and to stratify across tasks and also stratify across easy and hard tasks.
Many problems people would like to solve break down to training a large number of similar classifiers. Consider the problem, for instance, of disambiguating word senses. This could be posed as for a given word (say "read") picking which sense out of a dictionary like WordNet is meant. This might mean training 10,000 classifiers (one for each word) with 10 or so classes each (one for each sense.) You run into all sorts of problems with this, first of all that the required amount of training data is astronomical but that with so many sub-classification problems there are going to be many where class inbalance problems are terrible, where the disambiguation might really be impossible or that are plagued with taxonomic problems. (Those, however, can sometimes work in your favor... It might turn out that 3 out of those 10 word senses really mean the same thing so far as your system is concerned as well as the other 7 out of 10.)
Just like you don't need to become an expert in cryptography to add auth to or accept payments in your app, you won't need to be an expert in machine learning to make intelligent content recommendations or add contextual search.
As an example, we do this for object detection; you add & label a few hundred example images and we train a model and stand it up as an API endpoint (and have SDK's for running it locally in a Docker or web browser).
That abstraction & separation of concerns enables hackers to build a computer vision powered app in a couple of hours[1] just like Stripe lets you accept payments in a couple of hours or Auth0 lets you add auth in a couple of hours.