The Machine Learning Race Is Really a Data Race

I currently believe this is how most ad tech runs internally. Scammers scamming scammers.

So I'd say if you have a very specific area that you're investigating you have a very good chance of beating larger players that don't specialize as much as you can. Of course competing against Google in self-driving cars or machine translation might be a bad idea, but even in those areas there are small startups that produce impressive results (e.g. DeepL: https://www.deepl.com/en/translator). Also, big companies regularly exaggerate their capabilities as well (sometimes more than startups), just have a look at how IBM markets their Watson AI/ML solutions, and what they deliver in reality.

So personally I'd say it has never been that easy to build relevant and interesting ML/AI based solutions as a small team, and it is possible to beat large players if you have the right approach and the right (very narrow) problem.

Call me a cynic, but we can't even get companies to do the bare minimum to protect data that is centralized, and you expect them to do all that?

I'm not kidding when I say that your last sentence is a bigger, harder and more important task than the machine learning it is supposed to facilitate.

I was sure someone will say this. I think it only applies for advertising, commerce and insurance. When it comes to learning ML models on images, text and games, big corporations don't have a unique trove of data. They only have the data advantage when it comes to personal data.

The more important advantage big corporations have is hiring the best and hiring more ML scientists and engineers. The job demand far outstrips the offer.

Can you provide some references for this?

This seems to hold fairly well, with a few caveats. For example, without enough data you ar wasting your t8me with some techniques. Also “data quality” most often needs to include quality labeling in practice (but you may have meant that inclusively).

For example, let's say you want to build a speech rec engine. You need 15K hours of data to build/validate a model. How would you get that? You could farm it out to some people on mechanical turk and get 15K hours of audio transcribed. With enough money, you could duplicate the transcriptions enough times to actually be pretty sure about the quality of the data set. If you're clever and have a large enough dataset, segmentation generally gets you decent quality. The big gains come in when you have features not present. For example, google realized when you build a speech rec engine, you can include video data and image processing to actually use the way people move their mouths to significantly increase the quality of an automated transcription.

Business strategy is fundamentally about trade-offs, though, so we do need a caveat. Naturally, more choices and more insights can, at times, be a weakness. You always have to prioritize, and as data volume increases it doesn't seem the ability for quality prioritization grows in parallel (or at least, necessarily does).

The ideal approach would somehow allow this anonymized access to multiple large databases simultaneously. I don't know how you'd do that but if you claim Etherium would help, you'd arrive at buzzword nirvana even if you were wrong.

What moves specifically?

I think the key fields in AI will become simulator based learning (RL) and graph processing neural nets, because graphs can express any kind of highly dimensional data and are useful for reasoning tasks. They marry the symbolic and connectionist approaches. These two subdomains have had rapid evolution over the last couple of years. They also solve the data problem - in simulation you can produce as much data as you want, and graphs have combinatorial generalisation, thus they work on new configurations without retraining.

I could show someone a single photograph of an animal they have never seen before and they will recognize it forever, from different angles, in different lighting, in black and white, probably even from a silhouette.

The more I dig in to ML the more it seems like it's cheating, like a mathematical trick. At least the way we are using it.

I can't help but feel like it's a game of Pachinko with pixels instead of balls, and neural weights instead of pins and holes.

I'm no expert though so take my opinion with a grain of salt.

If we can devise ML algorithms capable of generalizing from substantially smaller datasets, while data will still be relevant, it will be a less important bottleneck in the process than it is now.

This is a result of the ML paradigm essentially starting new with whatever data set it is trying to "solve" but overall, the paradigm doesn't have to be that.

So by the metric of model performance (the only thing that matters here), what you're saying is that hiring more labelers is actually not linear.