I've worked on a dozen ml projects two of them before Alex net came out and I've never gone wrong by spending 80% of my time creating a dataset specific to the problem and then using whatever algorithm is top dog right now.
Labelled data is king.
I have a classification task where I can train multiple models to do automated evaluation in about 3 minutes using BERT + classical ML. The models are consistently good.
Sometimes you can do better fine-tuning the BERT model with your training set but a single round takes 30 minutes. The best fine-tuned models are about as good as my classical ML models but the results are not consistent and I haven't developed a reliable training procedure and if I did it would probably take 3 hours or more because I'd have to train multiple models with different parameters.
Even if I could get 82% AUC over 81% AUC I'm not so sure it is worth the trouble, and if I really felt I needed a better AUC (the number I live by, not the usually useless 'accuracy' and F1) I could develop a stacked model based on my simple classifier which shouldn't be too hard because of the rapid cycle time it makes possible.
My favorite arXiv papers are not the ones where people develop "cutting edge" methods but where people have a run-of-the-mill problem and apply a variety of run-of-the-mill methods. I find people like that frequently get results like mine so they're quite helpful.
Sometimes it's just ... hard. Apply some thought maybe before blindly parroting "profit!"
Reporter: "Why is it hard to cure cancer?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!"
Reporter: "Why is it hard to end World poverty?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!"
Reporter: "Why is it hard to build a warp engine?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!"
Reporter: "Why is it hard to wipe your ass using the left hand?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!"
You get the idea...
What you want to optimize for is the money amount that you make at some quantile of the probablity distribution of the profits; say, the profits that are guaranteed in the best, say, 3 %, 5 %, 10 % or even 20 % of all possible outcomes. With a probablity of 97 % (if you choose the best 3 % of the outcomes), you won't make sufficient money if you attempt to cure cancer to be worth the risk, so the financial incentive is not there.
TLDR: Financial incentives do matter, but work differently from how many people think that they are structured.
Put another way,if I were trying to do a start up in this space I'd spend 50% of my budget on marketing 25% on a third world data labelling sweatshop, 20% on data pipeline engineering and 5% on sexy ml stuff.
When researchers want to publish a paper, are they going to pay extra money for extra difficulty in publishing their paper? No, they'll just use whatever toy environment is free or already established and get that paper published!