Lets take a small toy example from one that came up a few days ago, the model that could predict 'heart-disease' from one heartbeat. Their data set came from two different sources: their 'disease state' and 'null state' patients had their cardiograms recorded via different instruments. They did some statistical re-sampling to get the data to 'match', so, a skeptics flags should already be raised. Second major issue: without resampling, they had an effective N of 30; they made the decision to 'slice' every ones cardio gram into thousands of examples. The RNN (i think it was an RNN), obviously, will need thousands of examples to train. They then randomly sampled (according to their train/validation/test split) from this distribution of 'beats' to train their model.

So they didn't do anything 'wrong' according to what myself, and what I would assume yourself were told when we did this or that course in ML. But actually, from an experimental design POV, this is clearly going to overfit. Even disregarding the resampling of the original data (its own, very suspect issue), the slicing alone is enough to realize 'Ah. This is horse crap'. Think about it like this. Say its a 80:10:10 split. Take a random sample of n heartbeats, 3 heartbeats long, from n=100k heartbeats. What is the probability of a heartbeat in the training data set not being very close (in time proximity) to a heartbeat in the validation/ test set? I don't have the time to work it out on paper, but its most likely that anything in the test/ validation set will be temporally adjacent to something in the training set. The probability of anything in the test/ validation being very different from something in the training dataset (say, 3-4 beat sets away in either direction) is very very unlikely. The vast majority of data in the test/validation dataset will have two direct neighbors, both used in training, the next most populated class will have 1 neighbor in one direction, and the almost none will be separated at great distance (even 2-3 beats isolated from something in the training dataset).

This issue central here is the lack of independence in the experimental design. They've created quasi-independence in their sampling methodology, but at the end of the day, they've still only got an n of 30 (not 300,000-3,000,0000). I get it. For most cases, one almost always has to create a condition of quasi-independence in ones data. To get these algorithms to work, you need lots of data.

Knowledge of what ML is, how it works under the hood (a bit), and how to implement it (more imp. imo), all matter in this space. What matters more (imo), is the mindset that can be developed by doing science as an intellectual exercise. Its good science to remain skeptical and be adherent to the evidence rather than our assumptions.