Fantastic point. Another thing about academic ml models is that they're commonly evaluated on in-sample data. For example, medical AI devices are commonly evaluated on data coming from the same source/hospital that the training data comes from (see https://www-cdn.law.stanford.edu/wp-content/uploads/2021/04/...). This leads to over-confidence in the performance because of overfitting of batch effects.