datasets are proxies, measurements are fuzzy.
This is something that is drilled into students learning statistics and I can't for the life of me figure out why this changed with respect to ML (best guess is not requiring statistics courses).
Datasets are proxies: they represent the world, but aren't. They should generally be seen as narrow subsets too. Your dataset quality and type matter a lot! Things like medical image datasets also have tons of correlating factors that can easily invalidate all results without you being aware. There are simple datasets we use to prove a concept (toys, mnist, cifar, etc). There are large scale datasets that have internal inconsistencies (imagenet, flowers). There are huge datasets that haven't been properly filtered/deduplicated (LAION). (there are also just shitty datasets (HumanEval)) Thinking of datasets as proxies helps internalize the frustration that literally every production engineer faces (even outside ML and software): real world results are inconsistent with lab results. Dataset engineering is an underappreciated art that is extremely difficult. But everyone needs to internalize that datasets are just a map, not the territory, and your navigation will only be as good as the map (many are poorly drawn maps, often on purpose)
Measurements are fuzzy: Benchmarkism is running rampant in the ML world and it baffles me that a field who's bottomline objective deals with alignment isn't able to align how we evaluate ourselves. No measurement is perfect, many are far from them. You can train two language models to the same NLL and one might sample well and the other outputs garbage. You can train two image models to have identical FIDs and one samples clearly and the other doesn't. Likelihood also doesn't guarantee sharpness and I can go on. You must think about the limitations of your measurements and know them in depth. This also has seem to have gotten away from us and people just use the measurement libraries and call it that. We've reached a point where ImageNet classification accuracy has decoupled from downstream performance (object detection and segmentation), and things like this are confusing to production people because just taking the model with the highest score doesn't always result in the best performing model for their work (even before we consider things like throughput and memory usage). It is a Goodhart problem through and through.
ML is at a serious point where we've gotten away from our basic stats learning. That's going to pose a real danger to society, not AGI. It's like handing powertools to chimps (who don't know how powertools work), it won't end well. But that is happening because we've shifted focus to meet targets, not to measure our work. Targets are easy, science is hard. Unless we bring these nuances back to our evaluation of works then we are just handing powertools to chimps without any quality assurance.