Most of the time, the surrogate metrics align with the unknown metrics when they are small, e.g. 80%. They can start to diverge at 90% for example. When I see people report a .5% improvement over a 98% accuracy and claim the statistical significance, I highly doubt about the real significance. The long tail is always longer than we thought. On the hand, there is no coincidence in the world. If some datapoints happen to be outliers that the metrics miscalculated, it might be that the problem isn't setup right. For example, missed some information in the data. In-person interview helps not just because people have a better judgement. It's because we can find out more details about that sample. The current AI/ML method lacks an active information seeking process. They don't have the freedom to explore the world beyond the defined state/reward space.