Sfm tests the calibration to the extreme and even single pixel calibration error will be highlighted as correlated reprojection error vector in most images. And that is assuming the bad calibration does not cause the system to fail outright. There is also the inbetween case where the system is only able to use small parts of the image.
The distortion field often visibly tells you if the estimation failed. It should be smooth, and monotonic, and you can draw it not just for where there are pixels but further and in higher resolution, and for regular lenses almost always highly symmetric. Looking at the distortion field you can see alot of problems that you could not otherwise. The most common problem is the monotonicity, since that constraint is very difficult to add to general optimizers. Since this means the distortion goes backwards, they are visible as sharp edges in the distortion field.