You can always compare two samples without controlling for confounding differences. However, you need to controlling for confounding differences when you want to explain the differences. When we say there is a difference in IQ between two groups, we are not saying the difference is genetic. The raw difference in IQ does have a meaning by itself.

I do agree with you that there are questions regarding to measurement issues and confounding issues. But it is hard to believe all the group differences can be explained away by solving these issues.