Okay, so statistics demonstrate the SAT only correlates with the first year of college...great, what do you have to replace it with?
We work with what we have, and take the benefits it gives us.
Systems can always be gamed, and that's an unfortunate (fortunate for some, I guess) reality.
My personal threshold attempts to minimize false positives even to the extent that I will pass on people that would probably have done a good job. I tune it this way because I think that bad hires are one of the worst things that can happen to a team. But, each individual has to calibrate this for themselves.
I'm not looking to solve the problem, I'm looking to make the best possible decision based on the information I have available.
1. You need large samples of hires because you must know what features actually correspond with good performance
2. You need to have a near perfect understanding of statistics
Once Google and friends collected a large enough sample they concluded their puzzles are not a good indicator. Similarly, people tend to have bias towards certain skills without any statistical evidence that they are correlated with performance. And statistics is all about removing the bias, which has proven time and time again is very hard. Otherwise you would be just gambling on instinct, which at best is just a sanity check.