Hacker News Reader

Comment by apengwin | Hacker News Reader

That biases admissions for people who can afford extra tutoring and preparation for the test.

Yes. Case in point, the daughter of the lady in my town who ran the SAT and SSAT prep courses got all 800s. ALL 800s. She was smart, but not that smart. It is possible to be completely prepared for the SATs.

A perfect score on the modern SAT is not very impressive. The bar has been lowered considerably in the last 40 years.

[edit: seems I was very clearly wrong about this]

Ah, there it is. The "this generation has it so much easier" mentality.

Got any citations to back this up? Or just feel like grousing about millennials?

Nope. Seems I was wrong.

With respect, what in the world are you talking about? A casual search on the topic shows that 2 million people took the SAT last year. Of those 2 million, fewer than 500 achieved a perfect score.

Being part of the 0.025% isn't impressive to you?

Yes. False memory on my part it seems. I was very clearly wrong. The current SAT is clearly selective enough for my very popular scheme.

Isn't the difficulty tuned to produce a histogram close to that of a normal distribution? Yes it's true that the contents of the test have changed (removing analogies, and then recently circa 2016 revamping the writing/reading sections to be more like the ACT) but fundamentally it doesn't really look that different to me. In fact if anything now the reading section relies more on understanding and analysis of passages rather than just memorizing words and definitions which if anything seems like slightly raising the bar.

Goodhart's law: "Once a metric becomes a target, it ceases to be a good measure."

Far from universally true. IQ tests like the SAT, GRE, GMAT etc. are all designed to be as close to ungameable as possible. There’s a limited amount they can do but the limit is very high indeed. The major, and unavoidable defect is the practice effect. The more you do something the better you get at it. But IQ tests are like running or weightlifting, there are beginner gains, there are tough gains that are ground out with a great deal of effort and there’s a limit you’re not going to surpass no matter how hard you try. O matter how many steroids I take or how hard I train I’m not going to beat Usain Bolt in the 100m. I’m probably not going to beat the average NHL player.

Goodhart’s Law is the opposite of true in quality control. What gets measured gets managed. Manufacturing defects in semiconductors, automobiles and I presume, most everything else have been trending down for decades. Businesses have KPIs for employees for a reason. Monomaniacal focus over a long period on one measure may not be the best idea, but it’s a great tool to have.

This is a great example of Goodhart's Law, though. If nobody studied for the SAT, it might (arguably) work fine as a measurement of collegiate potential. But because it became the metric for collegiate success, and because the "practice effect" means that practicing makes you do better on the test, many people study hard for it and greatly increase their likely score. The folks who don't study as much or as effectively for it for whatever reason will not do as well, as so you're left with a measurement of whether people studied for the SAT and not whether the test taker would be likely to succeed at college. So a good metric becomes a bad metric because it's chosen as the metric.

Not saying it isn't a problem but those better off have the resources to be educated better before in addition to factors like nutrition. Essentially the advantages aren't bias but a byproduct. The way to fix it of course is to raise the floor.

Reply on news.ycombinator.com