Hacker News Reader

This is junk science that can best be used to reinforce existing stratification or create/justify a self-fulfilling prophecy of success. Best to ignore it and move on.

Their method would miss the likes of Blake, Van Gogh, Galileo, ...

SMPY identifies outlier ability via standardized testing taken before age 13 (for cohorts other than the very last one). All of Study 1 was selected this way. This may have some very loose correlation with wealth.

Is there a correlation between SMPY eligibility and getting into elite universities? Probably, but it doesn’t seem like an improper one. Is there a correlation between elite universities and life outcome? Sure.

Does that make the longitudinal study from age 11/12 into adulthood useless or content-free? Not in my opinion.

Only Study 2 was done on cohort 5, which was selected as grad students already at elite universities.

Cohort 4 was the only state university sample, but even that one used students in the summer programme (which is tied to wealth).

I also realized that SAT scores are heavily impacted by SAT specific training/tutoring that costs a reasonable amount (because as far as I can tell - as a foreigner - the SAT tests rely more on having been taught the exact way - and tricks - to answer SAT questions in the time available. Honestly if your test is good it should not have any real time pressure for the vast majority of participants)

> I also realized that SAT scores are heavily impacted by SAT specific training/tutoring

It's difficult to realize something that isn't true. The effects of training on SAT scores have always been negligible, less than the measurement error associated with taking the test one time.

Most people scoring 700+ on SAT-M experience no time pressure on the test. (That could be read as tautological, but I’m fairly convinced that there is no real time pressure that would dramatically shift the “SMPY/SET eligible” from not. The questions are straightforward and there’s more than ample time provided for those with a realistic chance of scoring 700+.)

The extent of my tutoring was a public school guidance counselor explaining to our whole class who was taking the test that there was statistically no penalty for guessing and how the “A is greater, B is greater, C means equal, D means indeterminate” questions worked. That was 30 minutes at most.

I got a 760 on SAT-M and I definitely felt tremendous time pressure. I am convinced I could have gotten 800 with no time limit. At the same time someone much smarter than me could probably get an 800 with half the time given.

My high school did intense drilling for SATs as an after school thing. The average kid in my class had a 180 point improvement versus the PSAT as a result.

High SAT scores are evidence of high SAT score.

ah yes, the exorbitant cost of $35 for an SAT prep textbook

Well, even better, you don’t even need to purchase them—most public libraries have SAT prep books available for borrowing.

When I was in high school, I would check out whichever ones were available at the time and would photocopy the whole book. I drilled myself on the tests until I scored 1600s every time. Shows how flawed they are as a testing vehicle.

That's not evidence the tests are flawed. I was just having a very similar conversation with a friend who aced the physics olympiad. She said, "it's too easy to game, all I had to do was try old tests on my own". Well, hundreds to thousands of others were putting in more hours doing the same thing, in addition to enrolling in expensive prep programs, and they didn't do as well as her.

That is evidence in favor of the test's quality, not against it. There's no paying to win.

That’s a good point. You’re right if the aim of the test is to measure information retention, and not raw intelligence.

Never mind what it measures. My main point is that whatever it is, it's not simply "dollars spent on prep". Good tests provide an opening for the less well-off to succeed, despite the more well-off doing everything they can to rig it in their favor. My parents (and by extension me) owe a lot to them.

The evidence says that SAT prep courses do not make much of a difference:

"From 1981 to 1990, three separate analyses of all the prior studies were published in peer-reviewed journals. They found a coaching effect of 9 to 25 points on the SAT Verbal and of 15 to 25 points on the SAT Math. In 2004, Derek Briggs, using the National Education Longitudinal Study of 1988, found effects of 3 to 20 points for the SAT Verbal and 10 to 28 points for the SAT Math. Donald Powers and Donald Rock, using a nationally representative sample of students who took the SAT after its revisions in the mid-1990s, found an average coaching effect of 6 to 12 points on the SAT Verbal and 13 to 18 points on the SAT Math. Many studies tell nearly identical stories. On average, coaching raises scores by no more than a few dozen points, enough to sway college admissions in exceedingly few cases."

(from https://www.aei.org/articles/abolish-the-sat-2)

When I was tested in that first cohort, I received no coaching whatsoever. I was 12 years old.

> Ok so I see no controls for income/family wealth, etc

There is no reason to control for those, and doing so would be an 'Everest regression': "controlling for altitude, Mount Everest is actually warm today". What you are asking for is like reading about IQ and income and saying "ah, but did they control for education?" Controlling is not magic pixie dust.

> What you are asking for is like reading about IQ and income and saying "ah, but did they control for education?"

that thing that literally raises the measured IQ number?

you know that practice can raise your score, right

Right - but there is also a relationship between IQ and educational attainment. So controlling for it means that you're measuring something fundamentally different. That's fine, but you can't just control for things without having a valid theoretical construct to justify it. (Well, OK, you can, and people do, and it's great for getting p<0.05 if you don't care about getting meaningful answers about the underlying questions.)

it's a triviality to say that you can control for educational attainment (and opportunities for education as well as class) when accounting for differences in IQ: it's hard to do in practice because social science is complicated and requires ACTUALLY DOING THE SCIENCE so gwern's glib comment is just wrong here

clearly we're talking about the impact on IQ of whatever factors we make up to describe IQ, and obviously, and provenly, education (and therefore the ability to get education) is in play here, which gwern's glib rebuttal ignores

honestly it's disappointing when i see people with training in stats make these comments: I'm a fucking poet if I'm anything at all and yet I find myself drawn into these ridiculous discussions about stats (in which I have no formal training) where I feel like I'm one of the only people in the room who knows how to divy up numbers

also, to clarify things a bit: in my first comment i got a bit mixed up (fuck yeah lifelong learning disability) trying to make my point about lifelong attainment and the IQ analogy and made it sound like the paper in question was about IQ, which is incorrect.

The point stands however: controlling for educational background is a thing you can do in such a study as according to the factors I pointed to, and gwern's glib comment is wrong

No, your comment was glib. You suggest controlling for variables which are going to create overcontrolling and the sociologist's fallacy, and you know perfectly well that family background is not independent of IQ and attenuates the relationship and your suggestion creates only an uninterpretable mishmash of a marginal effect which includes some causal effects but quite arbitrarily excludes others which run through controlled variables. I did not choose my example of education/IQ/income casually, but picked one which is very common in the literature which people commit because they think they are being rigorous when in fact they're only fucking up their own results and making them less meaningful. One's intelligence is not independent of one's family or genetic background! This matters to testing and things which can change! (Because it gives you residual confounding, if nothing else.) By the way, the proper way to do what you may or may not want to do (because your comments are so glib it's unclear to me what DAG you have in mind or what causal effect you want) is sibling controls, not family background controls, and you will be unhappy as to the results when they have been done.

This is a total nothingburger, a long paragraph attacking a strawman of an argument you haven't even solicited the details of

The burden is on the person saying that 'such-and-such a thing should be done' to explain themselves. You failed to do that, and you have still failed to do that, and you spend your comments complaining about being criticized, so your comments remain nothingburgers.

I mean jesus, of course family background is not independent of IQ, but assessments of family background can obviously be informative in understanding how we get from there to here. To say that, quote, "One's intelligence is not independent of one's family or genetic background" is a flaccid truism. Nonetheless particular comparisons within assessments of family background can obviously in principle be informative even if they are hard to do.

Your suggestion - with this fringe talk about "the sociologist's fallacy - is to eliminate any explanation from any study which does not assign success to invariant psychological metrics.

> I mean jesus, of course family background is not independent of IQ

Then what exactly do you think you are doing when you rant about how they didn't control for it?

> Your suggestion - with this fringe talk about "the sociologist's fallacy - is to eliminate any explanation from any study which does not assign success to invariant psychological metrics.

Nothing fringe about it, I'm afraid. This is well-understood in the statistics fields like causality you seem so loathe to actually learn about.

The point is: when you control for something and then go on to make causal claims, you are not making a minor tweak. You are assuming an extremely implausible causal model and making strong inferences from there about what are real causes and what are not. You have entirely failed to argue for a causal model which would justify simply controlling away family background etc, and it seems from how your comments handwave it away in favor of nothingburgers, you don't even understand what you're arguing for.

Really Gerard, are you going to lecture me on this? I am aware of practice, and more importantly, I am aware the IQ gains from education are hollow, from the Milwaukee Project on to Ritchie's paper on education increases. The practice that raises your score does not affect all subtests the same: increasing your vocab does not affect your backwards digit span or reaction time, or increase your longevity or income or...

And that's not relevant anyway. Are you trying to suggest people only get into SMPY because their parents made them practice the SAT? In the 1970s and 1980s, hardly anyone would have even heard of SMPY before their kids took the test in school.

you know as well as I do that zip codes are more heritable than IQ

and predict results pretty well too

No, I don't, because they're not: are identical twins more likely to live in the same zip code as fraternal twins? Seriously, think about it for a second. Or I'll put it this way: how much more likely are identical twins adopted out to live in the same zip code?

Nor does zip code predict remotely as well as IQ.

Are we just making up definitions of heritability now?

http://bactra.org/weblog/520.html

The way I read the study is that within the top N% group, higher scores predict "eminence". Not the top N compared to everyone else.

> If anything I see selection for wealth: the samples in one of the bigger studies selected from “the 15 best STEM graduate training programmed”, which is broadly a proxy for wealth.

Not everything is about wealth.

STEM graduate school doesn't select for wealth at all. Undergraduate admissions sometimes do, but essentially zero of the wealth-based admits major in STEM. In fact in my experience the effect is the exact opposite of what you say: STEM takes in whoever can do the work, regardless of wealth, and helps produce new wealth. In every institution I've been in, STEM people had the lowest average family wealth by far.

>Obviously we would generally expect some correlation between skill when younger to 35 years later

Though this is obvious, it wasn't obvious that the tests given were actually showing who had some amounts of skill. Though, this should more accurately be called "some amount of ability to become prominent in a field," as like you say things other than skill may cause both high scores and success later.

The justification for using the SAT-M in underage testing is in fact that: because the material would not have yet been covered in any classes, it tests ability to come up with a correct answer to a novel problem rather than skill. (Remember, educators have long been convinced that any kind of acceleration of gifted kids is poison, and SMPY's results have been a major reason that gifted kids can do any acceleration these days, so they're not just covering it in some accelerated math class.)

They could be covering the material with private tutors or similar outside the classroom experience. That's why we can't say for certain the test is measuring skill.

Uh oh, that seems to be rather out of date. When I took the SAT-M in 7th grade I recall having already seen harder versions of all the questions, because of math competitions. The same went for almost everyone I knew that took it.

FWIW I took it in the 80s (also around 7th grade though I forget exactly) and it was not like that for me. I wish I'd had more contact with real math that early -- school was such a waste of time.

When you took it, you weren't part of this SMPY cohort starting in the 1970s. Most of these kids certainly were not taking 'math competitions' or 'private tutors', they were languishing in regular classes, and that's why Stanley used the SAT-M.

It would be interesting to compare the trajectories of the bottom 5% of Harvard grads with the top 5% of UMass grads.

As a thought exercise, you can ask yourself: would you prefer that your doctor be from the former or the latter group?

Definitely the latter. Bottom 5% of Harvard are likely legacies with little motivation - and unlikely to get into medical school.