Hacker News Reader

Comment by solveit | Hacker News Reader

I'm specifically not talking about the SAT and ACT since they're widely used by elite universities. Those tests top out way too early and can't reliably distinguish between the top 1% and the top 0.1% (The fraction of perfect scorers is much smaller than 1% but the standard error of measurement is way too large). The exams I have in mind are those like the science olympiads and the Cambridge STEP. STEP in particular is basically used for this purpose (olympiads are a slightly different beast).

View on HN

Are you trying to get students that can pass specific kinds of tests or students that will be successful? The two aren't strongly correlated. I'd argue successful builds more prestige for the university. And let's be real, everyone knows these tests are purposefully tricky. In the real world people don't give you multiple choices and try to trick you into picking the wrong answer and then laugh at you when you picked the one try tried to get you to pick (politics aside). In classes, unless your professor is an asshole, they aren't trying to trick you. Success depends on a lot of factors well beyond what these tests measure. This is why SAT and ACT don't measure success or even academic success (as in they don't strongly correlate [0][1] with success. This source should decent correlation with first year grades but not second).

I'm not sure why one would even think an academic test could determine success. One of the biggest lies many of us have been told is that if you're just smart enough you'll be successful. Sorry, but that's not how it works.

[0] https://edpolicyinca.org/publications/predicting-college-suc...

[1] https://www.latimes.com/california/story/2019-12-22/grades-v...

At the extreme end of the spectrum with quant hedge funds this actually becomes somewhat true. They expect strong academic understanding of math and make a killing by hiring those who do.

I have some useful background here that leads me to strongly disagree with you. I did well at these international science olympiads in high school. To do that I studied for the particular shape of the exam questions. I became friends with many other people who were high performers at these olympiads. Later on I got a PhD from a prestigious institution and I am now a scientist at another one. Throughout this time, I was very involved in the pedagogical work at these institutions.

One very important piece of insight I gained throughout that time is that after you use some exam to get the top 25% (ish) percent of students, the exam performance really does not matter: it does not correlate with scientific output or originality of work. If anything, I had to unlearn some of the skills that made me good at olympiads because they were severely limiting my creativity. At an olympiad you know there is a solution, while true research problems might be unsolvable and need to be approached differently.

TL;DR: Exams (selective or not, hard or not) are great at giving you the top 20% of students, but they are inherently terrible at telling you who among the top 20% will be a productive scientist or engineer.

Your anecdotal evidence is contradicted by wider evidence.

In particular the famous Benbow study [1] that established that even among the top percentile of the population there are massive differences: the top quarter percentile (99.75-99.99) was 2-3 times more likely to have authored academic research later in life, and about 1.5x more likely to have attended postgrad education, to have gone for a STEM degree, to have gone for a PhD than the bottom quarter of the top percent (99.00-99.24).

[1] https://www.gwern.net/docs/iq/smpy/1992-benbow.pdf

This is interesting, thanks for sharing it! It would take me some time to read it, but given that you seem the have already looked into this, I was wondering whether you have suggestion for a wider set of studies, a meta-study maybe.

I am asking, because if I have to choose between my empirical observations (anecdotal as they are, given it is n=1 observers), and a single piece of research without followups, I do feel justified to stick to what I have seen myself. But I am open to be convinced otherwise.

To update on the sibling comment I posted, after reading this study, I think it is pretty flawed. They indeed have an interesting population they have chosen to test, but to claim that they are the 1% top performers is incredibly misleading. The other 99% of their peers were never tested in the same fashion, which leads to obvious sampling and bias problems. This special population was treated differently from the other 99% of students from the very start of the study, so there is no surprise that new internal dynamics will appear among them, simply by virtue of being observed and treated as a separate group.

To start testing the validity of the claims in this paper a study needs to be performed where the strength of effect has to be considered, when comparing a similar one-percenter group and a larger ten-percenter group. My prediction is that the strength of effect would show only negligible differences, confirming my hunch that the special treatment[1] is what created the new subdivision in this new group.

More reading material on this topic would certainly be interesting, if you have anything in mind.

[1]: The special treatment in this paper being telling a kid "you are only a 7th grader, but you are as smart as a high schooler".

In the US the appropriate exams would likely be the qualifiers for the International Olympiads.

It those were used for admissions, I wouldn’t be surprised if there was a huge market for answers (from somewhere in the production chain), stand-in test takers, etc.

Reply on news.ycombinator.com